SOGO (and AD LDAP clients) not working after upgrade

indra · 2018-05-14 04:51:18 UTC

Can’t login to SOGO, it says invalid username or password.
When I go to users and groups on the webadmin there are no users listed and I get a red warning message on top : AccountProvider_Error_82

On the command when I try /usr/libexec/nethserver/list-users it says :
kinit: Cannot contact any KDC for realm ‘CHANGED-DOMAIN.COM’ while getting initial credentials
(82) GSSAPI Error (init): Unspecified GSS failure. Minor code may provide more information
Ticket expired

/var/log/messages keeps repeating this

May 14 06:47:01 ganesha [sssd[ldap_child[29987]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'. Unable to create GSSAPI-encrypted LDAP connection.
May 14 06:47:01 ganesha [sssd[ldap_child[29987]]]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'
May 14 06:47:11 ganesha [sssd[ldap_child[30040]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'. Unable to create GSSAPI-encrypted LDAP connection.
May 14 06:47:11 ganesha [sssd[ldap_child[30040]]]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'
May 14 06:47:11 ganesha [sssd[ldap_child[30043]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'. Unable to create GSSAPI-encrypted LDAP connection.
May 14 06:47:11 ganesha [sssd[ldap_child[30043]]]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'
May 14 06:47:16 ganesha dbus[1616]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service'
May 14 06:47:16 ganesha dbus-daemon: dbus[1616]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service'
May 14 06:47:16 ganesha systemd: Starting Time & Date Service...
May 14 06:47:16 ganesha dbus[1616]: [system] Successfully activated service 'org.freedesktop.timedate1'
May 14 06:47:16 ganesha dbus-daemon: dbus[1616]: [system] Successfully activated service 'org.freedesktop.timedate1'
May 14 06:47:16 ganesha systemd: Started Time & Date Service.
May 14 06:47:34 ganesha nfsidmap[30595]: nss_getpwnam: name 'admin@CHANGED-DOMAIN' does not map into domain 'noble-house.tk'
May 14 06:47:34 ganesha nfsidmap[30598]: nss_name_to_gid: name 'users@CHANGED-DOMAIN' does not map into domain 'noble-house.tk'
May 14 06:47:49 ganesha nmbd[2785]: [2018/05/14 06:47:49.738891,  0] ../source3/libsmb/nmblib.c:873(send_udp)
May 14 06:47:49 ganesha nmbd[2785]:  Packet send failed to 192.168.122.255(137) ERRNO=Operation not permitted
May 14 06:47:49 ganesha nmbd[2785]: [2018/05/14 06:47:49.739054,  0] ../source3/nmbd/nmbd_packets.c:179(send_netbios_packet)
May 14 06:47:49 ganesha nmbd[2785]:  send_netbios_packet: send_packet() to IP 192.168.122.255 port 137 failed
May 14 06:47:49 ganesha nmbd[2785]: [2018/05/14 06:47:49.739106,  0] ../source3/nmbd/nmbd_namequery.c:245(query_name)
May 14 06:47:49 ganesha nmbd[2785]:  query_name: Failed to send packet trying to query name NOBLE-HOUSE<1d>
May 14 06:48:01 ganesha systemd: Created slice User Slice of asterisk.
May 14 06:48:01 ganesha systemd: Starting User Slice of asterisk.
May 14 06:48:01 ganesha systemd: Started Session 50 of user asterisk.
May 14 06:48:01 ganesha systemd: Starting Session 50 of user asterisk.
May 14 06:48:01 ganesha systemd: Created slice User Slice of sogo.
May 14 06:48:01 ganesha systemd: Starting User Slice of sogo.
May 14 06:48:01 ganesha systemd: Started Session 51 of user sogo.
May 14 06:48:01 ganesha systemd: Starting Session 51 of user sogo.
May 14 06:48:01 ganesha systemd: Removed slice User Slice of asterisk.
May 14 06:48:01 ganesha systemd: Stopping User Slice of asterisk.
May 14 06:48:01 ganesha systemd: Removed slice User Slice of sogo.
May 14 06:48:01 ganesha systemd: Stopping User Slice of sogo.
May 14 06:48:01 ganesha [sssd[ldap_child[31200]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'. Unable to create GSSAPI-encrypted LDAP connection.
May 14 06:48:01 ganesha [sssd[ldap_child[31200]]]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'
May 14 06:48:38 ganesha [sssd[ldap_child[31815]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'. Unable to create GSSAPI-encrypted LDAP connection.
May 14 06:48:38 ganesha [sssd[ldap_child[31815]]]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'
May 14 06:48:38 ganesha [sssd[ldap_child[31818]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'. Unable to create GSSAPI-encrypted LDAP connection.
May 14 06:48:38 ganesha [sssd[ldap_child[31818]]]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'

stephdl · 2018-05-14 06:19:14 UTC

Did you upgrade to centos7.5 ?

indra · 2018-05-14 06:22:13 UTC

No just the regular update from the software center on the webadmin.
I have a Crostino subscription so I thought it was safe, usually I take a snapshot before but this time, of course, I did not

I pasted the message log from the upgrade here : https://paste.ee/p/0viWB

indra · 2018-05-14 08:21:42 UTC

Also nextcloud and xmpp are not available, please help

stephdl · 2018-05-14 08:41:39 UTC

Yes not relative to sogo, you cannot authenticate any ldap client

indra · 2018-05-14 08:43:28 UTC

Yes, so it seems. How should I best proceed to get it fixed ?
Any logs I should look into or commands I could try to pinpoint the problem, and more importantly find a solution.

stephdl · 2018-05-14 08:49:57 UTC

When you look to the line 400 of your pastebin, you can see that the event nethserver-dc-update upgraded the sambaAD VM to centos7.5.

If I understand correctly this could be the key of the problem

@davidep do i’m wrong ?

indra · 2018-05-14 08:52:35 UTC

What did it do that for , can and should I revert that upgrade?

stephdl · 2018-05-14 08:54:19 UTC

Keep calm, how many users have you on your system ?

indra · 2018-05-14 08:58:39 UTC

About 40 to 50 accounts of which like some 20 active users.

indra · 2018-05-14 09:00:41 UTC

Downgrade seems to be an option, but waiting on your reply

]# yum downgrade nethserver-dc.x86_64 
Loaded plugins: changelog, fastestmirror, nethserver_events
sb-base                                                                                                                  | 3.6 kB  00:00:00     
sb-centos-sclo-rh                                                                                                        | 3.0 kB  00:00:00     
sb-centos-sclo-sclo                                                                                                      | 2.9 kB  00:00:00     
sb-epel                                                                                                                  | 4.7 kB  00:00:00     
sb-extras                                                                                                                | 3.4 kB  00:00:00     
sb-nethserver-base                                                                                                       | 2.9 kB  00:00:00     
sb-nethserver-updates                                                                                                    | 4.1 kB  00:00:00     
sb-updates                                                                                                               | 3.4 kB  00:00:00     
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package nethserver-dc.x86_64 0:1.4.5-1.ns7 will be a downgrade
---> Package nethserver-dc.x86_64 0:1.5.0-1.ns7 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================
 Package                           Arch                       Version                           Repository                                 Size
================================================================================================================================================
Downgrading:
 nethserver-dc                     x86_64                     1.4.5-1.ns7                       sb-nethserver-updates                      13 M

Transaction Summary
================================================================================================================================================
Downgrade  1 Package

Total download size: 13 M
Is this ok [y/d/N]: n
Exiting on user command

davidep · 2018-05-14 11:48:26 UTC

I’m looking at the attached log…

indra · 2018-05-14 11:49:21 UTC

Thank you, we really need the SOGo calendar functionality.

davidep · 2018-05-14 12:28:20 UTC

So, all LDAP clients fail to connect with AD LDAP service: is it running? Any error in journal?

account-provider-test dump
ping $(config getprop nsdc IpAddress)
systemctl status nsdc
journalctl nsdc
journalctl -M nsdc

If the nsdc service is stopped try with:

systemctl start nsdc

…and see if any error occurs in journals

https://paste.ee/p/0viWB#s=0&l=439

It seems your system was updated from sb-* repositories as expected. However for a reason that needs to be dug more deeply the NSDC chroot was updated against 7.5 repositories.

So your NethServer is still at 7.4. Only nsdc chroot is rebased on 7.5.

indra · 2018-05-14 12:42:06 UTC

Everything seems to function, only journalctl nsdc returns :

Failed to add match 'nsdc': Invalid argument
Failed to add filters: Invalid argument

First three commands seems to return valid responses and last one shows many many pages of logs.
I’ll try to anonymize some and post them for you.

indra · 2018-05-14 12:57:04 UTC

~]# systemctl status nsdc
● nsdc.service - NethServer Domain Controller container
Loaded: loaded (/usr/lib/systemd/system/nsdc.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2018-05-14 05:27:08 CEST; 9h ago
Docs: man:systemd-nspawn(1)
Main PID: 2825 (systemd-nspawn)
Status: "Container running."
CGroup: /machine.slice/nsdc.service
├─2825 /usr/bin/systemd-nspawn --quiet --keep-unit --boot --network-bridge=br0 --machine=nsdc --capability=CAP_SYS_TIME
├─2829 /usr/lib/systemd/systemd
└─system.slice
├─samba.service
│ ├─5896 /usr/sbin/samba -i --debug-stderr
│ ├─6141 /usr/sbin/samba -i --debug-stderr
│ ├─6142 /usr/sbin/samba -i --debug-stderr
│ ├─6143 /usr/sbin/samba -i --debug-stderr
│ ├─6144 /usr/sbin/samba -i --debug-stderr
│ ├─6145 /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
│ ├─6146 /usr/sbin/samba -i --debug-stderr
│ ├─6147 /usr/sbin/samba -i --debug-stderr
│ ├─6148 /usr/sbin/samba -i --debug-stderr
│ ├─6150 /usr/sbin/samba -i --debug-stderr
│ ├─6151 /usr/sbin/samba -i --debug-stderr
│ ├─6152 /usr/sbin/samba -i --debug-stderr
│ ├─6153 /usr/sbin/samba -i --debug-stderr
│ ├─6154 /usr/sbin/samba -i --debug-stderr
│ ├─6155 /usr/sbin/samba -i --debug-stderr
│ ├─6158 /usr/sbin/samba -i --debug-stderr
│ ├─6159 /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
│ ├─6166 /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
│ ├─6168 /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
│ ├─6169 /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
│ └─6175 /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
├─console-getty.service
│ └─5854 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
├─systemd-logind.service
│ └─5818 /usr/lib/systemd/systemd-logind
├─dbus.service
│ └─5623 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
├─ntpd.service
│ └─5731 /usr/sbin/ntpd -u ntp:ntp -g
└─systemd-journald.service
└─5515 /usr/lib/systemd/systemd-journald

May 14 05:27:14 nethserver.ourdomain.com systemd-nspawn[2825]: [  OK  ] Started Network Service.
May 14 05:27:14 nethserver.ourdomain.com systemd-nspawn[2825]: [  OK  ] Reached target Network.
May 14 05:27:15 nethserver.ourdomain.com systemd-nspawn[2825]: [  OK  ] Started Samba domain controller daemon.
May 14 05:27:15 nethserver.ourdomain.com systemd-nspawn[2825]: Starting Samba domain controller daemon...
May 14 05:27:15 nethserver.ourdomain.com systemd-nspawn[2825]: [  OK  ] Reached target Multi-User System.
May 14 05:27:15 nethserver.ourdomain.com systemd-nspawn[2825]: [  OK  ] Reached target Graphical Interface.
May 14 05:27:15 nethserver.ourdomain.com systemd-nspawn[2825]: Starting Update UTMP about System Runlevel Changes...
May 14 05:27:15 nethserver.ourdomain.com systemd-nspawn[2825]: [  OK  ] Started Update UTMP about System Runlevel Changes.
May 14 05:27:16 nethserver.ourdomain.com systemd-nspawn[2825]: CentOS Linux 7 (Core)
May 14 05:27:16 nethserver.ourdomain.com systemd-nspawn[2825]: Kernel 3.10.0-693.21.1.el7.x86_64 on an x86_64

and an excerpt from the journalctl -M nsdc

davidep · 2018-05-14 13:12:27 UTC

Sorry it was:

journalctl -u nsdc

I’ve tried to reproduce the unexpected RPM installation from 7.5 repositories:

On installation, NSDC packages were downloaded from 7.4 as expected.
However if I run the nsdc update procedure upstream repositories (7.5) are used.

It does not seem a problem, as the nsdc process is running. However I found an issue with the KDC process of Samba DC:

journalctl -M nsdc -u samba

One of the lines:

May 14 14:59:38 nsdc-vm7.ad.dpnet.nethesis.it samba[158]: task_server_terminate: [kdc: krb5_init_context failed]

Do you have the same?

Yes!

https://paste.ee/p/q7swx#s=0&l=411

davidep · 2018-05-14 13:34:29 UTC

The workaround to the bad Samba DC startup, is edit/fixing krb5.conf under the nsdc chroot. Just run this command:

cp -vfp /var/lib/machines/nsdc/var/lib/samba/private/krb5.conf /var/lib/machines/nsdc/etc/krb5.conf

Then stop and start the samba DC process

systemctl -M nsdc stop samba
systemctl -M nsdc start samba

Now I must investigate why the nsdc container wants to download from 7.5 repositories

indra · 2018-05-14 14:31:07 UTC

Yes that solved it, thank you very much!

davidep · 2018-05-15 13:00:11 UTC

Filed an issue, the fix is available from nethserver-testing repo, but requires the following command to be applied:

 signal-event nethserver-dc-upgrade