SOGO (and AD LDAP clients) not working after upgrade

[indra]

Can’t login to SOGO, it says invalid username or password.
When I go to users and groups on the webadmin there are no users listed and I get a red warning message on top : AccountProvider_Error_82

On the command when I try /usr/libexec/nethserver/list-users it says :
kinit: Cannot contact any KDC for realm ‘CHANGED-DOMAIN.COM’ while getting initial credentials
(82) GSSAPI Error (init): Unspecified GSS failure. Minor code may provide more information
Ticket expired

/var/log/messages keeps repeating this

May 14 06:47:01 ganesha [sssd[ldap_child[29987]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'. Unable to create GSSAPI-encrypted LDAP connection.
May 14 06:47:01 ganesha [sssd[ldap_child[29987]]]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'
May 14 06:47:11 ganesha [sssd[ldap_child[30040]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'. Unable to create GSSAPI-encrypted LDAP connection.
May 14 06:47:11 ganesha [sssd[ldap_child[30040]]]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'
May 14 06:47:11 ganesha [sssd[ldap_child[30043]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'. Unable to create GSSAPI-encrypted LDAP connection.
May 14 06:47:11 ganesha [sssd[ldap_child[30043]]]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'
May 14 06:47:16 ganesha dbus[1616]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service'
May 14 06:47:16 ganesha dbus-daemon: dbus[1616]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service'
May 14 06:47:16 ganesha systemd: Starting Time & Date Service...
May 14 06:47:16 ganesha dbus[1616]: [system] Successfully activated service 'org.freedesktop.timedate1'
May 14 06:47:16 ganesha dbus-daemon: dbus[1616]: [system] Successfully activated service 'org.freedesktop.timedate1'
May 14 06:47:16 ganesha systemd: Started Time & Date Service.
May 14 06:47:34 ganesha nfsidmap[30595]: nss_getpwnam: name 'admin@CHANGED-DOMAIN' does not map into domain 'noble-house.tk'
May 14 06:47:34 ganesha nfsidmap[30598]: nss_name_to_gid: name 'users@CHANGED-DOMAIN' does not map into domain 'noble-house.tk'
May 14 06:47:49 ganesha nmbd[2785]: [2018/05/14 06:47:49.738891,  0] ../source3/libsmb/nmblib.c:873(send_udp)
May 14 06:47:49 ganesha nmbd[2785]:  Packet send failed to 192.168.122.255(137) ERRNO=Operation not permitted
May 14 06:47:49 ganesha nmbd[2785]: [2018/05/14 06:47:49.739054,  0] ../source3/nmbd/nmbd_packets.c:179(send_netbios_packet)
May 14 06:47:49 ganesha nmbd[2785]:  send_netbios_packet: send_packet() to IP 192.168.122.255 port 137 failed
May 14 06:47:49 ganesha nmbd[2785]: [2018/05/14 06:47:49.739106,  0] ../source3/nmbd/nmbd_namequery.c:245(query_name)
May 14 06:47:49 ganesha nmbd[2785]:  query_name: Failed to send packet trying to query name NOBLE-HOUSE<1d>
May 14 06:48:01 ganesha systemd: Created slice User Slice of asterisk.
May 14 06:48:01 ganesha systemd: Starting User Slice of asterisk.
May 14 06:48:01 ganesha systemd: Started Session 50 of user asterisk.
May 14 06:48:01 ganesha systemd: Starting Session 50 of user asterisk.
May 14 06:48:01 ganesha systemd: Created slice User Slice of sogo.
May 14 06:48:01 ganesha systemd: Starting User Slice of sogo.
May 14 06:48:01 ganesha systemd: Started Session 51 of user sogo.
May 14 06:48:01 ganesha systemd: Starting Session 51 of user sogo.
May 14 06:48:01 ganesha systemd: Removed slice User Slice of asterisk.
May 14 06:48:01 ganesha systemd: Stopping User Slice of asterisk.
May 14 06:48:01 ganesha systemd: Removed slice User Slice of sogo.
May 14 06:48:01 ganesha systemd: Stopping User Slice of sogo.
May 14 06:48:01 ganesha [sssd[ldap_child[31200]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'. Unable to create GSSAPI-encrypted LDAP connection.
May 14 06:48:01 ganesha [sssd[ldap_child[31200]]]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'
May 14 06:48:38 ganesha [sssd[ldap_child[31815]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'. Unable to create GSSAPI-encrypted LDAP connection.
May 14 06:48:38 ganesha [sssd[ldap_child[31815]]]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'
May 14 06:48:38 ganesha [sssd[ldap_child[31818]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'. Unable to create GSSAPI-encrypted LDAP connection.
May 14 06:48:38 ganesha [sssd[ldap_child[31818]]]: Cannot contact any KDC for realm 'CHANGED-DOMAIN.COM'

[stephdl]

Did you upgrade to centos7.5 ?

[indra]

No just the regular update from the software center on the webadmin.
I have a Crostino subscription so I thought it was safe, usually I take a snapshot before but this time, of course, I did not

I pasted the message log from the upgrade here : https://paste.ee/p/0viWB

[indra]

Also nextcloud and xmpp are not available, please help

[stephdl]

Yes not relative to sogo, you cannot authenticate any ldap client

[indra]

Yes, so it seems. How should I best proceed to get it fixed ?
Any logs I should look into or commands I could try to pinpoint the problem, and more importantly find a solution.

[stephdl]

When you look to the line 400 of your pastebin, you can see that the event nethserver-dc-update upgraded the sambaAD VM to centos7.5.

If I understand correctly this could be the key of the problem

@davidep do i’m wrong ?

[indra]

What did it do that for , can and should I revert that upgrade?

[stephdl]

Keep calm, how many users have you on your system ?

[indra]

About 40 to 50 accounts of which like some 20 active users.

[indra]

Downgrade seems to be an option, but waiting on your reply

]# yum downgrade nethserver-dc.x86_64 
Loaded plugins: changelog, fastestmirror, nethserver_events
sb-base                                                                                                                  | 3.6 kB  00:00:00     
sb-centos-sclo-rh                                                                                                        | 3.0 kB  00:00:00     
sb-centos-sclo-sclo                                                                                                      | 2.9 kB  00:00:00     
sb-epel                                                                                                                  | 4.7 kB  00:00:00     
sb-extras                                                                                                                | 3.4 kB  00:00:00     
sb-nethserver-base                                                                                                       | 2.9 kB  00:00:00     
sb-nethserver-updates                                                                                                    | 4.1 kB  00:00:00     
sb-updates                                                                                                               | 3.4 kB  00:00:00     
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package nethserver-dc.x86_64 0:1.4.5-1.ns7 will be a downgrade
---> Package nethserver-dc.x86_64 0:1.5.0-1.ns7 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================
 Package                           Arch                       Version                           Repository                                 Size
================================================================================================================================================
Downgrading:
 nethserver-dc                     x86_64                     1.4.5-1.ns7                       sb-nethserver-updates                      13 M

Transaction Summary
================================================================================================================================================
Downgrade  1 Package

Total download size: 13 M
Is this ok [y/d/N]: n
Exiting on user command

[davidep]

I’m looking at the attached log…

[indra]

Thank you, we really need the SOGo calendar functionality.

[davidep]

May 14 06:47:01 ganesha [sssd[ldap_child[29987]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm ‘CHANGED-DOMAIN.COM’. Unable to create GSSAPI-encrypted LDAP connection.

So, all LDAP clients fail to connect with AD LDAP service: is it running? Any error in journal?

account-provider-test dump
ping $(config getprop nsdc IpAddress)
systemctl status nsdc
journalctl nsdc
journalctl -M nsdc

If the nsdc service is stopped try with:

systemctl start nsdc

…and see if any error occurs in journals

I pasted the message log from the upgrade here : https://paste.ee/p/0viWB

https://paste.ee/p/0viWB#s=0&l=439

It seems your system was updated from sb-* repositories as expected. However for a reason that needs to be dug more deeply the NSDC chroot was updated against 7.5 repositories.

So your NethServer is still at 7.4. Only nsdc chroot is rebased on 7.5.

[indra]

Everything seems to function, only journalctl nsdc returns :

Failed to add match 'nsdc': Invalid argument
Failed to add filters: Invalid argument

First three commands seems to return valid responses and last one shows many many pages of logs.
I’ll try to anonymize some and post them for you.

[indra]

~]# systemctl status nsdc
● nsdc.service - NethServer Domain Controller container
Loaded: loaded (/usr/lib/systemd/system/nsdc.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2018-05-14 05:27:08 CEST; 9h ago
Docs: man:systemd-nspawn(1)
Main PID: 2825 (systemd-nspawn)
Status: "Container running."
CGroup: /machine.slice/nsdc.service
├─2825 /usr/bin/systemd-nspawn --quiet --keep-unit --boot --network-bridge=br0 --machine=nsdc --capability=CAP_SYS_TIME
├─2829 /usr/lib/systemd/systemd
└─system.slice
├─samba.service
│ ├─5896 /usr/sbin/samba -i --debug-stderr
│ ├─6141 /usr/sbin/samba -i --debug-stderr
│ ├─6142 /usr/sbin/samba -i --debug-stderr
│ ├─6143 /usr/sbin/samba -i --debug-stderr
│ ├─6144 /usr/sbin/samba -i --debug-stderr
│ ├─6145 /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
│ ├─6146 /usr/sbin/samba -i --debug-stderr
│ ├─6147 /usr/sbin/samba -i --debug-stderr
│ ├─6148 /usr/sbin/samba -i --debug-stderr
│ ├─6150 /usr/sbin/samba -i --debug-stderr
│ ├─6151 /usr/sbin/samba -i --debug-stderr
│ ├─6152 /usr/sbin/samba -i --debug-stderr
│ ├─6153 /usr/sbin/samba -i --debug-stderr
│ ├─6154 /usr/sbin/samba -i --debug-stderr
│ ├─6155 /usr/sbin/samba -i --debug-stderr
│ ├─6158 /usr/sbin/samba -i --debug-stderr
│ ├─6159 /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
│ ├─6166 /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
│ ├─6168 /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
│ ├─6169 /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
│ └─6175 /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
├─console-getty.service
│ └─5854 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
├─systemd-logind.service
│ └─5818 /usr/lib/systemd/systemd-logind
├─dbus.service
│ └─5623 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
├─ntpd.service
│ └─5731 /usr/sbin/ntpd -u ntp:ntp -g
└─systemd-journald.service
└─5515 /usr/lib/systemd/systemd-journald

May 14 05:27:14 nethserver.ourdomain.com systemd-nspawn[2825]: [  OK  ] Started Network Service.
May 14 05:27:14 nethserver.ourdomain.com systemd-nspawn[2825]: [  OK  ] Reached target Network.
May 14 05:27:15 nethserver.ourdomain.com systemd-nspawn[2825]: [  OK  ] Started Samba domain controller daemon.
May 14 05:27:15 nethserver.ourdomain.com systemd-nspawn[2825]: Starting Samba domain controller daemon...
May 14 05:27:15 nethserver.ourdomain.com systemd-nspawn[2825]: [  OK  ] Reached target Multi-User System.
May 14 05:27:15 nethserver.ourdomain.com systemd-nspawn[2825]: [  OK  ] Reached target Graphical Interface.
May 14 05:27:15 nethserver.ourdomain.com systemd-nspawn[2825]: Starting Update UTMP about System Runlevel Changes...
May 14 05:27:15 nethserver.ourdomain.com systemd-nspawn[2825]: [  OK  ] Started Update UTMP about System Runlevel Changes.
May 14 05:27:16 nethserver.ourdomain.com systemd-nspawn[2825]: CentOS Linux 7 (Core)
May 14 05:27:16 nethserver.ourdomain.com systemd-nspawn[2825]: Kernel 3.10.0-693.21.1.el7.x86_64 on an x86_64

and an excerpt from the journalctl -M nsdc

View Paste q7swx

[davidep]

Invalid argument

journalctl nsdc

Sorry it was:

journalctl -u nsdc

---

I’ve tried to reproduce the unexpected RPM installation from 7.5 repositories:

• On installation, NSDC packages were downloaded from 7.4 as expected.
• However if I run the nsdc update procedure upstream repositories (7.5) are used.

It does not seem a problem, as the nsdc process is running. However I found an issue with the KDC process of Samba DC:

journalctl -M nsdc -u samba

One of the lines:

May 14 14:59:38 nsdc-vm7.ad.dpnet.nethesis.it samba[158]: task_server_terminate: [kdc: krb5_init_context failed]

Do you have the same?

and an excerpt from the journalctl -M nsdc

Yes!

https://paste.ee/p/q7swx#s=0&l=411

[davidep]

kinit: Cannot contact any KDC for realm ‘CHANGED-DOMAIN.COM’ while getting initial credentials

May 14 14:59:38 nsdc-vm7.ad.dpnet.nethesis.it samba[158]: task_server_terminate: [kdc: krb5_init_context failed]

The workaround to the bad Samba DC startup, is edit/fixing krb5.conf under the nsdc chroot. Just run this command:

cp -vfp /var/lib/machines/nsdc/var/lib/samba/private/krb5.conf /var/lib/machines/nsdc/etc/krb5.conf

Then stop and start the samba DC process

systemctl -M nsdc stop samba
systemctl -M nsdc start samba

Now I must investigate why the nsdc container wants to download from 7.5 repositories

[indra]

Yes that solved it, thank you very much!

[davidep]

Filed an issue, the fix is available from nethserver-testing repo, but requires the following command to be applied:

 signal-event nethserver-dc-upgrade

Issue: Wrong YUM repository config for NSDC upgrade procedure