Lost contact to LDAP

[Ruaraidh (Gabor)]

NethServer Version: release 7.9.2009 (final)
Module: LDAP

Hi Everyone,

Since today i can not logon to SOGo and Nextcloud.
Guacamole works fine, also Zoneminder.

Nextcloud prompts immediately an error at launch: “internal server error”
SOGo does let me put in the credentials, but then says: “wrong username or passord”

I looked into the Nextcloud logs, and found lot of exception with the error:“message”:“Lost connection to LDAP server.”

In the message logs in only find:
core sogo-backup.sh: <0x0x1519e50[LDAPSource]> <NSException: 0x1455b70> NAME:LDAPException REASON:operation bind failed: Invalid credentials (0x31) INFO:{“error_code” = 49; login =

and

WARNING: Method [public void org.apache.guacamole.auth.jdbc.connection.ConnectionDirectory.add(org.apache.guacamole.net.auth.Identifiable) throws org.apache.guacamole.GuacamoleException] is synthetic and is being intercepted by [org.mybatis.guice.transactional.TransactionalMethodInterceptor@49908839]. This could indicate a bug. The method may be intercepted twice, or may not be intercepted at all.

Could anyone point me to the right direction?

[mrmarkuz (Markus Neuberger)]

@Ruaraidh

Guacamole works fine, also Zoneminder.

Do you use AD users login in Guacamole or Zoneminder?

Do you use local or remote AD? If remote, maybe the remote user changed or is locked?

Is the Active Directory connection working? The following outputs AD entries:

account-provider-test

Is mysql/mariadb running?

SOGo:

systemctl status mysqld -l

Nextcloud:

systemctl status rh-mariadb105-mariadb@nextcloud -l

As regards Nextcloud, there’s also a wiki entry about troubleshooting.

[Ruaraidh (Gabor)]

Hi Markus,

Thank you very muh for the response.

You are right, Guacamole and Zoneminder works as i don’t use LDAP on them.
Here are the results of the commands:

account-provider-test
ldap_sasl_bind(SIMPLE): Can’t contact LDAP server (-1)

systemctl status mysqld -l
● mysqld.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor prese t: disabled)
Active: inactive (dead) since Thu 2022-03-17 08:57:44 GMT; 11h ago
Main PID: 1542 (code=exited, status=0/SUCCESS)
Mar 17 08:55:31 systemd[1]: Starting MariaDB database server…
Mar 17 08:55:31 mariadb-prepare-db-dir[1222]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
Mar 17 08:55:31 mariadb-prepare-db-dir[1222]: If this is not the case, make sure the /var/lib/mysql is empty before running mariadb-prepare-db-dir.
Mar 17 08:55:32 mysqld_safe[1542]: 220317 08:55:32 mysqld_safe Logging to ‘/var/log/mariadb/mariadb.log’.
Mar 17 08:55:32 mysqld_safe[1542]: 220317 08:55:32 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Mar 17 08:55:37 systemd[1]: Started MariaDB database server.

systemctl status rh-mariadb105-mariadb@nextcloud -l
● rh-mariadb105-mariadb@nextcloud.service - MariaDB 10.5 database server
Loaded: loaded (/etc/systemd/system/rh-mariadb105-mariadb@.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/rh-mariadb105-mariadb@nextcloud.service.d
└─nethserver.conf
Active: active (running) since Thu 2022-03-17 18:05:00 GMT; 2h 5min ago
Docs: man:mysqld(8)
systemd - MariaDB Knowledge Base
Process: 16606 ExecStartPost=/usr/bin/scl enable $RH_MARIADB105_SCLS_ENABLED – /opt/rh/rh-mariadb105/root/usr/libexec/mysql-check-upgrade --defaults-group-suffix=.%I (code=exited, status=0/SUCCESS)
Process: 16544 ExecStartPre=/usr/bin/scl enable $RH_MARIADB105_SCLS_ENABLED – /opt/rh/rh-mariadb105/root/usr/libexec/mysql-prepare-db-dir --defaults-group-suffix=.%I %n (code=exited, status=0/SUCCESS)
Process: 16512 ExecStartPre=/usr/bin/scl enable $RH_MARIADB105_SCLS_ENABLED – /opt/rh/rh-mariadb105/root/usr/libexec/mysql-check-socket --defaults-group-suffix=.%I (code=exited, status=0/SUCCESS)
Process: 16503 ExecStartPre=/usr/bin/scl enable $RH_MARIADB105_SCLS_ENABLED – /usr/bin/scl_enabled rh-mariadb105 (code=exited, status=0/SUCCESS)
Main PID: 16588 (mysqld)
Status: “Taking your SQL requests now…”
CGroup: /system.slice/system-rh\x2dmariadb105\x2dmariadb.slice/rh-mariadb105-mariadb@nextcloud.service
└─16588 /opt/rh/rh-mariadb105/root/usr/libexec/mysqld --defaults-group-suffix=.nextcloud --basedir=/opt/rh/rh-mariadb105/root/usr

Mar 17 18:05:00 systemd[1]: Starting MariaDB 10.5 database server…
Mar 17 18:05:00 scl[16544]: Database MariaDB is probably initialized in /var/opt/rh/rh-mariadb105/lib/mysql-nextcloud already, nothing is done.
Mar 17 18:05:00 scl[16544]: If this is not the case, make sure the /var/opt/rh/rh-mariadb105/lib/mysql-nextcloud is empty before running mysql-prepare-db-dir.
Mar 17 18:05:00 mysqld-scl-helper[16588]: 2022-03-17 18:05:00 0 [Note] /opt/rh/rh-mariadb105/root/usr/libexec/mysqld (mysqld 10.5.9-MariaDB) starting as process 16588 …
Mar 17 18:05:00 systemd[1]: Started MariaDB 10.5 database server.

Is the main issue that mysql is not working?

[Ruaraidh (Gabor)]

Trying to start the Service but it fails:

Mar 17 20:24:31 control-service: mysqld start
Mar 17 20:24:31 systemd: Starting MariaDB database server…
Mar 17 20:24:31 mariadb-prepare-db-dir: Socket file /var/lib/mysql/mysql.sock exists.
Mar 17 20:24:31 mariadb-prepare-db-dir: Is another MySQL daemon already running with the same unix socket?
Mar 17 20:24:31 systemd: mysqld.service: control process exited, code=exited status=1
Mar 17 20:24:31 systemd: Failed to start MariaDB database server.
Mar 17 20:24:31 systemd: Unit mysqld.service entered failed state.
Mar 17 20:24:31 systemd: mysqld.service failed.

[Ruaraidh (Gabor)]

Mysql is now up again after this and a reboot:

mv /var/lib/mysql/mysql.sock /var/lib/mysql/mysql.sock.bak

The LDAP test still fails though witht he same error

[mrmarkuz (Markus Neuberger)]

@Ruaraidh

The LDAP test still fails though witht he same error

Is the NSDC service started?

systemctl status nsdc -l

[Ruaraidh (Gabor)]

yes, that is running:

systemctl status nsdc -l
● nsdc.service - NethServer Domain Controller container
Loaded: loaded (/usr/lib/systemd/system/nsdc.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2022-03-17 20:44:50 GMT; 34min ago
Docs: man:systemd-nspawn(1)
Main PID: 1214 (systemd-nspawn)
Status: “Container running.”
Tasks: 32
Memory: 196.5M
CGroup: /machine.slice/nsdc.service
├─1214 /usr/bin/systemd-nspawn --quiet --keep-unit --boot --network-bridge=br0 --machine=nsdc --capability=CAP_SYS_TIME
├─1219 /usr/lib/systemd/systemd
└─system.slice
├─samba.service
│ ├─2716 /usr/sbin/samba -i --debug-stderr
│ ├─2760 /usr/sbin/samba -i --debug-stderr
│ ├─2761 /usr/sbin/samba -i --debug-stderr
│ ├─2762 /usr/sbin/samba -i --debug-stderr
│ ├─2763 /usr/sbin/samba -i --debug-stderr
│ ├─2764 /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
│ ├─2765 /usr/sbin/samba -i --debug-stderr
│ ├─2766 /usr/sbin/samba -i --debug-stderr
│ ├─2767 /usr/sbin/samba -i --debug-stderr
│ ├─2768 /usr/sbin/samba -i --debug-stderr
│ ├─2769 /usr/sbin/samba -i --debug-stderr
│ ├─2770 /usr/sbin/samba -i --debug-stderr
│ ├─2771 /usr/sbin/samba -i --debug-stderr
│ ├─2772 /usr/sbin/samba -i --debug-stderr
│ ├─2773 /usr/sbin/samba -i --debug-stderr
│ ├─2774 /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
│ ├─2775 /usr/sbin/samba -i --debug-stderr
│ ├─2776 /usr/sbin/samba -i --debug-stderr
│ ├─2804 /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
│ ├─2805 /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
│ ├─2810 /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
│ ├─3321 /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
│ ├─3344 /usr/sbin/samba -i --debug-stderr
│ ├─7449 /usr/sbin/samba -i --debug-stderr
│ └─7462 /usr/sbin/samba -i --debug-stderr
├─console-getty.service
│ └─2714 /sbin/agetty --noclear --keep-baud console 115200,38400,9600 vt220
├─ntpd.service
│ └─2731 /usr/sbin/ntpd -u ntp:ntp -g
├─systemd-logind.service
│ └─2710 /usr/lib/systemd/systemd-logind
├─dbus.service
│ └─2645 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
└─systemd-journald.service
└─2273 /usr/lib/systemd/systemd-journald

Mar 17 20:44:52 systemd-nspawn[1214]: [ OK ] Reached target Network.
Mar 17 20:44:52 systemd-nspawn[1214]: [ OK ] Started Samba domain controller daemon.
Mar 17 20:44:52 systemd-nspawn[1214]: [ OK ] Started Login Service.
Mar 17 20:44:52 systemd-nspawn[1214]: [ OK ] Started Network Time Service.
Mar 17 20:44:52 systemd-nspawn[1214]: [ OK ] Reached target Multi-User System.
Mar 17 20:44:52 systemd-nspawn[1214]: [ OK ] Reached target Graphical Interface.
Mar 17 20:44:52 systemd-nspawn[1214]: Starting Update UTMP about System Runlevel Changes…
Mar 17 20:44:52 systemd-nspawn[1214]: [ OK ] Started Update UTMP about System Runlevel Changes.
Mar 17 20:44:53 systemd-nspawn[1214]: CentOS Linux 7 (Core)
Mar 17 20:44:53 systemd-nspawn[1214]: Kernel 3.10.0-1160.59.1.el7.x86_64 on an x86_64

Going to User and Groups, it does allows me to create a user without error.

[mrmarkuz (Markus Neuberger)]

OK, so I assume account-provider-test is working now.

Are there still issues with Nextcloud or SOGo? Can you login with an existing user a new one?

[Ruaraidh (Gabor)]

@mrmarkuz

account-provider-test

No it does not unfortunately. It gives the same error, and the Nextcloud, SOGo issue remaind.
Running journalctl -M nsdc i see no error at all the past 2 days.

This is after latest reboot:

• Reboot –
  Mar 17 20:44:51 systemd-journal[14]: Runtime journal is using 8.0M (max allowed 693.0M, trying to leave 1.0G free of 6.7G available → current limit 693.0M).
  Mar 17 20:44:51 systemd-journal[14]: Journal started
  Mar 17 20:44:51 systemd[1]: Starting Flush Journal to Persistent Storage…
  Mar 17 20:44:51 systemd-journal[14]: Permanent journal is using 280.0M (max allowed 4.0G, trying to leave 4.0G free of 472.7G available → current limit 4.0G).
  Mar 17 20:44:51 systemd-journal[14]: Time spent on flushing to /var is 40.379ms for 4 entries.
  Mar 17 20:44:51 systemd[1]: Started Flush Journal to Persistent Storage.
  Mar 17 20:44:51 systemd[1]: Starting Create Volatile Files and Directories…
  Mar 17 20:44:52 systemd[1]: Started Create Volatile Files and Directories.
  Mar 17 20:44:52 systemd[1]: Starting Update UTMP about System Boot/Shutdown…
  Mar 17 20:44:52 systemd[1]: Started Update UTMP about System Boot/Shutdown.
  Mar 17 20:44:52 systemd[1]: Reached target System Initialization.
  Mar 17 20:44:52 systemd[1]: Listening on NSDC container remote command server.
  Mar 17 20:44:52 systemd[1]: Started Daily Cleanup of Temporary Directories.
  Mar 17 20:44:52 systemd[1]: Reached target Timers.
  Mar 17 20:44:52 systemd[1]: Listening on D-Bus System Message Bus Socket.
  Mar 17 20:44:52 systemd[1]: Reached target Sockets.
  Mar 17 20:44:52 systemd[1]: Reached target Basic System.
  Mar 17 20:44:52 systemd[1]: Started D-Bus System Message Bus.
  Mar 17 20:44:52 systemd[1]: Starting Network Service…
  Mar 17 20:44:52 systemd[1]: Starting Login Service…
  Mar 17 20:44:52 systemd[1]: Starting Network Time Service…
  Mar 17 20:44:52 systemd[1]: Starting Permit User Sessions…
  Mar 17 20:44:52 systemd[1]: Started Permit User Sessions.
  Mar 17 20:44:52 systemd[1]: Started Console Getty.
  Mar 17 20:44:52 systemd[1]: Reached target Login Prompts.
  Mar 17 20:44:52 systemd-networkd[20]: host0 : Cannot configure IPv4 forwarding for interface host0: Read-only file system
  Mar 17 20:44:52 systemd-networkd[20]: host0 : Cannot configure IPv6 forwarding for interface: Read-only file system
  Mar 17 20:44:52 systemd-networkd[20]: Enumeration completed
  Mar 17 20:44:52 systemd[1]: Started Network Service.
  Mar 17 20:44:52 systemd[1]: Reached target Network.
  Mar 17 20:44:52 systemd-networkd[20]: host0 : gained carrier
  Mar 17 20:44:52 systemd-networkd[20]: host0 : link configured
  Mar 17 20:44:52 systemd[1]: Started Samba domain controller daemon.
  Mar 17 20:44:52 systemd[1]: Started Login Service.
  Mar 17 20:44:52 systemd-logind[21]: New seat seat0.
  Mar 17 20:44:52 ntpd[22]: ntpd 4.2.6p5@1.2349-o Thu Aug 8 11:47:59 UTC 2019 (1)
  Mar 17 20:44:52 systemd[1]: Started Network Time Service.
  Mar 17 20:44:52 systemd[1]: Reached target Multi-User System.
  Mar 17 20:44:52 systemd[1]: Reached target Graphical Interface.
  Mar 17 20:44:52 systemd[1]: Starting Update UTMP about System Runlevel Changes…
  Mar 17 20:44:52 ntpd[26]: proto: precision = 0.050 usec
  Mar 17 20:44:52 ntpd[26]: 0.0.0.0 c01d 0d kern kernel time sync enabled
  Mar 17 20:44:52 systemd[1]: Started Update UTMP about System Runlevel Changes.
  Mar 17 20:44:52 systemd[1]: Startup finished in 1.198s.
  Mar 17 20:44:52 ntpd[26]: MS-SNTP signd operations currently block ntpd degrading service to all clients.
  Mar 17 20:44:52 samba[25]: samba version 4.9.18 started.
  Mar 17 20:44:52 samba[25]: Copyright Andrew Tridgell and the Samba Team 1992-2018
  Mar 17 20:44:53 samba[25]: binary_smbd_main: samba: using ‘standard’ process model
  Mar 17 20:44:53 winbindd[42]: [2022/03/17 20:44:53.439996, 0] …/source3/winbindd/winbindd_cache.c:3161(initialize_winbindd_cache)
  Mar 17 20:44:53 winbindd[42]: initialize_winbindd_cache: clearing cache and re-creating with version number 2
  Mar 17 20:44:53 winbindd[42]: [2022/03/17 20:44:53.447703, 0] …/lib/util/become_daemon.c:136(daemon_ready)
  Mar 17 20:44:53 winbindd[42]: daemon_ready: daemon ‘winbindd’ finished starting up and ready to serve connections
  Mar 17 20:44:53 smbd[32]: [2022/03/17 20:44:53.493391, 0] …/lib/util/become_daemon.c:136(daemon_ready)
  Mar 17 20:44:53 smbd[32]: daemon_ready: daemon ‘smbd’ finished starting up and ready to serve connections
  Mar 17 20:59:55 systemd[1]: Starting Cleanup of Temporary Directories…
  Mar 17 20:59:55 systemd[1]: Started Cleanup of Temporary Directories.
  Mar 17 21:23:27 systemd[1]: Created slice system-nsdc\x2drun.slice.
  Mar 17 21:23:27 systemd[1]: Started nsdc-run worker process.
  Mar 17 21:23:28 systemd[1]: Started nsdc-run worker process.
  Mar 17 21:23:29 systemd[1]: Started nsdc-run worker process.
  Mar 17 21:23:29 systemd[1]: Started nsdc-run worker process.
  Mar 17 21:23:29 systemd[1]: Started nsdc-run worker process.
  Mar 17 21:23:30 systemd[1]: Started nsdc-run worker process.

[Ruaraidh (Gabor)]

Nextcloud log:
“YjOuI4clAUwVRirdGvucMgAAAAQ”,“level”:3,“time”:“2022-03-17T21:54:45+00:00”,“remoteAddr”:“x.x.x.x”,“user”:"–",“app”:“index”,“method”:“GET”,“url”:"/nextcloud/index.php/login",“message”:“Lost connection to LDAP server.”,“userAgent”:"Mozilla/5.

SOGo Log
Mar 17 21:56:27 sogod [2912]: [ERROR] <0x0x55f3736e2ac0[LDAPSource]> <NSException: 0x55f373a48720> NAME:LDAPException REASON:operation bind failed: Can’t contact LDAP server (0xFFFFFFFF) INFO:{“error_code” = “-1”; login = “ldapservice@AD.Somedomain.com”; }

[mrmarkuz (Markus Neuberger)]

I don’t understand that you can create a user but don’t have access to AD…

Were there recent updates? You may check /var/log/yum.log

Do you get warnings in cockpit about not started services?

Does this one work? It should display AD settings:

account-provider-test dump

Let’s check sssd config:

config show sssd

Is this the same password as above?

cat /var/lib/nethserver/secrets/ldapservice

Let’s try to reconfigure Nextcloud and SOGo:

signal-event nethserver-nextcloud-update

signal-event nethserver-sogo-update

[Ruaraidh (Gabor)]

@mrmarkuz

Were there recent updates? You may check /var/log/yum.log

The last updates were on 28th. I ran some updates this morning after the issue, in the hope it will fix something magically.

@mrmarkuz

Do you get warnings in cockpit about not started services?

No Warnings at all. I use still the old interface, but logging in to cockpit, shows no errors either.

@mrmarkuz

account-provider-test dump

account-provider-test dump
{
“BindDN” : “ldapservice@AD.fake.fake”,
“LdapURI” : “ldaps://ad.fake.fake”,
“DiscoverDcType” : “dns”,
“StartTls” : “”,
“port” : 636,
“host” : “ad.fake.fake”,
“isAD” : “1”,
“isLdap” : “”,
“UserDN” : “dc=ad,dc=fake,dc=fake”,
“GroupDN” : “dc=ad,dc=fake,dc=fake”,
“BindPassword” : “xxxxxxxxxxxxx”,
“BaseDN” : “dc=ad,dc=fake,dc=fake”,
“LdapUriDn” : “ldap:///dc%3Dad%2Cdc%3Dfake%2Cdc%3Dfake”
}

@mrmarkuz

config show sssd

config show sssd
sssd=service
AdDns=x.x.x.x
BindDN=ldapservice@AD.fake.fake
BindPassword=samepasswordasbefore
DiscoverDcType=dns
LdapURI=
Provider=ad
Realm=AD.fake.fake
ShellOverrideStatus=disabled
Workgroup=fake
status=enabled

@mrmarkuz

cat /var/lib/nethserver/secrets/ldapservice

Yes, same password.

@mrmarkuz

Let’s try to reconfigure Nextcloud and SOGo:

signal-event nethserver-nextcloud-update

signal-event nethserver-sogo-update

Ran those and finished without error, but same issue remained.
Though i was able to create the test user, i can not login with that account either. In SOGo it says the credentials are incorrect, and Nextcloud still internal error.

[Ruaraidh (Gabor)]

Stupid question, but should startls be disabled with port 636?

[mrmarkuz (Markus Neuberger)]

The only stupid questions are the ones that are not asked.

Yes, please disable STARTTLS when using ldaps, that could be the issue…but it seems already disabled.

[Ruaraidh (Gabor)]

Damn, startls was already disabled

[mrmarkuz (Markus Neuberger)]

To use ldapsearch to check AD see the wiki or following thread:

How to (re)enable TLS/STARTTLS in Nethserver AD

That’s true. With StartTLS it’s just ldap:// too, To test starttls: ldapsearch -Z -D CN=admin,CN=Users,DC=ad,DC=domain,DC=tld -w <PASSWORD> -b CN=Users,DC=ad,DC=domain,DC=tld -h <NSDC IP> To test ssl: ldapsearch -D CN=admin,CN=Users,DC=ad,DC=domain,DC=tld -w <PASSWORD> -b CN=Users,DC=ad,DC=domain,DC=tld -H ldaps://ad.domain.tld

@Ruaraidh

DiscoverDcType=dns
LdapURI=

I have following settings:

DiscoverDcType=ldapuri
LdapURI=ldaps://nsdc-testserver.ad.domain.local

Do you use an internal DNS server that maybe doesn’t point to the AD anymore?

Can you ping the NSDC IP address?

[Ruaraidh (Gabor)]

Hi Markus,

I am not really sure what exactly was the issue, but there was a rouge DHCP on the network. Once i did shut down that and rebooted the Nethserver the issue went away.
Thank you very much Markus to point me to the right direction and most of all staying on it even so late in the evening