Ugh! I broke Samba Active Directory (or maybe not)

Support
,
#1

NethServer Version: NethServer release 7.3.1611 Final

Hi everyone,

I guess April is turning out not to be my month. I had Nethserver up and working just fine with the samba active directory accounts provider. I could create users, ibays, login and get my email, etc… Just waiting for dust to settle on Nextcloud 11 before setting that up and having my new server all configured and perfect.

So last night I did the update to Nextcloud 11. I could login as the “admin” user just fine, but when I tried to login as myself it wouldn’t work. So I poked around in the admin login setup section and noticed it was having LDAP connection issues, tried to manually fix things, but in the end just couldn’t get it to work. A fresh start seemed like it would be a good idea so I uninstalled Nextcloud (using the server manager) and rebooted for good measure.

Now the trouble starts… I can’t login into the file shares as myself anymore and the output of net ads info shows:
ads_connect: No logon servers
ads_connect: No logon servers
Didn’t find the ldap server!

And, now it’s working again without me changing anything! About 15 to 20 minutes have passed since I started typing this and researching on the web. What the heck? I just jumped to a shell to retry the command and now I get the expected output. File sharing and email logins are working again. Sorry to be troublesome but now I have a question. Is there some kind of built in timeout or retry interval with the nsdc virtual machine that is used for active directory?

Controller provisioning fails with long domain name
#2

No, there isn’t!

When a module is uninstalled the remaining ones are reconfigured. This and/or rebooting could have fixed your issue…

Please search the nsdc journal for any clue:

journalctl -M nsdc
#3

Thanks Davide. Since my original post, I applied the latest CentOS updates, rebooted and now it appears broken for good. The server has been up for well over an hour and AD is still busted. So you are right, I must have gotten lucky after my previous reboot.

Here is the output from the command you gave:

Apr 13 15:48:16 nsdc-trs80.kuntzilla.com systemd[1]: Started Login Service.
Apr 13 15:48:16 nsdc-trs80.kuntzilla.com systemd-networkd[22]: host0           : Cannot configure IPv4 forwarding for interface host0: Read-only file system
Apr 13 15:48:16 nsdc-trs80.kuntzilla.com systemd-networkd[22]: host0           : Cannot configure IPv6 forwarding for interface: Read-only file system
Apr 13 15:48:16 nsdc-trs80.kuntzilla.com systemd-networkd[22]: Enumeration completed
Apr 13 15:48:16 nsdc-trs80.kuntzilla.com systemd[1]: Started Network Service.
Apr 13 15:48:16 nsdc-trs80.kuntzilla.com systemd[1]: Reached target Network.
Apr 13 15:48:16 nsdc-trs80.kuntzilla.com systemd[1]: Starting Network.
Apr 13 15:48:16 nsdc-trs80.kuntzilla.com systemd[1]: Started Samba domain controller daemon.
Apr 13 15:48:16 nsdc-trs80.kuntzilla.com systemd[1]: Starting Samba domain controller daemon...
Apr 13 15:48:16 nsdc-trs80.kuntzilla.com systemd[1]: Reached target Multi-User System.
Apr 13 15:48:16 nsdc-trs80.kuntzilla.com systemd[1]: Starting Multi-User System.
Apr 13 15:48:16 nsdc-trs80.kuntzilla.com systemd[1]: Reached target Graphical Interface.
Apr 13 15:48:16 nsdc-trs80.kuntzilla.com systemd[1]: Starting Graphical Interface.
Apr 13 15:48:16 nsdc-trs80.kuntzilla.com systemd-networkd[22]: host0           : gained carrier
Apr 13 15:48:16 nsdc-trs80.kuntzilla.com systemd-networkd[22]: host0           : link configured
Apr 13 15:48:16 nsdc-trs80.kuntzilla.com systemd[1]: Started Update UTMP about System Runlevel Changes.
Apr 13 15:48:16 nsdc-trs80.kuntzilla.com systemd[1]: Startup finished in 1.221s.
Apr 13 15:48:17 nsdc-trs80.kuntzilla.com samba[24]: samba version 4.4.5 started.
Apr 13 15:48:17 nsdc-trs80.kuntzilla.com samba[24]: Copyright Andrew Tridgell and the Samba Team 1992-2016
Apr 13 15:48:18 nsdc-trs80.kuntzilla.com samba[24]: samba: using 'standard' process model
Apr 13 15:48:18 nsdc-trs80.kuntzilla.com samba[24]: Failed to bind to ipv6:fdab:d31d:931c:0:1858:b7ff:fe80:10cf:389 - NT_STATUS_ADDRESS_NOT_ASSOCIATED
Apr 13 15:48:18 nsdc-trs80.kuntzilla.com samba[24]: task_server_terminate: [cldapd failed to setup interfaces]
Apr 13 15:48:18 nsdc-trs80.kuntzilla.com samba[24]: Failed to bind to fdab:d31d:931c:0:1858:b7ff:fe80:10cf:88 UDP - NT_STATUS_ADDRESS_NOT_ASSOCIATED
Apr 13 15:48:18 nsdc-trs80.kuntzilla.com samba[24]: task_server_terminate: [kdc failed to setup interfaces]
Apr 13 15:48:18 nsdc-trs80.kuntzilla.com samba[24]: samba_terminate: cldapd failed to setup interfaces
Apr 13 15:48:18 nsdc-trs80.kuntzilla.com samba[24]: samba_terminate: kdc failed to setup interfaces
Apr 13 15:48:19 nsdc-trs80.kuntzilla.com winbindd[38]: [2017/04/13 15:48:19.039464,  0] ../source3/winbindd/winbindd_cache.c:3245(initialize_winbindd_cache)
Apr 13 15:48:19 nsdc-trs80.kuntzilla.com winbindd[38]:   initialize_winbindd_cache: clearing cache and re-creating with version number 2
Apr 13 15:48:19 nsdc-trs80.kuntzilla.com winbindd[38]: [2017/04/13 15:48:19.604910,  0] ../lib/util/become_daemon.c:124(daemon_ready)
Apr 13 15:48:19 nsdc-trs80.kuntzilla.com winbindd[38]:   STATUS=daemon 'winbindd' finished starting up and ready to serve connections
Apr 13 15:48:19 nsdc-trs80.kuntzilla.com smbd[28]: [2017/04/13 15:48:19.865263,  0] ../lib/util/become_daemon.c:124(daemon_ready)
Apr 13 15:48:19 nsdc-trs80.kuntzilla.com smbd[28]:   STATUS=daemon 'smbd' finished starting up and ready to serve connections
Apr 13 15:48:20 nsdc-trs80.kuntzilla.com samba[24]: /usr/sbin/samba_dnsupdate: Traceback (most recent call last):
Apr 13 15:48:20 nsdc-trs80.kuntzilla.com samba[24]: /usr/sbin/samba_dnsupdate:   File "/usr/sbin/samba_dnsupdate", line 631, in <module>
Apr 13 15:48:20 nsdc-trs80.kuntzilla.com samba[24]: /usr/sbin/samba_dnsupdate:     get_credentials(lp)
Apr 13 15:48:20 nsdc-trs80.kuntzilla.com samba[24]: /usr/sbin/samba_dnsupdate:   File "/usr/sbin/samba_dnsupdate", line 123, in get_credentials
Apr 13 15:48:20 nsdc-trs80.kuntzilla.com samba[24]: /usr/sbin/samba_dnsupdate:     raise e
Apr 13 15:48:20 nsdc-trs80.kuntzilla.com samba[24]: /usr/sbin/samba_dnsupdate: RuntimeError: kinit for NSDC-TRS80$@KUNTZILLA.COM failed (Cannot contact any KDC for requested realm)
Apr 13 15:48:20 nsdc-trs80.kuntzilla.com samba[24]: /usr/sbin/samba_dnsupdate:
Apr 13 15:48:20 nsdc-trs80.kuntzilla.com samba[24]: ../source4/dsdb/dns/dns_update.c:295: Failed DNS update - NT_STATUS_ACCESS_DENIED
Apr 13 15:58:19 nsdc-trs80.kuntzilla.com samba[24]: /usr/sbin/samba_dnsupdate: Traceback (most recent call last):
Apr 13 15:58:19 nsdc-trs80.kuntzilla.com samba[24]: /usr/sbin/samba_dnsupdate:   File "/usr/sbin/samba_dnsupdate", line 631, in <module>
Apr 13 15:58:19 nsdc-trs80.kuntzilla.com samba[24]: /usr/sbin/samba_dnsupdate:     get_credentials(lp)
Apr 13 15:58:19 nsdc-trs80.kuntzilla.com samba[24]: /usr/sbin/samba_dnsupdate:   File "/usr/sbin/samba_dnsupdate", line 123, in get_credentials
Apr 13 15:58:19 nsdc-trs80.kuntzilla.com samba[24]: /usr/sbin/samba_dnsupdate:     raise e

I’m stumped at this point as to why it just stopped working. There appear to be some IP v6 errors listed, but I only use v4 on my network.

Domain controller IP unreachable
#4

An IPv4 conflict in your LAN?

 systemctl stop nsdc
 arp -d  $(config getprop nsdc IpAddr)
 ping $(config getprop nsdc IpAddr)
 systemctl start nsdc
#5

There shouldn’t be one. .2 is the address of nsdc and I have DHCP setup to use .100 through .253

The arp -d command returned no output, while the ping returns:
[root@trs80 ~]# ping $(config getprop nsdc IpAddress)
PING 192.168.67.2 (192.168.67.2) 56(84) bytes of data.
From 192.168.67.1 icmp_seq=1 Destination Host Unreachable
From 192.168.67.1 icmp_seq=2 Destination Host Unreachable

When I started nsdc again it still failed the DNS update.
Apr 13 18:25:28 nsdc-trs80.kuntzilla.com samba[25]: ../source4/dsdb/dns/dns_update.c:295: Failed DNS update - NT_STATUS_IO_DEVICE_ERROR

Just as a test (without changing anything) I did a systemctl stop nsdc and systemctl start nsdc, waited a bit and now it’s working again:

Apr 13 18:32:25 nsdc-trs80.kuntzilla.com systemd[1]: Starting Update UTMP about System Runlevel Changes...
Apr 13 18:32:25 nsdc-trs80.kuntzilla.com systemd-networkd[22]: host0           : gained carrier
Apr 13 18:32:25 nsdc-trs80.kuntzilla.com systemd-networkd[22]: host0           : link configured
Apr 13 18:32:25 nsdc-trs80.kuntzilla.com systemd[1]: Started Update UTMP about System Runlevel Changes.
Apr 13 18:32:25 nsdc-trs80.kuntzilla.com systemd[1]: Startup finished in 552ms.
Apr 13 18:32:25 nsdc-trs80.kuntzilla.com samba[25]: samba version 4.4.5 started.
Apr 13 18:32:25 nsdc-trs80.kuntzilla.com samba[25]: Copyright Andrew Tridgell and the Samba Team 1992-2016
Apr 13 18:32:25 nsdc-trs80.kuntzilla.com samba[25]: samba: using 'standard' process model
Apr 13 18:32:25 nsdc-trs80.kuntzilla.com winbindd[38]: [2017/04/13 18:32:25.987822,  0] ../source3/winbindd/winbindd_cache.c:3245(initialize_winbindd_cache)
Apr 13 18:32:25 nsdc-trs80.kuntzilla.com winbindd[38]:   initialize_winbindd_cache: clearing cache and re-creating with version number 2
Apr 13 18:32:26 nsdc-trs80.kuntzilla.com smbd[29]: [2017/04/13 18:32:26.637713,  0] ../lib/util/become_daemon.c:124(daemon_ready)
Apr 13 18:32:26 nsdc-trs80.kuntzilla.com smbd[29]:   STATUS=daemon 'smbd' finished starting up and ready to serve connections
Apr 13 18:32:26 nsdc-trs80.kuntzilla.com winbindd[38]: [2017/04/13 18:32:26.822212,  0] ../lib/util/become_daemon.c:124(daemon_ready)
Apr 13 18:32:26 nsdc-trs80.kuntzilla.com winbindd[38]:   STATUS=daemon 'winbindd' finished starting up and ready to serve connections
Apr 13 18:37:37 nsdc-trs80.kuntzilla.com systemd[1]: Starting Network Service...
Apr 13 18:37:37 nsdc-trs80.kuntzilla.com systemd-networkd[49]: host0           : Cannot configure IPv4 forwarding for interface host0: Read-only file system
Apr 13 18:37:37 nsdc-trs80.kuntzilla.com systemd-networkd[49]: host0           : Cannot configure IPv6 forwarding for interface: Read-only file system
Apr 13 18:37:37 nsdc-trs80.kuntzilla.com systemd-networkd[49]: Enumeration completed
Apr 13 18:37:37 nsdc-trs80.kuntzilla.com systemd[1]: Started Network Service.
Apr 13 18:37:37 nsdc-trs80.kuntzilla.com systemd-networkd[49]: host0           : link configured

net ads info gives me this:
LDAP server: 192.168.67.2
LDAP server name: nsdc-trs80.kuntzilla.com
Realm: KUNTZILLA.COM
Bind Path: dc=KUNTZILLA,dc=COM
LDAP port: 389
Server time: Thu, 13 Apr 2017 18:45:57 EDT
KDC server: 192.168.67.2
Server time offset: 0
Last machine account password change: Sun, 26 Feb 2017 15:50:03 EST

What the heck? Since it is not consistent I wonder if there is some kind of timing issue during the startup?

1 Like
#6

It’s possible!

Next week we could try to delay samba startup by setting After=network-online.target in samba unit configuration.

See also

https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

https://bugs.launchpad.net/ubuntu/+source/samba/+bug/1339434

#7

Ok, I will check out those web pages. I did some more checking in the logs last night and it appears that the trouble started on April 3rd with the DNS update errors. From Feb 26 through April 2 there are no errors in the log. Checking my system logs didn’t show any package updates around that time frame and I don’t remember changing any settings either.

This series of message keeps reoccurring in the log now:
Apr 14 06:22:32 nsdc-trs80.kuntzilla.com samba[25]: …/source4/dsdb/dns/dns_update.c:295: Failed DNS update - NT_STATUS_IO_DEVICE_ERROR
Apr 14 06:32:32 nsdc-trs80.kuntzilla.com samba[25]: …/source4/dsdb/dns/dns_update.c:295: Failed DNS update - NT_STATUS_IO_DEVICE_ERROR
Apr 14 06:42:32 nsdc-trs80.kuntzilla.com samba[25]: …/source4/dsdb/dns/dns_update.c:295: Failed DNS update - NT_STATUS_IO_DEVICE_ERROR
Apr 14 06:45:55 nsdc-trs80.kuntzilla.com systemd[1]: Starting Network Service…
Apr 14 06:45:55 nsdc-trs80.kuntzilla.com systemd-networkd[500]: host0 : Cannot configure IPv4 forwarding for interface host0: Read-only file system
Apr 14 06:45:55 nsdc-trs80.kuntzilla.com systemd-networkd[500]: host0 : Cannot configure IPv6 forwarding for interface: Read-only file system
Apr 14 06:45:55 nsdc-trs80.kuntzilla.com systemd-networkd[500]: Enumeration completed
Apr 14 06:45:55 nsdc-trs80.kuntzilla.com systemd-networkd[500]: host0 : link configured
Apr 14 06:45:55 nsdc-trs80.kuntzilla.com systemd[1]: Started Network Service.
Apr 14 06:52:32 nsdc-trs80.kuntzilla.com samba[25]: …/source4/dsdb/dns/dns_update.c:295: Failed DNS update - NT_STATUS_IO_DEVICE_ERROR
Apr 14 07:02:32 nsdc-trs80.kuntzilla.com samba[25]: …/source4/dsdb/dns/dns_update.c:295: Failed DNS update - NT_STATUS_IO_DEVICE_ERROR
Apr 14 07:09:11 nsdc-trs80.kuntzilla.com systemd[1]: Starting Network Service…
Apr 14 07:09:11 nsdc-trs80.kuntzilla.com systemd-networkd[519]: host0 : Cannot configure IPv4 forwarding for interface host0: Read-only file system
Apr 14 07:09:11 nsdc-trs80.kuntzilla.com systemd-networkd[519]: host0 : Cannot configure IPv6 forwarding for interface: Read-only file system
Apr 14 07:09:11 nsdc-trs80.kuntzilla.com systemd-networkd[519]: Enumeration completed
Apr 14 07:09:11 nsdc-trs80.kuntzilla.com systemd[1]: Started Network Service.
Apr 14 07:09:11 nsdc-trs80.kuntzilla.com systemd-networkd[519]: host0 : link configured

Looks like every 10 to 20 minutes it is trying to update DNS and then restarts the container’s networking I guess? I’m a novice when it comes to containers and Samba. Is it failing to update the DNS that is built into Samba or the DNSMasq(?) one that comes with Nethserver? trying to figure out where to look for the issues.

Thanks for trying to help me Davide, I really appreciate it.