Samba container fails to start at first (large HDD shares)

[kellerman (Aivis)]

Hello, I yesterday updated my NS8 machine that is running just Samba with AD services.
Now after restarting the VM (Rocky Linux) I get these logs in NS8 UI. The container starts and runs well, it just takes multiple retries for it to start. Anyone experiencing similar issue?

2025-08-20T03:06:06+03:00 [1:samba1:systemd] Started D-Bus User Message Bus.
2025-08-20T03:06:06+03:00 [1:samba1:] Ready
2025-08-20T03:06:06+03:00 [1:samba1:systemd] Created slice Slice /user.
2025-08-20T03:06:06+03:00 [1:samba1:systemd] libpod-43b52b70608152ebcb29f2e54a3b98fbf0b3c49ad862acba8b9cb6b27281827e.scope: unit configures an IP firewall, but not running as root.
2025-08-20T03:06:06+03:00 [1:samba1:systemd] (This warning is only shown for the first unit using IP firewalling.)
2025-08-20T03:06:06+03:00 [1:samba1:systemd] Started libcrun container.
2025-08-20T03:06:06+03:00 [1:samba1:podman] 43b52b70608152ebcb29f2e54a3b98fbf0b3c49ad862acba8b9cb6b27281827e
2025-08-20T03:06:06+03:00 [1:samba1:systemd] Started podman-pause-13686a14.scope.
2025-08-20T03:06:06+03:00 [1:samba1:systemd] Started TimescaleDB for Samba Audit logs.
2025-08-20T03:06:06+03:00 [1:samba1:systemd] Starting Samba DC and File Server...
2025-08-20T03:06:06+03:00 [1:samba1:timescaledb]
2025-08-20T03:06:06+03:00 [1:samba1:timescaledb] PostgreSQL Database directory appears to contain a database; Skipping initialization
2025-08-20T03:06:06+03:00 [1:samba1:timescaledb]
2025-08-20T03:06:06+03:00 [1:samba1:timescaledb] 2025-08-20 00:06:06.608 UTC [1] LOG:  starting PostgreSQL 17.5 on x86_64-pc-linux-musl, compiled by gcc (Alpine 14.2.0) 14.2.0, 64-bit
2025-08-20T03:06:06+03:00 [1:samba1:timescaledb] 2025-08-20 00:06:06.608 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 15432
2025-08-20T03:06:06+03:00 [1:samba1:timescaledb] 2025-08-20 00:06:06.608 UTC [1] LOG:  listening on IPv6 address "::", port 15432
2025-08-20T03:06:06+03:00 [1:samba1:timescaledb] 2025-08-20 00:06:06.616 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.15432"
2025-08-20T03:06:06+03:00 [1:samba1:timescaledb] 2025-08-20 00:06:06.630 UTC [24] LOG:  database system was shut down at 2025-08-20 00:05:34 UTC
2025-08-20T03:06:06+03:00 [1:samba1:timescaledb] 2025-08-20 00:06:06.661 UTC [1] LOG:  database system is ready to accept connections
2025-08-20T03:06:06+03:00 [1:samba1:systemd] Started API Module Daemon (AMLD).
2025-08-20T03:06:06+03:00 [1:samba1:timescaledb] 2025-08-20 00:06:06.670 UTC [27] LOG:  TimescaleDB background worker launcher connected to shared catalogs
2025-08-20T03:07:37+03:00 [1:samba1:systemd] samba-dc.service: start operation timed out. Terminating.
2025-08-20T03:07:37+03:00 [1:samba1:systemd] samba-dc.service: Control process exited, code=exited, status=1/FAILURE
2025-08-20T03:07:37+03:00 [1:samba1:podman] c45aa4ef209f5edc0d786170c90d34ecf88e5be9587f176913cf760ae74ce632
2025-08-20T03:07:37+03:00 [1:samba1:systemd] samba-dc.service: Failed with result 'timeout'.
2025-08-20T03:07:37+03:00 [1:samba1:systemd] Failed to start Samba DC and File Server.
2025-08-20T03:07:37+03:00 [1:samba1:systemd] samba-dc.service: Consumed 9.931s CPU time.
2025-08-20T03:07:37+03:00 [1:samba1:systemd] Reached target Main User Target.
2025-08-20T03:07:37+03:00 [1:samba1:systemd] Startup finished in 1min 31.432s.
2025-08-20T03:07:38+03:00 [1:samba1:systemd] samba-dc.service: Scheduled restart job, restart counter is at 1.
2025-08-20T03:07:38+03:00 [1:samba1:systemd] Stopped Samba DC and File Server.
2025-08-20T03:07:38+03:00 [1:samba1:systemd] samba-dc.service: Consumed 9.931s CPU time.
2025-08-20T03:07:38+03:00 [1:samba1:systemd] Starting Samba DC and File Server...
2025-08-20T03:08:07+03:00 [1:samba1:systemd] Starting Mark boot as successful...
2025-08-20T03:08:07+03:00 [1:samba1:systemd] Finished Mark boot as successful.
2025-08-20T03:09:08+03:00 [1:samba1:systemd] samba-dc.service: start operation timed out. Terminating.
2025-08-20T03:09:08+03:00 [1:samba1:systemd] samba-dc.service: Control process exited, code=exited, status=1/FAILURE
2025-08-20T03:09:08+03:00 [1:samba1:podman] 9ba393ac0f78f2c84057fa2b9013574b7601c1e2584ad0b2d6cc89ff8f9d9ee0
2025-08-20T03:09:08+03:00 [1:samba1:systemd] samba-dc.service: Failed with result 'timeout'.
2025-08-20T03:09:08+03:00 [1:samba1:systemd] Failed to start Samba DC and File Server.
2025-08-20T03:09:08+03:00 [1:samba1:systemd] samba-dc.service: Consumed 9.970s CPU time.
2025-08-20T03:09:08+03:00 [1:samba1:systemd] samba-dc.service: Scheduled restart job, restart counter is at 2.
2025-08-20T03:09:08+03:00 [1:samba1:systemd] Stopped Samba DC and File Server.
2025-08-20T03:09:08+03:00 [1:samba1:systemd] samba-dc.service: Consumed 9.970s CPU time.
2025-08-20T03:09:08+03:00 [1:samba1:systemd] Starting Samba DC and File Server...
2025-08-20T03:09:53+03:00 [1:samba1:systemd] Started libcrun container.
2025-08-20T03:09:53+03:00 [1:samba1:podman] 33516dbb6c6328446240ba7a440b1bb2fae2987b60b023ec306b14aa2d063f37
2025-08-20T03:09:53+03:00 [1:samba1:bash] /usr/bin/bash: connect: Connection refused
2025-08-20T03:09:53+03:00 [1:samba1:bash] /usr/bin/bash: line 1: /dev/tcp/10.44.1.4/53: Connection refused
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] # Global parameters
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] [global]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	bind interfaces only = Yes
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	interfaces = 127.0.0.1 xxx.xxx.xxx.xxx
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	ldap server require strong auth = No
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	ntlm auth = mschapv2-and-ntlmv2-only
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	obey pam restrictions = Yes
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	passdb backend = samba_dsdb
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	realm = AD.XXX.LOCAL
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	registry shares = Yes
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	server role = active directory domain controller
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	template homedir = /srv/homes/%U
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	workgroup = XXX
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	rpc_server:tcpip = no
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	rpc_daemon:spoolssd = embedded
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	rpc_server:spoolss = embedded
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	rpc_server:winreg = embedded
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	rpc_server:ntsvcs = embedded
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	rpc_server:eventlog = embedded
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	rpc_server:srvsvc = embedded
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	rpc_server:svcctl = embedded
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	rpc_server:default = external
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	winbindd:use external pipes = true
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	recycle:versions = yes
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	recycle:keeptree = yes
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	recycle:repository =
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	full_audit:failure = none
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	full_audit:success = none
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	full_audit:priority = INFO
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	full_audit:facility = LOCAL7
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	full_audit:prefix = %R|%I|%u|%S
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	acl_xattr:ignore system acls = yes
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	acl_xattr:security_acl_name = user.NTACL
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	idmap config * : backend = tdb
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	include = /etc/samba/include.conf
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	inherit owner = windows and unix
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	map archive = No
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	vfs objects = dfs_samba4 acl_xattr recycle full_audit
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] [sysvol]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	inherit owner = no
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	path = /var/lib/samba/sysvol
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	read only = No
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	acl_xattr:ignore system acls = no
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] [netlogon]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	path = /var/lib/samba/sysvol/ad.xxx.local/scripts
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	read only = No
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] [homes]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	browseable = No
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	comment = %u home directory
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	read only = No
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] [xxx]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	comment = AD xxx files
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	path = /srv/shares/xxx
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	read only = No
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] [xxx]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	comment = xxx
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	path = /srv/shares/xxx
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	read only = No
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] [xxx]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	comment = xxx backups
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	path = /srv/shares/xxx
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	read only = No
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] [xxx]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	comment = xxx
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	path = /srv/shares/xxx
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	read only = No
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] [xxx]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	comment = xxx
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	path = /srv/shares/xxx
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	read only = No
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] [xxx]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	comment = xxx
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	path = /srv/shares/xxx
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	read only = No
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] [xxx]
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	comment = xxx
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	path = /srv/shares/xxx
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 	read only = No
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 2025-08-20T00:09:54Z chronyd version 4.5 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 -DEBUG)
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 2025-08-20T00:09:54Z Disabled control of system clock
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 2025-08-20T00:09:54Z Loaded 0 symmetric keys
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] 2025-08-20T00:09:54Z MS-SNTP authentication enabled
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] [2025-08-20T00:09:54.162093] smart-multi-line: error opening smart-multi-line.fsm file; filename='/usr/share/syslog-ng/smart-multi-line.fsm', error='No such file or directory (2)'
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] [2025-08-20T00:09:54.162113] smart-multi-line: your smart-multi-line.fsm seems to be empty or non-existent, automatic multi-line log extraction will probably not work; filename='/usr/share/syslog-ng/smart-multi-line.fsm'
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] samba version 4.19.5-Ubuntu started.
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] Copyright Andrew Tridgell and the Samba Team 1992-2023
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] daemon 'samba' : Starting process...
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] /usr/sbin/smbd: smbd version 4.19.5-Ubuntu started.
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] /usr/sbin/smbd: Copyright Andrew Tridgell and the Samba Team 1992-2023
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] /usr/sbin/smbd: INFO: Profiling turned OFF from pid 26
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] /usr/sbin/winbindd: winbindd version 4.19.5-Ubuntu started.
2025-08-20T03:09:54+03:00 [1:samba1:samba-dc] /usr/sbin/winbindd: Copyright Andrew Tridgell and the Samba Team 1992-2023
2025-08-20T03:09:55+03:00 [1:samba1:samba-dc] /usr/bin/wsdd:935: DeprecationWarning: Testing an element's truth value will raise an exception in future versions.  Use specific 'len(elem)' or 'elem is not None' test instead.
2025-08-20T03:09:55+03:00 [1:samba1:samba-dc]   if scopes:
2025-08-20T03:09:58+03:00 [1:samba1:systemd] Started Samba DC and File Server.

[mrmarkuz (Markus Neuberger)]

I can’t reproduce.

@kellerman

samba-dc.service: start operation timed out. Terminating.

It seems samba hits a timeout. Is there high system load?

At least SSD disks are recommended, see System requirements — NS8 documentation

[kellerman (Aivis)]

There is about 1400 users and the system is a 4309Y Xeon with 10G of RAM allocated running under Proxmox and a Intel SATA SSD. When timing out, there seems to be no CPU load (I don’t know about disk load). Problem seem to have started after latest updates. At the moment of starting there is close to none user usage.

How can I debug Samba startup more verbose? Or maybe there is a need to modify the service timeout limit? Also there is some lines about failing connection to port 53 on 10.44.1.4 (which is ns8 IP).

[mrmarkuz (Markus Neuberger)]

Currently the timeout for samba-dc is set to TimeoutStopSec=70. (70 secs)
You could use following command to edit the samba-dc service file and increase the value:

runagent -m samba1 systemctl --user edit --full samba-dc

@kellerman

Also there is some lines about failing connection to port 53 on 10.44.1.4 (which is ns8 IP).

This is just a check if the service is up and can be ignored.

[kellerman (Aivis)]

Thanks for the advice @mrmarkuz
I changed the timeout to 150 and it still does the same thing, now just it takes even longer for it to start.

Doing

runagent -m samba1 systemctl --user restart samba-dc

restarts samba-dc just fine after it’s started.

Is it the systemd unit script thats hanging on some point (more likely) or Samba itself? How could I debug it?

[mrmarkuz (Markus Neuberger)]

I think it’s due to high IO load during boot of the server.
After booting there’s no high load anymore and the manual service restarts are fast and without timeout.
You could check the system löad after a reboot.
I didn’t test but maybe it helps to add for example ExecStartPre=sleep 60 in the samba-dc.service file before the container is started so Samba starts a minute later when there’s no high load anymore.

[kellerman (Aivis)]

It seems that it just creates lots of IO load when starting Samba. The user home directories are on a HDD RAID array (md0). md0 is being accessed at the startup time, which obviously is slower than SSD. Thats probably why ExecStartPre=sleep 60 didn’t help.

My idea is that it reads the home dirs for some reason (to re-check the permissions perhaps). Otherwise the server is snappy as hell.

Anyways… setting the timeout to anything larger than 90s in that file seems not to be respected. There is some more global systemd limit maybe?

[mrmarkuz (Markus Neuberger)]

@kellerman

The user home directories are on a HDD RAID array (md0).

All rootless containers are located in /home so that causes the high load on boot.
I think it’s not fully solvable without fast disks but as samba is starting it should be ok.

@kellerman

setting the timeout to anything larger than 90s in that file seems not to be respected. There is some more global systemd limit maybe?

Maybe other timeout options are working better, see systemd.service

[kellerman (Aivis)]

@mrmarkuz

All rootless containers are located in /home

It’s only the AD user home directories in my case that are mounted on a HDD RAID array (/home/samba1/.local/share/containers/storage/volumes/homes/). I know that samba checks for correct permissions on every share on boot.

/home and everything else is still mounted on the system SSD.

Anyways thanks for the answers @mrmarkuz ! I will try to make systemd to wait longer, so there are no problems further on.

So the solution was setting both TimeoutStopSec=x and TimeoutStartSec=x
I set it to 500s, which is plenty enough. It started first time now.