NS8 Core 3.19.1 released: required compatibility update for Rocky Linux and AlmaLinux 9.8

davidep · June 5, 2026, 3:18pm

We have released NS8 Core 3.19.1^[1].

A breaking change introduced by systemd in Rocky Linux 9.8 can cause application services to fail after an application restart or a node reboot. The same issue also affects AlmaLinux 9.8^[2].

To address the problem, Core 3.19.1 includes an important compatibility fix that is required before upgrading Rocky Linux or AlmaLinux nodes to version 9.8.

What you need to do

All NS8 clusters should update to Core 3.19.1.

Open:

Cluster Admin → Software Center → Updates

and update the Core component to version 3.19.1.

Debian-based nodes are not affected by the systemd change and can safely update to Core 3.19.1 without any additional considerations.

Updating the Core component does not upgrade the operating system to Rocky Linux 9.8 or AlmaLinux 9.8.

Existing Rocky Linux and AlmaLinux 9.7 systems continue to operate normally and can safely install application updates while remaining on 9.7.

Rocky Linux 9.8 availability

We are aware that Rocky Linux 9.8 includes important security fixes. However, to avoid unexpected service disruptions, NS8 Rocky Linux mirrors are still serving Rocky Linux 9.7.

To reduce the risk of accidental upgrades, NS8 Rocky Linux mirrors will remain on 9.7 until Tuesday, 2026-06-09. This provides a transition period for administrators to update their clusters to Core 3.19.1.

After that date, NS8 Rocky Linux mirrors will be synchronized with Rocky Linux 9.8.

Manual upgrade to Rocky Linux 9.8 before 2026-06-09

If you need to upgrade earlier, first install Core 3.19.1 and then temporarily enable the upstream Rocky Linux repositories:

dnf --enablerepo=baseos,appstream --disablerepo=ns-baseos,ns-appstream update -y

Do not perform this upgrade before installing Core 3.19.1.

Recovery procedure for accidentally upgraded nodes

If a node running Core 3.19.0 or earlier is accidentally upgraded to Rocky Linux or AlmaLinux 9.8, services will generally continue to run until an application restart or a node reboot occurs.

After a restart or reboot, application services may fail and Cluster Admin access may become unavailable.

To recover the node, manually repair the Traefik core application and then install Core 3.19.1.

First identify the Traefik service account:

grep traefik /etc/passwd

The added symbolic link redirects systemd %S placeholders used by NS8 applications to the previous state directory location. Without it, application services will likely fail to start after a restart or reboot.

Assuming the account is traefik1, run:

ln -sfv ../.config ~traefik1/.local/state
systemctl restart user@$(id -u traefik1)

Once Cluster Admin functionality has been restored, update the Core component to version 3.19.1, then reboot the node or run:

systemctl try-restart user@*

Information for NS8 application developers

Core 3.19.1 includes a compatibility workaround that restores the behavior expected by existing applications. The workaround allows current applications to continue working on Rocky Linux 9.8, AlmaLinux 9.8, and Debian 13, replacing the Debian-specific workaround that was previously released.

However, application developers should update their applications to use the correct state directory paths and avoid relying on the compatibility symlink introduced by the workaround.

Instructions and background information are available here:

github.com/NethServer/dev

Implement placeholder %E in Systemd services

opened 06:17PM - 04 Jun 26 UTC

DavidePrincipi

Newer Rocky Linux and Debian releases ship a breaking change in the way Systemd …expands the %S placeholder. This issue is described and solved with a symlink automatic creation in #8028. In old Systemd versions (up to Rocky Linux 9.8 and Debian 12) `%S` and `%E` placeholders both expand to `~/.config`. In recent releases RL 9.8, Debian 13 `%S` expands to `.local/state`, reflecting upstream changes. NS8 Systemd units often use `%S` as symbolic reference to `~/.config`. **Proposed solution** Replace `%S` with `%E` in Systemd service units, for all NS8 applications. This replacement is more robust than relying on a symlink created when the Systemd user session starts. **Alternative solutions** Do nothing and continue to rely on the symlink created by fix #8028. **Additional context** - Upstream patch https://git.rockylinux.org/staging/src-rhel/rpms/systemd/-/blob/c9s/SOURCES/1313-execute-add-support-for-XDG_STATE_HOME-for-placing-s.patch?ref_type=heads **See also** - Bug #8028 - Discussion https://mattermost.nethesis.it/nethesis/pl/jb7c1zs6atn5bjsswenua7hk6e - Forum fix release announcement https://community.nethserver.org/t/ns8-core-3-19-1-released-required-compatibility-update-for-rocky-linux-and-almalinux-9-8/27773

We recommend updating applications as soon as possible so they remain compatible with future operating system and systemd changes without requiring the workaround.

stephdl · June 5, 2026, 4:14pm

done…I am still alive

MadPatrick · June 5, 2026, 5:18pm

On Debian 13 no issues

pagaille · June 6, 2026, 10:31am

Just upgraded to Rocky 9.8 as detailed above (thanks @davidep). System up but one of my two nextcloud instances didn’t start correctly.

Symptom :

Failed to load environment files: No such file or directory
Failed to run 'start-pre' task: No such file or directory

~/.local/state/ was absent or empty. I had to create and link manually

mkdir -p ~/.local/state && ln -sfn ~/.config/state ~/.local/state/state'

Strangely nextcloud2 worked perfectly.

davidep · June 6, 2026, 1:27pm

Thanks for the report.

I notice the symbolic link you created differs from the one documented in the announcement. The documented workaround creates:

~/.local/state → ../.config

while your fix creates:

~/.local/state/state → ~/.config/state

Both may address the immediate issue, but they are not equivalent.

It would be useful to understand why the workaround was not applied to one of the two Nextcloud instances while the other started correctly. Could you check “journalctl” and/or “/var/log/messages” around the time of the upgrade and post any relevant messages related to the affected instance?

In particular, I’d like to see what happened when the instance was restarted and whether there were any errors while applying the compatibility fix.

pagaille · June 6, 2026, 6:59pm

Sure. Could please give me a pointer to a string that would help find the moment I started the upgrade ? There is so much noise in those logs, it’s difficult to find anything

davidep · June 6, 2026, 8:55pm

A good starting point would be to look for the systemd unit that triggers the compatibility workaround:

grep -A 10 User-State-Private /var/log/messages*

Please post the output. It should help determine whether the fix was applied correctly and why one Nextcloud instance behaved differently from the other.

pagaille · June 7, 2026, 6:47am

Here you are :

[root@ns8 ~]# id nextcloud1
uid=1005(nextcloud1) gid=1005(nextcloud1) groups=1005(nextcloud1)

[root@ns8 ~]# id nextcloud2
uid=1006(nextcloud2) gid=1006(nextcloud2) groups=1006(nextcloud2)




% grep -A 10 User-State-Private  /var/log/messages-20260607 
Jun  6 11:16:07 ns8 systemd[1]: Starting Create User-State-Private link...
Jun  6 11:16:07 ns8 systemd[1]: Starting User Runtime Directory /run/user/0...
Jun  6 11:16:07 ns8 systemd-logind[908]: New session 74 of user root.
Jun  6 11:16:07 ns8 systemd[1]: fix-xdg-state@0.service: Deactivated successfully.
Jun  6 11:16:07 ns8 systemd[1]: Finished Create User-State-Private link.
Jun  6 11:16:07 ns8 systemd[1]: Finished User Runtime Directory /run/user/0.
Jun  6 11:16:07 ns8 systemd[1]: Starting User Manager for UID 0...
Jun  6 11:16:08 ns8 redis[844358]: 1:M 06 Jun 2026 09:16:08.085 * 1 changes in 5 seconds. Saving...
Jun  6 11:16:08 ns8 redis[844358]: 1:M 06 Jun 2026 09:16:08.086 * Background saving started by pid 32
Jun  6 11:16:08 ns8 redis[844358]: 32:C 06 Jun 2026 09:16:08.089 * BGSAVE done, 300 keys saved, 0 keys skipped, 286299 bytes written.
Jun  6 11:16:08 ns8 systemd[845738]: Queued start job for default target Main User Target.
Jun  6 11:16:08 ns8 systemd[845738]: Created slice User Application Slice.
Jun  6 11:16:08 ns8 systemd[845738]: Mark boot as successful after the user session has run 2 minutes was skipped because of an unmet condition check (ConditionUser=!@system).
Jun  6 11:16:08 ns8 systemd[845738]: Started Daily Cleanup of User's Temporary Directories.
Jun  6 11:16:08 ns8 systemd[845738]: Reached target Paths.
--
Jun  6 11:50:25 ns8 systemd[1]: Starting Create User-State-Private link...
Jun  6 11:50:25 ns8 systemd[1]: Starting Create User-State-Private link...
Jun  6 11:50:25 ns8 systemd[1]: Starting Create User-State-Private link...
Jun  6 11:50:25 ns8 systemd[1]: Starting Create User-State-Private link...
Jun  6 11:50:25 ns8 systemd[1]: Starting Create User-State-Private link...
Jun  6 11:50:25 ns8 systemd[1]: Starting Create User-State-Private link...
Jun  6 11:50:25 ns8 systemd[1]: Starting Create User-State-Private link...
Jun  6 11:50:25 ns8 api-server[944]: NethServer cluster-admin UI:
Jun  6 11:50:25 ns8 api-server[944]:  - https://ns8.toucheatout.be/cluster-admin/
Jun  6 11:50:25 ns8 api-server[944]:  - https:///cluster-admin/
Jun  6 11:50:25 ns8 systemd[1]: Starting User Runtime Directory /run/user/1000...
Jun  6 11:50:25 ns8 systemd[1]: Starting User Runtime Directory /run/user/1001...
Jun  6 11:50:25 ns8 systemd[1]: Starting User Runtime Directory /run/user/1002...
Jun  6 11:50:25 ns8 systemd[1]: Starting User Runtime Directory /run/user/1003...
Jun  6 11:50:25 ns8 systemd[1]: Starting User Runtime Directory /run/user/1004...
Jun  6 11:50:25 ns8 systemd[1]: Starting User Runtime Directory /run/user/1005...
Jun  6 11:50:25 ns8 systemd[1]: Starting User Runtime Directory /run/user/1013...
--
Jun  6 11:50:25 ns8 systemd[1]: Starting Create User-State-Private link...
Jun  6 11:50:25 ns8 systemd[1]: Starting Create User-State-Private link...
Jun  6 11:50:25 ns8 systemd[1]: Starting Create User-State-Private link...
Jun  6 11:50:25 ns8 systemd[1]: Starting User Runtime Directory /run/user/1006...
Jun  6 11:50:25 ns8 systemd[1]: Starting User Runtime Directory /run/user/1008...
Jun  6 11:50:25 ns8 systemd[1]: Starting User Runtime Directory /run/user/1015...
Jun  6 11:50:25 ns8 systemd[1]: Started User Login Management.
Jun  6 11:50:25 ns8 bash[960]: '.local/state' -> '../.config'
Jun  6 11:50:25 ns8 systemd[1]: fix-xdg-state@1002.service: Deactivated successfully.
Jun  6 11:50:25 ns8 systemd[1]: Finished Create User-State-Private link.
Jun  6 11:50:25 ns8 bash[962]: '.local/state' -> '../.config'
Jun  6 11:50:25 ns8 systemd[1]: fix-xdg-state@1004.service: Deactivated successfully.
Jun  6 11:50:25 ns8 systemd[1]: Finished Create User-State-Private link.
Jun  6 11:50:25 ns8 systemd[1]: fix-xdg-state@1005.service: Deactivated successfully.
Jun  6 11:50:25 ns8 systemd[1]: Finished Create User-State-Private link.
Jun  6 11:50:25 ns8 bash[961]: '.local/state' -> '../.config'
Jun  6 11:50:25 ns8 systemd[1]: fix-xdg-state@1003.service: Deactivated successfully.
Jun  6 11:50:25 ns8 systemd[1]: Finished Create User-State-Private link.
Jun  6 11:50:25 ns8 bash[959]: '.local/state' -> '../.config'
Jun  6 11:50:25 ns8 systemd[1]: fix-xdg-state@1000.service: Deactivated successfully.
Jun  6 11:50:25 ns8 systemd[1]: Finished Create User-State-Private link.
Jun  6 11:50:25 ns8 bash[965]: '.local/state' -> '../.config'
Jun  6 11:50:25 ns8 systemd[1]: fix-xdg-state@1008.service: Deactivated successfully.
Jun  6 11:50:25 ns8 systemd[1]: Finished Create User-State-Private link.
Jun  6 11:50:25 ns8 systemd[1]: Finished User Runtime Directory /run/user/1000.
Jun  6 11:50:25 ns8 systemd[1]: Finished User Runtime Directory /run/user/1001.
Jun  6 11:50:25 ns8 systemd[1]: Finished User Runtime Directory /run/user/1002.
Jun  6 11:50:25 ns8 systemd[1]: Finished User Runtime Directory /run/user/1003.
Jun  6 11:50:25 ns8 systemd[1]: Finished User Runtime Directory /run/user/1004.
Jun  6 11:50:25 ns8 systemd[1]: Finished User Runtime Directory /run/user/1005.
Jun  6 11:50:25 ns8 systemd[1]: Finished User Runtime Directory /run/user/1013.
Jun  6 11:50:25 ns8 systemd[1]: Finished User Runtime Directory /run/user/1006.
Jun  6 11:50:25 ns8 systemd[1]: Finished User Runtime Directory /run/user/1008.
Jun  6 11:50:25 ns8 systemd[1]: Finished User Runtime Directory /run/user/1015.
--
Jun  6 11:50:25 ns8 systemd[1]: Finished Create User-State-Private link.
Jun  6 11:50:25 ns8 bash[967]: '.local/state' -> '../.config'
Jun  6 11:50:25 ns8 systemd[1]: fix-xdg-state@1013.service: Deactivated successfully.
Jun  6 11:50:25 ns8 systemd[1]: Finished Create User-State-Private link.
Jun  6 11:50:25 ns8 bash[968]: '.local/state' -> '../.config'
Jun  6 11:50:25 ns8 systemd[1]: fix-xdg-state@1006.service: Deactivated successfully.
Jun  6 11:50:25 ns8 systemd[1]: Finished Create User-State-Private link.
Jun  6 11:50:25 ns8 bash[964]: '.local/state' -> '../.config'
Jun  6 11:50:25 ns8 systemd[1]: fix-xdg-state@1001.service: Deactivated successfully.
Jun  6 11:50:25 ns8 systemd[1]: Finished Create User-State-Private link.
Jun  6 11:50:25 ns8 chronyd[984]: chronyd version 4.8 starting (+CMDMON +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +NTS +SECHASH +IPV6 +DEBUG)
Jun  6 11:50:25 ns8 chronyd[984]: Loaded 0 symmetric keys
Jun  6 11:50:25 ns8 chronyd[984]: Using right/UTC timezone to obtain leap second data
Jun  6 11:50:25 ns8 chronyd[984]: Frequency -21.807 +/- 0.198 ppm read from /var/lib/chrony/drift
Jun  6 11:50:25 ns8 chronyd[984]: Loaded seccomp filter (level 2)
Jun  6 11:50:25 ns8 systemd[1]: Started NTP client/server.
Jun  6 11:50:25 ns8 systemd[1]: var-lib-containers-storage-overlay-compat2401981803-merged.mount: Deactivated successfully.
Jun  6 11:50:26 ns8 kernel: evm: overlay not supported
Jun  6 11:50:26 ns8 podman[918]: 2026-06-06 11:50:26.961641893 +0200 CEST m=+1.547335227 system refresh
Jun  6 11:50:27 ns8 podman[917]: 2026-06-06 11:50:27.061311723 +0200 CEST m=+1.646974597 image pull 41f2dde5d28ac476434f6f84b246cec1da16297cf42996ed0812d5175645e583 ghcr.io/nethserver/restic:3.19.1
--
Jun  6 11:50:41 ns8 systemd[1]: Starting Create User-State-Private link...
Jun  6 11:50:41 ns8 systemd[1]: Starting User Runtime Directory /run/user/0...
Jun  6 11:50:41 ns8 systemd-logind[912]: New session 5 of user root.
Jun  6 11:50:41 ns8 systemd[1]: fix-xdg-state@0.service: Deactivated successfully.
Jun  6 11:50:41 ns8 systemd[1]: Finished Create User-State-Private link.
Jun  6 11:50:41 ns8 systemd[1]: Finished User Runtime Directory /run/user/0.
Jun  6 11:50:41 ns8 systemd[1]: Starting User Manager for UID 0...
Jun  6 11:50:41 ns8 systemd[6911]: Queued start job for default target Main User Target.
Jun  6 11:50:41 ns8 systemd[6911]: Created slice User Application Slice.
Jun  6 11:50:41 ns8 systemd[6911]: Mark boot as successful after the user session has run 2 minutes was skipped because of an unmet condition check (ConditionUser=!@system).
Jun  6 11:50:41 ns8 systemd[6911]: Started Daily Cleanup of User's Temporary Directories.
Jun  6 11:50:41 ns8 systemd[6911]: Reached target Paths.
Jun  6 11:50:41 ns8 systemd[6911]: Reached target Timers.
Jun  6 11:50:41 ns8 systemd[6911]: Starting D-Bus User Message Bus Socket...
Jun  6 11:50:41 ns8 systemd[6911]: Starting Create User's Volatile Files and Directories...
--
Jun  6 12:00:53 ns8 systemd[1]: Starting Create User-State-Private link...
Jun  6 12:00:53 ns8 redis[3632]: 34:C 06 Jun 2026 10:00:53.958 * BGSAVE done, 539 keys saved, 0 keys skipped, 515498 bytes written.
Jun  6 12:00:53 ns8 systemd[1]: fix-xdg-state@1005.service: Deactivated successfully.
Jun  6 12:00:53 ns8 systemd[1]: Finished Create User-State-Private link.
Jun  6 12:00:53 ns8 systemd[1]: Starting User Manager for UID 1005...
Jun  6 12:00:54 ns8 redis[3632]: 34:C 06 Jun 2026 10:00:54.042 * DB saved on disk
Jun  6 12:00:54 ns8 redis[3632]: 34:C 06 Jun 2026 10:00:54.043 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB
Jun  6 12:00:54 ns8 redis[3632]: 1:M 06 Jun 2026 10:00:54.052 * Background saving terminated with success
Jun  6 12:00:54 ns8 systemd[18559]: Queued start job for default target Main User Target.
Jun  6 12:00:54 ns8 systemd[18559]: Created slice User Application Slice.
Jun  6 12:00:54 ns8 systemd[18559]: Started Backup II to ToucheAtout S3 bucket.
Jun  6 12:00:54 ns8 systemd[18559]: Started Mark boot as successful after the user session has run 2 minutes.
Jun  6 12:00:54 ns8 systemd[18559]: Started Daily Cleanup of User's Temporary Directories.
Jun  6 12:00:54 ns8 systemd[18559]: Reached target Paths.
--
Jun  6 12:14:55 ns8 systemd[1]: Starting Create User-State-Private link...
Jun  6 12:14:55 ns8 traefik[6900]: 2a02:a03f:a96d:d901:21d4:d4e5:b846:654c - - [06/Jun/2026:10:14:55 +0000] "GET /cluster-admin/api/node/1/task/bf51e60d-cd93-4862-ba57-00aa106025ad/context HTTP/2.0" 200 291 "-" "-" 808 "cluster-admin-https@file" "http://127.0.0.1:9311" 1ms
Jun  6 12:14:55 ns8 systemd[1]: fix-xdg-state@1005.service: Deactivated successfully.
Jun  6 12:14:55 ns8 systemd[1]: Finished Create User-State-Private link.
Jun  6 12:14:55 ns8 systemd[1]: Starting User Manager for UID 1005...
Jun  6 12:14:55 ns8 systemd[30389]: Queued start job for default target Main User Target.
Jun  6 12:14:55 ns8 systemd[30389]: Created slice User Application Slice.
Jun  6 12:14:55 ns8 systemd[30389]: Started Backup II to ToucheAtout S3 bucket.
Jun  6 12:14:55 ns8 systemd[30389]: Started Mark boot as successful after the user session has run 2 minutes.
Jun  6 12:14:55 ns8 systemd[30389]: Started Daily Cleanup of User's Temporary Directories.
Jun  6 12:14:55 ns8 systemd[30389]: Reached target Paths.
Jun  6 12:14:55 ns8 systemd[30389]: Reached target Timers.
Jun  6 12:14:55 ns8 systemd[30389]: Starting D-Bus User Message Bus Socket...
Jun  6 12:14:55 ns8 systemd[30389]: Starting Create User's Volatile Files and Directories...
--
Jun  6 12:16:11 ns8 systemd[1]: Starting Create User-State-Private link...
Jun  6 12:16:11 ns8 traefik[6900]: 2a02:a03f:a96d:d901:21d4:d4e5:b846:654c - - [06/Jun/2026:10:16:11 +0000] "GET /cluster-admin/api/node/1/task/db2ef34b-00b7-427a-b0f5-7223ef6b618f/context HTTP/2.0" 200 292 "-" "-" 942 "cluster-admin-https@file" "http://127.0.0.1:9311" 2ms
Jun  6 12:16:11 ns8 traefik[6900]: 2a02:a03f:a96d:d901:21d4:d4e5:b846:654c - - [06/Jun/2026:10:16:11 +0000] "GET /cluster-admin/api/node/1/task/db2ef34b-00b7-427a-b0f5-7223ef6b618f/context HTTP/2.0" 200 292 "-" "-" 943 "cluster-admin-https@file" "http://127.0.0.1:9311" 1ms
Jun  6 12:16:11 ns8 systemd[1]: fix-xdg-state@1005.service: Deactivated successfully.
Jun  6 12:16:11 ns8 systemd[1]: Finished Create User-State-Private link.
Jun  6 12:16:11 ns8 systemd[1]: Starting User Manager for UID 1005...
Jun  6 12:16:11 ns8 systemd[31497]: Queued start job for default target Main User Target.
Jun  6 12:16:11 ns8 systemd[31497]: Created slice User Application Slice.
Jun  6 12:16:11 ns8 systemd[31497]: Started Backup II to ToucheAtout S3 bucket.
Jun  6 12:16:11 ns8 systemd[31497]: Started Mark boot as successful after the user session has run 2 minutes.
Jun  6 12:16:11 ns8 systemd[31497]: Started Daily Cleanup of User's Temporary Directories.
Jun  6 12:16:11 ns8 systemd[31497]: Reached target Paths.
Jun  6 12:16:11 ns8 systemd[31497]: Reached target Timers.
Jun  6 12:16:11 ns8 systemd[31497]: Starting D-Bus User Message Bus Socket...
Jun  6 12:16:11 ns8 systemd[31497]: Starting Create User's Volatile Files and Directories...
--
Jun  6 12:22:01 ns8 systemd[1]: Starting Create User-State-Private link...
Jun  6 12:22:01 ns8 systemd[1]: fix-xdg-state@1005.service: Deactivated successfully.
Jun  6 12:22:01 ns8 systemd[1]: Finished Create User-State-Private link.
Jun  6 12:22:01 ns8 systemd[1]: Starting User Manager for UID 1005...
Jun  6 12:22:01 ns8 qemu-ga[903]: info: guest-ping called
Jun  6 12:22:01 ns8 systemd[36367]: Queued start job for default target Main User Target.
Jun  6 12:22:01 ns8 systemd[36367]: Created slice User Application Slice.
Jun  6 12:22:01 ns8 systemd[36367]: Started Backup II to ToucheAtout S3 bucket.
Jun  6 12:22:01 ns8 systemd[36367]: Started Mark boot as successful after the user session has run 2 minutes.
Jun  6 12:22:01 ns8 systemd[36367]: Started Daily Cleanup of User's Temporary Directories.
Jun  6 12:22:01 ns8 systemd[36367]: Reached target Paths.
Jun  6 12:22:01 ns8 systemd[36367]: Reached target Timers.
Jun  6 12:22:01 ns8 systemd[36367]: Starting D-Bus User Message Bus Socket...
--
Jun  6 15:44:13 ns8 systemd[1]: Starting Create User-State-Private link...
Jun  6 15:44:13 ns8 systemd[1]: Starting User Runtime Directory /run/user/0...
Jun  6 15:44:13 ns8 systemd-logind[912]: New session 31 of user root.
Jun  6 15:44:13 ns8 systemd[1]: fix-xdg-state@0.service: Deactivated successfully.
Jun  6 15:44:13 ns8 systemd[1]: Finished Create User-State-Private link.
Jun  6 15:44:13 ns8 systemd[1]: Finished User Runtime Directory /run/user/0.
Jun  6 15:44:13 ns8 systemd[1]: Starting User Manager for UID 0...
Jun  6 15:44:13 ns8 systemd[122341]: Queued start job for default target Main User Target.
Jun  6 15:44:13 ns8 systemd[122341]: Created slice User Application Slice.
Jun  6 15:44:13 ns8 systemd[122341]: Mark boot as successful after the user session has run 2 minutes was skipped because of an unmet condition check (ConditionUser=!@system).
Jun  6 15:44:13 ns8 systemd[122341]: Started Daily Cleanup of User's Temporary Directories.
Jun  6 15:44:13 ns8 systemd[122341]: Reached target Paths.
Jun  6 15:44:13 ns8 systemd[122341]: Reached target Timers.
Jun  6 15:44:13 ns8 systemd[122341]: Starting D-Bus User Message Bus Socket...
Jun  6 15:44:13 ns8 systemd[122341]: Starting Create User's Volatile Files and Directories...

Here is the first tentative to restart nextcloud1 after the fix was applied (nextcloud2 started successfully).

grep -A 30 "User Manager for UID 1005" /var/log/messages-20260607 | head -50 
Jun  6 11:51:20 ns8 systemd[1]: Starting User Manager for UID 1005...
Jun  6 11:51:21 ns8 systemd[7998]: Queued start job for default target Main User Target.
Jun  6 11:51:21 ns8 systemd[7998]: Created slice User Application Slice.
Jun  6 11:51:21 ns8 systemd[7998]: Started Backup II to ToucheAtout S3 bucket.
Jun  6 11:51:21 ns8 systemd[7998]: Started Mark boot as successful after the user session has run 2 minutes.
Jun  6 11:51:21 ns8 systemd[7998]: Started Daily Cleanup of User's Temporary Directories.
Jun  6 11:51:21 ns8 systemd[7998]: Reached target Paths.
Jun  6 11:51:21 ns8 systemd[7998]: Reached target Timers.
Jun  6 11:51:21 ns8 systemd[7998]: Starting D-Bus User Message Bus Socket...
Jun  6 11:51:21 ns8 systemd[7998]: Starting Create User's Volatile Files and Directories...
Jun  6 11:51:21 ns8 systemd[7998]: Listening on D-Bus User Message Bus Socket.
Jun  6 11:51:21 ns8 systemd[7998]: Reached target Sockets.
Jun  6 11:51:21 ns8 systemd[7998]: Finished Create User's Volatile Files and Directories.
Jun  6 11:51:21 ns8 systemd[7998]: Reached target Basic System.
Jun  6 11:51:21 ns8 systemd[7998]: Starting Rootless module/nextcloud1 agent...
Jun  6 11:51:21 ns8 systemd[7998]: nextcloud.service: Failed to load environment files: No such file or directory
Jun  6 11:51:21 ns8 systemd[7998]: nextcloud.service: Failed to run 'start-pre' task: No such file or directory
Jun  6 11:51:21 ns8 systemd[7998]: nextcloud.service: Failed with result 'resources'.
Jun  6 11:51:21 ns8 systemd[7998]: Failed to start Podman nextcloud.service.
Jun  6 11:51:21 ns8 systemd[7998]: Dependency failed for Podman nextcloud-app.service.
Jun  6 11:51:21 ns8 systemd[7998]: nextcloud-app.service: Job nextcloud-app.service/start failed with result 'dependency'.
Jun  6 11:51:21 ns8 systemd[7998]: nextcloud-db.service: Failed to load environment files: No such file or directory
Jun  6 11:51:21 ns8 systemd[7998]: nextcloud-db.service: Failed to run 'start-pre' task: No such file or directory
Jun  6 11:51:21 ns8 systemd[7998]: nextcloud-db.service: Failed with result 'resources'.
Jun  6 11:51:21 ns8 systemd[7998]: Failed to start Podman nextcloud-db.service.
Jun  6 11:51:21 ns8 systemd[7998]: nextcloud-nginx.service: Failed to load environment files: No such file or directory
Jun  6 11:51:21 ns8 systemd[7998]: nextcloud-nginx.service: Failed to run 'start-pre' task: No such file or directory
Jun  6 11:51:21 ns8 systemd[7998]: nextcloud-nginx.service: Failed with result 'resources'.
Jun  6 11:51:21 ns8 systemd[7998]: Failed to start Podman nextcloud-nginx.service.
Jun  6 11:51:21 ns8 systemd[7998]: nextcloud-notify_push.service: Failed to load environment files: No such file or directory
Jun  6 11:51:21 ns8 systemd[7998]: nextcloud-notify_push.service: Failed to run 'start-pre' task: No such file or directory
--
Jun  6 11:51:21 ns8 systemd[1]: Started User Manager for UID 1005.
Jun  6 11:51:21 ns8 systemd[1]: Starting User Manager for UID 1008...
Jun  6 11:51:22 ns8 redis[3632]: 1:M 06 Jun 2026 09:51:22.007 * 1 changes in 5 seconds. Saving...
Jun  6 11:51:22 ns8 redis[3632]: 1:M 06 Jun 2026 09:51:22.008 * Background saving started by pid 24
Jun  6 11:51:22 ns8 redis[3632]: 24:C 06 Jun 2026 09:51:22.013 * BGSAVE done, 454 keys saved, 0 keys skipped, 436248 bytes written.
Jun  6 11:51:22 ns8 systemd[7998]: nextcloud-db.service: Scheduled restart job, restart counter is at 2.
Jun  6 11:51:22 ns8 systemd[7998]: nextcloud-nginx.service: Scheduled restart job, restart counter is at 2.
Jun  6 11:51:22 ns8 systemd[7998]: nextcloud-notify_push.service: Scheduled restart job, restart counter is at 2.
Jun  6 11:51:22 ns8 systemd[7998]: nextcloud.service: Scheduled restart job, restart counter is at 2.
Jun  6 11:51:22 ns8 systemd[7998]: nextcloud-redis.service: Scheduled restart job, restart counter is at 2.
Jun  6 11:51:22 ns8 systemd[7998]: Stopped Podman nextcloud-db.service.
Jun  6 11:51:22 ns8 systemd[7998]: Stopped Podman nextcloud-nginx.service.
Jun  6 11:51:22 ns8 systemd[7998]: Stopped Podman nextcloud-notify_push.service.
Jun  6 11:51:22 ns8 systemd[7998]: nextcloud-notify_push.service: Failed to load environment files: No such file or directory
Jun  6 11:51:22 ns8 systemd[7998]: nextcloud-notify_push.service: Failed to run 'start-pre' task: No such file or directory
Jun  6 11:51:22 ns8 systemd[7998]: nextcloud-notify_push.service: Failed with result 'resources'.
Jun  6 11:51:22 ns8 systemd[7998]: Failed to start Podman nextcloud-notify_push.service.
Jun  6 11:51:22 ns8 systemd[7998]: Stopped Podman nextcloud-redis.service.
.

davidep · June 7, 2026, 10:39am

Thanks, this is very useful.

For “nextcloud1” (UID 1005), I don’t see the expected output from the “ln -sfv” command when the compatibility fix was applied. For several other application users, including “nextcloud2” (UID 1006), the log contains lines like:

bash[968]: '.local/state' -> '../.config'
fix-xdg-state@1006.service: Deactivated successfully.

For UID 1005, I only see:

fix-xdg-state@1005.service: Deactivated successfully.
Finished Create User-State-Private link.

This may indicate that “~nextcloud1/.local/state” already existed at the time the workaround was applied, preventing the expected symlink from being created.

Could you check:

ls -ld ~nextcloud1/.local ~nextcloud1/.local/state
readlink -v ~nextcloud1/.local/state
stat  ~nextcloud1/.local/state

A few additional questions that may help explain the difference between “nextcloud1” and “nextcloud2”:

Did you ever manually create “~nextcloud1/.local/state”?
Is “nextcloud1” older than “nextcloud2”, for example migrated from an earlier NS8 release?
Was “nextcloud1” ever restored from backup?
Did you mount any additional disk, bind mount, or custom storage under “~nextcloud1/.local”, “~nextcloud1/.local/state”, or a subdirectory of that path?

At the moment my leading hypothesis is that something already existed at “~nextcloud1/.local/state” when the fix ran, causing a different result than for “nextcloud2”.

Please, check in nextcloud1 history the exact timestamp of that command:

runagent -m nextcloud1 history | grep mkdir

Or

grep mkdir ~nextcloud1/.bash_history

pagaille · June 7, 2026, 2:07pm

Sadly I fixed it manually following you’re advice this morning by doing rm -rf ~/.local/state && ln -s ../.config ~/.local/state; the history is lost I guess.

[root@ns8 ~]# ls -ld ~nextcloud1/.local ~nextcloud1/.local/state
drwx------. 3 nextcloud1 nextcloud1 4096 Jun 7 07:57 /home/nextcloud1/.local
lrwxrwxrwx. 1 nextcloud1 nextcloud1 10 Jun 7 07:57 /home/nextcloud1/.local/state → ../.config
[root@ns8 ~]# stat ~nextcloud1/.local/state
File: /home/nextcloud1/.local/state → ../.config
Size: 10 Blocks: 0 IO Block: 4096 symbolic link
Device: fd00h/64768d Inode: 2235378 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 1005/nextcloud1) Gid: ( 1005/nextcloud1)
Context: unconfined_u:object_r:gconf_home_t:s0
Access: 2026-06-07 07:57:29.723878079 +0200
Modify: 2026-06-07 07:57:25.270780669 +0200
Change: 2026-06-07 07:57:25.270780669 +0200
Birth: 2026-06-07 07:57:25.270780669 +0200
[root@ns8 ~]# readlink -v ~nextcloud1/.local/state
../.config

Maybe restoring a backup could give us an answer ? Sadly I didn’t took a snapshot before upgrading (actually I did but on a wrong VM )

49967 2026-06-06 12:18:40 runagent -m nextcloud1 bash -c ‘mkdir -p ~/.local/state && ln -sfn ~/.config/state ~/.local/state/state’

If needed I might create you an account so that you can ssh and investigate.

davidep · June 8, 2026, 3:12pm

Thank you for taking the time to investigate and provide all those details.

At this point, since the original ~nextcloud1/.local/state entry has been removed and replaced with the correct symlink, I don’t think we can learn much more from this specific node. The key evidence that could explain why the path existed in the first place is unfortunately no longer available.

From our phonehome statistics, we already have several dozen installations running Rocky Linux or AlmaLinux 9.8. With the current fix included in Core 3.19.1, it is important that we react promptly if applications fail to start after a distribution upgrade, so reports like yours are extremely valuable.

We have also prepared a second, more complex fix^[1] that handles pre-existing ~/.local/state paths. However, it does so by silently moving the existing path aside to a backup location before creating the expected symlink. While that would likely have avoided the issue you encountered, it would also hide the original cause and make future investigations more difficult.

For the moment, understanding whether there are still scenarios not covered by the current fix is more important than automatically masking them. Let’s see how things go over the next few days and collect additional feedback.

As always, creating a VM snapshot and/or an application backup before a major distribution upgrade is strongly recommended. It provides a straightforward rollback path and preserves valuable evidence if something unexpected happens.

According to the planned schedule, Rocky Linux 9.8 will start rolling out through the NS8 mirrors tomorrow. Please continue to report any startup failures or other anomalies after the upgrade so we can quickly identify and address any remaining edge cases.

fix: backup .local/state before creating the symlink by DavidePrincipi · Pull Request #1198 · NethServer/ns8-core · GitHub ↩︎

pagaille · June 10, 2026, 5:49am

Damn. One of my servers was configured to apply updates automatically and services are not starting anymore

I tried to update the core manually but it doesn’t work. What should

 run update-core --data '{"core_url":"ghcr.io/nethserver/core:latest","nodes":[1]}'
_acontrol_task request attempt failed (Error 111 connecting to cluster-leader:6379. Connect call failed ('127.0.0.1', 6379).). Retrying...
_acontrol_task request recovered successfully at attempt 2
_acontrol_task request attempt failed (Error 111 connecting to cluster-leader:6379. Connect call failed ('127.0.0.1', 6379).). Retrying...
_acontrol_task request recovered successfully at attempt 2
<7>run-scriptdir /var/lib/nethserver/cluster/update-core-pre-modules.d/
Running /var/lib/nethserver/cluster/update-core-pre-modules.d/50update_grants...
Task cluster/update-module run failed: {'output': '', 'error': 'Traceback (most recent call last):\n  File "/var/lib/nethserver/cluster/actions/update-module/50update", line 40, in <module>\n    ping_errors = agent.tasks.runp_brief([{"agent_id": f"module/{mid}", "action": "list-actions"} for mid in instances],\n                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/agent/pypkg/agent/tasks/run.py", line 61, in runp_brief\n    results = asyncio.run(_runp(tasks, **kwargs))\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/lib64/python3.11/asyncio/runners.py", line 190, in run\n    return runner.run(main)\n           ^^^^^^^^^^^^^^^^\n  File "/usr/lib64/python3.11/asyncio/runners.py", line 118, in run\n    return self._loop.run_until_complete(task)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/lib64/python3.11/asyncio/base_events.py", line 654, in run_until_complete\n    return future.result()\n           ^^^^^^^^^^^^^^^\n  File "/usr/local/agent/pypkg/agent/tasks/run.py", line 131, in _runp\n    return await asyncio.gather(*runners, return_exceptions=(len(tasks) > 1))\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/agent/pypkg/agent/tasks/run.py", line 137, in _run_with_protocol\n    return await run_redisclient(taskrq, **pconn)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/agent/pypkg/agent/tasks/redisclient.py", line 77, in run_redisclient\n    await _task_submission_check_client_idle(rdb, taskrq, kwargs[\'check_idle_time\'])\n  File "/usr/local/agent/pypkg/agent/tasks/redisclient.py", line 41, in _task_submission_check_client_idle\n    raise TaskSubmissionCheckFailed(f"Client \\"{taskrq[\'agent_id\']}\\" was not found")\nagent.tasks.exceptions.TaskSubmissionCheckFailed: Client "module/ldapproxy1" was not found\n', 'exit_code': 1}
<7>run-scriptdir /var/lib/nethserver/cluster/update-core-post-modules.d/
Running /var/lib/nethserver/cluster/update-core-post-modules.d/70domain_notification...
Running /var/lib/nethserver/cluster/update-core-post-modules.d/70metrics...
Running /var/lib/nethserver/cluster/update-core-post-modules.d/90disable_testing...
update-core failed in some core modules
  File "/var/lib/nethserver/cluster/actions/update-core/70update_modules", line 48, in <module>
    agent.assert_exp(update_module_errors == 0, 'update-core failed in some core modules')
""

davidep · June 10, 2026, 6:36am

This may be related or not to the rocky Linux 9.8 systemd issue.

One of your core modules has agent.service or the whole user session down. To avoid issues the upgrade-core action aborts immediately.

Check what’s running:

 runagent -l

It should match the list of installed core modules on that node, as documented here: Software center — NS8 documentation

davidep · June 10, 2026, 7:22am

I missed that on the smartphone screen.

The stopped module is ldapproxy1.

The above command is not correct: never use the latest tag, as it is intended for testing purposes only.

Since the operation was not forced, no update was actually performed. At this point, the only information we have gained is that ldapproxy1 is not running.

What is the currently installed core version on the node? Was core 3.19.1 already installed before the upgrade to Rocky Linux 9.8? Was the node rebooted?

pagaille · June 10, 2026, 9:25am

Solved, sorry I was in a hurry this morning and the message I wrote wasn’t sent.

I had to apply the fix to traefik AND ldapproxy indeed

ln -sfv ../.config ~traefik1/.local/state
systemctl restart user@$(id -u traefik1)
ln -sfv ../.config ~ldapproxy1/.local/state
systemctl restart user@$(id -u ldapproxy1)

Then i managed to update the core from the command line, then again from the ui (probably because of the latest tag as you said)

Some apps like mattermost still didn’t started, I had to upgrade them too.

Everything up and running.

thanks Davide

M

pagaille · June 10, 2026, 9:27am

Actually I found it in the official doc Updates | NS8 dev manual

EDIT / That’s the Dev manual !! : it was the first link found in google.

davidep · June 10, 2026, 10:04am

Good catch, and thanks for the reference! I’ve updated the Developer’s Manual.

Glad the workaround solved the upgrade issue.

One question: was Core 3.19.1 installed before the Rocky Linux 9.8 upgrade, or was the node still running an older Core version? That would help us understand why this second server hit the bug.

pagaille · June 10, 2026, 10:07am

Automatic OS update was enabled (the setting was found in cockpit), so I believe that 9.8 was applied and server rebooted, but not the core. That would explain why the fix wasn’t applied.

I disabled those automatic updates.

Since the operation was not forced, no update was actually performed.

So you mean that one should add –-force to the above command ? It isn’t mentioned in the dev doc either I believe.

davidep · June 10, 2026, 10:34am

I agree with your reconstruction of the timeline: it looks like Rocky Linux 9.8 was installed automatically before Core 3.19.1 had a chance to apply the compatibility fix.

Regarding the --force flag, let’s avoid going too far off-topic here. The flag is intentionally undocumented. It exists mainly for development, testing, and critical support scenarios where an update must be re-applied regardless of the current version.

Keeping it out of the public documentation also reduces the chance of AI-generated suggestions or copy/pasted commands using it unnecessarily on production systems.