After an automatic security update from Rocky Linux and a automatic reboot, Nethserver is no longer working; I can no longer log in and it seems that none of the services are working either. I urgently need help.
Hi Erwin, sorry for this.
Read carefully this thread and before applying the workarounds try to gather information, if possible.
Please share your findings then, thanks in advance
Hi David,
Thank you very much for your quick help, and sorry for the late reply – I had to rush off to an appointment and was just glad that Nethserver was working again.
The workarounds worked; I can log in to Nethserver again and all services seem to be running correctly.
Why wasn’t the NS8 Core 3.19.1 update installed automatically beforehand? If it had been, this problem wouldn’t have arisen at all – or am I mistaken?
Can I now update to Rocky Linux 9.8 without any issues, or could further problems arise?
Best, Erwin
Hi Erwin,
I’m glad to hear the workaround was successful and that you can access NS8 again and all services appear to be running normally.
Regarding your question, automatic DNF updates and NS8 Core updates are managed separately. Even when automatic operating system updates are enabled by the system administrator (by default they are manual), they do not automatically install NS8 Core updates. Core updates remain a manual operation unless the system has an active subscription that enables automated OS, core, and application updates.
My suspicion is that the Rocky Linux upgrade from 9.7 to 9.8 was installed automatically, while the NS8 Core was still at version 3.19.0. In that situation, the node would be exposed to the compatibility issue addressed by Core 3.19.1, which would explain the symptoms you experienced after the reboot.
Can you confirm which NS8 Core version was installed at the time Rocky Linux was upgraded to 9.8?
As for upgrading to Rocky Linux 9.8 now, once Core 3.19.1 (or later) is installed, the specific systemd compatibility issue should be resolved. However, let’s first verify the versions involved so we can better understand exactly what happened on your system.
Thanks in advance for the additional information.
Hi David,
I have an active subscription, which is precisely why I’m surprised that this update wasn’t installed automatically in good time.
I haven’t yet completed the installation of Rocky Linux 9.8, but it seems an urgent security update was installed automatically last night and the server was restarted afterwards. At that time, Core 3.19.0 was still active, which then led to this problem.
If the Core 3.19.1 update had been installed automatically beforehand, the problem wouldn’t have arisen in the first place.
I have installed the Core 3.19.1 update manually, so the way should now be clear for Rocky Linux 9.8.
Best, Erwin
Hi Erwin,
Thank you for the additional details.
A very similar issue has already been reported on another system with an active subscription. In that case, we found that a concurrent OS update mechanism was in place: the dnf-automatic package was installing Rocky Linux upstream updates overnight, independently of NS8’s own update workflow.
The problem is that the two update agents are not compatible with each other. While NS8’s apply-updates.timer is designed to coordinate application and operating system updates in the correct order, dnf-automatic can pull Rocky Linux updates in advance, bypassing those safeguards. This could explain why your system received the Rocky Linux update while Core 3.19.0 was still installed.
One possibility is that dnf-automatic was installed through Cockpit, as Cockpit can enable automatic operating system updates by installing and configuring this package.
Could you please share the output of the following commands?
-
Check if and when
dnf-automaticwas installed:dnf history info dnf-automatic -
Check which update timers are active and when they last ran:
systemctl list-timers
In particular, it would be useful to understand:
- Whether
dnf-automaticis installed and how it got there. - Whether it may have been enabled from Cockpit.
- Whether its timer was active at the time of the incident.
- Whether
apply-updates.timerwas running normally. - Whether
dnf-automaticupdate agent actually triggered the overnight reboot.
With that information we should be able to reconstruct the sequence of events and understand why the Core update was not applied before the Rocky Linux update.
Please share your findings when you have a chance. Thanks in advance.
Hi David,
In Cockpit, I set the automatic updates for every day to 6:30 a.m. and enabled Kerner Live Patching.
I specifically set it for 6:30 a.m. so it wouldn’t conflict with Nethserver’s nightly automatic updates.
systemctl list-timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Wed 2026-06-10 13:25:52 CEST 17min left Wed 2026-06-10 11:52:01 CEST 1h 15min ago dnf-makecache.timer dnf-makecache.service
Thu 2026-06-11 00:00:00 CEST 10h left Wed 2026-06-10 00:00:00 CEST 13h ago logrotate.timer logrotate.service
Thu 2026-06-11 00:00:00 CEST 10h left Wed 2026-06-10 00:00:00 CEST 13h ago mlocate-updatedb.timer mlocate-updatedb.service
Thu 2026-06-11 00:16:40 CEST 11h left Wed 2026-06-10 00:07:12 CEST 13h ago password-warning.timer password-warning.service
Thu 2026-06-11 00:47:33 CEST 11h left Wed 2026-06-10 00:46:59 CEST 12h ago phonehome.timer phonehome.service
Thu 2026-06-11 01:22:03 CEST 12h left Wed 2026-06-10 01:22:12 CEST 11h ago send-inventory.timer send-inventory.service
Thu 2026-06-11 02:52:29 CEST 13h left Wed 2026-06-10 02:52:29 CEST 10h ago apply-updates.timer apply-updates.service
Thu 2026-06-11 07:03:01 CEST 17h left Wed 2026-06-10 07:25:59 CEST 5h 41min ago dnf-automatic-install.timer dnf-automatic-install.service
Thu 2026-06-11 08:42:35 CEST 19h left Wed 2026-06-10 08:42:35 CEST 4h 25min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
9 timers listed.
dnf history info dnf-automatic
Upgraded python3-3.9.18-1.el9_3.1.x86_64 @@System
Upgrade tuned-2.22.1-1.el9.noarch @ns-baseos
Upgraded tuned-2.21.0-1.el9_3.noarch @@System
Upgrade libsepol-3.6-1.el9.x86_64 @ns-baseos
Upgraded libsepol-3.5-1.el9.x86_64 @@System
Upgrade libss-1.46.5-5.el9.x86_64 @ns-baseos
Upgraded libss-1.46.5-3.el9.x86_64 @@System
Upgrade libcom_err-1.46.5-5.el9.x86_64 @ns-baseos
Upgraded libcom_err-1.46.5-3.el9.x86_64 @@System
Upgrade e2fsprogs-libs-1.46.5-5.el9.x86_64 @ns-baseos
Upgraded e2fsprogs-libs-1.46.5-3.el9.x86_64 @@System
Upgrade e2fsprogs-1.46.5-5.el9.x86_64 @ns-baseos
Upgraded e2fsprogs-1.46.5-3.el9.x86_64 @@System
Upgrade cockpit-ws-311.1-1.el9.x86_64 @ns-baseos
Upgraded cockpit-ws-300.4-1.el9_3.x86_64 @@System
Upgrade cockpit-bridge-311.1-1.el9.x86_64 @ns-baseos
Upgraded cockpit-bridge-300.4-1.el9_3.x86_64 @@System
Upgrade cockpit-311.1-1.el9.x86_64 @ns-baseos
Upgraded cockpit-300.4-1.el9_3.x86_64 @@System
Upgrade cockpit-system-311.1-1.el9.noarch @ns-baseos
Upgraded cockpit-system-300.4-1.el9_3.noarch @@System
Upgrade libnl3-cli-3.9.0-1.el9.x86_64 @ns-baseos
Upgraded libnl3-cli-3.7.0-1.el9.x86_64 @@System
Upgrade libnl3-3.9.0-1.el9.x86_64 @ns-baseos
Upgraded libnl3-3.7.0-1.el9.x86_64 @@System
Upgrade libsemanage-3.6-1.el9.x86_64 @ns-baseos
Upgraded libsemanage-3.5-2.el9.x86_64 @@System
Upgrade libselinux-utils-3.6-1.el9.x86_64 @ns-baseos
Upgraded libselinux-utils-3.5-1.el9.x86_64 @@System
Upgrade libselinux-3.6-1.el9.x86_64 @ns-baseos
Upgraded libselinux-3.5-1.el9.x86_64 @@System
Upgrade iscsi-initiator-utils-iscsiuio-6.2.1.9-1.gita65a472.el9.x86_64 @ns-baseos
Upgraded iscsi-initiator-utils-iscsiuio-6.2.1.4-3.git2a8f9d8.el9.x86_64 @@System
Upgrade iscsi-initiator-utils-6.2.1.9-1.gita65a472.el9.x86_64 @ns-baseos
Upgraded iscsi-initiator-utils-6.2.1.4-3.git2a8f9d8.el9.x86_64 @@System
Upgrade dracut-squash-057-53.git20240104.el9.x86_64 @ns-baseos
Upgraded dracut-squash-057-44.git20230822.el9.x86_64 @@System
Upgrade dracut-network-057-53.git20240104.el9.x86_64 @ns-baseos
Upgraded dracut-network-057-44.git20230822.el9.x86_64 @@System
Upgrade dracut-config-rescue-057-53.git20240104.el9.x86_64 @ns-baseos
Upgraded dracut-config-rescue-057-44.git20230822.el9.x86_64 @@System
Upgrade dracut-057-53.git20240104.el9.x86_64 @ns-baseos
Upgraded dracut-057-44.git20230822.el9.x86_64 @@System
Upgrade chrony-4.5-1.el9.x86_64 @ns-baseos
Upgraded chrony-4.3-1.el9.x86_64 @@System
Upgrade numactl-libs-2.0.16-3.el9.x86_64 @ns-baseos
Upgraded numactl-libs-2.0.16-1.el9.x86_64 @@System
Upgrade openssh-server-8.7p1-38.el9.x86_64 @ns-baseos
Upgraded openssh-server-8.7p1-34.el9_3.3.x86_64 @@System
Upgrade openssh-clients-8.7p1-38.el9.x86_64 @ns-baseos
Upgraded openssh-clients-8.7p1-34.el9_3.3.x86_64 @@System
Upgrade openssh-8.7p1-38.el9.x86_64 @ns-baseos
Upgraded openssh-8.7p1-34.el9_3.3.x86_64 @@System
Upgrade python3-nftables-1:1.0.9-1.el9.x86_64 @ns-baseos
Upgraded python3-nftables-1:1.0.4-11.el9_3.x86_64 @@System
Upgrade nftables-1:1.0.9-1.el9.x86_64 @ns-baseos
Upgraded nftables-1:1.0.4-11.el9_3.x86_64 @@System
Upgrade librepo-1.14.5-2.el9.x86_64 @ns-baseos
Upgraded librepo-1.14.5-1.el9.x86_64 @@System
Upgrade cronie-anacron-1.5.7-11.el9.x86_64 @ns-baseos
Upgraded cronie-anacron-1.5.7-8.el9.x86_64 @@System
Upgrade cronie-1.5.7-11.el9.x86_64 @ns-baseos
Upgraded cronie-1.5.7-8.el9.x86_64 @@System
Upgrade python3-setools-4.4.4-1.el9.x86_64 @ns-baseos
Upgraded python3-setools-4.4.3-1.el9.x86_64 @@System
Upgrade smartmontools-1:7.2-9.el9.x86_64 @ns-baseos
Upgraded smartmontools-1:7.2-7.el9.x86_64 @@System
Upgrade p11-kit-trust-0.25.3-2.el9.x86_64 @ns-baseos
Upgraded p11-kit-trust-0.24.1-2.el9.x86_64 @@System
Upgrade p11-kit-0.25.3-2.el9.x86_64 @ns-baseos
Upgraded p11-kit-0.24.1-2.el9.x86_64 @@System
Upgrade elfutils-libs-0.190-2.el9.x86_64 @ns-baseos
Upgraded elfutils-libs-0.189-3.el9.x86_64 @@System
Upgrade elfutils-libelf-0.190-2.el9.x86_64 @ns-baseos
Upgraded elfutils-libelf-0.189-3.el9.x86_64 @@System
Upgrade elfutils-debuginfod-client-0.190-2.el9.x86_64 @ns-baseos
Upgraded elfutils-debuginfod-client-0.189-3.el9.x86_64 @@System
Upgrade elfutils-default-yama-scope-0.190-2.el9.noarch @ns-baseos
Upgraded elfutils-default-yama-scope-0.189-3.el9.noarch @@System
Upgrade libkcapi-hmaccalc-1.4.0-2.el9.x86_64 @ns-baseos
Upgraded libkcapi-hmaccalc-1.3.1-3.el9.x86_64 @@System
Upgrade libkcapi-1.4.0-2.el9.x86_64 @ns-baseos
Upgraded libkcapi-1.3.1-3.el9.x86_64 @@System
Upgrade libcurl-7.76.1-29.el9_4.x86_64 @ns-baseos
Upgraded libcurl-7.76.1-26.el9_3.3.x86_64 @@System
Upgrade curl-7.76.1-29.el9_4.x86_64 @ns-baseos
Upgraded curl-7.76.1-26.el9_3.3.x86_64 @@System
Upgrade openldap-2.6.6-3.el9.x86_64 @ns-baseos
Upgraded openldap-2.6.3-1.el9.x86_64 @@System
Upgrade libssh-config-0.10.4-13.el9.noarch @ns-baseos
Upgraded libssh-config-0.10.4-12.el9_3.noarch @@System
Upgrade libssh-0.10.4-13.el9.x86_64 @ns-baseos
Upgraded libssh-0.10.4-12.el9_3.x86_64 @@System
Upgrade python3-dbus-1.2.18-2.el9.0.1.x86_64 @ns-baseos
Upgraded python3-dbus-1.2.18-2.el9.x86_64 @@System
Upgrade initscripts-rename-device-10.11.6-1.el9.x86_64 @ns-baseos
Upgraded initscripts-rename-device-10.11.5-1.el9.x86_64 @@System
Upgrade kpartx-0.8.7-27.el9.x86_64 @ns-baseos
Upgraded kpartx-0.8.7-22.el9.x86_64 @@System
Upgrade device-mapper-multipath-libs-0.8.7-27.el9.x86_64 @ns-baseos
Upgraded device-mapper-multipath-libs-0.8.7-22.el9.x86_64 @@System
Upgrade device-mapper-multipath-0.8.7-27.el9.x86_64 @ns-baseos
Upgraded device-mapper-multipath-0.8.7-22.el9.x86_64 @@System
Upgrade pam-1.5.1-19.el9.x86_64 @ns-baseos
Upgraded pam-1.5.1-15.el9.x86_64 @@System
Upgrade fuse-common-3.10.2-8.el9.x86_64 @ns-baseos
Upgraded fuse-common-3.10.2-6.el9.x86_64 @@System
Upgrade libstdc+±11.4.1-3.el9.x86_64 @ns-baseos
Upgraded libstdc+±11.4.1-2.1.el9.x86_64 @@System
Upgrade libgomp-11.4.1-3.el9.x86_64 @ns-baseos
Upgraded libgomp-11.4.1-2.1.el9.x86_64 @@System
Upgrade libgcc-11.4.1-3.el9.x86_64 @ns-baseos
Upgraded libgcc-11.4.1-2.1.el9.x86_64 @@System
Upgrade libatomic-11.4.1-3.el9.x86_64 @ns-baseos
Upgraded libatomic-11.4.1-2.1.el9.x86_64 @@System
Upgrade glibc-langpack-en-2.34-100.el9.x86_64 @ns-baseos
Upgraded glibc-langpack-en-2.34-83.el9.12.x86_64 @@System
Upgrade glibc-gconv-extra-2.34-100.el9.x86_64 @ns-baseos
Upgraded glibc-gconv-extra-2.34-83.el9.12.x86_64 @@System
Upgrade glibc-common-2.34-100.el9.x86_64 @ns-baseos
Upgraded glibc-common-2.34-83.el9.12.x86_64 @@System
Upgrade glibc-2.34-100.el9.x86_64 @ns-baseos
Upgraded glibc-2.34-83.el9.12.x86_64 @@System
Upgrade util-linux-user-2.37.4-18.el9.x86_64 @ns-baseos
Upgraded util-linux-user-2.37.4-15.el9.x86_64 @@System
Upgrade util-linux-core-2.37.4-18.el9.x86_64 @ns-baseos
Upgraded util-linux-core-2.37.4-15.el9.x86_64 @@System
Upgrade util-linux-2.37.4-18.el9.x86_64 @ns-baseos
Upgraded util-linux-2.37.4-15.el9.x86_64 @@System
Upgrade libuuid-2.37.4-18.el9.x86_64 @ns-baseos
Upgraded libuuid-2.37.4-15.el9.x86_64 @@System
Upgrade libsmartcols-2.37.4-18.el9.x86_64 @ns-baseos
Upgraded libsmartcols-2.37.4-15.el9.x86_64 @@System
Upgrade libmount-2.37.4-18.el9.x86_64 @ns-baseos
Upgraded libmount-2.37.4-15.el9.x86_64 @@System
Upgrade libfdisk-2.37.4-18.el9.x86_64 @ns-baseos
Upgraded libfdisk-2.37.4-15.el9.x86_64 @@System
Upgrade libblkid-2.37.4-18.el9.x86_64 @ns-baseos
Upgraded libblkid-2.37.4-15.el9.x86_64 @@System
Upgrade xfsprogs-6.3.0-1.el9.x86_64 @ns-baseos
Upgraded xfsprogs-5.19.0-4.el9.x86_64 @@System
Upgrade NetworkManager-tui-1:1.46.0-4.el9_4.x86_64 @ns-baseos
Upgraded NetworkManager-tui-1:1.44.0-5.el9_3.x86_64 @@System
Upgrade NetworkManager-team-1:1.46.0-4.el9_4.x86_64 @ns-baseos
Upgraded NetworkManager-team-1:1.44.0-5.el9_3.x86_64 @@System
Upgrade NetworkManager-libnm-1:1.46.0-4.el9_4.x86_64 @ns-baseos
Upgraded NetworkManager-libnm-1:1.44.0-5.el9_3.x86_64 @@System
Upgrade NetworkManager-1:1.46.0-4.el9_4.x86_64 @ns-baseos
Upgraded NetworkManager-1:1.44.0-5.el9_3.x86_64 @@System
Upgrade python3-libdnf-0.69.0-8.el9.x86_64 @ns-baseos
Upgraded python3-libdnf-0.69.0-6.el9_3.x86_64 @@System
Upgrade python3-hawkey-0.69.0-8.el9.x86_64 @ns-baseos
Upgraded python3-hawkey-0.69.0-6.el9_3.x86_64 @@System
Upgrade libdnf-0.69.0-8.el9.x86_64 @ns-baseos
Upgraded libdnf-0.69.0-6.el9_3.x86_64 @@System
Upgrade lvm2-libs-9:2.03.23-2.el9.x86_64 @ns-baseos
Upgraded lvm2-libs-9:2.03.21-3.el9.x86_64 @@System
Upgrade lvm2-9:2.03.23-2.el9.x86_64 @ns-baseos
Upgraded lvm2-9:2.03.21-3.el9.x86_64 @@System
Upgrade device-mapper-libs-9:1.02.197-2.el9.x86_64 @ns-baseos
Upgraded device-mapper-libs-9:1.02.195-3.el9.x86_64 @@System
Upgrade device-mapper-event-libs-9:1.02.197-2.el9.x86_64 @ns-baseos
Upgraded device-mapper-event-libs-9:1.02.195-3.el9.x86_64 @@System
Upgrade device-mapper-event-9:1.02.197-2.el9.x86_64 @ns-baseos
Upgraded device-mapper-event-9:1.02.195-3.el9.x86_64 @@System
Upgrade device-mapper-9:1.02.197-2.el9.x86_64 @ns-baseos
Upgraded device-mapper-9:1.02.195-3.el9.x86_64 @@System
Upgrade libnvme-1.6-1.el9.x86_64 @ns-baseos
Upgraded libnvme-1.4-7.el9.x86_64 @@System
Upgrade rocky-repos-9.4-1.5.el9.noarch @ns-baseos
Upgraded rocky-repos-9.3-1.3.el9.noarch @@System
Upgrade rocky-release-9.4-1.5.el9.noarch @ns-baseos
Upgraded rocky-release-9.3-1.3.el9.noarch @@System
Upgrade rocky-gpg-keys-9.4-1.5.el9.noarch @ns-baseos
Upgraded rocky-gpg-keys-9.3-1.3.el9.noarch @@System
Upgrade grub2-tools-minimal-1:2.06-77.el9.x86_64 @ns-baseos
Upgraded grub2-tools-minimal-1:2.06-70.el9_3.2.rocky.0.5.x86_64 @@System
Upgrade grub2-tools-extra-1:2.06-77.el9.x86_64 @ns-baseos
Upgraded grub2-tools-extra-1:2.06-70.el9_3.2.rocky.0.5.x86_64 @@System
Upgrade grub2-tools-efi-1:2.06-77.el9.x86_64 @ns-baseos
Upgraded grub2-tools-efi-1:2.06-70.el9_3.2.rocky.0.5.x86_64 @@System
Upgrade grub2-tools-1:2.06-77.el9.x86_64 @ns-baseos
Upgraded grub2-tools-1:2.06-70.el9_3.2.rocky.0.5.x86_64 @@System
Upgrade grub2-efi-x64-1:2.06-77.el9.x86_64 @ns-baseos
Upgraded grub2-efi-x64-1:2.06-70.el9_3.2.rocky.0.5.x86_64 @@System
Upgrade grub2-common-1:2.06-77.el9.noarch @ns-baseos
Upgraded grub2-common-1:2.06-70.el9_3.2.rocky.0.5.noarch @@System
Upgrade jq-1.6-16.el9.x86_64 @ns-baseos
Upgraded jq-1.6-15.el9.x86_64 @@System
Upgrade fwupd-1.9.13-2.el9.x86_64 @ns-baseos
Upgraded fwupd-1.8.16-1.el9.rocky.0.3.x86_64 @@System
Upgrade oniguruma-6.9.6-1.el9.5.0.1.x86_64 @ns-baseos
Upgraded oniguruma-6.9.6-1.el9.5.x86_64 @@System
Upgrade PackageKit-glib-1.2.6-1.el9.x86_64 @ns-appstream
Upgraded PackageKit-glib-1.2.4-2.el9.x86_64 @@System
Upgrade PackageKit-1.2.6-1.el9.x86_64 @ns-appstream
Upgraded PackageKit-1.2.4-2.el9.x86_64 @@System
Upgrade container-selinux-3:2.229.0-1.el9.noarch @ns-appstream
Upgraded container-selinux-3:2.221.0-1.el9.noarch @@System
Upgrade checkpolicy-3.6-1.el9.x86_64 @ns-appstream
Upgraded checkpolicy-3.5-1.el9.x86_64 @@System
Upgrade fuse-overlayfs-1.13-1.el9.x86_64 @ns-appstream
Upgraded fuse-overlayfs-1.12-1.el9.x86_64 @@System
Upgrade conmon-2:2.1.10-1.el9.x86_64 @ns-appstream
Upgraded conmon-2:2.1.8-1.el9.x86_64 @@System
Upgrade python3-audit-3.1.2-2.el9.x86_64 @ns-appstream
Upgraded python3-audit-3.0.7-104.el9.x86_64 @@System
Upgrade libblockdev-utils-2.28-10.el9.x86_64 @ns-appstream
Upgraded libblockdev-utils-2.28-7.el9.x86_64 @@System
Upgrade libblockdev-swap-2.28-10.el9.x86_64 @ns-appstream
Upgraded libblockdev-swap-2.28-7.el9.x86_64 @@System
Upgrade libblockdev-part-2.28-10.el9.x86_64 @ns-appstream
Upgraded libblockdev-part-2.28-7.el9.x86_64 @@System
Upgrade libblockdev-mdraid-2.28-10.el9.x86_64 @ns-appstream
Upgraded libblockdev-mdraid-2.28-7.el9.x86_64 @@System
Upgrade libblockdev-lvm-2.28-10.el9.x86_64 @ns-appstream
Upgraded libblockdev-lvm-2.28-7.el9.x86_64 @@System
Upgrade libblockdev-loop-2.28-10.el9.x86_64 @ns-appstream
Upgraded libblockdev-loop-2.28-7.el9.x86_64 @@System
Upgrade libblockdev-fs-2.28-10.el9.x86_64 @ns-appstream
Upgraded libblockdev-fs-2.28-7.el9.x86_64 @@System
Upgrade libblockdev-crypto-2.28-10.el9.x86_64 @ns-appstream
Upgraded libblockdev-crypto-2.28-7.el9.x86_64 @@System
Upgrade libblockdev-2.28-10.el9.x86_64 @ns-appstream
Upgraded libblockdev-2.28-7.el9.x86_64 @@System
Upgrade libX11-common-1.7.0-9.el9.noarch @ns-appstream
Upgraded libX11-common-1.7.0-8.el9.noarch @@System
Upgrade python3-policycoreutils-3.6-2.1.el9.noarch @ns-appstream
Upgraded python3-policycoreutils-3.5-3.el9_3.noarch @@System
Upgrade policycoreutils-python-utils-3.6-2.1.el9.noarch @ns-appstream
Upgraded policycoreutils-python-utils-3.5-3.el9_3.noarch @@System
Upgrade rpm-plugin-systemd-inhibit-4.16.1.3-29.el9.x86_64 @ns-appstream
Upgraded rpm-plugin-systemd-inhibit-4.16.1.3-27.el9_3.x86_64 @@System
Upgrade slirp4netns-1.2.3-1.el9.x86_64 @ns-appstream
Upgraded slirp4netns-1.2.1-1.el9.x86_64 @@System
Upgrade wireguard-tools-1.0.20210914-3.el9.x86_64 @ns-appstream
Upgraded wireguard-tools-1.0.20210914-2.el9.x86_64 @@System
Upgrade tcpdump-14:4.99.0-9.el9.x86_64 @ns-appstream
Upgraded tcpdump-14:4.99.0-7.el9_3.1.x86_64 @@System
Upgrade python-unversioned-command-3.9.18-3.el9.noarch @ns-appstream
Upgraded python-unversioned-command-3.9.18-1.el9_3.1.noarch @@System
Upgrade criu-libs-3.19-1.el9.x86_64 @ns-appstream
Upgraded criu-libs-3.18-1.el9.x86_64 @@System
Upgrade criu-3.19-1.el9.x86_64 @ns-appstream
Upgraded criu-3.18-1.el9.x86_64 @@System
Upgrade crun-1.14.3-1.el9.x86_64 @ns-appstream
Upgraded crun-1.8.7-1.el9.x86_64 @@System
Upgrade bind-utils-32:9.16.23-18.el9_4.1.x86_64 @ns-appstream
Upgraded bind-utils-32:9.16.23-14.el9_3.4.x86_64 @@System
Upgrade bind-libs-32:9.16.23-18.el9_4.1.x86_64 @ns-appstream
Upgraded bind-libs-32:9.16.23-14.el9_3.4.x86_64 @@System
Upgrade bind-license-32:9.16.23-18.el9_4.1.noarch @ns-appstream
Upgraded bind-license-32:9.16.23-14.el9_3.4.noarch @@System
Upgrade libdrm-2.4.117-1.el9.x86_64 @ns-appstream
Upgraded libdrm-2.4.115-1.el9.x86_64 @@System
Upgrade cockpit-storaged-311.1-1.el9.noarch @ns-appstream
Upgraded cockpit-storaged-300.4-1.el9_3.noarch @@System
Upgrade cockpit-packagekit-311.1-1.el9.noarch @ns-appstream
Upgraded cockpit-packagekit-300.4-1.el9_3.noarch @@System
Upgrade python3-libsemanage-3.6-1.el9.x86_64 @ns-appstream
Upgraded python3-libsemanage-3.5-2.el9.x86_64 @@System
Upgrade python3-libselinux-3.6-1.el9.x86_64 @ns-appstream
Upgraded python3-libselinux-3.5-1.el9.x86_64 @@System
Upgrade librsvg2-tools-2.50.7-3.el9.x86_64 @ns-appstream
Upgraded librsvg2-tools-2.50.7-2.el9.x86_64 @@System
Upgrade librsvg2-2.50.7-3.el9.x86_64 @ns-appstream
Upgraded librsvg2-2.50.7-2.el9.x86_64 @@System
Upgrade tracer-common-1.1-2.el9.noarch @ns-appstream
Upgraded tracer-common-0.7.5-4.el9.noarch @@System
Upgrade python3-tracer-1.1-2.el9.noarch @ns-appstream
Upgraded python3-tracer-0.7.5-4.el9.noarch @@System
Upgrade libfprint-1.94.6-1.el9.x86_64 @ns-appstream
Upgraded libfprint-1.94.5-1.el9.x86_64 @@System
Upgrade rsyslog-relp-8.2310.0-4.el9.x86_64 @ns-appstream
Upgraded rsyslog-relp-8.2102.0-117.el9.x86_64 @@System
Upgrade rsyslog-logrotate-8.2310.0-4.el9.x86_64 @ns-appstream
Upgraded rsyslog-logrotate-8.2102.0-117.el9.x86_64 @@System
Upgrade rsyslog-gssapi-8.2310.0-4.el9.x86_64 @ns-appstream
Upgraded rsyslog-gssapi-8.2102.0-117.el9.x86_64 @@System
Upgrade rsyslog-gnutls-8.2310.0-4.el9.x86_64 @ns-appstream
Upgraded rsyslog-gnutls-8.2102.0-117.el9.x86_64 @@System
Upgrade rsyslog-8.2310.0-4.el9.x86_64 @ns-appstream
Upgraded rsyslog-8.2102.0-117.el9.x86_64 @@System
Upgrade nss-util-3.90.0-7.el9_4.x86_64 @ns-appstream
Upgraded nss-util-3.90.0-7.el9_3.x86_64 @@System
Upgrade nss-sysinit-3.90.0-7.el9_4.x86_64 @ns-appstream
Upgraded nss-sysinit-3.90.0-7.el9_3.x86_64 @@System
Upgrade nss-softokn-freebl-3.90.0-7.el9_4.x86_64 @ns-appstream
Upgraded nss-softokn-freebl-3.90.0-7.el9_3.x86_64 @@System
Upgrade nss-softokn-3.90.0-7.el9_4.x86_64 @ns-appstream
Upgraded nss-softokn-3.90.0-7.el9_3.x86_64 @@System
Upgrade nss-3.90.0-7.el9_4.x86_64 @ns-appstream
Upgraded nss-3.90.0-7.el9_3.x86_64 @@System
Upgrade nspr-4.35.0-7.el9_4.x86_64 @ns-appstream
Upgraded nspr-4.35.0-7.el9_3.x86_64 @@System
Upgrade open-vm-tools-12.3.5-2.el9.x86_64 @ns-appstream
Upgraded open-vm-tools-12.2.5-3.el9_3.2.x86_64 @@System
Upgrade netavark-2:1.10.3-1.el9.x86_64 @ns-appstream
Upgraded netavark-2:1.7.0-2.el9_3.x86_64 @@System
Upgrade containers-common-2:1-91.el9_4.x86_64 @ns-appstream
Upgraded containers-common-2:1-55.el9.x86_64 @@System
Upgrade aardvark-dns-2:1.10.0-3.el9_4.x86_64 @ns-appstream
Upgraded aardvark-dns-2:1.7.0-1.el9.x86_64 @@System
Upgrade webkit2gtk3-jsc-2.42.5-1.el9.x86_64 @ns-appstream
Upgraded webkit2gtk3-jsc-2.40.5-1.el9_3.1.x86_64 @@System
Upgrade libX11-1.7.0-9.el9.x86_64 @ns-appstream
Upgraded libX11-1.7.0-8.el9.x86_64 @@System
Upgrade fuse3-libs-3.10.2-8.el9.x86_64 @ns-appstream
Upgraded fuse3-libs-3.10.2-6.el9.x86_64 @@System
Upgrade fuse3-3.10.2-8.el9.x86_64 @ns-appstream
Upgraded fuse3-3.10.2-6.el9.x86_64 @@System
Upgrade python3.11-libs-3.11.7-1.el9.x86_64 @ns-appstream
Upgraded python3.11-libs-3.11.5-1.el9_3.x86_64 @@System
Upgrade python3.11-3.11.7-1.el9.x86_64 @ns-appstream
Upgraded python3.11-3.11.5-1.el9_3.x86_64 @@System
Upgrade python3.11-pip-wheel-22.3.1-5.el9.noarch @ns-appstream
Upgraded python3.11-pip-wheel-22.3.1-4.el9_3.1.noarch @@System
Upgrade fwupd-plugin-flashrom-1.9.13-2.el9.x86_64 @ns-appstream
Upgraded fwupd-plugin-flashrom-1.8.16-1.el9.rocky.0.3.x86_64 @@System
Scriptlet output:
1 warning: /etc/shadow created as /etc/shadow.rpmnew
2 Failed to set unit properties on pesign.service: Unit pesign.service not found.
Transaction ID : 2
Begin time : Fri Mar 8 09:27:45 2024
Begin rpmdb : 8ae32bb73aeb09256d54051c44b07068dd426917cd72c7f7e9c74a834e265d30
End time : Fri Mar 8 09:27:46 2024 (1 seconds)
End rpmdb : f9893f5dab3e019899298ad401b841859b922152594ed28bae0c1899cce55441 **
User : root
Return-Code : Success
Releasever :
Command Line :
Persistence : Unknown
Comment :
Packages Altered:
Install dnf-automatic-4.14.0-8.el9.noarch @baseos
I hope that helps. I have another appointment now and won’t be able to reply until tomorrow if you need anything else.
Best, Erwin
1 Like
Hi Erwin,
Thank you, that helps a lot.
From the information you provided, it appears we are looking at an exceptional combination of events:
- A breaking change was introduced upstream in systemd as part of the Rocky Linux 9.8 update.
dnf-automaticinstalled operating system updates independently of NS8’s update workflow.- The node rebooted before Core 3.19.1 had been installed.
Under those circumstances, the node could reach a state where Rocky Linux was already updated while the compatibility fix contained in Core 3.19.1 was still missing, triggering the issue you experienced.
The important point is that the conflict is not related to the time of day when the updates run. Setting dnf-automatic to 06:30 and apply-updates.timer to run overnight does not avoid the problem. The issue is that the two update agents see different DNF repository contents and make independent update decisions.
With an active subscription, only apply-updates.timer is designed to coordinate NS8 Core, applications, and operating system updates so that they are applied in the intended order. In contrast, dnf-automatic only sees the operating system updates and can install them before the corresponding NS8 Core update becomes available or is scheduled for installation.
Until we release a Core update that detects and blocks this situation, I recommend disabling dnf-automatic and any Cockpit-managed automatic DNF updates. With an active subscription, operating system updates are already handled by apply-updates.timer, so keeping both mechanisms enabled can lead to unexpected results.
Thank you for helping us investigate this. Your findings, together with the other report, are helping us better understand the conditions that triggered the problem.
Best,
Davide
1 Like