Netdata Ns8 not start

Hi everyone, I don’t know if after the last update of the ns8 core, the net app seems active, but the link no longer works reporting “bad gateway”.

1 Like

systemctl status netdata1

You could try to check log

journalctl -e -u netdata1

And restart if needed

systemctl restart netdata1

Check log after again and try to catch something

Hi @stephdl , the service was active , but it started working again with a restart .

Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.139+02:00 comm=netdata source=collector level=error tid=399 thread=P[cgroups]>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.150+02:00 comm=cgroup-name.sh source=collector level=info tid=2867 thread=cgr>
Jun 07 22:23:36 ns8 netdata1[79110]: curl: (3) URL using bad/illegal format or missing URL
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.192+02:00 comm=cgroup-name.sh source=collector level=warning tid=2872 thread=>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.196+02:00 comm=cgroup-name.sh source=collector level=info tid=2873 thread=cgr>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.196+02:00 comm=netdata source=daemon level=error tid=399 thread=P[cgroups] ms>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.199+02:00 comm=cgroup-network source=collector level=info errno="2, No such f>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.208+02:00 comm=cgroup-network-helper.sh source=collector level=info tid=2880 >
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.214+02:00 comm=cgroup-network source=collector level=error tid=2874 thread=cg>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.214+02:00 comm=cgroup-network source=collector level=error errno="22, Invalid>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.215+02:00 comm=netdata source=daemon level=error tid=399 thread=P[cgroups] ms>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.225+02:00 comm=cgroup-name.sh source=collector level=info tid=2892 thread=cgr>
Jun 07 22:23:36 ns8 netdata1[79110]: curl: (3) URL using bad/illegal format or missing URL
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.268+02:00 comm=cgroup-name.sh source=collector level=warning tid=2897 thread=>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.271+02:00 comm=cgroup-name.sh source=collector level=info tid=2898 thread=cgr>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.272+02:00 comm=netdata source=daemon level=error tid=399 thread=P[cgroups] ms>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.272+02:00 comm=netdata source=collector level=error tid=399 thread=P[cgroups]>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.282+02:00 comm=cgroup-name.sh source=collector level=info tid=2905 thread=cgr>
Jun 07 22:23:36 ns8 netdata1[79110]: curl: (3) URL using bad/illegal format or missing URL
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.324+02:00 comm=cgroup-name.sh source=collector level=warning tid=2910 thread=>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.327+02:00 comm=cgroup-name.sh source=collector level=info tid=2911 thread=cgr>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.328+02:00 comm=netdata source=daemon level=error tid=399 thread=P[cgroups] ms>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.330+02:00 comm=cgroup-network source=collector level=info errno="2, No such f>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.339+02:00 comm=cgroup-network-helper.sh source=collector level=info tid=2918 >
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.345+02:00 comm=cgroup-network source=collector level=error tid=2912 thread=cg>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.345+02:00 comm=cgroup-network source=collector level=error tid=2912 thread=cg>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.345+02:00 comm=netdata source=daemon level=error tid=399 thread=P[cgroups] ms>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.356+02:00 comm=cgroup-name.sh source=collector level=info tid=2930 thread=cgr>
Jun 07 22:23:36 ns8 netdata1[79110]: curl: (3) URL using bad/illegal format or missing URL
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.398+02:00 comm=cgroup-name.sh source=collector level=warning tid=2935 thread=>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.402+02:00 comm=cgroup-name.sh source=collector level=info tid=2936 thread=cgr>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.402+02:00 comm=netdata source=daemon level=error tid=399 thread=P[cgroups] ms>
Jun 07 22:23:36 ns8 netdata1[79110]: time=2024-06-07T22:23:36.402+02:00 comm=netdata source=collector level=error tid=399 thread=P[cgroups]>
Jun 07 22:23:38 ns8 netdata1[79110]: time=2024-06-07T22:23:38.626+02:00 comm=netdata source=health level=info tid=344 thread=HEALTH msg_id=>
Jun 07 22:23:38 ns8 netdata1[79110]: time=2024-06-07T22:23:38.626+02:00 comm=netdata source=health level=info tid=344 thread=HEALTH msg_id=>
Jun 07 22:23:38 ns8 netdata1[79110]: time=2024-06-07T22:23:38.627+02:00 comm=netdata source=health level=info tid=344 thread=HEALTH msg_id=>
Jun 07 22:23:38 ns8 netdata1[79110]: time=2024-06-07T22:23:38.627+02:00 comm=netdata source=health level=info tid=344 thread=HEALTH msg_id=>
Jun 07 22:25:10 ns8 netdata1[79110]: time=2024-06-07T22:25:10.500+02:00 comm=netdata source=daemon level=info tid=1 thread=netdata msg="SIG>
Jun 07 22:25:13 ns8 netdata1[79110]: time=2024-06-07T22:25:13.680+02:00 comm=netdata source=health level=info tid=344 thread=HEALTH msg_id=>
Jun 07 22:25:13 ns8 netdata1[79110]: time=2024-06-07T22:25:13.681+02:00 comm=netdata source=health level=info tid=344 thread=HEALTH msg_id=>
Jun 07 22:25:13 ns8 netdata1[79110]: time=2024-06-07T22:25:13.681+02:00 comm=netdata source=health level=info tid=344 thread=HEALTH msg_id=>
lines 960-1000/1000 (END)

There is something missing and maybe interesting

You could try

journalctl -u netdata1 > dump

It will make a dump of the log…easier for you to search

this is a bug, thank @france

we try to start net-data before redis

4 Likes

hello @france

could please test the fix

mainly it is to install netdata, check there is no regressions after configuration and restart the server

after each restart, netdata must be up and reachable

thank in advance

Hi @stephdl , ok I’ll try as soon as possible by today .

I have to perform what you wrote :sweat_smile: ?

  • install netdata and configure it
  • add-module ghcr.io/nethserver/netdata:1.1.1-dev.1
  • restart the server several times
  • netdata must be up and full configured after it
1 Like

exactly, now we start after redis, you could also demonstrate/prove it by

systemd-analyze plot > plot

then do to extract it

scp plot user@IP:/home/user

it is an html file

Ok, since there are 2 modules so as an instance is that okay?

hum yes I do not think to get twice this module is a breaker

just test the plot like I asked, you need to see netdata2 starting after redis

I did as you wrote to me. Anyway I have a snapshot of the server

I have the file him, how do I send it to you?

do like me, present it, opened with firefox

initrd
init.scope
-.slice
system.slice
system-systemd\x2dhibernate\x2dresume.slice
initrd-usr-fs.target
systemd-journald-dev-log.socket
systemd-journald.socket
systemd-vconsole-setup.service (196ms)
dracut-cmdline.service (87ms)
dracut-pre-udev.service (46ms)
sys-module-fuse.device (3.572s)
sys-module-configfs.device (3.479s)
dracut-initqueue.service (883ms)
sys-devices-platform-serial8250-tty-ttyS3.device (3.486s)
dev-ttyS3.device (3.486s)
sys-devices-platform-serial8250-tty-ttyS1.device (3.486s)
dev-ttyS1.device (3.486s)
sys-devices-platform-serial8250-tty-ttyS0.device (3.485s)
dev-ttyS0.device (3.485s)
sys-devices-platform-serial8250-tty-ttyS2.device (3.485s)
dev-ttyS2.device (3.485s)
dev-virtio\x2dports-org.qemu.guest_agent.0.device (3.488s)
sys-devices-pci0000:00-0000:00:08.0-virtio1-virtio\x2dports-vport1p1.device (3.488s)
dev-vport1p1.device (3.488s)
sys-devices-pci0000:00-0000:00:12.0-virtio2-net-ens18.device (3.372s)
sys-subsystem-net-devices-ens18.device (3.372s)
sys-devices-pci0000:00-0000:00:05.0-0000:01:01.0-virtio3-host0-target0:0:0-0:0:0:0-block-sda.device (3.207s)
dev-disk-by\x2did-scsi\x2d0QEMU_QEMU_HARDDISK_drive\x2dscsi0.device (3.207s)
dev-disk-by\x2dpath-pci\x2d0000:01:01.0\x2dscsi\x2d0:0:0:0.device (3.207s)
dev-disk-by\x2ddiskseq-1.device (3.207s)
dev-sda.device (3.207s)
dev-disk-by\x2dpath-pci\x2d0000:01:01.0\x2dscsi\x2d0:0:0:0\x2dpart1.device (3.204s)
dev-disk-by\x2did-scsi\x2d0QEMU_QEMU_HARDDISK_drive\x2dscsi0\x2dpart1.device (3.204s)
dev-disk-by\x2duuid-3b710d50\x2db7cc\x2d489c\x2dbcc1\x2d63c6cfd22c47.device (3.204s)
dev-disk-by\x2dpartuuid-597f82f4\x2d01.device (3.204s)
sys-devices-pci0000:00-0000:00:05.0-0000:01:01.0-virtio3-host0-target0:0:0-0:0:0:0-block-sda-sda1.device (3.204s)
dev-sda1.device (3.204s)
dev-sda2.device (3.236s)
dev-disk-by\x2dpartuuid-597f82f4\x2d02.device (3.236s)
dev-disk-by\x2did-scsi\x2d0QEMU_QEMU_HARDDISK_drive\x2dscsi0\x2dpart2.device (3.236s)
sys-devices-pci0000:00-0000:00:05.0-0000:01:01.0-virtio3-host0-target0:0:0-0:0:0:0-block-sda-sda2.device (3.236s)
dev-disk-by\x2dpath-pci\x2d0000:01:01.0\x2dscsi\x2d0:0:0:0\x2dpart2.device (3.236s)
dev-rl-root.device
dev-dm\x2d0.device
dev-mapper-rl\x2droot.device
sys-devices-virtual-block-dm\x2d0.device
dev-disk-by\x2did-dm\x2duuid\x2dLVM\x2dSmgXdiGvpgjdy1CLJB3A8V4EFIIC5FncQh1zX7PwPYtw2VSICzfxV8bdYtL6UFD8.device
dev-disk-by\x2duuid-dab676bd\x2d1780\x2d4e92\x2d9d29\x2d8e4995a25852.device
dev-disk-by\x2did-dm\x2dname\x2drl\x2droot.device
initrd-root-device.target
dev-disk-by\x2did-dm\x2dname\x2drl\x2dswap.device
dev-disk-by\x2duuid-9cd1f57b\x2deeb1\x2d4a63\x2dbc76\x2d5628c1f75852.device
sys-devices-virtual-block-dm\x2d1.device
dev-disk-by\x2did-dm\x2duuid\x2dLVM\x2dSmgXdiGvpgjdy1CLJB3A8V4EFIIC5FncJqQb3rArxLeGr7hPeEOdffuCscXWcKQ8.device
dev-dm\x2d1.device
dev-mapper-rl\x2dswap.device
dev-rl-swap.device
remote-fs-pre.target
dracut-pre-mount.service (7ms)
systemd-fsck-root.service (49ms)
initrd-root-fs.target
initrd-parse-etc.service (15ms)
initrd-fs.target
initrd.target
dracut-pre-pivot.service (39ms)
initrd-cleanup.service (31ms)
initrd-udevadm-cleanup-db.service (5ms)
initrd-switch-root.target
initrd-switch-root.service (973ms)
systemd
system-agent.slice
system-getty.slice
system-modprobe.slice
system-sshd\x2dkeygen.slice
system-wg\x2dquick.slice
user.slice
systemd-ask-password-console.path
systemd-ask-password-wall.path
proc-sys-fs-binfmt_misc.automount
cryptsetup.target
integritysetup.target
nss-lookup.target
paths.target
remote-fs.target
slices.target
veritysetup.target
dm-event.socket
lvm2-lvmpolld.socket
systemd-coredump.socket
systemd-initctl.socket
systemd-udevd-control.socket
systemd-udevd-kernel.socket
dev-mapper-rl\x2dswap.swap (43ms)
dev-hugepages.mount (43ms)
dev-mqueue.mount (41ms)
sys-kernel-debug.mount (41ms)
sys-kernel-tracing.mount (40ms)
kmod-static-nodes.service (35ms)
lvm2-monitor.service (192ms)
modprobe@drm.service (100ms)
nis-domainname.service (26ms)
systemd-journald.service (48ms)
systemd-modules-load.service (19ms)
systemd-network-generator.service (20ms)
systemd-remount-fs.service (44ms)
systemd-udev-trigger.service (121ms)
dev-disk-by\x2did-dm\x2duuid\x2dLVM\x2dSmgXdiGvpgjdy1CLJB3A8V4EFIIC5FncJqQb3rArxLeGr7hPeEOdffuCscXWcKQ8.swap
dev-disk-by\x2duuid-9cd1f57b\x2deeb1\x2d4a63\x2dbc76\x2d5628c1f75852.swap
dev-disk-by\x2did-dm\x2dname\x2drl\x2dswap.swap
dev-rl-swap.swap
dev-dm\x2d1.swap
swap.target
sys-fs-fuse-connections.mount (9ms)
sys-kernel-config.mount (8ms)
systemd-sysctl.service (32ms)
run-credentials-systemd\x2dsysctl.service.mount
systemd-hwdb-update.service (731ms)
systemd-journal-flush.service (62ms)
systemd-random-seed.service (61ms)
systemd-sysusers.service (71ms)
run-credentials-systemd\x2dsysusers.service.mount
systemd-tmpfiles-setup-dev.service (49ms)
run-credentials-systemd\x2dtmpfiles\x2dsetup\x2ddev.service.mount
local-fs-pre.target
systemd-udevd.service (77ms)
modprobe@configfs.service (13ms)
modprobe@fuse.service (3ms)
boot.mount (117ms)
dev-sr0.device
dev-disk-by\x2ddiskseq-3.device
dev-disk-by\x2dpath-pci\x2d0000:00:01.1\x2data\x2d2.0.device
dev-disk-by\x2dpath-pci\x2d0000:00:01.1\x2data\x2d2.device
dev-cdrom.device
dev-disk-by\x2did-ata\x2dQEMU_DVD\x2dROM_QM00003.device
sys-devices-pci0000:00-0000:00:01.1-ata2-host2-target2:0:0-2:0:0:0-block-sr0.device
dev-disk-by\x2did-lvm\x2dpv\x2duuid\x2dwizQLF\x2dKmWS\x2dbXkF\x2dM7yP\x2dIoUI\x2dadcB\x2dYMC1iX.device
dev-disk-by\x2duuid-26f38ac4\x2d68ad\x2d440e\x2da91b\x2d3e257fc8fbdf.device
dev-disk-by\x2did-dm\x2duuid\x2dLVM\x2dSmgXdiGvpgjdy1CLJB3A8V4EFIIC5FncTI4798rc1EPJUobY49kVrs8I5K9V78zs.device
dev-mapper-rl\x2dhome.device
sys-devices-virtual-block-dm\x2d2.device
dev-dm\x2d2.device
dev-rl-home.device
dev-disk-by\x2did-dm\x2dname\x2drl\x2dhome.device
home.mount (132ms)
local-fs.target
ldconfig.service (107ms)
systemd-boot-update.service (14ms)
systemd-tmpfiles-setup.service (71ms)
run-credentials-systemd\x2dtmpfiles\x2dsetup.service.mount
auditd.service (127ms)
systemd-journal-catalog-update.service (39ms)
systemd-update-done.service (9ms)
systemd-update-utmp.service (10ms)
sysinit.target
backup1-dnsmasq1.timer
dnf-makecache.timer
logrotate.timer
mlocate-updatedb.timer
phonehome.timer
systemd-tmpfiles-clean.timer
timers.target
dbus.socket
sssd-kcm.socket
sockets.target
dbus-broker.service (44ms)
basic.target
agent@cluster.service
agent@netdata1.service
agent@netdata2.service
agent@node.service
api-server.service (210ms)
chronyd.service (210ms)
dracut-shutdown.service (72ms)
firewalld.service (1.153s)
irqbalance.service
overlay-cleanup.service (1.042s)
qemu-guest-agent.service
rclone-webdav.service (1.344s)
sshd-keygen.target
nss-user-lookup.target
systemd-logind.service (239ms)
user-1002.slice
user-runtime-dir@1002.service (402ms)
user-1003.slice
user-1008.slice
user-1009.slice
user-1012.slice
user-1015.slice
user-1016.slice
user-runtime-dir@1003.service (355ms)
user-runtime-dir@1008.service (353ms)
user-runtime-dir@1009.service (351ms)
user-runtime-dir@1012.service (348ms)
user-runtime-dir@1015.service (346ms)
user-runtime-dir@1016.service (344ms)
user-1014.slice
user-runtime-dir@1014.service (334ms)
user-1001.slice
user-runtime-dir@1001.service (316ms)
user-1006.slice
user-runtime-dir@1006.service (307ms)
user-1000.slice
user-1004.slice
user-1007.slice
user-1013.slice
user-runtime-dir@1000.service (269ms)
user-runtime-dir@1004.service (268ms)
user-runtime-dir@1007.service (266ms)
user-runtime-dir@1013.service (264ms)
run-user-1001.mount
run-user-1008.mount
run-user-1016.mount
run-user-1009.mount
run-user-1014.mount
run-user-1002.mount
run-user-1015.mount
run-user-1003.mount
run-user-1012.mount
run-user-1013.mount
run-user-1007.mount
run-user-1004.mount
run-user-1000.mount
run-user-1006.mount
network-pre.target
NetworkManager.service (299ms)
machine.slice
libpod-67426583ebbe304bd4d4e0b88e1c00b6ecee55007a608b7cc82cb6c5174a8c4b.scope
dev-rfkill.device
sys-devices-virtual-misc-rfkill.device
systemd-rfkill.socket
network.target
NetworkManager-wait-online.service (670ms)
netdata1.service (3.176s)
sshd.service (71ms)
systemd-user-sessions.service (13ms)
crond.service
getty@tty1.service
getty.target
var-lib-containers-storage-overlay-2ec6c150fef1af178359a72b64d477d06660fd7b6dfee154be15e7361733538d-merged.mount
var-lib-containers-storage-overlay\x2dcontainers-67426583ebbe304bd4d4e0b88e1c00b6ecee55007a608b7cc82cb6c5174a8c4b-userdata-shm.mount
var-lib-containers-storage-overlay.mount
network-online.target
api-server-motd.service (18ms)
kdump.service (2.224s)
rsyslog.service (84ms)
wg-quick@wg0.service (108ms)
sys-devices-virtual-net-wg0.device
sys-subsystem-net-devices-wg0.device
redis.service (1.394s)
var-lib-containers-storage-overlay\x2dcontainers-c3319208b2357a8dfa8f1c8e10756d23de61f8d8eb7cc5fd9a407f86123b50d6-userdata-shm.mount
var-lib-containers-storage-overlay-0fb60f5e7612aa7410b996a5d46bbb579204aaa4611bed2b54b990a389efe56e-merged.mount
libpod-c3319208b2357a8dfa8f1c8e10756d23de61f8d8eb7cc5fd9a407f86123b50d6.scope
var-lib-containers-storage-overlay-d4688aac36585ec7131bc052b4f825a0686c46c32a46db141282984476473f81-merged.mount
var-lib-containers-storage-overlay\x2dcontainers-fb17495d1ca64d49652370e9b2d9fc026afce117998675de4a0c9752bd325cab-userdata-shm.mount
run-netns.mount
run-netns-netns\x2d0e546f30\x2d9c9d\x2dfe95\x2deb0e\x2dd8039f0afd7b.mount
netdata2.service (1.098s)
promtail.service (1.337s)
user@1000.service (1.184s)
user@1001.service (874ms)
user@1002.service (937ms)
user@1003.service (957ms)
user@1004.service (862ms)
user@1006.service (1.034s)
user@1007.service (854ms)
user@1008.service (964ms)
user@1009.service (1.012s)
user@1012.service (947ms)
user@1013.service (1.230s)
user@1014.service (756ms)
user@1015.service (959ms)
user@1016.service (794ms)
sys-devices-virtual-net-podman0.device
sys-subsystem-net-devices-podman0.device
sys-devices-virtual-net-veth0.device
sys-subsystem-net-devices-veth0.device
libpod-fb17495d1ca64d49652370e9b2d9fc026afce117998675de4a0c9752bd325cab.scope
var-lib-containers-storage-overlay\x2dcontainers-0308bf4c815eb163b9d20f0a788d4ef7a274dc3b46494066d0cb6ec7ce8bcb75-userdata-shm.mount
var-lib-containers-storage-overlay-09bb4ba7d7cc770a6b668c7400604cf38232001aa0f992756fb8fc6d6bea9e8c-merged.mount
libpod-0308bf4c815eb163b9d20f0a788d4ef7a274dc3b46494066d0cb6ec7ce8bcb75.scope
libpod-5714d3a3ec8aefda0745cca69c6f61674011b1cc4580bc0ac74b8015f93fb375.scope
multi-user.target

If so, it’s not good in the link I gave you, I copied it right there

netdata2 started after redis

Great, so it means the change you made worked! Do I have to do more tests for you now?

1 Like

no I release, except if netdata2 is not reachable from the web

If you mean the display , the link works on netdata2 opens the net data page.

1 Like