Netdata Failed after update

rmk · July 31, 2023, 2:43pm

Has anyone experiencing Netdata failed to work after recent update of netdata.x86_64 0:1.41.0-1.el7, netdata-conf.noarch 0:1.41.0-1.el7 , netdata-data.noarch 0:1.41.0-1.el7…

I tried flashing the /var/cache/netdata/dbengine … and then try to restart netdata, still not able to get it to start or work…
Any suggestion ?

giacomo · July 31, 2023, 2:52pm

Can you check if it’s the same error of Netdata Service failed since Update?

rmk · July 31, 2023, 2:54pm

I got the following error when I ran " systemctl status netdata "…after trying to restart.

 Active: activating (auto-restart) (Result: exit-code) since Mon 2023-07-31 07:37:17 PDT; 2s ago
  Process: 3013 ExecStart=/usr/sbin/netdata -P /run/netdata/netdata.pid -D (code=exited, status=233/RUNTIME_DIRECTORY)
  Process: 3011 ExecStartPre=/usr/bin/find /var/cache/netdata/ -type f -empty -delete (code=exited, status=233/RUNTIME_DIRECTORY)
  Process: 3008 ExecStartPre=/bin/chown -R netdata /run/netdata (code=exited, status=0/SUCCESS)
  Process: 3005 ExecStartPre=/bin/mkdir -p /run/netdata (code=exited, status=0/SUCCESS)
  Process: 3002 ExecStartPre=/bin/chown -R netdata /var/cache/netdata (code=exited, status=0/SUCCESS)
  Process: 3000 ExecStartPre=/bin/mkdir -p /var/cache/netdata (code=exited, status=0/SUCCESS)
 Main PID: 3013 (code=exited, status=233/RUNTIME_DIRECTORY)

Here are the errors from first 2 lines 
(code=exited, status=233/RUNTIME_DIRECTORY)
(code=exited, status=233/RUNTIME_DIRECTORY)

 Looks like the same issue...

giacomo · July 31, 2023, 3:28pm

Have you updated also nethserver-netdata?

Try:

yum clean all && yum update

Niwre · July 31, 2023, 3:37pm

Hi,
the problem is still present after that. What does nethserver-netdata updated mean? I installed the existing update via the cockpit this morning.

rmk · July 31, 2023, 3:39pm

Tried ```
yum clean all && yum update

      Updating:
 netdata              x86_64         1.41.0-1.el7            epel         3.5 M
 netdata-conf         noarch       1.41.0-1.el7            epel         155 k
 netdata-data         noarch       1.41.0-1.el7            epel          10 M

Not seeing the   Nethserver-netdata package

EddieA · July 31, 2023, 7:18pm

My Logwatch report showed this:

The following updates will be applied on Nethserver.BogoLinux.net:
================================================================================
 Package              Arch           Version                 Repository    Size
================================================================================
Updating:
 netdata              x86_64         1.41.0-1.el7            epel         3.5 M
 netdata-conf         noarch         1.41.0-1.el7            epel         155 k
 netdata-data         noarch         1.41.0-1.el7            epel          10 M

Transaction Summary
================================================================================
Upgrade  3 Packages
The updates were successfully applied

Did nothing:

No packages marked for update

Cheers.

PCXLan.es · July 31, 2023, 7:23pm

Same problem here. Log:

 netdata.service - Real time performance monitoring
   Loaded: loaded (/usr/lib/systemd/system/netdata.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/netdata.service.d
           `-cleanup.conf
   Active: activating (auto-restart) (Result: exit-code) since Mon 2023-07-31 21:19:06 CEST; 27s ago
  Process: 35905 ExecStart=/usr/sbin/netdata -P /run/netdata/netdata.pid -D (code=exited, status=233/RUNTIME_DIRECTORY)
  Process: 35901 ExecStartPre=/usr/bin/find /var/cache/netdata/ -type f -empty -delete (code=exited, status=233/RUNTIME_DIRECTORY)
  Process: 35891 ExecStartPre=/bin/chown -R netdata /run/netdata (code=exited, status=0/SUCCESS)
  Process: 35884 ExecStartPre=/bin/mkdir -p /run/netdata (code=exited, status=0/SUCCESS)
  Process: 35870 ExecStartPre=/bin/chown -R netdata /var/cache/netdata (code=exited, status=0/SUCCESS)
  Process: 35861 ExecStartPre=/bin/mkdir -p /var/cache/netdata (code=exited, status=0/SUCCESS)
 Main PID: 35905 (code=exited, status=233/RUNTIME_DIRECTORY)

Jul 31 21:19:06 server.domain.es systemd[1]: netdata.service: main process exited, code=exited, status=233/RUNTIME_DIRECTORY
Jul 31 21:19:06 server.domain.es systemd[1]: Unit netdata.service entered failed state.
Jul 31 21:19:06 server.domain.es systemd[1]: netdata.service failed.
Jul 31 21:19:36 server.domain.es systemd[1]: netdata.service holdoff time over, scheduling restart.
Jul 31 21:19:36 server.domain.es systemd[1]: Stopped Real time performance monitoring.
Jul 31 21:19:36 server.domain.es systemd[1]: Starting Real time performance monitoring...
Jul 31 21:19:36 server.domain.es systemd[38180]: Failed at step RUNTIME_DIRECTORY spawning /usr/bin/find: File exists
Jul 31 21:19:36 server.domain.es systemd[1]: Started Real time performance monitoring.
Jul 31 21:19:36 server.domain.es systemd[38183]: Failed at step RUNTIME_DIRECTORY spawning /usr/sbin/netdata: File exists
Jul 31 21:19:36 server.domain.es systemd[1]: netdata.service: main process exited, code=exited, status=233/RUNTIME_DIRECTORY
Jul 31 21:19:36 server.domain.es systemd[1]: Unit netdata.service entered failed state.
Jul 31 21:19:36 server.domain.es systemd[1]: netdata.service failed.

Have you updated also nethserver-netdata?

Try:
yum clean all && yum update

I always do yum clean all && yum update to update my server, and same packages had appeared:

================================================================================
 Package              Arch           Version                 Repository    Size
================================================================================
Updating:
 netdata              x86_64         1.41.0-1.el7            epel         3.5 M
 netdata-conf         noarch         1.41.0-1.el7            epel         155 k
 netdata-data         noarch         1.41.0-1.el7            epel          10 M

Transaction Summary
================================================================================
Upgrade  3 Packages

dnutan · July 31, 2023, 8:04pm

github.com/netdata/netdata

[Bug]: Netdata fails to start on Centos7/RHEL7 with "Failed at step RUNTIME_DIRECTORY"

opened 09:36AM - 22 Jun 23 UTC

closed 04:48PM - 22 Jun 23 UTC

ilyam8

bug area/packaging

### Bug description [Reported on Discord](https://discord.com/channels/84750228…0503590932/1121069723417849856/1121069723417849856): Netdata fails to start on RHEL7. I tested on Centos7 and confirm the problem. - systemctl status ```c $ systemctl status netdata ● netdata.service - Real time performance monitoring Loaded: loaded (/usr/lib/systemd/system/netdata.service; enabled; vendor preset: enabled) Active: activating (auto-restart) (Result: exit-code) since Thu 2023-06-22 12:00:22 EEST; 5s ago Process: 20673 ExecStart=/usr/sbin/netdata -P /run/netdata/netdata.pid -D (code=exited, status=233/RUNTIME_DIRECTORY) Process: 20670 ExecStartPre=/bin/chown -R netdata /run/netdata (code=exited, status=0/SUCCESS) Process: 20667 ExecStartPre=/bin/mkdir -p /run/netdata (code=exited, status=0/SUCCESS) Process: 20664 ExecStartPre=/bin/chown -R netdata /var/cache/netdata (code=exited, status=0/SUCCESS) Process: 20662 ExecStartPre=/bin/mkdir -p /var/cache/netdata (code=exited, status=0/SUCCESS) Main PID: 20673 (code=exited, status=233/RUNTIME_DIRECTORY) ``` - journal logs ``` Jun 21 19:18:35 shared-centos7 systemd[16829]: Failed at step RUNTIME_DIRECTORY spawning /usr/sbin/netdata: File exists ``` ### Expected behavior Netdata starts on Centos7 ### Steps to reproduce Install Netdata on Centos7 (native package) ### Installation method kickstart.sh ### System info ```shell /etc/centos-release:CentOS Linux release 7.9.2009 (Core) /etc/os-release:NAME="CentOS Linux" /etc/os-release:VERSION="7 (Core)" /etc/os-release:ID="centos" /etc/os-release:ID_LIKE="rhel fedora" /etc/os-release:VERSION_ID="7" /etc/os-release:PRETTY_NAME="CentOS Linux 7 (Core)" /etc/os-release:ANSI_COLOR="0;31" /etc/os-release:CPE_NAME="cpe:/o:centos:centos:7" /etc/os-release: /etc/os-release:CENTOS_MANTISBT_PROJECT="CentOS-7" /etc/os-release:CENTOS_MANTISBT_PROJECT_VERSION="7" /etc/os-release:REDHAT_SUPPORT_PRODUCT="centos" /etc/os-release:REDHAT_SUPPORT_PRODUCT_VERSION="7" /etc/os-release: /etc/redhat-release:CentOS Linux release 7.9.2009 (Core) /etc/system-release:CentOS Linux release 7.9.2009 (Core) ``` ### Netdata build info ```shell Version: netdata v1.40.0-32-nightly Configure options: '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--datadir=/usr/share' '--includedir=/usr/include' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-bundled-protobuf' '--prefix=/usr' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/usr/libexec' '--libdir=/usr/lib' '--with-zlib' '--with-math' '--with-user=netdata' '--disable-dependency-tracking' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro ' 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' 'PKG_CONFIG_PATH=:/usr/lib/pkgconfig:/usr/share/pkgconfig' Install type: binpkg-rpm Binary architecture: x86_64 Packaging distro: Features: dbengine: YES Native HTTPS: YES Netdata Cloud: YES ACLK: YES TLS Host Verification: YES Machine Learning: YES Stream Compression: NO HTTPD (h2o): YES Libraries: protobuf: YES (bundled) jemalloc: NO JSON-C: YES libcap: NO libcrypto: YES libm: YES tcalloc: NO zlib: YES Plugins: apps: YES cgroup Network Tracking: YES CUPS: NO debugfs: YES EBPF: YES IPMI: YES NFACCT: NO perf: YES slabinfo: YES Xen: NO Xen VBD Error Tracking: NO Exporters: AWS Kinesis: NO GCP PubSub: NO MongoDB: NO Prometheus Remote Write: YES Debug/Developer Features: Trace Allocations: NO ``` ### Additional info _No response_

if it’s the same problem described in the netdata bug report, it was supposedly fixed on 1.40.1, unless there’s some regression or a different cause.

giacomo · August 1, 2023, 8:19am

Thank you all for reporting.
Indeed this is an upstream issue, we tried to fix it with a workaround: Netdata 1.41 doesn't start · Issue #6760 · NethServer/dev · GitHub

Please try it and report the results here:

yum --enablerepo=nethserver-testing update nethserver-netdata

Niwre · August 1, 2023, 8:44am

@giacomo
After the update from nethserver-testing everything works fine again.
Thanks for the great support.

Best
Erwin

giacomo · August 1, 2023, 9:26am

The fix has been released.

rmk · August 1, 2023, 1:12pm

Thank you for the fix and update… it’s working now…
Much Appreciated !!!