Netdata service not started

[capote (Marko)]

NethServer Version: NethServer release 7.7.1908 (final)
Module: Firewall, Webproxy / Webfilter, File Server, IPS, nTopNG,

Hello, I cannot start netdata service.

[root@nethserver2 ~]# systemctl start netdata
[root@nethserver2 ~]# systemctl status netdata
● netdata.service - Real time performance monitoring
   Loaded: loaded (/usr/lib/systemd/system/netdata.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Tue 2020-02-25 20:09:02 CET; 3s ago
  Process: 3441 ExecStart=/usr/sbin/netdata -P /var/run/netdata/netdata.pid -D -W set global process scheduling policy keep -W set global OOM score keep (code=exited, status=1/FAILURE)
  Process: 3439 ExecStartPre=/bin/chown -R netdata:netdata /var/run/netdata (code=exited, status=0/SUCCESS)
  Process: 3435 ExecStartPre=/bin/mkdir -p /var/run/netdata (code=exited, status=0/SUCCESS)
  Process: 3432 ExecStartPre=/bin/chown -R netdata:netdata /var/cache/netdata (code=exited, status=0/SUCCESS)
  Process: 3430 ExecStartPre=/bin/mkdir -p /var/cache/netdata (code=exited, status=0/SUCCESS)
 Main PID: 3441 (code=exited, status=1/FAILURE)

Feb 25 20:09:02 nethserver2.lan.home systemd[1]: netdata.service: main process exited, code=exited, status=1/FAILURE
Feb 25 20:09:02 nethserver2.lan.home systemd[1]: Unit netdata.service entered failed state.
Feb 25 20:09:02 nethserver2.lan.home systemd[1]: netdata.service failed.
[root@nethserver2 ~]#

I removed and reinstalled all optional modules, the issue still exists.

What can I do’

[mrmarkuz (Markus Neuberger)]

Do you get some more detailed error messages in /var/log/netdata/*.log ?

[capote (Marko)]

## /var/log/netdata/error.log
empty

## /var/log/netdata/debug.log
empty

## /var/log/netdata/access.log
empty

## /var/log/netdata/error.log-20200221

last entries

2020-02-21 00:34:13: netdata ERROR : PLUGINSD[apps] : read failed: end of file
2020-02-21 00:34:13: netdata ERROR : PLUGINSD[apps] : ‘/usr/libexec/netdata/plugins.d/apps.plugin’ (pid 30309) disconnected after 68072 successful data collections (ENDs).
2020-02-21 00:34:13: netdata ERROR : PLUGIN[tc] : child pid 30299 killed by signal 15.
2020-02-21 00:34:13: netdata ERROR : PLUGINSD[apps] : child pid 30309 killed by signal 15.
2020-02-21 00:34:13: netdata INFO : PLUGINSD[apps] : ‘/usr/libexec/netdata/plugins.d/apps.plugin’ (pid 30309) was killed with SIGTERM. Disabling it.
2020-02-21 00:34:13: netdata INFO : PLUGINSD[apps] : thread with task id 30288 finished
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
2020-02-21 00:34:13: netdata ERROR : PLUGINSD[python.d] : read failed: end of file
2020-02-21 00:34:13: netdata ERROR : PLUGINSD[python.d] : ‘/usr/libexec/netdata/plugins.d/python.d.plugin’ (pid 30300) disconnected after 30562 successful data collections (ENDs).
2020-02-21 00:34:13: netdata ERROR : PLUGINSD[python.d] : child pid 30300 killed by signal 15.
2020-02-21 00:34:13: netdata INFO : PLUGINSD[python.d] : ‘/usr/libexec/netdata/plugins.d/python.d.plugin’ (pid 30300) was killed with SIGTERM. Disabling it.
2020-02-21 00:34:13: netdata INFO : PLUGINSD[python.d] : thread with task id 30294 finished
2020-02-21 00:34:14: netdata INFO : MAIN : EXIT: cleaning up the database…
2020-02-21 00:34:14: netdata INFO : MAIN : Cleaning up database [1 hosts(s)]…
2020-02-21 00:34:14: netdata INFO : MAIN : Cleaning up database of host ‘nethserver2.lan.home’…
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
2020-02-21 00:34:14: tc-qos-helper.sh: WARNING: Cannot find file ‘/etc/netdata/conf.d/tc-qos-helper.conf’.
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
2020-02-21 00:34:14: tc-qos-helper.sh: WARNING: Cannot find file ‘/etc/netdata/tc-qos-helper.conf’.
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
job-working-directory: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
2020-02-21 00:34:15: netdata INFO : MAIN : EXIT: stopping master threads…
2020-02-21 00:34:15: netdata INFO : MAIN : EXIT: Stopping master thread: PLUGIN[proc]
2020-02-21 00:34:15: netdata INFO : MAIN : EXIT: Stopping master thread: PLUGIN[diskspace]
2020-02-21 00:34:15: netdata INFO : MAIN : EXIT: Stopping master thread: PLUGIN[cgroups]
2020-02-21 00:34:15: netdata INFO : MAIN : EXIT: Stopping master thread: PLUGIN[tc]
2020-02-21 00:34:15: netdata INFO : MAIN : EXIT: Stopping master thread: PLUGIN[idlejitter]
2020-02-21 00:34:15: netdata INFO : MAIN : EXIT: Stopping master thread: STATSD
2020-02-21 00:34:15: netdata INFO : MAIN : EXIT: Stopping master thread: WEB_SERVER[static1]
2020-02-21 00:34:15: netdata INFO : PLUGIN[proc] : cleaning up…
2020-02-21 00:34:15: netdata INFO : PLUGIN[proc] : thread with task id 30276 finished
2020-02-21 00:34:15: netdata INFO : PLUGIN[tc] : cleaning up…
2020-02-21 00:34:15: netdata INFO : PLUGIN[tc] : TC: killing with SIGTERM tc-qos-helper process 16117
2020-02-21 00:34:15: netdata INFO : PLUGIN[tc] : TC: waiting for tc plugin child process pid 16117 to exit…
2020-02-21 00:34:15: netdata INFO : PLUGIN[tc] : thread with task id 30279 finished
2020-02-21 00:34:15: netdata INFO : STATSD : cleaning up…
2020-02-21 00:34:15: netdata INFO : STATSD : STATSD: stopping data collection thread 1…
2020-02-21 00:34:15: netdata INFO : STATSD : STATSD: closing sockets…
2020-02-21 00:34:15: netdata INFO : STATSD : STATSD: cleanup completed.
2020-02-21 00:34:15: netdata INFO : STATSD : thread with task id 30281 finished
2020-02-21 00:34:15: netdata INFO : STATSD_COLLECTOR[1] : cleaning up…
2020-02-21 00:34:15: netdata INFO : STATSD_COLLECTOR[1] : thread with task id 30297 finished
2020-02-21 00:34:15: netdata INFO : PLUGIN[idlejitter] : cleaning up…
2020-02-21 00:34:15: netdata INFO : PLUGIN[diskspace] : cleaning up…
2020-02-21 00:34:15: netdata INFO : PLUGIN[cgroups] : cleaning up…
2020-02-21 00:34:15: netdata INFO : PLUGIN[idlejitter] : thread with task id 30280 finished
2020-02-21 00:34:15: netdata INFO : MAIN : EXIT: Stopping master thread: PLUGINSD
2020-02-21 00:34:15: netdata INFO : PLUGIN[diskspace] : thread with task id 30277 finished
2020-02-21 00:34:15: netdata INFO : PLUGIN[cgroups] : thread with task id 30278 finished
2020-02-21 00:34:15: netdata INFO : MAIN : EXIT: Stopping master thread: HEALTH
2020-02-21 00:34:15: netdata INFO : MAIN : Waiting 9 threads to finish…
2020-02-21 00:34:15: netdata INFO : WEB_SERVER[static1] : freeing local web clients cache…
2020-02-21 00:34:15: netdata INFO : WEB_SERVER[static1] : stopped after 0 connects, 0 disconnects (max concurrent 0), 0 receptions and 0 sends
2020-02-21 00:34:15: netdata INFO : WEB_SERVER[static1] : stopping worker 2
2020-02-21 00:34:15: netdata INFO : HEALTH : cleaning up…
2020-02-21 00:34:15: netdata INFO : HEALTH : thread with task id 30285 finished
2020-02-21 00:34:15: netdata INFO : WEB_SERVER[static2] : freeing local web clients cache…
2020-02-21 00:34:15: netdata INFO : WEB_SERVER[static2] : stopped after 0 connects, 0 disconnects (max concurrent 0), 0 receptions and 0 sends
2020-02-21 00:34:15: netdata INFO : WEB_SERVER[static2] : thread with task id 30287 finished
2020-02-21 00:34:15: netdata INFO : WEB_SERVER[static1] : stopping worker 3
2020-02-21 00:34:15: netdata INFO : WEB_SERVER[static1] : stopping worker 4
2020-02-21 00:34:15: netdata INFO : WEB_SERVER[static1] : Waiting 3 static web threads to finish…
2020-02-21 00:34:15: netdata INFO : PLUGINSD : cleaning up…
2020-02-21 00:34:15: netdata INFO : PLUGINSD : cleanup completed.
2020-02-21 00:34:15: netdata INFO : PLUGINSD : thread with task id 30284 finished
2020-02-21 00:34:15: netdata INFO : WEB_SERVER[static4] : freeing local web clients cache…
2020-02-21 00:34:15: netdata INFO : WEB_SERVER[static4] : stopped after 0 connects, 0 disconnects (max concurrent 0), 0 receptions and 0 sends
2020-02-21 00:34:15: netdata INFO : WEB_SERVER[static4] : thread with task id 30296 finished
2020-02-21 00:34:15: netdata INFO : WEB_SERVER[static3] : freeing local web clients cache…
2020-02-21 00:34:15: netdata INFO : WEB_SERVER[static3] : stopped after 0 connects, 0 disconnects (max concurrent 0), 0 receptions and 0 sends
2020-02-21 00:34:15: netdata INFO : WEB_SERVER[static3] : thread with task id 30295 finished
2020-02-21 00:34:15: netdata INFO : WEB_SERVER[static1] : closing all web server sockets…
2020-02-21 00:34:15: netdata INFO : WEB_SERVER[static1] : all static web threads stopped.
2020-02-21 00:34:15: netdata INFO : WEB_SERVER[static1] : thread with task id 30283 finished
2020-02-21 00:34:15: netdata INFO : MAIN : All threads finished.
2020-02-21 00:34:15: netdata INFO : MAIN : EXIT: freeing database memory…
2020-02-21 00:34:15: netdata INFO : MAIN : Freeing all memory for host ‘nethserver2.lan.home’…
2020-02-21 00:34:15: netdata INFO : MAIN : SYSTEM_INFO: free 0x55d82743ce00
2020-02-21 00:34:15: netdata INFO : MAIN : EXIT: removing netdata PID file ‘/var/run/netdata/netdata.pid’…
2020-02-21 00:34:15: netdata INFO : MAIN : EXIT: all done - netdata is now exiting - bye bye…

## /var/log/netdata/access.log-20200221

last entries

2020-02-20 23:43:36: 159: 29879 ‘[localhost]:45804’ ‘CONNECTED’
2020-02-20 23:43:36: 159: 29879 ‘[localhost]:45804’ ‘DISCONNECTED’
2020-02-20 23:43:36: 159: 29879 ‘[localhost]:45804’ ‘DATA’ (sent/all = 27332/27332 bytes -0%, prep/sent/total = 79.14/12.03/91.17 ms) 200 ‘//api/v1/data’
2020-02-20 23:43:36: 160: 29879 ‘[localhost]:45810’ ‘CONNECTED’
2020-02-20 23:43:36: 160: 29879 ‘[localhost]:45810’ ‘DISCONNECTED’
2020-02-20 23:43:36: 160: 29879 ‘[localhost]:45810’ ‘DATA’ (sent/all = 23130/23130 bytes -0%, prep/sent/total = 9.10/12.73/21.83 ms) 200 ‘//api/v1/data’

[mrmarkuz (Markus Neuberger)]

Are all packages installed?

[root@testserver ~]# rpm -qa *netdata*
netdata-data-1.18.1-1.el7.noarch
nethserver-netdata-1.1.0-1.ns7.noarch
netdata-1.18.1-1.el7.x86_64
netdata-conf-1.18.1-1.el7.noarch

Did you try to restart the service?

systemctl restart netdata

[capote (Marko)]

@mrmarkuz

Are all packages installed?

yes

root@nethserver2 ~]# rpm -qa *netdata*
netdata-conf-1.18.1-1.el7.noarch
netdata-1.18.1-1.el7.x86_64
nethserver-netdata-1.1.0-1.ns7.noarch
netdata-data-1.18.1-1.el7.noarch
root@nethserver2 ~#

Did you try to restart the service?

sure

[capote (Marko)]

I reinstalled nethserver …

thank you for your support

[oxcar9 (Oscar)]

hi, how do you reinstall nethserver???, I tried download new iso form repository, first install centos minimal and then install nethserver, no way works

[root@fw ~]# systemctl status netdata
● netdata.service - Real time performance monitoring
Loaded: loaded (/usr/lib/systemd/system/netdata.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/netdata.service.d
└─cleanup.conf
Active: activating (auto-restart) (Result: exit-code) since jue 2023-06-01 18:23:31 -05; 26s ago
Process: 29105 ExecStart=/usr/sbin/netdata -P /run/netdata/netdata.pid -D (code=exited, status=1/FAILURE)
Process: 29102 ExecStartPre=/usr/bin/find /var/cache/netdata/ -type f -empty -delete (code=exited, status=0/SUCCESS)
Process: 29099 ExecStartPre=/bin/chown -R netdata /run/netdata (code=exited, status=0/SUCCESS)
Process: 29096 ExecStartPre=/bin/mkdir -p /run/netdata (code=exited, status=0/SUCCESS)
Process: 29093 ExecStartPre=/bin/chown -R netdata /var/cache/netdata (code=exited, status=0/SUCCESS)
Process: 29091 ExecStartPre=/bin/mkdir -p /var/cache/netdata (code=exited, status=0/SUCCESS)
Main PID: 29105 (code=exited, status=1/FAILURE)

jun 01 18:23:31 fw.gaval.pe systemd[1]: netdata.service: main process exited, code=exited, status=1/FAILURE
jun 01 18:23:31 fw.gaval.pe systemd[1]: Unit netdata.service entered failed state.
jun 01 18:23:31 fw.gaval.pe systemd[1]: netdata.service failed.

[mrmarkuz (Markus Neuberger)]

Hi Oscar,

and welcome to NethServer community. Let’s talk about the issue in just one thread:

Netdata cannot start correctly, web proxy, services

I recommend to install from NethServer ISO if possible. Did you update the system? Maybe it’s about netdata cache issues? https://community.nethserver.org/t/out-of-5-servers-the-netdata-status-doesnt-appear-optimal-and-1-keeps-crashing/21515/4?u=mrmarkuz

[capote (Marko)]

@oxcar9

hi, how do you reinstall nethserver???

The best way is to do a fresh installallation from the ISO.
Maybe you can try to reinstall as it is described in the user guides.

https://wiki.nethserver.org/doku.php?id=howto:useful_commands#reinstall_all_required_packages

[oxcar9 (Oscar)]

Thanks for answering!!!, I have downloaded 4 isos on different PCs, from the site https://github.com/NethServer/dev/releases/download/iso-7.9.2009/nethserver-7.9.2009-x86_64.iso

About netdata cache issues:

Try to reinstall from terminal:

As additional information, Nethserver is being installed in a VM on a Proxmox 7.3-3 server, normally I perform the tests on this server before doing it in a production environment.