Zabbix service don't start after update to 5.0.26

[pike (Michael Kicks)]

NethServer Version: 7.9
Module: Zabbix (courtesy of mr markuz)
After the update to Zabbix 5.0.26, the service does not start; currently the server is fully updated (even with openjdk 1.8.0whatever)

no yum operations make things happen different

# rpm -qa zabb*
zabbix-release-5.0-1.el7.noarch
zabbix-agent-5.0.26-1.el7.x86_64
zabbix-server-pgsql-5.0.26-1.el7.x86_64
zabbix-web-5.0.26-1.el7.noarch
zabbix-agent2-5.0.26-1.el7.x86_64

zabbix-agent is disabled, zabbix-agent2 is enabled as service.

Scrubbing among log, i found this row
zabbix-server.service: control process exited, code=exited status=1
I forgot: signal-event nethserver-zabbix-update didn’t change a thing

Edit: after stopping zabbix-agent2, i tried to start zabbix-server, than this appeared on the cockpit console
echo '{"action":"start","name":"zabbix-server"}' | /usr/bin/setsid /usr/bin/sudo /usr/libexec/nethserver/api/system-services/update | jq
as a “collateral” of dns.service_action_error

[mrmarkuz (Markus Neuberger)]

I can’t reproduce the issue.
Did you already check the logs in /var/log/zabbix/zabbix_* ?

Maybe this way you can see related errors:

systemctl status zabbix-server -l

[pike (Michael Kicks)]

Hope this helps…
systemctl status zabbix-server -l

● zabbix-server.service - Zabbix Server
   Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: protocol) since Mon 2022-08-01 11:43:27 CEST; 4s ago
  Process: 7167 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
 Main PID: 21627 (code=exited, status=0/SUCCESS)

Aug 01 11:43:27 not.your.interest systemd[1]: Failed to start Zabbix Server.
Aug 01 11:43:27 not.your.interest systemd[1]: Unit zabbix-server.service entered failed state.
Aug 01 11:43:27 not.your.interest systemd[1]: zabbix-server.service failed.

And into /var/log/zabbix/zabbix_server i found a lot of

zabbix_server [24039]: cannot create PID file [/var/run/zabbix/zabbix_server.pid]: [2] No such file or directory

Directory /zabbix was missing from /var/run then i created it, without changing a thing.

systemctl start zabbix-server
Job for zabbix-server.service failed. See "systemctl status zabbix-server.service" and "journalctl -xe" for details.

Then I asked to the suggested commands

● zabbix-server.service - Zabbix Server
   Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: protocol) since Mon 2022-08-01 11:47:44 CEST; 1s ago
  Process: 7858 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
 Main PID: 21627 (code=exited, status=0/SUCCESS)

Aug 01 11:47:44 not.your.interest systemd[1]: Failed to start Zabbix Server.
Aug 01 11:47:44 not.your.interest systemd[1]: Unit zabbix-server.service entered failed state.
Aug 01 11:47:44 not.your.interest systemd[1]: zabbix-server.service failed.

and

journalctl -xe
--
-- Unit zabbix-server.service has failed.
--
-- The result is failed.
Aug 01 11:47:31 not.your.interest systemd[1]: Unit zabbix-server.service entered failed state.
Aug 01 11:47:31 not.your.interest systemd[1]: zabbix-server.service failed.
Aug 01 11:47:31 not.your.interest polkitd[883]: Unregistered Authentication Agent for unix-process:7810:15265171 (system bus name :1.332, object path /org/freedesktop/Polic
Aug 01 11:47:42 not.your.interest systemd[1]: zabbix-server.service holdoff time over, scheduling restart.
Aug 01 11:47:42 not.your.interest systemd[1]: Stopped Zabbix Server.
-- Subject: Unit zabbix-server.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit zabbix-server.service has finished shutting down.
Aug 01 11:47:42 not.your.interest systemd[1]: Starting Zabbix Server...
-- Subject: Unit zabbix-server.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit zabbix-server.service has begun starting up.
Aug 01 11:47:44 not.your.interest systemd[1]: Can't open PID file /run/zabbix/zabbix_server.pid (yet?) after start: No such file or directory
Aug 01 11:47:44 not.your.interest systemd[1]: Failed to start Zabbix Server.
-- Subject: Unit zabbix-server.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit zabbix-server.service has failed.
--
-- The result is failed.
Aug 01 11:47:44 not.your.interest systemd[1]: Unit zabbix-server.service entered failed state.
Aug 01 11:47:44 not.your.interest systemd[1]: zabbix-server.service failed.
Aug 01 11:47:54 not.your.interest systemd[1]: zabbix-server.service holdoff time over, scheduling restart.
Aug 01 11:47:54 not.your.interest systemd[1]: Stopped Zabbix Server.
-- Subject: Unit zabbix-server.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit zabbix-server.service has finished shutting down.
Aug 01 11:47:54 not.your.interest systemd[1]: Starting Zabbix Server...
-- Subject: Unit zabbix-server.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit zabbix-server.service has begun starting up.

[mrmarkuz (Markus Neuberger)]

In /etc/zabbix/zabbix_server.conf you should have following line:

PidFile=/var/run/zabbix/zabbix_server.pid

Are you using the latest version of nethserver-zabbix?

[root@server2 ~]# rpm -q nethserver-zabbix
nethserver-zabbix-0.0.1-10.ns7.noarch

Did you set zabbix as owner for /var/run/zabbix ?

chown zabbix:zabbix /var/run/zabbix

Maybe it helps to reinstall the server:

yum reinstall zabbix-server

[pike (Michael Kicks)]

Thanks for your patience, time, effort.

@mrmarkuz

PidFile=/var/run/zabbix/zabbix_server.pid

row present into zammad_server.conf

@mrmarkuz

 rpm -q nethserver-zabbix
nethserver-zabbix-0.0.1-10.ns7.noarch

same output here

@mrmarkuz

Did you set zabbix as owner for /var/run/zabbix ?

I did not

ls -l /var/run/zabbix
total 4
-rw-r--r-- 1 zabbix zabbix 5 Aug  1 11:56 zabbix_agent2.pid

from /var/run

 ls -ll |grep zabbix
drwxr-xr-x  2 zabbix   zabbix     60 Aug  1 22:36 zabbix

Seemed unnecessary to me.
yum reinstall zabbix-server forced reinstall of zabbix-server-pgsql

yum reinstall zabbix-server
Loaded plugins: changelog, fastestmirror, nethserver_events
Determining fastest mirrors
epel/x86_64/metalink                                                                                                                                  |  30 kB  00:00:00
 * ce-base: it2.mirror.vhosting-it.com
 * ce-extras: it2.mirror.vhosting-it.com
 * ce-sclo-rh: it2.mirror.vhosting-it.com
 * ce-sclo-sclo: it2.mirror.vhosting-it.com
 * ce-updates: it2.mirror.vhosting-it.com
 * epel: it1.mirror.vhosting-it.com
 * nethforge: nethesis-mirror.nokken.co.uk
 * nethserver-base: nethesis-mirror.nokken.co.uk
 * nethserver-updates: nethesis-mirror.nokken.co.uk
ce-base/7/x86_64/signature                                                                                                                            |  811 B  00:00:00
ce-base/7/x86_64/signature                                                                                                                            | 3.6 kB  00:00:00 !!!
ce-extras/7/x86_64/signature                                                                                                                          |  811 B  00:00:00
ce-extras/7/x86_64/signature                                                                                                                          | 2.9 kB  00:00:00 !!!
ce-sclo-rh                                                                                                                                            | 3.0 kB  00:00:00
ce-sclo-sclo                                                                                                                                          | 3.0 kB  00:00:00
ce-updates/7/x86_64/signature                                                                                                                         |  811 B  00:00:00
ce-updates/7/x86_64/signature                                                                                                                         | 2.9 kB  00:00:00 !!!
epel                                                                                                                                                  | 4.7 kB  00:00:00
mrmarkuz/7/signature                                                                                                                                  |  490 B  00:00:00
mrmarkuz/7/signature                                                                                                                                  | 2.9 kB  00:00:00 !!!
nethforge/7/x86_64/signature                                                                                                                          |  836 B  00:00:00
nethforge/7/x86_64/signature                                                                                                                          | 3.6 kB  00:00:00 !!!
nethserver-base/7/x86_64/signature                                                                                                                    |  836 B  00:00:00
nethserver-base/7/x86_64/signature                                                                                                                    | 2.9 kB  00:00:00 !!!
nethserver-updates/7/x86_64/signature                                                                                                                 |  836 B  00:00:00
nethserver-updates/7/x86_64/signature                                                                                                                 | 3.6 kB  00:00:00 !!!
zabbix                                                                                                                                                | 2.9 kB  00:00:00
zabbix-frontend                                                                                                                                       | 2.9 kB  00:00:00
(1/3): epel/x86_64/updateinfo                                                                                                                         | 1.1 MB  00:00:00
(2/3): ce-sclo-rh/7/x86_64/primary_db                                                                                                                 | 3.4 MB  00:00:01
(3/3): epel/x86_64/primary_db                                                                                                                         | 7.0 MB  00:00:02
Resolving Dependencies
--> Running transaction check
---> Package zabbix-server-pgsql.x86_64 0:5.0.26-1.el7 will be reinstalled
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================================
 Package                                          Arch                                Version                                      Repository                           Size
=============================================================================================================================================================================
Reinstalling:
 zabbix-server-pgsql                              x86_64                              5.0.26-1.el7                                 zabbix                              3.1 M

Transaction Summary
=============================================================================================================================================================================
Reinstall  1 Package

Total download size: 3.1 M
Installed size: 7.1 M
Is this ok [y/d/N]: y
Downloading packages:
zabbix-server-pgsql-5.0.26-1.el7.x86_64.rpm                                                                                                           | 3.1 MB  00:00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : zabbix-server-pgsql-5.0.26-1.el7.x86_64                                                                                                                   1/1
  Verifying  : zabbix-server-pgsql-5.0.26-1.el7.x86_64                                                                                                                   1/1

Installed:
  zabbix-server-pgsql.x86_64 0:5.0.26-1.el7

Complete!

And now service runs as intended.

According to this Zabbix Official Repository packages was released 25/07, at least two days earlier than my upgrade.

Again, thanks for your patience, time, effort. I’m gonna create another post for the “solution”, but crediting it to you.

[pike (Michael Kicks)]

@mrmarkuz

Maybe it helps to reinstall the server:

yum reinstall zabbix-server

For my hiccup, this was the solution; the package zabbix-server-pgsql-5.0.26-1.el7.x86_64 was reinstalled after this command.

Credit due to @mrmarkuz for the troubleshooting and the correct hint for solving the problem.
(I also had to increase cache, but this was already covered here)