Zabbix 5.0 LTS has been released

Thanks for testing! :+1:

I only found that post_max_size = 16M is required and we use 8M by default.

https://www.zabbix.com/documentation/current/manual/installation/install#installing_zabbix_web_interface

Thanks for the pointer, this seems mandatory for updating the history.

This could be an issue as regards big databases because the update may take a long time and in worst case lead to an inconsistent db when interrupted…

1 Like

Hi Markus,

I found this link about $DB['DOUBLE_IEEE754'] = 'true'; Upgrade Zabbix (5.0, 5.2, 5.4) to 6.0 like a Pro [+6.0 to 6.4 guide]

Step 6: Patch DB and fix warning “database is not upgraded to use double precision values”

With Zabbix 5.0 float data type supports precision of approximately 15 digits and bigger range. This is by default for new installations, but if you are upgrading then a manual database upgrade patch must be applied.

Download the MySQL/MariaDB patch “ double.sql ” and apply it (PostGreSql users need to change “mysql” with “postgresql” in the URL path):

To download the patch:

# wget https://git.zabbix.com/projects/ZBX/repos/zabbix/raw/database/mysql/double.sql

I am using MariaDB-10.2 and the name of the database for Zabbix is zabbixDB.
I applied the patch with:

# mysql102 zabbixDB < double.sql

Before and after the patch to verify:

# mysql102 zabbixDB -e "show create table history;"

Also, when you start zabbix-sever for the first time after updating, you have to check the updating of the database by zabbix-server:

# cat /var/log/zabbix/zabbix_server.log | grep database

...
  6948:20200602:120547.195 current database version (mandatory/optional): 04000000/04000006
  6948:20200602:120547.195 starting automatic database upgrade
  6948:20200602:120547.384 completed 0% of database upgrade
  6948:20200602:120547.486 completed 1% of database upgrade
...
  6948:20200602:120602.950 completed 100% of database upgrade
  6948:20200602:120602.951 database upgrade fully completed
  6948:20200602:120602.962 database is not upgraded to use double precision values
#

The last line is there if you didn’t apply the patch.

Michel-André

4 Likes

Hi Markus,

Référence: Dmitry Lambert

Don’t underestimate Database tuning

Hey guys, It’s been a while for me posting here. Mostly because I am kind’a struggling with balancing content creation over Patreon and Youtube platforms. Reason is quite simple :slight_smile: I want to post for as large audience as possible, but at the same time my dear patreons are supporting me also in financial way, so they(you) deserve some extra right? :slight_smile: And another minus of Patreon platform is formatting of text… which is fine for just blogging, but just a nightmare to share any code snippets.

Alright, off from introduction and back to topic. Somehow, the most simple thing which has been pushed to Zabbix users in all possible ways and platforms still get’s underestimated. I really have a feeling that 9 out of 10 installations have their database configuration file with default settings. Trust me, It is straight road to performance problems with Zabbix, even if you are not ( and you won’t ) feel any of those right after your installation.

I understand that fine tuning kernel parameters, filesystem is basically a min/maxing, but database configuration tuning is a must!

Let me briefly explain what is happening when your database is not optimized. You deploy your Zabbix instance where database by default would weight probably around 10MB which will forgive any mistakes or lazyness of tuning. You have a default set of processes from Zabbix server, which probably varies around 20-30, Housekeeper runs in couple of seconds, you don’t see any slow queries in server log, all frontend pages are opening in a second, so everything is totally fine.

Thing is that database size will grow. Especially after a clean install, it simply must grow! How much? That strictly depends on NVPS ( New values processed by server ) and your history storage period. Let’s take one example based on simple formulas.

300 hosts
120 average items per host
36000 Total items
60 seconds average update interval
60 days storing history
1 year for trends 

Such setup just like any other will start only from 10-20 Mb of database and will grow very fast for next 60 days ( history retention period ) then growth will slow down, because housekeeper will start to delete data. So in general, after 2 months, we should have it like this:

600 NVPS
418G database

At this moment Housekeeper finally will start to work on a full load, because in next day ( 61 ) there will be 1 day of data to delete. Database size is much bigger, which worsens performance of Zabbix server and also housekeeper.

As a result we will eventually start to receive notifications from Trigger is more then 75% busy. In server log file we will start to notice Slow queries ( by default they are noted for every query that is slower then 3 seconds ) Initially most of those queries will be by housekeeper process with DELETE statements to database.

When housekeeper is not able to keep up with all the data ( You write more then housekeeper is able to delete ) Database growth will continue. When database size increases, housekeeper becomes even more slower. So it is able to delete even less data, so database grows even more. I believe you understood where this is going.

And it would not be that bad if all of this would affect only housekeeper. When housekeeper is slow and runs for many minutes, what it does is scanning History tables for data to delete. Remember that those are same tables where History syncers have to insert new data. While housekeeper is scanning, history syncer is not able to insert new data. So then we will start to receive other alerts. History syncer more then 75% busy. Nodata triggers storm, internal process graph busy graph would show most of processes 100% busy. Gaps in the graphs, slow queries in the log. And all of this would be corelated with cycles of Housekeeper, when eventually you will reach situation when housekeeper will run for 1 hour ( by default it cycles every hour ) so basically it will be running 24/7

You might in panic throw more memory on your database server, but without touching configuration file, it won’t help at all. Thing is that database will not utilize your server resoruces up to maximum. You need to allow database to do that.

Even now, simply open SSH to your database server and type free -h

If you see couple of Gigs listed as available memory, it means that your database could be using this memory to increase performance. But it’s not, simply because you did not allow it to do that.

I understand that tuning the database also required deeper knowledge of what to do, when and how much. That’s why I will post you just two websites that focus on most critical parameters that will dramatically improve performance of your database.

For MySQL/MariaDB use this URL:

For PostgreSQL use this URL: https://pgtune.leopard.in.ua/#/

Do the tuning guys!

Michel-André
P.S. I received this by email because I support financially (too small amount) Dmitri at Patreon: https://www.patreon.com/dmitryl

3 Likes

@michelandre

Hi Michel-André

I know you use MySQL for your Zabbix. @mrmarkuz modul uses PGSQL, which is what I use.
That Zabbix Housekeeping can’t cope with cleaning is fairly well known.

That means the Zabbix will grow quite fast, under normal conditions. If you have “unseen” issues, like what I had: Network wiring in the wall, installed 20 years ago with Cat6 cables…
Last year, those cables deteroriated from Gigabit down to about max 150 MBit/S… Errors on monitored Switches and PCs/Servers grew exponentially.

We are talking about a small office with 5 employees… The DB went to around 100 GB… My client had an electrician put in new wiring, so now that works. But wiring in the wall is often overlooked - it’s in the wall. But it’s nothing else than wiring you do not see, nor do you see any damages or corrossion…

Cleaning that up helped me get the necessary stuff together for PG, for MySQL there’s quite a lot of infos around…
I now have a script for regular cleaning, at least for PG…

Note: Optimizing the DB is still a necessity, but that doesn’t mean cleaning should be neglected…

Thx for the PGSQL optimization link…

My 2 cents
Andy

finally some time to test the upgrade from a 4.0.x,
1st try, after installed the upgrade, a yum update:

Resolving Dependencies
→ Running transaction check
—> Package zabbix-agent.x86_64 0:4.0.23-1.el7 will be updated
—> Package zabbix-agent.x86_64 0:5.0.2-1.el7 will be an update
—> Package zabbix-server-pgsql.x86_64 0:4.0.23-1.el7 will be updated
—> Package zabbix-server-pgsql.x86_64 0:5.0.2-1.el7 will be an update
—> Package zabbix-web.noarch 0:4.0.23-1.el7 will be updated
→ Processing Dependency: zabbix-web = 4.0.23-1.el7 for package: zabbix-web-pgsql-4.0.23-1.el7.noarch
—> Package zabbix-web.noarch 0:5.0.2-1.el7 will be an update
→ Finished Dependency Resolution
Error: Package: zabbix-web-pgsql-4.0.23-1.el7.noarch (@zabbix)
Requires: zabbix-web = 4.0.23-1.el7
Removing: zabbix-web-4.0.23-1.el7.noarch (@zabbix)
zabbix-web = 4.0.23-1.el7
Updated By: zabbix-web-5.0.2-1.el7.noarch (zabbix-frontend)
zabbix-web = 5.0.2-1.el7
Available: zabbix-web-5.0.0-1.el7.noarch (zabbix-frontend)
zabbix-web = 5.0.0-1.el7
Available: zabbix-web-5.0.1-1.el7.noarch (zabbix-frontend)
zabbix-web = 5.0.1-1.el7
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest

2nd try:
yum remove zabbix-web-*
yum install https://repo.zabbix.com/zabbix/5.0/rhel/7/x86_64/zabbix-release-5.0-1.el7.noarch.rpm

[zabbix-frontend]
...
enabled=1
...

yum --enablerepo=zabbix-frontend install https://mrmarkuz.dynu.net/mirror/devtest/nethserver-zabbix-0.0.1-8.ns7.noarch.rpm

yum clean all && yum update

seems ok (on a clean zabbix install) this seems a cleaner path to upgrade or am i missing something?

1 Like

I didn’t find a better upgrade method than remove and install yet. Thanks for testing!

I’m working on the module, I hope I can provide an update soon.

4 Likes

Hi Markus, thanks for your work. Great as always!! :smiley:

I did a clear install on a test vm and configured a pc with the agent.
Seems to work. Server and PC are communicating.

But I get some Warnings in the systeminformation field:

grafik

You configured a config-database “php” in witch I found 2 of this options. But when I change them to the suggested values and expand the template, nothing changes in Zabbix. Even after a reboot. :thinking:

2 Likes

Thanks for testing!

I am going to provide an update soon with correct php settings and also the database history setting, see Zabbix 5.0 LTS has been released - #20 by dz00te

This is the config db for the default php, not rh-php73.

In the meanwhile you could create following files to get rid of the warnings:

Create /etc/opt/rh/rh-php73/php-fpm.d/zabbix.conf with following content:

php_value[post_max_size] = 16M
php_value[max_execution_time] = 300
php_value[max_input_time] = 300
php_value[date.timezone] = Europe/Vienna

Edit /etc/zabbix/web/zabbix.conf.php and add

$DB['DOUBLE_IEEE754'] = true;

Stop zabbix-server:

systemctl stop zabbix-server

Add patch to db:

cat $(rpm -ql zabbix-server-pgsql | grep double.sql) | sudo -u zabbix psql zabbix

Restart services:

systemctl restart zabbix-server rh-php73-php-fpm

4 Likes

Worked perfectly!! Thanks!!

1 Like

tnx! works great! :+1: :vulcan_salute:

1 Like

I updated the testing package.

  • Included php settings, timezone is set to Nethserver timezone
  • Included DB settings
  • New images by @Andy_Wismer
  • Removed the old zabbix backup script as @syntaxerrormmm provides an RPM for easy deployment

Thanks for testing!

To install:

yum -y install https://repo.zabbix.com/zabbix/5.0/rhel/7/x86_64/zabbix-release-5.0-1.el7.noarch.rpm
yum -y --enablerepo=zabbix,zabbix-frontend install https://mrmarkuz.dynu.net/mirror/devtest/nethserver-zabbix-0.0.1-8.ns7.noarch.rpm

For updating you need to remove the old zabbix packages and install the new one as described here.

If you already tested you need to reinstall to get the new version:

yum -y --enablerepo=zabbix,zabbix-frontend reinstall https://mrmarkuz.dynu.net/mirror/devtest/nethserver-zabbix-0.0.1-8.ns7.noarch.rpm

6 Likes

I installed Zabbix Agents inside my Nethserver and Proxmox-System.
Both are active (stared and enabled) but inside of the Zabbix host configuration both are not connected.
Do you have a hint or me?

Best regards, Marko

1 Like

You need to open port 10050 TCP on the machines that have only the agent.

For the nethserver I did:

  1. Creating new service “zabbix-agent”
  2. I opend the firewall
  3. I created a local role

The Zabbix agent isn’t still accessible, but the firewall don’t rejects anymore. Last entry before FW-Konfiguration
Sep 14 17:30:36 nethserver kernel: Shorewall:loc2fw:REJECT:IN=br0 OUT= MAC=xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx SRC=192.168.3.210 DST=192.168.3.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=36828 DF PROTO=TCP SPT=57568 DPT=10050 WINDOW=29200 RES=0x00 SYN URGP=0

For the Proxmox system I have no idea how to open the port. The firewall is inactive.

The zabbix agent is running?

systemctl status zabbix-agent

Please check if zabbix-agent is listening:

netstat -tlpn | grep 10050

# systemctl status zabbix-agent 
zabbix-agent.service - Zabbix Agent
   Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-09-13 19:03:37 CEST; 23h ago


# netstat -tlpn | grep 10050
tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN      9496/zabbix_agentd
tcp6       0      0 :::10050                :::*                    LISTEN      9496/zabbix_agentd

Did you setup a firewall rule on the proxmox?

@capote

Hi Marko

Are both Zabbix Agents correctly configured with correct hostname (Needs to match the zabbix server entry, not necessarily the real hostname!) and IP (On GREEN!)?

I usually put both in double: under Active check and Server (For the server) and under SourceIP and ListenIP (for the Agent itself) in zabbix_agentd.conf…

My 2 cents
Andy

1 Like

Did you setup a firewall rule on the proxmox?

no, I don’t have activated the FW on the Proxmox, Should I do?
I am not enthusiastic about increasing the complexity…

No, it was just an idea. If it’s not active it should be ok too.

1 Like