NethServer Version: 7.9 Module: Nextcloud or backup
Had to reboot my Nethserver this morning after 100+ days of uptime, as an NFS mount had frozen and I wasn’t able to unmount the volume. After the system booted, I went back into Cockpit to restart the backup. And I’m seeing two strange things, which I suspect are related:
After over three hours’ of uptime, the pre-backup-data event is still running:
There are tons of MySQL 10.5 processes running, chewing up lots of CPU:
Not sure where to go from here, but this doesn’t seem right.
Edit: after four hours’ uptime, the pre-backup-data event finished, and the mysql processes went away. The backup failed because of a lock in the restic repo, but that’s easy enough to clear. Restarting the backup resulted in the pre-backup-data finishing much more quickly. Strange.
How many instances, tables and gigabytes is your DB server?
Also: did you ever considered to… have some sort of management/monitoring tool for MySQL to understand what it’s doing?
Not sure about the first two, but du -sh /var/lib/mysql reports 1.6 GB, and I count 24 directories in there, which would correspond to 24 databases.
Have not considered this so far, but it’s looking like it might be a good idea.
Really, it seems the problem has gone away (three subsequent backup jobs have completed without the pre-backup-data event taking an unusual amount of time), but I’m left with no idea what caused it. I guess I could investigate further by rebooting my production server and trying again (to see if it happens consistently), but I’m not inclined to reboot that server if I don’t need to (which is why it had 113 days of uptime yesterday before I rebooted it). But for some reason, that first pre-backup-data event after rebooting took four hours to complete, and based on the processes that were chewing up CPU time, it looks like is was the part where it was dumping the Nextcloud database that was doing it.
