After doing a bit of research it appears a new service has been added after the upgrade, “rh-mariadb105-mariadb@nextcloud”. I don’t remember seeing this service here prior to the upgrade, I can’t start the service, presumably because there is already a MariaDB service running on the server.
It’s causing major issues with doing nightly backups. They fail with the following in the error log:
Jun 24 00:02:57 network esmith::event[2398]: Event: pre-backup-data mainbackup
Jun 24 00:03:07 network esmith::event[2398]: Action: /etc/e-smith/events/pre-backup-data/S20nethserver-backup-config-predatabackup SUCCESS [10.120327]
Jun 24 00:03:08 network esmith::event[2398]: Action: /etc/e-smith/events/pre-backup-data/S50mysql-dump-tables SUCCESS [1.160179]
Jun 24 00:03:08 network esmith::event[2398]: Action: /etc/e-smith/events/pre-backup-data/S50nethserver-ibays-dump-acls SUCCESS [0.004412]
Jun 24 00:03:08 network esmith::event[2398]: Action: /etc/e-smith/events/pre-backup-data/S60nethserver-nextcloud-rh-mariadb105-dump-database FAILED: 2 [0.013348]
Jun 24 00:03:11 network esmith::event[2398]: Action: /etc/e-smith/events/pre-backup-data/S80nethserver-sogo-backup-users SUCCESS [2.930666]
Jun 24 00:03:11 network esmith::event[2398]: Event: pre-backup-data FAILED
EDIT: I was able to temporarily fix the issue and at least take a backup by going into the file (/etc/e-smith/events/pre-backup-data/S60nethserver-nextcloud-rh-mariadb105-dump-database), and commenting out it’s contents.
While not ideal, I was still able to grab a backup. Looking at the logs from the previous backup (June 23 2021), that backup task does not exist, as you can see below.
Jun 23 00:00:03 network esmith::event[3518]: Event: pre-backup-data mainbackup
Jun 23 00:00:13 network esmith::event[3518]: Action: /etc/e-smith/events/pre-backup-data/S20nethserver-backup-config-predatabackup SUCCESS [10.115935]
Jun 23 00:00:14 network esmith::event[3518]: Action: /etc/e-smith/events/pre-backup-data/S50mysql-dump-tables SUCCESS [1.141623]
Jun 23 00:00:14 network esmith::event[3518]: Action: /etc/e-smith/events/pre-backup-data/S50nethserver-ibays-dump-acls SUCCESS [0.004579]
Jun 23 00:00:21 network esmith::event[3518]: Action: /etc/e-smith/events/pre-backup-data/S80nethserver-sogo-backup-users SUCCESS [6.72923]
Jun 23 00:00:21 network esmith::event[3518]: Event: pre-backup-data SUCCESS
Yes, that’s because NC21 now requires a newer version of MySQL/MariaDB. It should be able to run alongside the previous version. But if it won’t start, that’s going to break Nextcloud completely (not to mention the problem you’re seeing with your backups).
Please delete the above post (as it contains the server’s hostname, which for security we want redacted)
Redirecting to /bin/systemctl status rh-mariadb105-mariadb@nextcloud.service
● rh-mariadb105-mariadb@nextcloud.service - MariaDB 10.5 database server
Loaded: loaded (/etc/systemd/system/rh-mariadb105-mariadb@.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/rh-mariadb105-mariadb@nextcloud.service.d
└─nethserver.conf
Active: failed (Result: exit-code) since Thu 2021-06-24 11:55:45 EDT; 3s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 23090 ExecStartPre=/usr/bin/scl enable $RH_MARIADB105_SCLS_ENABLED -- /opt/rh/rh-mariadb105/root/usr/libexec/mysql-check-socket --defaults-group-suffix=.%I (code=exited, status=1/FAILURE)
Process: 23081 ExecStartPre=/usr/bin/scl enable $RH_MARIADB105_SCLS_ENABLED -- /usr/bin/scl_enabled rh-mariadb105 (code=exited, status=0/SUCCESS)
Jun 24 11:55:45 network systemd[1]: Starting MariaDB 10.5 database server...
Jun 24 11:55:45 network scl[23090]: Socket file /var/lib/mysql/mysql.sock exists.
Jun 24 11:55:45 network scl[23090]: Is another MySQL daemon already running with the same unix socket?
Jun 24 11:55:45 network scl[23090]: Please, stop the process using the socket /var/lib/mysql/mysql.s...vice.
Jun 24 11:55:45 network systemd[1]: rh-mariadb105-mariadb@nextcloud.service: control process exited,...tus=1
Jun 24 11:55:45 network systemd[1]: Failed to start MariaDB 10.5 database server.
Jun 24 11:55:45 network systemd[1]: Unit rh-mariadb105-mariadb@nextcloud.service entered failed state.
Jun 24 11:55:45 network systemd[1]: rh-mariadb105-mariadb@nextcloud.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
The rh-mariadb105-mariadb@nextcloud service is failing to start, because it sees an existing /var/lib/mysql/mysql.sock (which I assume is from the “normal” mysql for the system). This seems to be causing a variety of brokenness as described further up-topic.
Jun 24 11:55:45 network scl[23090]: Socket file /var/lib/mysql/mysql.sock exists.
Jun 24 11:55:45 network scl[23090]: Is another MySQL daemon already running with the same unix socket?
Jun 24 11:55:45 network scl[23090]: Please, stop the process using the socket /var/lib/mysql/mysql.s...vice.
Jun 24 11:55:45 network systemd[1]: rh-mariadb105-mariadb@nextcloud.service: control process exited,...tus=1
Jun 24 11:55:45 network systemd[1]: Failed to start MariaDB 10.5 database server.
[root@prometheus ~]# /opt/rh/rh-mariadb105/root/bin/mysqlshow --socket="/var/run/rh-mariadb105-mariadb/nextcloud-mysql.sock"
+--------------------+
| Databases |
+--------------------+
| information_schema |
| mysql |
| nextcloud |
| performance_schema |
+--------------------+
[root@prometheus ~]# /opt/rh/rh-mariadb105/root/bin/mysql --socket="/var/run/rh-mariadb105-mariadb/nextcloud-mysql.sock"
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 146
Server version: 10.5.8-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]>