[SOLVED] Backup fails to dump rh-mariadb105 database

please consider the subscription if you want to support the development of NethServer

3 Likes

Well another issue has sprouted, I’m not sure if it’s related to the two other original issues, but since it’s tied to NextCloud just like the others, I’m posting it here. Please feel free to split it off if need be.

When attempting to load the NextCloud interface from it’s designated VirtualHost URL, we are greeted with the following:

Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

I have tried restarting the server, reinstalling NextCloud, and PHP-FPM (7.2 & 7.3).

When running the command:

sudo -u apache scl enable rh-php73 – php -d memory_limit=512M /usr/share/nextcloud/occ

From the CLI, I receive the following error:

[22:06] [network ~] # sudo -u apache scl enable rh-php73 – php -d memory_limit=512M /usr/share/nextcloud/occ
An unhandled exception has been thrown:
Doctrine\DBAL\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [1045] Access denied for user ‘oc_admin’@‘localhost’ (using password: YES) in /usr/share/nextcloud/lib/private/DB/Connection.php:85
Stack trace:
#0 /usr/share/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1486): OC\DB\Connection->connect()
#1 /usr/share/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1014): Doctrine\DBAL\Connection->getWrappedConnection()
#2 /usr/share/nextcloud/lib/private/DB/Connection.php(226): Doctrine\DBAL\Connection->executeQuery(‘SELECT * FROM ...', Array, Array, NULL) #3 /usr/share/nextcloud/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php(210): OC\DB\Connection->executeQuery('SELECT * FROM …’, Array, Array)
#4 /usr/share/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php(287): Doctrine\DBAL\Query\QueryBuilder->execute()
#5 /usr/share/nextcloud/lib/private/AppConfig.php(345): OC\DB\QueryBuilder\QueryBuilder->execute()
#6 /usr/share/nextcloud/lib/private/AppConfig.php(110): OC\AppConfig->loadConfigValues()
#7 /usr/share/nextcloud/lib/private/AppConfig.php(301): OC\AppConfig->getApps()
#8 /usr/share/nextcloud/lib/private/legacy/OC_App.php(957): OC\AppConfig->getValues(false, ‘installed_versi…’)
#9 /usr/share/nextcloud/lib/private/Server.php(682): OC_App::getAppVersions()
#10 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(155): OC\Server->OC{closure}(Object(OC\Server))
#11 /usr/share/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility{closure}(Object(Pimple\Container))
#12 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(122): Pimple\Container->offsetGet(‘OC\Memcache\Fac…’)
#13 /usr/share/nextcloud/lib/private/ServerContainer.php(137): OC\AppFramework\Utility\SimpleContainer->query(‘OC\Memcache\Fac…’, false)
#14 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(56): OC\ServerContainer->query(‘OC\Memcache\Fac…’)
#15 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(177): OC\AppFramework\Utility\SimpleContainer->get(‘OC\Memcache\Fac…’)
#16 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(155): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility{closure}(Object(OC\Server))
#17 /usr/share/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(114): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility{closure}(Object(Pimple\Container))
#18 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(122): Pimple\Container->offsetGet(‘OCP\ICacheFacto…’)
#19 /usr/share/nextcloud/lib/private/ServerContainer.php(137): OC\AppFramework\Utility\SimpleContainer->query(‘OCP\ICacheFacto…’, false)
#20 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(56): OC\ServerContainer->query(‘OCP\ICacheFacto…’)
#21 /usr/share/nextcloud/lib/private/Server.php(1031): OC\AppFramework\Utility\SimpleContainer->get(‘OCP\ICacheFacto…’)
#22 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(155): OC\Server->OC{closure}(Object(OC\Server))
#23 /usr/share/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility{closure}(Object(Pimple\Container))
#24 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(122): Pimple\Container->offsetGet(‘OCP\Lock\ILocki…’)
#25 /usr/share/nextcloud/lib/private/ServerContainer.php(137): OC\AppFramework\Utility\SimpleContainer->query(‘OCP\Lock\ILocki…’, false)
#26 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(56): OC\ServerContainer->query(‘OCP\Lock\ILocki…’)
#27 /usr/share/nextcloud/lib/private/Server.php(1990): OC\AppFramework\Utility\SimpleContainer->get(‘OCP\Lock\ILocki…’)
#28 /usr/share/nextcloud/lib/private/Files/View.php(119): OC\Server->getLockingProvider()
#29 /usr/share/nextcloud/lib/private/Server.php(430): OC\Files\View->__construct()
#30 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(155): OC\Server->OC{closure}(Object(OC\Server))
#31 /usr/share/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility{closure}(Object(Pimple\Container))
#32 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(122): Pimple\Container->offsetGet(‘OC\Files\Node\H…’)
#33 /usr/share/nextcloud/lib/private/ServerContainer.php(137): OC\AppFramework\Utility\SimpleContainer->query(‘OC\Files\Node\H…’, false)
#34 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(56): OC\ServerContainer->query(‘OC\Files\Node\H…’)
#35 /usr/share/nextcloud/lib/private/Server.php(1353): OC\AppFramework\Utility\SimpleContainer->get(‘OC\Files\Node\H…’)
#36 /usr/share/nextcloud/lib/base.php(588): OC\Server->boot()
#37 /usr/share/nextcloud/lib/base.php(1076): OC::init()
#38 /usr/share/nextcloud/console.php(49): require_once(‘/usr/share/next…’)
#39 /usr/share/nextcloud/occ(11): require_once(‘/usr/share/next…’)
#40 {main}[22:12] [network ~] #

I’ve checked all other logs, including the NextCloud log in the Cockpit interface and there are no logs anywhere other than the command log from above that show any type of issue. Before anyone asks to verify, yes MySQL (mariadb) is running, and so is the service rh-mariadb105-mariadb@nextcloud, which I assume is still requried for NextCloud (despite there already being an instance of MySQL running… Absolutely NOTHING has been changed that I am aware of (and as the only Administrator of this server, can verify no one else has logged in and changed anything).

Help would be greatly appreciated.

On NethServer the default mysql user for nextcloud is nextcloud.
What’s the outcome of these commands?

grep \'db /usr/share/nextcloud/config/config.php |grep -v dbpass
/opt/rh/rh-mariadb105/root/bin/mysql --socket="/var/run/rh-mariadb105-mariadb/nextcloud-mysql.sock" -e 'SELECT user FROM mysql.user'
mysql -e 'SELECT user FROM mysql.user'
occ check    # most likely this one will fail again
[10:09] [network ~] # grep \'db /usr/share/nextcloud/config/config.php |grep -v dbpass
  'dbtype' => 'mysql',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost:/run/rh-mariadb105-mariadb/nextcloud-mysql.sock',
  'dbport' => '3306',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'oc_admin',
[10:09] [network ~] # /opt/rh/rh-mariadb105/root/bin/mysql --socket="/var/run/rh-mariadb105-mariadb/nextcloud-mysql.sock" -e 'SELECT user FROM mysql.user'
+-------------+
| User        |
+-------------+
| mariadb.sys |
| mysql       |
| nextcloud   |
| root        |
+-------------+
[10:10] [network ~] # mysql -e 'SELECT user FROM mysql.user'
+-----------+
| user      |
+-----------+
| oc_admin  |
| root      |
| root      |
| canis     |
| nextcloud |
| oc_admin  |
| root      |
| roundcube |
| smbd      |
| sogo      |
| sympa     |
+-----------+
[10:10] [network ~] # occ check    # most likely this one will fail again
An unhandled exception has been thrown:
Doctrine\DBAL\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [1045] Access denied for user 'oc_admin'@'localhost' (using password: YES) in /usr/share/nextcloud/lib/private/DB/Connection.php:85
Stack trace:
#0 /usr/share/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1486): OC\DB\Connection->connect()
#1 /usr/share/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1014): Doctrine\DBAL\Connection->getWrappedConnection()
#2 /usr/share/nextcloud/lib/private/DB/Connection.php(226): Doctrine\DBAL\Connection->executeQuery('SELECT * FROM `...', Array, Array, NULL)
#3 /usr/share/nextcloud/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php(210): OC\DB\Connection->executeQuery('SELECT * FROM `...', Array, Array)
#4 /usr/share/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php(287): Doctrine\DBAL\Query\QueryBuilder->execute()
#5 /usr/share/nextcloud/lib/private/AppConfig.php(345): OC\DB\QueryBuilder\QueryBuilder->execute()
#6 /usr/share/nextcloud/lib/private/AppConfig.php(110): OC\AppConfig->loadConfigValues()
#7 /usr/share/nextcloud/lib/private/AppConfig.php(301): OC\AppConfig->getApps()
#8 /usr/share/nextcloud/lib/private/legacy/OC_App.php(957): OC\AppConfig->getValues(false, 'installed_versi...')
#9 /usr/share/nextcloud/lib/private/Server.php(682): OC_App::getAppVersions()
#10 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(155): OC\Server->OC\{closure}(Object(OC\Server))
#11 /usr/share/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#12 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(122): Pimple\Container->offsetGet('OC\\Memcache\\Fac...')
#13 /usr/share/nextcloud/lib/private/ServerContainer.php(137): OC\AppFramework\Utility\SimpleContainer->query('OC\\Memcache\\Fac...', false)
#14 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(56): OC\ServerContainer->query('OC\\Memcache\\Fac...')
#15 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(177): OC\AppFramework\Utility\SimpleContainer->get('OC\\Memcache\\Fac...')
#16 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(155): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(OC\Server))
#17 /usr/share/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(114): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#18 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(122): Pimple\Container->offsetGet('OCP\\ICacheFacto...')
#19 /usr/share/nextcloud/lib/private/ServerContainer.php(137): OC\AppFramework\Utility\SimpleContainer->query('OCP\\ICacheFacto...', false)
#20 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(56): OC\ServerContainer->query('OCP\\ICacheFacto...')
#21 /usr/share/nextcloud/lib/private/Server.php(1031): OC\AppFramework\Utility\SimpleContainer->get('OCP\\ICacheFacto...')
#22 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(155): OC\Server->OC\{closure}(Object(OC\Server))
#23 /usr/share/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#24 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(122): Pimple\Container->offsetGet('OCP\\Lock\\ILocki...')
#25 /usr/share/nextcloud/lib/private/ServerContainer.php(137): OC\AppFramework\Utility\SimpleContainer->query('OCP\\Lock\\ILocki...', false)
#26 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(56): OC\ServerContainer->query('OCP\\Lock\\ILocki...')
#27 /usr/share/nextcloud/lib/private/Server.php(1990): OC\AppFramework\Utility\SimpleContainer->get('OCP\\Lock\\ILocki...')
#28 /usr/share/nextcloud/lib/private/Files/View.php(119): OC\Server->getLockingProvider()
#29 /usr/share/nextcloud/lib/private/Server.php(430): OC\Files\View->__construct()
#30 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(155): OC\Server->OC\{closure}(Object(OC\Server))
#31 /usr/share/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#32 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(122): Pimple\Container->offsetGet('OC\\Files\\Node\\H...')
#33 /usr/share/nextcloud/lib/private/ServerContainer.php(137): OC\AppFramework\Utility\SimpleContainer->query('OC\\Files\\Node\\H...', false)
#34 /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(56): OC\ServerContainer->query('OC\\Files\\Node\\H...')
#35 /usr/share/nextcloud/lib/private/Server.php(1353): OC\AppFramework\Utility\SimpleContainer->get('OC\\Files\\Node\\H...')
#36 /usr/share/nextcloud/lib/base.php(588): OC\Server->boot()
#37 /usr/share/nextcloud/lib/base.php(1076): OC::init()
#38 /usr/share/nextcloud/console.php(49): require_once('/usr/share/next...')
#39 /usr/share/nextcloud/occ(11): require_once('/usr/share/next...')
#40 {main}[10:10] [network ~] #

You could try changing dbuser to nextcloud on the config file (although not sure if dbpassword will match, but you can find nextcloud mysql user password in
/var/lib/nethserver/secrets/nextcloud) or recreate oc_admin mysql user and priviledges on rh-mariadb105 database.

Wouldn’t that just cause the issue to come back again next time the update command is run for Nextcloud?

signal-event nethserver-nextcloud-update

Setting dbuser to nextcloud and dbpassword to the one in the pointed file should not cause problems with the update event.
Like you have it now Nextcloud is most likely not working as expected (you can try to access nextcloud but it seems it has no access to the database).

The settings using oc_admin user could come from a manual installation (or a failed one).

Can you list the databases present in both instances of mariadb/mysql?

We never did a manual installation. Nextcloud was originally installed using the Cockpit GUI, and has been updated from that each time. That why I am confused on this issue as I am. There hasn’t been any third-party configuration or modification.

Here are the databases for both instances of MariaDB

[13:31] [network ~] # /opt/rh/rh-mariadb105/root/bin/mysql --socket=“/var/run/rh-mariadb105-mariadb/nextcloud-mysql.sock” -e ‘show databases;’
±-------------------+
| Database |
±-------------------+
| information_schema |
| mysql |
| nextcloud |
| performance_schema |
±-------------------+
[13:31] [network ~] # mysql -e ‘show databases;’
±-------------------+
| Database |
±-------------------+
| information_schema |
| canis |
| mysql |
| performance_schema |
| roundcube |
| smbaudit |
| sogo |
| sympa |
| ticketsystem |
±-------------------+
[13:32] [network ~] #

The database is where it is supposed to be.
occ is not working
Access to nextcloud using ip or domain?
Access to nextcloud through virtual host?

After correcting the config.php, the issue was fixed, but still raises the question of how this happened? Like I stated, we never modified Nextcloud, it was never “manually installed”. So I’m extremely confused.

Maybe a partial/failed upgrade. Answer would we in the logs (most likely in /var/log/messages). Glad it’s working now.

could you please explain what you modified, unfortunately we do not know

dbuser and dbpassword, if I understood correctly.

Apologies. As stated above. I changed the username in config.php to “nextcloud”, and updated the password to the one that was displayed by using this command:
/var/lib/nethserver/secrets/nextcloud

Which seems to have corrected the original Internal Server Error.

Yes, for some reason with the NC21 update, these issues all started. The system was working fine up until that point.

I have some doubts on your statement, this is hard to admit, we should find in our code a change on the admin mysql user of the nextcloud database.

I respect your opinion, but as I stated. I am the only person with access to this system. So any changes made to the NextCloud configuration were made by the system. Aside from modules we’ve added using the NextCloud web GUI.

Our daily backup is failing yet again. So this issue is still ongoing.

Without evidences I cannot help you

we need log traces in /var/log/messages, maybe the issues do not come from nextcloud, else you can simulate the pre-backup-data by

/etc/e-smith/events/actions/nethserver-nextcloud-rh-mariadb105-dump-database

Check in /var/lib/nethserver/backup/rh-mariadb105/ if you have the backup, and the time of the backup

For any errors, go to /var/log/messages and print us the lines to understand and fix it.

You need to help us to help you :smiley:

I may end up putting my foot in my mouth on this one but this issue appears to have solved itself mysteriously. Almost as mysteriously as it started, it appears to be fixed. We updated the NethServer instance using the Web GUI and noticed there was a NextCloud update, after that the issue stopped popping up. I will continue to monitor it but I think we’re all set now strangely.

Apologies for the delay in my response, I wanted to wait an appropriate amount of time after doing the update and not seeing errors, to see if it popped up again, but so far no issues.

@cfd10

Hi

Quite often, especially stuff involving databases, and to some extent also stuff using LDAP:

A lot of users forget that it can take TIME to upgrade a Database, depending on size, it can easily take longer than a few seconds - say an hour or two… (My Zabbix database is around 15 GB large!).

Then people are surprised, when an Issue “solves” itself without their doing anything…

They did isplay patience - enough, so the database got upgraded / fixed in the background. A reboot or refresh, and surprise, everythings working again!

My 2 cents
Andy