Nextcloud didn't redirect to new database during upgrade

Support
, ,
11

I just sent you a link to the file… I was trying to not post too much.

12

Blind shot: maybe related to memory consumption.

1 Like
13

Will check tomorrow, the TZ is to go to bed here

14

I got the log, let me dive inside

15

please @Socs28 can you share with us about the server, its story and its hardware

16

the error happened at 21h45, this is the code in the action

# Create the database in rh-mariadb105 and import the dump from mariadb55 or exit on error
mysql --socket=/run/rh-mariadb105-mariadb/nextcloud-mysql.sock -e "CREATE DATABASE IF NOT EXISTS nextcloud;"
mysql --socket=/run/rh-mariadb105-mariadb/nextcloud-mysql.sock --database nextcloud < $tmp_sql
if [[ $? -ne 0 ]]; then
    exitOnError "[ERROR] Error while restoring nextcloud database to SCL mariadb105"
fi

we succeed to create the database and dumping the old DB, but we failed to restore the database mariadb55 to mariadb105, so mariadb105 is completely empty and you run on the mariadb55.

However between 17h38 and 21h45 your server was just waiting and did nothing. (at least in logs).

do we have more interesting things in (25 june 5h30PM-9h45PM)

/var/log/rh-mariadb105/*nextcloud.log*

17

please @Socs28 can you share with us about the server, its story and its hardware

The machine is running on a Proxmox VM with 2CPU’s 2cores each and 2G of RAM dedicated to this VM. I’ve raised the RAM to 4G, but didn’t take the chance to shut it down so that could activate before doing the upgrade.

/var/log/rh-mariadb105/nextcloud.log

I will send you a link to that file upload.

18

How many users you have on your VM, 2gb is a minimum

19

5 to 10 at a time

1 Like
20

got the nextcloud.log file, need to dive inside, but indeed the mysql server instance got an unattended issue. The good new is that you have the old database safe and running, we could think to try to upgrade again, also because you run a VM and you can do a snapshot if you want to go back. However I would like you make also a mysqldump of the mariadb55 before to migrate.

2021-06-25 17:38:15 0 [Note] /opt/rh/rh-mariadb105/root/usr/libexec/mysqld: ready for connections.
Version: '10.5.8-MariaDB'  socket: '/run/rh-mariadb105-mariadb/nextcloud-mysql.sock'  port: 0  MariaDB Server
2021-06-25 20:09:53 0 [Warning] InnoDB: A long semaphore wait:
--Thread 140303167837952 has waited at btr0cur.cc line 1480 for 241.00 seconds the semaphore:
SX-lock on RW-latch at 0x7f9aa421be90 created in file dict0dict.cc line 2161
a writer (thread id 140303193323264) has reserved it in mode  SX
number of readers 0, waiters flag 1, lock_word: 10000000
Last time write locked in file dict0stats.cc line 1969
2021-06-25 20:09:53 0 [Note] InnoDB: A semaphore wait:
--Thread 140303167837952 has waited at btr0cur.cc line 1480 for 241.00 seconds the semaphore:
SX-lock on RW-latch at 0x7f9aa421be90 created in file dict0dict.cc line 2161
a writer (thread id 140303193323264) has reserved it in mode  SX
number of readers 0, waiters flag 1, lock_word: 10000000
Last time write locked in file dict0stats.cc line 1969
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending reads 0, writes 0
InnoDB: ###### Diagnostic info printed to the standard error stream
1 Like
21

Yes, I will do a snapshot and a mysqldump first. What command will I run to get the database to migrate? In order to get Nextcloud out of the locked state, I ran the manual upgrade from occ so Nextcloud is already 21.0.2, but running on the 5.5 database.

1 Like
22

we need to investigate why it failed before to going on

this is the resources we gave to innodb, maybe in some cases we need more

/etc/opt/rh/rh-mariadb105/my.cnf.d/mariadb-server.cnf:30:innodb_large_prefix=on
/etc/opt/rh/rh-mariadb105/my.cnf.d/mariadb-server.cnf:31:innodb_file_format=barracuda
/etc/opt/rh/rh-mariadb105/my.cnf.d/mariadb-server.cnf:32:innodb_file_per_table=1
/etc/opt/rh/rh-mariadb105/my.cnf.d/mariadb-server.cnf:36:innodb_buffer_pool_size = 128M
/etc/opt/rh/rh-mariadb105/my.cnf.d/mariadb-server.cnf:37:innodb_buffer_pool_instances = 1
/etc/opt/rh/rh-mariadb105/my.cnf.d/mariadb-server.cnf:38:innodb_flush_log_at_trx_commit = 2
/etc/opt/rh/rh-mariadb105/my.cnf.d/mariadb-server.cnf:39:innodb_log_buffer_size = 32M
/etc/opt/rh/rh-mariadb105/my.cnf.d/mariadb-server.cnf:40:innodb_max_dirty_pages_pct = 90
23

in fact we introduced a lof of steps with tests before to go to the next migration step. We decided to let the nextcloud locked because it forces sysadmin to react and say hey there is something wrong.

24

We shared with the team and you are the only one with this issue, so the settings should be good for you too. I would like to make a second attempt

  • make a mysqldump of mariadb 55 (verify you have something inside and save it elsewhere)
    mysqldump nextcloud > nextcloud.sql
  • delete the mariadb105 nextcloud database
    /opt/rh/rh-mariadb105/root/bin/mysql --socket="/var/run/rh-mariadb105-mariadb/nextcloud-mysql.sock"
    then drop the database
    drop database nextcloud;
  • add more RAM
  • Do a snapshot of the VM
  • then reinstall the rpm
    yum reinstall nethserver-nextcloud
2 Likes
25

Thank you, just to be clear, the drop database nextcloud is on the maria105 database correct?

I am out of town tonight, so I’ll have to do this tomorrow evening (US/Pacific) I’ll get back with the results.

26

Absolutely, destroy the database inside mariadb105, verify it before but it should be empty

Of course keep safe the database of mariadb55

27

I am doing the migration. I’m noticing that the oc_filecache.ibd file is growing and growing in the new database. That file doesn’t exist in the 5.5 database folder. Currently it is 9.5G and growing.

28

If finished overnight, but again didn’t transition to the new database. This is the only entry in /var/log/rh-mariadb105/nextcloud.log

2021-07-05 0:38:06 11 [Warning] Aborted connection 11 to db: ‘nextcloud’ user: ‘root’ host: ‘localhost’ (Got an error writing communication packets)

It was still in maintenance mode, so I turned that off and we are running again on the 55 database. I am wondering if I am running out of hard drive space when it it is backing up and adding data to the new database. I have plenty of space, but not on that partition. I will try expanding that and running it again tonight.

1 Like
29

Yesterday evening, I shut the machine down and booted from Knoppix and rearanged my partitioning and mount points so that the whole /var directory is on my large virtual disk. That gave plenty of room for /var/opt/rh/rh-mariadb105/… to expand. After that, From logs, it looks like the entire process took from 6:45pm to 3:00am to finish, but it did finish, and Nextcloud is now running on the new database and the old one was removed.

Thank you very much for your help.

2 Likes
30

hourra

Thank you very much to give us the chance to help you.

1 Like