Email migration failure

maja2020 · January 3, 2025, 7:42pm

Today I tried to migratie email to my new ns8 cluster. Data copy went fine, finish migration failed. The last part of the log:

<7>systemctl --user -T reload dovecot
Enqueued anchor job 218 dovecot.service/reload.
<7>BEGIN
<7>INSERT INTO mynetworks (network) VALUES ('x.x.x.x')
<4>Skipping open relay policy for IP x.x.x.x: UNIQUE constraint failed: mynetworks.network
<7>INSERT INTO mynetworks (network) VALUES ('x.x.x.x')
<4>Skipping open relay policy for IP x.x.x.x: UNIQUE constraint failed: mynetworks.network
<7>COMMIT
<6>Set bypass map bypass_sender_domain.map: <domain1>, <domain2>, <domain3>, <domain4>,
<6>Set bypass map bypass_ip.map: x.x.x.x, x.x.x.x,
60cbe42d6592b9f6c51f0ae625ee226eadb1c347be5a15800e1616ae96c63dbb
""""removed ‘sogo.sql’
<f..T...... sogo.sql
mkdir: created directory ‘webroot’
Removed symlink /etc/systemd/system/multi-user.target.wants/sogod.service.
dr-xr-xr-x              6 2023/09/28 13:18:46 .
<7>podman-pull-missing ghcr.io/nethserver/rsync:3.4.0
<7>podman run --rm --privileged --network=host --workdir=/srv --env=RSYNCD_NETWORK=10.5.3.0/24 --env=RSYNCD_ADDRESS=cluster-localnode --env=RSYNCD_PORT=20005 --env=RSYNCD_USER=sogo1 --env=RSYNCD_PASSWORD=<passwd> --env=RSYNCD_SYSLOG_TAG=sogo1 --volume=/dev/log:/dev/log --replace --name=rsync-sogo1 --volume=/home/sogo1/.config/state:/srv/state --volume=mysql-data:/srv/volumes/mysql-data --volume=restic-cache:/srv/volumes/restic-cache ghcr.io/nethserver/rsync:3.4.0
mkdir: created directory 'initdb.d'
renamed 'sogo.sql' -> 'initdb.d/sogo.sql'
2025-01-03 18:59:37+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.6+maria~ubu2204 started.
2025-01-03 18:59:37+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2025-01-03 18:59:37+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.6+maria~ubu2204 started.
2025-01-03 18:59:37+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
2025-01-03 18:59:37 0 [Note] Starting MariaDB 10.11.6-MariaDB-1:10.11.6+maria~ubu2204 source revision fecd78b83785d5ae96f2c6ff340375be803cd299 as process 1
2025-01-03 18:59:37 0 [Warning] WSREP: Failed to guess base node address. Set it explicitly via wsrep_node_address.
2025-01-03 18:59:37 0 [Warning] WSREP: Failed to guess base node address. Set it explicitly via wsrep_node_address.
2025-01-03 18:59:37 0 [Warning] WSREP: Guessing address for incoming client connections failed. Try setting wsrep_node_incoming_address explicitly.
2025-01-03 18:59:37 0 [Note] WSREP: Node addr:
2025-01-03 18:59:37 0 [ERROR] mariadbd: Can't lock aria control file '/var/lib/mysql/aria_log_control' for exclusive use, error: 11. Will retry for 30 seconds
2025-01-03 19:00:07 0 [ERROR] mariadbd: Got error 'Could not get an exclusive lock; file is probably in use by another process' when trying to use aria control file '/var/lib/mysql/aria_log_control'
2025-01-03 19:00:07 0 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
2025-01-03 19:00:07 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2025-01-03 19:00:07 0 [Note] InnoDB: Number of transaction pools: 1
2025-01-03 19:00:07 0 [Note] InnoDB: Using SSE4.2 crc32 instructions
2025-01-03 19:00:07 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2025-01-03 19:00:07 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2025-01-03 19:00:07 0 [Note] InnoDB: Completed initialization of buffer pool
2025-01-03 19:00:07 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11
2025-01-03 19:00:07 0 [Note] InnoDB: Check that you do not already have another mariadbd process using the same InnoDB data or log files.
2025-01-03 19:00:07 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2025-01-03 19:00:07 0 [Note] InnoDB: Starting shutdown...
2025-01-03 19:00:07 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2025-01-03 19:00:07 0 [Note] Plugin 'FEEDBACK' is disabled.
2025-01-03 19:00:07 0 [ERROR] Could not open mysql.plugin table: "Unknown storage engine 'Aria'". Some plugins may be not loaded
2025-01-03 19:00:07 0 [ERROR] Failed to initialize plugins.
2025-01-03 19:00:07 0 [ERROR] Aborting
removed 'initdb.d/sogo.sql'
removed 'initdb.d/zz_sogo_restore.sh'
removed directory 'initdb.d/'

The ns7 machine is in a state where

a white screen “Site has moved to ”
dovecot and sogod are dead (systemctl status)

Sogo on ns8 is accessible, email is there, but display malformed.

Can I safely press the button “Abort migration” to restore ns7 mail functions?
How do I proceed?

mrmarkuz · January 3, 2025, 11:48pm

No, AFAIK it will just clean up the migration to be able to start over.

To restore the NS7 applications (removing the white screens & enable the services) you can follow the “Post-migration step back” in the nethserver-ns8-migration — NethServer 7 documentation

Good question, it seems that just the SOGo migration didn’t work. It’s easily possible to restore the NS7 so you can check what happened on the NS8 and start over another time.

Some ideas:

Did you try to reboot the NS8 or to restart SOGo as there’s an older thread about the same error which was solved by reboot: Guacamole unexpected internal error - mysqld stopped - #4 by mrmarkuz

Do you use SOGo just for mail? If this is the case you may just remove the not working SOGo and start over a fresh SOGo install. When mails and user domain are migrated correctly it should just work.
AFAIK SOGo has an export/import tool, maybe it can be used to transfer calendar/contacts to the NS8…

maja2020 · January 4, 2025, 8:28am

Thanks for replying.

Did you try to reboot the NS8 or to restart SOGo

The reboot worked. ns8-sogo is working now, but in an untrusted state.

follow the “Post-migration step back

Worked as well. Two working instances of Sogo, thats more than I expected.

The question is: migration broke. Since the last step is disabeling services on ns7, what specific action failed on ns8, do I have to repair that or can I just activate the ns8 install for users after syncing email?

Second option is to remove ns8-Sogo and/or ns8-email and migrate again.
Any insights for the next step?

mrmarkuz · January 4, 2025, 8:58am

As it seems that just the SOGo migration had some issues, I’d try to finish the migration to finally migrate mail and the account provider to the NS8.
If NS8 is working after migration, you’re ready.

If it doesn’t work as expected, there’s always the second option to reactivate all services on the NS7 and start over with a new migration.

maja2020 · January 4, 2025, 2:19pm

That failed, Checked ping to 10.5.3.1 (ns8 wireguard tunnel which works).
Seems the rsync target dows not repond, so the data sync does not finish.

rsync: failed to connect to 10.5.3.1 (10.5.3.1): Connection refused (111)

On the ns8 machine in the sogo1 module I see a running rsync-sogo1 container. (runagent -m sogo1, podman ps). The ns7 is working now.

I will remove the ns8 email and sogo modules, re-install ns7 migration and try again.

mrmarkuz · January 4, 2025, 2:56pm

Maybe it helps to remove the rsync container using

podman rm ...

and retry migration…

maja2020 · January 5, 2025, 4:45pm

This time the migration completed successfully. Thanks for the help.

stephdl · January 6, 2025, 8:03am

maja2020:

2025-01-03 18:59:37 0 [ERROR] mariadbd: Can't lock aria control file '/var/lib/mysql/aria_log_control' for exclusive use, error: 11. Will retry for 30 seconds
2025-01-03 19:00:07 0 [ERROR] mariadbd: Got error 'Could not get an exclusive lock; file is probably in use by another process' when trying to use aria control file '/var/lib/mysql/aria_log_control'

it seems the container itself failed to initialize, the database migration did not start