HELP! Cannot finish migration and NS7 mail is unavailable now!

So I just wanted to migrate my mail (with smart host) + webtop to NS8.
I connected from my NS7 VM to my NS8 clean instance.
Detected the services that I can migrate.
I chose to only migrate the above.
I clicked to start migration. OK.
I clicked to sync. OK.
I re-clicked to sync. OK but took way more time (!?)…
I clicked to finalize migration…

BOOM.

I got this that it failed and could not detect the error.
Gave me the following command to paste to console and got this:
I’ll recover my VM from backup now.

 echo '{"app":"nethserver-mail","action":"finish","migrationConfig":{"roundCubeVirtualHost":""}}' | /usr/bin/setsid /usr/bin/sudo /usr/libexec/nethserver/api/nethserver-ns8-migration/migration/update | jq
{
  "progress": "0.00",
  "time": "0.0",
  "exit": 0,
  "event": "migration-sync",
  "state": "running",
  "step": 0,
  "pid": 0,
  "action": ""
}
rsync: failed to connect to 10.5.4.1 (10.5.4.1): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(126) [sender=3.1.2]
rsync: failed to connect to 10.5.4.1 (10.5.4.1): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(126) [sender=3.1.2]
rsync: failed to connect to 10.5.4.1 (10.5.4.1): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(126) [sender=3.1.2]
rsync: failed to connect to 10.5.4.1 (10.5.4.1): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(126) [sender=3.1.2]
rsync: failed to connect to 10.5.4.1 (10.5.4.1): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(126) [sender=3.1.2]
rsync: failed to connect to 10.5.4.1 (10.5.4.1): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(126) [sender=3.1.2]
rsync: failed to connect to 10.5.4.1 (10.5.4.1): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(126) [sender=3.1.2]
rsync: failed to connect to 10.5.4.1 (10.5.4.1): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(126) [sender=3.1.2]
rsync: failed to connect to 10.5.4.1 (10.5.4.1): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(126) [sender=3.1.2]
rsync: failed to connect to 10.5.4.1 (10.5.4.1): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(126) [sender=3.1.2]
rsync: failed to connect to 10.5.4.1 (10.5.4.1): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(126) [sende

Also my NS8 is reporting this:

module/mail1/import-module

Task module/mail1/import-module failed validation
Copy to clipboard
Copy task trace

module/mail1/import-module
Task ID: Copy to clipboard
5608c325-759f-49b5-bcea-85a279ca851b

…I’ll recover my VM from backup now.

1 Like

This looks bad :frowning:

Wise decision

Well that does look like a bug @davidep as I had roundcube migration DISABLED.

1 Like

If you have the NS8 still running go to Audit trail page, paste it in the Audit info field. Filter by create-task Action:

image

Then search: you’d obtain the task submission data, like:

Please, copy and paste its output here!

1 Like

No everything is reset now.
I will do that though in case it happens again.
Thanks.

OK it failed again.
NS7 working fine (from BU), NS8 clean.
Migration module correctly detected NS8 fast and reported correctly the modules that could be migrated.
I disabled mattermost (and one other experimental) and only left mail that also includes imap connectors and webtop.
The initial “start migration” was rather fast.
The one time I pressed “sync”, took like 10 minutes. I thought was stuck as I never got a progress (stayed 0%). But eventually it returned me to the migration page.
So I pressed “finish”… Which started a new sync (!?)…
And this went BOOM after a few minutes:

Error
Synchronization failed

The following command has failed:
nethserver-ns8-migration/migration/update
Unfortunately we couldn't catch the exact error. If you want to help, please click on the button below to copy the failed command to the clipboard, paste it into the Terminal and submit command output to the developers.

The command is:

 echo '{"app":"nethserver-mail","action":"finish","migrationConfig":{"roundCubeVirtualHost":""}}' | /usr/bin/setsid /usr/bin/sudo /usr/libexec/nethserver/api/nethserver-ns8-migration/migration/update | jq

At this point, cockpit works fine in NS7 but mail sever is dead (or webtop …oh wait postfix is disabled, so mail is fully dead)…

In my NS8 I do not see any fails (at least after I pressed the “finish”), so I cannot do what you asked.

Last notifications (that are after I pressed finish) are “cluster/list-user-domains” twice, completed.

I will restore my NS7 from backup once more. :frowning: (I will keep the “broken” VM in case we need to get something from it)
I will keep NS8 as is for now, pause the VM, in case there is a log that can help us.

Things don’t look too bright to be honest.

Thank you for your time and for not giving up!

Mail services were stopped and disabled by the Finish step of the migration procedure.

The contents of /var/log/ns8-migration.log are visible from the migration tool UI: did you see it?

What version of Roundcube is installed?

You can save a copy of /var/log/messages, or dump the system journal to a file (e.g. run journalctl > /var/log/journal.out).

Also a copy of /var/lib/nethserver/api-server/audit.db may be useful.


If you can privately share the above files from ns7 and ns8, we can try to understand what’s going wrong.

I cannot give up, I need to eventually migrate.
I will get back to you with files.

I don’t remember anything about Roundcube as it was an experimental build and never used, this is why I didn’t choose to migrate.
Which is already a bug, as something that you don’t choose to migrate (or touch) shouldn’t bother the migration whatsoever.

In the end I might just remove/delete unneeded services from NS7 before I migrate - but we better make the migration process more resilient first, right?

1 Like

Yes right, however you’d help me if you check your Roundcube version and release number. In ns7 we have two different Roundcube versions:

  • nethserver-roundcubemail-next
  • nethserver-roundcubemail

Hum the mysql database are not the same IIRC

I think something might be wrong here as this is the list of apps that shows in my (restored) NS7:

…and here is a list of installed packages.
…erm… cannot attach text file…
but there is no roundcube in it…
…so…

(about the files you requested, I am still remotely and cannot bring the broken images online - will happen in the weekend)

The migration-tool checks the existence of this file to detect if Roundcube is present or not:

/etc/e-smith/db/configuration/defaults/roundcubemail/type

To find the RPM owning the file, run

rpm -qf /etc/e-smith/db/configuration/defaults/roundcubemail/type

There is no roundcubemail folder at all (I am not using roundcube, I use webtop).

I was probably on the wrong track with the Roundcube message…

What’s the hardware setup of the NS8? Can you exclude it is running out of resources?

It should not be running out of resources.
It is a KVM based VM (running on UNRAID).
The machine is a 6 core/12 thread CPU, 32GB RAM and the VM itself is on M.2.
The VM configuration is 4 threads and 2GB RAM for NS8 and 4 threads 4GB RAM for NS7.

hello mate, something new to test, we do not want to migrate nethserver-roundcubemail-next (from Nethforge, no worry, manual dump can been done) so we have changed the way how to detect nethserver-roundcubemail from nethserver-roundcubemail-next

The test is quite simple and is written here: Roundcubemail migration From NS7 to NS8 · Issue #6851 · NethServer/dev · GitHub

If you could help by testing and report to the bug report, thank in advance

1 Like

OK will do in the next couple of days probably.

So I need to install roundcube even though I don’t use it?
(I use Webtop)

1 Like

@stephdl can you please tell me… Do I still need to manually install that update? Because there was an official update yesterday and migration is already version 1.07.

Also again: I don’t use roundcube, I use webtop… will this also help me?

Should I just retry migration from scratch? Anything else? Please let me know how to proceed.

The rpm has not been released you have to install it manually with yum and the url

The upgrade concern only roundcubemail because we have defined a better target to know if roundcubemail is installed on your system

In short we do not want to migrate roundcubemail next if it is installed

Maybe that was the root cause of your issue even saying that we probably need the log on the ns8 to debug what is occuring

journalctl -f

Migration is a complex thing because the error could be locally on NS7 or remotely on NS8

clear - I will test tonight

1 Like