NS7 to NS8 Migration issue

jrieder · May 5, 2025, 8:20pm

NethServer Version: 7.9.2009(final) migration tool V1.2.1
NethServer Version: 8.4 ( Pre-built image)
Module: Nextcloud

After long hesitation I started the migration topic:
Setup a Testsystem with current NS7 server snapshot and fresh NS8 (from pre-built image).
NS7 Modules for migration: AD Account Provider, Nextcloud, Email incl. WebTop/connectors,
Everything as per migration instruction:

installed NS7 migration module
connected NS7 to NS8 (without TLS validation)
Start migration of Nextcloud

After a while Nextcloud Migration failed:

On NS7: nethserver-ns8-migration/migration/update

ns8-migration.log

=========== Join cluster Sun, 04 May 2025 19:51:52 +0200
Joined to cluster leader ns8.domain.org
----------- start nethserver-nextcloud Sun, 04 May 2025 19:53:12 +0200
Traceback (most recent call last):
  File "/usr/sbin/ns8-action", line 126, in <module>
    ret = wait_task(api_endpoint, payload['token'], task_id)
  File "/usr/sbin/ns8-action", line 62, in wait_task
    raise Exception("No server response")
Exception: No server response

On NS8:
Cluster/import-module failed

Logs:
task history:

=========== Join cluster Sun, 04 May 2025 19:51:52 +0200
Joined to cluster leader http://ns8.domain.org
----------- start nethserver-nextcloud Sun, 04 May 2025 19:53:12 +0200
Traceback (most recent call last):
  File "/usr/sbin/ns8-action", line 126, in <module>
    ret = wait_task(api_endpoint, payload['token'], task_id)
  File "/usr/sbin/ns8-action", line 62, in wait_task
    raise Exception("No server response")
Exception: No server response

cluster/import-module

Assertion failed
  File "/var/lib/nethserver/cluster/actions/import-module/50import", line 50, in <module>
    agent.assert_exp(add_module_result['exit_code'] == 0) # add-module is successful

cluster/add-module

<7>podman-pull-missing ghcr.io/nethserver/nextcloud:1.1.9
Trying to pull ghcr.io/nethserver/nextcloud:1.1.9...
Getting image source signatures
Copying blob sha256:a1cf445727b00d0d609ed6055054899538fec7ef42ed7cd510a20e4e5a6951cd
Copying config sha256:e74fecb8db3d6721330629715fe851111dd6dca53f79242d58148ac013717968
Writing manifest to image destination
e74fecb8db3d6721330629715fe851111dd6dca53f79242d58148ac013717968
<7>extract-ui ghcr.io/nethserver/nextcloud:1.1.9
Extracting container filesystem ui to /var/lib/nethserver/cluster/ui/apps/nextcloud1
ui/css/
ui/css/about~31ecd969.276957f6.css
ui/css/app~748942c6.35210893.css
ui/i18n/
ui/i18n/ar/
ui/i18n/ar/translation.json
ui/i18n/de/
ui/i18n/de/translation.json
ui/i18n/en/
ui/i18n/en/translation.json
ui/i18n/es/
ui/i18n/es/translation.json
ui/i18n/eu/
ui/i18n/eu/translation.json
ui/i18n/it/
ui/i18n/it/translation.json
ui/i18n/pt/
ui/i18n/pt/translation.json
ui/i18n/pt_BR/
ui/i18n/pt_BR/translation.json
ui/img/
ui/img/module_default_logo.bd323802.png
ui/index.html
ui/js/
ui/js/about~31ecd969.6a75658c.js
ui/js/about~31ecd969.6a75658c.js.map
ui/js/app~748942c6.862b1ea4.js
ui/js/app~748942c6.862b1ea4.js.map
ui/js/chunk-vendors~02576867.44c88c8d.js
ui/js/chunk-vendors~02576867.44c88c8d.js.map
ui/js/chunk-vendors~0605657e.7f254d6a.js
ui/js/chunk-vendors~0605657e.7f254d6a.js.map
ui/js/chunk-vendors~0f485567.52974094.js
ui/js/chunk-vendors~0f485567.52974094.js.map
ui/js/chunk-vendors~17faf02d.7f68aab6.js
ui/js/chunk-vendors~17faf02d.7f68aab6.js.map
ui/js/chunk-vendors~1d97ff09.290bda9d.js
ui/js/chunk-vendors~1d97ff09.290bda9d.js.map
ui/js/chunk-vendors~2a42e354.0df57e14.js
ui/js/chunk-vendors~2a42e354.0df57e14.js.map
ui/js/chunk-vendors~2aa62147.38204dba.js
ui/js/chunk-vendors~2aa62147.38204dba.js.map
ui/js/chunk-vendors~41d44f25.e7779fa1.js
ui/js/chunk-vendors~41d44f25.e7779fa1.js.map
ui/js/chunk-vendors~46852254.0b6d19c2.js
ui/js/chunk-vendors~46852254.0b6d19c2.js.map
ui/js/chunk-vendors~57473a66.f586d2ed.js
ui/js/chunk-vendors~57473a66.f586d2ed.js.map
ui/js/chunk-vendors~5bb1f863.3efba861.js
ui/js/chunk-vendors~5bb1f863.3efba861.js.map
ui/js/chunk-vendors~5eba3806.b8428291.js
ui/js/chunk-vendors~5eba3806.b8428291.js.map
ui/js/chunk-vendors~690b702c.fd888ae3.js
ui/js/chunk-vendors~690b702c.fd888ae3.js.map
ui/js/chunk-vendors~7274e1de.aaf08c8f.js
ui/js/chunk-vendors~7274e1de.aaf08c8f.js.map
ui/js/chunk-vendors~86f6b1bc.434ca0c9.js
ui/js/chunk-vendors~86f6b1bc.434ca0c9.js.map
ui/js/chunk-vendors~b5906859.deff14e4.js
ui/js/chunk-vendors~b5906859.deff14e4.js.map
ui/js/chunk-vendors~bc21d4b3.68b10b46.js
ui/js/chunk-vendors~bc21d4b3.68b10b46.js.map
ui/js/chunk-vendors~c8728516.dd1ccc58.js
ui/js/chunk-vendors~c8728516.dd1ccc58.js.map
ui/js/chunk-vendors~d2305125.6a99a4ff.js
ui/js/chunk-vendors~d2305125.6a99a4ff.js.map
ui/js/chunk-vendors~d9886323.1a1a66f6.js
ui/js/chunk-vendors~d9886323.1a1a66f6.js.map
ui/js/chunk-vendors~db300d2f.ddb8641c.js
ui/js/chunk-vendors~db300d2f.ddb8641c.js.map
ui/js/chunk-vendors~ec8c427e.5c1734c3.js
ui/js/chunk-vendors~ec8c427e.5c1734c3.js.map
ui/js/chunk-vendors~fdc6512a.81c5c86c.js
ui/js/chunk-vendors~fdc6512a.81c5c86c.js.map
ui/js/lang-ar-translation-json~3bbe8b71.e45170f6.js
ui/js/lang-ar-translation-json~3bbe8b71.e45170f6.js.map
ui/js/lang-de-translation-json~3c620948.f7547109.js
ui/js/lang-de-translation-json~3c620948.f7547109.js.map
ui/js/lang-en-translation-json~9b60384d.93bbef0a.js
ui/js/lang-en-translation-json~9b60384d.93bbef0a.js.map
ui/js/lang-es-translation-json~f7c30340.ca8ea994.js
ui/js/lang-es-translation-json~f7c30340.ca8ea994.js.map
ui/js/lang-eu-translation-json~8ed4c5c2.fb33634e.js
ui/js/lang-eu-translation-json~8ed4c5c2.fb33634e.js.map
ui/js/lang-it-translation-json~e043826f.f5eae8e5.js
ui/js/lang-it-translation-json~e043826f.f5eae8e5.js.map
ui/js/lang-pt-translation-json~45d767f3.1029da9e.js
ui/js/lang-pt-translation-json~45d767f3.1029da9e.js.map
ui/js/lang-pt_BR-translation-json~eef7148a.4de76209.js
ui/js/lang-pt_BR-translation-json~eef7148a.4de76209.js.map
ui/metadata.json
ui/shortcuts.json
755b73fdb7ec5a44216cfa0efd91db3d082687a3c293b015a3d1a27a419cb82b
Assertion failed
  File "/var/lib/nethserver/cluster/actions/add-module/50update", line 196, in <module>
    agent.assert_exp(create_module_result['exit_code'] == 0) # Ensure create-module is successful

cluster/create-module

Add to module/nextcloud1 environment REDIS_IMAGE=docker.io/redis:6.2.12-alpine
Add to module/nextcloud1 environment MARIADB_IMAGE=docker.io/mariadb:10.6.17
Add to module/nextcloud1 environment NGINX_IMAGE=docker.io/nginx:1.25.4-alpine
Add to module/nextcloud1 environment NEXTCLOUD_APP_IMAGE=ghcr.io/nethserver/nextcloud-app:1.1.9
<7>podman-pull-missing docker.io/redis:6.2.12-alpine docker.io/mariadb:10.6.17 docker.io/nginx:1.25.4-alpine ghcr.io/nethserver/nextcloud-app:1.1.9
Trying to pull docker.io/library/redis:6.2.12-alpine...
Getting image source signatures
Copying blob sha256:f18575e3694b6c5ca6860ac158b3015703562add69145fcc26dc6957c893a2b6
Copying blob sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3
Copying blob sha256:7aaf69037d81f66eec59c735ab5ab7e0a7bf9a2206560da4a7f34fb381758417
Copying blob sha256:029a81f05585f767fb7549af85a8f24479149e2a73710427a8775593fbe86159
Copying blob sha256:3b439d5c2f2545ae374ccbe318c8cd118537b041fc7eb52ecca6c4e147196e4d
Copying blob sha256:4a828495d25374d7219cc75933581a5c5bc43aaa09b549e81ce838467220a46e
Copying config sha256:b9cad9a5aff99063a48d14d26e9f31dcf1768b8e65c4687a8e3bc995c02749f3
Writing manifest to image destination
b9cad9a5aff99063a48d14d26e9f31dcf1768b8e65c4687a8e3bc995c02749f3
Trying to pull docker.io/library/mariadb:10.6.17...
Getting image source signatures
Copying blob sha256:d4c3c94e5e10ed15503bda7e145a3652ee935c0b2e9de9b5c98df7ec0a0cd925
Copying blob sha256:6467e8cb577a42e63091e8559bf107b4b06aac58496bad4cf46be17ed954c7de
Copying blob sha256:1447c5b36c4a3d4ba1909fd428116b61d48b86083c2cf46f3fa5bf82a2990799
Copying blob sha256:2812e67c5d20d24ae757aba63457f3a88edd7201c2a4da39eed13e84d581a048
Copying blob sha256:ab0bad438d182542ba2a34f828e2a1b8be312934d8c16551c01e3ca0addaefde
Copying blob sha256:b54fa3f565ecc150054e244823f992ab76f485c66ad492d7f1db1136dd3b66a4
Copying blob sha256:b91e4ecf04532260eaafe540fd133a7eefc86cab6cf074701b162b12b69b52f8
Copying blob sha256:f15e4e0bbeb42dec5cf88b160d47c4f263a4516fada4c06e0e123375035c45b6
Copying config sha256:d7a7b84914e9e8f30bd2d3a4114a65dcc076982658d77dce2471f7a8680556ab
Writing manifest to image destination
d7a7b84914e9e8f30bd2d3a4114a65dcc076982658d77dce2471f7a8680556ab
Trying to pull docker.io/library/nginx:1.25.4-alpine...
Getting image source signatures
Copying blob sha256:ed3e62e73b33c9cfa4b253060771e4a9eebb751ab438052f197e847b4553a9ac
Copying blob sha256:6eba808ac059320c42179a6590b021f8695d3f12c2afa8745e219f635acf19d4
Copying blob sha256:1d0dd2dc2265a581798226f7c79d134ac797f42db3f934dd4af1d38a6b89ce5c
Copying blob sha256:5126dce06df729f9a22956013e160f8b581d47095beec332d647a5c1119b2411
Copying blob sha256:619be1103602d98e1963557998c954c892b3872986c27365e9f651f5bc27cab8
Copying blob sha256:2b1ab92f023179da00446365a60daa60d72a1edeb697fb81811e086eba2e0170
Copying blob sha256:57038e85fbb88e96e34a84b125e568f540437561adb363fa791ff9e94e153dc1
Copying blob sha256:eec94c9845c062c9f3495ce861d2bd9507bd7a13710de6b89a195a6b089ea8a7
Copying config sha256:e289a478ace02cd72f0a71a5b2ec0594495e1fae85faa10aae3b0da530812608
Writing manifest to image destination
e289a478ace02cd72f0a71a5b2ec0594495e1fae85faa10aae3b0da530812608
Trying to pull ghcr.io/nethserver/nextcloud-app:1.1.9...
Getting image source signatures
Copying blob sha256:3cacdd319395c1cec327d721a2a97626ea9472d7984f1cb91d5be6cd377f647b
Copying blob sha256:41011b1fcd377e2454d430d54d3bbd4120076da6d6271da66ebda3ab97caffdb
Copying blob sha256:9bee848f9a490d3f1d6ac71a1b40f10168aa79ea3fc3cf58f9a08a6c3d0d8999
Copying blob sha256:fb310659793204646b49d0f71b75ac32ce6d9694aca4c52d22bb3a140c56ab0f
Copying blob sha256:6e5015e024232cf8074347bf2e0a0de6fd4390d58e0459e6150012acd0507ba6
Copying blob sha256:a9f5360b3bc6946ac1718623cbe9795852f3fead51a248a7680a061aba9912b1
Copying blob sha256:a61b7aa022d0670a3d42da68fbe8ab690b46427422156d0307446cd86fc750ba
Copying blob sha256:ec162902f37b70ae49a265bd30450773f089f88f88afb122699fcc8b8e2cd242
Copying blob sha256:471280db953474aa3ab470152861a34b883fe48ef8e6534110f31753192e3ec9
Copying blob sha256:5560943dcabd917a8c0bafe2a4ef0f3b1b613398a7984db8220bd2f5b67715fb
Copying blob sha256:671e5d3ee90f374af287c116b2b087db2f223b581b35960df04f47288795d6fc
Copying blob sha256:80cdcc6e22e01261c01e8bb061419560be1f053b1fdf1a35fa96641ce68f3bda
Copying blob sha256:83db1d6b83a20dfd7ba79f2544c14540f94fa3d822d6ad979201d6465f0f9973
Copying blob sha256:7a36440268a471e1f797986f80666e0db8114586d6c682a19c7c6679f5228f58
Copying blob sha256:ba9e9d0a39b5bbd6e5a58c901165d4a7e53f6496526c2289bcf5a52d97ffbe3b
Copying blob sha256:b234077e5e1941ab219d8a0a9091bb43541cb70211dd69ce4431d5189d4c87ad
Copying blob sha256:c5257e2cff0a4a8201b3efe45c5e912fdda40a8577c3ad076652657f9d6b19d7
Error: writing blob: storing blob to file "/var/tmp/container_images_storage4193935430/12": happened during read: unexpected EOF (while reconnecting: Get "https://ghcr.io/token?scope=repository%3Anethserver%2Fnextcloud-app%3Apull&service=ghcr.io": dial tcp: lookup ghcr.io on [fd10:1:1:1::1]:53: read udp [fd10:1:1:1::6]:57910->[fd10:1:1:1::1]:53: i/o timeout)
Traceback (most recent call last):
  File "/usr/local/agent/bin/podman-pull-missing", line 35, in <module>
    subprocess.run(['podman', 'pull', image_url]).check_returncode()
  File "/usr/lib64/python3.11/subprocess.py", line 502, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['podman', 'pull', 'ghcr.io/nethserver/nextcloud-app:1.1.9']' returned non-zero exit status 125.
Traceback (most recent call last):
  File "/usr/local/agent/actions/create-module/05pullimages", line 48, in <module>
    agent.run_helper('podman-pull-missing', *images).check_returncode()
  File "/usr/lib64/python3.11/subprocess.py", line 502, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '('podman-pull-missing', 'docker.io/redis:6.2.12-alpine', 'docker.io/mariadb:10.6.17', 'docker.io/nginx:1.25.4-alpine', 'ghcr.io/nethserver/nextcloud-app:1.1.9')' returned non-zero exit status 1.

Nextcloud seems partly installed - instance Nextcloud1 is shown - but details cannot be shown

Screenshot 2025-05-05 204408_ns8_installed_modules_nextcloud_instances399×446 18.2 KB

I started with a backup of the snapshots (both NS7 and NS8) once again - same failure on ns7 and ns8.

It seems an issue during downloading or storing, but I don’t know how to solve this and what is the real issue behind?
What is unclear to me is: the following points to failed IPv6 DS query to [fd10:1:1:1::1]:53 to the next-hop router (where DNSv6 is not setup). But why is suddenly an IPv6 address used here?

(while reconnecting: Get "https://ghcr.io/token?scope=repository%3Anethserver%2Fnextcloud-app%3Apull&service=ghcr.io": dial tcp: lookup ghcr.io on [fd10:1:1:1::1]:53: read udp [fd10:1:1:1::6]:57910->[fd10:1:1:1::1]:53: i/o timeout)

The entire test-network is setup for IPv4, IPv6 is only partly setup (as per default) but not really tested (although both NS7 ans NS8 servers have IPv6 addresses, also the next-hop router). But the migration starts, internet connection available, significant parts of module SW is downloaded already as seen in the logs - so download source must have been found before as well…?

A DNS query even using IPv6 internal address of test router is ok - answering with IPv4 address though:

[root@xxxx log]# dig @fd10:1:1:1::1 ghcr.io

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.16 <<>> @fd10:1:1:1::1 ghcr.io
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41729
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;ghcr.io.                       IN      A

;; ANSWER SECTION:
ghcr.io.                55      IN      A       140.82.121.33

;; Query time: 11 msec
;; SERVER: fd10:1:1:1::1#53(fd10:1:1:1::1)
;; WHEN: Mon May 05 22:01:01 CEST 2025
;; MSG SIZE  rcvd: 41

Is there any specific IPv6 usage on NS8 prebuild image?

Any hint is appreciated.
Many thanks!

mrmarkuz · May 6, 2025, 6:42am

There’s an updated migration tool 1.3.0 where this bug should be fixed:

See also Updates for NethServer 7 - #489 by nethbot and Task progress lost and timeout during migration · Issue #7319 · NethServer/dev · GitHub

jrieder · May 6, 2025, 7:08pm

Many thanks for this hint, received the download message of the new migration tool version also today! I installed the updated migration tool. Also installed at NS8 updated core version.
Restarted the entire migration. From GUI perspective the nextcloud migration was successful then.
In the ns8-migration.log however some error appears:

=========== Join cluster Tue, 06 May 2025 11:00:05 +0200
Joined to cluster leader xxx.org
----------- start nethserver-nextcloud Tue, 06 May 2025 11:01:10 +0200
[INFO] Created remote module instance nextcloud2
[INFO] App nethserver-nextcloud is bound to rsync://nextcloud2@10.5.4.1:20004, waiting for task module/nextcloud2/task/c
184ed6e-41dc-480a-803d-8ea51e5436cd
----------- sync nethserver-nextcloud Tue, 06 May 2025 12:23:21 +0200
----------- finish nethserver-nextcloud Tue, 06 May 2025 13:21:56 +0200
dr-xr-xr-x              6 2025/02/14 00:04:11 .
<7>podman-pull-missing ghcr.io/nethserver/rsync:3.7.0
Trying to pull ghcr.io/nethserver/rsync:3.7.0...
Getting image source signatures
Copying blob sha256:f269998746e3f861e7cda498a0bafff2391a265e1e9c3bd7f66e65879a5ba2f1
Copying blob sha256:b0f6f1c319a1570f67352d490370f0aeb5c0e67a087baf2d5f301ad51ec18858
Copying config sha256:e9bc75def280a1ce726652db04819de84dd809200e2b16ee051f1497d6d0b2a4
Writing manifest to image destination
e9bc75def280a1ce726652db04819de84dd809200e2b16ee051f1497d6d0b2a4
<7>podman volume create nextcloud-app-data
nextcloud-app-data
<7>podman run --rm --privileged --network=host --workdir=/srv --env=RSYNCD_NETWORK=10.5.4.0/24 --env=RSYNCD_ADDRESS=clus
ter-localnode --env=RSYNCD_PORT=20004 --env=RSYNCD_USER=nextcloud2 --env=RSYNCD_PASSWORD=2824863087852-86a3-4ec3-9904-f6
9654940307 --env=RSYNCD_SYSLOG_TAG=nextcloud2 --volume=/dev/log:/dev/log --replace --name=rsync-nextcloud2 --volume=/hom
e/nextcloud2/.config/state:/srv/state --volume=nextcloud-app-data:/srv/volumes/nextcloud-app-data ghcr.io/nethserver/rsy
nc:3.7.0
""_acontrol_task request attempt failed (WS reached EOF while waiting for module/traefik1/task/20f14a78-c028-4a42-a3a1-4
db7f9039155). Retrying...
_acontrol_task request recovered successfully at attempt 2
2025-05-06 11:22:19+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.6.17+maria~ubu2004 started.
2025-05-06 11:22:20+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2025-05-06 11:22:20+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.6.17+maria~ubu2004 started.
2025-05-06 11:22:21+00:00 [Note] [Entrypoint]: Initializing database files

PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
...... skipped .........

2025-05-06 11:22:22 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2025-05-06 11:22:22 0 [Note] InnoDB: 10.6.17 started; log sequence number 41354; transaction id 14
2025-05-06 11:22:22 0 [Note] Plugin 'FEEDBACK' is disabled.
2025-05-06 11:22:22 0 [Warning] 'user' entry 'root@4a9b3d7339f8' ignored in --skip-name-resolve mode.
2025-05-06 11:22:22 0 [Warning] 'proxies_priv' entry '@% root@4a9b3d7339f8' ignored in --skip-name-resolve mode.
2025-05-06 11:22:22 0 [Note] mariadbd: ready for connections.
Version: '10.6.17-MariaDB-1:10.6.17+maria~ubu2004'  socket: '/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distr
ibution
2025-05-06 11:22:23+00:00 [Note] [Entrypoint]: Temporary server started.
2025-05-06 11:22:27+00:00 [Note] [Entrypoint]: Creating database nextcloud
2025-05-06 11:22:27+00:00 [Note] [Entrypoint]: Creating user nextcloud
2025-05-06 11:22:27+00:00 [Note] [Entrypoint]: Giving user nextcloud access to schema nextcloud
2025-05-06 11:22:27+00:00 [Note] [Entrypoint]: Securing system users (equivalent to running mysql_secure_installation)

2025-05-06 11:22:27+00:00 [Note] [Entrypoint]: /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/
dump.sql


2025-05-06 11:22:45+00:00 [Note] [Entrypoint]: /usr/local/bin/docker-entrypoint.sh: sourcing /docker-entrypoint-initdb.d
/zz_stop.sh
2025-05-06 11:22:45 0 [Note] mariadbd (initiated by: unknown): Normal shutdown
2025-05-06 11:22:45 0 [Note] InnoDB: FTS optimize thread exiting.
2025-05-06 11:22:45 0 [Note] InnoDB: Starting shutdown...
2025-05-06 11:22:45 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2025-05-06 11:22:45 0 [Note] InnoDB: Restricted to 2028 pages due to innodb_buf_pool_dump_pct=25
2025-05-06 11:22:45 0 [Note] InnoDB: Buffer pool(s) dump completed at 250506 11:22:45
2025-05-06 11:22:46 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2025-05-06 11:22:46 0 [Note] InnoDB: Shutdown completed; log sequence number 197744345; transaction id 675
2025-05-06 11:22:46 0 [Note] mariadbd: Shutdown complete

parse error: Invalid numeric literal at line 2, column 8
parse error: Invalid numeric literal at line 2, column 8
parse error: Invalid numeric literal at line 2, column 8
parse error: Invalid numeric literal at line 2, column 8
{"domain":"ad.home.lan","host":"nextcloud.wf1.dynip.online","http2https":true,"lets_encrypt":false}Created symlink /home
/nextcloud2/.config/systemd/user/default.target.wants/nextcloud.service → /home/nextcloud2/.config/systemd/user/nextclou
d.service.
Created symlink /home/nextcloud2/.config/systemd/user/default.target.wants/nextcloud-redis.service → /home/nextcloud2/.c
onfig/systemd/user/nextcloud-redis.service.
Created symlink /home/nextcloud2/.config/systemd/user/default.target.wants/nextcloud-db.service → /home/nextcloud2/.conf
ig/systemd/user/nextcloud-db.service.
Created symlink /home/nextcloud2/.config/systemd/user/default.target.wants/nextcloud-app.service → /home/nextcloud2/.con
fig/systemd/user/nextcloud-app.service.
Created symlink /home/nextcloud2/.config/systemd/user/default.target.wants/nextcloud-nginx.service → /home/nextcloud2/.c
onfig/systemd/user/nextcloud-nginx.service.

first warnings:

2025-05-06 11:22:22 0 [Warning] 'user' entry 'root@4a9b3d7339f8' ignored in --skip-name-resolve mode.
2025-05-06 11:22:22 0 [Warning] 'proxies_priv' entry '@% root@4a9b3d7339f8' ignored in --skip-name-resolve mode.

then some error:

parse error: Invalid numeric literal at line 2, column 8
parse error: Invalid numeric literal at line 2, column 8
parse error: Invalid numeric literal at line 2, column 8
parse error: Invalid numeric literal at line 2, column 8

Do I have to worry about this?

And I guess I should have removed instance nextcloud1 @ NS8 server before I restarted the migration all over? (in order to avoid having two nextcloud instances running? Guess I need to delete nextcloud1 instance then manually.

Continuing Migration:
After nextcloud I started migration of mail, unfortunately again an error in the 2nd step “sync data”

Error appears almost immediate, each time I click the “sync data” step. Cannot reach the copy command button, hidded under the Event pop-up window.

ns8-migration.log:

""----------- start nethserver-mail Tue, 06 May 2025 13:41:30 +0200
mkdir: created directory ‘/var/lib/nethserver/nethserver-ns8-migration/nethserver-mail-getmail’
mkdir: created directory ‘/var/lib/nethserver/nethserver-ns8-migration/nethserver-webtop5’
Assertion failed
  File "/var/lib/nethserver/cluster/actions/import-module/50import", line 50, in <module>
    agent.assert_exp(add_module_result['exit_code'] == 0) # add-module is successful
[INFO] Created remote module instance imapsync1
[INFO] App nethserver-mail-getmail is bound to rsync://imapsync1@10.5.4.1:20006, waiting for task module/imapsync1/task/
539a5544-1f0b-4421-b07a-5330d3bcc233
[INFO] Created remote module instance mail1
[INFO] App nethserver-mail is bound to rsync://mail1@10.5.4.1:20007, waiting for task module/mail1/task/ef1c629a-10c9-46
06-ba4d-e47beb362b2d
----------- sync nethserver-mail Tue, 06 May 2025 17:39:56 +0200
----------- sync nethserver-mail Tue, 06 May 2025 17:40:10 +0200

On NS8 server: Applications imapsync, mail and webtop appear available. mail seems unconfigured (requesting to fill in mailserver hostname, primary email domain, ) when I move to mail app settings - which I dit not pursue further so far:

log for mail application @ NS8:

skipped some parts
2025-05-06T13:42:12+02:00 [1:mail1:systemd] Reached target Main User Target.
2025-05-06T13:42:12+02:00 [1:mail1:systemd] Startup finished in 287ms.
2025-05-06T13:42:13+02:00 [1:mail1:agent@mail1] task/module/mail1/e9a78dbc-745d-46ce-9f67-d359950f2bcb: create-module/05pullimages is starting
2025-05-06T13:42:13+02:00 [1:mail1:agent@mail1] Add to module/mail1 environment MAIL_DOVECOT_IMAGE=ghcr.io/nethserver/mail-dovecot:1.6.3
2025-05-06T13:42:13+02:00 [1:mail1:agent@mail1] Add to module/mail1 environment MAIL_POSTFIX_IMAGE=ghcr.io/nethserver/mail-postfix:1.6.3
2025-05-06T13:42:13+02:00 [1:mail1:agent@mail1] Add to module/mail1 environment MAIL_RSPAMD_IMAGE=ghcr.io/nethserver/mail-rspamd:1.6.3
2025-05-06T13:42:13+02:00 [1:mail1:agent@mail1] Add to module/mail1 environment MAIL_CLAMAV_IMAGE=ghcr.io/nethserver/mail-clamav:1.6.3
2025-05-06T13:42:13+02:00 [1:mail1:agent@mail1] podman-pull-missing ghcr.io/nethserver/mail-dovecot:1.6.3 ghcr.io/nethserver/mail-postfix:1.6.3 ghcr.io/nethserver/mail-rspamd:1.6.3 ghcr.io/nethserver/mail-clamav:1.6.3
2025-05-06T13:42:13+02:00 [1:mail1:systemd] Starting D-Bus User Message Bus...
2025-05-06T13:42:13+02:00 [1:mail1:dbus-broker-launch] Policy to allow eavesdropping in /usr/share/dbus-1/session.conf +31: Eavesdropping is deprecated and ignored
2025-05-06T13:42:13+02:00 [1:mail1:dbus-broker-launch] Policy to allow eavesdropping in /usr/share/dbus-1/session.conf +33: Eavesdropping is deprecated and ignored
2025-05-06T13:42:13+02:00 [1:mail1:systemd] Started D-Bus User Message Bus.
2025-05-06T13:42:13+02:00 [1:mail1:] Ready
2025-05-06T13:42:13+02:00 [1:mail1:systemd] Created slice Slice /user.
2025-05-06T13:42:13+02:00 [1:mail1:systemd] podman-pause-48c22bad.scope: unit configures an IP firewall, but not running as root.
2025-05-06T13:42:13+02:00 [1:mail1:systemd] (This warning is only shown for the first unit using IP firewalling.)
2025-05-06T13:42:13+02:00 [1:mail1:systemd] Started podman-pause-48c22bad.scope.
2025-05-06T13:42:14+02:00 [1:mail1:agent@mail1] Trying to pull ghcr.io/nethserver/mail-dovecot:1.6.3...
2025-05-06T13:42:34+02:00 [1:mail1:agent@mail1] time="2025-05-06T11:42:34Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://ghcr.io/nethserver/mail-dovecot:1.6.3: pinging container registry ghcr.io: Get \"https://ghcr.io/v2/\": dial tcp: lookup ghcr.io on [fd10:1:1:1::1]:53: read udp [fd10:1:1:1::6]:42152->[fd10:1:1:1::1]:53: i/o timeout"
2025-05-06T13:42:36+02:00 [1:mail1:agent@mail1] Getting image source signatures
2025-05-06T13:42:36+02:00 [1:mail1:agent@mail1] Copying blob sha256:b0f6f1c319a1570f67352d490370f0aeb5c0e67a087baf2d5f301ad51ec18858
2025-05-06T13:42:36+02:00 [1:mail1:agent@mail1] Copying blob sha256:d021d69dcf5666a934aa8f3b1b4c5773ccc335d6f14eaa60e3261dee86a6c028
2025-05-06T13:44:29+02:00 [1:mail1:systemd] Starting Mark boot as successful...
2025-05-06T13:44:29+02:00 [1:mail1:systemd] Finished Mark boot as successful.
2025-05-06T13:47:29+02:00 [1:mail1:systemd] Created slice User Background Tasks Slice.
2025-05-06T13:47:29+02:00 [1:mail1:systemd] Starting Cleanup of User's Temporary Files and Directories...
2025-05-06T13:47:29+02:00 [1:mail1:systemd] Finished Cleanup of User's Temporary Files and Directories.
2025-05-06T13:52:03+02:00 [1:mail1:agent@mail1] Copying config sha256:a66b452f9a064a65913503295eb00758e48554a6f1e6cc5fdb2601b0adcedcf3
2025-05-06T13:52:03+02:00 [1:mail1:agent@mail1] Writing manifest to image destination
2025-05-06T13:52:03+02:00 [1:mail1:agent@mail1] a66b452f9a064a65913503295eb00758e48554a6f1e6cc5fdb2601b0adcedcf3
2025-05-06T13:52:04+02:00 [1:mail1:agent@mail1] Trying to pull ghcr.io/nethserver/mail-postfix:1.6.3...
2025-05-06T13:52:05+02:00 [1:mail1:agent@mail1] Getting image source signatures
2025-05-06T13:52:05+02:00 [1:mail1:agent@mail1] Copying blob sha256:b0f6f1c319a1570f67352d490370f0aeb5c0e67a087baf2d5f301ad51ec18858
2025-05-06T13:52:05+02:00 [1:mail1:agent@mail1] Copying blob sha256:5dea46fcb4192fb109b02d27d3a61cd4652d1c0be12782af388c54e37730a309
2025-05-06T13:53:34+02:00 [1:mail1:agent@mail1] Copying config sha256:dc046ce6474549b2f45984356e5ee606d1dc3e4b2fa1dcd060a800544b24edd7
2025-05-06T13:53:34+02:00 [1:mail1:agent@mail1] Writing manifest to image destination
2025-05-06T13:53:34+02:00 [1:mail1:agent@mail1] dc046ce6474549b2f45984356e5ee606d1dc3e4b2fa1dcd060a800544b24edd7
2025-05-06T13:53:35+02:00 [1:mail1:agent@mail1] Trying to pull ghcr.io/nethserver/mail-rspamd:1.6.3...
2025-05-06T13:53:36+02:00 [1:mail1:agent@mail1] Getting image source signatures
2025-05-06T13:53:36+02:00 [1:mail1:agent@mail1] Copying blob sha256:b0f6f1c319a1570f67352d490370f0aeb5c0e67a087baf2d5f301ad51ec18858
2025-05-06T13:53:36+02:00 [1:mail1:agent@mail1] Copying blob sha256:75d92f77e47ef0cd1892f4e47601430170ee55399c793115f3a2f4a205f005bf
2025-05-06T13:58:39+02:00 [1:mail1:agent@mail1] Copying config sha256:87b416caf24f0a13e63c583fc49f49c7c5272941a3635dcd05ef5916fe788624
2025-05-06T13:58:39+02:00 [1:mail1:agent@mail1] Writing manifest to image destination
2025-05-06T13:58:39+02:00 [1:mail1:agent@mail1] 87b416caf24f0a13e63c583fc49f49c7c5272941a3635dcd05ef5916fe788624
2025-05-06T13:58:39+02:00 [1:mail1:agent@mail1] Trying to pull ghcr.io/nethserver/mail-clamav:1.6.3...
2025-05-06T13:58:40+02:00 [1:mail1:agent@mail1] Getting image source signatures
2025-05-06T13:58:40+02:00 [1:mail1:agent@mail1] Copying blob sha256:00c5f026def0af90f778750a826f1a97fb76326e3cbfcbe230951bfeaa4b9883
2025-05-06T13:58:40+02:00 [1:mail1:agent@mail1] Copying blob sha256:b0f6f1c319a1570f67352d490370f0aeb5c0e67a087baf2d5f301ad51ec18858
2025-05-06T14:03:52+02:00 [1:mail1:agent@mail1] Copying config sha256:255071aad1a1f1deb19746c556a75347a7d17f798789deed44f767dd8835f24f
2025-05-06T14:03:52+02:00 [1:mail1:agent@mail1] Writing manifest to image destination
2025-05-06T14:03:52+02:00 [1:mail1:agent@mail1] 255071aad1a1f1deb19746c556a75347a7d17f798789deed44f767dd8835f24f
2025-05-06T14:03:52+02:00 [1:mail1:agent@mail1] task/module/mail1/e9a78dbc-745d-46ce-9f67-d359950f2bcb: create-module/10selfadm_role is starting
2025-05-06T14:03:53+02:00 [1:mail1:agent@mail1] 1
2025-05-06T14:03:53+02:00 [1:mail1:agent@mail1] task/module/mail1/e9a78dbc-745d-46ce-9f67-d359950f2bcb: create-module/20dhgen is starting
2025-05-06T14:03:53+02:00 [1:mail1:agent@mail1] Created symlink /home/mail1/.config/systemd/user/timers.target.wants/dhgen.timer → /home/mail1/.config/systemd/user/dhgen.timer.
2025-05-06T14:03:53+02:00 [1:mail1:systemd] Reloading.
2025-05-06T14:03:53+02:00 [1:mail1:systemd] Started Timer of Diffie-Hellman group generator.
2025-05-06T14:03:53+02:00 [1:mail1:agent@mail1] task/module/mail1/e9a78dbc-745d-46ce-9f67-d359950f2bcb: create-module/20spam-expunge is starting
2025-05-06T14:03:53+02:00 [1:mail1:agent@mail1] Created symlink /home/mail1/.config/systemd/user/timers.target.wants/spam-expunge.timer → /home/mail1/.config/systemd/user/spam-expunge.timer.
2025-05-06T14:03:53+02:00 [1:mail1:systemd] Reloading.
2025-05-06T14:03:53+02:00 [1:mail1:systemd] Started Timer of spam-expunge procedure.
2025-05-06T14:03:53+02:00 [1:mail1:agent@mail1] task/module/mail1/e9a78dbc-745d-46ce-9f67-d359950f2bcb: create-module/30grants is starting
2025-05-06T14:03:54+02:00 [1:mail1:agent@mail1] 3
2025-05-06T14:03:54+02:00 [1:mail1:agent@mail1] 4
2025-05-06T14:03:54+02:00 [1:mail1:agent@mail1] 3
2025-05-06T14:03:54+02:00 [1:mail1:agent@mail1] task/module/mail1/e9a78dbc-745d-46ce-9f67-d359950f2bcb: create-module/90firewall is starting
2025-05-06T14:03:56+02:00 [1:mail1:agent@mail1] _acontrol_task request attempt failed (WS reached EOF while waiting for node/1/task/7bc80f84-296f-4f43-8552-e2abecb37ea7). Retrying...
2025-05-06T14:03:59+02:00 [1:mail1:agent@mail1] _acontrol_task request recovered successfully at attempt 2
2025-05-06T14:03:59+02:00 [1:mail1:agent@mail1] task/module/mail1/e9a78dbc-745d-46ce-9f67-d359950f2bcb: action "create-module" status is "completed" (0) at step 90firewall
2025-05-06T14:04:04+02:00 [1:mail1:agent@mail1] task/module/mail1/ef1c629a-10c9-4606-ba4d-e47beb362b2d: import-module/00validate_import is starting
2025-05-06T14:04:04+02:00 [1:mail1:agent@mail1] task/module/mail1/ef1c629a-10c9-4606-ba4d-e47beb362b2d: import-module/02pullrsync is starting
2025-05-06T14:04:05+02:00 [1:mail1:agent@mail1] podman-pull-missing ghcr.io/nethserver/rsync:3.7.0
2025-05-06T14:04:05+02:00 [1:mail1:agent@mail1] Trying to pull ghcr.io/nethserver/rsync:3.7.0...
2025-05-06T14:04:07+02:00 [1:mail1:agent@mail1] Getting image source signatures
2025-05-06T14:04:07+02:00 [1:mail1:agent@mail1] Copying blob sha256:f269998746e3f861e7cda498a0bafff2391a265e1e9c3bd7f66e65879a5ba2f1
2025-05-06T14:04:07+02:00 [1:mail1:agent@mail1] Copying blob sha256:b0f6f1c319a1570f67352d490370f0aeb5c0e67a087baf2d5f301ad51ec18858
2025-05-06T14:04:13+02:00 [1:mail1:agent@mail1] Copying config sha256:e9bc75def280a1ce726652db04819de84dd809200e2b16ee051f1497d6d0b2a4
2025-05-06T14:04:13+02:00 [1:mail1:agent@mail1] Writing manifest to image destination
2025-05-06T14:04:13+02:00 [1:mail1:agent@mail1] e9bc75def280a1ce726652db04819de84dd809200e2b16ee051f1497d6d0b2a4
2025-05-06T14:04:13+02:00 [1:mail1:agent@mail1] task/module/mail1/ef1c629a-10c9-4606-ba4d-e47beb362b2d: import-module/05create_volumes is starting
2025-05-06T14:04:13+02:00 [1:mail1:agent@mail1] podman volume create rspamd-redis
2025-05-06T14:04:13+02:00 [1:mail1:agent@mail1] rspamd-redis
2025-05-06T14:04:13+02:00 [1:mail1:agent@mail1] podman volume create dovecot-data
2025-05-06T14:04:13+02:00 [1:mail1:agent@mail1] dovecot-data
2025-05-06T14:04:13+02:00 [1:mail1:agent@mail1] task/module/mail1/ef1c629a-10c9-4606-ba4d-e47beb362b2d: import-module/10recvstate is starting
2025-05-06T14:04:14+02:00 [1:mail1:agent@mail1] podman run --rm --privileged --network=host --workdir=/srv --env=RSYNCD_NETWORK=10.5.4.0/24 --env=RSYNCD_ADDRESS=cluster-localnode --env=RSYNCD_PORT=20007 --env=RSYNCD_USER=mail1 --env=RSYNCD_PASSWORD=352311cadc670-fb30-4fcc-8f57-87688d47286d --env=RSYNCD_SYSLOG_TAG=mail1 --volume=/dev/log:/dev/log --replace --name=rsync-mail1 --volume=/home/mail1/.config/state:/srv/state --volume=rspamd-redis:/srv/volumes/rspamd-redis --volume=dovecot-data:/srv/volumes/dovecot-data ghcr.io/nethserver/rsync:3.7.0
2025-05-06T14:04:14+02:00 [1:mail1:systemd] Started libcrun container.
2025-05-06T14:04:14+02:00 [1:mail1:mail1] rsyncd version 3.4.0 starting, listening on port 20007
2025-05-06T14:04:16+02:00 [1:mail1:mail1] connect from UNDETERMINED (10.5.4.6)
2025-05-06T14:04:16+02:00 [1:mail1:mail1] module-list request from UNDETERMINED (10.5.4.6)
2025-05-06T14:19:29+02:00 [1:mail1:systemd] Starting Diffie-Hellman group generator...
2025-05-06T14:19:30+02:00 [1:mail1:dhgen] mkdir: created directory 'tls-certs'
2025-05-06T14:19:30+02:00 [1:mail1:dhgen] Generating DH parameters, 2048 bit long safe prime
2025-05-06T14:19:54+02:00 [1:mail1:dhgen] .................................................................................................................................................................+.................................................................................................................................................+....................................................................................................................................................+...................+....................................+...........+.................................................+.+.........................................................................................................................................................................................+......................+.....................................................................................................................................................................................+........................+..............................................................................................................................................................................................................................................................................................................................................+...........................................+...............................................................................................................................................................................................................................................................................................................+...........................+...............................................+.......................................................+.....................................+.............................................................................................................................................................................................................................+.........+...............................+..................................................................................................................................................................................................................................................................+................................................................................................................................................................................................................................................................................................+......+.......+..............+............................................................................+............................................................................................................................................................................................................................................................................................+................................................................+.................+..........................................................................................................................+.............................+..........................................................+.................................................................+...+............+..............................+................................................................................................+........................................+.......+...........................................................................................................................+.............................................................+....................................................+......................................+.............................+..............................................+........................................................................................................................................................+...........................................+...........................................................................................................................................+....................................................................................+..........................................................................................+..+.....+.......................................................+................................................................................................................+............................................................................................................................................................................................................................................+.......................................................................................................+..................+.......................................................................................................+..............................................................................................................................................................................................+...............................................................++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*
2025-05-06T14:19:54+02:00 [1:mail1:dhgen] renamed 'dh.pem.tmp' -> 'dh.pem'
2025-05-06T14:19:54+02:00 [1:mail1:systemd] postfix.service: Unit cannot be reloaded because it is inactive.
2025-05-06T14:19:54+02:00 [1:mail1:systemd] dovecot.service: Unit cannot be reloaded because it is inactive.
2025-05-06T14:19:54+02:00 [1:mail1:systemctl] postfix.service is not active, cannot reload.
2025-05-06T14:19:54+02:00 [1:mail1:systemctl] dovecot.service is not active, cannot reload.
2025-05-06T14:19:54+02:00 [1:mail1:systemd] Finished Diffie-Hellman group generator.
2025-05-06T14:19:54+02:00 [1:mail1:systemd] dhgen.service: Consumed 24.310s CPU time.
2025-05-06T17:39:56+02:00 [1:mail1:mail1] connect from UNDETERMINED (10.5.4.6)
2025-05-06T17:39:56+02:00 [1:mail1:mail1] rsync allowed access on module data from UNDETERMINED (10.5.4.6)
2025-05-06T17:39:56+02:00 [1:mail1:mail1] rsync to data/ from mail1@UNDETERMINED (10.5.4.6)
2025-05-06T17:39:56+02:00 [1:mail1:mail1] receiving file list
2025-05-06T17:39:56+02:00 [1:mail1:mail1] sent 24 bytes  received 57 bytes  total size 215
2025-05-06T17:40:10+02:00 [1:mail1:mail1] connect from UNDETERMINED (10.5.4.6)
2025-05-06T17:40:10+02:00 [1:mail1:mail1] rsync allowed access on module data from UNDETERMINED (10.5.4.6)
2025-05-06T17:40:10+02:00 [1:mail1:mail1] rsync to data/ from mail1@UNDETERMINED (10.5.4.6)
2025-05-06T17:40:10+02:00 [1:mail1:mail1] receiving file list
2025-05-06T17:40:10+02:00 [1:mail1:mail1] sent 24 bytes  received 57 bytes  total size 215
2025-05-06T17:56:06+02:00 [1:mail1:agent@mail1] task/module/mail1/e84d60ac-9abb-4bba-9fdc-6769e90ff01d: get-configuration/20read is starting
2025-05-06T17:56:06+02:00 [1:mail1:agent@mail1] task/module/mail1/d34b6432-3afb-426e-a2c7-8f7ba1d224c4: get-name/50get_name is starting
2025-05-06T17:56:06+02:00 [1:mail1:agent@mail1] task/module/mail1/66aa6b22-a8fa-4746-9115-ec1d05af4d67: get-status/20read is starting
2025-05-06T17:56:06+02:00 [1:mail1:agent@mail1] task/module/mail1/e2d610d5-63aa-44d3-b1ad-e9a11bab26df: get-filter-configuration/10get_filter_configuration is starting
2025-05-06T17:56:06+02:00 [1:mail1:agent@mail1] task/module/mail1/d34b6432-3afb-426e-a2c7-8f7ba1d224c4: action "get-name" status is "completed" (0) at step 50get_name
2025-05-06T17:56:07+02:00 [1:mail1:agent@mail1] task/module/mail1/e84d60ac-9abb-4bba-9fdc-6769e90ff01d: action "get-configuration" status is "completed" (0) at step validate-output.json
2025-05-06T17:56:07+02:00 [1:mail1:agent@mail1] task/module/mail1/e2d610d5-63aa-44d3-b1ad-e9a11bab26df: action "get-filter-configuration" status is "completed" (0) at step validate-output.json
2025-05-06T17:56:07+02:00 [1:mail1:agent@mail1] task/module/mail1/66aa6b22-a8fa-4746-9115-ec1d05af4d67: action "get-status" status is "completed" (0) at step validate-output.json
2025-05-06T17:56:07+02:00 [1:mail1:agent@mail1] task/module/mail1/7fbe0b2a-5662-4e9f-b6ed-817d3014e13c: get-defaults/10get_defaults is starting
2025-05-06T17:56:08+02:00 [1:mail1:agent@mail1] task/module/mail1/7fbe0b2a-5662-4e9f-b6ed-817d3014e13c: action "get-defaults" status is "completed" (0) at step validate-output.json
2025-05-06T17:58:00+02:00 [1:mail1:mail1] connect from UNDETERMINED (10.5.4.6)
2025-05-06T17:58:00+02:00 [1:mail1:mail1] rsync allowed access on module data from UNDETERMINED (10.5.4.6)
2025-05-06T17:58:00+02:00 [1:mail1:mail1] rsync to data/ from mail1@UNDETERMINED (10.5.4.6)
2025-05-06T17:58:00+02:00 [1:mail1:mail1] receiving file list
2025-05-06T17:58:00+02:00 [1:mail1:mail1] sent 24 bytes  received 57 bytes  total size 215
2025-05-06T20:29:19+02:00 [1:mail1:agent@mail1] task/module/mail1/46590c6d-54d7-4c65-b833-939ad84235f6: get-name/50get_name is starting
2025-05-06T20:29:19+02:00 [1:mail1:agent@mail1] task/module/mail1/31d86b8a-fe3a-4a8a-9df5-8b95d287954f: get-configuration/20read is starting
2025-05-06T20:29:19+02:00 [1:mail1:agent@mail1] task/module/mail1/f12b41bc-1b39-4041-bdc3-4c0d330b532a: get-status/20read is starting
2025-05-06T20:29:19+02:00 [1:mail1:agent@mail1] task/module/mail1/10f2f5a6-b1b1-4438-9224-ffb20d756274: get-filter-configuration/10get_filter_configuration is starting
2025-05-06T20:29:20+02:00 [1:mail1:agent@mail1] task/module/mail1/46590c6d-54d7-4c65-b833-939ad84235f6: action "get-name" status is "completed" (0) at step 50get_name
2025-05-06T20:29:20+02:00 [1:mail1:agent@mail1] task/module/mail1/31d86b8a-fe3a-4a8a-9df5-8b95d287954f: action "get-configuration" status is "completed" (0) at step validate-output.json
2025-05-06T20:29:20+02:00 [1:mail1:agent@mail1] task/module/mail1/10f2f5a6-b1b1-4438-9224-ffb20d756274: action "get-filter-configuration" status is "completed" (0) at step validate-output.json
2025-05-06T20:29:20+02:00 [1:mail1:agent@mail1] task/module/mail1/4c867301-6f58-4ba4-8366-08925542c2cd: get-defaults/10get_defaults is starting
2025-05-06T20:29:20+02:00 [1:mail1:agent@mail1] task/module/mail1/f12b41bc-1b39-4041-bdc3-4c0d330b532a: action "get-status" status is "completed" (0) at step validate-output.json
2025-05-06T20:29:21+02:00 [1:mail1:agent@mail1] task/module/mail1/4c867301-6f58-4ba4-8366-08925542c2cd: action "get-defaults" status is "completed" (0) at step validate-output.json

maybe again a timeout error?

2025-05-06T13:42:34+02:00 [1:mail1:agent@mail1] time="2025-05-06T11:42:34Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://ghcr.io/nethserver/mail-dovecot:1.6.3: pinging container registry ghcr.io: Get \"https://ghcr.io/v2/\": dial tcp: lookup ghcr.io on [fd10:1:1:1::1]:53: read udp [fd10:1:1:1::6]:42152->[fd10:1:1:1::1]:53: i/o timeout"

But no error message shown on GUI of NS8 server.

log for imapsync application @ NS8

also timeout errors

2025-05-06T13:50:45+02:00 [1:imapsync1:agent@imapsync1] Trying to pull ghcr.io/nethserver/rsync:3.7.0...
2025-05-06T13:51:21+02:00 [1:imapsync1:agent@imapsync1] time="2025-05-06T11:51:21Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://ghcr.io/nethserver/rsync:3.7.0: Get \"https://ghcr.io/token?scope=repository%3Anethserver%2Frsync%3Apull&service=ghcr.io\": dial tcp: lookup ghcr.io on [fd10:1:1:1::1]:53: read udp [fd10:1:1:1::6]:37597->[fd10:1:1:1::1]:53: i/o timeout"
2025-05-06T13:51:51+02:00 [1:imapsync1:agent@imapsync1] time="2025-05-06T11:51:51Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://ghcr.io/nethserver/rsync:3.7.0: pinging container registry ghcr.io: Get \"https://ghcr.io/v2/\": dial tcp: lookup ghcr.io on [fd10:1:1:1::1]:53: read udp [fd10:1:1:1::6]:49809->[fd10:1:1:1::1]:53: i/o timeout"

Not sure how to continue at best: abort mail migration and try it again? Should I delete then mail, imapsync and wepttop instances at NS8 before restarting the mail migration (when still the migration link between NS7 and NS8 remains established)?
Thank you!

mrmarkuz · May 6, 2025, 8:39pm

Yes, you can remove it from Software Center but usually you should migrate to a fresh installed NS8, see also NethServer 7 migration — NS8 documentation

Does it work to pull the image manually?

runagent -m imapsync2 podman pull ghcr.io/nethserver/rsync:3.7.0

No, when aborting an app migration the application instance is removed from NS8, see also NethServer 7 migration — NS8 documentation

jrieder · May 7, 2025, 8:51pm

running the command manually for instance#2, then with the instance #1

[root@XXXX ~]# runagent -m imapsync2 podman pull ghcr.io/nethserver/rsync:3.7.0
runagent: [FATAL] Cannot find module imapsync2 in the local node
[root@main-ns8 ~]#


[root@XXXX ~]# runagent -m imapsync1 podman pull ghcr.io/nethserver/rsync:3.7.0
Trying to pull ghcr.io/nethserver/rsync:3.7.0...
Getting image source signatures
Copying blob b0f6f1c319a1 skipped: already exists
Copying blob f269998746e3 skipped: already exists
Copying config e9bc75def2 done   |
Writing manifest to image destination
e9bc75def280a1ce726652db04819de84dd809200e2b16ee051f1497d6d0b2a4

same for dovecot where also the timeout occured.

[root@XXXXX ~]# runagent -m mail1 podman pull ghcr.io/nethserver/mail-dovecot:1.6.3
Trying to pull ghcr.io/nethserver/mail-dovecot:1.6.3...
Getting image source signatures
Copying blob d021d69dcf56 skipped: already exists
Copying blob b0f6f1c319a1 skipped: already exists
Copying config a66b452f9a done   |
Writing manifest to image destination
a66b452f9a064a65913503295eb00758e48554a6f1e6cc5fdb2601b0adcedcf3

What about this error in ns8-migration.log:

Assertion failed
  File "/var/lib/nethserver/cluster/actions/import-module/50import", line 50, in <module>
    agent.assert_exp(add_module_result['exit_code'] == 0) # add-module is successful

mrmarkuz · May 7, 2025, 9:12pm

Sorry, on my test VM it was imapsync2 so in your case imapsync1 is correct.

But this way it seems to work to pull the image.

I’m also wondering about the IPv6 addresses.

Does it help to change the DNS server?

jrieder · May 8, 2025, 5:16am

Not quite sure what to change on the DNS server. As said no IPv6 entries are configured, asking for AAAA record for ghcr.io will fail though but as unkown entries are forwarded to external DNS, I don’t think I can solve that.

[admin@XXXXX ~]$ dig  ghcr.io

; <<>> DiG 9.16.23-RH <<>> ghcr.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8619
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;ghcr.io.                       IN      A

;; ANSWER SECTION:
ghcr.io.                13      IN      A       140.82.121.34

;; Query time: 11 msec
;; SERVER: 10.0.1.1#53(10.0.1.1)
;; WHEN: Thu May 08 05:33:26 UTC 2025
;; MSG SIZE  rcvd: 41

[admin@XXXXX ~]$ dig  -6 -t AAAA ghcr.io

; <<>> DiG 9.16.23-RH <<>> -6 -t AAAA ghcr.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28813
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;ghcr.io.                       IN      AAAA

;; Query time: 11 msec
;; SERVER: fd10:1:1:1::1#53(fd10:1:1:1::1)
;; WHEN: Thu May 08 05:34:08 UTC 2025
;; MSG SIZE  rcvd: 25

But it looks to me that the images are all pulled and installed finally anyway? I would probably now nevertheless try to abort email migration and restart email migration to make sure the email app is fully installed and prepared on NS8 for starting “sync data”?

jrieder · May 8, 2025, 7:07am

abort mail migration also failed:

no entry in ns8-migration.log

but in /var/log/messages:

May  8 09:01:35 XXXX cockpit-bridge: /usr/libexec/nethserver/api/nethserver-ns8-migration/migration/update: line 73: /var/log/ns8-migration.log: Permission denied

Guess the migration tool cannot even log the error message Is this a bug of the migration tool?

Now I cannot start “sync data” as well - button is not clickable any more (only “Finish migration” - which I did not yet do, and “Abort migration” are clickable - visible via hovering the curser over the buttons).

Should I redo entire migration from a clean NS8 once again?

mrmarkuz · May 8, 2025, 8:49am

Are you logged in to cockpit on NS7 as another user (admin) instead of root? This could be the cause for the “permission denied” error of the migration tool.

jrieder · May 8, 2025, 9:44am

tried with user root in cockpit (although I never used NS7 like that):
On Email -migration:

“sync-data” again clickable - but maybe the new login was also releasing that - but same error in GUI (managed now to use the copy button via zooming)

 echo '{"app":"nethserver-mail","action":"sync"}' | /usr/bin/setsid /usr/bin/sudo /usr/libexec/nethserver/api/nethserver-ns8-migration/migration/update | jq

in /var/log/messages:

May  8 11:31:01 XXXXX cockpit-bridge: /usr/sbin/ns8-check-import: line 35: bind.env: No such file or directory

Maybe this is now a hint why sync does not start?

still no entry in ns8-migration.log except the single line

----------- sync nethserver-mail Thu, 08 May 2025 11:31:01 +0200

“abort migration” button
permission denied entry in /var/log/messages indeed disappeard with using Cockpit with user root

same error though

but in ns8-migration.log now an entry - but not really helpful

----------- abort nethserver-mail Thu, 08 May 2025 11:34:13 +0200

davidep · May 8, 2025, 9:51am

The Abort migration does two things:

Removes instances in NS8 cluster. You can do it by hand from the NS8 Software Center. E.g. remove Mail, Webtop, SOGo, Imapsync, Roundcube… any instance of the Mail stack the migration tool installed automatically.
Cleans the directories under /var/lib/nethserver/nethserver-ns8-migration/ corresponding to NS7 apps. You can do it manually as well: rm -rf /var/lib/nethserver/nethserver-ns8-migration/nethserver-someapp-dir

jrieder · May 8, 2025, 6:39pm

Thanks, I followed this manual procedure.
Then I could restart email migration.
Again failed (not even came to the sync).

----------- start nethserver-mail Thu, 08 May 2025 12:22:43 +0200
mkdir: created directory ‘/var/lib/nethserver/nethserver-ns8-migration/nethserver-webtop5’
mkdir: created directory ‘/var/lib/nethserver/nethserver-ns8-migration/nethserver-mail-getmail’
Assertion failed
  File "/var/lib/nethserver/cluster/actions/import-module/50import", line 50, in <module>
    agent.assert_exp(add_module_result['exit_code'] == 0) # add-module is successful
Assertion failed
  File "/var/lib/nethserver/cluster/actions/import-module/50import", line 50, in <module>
    agent.assert_exp(add_module_result['exit_code'] == 0) # add-module is successful

Then I deleted again the already installed mail modules at NS8 (imapsync, webtop, email) manuelly. I could again restart mail migration

[INFO] Created remote module instance webtop2
[INFO] App nethserver-webtop5 is bound to rsync://webtop2@10.5.4.1:20005, waiting for task module/webtop2/task/164587d1-529b-4ec8-ad18-615870cda02f
----------- start nethserver-mail Thu, 08 May 2025 18:12:20 +0200
Assertion failed
  File "/var/lib/nethserver/cluster/actions/import-module/50import", line 50, in <module>
    agent.assert_exp(add_module_result['exit_code'] == 0) # add-module is successful
[INFO] Created remote module instance mail3
[INFO] App nethserver-mail is bound to rsync://mail3@10.5.4.1:20002, waiting for task module/mail3/task/6d1e6e39-6d12-403d-b016-e0715ebb2c91

It did not finish. I reloaded the Cockpit GUI, the mail-migration pop-up disappeard, but all the buttons remain grayed out - not clickable. The icons at all mail related apps are showing “busy” - running circly. never stops. At NS8 only two out of 3 instanzed there (Webtop missing)

davidep · May 8, 2025, 7:16pm

The problem seems on ns8 side:

check ns8 logs
check availble disk space on ns8
check for ns8 temporary DNS issues

jrieder · May 9, 2025, 7:59pm

Thanks for the hints.
Lot of tracing and tests now performed.
Finally it turned out

there was a rate limit on the testlab router active (not known so far) which apperently led to congestion and TCP retransmissions especially when multiple downloads have been started in parallel - and maybe this also affected the link between NS7 and NS8 (not that sure about that), and thus the downloads were in general slow (total BW of about 100 to 120 kbit/s). If this was finally then only a timer issue on the migration scripts or if even packet loss occured I could not deduce.
there was some auto IPv6 config on NS8 server active - removed then all IPv6 config from the network links of NS8 server. This maybe explains why suddenly IPv6 DNS request was issued by NS8 server.

After solving 1) (Removed eventually the rate limit) and 2) :

Mail migration was then successfull - could not yet check all data and features - but main stuff seems to work
AD/Samba migration successfull in 2nd attempt (also here - could not yet check yet everything)

One feedback from that:
Local AD migration failed the first attempt as firewalld was still stopped (I stopped it during issue investigation, but not turned on again before AD migration started). Error was in
add to module/samba1 environment

Traceback (most recent call last):
  File "/var/lib/nethserver/node/actions/add-public-service/50add", line 50, in <module>
    raise ex
  File "/var/lib/nethserver/node/actions/add-public-service/50add", line 36, in <module>
    current_ports = set(subprocess.check_output(
                        ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['firewall-cmd', '--permanent', '--service=samba1', '--get-ports']' returned non-zero exit status 252.

Now two samba instances are appearing - but samba1 seems not configured and used. - not sure if this can be removed - seems not possible in SoftwareCenter the usual way (samba not shown in SW center - only in the apps menu)

Now I will check the rest…

But still many, many thanks for all help and hints so far…

mrmarkuz · May 9, 2025, 10:09pm

In Software Center Samba instances are just listed as core app.
The Samba user domains are shown on the “Domains and users” page and can be removed there.

mrmarkuz · May 20, 2025, 9:16am

The bug is fixed, it’s possible now to abort an app migration with a non-root user that’s member of the domain admins group, see Migration tool app abort error with non-root user · Issue #7449 · NethServer/dev · GitHub