Incomplete Migration - Nextcloud

Ted · April 22, 2024, 9:13pm

NethServer Version: NS8
Module: NS8 Migration / Nextcloud

Hi guys, very similar to this thread (Nextcloud migration not working - data directory not writable) my connected NS7 node kicks off a data sync process, but nothing seems to be happening (~8 hours). However, I don’t appear to be having any permissions errors.

Screenshot 2024-04-22 at 9.50.50 PM

I’d be happy enough to rsync the files manually, assuming that will work. The module appears to be running correctly on NS8 as near as I can tell. It’s just missing the data files. However, I don’t want to mess things up.

Any idea what is going wrong? Suggestions? Things to try?

Right now, when I click “Sync Data” in the migration tool, the progress bar pops up on the NS7 machine showing 0%. It stays for a few minutes and then says “Sync Complete.” The number never changes from 0%.

I notice in the logs that even though each of the nodes has a specific name, the request always comes from UNDETERMINED.

On the NS7 host, Nextcloud data is stored on a separate drive mounted at /home/cloudData. It’s using about 1.6TB.

On the NS8 host, I have /home mounted as a separate drive using 6.6G. Nextcloud is the ONLY installed module on the node, though it is a worker node in a larger cluster.

Thanks!

The NS7 log is effectively empty:

----------- sync nethserver-nextcloud Mon, 22 Apr 2024 19:26:55 +0000

I think this is the relevant log from the NS8 machine:

2024-04-22T20:26:57+01:00 [9:nextcloud1:systemd] Started podman-pause-7947a569.scope.
2024-04-22T20:26:58+01:00 [9:nextcloud1:agent@nextcloud1] task/module/nextcloud1/53ede537-b8e8-445f-92b7-e884ffb0a3e4: import-module/05create_volumes is starting
2024-04-22T20:26:58+01:00 [9:nextcloud1:agent@nextcloud1] task/module/nextcloud1/53ede537-b8e8-445f-92b7-e884ffb0a3e4: import-module/10recvstate is starting
2024-04-22T20:26:59+01:00 [9:nextcloud1:agent@nextcloud1] podman run --rm --privileged --network=host --workdir=/srv --env=RSYNCD_NETWORK=10.5.4.0/24 --env=RSYNCD_ADDRESS=cluster-localnode --env=RSYNCD_PORT=20010 --env=RSYNCD_USER=nextcloud1 --env=RSYNCD_PASSWORD=XXXX --env=RSYNCD_SYSLOG_TAG=nextcloud1 --volume=/dev/log:/dev/log --name=rsync-nextcloud1 --volume=/home/nextcloud1/.config/state:/srv/state --volume=nextcloud-app-data:/srv/volumes/nextcloud-app-data ghcr.io/nethserver/rsync:2.7.0
2024-04-22T20:26:59+01:00 [9:nextcloud1:podman] 2024-04-22 14:26:59.359176904 -0500 CDT m=+0.019580235 image pull  ghcr.io/nethserver/rsync:2.7.0
2024-04-22T20:26:59+01:00 [9:nextcloud1:podman]
2024-04-22T20:26:59+01:00 [9:nextcloud1:podman] 2024-04-22 14:26:59.493328403 -0500 CDT m=+0.153731704 container create 13e7b73552157fd52fb1afc6bf05df208e676a42830df0b5b1a5223dfe6abdbc (image=ghcr.io/nethserver/rsync:2.7.0, name=rsync-nextcloud1, io.buildah.version=1.23.1)
2024-04-22T20:26:59+01:00 [9:nextcloud1:systemd] Started libpod-13e7b73552157fd52fb1afc6bf05df208e676a42830df0b5b1a5223dfe6abdbc.scope - libcrun container.
2024-04-22T20:26:59+01:00 [9:nextcloud1:podman] 2024-04-22 14:26:59.532979821 -0500 CDT m=+0.193383132 container init 13e7b73552157fd52fb1afc6bf05df208e676a42830df0b5b1a5223dfe6abdbc (image=ghcr.io/nethserver/rsync:2.7.0, name=rsync-nextcloud1, io.buildah.version=1.23.1)
2024-04-22T20:26:59+01:00 [9:nextcloud1:podman] 2024-04-22 14:26:59.538323463 -0500 CDT m=+0.198726795 container start 13e7b73552157fd52fb1afc6bf05df208e676a42830df0b5b1a5223dfe6abdbc (image=ghcr.io/nethserver/rsync:2.7.0, name=rsync-nextcloud1, io.buildah.version=1.23.1)
2024-04-22T20:26:59+01:00 [9:nextcloud1:podman] 2024-04-22 14:26:59.53844944 -0500 CDT m=+0.198852741 container attach 13e7b73552157fd52fb1afc6bf05df208e676a42830df0b5b1a5223dfe6abdbc (image=ghcr.io/nethserver/rsync:2.7.0, name=rsync-nextcloud1, io.buildah.version=1.23.1)
2024-04-22T20:26:59+01:00 [9:nextcloud1:nextcloud1] rsyncd version 3.2.7 starting, listening on port 20010
2024-04-22T20:26:59+01:00 [9:nextcloud1:nextcloud1] connect from UNDETERMINED (10.5.4.10)
2024-04-22T20:27:00+01:00 [9:nextcloud1:nextcloud1] rsync allowed access on module data from UNDETERMINED (10.5.4.10)
2024-04-22T20:27:00+01:00 [9:nextcloud1:nextcloud1] rsync to data/ from nextcloud1@UNDETERMINED (10.5.4.10)
2024-04-22T20:27:00+01:00 [9:nextcloud1:nextcloud1] receiving file list
2024-04-22T20:27:02+01:00 [9:nextcloud1:nextcloud1] sent 24 bytes  received 57 bytes  total size 230
2024-04-22T20:27:03+01:00 [9:nextcloud1:nextcloud1] connect from UNDETERMINED (10.5.4.10)
2024-04-22T20:27:03+01:00 [9:nextcloud1:nextcloud1] rsync allowed access on module data from UNDETERMINED (10.5.4.10)
2024-04-22T20:27:04+01:00 [9:nextcloud1:nextcloud1] rsync to data/volumes/nextcloud-app-data/data/ from nextcloud1@UNDETERMINED (10.5.4.10)
2024-04-22T20:27:04+01:00 [9:nextcloud1:nextcloud1] receiving file list
2024-04-22T20:27:06+01:00 [9:nextcloud1:nextcloud1] sent 30 bytes  received 1768 bytes  total size 21323638
2024-04-22T20:27:06+01:00 [9:nextcloud1:nextcloud1] connect from UNDETERMINED (10.5.4.10)
2024-04-22T20:27:07+01:00 [9:nextcloud1:nextcloud1] rsync allowed access on module data from UNDETERMINED (10.5.4.10)
2024-04-22T20:27:07+01:00 [9:nextcloud1:nextcloud1] rsync to data/state/restore/ from nextcloud1@UNDETERMINED (10.5.4.10)
2024-04-22T20:27:07+01:00 [9:nextcloud1:nextcloud1] receiving file list
2024-04-22T20:27:09+01:00 [9:nextcloud1:nextcloud1] sent 892 bytes  received 903 bytes  total size 98939
2024-04-22T20:27:10+01:00 [9:nextcloud1:nextcloud1] connect from UNDETERMINED (10.5.4.10)
2024-04-22T20:27:10+01:00 [9:nextcloud1:nextcloud1] rsync allowed access on module data from UNDETERMINED (10.5.4.10)
2024-04-22T20:27:11+01:00 [9:nextcloud1:nextcloud1] rsync to data/state/ from nextcloud1@UNDETERMINED (10.5.4.10)
2024-04-22T20:27:11+01:00 [9:nextcloud1:nextcloud1] receiving file list
2024-04-22T20:27:12+01:00 [9:nextcloud1:nextcloud1] sent 17 bytes  received 63 bytes  total size 1679
2024-04-22T20:29:08+01:00 [9:nextcloud1:agent@nextcloud1] task/module/nextcloud1/0cf7373e-5dbe-413f-8336-f7ea6dcc3827: get-configuration/20read is starting
2024-04-22T20:29:53+01:00 [9:nextcloud1:agent@nextcloud1] task/module/nextcloud1/0cf7373e-5dbe-413f-8336-f7ea6dcc3827: action "get-configuration" status is "completed" (0) at step 20read
2024-04-22T20:34:42+01:00 [9:nextcloud1:agent@nextcloud1] task/module/nextcloud1/fe818807-6dfb-46bd-ab5f-d9208405f796: get-name/50get_name is starting
2024-04-22T20:34:42+01:00 [9:nextcloud1:agent@nextcloud1] task/module/nextcloud1/e5766b57-16b9-4aca-add4-943330c91654: get-configuration/20read is starting
2024-04-22T20:34:43+01:00 [9:nextcloud1:agent@nextcloud1] task/module/nextcloud1/fe818807-6dfb-46bd-ab5f-d9208405f796: action "get-name" status is "completed" (0) at step 50get_name
2024-04-22T20:34:46+01:00 [9:nextcloud1:agent@nextcloud1] task/module/nextcloud1/274233ac-3590-4be9-a0dc-a3ea77a0020e: get-configuration/20read is starting
2024-04-22T20:34:46+01:00 [9:nextcloud1:agent@nextcloud1] task/module/nextcloud1/23a4df0d-f266-4aae-9c48-a3708216bace: get-status/20read is starting
2024-04-22T20:34:48+01:00 [9:nextcloud1:agent@nextcloud1] task/module/nextcloud1/23a4df0d-f266-4aae-9c48-a3708216bace: action "get-status" status is "completed" (0) at step validate-output.json
2024-04-22T20:35:26+01:00 [9:nextcloud1:agent@nextcloud1] task/module/nextcloud1/e5766b57-16b9-4aca-add4-943330c91654: action "get-configuration" status is "completed" (0) at step 20read
2024-04-22T20:35:31+01:00 [9:nextcloud1:agent@nextcloud1] task/module/nextcloud1/274233ac-3590-4be9-a0dc-a3ea77a0020e: action "get-configuration" status is "completed" (0) at step 20read

Andy_Wismer · April 22, 2024, 10:44pm

Hi

This is probably the issue, as a second disk like this was never officially supported in NS7, even though it did work.

Better would be to unmount the disk, let it sync an “empty” folder, then transfer your data with rsync manually and remount the stuff where it’s needed.

→ Be careful, a container is a different animal than a full blown Linux, like NS7 was…

You may need to start from scratch…

My 2 cents
Andy

Ted · April 23, 2024, 5:38am

Thanks, @Andy_Wismer that makes sense. I’m on the road the next two days, but I’ll gove it a go when I return and post how it goes.

Ted · April 26, 2024, 2:55pm

Following your suggestion, I unmounted the source drive (/home/cloudData) and reran the sync. No indication of change, but since it no longer had data to move, I thought maybe that was intended behavior.
I remounted the drive and rsynced the data across into the /home/nextcloud1/.local/share/containers/storage/volumes/nextcloud-app-data/_data/data folder.
I tested nextcloud and received an error about the permissions on the data folder.
I tried to rerun the sync (in migration wizard) in the hopes that would fix potential permissions issues:

#  echo '{"app":"nethserver-nextcloud","action":"sync"}' | /usr/bin/setsid /usr/bin/sudo /usr/libexec/nethserver/api/nethserver-ns8-migration/migration/update | jq
{
  "progress": "0.00",
  "time": "0.0",
  "exit": 0,
  "event": "migration-sync",
  "state": "running",
  "step": 0,
  "pid": 0,
  "action": ""
}
{
  "pid": 0,
  "status": "failed",
  "event": "migration-sync"
}
{
  "id": "1714142540",
  "type": "ApiFailed",
  "message": "sync nethserver-nextcloud failed"
}

I manually changed the permissions on the data folder to match the permissions on the other files in /home/nextcloud1/.local/share/containers/storage/volumes/nextcloud-app-data/_data

chown <ID#>:<GID#> /home/nextcloud1/.local/share/containers/storage/volumes/nextcloud-app-data/_data/data

Then I reran the sync and… it actually ran rsync against the original data. It appears to me that the initial issue may have been incorrect permissions on the data folder when it was created by the migration wizard. Is that a possibility?

Andy_Wismer · April 26, 2024, 3:02pm

Hi Ted

As I’m not a dev, I’ll leave the decision about it to @davidep who might be interested in this probable bug you seem to have found…
He IS one of the key devs and is very interested in capturing bugs still in the wild!

My 2 cents
Andy

davidep · April 26, 2024, 3:24pm

What does it mean? Did you mount it as external storage with SMB?

The migration tool migrates only the NC primary storage contents, under /var/lib/nethserver/nextcloud.

Ted · April 26, 2024, 3:42pm

Hi @davidep

It’s been many years since I set this up, and I can’t find any notes on how I did it. However, there is a separate data directory configured in /usr/share/nextcloud/config/config.php.

In this case, /home/cloudData is a raid6 array which is local to the NS7 machine.

I understand that is the intent. However, I’m just reporting on my experience. When I ran the migration tool, the new data folder was created on the NS8 host with permissions such that the NS8 host lacked write permission – preventing data sync. Once I resolved this, the NS7 migration tool launched rsync on the NS8 host and attempted to sync the data (which I’d manually done by that point).

I have not yet clicked the finalize migration button, however, I am seeing this error:

Will the “Update needed” message be resolved by finalizing the migration? Or has something else gone wrong?

Thanks!

Ted · April 27, 2024, 5:11pm

Of course, one path forward would be to follow the NextCloud docs and manually update, but I am assuming that NS8 follows the NS7 model wherein I should only ever use the module updates and not the NextCloud internal update. Can anyone confirm or direct me here?

dnutan · April 27, 2024, 6:09pm

I would say you can safely run:

runagent -m nextcloud1 occ upgrade

It shall give more details about the state nextcloud is and the problem it is facing.

Correct as far as I know

Ted · April 27, 2024, 9:45pm

Thank you. This helped. I get the login screen now, but neither admin nor my user account can login. Every step reveals a new obstacle.

dnutan · April 28, 2024, 12:32pm

Does the nextcloud instance log show anything relevant (error/warning…)?
Still doesn’t work after waiting a bit more (just enough time for the services to be fully started)?

You can try to login again after resetting the admin password:

runagent -m nextcloud1 occ user:resetpassword admin

If we where on a movie I’d say something along the lines of “leave no stone unturned”

Ted · April 28, 2024, 3:39pm

Clearly trying to do too many different things at once, didn’t even think to check the logs.

Using the above command, I was able to login as the admin. It appears that none of my data came over. No users exist – hence why none of my passwords work.

I tried rerunning the data sync from the NS7 Migration tool, but now I get errors related to the read/write status of config.php.

2024-04-28T16:23:24+01:00 [9:nextcloud1:nextcloud-app] NOTICE: PHP message: [nextcloud][PHP][3] {"reqId":"x7w0DTnTmyv53SpBQFlC","level":3,"time":"2024-04-28T15:23:24+00:00","remoteAddr":"192.168.1.108","user":"--","app":"PHP","method":"GET","url":"/ocs/v2.php/apps/serverinfo/api/v1/basicdata?format=json","message":"OCP\\HintException: Configuration was not read or initialized correctly, not overwriting /var/www/html/config/config.php at /var/www/html/lib/private/Config.php#266","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36","version":"","data":{"app":"PHP"}}

It’s interesting because the Migration tool keeps creating files with a very strange UID. On the source server all of the files are UID 48 GID 48. However, post migration, they are all assigned to UID 165617 GID 165617 – though the correct user (nextcloud1) has ID 1004.

I tried fixing these file permissions manually using chmod, but then I get parse errors:

2024-04-28T16:37:40+01:00 [9:nextcloud1:agent@nextcloud1] parse error: Invalid numeric literal at line 1, column 5
2024-04-28T16:37:40+01:00 [9:nextcloud1:podman] 2024-04-28 10:37:40.522807868 -0500 CDT m=+0.303386094 container exec_died 39cc609193f46a4e9c14f6ae11adac6d9219ca0c4de1441218dcf1f07d75a36d (image=ghcr.io/nethserver/nextcloud-app:1.1.6, name=nextcloud-app, PODMAN_SYSTEMD_UNIT=nextcloud-app.service, io.buildah.version=1.23.1)

All this rock kicking is making my toes hurt!