Backup failing in NS8

danb35 · February 17, 2024, 2:08pm

I’m running the release version of NS8 on Debian 12. Still kind of testing it out, still migrating data from my current NS7 server, but obviously backup is pretty important even if its dashboard didn’t nag you.

So this morning I created an account with iDrive e2, set it up as a S3-compatible storage repository in NS8, and kicked off a backup. Almost 400 GB of data were transferred, but then NS8 reported a failure. “Copy task trace” results in this:

{"context":{"action":"run-backup","data":{"id":1},"extra":{"backupName":"Backup to iDrive e2","completion":{"extraTextParams":["backupName"],"i18nString":"backup.backup_completed_successfully"},"description":"Running Backup to iDrive e2","title":"Run backup"},"id":"b36ddd26-1cee-47c1-908c-442cf472edf1","parent":"","queue":"cluster/tasks","timestamp":"2024-02-17T14:03:13.108504158Z","user":"admin"},"status":"aborted","progress":50,"subTasks":[{"context":{"action":"run-backup","data":{"id":1},"extra":{"description":"run-backup agent action","isNotificationHidden":true,"title":"module/collabora1/run-backup"},"id":"5c4b5ae3-b5bd-48b6-a56a-e5e8c3fa0916","parent":"b36ddd26-1cee-47c1-908c-442cf472edf1"},"status":"completed","progress":100,"subTasks":[],"result":{"error":"","exit_code":0,"file":"task/module/collabora1/5c4b5ae3-b5bd-48b6-a56a-e5e8c3fa0916","output":""}},{"context":{"action":"run-backup","data":{"id":1},"extra":{"description":"run-backup agent action","isNotificationHidden":true,"title":"module/crowdsec1/run-backup"},"id":"50aaf84f-77aa-4c00-bce5-978bd2021f48","parent":"b36ddd26-1cee-47c1-908c-442cf472edf1"},"status":"completed","progress":100,"subTasks":[],"result":{"error":"","exit_code":0,"file":"task/module/crowdsec1/50aaf84f-77aa-4c00-bce5-978bd2021f48","output":""}},{"context":{"action":"run-backup","data":{"id":1},"extra":{"description":"run-backup agent action","isNotificationHidden":true,"title":"module/sogo1/run-backup"},"id":"9d50cf9b-7d5e-48da-a7ab-529b6a801701","parent":"b36ddd26-1cee-47c1-908c-442cf472edf1"},"status":"completed","progress":100,"subTasks":[],"result":{"error":"","exit_code":0,"file":"task/module/sogo1/9d50cf9b-7d5e-48da-a7ab-529b6a801701","output":""}},{"context":{"action":"run-backup","data":{"id":1},"extra":{"description":"run-backup agent action","isNotificationHidden":true,"title":"module/openldap1/run-backup"},"id":"0e8a5c35-f2bc-4ccf-810a-e80b9bc3d258","parent":"b36ddd26-1cee-47c1-908c-442cf472edf1"},"status":"completed","progress":100,"subTasks":[],"result":{"error":"","exit_code":0,"file":"task/module/openldap1/0e8a5c35-f2bc-4ccf-810a-e80b9bc3d258","output":""}},{"context":{"action":"run-backup","data":{"id":1},"extra":{"description":"run-backup agent action","isNotificationHidden":true,"title":"module/loki1/run-backup"},"id":"460a891c-cc78-4927-b0f4-2577698e4277","parent":"b36ddd26-1cee-47c1-908c-442cf472edf1"},"status":"completed","progress":100,"subTasks":[],"result":{"error":"","exit_code":0,"file":"task/module/loki1/460a891c-cc78-4927-b0f4-2577698e4277","output":""}},{"context":{"action":"run-backup","data":{"id":1},"extra":{"description":"run-backup agent action","isNotificationHidden":true,"title":"module/mail1/run-backup"},"id":"4357d5ac-6f62-410f-9642-5625ea4be7ff","parent":"b36ddd26-1cee-47c1-908c-442cf472edf1"},"status":"completed","progress":100,"subTasks":[],"result":{"error":"","exit_code":0,"file":"task/module/mail1/4357d5ac-6f62-410f-9642-5625ea4be7ff","output":""}},{"context":{"action":"run-backup","data":{"id":1},"extra":{"description":"run-backup agent action","isNotificationHidden":true,"title":"module/roundcubemail2/run-backup"},"id":"a9286127-ab0c-4c31-b02d-9cb6874cfbec","parent":"b36ddd26-1cee-47c1-908c-442cf472edf1"},"status":"completed","progress":100,"subTasks":[],"result":{"error":"","exit_code":0,"file":"task/module/roundcubemail2/a9286127-ab0c-4c31-b02d-9cb6874cfbec","output":""}},{"context":{"action":"run-backup","data":{"id":1},"extra":{"description":"run-backup agent action","isNotificationHidden":true,"title":"module/mariadb1/run-backup"},"id":"ddb54a3d-4d75-45a1-b8f3-f8ce8745bda3","parent":"b36ddd26-1cee-47c1-908c-442cf472edf1"},"status":"completed","progress":100,"subTasks":[],"result":{"error":"","exit_code":0,"file":"task/module/mariadb1/ddb54a3d-4d75-45a1-b8f3-f8ce8745bda3","output":""}},{"context":{"action":"run-backup","data":{"id":1},"extra":{"description":"run-backup agent action","isNotificationHidden":true,"title":"module/mail2/run-backup"},"id":"a59350af-e71c-461b-9ddf-fd02d0d9012b","parent":"b36ddd26-1cee-47c1-908c-442cf472edf1"},"status":"completed","progress":100,"subTasks":[],"result":{"error":"","exit_code":0,"file":"task/module/mail2/a59350af-e71c-461b-9ddf-fd02d0d9012b","output":""}},{"context":{"action":"run-backup","data":{"id":1},"extra":{"description":"run-backup agent action","isNotificationHidden":true,"title":"module/webserver1/run-backup"},"id":"90149f70-559f-4143-84e5-beb604bf5153","parent":"b36ddd26-1cee-47c1-908c-442cf472edf1"},"status":"completed","progress":100,"subTasks":[],"result":{"error":"","exit_code":0,"file":"task/module/webserver1/90149f70-559f-4143-84e5-beb604bf5153","output":""}},{"context":{"action":"run-backup","data":{"id":1},"extra":{"description":"run-backup agent action","isNotificationHidden":true,"title":"module/prometheus1/run-backup"},"id":"ebfe69b0-9b59-453f-86e2-7c958e279c92","parent":"b36ddd26-1cee-47c1-908c-442cf472edf1"},"status":"completed","progress":100,"subTasks":[],"result":{"error":"","exit_code":0,"file":"task/module/prometheus1/ebfe69b0-9b59-453f-86e2-7c958e279c92","output":""}},{"context":{"action":"run-backup","data":{"id":1},"extra":{"description":"run-backup agent action","isNotificationHidden":true,"title":"module/nextcloud1/run-backup"},"id":"7e5e2da3-2d4a-4a83-89bc-b46cf47c3f39","parent":"b36ddd26-1cee-47c1-908c-442cf472edf1"},"status":"completed","progress":100,"subTasks":[],"result":{"error":"","exit_code":0,"file":"task/module/nextcloud1/7e5e2da3-2d4a-4a83-89bc-b46cf47c3f39","output":""}},{"context":{"action":"run-backup","data":{"id":1},"extra":{"description":"run-backup agent action","isNotificationHidden":true,"title":"module/traefik1/run-backup"},"id":"6367b95b-133d-4bc1-9175-bd8bbd21daa0","parent":"b36ddd26-1cee-47c1-908c-442cf472edf1"},"status":"completed","progress":100,"subTasks":[],"result":{"error":"","exit_code":0,"file":"task/module/traefik1/6367b95b-133d-4bc1-9175-bd8bbd21daa0","output":""}},{"context":{"action":"run-backup","data":{"id":1},"extra":{"description":"run-backup agent action","isNotificationHidden":true,"title":"module/grafana1/run-backup"},"id":"95039c65-bd12-4ab8-aab6-85d3013621e0","parent":"b36ddd26-1cee-47c1-908c-442cf472edf1"},"status":"completed","progress":100,"subTasks":[],"result":{"error":"","exit_code":0,"file":"task/module/grafana1/95039c65-bd12-4ab8-aab6-85d3013621e0","output":""}},{"context":{"action":"run-backup","data":{"id":1},"extra":{"description":"run-backup agent action","isNotificationHidden":true,"title":"module/roundcubemail1/run-backup"},"id":"9eab4910-76e3-428b-b84c-86d245984c88","parent":"b36ddd26-1cee-47c1-908c-442cf472edf1"},"status":"completed","progress":100,"subTasks":[],"result":{"error":"","exit_code":0,"file":"task/module/roundcubemail1/9eab4910-76e3-428b-b84c-86d245984c88","output":""}}],"validated":true,"result":{"error":"2024/02/17 14:03:40 NOTICE: s3: s3 provider \"\" not known - please set correctly\n2024/02/17 14:03:40 ERROR : dump.json.gz.gpg: Failed to copy: AccessDenied: Access Denied.\n\tstatus code: 403, request id: 17B4AB67CB4AB0C9, host id: \n2024/02/17 14:03:40 ERROR : dump.json.gz.gpg: Post request rcat error: AccessDenied: Access Denied.\n\tstatus code: 403, request id: 17B4AB67CB4AB0C9, host id: \n2024/02/17 14:03:40 Failed to rcat: AccessDenied: Access Denied.\n\tstatus code: 403, request id: 17B4AB67CB4AB0C9, host id: \n","exit_code":1,"file":"task/cluster/b36ddd26-1cee-47c1-908c-442cf472edf1","output":""}}

When I click the “More info” slider, it shows this output:

2024/02/17 14:03:40 NOTICE: s3: s3 provider "" not known - please set correctly
2024/02/17 14:03:40 ERROR : dump.json.gz.gpg: Failed to copy: AccessDenied: Access Denied.
	status code: 403, request id: 17B4AB67CB4AB0C9, host id: 
2024/02/17 14:03:40 ERROR : dump.json.gz.gpg: Post request rcat error: AccessDenied: Access Denied.
	status code: 403, request id: 17B4AB67CB4AB0C9, host id: 
2024/02/17 14:03:40 Failed to rcat: AccessDenied: Access Denied.
	status code: 403, request id: 17B4AB67CB4AB0C9, host id:

…which makes it look like it’s looking for a s3 provider with a null string as its name. But that isn’t the case; I’ve not-so-creatively called the repository iDrive e2. Not sure where to go from here–thoughts on what to check?

Edit: I don’t know if it’s relevant or not, but since the error seems to relate to the cluster config file (which I’m guessing based on the .json.gz.gpg extension)–while the backup to e2 was running, I downloaded the cluster config to my own PC. That worked just fine, but I’m wondering if it might have confused the other backup task.

oneitonitram · February 17, 2024, 3:01pm

wait $15 storage per year for 1tb? am getting me some

danb35 · February 17, 2024, 3:17pm

$15 for year 1, $30/yr after that (as of now). I still want to get the Minio app up and running on my NAS so I can back up to that, but this works at least for the time being.

oneitonitram · February 17, 2024, 4:30pm

The minio app has an external route bug,
At the moment, but tit’s being tracked

danb35 · February 17, 2024, 4:31pm

I’m talking about the Minio app on TrueNAS, not on NS8–I don’t see a whole lot of benefit in backing up the NS8 server to itself.

danb35 · February 19, 2024, 11:28am

…but back to the subject of this thread, according to the backup schedule, two scheduled backups should have run since I configured it. I don’t see that there’s anything in the cluster-admin pages that directly tells me the status of the backups, but I don’t see a notification that they’ve failed. And if I go to the system logs page and search for backup, everything that comes up is looking good, though I still don’t see something that directly says the backup was successful. This is good, I guess?

davidep · February 19, 2024, 12:02pm

The last backup status is reported under Backup > Schedules > See details:

I think it’s worth adding this information to the manual: Update backup.rst · NethServer/ns8-docs@5f4dee5 · GitHub

Some applications also have a Backup card in their Status dashboard:

danb35 · February 19, 2024, 12:26pm

Indeed. I don’t think that’s quite the most intuitive place, but the information is there. I think there’s room on the card itself for a status indication, though, right here:
Screenshot 2024-02-19 at 07.23.48
Something like “Last run: 23:00 18 Feb 24, status: success”