Backup failing, again

Support
,
1

Creating a new thread, because it seems like a different issue than in NS8 backup failing on IDrive e2 S3. I’ve set up s3-compatible backup storage on iDrive e2, and backups for all apps are failing. Logs below (partial, because full logs are too long for this forum):

2024-11-27T05:56:06-05:00 [1:traefik1:systemd] backup3.service: Consumed 2.942s CPU time.
2024-11-27T05:56:06-05:00 [1:traefik1:agent@traefik1] See "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.
2024-11-27T05:56:06-05:00 [1:traefik1:agent@traefik1] Job for backup3.service failed because the control process exited with error code.
2024-11-27T05:56:06-05:00 [1:loki1:backup3] 2024/11/27 10:56:06 ERROR : 6ff8cb7c-0498-4cc0-ae46-2a4b3be11661.json: Failed to copy: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T05:56:06-05:00 [1:loki1:backup3] 2024/11/27 10:56:06 ERROR : 6ff8cb7c-0498-4cc0-ae46-2a4b3be11661.json: Post request rcat error: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T05:56:06-05:00 [1:loki1:backup3] 2024/11/27 10:56:06 Failed to rcat: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T05:56:06-05:00 [1:loki1:backup3] 	status code: 403, request id: 180BCDFC96955606, host id:
2024-11-27T05:56:06-05:00 [1:roundcubemail1:backup3] 2024/11/27 10:56:06 ERROR : 24e2f1f6-e3dc-4625-b814-801fb9a75f22.json: Failed to copy: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T05:56:06-05:00 [1:roundcubemail1:backup3] 2024/11/27 10:56:06 ERROR : 24e2f1f6-e3dc-4625-b814-801fb9a75f22.json: Post request rcat error: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T05:56:06-05:00 [1:roundcubemail1:backup3] 	status code: 403, request id: 180BCDFCA683AC67, host id:
2024-11-27T05:56:06-05:00 [1:roundcubemail1:backup3] 2024/11/27 10:56:06 Failed to rcat: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T05:56:06-05:00 [1:loki1:systemd] backup3.service: Main process exited, code=exited, status=1/FAILURE
2024-11-27T05:56:06-05:00 [1:loki1:systemd] backup3.service: Failed with result 'exit-code'.
2024-11-27T05:56:06-05:00 [1:loki1:systemd] backup3.service: Consumed 3.176s CPU time.
2024-11-27T05:56:06-05:00 [1:loki1:agent@loki1] Job for backup3.service failed because the control process exited with error code.
2024-11-27T05:56:06-05:00 [1:loki1:agent@loki1] See "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.
2024-11-27T05:56:06-05:00 [1:roundcubemail1:backup3] removed 'roundcubemail.sql'
2024-11-27T05:56:06-05:00 [1:roundcubemail1:systemd] backup3.service: Main process exited, code=exited, status=1/FAILURE
2024-11-27T05:56:06-05:00 [1:roundcubemail1:systemd] backup3.service: Failed with result 'exit-code'.
2024-11-27T05:56:06-05:00 [1:roundcubemail1:systemd] backup3.service: Consumed 3.349s CPU time.
2024-11-27T05:56:06-05:00 [1:roundcubemail1:agent@roundcubemail1] See "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.
2024-11-27T05:56:06-05:00 [1:roundcubemail1:agent@roundcubemail1] Job for backup3.service failed because the control process exited with error code.
2024-11-27T05:56:07-05:00 [1:mariadb1:backup3] Please note that knowledge of your password is required to access
2024-11-27T05:56:07-05:00 [1:mariadb1:backup3] the repository. Losing your password means that your data is
2024-11-27T05:56:07-05:00 [1:mariadb1:backup3] irrecoverably lost.
2024-11-27T05:56:07-05:00 [1:mariadb1:backup3] created restic repository d4f28efd29 at s3:n4p9.m12.idrivee2-2.com/ns8/mariadb/c3320cfd-3684-41fb-96d0-939215ca0599
2024-11-27T05:56:07-05:00 [1:mariadb1:backup3] restic backup state/environment --files-from=/etc/state-include.conf
2024-11-27T05:56:07-05:00 [1:joplin1:backup3] 	status code: 403, request id: 180BCDFCF164CD74, host id:
2024-11-27T05:56:07-05:00 [1:joplin1:backup3] 2024/11/27 10:56:07 ERROR : 0c21e8b8-a541-44e4-a679-144d50e73906.json: Failed to copy: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T05:56:07-05:00 [1:joplin1:backup3] 2024/11/27 10:56:07 ERROR : 0c21e8b8-a541-44e4-a679-144d50e73906.json: Post request rcat error: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T05:56:07-05:00 [1:joplin1:backup3] 2024/11/27 10:56:07 Failed to rcat: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T05:56:08-05:00 [1:joplin1:systemd] backup3.service: Main process exited, code=exited, status=1/FAILURE
2024-11-27T05:56:08-05:00 [1:joplin1:systemd] backup3.service: Failed with result 'exit-code'.
2024-11-27T05:56:08-05:00 [1:joplin1:systemd] backup3.service: Consumed 3.155s CPU time.
2024-11-27T05:56:08-05:00 [1:joplin1:agent@joplin1] See "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.
2024-11-27T05:56:08-05:00 [1:joplin1:agent@joplin1] Job for backup3.service failed because the control process exited with error code.
2024-11-27T05:56:08-05:00 [1:matomo1:backup3] Dirs:         1515 new,     0 changed,     0 unmodified
2024-11-27T05:56:08-05:00 [1:matomo1:backup3] snapshot 0024c5f2 saved
2024-11-27T05:56:08-05:00 [1:matomo1:backup3] Files:       11473 new,     0 changed,     0 unmodified
2024-11-27T05:56:08-05:00 [1:matomo1:backup3] processed 11473 files, 374.806 MiB in 0:13
2024-11-27T05:56:08-05:00 [1:matomo1:backup3] Added to the repository: 208.233 MiB (86.049 MiB stored)
2024-11-27T05:56:08-05:00 [1:matomo1:backup3] restic forget --prune --keep-last=5
2024-11-27T05:56:09-05:00 [1:mariadb1:backup3] no parent snapshot found, will read all files
2024-11-27T05:56:09-05:00 [1:matomo1:backup3]                                                                       /srv/volumes/matomo-app
2024-11-27T05:56:09-05:00 [1:matomo1:backup3]                                                                       /srv/volumes/mysql-data
2024-11-27T05:56:09-05:00 [1:matomo1:backup3] Applying Policy: keep 5 latest snapshots
2024-11-27T05:56:09-05:00 [1:matomo1:backup3] -------------------------------------------------------------------------------------------------
2024-11-27T05:56:09-05:00 [1:matomo1:backup3]                                                                       /srv/volumes/configs
2024-11-27T05:56:09-05:00 [1:matomo1:backup3]                                                                       /srv/volumes/matomo_plugins
2024-11-27T05:56:09-05:00 [1:matomo1:backup3] keep 1 snapshots:
2024-11-27T05:56:09-05:00 [1:matomo1:backup3] 1 snapshots
2024-11-27T05:56:09-05:00 [1:matomo1:backup3]                                                                       /srv/volumes/matomo_logs
2024-11-27T05:56:09-05:00 [1:matomo1:backup3]                                                                       /srv/state/matomo.sql
2024-11-27T05:56:09-05:00 [1:matomo1:backup3] 0024c5f2  2024-11-27 10:55:54  ns8                     last snapshot  /srv/state/environment
2024-11-27T05:56:09-05:00 [1:matomo1:backup3] ID        Time                 Host        Tags        Reasons        Paths
2024-11-27T05:56:10-05:00 [1:matomo1:backup3] restic stats --json latest
2024-11-27T05:56:11-05:00 [1:mariadb1:backup3] Files:           6 new,     0 changed,     0 unmodified
2024-11-27T05:56:11-05:00 [1:mariadb1:backup3] processed 6 files, 146.145 MiB in 0:02
2024-11-27T05:56:11-05:00 [1:mariadb1:backup3] snapshot bd1e6dfc saved
2024-11-27T05:56:11-05:00 [1:mariadb1:backup3] Added to the repository: 146.155 MiB (15.850 MiB stored)
2024-11-27T05:56:11-05:00 [1:mariadb1:backup3] Dirs:            6 new,     0 changed,     0 unmodified
2024-11-27T05:56:11-05:00 [1:mariadb1:backup3] restic forget --prune --keep-last=5
2024-11-27T05:56:12-05:00 [1:nextcloud1:backup3] restic snapshots
2024-11-27T05:56:12-05:00 [1:nextcloud1:backup3] s3:n4p9.m12.idrivee2-2.com/ns8/nextcloud/f88154ae-2520-4876-8295-6052234af97b
2024-11-27T05:56:12-05:00 [1:nextcloud1:backup3] Is there a repository at the following location?
2024-11-27T05:56:12-05:00 [1:nextcloud1:backup3] Fatal: unable to open config file: Stat: The specified key does not exist.
2024-11-27T05:56:12-05:00 [1:matomo1:backup3] 2024/11/27 10:56:12 Failed to rcat: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T05:56:12-05:00 [1:matomo1:backup3] 2024/11/27 10:56:12 ERROR : 5ded4a26-8c9d-43fd-b228-cb3f2f5462d4.json: Failed to copy: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T05:56:12-05:00 [1:matomo1:backup3] 2024/11/27 10:56:12 ERROR : 5ded4a26-8c9d-43fd-b228-cb3f2f5462d4.json: Post request rcat error: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T05:56:12-05:00 [1:matomo1:backup3] 	status code: 403, request id: 180BCDFE172C2B65, host id:
2024-11-27T05:56:12-05:00 [1:nextcloud1:backup3] Initializing repository 26f0223d-1dcc-591c-898d-874477683382 at path nextcloud/f88154ae-2520-4876-8295-6052234af97b
2024-11-27T05:56:12-05:00 [1:nextcloud1:backup3] restic init
2024-11-27T05:56:12-05:00 [1:matomo1:backup3] removed 'matomo.sql'
2024-11-27T05:56:13-05:00 [1:mariadb1:backup3]                                                                       /srv/state/restore/mysql.sql
2024-11-27T05:56:13-05:00 [1:mariadb1:backup3] ---------------------------------------------------------------------------------------------------------
2024-11-27T05:56:13-05:00 [1:mariadb1:backup3]                                                                       /srv/volumes/config.user.inc.php
2024-11-27T05:56:13-05:00 [1:mariadb1:backup3]                                                                       /srv/state/restore/phpmyadmin.sql
2024-11-27T05:56:13-05:00 [1:mariadb1:backup3]                                                                       /srv/volumes/mysql-conf.d
2024-11-27T05:56:13-05:00 [1:mariadb1:backup3] Applying Policy: keep 5 latest snapshots
2024-11-27T05:56:13-05:00 [1:mariadb1:backup3]                                                                       /srv/state/restore/sys.sql
2024-11-27T05:56:13-05:00 [1:mariadb1:backup3] bd1e6dfc  2024-11-27 10:56:08  ns8                     last snapshot  /srv/state/environment
2024-11-27T05:56:13-05:00 [1:mariadb1:backup3] ID        Time                 Host        Tags        Reasons        Paths
2024-11-27T05:56:13-05:00 [1:mariadb1:backup3] 1 snapshots
2024-11-27T05:56:13-05:00 [1:mariadb1:backup3]                                                                       /srv/state/secrets/passwords.secret
2024-11-27T05:56:13-05:00 [1:mariadb1:backup3]                                                                       /srv/state/restore/weewx.sql
2024-11-27T05:56:13-05:00 [1:mariadb1:backup3] keep 1 snapshots:
2024-11-27T05:56:13-05:00 [1:matomo1:systemd] backup3.service: Main process exited, code=exited, status=1/FAILURE
2024-11-27T05:56:13-05:00 [1:matomo1:systemd] backup3.service: Failed with result 'exit-code'.
2024-11-27T05:56:13-05:00 [1:matomo1:systemd] backup3.service: Consumed 3.337s CPU time.
2024-11-27T05:56:13-05:00 [1:matomo1:agent@matomo1] See "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.
2024-11-27T05:56:13-05:00 [1:matomo1:agent@matomo1] Job for backup3.service failed because the control process exited with error code.
2024-11-27T05:56:13-05:00 [1:mariadb1:backup3] restic stats --json latest
2024-11-27T05:56:15-05:00 [1:nextcloud1:backup3] irrecoverably lost.
2024-11-27T05:56:15-05:00 [1:nextcloud1:backup3] the repository. Losing your password means that your data is
2024-11-27T05:56:15-05:00 [1:nextcloud1:backup3] Please note that knowledge of your password is required to access
2024-11-27T05:56:15-05:00 [1:nextcloud1:backup3] created restic repository 5dd28b443a at s3:n4p9.m12.idrivee2-2.com/ns8/nextcloud/f88154ae-2520-4876-8295-6052234af97b
2024-11-27T05:56:15-05:00 [1:nextcloud1:backup3] restic backup state/environment --files-from=/etc/state-include.conf
2024-11-27T05:56:15-05:00 [1:mariadb1:backup3] 2024/11/27 10:56:15 ERROR : c3320cfd-3684-41fb-96d0-939215ca0599.json: Post request rcat error: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T05:56:15-05:00 [1:mariadb1:backup3] 	status code: 403, request id: 180BCDFEBD7F222F, host id:
2024-11-27T05:56:15-05:00 [1:mariadb1:backup3] 2024/11/27 10:56:15 Failed to rcat: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T05:56:15-05:00 [1:mariadb1:backup3] 2024/11/27 10:56:15 ERROR : c3320cfd-3684-41fb-96d0-939215ca0599.json: Failed to copy: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T05:56:15-05:00 [1:mariadb1:backup3] removed 'restore/phpmyadmin.sql'
2024-11-27T05:56:15-05:00 [1:mariadb1:backup3] removed directory 'restore/'
2024-11-27T05:56:15-05:00 [1:mariadb1:backup3] removed 'restore/sys.sql'
2024-11-27T05:56:15-05:00 [1:mariadb1:backup3] removed 'restore/mysql.sql'
2024-11-27T05:56:15-05:00 [1:mariadb1:backup3] removed 'restore/weewx.sql'
2024-11-27T05:56:15-05:00 [1:mariadb1:systemd] backup3.service: Main process exited, code=exited, status=1/FAILURE
2024-11-27T05:56:15-05:00 [1:mariadb1:systemd] backup3.service: Failed with result 'exit-code'.
2024-11-27T05:56:15-05:00 [1:mariadb1:systemd] backup3.service: Consumed 5.618s CPU time.
2024-11-27T05:56:15-05:00 [1:mariadb1:agent@mariadb1] See "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.
2024-11-27T05:56:15-05:00 [1:mariadb1:agent@mariadb1] Job for backup3.service failed because the control process exited with error code.
2024-11-27T05:56:16-05:00 [1:nextcloud1:backup3] no parent snapshot found, will read all files
2024-11-27T06:07:25-05:00 [1:mail1:backup3] Added to the repository: 29.094 GiB (18.184 GiB stored)
2024-11-27T06:07:25-05:00 [1:mail1:backup3] Dirs:         2519 new,     0 changed,     0 unmodified
2024-11-27T06:07:25-05:00 [1:mail1:backup3] processed 437629 files, 29.984 GiB in 11:26
2024-11-27T06:07:25-05:00 [1:mail1:backup3] Files:       437629 new,     0 changed,     0 unmodified
2024-11-27T06:07:25-05:00 [1:mail1:backup3] snapshot 9c0316e8 saved
2024-11-27T06:07:26-05:00 [1:mail1:backup3] restic forget --prune --keep-last=5
2024-11-27T06:07:27-05:00 [1:mail1:backup3] 1 snapshots
2024-11-27T06:07:27-05:00 [1:mail1:backup3]                                                                       /srv/state/environment
2024-11-27T06:07:27-05:00 [1:mail1:backup3]                                                                       /srv/state/freshclam.timer.is-enabled
2024-11-27T06:07:27-05:00 [1:mail1:backup3]                                                                       /srv/volumes/postfix-custom
2024-11-27T06:07:27-05:00 [1:mail1:backup3]                                                                       /srv/volumes/rspamd-data
2024-11-27T06:07:27-05:00 [1:mail1:backup3]                                                                       /srv/volumes/dovecot-dict
2024-11-27T06:07:27-05:00 [1:mail1:backup3]                                                                       /srv/volumes/rspamd-override
2024-11-27T06:07:27-05:00 [1:mail1:backup3] ID        Time                 Host        Tags        Reasons        Paths
2024-11-27T06:07:27-05:00 [1:mail1:backup3]                                                                       /srv/volumes/dovecot-data
2024-11-27T06:07:27-05:00 [1:mail1:backup3]                                                                       /srv/volumes/dovecot-custom
2024-11-27T06:07:27-05:00 [1:mail1:backup3] 9c0316e8  2024-11-27 10:55:58  ns8                     last snapshot  /srv/state/clamav.service.is-enabled
2024-11-27T06:07:27-05:00 [1:mail1:backup3]                                                                       /srv/volumes/rspamd-redis
2024-11-27T06:07:27-05:00 [1:mail1:backup3]                                                                       /srv/state/dovecot.env
2024-11-27T06:07:27-05:00 [1:mail1:backup3]                                                                       /srv/state/pcdb-dump.sql
2024-11-27T06:07:27-05:00 [1:mail1:backup3]                                                                       /srv/state/rspamd.env
2024-11-27T06:07:27-05:00 [1:mail1:backup3] -----------------------------------------------------------------------------------------------------------
2024-11-27T06:07:27-05:00 [1:mail1:backup3] Applying Policy: keep 5 latest snapshots
2024-11-27T06:07:27-05:00 [1:mail1:backup3] keep 1 snapshots:
2024-11-27T06:07:27-05:00 [1:mail1:backup3] restic stats --json latest
2024-11-27T06:07:38-05:00 [1:mail1:backup3] 	status code: 403, request id: 180BCE9DCD52A741, host id:
2024-11-27T06:07:38-05:00 [1:mail1:backup3] 2024/11/27 11:07:38 ERROR : 7af5cec7-c9aa-45d5-93fc-5ac4fbc2ffd8.json: Failed to copy: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T06:07:38-05:00 [1:mail1:backup3] 	status code: 403, request id: 180BCE9DCD52A741, host id:
2024-11-27T06:07:38-05:00 [1:mail1:backup3] 2024/11/27 11:07:38 ERROR : 7af5cec7-c9aa-45d5-93fc-5ac4fbc2ffd8.json: Post request rcat error: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T06:07:38-05:00 [1:mail1:backup3] 2024/11/27 11:07:38 Failed to rcat: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T06:07:38-05:00 [1:mail1:backup3] Cleaning up Mail state:
2024-11-27T06:07:38-05:00 [1:mail1:backup3] removed 'pcdb-dump.sql'
2024-11-27T06:07:38-05:00 [1:mail1:backup3] removed 'clamav.service.is-enabled'
2024-11-27T06:07:38-05:00 [1:mail1:backup3] removed 'freshclam.timer.is-enabled'
2024-11-27T06:07:39-05:00 [1:mail1:systemd] backup3.service: Main process exited, code=exited, status=1/FAILURE
2024-11-27T06:07:39-05:00 [1:mail1:systemd] backup3.service: Failed with result 'exit-code'.
2024-11-27T06:07:39-05:00 [1:mail1:systemd] backup3.service: Consumed 5.042s CPU time.
2024-11-27T06:07:39-05:00 [1:mail1:agent@mail1] See "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.
2024-11-27T06:07:39-05:00 [1:mail1:agent@mail1] Job for backup3.service failed because the control process exited with error code.
2024-11-27T07:18:53-05:00 [1:nextcloud1:backup3] processed 122473 files, 266.602 GiB in 1:22:37
2024-11-27T07:18:53-05:00 [1:nextcloud1:backup3] Dirs:        17556 new,     0 changed,     0 unmodified
2024-11-27T07:18:53-05:00 [1:nextcloud1:backup3] snapshot a0ff46f6 saved
2024-11-27T07:18:53-05:00 [1:nextcloud1:backup3] Added to the repository: 247.000 GiB (243.358 GiB stored)
2024-11-27T07:18:53-05:00 [1:nextcloud1:backup3] Files:       122473 new,     0 changed,     0 unmodified
2024-11-27T07:18:53-05:00 [1:nextcloud1:backup3] restic forget --prune --keep-last=5
2024-11-27T07:18:55-05:00 [1:nextcloud1:backup3] Applying Policy: keep 5 latest snapshots
2024-11-27T07:18:55-05:00 [1:nextcloud1:backup3]                                                                       /srv/state/environment
2024-11-27T07:18:55-05:00 [1:nextcloud1:backup3] 1 snapshots
2024-11-27T07:18:55-05:00 [1:nextcloud1:backup3]                                                                       /srv/volumes/nextcloud-app-data
2024-11-27T07:18:55-05:00 [1:nextcloud1:backup3] keep 1 snapshots:
2024-11-27T07:18:55-05:00 [1:nextcloud1:backup3] -----------------------------------------------------------------------------------------------------
2024-11-27T07:18:55-05:00 [1:nextcloud1:backup3]                                                                       /srv/state/restore/dump.sql
2024-11-27T07:18:55-05:00 [1:nextcloud1:backup3] a0ff46f6  2024-11-27 10:56:15  ns8                     last snapshot  /srv/state/config.json
2024-11-27T07:18:55-05:00 [1:nextcloud1:backup3] ID        Time                 Host        Tags        Reasons        Paths
2024-11-27T07:18:55-05:00 [1:nextcloud1:backup3] restic stats --json latest
2024-11-27T07:19:01-05:00 [1:nextcloud1:backup3] 	status code: 403, request id: 180BD28302F525BE, host id:
2024-11-27T07:19:01-05:00 [1:nextcloud1:backup3] 2024/11/27 12:19:01 ERROR : f88154ae-2520-4876-8295-6052234af97b.json: Failed to copy: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T07:19:01-05:00 [1:nextcloud1:backup3] 	status code: 403, request id: 180BD28302F525BE, host id:
2024-11-27T07:19:01-05:00 [1:nextcloud1:backup3] 2024/11/27 12:19:01 ERROR : f88154ae-2520-4876-8295-6052234af97b.json: Post request rcat error: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T07:19:01-05:00 [1:nextcloud1:backup3] 	status code: 403, request id: 180BD28302F525BE, host id:
2024-11-27T07:19:01-05:00 [1:nextcloud1:backup3] 2024/11/27 12:19:01 Failed to rcat: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T07:19:01-05:00 [1:nextcloud1:systemd] backup3.service: Main process exited, code=exited, status=1/FAILURE
2024-11-27T07:19:01-05:00 [1:nextcloud1:systemd] backup3.service: Failed with result 'exit-code'.
2024-11-27T07:19:01-05:00 [1:nextcloud1:systemd] backup3.service: Consumed 19.559s CPU time.
2024-11-27T07:19:01-05:00 [1:nextcloud1:agent@nextcloud1] Job for backup3.service failed because the control process exited with error code.
2024-11-27T07:19:01-05:00 [1:nextcloud1:agent@nextcloud1] See "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.
2024-11-27T07:19:02-05:00 [1::agent@cluster] Task module/joplin1/run-backup run failed: {'output': '', 'error': 'Job for backup3.service failed because the control process exited with error code.\nSee "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.\n', 'exit_code': 1}
2024-11-27T07:19:02-05:00 [1::agent@cluster] Task module/crowdsec1/run-backup run failed: {'output': '', 'error': 'Job for backup3-crowdsec1.service failed because the control process exited with error code.\nSee "systemctl status backup3-crowdsec1.service" and "journalctl -xeu backup3-crowdsec1.service" for details.\n', 'exit_code': 1}
2024-11-27T07:19:02-05:00 [1::agent@cluster] Task module/webtop1/run-backup run failed: {'output': '', 'error': 'Job for backup3.service failed because the control process exited with error code.\nSee "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.\n', 'exit_code': 1}
2024-11-27T07:19:02-05:00 [1::agent@cluster] Task module/dokuwiki1/run-backup run failed: {'output': '', 'error': 'Job for backup3.service failed because the control process exited with error code.\nSee "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.\n', 'exit_code': 1}
2024-11-27T07:19:02-05:00 [1::agent@cluster] Task module/grafana1/run-backup run failed: {'output': '', 'error': 'Job for backup3.service failed because the control process exited with error code.\nSee "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.\n', 'exit_code': 1}
2024-11-27T07:19:02-05:00 [1::agent@cluster] Task module/mail1/run-backup run failed: {'output': '', 'error': 'Job for backup3.service failed because the control process exited with error code.\nSee "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.\n', 'exit_code': 1}
2024-11-27T07:19:02-05:00 [1::agent@cluster] Task module/loki1/run-backup run failed: {'output': '', 'error': 'Job for backup3.service failed because the control process exited with error code.\nSee "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.\n', 'exit_code': 1}
2024-11-27T07:19:02-05:00 [1::agent@cluster] Task module/openldap1/run-backup run failed: {'output': '', 'error': 'Job for backup3.service failed because the control process exited with error code.\nSee "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.\n', 'exit_code': 1}
2024-11-27T07:19:02-05:00 [1::agent@cluster] Task module/lam1/run-backup run failed: {'output': '', 'error': 'Job for backup3.service failed because the control process exited with error code.\nSee "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.\n', 'exit_code': 1}
2024-11-27T07:19:02-05:00 [1::agent@cluster] Task module/collabora1/run-backup run failed: {'output': '', 'error': 'Job for backup3.service failed because the control process exited with error code.\nSee "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.\n', 'exit_code': 1}
2024-11-27T07:19:02-05:00 [1::agent@cluster] Task module/roundcubemail1/run-backup run failed: {'output': '', 'error': 'Job for backup3.service failed because the control process exited with error code.\nSee "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.\n', 'exit_code': 1}
2024-11-27T07:19:02-05:00 [1::agent@cluster] Task module/prometheus1/run-backup run failed: {'output': '', 'error': 'Job for backup3.service failed because the control process exited with error code.\nSee "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.\n', 'exit_code': 1}
2024-11-27T07:19:02-05:00 [1::agent@cluster] Task module/mariadb1/run-backup run failed: {'output': '', 'error': 'Job for backup3.service failed because the control process exited with error code.\nSee "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.\n', 'exit_code': 1}
2024-11-27T07:19:02-05:00 [1::agent@cluster] Task module/traefik1/run-backup run failed: {'output': '', 'error': 'Job for backup3.service failed because the control process exited with error code.\nSee "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.\n', 'exit_code': 1}
2024-11-27T07:19:02-05:00 [1::agent@cluster] Task module/nextcloud1/run-backup run failed: {'output': '', 'error': 'Job for backup3.service failed because the control process exited with error code.\nSee "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.\n', 'exit_code': 1}
2024-11-27T07:19:02-05:00 [1::agent@cluster] Task module/webserver1/run-backup run failed: {'output': '', 'error': 'Job for backup3.service failed because the control process exited with error code.\nSee "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.\n', 'exit_code': 1}
2024-11-27T07:19:02-05:00 [1::agent@cluster] Task module/matomo1/run-backup run failed: {'output': '', 'error': 'Job for backup3.service failed because the control process exited with error code.\nSee "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.\n', 'exit_code': 1}
2024-11-27T07:19:02-05:00 [1::agent@cluster] Task module/sogo1/run-backup run failed: {'output': '', 'error': 'Job for backup3.service failed because the control process exited with error code.\nSee "systemctl --user status backup3.service" and "journalctl --user -xeu backup3.service" for details.\n', 'exit_code': 1}

I’m seeing what seem to be authentication errors to the provider, but the credentials are definitely correct–this job sent ~250 GB of data to that provider.

2

As said in the other thread, the error seems to come from Rclone, whilst (I don’t know why and how) Restic succeeds to write the backup.

To investigate further what’s happening you may use the following commands from the app agent environment, e.g.:

runagent -m loki1 # get a shell
restic-wrapper --help
restic-wrapper --show
restic-wrapper --backup 1 stats latest
rclone-wrapper                   # prints help
rclone-wrapper 1 lsd REMOTE_PATH # write it exactly as-is

BTW restic-wrapper is useful to clean a stale lock on the remote repository.

3 
[loki1@ns8 state]$ restic-wrapper --show
Destinations:
- 4758961e-b9aa-5833-82b6-94ff3a69f285 DO Spaces (s3:nyc3.digitaloceanspaces.com/ns8-prod)
- 99299de6-f922-562a-965c-579265604a16 iDrive e2 (s3:n4p9.m12.idrivee2-2.com/ns8-prod)
Scheduled backups:
- 2 Backup to iDrive e2, destination UUID 99299de6-f922-562a-965c-579265604a16
- 4 Backup to DO Spaces, destination UUID 4758961e-b9aa-5833-82b6-94ff3a69f285
[loki1@ns8 state]$ restic-wrapper --backup 2 stats latest
restic stats latest
scanning...
Stats in restore-size mode:
     Snapshots processed:  1
        Total File Count:  1430
              Total Size:  61.695 MiB
[loki1@ns8 state]$ rclone-wrapper 2 lsd REMOTE_PATH
           0 2000-01-01 00:00:00        -1 collabora
           0 2000-01-01 00:00:00        -1 crowdsec
           0 2000-01-01 00:00:00        -1 dokuwiki
           0 2000-01-01 00:00:00        -1 grafana
           0 2000-01-01 00:00:00        -1 joplin
           0 2000-01-01 00:00:00        -1 lam
           0 2000-01-01 00:00:00        -1 loki
           0 2000-01-01 00:00:00        -1 mail
           0 2000-01-01 00:00:00        -1 mariadb
           0 2000-01-01 00:00:00        -1 matomo
           0 2000-01-01 00:00:00        -1 nextcloud
           0 2000-01-01 00:00:00        -1 openldap
           0 2000-01-01 00:00:00        -1 prometheus
           0 2000-01-01 00:00:00        -1 roundcubemail
           0 2000-01-01 00:00:00        -1 sogo
           0 2000-01-01 00:00:00        -1 traefik
           0 2000-01-01 00:00:00        -1 webserver
           0 2000-01-01 00:00:00        -1 webtop
[loki1@ns8 state]$

The backup to Digital Ocean Spaces, which I set up about an hour ago, completes without issues. They’re pretty expensive, though.

4

The Access Denied error can be misleading. For other providers there was simply a bad Rclone configuration. Can you share the destination configuration?

 api-cli run list-backup-repositories

Remove any sensitive data

5 
[root@ns8 ~]# api-cli run list-backup-repositories
Warning: using user "cluster" credentials from the environment
{"repositories": [{"id": "4758961e-b9aa-5833-82b6-94ff3a69f285", "provider": "generic-s3", "name": "DO Spaces", "url": "s3:nyc3.digitaloceanspaces.com/ns8-prod", "password": "(redacted))", "parameters": {"aws_access_key_id": "(redacted)", "aws_secret_access_key": "(redacxted)"}}, {"id": "99299de6-f922-562a-965c-579265604a16", "provider": "generic-s3", "name": "iDrive e2", "url": "s3:n4p9.m12.idrivee2-2.com/ns8-prod", "password": "(redacted)", "parameters": {"aws_access_key_id": "(redacted)", "aws_secret_access_key": "(redacted)"}}], "password_exists": true}
6

Rclone has a dedicated driver for iDrive: https://rclone.org/s3/#idrive-e2.

You can try to patch /usr/local/agent/bin/rclone-wrapper. I guess it now falls back to S3 “Other” driver which doesn’t suit your case. I’ll be happy to fix that script! Thanks in advance!

7

So I guess that’d look something like:

    elif orepo['provider'] == 'generic-s3' and 'idrivee2' in s3_endpoint:
        rclone_env['RCLONE_S3_PROVIDER'] = 'IDrive'

?

1 Like
8

Added those lines to rclone-wrapper, but I’m still seeing what appear to be the same errors:

2024-11-27T09:59:37-05:00 [1:nextcloud1:backup2] restic backup state/environment --files-from=/etc/state-include.conf
2024-11-27T09:59:38-05:00 [1:mariadb1:backup2] 2024/11/27 14:59:38 ERROR : c3320cfd-3684-41fb-96d0-939215ca0599.json: Failed to copy: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T09:59:38-05:00 [1:mariadb1:backup2] 	status code: 403, request id: 180BDB46D4419D1F, host id:
2024-11-27T09:59:38-05:00 [1:mariadb1:backup2] 2024/11/27 14:59:38 Failed to rcat: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T09:59:38-05:00 [1:mariadb1:backup2] 2024/11/27 14:59:38 ERROR : c3320cfd-3684-41fb-96d0-939215ca0599.json: Post request rcat error: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T09:59:39-05:00 [1:mariadb1:backup2] removed 'restore/sys.sql'
2024-11-27T09:59:39-05:00 [1:mariadb1:backup2] removed 'restore/mysql.sql'
2024-11-27T09:59:39-05:00 [1:mariadb1:backup2] removed 'restore/weewx.sql'
2024-11-27T09:59:39-05:00 [1:mariadb1:backup2] removed 'restore/phpmyadmin.sql'
2024-11-27T09:59:39-05:00 [1:mariadb1:backup2] removed directory 'restore/'
2024-11-27T09:59:39-05:00 [1:mariadb1:systemd] backup2.service: Main process exited, code=exited, status=1/FAILURE
2024-11-27T09:59:39-05:00 [1:mariadb1:systemd] backup2.service: Failed with result 'exit-code'.
2024-11-27T09:59:39-05:00 [1:mariadb1:systemd] backup2.service: Consumed 4.897s CPU time.
2024-11-27T09:59:39-05:00 [1:mariadb1:agent@mariadb1] Job for backup2.service failed because the control process exited with error code.
2024-11-27T09:59:39-05:00 [1:mariadb1:agent@mariadb1] See "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.
2024-11-27T09:59:39-05:00 [1:nextcloud1:backup2] using parent snapshot 653960d5
2024-11-27T09:59:50-05:00 [1:nextcloud1:backup2] Added to the repository: 14.453 MiB (4.278 MiB stored)
2024-11-27T09:59:50-05:00 [1:nextcloud1:backup2] processed 122473 files, 266.602 GiB in 0:11
2024-11-27T09:59:50-05:00 [1:nextcloud1:backup2] Dirs:            0 new,    11 changed, 17545 unmodified
2024-11-27T09:59:50-05:00 [1:nextcloud1:backup2] snapshot 5ce5d0b3 saved
2024-11-27T09:59:50-05:00 [1:nextcloud1:backup2] Files:           1 new,     7 changed, 122465 unmodified
2024-11-27T09:59:50-05:00 [1:nextcloud1:backup2] restic forget --prune --keep-last=5
2024-11-27T09:59:51-05:00 [1:nextcloud1:backup2] repo already locked, waiting up to 0s for the lock
2024-11-27T09:59:51-05:00 [1:nextcloud1:backup2] the `unlock` command can be used to remove stale locks
2024-11-27T09:59:51-05:00 [1:nextcloud1:backup2] lock was created at 2024-11-22 16:07:00 (118h52m50.857569728s ago)
2024-11-27T09:59:51-05:00 [1:nextcloud1:backup2] storage ID c3f432bf
2024-11-27T09:59:51-05:00 [1:nextcloud1:backup2] unable to create lock in backend: repository is already locked by PID 1 on ns8.familybrown.org by root (UID 0, GID 0)
2024-11-27T09:59:51-05:00 [1:nextcloud1:backup2] [ERROR] restic backup failed. Command '['podman', 'run', '-i', '--rm', '--name=restic-nextcloud1-535745', '--privileged', '--network=host', '--volume=restic-cache:/var/cache/restic', '--log-driver=none', '-e', 'RESTIC_PASSWORD', '-e', 'RESTIC_CACHE_DIR', '-e', 'RESTIC_REPOSITORY', '-e', 'AWS_ACCESS_KEY_ID', '-e', 'AWS_SECRET_ACCESS_KEY', 'ghcr.io/nethserver/restic:3.2.3', 'forget', '--prune', '--keep-last=5']' returned non-zero exit status 1.
2024-11-27T09:59:51-05:00 [1:nextcloud1:backup2] restic stats --json latest
2024-11-27T10:00:03-05:00 [1:mail1:backup2] Files:         309 new,   219 changed, 437208 unmodified
2024-11-27T10:00:03-05:00 [1:mail1:backup2] processed 437736 files, 29.992 GiB in 0:41
2024-11-27T10:00:03-05:00 [1:mail1:backup2] snapshot 6c08904a saved
2024-11-27T10:00:03-05:00 [1:mail1:backup2] Added to the repository: 142.865 MiB (76.142 MiB stored)
2024-11-27T10:00:03-05:00 [1:mail1:backup2] Dirs:            0 new,   227 changed,  2292 unmodified
2024-11-27T10:00:03-05:00 [1:mail1:backup2] restic forget --prune --keep-last=5
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-data
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-override
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/state/pcdb-dump.sql
2024-11-27T10:00:05-05:00 [1:mail1:backup2] Applying Policy: keep 5 latest snapshots
2024-11-27T10:00:05-05:00 [1:mail1:backup2] 192684ce  2024-11-25 04:00:06  ns8                     last snapshot  /srv/state/clamav.service.is-enabled
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-data
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/postfix-custom
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-redis
2024-11-27T10:00:05-05:00 [1:mail1:backup2] 650754eb  2024-11-26 04:00:26  ns8                     last snapshot  /srv/state/clamav.service.is-enabled
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/state/rspamd.env
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-dict
2024-11-27T10:00:05-05:00 [1:mail1:backup2] -----------------------------------------------------------------------------------------------------------
2024-11-27T10:00:05-05:00 [1:mail1:backup2] keep 5 snapshots:
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/state/environment
2024-11-27T10:00:05-05:00 [1:mail1:backup2] ID        Time                 Host        Tags        Reasons        Paths
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/state/dovecot.env
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/state/freshclam.timer.is-enabled
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-custom
2024-11-27T10:00:05-05:00 [1:mail1:backup2] a0652233  2024-11-26 13:57:56  ns8                     last snapshot  /srv/state/clamav.service.is-enabled
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-redis
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/state/dovecot.env
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/postfix-custom
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-dict
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-override
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/state/environment
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/state/freshclam.timer.is-enabled
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/state/pcdb-dump.sql
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-custom
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-data
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/state/rspamd.env
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-data
2024-11-27T10:00:05-05:00 [1:mail1:backup2] 6c08904a  2024-11-27 14:59:20  ns8                     last snapshot  /srv/state/clamav.service.is-enabled
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/state/rspamd.env
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/state/environment
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/state/pcdb-dump.sql
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/state/dovecot.env
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-data
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-redis
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/postfix-custom
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-data
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-override
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-dict
2024-11-27T10:00:05-05:00 [1:mail1:backup2] 148f007a  2024-11-27 04:00:06  ns8                     last snapshot  /srv/state/clamav.service.is-enabled
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-custom
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/state/freshclam.timer.is-enabled
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-override
2024-11-27T10:00:05-05:00 [1:mail1:backup2] 1 snapshots
2024-11-27T10:00:05-05:00 [1:mail1:backup2] remove 1 snapshots:
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-data
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-redis
2024-11-27T10:00:05-05:00 [1:mail1:backup2] keep 1 snapshots:
2024-11-27T10:00:05-05:00 [1:mail1:backup2] -----------------------------------------------------------------------------------------------------------
2024-11-27T10:00:05-05:00 [1:mail1:backup2] 5 snapshots
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/postfix-custom
2024-11-27T10:00:05-05:00 [1:mail1:backup2] --------------------------------------------------------------------------------------------
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                        /srv/volumes/rspamd-redis
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                        /srv/volumes/rspamd-override
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                        /srv/state/dovecot.env
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                        /srv/volumes/rspamd-data
2024-11-27T10:00:05-05:00 [1:mail1:backup2] a86352da  2024-11-24 04:00:08  ns8                     /srv/state/clamav.service.is-enabled
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                        /srv/volumes/dovecot-dict
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                        /srv/state/environment
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                        /srv/volumes/dovecot-data
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                        /srv/volumes/dovecot-custom
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                        /srv/state/freshclam.timer.is-enabled
2024-11-27T10:00:05-05:00 [1:mail1:backup2] ID        Time                 Host        Tags        Paths
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-dict
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                        /srv/state/rspamd.env
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                        /srv/state/pcdb-dump.sql
2024-11-27T10:00:05-05:00 [1:mail1:backup2] ID        Time                 Host        Tags        Reasons        Paths
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                        /srv/volumes/postfix-custom
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/state/environment
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/state/freshclam.timer.is-enabled
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-redis
2024-11-27T10:00:05-05:00 [1:mail1:backup2] -----------------------------------------------------------------------------------------------------------
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/state/pcdb-dump.sql
2024-11-27T10:00:05-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-data
2024-11-27T10:00:05-05:00 [1:mail1:backup2] 1 snapshots
2024-11-27T10:00:05-05:00 [1:mail1:backup2] e36e1956  2024-11-22 12:56:36  ns8                     last snapshot  /srv/state/clamav.service.is-enabled
2024-11-27T10:00:05-05:00 [1:mail1:backup2] [0:00] 100.00%  1 / 1 files deleted
2024-11-27T10:00:05-05:00 [1:mail1:backup2] loading indexes...
2024-11-27T10:00:05-05:00 [1:mail1:backup2] 1 snapshots have been removed, running prune
2024-11-27T10:00:05-05:00 [1:mail1:backup2] loading all snapshots...
2024-11-27T10:00:05-05:00 [1:mail1:backup2] finding data that is still in use for 6 snapshots
2024-11-27T10:00:07-05:00 [1:nextcloud1:backup2] 	status code: 403, request id: 180BDB4D709EAD60, host id:
2024-11-27T10:00:07-05:00 [1:nextcloud1:backup2] 2024/11/27 15:00:07 ERROR : f88154ae-2520-4876-8295-6052234af97b.json: Failed to copy: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T10:00:07-05:00 [1:nextcloud1:backup2] 2024/11/27 15:00:07 ERROR : f88154ae-2520-4876-8295-6052234af97b.json: Post request rcat error: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T10:00:07-05:00 [1:nextcloud1:backup2] 2024/11/27 15:00:07 Failed to rcat: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T10:00:07-05:00 [1:nextcloud1:systemd] backup2.service: Main process exited, code=exited, status=1/FAILURE
2024-11-27T10:00:07-05:00 [1:nextcloud1:systemd] backup2.service: Failed with result 'exit-code'.
2024-11-27T10:00:07-05:00 [1:nextcloud1:systemd] backup2.service: Consumed 8.457s CPU time.
2024-11-27T10:00:07-05:00 [1:nextcloud1:agent@nextcloud1] Job for backup2.service failed because the control process exited with error code.
2024-11-27T10:00:07-05:00 [1:nextcloud1:agent@nextcloud1] See "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.
2024-11-27T10:00:08-05:00 [1:mail1:backup2] [0:02] 100.00%  6 / 6 snapshots
2024-11-27T10:00:08-05:00 [1:mail1:backup2] searching used packs...
2024-11-27T10:00:08-05:00 [1:mail1:backup2] collecting packs for deletion and repacking
2024-11-27T10:00:09-05:00 [1:mail1:backup2] [0:00] 100.00%  1158 / 1158 packs processed
2024-11-27T10:00:09-05:00 [1:mail1:backup2] to repack:           109 blobs / 9.398 MiB
2024-11-27T10:00:09-05:00 [1:mail1:backup2] this removes:         62 blobs / 4.693 MiB
2024-11-27T10:00:09-05:00 [1:mail1:backup2] to delete:            32 blobs / 38.628 MiB
2024-11-27T10:00:09-05:00 [1:mail1:backup2] total prune:          94 blobs / 43.321 MiB
2024-11-27T10:00:09-05:00 [1:mail1:backup2] remaining:        446137 blobs / 18.558 GiB
2024-11-27T10:00:09-05:00 [1:mail1:backup2] unused size after prune: 44.513 MiB (0.23% of remaining size)
2024-11-27T10:00:09-05:00 [1:mail1:backup2] repacking packs
2024-11-27T10:00:11-05:00 [1:mail1:backup2] [0:02] 100.00%  2 / 2 packs repacked
2024-11-27T10:00:11-05:00 [1:mail1:backup2] rebuilding index
2024-11-27T10:00:17-05:00 [1:mail1:backup2] [0:06] 100.00%  1154 / 1154 packs processed
2024-11-27T10:00:17-05:00 [1:mail1:backup2] deleting obsolete index files
2024-11-27T10:00:17-05:00 [1:mail1:backup2] [0:00] 100.00%  4 / 4 files deleted
2024-11-27T10:00:17-05:00 [1:mail1:backup2] removing 5 old packs
2024-11-27T10:00:17-05:00 [1:mail1:backup2] done
2024-11-27T10:00:17-05:00 [1:mail1:backup2] [0:00] 100.00%  5 / 5 files deleted
2024-11-27T10:00:17-05:00 [1:mail1:backup2] restic stats --json latest
2024-11-27T10:00:28-05:00 [1:mail1:backup2] 	status code: 403, request id: 180BDB5271CC8032, host id:
2024-11-27T10:00:28-05:00 [1:mail1:backup2] 2024/11/27 15:00:28 Failed to rcat: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T10:00:28-05:00 [1:mail1:backup2] 2024/11/27 15:00:28 ERROR : 7af5cec7-c9aa-45d5-93fc-5ac4fbc2ffd8.json: Failed to copy: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T10:00:28-05:00 [1:mail1:backup2] 2024/11/27 15:00:28 ERROR : 7af5cec7-c9aa-45d5-93fc-5ac4fbc2ffd8.json: Post request rcat error: failed to prepare upload: AccessDenied: Access Denied.
2024-11-27T10:00:28-05:00 [1:mail1:backup2] Cleaning up Mail state:
2024-11-27T10:00:28-05:00 [1:mail1:backup2] removed 'pcdb-dump.sql'
2024-11-27T10:00:28-05:00 [1:mail1:backup2] removed 'clamav.service.is-enabled'
2024-11-27T10:00:28-05:00 [1:mail1:backup2] removed 'freshclam.timer.is-enabled'
2024-11-27T10:00:29-05:00 [1:mail1:systemd] backup2.service: Main process exited, code=exited, status=1/FAILURE
2024-11-27T10:00:29-05:00 [1:mail1:systemd] backup2.service: Failed with result 'exit-code'.
2024-11-27T10:00:29-05:00 [1:mail1:systemd] backup2.service: Consumed 3.119s CPU time.
2024-11-27T10:00:29-05:00 [1:mail1:agent@mail1] Job for backup2.service failed because the control process exited with error code.
2024-11-27T10:00:29-05:00 [1:mail1:agent@mail1] See "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.
2024-11-27T10:00:29-05:00 [1::agent@cluster] Task module/mail1/run-backup run failed: {'output': '', 'error': 'Job for backup2.service failed because the control process exited with error code.\nSee "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.\n', 'exit_code': 1}
2024-11-27T10:00:29-05:00 [1::agent@cluster] Task module/collabora1/run-backup run failed: {'output': '', 'error': 'Job for backup2.service failed because the control process exited with error code.\nSee "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.\n', 'exit_code': 1}
2024-11-27T10:00:29-05:00 [1::agent@cluster] Task module/loki1/run-backup run failed: {'output': '', 'error': 'Job for backup2.service failed because the control process exited with error code.\nSee "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.\n', 'exit_code': 1}
2024-11-27T10:00:29-05:00 [1::agent@cluster] Task module/webtop1/run-backup run failed: {'output': '', 'error': 'Job for backup2.service failed because the control process exited with error code.\nSee "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.\n', 'exit_code': 1}
2024-11-27T10:00:29-05:00 [1::agent@cluster] Task module/crowdsec1/run-backup run failed: {'output': '', 'error': 'Job for backup2-crowdsec1.service failed because the control process exited with error code.\nSee "systemctl status backup2-crowdsec1.service" and "journalctl -xeu backup2-crowdsec1.service" for details.\n', 'exit_code': 1}
2024-11-27T10:00:29-05:00 [1::agent@cluster] Task module/dokuwiki1/run-backup run failed: {'output': '', 'error': 'Job for backup2.service failed because the control process exited with error code.\nSee "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.\n', 'exit_code': 1}
2024-11-27T10:00:29-05:00 [1::agent@cluster] Task module/grafana1/run-backup run failed: {'output': '', 'error': 'Job for backup2.service failed because the control process exited with error code.\nSee "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.\n', 'exit_code': 1}
2024-11-27T10:00:29-05:00 [1::agent@cluster] Task module/joplin1/run-backup run failed: {'output': '', 'error': 'Job for backup2.service failed because the control process exited with error code.\nSee "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.\n', 'exit_code': 1}
2024-11-27T10:00:29-05:00 [1::agent@cluster] Task module/lam1/run-backup run failed: {'output': '', 'error': 'Job for backup2.service failed because the control process exited with error code.\nSee "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.\n', 'exit_code': 1}
2024-11-27T10:00:29-05:00 [1::agent@cluster] Task module/traefik1/run-backup run failed: {'output': '', 'error': 'Job for backup2.service failed because the control process exited with error code.\nSee "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.\n', 'exit_code': 1}
2024-11-27T10:00:29-05:00 [1::agent@cluster] Task module/sogo1/run-backup run failed: {'output': '', 'error': 'Job for backup2.service failed because the control process exited with error code.\nSee "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.\n', 'exit_code': 1}
2024-11-27T10:00:29-05:00 [1::agent@cluster] Task module/mariadb1/run-backup run failed: {'output': '', 'error': 'Job for backup2.service failed because the control process exited with error code.\nSee "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.\n', 'exit_code': 1}
2024-11-27T10:00:29-05:00 [1::agent@cluster] Task module/roundcubemail1/run-backup run failed: {'output': '', 'error': 'Job for backup2.service failed because the control process exited with error code.\nSee "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.\n', 'exit_code': 1}
2024-11-27T10:00:29-05:00 [1::agent@cluster] Task module/prometheus1/run-backup run failed: {'output': '', 'error': 'Job for backup2.service failed because the control process exited with error code.\nSee "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.\n', 'exit_code': 1}
2024-11-27T10:00:29-05:00 [1::agent@cluster] Task module/openldap1/run-backup run failed: {'output': '', 'error': 'Job for backup2.service failed because the control process exited with error code.\nSee "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.\n', 'exit_code': 1}
2024-11-27T10:00:29-05:00 [1::agent@cluster] Task module/webserver1/run-backup run failed: {'output': '', 'error': 'Job for backup2.service failed because the control process exited with error code.\nSee "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.\n', 'exit_code': 1}
2024-11-27T10:00:29-05:00 [1::agent@cluster] Task module/matomo1/run-backup run failed: {'output': '', 'error': 'Job for backup2.service failed because the control process exited with error code.\nSee "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.\n', 'exit_code': 1}
2024-11-27T10:00:29-05:00 [1::agent@cluster] Task module/nextcloud1/run-backup run failed: {'output': '', 'error': 'Job for backup2.service failed because the control process exited with error code.\nSee "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.\n', 'exit_code': 1}
9

No idea for now about what is it missing. Just to give you another useful command to run the backup procedure during experiments:

runagent -m loki1 module-backup 1    # run backup ID 1 for loki1
10

What’s strange, and a little frustrating, is that the backup worked for at least a couple of days. I made no change to the backup configuration, but it then stopped working. At one point, it was successfully backing up everything but Nextcloud; now it isn’t backing up anything at all (to iDrive; it does to DO Spaces).

11

If access key does not have ALL buckets access, you must add the following to your configuration (As the limited access key does not have bucket creation permissions): no_check_bucket = true Optionally instead of the config line, you may use the following flag: --s3-no-check-bucket

Maybe we are missing a “no_check_bucket” option for Rclone. See Rclone - IDrive® e2 official documentation. Server side encryption must be disabled (data is already encrypted by Restic).

Please try the following patch:

diff --git a/core/imageroot/usr/local/agent/bin/rclone-wrapper b/core/imageroot/usr/local/agent/bin/rclone-wrapper
index abcfafc3..6585168b 100755
--- a/core/imageroot/usr/local/agent/bin/rclone-wrapper
+++ b/core/imageroot/usr/local/agent/bin/rclone-wrapper
@@ -81,6 +81,9 @@ elif uscheme == "s3":
         rclone_env['RCLONE_S3_PROVIDER'] = 'Synology'
         rclone_env['RCLONE_S3_REGION'] = orepo.get("aws_default_region", extract_region_code(s3_endpoint, 0))
         rclone_env['RCLONE_S3_NO_CHECK_BUCKET'] = "1"
+    elif orepo['provider'] == 'generic-s3' and 'idrivee2' in s3_endpoint:
+        rclone_env['RCLONE_S3_PROVIDER'] = 'IDrive'
+        rclone_env['RCLONE_S3_NO_CHECK_BUCKET'] = "1"
     else:
         rclone_env['RCLONE_S3_PROVIDER'] = 'Other'
12

It’s an improvement. Previously, all apps’ backup failed; now only mail and nextcloud do. Logs:

2024-11-28T06:22:58-05:00 [1:nextcloud1:backup2] Repository 99299de6-f922-562a-965c-579265604a16 is present at path nextcloud/f88154ae-2520-4876-8295-6052234af97b
2024-11-28T06:22:58-05:00 [1:nextcloud1:backup2] restic backup state/environment --files-from=/etc/state-include.conf
2024-11-28T06:23:00-05:00 [1:nextcloud1:backup2] using parent snapshot f73547be
2024-11-28T06:23:00-05:00 [1:mariadb1:backup2] removed 'restore/weewx.sql'
2024-11-28T06:23:00-05:00 [1:mariadb1:backup2] removed 'restore/sys.sql'
2024-11-28T06:23:00-05:00 [1:mariadb1:backup2] removed 'restore/phpmyadmin.sql'
2024-11-28T06:23:00-05:00 [1:mariadb1:backup2] removed 'restore/mysql.sql'
2024-11-28T06:23:00-05:00 [1:mariadb1:backup2] removed directory 'restore/'
2024-11-28T06:23:00-05:00 [1:mariadb1:systemd] backup2.service: Consumed 5.124s CPU time.
2024-11-28T06:23:11-05:00 [1:nextcloud1:backup2] Dirs:            0 new,     3 changed, 17553 unmodified
2024-11-28T06:23:11-05:00 [1:nextcloud1:backup2] Files:           0 new,     1 changed, 122472 unmodified
2024-11-28T06:23:11-05:00 [1:nextcloud1:backup2] Added to the repository: 11.212 MiB (3.323 MiB stored)
2024-11-28T06:23:11-05:00 [1:nextcloud1:backup2] processed 122473 files, 266.602 GiB in 0:12
2024-11-28T06:23:11-05:00 [1:nextcloud1:backup2] snapshot a4b44ac5 saved
2024-11-28T06:23:11-05:00 [1:nextcloud1:backup2] restic forget --prune --keep-last=5
2024-11-28T06:23:12-05:00 [1:nextcloud1:backup2] repo already locked, waiting up to 0s for the lock
2024-11-28T06:23:13-05:00 [1:nextcloud1:backup2] lock was created at 2024-11-22 16:07:00 (139h16m12.177727801s ago)
2024-11-28T06:23:13-05:00 [1:nextcloud1:backup2] storage ID c3f432bf
2024-11-28T06:23:13-05:00 [1:nextcloud1:backup2] the `unlock` command can be used to remove stale locks
2024-11-28T06:23:13-05:00 [1:nextcloud1:backup2] unable to create lock in backend: repository is already locked by PID 1 on ns8.familybrown.org by root (UID 0, GID 0)
2024-11-28T06:23:13-05:00 [1:nextcloud1:backup2] [ERROR] restic backup failed. Command '['podman', 'run', '-i', '--rm', '--name=restic-nextcloud1-613312', '--privileged', '--network=host', '--volume=restic-cache:/var/cache/restic', '--log-driver=none', '-e', 'RESTIC_PASSWORD', '-e', 'RESTIC_CACHE_DIR', '-e', 'RESTIC_REPOSITORY', '-e', 'AWS_ACCESS_KEY_ID', '-e', 'AWS_SECRET_ACCESS_KEY', 'ghcr.io/nethserver/restic:3.2.3', 'forget', '--prune', '--keep-last=5']' returned non-zero exit status 1.
2024-11-28T06:23:13-05:00 [1:nextcloud1:backup2] restic stats --json latest
2024-11-28T06:23:19-05:00 [1:mail1:backup2] Added to the repository: 59.527 MiB (29.474 MiB stored)
2024-11-28T06:23:19-05:00 [1:mail1:backup2] snapshot 8787d574 saved
2024-11-28T06:23:19-05:00 [1:mail1:backup2] processed 438059 files, 30.024 GiB in 0:37
2024-11-28T06:23:19-05:00 [1:mail1:backup2] Dirs:            0 new,   232 changed,  2287 unmodified
2024-11-28T06:23:19-05:00 [1:mail1:backup2] Files:          51 new,    41 changed, 437967 unmodified
2024-11-28T06:23:19-05:00 [1:mail1:backup2] restic forget --prune --keep-last=5
2024-11-28T06:23:20-05:00 [1:nextcloud1:systemd] backup2.service: Main process exited, code=exited, status=1/FAILURE
2024-11-28T06:23:20-05:00 [1:nextcloud1:systemd] backup2.service: Failed with result 'exit-code'.
2024-11-28T06:23:20-05:00 [1:nextcloud1:systemd] backup2.service: Consumed 8.565s CPU time.
2024-11-28T06:23:20-05:00 [1:nextcloud1:agent@nextcloud1] See "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.
2024-11-28T06:23:20-05:00 [1:nextcloud1:agent@nextcloud1] Job for backup2.service failed because the control process exited with error code.
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-dict
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-data
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-custom
2024-11-28T06:23:20-05:00 [1:mail1:backup2] -----------------------------------------------------------------------------------------------------------
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-redis
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/state/rspamd.env
2024-11-28T06:23:20-05:00 [1:mail1:backup2] ID        Time                 Host        Tags        Reasons        Paths
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/state/freshclam.timer.is-enabled
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/state/pcdb-dump.sql
2024-11-28T06:23:20-05:00 [1:mail1:backup2] 148f007a  2024-11-27 04:00:06  ns8                     last snapshot  /srv/state/clamav.service.is-enabled
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-data
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/postfix-custom
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/state/environment
2024-11-28T06:23:20-05:00 [1:mail1:backup2] keep 5 snapshots:
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/state/dovecot.env
2024-11-28T06:23:20-05:00 [1:mail1:backup2] a0652233  2024-11-26 13:57:56  ns8                     last snapshot  /srv/state/clamav.service.is-enabled
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-override
2024-11-28T06:23:20-05:00 [1:mail1:backup2] Applying Policy: keep 5 latest snapshots
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-data
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-custom
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-override
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/state/freshclam.timer.is-enabled
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-dict
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-data
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-redis
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/state/rspamd.env
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/state/environment
2024-11-28T06:23:20-05:00 [1:mail1:backup2] 6c08904a  2024-11-27 14:59:20  ns8                     last snapshot  /srv/state/clamav.service.is-enabled
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/state/dovecot.env
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/state/pcdb-dump.sql
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/postfix-custom
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/postfix-custom
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-dict
2024-11-28T06:23:20-05:00 [1:mail1:backup2] 2cac7bc0  2024-11-28 04:00:06  ns8                     last snapshot  /srv/state/clamav.service.is-enabled
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/state/pcdb-dump.sql
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-data
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/state/freshclam.timer.is-enabled
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-custom
2024-11-28T06:23:20-05:00 [1:mail1:backup2] 8787d574  2024-11-28 11:22:41  ns8                     last snapshot  /srv/state/clamav.service.is-enabled
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-override
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/state/environment
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/state/dovecot.env
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/state/rspamd.env
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-data
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-redis
2024-11-28T06:23:20-05:00 [1:mail1:backup2] ID        Time                 Host        Tags        Reasons        Paths
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-dict
2024-11-28T06:23:20-05:00 [1:mail1:backup2] 650754eb  2024-11-26 04:00:26  ns8                     /srv/state/clamav.service.is-enabled
2024-11-28T06:23:20-05:00 [1:mail1:backup2] 5 snapshots
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/postfix-custom
2024-11-28T06:23:20-05:00 [1:mail1:backup2] 1 snapshots
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                        /srv/volumes/postfix-custom
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-data
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-override
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-redis
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                        /srv/state/dovecot.env
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                        /srv/state/environment
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                        /srv/state/freshclam.timer.is-enabled
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                        /srv/state/pcdb-dump.sql
2024-11-28T06:23:20-05:00 [1:mail1:backup2] -----------------------------------------------------------------------------------------------------------
2024-11-28T06:23:20-05:00 [1:mail1:backup2] --------------------------------------------------------------------------------------------
2024-11-28T06:23:20-05:00 [1:mail1:backup2] keep 1 snapshots:
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                        /srv/state/rspamd.env
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                        /srv/volumes/rspamd-redis
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                        /srv/volumes/rspamd-override
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                        /srv/volumes/dovecot-custom
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                        /srv/volumes/dovecot-data
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                        /srv/volumes/dovecot-dict
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                        /srv/volumes/rspamd-data
2024-11-28T06:23:20-05:00 [1:mail1:backup2] ID        Time                 Host        Tags        Paths
2024-11-28T06:23:20-05:00 [1:mail1:backup2] remove 1 snapshots:
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/state/environment
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/state/pcdb-dump.sql
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/rspamd-redis
2024-11-28T06:23:20-05:00 [1:mail1:backup2] 1 snapshots
2024-11-28T06:23:20-05:00 [1:mail1:backup2] -----------------------------------------------------------------------------------------------------------
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/state/freshclam.timer.is-enabled
2024-11-28T06:23:20-05:00 [1:mail1:backup2]                                                                       /srv/volumes/dovecot-data
2024-11-28T06:23:20-05:00 [1:mail1:backup2] e36e1956  2024-11-22 12:56:36  ns8                     last snapshot  /srv/state/clamav.service.is-enabled
2024-11-28T06:23:20-05:00 [1:mail1:backup2] 1 snapshots have been removed, running prune
2024-11-28T06:23:20-05:00 [1:mail1:backup2] [0:00] 100.00%  1 / 1 files deleted
2024-11-28T06:23:20-05:00 [1:mail1:backup2] loading indexes...
2024-11-28T06:23:21-05:00 [1:mail1:backup2] loading all snapshots...
2024-11-28T06:23:21-05:00 [1:mail1:backup2] finding data that is still in use for 6 snapshots
2024-11-28T06:23:24-05:00 [1:mail1:backup2] [0:02] 100.00%  6 / 6 snapshots
2024-11-28T06:23:24-05:00 [1:mail1:backup2] searching used packs...
2024-11-28T06:23:24-05:00 [1:mail1:backup2] collecting packs for deletion and repacking
2024-11-28T06:23:24-05:00 [1:mail1:backup2] [0:00] 100.00%  1158 / 1158 packs processed
2024-11-28T06:23:25-05:00 [1:mail1:backup2] this removes:         43 blobs / 4.063 MiB
2024-11-28T06:23:25-05:00 [1:mail1:backup2] total prune:          78 blobs / 49.011 MiB
2024-11-28T06:23:25-05:00 [1:mail1:backup2] unused size after prune: 64.298 MiB (0.34% of remaining size)
2024-11-28T06:23:25-05:00 [1:mail1:backup2] to repack:           115 blobs / 9.902 MiB
2024-11-28T06:23:25-05:00 [1:mail1:backup2] remaining:        446537 blobs / 18.570 GiB
2024-11-28T06:23:25-05:00 [1:mail1:backup2] to delete:            35 blobs / 44.948 MiB
2024-11-28T06:23:25-05:00 [1:mail1:backup2] repacking packs
2024-11-28T06:23:26-05:00 [1:mail1:backup2] [0:01] 100.00%  2 / 2 packs repacked
2024-11-28T06:23:26-05:00 [1:mail1:backup2] rebuilding index
2024-11-28T06:23:29-05:00 [1:mail1:backup2] [0:03] 100.00%  1154 / 1154 packs processed
2024-11-28T06:23:29-05:00 [1:mail1:backup2] deleting obsolete index files
2024-11-28T06:23:29-05:00 [1:mail1:backup2] [0:00] 100.00%  4 / 4 files deleted
2024-11-28T06:23:29-05:00 [1:mail1:backup2] removing 5 old packs
2024-11-28T06:23:29-05:00 [1:mail1:backup2] [0:00] 100.00%  5 / 5 files deleted
2024-11-28T06:23:29-05:00 [1:mail1:backup2] done
2024-11-28T06:23:29-05:00 [1:mail1:backup2] restic stats --json latest
2024-11-28T06:23:41-05:00 [1:mail1:backup2] Cleaning up Mail state:
2024-11-28T06:23:41-05:00 [1:mail1:backup2] removed 'pcdb-dump.sql'
2024-11-28T06:23:41-05:00 [1:mail1:backup2] removed 'freshclam.timer.is-enabled'
2024-11-28T06:23:41-05:00 [1:mail1:backup2] removed 'clamav.service.is-enabled'
2024-11-28T06:23:41-05:00 [1:mail1:systemd] backup2.service: Consumed 3.149s CPU time.
2024-11-28T06:23:41-05:00 [1::agent@cluster] Task module/nextcloud1/run-backup run failed: {'output': '', 'error': 'Job for backup2.service failed because the control process exited with error code.\nSee "systemctl --user status backup2.service" and "journalctl --user -xeu backup2.service" for details.\n', 'exit_code': 1}

I don’t see an error with mail, but Nextcloud looks like the repository is locked. Ran restic-wrapper --backup 2 unlock which appears to have completed successfully. Ran the backup again, and it also appears to have completed successfully.

1 Like
13

…and that is the case; this key only has access to this bucket.

But given how many provider-specific drivers rclone has, S3 kind of sounds like a standard that isn’t.

14

I had to reapply the patch after the core update, but with that patch, backups to iDrive are continuing to be successful–it seems that patch did it.

1 Like
15

The Rclone configuration fix is available in core 3.2.5

2 Likes