Backup failing on one app

Support
,
1

Starting last week, one app, BackupPC has been failing daily:

2024-12-18T01:00:06-08:00 [1:backuppc2:systemd] Starting Backup to SMB destination...
2024-12-18T01:00:10-08:00 [1:backuppc2:backup3] restic --option=rclone.program=/usr/local/bin/rclone-wrapper snapshots
2024-12-18T01:00:12-08:00 [1:backuppc2:systemd] Started libcrun container.
2024-12-18T01:01:00-08:00 [1:backuppc2:systemd] libpod-e4c5a64292489ee2536cce27e329edd8d5c92043f86593c29c7f8a5973319c91.scope: Consumed 1.703s CPU time.
2024-12-18T01:01:03-08:00 [1:backuppc2:backup3] Repository d3cd492b-0703-5483-812b-b075e041f809 is present at path backuppc/dd78f4dd-97c2-459e-bc23-c6a34f9f6213
2024-12-18T01:01:03-08:00 [1:backuppc2:backup3] restic --option=rclone.program=/usr/local/bin/rclone-wrapper backup --no-scan state/environment --files-from=/etc/state-include.conf
2024-12-18T01:01:05-08:00 [1:backuppc2:systemd] Started libcrun container.
2024-12-18T01:01:05-08:00 [1:backuppc2:conmon] conmon 9e90db16d8a1ce55850c <nwarn>: runtime stderr: cannot stat `/proc/self/fd/100`: Input/output error
2024-12-18T01:01:05-08:00 [1:backuppc2:conmon] conmon 9e90db16d8a1ce55850c <error>: Failed to create container: exit status 1
2024-12-18T01:01:07-08:00 [1:backuppc2:backup3] Error: OCI runtime error: crun: cannot stat `/proc/self/fd/100`: Input/output error
2024-12-18T01:01:07-08:00 [1:backuppc2:backup3] [ERROR] restic backup failed. Command '['podman', 'run', '-i', '--rm', '--name=restic-backuppc2-309188', '--privileged', '--network=host', '--volume=restic-cache:/var/cache/restic', '--log-driver=none', '-e', 'RESTIC_PASSWORD', '-e', 'RESTIC_CACHE_DIR', '-e', 'RESTIC_REPOSITORY', '-e', 'RCLONE_SMB_HOST', '-e', 'RCLONE_SMB_USER', '-e', 'RCLONE_SMB_PASS', '-e', 'RCLONE_SMB_DOMAIN', '--workdir=/srv', '--volume=/home/backuppc2/.config/state:/srv/state', '--volume=logs:/srv/volumes/logs', '--volume=conf:/srv/volumes/conf', '--volume=home:/srv/volumes/home', '--volume=data:/srv/volumes/data', '--volume=/home/backuppc2/.config/etc/state-include.conf:/etc/state-include.conf:ro', 'ghcr.io/nethserver/restic:3.3.1', '--option=rclone.program=/usr/local/bin/rclone-wrapper', 'backup', '--no-scan', 'state/environment', '--files-from=/etc/state-include.conf']' returned non-zero exit status 126.
2024-12-18T01:01:07-08:00 [1:backuppc2:backup3] restic --option=rclone.program=/usr/local/bin/rclone-wrapper stats --json latest
2024-12-18T01:01:09-08:00 [1:backuppc2:systemd] Started libcrun container.
2024-12-18T01:01:22-08:00 [1:backuppc2:systemd] libpod-ac9341b9e4d6b539d66290b0ca3f5f09111b2a35fb2b6a6cbcc8eee3d3f5c33a.scope: Consumed 7.659s CPU time.
2024-12-18T01:01:25-08:00 [1:backuppc2:systemd] Started libcrun container.
2024-12-18T01:01:25-08:00 [1:backuppc2:conmon] conmon e5c2b049608c9ec17d74 <nwarn>: Failed to open cgroups file: /sys/fs/cgroup/user.slice/user-1007.slice/user@1007.service/user.slice/libpod-e5c2b049608c9ec17d7434e244b3ce4bcbf508872cf765b5a415c33f7b16ed3e.scope/container/memory.events
2024-12-18T01:01:27-08:00 [1:backuppc2:systemd] backup3.service: Main process exited, code=exited, status=1/FAILURE
2024-12-18T01:01:27-08:00 [1:backuppc2:systemd] backup3.service: Failed with result 'exit-code'.
2024-12-18T01:01:27-08:00 [1:backuppc2:systemd] Failed to start Backup to SMB destination.

I’m not sure if this coincided with any core updates, as I remember there’s been a few lately, but not exactly when. The system itself has been up for nearly a month and no underlying OS updates at that time.

=== Update ===

I just noticed that the backup is trying to include: ‘–volume=data:/srv/volumes/data’ But from this version, it shouldn’t be included in the local backup any more.

=== Further Update ===

It looks like the iSCSI connection to my NAS is broken, so the directory backing the /srv/volumes/data is no longer accessible:

root@ns8:~# ls -l /mnt/
ls: cannot access '/mnt/backupdata': Input/output error
total 0
d????????? ? ? ? ?            ? backupdata
root@ns8:~#

Which explains why this is failing when trying to back that up.

But not why it’s trying to, based on @stephdl 's update to not back up data.

Cheers.

2

hello @EddieA

Reading this I think that we do not use the file state-include.conf when we backup, we list the volumes and we backup them

We should list the volumes inside the file state-include.conf like we do for the restoration Code search results · GitHub

Is that a bug @davidep the data volume not listed in the state-include.conf is backuped as we could find in

2024-12-18T01:01:07-08:00 [1:backuppc2:backup3] [ERROR] restic backup failed. Command '['podman', 'run', '-i', '--rm', '--name=restic-backuppc2-309188', '--privileged', '--network=host', '--volume=restic-cache:/var/cache/restic', '--log-driver=none', '-e', 'RESTIC_PASSWORD', '-e', 'RESTIC_CACHE_DIR', '-e', 'RESTIC_REPOSITORY', '-e', 'RCLONE_SMB_HOST', '-e', 'RCLONE_SMB_USER', '-e', 'RCLONE_SMB_PASS', '-e', 'RCLONE_SMB_DOMAIN', '--workdir=/srv', '--volume=/home/backuppc2/.config/state:/srv/state', '--volume=logs:/srv/volumes/logs', '--volume=conf:/srv/volumes/conf', '--volume=home:/srv/volumes/home', '--volume=data:/srv/volumes/data', '--volume=/home/backuppc2/.config/etc/state-include.conf:/etc/state-include.conf:ro', 'ghcr.io/nethserver/restic:3.3.1', '--option=rclone.program=/usr/local/bin/rclone-wrapper', 'backup', '--no-scan', 'state/environment', '--files-from=/etc/state-include.conf']' returned non-zero exit status 126.

3

@stephdl Looking at the backed up data, it’s not backing up the DATA volume following the change you did:

image

I didn’t go through every single folder in the data directory, but peeking into a few directories the latest date I could find for what looks like coming from DATA (based on size) is around 11/10/2024, which was the date I updated the app to include your changes.

Now as to why those files haven’t been deleted is another question, as I only have the retention set at 14 days.

Cheers.

4

ok so good to know it works :stuck_out_tongue: