NethServer Version: 8
Module: Restic Backup
Hi
Restic Backup has been working for several months now, without any errors.
I now have an error with Restic Backup since 25.12.2024.
It seems that “loki1” has a lock set on that date which has never been removed.
This is extremly misleading especially considering no backup has actually been done since 5+ days…:
My System:
The error:
Log Details:
(AFAIK, this contains the exact error…)
Task module/loki1/run-backup run failed: {‘output’: ‘repo already locked, waiting up to 0s for the lock\n’, ‘error’: “restic --option=rclone.program=/usr/local/bin/rclone-wrapper snapshots\nRepository 26e5f4f9-54f6-547f-a57e-b797a1e885ad is present at path loki/161416fb-47ac-4bc3-912c-a162e603b64a\nrestic --option=rclone.program=/usr/local/bin/rclone-wrapper backup --json state/environment --files-from=/etc/state-include.conf\nrestic --option=rclone.program=/usr/local/bin/rclone-wrapper forget --prune --keep-last=7\nunable to create lock in backend: repository is already locked by PID 1 on awr9-ns8 by root (UID 0, GID 0)\nlock was created at 2024-12-25 01:00:44 (145h58m51.551888623s ago)\nstorage ID 45a09ebc\n***the unlock command can be used to remove stale locks***n[ERROR] restic backup failed. Command ‘[‘podman’, ‘run’, ‘-i’, ‘–rm’, ‘–name=restic-loki1-4785’, ‘–privileged’, ‘–network=host’, ‘–volume=restic-cache:/var/cache/restic’, ‘–log-driver=none’, ‘-e’, ‘RESTIC_PASSWORD’, ‘-e’, ‘RESTIC_CACHE_DIR’, ‘-e’, ‘RESTIC_REPOSITORY’, ‘-e’, ‘RCLONE_SMB_HOST’, ‘-e’, ‘RCLONE_SMB_USER’, ‘-e’, ‘RCLONE_SMB_PASS’, ‘-e’, ‘RCLONE_SMB_DOMAIN’, ‘Package restic · GitHub’, ‘–option=rclone.program=/usr/local/bin/rclone-wrapper’, ‘forget’, ‘–prune’, ‘–keep-last=7’]’ returned non-zero exit status 1.\nrestic --option=rclone.program=/usr/local/bin/rclone-wrapper stats --json latest\n”, ‘exit_code’: 1}
The “full” log for the whole tasks, detail tasks can be provided, but I do not think they are needed.
Task module/roundcubemail1/run-backup run failed: {‘output’: ‘’, ‘error’: ‘restic --option=rclone.program=/usr/local/bin/rclone-wrapper snapshots\nRepository 26e5f4f9-54f6-547f-a57e-b797a1e885ad is present at path roundcubemail/9d6ea8f4-75f6-4434-a3c2-a2ea2c31ff34\nrestic --option=rclone.program=/usr/local/bin/rclone-wrapper backup --json state/environment --files-from=/etc/state-include.conf\ntime=“2024-12-31T04:20:15+01:00” level=error msg=“Cleaning up volume (581f91e91a7bf6564e653172ce4aa47e4bb8840ef5563f3c5b85811cfd2279b1): volume 581f91e91a7bf6564e653172ce4aa47e4bb8840ef5563f3c5b85811cfd2279b1 is being used by the following container(s): 05126e68fc83a39da9aad1832b37f6b6be83e4fa0114745936e9a4dd3d778472: volume is being used”\nError: crun: error stat'ing file /home/roundcubemail1/.local/share/containers/storage/volumes/581f91e91a7bf6564e653172ce4aa47e4bb8840ef5563f3c5b85811cfd2279b1/_data: No such file or directory: OCI runtime attempted to invoke a command that was not found\n<3>Restic restore command failed with exit code 127.\n’, ‘exit_code’: 1}
Task module/dokuwiki1/run-backup run failed: {‘output’: ‘’, ‘error’: ‘restic --option=rclone.program=/usr/local/bin/rclone-wrapper snapshots\nRepository 26e5f4f9-54f6-547f-a57e-b797a1e885ad is present at path dokuwiki/62d4bf56-a0de-47fa-90f8-8d2a5f67688c\nrestic --option=rclone.program=/usr/local/bin/rclone-wrapper backup --json state/environment --files-from=/etc/state-include.conf\ntime=“2024-12-31T04:20:13+01:00” level=error msg=“Cleaning up volume (e4c4eaaacaf28678eadf1b397bbf9524a034a856560289e1a98e90d154c1f286): volume e4c4eaaacaf28678eadf1b397bbf9524a034a856560289e1a98e90d154c1f286 is being used by the following container(s): 2423cf0207656a70545dadfec96837caec983da0071800004b115fd840bdb797: volume is being used”\nError: crun: error stat'ing file /home/dokuwiki1/.local/share/containers/storage/volumes/e4c4eaaacaf28678eadf1b397bbf9524a034a856560289e1a98e90d154c1f286/_data: No such file or directory: OCI runtime attempted to invoke a command that was not found\n<3>Restic restore command failed with exit code 127.\n’, ‘exit_code’: 1}
Task module/guacamole1/run-backup run failed: {‘output’: ‘Dumping guacamole postgres database\n’, ‘error’: ‘restic --option=rclone.program=/usr/local/bin/rclone-wrapper snapshots\nRepository 26e5f4f9-54f6-547f-a57e-b797a1e885ad is present at path guacamole/8e3efbee-b5f5-4f6a-9e56-bf232c67bd9a\nrestic --option=rclone.program=/usr/local/bin/rclone-wrapper backup --json state/environment --files-from=/etc/state-include.conf\ntime=“2024-12-31T04:20:19+01:00” level=error msg=“Cleaning up volume (be1212910a5798ab009a52ffcefce99f54cfcb90cac24274a98dbc007fea9a22): volume be1212910a5798ab009a52ffcefce99f54cfcb90cac24274a98dbc007fea9a22 is being used by the following container(s): b37ed86c820e7ee58fd18ce5717775fb217c53e352e88a6f661a7cf074a44710: volume is being used”\nError: crun: error stat'ing file /home/guacamole1/.local/share/containers/storage/volumes/be1212910a5798ab009a52ffcefce99f54cfcb90cac24274a98dbc007fea9a22/_data: No such file or directory: OCI runtime attempted to invoke a command that was not found\n<3>Restic restore command failed with exit code 127.\n’, ‘exit_code’: 1}
Task module/nextcloud1/run-backup run failed: {‘output’: ‘’, ‘error’: ‘restic --option=rclone.program=/usr/local/bin/rclone-wrapper snapshots\nRepository 26e5f4f9-54f6-547f-a57e-b797a1e885ad is present at path nextcloud/180f3d32-5e3c-49a3-953a-a569f5d07b66\nrestic --option=rclone.program=/usr/local/bin/rclone-wrapper backup --json state/environment --files-from=/etc/state-include.conf\ntime=“2024-12-31T04:20:44+01:00” level=error msg=“Cleaning up volume (7fb1abbca9ac7deb7d59bff0aa1061b31192016755267e2bdc12538f0c567158): volume 7fb1abbca9ac7deb7d59bff0aa1061b31192016755267e2bdc12538f0c567158 is being used by the following container(s): 3db141e93b9278dff285caed85695cb47dcee42a9a7fffb4c40cd3c0b0e14ef4: volume is being used”\nError: crun: error stat'ing file /home/nextcloud1/.local/share/containers/storage/volumes/7fb1abbca9ac7deb7d59bff0aa1061b31192016755267e2bdc12538f0c567158/_data: No such file or directory: OCI runtime attempted to invoke a command that was not found\n<3>Restic restore command failed with exit code 127.\n’, ‘exit_code’: 1}
Task module/grafana1/run-backup run failed: {‘output’: ‘’, ‘error’: ‘restic --option=rclone.program=/usr/local/bin/rclone-wrapper snapshots\nRepository 26e5f4f9-54f6-547f-a57e-b797a1e885ad is present at path grafana/ad7fc7d7-e594-406a-af78-2d747d4cec22\nrestic --option=rclone.program=/usr/local/bin/rclone-wrapper backup --json state/environment\ntime=“2024-12-31T04:20:15+01:00” level=error msg=“Cleaning up volume (59bd6b944aff4fd2279be8f73412b513f76cdf07af3f78b6d7e387984934c3fc): volume 59bd6b944aff4fd2279be8f73412b513f76cdf07af3f78b6d7e387984934c3fc is being used by the following container(s): be60545a8b14364f7c00eca8031a7793eb7edf08d64f9f7a950782242a923139: volume is being used”\nError: crun: error stat'ing file /home/grafana1/.local/share/containers/storage/volumes/59bd6b944aff4fd2279be8f73412b513f76cdf07af3f78b6d7e387984934c3fc/_data: No such file or directory: OCI runtime attempted to invoke a command that was not found\n<3>Restic restore command failed with exit code 127.\n’, ‘exit_code’: 1}
Task module/webserver1/run-backup run failed: {‘output’: ‘’, ‘error’: ‘restic --option=rclone.program=/usr/local/bin/rclone-wrapper snapshots\nRepository 26e5f4f9-54f6-547f-a57e-b797a1e885ad is present at path webserver/6f384572-698b-4c85-b9d7-a1b7bf9f4769\nrestic --option=rclone.program=/usr/local/bin/rclone-wrapper backup --json state/environment --files-from=/etc/state-include.conf\ntime=“2024-12-31T04:20:16+01:00” level=error msg=“Cleaning up volume (2cab2b98b53c1e096c1c896bef79d7bd0f9211581370988c8433f60d0f1306fa): volume 2cab2b98b53c1e096c1c896bef79d7bd0f9211581370988c8433f60d0f1306fa is being used by the following container(s): fe3d1378dc2b71965564b8e25ff264a7a4c5db47a5948a17afe7156b7999bca2: volume is being used”\nError: crun: error stat'ing file /home/webserver1/.local/share/containers/storage/volumes/2cab2b98b53c1e096c1c896bef79d7bd0f9211581370988c8433f60d0f1306fa/_data: No such file or directory: OCI runtime attempted to invoke a command that was not found\n<3>Restic restore command failed with exit code 127.\n’, ‘exit_code’: 1}
Task module/lam1/run-backup run failed: {‘output’: ‘’, ‘error’: ‘restic --option=rclone.program=/usr/local/bin/rclone-wrapper snapshots\nRepository 26e5f4f9-54f6-547f-a57e-b797a1e885ad is present at path lam/dfaf187f-54b4-40e3-9693-ffd79a968f06\nrestic --option=rclone.program=/usr/local/bin/rclone-wrapper backup --json state/environment --files-from=/etc/state-include.conf\ntime=“2024-12-31T04:20:18+01:00” level=error msg=“Cleaning up volume (65386d69f1e086291935142f46002460a8842b0cab3f2e269b5dfa7f5eed7a3b): volume 65386d69f1e086291935142f46002460a8842b0cab3f2e269b5dfa7f5eed7a3b is being used by the following container(s): 1afea9a5b04a0bfd74892ef39a5e9e7bf23b6fa8e070e0526e472c7fea8cc102: volume is being used”\nError: crun: error stat'ing file /home/lam1/.local/share/containers/storage/volumes/65386d69f1e086291935142f46002460a8842b0cab3f2e269b5dfa7f5eed7a3b/_data: No such file or directory: OCI runtime attempted to invoke a command that was not found\n<3>Restic restore command failed with exit code 127.\n’, ‘exit_code’: 1}
Task module/loki1/run-backup run failed: {‘output’: ‘repo already locked, waiting up to 0s for the lock\n’, ‘error’: “restic --option=rclone.program=/usr/local/bin/rclone-wrapper snapshots\nRepository 26e5f4f9-54f6-547f-a57e-b797a1e885ad is present at path loki/161416fb-47ac-4bc3-912c-a162e603b64a\nrestic --option=rclone.program=/usr/local/bin/rclone-wrapper backup --json state/environment --files-from=/etc/state-include.conf\nrestic --option=rclone.program=/usr/local/bin/rclone-wrapper forget --prune --keep-last=7\nunable to create lock in backend: repository is already locked by PID 1 on awr9-ns8 by root (UID 0, GID 0)\nlock was created at 2024-12-25 01:00:44 (146h20m9.695485644s ago)\nstorage ID 45a09ebc\nthe unlock command can be used to remove stale locks\n[ERROR] restic backup failed. Command ‘[‘podman’, ‘run’, ‘-i’, ‘–rm’, ‘–name=restic-loki1-13058’, ‘–privileged’, ‘–network=host’, ‘–volume=restic-cache:/var/cache/restic’, ‘–log-driver=none’, ‘-e’, ‘RESTIC_PASSWORD’, ‘-e’, ‘RESTIC_CACHE_DIR’, ‘-e’, ‘RESTIC_REPOSITORY’, ‘-e’, ‘RCLONE_SMB_HOST’, ‘-e’, ‘RCLONE_SMB_USER’, ‘-e’, ‘RCLONE_SMB_PASS’, ‘-e’, ‘RCLONE_SMB_DOMAIN’, ‘Package restic · GitHub’, ‘–option=rclone.program=/usr/local/bin/rclone-wrapper’, ‘forget’, ‘–prune’, ‘–keep-last=7’]’ returned non-zero exit status 1.\nrestic --option=rclone.program=/usr/local/bin/rclone-wrapper stats --json latest\n”, ‘exit_code’: 1}
Task module/mariadb1/run-backup run failed: {‘output’: ‘’, ‘error’: ‘restic --option=rclone.program=/usr/local/bin/rclone-wrapper snapshots\nRepository 26e5f4f9-54f6-547f-a57e-b797a1e885ad is present at path mariadb/52f1172c-725b-4348-9d3b-84e6e133121e\nrestic --option=rclone.program=/usr/local/bin/rclone-wrapper backup --json state/environment --files-from=/etc/state-include.conf\ntime=“2024-12-31T04:20:15+01:00” level=error msg=“Cleaning up volume (8dc5e9d0f598ce91bf44401b7f2174fcdc9e46c93dc49c779f61bfdf5a34222d): volume 8dc5e9d0f598ce91bf44401b7f2174fcdc9e46c93dc49c779f61bfdf5a34222d is being used by the following container(s): 6f76f88ebc2371c8fc8f82e31cf6f41d39c2053e32d815075fd3fe2b068b5c3b: volume is being used”\nError: crun: error stat'ing file /home/mariadb1/.local/share/containers/storage/volumes/8dc5e9d0f598ce91bf44401b7f2174fcdc9e46c93dc49c779f61bfdf5a34222d/_data: No such file or directory: OCI runtime attempted to invoke a command that was not found\n<3>Restic restore command failed with exit code 127.\n’, ‘exit_code’: 1}
9
Any ideas or suggestions to remove the lock file(s)?
@stephdl
I wanted a full backup before testing Webserver new installation…
My 2 cents
Andy