App backup failed

dan · October 21, 2026, 5:05pm

With current NS8, I’m getting a recurrent failure on one of my backup tasks:

“See details” really doesn’t add anything:

I’m pretty sure at some point in the past I saw an error that the restic repo was locked, but I can’t seem to find that right now. How can I determine the cause of this failure?

davidep · October 21, 2026, 5:16pm

What is the destination type of NAS?

Try to search errors in System logs from 1:00 AM to the Completed timestamp.

Also, make sure you have the latest core version 3.23.2 installed, since it tweaks internal restic timeouts that may impact your backup, including locks management.

dan · October 21, 2026, 6:08pm

Looks like my subscription hasn’t picked it up yet; I’m on 3.23.0

TrueNAS, on an SMB share, using Tailscale.

Nothing relevant in the node log when searching for “Error.” Searching for “grafana” came up with this:

2026-10-21T01:07:56-04:00 [1::runagent] grafana1: starting backup 6 at Wed Oct 21 01:07:56 2026
2026-10-21T01:07:56-04:00 [1::systemd] Started Session c116 of User grafana1.
2026-10-21T01:07:56-04:00 [1::runuser] pam_unix(runuser-l:session): session opened for user grafana1(uid=1004) by (uid=0)
2026-10-21T01:07:57-04:00 [1:grafana1:systemd] Started podman-1044949.scope.
2026-10-21T01:07:57-04:00 [1:grafana1:systemd] Started podman-1044959.scope.
2026-10-21T01:07:57-04:00 [1:grafana1:podman] 2026-10-21 01:07:57.763159641 -0400 EDT m=+0.090675687 image pull 6c9111eb318cc2d6aed53463bb9441d36df1bd59f8b7db0f53879fcff2c89c73 ghcr.io/nethserver/restic:3.23.0
2026-10-21T01:07:57-04:00 [1:grafana1:podman] 2026-10-21 01:07:57.821302265 -0400 EDT m=+0.148818301 container create 5ac008f4a4564524f08beb44aa84e391e731ad9edef2d346b99102096878b97e (image=ghcr.io/nethserver/restic:3.23.0, name=restic-grafana1-1044940, io.buildah.version=1.33.7)
2026-10-21T01:07:57-04:00 [1:grafana1:systemd] Started libpod-conmon-5ac008f4a4564524f08beb44aa84e391e731ad9edef2d346b99102096878b97e.scope.
2026-10-21T01:07:57-04:00 [1:grafana1:systemd] Started libcrun container.
2026-10-21T01:07:57-04:00 [1::kernel] xfs filesystem being remounted at /home/grafana1/.local/share/containers/storage/overlay/ff0255e27027d847bebaefefec838eab9eba8706858628667e47585db23a3815/merged/var/cache/restic supports timestamps until 2038 (0x7fffffff)
2026-10-21T01:07:57-04:00 [1:grafana1:podman] 2026-10-21 01:07:57.993404222 -0400 EDT m=+0.320920278 container init 5ac008f4a4564524f08beb44aa84e391e731ad9edef2d346b99102096878b97e (image=ghcr.io/nethserver/restic:3.23.0, name=restic-grafana1-1044940, io.buildah.version=1.33.7)
2026-10-21T01:07:58-04:00 [1:grafana1:podman] 2026-10-21 01:07:58.011058521 -0400 EDT m=+0.338574547 container start 5ac008f4a4564524f08beb44aa84e391e731ad9edef2d346b99102096878b97e (image=ghcr.io/nethserver/restic:3.23.0, name=restic-grafana1-1044940, io.buildah.version=1.33.7)
2026-10-21T01:07:58-04:00 [1:grafana1:podman] 2026-10-21 01:07:58.016358561 -0400 EDT m=+0.343874607 container attach 5ac008f4a4564524f08beb44aa84e391e731ad9edef2d346b99102096878b97e (image=ghcr.io/nethserver/restic:3.23.0, name=restic-grafana1-1044940, io.buildah.version=1.33.7)
2026-10-21T01:08:03-04:00 [1:grafana1:podman] 2026-10-21 01:08:03.725404861 -0400 EDT m=+6.052920907 container died 5ac008f4a4564524f08beb44aa84e391e731ad9edef2d346b99102096878b97e (image=ghcr.io/nethserver/restic:3.23.0, name=restic-grafana1-1044940, io.buildah.version=1.33.7)
2026-10-21T01:08:03-04:00 [1:grafana1:podman] 2026-10-21 01:08:03.822714821 -0400 EDT m=+6.150230847 container remove 5ac008f4a4564524f08beb44aa84e391e731ad9edef2d346b99102096878b97e (image=ghcr.io/nethserver/restic:3.23.0, name=restic-grafana1-1044940, io.buildah.version=1.33.7)
2026-10-21T01:08:03-04:00 [1:grafana1:runagent] Repository is present at path 4bc4957e-f676-5cff-a365-7ee570f623bd/grafana/9d8a0dd9-bc23-4fd9-941d-713d61f12ce6
2026-10-21T01:08:03-04:00 [1:grafana1:systemd] Started podman-1045095.scope.
2026-10-21T01:08:04-04:00 [1:grafana1:podman] 2026-10-21 01:08:04.016653422 -0400 EDT m=+0.145169010 image pull 6c9111eb318cc2d6aed53463bb9441d36df1bd59f8b7db0f53879fcff2c89c73 ghcr.io/nethserver/restic:3.23.0
2026-10-21T01:08:04-04:00 [1:grafana1:podman] 2026-10-21 01:08:04.04309901 -0400 EDT m=+0.171614578 container create 5aae5447697d27e97ae357a2d9130efadf689b6fd588d620b16983cc019acddc (image=ghcr.io/nethserver/restic:3.23.0, name=restic-grafana1-1044940, io.buildah.version=1.33.7)
2026-10-21T01:08:04-04:00 [1:grafana1:systemd] Started libpod-conmon-5aae5447697d27e97ae357a2d9130efadf689b6fd588d620b16983cc019acddc.scope.
2026-10-21T01:08:04-04:00 [1:grafana1:systemd] Started libcrun container.
2026-10-21T01:08:04-04:00 [1::kernel] xfs filesystem being remounted at /home/grafana1/.local/share/containers/storage/overlay/b87ee6d78aa6bee2d7081da1a69dfb25d491887b52c5c8d4480427ffd0567157/merged/srv/state supports timestamps until 2038 (0x7fffffff)
2026-10-21T01:08:04-04:00 [1::kernel] xfs filesystem being remounted at /home/grafana1/.local/share/containers/storage/overlay/b87ee6d78aa6bee2d7081da1a69dfb25d491887b52c5c8d4480427ffd0567157/merged/srv/volumes/grafana-data supports timestamps until 2038 (0x7fffffff)
2026-10-21T01:08:04-04:00 [1::kernel] xfs filesystem being remounted at /home/grafana1/.local/share/containers/storage/overlay/b87ee6d78aa6bee2d7081da1a69dfb25d491887b52c5c8d4480427ffd0567157/merged/var/cache/restic supports timestamps until 2038 (0x7fffffff)
2026-10-21T01:08:04-04:00 [1:grafana1:podman] 2026-10-21 01:08:04.218733071 -0400 EDT m=+0.347248660 container init 5aae5447697d27e97ae357a2d9130efadf689b6fd588d620b16983cc019acddc (image=ghcr.io/nethserver/restic:3.23.0, name=restic-grafana1-1044940, io.buildah.version=1.33.7)
2026-10-21T01:08:04-04:00 [1:grafana1:podman] 2026-10-21 01:08:04.231255441 -0400 EDT m=+0.359771019 container start 5aae5447697d27e97ae357a2d9130efadf689b6fd588d620b16983cc019acddc (image=ghcr.io/nethserver/restic:3.23.0, name=restic-grafana1-1044940, io.buildah.version=1.33.7)
2026-10-21T01:08:04-04:00 [1:grafana1:podman] 2026-10-21 01:08:04.238664576 -0400 EDT m=+0.367180164 container attach 5aae5447697d27e97ae357a2d9130efadf689b6fd588d620b16983cc019acddc (image=ghcr.io/nethserver/restic:3.23.0, name=restic-grafana1-1044940, io.buildah.version=1.33.7)
2026-10-21T01:08:08-04:00 [1:grafana1:runagent] using parent snapshot 64febd82
2026-10-21T01:08:11-04:00 [1:grafana1:runagent] snapshot d028e43c saved
2026-10-21T01:08:11-04:00 [1:grafana1:runagent] Files:           0 new,     1 changed,     0 unmodified
2026-10-21T01:08:11-04:00 [1:grafana1:runagent] Dirs:            0 new,     1 changed,     0 unmodified
2026-10-21T01:08:11-04:00 [1:grafana1:runagent] Added to the repository: 950 B (771 B stored)
2026-10-21T01:08:11-04:00 [1:grafana1:runagent] processed 1 files, 866 B in 0:06
2026-10-21T01:08:11-04:00 [1:grafana1:podman] 2026-10-21 01:08:11.908625325 -0400 EDT m=+8.037140893 container died 5aae5447697d27e97ae357a2d9130efadf689b6fd588d620b16983cc019acddc (image=ghcr.io/nethserver/restic:3.23.0, name=restic-grafana1-1044940, io.buildah.version=1.33.7)
2026-10-21T01:08:11-04:00 [1:grafana1:podman] 2026-10-21 01:08:11.98137546 -0400 EDT m=+8.109891028 container remove 5aae5447697d27e97ae357a2d9130efadf689b6fd588d620b16983cc019acddc (image=ghcr.io/nethserver/restic:3.23.0, name=restic-grafana1-1044940, io.buildah.version=1.33.7)
2026-10-21T01:08:12-04:00 [1::runuser] pam_unix(runuser-l:session): session closed for user grafana1
2026-10-21T01:08:13-04:00 [1:grafana1:grafana] logger=dashboard-service t=2026-10-21T05:08:13.332092821Z level=info msg="No last resource version found, starting from scratch" orgID=1
2026-10-21T01:08:22-04:00 [1::runagent] [ERROR] retention policy failed for 4bc4957e-f676-5cff-a365-7ee570f623bd/grafana/9d8a0dd9-bc23-4fd9-941d-713d61f12ce6: Command '['podman', 'run', '--interactive', '--rm', '--replace', '--name=restic-node1-1040477', '--privileged', '--network=host', '--env=RESTIC_*', '--volume=restic-cache:/var/cache/restic', '--log-driver=none', 'ghcr.io/nethserver/restic:3.23.0', 'forget', '--prune', '--keep-last=5']' returned non-zero exit status 11.
2026-10-21T01:08:23-04:00 [1::runagent] grafana1: backup 6 failed (exit code 1)