NS8 `run-backup` retention fails on stale restic lock (container PID 1)

Support
,
1

Hi Davide,

I had a failure on my usb local backup a week ago and thought I’d debug why it kept failing. I’m running a single-node NS8, Core 3.x, local USB backup via rclone-gateway WebDAV.

Symptom: Scheduled backup marks samba1 failed; ~300 GB restic backup succeeds; retention (forget --prune) fails with exit 11. Redis shows errors: 1 but data is restorable. UI History shows no error detail.

Stale lock: repository is already locked by PID 1 on motueka — lock from an interrupted run (~Aug 4). Plain restic unlock does not remove it (host PID 1 = systemd still alive). unlock --remove-all clears it.

Repro: agent.run_restic → forget --prune after successful module-backup fails until --remove-all. Manual restic-wrapper forget sometimes appeared OK; run-backup path uses agent.run_restic with 0s lock wait.

Local workaround: Retry retention after unlock --remove-all in run_retention() (or always before prune).

Happy to open a GitHub issue on ns8-core if useful — not urgent on our side after workaround. My patch as below:

def run_retention(rdb, repository, repopath, retention):
    """Apply retention policy after a successful backup. Returns True on success, False on failure."""
    forget_args = ["forget", "--prune", f"--keep-last={retention}"]
    try:
        agent.run_restic(rdb, repository, repopath, [], forget_args, check=True)
        return True
    except Exception as ex:
        print(f"[ERROR] retention policy failed for {repository}/{repopath}: {ex}", file=sys.stderr)
        print("Retrying after stale lock cleanup...", file=sys.stderr)
        try:
            agent.run_restic(rdb, repository, repopath, [], ["unlock", "--remove-all"], check=False)
            agent.run_restic(rdb, repository, repopath, [], forget_args, check=True)
            return True
        except Exception as ex2:
            print(f"[ERROR] retention policy failed after unlock retry for {repository}/{repopath}: {ex2}", file=sys.stderr)
            return False

It may need a bit of improvement as done rather quickly.. and only a couple of test runs, thanks for you time,

Turbond
(aka Mad Matt)

2

Hi Matthew,

Glad the workaround is holding up on your side (and will work until next core release overwrites it), and thanks for the detailed report. Can I promote this discussion to a public thread?

I note that PID 1 here isn’t host systemd, it’s the container’s own PID 1 (restic itself runs as PID 1 inside its namespace). Restic’s staleness check works by trying to see if that PID is still alive, and inside a PID-namespaced container PID 1 always looks “alive” from restic’s own point of view, or from a subsequent container/process reusing PID 1. That’s a plausible reason plain unlock fails to recognize the lock as stale on a locally-attached repo.

We’re planning to disable the PID namespace in a future release, which should make this comparison meaningful again. Stay tuned!

3

Yes no problem… More than happy too.