I like the backup system in NethServer NS8, except for local backup. I’ve noticed that if a USB disk fails or disconnects, the backup can write to local disk without any error — which isn’t ideal. I saw this after a lightning storm and was lucky to recover data from a failing drive (bare-metal install).
I took my original rsync off-site backup script, modified it with some AI help and hardening, and turned it into a root tool that monitors for disconnection, and makes the backup fail if the USB target isn’t really there — instead of silently falling back to local disk. The script lets you configure USB disks by UUID, mount, and bind. A cron job (default: every five minutes) checks that the USB is actually connected and locks access when it isn’t, so backups can’t write to the wrong place.
I’m sure there are nicer ways to do this, but it’s a starting point.
Thanks for raising this issue and for sharing your script.
I think the documented procedure[1] already covers the main failure scenarios. If not, it’s a bug!
When the backup volume is created with a specific device, for example using --opt=device=/dev/disk/by-id/..., Podman expects that device to be present when mounting the volume. After a reboot, if the USB disk is missing, the volume mount fails, rclone-webdav.service does not start, and backups cannot run.
Likewise, if the disk is unplugged while in use, I/O errors are returned and the backup job should fail rather than silently continue elsewhere.
Where things can become more complicated is when a USB disk is exposed through a bind mount. In that case, if the mount disappears and the bind target falls back to the underlying directory, I can see how data might end up on the local filesystem instead. For that setup, a monitoring script like yours could indeed be useful.
My recommendation would be to use the documented by-id volume configuration whenever possible, as it avoids most of these issues by design.