Help diagnosing Restic over SMB error on Hetzner storage

pagaille · November 10, 2021, 12:55pm

Hi there,

Still struggling to configure Nethserver to hook to Hetzner Robot Storage for Restic Backups.

Have a look :

created restic repository 83b5fdb25c at /mnt/backup-restic_ecowez

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
created new cache in /var/lib/nethserver/backup/restic/restic_ecowez
no parent snapshot found, will read all files
Save(<data/9ae3a941fd>) returned error, retrying after 552.330144ms: Sync: sync /mnt/backup-restic_ecowez/data/9a/9ae3a941fd5960275323e5e44f7deaf660b6cd1cb2621c7b3c19522860a0c1c1: input/output error
Save(<data/92ebbe2fa4>) returned error, retrying after 720.254544ms: Sync: sync /mnt/backup-restic_ecowez/data/92/92ebbe2fa4cf6441110859e93bea00ca9a9614c31b8332d80a558dfc6ca42e21: input/output error
Save(<data/d0819b9c69>) returned error, retrying after 582.280027ms: Sync: sync /mnt/backup-restic_ecowez/data/d0/d0819b9c69bb77b83defb4d412be02948dc46669214d334a1e86c5e0adee3ef7: input/output error
Save(<data/9ae3a941fd>) returned error, retrying after 703.28564ms: OpenFile: open /mnt/backup-restic_ecowez/data/9a/9ae3a941fd5960275323e5e44f7deaf660b6cd1cb2621c7b3c19522860a0c1c1: file exists
Save(<data/d0819b9c69>) returned error, retrying after 693.478123ms: OpenFile: open /mnt/backup-restic_ecowez/data/d0/d0819b9c69bb77b83defb4d412be02948dc46669214d334a1e86c5e0adee3ef7: file exists
Save(<data/92ebbe2fa4>) returned error, retrying after 890.117305ms: OpenFile: open /mnt/backup-restic_ecowez/data/92/92ebbe2fa4cf6441110859e93bea00ca9a9614c31b8332d80a558dfc6ca42e21: file exists
Save(<data/9ae3a941fd>) returned error, retrying after 636.341646ms: OpenFile: open /mnt/backup-restic_ecowez/data/9a/9ae3a941fd5960275323e5e44f7deaf660b6cd1cb2621c7b3c19522860a0c1c1: file exists
Save(<data/d0819b9c69>) returned error, retrying after 738.584161ms: OpenFile: open /mnt/backup-restic_ecowez/data/d0/d0819b9c69bb77b83defb4d412be02948dc46669214d334a1e86c5e0adee3ef7: file exists
Save(<data/92ebbe2fa4>) returned error, retrying after 671.590708ms: OpenFile: open /mnt/backup-restic_ecowez/data/92/92ebbe2fa4cf6441110859e93bea00ca9a9614c31b8332d80a558dfc6ca42e21: file exists
Save(<data/9ae3a941fd>) returned error, retrying after 1.351538765s: OpenFile: open /mnt/backup-restic_ecowez/data/9a/9ae3a941fd5960275323e5e44f7deaf660b6cd1cb2621c7b3c19522860a0c1c1: file exists
Save(<data/d0819b9c69>) returned error, retrying after 1.713171311s: OpenFile: open /mnt/backup-restic_ecowez/data/d0/d0819b9c69bb77b83defb4d412be02948dc46669214d334a1e86c5e0adee3ef7: file exists
Save(<data/92ebbe2fa4>) returned error, retrying after 2.216767434s: OpenFile: open /mnt/backup-restic_ecowez/data/92/92ebbe2fa4cf6441110859e93bea00ca9a9614c31b8332d80a558dfc6ca42e21: file exists
Save(<data/9ae3a941fd>) returned error, retrying after 1.807980427s: OpenFile: open /mnt/backup-restic_ecowez/data/9a/9ae3a941fd5960275323e5e44f7deaf660b6cd1cb2621c7b3c19522860a0c1c1: file exists
Save(<data/d0819b9c69>) returned error, retrying after 2.22916351s: OpenFile: open /mnt/backup-restic_ecowez/data/d0/d0819b9c69bb77b83defb4d412be02948dc46669214d334a1e86c5e0adee3ef7: file exists
Save(<data/92ebbe2fa4>) returned error, retrying after 2.070709754s: OpenFile: open /mnt/backup-restic_ecowez/data/92/92ebbe2fa4cf6441110859e93bea00ca9a9614c31b8332d80a558dfc6ca42e21: file exists
Save(<data/9ae3a941fd>) returned error, retrying after 3.67875363s: OpenFile: open /mnt/backup-restic_ecowez/data/9a/9ae3a941fd5960275323e5e44f7deaf660b6cd1cb2621c7b3c19522860a0c1c1: file exists
Save(<data/d0819b9c69>) returned error, retrying after 2.973082793s: OpenFile: open /mnt/backup-restic_ecowez/data/d0/d0819b9c69bb77b83defb4d412be02948dc46669214d334a1e86c5e0adee3ef7: file exists
Save(<data/92ebbe2fa4>) returned error, retrying after 3.011308614s: OpenFile: open /mnt/backup-restic_ecowez/data/92/92ebbe2fa4cf6441110859e93bea00ca9a9614c31b8332d80a558dfc6ca42e21: file exists
Save(<data/9ae3a941fd>) returned error, retrying after 6.715255694s: OpenFile: open /mnt/backup-restic_ecowez/data/9a/9ae3a941fd5960275323e5e44f7deaf660b6cd1cb2621c7b3c19522860a0c1c1: file exists
Save(<data/d0819b9c69>) returned error, retrying after 4.092384182s: OpenFile: open /mnt/backup-restic_ecowez/data/d0/d0819b9c69bb77b83defb4d412be02948dc46669214d334a1e86c5e0adee3ef7: file exists
Save(<data/92ebbe2fa4>) returned error, retrying after 4.004869008s: OpenFile: open /mnt/backup-restic_ecowez/data/92/92ebbe2fa4cf6441110859e93bea00ca9a9614c31b8332d80a558dfc6ca42e21: file exists
Save(<data/d0819b9c69>) returned error, retrying after 7.354397612s: OpenFile: open /mnt/backup-restic_ecowez/data/d0/d0819b9c69bb77b83defb4d412be02948dc46669214d334a1e86c5e0adee3ef7: file exists
Save(<data/92ebbe2fa4>) returned error, retrying after 9.146728339s: OpenFile: open /mnt/backup-restic_ecowez/data/92/92ebbe2fa4cf6441110859e93bea00ca9a9614c31b8332d80a558dfc6ca42e21: file exists
Save(<data/9ae3a941fd>) returned error, retrying after 11.639721773s: OpenFile: open /mnt/backup-restic_ecowez/data/9a/9ae3a941fd5960275323e5e44f7deaf660b6cd1cb2621c7b3c19522860a0c1c1: file exists
Save(<data/d0819b9c69>) returned error, retrying after 10.163325309s: OpenFile: open /mnt/backup-restic_ecowez/data/d0/d0819b9c69bb77b83defb4d412be02948dc46669214d334a1e86c5e0adee3ef7: file exists
Save(<data/92ebbe2fa4>) returned error, retrying after 10.214177147s: OpenFile: open /mnt/backup-restic_ecowez/data/92/92ebbe2fa4cf6441110859e93bea00ca9a9614c31b8332d80a558dfc6ca42e21: file exists
Save(<data/9ae3a941fd>) returned error, retrying after 16.05103845s: OpenFile: open /mnt/backup-restic_ecowez/data/9a/9ae3a941fd5960275323e5e44f7deaf660b6cd1cb2621c7b3c19522860a0c1c1: file exists
Save(<data/d0819b9c69>) returned error, retrying after 13.581705599s: OpenFile: open /mnt/backup-restic_ecowez/data/d0/d0819b9c69bb77b83defb4d412be02948dc46669214d334a1e86c5e0adee3ef7: file exists
Save(<data/92ebbe2fa4>) returned error, retrying after 26.244032286s: OpenFile: open /mnt/backup-restic_ecowez/data/92/92ebbe2fa4cf6441110859e93bea00ca9a9614c31b8332d80a558dfc6ca42e21: file exists
Save(<data/9ae3a941fd>) returned error, retrying after 23.002952479s: OpenFile: open /mnt/backup-restic_ecowez/data/9a/9ae3a941fd5960275323e5e44f7deaf660b6cd1cb2621c7b3c19522860a0c1c1: file exists
Fatal: unable to save snapshot: OpenFile: open /mnt/backup-restic_ecowez/data/d0/d0819b9c69bb77b83defb4d412be02948dc46669214d334a1e86c5e0adee3ef7: file exists
Backup failed

Then I tried to delete the storage (still through the SMB mounted drive) :

[root@cloud backup-restic_ecowez]# rm -Rf *
rm: cannot remove ‘data/d0’: Directory not empty
rm: cannot remove ‘data/9a’: Directory not empty
rm: cannot remove ‘data/92’: Directory not empty
[root@cloud backup-restic_ecowez]# ls -lai data/d0                                                                                                                                  
total 1
10980450417000105048 drwxr-xr-x 2 root root       0 Nov  8 22:29 .
10980450352032596608 drwxr-xr-x 2 root root       0 Nov  8 22:56 ..
10980450635096375320 -rwxr-xr-x 0 root root 4364031 Nov  8 22:29 d0819b9c69bb77b83defb4d412be02948dc46669214d334a1e86c5e0adee3ef7
[root@cloud backup-restic_ecowez]# 
[root@cloud backup-restic_ecowez]# ls -lai data/9a
total 1
10980450400401363688 drwxr-xr-x 2 root root       0 Nov  8 22:49 .
10980450352032596608 drwxr-xr-x 2 root root       0 Nov  8 22:56 ..
10980450635244960144 -rwxr-xr-x 0 root root 4301436 Nov  8 22:29 9ae3a941fd5960275323e5e44f7deaf660b6cd1cb2621c7b3c19522860a0c1c1
[root@cloud backup-restic_ecowez]# ls -lai data/92
total 1
10980450397936811040 drwxr-xr-x 2 root root       0 Nov  8 22:55 .
10980450352032596608 drwxr-xr-x 2 root root       0 Nov  8 22:56 ..
10980450635095801320 -rwxr-xr-x 0 root root 4527498 Nov  8 22:29 92ebbe2fa4cf6441110859e93bea00ca9a9614c31b8332d80a558dfc6ca42e21

The offending files/folders are precisely the ones reported by restic in its log, i.e :

[Restic] Fatal: unable to save snapshot: OpenFile: open /mnt/backup-restic_ecowez/data/**d0/d0819b9c69bb77b83defb4d412be02948dc46669214d334a1e86c5e0adee3ef7**: file exists

Please note that accessing the storage through webdav allowed me to delete the offending folders and files.

Hetzner doesn’t look impressed and claims that everything is OK.

Any idea ?

Txs

pike · November 10, 2021, 1:43pm

Maybe Hetzner do not allow your process to have the rights that restic would love?
Maybe Hetzner is lower than what restic expect?

pagaille · November 10, 2021, 1:57pm

Sure but which one ? And how come that the file could be created but not deleted nor accessed through the same interface ?

Andy_Wismer · November 10, 2021, 2:42pm

Hi

On such hosting environments, such access (Storage/Backup) is often done as root, at the end of the copy/transfer the permission is set to your user/group…

If the job couldn’t complete, you’re left with an “orphan” (A file with no correct ownership). You see the file, but can’t access or delete it, as the ownership is wrong.

Rsync also works this way, and sets perms at the end of a transfer.

My 2 cents
Andy

pagaille · November 10, 2021, 3:33pm

But why would the job fail to complete ? Because Restic can’t change the rights after the transfer ?

Edit : indeed, it does change the permissions, and it’s hard coded

Andy_Wismer · November 10, 2021, 3:37pm

@pagaille

A work-around would be to test a manual connection and copy or rsync the stuff backup also copies, eg databases, files, etc from /var/lib/nethserver/.

You can also test which works better, scp, rsync, nfs or whatever…

If it works, a script job backup is better than absolutly nothing!

Not an ideal solution, but does give you backups and time to get this correctly solved…

pagaille · November 10, 2021, 3:45pm

Indeed. My idea was hacking nethserver’s code to make it compatible with passive / pure SFTP servers (and not SFTP over SSH). Not sure I’ll have the time. And there is also the step where a specific public key is sent to the server that should be handled (probably deleted).

Maybe @filippo_carletti could share his views about this, I could submit a PR if I succeed.

Andy_Wismer · November 10, 2021, 3:47pm

I could provide you a sample code, I have such in operation…
This can be adapted as needed…

My 2 cents
Andy

filippo_carletti · November 10, 2021, 3:54pm

Restic over SMB (maybe everything over SMB) has proved to be “limited” many times. I no longer use SMB/CIFS (I prefer NFS).

AFAIR, SFTP over SSH is only needed by rsync, but I can’t test restic now.

@pagaille could you show us some errors or logs about restic over sftp?

pagaille · November 11, 2021, 8:11pm

Indeed

Actually I can’t. Wanted to use SFTP but that storage doesn’t support SFTP over SSH (NS uses that way to connect and assumes that the storage is smart and supports executing commands). The storage I want to connect is dumb and only allows pure SFTP.

Thats why I proposed to try to implement a pure SFTP backup connector.

My storage doesn’t support NFS either. I tried Webdav but the performance is extremely bad, no idea why.

pagaille · November 11, 2021, 8:14pm

Thanks Andy. You know I’m a GUI addict and I don’t want to handle all the modules backup singularities. For the time being I switched to Duplicity over SMB, better than nothing.

pagaille · December 15, 2021, 3:05pm

Update : tried again the combo Restic + Webdav : it works perfectly. There reason why performance is terrible with rsync is probably due to the millions of little files transferred, where restic packages them into bigger chunks.

cc @capote