This looks scary and really alpha. Let’s hope the backup itself is working a expected 
I’ll test on my side ASAP.
Matthieu G. (en mode mobile)
Sorry for having captured old thread, but at this moment i didn’t know that would lead so far.
Therefore thanks for having split into new thread.
It’s really not my intention to scare the whole community, by saying that backup-restore-data in nethserver not working correctly.
Maybe it’s just for me, that I’m doing something wrong, and just have tested with local USB-Disk.
My hope is that for other users or other targets work without problems.
Let’s wait if someone can confirm.
Good news for restic.
Restore from restic works with all types of files (spaces,ä,ö,ü,etc.).
Only problem remains that restored directories are changed to owner root and original group, but without no permissions for the group.
Files in these folders are restored correctly with original owner and group permissions, but not accessible for samba users because parent folder is not browsable without permissions.
Workaround is to recreate the directories, where to restore the files in, manually in windows explorer from connected samba user before restoring, or after restoring reset permissions on shared folder in server manager or cockpit.
This solution works for me, but still want to clarify problems with rsync.
I can’t reproduce this problem.
Could you please share your exact configuration?
Tested on virtual machine before experimentation on working server.
VM with 4 GB Ram, 60 GB VDisk for Nethserver, 10 GB VDisk for Backup.
Nethserver 7.7.1908 all updated incl. nethserver-backup-data 1.6.5 and nethserver-restore-data 2.0.2
domain: test.vm
servername: nserver.test.vm
user: mario@test.vm
group: buero@test.vm
logged in old server manager with root@nserver.test.vm
created shared folder: test
owner group: buero@test.vm
write permissions for owner group
guest access no
browsable yes
no other ACL’s
checked with WinSCP or putty to /var/lib/nethserver/ibay/test shows: owner root, group buero@test.vm, permissions rwxrws—
connected as netdrive in Windows Explorer letter T: (\nserver\test) without problems
created some folders Test 1, Test 2, Test 3 from Windows Explorer
checked with WinSCP to /var/lib/nethserver/ibay/test/Test 1 shows: owner mario@test.vm, group buero@test.vm, permissions rwxrws— (same for Test 2 and Test 3)
created new some files (Textdocument 1.txt, Textdocument 2.txt) and copied some files (…xls, …doc, …pdf) in these folders from Windows Explorer
checked with WinSCP all files in /var/lib/nethserver/ibay/test/Test 1/ shows: owner mario@test.vm, group buero@test.vm, permissions rw-rw---- (same for Test 2 and Test 3)
from Cockpit partitioned virtual Disk 10 GB as GPT, formatted ext4, named backupdisk, mounted default (so result in not be mounted, otherwise will not be selectable as backup targed)
create data backup schedule
local disk check: “backupdisk: ext4 | 10G”
engine: restic
name: testrestic
Run Now working fine without errors
local disk seems automatically temporarily mounted to /mnt/backup-testrestic
view last log shows Backup Status Success
Repeated same steps for testing with rsync (name testrsync) and duplicity (name testduplicity)
After that in Windows Explorer renamed folder Test 1 (to Test 1 old), deleted folder Test 2, and deleted files in folder Test 3 (remaining folder Test 3)
in Cockpit opening applications-Restore Data-settings
choose backup “testrestic | USB-restic”
choose mode files or directories
searching for “test”
file tree shows folders Test 1, Test 2, Test 3 and files within there
selecting folders Test 1, Test 2, Test 3 by clicking automatically marking all files within there too
selecting overwrite and starting restore
in Windows Explorer showing up restored folders Test 1, Test 2 and remaining folder Test 3
Possible access only to folder Test 3 and files within there
Attempt to access folder Test1 and Test 2 not permitted
checked with WinSCP to /var/lib/nethserver/ibay/test/Test 1/ shows: owner root, group buero@test.vm, permissions rwx–s— (same for Test 2), so no read/write permissions for the group
remaining folder Test 3 still with original owner mario@test.vm, group buero@test.vm, permissions rwxrws— and therefore accessible
all files in these folders restored with correct owner mario@test.vm, group buero@test.vm, permissions rw-rw----
So problem only with restored folders, not with files and not with remaining folders
Same procedure with rsync not restoring completely nothing
With duplicity not restoring files/folders with “ä,ö,ü,etc.”, so not practicable in Germany
Funny is duplicity restoring folders with owner root, group buero@test.vm, permissions rwxr-sr-x, which means read for group (progress to restic but still no write) and also read for guest (formerly forbidden)
Sorry for very long description, but hope someone can repeat the steps and confirm or contradict.
@filippo_carletti what else configuration do you need, an how to upload ?
I’m still newbie in this forum
Thanks for the detailed explanation.
It seems that we are having multiple issues, let’s check them one by one.
First of all, backups are good, only restore is not working.
-
rsync
I had found problems only when using rsync: restore was broken, but I fixed them.
So, I’d kindly ask to repeat restore tests for rsync after installation of updated packages from the testing repository (see https://github.com/NethServer/dev/issues/5908).
Type in Terminal:
yum -y --enablerepo=nethserver-testing update nethserver-backup-data nethserver-restore-data
Backups are good, only restore is not working. -
restic permissions
I can’t reproduce the permission problem.
Could you please use Terminal to copy here the output of the command:
ls -l /var/lib/nethserver/ibay/test/
before and after restore? -
Duplicity with “strange” characters
I can’t reproduce, see screenshot.
Thanks @filippo_carletti for your efforts and help.
-
rsync now working perfectly, restoring all files and folders with original permissions
-
restic permissions (original, after renaming / deleting folders, after restore)
- Duplicity still not works for me with “strange” characters. Your screenshot seems to be from rsync ?
BTW please take a look at default for volume size under advanced options. Default value 2 is correct or should be 200 ?
So RSYNC is the way for me to go
Are these packages from testing repository ready for professional use or you have to work on it ?
Thanks again @filippo_carletti, you are great !
I can’t still reproduce it. Could you please use ls -la to show parent directory permissions?
I’m sorry, my mistake. I reproduced the problem, the fix is complex, we need some time.
Yes, those packages are identical to the production release, plus the fix for rsync.
They will automatically update to stable next week.
1 Like
this is original permissions
[root@nserver ~]# ls -la /var/lib/nethserver/ibay/test
total 0
drwxrws— 5 root buero@test.vm 48 Nov 14 18:22 .
drwxrwxr-x. 3 root root 18 Nov 12 18:47 …
drwxrws— 2 mario@test.vm buero@test.vm 158 Nov 12 19:04 Test 1
drwxrws— 2 mario@test.vm buero@test.vm 158 Nov 13 21:28 Test 2
drwxrws— 2 mario@test.vm buero@test.vm 241 Nov 13 21:41 Test 3
[root@nserver ~]#
and this is after restore from restic backup
[root@nserver ~]# ls -la /var/lib/nethserver/ibay/test
total 0
drwxrws— 6 root buero@test.vm 66 Nov 14 18:28 .
drwxrwxr-x. 3 root root 18 Nov 12 18:47 …
drwx–S— 2 root buero@test.vm 158 Nov 14 18:28 Test 1
drwxrws— 2 mario@test.vm buero@test.vm 158 Nov 12 19:04 Test 1 old
drwx–S— 2 root buero@test.vm 158 Nov 14 18:27 Test 2
drwxrws— 2 mario@test.vm buero@test.vm 241 Nov 14 18:28 Test 3
[root@nserver ~]#
Hope it helps you
Thanks for the details, I’ll try to reproduce the problem tomorrow, using a shared folder like yours.
Meanwhile I fixed the duplicity problem with “strange” characters in filenames
2 Likes
I fully reproduced your setup, but I still can’t reproduce the problem. I restored a directory and it had its permissions and owner correctly restored.
1 Like
Thanks for your efforts.
For me the problem remains on actual setup.
I’ll try to make a new test setup as soon as possible.
At the moment we have a lot work, so give me a few days.
1 Like
Created totally new setup for testing.
Made shared folder “test” with owning group “buero@test.vm” and write permissions for owning group.
Copied a few folders with subfolders and files in there.
Made backup with restic.
Deleted all content from shared folder “test”.
Restored all data from shared folder “test”.
All folders and subfolders and files reappeared in shared folder “test”.
But sorry to say, not accessible from connected samba user.
Problem is still the same, all directly subfolders from shared folder “test” were restored with owner “root” and group “buero@test.vm” but without any permissions for the group.
All further subfolders and files have restored original owner and group and permissions for both.
Sorry, but no more ideas what else to test.
Tell me if you need other informations.
Meanwhile I’m happy with rsync.
Today updated to “nethserver-backup-data 1.7.0-1” and “nethserver-rstore-data 2.0.3-1”.
Restic still not working for me, all folders are restored with owner changed to “root” and no group permissions.
But something has changed since update, because before just the first directly subfolder of shared folder was restored with wrong owner root, but now after update even all subsequent subfolders are changed to owner root and no group permissions.
Rsync still working correctly for me.
Could the File server > Settings option for the shared folder be affecting the restored directories and files?
When directory is created in share folder:
- Grant full control to the owner of the parent dir
- Grant full control to the creator
Tested with both settings.
Result ist the same for both settings.
Don’t think it relies on this, because rsync works well.
Could you reproduce my issue, or it’s me doing something wrong?
Yes, I can reproduce the problem on restored folders.
Before restore:
[root@server ~]# ls -al /var/lib/nethserver/ibay/test/*
-rw-rw---- 1 mario@domain.tld buero@domain.tld 45065 Mar 3 2018 /var/lib/nethserver/ibay/test/renamed Screenshot from 2018-03-03 11-31-54.png
/var/lib/nethserver/ibay/test/folder one:
total 0
drwxrws--- 2 mario@domain.tld buero@domain.tld 6 Nov 24 21:32 .
drwxrws--- 6 root buero@domain.tld 121 Nov 24 21:32 ..
/var/lib/nethserver/ibay/test/test 1:
total 24
drwxrws--- 3 mario@domain.tld buero@domain.tld 160 Nov 24 21:33 .
drwxrws--- 6 root buero@domain.tld 121 Nov 24 21:32 ..
-rw-rw---- 1 mario@domain.tld buero@domain.tld 159 Nov 12 00:00 2019-11-12-00-00-03.043-VirtualBox-22683.log
-rw-rw---- 1 mario@domain.tld buero@domain.tld 168 Nov 16 15:57 2019-11-16-15-57-40.062-VirtualBox-17451.log
-rw-rw---- 1 mario@domain.tld buero@domain.tld 14520 Nov 9 21:27 disabled field bypass.png
drwxrws--- 2 mario@domain.tld buero@domain.tld 6 Nov 24 21:33 subtest 1
/var/lib/nethserver/ibay/test/test 2:
total 20
drwxrws--- 2 mario@domain.tld buero@domain.tld 47 Nov 24 21:33 .
drwxrws--- 6 root buero@domain.tld 121 Nov 24 21:32 ..
-rw-rw---- 1 mario@domain.tld buero@domain.tld 16874 Mar 30 2018 onlyoffice-error-after-update.png
/var/lib/nethserver/ibay/test/test 3:
total 300
drwxrws--- 2 mario@domain.tld buero@domain.tld 51 Nov 24 21:33 .
drwxrws--- 6 root buero@domain.tld 121 Nov 24 21:32 ..
-rw-rw---- 1 mario@domain.tld buero@domain.tld 306781 Nov 5 13:15 Service Mode & Tools Instructions.pdf
After renaming/deleting some files and folders and doing a restore:
[root@server ~]# ls -al /var/lib/nethserver/ibay/test/*
-rw-rw---- 1 mario@domain.tld buero@domain.tld 45065 Mar 3 2018 /var/lib/nethserver/ibay/test/renamed Screenshot from 2018-03-03 11-31-54.png
-rw-rw---- 1 mario@domain.tld buero@domain.tld 45065 Mar 3 2018 /var/lib/nethserver/ibay/test/rerenamed Screenshot from 2018-03-03 11-31-54.png
/var/lib/nethserver/ibay/test/folder one:
total 0
drwxrws--- 2 mario@domain.tld buero@domain.tld 6 Nov 24 21:32 .
drwxrws--- 7 root buero@domain.tld 196 Nov 24 21:40 ..
/var/lib/nethserver/ibay/test/test 1:
total 24
drwxrws--- 3 mario@domain.tld buero@domain.tld 160 Nov 24 21:40 .
drwxrws--- 7 root buero@domain.tld 196 Nov 24 21:40 ..
-rw-rw---- 1 mario@domain.tld buero@domain.tld 159 Nov 12 00:00 2019-11-12-00-00-03.043-VirtualBox-22683.log
-rw-rw---- 1 mario@domain.tld buero@domain.tld 168 Nov 16 15:57 2019-11-16-15-57-40.062-VirtualBox-17451.log
-rw-rw---- 1 mario@domain.tld buero@domain.tld 14520 Nov 9 21:27 disabled field bypass.png
drwxrws--- 2 mario@domain.tld buero@domain.tld 6 Nov 24 21:33 subtest 1
/var/lib/nethserver/ibay/test/test 2:
total 20
drwx--S--- 2 root buero@domain.tld 47 Nov 24 21:40 .
drwxrws--- 7 root buero@domain.tld 196 Nov 24 21:40 ..
-rw-rw---- 1 mario@domain.tld buero@domain.tld 16874 Mar 30 2018 onlyoffice-error-after-update.png
/var/lib/nethserver/ibay/test/test 3:
total 300
drwx--S--- 2 root buero@domain.tld 51 Nov 24 21:40 .
drwxrws--- 7 root buero@domain.tld 196 Nov 24 21:40 ..
-rw-rw---- 1 mario@domain.tld buero@domain.tld 306781 Nov 5 13:15 Service Mode & Tools Instructions.pdf
/var/lib/nethserver/ibay/test/test 3.old:
total 300
drwxrws--- 2 mario@domain.tld buero@domain.tld 51 Nov 24 21:33 .
drwxrws--- 7 root buero@domain.tld 196 Nov 24 21:40 ..
-rw-rw---- 1 mario@domain.tld buero@domain.tld 306781 Nov 5 13:15 Service Mode & Tools Instructions.pdf
[root@server ~]# getfacl /var/lib/nethserver/ibay/test/
getfacl: Removing leading '/' from absolute path names
# file: var/lib/nethserver/ibay/test/
# owner: root
# group: buero@domain.tld
# flags: -s-
user::rwx
group::rwx
other::---
[root@server ~]# getfacl /var/lib/nethserver/ibay/test/*
getfacl: Removing leading '/' from absolute path names
# file: var/lib/nethserver/ibay/test/folder one
# owner: mario@domain.tld
# group: buero@domain.tld
# flags: -s-
user::rwx
group::rwx
other::---
# file: var/lib/nethserver/ibay/test/renamed Screenshot from 2018-03-03 11-31-54.png
# owner: mario@domain.tld
# group: buero@domain.tld
user::rw-
group::rw-
other::---
# file: var/lib/nethserver/ibay/test/rerenamed Screenshot from 2018-03-03 11-31-54.png
# owner: mario@domain.tld
# group: buero@domain.tld
user::rw-
group::rw-
other::---
# file: var/lib/nethserver/ibay/test/test 1
# owner: mario@domain.tld
# group: buero@domain.tld
# flags: -s-
user::rwx
group::rwx
other::---
# file: var/lib/nethserver/ibay/test/test 2
# owner: root
# group: buero@domain.tld
# flags: -s-
user::rwx
group::---
other::---
# file: var/lib/nethserver/ibay/test/test 3
# owner: root
# group: buero@domain.tld
# flags: -s-
user::rwx
group::---
other::---
# file: var/lib/nethserver/ibay/test/test 3.old
# owner: mario@domain.tld
# group: buero@domain.tld
# flags: -s-
user::rwx
group::rwx
other::---
From the command line (with user root), restoring directly through restic -r ..., restore-data, restore-file and restore-data-restic permissions are preserved. From restore data UI, without checking overwrite option, previously deleted folders are not restored. Checking overwrite option, folders are restored but don’t retain original permissions.
1 Like
I’ve found the issue:
@edoardo_spadoni is looking for a workaround.
I’ll try to have a fix from upstream.
2 Likes
1 Like