[root@testserver ~]# /etc/e-smith/events/actions/mount-cifs && ls /mnt/backup/testserver/
2018-07-06-010120 backup-config.tar.xz backup.marker
[root@testserver ~]# restore-data
Restore started at 2018-07-06 15:35:09
Event pre-restore-data: SUCCESS
/usr/bin/rsync -D --compress --numeric-ids --links --hard-links --one-file-system --itemize-changes --times --recursive --perms --owner --group --stats --human-readable /mnt/backup/testserver/latest/ /
rsync: change_dir "/mnt/backup/testserver/latest" failed: No such file or directory (2)
Number of files: 0
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 20
Total bytes received: 12
sent 20 bytes received 12 bytes 64.00 bytes/sec
total size is 0 speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
Action '/etc/e-smith/events/actions/restore-data-rsync': SUCCESS
Event post-restore-data: SUCCESS
Restore ended at 2018-07-06 15:35:12
Time elapsed: 0 hours, 0 minutes, 3 seconds
This is the real error: inside the directory the scripts should create a file named latest which is an hard link pointing to the latest snapshot.
Does you remote fs support hard links?
Samba doesn’t support symlinks (and it also has some problems with hardlinks) without proper insecure configuration.
I wasn’t even able to have working symlink with special config.
So I guess we should add a note to not use CIFS mounts on NethServer with rsync backup.
I started to test today ! Here are some comments and some reporting :
Regarding the naming (“Main”, “secondary”, "single, “multiple”, …) considerations, why is this differentiation simply needed ? I find this differentiation really misleading, especially without an UI. For me a single backup should really be nothing more than a multiple backup with only one configuration. Keeping the “old” way of doing things will only introduce confusion and doesn’t seem to have any advantage except maybe some backward compatibility (?)
Regarding the documentation : the “Single backup” configuration states that the BackupTime key should be expressed in the hh:mm form, while the “Multiple backup” entry should be in the cron syntax. Again, this is misleading.
Entry CleanupOlderThan is (should !) useless when used with the rsync backend, it should be specified in the doc. I can open a PR if someone confirms that.
duc index : I’m still not convinced about this I find this tool being really ressource hungry and not optimised (the index must be entirely rebuilt each time a backup is launched, imagine indexing a multi-terra file server being backuped each hour !).
For the rsync backend the duc index is even not useful when restoring since the file list can be easily built by listing the files in the “latest” backup folder.
For the restic backend a snapshot listing can also be done using restic ls -l latest
Conclusion : for me this index should only be created when needed if needed at all.
I tried to configure my system using this configuration :
We can’t remove it, we must preserve backward compatibility with thousands servers.
But we can hide the single backup when we will implement a new UI using cockpit.
I agree, but this is the most flexible implementation. We will hide also this complexity.
Okay. Still not sure to see the backward compatibily issue (looks easier to migrate old settings to fit the new backup system) but I trust you.
Don’t forget the documentation.
Remember we already share some thoughts about this :
We could also use some Lucene on the fly indexing if we really want the UI be super responsive instead of building a file list when loading the UI page. I’m not sure it’s worth the trouble : someone wanting to restore some files will easily accept to wait some seconds when the list of files is being built.
I just merged nethserver-restic and nethserver-rsync inside nethserver-backup-data package.
restic binary has been moved to a restic rpm which can be eventually switched for non-x86_64 architectures.
If you already installed the packages from testing, you need to execute the following:
TESTS test case 1: OK (but last backup info not shown on dashboard, for this and for any other Single Backup test case) test case 2(a-c)(restic; cifs, nfs, webdav): OK test case 3(a-c)(rsync; nfs): OK (chown failures on previous test were due to bad nfs config on destination) test case 3(a-c)(rsync; cifs, webdav): FAILED
Backup is done
Can list backup files (backup-data-list)
Cannot restore files: from server-manager it says the folders were restored but they were not. From CLI reports symlink problem.
results of restore-data command
# restore-data
Restore started at 2018-07-21 20:16:03
Event pre-restore-data: SUCCESS
rsync: change_dir "/mnt/backup/server/latest" failed: No such file or directory (2)
Number of files: 0
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 20
Total bytes received: 12
sent 20 bytes received 12 bytes 64.00 bytes/sec
total size is 0 speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
Action '/etc/e-smith/events/actions/restore-data-rsync': SUCCESS
Result of test case 3 with rsync and webdav
rsync: failed to set permissions on "/mnt/backup/server/2018-07-25-151336/var/lib/nethserver/ibay/sharedfolder": Invalid argument (22)
rsync: mkstemp "/mnt/backup/server/2018-07-25-151336/var/lib/nethserver/nextcloud/.htaccess.EMiL21" failed: Invalid argument (22)
ln: failed to create symbolic link ‘/mnt/backup/server/latest’: Function not implemented
Backup failed
Action 'backup-data-rsync ': FAIL
Backup status: FAIL
rsync_tmbackup: Backup completed without errors.
ln: failed to create symbolic link ‘/mnt/backup/server/latest’: Operation not supported
Action 'backup-data-rsync ': SUCCESS
General problems
Using single backup with any engine /var/log/last-backup.log is not created.
Log also shows Requested path not found:
esmith::event[4719]: Action: /etc/e-smith/events/pre-backup-data/S20nethserver-backup-config-predatabackup SUCCESS [1.203213]
esmith::event[4719]: Requested path not found
Many problems using webdav on single backup mode with any engine. I suspect the webdav server. Reports of open files exceeding max cache size, problems removing locks, or duplicity’s remote manifest not matching local one. (EDIT: using a different webdav local destination server, no problems).
Restic deletes snapshots according to the retention policy, but disk space is not reclaimed automatically, you have to run a prune operation.
See https://restic.net/blog/2016-08-22/removing-snapshots for details.
NethServer forces pruning of snapshots to reclaim space on every backup, to mimic duplicity.
But prune is a really expensive operation. Real life example: a backup of about 250G with 2.5 million files and about 1G of changes every day is completed in less than 10 minutes on a 30 gbit/s link.
The same backup takes about 2.5 hours to be pruned.
And, due to deduplication, the prune operation frees a little space. I think we are wasting resources (cpu and time) for a very little benefit.
I propose to run prune only once in a while (maybe every week) as a different cron job.
We could add an option to select the prune frequency, but I don’t like to add another option.