Backup-data: multiple schedule and backends

[giacomo (Giacomo Sanchietti)]

Test case 1 (Primary backup) - Success but last backup end time is wrong(UTC instead of local timezone) in dashboard after testing update:

The dashboard had few issues, it should be fixed now.

Test case 3a - green banner after restore but testfiles in directory root were not restored

I think that actually was correctly restored, but the UI didn’t report the right path.

Test case 3b

I can’t reproduce it.
Could you please post the output of:

config show backup-data
/etc/e-smith/events/actions/mount-<yourfs> && ls /mnt/backup/<hostname>/
restore-data

Reported issues should be fixed.

[mrmarkuz (Markus Neuberger)]

Here you are…

[root@testserver ~]# config show backup-data
backup-data=configuration
    BackupTime=0:06
    CleanupOlderThan=28D
    FullDay=0
    IncludeLogs=disabled
    LogFile=/var/log/last-backup.log
    Mount=/mnt/backup
    NFSHost=
    NFSShare=
    Notify=error
    NotifyFrom=
    NotifyTo=root@localhost
    Program=rsync
    SMBHost=192.168.1.100
    SMBLogin=
    SMBPassword=
    SMBShare=backuprsync
    Type=incremental
    USBLabel=
    VFSType=cifs
    VolSize=250
    WebDAVLogin=
    WebDAVPassword=
    WebDAVUrl=
    status=enabled

[root@testserver ~]# /etc/e-smith/events/actions/mount-cifs && ls /mnt/backup/testserver/
2018-07-06-010120  backup-config.tar.xz  backup.marker

[root@testserver ~]# restore-data
Restore started at 2018-07-06 15:35:09
Event pre-restore-data: SUCCESS
/usr/bin/rsync -D --compress --numeric-ids --links --hard-links --one-file-system --itemize-changes --times --recursive --perms --owner --group --stats --human-readable /mnt/backup/testserver/latest/  /
rsync: change_dir "/mnt/backup/testserver/latest" failed: No such file or directory (2)

Number of files: 0
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 20
Total bytes received: 12

sent 20 bytes  received 12 bytes  64.00 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
Action '/etc/e-smith/events/actions/restore-data-rsync': SUCCESS
Event post-restore-data: SUCCESS
Restore ended at 2018-07-06 15:35:12
Time elapsed: 0 hours, 0 minutes, 3 seconds

[giacomo (Giacomo Sanchietti)]

[root@testserver ~]# /etc/e-smith/events/actions/mount-cifs && ls /mnt/backup/testserver/ 2018-07-06-010120 backup-config.tar.xz backup.marker

This is the real error: inside the directory the scripts should create a file named latest which is an hard link pointing to the latest snapshot.
Does you remote fs support hard links?

[mrmarkuz (Markus Neuberger)]

Does you remote fs support hard links?

It’s a Nethserver samba share, so xfs under cifs but it seems not to support hard links. I tried some samba options but no success.

ln: failed to create symbolic link ‘/mnt/backup/testserver/latest’: Operation not supported

[giacomo (Giacomo Sanchietti)]

It’s a Nethserver samba share, so xfs under cifs but it seems not to support hard links. I tried some samba options but no success.

This is quite interesting! It’s probably some missing samba option…

I will test it right on Monday!

[giacomo (Giacomo Sanchietti)]

Samba doesn’t support symlinks (and it also has some problems with hardlinks) without proper insecure configuration.
I wasn’t even able to have working symlink with special config.

So I guess we should add a note to not use CIFS mounts on NethServer with rsync backup.

What do you think @mrmarkuz ?

[mrmarkuz (Markus Neuberger)]

Fully agree, so users are warned.

[giacomo (Giacomo Sanchietti)]

Fully agree, so users are warned.

Done!

Anyone else want to join the testing? If you have concerns, I just have to say that we have this implementation on production since a couple of weeks!

[pagaille (Matthieu Gaillet)]

I started to test today ! Here are some comments and some reporting :

• Regarding the naming (“Main”, “secondary”, "single, “multiple”, …) considerations, why is this differentiation simply needed ? I find this differentiation really misleading, especially without an UI. For me a single backup should really be nothing more than a multiple backup with only one configuration. Keeping the “old” way of doing things will only introduce confusion and doesn’t seem to have any advantage except maybe some backward compatibility (?)

• Regarding the documentation : the “Single backup” configuration states that the BackupTime key should be expressed in the hh:mm form, while the “Multiple backup” entry should be in the cron syntax. Again, this is misleading.

• Entry CleanupOlderThan is (should !) useless when used with the rsync backend, it should be specified in the doc. I can open a PR if someone confirms that.

• duc index : I’m still not convinced about this I find this tool being really ressource hungry and not optimised (the index must be entirely rebuilt each time a backup is launched, imagine indexing a multi-terra file server being backuped each hour !).
  For the rsync backend the duc index is even not useful when restoring since the file list can be easily built by listing the files in the “latest” backup folder.
  For the restic backend a snapshot listing can also be done using restic ls -l latest
  Conclusion : for me this index should only be created when needed if needed at all.

• I tried to configure my system using this configuration :

Primary backup disabled :

# config show backup-data
backup-data=configuration
BackupTime=23:00
CleanupOlderThan=14D
FullDay=1
IncludeLogs=disabled
LogFile=/var/log/last-backup.log
Mount=/mnt/backup
NFSHost=
NFSShare=
Notify=always
NotifyFrom=
NotifyTo=root@localhost
Program=rsync
SMBHost=
SMBLogin=
SMBPassword=
SMBShare=
Type=incremental
USBLabel=external
VFSType=usb
VolSize=250
WebDAVLogin=
WebDAVPassword=
WebDAVUrl=
status=disabled

Secondary backup configured to run every hour and to backup to the usb disk labelled “external” using rsync backend :

# db backups show
t1=rsync
    BackupTime=0 * * * *
    CleanupOlderThan=default
    Notify=always
    NotifyFrom=
    NotifyTo=root@localhost
    USBLabel=external
    VFSType=usb
    status=enabled

Then notified the system about this new backup scheme :

signal-event nethserver-backup-data-save t1

I launched the backup manually and it successfully found my previous backups an continued to incrementally backup nice !

I’ll report later if needed.

THANKS for the good work !

[pagaille (Matthieu Gaillet)]

Update : looks like if the main backup is disabled, the cron job for the secondary backup isn’t created, therefore the backup doesn’t start.

[giacomo (Giacomo Sanchietti)]

Keeping the “old” way of doing things will only introduce confusion and doesn’t seem to have any advantage except maybe some backward compatibility

We can’t remove it, we must preserve backward compatibility with thousands servers.
But we can hide the single backup when we will implement a new UI using cockpit.

Again, this is misleading.

I agree, but this is the most flexible implementation. We will hide also this complexity.

Entry CleanupOlderThan is (should !) useless when used with the rsync backend, it should be specified in the doc. I can open a PR if someone confirms that.

Already removed in this commit: https://github.com/NethServer/nethserver-rsync/commit/2da9d79eff92f5dc0ee92a4e11b6b85a74487a8e
(Even if upstream is implementing it, but there are some bugs).

I forgot to remove it from the README, just done.

duc index : I’m still not convinced about this

No one is. It’s bad hack for a problem without a simple solution.

the index must be entirely rebuilt each time a backup is launched

No is not: the index is incremental, but still it’s way too slow!

For the rsync backend the duc index is even not useful when restoring since the file list can be easily built by listing the files in the “latest” backup folder.

This is true from the command line, but not from the UI which currently needs to create a tree.

Update : looks like if the main backup is disabled, the cron job for the secondary backup isn’t created, therefore the backup doesn’t start.

This is an error inside the cron.d template, already fixed. Thank you for reporting!

[pike (Michael Kicks)]

question time…
Restic seems most interesting engine. Is there any plan to integrate a REST server into NethServer?

[giacomo (Giacomo Sanchietti)]

Restic seems most interesting engine. Is there any plan to integrate a REST server into NethServer?

Not officially for now, but you can find the instructions here: https://github.com/NethServer/nethserver-restic/blob/master/README.rst

[pagaille (Matthieu Gaillet)]

Hi @giacomo

We can’t remove it, we must preserve backward compatibility with thousands servers.
But we can hide the single backup when we will implement a new UI using cockpit.

Okay. Still not sure to see the backward compatibily issue (looks easier to migrate old settings to fit the new backup system) but I trust you.

I forgot to remove it from the README, just done.

Don’t forget the documentation.

duc index : I’m still not convinced about this

No one is. It’s bad hack for a problem without a simple solution.

Remember we already share some thoughts about this :

Duplicity replacement --> rsync Time machine-like backups

I agree. My general feeling is that we don’t really need to create any index of any sort regarding backups. Just browse the files & folders tree on the backup disk, done. I believe the way Duplicity stored the files on the disk made the indexing step mandatory. But it is not needed here.

Duplicity replacement --> rsync Time machine-like backups

on the other hand I guess that we can use the log file (if you want to have the files parsed and indexed) Rsync can output the activity or even generate the listing in a file in a specified format, if I’m not mistaking. This way you have the index files generated from the log. Easy to parse and use. Even tar (or most archivers) can generate a listing of the files (including the paths) if needed.

We could also use some Lucene on the fly indexing if we really want the UI be super responsive instead of building a file list when loading the UI page. I’m not sure it’s worth the trouble : someone wanting to restore some files will easily accept to wait some seconds when the list of files is being built.

This is an error inside the cron.d template, already fixed. Thank you for reporting!

My pleasure !

[giacomo (Giacomo Sanchietti)]

I just merged nethserver-restic and nethserver-rsync inside nethserver-backup-data package.
restic binary has been moved to a restic rpm which can be eventually switched for non-x86_64 architectures.

If you already installed the packages from testing, you need to execute the following:

yum --enablerepo=nethserver-testing clean all
rpm -e nethserver-restic nethserver-rsync --nodeps
yum --enablerepo=nethserver-testing update \*backup\*

[hector (Héctor Pérez)]

Ok, this is a work for my new virtualization environment (an old CompaqCQ45 with 8Gbytes of ram), so hands on!!!

[dnutan (Marc)]

Components

• nethserver-backup-data-1.3.4-1.53.g7dc6bc3.ns7.noarch
• nethserver-restore-data-1.2.4-1.2.g5c84689.ns7.noarch

TESTS
test case 1: OK (but last backup info not shown on dashboard, for this and for any other Single Backup test case)
test case 2(a-c) (restic; cifs, nfs, webdav): OK
test case 3(a-c) (rsync; nfs): OK (chown failures on previous test were due to bad nfs config on destination)
test case 3(a-c) (rsync; cifs, webdav): FAILED

• Backup is done
• Can list backup files (backup-data-list)
• Cannot restore files: from server-manager it says the folders were restored but they were not. From CLI reports symlink problem.

results of restore-data command

# restore-data 
Restore started at 2018-07-21 20:16:03
Event pre-restore-data: SUCCESS
rsync: change_dir "/mnt/backup/server/latest" failed: No such file or directory (2)

Number of files: 0
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 20
Total bytes received: 12

sent 20 bytes  received 12 bytes  64.00 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
Action '/etc/e-smith/events/actions/restore-data-rsync': SUCCESS

Result of test case 3 with rsync and webdav

rsync: failed to set permissions on "/mnt/backup/server/2018-07-25-151336/var/lib/nethserver/ibay/sharedfolder": Invalid argument (22)
rsync: mkstemp "/mnt/backup/server/2018-07-25-151336/var/lib/nethserver/nextcloud/.htaccess.EMiL21" failed: Invalid argument (22)
ln: failed to create symbolic link ‘/mnt/backup/server/latest’: Function not implemented
Backup failed
Action 'backup-data-rsync ': FAIL
Backup status: FAIL

Cause of failure:
As reported and explained before, symlink error:

rsync_tmbackup: Backup completed without errors.
ln: failed to create symbolic link ‘/mnt/backup/server/latest’: Operation not supported
Action 'backup-data-rsync ': SUCCESS

---

General problems
Using single backup with any engine /var/log/last-backup.log is not created.
Log also shows Requested path not found:

esmith::event[4719]: Action: /etc/e-smith/events/pre-backup-data/S20nethserver-backup-config-predatabackup SUCCESS [1.203213]
esmith::event[4719]: Requested path not found

Many problems using webdav on single backup mode with any engine. I suspect the webdav server. Reports of open files exceeding max cache size, problems removing locks, or duplicity’s remote manifest not matching local one. (EDIT: using a different webdav local destination server, no problems).

[dnutan (Marc)]

MULTIPLE BACKUPS TEST

Questions

• Multiple Backups + Duplicity: Are these properties supported: Type, FullDay, VolSize? Yes
• Is IncludeLogs prop supported by Multiple Backups?

---

test case 4 (duplicity; cifs): FAILED

Details of test case 4 (duplicity; cifs)

# db backups show duplicifs 
duplicifs=duplicity
    BackupTime=0 23 * * *
    CleanupOlderThan=10D
    FullDay=6
    Notify=error
    NotifyFrom=
    NotifyTo=root@localhost
    SMBHost=192.168.1.140
    SMBLogin=nethserver
    SMBPassword=********
    SMBShare=samba
    Type=incremental
    VFSType=cifs
    VolSize=250
    status=enabled

# backup-data -b duplicifs
Backup: duplicifs
Backup started at 2018-07-25 22:48:14
Pre backup scripts status: SUCCESS
umount: /mnt/backup-duplicifs: not mounted
Backup directory is not mounted
Can't mount /mnt/backup-duplicifs
Action 'backup-data-duplicity duplicifs': FAIL
Backup status: FAIL

# /etc/e-smith/events/actions/mount-cifs duplicifs
# touch /mnt/backup-duplicifs/file.txt
# ls /mnt/backup-duplicifs/
file.txt

Log:

Jul 25 23:01:39 server esmith::event[17604]: Event: pre-backup-data
Jul 25 23:01:39 server esmith::event[17604]: ===== Report for configuration backup =====
Jul 25 23:01:39 server esmith::event[17604]: 
Jul 25 23:01:39 server esmith::event[17604]: Backup started at 2018-07-25 23:01:39
Jul 25 23:01:39 server esmith::event[17607]: Event: pre-backup-config
Jul 25 23:01:39 server esmith::event[17607]: expanding /etc/backup-config.d/nethserver-sssd.include
Jul 25 23:01:39 server esmith::event[17607]: Action: /etc/e-smith/events/actions/generic_template_expand SUCCESS [0.191435]
Jul 25 23:01:39 server esmith::event[17607]: Action: /etc/e-smith/events/pre-backup-config/S20nethserver-directory-dump-ldap SUCCESS [0.08888]
Jul 25 23:01:39 server esmith::event[17607]: Action: /etc/e-smith/events/pre-backup-config/S40nethserver-sssd-backup-tdb SUCCESS [0.00395]
Jul 25 23:01:40 server esmith::event[17607]: Action: /etc/e-smith/events/pre-backup-config/S50nethserver-backup-config-list-packages SUCCESS [0.693899]
Jul 25 23:01:40 server esmith::event[17607]: Event: pre-backup-config SUCCESS
Jul 25 23:01:40 server esmith::event[17604]: Event pre-backup-config: SUCCESS
Jul 25 23:01:40 server esmith::event[17604]: Action backup-config-execute: SUCCESS
Jul 25 23:01:40 server esmith::event[17708]: Event: post-backup-config
Jul 25 23:01:40 server esmith::event[17708]: Event: post-backup-config SUCCESS
Jul 25 23:01:40 server esmith::event[17604]: Event post-backup-config: SUCCESS
Jul 25 23:01:40 server esmith::event[17604]: Backup status: SUCCESS
Jul 25 23:01:40 server esmith::event[17604]: Backup ended at 2018-07-25 23:01:40
Jul 25 23:01:40 server esmith::event[17604]: Time elapsed: 0 hours, 0 minutes, 1 seconds
Jul 25 23:01:40 server esmith::event[17604]: Action: /etc/e-smith/events/pre-backup-data/S20nethserver-backup-config-predatabackup SUCCESS [1.305231]
Jul 25 23:01:41 server esmith::event[17604]: Requested path not found
Jul 25 23:01:41 server esmith::event[17604]: Action: /etc/e-smith/events/pre-backup-data/S20nethserver-restore-data-duc-index SUCCESS [0.661085]
Jul 25 23:01:41 server esmith::event[17604]: Action: /etc/e-smith/events/pre-backup-data/S50mysql-dump-tables SUCCESS [0.191907]
Jul 25 23:01:41 server esmith::event[17604]: Action: /etc/e-smith/events/pre-backup-data/S50nethserver-ibays-dump-acls SUCCESS [0.003299]
Jul 25 23:01:41 server esmith::event[17604]: Action: /etc/e-smith/events/pre-backup-data/S70mount-cifs SUCCESS [0.094699]
Jul 25 23:01:41 server esmith::event[17604]: Action: /etc/e-smith/events/pre-backup-data/S70mount-nfs SUCCESS [0.094283]
Jul 25 23:01:41 server esmith::event[17604]: Action: /etc/e-smith/events/pre-backup-data/S70mount-usb SUCCESS [0.091106]
Jul 25 23:01:42 server esmith::event[17604]: Action: /etc/e-smith/events/pre-backup-data/S70mount-webdav SUCCESS [0.094998]
Jul 25 23:01:42 server esmith::event[17604]: Event: pre-backup-data SUCCESS

Cause of failure: /mnt/backup is mounted instead of the specific mountpoint for the backup task, even if Mount property for the backup job is set.

[filippo_carletti]

Is IncludeLogs prop supported by Multiple Backups?

Yes, multiple backups honor the main backup settings, unless customized.

[filippo_carletti]

Restic retention

Restic deletes snapshots according to the retention policy, but disk space is not reclaimed automatically, you have to run a prune operation.
See https://restic.net/blog/2016-08-22/removing-snapshots for details.

NethServer forces pruning of snapshots to reclaim space on every backup, to mimic duplicity.
But prune is a really expensive operation. Real life example: a backup of about 250G with 2.5 million files and about 1G of changes every day is completed in less than 10 minutes on a 30 gbit/s link.
The same backup takes about 2.5 hours to be pruned.

And, due to deduplication, the prune operation frees a little space. I think we are wasting resources (cpu and time) for a very little benefit.

I propose to run prune only once in a while (maybe every week) as a different cron job.
We could add an option to select the prune frequency, but I don’t like to add another option.

What do you think?