Backup-data: multiple schedule and backends

Prune could (or could not) overlap time on backup via restic backend. Which could generate an error.
I’m asking if could be possible (or logical) have the option “prune before backup”, or add a field/option for “on which occasion restic backup should be pruned”…
@giacomo is week the only timespace? Could be more time-to-gained-space efficient prune once a month?

It’s almost what I was proposing. The only difference is that prune will take place after the backup (not before).

I guess yes on machines where only few files are deleted. I was proposing it on a weekly timing because the implementation is very very easy :wink:

IIRC /var/log/backup was empty when I tried, and /var/log/last-backup.log was referenced on stdout/stderr after backup failed or completed. Also this log was used to read the last backup data shown on dashboard for duplicity engine.

1 Like

The log is generated only if the backup is invoked using backup-data-wrapper, otherwise the output is sent to stdout.
I also removed the wrong reference to the old unused /var/log/last-backup.log file.

This part has been already implemented using hooks (https://github.com/NethServer/nethserver-backup-data/blob/master/root/etc/backup-data.hooks/backup-dashboard-status).

I also tried to fix the test case 4, but it’s a very big refactor and the packages needs a full QA again :weary:
If you still have the machine could you please verify the bug is gone at least for the test case 4?
Use the latest package from this PR: https://github.com/NethServer/nethserver-backup-data/pull/22

As a side note, this refactor needs also a new PR for nethserver-duc which should now exclude all backup mount directories like /mnt/backup-* (I hope @edoardo_spadoni can help here).

Components
nethserver-backup-data-1.3.4-1.61.pr22.ge55f992.ns7.noarch

Now mount is working as expected. :wink:

Single Backup Mode: last backup end time on dashboard is off by 2 hours (ex. 20:07). Log filename and content has the correct time (ex. 22:07, same as date command). Backup called from backup-data-wrapper.

MULTIPLE BACKUPS
test case 4 (duplicity, restic; cifs, nfs):

  • if custom inclusion is set: global backup inclusion not respected. Only files on custom inclusion are backed up.
  • backup-data-list worked
  • concurrent backups not tested

This should depends on the different time zones for PHP an the system.

This is the expected behavior (https://github.com/NethServer/nethserver-backup-data#multiple-backups-2) maybe we should improve the doc.

Thank you Marc, as always your testing is really appreciated and extremely useful!

multiple backups read the same configuration of the single backup.
(…) file will override the list on included and excluded files from the single backup

OK, so /etc/backup-data/*.include takes preference over /etc/backup-data.d/custom.include

Sorry, by using “global inclusion” wording didn’t make myself clear. If custom inclusion is set for the multiple backup job then NOTHING else is backed up (no /var/lib/nethserver/* …)

It’s just like this. I will try to explain myself with an example.
Let’s configure duplicifs multiple backup and create a file named /etc/backup-data/duplicifs.include.
The backup will contain only files listed inside /etc/backup-data/duplicifs.include, except the global excludes.
If we create a file named /etc/backup-data/duplicifs.exclude, list of exclude files is read only from this file.

Why this implementation? Because it’s very flexible and allow the creation of backups with limited data set.
You can have the single backup saving everything, and a new mailbackup which backups only the mail every hour.
To achieve this you need to create two files:

  1. /etc/backup-data/backup.include which contains only:
    /var/lib/nethserver/vmail
    
  2. an empty /etc/backup-data/backup.exclude

If something is still unclear, please ask! :slight_smile:

1 Like

OK, didn’t get the idea from docs, now I understand. Thanks.

1 Like

Just added the same example to the administrator manual.

After some tests, we also implemented the restic Prune option once a week.
On our backup, a daily prune takes around 2 hours; if it’s execute once a week it takes around 6 hours.
If the prune is executed less often (like once a month), the process will be really slow and could clash with the next backup job. (/cc @pike @m.traeumner).

test case 4 (a-c) (duplicity, restic; cifs, nfs): OK
test case 4 (a-c) (rsync; nfs): OK
test case 5 (restic, rsync; sftp): OK

2 Likes

If it is possible to save an NFS/SAMBA to an RDX tape, which has 5 different tapes in a /dev/sdc1, I have to create 5 jobs, which is not possible, because there could only be one tape per day. Would I have to do a full backup every time?

Sorry but tape is not supported, but I know @filippo_carletti would like to add a custom script for it.
Just try to add your tar-based script inside the post-backup event.

1 Like

RDX is not a real tape, but a USB drive with removable disks, it is only recognized as tape by some backup software. I have now solved my one cronjob that performs a rsync and with the Sogo tool I make a backup to the medium.

RDX should be configured as USB.

1 Like

@kunstlust does RDX support eject of the “cartrdige” from shell?

Yes eject /dev/sdx

Hey @giacomo,

I just configured a simple (not multiple) rsync backup to a NFS share, launched backup-data -b mybackup some times and noticed those lines at the beginning of the /root/.rsync_tmbackup/xxx.log file :slight_smile:

2018/09/17 21:40:47 [14466] building file list                                                                                                                                          │
2018/09/17 21:40:47 [14466] rsync: chown "/mnt/backup-TimeMachine/cloud/2018-09-17-214047/root" failed: Operation not permitted (1)                                                 │
2018/09/17 21:40:47 [14466] rsync: chown "/mnt/backup-TimeMachine/cloud/2018-09-17-214047/root/.byobu" failed: Operation not permitted (1)        

And so on for all folders already existing on the NFS share.

Don’t know what to think. You ?

It could depend on permission mapping on the NFS.

Try to mount the filesystem, anche with ‘stat’ what are the file permissions.

stat /mnt/backup-backup-data/cloud/2018-09-18-201709/root/LOG_imapsync/2018_09_17_15_44_44_olivia\@lebrass.be.txt 
  File: ‘/mnt/backup-backup-data/cloud/2018-09-18-201709/root/LOG_imapsync/2018_09_17_15_44_44_olivia@lebrass.be.txt’
  Size: 11076           Blocks: 23         IO Block: 262144 regular file
Device: 29h/41d Inode: 326420      Links: 1
Access: (0600/-rw-------)  Uid: (  100/ UNKNOWN)   Gid: (  100/   users)
Access: 1970-01-01 07:10:45.000000000 +0100
Modify: 2018-09-18 20:17:21.492797411 +0200
Change: 2018-09-18 20:17:21.504797569 +0200
 Birth: -