Backup-data: multiple schedule and backends

pike · July 30, 2018, 2:33pm

Prune could (or could not) overlap time on backup via restic backend. Which could generate an error.
I’m asking if could be possible (or logical) have the option “prune before backup”, or add a field/option for “on which occasion restic backup should be pruned”…
@giacomo is week the only timespace? Could be more time-to-gained-space efficient prune once a month?

giacomo · July 30, 2018, 3:25pm

It’s almost what I was proposing. The only difference is that prune will take place after the backup (not before).

I guess yes on machines where only few files are deleted. I was proposing it on a weekly timing because the implementation is very very easy

dnutan · July 31, 2018, 11:36am

IIRC /var/log/backup was empty when I tried, and /var/log/last-backup.log was referenced on stdout/stderr after backup failed or completed. Also this log was used to read the last backup data shown on dashboard for duplicity engine.

giacomo · July 31, 2018, 5:05pm

The log is generated only if the backup is invoked using backup-data-wrapper, otherwise the output is sent to stdout.
I also removed the wrong reference to the old unused /var/log/last-backup.log file.

This part has been already implemented using hooks (nethserver-backup-data/root/etc/backup-data.hooks/backup-dashboard-status at master · NethServer/nethserver-backup-data · GitHub).

I also tried to fix the test case 4, but it’s a very big refactor and the packages needs a full QA again
If you still have the machine could you please verify the bug is gone at least for the test case 4?
Use the latest package from this PR: Refactor mount by gsanchietti · Pull Request #22 · NethServer/nethserver-backup-data · GitHub

As a side note, this refactor needs also a new PR for nethserver-duc which should now exclude all backup mount directories like /mnt/backup-* (I hope @edoardo_spadoni can help here).

dnutan · July 31, 2018, 5:21pm

Components
nethserver-backup-data-1.3.4-1.61.pr22.ge55f992.ns7.noarch

Now mount is working as expected.

Single Backup Mode: last backup end time on dashboard is off by 2 hours (ex. 20:07). Log filename and content has the correct time (ex. 22:07, same as date command). Backup called from backup-data-wrapper.

MULTIPLE BACKUPS
test case 4 (duplicity, restic; cifs, nfs):

if custom inclusion is set: global backup inclusion not respected. Only files on custom inclusion are backed up.
backup-data-list worked
concurrent backups not tested

giacomo · August 1, 2018, 6:42am

This should depends on the different time zones for PHP an the system.

This is the expected behavior (GitHub - NethServer/nethserver-backup-data) maybe we should improve the doc.

Thank you Marc, as always your testing is really appreciated and extremely useful!

dnutan · August 1, 2018, 7:29am

multiple backups read the same configuration of the single backup.
(…) file will override the list on included and excluded files from the single backup

OK, so /etc/backup-data/*.include takes preference over /etc/backup-data.d/custom.include

Sorry, by using “global inclusion” wording didn’t make myself clear. If custom inclusion is set for the multiple backup job then NOTHING else is backed up (no /var/lib/nethserver/* …)

giacomo · August 1, 2018, 7:52am

It’s just like this. I will try to explain myself with an example.
Let’s configure duplicifs multiple backup and create a file named /etc/backup-data/duplicifs.include.
The backup will contain only files listed inside /etc/backup-data/duplicifs.include, except the global excludes.
If we create a file named /etc/backup-data/duplicifs.exclude, list of exclude files is read only from this file.

Why this implementation? Because it’s very flexible and allow the creation of backups with limited data set.
You can have the single backup saving everything, and a new mailbackup which backups only the mail every hour.
To achieve this you need to create two files:

/etc/backup-data/backup.include which contains only:
```
/var/lib/nethserver/vmail
```
an empty /etc/backup-data/backup.exclude

If something is still unclear, please ask!

dnutan · August 1, 2018, 7:56am

OK, didn’t get the idea from docs, now I understand. Thanks.

giacomo · August 1, 2018, 12:47pm

Just added the same example to the administrator manual.

After some tests, we also implemented the restic Prune option once a week.
On our backup, a daily prune takes around 2 hours; if it’s execute once a week it takes around 6 hours.
If the prune is executed less often (like once a month), the process will be really slow and could clash with the next backup job. (/cc @pike @m.traeumner).

dnutan · August 1, 2018, 2:02pm

test case 4 (a-c) (duplicity, restic; cifs, nfs): OK
test case 4 (a-c) (rsync; nfs): OK
test case 5 (restic, rsync; sftp): OK

kunstlust · August 14, 2018, 10:01am

If it is possible to save an NFS/SAMBA to an RDX tape, which has 5 different tapes in a /dev/sdc1, I have to create 5 jobs, which is not possible, because there could only be one tape per day. Would I have to do a full backup every time?

giacomo · August 15, 2018, 4:51pm

Sorry but tape is not supported, but I know @filippo_carletti would like to add a custom script for it.
Just try to add your tar-based script inside the post-backup event.

kunstlust · August 16, 2018, 5:53am

RDX is not a real tape, but a USB drive with removable disks, it is only recognized as tape by some backup software. I have now solved my one cronjob that performs a rsync and with the Sogo tool I make a backup to the medium.

filippo_carletti · August 16, 2018, 8:45am

RDX should be configured as USB.

pike · August 17, 2018, 1:10pm

@kunstlust does RDX support eject of the “cartrdige” from shell?

kunstlust · August 22, 2018, 6:05am

Yes eject /dev/sdx

pagaille · September 17, 2018, 7:51pm

Hey @giacomo,

I just configured a simple (not multiple) rsync backup to a NFS share, launched backup-data -b mybackup some times and noticed those lines at the beginning of the /root/.rsync_tmbackup/xxx.log file

2018/09/17 21:40:47 [14466] building file list                                                                                                                                          │
2018/09/17 21:40:47 [14466] rsync: chown "/mnt/backup-TimeMachine/cloud/2018-09-17-214047/root" failed: Operation not permitted (1)                                                 │
2018/09/17 21:40:47 [14466] rsync: chown "/mnt/backup-TimeMachine/cloud/2018-09-17-214047/root/.byobu" failed: Operation not permitted (1)

And so on for all folders already existing on the NFS share.

Don’t know what to think. You ?

giacomo · September 18, 2018, 11:19am

It could depend on permission mapping on the NFS.

Try to mount the filesystem, anche with ‘stat’ what are the file permissions.

pagaille · September 18, 2018, 6:20pm

stat /mnt/backup-backup-data/cloud/2018-09-18-201709/root/LOG_imapsync/2018_09_17_15_44_44_olivia\@lebrass.be.txt 
  File: ‘/mnt/backup-backup-data/cloud/2018-09-18-201709/root/LOG_imapsync/2018_09_17_15_44_44_olivia@lebrass.be.txt’
  Size: 11076           Blocks: 23         IO Block: 262144 regular file
Device: 29h/41d Inode: 326420      Links: 1
Access: (0600/-rw-------)  Uid: (  100/ UNKNOWN)   Gid: (  100/   users)
Access: 1970-01-01 07:10:45.000000000 +0100
Modify: 2018-09-18 20:17:21.492797411 +0200
Change: 2018-09-18 20:17:21.504797569 +0200
 Birth: -