Backup-data: multiple schedule and backends

(Alessio Fattorini) #1

@giacomo and @edoardo_spadoni are working on improving the current implementation of backup-data in order to add some requested features like:

  • new storage backends
  • encryption
  • multiple scheduled backups.

We know that NethServer users have been waiting a long time for that features. Finally, we did it but that’s just a first implementation. We need your feedback to improve and finalize it :man_factory_worker:

The new implementation is based on :three: different backends:

  • duplicity
  • restic
  • rsync (TimeMachine)

Features of multiple backups are:

  • a backup can be scheduled multiple time a day
  • customization of include and exclude
  • notifications
  • multiple backends: duplicity and restic
  • retention policy

Restic specific features:

  • storage backends: sFTP, S3, B2, REST server
  • encryption

Rsync specific features:

  • storage backends: sFTP

Known Limitations:

  • no backup/restore UI for multiple backups
  • restore-config from backup-data has been removed
  • webdav is not supported by multiple backup
  • disk usage report is not supported by multiple backup
  • push configuration backup to backup directory is not supported by multiple backup

@giacomo can we test something already?

For further information take a look at https://github.com/NethServer/dev/issues/5538

Help hacking backup script for rsync
We Are NethServer - Community Overview - Sep 18
Help hacking backup script for rsync
(Giacomo Sanchietti) #2

Everything is ready to be tested!
Also you can read a stub of the new manual section: http://docs.nethserver.org/en/latest/backup.html

@davidep already suggested to change the name from “Multiple backup” to “Secondary”. Even if I think we could change the name of feature, I’m not fully convinced to call it “Secondary”.
What do you think @filippo_carletti?
(This is the PR for the manual: https://github.com/NethServer/docs/pull/338#pullrequestreview-133634470)

I would like to thanks @pagaille for the preliminary work on the rsync part!

We tried to fulfill most of the requests gathered from the community, are we missing something?
I’d also like to hear opinions from @m.traeumner @dnutan @Ctek @etino @dz00te @danb35 @maxbet @JOduMonT @AndreLinux

(Matthieu Gaillet) #3

WOAH !!! Can’t wait to test this !!

(Alessio Fattorini) #4

Hey would you to steal my job mentiong people? :smile:
Good job BTW :ok_hand:

(Matthieu Gaillet) #5

As far as I can see, the UI is not updated at all - yet - ?

(Filippo Carletti) #6

I’d say “multiple backups” (plural) or “secondary backup” singular (but it may give the false idea that you can have only one secondary backup).
See:

(Alessio Fattorini) #7

Good point. I’d prefer “multiple backups” it sounds more powerful as well :wink:

(Marc) #8

Do you have in mind to extend restic with rclone? Just curious.
About encrypted rsync I recall rsyncrypto, but never tried it.

(Jonathan Dumont) #9

:back::up:

Restic :+1:t6:

For me this one is a big +

  1. Encryption
  2. Snapshot
  3. about to be drop almost anywhere

About the name :point_up_2:

Personally I prefer Multiple backup than Secondary
1st: If I only use this solution, it become my First Backup not my Secondary :wink:
2nd: If I decide to use all of them Rsync become my Third Backup, not my Secondary :stuck_out_tongue:

@giacomo you definitely make my day with this news.

(Giacomo Sanchietti) #10

It’s not and it will not be updated in the near future.
We have plan to implement it directly into cockpit, later this year.

Not for now, because the configuration is a little bit complex.
But if some users needs extra storage backend we could try to add it!

(Michael Träumner) #11

I think the header should only be backup and the tasks should be named by the users.

So one user can name the first task as backup to external USB and a second one as backup to network.
And other user would name his first task with backup weekdays and the second one with backup weekends. Internal you could link them to

task1; task2; task3; and so on

always use different destinations for each engine

Can somebody explain me why?

(Giacomo Sanchietti) #12

We can’t, because the “multiple backup” feature is not replacing the primary one. So we need a name to differentiate both features.

Because the backup engine needs to create its own directory structure which can conflict with other engines.
Moreover, let’s take the example of restic. The repository is initialized on first backup and encrypted with a key. If another restic job tries to access the same repository, the software will not initialize a new repository (because we already have it), but will fail accessing it since it will use a different encryption key.

(Michael Kicks) #13

Another little question: there could be

  • same backend
  • different time schedules with different destinations and different email notifications

I am asking thinking about a scenario with a second backup hive (usb + NAS, for instance) which should be cared by a different person/team…

(Matthieu Gaillet) #14

What about multiple targets ?

(Michael Kicks) #15

destinations = targets?

(Michael Träumner) #16

So I would prefer

enhanced Backup

(Giacomo Sanchietti) #17

Sorry, I didn’t get. What is the question?

It’s not only about targets, but even schedule … :thinking:

Not bad! @filippo_carletti @davidep

(Michael Kicks) #18

My bad…

Is possible to select a backend but create different time schedules with different destinations and different email address notifications?

Also: notifications could be sent in mixed directions (into mailserver and out to different email addresses?)
This should be easier to get… :frowning:
Sorry

(Giacomo Sanchietti) #19

Yes.
The only limit is not to use the same destination storage for concurrent jobs and different engines.

Yes, notifications are customizable for each backup (see this example: http://docs.nethserver.org/projects/nethserver-devel/en/latest/nethserver-backup-data.html#adding-a-backup)

(Michael Kicks) #20

Thanks :slight_smile: