Multi Database Backup for NS8

Development
, ,
1

There is nothing like too many backups.
And nothing is better than a testable verifiable backup.

With the Implementation OF nethserver 8, We make use of a container based model,
In this new model, for backups, we backup the enire container, which we can restore at a point in time the backup was made.

I saw somwehre in the community someone mentioned something to do with sequential backups, And then i remembered.

PostreSQL comes with a builtin Incremental Backup and Point-inTime Recovery

Now in NEthserver 7, most Modules would utilize the same database server to create and store databases, and it would be easier to just backup the entire database server, and incase of failure for whatever reason, you can not lose your data and files.

In Nethserver 8 on the other hand, an inherent challenge is presented, whereby each module/Application comes with its own database server in the form of a docker container.

MEaning, if a NS8 instance has 10 Applications hosted(which many are there) We also have a situation whereby we have 10 database servers, each with own database and might require to be backed up SMH.

the provided backup and restore utility in NS8 is great, dont get me wrong, but sometimes you just need the backup of the database, and could care less about everything else.

Well, with docker it turns out some ENV could also be critical and worth backing up, Especially the ones that encrypt database data. WHOSH(security but mehn)

So this got me thinking, Alot… and i came accross this utility>
tiredofit/docker-db-backup: Backup multiple database types on a scheduled basis with many customizable options (github.com)

From the outlook, it allows you to backup mutliple databases accorss mutliple docker containers, accross mutliple database systems PGSql, Mysql, MOngoDB and Redis(atleast the ones most NS8 apps use)

I have not gone into great details on the tool, but looking into it, in corelation with the Questions above, that many users might have. I think it might worth looking into by the core dev team, dont you think.

if there is another solution to this conondrum, I am all ears, after all this is a comunity, and everyone is invited to offer their suggestions.

tag @davidep (did not want to tag all dev teams)

2

@oneitonitram

I think this is ONLY an issue for Developers.

All my clients use the databases as given - and Backups as made.
If two Apps happen to use PostgreSQL, fine, we have Backups of each App if needed.

This is what the client expects, and also what he gets.

Most users don’t care about the Database - the App as such must work, including database and data therein.

→ If a user / company / institution has to worry about such details, it implies that the system is unstable, not more, not less.

I don’t see the system as such as unstable. The NS7 migration still has issues, but the NS8 system as such seems rock solid.

So why waste resources?

My 2 cents
Andy

1 Like
3

Waste? Waste what resources. TBH i am happy with NS8 backup and restore, it just gets things done. I can Backup a say, dolibarr, and restore a Dolibarr either on the same node or another Node.

In Large companies, Data is Key.
Not only is there Image snapshorts of the Entire VM or Servers, ther is also seprate RSYNC of Files, and Equally, seprate Backups of the Database. the Database in this Case is Utilized in more than one Scenario.

Now NS8 is great, however for complex deployments, One might need to migrate an existing running system, for this, just a backup of the database might save alot.

Developers and System Admins alike. and you are right i am having a more broader outlook at this case

4

I have all of this anyways.

So why would I need this:

in addition?

NS8 is not intended for an Enterprise environment. ATM, there are still way too many Issues here, and
Enterprise wants something else anyways…
With Humans, Enterprise HR prefers people around 25 years old, and bringing along 20 years of experience in the business…

You did ask for feedback.

My 2 cents
Andy

1 Like
5

and i love it…

1 Like
6

If I think about my preferred applications (e.g. Nextcloud and Webtop), their DBs are not accessible by external backup tools because their TCP port is not published.

By system design, applications are isolated: this choice makes difficult to centralize tasks, like managing the backup of parts of them.

7

Also another question some of the applications you just pass the database credentials without Database name. So in this case the back up is internally made in the application such as running

php artisan db:backup

the one above is just an example
then the db is exported to a specific folder in the application

1) Altering Back up Process through Actions

Is there a way to alter the normal back up process and add additional bash files to be able to succeed with the back up through the internal db backup

2) Button on the UI

The second way, Adding a button for back up database and map it into actions
So that when the button is clicked it run the action to back up the database

So if this is the case how can we connect to the scheduler and make it back up weekly or daily just as the system back up

Also if there are any other ways to handle back up you can give ideas
@davidep

8

Also if there is a restoration action that would be great

9

Well, the backup procedure integrated with NS8 core can be extended as wanted by the app developer.

The Dev’s manual does not explain it well, it just describes how it works from the Core point of view: Backup & Restore | NS8 dev manual

Each module can implement module-dump-state and module-cleanup-state to prepare/cleanup the data that has to be included in the backup.

That means you can drop the two commands inside the app ~/.config/bin/ directory and they are executed during the backup procedure.

Those command dump and cleanup additional files that can be included in the backup set (see Backup & Restore | NS8 dev manual).

2 Likes
10

Thank you let me look at it
Is there a way i can contribute to the docs
the developer docs i would be happy to do that

11

Contributions are welcome: just open a pull request on GitHub!

12

@ oneitonitram

In order to obtain a separate backup of the respective DB (independent of NS8 strategies), the respective DB does not necessarily have to be accessed from outside. It would be sufficient if the respective container itself created DB dumps - which can then be “pulled” from outside.

Under Nethserver7, for example, a dump of the SOGo DB was always created, which was then included in the entire backup. I always saved this separately using rsync so that individual structures could be restored from it if necessary (e.g. individual users, calendars or address books).

The NS8 backup procedures may be as stable as possible, but there are many more situations in which only certain data should be restored than situations in which the entire container must (or should) be restored.

For this reason, I have always created a complete rsync backup of all services and data, from which I can grab individual files if necessary. Of course, this only works properly for DBs if there are already dumps from the DBs to sync away…

1 Like
13

Our current research is based on alternative backup options for NS8, We Have been hard at work in having these done, its hectic, but i think we have some light at the end of the tunnel.