How to plan a recovery?

thorsten · August 30, 2020, 3:40pm

NethServer Version: 7
Module: different

Hi,

I played (to much) with my productive system. I installed additional software e.g. nethserver modules and / community modules as well as third party software. My most important concerns are:

not clean removal of asterisk
not clean removal of docker
not clean removal of mariadb103
plex media server (besides the fact that I think it was removed clean)

Now I decided to set up a clean NS installation and to set up (some) of my individual configurations manually. What would be a good backup / restore strategy to keep

installed modules
domain / AD / user / group settings / computers
emails and aliases
file shares
DHCP and DNS Settings
additional event to transfer the LE certificate to internal servers
sogo settings, calenders and contacts
some (not really important) websites (www, matomo)
some important reverse proxy settings
a “house technique DB” within MariaDB with scripts writing climate data to it
NFS shares accessible within the intranet
Nextcloud (and Mattermost)

TIA
Thorsten

Andy_Wismer · August 30, 2020, 5:25pm

@thorsten

Hallo Thorsten

For my Home NethServer and for my clients, I use the following:
All Backups are basically 7 generations, one for each day.

As my NethServers are running in Proxmox, first Backup is a Proxmox backup, the whole machine, daily. This is also the fastest for disaster recovery.

The next one is NethServer doing it’s own Backup (mostly duplicity) to a local NAS.
This is incremental, with full backups on Sundays.

Some of my clients have twice daily backups, again both levels (Proxmox & NethServer).
These backups do not run at the same time, but slightly offset…

Both Backups are also Offsite, but this is handled by the NAS.

Most of my NethServers which are running also as Fileserver have a third Backup, a Rsync script which syncs the whole file folders into the NAS share “Backup”, there in Folders:
1_Monday
2_Tuesday
3_Wednesday
and so on.

These are read only to all users and are intended for quick restore of files without admin support.
The Backup Folder is only writable by root, not by any user. This Folder is also synched Offsite.

The whole NAS backs itself up (Synology: Hyperbackup) to a local USB3 Seagate Backup Hub Plus (8 or 10 TB).

This has covered me for disasters (cryptolockers don’t even take me an hour!), simple human error, file loss and other “Gotchas”!
The big speciality is several generations, second level not accessible by ANY PC, not even for a domain admin! This protects stuff from Cryptolocker/Ransomware.

I do have NethServers with a simpler regime, but these are mostly friends home servers, or test environments…

But for my clients, I do prefer being on the safe side!

My 2 cents
Andy

stephdl · August 30, 2020, 7:11pm

yum remove nethserver-rh-mariadb103 rh-mariadb103\* nethserver-docker container-selinux containerd.io docker-ce\*

thorsten · August 31, 2020, 6:40am

Dear

thank you for your comprehensive answer on how to schedule backups. In my case, I have all those backups - from Proxmox (2x Weekly Snapshot to different storages inside / outside) as well as from within Nethserver (Weekly full + Daily incremental) and some important databases (ecoDMS DB -> Nethserver -> Drobbox external). However, If I restore my Nethserver VM from Proxmox, I would get exactly the same VM as I had before.

My question is more how to “selectively restore modules and related Data from a full backup” or “how to plan a backup to do so.” Concrete, I do not want my user to notice that I restore the main AD, mail, files etc. I want to avoid removal / re-join of computers to the AD - I simply want to restore the AD module setup to another VM.

TIA
Thorsten

Andy_Wismer · August 31, 2020, 8:10am

@thorsten

I think a lot of people confuse Snapshots and Backups…
Reading your Post:

You start the sentence with "Backups, but then talk about “Snapshots”!!!

A snapshot is NOT to be confused with a backup!
A Backup is independent of anything else, it only needs a Proxmox (Or KVM) to run.
A Snapshot is a Snapshot, and needs the VM as such to run.

That’s what I understand under Disaster Recovery, you get the VM running, AD running, and all data / pw valid at the time of the Backup. I did say i do this daily on all NethServers, and some more often than that.
No User notices anything about AD. Theoretically, a PC could have changed it’s SID-Password with the AD, and would need to reconnect to the AD, but this is VERY VERY rare! We are talking about less than 24h for the PC to have changed it’s SID-PW for AD!

Databackups can/will get you the latest data versions available. For AD on NethServer, you can use Configuration Backup/Restore.

As mentionned above, important files/folders, but also databases (PG and MariaDB) are done seperately, with a cron job, on all my NethServers where this is important. The cron job started out before SME-Server, was adapted for NethServer, and is still running very well after 20y!

Strictly said, this has nothing to do with Backups!

This is AD Migration to another host, something also not trivial in MS. You need to join another host as AD, then remove the other AD Server, meaning you need both running, even if only for a short while or a day or two.

This is also not a recovery, as you want the AD on a different host. (You do not want the AD recovered as it was running…).

There are ways and procedures to do this, but the subject is not backup / restore or recovery.
As the word itself entails, recovery means recovering a system as it was running! Could be on different hardware (replacement) but the Software / System is configured as before!

Note:
My Proxmox are configured that I could even restore an Intel VM to an AMD based Proxmox, and it would still run without issues!

My 2 cents
Andy

thorsten · September 1, 2020, 9:12pm

No match to remove, but still services available within server manager. For asterisk it is even worse. Today I got the following notice:

The following updates will be applied on ebb-s01

Package Arch Version Repository Size

Updating:
asterisk13-addons-core x86_64 13.34.0-1.ns7 nethserver-updates 29 k
asterisk13-addons-mysql x86_64 13.34.0-1.ns7 nethserver-updates 41 k
asterisk13-core x86_64 13.34.0-1.ns7 nethserver-updates 4.5 M
asterisk13-odbc x86_64 13.34.0-1.ns7 nethserver-updates 66 k
asterisk13-resample x86_64 13.34.0-1.ns7 nethserver-updates 18 k
asterisk13-speex x86_64 13.34.0-1.ns7 nethserver-updates 15 k
asterisk13-voicemail-odbcstorage x86_64 13.34.0-1.ns7 nethserver-updates 92 k

Transaction Summary

Upgrade 7 Packages

/etc/cron.daily/0yum-daily.cron:

asterisk.service is not a native service, redirecting to /sbin/chkconfig.
Executing /sbin/chkconfig asterisk off
/etc/cron.daily/logrotate:

error: error opening /var/log/ufdbguard/ufdbguardd.log-20200412.gz: No such file or directory

thorsten · September 1, 2020, 9:14pm

In generall, my question is: can I selectively control

the rsync-migrate script as desribed herein:

https://docs.nethserver.org/en/v7/migration.html#migration-with-rsync

I really want to make sure that I get rid e.g. from my “asterisk ghosts”.

TIA

mrmarkuz · September 2, 2020, 9:37am

The rsync-migrate script is to migrate from SME.

I’d try to remove the asterisk ghosts first:

backup your server and yum remove "*asterisk*"
search for asterisk in the cronjobs and remove if necessary

If it does not work it should be enough to migrate by recovering from a backup.

You could just setup a new server and selectively rsync to it, see Lost users on NethServer v6.10 -> v7.8 upgrade - #6 by Andy_Wismer

nio707 · February 9, 2021, 5:41am

Hi Andy,

Thanks for all you help, and I have setup the NS over Proxmox in the production environment in one of our unit, Thanks for your all tips by following you. This setup was done very quickly due to some urgency. Now its time to clear some of my doubts, about your setup.

I got a Single Server setup with NS7 over Proxmox. Allocated 100Gb for Proxmox, and 1Tb for NS in the start from 2Tb of the Space available of our Dell Server ( Has 16Gb Ram, 8 cores). Arranged a Synollogy NAS with 4Tb space, and 4 Tb USB portable HDD. Have created two NFS share in NAS N-backup and P-backup and a shared folder. From Proxmox taken full backup on NAS in P-backup, and from NS Full backup and latter incremental (dublicity) in N-backup on NAS.

Some of my doubts are

When you refer to Snapshot of Proxmox, where does the snapshot is stored (Dell Server or NAS), since it looks like the fastest to do and rollback. And how and where to store the configuration of the Proxmox, to restore if the whole Proxmox needs to be reinstalled. Do I need the NS7 on the Proxmox host to shutdown for safe backup.
For the Second level i.e NS backup (dublicity), the 250mb files keeps on creating. How to manage the space because eventually it will fill the whole NAS space.
For the Fileserver backup to NAS Shared folder(Only Admin) have the write permission which Rsync script are you referring to and pls elaborate for the week day wise script setup (Does it means it will required 7X space)
Is it possible to add USB portable HDD to the server directly, and have another Proxmox backup and Snapshots on it. For the backup I can see the NFS share, but where does the Snapshots are going. Why do we need the USB portable HDD to backup the NAS itself, when it have twin HDD(Raid).
I can see the usb hdd as /dev/sda but could not add as LVM or LVM thin storage.
2221005×347 67.9 KB

When I try to add as LVM or Directory it shows no disk free.