Disaster Recovery Issues

[Jimbo (Jim Blake)]

I have run Nethserver on a Dell Precision T3500 for years.Its got two 1Tb drives in soft RAID and is backed up nightly. It’s fully up-to-date with Nethserver 7.9.

All was well till yesterday when the PSU decided to take a holiday. I don’t (yet) have a replacement PSU, so I thought I’d use the backups to rebuild on different hardware until the proper server could be reinstated (PSU on order).

I installed the Netherver base on the new hardware, then loaded all the applications that the original server had. I also ran all the updates. At that point I had what I thought was a good base install, so I got the config backup and loaded that, then loaded the data backup.

However, something is wrong: If I attempt to go to email, roundcube tells me it cannot reach the SQL server (which was loaded, and I thought was configured) and the websites the server is supposed to be serving are not available (/var/lib/nethserver/vhosts is empty)

I’m doing something wrong, but having read the documenation, I can’t see what. The substitute server should be OK, its a similar speed Xeon with more memory than the defective one. The only real difference is that the replacement device has smaller discs (650Gb). For some reason the original build used the discs in RAID without any intervention on my part, but when I built the substitute, there is no RAID, its JBOD.

From the description above, firstly am I following the correct process to recover, and secondly what debugging should I be doing to find what is wrong. The problem is all of my own making: the server has been so reliable for so long that I’ve forgotten a lot about anything other than basic operations.

Thanks for any help

Jim

[Andy_Wismer (André Wismer)]

@Jimbo

Hi Jim

I’m not an expert on NethServer Backups - my clients NethServers DO backups daily, but as all are virtual, I use Proxmox for disaster recovery…

But for anyone willing to help, it would help to know which of the three options for backup you’re using… I assume you’re using duplicity, but it could be you’re using restic engine or the third one?

And while you’re at it… I’d suggest trying out a virtual solution like Proxmox for disaster recovery…
My 30 clients all are using Proxmox underneath, and I have no issues transfering a VM from one Proxmox to another (Even Intel / AMD doesn’t matter in my case!).

A lot of people, if not the most here on this forum use virtualization underneath. Nowadays, one does not suggest to build a hardware server, it’s all virtual! My favorite is Proxmox, but others use VMWare, Xen or Hyper-V. A Workstation virtualization like Virtualbox is NOT the way to go!

My 2 cents
Andy

[Jimbo (Jim Blake)]

I’m using duplicity. I might migrate to Proxmox underlay after I recover the current situation, seems like a lot easier, but Nethserver has been solid as a rock up till now and there just hasn’t been the incentive

Jim

[mark_nl (Mark Verlinde)]

Hi Jim,

Only did a disaster recovery once, as a practice run, a long time ago…
Looking through my notes there are two topics with a explanation mark.

@Jimbo

I installed the Netherver base on the new hardware, then loaded all the applications that the original server had. I also ran all the updates. At that point I had what I thought was a good base install, so I got the config backup and loaded that, then loaded the data backup.

1. On the new system only install the base and update this : do not install modules.
   Than do the configuration restore
   (notes are a bit cryptic why it is related to the passwords / secrets of the former system needed to access databases and so on)

2. After the the system ran the configuration restore (this process installs the modules) you do the second phase of the disaster recovery which is restoring the actual data.
   (again notes are cryptic on why: IIRC with the data restore the actual db-bases and so on are restored, replacing the empty place holders created during the configuration restore)

[Jimbo (Jim Blake)]

Hi Mark,

Thanks for that, I was wondering, seems a bit pointless having to do a module reinstore from memory when the config backup could do it, but I thought I’d read that was the way to do it. I’m going to have another crack at this, and then I’ll let you know how I get on.

Thanks

Jim

[Jimbo (Jim Blake)]

Now its getting personal! I got two 620GB disks on the system, thats all I had available (it was originally running with 2 x 1Tb disks, but I’m keeping them to one side, once I get the PSU on the originl server fixed, I hope they will just come back up). Anyway, I did the config reload…worked fine. Did the Data reload, and it crashed on insufficent disk space. The / mount point is set up with 50G and is full. the /home mount point is set up with 1.7TB and is empty. I let the install program partition the drives, that now looks to be a mistake! Sorry to keep coming back with additional weirdness, but has anyone got any ideas? BTW, Roundcube and the mail now works, but I have none of the old mail messages and no address book. Also no websites

Thanks

Jim

[Andy_Wismer (André Wismer)]

@Jimbo

Hi Jim

AFAIK, and I’ve done more than my share of NethServer installs, the NethServer install programm will set up a single partition for usage. If you have two same disks, RAID1 (Mirror) can be used (It’s offered). Default is a small /boot partition, a LVM partition mapped to / and a swap partition.

I’ve NEVER seen a RAID0 being created with NethServer. And /home is NOT from NethServer, as NethServer leaves /home empty (It’s not used at all within NethServer)!

I’ve seen such using Centos7 as a base…

So WHAT exactly are you installing?

A home partition of 1.7 TB and a root partition of 50 GB strongly suggests a RAID0 being created of 2 650 GB Disks (Without any redundancy!). And /home is so untypical of NethServer!

Make sure (If using 2 disks) that you setup up as RAID1 (software raid, mirroring) - and do not do any manual partitioning! Let NethServer handle that for you!

Also let NethServer handle the Modules for you, at least the NethServer included ones. The additional Modules from nethForge (User Contributed Modules) still need manual installation (before restore!)…

My 2 cents
Andy

[dnutan (Marc)]

https://docs.nethserver.org/en/v7/disaster_recovery.html#disaster-recovery
The backup should come from a NethServer having the same operating system version.
Third-party repositories are not restored (install them before proceeding with the restore).

Guide / Steps to Disaster Recovery of Nethserver with a Dokuwiki and Zabbix installation

Hi everyone, recently I needed to move my Nethserver to a new instance. To make a long story short, I assigned too much space to my original installation and wanted to shrink the drive down which is not possible because of the XFS Filesystem. The only way to do this was to backup the original server and bring everything over to a new instance. Of course I experienced issues a long the way. This is why I am sharing the steps of what I did to get that done. My hope is that, if any of you are faced…

[Jimbo (Jim Blake)]

OK, Thanks guys! I got it all back and running last night, no data lost and everything looks OK. Now I remember why I chose to run NethServer, the level of support you have all given has been superb, even to someone like me who should know his systems better. Two things caused problems: 1, the recovery sequence and 2, the rebuilt disc structure and I got swift and competent help on both.

Now its time to look at Proxmox

[Andy_Wismer (André Wismer)]

@Jimbo

Hi Jim

If you have any questions, need any tips about Proxmox, don’t hesitate to ask!
Either direct in the forum here, or with a PM!

You won’t regret virtualising your Environment!

A few ideas / samples from my clients:

My home LAN:

My 2 cents
Andy

[m.traeumner (Michael Träumner)]

@Jimbo

OK, Thanks guys! I got it all back and running last night, no data lost and everything looks OK

That sounds good. Can you mark the answer which helped you as solution please? This could help others to find a solution .

Howto mark a topic as solved

Now we have an amazing feature on our community platform which helps to mark a topic as SOLVED As the OP (topic creator) you will have a button that allows you to accept an answer [image] Accepted answers are highlighted in the topic list: [image] Accepted answers are highlighted on the initial topic: [image] This kind of action is enabled only for topic into the Support category Happy support!