Restore Config and Data to new NS on PVE

NethServer Version: 7.9

I want to move a bare metal NS installation to Proxmox VE (also a local device). Target NS on PVE is a fresh install with modules installed as on source NS. Target NS and source NS were updated on the same day, so all software versions should be identical.

As far as I understand it, I first recover configuration backup and then data backup (from usb device).

However, last weekend, I couldn`t restore the configuration, it aborted after step 2 (of 2) without details on why it failed.

I had ticked “install packages” and “restore network” (source NS was plugged out then).

Anything I need to know before giving it another try?

The “desaster recovery” section in the manual is well done, except: real desaster begins, when desaster recovery fails…

It’s not necessary to install modules, it’s done during the restore process.

Please check /var/log/messages for errors.
Maybe it’s just a minor issue, so I’d try to finish the restore process and check if it works afterwards.

You may check if the interfaces are mapped correctly.

Bare metal (source installation) was shut down before restoring configuration on the PVE (destination) installation?

@Markus: unfortunately those logs are no longer in the file, but I will look there next time. There’s only one interface/NIC, and I tried both, mapping it immiadiately or later.

@Michael: I pulled the network plug of source NS, so it was completely disconnected, I am aware that restoring a configuration in the same LAN would result in IP conflicts, at least for the DC.

How vital is it for the restoring to have an internet connection? When I tried, I was online, but I could also try restoring offline, and connecting the target NS to LAN and online after completing successfully.

Is it possible to restore from command line? (If it’s offline, I can`t access Cockpit, but shell via PVE…)

@sternkrabbe AFAIK…

It’s a bit more tricky than that. Restoring NethServer and NSDC are assuming that there will be NO other machine on the LAN on their space, therefore the ip conflict with the “running” device will be immediate, during network restore phase, which is one of the firsts.
Moreover, your PVE must allow promiscuos mode for correcly running NSDC and make the clients reach the container.

AFAIK is necessary. during configuration restore, the restoring NS do not have any updated module and package (except for the kernel, which is updated into final part of the install), so it will download and install latest versions of backupped server installs during the second phase of the configuration restore (after NICs role/card assign).

AFAIK yes, should be written into documentation. Backup and restore modules are “WEBGUIs” for shell commands.

top tip IMVHO: install manually and update the nethserver-backup module.

Today restoring config worked so far, although no users/groups were created so far. I don`t know if this is part of data-restore.

Restoring data fails:

Apr  3 18:12:51 neth cockpit-bridge: Running as unit cockpit-restore-data-macminibackup.service.
Apr  3 18:12:51 neth systemd: Started /sbin/e-smith/restore-data -b macminibackup.
Apr  3 18:12:51 neth restore-data: Restore started at 2022-04-03 18:12:51
Apr  3 18:12:51 neth cockpit-bridge: Use of uninitialized value in string eq at /usr/libexec/nethserver/api/system-backup/read line 114.
Apr  3 18:12:51 neth esmith::event[12848]: Event: pre-restore-data macminibackup
Apr  3 18:12:51 neth esmith::event[12848]: Event: pre-restore-data SUCCESS
Apr  3 18:12:51 neth restore-data: Event pre-restore-data: SUCCESS
Apr  3 18:12:51 neth restore-data: Local and Remote metadata are synchronized, no sync needed.
Apr  3 18:12:51 neth restore-data: Last full backup date: Sun Apr  3 17:17:21 2022
Apr  3 18:12:51 neth restore-data: Action '/etc/e-smith/events/actions/restore-data-duplicity -b macminibackup': SUCCESS
Apr  3 18:12:51 neth esmith::event[12866]: Event: post-restore-data macminibackup
Apr  3 18:12:51 neth esmith::event[12866]: [ERROR] Error the nextcloud SCL mariadb105 dump does not exist
Apr  3 18:12:51 neth esmith::event[12866]: Error while restoring nextcloud database to SCL mariadb105: "exit 1" command filed with exit code 1.
Apr  3 18:12:51 neth esmith::event[12866]: Action: /etc/e-smith/events/post-restore-data/S03nethserver-nextcloud-restore-rh-mariadb105-database FAILED: 1 [0.00285]
Apr  3 18:12:51 neth esmith::event[12866]: expanding /etc/nethserver/cockpit.otp.force
Apr  3 18:12:51 neth esmith::event[12866]: expanding /etc/nethserver/otp.secret
Apr  3 18:12:51 neth esmith::event[12866]: expanding /etc/nethserver/sshd.otp.force
Apr  3 18:12:51 neth esmith::event[12866]: expanding /etc/pam.d/cockpit
Apr  3 18:12:51 neth esmith::event[12866]: Action: /etc/e-smith/events/actions/generic_template_expand SUCCESS [0.084689]
Apr  3 18:12:51 neth esmith::event[12866]: [WARNING] The backup file is not available
Apr  3 18:12:51 neth esmith::event[12866]: Action: /etc/e-smith/events/post-restore-data/S50nethserver-ejabberd-restore SUCCESS [0.004914]
Apr  3 18:12:51 neth esmith::event[12866]: Action: /etc/e-smith/events/post-restore-data/S55nethserver-ibays-remap-locals SUCCESS [0.041034]
Apr  3 18:12:51 neth esmith::event[12866]: Action: /etc/e-smith/events/post-restore-data/S60nethserver-collectd-ns6upgrade SUCCESS [0.003483]
Apr  3 18:12:51 neth esmith::event[12866]: Action: /etc/e-smith/events/post-restore-data/S60nethserver-ibays-restore SUCCESS [0.003444]
Apr  3 18:12:52 neth esmith::event[12866]: Action: /etc/e-smith/events/post-restore-data/S60nethserver-samba-ns6upgrade-ibays SUCCESS [0.041815]
Apr  3 18:12:52 neth esmith::event[12866]: Action: /etc/e-smith/events/post-restore-data/S60nethserver-sssd-ns6homes SUCCESS [0.002125]
Apr  3 18:12:52 neth systemd: Reloading.
Apr  3 18:12:52 neth systemd: [/usr/lib/systemd/system/netdata.service:56] Failed to parse capability in bounding/ambient set, ignoring: CAP_PERFMON
Apr  3 18:12:52 neth systemd: [/usr/lib/systemd/system/netdata.service:73] Unknown lvalue 'ProtectControlGroups' in section 'Service'
Apr  3 18:12:52 neth esmith::event[12866]: [INFO] service ejabberd restart
Apr  3 18:12:52 neth systemd: Stopping ejabberd XMPP Server...
Apr  3 18:12:54 neth epmd: epmd: got KILL_REQ - terminates normal
Apr  3 18:12:54 neth systemd: Stopped ejabberd XMPP Server.
Apr  3 18:12:54 neth systemd: Starting ejabberd XMPP Server...
Apr  3 18:12:56 neth systemd: Started ejabberd XMPP Server.
Apr  3 18:12:56 neth esmith::event[12866]: Action: /etc/e-smith/events/actions/adjust-services SUCCESS [4.826644]
Apr  3 18:12:56 neth esmith::event[12866]: Action: /etc/e-smith/events/post-restore-data/S95nethserver-inventory-send SUCCESS [0.11402]
Apr  3 18:12:56 neth esmith::event[12866]: [NOTICE] forcibly unmask shorewall service
Apr  3 18:12:57 neth systemd: Reloading.
Apr  3 18:12:57 neth systemd: [/usr/lib/systemd/system/netdata.service:56] Failed to parse capability in bounding/ambient set, ignoring: CAP_PERFMON
Apr  3 18:12:57 neth systemd: [/usr/lib/systemd/system/netdata.service:73] Unknown lvalue 'ProtectControlGroups' in section 'Service'
Apr  3 18:12:57 neth esmith::event[12866]: Action: /etc/e-smith/events/post-restore-data/S98nethserver-firewall-base-enable SUCCESS [0.142579]
Apr  3 18:12:57 neth esmith::event[12866]: Action: /etc/e-smith/events/post-restore-data/S99nethserver-backup-config-clearupgradeflag SUCCESS [0.00277]
Apr  3 18:12:57 neth esmith::event[12866]: Event: post-restore-data FAILED
Apr  3 18:12:57 neth systemd: cockpit-restore-data-macminibackup.service: main process exited, code=exited, status=1/FAILURE
Apr  3 18:12:57 neth restore-data: Restore: FAIL
Apr  3 18:12:57 neth systemd: Unit cockpit-restore-data-macminibackup.service entered failed state.
Apr  3 18:12:57 neth systemd: cockpit-restore-data-macminibackup.service failed.

Looks like an issue with nextcloud. As far as I can tell no data is recovered at all. Cockpit tells me that it was a success.

It seems there’s an issue about (finding) the backup files:

Please check the system backup settings and if the usb device is accessible.

Yes, see manual.

1 Like

command line restore: this question was aimed at configuration restore.

usb device was accessible, it was listed by lsblk, and on an attempt before, I encountered different errors when it was not attached (usb port number changed on PVE and was no longer connected to guest).

So maybe the database is not part of the backup. I assumed it`s path would be added to backup on installing nextcloud.

excluded paths:

/var/lib/nethserver/openvpn-tunnels
/var/lib/nethserver/certs
/var/lib/rspamd/dkim_whitelist.inc.local
/var/lib/rspamd/rspamd_dynamic
/var/lib/nethserver/backup/restic/
/var/lib/nethserver/db
/var/lib/rspamd/mime_types.inc.local
/var/lib/rspamd/surbl-whitelist.inc.local
/var/lib/nethserver/ibay/Unterhaltung
/root/.ssh
/var/log/lastlog
/var/lib/rspamd/spf_whitelist.inc.local
/var/lib/rspamd/2tld.inc.local
/var/lib/nethserver/secrets
/var/lib/rspamd/dmarc_whitelist.inc.local
/var/lib/nethserver/backup/duplicity/
/var/lib/rspamd/spf_dkim_whitelist.inc.local

included paths:

/var/lib/redis/rspamd
/etc/dokuwiki
/GSRpool-/
/var/lib/dokuwiki
/var/lib/rspamd
/var/lib/nethserver
/var/lib/collectd
/root
/GSRpool/Backup
/GSRpool/Programme-
/GSRpool/GartenSR-
/GSRpool/dokuwiki-
/usr/share/nextcloud/config/config.php
/usr/share/dokuwiki
/var/lib/nethserver/ibay

Here you are.

It should be part of the backup by default.
Was the old system up to date? Nextcloud now uses a newer database engine and if you had an older system, this could be an issue.

Both machines were manually updated on the very same day a few weeks ago when I started migration. Nextcloud on Source NS is version 23.0.2, so pretty up to date.

But I can see that the size of my backup is way too small. It does definitely not contain all data of the included paths. Reports report success all the time.

The standard backup unfortunately relies on a ovpn tunnel. This is tricky as I cannot access it without really big efforts (like carrying machines from one place to another).

There is a big difference in size between both backups:

[root@neth backup]# cat disk_usage-macminibackup
{"used":"20096","pcent":0.01,"avail":"245533056","size":"245553152"}
[root@neth backup]# cat disk_usage-backup-data
{"used":"360000","pcent":2.43,"avail":"14449088","size":"14809088"}

Apparently I need to fix this backup-configuration:

# /sbin/e-smith/backup-data-list -b macminibackup
Achtung, es wurden Signaturen ohne dazugehörige Sicherungsdaten gefunden
Entfernte Metadaten werden zum lokalen Puffer synchronisiert …
Lokal /var/lib/nethserver/backup/duplicity/macminibackup/41142746bec0c818a30a48d097eff27d/duplicity-full-signatures.20220403T151721Z.sigtar.gz wird gelöscht (keine Auswirkungen am Hintergrundprogramm).
Lokal /var/lib/nethserver/backup/duplicity/macminibackup/41142746bec0c818a30a48d097eff27d/duplicity-full.20220403T151721Z.manifest wird gelöscht (keine Auswirkungen am Hintergrundprogramm).
Letzte vollständige Sicherung: Keine
Traceback (innermost last):
  File "/usr/bin/duplicity", line 1581, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1567, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1419, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1502, in do_backup
    list_current(col_stats)
  File "/usr/bin/duplicity", line 708, in list_current
    path_iter = diffdir.get_combined_path_iter(sig_chain.get_fileobjs(time))
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 539, in get_fileobjs
    return [filename_to_fileobj(f) for f in self.get_filenames(time)]
  File "/usr/lib64/python2.7/site-packages/duplicity/collections.py", line 536, in filename_to_fileobj
    return sig_dp.filtered_open("rb")
  File "/usr/lib64/python2.7/site-packages/duplicity/path.py", line 774, in filtered_open
    return gzip.GzipFile(self.name, mode)
  File "/usr/lib64/python2.7/gzip.py", line 94, in __init__
    fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb')
 IOError: [Errno 2] No such file or directory: '/var/lib/nethserver/backup/duplicity/macminibackup/41142746bec0c818a30a48d097eff27d/duplicity-full-signatures.20220403T151721Z.sigtar.gz'

(Last error is funny, as it deleted the file itself)

However, it says that there is no full backup in place. So I will try to fix this and report.

1 Like

Backup size now is 181.5 GB. Guess, that´s what I need… I will give it another try next weekend.

1 Like

Looks like it worked… finally. But it really was a pain to get here.

Without internet connection, config backup does not work, whatever settings you use.
Cockpit gives you absolute no real information, what is going on, and /var/log/messages gives you too much information.
DC restore was pure luck, needed manually reinstalling the nsdc module.

If your server fails, there is a fair chance that you can recover user data from the ns backups. But don`t expect to recover the server itself in a day or two. I am glad to have it on proxmox now.

1 Like

@sternkrabbe thanks for your experience. Mine was a bit different.

Context:

  • no NSDC container
  • no connected services externally
  • backup on SMB share

I moved around at least 15 servers, among hardware and from physical to virtual.

These were my steps.

  • refresh the backup from the source server while
  • installing “new server” from ISO, then reboot
  • download a config from the source server (not using firefox, and anyway i used it); optional, file can be found into data backup SMB folder
  • install/update the backup module on the new server
  • shutdown the old server
  • load the configuration file into new server
  • manage the network interfaces/roles attribution
  • waited the configuration restore (with package install/update)
  • run data restore

And every time worked as intended. At least, lately.

I mean, if you want to be a pro in restore, do it many times.

1 Like

8 posts were split to a new topic: SFTP backup is deleted on disabling when backup target is unreachable