Disaster recovery - mailfolders

Support
, , ,
1

NethServer Version: 7.7.1908
Module: backup and mail

I just tested a disaster recovery on another VM with a clean install, same version. This seemed to work and all mailfolders and users are recreated. But, when I try to login with a mailuser to IMAP the logs show:

Feb 20 22:36:31 cloud dovecot: imap(user@mydomain.com): Namespace '': stat(/var/lib/nethserver/vmail/user@mydomain.com/Maildir) failed: Permission denied (euid=984(vmail) egid=981(vmail) missing +x perm: /var/lib/nethserver/vmail, dir owned by 985:982 mode=0700) in=0 out=382

Now, it seems the user is recreated with a different UID? Or the user ā€˜vmailā€™ is set with a different UID? Is this something I can prevent somehow?

2

How did you recovered your server as ā€œdisaster recoveryā€?
Please, a list of stepsā€¦

3

Right, of course.

  1. I did a clean install of 7.7.1908 on another VM.
    1a) I updated both servers before creating the backup and restoring it on the new server.
    1b) I did NOT change the hostname of the new and fresh Nethserver install.
  2. I downloaded the config from the 'production" server and restored it on the fresh Nethserver.
  3. I connected the fresh server to the same backup location where a recent restic backup from the production server is located.
  4. I restored that backup and reconfigured the interfaces afterwards.
  • I restored the above all from the GUI.

Now, all users and packages where restored as expected. Only when I try to connect to the IMAP server , I get the above logs about permission problems.

4

Between 1 and 2 did you fully updated your server before restore configuration?
Also: hostname for ā€œanother VMā€ is the same of the old/dead one? (according to disaster recovery scenario)

5

Between 1 and 2 did you fully updated your server before restore configuration?
Also: hostname for ā€œanother VMā€ is the same of the old/dead one? (according to disaster recovery scenario)

I did a full update on both machines before creating the update and after that, restoring it.
I did not change the hostname beforehand. The hostname of the new VM did change after restoring the config on it.

Should I retry with a new install and change the hostname beforehand you think?

6

Hi

I donā€™t think changing the hostname beforehand would help, as the name is changed during the restore to match what it wasā€¦

I see this as a (small) bug: A full restore to a newly installed NethServer should just work.

The UIDs should be the same as on the old server. Otherwise a lot of problems can and will happen, like Windows Access to AD/Filesā€¦

The predesessor of NethServer, SME-Server had a very well working backup. The main issue there was, the backup ONLY restored standard stuff. Any Add-Ons or Plug-Ins had to be manually installed BEFORE the restore, then it would work!
However, if you had stuff on the server no longer available (expired/replaced software, repos or other stuff), you had a big pronblem. NethServer solves this nicely by installing all software anew from the Software Repo. Other, manually installed stuff would still have to be handled separately.

My 2 cents
Andy

7

AFAIK the restore of the whole server (not config) itā€™s read only if the hostname of the backup matches the hostname of the restore target.
As docs statedā€¦
https://docs.nethserver.org/en/v7/disaster_recovery.html
1 install ā€œnewā€ machine
2 restore config uploading the file of configuration
3 map interfaces logical with physical
4 start restore config
4 opt - login back if IP changed
5 go to data backup
6 start restore data.

Is what you run into, @jelle?

1 Like
8

@pike

Youā€™re right there, a full restore NEEDS the same hostname as the ā€œoldā€ server. (Not the same IPā€¦)

Iā€™d also go for a full restore in this case.

That is also valid for a lot of other systems, even on my Macbook. (I can migrate the TimeMachine backup to a new running maschine, we in IT call that a restore, but technically itā€™s NOT a restore, itā€™s a migration, as the hardware is new! So the tool isnā€™t called TimeMachine anymore, itā€™s called Migration utility or something like that)ā€¦

Legalities and fine printā€¦ :wink:

My 2 cents
Andy

9

Thanks all for your replies.
Iā€™ve tested again from scratch.

  1. Unattended install of Nethserver 7.7.1908 on a VM
  2. Upgraded all software
  3. Changed the hostname to the same as original server
  4. Uploaded the config and restored it
  5. Restored the data-backup from original server

Now, Iā€™ve noticed that after the config-restore, I could access the IMAP folder of user1 which was correctly imported from the config. Of course the IMAP folder was still empty because I still had to do the data-restore.
After the data-restore, the data was indeed restored, but I get the same permission error on the IMAP folder (see above). So, this bug is introduced with the data-restore. Iā€™m not sure how to proceed. Of course I need a working backup functionality before I take the original server in ā€˜productionā€™.
Hope you can help.

10

What backup-data engine are you using? (restic, duplicity or rsync) Where? (CIFS, SFTP, etc).
I suspect that we may have permission problems using rsync over sftp with a non-root user over some destinations.

11

Iā€™m using restic over webdav.

Sidenote: Iā€™m getting the logs mailed after a successful backup of the original server and I noticed:

ā€¦
Pre backup scripts status: SUCCESS
/sbin/mount.davfs: Warning: canā€™t write entry into mtab, but will mount the file system anyway at /etc/e-smith/events/actions/mount-webdav line 76.
ā€¦

(But, on second thought, this might be due to concurrent backups running at same time and same webdav share since I restored the second server. Will shutdown backups on the new server and see what happens with the original backup routine.)

12

Could you show the permissions?

find /var/lib/nethserver/vmail/user@mydomain.com/Maildir -type d -ls

13

Yes.

The permissions on the original server:

268885867 4 drwx------ 11 vmail vmail 4096 Feb 20 22:45 /var/lib/nethserver/vmail/user@mydomain.com/Maildir
1168969 0 drwx------ 5 vmail vmail 156 Feb 13 00:11 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Trash
268885870 4 drwx------ 2 vmail vmail 4096 Feb 12 20:57 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Trash/cur
538146571 0 drwx------ 2 vmail vmail 6 Feb 12 20:23 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Trash/new
805934518 0 drwx------ 2 vmail vmail 6 Feb 12 20:57 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Trash/tmp
1168971 4 drwx------ 2 vmail vmail 4096 Feb 12 20:36 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/cur
268885869 0 drwx------ 2 vmail vmail 6 Feb 12 20:36 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/new
538146574 0 drwx------ 2 vmail vmail 6 Feb 12 20:36 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/tmp
805963715 0 drwx------ 5 vmail vmail 108 Feb 1 20:38 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Junk
1168973 0 drwx------ 2 vmail vmail 6 Feb 1 20:38 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Junk/cur
268885875 0 drwx------ 2 vmail vmail 6 Feb 1 20:38 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Junk/new
538146575 0 drwx------ 2 vmail vmail 6 Feb 1 20:38
ā€¦

On the new restored server:

1243732 4 drwx------ 11 _rspamd dovenull 4096 Feb 20 22:45 /var/lib/nethserver/vmail/user@mydomain.com/Maildir
102526790 0 drwx------ 5 _rspamd dovenull 156 Feb 13 00:11 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Trash
1476870 4 drwx------ 2 _rspamd dovenull 4096 Feb 12 20:57 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Trash/cur
34502537 0 drwx------ 2 _rspamd dovenull 6 Feb 12 20:23 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Trash/new
67677148 0 drwx------ 2 _rspamd dovenull 6 Feb 12 20:57 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Trash/tmp
103047519 4 drwx------ 2 _rspamd dovenull 4096 Feb 12 20:36 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/cur
1476869 0 drwx------ 2 _rspamd dovenull 6 Feb 12 20:36 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/new
34502538 0 drwx------ 2 _rspamd dovenull 6 Feb 12 20:36 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/tmp
67677149 0 drwx------ 5 _rspamd dovenull 108 Feb 1 20:38 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Junk
102024097 0 drwx------ 2 _rspamd dovenull 6 Feb 1 20:38 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Junk/cur
865859 0 drwx------ 2 _rspamd dovenull 6 Feb 1 20:38 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Junk/new
34502528 0 drwx------ 2 _rspamd dovenull 6 Feb 1 20:38
ā€¦

14

Could you also give user id of the vmail and _rspmad users in both systems?
Mine for reference:

[root@ns7 ~]# getent passwd vmail
vmail:x:985:983:Virtual mailboxes owner:/var/lib/nethserver/vmail:/sbin/nologin
[root@ns7 ~]# getent passwd _rspamd
_rspamd:x:989:984:Rspamd user:/var/lib/rspamd:/bin/false
15

Original server:

[root@cloud ~]# getent passwd vmail
vmail:x:985:982:Virtual mailboxes owner:/var/lib/nethserver/vmail:/sbin/nologin
[root@cloud ~]# getent passwd _rspamd
_rspamd:x:986:980:Rspamd user:/var/lib/rspamd:/bin/false

Restored server:

[root@cloud ~]# getent passwd vmail
vmail:x:984:981:Virtual mailboxes owner:/var/lib/nethserver/vmail:/sbin/nologin
[root@cloud ~]# getent passwd _rspamd
_rspamd:x:985:979:Rspamd user:/var/lib/rspamd:/bin/false

So, obviously the uids are changed after the data-restore.

16

Iā€™ve also tried with rsync over SFTP and Restic over SFTP. Both have the same result.

17

UPDATE:
After a lot of testing, restoring, etc, I now tried using the old GUI to:

  1. Create a config backup from original server
  2. Create a data-backup to a WEBDAV share
  1. Restore the config (in old GUI) in the testserver
  2. Restore the data-backup from the CLI with ā€œrestore-data -b backup-dataā€

This actually was successfull! Everything was restored with the right permissions.
So, there is something in the Cockpit GUI that runs the restore scripts differently that causes this. Iā€™m very curious if anyone can reproduce this behaviour.

Q: Can I restore a Restic backup from the CLI? Iā€™d like to test if that works better too.

1 Like
18

What the hā€¦ Glad you managed to find a way but thatā€™s a lot of pain for finding whatā€™s different into Cockpit restore than the shell.

19

Iā€™m still trying to reproduce the problem, without success.
Maybe we are looking in the wrong spot, restore-data is ok, but the system you are restoring on has something that changes the vmail user id.

All backups can be restored from the shell, using their name. Example:

restore-data -b my_restic_backup

20

Do Cockpit and NethGUI use the same user for start the shell command?