Disaster recovery - mailfolders

NethServer Version: 7.7.1908
Module: backup and mail

I just tested a disaster recovery on another VM with a clean install, same version. This seemed to work and all mailfolders and users are recreated. But, when I try to login with a mailuser to IMAP the logs show:

Feb 20 22:36:31 cloud dovecot: imap(user@mydomain.com): Namespace '': stat(/var/lib/nethserver/vmail/user@mydomain.com/Maildir) failed: Permission denied (euid=984(vmail) egid=981(vmail) missing +x perm: /var/lib/nethserver/vmail, dir owned by 985:982 mode=0700) in=0 out=382

Now, it seems the user is recreated with a different UID? Or the user ‘vmail’ is set with a different UID? Is this something I can prevent somehow?

How did you recovered your server as “disaster recovery”?
Please, a list of steps…

Right, of course.

  1. I did a clean install of 7.7.1908 on another VM.
    1a) I updated both servers before creating the backup and restoring it on the new server.
    1b) I did NOT change the hostname of the new and fresh Nethserver install.
  2. I downloaded the config from the 'production" server and restored it on the fresh Nethserver.
  3. I connected the fresh server to the same backup location where a recent restic backup from the production server is located.
  4. I restored that backup and reconfigured the interfaces afterwards.
  • I restored the above all from the GUI.

Now, all users and packages where restored as expected. Only when I try to connect to the IMAP server , I get the above logs about permission problems.

Between 1 and 2 did you fully updated your server before restore configuration?
Also: hostname for “another VM” is the same of the old/dead one? (according to disaster recovery scenario)

Between 1 and 2 did you fully updated your server before restore configuration?
Also: hostname for “another VM” is the same of the old/dead one? (according to disaster recovery scenario)

I did a full update on both machines before creating the update and after that, restoring it.
I did not change the hostname beforehand. The hostname of the new VM did change after restoring the config on it.

Should I retry with a new install and change the hostname beforehand you think?

Hi

I don’t think changing the hostname beforehand would help, as the name is changed during the restore to match what it was…

I see this as a (small) bug: A full restore to a newly installed NethServer should just work.

The UIDs should be the same as on the old server. Otherwise a lot of problems can and will happen, like Windows Access to AD/Files…

The predesessor of NethServer, SME-Server had a very well working backup. The main issue there was, the backup ONLY restored standard stuff. Any Add-Ons or Plug-Ins had to be manually installed BEFORE the restore, then it would work!
However, if you had stuff on the server no longer available (expired/replaced software, repos or other stuff), you had a big pronblem. NethServer solves this nicely by installing all software anew from the Software Repo. Other, manually installed stuff would still have to be handled separately.

My 2 cents
Andy

AFAIK the restore of the whole server (not config) it’s read only if the hostname of the backup matches the hostname of the restore target.
As docs stated…
https://docs.nethserver.org/en/v7/disaster_recovery.html
1 install “new” machine
2 restore config uploading the file of configuration
3 map interfaces logical with physical
4 start restore config
4 opt - login back if IP changed
5 go to data backup
6 start restore data.

Is what you run into, @jelle?

1 Like

@pike

You’re right there, a full restore NEEDS the same hostname as the “old” server. (Not the same IP…)

I’d also go for a full restore in this case.

That is also valid for a lot of other systems, even on my Macbook. (I can migrate the TimeMachine backup to a new running maschine, we in IT call that a restore, but technically it’s NOT a restore, it’s a migration, as the hardware is new! So the tool isn’t called TimeMachine anymore, it’s called Migration utility or something like that)…

Legalities and fine print… :wink:

My 2 cents
Andy

Thanks all for your replies.
I’ve tested again from scratch.

  1. Unattended install of Nethserver 7.7.1908 on a VM
  2. Upgraded all software
  3. Changed the hostname to the same as original server
  4. Uploaded the config and restored it
  5. Restored the data-backup from original server

Now, I’ve noticed that after the config-restore, I could access the IMAP folder of user1 which was correctly imported from the config. Of course the IMAP folder was still empty because I still had to do the data-restore.
After the data-restore, the data was indeed restored, but I get the same permission error on the IMAP folder (see above). So, this bug is introduced with the data-restore. I’m not sure how to proceed. Of course I need a working backup functionality before I take the original server in ‘production’.
Hope you can help.

What backup-data engine are you using? (restic, duplicity or rsync) Where? (CIFS, SFTP, etc).
I suspect that we may have permission problems using rsync over sftp with a non-root user over some destinations.

I’m using restic over webdav.

Sidenote: I’m getting the logs mailed after a successful backup of the original server and I noticed:


Pre backup scripts status: SUCCESS
/sbin/mount.davfs: Warning: can’t write entry into mtab, but will mount the file system anyway at /etc/e-smith/events/actions/mount-webdav line 76.

(But, on second thought, this might be due to concurrent backups running at same time and same webdav share since I restored the second server. Will shutdown backups on the new server and see what happens with the original backup routine.)

Could you show the permissions?

find /var/lib/nethserver/vmail/user@mydomain.com/Maildir -type d -ls

Yes.

The permissions on the original server:

268885867 4 drwx------ 11 vmail vmail 4096 Feb 20 22:45 /var/lib/nethserver/vmail/user@mydomain.com/Maildir
1168969 0 drwx------ 5 vmail vmail 156 Feb 13 00:11 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Trash
268885870 4 drwx------ 2 vmail vmail 4096 Feb 12 20:57 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Trash/cur
538146571 0 drwx------ 2 vmail vmail 6 Feb 12 20:23 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Trash/new
805934518 0 drwx------ 2 vmail vmail 6 Feb 12 20:57 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Trash/tmp
1168971 4 drwx------ 2 vmail vmail 4096 Feb 12 20:36 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/cur
268885869 0 drwx------ 2 vmail vmail 6 Feb 12 20:36 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/new
538146574 0 drwx------ 2 vmail vmail 6 Feb 12 20:36 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/tmp
805963715 0 drwx------ 5 vmail vmail 108 Feb 1 20:38 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Junk
1168973 0 drwx------ 2 vmail vmail 6 Feb 1 20:38 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Junk/cur
268885875 0 drwx------ 2 vmail vmail 6 Feb 1 20:38 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Junk/new
538146575 0 drwx------ 2 vmail vmail 6 Feb 1 20:38

On the new restored server:

1243732 4 drwx------ 11 _rspamd dovenull 4096 Feb 20 22:45 /var/lib/nethserver/vmail/user@mydomain.com/Maildir
102526790 0 drwx------ 5 _rspamd dovenull 156 Feb 13 00:11 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Trash
1476870 4 drwx------ 2 _rspamd dovenull 4096 Feb 12 20:57 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Trash/cur
34502537 0 drwx------ 2 _rspamd dovenull 6 Feb 12 20:23 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Trash/new
67677148 0 drwx------ 2 _rspamd dovenull 6 Feb 12 20:57 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Trash/tmp
103047519 4 drwx------ 2 _rspamd dovenull 4096 Feb 12 20:36 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/cur
1476869 0 drwx------ 2 _rspamd dovenull 6 Feb 12 20:36 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/new
34502538 0 drwx------ 2 _rspamd dovenull 6 Feb 12 20:36 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/tmp
67677149 0 drwx------ 5 _rspamd dovenull 108 Feb 1 20:38 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Junk
102024097 0 drwx------ 2 _rspamd dovenull 6 Feb 1 20:38 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Junk/cur
865859 0 drwx------ 2 _rspamd dovenull 6 Feb 1 20:38 /var/lib/nethserver/vmail/user@mydomain.com/Maildir/.Junk/new
34502528 0 drwx------ 2 _rspamd dovenull 6 Feb 1 20:38

Could you also give user id of the vmail and _rspmad users in both systems?
Mine for reference:

[root@ns7 ~]# getent passwd vmail
vmail:x:985:983:Virtual mailboxes owner:/var/lib/nethserver/vmail:/sbin/nologin
[root@ns7 ~]# getent passwd _rspamd
_rspamd:x:989:984:Rspamd user:/var/lib/rspamd:/bin/false

Original server:

[root@cloud ~]# getent passwd vmail
vmail:x:985:982:Virtual mailboxes owner:/var/lib/nethserver/vmail:/sbin/nologin
[root@cloud ~]# getent passwd _rspamd
_rspamd:x:986:980:Rspamd user:/var/lib/rspamd:/bin/false

Restored server:

[root@cloud ~]# getent passwd vmail
vmail:x:984:981:Virtual mailboxes owner:/var/lib/nethserver/vmail:/sbin/nologin
[root@cloud ~]# getent passwd _rspamd
_rspamd:x:985:979:Rspamd user:/var/lib/rspamd:/bin/false

So, obviously the uids are changed after the data-restore.

I’ve also tried with rsync over SFTP and Restic over SFTP. Both have the same result.

UPDATE:
After a lot of testing, restoring, etc, I now tried using the old GUI to:

  1. Create a config backup from original server
  2. Create a data-backup to a WEBDAV share

  1. Restore the config (in old GUI) in the testserver
  2. Restore the data-backup from the CLI with “restore-data -b backup-data”

This actually was successfull! Everything was restored with the right permissions.
So, there is something in the Cockpit GUI that runs the restore scripts differently that causes this. I’m very curious if anyone can reproduce this behaviour.

Q: Can I restore a Restic backup from the CLI? I’d like to test if that works better too.

1 Like

What the h… Glad you managed to find a way but that’s a lot of pain for finding what’s different into Cockpit restore than the shell.

I’m still trying to reproduce the problem, without success.
Maybe we are looking in the wrong spot, restore-data is ok, but the system you are restoring on has something that changes the vmail user id.

All backups can be restored from the shell, using their name. Example:

restore-data -b my_restic_backup

Do Cockpit and NethGUI use the same user for start the shell command?