Sage Backup Fails

joseoliveira · January 22, 2021, 2:46pm

NethServer Version: your_version
NethServer release 7.9.2009 (final)
Module: your_module
Backup

Dear Support,

Please, could someone help?
I’m getting this error:

Backup: backup-02
Backup started at 2021-01-22 14:30:22
Pre backup scripts status: SUCCESS
mkdir /mnt/backup-backup-02: File exists at /etc/e-smith/events/actions/mount-cifs line 68.
Can’t mount /mnt/backup-backup-02
Action ‘backup-data-duplicity backup-02’: FAIL
Backup status: FAIL

CIFS backup storage is an Ubuntu 20.04. Test passes without an error.
First backup was done on the 15h of January

I thank you in advance for your help.

Andy_Wismer · January 22, 2021, 2:49pm

@joseoliveira

Hi

I have the strong feeling you do not have enough permissions in the Ubunte Backup Share…

The error indicates Nethserver could not rename / change a file / directory.
And 15th was exactly a week ago, 7 days… Incremental Backup?

My 2 cents
Andy

joseoliveira · January 22, 2021, 5:16pm

Hi Andy,

Thank you for your update.
User used on backup has read and write rights on Ubuntu server. I tested that before setting the backup policy.
Besides, the error mounting /mnt/backup-backup02 seems more a local error on the Nethserver side…
Backup is using Duplicity
Type: Full
Cleanup backups older than 1 day

Kind regards,
Jose

Andy_Wismer · January 22, 2021, 5:33pm

Try erasing whatever is in the backup folder and doing starting a backup…
If it can do the backup, there’s your permissions issue: It may also need execute rights (or something else).

Try and give some feedback…

My 2 cents
Andy

dnutan · January 22, 2021, 8:55pm

Maybe the mountpoint was not cleanly unmounted. You can check the mount directory on NethServer (or reboot the server as in Windows user-land).

joseoliveira · January 22, 2021, 8:56pm

Hi Andy,

I deleted all files on SMB share: same error.
I deleted backup profile and setup a new one: same error.
I deleted backup profile again and tried a Restic backup:
Backup: backup01
Backup started at 2021-01-22 20:49:32
Pre backup scripts status: SUCCESS
umount: /mnt/backup-backup01: not mounted
Backup directory is not mounted
Can’t initialize restic repository
Action ‘backup-data-restic backup01’: FAIL
Backup status: FAIL

I’m keeping on investigating this.

Kind regards,
Jose

Andy_Wismer · January 22, 2021, 9:07pm

@joseoliveira

Hi

Since you’re using a linux Ubuntu box to store your backups - why not switch to NFS instead of CIFS?
NFS is much faster than the - for Linux - unnatural SMB…

Try it, and check the NFS permissions. I’m quite sure this will work, and be much easier in the end…

That’s what I’m using at home.

My 2 cents
Andy

joseoliveira · January 22, 2021, 10:31pm

Hi Andy,

Trying with NFS now… NFS and SMB directory have 777 permissions:

Backup: backup01
Backup started at 2021-01-22 22:23:57
Pre backup scripts status: SUCCESS
umount: /mnt/backup-backup01: not mounted
Backup directory is not mounted at /etc/e-smith/events/actions/mount-nfs line 88.
Can’t mount /mnt/backup-backup01
Action ‘backup-data-duplicity backup01’: FAIL
Backup status: FAIL

Syslog:
Jan 22 22:23:59 v-server-1 esmith::event[12811]: ===== Report for configuration backup =====
Jan 22 22:23:59 v-server-1 esmith::event[12811]: Backup started at 2021-01-22 22:23:59
Jan 22 22:23:59 v-server-1 esmith::event[12818]: Event: pre-backup-config
Jan 22 22:23:59 v-server-1 esmith::event[12818]: expanding /etc/backup-config.d/nethserver-sssd.include
Jan 22 22:23:59 v-server-1 esmith::event[12818]: Action: /etc/e-smith/events/actions/generic_template_expand SUCCESS [0.278931]
Jan 22 22:24:00 v-server-1 esmith::event[12818]: Action: /etc/e-smith/events/pre-backup-config/S40nethserver-dc-pre-backup SUCCESS [1.093917]
Jan 22 22:24:00 v-server-1 esmith::event[12818]: Action: /etc/e-smith/events/pre-backup-config/S40nethserver-sssd-backup-tdb SUCCESS [0.015185]
Jan 22 22:24:02 v-server-1 esmith::event[12818]: Action: /etc/e-smith/events/pre-backup-config/S50nethserver-backup-config-list-packages SUCCESS [2.069492]
Jan 22 22:24:02 v-server-1 esmith::event[12818]: Event: pre-backup-config SUCCESS
Jan 22 22:24:02 v-server-1 esmith::event[12811]: Event pre-backup-config: SUCCESS
Jan 22 22:24:15 v-server-1 esmith::event[12811]: Action backup-config-execute: SUCCESS
Jan 22 22:24:15 v-server-1 esmith::event[12955]: Event: post-backup-config
Jan 22 22:24:15 v-server-1 esmith::event[12955]: Action: /etc/e-smith/events/post-backup-config/S40nethserver-dc-post-backup SUCCESS [0.071741]
Jan 22 22:24:15 v-server-1 esmith::event[12955]: Event: post-backup-config SUCCESS
Jan 22 22:24:15 v-server-1 esmith::event[12811]: Event post-backup-config: SUCCESS
Jan 22 22:24:15 v-server-1 esmith::event[12811]: Backup status: SUCCESS
Jan 22 22:24:15 v-server-1 esmith::event[12811]: Backup ended at 2021-01-22 22:24:15
Jan 22 22:24:15 v-server-1 esmith::event[12811]: Time elapsed: 0 hours, 0 minutes, 16 seconds
Jan 22 22:24:15 v-server-1 esmith::event[12811]: Action: /etc/e-smith/events/pre-backup-data/S20nethserver-backup-config-predatabackup SUCCESS [16.276955]
Jan 22 22:25:43 v-server-1 esmith::event[12811]: Action: /etc/e-smith/events/pre-backup-data/S50nethserver-ibays-dump-acls SUCCESS [88.417486]
Jan 22 22:25:43 v-server-1 esmith::event[12811]: Event: pre-backup-data SUCCESS
Jan 22 22:25:49 v-server-1 systemd: cockpit-backup-data-backup01.service: main process exited, code=exited, status=1/FAILURE
Jan 22 22:25:49 v-server-1 systemd: Unit cockpit-backup-data-backup01.service entered failed state.
Jan 22 22:25:49 v-server-1 systemd: cockpit-backup-data-backup01.service failed.

[root@v-server-1 log]# systemctl status cockpit-backup-data-backup01
● cockpit-backup-data-backup01.service - /sbin/e-smith/backup-data-wrapper backup01
Loaded: loaded (/run/systemd/system/cockpit-backup-data-backup01.service; static; vendor preset: disabled)
Drop-In: /run/systemd/system/cockpit-backup-data-backup01.service.d
└─50-Description.conf, 50-Environment.conf, 50-ExecStart.conf
Active: failed (Result: exit-code) since Fri 2021-01-22 22:25:49 GMT; 4min 44s ago
Process: 12799 ExecStart=/sbin/e-smith/backup-data-wrapper backup01 (code=exited, status=1/FAILURE)
Main PID: 12799 (code=exited, status=1/FAILURE)

Jan 22 22:24:15 v-server-1.h2.local esmith::event[12811]: Event post-backup-config: SUCCESS
Jan 22 22:24:15 v-server-1.h2.local esmith::event[12811]: Backup status: SUCCESS
Jan 22 22:24:15 v-server-1.h2.local esmith::event[12811]: Backup ended at 2021-01-22 22:24:15
Jan 22 22:24:15 v-server-1.h2.local esmith::event[12811]: Time elapsed: 0 hours, 0 minutes, 16 seconds
Jan 22 22:24:15 v-server-1.h2.local esmith::event[12811]: Action: /etc/e-smith/events/pre-backup-data/S20nethserver-backup-config-predatabackup SUCCESS [16.276955]
Jan 22 22:25:43 v-server-1.h2.local esmith::event[12811]: Action: /etc/e-smith/events/pre-backup-data/S50nethserver-ibays-dump-acls SUCCESS [88.417486]
Jan 22 22:25:43 v-server-1.h2.local esmith::event[12811]: Event: pre-backup-data SUCCESS
Jan 22 22:25:49 v-server-1.h2.local systemd[1]: cockpit-backup-data-backup01.service: main process exited, code=exited, status=1/FAILURE
Jan 22 22:25:49 v-server-1.h2.local systemd[1]: Unit cockpit-backup-data-backup01.service entered failed state.
Jan 22 22:25:49 v-server-1.h2.local systemd[1]: cockpit-backup-data-backup01.service failed.

Any thoughts?

Kind regards,
Jose

Andy_Wismer · January 22, 2021, 10:45pm

Test 1)

If you manually try to mount the NFS share (To any newly created folder as a test) does it work?

Test 2)
Can you mount it manually to the mount point actually used by backup? (/mnt/backup-backup01)

If mounting works, can you create / delete files there (eg touch test or nano test…)?

joseoliveira · January 22, 2021, 11:03pm

Hi Andy,

Yes to both.
Here is the interesting part:
When I created Backup01 for NFS mount wasn’t there (there was one for SMB)…
Now, please, look at this:
Under: /mnt directory
drwxr-xr-x 2 root root 6 Jan 22 20:47 backup-backup01
d??? ? ? ? ? ? backup-backup-02
Both were created by Cockpit…
If I mount manually the NFS storage on /mnt/backup-backup01 and start the backup it keeps on going.
Definitely, there’s a problem with the backup module code regarding the mounts (nfs and cifs).
I will do more investigation and post results tomorrow. I needed desperately that backup!

Kind regards,
Jose

Andy_Wismer · January 22, 2021, 11:06pm

Hi

Seems to be a bug here with Cockpit

Maybe some help here, @davidep, @giacomo ?

Thanks
Andy

Andy_Wismer · January 22, 2021, 11:09pm

Hi

One last idea:

Pls delete all configured backups…
Also delete any “backup” folder under /mnt.

Now use the old server manager NethGUI (If your server is newly installed with 7.9.2009 you’ll need to install this from Software Center). This uses Port 980.

Create a backup in there. Not any choices, it uses duplicity by default.

Give Feedback / logs…

My 2 cents
Andy

joseoliveira · January 22, 2021, 11:22pm

Hi Andy,

I tried something similar as well…
On the old WUI, the dashboard showed an error and none was copied - I don’t believe that a 2.2 TB backup was done in less than 5 minutes! Syslog doesn’t show any error.
Note: I didn’t purge the backup profiled setup on Cockpit.
As soon as I run the backup and increase storage on my hypervisor I will be back with more tests and results…

Kind regards,
Jose

Andy_Wismer · January 22, 2021, 11:27pm

@joseoliveira

Hi

I think the backup share must be purged first, as you’ld either run into a race condition (two or more tasks competing for the same resource…) or a permissions issue…

It’s late here, but I’ll be around tomorrow… Good night!

My 2 cents
Andy

giacomo · January 25, 2021, 7:52am

This is seems to me a problem on the NAS side, I’m not aware of any bug on the backup mount part but … who knows

You could try to execute the backup from command line to gather more info:

backup-data -b <backup_name>

Also take a look at /var/log/messages.
Finally, you could try to invoke manually the mount actions like this one.

joseoliveira · January 28, 2021, 9:24pm

Hi Andy and Giacomo,

I do apologise for late update.
I was trying to run a critical backup (2.2 TB) and I had to wait until I got 2 successful backups…
I’m following Giacomo’s suggestion and initiated a backup from command line minutes ago…
As soon as I get the results I will post them here.

Thank you for your patience.

Kind regards,
Jose