Duplicity error on full backup to NFS share

Guus · May 31, 2017, 4:09pm

L.s.

After setting up a nfs share, which I can acces over W10-client, tried to backup-data to this share but getting errors which are duplicity related.

Thank you for any clues

Guus

===== Report for data backup =====

Backup started at 2017-05-31 17:37:11
Pre backup scripts status: SUCCESS
Backup script status: ERROR

Extract from log file /var/log/backup-data.log:

2017-05-31 17:37:11 - START - Backup data started
2017-05-31 17:37:25 - STEP - pre-backup-done done
2017-05-31 17:37:25 - ERROR - Backup failed, see /var/log/last-backup.log for details - 7680
2017-05-31 17:37:25 - ERROR - Action backup-data-duplicity failed - 1

Extract from log file /var/log/last-backup.log:

Reading globbing filelist /tmp/wARS7ua9CR
Traceback (most recent call last):
_ File “/usr/bin/duplicity”, line 1546, in _
_ with_tempdir(main)_
_ File “/usr/bin/duplicity”, line 1540, in with_tempdir_
_ fn()_
_ File “/usr/bin/duplicity”, line 1375, in main_
_ action = commandline.ProcessCommandLine(sys.argv[1:])_
_ File “/usr/lib64/python2.7/site-packages/duplicity/commandline.py”, line 1154, in ProcessCommandLine_
_ set_selection()_
_ File “/usr/lib64/python2.7/site-packages/duplicity/commandline.py”, line 978, in set_selection_
_ sel.ParseArgs(select_opts, select_files)_
_ File “/usr/lib64/python2.7/site-packages/duplicity/selection.py”, line 272, in ParseArgs_
_ for sf in self.filelist_globbing_get_sfs(filelists[filelists_index], 1, arg):_
_ File “/usr/lib64/python2.7/site-packages/duplicity/selection.py”, line 371, in filelist_globbing_get_sfs_
_ yield self.glob_get_sf(line, include)_
_ File “/usr/lib64/python2.7/site-packages/duplicity/selection.py”, line 434, in glob_get_sf_
_ sel_func = self.glob_get_filename_sf(glob_str, include)_
_ File “/usr/lib64/python2.7/site-packages/duplicity/selection.py”, line 502, in glob_get_filename_sf_
_ raise FilePrefixError(filename)_
FilePrefixError: +

dnutan · June 3, 2017, 4:59pm

This error could be caused by a blank line in a list of included paths to backup.
Does /var/log/messages shows any additional information at the time the backup failed? (eg. Requested path not found)

Can you report the output of this command:

grep -rc "^$" /etc/backup-data.d/

Guus · June 4, 2017, 9:41am

In the messages log I only see following information regarding backup data:

Jun 4 00:15:02 yz2nsrv esmith::event[7409]: Action: /etc/e-smith/events/pre-backup-config/S40nethserver-dc-pre-backup SUCCESS [0.625871]
Jun 4 00:15:02 yz2nsrv esmith::event[7409]: Action: /etc/e-smith/events/pre-backup-config/S40nethserver-sssd-backup-tdb SUCCESS [0.009506]
Jun 4 00:15:03 yz2nsrv esmith::event[7409]: Action: /etc/e-smith/events/pre-backup-config/S50nethserver-backup-config-list-packages SUCCESS [1.071455]
Jun 4 00:15:03 yz2nsrv esmith::event[7409]: Event: pre-backup-config SUCCESS
Jun 4 00:15:10 yz2nsrv esmith::event[7733]: Event: post-backup-config
Jun 4 00:15:10 yz2nsrv esmith::event[7733]: Action: /etc/e-smith/events/post-backup-config/S40nethserver-dc-post-backup SUCCESS [0.029241]
Jun 4 00:15:10 yz2nsrv esmith::event[7733]: Event: post-backup-config SUCCESS

Output of grep is as follows:
grep -rc “^$” /etc/backup-data.d/
/etc/backup-data.d/nethserver-base.exclude:0
/etc/backup-data.d/nethserver-base.include:0
/etc/backup-data.d/nextcloud.include:0
/etc/backup-data.d/custom.exclude:0
/etc/backup-data.d/custom.include:0
/etc/backup-data.d/nethserver-backup-data.exclude:0
/etc/backup-data.d/nethserver-collectd.include:0
/etc/backup-data.d/nethserver-virtualhosts.include:1

But the virtualhosts.include file looks empty although there are 3 virtualhosts made, but not installed till now

I’ve remove the old file and create a new empty one. It seems that now the backup is up and running since I didn’t get the error message within second.

I’ll keep you informed about the results.

In between result:
first duplicity files are now visible in the nfs share directory, problem therefore solved.

Thanks for all support.

Regards Guus

dnutan · June 4, 2017, 10:35am

Glad it’s working again!

Would you mind to run another test, just to see if this error will come back?

expand-template /etc/backup-data.d/nethserver-virtualhosts.include
grep -c "^$" /etc/backup-data.d/nethserver-virtualhosts.include

Guus · June 4, 2017, 8:59pm

Hello Marc,

run the test you suggested .

Outcome is as follows (grep…) which leads to the conclusion that once again the file nethserver-virtualhosts.include has the parameter 1L, 1C allthough it looks like a empty file, so it has returned to the original situation, causing the error of data-backup.
grep -rc “^$” /etc/backup-data.d/
/etc/backup-data.d/nethserver-base.exclude:0
/etc/backup-data.d/nethserver-base.include:0
/etc/backup-data.d/nextcloud.include:0
/etc/backup-data.d/custom.exclude:0
/etc/backup-data.d/custom.include:0
/etc/backup-data.d/nethserver-backup-data.exclude:0
/etc/backup-data.d/nethserver-collectd.include:0
/etc/backup-data.d/nethserver-virtualhosts.include.old:1
/etc/backup-data.d/nethserver-virtualhosts.include:1

Momentary the first full backup is still running, with a total size of 600 GB I’am at about 50% after 12 hours

Realizing this I suggest it should be possible to run a full backup by hand and after this only incrementals till one decides a new full backup is usefull. A full backup on weekly basis taking about 24 hours to generate is some what massive.

Regards,

Guus

dnutan · June 4, 2017, 10:41pm

So we are left to figure out the cause.
This is the perl code filling nethserver-virtualhost.include file:

github.com

NethServer/nethserver-virtualhosts/blob/7f464ea6be9ec77856933571fccc0680cdc48f68/root/etc/e-smith/templates/etc/backup-data.d/nethserver-virtualhosts.include/10base

{
    $OUT = '';
    foreach my $dir (glob('/var/www/html/*')) {
        my $rpm = `LANG=C /usr/bin/rpm -qf $dir`;
        if ($rpm =~ /is not owned by any package/) {
            $OUT .= "$dir\n";
        }
    }
    $OUT =~ s/\s+$//;
}

I think a list of /var/www/html directory will help (either ls -d1 /var/www/html/* or readlink -f /var/www/html/*), as long as there’s no personal information you don’t want to disclose.

Guus · June 5, 2017, 8:03am

Hello Marc,

Let me explain my situation. I’ve already prepared the virtual hosts but up till now not put any files in these directories. Therefore the outcome of your commands is as follows:

ls -d1 /var/www/html/*
ls: cannot access /var/www/html/*: No such file or directory

readlink -f /var/www/html/*
/var/www/html/*

Hope this helpd you

Regards,

Guus

dnutan · June 5, 2017, 10:03am

Yes, it helps. I can reproduce the error, and it’s the same problem @Andrey_Romakhin is having.
I might be wrong but maybe the $OUT declaration has something to do, as it’s not undefined when the directory is empty?

I changed this topic to the #bug category. I’m sure @giacomo will look into that promptly, as time allows him.

giacomo · June 6, 2017, 6:40am

I will continue the discussion here, since we are talking of the same problem: