Backup multiple USB disks

backup
v7
(Bouke Henstra) #1

NethServer Version: 7.3.1611
Module: Backup

I have noticed the following posts:

I still have some questions. I guess my question is regarding to Duplicity. I decided to create this new topic.

The first question is regarding to the use of multiple USB disks. Does it work out of the box?

I have noticed that the next backup fails when I remove the back-up files from the USB disk (/mnt/backup). I have worked around this by removing the folder in [ /var/lib/nethserver/backup/duplicity/ ]. The backup works again, by executing the [ /sbin/e-smith/backup-data ] command.

I am currently re-running the first back-up to reproduce the above. I just want to verify if the behaviour is consequent (I am using this product for the first time).

What I would like to know is if there is anyone who is using the backup as is (out of the box) with multiple USB hard disks. Is the above behaviour normal? Or am I dealing with a coincidence? Please bear with me - as the backup process takes several hours (I can test it myself but it takes long - some help of an experienced user would be very much appreciated).

The second question is with regards to compression ratio. I have noticed that the files produced by Duplicity are tar.gz volumes. What is the best way to increase the compression ratio? I guess I could pass the environment variable “GZIP=-9” system wide - but what is best practice? How am I supposed to accomplish this on a system running Nethserver?

By the way: is there a reason to use volumes of 250M? Is it possible to increase these to eg 1000M?

I am hoping to use the backup module as is with multiple USB hard disks. I am willing to add a hard disk and use rsync if required… but I prefer an out of the box experience.

(Filippo Carletti) #2

I’m not sure that using multiple disk with incremental backups could work. I prefer use a single disk as destination and copy its content to multiple usb disks.

Lower values are helpful when the backup is restarted (too hard to explain for me, please read duplicity documentation).
It can be changed:
config setprop backup-data VolSize=1000
Your next backup will have bigger slices.

I don’t know how to control compression ratio and a quick search didn’t find more details.

(Stefano Zamboni) #3

AFAIR there’s no way to change compression ratio

(Bouke Henstra) #4

Many thanks for all your help!

I have connected two USB3 drives to the Nethserver.
1x 4TB disk with a red chassis (this one remains connected to the server permanently)
1x 3TB disk - we have 5 of these which will be changed every working day

I ended up with two scripts:

/opt/backup/pre-backup-data.sh
/opt/backup/post-backup-data.sh

In /etc/e-smith/events/pre-backup-data/ I have created a symbolic link:
S60-pre-backup-data.sh -> /opt/backup/pre-backup-data.sh

In /etc/e-smith/events/post-backup-data/ I have created a symbolic link:
S79-post-backup-data.sh -> /opt/backup/post-backup-data.sh

pre-backup-data.sh

#!/bin/bash


####################################################################################################
# WHAT DISK?
# We required two mounted disks: SDC and SDD
# The 4.00 TB disk has to be mounted to /mnt/backup
# The 3.00 TB disk has to be mounted to /mnt/sync

dmesg | grep "sdc" | grep "3.00" | grep "TB" || backup="/dev/sdc1"
dmesg | grep "sdc" | grep "4.00" | grep "TB" || sync="/dev/sdc1"

dmesg | grep "sdd" | grep "3.00" | grep "TB" || backup="/dev/sdd1"
dmesg | grep "sdd" | grep "4.00" | grep "TB" || sync="/dev/sdd1"

####################################################################################################
# MOUNT SYNC DISK
# This is the disk which should be changed on working days by the backup administrator.

if [ $(mount | grep -c /mnt/sync) != 1 ]
then
        mount $sync /mnt/sync || exit 1
        echo "/mnt/sync is now mounted"
else
        echo "/mnt/sync already mounted"
fi

####################################################################################################
# MOUNT BACKUP DISK
# This is the disk which is connected to the server permanently

if [ $(mount | grep -c /mnt/backup) != 1 ]
then
        mount $backup /mnt/backup || exit 1
        echo "/mnt/backup is now mounted"
else
        echo "/mnt/backup already mounted"
fi

post-backup-data.sh

#!/bin/bash

####################################################################################################
# MOUNT DISKS
# We will run the pre-backup.sh script as this script is only used to mount the disks

/opt/backup/pre-backup-data.sh

####################################################################################################
# SYNC DIRS

  #timeStamp
  myTimeStamp=$(date -d "today" +"%Y-%m-%d-%Hh%M")

  #check log dir
   if [ ! -d "/var/log/rsync" ]; then
   mkdir "/var/log/rsync"
   fi

  echo "Sync /mnt/backup to /mnt/sync"
  rsync -a --log-file="/var/log/rsync/backup-data-$myTimeStamp.log" --delete --exclude="lost+found" /mnt/backup /mnt/sync
  if [[ $? -gt 0 ]]
  then
   #error
   echo "Er is een serieus probleem met de synchronisatie van de backup op SERVER1! Er dient direct actie ondernomen te worden om dit op te lossen." > /tmp/rsync-mail.txt
   echo "" >> /tmp/rsync-mail.txt
   echo "Log file:" >> /tmp/rsync-mail.txt
   cat /var/log/rsync/backup-data-$myTimeStamp.log >> /tmp/rsync-mail.txt
   mail -r admin@ourcustomer.inet -c support@yeahright.inet -s "Our Customer: RSync SERVER1 Failure /mnt/backup to /mnt/sync" maillog@ourcustomer.inet < /tmp/rsync-mail.txt
  else
   #success
  echo "De synchronisatie van de backup op SERVER1 is gelukt. De USB disk mag omgewisseld worden. Alvast bedankt en een fijne dag toegewenst." > /tmp/rsync-mail.txt
  echo "" >> /tmp/rsync-mail.txt
  echo "Log file:" >> /tmp/rsync-mail.txt
  cat /var/log/rsync/backup-data-$myTimeStamp.log >> /tmp/rsync-mail.txt
  mail -r admin@ourcustomer.inet -s "Our Customer: RSync SERVER1 Success /mnt/backup to /mnt/sync" maillog@ourcustomer.inet < /tmp/rsync-mail.txt
  fi

####################################################################################################
# UMOUNT SYNC DISK
# This is the disk which should be changed by the backup admin every working day

if [ $(mount | grep -c /mnt/sync) != 1 ]
then
        echo "/mnt/sync is not mounted"
else
        umount /mnt/sync || exit 1
        echo "/mnt/sync is now unmounted"
fi

I have changed the email addresses and the name of our customer (and so forth).

I am hoping the above is helpful for others who are struggling with back-ups.

(Alessio Fattorini) #5

Ehi man, are you still using these scripts above? Does it work?