Errors in Working Backup Logs

Support
1

Hey folks,

I’ve got a nightly rsync data-backup running between Nethserver 1 and Nethserver 2. Data set looks complete, and I’m pretty happy with how it’s functioning, but I’m seeing some concerning things in the logs and I wanted to see if anyone had any insights on what might be causing them.

The internet connection between these two servers is VERY slow. So the backups frequently take a very long time.

As you can see in the logs below there are several “connection refused/timeout” and “no such file” messages. In case it matters, I have sanitized the logs a bit, and sshd is not running on port 22.

Thanks for having a look!
Ted

Completed backup from yesterday:

/var/log/backup/backup-Nethserver1_backup_system-202102282200.log

rsync error: unexplained error (code 255) at io.c(820) [sender=3.1.2]
rsync_tmbackup: Backup completed without errors.
ssh: connect to host Nethserver2 port 22: Connection refused
ssh: connect to host Nethserver2 port 22: Connection refused
ssh: connect to host Nethserver2 port 22: Connection refused
Action 'backup-data-rsync Nethserver1_backup_system': SUCCESS
Post backup scripts status: SUCCESS
Backup status: SUCCESS
Backup ended at 2021-03-01 13:07:38

Time elapsed: 15 hours, 7 minutes, 37 seconds

Backup in progress from today:

/var/log/backup/backup-Nethserver1_backup_system-202103012200.log

Backup: Nethserver1_backup_system
Backup started at 2021-03-01 22:00:02
Pre backup scripts status: SUCCESS
df: ‘’: No such file or directory
rsync_tmbackup: Previous backup found - doing incremental backup from user@Nethserver2:/storage/Backup/2021-02-28-220017
rsync_tmbackup: Creating destination user@Nethserver2:/storage/Backup/2021-03-01-220023
ssh: connect to host Nethserver2 port 22: Connection timed out
rsync_tmbackup: Starting backup...
rsync_tmbackup: From: /
rsync_tmbackup: To:   user@Nethserver2:/storage/Backup/2021-03-01-220023/
/usr/bin/rsync_tmbackup: line 547: -D --numeric-ids --links --hard-links --one-file-system  --times --recursive --perms --owner --group --stats --human-readable --files-from=/tmp/qYwHAlvmCq  --exclude-from=/tmp/kfK33Ns0Ev =-D --numeric-ids --links --hard-links --one-file-system  --times --recursive --perms --owner --group --stats --human-readable --files-from=/tmp/qYwHAlvmCq  --exclude-from=/tmp/kfK33Ns0Ev  --compress: No such file or directory
rsync_tmbackup: Running command:
rsync_tmbackup: rsync  -e 'ssh -p 22 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null' -D --numeric-ids --links --hard-links --one-file-system  --times --recursive --perms --owner --group --stats --human-readable --files-from=/tmp/qYwHAlvmCq  --exclude-from=/tmp/kfK33Ns0Ev  --log-file '/root/.rsync_tmbackup/2021-03-01-220452.log' --link-dest='/storage/Backup/2021-02-28-220017' -- '/' 'user@Nethserver2:/storage/Backup/2021-03-01-220023/'
2

Can you connect to Nethserver 2 via SSH manually?

Is there maybe fail2ban blocking SSH on Nethserver 2? It looks like in yesterdays backup the connection was just refused whereas today it seems not available anymore (timed out)

3

Great question. IPS is turned on, but not Fail2Ban. Also, the backups are actually completing. I can see the files appearing on Nethserver2 correctly.

4

Maybe the errors occur just because of slow connection timeouts.

5

That would explain the timeouts. Honestly, I’m most concerned with the “no such file” errors. Every day’s log begins with the “df ‘’ no such file” If that’s expected behavior, great, but I hate to take chances with a backup. :slight_smile:

6

If the connection is to slow to provide the access to the file in time you may get that error too.
Please check if the backup is there completely (check size).
You may also try an internal backup to check if the error still occurs.

7

I’ve been checking file sizes. Everything looks good there. I’ll try the local backup and see what happens.

1 Like
8

Just a follow up in case anyone else is tracking this. It turns out that Nethserver2, running on a Pi4, is having issues with it’s disk array (a Terramaster D5-300C.

kernel: usb 2-1.3: reset SuperSpeed Gen 1 USB device number 11 using xhci_hcd

These usb errors seem to be common to Pi4s and to many devices using ASMEDIA chips – so both the Pi and the Terramaster. The common solution appears to be using the kernel usbquirks setup. But I can’t figure out how to do that on CentOS. :frowning: Usually, it’s done with a module setup, but unless I’m mistaken, CentOS includes usb drivers in the kernel instead of including them as modules.

It could be that the backup errors are unrelated to this USB issue, but I figure solving the USB issue is step one and I can see if the backup issues continue once it’s fixed.