Backup performance issues

danb35 · June 30, 2020, 7:50pm

NethServer Version: 7.8
Module: backup-data

As I posted in a different topic, I’m having a strange performance issue with my weekly full data backup job this week. The backup kicks off at 01.00 Sunday, and in past weeks, has finished around 14.00-15.00 that same day (taking 13-14 hours). The backup that started this past Sunday is still running (so over 60 hours so far), and is only about 60% complete. I’m puzzled, because I haven’t changed anything at all on the server, and I’ve upgraded my home Internet connection–I’d expect the backup should be faster, not slower.

The backup job is an old-fashioned Duplicity job, configured in the old server-manager before Cockpit or multiple backups were a thing. It’s backing up over NFS to my FreeNAS box. My Neth server is in a Contabo VPS; that system has an OpenVPN connection to my home network. My Internet connection had been 150 Mbps down/20 up; last week I upgraded it to 500 down/40 up. I am getting the advertised performance out of this connection as well.

A few days ago, I set up a second backup job using Restic to DigitalOcean Spaces (which I’m already using for uploads and backups of my Discourse fora). That seems to work pretty nicely, but it doesn’t look like I can set up two Restic jobs, one to DO and the other to my FreeNAS. From the manual:

…which makes it sound like I can’t have two backup jobs with the same engine to different destinations.

So, what to do here? I’m open to troubleshooting this backup, or to just defining a new job with a different engine, but “don’t bother keeping a backup on the FreeNAS box” isn’t an acceptable option.

giacomo · July 1, 2020, 6:43am

Sorry to hear that, but sometime duplicity is really hard to debug.
As a try, you could try to clean up the backup destination and the local Duplicity cache, finally restart the backup.

Mmm not really. It means that you should not use the same destination directory with 2 different engines, like restic and duplicity writing to the same cifs share or S3 bucket.
I’m not a native English speaker, and I would gladly rephrase the manual to better explain the concept.
Any hint on this?

Andy_Wismer · July 1, 2020, 6:58am

@giacomo
@danb35

Hi

Dan IS an American lawyer, and can easily handle “Legalese” in american english. Legalese, even if not always fully understood by laymen (People not working in the legal business), is VERY precise…

Then again, Dan does speak and write excellent american english, which us non laymen easily understand!

Note: The differences between english and american english are minor, but they are there…
Examples: gas / petrol, pants / trowsers, cell / mobile.
However, in IT the differences are less than elsewhere in the english language…

Maybe, time permitting, Dan could fly over the manual and point out ambiguous terms or definations… Finding those ambiguous points is something lawyers are very good at!

My 2 cents
Andy

Even and especially in Italy: If you want a legal bulletproof contract, let a good lawyer check the whole thing through!

danb35 · July 1, 2020, 9:59am

Backup destination is easy enough, but where is the local Duplicity cache?

Ah, that makes more sense.

giacomo · July 1, 2020, 10:01am

The cache is located at /var/lib/nethserver/backup/duplicity/<backup_name>/

federico.ballarini · July 5, 2020, 2:01pm

Take a look to restic engine!

danb35 · July 5, 2020, 4:43pm

Even Restic didn’t help. Fundamentally, the problem seems to be one of network throughput between my Neth box and my FreeNAS server, and it’s pretty strange. My network connection at home is fine, and is considerably faster than it was two weeks ago, both up and down. My Neth box is a VPS at Contabo. Both report decent connection speeds:

[root@neth ~]# speedtest-cli 
Retrieving speedtest.net configuration...
Testing from Contabo GmbH (REDACTED)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Contabo (Nuremberg) [7.56 km]: 2.327 ms
Testing download speed................................................................................
Download: 98.70 Mbit/s
Testing upload speed....................................................................................................
Upload: 66.06 Mbit/s

…and from a random box at home:

root@jdownloader:~ # speedtest-cli 
Retrieving speedtest.net configuration...
Testing from Comcast Business (REDACTED)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Pineland Telephone Co-Op (Metter, GA) [91.13 km]: 76.912 ms
Testing download speed................................................................................
Download: 315.12 Mbit/s
Testing upload speed......................................................................................................
Upload: 22.95 Mbit/s

But bandwidth between the two systems just isn’t good at all:

[root@neth ~]# iperf3 -c 192.168.1.10
Connecting to host 192.168.1.10, port 5201
[  4] local 192.168.3.100 port 54786 connected to 192.168.1.10 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   607 KBytes  4.97 Mbits/sec    0    158 KBytes       
[  4]   1.00-2.00   sec   694 KBytes  5.68 Mbits/sec    4    112 KBytes       
[  4]   2.00-3.00   sec   252 KBytes  2.06 Mbits/sec   38   57.8 KBytes       
[  4]   3.00-4.00   sec   441 KBytes  3.62 Mbits/sec    3   48.6 KBytes       
[  4]   4.00-5.00   sec   189 KBytes  1.55 Mbits/sec    0   55.2 KBytes       
[  4]   5.00-6.00   sec   378 KBytes  3.10 Mbits/sec    0   55.2 KBytes       
[  4]   6.00-7.00   sec   378 KBytes  3.10 Mbits/sec    0   57.8 KBytes       
[  4]   7.00-8.00   sec   189 KBytes  1.55 Mbits/sec    0   63.0 KBytes       
[  4]   8.00-9.00   sec   378 KBytes  3.10 Mbits/sec    0   67.0 KBytes       
[  4]   9.00-10.00  sec   441 KBytes  3.62 Mbits/sec   12   57.8 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  3.86 MBytes  3.23 Mbits/sec   57             sender
[  4]   0.00-10.00  sec  3.43 MBytes  2.88 Mbits/sec                  receiver

iperf Done.

The connection between the Neth box and my home LAN is over the same OpenVPN link it’s been using for the last two years, with the same pfSense router I’ve had for longer than that. Bouncing the connection on my Neth box hasn’t helped. My pfSense box certainly doesn’t seem to be resource-constrained (CPU load never seems to exceed 10%).

Andy_Wismer · July 5, 2020, 5:51pm

@danb35

Hi Dan

If you - for testing purposes - set up an IPsec connection between your Contabo NethServer and your PFsense at home and see how good is the speed over IPsec…

OpenVPN and IPsec are both about the same effort to set up…

I’ve seen issues with Site2Site with OpenVPN which didn’t appear with IPsec…
RoadWarrior OpenVPN is generally better.

My 2 cents
Andy

danb35 · July 5, 2020, 6:39pm

Seems like a good test. I haven’t made any changes to the OpenVPN configuration (and I’d just exported a client config file from pfSense and configured it on the CLI for Neth–this one wasn’t configured through the GUI, as the GUI doesn’t allow upload of .ovpn configuration files), so I wouldn’t expect it to suddenly be the problem, but it’s definitely worth ruling things out.

So, if I can figure out how to get IPSec up and running, I’ll give it a shot–though it isn’t going too well; the settings the pfSense docs recommend don’t seem to exist on Neth.

Andy_Wismer · July 5, 2020, 6:42pm

@danb35

Simplest test would be a quick and dirty SSH Tunnel…
But would give you a comparison how traffic flows (speed, etc).

PFsense, like OPNsense (Which I use) should also accept “lesser” settings, eg older, simpler IKEv1… Even in Law, it doesn’t have to be a “Textbook” response, as long as it’s legit!

IKEv1 and IKEv2 should work on both ends (NethServer and PFsense).
Try IKEv1 with Agressive mode and PFS (Perfect Forward Secrecy). NAT should not be needed.
IKEv2 should also work without problems.

-> The most problems come from Local and Remote Identifiers. This is often, but not always the WAN IP. OPNsense has an option to not verify the other side, I’m not sure bit think PFsense has this too.

If I recall right, at home you don’t have a static IP, but a DynDNS name will work as well - or for a quick one off test, you can just put in the IP by hand (even if it might change by tomorrow,).

My 2 cents
Andy

danb35 · July 5, 2020, 7:35pm

Of course, but there’s the matter of knowing which settings to choose–and compared to OpenVPN (at the CLI, not through the Neth GUI), it’s a royal pain. With OpenVPN, I set up the connection on pfSense, create a client connection, and download the .ovpn file that defines that connection. Drop that file in /etc/openvpn/client/pfsense.conf, and I’m good to go–just systemctl start openvpn-client@pfsense. Of course, it’s not nearly that easy if I want to do it through the GUI; for some inexplicable reason, the only OpenVPN config you can upload into the GUI is some .json nonsense that nothing other than another Neth server uses.

With IPSec, there are about a dozen settings, they all need to match, and the manual has nothing whatsoever to say about how to configure it. But I’ve managed to get it connected, even if I can’t get traffic to route reliably though the connection yet.

For future reference, the trick to get it connected is that not only do all the various protocols need to match, so do the local and remote identifiers. Neth pre-fills these with @<connection_name>.remote and @<connection_name>.local, which don’t appear to be acceptable values for pfSense–replacing these with the respective IP addresses allows the connection to succeed. (which I see you’ve now edited your post to add, but that information wasn’t there initially). But yes, I do have a static IP at home.

But though it’s connected, not all traffic works. From my home LAN, I can ping the Neth box via its IP on the “dummy0” network (192.168.4.1). I can browse to the server manager at that IP address. I can run iperf3 (and yes, the speed is much better–shown below–even if quite erratic).

But from the Neth box I can’t ping to any IP on my LAN, or otherwise make any connection to my LAN. Hmmm. Probably a routing problem, but I’m not sure how much further I’m going to go right now.

iperf3 results:

root@freenas2[~]# iperf3 -c 192.168.4.1  
Connecting to host 192.168.4.1, port 5201
[  5] local 192.168.1.10 port 49589 connected to 192.168.4.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   501 KBytes  4.11 Mbits/sec    0    221 KBytes       
[  5]   1.00-2.00   sec  3.28 MBytes  27.5 Mbits/sec  378    360 KBytes       
[  5]   2.00-3.00   sec  1.77 MBytes  14.9 Mbits/sec   86   7.11 KBytes       
[  5]   3.00-4.00   sec   664 KBytes  5.44 Mbits/sec    0    434 KBytes       
[  5]   4.00-5.00   sec  1.92 MBytes  16.1 Mbits/sec    0    441 KBytes       
[  5]   5.00-6.00   sec  1.97 MBytes  16.5 Mbits/sec    1   1.41 KBytes       
[  5]   6.00-7.00   sec  1.36 MBytes  11.4 Mbits/sec   14    227 KBytes       
[  5]   7.00-8.00   sec  1.10 MBytes  9.19 Mbits/sec    0    234 KBytes       
[  5]   8.00-9.00   sec  1.30 MBytes  10.9 Mbits/sec    1    122 KBytes       
[  5]   9.00-10.00  sec   793 KBytes  6.49 Mbits/sec    1   63.6 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  14.6 MBytes  12.3 Mbits/sec  481             sender
[  5]   0.00-10.00  sec  13.1 MBytes  11.0 Mbits/sec                  receiver

Andy_Wismer · July 5, 2020, 7:41pm

@danb35

Yes, admit guilty to checking the NethServer docs - very bleak on IPsec, and even right now setting up a second NethServer VM) at home, this one with 2 NICs…

But now i want to see how NethServer fares with IPsec / OpenVPN.

NethServer-Test is up, but still needs all software & updates. Then I can begin to setup IPsec…

I’d like to be able to support others on this subject - and I really need to setup NethServer as Firewall / VPN and play the whole thing through myself…

Andy

Andy_Wismer · July 5, 2020, 7:55pm

@danb35

This is the first time I’ve set up a NethServer with 2 NICs from the start. Compared to just one NIC and another box as firewall - a real pita…

The WAN was set to green, the LAN had no IP, needed to set a temp IP twice (Timeout too fast), but it’s downloading software & updates now …

Andy_Wismer · July 5, 2020, 8:31pm

@danb35

Hi

I got a VPN working between a Test-NethServer at home and an OPNsense at a clients in Germany…

Newer Cockpit:

Note:

What’s very special here is that my Test-nethServer has only a double-NAT connection to the Internet. What it “sees” as Internet is actually the OPT2 on my OPNsense, configured as “DHCP Internet” with 192.168.29.x (RED on Test-NethServer). As long as the NethServer initializes the connection, it’s up and stays up.

This shows the situ, left would be your hosted Nethserver, right home.
(Think of the left OPNsense as your hosters internal firewall - this one does no VPN…)

NethServer network setup:

The other side can’t init the VPN, as my OPNsense doesn’t know about this VPN.
And Double NAT stops things there…

I had to temporarily switch my Macbook to use my mobile Hotspot to finish of the other side, as 2 VPNs to the same target won’t work! May also be your routing problem, if your OpenVPN is still “active”.

For the other side I had to manually use the NethServers internal WAN (192.168.29.202) as Remote Identifier… Local and Remote Identifiers are often: WAN IP, an IP Adress, eMail. These need to be allowed on both sides (Not a hardcoded verificator…) and as you say need to be accordingly set (over cross).

This Test-nethServer is a placeholder for your hosted NethServer, and the other Side is a placeholder for your home. (It also has a static IP). And the Firewall are very similiar (OPNsense here, you use PFsense). And I can ping the NAS on the other side from the NethServer, which is basically what I think is what you want to do (Backups…).

The NAS is 192.168.43.70 on the other side…

I started out with the old Dashboard, but soon switched over to the newer Cockpit, which actually allows you to set up IKEv1 or IKEv2. I’m using IKEv2 here.

Hope this helps.

My 2 cents
Andy

Sorry, but I can’t get Cockpit to display in English, but I think it should be clear nevertheless…

danb35 · July 5, 2020, 9:01pm

Gut das ich Deutsch lese. Ich hab’s für heute aufgegeben; morgen versuche ich weiter.

Andy_Wismer · July 5, 2020, 9:03pm

@danb35

Never underestimate a good lawyer!

Good night, & good luck tomorrow!

Andy

PS:

In this special case, correct would be:

Gut, dass ich Deutsch lese.

A comma after gut and the dass with 2 “s” is equivalent to the english “that”. One “s” is just a “the”.
(Good, that I read german…)

Great PIC of Andromeda you made, also the one in Edinburgh!

I made a few of Mars two years ago, when it was really close, but I could have used a better cam then using my aged iPhone. The new one is MUCH better with night fotos.

If you ever come to Switzerland, do contact me first. As I have friends / relatives all over the world, I’ve become quite a tourist guide in the meantime.

Besides the well known swiss alps, see for example “Aareschlucht” in Google, then switch to pictures… It’s very impressive - the people who 50 or more years ago built the path in the sheer rock wall, the whole scenary, and when you emerge from the gorge (both ends) the fantastic swiss alps as your panorama!

Or this place for eating - great food, decent prices. But find a place in the US with such a history?
Even in older Europe, due to the ravages of two world wars, very hard to find in the west. And in the east the state owned everything!

I actually personally know Susanne, who now runs the Restaurant.

danb35 · July 6, 2020, 12:46pm

…and I’ve thereby demonstrated I don’t write it very well. It’s been 30 years since I studied it much, so I might have a bit of an excuse for being a little rusty.

Thanks, though I can see it has lots of room for improvement. Biggest issue is “more light.” But I ran across a video on Youtube a little while ago showing you can photograph some deep-sky objects with just a DSLR, lens, and tripod. No telescope, no tracker, no EQ mount, etc. So I gave it a try, and I’m happy with the result, but it’s clear I have a long way to go. Which may well result in buying lots of gear. Oh well…

I was actually there last summer, though only for a morning in Basel–it was the end point of a Rhine (or Rhein) river cruise. But all we were doing there was going to the airport. Last time I spent any real time there was, again, almost 30 years ago.

And the VPN issue really is orthogonal, so I’ve started a new topic on that: IPsec VPN routing problems