IPsec VPN routing problems

danb35 · July 6, 2020, 12:59pm

NethServer Version: 7.8
Module: IPsec VPN

tl;dr

My Neth server can’t route traffic over an IPsec VPN connection I’ve established through the GUI to my pfSense box.

Background/status

In trying to troubleshoot my backup performance issues, @Andy_Wismer suggested testing over an IPsec VPN connection, rather than the OpenVPN connection I’ve been using. Once I got the connection set up (and TFM is no help at all in this), I see it’s sometimes (though not consistently) showing better performance on iperf3, but it isn’t able to reach hosts on my LAN over that connection. Hosts on my LAN can ping it, run iperf3 against it, and even access the web GUI via the IPsec VPN connection, but the Neth box isn’t able to initiate any connection to hosts on my LAN over this connection.

Configuration

My Neth server is hosted in a Contabo VPS. Its “real” interface (with the public IP address assigned by Contabo) is Red, and I’ve created a dummy interface of dummy0 with an IP of 192.168.4.1/24, designating it green.

I have a pfSense router at home, and my home LAN is on 192.168.1.0/24. The router creates a separate subnet for OpenVPN on 192.168.3.0/24. I’ve used the router’s interface to create a connection profile for Neth at 192.168.3.100/24, dropped that .ovpn file into /etc/openvpn/client/pfsense.conf, and enabled the connection with systemctl enable --now openvpn-client@pfsense. (Doing this through the GUI would have been much trickier; last I checked the GUI didn’t support uploading the standard .ovpn config file, but instead required some .json thing that only another Neth server would generate.) That connection has been working well for as long as I’ve been running the Neth server there.

Then came IPsec. Compared to OpenVPN, IPsec is a pain to configure. There are about a dozen settings, they all need to match between local and remote (including, critically, the local and remote identifiers, which Neth defaults to completely inappropriate values), there’s no documentation on the Neth end about what settings are expected or supported, and the (much more thorough) pfSense docs recommend settings that Neth doesn’t support at all. But that rant aside, with a bit of experimentation, I got a connection set up.

That connection is configured as a net-to-net tunnel, between 192.168.1.0/24 on my LAN side and 192.168.4.0/24 on the Neth side. And from my LAN side, it seems to be working OK. I can reach the Neth server’s web GUI, I can ping it, I can hit it with iperf3, it’s all good. But not so from the Neth end.

From the Neth end, I can’t reach anything on my LAN–I can’t so much as ping 192.168.1.1, the IP of my pfSense router. Nor can I ping anything else on my LAN, nor can I connect to them in any other way (like using iperf).

Screen shots

pfSense configuration

Neth configuration

Conclusions

From the fact that my LAN can connect to the Neth box, but my Neth box can’t connect to hosts on my LAN, I surmise that there’s a routing problem–my router (and therefore my LAN) knows to route traffic to 192.168.4.0/24 through the IPsec link, but the Neth box doesn’t know to route traffic to 192.168.1.0/24 through that same link. But having determined that, I’m not at all sure how to fix it. Help?

Andy_Wismer · July 6, 2020, 1:20pm

@danb35

Strange, that my quick and dirty Test-NethServer has no routing problems to the other side of the IPsec network. Performance from both sides over VPN seems quite good.

My Test-NethServer is a VM in Proxmox, the LAN (Green) of this NethServer is on vmbr2, an internal bridge (Virtual Switch) in Proxmox not (yet) connected to anything. The Nethserver has no issues pinging the other side of the VPN and can actually connect to almost anything there (NAS, Printer, PCs etc). I even tested a NFS connection, after adding the internal IP of my NethServer to the share in question on that NAS - it can mount the drive - no issues!

So the routing “Out of the box” through a VPN seems to work.

Just for info.

Andy

pike · July 6, 2020, 1:51pm

@danb35 feel free to correct anything wrong with my “small version”.

pfSense
Unknown/not reported Red
192.168.1.0/24 Green
192.168.3.0/24 OpenVPN
192.168.4.0/24 Ipsec to NethServer

NethServer VPS (contabo)
Unknown/not reported Red
192.168.3.100/32 Ip Address as OpenVPN Client
192.168.4.1/24 Green
192.168.1.0/24 IPSec to pfSense

My few suggestions…

Do not use “Auto” setting on any side. Specify every detail: IKE version, cypher, integrity, DH group, key lifetime. In my experience, it will largely reduce the possibility of acceptance failure and/ore unexplained (without messing with logs) disconnections. Some other IPSec implementation allow multiple associations for cypher/integrity as proposal, they useful when you are migrating multiple endpoints, but AFAIK NethServer do not allow that
If you are not on dedicated connection, if available, DPD is not optional
check routing table: i don’t know if NethServer will allow the access to 192.168.1.0/24 outside the OpenVPN connection (it was the first connection to route to 192.168.1.0/24, unless you used NAT)
verify if 192.168.1.0/24 is among trusted networks in Nethserver
double check firewall rules: NethServer should automatically allow connection/data transfer between VPN segment and Green, but sometimes is not a bad idea verify and add some useful rules from the 192.168.1.0/24 subnet and VPN zone.

As option consider to temporary delete the OpenVPN on both sides and/or remove only on the nethserver (thus is a client)

Last but not least: AFAIK Nethserver act as IpSec initiator, cannot be configured as responder…

Andy_Wismer · July 6, 2020, 2:01pm

@danb35

Hi Dan

I fired up my Test-NethServer again, just to take a peek at it’s routing:

This is when the VPN is up & active. When the VPN is not up, that route disappears…
(The last of the blue maked lines…)

If the automatic routing in NethServer does not work, you may need to check if your OpenVPN config is interfereing… Both point to the same IP Network as Target.
You may try a manual routing entry, using the conventional linux commands…

Small reference:
https://www.cyberciti.biz/faq/linux-route-add/

I can second Pike’s comment to “Automatic”, it’s never what you really want!
I set all attributes on both sides by hand, and get a more secure yet fast network!

In your case, with static IP on both sides of the VPN, it doesn’t matter about initiator, as long as the nethserver starts that (My Test-Nethserver is also the initiator), it’ll work.
On the Home end, the PFsense sees itself as router, and your LAN hosts regard that PFsense as default gateway, implying “That guy knows where to forward my packets…”.

Andy

danb35 · July 6, 2020, 2:21pm

Yeah, when the OpenVPN connection is up, it establishes a route for 192.168.1.0/24 (via 192.168.3.1, pfSense’s IP on that network), but when it’s down, that route isn’t there:

So it seems like the obvious thing to do is to set up a route for 192.168.1.0/24 via whatever the pfSense box’s IP is on the 192.168.4 subnet. The only problem with that is that it doesn’t have one. Setting the route via 4.1 doesn’t work:

[root@neth ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         173.249.19.1    0.0.0.0         UG    0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 dummy0
169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 eth0
173.249.19.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.1.0     192.168.4.1     255.255.255.0   UG    0      0        0 dummy0
192.168.4.0     0.0.0.0         255.255.255.0   U     0      0        0 dummy0
192.168.10.0    192.168.10.2    255.255.255.0   UG    0      0        0 tunrw
192.168.10.2    0.0.0.0         255.255.255.255 UH    0      0        0 tunrw
[root@neth ~]# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
^C
--- 192.168.1.1 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5000ms

[root@neth ~]#

pike · July 6, 2020, 3:24pm

I am a routing sub noob, but AFAIK currently NethServer can not manage two paths for the same subnet.
IMVHO you should…

do a manual configuration backup, call it whenever you want.
delete OpenVPN client connection on NethServer
delete the IPSec Tunnel on NethServer
apply both settings
check routes on NethServer
check routes on PfSense too

then… make a decision based on routing status

If PfSense still have the route to 192.168.4.1/24 using tun, you should delete both connections also on PFSense
If PFSense do not have the route to 192.168.4.1 using tun, you can re-create the IpSec tunnel on NethServer

These are the tests that i’ll do…

Andy_Wismer · July 6, 2020, 3:24pm

@danb35

As Pike suggested, remove the OpenVPN, maybe even remove both VPNs (IPsec too).
I’d at least make a config-save on the NethServer beforehand.
Then recreate the IPsec connection, but manually specify all Specs.
I think it will “just work” after that.

Andy

danb35 · July 6, 2020, 3:50pm

OK, I deleted the IPsec VPN, and did systemctl disable openvpn-client@pfsense (remember, the OpenVPN configuration was all at the CLI because the GUI is brain-dead in that regard). Then rebooted.

On reboot, of course, neither connection exists. I then reconfigured the IPsec connection, manually specifying the Phase 1 and 2 parameters to match what I’d configured on pfSense (and the defaults there are just awful). It connects just fine, but still has no idea how to route to 192.168.1.0/24:

[root@neth ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         173.249.19.1    0.0.0.0         UG    0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 dummy0
169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 eth0
173.249.19.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.4.0     0.0.0.0         255.255.255.0   U     0      0        0 dummy0
192.168.10.0    192.168.10.2    255.255.255.0   UG    0      0        0 tunrw
192.168.10.2    0.0.0.0         255.255.255.255 UH    0      0        0 tunrw
[root@neth ~]# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
^C
--- 192.168.1.1 ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 6000ms

[root@neth ~]#

Andy_Wismer · July 6, 2020, 4:35pm

@danb35

Could you show the output of ifconfig from your hosted server?

Not quite understanding this network:
192.168.10.2…

173.249.19.0 This would be the “public” network, aka Internet.

Andy_Wismer · July 6, 2020, 4:45pm

@danb35

Try this:

route add 192.168.1.0 gw 173.249.19.1 eth0

aa in my home Tes-Nethserver routing (When VPN is up):
192.168.43.0 192.168.29.1 255.255.255.0 UG 0 0 0 eth1
The router 192.168.29.1 is outside of the NethServer, in your case the hosters gateway…

In your routing table above,

192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

points to the right Interface, but no “next hop” is given.
A “next hop” is the next router responsible for further routing the traffic, needed if the other side isn’t in the same network. Here, as in my home Test-NethServer, this corresponds to the default gateway.

Note:
This is not persistant after reboot.
To make it persistant:
nano /etc/sysconfig/network

or use the static route in the Web-GUI…

Andy

danb35 · July 6, 2020, 4:51pm

[root@neth ~]# ifconfig
dummy0: flags=195<UP,BROADCAST,RUNNING,NOARP>  mtu 1500
        inet 192.168.4.1  netmask 255.255.255.0  broadcast 192.168.4.255
        inet6 fe80::49b:99ff:fe72:c372  prefixlen 64  scopeid 0x20<link>
        ether 06:9b:99:72:c3:72  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3  bytes 210 (210.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 173.249.19.149  netmask 255.255.255.0  broadcast 173.249.19.255
        inet6 fe80::250:56ff:fe3d:7155  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:3d:71:55  txqueuelen 1000  (Ethernet)
        RX packets 4318  bytes 1762485 (1.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7856  bytes 6146470 (5.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 2815  bytes 391922 (382.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2815  bytes 391922 (382.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tunrw: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1500
        inet 192.168.10.1  netmask 255.255.255.255  destination 192.168.10.2
        inet6 fe80::d7a:144a:6493:7cc6  prefixlen 64  scopeid 0x20<link>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 100  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3  bytes 144 (144.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The .10 network is for the Neth box to act as an OpenVPN server, though I’ve mostly abandoned that.

[root@neth ~]# route add 192.168.1.0 gw 173.249.19.1 eth0
[root@neth ~]# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
^C
--- 192.168.1.1 ping statistics ---
9 packets transmitted, 0 received, 100% packet loss, time 7999ms

[root@neth ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         173.249.19.1    0.0.0.0         UG    0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 dummy0
169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 eth0
173.249.19.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.1.0     173.249.19.1    255.255.255.255 UGH   0      0        0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.4.0     0.0.0.0         255.255.255.0   U     0      0        0 dummy0
192.168.10.0    192.168.10.2    255.255.255.0   UG    0      0        0 tunrw
192.168.10.2    0.0.0.0         255.255.255.255 UH    0      0        0 tunrw

Andy_Wismer · July 6, 2020, 4:54pm

This needs to be removed first:

And the VPN should be up when putting in the route…

(It should do that automatically, and the above route should NOT be in the table, VPN up or not…)

192.168.1.0     173.249.19.1    255.255.255.255 UGH   0      0        0 eth0

The Netmask should be /24 here, not like shown /32. (/32 is a single IP, your PFsense at home, but 0 will not work with a /32 mask…)

danb35 · July 6, 2020, 5:03pm

I’m pretty sure it’s the VPN that’s putting that route in the table. Here’s the output of route -n after deleting the route you gave, and disabling the IPsec connection:

[root@neth ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         173.249.19.1    0.0.0.0         UG    0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 dummy0
169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 eth0
173.249.19.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.4.0     0.0.0.0         255.255.255.0   U     0      0        0 dummy0
192.168.10.0    192.168.10.2    255.255.255.0   UG    0      0        0 tunrw
192.168.10.2    0.0.0.0         255.255.255.255 UH    0      0        0 tunrw

…and here it is after re-enabling the IPsec connection:

[root@neth ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         173.249.19.1    0.0.0.0         UG    0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 dummy0
169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 eth0
173.249.19.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.4.0     0.0.0.0         255.255.255.0   U     0      0        0 dummy0
192.168.10.0    192.168.10.2    255.255.255.0   UG    0      0        0 tunrw
192.168.10.2    0.0.0.0         255.255.255.255 UH    0      0        0 tunrw

Note the null route for 192.168.1.0 added when the IPsec connection is enabled.

Andy_Wismer · July 6, 2020, 5:05pm

@danb35

Now the routing table looks correct!

Ping work with VPN?

192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
is not the same as before:

192.168.1.0 173.249.19.1 255.255.255.255 UGH 0 0 0 eth0
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

danb35 · July 6, 2020, 5:06pm

[root@neth ~]# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
^C
--- 192.168.1.1 ping statistics ---
13 packets transmitted, 0 received, 100% packet loss, time 12002ms

 dan@Dan-MacBook-Pro-2453  ~  ping 192.168.4.1
PING 192.168.4.1 (192.168.4.1): 56 data bytes
64 bytes from 192.168.4.1: icmp_seq=0 ttl=63 time=132.479 ms
64 bytes from 192.168.4.1: icmp_seq=1 ttl=63 time=132.961 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=63 time=133.211 ms
64 bytes from 192.168.4.1: icmp_seq=3 ttl=63 time=132.849 ms
^C
--- 192.168.4.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 132.479/132.875/133.211/0.264 ms

Andy_Wismer · July 6, 2020, 5:07pm

The gateway is still missing in the line

192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

0.0.0.0 will not work as a “route”.

danb35 · July 6, 2020, 5:08pm

I know. But that’s the route Neth adds.

Andy_Wismer · July 6, 2020, 5:08pm

Somehow, maybe due to the “dummy” interface, NethServer is not adding in the needed gateway to this route…

danb35 · July 6, 2020, 5:11pm

Clearly. But deleting the route that Neth added, and re-adding the route you suggested, still isn’t working:

[root@neth ~]# ip route del 192.168.1.0/24 via 0.0.0.0
[root@neth ~]# ip route add 192.168.1.0/24 via 173.249.19.1
[root@neth ~]# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
^C
--- 192.168.1.1 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4037ms

[root@neth ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         173.249.19.1    0.0.0.0         UG    0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 dummy0
169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 eth0
173.249.19.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.1.0     173.249.19.1    255.255.255.0   UG    0      0        0 eth0
192.168.4.0     0.0.0.0         255.255.255.0   U     0      0        0 dummy0
192.168.10.0    192.168.10.2    255.255.255.0   UG    0      0        0 tunrw
192.168.10.2    0.0.0.0         255.255.255.255 UH    0      0        0 tunrw
[root@neth ~]#

Andy_Wismer · July 6, 2020, 5:15pm

Do you have a link how you did the dummy interface?
I could try that on my Test-NethServer.
I’m taking the easy route at the moment, a Proxmox VM isn’t the same as a hosted server, and my adding a virtual, but still “real” NIC is easier…