IPsec VPN routing problems

[pike (Michael Kicks)]

I am a routing sub noob, but AFAIK currently NethServer can not manage two paths for the same subnet.
IMVHO you should…

• do a manual configuration backup, call it whenever you want.
• delete OpenVPN client connection on NethServer
• delete the IPSec Tunnel on NethServer
• apply both settings
• check routes on NethServer
• check routes on PfSense too

then… make a decision based on routing status

• If PfSense still have the route to 192.168.4.1/24 using tun, you should delete both connections also on PFSense
• If PFSense do not have the route to 192.168.4.1 using tun, you can re-create the IpSec tunnel on NethServer

These are the tests that i’ll do…

[Andy_Wismer (André Wismer)]

@danb35

As Pike suggested, remove the OpenVPN, maybe even remove both VPNs (IPsec too).
I’d at least make a config-save on the NethServer beforehand.
Then recreate the IPsec connection, but manually specify all Specs.
I think it will “just work” after that.

Andy

[danb35 (Dan)]

@Andy_Wismer

As Pike suggested, remove the OpenVPN, maybe even remove both VPNs (IPsec too).

OK, I deleted the IPsec VPN, and did systemctl disable openvpn-client@pfsense (remember, the OpenVPN configuration was all at the CLI because the GUI is brain-dead in that regard). Then rebooted.

On reboot, of course, neither connection exists. I then reconfigured the IPsec connection, manually specifying the Phase 1 and 2 parameters to match what I’d configured on pfSense (and the defaults there are just awful). It connects just fine, but still has no idea how to route to 192.168.1.0/24:

[root@neth ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         173.249.19.1    0.0.0.0         UG    0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 dummy0
169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 eth0
173.249.19.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.4.0     0.0.0.0         255.255.255.0   U     0      0        0 dummy0
192.168.10.0    192.168.10.2    255.255.255.0   UG    0      0        0 tunrw
192.168.10.2    0.0.0.0         255.255.255.255 UH    0      0        0 tunrw
[root@neth ~]# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
^C
--- 192.168.1.1 ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 6000ms

[root@neth ~]# 

[Andy_Wismer (André Wismer)]

@danb35

Could you show the output of ifconfig from your hosted server?

Not quite understanding this network:
192.168.10.2…

173.249.19.0 This would be the “public” network, aka Internet.

[Andy_Wismer (André Wismer)]

@danb35

Try this:

route add 192.168.1.0 gw 173.249.19.1 eth0

aa in my home Tes-Nethserver routing (When VPN is up):
192.168.43.0 192.168.29.1 255.255.255.0 UG 0 0 0 eth1
The router 192.168.29.1 is outside of the NethServer, in your case the hosters gateway…

In your routing table above,

192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

points to the right Interface, but no “next hop” is given.
A “next hop” is the next router responsible for further routing the traffic, needed if the other side isn’t in the same network. Here, as in my home Test-NethServer, this corresponds to the default gateway.

Note:
This is not persistant after reboot.
To make it persistant:
nano /etc/sysconfig/network

or use the static route in the Web-GUI…

Andy

[danb35 (Dan)]

@Andy_Wismer

Could you show the output of ifconfig from your hosted server?

[root@neth ~]# ifconfig
dummy0: flags=195<UP,BROADCAST,RUNNING,NOARP>  mtu 1500
        inet 192.168.4.1  netmask 255.255.255.0  broadcast 192.168.4.255
        inet6 fe80::49b:99ff:fe72:c372  prefixlen 64  scopeid 0x20<link>
        ether 06:9b:99:72:c3:72  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3  bytes 210 (210.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 173.249.19.149  netmask 255.255.255.0  broadcast 173.249.19.255
        inet6 fe80::250:56ff:fe3d:7155  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:3d:71:55  txqueuelen 1000  (Ethernet)
        RX packets 4318  bytes 1762485 (1.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7856  bytes 6146470 (5.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 2815  bytes 391922 (382.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2815  bytes 391922 (382.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tunrw: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1500
        inet 192.168.10.1  netmask 255.255.255.255  destination 192.168.10.2
        inet6 fe80::d7a:144a:6493:7cc6  prefixlen 64  scopeid 0x20<link>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 100  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3  bytes 144 (144.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

@Andy_Wismer

192.168.10.2…

The .10 network is for the Neth box to act as an OpenVPN server, though I’ve mostly abandoned that.

@Andy_Wismer

route add 192.168.1.0 gw 173.249.19.1 eth0

[root@neth ~]# route add 192.168.1.0 gw 173.249.19.1 eth0
[root@neth ~]# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
^C
--- 192.168.1.1 ping statistics ---
9 packets transmitted, 0 received, 100% packet loss, time 7999ms

[root@neth ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         173.249.19.1    0.0.0.0         UG    0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 dummy0
169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 eth0
173.249.19.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.1.0     173.249.19.1    255.255.255.255 UGH   0      0        0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.4.0     0.0.0.0         255.255.255.0   U     0      0        0 dummy0
192.168.10.0    192.168.10.2    255.255.255.0   UG    0      0        0 tunrw
192.168.10.2    0.0.0.0         255.255.255.255 UH    0      0        0 tunrw

[Andy_Wismer (André Wismer)]

This needs to be removed first:

@danb35

192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

And the VPN should be up when putting in the route…

(It should do that automatically, and the above route should NOT be in the table, VPN up or not…)

192.168.1.0     173.249.19.1    255.255.255.255 UGH   0      0        0 eth0

The Netmask should be /24 here, not like shown /32. (/32 is a single IP, your PFsense at home, but 0 will not work with a /32 mask…)

[danb35 (Dan)]

@Andy_Wismer

(It should do that automatically, and the above route should NOT be in the table, VPN up or not…)

I’m pretty sure it’s the VPN that’s putting that route in the table. Here’s the output of route -n after deleting the route you gave, and disabling the IPsec connection:

[root@neth ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         173.249.19.1    0.0.0.0         UG    0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 dummy0
169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 eth0
173.249.19.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.4.0     0.0.0.0         255.255.255.0   U     0      0        0 dummy0
192.168.10.0    192.168.10.2    255.255.255.0   UG    0      0        0 tunrw
192.168.10.2    0.0.0.0         255.255.255.255 UH    0      0        0 tunrw

…and here it is after re-enabling the IPsec connection:

[root@neth ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         173.249.19.1    0.0.0.0         UG    0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 dummy0
169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 eth0
173.249.19.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.4.0     0.0.0.0         255.255.255.0   U     0      0        0 dummy0
192.168.10.0    192.168.10.2    255.255.255.0   UG    0      0        0 tunrw
192.168.10.2    0.0.0.0         255.255.255.255 UH    0      0        0 tunrw

Note the null route for 192.168.1.0 added when the IPsec connection is enabled.

[Andy_Wismer (André Wismer)]

@danb35

Now the routing table looks correct!

Ping work with VPN?

192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
is not the same as before:

192.168.1.0 173.249.19.1 255.255.255.255 UGH 0 0 0 eth0
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

[danb35 (Dan)]

[root@neth ~]# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
^C
--- 192.168.1.1 ping statistics ---
13 packets transmitted, 0 received, 100% packet loss, time 12002ms

 dan@Dan-MacBook-Pro-2453  ~  ping 192.168.4.1
PING 192.168.4.1 (192.168.4.1): 56 data bytes
64 bytes from 192.168.4.1: icmp_seq=0 ttl=63 time=132.479 ms
64 bytes from 192.168.4.1: icmp_seq=1 ttl=63 time=132.961 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=63 time=133.211 ms
64 bytes from 192.168.4.1: icmp_seq=3 ttl=63 time=132.849 ms
^C
--- 192.168.4.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 132.479/132.875/133.211/0.264 ms

[Andy_Wismer (André Wismer)]

The gateway is still missing in the line

192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

0.0.0.0 will not work as a “route”.

[danb35 (Dan)]

@Andy_Wismer

0.0.0.0 will not work as a “route”.

I know. But that’s the route Neth adds.

[Andy_Wismer (André Wismer)]

Somehow, maybe due to the “dummy” interface, NethServer is not adding in the needed gateway to this route…

[danb35 (Dan)]

Clearly. But deleting the route that Neth added, and re-adding the route you suggested, still isn’t working:

[root@neth ~]# ip route del 192.168.1.0/24 via 0.0.0.0
[root@neth ~]# ip route add 192.168.1.0/24 via 173.249.19.1
[root@neth ~]# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
^C
--- 192.168.1.1 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4037ms

[root@neth ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         173.249.19.1    0.0.0.0         UG    0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 dummy0
169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 eth0
173.249.19.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.1.0     173.249.19.1    255.255.255.0   UG    0      0        0 eth0
192.168.4.0     0.0.0.0         255.255.255.0   U     0      0        0 dummy0
192.168.10.0    192.168.10.2    255.255.255.0   UG    0      0        0 tunrw
192.168.10.2    0.0.0.0         255.255.255.255 UH    0      0        0 tunrw
[root@neth ~]# 

[Andy_Wismer (André Wismer)]

Do you have a link how you did the dummy interface?
I could try that on my Test-NethServer.
I’m taking the easy route at the moment, a Proxmox VM isn’t the same as a hosted server, and my adding a virtual, but still “real” NIC is easier…

[danb35 (Dan)]

@Andy_Wismer

Do you have a link how you did the dummy interface?

It’s been a couple of years, but I think it was this:

Create a dummy interface when you only have 1 NIC available

This is a compilation of 2 discussions on these forums: Dummy Interface (virtual NIC) Virtual network interface for virtual machines Thanks to @asl and @jstammi for doing great work on this topic! add a file for your dummy interface in /etc/sysconfig/network-scripts/ let’s call it ifcfg-dummy0 [image] the kernel module dummy (which is need for dummy devices) is not loaded per default, so we need to add this during server start. Ceate a file dummy.conf inside /etc/modules-load.d. This f…

[Andy_Wismer (André Wismer)]

Ok, I’ll set up a new Test-NethServer, using this instruction and see how far I get.
I’ll get back to you, but might be in 1-2 hours.
I’ll try to connect to the same Site I’m presently using.

Dinnertime here…

[danb35 (Dan)]

@Andy_Wismer

Dinnertime here

Yeah, I have some actual work to do too. Thanks for all the help, though, and at least it seems like I wasn’t missing something totally obvious. For the time being, I’m going to disable IPsec, delete that route via 173.249.19.1, and re-enable OpenVPN.

[Andy_Wismer (André Wismer)]

Something in your setup is screwed up, now just need to pinpoint the problem…
Step one in solving an issue!

[Andy_Wismer (André Wismer)]

@danb35

Hi Dan

Took a while, but i did want to replicate as accurate as possible your situation.

So I had to first grab me the Centos 7 minimal, and set up a basic centos like on a hoster.
This time I only allocated one network interface.

I followed more or less exactly this instruction - judging from the IP, most likely the same as you did…
https://wiki.nethserver.org/doku.php?id=virtual_network_interface&s[]=dummy

Installed Centos 7 minimal:

Then added the “dummy”.

Configured the VPN - as yesterday, but with the relevant differences:

First time up, however:

I did forget first time to adapt the target network on the other side, corrected.
It still took several restarts on the VPN (on both sides!) entailing me having to switch several times from Home WLan to Mobile Hotspot WLan…

And things start to look better:

Pinging from Side B to my “hosted” NethServer worked:

Pinging the other side took one or two restarts, but finally worked:

The route to Side B (192.168.43.0/24) gets set automatically - and also removed, when the VPN is stopped.

Bottom Line:

• IPsec does work from such a “hosted” setup.
• It does seem faster than OpenVPN - but I’d need to do some serious tests before making a definitive statement.
• I will (later on) do a backup on the other side with OpenVPN and one with IPsec, just to compare. I do need to create a backup destination on the other side’s NAS.
• This whole setup was created using Centos 7.8.2003 minimal (ISO) ob Proxmox, also with double NAT.
• The Other end uses OPNsense, virtually feature and setup identical to a current PFsense.

Ergo (The BIG conclusion):

Either there’s a difference somewhere on the way from your setup to mine, implying differences in Versions at the time of installation, accumulated till current today.

Maybe software installations / testing / OpenVPN / whatever unknown is interfering with the correct setup of IPsec and routing on your setup. This could be some minor, easily overlooked detail, like wrong permissions on some routing file set by starting the VPN. And not able to remove or correct the settings therein…

It could also be the digitalised version of Murphy, throwing a digital wrench into the gearbox just for the kicks…
\
\
\
Possible Solutions?

If this were my server, I’d want it fixed, just not to run into lurking issues in the future.
And routing / VPN I consider basic musts!

Complete reinstallation with Centos 7.8.2003, install NethServer over this.
Test Backup / VPN
Restore Data and config and re-test VPN

Depending what you actually have / use and run on your external NethServer and also if you used that box to run tests, it might be worth it!

I leave it up to you to decide, I just wanted to try to collect the facts, as identical to your environment as possible. As in good law: Present the pure facts, envision and present the follow up logic, and let the court decide if this is sound!

Installing a minimal OS like Centos, and NethServer over it is not quite the same thing as installing NethServer directly, but is certainly much closer than eg a debian install, then Proxmox over that. Just bootable ZFS stuff alone can be a headache in Debian, whereas in Proxmox this is really smooth…

So it’s not really a bug, but something “irregular” on your system not allowing correct routing…

My 2 cents
Andy