NethServer Version: 7.5.1804
Hello,
I’m trying to connect Nethserver to an Ubiquiti Edgerouter and I have this errors:
next payload type of ISAKMP Hash Payload has an unknown value: different values
malformed payload in packet
I don’t know how to troubleshoot. I don’t at which stage is the problem and what is the problem. I tried several combinations of encryption, but no success so far.
What is your suggestion?
Istvan
I assume you connect directly from your NS to the ubiquiti router, both have public IPs, no port forwarding.
You can get the IPSEC log on Nethserver with:
journalctl -u ipsec.service
In general you need same settings(encryption, compression, PFS etc.) on both sides.
I did some research and found that it may be a wrong PSK or a firmware issue:
Nethserver IPSEC Docs:
http://docs.nethserver.org/en/v7/vpn.html#ipsec
http://docs.nethserver.org/projects/nethserver-devel/en/v7/nethserver-ipsec-tunnels.html
At both ends I have a public IP. At NS I have PPPoE and at EdgeRouter I have static IP.
The article you posted is about connecting two EdgeRouters.
I triple-checked the settings at both ends and they are the same. At least can you tell from this error at which phase is the problem?
You probably need to change the encryption and hash algorithms which must be the same on both endpoints.
The error may occur in both phases as far as I understood but you should see it in the logs:
Nethserver:
journalctl -u ipsec.service
Ubiquiti:
show vpn log
Feel free to share your anonymized settings:
Nethserver:
db vpn show
Ubiquiti:
show vpn
Try to go a bit deeper into settings.
Also, as rule of thumb: keep the system aligned as date and time. I had a working IpSec configuration that stopped working with the wrong time.
The Nethserver logs are in a continous loop:
Aug 16 18:55:32 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1410: deleting state (STATE_MAIN_I3)
Aug 16 18:55:32 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1411: STATE_MAIN_I2: sent MI2, expecting MR2
Aug 16 18:55:32 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1411: STATE_MAIN_I3: sent MI3, expecting MR3
Aug 16 18:55:32 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1411: next payload type of ISAKMP Hash Payload has an unknown value: 67 (0x43)
Aug 16 18:55:32 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1411: malformed payload in packet
Aug 16 18:55:33 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1411: STATE_MAIN_I3: retransmission; will wait 0.5 seconds for response
Aug 16 18:55:33 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1411: STATE_MAIN_I3: retransmission; will wait 1 seconds for response
Aug 16 18:55:34 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1411: STATE_MAIN_I3: retransmission; will wait 2 seconds for response
Aug 16 18:55:36 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1411: STATE_MAIN_I3: retransmission; will wait 4 seconds for response
Aug 16 18:55:40 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1411: STATE_MAIN_I3: retransmission; will wait 8 seconds for response
Aug 16 18:55:48 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1411: STATE_MAIN_I3: retransmission; will wait 16 seconds for response
Aug 16 18:56:04 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1411: STATE_MAIN_I3: retransmission; will wait 32 seconds for response
Aug 16 18:56:36 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1411: STATE_MAIN_I3: 60 second timeout exceeded after 7 retransmits. Possible authentication failure: no acceptable response to our first encrypted message
Aug 16 18:56:36 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1411: starting keying attempt 2 of an unlimited number
Aug 16 18:56:36 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1412: initiating Main Mode to replace #1411
Aug 16 18:56:36 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1411: deleting state (STATE_MAIN_I3)
Aug 16 18:56:36 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1412: STATE_MAIN_I2: sent MI2, expecting MR2
Aug 16 18:56:36 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1412: STATE_MAIN_I3: sent MI3, expecting MR3
Aug 16 18:56:36 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1412: next payload type of ISAKMP Hash Payload has an unknown value: 137 (0x89)
Aug 16 18:56:36 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1412: malformed payload in packet
Aug 16 18:56:37 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1412: STATE_MAIN_I3: retransmission; will wait 0.5 seconds for response
Aug 16 18:56:37 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1412: STATE_MAIN_I3: retransmission; will wait 1 seconds for response
Aug 16 18:56:38 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1412: STATE_MAIN_I3: retransmission; will wait 2 seconds for response
Aug 16 18:56:40 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1412: STATE_MAIN_I3: retransmission; will wait 4 seconds for response
Aug 16 18:56:44 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1412: STATE_MAIN_I3: retransmission; will wait 8 seconds for response
Aug 16 18:56:52 gw.domain.com pluto[30390]: “myipsec-vpn_ipsec-tunnel/1x1” #1412: STATE_MAIN_I3: retransmission; will wait 16 seconds for response
On the Ubiquity show vpn log doesn’t say much. There are several identical lines:
Aug 16 15:53:32 10[IKE] <586> x.x.x.x is initiating a Main Mode IKE_SA
on Nethserver
db vpn show
myipsec-vpn=ipsec-tunnel
compress=no
dpdaction=hold
esp=custom
espcipher=3des
esphash=md5
esppfsgroup=modp1024
ike=custom
ikecipher=3des
ikehash=md5
ikelifetime=28800
ikepfsgroup=modp1024
left=%ppp0
leftid=@myipsec-vpn.local
leftsubnets=192.168.18.0/24
pfs=no
psk=some_random_text
right=ubi.quiti.ip.address
rightid=@myipsec-vpn.remote
rightsubnets=192.168.1.0/24
salifetime=3600
status=enabled
On Ubiquiti
show vpn
ipsec {
auto-firewall-nat-exclude enable
esp-group FOO0 {
compression disable
lifetime 3600
mode tunnel
pfs disable
proposal 1 {
encryption 3des
hash md5
}
}
ike-group FOO0 {
ikev2-reauth no
key-exchange ikev1
lifetime 28800
proposal 1 {
dh-group 2
encryption 3des
hash md5
}
}
site-to-site {
peer neth.server.ip.address {
authentication {
mode pre-shared-secret
pre-shared-secret some_random_text
}
connection-type initiate
description @myipsec-vpn.remote
ike-group FOO0
ikev2-reauth inherit
local-address ubi.quiti.ip.adress
tunnel 1 {
allow-nat-networks disable
allow-public-networks disable
esp-group FOO0
local {
prefix 192.168.1.0/24
}
remote {
prefix 192.168.18.0/24
}
}
}
}
}
I don’t use ubiquiti edgerouter but maybe I found some errors:
I don’t think the description on ubiquiti matches the ID on Nethserver. It’s just a long description of the IPSEC network whereas the ID identifies server and client. Others used IP address or hostname as ID:
On Ubiquiti you may change ikev1 to ikev2.
Disabled DPD on Nethserver sets it to hold.
To set DPD to hold on ubiquiti:
set vpn ipsec ike-group FOO0 dead-peer-detection action hold
This is not necessary but just to check if it works:
NOTE: There is no need for DPD when IKEv2 is used, as it has a built-in keep-alive mechanism.
Did you enable P2 offload on ubiquiti:
set system offload ipsec enable
Did you set all necessary firewall rules on ubiquiti?
Docs to configure ubiquiti IPSEC VPN: