High memory usage - SWAP

prostream · July 23, 2017, 10:49am

So i figured out that setting up the ipsec tunnel correctly wasn´t the solution. After 4 Days my RAM Usage is that high again:

Any further ideas?

dnutan · July 23, 2017, 11:48am

Anything relevant from:

journalctl -u ipsec.service

prostream · July 23, 2017, 12:56pm

Very many entries like this:

"Site-to-Site"[1] 79.235.11.89 #11118: Aggressive mode peer ID is ID_FQDN: '@cloud.myipsecpartner.de'
Jul 19 13:05:45 net01.myserver.co pluto[1866]: "Site-to-Site"[1] 79.235.11.89 #11118: EXPECTATION FAILED at /builddir/build/BUILD/libreswan-3.15/programs/pluto/ikev1.c:2843: r != NULL
Jul 19 13:05:45 net01.myserver.co pluto[1866]: "Site-to-Site"[1] 79.235.11.89 #11118: no suitable connection for peer '@cloud.myipsecpartner.de'
Jul 19 13:05:45 net01.myserver.co pluto[1866]: "Site-to-Site"[1] 79.235.11.89 #11118: initial Aggressive Mode packet claiming to be from @myipsecpartner.no-ip.org on 79.235.11.89 but no connection has been authorized
Jul 19 13:05:45 net01.myserver.co pluto[1866]: "Site-to-Site"[1] 79.235.11.89 #11118: sending notification INVALID_ID_INFORMATION to 79.235.11.89:500
Jul 19 13:05:45 net01.myserver.co pluto[1866]: packet from 79.235.11.89:500: received Vendor ID payload [XAUTH]
Jul 19 13:05:45 net01.myserver.co pluto[1866]: packet from 79.235.11.89:500: received Vendor ID payload [Dead Peer Detection]
Jul 19 13:05:45 net01.myserver.co pluto[1866]: packet from 79.235.11.89:500: received Vendor ID payload [RFC 3947]
Jul 19 13:05:45 net01.myserver.co pluto[1866]: packet from 79.235.11.89:500: ignoring Vendor ID payload [draft-ietf-ipsec-nat-t-ike-03]
Jul 19 13:05:45 net01.myserver.co pluto[1866]: packet from 79.235.11.89:500: ignoring unknown Vendor ID payload [a2226fc364500f5634ff77db3b74f41b]
Jul 19 13:05:45 net01.myserver.co pluto[1866]: packet from 79.235.11.89:500: IKEv1 Aggressive Mode with PSK is vulnerable to dictionary attacks and is cracked on large scale by TLA's
Jul 19 13:05:45 net01.myserver.co pluto[1866]: "Site-to-Site"[1] 79.235.11.89 #11119: Aggressive mode peer ID is ID_FQDN: '@cloud.myipsecpartner.de'
Jul 19 13:05:45 net01.myserver.co pluto[1866]: "Site-to-Site"[1] 79.235.11.89 #11119: EXPECTATION FAILED at /builddir/build/BUILD/libreswan-3.15/programs/pluto/ikev1.c:2843: r != NULL
Jul 19 13:05:45 net01.myserver.co pluto[1866]: "Site-to-Site"[1] 79.235.11.89 #11119: no suitable connection for peer '@cloud.myipsecpartner.de'
Jul 19 13:05:45 net01.myserver.co pluto[1866]: "Site-to-Site"[1] 79.235.11.89 #11119: initial Aggressive Mode packet claiming to be from @myipsecpartner.no-ip.org on 79.235.11.89 but no connection has been authorized
Jul 19 13:05:45 net01.myserver.co pluto[1866]: "Site-to-Site"[1] 79.235.11.89 #11119: sending notification INVALID_ID_INFORMATION to 79.235.11.89:500
Jul 19 13:05:46 net01.myserver.co pluto[1866]: packet from 79.235.11.89:500: received Vendor ID payload [XAUTH]
Jul 19 13:05:46 net01.myserver.co pluto[1866]: packet from 79.235.11.89:500: received Vendor ID payload [Dead Peer Detection]
Jul 19 13:05:46 net01.myserver.co pluto[1866]: packet from 79.235.11.89:500: received Vendor ID payload [RFC 3947]
Jul 19 13:05:46 net01.myserver.co pluto[1866]: packet from 79.235.11.89:500: ignoring Vendor ID payload [draft-ietf-ipsec-nat-t-ike-03]
Jul 19 13:05:46 net01.myserver.co pluto[1866]: packet from 79.235.11.89:500: ignoring unknown Vendor ID payload [a2226fc364500f5634ff77db3b74f41b]
Jul 19 13:05:46 net01.myserver.co pluto[1866]: packet from 79.235.11.89:500: IKEv1 Aggressive Mode with PSK is vulnerable to dictionary attacks and is cracked on large scale by TLA's
Jul 19 13:05:46 net01.myserver.co pluto[1866]: "Site-to-Site"[1] 79.235.11.89 #11120: Aggressive mode peer ID is ID_FQDN: '@cloud.myipsecpartner.de'
Jul 19 13:05:46 net01.myserver.co pluto[1866]: "Site-to-Site"[1] 79.235.11.89 #11120: EXPECTATION FAILED at /builddir/build/BUILD/libreswan-3.15/programs/pluto/ikev1.c:2843: r != NULL
Jul 19 13:05:46 net01.myserver.co pluto[1866]: "Site-to-Site"[1] 79.235.11.89 #11120: no suitable connection for peer '@cloud.myipsecpartner.de'
Jul 19 13:05:46 net01.myserver.co pluto[1866]: "Site-to-Site"[1] 79.235.11.89 #11120: initial Aggressive Mode packet claiming to be from @myipsecpartner.no-ip.org on 79.235.11.89 but no connection has been authorized
Jul 19 13:05:46 net01.myserver.co pluto[1866]: "Site-to-Site"[1] 79.235.11.89 #11120: sending notification INVALID_ID_INFORMATION to 79.235.11.89:500
Jul 19 13:05:46 net01.myserver.co pluto[1866]: packet from 79.235.11.89:500: received Vendor ID payload [XAUTH]
Jul 19 13:05:46 net01.myserver.co pluto[1866]: packet from 79.235.11.89:500: received Vendor ID payload [Dead Peer Detection]
Jul 19 13:05:46 net01.myserver.co pluto[1866]: packet from 79.235.11.89:500: received Vendor ID payload [RFC 3947]

I´m wondering about these entries because “myipsecpartner.no-ip.org” is an old domain and does not appear in any configuration file.

Can you tell me which configurationfile openswan uses? /etc/ipsec.conf or /ipsec/ipsec.d/tunnels.conf

Stefano_Zamboni · July 23, 2017, 1:12pm

it’s normal and a good thing… you have to worry only if you’ll see your swap usage increase…

Linux is very efficient and it uses all the available ram…

dnutan · July 23, 2017, 1:50pm

in nethserver libreswan is using /etc/ipsec.d/tunnels.conf (handled through e-smith templates)

prostream · July 23, 2017, 1:55pm

So the conf file /etc/ipsec.conf and ipsec.secrets are not necessary? Deleting the files and restarting ipsec (service ipsec restart) gives:

 [root@net01 ~]# systemctl status ipsec.service
    ● ipsec.service - Internet Key Exchange (IKE) Protocol Daemon for IPsec
       Loaded: loaded (/usr/lib/systemd/system/ipsec.service; enabled; vendor preset: disabled)
       Active: failed (Result: start-limit) since Sun 2017-07-23 15:52:10 CEST; 40s ago
      Process: 1407 ExecStopPost=/usr/sbin/ipsec --stopnflog (code=exited, status=0/SUCCESS)
      Process: 1404 ExecStopPost=/sbin/ip xfrm state flush (code=exited, status=0/SUCCESS)
      Process: 1401 ExecStopPost=/sbin/ip xfrm policy flush (code=exited, status=0/SUCCESS)
      Process: 1389 ExecStop=/usr/libexec/ipsec/whack --shutdown (code=exited, status=0/SUCCESS)
      Process: 1423 ExecStartPre=/usr/libexec/ipsec/addconn --config /etc/ipsec.conf --checkconfig (code=exited, status=3)
     Main PID: 5006 (code=exited, status=0/SUCCESS)

    Jul 23 15:52:10 net01.myserver.co systemd[1]: ipsec.service: control process exited, code=exited status=3
    Jul 23 15:52:10 net01.myserver.co systemd[1]: Failed to start Internet Key Exchange (IKE) Protocol Daemon for IPsec.
    Jul 23 15:52:10 net01.myserver.co systemd[1]: Unit ipsec.service entered failed state.
    Jul 23 15:52:10 net01.myserver.co systemd[1]: ipsec.service failed.
    Jul 23 15:52:10 net01.myserver.co systemd[1]: ipsec.service holdoff time over, scheduling restart.
    Jul 23 15:52:10 net01.myserver.co systemd[1]: start request repeated too quickly for ipsec.service
    Jul 23 15:52:10 net01.myserver.co systemd[1]: Failed to start Internet Key Exchange (IKE) Protocol Daemon for IPsec.
    Jul 23 15:52:10 net01.myserver.co systemd[1]: Unit ipsec.service entered failed state.
    Jul 23 15:52:10 net01.myserver.co systemd[1]: ipsec.service failed.

dnutan · July 23, 2017, 2:04pm

It’s necessary but NethServer stores the config under /etc/ipsec.d/, leaving the default conf file untouched
(NethServer settings are stored in the vpn database)

prostream · July 23, 2017, 2:38pm

Ah okay. Does anybody know how to delete the old domain from nethserver vpn database?

dnutan · July 23, 2017, 3:04pm

Do you see the old settings here?

db vpn show    # (don't share the psk)

prostream · July 23, 2017, 3:31pm

yes i can see the old settings

dnutan · July 23, 2017, 3:39pm

If it doesn’t show up on the UI, you can delete it with:

db vpn delete yourtunnelname
signal-event nethserver-ipsec-tunnels-update

Make sure to choose the tunnel you want to delete.
If you don’t want to delete the tunnel but a specific property, the manual and the wiki have more information about the databases.

prostream · July 23, 2017, 4:45pm

Okay i was wrong. I can´t see the old setting with “db vpn show”… I was confused by a command before… Is there any other possible reason why it shows the old config in “journalctl -u ipsec.service” ?

EddieA · July 23, 2017, 7:29pm

I think back to basics here. Find out what is using the swap space and not just try and guess. Here is a very simple script:

for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r

Or this question on Stackoverflow has a few scripts to try.

Or another page with various methods to investigate swap space.

Cheers.

prostream · July 25, 2017, 11:27am

Thank you! I´ll try this script when my swap RAM is getting in usage again.

prostream · August 4, 2017, 11:06am

I realized that my Swap RAM gets higher and i runned your script. It shows this:

[root@net01 ~]# for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r
xfs-reclaim/sda
xfs-reclaim/dm-
xfs_mru_cache
xfs-log/sda1
xfs-log/dm-0
xfs-eofblocks/s
xfs-eofblocks/d
xfs-data/sda1
xfs-data/dm-0
xfs-conv/sda1
xfs-conv/dm-0
xfs-cil/sda1
xfs-cil/dm-0
xfs-buf/sda1
xfs-buf/dm-0
xfsalloc
xfsaild/sda1
xfsaild/dm-0
writeback
watchdog/3
watchdog/2
watchdog/1
watchdog/0
vballoon
tuned 0 kB
ttm_swap
tlsmgr 0 kB
systemd-udevd 0 kB
systemd-logind 0 kB
systemd-journal 0 kB
systemd 0 kB
sshd 0 kB
sshd 0 kB
sort 0 kB
smwingsd 0 kB
smartd 0 kB
scsi_tmf_1
scsi_tmf_0
scsi_eh_1
scsi_eh_0
rsyslogd 0 kB
rpciod
rcu_sched
rcu_bh
qmgr 0 kB
pppoe-connect 0 kB
pppd 0 kB
polkitd 0 kB
_pluto_adns 0 kB
pluto 0 kB
pickup 0 kB
openvpn 0 kB
netns
migration/3
migration/2
migration/1
migration/0
md
master 0 kB
lvmetad 0 kB
kworker/u8:1
kworker/u8:0
kworker/3:2H
kworker/3:2
kworker/3:1H
kworker/3:1
kworker/2:2
kworker/2:1H
kworker/2:1
kworker/2:0H
kworker/2:0
kworker/1:3
kworker/1:1H
kworker/1:1
kworker/1:0H
kworker/1:0
kworker/0:1H
kworker/0:1
kworker/0:0H
kworker/0:0
kthrotld
kthreadd
kswapd0
ksoftirqd/3
ksoftirqd/2
ksoftirqd/1
ksoftirqd/0
ksmd
kpsmoused
kmpath_rdacd
kintegrityd
khungtaskd
khugepaged
kdmflush
kdmflush
kdevtmpfs
kblockd
kauditd
irqbalance 0 kB
ipv6_addrconf
httpd 0 kB
httpd 0 kB
httpd 0 kB
httpd 0 kB
httpd 0 kB
httpd 0 kB
httpd 0 kB
httpd 0 kB
httpd 0 kB
httpd 0 kB
httpd 0 kB
httpd 0 kB
httpd 0 kB
httpd 0 kB
httpd 0 kB
httpd 0 kB
httpd 0 kB
httpd 0 kB
httpd 0 kB
httpd 0 kB
httpd 0 kB
httpd 0 kB
gssproxy 0 kB
fsnotify_mark
fail2ban-server 0 kB
dnsmasq 0 kB
deferwq
dbus-daemon 0 kB
crypto
crond 0 kB
cifsiod
chronyd 0 kB
bioset
bioset
bioset
bash 0 kB
bash 0 kB
bandwidthd 0 kB
bandwidthd
awk 0 kB
auditd 0 kB
ata_sff
agetty 0 kB
addconn

My RAM at the moment:

Stefano_Zamboni · August 4, 2017, 6:23pm

Your server is fine
It isn’t using any swap, it’s a good thing
Don’t worry