Hello friends!
I have a problem with my system load, as soon as I activate the SPAM filter under the mail server, my system load keeps increasing until the system is almost paralyzed.
After a reboot from it runs again up to 7 days.
As soon as I turn off the spam filter, the system load is completely normal.
Greetings and a Merry Christmas!
Gerald
Could you please provide a sort of system details about your installation?
And have a look at your messages.log please.
Ok, sure.
So as basis serves an Intel Celeron CPU J1900 with 8GB RAM us as SSD and a conventional hard drive as RAID for the directory / var.
2x NIC (green and red)
Software:
NethServer 7.4.1798 (current)
The content of the log is coming soon …
And here is an excerpt of the message.log, about 5 minutes:
What does he always do with a SOGO User ??
Dec 21 19:47:01 openzwo systemd: Stopping User Slice of apache.
Dec 21 19:47:02 openzwo systemd: Removed slice User Slice of sogo.
Dec 21 19:47:02 openzwo systemd: Stopping User Slice of sogo.
Dec 21 19:47:04 openzwo systemd-logind: Removed session c427.
Dec 21 19:47:04 openzwo systemd: Removed slice User Slice of stb@nandlnet.de.
Dec 21 19:47:04 openzwo systemd: Stopping User Slice of stb@nandlnet.de.
Dec 21 19:47:21 openzwo smbd[18126]: [2017/12/21 19:47:21.751080, 0] ../lib/param/loadparm.c:782(lpcfg_map_parameter)
Dec 21 19:47:21 openzwo smbd[18126]: Unknown parameter encountered: "share modes"
Dec 21 19:47:21 openzwo smbd[18126]: [2017/12/21 19:47:21.751230, 0] ../lib/param/loadparm.c:1791(lpcfg_do_service_parameter)
Dec 21 19:47:21 openzwo smbd[18126]: Ignoring unknown parameter "share modes"
Dec 21 19:47:22 openzwo systemd: Created slice User Slice of stb@nandlnet.de.
Dec 21 19:47:22 openzwo systemd: Starting User Slice of stb@nandlnet.de.
Dec 21 19:47:22 openzwo systemd-logind: New session c428 of user stb@nandlnet.de.
Dec 21 19:47:22 openzwo systemd: Started Session c428 of user stb@nandlnet.de.
Dec 21 19:47:22 openzwo systemd: Starting Session c428 of user stb@nandlnet.de.
Dec 21 19:47:41 openzwo clamd: SelfCheck: Database status OK.
Dec 21 19:47:41 openzwo clamd[10077]: SelfCheck: Database status OK.
Dec 21 19:48:01 openzwo systemd: Created slice User Slice of apache.
Dec 21 19:48:01 openzwo systemd: Starting User Slice of apache.
Dec 21 19:48:01 openzwo systemd: Started Session 18018 of user apache.
Dec 21 19:48:01 openzwo systemd: Starting Session 18018 of user apache.
Dec 21 19:48:01 openzwo systemd: Created slice User Slice of sogo.
Dec 21 19:48:01 openzwo systemd: Starting User Slice of sogo.
Dec 21 19:48:01 openzwo systemd: Started Session 18019 of user sogo.
Dec 21 19:48:01 openzwo systemd: Starting Session 18019 of user sogo.
Dec 21 19:48:01 openzwo systemd: Removed slice User Slice of apache.
Dec 21 19:48:01 openzwo systemd: Stopping User Slice of apache.
Dec 21 19:48:01 openzwo systemd: Removed slice User Slice of sogo.
Dec 21 19:48:01 openzwo systemd: Stopping User Slice of sogo.
Dec 21 19:48:47 openzwo dnsmasq-dhcp[9432]: DHCPREQUEST(br0) 192.168.200.236 c4:57:6e:78:14:18
Dec 21 19:48:47 openzwo dnsmasq-dhcp[9432]: DHCPACK(br0) 192.168.200.236 c4:57:6e:78:14:18
Dec 21 19:49:02 openzwo systemd: Created slice User Slice of apache.
Dec 21 19:49:02 openzwo systemd: Starting User Slice of apache.
Dec 21 19:49:02 openzwo systemd: Started Session 18020 of user apache.
Dec 21 19:49:02 openzwo systemd: Starting Session 18020 of user apache.
Dec 21 19:49:02 openzwo systemd: Created slice User Slice of sogo.
Dec 21 19:49:02 openzwo systemd: Starting User Slice of sogo.
Dec 21 19:49:02 openzwo systemd: Started Session 18021 of user sogo.
Dec 21 19:49:02 openzwo systemd: Starting Session 18021 of user sogo.
Dec 21 19:49:02 openzwo systemd: Removed slice User Slice of apache.
Dec 21 19:49:02 openzwo systemd: Stopping User Slice of apache.
Dec 21 19:49:02 openzwo dnsmasq-dhcp[9432]: DHCPREQUEST(br0) 192.168.200.19 90:cd:b6:8d:49:30
Dec 21 19:49:02 openzwo dnsmasq-dhcp[9432]: DHCPACK(br0) 192.168.200.19 90:cd:b6:8d:49:30 dcp
Dec 21 19:49:02 openzwo systemd: Removed slice User Slice of sogo.
Dec 21 19:49:02 openzwo systemd: Stopping User Slice of sogo.
Dec 21 19:50:01 openzwo systemd: Created slice User Slice of apache.
Dec 21 19:50:01 openzwo systemd: Starting User Slice of apache.
Dec 21 19:50:01 openzwo systemd: Started Session 18022 of user apache.
Dec 21 19:50:01 openzwo systemd: Starting Session 18022 of user apache.
Dec 21 19:50:01 openzwo systemd: Created slice User Slice of sogo.
Dec 21 19:50:01 openzwo systemd: Starting User Slice of sogo.
Dec 21 19:50:01 openzwo systemd: Started Session 18023 of user sogo.
Dec 21 19:50:01 openzwo systemd: Starting Session 18023 of user sogo.
Dec 21 19:50:01 openzwo systemd: Removed slice User Slice of apache.
Dec 21 19:50:01 openzwo systemd: Stopping User Slice of apache.
Dec 21 19:50:02 openzwo systemd: Removed slice User Slice of sogo.
Dec 21 19:50:02 openzwo systemd: Stopping User Slice of sogo.
Dec 21 19:50:31 openzwo systemd: Removed slice User Slice of root.
Dec 21 19:50:31 openzwo systemd: Stopping User Slice of root.
Dec 21 19:51:01 openzwo systemd: Created slice User Slice of apache.
Dec 21 19:51:01 openzwo systemd: Starting User Slice of apache.
Dec 21 19:51:01 openzwo systemd: Started Session 18024 of user apache.
Dec 21 19:51:01 openzwo systemd: Starting Session 18024 of user apache.
Dec 21 19:51:01 openzwo systemd: Created slice User Slice of sogo.
Dec 21 19:51:01 openzwo systemd: Starting User Slice of sogo.
Dec 21 19:51:01 openzwo systemd: Started Session 18025 of user sogo.
Dec 21 19:51:01 openzwo systemd: Starting Session 18025 of user sogo.
Dec 21 19:51:01 openzwo systemd: Removed slice User Slice of apache.
Dec 21 19:51:01 openzwo systemd: Stopping User Slice of apache.
Dec 21 19:51:02 openzwo systemd: Removed slice User Slice of sogo.
Dec 21 19:51:02 openzwo systemd: Stopping User Slice of sogo.
Dec 21 19:51:35 openzwo httpd: [NOTICE] Nethgui\Authorization\User: user `root` authenticated
Dec 21 19:52:01 openzwo systemd: Created slice User Slice of apache.
Dec 21 19:52:01 openzwo systemd: Starting User Slice of apache.
Dec 21 19:52:01 openzwo systemd: Started Session 18026 of user apache.
Dec 21 19:52:01 openzwo systemd: Starting Session 18026 of user apache.
Dec 21 19:52:01 openzwo systemd: Created slice User Slice of sogo.
Dec 21 19:52:01 openzwo systemd: Starting User Slice of sogo.
Dec 21 19:52:01 openzwo systemd: Started Session 18027 of user sogo.
Dec 21 19:52:01 openzwo systemd: Starting Session 18027 of user sogo.
Dec 21 19:52:01 openzwo systemd: Removed slice User Slice of apache.
Dec 21 19:52:01 openzwo systemd: Stopping User Slice of apache.
Dec 21 19:52:02 openzwo systemd: Removed slice User Slice of sogo.
Dec 21 19:52:02 openzwo systemd: Stopping User Slice of sogo.Check the sogo wiki page, you have a trick to hide the systemd log noise about sogo. It is normal log traces and not relevant to your problem
You could also have a look here:
I don’t see anything about this behavior at your log. Perhaps somebody else has another idea.
You may try “top” on command line to see which processes are paralyzing your system.
Please provide the part of /var/log/messages when activating the spam filter, so something like this should be in the logfile:
Dec 22 12:40:44 server esmith::event[3268]: Event: nethserver-mail-filter-save SUCCESS
So, have now reduced the log messages from SOGO to a minimum - as described in the wiki!
@stephdl 
Have the spam filter started again and must now wait first, 1 - 7 days until the system slows down again.
As I said, the system load increases slowly, as soon as you turn off the spam filter, it drops back to a normal value.
@mrmarkuz
yes the entry was there before at every start and is now back, so it must be so 
greetings
Gerald
check log when the server become unresponsive, or a bit before 
Hello,
After about 24 hours of operation, the system load increases again strongly.
Here is the excerpt from the running processes.
It runs again an Apache process with nasty performance data, 394% CPU power, etc …
When I turn off the system continues to run with normal values … it also runs normally again when I turn off the spam filter.
Tasks: 538 total, 2 running, 457 sleeping, 0 stopped, 79 zombie
%Cpu(s): 99.1 us, 0.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem : 7902388 total, 133332 free, 4298132 used, 3470924 buff/cache
KiB Swap: 6291452 total, 6285016 free, 6436 used. 3123856 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29959 apache 20 0 424460 40120 2328 S 394.1 0.5 3710:09 j
3154 root 20 0 168408 2784 1592 R 1.0 0.0 0:00.42 top
28338 stb@nan+ 20 0 464224 9052 6320 S 0.7 0.1 0:48.81 smbd
1622 root 20 0 767244 1460 780 S 0.3 0.0 0:06.31 c-icap
1727 sogo 20 0 364908 17112 5264 S 0.3 0.2 2:08.48 sogod
2124 mysql 20 0 2690952 158652 9148 S 0.3 2.0 1:50.96 mysqld
2500 apache 20 0 36944 4356 1416 S 0.3 0.1 0:00.09 /usr/sbin/httpd
3132 root 20 0 154804 6000 4644 S 0.3 0.1 0:00.20 sshd
5261 apache 20 0 36944 4452 1512 S 0.3 0.1 0:03.58 /usr/sbin/httpd
17536 apache 20 0 36944 4452 1512 S 0.3 0.1 0:01.98 /usr/sbin/httpd
I wish you all a merry Christmas!
And that all in peace with your loved ones can enjoy the time!
Gerald
Do you know something about the j process killing your CPU? You may try to identify it with “ps -aux | grep PID”…
mmmhhh, so that’s not directly meaningful to me:
[root@openzwo ~]# ps -aux | grep PID3702
root 26702 0.0 0.0 112660 976 pts/0 S+ 11:46 0:00 grep --color=auto PID3702
The Apache process is back at 396% CPU utilization …
top - 11:49:54 up 1 day, 2:17, 1 user, load average: 4.06, 4.10, 4.13
Tasks: 428 total, 2 running, 420 sleeping, 0 stopped, 6 zombie
%Cpu(s): 99.6 us, 0.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.2 si, 0.0 st
KiB Mem : 7902388 total, 198068 free, 4015188 used, 3689132 buff/cache
KiB Swap: 6291452 total, 6273300 free, 18152 used. 3411376 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3702 apache 20 0 424460 39960 2332 S 396.1 0.5 3710:05 j
26925 root 20 0 168392 2660 1592 R 1.0 0.0 0:00.14 top
774 root 20 0 276408 11356 4328 S 0.3 0.1 0:22.24 rsyslogd
2755 apache 20 0 36944 4456 1512 S 0.3 0.1 0:14.77 /usr/sbin/httpd
11656 apache 20 0 36944 4452 1512 S 0.3 0.1 0:14.65 /usr/sbin/httpd
14114 apache 20 0 36944 4456 1512 S 0.3 0.1 0:12.84 /usr/sbin/httpd
14511 apache 20 0 36944 4460 1512 S 0.3 0.1 0:13.49 /usr/sbin/httpd
19945 squid 20 0 183204 75340 7976 S 0.3 1.0 6:17.90 squidSorry, wrong params, please try
ps -fp PID of j process
Please show us the output of
lsof -p 3702
substitute 3702 with the pid of the “j” process.
Good Morning
here the current data of the day:
Tasks: 329 total, 2 running, 326 sleeping, 0 stopped, 1 zombie
%Cpu(s): 97.4 us, 0.4 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 2.2 si, 0.0 st
KiB Mem : 7902388 total, 300096 free, 3640940 used, 3961352 buff/cache
KiB Swap: 6291452 total, 6241704 free, 49748 used. 3754092 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31834 apache 20 0 424460 11260 1844 S 395.7 0.1 579:16.96 j
2672 root 20 0 555768 13644 4496 S 1.0 0.2 0:57.06 samba
2670 root 20 0 552036 17920 6600 S 0.7 0.2 1:33.48 samba
11422 root 20 0 168276 2556 1588 R 0.7 0.0 0:00.07 top
19945 squid 20 0 190016 91084 7976 S 0.7 1.2 7:58.21 squid
1411 redis 20 0 142912 5788 1432 S 0.3 0.1 3:31.15 redis-server
1680 sogo 20 0 364908 17168 5284 S 0.3 0.2 3:16.85 sogod
2169 mysql 20 0 2685120 140856 10240 S 0.3 1.8 3:16.09 mysqld
[root@openzwo ~]# ps -fp 31834
UID PID PPID C STIME TTY TIME CMD
apache 31834 1 99 07:35 ? 09:43:02 -bash
[root@openzwo ~]#
[root@openzwo ~]# lsof -p 31834
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
j 31834 apache cwd DIR 8,1 161 26240475 /tmp
j 31834 apache rtd DIR 8,1 4096 64 /
j 31834 apache txt REG 8,1 3876568 26246876 /tmp/j
j 31834 apache mem REG 8,1 111080 26293471 /usr/lib64/libresolv-2.17.so
j 31834 apache mem REG 8,1 27776 25864019 /usr/lib64/libnss_dns-2.17.so
j 31834 apache mem REG 8,1 164112 25752629 /usr/lib64/ld-2.17.so
j 31834 apache mem REG 8,1 2127336 25166636 /usr/lib64/libc-2.17.so
j 31834 apache mem REG 8,1 62184 26293453 /usr/lib64/libnss_files-2.17.so
j 31834 apache 0r CHR 1,3 0t0 1028 /dev/null
j 31834 apache 1w CHR 1,3 0t0 1028 /dev/null
j 31834 apache 2w CHR 1,3 0t0 1028 /dev/null
j 31834 apache 3u a_inode 0,9 0 5901 [eventfd]
j 31834 apache 4u a_inode 0,9 0 5901 [eventpoll]
j 31834 apache 5u a_inode 0,9 0 5901 [timerfd]
j 31834 apache 6r FIFO 0,8 0t0 3514077 pipe
j 31834 apache 7w FIFO 0,8 0t0 3514077 pipe
j 31834 apache 8u IPv4 3514165 0t0 TCP openzwo.fritz.box:53296->static.12.31.201.138.clients.your-server.de:dec-notes (ESTABLISHED)
[root@openzwo ~]#
So the IP address refers to a Hetzner server, from the port is assigned a speed test …
Many greetings
I think that you will find a file named j in /tmp.
I suggest that you analyze it at virustotal.com.
Could you please send it to us, too?
Thank you,
had searched for the file this morning, but in the tmp directory was nothing like that.
I have now blocked the port on nethserver and additionally in the Fritzbox.
In addition, I have deleted all the content from the tmp directory.
Let’s see what happens.
But how can something like that happen, that something like that fits into the system?
And why is the system again “almost” nrmal when the SPAM filter is switched off …
greetings
Gerald
I don’t think that system load is linked to the spam filter. The load is generated by the j process, which is unknown and suspicious.
We will need to analyze the system. Which software are you running on NethServer? Any app like wordpress? Or other php software? Custom made?
There are a total of three services that are not original.
Externally Wordpress runs once
Internally run two additional services as a virtual server:
phpmyadmin
and a trial platform with admidio (this is a club software) “admidio.org”