Virus/Malware and Systemload

mail

(Gerald) #1

Hello friends!

I have a problem with my system load, as soon as I activate the SPAM filter under the mail server, my system load keeps increasing until the system is almost paralyzed.

After a reboot from it runs again up to 7 days.

As soon as I turn off the spam filter, the system load is completely normal.

Greetings and a Merry Christmas!

Gerald


(Michael Kicks) #2

Could you please provide a sort of system details about your installation?


(Michael Träumner) #3

And have a look at your messages.log please.


(Gerald) #4

Ok, sure.

So as basis serves an Intel Celeron CPU J1900 with 8GB RAM us as SSD and a conventional hard drive as RAID for the directory / var.
2x NIC (green and red)

Software:

NethServer 7.4.1798 (current)

  • DHCP
  • DNS
  • SAMBA DC
  • Email
  • SOGO

The content of the log is coming soon …


(Gerald) #5

And here is an excerpt of the message.log, about 5 minutes:

What does he always do with a SOGO User ??

Dec 21 19:47:01 openzwo systemd: Stopping User Slice of apache.
Dec 21 19:47:02 openzwo systemd: Removed slice User Slice of sogo.
Dec 21 19:47:02 openzwo systemd: Stopping User Slice of sogo.
Dec 21 19:47:04 openzwo systemd-logind: Removed session c427.
Dec 21 19:47:04 openzwo systemd: Removed slice User Slice of stb@nandlnet.de.
Dec 21 19:47:04 openzwo systemd: Stopping User Slice of stb@nandlnet.de.
Dec 21 19:47:21 openzwo smbd[18126]: [2017/12/21 19:47:21.751080,  0] ../lib/param/loadparm.c:782(lpcfg_map_parameter)
Dec 21 19:47:21 openzwo smbd[18126]:  Unknown parameter encountered: "share modes"
Dec 21 19:47:21 openzwo smbd[18126]: [2017/12/21 19:47:21.751230,  0] ../lib/param/loadparm.c:1791(lpcfg_do_service_parameter)
Dec 21 19:47:21 openzwo smbd[18126]:  Ignoring unknown parameter "share modes"
Dec 21 19:47:22 openzwo systemd: Created slice User Slice of stb@nandlnet.de.
Dec 21 19:47:22 openzwo systemd: Starting User Slice of stb@nandlnet.de.
Dec 21 19:47:22 openzwo systemd-logind: New session c428 of user stb@nandlnet.de.
Dec 21 19:47:22 openzwo systemd: Started Session c428 of user stb@nandlnet.de.
Dec 21 19:47:22 openzwo systemd: Starting Session c428 of user stb@nandlnet.de.
Dec 21 19:47:41 openzwo clamd: SelfCheck: Database status OK.
Dec 21 19:47:41 openzwo clamd[10077]: SelfCheck: Database status OK.
Dec 21 19:48:01 openzwo systemd: Created slice User Slice of apache.
Dec 21 19:48:01 openzwo systemd: Starting User Slice of apache.
Dec 21 19:48:01 openzwo systemd: Started Session 18018 of user apache.
Dec 21 19:48:01 openzwo systemd: Starting Session 18018 of user apache.
Dec 21 19:48:01 openzwo systemd: Created slice User Slice of sogo.
Dec 21 19:48:01 openzwo systemd: Starting User Slice of sogo.
Dec 21 19:48:01 openzwo systemd: Started Session 18019 of user sogo.
Dec 21 19:48:01 openzwo systemd: Starting Session 18019 of user sogo.
Dec 21 19:48:01 openzwo systemd: Removed slice User Slice of apache.
Dec 21 19:48:01 openzwo systemd: Stopping User Slice of apache.
Dec 21 19:48:01 openzwo systemd: Removed slice User Slice of sogo.
Dec 21 19:48:01 openzwo systemd: Stopping User Slice of sogo.
Dec 21 19:48:47 openzwo dnsmasq-dhcp[9432]: DHCPREQUEST(br0) 192.168.200.236 c4:57:6e:78:14:18
Dec 21 19:48:47 openzwo dnsmasq-dhcp[9432]: DHCPACK(br0) 192.168.200.236 c4:57:6e:78:14:18
Dec 21 19:49:02 openzwo systemd: Created slice User Slice of apache.
Dec 21 19:49:02 openzwo systemd: Starting User Slice of apache.
Dec 21 19:49:02 openzwo systemd: Started Session 18020 of user apache.
Dec 21 19:49:02 openzwo systemd: Starting Session 18020 of user apache.
Dec 21 19:49:02 openzwo systemd: Created slice User Slice of sogo.
Dec 21 19:49:02 openzwo systemd: Starting User Slice of sogo.
Dec 21 19:49:02 openzwo systemd: Started Session 18021 of user sogo.
Dec 21 19:49:02 openzwo systemd: Starting Session 18021 of user sogo.
Dec 21 19:49:02 openzwo systemd: Removed slice User Slice of apache.
Dec 21 19:49:02 openzwo systemd: Stopping User Slice of apache.
Dec 21 19:49:02 openzwo dnsmasq-dhcp[9432]: DHCPREQUEST(br0) 192.168.200.19 90:cd:b6:8d:49:30
Dec 21 19:49:02 openzwo dnsmasq-dhcp[9432]: DHCPACK(br0) 192.168.200.19 90:cd:b6:8d:49:30 dcp
Dec 21 19:49:02 openzwo systemd: Removed slice User Slice of sogo.
Dec 21 19:49:02 openzwo systemd: Stopping User Slice of sogo.
Dec 21 19:50:01 openzwo systemd: Created slice User Slice of apache.
Dec 21 19:50:01 openzwo systemd: Starting User Slice of apache.
Dec 21 19:50:01 openzwo systemd: Started Session 18022 of user apache.
Dec 21 19:50:01 openzwo systemd: Starting Session 18022 of user apache.
Dec 21 19:50:01 openzwo systemd: Created slice User Slice of sogo.
Dec 21 19:50:01 openzwo systemd: Starting User Slice of sogo.
Dec 21 19:50:01 openzwo systemd: Started Session 18023 of user sogo.
Dec 21 19:50:01 openzwo systemd: Starting Session 18023 of user sogo.
Dec 21 19:50:01 openzwo systemd: Removed slice User Slice of apache.
Dec 21 19:50:01 openzwo systemd: Stopping User Slice of apache.
Dec 21 19:50:02 openzwo systemd: Removed slice User Slice of sogo.
Dec 21 19:50:02 openzwo systemd: Stopping User Slice of sogo.
Dec 21 19:50:31 openzwo systemd: Removed slice User Slice of root.
Dec 21 19:50:31 openzwo systemd: Stopping User Slice of root.
Dec 21 19:51:01 openzwo systemd: Created slice User Slice of apache.
Dec 21 19:51:01 openzwo systemd: Starting User Slice of apache.
Dec 21 19:51:01 openzwo systemd: Started Session 18024 of user apache.
Dec 21 19:51:01 openzwo systemd: Starting Session 18024 of user apache.
Dec 21 19:51:01 openzwo systemd: Created slice User Slice of sogo.
Dec 21 19:51:01 openzwo systemd: Starting User Slice of sogo.
Dec 21 19:51:01 openzwo systemd: Started Session 18025 of user sogo.
Dec 21 19:51:01 openzwo systemd: Starting Session 18025 of user sogo.
Dec 21 19:51:01 openzwo systemd: Removed slice User Slice of apache.
Dec 21 19:51:01 openzwo systemd: Stopping User Slice of apache.
Dec 21 19:51:02 openzwo systemd: Removed slice User Slice of sogo.
Dec 21 19:51:02 openzwo systemd: Stopping User Slice of sogo.
Dec 21 19:51:35 openzwo httpd: [NOTICE] Nethgui\Authorization\User: user `root` authenticated
Dec 21 19:52:01 openzwo systemd: Created slice User Slice of apache.
Dec 21 19:52:01 openzwo systemd: Starting User Slice of apache.
Dec 21 19:52:01 openzwo systemd: Started Session 18026 of user apache.
Dec 21 19:52:01 openzwo systemd: Starting Session 18026 of user apache.
Dec 21 19:52:01 openzwo systemd: Created slice User Slice of sogo.
Dec 21 19:52:01 openzwo systemd: Starting User Slice of sogo.
Dec 21 19:52:01 openzwo systemd: Started Session 18027 of user sogo.
Dec 21 19:52:01 openzwo systemd: Starting Session 18027 of user sogo.
Dec 21 19:52:01 openzwo systemd: Removed slice User Slice of apache.
Dec 21 19:52:01 openzwo systemd: Stopping User Slice of apache.
Dec 21 19:52:02 openzwo systemd: Removed slice User Slice of sogo.
Dec 21 19:52:02 openzwo systemd: Stopping User Slice of sogo.

(Stéphane de Labrusse) #6

Check the sogo wiki page, you have a trick to hide the systemd log noise about sogo. It is normal log traces and not relevant to your problem


(Michael Träumner) #7

You could also have a look here:

Sogo floods messages.log

I don’t see anything about this behavior at your log. Perhaps somebody else has another idea.


(Markus Neuberger) #8

You may try “top” on command line to see which processes are paralyzing your system.

Please provide the part of /var/log/messages when activating the spam filter, so something like this should be in the logfile:

Dec 22 12:40:44 server esmith::event[3268]: Event: nethserver-mail-filter-save SUCCESS


(Gerald) #9

So, have now reduced the log messages from SOGO to a minimum - as described in the wiki!
@stephdl :slight_smile:

Have the spam filter started again and must now wait first, 1 - 7 days until the system slows down again.

As I said, the system load increases slowly, as soon as you turn off the spam filter, it drops back to a normal value.

@mrmarkuz
yes the entry was there before at every start and is now back, so it must be so :wink:

greetings
Gerald


(Stéphane de Labrusse) #10

check log when the server become unresponsive, or a bit before :stuck_out_tongue:


(Gerald) #11

Hello,

After about 24 hours of operation, the system load increases again strongly.

Here is the excerpt from the running processes.

It runs again an Apache process with nasty performance data, 394% CPU power, etc …
When I turn off the system continues to run with normal values … it also runs normally again when I turn off the spam filter.

Tasks: 538 total,   2 running, 457 sleeping,   0 stopped,  79 zombie
%Cpu(s): 99.1 us,  0.5 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.4 si,  0.0 st
KiB Mem :  7902388 total,   133332 free,  4298132 used,  3470924 buff/cache
KiB Swap:  6291452 total,  6285016 free,     6436 used.  3123856 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                           
29959 apache    20   0  424460  40120   2328 S 394.1  0.5   3710:09 j                                                                                                                 
 3154 root      20   0  168408   2784   1592 R   1.0  0.0   0:00.42 top                                                                                                               
28338 stb@nan+  20   0  464224   9052   6320 S   0.7  0.1   0:48.81 smbd                                                                                                              
 1622 root      20   0  767244   1460    780 S   0.3  0.0   0:06.31 c-icap                                                                                                            
 1727 sogo      20   0  364908  17112   5264 S   0.3  0.2   2:08.48 sogod                                                                                                             
 2124 mysql     20   0 2690952 158652   9148 S   0.3  2.0   1:50.96 mysqld                                                                                                            
 2500 apache    20   0   36944   4356   1416 S   0.3  0.1   0:00.09 /usr/sbin/httpd                                                                                                   
 3132 root      20   0  154804   6000   4644 S   0.3  0.1   0:00.20 sshd                                                                                                              
 5261 apache    20   0   36944   4452   1512 S   0.3  0.1   0:03.58 /usr/sbin/httpd                                                                                                   
17536 apache    20   0   36944   4452   1512 S   0.3  0.1   0:01.98 /usr/sbin/httpd    

I wish you all a merry Christmas!
And that all in peace with your loved ones can enjoy the time!

Gerald


(Markus Neuberger) #12

Do you know something about the j process killing your CPU? You may try to identify it with “ps -aux | grep PID”…


(Gerald) #13

mmmhhh, so that’s not directly meaningful to me:
[root@openzwo ~]# ps -aux | grep PID3702
root 26702 0.0 0.0 112660 976 pts/0 S+ 11:46 0:00 grep --color=auto PID3702

The Apache process is back at 396% CPU utilization …

top - 11:49:54 up 1 day,  2:17,  1 user,  load average: 4.06, 4.10, 4.13
Tasks: 428 total,   2 running, 420 sleeping,   0 stopped,   6 zombie
%Cpu(s): 99.6 us,  0.2 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem :  7902388 total,   198068 free,  4015188 used,  3689132 buff/cache
KiB Swap:  6291452 total,  6273300 free,    18152 used.  3411376 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                          
 3702 apache    20   0  424460  39960   2332 S 396.1  0.5   3710:05 j                                                                                                                
26925 root      20   0  168392   2660   1592 R   1.0  0.0   0:00.14 top                                                                                                              
  774 root      20   0  276408  11356   4328 S   0.3  0.1   0:22.24 rsyslogd                                                                                                         
 2755 apache    20   0   36944   4456   1512 S   0.3  0.1   0:14.77 /usr/sbin/httpd                                                                                                  
11656 apache    20   0   36944   4452   1512 S   0.3  0.1   0:14.65 /usr/sbin/httpd                                                                                                  
14114 apache    20   0   36944   4456   1512 S   0.3  0.1   0:12.84 /usr/sbin/httpd                                                                                                  
14511 apache    20   0   36944   4460   1512 S   0.3  0.1   0:13.49 /usr/sbin/httpd                                                                                                  
19945 squid     20   0  183204  75340   7976 S   0.3  1.0   6:17.90 squid

(Markus Neuberger) #14

Sorry, wrong params, please try

ps -fp PID of j process


(Filippo Carletti) #15

Please show us the output of
lsof -p 3702
substitute 3702 with the pid of the “j” process.


(Gerald) #16

Good Morning

here the current data of the day:

Tasks: 329 total,   2 running, 326 sleeping,   0 stopped,   1 zombie
%Cpu(s): 97.4 us,  0.4 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  2.2 si,  0.0 st
KiB Mem :  7902388 total,   300096 free,  3640940 used,  3961352 buff/cache
KiB Swap:  6291452 total,  6241704 free,    49748 used.  3754092 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                           
31834 apache    20   0  424460  11260   1844 S 395.7  0.1 579:16.96 j                                                                                                                 
 2672 root      20   0  555768  13644   4496 S   1.0  0.2   0:57.06 samba                                                                                                             
 2670 root      20   0  552036  17920   6600 S   0.7  0.2   1:33.48 samba                                                                                                             
11422 root      20   0  168276   2556   1588 R   0.7  0.0   0:00.07 top                                                                                                               
19945 squid     20   0  190016  91084   7976 S   0.7  1.2   7:58.21 squid                                                                                                             
 1411 redis     20   0  142912   5788   1432 S   0.3  0.1   3:31.15 redis-server                                                                                                      
 1680 sogo      20   0  364908  17168   5284 S   0.3  0.2   3:16.85 sogod                                                                                                             
 2169 mysql     20   0 2685120 140856  10240 S   0.3  1.8   3:16.09 mysqld      



[root@openzwo ~]# ps -fp 31834
UID        PID  PPID  C STIME TTY          TIME CMD
apache   31834     1 99 07:35 ?        09:43:02 -bash                                                                                                                                
[root@openzwo ~]# 


[root@openzwo ~]# lsof -p 31834
COMMAND   PID   USER   FD      TYPE  DEVICE SIZE/OFF     NODE NAME
j       31834 apache  cwd       DIR     8,1      161 26240475 /tmp
j       31834 apache  rtd       DIR     8,1     4096       64 /
j       31834 apache  txt       REG     8,1  3876568 26246876 /tmp/j
j       31834 apache  mem       REG     8,1   111080 26293471 /usr/lib64/libresolv-2.17.so
j       31834 apache  mem       REG     8,1    27776 25864019 /usr/lib64/libnss_dns-2.17.so
j       31834 apache  mem       REG     8,1   164112 25752629 /usr/lib64/ld-2.17.so
j       31834 apache  mem       REG     8,1  2127336 25166636 /usr/lib64/libc-2.17.so
j       31834 apache  mem       REG     8,1    62184 26293453 /usr/lib64/libnss_files-2.17.so
j       31834 apache    0r      CHR     1,3      0t0     1028 /dev/null
j       31834 apache    1w      CHR     1,3      0t0     1028 /dev/null
j       31834 apache    2w      CHR     1,3      0t0     1028 /dev/null
j       31834 apache    3u  a_inode     0,9        0     5901 [eventfd]
j       31834 apache    4u  a_inode     0,9        0     5901 [eventpoll]
j       31834 apache    5u  a_inode     0,9        0     5901 [timerfd]
j       31834 apache    6r     FIFO     0,8      0t0  3514077 pipe
j       31834 apache    7w     FIFO     0,8      0t0  3514077 pipe
j       31834 apache    8u     IPv4 3514165      0t0      TCP openzwo.fritz.box:53296->static.12.31.201.138.clients.your-server.de:dec-notes (ESTABLISHED)
[root@openzwo ~]# 

So the IP address refers to a Hetzner server, from the port is assigned a speed test …

Many greetings


(Filippo Carletti) #17

I think that you will find a file named j in /tmp.
I suggest that you analyze it at virustotal.com.
Could you please send it to us, too?


(Gerald) #18

Thank you,

had searched for the file this morning, but in the tmp directory was nothing like that.
I have now blocked the port on nethserver and additionally in the Fritzbox.

In addition, I have deleted all the content from the tmp directory.

Let’s see what happens.

But how can something like that happen, that something like that fits into the system?
And why is the system again “almost” nrmal when the SPAM filter is switched off …

greetings
Gerald


(Filippo Carletti) #19

I don’t think that system load is linked to the spam filter. The load is generated by the j process, which is unknown and suspicious.
We will need to analyze the system. Which software are you running on NethServer? Any app like wordpress? Or other php software? Custom made?


(Gerald) #20

There are a total of three services that are not original.

Externally Wordpress runs once

Internally run two additional services as a virtual server:
phpmyadmin
and a trial platform with admidio (this is a club software) “admidio.org