NethServer Version: 7 final Module: main
Hello,
today my server was shutdown unexpected. It runs on a virtual mashine with kvm at ubuntu 14.04, a second virtual mashine with a nethserver too has no problems.
Here are the last entries of message.log before shutdown:
Feb 10 08:01:23 groupware systemd: Created slice user-0.slice.
Feb 10 08:01:23 groupware systemd: Starting user-0.slice.
Feb 10 08:01:23 groupware systemd: Started Session 2299 of user root.
Feb 10 08:01:23 groupware systemd: Starting Session 2299 of user root.
Feb 10 08:01:24 groupware systemd: Removed slice user-0.slice.
Feb 10 08:01:24 groupware systemd: Stopping user-0.slice. Feb 10 08:03:28 groupware clamd: SelfCheck: Database status OK.
Feb 10 04:01:01 groupware systemd: Created slice user-0.slice.
Feb 10 04:01:01 groupware systemd: Starting user-0.slice.
Feb 10 04:01:01 groupware systemd: Started Session 2288 of user root.
Feb 10 04:01:01 groupware systemd: Starting Session 2288 of user root.
Feb 10 04:01:01 groupware systemd: Removed slice user-0.slice.
Feb 10 04:01:01 groupware systemd: Stopping user-0.slice.
Feb 10 04:03:10 groupware clamd: SelfCheck: Database status OK.
Feb 10 04:10:01 groupware systemd: Created slice user-0.slice.
Feb 10 04:10:01 groupware systemd: Starting user-0.slice.
Feb 10 04:10:01 groupware systemd: Started Session 2289 of user root.
Feb 10 04:10:01 groupware systemd: Starting Session 2289 of user root.
Feb 10 04:13:10 groupware clamd: SelfCheck: Database status OK.
Feb 10 04:23:10 groupware clamd: SelfCheck: Database status OK.
Feb 10 04:33:10 groupware clamd: SelfCheck: Database status OK.
Feb 10 04:39:27 groupware kernel: br0: received packet on ens3 with own address as source address
Feb 10 04:43:10 groupware clamd: SelfCheck: Database status OK.
Feb 10 04:45:01 groupware systemd: Started Session 2290 of user root.
Feb 10 04:45:01 groupware systemd: Starting Session 2290 of user root.
Feb 10 04:53:10 groupware clamd: SelfCheck: Database modification detected. Forcing reload.
Feb 10 04:53:12 groupware clamd: Reading databases from /var/lib/clamav
Feb 10 04:53:32 groupware clamd: Database correctly reloaded (5899812 signatures)
Feb 10 05:01:01 groupware systemd: Started Session 2291 of user root.
Feb 10 05:01:01 groupware systemd: Starting Session 2291 of user root.
Feb 10 05:03:32 groupware clamd: SelfCheck: Database status OK.
Feb 10 05:10:01 groupware systemd: Started Session 2292 of user root.
Feb 10 05:10:01 groupware systemd: Starting Session 2292 of user root.
Feb 10 05:10:58 groupware systemd: Removed slice user-0.slice.
Feb 10 05:10:58 groupware systemd: Stopping user-0.slice.
Feb 10 05:13:32 groupware clamd: SelfCheck: Database status OK.
Feb 10 05:23:32 groupware clamd: SelfCheck: Database status OK.
Feb 10 05:33:32 groupware clamd: SelfCheck: Database status OK.
Feb 10 05:39:27 groupware kernel: br0: received packet on ens3 with own address as source address
Feb 10 05:43:32 groupware clamd: SelfCheck: Database status OK.
Feb 10 05:45:01 groupware systemd: Created slice user-0.slice.
Feb 10 05:45:01 groupware systemd: Starting user-0.slice.
Feb 10 05:45:01 groupware systemd: Started Session 2293 of user root.
Feb 10 05:45:01 groupware systemd: Starting Session 2293 of user root.
Feb 10 05:47:56 groupware systemd: Removed slice user-0.slice.
Feb 10 05:47:56 groupware systemd: Stopping user-0.slice.
Feb 10 05:53:32 groupware clamd: SelfCheck: Database modification detected. Forcing reload.
Feb 10 05:53:33 groupware clamd: Reading databases from /var/lib/clamav
Feb 10 05:53:52 groupware clamd: Database correctly reloaded (5899816 signatures)
Feb 10 06:00:01 groupware systemd: Created slice user-0.slice.
Feb 10 06:00:01 groupware systemd: Starting user-0.slice.
Feb 10 06:00:01 groupware systemd: Started Session 2294 of user root.
Feb 10 06:00:01 groupware systemd: Starting Session 2294 of user root.
Feb 10 06:01:01 groupware systemd: Started Session 2295 of user root.
Feb 10 06:01:01 groupware systemd: Starting Session 2295 of user root.
Feb 10 06:03:52 groupware clamd: SelfCheck: Database status OK.
Feb 10 06:13:52 groupware clamd: SelfCheck: Database status OK.
Feb 10 06:22:07 groupware freshclam[9181]: ClamAV update process started at Fri Feb 10 06:22:07 2017
Feb 10 06:22:07 groupware freshclam[9181]: main.cvd is up to date (version: 57, sigs: 4218790, f-level: 60, builder: amishhammer)
Feb 10 06:22:08 groupware freshclam[9181]: getfile: daily-23035.cdiff not found on database.clamav.net (IP: 62.245.181.53)
Feb 10 06:22:08 groupware freshclam[9181]: getpatch: Can’t download daily-23035.cdiff from database.clamav.net
Feb 10 06:22:08 groupware freshclam[9181]: Trying host database.clamav.net (62.27.56.14)…
Feb 10 06:22:09 groupware freshclam[9181]: Downloading daily-23035.cdiff [100%]
Feb 10 06:22:17 groupware freshclam[9181]: daily.cld updated (version: 23035, sigs: 1535322, f-level: 63, builder: neo)
Feb 10 06:22:17 groupware freshclam[9181]: bytecode.cld is up to date (version: 290, sigs: 55, f-level: 63, builder: neo)
Feb 10 06:22:23 groupware freshclam[9181]: Database updated (5754167 signatures) from database.clamav.net (IP: 62.27.56.14)
Feb 10 06:22:24 groupware systemd: Removed slice user-0.slice.
Feb 10 06:22:24 groupware systemd: Stopping user-0.slice.
Feb 10 06:22:25 groupware clamd: Reading databases from /var/lib/clamav
Feb 10 06:22:44 groupware clamd: Database correctly reloaded (5900242 signatures)
Feb 10 06:32:44 groupware clamd: SelfCheck: Database status OK.
Feb 10 06:39:28 groupware kernel: br0: received packet on ens3 with own address as source address
Feb 10 06:42:44 groupware clamd: SelfCheck: Database status OK.
Feb 10 06:45:01 groupware systemd: Created slice user-0.slice.
Feb 10 06:45:01 groupware systemd: Starting user-0.slice.
Feb 10 06:45:01 groupware systemd: Started Session 2296 of user root.
Feb 10 06:45:01 groupware systemd: Starting Session 2296 of user root.
Feb 10 06:52:35 groupware kernel: IPv4: host 192.168.46.131/if3 ignores redirects for 104.108.168.41 to 192.168.46.240
Feb 10 06:52:44 groupware clamd: SelfCheck: Database status OK.
Feb 10 06:52:51 groupware kernel: IPv4: host 192.168.46.132/if3 ignores redirects for 104.108.168.41 to 192.168.46.240
Feb 10 06:53:11 groupware kernel: IPv4: host 192.168.46.110/if3 ignores redirects for 40.77.226.250 to 192.168.46.240
Feb 10 06:53:19 groupware kernel: IPv4: host 192.168.46.128/if3 ignores redirects for 88.221.117.184 to 192.168.46.240
Feb 10 06:53:36 groupware kernel: IPv4: host 192.168.46.131/if3 ignores redirects for 88.221.117.184 to 192.168.46.240
Feb 10 06:53:53 groupware kernel: IPv4: host 192.168.46.132/if3 ignores redirects for 88.221.117.184 to 192.168.46.240
Feb 10 06:54:22 groupware systemd: Removed slice user-0.slice.
Feb 10 06:54:22 groupware systemd: Stopping user-0.slice.
Feb 10 06:58:38 groupware kernel: IPv4: host 192.168.46.131/if3 ignores redirects for 88.221.117.184 to 192.168.46.240
Feb 10 06:58:54 groupware kernel: IPv4: host 192.168.46.132/if3 ignores redirects for 104.106.230.10 to 192.168.46.240
Feb 10 06:59:20 groupware kernel: IPv4: host 192.168.46.128/if3 ignores redirects for 88.221.117.210 to 192.168.46.240
Feb 10 06:59:44 groupware kernel: IPv4: host 192.168.46.132/if3 ignores redirects for 104.108.150.142 to 192.168.46.240
Feb 10 07:00:09 groupware kernel: IPv4: host 192.168.46.110/if3 ignores redirects for 104.68.221.213 to 192.168.46.240
Feb 10 07:01:01 groupware systemd: Created slice user-0.slice.
Feb 10 07:01:01 groupware systemd: Starting user-0.slice.
Feb 10 07:01:01 groupware systemd: Started Session 2297 of user root.
Feb 10 07:01:01 groupware systemd: Starting Session 2297 of user root.
Feb 10 07:01:01 groupware systemd: Removed slice user-0.slice.
Feb 10 07:01:01 groupware systemd: Stopping user-0.slice.
Feb 10 07:02:44 groupware clamd: SelfCheck: Database modification detected. Forcing reload.
Feb 10 07:02:45 groupware clamd: Reading databases from /var/lib/clamav
Feb 10 07:03:04 groupware clamd: Database correctly reloaded (5900237 signatures)
Feb 10 07:05:09 groupware kernel: IPv4: host 192.168.46.142/if3 ignores redirects for 88.221.117.184 to 192.168.46.240
Feb 10 07:10:11 groupware kernel: IPv4: host 192.168.46.142/if3 ignores redirects for 88.221.117.210 to 192.168.46.240
Feb 10 07:13:04 groupware clamd: SelfCheck: Database status OK.
Feb 10 07:13:34 groupware kernel: IPv4: host 192.168.46.142/if3 ignores redirects for 104.108.191.202 to 192.168.46.240
Feb 10 07:15:43 groupware kernel: IPv4: host 192.168.46.131/if3 ignores redirects for 2.23.59.13 to 192.168.46.240
Feb 10 07:17:31 groupware kernel: IPv4: host 192.168.46.107/if3 ignores redirects for 40.77.226.250 to 192.168.46.240
Feb 10 07:19:04 groupware kernel: IPv4: host 192.168.46.107/if3 ignores redirects for 88.221.117.184 to 192.168.46.240
Feb 10 07:19:18 groupware kernel: IPv4: host 192.168.46.157/if3 ignores redirects for 88.221.117.210 to 192.168.46.240
Feb 10 07:20:06 groupware kernel: IPv4: host 192.168.46.107/if3 ignores redirects for 88.221.117.210 to 192.168.46.240
Feb 10 07:20:13 groupware kernel: IPv4: host 192.168.46.157/if3 ignores redirects for 104.108.168.41 to 192.168.46.240
Feb 10 07:23:04 groupware clamd: SelfCheck: Database status OK.
Feb 10 07:25:16 groupware kernel: IPv4: host 192.168.46.157/if3 ignores redirects for 104.108.150.142 to 192.168.46.240
Feb 10 07:28:10 groupware kernel: IPv4: host 192.168.46.107/if3 ignores redirects for 40.77.226.250 to 192.168.46.240
Feb 10 07:30:17 groupware kernel: IPv4: host 192.168.46.148/if3 ignores redirects for 104.108.168.41 to 192.168.46.240
Feb 10 07:31:18 groupware kernel: IPv4: host 192.168.46.148/if3 ignores redirects for 88.221.117.184 to 192.168.46.240
Feb 10 07:33:04 groupware clamd: SelfCheck: Database status OK.
Feb 10 07:36:21 groupware kernel: IPv4: host 192.168.46.148/if3 ignores redirects for 88.221.117.210 to 192.168.46.240
Feb 10 07:36:44 groupware kernel: IPv4: host 192.168.46.101/if3 ignores redirects for 88.221.117.186 to 192.168.46.240
Feb 10 07:38:37 groupware kernel: IPv4: host 192.168.46.101/if3 ignores redirects for 104.68.221.213 to 192.168.46.240
Feb 10 07:39:15 groupware kernel: IPv4: host 192.168.46.122/if3 ignores redirects for 40.77.226.249 to 192.168.46.240
Feb 10 07:39:23 groupware kernel: IPv4: host 192.168.46.114/if3 ignores redirects for 104.108.150.142 to 192.168.46.240
Feb 10 07:39:28 groupware kernel: br0: received packet on ens3 with own address as source address
Feb 10 07:39:51 groupware kernel: IPv4: host 192.168.46.122/if3 ignores redirects for 88.221.117.210 to 192.168.46.240
Feb 10 07:40:53 groupware kernel: IPv4: host 192.168.46.122/if3 ignores redirects for 88.221.117.184 to 192.168.46.240
I’ve found my error at the system.log of the host.
Out of memory: Kill process 11860 (qemu-system-x86) score 245 or sacrifice child
Killed process 11860 (qemu-system-x86) total-vm:5588372kB, anon-rss:2999628kB, file-rss:0kB
If other people have the same issue, here is what I’ve done to prohibit killing the process.
setup the following cronjob:
*/1 * * * * root pgrep -f qemu-system-x86_64 | while read PID; do echo -17 > /proc/$PID/oom_adj; done
It reads the PID of the qemu task and puts a “-17” to the oom_adj file every minute. “-17” at oom_adj file means, this process wouldn’t be killed by low memory problem. Because the PID would change by Server / Process restarting you have to do a cronjob.