Multiwan Down -Only one Connection- Works Sporadically

Setup:

3 inbound Internet connections setup in ‘Balance’ mode(50, 30, 20) with One static and two DHCP.

Everything has been working fine for over a month now in this mode until this morning when two of the links were showing as ‘DOWN’. I was curious and decided to check the connections by connecting them directly to a computer, and the connection was up. Same with the other one.

I tried doing some troubleshooting by ‘Releasing Role’ on the two that were not working. I get the following error when I try to do that.

Task completed with errors
Template /¦ #69 (exit status 1)
expansion of /¦ failed

As of now only one connection is up and internet is sporadic (30 secs on and 20 seconds off).

–Tried re-installing Firewall and still the same symptoms…

May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/actions
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/hosts
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/interfaces
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/maclist
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/mangle
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/masq
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/modules
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/nat
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/policy
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/providers
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/rtrules
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/rules
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/shorewall.conf
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/stoppedrules
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/tcinterfaces
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/tcpri
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/tunnels
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/zones
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/snat
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/findgw
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/helpers
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/firehol/fireqos.conf
May 26 07:59:07 gateway esmith::event[18944]: Action: /etc/e-smith/events/actions/generic_template_expand SUCCESS [0.624607]
May 26 07:59:07 gateway systemd: Reloading.
May 26 07:59:08 gateway sssd[be[vmakol.lan]]: Backend is online
May 26 07:59:08 gateway FireQOS[19293]: Cleared all QOS on all interfaces
May 26 07:59:09 gateway FireQOS[19341]: QoS applied ok (39 tc commands applied)
May 26 07:59:09 gateway root: Shorewall reloaded
May 26 07:59:09 gateway esmith::event[18944]: [NOTICE] Shorewall restart
May 26 07:59:09 gateway esmith::event[18944]: Action: /etc/e-smith/events/nethserver-firewall-base-save/S89nethserver-shorewall-restart SUCCESS [2.176035]
May 26 07:59:09 gateway systemd: Reloading.
May 26 07:59:09 gateway esmith::event[18944]: [INFO] service lsm restart
May 26 07:59:09 gateway systemd: Stopping LSM is the link status monitor…
May 26 07:59:09 gateway systemd: Started LSM is the link status monitor.
May 26 07:59:09 gateway systemd: Starting LSM is the link status monitor…
May 26 07:59:09 gateway esmith::event[18944]: Action: /etc/e-smith/events/actions/adjust-services SUCCESS [0.418427]
May 26 07:59:09 gateway esmith::event[18944]: Event: nethserver-firewall-base-save SUCCESS
May 26 07:59:09 gateway esmith::event[18943]: Action: /etc/e-smith/events/firewall-adjust/S20firewall-adjust SUCCESS [3.554889]
May 26 07:59:09 gateway esmith::event[18943]: Event: firewall-adjust SUCCESS
May 26 07:59:57 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 26 07:59:57 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 07:59:57 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
May 26 07:59:57 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:00:00 gateway esmith::event[20224]: Event: wan-uplink-update down SC root 0 0 0 0 0 0 0 0 unknown 1527314400
May 26 08:00:00 gateway esmith::event[20223]: Event: wan-uplink-update down Hola root 0 0 0 0 0 0 0 0 unknown 1527314400
May 26 08:00:00 gateway esmith::event[20222]: Event: wan-uplink-update up AeroM root 0 0 0 0 0 0 0 0 unknown 1527314400
May 26 08:00:00 gateway Shorewall[20236]: Enabling device enp2s0f1
May 26 08:00:01 gateway Shorewall[20243]: Disabling device enp3s0f1
May 26 08:00:01 gateway Shorewall[20250]: Disabling device enp3s0f0
May 26 08:00:01 gateway esmith::event[20222]: Action: /etc/e-smith/events/wan-uplink-update/S50nethserver-shorewall-wan-update SUCCESS [0.331084]
May 26 08:00:01 gateway esmith::event[20222]: Event: wan-uplink-update SUCCESS
May 26 08:00:05 gateway evebox: 2018-05-26 08:00:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 61
May 26 08:01:01 gateway esmith::event[20223]: Action: /etc/e-smith/events/wan-uplink-update/S50nethserver-shorewall-wan-update SUCCESS [60.404164]
May 26 08:01:01 gateway esmith::event[20223]: Event: wan-uplink-update SUCCESS
May 26 08:01:01 gateway esmith::event[20224]: Action: /etc/e-smith/events/wan-uplink-update/S50nethserver-shorewall-wan-update SUCCESS [60.406025]
May 26 08:01:01 gateway esmith::event[20224]: Event: wan-uplink-update SUCCESS
May 26 08:01:01 gateway systemd: Started Session 21 of user root.
May 26 08:01:01 gateway systemd: Starting Session 21 of user root.
May 26 08:01:05 gateway evebox: 2018-05-26 08:01:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:01:10 gateway dhclient[19805]: DHCPREQUEST on enp3s0f1 to 192.168.1.1 port 67 (xid=0x7a82b648)
May 26 08:01:10 gateway dhclient[19805]: DHCPACK from 192.168.1.1 (xid=0x7a82b648)
May 26 08:01:11 gateway chronyd[14790]: Selected source 81.19.96.148
May 26 08:01:12 gateway dhclient[19805]: bound to 192.168.1.95 – renewal in 269 seconds.
May 26 08:01:42 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 26 08:01:42 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:01:42 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
May 26 08:01:42 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:02:05 gateway evebox: 2018-05-26 08:02:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:03:05 gateway evebox: 2018-05-26 08:03:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:03:27 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 26 08:03:27 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:03:27 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
May 26 08:03:27 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:04:05 gateway evebox: 2018-05-26 08:04:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:05:05 gateway evebox: 2018-05-26 08:05:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:05:13 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 26 08:05:13 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:05:13 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
May 26 08:05:13 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:05:41 gateway dhclient[19805]: DHCPREQUEST on enp3s0f1 to 192.168.1.1 port 67 (xid=0x7a82b648)
May 26 08:05:41 gateway dhclient[19805]: DHCPACK from 192.168.1.1 (xid=0x7a82b648)
May 26 08:05:43 gateway dhclient[19805]: bound to 192.168.1.95 – renewal in 209 seconds.
May 26 08:06:05 gateway evebox: 2018-05-26 08:06:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:06:29 gateway dnsmasq-dhcp[16903]: DHCPRELEASE(enp2s0f0) 192.168.1.101 00:25:56:4a:ed:b5
May 26 08:06:31 gateway dnsmasq-tftp[16903]: file /var/lib/tftpboot/fp-net.cfg not found
May 26 08:06:40 gateway dnsmasq-tftp[16903]: file /var/lib/tftpboot/router.cfg not found
May 26 08:06:40 gateway dnsmasq-dhcp[16903]: DHCPDISCOVER(enp2s0f0) 00:25:56:4a:ed:b5
May 26 08:06:40 gateway dnsmasq-dhcp[16903]: DHCPOFFER(enp2s0f0) 192.168.1.101 00:25:56:4a:ed:b5
May 26 08:06:41 gateway dnsmasq-tftp[16903]: file /var/lib/tftpboot/fp-net.cfg not found
May 26 08:06:42 gateway dnsmasq-dhcp[16903]: DHCPREQUEST(enp2s0f0) 192.168.1.101 00:25:56:4a:ed:b5
May 26 08:06:42 gateway dnsmasq-dhcp[16903]: DHCPACK(enp2s0f0) 192.168.1.101 00:25:56:4a:ed:b5

Sorry, I don’t know what might be causing your problem. But…are you using a virtualised system? If so, what VM platform are you using and what NIC drivers is your NethServer using?

(I’m having problems of martian packets on a virtualised NethServer that using Omnios & KVM with virtio NICs).

Thanks
Bob

I am not using anything Virtualized. Its all on a pretty decent physical server.

I reinstalled NS. I have not applied any updates yet…just updated the network connections, installed basic firewall, and was setting up port forwarding when I got the following when saving the settings.

Task completed with errors
Template /etc/shorewall/masq #12 (exit status 1)
expansion of /etc/shorewall/masq failed
Configuring shorewall #29 (exit status 1)
Compiling using Shorewall 5.0.14.1…
WARNING: Unknown capability (CPU_FANOUT) ignored /etc/shorewall/capabilities (line 20)
WARNING: Unknown capability (NETMAP_TARGET) ignored /etc/shorewall/capabilities (line 62)
WARNING: Unknown capability (NFLOG_SIZE) ignored /etc/shorewall/capabilities (line 66)
WARNING: Unknown capability (RESTORE_WAIT_OPTION) ignored /etc/shorewall/capabilities (line 84)
Processing /etc/shorewall/params …
Processing /etc/shorewall/shorewall.conf…
WARNING: Unknown configuration option (BLACKLIST_DEFAULT) ignored /etc/shorewall/shorewall.conf (line 113)
Loading Modules…
Compiling /etc/shorewall/zones…
Compiling /etc/shorewall/interfaces…
Determining Hosts in Zones…
Locating Action Files…
Compiling /etc/shorewall/policy…
Running /etc/shorewall/initdone…
Adding Anti-smurf Rules
Adding rules for DHCP
Compiling TCP Flags filtering…
Compiling Kernel Route Filtering…
Compiling Martian Logging…
Compiling /etc/shorewall/providers…
Compiling /etc/shorewall/snat…
Compiling MAC Filtration – Phase 1…
Compiling /etc/shorewall/rules…
Compiling /etc/shorewall/conntrack…
Compiling MAC Filtration – Phase 2…
Applying Policies…
Compiling /usr/share/shorewall/action.Broadcast for chain Broadcast…
ERROR: Invalid parameter (DROP),Multicast(DROP) /usr/share/shorewall/action.Broadcast (line 1)
from (line EOF)

Reinstalled NS, applied the updates, configured multiwan with no errors or warning, but the symptoms are still the same. Very sporadic net access…
May 29 09:02:37 gateway kernel: IPv4: martian source 255.255.255.255 from 77.237.246.37, on dev enp3s0f1
May 29 09:02:39 gateway kernel: IPv4: martian source 255.255.255.255 from 217.71.204.119, on dev enp3s0f1
May 29 09:12:15 gateway shorewall: Compiling Martian Logging…
May 29 09:12:15 gateway shorewall: Setting up Martian Logging…
May 29 09:15:29 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:29 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:30 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:30 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:30 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:30 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:30 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:30 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:31 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:31 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:34 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:34 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:35 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:35 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:35 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:35 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:35 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:35 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:36 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:36 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:39 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:39 gateway kernel: IPv4: martian source 192.168.1.92 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:40 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:40 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:40 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:40 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:40 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:40 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:41 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:41 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 11:23:39 gateway shorewall: Compiling Martian Logging…
May 29 11:23:39 gateway shorewall: Setting up Martian Logging…
May 29 11:59:09 gateway shorewall: Compiling Martian Logging…
May 29 11:59:09 gateway shorewall: Setting up Martian Logging…
May 29 12:00:03 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 29 12:00:03 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
May 29 12:01:51 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 29 12:01:51 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0

Looking at a recurrent IP address I suspect a malware attack or infection.
You can check active processes, opened connections, logs and evebox/IPS for suspicious activity.

Below are the processes. I don’t see anything out of the ordinary in the log files.

[root@gateway html]# top
top - 08:40:49 up 20:42,  1 user,  load average: 0.04, 0.04, 0.05
Tasks: 227 total,   1 running, 226 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni, 99.5 id,  0.0 wa,  0.0 hi,  0.4 si,  0.0 st
KiB Mem : 16253188 total, 13276540 free,  1331476 used,  1645172 buff/cache
KiB Swap:  8257532 total,  8257532 free,        0 used. 14231648 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
20730 root      20   0  162108   2428   1584 R   0.7  0.0   0:00.07 top
   86 root      20   0       0      0      0 S   0.3  0.0   0:14.51 ksoftirqd/15
  113 root      20   0       0      0      0 S   0.3  0.0   0:03.93 kworker/14:1
  574 root      20   0       0      0      0 S   0.3  0.0   0:10.01 kworker/3:2
11331 root      20   0  262892   6108   4840 S   0.3  0.0   0:17.47 rsyslogd
22642 root      20   0       0      0      0 S   0.3  0.0   0:02.99 kworker/5:1
30797 suricata  20   0 1637472 112032   7936 S   0.3  0.7   1:11.13 evebox
    1 root      20   0  192036   4972   2616 S   0.0  0.0   0:18.53 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.01 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:01.77 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    6 root      20   0       0      0      0 S   0.0  0.0   0:04.53 kworker/u64:0
    8 root      rt   0       0      0      0 S   0.0  0.0   0:00.22 migration/0
    9 root      20   0       0      0      0 S   0.0  0.0   0:02.86 rcu_bh
   10 root      20   0       0      0      0 S   0.0  0.0   4:19.94 rcu_sched
   11 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 lru-add-drain
   12 root      rt   0       0      0      0 S   0.0  0.0   0:00.74 watchdog/0
   13 root      rt   0       0      0      0 S   0.0  0.0   0:00.18 watchdog/1
   14 root      rt   0       0      0      0 S   0.0  0.0   0:00.32 migration/1
   15 root      20   0       0      0      0 S   0.0  0.0   0:02.99 ksoftirqd/1
   17 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
   19 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 watchdog/2
   20 root      rt   0       0      0      0 S   0.0  0.0   0:00.24 migration/2
   21 root      20   0       0      0      0 S   0.0  0.0   0:01.43 ksoftirqd/2
   23 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:0H
   24 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 watchdog/3
   25 root      rt   0       0      0      0 S   0.0  0.0   0:00.53 migration/3
   26 root      20   0       0      0      0 S   0.0  0.0   0:02.27 ksoftirqd/3
   28 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:0H
   29 root      rt   0       0      0      0 S   0.0  0.0   0:00.18 watchdog/4
   30 root      rt   0       0      0      0 S   0.0  0.0   0:00.38 migration/4
   31 root      20   0       0      0      0 S   0.0  0.0   0:01.72 ksoftirqd/4
   33 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/4:0H
   34 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 watchdog/5
   35 root      rt   0       0      0      0 S   0.0  0.0   0:00.22 migration/5
   36 root      20   0       0      0      0 S   0.0  0.0   0:03.31 ksoftirqd/5
   38 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/5:0H
   39 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 watchdog/6
   40 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 migration/6
   41 root      20   0       0      0      0 S   0.0  0.0   0:00.24 ksoftirqd/6
   43 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/6:0H
   44 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 watchdog/7
   45 root      rt   0       0      0      0 S   0.0  0.0   0:00.21 migration/7
   46 root      20   0       0      0      0 S   0.0  0.0   0:02.05 ksoftirqd/7
   48 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/7:0H
   49 root      rt   0       0      0      0 S   0.0  0.0   0:00.16 watchdog/8
   50 root      rt   0       0      0      0 S   0.0  0.0   0:00.14 migration/8
   51 root      20   0       0      0      0 S   0.0  0.0   0:00.56 ksoftirqd/8
   53 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/8:0H
   54 root      rt   0       0      0      0 S   0.0  0.0   0:00.19 watchdog/9
   55 root      rt   0       0      0      0 S   0.0  0.0   0:00.19 migration/9
   56 root      20   0       0      0      0 S   0.0  0.0   0:00.07 ksoftirqd/9

Problem isolated. We were able to replicate the issue using ClearOS as well. We then decided to troubleshoot the issue by process of elimination by removing one switch at a time from the network. Once we identified the switch, we were able to narrow it down to a RHAT server, provided by Oracle Corp.
We were using it to test their new application development tool.
I am suspecting malware but we haven’t looked into it yet. I am glad to post that its nothing to do with NS, and the problem is isolated.

2 Likes