Multiwan Down -Only one Connection- Works Sporadically

multiwan
(Vinit Makol) #1

Setup:

3 inbound Internet connections setup in ‘Balance’ mode(50, 30, 20) with One static and two DHCP.

Everything has been working fine for over a month now in this mode until this morning when two of the links were showing as ‘DOWN’. I was curious and decided to check the connections by connecting them directly to a computer, and the connection was up. Same with the other one.

I tried doing some troubleshooting by ‘Releasing Role’ on the two that were not working. I get the following error when I try to do that.

Task completed with errors
Template /¦ #69 (exit status 1)
expansion of /¦ failed

As of now only one connection is up and internet is sporadic (30 secs on and 20 seconds off).

–Tried re-installing Firewall and still the same symptoms…

May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/actions
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/hosts
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/interfaces
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/maclist
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/mangle
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/masq
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/modules
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/nat
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/policy
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/providers
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/rtrules
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/rules
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/shorewall.conf
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/stoppedrules
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/tcinterfaces
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/tcpri
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/tunnels
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/zones
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/snat
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/findgw
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/helpers
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/firehol/fireqos.conf
May 26 07:59:07 gateway esmith::event[18944]: Action: /etc/e-smith/events/actions/generic_template_expand SUCCESS [0.624607]
May 26 07:59:07 gateway systemd: Reloading.
May 26 07:59:08 gateway sssd[be[vmakol.lan]]: Backend is online
May 26 07:59:08 gateway FireQOS[19293]: Cleared all QOS on all interfaces
May 26 07:59:09 gateway FireQOS[19341]: QoS applied ok (39 tc commands applied)
May 26 07:59:09 gateway root: Shorewall reloaded
May 26 07:59:09 gateway esmith::event[18944]: [NOTICE] Shorewall restart
May 26 07:59:09 gateway esmith::event[18944]: Action: /etc/e-smith/events/nethserver-firewall-base-save/S89nethserver-shorewall-restart SUCCESS [2.176035]
May 26 07:59:09 gateway systemd: Reloading.
May 26 07:59:09 gateway esmith::event[18944]: [INFO] service lsm restart
May 26 07:59:09 gateway systemd: Stopping LSM is the link status monitor…
May 26 07:59:09 gateway systemd: Started LSM is the link status monitor.
May 26 07:59:09 gateway systemd: Starting LSM is the link status monitor…
May 26 07:59:09 gateway esmith::event[18944]: Action: /etc/e-smith/events/actions/adjust-services SUCCESS [0.418427]
May 26 07:59:09 gateway esmith::event[18944]: Event: nethserver-firewall-base-save SUCCESS
May 26 07:59:09 gateway esmith::event[18943]: Action: /etc/e-smith/events/firewall-adjust/S20firewall-adjust SUCCESS [3.554889]
May 26 07:59:09 gateway esmith::event[18943]: Event: firewall-adjust SUCCESS
May 26 07:59:57 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 26 07:59:57 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 07:59:57 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
May 26 07:59:57 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:00:00 gateway esmith::event[20224]: Event: wan-uplink-update down SC root 0 0 0 0 0 0 0 0 unknown 1527314400
May 26 08:00:00 gateway esmith::event[20223]: Event: wan-uplink-update down Hola root 0 0 0 0 0 0 0 0 unknown 1527314400
May 26 08:00:00 gateway esmith::event[20222]: Event: wan-uplink-update up AeroM root 0 0 0 0 0 0 0 0 unknown 1527314400
May 26 08:00:00 gateway Shorewall[20236]: Enabling device enp2s0f1
May 26 08:00:01 gateway Shorewall[20243]: Disabling device enp3s0f1
May 26 08:00:01 gateway Shorewall[20250]: Disabling device enp3s0f0
May 26 08:00:01 gateway esmith::event[20222]: Action: /etc/e-smith/events/wan-uplink-update/S50nethserver-shorewall-wan-update SUCCESS [0.331084]
May 26 08:00:01 gateway esmith::event[20222]: Event: wan-uplink-update SUCCESS
May 26 08:00:05 gateway evebox: 2018-05-26 08:00:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 61
May 26 08:01:01 gateway esmith::event[20223]: Action: /etc/e-smith/events/wan-uplink-update/S50nethserver-shorewall-wan-update SUCCESS [60.404164]
May 26 08:01:01 gateway esmith::event[20223]: Event: wan-uplink-update SUCCESS
May 26 08:01:01 gateway esmith::event[20224]: Action: /etc/e-smith/events/wan-uplink-update/S50nethserver-shorewall-wan-update SUCCESS [60.406025]
May 26 08:01:01 gateway esmith::event[20224]: Event: wan-uplink-update SUCCESS
May 26 08:01:01 gateway systemd: Started Session 21 of user root.
May 26 08:01:01 gateway systemd: Starting Session 21 of user root.
May 26 08:01:05 gateway evebox: 2018-05-26 08:01:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:01:10 gateway dhclient[19805]: DHCPREQUEST on enp3s0f1 to 192.168.1.1 port 67 (xid=0x7a82b648)
May 26 08:01:10 gateway dhclient[19805]: DHCPACK from 192.168.1.1 (xid=0x7a82b648)
May 26 08:01:11 gateway chronyd[14790]: Selected source 81.19.96.148
May 26 08:01:12 gateway dhclient[19805]: bound to 192.168.1.95 – renewal in 269 seconds.
May 26 08:01:42 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 26 08:01:42 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:01:42 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
May 26 08:01:42 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:02:05 gateway evebox: 2018-05-26 08:02:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:03:05 gateway evebox: 2018-05-26 08:03:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:03:27 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 26 08:03:27 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:03:27 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
May 26 08:03:27 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:04:05 gateway evebox: 2018-05-26 08:04:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:05:05 gateway evebox: 2018-05-26 08:05:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:05:13 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 26 08:05:13 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:05:13 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
May 26 08:05:13 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:05:41 gateway dhclient[19805]: DHCPREQUEST on enp3s0f1 to 192.168.1.1 port 67 (xid=0x7a82b648)
May 26 08:05:41 gateway dhclient[19805]: DHCPACK from 192.168.1.1 (xid=0x7a82b648)
May 26 08:05:43 gateway dhclient[19805]: bound to 192.168.1.95 – renewal in 209 seconds.
May 26 08:06:05 gateway evebox: 2018-05-26 08:06:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:06:29 gateway dnsmasq-dhcp[16903]: DHCPRELEASE(enp2s0f0) 192.168.1.101 00:25:56:4a:ed:b5
May 26 08:06:31 gateway dnsmasq-tftp[16903]: file /var/lib/tftpboot/fp-net.cfg not found
May 26 08:06:40 gateway dnsmasq-tftp[16903]: file /var/lib/tftpboot/router.cfg not found
May 26 08:06:40 gateway dnsmasq-dhcp[16903]: DHCPDISCOVER(enp2s0f0) 00:25:56:4a:ed:b5
May 26 08:06:40 gateway dnsmasq-dhcp[16903]: DHCPOFFER(enp2s0f0) 192.168.1.101 00:25:56:4a:ed:b5
May 26 08:06:41 gateway dnsmasq-tftp[16903]: file /var/lib/tftpboot/fp-net.cfg not found
May 26 08:06:42 gateway dnsmasq-dhcp[16903]: DHCPREQUEST(enp2s0f0) 192.168.1.101 00:25:56:4a:ed:b5
May 26 08:06:42 gateway dnsmasq-dhcp[16903]: DHCPACK(enp2s0f0) 192.168.1.101 00:25:56:4a:ed:b5

Getting lots of martian packets
Ntop misterious traffic
(bob) #2

Sorry, I don’t know what might be causing your problem. But…are you using a virtualised system? If so, what VM platform are you using and what NIC drivers is your NethServer using?

(I’m having problems of martian packets on a virtualised NethServer that using Omnios & KVM with virtio NICs).

Thanks
Bob

(Vinit Makol) #3

I am not using anything Virtualized. Its all on a pretty decent physical server.

(Vinit Makol) #5

I reinstalled NS. I have not applied any updates yet…just updated the network connections, installed basic firewall, and was setting up port forwarding when I got the following when saving the settings.

Task completed with errors
Template /etc/shorewall/masq #12 (exit status 1)
expansion of /etc/shorewall/masq failed
Configuring shorewall #29 (exit status 1)
Compiling using Shorewall 5.0.14.1…
WARNING: Unknown capability (CPU_FANOUT) ignored /etc/shorewall/capabilities (line 20)
WARNING: Unknown capability (NETMAP_TARGET) ignored /etc/shorewall/capabilities (line 62)
WARNING: Unknown capability (NFLOG_SIZE) ignored /etc/shorewall/capabilities (line 66)
WARNING: Unknown capability (RESTORE_WAIT_OPTION) ignored /etc/shorewall/capabilities (line 84)
Processing /etc/shorewall/params …
Processing /etc/shorewall/shorewall.conf…
WARNING: Unknown configuration option (BLACKLIST_DEFAULT) ignored /etc/shorewall/shorewall.conf (line 113)
Loading Modules…
Compiling /etc/shorewall/zones…
Compiling /etc/shorewall/interfaces…
Determining Hosts in Zones…
Locating Action Files…
Compiling /etc/shorewall/policy…
Running /etc/shorewall/initdone…
Adding Anti-smurf Rules
Adding rules for DHCP
Compiling TCP Flags filtering…
Compiling Kernel Route Filtering…
Compiling Martian Logging…
Compiling /etc/shorewall/providers…
Compiling /etc/shorewall/snat…
Compiling MAC Filtration – Phase 1…
Compiling /etc/shorewall/rules…
Compiling /etc/shorewall/conntrack…
Compiling MAC Filtration – Phase 2…
Applying Policies…
Compiling /usr/share/shorewall/action.Broadcast for chain Broadcast…
ERROR: Invalid parameter (DROP),Multicast(DROP) /usr/share/shorewall/action.Broadcast (line 1)
from (line EOF)

(Vinit Makol) #6

Reinstalled NS, applied the updates, configured multiwan with no errors or warning, but the symptoms are still the same. Very sporadic net access…
May 29 09:02:37 gateway kernel: IPv4: martian source 255.255.255.255 from 77.237.246.37, on dev enp3s0f1
May 29 09:02:39 gateway kernel: IPv4: martian source 255.255.255.255 from 217.71.204.119, on dev enp3s0f1
May 29 09:12:15 gateway shorewall: Compiling Martian Logging…
May 29 09:12:15 gateway shorewall: Setting up Martian Logging…
May 29 09:15:29 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:29 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:30 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:30 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:30 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:30 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:30 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:30 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:31 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:31 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:34 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:34 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:35 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:35 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:35 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:35 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:35 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:35 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:36 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:36 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:39 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:39 gateway kernel: IPv4: martian source 192.168.1.92 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:40 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:40 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:40 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:40 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:40 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:40 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:41 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:41 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 11:23:39 gateway shorewall: Compiling Martian Logging…
May 29 11:23:39 gateway shorewall: Setting up Martian Logging…
May 29 11:59:09 gateway shorewall: Compiling Martian Logging…
May 29 11:59:09 gateway shorewall: Setting up Martian Logging…
May 29 12:00:03 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 29 12:00:03 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
May 29 12:01:51 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 29 12:01:51 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0

(Marc) #7

Looking at a recurrent IP address I suspect a malware attack or infection.
You can check active processes, opened connections, logs and evebox/IPS for suspicious activity.

(Vinit Makol) #8

Below are the processes. I don’t see anything out of the ordinary in the log files.

[root@gateway html]# top
top - 08:40:49 up 20:42,  1 user,  load average: 0.04, 0.04, 0.05
Tasks: 227 total,   1 running, 226 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni, 99.5 id,  0.0 wa,  0.0 hi,  0.4 si,  0.0 st
KiB Mem : 16253188 total, 13276540 free,  1331476 used,  1645172 buff/cache
KiB Swap:  8257532 total,  8257532 free,        0 used. 14231648 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
20730 root      20   0  162108   2428   1584 R   0.7  0.0   0:00.07 top
   86 root      20   0       0      0      0 S   0.3  0.0   0:14.51 ksoftirqd/15
  113 root      20   0       0      0      0 S   0.3  0.0   0:03.93 kworker/14:1
  574 root      20   0       0      0      0 S   0.3  0.0   0:10.01 kworker/3:2
11331 root      20   0  262892   6108   4840 S   0.3  0.0   0:17.47 rsyslogd
22642 root      20   0       0      0      0 S   0.3  0.0   0:02.99 kworker/5:1
30797 suricata  20   0 1637472 112032   7936 S   0.3  0.7   1:11.13 evebox
    1 root      20   0  192036   4972   2616 S   0.0  0.0   0:18.53 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.01 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:01.77 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    6 root      20   0       0      0      0 S   0.0  0.0   0:04.53 kworker/u64:0
    8 root      rt   0       0      0      0 S   0.0  0.0   0:00.22 migration/0
    9 root      20   0       0      0      0 S   0.0  0.0   0:02.86 rcu_bh
   10 root      20   0       0      0      0 S   0.0  0.0   4:19.94 rcu_sched
   11 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 lru-add-drain
   12 root      rt   0       0      0      0 S   0.0  0.0   0:00.74 watchdog/0
   13 root      rt   0       0      0      0 S   0.0  0.0   0:00.18 watchdog/1
   14 root      rt   0       0      0      0 S   0.0  0.0   0:00.32 migration/1
   15 root      20   0       0      0      0 S   0.0  0.0   0:02.99 ksoftirqd/1
   17 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
   19 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 watchdog/2
   20 root      rt   0       0      0      0 S   0.0  0.0   0:00.24 migration/2
   21 root      20   0       0      0      0 S   0.0  0.0   0:01.43 ksoftirqd/2
   23 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:0H
   24 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 watchdog/3
   25 root      rt   0       0      0      0 S   0.0  0.0   0:00.53 migration/3
   26 root      20   0       0      0      0 S   0.0  0.0   0:02.27 ksoftirqd/3
   28 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:0H
   29 root      rt   0       0      0      0 S   0.0  0.0   0:00.18 watchdog/4
   30 root      rt   0       0      0      0 S   0.0  0.0   0:00.38 migration/4
   31 root      20   0       0      0      0 S   0.0  0.0   0:01.72 ksoftirqd/4
   33 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/4:0H
   34 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 watchdog/5
   35 root      rt   0       0      0      0 S   0.0  0.0   0:00.22 migration/5
   36 root      20   0       0      0      0 S   0.0  0.0   0:03.31 ksoftirqd/5
   38 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/5:0H
   39 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 watchdog/6
   40 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 migration/6
   41 root      20   0       0      0      0 S   0.0  0.0   0:00.24 ksoftirqd/6
   43 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/6:0H
   44 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 watchdog/7
   45 root      rt   0       0      0      0 S   0.0  0.0   0:00.21 migration/7
   46 root      20   0       0      0      0 S   0.0  0.0   0:02.05 ksoftirqd/7
   48 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/7:0H
   49 root      rt   0       0      0      0 S   0.0  0.0   0:00.16 watchdog/8
   50 root      rt   0       0      0      0 S   0.0  0.0   0:00.14 migration/8
   51 root      20   0       0      0      0 S   0.0  0.0   0:00.56 ksoftirqd/8
   53 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/8:0H
   54 root      rt   0       0      0      0 S   0.0  0.0   0:00.19 watchdog/9
   55 root      rt   0       0      0      0 S   0.0  0.0   0:00.19 migration/9
   56 root      20   0       0      0      0 S   0.0  0.0   0:00.07 ksoftirqd/9
(Vinit Makol) #9

Problem isolated. We were able to replicate the issue using ClearOS as well. We then decided to troubleshoot the issue by process of elimination by removing one switch at a time from the network. Once we identified the switch, we were able to narrow it down to a RHAT server, provided by Oracle Corp.
We were using it to test their new application development tool.
I am suspecting malware but we haven’t looked into it yet. I am glad to post that its nothing to do with NS, and the problem is isolated.

Received packet with own address as source address