3 inbound Internet connections setup in ‘Balance’ mode(50, 30, 20) with One static and two DHCP.
Everything has been working fine for over a month now in this mode until this morning when two of the links were showing as ‘DOWN’. I was curious and decided to check the connections by connecting them directly to a computer, and the connection was up. Same with the other one.
I tried doing some troubleshooting by ‘Releasing Role’ on the two that were not working. I get the following error when I try to do that.
Task completed with errors
Template /¦ #69 (exit status 1)
expansion of /¦ failed
As of now only one connection is up and internet is sporadic (30 secs on and 20 seconds off).
–Tried re-installing Firewall and still the same symptoms…
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/actions
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/hosts
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/interfaces
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/maclist
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/mangle
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/masq
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/modules
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/nat
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/policy
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/providers
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/rtrules
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/rules
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/shorewall.conf
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/stoppedrules
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/tcinterfaces
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/tcpri
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/tunnels
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/zones
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/snat
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/findgw
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/helpers
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/firehol/fireqos.conf
May 26 07:59:07 gateway esmith::event[18944]: Action: /etc/e-smith/events/actions/generic_template_expand SUCCESS [0.624607]
May 26 07:59:07 gateway systemd: Reloading.
May 26 07:59:08 gateway sssd[be[vmakol.lan]]: Backend is online
May 26 07:59:08 gateway FireQOS[19293]: Cleared all QOS on all interfaces
May 26 07:59:09 gateway FireQOS[19341]: QoS applied ok (39 tc commands applied)
May 26 07:59:09 gateway root: Shorewall reloaded
May 26 07:59:09 gateway esmith::event[18944]: [NOTICE] Shorewall restart
May 26 07:59:09 gateway esmith::event[18944]: Action: /etc/e-smith/events/nethserver-firewall-base-save/S89nethserver-shorewall-restart SUCCESS [2.176035]
May 26 07:59:09 gateway systemd: Reloading.
May 26 07:59:09 gateway esmith::event[18944]: [INFO] service lsm restart
May 26 07:59:09 gateway systemd: Stopping LSM is the link status monitor…
May 26 07:59:09 gateway systemd: Started LSM is the link status monitor.
May 26 07:59:09 gateway systemd: Starting LSM is the link status monitor…
May 26 07:59:09 gateway esmith::event[18944]: Action: /etc/e-smith/events/actions/adjust-services SUCCESS [0.418427]
May 26 07:59:09 gateway esmith::event[18944]: Event: nethserver-firewall-base-save SUCCESS
May 26 07:59:09 gateway esmith::event[18943]: Action: /etc/e-smith/events/firewall-adjust/S20firewall-adjust SUCCESS [3.554889]
May 26 07:59:09 gateway esmith::event[18943]: Event: firewall-adjust SUCCESS
May 26 07:59:57 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 26 07:59:57 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 07:59:57 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
May 26 07:59:57 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:00:00 gateway esmith::event[20224]: Event: wan-uplink-update down SC root 0 0 0 0 0 0 0 0 unknown 1527314400
May 26 08:00:00 gateway esmith::event[20223]: Event: wan-uplink-update down Hola root 0 0 0 0 0 0 0 0 unknown 1527314400
May 26 08:00:00 gateway esmith::event[20222]: Event: wan-uplink-update up AeroM root 0 0 0 0 0 0 0 0 unknown 1527314400
May 26 08:00:00 gateway Shorewall[20236]: Enabling device enp2s0f1
May 26 08:00:01 gateway Shorewall[20243]: Disabling device enp3s0f1
May 26 08:00:01 gateway Shorewall[20250]: Disabling device enp3s0f0
May 26 08:00:01 gateway esmith::event[20222]: Action: /etc/e-smith/events/wan-uplink-update/S50nethserver-shorewall-wan-update SUCCESS [0.331084]
May 26 08:00:01 gateway esmith::event[20222]: Event: wan-uplink-update SUCCESS
May 26 08:00:05 gateway evebox: 2018-05-26 08:00:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 61
May 26 08:01:01 gateway esmith::event[20223]: Action: /etc/e-smith/events/wan-uplink-update/S50nethserver-shorewall-wan-update SUCCESS [60.404164]
May 26 08:01:01 gateway esmith::event[20223]: Event: wan-uplink-update SUCCESS
May 26 08:01:01 gateway esmith::event[20224]: Action: /etc/e-smith/events/wan-uplink-update/S50nethserver-shorewall-wan-update SUCCESS [60.406025]
May 26 08:01:01 gateway esmith::event[20224]: Event: wan-uplink-update SUCCESS
May 26 08:01:01 gateway systemd: Started Session 21 of user root.
May 26 08:01:01 gateway systemd: Starting Session 21 of user root.
May 26 08:01:05 gateway evebox: 2018-05-26 08:01:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:01:10 gateway dhclient[19805]: DHCPREQUEST on enp3s0f1 to 192.168.1.1 port 67 (xid=0x7a82b648)
May 26 08:01:10 gateway dhclient[19805]: DHCPACK from 192.168.1.1 (xid=0x7a82b648)
May 26 08:01:11 gateway chronyd[14790]: Selected source 81.19.96.148
May 26 08:01:12 gateway dhclient[19805]: bound to 192.168.1.95 – renewal in 269 seconds.
May 26 08:01:42 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 26 08:01:42 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:01:42 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
May 26 08:01:42 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:02:05 gateway evebox: 2018-05-26 08:02:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:03:05 gateway evebox: 2018-05-26 08:03:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:03:27 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 26 08:03:27 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:03:27 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
May 26 08:03:27 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:04:05 gateway evebox: 2018-05-26 08:04:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:05:05 gateway evebox: 2018-05-26 08:05:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:05:13 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 26 08:05:13 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:05:13 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
May 26 08:05:13 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:05:41 gateway dhclient[19805]: DHCPREQUEST on enp3s0f1 to 192.168.1.1 port 67 (xid=0x7a82b648)
May 26 08:05:41 gateway dhclient[19805]: DHCPACK from 192.168.1.1 (xid=0x7a82b648)
May 26 08:05:43 gateway dhclient[19805]: bound to 192.168.1.95 – renewal in 209 seconds.
May 26 08:06:05 gateway evebox: 2018-05-26 08:06:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:06:29 gateway dnsmasq-dhcp[16903]: DHCPRELEASE(enp2s0f0) 192.168.1.101 00:25:56:4a:ed:b5
May 26 08:06:31 gateway dnsmasq-tftp[16903]: file /var/lib/tftpboot/fp-net.cfg not found
May 26 08:06:40 gateway dnsmasq-tftp[16903]: file /var/lib/tftpboot/router.cfg not found
May 26 08:06:40 gateway dnsmasq-dhcp[16903]: DHCPDISCOVER(enp2s0f0) 00:25:56:4a:ed:b5
May 26 08:06:40 gateway dnsmasq-dhcp[16903]: DHCPOFFER(enp2s0f0) 192.168.1.101 00:25:56:4a:ed:b5
May 26 08:06:41 gateway dnsmasq-tftp[16903]: file /var/lib/tftpboot/fp-net.cfg not found
May 26 08:06:42 gateway dnsmasq-dhcp[16903]: DHCPREQUEST(enp2s0f0) 192.168.1.101 00:25:56:4a:ed:b5
May 26 08:06:42 gateway dnsmasq-dhcp[16903]: DHCPACK(enp2s0f0) 192.168.1.101 00:25:56:4a:ed:b5
Sorry, I don’t know what might be causing your problem. But…are you using a virtualised system? If so, what VM platform are you using and what NIC drivers is your NethServer using?
(I’m having problems of martian packets on a virtualised NethServer that using Omnios & KVM with virtio NICs).
I reinstalled NS. I have not applied any updates yet…just updated the network connections, installed basic firewall, and was setting up port forwarding when I got the following when saving the settings.
Reinstalled NS, applied the updates, configured multiwan with no errors or warning, but the symptoms are still the same. Very sporadic net access…
May 29 09:02:37 gateway kernel: IPv4: martian source 255.255.255.255 from 77.237.246.37, on dev enp3s0f1
May 29 09:02:39 gateway kernel: IPv4: martian source 255.255.255.255 from 217.71.204.119, on dev enp3s0f1
May 29 09:12:15 gateway shorewall: Compiling Martian Logging…
May 29 09:12:15 gateway shorewall: Setting up Martian Logging…
May 29 09:15:29 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:29 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:30 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:30 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:30 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:30 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:30 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:30 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:31 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:31 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:34 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:34 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:35 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:35 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:35 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:35 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:35 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:35 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:36 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:36 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:39 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:39 gateway kernel: IPv4: martian source 192.168.1.92 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:40 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:40 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:40 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:40 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:40 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:40 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:41 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 09:15:41 gateway kernel: IPv4: martian source 255.255.255.255 from 192.168.1.1, on dev enp3s0f0
May 29 11:23:39 gateway shorewall: Compiling Martian Logging…
May 29 11:23:39 gateway shorewall: Setting up Martian Logging…
May 29 11:59:09 gateway shorewall: Compiling Martian Logging…
May 29 11:59:09 gateway shorewall: Setting up Martian Logging…
May 29 12:00:03 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 29 12:00:03 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
May 29 12:01:51 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 29 12:01:51 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
Looking at a recurrent IP address I suspect a malware attack or infection.
You can check active processes, opened connections, logs and evebox/IPS for suspicious activity.
Problem isolated. We were able to replicate the issue using ClearOS as well. We then decided to troubleshoot the issue by process of elimination by removing one switch at a time from the network. Once we identified the switch, we were able to narrow it down to a RHAT server, provided by Oracle Corp.
We were using it to test their new application development tool.
I am suspecting malware but we haven’t looked into it yet. I am glad to post that its nothing to do with NS, and the problem is isolated.