Setup:
3 inbound Internet connections setup in ‘Balance’ mode(50, 30, 20) with One static and two DHCP.
Everything has been working fine for over a month now in this mode until this morning when two of the links were showing as ‘DOWN’. I was curious and decided to check the connections by connecting them directly to a computer, and the connection was up. Same with the other one.
I tried doing some troubleshooting by ‘Releasing Role’ on the two that were not working. I get the following error when I try to do that.
Task completed with errors
Template /¦ #69 (exit status 1)
expansion of /¦ failed
As of now only one connection is up and internet is sporadic (30 secs on and 20 seconds off).
–Tried re-installing Firewall and still the same symptoms…
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/actions
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/hosts
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/interfaces
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/maclist
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/mangle
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/masq
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/modules
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/nat
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/policy
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/providers
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/rtrules
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/rules
May 26 07:59:06 gateway esmith::event[18944]: expanding /etc/shorewall/shorewall.conf
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/stoppedrules
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/tcinterfaces
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/tcpri
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/tunnels
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/zones
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/snat
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/findgw
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/shorewall/helpers
May 26 07:59:07 gateway esmith::event[18944]: expanding /etc/firehol/fireqos.conf
May 26 07:59:07 gateway esmith::event[18944]: Action: /etc/e-smith/events/actions/generic_template_expand SUCCESS [0.624607]
May 26 07:59:07 gateway systemd: Reloading.
May 26 07:59:08 gateway sssd[be[vmakol.lan]]: Backend is online
May 26 07:59:08 gateway FireQOS[19293]: Cleared all QOS on all interfaces
May 26 07:59:09 gateway FireQOS[19341]: QoS applied ok (39 tc commands applied)
May 26 07:59:09 gateway root: Shorewall reloaded
May 26 07:59:09 gateway esmith::event[18944]: [NOTICE] Shorewall restart
May 26 07:59:09 gateway esmith::event[18944]: Action: /etc/e-smith/events/nethserver-firewall-base-save/S89nethserver-shorewall-restart SUCCESS [2.176035]
May 26 07:59:09 gateway systemd: Reloading.
May 26 07:59:09 gateway esmith::event[18944]: [INFO] service lsm restart
May 26 07:59:09 gateway systemd: Stopping LSM is the link status monitor…
May 26 07:59:09 gateway systemd: Started LSM is the link status monitor.
May 26 07:59:09 gateway systemd: Starting LSM is the link status monitor…
May 26 07:59:09 gateway esmith::event[18944]: Action: /etc/e-smith/events/actions/adjust-services SUCCESS [0.418427]
May 26 07:59:09 gateway esmith::event[18944]: Event: nethserver-firewall-base-save SUCCESS
May 26 07:59:09 gateway esmith::event[18943]: Action: /etc/e-smith/events/firewall-adjust/S20firewall-adjust SUCCESS [3.554889]
May 26 07:59:09 gateway esmith::event[18943]: Event: firewall-adjust SUCCESS
May 26 07:59:57 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 26 07:59:57 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 07:59:57 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
May 26 07:59:57 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:00:00 gateway esmith::event[20224]: Event: wan-uplink-update down SC root 0 0 0 0 0 0 0 0 unknown 1527314400
May 26 08:00:00 gateway esmith::event[20223]: Event: wan-uplink-update down Hola root 0 0 0 0 0 0 0 0 unknown 1527314400
May 26 08:00:00 gateway esmith::event[20222]: Event: wan-uplink-update up AeroM root 0 0 0 0 0 0 0 0 unknown 1527314400
May 26 08:00:00 gateway Shorewall[20236]: Enabling device enp2s0f1
May 26 08:00:01 gateway Shorewall[20243]: Disabling device enp3s0f1
May 26 08:00:01 gateway Shorewall[20250]: Disabling device enp3s0f0
May 26 08:00:01 gateway esmith::event[20222]: Action: /etc/e-smith/events/wan-uplink-update/S50nethserver-shorewall-wan-update SUCCESS [0.331084]
May 26 08:00:01 gateway esmith::event[20222]: Event: wan-uplink-update SUCCESS
May 26 08:00:05 gateway evebox: 2018-05-26 08:00:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 61
May 26 08:01:01 gateway esmith::event[20223]: Action: /etc/e-smith/events/wan-uplink-update/S50nethserver-shorewall-wan-update SUCCESS [60.404164]
May 26 08:01:01 gateway esmith::event[20223]: Event: wan-uplink-update SUCCESS
May 26 08:01:01 gateway esmith::event[20224]: Action: /etc/e-smith/events/wan-uplink-update/S50nethserver-shorewall-wan-update SUCCESS [60.406025]
May 26 08:01:01 gateway esmith::event[20224]: Event: wan-uplink-update SUCCESS
May 26 08:01:01 gateway systemd: Started Session 21 of user root.
May 26 08:01:01 gateway systemd: Starting Session 21 of user root.
May 26 08:01:05 gateway evebox: 2018-05-26 08:01:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:01:10 gateway dhclient[19805]: DHCPREQUEST on enp3s0f1 to 192.168.1.1 port 67 (xid=0x7a82b648)
May 26 08:01:10 gateway dhclient[19805]: DHCPACK from 192.168.1.1 (xid=0x7a82b648)
May 26 08:01:11 gateway chronyd[14790]: Selected source 81.19.96.148
May 26 08:01:12 gateway dhclient[19805]: bound to 192.168.1.95 – renewal in 269 seconds.
May 26 08:01:42 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 26 08:01:42 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:01:42 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
May 26 08:01:42 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:02:05 gateway evebox: 2018-05-26 08:02:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:03:05 gateway evebox: 2018-05-26 08:03:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:03:27 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 26 08:03:27 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:03:27 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
May 26 08:03:27 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:04:05 gateway evebox: 2018-05-26 08:04:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:05:05 gateway evebox: 2018-05-26 08:05:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:05:13 gateway kernel: IPv4: martian source 23.247.54.44 from 224.168.1.11, on dev enp2s0f0
May 26 08:05:13 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:05:13 gateway kernel: IPv4: martian source 23.247.54.44 from 0.168.1.11, on dev enp2s0f0
May 26 08:05:13 gateway kernel: ll header: 00000000: 18 a9 05 65 64 64 08 00 27 1d 95 ae 08 00 …edd…’…
May 26 08:05:41 gateway dhclient[19805]: DHCPREQUEST on enp3s0f1 to 192.168.1.1 port 67 (xid=0x7a82b648)
May 26 08:05:41 gateway dhclient[19805]: DHCPACK from 192.168.1.1 (xid=0x7a82b648)
May 26 08:05:43 gateway dhclient[19805]: bound to 192.168.1.95 – renewal in 209 seconds.
May 26 08:06:05 gateway evebox: 2018-05-26 08:06:05 (evefileprocessor.go:176) – Total: 0; last minute: 0; EOFs: 60
May 26 08:06:29 gateway dnsmasq-dhcp[16903]: DHCPRELEASE(enp2s0f0) 192.168.1.101 00:25:56:4a:ed:b5
May 26 08:06:31 gateway dnsmasq-tftp[16903]: file /var/lib/tftpboot/fp-net.cfg not found
May 26 08:06:40 gateway dnsmasq-tftp[16903]: file /var/lib/tftpboot/router.cfg not found
May 26 08:06:40 gateway dnsmasq-dhcp[16903]: DHCPDISCOVER(enp2s0f0) 00:25:56:4a:ed:b5
May 26 08:06:40 gateway dnsmasq-dhcp[16903]: DHCPOFFER(enp2s0f0) 192.168.1.101 00:25:56:4a:ed:b5
May 26 08:06:41 gateway dnsmasq-tftp[16903]: file /var/lib/tftpboot/fp-net.cfg not found
May 26 08:06:42 gateway dnsmasq-dhcp[16903]: DHCPREQUEST(enp2s0f0) 192.168.1.101 00:25:56:4a:ed:b5
May 26 08:06:42 gateway dnsmasq-dhcp[16903]: DHCPACK(enp2s0f0) 192.168.1.101 00:25:56:4a:ed:b5