Shorewall issues

You may try network recovery:

network-recovery

To set the role of a interface:

db networks setprop eth0 role red

Reload network config:

signal-event interface-update

See the documentation for more details.

Seems hetzner VPS uses a special network configuration:

I think it is related to using more network interfaces in the same network.

Thanks. I found the snippet in that Networking documentation that explains how to reset by deleting the interfaces from the DB, deleting any startup scripts, and then restarting the network. When I do that, the server is accessible again and I don’t see any shorewall spam in the console.

Once I run the command to set eth0 to red again, I start getting shorewall spam and nothing is accessible. Even doing shorewall clear in this case doesn’t seem to help the system become available again… So it seems that something in the network configuration is really messed up and the only way to make it work is to get Neth to not touch it by deleting all its configs.

How can I start fresh?

I don’t know hetzner but maybe you have a backup?
I assume you just need to fire up a new centos image and reinstall NethServer.

But if it works with green and public IP I’d work with that configuration. If you really need a second interface (red and green), please follow these steps.

Sorry for my quick last post. Was in a bit of a panic mode. I’ll give this a try and see if I can get things working reliably with a single WAN interface set to green.

I assume that what you mentioned above, as well as what the other Hetzner post is talking about, is that the Hetzner “private network” has a slightly strange configuration.

Once I get things working well with the Green WAN I will follow up with them about the specific configuration of this private network. If there is nothing conclusive from them then I will use the dummy interface as you mentioned.

As always, thanks for your support! :slight_smile:

1 Like

After lots of server restarts, I have things almost working at a minimal level. Here’s what I did:

  • Followed the instructions here to reset the interfaces in the db
  • rebooted the server
  • ran shorewall clear

at this point, I could now access Neth again, but my network was not configured.

I thought I saw in the logs somewhere that some network bridges were still set up for eth1 (the network interface I removed)

Then I did this:

  • Removed Docker
  • Removed VPN
  • Removed Firewall
  • Added my eth0 as a Green DHCP (it never worked when setting it as static)

Now I can access my server and it starts normally, but I notice that shorewall is failing to start:

-- Unit shorewall.service has begun starting up.
Jul 26 20:46:41 neth.mydomain.com shorewall[5552]: Compiling using Shorewall 5.1.10.2...
Jul 26 20:46:41 neth.mydomain.com shorewall[5552]: Processing /etc/shorewall/params ...
Jul 26 20:46:41 neth.mydomain.com shorewall[5552]: Processing /etc/shorewall/shorewall.conf...
Jul 26 20:46:41 neth.mydomain.com shorewall[5552]: Loading Modules...
Jul 26 20:46:42 neth.mydomain.com kernel: nf_log: can't load ipt_ULOG, conflicting nfnetlink_log already loaded
Jul 26 20:46:42 neth.mydomain.com kernel: ipt_ULOG: ULOG: fail to register logger.
Jul 26 20:46:42 neth.mydomain.com shorewall[5552]: ERROR: Unable to find tcstart file /etc/shorewall/modules (EOF)
Jul 26 20:46:42 neth.mydomain.com root[5875]: ERROR:Shorewall start failed
Jul 26 20:46:42 neth.mydomain.com systemd[1]: shorewall.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 26 20:46:42 neth.mydomain.com systemd[1]: Failed to start Shorewall IPv4 firewall.
-- Subject: Unit shorewall.service has failed

I haven’t done any custom configurations to Shorewall, and I think I have now removed all the applications that have done significant changes. So it comes back to my original question: How do I reset the shorewall configuration?

It seems there are issues with shorewall modules. Maybe the system has a customized kernel?

You may check your kernel version with uname -r

[root@testserver ~]# uname -r
3.10.0-1127.13.1.el7.x86_64

Did you update the system?
See also Firewall not start

Mine shows the same, but I did add the dummy module per the instructions. I guess that the firewall needs to be told about this as the other article mentions?

No, this is only needed if you use another kernel.
I’d try to make it work with one interface first. If everythings ok, add the dummy virtual interface.

I’ve removed the dummy module and still getting the same error. I searched google and I see a gist from you with the same error message… So how did you fix it? :slight_smile:

Ah, but this isn’t an error. I see the error a few lines down:
ERROR: Unable to find tcstart file /etc/shorewall/modules (EOF)

This gist was about ndpi, did you setup firewall rules?

I didn’t set up any firewall rules except for a port forward and whatever the VPN app sets up. I uninstalled both the Firewall app and the VPN app, so I assumed they would clean up after themselves. Is there some additional clean-up I need to do?

Please check if there are still entries in the db:

db fwrules show
db portforward show

To debug shorewall and hopefully find out more about the error (see also Shorewall docs):

shorewall restart -T

Reconfigure firewall for production after debugging:

signal-event firewall-adjust

As regards the static IP issue: (Warning! This may break your network again.)
Just go through the steps as described here.

If it doesn’t work you could try this one:

db networks setprop eth0 bootproto static onboot yes userctl no ipaddr <YOUR IPv4 ADDRESS> netmask 255.255.255.255 dns1 213.133.98.98 dns2 213.133.99.99 ipv6addr <IPv6 FROM SUBNET LIKE 2a01:4f8:0::1>/64 ipv6init yes ipv6_defaultgw fe80::1%eth0 ipv6_defaultdev eth0 role green

You need to replace <IPv4 ADDRESS> and <IPv6 FROM SUBNET LIKE 2a01:4f8:0::1> with the IPs from your configuration:

To get the actual network config:

ip a

Apply config:

signal-event interface-update

empty

Similar error with a bit more verbosity:

# shorewall restart -T
Compiling using Shorewall 5.1.10.2...
Processing /etc/shorewall/params ...
Processing /etc/shorewall/shorewall.conf...
Loading Modules...
   ERROR: Unable to find tcstart file /etc/shorewall/modules (EOF) at /usr/share/perl5/vendor_perl/Shorewall/Config.pm line 1561.
        Shorewall::Config::fatal_error('Unable to find tcstart file') called at /usr/share/perl5/vendor_perl/Shorewall/Config.pm line 6816
        Shorewall::Config::get_configuration(0, 0, 0, 0) called at /usr/share/perl5/vendor_perl/Shorewall/Compiler.pm line 698
        Shorewall::Compiler::compiler('script', '/var/lib/shorewall/.restart', 'directory', '', 'verbosity', 1, 'timestamp', 0, 'debug', ...) called at /usr/libexec/shorewall/compiler.pl line 144

Those lines seem to show:

    $val = "\L$config{TC_ENABLED}";

    if ( $val eq 'yes' ) {
        my $file = find_file 'tcstart';
        fatal_error "Unable to find tcstart file" unless -f $file;
        $globals{TC_SCRIPT} = $file;

and in my shorewall.conf file I see these seemingly relevant lines:

TC=
...
CLEAR_TC=Yes
...
TC_ENABLED=Yes
...
TC_EXPERT=No
...
TC_PRIOMAP="2 3 3 3 2 3 1 1 2 2 2 2 2 2 2 2"

Just to clarify, did you already execute signal-event firewall-adjust ?

Do the files exist and have content?

ls -l /etc/shorewall/tcstart /etc/shorewall/modules

These settings are ok and shorewall.conf is templated.

I didn’t before, but I just did it now and then checked again.

tcstart doesn’t exist, modules does exist

You may try to create the missing file /etc/shorewall/tcstart with following content:

# Cleanup current TC configuration
fireqos clear_all_qos

# Apply new TC configuration
fireqos start

Set permissions:

chmod 755 /etc/shorewall/tcstart

Restart firewall:

signal-event firewall-adjust

I did this and I still got this message

Jul 27 22:24:57 neth.mydomain.com shorewall[2317]: ERROR: Unable to find tcstart file /etc/shorewall/modules (EOF)
Jul 27 22:24:57 neth.mydomain.com systemd[1]: shorewall.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 27 22:24:57 neth.mydomain.com systemd[1]: Failed to start Shorewall IPv4 firewall.
Jul 27 22:24:57 neth.mydomain.com systemd[1]: Unit shorewall.service entered failed state.
Jul 27 22:24:57 neth.mydomain.com systemd[1]: shorewall.service failed.

However, when I ran shorewall restart -T it showed success. so then I did systemctl start shorewall and it started just fine.

I do notice these permissions on modules, is it correct?

-rw-r--r--. 1 root root 975 Jul 27 22:35 /etc/shorewall/modules
-rwxr-xr-x  1 root root 101 Jul 28 22:01 /etc/shorewall/tcstart

So… now the firewall is running again, and I’m getting the same usual spam even though I just have eth0 in ifconfig. However, I do notice that in my db networks show I have an extra entry:

[root@neth ~]# db networks show
eth0=ethernet
    bootproto=dhcp
    gateway=
    ipaddr=12.34.56.78
    netmask=255.255.255.255
    nslabel=
    role=green
ppp0=xdsl-disabled
    AuthType=auto
    FwInBandwidth=
    FwOutBandwidth=
    Password=
    name=PPPoE
    provider=xDSL provider
    role=red
    user=

You may try to configure the static IP, see Shorewall issues

Yes.

This is ok.

Thank you for all your help.

These days aren’t a good time for the system to go down so in a couple days I will give the static IP a try and report back.

In the end, this was actually a combination of 3 issues.

1. There was unnecessary shorewall spam at the console

It turns out I already raised this as a separate issue. I have put the answer in the other thread.

2. Hetzner requires static routes to allow the static IPs to work

Details (it's kind of long!)

Thanks to @mrmarkuz for figuring this one out too and for improvement from @filippo_carletti. Hetzner has this well documented here , but unfortunately it’s not so simple for us on NethServer. We will need to make some changes to the e-smith database to make sure the static route is being added so our static IP can work. Basically, it means that we can’t use the Web GUI for managing static IPs. It is really unfortunate that Hetzner designs it this way, and this makes their competitor Contabo more attractive for doing a simple NethServer install.

Nevertheless, to fix the routes we need to do the following:

  • Manually edit the e-smith database for your external interface (eth0 by default)
db networks setprop eth0 bootproto none onboot yes userctl no ipv6addr <IPV6_Address>::1/64 ipv6init yes ipv6_defaultgw fe80::1%eth0 ipv6_defaultdev eth0 role green hwaddr <Eth0_Mac_Address> ipaddr <IPV4_Address> netmask 255.255.255.255

Get your IPv4 and IPv6 addreses from your Hetzner control panel:
image

If you’re like me and IPv6 is new to you, then just grab the IPv6 that is shown in your control panel and put the AAAA:BBB:CCC:DDDD it in the script above exactly as I have shown. If you know what you’re doing, then you can pick your own sub-address.

  • Remove the gateway property from eth0
db networks delprop eth0 gateway
  • Add the static route through a command:
db routes set 0.0.0.0/0 static Description “default gw” Device eth0 Metric “” Router 172.31.1.1

or through the web interface:


Now is the time to cross your fingers and hope things work. Make sure you’re ready for some down time and debugging.

  • Trigger the interface update
signal-event interface-update

Things should work. If not, try rebooting your server before playing with any other configurations.

3. If the Nextcloud script in interface-update fails it will hang and prevent the operation from finishing which will leave the network down
In my case, the Nextcloud interface-update script failed because my Nextcloud configuration has the data drive on a network share. I’ve opened a separate issue to discuss that, and my current fix.

Edit: Updated method for setting static route based on suggestion from @filippo_carletti in this issue:

2 Likes