Broken 7.6-1810 - how to reset shorewall

This morning, I was logged into the server over SSH, from the local network. Checking logs and manually adding some IPs to the drop list (this server has been getting a lot of attempts to log into the IMAP server from remote - mainly Chinese systems.

Quite suddenly, I found my terminal was frozen. A machine that I was doing Windows updates started reporting no internet connection. At first I thought the internet connection had gone down (even though my laptop was on the local net), but restarting the router did nothing. I then found that I could not access the server at all, other than from the console (fortunately, there is a KVM that allows me to connect direct to it). I tried restarting the server. No change. DHCP was still working, but I could not ping the server, connect to the web interface or use SSH to connect. From the server console, I could not connect to other machines in the network, or ping any external sites.

Since the only thing I was doing when this happened was reading logs and entering Shorewall commands, I assume that I must have upset Shorewall to cause the problem. Is there a way that I can reset Shorewall to defaults and see if that clears the problem? Going through the Shorewall docs hasn’t helped much so far, and I’d prefer to repair rather than have to set up another install and restore the backups.

With

shorewall clear

you can disable shorewall to check if it’s a shorewall problem.

To restart and reconfigure the firewall:

signal-event firewall-adjust

Could it be fail2ban is blocking? To list bans execute:

fail2ban-listban

1 Like

Thanks. I’ll try that when I get back to the site tomorrow. I did wonder about fail2ban, but didn’t know how to check it from the command line.

Its one thing that I rather like SME for - I can do most reconfigaration from its console, using the ncurses interface. Not something I can do (or even check) easily with Nethserver.

Being a log time user of SME, I rarely used the ncurses interface, but you make me think about it: it may have value for some “emergency” situations.

@dev_team , do you think we could evaluate a text based system for emergency repairs (and troubleshooting)?

It’s in the list of desiderata since ns6 :grimacing:

Jokes apart I think there’s a reason because we didn’t it yet. “Emergency situations” should lead to bug fixes not to emergency tools implementation.

If something is broken and you don’t know how to fix it ask for Support. We have a great support here!

1 Like

Shorewall check/restart
Launch the magic/hidden action to reconfigure
Reconfigure the network

It could be fun, yes

:slight_smile: I can’t say I used it often. But its a good quick basic configuration when first installing an SME system, and also for a quick way of setting up hardware changes (NIC failed, replace it and use the configuration interface to set up the new one…). While I can do a lot of that type of work from the command line, I don’t yet know Nethserver well enough to do that same. Which means that things can get difficult if I can’t connect to a green interface to use the dashboard.

I have to agree - community support here is very good indeed. But I do think that something like the SME configuration interface has a place. It provides a quick way of configuring things and making quick repairs. It also gives something that does not require a network connection, which is something I have missed several times in Nethserver.

A few months ago, I changed one NIC in a test Nethserver install. I made a mistake in how I tried to reconfigure the green interface and ended up with a machine I could no longer access from the network. While I did eventually find how CentOS 7 sets network interfaces, I must have missed something, since even after manually editing the network files, I could not connect to the web interface. Not a bug, rather my fat fingers…

In that instance, it was quicker just to reinstall, but in the case of a production server, I would really like a something that allows me to make those sort of changes without having to do a new install and restore from the backup.

:slight_smile: That said, doing a restore from backup is something I should do anyway - I never consider a backup valid until I know I can restore it…

1 Like

:slight_smile: its the reconfigure the network stage I’m a bit concerned about.

This is an interesting scenario. To recover it, a signal-event interface-update is not enough.

So the “emergency tool” here means: I want to configure a temporary network interface from the console with a ncurses-based interface. Do I understand well?

I guess recovering a bad network config is a common mistake and such tool can be useful…

What do you think?

1 Like

That would certainly be useful. While I’m almost exclusively a Linux user, I graduated from Slackware to Arch, which I’ve been running for several years. It took me a while to find where Cent OS stores network configuration information. Even though I was then able to edit the files manually, I must have been missing something, because I still couldn’t connect to the server. So something that would enable basic configuration to be done from the local console would certainly be something I would be interested in.

Finally back on site. shorewall clear allowed me to ssh back into the server, and also opened up the interface to the internet. Great start…

When I used signal-even firewall-adjust, things don’t look so rosy - my ssh terminal session locked up instantly. I tried fail2ban-listban, which showed all the jails as empty. But at the end of that, there was a very long list of banned IPs. I think that must be causing main problem. I’ve not yet found where shorewall stores its list of banned IPs. I’m hoping if I can find that, I may be able to clear it and get the system back to normal.

Can anyone tell me where Shorewall stores its list of exclusions? If nothing else, I would like to be able to clear that list manually when I need to .

A console wizard for reconfigure network could ease at lot of user cases:

  • Change of network card (for upgrade or fault)
  • Change of mainboard
  • Change of hypervisor
  • Upgrade of hypervisor (if the interface, interface driver or macaddress cannot be reproduced in newer version)
  • Change of network structure due to broken external devices (switches with VLAN setup)

A post was split to a new topic: Basic Network configuration recovery tool

:frowning: Oh well, back to the start…

After using shorewall clear I was able to get back to the server with SSH and use the server manager web interface. The server was able to ping outside sites, so both the outgoing connection and DNS was functioning. What I didn’t realise until a bit later was that nothing on the internal network could get out to the internet. From my laptop, I could ping other devices on the internal netwwork, but I couldn’t ping anything out of the local network.

I also found that as soon as I used signal-event firewall-adjust the system went back to not being contactable from the internal network, and was also unable to contact the internet.

Running fail2ban-listban listed all the fail2ban jails, each showing no current bans, but that was followed by a long list of blocked IPs. Some of them at least, I set manually using shorewall, but there were far more than I ever set to be dropped. Thinking that shorewall might be blocking things due to running out of resources, I would have liked to clear the blocked list, but haven’t so far found a command to do so.

Since I needed to get the system back up, I ended up doing a fresh install and loading the configuration backup from early Monday morning. I’ve used a fresh hard drive, and will pull the old drive tomorrow morning in the hope of working out the problem. I’ve not yet been able to reload the most recent data backup - while the config backup pulled in a lot of things and set up most things correctly, I had to manually configure NFS, so other than copy the data backup to a USB drive, I’ve not been able to do much with it. I’ll start a new thread regarding the disaster recovery procedures, but for the moment, the system is running and capable of both sending and receiving email.

Can anyone tell me what command would be needed to clear the blocked IP list? If I can do that, I hope to find that the connectivity problem goes away. I’m still not sure how the list of blocked IPs got as long as it did, and I’d like to be in a position to clear it if necessary in future.

See [Solved] How to reset the database used by fail2ban? - #2 by stef

1 Like

Thanks. That’s useful, but what I’m looking for in this case is a way of clearing Shorewall itslef. The system that is having problems has been under quite heavy attack, much of which hasn’t come from multiple IPs in a single net. As a result, the attacks aren’t being blocked by fail2ban. So as well as blocks put in place by fail2ban, I’ve used

shorewall drop from <IP or net>

I suspect that the problem the system had was at least in part because I’d added a number of individual bans. They are stored somewhere, but I’ve not yet been able to find a way to clear them. I’d like to test the theory by preventing the existing bans being reloaded when the system is restarted.

I’m also looking for alternate methods of defending against the attacks - I’ve set up logwatch on the system. Just as an example, here is the dovecot section of the PAM area from last night:

dovecot: Authentication Failures: [ken@xxxxxx.co.uk](mailto:ken@xxxxx.co.uk): 82 Time(s) ken: 57 Time(s) [admin@xxxxx.co.uk](mailto:admin@xxxx.co.uk): 48 Time(s) info: 5 Time(s) [info@xxxxxx.co.uk](mailto:info@xxxxxx.co.uk): 5 Time(s) [paul@xxxxx.co.uk](mailto:paul@xxxxxxx.co.uk): 1 Time(s)

While passwords are reasonably secure, I’m concerned with the sheer level of the attacks and the fact that many come from IPs very close together.

Any suggestions for tackling this would be very useful…

You could investigate shorewall and iptables documentation.
Another way found with trial and error method (which might not be the proper one or recommended) could be:

shorewall stop
mv /var/lib/shorewall/.iptables-restore-input{,.bkp}    # mv instead of rm, just in case
mv /var/lib/shorewall/.dynamic{,.bkp}    # mv instead of rm, just in case
shorewall start    # or signal-event firewall-adjust

Thanks. I’ll give that a try. I’m reading the Shorewall docs, but its fairly slow going…