Shorewall doesn't start after reboot - urgent!

flatspin · November 18, 2021, 11:00am

NethServer Version: 7.2.2009
Module: firewall

Hi friends,

I updated my proxmoxserver this morning and rebooted it. Before I installed some updates on Nethserver.
Since then, shorewall doesn’t start automatically with systemboot.
When starting shorewall manually, it shows running, but I can’t get the users and groups in cockpit.

Nethupdates this morning:

Nov 18 07:56:49 Updated: clamav-filesystem-0.103.4-1.el7.noarch
Nov 18 07:56:50 Updated: clamav-lib-0.103.4-1.el7.x86_64
Nov 18 07:56:50 Updated: clamav-update-0.103.4-1.el7.x86_64
Nov 18 07:56:50 Updated: httpd-tools-2.4.6-97.el7.centos.2.x86_64
Nov 18 07:56:51 Updated: httpd-2.4.6-97.el7.centos.2.x86_64
Nov 18 07:56:51 Updated: 1:mod_ssl-2.4.6-97.el7.centos.2.x86_64
Nov 18 07:56:52 Updated: clamav-0.103.4-1.el7.x86_64
Nov 18 07:56:52 Updated: clamd-0.103.4-1.el7.x86_64

Shorewall status:

[root@nethserver ~]# systemctl status shorewall -l
● shorewall.service - Shorewall IPv4 firewall
   Loaded: loaded (/usr/lib/systemd/system/shorewall.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/shorewall.service.d
           └─blacklist.conf
        /usr/lib/systemd/system/shorewall.service.d
           └─nethserver-firewall-base.conf
   Active: failed (Result: exit-code) since Thu 2021-11-18 11:44:03 CET; 3min 16s ago
  Process: 1530 ExecStart=/usr/sbin/shorewall $OPTIONS start $STARTOPTIONS (code=exited, status=143)
  Process: 1360 ExecStartPre=/usr/share/nethserver-blacklist/load-ipsets (code=exited, status=0/SUCCESS)
 Main PID: 1530 (code=exited, status=143)

Nov 18 11:44:02 nethserver.jeckel.local shorewall[1530]: bye...
Nov 18 11:44:02 nethserver.jeckel.local shorewall[1530]: IPv4 Forwarding Enabled
Nov 18 11:44:02 nethserver.jeckel.local shorewall[1530]: Processing /etc/shorewall/restored ...
Nov 18 11:44:02 nethserver.jeckel.local shorewall[1530]: done.
Nov 18 11:44:02 nethserver.jeckel.local shorewall[1530]: Shorewall restored from /var/lib/shorewall/restore
Nov 18 11:44:03 nethserver.jeckel.local shorewall[1530]: /usr/share/shorewall/lib.common: line 93:  2883 Terminated                                                                                    $SHOREWALL_SHELL $script $options $@
Nov 18 11:44:03 nethserver.jeckel.local systemd[1]: shorewall.service: main process exited, code=exited, status=143/n/a
Nov 18 11:44:03 nethserver.jeckel.local systemd[1]: Failed to start Shorewall IPv4 firewall.
Nov 18 11:44:03 nethserver.jeckel.local systemd[1]: Unit shorewall.service entered failed state.
Nov 18 11:44:03 nethserver.jeckel.local systemd[1]: shorewall.service failed.

I need help, because sogo is our central calendar and we need it and it cant authenticate without users and groups.

Any thoughts about this??

TIA Ralf

Andy_Wismer · November 18, 2021, 12:36pm

@flatspin

Hi

Are you using LDAP or AD?
Is the account provider running?

The list of users is NOT dependent on Shorewall running, it depends on your Account Provider.
SoGo needs the Account Provider, otherwise it doesn’t have any users for mail / calendar or adressbooks…
SoGo will need Shorewall to get or send mail, but first see that you have users and groups!

BTW, I also had updates on my Proxmox at home - including a new kernel and a reboot.
Also for about half of my clients, I’ve done the Proxmox update. My clients had no issues so far.

My 2 cents
Andy

flatspin · November 18, 2021, 2:05pm

Hi Andy,

thanks for reply. Yes SSSD is running.

dnutan · November 18, 2021, 2:09pm

Does the /var/log/shorewall-init.log* has more details on the failed start?

dnutan · November 18, 2021, 2:23pm

or journalctl -u shorewall

flatspin · November 18, 2021, 2:42pm

That was a great hint!! Thanks!

I found iptables-restore v1.4.21: Set geo-whitelist doesn't exist.

I disabled geoip in threat shield and restarted shorewall.
Now all users are back in cockpit and sogo authentication works again.

@stephdl and @filippo_carletti is there a bug in geoip?

stephdl · November 18, 2021, 2:51pm

created without condition by the load-geoips, you have to dive better or during the update to understand why

github.com

NethServer/nethserver-blacklist/blob/master/root/usr/share/nethserver-blacklist/load-geoips#L88

    
      
              debug "Resetting ipset $j"
              /sbin/ipset $quiet $action $j
          done
          
          
if [ "$STATUS" == "disabled" ]; then
              debug "Black list disabled, exit after clean"
              exit 0
          fi
          
          
debug "Creating global whitelist"
          /sbin/ipset create -exist geo-whitelist hash:net
          if [ -f $BLDIR/whitelist.global ]; then
              debug "Loading global whitelist"
              cat $BLDIR/whitelist.global | sed '/^#/d; /^$/d' | sed "s/^/add geo-whitelist /" | /sbin/ipset restore
          fi
          
          
for l in $LISTS; do
              file=''
              type=''
              file_netset="$BLDIR/$l".netset
              file_ipset="$BLDIR/$l".ipset

flatspin · November 18, 2021, 4:25pm

o.k. I did the create command manually and the enabled deoip again.
Now everything works again. So no bug.

But what could have happend, that it lost the entries (geo-whitelist,geo-cn and geo-ru)?

stephdl · November 18, 2021, 4:28pm

/var/log/messages could maybe explain us better, search before and after the update, when you trigger the shorewall restart it drops normally all ipset before to build them again IIRC

flatspin · November 18, 2021, 4:39pm

Sorry I have to crawl back. After reboot same problem again.
So the set are not saved. Should I have done ipset save after create?
I’ll disable geoip for the moment until I’ve some time to dive in it.

Thanks for your help Stephan!

stephdl · November 18, 2021, 7:48pm

good new I can reproduce

stephdl · November 18, 2021, 8:18pm

add to /etc/systemd/system/shorewall.service.d/blacklist.conf
ExecStartPre=/usr/share/nethserver-blacklist/load-geoips

this is the example

# /etc/systemd/system/shorewall.service.d/blacklist.conf
[Service]
ExecStartPre=/usr/share/nethserver-blacklist/load-ipsets
ExecStartPre=/usr/share/nethserver-blacklist/load-geoips

then systemctl daemon-reload

try the reboot then, the ipset will be created after the reboot, think to enable again the geoip before to reboot

stephdl · November 18, 2021, 8:22pm

flatspin · November 19, 2021, 7:27am

Good morning Stephane,

tried your solution and I confirm that it works.
After reboot geoips are loaded and shorewall works as expected.

I marked this one now as solution, as my one was only a workaround.

Thanks a lot!!