Nethserver-fail2ban-0.10 loves IPSET

[stephdl (Stéphane de Labrusse)]

Nethserver-fail2ban comes with a lot of good improvements, that will make it faster, better, I speak about IPSET.

Actually we used a custom json database to store the bans for the recidive jail,. This is the past, now we use shorewall-ipset as a banaction with FAILBAN-0.10.4.

We are in the fifth dimensions

We expect no issues, @giacomo and I, have worked hard on it…

After the Upgrade you could make some basic checks

• shorewall show dynamic must give back nothing more IP banned

• ipset -L -name will give back all enabled jail

• iptables -L you must find at the end the jails of fail2ban (something like f2b-sshd)

  DROP all – anywhere anywhere match-set f2b-apache-noscript src
  DROP all – anywhere anywhere match-set f2b-apache-overflows src

• ipset -L will gives you all bans in the sets (something like f2b-sshd), you will see the banned IP with a timeout before release.

• cockpit is expected to work (minor changes, the status switch of jails is removed, if no bans, you have a message to explain why)

• cockpit is protected by the pam-generic jail

• no changes in nethgui, but you could test about regressions

• check /var/log/fail2ban.log

as never a big upgrade of the engine…

yum upgrade nethserver-fail2ban --enablerepo=nethserver-testing

this is the GH issue

Issue: Fail2ban: Upgrade to fail2ban-0.10

[giacomo (Giacomo Sanchietti)]

I have already it on 4 productions server, no problems so far!
Also performances have drastically improved.
In our server the fail2ban restart took more than 1.30 minutes, now it restarts in less than 2 seconds

[saitobenkei]

@giacomo

In our server the fail2ban restart took more than 1.30 minutes, now it restarts in less than 2 seconds

I want it!

[stephdl (Stéphane de Labrusse)]

me on my two main servers…on one, 650 bans in recidive…they are well in their jail

[stephdl (Stéphane de Labrusse)]

Do I have mentioned that fail2ban-0.10 is ipv6 compatible, it is the latest trend

Some testers please, we have worked hard

[dnutan (Marc)]

`fail2ban-unban` command not found.

# locate fail2ban-unban
/etc/e-smith/events/actions/fail2ban-unban
/etc/e-smith/events/actions/nethserver-fail2ban-unban-whitelist
/etc/e-smith/events/nethserver-fail2ban-save/S10nethserver-fail2ban-unban-whitelist
/etc/e-smith/events/nethserver-fail2banUnBan-save/S10fail2ban-unban

fail2ban log shows this notice:

fail2ban.filtersystemd [17187]: NOTICE Jail started without ‘journalmatch’ set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.

Updating fail2ban showed some errors but it seems they are no more afterwards, so maybe there is nothing to worry about.

Dec  1 10:36:13 server systemd: Started Fail2Ban Service.
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,327 fail2ban.configreader   [20650]: ERROR   Found no accessible config files for 'action.d/shorewall-nethserver' under /etc/fail2ban
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,327 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,327 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'sshd'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,328 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,328 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'apache-auth'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,329 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,329 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'apache-badbots'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,330 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,330 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'apache-noscript'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,331 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,331 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'apache-overflows'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,335 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,335 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'apache-nohome'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,336 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,336 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'apache-fakegooglebot'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,337 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,337 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'apache-modsecurity'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,338 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,338 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'apache-shellshock'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,339 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,339 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'nginx-http-auth'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,341 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,341 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'nginx-botsearch'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,343 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,343 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'postfix'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,345 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,345 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'postfix-rbl'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,347 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,347 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'dovecot'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,348 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,348 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'sieve'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,350 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,350 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'postfix-sasl'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,351 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,351 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'mysqld-auth'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,353 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,353 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'recidive'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,354 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,354 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'pam-generic'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,355 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,356 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'httpd-admin'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,357 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,357 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'pam-generic-nethserver'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,358 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,358 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'postfix-ddos'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,359 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,359 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'postfix-sasl-abuse'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,360 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,360 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'apache-scan'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,361 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,361 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'phpmyadmin'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,362 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,362 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'dovecot-nethserver'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,363 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,363 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'nextcloud-auth'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,364 fail2ban.jailreader     [20650]: ERROR   Unable to read action 'shorewall-nethserver'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,364 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'rspamd'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,365 fail2ban.configreader   [20650]: ERROR   Found no accessible config files for 'filter.d/sshd-ddos' under /etc/fail2ban
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,365 fail2ban.jailreader     [20650]: ERROR   Unable to read the filter 'sshd-ddos'
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,365 fail2ban.jailsreader    [20650]: ERROR   Errors in jail 'sshd-ddos'. Skipping...
Dec  1 10:36:13 server fail2ban-server: 2019-12-01 10:36:13,367 fail2ban                [20650]: ERROR   Async configuration of server failed
Dec  1 10:36:13 server systemd: fail2ban.service: main process exited, code=exited, status=255/n/a
Dec  1 10:36:13 server fail2ban-client: 2019-12-01 10:36:13,405 fail2ban                [20653]: ERROR   Failed to access socket path: /var/run/fail2ban/fail2ban.sock. Is fail2ban running?
Dec  1 10:36:13 server systemd: fail2ban.service: control process exited, code=exited status=255
Dec  1 10:36:13 server systemd: Unit fail2ban.service entered failed state.
Dec  1 10:36:13 server systemd: fail2ban.service failed.
Dec  1 10:36:17 server esmith::event[20662]: Event: nethserver-fail2ban-update
Dec  1 10:36:17 server esmith::event[20662]: Migrating existing database configuration
(...)

Recidive detected in log by fail2ban.filter but not shown within ipset -L nor fail2ban-listban:

fail2ban.filter         [4248]: INFO    [recidive] Found 10.0.0.67 - 2019-12-01 11:16:48

If possible, on the manual specify that stats are not in real-time, as people will complain about stats mismatch.

Feature: not really necessary but on the unban list, don’t know if people will find it useful to list the cause of the ban (which jail(s) were matched by the IP address).

[stephdl (Stéphane de Labrusse)]

Thank

@dnutan

fail2ban-unban command not found.

yep removed by @giacomo maybe we can introduced again the wrapper, people are waiting for it.

now it is : fail2ban-client unban 1.1.1.1

@dnutan

fail2ban log shows this notice:

fail2ban.filtersystemd [17187]: NOTICE Jail started without ‘journalmatch’ set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.

these notices were already existing, I suspect a systemd issue, but the jails are workable

@dnutan

Updating fail2ban showed some errors but it seems they are no more afterwards, so maybe there is nothing to worry about.

yep we removed this custom action shorewall-nethserver, we go to base shorewall-ipset, it seems comprehensible. We could check maybe to hide this log noise

@dnutan

Recidive detected in log by fail2ban.filter but not shown within ipset -L nor fail2ban-listban :

you pointed a recidive found, not a recidive ban, so it is normal, after several recidive found, you have a recidive ban

@dnutan

If possible, on the manual specify that stats are not in real-time, as people will complain about stats mismatch.

stats works differently and are more accurate, before for each ban, we store the information inside a json file, but now we query directly the sqlite db, it is the real information, my custom json db was wrong, each restart of the service and you have the ban and unban IP added. Now the cron does the query at 23h45.

@dnutan

Feature: not really necessary but on the unban list, don’t know if people will find it useful to list the cause of the ban (which jail(s) were matched by the IP address).

yes this could be cool, but often it is recidive

[dnutan (Marc)]

@stephdl

We could check maybe to hide this log noise

Don’t waste time on it.

@stephdl

you pointed a recidive found , not a recidive ban , so it is normal, after several recidive found, you have a recidive ban

guilty as charged

[stephdl (Stéphane de Labrusse)]

Fail2ban-unban is a wrapper to unban

/usr/bin/fail2ban-unban is back

[giacomo (Giacomo Sanchietti)]

@dnutan

fail2ban-unban command not found.

You can obtain the same things with fail2ban-client command, there is no need to re-implement something which already exists.

@dnutan

If possible, on the manual specify that stats are not in real-time, as people will complain about stats mismatch.

I agree, the UI should state something like “stats updated at xxxx”.
I’d like to release the existing and move this feature (with eventually the unban command) to next release.

[stephdl (Stéphane de Labrusse)]

[stephdl (Stéphane de Labrusse)]

by the way, fail2ban0.10 is released

[giacomo (Giacomo Sanchietti)]

I’d change the label to “Last statistics update”.

@dnutan your favorite script is back https://github.com/NethServer/dev/issues/5973 ;D

[stephdl (Stéphane de Labrusse)]

Please verify the statistics date on GH

Issue: Fail2ban: Display the date of the last statistic update