Since upgrade to 7.5, no more fail2ban recidive jail?

NethServer Version: 7.5
Module: fail2ban

I upgraded my Neth server yesterday from 7.4 to 7.5. Except for needing to manually restart the httpd-admin service, everything seemed to work just fine. But I’m noticing a lot of fail2ban emails on what look like some pretty familiar IP addresses (or at least networks), and they seem to be coming up over and over again. Investigation appears to show that I don’t have a recidive jail any more.

[root@neth ~]# config show fail2ban
fail2ban=service
    ApacheAuth_status=true
    ApacheBadbots_status=true
    ApacheBotsearch_status=false
    ApacheFakegooglebot_status=true
    ApacheModsecurity_status=true
    ApacheNohome_status=true
    ApacheNoscript_status=true
    ApacheOverflows_status=true
    ApachePhpMyAdmin_status=true
    ApacheScan_status=true
    ApacheShellshock_status=true
    Apache_MaxRetry=
    BanAction=shorewall-nethserver
    BanLocalNetwork=disabled
    BanTime=1800
    CustomDestemail=admin@familybrown.org
    Dovecot_MaxRetry=
    Dovecot_status=true
    EjabberAuth_status=true
    Ejabber_MaxRetry=
    FindTime=900
    HttpdAdmin_MaxRetry=
    HttpdAdmin_status=true
    IgnoreIP=
    LogLevel=INFO
    Mail=enabled
    MailJailState=disabled
    MaxRetry=3
    MysqldAuth_status=true
    Mysqld_MaxRetry=
    Nextcloud_MaxRetry=
    Nextcloud_status=true
    NginxBotSearch_status=true
    NginxHttpAuth_status=true
    Nginx_MaxRetry=
    OpenVpnAuth_status=true
    OpenVpn_MaxRetry=
    Owncloud_MaxRetry=
    Owncloud_status=true
    PamGeneric_MaxRetry=
    PamGeneric_status=true
    PostfixRbl_status=true
    Postfix_MaxRetry=
    Postfix_status=true
    Recidive_MaxRetry=
    Recidive_Perpetual=disabled
    Recidive_status=true
    Roundcube_MaxRetry=
    Roundcube_status=true
    Sieve_MaxRetry=
    Sieve_status=true
    SogoAuth_status=true
    Sogo_MaxRetry=
    SshdDdos_status=true
    Sshd_MaxRetry=
    Sshd_status=true
    Urbackup_MaxRetry=
    Urbackup_status=true
    Vsftpd_MaxRetry=
    Vsftpd_status=true
    status=enabled

This says that Recidive_status is enabled, but fail2ban-client doesn’t list it as an available jail:

[root@neth ~]# fail2ban-client status
Status
|- Number of jail:	23
`- Jail list:	apache-auth, apache-badbots, apache-fakegooglebot, apache-modsecurity, apache-nohome, apache-noscript, apache-overflows, apache-scan, apache-shellshock, dovecot, dovecot-nethserver, httpd-admin, mysqld-auth, nextcloud-auth, pam-generic, pam-generic-nethserver, phpmyadmin, postfix, postfix-ddos, postfix-rbl, sieve, sshd, sshd-ddos

If I do fail2ban-listban, I get more information, but again no indication of a recidive jail:

[root@neth ~]# fail2ban-listban

If you want more information on a jail, do : fail2ban-client status {JailName}

Status of Jails
---------------
 apache-auth Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 apache-badbots Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 apache-fakegooglebot Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 apache-modsecurity Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 apache-nohome Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 apache-noscript Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 apache-overflows Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 apache-scan Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 apache-shellshock Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 dovecot Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 dovecot-nethserver Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 httpd-admin Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 mysqld-auth Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 nextcloud-auth Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 pam-generic Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 pam-generic-nethserver Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 phpmyadmin Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 postfix Jail enabled
    - Currently banned: 2     - Total banned after service start: 30
    - Banned IP: 208.53.48.218 108.60.195.213
 postfix-ddos Jail enabled
    - Currently banned: 0     - Total banned after service start: 4
    - Banned IP: 
 postfix-rbl Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 sieve Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 sshd Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 
 sshd-ddos Jail enabled
    - Currently banned: 0     - Total banned after service start: 0
    - Banned IP: 

List of all banned IP: 
Shorewall 5.1.10.2 Chain dynamic at neth.familybrown.org - Sat Jun 23 21:13:17 EDT 2018

Counters reset Sat Jun 23 08:09:41 EDT 2018

Chain dynamic (8 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      *       208.53.48.218        0.0.0.0/0           
    0     0 DROP       all  --  *      *       108.60.195.213       0.0.0.0/0           

/etc/fail2ban/filter.d/recidive.conf is present. However, /etc/fail2ban/jail.local seems to be the problem:

#recidive not used on this server

Why’s that? If I take a look at /etc/e-smith/templates/etc/fail2ban/jail.local/10recidive (BTW, it seems pretty odd that all the fragments in this directory start with 10), it starts with

{
return "\n#recidive not used on this server"
           unless (-f '/var/log/fail2ban.log');

…and though my understanding of Perl is nearly nonexistent, I think that means that what I’m seeing in my jail.local file should only be there if /var/log/fail2ban.log isn’t present.

[root@neth log]# ll /var/log/fail2ban.log
-rw------- 1 root root 115112 Jun 23 21:12 /var/log/fail2ban.log

Hmmm. Maybe the log file just wasn’t there when the template was expanded. A signal-event nethserver-fail2ban-update will fix that, right? Nope, I do that, and jail.local still has the same line about recidive.

Am I missing something? Or is this a bug?

Hi @danb35,

Did you enable “Recidive jail is perpetual” in Fail2Ban -> Configuration -> Advanced?

No:


…but Recidive itself is enabled, and from the template fragment, it should work just fine without having to be perpetual. But let’s try it:

[recidive]
enabled   = true
logpath   = /var/log/fail2ban.log
maxretry  = 6
banaction = shorewall-nethserver
bantime   = -1
findtime  = 84600

Yeah, I’m pretty sure that’s a bug.

Before this option, “Recidive jail is perpetual”, after a while (I think after “Ban time”), all bad IPs, as you said, “… seem to be coming up over and over again.”.

EDIT:

Fail2ban — NethServer 7 Final

“Recidive jail is perpetual
When an IP goes several time in jail, the recidive jail bans it for a much longer time. If enabled, it is perpetual.”

Normal behavior (i.e., the way it worked pre-7.5, and IMO the way it should work):

  • After N failures on a particular server, $IP is banned for $BanTime (which defaults to 30 minutes), and then is allowed to reconnect.
  • After X bans of the same IP within Y time, $IP is deemed a recidivist, and is now banned for a much longer time (one week by default, or 336 x $BanTime if $BanTime has been set). Once that time expires, $IP is allowed to reconnect.

With “Recidive jail is perpetual” checked, though, it works like this:

  • After N failures on a particular server, $IP is banned for $BanTime (which defaults to 30 minutes), and then is allowed to reconnect.
  • After X bans of the same IP within Y time, $IP is banned permanently.

Not the same thing, and not really the desired behavior. The problem is that until I checked the “perpetual” box, the recidive jail was disabled completely (despite the Enabled property being set).

That would have been a simple enough bug, but now there’s a wrinkle. I went back to the server manager and unchecked the perpetual box. Hit Submit again, and now the recidive jail is listed as it should be. So, now working for me, but the failure remains unexplained. I’ll chalk it up to an upgrade bug, but it does still seem like a bug.

I don’t remember having this problem.
I’m sure @stephdl will check this and will give the proper answer.

BR,
Gabriel

could be written also

return "\n#recidive not used on this server"
           if  ( ! -f '/var/log/fail2ban.log');

in short :

print '# comment' if '/var/log/fail2ban.log' doesn't exist

or

print '# comment' unless '/var/log/fail2ban.log' exists

I do not know much language with unless I love it :smiley:

Why to do this, in fact if you start one jail without the relevant log, then the entire fail2ban service won’t start…hence I protect each jail

The question now is why the fail2ban’s log was not there when you expanded the template the last time…Like you saw yourself, when you set the recidive perpetual, the recidive jail appeared in the configuration file…so there is no bug.

After each time you save the configuration in the fail2ban panel or after each runlevel-adjust (after each nethserver-*.rpm installation) the fail2ban templates are expanded and the service is restarted.

to debug

ll /var/log/fail2ban*

check in /var/log/fail2ban.log what is the first lines and when they appeared…try to understand for what they were relevant ???

Unless the upgrade took place right in the middle of log rotation or some such, it was there. And signal-event nethserver-fail2ban-update, at least twice (after confirming the log file was there), didn’t change it.

[root@neth ~]# ll /var/log/fail2ban*
-rw------- 1 root root 175226 Jun 24 02:05 /var/log/fail2ban.log
-rw------- 1 root root 237068 May 27 09:07 /var/log/fail2ban.log-20180527
-rw------- 1 root root 202822 Jun  4 04:13 /var/log/fail2ban.log-20180604
-rw------- 1 root root  69256 Jun 10 09:18 /var/log/fail2ban.log-20180610
-rw------- 1 root root 262845 Jun 18 06:38 /var/log/fail2ban.log-20180618
[root@neth ~]# head /var/log/fail2ban.log
2018-06-23 04:06:23,056 fail2ban.server         [10577]: INFO    Changed logging target to /var/log/fail2ban.log for Fail2ban v0.9.7
2018-06-23 04:06:23,057 fail2ban.database       [10577]: INFO    Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'
2018-06-23 04:06:23,060 fail2ban.jail           [10577]: INFO    Creating new jail 'sshd'
2018-06-23 04:06:23,119 fail2ban.jail           [10577]: INFO    Jail 'sshd' uses systemd {}
2018-06-23 04:06:23,140 fail2ban.jail           [10577]: INFO    Initiated 'systemd' backend
2018-06-23 04:06:23,142 fail2ban.filter         [10577]: INFO    Set maxRetry = 3
2018-06-23 04:06:23,145 fail2ban.filter         [10577]: INFO    Set jail log file encoding to UTF-8
2018-06-23 04:06:23,145 fail2ban.actions        [10577]: INFO    Set banTime = 1800
2018-06-23 04:06:23,146 fail2ban.filter         [10577]: INFO    Set findtime = 900
2018-06-23 04:06:23,147 fail2ban.filter         [10577]: INFO    Set maxlines = 10

for this we could use a wildcard (/var/log/fail2ban.log*') but i’m not sure it worths it…log rotation are at sunday 4H AM, you should sleep :slight_smile:

could you reproduce

go to /etc/fail2ban/jail.local

remove the recidive setting then

signal-event nethserver-fail2ban-save

or

signal-event nethserver-fail2ban-update

…on second look… The current log file should have everything from 06:38 18 June forward. But the first lines showing using head are stamped 23 Jun 04:06. Curious.

[root@neth ~]# tail /var/log/fail2ban.log-20180618
2018-06-18 06:21:37,979 fail2ban.filter         [31568]: INFO    [postfix-ddos] Found 185.234.216.190
2018-06-18 06:27:37,952 fail2ban.filter         [31568]: INFO    [apache-fakegooglebot] Ignore 66.249.66.218 by command
2018-06-18 06:27:38,078 fail2ban.filter         [31568]: INFO    [apache-fakegooglebot] Ignore 66.249.66.214 by command
2018-06-18 06:27:38,198 fail2ban.filter         [31568]: INFO    [apache-fakegooglebot] Ignore 66.249.66.2 by command
2018-06-18 06:27:38,674 fail2ban.filter         [31568]: INFO    [apache-auth] Found 66.249.66.2
2018-06-18 06:27:38,674 fail2ban.filter         [31568]: INFO    [apache-auth] Found 66.249.66.2
2018-06-18 06:27:39,320 fail2ban.filter         [31568]: INFO    [apache-fakegooglebot] Ignore 66.249.66.2 by command
2018-06-18 06:30:11,032 fail2ban.filter         [31568]: INFO    [postfix-ddos] Found 185.234.216.190
2018-06-18 06:35:21,246 fail2ban.filter         [31568]: INFO    [apache-auth] Found 85.72.48.43
2018-06-18 06:38:53,729 fail2ban.filter         [31568]: INFO    [postfix-ddos] Found 185.234.216.190

The time zone where I am at the moment is 13 hours different from the time zone set on my server (which itself is 5 hours from the time zone where the server is physically located).

OK:

[root@neth ~]# config setprop fail2ban Recidive_Perpetual enabled
[root@neth ~]# signal-event nethserver-fail2ban-update
[root@neth ~]# nano /etc/fail2ban/jail.local 

[recidive]
enabled   = true
logpath   = /var/log/fail2ban.log
maxretry  = 6
banaction = shorewall-nethserver
bantime   = 604800
findtime  = 84600

…and looking inside /etc/e-smith/events/nethserver-fail2ban-update/ and /etc/e-smith/events/nethserver-fail2ban-save/, the reason becomes clear: the -update event has no services2adjust nor templates2expand directories. The -save event has both.

1 Like

oups yes you are right for the -update, since the runlevel-adjust and the firewall-adjust are called, I did not add them twice

yes this could advocating to a wildcard