How to analyze who triggers fail2ban

capote · April 19, 2021, 9:39am

Regarding this thread we should try collect know how to analyze which client triggers fail2ban.
In my mind that summarize know how about:

Some reproducible strategies to find the root causes of bans
most common commands for analysis of the relevant log files
prevention against the most common scenarios to lock yourself out.

I will progressively consolidate the answers here in the first posting.

Use Case #1
1. Architecture

root server within an external data center
2 network interfaces with WAN-IP (RED) and dummy LAN-IP (Green)
installed IPS, Thread Shield
own LAN with more than 30 clients behind a DSL router with dynamically changed IP address they use Nethsever for…
- Mail
- authentication for some Nethserver provided applications like FileServer, DokuWiki, NextCloud, Webmail, Wordpress, VPN (Road Warrior)
  2. fail2ban configuration
active jails:
- apache-auth
- apache-badbots
- apache-botsearch
- apache-fakegooglebot
- apache-modsecurity
- apache-nohome
- apache-noscript
- apache-overflows
- apache-scan
- apache-shellshock
- dovecot
- dovecot-nethserver
- ejabberd-auth
- httpd-admin
- mysqld-auth
- nextcloud-auth
- openvpn
- pam-generic
- pam-generic-nethserver
- phpmyadmin
- postfix
- postfix-ddos
- postfix-sasl-abuse
- recidive
- rspamd
- sieve
- sogo-auth
- sshd
- vsftpd
whitelisting: yes
static ban time
log level:INFO
Number of attempts: 3
time span: 10 Minutes
ban time: 15 Minutes

3. fai2ban behavior

fail2ban bans my own LAN because one of my clients hammers against fail2ban. That’s why analysis and administration is totaly blocked
Assumption 1: Typo within mail client logon data
Assumption 2: user changed his own password via cockpit and the credentials within mail clients become invalid

4. work around:

connect administrator client via mobile phone to internet to access to the root server
whitelisting the WAN IP of the DSL-Router (unfortunately each time if DSL-Router changes his IP-address)
preventive renewal of the mail login data for all clients, as it was not possible to identify the triggering client quickly

5. Analysis Procedure:

Identify banned IP and whitelist and unblock it if it is identical to your own WAN IP
Determination of the applicable jail / service and trigger time
Determination of the applicable local client
tbd

6. Analysis commands:
6.1 Analysis of triggerd jails
# cat /var/log/fail2ban.log | grep "Ban my.dynamic.wan.ip"

6.1 Analysis of current postfix/smtpd jail

# fail2ban-regex /var/log/maillog /etc/fail2ban/filter.d/postfix.conf --print-all-matched
# fail2ban-regex /var/log/maillog /etc/fail2ban/filter.d/postfix-ddos.conf --print-all-matched

6.2 Analysis of earlier postfix/smtpd jail

# fail2ban-regex /var/log/maillog-yyyymmdd /etc/fail2ban/filter.d/postfix.conf --print-all-matched
# fail2ban-regex /var/log/maillog-yyyymmdd /etc/fail2ban/filter.d/postfix-ddos.conf --print-all-matched

Matrix of log files corresponding to jails

|       Jail                |       Logs                    |
| apache-auth               | /var/log/httpd/*error_log     |    
| apache-badbots            | /var/log/httpd/access_log     |
| apache-botsearch          | /var/log/httpd/*error_log     |
| apache-fakegooglebot      | /var/log/httpd/access_log     |
| apache-modsecurity        | /var/log/httpd/*error_log     |
| apache-nohome             | /var/log/httpd/*error_log     |
| apache-noscript           | /var/log/httpd/*error_log     | 
| apache-overflows          | /var/log/httpd/*error_log     |
| apache-scan               | /var/log/httpd/*error_log.    |
| apache-shellshock         | /var/log/httpd/*error_log     |
| dovecot                   | /var/log/imap                 |
| dovecot-nethserver        | /var/log/imap                 | 
| ejabberd-auth             | /var/log/ejabberd/ejabberd.log|
| httpd-admin               |/var/log/httpd-admin/access_log|
| mysqld-auth               | /var/log/mariadb/mariadb.log  |
| nextcloud-auth            |/var/lib/nethserver/nextcloud/nextcloud.log |
| openvpn                   | /var/log/openvpn/openvpn.log  |
| pam-generic               | /var/log/secure               |
| pam-generic-nethserver    | /var/log/secure               |
| phpmyadmin                | /var/log/httpd/access_log     |
| postfix                   | /var/log/maillog              |
| postfix/smtp              | /var/log/maillog              |
| postfix-ddos              | /var/log/maillog              |
| postfix-sasl-abuse        | /var/log/maillog              |
| recidive                  | /var/log/fail2ban.log         |
| rspamd                    |/var/log/httpd-admin/access_log|
| sieve                     | %(dovecot_log)s               |
| sogo-auth                 | /var/log/sogo/sogo.log        |
| sshd                      | %(sshd_log)s                  |
| vsftpd                    | /var/log/vsftpd.log           |
| webtop                    |/var/log/webtop/webtop_auth.log|

Please correct and add with your suggestions.

stephdl · April 19, 2021, 10:54am

I would add some feed for thoughts

If the remote gateway is a static IP, add it to the whitelist
Create a vpn tunnel and add the remote local IP to the well know network (cc @pagaille)
If the server is to authenticate users, try the incremental ban time, better than the recidive where people goes to jail for a long time.
FindTime: 900,BanTime:60, RecidiveBan:incremental

In order to find what client has triggered the jail, the idea is

Go to fail2ban.log and find the jail
use fail2ban-regex with the relevant log (written in jail.local) and the relevant filter (same as the jail name)

yummiweb · April 19, 2021, 8:18pm

I could imagine a script that looked onto an (external) dns (maybe dynamic dns), find out the (daily) ip adress to set this ip onto the fail2ban whitelist for a predefined time or until the dns points to an other ip.

capote · April 19, 2021, 10:29pm

IMHO ddclient: module:ddclient [NethServer Wiki].
I use it to get the current dynamic address on my DSL router for the Nethserer VPN.

stephdl · April 20, 2021, 7:11am

Question: does the remote gateway (wan-ip) are whitelisted inside your fail2ban

capote · April 20, 2021, 7:32am

At me, yes. But whitelisting is obsolete when my DSL router changes its dynamic IP address.

stephdl · April 20, 2021, 7:38am

I always refused to add a domain name to the whitelist but I am quite sure it could work … Sometime a developer does not reach the need of sysadmin

:-?

capote · April 20, 2021, 8:18am

I’m having a hard time understanding how it could help.

Do you mean the following mechanism I practice for VPN dial-up to my DSL router?

my DNS provider determines via ddclient my current dyamic address at the gateway.
my DNS provider assigns this as A-record to a subdomain like vpn.mydomain.tld
I whithelist the subdomain vpn.mydomain.tld with the current dynamically assigned IP address

stephdl · April 20, 2021, 9:51am

I mean in /etc/fail2ban/jail.local add to

ignoreip = 127.0.0.1 stephdl.dynamic.ip.provider.net

restart fail2ban then

2021-04-20 11:45:40,422 fail2ban.filter [26521]: INFO [sogo-auth] Ignore 90.1.xxx.xxx by dns

Then your dynamic IP will be ignored by fail2ban but the UI will create an error because the validation accept only IP or network

@giacomo @filippo_carletti what do you think if we could accept dynamic domain inside the UI of fail2ban

Like this we could be sure to be not banned when you do not have a static IP ?

filippo_carletti · April 20, 2021, 9:54am

fail2ban accepts hostnames, I don’t know why we limited the interface to IPs only.

stephdl · April 20, 2021, 9:55am

At the start I even refused to add CIDR

stephdl · April 20, 2021, 10:45am

@capote it is coming

https://github.com/NethServer/nethserver-fail2ban/pull/113

With this you can add FQDN or hostname

capote · April 20, 2021, 12:42pm

This is the same as I thought. The difference is that I don’t need a separate subdomain like dynip.mydomain.de because I have one for VPN dialup called vpn.mydomain.de.

capote · April 20, 2021, 12:51pm

One big request…
Could you please integrate the ability to comment on the whitelisted IPs, CIDR or domains so that everyone can document the reason for white listening, as…

#Admins LAN
stephdl.dynamic.ip.provider.ne
#uptimerobot 1
72.55.136.192/28
#anything else
11.22.33.44

stephdl · April 20, 2021, 12:55pm

inside textarea I think it is not possible

capote · April 20, 2021, 12:57pm

Too bad, I thought it would be as simple as in any script. But I don’t understand any of it either.

stephdl · April 20, 2021, 1:00pm

web development is never easy, as well how to understand the good UI. Imagine comments inside a textarea, after all everything is possible I could create modals to fill the whitelist with a table like for the unban, but please go to github and check how many hundred line of code I will add for something that is a hole in the security…what withlist is

capote · April 20, 2021, 1:16pm

capote:

Matrix of log files corresponding to jails

| Jail              | Logs                        |
| postfix/smtp      | /var/log/maillog            |
| apache-auth       | /var/log/httpd/access_log   |
| tbd               | tbd                         |

Can anyone contribute additional relationships between triggered jail and log files to be analyzed (where more in-depth analysis would be needed)??

stephdl · April 20, 2021, 3:54pm

everything can be found in /etc/fail2ban/jail.local

dnutan · April 20, 2021, 9:07pm

Unrelated to fail2ban but a quick & dirty script to get the changing IP address of a domain (lets say of another of your publicly reachable hosts) could be something like this:

#!/bin/bash
#
# Update host object IP address (to be used with firewall rules, i.e. for public domain)
# (Requires: dig, ipcalc, e-smith commands)
#

# get IP address
ip=$( /usr/bin/basename $( /usr/bin/dig +short domain.tld A | tr '\n' '/' ) 2>&- )

# validate IPv4 address
if ! /usr/bin/ipcalc -c "$ip"; then exit; fi

# get stored IP address (if any) # hard-coded value
host='hostnamex'
oldip=$( /usr/sbin/e-smith/db hosts getprop $host IpAddress )

# set firewall object to the new IP address
if [ "$ip" = "$oldip" ]
then
   exit
else
   /usr/sbin/e-smith/db hosts set $host host IpAddress $ip
   /usr/sbin/e-smith/signal-event firewall-adjust
fi

For instance to be used with a cron job to update the host ip address used on some firewall rules.
Probably there are better ways to do it but there’s that.