Fail2ban does huge logs

stephdl · October 31, 2017, 7:36pm

I’m setting a new vm specific for fail2ban, you can downgrade nethserver-fail2ban if you want

yum downgrade nethserver-fail2ban

stephdl · November 2, 2017, 6:51am

[root@ns7dev9 ~]# ll /var/log/httpd*
/var/log/httpd:
total 60
-rw-r--r--  1 root root     0 Nov  2 03:16 access_log
-rw-r--r--. 1 root root  3020 Jun 17 19:11 access_log-20171101
-rw-r--r--  1 root root   169 Nov  1 06:35 access_log-20171102
-rw-r--r--  1 root root   517 Nov  2 03:16 error_log
-rw-r--r--. 1 root root 14935 Nov  1 03:55 error_log-20171101
-rw-r--r--  1 root root   633 Nov  2 03:16 error_log-20171102
-rw-r--r--  1 root root     0 Nov  2 03:16 ssl_access_log
-rw-r--r--. 1 root root   860 Nov  1 06:34 ssl_access_log-20171102
-rw-r--r--  1 root root   320 Nov  2 03:16 ssl_error_log
-rw-r--r--. 1 root root  9262 Oct 31 23:00 ssl_error_log-20171101
-rw-r--r--  1 root root   585 Nov  1 06:34 ssl_error_log-20171102
-rw-r--r--  1 root root     0 Nov  2 03:16 ssl_request_log
-rw-r--r--. 1 root root  1112 Nov  1 06:34 ssl_request_log-20171102

/var/log/httpd-admin:
total 48
-rw-r--r--  1 root root     0 Nov  2 03:16 access_log
-rw-r--r--. 1 root root 28396 Oct 31 22:47 access_log-20171101
-rw-r--r--  1 root root  2464 Nov  1 06:35 access_log-20171102
-rw-r--r--. 1 root root 14067 Nov  1 06:16 error_log-20171101
[root@ns7dev9 ~]# ll /var/log/fail2ban.log*
-rw------- 1 root root    108 Nov  2 03:16 /var/log/fail2ban.log
-rw------- 1 root root 316160 Nov  1 06:52 /var/log/fail2ban.log-20171102

confirmed that httpd and fail2ban still continue to write to the log with the same file descriptor. After the log rotation, the *.log is empty and a new file with the date in the name is used as a log file.

copytruncate could be the way but I know it is used for sogo, and I can see the same problem with it…
A workaround could be to create a /etc/logrotate.d/zzz-fail2ban.log configuration and reload fail2ban after the log rotation.

rotation are launched following an alphabetical order…

Really tricky but I would better to understand and fix why nethserver services continue to write on the same file descriptor log

filippo_carletti · November 2, 2017, 7:59am

AFAIK, it happens only with httpd-admin and only once in a few weeks.

# fuser -v /var/log/httpd-admin/access_log*
                     USER        PID ACCESS COMMAND
/var/log/httpd-admin/access_log-20171031:
                     srvmgr     2514 F.... httpd

stephdl · November 2, 2017, 10:04am

I tried to put a vm with a daily rotation…yesterday yet i have had empty logs (sogo, httpd-admin, httpd, fail2ban) for ns7.

I did not test it on ns6 if people can monitor their logs ?

Jclendineng · November 2, 2017, 1:44pm

Correct httpd, also fail2ban, httpd happens every day every 16 hours or so, fail2ban every couple weeks.

Edit: I just checked my log after clearing and reloading fail2ban yesterday. Log is 100 megs and here is the gist. I am getting fail2ban and sogo errors now, no httpd…sorry about the size I just copied the entire log before clearing it.

Edit2: Ok I broke github let me try something different. Pretty sure they blocked me LOL. Ok after 6gb ram usage trying to copy the log I crashed everything, so I cleared and let the log repopulate. This is a good sample, the entire log after a couple minutes.

gist.github.com

https://gist.github.com/anonymous/b370060448dc2f19a31dec919e3c296f

gistfile1.txt

2017-11-02 09:56:13,349 fail2ban.filterpoll     [17200]: ERROR   Unable to get stat on /var/log/sogo/sogo.log-20171026.gz because of: [Errno 2] No such file or directory: '/var/log/sogo/sogo.log-20171026.gz'
2017-11-02 09:56:13,349 fail2ban.filterpoll     [17200]: ERROR   Unable to get stat on /var/log/sogo/sogo.log-20171031 because of: [Errno 2] No such file or directory: '/var/log/sogo/sogo.log-20171031'
2017-11-02 09:56:13,349 fail2ban.filterpoll     [17200]: WARNING Too many errors. Setting the jail idle
2017-11-02 09:56:13,349 fail2ban.filterpoll     [17200]: ERROR   Unable to get stat on /var/log/sogo/sogo.log-20171025.gz because of: [Errno 2] No such file or directory: '/var/log/sogo/sogo.log-20171025.gz'
2017-11-02 09:56:13,349 fail2ban.filterpoll     [17200]: WARNING Too many errors. Setting the jail idle
2017-11-02 09:56:13,783 fail2ban.filterpoll     [17200]: ERROR   Unable to get stat on /var/log/fail2ban.log-20171030 because of: [Errno 2] No such file or directory: '/var/log/fail2ban.log-20171030'
2017-11-02 09:56:14,350 fail2ban.filterpoll     [17200]: ERROR   Unable to get stat on /var/log/sogo/sogo.log-20171026.gz because of: [Errno 2] No such file or directory: '/var/log/sogo/sogo.log-20171026.gz'
2017-11-02 09:56:14,350 fail2ban.filterpoll     [17200]: ERROR   Unable to get stat on /var/log/sogo/sogo.log-20171031 because of: [Errno 2] No such file or directory: '/var/log/sogo/sogo.log-20171031'
2017-11-02 09:56:14,350 fail2ban.filterpoll     [17200]: ERROR   Unable to get stat on /var/log/sogo/sogo.log-20171025.gz because of: [Errno 2] No such file or directory: '/var/log/sogo/sogo.log-20171025.gz'
2017-11-02 09:56:14,784 fail2ban.filterpoll     [17200]: ERROR   Unable to get stat on /var/log/fail2ban.log-20171030 because of: [Errno 2] No such file or directory: '/var/log/fail2ban.log-20171030'

This file has been truncated. show original

stephdl · November 2, 2017, 11:08pm

upgrade fail2ban in few hours, I reverted the patch with the wildcard

Jclendineng · November 4, 2017, 3:59am

Nothing in the log since I cleared it after posting that log.

Jclendineng · November 10, 2017, 2:06pm

It seems to be fixed, logs are staying nice. Awesome!