How to do email statistics

NethServer Version: NethServer 7.9.2009
Module: email

As I migrated from SME to NS I’m missing a tool for email statistics. Is there any chance or tool like

Email Statistics

Mainly it’s scripting based on usual linux programs. We collect (copy) every email with maillog - maybe someone could point me in a direction to extract all sender and receiver email-adresses. Or do I overlook something which is already there?

Thx for any help in advance,
stefan

Hi Stefan,
awstats and rSpamd-Statistics?

Sincerely, Marko

2 Likes

Hi Marko,

thank you for joining in.

Basically the intention is to get a yearly report of all emails coming in (not the rejected ones) with the full email-adress i.e. john.doe@example.com. As well as all adresses the users sent to anyone. Also with the full email-adress.

I tried and struggled around with nethserver-awstats. It shows only one month (December, but no other). Even when I pick another let’s say November, nothing is dating up in the stats. In the default the config is pointing to /var/log/maillog. Of course there a a few older rotated other maillog files. It seems, they are not examined. Unfortunately googling around is not helping much - have you got any idea?

Best regrads,
stefan

please check it again. You can select “yearly”


and check
image

best regards, Marko

1 Like

Tried this already. No luck. Only December is displayed.

Best regards,
stefan

I believe, you should scroll down, compare the lists with different selection (esp. Dec vs. YEAR)
Only the second section constantly displays monthly statistics.

Sorry. Not working on this server. No matter what I choose - only December is displayed.

maybe @stephdl can help

The awstats FAQ explains how to process old logfiles.

@mrmarkuz

Thank you for hopping on.

I tried this:

First I merged all maillog files in another file with:

perl /usr/share/awstats/tools/logresolvemerge.pl /var/log/maillog* > /var/log/test

Then I run:

perl /usr/share/awstats/wwwroot/cgi-bin/awstats.pl -config=awstats.maillog.conf -update
Create/Update database for config “/etc/awstats/awstats.maillog.conf” by AWStats version 7.8 (build 20200416)
From data in log file “perl /usr/share/awstats/tools/maillogconvert.pl standard < /var/log/test |”…
Phase 1 : First bypass old records, searching new record…
Searching new records from beginning of log file…
Phase 2 : Now process new records (Flush history on disk after 20000 hosts)…
Jumped lines in file: 0
Parsed lines in file: 23668
Found 0 dropped records,
Found 0 comments,
Found 0 blank records,
Found 22537 corrupted records,
Found 0 old records,
Found 1131 new qualified records.

What does it mean, that there are the 22537 corrupted records? Now awstats is showing some statistc - I feel there’s missing a lot email.

edit:

I purged the files in /var/lib/awstats/

Running

perl /usr/share/awstats/wwwroot/cgi-bin/awstats.pl -config=awstats.maillog.conf -update -showcorrupted

brings up i.e.

Corrupted record (date 20211210154211 lower than 20211212224502-20000): 2021-12-10 15:42:11

O.k. seems the merged file has to be in a chronical order…

1 Like

Here are some hints about corrupted records.

Here’s what I did:

  1. created a bash-script i.e. mail_sort.sh to sort the old logs:

#!/bin/bash
for file in maillog-* ; do
sort -k1,1rn < “$file”
done > test

  1. In the /etc/awstats config file for maillog I changed for the initial run to
    LogFile=“perl /usr/share/awstats/tools/maillogconvert.pl standard < /var/log/test |”

Only a few errors ocurred, that’s o.k. for me.

  1. I reverted the awstats.maillog.conf back to
    LogFile=“perl /usr/share/awstats/tools/maillogconvert.pl standard < /var/log/maillog |”

That seemed to do the trick.

Thanks to everybody for helping with this.

Best regards and merry christmas,
stefan

3 Likes