Email 2 with rspamd 1.7.x

[davidep]

@stephdl fixed the rspamd configuration for the latest release of rspamd: 1.7.1

It is available from the testing repository:

• nethserver-mail2-filter-2.0.0-1.3.gae5c436.ns7.noarch
• rspamd-1.7.1-1.x86_64

The 1.7 branch has a new bayes DB format. To train the new bayes DB with messages in the Junk folder run the shell following script

for USER in /var/lib/nethserver/vmail/*; do
  USER=$(basename $USER)

  echo "Loading $USER.."
 
  doveadm search -u $USER mailbox Junk ALL | 

  while read guid uid; do
     doveadm -f pager fetch -u $USER text mailbox-guid $guid uid $uid \
         | sed '1d;$d' \
         | rspamc -h localhost:11334 learn_spam
  done

done

[mrmarkuz]

Updated 2 servers and first tests worked! Spam learning works too.

[davidep]

How is it going with rspamd 1.7.1? Is everything working fine? /cc @stephdl @filippo_carletti @mrmarkuz

[davidep]

@stephdl I found this deprecation warning in /etc/rspamd/metrics.conf

# DEPRECATION WARNING!!
# This file is deprecated since 1.7
# Please use actions.conf and groups.conf files instead

The package in testing still has /etc/e-smith/templates/etc/rspamd/override.d/metrics.conf/10base: shall we remove it?

[stephdl]

actions.conf

I missed that. We use groups.conf now, but still use metrics§.conf although it is still workable but deprecated. We ought to move the metrics.conf to action.conf

[mrmarkuz]

On one server rspamd 1.7.1 is running, it seems to work but I see these errors in web UI (maybe I should just update to 1.7.2):

28.3.2018, 18:55:06 controller 20907 csession 91aaae got IO timeout with server fuzzy.rspamd.com(8.9.1.9), after 3 retransmits

On another server rspamd 1.7.2 is running, working fine without errors.

[stephdl]

28.3.2018, 18:55:06 controller 20907 csession 91aaae got IO timeout with server fuzzy.rspamd.com(8.9.1.9), after 3 retransmits

this should be a network issue to contact the website fuzzy.rspamd.com (spam database)

[stephdl]

@davidep @filippo_carletti with rspamd 1.7.2 we have two options cool in actions.conf

unknown_weight = 1.0; # Enable if need to set score for all symbols implicitly
# Each new symbol is added multiplied by gf^N, where N is the number of spammy symbols
#grow_factor = 1.1;

I would like to enable the firts one

unknown_weight = 1.0;

if a symbol does not get a weight, the weight is 1

[stephdl]

@dev_team something has changed with 1.7.0 in a good and bad direction

my bug https://github.com/vstakhov/rspamd/issues/1968 is solved

this is the good thing

the bad thing now could drive us to change our way of mind:

Indeed the UI gets precedence over static by the dynamic map, so once a new settings is put in the web UI, static settings in configuration files are not used anymore.

Therefore we are going to a full rspamd UI I think, maybe not if we want to change the spam subject message but probably one day we could be able to do it in rspamd.

Actually the new way if you want to only add header and no rewrite subject, the way is to blank the value rewrite subject and save in the web UI of rspamd.

All new settings are saved in the dynamic map that you can edit online in the configuration tab (/var/lib/rspamd/rspamd_dynamic)

I know it is a new way that can change a lot of our habits maybe more for people using the CLI

I need to think on it

[giacomo]

Indeed the UI gets precedence over static by the dynamic map, so once a new settings is put in the web UI, static settings in configuration files are not used anymore.

I didn’t study much the whole rspamd thing, so I need an example to fully understand the problem:

1. configure spam threshold to from server manager
2. change spam threshold from rspamd UI
3. change again the spam threshold from server manager

In this case, the real enforced configuration is the one set on point 2.
Do I correctly understand?

If yes, I don’t think this is a problem but after point 2, the server manager page should be blocked and display a warning (or something similar).

[stephdl]

In this case, the real enforced configuration is the one set on point 2.

Do I correctly understand

Yes once settings set in rspamd UI it is mandatory. I think we could remove the threshold in nethgui

[giacomo]

I agree with you.

IMHO options available in rspamd UI should be removed from the server manager.
Also a label like this should be added: “Options x,y,z can be configured using http://rpsmadUI”

[davidep]

I liked the idea to “disable” the mail-filter tab if rspamd takes the control. If we plan to replace Email with Email 2 as automated RPM update during ns7 lifecycle, I prefer the UI is not changed.

[giacomo]

I prefer the UI is not changed.

If the UI is misleading, it must be changed.
For example, if after the upgrade the UI has no real effect on the configuration, we can’t leave everything as is.
But we should inform the user about the change using something like the label suggested above

[stephdl]

The dynamic map takes precedence now and we can not disable it because it stores all modified settings in the rspamd ui

If we have two settings place, it drives to issues

[davidep]

I think we all agree. If we cannot control rspamd settings from server manager, the server manager must reflect this situation consistently. What I want to suggest is not change the current UI widgets and labels to avoid “upgrade confusion”; just change the style to “read only/disabled” and put a link to rspamd UI.

It must be clear what is the new place where the thresholds can be tweaked.

[stephdl]

The dynamic map takes precedence now and we can not disable it because it stores all modified settings in the rspamd ui

But we can play with curl and modify dynamically the rspamd_dynamic map by an action

curl 'https://rspamd:PASSWORD@127.0.0.1:980/rspamd/saveactions' --data '[20,9,8.8,7]' -k

order is spam/rewrite_subject/probably_spam/Greylist

the password is stored in /var/lib/nethserver/secrets/rspamd

kudo to @giacomo and @davidep for the idea

[stephdl]

@dev_team we have some change I would like to talk

if you look this snippet from /usr/share/rspamd/www/js/app/config.js

        function saveActions(callback) {
            var elts = loadActionsFromForm();
            // String to array for comparison
            var eltsArray = JSON.parse(loadActionsFromForm());
            if (eltsArray[0] < 0) {
                rspamd.alertMessage("alert-modal alert-error", "Spam can not be negative");
            } else if (eltsArray[1] < 0) {
                rspamd.alertMessage("alert-modal alert-error", "Rewrite subject can not be negative");
            } else if (eltsArray[2] < 0) {
                rspamd.alertMessage("alert-modal alert-error", "Probable spam can not be negative");
            } else if (eltsArray[3] < 0) {
                rspamd.alertMessage("alert-modal alert-error", "Greylist can not be negative");
            } else if (
                (eltsArray[2] === null || eltsArray[3] < eltsArray[2]) &&
                (eltsArray[1] === null || eltsArray[2] < eltsArray[1]) &&
                (eltsArray[0] === null || eltsArray[1] < eltsArray[0])
            ) {
                callback("saveactions", null, null, "POST", {}, {
                    data: elts,
                    dataType: "json"
                });
            } else {
                rspamd.alertMessage("alert-modal alert-error", "Incorrect order of metric actions threshold");
            }
        } 

you saw that now the rewrite_subject action gets its own score and you cannot set the same score than the add_header because it breaks the UI. For now I added 0.1 more than the add_header but we can imagine to get a specific threshold for the rewrite_subject.

What do you think ?

[davidep]

Ok, but add 0.01: I saw many 0.1 scores around. An order of magnitude less is safer.

BTW we could expose a new threshold widget too…

[stephdl]

Ok I can import the parameters (reject, add_header,grey) from a curl action in nethgui, I would like to save it to rspamd now … just FYI

of course I have to play with nethgui a bit