Whitelist and blacklist not effective with POP3 connector

davidep · February 7, 2020, 4:24pm

now that we have a reasonable fix for the discared mails we can look at this one too… /cc @stephdl

Probably the whitelist/blacklist cannot work with getmail because they are evaluated only with a SMTP envelope… Let’s try to reproduce it.

stephdl · February 7, 2020, 4:33pm

I am quite confident that getMail is whitelist aware, I can find it in side the maillog that @wbilger gave us

davidep · February 7, 2020, 4:36pm

I would be so happy if you are right!

not-a-bug is far better than wont-fix

stephdl · February 7, 2020, 4:39pm

<6467506.1580747919703.JavaMail.wwwrun@localmail.loss.com>: set pre-result to 'no action' (no score): 'Matched map: FROM_SUBDOMAINS_WHITELIST' from multimap(1)
Feb  3 11:40:19 lrtserv-data rspamd[26933]: <8d008f>; csession; rspamd_task_write_log: id: <6467506.1580747919703.JavaMail.wwwrun@localmail.londonroos.com>, ip: 127.0.0.1, from: <tonia@londos.com>, (default: F (no action): [0.00/15.00] [FROM_SUBDOMAINS_WHITELIST(0.00){domain.com;}]), len: 1404, time: 3.217ms, dns req: 0, digest: <8e42e44498f4cc7e80faed013c49ded4>, mime_rcpts: <sam@domain.com>, file: stdin, forced: no action "Matched map: FROM_SUBDOMAINS_WHITELIST"; score=nan (set by multimap)

wbilger · February 7, 2020, 4:46pm

I think so, this was when first trying to find the soft reject issue, but I corrected after that a re bulk-import of whitelist settings overwrote what I had in there. So I think this was a mistake on my part.

stephdl · February 7, 2020, 4:51pm

but @davidep when you reject the email and you delete it with getmail from the remote server …what it happen ?

 Feb  7 17:49:14 prometheus rspamd[31990]: <1b5e12>; csession; rspamd_add_passthrough_result: <37B8F13771AD4CBEA4F52A0D3AF2B154@EMEA.FS.UTC.com>: set pre-result to 'reject' (no score): 'Sender email address rejected' from force_actions(1)
Feb  7 17:49:29 prometheus rspamd[31990]: <1b5e12>; lua; clamav.lua:119: clamav: failed to scan, maximum retransmits exceed
Feb  7 17:49:29 prometheus rspamd[31990]: <1b5e12>; lua; common.lua:107: clamav: result - FAILED with error: "failed to scan and retransmits exceed - score: 0"
Feb  7 17:49:29 prometheus rspamd[31990]: <1b5e12>; csession; rspamd_task_write_log: id: <37B8F13771AD4CBEA4F52A0D3AF2B154@EMEA.FS.UTC.com>, ip: 127.0.0.1, from: <devoir.de.conseil@chubb.fr>, (default: T (reject): [-0.10/19.90] [MIME_GOOD(-0.10){multipart/mixed;text/plain;},CLAM_VIRUS_FAIL(0.00){failed to scan and retransmits exceed;},FORCE_ACTION_REJECT_FROM_BLACKLIST(0.00){reject;},FROM_BLACKLIST(0.00){devoir.de.conseil@chubb.fr;},FROM_EQ_ENVFROM(0.00){},FROM_NO_DN(0.00){},HAS_REPLYTO(0.00){devoir.de.conseil@chubb.fr;},MIME_TRACE(0.00){0:+;1:+;2:~;},REPLYTO_EQ_FROM(0.00){}]), len: 357882, time: 15006.329ms, dns req: 0, digest: <7b97fe11554dad5fc5d7ff49a96d5aab>, mime_rcpts: <luc.fabre@chubb.fr,maisonderetraite.sauveterre@wanadoo.fr,stephane@de-labrusse.fr...>, file: stdin, forced: reject "Sender email address rejected"; score=nan (set by force_actions)
Feb  7 17:49:29 prometheus rspamd[31990]: <1b5e12>; csession; rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned, 0 regexps matched, 184 regexps total, 0 regexps cached, 0B scanned using pcre, 0B scanned total
Feb  7 17:49:30 prometheus dovecot: lda(toto@de-labrusse.fr): sieve: msgid=<37B8F13771AD4CBEA4F52A0D3AF2B154@EMEA.FS.UTC.com>: marked message to be discarded if not explicitly delivered (discard action)
Feb  7 17:49:30 prometheus getmail: msg  4/31 (362433 bytes) msgid 597083610/36139 from <devoir.de.conseil@chubb.fr> delivered to MDA_external command dovecot-lda (), deleted

davidep · February 7, 2020, 4:56pm

By your log, if a message hits a blacklist:

it is delivered and discarded by sieve
it is expunged from the remote server

It is destroyed, as expected. Good news so.

The blacklist works. What about the whitelist?

wbilger · February 8, 2020, 1:47pm

I have mydomain.com in my whitelist.
I just sent a test message to one of my users with a spam signature in the content of the message, and the message was rejected (message was retrieved by getmail). Is the whitelist supposed to completely bypass spam filtering?

stephdl · February 8, 2020, 1:59pm

Yes, as ever we need log lines, what is the spam signature

wbilger · February 8, 2020, 2:46pm

I have this in the body of the message;

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

mydomain.com is in Allow From whitelist
Here’s the log;

Feb 8 09:40:03 lrtserv-data rspamd[2939]: ; csession; rspamd_controller_check_password: allow unauthorized connection from a trusted IP 127.0.0.1
Feb 8 09:40:03 lrtserv-data rspamd[2939]: ; csession; rspamd_message_parse: loaded message; id: 7e7a9f7842a1f264ff5467a814cd3c56@mydomain.com; queue-id: ; size: 1085; checksum:
Feb 8 09:40:03 lrtserv-data rspamd[2939]: ; csession; rspamd_check_gtube: gtube reject pattern has been found in part of length 69
Feb 8 09:40:03 lrtserv-data rspamd[2939]: ; csession; rspamd_add_passthrough_result: 7e7a9f7842a1f264ff5467a814cd3c56@mydomain.com: set pre-result to ‘reject’ (15.00): ‘Gtube pattern’ from GTUBE(3)
Feb 8 09:40:03 lrtserv-data rspamd[2939]: ; csession; rspamd_task_write_log: id: 7e7a9f7842a1f264ff5467a814cd3c56@mydomain.com, ip: 127.0.0.1, from: test@mydomain.com, (default: S (reject): [15.00/15.00] [GTUBE(0.00){}]), len: 1085, time: 0.966ms, dns req: 0, digest: , mime_rcpts: wayne@mydomain.com, file: stdin, forced: reject “Gtube pattern”; score=15.00 (set by GTUBE)
Feb 8 09:40:03 lrtserv-data rspamd[2939]: ; csession; rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned, 0 regexps matched, 184 regexps total, 0 regexps cached, 0B scanned using pcre, 0B scanned total
Feb 8 09:40:03 lrtserv-data dovecot: lda(wayne@mydomain.com): sieve: msgid=7e7a9f7842a1f264ff5467a814cd3c56@mydomain.com: marked message to be discarded if not explicitly delivered (discard action)
Feb 8 09:40:03 lrtserv-data getmail: msg 103/103 (1113 bytes) msgid 1580749373/126 from test@mydomain.com delivered to MDA_external command dovecot-lda ()
Feb 8 09:40:17 lrtserv-data rspamd[2939]: ; lua; bayes_expiry.lua:437: finished expiry step 674: 1001 items checked, 121 significant (4 made persistent), 0 insignificant (0 ttls set), 1 common (0 discriminated), 879 infrequent (33 ttls set), 7 mean, 43 std
Feb 8 09:40:21 lrtserv-data clamd[3050]: SelfCheck: Database status OK.
Feb 8 09:40:34 lrtserv-data rspamd[2939]: <3e3c2f>; csession; rspamd_controller_check_password: allow unauthorized connection from a trusted IP 127.0.0.1
Feb 8 09:40:34 lrtserv-data rspamd[2939]: <04707d>; csession; rspamd_controller_check_password: allow unauthorized connection from a trusted IP 127.0.0.1
Feb 8 09:41:51 lrtserv-data rspamd[2939]: ; lua; bayes_expiry.lua:437: finished expiry step 675: 999 items checked, 135 significant (13 made persistent), 1 insignificant (0 ttls set), 0 common (0 discriminated), 863 infrequent (9 ttls set), 5 mean, 19 std

stephdl · February 8, 2020, 3:42pm

It is normal, the spam signature is a test and it is checked before all filters

This spam signature is a test for rspamd and a mandatory to check before all others map/modules

So nothing to declare, to do , to check

However we have talked about getMail with @davidep this friday, we think to display a banner, be aware you can loose an email if you delete the emails from the remote server and the fetched emails has been rejected by ourself.