Email module transition to Rspamd

mail2
rspamd

(Juan Carlos Fernandez) #1

NethServer Version: 7.5.1804 (beta)
Module: EMail2

I just did the Email module transition to Rspamd instructions for the Email module upgrade. After installing and rebooting I’m getting the following errors:

-- Logs begin at Mon 2018-06-04 18:51:45 CDT, end at Mon 2018-06-04 18:57:47 CDT. --
Jun 04 22:51:48 hermod.dcserver.local smartd[570]: DEVICESCAN failed: glob(3) aborted matching pattern /dev/discs/disc*
Jun 04 22:51:48 hermod.dcserver.local smartd[570]: In the system's table of devices NO devices found to scan
Jun 04 22:51:48 hermod.dcserver.local systemd[1]: Dependency failed for Network Manager Wait Online.
Jun 04 22:52:08 hermod.dcserver.local rspamd[1352]: <xa6qqy>; map; rspamd_map_dns_callback: cannot resolve rspamd.com
Jun 04 22:52:08 hermod.dcserver.local rspamd[1352]: <fdp86m>; map; rspamd_map_dns_callback: cannot resolve maps.rspamd.com
Jun 04 22:52:08 hermod.dcserver.local rspamd[1352]: <pr5857>; map; rspamd_map_dns_callback: cannot resolve updates.rspamd.com
Jun 04 22:52:08 hermod.dcserver.local rspamd[1352]: <fdp86m>; map; rspamd_map_dns_callback: cannot resolve maps.rspamd.com
Jun 04 22:52:08 hermod.dcserver.local rspamd[1352]: <k8f914>; map; rspamd_map_dns_callback: cannot resolve maps.rspamd.com
Jun 04 22:52:08 hermod.dcserver.local rspamd[1352]: <k8f914>; map; rspamd_map_dns_callback: cannot resolve maps.rspamd.com
Jun 04 22:52:08 hermod.dcserver.local rspamd[1352]: <fdp86m>; map; rspamd_map_dns_callback: cannot resolve maps.rspamd.com
Jun 04 22:52:08 hermod.dcserver.local rspamd[1352]: <fdp86m>; map; rspamd_map_dns_callback: cannot resolve maps.rspamd.com

Any idea? Dear all mighty google doesn’t give me any solid hint


(Juan Carlos Fernandez) #2

@dnutan thanks for tagging, I completely forgot it.


(Eddie Atherton) #3

That looks like a DNS issue. Can you resolve any external domains.

Cheers.


(Juan Carlos Fernandez) #4

Thanks for your help, this is a DNS response using dig

dig maps.rspamd.com

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> maps.rspamd.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49771
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1280
;; QUESTION SECTION:
;maps.rspamd.com.		IN	A

;; ANSWER SECTION:
maps.rspamd.com.	59	IN	A	88.99.142.95

;; Query time: 483 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Jun 05 08:14:21 CDT 2018
;; MSG SIZE  rcvd: 60

dig updates.rspamd.com

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> updates.rspamd.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34464
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1280
;; QUESTION SECTION:
;updates.rspamd.com.		IN	A

;; ANSWER SECTION:
updates.rspamd.com.	59	IN	A	88.99.142.95

;; Query time: 320 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Jun 05 08:14:34 CDT 2018
;; MSG SIZE  rcvd: 63

As you can see my server is able to reach those addresses


(Stéphane de Labrusse) #5

please try to restart and reconfigure rspam

signal-event nethserver-mail-filter-save

If the error occurs from time to time it is minor issue, if permanent you have something bad


(Davide Principi) #6

IIRC rspamd relies on unbound as local dns resolver. It does not forward DNS queries with dnsmasq, through the system resolv.conf.

This test does not pass through unbound


(Stéphane de Labrusse) #7

I read from time to time this issue. Does our setting could be the key ???


(Juan Carlos Fernandez) #8

This is before following @stephdl advice:

https://pastebin.com/raw/YMMuTf2z

Tomorrow I will have another look and see if @stephdl hack did the trick, thanks you all for the support.


(Juan Carlos Fernandez) #10

Ok, and how can I test if Unbound DNS resolves correctly?


(Davide Principi) #11

You could start by specifying the unbound port to dig

dig -p 10053 maps.rspamd.com @127.0.0.1

Rspamd is configured to talk to unbound here:

https://github.com/NethServer/nethserver-mail/blob/a89248723039e5e38e033a2c2088b410d668dadd/filter/etc/e-smith/templates/etc/rspamd/rspamd.conf/20Options#L21-L26

@stephdl, maybe the timout (1s) is too low? /cc @filippo_carletti


(Filippo Carletti) #12

The correct syntax is:

dig @127.0.0.1 -p 10053 maps.rspamd.com


(Juan Carlos Fernandez) #13

Both syntaxes give me the same output:

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> @127.0.0.1 -p 10053 maps.rspamd.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29136
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;maps.rspamd.com.		IN	A

;; ANSWER SECTION:
maps.rspamd.com.	60	IN	A	88.99.142.95

;; Query time: 697 msec
;; SERVER: 127.0.0.1#10053(127.0.0.1)
;; WHEN: Wed Jun 06 11:29:51 CDT 2018
;; MSG SIZE  rcvd: 60

One more thing, the error I’ve been repeated lately (From Jun 05 08:07:34 to Jun 06 11:30:43 and still kicking in) is this one:

Jun 06 11:30:43 hermod.dcserver.local rspamd[19453]: <d08888>; proxy; fuzzy_check_timer_callback: got IO timeout with server fuzzy.rspamd.com(88.99.142.95:11335), after 1 retransmits

If I underestand correctly to check this error I should do:

dig @127.0.0.1 -p 11335 fuzzy.rspamd.com

Or

dig -p 11335 fuzzy.rspamd.com @127.0.0.1

Both syntaxes give me a timeout, this is the output in both cases:

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> @127.0.0.1 -p 11335 fuzzy.rspamd.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

@davidep I don’t see any reference about this in the git link you gave me. Could you give me some insights, so I can help?


(Davide Principi) #14

The link was for Stephane and Filippo :wink:

Do we need to increase the request timeout parameter?

the timeout is after 1 retrasmits. But it should do up to 5 retransmits, according to our config. So maybe it is not an issue. Just rspamd is a bit talky about that…

I don’t know if it’s safe, but you could try to increase the dns timeout to 5 seconds in rspamd.conf


(Marc) #15

(Juan Carlos Fernandez) #16

Thanks for opening an issue @dnutan


(Marc) #17

It wasn’t me. Just found someone reported the same issue and the developer said it should be fixed in rspamd 1.7.5
Which version do you have?


(Juan Carlos Fernandez) #18

According to the Web UI and this rspamadm --version I have version 1.7.5


(Juan Carlos Fernandez) #19

BTW, does anyone else has this issue?


(Marc) #20

@PCXLan.es is having the same issue:


(Jon) #21

Maybe because of this?

Rspamd

Fast, free and open-source spam filtering system.
Rspamd 1.7.6 has been released

15 Jun 2018

We have released Rspamd 1.7.6 today. There are no incompatible changes introduced with this version to our best knowledge.
The most important features and fixes

Fix multiple neural networks support: it is now possible to learn multiple neural networks with different settings as documented
Rework rspamadm to use mostly Lua for subcommands for better documentation and extensions support
Add pubkey checks for dkim_signing module (#2277)
DMARC reports are now compressed using gzip as suggested by RFC
Settings module can now skip message processing to improve performance
Bayes classifier now consider more metatokens from the headers
ED25519 DKIM signatures are now supported
Fixed serious issues with composites, maps and other components
Major memory leak hunting and eliminating (especially those that occurs during reload)
Add more tests and allow to create fake DNS records to make certain tests self-contained (e.g. DKIM or DMARC)

Full list of the meaningful changes

[CritFix] Fix multiple neural networks support
[Feature] Add decryption function to keypair command
[Feature] Add gzip compression for HTTP requests in elastic module
[Feature] Add gzip methods to lua util
[Feature] Add maps based on Top Level Domains
[Feature] Add pubkey checks for dkim_signing
[Feature] Add support of fake DNS records
[Feature] Add tool to encrypt files
[Feature] Allow to add symbols using settings directly
[Feature] Allow to match private and public keys for DKIM signatures
[Feature] Allow to set task flags via settings
[Feature] Allow to specify fake DNS address from the config
[Feature] Implement signatures verification using rspamadm keypair
[Feature] Implement signing using rspamadm keypair
[Feature] Improve error reporting for DKIM key access issues
[Feature] Provide $HOSTNAME variable in UCL
[Feature] Rework levenshtein distance computation
[Feature] Split message parsing and processing
[Feature] Support ED25519 DKIM signatures
[Feature] Support encrypted configs in UCL
[Feature] Suppress duplicate warning on very large radix tries
[Feature] Use OSB to combine header names
[Fix] Cleanup maps data on shutdown
[Fix] Fix ‘~’ behaviour in composites
[Fix] Fix HTTP maps updates
[Fix] Fix NIST signatures
[Fix] Fix RFC822 comments when processing a mime address
[Fix] Fix double free
[Fix] Fix dynamic settings application
[Fix] Fix for CommuniGate Pro maillist
[Fix] Fix keypair creation method to actually create keypair…
[Fix] Fix matching patterns with no paths
[Fix] Fix memory leak in parsing comments
[Fix] Fix parsing of urls with numeric password
[Fix] Fix plugins intialisation in configwizard
[Fix] Fix potential crash on reload
[Fix] Fix potential race condition for a finished HTTP connections
[Fix] Fix race-condition leak on processes reload
[Fix] Fix signing in openssl mode
[Fix] Free language detector structures
[Fix] Relax alignment requirements
[Fix] Send DMARC reports compressed
[Fix] Try to fix leak in dmarc module
[Fix] Try to plug memory leak in metric exporter
[Project] Convert rspamadm subcommands to Lua
[WebUI] Display smtp sender/recipient in history
[WebUI] Fix elements disabling in “Symbols” tab
[WebUI] Limit recipients list in history column to 3
[WebUI] Match envelope and mime addresses following in arbitrary order
[WebUI] Update column header
[WebUI] Wrap addresses in history

News: Rspamd 1.7.6 has been released
https://rspamd.com/announce/2018/06/15/rspamd-1.7.6.html