NethServer Version: 7.9.2009
Module: Email
Hello,
Having some problems here… Background: All started as our ISP set 30 mail (port 25) per minute limit in their firewall, that should be more than sufficient, but we got blocked many times in the last few months. First there were DDoS attacks trying to send dozens of emails to not existing users at our domain, and actually fail2ban got them after 3 attempts but until it read the log files and applied the filter we already exceeded the limit by responding to these. Fortunately after reorganizing the whole network and setting IP blacklists in pFsense, this problem has gone for now.
Currently we have a strange problem: whenever DHL sends an automated email (arrives without a problem), postfix goes crazy, tries to connect to all of the mail servers (to do some feedback??). It is always some kind of response to the incoming e-mail that triggers the filter at ISP - 4th time in the last 4 work days.
The problem is, I could not find any good flowchart or description about what really happens when an e-mail arrives. First I thought it might be a DMARC report due to some misconfiguration, but SPF, DKIM is valid (according to rSpamd, but also checked SPF manually).
I have packet capture on port 25 for the last time it happened, but as a non-expert all I can see is that after the e-mail arrives well from an IP, a second later there are dozens of SYN connections to several other IPs in a totally different IP range, and then FIN, ACK to the same IPs and the filter bans us, rest of the FIN, ACK ends with TCP tries to do its job.
What I have figured out is that all of these IPs are resolved by the DNS records for the MX domains:
> dhl.com
Server: UnKnown
Address: 10.10.10.10
Non-authoritative answer:
dhl.com MX preference = 5, mail exchanger = mx1.dhl.iphmx.com
dhl.com MX preference = 10, mail exchanger = mx2.dhl.iphmx.com
mx1.dhl.iphmx.com internet address = 68.232.135.99
mx1.dhl.iphmx.com internet address = 68.232.129.198
mx1.dhl.iphmx.com internet address = 68.232.148.170
mx1.dhl.iphmx.com internet address = 68.232.142.218
mx1.dhl.iphmx.com internet address = 68.232.148.169
mx1.dhl.iphmx.com internet address = 68.232.142.49
mx1.dhl.iphmx.com internet address = 68.232.143.139
mx1.dhl.iphmx.com internet address = 68.232.148.171
mx1.dhl.iphmx.com internet address = 68.232.143.176
mx1.dhl.iphmx.com internet address = 68.232.130.32
mx1.dhl.iphmx.com internet address = 68.232.142.236
mx1.dhl.iphmx.com internet address = 68.232.129.11
mx1.dhl.iphmx.com internet address = 68.232.142.240
mx1.dhl.iphmx.com internet address = 68.232.129.199
mx1.dhl.iphmx.com internet address = 68.232.135.103
mx1.dhl.iphmx.com internet address = 68.232.141.220
mx1.dhl.iphmx.com internet address = 68.232.143.21
mx1.dhl.iphmx.com internet address = 68.232.135.98
mx1.dhl.iphmx.com internet address = 68.232.135.101
mx1.dhl.iphmx.com internet address = 68.232.141.53
Possibly not the best way DHL does load balancing (is this even valid)?
Gmail for reference:
> gmail.com
Server: UnKnown
Address: 10.10.10.10
Non-authoritative answer:
gmail.com MX preference = 5, mail exchanger = gmail-smtp-in.l.google.com
gmail.com MX preference = 30, mail exchanger = alt3.gmail-smtp-in.l.google.com
gmail.com MX preference = 40, mail exchanger = alt4.gmail-smtp-in.l.google.com
gmail.com MX preference = 20, mail exchanger = alt2.gmail-smtp-in.l.google.com
gmail.com MX preference = 10, mail exchanger = alt1.gmail-smtp-in.l.google.com
gmail-smtp-in.l.google.com internet address = 108.177.127.27
gmail-smtp-in.l.google.com AAAA IPv6 address = 2a00:1450:4013:c07::1a
alt3.gmail-smtp-in.l.google.com internet address = 142.250.157.27
alt3.gmail-smtp-in.l.google.com AAAA IPv6 address = 2404:6800:4008:c13::1a
alt4.gmail-smtp-in.l.google.com internet address = 74.125.199.26
alt4.gmail-smtp-in.l.google.com AAAA IPv6 address = 2607:f8b0:400e:c02::1a
alt2.gmail-smtp-in.l.google.com internet address = 74.125.200.26
alt2.gmail-smtp-in.l.google.com AAAA IPv6 address = 2404:6800:4003:c00::1a
alt1.gmail-smtp-in.l.google.com internet address = 142.250.150.26
alt1.gmail-smtp-in.l.google.com AAAA IPv6 address = 2a00:1450:4010:c1c::1a
Anyhow, it looks like the problem could be solved if postfix would use only 1 IP from the many it receives from the DNS query, or by limiting the rate it does whatever it does.
For limiting, I made some custom changes to the config:
/etc/e-smith/templates-custom/etc/postfix/main.cf/99smtplimit
#default_process_limit = 10
default_destination_concurrency_limit = 2
smtp_destination_recipient_limit = 2
smtp_destination_concurrency_limit = 2
smtp_destination_rate_delay = 10s
/etc/e-smith/templates-custom/etc/postfix/master.cf/99smtplimit
smtp inet n - n - 10 smtpd
-o smtpd_helo_required=yes
-o strict_rfc821_envelopes=yes
But these did not help - I guess these are for outgoing emails only, not for responses to incoming emails. Still, it would be only a workaround by slowing the whole thing down.
Maybe we could add something to the Nethserver config that is possibly missing or misconfigured? Or is this a postfix issue and I should ask them instead?
So the questions:
- Could we force postfix to pick 1 IP from this list, and use only that one instead of flooding all of them? Or even handle the DNS records manually for the few problematic domains.
- Alternatively, could we limit the response rate of these incoming feedbacks?
- Or make this parallel response sequential - that only continues if the first IP fails?
Bonus:
- What does postfix respond to (or ask from) all of these IPs?
Thanks for any input, I am pretty much lost and annoyed due to this long lasting problem that strikes back every time I think I have a solution…
