E-Mail wrongly delivered to account given in "Accept unknown recipients"

uliversal · February 16, 2017, 7:52pm

NethServer release 7.3.1611, nethserver-mail-server

Deliver locally: if the checkbox “Accept unknown recipients” is checked and an account is given, all E-Mail is wrongly delivered to this account and not the original (existing!) recipient.

How to reproduce:

1 - Create Email domain “xyz.intern”, choose “Deliver locally”, don’t check “Accept unknown recipients”. In my server that’s the only Email domain.

2 - Create user “test@xyz.intern” (admin and administrator user already exist)

3 - Send an E-Mail (e.g. with sendmail) to “test@xyz.intern” (sendmail test@xyz.intern < sometext.txt)

4 - Look at /var/log/maillog: E-Mail is correctly delivered to "test@xyz.intern"

Feb 16 19:58:43 zeus postfix/pickup[6461]: EE026401268CD: uid=0 from=<root>
Feb 16 19:58:43 zeus postfix/cleanup[6467]: EE026401268CD: message-id=<20170216185843.EE026401268CD@zeus.xyz.intern>
Feb 16 19:58:43 zeus postfix/qmgr[6462]: EE026401268CD: from=<root@zeus.xyz.intern>, size=878, nrcpt=1 (queue active)
Feb 16 19:58:43 zeus dovecot: lmtp(6470): Connect from local
Feb 16 19:58:44 zeus dovecot: lmtp(6470, test@xyz.intern): BQB7O2P2pVhGGQAA4kELwg: sieve: msgid=<20170216185843.EE026401268CD@zeus.xyz.intern>: stored mail into mailbox 'INBOX'
Feb 16 19:58:44 zeus postfix/lmtp[6469]: EE026401268CD: to=<test@xyz.intern>, relay=zeus.xyz.intern[/var/run/dovecot/lmtp], delay=0.24, delays=0.12/0/0/0.12, dsn=2.0.0, status=sent (250 2.0.0 <test@xyz.intern> BQB7O2P2pVhGGQAA4kELwg Saved)
Feb 16 19:58:44 zeus dovecot: lmtp(6470): Disconnect from local: Successful quit
Feb 16 19:58:44 zeus postfix/qmgr[6462]: EE026401268CD: removed

5 - Update Email domain. Check “Accept unknown recipients”, set “Deliver to” = “administrator@xyz.intern”.

6 - Send an E-Mail to “test@xyz.intern” (same as before).

7 - Look at /var/log/maillog: E-Mail is wrongly delivered to "administrator@xyz.intern"

Feb 16 20:00:19 zeus postfix/pickup[6637]: B43F7401268D3: uid=0 from=<root>
Feb 16 20:00:19 zeus postfix/cleanup[6646]: B43F7401268D3: message-id=<20170216190019.B43F7401268D3@zeus.xyz.intern>
Feb 16 20:00:19 zeus postfix/qmgr[6638]: B43F7401268D3: from=<root@zeus.xyz.intern>, size=878, nrcpt=1 (queue active)
Feb 16 20:00:19 zeus dovecot: lmtp(6649): Connect from local
Feb 16 20:00:19 zeus dovecot: lmtp(6649, administrator@xyz.intern): +hNDLsP2pVj5GQAA4kELwg: sieve: msgid=<20170216190019.B43F7401268D3@zeus.xyz.intern>: stored mail into mailbox 'INBOX'
Feb 16 20:00:19 zeus postfix/lmtp[6648]: B43F7401268D3: to=<administrator@xyz.intern>, orig_to=<test@xyz.intern>, relay=zeus.xyz.intern[/var/run/dovecot/lmtp], delay=0.27, delays=0.13/0/0/0.14, dsn=2.0.0, status=sent (250 2.0.0 <administrator@xyz.intern> +hNDLsP2pVj5GQAA4kELwg Saved)
Feb 16 20:00:19 zeus dovecot: lmtp(6649): Disconnect from local: Successful quit
Feb 16 20:00:19 zeus postfix/qmgr[6638]: B43F7401268D3: removed

Repeated the setup (mail domain/users) many times, every time the same behaviour.

uliversal · February 16, 2017, 8:00pm

A notice: if I add test@xyz.de in the accounts section of /etc/postfix/virtual

#
# 20accounts -- ensure local delivery for our accounts
#
test@xyz.intern test@xyz.intern

and do a

postmap /etc/postfix/virtual

and again send an E-Mail to test@xyz.de, it is delivered correctly. Maybe this helps in solving the problem?

davidep · February 16, 2017, 8:46pm

Great job @uliversal!

Is this already reported in your post about multi drop getmail setup?

uliversal · February 17, 2017, 6:43am

Thank you @davidep!
Your’re right, this is already reported in the post you mentioned:

But for this bug report I have isolated the problem and led it back to a simple and reproducible scenario. I hope this is helpful for solving the problem.

davidep · February 17, 2017, 8:05am

I really appreciate your effort! I’ve filed a new bug, next week I hope I’ll get the time to fix it!

davidep · February 21, 2017, 10:07am

Absolutely! I need your help to test the bug fix for the catchall account! It comes with the support for the primary domain record removal. I’ve completed the dovecot/postfix configuration and I’m working on the web interface validators. Just a few hours…

davidep · February 21, 2017, 5:10pm

You can install the RPM from nethserver-testing repository:

yum --enablerepo=nethserver-testing update nethserver-mail-server-1.10.8-1.5.g8683219.ns7.noarch

alefattorini · February 21, 2017, 5:19pm

Such a quick solution WDYT @uliversal ?
@davidep is your man

uliversal · February 21, 2017, 8:19pm

Too quick for me!
I’m totally surprised, great!

So I turned on the server, sat at my notebook and tested…

[EDIT: forgot to mention that I installed the testing RPM]

First of all, I tried to reset NethServer. But sadly with some problems (but they have been still there yesterday evening, before installing the testing RPM):
I deinstalled Samba, did a Factory reset (as described here and also a /etc/e-smith/events/actions/system-adjust). After that I removed the bridge for the Active Directory.
When trying to change the Server name (to zeus.xyz.intern), I got the following error:

Task completed with errors
Adjust service nsdc #309 (exit status 1)
failed

As I don’t have the time today for a complete reinstall, I did go one with re-installing Samba Active Directory.
After that I tried to set a password for admin and administrator, but that failed with

ERROR 404
404 - Not foundadmin@xyz.intern
1322148399

I ignored that and set up three new users: mickey@xyz.intern, goofy@xyz.intern and test@xyz.intern.

Mail domain was updated as seen in the following image (Deliver locally, checked Accept unknown recipients and Deliver to test@xyz.intern):

Test case 1: do an internal sendmail to mickey@xyz.intern, expected result: E-Mail in Mickey’s mailbox. Result: OK!

Test case 2: do an internal sendmail to goofy@xyz.intern, expected result: E-Mail in Goofy’s mailbox. Result: OK!

Test case 3: do an internal sendmail to pluto@xyz.intern, expected result: E-Mail in in Test’s mailbox. Result: OK!

Mail domain was updated as seen in the following image (Deliver locally, unchecked Accept unknown recipients):

Test case 4: do an internal sendmail to mickey@xyz.intern, expected result: E-Mail in Mickey’s mailbox. Result: FAIL, failure E-Mail (unknown recipient) sent to root@zeus.xyz.intern (as sender of the E-Mail)

Test case 5: do an internal sendmail to goofy@xyz.intern, expected result: E-Mail in Goofy’s mailbox. Result: FAIL, failure E-Mail as in test case 4

Test case 6: do an internal sendmail to pluto@xyz.intern, expected result: E-Mail returned to sender, because user does not exist. Result: OK

Excpert of /var/log/maillog

Feb 21 21:01:55 zeus postfix/pickup[30660]: E79EC4013B48B: uid=0 from=<root>
Feb 21 21:01:55 zeus postfix/cleanup[30693]: E79EC4013B48B: message-id=<20170221200155.E79EC4013B48B@zeus.xyz.intern>
Feb 21 21:01:56 zeus postfix/qmgr[30661]: E79EC4013B48B: from=<root@zeus.xyz.intern>, size=842, nrcpt=1 (queue active)
Feb 21 21:01:56 zeus dovecot: lmtp(30696): Connect from local
Feb 21 21:01:56 zeus postfix/lmtp[30695]: E79EC4013B48B: to=<mickey@xyz.intern>, relay=zeus.xyz.intern[/var/run/dovecot/lmtp], delay=0.32, delays=0.24/0/0/0.08, dsn=5.1.1, status=bounced (host zeus.xyz.intern[/var/run/dovecot/lmtp] said: 550 5.1.1 <mickey@xyz.intern> User doesn't exist: mickey@xyz.intern (in reply to RCPT TO command))
Feb 21 21:01:56 zeus dovecot: lmtp(30696): Disconnect from local: Successful quit
Feb 21 21:01:56 zeus postfix/cleanup[30693]: 19EA5401BFB4A: message-id=<20170221200156.19EA5401BFB4A@zeus.xyz.intern>
Feb 21 21:01:56 zeus postfix/bounce[30699]: E79EC4013B48B: sender non-delivery notification: 19EA5401BFB4A
Feb 21 21:01:56 zeus postfix/qmgr[30661]: 19EA5401BFB4A: from=<>, size=2793, nrcpt=1 (queue active)
Feb 21 21:01:56 zeus postfix/qmgr[30661]: E79EC4013B48B: removed
Feb 21 21:01:56 zeus dovecot: lmtp(30696): Connect from local
Feb 21 21:01:56 zeus dovecot: lmtp(30696, root@zeus.xyz.intern): LnVgAbScrFjodwAAzzd1uA: sieve: msgid=<20170221200156.19EA5401BFB4A@zeus.xyz.intern>: stored mail into mailbox 'INBOX'
Feb 21 21:01:56 zeus postfix/lmtp[30695]: 19EA5401BFB4A: to=<root@zeus.xyz.intern>, relay=zeus.xyz.intern[/var/run/dovecot/lmtp], delay=0.18, delays=0.07/0/0/0.11, dsn=2.0.0, status=sent (250 2.0.0 <root@zeus.xyz.intern> LnVgAbScrFjodwAAzzd1uA Saved)
Feb 21 21:01:56 zeus dovecot: lmtp(30696): Disconnect from local: Successful quit
Feb 21 21:01:56 zeus postfix/qmgr[30661]: 19EA5401BFB4A: removed

Excerpt of the failure notification:

Return-Path: <>
Delivered-To: <root@zeus.xyz.intern>
Received: from zeus.xyz.intern
        by zeus.xyz.intern (Dovecot) with LMTP id 4upNHYugrFhjeAAAzzd1uA
        for <root@zeus.xyz.intern>; Tue, 21 Feb 2017 21:18:19 +0100
Received: by zeus.xyz.intern (Postfix)
        id 8D303401268F8; Tue, 21 Feb 2017 21:18:19 +0100 (CET)
Date: Tue, 21 Feb 2017 21:18:19 +0100 (CET)
From: MAILER-DAEMON@zeus.xyz.intern (Mail Delivery System)
Subject: Undelivered Mail Returned to Sender
To: root@zeus.xyz.intern
Auto-Submitted: auto-replied
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
        boundary="62F9F4013B48B.1487708299/zeus.xyz.intern"
Message-Id: <20170221201819.8D303401268F8@zeus.xyz.intern>

This is a MIME-encapsulated message.

--62F9F4013B48B.1487708299/zeus.xyz.intern
Content-Description: Notification
Content-Type: text/plain; charset=us-ascii

This is the mail system at host zeus.xyz.intern.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

                   The mail system

<mickey@xyz.intern>: host zeus.xyz.intern[/var/run/dovecot/lmtp] said: 550
    5.1.1 <mickey@xyz.intern> User doesn't exist: mickey@xyz.intern (in reply
    to RCPT TO command)

@davidep, can you check the test cases 4 and 5 in your system? If they fail because of my crappy NethServer configuration, I have to reinstall the complete server and re-test them. (But not today, I have to care about my one year old daughter…)

I think tomorrow evening I will have time to go on with the tests (second mail domain xyz.de, remove initial mail domain, check Multidrop POP3 connector)

By the way: do you have any standards/requirements/specifications for testing?

[Edit: revised test case 6, corrected spelling mistakes]

alefattorini · February 21, 2017, 9:09pm

@quality_team is what you’re looking for. /cc @dz00te

davidep · February 22, 2017, 8:59am

Hi @uliversal, many thanks for your test! Luckily, I couldn’t reproduce the failed cases above

sendmail -bv nonexists@dpnet.nethesis.it first.user@dpnet.nethesis.it admin@dpnet.nethesis.it

Output:

This is the mail system at host vm5.dpnet.nethesis.it.

Enclosed is the mail delivery report that you requested.

                   The mail system

: delivery via
    vm5.dpnet.nethesis.it[/var/run/dovecot/lmtp]: 250 2.1.5 OK

: delivery via
    vm5.dpnet.nethesis.it[/var/run/dovecot/lmtp]: 250 2.1.5 OK

: delivery via
    vm5.dpnet.nethesis.it[/var/run/dovecot/lmtp]: host
    vm5.dpnet.nethesis.it[/var/run/dovecot/lmtp] said: 550 5.1.1
     User doesn't exist:
    nonexists@dpnet.nethesis.it (in reply to RCPT TO command)

BTW, usually we configure applications to send mail via TCP 587 submission port to enable spam filter, virus scanner and disclaimer text features.

uliversal · February 22, 2017, 9:14am

Ciao @davidep, thank you very much. So I have to reinstall my NethServer for meaningful tests

Why do you mention the TCP port 587?

davidep · February 22, 2017, 9:17am

The sendmail command drops the message directly in Postfix queue via a local unix socket (I suppose). This is not the habitual route of messages coming from the internet, LAN clients or local applications such as Roundcube, SOGo, WebTop…

It is not wrong, but it is only a partial test coverage!

davidep · February 22, 2017, 9:19am

Sometimes reinstall takes less time and effort, and leads to a better result than troubleshooting a problem!

For testing purposes I recommend a VM, of course!

alefattorini · February 22, 2017, 11:05am

Snapshotting the VM before the testing rpms installation and reverting if needed

dcoelho · February 22, 2017, 11:18am

come on, reinstalling a linux machine is like a walk in the park. Courage and good work.

uliversal · February 22, 2017, 8:22pm

@dcoelho:
A walk in a huge park…
A long walk in a huge park…
have seen many interesting trees in this one and a half our…

@davidep: the test cases 4 and 5 also look very good in the new installation! Well done!

Update:

Test case 4: do an internal sendmail to mickey@xyz.intern, expected result: E-Mail in Mickey’s mailbox. Result: OK!

Test case 5: do an internal sendmail to goofy@xyz.intern, expected result: E-Mail in Goofy’s mailbox. Result: OK!

I will test the other issues tomorrow (second mail domain xyz.de, remove initial mail domain, check Multidrop POP3 connector)

(never used and installed a VM so far, this is a task for a long weekend)

davidep · February 22, 2017, 8:34pm

Great news!! Awesome

You’ll still need Filippo’s tweak to getmail setup!

dcoelho · February 23, 2017, 10:33am

@uliversal maybe this link can help you with VM:
How-To: Create a NethServer instance in VirtualBox machine

have a nice “long weekend”.

uliversal · February 23, 2017, 7:48pm

Going on…
Now I deleted the Email Domain “xyz.intern”. So there is no more Email domain.

Test case 7: internally send a mail to mickey@xyz,intern (existing user); expected result: E-Mail returned to sender; actual result: E-Mail in Mickey’s mailbox

Test case 8: internally send a mail to pluto@xyz,intern (not-existing user); expected result: E-Mail returned to sender; actual result: E-Mail goes to nirvana (can’t see where it is delivered, roundcube says: “successfully sent”)

@davidep: please have a look at test case 7 and 8 if my expected result or the actual result is the defined functionality. As I see know the mail server doesn’t care about the missing Email domain?

Now I created a new Email Domain “xyz.de” and for each of the defined users (mickey, goofy and catchall) a Mail alias (e.g. Email address: mickey@[all domains], Destinations: mickey@xyz.intern.
If I now open roundcube and change the “Email” in Settings/Identities/mickey@xyz.intern to mickey@xyz.de I can send an E-Mail as mickey@xyz.de.

One flaw: in roundcube the suggestion in the “To” input field is still the @xyz.intern address (e.g. goofy@xyz.intern when I typed “go”). The Mail alias (e.g. goofy@xyz.de) doesn’t appear in the suggestions.

That’s very odd for the normal user, I expect. Is there a chance to remove the addresses being in an deleted Email domain from the suggestions and display the Email aliases instead?

What I also don’t like in this scenario (with the xyz.intern domain): The external E-Mail address (e.g. mickey@xyz.de) is not stored in the user object. So if a user is created/deleted you always have to consider creating/deleting the external Mail alias. In my opinion it would be better to have one (or more) additional field(s) in the user object where I can put E-Mail address(es).

To be continued…