Discarding duplicate emails send to groups

So I created three user groups, and added myself to all three of them, then sent an email to all three groups. I received 3 copies of the email. Correct me if I’m wrong, but my assumption is that this configuration should make it so I only receive 1 copy.

Here are the logs from /var/log/imap

Sep 20 13:43:59 network dovecot: imap(brad.trammell@domain.com): save: box=Sent, uid=1634, msgid=<030d01d7ae47$18861370$49923a50$@domain.com>, from="Brad J. Trammell" <brad.trammell@domain.com>, subject=Test, flags=(\Seen)
Sep 20 13:43:59 network dovecot: imap-login: Login: user=<brad.trammell>, method=PLAIN, rip=XX.XX.XX.XX, lip=XX.XX.XX.XX, mpid=47528, TLS, session=<nhR203DMGURHc5Yi>
Sep 20 13:44:00 network dovecot: imap(brad.trammell@domain.com): Connection closed (UID FETCH finished 0.035 secs ago) in=342 out=6737
Sep 20 13:44:00 network dovecot: imap-login: Login: user=<brad.trammell>, method=PLAIN, rip=XX.XX.XX.XX, lip=XX.XX.XX.XX, mpid=47550, TLS, session=<cVGC03DM8rhHc5Yi>
Sep 20 13:44:01 network dovecot: imap-login: Login: user=<brad.trammell>, method=PLAIN, rip=XX.XX.XX.XX, lip=XX.XX.XX.XX, mpid=47567, TLS, session=<nZCL03DMkPpHc5Yi>
Sep 20 13:44:01 network dovecot: imap(brad.trammell@domain.com): Connection closed (UID FETCH finished 0.431 secs ago) in=244 out=11960
Sep 20 13:44:01 network dovecot: imap(brad.trammell@domain.com): Connection closed (SEARCH finished 0.022 secs ago) in=388 out=2400
Sep 20 13:44:06 network dovecot: imap(brad.trammell@domain.com): Connection closed (IDLE running for 0.001 + waiting input for 6.162 secs, 2 B in + 10 B out, state=wait-input) in=290 out=16271
Sep 20 13:44:06 network dovecot: imap-login: Login: user=<brad.trammell>, method=PLAIN, rip=XX.XX.XX.XX, lip=XX.XX.XX.XX, mpid=47774, TLS, session=<3FHZ03DMFmhHc5Yi>
Sep 20 13:44:06 network dovecot: imap-login: Login: user=<brad.trammell>, method=PLAIN, rip=XX.XX.XX.XX, lip=XX.XX.XX.XX, mpid=47779, TLS, session=<rr7d03DM6MFHc5Yi>
Sep 20 13:44:06 network dovecot: imap(brad.trammell@domain.com): Connection closed (SEARCH finished 0.023 secs ago) in=457 out=2445
Sep 20 13:44:12 network dovecot: imap(brad.trammell@domain.com): Logged out in=6951 out=49129
Sep 20 13:44:13 network dovecot: imap-login: Login: user=<brad.trammell@domain.com>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=47932, secured, session=<PklB1HDM+IJ/AAAB>
Sep 20 13:44:15 network dovecot: imap(brad.trammell@domain.com): Logged out in=1535 out=36078

Any clues on what to do next?

@support_team
Any idea here?

As explained in the above link, from dovecot 2.2.18 the duplicate plugin is named just “duplicate” (instead of vnd.dovecot.duplicate) and available by default so the custom template /etc/e-smith/templates-custom/etc/dovecot/dovecot.conf/60lmtp is not needed anymore.

So creating the file /var/lib/nethserver/sieve-scripts/after.sieve with following content should be enough.

#  do not deliver duplicate mails

require "duplicate";

if duplicate {
    discard;
    stop;
}

To be sure restart the mailserver services (maybe not necessary):

signal-event nethserver-mail-server-update

Now mails to groups with same members are delivered only once. I tested with Roundcube and Thunderbird.

6 Likes

I’m not sure why, but after following these steps, I am still receiving duplicates. Should I do a full reboot of the server as well?

I tested it on another server, no reboot or service restart needed. After creating /var/lib/nethserver/sieve-scripts/after.sieve it just worked.

For testing I created 2 groups group1 and group2 and user markus is member of those groups.
I sent a testmail from an external mail account to group1, group2 and markus and markus gets 1 mail.

In /var/log/maillog I can see 1 mail stored in the inbox but the next 2 are discarded:

Sep 28 21:25:57 nethserver dovecot: lmtp(markus@mrmarkuz.org): IADPM0VsU2HsRwAAGa89gA: sieve: msgid=<0909d95b-0ed0-f24e-bf83-cfdc67a2811b@domain.at>: stored mail into mailbox 'INBOX'
Sep 28 21:25:57 nethserver postfix/lmtp[18411]: 10B425C05A9: to=<markus@mrmarkuz.org>, relay=nethserver.mrmarkuz.org[/var/run/dovecot/lmtp], delay=0.92, delays=0.83/0.01/0.02/0.06, dsn=2.0.0, status=sent (250 2.0.0 <markus@mrmarkuz.org> IADPM0VsU2HsRwAAGa89gA Saved)
Sep 28 21:25:57 nethserver dovecot: lmtp(markus@mrmarkuz.org): IADPM0VsU2HsRwAAGa89gA:2: sieve: msgid=<0909d95b-0ed0-f24e-bf83-cfdc67a2811b@domain.at>: marked message to be discarded if not explicitly delivered (discard action)
Sep 28 21:25:57 nethserver postfix/lmtp[18411]: 10B425C05A9: to=<markus@mrmarkuz.org>, orig_to=<devs@mrmarkuz.org>, relay=nethserver.mrmarkuz.org[/var/run/dovecot/lmtp], delay=0.93, delays=0.83/0.01/0.02/0.06, dsn=2.0.0, status=sent (250 2.0.0 <markus@mrmarkuz.org> IADPM0VsU2HsRwAAGa89gA:2 Saved)
Sep 28 21:25:57 nethserver dovecot: lmtp(markus@mrmarkuz.org): IADPM0VsU2HsRwAAGa89gA:3: sieve: msgid=<0909d95b-0ed0-f24e-bf83-cfdc67a2811b@domain.at>: marked message to be discarded if not explicitly delivered (discard action)

I sent a test to my 3 groups. itemailtest1, itemailtest2 and itemailtest3. And received it three times. Here is the log from my /var/log/maillog

Sep 28 15:55:06 network rspamd[3915]: <9fc7ef>; proxy; rspamd_mime_part_detect_language: detected part language: en
Sep 28 15:55:06 network rspamd[3915]: <9fc7ef>; proxy; rspamd_mime_part_detect_language: detected part language: en
Sep 28 15:55:06 network rspamd[3915]: <9fc7ef>; lua; settings.lua:363: <0b1801d7b4a2$bc6adaa0$35408fe0$@domain.com> apply static settings authenticated (id = 1937017268); authenticated matched; priority high
Sep 28 15:55:06 network rspamd[3915]: <9fc7ef>; proxy; rspamd_add_passthrough_result: <0b1801d7b4a2$bc6adaa0$35408fe0$@domain.com>: set pre-result to 'no action' (no score): 'Matched map: FROM_SUBDOMAINS_WHITELIST' from multimap(1)
Sep 28 15:55:06 network rspamd[3915]: <9fc7ef>; proxy; rspamd_task_write_log: id: <0b1801d7b4a2$bc6adaa0$35408fe0$@domain.com>, qid: <126845A45>, ip: XX.XX.XX.XX, user: brad.trammell, from: <brad.trammell@domain.com>, (default: F (no action): [0.00/20.00] [ASN(0.00){asn:701, ipnet:71.115.128.0/17, country:US;},FROM_SUBDOMAINS_WHITELIST(0.00){domain.com;}]), len: 3724, time: 1.785ms, dns req: 1, digest: <04ad92e139d5fc81009aef3afaf3f045>, rcpts: <itemailtest1@domain.com,itemailtest2@domain.com,itemailtest3@domain.com>, mime_rcpts: <itemailtest1@domain.com,itemailtest2@domain.com,itemailtest3@domain.com...>, forced: no action "Matched map: FROM_SUBDOMAINS_WHITELIST"; score=nan (set by multimap), settings_id: authenticated
Sep 28 15:55:06 network rspamd[3915]: <9fc7ef>; proxy; rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned, 0 regexps matched, 174 regexps total, 0 regexps cached, 0B scanned using pcre, 0B scanned total
Sep 28 15:55:06 network opendkim[2508]: 126845A45: DKIM-Signature field added (s=default, d=domain.com)
Sep 28 15:55:06 network postfix/qmgr[3880]: 126845A45: from=<brad.trammell@domain.com>, size=3959, nrcpt=6 (queue active)
Sep 28 15:55:06 network dovecot: lmtp(29798): Connect from local
Sep 28 15:55:06 network dovecot: lmtp(brad.trammell@domain.com): save: box=INBOX, uid=10686, msgid=<0b1801d7b4a2$bc6adaa0$35408fe0$@domain.com>, from="Brad J. Trammell" <brad.trammell@domain.com>, subject=Duplicate Test, flags=()
Sep 28 15:55:06 network dovecot: lmtp(brad.trammell@domain.com): msgid=<0b1801d7b4a2$bc6adaa0$35408fe0$@domain.com>: saved mail to INBOX
Sep 28 15:55:06 network postfix/lmtp[29797]: 126845A45: to=<brad.trammell@domain.com>, orig_to=<itemailtest1@domain.com>, relay=network.domain.com[/var/run/dovecot/lmtp], delay=0.19, delays=0.14/0.02/0.01/0.02, dsn=2.0.0, status=sent (250 2.0.0 <brad.trammell@domain.com> 8J/5DBpzU2FmdAAAcGdNhg Saved)
Sep 28 15:55:06 network dovecot: lmtp(brad.trammell@domain.com): save: box=INBOX, uid=10687, msgid=<0b1801d7b4a2$bc6adaa0$35408fe0$@domain.com>, from="Brad J. Trammell" <brad.trammell@domain.com>, subject=Duplicate Test, flags=()
Sep 28 15:55:06 network dovecot: lmtp(brad.trammell@domain.com): msgid=<0b1801d7b4a2$bc6adaa0$35408fe0$@domain.com>: saved mail to INBOX
Sep 28 15:55:06 network postfix/lmtp[29797]: 126845A45: to=<brad.trammell@domain.com>, orig_to=<itemailtest2@domain.com>, relay=network.domain.com[/var/run/dovecot/lmtp], delay=0.2, delays=0.14/0.02/0.01/0.02, dsn=2.0.0, status=sent (250 2.0.0 <brad.trammell@domain.com> 8J/5DBpzU2FmdAAAcGdNhg:2 Saved)
Sep 28 15:55:06 network dovecot: lmtp(brad.trammell@domain.com): save: box=INBOX, uid=10688, msgid=<0b1801d7b4a2$bc6adaa0$35408fe0$@domain.com>, from="Brad J. Trammell" <brad.trammell@domain.com>, subject=Duplicate Test, flags=()
Sep 28 15:55:06 network dovecot: lmtp(brad.trammell@domain.com): msgid=<0b1801d7b4a2$bc6adaa0$35408fe0$@domain.com>: saved mail to INBOX
Sep 28 15:55:06 network postfix/lmtp[29797]: 126845A45: to=<brad.trammell@domain.com>, orig_to=<itemailtest3@domain.com>, relay=network.domain.com[/var/run/dovecot/lmtp], delay=0.2, delays=0.14/0.02/0.01/0.03, dsn=2.0.0, status=sent (250 2.0.0 <brad.trammell@domain.com> 8J/5DBpzU2FmdAAAcGdNhg:3 Saved)
Sep 28 15:55:06 network dovecot: lmtp(email.test@domain.com): save: box=INBOX, uid=17, msgid=<0b1801d7b4a2$bc6adaa0$35408fe0$@domain.com>, from="Brad J. Trammell" <brad.trammell@domain.com>, subject=Duplicate Test, flags=()
Sep 28 15:55:06 network dovecot: lmtp(email.test@domain.com): msgid=<0b1801d7b4a2$bc6adaa0$35408fe0$@domain.com>: saved mail to INBOX
Sep 28 15:55:06 network postfix/lmtp[29797]: 126845A45: to=<email.test@domain.com>, orig_to=<itemailtest1@domain.com>, relay=network.domain.com[/var/run/dovecot/lmtp], delay=0.2, delays=0.14/0.02/0.01/0.03, dsn=2.0.0, status=sent (250 2.0.0 <email.test@domain.com> 8J/5DBpzU2FmdAAAcGdNhg:4 Saved)
Sep 28 15:55:06 network dovecot: lmtp(email.test@domain.com): save: box=INBOX, uid=18, msgid=<0b1801d7b4a2$bc6adaa0$35408fe0$@domain.com>, from="Brad J. Trammell" <brad.trammell@domain.com>, subject=Duplicate Test, flags=()
Sep 28 15:55:06 network dovecot: lmtp(email.test@domain.com): msgid=<0b1801d7b4a2$bc6adaa0$35408fe0$@domain.com>: saved mail to INBOX
Sep 28 15:55:06 network postfix/lmtp[29797]: 126845A45: to=<email.test@domain.com>, orig_to=<itemailtest2@domain.com>, relay=network.domain.com[/var/run/dovecot/lmtp], delay=0.2, delays=0.14/0.02/0.01/0.03, dsn=2.0.0, status=sent (250 2.0.0 <email.test@domain.com> 8J/5DBpzU2FmdAAAcGdNhg:5 Saved)
Sep 28 15:55:06 network dovecot: lmtp(email.test@domain.com): save: box=INBOX, uid=19, msgid=<0b1801d7b4a2$bc6adaa0$35408fe0$@domain.com>, from="Brad J. Trammell" <brad.trammell@domain.com>, subject=Duplicate Test, flags=()
Sep 28 15:55:06 network dovecot: lmtp(email.test@domain.com): msgid=<0b1801d7b4a2$bc6adaa0$35408fe0$@domain.com>: saved mail to INBOX
Sep 28 15:55:06 network postfix/lmtp[29797]: 126845A45: to=<email.test@domain.com>, orig_to=<itemailtest3@domain.com>, relay=network.domain.com[/var/run/dovecot/lmtp], delay=0.21, delays=0.14/0.02/0.01/0.03, dsn=2.0.0, status=sent (250 2.0.0 <email.test@domain.com> 8J/5DBpzU2FmdAAAcGdNhg:6 Saved)
Sep 28 15:55:06 network dovecot: lmtp(29798): Disconnect from local: Successful quit
Sep 28 15:55:06 network postfix/qmgr[3880]: 126845A45: removed

Which mail client do you use for testing?

I have tested in both SoGO Webmail, and Outlook 2019. Both are doing the same thing.

Did you remove the custom template /etc/e-smith/templates-custom/etc/dovecot/dovecot.conf/60lmtp and applied the config with

signal-event nethserver-mail-server-update

Do we have the same dovecot versions?

[root@nethserver ~]# dovecot --version
2.2.36 (1f10bfa63)

Did you already test from an external mail account that’s not hosted on your server?

Yes. Just to be sure I checked again, that file does not exist.

Dovecot version appears to be the same.

[19:30] [network ~] # dovecot --version
2.2.36 (1f10bfa63)

Strangely enough, it appears to be working if it’s an external email address, but not an internal address sending to internal groups. Which defeats the purpose, as about 90% of all emails are internally to the groups.

Here it works internally with SOGo and Outlook 2016. That’s really weird.

Do you use internal domain names in mail addresses like mynetwork.local which cannot be resolved by public DNS?

Do you use Nethserver as Gateway (red and green interface) or Server (only green one) ? Here it works here on “Server mode” Nethservers.

Our active directory domain name is ad.domain.com, and our email domain is domain.com, where domain.com is our actual public domain name, but we don’t use any type of .local or .lan domain name for emails.

Nethserver is not our gateway for the LAN, as it sits behind a Ubiquiti USG-3, but it does have it’s own public IP address (it has two NIC, one internal LAN, and one WAN NIC), so I guess you’d call that Gateway mode.

Are you using local samba AD on the Nethserver or is there a Windows Server providing the AD?

It is SAMBA Active Directory provided by Nethserver.

can confirm it’s working on my system both external and internal

do you have anything else in /etc/e-smith/templates-custom/etc/dovecot/dovecot.conf/ apart from
40namespaces maybey you acciedentaly saved a misspelled 60lmtp
ie /etc/e-smith/templates-custom/etc/dovecot/dovecot.conf/6lmtp

[00:57] [network dovecot.conf] # ls -la /etc/e-smith/templates-custom/etc/dovecot/dovecot.conf/
total 0
drwxr-xr-x 2 root root  6 Sep 28 19:29 .
drwxr-xr-x 3 root root 26 Sep 20 03:01 ..
[00:57] [network dovecot.conf] #

According to the ls command that directory is completely empty.

Just for giggles, I did a search for files containing the word ‘duplicate’, in the e-smith folder and it’s sub-directories, and the search turned up no additional files with a similar line or filtering in there.

[01:14] [network ~] # grep -rnw '/etc/e-smith/' -e 'duplicate'
/etc/e-smith/events/actions/nethserver-samba-ns6sidfix:29:# Fix duplicate SID for local machine and domain (required for ns6 upgraded systems)
/etc/e-smith/templates/etc/shorewall/policy/20policy:24:        $zones{$zone} = ''; # avoid duplicate policies
/etc/e-smith/templates/etc/rspamd/rspamd.conf/30Logging:8:    .include(try=true; priority=1,duplicate=merge) "$LOCAL_CONFDIR/local.d/logging.inc"
/etc/e-smith/templates/etc/pulledpork/pulledpork.conf/20options:103:# de-dupe any duplicate IPs from different sources.
[01:14] [network ~] #

I’m honestly not sure where to go from here.

in case it helps this is a copy of my 40namespace

#
# Namespace setup
#

# Private mailboxes
namespace ROOT \{
  type = private
  separator = /
  prefix =
  # location defaults to mail_location.
  inbox = yes
  subscriptions = yes

  # Commonly used folders:
  mailbox Trash \{
    special_use = \Trash
    auto = no
  \}
  mailbox Drafts \{
    special_use = \Drafts
    auto = no
  \}
  mailbox Sent \{
    special_use = \Sent
    auto = no
  \}
  mailbox "Sent Messages" \{
    special_use = \Sent
    auto = no
  \}

\}

# Shared mailboxes are { $dovecot{SharedMailboxesStatus} }
namespace SHARED_USERS \{
  type = shared
  disabled = { $dovecot{SharedMailboxesStatus} eq 'enabled' ? 'no' : 'yes' }
  separator = /
  prefix = Shared/%%n@{{ $DomainName }}/
{
  $location = 'location = maildir:/var/lib/nethserver/vmail/%%u/Maildir';
  if (($dovecot{'SharedSeen'} || 'disabled' ) eq 'disabled') {
    $location .= ':INDEXPVT=~/Maildir/shared/%%u';
  }
  $OUT .= "  $location";
}
  subscriptions = no
  list = children
\}

# Public mailboxes
namespace PUBLIC \{
  type = public
  separator = /
  prefix = Public/
  subscriptions = no
  list = children
  location = maildir:/var/lib/nethserver/vmail/vmail/Maildir:INDEXPVT=~/Maildir/public
\}

# Enable acl plugin for shared mailboxes,
# and listescape to extend allowable characters in mailbox names
mail_plugins = $mail_plugins acl listescape {
    $OUT = '';
    if(($dovecot{'FtsLuceneStatus'} || '') eq 'enabled') {
        $OUT = 'fts fts_lucene';
    }
}
protocol imap \{
  mail_plugins = $mail_plugins imap_acl
\}

plugin \{
  acl = vfile
  acl_shared_dict = file:/var/lib/nethserver/vmail/shared-mailboxes.db
  {
    $OUT = '';
    if(($dovecot{'FtsLuceneStatus'} || '') eq 'enabled') {
        $OUT = "fts = lucene\n  fts_lucene = whitespace_chars=@.";
    }
  }
\}

service dict \{
  unix_listener dict \{
    mode = 0600
    user = vmail
  \}
\}

service imap-login \{
   unix_listener imap-ipc \{
     group = root
     user = $default_internal_user
     mode = 0600
   \}
\}

# auth_debug = yes
# auth_verbose = yes

the above file i know has to do with creating the mailboxes i dont know if yours is set differently

The file you sent is identical to the one on the server in question.