Discarding duplicate emails send to groups

NethServer Version: 7.2009
Module: Email/Groups

I’m not sure if that has been asked before, but I was unable to find it anywhere. I’m having an issue with duplicate emails being sent to groups, particularly multiple groups that contain some of the same users, and my users are starting to complain about multiple copies of the same email.

For example, lets say I have 3 email groups. sales@domain.com, support@domain.com, billing@domain.com. Now for the purpose of this example, Sales and Billing would have relatively the same users. Plus or minus a few.

The issue we are running into is that if I am a member of Sales, and I send an email to Sales, CC’ing Billing, and I’m on that as well, I will receive 2 copies of the email I just sent to those two lists. One for each of them. Now normally this wouldn’t be a problem, if you had email groups that had unique user memberships, but in my (and probably other peoples case too), the biggest problem we’re having is that when a user sends an email to a large group of users, and those users reply, the email doubles, triples, quadruples, etc. with each reply. Turning one or two emails, into 10, 30, 50 emails.

Is there a way programmatically, or other to have Nethserver discard duplicate emails being sent to the same user, when they are members of multiple groups attached to the email regardless of what group it came into?

This is not the answer you were looking for, but maybe can be a “client side” option, if the user is already Thunderbird adopter.


The addon is able to research into a single folder if there are any kind duplicate messages, allowing to keep different messages. Duplicates can be moved o straight deleted.

It’s not… fast.
At least, it was not fast into a local folder with more than 10k messages on SSD disk, during the lookup phase. Deletion was quite quick.

As any Thunderbird extension, it’s crossOS (MacOS, Linux, Windows)

Could this be a good place to start?

https://wiki.dovecot.org/Tools/Doveadm/Deduplicate

5 Likes

question is: may this nice Dovecot tool can be integrated into Nethserver and activated by roundcube/webtop?

2 Likes

The right thing is that it should work regardless of the Client used.

Most our users are using mobile clients.

It should just be a matter of adding it to the templates.

If you are using a reasonably recent Dovecot with Pigeonhole implementation of Sieve , there is a special extension that can be used for on-delivery deduplication. In Dovecot older than 2.2.18, the name of this extension is “vnd.dovecot.duplicate” and it’s disabled by default, in newer releases the extension is called “duplicate” and should be already enabled, but enabling it explicitly won’t harm.

That is exactly what I’m looking for.

Tried this, can’t seem to get it integrated into nethserver, or working.

I use a custom template. Ping me next week, I can’t access the file now

4 Likes

Apparently this is built into Dovecot, it just has to be enabled.

https://wiki2.dovecot.org/Pigeonhole/Sieve/Extensions/Duplicate

@filippo_carletti, I will reach out next week to get a look at your template.

The customization is a bit too complex, I’m open to suggestions on how to improve it. We could also evaluate having it as a standard option, but my opinion is that if you need deduplication you are “abusing” groups instead of preferring shared/public folders.

You need two files:

  1. /var/lib/nethserver/sieve-scripts/after.sieve
#  do not deliver duplicate mails

require "vnd.dovecot.duplicate";

if duplicate {
    discard;
    stop;
}
  1. /etc/e-smith/templates-custom/etc/dovecot/dovecot.conf/60lmtp to overwrite the original file to add the dovecot extension (copy the original file and modify the extensions line as below)


    sieve_extensions = +imapflags +editheader +vnd.dovecot.duplicate

Am I wrong or… deduplication plugins acts at deliver? This is quite different of what I understood… The extensions suggested on Thunderbird acts “on call” on a single folder (IMAP or Local) allowing the choice for which message keep and what to do with the “other/s”