Discarding duplicate emails send to groups

Support
1

NethServer Version: 7.2009
Module: Email/Groups

I’m not sure if that has been asked before, but I was unable to find it anywhere. I’m having an issue with duplicate emails being sent to groups, particularly multiple groups that contain some of the same users, and my users are starting to complain about multiple copies of the same email.

For example, lets say I have 3 email groups. sales@domain.com, support@domain.com, billing@domain.com. Now for the purpose of this example, Sales and Billing would have relatively the same users. Plus or minus a few.

The issue we are running into is that if I am a member of Sales, and I send an email to Sales, CC’ing Billing, and I’m on that as well, I will receive 2 copies of the email I just sent to those two lists. One for each of them. Now normally this wouldn’t be a problem, if you had email groups that had unique user memberships, but in my (and probably other peoples case too), the biggest problem we’re having is that when a user sends an email to a large group of users, and those users reply, the email doubles, triples, quadruples, etc. with each reply. Turning one or two emails, into 10, 30, 50 emails.

Is there a way programmatically, or other to have Nethserver discard duplicate emails being sent to the same user, when they are members of multiple groups attached to the email regardless of what group it came into?

2

This is not the answer you were looking for, but maybe can be a “client side” option, if the user is already Thunderbird adopter.


The addon is able to research into a single folder if there are any kind duplicate messages, allowing to keep different messages. Duplicates can be moved o straight deleted.

It’s not… fast.
At least, it was not fast into a local folder with more than 10k messages on SSD disk, during the lookup phase. Deletion was quite quick.

As any Thunderbird extension, it’s crossOS (MacOS, Linux, Windows)

3

Could this be a good place to start?

https://wiki.dovecot.org/Tools/Doveadm/Deduplicate

5 Likes
4

question is: may this nice Dovecot tool can be integrated into Nethserver and activated by roundcube/webtop?

2 Likes
5

The right thing is that it should work regardless of the Client used.

1 Like
6

Most our users are using mobile clients.

7

It should just be a matter of adding it to the templates.

If you are using a reasonably recent Dovecot with Pigeonhole implementation of Sieve , there is a special extension that can be used for on-delivery deduplication. In Dovecot older than 2.2.18, the name of this extension is “vnd.dovecot.duplicate” and it’s disabled by default, in newer releases the extension is called “duplicate” and should be already enabled, but enabling it explicitly won’t harm.

That is exactly what I’m looking for.

8

Tried this, can’t seem to get it integrated into nethserver, or working.

9

I use a custom template. Ping me next week, I can’t access the file now

5 Likes
10

Apparently this is built into Dovecot, it just has to be enabled.

https://wiki2.dovecot.org/Pigeonhole/Sieve/Extensions/Duplicate

@filippo_carletti, I will reach out next week to get a look at your template.

11

The customization is a bit too complex, I’m open to suggestions on how to improve it. We could also evaluate having it as a standard option, but my opinion is that if you need deduplication you are “abusing” groups instead of preferring shared/public folders.

You need two files:

  1. /var/lib/nethserver/sieve-scripts/after.sieve
#  do not deliver duplicate mails

require "vnd.dovecot.duplicate";

if duplicate {
    discard;
    stop;
}
  1. /etc/e-smith/templates-custom/etc/dovecot/dovecot.conf/60lmtp to overwrite the original file to add the dovecot extension (copy the original file and modify the extensions line as below)
    ...
    sieve_extensions = +imapflags +editheader +vnd.dovecot.duplicate
    ...
2 Likes
12

Am I wrong or… deduplication plugins acts at deliver? This is quite different of what I understood… The extensions suggested on Thunderbird acts “on call” on a single folder (IMAP or Local) allowing the choice for which message keep and what to do with the “other/s”

13

Most of my users are using Mobile clients such as Outlook for Mobile or BlueMail. Client plugins won’t help in my situation.

14

For the use case I have this system in production for, shared & public folders are not feasible unless they are displayed only to the users specified. Groups in my production environment are used for security groups, and email distribution.

I’m open for having this be a standard option. I think it would make life a lot easier for other users who may have a similar issue.

15

This file or the parent directory doesn’t exist on either of my two production systems. Is it safe to create it and not have it cause any issues?

16

Maybe I already read it?

Yeah, I did.

I’m sorry if it does not fit your case, but maybe could be a good enough solution for someone else.
And anyway, FWIW i share some of the concerns expressed by @filippo_carletti about sub-optimal use of groups of users.
Anyway, acting on delivery messages to the mailboxes, instead of post-processing them with appropriated rules (NethServer use Sieve, but you can still look for procmail if you’re looking for something a bit more… seasoned) can achieve something… similar after the message delivery to the inbox and avoiding some “messages disappearing” cases due to delivery rules…

17

Yes, it’s safe (more info here).

After creating the file, apply the new config with:

signal-event nethserver-mail-server-update
1 Like
18

I have implemented this and am still receiving duplicates. I’m wondering if maybe there is an error somewhere?

19

I don’t know, I’ve never tried it :slight_smile:
Let’s wait @filippo_carletti we he will have time (he’s currently on vacation)

20

I know the customization is working, but I can’t access the system where it is installed now to double-check.
@bradjtrammell have a look at /var/log/imap. I’ll look into this next week if you can’t fix it (please remind me).

3 Likes