Pop3 connector - emails re-dowloaded

pop3connector
mail
v7

(Jose Manuel Insausti) #1

NethServer Version: NethServer release 7.6.1810 (final)
Module: get_mail

Versión del núcleo
3.10.0-957.1.3.el7.x86_64

I’m using the POP3 Connector to download my ISP’s emails.

Since yesterday, the emails are downloaded over and over again. It looks like it doesn’t mark them as unloaded.

In messages I see the download and I see that difference new and old.


(Markus Neuberger) #2

Just a blind shot:

Could it be you have a full mailbox or are over quota?


(Stéphane de Labrusse) #3

Indeed there was something strange last days on the same topic


(LR) #4

i deleted the mails from the provider and set the retrieval time to 60 minutes


(LR) #5

i have done a backup of the server configuration before the update. so i’m not so sure what caused the problem anymore, because the error always had a latent after changes.

https://community.nethserver.org/t/pop3-connector-no-longer-deletes-mails/11474/2


(Jose Manuel Insausti) #6

It is not a problem of the quota. no user exceeds 30%


(Marc) #7

Can you identify duplicates in the logs (msgid)?
Do /var/lib/getmail/oldmail* files get updated?


(Jose Manuel Insausti) #8

I see the following in one of the files

oldmail-mail.dominio.com-993-ttintoayo@dominio.com-INBOX
1240220693/41528^@1543332001
1240220693/41529^@1543397401
1240220693/41522^@1542893701
1240220693/41523^@1542899102
1240220693/41520^@1542879602
1240220693/41521^@1542887101
1240220693/41526^@1543312201
1240220693/41527^@1543320301
1240220693/41524^@1542966301
1240220693/41525^@1543231802
1240220693/41548^@1544565901
1240220693/41549^@1544603402
1240220693/41540^@1543917302
1240220693/41541^@1544018101
1240220693/41542^@1544018101
1240220693/41543^@1544176501
1240220693/41544^@1544433002
1240220693/41545^@1544434502
1240220693/41546^@1544527201

I have configured the pop3 service to delete messages immediately upon download.

neither does it eliminate them from the mail provider.


(Samuel Grenier) #9

Sorry for my bad english.

After update to 7.6.1810, I have the same problem, emails are re-downloaded with a dovecot message error delivery in logs. If I set disk space to unlimited the problem disappears, but I have no quota anymore. Is there any solution ?


(Marc) #10

This seems a #bug. I’ve done some tests, and as @kunstlust and @samuel_grenier found, when a quota is set under Email -> Mailboxes -> disk space: apply quota, we get a warning and email messages are not deleted or are fetched again. (what follows is an excerpt):

Global setting mail_plugins won’t change the setting inside an earlier filter at /etc/dovecot/dovecot.conf

Problems might come from declaration of mail_plugins global setting in dovecot.conf…:

# Enable acl plugin for shared mailboxes, 
# and listescape to extend allowable characters in mailbox names
mail_plugins = $mail_plugins acl listescape fts fts_lucene
protocol imap {
  mail_plugins = $mail_plugins imap_acl
}

#
# Quota configuration is enabled
#
mail_plugins = $mail_plugins quota   # <----

protocol imap {
  mail_plugins = $mail_plugins quota imap_quota
}
[root@server ~]# dovecot -n
# 2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.24 (124e06aa)
doveconf: Warning: /etc/dovecot/dovecot.conf line 276: Global setting mail_plugins won't change the setting inside an earlier filter at /etc/dovecot/dovecot.conf line 202 (if this is intentional, avoid this warning by moving the global setting before /etc/dovecot/dovecot.conf line 202)
doveconf: Warning: /etc/dovecot/dovecot.conf line 276: Global setting mail_plugins won't change the setting inside an earlier filter at /etc/dovecot/dovecot.conf line 202 (if this is intentional, avoid this warning by moving the global setting before /etc/dovecot/dovecot.conf line 202)
doveconf: Warning: service auth { client_limit=2000 } is lower than required under max. load (2400)
# OS: Linux 3.10.0-957.1.3.el7.x86_64 x86_64 CentOS Linux release 7.6.1810 (Core)  
...

https://marc.info/?l=getmail&m=154368012306671&w=2
there is some sort of problem suddenly happened error when getmail hands over to dovecot deliver and it reports error which stops getmail sending delete command I guess

Yes. If your MDA (delivery program) fails, getmail doesn’t tell the server to delete the message, so it can try again next time.

https://marc.info/?l=getmail&m=152753031028563&w=2
As dovecot-lda send these warnings to STDERR, getmails seems to treat them as error and DOES NOT DELETE the getmail-ed msgs anymore.

Similar to this older one: https://www.dovecot.org/list/dovecot/2016-April/103995.html


Pop3 connector no longer deletes mails
(Stéphane de Labrusse) #11

thank a lot for diving, card created https://github.com/orgs/NethServer/projects/1#card-15934419


(Stéphane de Labrusse) #12

ok for what I saw this should be related to dovecot, at least it is the major bump in the rpm version

[root@prometheus ~]# rpm -qa dovecot getmail
getmail-5.6-1.el7.noarch
dovecot-2.2.10-8.el7.x86_64

#upgraded to centos7.6

[root@ns7loc12 ~]# rpm -qa dovecot getmail
getmail-5.6-1.el7.noarch
dovecot-2.2.36-3.el7.x86_64

(Stéphane de Labrusse) #13

we have also this in /var/log/messages

Dec 20 22:01:47 ns7loc12 systemd: PID file /var/run/dovecot/master.pid not readable (yet?) after start.


(Stéphane de Labrusse) #14

ok at least it clear now, when you trigger getmail manually

  msg    20/10669 (18537 bytes), delivery error (command dovecot-lda 6685 wrote to stderr: doveconf: Warning: /etc/dovecot/dovecot.conf line 262: Global setting mail_plugins won't change the setting inside an earlier filter at /etc/dovecot/dovecot.conf line 188 (if this is intentional, avoid this warning by moving the global setting before /etc/dovecot/dovecot.conf line 188))
Delivery error (command dovecot-lda 6689 wrote to stderr: doveconf: Warning: /etc/dovecot/dovecot.conf line 262: Global setting mail_plugins won't change the setting inside an earlier filter at /etc/dovecot/dovecot.conf line 188 (if this is intentional, avoid this warning by moving the global setting before /etc/dovecot/dovecot.conf line 188))

(Stéphane de Labrusse) #15

in dovecot.conf

service auth {
-    client_limit = 2000
+    client_limit = 2500
}


- mail_plugins = $mail_plugins acl listescape fts fts_lucene 
+ mail_plugins = $mail_plugins acl listescape fts fts_lucene quota

protocol imap {
-  mail_plugins = $mail_plugins imap_acl
+ mail_plugins = $mail_plugins imap_acl quota imap_quota
}


#
# Quota configuration is enabled
#
- mail_plugins = $mail_plugins quota

- protocol imap {
-  mail_plugins = $mail_plugins quota imap_quota
- }

/etc/systemd/system/dovecot.service.d/limits.conf

[Service]
-  LimitNOFILE=2000
+ LimitNOFILE=2500
+ Group=vmail

systemctl daemon-reload
systemctl restart dovecot

in /etc/cron.d/getmail you have the command line to start manually getmail and check if no error

now here : msg 26/10669 (1707 bytes) delivered, deleted

/usr/bin/flock -n -E 0 /var/lib/getmail/stephane.xxx@xxx.xx.cfg -c "/usr/bin/getmail --getmaildir /var/lib/getmail/ --rcfile /var/lib/getmail/stephane.xxx@xxx.xx.cfg "

I will try to patch tomorrow


(Stéphane de Labrusse) #16

a patch to test for those are concerned with this bug

yum install http://packages.nethserver.org/nethserver/7.6.1810/autobuild/x86_64/Packages/nethserver-mail-server-2.4.2-1.3.pr99.gd760317.ns7.noarch.rpm

to go back at the original

yum downgrade nethserver-mail-server


(Stéphane de Labrusse) #17

Patch accepted, we need to verify it…Xmas excuses are not accepted :smiley:

Really thank you in advance.

https://github.com/NethServer/dev/issues/5679


(Marc) #18

On the QA note, is nethserver-getmail needed or it’s nethserver-mail-getmail?
After a quick test it seems to work, mail is deleted from server and not duplicated.


(Stéphane de Labrusse) #19

yep nethserver-mail-getmail, sorry for the mistake and tk a lot @dnutan you are my hero

just a question you upgraded nethserver-mail-server from nethserver-testing


(Marc) #20

Yes, upgraded all nethserver-mail-* packages from testing repo.