If I didn’t understand wrong, sync status of getmail is in /var/lib/getmail. If you migrate a server, configuration is correct but your server will resync all emails. Why we don’t backup this folder?
@davidep Could you explain me? 
Would you demonstrate it is a bug?
1 Like
Yes, I think it’s a bug.
Reproduce it without restore a server is simple:
- rm -f /var/lib/getmail/*
- signal-event nethserver-mail-getmail-update (configuration is recreated, but no old emails already downloaded - it’s the same situation when you restore a server)
- force download of getmail
- all emails are re-download
We need to add a backup-data.include for getmail and backup /var/lib/getmail folder.
I’ll open the bug asap.
@davidep do you think we have to backup all /var/lib/getmail folder or we have to exclude from it the .cfg files?
All of this sounds like a dejavou… 
Did you look also at the git changelog or past issues?
I can search if there is something related, but in the code there isn’t backup-data configuration.
I’d expect config backup instead
1 Like
Ok 
In the code there isn’t backup-config.d folder.
Do you think this folder should be backupped in config backup?
Time for a lasagna now. Tomorrow I’ll look at it. In the meanwhile I suggest you book a ticket to Fosdem!
1 Like
Have a good lunch! 
1 Like
There seem to be no backup of the getmail state file. I think I’ve already discussed this issue in the past with @giacomo or maybe @filippo_carletti.
Just some of thoughts:
- if you backup up the getmail state in the configuration backup, it will change every day. It is not “configuration” it is the state of the getmail application: it is not the right place for it.
- if you put it in backup-data it could be uneffective or even disruptive, as the getmail service is started just after the configuration is restored. It starts to fetch remote messages then at a certain point its state file gets overwritten.
- if you do not configure the backup for the app state at all, as we’re doing, some messages could be downloaded again.
I’d keep the current implementation and fix the documentation to clarify the getmail behavior during restore. I’d also suggest to keep a short message retention time, to avoid double downloads. We can also investigate how to implement a de-duplication procedure, as messages have their uinique IDs.
2 Likes
We can reach a compromise by disabling getmail connectors, until post-restore-data is launched. Save active getmail connectors before disabling them in a file and then activate them after post-restore-data. We can save the state in backup-data and they will starts sync after post-restore-data, so no overwrite.
2 Likes
@davidep Do you think can I add a sync into HotSync package of /var/lib/getmail folder? I think that with HotSync should be synced also this folder, to avoid all emails re-download when you put slave in production.
Yes, that’s a strategy we implemented somewhere else: start something after post-restore-data. But it’s complex and may lead to issues if post-restore-data has to be fired manually.
Moreover, if you save the state in the data backup the state becomes stale quickly as the backup frequency is daily, whilst the getmail fetch occurs in few minutes. That means that some messages from the remote mail server are downloaded again: we can’t work around this limitation completely.
HotSync runs more frequently: theoretically it can reduce the number of double downloads.
However I’m afraid we’re going to add something to a complex system that is not a real solution of the original issue. Instead we have the risk of introducing regression bugs.
The easiest thing to do to mitigate the limitation is to lower the remote server retention time. Once a message has been downloaded it should be expunged as quickly as possible. I’d say, immediately.
This issue originates from the weakness of the “fetchmail” paradigm: the protocols it relies on, (POP3, IMAP) are not designed to transport messages, like SMTP is.