Nethserver-piler needs your tests

stephdl · February 8, 2021, 6:05pm

I created a rpm to install mailpiler in three docker containers managed by docker-compose, you will retrieve some documentation at : https://mailpiler.com/documentation/

yum install http://mirror.de-labrusse.fr/NethDev/nethserver-piler/nethserver-piler-0.1.0-1.7.g4ba4db9.ns7.sdl.x86_64.rpm

now I need you test to break it and your review please @jsuto

You can import email with the script piler-import-email
You can change the vhost or use piler.DefaultDomainOfServer

config setprop piler Vhost sub.domain.org
signal-event nethserver-piler-update

you have to do a Bcc of each domain to send emails to piler

db domains setprop nethservertest.org AlwaysBccAddress archive@piler.nethservertest.org  AlwaysBccStatus enabled
signal-event domain-modify nethservertest.org

each real user of the system can login (IMAP authentication), else

admin@local:pilerrocks
auditor@local:auditor

The backup is taking in to account

you have a piler wrapper to manage docker

[root@ns7loc11 ~]# piler
The commands are :
    piler status : retrieve the status of piler container
    piler bash : start a shell inside the container
    piler restart: Restart the piler container
    piler start: Start the piler container
    piler stop: Stop the piler container
    piler ps: Container information
    piler log: Display the error log of the container

Each user can restore his emails from the archive

jsuto · February 8, 2021, 6:27pm

Thank you, Stephane, I’ll try to break it I’ll keep you posted.

jsuto · February 8, 2021, 9:14pm

After a quick review, it’s a nice job. The imap authentication, and mail restore over imap indeed both work.
By setting the per domain bcc, postfix forwards emails to piler in the container.

A few polishing ideas:

set the antispam header in piler.conf in the spam_header_line variable. I can see nethserver uses rspamd. I assume it sets a specific mail header in case of a spam. By setting it in piler.conf, piler is able to indicate in the gui that it’s a spam message
“piler status” can’t find the health template for memcached and mysql

piler_piler_1 status

healthy

piler_mysql_1 status

Template parsing error: template: :1:8: executing “” at <.State.Health.Status>: map has no entry for key “Health”

piler_memcached_1 status

Template parsing error: template: :1:8: executing “” at <.State.Health.Status>: map has no entry for key “Health”

I’ll find some health checks for them, and let you know

when you fix the piler’s vhost name with commands you mentioned then the $config[‘SITE_URL’] parameter changes accordingly in /etc/piler/config-site.php, however, $config[SITE_NAME_CONST] values doesn’t change, it remains the default, ie. piler.domain.

Anyway, I think you did a great job so far. Perhaps one more question: how to handle piler version upgrades? Usually it would involve a new docker image, and once in a while a database schema fix, eg. adding a new table or similar.

Btw. docker images. I’m not sure if nethserver has any docker image repository. I currently use the docker hub, however, note that they have placed some limitations for the freemium users (like me), see https://www.docker.com/blog/docker-hub-image-retention-policy-delayed-and-subscription-updates/ and https://docs.docker.com/docker-hub/download-rate-limit/

jsuto · February 8, 2021, 9:14pm

JFrog (and probably other docker registry providers) offer some other option for this mentioned limitations, see https://jfrog.com/blog/get-around-docker-download-limits-jfrog-artifactory/ for more

stephdl · February 8, 2021, 10:15pm

good idea

need to check but in my VM I found piler unhealthy due to curl error, need to verify, thank

maybe a feature, I saw that piler does not like when the piler hostname was not defined in the /etc/hosts. something more to check

I thought to upgrade the docker-compose template we have in /etc/e-smith/templates/usr/share/piler/docker-compose.yml, so yes change the docker image accordingly of your upgrade, relative to the mysql upgrade we could start a script inside the mysql docker image like I did for the email importation. All of this could be done by a new nethserver-piler rpm version.

That’s not a good news

not aware, need to check how to host a docker repository on my server or elsewhere.

stephdl · February 8, 2021, 10:34pm

relative to piler health status @jsuto

 [root@ns7loc11 ~]# piler status
# piler_piler_1 status
unhealthy

I go to the container

[root@ns7loc11 ~]# docker exec -ti piler_piler_1 /bin/bash
root@0194e80c376f:/# curl smtp://localhost
bash: curl: command not found

if Install curl in the container, then I have another error

root@0194e80c376f:/# curl -v smtp://localhost
*   Trying 127.0.0.1:25...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 25 (#0)
< 220 piler.nethservertest.org ESMTP
> EHLO 0194e80c376f
< 250-piler.nethservertest.org
< 250-PIPELINING
< 250-STARTTLS
< 250-SIZE
< 250 8BITMIME
> HELP
< 502 Command not implemented
* Command failed: 502
> QUIT
< 221 piler.nethservertest.org Goodbye
* Closing connection 0
curl: (56) Command failed: 502

maybe I did something wrong, however it seems that curl is the status to command to see if the container is in a bad or good state.

stephdl · February 8, 2021, 10:39pm

There is currently around 19000 servers, however since it is docker based and currently not an official module, it will have less people.

https://www.nethserver.org/phone-home/index.html

jsuto · February 9, 2021, 4:45am

No, you did that right. I forgot to add both curl and support for the HELP command. Yesterday I fixed the image, so pull sutoj/piler:1.3.10 again, and it should be fine.

jsuto · February 9, 2021, 6:52am

I’ve added a health check to the mysql container, see https://bitbucket.org/jsuto/piler/commits/9dbfbb74af8224fa9a8c04aff59e7f7444a5aabe for the details.

However, I couldn’t get any meaningful health check possibility using the memcached container. It could be enhanced by adding a package like nc to do some periodic checks, but I think it’s much simpler to fix /usr/bin/piler script to use docker top to see if memcached is running, eg.

echo “# piler_memcached_1 status”
if [[ $(docker top piler_memcached_1 | grep -c memcached ) -eq 1 ]]; then echo “healthy”; else echo “unhealthy”; fi

Also note that I’ve fixed the mariadb version to 10.5 in docker-compose.yaml

stephdl · February 9, 2021, 6:53am

There is a data migration to that new version ?

For piler wrapper no problem to fix it on my side.

jsuto · February 9, 2021, 6:54am

Not sure, hopefully no, since it’s a minor version update.

stephdl · February 9, 2021, 6:56am

Thank

jsuto · February 9, 2021, 7:25am

This is what we need in piler.conf to recognize spams tagged by rspamd.

spam_header_line=X-Spam: Yes

stephdl · February 9, 2021, 9:14am

Blind shot, we reject email flagged as spam by rspamd, normally it should not go to the archive.

I can add it, it doesn’t harm

jsuto · February 9, 2021, 9:34am

If that’s the case, then it’s not necessary. However, I think there’s a threshold to mark the message and spam (6.0 points), and another to deny it (20.0). Spam emails under 20.0 may be delivered to the archive.

stephdl · February 9, 2021, 2:33pm

you are right, however we do not use the default rspamd flag due to old compatibility we must get with spamassassin

X-Spamd-Result: default: False [11.12 / 19.90];
	 HAS_REPLYTO(0.00)[info@top-partenaire.com];
	 BAYES_SPAM(0.74)[78.53%];
	 REPLYTO_DN_EQ_FROM_DN(0.00)[];
	 HAS_X_SOURCE(0.00)[];
	 REPLYTO_ADDR_EQ_FROM(0.00)[];
	 TO_DN_NONE(0.00)[];
	 DKIM_TRACE(0.00)[aru2l.org:+,top-partenaire.com:+];
	 MX_GOOD(-0.01)[];
	 FROM_EXCESS_QP(1.20)[];
	 FORGED_SENDER(0.30)[info@top-partenaire.com,50354=4477cac1-619b-4df6-92ab-08b782dd5d25=117=11249971@e.top-partenaire.com];
	 HAS_X_ANTIABUSE(0.00)[];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+,1:+,2:~];
	 ASN(0.00)[asn:50474, ipnet:109.234.163.0/24, country:FR];
	 FROM_NEQ_ENVFROM(0.00)[info@top-partenaire.com,50354=4477cac1-619b-4df6-92ab-08b782dd5d25=117=11249971@e.top-partenaire.com];
	 HAS_X_AS(0.00)[contact@aru2l.org];
	 FORGED_RECIPIENTS(2.00)[m:contact@aru2l.org,s:stephane@de-labrusse.fr];
	 R_SPF_FAIL(1.00)[-all];
	 R_DKIM_ALLOW(-0.20)[aru2l.org:s=default,top-partenaire.com:s=default1k];
	 FROM_HAS_DN(0.00)[];
	 MIME_GOOD(-0.10)[multipart/alternative,text/plain];
	 HTML_SHORT_LINK_IMG_3(0.50)[];
	 DMARC_NA(0.00)[top-partenaire.com];
	 HAS_LIST_UNSUB(-0.01)[];
	 RCPT_COUNT_ONE(0.00)[1];
	 MANY_INVISIBLE_PARTS(1.00)[10];
	 REPLYTO_EXCESS_QP(1.20)[];
	 VIOLATED_DIRECT_SPF(3.50)[];
	 HAS_X_GMSV(0.00)[];
	 IP_REPUTATION_SPAM(0.00)[asn: 50474(0.02), country: FR(0.00), ip: 109.234.163.71(0.00)]
X-Rspamd-Queue-Id: 2F41118B84208
X-Spam-Flag: Yes

But this can be a choice of the sysadmin, because they are in a range that it is not a spam because it is rejected immediately and it could be a false positive (good email with object renamed with SPAM and moved to junk).

good shot

jsuto · February 9, 2021, 2:46pm

I see. The point is that piler supports a single spam header value.

stephdl · February 9, 2021, 2:53pm

not a problem, we have just one X-Spam-Flag: Yes

stephdl · February 9, 2021, 3:04pm

one question, after I change the piler configuration in the volume piler_piler_etc/_data/piler.conf do you think piler needs to be restarted ? the documentation doesn’t state on it

https://www.mailpiler.org/wiki/current:antivirus-antispam

jsuto · February 9, 2021, 3:07pm

Yes, you need to restart it to re-read piler.conf values.

What would be the next step from my side?

Nethserver-piler needs your tests

piler_piler_1 status

piler_mysql_1 status

piler_memcached_1 status