Nethserver-piler needs your tests

I created a rpm to install mailpiler in three docker containers managed by docker-compose, you will retrieve some documentation at : https://mailpiler.com/documentation/

yum install http://mirror.de-labrusse.fr/NethDev/nethserver-piler/nethserver-piler-0.1.0-1.7.g4ba4db9.ns7.sdl.x86_64.rpm

now I need you test to break it and your review please @jsuto

You can import email with the script piler-import-email
You can change the vhost or use piler.DefaultDomainOfServer

config setprop piler Vhost sub.domain.org
signal-event nethserver-piler-update

you have to do a Bcc of each domain to send emails to piler

db domains setprop nethservertest.org AlwaysBccAddress archive@piler.nethservertest.org  AlwaysBccStatus enabled
signal-event domain-modify nethservertest.org

each real user of the system can login (IMAP authentication), else

admin@local:pilerrocks
auditor@local:auditor

The backup is taking in to account

you have a piler wrapper to manage docker

[root@ns7loc11 ~]# piler
The commands are :
    piler status : retrieve the status of piler container
    piler bash : start a shell inside the container
    piler restart: Restart the piler container
    piler start: Start the piler container
    piler stop: Stop the piler container
    piler ps: Container information
    piler log: Display the error log of the container

Each user can restore his emails from the archive

5 Likes

Thank you, Stephane, I’ll try to break it :slight_smile: I’ll keep you posted.

3 Likes

After a quick review, it’s a nice job. The imap authentication, and mail restore over imap indeed both work.
By setting the per domain bcc, postfix forwards emails to piler in the container.

A few polishing ideas:

  • set the antispam header in piler.conf in the spam_header_line variable. I can see nethserver uses rspamd. I assume it sets a specific mail header in case of a spam. By setting it in piler.conf, piler is able to indicate in the gui that it’s a spam message

  • “piler status” can’t find the health template for memcached and mysql

    piler_piler_1 status

    healthy

    piler_mysql_1 status

    Template parsing error: template: :1:8: executing “” at <.State.Health.Status>: map has no entry for key “Health”

    piler_memcached_1 status

    Template parsing error: template: :1:8: executing “” at <.State.Health.Status>: map has no entry for key “Health”

I’ll find some health checks for them, and let you know

  • when you fix the piler’s vhost name with commands you mentioned then the $config[‘SITE_URL’] parameter changes accordingly in /etc/piler/config-site.php, however, $config[SITE_NAME_CONST] values doesn’t change, it remains the default, ie. piler.domain.

Anyway, I think you did a great job so far. Perhaps one more question: how to handle piler version upgrades? Usually it would involve a new docker image, and once in a while a database schema fix, eg. adding a new table or similar.

Btw. docker images. I’m not sure if nethserver has any docker image repository. I currently use the docker hub, however, note that they have placed some limitations for the freemium users (like me), see https://www.docker.com/blog/docker-hub-image-retention-policy-delayed-and-subscription-updates/ and https://docs.docker.com/docker-hub/download-rate-limit/

5 Likes

JFrog (and probably other docker registry providers) offer some other option for this mentioned limitations, see https://jfrog.com/blog/get-around-docker-download-limits-jfrog-artifactory/ for more

3 Likes

good idea

need to check but in my VM I found piler unhealthy due to curl error, need to verify, thank

maybe a feature, I saw that piler does not like when the piler hostname was not defined in the /etc/hosts. something more to check

I thought to upgrade the docker-compose template we have in /etc/e-smith/templates/usr/share/piler/docker-compose.yml, so yes change the docker image accordingly of your upgrade, relative to the mysql upgrade we could start a script inside the mysql docker image like I did for the email importation. All of this could be done by a new nethserver-piler rpm version.

That’s not a good news :smiley:

not aware, need to check how to host a docker repository on my server or elsewhere.

relative to piler health status @jsuto

 [root@ns7loc11 ~]# piler status
# piler_piler_1 status
unhealthy

I go to the container

[root@ns7loc11 ~]# docker exec -ti piler_piler_1 /bin/bash
root@0194e80c376f:/# curl smtp://localhost
bash: curl: command not found

if Install curl in the container, then I have another error

root@0194e80c376f:/# curl -v smtp://localhost
*   Trying 127.0.0.1:25...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 25 (#0)
< 220 piler.nethservertest.org ESMTP
> EHLO 0194e80c376f
< 250-piler.nethservertest.org
< 250-PIPELINING
< 250-STARTTLS
< 250-SIZE
< 250 8BITMIME
> HELP
< 502 Command not implemented
* Command failed: 502
> QUIT
< 221 piler.nethservertest.org Goodbye
* Closing connection 0
curl: (56) Command failed: 502

maybe I did something wrong, however it seems that curl is the status to command to see if the container is in a bad or good state.

There is currently around 19000 servers, however since it is docker based and currently not an official module, it will have less people.

https://www.nethserver.org/phone-home/index.html

No, you did that right. I forgot to add both curl and support for the HELP command. Yesterday I fixed the image, so pull sutoj/piler:1.3.10 again, and it should be fine.

1 Like

I’ve added a health check to the mysql container, see https://bitbucket.org/jsuto/piler/commits/9dbfbb74af8224fa9a8c04aff59e7f7444a5aabe for the details.

However, I couldn’t get any meaningful health check possibility using the memcached container. It could be enhanced by adding a package like nc to do some periodic checks, but I think it’s much simpler to fix /usr/bin/piler script to use docker top to see if memcached is running, eg.

echo “# piler_memcached_1 status”
if [[ $(docker top piler_memcached_1 | grep -c memcached ) -eq 1 ]]; then echo “healthy”; else echo “unhealthy”; fi

Also note that I’ve fixed the mariadb version to 10.5 in docker-compose.yaml

1 Like

There is a data migration to that new version ?

For piler wrapper no problem to fix it on my side.

Not sure, hopefully no, since it’s a minor version update.

Thank :slight_smile:

This is what we need in piler.conf to recognize spams tagged by rspamd.

spam_header_line=X-Spam: Yes

1 Like

Blind shot, we reject email flagged as spam by rspamd, normally it should not go to the archive.

I can add it, it doesn’t harm

If that’s the case, then it’s not necessary. However, I think there’s a threshold to mark the message and spam (6.0 points), and another to deny it (20.0). Spam emails under 20.0 may be delivered to the archive.

1 Like

you are right, however we do not use the default rspamd flag due to old compatibility we must get with spamassassin

X-Spamd-Result: default: False [11.12 / 19.90];
	 HAS_REPLYTO(0.00)[info@top-partenaire.com];
	 BAYES_SPAM(0.74)[78.53%];
	 REPLYTO_DN_EQ_FROM_DN(0.00)[];
	 HAS_X_SOURCE(0.00)[];
	 REPLYTO_ADDR_EQ_FROM(0.00)[];
	 TO_DN_NONE(0.00)[];
	 DKIM_TRACE(0.00)[aru2l.org:+,top-partenaire.com:+];
	 MX_GOOD(-0.01)[];
	 FROM_EXCESS_QP(1.20)[];
	 FORGED_SENDER(0.30)[info@top-partenaire.com,50354=4477cac1-619b-4df6-92ab-08b782dd5d25=117=11249971@e.top-partenaire.com];
	 HAS_X_ANTIABUSE(0.00)[];
	 RCVD_COUNT_ZERO(0.00)[0];
	 MIME_TRACE(0.00)[0:+,1:+,2:~];
	 ASN(0.00)[asn:50474, ipnet:109.234.163.0/24, country:FR];
	 FROM_NEQ_ENVFROM(0.00)[info@top-partenaire.com,50354=4477cac1-619b-4df6-92ab-08b782dd5d25=117=11249971@e.top-partenaire.com];
	 HAS_X_AS(0.00)[contact@aru2l.org];
	 FORGED_RECIPIENTS(2.00)[m:contact@aru2l.org,s:stephane@de-labrusse.fr];
	 R_SPF_FAIL(1.00)[-all];
	 R_DKIM_ALLOW(-0.20)[aru2l.org:s=default,top-partenaire.com:s=default1k];
	 FROM_HAS_DN(0.00)[];
	 MIME_GOOD(-0.10)[multipart/alternative,text/plain];
	 HTML_SHORT_LINK_IMG_3(0.50)[];
	 DMARC_NA(0.00)[top-partenaire.com];
	 HAS_LIST_UNSUB(-0.01)[];
	 RCPT_COUNT_ONE(0.00)[1];
	 MANY_INVISIBLE_PARTS(1.00)[10];
	 REPLYTO_EXCESS_QP(1.20)[];
	 VIOLATED_DIRECT_SPF(3.50)[];
	 HAS_X_GMSV(0.00)[];
	 IP_REPUTATION_SPAM(0.00)[asn: 50474(0.02), country: FR(0.00), ip: 109.234.163.71(0.00)]
X-Rspamd-Queue-Id: 2F41118B84208
X-Spam-Flag: Yes

But this can be a choice of the sysadmin, because they are in a range that it is not a spam because it is rejected immediately and it could be a false positive (good email with object renamed with SPAM and moved to junk).

good shot

I see. The point is that piler supports a single spam header value.

1 Like

not a problem, we have just one X-Spam-Flag: Yes

one question, after I change the piler configuration in the volume piler_piler_etc/_data/piler.conf do you think piler needs to be restarted ? the documentation doesn’t state on it

https://www.mailpiler.org/wiki/current:antivirus-antispam

Yes, you need to restart it to re-read piler.conf values.

What would be the next step from my side?

1 Like