Nethserver-piler needs your tests

stephdl · February 9, 2021, 3:13pm

let me check I make some changes…thank again

stephdl · February 9, 2021, 6:05pm

[root@ns7loc11 ~]# /usr/bin/piler-import-email
The commands are :
    -a : import all accounts
    -l : list all accounts of the server
    -u account : import all email of a list of users(comma separated : user1,user2)

stephdl · February 9, 2021, 7:03pm

quick check relative to sql upgrade : https://www.mailpiler.org/wiki/current:upgrade

to export

[root@ns7loc11 ~]# docker exec piler_mysql_1 /usr/bin/mysqldump -u piler --password=ozN_22MF0XXgIJXl piler > backup.sql

to import

[root@ns7loc11 ~]# docker exec piler_mysql_1 /usr/bin/mysqldump -u piler --password=ozN_22MF0XXgIJXl piler < backup.sql

Yes we could I think alter the tables by a script with an upgrade.

Thinking loudly but maybe I could do a backup of the db with data backup, I save the full volume of docker sql but a sql dump could help also.

jsuto · February 9, 2021, 7:33pm

Also note that a regular backup of the mysql piler database is crucial, as well as the data on the docker volumes. If piler gets to the docs section, it’s worth to emphasize the importance of the backup.

Not sure if it’s a viable deployment model for nethserver users and customers, but even though piler is pretty flexible, and you can deploy it to the same host where the mail server is, I usually advocate to deploy the archive to another host, probably running nethserver as well.

stephdl · February 9, 2021, 7:51pm

We aim to design our software as something modular, most of time we try to hide the complexity to the system administrator. I try to bundle a workable solution with flexibility, I use to say that the complexity is for the developer.

Actually I require the email stack, we could imagine to not require it and following if the templates detects the email stack, then :

if the email stack is installed go to aqua, and use a BCC to the container
if the email stack is not installed go to ~~docker0~~ aqua but use TCP Port 2525:25,8080:80,4430:443.
The last configuration will need to enable a LDAP authentication I think to the remote account provider, or we could still use IMAP (after all IMAP takes its account from the account provider). ~~However I am not sure that docker0 works well with shorewall.~~ We can still use aqua however an entry to /etc/postfix/transport is a necessary :

/etc/postfix/transport:
neth-archive.acts.hu smtp:127.0.0.1:2525

That will add a layer of complexity but I recall a WAPT requirement and the developers state that they want nothing other software on the host.

Obviously this module will have a documentation, I think mostly specific to the NethServer installation, we have a wiki: https://wiki.nethserver.org/doku.php?id=modules.

jsuto · February 9, 2021, 8:15pm

IMAP authentication works for sure, however you can assign only the user’s primary email address in the piler GUI. Nowadays it’s common for users to have even some mail aliases, and being part of some mailing lists. The IMAP authentication alone cannot provide these extra stuff for piler.

There’s a workaround, though. The piler gui supports pre- and post authentication hooks, and allows you to create an arbitrary logic to assign more email addresses to the logged in user. See https://www.mailpiler.org/wiki/current:custom-authentication

LDAP authentication may provide a much richer context for the user, eg. email aliases, group memberships, etc. I think it’s fine to start with imap authentication, and then later add ldap auth.

Perhaps a nice item to the nethserver’s roadmap is allowing even postfix to pull its user, domain, alias db from ldap.

stephdl · February 10, 2021, 3:42pm

Hello @jsuto

I am listening your knowledge, I have a valid openldap configuration

$config['ENABLE_LDAP_AUTH'] = 1;
$config['LDAP_HOST'] = 'ldaps://ns7loc11.nethservertest.org';
$config['LDAP_HELPER_DN'] = 'cn=ldapservice,dc=directory,dc=nh';
$config['LDAP_HELPER_PASSWORD'] = 'V_85617fr2bK3Csj';
$config['LDAP_MAIL_ATTR'] = 'mail';
// set this if you want to limit the scope of the ldap query
$config['LDAP_BASE_DN'] = 'dc=directory,dc=nh';
$config['LDAP_ACCOUNT_OBJECTCLASS'] = 'posixAccount';
// admin@nethservertest.org is admin of piler
$config['LDAP_ADMIN_MEMBER_DN'] = 'uid=admin,ou=People,dc=directory,dc=nh';
// members of PilerAuditor are auditors
$config['LDAP_AUDITOR_MEMBER_DN'] = 'cn=PilerAuditor,ou=Groups,dc=directory,dc=nh';

However how to add in /etc/ldap/ldap.conf (obviously in the piler container)

TLS_REQCERT never

testing now with samba AD

jsuto · February 10, 2021, 4:18pm

I was reluctant to add “TLS_REQCERT never” to the image, because it’s only necessary if you have a self signed certificate. I think it’s best to have a fixed ldap.conf as part of the piler rpm package, eg. right next to the docker-compose.yaml file with the following content:

TLS_CACERT	/etc/ssl/certs/ca-certificates.crt
TLS_REQCERT never

Then mount it to the piler container, eg.

volumes:
  - piler_etc:/etc/piler
  - piler_var:/var/piler
  - /usr/...../ldap.conf:/etc/ldap/ldap.conf:ro

stephdl · February 10, 2021, 4:37pm

yes

stephdl · February 10, 2021, 6:08pm

samba AD

$config['ENABLE_LDAP_AUTH'] = 1;
$config['LDAP_HOST'] = 'ldaps://nsdc-ns7loc12.ad.nethservertest.org';
$config['LDAP_HELPER_DN'] = 'ldapservice@AD.NETHSERVERTEST.ORG';
$config['LDAP_HELPER_PASSWORD'] = 'k7aL_gMQPatrENJJ';
$config['LDAP_ACCOUNT_OBJECTCLASS'] = 'user';
$config['LDAP_MAIL_ATTR'] = 'userPrincipalName';
$config['LDAP_DISTRIBUTIONLIST_OBJECTCLASS'] = 'group';
$config['LDAP_DISTRIBUTIONLIST_ATTR'] = 'member';
// set this if you want to limit the scope of the ldap query
$config['LDAP_BASE_DN'] = 'dc=ad,dc=nethservertest,dc=org';
// admin@nethservertest.org is admin of piler
$config['LDAP_ADMIN_MEMBER_DN'] = 'CN=admin,CN=Users,DC=ad,DC=nethservertest,DC=org';
// members of PilerAuditor are auditors
$config['LDAP_AUDITOR_MEMBER_DN'] = 'CN=famille,CN=Users,DC=ad,DC=nethservertest,DC=org';

don’t ever forget : account-provider-test dump

stephdl · February 10, 2021, 8:22pm

I think I have an issue with openldap to retrieve the group for auditor, it is workable with samba AD. This is the slapcat output, we can see the memberUid without domain DC|DN

dn: cn=famille,ou=Groups,dc=directory,dc=nh
gidNumber: 1002
cn: famille
objectClass: posixGroup
structuralObjectClass: posixGroup
entryUUID: f4126658-d67e-103a-9353-618194e739f1
creatorsName: cn=libuser,dc=directory,dc=nh
createTimestamp: 20201219194847Z
memberUid: stephane
memberUid: helene
memberUid: maxime
entryCSN: 20201219194955.372605Z#000000#000#000000
modifiersName: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
modifyTimestamp: 20201219194955Z

this is what I added to piler to authenticate, I can login but obviously the group is not retrieved, I think it is because piler use user@domain.com and not user

$config['ENABLE_LDAP_AUTH'] = 1;
$config['LDAP_HOST'] = 'ldaps://ns7loc11.nethservertest.org';
$config['LDAP_HELPER_DN'] = 'cn=ldapservice,dc=directory,dc=nh';
$config['LDAP_HELPER_PASSWORD'] = 'V_85617fr2bK3Csj';
$config['LDAP_MAIL_ATTR'] = 'mail';
$config['LDAP_BASE_DN'] = 'dc=directory,dc=nh';
$config['LDAP_ACCOUNT_OBJECTCLASS'] = 'posixAccount';
$config['LDAP_DISTRIBUTIONLIST_ATTR'] = 'memberUid';
$config['LDAP_DISTRIBUTIONLIST_OBJECTCLASS'] = 'posixGroup';

// admin.com is admin of piler
$config['LDAP_ADMIN_MEMBER_DN'] = 'uid=admin,ou=People,dc=directory,dc=nh';
// members of PilerAuditor are auditors
$config['LDAP_AUDITOR_MEMBER_DN'] = 'cn=famille,ou=Groups,dc=directory,dc=nh';

will update my rpm…maybe I will need a hand on this

stephdl · February 10, 2021, 8:31pm

yum install http://mirror.de-labrusse.fr/NethDev/nethserver-piler/nethserver-piler-0.1.0-1.13.gd254596.ns7.sdl.x86_64.rpm

you have to install an account provider, ldap is the default one to change it

config setprop piler Authentication imap
signal-event nethserver-piler-update

the admin of the system is the admin of piler, the group PilerAuditor is the group of auditor (broken with openldap for now)

stephdl · February 11, 2021, 5:43pm

yum install http://mirror.de-labrusse.fr/NethDev/nethserver-piler/nethserver-piler-0.1.0-1.17.g1711166.ns7.sdl.x86_64.rpm

you have to install an account provider,back to imap because ldap still doe not work well

to use ldap do

config setprop piler Authentication ldap
systemctl restart piler

with openldap, piler can authenticate the user, grant him admin right if it is admin, retrieve its email address, the issue is piler cannot retrieve the group

with samba AD it is worth strange, piler can authenticate the user but piler does not know the email address and the group. This is a big issue because without email address, piler is completely lost.

the ldap configuration

github.com

stephdl/nethserver-piler/blob/ns7/root/etc/e-smith/templates/var/lib/docker/volumes/piler_piler_etc/_data/config-site.php/10base

define('SITE_NAME_CONST', 'SITE_NAME');

$config[SITE_NAME_CONST] = 'piler.{$DomainName}';
$config['SITE_URL'] = 'https://{$piler{'Vhost'} || piler.'.'.$DomainName}/';

$config['SMTP_DOMAIN'] = $config[SITE_NAME_CONST];
$config['SMTP_FROMADDR'] = 'no-reply@' . $config[SITE_NAME_CONST];
$config['ADMIN_EMAIL'] = 'admin@' . $config[SITE_NAME_CONST];

$config['DB_PASSWORD'] = 'MYSQL_PASSWORD';

$config['DECRYPT_BINARY'] = '/usr/bin/pilerget';
$config['DECRYPT_ATTACHMENT_BINARY'] = '/usr/bin/pileraget';
$config['PILER_BINARY'] = '/usr/sbin/piler';
$config['DB_HOSTNAME'] = 'mysql';
{
use NethServer::Password;
my $MysqlPilerPassword = NethServer::Password::store('PilerMysqlUserPassword') || die('Could not generate PilerMysqlUserPassword password!');
$OUT.="\$config['DB_PASSWORD'] = '$MysqlPilerPassword';\n";
}

This file has been truncated. show original

I think I need some advices @jsuto

this is the output of samba : https://gist.github.com/stephdl/109d4cbbc34fd3d8dcdc56218594599d

[root@ns7loc13 ~]# net ads search -P cn=stephane
Got 1 replies

objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: stephane
givenName: stephane de labrusse
instanceType: 4
whenCreated: 20201219201513.0Z
displayName: stephane de labrusse
uSNCreated: 3819
name: stephane
objectGUID: d9f0d850-2d48-49c9-8a40-59b6e1cd7a62
badPwdCount: 0
codePage: 0
countryCode: 0
badPasswordTime: 0
lastLogoff: 0
lastLogon: 0
primaryGroupID: 513
objectSid: S-1-5-21-2352814451-3700784286-2587900704-1107
accountExpires: 9223372036854775807
logonCount: 0
sAMAccountName: stephane
sAMAccountType: 805306368
objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=ad,DC=nethservertest,DC=org
loginShell: /usr/libexec/openssh/sftp-server
unixHomeDirectory: /var/lib/nethserver/home/stephane
userPrincipalName: stephane@nethservertest.org
memberOf: CN=famille,CN=Users,DC=ad,DC=nethservertest,DC=org
userAccountControl: 66048
pwdLastSet: 132528825164467320
lastLogonTimestamp: 132575382823831570
whenChanged: 20210211173122.0Z
uSNChanged: 3859
distinguishedName: CN=stephane,CN=Users,DC=ad,DC=nethservertest,DC=org

[root@ns7loc13 ~]# 
[root@ns7loc13 ~]# net ads search -P cn=famille
Got 1 replies

objectClass: top
objectClass: group
cn: famille
instanceType: 4
whenCreated: 20201219201432.0Z
uSNCreated: 3818
name: famille
objectGUID: b7be0ea0-e8bd-4048-a401-4289f9207016
objectSid: S-1-5-21-2352814451-3700784286-2587900704-1106
sAMAccountName: famille
sAMAccountType: 268435456
groupType: -2147483646
objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=ad,DC=nethservertest,DC=org
member: CN=maxime,CN=Users,DC=ad,DC=nethservertest,DC=org
member: CN=stephane,CN=Users,DC=ad,DC=nethservertest,DC=org
member: CN=helene,CN=Users,DC=ad,DC=nethservertest,DC=org
whenChanged: 20201219201658.0Z
uSNChanged: 3845
distinguishedName: CN=famille,CN=Users,DC=ad,DC=nethservertest,DC=org

this is the output of openldap : https://gist.github.com/stephdl/c970babae3b19afd1302a59ae4f3e821

dn: cn=famille,ou=Groups,dc=directory,dc=nh
gidNumber: 1002
cn: famille
objectClass: posixGroup
structuralObjectClass: posixGroup
entryUUID: f4126658-d67e-103a-9353-618194e739f1
creatorsName: cn=libuser,dc=directory,dc=nh
createTimestamp: 20201219194847Z
memberUid: stephane
memberUid: helene
memberUid: maxime
entryCSN: 20201219194955.372605Z#000000#000#000000
modifiersName: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
modifyTimestamp: 20201219194955Z

dn: uid=stephane,ou=People,dc=directory,dc=nh
uidNumber: 1001
gidNumber: 1000
uid: stephane
shadowMax: 99999
shadowWarning: 7
shadowMin: 0
loginShell: /usr/libexec/openssh/sftp-server
homeDirectory: /var/lib/nethserver/home/stephane
shadowInactive: -1
shadowExpire: -1
gecos: stephane de labrusse
shadowLastChange: 18615
shadowFlag: -1
cn: stephane de labrusse
sn: stephane
objectClass: posixAccount
objectClass: shadowAccount
objectClass: inetOrgPerson
structuralObjectClass: inetOrgPerson
entryUUID: 020142ca-d67f-103a-9354-618194e739f1
creatorsName: cn=libuser,dc=directory,dc=nh
createTimestamp: 20201219194910Z
mail: stephane@nethservertest.org
userPassword:: e0NSWVBUfSQ2JEZHdzhQaWJvLmEvS2pHc2wkcXdhWUtHV0l0ZnBMakF3MWlla
 G1ZV05SdDFRVjVFYVpyU1AudnJxYllpcGdZSmsxZEoxUW4xVXMxMnIuVW9sWGxZM1YvNUhLZmZ6
 SFN3ZGZvVGJZNy8=
entryCSN: 20201219194912.524513Z#000000#000#000000
modifiersName: cn=libuser,dc=directory,dc=nh
modifyTimestamp: 20201219194912Z

jsuto · February 11, 2021, 8:51pm

Piler first tries to query your email addresses associated with the user ldap object. Then it uses both your email and your user’s object’s DN value to find the groups you are member of. When I tried to authenticate against the local openldap db I found that the group object doesn’t have any email addresses.

I can see that the famille group has 3 members, but no email address. The same is true for the samba group object. Btw. did you enable email address for these groups? If so, then perhaps the group object should or might be extended with an email field.

Btw. just to show what my openldap stuff looks like. My group has no email address field either. However, I used the “description” ldap field to hold an email address, and added “description” to the mailattrs variable in config-site.php. I know, it’s ridiculous and it sucks, but it worked

I think I need to extend the group object with some proper email attributes. Just didn’t have the time to work on it. Perhaps the nethserver team has an ldap expert.

stephdl · February 12, 2021, 7:42am

@davidep is the father of samba AD, not sure for openldap but he put his hands inside.

Davidep could you advice, the link Nethserver-piler needs your tests

I put the configuration of piler to authenticate and the output of LDAP/AD

mark_nl · February 14, 2021, 6:49am

Hi @stephdl, (Thinking out loud, maybe it helps…)

I do not understands why you do not use the configuration vars provided by NethServer::SSSD like $sssd->userDN() and $sssd->groupDN()

EDIT Seems it does on ldap
~~more over IIRC NS does not populate the email field (line 49) in ldap nor AD.~~

stephdl · February 14, 2021, 7:01am

I use it :-?

github.com

stephdl/nethserver-piler/blob/1711166b1d589d646838077b991cc88b0d0d3668/root/etc/e-smith/templates/var/lib/docker/volumes/piler_piler_etc/_data/config-site.php/10base#L38


      
              } elsif ($piler{'Authentication'} eq 'ldap') {
                  use NethServer::SSSD;
                  my $sssd = new NethServer::SSSD();
                  my $bindPassword = $sssd->bindPassword();
                  my $baseDN = $sssd->baseDN();
                  # Piler is case sensitive for ldap filter
                  my $UPbaseDN = $baseDN ;
                  $UPbaseDN =~ s/dc=/DC=/g;
                  my $bindDN = $sssd->bindDN();
                  my $port = $sssd->port();
                  my $userDN = $sssd->userDN();
                  my $host = $sssd->host();
                  my $ldap = $sssd->isLdap();
          
                  if ($sssd->isLdap()) {
                      $host = ($host eq '127.0.0.1') ? $SystemName.'.'.$DomainName : $host;
                      $OUT.= qq(
          \$config['ENABLE_LDAP_AUTH'] = 1;
          \$config['LDAP_HOST'] = 'ldaps://$host';
          \$config['LDAP_HELPER_DN'] = '$bindDN';
          \$config['LDAP_HELPER_PASSWORD'] = '$bindPassword';

stephdl · February 14, 2021, 7:02am

I used it only one time for dokuwiki

github.com

stephdl/nethserver-dokuwiki/blob/f40d15f6d1888e6a278f29782a81b24821ee4824/root/etc/e-smith/templates/etc/dokuwiki/local.protected.php/10auth#L13


      
          use esmith::util;
          use NethServer::SSSD;
          my $sssd = new NethServer::SSSD();
          
          my $quotedBindPass = $sssd->bindPassword();
          $quotedBindPass =~ s/\'/\\'/g;
          
          #specific OpenLDAP
          my $server = $sssd->ldapURI() .':'. $sssd->port();
          my $usertree = $sssd->userDN();
          my $grouptree = $sssd->groupDN();
          my $binddn = $sssd->bindDN();
          my $startTls = $sssd->startTls() || '0';
          
          #specific Samba4 AD
          my $host = $sssd->ldapURI();
          my $baseDN = $sssd->baseDN();
          $tls = $sssd->startTls();
          
          #retrieve the dokuwiki authentication
          my $auth = $dokuwiki{'Authentication'} || 'ldap';

mark_nl · February 14, 2021, 7:06am

Must be overseeing it:
do not find where de scalar (var) my $userDN is used…

EDIT

we use it in SOGo too

mark_nl · February 14, 2021, 7:22am

do I understand correctly piler is case sensitive only for AD-ldap filter?