Certs from Letsencrypt renewal not propagated to https and Admin servers

EddieA · August 18, 2017, 11:53pm

Tried to log on to my local Admin UI today and Chrome threw a certificate error. Hmmmm.

Looking into it, it looks like after the automatic renew of the certificates on July 19th the new certificates weren’t propagated though to the files in the /etc/pki tree, which is where the https and Admin UI servers grab the certs from:

[root@Nethserver ~]# ls -lrt /etc/letsencrypt/live/nethserver.bogolinux.net/
total 4
-rw-r--r-- 1 root root 543 Mar 19 18:26 README
lrwxrwxrwx 1 root root  51 Jul 19 03:33 privkey.pem -> ../../archive/nethserver.bogolinux.net/privkey5.pem
lrwxrwxrwx 1 root root  53 Jul 19 03:33 fullchain.pem -> ../../archive/nethserver.bogolinux.net/fullchain5.pem
lrwxrwxrwx 1 root root  49 Jul 19 03:33 chain.pem -> ../../archive/nethserver.bogolinux.net/chain5.pem
lrwxrwxrwx 1 root root  48 Jul 19 03:33 cert.pem -> ../../archive/nethserver.bogolinux.net/cert5.pem
[root@Nethserver ~]#
[root@Nethserver ~]# ls -lrt /etc/letsencrypt/archive/nethserver.bogolinux.net/
total 80
-rw-r--r-- 1 root root 1708 Mar 19 18:26 privkey1.pem
-rw-r--r-- 1 root root 3469 Mar 19 18:26 fullchain1.pem
-rw-r--r-- 1 root root 1647 Mar 19 18:26 chain1.pem
-rw-r--r-- 1 root root 1822 Mar 19 18:26 cert1.pem
-rw-r--r-- 1 root root 1704 Mar 20 10:21 privkey2.pem
-rw-r--r-- 1 root root 3489 Mar 20 10:21 fullchain2.pem
-rw-r--r-- 1 root root 1647 Mar 20 10:21 chain2.pem
-rw-r--r-- 1 root root 1842 Mar 20 10:21 cert2.pem
-rw-r--r-- 1 root root 1708 Mar 20 12:00 privkey3.pem
-rw-r--r-- 1 root root 3526 Mar 20 12:00 fullchain3.pem
-rw-r--r-- 1 root root 1647 Mar 20 12:00 chain3.pem
-rw-r--r-- 1 root root 1879 Mar 20 12:00 cert3.pem
-rw-r--r-- 1 root root 1708 May 20 03:38 privkey4.pem
-rw-r--r-- 1 root root 1879 May 20 03:38 cert4.pem
-rw-r--r-- 1 root root 3526 May 20 03:38 fullchain4.pem
-rw-r--r-- 1 root root 1647 May 20 03:38 chain4.pem
-rw-r--r-- 1 root root 1704 Jul 19 03:33 privkey5.pem
-rw-r--r-- 1 root root 3526 Jul 19 03:33 fullchain5.pem
-rw-r--r-- 1 root root 1647 Jul 19 03:33 chain5.pem
-rw-r--r-- 1 root root 1879 Jul 19 03:33 cert5.pem
[root@Nethserver ~]#
[root@Nethserver ~]# ls -l /etc/pki/tls/certs/
total 24
lrwxrwxrwx  1 root root   49 Jun 13 18:22 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx  1 root root   55 Jun 13 18:22 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
-rw-------  1 root root 3527 May 22 12:05 httpd-admin.crt
-rw-------  1 root root 3527 May 22 12:05 localhost.crt
-rwxr-xr-x  1 root root  610 Feb 20 06:41 make-dummy-cert
-rw-r--r--  1 root root 2388 Feb 20 06:41 Makefile
-rw-r--r--. 1 root root 1484 Mar  1 19:17 NSRV.crt
-rwxr-xr-x  1 root root  829 Feb 20 06:41 renew-dummy-cert
[root@Nethserver ~]#

[root@Nethserver ~]# grep SSLCertificate /etc/httpd/conf.d/ssl.conf
# Point SSLCertificateFile at a PEM encoded certificate.  If
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
#   Point SSLCertificateChainFile at a file containing the
#   the referenced file can be the same as SSLCertificateFile
#SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt
[root@Nethserver ~]#
[root@Nethserver ~]# grep SSLCertificate /etc/httpd/admin-conf/httpd.conf
SSLCertificateChainFile /etc/letsencrypt/live/nethserver.bogolinux.net/chain.pem
SSLCertificateFile /etc/pki/tls/certs/httpd-admin.crt
SSLCertificateKeyFile /etc/pki/tls/private/httpd-admin.key
[root@Nethserver ~]#

httpd-admin.crt and localhost.crt are taken from the xxx4 set of certificates which expired today.

Cheers.

phonon112358 · August 20, 2017, 1:57am

I have the same issue but no direct solution…

giacomo · August 21, 2017, 6:56am

IIRC the httpd-admin daemon is reloaded asynchronously.
See this commit:

If you need to force the full reload, please use:

signal-event nethserver-httpd-admin-update

Edit: this is the expected behavior, since reload the httpd-daemon when the users are connected can have unexpected behavior. For example, ajax calls will fail in case of certificate change.

phonon112358 · August 21, 2017, 1:17pm

thanks!

EddieA · August 21, 2017, 3:46pm

Yes, it looks like that happened when I loaded that particular fix on May 22nd based on the dates of the http-admin.crt and localhost.crt.

But it appears not to have been done when Letsencrypt renewed the certificates. Or are you saying that I always have to remember when the certificates are renewed automatically and then manually run a command to activate them. Not really a solution.

Cheers.

giacomo · August 22, 2017, 7:14am

After renewal, Let’s Encrypt script will fire the certificate-update event:

github.com

NethServer/nethserver-letsencrypt/blob/master/root/usr/libexec/nethserver/letsencrypt-certs#L159


 exit 0;
}


# Renew certificate for all domains
renew(\@domains);


if ($modified > 0) {
if ($verbose) {
   print "Executing certificate-update event...\n";
}
if(esmith::event::event_signal('certificate-update') == 0) {
    exit 1;
}
}


exit 0;

The event restarts httpd-admin:

github.com

NethServer/nethserver-httpd-admin/blob/master/createlinks#L52


event_services('nethserver-httpd-admin-update', qw(
 httpd-admin reload
));
 
#
# certificate-update event
#
event_templates('certificate-update', @certificates);
event_templates('certificate-update', '/etc/httpd/admin-conf/httpd.conf');


event_actions('certificate-update', qw(
    nethserver-httpd-admin-asyncreload 99
));


#
# system-init event
#
event_actions("system-init", qw(
enable-firstconfigwizard 20
));

If you feel there is a bug, please post some extract from logs to track down the problem.

phonon112358 · August 22, 2017, 12:18pm

Somehow it seems to me that the last renewal of letsencrypt certificate didn’t fire the certificate-update event…
I had to fire it manually to get the LDAP server working again (see Not able to Update User Password or Add new User)…
I don’t know which logs are interesting for you…

giacomo · August 22, 2017, 1:45pm

Check /var/log/messages and /var/log/cron to see if something failed.

Please also be sure to have latest packages installed.

phonon112358 · August 22, 2017, 2:06pm

Ahhh… now I remember! I did the renewal manually via the command line.
hence, there is no real bug of Nethserver - I just have to remember that I have to perform the certificate-update event if I manually renew the certificate in future.

Thus, at least for me, everything is fine…!

EddieA · August 22, 2017, 4:33pm

It looks like that didn’t happen, at least for me.

Running the event manually did update the certificates:

[root@Nethserver certificate-update]# ls -l /etc/pki/tls/certs/
total 24
lrwxrwxrwx  1 root root   49 Jun 13 18:22 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx  1 root root   55 Jun 13 18:22 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
-rw-------  1 root root 3527 Aug 22 09:27 httpd-admin.crt
-rw-------  1 root root 3527 Aug 22 09:27 localhost.crt
-rwxr-xr-x  1 root root  610 Feb 20  2017 make-dummy-cert
-rw-r--r--  1 root root 2388 Feb 20  2017 Makefile
-rw-r--r--. 1 root root 1484 Mar  1 19:17 NSRV.crt
-rwxr-xr-x  1 root root  829 Feb 20  2017 renew-dummy-cert
[root@Nethserver certificate-update]#

Unfortunately NS only keeps the previous 4 weeks of logs and this happened prior to those, so I can’t check what was reported.

But, I will check next month when Letsencrypt is due to renew again.

Cheers.

filippo_carletti · August 23, 2017, 7:23am

By default. You can customize log rotation:
http://docs.nethserver.org/projects/nethserver-devel/en/v6/logs.html

These instructions also apply to version 7, manual here:
http://docs.nethserver.org/projects/nethserver-devel/en/v7/nethserver-base.html#log-retention-and-rotation

EddieA · September 28, 2017, 7:37am

Nope, not seeing that in the logs. Unfortunately I re-booted after the certificates were renewed, but before I remembered to go grab all the information on this. So, because of that, the nethserver-httpd-admin-update event did run, but only updated the Admin cert and not the https one:

[root@Nethserver ~]# cd /etc/pki/tls/certs/
[root@Nethserver certs]# ls -l
total 24
lrwxrwxrwx  1 root root   49 Sep 27 23:27 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx  1 root root   55 Sep 27 23:27 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
-rw-------  1 root root 3527 Sep 27 23:37 httpd-admin.crt
-rw-------  1 root root 3527 Aug 22 09:27 localhost.crt
-rwxr-xr-x  1 root root  610 Aug  3 23:41 make-dummy-cert
-rw-r--r--  1 root root 2516 Aug  3 23:41 Makefile
-rw-r--r--. 1 root root 1484 Mar  1  2017 NSRV.crt
-rwxr-xr-x  1 root root  829 Aug  3 23:41 renew-dummy-cert
[root@Nethserver certs]#

[root@Nethserver ~]# ls -lrt /var/log//letsencrypt/
total 1624
---  snip lots ---
-rw-r--r-- 1 root root  7580 Sep 15 03:22 letsencrypt.log.12
-rw-r--r-- 1 root root  7580 Sep 16 03:33 letsencrypt.log.11
-rw-r--r-- 1 root root 55265 Sep 17 03:14 letsencrypt.log.10
-rw-r--r-- 1 root root  7580 Sep 18 03:50 letsencrypt.log.9
-rw-r--r-- 1 root root  7580 Sep 19 03:42 letsencrypt.log.8

(Side issue: these logs need to be trimmed somehow. I’m currently up to 198).

From /var/log/cron:

Sep 17 03:14:01 Nethserver anacron[27151]: Job `cron.daily' started
Sep 17 03:14:01 Nethserver run-parts(/etc/cron.daily)[27286]: starting 0logwatch
Sep 17 03:14:05 Nethserver run-parts(/etc/cron.daily)[27702]: finished 0logwatch
Sep 17 03:14:05 Nethserver run-parts(/etc/cron.daily)[27286]: starting backup-co
nfig-cron
Sep 17 03:14:12 Nethserver run-parts(/etc/cron.daily)[28125]: finished backup-co
nfig-cron
Sep 17 03:14:12 Nethserver run-parts(/etc/cron.daily)[27286]: starting collectd_
cleanup
Sep 17 03:14:12 Nethserver run-parts(/etc/cron.daily)[28134]: finished collectd_
cleanup
Sep 17 03:14:12 Nethserver run-parts(/etc/cron.daily)[27286]: starting duc-index
Sep 17 03:14:19 Nethserver run-parts(/etc/cron.daily)[28151]: finished duc-index
Sep 17 03:14:19 Nethserver run-parts(/etc/cron.daily)[27286]: starting logrotate
Sep 17 03:14:19 Nethserver run-parts(/etc/cron.daily)[28159]: finished logrotate
Sep 17 03:14:19 Nethserver run-parts(/etc/cron.daily)[27286]: starting man-db.cr
on
Sep 17 03:14:21 Nethserver run-parts(/etc/cron.daily)[28170]: finished man-db.cr
on
Sep 17 03:14:21 Nethserver run-parts(/etc/cron.daily)[27286]: starting nethserve
r-check-builtin-certificate
Sep 17 03:14:21 Nethserver run-parts(/etc/cron.daily)[28181]: finished nethserve
r-check-builtin-certificate
Sep 17 03:14:21 Nethserver run-parts(/etc/cron.daily)[27286]: starting nethserve
r-check-uploaded-certificates
Sep 17 03:14:21 Nethserver run-parts(/etc/cron.daily)[28205]: finished nethserve
r-check-uploaded-certificates
Sep 17 03:14:21 Nethserver run-parts(/etc/cron.daily)[27286]: starting nethserve
r-letsencrypt-certs
Sep 17 03:14:31 Nethserver run-parts(/etc/cron.daily)[28220]: finished nethserve
r-letsencrypt-certs
Sep 17 03:14:31 Nethserver run-parts(/etc/cron.daily)[27286]: starting nethserve
r-mail-spam-expunge
Sep 17 03:14:32 Nethserver run-parts(/etc/cron.daily)[28253]: finished nethserve
r-mail-spam-expunge
Sep 17 03:14:32 Nethserver anacron[27151]: Job `cron.daily' terminated
Sep 17 03:14:32 Nethserver anacron[27151]: Normal exit (1 job run)

And from /var/log/messages:

Sep 17 03:10:01 Nethserver systemd: Starting Session 19596 of user root.
Sep 17 03:14:05 Nethserver esmith::event[27716]: Event: pre-backup-config
Sep 17 03:14:05 Nethserver esmith::event[27716]: expanding /etc/backup-config.d/
nethserver-sssd.include
Sep 17 03:14:05 Nethserver dbus[1046]: [system] Activating via systemd: service
name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service'
Sep 17 03:14:05 Nethserver dbus-daemon: dbus[1046]: [system] Activating via syst
emd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedat
e1.service'
Sep 17 03:14:05 Nethserver systemd: Starting Time & Date Service...
Sep 17 03:14:05 Nethserver dbus[1046]: [system] Successfully activated service 'org.freedesktop.timedate1'
Sep 17 03:14:05 Nethserver dbus-daemon: dbus[1046]: [system] Successfully activated service 'org.freedesktop.timedate1'
Sep 17 03:14:05 Nethserver systemd: Started Time & Date Service.
Sep 17 03:14:05 Nethserver esmith::event[27716]: Action: /etc/e-smith/events/actions/generic_template_expand SUCCESS [0.223171]
Sep 17 03:14:07 Nethserver esmith::event[27716]: Action: /etc/e-smith/events/pre-backup-config/S40nethserver-dc-pre-backup SUCCESS [1.714494]
Sep 17 03:14:07 Nethserver esmith::event[27716]: Action: /etc/e-smith/events/pre-backup-config/S40nethserver-sssd-backup-tdb SUCCESS [0.03083]
Sep 17 03:14:08 Nethserver esmith::event[27716]: Action: /etc/e-smith/events/pre-backup-config/S50nethserver-backup-config-list-packages SUCCESS [0.750751]
Sep 17 03:14:08 Nethserver esmith::event[27716]: Event: pre-backup-config SUCCESS
Sep 17 03:14:12 Nethserver esmith::event[28089]: Event: post-backup-config
Sep 17 03:14:12 Nethserver esmith::event[28089]: Action: /etc/e-smith/events/post-backup-config/S40nethserver-dc-post-backup SUCCESS [0.040589]
Sep 17 03:14:12 Nethserver esmith::event[28089]: Event: post-backup-config SUCCESS
Sep 17 03:14:25 Nethserver clamd: SelfCheck: Database status OK.
Sep 17 03:14:32 Nethserver systemd: Removed slice user-0.slice.
Sep 17 03:14:32 Nethserver systemd: Stopping user-0.slice.
Sep 17 03:15:01 Nethserver systemd: Created slice user-0.slice.

Running openssl against port 980 and 443 shows 980 using the latest cert, as (kinda) expected because the admin even ran as part of the reboot, but 443 still shows the old cert.

Maybe next time I’ll remember to capture everything before a reboot.

Cheers.

EddieA · November 18, 2017, 12:17am

OK, my system did the latest renewal, and yet again, the current certs weren’t refreshed:

[root@Nethserver ~]# ls -lrt /var/log/letsencrypt/
total 1880
-rw-r--r-- 1 root root     0 Mar 19  2017 letsencrypt.log.249
-rw-r--r-- 1 root root 31851 Mar 19  2017 letsencrypt.log.248
-- snip loads -- -- Maybe there should be a tidy-up job for this directory --
-rw-r--r-- 1 root root  3625 Nov 14 03:48 letsencrypt.log.3
-rw-r--r-- 1 root root  3626 Nov 15 03:46 letsencrypt.log.2
-rw-r--r-- 1 root root 57115 Nov 16 03:20 letsencrypt.log.1
-rw-r--r-- 1 root root  3626 Nov 17 03:20 letsencrypt.log
[root@Nethserver ~]#

[root@Nethserver ~]# ls -lrt /etc/letsencrypt/live/nethserver.bogolinux.net/
total 4
-rw-r--r-- 1 root root 543 Mar 19  2017 README
lrwxrwxrwx 1 root root  51 Nov 16 03:20 privkey.pem -> ../../archive/nethserver.bogolinux.net/privkey7.pem
lrwxrwxrwx 1 root root  53 Nov 16 03:20 fullchain.pem -> ../../archive/nethserver.bogolinux.net/fullchain7.pem
lrwxrwxrwx 1 root root  49 Nov 16 03:20 chain.pem -> ../../archive/nethserver.bogolinux.net/chain7.pem
lrwxrwxrwx 1 root root  48 Nov 16 03:20 cert.pem -> ../../archive/nethserver.bogolinux.net/cert7.pem
[root@Nethserver ~]# ls -lrt /etc/letsencrypt/archive/nethserver.bogolinux.net/
total 112
-- snip a bunch of lines --
-rw-r--r-- 1 root root 1704 Sep 17 03:14 privkey6.pem
-rw-r--r-- 1 root root 3526 Sep 17 03:14 fullchain6.pem
-rw-r--r-- 1 root root 1647 Sep 17 03:14 chain6.pem
-rw-r--r-- 1 root root 1879 Sep 17 03:14 cert6.pem
-rw-r--r-- 1 root root 1704 Nov 16 03:20 privkey7.pem
-rw-r--r-- 1 root root 3526 Nov 16 03:20 fullchain7.pem
-rw-r--r-- 1 root root 1647 Nov 16 03:20 chain7.pem
-rw-r--r-- 1 root root 1879 Nov 16 03:20 cert7.pem
[root@Nethserver ~]# ls -lrt /etc/pki/tls/certs/
total 24
-rw-r--r--. 1 root root 1484 Mar  1  2017 NSRV.crt
-rwxr-xr-x  1 root root  829 Aug  3 23:41 renew-dummy-cert
-rw-r--r--  1 root root 2516 Aug  3 23:41 Makefile
-rwxr-xr-x  1 root root  610 Aug  3 23:41 make-dummy-cert
lrwxrwxrwx  1 root root   55 Sep 27 23:27 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
lrwxrwxrwx  1 root root   49 Sep 27 23:27 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
-rw-------  1 root root 3527 Sep 27 23:37 httpd-admin.crt
-rw-------  1 root root 3527 Oct  4 14:09 localhost.crt
[root@Nethserver ~]#

From /var/log/cron:

Nov 16 03:20:01 Nethserver anacron[22698]: Job `cron.daily' started
Nov 16 03:20:01 Nethserver run-parts(/etc/cron.daily)[22837]: starting 0logwatch
Nov 16 03:20:06 Nethserver run-parts(/etc/cron.daily)[23274]: finished 0logwatch
Nov 16 03:20:06 Nethserver run-parts(/etc/cron.daily)[22837]: starting backup-config-cron
Nov 16 03:20:13 Nethserver run-parts(/etc/cron.daily)[23735]: finished backup-config-cron
Nov 16 03:20:13 Nethserver run-parts(/etc/cron.daily)[22837]: starting collectd_cleanup
Nov 16 03:20:13 Nethserver run-parts(/etc/cron.daily)[23744]: finished collectd_cleanup
Nov 16 03:20:13 Nethserver run-parts(/etc/cron.daily)[22837]: starting duc-index
Nov 16 03:20:18 Nethserver run-parts(/etc/cron.daily)[23758]: finished duc-index
Nov 16 03:20:18 Nethserver run-parts(/etc/cron.daily)[22837]: starting logrotate
Nov 16 03:20:19 Nethserver run-parts(/etc/cron.daily)[23766]: finished logrotate
Nov 16 03:20:19 Nethserver run-parts(/etc/cron.daily)[22837]: starting man-db.cron
Nov 16 03:20:21 Nethserver run-parts(/etc/cron.daily)[23921]: finished man-db.cron
Nov 16 03:20:21 Nethserver run-parts(/etc/cron.daily)[22837]: starting nethserver-check-builtin-certificate
Nov 16 03:20:21 Nethserver run-parts(/etc/cron.daily)[23932]: finished nethserver-check-builtin-certificate
Nov 16 03:20:21 Nethserver run-parts(/etc/cron.daily)[22837]: starting nethserver-check-uploaded-certificates
Nov 16 03:20:21 Nethserver run-parts(/etc/cron.daily)[23956]: finished nethserver-check-uploaded-certificates
Nov 16 03:20:21 Nethserver run-parts(/etc/cron.daily)[22837]: starting nethserver-letsencrypt-certs
Nov 16 03:20:30 Nethserver run-parts(/etc/cron.daily)[23972]: finished nethserver-letsencrypt-certs
Nov 16 03:20:30 Nethserver run-parts(/etc/cron.daily)[22837]: starting nethserver-mail-spam-expunge
Nov 16 03:20:31 Nethserver run-parts(/etc/cron.daily)[24005]: finished nethserver-mail-spam-expunge
Nov 16 03:20:31 Nethserver anacron[22698]: Job `cron.daily' terminated

And lastly, from /var/log/messages:

Nov 16 03:15:05 Nethserver nmbd[1703]:  This response was from IP 192.168.0.1, reporting an IP address of 192.168.0.1.
Nov 16 03:20:06 Nethserver esmith::event[23287]: Event: pre-backup-config
Nov 16 03:20:06 Nethserver esmith::event[23287]: expanding /etc/backup-config.d/nethserver-sssd.include
Nov 16 03:20:06 Nethserver dbus[1071]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service'
Nov 16 03:20:06 Nethserver dbus-daemon: dbus[1071]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service'
Nov 16 03:20:06 Nethserver systemd: Starting Time & Date Service...
Nov 16 03:20:06 Nethserver dbus[1071]: [system] Successfully activated service 'org.freedesktop.timedate1'
Nov 16 03:20:06 Nethserver dbus-daemon: dbus[1071]: [system] Successfully activated service 'org.freedesktop.timedate1'
Nov 16 03:20:06 Nethserver systemd: Started Time & Date Service.
Nov 16 03:20:06 Nethserver esmith::event[23287]: Action: /etc/e-smith/events/actions/generic_template_expand SUCCESS [0.187284]
Nov 16 03:20:07 Nethserver nmbd[1703]: [2017/11/16 03:20:07.940611,  0] ../source3/nmbd/nmbd_namequery.c:109(query_name_response)
Nov 16 03:20:07 Nethserver nmbd[1703]:  query_name_response: Multiple (2) responses received for a query on subnet 192.168.0.254 for name DISCWORLD<1d>.
Nov 16 03:20:07 Nethserver nmbd[1703]:  This response was from IP 192.168.0.1, reporting an IP address of 192.168.0.1.
Nov 16 03:20:08 Nethserver esmith::event[23287]: Action: /etc/e-smith/events/pre-backup-config/S40nethserver-dc-pre-backup SUCCESS [1.479885]
Nov 16 03:20:08 Nethserver esmith::event[23287]: Action: /etc/e-smith/events/pre-backup-config/S40nethserver-sssd-backup-tdb SUCCESS [0.00547]
Nov 16 03:20:09 Nethserver esmith::event[23287]: Action: /etc/e-smith/events/pre-backup-config/S50nethserver-backup-config-list-packages SUCCESS [0.839394]
Nov 16 03:20:09 Nethserver esmith::event[23287]: Event: pre-backup-config SUCCESS
Nov 16 03:20:13 Nethserver esmith::event[23700]: Event: post-backup-config
Nov 16 03:20:13 Nethserver esmith::event[23700]: Action: /etc/e-smith/events/post-backup-config/S40nethserver-dc-post-backup SUCCESS [0.030478]
Nov 16 03:20:13 Nethserver esmith::event[23700]: Event: post-backup-config SUCCESS
Nov 16 03:25:07 Nethserver nmbd[1703]: [2017/11/16 03:25:07.625975,  0] ../source3/nmbd/nmbd_namequery.c:109(query_name_response)

Is there anything else you need captured.

Cheers.

EddieA · November 27, 2017, 7:06pm

Is the information posted above sufficient. If not, what else do you need.

Cheers.

giacomo · November 28, 2017, 1:16pm

You need to cut & paste logs relevant to Let’s encrypt renewal. Did you see a certificate-update event inside the logs?

That is the script which does the restart job of all needed services.

EddieA · November 28, 2017, 4:33pm

Here’s the relevant log from /var/log/letsencrypt. It’s the end of the log after the new certificates are stored:

2017-11-16 11:20:30,127:DEBUG:certbot.storage:Writing new private key to /etc/letsencrypt/archive/nethserver.bogolinux.net/privkey7.pem.
2017-11-16 11:20:30,127:DEBUG:certbot.storage:Writing certificate to /etc/letsencrypt/archive/nethserver.bogolinux.net/cert7.pem.
2017-11-16 11:20:30,127:DEBUG:certbot.storage:Writing chain to /etc/letsencrypt/archive/nethserver.bogolinux.net/chain7.pem.
2017-11-16 11:20:30,127:DEBUG:certbot.storage:Writing full chain to /etc/letsencrypt/archive/nethserver.bogolinux.net/fullchain7.pem.
2017-11-16 11:20:30,150:DEBUG:certbot.plugins.selection:Requested authenticator webroot and installer <certbot.cli._Default object at 0x256dc50>
2017-11-16 11:20:30,150:DEBUG:certbot.cli:Default Detector is Namespace(account=<certbot.cli._Default object at 0x253e5d0>, agree_dev_preview=None, allow_subset_of_names=<certbot.cli._Default object at 0x2565810>, apache=<certbot.cli._Default object at 0x256df50>, authenticator='webroot', break_my_certs=<certbot.cli._Default object at 0x256b550>, cert_path=<certbot.cli._Default object at 0x256d450>, certname=<certbot.cli._Default object at 0x253e110>, chain_path=<certbot.cli._Default object at 0x256d750>, checkpoints=<certbot.cli._Default object at 0x2540f10>, config_dir=<certbot.cli._Default object at 0x256d850>, config_file=None, configurator=<certbot.cli._Default object at 0x256dc50>, csr=<certbot.cli._Default object at 0x2540d10>, debug=<certbot.cli._Default object at 0x253ee10>, debug_challenges=<certbot.cli._Default object at 0x253ef10>, deploy_hook=<certbot.cli._Default object at 0x2540590>, dialog=None, directory_hooks=<certbot.cli._Default object at 0x2540790>, dns_cloudflare=<certbot.cli._Default object at 0x256e490>, dns_cloudxns=<certbot.cli._Default object at 0x256e590>, dns_digitalocean=<certbot.cli._Default object at 0x256e690>, dns_dnsimple=<certbot.cli._Default object at 0x256e790>, dns_dnsmadeeasy=<certbot.cli._Default object at 0x256e890>, dns_google=<certbot.cli._Default object at 0x256e990>, dns_luadns=<certbot.cli._Default object at 0x256ea90>, dns_nsone=<certbot.cli._Default object at 0x256eb90>, dns_rfc2136=<certbot.cli._Default object at 0x256ec90>, dns_route53=<certbot.cli._Default object at 0x256ed90>, domains='www.BogoLinux.net', dry_run=<certbot.cli._Default object at 0x253e210>, duplicate=<certbot.cli._Default object at 0x253e3d0>, eff_email=<certbot.cli._Default object at 0x253e610>, email='root@bogolinux.net', expand=<certbot.cli._Default object at 0x253e910>, force_interactive=<certbot.cli._Default object at 0x2565ed0>, fullchain_path=<certbot.cli._Default object at 0x256d650>, func=<function certonly at 0x215ec08>, hsts=<certbot.cli._Default object at 0x256ba50>, http01_address=<certbot.cli._Default object at 0x256b450>, http01_port=<certbot.cli._Default object at 0x256b350>, ifaces=<certbot.cli._Default object at 0x256d250>, init=<certbot.cli._Default object at 0x256d050>, installer=<certbot.cli._Default object at 0x256dc50>, key_path=<certbot.cli._Default object at 0x256d550>, logs_dir=<certbot.cli._Default object at 0x256da50>, manual=<certbot.cli._Default object at 0x256e290>, manual_auth_hook=<certbot.cli._Default object at 0x256eed0>, manual_cleanup_hook=<certbot.cli._Default object at 0x2570050>, manual_public_ip_logging_ok=<certbot.cli._Default object at 0x2570150>, max_log_backups=<certbot.cli._Default object at 0x2565cd0>, must_staple=<certbot.cli._Default object at 0x256b750>, nginx=<certbot.cli._Default object at 0x256e090>, no_bootstrap=<certbot.cli._Default object at 0x253eb10>, no_self_upgrade=<certbot.cli._Default object at 0x253ea10>, no_verify_ssl=<certbot.cli._Default object at 0x256b050>, noninteractive_mode=True, num=<certbot.cli._Default object at 0x2540a10>, os_packages_only=<certbot.cli._Default object at 0x253e1d0>, post_hook=<certbot.cli._Default object at 0x2540390>, pre_hook=<certbot.cli._Default object at 0x2540290>, pref_challs=<certbot.cli._Default object at 0x2540190>, prepare=<certbot.cli._Default object at 0x256d150>, quiet=True, reason=<certbot.cli._Default object at 0x2540e10>, redirect=<certbot.cli._Default object at 0x256b850>, register_unsafely_without_email=<certbot.cli._Default object at 0x253e310>, reinstall=<certbot.cli._Default object at 0x253e810>, renew_by_default=<certbot.cli._Default object at 0x2565d50>, renew_hook=<certbot.cli._Default object at 0x2540490>, renew_with_new_domains=<certbot.cli._Default object at 0x2565b50>, rsa_key_size=<certbot.cli._Default object at 0x256b650>, server=<certbot.cli._Default object at 0x256db50>, staging=<certbot.cli._Default object at 0x253ed10>, standalone=<certbot.cli._Default object at 0x256e190>, standalone_supported_challenges=<certbot.cli._Default object at 0x2570290>, staple=<certbot.cli._Default object at 0x256be50>, strict_permissions=<certbot.cli._Default object at 0x2540090>, text_mode=True, tls_sni_01_address=<certbot.cli._Default object at 0x256b250>, tls_sni_01_port=<certbot.cli._Default object at 0x256b150>, tos=True, uir=<certbot.cli._Default object at 0x256bc50>, update_registration=<certbot.cli._Default object at 0x253e410>, user_agent=<certbot.cli._Default object at 0x2540b10>, user_agent_comment=<certbot.cli._Default object at 0x2540bd0>, validate_hooks=<certbot.cli._Default object at 0x2540690>, verb='certonly', verbose_count=<certbot.cli._Default object at 0x2565ad0>, webroot=True, webroot_map=<certbot.cli._Default object at 0x2570490>, webroot_path='/var/www/html/', work_dir=<certbot.cli._Default object at 0x256d950>)
2017-11-16 11:20:30,342:DEBUG:certbot.storage:Writing new config /etc/letsencrypt/renewal/nethserver.bogolinux.net.conf.new.
2017-11-16 11:20:30,344:DEBUG:certbot.reporter:Reporting to user: Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/nethserver.bogolinux.net/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/nethserver.bogolinux.net/privkey.pem
Your cert will expire on 2018-02-14. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew"
2017-11-16 11:20:30,344:DEBUG:certbot.reporter:Reporting to user: If you like Certbot, please consider supporting our work by:

Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
Donating to EFF:                    https://eff.org/donate-le

Not sure what other logs are needed, as I pasted the relevant sections from /var/log/cron and /var/log/messages earlier.

Cheers.

giacomo · November 29, 2017, 11:11am

Please try to find the certificate-update event inside logs:

grep certificate-update /var/log/messages*

Honestly I can’t see any error on your logs.

Edit: just a question: is Let’s Encrypt cert set as default for all services?

EddieA · November 29, 2017, 5:34pm

[root@Nethserver ~]# grep certificate-update /var/log/messages*
[root@Nethserver ~]#

The /var/log/messages I pasted covers the period that the cron process ran and it shows that certificate-update was not run as part of the renewal process.

I thought so. Is this sufficient:

[root@Nethserver ~]# openssl s_client -connect localhost:980
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = bogolinux.net
verify return:1
---
Certificate chain
 0 s:/CN=bogolinux.net
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---

[root@Nethserver ~]# openssl s_client -connect localhost:443
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = bogolinux.net
verify return:1
---
Certificate chain
 0 s:/CN=bogolinux.net
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---

Cheers.

EddieA · November 29, 2017, 6:28pm

I wonder if this issue is still in play here ??

*** UPDATE ***

Yeah, it looks like it:

[root@Nethserver db]# config getprop pki LetsEncryptDomains
BogoLinux.net,Nethserver.BogoLinux.net,smtp.BogoLinux.net,www.BogoLinux.net
[root@Nethserver db]# ls -l /etc/letsencrypt/live/
total 0
drwxr-xr-x 2 root root 93 Nov 16 03:20 nethserver.bogolinux.net
[root@Nethserver db]#

So this part of the renew code doesn’t get executed:

if ($modified > 0) {
   if ($verbose) {
      print "Executing certificate-update event...\n";
   }
   if(esmith::event::event_signal('certificate-update') == 0) {
       exit 1;
   }
}

*** 2nd UPDATE ***

Would this work correctly even if the case wasn’t an issue:

# file paths
my $crt = $crtdir."/live/".${$domains}[0]."/cert.pem";

If I’m reading that correctly, then it would use the first property entry to construct the directory name, which is just the domain, but the actual directory name is the FQDN of the server.

Cheers.

EddieA · December 1, 2017, 9:42pm

No it doesn’t. It behaves exactly like I suggest in my final comment.

# file paths
my $crt = $crtdir."/live/".${$domains}[0]."/cert.pem";
print $crt."\n";
# read the date of certificate link before renewal
my $tmp = stat($crt);
my $before = defined($tmp) ? $tmp->mtime : 0;
print $before."\n";


[root@Nethserver ~]# ./letsencrypt-certs
/etc/letsencrypt//live/BogoLinux.net/cert.pem
0
[root@Nethserver ~]#

So there are two issues, either one would cause the “certificate-update” to not run.

Cheers.