All certificates are expired or expiring - nothing renewing

[Nuke]

NethServer Version: NS8 up to date
Module: traefik1 v3.2; core 3.9

This afternoon my wife, daughter and I have noticed that we can’t reach Nextcloud or mail (neither webmail/roundcube or NS8 mail). The certificates have expired as of ~45-60 minute ago.

The GUI console shows green lights for all certificates but each app is getting a pop-up that the certificate from Let’s Encrypt has expired.

I’ve been reading through other posts but it doesn’t really look like any similar issues. If I missed a post, please direct me to the right topic.

Is there a way to force an update of certificates? If yes, where do I find the instructions.

Before someone asks, port 80 is open on my OPNsense firewall so it should be able to get certificates from 443 or 80.

Thanks.

[mrmarkuz (Markus Neuberger)]

You may need to manually delete the certs from the traefik config to make the renewal work again.

SSL certificate error ERR_CERT_DATE_INVALID

This list of names is very long. Apart from nv.twr.cz I think you can safely remove also the names for nextcloud, collabora and mattermost because they are web applications, they define an HTTP route, and their names do not be to be requested also from TLS certificates page with Let’s Encrypt. If Let’s Encrypt requirements are met for all of those names, manually delete redundant entries from the Traefik’s configuration with a simple text editor, e.g.: runagent -m traefik1 vi configs/_defaul…

[Nuke]

Hi @mrmarkuz

So if I go and manually delete all the certificates from Traefik’s configuration, does the GUI know and update the missing certificates that are in the GUI?

I think I’ll look to make a back up of this _default_cert.yml file before going to delete anything but if I do delete everything that has expired, how do I get it to update with a new certificate?

Is there some cli script or command to manually update all the certificates?

Is there something I can do to make sure that it refreshes again prior to the next 3 months so I don’t end up at the same point in September?

Thanks again.

[Nuke]

About 2 months ago I posted about not being able to delete a certificate for a domain that was created in error. I could never get that certificate to delete. So now I see from the error messages that this domain is causing the “error renewing certificates”.

Since that intranet.domain.tld is referenced through the other domains nothing updates.

Well this sucks.

# runagent -m traefik1 cat configs/_default_cert.yml
tls:
  stores:
    default:
      defaultGeneratedCert:
        resolver: acmeServer
        domain:
          main: webmail.domain.tld
          sans:
          - domain.tld
          - www.domain2.tld
          - ad.domain.tld
          - collabora.domain.tld
          - suitecrm.domain.tld
          - www.domain.tld
          - cloud.domain.tld
          - domain2.tld
          - ns8.domain.tld
          - **intranet.domain.tld**
          - mail.domain.tld

This looks wrong after looking at the traefik tls documentation. main should be the top level domain and sans should be the subdomains. The little f**ker intranet.domain.tld is still there.

# journalctl --grep acmeCA
Jun 26 19:16:47 ns8 traefik[948310]: 2025-06-26T23:16:47Z INF Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:16:49 ns8 traefik[948310]: 2025-06-26T23:16:49Z INF Renewing certificate from LE : {Main:intranet.domain.tld SANs:[]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:16:56 ns8 traefik[948310]: 2025-06-26T23:16:56Z ERR Error renewing certificate from LE: {intranet.domain.tld []} error="error: one or more domains had a problem:\n[intranet.domain.tld] invalid authorization: acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for intranet.domain.tld - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for intranet.domain.tld - check that a DNS record exists for this domain\n" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:16:56 ns8 traefik[948310]: 2025-06-26T23:16:56Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[intranet.domain.tld ns8.domain.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:18:53 ns8 traefik[948310]: 2025-06-26T23:18:53Z ERR Error renewing certificate from LE: {webmail.domain.tld [intranet.domain.tld ns8.domain.tld]} error="error: one or more domains had a problem:\n[intranet.domain.tld] invalid authorization: acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for intranet.domain.tld - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for intranet.domain.tld - check that a DNS record exists for this domain\n" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:18:53 ns8 traefik[948310]: 2025-06-26T23:18:53Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[cloud.domain.tld intranet.domain.tld ns8.domain.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:19:42 ns8 traefik[948310]: 2025-06-26T23:19:42Z ERR Error renewing certificate from LE: {webmail.domain.tld [cloud.domain.tld intranet.domain.tld ns8.domain.tld]} error="error: one or more domains had a problem:\n[intranet.domain.tld] invalid authorization: acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for intranet.domain.tld - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for intranet.domain.tld - check that a DNS record exists for this domain\n" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:19:42 ns8 traefik[948310]: 2025-06-26T23:19:42Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[cloud.domain.tld intranet.domain.tld ns8.domain.tld suitecrm.domain.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:38 ns8 traefik[948310]: 2025-06-26T23:20:38Z ERR Error renewing certificate from LE: {webmail.domain.tld [cloud.domain.tld intranet.domain.tld ns8.domain.tld suitecrm.domain.tld]} error="error: one or more domains had a problem:\n[intranet.domain.tld] invalid authorization: acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for intranet.domain.tld - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for intranet.domain.tld - check that a DNS record exists for this domain\n" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:38 ns8 traefik[948310]: 2025-06-26T23:20:38Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[intranet.domain.tld ad.domain.tld suitecrm.domain.tld ns8.domain.tld cloud.domain.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z ERR Error renewing certificate from LE: {webmail.domain.tld [intranet.domain.tld ad.domain.tld suitecrm.domain.tld ns8.domain.tld cloud.domain.tld]} error="error: one or more domains had a problem:\n[intranet.domain.tld] invalid authorization: acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for intranet.domain.tld - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for intranet.domain.tld - check that a DNS record exists for this domain\n" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[ad.domain.tld suitecrm.domain.tld ns8.domain.tld intranet.domain.tld collabora.domain.tld cloud.domain.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z ERR Error renewing certificate from LE: {webmail.domain.tld [ad.domain.tld suitecrm.domain.tld ns8.domain.tld intranet.domain.tld collabora.domain.tld cloud.domain.tld]} error="acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: too many failed authorizations (5) for \"intranet.domain.tld\" in the last 1h0m0s, retry after 2025-06-26 23:28:52 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[intranet.domain.tld ad.domain.tld ns8.domain.tld mail.domain.tld cloud.domain.tld collabora.domain.tld suitecrm.domain.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z ERR Error renewing certificate from LE: {webmail.domain.tld [intranet.domain.tld ad.domain.tld ns8.domain.tld mail.domain.tld cloud.domain.tld collabora.domain.tld suitecrm.domain.tld]} error="acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: too many failed authorizations (5) for \"intranet.domain.tld\" in the last 1h0m0s, retry after 2025-06-26 23:28:53 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[ad.domain.tld collabora.domain.tld cloud.domain.tld domain.tld intranet.domain.tld mail.domain.tld suitecrm.domain.tld ns8.domain.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z ERR Error renewing certificate from LE: {webmail.domain.tld [ad.domain.tld collabora.domain.tld cloud.domain.tld domain.tld intranet.domain.tld mail.domain.tld suitecrm.domain.tld ns8.domain.tld]} error="acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: too many failed authorizations (5) for \"intranet.domain.tld\" in the last 1h0m0s, retry after 2025-06-26 23:28:51 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[ad.domain.tld cloud.domain.tld suitecrm.domain.tld intranet.domain.tld domain.tld collabora.domain.tld mail.domain.tld ns8.domain.tld domain2.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z ERR Error renewing certificate from LE: {webmail.domain.tld [ad.domain.tld cloud.domain.tld suitecrm.domain.tld intranet.domain.tld domain.tld collabora.domain.tld mail.domain.tld ns8.domain.tld domain2.tld]} error="acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: too many failed authorizations (5) for \"intranet.domain.tld\" in the last 1h0m0s, retry after 2025-06-26 23:29:04 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[ad.domain.tld cloud.domain.tld domain.tld intranet.domain.tld collabora.domain.tld suitecrm.domain.tld domain2.tld www.domain.tld mail.domain.tld ns8.domain.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z ERR Error renewing certificate from LE: {webmail.domain.tld [ad.domain.tld cloud.domain.tld domain.tld intranet.domain.tld collabora.domain.tld suitecrm.domain.tld domain2.tld www.domain.tld mail.domain.tld ns8.domain.tld]} error="acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: too many failed authorizations (5) for \"intranet.domain.tld\" in the last 1h0m0s, retry after 2025-06-26 23:29:02 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[domain.tld www.domain2.tld ad.domain.tld collabora.domain.tld suitecrm.domain.tld www.domain.tld cloud.domain.tld domain2.tld ns8.domain.tld intranet.domain.tld mail.domain.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z ERR Error renewing certificate from LE: {webmail.domain.tld [domain.tld www.domain2.tld ad.domain.tld collabora.domain.tld suitecrm.domain.tld www.domain.tld cloud.domain.tld domain2.tld ns8.domain.tld intranet.domain.tld mail.domain.tld]} error="acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: too many failed authorizations (5) for \"intranet.domain.tld\" in the last 1h0m0s, retry after 2025-06-26 23:28:53 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme

It looks like its looking for intranet.domain.tld but since I removed it from my DNS, it isn’t working. So adding these back to both local and external DNS.

A little while later … after reading some other posts

runagent -m traefik1
cd acme
<acme.json jq --arg domain "intranet.domain.tld" '.acmeServer.Certificates |= map(select(.domain.main != $domain and ((.domain.sans//[])|contains([$domain])|not)))' >acme.json.acmejson-notify
cat acme.json.acmejson-notify >acme.json
systemctl --user restart traefik

This removed the intranet.domain.tld temporarily from the console GUI. The GUI shows intranet.domain.tld as not obtained. Then I tried to do the delete from the GUI but it still doesn’t work. And when it refreshes, the intranet.domain.tld is back … aaaaaahhhh HELP!

Too tired … time for bed.

[Nuke]

Grrr. Added the “sync_timeout”:“60” but no joy

# api-cli run module/traefik1/delete-certificate --data '{"fqdn":"intranet.domain.tld","type":"internal","sync_timeout":60}'
<3>Timeout after about 60 seconds. Certificate not obtained for [....].<3>

Sleeeeep time.

[mrmarkuz (Markus Neuberger)]

Does it help to remove the wrong domain from configs/_default_cert.yml ?

[Nuke]

Hi @mrmarkuz
Thank you for the suggestion.
I removed the wrong domain from configs/_default_cert.yml

# runagent -m traefik1
$ cd configs/
$ vim _default_cert.yml (deleted the wrong domain)
$ cd ..
$ cd acme/
$ <acme.json jq --arg domain "intranet.domain.tld" '.acmeServer.Certificates |= map(select(.domain.main != $domain and ((.domain.sans//[])|contains([$domain])|not)))' >acme.json.acmejson-notify
$ cat acme.json.acmejson-notify >acme.json
$ systemctl --user restart traefik
$ exit

And now there is no wrong domain in the console GUI. Woo hoo!
Now to check if the certificates are updating. I’m not sure about how to do that but I’ll check the systemctl journal and see if there are still acme errors.

[Nuke]

No acme errors in the journalctl anymore.

I’m crossing my fingers that this has fixed the issue. I think the certificates are updating but I’m not sure. Next, I need to figure out how to check the certificates and the dates of the certificates.

Thanks again for your help, @mrmarkuz

[mrmarkuz (Markus Neuberger)]

You could use your browser or curl to check certs and dates:

SSL certificate error ERR_CERT_DATE_INVALID

It can be an expired certificate? You can inspect the certificate expiration dates with your browser, or a command like curl -v https://mattermost.example.org 2>&1 >/dev/null | grep '^\* ' As I just wrote the update for another Bug, I ask you: do you have Webtop installed on the same node? See Latest issue with LetsEncrypt.

[Nuke]

Thank you, I’ll try that.

Is there documentation on the correct structure for certificates with Traefik? From the earlier post, I believe that main should be domain.tld and not webmail.domain.tld?
Copied from above …

# runagent -m traefik1 cat configs/_default_cert.yml
tls:
  stores:
    default:
      defaultGeneratedCert:
        resolver: acmeServer
        domain:
          main: webmail.domain.tld
          sans:
          - domain.tld
          - www.domain2.tld
          - ad.domain.tld
          - collabora.domain.tld
          - suitecrm.domain.tld
          - www.domain.tld
          - cloud.domain.tld
          - domain2.tld
          - ns8.domain.tld
          - mail.domain.tld

[mrmarkuz (Markus Neuberger)]

No, that’s ok.
The cert that was issued first (webmail in your case) is main.
The others are sans.

[Nuke]

OK, thank you for the clarification.

Again, @mrmarkuz thanks for your help.