All certificates are expired or expiring - nothing renewing

Support
, , ,
4

About 2 months ago I posted about not being able to delete a certificate for a domain that was created in error. I could never get that certificate to delete. So now I see from the error messages that this domain is causing the “error renewing certificates”.

Since that intranet.domain.tld is referenced through the other domains nothing updates.

Well this sucks.

# runagent -m traefik1 cat configs/_default_cert.yml
tls:
  stores:
    default:
      defaultGeneratedCert:
        resolver: acmeServer
        domain:
          main: webmail.domain.tld
          sans:
          - domain.tld
          - www.domain2.tld
          - ad.domain.tld
          - collabora.domain.tld
          - suitecrm.domain.tld
          - www.domain.tld
          - cloud.domain.tld
          - domain2.tld
          - ns8.domain.tld
          - **intranet.domain.tld**
          - mail.domain.tld

This looks wrong after looking at the traefik tls documentation. main should be the top level domain and sans should be the subdomains. The little f**ker intranet.domain.tld is still there.

# journalctl --grep acmeCA
Jun 26 19:16:47 ns8 traefik[948310]: 2025-06-26T23:16:47Z INF Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:16:49 ns8 traefik[948310]: 2025-06-26T23:16:49Z INF Renewing certificate from LE : {Main:intranet.domain.tld SANs:[]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:16:56 ns8 traefik[948310]: 2025-06-26T23:16:56Z ERR Error renewing certificate from LE: {intranet.domain.tld []} error="error: one or more domains had a problem:\n[intranet.domain.tld] invalid authorization: acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for intranet.domain.tld - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for intranet.domain.tld - check that a DNS record exists for this domain\n" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:16:56 ns8 traefik[948310]: 2025-06-26T23:16:56Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[intranet.domain.tld ns8.domain.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:18:53 ns8 traefik[948310]: 2025-06-26T23:18:53Z ERR Error renewing certificate from LE: {webmail.domain.tld [intranet.domain.tld ns8.domain.tld]} error="error: one or more domains had a problem:\n[intranet.domain.tld] invalid authorization: acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for intranet.domain.tld - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for intranet.domain.tld - check that a DNS record exists for this domain\n" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:18:53 ns8 traefik[948310]: 2025-06-26T23:18:53Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[cloud.domain.tld intranet.domain.tld ns8.domain.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:19:42 ns8 traefik[948310]: 2025-06-26T23:19:42Z ERR Error renewing certificate from LE: {webmail.domain.tld [cloud.domain.tld intranet.domain.tld ns8.domain.tld]} error="error: one or more domains had a problem:\n[intranet.domain.tld] invalid authorization: acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for intranet.domain.tld - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for intranet.domain.tld - check that a DNS record exists for this domain\n" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:19:42 ns8 traefik[948310]: 2025-06-26T23:19:42Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[cloud.domain.tld intranet.domain.tld ns8.domain.tld suitecrm.domain.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:38 ns8 traefik[948310]: 2025-06-26T23:20:38Z ERR Error renewing certificate from LE: {webmail.domain.tld [cloud.domain.tld intranet.domain.tld ns8.domain.tld suitecrm.domain.tld]} error="error: one or more domains had a problem:\n[intranet.domain.tld] invalid authorization: acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for intranet.domain.tld - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for intranet.domain.tld - check that a DNS record exists for this domain\n" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:38 ns8 traefik[948310]: 2025-06-26T23:20:38Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[intranet.domain.tld ad.domain.tld suitecrm.domain.tld ns8.domain.tld cloud.domain.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z ERR Error renewing certificate from LE: {webmail.domain.tld [intranet.domain.tld ad.domain.tld suitecrm.domain.tld ns8.domain.tld cloud.domain.tld]} error="error: one or more domains had a problem:\n[intranet.domain.tld] invalid authorization: acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for intranet.domain.tld - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for intranet.domain.tld - check that a DNS record exists for this domain\n" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[ad.domain.tld suitecrm.domain.tld ns8.domain.tld intranet.domain.tld collabora.domain.tld cloud.domain.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z ERR Error renewing certificate from LE: {webmail.domain.tld [ad.domain.tld suitecrm.domain.tld ns8.domain.tld intranet.domain.tld collabora.domain.tld cloud.domain.tld]} error="acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: too many failed authorizations (5) for \"intranet.domain.tld\" in the last 1h0m0s, retry after 2025-06-26 23:28:52 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[intranet.domain.tld ad.domain.tld ns8.domain.tld mail.domain.tld cloud.domain.tld collabora.domain.tld suitecrm.domain.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z ERR Error renewing certificate from LE: {webmail.domain.tld [intranet.domain.tld ad.domain.tld ns8.domain.tld mail.domain.tld cloud.domain.tld collabora.domain.tld suitecrm.domain.tld]} error="acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: too many failed authorizations (5) for \"intranet.domain.tld\" in the last 1h0m0s, retry after 2025-06-26 23:28:53 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[ad.domain.tld collabora.domain.tld cloud.domain.tld domain.tld intranet.domain.tld mail.domain.tld suitecrm.domain.tld ns8.domain.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z ERR Error renewing certificate from LE: {webmail.domain.tld [ad.domain.tld collabora.domain.tld cloud.domain.tld domain.tld intranet.domain.tld mail.domain.tld suitecrm.domain.tld ns8.domain.tld]} error="acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: too many failed authorizations (5) for \"intranet.domain.tld\" in the last 1h0m0s, retry after 2025-06-26 23:28:51 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[ad.domain.tld cloud.domain.tld suitecrm.domain.tld intranet.domain.tld domain.tld collabora.domain.tld mail.domain.tld ns8.domain.tld domain2.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z ERR Error renewing certificate from LE: {webmail.domain.tld [ad.domain.tld cloud.domain.tld suitecrm.domain.tld intranet.domain.tld domain.tld collabora.domain.tld mail.domain.tld ns8.domain.tld domain2.tld]} error="acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: too many failed authorizations (5) for \"intranet.domain.tld\" in the last 1h0m0s, retry after 2025-06-26 23:29:04 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[ad.domain.tld cloud.domain.tld domain.tld intranet.domain.tld collabora.domain.tld suitecrm.domain.tld domain2.tld www.domain.tld mail.domain.tld ns8.domain.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z ERR Error renewing certificate from LE: {webmail.domain.tld [ad.domain.tld cloud.domain.tld domain.tld intranet.domain.tld collabora.domain.tld suitecrm.domain.tld domain2.tld www.domain.tld mail.domain.tld ns8.domain.tld]} error="acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: too many failed authorizations (5) for \"intranet.domain.tld\" in the last 1h0m0s, retry after 2025-06-26 23:29:02 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z INF Renewing certificate from LE : {Main:webmail.domain.tld SANs:[domain.tld www.domain2.tld ad.domain.tld collabora.domain.tld suitecrm.domain.tld www.domain.tld cloud.domain.tld domain2.tld ns8.domain.tld intranet.domain.tld mail.domain.tld]} acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme
Jun 26 19:20:53 ns8 traefik[948310]: 2025-06-26T23:20:53Z ERR Error renewing certificate from LE: {webmail.domain.tld [domain.tld www.domain2.tld ad.domain.tld collabora.domain.tld suitecrm.domain.tld www.domain.tld cloud.domain.tld domain2.tld ns8.domain.tld intranet.domain.tld mail.domain.tld]} error="acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: too many failed authorizations (5) for \"intranet.domain.tld\" in the last 1h0m0s, retry after 2025-06-26 23:28:53 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=acmeServer.acme

It looks like its looking for intranet.domain.tld but since I removed it from my DNS, it isn’t working. So adding these back to both local and external DNS.

A little while later … after reading some other posts

runagent -m traefik1
cd acme
<acme.json jq --arg domain "intranet.domain.tld" '.acmeServer.Certificates |= map(select(.domain.main != $domain and ((.domain.sans//[])|contains([$domain])|not)))' >acme.json.acmejson-notify
cat acme.json.acmejson-notify >acme.json
systemctl --user restart traefik

This removed the intranet.domain.tld temporarily from the console GUI. The GUI shows intranet.domain.tld as not obtained. Then I tried to do the delete from the GUI but it still doesn’t work. And when it refreshes, the intranet.domain.tld is back … aaaaaahhhh :unamused: HELP!

Too tired … time for bed.