NethServer Version: NS8 on Rocky Linux 9.4
Module: TLS Certificates
Repositories: default and nethforge, no testing enabled
The second problem I ran into has to do with the Let’s Encrypt certificates. They all stay in the Status “Not obtained”. The certificates in NS7 are removed. I can’t find any pointers on the reason of this. Maybe they should be in the loki logs, but they are unfortunately not working. Where can I look for clues?
IIRC please check / disable IPV6 both DNS provider and locally.
HTH
I am trying to debug the LE problem. Could it be a problem that I named my NS8 servers within the internal domain (servername.int.pa3hfj.nl)?
In this example I try to obtain a certificate for pb6bb.pi4zwn.nl on the leader node server2.int.pa3hfj.nl. For both domainnames there is an entry in the public dns server of the domain name provider, pointing to my public internet address, and port 80 is forwarded to the leader node. I changed the ACME URL to staging.
Below are the messages with journalctl on the server, with grep on “acme” and “certificate”, but I gather these are the most important ones:
Jun 28 14:14:43 server2 traefik[2983]: time=“2024-06-28T12:14:43Z” level=error msg=“HTTP challenge is not enabled” entryPointName=http routerName=acme-http@internal
Jun 28 14:14:43 server2 traefik[2983]: time=“2024-06-28T12:14:43Z” level=error msg=“the router openwebrx.pa3hfj.nl-https@file uses a non-existent resolver: acmeServer”
Jun 28 14:14:43 server2 traefik[2983]: time=“2024-06-28T12:14:43Z” level=error msg=“the router wordpress1-https@file uses a non-existent resolver: acmeServer”
Jun 28 14:14:43 server2 traefik[2983]: time=“2024-06-28T12:14:43Z” level=error msg=“the router log.pa3hfj.nl-https@file uses a non-existent resolver: acmeServer”
Jun 28 14:14:43 server2 traefik[2983]: time=“2024-06-28T12:14:43Z” level=error msg=“the router nvr.pa3hfj.nl-https@file uses a non-existent resolver: acmeServer”
Jun 28 14:14:43 server2 traefik[2983]: time=“2024-06-28T12:14:43Z” level=error msg=“the router nextcloud1-https@file uses a non-existent resolver: acmeServer”
Jun 28 14:14:43 server2 traefik[2983]: time=“2024-06-28T12:14:43Z” level=error msg=“the router certificate-pb6bb.pi4zwn.nl@file uses a non-existent resolver: acmeServer”
Jun 28 14:14:43 server2 traefik[2983]: time=“2024-06-28T12:14:43Z” level=error msg=“the router wordpress2-https@file uses a non-existent resolver: acmeServer”
Jun 28 14:14:43 server2 traefik[2983]: time=“2024-06-28T12:14:43Z” level=error msg=“the router certificate-server2.int.pa3hfj.nl@file uses a non-existent resolver: acmeServer”
Jun 28 14:14:43 server2 traefik[2983]: time=“2024-06-28T12:14:43Z” level=error msg=“the router certificate-mail.pa3hfj.nl@file uses a non-existent resolver: acmeServer”
Jun 28 14:14:43 server2 traefik[2983]: time=“2024-06-28T12:14:43Z” level=error msg=“the router ha.pa3hfj.nl-https@file uses a non-existent resolver: acmeServer”
Jun 28 14:14:41 server2 agent@traefik1[2694]: task/module/traefik1/11c9a0f8-ac44-4c25-835c-dc0c1ff089cd: set-certificate/20writeconfig is starting
Jun 28 14:14:41 server2 agent@traefik1[2694]: task/module/traefik1/31b54ee3-d38e-47e5-a5f5-f94bd4d49069: list-certificates/20readconfig is starting
Jun 28 14:14:41 server2 agent@traefik1[2694]: task/module/traefik1/11c9a0f8-ac44-4c25-835c-dc0c1ff089cd: set-certificate/21waitsync is starting
Jun 28 14:14:41 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:41 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-mail.pa3hfj.nl@file HTTP/1.1" 200 346 "-" "-" 8423 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:41 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:41 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-server2.int.pa3hfj.nl@file HTTP/1.1" 200 367 "-" "-" 8424 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:41 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:41 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 404 65 "-" "-" 8432 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:41 server2 agent@traefik1[2694]: task/module/traefik1/31b54ee3-d38e-47e5-a5f5-f94bd4d49069: action "list-certificates" status is "completed" (0) at step validate-output.json
Jun 28 14:14:42 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:42 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 404 65 "-" "-" 8437 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:43 server2 traefik[2983]: time="2024-06-28T12:14:43Z" level=error msg="HTTP challenge is not enabled" entryPointName=http routerName=acme-http@internal
Jun 28 14:14:43 server2 traefik[2983]: time="2024-06-28T12:14:43Z" level=error msg="the router openwebrx.pa3hfj.nl-https@file uses a non-existent resolver: acmeServer"
Jun 28 14:14:43 server2 traefik[2983]: time="2024-06-28T12:14:43Z" level=error msg="the router wordpress1-https@file uses a non-existent resolver: acmeServer"
Jun 28 14:14:43 server2 traefik[2983]: time="2024-06-28T12:14:43Z" level=error msg="the router log.pa3hfj.nl-https@file uses a non-existent resolver: acmeServer"
Jun 28 14:14:43 server2 traefik[2983]: time="2024-06-28T12:14:43Z" level=error msg="the router nvr.pa3hfj.nl-https@file uses a non-existent resolver: acmeServer"
Jun 28 14:14:43 server2 traefik[2983]: time="2024-06-28T12:14:43Z" level=error msg="the router nextcloud1-https@file uses a non-existent resolver: acmeServer"
Jun 28 14:14:43 server2 traefik[2983]: time="2024-06-28T12:14:43Z" level=error msg="the router certificate-pb6bb.pi4zwn.nl@file uses a non-existent resolver: acmeServer"
Jun 28 14:14:43 server2 traefik[2983]: time="2024-06-28T12:14:43Z" level=error msg="the router wordpress2-https@file uses a non-existent resolver: acmeServer"
Jun 28 14:14:43 server2 traefik[2983]: time="2024-06-28T12:14:43Z" level=error msg="the router certificate-server2.int.pa3hfj.nl@file uses a non-existent resolver: acmeServer"
Jun 28 14:14:43 server2 traefik[2983]: time="2024-06-28T12:14:43Z" level=error msg="the router certificate-mail.pa3hfj.nl@file uses a non-existent resolver: acmeServer"
Jun 28 14:14:43 server2 traefik[2983]: time="2024-06-28T12:14:43Z" level=error msg="the router ha.pa3hfj.nl-https@file uses a non-existent resolver: acmeServer"
Jun 28 14:14:43 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:43 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8438 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:43 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:43 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8439 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:43 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:43 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8440 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:43 server2 crowdsec2[156829]: time="2024-06-28T12:14:43Z" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-06-28T12:14:43Z\" level=error msg=\"HTTP challenge is not enabled\" entryPointName=http routerName=acme-http@internal"
Jun 28 14:14:43 server2 crowdsec2[156829]: time="2024-06-28T12:14:43Z" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-06-28T12:14:43Z\" level=error msg=\"the router openwebrx.pa3hfj.nl-https@file uses a non-existent resolver: acmeServer\""
Jun 28 14:14:43 server2 crowdsec2[156829]: time="2024-06-28T12:14:43Z" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-06-28T12:14:43Z\" level=error msg=\"the router wordpress1-https@file uses a non-existent resolver: acmeServer\""
Jun 28 14:14:43 server2 crowdsec2[156829]: time="2024-06-28T12:14:43Z" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-06-28T12:14:43Z\" level=error msg=\"the router log.pa3hfj.nl-https@file uses a non-existent resolver: acmeServer\""
Jun 28 14:14:43 server2 crowdsec2[156829]: time="2024-06-28T12:14:43Z" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-06-28T12:14:43Z\" level=error msg=\"the router nvr.pa3hfj.nl-https@file uses a non-existent resolver: acmeServer\""
Jun 28 14:14:43 server2 crowdsec2[156829]: time="2024-06-28T12:14:43Z" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-06-28T12:14:43Z\" level=error msg=\"the router nextcloud1-https@file uses a non-existent resolver: acmeServer\""
Jun 28 14:14:43 server2 crowdsec2[156829]: time="2024-06-28T12:14:43Z" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-06-28T12:14:43Z\" level=error msg=\"the router certificate-pb6bb.pi4zwn.nl@file uses a non-existent resolver: acmeServer\""
Jun 28 14:14:43 server2 crowdsec2[156829]: time="2024-06-28T12:14:43Z" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-06-28T12:14:43Z\" level=error msg=\"the router wordpress2-https@file uses a non-existent resolver: acmeServer\""
Jun 28 14:14:43 server2 crowdsec2[156829]: time="2024-06-28T12:14:43Z" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-06-28T12:14:43Z\" level=error msg=\"the router certificate-server2.int.pa3hfj.nl@file uses a non-existent resolver: acmeServer\""
Jun 28 14:14:43 server2 crowdsec2[156829]: time="2024-06-28T12:14:43Z" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-06-28T12:14:43Z\" level=error msg=\"the router certificate-mail.pa3hfj.nl@file uses a non-existent resolver: acmeServer\""
Jun 28 14:14:43 server2 crowdsec2[156829]: time="2024-06-28T12:14:43Z" level=error msg="UnmarshalJSON : invalid character 'i' in literal true (expecting 'r')" line="time=\"2024-06-28T12:14:43Z\" level=error msg=\"the router ha.pa3hfj.nl-https@file uses a non-existent resolver: acmeServer\""
Jun 28 14:14:44 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:44 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8442 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:45 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:45 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8444 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:46 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:46 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8447 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:47 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:47 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8449 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:48 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:48 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8451 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:49 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:49 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8453 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:50 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:50 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8458 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:51 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:51 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8462 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:52 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:52 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8464 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:53 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:53 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8466 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:54 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:54 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8468 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:55 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:55 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8470 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:56 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:56 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8473 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:57 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:57 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8475 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:58 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:58 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8478 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:14:59 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:14:59 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8480 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:15:00 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:15:00 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8482 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:15:01 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:15:01 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8485 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:15:02 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:15:02 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8487 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:15:03 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:15:03 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8489 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:15:04 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:15:04 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8491 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:15:05 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:15:05 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8493 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:15:06 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:15:06 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8496 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:15:07 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:15:07 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8498 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:15:08 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:15:08 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8500 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:15:09 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:15:09 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8502 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:15:10 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:15:10 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8505 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:15:11 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:15:11 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8508 "ApisEndpointHttp@file" "-" 0ms
<truncated>
Jun 28 14:16:25 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:25 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8685 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:26 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:26 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8687 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:27 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:27 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8690 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:28 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:28 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8693 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:29 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:29 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8695 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:30 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:30 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8697 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:31 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:31 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8699 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:32 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:32 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8702 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:33 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:33 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8704 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:34 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:34 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8706 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:35 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:35 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8708 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:36 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:36 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8712 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:37 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:37 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8736 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:37 server2 agent@traefik1[2694]: task/module/traefik1/dc6be428-fbe5-46ab-aa20-e6bc93604941: list-certificates/20readconfig is starting
Jun 28 14:16:37 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:37 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-mail.pa3hfj.nl@file HTTP/1.1" 200 346 "-" "-" 8756 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:37 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:37 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8757 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:37 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:37 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-server2.int.pa3hfj.nl@file HTTP/1.1" 200 367 "-" "-" 8758 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:37 server2 agent@traefik1[2694]: task/module/traefik1/dc6be428-fbe5-46ab-aa20-e6bc93604941: action "list-certificates" status is "completed" (0) at step validate-output.json
Jun 28 14:16:38 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:38 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8769 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:39 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:39 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8771 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:40 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:40 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8773 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:41 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:41 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8775 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:42 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:42 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8778 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:43 server2 traefik[2983]: 127.0.0.1 - - [28/Jun/2024:12:16:43 +0000] "GET /0148c706-7109-42e2-9c37-f383560a29de/api/http/routers/certificate-pb6bb.pi4zwn.nl@file HTTP/1.1" 200 349 "-" "-" 8780 "ApisEndpointHttp@file" "-" 0ms
Jun 28 14:16:44 server2 agent@traefik1[2694]: task/module/traefik1/11c9a0f8-ac44-4c25-835c-dc0c1ff089cd: action "set-certificate" status is "aborted" (2) at step 21waitsync
@davidep: Would it be an option to reinitiate / reinstall Traefik (as we earlier did with Loki for solving another problem) to get this going? I really need to get the LE certificates working for some websites, nextcloud and such…
I never saw this error, but can be a symptom of corrupted acme.json contents, or bad file permissions.
To check permissions:
runagent -m traefik1 podman exec traefik ls -Rl /etc/traefik/acme
Should be
/etc/traefik/acme:
total 0
-rw------- 1 root root 0 Jul 1 09:27 acme.json
If they are correct, and you have few LE host names (so the request limit is far from being hit) let’s try to erase acme.json:
runagent -m traefik1 podman exec traefik sh -c 'umask 077; echo {} >/etc/traefik/acme/acme.json'
runagent -m traefik1 systemctl --user restart traefik
Hello Davide, once again thank you for helping me!
The first command
runagent -m traefik1 podman exec traefik ls -Rl /etc/traefik/acme
exits with:
ls: can't open '/etc/traefik/acme': Permission denied
So probably a permission problem.
This is strange. Just to be sure, did you change directory or file ownership from the command line?
This command can fix the access error if it’s an ownership issue:
chown -v -R traefik1:traefik1 ~traefik1/.local/share/containers/storage/volumes/traefik-acme/_data
[root@server2 ~]# chown -v -R traefik1:traefik1 ~traefik1/.local/share/containers/storage/volumes/traefik-acme/_data
ownership of '/home/traefik1/.local/share/containers/storage/volumes/traefik-acme/_data/acme.json' retained as traefik1:traefik1
ownership of '/home/traefik1/.local/share/containers/storage/volumes/traefik-acme/_data' retained as traefik1:traefik1
Still a Permission denied on /etc/traefik/acme:
[root@server2 ~]# runagent -m traefik1 podman exec traefik ls -Rl /etc/traefik
/etc/traefik:
total 16
drwxr-xr-x 2 root root 23 Jun 30 19:20 acme
drwxr-xr-x 2 root root 4096 Jul 1 10:51 configs
drwxr-xr-x 2 root root 6 Jun 19 20:00 custom_certificates
-rw-r--r-- 1 root root 1960 Jun 19 20:00 selfsigned.crt
-rw------- 1 root root 3268 Jun 19 20:00 selfsigned.key
-rw-r--r-- 1 root root 543 Jul 1 12:49 traefik.yaml
/etc/traefik/acme:
ls: can't open '/etc/traefik/acme': Permission denied
total 0
/etc/traefik/configs:
total 64
-rw-r--r-- 1 root root 484 Jun 19 20:05 _api.yml
-rw-r--r-- 1 root root 839 Jun 19 20:00 _api_server.yml
-rw-r--r-- 1 root root 145 Jun 19 20:00 _default_cert.yml
-rw-r--r-- 1 root root 120 Jun 19 20:00 _http2https.yml
-rw-r--r-- 1 root root 536 Jul 1 10:51 collabora1.yml
-rw-r--r-- 1 root root 613 Jun 30 18:24 ha.pa3hfj.nl.yml
-rw-r--r-- 1 root root 539 Jul 1 08:14 log.pa3hfj.nl.yml
-rw-r--r-- 1 root root 511 Jun 23 17:28 log.pi4zwn.nl.yml
-rw-r--r-- 1 root root 1084 Jun 23 19:53 mail7-rspamd.yml
-rw-r--r-- 1 root root 512 Jun 30 18:22 nextcloud1.yml
-rw-r--r-- 1 root root 619 Jun 30 18:26 nvr.pa3hfj.nl.yml
-rw-r--r-- 1 root root 560 Jun 30 18:26 openwebrx.pa3hfj.nl.yml
-rw-r--r-- 1 root root 526 Jun 30 18:22 roundcubemail7.yml
-rw-r--r-- 1 root root 762 Jun 23 10:21 samba1-amld.yml
-rw-r--r-- 1 root root 474 Jun 30 20:10 wordpress2.yml
-rw-r--r-- 1 root root 536 Jun 30 21:15 wordpress4.yml
/etc/traefik/custom_certificates:
total 0
It can be a selinux problem
aureport -a | tail
Did you mount some disk on /home or anything else?
Yes, home is mounted on a separate disk! (in proxmox). I did not realize that could be a problem 
.
[root@server2 ~]# aureport -a | tail
87. 06/30/24 20:26:55 dnsmasq system_u:system_r:container_t:s0:c837,c890 93 fifo_file setattr system_u:system_r:container_runtime_t:s0 denied 442
88. 06/30/24 20:33:58 dnsmasq system_u:system_r:container_t:s0:c410,c850 93 fifo_file setattr system_u:system_r:container_runtime_t:s0 denied 449
89. 07/01/24 04:00:01 qemu-ga system_u:system_r:virt_qemu_ga_t:s0 257 dir search system_u:object_r:container_var_lib_t:s0 denied 689
90. 07/01/24 04:00:01 qemu-ga system_u:system_r:virt_qemu_ga_t:s0 257 dir search system_u:object_r:container_var_lib_t:s0 denied 690
91. 07/01/24 04:00:02 qemu-ga system_u:system_r:virt_qemu_ga_t:s0 257 dir search system_u:object_r:container_var_lib_t:s0 denied 693
92. 07/01/24 13:24:09 ls system_u:system_r:container_t:s0:c492,c724 2 dir read unconfined_u:object_r:data_home_t:s0 denied 848
93. 07/01/24 14:49:47 traefik system_u:system_r:container_t:s0:c19,c409 257 file read unconfined_u:object_r:data_home_t:s0 denied 934
94. 07/01/24 14:52:45 traefik system_u:system_r:container_t:s0:c973,c1016 257 file read unconfined_u:object_r:data_home_t:s0 denied 942
95. 07/01/24 14:54:44 ls system_u:system_r:container_t:s0:c973,c1016 2 dir read unconfined_u:object_r:data_home_t:s0 denied 951
96. 07/01/24 14:55:50 ls system_u:system_r:container_t:s0:c973,c1016 2 dir read unconfined_u:object_r:data_home_t:s0 denied 968
It is not a problem until you forget to copy SELinux labels 
Try this procedure:
touch /.autorelabel
reboot
If you want to avoid reboot:
restorecon -n -r -v /home
…And repeat again without -n (dry run flag).
touch /.autorelabel
reboot
This did it! I would never have found this out by myself. As it turns out I have yet a lot to learn on NS8. So a big thank you is (once again) in order for @davidep!
I have one (hopefully for now) last question: while I had LE staging activated some certificates are not valid for official use. Deleting the certificate in the respective module settings and later on activating them again doesn’t trigger the creation of a new non-staging certificate. How can I trigger it manually?