Custom ACME Server - tls error to server

NethServer Version: 8 - prebuilt official image
Module: Traefik

Hello!
Great product first of all! I have got small problem with acme servers. I have added my custom acme server in UI which is based on step ca hosted on Kubernetes. The problem that I am having right now is the tls certificate verification when connecting to acme server. I have checked docs and I could not find anything related to importing custom root ca to Traefik module to be able to call my acme server. I have added my custom root ca to underlying system which is rocky linux 9, but It looks like it has no affection on it (after import I am doing curl to my acme server without any problem on the underlying server).

The error coming from traefik is:

2024-10-06T23:07:05+02:00 [1:traefik1:traefik] time="2024-10-06T21:07:05Z" level=error msg="Unable to obtain ACME certificate for domains \"nethserver.home.lan\"" ACME CA="https://ca.home.lan/acme/homelab-lan-acme/directory" providerName=acmeServer.acme routerName=certificate-nethserver.home.lan@file error="cannot get ACME client get directory at 'https://ca.home.lan/acme/homelab-lan-acme/directory': Get \"https://ca.home.lan/acme/homelab-lan-acme/directory\": tls: failed to verify certificate: x509: certificate signed by unknown authority" rule="Host(`nethserver.home.lan`) && Path(`/0d746074-e85b-4b34-b143-aeaf23e8b581`)"

Hi Michal, welcome to our community!

I use to switch to the Let’s Encrypt Staging server for testing. In your situation, it seems like the custom CA root certificate would need to be installed into Traefik’s container, but I’m unsure of the exact steps and there’s no documentation about how to do it.

I’ll ping @Tbaile and @Amygos, maybe they can provide some guidance here!

Hi!
Thank you for response. As far as I know there might be two solutions:

  • replacing container ca-certificates.crt bundle with the on containing my root ca
  • Using LEGO_CA_CERTIFICATES or LEGO_CA_SYSTEM_CERT_POOL environment variable

Hello Michal!
I am going to be honest, I have no clue on what are the steps to do to implement such a thing on Traefik, but below is a possible solution. A tag team with @davidep and @Amygos might needed :smile:

Seems that a custom directory for drop-in of certificate files is in the module (it’s tecnically used by manual certificate upload, but will try). Additionally it’s needed to edit the environment file in the module.

Accessing the module with runagent -m traefik<numeric instance> gives you access to custom_certificates directory, you can drop your certificates there.
Then, edit the environment file adding LEGO_CA_CERTIFICATES=/etc/traefik/custom_certificates/<certificate>.
Now restart the module using systemctl --user restart traefik

Hope for the best :crossed_fingers: :grimacing:

1 Like

Hello!
Sorry for late response. Unfortunately this did not helped. It looks like the environment files is loaded either to late in container or right straight not used which is unlike. I studied step ca docs and they suggest to use LEGO_CA_CERTIFICATES, but in slightly different manier. I see that there are two things I traefik1 state folder. The agent.env file and manual_flags. Maybe one of which would make sense. In the following code listing I present you what I have added to environment file.

LEGO_CA_CERTIFICATES=/etc/traefik/custom_certificates/root-homelab.crt

Beside the systemctl call I also did the reboot of whole system to be sure.

Thank you very much for your help and interest in this topic.

Hi Michal and thank you for trying!

Ok, so changing environment of the container doesn’t help, so we need to create a environment entry per se.
Try this:

  1. After logging into traefik module, edit the file ../systemd/user/traefik.service
  2. Add the following line after the last --volume entry (here a gist to illustrate the edited file)
  --env=LEGO_CA_CERTIFICATES=/etc/traefik/custom_certificates/root-homelab.crt \
  1. Reload systemd with systemctl --user daemon-reload
  2. Restart traefik service with systemctl --user restart traefik

NOTE: this won’t persists module updates, a manual change in the module might be needed in order to implement such feature.

4 Likes

Thank you for help. I can confirm that this solution worked. Now I can obtain certificates without any problems. It would be great I think for organisations to make it easier. For HomeLab It is totally fine, as It is not ciritcall infrastructure. I think that It would be great to make those modules (like traefik) to use system certificates store.

3 Likes

Happy to help!

This is tricky and error-prone, different OSs might use different locations and might break the module all together.

A viable option is to add a custom env file in the module that persists between updates. For the “system certificate store” section, It’s still possible to symlink the files from the OS specific directory to the Traefik module one, solving the issue of having multiple certificates around.

Maybe @davidep can add a card to implement such feature (for datacenters it might be useful to customize this settings)

Edit:
Card NethServer · GitHub

1 Like