Nethserver Letsencrypt - Timeout during connect

Support
1

Hi everyone,

Following my installation of Dokuwiki on Nethserver and using the AD as its user base, I decided to extend that to other servers including OPNSense. After asking a few questions this would require me to purchase a domain name and have Letsencrypt produce a valid certificate that I would need to use on Nethserver and then copy to that same certificate to the AD. Lets call my domain name “MyDomain.net”. So the idea is to produce a certificate for that domain name. All my network is set with that domain name currently but using the original certificates which are not valid certs.

I was recommended to use Cloudflare as my DNS manager for my domain name. I purchased a name from Google and transfered the management to Cloudflare. My ISP does not attribute me a fixed IP so I would have to use Duckdns to update my IP address when it changes. That would prevent me from using any kind of A record in Cloudflare which requires an IP. So I thought no problem, I would link my domain name with a CNAME something like this, CNAME : mydomain.net -> mydomain.duckdns.org. I didn’t use the proxy as this seemed to cause me some issues and it is in DNS only.

I had to wait a while for all this to propagate and be seen my DNS servers. Eventually I used NSlookup and noticed something interesting : nslookup -query=a mydomain.net 8.8.8.8 it would return the IP address assigned to mydomain.duckdns.org although I didn’t set any A record. When I tried to query the cname, it returned this :

image

So I guess something is wrong as it appears to point to 8.8.8.8

I decided to try to get a certificate anyways to see what would happen through Nethserver but it failed with the mydomain.net.

I tried to access my network (which has OPNSense in front of it) with https://mydomain.net, which didn’t work as I was returned a page stating : “A potential DNS Rebind attack has been detected”. So I wonder if that could be the source of my problems but not entirely sure.

I looked at the letsencrypt logs and saw this :

Domain: mydomain.net
Type:   connection
Detail: Fetching http://mydomain.net/.well-known/acme-challenge/nPDkTl7rDJXeVIEFsuUua35e_g35Y3jfTaucstaIocA: Timeout during connect (likely firewall problem)

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. Additionally, please check that your computer has a publicly routable IP address and that no firewalls are preventing the server from communicating with the client. If you're using the webroot plugin, you should also verify that you are serving files from the webroot path you provided.

As additional information there is a port forwarding rule in place that allows the passage of traffic for ports 80 / 443. I thought I would mention that since the message does state it could be a firewall issue.

I’m hoping some of you could help me understand what I am not seeing. I am not a DNS expert so I am not sure if my problem could be because I do not have an A record and frankly don’t know how I would make this work since I would have to manually update my A record every time my ISP attributes a new one, hence the reason why I used a CNAME. Also, seeing that in the logs under DETAIL it tries to fetch to mydomain.net which OPNSense detected as a dns rebind attack, possibly this is the problem but I am not sure.

If anyone has any ideas or suggestions, anything that could help really, I would appreciate it.

Thanks in advance.

1 Like
2

Update : I did fix the rebind attack issue by specifying an alternate host name in OPNSense which fixed that. I can now access OPNSense with my domain name but that still doesn’t fix the issue. The message is still the same.

Update 2 : Paying a closer look at the URL, I’m guessing I have to route all data coming from 80 / 443 to Nethserver when mydomain.net is hit…

Update 3 : I did route all the traffic coming from mydomain.net to nethserver and checked I could see the acme-challenge folder and I do. I don’t have access to what is inside of course that is most likely protected. I’m not convinced the problem is with the firewall anymore. Just to be clear, if I hit : http://mydomain.net/.well-known/ I see the acme-challenge folder but I can’t go inside as it is protected. All that to say, the outside can see my nethserver. I dunno what the issue is.

Anyways if any of you have any ideas or had to deal with similar issues in the past, let me know.

Thanks

3

I noticed this in the log…

2021-01-09 11:57:56,668:DEBUG:certbot._internal.plugins.webroot:Attempting to save validation to /var/www/html/.well-known/acme-challenge/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

But the folder does not exist. Is there something I should add to the web server?

I checked the web server logs and found this :

[Sat Jan 09 11:58:50.284378 2021] [autoindex:error] [pid 12029] [client 192.168.20.40:53155] AH01276: Cannot serve directory /var/www/html/.well-known/acme-challenge/: No matching DirectoryIndex (index.html,index.php) found, and server-generated directory index forbidden by Options directive

So I guess I am missing something. But I would imagine using Letsencrypt from the UI would actually do what it needs for me? Or is there a config step I missed somewhere?

4

Update : So the .well-know directory is there and the “key” is created. So I guess that works so the problem is not there…

Trying to access http://mydomain.net/.well-known/acme-challenge returns a 403. The logs says there is a timeout returned trying to access the key, so http://dart8084.net/.well-known/acme-challenge/xxxxxxx. Given the 403 I suppose that is why but not sure how to fix this.

I’m out of ideas… Anyone have a clue?

5

@support_team anyone?

6

It’s important to know the exact configuration here. @tessierp, is your Neth box directly on the net, or is it behind your OPNsense box? If the latter, how is the OPNsense box configured? Are you forwarding ports 80 and 443 to the Neth box, or are you using HAProxy on the OPNsense box as a reverse proxy?

7

Sorry everyone… I didn’t write an update as I am trying to figure out a few things. Your assumption is correct, Nethserver sits behind OPNSense.

So the problem was I didn’t forward the ports 80 and 443 to Nethserver and I was able to get a certificate.

I intend to write a follow-up soon based on what I am finding out myself along with others who are kind enough to help me with a bunch of questions. Hopefully this helps someone else.

Stay tuned for more!

8

That will definitely do it. Going forward, there are a few options:

  • You can keep doing what you’re now doing–keep 80 and 443 forwarded to the Neth box. Let’s Encrypt will need to reach your server on port 80 every time the cert is renewed.
  • You could terminate TLS on the OPNsense box rather than the Neth box using HAProxy. In that case, OPNsense would obtain and maintain the certs, and would communicate with the Neth box on the backend (unencrypted)
  • You could keep 80 and 443 closed, and instead use DNS validation to get your certs. Neth doesn’t support this by default, but it isn’t that hard to implement:

https://wiki.nethserver.org/doku.php?id=userguide:let_s_encrypt_for_internal_servers
https://wiki.nethserver.org/doku.php?id=userguide:let_s_encrypt_acme-dns

3 Likes
9

I will have to do some research into those options. The first one was easiest but it does limit me in the future where I assign a machine to ports 80 and 443 and there can be only one.

Having OPNSense manage the certs could be an option but not sure how that would work and if it can be automated like Nethserver does. That could mean, I guess, that I would have to push every certificate renewals to the appropriate machines on the network. And yes SSL termination could be a solution to consider as I don’t need encryption internally… Not sure how to make that work though. Must do some research.

Last option to use DNS validation is interesting and not sure exactly what it means… I will have to consult the links you sent me. Thanks for that.

10

No, the idea is that all the encryption/decryption and cert management would happen on the OPNsense box. It has an ACME client (acme.sh, I believe) that can obtain and renew certs for whatever you need. It also runs a reverse proxy (HAProxy) which handles all the TLS stuff, and then communicates with any of your backend servers (Neth and/or anything else) unencrypted. The backend stuff doesn’t need a cert at all. Of course, that assumes that you trust your LAN. But if you want to access more than one thing behind the OPNsense box (e.g., I’m running Bitwarden on one VM and Ombi on another), this is the way to go. The OPNsense docs on this are pretty good.

In short, it lets you validate domain control by creating DNS records rather than present a token file via HTTP. Very handy for internal hosts (I use it on probably a couple of dozen devices on my LAN), as it doesn’t require access from the Internet to that device, but it does require a DNS host that supports programmatic updates.

11

That is all interesting. I understand what you are suggesting and certainly is an interesting option. I am just a bit limited in my knowledge on how to do all that. When you are considering options you also need to have a good understanding of each one. Right now, the first option is clear to me but not ideal since it is limiting.

A lot of good information, thank you! That being said, still need to do a lot of digging. I’m not a network guy, just a Software Dev. :slight_smile:

12

I read on DNS validation. It is interesting, the use of an API to get the certs indirectly. I use Cloudflare since yesterday so I don’t have to host my own acme-dns. This would prevent me exposing ports 40, 443 to nethserver directly. Interesting option.

But I think my best option is to learn how to do all this from OPNSense, handle the renewal of certs there, reverse proxy to my internal servers unencrypted.

13

Heck, I’m neither of these–I’m a lawyer.

1 Like
14

Nice! So networking and servers is a hobby or just wanted to maintain your own stuff at home is that how you got into it?

15

Some of both. I’ve always been a computer geek–I started with a VIC-20, started playing with Linux in the early '90s, registered a domain and set up my own mail server (using what was then called e-smith–ancestor to Nethserver) over 20 years ago. I’ve basically been running a server for as long as I’ve had high-speed Internet, even if my primary Neth server is now a VPS on another continent. And now I have a server rack burning about a kilowatt 24x7–the hobby gets expensive.

16

:laughing: It does get expensive. I’m trying to be careful with that. Right now I have a small size server using Proxmox in a 2U case with a Ryzen 3900X. Not a lot of storage but enough to run a few servers. My next expense probably will be my own NAS system using TrueNAS or UNRAID. And yes I do know UNRAID is probably not like a true NAS system but I like some of its flexibilities. All that being said, I am not decided and I know this will cost me. I do have a QNAP server right now but it is slow and aging.

Like you said, a expensive hobby!

17

I was using FreeNAS for a while on 3 x 2 TB disks in RAIDZ1, and it was good. Then I learned about Plex, Radarr, Sonarr, etc. My current server has 32 disks installed, roughly 150 TB of storage…

Most of my equipment is used enterprise gear, which saves on acquisition cost, but doesn’t tend to be as energy-efficient as something newer. As a result, the electric bill for my workshop (which houses the server rack) is higher than it is for my house.

2 Likes
18

Wow, that is A LOT of storage. You have 6 TB drives in there? One of the reason I’m considering UNRAID is because expanding a FreeNAS (now TrueNAS) is not very flexible because of how the VDEVs work.

Yeah enterprise equipment is probably more efficient now, the most recent gear. Older gear requires more power. Another problem I see is noise.

I prefer to build my own. Sure, most of it is not enterprise grade but… Still works :slight_smile:

19

Everything from 3TB to 12TB, in two pools–one of striped mirrors (4 x 2 x 3 TB), one of 4 x 6-disk RAIDZ2 vdevs with 4, 6, 8, and 12 TB disks.

Not as flexible as UnRAID, but it certainly can be done. Not too bad with a bit of planning, but it does take some thought.

And heat. Which are two big reasons the rack is now in a separate building, 50+’ from the house.

1 Like
20

Just out of curiosity, have you ever tried to get a wildcard cert (*.mydomain.org) from LetsEncrypt using Cloudflare as your DNS? I’m trying to do that right now with OPNSense and it fails all the time. But requesting a cert for a FQDN works OK.