Nextcloud letsencrypt certificate

tmp501 · October 1, 2020, 5:27pm

finally had the nerve to set the one cert as default and all is cool now. so the address is https://next.domain.org/nexcloud. all my devices are set up for an address without the /nextcloud. is there a easy way to redirect that url to https://next.domain.org ? thanks

Andy_Wismer · October 1, 2020, 5:55pm

@tmp501

Hi

It’s safer and more reliable if you erase the old “server” and create a new connection, especially on your mobile devices… (Also on PCs, Macs…)

My 2 cents
Andy

tmp501 · October 1, 2020, 6:09pm

yes thanks. just did that. all good.

tmp501 · January 19, 2021, 5:21pm

hey people,

i got a new cert issue. apparently my two certs did not get renewed and expired today (no email). requesting a new one threw me an error. got frustrated and removed the neth-letsencrypt rpm, the files belonging to the two domains under /etc/letsencrypt - reinstalled neth-letsencrypt - reboot and same error. the dns/domain/ip entries are correct - they worked up untill today. any ideas?
ps.: i upgraded to the latest release after the last reboot.

just saw that the http service is not running…

dns.service_action_error
The following command has failed:
`system-services/update`
Unfortunately we couldn't catch the exact error. If you want to help, please click on the button below to copy the failed command to the clipboard, paste it into the Terminal and submit command output to the developers.

http is up again. requesting a new cert still throws the same error

   Domain: funk.domain.org
Type:   connection
Detail: Fetching http://funk.domain.org/.well-known/acme-challenge/6J8rvDQ4YpLG9IITgDx5d5QTyB_qNDZoJ8hq3UOe9xY: Connection refused

Domain: next.domain.org
Type:   connection
Detail: Fetching http://next.domain.org/.well-known/acme-challenge/4gxyXW5b861TFdH27cRaF0Tq_AkwisO8YDFa7jKYpCU: Connection refused

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. Additionally, please check that your computer has a publicly routable IP address and that no firewalls are preventing the server from communicating with the client. If you're using the webroot plugin, you should also verify that you are serving files from the webroot path you provided.
2021-01-19 18:05:14,100:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
AuthorizationError: Some challenges have failed.

2021-01-19 18:05:14,100:DEBUG:certbot._internal.error_handler:Calling registered functions
2021-01-19 18:05:14,100:INFO:certbot._internal.auth_handler:Cleaning up challenges
2021-01-19 18:05:14,101:DEBUG:certbot._internal.plugins.webroot:Removing /var/www/html/.well-known/acme-challenge/6J8rvDQ4YpLG9IITgDX5d1QTyB_qNDZoJ8hq3UOe9xY
2021-01-19 18:05:14,102:DEBUG:certbot._internal.plugins.webroot:Removing /var/www/html/.well-known/acme-challenge/4gxyXW5b811TFdH27YRaF0Tq_AkwisO8YDFa7jKYpCU
2021-01-19 18:05:14,102:DEBUG:certbot._internal.plugins.webroot:Removing /var/www/html/.well-known/acme-challenge/8BpVN8ygR1IScbDHtc9aI2MWdLP1jyoU7sBHCguiwrE
2021-01-19 18:05:14,104:DEBUG:certbot._internal.plugins.webroot:All challenges cleaned up
2021-01-19 18:05:14,104:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 9, in <module>
    load_entry_point('certbot==1.10.1', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 1412, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 1293, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 134, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/client.py", line 441, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/client.py", line 374, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/client.py", line 421, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
AuthorizationError: Some challenges have failed.
2021-01-19 18:05:14,107:ERROR:certbot._internal.log:Some challenges have failed

danb35 · January 19, 2021, 5:41pm

There’s your problem–your server is refusing connections from the outside world. Fix that, and your cert should issue normally.

tmp501 · January 19, 2021, 5:46pm

hmm… ok now what?

found something… httpd-admin is not running…

httpd-admin.service - Server Manager UI httpd instance
   Loaded: loaded (/usr/lib/systemd/system/httpd-admin.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2021-01-19 18:47:09 CET; 4s ago
     Docs: https://github.com/NethServer/nethserver-httpd-admin
  Process: 24251 ExecStart=/usr/sbin/httpd -f /etc/httpd/admin-conf/httpd.conf -c MaxConnectionsPerChild 12 -DFOREGROUND (code=exited, status=1/FAILURE)
 Main PID: 24251 (code=exited, status=1/FAILURE)

Jan 19 18:47:09 domain.org systemd[1]: Started Server Manager UI httpd instance.
Jan 19 18:47:09 domain.org httpd[24251]: AH00526: Syntax error on line 62 of /etc/httpd/admin-conf/httpd.conf:
Jan 19 18:47:09 tank.domain.org httpd[24251]: SSLCertificateChainFile: file '/etc/letsencrypt/live/next.domain.org/chain.pem' does not exist or is empty
Jan 19 18:47:09 tank.domain.org systemd[1]: httpd-admin.service: main process exited, code=exited, status=1/FAILURE
Jan 19 18:47:09 tank.domain.org systemd[1]: Unit httpd-admin.service entered failed state.
Jan 19 18:47:09 tank.domain.org systemd[1]: httpd-admin.service failed.

this file /etc/httpd/admin-conf/httpd.conf still has entries to my old/deleted certs…

tmp501 · January 19, 2021, 7:03pm

httpd-admin is running again… but i cant figure out why or what is preventing letsencrypt access from the outside world. my nextcloud instance was reachable before the cert expired. but its an explanation why my certs didn’t get renewed…

seems like its only complaining about one of my certs…weird it was working…
/usr/libexec/nethserver/letsencrypt-certs -d next.domain.org,funk.domain.org -v

Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/10202135823:
{
  "protected": "eyJub25jZSI6ICIwMTA0dE5xT19tYnNvSExqLXRWRHVwamtHVDBiSWowZnVjZy1NYXZYY1gfRjcyYyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHotdjMvMTAyMDIxMzU4MjMiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvOTgwNDUxODgiLCAiYWxnIjogIlJTMjU2In0", 
  "payload": "", 
  "signature": "kmsGKpSBGQiMTB4twviu8xlQLkEH7Tk0YrZGUBN2ItGqLuPJabE3e2OlzAdFGnds2lZF26-B_FIGwdekD6f8MKH-BZ6mfyLh0ixqVP4P66eNLhr36R5bbbEuErAys3JvF6GR68PPHr_vSoJO98DXyq97wFEmdfgUo7iKSfKfUiVDyjNP_mEwOWZYJBMO_3Tbh0IS3gu-78rEjuorQkvyXrbkvCJkQXshKX-FCKYKzQrsig3_nkLjSxvf8cPPNVw_sj_v-MTgoAvAbTNe3UeF1ZPYJznbeZnxy8eSseVK-xb8aV0J_umA1PK6yO_CAnYvOYiMpx_ozWGIzWtArtvwNQ"
}
"POST /acme/authz-v3/10202135823 HTTP/1.1" 200 1547
Received response:
HTTP 200
content-length: 1547
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
boulder-requester: 98045188
date: Tue, 19 Jan 2021 19:11:30 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0103zp8Z7QaxFXvSN4sU3K4E4h8bgfVkruxUCp1SF0Pueo

{
  "identifier": {
    "type": "dns",
    "value": "funk.domain.org"
  },
  "status": "invalid",
  "expires": "2021-01-26T19:11:24Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "Invalid response from https://funk.domain.org/.well-known/acme-challenge/mxb0z_sMfn9m1PqMmTmGhS1q2lG_m5mvFTRTLtj0pW8 [PUBLIC IP]: \"\\n\u003c!doctype html\u003e\\n\u003chtml lang=\\\"en\\\"\u003e\\n\u003chead\u003e\\n  \u003ctitle\u003eNot Found\u003c/title\u003e\\n\u003c/head\u003e\\n\u003cbody\u003e\\n  \u003ch1\u003eNot Found\u003c/h1\u003e\u003cp\u003eThe requested resource\"",
        "status": 403
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/10202135823/oNWpTQ",
      "token": "mxb0z_sMfn9m1PqMmTmGhgf1q2lG_m5mvFTRTLtj0pW8",
      "validationRecord": [
        {
          "url": "http://funk.domain.org/.well-known/acme-challenge/mxb0z_sMfn9m1PqMmTmGhS1q2lG_m5mvFTRTLtj0pW8",
          "hostname": "funk.domain.org",
          "port": "80",
          "addressesResolved": [
            "PUBLIC IP"
          ],
          "addressUsed": "PUBLIC IP"
        },
        {
          "url": "https://funk.domain.org/.well-known/acme-challenge/mxb0z_sMfn9m1PqMmTmGhS1q2lG_m5mvFTRTLtj0pW8",
          "hostname": "funk.domain.org",
          "port": "443",
          "addressesResolved": [
            "PUBLIC IP"
          ],
          "addressUsed": "PUBLIC IP"
        }
      ]
    }
  ]
}
Storing nonce: 0103zp8Z7QaxFXvSN4sU3K4E4h8fgTIbVkruxUCp1SF0Pueo
Challenge failed for domain funk.domain.org
http-01 challenge for funk.domain.org
Reporting to user: The following errors were reported by the server:

Domain: funk.domain.org
Type:   unauthorized
Detail: Invalid response from https://funk.domain.org/.well-known/acme-challenge/mxb0z_sMfn9m1PqMmTmGhS1q2lG_m5mvFTRTLtj0pW8 [PUBLIC IP]: "\n<!doctype html>\n<html lang=\"en\">\n<head>\n  <title>Not Found</title>\n</head>\n<body>\n  <h1>Not Found</h1><p>The requested resource"

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
AuthorizationError: Some challenges have failed.

yep, could install the nexcloud cert, its the second (combined) cert for funkwhale that is failing.

Andy_Wismer · January 19, 2021, 7:30pm

@tmp501

Hi Philip

I have a hunch:

Check if the DNS for the problem domain is still working, or if there is a dyndns issue here…
That entry should point to your NethServer (External IP)…

My 2 cents
Andy

tmp501 · January 20, 2021, 11:39am

hey andy, thanks for the reply.
still kinda stuck here, how would i test the dns entries? from what i can see it got the right ip(replaced ip with public ip text). and i could get a cert for nextcloud which uses the same public ip. just a different sub domain. when i first installed funkwhale i had two seperate certs, then switch to one combined cert (i had to disable the cert lines in the funkwhale apache config). maybe i should try a seperate cert again maybe with a different domain name? ill keep on fiddling around this afternoon…

   The following errors were reported by the server:
   Domain: funk.domain.org
   Type:   unauthorized
   Detail: Invalid response from
   https://funk.domain.org/.well-known/acme-challenge/-ONxNrXWJfWzqxUQI2jjT2l-9x0sCbTpXUE54RTHVAE
   [PUBLIC IP]: "\n<!doctype html>\n<html lang=\"en\">\n<head>\n
   <title>Not Found</title>\n</head>\n<body>\n  <h1>Not
   Found</h1><p>The requested resource"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

Andy_Wismer · January 20, 2021, 11:46am

@tmp501

Hi

You could test the DNS from your PC / Notebook in the following form:

nslookup [-query=any] hostname-to-be-queried [DNS server]

Anything within [] is optional…
If you do not specify a DNS Server like Google, your Internal DNS (Router? NethServer?) will be used, maybe falsifiying results…

I’d suggest simply as follows:

nslookup domain.org 8.8.8.8

or

nslookup -query=any domain.org 8.8.8.8

nslookup -query=a domain.org 8.8.8.8

nslookup -query=cname domain.org 8.8.8.8

or

If you have a static IP and want to see what DNS-Name that points to:

nslookup -query=ptr YOUR-IP 8.8.8.8

My 2 cents
Andy

tmp501 · January 20, 2021, 11:56am

hmm… looks good to me…

nslookup -query=any funk.domain.org 8.8.8.8
Server:         8.8.8.8
Address:        8.8.8.8#53

Non-authoritative answer:
Name:   funk.domain.org
Address: CORRECT PUBLIC IP

Authoritative answers can be found from:

Andy_Wismer · January 20, 2021, 12:22pm

Now you need to test with eg a mobile phone / using a mobile hotspot and see if the server is actually accessible… (From the Outside, not from your LAN…)

danb35 · January 20, 2021, 12:39pm

But it’s been shown that it is:

   The following errors were reported by the server:
   Domain: funk.domain.org
   Type:   unauthorized
   Detail: Invalid response from
   https://funk.domain.org/.well-known/acme-challenge/-ONxNrXWJfWzqxUQI2jjT2l-9x0sCbTpXUE54RTHVAE
   [PUBLIC IP]: "\n<!doctype html>\n<html lang=\"en\">\n<head>\n
   <title>Not Found</title>\n</head>\n<body>\n  <h1>Not
   Found</h1><p>The requested resource"

That’s not a “connection refused”; it means the LE servers were able to connect to his system (well, a system–and if the public IP is correct, it should be his system) and it responded with a suitable HTTP response. But there are lots of problems:

The response is a 404–the token can’t be found
That response is coming from nginx–why? Neth uses Apache as its web server.
next.domain gives the default Neth start page, not Nextcloud.
The cert being served on next.domain was issued about 18 hours ago and is valid–good. But it doesn’t include funk.domain, and funk.domain is serving the same cert–so it’s giving a cert error.
If you bypass the cert error on funk.domain, you get

So there’s a lot going on, much of it wrong–but I think the key issue is the second: why is Let’s Encrypt hitting an Nginx server?

Andy_Wismer · January 20, 2021, 12:42pm

@danb35

That’s implying the “right” server, as you say, something here (nginx) is not part of what can be expected on a Nethserver… (I did use “the” and not “a server”…)

Nginx CAN be installed, but you’re more or less on your own there…

My 2 cents
Andy

danb35 · January 20, 2021, 12:45pm

…but the responses (i.e., the pages served) are exactly what would be expected.

But I just reviewed the thread, and realized that OP isn’t serving Nextcloud on next.domain, but on next.domain/nextcloud. That comes up for me just fine with a login page, secured by the same 18-hour-old cert. Looks like it’s working perfectly. The issue is with the funk.domain virtual host; there doesn’t seem to be any problem at all with Nextcloud.

Edit: A little further investigation reveals that next. and funk. both resolve to the same IP address (good), and that’s the same IP address that the LE servers were using for funk. (also good). But despite the text of the error saying “Not Found” (which would suggest a 404), the error is actually 403, unauthorized. Digging a little further with curl indicates that whatever software is supposed to be running on funk.domain is identified as “uvicorn” by curl–so funk.domain must be reverse proxying to that software. So perhaps the answer to:

is that OP has a Nginx reverse proxy (either on the Neth box itself, or on some other system on his LAN) that’s directing traffic. I’m not sure if curl would detect that (it reports next.domain being served by Apache 2.4.6, exactly what you’d expect on Nethserver), though.

But in any case, the problem is entirely with the funk. subdomain/virtual host. And there are problems:

There’s never been a cert issued for funk.domain
funk.domain is serving a cert that doesn’t have that name on it (it only has next.domain), causing certificate errors
When you ignore the cert error, whatever software is supposed to be serving funk.domain is dead/broken/misconfigured.

As far as I can tell, funk.domain is responding appropriately to requests for tokens (I get a 404, which is what you’d expect when certbot isn’t in the middle of a run). But I think we need some clarification from OP about what the setup is.

tmp501 · January 20, 2021, 1:24pm

thats weird. i cant recall haveing installed ngnix at any point…

rpm -qi ngnix  
package ngnix is not installed

ill dig into it a bit later… was thinking about trying to install a new cert…see if that works…
funk.domain.org should be running (and it is up and running) a funkwhale instance
Howto install Funkwhale 0.20.1

excerpt of zzz_funkwhale.conf

# HTTP requests redirected to HTTPS
<VirtualHost *:80>
   ServerName ${funkwhale-sn}

# Default is to force https
   RewriteEngine on
   RewriteCond %{SERVER_NAME} =${funkwhale-sn}
   RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]

   <Location "/.well-known/acme-challenge/">
      Options None
      Require all granted
   </Location>
</VirtualHost>


<IfModule mod_ssl.c>
<VirtualHost *:443>
   ServerName ${funkwhale-sn}

   # Path to ErrorLog and access log
   ErrorLog /var/log/funkwhale/error.log
   CustomLog /var/log/funkwhale/access.log combined

#Header always set Service-Worker-Allowed "/"

   # TLS
   # Feel free to use your own configuration for SSL here or simply remove the
   # lines and move the configuration to the previous server block if you
   # don't want to run funkwhale behind https (this is not recommended)
   # have a look here for let's encrypt configuration:
   # https://certbot.eff.org/lets-encrypt/debianstretch-apache.html
   SSLEngine on
   SSLProxyEngine On
#   SSLCertificateFile /etc/letsencrypt/live/${funkwhale-sn}/fullchain.pem
#   SSLCertificateKeyFile /etc/letsencrypt/live/${funkwhale-sn}/privkey.pem
#   Include /etc/letsencrypt/options-ssl-apache.conf

   # Tell the api that the client is using https
   RequestHeader set X-Forwarded-Proto "https"

   # Configure Proxy settings
   # ProxyPreserveHost pass the original Host header to the backend server
   ProxyVia On
   ProxyPreserveHost On
   <IfModule mod_remoteip.c>
      RemoteIPHeader X-Forwarded-For
   </IfModule>

   # Turning ProxyRequests on and allowing proxying from all may allow
   # spammers to use your proxy to send email.
   ProxyRequests Off

   <Proxy *>

thanks for your help guys!

danb35 · January 20, 2021, 1:37pm

Why do you have funk.domain redirecting to SSL, when you’ve never had a cert for funk.domain?

tmp501 · January 20, 2021, 1:54pm

i had one… for some reason my two certs did not get renewed, and i found out the day they expired. requesting them again through the NS UI did not work (pretty much the same error). then i did probably a stupid thing and did what one user on NS forums said worked for him. purging the letsencryp certs reinstalling neth-letsencrypt and start from scratch.

for the lookup thing. i checked if its reachable through a dns lookup website and all checked out fine.

same error requesting a brand new domain name… so i guess something is off in the apache zzz_funkwhale.conf ?

danb35 · January 20, 2021, 2:53pm

My mistake–the tool I was using only looks at certs that might be relevant for rate limits, so it didn’t see the older one.

I’m thinking so–it isn’t pointing /.well-known/acme-challenge/ to the right place.

tmp501 · January 20, 2021, 6:27pm

phew! its working now…i think one of those funkwhale services were stuck somehow. i just switched funkwhales settings back to my original domain name, then it acted up not wanting to start services… whatever it was i endet up using this funkwhale apache config

#<VirtualHost *:80>
#   ServerName ${funkwhale-sn}
#   RedirectMatch 301 ^(?!/.well-known/acme-challenge/).* https://${funkwhale-sn}
#</VirtualHost>

# HTTP requests redirected to HTTPS
<VirtualHost *:80>
   ServerName ${funkwhale-sn}

# Default is to force https
   RewriteEngine on
   RewriteCond %{SERVER_NAME} =${funkwhale-sn}
   RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]

   <Location "/.well-known/acme-challenge/">
      Options None
      Require all granted
   </Location>
</VirtualHost>


<IfModule mod_ssl.c>
<VirtualHost *:443>
   ServerName ${funkwhale-sn}

   # Path to ErrorLog and access log
   ErrorLog /var/log/funkwhale/error.log
   CustomLog /var/log/funkwhale/access.log combined

#Header always set Service-Worker-Allowed "/"

   # TLS
   # Feel free to use your own configuration for SSL here or simply remove the
   # lines and move the configuration to the previous server block if you
   # don't want to run funkwhale behind https (this is not recommended)
   # have a look here for let's encrypt configuration:
   # https://certbot.eff.org/lets-encrypt/debianstretch-apache.html
   SSLEngine on
   SSLProxyEngine On
#   SSLCertificateFile /etc/letsencrypt/live/${funkwhale-sn}/fullchain.pem
#   SSLCertificateKeyFile /etc/letsencrypt/live/${funkwhale-sn}/privkey.pem
#  Include /etc/letsencrypt/options-ssl-apache.conf

the commented virtualhost section is the one suggested in the NS funkwhale thread. the virtualhost section that is active right now is how the FW people do it nowadays. the certfile references are disabled because i am using this combined NS cert thingy.
thanks you all - still loving NS!