NethServer Version: 8
Module: Core, Traefik, Roundcubemail
Using cluster-admin I did this morning a NS8 software update. Core, mail, imapsync and traefik got updated to latest version.
After update my mailaddress to roundcubemail shows error “404 page not found”, directly from local lan as from internet. cluster-admin was working find. I did restart of NS8. Now also cluster-admin was also giving “404 page not found error”. After around 5-10 minutes this was suddenly solved and I could access cluster-admin again. Accessing webmail was still showing "404 page not found error’. No issues accessing mail via android app. Sending and receiving mail also works fine via android app. Looking though system log and roundcubemail log via cluster-admin doesn’t give me any clear hints.
Now again suddenly after 2 hours no access to cluster-admin: “404 page not found” (http or https). I can access NS8 server via telnet session. Also Cockpit access to NS8 server is still working. NS8 is running as a VM on ESXI. OPNsense (also VM) is used as firewall. OPNsense is used for the letsencrypt certificate management and TLS termination (roundcubemail) and HAproxy within OPNsense for routing to NS8. Roundcube mail app in NS8 is configured accordingly. All was working stable last few months since NS8 install. Any help appreciated.
Hi,
welcome to NethServer community.
Is there high load on the server?
It seems that traefik is up as it shows the 404 error but the HTTP route for roundcube isn’t working while others seem to work sometimes.
Please check the HTTP routes in the NS8 settings, your apps should be listed there. Maybe the roundcube FQDN is used twice? An app FQDN must not be the NS8 FQDN.
Did you already try to save the settings of the roundcube app to reconfigure it?
Thanks for thinking with me.
No high load on server.
HTTP routes in NS8 settings are available for roundcubemail
Request LE certificate and Force HTTPS are disabled in roundcubemail settings. Changed settings, saved and back to original. Same 404 issue.
Cluster-admin looks stable now.
Roundcubemail was working fine before installing latest Software Center updates.
Config now:
core 3.6.0
traefik 3.0.0
mail 1.6.0
I also have a NS-Test server running in same setup but not yet updated:
core 3.4.1
traefik 2.2.5
mail 1.5.0
roundcubemail v2.0.6
Here no 404 error but working roundcube webmail.
I also tried the “Trust Forward Headers” (Trust of forwarded headers in Traefik · Issue #7305 · NethServer/dev · GitHub)
I can see now the original ip address instead of the gateway address (OPNsense) in the Traefik logging but I didn’t solve the 404 issue.
Just to be sure, is it about following “404 page not found” error?
I still think there’s an error regarding the roundcube http route because traefik seems running as it shows the “404 Page not found” which means that the route is wrong. If the route is correct and the app isn’t started you’d get a “Bad Gateway” error.
Please recheck if DNS is ok on client and server and the name in the URL is really the right one and points to the right IP of the NS8, for example using
nslookup webmail.domain.tld
Please check if the NS8 hostname and the used app FQDNs are unique.
Get FQDN of NS8:
hostname -f
Get FQDNs of the apps:
runagent -m traefik1 bash -c "grep -h rule: configs/* | uniq | sort"
Let’s check traefik roundcube configs:
runagent -m traefik1 tail -n +1 configs/roundcubemail1.yml configs.v2/roundcubemail1.yml
Here's the result on my NS8 to compare
==> configs/roundcubemail2.yml <==
http:
services:
roundcubemail2:
loadBalancer:
servers:
- url: http://127.0.0.1:20017
routers:
roundcubemail2-http:
rule: Host(`webmail.ns8rockytest.com`)
priority: '2'
entryPoints:
- http
service: roundcubemail2
roundcubemail2-https:
rule: Host(`webmail.ns8rockytest.com`)
priority: '2'
entryPoints:
- https
service: roundcubemail2
tls: {}
==> configs.v2/roundcubemail2.yml <==
http:
services:
roundcubemail2:
loadBalancer:
servers:
- url: http://127.0.0.1:20017
routers:
roundcubemail2-http:
rule: Host(`webmail.ns8rockytest.com`)
priority: '2'
entryPoints: http,https
service: roundcubemail2
roundcubemail2-https:
rule: Host(`webmail.ns8rockytest.com`)
priority: '2'
entryPoints: http,https
service: roundcubemail2
tls:
domains:
- main: webmail.ns8rockytest.com
You may also check if roundcube is reachable in NS8:
Get the url of the roundcube HTTP route:
and test it on the NS8:
[root@ns8rockytest ~]# curl 127.0.0.1:20017 --stderr - | grep Roundcube
<meta http-equiv="content-type" content="text/html; charset=UTF-8"><title>Roundcube Webmail
Hi Markus, thanks for your extensive answer.
As there was no Roundcubemail app update I suspected a issue in the Traefik changes for my configuration. I just started to compare the yml files of my production and (not upgraded) Test server. It showed a difference in the rouncubemail1.yml’s (what you also can see in your “Let’s check traefik roundcube configs:”). In the old config the router “roundcubemail1-http” has “entryPoints: http,https” and in the new only “entryPoints: - http”. Same for router “roundcubemail1-https” but other way around. So I changed for both entryPoints this (back) to both “- http and -https”. Restarted Traefik and problem solved, Roundcubemail is working again!
I don’t know Traefik good enough. I know HAproxy a little bit (using it in OPNsense), Traefik looks a little bit like it but I don’t yet understand what exactly caused the issue.
Apparently my configuration: using OPNsense for letsencrypt certificate handling and TLS termination (also for my roundcube hostname), passing this http-unencrypted via HAproxy (as OPNsense service) to my NS8 server doesn’t work with the latest Traefik update to V3.0.0
Hopefully this proxy-after-proxy “bug” is getting solved in one of the next updates.
A still happy NS8 user.
Hi @Joost
In my experience (extensive OPNsense use…), the nginx works better as a reverse Proxy on OPNsense than HAProxy, and also supports more options.
My 2 cents
Andy
I’m glad that it works again but I can’t reproduce or understand it yet. You reverted the changes from the traefik update but when I try the same I get the “404 page not found” error.
Did you edit configs/roundcubemail1.yml or configs.v2/roundcubemail1.yml?
Maybe just a restart of traefik without changing the traefik config would have been enough?
After a configuration change or an update of roundcube or traefik the roundcube traefik config file will be rewritten.
I only changed the “configs/roundcubemail1.yml”
If I change it back to the original settings and restart Traefik I get the 404 again.
I see that I only have to add the “https” as extra entryPoint to the “roundcubemail1-http” part to get it working.
My working “configs/roundcubemail1.yml”:
http:
services:
roundcubemail1:
loadBalancer:
servers:
- url: http://127.0.0.1:20010
routers:
roundcubemail1-http:
rule: Host(`mymail.nl`)
priority: 2
service: roundcubemail1
entryPoints:
- http
- https
roundcubemail1-https:
rule: Host(`mymail.nl`)
priority: 2
service: roundcubemail1
entryPoints:
- https
tls:
domains:
- main: mymail.nl
OK, I changed it in a wrong way so that change works here too but the routers are separated by http/https so the additional https entrypoint for the http router looks wrong to me.
Maybe it’s needed due to the haproxy configuration? Is it maybe wrongly using https to connect to port 80?
Hi Markus,
You were right. HAproxy was wrongly using using port 443 to connect to NS8.
I did a “packet capture” in OPNsense and loaded it in Wireshark: plaintext Roundcubemail data over port 443.
My assumption that TLS termination in HAproxy would mean that default port 80 would be used for the backend server connection was wrong. HAproxy documentation says about this: “ is an optional port specification. If set, all connections will be sent to this port. If unset, the same port the client connected to will be used.” I added “:80” to the HAproxy NS8 backend server definition. Changed the “configs/roundcubemail1.yml” to original settings with Traefik restart and Roundcubemail just works. Apologies for bug-blaming NS8. Thanks Markus for putting me in the right direction.