Can't access configured service locally

djx · June 24, 2020, 10:02pm

NethServer Version: 7.8.2003
Module: services / firewall

I am running a Docker container on port 8443. I can access this externally at: https://meet.mydomain.com:8443

I can see that the service is set up like this:

and that the port is being listened on:

[root@neth conf.d]# netstat -plantu | grep "8443"
tcp6       0      0 :::8443                 :::*                    LISTEN      3833/docker-proxy

However, I noticed that any sort of local access fails. I am trying to set up a reverse proxy to host this service on my sub-domain without the extra port but it always gives an error that the service is unavailable, even though it’s accessible when I access the link directly (port 8443)

I tested with lynx and it seems that I am unable to connect to port 8443 locally on any interface, I have tried my WAN IP, my LAN IP, 127.0.0.1, the docker bridge IP, etc. All of these fail to connect

I was running tcpdump on all interfaces while making these attempts and there were no packets received, which leads me to believe it’s a firewall issue

[root@neth docker-jitsi-meet]# tcpdump -vv -i any port 8443
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes

Please point out my stupid mistake!

pike · June 25, 2020, 4:05am

Routing?

djx · June 25, 2020, 10:32am

Thanks - I checked this with ip route get and it is recognized as a local route.

I checked the Shorewall logs when doing lynx https://10.0.0.2:8443 and I see this:

Jun 25 18:29:27 neth kernel: Shorewall:INPUT:REJECT:IN=br-c182c5c8c166 OUT= PHYSIN=vethc60c9cb MAC=02:42:53:c4:84:17:02:42:ac:12:00:04:08:00 SRC=172.18.0.4 DST=10.0.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=443 DPT=44352
 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Jun 25 18:29:28 neth kernel: Shorewall:INPUT:REJECT:IN=br-c182c5c8c166 OUT= PHYSIN=vethc60c9cb MAC=02:42:53:c4:84:17:02:42:ac:12:00:04:08:00 SRC=172.18.0.4 DST=10.0.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=443 DPT=44352
 WINDOW=28960 RES=0x00 ACK SYN URGP=0
Jun 25 18:29:30 neth kernel: Shorewall:INPUT:REJECT:IN=br-c182c5c8c166 OUT= PHYSIN=vethc60c9cb MAC=02:42:53:c4:84:17:02:42:ac:12:00:04:08:00 SRC=172.18.0.4 DST=10.0.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=443 DPT=44352
 WINDOW=28960 RES=0x00 ACK SYN URGP=0

So it seems like shorewall is rejecting this traffic. Is this expected by default?

mrmarkuz · June 25, 2020, 3:22pm

No, local access to published ports or directly to aqua container ports should just work.
I assume you are using the jitsi container from your howto, I’d like to use it to setup jitsi for testing asap.
I tested with a simple nginx container by running

docker run --name mynginx1 -p 8080:80 -d nginx

Then I put it to aqua network via portainer like you described in the howto and the test connection worked.

[root@testserver ~]# curl localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

curl 172.28.0.2 is working too (direct access to nginx container)

Did you put the container to the aqua network? Because br-... seems to be another network.

djx · June 25, 2020, 5:56pm

You are correct that I’m using the steps from my howto.

I’m not sure what this br* network is, but it appears that’s where the traffic is coming from whenever I do commands in the shell.

My network is very simple and I have not (to my knowledge) done anything special:

Yet, when I look at my ifconfig I can see the br* interface and a lot of other veth interfaces and I’m not sure what they are for. I’ve pasted the list below.

Based on your comment I tried adding the docker-jitsi-meet_web container to the aqua network and suddenly my commands started to work - including the reverse proxy. Previously, only the docker-jitsi-meet_prosody container was on aqua for the LDAP connection because I don’t expose my LDAP to the WAN.

I understand why https://10.0.0.2:8443 does not work without the container being on aqua.

I do not understand why https://meet.mydomain.com:8443 does not work without aqua. This should not be attempting to do a local connection, but an external connection - which we know works because I can access the site externally.

Thanks for your help so far - now I can debug the BOSH connectivity errors with the reverse proxy! I would appreciate if you can help me understand the other network interfaces and why the external URL failed without the container being on the aqua network.

aqua0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.28.0.1  netmask 255.255.0.0  broadcast 172.28.255.255
        inet6 fe80::42:62ff:fe41:506b  prefixlen 64  scopeid 0x20<link>
        ether 02:42:62:41:50:6b  txqueuelen 0  (Ethernet)
        RX packets 308290  bytes 178405806 (170.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 320268  bytes 83195868 (79.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

br-c182c5c8c166: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.18.0.1  netmask 255.255.0.0  broadcast 172.18.255.255
        inet6 fe80::42:53ff:fec4:8417  prefixlen 64  scopeid 0x20<link>
        ether 02:42:53:c4:84:17  txqueuelen 0  (Ethernet)
        RX packets 308290  bytes 178405806 (170.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 320268  bytes 83195868 (79.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:a2:27:64:d1  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 99.99.99.99  netmask 255.255.255.255  broadcast 99.99.99.99
        inet6 fe80::9400:ff:fe33:f5dd  prefixlen 64  scopeid 0x20<link>
        ether 99:99:99:99:99:99  txqueuelen 1000  (Ethernet)
        RX packets 83158300  bytes 33063322710 (30.7 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 87569755  bytes 47684138251 (44.4 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.0.0.2  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 fe80::8400:ff:fe33:f5e2  prefixlen 64  scopeid 0x20<link>
        ether 86:00:00:33:f5:e2  txqueuelen 1000  (Ethernet)
        RX packets 12  bytes 2924 (2.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 100  bytes 8280 (8.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
		
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 26521085  bytes 3491519506 (3.2 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 26521085  bytes 3491519506 (3.2 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth261b4ab: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::8405:96ff:fe06:c8c4  prefixlen 64  scopeid 0x20<link>
        ether 86:05:96:06:c8:c4  txqueuelen 0  (Ethernet)
        RX packets 54182  bytes 56671103 (54.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 63778  bytes 33870335 (32.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth531baff: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::94e8:deff:feb8:2bfd  prefixlen 64  scopeid 0x20<link>
        ether 96:e8:de:b8:2b:fd  txqueuelen 0  (Ethernet)
        RX packets 306728  bytes 182604534 (174.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 318693  bytes 55062554 (52.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth78258f2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::c14:63ff:fe33:d072  prefixlen 64  scopeid 0x20<link>
        ether 0e:14:63:33:d0:72  txqueuelen 0  (Ethernet)
        RX packets 26267  bytes 5700784 (5.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 17181  bytes 26371494 (25.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth88c947b: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::c865:c2ff:fee9:8f1a  prefixlen 64  scopeid 0x20<link>
        ether ca:65:c2:e9:8f:1a  txqueuelen 0  (Ethernet)
        RX packets 2406804  bytes 1659731336 (1.5 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2171096  bytes 1692068536 (1.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethc60c9cb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::3c0f:8aff:fe1e:82a7  prefixlen 64  scopeid 0x20<link>
        ether 3e:0f:8a:1e:82:a7  txqueuelen 0  (Ethernet)
        RX packets 48202  bytes 193021318 (184.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 34478  bytes 8544938 (8.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0