Server can no longer connect to Docker Containers

Support
1

Hi folks,

Thanks for all the work maintaining this excellent product! I was reading about NS8 yesterday and it sounds super exciting! :slight_smile:

I have a number of docker containers running on a nethserver server on my internal network. They are exposed to the internet via reverse proxies on a second nethserver running on a raspberry pi 4.

Recently, something changed. I’m guessing in an update, since I haven’t done much else, but I did do some hardening in response to some DOS/SSH attacks recently and maybe I’ve tripped myself up here.

All of the sudden, the pi-server can no longer connect to the docker containers. It just times out. It still has solid connectivity to the server hosting the docker containers, though. I’ve checked firewall, IPS, Fail2ban, etc. I can’t seem to find the problem. All other clients on the internal network can talk to the docker containers. Has something changed recently that might explain this? Perhaps in the way macvlan works?

Detailed Troubleshooting

Here’s how my network looks:

192.168.7.254 - Pi-Server (Nethserver 7.9.2009 ARM7) - Internet connected via port mapping to a non-standard SSH port, and port 443.

192.168.7.55 - Internal Server (Nethserver 7.9.2009 AMD64)

  • Docker Containers:

192.168.7.251 - Calibre-web - macvlan port 2083

192.168.7.250 - Foundry VTT - macvlan port 30000

192.168.7.249 - Emby - macvlan port 8096

Here’s curl running from the Pi-server command line to the internal server hosting Docker:

# curl -vI http://192.168.7.55:9090* About to connect() to 192.168.7.55 port 9090 (#0)
* Trying 192.168.7.55...
* Connected to 192.168.7.55 (192.168.7.55) port 9090 (#0)
> HEAD / HTTP/1.1> User-Agent: curl/7.29.0
> Host: 192.168.7.55:9090> Accept: */* >
< HTTP/1.1 301 Moved PermanentlyHTTP/1.1 301 Moved Permanently
< Content-Type: text/html
Content-Type: text/html
< Location: https://192.168.7.55:9090/
Location: https://192.168.7.55:9090/
< Content-Length: 73
Content-Length: 73
< X-DNS-Prefetch-Control: off
X-DNS-Prefetch-Control: off
< Referrer-Policy: no-referrer
Referrer-Policy: no-referrer
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
<
* Connection #0 to host 192.168.7.55 left intact

Here’s the same thing attempting to contact one of the docker containers:

# curl -vI http://192.168.7.251:8083

* About to connect() to 192.168.7.251 port 8083 (#0)
* Trying 192.168.7.251...
* Connection timed out
* Failed connect to 192.168.7.251:8083; Connection timed out
* Closing connection 0
curl: (7) Failed connect to 192.168.7.251:8083; 
Connection timed out

Here’s a log entry from the reverse proxy. It shows the same time out:

[proxy_http:error] [pid 11179] [client 51.158.109.3:36438] AH01114: HTTP: failed to make connection to backend: 192.168.7.251

And finally, a screenshot showing connectivity from another client:

Screen Shot 2022-05-30 at 4.56.21 PM
Screen Shot 2022-05-30 at 4.56.21 PM1370×840 65.3 KB

NethServer Version: 7.9.2009
Module: Docker?

2

Did you update docker as explained in the docs?

Please share the docker config:

config show docker

Did you check /var/log/firewall.log on the raspi?

2 Likes
3

I did update docker following the docs when I set this up. Are you suggesting I update it again?

Here’s the config:

# config show docker
docker=service
    DirectLvmDevice=
    IpAddress=172.28.0.1
    Network=172.28.0.0/16
    bridgeAeria=
    enableRepository=enabled
    macVlanGateway=192.168.7.1
    macVlanLocalNetwork=192.168.7.0/24
    macVlanNetwork=192.168.7.248/29
    macVlanNic=br0
    status=enabled

I went ahead and tailed the /var/log/firewall.log on the pi while trying to connect. Nothing pops in the logs.

4

The macvlan subnet 192.168.7.248/29 allows an IP range from 192.168.7.249 to 192.168.7.254 which also covers the raspi IP and I think that’s the issue.
You could change the macvlan to 192.168.7.240/29 (allowing range 241-246 for containers) and change the container IPs or change the raspi IP.

No, if you already updated, it should be ok. To get updates with the system updates, you could enable the repository.

3 Likes
5

Thanks. I’ll give that a go and update with results. :slight_smile:

1 Like
6

That was indeed the answer. I changed the macvlan subnet, moved the docker containers, and updated the reverse proxies. Everything is working again. Thank you so much!

1 Like