Portainer config issues

bwdjames · March 21, 2020, 4:51pm

NethServer Version: 7.7.1908
Module: nethserver-docker

I have followed the instructions at GitHub - NethServer/nethserver-docker: Docker configuration for NethServer and I get the following 503 Error whe nI go to https://192.168.1.1:980/portainer/ :

Service Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity
problems. Please try again later.

Think I’m missing something stupidly small in the instructions.

I am a little unsure if the following would have any effect:

The default Docker bridged network is disabled, as long as the iptables mangling feature.

mrmarkuz · March 21, 2020, 5:09pm

I got it working with these steps (Installation section):

stephdl · March 21, 2020, 5:10pm

you are not on the good GH branch, the rpm must built

mrmarkuz · March 21, 2020, 5:11pm

I uploaded a build to my repo for easier installation.

bwdjames · March 21, 2020, 8:54pm

The problem is that when I run the command db networks show aqua, it does not show anything. Its like the aqua zone is not created and neither is the firewall rules.

What are the required commands to achieve this?

mrmarkuz · March 21, 2020, 9:09pm

It’s not in the networks db, it’s only configured in shorewall.

Check /etc/shorewall/zones and /etc/shorewall/policy.

I used to change the policy with a custom template in the nxfilter/pihole howto thread to need no firewall rules for easy testing and because the aqua zone seems not selectable in both server managers.

It looks like it worked some time ago:

EDIT:

I couldn’t manage it via the GUI.
As a workaround you may copy the 65aqua fragment and add wanted firewall rules with a custom template.

mkdir -p /etc/e-smith/templates-custom/etc/shorewall/rules

cp /etc/e-smith/templates/etc/shorewall/rules/65aqua /etc/e-smith/templates-custom/etc/shorewall/rules/

Edit /etc/e-smith/templates-custom/etc/shorewall/rules/65aqua and edit/add your firewall rules.

To apply the config:

signal-event firewall-adjust

bwdjames · March 24, 2020, 11:37am

I don’t actually know what firewall rules I should be adding for this to work right now or how to add them on the command line.

Is there a default set which you would recommend?

I was thinking of tackling or looking at this from a different perspective: what are the implications of having docker accessible in the Green Zone instead of the Aqua Zone?

mrmarkuz · March 24, 2020, 4:11pm

It depends on the ports you need for your containers.
For testing I’d go with open policy like that:

/etc/e-smith/templates/etc/shorewall/policy/35aqua should look like that:

#
# 35aqua -- the Docker network policy
#
aqua net ACCEPT
$FW aqua ACCEPT
aqua $FW ACCEPT
loc aqua ACCEPT

Rules configuration is really simple:

For example /etc/e-smith/templates/etc/shorewall/rules/65aqua

#
# 65aqua Accept ping from aqua
#

Ping/ACCEPT    aqua            $FW


#
# 65aqua -- Rules for Docker containers
#

?COMMENT aqua
ACCEPT  aqua    $FW     tcp     3306

The first rule accepts ping from aqua to the firewall.
The second rule accepts mariadb from firewall to aqua

Here are some commonly used rules:

https://shorewall.org/two-interface.htm#DNS

Don’t forget to apply changes with

signal-event firewall-adjust

Next to bad security, I don’t know if that would be possible easily. Think of DHCP server vs docker setting IPs but never tested.

bwdjames · March 24, 2020, 4:53pm

Thanks @mrmarkuz, definitely has some of those FW rules wrong or a few missing FW rules. I have corrected those as per your example and still getting the 503.

Think I am loosing my marbles and missing a couple of key points on this one…

mrmarkuz · March 24, 2020, 5:02pm

It think it’s not a firewall problem as you get the 503 error. It’s not a rejected connection.

Check if docker is running:

systemctl status docker -l

Check if your containers are running, portainer is just a container:

docker ps -a

Is the aqua interface UP?

ip a s aqua0

You may reconfigure and restart with

signal-event nethserver-docker-update

and check /var/log/messages afterwards.

EDIT:

Did you enable nethserver-docker?

config setprop docker status enabled
signal-event nethserver-docker-update

bwdjames · March 24, 2020, 9:34pm

Mar 24 21:24:43 server.thenesbitts.net dockerd[28597]: time=“2020-03-24T21:24:43.229507950Z” level=info msg=“IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]”
Mar 24 21:24:43 server.thenesbitts.net dockerd[28597]: time=“2020-03-24T21:24:43.273699678Z” level=error msg=“ca281aba5e2a3a49cfb76559bab48538b31c005ab7965f7f2dda1885b0c4ce1a cleanup: failed to delete container from containerd: no such container”
Mar 24 21:24:43 server.thenesbitts.net dockerd[28597]: time=“2020-03-24T21:24:43.273764739Z” level=error msg=“Failed to start container ca281aba5e2a3a49cfb76559bab48538b31c005ab7965f7f2dda1885b0c4ce1a: network cc0cbf3f87550dd62a17593440f3911b0031634a87db104c3a8cb2b69cc3ee86 not found”
Mar 24 21:24:43 server.thenesbitts.net dockerd[28597]: time=“2020-03-24T21:24:43.811933166Z” level=info msg=“No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]”
Mar 24 21:24:43 server.thenesbitts.net dockerd[28597]: time=“2020-03-24T21:24:43.811974956Z” level=info msg=“IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]”
Mar 24 21:24:45 server.thenesbitts.net dockerd[28597]: time=“2020-03-24T21:24:45.983636380Z” level=info msg=“Loading containers: done.”
Mar 24 21:24:46 server.thenesbitts.net dockerd[28597]: time=“2020-03-24T21:24:46.394948171Z” level=info msg=“Docker daemon” commit=afacb8b graphdriver(s)=overlay2 version=19.03.8
Mar 24 21:24:46 server.thenesbitts.net dockerd[28597]: time=“2020-03-24T21:24:46.395793670Z” level=info msg=“Daemon has completed initialization”
Mar 24 21:24:46 server.thenesbitts.net dockerd[28597]: time=“2020-03-24T21:24:46.581339725Z” level=info msg=“API listen on /var/run/docker.sock”
Mar 24 21:24:46 server.thenesbitts.net systemd[1]: Started Docker Application Container Engine.

docker ps -a

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c023fbed0877 pihole/pihole:latest “/s6-init” 5 hours ago Up About a minute (healthy) 53/udp, 53/tcp, 80/tcp, 443/tcp, 67/udp pihole
5a98f45d8dd8 packetworks/nxfilter-base:latest “/nxfilter/bin/start…” 5 hours ago Up About a minute nxfilter
ca281aba5e2a portainer/portainer “/portainer” 3 days ago Exited (2) 3 days ago portainer

ip a s aqua0

6: aqua0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:cc:b1:4b:4f brd ff:ff:ff:ff:ff:ff
inet 172.28.5.254/16 brd 172.28.255.255 scope global aqua0
valid_lft forever preferred_lft forever
inet6 fe80::42:ccff:feb1:4b4f/64 scope link
valid_lft forever preferred_lft forever

I am getting the following in /var/log/messages:

Mar 24 21:24:43 server dockerd: time=“2020-03-24T21:24:43.273699678Z” level=error msg=“ca281aba5e2a3a49cfb76559bab48538b31c005ab7965f7f2dda1885b0c4ce1a cleanup: failed to delete container from containerd: no such container”
Mar 24 21:24:43 server dockerd: time=“2020-03-24T21:24:43.273764739Z” level=error msg=“Failed to start container ca281aba5e2a3a49cfb76559bab48538b31c005ab7965f7f2dda1885b0c4ce1a: network cc0cbf3f87550dd62a17593440f3911b0031634a87db104c3a8cb2b69cc3ee86 not found”
Mar 24 21:24:43 server dockerd: time=“2020-03-24T21:24:43.811933166Z” level=info msg=“No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]”
Mar 24 21:24:43 server dockerd: time=“2020-03-24T21:24:43.811974956Z” level=info msg=“IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]”

And the docker logs is showing a lot of the following:

2020/03/21 20:14:40 Migrating database from version 0 to 22.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x88722a]

Still trying to figure out how to fix this now

mrmarkuz · March 24, 2020, 9:37pm

The portainer container is not running, try the following:

docker start ca281aba5e2a

bwdjames · March 25, 2020, 11:02am

Ah! So if I run that command I get the following:

Error response from daemon: network cc0cbf3f87550dd62a17593440f3911b0031634a87db104c3a8cb2b69cc3ee86 not found
Error: failed to start containers: ca281aba5e2a

I’m not sure exactly how to fix that with breaking everything.

mrmarkuz · March 25, 2020, 5:20pm

OK, let’s remove portainer container and recreate it.

Be sure to get the right container ID of portainer:

docker ps -a

Remove it!

docker rm -f <container ID>

Check if it’s deleted:

docker ps -a

Just to be sure we delete the image too, get the image ID:

docker images -a

Remove images:

docker rmi -f <image> ID

Recreate portainer container (it’s recreated if it doesn’t exist):

signal-event nethserver-docker-update

Check if portainer is started now:

docker ps -a

bwdjames · March 26, 2020, 9:08am

@mrmarkuz Thanks for that, that has fixed the one issue and its at least starting now and not complaining about missing/incorrect network.

Now the issue is a couple of seconds after it has successfully start, it is restarting the container and I can see the following in the logs?

    /home/vsts/work/1/s/api/cmd/portainer/main.go:521 +0x99
    2020/03/26 09:04:03 Migrating database from version 0 to 22.
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x88722a]

Not sure if this is the cause of the restarts?

Andy_Wismer · March 26, 2020, 9:50am

Hi

It could also be s simple defective memory chip…
Check that out using the basis BIOS, that often has options for a memory check.
Or a linux Live CD like SystemrescueCD…

Had a stumper like that a while ago. it was one of the higher RAM chips, and only “barfed” when used… (Barf=core dump)…

The chip reports to the BIOS the amount of RAM, real, available RAM is the less due to the defective chip, and…

My 2 cents
Andy

mrmarkuz · March 26, 2020, 2:14pm

I think you’re hitting this one:

You need to delete the content of the local portainer directory and restart:

rm -Rf /var/lib/nethserver/portainer/*
signal-event nethserver-docker-update

bwdjames · March 26, 2020, 4:44pm

@mrmarkuz it is working now. Thank you for all of your help!

pagaille · May 3, 2020, 12:32pm

Hey @mrmarkuz

I’m playing again with nethserver-docker and also have some issues.

In my case I see that the docker service is stuck in a start / restart loop. I already checked your advices but no result.

Any other idea ?

Here are the looping messages :

May  3 14:29:51 mattlabs kernel: XFS (dm-3): Mounting V5 Filesystem
May  3 14:29:52 mattlabs kernel: XFS (dm-3): Ending clean mount
May  3 14:29:52 mattlabs kernel: aqua0: port 1(veth4b6586b) entered blocking state
May  3 14:29:52 mattlabs kernel: aqua0: port 1(veth4b6586b) entered disabled state
May  3 14:29:52 mattlabs kernel: device veth4b6586b entered promiscuous mode
May  3 14:29:52 mattlabs kernel: IPv6: ADDRCONF(NETDEV_UP): veth4b6586b: link is not ready
May  3 14:29:52 mattlabs NetworkManager[1002]: <info>  [1588508992.0608] manager: (veth2087142): new Veth device (/org/freedesktop/NetworkManager/Devices/228)
May  3 14:29:52 mattlabs NetworkManager[1002]: <info>  [1588508992.0652] manager: (veth4b6586b): new Veth device (/org/freedesktop/NetworkManager/Devices/229)
May  3 14:29:52 mattlabs dockerd: time="2020-05-03T14:29:52.143685906+02:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
May  3 14:29:52 mattlabs dockerd: time="2020-05-03T14:29:52.143810918+02:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
May  3 14:29:52 mattlabs containerd: time="2020-05-03T14:29:52.403520601+02:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/c48911ae7d7d10a1f815d7e53314bba92dd537d5b710e5018ac6d7fa0a2940f9/shim.sock" debug=false pid=26728
May  3 14:29:52 mattlabs avahi-daemon[14690]: Withdrawing workstation service for veth2087142.
May  3 14:29:52 mattlabs kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
May  3 14:29:52 mattlabs kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
May  3 14:29:52 mattlabs kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth4b6586b: link becomes ready
May  3 14:29:52 mattlabs kernel: aqua0: port 1(veth4b6586b) entered blocking state
May  3 14:29:52 mattlabs kernel: aqua0: port 1(veth4b6586b) entered forwarding state
May  3 14:29:52 mattlabs NetworkManager[1002]: <info>  [1588508992.8035] device (veth4b6586b): carrier: link connected
May  3 14:29:52 mattlabs NetworkManager[1002]: <info>  [1588508992.8047] device (aqua0): carrier: link connected
May  3 14:29:52 mattlabs sh: sendto: Operation not permitted
May  3 14:29:52 mattlabs sh: sendto: Operation not permitted
May  3 14:29:52 mattlabs sh: sendto: Operation not permitted
May  3 14:29:52 mattlabs sh: sendto: Operation not permitted
May  3 14:29:53 mattlabs containerd: time="2020-05-03T14:29:53.852294314+02:00" level=info msg="shim reaped" id=c48911ae7d7d10a1f815d7e53314bba92dd537d5b710e5018ac6d7fa0a2940f9
May  3 14:29:53 mattlabs dockerd: time="2020-05-03T14:29:53.864887501+02:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
May  3 14:29:53 mattlabs kernel: aqua0: port 1(veth4b6586b) entered disabled state
May  3 14:29:53 mattlabs NetworkManager[1002]: <info>  [1588508993.9313] manager: (veth2087142): new Veth device (/org/freedesktop/NetworkManager/Devices/230)
May  3 14:29:53 mattlabs kernel: aqua0: port 1(veth4b6586b) entered disabled state
May  3 14:29:54 mattlabs avahi-daemon[14690]: Withdrawing workstation service for veth2087142.
May  3 14:29:54 mattlabs kernel: device veth4b6586b left promiscuous mode
May  3 14:29:54 mattlabs kernel: aqua0: port 1(veth4b6586b) entered disabled state
May  3 14:29:54 mattlabs avahi-daemon[14690]: Withdrawing workstation service for veth4b6586b.
May  3 14:29:54 mattlabs NetworkManager[1002]: <info>  [1588508994.0248] device (veth4b6586b): released from master device aqua0
May  3 14:29:54 mattlabs kernel: XFS (dm-3): Unmounting Filesystem
May  3 14:30:04 mattlabs kernel: sdc: sdc1
May  3 14:30:53 mattlabs kernel: XFS (dm-3): Mounting V5 Filesystem
May  3 14:30:54 mattlabs kernel: XFS (dm-3): Ending clean mount
May  3 14:30:54 mattlabs NetworkManager[1002]: <info>  [1588509054.1235] manager: (vethfe190a0): new Veth device (/org/freedesktop/NetworkManager/Devices/231)
May  3 14:30:54 mattlabs NetworkManager[1002]: <info>  [1588509054.1351] manager: (veth04143fe): new Veth device (/org/freedesktop/NetworkManager/Devices/232)
May  3 14:30:54 mattlabs kernel: aqua0: port 1(veth04143fe) entered blocking state
May  3 14:30:54 mattlabs kernel: aqua0: port 1(veth04143fe) entered disabled state
May  3 14:30:54 mattlabs kernel: device veth04143fe entered promiscuous mode
May  3 14:30:54 mattlabs kernel: IPv6: ADDRCONF(NETDEV_UP): veth04143fe: link is not ready
May  3 14:30:54 mattlabs dockerd: time="2020-05-03T14:30:54.229504295+02:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
May  3 14:30:54 mattlabs dockerd: time="2020-05-03T14:30:54.229552967+02:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
May  3 14:30:54 mattlabs containerd: time="2020-05-03T14:30:54.489196538+02:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/c48911ae7d7d10a1f815d7e53314bba92dd537d5b710e5018ac6d7fa0a2940f9/shim.sock" debug=false pid=27944
May  3 14:30:54 mattlabs avahi-daemon[14690]: Withdrawing workstation service for vethfe190a0.
May  3 14:30:54 mattlabs kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
May  3 14:30:54 mattlabs kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
May  3 14:30:54 mattlabs kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth04143fe: link becomes ready
May  3 14:30:54 mattlabs kernel: aqua0: port 1(veth04143fe) entered blocking state
May  3 14:30:54 mattlabs kernel: aqua0: port 1(veth04143fe) entered forwarding state
May  3 14:30:54 mattlabs NetworkManager[1002]: <info>  [1588509054.8667] device (veth04143fe): carrier: link connected
May  3 14:30:54 mattlabs NetworkManager[1002]: <info>  [1588509054.8676] device (aqua0): carrier: link connected
May  3 14:30:55 mattlabs sh: sendto: Operation not permitted
May  3 14:30:55 mattlabs sh: sendto: Operation not permitted
May  3 14:30:55 mattlabs sh: sendto: Operation not permitted
May  3 14:30:55 mattlabs sh: sendto: Operation not permitted
May  3 14:30:56 mattlabs containerd: time="2020-05-03T14:30:56.256034490+02:00" level=info msg="shim reaped" id=c48911ae7d7d10a1f815d7e53314bba92dd537d5b710e5018ac6d7fa0a2940f9
May  3 14:30:56 mattlabs dockerd: time="2020-05-03T14:30:56.266361783+02:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
May  3 14:30:56 mattlabs kernel: aqua0: port 1(veth04143fe) entered disabled state
May  3 14:30:56 mattlabs NetworkManager[1002]: <info>  [1588509056.3503] manager: (vethfe190a0): new Veth device (/org/freedesktop/NetworkManager/Devices/233)
May  3 14:30:56 mattlabs kernel: aqua0: port 1(veth04143fe) entered disabled state
May  3 14:30:56 mattlabs avahi-daemon[14690]: Withdrawing workstation service for vethfe190a0.
May  3 14:30:56 mattlabs kernel: device veth04143fe left promiscuous mode
May  3 14:30:56 mattlabs kernel: aqua0: port 1(veth04143fe) entered disabled state
May  3 14:30:56 mattlabs avahi-daemon[14690]: Withdrawing workstation service for veth04143fe.
May  3 14:30:56 mattlabs NetworkManager[1002]: <info>  [1588509056.4243] device (veth04143fe): released from master device aqua0
May  3 14:30:56 mattlabs kernel: XFS (dm-3): Unmounting Filesystem

Thanks for helping.

Matt

pagaille · May 3, 2020, 3:13pm

Solved. It was actually an oooold version of portainer that was cycling because of an unhandled database update error. I ended up cleaning everything by hand (docker rm portainer wasn’t enough) and restarting from scratch.