pagaille
June 1, 2020, 6:28pm
61
… Or not.
pihole just crashed. I’ll put in on the aqua network. There is definitively something not kasher with the DHCP plugin.
Jun 1 20:22:04 mattlabs NetworkManager[838]: <info> [1591035724.9146] manager: (21640da87998-dh): new Veth device (/org/freedesktop/NetworkManager/Devices/151)
Jun 1 20:22:04 mattlabs NetworkManager[838]: <info> [1591035724.9167] manager: (dh-21640da87998): new Veth device (/org/freedesktop/NetworkManager/Devices/152)
Jun 1 20:22:15 mattlabs avahi-daemon[854]: Withdrawing workstation service for dh-21640da87998.
Jun 1 20:22:15 mattlabs avahi-daemon[854]: Withdrawing workstation service for 21640da87998-dh.
Jun 1 20:22:15 mattlabs dockerd: time="2020-06-01T20:22:15+02:00" level=error msg="<local> - - [01/Jun/2020 18:22:15] \"POST /NetworkDriver.CreateEndpoint HTTP/1.1\" 0 -" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:15 mattlabs dockerd: time="2020-06-01T20:22:15.580091747+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554/net-dhcp.sock/NetworkDriver.CreateEndpoint: Post http://%2Frun%2Fdocker%2Fplugins%2Fa683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554%2Fnet-dhcp.sock/NetworkDriver.CreateEndpoint: net/http: HTTP/1.x transport connection broken: malformed HTTP status code \"0\", retrying in 1s"
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg="400 Bad Request: The browser (or proxy) sent a request that this server could not understand." plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg="Traceback (most recent call last):" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" File \"/usr/local/lib/python3.7/site-packages/werkzeug/wrappers/json.py\", line 119, in get_json" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" rv = self.json_module.loads(data)" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" File \"/usr/local/lib/python3.7/site-packages/flask/json/__init__.py\", line 253, in loads" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" return _json.loads(s, **kwargs)" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" File \"/usr/local/lib/python3.7/json/__init__.py\", line 361, in loads" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" return cls(**kw).decode(s)" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" File \"/usr/local/lib/python3.7/json/decoder.py\", line 337, in decode" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" obj, end = self.raw_decode(s, idx=_w(s, 0).end())" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" File \"/usr/local/lib/python3.7/json/decoder.py\", line 355, in raw_decode" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" raise JSONDecodeError(\"Expecting value\", s, err.value) from None" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg="json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg="During handling of the above exception, another exception occurred:" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg="Traceback (most recent call last):" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" File \"/usr/local/lib/python3.7/site-packages/flask/app.py\", line 1949, in full_dispatch_request" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" rv = self.dispatch_request()" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" File \"/usr/local/lib/python3.7/site-packages/flask/app.py\", line 1935, in dispatch_request" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" return self.view_functions[rule.endpoint](**req.view_args)" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" File \"/opt/plugin/net_dhcp/network.py\", line 134, in create_endpoint" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" req = request.get_json(force=True)" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" File \"/usr/local/lib/python3.7/site-packages/werkzeug/wrappers/json.py\", line 128, in get_json" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" rv = self.on_json_loading_failed(e)" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" File \"/usr/local/lib/python3.7/site-packages/flask/wrappers.py\", line 27, in on_json_loading_failed" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg=" raise BadRequest()" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg="werkzeug.exceptions.BadRequest: 400 Bad Request: The browser (or proxy) sent a request that this server could not understand." plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16+02:00" level=error msg="<local> - - [01/Jun/2020 18:22:16] \"POST /NetworkDriver.CreateEndpoint HTTP/1.1\" 500 -" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
Jun 1 20:22:16 mattlabs dockerd: time="2020-06-01T20:22:16.701214975+02:00" level=warning msg="error locating sandbox id 6770c06611b0e9a7cc3b745e5f554a0454d972abb300870e29c12b73f0741a34: sandbox 6770c06611b0e9a7cc3b745e5f554a0454d972abb300870e29c12b73f0741a34 not found"
Jun 1 20:22:16 mattlabs kernel: XFS (dm-3): Unmounting Filesystem
Jun 1 20:22:17 mattlabs dockerd: time="2020-06-01T20:22:17.403769275+02:00" level=error msg="50c149a17d3cf01133270162dd542090b7962615442687829d55876d0f5c28b0 cleanup: failed to delete container from containerd: no such container"
Jun 1 20:22:17 mattlabs dockerd: time="2020-06-01T20:22:17.517888697+02:00" level=error msg="restartmanger wait error: failed to create endpoint pihole on network aeria: NetworkDriver.CreateEndpoint: 400 Bad Request: The browser (or proxy) sent a request that this server could not understand."
stephdl
June 1, 2020, 6:36pm
62
we could patch the systemd docker service to use ExecStartPost with a script, check the plugin is enable, if disabled then enabled it.
Tricky but it should work…
BTW, does it is reproducible before to code ???
pagaille
June 1, 2020, 6:49pm
63
I think so. I had the problem yesterday and ended up cleaning everything, therefore the update was applied again leading to the same result.
However I’m wondering if the DHCP plugin is compatible with the latest version of docker : see my last report, it is far from reliable (containers are stopping after some time and this is clearly linked to the plugin).
1 Like
stephdl
June 1, 2020, 6:52pm
64
I crashed my docker after the update tonight
stephdl
June 1, 2020, 7:26pm
65
Not sure the plugin is responsible why the update failed, I can see that we have a rpm from ce-base with errors.
Could you reproduce ?
=====================================================================================================================================
Package Arch Version Repository Size
=====================================================================================================================================
Updating:
containerd.io x86_64 1.2.13-3.2.el7 docker-ce-stable 25 M
docker-ce x86_64 3:19.03.11-3.el7 docker-ce-stable 24 M
docker-ce-cli x86_64 1:19.03.11-3.el7 docker-ce-stable 38 M
python-backports-ssl_match_hostname noarch 3.5.0.1-1.el7 ce-base 13 k
Transaction Summary
=====================================================================================================================================
Upgrade 4 Packages
Total download size: 88 M
Downloading packages:
No Presto metadata available for docker-ce-stable
No Presto metadata available for ce-base
(1/4): containerd.io-1.2.13-3.2.el7.x86_64.rpm | 25 MB 00:00:02
(2/4): docker-ce-19.03.11-3.el7.x86_64.rpm | 24 MB 00:00:02
(3/4): python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch.rpm | 13 kB 00:00:00
(4/4): docker-ce-cli-19.03.11-3.el7.x86_64.rpm | 38 MB 00:00:01
-------------------------------------------------------------------------------------------------------------------------------------
Total 21 MB/s | 88 MB 00:00:04
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : containerd.io-1.2.13-3.2.el7.x86_64 1/8
Updating : 1:docker-ce-cli-19.03.11-3.el7.x86_64 2/8
Updating : 3:docker-ce-19.03.11-3.el7.x86_64 3/8
Updating : python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch 4/8
Error unpacking rpm package python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch
error: unpacking of archive failed on file /usr/lib/python2.7/site-packages/backports.ssl_match_hostname-3.5.0.1-py2.7.egg-info: cpio: rename
error: python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch: install failed
Cleanup : 3:docker-ce-19.03.8-3.el7.x86_64 5/8
Cleanup : containerd.io-1.2.6-3.3.el7.x86_64 6/8
Cleanup : 1:docker-ce-cli-19.03.8-3.el7.x86_64 7/8
error: python-backports-ssl_match_hostname-3.4.0.2-4.el7.noarch: erase skipped
Verifying : 1:docker-ce-cli-19.03.11-3.el7.x86_64 1/8
Verifying : containerd.io-1.2.13-3.2.el7.x86_64 2/8
Verifying : 3:docker-ce-19.03.11-3.el7.x86_64 3/8
python-backports-ssl_match_hostname-3.4.0.2-4.el7.noarch was supposed to be removed but is not!
Verifying : python-backports-ssl_match_hostname-3.4.0.2-4.el7.noarch 4/8
Verifying : 1:docker-ce-cli-19.03.8-3.el7.x86_64 5/8
Verifying : containerd.io-1.2.6-3.3.el7.x86_64 6/8
Verifying : 3:docker-ce-19.03.8-3.el7.x86_64 7/8
Verifying : python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch 8/8
Updated:
containerd.io.x86_64 0:1.2.13-3.2.el7 docker-ce.x86_64 3:19.03.11-3.el7 docker-ce-cli.x86_64 1:19.03.11-3.el7
Failed:
python-backports-ssl_match_hostname.noarch 0:3.4.0.2-4.el7 python-backports-ssl_match_hostname.noarch 0:3.5.0.1-1.el7
Complete!
[root@NS7 ~]# docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
I must restart the server if I want the service UP
Once the server restarted, the plugin is disabled
[root@NS7 ~]# docker plugin enable devplayer0/net-dhcp
devplayer0/net-dhcp
once the plugin is enabled, then pihole can be started, or created again.
1 Like
pagaille
June 1, 2020, 8:26pm
66
Nope, no error like this on my side.
I also was able to start the aqua-linked containers again after reboot, and Aeria-linked after enabling the plugin but I remarked multiple occurence (every 60s) of messages seemingly related to the API :
**Jun 1 20:16** :40 mattlabs dockerd: time="2020-06-01T20:16:40.296752070+02:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
**Jun 1 20:16** :40 mattlabs dockerd: time="2020-06-01T20:16:40+02:00" level=error msg="<local> - - [01/Jun/2020 18:16:40] \"POST /NetworkDriver.RevokeExternalConnectivity HTTP/1.1\" 404 -" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
**Jun 1 20:16** :40 mattlabs dockerd: time="2020-06-01T20:16:40+02:00" level=error msg="<local> - - [01/Jun/2020 18:16:40] \"POST /NetworkDriver.Leave HTTP/1.1\" 200 -" plugin=a683e7c66cf7e39e855fbc9f91e2f195f78a8eeb9cfb7495f9d30de28e6dc554
And after some times the container stopped.
stephdl
June 2, 2020, 2:10pm
67
I found your post about devplayer0/net-dhcp:latest on docker.com , and I must admit I retain support lesson, use a default network, not a custom plugin. This plugin is nice, but probably not maintained anymore, or maybe soon…who knows.
The upgrade to docker
containerd.io x86_64 1.2.13-3.2.el7
docker-ce x86_64 3:19.03.11-3.el7
docker-ce-cli x86_64 1:19.03.11-3.el7
It breaks the docker service once aeria is created, a restart of the server is needed. If you update directly to the version above, you can install the plugin, it works. Definitely there is something wrong.
For now it is still experimental so I have no problem to go back, even for what I have in mind I have a spare wheel that I would share.
MACVLAN
A bit complicated to make it works, but the goal is the same, get an IP a real network
1 Like
pagaille
June 2, 2020, 3:27pm
68
I agree. MACVLAN was also in my plans. And I’m sure that somehow we’ll be able to make it work by simply issuing yum install nethserver-docker
bwdjames
June 2, 2020, 3:59pm
69
Just wondering if this would also work on a the Raspberry Pi version of NethServer?
Being a little cheeky with this one…
1 Like
stephdl
June 2, 2020, 4:02pm
70
it seems that the host cannot ping/reach the container, a normal behaviour for macvlan… its a shame
mark_nl
June 2, 2020, 4:55pm
71
No __ Maybe ??
There are no stable containers available for centos:…https://download.docker.com/linux/centos/7/armv7l/
A Rasbian container might work:https://download.docker.com/linux/raspbian/
bwdjames
June 2, 2020, 5:27pm
72
Thanks @mark_nl , I was a little unsure.
And flipping heck! For the nightly build, the last one was 21st January 2020… Somehow I don’t think that it can be classed as nightly…
Would a Rasbian container work on Centos?
mark_nl
June 2, 2020, 5:34pm
73
After a bit more investigation, this module won’t work on the arm32 bit;
docker-ce (dependency of nethserver-docker) is missing…
2 Likes
bwdjames
June 2, 2020, 5:39pm
74
Thanks for checking @mark_nl . Bit of a pity…
mrmarkuz
June 2, 2020, 9:00pm
75
I hope this solves the plugin issue:
devplayer0:master ← devplayer0:golang
opened 03:39PM - 19 May 20 UTC
I'm gonna try rewriting this in Go, hopefully should make everything a bit more … stable. Might finally get around to checking out #2 too.
Fixes #13.
How long did it take until it stopped?
Did you set a restart policy for the container?
It can be done on cli with --restart=unless-stopped or with portainer in the “Container details”.
1 Like
stephdl
June 2, 2020, 9:05pm
76
I am about to add a new network macvlan based, all previous network will continue to exist, the sysadmin will make a choice
2 Likes
pagaille
June 3, 2020, 7:54am
77
I don’t remember. I few hours I believe. Do you also notice the error messages that shows every 60s in the logs ?
Anyway, I agree with @stephdl : we currently can’t rely on this plugin, at least for mission critical containers like a DNS server.
giacomo
June 3, 2020, 8:01am
78
Probably, the RPM has been uploaded before the comps merge.
@davidep has already fixed it (it will be synced soon to all mirrors)!
3 Likes
pagaille
June 3, 2020, 9:02am
79
Referring to your script (https://github.com/stephdl/nethserver-pihole/blob/ns7/root/etc/e-smith/events/actions/nethserver-pihole-docker-creation )
Why do you map all those ports to the host ?
-p 53:53/tcp \
-p 53:53/udp \
-p 80:80/tcp \
-p 443:443/tcp \
-p 67:67/udp \
Port 53 is obviously taken by dnsmasq on the NS host…
stephdl
June 3, 2020, 12:33pm
80
still in dev, indeed I do not need to map the port because it is on a specific IP, I think I can remove it
there is something I don’t get here