NethServer Version: 7.9.2009
Module: nethserver-docker (1.0.13)
Hi *,
I am a little lost regarding an issue since the last package update of Nethserver components. Whenever the adjust-firewall or another Docker-influencing command runs (e.g. signal-event nethserver-docker-update), all containers fail to restart. I need to restart the whole server in order to get them running again. I was able to extract the logs but looking into it, I don’t find any information which might be helpful. If anyone has experienced a similar issue or has any idea what might fix it, I’m glad for any help.
The log:
Jul 3 17:26:04 neth esmith::event[10107]: Event: nethserver-docker-update
Jul 3 17:26:04 neth esmith::event[10107]: Action: /etc/e-smith/events/nethserver-docker-update/S00initialize-default-databases SUCCESS [0.755103]
Jul 3 17:26:04 neth esmith::event[10107]: expanding /etc/docker/docker.conf
Jul 3 17:26:05 neth esmith::event[10107]: expanding /etc/yum.repos.d/docker-ce.repo
Jul 3 17:26:05 neth esmith::event[10107]: Action: /etc/e-smith/events/nethserver-docker-update/S10nethserver-docker-create-logs SUCCESS [0.070673]
Jul 3 17:26:05 neth esmith::event[10107]: Action: /etc/e-smith/events/nethserver-docker-update/S20nethserver-docker-create-aeria SUCCESS [0.136032]
Jul 3 17:26:05 neth esmith::event[10107]: Action: /etc/e-smith/events/nethserver-docker-update/S20nethserver-docker-create-network SUCCESS [0.242492]
Jul 3 17:26:05 neth esmith::event[10107]: Action: /etc/e-smith/events/nethserver-docker-update/S20nethserver-docker-macvlan-creation SUCCESS [0.310743]
Jul 3 17:26:06 neth esmith::event[10107]: [INFO] service docker restart
Jul 3 17:26:06 neth dockerd: time="2023-07-03T17:26:06.080411153+07:00" level=info msg="Processing signal 'terminated'"
Jul 3 17:26:06 neth dockerd: time="2023-07-03T17:26:06.124362108+07:00" level=info msg="Daemon shutdown complete"
Jul 3 17:26:06 neth dockerd: time="2023-07-03T17:26:06.124498594+07:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=moby
Jul 3 17:26:06 neth dockerd: time="2023-07-03T17:26:06.124506180+07:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
Jul 3 17:26:06 neth dockerd: time="2023-07-03T17:26:06.232153938+07:00" level=info msg="Starting up"
Jul 3 17:26:06 neth dockerd: time="2023-07-03T17:26:06.349981477+07:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Jul 3 17:26:06 neth dockerd: time="2023-07-03T17:26:06.727804422+07:00" level=info msg="Loading containers: start."
Jul 3 17:26:10 neth dockerd: time="2023-07-03T17:26:10.277222913+07:00" level=error msg="failed to populate fields for osl sandbox e320ae992b4092148874fe514cfd925e201230e5cc94f53195a8e5163421e0c6"
Jul 3 17:26:10 neth dockerd: time="2023-07-03T17:26:10.295711094+07:00" level=error msg="failed to populate fields for osl sandbox 0f90075be3dc051a52272e773ae535cff27e785df22322f696e3d8f03d9a29a4"
Jul 3 17:26:10 neth dockerd: time="2023-07-03T17:26:10.313221248+07:00" level=error msg="failed to populate fields for osl sandbox 24a42a2ac1757193e5218b5b23d6bc1ef0585a73895e94fc48c7d2ad12647568"
Jul 3 17:26:10 neth dockerd: time="2023-07-03T17:26:10.333370275+07:00" level=error msg="failed to populate fields for osl sandbox 5fe029cf27ce7feebc2191ad54597407bf7c78c624557c56b6d5f0d53e744c9b"
Jul 3 17:26:10 neth dockerd: time="2023-07-03T17:26:10.350685152+07:00" level=error msg="failed to populate fields for osl sandbox 7082050887f46469dca8f510f3d527332966994d90a03562339cb804135ae954"
Jul 3 17:26:10 neth dockerd: time="2023-07-03T17:26:10.368372837+07:00" level=error msg="failed to populate fields for osl sandbox 7400906eb4de1c1bd0cac990af5eadf3ae83b8c4c5d854ba4c7ec74a1a891d9e"
Jul 3 17:26:10 neth dockerd: time="2023-07-03T17:26:10.384680248+07:00" level=error msg="failed to populate fields for osl sandbox 82f48bc3bd9013abcef29e6eee534385539446c60801d72ba088a6c4d0e3d54c"
Jul 3 17:26:10 neth dockerd: time="2023-07-03T17:26:10.402194973+07:00" level=error msg="failed to populate fields for osl sandbox d0f49def6670374c2b85f3ff33fb37ddecc84654134fab5a6eae2abf46cfa208"
Jul 3 17:26:10 neth dockerd: time="2023-07-03T17:26:10.417747455+07:00" level=error msg="failed to populate fields for osl sandbox fa5bb0dfeae6d7b16480db5cb17bfee46cbf5b3c5c3211bdb65dfcb16505ac7e"
Jul 3 17:26:10 neth dockerd: time="2023-07-03T17:26:10.433758088+07:00" level=error msg="failed to populate fields for osl sandbox 05c0421b723746d37a5b85c02c4e7f7dce7911e2ee386221f23cf34b8a76ae43"
Jul 3 17:26:10 neth dockerd: time="2023-07-03T17:26:10.451926610+07:00" level=error msg="failed to populate fields for osl sandbox 33db774013e81906add5d51e409371f1cc58d8620b5e7f35c26cff281ed3db3b"
Jul 3 17:26:10 neth dockerd: time="2023-07-03T17:26:10.468246975+07:00" level=error msg="failed to populate fields for osl sandbox 3e8f798445c2eaa80286068ccee1504896c9390ebcb2634505994cc068268aba"
Jul 3 17:26:10 neth dockerd: time="2023-07-03T17:26:10.492881686+07:00" level=error msg="failed to populate fields for osl sandbox 687a0f7047ee11e46465619c078c775768ba271eb941d15d1186db18241490e9"
Jul 3 17:26:10 neth dockerd: time="2023-07-03T17:26:10.494025976+07:00" level=info msg="there are running containers, updated network configuration will not take affect"
Jul 3 17:26:10 neth dockerd: time="2023-07-03T17:26:10.550769415+07:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Jul 3 17:26:10 neth dockerd: time="2023-07-03T17:26:10.550807915+07:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Jul 3 17:26:10 neth dockerd: time="2023-07-03T17:26:10.976475053+07:00" level=info msg="Loading containers: done."
Jul 3 17:26:11 neth dockerd: time="2023-07-03T17:26:10.999856702+07:00" level=info msg="Docker daemon" commit=659604f graphdriver=overlay2 version=24.0.2
Jul 3 17:26:11 neth dockerd: time="2023-07-03T17:26:10.999953232+07:00" level=info msg="Daemon has completed initialization"
Jul 3 17:26:11 neth dockerd: time="2023-07-03T17:26:11.147260740+07:00" level=info msg="API listen on /var/run/docker.sock"
Jul 3 17:26:11 neth dockerd: time="2023-07-03T17:26:11.758743349+07:00" level=info msg="ignoring event" container=334f1804ea8ebcb91a10efbd0b1cb7ef4fbfa8f7f11fa01554b3e1a51fc4f820 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jul 3 17:26:11 neth dockerd: time="2023-07-03T17:26:11.773407190+07:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=334f1804ea8ebcb91a10efbd0b1cb7ef4fbfa8f7f11fa01554b3e1a51fc4f820 daemonShuttingDown=false error="restart canceled" execDuration=1h4m0.292362084s exitStatus="{0 2023-07-03 10:26:11.731082566 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
Jul 3 17:26:11 neth dockerd: time="2023-07-03T17:26:11.843023008+07:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Jul 3 17:26:11 neth dockerd: time="2023-07-03T17:26:11.843244092+07:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Jul 3 17:26:12 neth dockerCreateMacVlan0: 334f1804ea8e
Jul 3 17:26:13 neth dockerd: time="2023-07-03T17:26:13.809199056+07:00" level=info msg="ignoring event" container=3ae569a5eef102f8a35b6627b43c64178a4ab9155df6298cacc38c8a4280edca module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jul 3 17:26:13 neth dockerd: time="2023-07-03T17:26:13.828888949+07:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=3ae569a5eef102f8a35b6627b43c64178a4ab9155df6298cacc38c8a4280edca daemonShuttingDown=false error="restart canceled" execDuration=1h3m55.51871995s exitStatus="{130 2023-07-03 10:26:13.601572795 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
Jul 3 17:26:15 neth dockerd: time="2023-07-03T17:26:15.399867306+07:00" level=warning msg="macvlan driver does not support port mappings"
Jul 3 17:26:15 neth dockerd: time="2023-07-03T17:26:15.399898336+07:00" level=warning msg="macvlan driver does not support port exposures"
Jul 3 17:26:16 neth dockerd: time="2023-07-03T17:26:16.699459070+07:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Jul 3 17:26:16 neth dockerd: time="2023-07-03T17:26:16.699495368+07:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Jul 3 17:26:17 neth dockerd: time="2023-07-03T17:26:17.379221731+07:00" level=error msg="stream copy error: reading from a closed fifo"
Jul 3 17:26:17 neth dockerd: time="2023-07-03T17:26:17.379246851+07:00" level=error msg="stream copy error: reading from a closed fifo"
Jul 3 17:26:17 neth dockerd: time="2023-07-03T17:26:17.683213138+07:00" level=error msg="3ae569a5eef102f8a35b6627b43c64178a4ab9155df6298cacc38c8a4280edca cleanup: failed to delete container from containerd: container \"3ae569a5eef102f8a35b6627b43c64178a4ab9155df6298cacc38c8a4280edca\" in namespace \"moby\": not found"
Jul 3 17:26:17 neth dockerd: time="2023-07-03T17:26:17.893983104+07:00" level=error msg="Handler for POST /v1.43/containers/3ae569a5eef1/restart returned error: Cannot restart container 3ae569a5eef1: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: failed to add interface veth50cb11d to sandbox: failed to set link up: address already in use: unknown"
Jul 3 17:26:17 neth dockerCreateMacVlan0: Error response from daemon: Cannot restart container 3ae569a5eef1: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: failed to add interface veth50cb11d to sandbox: failed to set link up: address already in use: unknown
Jul 3 17:26:17 neth systemd: docker.service: control process exited, code=exited status=1
Jul 3 17:26:17 neth dockerd: time="2023-07-03T17:26:17.899513782+07:00" level=info msg="Processing signal 'terminated'"
Jul 3 17:26:17 neth dockerd: time="2023-07-03T17:26:17.941657963+07:00" level=info msg="Daemon shutdown complete"
Jul 3 17:26:17 neth systemd: Unit docker.service entered failed state.
Jul 3 17:26:17 neth systemd: docker.service failed.
Jul 3 17:26:17 neth esmith::event[10107]: Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.