The system log in NS8 is empty, regardless of the time period. I can’t say since when this has been the case, as I first took care of the migration from NS7 to NS8 and the configuration of NS8. Now I wanted to have a look at the logs and realized that everything is empty.
Is it possible to fix this, I have just finished the migration to NS8?
To gather more information, in the leader node, get the list of running modules:
loginctl list-users
A loki1 should be among them. Check the status of Loki, which is the backend of the System logs page; it collects logs from all cluster nodes.
runagent -m loki1 systemctl status --user loki-server
runagent -m loki1 podman ps
Then check the status of the node promtail.service, which is a service running on every node; it sends the node journal to the central Loki instance.
systemctl status promtail
The system journal can still have some information
journalctl _UID=$(id -u loki1)
journalctl -u promtail.service
More log insights: Logs | NS8 dev manual
2 Likes
loginctl list-users
UID USER LINGER STATE
0 root no active
1001 traefik1 yes lingering
1002 loki1 yes lingering
1003 ldapproxy1 yes lingering
1004 webtop1 yes lingering
1005 mail1 yes lingering
1006 imapsync1 yes lingering
1007 nextcloud1 yes lingering
1008 samba1 yes lingering
1009 roundcubemail1 yes lingering
1010 webserver1 yes lingering
1011 collabora1 yes lingering
12 users listed.
runagent -m loki1 systemctl status --user loki
loki.service - Loki pod service
Loaded: loaded (/home/loki1/.config/systemd/user/loki.service; enabled; preset: disabled)
Active: failed (Result: start-limit-hit) since Wed 2024-04-03 08:33:26 CEST; 1 day 1h ago
Duration: 1.059s
Process: 5907 ExecStartPre=/bin/rm -f /run/user/1002/loki.pid /run/user/1002/loki.pod-id (code=exited, status=0/SUCCESS)
Process: 5908 ExecStartPre=/usr/bin/podman pod create --infra-conmon-pidfile /run/user/1002/loki.pid --pod-id-file /run/user/1002/loki.pod-id --name loki --publish ${LOKI_HTTP_PORT}:${LOKI_HTT>
Process: 5918 ExecStart=/usr/bin/podman pod start --pod-id-file /run/user/1002/loki.pod-id (code=exited, status=0/SUCCESS)
Process: 6077 ExecStop=/usr/bin/podman pod stop --ignore --pod-id-file /run/user/1002/loki.pod-id -t 10 (code=exited, status=0/SUCCESS)
Process: 6096 ExecStopPost=/usr/bin/podman pod rm --ignore -f --pod-id-file /run/user/1002/loki.pod-id (code=exited, status=0/SUCCESS)
Main PID: 5948 (code=exited, status=0/SUCCESS)
CPU: 189ms
runagent -m loki1 podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
systemctl status promtail
promtail.service - Promtail logs collector for Loki
Loaded: loaded (/etc/systemd/system/promtail.service; enabled; preset: disabled)
Active: active (running) since Wed 2024-04-03 08:32:58 CEST; 1 day 1h ago
Process: 1874 ExecStartPre=/bin/rm -f /run/promtail.pid /run/promtail.cid (code=exited, status=0/SUCCESS)
Process: 1886 ExecStartPre=mkdir -vp promtail (code=exited, status=0/SUCCESS)
Process: 1887 ExecStartPre=/usr/local/bin/runagent -m node /var/lib/nethserver/node/bin/generate-promtail-config (code=exited, status=0/SUCCESS)
Process: 2180 ExecStart=/usr/bin/podman run --conmon-pidfile=/run/promtail.pid --cidfile=/run/promtail.cid --cgroups=no-conmon --replace --name=promtail --privileged --network=host --detach -->
Main PID: 2263 (conmon)
Tasks: 1 (limit: 100458)
Memory: 2.2M
CPU: 1.108s
CGroup: /system.slice/promtail.service
└─2263 /usr/bin/conmon --api-version 1 -c f44d2ff3e13374134afac2e8f5f8da539c89e134701a4d51e25573f5fd5fec53 -u f44d2ff3e13374134afac2e8f5f8da539c89e134701a4d51e25573f5fd5fec53 -r /usr/>
Apr 04 09:44:59 nethserver8 promtail[2263]: level=warn ts=2024-04-04T07:44:59.852412625Z caller=client.go:419 component=client host=10.5.4.1:20001 msg="error sending batch, will retry" status=-1 t>
Apr 04 09:46:24 nethserver8 promtail[2263]: level=warn ts=2024-04-04T07:46:24.348147834Z caller=client.go:419 component=client host=10.5.4.1:20001 msg="error sending batch, will retry" status=-1 t>
Apr 04 09:48:59 nethserver8 promtail[2263]: level=warn ts=2024-04-04T07:48:59.299394008Z caller=client.go:419 component=client host=10.5.4.1:20001 msg="error sending batch, will retry" status=-1 t>
Apr 04 09:48:59 nethserver8 promtail[2263]: level=error ts=2024-04-04T07:48:59.299495261Z caller=client.go:430 component=client host=10.5.4.1:20001 msg="final error sending batch" status=-1 tenant>
Apr 04 09:48:59 nethserver8 promtail[2263]: level=warn ts=2024-04-04T07:48:59.332753091Z caller=client.go:419 component=client host=10.5.4.1:20001 msg="error sending batch, will retry" status=-1 t>
Apr 04 09:49:00 nethserver8 promtail[2263]: level=warn ts=2024-04-04T07:49:00.144236505Z caller=client.go:419 component=client host=10.5.4.1:20001 msg="error sending batch, will retry" status=-1 t>
Apr 04 09:49:01 nethserver8 promtail[2263]: level=warn ts=2024-04-04T07:49:01.351495124Z caller=client.go:419 component=client host=10.5.4.1:20001 msg="error sending batch, will retry" status=-1 t>
Apr 04 09:49:03 nethserver8 promtail[2263]: level=warn ts=2024-04-04T07:49:03.536103392Z caller=client.go:419 component=client host=10.5.4.1:20001 msg="error sending batch, will retry" status=-1 t>
Apr 04 09:49:07 nethserver8 promtail[2263]: level=warn ts=2024-04-04T07:49:07.92934567Z caller=client.go:419 component=client host=10.5.4.1:20001 msg="error sending batch, will retry" status=-1 te>
Apr 04 09:49:22 nethserver8 promtail[2263]: level=warn ts=2024-04-04T07:49:22.644602914Z caller=client.go:419 component=client host=10.5.4.1:20001 msg="error sending batch, will retry" status=-1 t>
lines 1-24/24 (END)
journalctl _UID=$(id -u loki1)
Apr 04 06:10:32 nethserver8 systemd[1876]: backup3.service: Main process exited, code=exited, status=1/FAILURE
Apr 04 06:10:32 nethserver8 systemd[1876]: backup3.service: Failed with result 'exit-code'.
Apr 04 06:10:32 nethserver8 systemd[1876]: Failed to start Sicherung nach Backup 2 SMB.
Apr 04 06:10:32 nethserver8 systemd[1876]: backup3.service: Consumed 1.421s CPU time.
Apr 04 08:38:12 nethserver8 systemd[1876]: Starting Cleanup of User's Temporary Files and Directories...
Apr 04 08:38:12 nethserver8 systemd[1876]: Finished Cleanup of User's Temporary Files and Directories.
Apr 04 09:23:30 nethserver8 systemd[1876]: Started podman-176838.scope.
Apr 04 09:35:49 nethserver8 systemd[1876]: Started podman-180436.scope.
journalctl -u promtail.service
Apr 04 09:12:54 nethserver8 promtail[2263]: level=warn ts=2024-04-04T07:12:54.476445109Z caller=client.go:419 component=client host=10.5.4.1:20001 msg="error sending batch, will retry" status=-1 t>
...
Apr 04 09:39:52 nethserver8 promtail[2263]: level=warn ts=2024-04-04T07:39:52.830666521Z caller=client.go:419 component=client host=10.5.4.1:20001 msg="error sending batch, will retry" status=-1 t>
There seems to be a problem with the promtail.service?
Just a little hint, I also have Cockpit installed, could it have something to do with that? Logs are displayed in Cockpit.
It seems the Pod is down. Let’s try to start it again
runagent -m loki1 systemctl start --user loki
My previous check command wasn’t correct. Fixed as it should be:
runagent -m loki1 systemctl status --user loki-server
Promtail can’t contact Loki. The problem seems to be on the Loki’s side.
Cockpit shows the system journal. Depending on the distro defaults it can be stored in the disk or in memory.
runagent -m loki1 systemctl start --user loki
runagent -m loki1 systemctl status --user loki-server
loki-server.service - Loki server
Loaded: loaded (/home/loki1/.config/systemd/user/loki-server.service; disabled; preset: disabled)
Active: inactive (dead) since Thu 2024-04-04 11:48:02 CEST; 547ms ago
Duration: 308ms
Process: 208533 ExecStartPre=/bin/rm -f /run/user/1002/loki-server.pid /run/user/1002/loki-server.ctr-id (code=exited, status=0/SUCCESS)
Process: 208535 ExecStart=/usr/bin/podman run --detach --conmon-pidfile=/run/user/1002/loki-server.pid --cidfile=/run/user/1002/loki-server.ctr-id --cgroups=no-conmon --pod-id-file /run/user/1002/loki.> Process: 208578 ExecStop=/usr/bin/podman stop --ignore --cidfile /run/user/1002/loki-server.ctr-id -t 10 (code=exited, status=0/SUCCESS)
Process: 208609 ExecStopPost=/usr/bin/podman rm --ignore -f --cidfile /run/user/1002/loki-server.ctr-id (code=exited, status=0/SUCCESS)
Main PID: 208557 (code=exited, status=0/SUCCESS)
CPU: 167ms
lines 1-10/10 (END)
The loki-server unit is in a crash loop. There should be some evidence in the journal, search in reverse order with
journalctl -r _UID=$(id -u loki1)
or no filter at all, (reversed time order)
journalctl -r
Apr 04 11:53:26 nethserver8 systemd[1876]: traefik.service: Job traefik.service/start failed with result 'dependency'.
Apr 04 11:53:26 nethserver8 systemd[1876]: Dependency failed for Loki frontend HTTP proxy.
Apr 04 11:53:26 nethserver8 systemd[1876]: loki-server.service: Job loki-server.service/start failed with result 'dependency'.
Apr 04 11:53:26 nethserver8 systemd[1876]: Dependency failed for Loki server.
Apr 04 11:53:26 nethserver8 systemd[1876]: Failed to start Loki pod service.
Apr 04 11:53:26 nethserver8 systemd[1876]: loki.service: Failed with result 'start-limit-hit'.
Apr 04 11:53:26 nethserver8 systemd[1876]: loki.service: Start request repeated too quickly.
Apr 04 11:53:26 nethserver8 systemd[1876]: Stopped Loki pod service.
Apr 04 11:53:26 nethserver8 podman[210411]: 4795c30b9b894c5ecb4625aaf28d5b12b2be404db34fe6e3adbfe6a916661b48
Apr 04 11:53:26 nethserver8 systemd[1876]: Removed slice cgroup user-libpod_pod_4795c30b9b894c5ecb4625aaf28d5b12b2be404db34fe6e3adbfe6a916661b48.slice.
Apr 04 11:53:26 nethserver8 podman[210392]: 4795c30b9b894c5ecb4625aaf28d5b12b2be404db34fe6e3adbfe6a916661b48
Apr 04 11:53:26 nethserver8 systemd[1876]: Stopping Loki pod service...
Apr 04 11:53:26 nethserver8 systemd[1876]: Stopped Loki server.
Apr 04 11:53:26 nethserver8 podman[210382]: 98956d1f242022ebcf8ba934548d4cf81935f517e10c5f03c6563263c697798e
Apr 04 11:53:26 nethserver8 podman[210363]: 98956d1f242022ebcf8ba934548d4cf81935f517e10c5f03c6563263c697798e
Apr 04 11:53:25 nethserver8 loki-server[210340]: level=error ts=2024-04-04T09:53:25.666556585Z caller=mapper.go:42 msg="unable to read rules directory" path=/loki/rules-temp err="open /loki/rules->
Apr 04 11:53:25 nethserver8 systemd[1876]: Stopped Loki frontend HTTP proxy.
Apr 04 11:53:25 nethserver8 systemd[1876]: Stopping Loki server...
Apr 04 11:53:25 nethserver8 systemd[1876]: traefik.service: Scheduled restart job, restart counter is at 28.
Apr 04 11:53:25 nethserver8 loki-server[210340]: level=error ts=2024-04-04T09:53:25.372210947Z caller=mapper.go:42 msg="unable to read rules directory" path=/loki/rules-temp err="open /loki/rules->
Apr 04 11:53:25 nethserver8 systemd[1876]: Failed to start Loki frontend HTTP proxy.
Apr 04 11:53:25 nethserver8 systemd[1876]: traefik.service: Failed with result 'exit-code'.
Apr 04 11:53:25 nethserver8 systemd[1876]: traefik.service: Control process exited, code=exited, status=125/n/a
Apr 04 11:53:25 nethserver8 podman[210328]: Error: removing storage for container "traefik": unmounting "/home/loki1/.local/share/containers/storage/overlay/0713907cbc9254eb6a76a4a184ff4d0d5d8af2e>
Apr 04 11:53:25 nethserver8 podman[210328]: time="2024-04-04T11:53:25+02:00" level=warning msg="Unmounting container \"traefik\" while attempting to delete storage: unmounting \"/home/loki1/.local>
Apr 04 11:53:25 nethserver8 systemd[1876]: Started Loki server.
Apr 04 11:53:25 nethserver8 podman[210319]: 98956d1f242022ebcf8ba934548d4cf81935f517e10c5f03c6563263c697798e
Apr 04 11:53:25 nethserver8 systemd[1876]: Started libcrun container.
Apr 04 11:53:25 nethserver8 systemd[1876]: Starting Loki frontend HTTP proxy...
Apr 04 11:53:25 nethserver8 systemd[1876]: Starting Loki server...
Apr 04 11:53:25 nethserver8 systemd[1876]: Started Loki pod service.
Apr 04 11:53:25 nethserver8 podman[210283]: 4795c30b9b894c5ecb4625aaf28d5b12b2be404db34fe6e3adbfe6a916661b48
Apr 04 11:53:25 nethserver8 systemd[1876]: Started libcrun container.
Apr 04 11:53:25 nethserver8 podman[210270]: 4795c30b9b894c5ecb4625aaf28d5b12b2be404db34fe6e3adbfe6a916661b48
Apr 04 11:53:25 nethserver8 systemd[1876]: Created slice cgroup user-libpod_pod_4795c30b9b894c5ecb4625aaf28d5b12b2be404db34fe6e3adbfe6a916661b48.slice.
Apr 04 11:53:25 nethserver8 systemd[1876]: Starting Loki pod service...
Apr 04 11:53:25 nethserver8 systemd[1876]: Stopped Loki pod service.
Apr 04 11:53:25 nethserver8 podman[210258]: 230b5dafc3cf1314af5ad8c6bcc65d914802dc5bd63414c8187ff7412037bec9
Apr 04 11:53:25 nethserver8 systemd[1876]: Removed slice cgroup user-libpod_pod_230b5dafc3cf1314af5ad8c6bcc65d914802dc5bd63414c8187ff7412037bec9.slice.
Apr 04 11:53:25 nethserver8 podman[210234]: 230b5dafc3cf1314af5ad8c6bcc65d914802dc5bd63414c8187ff7412037bec9
Apr 04 11:53:25 nethserver8 systemd[1876]: Stopping Loki pod service...
Apr 04 11:53:25 nethserver8 systemd[1876]: Stopped Loki server.
Apr 04 11:53:25 nethserver8 podman[210224]: a28f61cc07faaa608d46fdff2218b2772bcd18fb6aa52d73c7484b10a9cdf804
Apr 04 11:53:24 nethserver8 podman[210203]: a28f61cc07faaa608d46fdff2218b2772bcd18fb6aa52d73c7484b10a9cdf804
Apr 04 11:53:24 nethserver8 loki-server[210183]: level=error ts=2024-04-04T09:53:24.416623227Z caller=mapper.go:42 msg="unable to read rules directory" path=/loki/rules-temp err="open /loki/rules->
Apr 04 11:53:24 nethserver8 systemd[1876]: Stopped Loki frontend HTTP proxy.
Apr 04 11:53:24 nethserver8 systemd[1876]: Stopping Loki server...
Apr 04 11:53:24 nethserver8 systemd[1876]: traefik.service: Scheduled restart job, restart counter is at 27.
Apr 04 11:53:24 nethserver8 loki-server[210183]: level=error ts=2024-04-04T09:53:24.270888299Z caller=mapper.go:42 msg="unable to read rules directory" path=/loki/rules-temp err="open /loki/rules->
Apr 04 11:53:24 nethserver8 systemd[1876]: Failed to start Loki frontend HTTP proxy.
Apr 04 11:53:24 nethserver8 systemd[1876]: traefik.service: Failed with result 'exit-code'.
Apr 04 11:53:24 nethserver8 systemd[1876]: traefik.service: Control process exited, code=exited, status=125/n/a
Apr 04 11:53:24 nethserver8 podman[210170]: Error: removing storage for container "traefik": unmounting "/home/loki1/.local/share/containers/storage/overlay/0713907cbc9254eb6a76a4a184ff4d0d5d8af2e>
Apr 04 11:53:24 nethserver8 podman[210170]: time="2024-04-04T11:53:24+02:00" level=warning msg="Unmounting container \"traefik\" while attempting to delete storage: unmounting \"/home/loki1/.local>
Apr 04 11:53:24 nethserver8 systemd[1876]: Started Loki server.
Apr 04 11:53:24 nethserver8 podman[210161]: a28f61cc07faaa608d46fdff2218b2772bcd18fb6aa52d73c7484b10a9cdf804
Apr 04 11:53:24 nethserver8 systemd[1876]: Started libcrun container.
Apr 04 11:53:24 nethserver8 systemd[1876]: Starting Loki frontend HTTP proxy...
Apr 04 11:53:24 nethserver8 systemd[1876]: Starting Loki server...
Apr 04 11:53:24 nethserver8 systemd[1876]: Started Loki pod service.
Apr 04 11:53:24 nethserver8 podman[210125]: 230b5dafc3cf1314af5ad8c6bcc65d914802dc5bd63414c8187ff7412037bec9
Apr 04 11:53:24 nethserver8 systemd[1876]: Started libcrun container.
Apr 04 11:53:24 nethserver8 podman[210115]: 230b5dafc3cf1314af5ad8c6bcc65d914802dc5bd63414c8187ff7412037bec9
Apr 04 11:53:24 nethserver8 systemd[1876]: Created slice cgroup user-libpod_pod_230b5dafc3cf1314af5ad8c6bcc65d914802dc5bd63414c8187ff7412037bec9.slice.
Apr 04 11:53:24 nethserver8 systemd[1876]: Starting Loki pod service...
Apr 04 11:53:24 nethserver8 systemd[1876]: Stopped Loki pod service.
Apr 04 11:53:24 nethserver8 podman[210104]: 0369eb1627fddad1f0c3d0fe37ca6c02e6684feffee27d75171ff61b44c35caa
Apr 04 11:53:24 nethserver8 systemd[1876]: Removed slice cgroup user-libpod_pod_0369eb1627fddad1f0c3d0fe37ca6c02e6684feffee27d75171ff61b44c35caa.slice.
Apr 04 11:53:24 nethserver8 podman[210085]: 0369eb1627fddad1f0c3d0fe37ca6c02e6684feffee27d75171ff61b44c35caa
Apr 04 11:53:24 nethserver8 conmon[210006]: conmon 265e8ff5b48720d7fd36 <nwarn>: Failed to open cgroups file: /sys/fs/cgroup/user.slice/user-1002.slice/user@1002.service/user.slice/user-libpod_pod>
Apr 04 11:53:23 nethserver8 systemd[1876]: Stopping Loki pod service...
I don’t know why this happen. If you can download again ~150MB of images I just propose the following workaround.
First of all, enter the loki1 application environment
runagent -m loki1
Then run the following commands
podman volume export -o loki-server-data.tar loki-server-data
podman system reset -f
podman volume create loki-server-data
podman volume import loki-server-data loki-server-data.tar
systemctl --user start loki
Then if this shows that three containers are running, go to the Logs page and try again searching.
podman ps
runagent -m loki1
runagent: [INFO] starting bash -l
runagent: [INFO] working directory: /home/loki1/.config/state
[loki1@nethserver8 state]$ podman volume export -o loki-server-data.tar loki-server-data
[loki1@nethserver8 state]$ podman system reset -f
ERRO[0000] no container with name or ID “193be43d1312bbbdd4d856ca6c9e17d1d972028247faa078f2f29cb08c6b1690” found: no such container
ERRO[0000] 2 errors occurred:
* image used by 193be43d1312bbbdd4d856ca6c9e17d1d972028247faa078f2f29cb08c6b1690: image is in use by a container: consider listing external containers and force-removing image
* image used by 62788ef441948c541c25b7859de16d646be1aab4e81592f26ebd3e55699e5a08: image is in use by a container: consider listing external containers and force-removing image
[loki1@nethserver8 state]$ podman volume create loki-server-data
Error: volume with name loki-server-data already exists: volume already exists
[loki1@nethserver8 state]$ podman volume import loki-server-data loki-server-data.tar
[loki1@nethserver8 state]$ systemctl --user start loki
[loki1@nethserver8 state]$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[loki1@nethserver8 state]$
Always in
runagent -m loki1
As the error message suggests, let’s try to force-removing the images
podman rmi -f -a
Then execute again
podman system reset -f
podman volume create loki-server-data
podman volume import loki-server-data loki-server-data.tar
systemctl --user start loki
This worked, unfortunately the log is still empty.
runagent -m loki1
runagent: [INFO] starting bash -l
runagent: [INFO] working directory: /home/loki1/.config/state
[loki1@nethserver8 state]$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f4319cc8153e localhost/podman-pause:4.6.1-1709719721 10 minutes ago Up 10 minutes 0.0.0.0:20001->20001/tcp d2369bd1427e-infra
d913babf32af docker.io/grafana/loki:2.2.1 -config.file=/etc… 10 minutes ago Up 10 minutes 0.0.0.0:20001->20001/tcp loki-server
6a031f402ee9 Docker --entryPoints.lok… 10 minutes ago Up 10 minutes 0.0.0.0:20001->20001/tcp traefik
[loki1@nethserver8 state]$
journalctl -r _UID=$(id -u loki1)
Apr 04 16:27:18 nethserver8 systemd[1876]: Started podman-269443.scope.
Apr 04 16:20:23 nethserver8 systemd[1876]: Started podman-264270.scope.
Apr 04 16:16:34 nethserver8 systemd[1876]: Started podman-263754.scope.
Apr 04 16:16:26 nethserver8 traefik[263733]: time="2024-04-04T14:16:26Z" level=info msg="Configuration loaded from flags."
Apr 04 16:16:26 nethserver8 loki-server[263722]: level=error ts=2024-04-04T14:16:26.419326997Z caller=mapper.go:42 msg="unable to read rules directory" path=/loki/rules-temp err="open /loki/rules->
Apr 04 16:16:26 nethserver8 systemd[1876]: Started Loki frontend HTTP proxy.
Apr 04 16:16:26 nethserver8 podman[263616]: 6a031f402ee9bd39c15071a6daa6afbb9d2cdb514cc44e1730a63856a653d4dd
Apr 04 16:16:26 nethserver8 systemd[1876]: Started libcrun container.
Apr 04 16:16:26 nethserver8 systemd[1876]: Started Loki server.
Apr 04 16:16:26 nethserver8 podman[263608]: d913babf32af166c034de21346499e907e6cc6899a80ba3096a64c9e979e69e7
Apr 04 16:16:26 nethserver8 systemd[1876]: Started libcrun container.
Apr 04 16:16:26 nethserver8 podman[263608]: Writing manifest to image destination
Apr 04 16:16:26 nethserver8 podman[263608]: Copying config sha256:727c39682956d63917122ed8b23b916821f7c850bed426ace01fabfe81530790
Apr 04 16:16:25 nethserver8 podman[263616]: Writing manifest to image destination
Apr 04 16:16:25 nethserver8 podman[263616]: Copying config sha256:de1a7c9d5d63d8ab27b26f16474a74e78d252007d3a67ff08dcbad418eb335ae
Apr 04 16:16:24 nethserver8 podman[263608]: Copying blob sha256:586f67fcd52ef2996deebfe0c50f19ca0b2c59660e6efa9c76e580580dd8a5f9
Apr 04 16:16:24 nethserver8 podman[263608]: Copying blob sha256:31603596830fc7e56753139f9c2c6bd3759e48a850659506ebfb885d1cf3aef5
Apr 04 16:16:24 nethserver8 podman[263608]: Copying blob sha256:753793ea21f66410ed4b05c828216728b6521b13a1de5939203258636f11eed5
Apr 04 16:16:24 nethserver8 podman[263608]: Copying blob sha256:e4f5d1b1114583aca9dafec1f0cc4f8a21ae15f5ea04b0ac236c148e09aa5f54
Apr 04 16:16:24 nethserver8 podman[263608]: Copying blob sha256:e94f8da5a2cba4300c6e74d1eef04f2cc7f39cb1d08303f803f119fe3e06ca0f
Apr 04 16:16:24 nethserver8 podman[263608]: Copying blob sha256:858024693c419c0dd99a6d6ea88967d0f74cb3bdace948a115a6b7584019f644
Apr 04 16:16:24 nethserver8 podman[263608]: Copying blob sha256:8818f75b6e26a7de66dd46182944abe3a695bcafe01f1bf4cffcab489b34f960
Apr 04 16:16:24 nethserver8 podman[263608]: Getting image source signatures
Apr 04 16:16:24 nethserver8 podman[263616]: Copying blob sha256:5f6722e60c2f6c55424dadebe886f88ba1b903df075b00048427439abb91b85a
Apr 04 16:16:24 nethserver8 podman[263616]: Copying blob sha256:fe4701c53ae539044a129428575f42a0e0aa5e923d04b97466915bf45f5df0e3
Apr 04 16:16:24 nethserver8 podman[263616]: Copying blob sha256:3abdcd3bb40ca29c232ad12d1f2cba6efcb28e8d8ab7e5787ad2771b4e3862b0
Apr 04 16:16:24 nethserver8 podman[263616]: Copying blob sha256:ddad3d7c1e96adf9153f8921a7c9790f880a390163df453be1566e9ef0d546e0
Apr 04 16:16:24 nethserver8 podman[263616]: Getting image source signatures
Apr 04 16:16:22 nethserver8 podman[263608]: Trying to pull docker.io/grafana/loki:2.2.1...
Apr 04 16:16:22 nethserver8 podman[263616]: Trying to pull docker.io/library/traefik:v2.4...
Apr 04 16:16:22 nethserver8 systemd[1876]: Starting Loki frontend HTTP proxy...
Apr 04 16:16:22 nethserver8 systemd[1876]: Starting Loki server...
Apr 04 16:16:22 nethserver8 systemd[1876]: Started Loki pod service.
Apr 04 16:16:22 nethserver8 podman[263571]: d2369bd1427e5c55197399a2a1a892e179ad044270d811d5f0b432832d0e7224
Apr 04 16:16:22 nethserver8 systemd[1876]: Started libcrun container.
Apr 04 16:16:22 nethserver8 podman[263517]: d2369bd1427e5c55197399a2a1a892e179ad044270d811d5f0b432832d0e7224
Apr 04 16:16:22 nethserver8 systemd[1876]: Created slice cgroup user-libpod_pod_d2369bd1427e5c55197399a2a1a892e179ad044270d811d5f0b432832d0e7224.slice.
Apr 04 16:16:22 nethserver8 systemd[1876]: Starting Loki pod service...
Apr 04 16:16:13 nethserver8 systemd[1876]: Started podman-263502.scope.
Apr 04 16:16:05 nethserver8 systemd[1876]: Started podman-pause-0816637a.scope.
Apr 04 16:16:05 nethserver8 systemd[1876]: Started podman-263484.scope.
Apr 04 16:15:58 nethserver8 systemd[1876]: Started podman-263429.scope.
Apr 04 16:15:50 nethserver8 systemd[1876]: Started podman-263412.scope.
Apr 04 15:29:54 nethserver8 systemd[1876]: Started podman-258870.scope.
Apr 04 15:29:48 nethserver8 systemd[1876]: traefik.service: Job traefik.service/start failed with result 'dependency'.
Apr 04 15:29:48 nethserver8 systemd[1876]: Dependency failed for Loki frontend HTTP proxy.
Apr 04 15:29:48 nethserver8 systemd[1876]: loki-server.service: Job loki-server.service/start failed with result 'dependency'.
Apr 04 15:29:48 nethserver8 systemd[1876]: Dependency failed for Loki server.
Apr 04 15:29:48 nethserver8 systemd[1876]: Failed to start Loki pod service.
Apr 04 15:29:48 nethserver8 systemd[1876]: loki.service: Failed with result 'start-limit-hit'.
Apr 04 15:29:48 nethserver8 systemd[1876]: loki.service: Start request repeated too quickly.
Apr 04 15:29:48 nethserver8 systemd[1876]: Stopped Loki pod service.
Apr 04 15:29:48 nethserver8 podman[258860]: 6905e54a7713e9fdae49b4afa485a457cb368cb48a2aacad1790b31860b23837
Apr 04 15:29:48 nethserver8 systemd[1876]: Removed slice cgroup user-libpod_pod_6905e54a7713e9fdae49b4afa485a457cb368cb48a2aacad1790b31860b23837.slice.
Apr 04 15:29:48 nethserver8 podman[258845]: 6905e54a7713e9fdae49b4afa485a457cb368cb48a2aacad1790b31860b23837
Apr 04 15:29:48 nethserver8 systemd[1876]: Stopping Loki pod service...
Apr 04 15:29:48 nethserver8 systemd[1876]: Stopped Loki server.
Apr 04 15:29:48 nethserver8 systemd[1876]: Stopped Loki frontend HTTP proxy.
Apr 04 15:29:48 nethserver8 systemd[1876]: traefik.service: Scheduled restart job, restart counter is at 38.
Apr 04 15:29:48 nethserver8 podman[258808]: Trying to pull docker.io/grafana/loki:2.2.1...
Apr 04 15:29:48 nethserver8 systemd[1876]: Failed to start Loki frontend HTTP proxy.
Apr 04 15:29:48 nethserver8 systemd[1876]: traefik.service: Failed with result 'exit-code'.
Apr 04 15:29:48 nethserver8 systemd[1876]: traefik.service: Control process exited, code=exited, status=125/n/a
Apr 04 15:29:48 nethserver8 podman[258817]: Error: removing storage for container "traefik": unmounting "/home/loki1/.local/share/containers/storage/overlay/0713907cbc9254eb6a76a4a184ff4d0d5d8af2e>
Apr 04 15:29:48 nethserver8 podman[258817]: time="2024-04-04T15:29:48+02:00" level=warning msg="Unmounting container \"traefik\" while attempting to delete storage: unmounting \"/home/loki1/.local>
Apr 04 15:29:48 nethserver8 systemd[1876]: Starting Loki frontend HTTP proxy...
Apr 04 15:29:48 nethserver8 systemd[1876]: Starting Loki server...
Apr 04 15:29:48 nethserver8 systemd[1876]: Started Loki pod service.
Apr 04 15:29:48 nethserver8 podman[258772]: 6905e54a7713e9fdae49b4afa485a457cb368cb48a2aacad1790b31860b23837
Apr 04 15:29:48 nethserver8 systemd[1876]: Started libcrun container.
Apr 04 15:29:48 nethserver8 podman[258762]: 6905e54a7713e9fdae49b4afa485a457cb368cb48a2aacad1790b31860b23837
Apr 04 15:29:48 nethserver8 systemd[1876]: Created slice cgroup user-libpod_pod_6905e54a7713e9fdae49b4afa485a457cb368cb48a2aacad1790b31860b23837.slice.
Apr 04 15:29:47 nethserver8 systemd[1876]: Starting Loki pod service...
Apr 04 15:29:47 nethserver8 systemd[1876]: Stopped Loki pod service.
Apr 04 15:29:47 nethserver8 podman[258750]: 9ffc4a1a6ac8fbdefca7706f615d7ef37ad09616a9beb2dd0dc3d97fcbe314b1
Apr 04 15:29:47 nethserver8 systemd[1876]: Removed slice cgroup user-libpod_pod_9ffc4a1a6ac8fbdefca7706f615d7ef37ad09616a9beb2dd0dc3d97fcbe314b1.slice.
Apr 04 15:29:47 nethserver8 podman[258729]: 9ffc4a1a6ac8fbdefca7706f615d7ef37ad09616a9beb2dd0dc3d97fcbe314b1
Apr 04 15:29:47 nethserver8 systemd[1876]: Stopping Loki pod service...
Apr 04 15:29:47 nethserver8 systemd[1876]: Stopped Loki server.
Apr 04 15:29:47 nethserver8 systemd[1876]: Stopped Loki frontend HTTP proxy.
Apr 04 15:29:47 nethserver8 systemd[1876]: traefik.service: Scheduled restart job, restart counter is at 37.
Apr 04 15:29:47 nethserver8 podman[258693]: Trying to pull docker.io/grafana/loki:2.2.1...
Apr 04 15:29:47 nethserver8 systemd[1876]: Failed to start Loki frontend HTTP proxy.
Apr 04 15:29:47 nethserver8 systemd[1876]: traefik.service: Failed with result 'exit-code'.
Apr 04 15:29:47 nethserver8 systemd[1876]: traefik.service: Control process exited, code=exited, status=125/n/a
Apr 04 15:29:47 nethserver8 podman[258703]: Error: removing storage for container "traefik": unmounting "/home/loki1/.local/share/containers/storage/overlay/0713907cbc9254eb6a76a4a184ff4d0d5d8af2e>
Apr 04 15:29:47 nethserver8 podman[258703]: time="2024-04-04T15:29:47+02:00" level=warning msg="Unmounting container \"traefik\" while attempting to delete storage: unmounting \"/home/loki1/.local>
Apr 04 15:29:47 nethserver8 systemd[1876]: Starting Loki frontend HTTP proxy...
Apr 04 15:29:47 nethserver8 systemd[1876]: Starting Loki server...
Apr 04 15:29:47 nethserver8 systemd[1876]: Started Loki pod service.
Apr 04 15:29:47 nethserver8 podman[258655]: 9ffc4a1a6ac8fbdefca7706f615d7ef37ad09616a9beb2dd0dc3d97fcbe314b1
Apr 04 15:29:47 nethserver8 systemd[1876]: Started libcrun container.
Apr 04 15:29:47 nethserver8 podman[258645]: 9ffc4a1a6ac8fbdefca7706f615d7ef37ad09616a9beb2dd0dc3d97fcbe314b1
Apr 04 15:29:47 nethserver8 systemd[1876]: Created slice cgroup user-libpod_pod_9ffc4a1a6ac8fbdefca7706f615d7ef37ad09616a9beb2dd0dc3d97fcbe314b1.slice.
Apr 04 15:29:47 nethserver8 systemd[1876]: Starting Loki pod service...
Apr 04 15:29:47 nethserver8 systemd[1876]: Stopped Loki pod service.
Apr 04 15:29:47 nethserver8 podman[258634]: 76a78f9498d457e4a74d042389593ab926666ccf6bfe05934c1eca592b7b5f48
Apr 04 15:29:47 nethserver8 systemd[1876]: Removed slice cgroup user-libpod_pod_76a78f9498d457e4a74d042389593ab926666ccf6bfe05934c1eca592b7b5f48.slice.
Apr 04 15:29:47 nethserver8 podman[258615]: 76a78f9498d457e4a74d042389593ab926666ccf6bfe05934c1eca592b7b5f48
Apr 04 15:29:47 nethserver8 systemd[1876]: Stopping Loki pod service...
Apr 04 15:29:47 nethserver8 systemd[1876]: Stopped Loki server.
Apr 04 15:29:47 nethserver8 systemd[1876]: Stopped Loki frontend HTTP proxy.
Apr 04 15:29:47 nethserver8 systemd[1876]: traefik.service: Scheduled restart job, restart counter is at 36.
Apr 04 15:29:47 nethserver8 podman[258576]: Trying to pull docker.io/grafana/loki:2.2.1...
Apr 04 15:29:47 nethserver8 systemd[1876]: Failed to start Loki frontend HTTP proxy.
Apr 04 15:29:47 nethserver8 systemd[1876]: traefik.service: Failed with result 'exit-code'.
Apr 04 15:29:47 nethserver8 systemd[1876]: traefik.service: Control process exited, code=exited, status=125/n/a
Apr 04 15:29:47 nethserver8 podman[258586]: Error: removing storage for container "traefik": unmounting "/home/loki1/.local/share/containers/storage/overlay/0713907cbc9254eb6a76a4a184ff4d0d5d8af2e>
Apr 04 15:29:47 nethserver8 podman[258586]: time="2024-04-04T15:29:47+02:00" level=warning msg="Unmounting container \"traefik\" while attempting to delete storage: unmounting \"/home/loki1/.local>
Apr 04 15:29:47 nethserver8 systemd[1876]: Starting Loki frontend HTTP proxy...
Apr 04 15:29:47 nethserver8 systemd[1876]: Starting Loki server...
Apr 04 15:29:47 nethserver8 systemd[1876]: Started Loki pod service.
Apr 04 15:29:47 nethserver8 podman[258539]: 76a78f9498d457e4a74d042389593ab926666ccf6bfe05934c1eca592b7b5f48
Apr 04 15:29:47 nethserver8 systemd[1876]: Started libcrun container.
Apr 04 15:29:47 nethserver8 podman[258529]: 76a78f9498d457e4a74d042389593ab926666ccf6bfe05934c1eca592b7b5f48
Apr 04 15:29:47 nethserver8 systemd[1876]: Created slice cgroup user-libpod_pod_76a78f9498d457e4a74d042389593ab926666ccf6bfe05934c1eca592b7b5f48.slice.
Apr 04 15:29:47 nethserver8 systemd[1876]: Starting Loki pod service...
Apr 04 15:29:47 nethserver8 systemd[1876]: Stopped Loki pod service.
Apr 04 15:29:47 nethserver8 podman[258518]: 84d9ccc4032f7b42d703cbcba5007237182f41fa8d7ac84cf18c188ba09380fe
Apr 04 15:29:47 nethserver8 systemd[1876]: Removed slice cgroup user-libpod_pod_84d9ccc4032f7b42d703cbcba5007237182f41fa8d7ac84cf18c188ba09380fe.slice.
Apr 04 15:29:47 nethserver8 podman[258497]: 84d9ccc4032f7b42d703cbcba5007237182f41fa8d7ac84cf18c188ba09380fe
Apr 04 15:29:46 nethserver8 systemd[1876]: Stopping Loki pod service...
Apr 04 15:29:46 nethserver8 systemd[1876]: Stopped Loki server.
Apr 04 15:29:46 nethserver8 systemd[1876]: Stopped Loki frontend HTTP proxy.
Apr 04 15:29:46 nethserver8 systemd[1876]: traefik.service: Scheduled restart job, restart counter is at 35.
Apr 04 15:29:46 nethserver8 podman[258460]: Trying to pull docker.io/grafana/loki:2.2.1...
Apr 04 15:29:46 nethserver8 systemd[1876]: Failed to start Loki frontend HTTP proxy.
Apr 04 15:29:46 nethserver8 systemd[1876]: traefik.service: Failed with result 'exit-code'.
Apr 04 15:29:46 nethserver8 systemd[1876]: traefik.service: Control process exited, code=exited, status=125/n/a
Apr 04 15:29:46 nethserver8 podman[258469]: Error: removing storage for container "traefik": unmounting "/home/loki1/.local/share/containers/storage/overlay/0713907cbc9254eb6a76a4a184ff4d0d5d8af2e>
Apr 04 15:29:46 nethserver8 podman[258469]: time="2024-04-04T15:29:46+02:00" level=warning msg="Unmounting container \"traefik\" while attempting to delete storage: unmounting \"/home/loki1/.local>
Apr 04 15:29:46 nethserver8 systemd[1876]: Starting Loki frontend HTTP proxy...
Apr 04 15:29:46 nethserver8 systemd[1876]: Starting Loki server...
Apr 04 15:29:46 nethserver8 systemd[1876]: Started Loki pod service.
Apr 04 15:29:46 nethserver8 podman[258423]: 84d9ccc4032f7b42d703cbcba5007237182f41fa8d7ac84cf18c188ba09380fe
Apr 04 15:29:46 nethserver8 systemd[1876]: Started libcrun container.
Apr 04 15:29:46 nethserver8 podman[258414]: 84d9ccc4032f7b42d703cbcba5007237182f41fa8d7ac84cf18c188ba09380fe
Apr 04 15:29:46 nethserver8 systemd[1876]: Created slice cgroup user-libpod_pod_84d9ccc4032f7b42d703cbcba5007237182f41fa8d7ac84cf18c188ba09380fe.slice.
Apr 04 15:29:46 nethserver8 systemd[1876]: Starting Loki pod service...
Apr 04 15:29:46 nethserver8 systemd[1876]: Stopped Loki pod service.
Apr 04 15:29:46 nethserver8 podman[258402]: 04abb9d3d7adf0dfe7dd876b117ad4c97161b0c0030c595005458d6a13cb6397
Apr 04 15:29:46 nethserver8 systemd[1876]: Removed slice cgroup user-libpod_pod_04abb9d3d7adf0dfe7dd876b117ad4c97161b0c0030c595005458d6a13cb6397.slice.
Apr 04 15:29:46 nethserver8 podman[258380]: 04abb9d3d7adf0dfe7dd876b117ad4c97161b0c0030c595005458d6a13cb6397
Apr 04 15:29:46 nethserver8 systemd[1876]: Stopping Loki pod service...
Apr 04 15:29:46 nethserver8 systemd[1876]: Stopped Loki server.
Apr 04 15:29:46 nethserver8 systemd[1876]: Stopped Loki frontend HTTP proxy.
Apr 04 15:29:46 nethserver8 systemd[1876]: traefik.service: Scheduled restart job, restart counter is at 34.
Apr 04 15:29:46 nethserver8 podman[258342]: Trying to pull docker.io/grafana/loki:2.2.1...
Apr 04 15:29:46 nethserver8 systemd[1876]: Failed to start Loki frontend HTTP proxy.
Apr 04 15:29:46 nethserver8 systemd[1876]: traefik.service: Failed with result 'exit-code'.
Apr 04 15:29:46 nethserver8 systemd[1876]: traefik.service: Control process exited, code=exited, status=125/n/a
Apr 04 15:29:46 nethserver8 podman[258351]: Error: removing storage for container "traefik": unmounting "/home/loki1/.local/share/containers/storage/overlay/0713907cbc9254eb6a76a4a184ff4d0d5d8af2e>
Apr 04 15:29:46 nethserver8 podman[258351]: time="2024-04-04T15:29:46+02:00" level=warning msg="Unmounting container \"traefik\" while attempting to delete storage: unmounting \"/home/loki1/.local>
Apr 04 15:29:45 nethserver8 systemd[1876]: Starting Loki frontend HTTP proxy...
Apr 04 15:29:45 nethserver8 systemd[1876]: Starting Loki server...
Apr 04 15:29:45 nethserver8 systemd[1876]: Started Loki pod service.
Apr 04 15:29:45 nethserver8 podman[258306]: 04abb9d3d7adf0dfe7dd876b117ad4c97161b0c0030c595005458d6a13cb6397
Apr 04 15:29:45 nethserver8 systemd[1876]: Started libcrun container.
Apr 04 15:29:45 nethserver8 podman[258297]: 04abb9d3d7adf0dfe7dd876b117ad4c97161b0c0030c595005458d6a13cb6397
Apr 04 15:29:45 nethserver8 systemd[1876]: Created slice cgroup user-libpod_pod_04abb9d3d7adf0dfe7dd876b117ad4c97161b0c0030c595005458d6a13cb6397.slice.
Apr 04 15:29:45 nethserver8 systemd[1876]: Starting Loki pod service...
Apr 04 15:29:36 nethserver8 systemd[1876]: Started podman-258286.scope.
Apr 04 15:29:24 nethserver8 systemd[1876]: Started podman-pause-b51ca8dc.scope.
Apr 04 15:29:24 nethserver8 systemd[1876]: Started podman-258273.scope.
Apr 04 15:29:16 nethserver8 systemd[1876]: Started podman-258252.scope.
Apr 04 15:29:07 nethserver8 systemd[1876]: Started podman-258225.scope.
Apr 04 15:26:12 nethserver8 systemd[1876]: Started podman-257570.scope.
Apr 04 15:26:07 nethserver8 systemd[1876]: traefik.service: Job traefik.service/start failed with result 'dependency'.
Apr 04 15:26:07 nethserver8 systemd[1876]: Dependency failed for Loki frontend HTTP proxy.
Apr 04 15:26:07 nethserver8 systemd[1876]: loki-server.service: Job loki-server.service/start failed with result 'dependency'.
Apr 04 15:26:07 nethserver8 systemd[1876]: Dependency failed for Loki server.
Apr 04 15:26:07 nethserver8 systemd[1876]: Failed to start Loki pod service.
Apr 04 15:26:07 nethserver8 systemd[1876]: loki.service: Failed with result 'start-limit-hit'.
Apr 04 15:26:07 nethserver8 systemd[1876]: loki.service: Start request repeated too quickly.
Apr 04 15:26:07 nethserver8 systemd[1876]: Stopped Loki pod service.
Apr 04 15:26:07 nethserver8 podman[257551]: 7f0e01eb12c01339efe474279c517ece057ebf745ad5d8d2f6b409cc0892ffee
Apr 04 15:26:07 nethserver8 systemd[1876]: Removed slice cgroup user-libpod_pod_7f0e01eb12c01339efe474279c517ece057ebf745ad5d8d2f6b409cc0892ffee.slice.
Apr 04 15:26:07 nethserver8 podman[257532]: 7f0e01eb12c01339efe474279c517ece057ebf745ad5d8d2f6b409cc0892ffee
Apr 04 15:26:07 nethserver8 systemd[1876]: Stopping Loki pod service...
Apr 04 15:26:07 nethserver8 systemd[1876]: Stopped Loki server.
Apr 04 15:26:07 nethserver8 systemd[1876]: Stopped Loki frontend HTTP proxy.
Apr 04 15:26:07 nethserver8 systemd[1876]: traefik.service: Scheduled restart job, restart counter is at 33.
Apr 04 15:26:06 nethserver8 podman[257493]: Trying to pull docker.io/grafana/loki:2.2.1...
Apr 04 15:26:06 nethserver8 systemd[1876]: Failed to start Loki frontend HTTP proxy.
...skipping...Apr 04 15:26:05 nethserver8 podman[257210]: 9c4fe63b796171b0f462483eca1d71b80fe6b1ed9f2d3ab4a85e0a620a2385d6
Apr 04 15:26:05 nethserver8 systemd[1876]: Removed slice cgroup user-libpod_pod_9c4fe63b796171b0f462483eca1d71b80fe6b1ed9f2d3ab4a85e0a620a2385d6.slice.
Apr 04 15:26:05 nethserver8 podman[257189]: 9c4fe63b796171b0f462483eca1d71b80fe6b1ed9f2d3ab4a85e0a620a2385d6
Apr 04 15:26:05 nethserver8 systemd[1876]: Stopping Loki pod service...
Apr 04 15:26:05 nethserver8 systemd[1876]: Stopped Loki server.
Apr 04 15:26:05 nethserver8 systemd[1876]: Stopped Loki frontend HTTP proxy.
Apr 04 15:26:05 nethserver8 systemd[1876]: traefik.service: Scheduled restart job, restart counter is at 30.
Apr 04 15:26:05 nethserver8 podman[257150]: Trying to pull docker.io/grafana/loki:2.2.1...
Apr 04 15:26:05 nethserver8 systemd[1876]: Failed to start Loki frontend HTTP proxy.
Apr 04 15:26:05 nethserver8 systemd[1876]: traefik.service: Failed with result 'exit-code'.
Apr 04 15:26:05 nethserver8 systemd[1876]: traefik.service: Control process exited, code=exited, status=125/n/a
Apr 04 15:26:05 nethserver8 podman[257158]: Error: removing storage for container "traefik": unmounting "/home/loki1/.local/share/containers/storage/overlay/0713907cbc9254eb6a76a4a184ff4d0d5d8af2e>
Apr 04 15:26:05 nethserver8 podman[257158]: time="2024-04-04T15:26:05+02:00" level=warning msg="Unmounting container \"traefik\" while attempting to delete storage: unmounting \"/home/loki1/.local>
Apr 04 15:26:05 nethserver8 systemd[1876]: Starting Loki frontend HTTP proxy...
Apr 04 15:26:05 nethserver8 systemd[1876]: Starting Loki server...
Apr 04 15:26:05 nethserver8 systemd[1876]: Started Loki pod service.
Apr 04 15:26:04 nethserver8 podman[257114]: 9c4fe63b796171b0f462483eca1d71b80fe6b1ed9f2d3ab4a85e0a620a2385d6
Apr 04 15:26:04 nethserver8 systemd[1876]: Started libcrun container.
Apr 04 15:26:04 nethserver8 podman[257104]: 9c4fe63b796171b0f462483eca1d71b80fe6b1ed9f2d3ab4a85e0a620a2385d6
Apr 04 15:26:04 nethserver8 systemd[1876]: Created slice cgroup user-libpod_pod_9c4fe63b796171b0f462483eca1d71b80fe6b1ed9f2d3ab4a85e0a620a2385d6.slice.
Apr 04 15:26:04 nethserver8 systemd[1876]: Starting Loki pod service...
Apr 04 15:26:04 nethserver8 systemd[1876]: Stopped Loki pod service.
Apr 04 15:26:04 nethserver8 podman[257093]: bca3634dcf2372430348eb0b105d0bacd88288627d59401c50b328c3b16f0699
Apr 04 15:26:04 nethserver8 systemd[1876]: Removed slice cgroup user-libpod_pod_bca3634dcf2372430348eb0b105d0bacd88288627d59401c50b328c3b16f0699.slice.
Apr 04 15:26:04 nethserver8 podman[257070]: bca3634dcf2372430348eb0b105d0bacd88288627d59401c50b328c3b16f0699
Apr 04 15:26:04 nethserver8 conmon[257025]: conmon 3b92d92532400b730805 <nwarn>: Failed to open cgroups file: /sys/fs/cgroup/user.slice/user-1002.slice/user@1002.service/user.slice/user-libpod_pod>
Apr 04 15:26:04 nethserver8 systemd[1876]: Stopping Loki pod service...
Apr 04 15:26:04 nethserver8 systemd[1876]: Stopped Loki server.
Apr 04 15:26:04 nethserver8 systemd[1876]: Stopped Loki frontend HTTP proxy.
Apr 04 15:26:04 nethserver8 systemd[1876]: traefik.service: Scheduled restart job, restart counter is at 29.
Apr 04 15:26:04 nethserver8 podman[257031]: Trying to pull docker.io/grafana/loki:2.2.1...
Apr 04 15:26:04 nethserver8 systemd[1876]: Failed to start Loki frontend HTTP proxy.
Apr 04 15:26:04 nethserver8 systemd[1876]: traefik.service: Failed with result 'exit-code'.
Apr 04 15:26:04 nethserver8 systemd[1876]: traefik.service: Control process exited, code=exited, status=125/n/a
Apr 04 15:26:04 nethserver8 podman[257042]: Error: removing storage for container "traefik": unmounting "/home/loki1/.local/share/containers/storage/overlay/0713907cbc9254eb6a76a4a184ff4d0d5d8af2e>
Apr 04 15:26:04 nethserver8 podman[257042]: time="2024-04-04T15:26:04+02:00" level=warning msg="Unmounting container \"traefik\" while attempting to delete storage: unmounting \"/home/loki1/.local>
Apr 04 15:26:04 nethserver8 systemd[1876]: Starting Loki frontend HTTP proxy...
Apr 04 15:26:04 nethserver8 systemd[1876]: Starting Loki server...
Apr 04 15:26:04 nethserver8 systemd[1876]: Started Loki pod service.
Apr 04 15:26:04 nethserver8 podman[256995]: bca3634dcf2372430348eb0b105d0bacd88288627d59401c50b328c3b16f0699
Apr 04 15:26:04 nethserver8 systemd[1876]: Started libcrun container.
Apr 04 15:26:04 nethserver8 podman[256985]: bca3634dcf2372430348eb0b105d0bacd88288627d59401c50b328c3b16f0699
Apr 04 15:26:04 nethserver8 systemd[1876]: Created slice cgroup user-libpod_pod_bca3634dcf2372430348eb0b105d0bacd88288627d59401c50b328c3b16f0699.slice.
Apr 04 15:26:04 nethserver8 systemd[1876]: Starting Loki pod service...
Apr 04 15:25:54 nethserver8 systemd[1876]: Started podman-256516.scope.
Apr 04 15:25:42 nethserver8 systemd[1876]: Started podman-pause-194cbbf2.scope.
Apr 04 15:25:42 nethserver8 systemd[1876]: Started podman-256494.scope.
Apr 04 15:25:23 nethserver8 systemd[1876]: Started podman-256436.scope.
I have the same error message here, but everything works 
We can ignore it.
Try to restart promtail 
systemctl restart promtail