NS8 - Issue after core updates

yummiweb · March 11, 2023, 6:03pm

I installed the latest core updates on my NS8 test system.

According to the message display, the updates went through without any problems.

Everything worked right after the update. After a restart, however, connections to the web interfaces no longer work. This seems to affect all web services.

A connection with SSH is still possible.

I went back to the last snapshot and tried several times, the problem occurs after every update with a subsequent restart.

The network configuration or hostname hasn’t changed, I would imagine there is a problem with traefik?

Regards
Yummiweb

Edit: The underlying system is Debian 11

yummiweb · March 11, 2023, 6:31pm

A test with “lsof -i:443” shows no result, port closed? 22, 25, 587 etc. seem open.

danb35 · March 11, 2023, 6:51pm

I don’t have an answer, but I encountered the same thing the other day under Rocky 9. Since I was dealing with other issues anyway, my resolution was to just blow it away and start over, which probably isn’t what you want to do.

yummiweb · March 11, 2023, 7:10pm

Well, the snapshots before the core updates still work. But not anymore after these core updates.

Traffic now seems to be going through port 443 as well, but not from me:

“lsof -i:443
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
slirp4net 1083 loki1 8u IPv4 20319 0t0 TCP myservername:43388->ec2-15-236-36-251.eu-west-3.compute.amazonaws.com:https (ESTABLISHED)
slirp4net 1083 loki1 11u IPv4 19315 0t0 TCP myservername:59222->lb-140-82-121-6-fra.github.com:https (ESTABLISHED)”

yummiweb · March 11, 2023, 7:11pm

Would the (problematic) core updates already be included if I completely reinstall NS8?

danb35 · March 11, 2023, 7:40pm

I’d assume so. My installation now doesn’t show updates available, and its web UI works fine.

stephdl · March 12, 2023, 7:44am

does traefik is up ?

ssh traefik1@localhost

systemctl status --user traefik

podman ps

something interesting in log with root user

journalctl

stephdl · March 12, 2023, 7:49am

need to test it, sometimes we have side effect with the podman version between debian (so much old) and centos like

stephdl · March 12, 2023, 7:57am

workable on debian 11

stephdl · March 12, 2023, 7:58am

yummiweb · March 12, 2023, 9:17am

ssh traefik1@localhost

The SSH connection asks for the password for “traefik”.

systemctl status --user traefik

Failed to connect to bus: $DBUS_SESSION_BUS_ADDRESS and $XDG_RUNTIME_DIR not defined (consider using --machine=<user>@.host --user to connect to bus of other user)

podman ps

CONTAINER ID  IMAGE                              COMMAND               CREATED       STATUS           PORTS       NAMES
814f3984b9b4  ghcr.io/nethserver/redis:0.0.31    redis-server /dat...  14 hours ago  Up 14 hours ago              redis
765c8ffd4ab8  docker.io/grafana/promtail:2.7.3   -config.file=/etc...  14 hours ago  Up 14 hours ago              promtail1
57facee6bc5e  ghcr.io/nethserver/samba-dc:0.0.6                        14 hours ago  Up 14 hours ago              samba1

yummiweb · March 12, 2023, 9:18am

cat /etc/debian_version

11.6

yummiweb · March 12, 2023, 9:35am

something interesting in log with root user

journalctl

Do you have a hint what I could narrow the journal output to (grep)?

yummiweb · March 12, 2023, 10:38am

Here are a few excerpts from “journalctl | grep error”:

cluster[312]: Task queue pop error: dial tcp 127.0.0.1:6379: connect: connection refused
samba1[315]: Task queue pop error: dial tcp 127.0.0.1:6379: connect: connection refused
promtail1[314]: Task queue pop error: dial tcp 127.0.0.1:6379: connect: connection refused
bggsrv node[313]: Task queue pop error: dial tcp 127.0.0.1:6379: connect: connection refused

ldapproxy1[530]: Task queue pop error: dial tcp 127.0.0.1:6379: connect: connection refused
mail1[539]: Task queue pop error: dial tcp 127.0.0.1:6379: connect: connection refused
ldapproxy1[530]: Task queue pop error: dial tcp 127.0.0.1:6379: connect: connection refused
mail1[539]: Task queue pop error: dial tcp 127.0.0.1:6379: connect: connection refused
loki1[546]: Task queue pop error: dial tcp 127.0.0.1:6379: connect: connection refused

traefik1[576]: Task queue pop error: dial tcp 127.0.0.1:6379: connect: connection refused
traefik1[576]: Task queue pop error: dial tcp 127.0.0.1:6379: connect: connection refused
traefik1[865]: Error: error reading CIDFile: open /run/user/1001/traefik.ctr-id: no such file or directory
traefik1[947]: Error: error reading CIDFile: open /run/user/1001/traefik.ctr-id: no such file or directory
traefik1[1066]: Error: error reading CIDFile: open /run/user/1001/traefik.ctr-id: no such file or directory

traefik1[576]: Task queue pop error: dial tcp 127.0.0.1:6379: connect: connection refused

yummiweb · March 12, 2023, 11:05am

Sorry I hadn’t tried this under “root”.

Under “root” I can log in via ssh as “trafik1”.

stephdl · March 12, 2023, 11:31am

To narrow

journalctl -t trafik1

Everything relative to the traefik1 module will be displayed

It seems that traefik ldapproxy and mail cannot contact redis on the port 6379

Did you try to restart redis

systemctl restart redis

yummiweb · March 12, 2023, 12:16pm

“journalctl -t traefik1”
returns for the current session:

Mär 12 12:40:02 myservername traefik1[608]: Task queue pop error: dial tcp 127.0.0.1:6379: connect: connection refused
Mär 12 12:40:02 myservername traefik1[608]: Task queue pop error: dial tcp 127.0.0.1:6379: connect: connection refused
Mär 12 12:40:02 myservername traefik1[712]: Error: statfs /home/traefik1/.config/state/configs: no such file or directory
Mär 12 12:40:02 myservername traefik1[887]: Error: error reading CIDFile: open /run/user/1001/traefik.ctr-id: no such file or directory
Mär 12 12:40:02 myservername traefik1[920]: Error: statfs /home/traefik1/.config/state/configs: no such file or directory
Mär 12 12:40:02 myservername traefik1[952]: Error: error reading CIDFile: open /run/user/1001/traefik.ctr-id: no such file or directory
Mär 12 12:40:02 myservername traefik1[1021]: Error: statfs /home/traefik1/.config/state/configs: no such file or directory
Mär 12 12:40:02 myservername traefik1[1054]: Error: error reading CIDFile: open /run/user/1001/traefik.ctr-id: no such file or directory
Mär 12 12:40:02 myservername traefik1[1092]: Error: statfs /home/traefik1/.config/state/configs: no such file or directory
Mär 12 12:40:02 myservername traefik1[1135]: Error: error reading CIDFile: open /run/user/1001/traefik.ctr-id: no such file or directory
Mär 12 12:40:03 myservername traefik1[1177]: Error: statfs /home/traefik1/.config/state/configs: no such file or directory
Mär 12 12:40:03 myservername traefik1[1255]: Error: error reading CIDFile: open /run/user/1001/traefik.ctr-id: no such file or directory

After “systemctl restart redis” I get with “journalctl -t traefik1” (new messages):

Mär 12 13:10:40 myservername traefik1[608]: redis: 2023/03/12 13:10:40 pubsub.go:159: redis: discarding bad PubSub connection: EOF
Mär 12 13:10:40 myservername traefik1[608]: Task queue pop error: dial tcp 127.0.0.1:6379: connect: connection refused

Note: it is not important for me to solve this problem, I can also reinstall NS8, it is only a test system. But I would like to understand what is going on and how to troubleshoot such issues in NS8.

stephdl · March 12, 2023, 5:52pm

Can you connect to redis

In root

redis-cli

yummiweb · March 12, 2023, 6:26pm

Yes, I get the following (prompt):

127.0.0.1:6379>

stdenis · March 14, 2023, 4:23pm

Hi,
You’re not alone, I have the same error:

Mar 14 12:15:42 lvvi-neth8-1 traefik1[2706]: Error: statfs /home/traefik1/.config/state/configs: no such file or directory
Mar 14 12:15:43 lvvi-neth8-1 traefik1[2740]: Error: error reading CIDFile: open /run/user/1000/traefik.ctr-id: no such file or directory

Also running on a Debian 11 VM.
SSH and redis acces are ok.