Cannot access cockpit Web UI (port 9090)

ertank · April 18, 2020, 5:50am

NethServer Version: 7.7.1908
Module: Web UI

Hello,

We are trying to move out NethServer to our new server running VMware ESXi 6.7. So, we have manually installed vmware tools on the system. OS selected as CentOS 7 64Bit when setting up VM.

It is a fresh installation we are doing. We setup several things in server like hostname, SSH, etc. We rebooted. Now, we cannot get access to Web UI. We get connection timeouts.

This is not the first time we are facing that problem. We could not figure what’s causing it and installed NethServer from scratch several times.

We can have access by SSH, we can get http://IP, we can get http://IP:980 we cannot get https://IP:9090

Our searches on forum shows similar threads. They seemingly having httpd-admin not started and on our system it seems to be running. Just not listening port 9090

[root@neth ~]# systemctl -l status httpd-admin
● httpd-admin.service - Server Manager UI httpd instance
   Loaded: loaded (/usr/lib/systemd/system/httpd-admin.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2020-04-18 09:05:57 +03; 38min left
     Docs: https://github.com/NethServer/nethserver-httpd-admin
 Main PID: 1423 (httpd)
   CGroup: /system.slice/httpd-admin.service
           ├─1423 /usr/sbin/httpd -f /etc/httpd/admin-conf/httpd.conf -c MaxConnectionsPerChild 12 -DFOREGROUND
           ├─1579 /usr/sbin/httpd -f /etc/httpd/admin-conf/httpd.conf -c MaxConnectionsPerChild 12 -DFOREGROUND
           ├─1580 /usr/sbin/httpd -f /etc/httpd/admin-conf/httpd.conf -c MaxConnectionsPerChild 12 -DFOREGROUND
           ├─1581 /usr/sbin/httpd -f /etc/httpd/admin-conf/httpd.conf -c MaxConnectionsPerChild 12 -DFOREGROUND
           ├─1585 /usr/sbin/httpd -f /etc/httpd/admin-conf/httpd.conf -c MaxConnectionsPerChild 12 -DFOREGROUND
           └─1587 /usr/sbin/httpd -f /etc/httpd/admin-conf/httpd.conf -c MaxConnectionsPerChild 12 -DFOREGROUND

Apr 18 09:05:57 neth.server.com systemd[1]: Started Server Manager UI httpd instance.
[root@neth ~]# netstat -l|grep 980
tcp6       0      0 [::]:980                [::]:*                  LISTEN
[root@neth ~]# netstat -l|grep 9090

There is no firewall installed, yet. We installed so many times that I might be wrong about that.
There is single ethernet connected to VM.
Network is set as LAN and green with fixed IP number.
We obviously can ping IP number.

One thing we suspect is that it happens after switching from DHCP assigned IP number to manually assigned static IP number.

Though we could not see what maybe problem.

Any help is appreciated.

Thanks & regards,
Ertan

Andy_Wismer · April 18, 2020, 5:53am

Hello Ertan

Have you tried to use a “reserved” DHCP when setting up that NethServer - with the reserved DHCP lease coressponding to the later IP? That means even when you switch from DHCP to fixed IP, it remains the same…

Maybe worth a try…
Andy

ertank · April 18, 2020, 5:59am

Hello,

We did not do that as it will be NethServer which is to serve as DHCP Server after everything complete.

Thanks.

Andy_Wismer · April 18, 2020, 6:12am

Hi

Maybe I read wrong or understood wrong:

Before setting up my NethServer, usually the router/firewall is my DHCP server…

Andy

ertank · April 18, 2020, 6:24am

Hello,

We do have a DHCP server. It is our old NethServer running on another physical server right now. DHCP assignments are coming from it.

We have ADSL modem in bridge mode. Once we finish installing new NethServer, old one is going to retire and ADSL modem will be physically connected to new NethServer. In the end, new NethServer will be serving as our router/firewall and DHCP server.

As old server will retire, we cannot use DHCP assigned static IP numbers from it.

We also are not 100% sure it is manual IP assignment that is causing Web UI to fail. We suspect that it might be.

Thanks.

Andy_Wismer · April 18, 2020, 6:50am

It doesn’t matter if that old server is going to be switched off…
After setting up, your new Nethserver will have a DHCP reserved IP, which you can change to manual - using the same IP.
After this, the new NethServer doesn’t need the old NethServer or DHCP from the old NethServer…

But all in all - you might just setup your NethServer right from the beginning with a fixed IP, the one you want to use…

stephdl · April 18, 2020, 7:29am

systemctl status cockpit

check if the service is up, cockpit uses 9090

ertank · April 18, 2020, 7:32am

That seems to be dead

[root@neth ~]# systemctl status cockpit
● cockpit.service - Cockpit Web Service
   Loaded: loaded (/usr/lib/systemd/system/cockpit.service; static; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:cockpit-ws(8)
[root@neth ~]#

stephdl · April 18, 2020, 7:33am

systemctl start cockpit

and check what it is wrong in logs

Andy_Wismer · April 18, 2020, 7:39am

It doesn’t matter if that old server is going to be switched off…
After setting up, your new Nethserver will have a DHCP reserved IP, which you can change to manual - using the same IP.
After this, the new NethServer doesn’t need the old NethServer or DHCP from the old NethServer…

But all in all - you might just setup your NethServer right from the beginning with a fixed IP, the one you want to use…

ertank · April 18, 2020, 7:46am

Before I read your reply, I tried to start it by using

systemctl restart cockpit

I have following in log files

Apr 18 10:32:44 neth systemd: Reloading.
Apr 18 10:32:53 neth systemd: Starting Cockpit Web Service...
Apr 18 10:32:53 neth systemd: Started Cockpit Web Service.
Apr 18 10:32:53 neth cockpit-ws: Using certificate: /etc/cockpit/ws-certs.d/99-nethserver.cert
Apr 18 10:33:05 neth CommAmqpListener[9966]: Initializing CommAmqpListener
Apr 18 10:34:32 neth systemd: Started Session c4 of user root.

I see status as started now.

[root@neth log]# systemctl status cockpit
● cockpit.service - Cockpit Web Service
   Loaded: loaded (/usr/lib/systemd/system/cockpit.service; static; vendor preset: disabled)
   Active: inactive (dead) since Sat 2020-04-18 10:34:24 +03; 3min 7s ago
     Docs: man:cockpit-ws(8)
  Process: 9921 ExecStart=/usr/libexec/cockpit-ws (code=exited, status=0/SUCCESS)
  Process: 9919 ExecStartPre=/usr/sbin/remotectl certificate --ensure --user=root --group=cockpit-ws --selinux-type=etc_t (code=exited, status=0/SUCCESS)
 Main PID: 9921 (code=exited, status=0/SUCCESS)

Apr 18 10:32:53 neth.server.com systemd[1]: Starting Cockpit Web Service...
Apr 18 10:32:53 neth.server.com systemd[1]: Started Cockpit Web Service.
Apr 18 10:32:53 neth.server.com cockpit-ws[9921]: Using certificate: /etc/cockpit/ws-certs.d/99-nethserver.cert

However, I still cannot access to Web UI. Though, I see below in /var/log/messages if that means anything at all

Apr 18 10:35:35 neth journal: %Y-%m-%d %H:%M:%S,561[9973]: CommAmqpListener: [CCafException] AmqpCommon::validateStatus() amqp_socket_open_noblock: hostname lookup failed
Apr 18 10:38:05 neth CommAmqpListener[10286]: Initializing CommAmqpListener

We are not using a FQDN as hostname.

stephdl · April 18, 2020, 7:48am

sorry but the hour of death is at Sat 2020-04-18 10:34:24

stephdl · April 18, 2020, 7:50am

journalctl -u cockpit

check in /var/log/messages too

ertank · April 18, 2020, 7:53am

All relevant lines from /var/log/messages near/at that hour is below. Just to clarify, I did not remove any lines in between from below log entries. There is no other log entry other than below.

Apr 18 10:32:44 neth systemd: Reloading.
Apr 18 10:32:53 neth systemd: Starting Cockpit Web Service...
Apr 18 10:32:53 neth systemd: Started Cockpit Web Service.
Apr 18 10:32:53 neth cockpit-ws: Using certificate: /etc/cockpit/ws-certs.d/99-nethserver.cert
Apr 18 10:33:05 neth CommAmqpListener[9966]: Initializing CommAmqpListener
Apr 18 10:34:32 neth systemd: Started Session c4 of user root.
Apr 18 10:35:35 neth journal: %Y-%m-%d %H:%M:%S,561[9973]: CommAmqpListener: [CCafException] AmqpCommon::validateStatus() amqp_socket_open_noblock: hostname lookup failed
Apr 18 10:38:05 neth CommAmqpListener[10286]: Initializing CommAmqpListener

journal result is as following

[root@neth log]# journalctl -u cockpit
-- Logs begin at Sat 2020-04-18 08:22:49 +03, end at Sat 2020-04-18 10:50:36 +03. --
Apr 18 10:32:53 neth.server.com systemd[1]: Starting Cockpit Web Service...
Apr 18 10:32:53 neth.server.com systemd[1]: Started Cockpit Web Service.
Apr 18 10:32:53 neth.server.com cockpit-ws[9921]: Using certificate: /etc/cockpit/ws-certs.d/99-nethserver.cert

Nothing else that I can see. I am not sure what additional log file(s) to check for more clues.

Andy_Wismer · April 18, 2020, 7:55am

@ertank
@stephdl

Hi

I think this is an issue of changing the IP AFTER setup of NethServer…

I could be wrong, but I imagine that cockpit / fw / has the old IP set (DHCP during setup).

My 2 cents
Andy

stephdl · April 18, 2020, 8:11am

config show cockpit.socket

ertank · April 18, 2020, 8:40am

[root@neth log]# config show cockpit.socket
cockpit.socket=configuration
HideUninstall=disabled
LimitAccess=31.206.45.188
Pins=
Shortcuts=
ShowHints=enabled
UserSettingsGrantAccess=disabled
UserSettingsPage=disabled
UserSettingsPageAlias=/user-settings
access=green,red
delegation=
[root@neth log]#

stephdl · April 18, 2020, 9:01am

nothing bad there

mrmarkuz · April 18, 2020, 1:46pm

Did you try

signal-event nethserver-cockpit-update

and check logs afterwards?

Which packages are installed?

rpm -qa --qf "%{NAME}-%{VERSION}\n" "nethserver-*" | sort

Maybe reinstall cockpit helps?

yum reinstall nethserver-cockpit cockpit cockpit-bridge cockpit-storaged cockpit-system cockpit-ws

ertank · April 18, 2020, 2:20pm

I was accessing new NethServer over VPN connection for setting it up.
I catch a colleague at office and he told me he can connect to Web UI just fine.
After he did complete some configuration.
Now I can access Web UI, too.
We did nothing particular. Maybe a few restarts.

If that is to happen again, I will also check your suggestions @mrmarkuz

We have just 3 applications installed.
-Firewall (that was not installed when I had broken Web UI)
-OpenVPN
-Web server (that is a standard installation I assume)

Thanks.