Mattermost service stopped

Bug
1

NethServer Version: 7.9.2009
Module: Mattermost

Hi all Nethserver fans!
Mattermost has just be updated, but cannot start anymore. I have the following:
systemctl status mattermost
● mattermost.service - Mattermost
Loaded: loaded (/usr/lib/systemd/system/mattermost.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Thu 2021-01-21 07:28:36 CET; 14s ago
Process: 8912 ExecStart=/opt/mattermost/bin/mattermost (code=exited, status=1/FAILURE)
Main PID: 8912 (code=exited, status=1/FAILURE)

Jan 21 07:28:35 #mysite# systemd[1]: mattermost.service failed.
Jan 21 07:28:36 #mysite# systemd[1]: mattermost.service holdoff time over, scheduling restart.
Jan 21 07:28:36 #mysite# systemd[1]: Stopped Mattermost.
Jan 21 07:28:36 #mysite# systemd[1]: start request repeated too quickly for mattermost.service
Jan 21 07:28:36 #mysite# systemd[1]: Failed to start Mattermost.
Jan 21 07:28:36 #mysite# systemd[1]: Unit mattermost.service entered failed state.
Jan 21 07:28:36 #mysite# systemd[1]: mattermost.service failed.

Any idea on how to solve this?
Thanks

2

Please check the logfiles /var/log/messages and /opt/mattermost/logs/* for more detailed errors.

Maybe the database is not running?

systemctl status rh-postgresql12-postgresql

3

The database is running. For Mattermost, I have:
Jan 21 07:28:36 srv.vache.ovh systemd[1]: mattermost.service holdoff time over, scheduling restart.
Jan 21 07:28:36 srv.vache.ovh systemd[1]: Stopped Mattermost.
Jan 21 07:28:36 srv.vache.ovh systemd[1]: start request repeated too quickly for mattermost.service
Jan 21 07:28:36 srv.vache.ovh systemd[1]: Failed to start Mattermost.
Jan 21 07:28:36 srv.vache.ovh systemd[1]: Unit mattermost.service entered failed state.
Jan 21 07:28:36 srv.vache.ovh systemd[1]: mattermost.service failed.

4

Is there nothing more in the logfiles, like why mattermost is not starting? Is /opt/mattermost/logs/mattermost.log empty?

You may try to reconfigure mattermost:

signal-event nethserver-mattermost-update

5

I have this repeated:
{“level”:“error”,“ts”:1611209986.8674495,“caller”:“sqlstore/supplier.go:258”,“msg”:“Failed to ping DB”,“error”:“dial tcp 127.0.0.1:5432: connect: connection refused”,“retrying in seconds”:10}

6

Hm, it seems mattermost wants the postgresql db on port 5432 but postgresql 12 on Neth uses port 55434 so mattermost wasn’t correctly preconfigured.

Did you already try

signal-event nethserver-mattermost-update

to reconfigure mattermost?

Which version of mattermost do you use?

rpm -qa "*mattermost*"

Please recheck the DB settings in /opt/mattermost/config/config.json, they should look like this, the password is in /var/lib/nethserver/secrets/mattermost

    "SqlSettings": {
        "DriverName": "postgres",
        "DataSource": "postgres://mattuser:PASSWORD@localhost:55434/mattermost?sslmode=disable\u0026connect_timeout=10",
7

nethserver-mattermost-1.6.1-1.ns7.x86_64
and
“SqlSettings”:{“MaxOpenConns”:300,“DriverName”:“postgres”,“QueryTimeout”:30,“ConnMaxLifetimeMilliseconds”:3600000,“DataSourceSearchReplicas”:[],“DisableDatabaseSearch”:false,“DataSource”:“postgres://mattuser:psw@localhost:55434/mattermost?sslmode=disable&connect_timeout=10”
I go for reconfiguration?

8

After reconfiguration:
Jan 21 16:59:30 srv.vache.ovh systemd[1]: Started Mattermost.
Jan 21 16:59:31 srv.vache.ovh systemd[1]: mattermost.service: main process exited, code=exited, status=1/FAILURE
Jan 21 16:59:31 srv.vache.ovh systemd[1]: Unit mattermost.service entered failed state.
Jan 21 16:59:31 srv.vache.ovh systemd[1]: mattermost.service failed.
Jan 21 16:59:31 srv.vache.ovh systemd[1]: mattermost.service holdoff time over, scheduling restart.
Jan 21 16:59:31 srv.vache.ovh systemd[1]: Stopped Mattermost.
Jan 21 16:59:31 srv.vache.ovh systemd[1]: start request repeated too quickly for mattermost.service
Jan 21 16:59:31 srv.vache.ovh systemd[1]: Failed to start Mattermost.
Jan 21 16:59:31 srv.vache.ovh systemd[1]: Unit mattermost.service entered failed state.
Jan 21 16:59:31 srv.vache.ovh systemd[1]: mattermost.service failed.

9

Do you still see these lines in /opt/mattermost/logs/ after the reconfiguration?

Lets check if we are able to login to the database and check if the mattermost db exists:

Login as postgres user (no password needed):

su - postgres

Enable psql 12 environment:

scl enable rh-postgresql12 bash

Connect to postgres on port 55434

psql -p 55434

List databases:

\l

You should see a line like this:

 mattermost | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres         +

Quit and exit:

\q

exit (2 times)

10

First, I have no additional line in /opt/mattermost/logs/
I check the database

11

The database exits as expected

12

Hi,

i have exactly the same problem after updating mattermost this morning. After a restart of the server, Mattermost Service is down.

Regards

Uwe

13

I can’t reproduce it, mattermost starting here without problems.

Maybe the port is in use by another application?

netstat -tlpn | grep 8065

Does the mattermost user exist?

getent passwd mattermost

Maybe it’s a permission issue, check if mattermost is allowed:

[root@server2 ~]# ls -l /var/lib/nethserver/mattermost/
total 0
drwxr-xr-x 2 postgres   postgres   28 Jan 20 11:59 backup
drwxr-xr-x 3 mattermost mattermost 19 Jan 21 20:41 data
[root@server2 ~]# ls -l /var/lib/nethserver/mattermost/data
total 4
drwxr-xr-- 17 mattermost mattermost 4096 Sep 27 21:57 users

Check logs:

1 Like
14

[root@teamserver ~]# netstat -tlpn | grep 8065
[root@teamserver ~]# getent passwd mattermos
[root@teamserver ~]# ls -l /var/lib/nethserver/mattermost/
total 4
drwxr-xr-x 2 postgres postgres 28 Jan 20 11:59 backup
drwxr-xr-x 57 mattermost mattermost 4096 Jan 20 11:59 data
[root@teamserver ~]# ls -l /var/lib/nethserver/mattermost/data
total 0
drwxr-x— 3 mattermost mattermost 19 Jul 23 16:11 20200723
drwxr-x— 3 mattermost mattermost 19 Aug 8 11:56 20200808
drwxr-x— 3 mattermost mattermost 19 Aug 12 09:52 20200812
drwxr-x— 3 mattermost mattermost 19 Aug 18 11:36 20200818
drwxr-x— 3 mattermost mattermost 19 Aug 23 10:20 20200823
drwxr-x— 3 mattermost mattermost 19 Aug 24 08:57 20200824
drwxr-x— 3 mattermost mattermost 19 Aug 26 13:15 20200826
drwxr-x— 3 mattermost mattermost 19 Aug 29 13:33 20200829
drwxr-x— 3 mattermost mattermost 19 Aug 30 15:18 20200830
drwxr-x— 3 mattermost mattermost 19 Sep 1 18:40 20200901
drwxr-x— 3 mattermost mattermost 19 Sep 10 10:01 20200910
drwxr-x— 3 mattermost mattermost 19 Sep 12 12:50 20200912
drwxr-x— 3 mattermost mattermost 19 Sep 17 17:58 20200917
drwxr-x— 3 mattermost mattermost 19 Sep 18 12:30 20200918
drwxr-x— 3 mattermost mattermost 19 Sep 26 10:40 20200926
drwxr-x— 3 mattermost mattermost 19 Sep 27 17:57 20200927
drwxr-x— 3 mattermost mattermost 19 Sep 29 16:19 20200929
drwxr-x— 3 mattermost mattermost 19 Oct 3 13:44 20201003
drwxr-x— 3 mattermost mattermost 19 Oct 5 21:03 20201005
drwxr-x— 3 mattermost mattermost 19 Oct 8 10:44 20201008
drwxr-x— 3 mattermost mattermost 19 Oct 17 10:15 20201017
drwxr-x— 3 mattermost mattermost 19 Oct 24 15:18 20201024
drwxr-x— 3 mattermost mattermost 19 Oct 26 12:36 20201026
drwxr-x— 3 mattermost mattermost 19 Oct 29 18:35 20201029
drwxr-x— 3 mattermost mattermost 19 Oct 30 20:27 20201030
drwxr-x— 3 mattermost mattermost 19 Nov 2 12:21 20201102
drwxr-x— 3 mattermost mattermost 19 Nov 5 09:10 20201105
drwxr-x— 3 mattermost mattermost 19 Nov 7 16:46 20201107
drwxr-x— 3 mattermost mattermost 19 Nov 8 08:46 20201108
drwxr-x— 3 mattermost mattermost 19 Nov 9 10:49 20201109
drwxr-x— 3 mattermost mattermost 19 Nov 11 09:40 20201111
drwxr-x— 3 mattermost mattermost 19 Nov 12 10:07 20201112
drwxr-x— 3 mattermost mattermost 19 Nov 14 10:14 20201114
drwxr-x— 3 mattermost mattermost 19 Nov 15 11:06 20201115
drwxr-x— 3 mattermost mattermost 19 Nov 17 12:21 20201117
drwxr-x— 3 mattermost mattermost 19 Nov 21 10:51 20201121
drwxr-x— 3 mattermost mattermost 19 Nov 27 11:41 20201127
drwxr-x— 3 mattermost mattermost 19 Dec 7 11:10 20201207
drwxr-x— 3 mattermost mattermost 19 Dec 8 18:10 20201208
drwxr-x— 3 mattermost mattermost 19 Dec 10 09:44 20201210
drwxr-x— 3 mattermost mattermost 19 Dec 12 09:32 20201212
drwxr-x— 3 mattermost mattermost 19 Dec 13 16:48 20201213
drwxr-x— 3 mattermost mattermost 19 Dec 16 11:17 20201216
drwxr-x— 3 mattermost mattermost 19 Dec 20 10:36 20201220
drwxr-x— 3 mattermost mattermost 19 Dec 21 19:21 20201221
drwxr-x— 3 mattermost mattermost 19 Dec 25 08:57 20201225
drwxr-x— 3 mattermost mattermost 19 Dec 26 17:47 20201226
drwxr-x— 3 mattermost mattermost 19 Dec 31 11:35 20201231
drwxr-x— 3 mattermost mattermost 19 Jan 2 13:49 20210102
drwxr-x— 3 mattermost mattermost 19 Jan 6 17:14 20210106
drwxr-x— 3 mattermost mattermost 19 Jan 12 20:32 20210112
drwxr-x— 3 mattermost mattermost 19 Jan 15 16:16 20210115
drwxr-x— 3 mattermost mattermost 19 Jan 18 10:51 20210118
drwxr-x— 2 mattermost mattermost 23 Jun 19 2020 brand
drwxr-x— 8 mattermost mattermost 210 Dec 26 12:20 users

15

Is it a typo (mattermos instead of mattermost) or did you get no result?

It should look like this:

[root@server2 ~]# getent passwd mattermost
mattermost:x:977:973:Mattermost user:/var/lib/nethserver/mattermost:/sbin/nologin
16

For me:
getent passwd mattermost
mattermost:x:979:974:Mattermost user:/var/lib/nethserver/mattermost:/sbin/nologin
'# ls -l /var/lib/nethserver/mattermost/
total 0
drwxr-xr-x 2 postgres postgres 28 Jan 20 11:59 backup
drwxr-xr-x 5 mattermost mattermost 47 Jan 20 11:59 data
'# ls -l /var/lib/nethserver/mattermost/data
total 4
drwxr-x— 2 mattermost mattermost 6 Apr 20 2020 brand
drwxr-x— 2 mattermost mattermost 50 Nov 20 13:31 plugins
drwxr-x— 12 mattermost mattermost 4096 Nov 20 12:18 users

17

Yes, it was a typo. Here is the result:

mattermost:x:983:978:Mattermost user:/var/lib/nethserver/mattermost:/sbin/nologi n

18

In the /var/log/messages, nothing except the errors already reported

19

Please try to run mattermost from cli:

sudo -u mattermost /opt/mattermost/bin/mattermost

20

Error: failed to load configuration: failed to create store: unable to load on store creation: parsing error at line 1, character 13860: invalid character ‘a’ after top-level value

1 Like