Matrix-Synapse Docker Database Connection Error

loic · August 19, 2023, 8:26pm

Hi, thanks for the synapse-docker-synapse-1 synapse-docker-synapse-1 | synapse-docker-synapse-1 | synapse-docker-synapse-1 | synapse-docker-synapse-1 | synapse-docker-synapse-1 | synapse-docker-synapse-1 | synapse-docker-synapse-1 | synapse-docker-synapse-1 | synapse-docker-synapse-1 | synapse-docker-synapse-1 | synapse-docker-synapse-1 | synapse-docker-synapse-1 | synapse-docker-synapse-1 | synapse-docker-synapse-1 | synapse-docker-synapse-1 | synapse-docker-synapse-1 | synapse-docker-synapse-1 | synapse-docker-synapse-1 | tuto. I have gone through most of the steps, but synapse does not start:
| Error during initialisation:
Traceback (most recent call last):
File “/usr/local/lib/python3.11/site-packages/synapse/app/homeserver.py”, line 353, in setup
hs.setup()
File “/usr/local/lib/python3.11/site-packages/synapse/server.py”, line 339, in setup
self.datastores = Databases(self.DATASTORE_CLASS, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/site-packages/synapse/storage/databases/init.py”, line 65, in init
with make_conn(database_config, engine, “startup”) as db_conn:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/site-packages/synapse/storage/database.py”, line 161, in make_conn
native_db_conn = engine.module.connect(**db_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/site-packages/psycopg2/init.py”, line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host “db” (172.28.0.2) and accepting
TCP/IP connections on port 5432?

Any idea on why and how to solve. It seems to be the access to the database.
Thanks

mrmarkuz · August 20, 2023, 7:15pm

Is the synapse db container running? Please check the logs of the synapse db container in Portainer or on CLI:

docker logs synapse-docker-db-1

loic · August 20, 2023, 8:23pm

I have this:
PostgreSQL Database directory appears to contain a database; Skipping initialization

2023-08-19 20:06:07.170 UTC [1] LOG: starting PostgreSQL 15.4 (Debian 15.4-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2023-08-19 20:06:07.170 UTC [1] LOG: listening on IPv4 address “0.0.0.0”, port 5432
2023-08-19 20:06:07.170 UTC [1] LOG: listening on IPv6 address “::”, port 5432
2023-08-19 20:06:07.177 UTC [1] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.5432”
2023-08-19 20:06:07.185 UTC [29] LOG: database system was shut down at 2023-08-19 20:05:57 UTC
2023-08-19 20:06:07.219 UTC [1] LOG: database system is ready to accept connections
2023-08-19 20:11:07.287 UTC [27] LOG: checkpoint starting: time
2023-08-19 20:11:07.295 UTC [27] LOG: checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.002 s, sync=0.002 s, total=0.008 s; sync files=2, longest=0.002 s, average=0.001 s; distance=0 kB, estimate=0 kB

mrmarkuz · August 21, 2023, 7:10pm

The database container log looks good but it’s not reachable from the synapse container.

Is it reachable from localhost?

ping 172.28.0.2

Are both containers part of the aqua network?

docker network inspect aqua

BTW, I moved our posts to a new thread.

loic · August 21, 2023, 7:48pm

Thank you Markus for managing my posts
The ping is fine
Aqua:

{
“Name”: “aqua”,
“Id”: “294df7320af9676d2ce8a615ebde3edc967c380c388557c4baf753951f3235f1”,
“Created”: “2023-08-19T12:05:48.694985299+02:00”,
“Scope”: “local”,
“Driver”: “bridge”,
“EnableIPv6”: false,
“IPAM”: {
“Driver”: “default”,
“Options”: {},
“Config”: [
{
“Subnet”: “172.28.0.0/16”,
“Gateway”: “172.28.0.1”
}
]
},
“Internal”: false,
“Attachable”: false,
“Ingress”: false,
“ConfigFrom”: {
“Network”: “”
},
“ConfigOnly”: false,
“Containers”: {
“1f662e4fe44a7a811fd9be4fb9e4fdd0f705a44d97464c7e83a49069dc8e829d”: {
“Name”: “portainer”,
“EndpointID”: “a893f4b75ee76b3620ebc30c9c0aa7b00e54e69d2f5870a9058959eb8c4987d6”,
“MacAddress”: “02:42:ac:1c:ff:fe”,
“IPv4Address”: “172.28.255.254/16”,
“IPv6Address”: “”
},
“b67e50affe4af9bccab396b810701a2fd477296d3eb89d12b4b08396732972c5”: {
“Name”: “synapse-docker-db-1”,
“EndpointID”: “bdfe9b2f0ab6d95673780079a04782b501baafd7bf44c1f4faf46df954b57df9”,
“MacAddress”: “02:42:ac:1c:00:02”,
“IPv4Address”: “172.28.0.2/16”,
“IPv6Address”: “”
},
“de9f3d1304f02b373ffdc1da525c8e9dc2812b5a1498ccde9cee6d362f25aaa4”: {
“Name”: “mautrix-whatsapp”,
“EndpointID”: “bf6ed93a091e770b375501e56fd058c6e061976b4b2ba676ea930875367e9049”,
“MacAddress”: “02:42:ac:1c:00:03”,
“IPv4Address”: “172.28.0.3/16”,
“IPv6Address”: “”
}
},
“Options”: {
“com.docker.network.bridge.name”: “aqua0”
},
“Labels”: {}
}

mrmarkuz · August 21, 2023, 8:07pm

You’re welcome.

The synapse container is not part of the aqua network.

To quickly add it (doesn’t work anymore after restart/reboot):

docker network connect aqua synapse-docker-synapse-1

To add it permanently you need to define aqua as default network in the /opt/synapse-docker/docker-compose.yml. Here’s my compose file to compare, at the bottom you can find the default networks set to aqua.

/opt/synapse-docker/docker-compose.yml

version: '3.7'

services:

  synapse:
    image: docker.io/matrixdotorg/synapse:latest
    restart: unless-stopped
    volumes:
      - ./synapse-data:/data
      - ./shared_secret_authenticator.py:/usr/local/lib/python3.9/site-packages/shared_secret_authenticator.py
    depends_on:
      - db
    ports:
      - '127.0.0.1:8008:8008/tcp'
  db:
    image: docker.io/postgres:latest
    restart: unless-stopped
    environment:
      - POSTGRES_USER=synapse
      - POSTGRES_PASSWORD=synapse
      - POSTGRES_INITDB_ARGS=--encoding='UTF8' --lc-collate='C' --lc-ctype='C'
    volumes:
      - ./postgres-data:/var/lib/postgresql/data

  mautrix-whatsapp:
    container_name: mautrix-whatsapp
    image: dock.mau.dev/mautrix/whatsapp:latest
    restart: unless-stopped
    volumes:
    - ./mautrix-whatsapp:/data

  mautrix-signal:
    container_name: mautrix-signal
    image: dock.mau.dev/mautrix/signal
    restart: unless-stopped
    volumes:
    - ./mautrix-signal:/data
    - ./signald:/signald
    depends_on:
    - signald

  signald:
    container_name: signald
    image: docker.io/signald/signald
    restart: unless-stopped
    volumes:
    - ./signald:/signald

  mautrix-telegram:
    container_name: mautrix-telegram
    image: dock.mau.dev/mautrix/telegram
    restart: unless-stopped
    volumes:
    - ./mautrix-telegram:/data

  synapse-admin:
    container_name: synapse-admin
    hostname: synapse-admin
    image: awesometechnologies/synapse-admin
    #build:
     # context: https://github.com/Awesome-Technologies/synapse-admin.git
      # args:
      #   - NODE_OPTIONS="--max_old_space_size=1024"
    ports:
      - "8095:80"
    restart: unless-stopped

networks:
  default:
    name: aqua
    external: true

Restart synapse containers using docker-compose:

cd /opt/synapse-docker
docker-compose down && docker-compose up -d

loic · August 21, 2023, 8:16pm

I had the network part, but I copy-pasted yours. I get:

services.networks Additional property default is not allowed

loic · August 21, 2023, 8:18pm

docker network connect aqua synapse-docker-synapse-1
includes the synapse-1 into aqua

        "2d41db427967b172d3622f9df0dbf9f712a1d38c7139c14115aff82fec65ae34": {
            "Name": "synapse-docker-synapse-1",
            "EndpointID": "271c4939accef1a6cd320149f1656e2e926912aca811115177643c0da4009a56",
            "MacAddress": "02:42:ac:1c:00:04",
            "IPv4Address": "172.28.0.4/16",
            "IPv6Address": ""

So there is a problem in my docker-compose.yml

mrmarkuz · August 21, 2023, 8:28pm

Please check the indents, “networks” should be top level like “services”, it seems you put networks to the services.

services:
  synapse:
...
...
...
networks:
  default:
...

loic · August 21, 2023, 8:30pm

that was it! Thank you very much. I check my other settings and if everything works.

loic · August 28, 2023, 5:13pm

Hi Markus, Hi everyone,
Sorry to come back, but that was not the reason. I have reset the file and synapse keeps restarting. I have uninstalled and reinstalled the containers, and I have the same behavior. I have two questions to try to understand:

When I test the ports, I have nothing for db:

[root@srv synapse-docker]# docker port synapse-docker-synapse-1
8008/tcp → 127.0.0.1:8008
[root@srv synapse-docker]# docker port synapse-docker-db-1
[root@srv synapse-docker]#

The howto says to grant ownership to user 991:991. what is that user? I don’t have it
Thanks

mrmarkuz · August 28, 2023, 7:15pm

Please check the logs of the synapse container for more details about the restart

That’s ok. No redirected ports needed because the containers talk directly to each other over the aqua network.

It’s a container user, see also Docker

In my case 991:991 points to zabbix:cgred locally:

[root@server synapse-docker]# ls -l
total 16
drwxr-xr-x  2    1337  1337   50 Jan 21  2023 bridge
-rw-r--r--  1 root    root  1735 Jan 31  2023 docker-compose.yml
drwxr-xr-x  2    1337  1337   50 Aug 27 23:42 mautrix-signal
drwxr-xr-x  2    1337  1337   50 Aug 27 23:42 mautrix-telegram
drwxr-xr-x  3    1337  1337   62 Aug 27 23:42 mautrix-whatsapp
drwx------ 19 polkitd root  4096 Aug 27 23:42 postgres-data
-rw-r--r--  1 root    root  4625 Jan 20  2023 shared_secret_authenticator.py
drwxr-xr-x  6    1337  1337  106 Aug 27 23:42 signald
drwxr-xr-x  3 zabbix  cgred  301 Jan 21  2023 synapse-data
[root@server synapse-docker]# ls -l synapse-data/
total 32
-rw-r--r-- 1 root   root  3057 Mar 16 10:41 homeserver.yaml
-rw-r--r-- 1 root   root   479 Jan 19  2023 matrix.mrmarkuz.domain.tld.log.config
-rw-r--r-- 1 root   root    59 Jan 19  2023 matrix.mrmarkuz.domain.tld.signing.key
drwxr-xr-x 6 zabbix cgred   97 Jan 19  2023 media_store
-rw-r--r-- 1 zabbix cgred  550 Jan 21  2023 registration_signal.yaml
-rw-r--r-- 1 root   root   632 Jan 21  2023 registration_telegram.yaml
-rw------- 1 zabbix cgred  526 Jan 21  2023 registration.yaml

loic · August 30, 2023, 2:40pm

Mine is memcached cgred 991:986

loic · August 30, 2023, 2:41pm

I always get an access problem to the db container:

Error during initialisation:
Traceback (most recent call last):
File “/usr/local/lib/python3.11/site-packages/synapse/app/homeserver.py”, line 353, in setup
hs.setup()
File “/usr/local/lib/python3.11/site-packages/synapse/server.py”, line 339, in setup
self.datastores = Databases(self.DATASTORE_CLASS, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/site-packages/synapse/storage/databases/init.py”, line 65, in init
with make_conn(database_config, engine, “startup”) as db_conn:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/site-packages/synapse/storage/database.py”, line 161, in make_conn
native_db_conn = engine.module.connect(**db_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/site-packages/psycopg2/init.py”, line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host “db” (172.28.0.3) and accepting
TCP/IP connections on port 5432?

mrmarkuz · August 30, 2023, 4:24pm

Let’s list the databases:

docker exec -it synapse-docker-db-1 psql -U synapse -c "\l"

To check the connection I installed nmap in the container.

Go into the synapse container:

docker exec -it synapse-docker-synapse-1 bash

Update packages:

apt update

Install nmap:

apt install nmap

Test connection, nmap should show the open port 5432 of postgresql:

nmap db

Remove nmap:

apt autoremove nmap

It should be 991:991.

loic · August 30, 2023, 4:53pm

                                       List of databases
Name | Owner | Encoding | Collate | Ctype | ICU Locale | Locale Provider | Access privilege
s
-----------±--------±---------±--------±------±-----------±----------------±-----------------

mautrix | synapse | UTF8 | C | C | | libc |
postgres | synapse | UTF8 | C | C | | libc |
synapse | synapse | UTF8 | C | C | | libc |
template0 | synapse | UTF8 | C | C | | libc | =c/synapse
+
| | | | | | | synapse=CTc/synap
se
template1 | synapse | UTF8 | C | C | | libc | =c/synapse
+
| | | | | | | synapse=CTc/synap
se

mrmarkuz · August 30, 2023, 4:55pm

OK, the engine is running and the DBs are there, so please also check the connection from the synapse container as described above.

loic · August 30, 2023, 4:57pm

I couldn’t log into synapse because it is constantly restarting. Login into db container gives:

Nmap scan report for db (172.28.0.4)
Host is up (0.000050s latency).
rDNS record for 172.28.0.4: c873e01769e4
Not shown: 999 closed tcp ports (reset)
PORT STATE SERVICE
5432/tcp open postgresql

Nmap done: 1 IP address (1 host up) scanned in 0.37 seconds

loic · August 30, 2023, 5:08pm

It is 991:991, just checked

mrmarkuz · August 30, 2023, 6:39pm

In your synapse container db points to 172.28.0.3 but in the database container db points to 172.28.0.4? Did you restart the containers?

Matrix-Synapse Docker Database Connection Error

Name | Owner | Encoding | Collate | Ctype | ICU Locale | Locale Provider | Access privilege s -----------±--------±---------±--------±------±-----------±----------------±-----------------

Name | Owner | Encoding | Collate | Ctype | ICU Locale | Locale Provider | Access privilege
s
-----------±--------±---------±--------±------±-----------±----------------±-----------------