Install Matrix-Synapse including Whatsapp bridge using docker-compose

mrmarkuz · January 19, 2023, 10:38pm

This howto explains how to install the matrix-synapse server including the mautrix-whatsapp bridge. Maybe other bridges will follow. Thanks to @robb for inspiring and testing.

I ended up using docker-compose because it was the only way for me to make the database container work.

Download and install docker-compose:

curl -L https://github.com/docker/compose/releases/download/v2.15.0/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

Create the synapse-docker directory:

mkdir /opt/synapse-docker

Create /opt/synapse-docker/docker-compose.yml with following content:

version: '3.7'

services:

  synapse:
    image: docker.io/matrixdotorg/synapse:latest
    restart: unless-stopped
    volumes:
      - ./synapse-data:/data
    depends_on:
      - db
    ports:
      - '127.0.0.1:8008:8008/tcp'
  db:
    image: docker.io/postgres:latest
    restart: unless-stopped
    environment:
      - POSTGRES_USER=synapse
      - POSTGRES_PASSWORD=synapse
      - POSTGRES_INITDB_ARGS=--encoding='UTF8' --lc-collate='C' --lc-ctype='C'
    volumes:
      - ./postgres-data:/var/lib/postgresql/data

  mautrix-whatsapp:
    container_name: mautrix-whatsapp
    image: dock.mau.dev/mautrix/whatsapp:latest
    restart: unless-stopped
    volumes:
    - ./mautrix-whatsapp:/data

networks:
  default:
    name: aqua
    external: true

Create /etc/httpd/conf.d/a_synapse.conf with following content and edit the ServerName to match your domain:

<VirtualHost *:443>
        SSLEngine on
        ServerName matrix.example.com
        AllowEncodedSlashes NoDecode
        ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon
        ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix
</VirtualHost>

Listen 8448

<VirtualHost *:8448>
        SSLEngine on
        AllowEncodedSlashes NoDecode
        ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon
        ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix
</VirtualHost>

Restart httpd to apply the config:

systemctl restart httpd

Enter the synapse-docker dir, else docker-compose won’t work:

cd /opt/synapse-docker

Run the below Docker command to generate the synapse config file, edit SYNAPSE_SERVER_NAME to match your domain:

docker run -it --rm \
    -v $PWD/synapse-data:/data \
    -e SYNAPSE_SERVER_NAME=matrix.example.com \
    -e SYNAPSE_REPORT_STATS=no \
    matrixdotorg/synapse:latest generate

Remove/comment out the sqlite database and add the postgresql database section in synapse-data/homeserver.yaml so it looks like

#database:
#  name: sqlite3
#  args:
#    database: /data/homeserver.db
database:
  name: psycopg2
  args:
    user: synapse
    password: synapse
    database: synapse
    host: db # The service name defined in your docker-compose file
    cp_min: 5
    cp_max: 10

Run the following Docker command to generate mautrix-whatsapp config in mautrix-whatsapp:

docker run --rm -v $PWD/mautrix-whatsapp:/data:z dock.mau.dev/mautrix/whatsapp:latest

Edit mautrix-whatsapp/config.yaml and change addresses, domain and uri:

homeserver address: https://matrix.example.com
homeserver domain: matrix.example.com
appservice address: http://mautrix-whatsapp:29318
database uri: postgres://synapse:synapse@db/mautrix?sslmode=disable
Set your domain at the permissions section:

    permissions:
        "*": relay
        "matrix.example.com": user
        "@admin:matrix.example.com": admin

Run the following Docker command to generate registration file

docker run --rm -v $PWD/mautrix-whatsapp:/data:z dock.mau.dev/mautrix/whatsapp:latest

Copy the registration to a directory where synapse can read it and set owner:

cp mautrix-whatsapp/registration.yaml synapse-data/
chown 991:991 synapse-data/registration.yaml

Add the following registration entry at the end of synapse-data/homeserver.yaml

app_service_config_files:
- /data/registration.yaml

Add the firewall rule for port 8448 (used for federation):

config set synapse service status enabled TCPPort 8448 access red,green
signal-event firewall-adjust

Start the containers using docker-compose:

docker-compose up -d

Wait until containers are up or the following command won’t work because it needs the database container.

To stop the containers if needed, eg for restarting:

docker-compose down

Create mautrix database:

docker exec -it synapse-docker-db-1 psql -U synapse
CREATE DATABASE mautrix
 ENCODING 'UTF8'
 LC_COLLATE='C'
 LC_CTYPE='C'
 template=template0
 OWNER synapse;
\q

Install Element web client v1.11.20, check Releases · vector-im/element-web · GitHub for new releases:

wget https://github.com/vector-im/element-web/releases/download/v1.11.20/element-v1.11.20.tar.gz
tar -xzf element-v1.11.20.tar.gz
mv element-v1.11.20 /var/www/html/element
cp /var/www/html/element/config.sample.json /var/www/html/element/config.json
rm -f element-v1.11.20.tar.gz

Run the following Docker command to create a user:

docker exec -it synapse-docker-synapse-1 register_new_matrix_user http://localhost:8008 -c /data/homeserver.yaml

Login to the element web client at https://matrix.example.com/element

To connect to the Whatsapp bridge you need to start a chat to @whatsappbot:matrix.example.com.

help shows some information about usable commands.

login shows a QR Code to connect WhatsApp Web to the bridge. After connecting the bridge should work and you get the WA messages in your element client.

Synapse-admin

To make the synapse admin api work, we need to add a location directive to /etc/httpd/conf.d/a_synapse.conf:

<VirtualHost *:443>
        SSLEngine on
        ServerName example.com
        AllowEncodedSlashes NoDecode
        ProxyPreserveHost on
        ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon
        ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix

        <Location /_synapse/admin>
          Require all denied
          # Change the network as needed, we don't want to allow the synapse admin api to public.
          Require ip 192.168.1.0/24
          ProxyPass http://127.0.0.1:8008/_synapse/admin nocanon
          ProxyPassReverse http://127.0.0.1:8008/_synapse/admin
        </Location>

</VirtualHost>

The synapse-admin service needs to be added to the docker-compose.yml: (in this example it’s added after the synapse service)

services:

  synapse:
    image: docker.io/matrixdotorg/synapse:latest
    restart: unless-stopped
    volumes:
      - ./synapse-data:/data
      - ./shared_secret_authenticator.py:/usr/local/lib/python3.9/site-packages$
    depends_on:
      - db
    ports:
      - '127.0.0.1:8008:8008/tcp'

  synapse-admin:
    container_name: synapse-admin
    hostname: synapse-admin
    image: awesometechnologies/synapse-admin
    ports:
      - "8095:80"
    restart: unless-stopped

A reverse proxy is needed to reach the site:

Start the synapse admin:

docker-compose up -d synapse-admin

Now the Synapse Admin should be reachable via the reverse proxy. Login is possible with synapse admins only.

LayLow · January 21, 2023, 1:46am

And another gem is born. Thanks for that @mrmarkuz !

LayLow · January 21, 2023, 2:11am

from what file pls?

Is there perhaps an extra step in between ‘Generate…’ SYNAPSE_SERVER_NAME config file and ‘Remove/comment out…’?

Is 'Generate creating text file with nano or executing a command?

ps I replace in my test SYNAPSE_SERVER_NAME with ‘synapse_chat.myserver.com’, so lower case. Is that ok?

Thanks again

mrmarkuz · January 21, 2023, 10:36am

Thanks for testing. I’d like to add more bridges so I’m going to reorganize this howto to make the installation easier.

From synapse-data/homeserver.yaml

OK, that’s really confusing. I’m going to edit my post to make it more clear.

The synapse config file synapse-data/homeserver.yaml is created by starting the synapse container once by running docker run ....

There’s no extra step, after creating the file you can edit it.

Sorry, I meant to replace matrix.example.com with your domain so it should look like

docker run -it --rm \
    -v $PWD/synapse-data:/data \
    -e SYNAPSE_SERVER_NAME=synapse_chat.myserver.com \
    -e SYNAPSE_REPORT_STATS=no \
    matrixdotorg/synapse:latest generate

LayLow · January 22, 2023, 4:28pm

Should this not be ‘synapse-docker_db_1’

and

synapse-docker_synapse_1

I vaguely recall something was discussed about docker versions using different syntaxes…

Just a note. Anyway, element is up and running and so are the containers. But I can only register with the Matrix server and not on MY server/domain. New things to explore for me

LayLow · January 26, 2023, 8:02pm

registration.yaml is copied to synapse-data directory, but the original registration.yaml get’s 991 ownership? Odd or me not understanding?

mrmarkuz · January 26, 2023, 8:06pm

You are right, it should be

chown 991:991 synapse-data/registration.yaml

I corrected it.

LayLow · January 26, 2023, 9:46pm

I guess this should be ‘- /synapse-data/registration.yaml’ ?

/me hiding…

mrmarkuz · January 26, 2023, 9:47pm

No, it’s the path in the container.

From the docker-compose.yml - the local dir synapse-data points to the data dir in the container:

LayLow · January 28, 2023, 5:58pm

Maybe someone has a clue (@robb perhaps?) I followe d this how-to to the letter, but I am unable to create an account when accessing element and changing the homeserver to my domain (chat.myserver.com). It would display the below error message. Any clue what I am missing or do I expect something that is not possible? Thanks!

michelandre · January 28, 2023, 6:12pm

Hi @LayLow

If I remember well:

The file config.json should contain your server name as:

{
    "default_server_config": {
        "m.homeserver": {
            "base_url": "https://chat.myserver.com",
            "server_name": "chat.myserver.com"
        },
        "m.identity_server": {
            "base_url": "https://vector.im"
        }
    },
...

The file ...well-known/matrix/server should contain;

{ “m.server”: “chat.myserver.com:443” }

The port 443 may not be required.

At: https://federationtester.matrix.org/ , enter your domain for a test:
chat.myserver.com => Go.

Michel-André

P.S.
homeserver.yaml must contain:

enable_registration: true

P.P.S.

Did you clear the brower cache after changing the name.
You can register to https://chat.mozilla.org/ to make some tests.

LayLow · January 28, 2023, 7:46pm

Mercie Bien Michel-André,

Ok, where would this file be (path) please?

This file would be at the default apache path of NS I guess?

That shows me that I have very poor skills

Thanks for the links and assist. I’ll report back!

mrmarkuz · January 28, 2023, 8:28pm

Please also check the container logs for errors by using portainer or command line.

Show logs of db container for example:

docker logs synapse-docker-db-1

Show all synapse logs using docker-compose:

docker-compose logs

michelandre · January 28, 2023, 8:34pm

Hi @LayLow

The config.json in under the “client” element and you copy it from config.sample.json in the same directory or just rename it to config.json.

The file .well-known/matrix/server is in the default web directory: /var/www/html/.well-known/matrix/server.

When login in and the server name is not right you have to use https: i.e. https://chat.myserver.com

Don’t forget to clear the browser cache…

You can have a look at my document (translated in English by a web page which I don’t remember). The translation is not bad, I didn’t do any correction:
https://www.micronator.org/affaires/PDF/NethServer-701/RF-232_Cours_NethServer-701-01_Matrix_Synapse_2021-03-30_12h51_CHAT-EN.pdf.

Michel-André

LayLow · January 28, 2023, 9:59pm

Please see synapse_1 | Starting synapse with args -m synapse.app.homeserver --con - Pastebin.com

There is an error somewhere. I also learned that a yaml file CAN NOT contain any tabs, but idents have to be made by spaces.

See anything?

LayLow · January 28, 2023, 10:00pm

Hi @michelandre ,

Thanks so much for the details. Let me digest!

Thanks!

mrmarkuz · January 28, 2023, 10:07pm

There seem to be errors in your homeserver.yaml, check lines 12 and 29:

yaml.parser.ParserError: while parsing a block mapping
in "/data/homeserver.yaml", line 12, column 1
expected <block end>, but found '<block mapping start>'
in "/data/homeserver.yaml", line 29, column 3

You may also share your homeserver.yaml.

LayLow · January 28, 2023, 10:45pm

Please see DM. I will share the outcome here

danb35 · January 28, 2023, 11:15pm

Nope, no tabs, and indentation is critical. An indent of two spaces means something completely different than an indent of four spaces.

LayLow · January 28, 2023, 11:30pm

Ok…

I had a typo. I used ‘arg’ instead of ‘args’
I had to add suppress_key_server_warning: true (homeserver.yaml)
had to add enable_registration_without_verification: true
Got rid of 4 lines with tabs and replace the with space to the correct position

Obviously verification should be required to prevent spam and bots etc. ut for this testing purpose is allowed me to finally register.

The command ‘docker-compose logs’ is very informative.

Thanks for this time around y’all.