Matrix - synapse

This is a WIP for creating an install howt for Matrix - Synapse messaging server on NethServer 7.
A startingpoint could be: https://www.vultr.com/docs/create-a-chat-server-using-matrix-synapse-and-riot-on-centos-7
and the official install howto on github: synapse/INSTALL.md at master · matrix-org/synapse · GitHub
please feel free to add in this topic steps to install on NethServer.
This is a wikified post.

The goal is to have a Matrix - Synapse module for NethServer with the option to both have access with a local accountprovider (either LDAP or Samba4 AD) as internal messaging server AND the option to have conversations with other Matrix - Synapse servers through federation

ToDo

TEST : - Voice/video chat with TURN server
Element Call

Prerequisites

You need a valid letsencrypt cert to let the clients connect. You may use Matrix Synapse with your default domain but for avoiding virtualhost conflicts I recommend to use a separate (sub)domain like matrix.example.org

I tested with Riot client on Windows and Android. Encryption is working.

Installing needed packages and dev tools

yum -y install libtiff-devel libjpeg-devel libzip-devel freetype-devel lcms2-devel libwebp-devel tcl-devel tk-devel redhat-rpm-config python-virtualenv libffi-devel openssl-devel postgresql-devel
yum -y groupinstall "Development Tools"

Install postgres 10.0, create DB and user synapse

You may replace SECRET with a more secure password.

yum -y install rh-postgresql10
scl enable rh-postgresql10 bash
postgresql-setup --initdb --port=55433
exit
cat << EOF > /var/opt/rh/rh-postgresql10/lib/pgsql/data/pg_hba.conf
# "local" is for Unix domain socket connections only
local   all         postgres                          ident
local   all         all                               md5
# IPv4 local connections:
host    all         all         127.0.0.1/32          md5
# IPv6 local connections:
host    all         all         ::1/128               md5
EOF
systemctl enable rh-postgresql10-postgresql --now
su - postgres -c "scl enable rh-postgresql10 -- psql --port=55433"
create user synapse with encrypted password 'SECRET';
CREATE DATABASE synapse
 ENCODING 'UTF8'
 LC_COLLATE='C'
 LC_CTYPE='C'
 template=template0
 OWNER synapse;
\q
Alternative install of postgres92 via nethserver-postgresql
yum -y install nethserver-postgresql postgresql-devel
su - postgres
createuser synapse
psql -c "alter user synapse with encrypted password 'SECRET';"
psql
CREATE DATABASE synapse
 ENCODING 'UTF8'
 LC_COLLATE='C'
 LC_CTYPE='C'
 template=template0
 OWNER synapse;
\q
exit

Install synapse

yum -y install rh-python36
mkdir -p /opt/synapse
scl enable rh-python36 bash
virtualenv -p python3.6 /opt/synapse
source /opt/synapse/bin/activate
pip install --upgrade pip
pip install --upgrade setuptools
pip install matrix-synapse
pip install Jinja2
alternative install with python2.7
mkdir -p /opt/synapse
virtualenv -p python2.7 /opt/synapse
source /opt/synapse/bin/activate
pip install --upgrade pip
pip install --upgrade setuptools
pip install matrix-synapse
pip install Jinja2

Install postgres client libs

pip install matrix-synapse[postgres]

Create config file

Replace example.org with the domain you want to use:

cd /opt/synapse
python -m synapse.app.homeserver \
    --server-name example.org \
    --config-path homeserver.yaml \
    --generate-config \
    --report-stats=no

Edit homeserver.yaml to use postgresql, search for database:, edit name and add/edit the args as needed:

database:
    name: psycopg2
    args:
        user: synapse
        password: SECRET
        database: synapse
# specify port because rh-postgresql10 doesn't run on default 5432 port
        host: localhost
        port: 55433
        cp_min: 5
        cp_max: 10

Configure httpd

We use a reverse proxy to make synapse accessible at https://example.org. As it has some special options it cannot be added via web UI. Don’t forget to replace example.org with the domain you use.

cat << EOF > /etc/httpd/conf.d/a_synapse.conf
<VirtualHost *:443>
        SSLEngine on
        ServerName example.org

        AllowEncodedSlashes NoDecode
        ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon
        ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix
</VirtualHost>

Listen 8448

<VirtualHost *:8448>
        SSLEngine on

        AllowEncodedSlashes NoDecode
        ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon
        ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix
</VirtualHost>
EOF

Restart httpd to apply the new config:

systemctl restart httpd

Open port for synapse federation

config set synapse service status enabled TCPPort 8448 access red,green
signal-event firewall-adjust

Create, enable and start systemd service

cat << EOF > /etc/systemd/system/matrix-synapse.service
[Unit]
Description=Matrix Synapse service
After=network.target

[Service]
Type=forking
WorkingDirectory=/opt/synapse/
ExecStart=/opt/synapse/bin/synctl start
ExecStop=/opt/synapse/bin/synctl stop
ExecReload=/opt/synapse/bin/synctl restart
Restart=always
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=synapse

[Install]
WantedBy=multi-user.target
EOF

systemctl enable matrix-synapse --now

Test synapse

Browse to https://example.org/_matrix/key/v2/server/auto and you should get JSON output like

{"old_verify_keys":{},"server_name":"example.org"...

Register local synapse users

To register a new (admin) user:

register_new_matrix_user -c homeserver.yaml http://localhost:8008

Now you are ready to connect to the server with a matrix synapse client like the Riot web app.

Element web app

wget https://github.com/vector-im/element-web/releases/download/v1.10.11/element-v1.10.11.tar.gz
tar -xzf element-v1.10.11.tar.gz
mv element-v1.10.11 /var/www/html/element
cp /var/www/html/element/config.sample.json /var/www/html/element/config.json
rm -f element-v1.10.11.tar.gz

Edit /var/www/html/element/config.json to match your domain:

    "default_server_config": {
        "m.homeserver": {
            "base_url": "https://example.org",
            "server_name": "example.org"
        },
        "m.identity_server": {
            "base_url": "https://matrix.org"

Browse to https://example.org/element and login to your web chat app.

Federation test

If you’re able to connect to room #synapse:matrix.org, federation should work.
You may test with federation tester too: https://federationtester.matrix.org/

TurnServer

If you have epel disabled:

yum --enablerepo=epel install coturn

yum install nano coturn
openssl rand -hex 32

nano /etc/coturn/turnserver.conf

listening-port=3478
fingerprint
use-auth-secret
static-auth-secret=<yourChosen/GeneratedSecret>
realm=server.domain.tld
bps-capacity=0
stale-nonce=600
no-multicast-peers
config set coturn service status enabled TCPPort 3478 UDPPort 3478 access public
signal-event firewall-adjust
systemctl enable --now coturn
systemctl restart coturn

Coturn Synapse setup

Your homeserver configuration file needs the following extra keys:

  1. " turn_uris ": This needs to be a yaml list of public-facing URIs for your TURN server to be given out to your clients. Add separate entries for each transport your TURN server supports.
  2. " turn_shared_secret ": This is the secret shared between your homeserver and your TURN server, so you should set it to the same string you used in turnserver.conf.
  3. " turn_user_lifetime ": This is the amount of time credentials generated by your homeserver are valid for (in milliseconds). Shorter times offer less potential for abuse at the expense of increased traffic between web clients and your homeserver to refresh credentials. The TURN REST API specification recommends one day (86400000).
  4. " turn_allow_guests ": Whether to allow guest users to use the TURN server. This is enabled by default, as otherwise VoIP will not work reliably for guests. However, it does introduce a security risk as it lets guests connect to arbitrary endpoints without having gone through a CAPTCHA or similar to register a real account.

As an example, here is the relevant section of the config file for matrix.org . The turn_uris are appropriate for TURN servers listening on the default ports, with no TLS.

turn_uris: [ "turn:turn.matrix.org?transport=udp", "turn:turn.matrix.org?transport=tcp" ]
turn_shared_secret: "n0t4ctuAllymatr1Xd0TorgSshar3d5ecret4obvIousreAsons"
turn_user_lifetime: 86400000
turn_allow_guests: True

After updating the homeserver configuration, you must restart synapse:
systemctl restart matrix-synapse
Reference: synapse/turn-howto.md at develop · matrix-org/synapse (github.com)

LDAP/AD

You may use LDAP/AD as userbase. This way both, local synapse users and LDAP/AD users, are possible.

For remote AD/LDAP you need to use LDAPS and port 636 and for AD maybe a cert, I didn’t test remote AD/LDAP…

Install the ldap module:

pip install matrix-synapse-ldap3

Uncomment the password providers settings in /opt/synapse/homeserver.yaml (search for password_providers:) and edit to your needs.

Replace BINDPASSWORD with your AD/LDAP bind user password.

Local LDAP

password_providers:
    - module: "ldap_auth_provider.LdapAuthProvider"
      config:
        enabled: true
        uri: "ldap://localhost:389"
        start_tls: true
        base: "ou=People,dc=directory,dc=nh"
        attributes:
           uid: "uid"
           mail: "email"
           name: "sn"
        bind_dn: "cn=ldapservice,dc=directory,dc=nh"
        bind_password: BINDPASSWORD

Local AD

password_providers:
    - module: "ldap_auth_provider.LdapAuthProvider"
      config:
        enabled: true
        uri: "ldap://nsdc-testserver.ad.domain.local"
        start_tls: true
        base: "dc=ad,dc=domain,dc=local"
        attributes:
           uid: "cn"
           mail: "email"
           name: "samaccountname"
        bind_dn: "ldapservice@ad.domain.local"
        bind_password: BINDPASSWORD

Apply config:

systemctl restart matrix-synapse

Bridges

matrix-appservice-discord - work in progress

Get nodejs 10 and matrix and compile:

cd ~
yum install rh-nodejs10
git clone https://github.com/Half-Shot/matrix-appservice-discord.git
cd matrix-appservice-discord
scl enable rh-nodejs10 bash
npm install
npm run build
cp config/config.sample.yaml ./config.yaml

Edit config.yaml to set your domain:

bridge:
    domain: "example.com"
    homeserverUrl: "https://example.com"

Generate discord-registration.yaml and copy to right place:

node build/src/discordas.js -r -u "http://localhost:9005" -c config.yaml
cp discord-registration.yaml /opt/synapse/

Apply discord bridge to synapse config:

Edit /opt/synapse/homeserver.yaml and add the

app_service_config_files:
  - "discord-registration.yaml"

Now I’m stuck at the docker run.

docker run -v /root/matrix-appservice-discord:/data halfshot/matrix-appservice-discord

I get following error. It seems the container does not get an IP address.

Mar-20 21:34:40.721 [ClientFactory] error: Could not login as the bot user. This is bad! Error: getaddrinfo EAI_AGAIN discordapp.com
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)

mautrix-telegram - work in progress

This is an example for a simple matrix bridge.

yum -y install http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-5.el7.nux.noarch.rpm
yum -y install ffmpeg
su - postgres -c "scl enable rh-postgresql10 -- psql --port=55433"
create user mautrixtelegram with encrypted password 'SECRET';
CREATE DATABASE mautrixtelegram
 ENCODING 'UTF8'
 LC_COLLATE='C'
 LC_CTYPE='C'
 template=template0
 OWNER mautrixtelegram;
\q
mkdir /opt/mautrix-telegram
cd /opt/mautrix-telegram
scl enable rh-python36 bash
virtualenv .
source ./bin/activate
pip install --upgrade mautrix-telegram[all]
cp example-config.yaml config.yaml

Create a telegram bot at Bots: An introduction for developers to get api id and hash.

Open config.yaml and configure homeserver settings, api_id/api_hash, look for example.org and replace with your domain.

Add postgres settings:

database: postgres://mautrixtelegram:SECRET@localhost:55433/mautrixtelegram

Create registration file with following command:

python -m mautrix_telegram -g

Edit ../synapse/homeserver.yaml (The matrix homeserver config):

app_service_config_files:
  - "discord-registration.yaml"
  - "/opt/mautrix-telegram/registration.yaml"

Create the database with alembic upgrade head . If you have a custom config path, use alembic -x config=/path/to/config.yaml upgrade head

Restart matrix-synapse and start mautrix_telegram

systemctl restart matrix-synapse
python -m mautrix_telegram

Go to your Telegram and talk to your bot and check the output on the screen.

Clean routine

Delete Matrix Synapse to make reinstall possible if something failed badly:

systemctl stop matrix-synapse
rm -Rf /opt/synapse
su - postgres -c "scl enable rh-postgresql10 -- psql --port=55433"
drop database synapse;
drop role synapse;
\q

Updating Element Web

v1.10.11

rm -rf /var/www/html/element/config.bkp.json
cp /var/www/html/element/config.json /var/www/html/element/config.bkp.json
wget https://github.com/vector-im/element-web/releases/download/v1.10.11/element-v1.10.11.tar.gz
tar -xzf element-v1.10.11.tar.gz
mv element-v1.10.11.tar.gz /var/www/html/element
cp /var/www/html/element/config.bkp.json /var/www/html/element/config.json
rm -f element-v1.10.11.tar.gz
5 Likes

A big THANK YOU @mrmarkuz for the first steps of this howto.
Would it be possible to add a reverse proxy config so LE certs can be used behind a router on a private subnet?
oops… you already have added that
@mrmarkuz: did you also see the warning that puthon 2.7 will be deprecated:

DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at Release process - pip documentation v24.1.dev0

1 Like

Yes, it’s already on the todo list, I’ll give it a try with python3 asap. Postgres 9.5 is recommended and we use 9.2. but it’s working so far.

i think ill wait for the module…

tried following the instructions multiple times. i cant get past

======
Job for matrix-synapse.service failed because the control process exited with error code. S ee "systemctl status matrix-synapse.service" and "journalctl -xe" for details.

========
Not sure where am going wrong.

looking forward to testing that.

EDIT:
i was able to resolve that one.
issue was

had not been defined properly… thanks for the clean script

You may run synctl start to start synapse manually and check the output for errors.
Or check with systemctl status matrix-synapse -l

I put all lines that need to be performed in terminal between [code] tags. Am I right this will avoid parsing problems?

so far everything has worked, to the point i can login to riot.

But, the testing of synapse returns 404.
while login to riot on local says cannot reach homeserver.

what could be the error?

Is matrix synapse running?
systemctl status matrix-synapse -l
Is rh-postgresql10-postgresql running?
There are log files in /opt/synapse, please check them for errors…
If you copy/paste to terminal be careful that every command runs because systemctl commands cut off following commands.

looks like all is ok…
but still not able to have online test working. that weired

Hm I think I forgot the firewall settings.

Please try:

config set synapse service status enabled TCPPort 8448 access green,red
signal-event firewall-adjust

i have run the command. still getting same behavior

Thanks for testing, the problem was the httpd conf file order, I’ll update the howto…

2 Likes

what is identity provider, i saw a notification about it not having bein genabled.

Please try to change the identity server in riot config /var/www/html/riot/config.json to matrix.org instead of your domain and check if the notification goes away. I assumed that our server is an identity server too but maybe I was wrong…

"m.identity_server": {
            "base_url": "https://matrix.org"
1 Like

now it works great,
i want to test ldap and users created on nethserver if they will be able to automatically login.
then i test if the video and voice call functions work.

2 Likes

@robb I don’t know if parsing is really easier with code tags. Most of the lines are not source code (bash commands/config files) so the syntax highlighting does not have the intended effect.

I had some parsing issues, mostly because of extra spaces in the txt formatting

I am running into a small issue: On a subscription server the repo where Development Tools reside is not configured. What repo do I need to add?

Hm, it’s a default centos yum group of packages.

What do you get?

yum group list

yum groupinfo "Development Tools"

Maybe this works:

yum --setopt=group_package_types=mandatory,default,optional groupinstall "Development Tools"

Source:

It’s a subscription server. Only nethserver repo’s are listed
@davidep @giacomo any suggestions?

Loaded plugins: changelog, fastestmirror, nethserver_events
Loading mirror speeds from cached hostfile
 * sb-base: u2.nethserver.com
 * sb-centos-sclo-rh: u2.nethserver.com
 * sb-centos-sclo-sclo: u2.nethserver.com
 * sb-epel: u2.nethserver.com
 * sb-extras: u2.nethserver.com
 * sb-nethserver-base: u2.nethserver.com
 * sb-nethserver-updates: u2.nethserver.com
 * sb-updates: u2.nethserver.com
Installed Groups:
   Backup
   Backup restore
   Basic firewall
   Dutch language
   Email
   FTP server
   Fail2ban
   Instant messaging
   MariaDB (MySQL) server
   Mattermost
   Netdata
   NethServer subscription
   New Server Manager (Beta)
   Nextcloud
   OpenVPN
   Report (Beta)
   SMTP proxy
   Statistics
   Web hosting
Available Groups:
   Bandwidth monitor
   Dedalo Hotspot
   Deep packet inspection (DPI)
   Fax server
   File server
   French language
   German language
   Greek language
   Hungarian language
   IPsec tunnels
   Intrusion Prevention System
   Italian language
   POP3 connector
   POP3 proxy
   Portuguese language
   Print server
   Reverse proxy
   Romanian language
   Roundcube web mail
   Russian language
   SNMP server
   Serbian language
   Spanish language
   Turkish language
   UPS support
   VoIP PBX
   Web filter
   Web proxy
   WebTop 5 groupware
Done
yum groupinfo "Development Tools"
Loaded plugins: changelog, fastestmirror, nethserver_events
Loading mirror speeds from cached hostfile
 * sb-base: u2.nethserver.com
 * sb-centos-sclo-rh: u2.nethserver.com
 * sb-centos-sclo-sclo: u2.nethserver.com
 * sb-epel: u2.nethserver.com
 * sb-extras: u2.nethserver.com
 * sb-nethserver-base: u2.nethserver.com
 * sb-nethserver-updates: u2.nethserver.com
 * sb-updates: u2.nethserver.com
Warning: group/environment Development Tools does not exist.

has anyone managed to get ldap local users to be able to login into the app?