Mattermost outgoing webhook cannot reach n8n on the same node

Support
,
1

I am trying to set up an outgoing webhook from mattermost to n8n running on the same NS8 node, but it is not working. Webhooks sent to n8n from other machines in LAN do work well.
I suspect this may have something to do with inter-app communication within the same NS8 node, perhaps related to what is described here: Workfull weekend for tebbiworld repository - updates and new modules - #5 by thorsten
However, this is a bit beyond my knowledge of Podman/NS8 networking.

Is there a supported way to make Mattermost outgoing webhooks reach n8n when both applications are running on the same NS8 node?

2

In between, after some research, I have managed to make it work by inserting

169.254.1.2 n8n.mydomain.my

into the mattermost-app hosts file by running

MM_PID=$(/usr/local/bin/runagent -m mattermost1 podman inspect -f '{{.State.Pid}}' mattermost-app)
cp /proc/$MM_PID/root/etc/hosts /root/mattermost-hosts.backup
echo '169.254.1.2 n8n.mydomain.my' >> /proc/$MM_PID/root/etc/hosts

It works now, but it is just a temporary solution. Is there any recommendation for permanent one?

Allow untrusted internal connections to: in Mattermost System Console / Developer is set to
host.containers.internal n8n.mydomain.my

3

You could add --add-host n8n.mydomain.my:host-gateway to the service file to preconfigure the containers hosts file.

Edit the systemd service:

runagent -m mattermost1 systemctl --user edit --full mattermost

Add

--add-host n8n.mydomain.my:host-gateway \

so it looks like

...
ExecStartPre=/usr/bin/podman pod create --infra-conmon-pidfile %t/mattermost.pid \
    --pod-id-file %t/mattermost.pod-id \
    --name mattermost \
    --publish 127.0.0.1:${TCP_PORT}:8065 \
    "$POD_EXTRA_ARGS" \
    --add-host n8n.mydomain.my:host-gateway \
    --replace
...

Restart mattermost:

runagent -m mattermost1 systemctl --user restart mattermost

This way the pasta network is used.
This change is persistent until the next update or restore as it’s not included in the backup.

BTW, if you have some time, please help testing n8n, see NS8-n8n: Nodemation App - #13 by mrmarkuz

4

Thank you @mrmarkuz. Is it planned to include this in future updates?

5

I’m not sure. I think it’s still discussed.
When using the current pasta network, we need the N8N FQDN that accesses Mattermost so there’s no general fix that fits all cases.
When using the old slirp4netns network, the N8N FQDN isn’t needed but Mattermost is able to connect to every localhost service on the host which may be too much from a security perspective.

To make the changes surviving an update in the meanwhile, it’s possible to create a systemd override. This way the original systemd file may be overwritten but the override is kept after an update.
Here’s another override example: SOGo new features: configuration template and access/configure button - #33 by mrmarkuz

Create and edit the override.conf file:

systemctl --user edit mattermost

The override.conf file should look like:

### Editing /home/mattermost1/.config/systemd/user/mattermost.service.d/override.conf
### Anything between here and the comment below will become the new contents of the file

[Service]
ExecStartPre=
ExecStartPre=/bin/rm -f %t/mattermost.pid %t/mattermost.pod-id
ExecStartPre=/usr/bin/podman pod create --infra-conmon-pidfile %t/mattermost.pid \
    --pod-id-file %t/mattermost.pod-id \
    --name mattermost \
    --publish 127.0.0.1:${TCP_PORT}:8065 \
    "$POD_EXTRA_ARGS" \
    --add-host n8n.mydomain.my:host-gateway \
    --replace

### Lines below this comment will be discarded
...

There’s still a culprit. When an update includes a changed ExecStartPre= line in the original systemd file, it’s not applied due to the override which may in the worst case break the app.

6

Thank you @mrmarkuz for a somewhat more permanent solution.
Anyway, I would still suggest that NS8 provide a more standard - ideally WebGUI-configurable - way to permanently allow applications running on the same node to communicate with each other through their standard documented endpoints.

7

Looking at the pod definition in n8n.service, the app is “listen-local”: its port is bound to 127.0.0.1, so it is unreachable from the node’s other interfaces.

I’m not sure whether n8n would be less secure listening on all interfaces, but the VPN IP address (e.g. 10.5.4.1) would be simpler to reach from other apps and to integrate with them.

Choosing a listen-local or listen-all policy is the developer’s call; it is not something to expose in the UI.

Now let’s look at Mattermost. It sends webhooks, so it needs a network configuration that allows connecting to apps on the same node. Since n8n is currently listen-local, the only route to it is through the node’s Traefik instance. The n8n FQDN must resolve to a Traefik address other than the node’s main IP. Pasta assigns that IP to the container itself, so connections to it never leave the Mattermost container. A simple, persistent workaround is a line in the node’s /etc/hosts:

10.5.4.1  n8n.example.com     # use the node VPN IP address to reach Traefik

Then restart the Mattermost app so that it picks up the /etc/hosts change. Keep n8n.example.com (or 10.5.4.1) in Mattermost’s “Allow untrusted internal connections” setting.

If n8n switches to a listen-all policy, the /etc/hosts change is no longer required: the Mattermost webhook can point straight to http://10.5.4.1:<n8n port> or http://cluster-localnode:<n8n port>.

8

Thank you, confirmed — this solution works.