Thanks to @mrmarkuz for helping figure this one out. I’m doing my hosting on Hetzner, and it seems they do some customizations to the base CentOS image which can conflict with NethServer (more on that later).
The CentOS image on Hetzner makes a change to the standard Kernel logging settings which causes Shorewall messages to be put directly in the console, instead of in /var/log/messages where they belong.
To fix it, edit /etc/sysctl.conf and add this at the end:
kernel.printk = 4 4 1 7
And then activate it by running sysctl -p