[Solved] Mail problems after update

paul_marwick · April 11, 2019, 10:24am

One my recently migrated Nethserver 7.6 machine, I ran a small update a little while ago. Results are catastrophic and I don’t know how to revert them. Update was as follows:

   Apr 11 10:34:28 Updated: nethserver-mail-common-2.5.0-1.ns7.noarch
   Apr 11 10:34:30 Updated: rspamd-1.9.1-1.x86_64
   Apr 11 10:34:31 Updated: nethserver-mail-filter-2.5.0-1.ns7.noarch
   Apr 11 10:34:32 Updated: nethserver-mail-quarantine-2.5.0-1.ns7.noarch
   Apr 11 10:34:32 Updated: nethserver-mail-server-2.5.0-1.ns7.noarch
   Apr 11 10:34:33 Updated: nethserver-mail-disclaimer-2.5.0-1.ns7.noarch
   Apr 11 10:34:34 Updated: nethserver-mail-smarthost-2.5.0-1.ns7.noarchbn

As of now, all incoming mail seems to be being blocked, and none of the accounts I’ve tried can send mail either. From Seamonkey, I get “login to mail.brillcomputers.co.uk failed” and a prompt to enter a new password. Doing so does nothing. I’ve change my password from the web interface, but that doesn’t help at all.

This is a real show sotpper - I don’t have the knowledge to revert this mornings changes, and this is a production server. Help…

For the moment, I think I have no choice but to put the SME server back in place. But this sort of disaster is something that I cannot afford to have happen anywhere else.

pike · April 11, 2019, 10:26am

@dev_team calling
and @paul_marwick consider the option for a subscription to avoid a too early update to get to your server and mess up with it.

federico.ballarini · April 11, 2019, 10:27am

The sender receive a notification or nothing?
Can you check through rspamd if email results blocked or sent? Check also email queue.
Users can login into roundcube?

stephdl · April 11, 2019, 10:32am

Yep we need also logs lines in /var/log/maillog, pleas do use a pastebin to display the lig nicely

giacomo · April 11, 2019, 10:40am

If you are on hurry, revert the updates.

Find the yum transaction:
```
yum history
```
Pick the right transaction and undo it:
```
yum history undo <transaction>
```

paul_marwick · April 11, 2019, 12:48pm

@pike

Not my decision. Not the owner of the network, just the admin. Might
consider it, but Neth has to prove itself first.

pike · April 11, 2019, 12:53pm

Therefore, as admin, disable automatic updates and review it manually every time.

paul_marwick · April 11, 2019, 2:28pm

federico.ballarini:

paul_marwick:

As of now, all incoming mail seems to be being blocked, and none of the accounts I’ve tried can send mail either. From Seamonkey, I get “login to mail.brillcomputers.co.uk failed” and a prompt to enter a new password. Doing so does nothing. I’ve change my password from the web interface, but that doesn’t help at all.

1. The sender receive a notification or nothing?

Nothing. I sent a test message from my Gmail account - it didn’t arrive, nor did it bounce.

2. Can you check through rspamd if email results blocked or sent? Check also email queue.

There doesn’t seem to be an rspamd log, though there is a directory under /var/log for it. Looking through /var/log/messages, I see this, just after the update completed:

Apr 11 10:34:28 barracuda useradd[4211]: failed adding user '_rspamd', exit code: 9

It seems to have failed to read its config and failed at that point

[quote]3. Users can login into roundcube?

No roundcube installed on this server. I was intending to install sogo eventually, but wanted to sort other things out first. Users were able to log in and read mail, but unable to send anything, and no new mail was being accepted.

I’ve put a cut version of the maillog here: https://pastebin.com/tiCB8rFL
I’m not going to be able to get back to the site until Sunday afternoon at the earliest, and since I’ve had to put the SME server back up, the Nethserver isn’t online at the moment - I would have to change IP assignments before I can put it back up.

paul_marwick · April 11, 2019, 2:42pm

Thanks - if all else fails, I can try that when I get back to the site.

paul_marwick · April 11, 2019, 2:43pm

No automatic updates on the system. Not at all sure what “manual review” I could do on packages straight from the neth repos.

stephdl · April 11, 2019, 3:04pm

Apr 11 11:25:18 barracuda postfix/smtpd[7880]: warning: connect to Milter service unix:/var/run/rspamd/worker-proxy: No such file or directory

Restart rspamd it is down

federico.ballarini · April 11, 2019, 3:49pm

systemctl start rspamd

If you want to enable it: systemctl enable rspamd

Post error if something is wrong

GG_jr · April 11, 2019, 4:05pm

Hi guys,

I made those updates this morning, on two dedicated email servers.
Nothing wrong till now.

Best regards,
Gabriel

stephdl · April 11, 2019, 4:39pm

just did myself, and nothing wrong

systemctl status rspamd
● rspamd.service - rapid spam filtering system
   Loaded: loaded (/usr/lib/systemd/system/rspamd.service; enabled; vendor preset: enabled)
  Drop-In: /usr/lib/systemd/system/rspamd.service.d
           └─nethserver.conf
   Active: active (running) since Thu 2019-04-11 18:17:58 CEST; 20min ago
     Docs: https://rspamd.com/doc/
 Main PID: 7034 (rspamd)
   CGroup: /system.slice/rspamd.service
           ├─7034 rspamd: main process
           ├─7154 rspamd: log_helper process
           ├─7155 rspamd: fuzzy process (localhost:11335)
           ├─7156 rspamd: rspamd_proxy process (/var/run/rspamd/worker-proxy mode=0770 owner=_rspamd group=mail)
           └─7157 rspamd: controller process (127.0.0.1:11334)


[root@prometheus ~]# rpm -qa | grep nethserver-mail
nethserver-mail-p3scan-2.5.0-1.ns7.noarch
nethserver-mail-smarthost-2.5.0-1.ns7.noarch
nethserver-mail-filter-2.5.0-1.ns7.noarch
nethserver-mail-getmail-2.5.0-1.ns7.noarch
nethserver-mail-server-2.5.0-1.ns7.noarch
nethserver-mail-disclaimer-2.5.0-1.ns7.noarch
nethserver-mail-common-2.5.0-1.ns7.noarch

davidep · April 11, 2019, 5:13pm

Same here!

Otherwise no release would have been done!

paul_marwick · April 11, 2019, 6:15pm

I’ve been going through /var/log/messages. I think I’ve found the problem, though I have no idea what could have cause it. As below:

   Apr 11 10:34:39 barracuda rspamd: 2019-04-11 10:34:39 #4264(main) cfg; rspamd_config_read: failed    to load config: ucl parser error: error while parsing /etc/rspamd/local.d/multimap.conf: line: 101, column: 10 - 'invalid character in a key', character: '0x0a'
Apr 11 10:34:39 barracuda systemd: rspamd.service: main process exited, code=exited, status=1/FAILURE
Apr 11 10:34:39 barracuda systemd: Unit rspamd.service entered failed state.
Apr 11 10:34:39 barracuda systemd: rspamd.service failed.
Apr 11 10:34:39 barracuda systemd: rspamd.service holdoff time over, scheduling restart.
Apr 11 10:34:39 barracuda systemd: Stopped rapid spam filtering system.
Apr 11 10:34:39 barracuda systemd: Started rapid spam filtering system.

Looks as though something corrupted the rspamd config. No idea why that should have happened, but hopefully that will be reasonably easy to fix. Then I’ll just have to problem of getting the mail back in sync after several days of running the old SME server.

stephdl · April 11, 2019, 6:44pm

   Apr 11 10:34:39 barracuda rspamd: 2019-04-11 10:34:39 #4264(main) cfg; rspamd_config_read: failed    to load config: ucl parser error: error while parsing /etc/rspamd/local.d/multimap.conf: line: 101, column: 10 - 'invalid character in a key', character: '0x0a'

the line 101 doesn’t exist on my server…could you paste the raw of that config file

dnutan · April 11, 2019, 8:05pm

0x0a corresponds to a new line character (\n).

paul_marwick · April 11, 2019, 8:31pm

stephdl:

   Apr 11 10:34:39 barracuda rspamd: 2019-04-11 10:34:39 #4264(main) cfg; rspamd_config_read: failed    to load config: ucl parser error: error while parsing /etc/rspamd/local.d/multimap.conf: line: 101, column: 10 - 'invalid character in a key', character: '0x0a'

the line 101 doesn’t exist on my server…could you paste the raw of that config file

Hopefully be able to sometime tomorrow - I had to pull the server off the network because of IP clashes. I’m hoping to get to the site late tomorrow morning at least for a little while, at which point I’ll be able to pull a copy.

paul_marwick · April 11, 2019, 8:32pm

Yes. I recognised it. From the error, it sounds as though something has split a line. I’m not sure what could have happened - it was working fine before the update.