Squidclamav doesn't start - hangs during acitvating and takes 100% CPU (1 core)

flatspin · January 10, 2019, 1:29pm

NethServer Version: 7.5.1804 final
Module: webfilter / squidclamav

I wanted to test update clamav from 0.100.2 to 0.101.1 on a test-vm, but took accidentally the ssh-window of my production machine. I reverted back to clamav 0.100.2.

# rpm -qa clam*
clamav-filesystem-0.100.2-2.el7.noarch
clamav-lib-0.100.2-2.el7.x86_64
clamav-0.100.2-2.el7.x86_64
clamav-update-0.100.2-2.el7.x86_64
clamav-unofficial-sigs-3.7.2-1.el7.noarch
clamd-0.100.2-2.el7.x86_64
clamav-data-0.100.2-2.el7.noarch
clamav-scanner-systemd-0.100.2-2.el7.x86_64
clamav-server-systemd-0.100.2-2.el7.x86_64

But since then I can’t start webfilter-antivirus.

I removed nethserver-squidclamav and reinstalled it, but no success.
I removed the hole nethserver-webfilter-module and reinstelld it, but no success.
Same with nethserver-antivirus.

#systemctl status squidclamav
● clamd@squidclamav.service - clamd scanner (squidclamav) daemon
   Loaded: loaded (/usr/lib/systemd/system/clamd@.service; static; vendor preset: disabled)
  Drop-In: /etc/systemd/system/clamd@squidclamav.service.d
           └─c-icap.conf
   Active: activating (start) since Thu 2019-01-10 13:56:44 CET; 1min 22s ago
     Docs: man:clamd(8)
           man:clamd.conf(5)
           https://www.clamav.net/documents/
  Control: 23824 (clamd)
   CGroup: /system.slice/system-clamd.slice/clamd@squidclamav.service
           └─23824 /usr/sbin/clamd -c /etc/clamd.d/squidclamav.conf

Jan 10 13:56:44 nethserver.jeckel.local systemd[1]: Starting clamd scanner (squidclamav) daemon...
Jan 10 13:56:44 nethserver.jeckel.local clamd[23824]: Received 0 file descriptor(s) from systemd.
Jan 10 13:56:44 nethserver.jeckel.local clamd[23824]: clamd daemon 0.100.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Jan 10 13:56:44 nethserver.jeckel.local clamd[23824]: Running as user c-icap (UID 987, GID 982)
Jan 10 13:56:44 nethserver.jeckel.local clamd[23824]: Log file size limited to 1048576 bytes.
Jan 10 13:56:44 nethserver.jeckel.local clamd[23824]: Reading databases from /var/lib/squidclamav
Jan 10 13:56:44 nethserver.jeckel.local clamd[23824]: Not loading PUA signatures.
Jan 10 13:56:44 nethserver.jeckel.local clamd[23824]: Bytecode: Security mode set to "TrustSigned".

When I kill this task, c-icap server is reported running

#systemctl status c-icap
  ● c-icap.service - C-ICAP Server
       Loaded: loaded (/usr/lib/systemd/system/c-icap.service; enabled; vendor preset: disabled)
      Drop-In: /etc/systemd/system/c-icap.service.d
               └─squidclamav.conf
       Active: active (running) since Thu 2019-01-10 13:58:27 CET; 11min ago
      Process: 24088 ExecStart=/usr/sbin/c-icap $OPTIONS (code=exited, status=0/SUCCESS)
     Main PID: 24089 (c-icap)
        Tasks: 37
       CGroup: /system.slice/c-icap.service
               ├─24089 /usr/sbin/c-icap -f /etc/c-icap/c-icap.conf
               ├─24090 /usr/sbin/c-icap -f /etc/c-icap/c-icap.conf
               ├─24091 /usr/sbin/c-icap -f /etc/c-icap/c-icap.conf
               └─24092 /usr/sbin/c-icap -f /etc/c-icap/c-icap.conf

    Jan 10 13:58:27 nethserver.jeckel.local systemd[1]: Starting C-ICAP Server...
    Jan 10 13:58:27 nethserver.jeckel.local systemd[1]: Started C-ICAP Server.

Webfilter itself works. Only the antivirus doesn’t work!

Any advice for me to bring back this featur @giacomo or @filippo_carletti ?
Thanks a lot.

giacomo · January 10, 2019, 1:48pm

Try to manually start the clamd instance and see if there is any relevant error:

/usr/sbin/clamd --debug -F -c /etc/clamd.d/squidclamav.conf

flatspin · January 10, 2019, 3:25pm

Hi giacomo, thanks for reply.

Started manually

From messages log:

Jan 10 16:04:57 nethserver clamd[14180]: Received 0 file descriptor(s) from systemd.
Jan 10 16:04:57 nethserver clamd[14180]: clamd daemon 0.100.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Jan 10 16:04:57 nethserver clamd[14180]: Running as user c-icap (UID 987, GID 982)
Jan 10 16:04:57 nethserver clamd[14180]: Log file size limited to 1048576 bytes.
Jan 10 16:04:57 nethserver clamd[14180]: Reading databases from /var/lib/squidclamav
Jan 10 16:04:57 nethserver clamd[14180]: Not loading PUA signatures.
Jan 10 16:04:57 nethserver clamd[14180]: Bytecode: Security mode set to "TrustSigned".

But still CPU 100% . Where to find the debug info?

flatspin · January 10, 2019, 4:14pm

When I start clamd manually with a minimal config

User clanscan
LocalSocket /var/run/cland.scan/clamd.sock

clamd starts normally and I get this:

Limits: Global size limit set to 104857600 bytes.
Limits: File size limit set to 26214400 bytes.
Limits: Recursion level limit set to 16.
Limits: Files limit set to 10000.
Limits: MaxEmbeddedPE limit set to 10485760 bytes.
Limits: MaxHTMLNormalize limit set to 10485760 bytes.
Limits: MaxHTMLNoTags limit set to 2097152 bytes.
Limits: MaxScriptNormalize limit set to 5242880 bytes.
Limits: MaxZipTypeRcg limit set to 1048576 bytes.
Limits: MaxPartitions limit set to 50.
Limits: MaxIconsPE limit set to 100.
Limits: MaxRecHWP3 limit set to 16.
Limits: PCREMatchLimit limit set to 100000.
Limits: PCRERecMatchLimit limit set to 5000.
Limits: PCREMaxFileSize limit set to 26214400.
Archive support enabled.
BlockMax heuristic detection disabled.
Algorithmic detection enabled.
Portable Executable support enabled.
ELF support enabled.
Mail files support enabled.
OLE2 support enabled.
PDF support enabled.
SWF support enabled.
HTML support enabled.
XMLDOCS support enabled.
HWP3 support enabled.
Self checking every 600 seconds.

EDIT:
@giacomo thanks for the hint with manually starting.

I found in /var/lib/squidclamav (= DatabaseDirectory) a lot of file like main.cld, and tmp files.
I deleted them and left only the same as on a fresh install. When I then start manually it works.

But when I try to start it via GUI, the files where downloaded again and it won’t start.

Do you know which part to check to prevent downloading files when start squidclamav from GUI?

TIA Ralf

flatspin · January 11, 2019, 7:26am

@giacomo found the solution: signal-event nethserver-squidclamav-update did the trick.
Now everything is in the right place again and services are running from GUI again.

giacomo · January 11, 2019, 8:57am

Good to hear!
I just tried to reproduce the problem but I failed

flatspin · January 11, 2019, 10:42am

I think it was because of subscription.
As I updated I was wondering why I had to enable epel separately, but my fingers were quicker then my brain.

tmb · March 19, 2019, 5:21pm

With “Graphs” and “top” in SSH I noticed since two days (could have been longer) that Clamd is at 100% every afternoon more than two hours. Your solution seems to resolve the issue on my NS too. Thanks

tmb · March 20, 2019, 10:50am

Oops - the issue is back again (100% Cpu load from clamd) this morning after reboot so the command signal-event nethserver-squidclamav-update did not solve the issue - only temporarily (NS7.6)

flatspin · March 20, 2019, 12:17pm

Hi Thomas,

sorry to hear that. Please post output of:

rpm -qa | grep clam
systemctl status clamd@squidclamav

Also please verify that it’s clamd@squidclamav (/usr/sbin/clamd -c /etc/clam.d/squidclamav.conf) that hangs.
Did you try to update clamav manually?

tmb · March 20, 2019, 3:13pm

Since it was annoying I removed nethserver-squidclamav

After a new install I see the same 100% load. Interesting is that now the NS web interface shows an error when I try to enable Antivirus under Web Content Filter.

rpm -qa | grep clam

clamav-update-0.101.1-1.el7.x86_64
squidclamav-6.16-1.ns7.x86_64
clamav-scanner-systemd-0.101.1-1.el7.x86_64
nethserver-squidclamav-3.0.0-1.ns7.noarch
clamav-lib-0.101.1-1.el7.x86_64
clamav-0.101.1-1.el7.x86_64
clamav-filesystem-0.101.1-1.el7.noarch
clamav-server-systemd-0.101.1-1.el7.x86_64
clamav-unofficial-sigs-5.6.2-3.el7.noarch
clamd-0.101.1-1.el7.x86_64
clamav-data-0.101.1-1.el7.noarch

systemctl status clamd@squidclamav

clamd@squidclamav.service - clamd scanner (squidclamav) daemon
   Loaded: loaded (/usr/lib/systemd/system/clamd@.service; static; vendor preset: disabled)
  Drop-In: /etc/systemd/system/clamd@squidclamav.service.d
           └─c-icap.conf
   Active: activating (start) since Wed 2019-03-20 15:58:30 CET; 21s ago
     Docs: man:clamd(8)
           man:clamd.conf(5)
           https://www.clamav.net/documents/
  Control: 24764 (clamd)
   CGroup: /system.slice/system-clamd.slice/clamd@squidclamav.service
           └─24764 /usr/sbin/clamd -c /etc/clamd.d/squidclamav.conf

... systemd[1]: Starting clamd scanner (squidclamav) daemon...
... clamd[24764]: Received 0 file descriptor(s) from systemd.
... clamd[24764]: clamd daemon 0.101.1 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
... clamd[24764]: Running as user c-icap (UID 992, GID 987)
... clamd[24764]: Log file size limited to 1048576 bytes.
... gw.boudnik.de clamd[24764]: Reading databases from /var/lib/squidclamav
... clamd[24764]: Not loading PUA signatures.
... clamd[24764]: Bytecode: Security mode set to "TrustSigned".

flatspin · March 20, 2019, 4:37pm

Please stop clamd
systemctl stop clamd@squidclamav
and start it manualy
/usr/sbin/clamd --debug -F -c /etc/clamd.d/squidclamav.conf
and see if there are any relevant error.
Please check directory /var/lib/squidclamav if there are all files symlinks to /var/lib/clamav.

tmb · April 13, 2019, 11:06am

Thanks for the hint. As I wrote above I had disabled clamav since I could not find the error. Today I had some time and reinstalled it. Although a newer version it is still the same issue. clamd shows 95% CPU usage in ‘top’.

I initiated:

and got the following warning:

LibClamAV Warning: Detected duplicate databases /var/lib/squidclamav/bytecode.cvd and /var/lib/squidclamav/bytecode.cld, please manually remove one of them

I deleted the later of the two files. And checked in /var/lib/squidclamav if all files are symlinks.
Except for one file ‘daily.cvd’ which seems to be an archive file all the other files are symlinks.

At the moment it looks ok. Will report in case the issue comes back tomorrow after reboot.