Bug 12317 - Service startup timeout causing endless loop with 100% CPU usage
Service startup timeout causing endless loop with 100% CPU usage
Status: CLOSED FIXED
Product: ClamAV
Classification: ClamAV
Component: clamd
0.101.2
x86_64 GNU/Linux
: P3 normal
: 0.101.0
Assigned To: ClamAV team
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2019-04-05 13:54 EDT by Hrvoje
Modified: 2019-04-07 16:08 EDT (History)
2 users (show)

See Also:
QA Contact:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Hrvoje 2019-04-05 13:54:19 EDT
After updating to version 0.101.2 on CentOS 7 (Linux 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux) the service seems to get stuck on slower machines in an infinite attempting to start loop consuming 100% CPU in the process.

from /var/log/messages
-----------------
Apr  5 18:50:45 localhost systemd: Starting Generic clamav scanner daemon...
Apr  5 18:50:48 localhost clamd[31880]: Received 0 file descriptor(s) from systemd.
Apr  5 18:50:48 localhost clamd[31880]: clamd daemon 0.101.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Apr  5 18:50:48 localhost clamd[31880]: Running as user clamscan (UID 993, GID 991)
Apr  5 18:50:48 localhost clamd[31880]: Log file size limited to 1048576 bytes.
Apr  5 18:50:48 localhost clamd[31880]: Reading databases from /var/lib/clamav
Apr  5 18:50:48 localhost clamd[31880]: Not loading PUA signatures.
Apr  5 18:50:48 localhost clamd[31880]: Bytecode: Security mode set to "TrustSigned".
Apr  5 18:52:15 localhost systemd: clamd@scan.service start operation timed out. Terminating.
Apr  5 18:52:16 localhost systemd: Failed to start Generic clamav scanner daemon.
Apr  5 18:52:16 localhost systemd: Unit clamd@scan.service entered failed state.
Apr  5 18:52:16 localhost systemd: clamd@scan.service failed.
Apr  5 18:52:16 localhost systemd: clamd@scan.service holdoff time over, scheduling restart.
Apr  5 18:52:16 localhost systemd: Stopped Generic clamav scanner daemon.
-----------------
Apr  5 18:52:16 localhost systemd: Starting Generic clamav scanner daemon...
Apr  5 18:52:16 localhost clamd[31896]: Received 0 file descriptor(s) from systemd.
Apr  5 18:52:16 localhost clamd[31896]: clamd daemon 0.101.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Apr  5 18:52:16 localhost clamd[31896]: Running as user clamscan (UID 993, GID 991)
Apr  5 18:52:16 localhost clamd[31896]: Log file size limited to 1048576 bytes.
Apr  5 18:52:16 localhost clamd[31896]: Reading databases from /var/lib/clamav
Apr  5 18:52:16 localhost clamd[31896]: Not loading PUA signatures.
Apr  5 18:52:16 localhost clamd[31896]: Bytecode: Security mode set to "TrustSigned".
Apr  5 18:53:46 localhost systemd: clamd@scan.service start operation timed out. Terminating.
Apr  5 18:53:46 localhost systemd: Failed to start Generic clamav scanner daemon.
Apr  5 18:53:46 localhost systemd: Unit clamd@scan.service entered failed state.
Apr  5 18:53:46 localhost systemd: clamd@scan.service failed.
Apr  5 18:53:46 localhost systemd: clamd@scan.service holdoff time over, scheduling restart.
Apr  5 18:53:46 localhost systemd: Stopped Generic clamav scanner daemon.
-----------------
Apr  5 18:53:46 localhost systemd: Starting Generic clamav scanner daemon
...
...
Comment 1 Micah Snyder 2019-04-05 14:04:35 EDT
We appear to have a signature database issue today causing extremely slow load time for the database.  I think this may be causing your slower machines to time out before they are able to finish loading.
Comment 2 Hrvoje 2019-04-05 14:05:18 EDT
Additional note... running it manually the service does start up in the end, but it takes longer then the systemd default service start timeout of 90s.
Comment 3 Micah Snyder 2019-04-05 16:15:35 EDT
We'll be dropping a bunch of Phishtank.Phishing signatures from the database in tomorrow's update.  I believe these are responsible for the poor load and scan time performance.

Please let me know after tomorrow's update if your situation is resolved.

Regards,
Micah
Comment 4 JMGorro 2019-04-05 17:06:14 EDT
I'm experiencing something similar since today midday (approx) GMT.

clamav-db-0.100.2-1.el6.x86_64
clamav-0.100.2-1.el6.x86_64
ClamAV update process started at Fri Apr  5 23:02:09 2019
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Local version: 0.100.2 Recommended version: 0.101.2
DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav
main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
daily.cvd is up to date (version: 25410, sigs: 1552552, f-level: 63, builder: raynman)
bytecode.cld is up to date (version: 328, sigs: 94, f-level: 63, builder: neo)
centos-release-6-10.el6.centos.12.3.x86_64
epel-release-6-8.noarch
Linux 2.6.32-754.11.1.el6.x86_64 #1 SMP Tue Feb 26 15:38:56 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Running inside MailScanner for long time always good performance. The failure has appeared today.

Hope this will be helpful.
Comment 5 Hrvoje 2019-04-07 06:24:33 EDT
It seems to be better today:
Apr  7 12:20:31 localhost systemd: Starting Generic clamav scanner daemon...
...
Apr  7 12:21:16 localhost systemd: Started Generic clamav scanner daemon.

45s
Comment 6 Micah Snyder 2019-04-07 16:08:04 EDT
(In reply to Hrvoje from comment #5)
> It seems to be better today:
> Apr  7 12:20:31 localhost systemd: Starting Generic clamav scanner daemon...
> ...
> Apr  7 12:21:16 localhost systemd: Started Generic clamav scanner daemon.
> 
> 45s

I'm glad to hear it.  I'm sorry about the trouble.

Closing this ticket now that things are working again.