Bugzilla – Bug 12009
Private Local Mirror - Detected duplicate databases
Last modified: 2020-02-19 09:44:17 EST
Hello, I am having issue with setting up a private mirror on my network. I have an apache server with those files : main.cvd bytecode.cvd and daily.cvd Freshclam always report warning/error messages when : - there is no cld files on the apache web server, then I have multiples 404 on my web apache logs for cld files, - When there is cld files on the mirror, the client return a warinong reporting duplicate files. I'm running on debian 8 : # dpkg --list | grep fresh ii clamav-freshclam 0.99.2+dfsg-0+deb8u2 amd64 anti-virus utility for Unix - virus database update utility Please see my configuration, and the messages returned by freshclam in both cases. -rw-r--r-- 1 www-data www-data 117892267 07.06.2017 23:38 main.cvd -rw-r--r-- 1 www-data www-data 153228 07.12.2017 03:17 bytecode.cvd lrwxrwxrwx 1 root root 8 05.01.2018 10:14 main.cld -> main.cvd lrwxrwxrwx 1 root root 9 05.01.2018 10:14 daily.cld -> daily.cvd lrwxrwxrwx 1 root root 12 05.01.2018 10:14 bytecode.cld -> bytecode.cvd -rw-r--r-- 1 www-data www-data 43804052 10.01.2018 06:17 daily.cvd My "/etc/clamav/freshclam.conf" is : AllowSupplementaryGroups false Bytecode true Checks 24 CompressLocalDatabase no ConnectTimeout 30 DNSDatabaseInfo current.cvd.clamav.net DatabaseDirectory /var/lib/clamav DatabaseMirror mirror.recette.local DatabaseOwner clamav Debug false Foreground false LogFacility LOG_LOCAL6 LogFileMaxSize 0 LogRotate true LogSyslog true LogTime yes LogVerbose false MaxAttempts 5 PidFile /var/run/clamav/freshclam.pid PrivateMirror mirror.recette.local ReceiveTimeout 30 ScriptedUpdates no TestDatabases yes UpdateLogFile /var/log/clamav/freshclam.log # freshclam ClamAV update process started at Sat Jan 13 13:07:47 2018 Reading CVD header (main.cld): OK (IMS) Reading CVD header (main.cvd): OK (IMS) main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr) Reading CVD header (daily.cld): OK Downloading daily.cld [100%] daily.cld updated (version: 24217, sigs: 1823727, f-level: 63, builder: neo) Reading CVD header (bytecode.cld): OK bytecode.cvd is up to date (version: 319, sigs: 75, f-level: 63, builder: neo) [LibClamAV] Detected duplicate databases /var/lib/clamav/daily.cvd and /var/lib/clamav/daily.cld. The /var/lib/clamav/daily.cvd database is older and will not be loaded, you should manually remove it from the database directory. Database updated (6390051 signatures) from mirror.recette.local (IP: 192.168.1.15) If I remove cld files from my private mirror : # freshclam ClamAV update process started at Sat Jan 13 13:10:18 2018 Downloading main.cld [100%] main.cld updated (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr) Downloading daily.cld [100%] daily.cld updated (version: 24217, sigs: 1823727, f-level: 63, builder: neo) WARNING: getfile: bytecode.cld not found on mirror.recette.local (IP: 192.168.1.15) WARNING: Can't download bytecode.cld from mirror.recette.local Downloading bytecode.cvd [100%] bytecode.cvd updated (version: 319, sigs: 75, f-level: 63, builder: neo) Database updated (6390051 signatures) from mirror.recette.local (IP: 192.168.1.15)
From the error message: it looks like- you should be removing The /var/lib/clamav/daily.cvd instead of the cld when everything is downloaded. I would try that first, after doing a freshclam on the server- to see if that message disappears.
Here is what I tried : - on the web server rename *.cvd files to *.cld : -rw-r--r-- 1 www-data www-data 153228 07.12.2017 03:17 bytecode.cld -rw-r--r-- 1 www-data www-data 44074218 22.01.2018 14:20 daily.cld -rw-r--r-- 1 www-data www-data 117892267 07.06.2017 23:38 main.cld - purge client /var/lib/clamav/ folder - run freshclam once it works : # freshclam ClamAV update process started at Mon Jan 22 17:00:44 2018 Downloading main.cld [100%] main.cld updated (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr) Downloading daily.cld [100%] daily.cld updated (version: 24245, sigs: 1832278, f-level: 63, builder: neo) Downloading bytecode.cld [100%] bytecode.cld updated (version: 319, sigs: 75, f-level: 63, builder: neo) Database updated (6398602 signatures) from raproxy-vip.recette.local (IP: 192.168.8.15) Run freshclam seconde time I still get file not found but for the cvd this time : # freshclam ClamAV update process started at Mon Jan 22 17:02:46 2018 Reading CVD header (main.cld): OK (IMS) Reading CVD header (main.cvd): WARNING: main.cvd not found on remote server main.cld is up to date (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr) Reading CVD header (daily.cld): OK daily.cld is up to date (version: 24245, sigs: 1832278, f-level: 63, builder: neo) Reading CVD header (bytecode.cld): OK bytecode.cld is up to date (version: 319, sigs: 75, f-level: 63, builder: neo So whenever I miss cvd or cld file on my web server freshclam return a warning. I don't get it why there is this behaviour, because when I provide both files, the client side isn't happy too...
Actually the return value looks ok. It returns a cvd struct if there is a new cld/cvd, else NULL. To determine if it's up to date, looks to me like the &ims out variable is set to a 0 instead of a 1 to indicate that the CLD is up-to-date. Unfortunately, the value of ims isn't checked when remote_cvdhead() returns NULL. -1 would indicate failure, 0 would indicate up to date, and 1 would indicate there is a newer version. It looks like it should be an easy fix. Also worth noting that when making the fix, ims should probably be initialized to -1 inside of the call to remote_cvdhead() instead of depending on the caller to properly initialize it before the call. Tom, do you know what ims is an acronym for? I'm really curious. Maybe Tom can also figure out a bandaid that'll make your situation work while you wait for a new version of ClamAV. We should have a beta for version 0.100.0 coming out soon, and hopefully in a few weeks we'll have another patch release for 0.99(.4) which we may be able to sneak this fix into.
Ok I put in a fix in 1cc4d6bc699b5babafc41955ea53bd3fe169460a. It's a small change, and I feel very comfortable about the logic but I haven't actually tested with a private mirror of my own yet. I will see to it that this change gets integrated into the next patch release for 0.99(.4), pending testing.
I just verified that the fix made a difference. The situation occurs when the modified timestamp on the client matches to the second the modified timestamp on the private mirror. Freshclam queries for the .cld file with the "if-modified-by" timestamp, and because it matches exactly, the mirror will respond with the "304 Not Modified" message containing the matching timestamp. This can get a little dicey with different timezones, but if the timestamps match exactly, freshclam will receive the 304 message. I have verified that when this happens, without the fix the output from freshclam looks something like this: micasnyd@avavpn-043:~/workspace/clamav-build-d$ bin/freshclam -v --debug --show-progress Current working dir is /Users/micasnyd/workspace/clamav-build-d/share/clamav Max retries == 3 ClamAV update process started at Wed Feb 14 02:54:53 2018 Using IPv6 aware code ... If-Modified-Since: Wed, 14 Feb 2018 01:22:50 GMT Reading CVD header (daily.cld): Connected to 0.0.0.0 (IP: 0.0.0.0). Trying to retrieve CVD header of http://0.0.0.0/daily.cld OK (IMS) If-Modified-Since: Wed, 14 Feb 2018 01:22:50 GMT Reading CVD header (daily.cvd): Connected to 0.0.0.0 (IP: 0.0.0.0). Trying to retrieve CVD header of http://0.0.0.0/daily.cvd WARNING: daily.cvd not found on remote server daily.cld is up to date (version: 24313, sigs: 1852275, f-level: 63, builder: neo) ... But with the patch, it looks more like this: micasnyd@avavpn-104:~/workspace/clamav-devel-c$ ../clamav-build-c/bin/freshclam -v --debug --show-progress Current working dir is /Users/micasnyd/workspace/clamav-build-c/share/clamav Max retries == 3 ClamAV update process started at Wed Feb 14 02:54:04 2018 Using IPv6 aware code ... If-Modified-Since: Wed, 14 Feb 2018 01:22:50 GMT Reading CVD header (daily.cld): Connected to 0.0.0.0 (IP: 0.0.0.0). Trying to retrieve CVD header of http://0.0.0.0/daily.cld OK (IMS) daily.cld is up to date (version: 24313, sigs: 1852275, f-level: 63, builder: neo) ... Note that with the fix, the freshclam program does not attempt to download daily.cvd when the query for daily.cld came back with the "304 not modified" response -- denoted in the log with "OK (IMS)".