Bug 12009 - Private Local Mirror - Detected duplicate databases
Private Local Mirror - Detected duplicate databases
Status: CLOSED FIXED
Product: ClamAV
Classification: ClamAV
Component: freshclam
0.99.2
x86_64 GNU/Linux
: P3 normal
: ---
Assigned To: Micah Snyder
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2018-01-22 04:45 EST by Hugo
Modified: 2020-02-19 09:44 EST (History)
1 user (show)

See Also:
QA Contact:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Hugo 2018-01-22 04:45:43 EST
Hello,

I am having issue with setting up a private mirror on my network. 
I have an apache server with those files : main.cvd bytecode.cvd and daily.cvd
Freshclam always report warning/error messages when :
- there is no cld files on the apache web server, then I have multiples 404 on my web apache logs for cld files,
- When there is cld files on the mirror, the client return a warinong reporting duplicate files.

I'm running on debian 8 :
# dpkg --list | grep fresh
ii  clamav-freshclam                  0.99.2+dfsg-0+deb8u2               amd64        anti-virus utility for Unix - virus database update utility


Please see my configuration, and the messages returned by freshclam in both cases.

-rw-r--r-- 1 www-data www-data 117892267 07.06.2017 23:38 main.cvd
-rw-r--r-- 1 www-data www-data    153228 07.12.2017 03:17 bytecode.cvd
lrwxrwxrwx 1 root     root             8 05.01.2018 10:14 main.cld -> main.cvd
lrwxrwxrwx 1 root     root             9 05.01.2018 10:14 daily.cld -> daily.cvd
lrwxrwxrwx 1 root     root            12 05.01.2018 10:14 bytecode.cld -> bytecode.cvd
-rw-r--r-- 1 www-data www-data  43804052 10.01.2018 06:17 daily.cvd


My "/etc/clamav/freshclam.conf" is :


AllowSupplementaryGroups false
Bytecode true
Checks 24
CompressLocalDatabase no
ConnectTimeout 30
DNSDatabaseInfo current.cvd.clamav.net
DatabaseDirectory /var/lib/clamav
DatabaseMirror mirror.recette.local
DatabaseOwner clamav
Debug false
Foreground false
LogFacility LOG_LOCAL6
LogFileMaxSize 0
LogRotate true
LogSyslog true
LogTime yes
LogVerbose false
MaxAttempts 5
PidFile /var/run/clamav/freshclam.pid
PrivateMirror mirror.recette.local
ReceiveTimeout 30
ScriptedUpdates no
TestDatabases yes
UpdateLogFile /var/log/clamav/freshclam.log


# freshclam

ClamAV update process started at Sat Jan 13 13:07:47 2018
Reading CVD header (main.cld): OK (IMS)
Reading CVD header (main.cvd): OK (IMS)
main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
Reading CVD header (daily.cld): OK
Downloading daily.cld [100%]
daily.cld updated (version: 24217, sigs: 1823727, f-level: 63, builder: neo)
Reading CVD header (bytecode.cld): OK
bytecode.cvd is up to date (version: 319, sigs: 75, f-level: 63, builder: neo)
[LibClamAV] Detected duplicate databases /var/lib/clamav/daily.cvd and /var/lib/clamav/daily.cld. The /var/lib/clamav/daily.cvd database is older and will not be loaded, you should manually remove it from the database directory.
Database updated (6390051 signatures) from mirror.recette.local (IP: 192.168.1.15)

If I remove cld files from my private mirror : 

# freshclam
ClamAV update process started at Sat Jan 13 13:10:18 2018
Downloading main.cld [100%]
main.cld updated (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
Downloading daily.cld [100%]
daily.cld updated (version: 24217, sigs: 1823727, f-level: 63, builder: neo)
WARNING: getfile: bytecode.cld not found on mirror.recette.local (IP: 192.168.1.15)
WARNING: Can't download bytecode.cld from mirror.recette.local
Downloading bytecode.cvd [100%]
bytecode.cvd updated (version: 319, sigs: 75, f-level: 63, builder: neo)
Database updated (6390051 signatures) from mirror.recette.local (IP: 192.168.1.15)
Comment 1 Tom 2018-01-22 09:22:26 EST
From the error message:
it looks like- you should be removing
The /var/lib/clamav/daily.cvd instead of the cld when everything is downloaded.

I would try that first, after doing a freshclam on the server- to see if that message disappears.
Comment 2 Hugo 2018-01-22 11:05:16 EST
Here is what I tried : 

- on the web server rename *.cvd files to *.cld :
-rw-r--r-- 1 www-data www-data    153228 07.12.2017 03:17 bytecode.cld
-rw-r--r-- 1 www-data www-data  44074218 22.01.2018 14:20 daily.cld
-rw-r--r-- 1 www-data www-data 117892267 07.06.2017 23:38 main.cld

- purge client /var/lib/clamav/ folder 
- run freshclam once it works  :

# freshclam
ClamAV update process started at Mon Jan 22 17:00:44 2018
Downloading main.cld [100%]
main.cld updated (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
Downloading daily.cld [100%]
daily.cld updated (version: 24245, sigs: 1832278, f-level: 63, builder: neo)
Downloading bytecode.cld [100%]
bytecode.cld updated (version: 319, sigs: 75, f-level: 63, builder: neo)
Database updated (6398602 signatures) from raproxy-vip.recette.local (IP: 192.168.8.15)

Run freshclam seconde time I still get file not found but for the cvd this time : 

# freshclam
ClamAV update process started at Mon Jan 22 17:02:46 2018
Reading CVD header (main.cld): OK (IMS)
Reading CVD header (main.cvd): WARNING: main.cvd not found on remote server
main.cld is up to date (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
Reading CVD header (daily.cld): OK
daily.cld is up to date (version: 24245, sigs: 1832278, f-level: 63, builder: neo)
Reading CVD header (bytecode.cld): OK
bytecode.cld is up to date (version: 319, sigs: 75, f-level: 63, builder: neo


So whenever I miss cvd or cld file on my web server freshclam return a warning. 
I don't get it why there is this behaviour, because when I provide both files, the client side isn't happy too...
Comment 4 Micah Snyder 2018-01-27 10:05:19 EST
Actually the return value looks ok.  It returns a cvd struct if there is a new cld/cvd, else NULL.  To determine if it's up to date, looks to me like the &ims out variable is set to a 0 instead of a 1 to indicate that the CLD is up-to-date.  

Unfortunately, the value of ims isn't checked when remote_cvdhead() returns NULL.  -1 would indicate failure, 0 would indicate up to date, and 1 would indicate there is a newer version.  It looks like it should be an easy fix.  

Also worth noting that when making the fix, ims should probably be initialized to -1 inside of the call to remote_cvdhead() instead of depending on the caller to properly initialize it before the call. 

Tom, do you know what ims is an acronym for?  I'm really curious. 

Maybe Tom can also figure out a bandaid that'll make your situation work while you wait for a new version of ClamAV.  We should have a beta for version 0.100.0 coming out soon, and hopefully in a few weeks we'll have another patch release for 0.99(.4) which we may be able to sneak this fix into.
Comment 5 Micah Snyder 2018-01-31 12:56:06 EST
Ok I put in a fix in 1cc4d6bc699b5babafc41955ea53bd3fe169460a.
It's a small change, and I feel very comfortable about the logic but I haven't actually tested with a private mirror of my own yet. 

I will see to it that this change gets integrated into the next patch release for 0.99(.4), pending testing.
Comment 6 Micah Snyder 2018-02-13 22:05:49 EST
I just verified that the fix made a difference.

The situation occurs when the modified timestamp on the client matches to the second the modified timestamp on the private mirror.  Freshclam queries for the .cld file with the "if-modified-by" timestamp, and because it matches exactly, the mirror will respond with the "304 Not Modified" message containing the matching timestamp. This can get a little dicey with different timezones, but if the timestamps match exactly, freshclam will receive the 304 message.  

I have verified that when this happens, without the fix the output from freshclam looks something like this:

micasnyd@avavpn-043:~/workspace/clamav-build-d$ bin/freshclam -v --debug --show-progress
Current working dir is /Users/micasnyd/workspace/clamav-build-d/share/clamav
Max retries == 3
ClamAV update process started at Wed Feb 14 02:54:53 2018
Using IPv6 aware code
...
If-Modified-Since: Wed, 14 Feb 2018 01:22:50 GMT
Reading CVD header (daily.cld): Connected to 0.0.0.0 (IP: 0.0.0.0).
Trying to retrieve CVD header of http://0.0.0.0/daily.cld
OK (IMS)
If-Modified-Since: Wed, 14 Feb 2018 01:22:50 GMT
Reading CVD header (daily.cvd): Connected to 0.0.0.0 (IP: 0.0.0.0).
Trying to retrieve CVD header of http://0.0.0.0/daily.cvd
WARNING: daily.cvd not found on remote server
daily.cld is up to date (version: 24313, sigs: 1852275, f-level: 63, builder: neo)
...


But with the patch, it looks more like this:
micasnyd@avavpn-104:~/workspace/clamav-devel-c$ ../clamav-build-c/bin/freshclam -v --debug --show-progress
Current working dir is /Users/micasnyd/workspace/clamav-build-c/share/clamav
Max retries == 3
ClamAV update process started at Wed Feb 14 02:54:04 2018
Using IPv6 aware code
...
If-Modified-Since: Wed, 14 Feb 2018 01:22:50 GMT
Reading CVD header (daily.cld): Connected to 0.0.0.0 (IP: 0.0.0.0).
Trying to retrieve CVD header of http://0.0.0.0/daily.cld
OK (IMS)
daily.cld is up to date (version: 24313, sigs: 1852275, f-level: 63, builder: neo)
...

Note that with the fix, the freshclam program does not attempt to download daily.cvd when the query for daily.cld came back with the "304 not modified" response -- denoted in the log with "OK (IMS)".