Bugzilla – Bug 12217
MD5 and SHA256 file whitelist doesn't work
Last modified: 2020-08-10 19:50:10 EDT
Created attachment 7481 [details] Sample signature, whitelist and malicious file. Hello, I noticed a problem in ClamAV 0.100.2 that I can't reproduce on machine with older ClamAV (0.99). It appears that MD5 and SHA256 file whistelist doesn't work in ClamAV 0.100.2 if the file being scanned is normalized to multiple files (e.g. nocomment.html and notags.html). Steps to reproduce: 1. unpack the attached archive (which contains a sample sigs.ndb, whitelist.fp and dummy.php malicious file) 2. scan the file using provided signature and fingerprint database $ clamscan -d /path/to/extracted/directory/rules --debug --leave-temps /path/to/extracted/directory/dummy.php Expected result: Scanned file should not be reported as infected Actual result: Scanned file is resported as infected. Debug output printed to stdout clearly shows that whitelist.fp was loaded and it also shows that the file was recognized as infected <snip> LibClamAV debug: cli_scanhtml: using tempdir /tmp/clamav-568b62edac6769b83d50b7b810e09a71.tmp LibClamAV debug: FP SIGNATURE: 00000000000000000000000000000000:188:Malicious.PHP.134.UNOFFICIAL <snip> The interesting part is that ClamAV didn't detect the correct MD5 sum of the file and that it displayed it as "00000000000000000000000000000000". The file size is also wrong. In debug output it's show as 188 bytes while the actual file size is 190 bytes. 188 bytes is actually the size of normalized file (which was left behind in /tmp/clamav-568b62edac6769b83d50b7b810e09a71.tmp/nocomment.html)
Hi Sasa / Sale, Sorry about the delay. Thanks for the well written report and excellent sample attachments. I just tested with 0.101 and 0.99.4 and confirmed what you're seeing. I haven't had the chance to debug it yet to identify a solution. I will add it to our task list. Regards, Micah
(In reply to Micah Snyder from comment #1) > Hi Sasa / Sale, > > Sorry about the delay. Thanks for the well written report and excellent > sample attachments. I just tested with 0.101 and 0.99.4 and confirmed what > you're seeing. > > I haven't had the chance to debug it yet to identify a solution. I will add > it to our task list. > > Regards, > Micah Thank you for the update, Micah. I'm looking forward to further news regarding this issue. I'll gladly help with further testing and providing with more information if necessary. Cheers!
Hello, are there any news in regards to this bug report? I can reproduce this problem with ClamAV 0.101.1
I'm sorry. No news yet. Thanks for bumping the ticket though.
Hi, I've also recently encountered same bug with version 0.101.4 clamscan --version ClamAV 0.101.4/25580/Sun Sep 22 08:23:13 2019 I have tried creating both md5 and sha256 signatures ignores and in both cases they didn't work. As a bit of extra effort I've tried on a clean clamav install reproduction steps that Sasa mentioned, and I can confirm that same holds true for 101.4 still. cat /etc/redhat-release CentOS Linux release 7.7.1908 (Core) Clamav was install from epel repository version 0.101.4 If you need any additional info from my side, do let me know.
Hi Branko, Thanks for confirming the issue. What file type are you trying to whitelist? I haven't dug into this issue very far yet, but I have a sneaking suspicion that results may vary depending on file type. Micah
For reference, I think Mark was reporting the same issue here: https://lists.gt.net/clamav/devel/76131
Hi Micah, tnx for feedback. I first encountered this error while trying to white list some PHP files. My latest attempt was white listing this file: https://github.com/wp-plugins/wordfence/blob/master/lib/wordfenceClass.php As I do use some third party sigs that detect this as false positive.
Thanks Branko, that will be helpful for testing.
The bug is in version 0.102.2/25824/Tue May 26 14:27:30 2020 still present. Is there a workaround at this time?
(In reply to Drakano from comment #10) > The bug is in version 0.102.2/25824/Tue May 26 14:27:30 2020 still present. > Is there a workaround at this time? Unfortunately, there's no workaround that I know of. It's a shame, because we heavily utilized this feature.
We're currently working on a fix for this. The bug is pretty deep and there were a few issues involved. The FP signatures are hash signatures checked against the current file being scanned. However, if the alert itself comes from a scan of an embedded file (or a normalized buffer from a file, like HTML), then the hash in the .fp file (of the outer file / HTML file) never matches the interior or normalized file. Further, we found that for normalized files, hashes were never being calculated in the first place, resulting in 00000000000000000000000000000000 hashes (like the one in the first comment. To fix this issue I moved the file hash to our file map structure, so that any time a new file map is created, the content is hashed immediately and will always be accessible for the FP check. I’ve also been working on related file map improvements to include recording file names (where applicable). This work is all completed but the merge is pending fixup & merge of work from a public contribution that will likely have merge conflicts if I merge my stuff first. Mickey started to work on having the cli_checkfp() function traverse our file map recursion to check every parent file map hash so that FP signatures would be honored for outer files. This unfortunately uncovered an issue with inconsistencies in how scan recursion is being tracked. A part of her task is now to add fmap recursion tracking that is separate from the archive recursion tracking, as each file may contain more than 1 fmap. This work is necessary so FP checking doesn't just check the hash of normalized or embedded files, but also of the outer files. When this is complete, the bug should be fixed. In summary, we’re working on it and it will be fixed in the 0.103 release.
Hi all, This issue should now be fixed. Thanks for your patience. Allowing FP's should work correctly in the upcoming 0.103 release. Part 1: https://github.com/Cisco-Talos/clamav-devel/commit/cbe2cba4d1391fa9960ba23fca7f033caf9eb9bd#diff-5d5a25240c5110efefb695f0a50f5f7b Part 2: https://github.com/Cisco-Talos/clamav-devel/commit/9ea3b93018acf33fe3253dbb9817382a6df3e0ea Regards, Micah