Bugzilla – Bug 10971
can't scan a files larger than available physical memory.
Last modified: 2018-06-09 08:58:24 EDT
Created attachment 6606 [details] RFC patch for the problem. There is a bug reported [0] where the user can't scan a file because it is larger than the physical available memory on this computer. On unix platform the fmap_check_empty() function maps the complete file by allocating the memory via mmap() with the write flag on. On systems with not a lot of memory it means that a file can't be scanned. For instance a 1GiB file on a system with 512MiB of memory. This patch changes the behaviour to what we have on win32: - the malloc() cl_fmap_t structure - mmap() the file as RO only. Since it content is backed by file on disk we mmap() any file as long as our virtual memory space is large enough. That means this will fail on 32bit architectures which try scan a file which is ~2.8GiB. This failed before as well but now we aleast not limited by our physical memory. This is my proposal. I tested it on x86-64 with 512MiB of RAM with 1.5GiB file via clamscan. I wanted some feedback before spending more time on this. We still have the problem on 32bit Platforms for files >3GiB. It would help to allocate only small chunks (say 4MiB) of the file instead of everything. Then if other pieces of the file are required it could get re-mapped(). Just a though. [0] https://bugs.debian.org/740059 Sebastian
Thanks for sending a patch, scheduled to evaluate for upcoming release.
Could this be included in 0.98.5?
any ETA for 0.99? Something like this summer, Xmas, next year ?
I am interested by this issue too, we are running clamav on VMs which don't always have enough memory to scan large files with the whole file in memory. Maybe a partial scan could be possible ? Best regards Loïc
For review and comment by Mickey. Targeting 0.99.3.
We like the idea and effort here, but the current implementation breaks our valgrind unit tests when testing bytecode functionality. Additionally, the general use-case for this isn't entirely clear and changing such a crucial piece of the library may have serious implications and needs heavy regression testing. As things stand, if we move forward with this patch it will likely be as a configuration option specifically intended for scanning in low physical memory environments.
Sorry, we are going to have to look at doing this another way based on comment 6. I will move this to the feature request queue.
Maybe we could start with a graceful failure by replacing the current output LibClamAV Warning: fmap: map allocation failed LibClamAV Error: CRITICAL: fmap() failed with a message conforming to the current FOUND format /path/to/file: file larger than available memory NOTSCANNED