#274 Possible memory leak during filtering

[chris]

I was checking the AbsProbe.cpp file, and more specifically the ApplyFilters method. Since items are created and registered during CollectItems, I was wondering if memory could be lost during filtering.

Method uses the std::remove_if feature for deleting items from the newly populated vector. While this will remove the entries from collection, underlying memory may not be reclaimed.

I didn't check the filtering operation from AbsObjectCollector.cpp.

[Michael Chisholm]

I think you're right about AbsProbe.cpp. But in AbsObjectCollector.cpp, I think items have already been cached when the filtering happens, so that memory is reclaimed when the cache is cleared.

[chris]

Thank you for the quick feedback. You must be right for AbsObjectCollector.cpp as I didn't realize caching would prevent from wasting memory here.

I would appreciate your feeling about one possible improvement for the items collect operation. The current implementation is clear and works fine but some probes may require GB of memory. For instance, probes using FileFinder such as FileProbe may drive to out of memory situations very quickly on Linux devices.

I tested different OVAL contents (through SCAP processing) and was unable to complete most of them because my Linux device includes hundreds of packages. I tried reducing the memory usage using a different approach.

The current implementation collects items and then applies filters. I think that most of the probes should continue this way but others may take advantage of "early filtering". I added the following protected method in AbsProbe.h:

bool IsFiltered (Item* item, FilterVector* filters);

with implementation in AbsProbe.cpp:

bool AbsProbe::IsFiltered (Item* item, FilterVector* filters)
{
    for (FilterVector::iterator filter_it=filters->begin (); 
         filter_it!=filters->end (); ++filter_it)
    {
        FilterFunctor filtered (*filter_it);

        if (filtered (item))
        {
            return true;
        }
    }

    return false;
}

As a consequence, any real probe may use this method before adding items into the items collection. For example, I tried with FileProbe::CollectItems:

ItemVector* FileProbe::CollectItems(Object* object) {

    FilterVector* filters = object->GetFilters();
    ItemVector *collectedItems = new ItemVector();

    ...
}

and replace every

collectedItems->push_back(item);

with

if (IsFiltered (item, filters))
{
    delete item;
}
else
{
    collectedItems->push_back(item);
}

Do you think this approach would drive to acceptable results? Idea is just to avoid collecting items for very limited probes.

Thank you

[Michael Chisholm]

Yes, it makes sense. I think this sort of filter-as-you-go optimization has been described on the oval-developer-list as well.