#72 Memory management problem in 2.10

[Peter Williams]

Hi André,

I recently tried upgrading aoflagger to version 2.10 for my VLA data processing. Unfortunately, I find that for at least one of my current datasets, the program now fails with a std::bad_alloc exception as processing initializes. If I downgrade to 2.9, the same data set is processed without problems. I'm running with 12 threads on a machine with ~20 GB of memory and the dataset in question is 15 GB. The program doesn't print any messages about which method it's using to read the dataset.

Unfortunately I get this problem in my production environment, but not my development workstation, so (1) the problem is tied to the computing environment and (2) the tools I have available to debug things are a bit more limited. Do you have any suggestions as to how to diagnose what's going wrong? I should be able to run it in a debugger and at least identify where exactly the exception is getting thrown, and try different read methods, but I need to find a time when I can temporarily re-break my dataset processing.

Cheers,

Peter

[Peter Williams]

The crash occurs regardless of whether I specify -direct-read, -indirect-read, or -memory-read.

Backtrace (I think this is the crashing thread; thread apply all bt produces a whooole lot of text):

#0  0x00007ffff518968a in rfiStrategy::FrequencySelectionAction::Perform(rfiStrategy::ArtifactSet&, ProgressListener&)
    () from /users/pwilliam/env/bin/../lib/libaoflagger.so.0
#1  0x00007ffff52180ee in rfiStrategy::CombineFlagResults::Perform(rfiStrategy::ArtifactSet&, ProgressListener&) ()
   from /users/pwilliam/env/bin/../lib/libaoflagger.so.0
#2  0x00007ffff5208190 in rfiStrategy::ActionBlock::Perform(rfiStrategy::ArtifactSet&, ProgressListener&) ()
   from /users/pwilliam/env/bin/../lib/libaoflagger.so.0
#3  0x00007ffff521b0dc in rfiStrategy::ForEachPolarisationBlock::performDirectPolarizations(rfiStrategy::ArtifactSet&, ProgressListener&) () from /users/pwilliam/env/bin/../lib/libaoflagger.so.0
#4  0x00007ffff521e077 in rfiStrategy::ForEachPolarisationBlock::Perform(rfiStrategy::ArtifactSet&, ProgressListener&)
    () from /users/pwilliam/env/bin/../lib/libaoflagger.so.0
#5  0x00007ffff5208190 in rfiStrategy::ActionBlock::Perform(rfiStrategy::ArtifactSet&, ProgressListener&) ()
   from /users/pwilliam/env/bin/../lib/libaoflagger.so.0
#6  0x00007ffff5179814 in rfiStrategy::ForEachBaselineAction::PerformFunction::operator()() ()
   from /users/pwilliam/env/bin/../lib/libaoflagger.so.0
#7  0x00007ffff072bc5c in execute_native_thread_routine_compat ()
    at /home/msarahan/miniconda2/conda-bld/compilers_linux-64_1507259624353/work/.build/src/gcc-7.2.0/libstdc++-v3/src/c++11/thread.cc:110
#8  0x0000003f1d807aa1 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003f1d0e8bcd in clone () from /lib64/libc.so.6

[André Offringa]

Hi Peter, thanks for reporting this. I think I found the issue in recent changes to the FrequencySelectionAction code (with the help of your stack trace -- thanks a lot for that). There was indeed a problem, causing an object to be deallocated but used after that. It's not uncommon that this causes bad_alloc errors on some platforms, while other platform (mostly) don't show an issue, so this could very well be the issue.

I've committed the fix to Git. Could you test the fix? I might release a version a bit sooner to fix this issue, since this might well affect other platforms (it didn't affect any of mine though).

[Peter Williams]

Fix confirmed! Thanks for the prompt reply. (Also it's super lucky that I saw it, since I didn't get any email notification from SourceForge about your reply — just happened to check back here since I needed to update my aoflagger build for an entirely different reason. I am not a big fan of the GitHub monoculture but I would say that its communication tools are much better ...)

[André Offringa]

Great, thanks for the confirmation! I like GitHub a bit better too. Someday I might switch, but so far SourceForge works fine for me.