ncea: ERROR nco_malloc() unable to allocate

Developers
IITM
2011-11-18
2013-10-17
  • IITM
    IITM
    2011-11-18

    Can we have a solution to the below mentioned problem related to NCO malloc failure.

    ncea: ERROR nco_malloc() unable to allocate 970467840 bytes
    ncea: INFO NCO has reported a malloc() failure. malloc() failures usually indicate that your machine does not have enough free memory (RAM+swap) to perform the requested operation. As such, malloc() failures result from the physical limitations imposed by your hardware. Read http://nco.sf.net/nco.html#mmr for a description of NCO memory usage. There are two workarounds in this scenario. One is to process your data in smaller chunks. The other is to use a machine with more free memory.

    Large tasks may uncover memory leaks in NCO. This is likeliest to occur with ncap. ncap scripts are completely dynamic and may be of arbitrary length and complexity. A script that contains many thousands of operations may uncover a slow memory leak even though each single operation consumes little additional memory. Memory leaks are usually identifiable by their memory usage signature. Leaks cause peak memory usage to increase monotonically with time regardless of script complexity. Slow leaks are very difficult to find. Sometimes a malloc() failure is the only noticeable clue to their existance. If you have good reasons to believe that your malloc() failure is ultimately due to an NCO memory leak (rather than inadequate RAM on your system), then we would be very interested in receiving a detailed bug report.

     
  • Charlie Zender
    Charlie Zender
    2011-11-19

    As the message says "There are two workarounds in this scenario. One is to process your data in smaller chunks. The other is to use a machine with more free memory."  Of course a longer term solution would be to solve this in software, but that would entail a significant code re-write, i.e., require dedicated time/funding.
    cz