Help save net neutrality! Learn more.
Close

How can I determine if my installation of NCO has LFS cooked in?

Help
2014-01-08
2014-01-08
  • James Adams

    James Adams - 2014-01-08

    I'm trying to perform a dimension permutation on a very large NetCDF file (~12 GB) and it's croaking before completion with a single "Killed" message so I'm not really sure yet what's going wrong. I wonder if maybe I'm overwhelming the operator (ncpdq) because it doesn't have LFS cooked in? I didn't perform the NCO installation myself so I'm trying to determine if this is the issue by probing the NCO installation if possible, and my question is how can I determine whether or not LFS was configured in my current NCO installation? Maybe I don't even need LFS configured in NCO since it looks like it's only really necessary to support versions of NetCDF prior to 3.6 and I have version 4.1.1-3 installed. Here's what I can see in terms of the installed versions of NetCDF and NCO when I run 'yum list' on my CentOS machine:

    nco.i686                    4.0.5-2.el6                   epel
    nco.x86_64                  4.0.5-2.el6                   epel
    nco-devel.i686              4.0.5-2.el6                   epel
    nco-devel.x86_64            4.0.5-2.el6                   epel
    nco-static.x86_64           4.0.5-2.el6                   epel
    netcdf.x86_64               4.1.1-3.el6.4                 @epel_base
    netcdf-devel.x86_64         4.1.1-3.el6.4                 @epel_base
    netcdf.i686                 4.1.1-3.el6.4                 epel
    netcdf-devel.i686           4.1.1-3.el6.4                 epel
    netcdf-perl.x86_64          1.2.4-2.el6                   epel
    netcdf-static.x86_64        4.1.1-3.el6.4                 epel
    netcdf4-python.x86_64       1.0.2-1.el6                   epel
    

    Thanks in advance for any suggestions.

    --James

     
  • Charlie Zender

    Charlie Zender - 2014-01-08

    Hello James,

    Run ncks -r and check the output line for "netCDF3 64-bit files".
    This is LFS.
    Most likely it is supported.
    The "Killed" message has been seen when a process exceeds the allowed memory limits. A 12 GB file could do this. Maybe break it into smaller pieces first?
    cz

     
  • James Adams

    James Adams - 2014-01-08

    Thanks a lot for the quick help, Charlie.

    Everything looks good in terms of support for LFS so I suspect you're right about exceeding my memory allocation on this virtual machine. I've thought about using NCO to chop the NetCDF into smaller pieces but I may run into the same issue trying that approach. Probably I should just rewrite my code that creates the NetCDF file such that it instead creates several smaller files, as well as figuring out how to take advantage of scaling my float values down to integers since the original ASCII data only goes out to two decimal places.

    --James

     

Log in to post a comment.