I'm trying to perform a dimension permutation on a very large NetCDF file (~12 GB) and it's croaking before completion with a single "Killed" message so I'm not really sure yet what's going wrong. I wonder if maybe I'm overwhelming the operator (ncpdq) because it doesn't have LFS cooked in? I didn't perform the NCO installation myself so I'm trying to determine if this is the issue by probing the NCO installation if possible, and my question is how can I determine whether or not LFS was configured in my current NCO installation? Maybe I don't even need LFS configured in NCO since it looks like it's only really necessary to support versions of NetCDF prior to 3.6 and I have version 4.1.1-3 installed. Here's what I can see in terms of the installed versions of NetCDF and NCO when I run 'yum list' on my CentOS machine:
nco.i686 4.0.5-2.el6 epel
nco.x86_64 4.0.5-2.el6 epel
nco-devel.i686 4.0.5-2.el6 epel
nco-devel.x86_64 4.0.5-2.el6 epel
nco-static.x86_64 4.0.5-2.el6 epel
netcdf.x86_64 4.1.1-3.el6.4 @epel_base
netcdf-devel.x86_64 4.1.1-3.el6.4 @epel_base
netcdf.i686 4.1.1-3.el6.4 epel
netcdf-devel.i686 4.1.1-3.el6.4 epel
netcdf-perl.x86_64 1.2.4-2.el6 epel
netcdf-static.x86_64 4.1.1-3.el6.4 epel
netcdf4-python.x86_64 1.0.2-1.el6 epel
Thanks in advance for any suggestions.
Run ncks -r and check the output line for "netCDF3 64-bit files".
This is LFS.
Most likely it is supported.
The "Killed" message has been seen when a process exceeds the allowed memory limits. A 12 GB file could do this. Maybe break it into smaller pieces first?
Thanks a lot for the quick help, Charlie.
Everything looks good in terms of support for LFS so I suspect you're right about exceeding my memory allocation on this virtual machine. I've thought about using NCO to chop the NetCDF into smaller pieces but I may run into the same issue trying that approach. Probably I should just rewrite my code that creates the NetCDF file such that it instead creates several smaller files, as well as figuring out how to take advantage of scaling my float values down to integers since the original ASCII data only goes out to two decimal places.
Log in to post a comment.