Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

ncrcat "ERROR Error code is -40" failure for necdf4 file with more unlimited dimensions

Help
rmla
2014-06-19
4 days ago
  • rmla
    rmla
    2014-06-19

    Some slicing of the file works other not..
    1) it works:
    ncrcat -O -d time,0,1 71355.ecmf.1.nc 71355.ecmf.1.nc.slice
    2) it doesn't:
    ncrcat -O -d time,0,0 71355.ecmf.1.nc 71355.ecmf.1.nc.slice
    ncrcat -O -d time,1,1 71355.ecmf.1.nc 71355.ecmf.1.nc.slice

    If the 'parameter' dim is not UNLIMITED than slicing works without problems. Thank you for any opinion.

    --> export NCO_VERSION=4.3.1 ; use nco
    nco version 4.3.1 support
    Note that these are 64 bit mode versions.
    nco, version 4.3.1 enabled (PATH, MANPATH, NCO, NCO_INCLUDE, NCO_LIB)
    Look at http://www.ecmwf.int/services/computing/docs/data_formats/README_NCO.html
    for more information.

    --> ncdump -c 71355.ecmf.1.nc
    netcdf \71355.ecmf.1 {
    dimensions:
    run = 4 ;
    step = UNLIMITED ; // (61 currently)
    number = UNLIMITED ; // (51 currently)
    point = 3 ;
    npc = 6 ;
    ngr = 4 ;
    strlen1 = 10 ;
    strlen2 = 10 ;
    time = UNLIMITED ; // (2 currently)
    parameter = UNLIMITED ; // (2 currently)
    variables:
    int time(time) ;
    time:units = "hours since 2000-01-01 00:00:00 UTC" ;
    time:origin = "01-JAN-2000 00:00:00 UTC" ;
    time:calendar = "gregorian" ;
    char parameter_list(parameter, strlen1) ;
    parameter_list:missing_value = "" ;
    parameter_list:fill_value = "" ;
    char type_list(time, parameter, run, number, strlen2) ;
    type_list:missing_value = "" ;
    type_list:fill_value = "" ;
    char step_range_list(time, parameter, run, step, strlen2) ;
    step_range_list:missing_value = "" ;
    step_range_list:fill_value = "" ;
    double sfc_2_2t(time, run, step, number, ngr) ;
    sfc_2_2t:units = "K" ;
    sfc_2_2t:long_name = "2 metre temperature" ;
    sfc_2_2t:short_name = "2t" ;
    sfc_2_2t:missing_value = 1.e+30 ;
    sfc_2_2t:fill_value = 1.e+30 ;
    double sfc_2_2d(time, run, step, number, ngr) ;
    sfc_2_2d:units = "K" ;
    sfc_2_2d:long_name = "2 metre dewpoint temperature" ;
    sfc_2_2d:short_name = "2d" ;
    sfc_2_2d:missing_value = 1.e+30 ;
    sfc_2_2d:fill_value = 1.e+30 ;
    data:
    time = 110280, 110376 ;
    }

    KO case when slicing over -d time,0,0 (or -d time,1,1)

    --> ncrcat -O -d time,0,0 71355.ecmf.1.nc 71355.ecmf.1.nc.slice
    nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_put_var1()
    nco_err_exit(): ERROR Error code is -40. Translation into English with nc_strerror(-40) is "NetCDF: Index exceeds dimension bound"
    nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)
    --> ncrcat -O -d time,0,1 71355.ecmf.1.nc 71355.ecmf.1.nc.slice

    OK case when slicing over both existing time records

    --> ncrcat -O -d time,0,1 71355.ecmf.1.nc 71355.ecmf.1.nc.slice
    --> ncdump -c 71355.ecmf.1.nc.slice
    netcdf \71355.ecmf.1.nc {
    dimensions:
    run = 4 ;
    step = 61 ;
    number = 51 ;
    ngr = 4 ;
    strlen1 = 10 ;
    strlen2 = 10 ;
    time = 2 ;
    parameter = UNLIMITED ; // (2 currently)
    variables:
    int time(time) ;
    time:units = "hours since 2000-01-01 00:00:00 UTC" ;
    time:origin = "01-JAN-2000 00:00:00 UTC" ;
    time:calendar = "gregorian" ;
    char parameter_list(parameter, strlen1) ;
    parameter_list:missing_value = "" ;
    parameter_list:fill_value = "" ;
    char type_list(time, parameter, run, number, strlen2) ;
    type_list:missing_value = "" ;
    type_list:fill_value = "" ;
    char step_range_list(time, parameter, run, step, strlen2) ;
    step_range_list:missing_value = "" ;
    step_range_list:fill_value = "" ;
    double sfc_2_2t(time, run, step, number, ngr) ;
    sfc_2_2t:units = "K" ;
    sfc_2_2t:long_name = "2 metre temperature" ;
    sfc_2_2t:short_name = "2t" ;
    sfc_2_2t:missing_value = 1.e+30 ;
    sfc_2_2t:fill_value = 1.e+30 ;
    double sfc_2_2d(time, run, step, number, ngr) ;
    sfc_2_2d:units = "K" ;
    sfc_2_2d:long_name = "2 metre dewpoint temperature" ;
    sfc_2_2d:short_name = "2d" ;
    sfc_2_2d:missing_value = 1.e+30 ;
    sfc_2_2d:fill_value = 1.e+30 ;

    // global attributes:
    :history = "Thu Jun 19 16:02:20 2014: ncrcat -O -d time,0,1 71355.ecmf.1.nc 71355.ecmf.1.nc.slice" ;
    :nco_openmp_thread_number = 1 ;
    data:

    time = 110280, 110376 ;
    }

     
    Last edit: rmla 2014-06-19
    Attachments
  • Charlie Zender
    Charlie Zender
    2014-06-19

    Dear rmla,

    Your commands all work without reporting errors on the file provided with the current NCO version. Thus at least some of the problems you experienced were fixed between 4.3.1 and 4.4.4. However, I carefully examined the output and
    there is a problem with the output hyperslabs. I will post again
    when I have a simpler test case and it is clearer to me what is going wrong.
    cz
    p.s. The links you posted do not work for me.

     
  • Charlie Zender
    Charlie Zender
    2014-06-24

    please post the file with the exact command that does not work as expected.
    cz

     
  • Charlie Zender
    Charlie Zender
    2014-06-24

    sorry, last message not meant for this thread. still waiting on unidata to confirm my bug report, which i include below to show why this problem may take some time to solve (because ncgen cannot create a workable test file that is similar to your file):

    Hi,

    Latest ncgen seems to fail to create netCDF4 file from this CDL file
    produced by ncdump from a presumably valid netCDF4 file that
    contains multiple record dimensions (MRD):

    zender@givre:~$ ncks -O -d time,0,1 -d run,0,1 -d step,0,1 -d number,0,1 -d ngr,0,1 ${DATA}/hdf/71355.ecmf.1.nc ~/mrd.nc
    zender@givre:~$ ncdump ~/mrd.nc > ~/mrd.cdl
    zender@givre:~$ ncgen -k netCDF-4 -b -o ~/mrd2.nc ~/mrd.cdl
    assertion failure: con->nctype == NC_COMPOUND

    Aborted (core dumped)
    zender@givre:~$ ncgen -v
    ncgen: option requires an argument -- 'v'
    Usage: ncgen [ -b ] [ -c ] [ -f ] [ -k kind ] [ -x ] [-S struct-format] [-M <name> [ -o outfile] [ file ... ]
    netcdf library version 4.3.3-rc1 of Jun 5 2014 13:50:57 $

    mrd.nc and mrd.cdl are attached.

    Can you reproduce the problem?
    If so, please let me know when there is a fix.

    Thanks!
    c

     
  • rmla
    rmla
    2014-07-18

    Dear Charlie,
    Thank you very much for your effort.
    Please let me know how dangerous is to use that kind of slicing and NCO version 4.4.4. as you mentioned some problem with output hyperslabs. Currently the file are created, appended etc with python script and ncrcat should be used for simple and fast slicing of potentially big final files according to user's request.
    Best regards,
    Richard

     
  • Charlie Zender
    Charlie Zender
    2014-07-18

    Richard,
    My bug report (above) to Unidata seems not to have been received.
    I will resubmit it.
    I am traveling for ~2 weeks and unlikely to learn more until I return.
    Until then, I'm unsure how "dangerous" it is.
    My advice is check before believing your answers on hyperslabbing multiple record dimension variables.
    Will post again when I learn more.
    cz

     
  • Charlie Zender
    Charlie Zender
    4 days ago

    Richard,
    Re-submitted bug report (FJV-105162). Unidata acknowledges a bug in ncgen/ncdump. When they commit a patch that fixes that bug, we'll be able to get to work on your use case...
    c