#51 problems using iopendap urls

closed-accepted
None
7
2012-03-30
2012-01-05
gjvo
No

SInce I upgraded nco to version 4.0.8 I am no longer able to handle remote files with opendap.

On my workstation (Fedora 14) I get

[bhw330] [20:51] ~$ where ncks
/usr/bin/ncks
[bhw330] [20:51] ~$ ncks --version
NCO netCDF Operators version "4.0.8" last modified 2011/04/26 built Jun 23 2011 on nowhere by mockbuild
ncks version 4.0.8
[bhw330] [20:51] ~$ ncks -v tas http://ensembles.ecmwf.int/thredds/dodsC/ensembles/stream1/atmospheric/monthly /tmp/aap.nc
Segmentation fault (core dumped)
[bhw330] [20:51] ~$ gdb /usr/bin/ncks
GNU gdb (GDB) Fedora (7.2-51.fc14)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/ncks...(no debugging symbols found)...done.
Missing separate debuginfos, use: debuginfo-install nco-4.0.8-2.fc14.i686
(gdb) run -v tas http://ensembles.ecmwf.int/thredds/dodsC/ensembles/stream1/atmospheric/monthly /tmp/aap.nc
Starting program: /usr/bin/ncks -v tas http://ensembles.ecmwf.int/thredds/dodsC/ensembles/stream1/atmospheric/monthly /tmp/aap.nc
[Thread debugging using libthread_db enabled]
[New Thread 0x4023cb70 (LWP 17181)]
[Thread 0x4023cb70 (LWP 17181) exited]
[New Thread 0x4023cb70 (LWP 17182)]
[Thread 0x4023cb70 (LWP 17182) exited]

Program received signal SIGSEGV, Segmentation fault.
0x00834f8c in __memcpy_ssse3_rep () from /lib/libc.so.6
(gdb) where
#0 0x00834f8c in __memcpy_ssse3_rep () from /lib/libc.so.6
#1 0x06d764d9 in ocfetch () from /usr/lib/libnetcdf.so.6
#2 0x06d6451d in oc_fetch () from /usr/lib/libnetcdf.so.6
#3 0x06d79b7c in dap_oc_fetch () from /usr/lib/libnetcdf.so.6
#4 0x06d8029d in buildcachenode3 () from /usr/lib/libnetcdf.so.6
#5 0x06d71c09 in nc3d_getvarx () from /usr/lib/libnetcdf.so.6
#6 0x06d650cb in nc3d_getvara () from /usr/lib/libnetcdf.so.6
#7 0x06d65172 in nc3d_get_vara_float () from /usr/lib/libnetcdf.so.6
#8 0x06d7b5c6 in nc3_get_vara_float () from /usr/lib/libnetcdf.so.6
#9 0x06d2e27e in ?? () from /usr/lib/libnetcdf.so.6
#10 0x0094e043 in nco_get_vara () from /usr/lib/libnco-4.0.8.so
#11 0x00967bf5 in nco_cpy_var_val () from /usr/lib/libnco-4.0.8.so
#12 0x0804abc3 in ?? ()
#13 0x00720e36 in __libc_start_main () from /lib/libc.so.6
#14 0x08049781 in ?? ()

On my server (climexp.knmi.nl, RHEL, I am not sure which version) I had to install nco myself from source, which went reasonably smoothly (thanks). There I get

[bvlclim] [20:54] ~$ where ncks
/home/oldenbor/bin/ncks
/home/oldenbor/climexp/bin/ncks
[bvlclim] [20:56] ~$ ncks --version
NCO netCDF Operators version "4.0.8" last modified 2011/04/26 built Jan 5 2012 on bvlclim.knmi.nl by oldenbor
ncks version 4.0.8
[bvlclim] [20:55] ~$ ncks -v tas http://ensembles.ecmwf.int/thredds/dodsC/ensembles/stream1/atmospheric/monthly /tmp/aap.nc
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_get_vara()
nco_err_exit(): ERROR Error code is -70. Translation into English with nc_strerror(-70) is "NetCDF: DAP server error"
nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)

nco was built against netcdf-4.1.3 with built-in opendap support:

[bvlclim] [20:56] ~$ nc-config --version
netCDF 4.1.3
[bvlclim] [20:56] ~$ nc-config --has-dap
yes

The same opendap server responds normally to

[bvlclim] [21:09] ~$ ncdump -h http://ensembles.ecmwf.int/thredds/dodsC/ensembles/stream1/atmospheric/monthly
netcdf monthly {
dimensions:
ensemble = 62 ;
latitude = 71 ;
leadtime = 231 ;
...

My problem is that the climexp.knmi.nl web site uses ncks to get at external data, so this functionality is broken as long as I do not know how to fix this. Do you have any ideas?

Discussion

  • Charlie Zender
    Charlie Zender
    2012-01-05

    Unfortunately, I get the same results as you when using the latest NCO's ncks and netCDF's ncdump.
    However, I looked a bit further and this seems be a DAP server problem, i.e., something is amiss with ECMWF DAP server.
    I say this because ncdump -h works as you showed, but ncdump -v tas fails just like ncks.
    Moreover, ncks -m works just like ncdump -h.
    Furthermore, if you use ncks to print the data to screen, rather than a netCDF file, as follows

    ncks -D 3 -v tas http://ensembles.ecmwf.int/thredds/dodsC/ensembles/stream1/atmospheric/monthly

    you will see that the DAP server starts to provide data before the error occurs.
    So my theory is that something in the DAP library that NCO links to, or something in the DAP server at ECMWF, has
    changed in a way that breaks this command that should work. I am going to submit this as a bug report to Unidata.
    They will hopefully let me know if this is an issue on the NCO end, the client DAP library, or the server DAP.
    When I know more, I will post a follow-up.
    cz

     
  • Charlie Zender
    Charlie Zender
    2012-01-05

    • priority: 5 --> 7
    • assigned_to: nobody --> zender
     
  • gjvo
    gjvo
    2012-01-05

    I think it is more likely a problem in the netcdf4 library than at ECMWF, as
    1) I get similar problems with another opendap server that used to work:
    ncks -p http://ensemblesrt3.dmi.dk/cgi-bin/nph-dods/data/A1B/ KNMI-RACMO2_A1B_ECHAM5-r3_MM_25km-CRU_tas.nc.gz -v tas /tmp/aap.nc
    I also tried the NCEP opendap server, but it times out during the US day (as usual).
    2) I used to built with the separate nc_dap library, so there have been major changes there.

     
  • Charlie Zender
    Charlie Zender
    2012-03-30

    • status: open --> closed-accepted