also works. So what I'm not too sure about is whether this is an nco issue or something different? Any pointers appreciated.
Thanks
James
[for ref: I'm using the netcdf libraries from an anaconda install. I built nco using these - but the ncks seg faulted even on a simple ncks -r call, so I did a yum install nco instead]
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
This appears to be a permissions problem, that in turn may (not sure) point to inadequate authorization functionality in NCO. The first thing is to get ncks reading (not copying) from ESGF. Appended below is the lengthy error message I get. It boils down to an authorization error. I'm unfamiliar with exactly the syntax or handshakes ESGF needs to work with DAP. Any pointers appreciated. Try to get either of these commands working by including your ESGF login/password somewhere in the URL:
The following URL requires authentication: http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1.dds
Please enter your OpenID and you will be redirected to the login page at that site
<input type="checkbox" name="rememberOpenid" checked="checked" �=""/> Remember my OpenID on this computer
</form>
After logging in, you will be redirected to: http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1.dds
</body> </html>
ncks: INFO DAP-access to http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1 failed with error code -78. Translation into English with nc_strerror(-78) is "NetCDF: Authorization failure"
ncks: WARNING not searching for http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1 on remote filesystem, using local file hredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1 instead
ERROR: ncoopen() unable to open file "thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1"
ERROR NC_ENOTNC Not a netCDF file
HINT: NC_ENOTNC errors can occur for many reasons. If your use-case matches one of the four listed below, take the corrective action indicated:
1. An NCO operator linked only to the classic netCDF3 library attempts to read netCDF4 (or HDF4 or HDF5) files. However, this executable seems to have been built with the capability to manipulate netCDF4 files, so it is unlikely that this command failed only because the input datasets are netCDF4 format. Something else is going wrong.
2. NCO attempts to utilize diskless (i.e., RAM) files. In this case remove the diskless switches (e.g., --ram or --diskless) and then re-issue the command.
3. NCO attempts to read other filetypes (HDF4, HDF-EOS2, pnetCDF) for which support must be (but was not) enabled at netCDF build-time. In this case it may be possible to access the input files using NCO if NCO is first re-linked to a version of netCDF configured with the --enable-hdf4 option. This is a non-standard netCDF build option described here: http://www.unidata.ucar.edu/software/netcdf/docs/build_hdf4.html
4. Access to a DAP URL fails, and the backup method of downloading the URL using wget obtains a data aggregation file (e.g., a .ncml file) instead of an actual netCDF file. In this case the problem is with the DAP server or URL.
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): ncoopen()
nco_err_exit(): ERROR Error code is -51. Translation into English with nc_strerror(-51) is "NetCDF: Unknown file format"
nco_err_exit(): ERROR NCO will now exit with system call abort()
Aborted (core dumped)
The following URL requires authentication: http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1.dds
Please enter your OpenID and you will be redirected to the login page at that site
<input type="checkbox" name="rememberOpenid" checked="checked" �=""/> Remember my OpenID on this computer
</form>
After logging in, you will be redirected to: http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1.dds
My credentials are authorized by using a myproxyclient call before accessing the ESGF. This gives me a 12hour period in which to access data. ncdump and nccopy work fine. I guess you're getting the errors above because you first need to sign in through a proxy login... I was just a little lost by the fact an nco call would fail with a DAP server error. I'll do a little more digging when I get a spare moment. I'll also double check an ncks call to a public facing thredds server.
Thanks
James
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
ERROR: nco__open() unable to open file "http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc"
ERROR NC_EIO Generic IO error
HINT: NC_EIO errors can occur when NCO tries to read a file through a non-existant DAP location. Then NCO automatically tries to retrieve the file
her method, e.g., searching for a file of the same name in the current directory. This can trigger a subsequent error. When debugging the problem,
s the originating error (from NC_EIO).
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco__open()
nco_err_exit(): ERROR Error code is -68. Translation into English with nc_strerror(-68) is "NetCDF: I/O failure"
nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)
I'm using the binaries installation for windows:
NCO netCDF Operators version "4.6.0" built by USER on HOSTNAME at May 12 2016 17:17:59
ncks version "4.6.0"
Linked to netCDF library version 4.3.2, compiled Feb 29 2016 01:53:06
Copyright (C) 1995--2016 Charlie Zender
This program is part of NCO, the netCDF Operators.
NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO WARRANTY
You may redistribute and/or modify NCO under the terms of the
GNU General Public License (GPL) Version 3 with exceptions described in the LICENSE file
GPL: http://www.gnu.org/copyleft/gpl.html
LICENSE: https://github.com/nco/nco/tree/master/LICENSE
Homepage: http://nco.sf.net
User Guide: http://nco.sf.net/nco.html
Configuration Option: Active? Meaning or Reference:
Check _FillValue Yes http://nco.sf.net/nco.html#mss_val
Check missing_value No http://nco.sf.net/nco.html#mss_val
DAP clients Yes http://nco.sf.net/nco.html#dap
Debugging: Custom No Pedantic, bounds checking (slowest execution)
Debugging: Symbols No Produce symbols for debuggers (e.g., dbx, gdb)
ESMF Library No http://nco.sf.net/nco.html#esmf
GNU Scientific Library Yes http://nco.sf.net/nco.html#gsl
HDF4 support Unknown http://nco.sf.net/nco.html#hdf4
Internationalization No http://nco.sf.net/nco.html#i18n (pre-alpha)
MPI parallelization No http://nco.sf.net/nco.html#mpi (beta)
netCDF3 64-bit files Yes http://nco.sf.net/nco.html#lfs
netCDF4/HDF5 available Yes http://nco.sf.net/nco.html#nco4
netCDF4/HDF5 enabled Yes http://nco.sf.net/nco.html#nco4
OpenMP SMP threading No http://nco.sf.net/nco.html#omp
Optimization: run-time No Fastest execution possible (slowest compilation)
Parallel netCDF3 No http://nco.sf.net/nco.html#pnetcdf (pre-alpha)
Regular Expressions No http://nco.sf.net/nco.html#rx
Shared libraries built No Small, dynamically linked executables
Static libraries built No Large executables with private namespaces
UDUnits conversions Yes http://nco.sf.net/nco.html#udunits
UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits
AP Lang
Regards
Antonio
Last edit: Antonio S. Cofińo 2016-06-09
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have drill-down which directive from the .dodsrc is originating the problem and is was due the following one:
HTTP.SSL.CAPATH
The rest of .dodsrc directive are working.
The NCO binaries are linked statically with the netcdf-c library, but this static netcdf-c libraries are compiled with libcurl (for DAP support) but with no HTTPS protocol support:
Protocol "https" not supported or disabled in libcurl
I have compiled in Linux the NCO 4.6.0 agains the Netcdf 4.4.0 with DAP support and SSL and they work against ESGF data using SSL certifiates. Also cookies and redirection capabilities are working.
Hello,
I've been trying to sub set CMIP5 data from the ESGF portal with little success:
ncks -d lon,0,20,1 -d lat,170,190,1 http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1 /projectsa/Eurobasin/testing.nc
syntax error
context: ^
ncks: INFO DAP-access to http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1 failed with error code -70. Translation into English with nc_strerror(-70) is "NetCDF: DAP server error"
ncks: HINT DAP-access error code indicates that URL does exist, so error appears to be in DAP server. Reasons for this could include an aggregation server (e.g., GDS or THREDDS) that receives a URL that is valid according to the aggregation metadata but the implied back-end dataset may be temporarily unavailable.
however I can ncdump said file, and as an alternative method:
nccopy 'http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1?lon[0:1:20],lat[170:1:190],time,lev,uo[0:1:1128][0:1:39][170:1:190][0:1:20]' out.nc
also works. So what I'm not too sure about is whether this is an nco issue or something different? Any pointers appreciated.
Thanks
James
[for ref: I'm using the netcdf libraries from an anaconda install. I built nco using these - but the ncks seg faulted even on a simple ncks -r call, so I did a yum install nco instead]
Hi James,
This appears to be a permissions problem, that in turn may (not sure) point to inadequate authorization functionality in NCO. The first thing is to get ncks reading (not copying) from ESGF. Appended below is the lengthy error message I get. It boils down to an authorization error. I'm unfamiliar with exactly the syntax or handshakes ESGF needs to work with DAP. Any pointers appreciated. Try to get either of these commands working by including your ESGF login/password somewhere in the URL:
ncks -M http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1
ncdump -h http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1
Clearly nccopy finds what it needs. Not sure why. Suggestions welcome!
I've added ESGF access as TODO nco1123.
cz
zender@roulee:~$ ncks -M http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1
syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
Data Access Login
The following URL requires authentication:
http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1.dds
Please enter your OpenID and you will be redirected to the login page at that site<form method="post" action="/esg-orp/j_spring_openid_security_check.htm;jsessionid=D670C2A58E40A3B27F47E3E33D07488A"> <script language="javascript"> function sanitize() { openidElement = document.getElementById("openid_identifier"); openid = openidElement.value; openid = openid.replace("http:","https:") .replace(/^\s\s/, '').replace(/\s\s$/, ''); openidElement.value = openid; } </script>
After logging in, you will be redirected to:
http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1.dds
ncks: INFO DAP-access to http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1 failed with error code -78. Translation into English with nc_strerror(-78) is "NetCDF: Authorization failure"
ncks: WARNING not searching for http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1 on remote filesystem, using local file hredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1 instead
ERROR: ncoopen() unable to open file "thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1"
ERROR NC_ENOTNC Not a netCDF file
HINT: NC_ENOTNC errors can occur for many reasons. If your use-case matches one of the four listed below, take the corrective action indicated:
1. An NCO operator linked only to the classic netCDF3 library attempts to read netCDF4 (or HDF4 or HDF5) files. However, this executable seems to have been built with the capability to manipulate netCDF4 files, so it is unlikely that this command failed only because the input datasets are netCDF4 format. Something else is going wrong.
2. NCO attempts to utilize diskless (i.e., RAM) files. In this case remove the diskless switches (e.g., --ram or --diskless) and then re-issue the command.
3. NCO attempts to read other filetypes (HDF4, HDF-EOS2, pnetCDF) for which support must be (but was not) enabled at netCDF build-time. In this case it may be possible to access the input files using NCO if NCO is first re-linked to a version of netCDF configured with the --enable-hdf4 option. This is a non-standard netCDF build option described here: http://www.unidata.ucar.edu/software/netcdf/docs/build_hdf4.html
4. Access to a DAP URL fails, and the backup method of downloading the URL using wget obtains a data aggregation file (e.g., a .ncml file) instead of an actual netCDF file. In this case the problem is with the DAP server or URL.
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): ncoopen()
nco_err_exit(): ERROR Error code is -51. Translation into English with nc_strerror(-51) is "NetCDF: Unknown file format"
nco_err_exit(): ERROR NCO will now exit with system call abort()
Aborted (core dumped)
zender@roulee:~$ ncdump -h http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1
syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
Data Access Login
The following URL requires authentication:
http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1.dds
Please enter your OpenID and you will be redirected to the login page at that site<form method="post" action="/esg-orp/j_spring_openid_security_check.htm;jsessionid=43F667586EC43177645D485549B57812"> <script language="javascript"> function sanitize() { openidElement = document.getElementById("openid_identifier"); openid = openidElement.value; openid = openid.replace("http:","https:") .replace(/^\s\s/, '').replace(/\s\s$/, ''); openidElement.value = openid; } </script>
After logging in, you will be redirected to:
http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1.dds
ncdump: http://esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1: NetCDF: Authorization failure
The first thing to try is your usernamepassword in
ncks -M http://username:password@esgf-data2.ceda.ac.uk/thredds/dodsC/cmip5.output1.MOHC.HadGEM2-ES.rcp85.mon.ocean.Omon.r1i1p1.uo.20111014.aggregation.1
If that doesn't work, try to glean some tips from
https://www.unidata.ucar.edu/software/netcdf/docs/auth.html
cz
Charlie,
My credentials are authorized by using a myproxyclient call before accessing the ESGF. This gives me a 12hour period in which to access data. ncdump and nccopy work fine. I guess you're getting the errors above because you first need to sign in through a proxy login... I was just a little lost by the fact an nco call would fail with a DAP server error. I'll do a little more digging when I get a spare moment. I'll also double check an ncks call to a public facing thredds server.
Thanks
James
Any progresson this?
I'm trying to:
but with no success:
I'm using the binaries installation for windows:
Regards
Antonio
Last edit: Antonio S. Cofińo 2016-06-09
Hi Antonio,
I tried the above url with nco-4.5.2 and it worked fine
by that i mean:
ncks -m http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc
Also i tried:
ncdump http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc
and this works. DAP requries a lot of libs please check your netcdf install. If ncdump doesnt work then ncks will also fail
...Henry
Henry,
I have tested the ncdump and it's working, I forgot to mention in my message.
After your e-mail I have discovered that the problems is with the .dodsrc.
I have this .dodsrc to enable SSL authentication against a ESGF endpoint. If I remove the .dodsrc both the ncdump and ncks are working with the URL:
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis2/pressure/air.2014.nc
I have drill-down which directive from the .dodsrc is originating the problem and is was due the following one:
HTTP.SSL.CAPATH
The rest of .dodsrc directive are working.
The NCO binaries are linked statically with the netcdf-c library, but this static netcdf-c libraries are compiled with libcurl (for DAP support) but with no HTTPS protocol support:
Protocol "https" not supported or disabled in libcurl
Antonio
Last edit: Antonio S. Cofińo 2016-06-12
BTW
I have compiled in Linux the NCO 4.6.0 agains the Netcdf 4.4.0 with DAP support and SSL and they work against ESGF data using SSL certifiates. Also cookies and redirection capabilities are working.
More details in
http://www.unidata.ucar.edu/software/netcdf/docs/auth.html
Regards
Antonio