I am trying to access data from this OPeNDAP server: https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-MONTHLY.html
using NCKS, but I cannot for the life of me figure out how I should specify the source. All the other examples I have seen have a .nc file sitting on the server, but not this particular server. I'm clearly doing something wrong, but can't quite figure out what....
The .html extension takes you to an OPeNDAP Dataset Query Form, which will let you look at the variable names, dimensions, attributes, etc., with checkboxes to help you form OPeNDAP queries. However, no netCDF program will understand that form.
Normally, to download a whole "file" as netCDF from a server, you would do the following: strip off the .html and use that as your input file to nccopy, e.g.,:
nccopy https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-MONTHLY out.nc
HOWEVER, that does not work in this case because the files have been aggregated over time into one big virtual dataset. As a result, the whole dataset is 6.6 Billion points, which blows out the memory in nccopy (and which you probably don't want anyway). In order to get, say, just the first time slice for one variable, use this syntax (the HTML form is really helpful at figuring this out, by the way):
The array indices are [start_index:stride:end_index]. Note that you can also extract spatial subsets by changing the 2nd and 3rd ranges, and (IMPORTANT!) adding the same ranges to lat and lon, e.g.,
Hi Mark,
I don't know what this server needs in order to produce a netCDF file.
Maybe it's not you, maybe it's the server?
Anyone know what the filename argument should be to get a netCDF file here?
cz
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
BTW, the HTML form gives the Data URL, which is normally what you would feed to ncks. However, ncks gives the same error as you see WITH the .html. On the other hand, if I do something similar with our own THREDDS server,
ncks -v nlon 'http://disc2.nascom.nasa.gov/thredds/dodsC/aggregation/TRMM_3B43/TRMM_3B43_Aggregation.ncml.ncml' out.nc
it works fine. I think ncks would probably be fine if it went ahead and used the URL as entered instead of erroring out because it doesn't like the URL.
Is it possible that ncks is not recognizing the https:// prefix as a valid OPeNDAP URL? Or wants to see some suffix/any suffix?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Our posts crossed. DAP syntax confuses me. Your most recent post helped me understand why NCO failed here: NCO expects DAP URLs to begin with http:// not https://. That's a simple thing to fix. I just committed a fix which does indeed work as expected:
zender@roulee:~$ ncks -D 4 -O https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-MONTHLY -d lat,0 -d lon,1 -d time,0 -v chlor_a ~/foo.nc
ncks: INFO nco_fl_mk_lcl() successfully accessed this file using the DAP protocol
ncks: INFO Extended filetype of https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-MONTHLY is NC_FORMAT_DAP2, mode = 0
ncks: CONVENTION File "Conventions" attribute is "CF-1.6"
ncks: TIMER Metadata setup and file layout before main loop took 0.26 s
ncks: INFO Moving /home/zender/foo.nc.pid14351.ncks.tmp to /home/zender/foo.nc...done
ncks: TIMER Wallclock-elapsed time for command is 0.28 s
This fix (allowing DAP servers that begin with https://) is now in the main trunk and will be in 4.4.5. Please test it, people :)
cz
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Just verifying that NCO hyperslabbing works as expected. It seeems so.
Hopefully this notation is easy for users to formulate themselves. Still embarassed that we never tested https:// URLs for DAP access until now. Oh well.
cz
Thanks for checking this out, and patching it up. Isn't open source wonderful?
I was able to get it working without problems for the monthly version of the data - everything seems to be working fine there. But extracting data from the daily version didn't prove to be so successful - I get a seg fault:
./ncks -D 4 https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-DAILY -d lat,43.0,48.0 -d lon,-5.0,-1.0 -v chlor_a Bob_Anchovy.nc --overwrite
ncks: INFO nco_fl_mk_lcl() successfully accessed this file using the DAP protocol
ncks: INFO Extended filetype of https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-DAILY is NC_FORMAT_UNDEFINED, mode = 0
ncks: CONVENTION File "Conventions" attribute is "CF-1.6"
ncks: TIMER Metadata setup and file layout before main loop took 0.33 s
Segmentation fault
Are you able to reproduce this error? Trying to extract temporal subsets of the daily data also works, but not the whole data set....
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
for me this command proceeds farther then dies with a different error. the failure is fairly inscrutable because "DAP failure" means the error occurs in a manner that is hidden from the NCO client. However, based on Chris' description of this dataset, it seems likely that the DAP server barfs because the requested size is too big. Sure enough, when one hyperslabs a subset of time (second command below), then the command works (for me). As to why your command fails for you at an earlier stage, that may be due to us using different netCDF libraries. Mine is netCDF 4.3.3-rc1, which may be more fault-tolerant than yours.
zender@roulee:~$ ncks -O -D 4 https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-DAILY -d lat,43.0,48.0 -d lon,-5.0,-1.0 -v chlor_a ~/foo.nc
ncks: INFO nco_fl_mk_lcl() successfully accessed this file using the DAP protocol
ncks: INFO Extended filetype of https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-DAILY is NC_FORMAT_DAP2, mode = 0
ncks: CONVENTION File "Conventions" attribute is "CF-1.6"
ncks: TIMER Metadata setup and file layout before main loop took 0.31 s
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_get_vara()
nco_err_exit(): ERROR Error code is -66. Translation into English with nc_strerror(-66) is "NetCDF: DAP failure"
nco_err_exit(): ERROR NCO will now exit with system call abort()
Aborted
zender@roulee:~$ ncks -O -D 4 https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-DAILY -d lat,43.0,48.0 -d lon,-5.0,-1.0 -d time,73 -v chlor_a ~/foo.nc
ncks: INFO nco_fl_mk_lcl() successfully accessed this file using the DAP protocol
ncks: INFO Extended filetype of https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-DAILY is NC_FORMAT_DAP2, mode = 0
ncks: CONVENTION File "Conventions" attribute is "CF-1.6"
ncks: TIMER Metadata setup and file layout before main loop took 0.28 s
ncks: INFO Moving /home/zender/foo.nc.pid18212.ncks.tmp to /home/zender/foo.nc...done
ncks: TIMER Wallclock-elapsed time for command is 0.30 s
ender@roulee:~$ ncks --lbr
Linked to netCDF library version 4.3.3-rc1, compiled May 18 2014 11:09:34
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
I am trying to access data from this OPeNDAP server:
https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-MONTHLY.html
using NCKS, but I cannot for the life of me figure out how I should specify the source. All the other examples I have seen have a .nc file sitting on the server, but not this particular server. I'm clearly doing something wrong, but can't quite figure out what....
Here's a typical error:
ncks https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-MONTHLY.html -v time -o foo.nc
ncks: ERROR file https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-MONTHLY.html neither exists locally nor matches remote filename patterns
Any ideas?
Mark
The .html extension takes you to an OPeNDAP Dataset Query Form, which will let you look at the variable names, dimensions, attributes, etc., with checkboxes to help you form OPeNDAP queries. However, no netCDF program will understand that form.
Normally, to download a whole "file" as netCDF from a server, you would do the following: strip off the .html and use that as your input file to nccopy, e.g.,:
nccopy https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-MONTHLY out.nc
HOWEVER, that does not work in this case because the files have been aggregated over time into one big virtual dataset. As a result, the whole dataset is 6.6 Billion points, which blows out the memory in nccopy (and which you probably don't want anyway). In order to get, say, just the first time slice for one variable, use this syntax (the HTML form is really helpful at figuring this out, by the way):
nccopy 'https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-MONTHLY?lat,lon,time[0:1:0],chlor_a[0:1:0][0:1:4319][0:1:8639]' out.nc
The array indices are [start_index:stride:end_index]. Note that you can also extract spatial subsets by changing the 2nd and 3rd ranges, and (IMPORTANT!) adding the same ranges to lat and lon, e.g.,
nccopy 'https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-MONTHLY?lat[2000:1:2010],lon[4000:1:4010],time[0:1:0],chlor_a[0:1:0][2000:1:2010][4000:1:4010]' out.nc
Which turns out to be a small area in the vicinity of 6 N, 13 W.
Hi Mark,
I don't know what this server needs in order to produce a netCDF file.
Maybe it's not you, maybe it's the server?
Anyone know what the filename argument should be to get a netCDF file here?
cz
BTW, the HTML form gives the Data URL, which is normally what you would feed to ncks. However, ncks gives the same error as you see WITH the .html. On the other hand, if I do something similar with our own THREDDS server,
ncks -v nlon 'http://disc2.nascom.nasa.gov/thredds/dodsC/aggregation/TRMM_3B43/TRMM_3B43_Aggregation.ncml.ncml' out.nc
it works fine. I think ncks would probably be fine if it went ahead and used the URL as entered instead of erroring out because it doesn't like the URL.
Is it possible that ncks is not recognizing the https:// prefix as a valid OPeNDAP URL? Or wants to see some suffix/any suffix?
Thanks Chris! Now that I better understand this DAP feature, it is on my radar to improve NCO support for it.
cz
Our posts crossed. DAP syntax confuses me. Your most recent post helped me understand why NCO failed here: NCO expects DAP URLs to begin with http:// not https://. That's a simple thing to fix. I just committed a fix which does indeed work as expected:
zender@roulee:~$ ncks -D 4 -O https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-MONTHLY -d lat,0 -d lon,1 -d time,0 -v chlor_a ~/foo.nc
ncks: INFO nco_fl_mk_lcl() successfully accessed this file using the DAP protocol
ncks: INFO Extended filetype of https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-MONTHLY is NC_FORMAT_DAP2, mode = 0
ncks: CONVENTION File "Conventions" attribute is "CF-1.6"
ncks: TIMER Metadata setup and file layout before main loop took 0.26 s
ncks: INFO Moving /home/zender/foo.nc.pid14351.ncks.tmp to /home/zender/foo.nc...done
ncks: TIMER Wallclock-elapsed time for command is 0.28 s
This fix (allowing DAP servers that begin with https://) is now in the main trunk and will be in 4.4.5. Please test it, people :)
cz
Just verifying that NCO hyperslabbing works as expected. It seeems so.
Hopefully this notation is easy for users to formulate themselves. Still embarassed that we never tested https:// URLs for DAP access until now. Oh well.
cz
zender@roulee:~$ ncks -u -H -O https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-MONTHLY -d lat,0.0 -d lon,180. -d time,73 -v chlor_a
time[73] lat[2159]=0.0208333 lon[8639]=179.979 chlor_a[2743372799]=0.209975 milligram m-3
lat[2159]=0.0208333 degrees_north
lon[8639]=179.979 degrees_east
time[73] maxStrlen64[0] time[4672--4692]="2003-10-01T00:00:00Z"
Hi,
Thanks for checking this out, and patching it up. Isn't open source wonderful?
I was able to get it working without problems for the monthly version of the data - everything seems to be working fine there. But extracting data from the daily version didn't prove to be so successful - I get a seg fault:
./ncks -D 4 https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-DAILY -d lat,43.0,48.0 -d lon,-5.0,-1.0 -v chlor_a Bob_Anchovy.nc --overwrite
ncks: INFO nco_fl_mk_lcl() successfully accessed this file using the DAP protocol
ncks: INFO Extended filetype of https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-DAILY is NC_FORMAT_UNDEFINED, mode = 0
ncks: CONVENTION File "Conventions" attribute is "CF-1.6"
ncks: TIMER Metadata setup and file layout before main loop took 0.33 s
Segmentation fault
Are you able to reproduce this error? Trying to extract temporal subsets of the daily data also works, but not the whole data set....
for me this command proceeds farther then dies with a different error. the failure is fairly inscrutable because "DAP failure" means the error occurs in a manner that is hidden from the NCO client. However, based on Chris' description of this dataset, it seems likely that the DAP server barfs because the requested size is too big. Sure enough, when one hyperslabs a subset of time (second command below), then the command works (for me). As to why your command fails for you at an earlier stage, that may be due to us using different netCDF libraries. Mine is netCDF 4.3.3-rc1, which may be more fault-tolerant than yours.
zender@roulee:~$ ncks -O -D 4 https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-DAILY -d lat,43.0,48.0 -d lon,-5.0,-1.0 -v chlor_a ~/foo.nc
ncks: INFO nco_fl_mk_lcl() successfully accessed this file using the DAP protocol
ncks: INFO Extended filetype of https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-DAILY is NC_FORMAT_DAP2, mode = 0
ncks: CONVENTION File "Conventions" attribute is "CF-1.6"
ncks: TIMER Metadata setup and file layout before main loop took 0.31 s
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_get_vara()
nco_err_exit(): ERROR Error code is -66. Translation into English with nc_strerror(-66) is "NetCDF: DAP failure"
nco_err_exit(): ERROR NCO will now exit with system call abort()
Aborted
zender@roulee:~$ ncks -O -D 4 https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-DAILY -d lat,43.0,48.0 -d lon,-5.0,-1.0 -d time,73 -v chlor_a ~/foo.nc
ncks: INFO nco_fl_mk_lcl() successfully accessed this file using the DAP protocol
ncks: INFO Extended filetype of https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v1.0-DAILY is NC_FORMAT_DAP2, mode = 0
ncks: CONVENTION File "Conventions" attribute is "CF-1.6"
ncks: TIMER Metadata setup and file layout before main loop took 0.28 s
ncks: INFO Moving /home/zender/foo.nc.pid18212.ncks.tmp to /home/zender/foo.nc...done
ncks: TIMER Wallclock-elapsed time for command is 0.30 s
ender@roulee:~$ ncks --lbr
Linked to netCDF library version 4.3.3-rc1, compiled May 18 2014 11:09:34