Extract netcdf data from an irregular grid

2014-06-23
2014-11-12
  • Alan Guillouzo

    Alan Guillouzo - 2014-06-23

    Hi all,

    For my job activities, i have to deal for the first time with netcdf files, i never used these data before, and i am not familiar at all with IT or command line, that's why i create this topic to ask some help...

    I want to extract data from this dataset (irregular grid)
    ftp://ftp.ifremer.fr/PREVIMER_WW3/NORGAS-UG_V3/best_estimate/2014/

    description of a similar product below
    http://tds1.ifremer.fr/thredds/dodsC/PREVIMER-MANCHEGASCOGNENORD-WW3_NORGAS_UG-FOR_FULL_TIME_SERIE.das

    I installed nco command language and ncdump. I don't know so far how to use them to get data I want. I want to extract hs value for coordinates between 49,48°N/49,52°N/1,18°W/1,12°W and then convert the .nc file in .xlsx.

    Is there an easy way to get this information for a file ? It might look easy for you all, but for me it's kind of really tricky..

    well a little help would be much appreciated..:)!

    thanks

    Alan

     
  • Charlie Zender

    Charlie Zender - 2014-06-23

    Hi Alan,
    these seem to be data from the WaveWatch model. lon/lat are stored in a 1-D array called nodenumber. This is the perfect application for the NCO -X switch.
    Get a netCDF file with the variables you want, including lat/lon, and then
    hyperslabe with e.g.,
    ncks -X 348.,352.,49.,48 in.nc out.nc
    http://nco.sf.net/nco.html#-X
    To convert result from .nc to .xlsx you'll have to search elsewhere.
    There are spreadsheet modules that will do this.
    ncks can also dump in csv format.
    cz

     
    • romzi04

      romzi04 - 2014-10-29

      Hi CZ,

      I am also working on the wave model WaveWatch, and I planned to use the ncks -X option to subset data from a specific region of my grid over time dimension.

      From the documentation I noticed the variable that you want to extract needs to have the "coordinates" attributes set to the variables referring the longitude and latitude.

      Hence I extract data to a single time frame just for testing.
      Then I add the coordinates attribute by:
      ncatted -O -a coordinates,hs,c,c,"longitude latitude" out.nc

      Finally I try to subset the data with:
      ncks --auxiliary 9.,10.,43.,44. -v hs out.nc out2.nc
      or
      ncks -X 9.,10.,43.,44. -v hs out.nc out2.nc

      Unfortunately, I obtain the same dataset with all the cells (nodes in my case).

      Do you think I missed something or it could be an issue from my NCO manually compilation ?

      Thanks,
      R.

      Ps. I work on nco 4.4.4

       
      Last edit: romzi04 2014-10-29
      • Charlie Zender

        Charlie Zender - 2014-10-29

        Hi R.,
        I can reproduce the problem you reported.
        My guess is that this has to do with the degenerate time dimension in hs. Other issues prevent me from testing this (ncwa fails to average-out time). So, multiple things may be going on. Will post again when I understand more.
        cz

         
        • romzi04

          romzi04 - 2014-10-29

          Many thanks.

          Do you want I upload a nc file with several time frames instead of the one extracted ?
          I keep testing from my side but without solution so far.

          R.

           
          • Charlie Zender

            Charlie Zender - 2014-10-29

            If you can produce a file that does not contain a time dimension,
            then post that I can test my hypothesis. i.e., hs(node) instead of hs(time,node).
            cz

             
            • romzi04

              romzi04 - 2014-10-30

              My ncwa works perfectly on the first uploaded file :

              Here is the freshly created file..but ncks -X still doesn't work here.

              R.

               
              • Charlie Zender

                Charlie Zender - 2014-10-31

                Yes, my ncwa had a spurious problem. Now I see that -X is broken. Apparently it hyperslabs the coordinate as intended yet does not hyperslab the variable as intended. The problem could be worse, and I have an idea when it was broken. Will post again when we've made progress fixing it.
                cz

                 
                • Charlie Zender

                  Charlie Zender - 2014-11-04

                  FYI we are working on fixing -X. Regression test has been added to demonstrate problem you encountered.

                   
                  • romzi04

                    romzi04 - 2014-11-05

                    Many thanks CZ ! I'm looking forward then :)

                    R.

                     
                    • Charlie Zender

                      Charlie Zender - 2014-11-10

                      Romzi,
                      Your use case should now work with the latest committed code.
                      Please test if possible. This will be in 4.4.7.
                      There are still a few more kinks to smooth out with auxiliary coordinates that we expect to fix soon. But your use-case should work now.
                      cz

                       
                      • Charlie Zender

                        Charlie Zender - 2014-11-11

                        All known auxiliary coordinate problems have been resolved. Feedback welcome.
                        cz

                         
                      • romzi04

                        romzi04 - 2014-11-12

                        Thanks Charlie, I'll test asap. Do I have to access the CVS repository or it has been packaged into the 4.4.6 tgz available ?
                        R.

                         
                        • Charlie Zender

                          Charlie Zender - 2014-11-12

                          Only available via CVS.
                          Cz

                           
                          • romzi04

                            romzi04 - 2014-11-12

                            Hi Charlie -

                            Things are working as expected ! From now -X does work with the file ; I tested on a) a single slice, then b) many of them through the degenerate time dimension. All tests are successful.

                            Many thanks.
                            Romain

                             
  • Alan Guillouzo

    Alan Guillouzo - 2014-06-24

    Hi Charlie,

    Thanks for your help and fast reply. I did extract variables i wanted (lat,long,hs,dir) in a new netcdf file with the command ncks -v in.nc out.nc

    Unfortunately, no way to subset another .nc according to the coordinates data i am asking for.

    When i run the command below, it actually creates a new file but it's the same size that the input file:
    ncks -X -1.18,-1.12,49.48,49.52 test1.nc test2.nc
    If i test with ncks -h, i can see there is still values for all nodes.

    I tried as well this command :
    ncks -d longitude,-1.18,-1.12 latitude,49.0,50.0 test1.nc test2.nc
    It returns => "ncks: ERROR dimension longitude is not in input file"

    Thanks again for your help
    Alan

     
  • Charlie Zender

    Charlie Zender - 2014-06-24

    please post the file with the exact command that does not work as expected.
    cz

     
  • Alan Guillouzo

    Alan Guillouzo - 2014-06-24

    It's ok, i get information i want with this command :
    ncks -C -d node,78630,78630 -v latitude,longitude,hs,dir in.nc > out.txt

    78630 represent coordinates where i want to extract data and create a time serie.

    I have .nc file dataset (same model, samevariables) for a period of six months, one file means one hour of data (i have 4000 files), and i am wondering though whether it's possible to concatenate all of them to make a single .nc file instead of making a routine?

    Have a good one
    Alan

     

Log in to post a comment.