Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

Extract netcdf data from an irregular grid

2014-06-23
2014-11-12
  • Alan Guillouzo
    Alan Guillouzo
    2014-06-23

    Hi all,

    For my job activities, i have to deal for the first time with netcdf files, i never used these data before, and i am not familiar at all with IT or command line, that's why i create this topic to ask some help...

    I want to extract data from this dataset (irregular grid)
    ftp://ftp.ifremer.fr/PREVIMER_WW3/NORGAS-UG_V3/best_estimate/2014/

    description of a similar product below
    http://tds1.ifremer.fr/thredds/dodsC/PREVIMER-MANCHEGASCOGNENORD-WW3_NORGAS_UG-FOR_FULL_TIME_SERIE.das

    I installed nco command language and ncdump. I don't know so far how to use them to get data I want. I want to extract hs value for coordinates between 49,48°N/49,52°N/1,18°W/1,12°W and then convert the .nc file in .xlsx.

    Is there an easy way to get this information for a file ? It might look easy for you all, but for me it's kind of really tricky..

    well a little help would be much appreciated..:)!

    thanks

    Alan

     
  • Charlie Zender
    Charlie Zender
    2014-06-23

    Hi Alan,
    these seem to be data from the WaveWatch model. lon/lat are stored in a 1-D array called nodenumber. This is the perfect application for the NCO -X switch.
    Get a netCDF file with the variables you want, including lat/lon, and then
    hyperslabe with e.g.,
    ncks -X 348.,352.,49.,48 in.nc out.nc
    http://nco.sf.net/nco.html#-X
    To convert result from .nc to .xlsx you'll have to search elsewhere.
    There are spreadsheet modules that will do this.
    ncks can also dump in csv format.
    cz

     
    • romzi04
      romzi04
      2014-10-29

      Hi CZ,

      I am also working on the wave model WaveWatch, and I planned to use the ncks -X option to subset data from a specific region of my grid over time dimension.

      From the documentation I noticed the variable that you want to extract needs to have the "coordinates" attributes set to the variables referring the longitude and latitude.

      Hence I extract data to a single time frame just for testing.
      Then I add the coordinates attribute by:
      ncatted -O -a coordinates,hs,c,c,"longitude latitude" out.nc

      Finally I try to subset the data with:
      ncks --auxiliary 9.,10.,43.,44. -v hs out.nc out2.nc
      or
      ncks -X 9.,10.,43.,44. -v hs out.nc out2.nc

      Unfortunately, I obtain the same dataset with all the cells (nodes in my case).

      Do you think I missed something or it could be an issue from my NCO manually compilation ?

      Thanks,
      R.

      Ps. I work on nco 4.4.4

       
      Last edit: romzi04 2014-10-29
      Attachments
      • Charlie Zender
        Charlie Zender
        2014-10-29

        Hi R.,
        I can reproduce the problem you reported.
        My guess is that this has to do with the degenerate time dimension in hs. Other issues prevent me from testing this (ncwa fails to average-out time). So, multiple things may be going on. Will post again when I understand more.
        cz

         
        • romzi04
          romzi04
          2014-10-29

          Many thanks.

          Do you want I upload a nc file with several time frames instead of the one extracted ?
          I keep testing from my side but without solution so far.

          R.

           
          • Charlie Zender
            Charlie Zender
            2014-10-29

            If you can produce a file that does not contain a time dimension,
            then post that I can test my hypothesis. i.e., hs(node) instead of hs(time,node).
            cz

             
            • romzi04
              romzi04
              2014-10-30

              My ncwa works perfectly on the first uploaded file :

              Here is the freshly created file..but ncks -X still doesn't work here.

              R.

               
              Attachments
              • Charlie Zender
                Charlie Zender
                2014-10-31

                Yes, my ncwa had a spurious problem. Now I see that -X is broken. Apparently it hyperslabs the coordinate as intended yet does not hyperslab the variable as intended. The problem could be worse, and I have an idea when it was broken. Will post again when we've made progress fixing it.
                cz

                 
                • Charlie Zender
                  Charlie Zender
                  2014-11-04

                  FYI we are working on fixing -X. Regression test has been added to demonstrate problem you encountered.

                   
                  • romzi04
                    romzi04
                    2014-11-05

                    Many thanks CZ ! I'm looking forward then :)

                    R.

                     
                    • Charlie Zender
                      Charlie Zender
                      2014-11-10

                      Romzi,
                      Your use case should now work with the latest committed code.
                      Please test if possible. This will be in 4.4.7.
                      There are still a few more kinks to smooth out with auxiliary coordinates that we expect to fix soon. But your use-case should work now.
                      cz

                       
                      • Charlie Zender
                        Charlie Zender
                        2014-11-11

                        All known auxiliary coordinate problems have been resolved. Feedback welcome.
                        cz

                         
                      • romzi04
                        romzi04
                        2014-11-12

                        Thanks Charlie, I'll test asap. Do I have to access the CVS repository or it has been packaged into the 4.4.6 tgz available ?
                        R.

                         
                        • Charlie Zender
                          Charlie Zender
                          2014-11-12

                          Only available via CVS.
                          Cz

                           
                          • romzi04
                            romzi04
                            2014-11-12

                            Hi Charlie -

                            Things are working as expected ! From now -X does work with the file ; I tested on a) a single slice, then b) many of them through the degenerate time dimension. All tests are successful.

                            Many thanks.
                            Romain

                             
  • Alan Guillouzo
    Alan Guillouzo
    2014-06-24

    Hi Charlie,

    Thanks for your help and fast reply. I did extract variables i wanted (lat,long,hs,dir) in a new netcdf file with the command ncks -v in.nc out.nc

    Unfortunately, no way to subset another .nc according to the coordinates data i am asking for.

    When i run the command below, it actually creates a new file but it's the same size that the input file:
    ncks -X -1.18,-1.12,49.48,49.52 test1.nc test2.nc
    If i test with ncks -h, i can see there is still values for all nodes.

    I tried as well this command :
    ncks -d longitude,-1.18,-1.12 latitude,49.0,50.0 test1.nc test2.nc
    It returns => "ncks: ERROR dimension longitude is not in input file"

    Thanks again for your help
    Alan

     
  • Charlie Zender
    Charlie Zender
    2014-06-24

    please post the file with the exact command that does not work as expected.
    cz

     
  • Alan Guillouzo
    Alan Guillouzo
    2014-06-24

    It's ok, i get information i want with this command :
    ncks -C -d node,78630,78630 -v latitude,longitude,hs,dir in.nc > out.txt

    78630 represent coordinates where i want to extract data and create a time serie.

    I have .nc file dataset (same model, samevariables) for a period of six months, one file means one hour of data (i have 4000 files), and i am wondering though whether it's possible to concatenate all of them to make a single .nc file instead of making a routine?

    Have a good one
    Alan