Convert HDF to Netcdf

Help
luis
2014-03-11
2014-08-28
  • luis
    luis
    2014-03-11

    Hi
    I need help to convert HDF from OceanColour to Netcdf.
    I used the NCO but i dont have sucess.
    The error is this

    @ubuntu:~$ ncks -hdf4 /home/luis/Desktop/T2013001201300L3m_8D_SST_4.hdf /home/luis/Desktop/t.nc
    ncks: ERROR in hyperslab specification for dimension f4
    ncks: HINT Conform request to hyperslab documentation at http://nco.sf.net/nco.html#hyp

    I need help.
    Thanks

    Luis

     
  • Pedro Vicente
    Pedro Vicente
    2014-03-11

    Can you post the file somewhere or send it to pvicente@uci.edu?
    Pedro

     
  • Charlie Zender
    Charlie Zender
    2014-03-11

    Your syntax is mistaken, use two dashes on long option "--hdf4":

    ncks --hdf4 /home/luis/Desktop/T2013001201300L3m_8D_SST_4.hdf /home/luis/Desktop/t.nc

    cz

     
  • luis
    luis
    2014-03-12

    luis@ubuntu:~$ ncks --hdf4 /home/luis/Desktop/T2013001201300L3m_8D_SST_4.hdf /home/luis/Desktop/t.nc
    ncks: unrecognized option '--hdf4'
    ncks command line options cheat-sheet:
    ncks [-3] [-4] [-A] [-a] [-B] [-b fl_bnr] [-C] [-c] [--cnk_dmn nm,sz] [--cnk_map map] [--cnk_plc plc] [--cnk_scl sz] [-D dbg_lvl] [-d ...] [-F] [-H] [-h] [--hdr_pad nbr] [-L lvl] [-l path] [-m] [-M] [-O] [-o out.nc] [-P] [-p path] [-Q] [-q] [-R] [-r] [-s format] [-u] [-v ...] [-X box] [-x] in.nc [[out.nc]]

    -3, --3, --fl_fmt=classic Output file in netCDF3 CLASSIC storage format
    -A, --apn, --append Append to existing output file, if any
    -a, --abc, --alphabetize Disable alphabetization of extracted variables
    -B, --bnr, --binary Write data to unformatted binary file
    -b, --fl_bnr, --binary-file fl_bnr Unformatted binary file to write
    -C, --nocoords Associated coordinate variables should not be processed
    -c, --crd, --coords Coordinate variables will all be processed
    --cnk_dmn, --chunk_dimension nm,sz Chunksize of dimension nm is sz
    --cnk_map, --chunk_map map Chunking map [dmn,rd1,prd,xpl]
    --cnk_plc, --chunk_policy plc Chunking policy [all,g2d,g3d,xpl,uck]
    --cnk_scl, --chunk_scalar sz Chunksize scalar (for all dimensions)
    -D, --dbg_lvl, --debug-level dbg_lvl Debugging level
    -d, --dmn, --dimension dim,[min][,[max]][,[stride]] Dimension's limits and stride in hyperslab
    -F, --ftn, --fortran Fortran indexing conventions (1-based) for I/O
    -H, --huh, --hmm Toggle printing data
    -h, --hst, --history Do not append to "history" global attribute
    --hdr_pad, --header_pad Pad output header with nbr bytes
    -l, --lcl, --local path Local storage path for remotely-retrieved files
    -M, --Mtd, --Metadata Toggle printing global metadata
    -m, --mtd, --metadata Toggle printing variable metadata
    -o, --output, --fl_out fl_out Output file name (or use last positional argument)
    -O, --ovr, --overwrite Overwrite existing output file, if any
    -P, --prn, --print Print data, metadata, and units. Abbreviation for -C -H -M -m -u.
    -p, --pth, --path path Path prefix for all input filenames
    -Q, Toggle printing of dimension indices and coordinate values
    -q, --quiet Turn off all printing to screen
    -R, --rtn, --retain Retain remotely-retrieved files after use
    -r, --revision, --vrs, --version Compile-time configuration and/or program version
    -s, --sng_fmt, --string format String format for text output
    -u, --units Toggle printing units of variables, if any
    -v, --variable var1[,var2[...]] Variable(s) to process (regular expressions supported)
    -X, --auxiliary lon_min,lon_max,lat_min,lat_max Auxiliary coordinate bounding box
    -x, --xcl, --exclude Extract all variables EXCEPT those specified with -v
    in.nc Input file name
    out.nc Output file name (or use -o switch)

    Eight ways to find more help on ncks and/or NCO:
    1. Concrete ncks examples: http://nco.sf.net/nco.html#xmp_ncks
    2. Reference manual for ncks: http://nco.sf.net/nco.html#ncks
    3. User's Guide: http://nco.sf.net#RTFM
    4. Man pages: 'man ncks', 'man nco'
    5. Homepage: http://nco.sf.net
    6. FAQ: http://nco.sf.net#FAQ
    7. Help Forum: http://sf.net/forum/forum.php?forum_id=9830
    8. NCO-related publications: http://nco.sf.net#pub
    Post questions, suggestions, patches at http://sf.net/projects/nco
    luis@ubuntu:~$

    dont work..result in this..

     
    • ncks fl.hdf fl.nc # Convert HDF4->netCDF4 (NCO 4.4.0+, netCDF 4.3.1+)
      ncks --hdf4 fl.hdf fl.nc # Convert HDF4->netCDF4 (NCO 4.3.7-4.3.9)

      On Wed, Mar 12, 2014 at 6:57 AM, luis luisgeotig@users.sf.net wrote:

      luis@ubuntu:~$ ncks --hdf4 /home/luis/Desktop/T2013001201300L3m_8D_SST_4.hdf
      /home/luis/Desktop/t.nc
      ncks: unrecognized option '--hdf4'
      ncks command line options cheat-sheet:
      ncks [-3] [-4] [-A] [-a] [-B] [-b fl_bnr] [-C] [-c] [--cnk_dmn nm,sz]
      [--cnk_map map] [--cnk_plc plc] [--cnk_scl sz] [-D dbg_lvl] [-d ...] [-F]
      [-H] [-h] [--hdr_pad nbr] [-L lvl] [-l path] [-m] [-M] [-O] [-o out.nc] [-P]
      [-p path] [-Q] [-q] [-R] [-r] [-s format] [-u] [-v ...] [-X box] [-x] in.nc
      [[out.nc]]

      -3, --3, --fl_fmt=classic Output file in netCDF3 CLASSIC storage format
      -A, --apn, --append Append to existing output file, if any
      -a, --abc, --alphabetize Disable alphabetization of extracted variables
      -B, --bnr, --binary Write data to unformatted binary file
      -b, --fl_bnr, --binary-file fl_bnr Unformatted binary file to write
      -C, --nocoords Associated coordinate variables should not be processed
      -c, --crd, --coords Coordinate variables will all be processed
      --cnk_dmn, --chunk_dimension nm,sz Chunksize of dimension nm is sz
      --cnk_map, --chunk_map map Chunking map [dmn,rd1,prd,xpl]
      --cnk_plc, --chunk_policy plc Chunking policy [all,g2d,g3d,xpl,uck]
      --cnk_scl, --chunk_scalar sz Chunksize scalar (for all dimensions)
      -D, --dbg_lvl, --debug-level dbg_lvl Debugging level
      -d, --dmn, --dimension dim,[min][,[max]][,[stride]] Dimension's limits and
      stride in hyperslab
      -F, --ftn, --fortran Fortran indexing conventions (1-based) for I/O
      -H, --huh, --hmm Toggle printing data
      -h, --hst, --history Do not append to "history" global attribute
      --hdr_pad, --header_pad Pad output header with nbr bytes
      -l, --lcl, --local path Local storage path for remotely-retrieved files
      -M, --Mtd, --Metadata Toggle printing global metadata
      -m, --mtd, --metadata Toggle printing variable metadata
      -o, --output, --fl_out fl_out Output file name (or use last positional
      argument)
      -O, --ovr, --overwrite Overwrite existing output file, if any
      -P, --prn, --print Print data, metadata, and units. Abbreviation for -C -H
      -M -m -u.
      -p, --pth, --path path Path prefix for all input filenames
      -Q, Toggle printing of dimension indices and coordinate values
      -q, --quiet Turn off all printing to screen
      -R, --rtn, --retain Retain remotely-retrieved files after use
      -r, --revision, --vrs, --version Compile-time configuration and/or program
      version
      -s, --sng_fmt, --string format String format for text output
      -u, --units Toggle printing units of variables, if any
      -v, --variable var1[,var2[...]] Variable(s) to process (regular expressions
      supported)
      -X, --auxiliary lon_min,lon_max,lat_min,lat_max Auxiliary coordinate
      bounding box
      -x, --xcl, --exclude Extract all variables EXCEPT those specified with -v
      in.nc Input file name
      out.nc Output file name (or use -o switch)

      Eight ways to find more help on ncks and/or NCO:
      1. Concrete ncks examples: http://nco.sf.net/nco.html#xmp_ncks
      2. Reference manual for ncks: http://nco.sf.net/nco.html#ncks
      3. User's Guide: http://nco.sf.net#RTFM
      4. Man pages: 'man ncks', 'man nco'
      5. Homepage: http://nco.sf.net
      6. FAQ: http://nco.sf.net#FAQ
      7. Help Forum: http://sf.net/forum/forum.php?forum_id=9830
      8. NCO-related publications: http://nco.sf.net#pub
      Post questions, suggestions, patches at http://sf.net/projects/nco
      luis@ubuntu:~$

      dont work..result in this..


      Convert HDF to Netcdf


      Sent from sourceforge.net because you indicated interest in
      https://sourceforge.net/p/nco/discussion/9830/

      To unsubscribe from further messages, please visit
      https://sourceforge.net/auth/subscriptions/

      --
      Dr. Richard P. Signell (508) 457-2229
      USGS, 384 Woods Hole Rd.
      Woods Hole, MA 02543-1598

       
  • Probably a NCO version issue. On the documentation at
    http://nco.sourceforge.net/nco.html#ncks-netCDF-Kitchen-Sink
    it says:

    Converting HDF4 files to netCDF: Since NCO reads HDF4 files natively, it is now easy to convert HDF4 files to netCDF files directly, e.g.,

    ncks fl.hdf fl.nc # Convert HDF4->netCDF4 (NCO 4.4.0+, netCDF 4.3.1+)
    ncks --hdf4 fl.hdf fl.nc # Convert HDF4->netCDF4 (NCO 4.3.7-4.3.9)
    The most efficient and accurate way to convert HDF4 data to netCDF format is to convert to netCDF4 using NCO as above. Many HDF4 producers (NASA!) love to use netCDF4 types, e.g., unsigned bytes, so this procedure is the most typical. Conversion of HDF4 to netCDF4 as above suffices when the data will only be processed by NCO and other netCDF4-aware tools.

     
  • luis
    luis
    2014-03-12

    I have the version 3.9.9-1, avaible on Ubuntu Software Center.

     
  • Yipes, In July, your version will be 5 years old !

    Time to upgrade!

     
  • luis
    luis
    2014-03-12

    Yes,but isn-t avaibale on Ubuntu Software center and i dont install for other way NCO

     
  • Charlie Zender
    Charlie Zender
    2014-03-12

    Thanks, Rich, you are right that this is a version issue. It's hard to summarize in one line. The upshot is that NCO 4.3.7+ is required, along with netCDF built with HDF4 support. Sorry, Luis, you may need to enlist a sysadmin to build these packages for you.
    cz

     
  • luis
    luis
    2014-03-12

    Ok.
    Its a big help, i will try install this version and solve the problems.
    Thanks

    luis

     
  • luis
    luis
    2014-03-19

    Hi again
    I don't did convert yet.
    The error is
    luis@ubuntu:~$ cd /home/luis/Desktop
    luis@ubuntu:~/Desktop$ ncks T2004001200400_L3m_8D_CHL_chlor_a_4km.hdf a.hdf
    ERROR: ncoopen() unable to open file "T2004001200400_L3m_8D_CHL_chlor_a_4km.hdf"
    ERROR NC_ENOTNC Not a netCDF file
    HINT: NC_ENOTNC errors can occur for many reasons. If your use case matches one of the four listed below, take the corrective action indicated:
    1. An NCO operator linked to the netCDF3 library attempts to read netCDF4 files. Are your input files netCDF4 format? (http://nco.sf.net/nco.html#fmt_inq shows how to tell.) If so then installing or re-building a netCDF4-compatible version of NCO should solve this problem. First upgrade netCDF to version 4.x, then install NCO using those netCDF 4.x libraries.
    2. NC_ENOTNC can occur when users attempt to utilize diskless (i.e., RAM) files. In this case remove the diskless switches (e.g., --ram or --diskless) and then re-issue the command.
    2. NCO attempts to utilize diskless (i.e., RAM) files. In this case remove the diskless switches (e.g., --ram or --diskless) and then re-issue the command.
    3. NCO attempts to read other non-supported filetypes, e.g., HDF4, HDF-EOS2. In this case it may be possible to access the input files using NCO if NCO is first re-linked to a version of netCDF configured with the --enable-hdf4 option. This is a non-standard netCDF build option described here: http://www.unidata.ucar.edu/software/netcdf/docs/build_hdf4.html
    4. Access to a DAP URL fails, and the backup method of downloading the URL using wget obtains a data aggregation file (e.g., a .ncml file) instead of an actual netCDF file. In this case the problem is with the DAP server or URL.
    nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco
    open()
    nco_err_exit(): ERROR Error code is -51. Translation into English with nc_strerror(-51) is "NetCDF: Unknown file format"
    nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)

    The version is 4.4.2

     
  • Charlie Zender
    Charlie Zender
    2014-03-19

    Did you read and follow the instructions for problem #3 above?
    3. NCO attempts to read other non-supported filetypes, e.g., HDF4, HDF-EOS2. In this case it may be possible to access the input files using NCO if NCO is first re-linked to a version of netCDF configured with the --enable-hdf4 option. This is a non-standard netCDF build option described here: http://www.unidata.ucar.edu/software/netcdf/docs/build_hdf4.html
    cz

     
  • Hi

    I want to convert MODIS HDF files into a NetCDF format. I have not compiled the NCO operators with large file size capability and I am getting the following error:

    ncks: nc4file.c:2696: NC4_open: Assertion `0' failed.
    Aborted (core dumped)

    I have used these commands for performing the conversion:
    ~$ ncpdq --hdf_upk -P xst_new MOD05_L2.A2000243.2000.005.2006265220542.hdf modis.nc

    ~$ ncks --hdf4 /home/nghajarnia/MOD05_L2.A2000243.2000.005.2006265220542.hdf /home/nghajarnia/MODIS.nc

    Can you please help what must i do?

    Thank you
    Navid

     
    Last edit: Navid Ghajarnia 2014-08-28
  • Charlie Zender
    Charlie Zender
    2014-08-28

    it may be that your Ubuntu does not have sufficiently new NCO and netCDF4.
    put the HDF file where we can grab it, and i will try to convert it with the latest NCO to test that hypothesis.
    cz