Menu

ncatted > 20 min per monthly files ??

Help
2017-05-24
2017-05-25
  • Joana de Medeiros

    Dear all (I am new at this forum) , I am using ncatted to change some attributes such as units and names - see bellow - Each file is 1.1GB monthly files with daily fields for 7 variables, it is taking more than 20 min (and still counting) per file. It is actually taking more time than the bicubic interpolation I did for each file. Any reason why this long ? Some feedback would be much appreciated as at this rate it will take more than a week to complete all 456 files (12monthsx38years) . Kind regards. Joana
    ~~~

    !/usr/bin/sh

    for file in EU_MERRA2_sfc_5km_*.nc
    do

    change name atts

    ncatted -O -a long_name,W10,o,c,wind_speed $file
    ncatted -O -a standard_name,W10,o,c,wind_speed $file

    change units

    ncatted -O -a units,Tmax,m,c,"degrees celsius" $file
    ncatted -O -a units,Tmin,m,c,"degrees celsius" $file]
    ncatted -O -a units,Tmean,m,c,"degrees celsius" $file
    ncatted -O -a units,TP,m,c,"millimeters" $file
    done

     
  • Joana de Medeiros

    It took 25 mins, and outputed the error "ncatted: ERROR file EU_MERRA2_sfc_5km_198001.nc] neither exists locally nor matches remote filename patterns"

    After I did an ncdump -h on the file that exists, it changed all but Tmin :
    ncdump -h EU_MERRA2_sfc_5km_198001.nc
    netcdf EU_MERRA2_sfc_5km_198001 {
    dimensions:
    lon = 1500 ;
    lat = 900 ;
    time = UNLIMITED ; // (31 currently)
    variables:
    double lon(lon) ;
    lon:standard_name = "longitude" ;
    lon:long_name = "longitude" ;
    lon:units = "degrees_east" ;
    lon:axis = "X" ;
    double lat(lat) ;
    lat:standard_name = "latitude" ;
    lat:long_name = "latitude" ;
    lat:units = "degrees_north" ;
    lat:axis = "Y" ;
    double time(time) ;
    time:standard_name = "time" ;
    time:long_name = "time" ;
    time:units = "days since 1980-1-1 00:00:00" ;
    time:calendar = "standard" ;
    time:axis = "T" ;
    float TP(time, lat, lon) ;
    TP:long_name = "total_precipitation" ;
    TP:units = "millimeters" ;
    TP:_FillValue = 1.e+15f ;
    TP:missing_value = 1.e+15f ;
    float Tmax(time, lat, lon) ;
    Tmax:standard_name = "2-meter_air_temperature" ;
    Tmax:long_name = "2-meter_air_temperature" ;
    Tmax:units = "degrees celsius" ;
    Tmax:_FillValue = 1.e+15f ;
    Tmax:missing_value = 1.e+15f ;
    Tmax:fmissing_value = 1.e+15f ;
    Tmax:vmax = 1.e+15f ;
    Tmax:vmin = -1.e+15f ;
    float Tmean(time, lat, lon) ;
    Tmean:standard_name = "2-meter_air_temperature" ;
    Tmean:long_name = "2-meter_air_temperature" ;
    Tmean:units = "degrees celsius" ;
    Tmean:_FillValue = 1.e+15f ;
    Tmean:missing_value = 1.e+15f ;
    Tmean:fmissing_value = 1.e+15f ;
    Tmean:vmax = 1.e+15f ;
    Tmean:vmin = -1.e+15f ;
    float Tmin(time, lat, lon) ;
    Tmin:standard_name = "2-meter_air_temperature" ;
    Tmin:long_name = "2-meter_air_temperature" ;
    Tmin:units = "K" ;
    Tmin:_FillValue = 1.e+15f ;
    Tmin:missing_value = 1.e+15f ;
    Tmin:fmissing_value = 1.e+15f ;
    Tmin:vmax = 1.e+15f ;
    Tmin:vmin = -1.e+15f ;
    float U10(time, lat, lon) ;
    U10:standard_name = "10-meter_eastward_wind" ;
    U10:long_name = "10-meter_eastward_wind" ;
    U10:units = "m s-1" ;
    U10:_FillValue = 1.e+15f ;
    U10:missing_value = 1.e+15f ;
    U10:fmissing_value = 1.e+15f ;
    U10:vmax = 1.e+15f ;
    U10:vmin = -1.e+15f ;
    float V10(time, lat, lon) ;
    V10:standard_name = "10-meter_northward_wind" ;
    V10:long_name = "10-meter_northward_wind" ;
    V10:units = "m s-1" ;
    V10:_FillValue = 1.e+15f ;
    V10:missing_value = 1.e+15f ;
    V10:fmissing_value = 1.e+15f ;
    V10:vmax = 1.e+15f ;
    V10:vmin = -1.e+15f ;
    float W10(time, lat, lon) ;
    W10:standard_name = "wind_speed" ;
    W10:long_name = "wind_speed" ;
    W10:units = "m s-1" ;
    W10:_FillValue = 1.e+15f ;
    W10:missing_value = 1.e+15f ;
    W10:fmissing_value = 1.e+15f ;
    W10:vmax = 1.e+15f ;
    W10:vmin = -1.e+15f ;

    many thanks in advance
    J

     
  • Joana de Medeiros

    Sorry the Tmin had the "]" at the end, but still 25 min per file is too long !
    J

     
    • Charlie Zender

      Charlie Zender - 2017-05-24

      Two suggestions: Combine all your edits into one command with multiple -a options (read the manual if that's unclear). That should speed things up by a factor of about five. You might also consider adding more header space with --hdr (read the manual) since otherwise the addition of attributes may require unnecessary file copying.

       
  • Joana de Medeiros

    thanks, will do
    J

     
  • Joana de Medeiros

    morning,
    I used this

    ncatted --hdr -O -a long_name,W10,o,c,wind_speed -a standard_name,W10,o,c,wind_speed -a units,Tmax,m,c,"degrees celsius" -a units,Tmin,m,c,"degrees celsius" -a units,Tmean,m,c,"degrees celsius" -a units,TP,m,c,"millimeters" $file
    

    I had the following error :

    nco_sng_cnv_err(): ERROR an NCO function or main program attempted to convert the user-defined string "-O" to an integer-type using the standard C-library function "strtoul()". This function stopped converting the input string when it encountered the illegal (i.e., non-numeric or non-integer) character '-'. This probably indicates a syntax error by the user. Please check the argument syntax and re-try the command. Exiting...
    nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_sng_cnv_err()
    nco_err_exit(): ERROR Error code is 0. This indicates an error occurred outside of the netCDF layer, i.e., in NCO code or in a system call.
    nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)
    

    Thanks in advance
    p.s: couldn't find --hdr examples, so might be doing something wrong !

     
  • Joana de Medeiros

    Thanks for the advice,
    I did

    ncatted --hdr 2 -O -a long_name,W10,o,c,wind_speed -a standard_name,W10,o,c,wind_speed -a units,Tmax,m,c,"degrees celsius" -a units,Tmin,m,c,"degrees celsius" -a units,Tmean,m,c,"degrees celsius" -a units,TP,m,c,"millimeters" $file
    

    It just shortened by 5 mins, it takes 20min per file, still quite considerate amount of time just to edit ...
    J

     
  • Charlie Zender

    Charlie Zender - 2017-05-25

    First, pad the header with 10k of space, e.g., --hdr_pad=10000. Other than that I have no more suggestions. You are right, 20 min is too long. It's mysterious why it takes so long.

     
  • Joana de Medeiros

    thanks
    I suspect the size and res of the grid,
    I did South America with the same 5km res and was very quick, this is full Europe, 5km res...I can only think of that, size of files ....
    Thank you, at least I have one done, will leave EU running during the weekend !

    Many thanks
    J

     

Log in to post a comment.