curl, rate limiting ncks/OpenDAP retrievals

  • bjell

    bjell - 2012-12-05

    To avoid soaking up all available bandwidth when retrieving large data sets, we would like to be able to invoke ncks with an option to limit rate. As curl libraries are used for this (I think), I suspect that a mechanism to pass options to the curl call is all that would be needed, so the curl option "-limit-rate " could the utilized.

    Does this functionality currently exist within nco?

  • Charlie Zender

    Charlie Zender - 2012-12-05

    NCO can access remote files in about five different ways

    one of which is Opendap which, I think you're right, uses libcurl.a.
    You are certain you are accessing via Opendap?
    If so, I am happy to implement any rate-throttling options but someone
    will need to tell me what these are because I am clueless.  
    FYI, NCO accesses DAP through the netCDF client library and there is
    no way to pass bandwidth requests through the netCDF API.
    Someone will have to dig a bit deeper to learn how to request DAP
    to slow-down/speed-up.


  • bjell

    bjell - 2012-12-05

    Thanks for the response, cz.

    Yes, OpenDAP.
    Example command to retrieve global ocean surface currents, 6 hour forecast :

    ncks -v lon,lat,u_velocity,v_velocity -C -6 -d time,2,2,1 -l 2012120400 -p rtofs_glo_2ds_forecast_3hrly_prog -o

    The curl limit-rate portion of the curl man page:

    --limit-rate <speed>
        Specify the maximum transfer rate you want curl to use. This feature is useful if you have a limited pipe and you'd like your transfer not to use your entire bandwidth.
        The given speed is measured in bytes/second, unless a suffix is appended. Appending 'k' or 'K' will count the number as kilobytes, 'm' or M' makes it megabytes, while 'g' or 'G' makes it gigabytes. Examples: 200K, 3m and 1G.
        The given rate is the average speed counted during the entire transfer. It means that curl might use higher transfer speeds in short bursts, but over time it uses no more than the given rate.
        If you also use the -Y/--speed-limit option, that option will take precedence and might cripple the rate-limiting slightly, to help keeping the speed-limit logic working.
        If this option is used several times, the last one will be used.

    But, if passing this one along to curl is possible, it might serve everyone well if any curl option could potentially be passed through. Maybe, for example:

    ncks [b]--curl-opt "limit-rate 100k"[/b] -v lon,lat,u_velocity,v_velocity -C -6 -d time,2,2,1 -l 2012120400 -p rtofs_glo_2ds_forecast_3hrly_prog -o

    That way, other options and/or multiple options could be employed.

    As to DAP slow-down/speed-up, I haven't the foggiest idea how…

    And, if there is no way of throttling bandwidth usage via NCO/DAP/netCDF/etc. well, then we'll just have to look into tc traffic shaping downloads on the port. (Not as simple as passing an option for a single instance of ncks, but probably doable…I think.)

  • bjell

    bjell - 2012-12-05

    Argh, the example hypothetical command would have to be:

    ncks --curl-opt "--limit-rate 100k" -v lon,lat,u_velocity,v_velocity -C -6 -d time,2,2,1 -l 2012120400 -p rtofs_glo_2ds_forecast_3hrly_prog -o
  • Charlie Zender

    Charlie Zender - 2012-12-05

    I understand what you want and implementing the argument passing to NCO is trivial. What I need someone to explain to me, is how to pass that info through to the DAP server. DAP supports things called "constraint expressions" that may be useful, but I don't know enough about it. Please contact a DAP expert (Unidata? Jim Gallagher?) and keep me posted if you learn a viable method that NCO can implement. Curl is hidden pretty deep in the NCO software stack and I'm just unsure there's anyway to directly access those very nice features you found in the man page.

  • bjell

    bjell - 2012-12-05

    I'm not sure they have to be passed to the DAP server… (In my mind, I am figuring on the classic server/client roles, here.)
    The usual rate limiting options in curl, wget, scp, etc. are purely client side throttling.

    I totally do not grasp the issue… (possible!)

  • Dave Allured

    Dave Allured - 2013-01-02

    bjell, as a large downloader, I agree that your idea for rate limiting would be a nice feature.

    The NCO software stack includes the libcurl C interface, *not* the curl command line utility.  Options such as -limit rate are parsed only at the curl command line level.  Therefore, implementing your vision of -curl-opt would be a lot of work.

    The NetCDF implementation of OpenDAP already includes an optional config file ".dodsrc" that can be used to adjust certain HTTP parameters.  I think this would be the easiest and most appropriate place to add rate limiting.  This way, you would not need to add anything to either NCO or the NetCDF API.  Also, you would get the benefit of rate limiting for all NetCDF applications, not just NCO.

    What is needed is to add .dodsrc support for these two libcurl options: CURLOPT_MAX_SEND_SPEED_LARGE and CURLOPT_MAX_RECV_SPEED_LARGE.  Yes, this means you can independently adjust the upload and download speeds.  Reference page for curl_easy_setopt:

    If anyone has the time and interest to write a patch, the central code supporting .dodsrc is oc/ocrc.c in the NetCDF source distribution.  HTH.


  • Charlie Zender

    Charlie Zender - 2013-01-03

    These suggestions are novel to me and I hope they prove to satisfy the poster's needs.
    Thanks for taking the time to explain and give the specifics about which options to set and where to look for more information.


Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

No, thanks