Slow ncrcat in NCO 4.4.2

Help
2014-06-30
2014-07-02
  • Richard Valent

    Richard Valent - 2014-06-30

    Charlie and NCO support - we need some help or advice on this problem:

    On Yellowstone,we are seeing slow timings for NCO 4.4.2 ncrcat, as compared
    to NCO 4.3.4. See the ncrcat command below, along with the timings. The
    15 input files wrfout_ data are too large to attach: each 0.7 GB in size.

    The files are in Yellowstone directory /glade/scratch/valent/amy. Charlie
    you have an account there. Let us know if there is anything else you need,
    account-wise etc..

    Thank you. --Dick Valent

    Here is the command we timed:
    ncrcat -d south_north,55 -d south_north_stag,55 -d west_east,68 *
    -d west_east_stag,68 wrfout_d03_2013-03 foo.nc
    http://foo.nc/

    Yellowstone single thread
    4.3.4 time -- 0.294u 3.718s 0:11.97 33.4% 0+0k 0+0io 0pf+0w
    4.4.2 time -- 881.394u 18624.248s 5:26:00.11 99.7% 0+0k 0+0io 0pf+0w

    Yellowstone 15 threads
    4.3.4 time -- 0.303u 3.838s 0:12.31 33.5% 0+0k 0+0io 0pf+0
    4.4.2 time -- 920.082u 18498.078s 5:25:19.47 99.4% 0+0k 0+0io 0pf+0w

     
  • Charlie Zender

    Charlie Zender - 2014-06-30

    Hi Dick,

    Thank you for using SF forums. As I mentioned offline, the latest NCO, 4.4.4,
    seems to have reasonable timings, as good as 4.3.4 according to your results:

    zender@yslogin1:~$ time ncrcat -O -d south_north,55 -d south_north_stag,55 -d west_east,68 -d west_east_stag,68 /glade/scratch/valent/amy/wrfout_d03_2013-03* ~/foo.nc

    real 0m9.702s
    user 0m0.604s
    sys 0m2.163s

    I believe this is the same command you executed, although SF messes with the formatting a bit.

    I'm unsure why 4.4.2 has this problem which clearly makes ncrcat too slow for some use cases. I recommend installing 4.4.4. It's available in ~zender/[bin,lib]/LINUXAMD64 if you want to test it.

    As mentioned, access to some HPSS space would be appreciated for other testing.

    Best,
    c

     
  • Richard Valent

    Richard Valent - 2014-07-01

    Hi Charlie; you are welcome, re: using the SF forums. I am glad to know how to do that.

    I will tell our consulting group that NCO 4.4.4 has reasonable timings, and we'll see about installing it.

    I checked your Yellowstone accounts; you appear to have access to HPSS, and enough units in your allocations to use it.

    You should be able to logon to Yellowstone and execute command "hsi -d4 ls /home/zender" to see the contents of your home archival system. We provide an introduction to our HPSS system here http://www2.cisl.ucar.edu/docs/hpss.

    When you've tried to access it, let me know what happens. If it fails, send me the output of the above hsi command; it will contain information that will help us understand the problem.

    Again, thank you. --Dick Valent

     
  • Charlie Zender

    Charlie Zender - 2014-07-02

    Hi Dick,
    Thanks for the info about HPSS.
    It turns out that CISL recently deleted my HPSS files which were charged to an old project of mine. I didn't know I had a new project number to which I could have changed their ownership. Live'n'learn.
    c

     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks