Slow ncrcat in NCO 4.4.2

  • Richard Valent

    Richard Valent - 2014-06-30

    Charlie and NCO support - we need some help or advice on this problem:

    On Yellowstone,we are seeing slow timings for NCO 4.4.2 ncrcat, as compared
    to NCO 4.3.4. See the ncrcat command below, along with the timings. The
    15 input files wrfout_ data are too large to attach: each 0.7 GB in size.

    The files are in Yellowstone directory /glade/scratch/valent/amy. Charlie
    you have an account there. Let us know if there is anything else you need,
    account-wise etc..

    Thank you. --Dick Valent

    Here is the command we timed:
    ncrcat -d south_north,55 -d south_north_stag,55 -d west_east,68 *
    -d west_east_stag,68 wrfout_d03_2013-03

    Yellowstone single thread
    4.3.4 time -- 0.294u 3.718s 0:11.97 33.4% 0+0k 0+0io 0pf+0w
    4.4.2 time -- 881.394u 18624.248s 5:26:00.11 99.7% 0+0k 0+0io 0pf+0w

    Yellowstone 15 threads
    4.3.4 time -- 0.303u 3.838s 0:12.31 33.5% 0+0k 0+0io 0pf+0
    4.4.2 time -- 920.082u 18498.078s 5:25:19.47 99.4% 0+0k 0+0io 0pf+0w

  • Charlie Zender

    Charlie Zender - 2014-06-30

    Hi Dick,

    Thank you for using SF forums. As I mentioned offline, the latest NCO, 4.4.4,
    seems to have reasonable timings, as good as 4.3.4 according to your results:

    zender@yslogin1:~$ time ncrcat -O -d south_north,55 -d south_north_stag,55 -d west_east,68 -d west_east_stag,68 /glade/scratch/valent/amy/wrfout_d03_2013-03* ~/

    real 0m9.702s
    user 0m0.604s
    sys 0m2.163s

    I believe this is the same command you executed, although SF messes with the formatting a bit.

    I'm unsure why 4.4.2 has this problem which clearly makes ncrcat too slow for some use cases. I recommend installing 4.4.4. It's available in ~zender/[bin,lib]/LINUXAMD64 if you want to test it.

    As mentioned, access to some HPSS space would be appreciated for other testing.


  • Richard Valent

    Richard Valent - 2014-07-01

    Hi Charlie; you are welcome, re: using the SF forums. I am glad to know how to do that.

    I will tell our consulting group that NCO 4.4.4 has reasonable timings, and we'll see about installing it.

    I checked your Yellowstone accounts; you appear to have access to HPSS, and enough units in your allocations to use it.

    You should be able to logon to Yellowstone and execute command "hsi -d4 ls /home/zender" to see the contents of your home archival system. We provide an introduction to our HPSS system here

    When you've tried to access it, let me know what happens. If it fails, send me the output of the above hsi command; it will contain information that will help us understand the problem.

    Again, thank you. --Dick Valent

  • Charlie Zender

    Charlie Zender - 2014-07-02

    Hi Dick,
    Thanks for the info about HPSS.
    It turns out that CISL recently deleted my HPSS files which were charged to an old project of mine. I didn't know I had a new project number to which I could have changed their ownership. Live'n'learn.


Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

No, thanks