Charlie and NCO support - we need some help or advice on this problem:
On Yellowstone,we are seeing slow timings for NCO 4.4.2 ncrcat, as compared
to NCO 4.3.4. See the ncrcat command below, along with the timings. The
15 input files wrfout_ data are too large to attach: each 0.7 GB in size.
The files are in Yellowstone directory /glade/scratch/valent/amy. Charlie
you have an account there. Let us know if there is anything else you need,
account-wise etc..
Thank you. --Dick Valent
Here is the command we timed: ncrcat -d south_north,55 -d south_north_stag,55 -d west_east,68 * -d west_east_stag,68 wrfout_d03_2013-03 foo.nc http://foo.nc/
Yellowstone single thread
4.3.4 time -- 0.294u 3.718s 0:11.97 33.4% 0+0k 0+0io 0pf+0w
4.4.2 time -- 881.394u 18624.248s 5:26:00.11 99.7% 0+0k 0+0io 0pf+0w
Thank you for using SF forums. As I mentioned offline, the latest NCO, 4.4.4,
seems to have reasonable timings, as good as 4.3.4 according to your results:
I believe this is the same command you executed, although SF messes with the formatting a bit.
I'm unsure why 4.4.2 has this problem which clearly makes ncrcat too slow for some use cases. I recommend installing 4.4.4. It's available in ~zender/[bin,lib]/LINUXAMD64 if you want to test it.
As mentioned, access to some HPSS space would be appreciated for other testing.
Best,
c
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Charlie; you are welcome, re: using the SF forums. I am glad to know how to do that.
I will tell our consulting group that NCO 4.4.4 has reasonable timings, and we'll see about installing it.
I checked your Yellowstone accounts; you appear to have access to HPSS, and enough units in your allocations to use it.
You should be able to logon to Yellowstone and execute command "hsi -d4 ls /home/zender" to see the contents of your home archival system. We provide an introduction to our HPSS system here http://www2.cisl.ucar.edu/docs/hpss.
When you've tried to access it, let me know what happens. If it fails, send me the output of the above hsi command; it will contain information that will help us understand the problem.
Again, thank you. --Dick Valent
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Dick,
Thanks for the info about HPSS.
It turns out that CISL recently deleted my HPSS files which were charged to an old project of mine. I didn't know I had a new project number to which I could have changed their ownership. Live'n'learn.
c
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Charlie and NCO support - we need some help or advice on this problem:
On Yellowstone,we are seeing slow timings for NCO 4.4.2 ncrcat, as compared
to NCO 4.3.4. See the ncrcat command below, along with the timings. The
15 input files wrfout_ data are too large to attach: each 0.7 GB in size.
The files are in Yellowstone directory /glade/scratch/valent/amy. Charlie
you have an account there. Let us know if there is anything else you need,
account-wise etc..
Thank you. --Dick Valent
Here is the command we timed:
ncrcat -d south_north,55 -d south_north_stag,55 -d west_east,68 *
-d west_east_stag,68 wrfout_d03_2013-03 foo.nc
http://foo.nc/
Yellowstone single thread
4.3.4 time -- 0.294u 3.718s 0:11.97 33.4% 0+0k 0+0io 0pf+0w
4.4.2 time -- 881.394u 18624.248s 5:26:00.11 99.7% 0+0k 0+0io 0pf+0w
Yellowstone 15 threads
4.3.4 time -- 0.303u 3.838s 0:12.31 33.5% 0+0k 0+0io 0pf+0
4.4.2 time -- 920.082u 18498.078s 5:25:19.47 99.4% 0+0k 0+0io 0pf+0w
Hi Dick,
Thank you for using SF forums. As I mentioned offline, the latest NCO, 4.4.4,
seems to have reasonable timings, as good as 4.3.4 according to your results:
zender@yslogin1:~$ time ncrcat -O -d south_north,55 -d south_north_stag,55 -d west_east,68 -d west_east_stag,68 /glade/scratch/valent/amy/wrfout_d03_2013-03* ~/foo.nc
real 0m9.702s
user 0m0.604s
sys 0m2.163s
I believe this is the same command you executed, although SF messes with the formatting a bit.
I'm unsure why 4.4.2 has this problem which clearly makes ncrcat too slow for some use cases. I recommend installing 4.4.4. It's available in ~zender/[bin,lib]/LINUXAMD64 if you want to test it.
As mentioned, access to some HPSS space would be appreciated for other testing.
Best,
c
Hi Charlie; you are welcome, re: using the SF forums. I am glad to know how to do that.
I will tell our consulting group that NCO 4.4.4 has reasonable timings, and we'll see about installing it.
I checked your Yellowstone accounts; you appear to have access to HPSS, and enough units in your allocations to use it.
You should be able to logon to Yellowstone and execute command "hsi -d4 ls /home/zender" to see the contents of your home archival system. We provide an introduction to our HPSS system here http://www2.cisl.ucar.edu/docs/hpss.
When you've tried to access it, let me know what happens. If it fails, send me the output of the above hsi command; it will contain information that will help us understand the problem.
Again, thank you. --Dick Valent
Hi Dick,
Thanks for the info about HPSS.
It turns out that CISL recently deleted my HPSS files which were charged to an old project of mine. I didn't know I had a new project number to which I could have changed their ownership. Live'n'learn.
c