Menu
▾
▴
Re: [Opengrads-devel] 4.1.1 vs. 4.1.2
|
From: Arlindo da S. <arl...@na...> - 2011-10-09 22:41:56
|
On Fri, Oct 7, 2011 at 12:27 PM, Huddleston, John <
Hud...@ci...> wrote:
> Jennifer****
>
> ** **
>
> While creating the AMSR CTL file I did some testing on the Ubuntu Linux
> build of GrADS and found that version 2.0.0 was faster than both 2.0a8 and
> 2.0a9 by some 20-30%.****
>
> ** **
>
> So, I did the same run on Windows and it was horribly slow. I re-compiled
> GrADS with the GNU profiler and found that the HDF5 HAPatom_object was the
> culprit.****
>
> ** **
>
> % cumulative self self total****
>
> time seconds seconds calls ms/call ms/call name****
>
> 99.21 85.51 85.51 HAPatom_object****
>
> 0.15 85.64 0.13 475492 0.00 0.00 gahrow****
>
> 0.07 85.70 0.06 475544 0.00 0.00 gree****
>
> 0.06 85.75 0.05 HTPinquire****
>
> 0.05 85.79 0.04 76373 0.00 0.00 gxfill****
>
> 0.03 85.82 0.03 475559 0.00 0.00 galloc****
>
> 0.03 85.85 0.03 DFKconvert****
>
> ** **
>
> I checked and I was using HDF5-1.8.6; so, I’m building HDF5-1.8.7 for
> Cygwin and will rebuild GrADS 2.0.0.****
>
> **
>
Please let me know that you find. I have tried my windows build of 2.0.a9 on
this large netcdf-4 file (with compression)
ftp://gmao_ops@ftp.nccs.nasa.gov/fp/das/Y2011/M10/D01/DAS.ops.asm.inst3_3d_asm_Np.GEOS572.20111001_0000.V01.nc4
and found that
lats4d -i DAS.ops.asm.inst3_3d_asm_Np.GEOS572.20111001_0000.V01.nc4 -format
stats
ran slightly faster on Windows 7 than Mac OS X 10.6 (48.6s vs 51.6s) on the
same hardware (different boots). This was based on HDF5 1.8.4 and NetCDF
4.1.1 that I used for the 2.0.a9 builds.
I will now build 2.0.0 with new libraries and rerun the benchmark.
Arlindo
> **
>
> John****
>
> ** **
>
> ** **
>
> ** **
>
> *From:* Jennifer Adams [mailto:jm...@co...]
> *Sent:* Wednesday, October 05, 2011 3:39 PM
> *To:* Arlindo da Silva
> *Cc:* Brian Doty; Huddleston, John
> *Subject:* Re: 4.1.1 vs. 4.1.2****
>
> ** **
>
> I think netcdf-4.1.2 (with zlib-1.2.5 and hdf5-1.8.7) is what we should use
> for GrADS 2.0.0. We've been using 4.1.2 at COLA for just about a year. I
> have made new builds for darwin and CentOS and posted them to our FTP
> server, and I changed my supplibs.html page too. Arlindo, thanks for
> bringing this bug to my attention. John, I guess you have no extra work
> since your builds are already linked with 4.1.2. ****
>
> --Jennifer****
>
> ** **
>
> p.s. Here is what Dennis Heimbigner wrote about the performance bug: ****
>
> "The performance bug has been in the code since almost****
>
> the beginning of opendap support. It is possible
> (even probable) that you will not encounter it.
> It occurs when you read a large variable and specify
> it as a constraint in the url. E.g. http://x.com/y.nc?var[0:1]
> The bug is that the code ignores the [0:1] and reads
> the whole variable. The result is correct, but unnecessary
> data is read." ****
>
> ** **
>
> I don't exactly know whether that syntax he shows is different from the
> requests in the gds log that look like this:****
>
> "...GET /gfs.dods?ps.ps[0][0:360][0:719] ..."****
>
> Maybe they ARE different and that's why we don't encounter a performance
> change between old versions and the new snapshot. In any event, I think
> 4.1.2 is the way to go for now. ****
>
> ** **
>
> ** **
>
> On Oct 5, 2011, at 1:20 PM, Arlindo da Silva wrote:****
>
>
>
> ****
>
> ** **
>
> On Wed, Oct 5, 2011 at 11:51 AM, Jennifer Adams <jm...@co...> wrote:
> ****
>
> Hi, Arlindo -- ****
>
> The release notes for 4.1.2 (
> http://www.unidata.ucar.edu/software/netcdf/release-notes-4.1.2.html )
> mention a lot of items that are relevant for GrADS, one especially that was
> pointed out by me (speedup in opening of large files). I will definitely not
> be going to back to 4.1.1. We've been using 4.1.2 at COLA for a long time.
> ****
>
> ** **
>
> ** **
>
> This is good to know. If you have been using 4.1.2 for sometime than this
> is proof of stability. I'll follow your lead.****
>
> ****
>
> Here are the 7 tests I ran with the two builds -- I could not detect any
> difference in the time it took for these tasks to run when comparing 4.1.2
> and 4.2.snapshot.****
>
> ** **
>
> I share your concern about 4.2. First, it is very new, and second, a large
> portion of the opendap codebase was apparently rewritten. Let me know when
> you make the final decision about 4.1.2 and I'll update my supplibs
> accordingly.****
>
> ** **
>
> Cheers!****
>
> ** **
>
> Arlindo****
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
> ****
>
> ****
>
> --Jennifer****
>
> ** **
>
> 'reinit'****
>
> '!date'****
>
> ** **
>
> if (0)****
>
> * The original bug****
>
> 'sdfopen
> http://opendap.gsfc.nasa.gov:9090/dods/GEOS-5/fp/0.25_deg/assim/inst1_2d_hwl_Nx'
> ****
>
> 'set x 1 1152'****
>
> 'd slp'****
>
> say result****
>
> endif****
>
> ** **
>
> if (0)****
>
> * NOMADS GDS****
>
> 'sdfopen http://nomads.ncep.noaa.gov:9090/dods/gfs/gfs20111003/gfs_00z'**
> **
>
> 'set x 1 360'****
>
> 'set y 1 181'****
>
> 'set z 1 26'****
>
> 'set t 1 65'****
>
> 'define foo = hgtprs'****
>
> endif****
>
> ** **
>
> if (0)****
>
> * COLA GDS, Ensembles****
>
> 'sdfopen http://monsoondata.org:9090/dods/gfsens/gfsens.2011100400'****
>
> 'set x 1 360'****
>
> 'set y 1 181'****
>
> 'set z 1 7'****
>
> 'set e 1 22'****
>
> 'define foo = z'****
>
> endif****
>
> ** **
>
> if (0)****
>
> * Netcdf4 high res data behind GDS****
>
> 'sdfopen http://monsoondata.org:9090/dods/nicam/sst'****
>
> 'set x 1 5120'****
>
> 'set y 1 2556'****
>
> 'd ave(sst,t=1,t=8)'****
>
> endif****
>
> ** **
>
> if (0)****
>
> * Netcdf4 high res data on local disk****
>
> 'open /data/hdf5/grids/sst.ctl'****
>
> 'set x 1 5120'****
>
> 'set y 1 2556'****
>
> 'd ave(sst,t=1,t=8)'****
>
> endif****
>
> ** **
>
> if (0)****
>
> * Local file, PDEF'd****
>
> 'open /data/netcdf/gswp/Albedo_mean_csu.ctl'****
>
> 'set x 1 360'****
>
> 'set y 1 150'****
>
> 'set z 1'****
>
> 'set t 1 12'****
>
> 'define foo = albedo'****
>
> endif****
>
> ** **
>
> if (1)****
>
> * Classic netcdf****
>
> 'sdfopen /data/netcdf/rean/air.mon.mean.nc'****
>
> 'set x 1 144'****
>
> 'set y 1 73'****
>
> 'set z 1 17'****
>
> 'set t 1 349'****
>
> 'define foo = air'****
>
> endif****
>
> ** **
>
> '!date'****
>
> ** **
>
> ** **
>
> On Oct 5, 2011, at 10:26 AM, Arlindo da Silva wrote:****
>
>
>
> ****
>
> On Tue, Oct 4, 2011 at 5:06 PM, Jennifer Adams <jm...@co...> wrote:*
> ***
>
> Hi, Dennis -- ****
>
> Using the same GrADS source code, I have created two builds, one linked
> with netcdf-4.1.2 and the other netcdf-4.2-snapshot2011100320. Both use
> hdf5-1.8.7. The bug that showed up with 4.1.3 does not reproduce with either
> build. Furthermore, I am unable to detect any performance difference with
> these two builds, having tested some bulky I/O requests via OPeNDAP from
> several different GDS servers, and also from local classic netcdf files and
> very large compressed netcdf4 files. I am inclined to use 4.1.2 for my GrADS
> release. Can you give me an example for how to test the performance
> enhancements in the 4.2 snapshot? ****
>
> ** **
>
> I have a similar experience. Comparing builds with older 4.1.1 and most
> recent 4.2 snapshot (with subsetting) gives essentially the same timing (if
> anything, 4.1.1 is a tiny bit faster: 49.69 vs 45.63, but this difference
> could be "natural variability"). ****
>
> ** **
>
> Jennifer: BTW, here is my grads benchmark:****
>
> ** **
>
> lats4d -i
> http://opendap.gsfc.nasa.gov:9090/dods/GEOS-5/fp/0.25_deg/assim/inst1_2d_hwl_Nx-format stats -lat 0 30 -ntimes 10
> ****
>
> ** **
>
> Since 4.1.1 has been working for us very reliably for over a year, I am
> very tempted to stick with it in name of stability. Are there any major bug
> fixes going from 4.1.1 to 4.1.2?****
>
> ** **
>
> Arlindo****
>
> ** **
>
> ****
>
> ** **
>
> ****
>
> ** **
>
> ** **
>
> --****
>
> Jennifer M. Adams****
>
> IGES/COLA****
>
> 4041 Powder Mill Road, Suite 302****
>
> Calverton, MD 20705****
>
> jm...@co...****
>
> ** **
>
>
>
> ****
>
> ** **
>
> ** **
>
> ** **
>
> --****
>
> Jennifer M. Adams****
>
> IGES/COLA****
>
> 4041 Powder Mill Road, Suite 302****
>
> Calverton, MD 20705****
>
> jm...@co...****
>
> ** **
>
>
>
> ****
>
> ** **
>
|
