Menu
▾
▴
Re: [Opengrads-devel] 4.1.1 vs. 4.1.2
|
From: Arlindo da S. <da...@al...> - 2011-10-11 16:35:21
|
On Mon, Oct 10, 2011 at 7:47 AM, Huddleston, John < Hud...@ci...> wrote: > Hi Arlindo > > The AMSR data was stored in HDF4 format; and, the HDF4r3 version works > best. I sent a report to the hdfgroup and they have started a help > ticket. It took 1.2 seconds to process the AMSR HDF4 file with HDF4r3 and > it took nearly two minutes using later HDF4 libraries. > > The GrADS-2.0.0 was faster than GrADS-2.0a9 and a8 on a Linux Ubuntuserver. It was about 2Xfaster than Windows 7. The Ubuntuserver has single processor 2.8GHzwith 3GBof RAM. The Windows 7 has two 3.6GHzprocessors with 8 GB RAM. You'd expect the Windows version to be faster. I'm > working on a MinGW native windows build. All the HDF4, HDF5, and NetCDFapplications under MingWrun 10xfaster than equivalent Ubuntuversions (which is what you would expect from the hardware difference.) > > What are your plans for X11? It is possible to have stable X11 builds on mingwin as demonstrated by the Xming guy. However, I was not able to convince him to share his libraries with me. Having built X11 on Windows in the past (very early on I maintained X11 cygwin builds), I don't look forward to the task... Time would be better spent replacing gxX.c with a native gxCairo.c. (This is different from what Jennifer is doing for 2.1 as she is using the X11 surface with cairo.) One should be able to erase any trace of X from the code, but it is not as easy because X11 is not restricted to gxX.c. The problem would be more doable if one would drop the older X11 widgets and the Athena GUI features in GrADS. I'd start adapting the gxJava.c which part of GrADSVM... > e.g. See http://vista.cira.colostate.edu/nco/Grib2/ WGRIB2 executables I > build for Wesley. > > A couple of years back I also had a mini-supplibs built with mingwin that I used for wgrib2, there is a binary on sf.net: http://sourceforge.net/projects/opengrads/files/wgrib2/0.1.7.8e/ The only think I found annoying with the mingwin ports is the handling of the "\" vs "/" in directory names... But I agreee, It would be nice to convert grads to mingwin. Cheers! Arlindo > John > > ------------------------------ > *From:* mor...@gm... [mor...@gm...] On Behalf Of > Arlindo da Silva [arl...@na...] > *Sent:* Sunday, October 09, 2011 4:41 PM > *To:* Huddleston, John > *Cc:* ope...@li... > > *Subject:* Re: 4.1.1 vs. 4.1.2 > > On Fri, Oct 7, 2011 at 12:27 PM, Huddleston, John < > Hud...@ci...> wrote: > >> Jennifer**** >> >> **** >> >> While creating the AMSR CTL file I did some testing on the Ubuntu Linux >> build of GrADS and found that version 2.0.0 was faster than both 2.0a8 and >> 2.0a9 by some 20-30%.**** >> >> **** >> >> So, I did the same run on Windows and it was horribly slow. I re-compiled >> GrADS with the GNU profiler and found that the HDF5 HAPatom_object was the >> culprit.**** >> >> **** >> >> % cumulative self self total**** >> >> time seconds seconds calls ms/call ms/call name**** >> >> 99.21 85.51 85.51 HAPatom_object**** >> >> 0.15 85.64 0.13 475492 0.00 0.00 gahrow**** >> >> 0.07 85.70 0.06 475544 0.00 0.00 gree**** >> >> 0.06 85.75 0.05 HTPinquire**** >> >> 0.05 85.79 0.04 76373 0.00 0.00 gxfill**** >> >> 0.03 85.82 0.03 475559 0.00 0.00 galloc**** >> >> 0.03 85.85 0.03 DFKconvert**** >> >> **** >> >> I checked and I was using HDF5-1.8.6; so, I’m building HDF5-1.8.7 for >> Cygwin and will rebuild GrADS 2.0.0.**** >> >> ** >> > > > Please let me know that you find. I have tried my windows build of 2.0.a9 > on this large netcdf-4 file (with compression) > > > ftp://gma...@ft.../fp/das/Y2011/M10/D01/DAS.ops.asm.inst3_3d_asm_Np.GEOS572.20111001_0000.V01.nc4 > > and found that > > lats4d -i DAS.ops.asm.inst3_3d_asm_Np.GEOS572.20111001_0000.V01.nc4 -format > stats > > ran slightly faster on Windows 7 than Mac OS X 10.6 (48.6s vs 51.6s) on the > same hardware (different boots). This was based on HDF5 1.8.4 and NetCDF > 4.1.1 that I used for the 2.0.a9 builds. > > I will now build 2.0.0 with new libraries and rerun the benchmark. > > Arlindo > > > > > >> ** >> >> John**** >> >> **** >> >> **** >> >> **** >> >> *From:* Jennifer Adams [mailto:jm...@co...] >> *Sent:* Wednesday, October 05, 2011 3:39 PM >> *To:* Arlindo da Silva >> *Cc:* Brian Doty; Huddleston, John >> *Subject:* Re: 4.1.1 vs. 4.1.2**** >> >> **** >> >> I think netcdf-4.1.2 (with zlib-1.2.5 and hdf5-1.8.7) is what we should >> use for GrADS 2.0.0. We've been using 4.1.2 at COLA for just about a year. I >> have made new builds for darwin and CentOS and posted them to our FTP >> server, and I changed my supplibs.html page too. Arlindo, thanks for >> bringing this bug to my attention. John, I guess you have no extra work >> since your builds are already linked with 4.1.2. **** >> >> --Jennifer**** >> >> **** >> >> p.s. Here is what Dennis Heimbigner wrote about the performance bug: **** >> >> "The performance bug has been in the code since almost**** >> >> the beginning of opendap support. It is possible >> (even probable) that you will not encounter it. >> It occurs when you read a large variable and specify >> it as a constraint in the url. E.g. http://x.com/y.nc?var[0:1] >> The bug is that the code ignores the [0:1] and reads >> the whole variable. The result is correct, but unnecessary >> data is read." **** >> >> **** >> >> I don't exactly know whether that syntax he shows is different from the >> requests in the gds log that look like this:**** >> >> "...GET /gfs.dods?ps.ps[0][0:360][0:719] ..."**** >> >> Maybe they ARE different and that's why we don't encounter a performance >> change between old versions and the new snapshot. In any event, I think >> 4.1.2 is the way to go for now. **** >> >> **** >> >> **** >> >> On Oct 5, 2011, at 1:20 PM, Arlindo da Silva wrote:**** >> >> >> >> **** >> >> **** >> >> On Wed, Oct 5, 2011 at 11:51 AM, Jennifer Adams <jm...@co...> >> wrote:**** >> >> Hi, Arlindo -- **** >> >> The release notes for 4.1.2 ( >> http://www.unidata.ucar.edu/software/netcdf/release-notes-4.1.2.html ) >> mention a lot of items that are relevant for GrADS, one especially that was >> pointed out by me (speedup in opening of large files). I will definitely not >> be going to back to 4.1.1. We've been using 4.1.2 at COLA for a long time. >> **** >> >> **** >> >> **** >> >> This is good to know. If you have been using 4.1.2 for sometime than this >> is proof of stability. I'll follow your lead.**** >> >> **** >> >> Here are the 7 tests I ran with the two builds -- I could not detect >> any difference in the time it took for these tasks to run when comparing >> 4.1.2 and 4.2.snapshot.**** >> >> **** >> >> I share your concern about 4.2. First, it is very new, and second, a large >> portion of the opendap codebase was apparently rewritten. Let me know when >> you make the final decision about 4.1.2 and I'll update my supplibs >> accordingly.**** >> >> **** >> >> Cheers!**** >> >> **** >> >> Arlindo**** >> >> **** >> >> **** >> >> **** >> >> **** >> >> **** >> >> **** >> >> --Jennifer**** >> >> **** >> >> 'reinit'**** >> >> '!date'**** >> >> **** >> >> if (0)**** >> >> * The original bug**** >> >> 'sdfopen >> http://opendap.gsfc.nasa.gov:9090/dods/GEOS-5/fp/0.25_deg/assim/inst1_2d_hwl_Nx' >> **** >> >> 'set x 1 1152'**** >> >> 'd slp'**** >> >> say result**** >> >> endif**** >> >> **** >> >> if (0)**** >> >> * NOMADS GDS**** >> >> 'sdfopen http://nomads.ncep.noaa.gov:9090/dods/gfs/gfs20111003/gfs_00z'* >> *** >> >> 'set x 1 360'**** >> >> 'set y 1 181'**** >> >> 'set z 1 26'**** >> >> 'set t 1 65'**** >> >> 'define foo = hgtprs'**** >> >> endif**** >> >> **** >> >> if (0)**** >> >> * COLA GDS, Ensembles**** >> >> 'sdfopen http://monsoondata.org:9090/dods/gfsens/gfsens.2011100400'**** >> >> 'set x 1 360'**** >> >> 'set y 1 181'**** >> >> 'set z 1 7'**** >> >> 'set e 1 22'**** >> >> 'define foo = z'**** >> >> endif**** >> >> **** >> >> if (0)**** >> >> * Netcdf4 high res data behind GDS**** >> >> 'sdfopen http://monsoondata.org:9090/dods/nicam/sst'**** >> >> 'set x 1 5120'**** >> >> 'set y 1 2556'**** >> >> 'd ave(sst,t=1,t=8)'**** >> >> endif**** >> >> **** >> >> if (0)**** >> >> * Netcdf4 high res data on local disk**** >> >> 'open /data/hdf5/grids/sst.ctl'**** >> >> 'set x 1 5120'**** >> >> 'set y 1 2556'**** >> >> 'd ave(sst,t=1,t=8)'**** >> >> endif**** >> >> **** >> >> if (0)**** >> >> * Local file, PDEF'd**** >> >> 'open /data/netcdf/gswp/Albedo_mean_csu.ctl'**** >> >> 'set x 1 360'**** >> >> 'set y 1 150'**** >> >> 'set z 1'**** >> >> 'set t 1 12'**** >> >> 'define foo = albedo'**** >> >> endif**** >> >> **** >> >> if (1)**** >> >> * Classic netcdf**** >> >> 'sdfopen /data/netcdf/rean/air.mon.mean.nc'**** >> >> 'set x 1 144'**** >> >> 'set y 1 73'**** >> >> 'set z 1 17'**** >> >> 'set t 1 349'**** >> >> 'define foo = air'**** >> >> endif**** >> >> **** >> >> '!date'**** >> >> **** >> >> **** >> >> On Oct 5, 2011, at 10:26 AM, Arlindo da Silva wrote:**** >> >> >> >> **** >> >> On Tue, Oct 4, 2011 at 5:06 PM, Jennifer Adams <jm...@co...> wrote: >> **** >> >> Hi, Dennis -- **** >> >> Using the same GrADS source code, I have created two builds, one linked >> with netcdf-4.1.2 and the other netcdf-4.2-snapshot2011100320. Both use >> hdf5-1.8.7. The bug that showed up with 4.1.3 does not reproduce with either >> build. Furthermore, I am unable to detect any performance difference with >> these two builds, having tested some bulky I/O requests via OPeNDAP from >> several different GDS servers, and also from local classic netcdf files and >> very large compressed netcdf4 files. I am inclined to use 4.1.2 for my GrADS >> release. Can you give me an example for how to test the performance >> enhancements in the 4.2 snapshot? **** >> >> **** >> >> I have a similar experience. Comparing builds with older 4.1.1 and most >> recent 4.2 snapshot (with subsetting) gives essentially the same timing (if >> anything, 4.1.1 is a tiny bit faster: 49.69 vs 45.63, but this difference >> could be "natural variability"). **** >> >> **** >> >> Jennifer: BTW, here is my grads benchmark:**** >> >> **** >> >> lats4d -i >> http://opendap.gsfc.nasa.gov:9090/dods/GEOS-5/fp/0.25_deg/assim/inst1_2d_hwl_Nx-format stats -lat 0 30 -ntimes 10 >> **** >> >> **** >> >> Since 4.1.1 has been working for us very reliably for over a year, I am >> very tempted to stick with it in name of stability. Are there any major bug >> fixes going from 4.1.1 to 4.1.2?**** >> >> **** >> >> Arlindo**** >> >> **** >> >> **** >> >> **** >> >> **** >> >> **** >> >> **** >> >> --**** >> >> Jennifer M. Adams**** >> >> IGES/COLA**** >> >> 4041 Powder Mill Road, Suite 302**** >> >> Calverton, MD 20705**** >> >> jm...@co...**** >> >> **** >> >> >> >> **** >> >> **** >> >> **** >> >> **** >> >> --**** >> >> Jennifer M. Adams**** >> >> IGES/COLA**** >> >> 4041 Powder Mill Road, Suite 302**** >> >> Calverton, MD 20705**** >> >> jm...@co...**** >> >> **** >> >> >> >> **** >> >> **** >> > > > > ------------------------------------------------------------------------------ > All the data continuously generated in your IT infrastructure contains a > definitive record of customers, application performance, security > threats, fraudulent activity and more. Splunk takes this data and makes > sense of it. Business sense. IT sense. Common sense. > http://p.sf.net/sfu/splunk-d2d-oct > _______________________________________________ > Opengrads-devel mailing list > Ope...@li... > https://lists.sourceforge.net/lists/listinfo/opengrads-devel > > -- Arlindo da Silva da...@al... |
