Menu
▾
▴
Re: [Opengrads-devel] Dat File re-reads
|
From: Mike F. <mfi...@gm...> - 2011-10-13 15:00:44
|
thanks Brian and all for the good discussion regarding performance. i too generate order 10,000 images / day and would hopefully benefit from the cache. i'll build and see what happens. best /r mike On 10/13/11 2:56 PM, Brian Doty wrote: > Ok, not too hard, I've added some basic caching of the map file to gxwmap.c. The changes > are all isolated to gxwmap.c for now. I kinda did this quick so I haven't done a whole > lot of testing on it. The cache lasts for the entire grads session. I don't think this > hurts performance any for those cases when grads is just being run to do one plot. This > mod is applied to the gxwmap.c from the 2.0.0 code base -- but we will not include it in > 2.0; it will be included in 2.1. Right now the CACHEMAX is set to 2MB which will cache > lowres, mres, and hires. If you have your own map files that are larger, you may want to > increase the CACHEMAX value. This will have no affect on shapefile usage. > > Gary, I'm not really sure this will help your situation; it will depend on your system > environment. Arlindo's suggestion to set up some system-wide solution like a ramdisk may > be more along the lines of what you need... Brian > > > > > On Oct 12, 2011, at 8:38 PM, Love, Mr. Gary, Contractor, Code 7542 wrote: > >> Arlindo, >> >> We have some solid state disks that function as local disks because the global file >> systems running gpfs are too slow. So our challenge is how to accomplish how to do the >> "shared memory" which we also thought of using. >> >> Do you know if anyone else has this problem and who could benefit from the solution? >> >> Thanks for your input. I'll ask for more when we get into the project. >> >> Gary >> >> From: Arlindo da Silva [mailto:da...@al...] >> Sent: Wednesday, October 12, 2011 4:41 PM >> To: ope...@li... >> Cc: Brian Doty; Frost, Mr. Michael >> Subject: Re: [Opengrads-devel] Dat File re-reads >> >> On Wed, Oct 12, 2011 at 3:16 PM, Love, Mr. Gary, Contractor, Code 7542 >> <gar...@nr...> wrote: >> Hi Brian, Jennifer, >> >> We use GrADS extensively in a production mode and have found when profiling GrADS runs >> that 2/3 rds of the data reads are the coastlines and font files. This may not sound >> like much, but at FNMOC when running 100's of charts for each of 32 multi-grid projects, >> the coastline and font reads amount to terabytes for each 12 hour set of runs. >> >> The re-reads occur for each and every display of a variable from the same file after it >> is opened. We are going to try to store these files in memory after the first read. >> >> My first question is whether anyone else has looked at this and has a solution? My next >> questions, do you have any ideas or guidance to help us do this? >> >> I know that GrADS was designed to operate well on small platforms with little memory, so >> our goal is to implement a memory caching capability that would be controlled by a 'set' >> command and would be moderated by the amount of memory available. >> >> As Brian mentions, the map and font database are relatively small. So, it would be very >> feasible to setup a ramdisk to hold these files which would probably give you a nice >> performance boost with virtually no programming involved. Depending on your system >> architecture, this ramdisk could effectively function as "shared memory" used by multiple >> cores. (Because grads is not thread safe, having an explicit memory buffer to hold this >> would require that each grads process duplicate the memory necessary to hold the >> maps/font databases.) One hardware solution is to place grads and its data on a solid >> state disk, these are quite affordable these days. >> >> Just my 2 lazy cents. >> >> Arlindo >> >> -- >> Arlindo da Silva >> da...@al... > > > > ------------------------------------------------------------------------------ > All the data continuously generated in your IT infrastructure contains a > definitive record of customers, application performance, security > threats, fraudulent activity and more. Splunk takes this data and makes > sense of it. Business sense. IT sense. Common sense. > http://p.sf.net/sfu/splunk-d2d-oct > > > _______________________________________________ > Opengrads-devel mailing list > Ope...@li... > https://lists.sourceforge.net/lists/listinfo/opengrads-devel |
