Re: [Opengrads-devel] Dat File re-reads

[Brian D. <do...@co...>]

Dear Gary, yes if you make CACHEMAX larger than 9671652, the larger  
map file will be cached.  I am not sure about the font files.  The  
intent of the old code is that they should only be getting read  
once.   I will take a look again at the code -- there may be a bug  
there... Brian

On Oct 13, 2011, at 2:58 PM, Love, Mr. Gary, Contractor, Code 7542  
wrote:

> Hi Brian,
>
> I think this solution is exactly on point and I will test it for  
> number of file reads and bytes read.  This will be especially useful  
> when I convert to creating a series of plots within the same grads  
> sessions.  I will be increasing CACHEMAX because we frequently use  
> the worldmap file for the ultra-high resolution coastlines.  Since  
> it is  9671652 bytes, is 10MB big enough for CACHEMAX?
>
> When we profiled grads, an equal amount of bytes were read from the  
> font files as were read from the coastline files.  I guess every  
> draw string must invoke a font read. Do you have a similar fix for  
> the font problem?
>
> Thanks,
> Gary
>
> From: Brian Doty [mailto:do...@co...]
> Sent: Thursday, October 13, 2011 7:56 AM
> To: Love, Mr. Gary, Contractor, Code 7542
> Cc: ope...@li...; Frost, Mr. Michael
> Subject: Re: [Opengrads-devel] Dat File re-reads
>
> Ok, not too hard, I've added some basic caching of the map file to
> gxwmap.c.  The changes are all isolated to gxwmap.c for now.  I kinda
> did this quick so I haven't done a whole lot of testing on it.  The
> cache lasts for the entire grads session.  I don't think this hurts
> performance any for those cases when grads is just being run to do one
> plot.  This mod is applied to the gxwmap.c from the 2.0.0 code base --
> but we will not include it in 2.0; it will be included in 2.1.  Right
> now the CACHEMAX is set to 2MB which will cache lowres, mres, and
> hires.  If you have your own map files that are larger, you may want
> to increase the CACHEMAX value.  This will have no affect on shapefile
> usage.
>
> Gary, I'm not really sure this will help your situation; it will
> depend on your system environment.  Arlindo's suggestion to set up
> some system-wide solution like a ramdisk may be more along the lines
> of what you need... Brian
>
>
>
>
> On Oct 12, 2011, at 8:38 PM, Love, Mr. Gary, Contractor, Code 7542
> wrote:
>
> > Arlindo,
> >
> > We have some solid state disks that function as local disks because
> > the global file systems running gpfs are too slow.  So our challenge
> > is how to accomplish how to do the “shared memory” which we also
> > thought of using.
> >
> > Do you know if anyone else has this problem and who could benefit
> > from the solution?
> >
> > Thanks for your input.  I’ll ask for more when we get into the
> > project.
> >
> > Gary
> >
> > From: Arlindo da Silva [mailto:da...@al...]
> > Sent: Wednesday, October 12, 2011 4:41 PM
> > To: ope...@li...
> > Cc: Brian Doty; Frost, Mr. Michael
> > Subject: Re: [Opengrads-devel] Dat File re-reads
> >
> > On Wed, Oct 12, 2011 at 3:16 PM, Love, Mr. Gary, Contractor, Code
> > 7542 <gar...@nr...> wrote:
> > Hi Brian, Jennifer,
> >
> > We use GrADS extensively in a production mode and have found when
> > profiling GrADS runs that 2/3 rds of the data reads are the
> > coastlines and font files.  This may not sound like much, but at
> > FNMOC when running 100's of charts for each of 32 multi-grid
> > projects, the coastline and font reads amount to terabytes for each
> > 12 hour set of runs.
> >
> > The re-reads occur for each and every display of a variable from the
> > same file after it is opened.  We are going to try to store these
> > files in memory after the first read.
> >
> > My first question is whether anyone else has looked at this and has
> > a solution?  My next questions, do you have any ideas or guidance to
> > help us do this?
> >
> > I know that GrADS was designed to operate well on small platforms
> > with little memory, so our goal is to implement a memory caching
> > capability that would be controlled by a 'set' command and would be
> > moderated by the amount of memory available.
> >
> > As Brian mentions, the map and font database are relatively small.
> > So, it would be very feasible to setup a ramdisk to hold these files
> > which would probably give you a nice performance boost with
> > virtually no programming involved. Depending on your system
> > architecture, this ramdisk could effectively function as "shared
> > memory" used by multiple cores. (Because grads is not thread safe,
> > having an explicit memory buffer to hold this would require that
> > each grads process duplicate the memory necessary to hold the maps/
> > font databases.)  One hardware solution is to place grads and its
> > data on a solid state disk, these are quite affordable these days.
> >
> >   Just my 2 lazy cents.
> >
> >        Arlindo
> >
> > --
> > Arlindo da Silva
> > da...@al...
>