Re: [Opengrads-devel] Dat File re-reads

[Love, M. G. C. C. 7. <gar...@nr...>]

Brian,

If I use the ‘reinit’ command will this force a re-read of the map files?

Happy Thanksgiving,
Gary

From: Brian Doty [mailto:do...@co...]
Sent: Thursday, October 13, 2011 7:56 AM
To: Love, Mr. Gary, Contractor, Code 7542
Cc: ope...@li...; Frost, Mr. Michael
Subject: Re: [Opengrads-devel] Dat File re-reads

Ok, not too hard, I've added some basic caching of the map file to
gxwmap.c.  The changes are all isolated to gxwmap.c for now.  I kinda
did this quick so I haven't done a whole lot of testing on it.  The
cache lasts for the entire grads session.  I don't think this hurts
performance any for those cases when grads is just being run to do one
plot.  This mod is applied to the gxwmap.c from the 2.0.0 code base --
but we will not include it in 2.0; it will be included in 2.1.  Right
now the CACHEMAX is set to 2MB which will cache lowres, mres, and
hires.  If you have your own map files that are larger, you may want
to increase the CACHEMAX value.  This will have no affect on shapefile
usage.

Gary, I'm not really sure this will help your situation; it will
depend on your system environment.  Arlindo's suggestion to set up
some system-wide solution like a ramdisk may be more along the lines
of what you need... Brian



On Oct 12, 2011, at 8:38 PM, Love, Mr. Gary, Contractor, Code 7542
wrote:

> Arlindo,
>
> We have some solid state disks that function as local disks because
> the global file systems running gpfs are too slow.  So our challenge
> is how to accomplish how to do the “shared memory” which we also
> thought of using.
>
> Do you know if anyone else has this problem and who could benefit
> from the solution?
>
> Thanks for your input.  I’ll ask for more when we get into the
> project.
>
> Gary
>
> From: Arlindo da Silva [mailto:da...@al...]
> Sent: Wednesday, October 12, 2011 4:41 PM
> To: ope...@li...<mailto:ope...@li...>
> Cc: Brian Doty; Frost, Mr. Michael
> Subject: Re: [Opengrads-devel] Dat File re-reads
>
> On Wed, Oct 12, 2011 at 3:16 PM, Love, Mr. Gary, Contractor, Code
> 7542 <gar...@nr...<mailto:gar...@nr...>> wrote:
> Hi Brian, Jennifer,
>
> We use GrADS extensively in a production mode and have found when
> profiling GrADS runs that 2/3 rds of the data reads are the
> coastlines and font files.  This may not sound like much, but at
> FNMOC when running 100's of charts for each of 32 multi-grid
> projects, the coastline and font reads amount to terabytes for each
> 12 hour set of runs.
>
> The re-reads occur for each and every display of a variable from the
> same file after it is opened.  We are going to try to store these
> files in memory after the first read.
>
> My first question is whether anyone else has looked at this and has
> a solution?  My next questions, do you have any ideas or guidance to
> help us do this?
>
> I know that GrADS was designed to operate well on small platforms
> with little memory, so our goal is to implement a memory caching
> capability that would be controlled by a 'set' command and would be
> moderated by the amount of memory available.
>
> As Brian mentions, the map and font database are relatively small.
> So, it would be very feasible to setup a ramdisk to hold these files
> which would probably give you a nice performance boost with
> virtually no programming involved. Depending on your system
> architecture, this ramdisk could effectively function as "shared
> memory" used by multiple cores. (Because grads is not thread safe,
> having an explicit memory buffer to hold this would require that
> each grads process duplicate the memory necessary to hold the maps/
> font databases.)  One hardware solution is to place grads and its
> data on a solid state disk, these are quite affordable these days.
>
>   Just my 2 lazy cents.
>
>        Arlindo
>
> --
> Arlindo da Silva
> da...@al...<mailto:da...@al...>