Re: [Opengrads-devel] Dat File re-reads

[Brian D. <do...@co...>]

Hi Gary, what you report is of concern and should be fixed.  You are  
correct that the old rationale for the way it works no longer applies.

The map and font files that come with grads are quite small by today's  
standards.  Even the hires data is only 1.7MB.  If you have your own  
map files they may be bigger.

Looking at the map code in gxwmap.c, the entry point is function  
gxdmap.  My approach to caching the map files would be to replace the  
calls to fopen, fclose, fread, and fseek that appear in gxdmap with  
calls to some new routines that would be put in gxwmap.c.  The new  
open routine would read the file in its entirety and put it in some  
dynamically-allocated memory buffer -- if it is smaller than some  
threshold, say.  Then the read and seek routines would just pull data  
out of the memory buffer, or do the I/O if the memory buffer wasn't  
allocated.  The open routine would also need to detect if the file  
that is being opened is already in memory.  So the memory buffer  
should be a link list of structures where each structure contains the  
file name and the pointer to the data cache.  It is probably  
sufficient to just test on the full path name; it seems unlikely  
people would point to different locations of the same file within one  
grads session.  There probably needs to also be a routine to free up  
all the memory and reset; perhaps on "reinit" this should be done?

For the fonts -- don't these get read in once?  Well it's been 20  
years so I'll take a look... in gxchpl.c the string plotting all goes  
through the gxchpl entry point.  That calls an internal routine  
gxchgc, which checks some pointer and reads in the font data for that  
font if the pointer is NULL.  Looks like the font data is read once,  
the first time the font is used.   Your stats are showing a ton of  
reads to these files?  That is odd and may be worth looking into  
further... Brian


On Oct 12, 2011, at 3:16 PM, Love, Mr. Gary, Contractor, Code 7542  
wrote:

> Hi Brian, Jennifer,
>
> We use GrADS extensively in a production mode and have found when  
> profiling GrADS runs that 2/3 rds of the data reads are the  
> coastlines and font files.  This may not sound like much, but at  
> FNMOC when running 100's of charts for each of 32 multi-grid  
> projects, the coastline and font reads amount to terabytes for each  
> 12 hour set of runs.
>
> The re-reads occur for each and every display of a variable from the  
> same file after it is opened.  We are going to try to store these  
> files in memory after the first read.
>
> My first question is whether anyone else has looked at this and has  
> a solution?  My next questions, do you have any ideas or guidance to  
> help us do this?
>
> I know that GrADS was designed to operate well on small platforms  
> with little memory, so our goal is to implement a memory caching  
> capability that would be controlled by a 'set' command and would be  
> moderated by the amount of memory available.
>
> Thanks,
> Gary