Re: [Opengrads-devel] Dat File re-reads

[Brian D. <do...@co...>]

Hi Gary, these days the map files should be cached, at least the the  
smaller ones.  I will put this code in to gxwmap.c. This will benefit  
COLA and others.  With the new distributed parallel file systems, the  
old assumptions about system file i/o caching are no longer valid in  
many cases... Brian

On Oct 12, 2011, at 8:38 PM, Love, Mr. Gary, Contractor, Code 7542  
wrote:

> Arlindo,
>
> We have some solid state disks that function as local disks because  
> the global file systems running gpfs are too slow.  So our challenge  
> is how to accomplish how to do the “shared memory” which we also  
> thought of using.
>
> Do you know if anyone else has this problem and who could benefit  
> from the solution?
>
> Thanks for your input.  I’ll ask for more when we get into the  
> project.
>
> Gary
>
> From: Arlindo da Silva [mailto:da...@al...]
> Sent: Wednesday, October 12, 2011 4:41 PM
> To: ope...@li...
> Cc: Brian Doty; Frost, Mr. Michael
> Subject: Re: [Opengrads-devel] Dat File re-reads
>
> On Wed, Oct 12, 2011 at 3:16 PM, Love, Mr. Gary, Contractor, Code  
> 7542 <gar...@nr...> wrote:
> Hi Brian, Jennifer,
>
> We use GrADS extensively in a production mode and have found when  
> profiling GrADS runs that 2/3 rds of the data reads are the  
> coastlines and font files.  This may not sound like much, but at  
> FNMOC when running 100's of charts for each of 32 multi-grid  
> projects, the coastline and font reads amount to terabytes for each  
> 12 hour set of runs.
>
> The re-reads occur for each and every display of a variable from the  
> same file after it is opened.  We are going to try to store these  
> files in memory after the first read.
>
> My first question is whether anyone else has looked at this and has  
> a solution?  My next questions, do you have any ideas or guidance to  
> help us do this?
>
> I know that GrADS was designed to operate well on small platforms  
> with little memory, so our goal is to implement a memory caching  
> capability that would be controlled by a 'set' command and would be  
> moderated by the amount of memory available.
>
> As Brian mentions, the map and font database are relatively small.  
> So, it would be very feasible to setup a ramdisk to hold these files  
> which would probably give you a nice performance boost with  
> virtually no programming involved. Depending on your system  
> architecture, this ramdisk could effectively function as "shared  
> memory" used by multiple cores. (Because grads is not thread safe,  
> having an explicit memory buffer to hold this would require that  
> each grads process duplicate the memory necessary to hold the maps/ 
> font databases.)  One hardware solution is to place grads and its  
> data on a solid state disk, these are quite affordable these days.
>
>   Just my 2 lazy cents.
>
>        Arlindo
>
> -- 
> Arlindo da Silva
> da...@al...