Re: [Opengrads-devel] Dat File re-reads -- a small tweak

[Love, M. G. C. C. 7. <gar...@nr...>]

Thanks guys, the response has been tremendous!  Gary

From: Michael Fiorino [mailto:Mic...@no...]
Sent: Thursday, October 13, 2011 2:13 PM
To: ope...@li...; Jennifer Adams
Cc: Brian Doty; Love, Mr. Gary, Contractor, Code 7542; Frost, Mr. Michael
Subject: Re: [Opengrads-devel] Dat File re-reads -- a small tweak

hi Brian,

i've added your gxwmap.c to our opengrads/src (the only diff was your
caching code) and set

#define CACHEMAX 12000000

to handle the hires map as Gary suggested

i also added a tweak to 'set grid' to allow setting line thickness,
e.g.,

'set grid on 5 15 5'

to make the default grid a 5 thickness vice the default of 1

the mods are in grads.h, grads.c, gauser.c and gagx.c (attached).

that's the only tweaking i've done with the code the last couple of
years.  i had added 'set z 1 last' but agree that it's not really proper
to support that option since 'z' is not the same kind of coordinate as
't'

best /r mike



On 10/13/11 7:23 PM, Brian Doty wrote:
> Hi Gary, I checked the font
      behavior, and it seems to be working as

      > intended. I put a debug write statement in the routine that
      reads the

      > font file, and it indicates it only gets called once, when a


      > particular font is first used. I was testing with the COLA
      2.0.0 code

      > base. If you want to test with your version, you can put a
      debug

      > write statement at the beginning of the function gxchrd in
      the

      > gxchpl.c source file; gxchrd is the routine that actually
      reads in a

      > font file... Brian

      >

      > On Oct 13, 2011, at 2:58 PM, Love, Mr. Gary, Contractor, Code
      7542

      > wrote:

      >

      >> Hi Brian,

      >>

      >> I think this solution is exactly on point and I will test
      it for

      >> number of file reads and bytes read. This will be
      especially useful

      >> when I convert to creating a series of plots within the
      same grads

      >> sessions. I will be increasing CACHEMAX because we
      frequently use

      >> the worldmap file for the ultra-high resolution
      coastlines. Since

      >> it is 9671652 bytes, is 10MB big enough for CACHEMAX?

      >>

      >> When we profiled grads, an equal amount of bytes were
      read from the

      >> font files as were read from the coastline files. I guess
      every

      >> draw string must invoke a font read. Do you have a
      similar fix for

      >> the font problem?

      >>

      >> Thanks,

      >> Gary

      >>

      >> From: Brian Doty [mailto:do...@co...]

      >> Sent: Thursday, October 13, 2011 7:56 AM

      >> To: Love, Mr. Gary, Contractor, Code 7542

      >> Cc: ope...@li...<mailto:ope...@li...>; Frost, Mr.
     Michael

      >> Subject: Re: [Opengrads-devel] Dat File re-reads

      >>

      >> Ok, not too hard, I've added some basic caching of the
      map file to

      >> gxwmap.c. The changes are all isolated to gxwmap.c for
      now. I kinda

      >> did this quick so I haven't done a whole lot of testing
      on it. The

      >> cache lasts for the entire grads session. I don't think
      this hurts

      >> performance any for those cases when grads is just being
      run to do one

      >> plot. This mod is applied to the gxwmap.c from the 2.0.0
      code base --

      >> but we will not include it in 2.0; it will be included in
      2.1. Right

      >> now the CACHEMAX is set to 2MB which will cache lowres,
      mres, and

      >> hires. If you have your own map files that are larger,
      you may want

      >> to increase the CACHEMAX value. This will have no affect
      on shapefile

      >> usage.

      >>

      >> Gary, I'm not really sure this will help your situation;
      it will

      >> depend on your system environment. Arlindo's suggestion
      to set up

      >> some system-wide solution like a ramdisk may be more
      along the lines

      >> of what you need... Brian

      >>

      >>

      >>

      >>

      >> On Oct 12, 2011, at 8:38 PM, Love, Mr. Gary, Contractor,
      Code 7542

      >> wrote:

      >>

      >>> Arlindo,

      >>>

      >>> We have some solid state disks that function as local
      disks because

      >>> the global file systems running gpfs are too slow. So
      our challenge

      >>> is how to accomplish how to do the shared memory
      which we also

      >>> thought of using.

      >>>

      >>> Do you know if anyone else has this problem and who
      could benefit

      >>> from the solution?

      >>>

      >>> Thanks for your input. Ill ask for more when we get
      into the

      >>> project.

      >>>

      >>> Gary

      >>>

      >>> From: Arlindo da Silva [mailto:da...@al...]

      >>> Sent: Wednesday, October 12, 2011 4:41 PM

      >>> To: ope...@li...<mailto:ope...@li...>

      >>> Cc: Brian Doty; Frost, Mr. Michael

      >>> Subject: Re: [Opengrads-devel] Dat File re-reads

      >>>

      >>> On Wed, Oct 12, 2011 at 3:16 PM, Love, Mr. Gary,
      Contractor, Code

      >>> 7542 <gar...@nr...><mailto:gar...@nr...> wrote:

      >>> Hi Brian, Jennifer,

      >>>

      >>> We use GrADS extensively in a production mode and
      have found when

      >>> profiling GrADS runs that 2/3 rds of the data reads
      are the

      >>> coastlines and font files. This may not sound like
      much, but at

      >>> FNMOC when running 100's of charts for each of 32
      multi-grid

      >>> projects, the coastline and font reads amount to
      terabytes for each

      >>> 12 hour set of runs.

      >>>

      >>> The re-reads occur for each and every display of a
      variable from the

      >>> same file after it is opened. We are going to try to
      store these

      >>> files in memory after the first read.

      >>>

      >>> My first question is whether anyone else has looked
      at this and has

      >>> a solution? My next questions, do you have any ideas
      or guidance to

      >>> help us do this?

      >>>

      >>> I know that GrADS was designed to operate well on
      small platforms

      >>> with little memory, so our goal is to implement a
      memory caching

      >>> capability that would be controlled by a 'set'
      command and would be

      >>> moderated by the amount of memory available.

      >>>

      >>> As Brian mentions, the map and font database are
      relatively small.

      >>> So, it would be very feasible to setup a ramdisk to
      hold these files

      >>> which would probably give you a nice performance
      boost with

      >>> virtually no programming involved. Depending on your
      system

      >>> architecture, this ramdisk could effectively function
      as "shared

      >>> memory" used by multiple cores. (Because grads is not
      thread safe,

      >>> having an explicit memory buffer to hold this would
      require that

      >>> each grads process duplicate the memory necessary to
      hold the maps/

      >>> font databases.) One hardware solution is to place
      grads and its

      >>> data on a solid state disk, these are quite
      affordable these days.

      >>>

      >>> Just my 2 lazy cents.

      >>>

      >>> Arlindo

      >>>

      >>> --

      >>> Arlindo da Silva

      >>> da...@al...<mailto:da...@al...>

      >>

      >

      >

      >
------------------------------------------------------------------------------

      > All the data continuously generated in your IT infrastructure
      contains a

      > definitive record of customers, application performance,
      security

      > threats, fraudulent activity and more. Splunk takes this data
      and makes

      > sense of it. Business sense. IT sense. Common sense.

      > http://p.sf.net/sfu/splunk-d2d-oct

      > _______________________________________________

      > Opengrads-devel mailing list

      > Ope...@li...<mailto:Ope...@li...>

      > https://lists.sourceforge.net/lists/listinfo/opengrads-devel