Menu
▾
▴
opengrads-devel
|
From: Love, M. G. C. C. 7. <gar...@nr...> - 2011-10-12 19:16:50
|
Hi Brian, Jennifer, We use GrADS extensively in a production mode and have found when profiling GrADS runs that 2/3 rds of the data reads are the coastlines and font files. This may not sound like much, but at FNMOC when running 100's of charts for each of 32 multi-grid projects, the coastline and font reads amount to terabytes for each 12 hour set of runs. The re-reads occur for each and every display of a variable from the same file after it is opened. We are going to try to store these files in memory after the first read. My first question is whether anyone else has looked at this and has a solution? My next questions, do you have any ideas or guidance to help us do this? I know that GrADS was designed to operate well on small platforms with little memory, so our goal is to implement a memory caching capability that would be controlled by a 'set' command and would be moderated by the amount of memory available. Thanks, Gary |
|
From: Arlindo da S. <da...@al...> - 2011-10-12 23:41:10
|
On Wed, Oct 12, 2011 at 3:16 PM, Love, Mr. Gary, Contractor, Code 7542 <
gar...@nr...> wrote:
> Hi Brian, Jennifer,
>
> We use GrADS extensively in a production mode and have found when profiling
> GrADS runs that 2/3 rds of the data reads are the coastlines and font files.
> This may not sound like much, but at FNMOC when running 100's of charts for
> each of 32 multi-grid projects, the coastline and font reads amount to
> terabytes for each 12 hour set of runs.
>
> The re-reads occur for each and every display of a variable from the same
> file after it is opened. We are going to try to store these files in memory
> after the first read.
>
> My first question is whether anyone else has looked at this and has a
> solution? My next questions, do you have any ideas or guidance to help us
> do this?
>
> I know that GrADS was designed to operate well on small platforms with
> little memory, so our goal is to implement a memory caching capability that
> would be controlled by a 'set' command and would be moderated by the amount
> of memory available.
>
As Brian mentions, the map and font database are relatively small. So, it
would be very feasible to setup a ramdisk to hold these files which would
probably give you a nice performance boost with virtually no programming
involved. Depending on your system architecture, this ramdisk could
effectively function as "shared memory" used by multiple cores. (Because
grads is not thread safe, having an explicit memory buffer to hold this
would require that each grads process duplicate the memory necessary to hold
the maps/font databases.) One hardware solution is to place grads and its
data on a solid state disk, these are quite affordable these days.
Just my 2 lazy cents.
Arlindo
--
Arlindo da Silva
da...@al...
|
|
From: Love, M. G. C. C. 7. <gar...@nr...> - 2011-10-13 00:38:35
|
Arlindo,
We have some solid state disks that function as local disks because the global file systems running gpfs are too slow. So our challenge is how to accomplish how to do the "shared memory" which we also thought of using.
Do you know if anyone else has this problem and who could benefit from the solution?
Thanks for your input. I'll ask for more when we get into the project.
Gary
From: Arlindo da Silva [mailto:da...@al...]
Sent: Wednesday, October 12, 2011 4:41 PM
To: ope...@li...
Cc: Brian Doty; Frost, Mr. Michael
Subject: Re: [Opengrads-devel] Dat File re-reads
On Wed, Oct 12, 2011 at 3:16 PM, Love, Mr. Gary, Contractor, Code 7542 <gar...@nr...<mailto:gar...@nr...>> wrote:
Hi Brian, Jennifer,
We use GrADS extensively in a production mode and have found when profiling GrADS runs that 2/3 rds of the data reads are the coastlines and font files. This may not sound like much, but at FNMOC when running 100's of charts for each of 32 multi-grid projects, the coastline and font reads amount to terabytes for each 12 hour set of runs.
The re-reads occur for each and every display of a variable from the same file after it is opened. We are going to try to store these files in memory after the first read.
My first question is whether anyone else has looked at this and has a solution? My next questions, do you have any ideas or guidance to help us do this?
I know that GrADS was designed to operate well on small platforms with little memory, so our goal is to implement a memory caching capability that would be controlled by a 'set' command and would be moderated by the amount of memory available.
As Brian mentions, the map and font database are relatively small. So, it would be very feasible to setup a ramdisk to hold these files which would probably give you a nice performance boost with virtually no programming involved. Depending on your system architecture, this ramdisk could effectively function as "shared memory" used by multiple cores. (Because grads is not thread safe, having an explicit memory buffer to hold this would require that each grads process duplicate the memory necessary to hold the maps/font databases.) One hardware solution is to place grads and its data on a solid state disk, these are quite affordable these days.
Just my 2 lazy cents.
Arlindo
--
Arlindo da Silva
da...@al...<mailto:da...@al...>
|
|
From: Arlindo da S. <da...@al...> - 2011-10-13 02:36:49
|
On Wed, Oct 12, 2011 at 8:38 PM, Love, Mr. Gary, Contractor, Code 7542 < gar...@nr...> wrote: > Arlindo,**** > > ** ** > > We have some solid state disks that function as local disks because the > global file systems running gpfs are too slow. > gpfs has been a big disappointment... > So our challenge is how to accomplish how to do the “shared memory” which > we also thought of using.**** > > ** > Do you mean standard "Unix shared memory" or "openMP" type of shared memory parallel programing? I have looked into Unix Shared Memory as an approach for UDF data exchange but found it a bit too clumsy to program into. (I take it back: it is much simpler if you only need to read from it.) But, really, I'd try the ramdisk approach first because it is so simple. I am assuming you are on a linux platform, correct? > ** > > Do you know if anyone else has this problem and who could benefit from the > solution?**** > > ** > I used a similar approach recently. As you know, gxyat() is both an off-line translator or grads metafiles into png, pdf, etc, as well as a replacement for printim. I use absolutely the same code for both. In the on-line (printim replacement) I have #define fread udx_mread where udx_mread() is a function that works pretty much like fread() but instead of reading from a file it reads from the grads graphics (memory) buffer. Now, this makes wonder. Do you really want to cache the map/font database or the rendering "strokes" associated with it? For example, we could have this functionality: ga-> enable gx_cache joseph ga-> draw map ga-> disable gx_cache where "joseph" is the name of the gx cache. Next time you wanted that map drawn you could simply draw it ga-> draw gx_cache joseph and the map would be redrawn, possibly even with clipping, without reading any file. This would be trivial to implement; all you would need to do is to make a copy of the graphics buffer between enable/disable gx_cache. Arlindo > ** > > Thanks for your input. I’ll ask for more when we get into the project.*** > * > > ** ** > > Gary**** > > ** ** > > *From:* Arlindo da Silva [mailto:da...@al...] > *Sent:* Wednesday, October 12, 2011 4:41 PM > *To:* ope...@li... > *Cc:* Brian Doty; Frost, Mr. Michael > *Subject:* Re: [Opengrads-devel] Dat File re-reads**** > > ** ** > > On Wed, Oct 12, 2011 at 3:16 PM, Love, Mr. Gary, Contractor, Code 7542 < > gar...@nr...> wrote:**** > > Hi Brian, Jennifer, > > We use GrADS extensively in a production mode and have found when profiling > GrADS runs that 2/3 rds of the data reads are the coastlines and font files. > This may not sound like much, but at FNMOC when running 100's of charts for > each of 32 multi-grid projects, the coastline and font reads amount to > terabytes for each 12 hour set of runs. > > The re-reads occur for each and every display of a variable from the same > file after it is opened. We are going to try to store these files in memory > after the first read. > > My first question is whether anyone else has looked at this and has a > solution? My next questions, do you have any ideas or guidance to help us > do this? > > I know that GrADS was designed to operate well on small platforms with > little memory, so our goal is to implement a memory caching capability that > would be controlled by a 'set' command and would be moderated by the amount > of memory available.**** > > ** ** > > As Brian mentions, the map and font database are relatively small. So, it > would be very feasible to setup a ramdisk to hold these files which would > probably give you a nice performance boost with virtually no programming > involved. Depending on your system architecture, this ramdisk could > effectively function as "shared memory" used by multiple cores. (Because > grads is not thread safe, having an explicit memory buffer to hold this > would require that each grads process duplicate the memory necessary to hold > the maps/font databases.) One hardware solution is to place grads and its > data on a solid state disk, these are quite affordable these days.**** > > ** ** > > Just my 2 lazy cents.**** > > ** ** > > Arlindo**** > > **** > > -- > Arlindo da Silva > da...@al...**** > > > ------------------------------------------------------------------------------ > All the data continuously generated in your IT infrastructure contains a > definitive record of customers, application performance, security > threats, fraudulent activity and more. Splunk takes this data and makes > sense of it. Business sense. IT sense. Common sense. > http://p.sf.net/sfu/splunk-d2d-oct > _______________________________________________ > Opengrads-devel mailing list > Ope...@li... > https://lists.sourceforge.net/lists/listinfo/opengrads-devel > > -- Arlindo da Silva da...@al... |
|
From: Brian D. <do...@co...> - 2011-10-13 13:54:36
|
Hi Gary, these days the map files should be cached, at least the the smaller ones. I will put this code in to gxwmap.c. This will benefit COLA and others. With the new distributed parallel file systems, the old assumptions about system file i/o caching are no longer valid in many cases... Brian On Oct 12, 2011, at 8:38 PM, Love, Mr. Gary, Contractor, Code 7542 wrote: > Arlindo, > > We have some solid state disks that function as local disks because > the global file systems running gpfs are too slow. So our challenge > is how to accomplish how to do the “shared memory” which we also > thought of using. > > Do you know if anyone else has this problem and who could benefit > from the solution? > > Thanks for your input. I’ll ask for more when we get into the > project. > > Gary > > From: Arlindo da Silva [mailto:da...@al...] > Sent: Wednesday, October 12, 2011 4:41 PM > To: ope...@li... > Cc: Brian Doty; Frost, Mr. Michael > Subject: Re: [Opengrads-devel] Dat File re-reads > > On Wed, Oct 12, 2011 at 3:16 PM, Love, Mr. Gary, Contractor, Code > 7542 <gar...@nr...> wrote: > Hi Brian, Jennifer, > > We use GrADS extensively in a production mode and have found when > profiling GrADS runs that 2/3 rds of the data reads are the > coastlines and font files. This may not sound like much, but at > FNMOC when running 100's of charts for each of 32 multi-grid > projects, the coastline and font reads amount to terabytes for each > 12 hour set of runs. > > The re-reads occur for each and every display of a variable from the > same file after it is opened. We are going to try to store these > files in memory after the first read. > > My first question is whether anyone else has looked at this and has > a solution? My next questions, do you have any ideas or guidance to > help us do this? > > I know that GrADS was designed to operate well on small platforms > with little memory, so our goal is to implement a memory caching > capability that would be controlled by a 'set' command and would be > moderated by the amount of memory available. > > As Brian mentions, the map and font database are relatively small. > So, it would be very feasible to setup a ramdisk to hold these files > which would probably give you a nice performance boost with > virtually no programming involved. Depending on your system > architecture, this ramdisk could effectively function as "shared > memory" used by multiple cores. (Because grads is not thread safe, > having an explicit memory buffer to hold this would require that > each grads process duplicate the memory necessary to hold the maps/ > font databases.) One hardware solution is to place grads and its > data on a solid state disk, these are quite affordable these days. > > Just my 2 lazy cents. > > Arlindo > > -- > Arlindo da Silva > da...@al... |
|
From: Brian D. <do...@co...> - 2011-10-13 14:56:34
Attachments:
gxwmap.c
|
Ok, not too hard, I've added some basic caching of the map file to gxwmap.c. The changes are all isolated to gxwmap.c for now. I kinda did this quick so I haven't done a whole lot of testing on it. The cache lasts for the entire grads session. I don't think this hurts performance any for those cases when grads is just being run to do one plot. This mod is applied to the gxwmap.c from the 2.0.0 code base -- but we will not include it in 2.0; it will be included in 2.1. Right now the CACHEMAX is set to 2MB which will cache lowres, mres, and hires. If you have your own map files that are larger, you may want to increase the CACHEMAX value. This will have no affect on shapefile usage. Gary, I'm not really sure this will help your situation; it will depend on your system environment. Arlindo's suggestion to set up some system-wide solution like a ramdisk may be more along the lines of what you need... Brian |
|
From: Mike F. <mfi...@gm...> - 2011-10-13 15:00:44
|
thanks Brian and all for the good discussion regarding performance. i too generate order 10,000 images / day and would hopefully benefit from the cache. i'll build and see what happens. best /r mike On 10/13/11 2:56 PM, Brian Doty wrote: > Ok, not too hard, I've added some basic caching of the map file to gxwmap.c. The changes > are all isolated to gxwmap.c for now. I kinda did this quick so I haven't done a whole > lot of testing on it. The cache lasts for the entire grads session. I don't think this > hurts performance any for those cases when grads is just being run to do one plot. This > mod is applied to the gxwmap.c from the 2.0.0 code base -- but we will not include it in > 2.0; it will be included in 2.1. Right now the CACHEMAX is set to 2MB which will cache > lowres, mres, and hires. If you have your own map files that are larger, you may want to > increase the CACHEMAX value. This will have no affect on shapefile usage. > > Gary, I'm not really sure this will help your situation; it will depend on your system > environment. Arlindo's suggestion to set up some system-wide solution like a ramdisk may > be more along the lines of what you need... Brian > > > > > On Oct 12, 2011, at 8:38 PM, Love, Mr. Gary, Contractor, Code 7542 wrote: > >> Arlindo, >> >> We have some solid state disks that function as local disks because the global file >> systems running gpfs are too slow. So our challenge is how to accomplish how to do the >> "shared memory" which we also thought of using. >> >> Do you know if anyone else has this problem and who could benefit from the solution? >> >> Thanks for your input. I'll ask for more when we get into the project. >> >> Gary >> >> From: Arlindo da Silva [mailto:da...@al...] >> Sent: Wednesday, October 12, 2011 4:41 PM >> To: ope...@li... >> Cc: Brian Doty; Frost, Mr. Michael >> Subject: Re: [Opengrads-devel] Dat File re-reads >> >> On Wed, Oct 12, 2011 at 3:16 PM, Love, Mr. Gary, Contractor, Code 7542 >> <gar...@nr...> wrote: >> Hi Brian, Jennifer, >> >> We use GrADS extensively in a production mode and have found when profiling GrADS runs >> that 2/3 rds of the data reads are the coastlines and font files. This may not sound >> like much, but at FNMOC when running 100's of charts for each of 32 multi-grid projects, >> the coastline and font reads amount to terabytes for each 12 hour set of runs. >> >> The re-reads occur for each and every display of a variable from the same file after it >> is opened. We are going to try to store these files in memory after the first read. >> >> My first question is whether anyone else has looked at this and has a solution? My next >> questions, do you have any ideas or guidance to help us do this? >> >> I know that GrADS was designed to operate well on small platforms with little memory, so >> our goal is to implement a memory caching capability that would be controlled by a 'set' >> command and would be moderated by the amount of memory available. >> >> As Brian mentions, the map and font database are relatively small. So, it would be very >> feasible to setup a ramdisk to hold these files which would probably give you a nice >> performance boost with virtually no programming involved. Depending on your system >> architecture, this ramdisk could effectively function as "shared memory" used by multiple >> cores. (Because grads is not thread safe, having an explicit memory buffer to hold this >> would require that each grads process duplicate the memory necessary to hold the >> maps/font databases.) One hardware solution is to place grads and its data on a solid >> state disk, these are quite affordable these days. >> >> Just my 2 lazy cents. >> >> Arlindo >> >> -- >> Arlindo da Silva >> da...@al... > > > > ------------------------------------------------------------------------------ > All the data continuously generated in your IT infrastructure contains a > definitive record of customers, application performance, security > threats, fraudulent activity and more. Splunk takes this data and makes > sense of it. Business sense. IT sense. Common sense. > http://p.sf.net/sfu/splunk-d2d-oct > > > _______________________________________________ > Opengrads-devel mailing list > Ope...@li... > https://lists.sourceforge.net/lists/listinfo/opengrads-devel |
|
From: Love, M. G. C. C. 7. <gar...@nr...> - 2011-11-23 21:18:25
|
Brian, If I use the ‘reinit’ command will this force a re-read of the map files? Happy Thanksgiving, Gary From: Brian Doty [mailto:do...@co...] Sent: Thursday, October 13, 2011 7:56 AM To: Love, Mr. Gary, Contractor, Code 7542 Cc: ope...@li...; Frost, Mr. Michael Subject: Re: [Opengrads-devel] Dat File re-reads Ok, not too hard, I've added some basic caching of the map file to gxwmap.c. The changes are all isolated to gxwmap.c for now. I kinda did this quick so I haven't done a whole lot of testing on it. The cache lasts for the entire grads session. I don't think this hurts performance any for those cases when grads is just being run to do one plot. This mod is applied to the gxwmap.c from the 2.0.0 code base -- but we will not include it in 2.0; it will be included in 2.1. Right now the CACHEMAX is set to 2MB which will cache lowres, mres, and hires. If you have your own map files that are larger, you may want to increase the CACHEMAX value. This will have no affect on shapefile usage. Gary, I'm not really sure this will help your situation; it will depend on your system environment. Arlindo's suggestion to set up some system-wide solution like a ramdisk may be more along the lines of what you need... Brian On Oct 12, 2011, at 8:38 PM, Love, Mr. Gary, Contractor, Code 7542 wrote: > Arlindo, > > We have some solid state disks that function as local disks because > the global file systems running gpfs are too slow. So our challenge > is how to accomplish how to do the “shared memory” which we also > thought of using. > > Do you know if anyone else has this problem and who could benefit > from the solution? > > Thanks for your input. I’ll ask for more when we get into the > project. > > Gary > > From: Arlindo da Silva [mailto:da...@al...] > Sent: Wednesday, October 12, 2011 4:41 PM > To: ope...@li...<mailto:ope...@li...> > Cc: Brian Doty; Frost, Mr. Michael > Subject: Re: [Opengrads-devel] Dat File re-reads > > On Wed, Oct 12, 2011 at 3:16 PM, Love, Mr. Gary, Contractor, Code > 7542 <gar...@nr...<mailto:gar...@nr...>> wrote: > Hi Brian, Jennifer, > > We use GrADS extensively in a production mode and have found when > profiling GrADS runs that 2/3 rds of the data reads are the > coastlines and font files. This may not sound like much, but at > FNMOC when running 100's of charts for each of 32 multi-grid > projects, the coastline and font reads amount to terabytes for each > 12 hour set of runs. > > The re-reads occur for each and every display of a variable from the > same file after it is opened. We are going to try to store these > files in memory after the first read. > > My first question is whether anyone else has looked at this and has > a solution? My next questions, do you have any ideas or guidance to > help us do this? > > I know that GrADS was designed to operate well on small platforms > with little memory, so our goal is to implement a memory caching > capability that would be controlled by a 'set' command and would be > moderated by the amount of memory available. > > As Brian mentions, the map and font database are relatively small. > So, it would be very feasible to setup a ramdisk to hold these files > which would probably give you a nice performance boost with > virtually no programming involved. Depending on your system > architecture, this ramdisk could effectively function as "shared > memory" used by multiple cores. (Because grads is not thread safe, > having an explicit memory buffer to hold this would require that > each grads process duplicate the memory necessary to hold the maps/ > font databases.) One hardware solution is to place grads and its > data on a solid state disk, these are quite affordable these days. > > Just my 2 lazy cents. > > Arlindo > > -- > Arlindo da Silva > da...@al...<mailto:da...@al...> |
|
From: Love, M. G. C. C. 7. <gar...@nr...> - 2011-10-13 18:58:59
|
Hi Brian, I think this solution is exactly on point and I will test it for number of file reads and bytes read. This will be especially useful when I convert to creating a series of plots within the same grads sessions. I will be increasing CACHEMAX because we frequently use the worldmap file for the ultra-high resolution coastlines. Since it is 9671652 bytes, is 10MB big enough for CACHEMAX? When we profiled grads, an equal amount of bytes were read from the font files as were read from the coastline files. I guess every draw string must invoke a font read. Do you have a similar fix for the font problem? Thanks, Gary From: Brian Doty [mailto:do...@co...] Sent: Thursday, October 13, 2011 7:56 AM To: Love, Mr. Gary, Contractor, Code 7542 Cc: ope...@li...; Frost, Mr. Michael Subject: Re: [Opengrads-devel] Dat File re-reads Ok, not too hard, I've added some basic caching of the map file to gxwmap.c. The changes are all isolated to gxwmap.c for now. I kinda did this quick so I haven't done a whole lot of testing on it. The cache lasts for the entire grads session. I don't think this hurts performance any for those cases when grads is just being run to do one plot. This mod is applied to the gxwmap.c from the 2.0.0 code base -- but we will not include it in 2.0; it will be included in 2.1. Right now the CACHEMAX is set to 2MB which will cache lowres, mres, and hires. If you have your own map files that are larger, you may want to increase the CACHEMAX value. This will have no affect on shapefile usage. Gary, I'm not really sure this will help your situation; it will depend on your system environment. Arlindo's suggestion to set up some system-wide solution like a ramdisk may be more along the lines of what you need... Brian On Oct 12, 2011, at 8:38 PM, Love, Mr. Gary, Contractor, Code 7542 wrote: > Arlindo, > > We have some solid state disks that function as local disks because > the global file systems running gpfs are too slow. So our challenge > is how to accomplish how to do the “shared memory” which we also > thought of using. > > Do you know if anyone else has this problem and who could benefit > from the solution? > > Thanks for your input. I’ll ask for more when we get into the > project. > > Gary > > From: Arlindo da Silva [mailto:da...@al...] > Sent: Wednesday, October 12, 2011 4:41 PM > To: ope...@li...<mailto:ope...@li...> > Cc: Brian Doty; Frost, Mr. Michael > Subject: Re: [Opengrads-devel] Dat File re-reads > > On Wed, Oct 12, 2011 at 3:16 PM, Love, Mr. Gary, Contractor, Code > 7542 <gar...@nr...<mailto:gar...@nr...>> wrote: > Hi Brian, Jennifer, > > We use GrADS extensively in a production mode and have found when > profiling GrADS runs that 2/3 rds of the data reads are the > coastlines and font files. This may not sound like much, but at > FNMOC when running 100's of charts for each of 32 multi-grid > projects, the coastline and font reads amount to terabytes for each > 12 hour set of runs. > > The re-reads occur for each and every display of a variable from the > same file after it is opened. We are going to try to store these > files in memory after the first read. > > My first question is whether anyone else has looked at this and has > a solution? My next questions, do you have any ideas or guidance to > help us do this? > > I know that GrADS was designed to operate well on small platforms > with little memory, so our goal is to implement a memory caching > capability that would be controlled by a 'set' command and would be > moderated by the amount of memory available. > > As Brian mentions, the map and font database are relatively small. > So, it would be very feasible to setup a ramdisk to hold these files > which would probably give you a nice performance boost with > virtually no programming involved. Depending on your system > architecture, this ramdisk could effectively function as "shared > memory" used by multiple cores. (Because grads is not thread safe, > having an explicit memory buffer to hold this would require that > each grads process duplicate the memory necessary to hold the maps/ > font databases.) One hardware solution is to place grads and its > data on a solid state disk, these are quite affordable these days. > > Just my 2 lazy cents. > > Arlindo > > -- > Arlindo da Silva > da...@al...<mailto:da...@al...> |
|
From: Brian D. <do...@co...> - 2011-10-13 19:10:29
|
Dear Gary, yes if you make CACHEMAX larger than 9671652, the larger map file will be cached. I am not sure about the font files. The intent of the old code is that they should only be getting read once. I will take a look again at the code -- there may be a bug there... Brian On Oct 13, 2011, at 2:58 PM, Love, Mr. Gary, Contractor, Code 7542 wrote: > Hi Brian, > > I think this solution is exactly on point and I will test it for > number of file reads and bytes read. This will be especially useful > when I convert to creating a series of plots within the same grads > sessions. I will be increasing CACHEMAX because we frequently use > the worldmap file for the ultra-high resolution coastlines. Since > it is 9671652 bytes, is 10MB big enough for CACHEMAX? > > When we profiled grads, an equal amount of bytes were read from the > font files as were read from the coastline files. I guess every > draw string must invoke a font read. Do you have a similar fix for > the font problem? > > Thanks, > Gary > > From: Brian Doty [mailto:do...@co...] > Sent: Thursday, October 13, 2011 7:56 AM > To: Love, Mr. Gary, Contractor, Code 7542 > Cc: ope...@li...; Frost, Mr. Michael > Subject: Re: [Opengrads-devel] Dat File re-reads > > Ok, not too hard, I've added some basic caching of the map file to > gxwmap.c. The changes are all isolated to gxwmap.c for now. I kinda > did this quick so I haven't done a whole lot of testing on it. The > cache lasts for the entire grads session. I don't think this hurts > performance any for those cases when grads is just being run to do one > plot. This mod is applied to the gxwmap.c from the 2.0.0 code base -- > but we will not include it in 2.0; it will be included in 2.1. Right > now the CACHEMAX is set to 2MB which will cache lowres, mres, and > hires. If you have your own map files that are larger, you may want > to increase the CACHEMAX value. This will have no affect on shapefile > usage. > > Gary, I'm not really sure this will help your situation; it will > depend on your system environment. Arlindo's suggestion to set up > some system-wide solution like a ramdisk may be more along the lines > of what you need... Brian > > > > > On Oct 12, 2011, at 8:38 PM, Love, Mr. Gary, Contractor, Code 7542 > wrote: > > > Arlindo, > > > > We have some solid state disks that function as local disks because > > the global file systems running gpfs are too slow. So our challenge > > is how to accomplish how to do the “shared memory” which we also > > thought of using. > > > > Do you know if anyone else has this problem and who could benefit > > from the solution? > > > > Thanks for your input. I’ll ask for more when we get into the > > project. > > > > Gary > > > > From: Arlindo da Silva [mailto:da...@al...] > > Sent: Wednesday, October 12, 2011 4:41 PM > > To: ope...@li... > > Cc: Brian Doty; Frost, Mr. Michael > > Subject: Re: [Opengrads-devel] Dat File re-reads > > > > On Wed, Oct 12, 2011 at 3:16 PM, Love, Mr. Gary, Contractor, Code > > 7542 <gar...@nr...> wrote: > > Hi Brian, Jennifer, > > > > We use GrADS extensively in a production mode and have found when > > profiling GrADS runs that 2/3 rds of the data reads are the > > coastlines and font files. This may not sound like much, but at > > FNMOC when running 100's of charts for each of 32 multi-grid > > projects, the coastline and font reads amount to terabytes for each > > 12 hour set of runs. > > > > The re-reads occur for each and every display of a variable from the > > same file after it is opened. We are going to try to store these > > files in memory after the first read. > > > > My first question is whether anyone else has looked at this and has > > a solution? My next questions, do you have any ideas or guidance to > > help us do this? > > > > I know that GrADS was designed to operate well on small platforms > > with little memory, so our goal is to implement a memory caching > > capability that would be controlled by a 'set' command and would be > > moderated by the amount of memory available. > > > > As Brian mentions, the map and font database are relatively small. > > So, it would be very feasible to setup a ramdisk to hold these files > > which would probably give you a nice performance boost with > > virtually no programming involved. Depending on your system > > architecture, this ramdisk could effectively function as "shared > > memory" used by multiple cores. (Because grads is not thread safe, > > having an explicit memory buffer to hold this would require that > > each grads process duplicate the memory necessary to hold the maps/ > > font databases.) One hardware solution is to place grads and its > > data on a solid state disk, these are quite affordable these days. > > > > Just my 2 lazy cents. > > > > Arlindo > > > > -- > > Arlindo da Silva > > da...@al... > |
|
From: Brian D. <do...@co...> - 2011-10-13 19:24:03
|
Hi Gary, I checked the font behavior, and it seems to be working as intended. I put a debug write statement in the routine that reads the font file, and it indicates it only gets called once, when a particular font is first used. I was testing with the COLA 2.0.0 code base. If you want to test with your version, you can put a debug write statement at the beginning of the function gxchrd in the gxchpl.c source file; gxchrd is the routine that actually reads in a font file... Brian On Oct 13, 2011, at 2:58 PM, Love, Mr. Gary, Contractor, Code 7542 wrote: > Hi Brian, > > I think this solution is exactly on point and I will test it for > number of file reads and bytes read. This will be especially useful > when I convert to creating a series of plots within the same grads > sessions. I will be increasing CACHEMAX because we frequently use > the worldmap file for the ultra-high resolution coastlines. Since > it is 9671652 bytes, is 10MB big enough for CACHEMAX? > > When we profiled grads, an equal amount of bytes were read from the > font files as were read from the coastline files. I guess every > draw string must invoke a font read. Do you have a similar fix for > the font problem? > > Thanks, > Gary > > From: Brian Doty [mailto:do...@co...] > Sent: Thursday, October 13, 2011 7:56 AM > To: Love, Mr. Gary, Contractor, Code 7542 > Cc: ope...@li...; Frost, Mr. Michael > Subject: Re: [Opengrads-devel] Dat File re-reads > > Ok, not too hard, I've added some basic caching of the map file to > gxwmap.c. The changes are all isolated to gxwmap.c for now. I kinda > did this quick so I haven't done a whole lot of testing on it. The > cache lasts for the entire grads session. I don't think this hurts > performance any for those cases when grads is just being run to do one > plot. This mod is applied to the gxwmap.c from the 2.0.0 code base -- > but we will not include it in 2.0; it will be included in 2.1. Right > now the CACHEMAX is set to 2MB which will cache lowres, mres, and > hires. If you have your own map files that are larger, you may want > to increase the CACHEMAX value. This will have no affect on shapefile > usage. > > Gary, I'm not really sure this will help your situation; it will > depend on your system environment. Arlindo's suggestion to set up > some system-wide solution like a ramdisk may be more along the lines > of what you need... Brian > > > > > On Oct 12, 2011, at 8:38 PM, Love, Mr. Gary, Contractor, Code 7542 > wrote: > > > Arlindo, > > > > We have some solid state disks that function as local disks because > > the global file systems running gpfs are too slow. So our challenge > > is how to accomplish how to do the “shared memory” which we also > > thought of using. > > > > Do you know if anyone else has this problem and who could benefit > > from the solution? > > > > Thanks for your input. I’ll ask for more when we get into the > > project. > > > > Gary > > > > From: Arlindo da Silva [mailto:da...@al...] > > Sent: Wednesday, October 12, 2011 4:41 PM > > To: ope...@li... > > Cc: Brian Doty; Frost, Mr. Michael > > Subject: Re: [Opengrads-devel] Dat File re-reads > > > > On Wed, Oct 12, 2011 at 3:16 PM, Love, Mr. Gary, Contractor, Code > > 7542 <gar...@nr...> wrote: > > Hi Brian, Jennifer, > > > > We use GrADS extensively in a production mode and have found when > > profiling GrADS runs that 2/3 rds of the data reads are the > > coastlines and font files. This may not sound like much, but at > > FNMOC when running 100's of charts for each of 32 multi-grid > > projects, the coastline and font reads amount to terabytes for each > > 12 hour set of runs. > > > > The re-reads occur for each and every display of a variable from the > > same file after it is opened. We are going to try to store these > > files in memory after the first read. > > > > My first question is whether anyone else has looked at this and has > > a solution? My next questions, do you have any ideas or guidance to > > help us do this? > > > > I know that GrADS was designed to operate well on small platforms > > with little memory, so our goal is to implement a memory caching > > capability that would be controlled by a 'set' command and would be > > moderated by the amount of memory available. > > > > As Brian mentions, the map and font database are relatively small. > > So, it would be very feasible to setup a ramdisk to hold these files > > which would probably give you a nice performance boost with > > virtually no programming involved. Depending on your system > > architecture, this ramdisk could effectively function as "shared > > memory" used by multiple cores. (Because grads is not thread safe, > > having an explicit memory buffer to hold this would require that > > each grads process duplicate the memory necessary to hold the maps/ > > font databases.) One hardware solution is to place grads and its > > data on a solid state disk, these are quite affordable these days. > > > > Just my 2 lazy cents. > > > > Arlindo > > > > -- > > Arlindo da Silva > > da...@al... > |
|
From: Michael F. <mic...@no...> - 2011-10-14 18:04:52
|
hi Brian and Jennifer,
i noticed a diff in 'q shades' in gauser -- now using snprintf... this
tripped up my python the parses the output, just curious if this was
intended? not clear it really in is "<=" vice "<". the right hand side is the new gauser.c
the left older.
have a good weekend.
best /r mike
else if (cmpwrd(arg,"shades")) { | else if (cmpwrd(arg,"shades")) {
if (pcm->shdcnt<1) { | if (pcm->shdcnt<1) {
gaprnt(2,"None\n"); | gaprnt(2,"None\n");
} else { | } else {
sprintf(pout,"Number of levels = %i\n",pcm->shdcnt); | snprintf(pout,255,"Number of levels = %i\n",pcm->shdcnt);
gaprnt(2,pout); | gaprnt(2,pout);
for (i=0; i<pcm->shdcnt; i++) { | for (i=0; i<pcm->shdcnt; i++) {
if (i==0) | if (i==0)
sprintf (pout,"%i < %g\n",pcm->shdcls[i],pcm->shdlvs[1]); | snprintf(pout,255,"%i <= %g\n",pcm->shdcls[i],pcm->shdlvs[1]);
else if (i==pcm->shdcnt-1) | else if (i==pcm->shdcnt-1)
sprintf (pout,"%i %g >\n",pcm->shdcls[i],pcm->shdlvs[i]); | snprintf(pout,255,"%i %g >\n",pcm->shdcls[i],pcm->shdlvs[i]);
else | else
sprintf (pout,"%i %g %g\n",pcm->shdcls[i],pcm->shdlvs[i],pcm->shdlvs[i+1]); | snprintf(pout,255,"%i %g %g\n",pcm->shdcls[i],pcm->shdlvs[i],pcm->shdlvs[i+1]);
gaprnt(2,pout); | gaprnt(2,pout);
} | }
} | }
} | }
i've added your gxwmap.c to our opengrads/src (the only diff was your
caching code) and set
#define CACHEMAX 12000000
to handle the hires map as Gary suggested
i also added a tweak to 'set grid' to allow setting line thickness,
e.g.,
'set grid on 5 15 5'
to make the default grid a 5 thickness vice the default of 1
the mods are in grads.h, grads.c, gauser.c and gagx.c (attached).
that's the only tweaking i've done with the code the last couple of
years. i had added 'set z 1 last' but agree that it's not really proper
to support that option since 'z' is not the same kind of coordinate as
't'
best /r mike
On 10/13/11 7:23 PM, Brian Doty wrote:
> Hi Gary, I checked the font behavior, and it seems to be working as
> intended. I put a debug write statement in the routine that reads the
> font file, and it indicates it only gets called once, when a
> particular font is first used. I was testing with the COLA 2.0.0 code
> base. If you want to test with your version, you can put a debug
> write statement at the beginning of the function gxchrd in the
> gxchpl.c source file; gxchrd is the routine that actually reads in a
> font file... Brian
>
> On Oct 13, 2011, at 2:58 PM, Love, Mr. Gary, Contractor, Code 7542
> wrote:
>
>> Hi Brian,
>>
>> I think this solution is exactly on point and I will test it for
>> number of file reads and bytes read. This will be especially useful
>> when I convert to creating a series of plots within the same grads
>> sessions. I will be increasing CACHEMAX because we frequently use
>> the worldmap file for the ultra-high resolution coastlines. Since
>> it is 9671652 bytes, is 10MB big enough for CACHEMAX?
>>
>> When we profiled grads, an equal amount of bytes were read from the
>> font files as were read from the coastline files. I guess every
>> draw string must invoke a font read. Do you have a similar fix for
>> the font problem?
>>
>> Thanks,
>> Gary
>>
>> From: Brian Doty [mailto:do...@co...]
>> Sent: Thursday, October 13, 2011 7:56 AM
>> To: Love, Mr. Gary, Contractor, Code 7542
>> Cc: ope...@li...; Frost, Mr. Michael
>> Subject: Re: [Opengrads-devel] Dat File re-reads
>>
>> Ok, not too hard, I've added some basic caching of the map file to
>> gxwmap.c. The changes are all isolated to gxwmap.c for now. I kinda
>> did this quick so I haven't done a whole lot of testing on it. The
>> cache lasts for the entire grads session. I don't think this hurts
>> performance any for those cases when grads is just being run to do one
>> plot. This mod is applied to the gxwmap.c from the 2.0.0 code base --
>> but we will not include it in 2.0; it will be included in 2.1. Right
>> now the CACHEMAX is set to 2MB which will cache lowres, mres, and
>> hires. If you have your own map files that are larger, you may want
>> to increase the CACHEMAX value. This will have no affect on shapefile
>> usage.
>>
>> Gary, I'm not really sure this will help your situation; it will
>> depend on your system environment. Arlindo's suggestion to set up
>> some system-wide solution like a ramdisk may be more along the lines
>> of what you need... Brian
>>
>>
>>
>>
>> On Oct 12, 2011, at 8:38 PM, Love, Mr. Gary, Contractor, Code 7542
>> wrote:
>>
>>> Arlindo,
>>>
>>> We have some solid state disks that function as local disks because
>>> the global file systems running gpfs are too slow. So our challenge
>>> is how to accomplish how to do the shared memory which we also
>>> thought of using.
>>>
>>> Do you know if anyone else has this problem and who could benefit
>>> from the solution?
>>>
>>> Thanks for your input. Ill ask for more when we get into the
>>> project.
>>>
>>> Gary
>>>
>>> From: Arlindo da Silva [mailto:da...@al...]
>>> Sent: Wednesday, October 12, 2011 4:41 PM
>>> To: ope...@li...
>>> Cc: Brian Doty; Frost, Mr. Michael
>>> Subject: Re: [Opengrads-devel] Dat File re-reads
>>>
>>> On Wed, Oct 12, 2011 at 3:16 PM, Love, Mr. Gary, Contractor, Code
>>> 7542 <gar...@nr...> wrote:
>>> Hi Brian, Jennifer,
>>>
>>> We use GrADS extensively in a production mode and have found when
>>> profiling GrADS runs that 2/3 rds of the data reads are the
>>> coastlines and font files. This may not sound like much, but at
>>> FNMOC when running 100's of charts for each of 32 multi-grid
>>> projects, the coastline and font reads amount to terabytes for each
>>> 12 hour set of runs.
>>>
>>> The re-reads occur for each and every display of a variable from the
>>> same file after it is opened. We are going to try to store these
>>> files in memory after the first read.
>>>
>>> My first question is whether anyone else has looked at this and has
>>> a solution? My next questions, do you have any ideas or guidance to
>>> help us do this?
>>>
>>> I know that GrADS was designed to operate well on small platforms
>>> with little memory, so our goal is to implement a memory caching
>>> capability that would be controlled by a 'set' command and would be
>>> moderated by the amount of memory available.
>>>
>>> As Brian mentions, the map and font database are relatively small.
>>> So, it would be very feasible to setup a ramdisk to hold these files
>>> which would probably give you a nice performance boost with
>>> virtually no programming involved. Depending on your system
>>> architecture, this ramdisk could effectively function as "shared
>>> memory" used by multiple cores. (Because grads is not thread safe,
>>> having an explicit memory buffer to hold this would require that
>>> each grads process duplicate the memory necessary to hold the maps/
>>> font databases.) One hardware solution is to place grads and its
>>> data on a solid state disk, these are quite affordable these days.
>>>
>>> Just my 2 lazy cents.
>>>
>>> Arlindo
>>>
>>> --
>>> Arlindo da Silva
>>> da...@al...
>>
>
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2d-oct
> _______________________________________________
> Opengrads-devel mailing list
> Ope...@li...
> https://lists.sourceforge.net/lists/listinfo/opengrads-devel
|
|
From: Jennifer A. <jm...@CO...> - 2011-10-14 20:10:43
|
The change from sprintf to snprintf was implemented throughout the source code -- I guess it is less prone to memory overrides this way. The other change, from '<' to '<=' is deliberate and meaningful -- the 'q shades' output did not reflect the algorithm for doing the shaded contouring. This was a mod checked in on June 10, 2010, so it was in 2.0.a9. The ChangeLog says, "Fixed color selection for shaded plots: lev1 < val <= lev2". --Jennifer On Oct 14, 2011, at 2:04 PM, Michael Fiorino wrote: > hi Brian and Jennifer, > > i noticed a diff in 'q shades' in gauser -- now using snprintf... > this > tripped up my python the parses the output, just curious if this was > intended? not clear it really in is "<=" vice "<". the right hand > side is the new gauser.c > the left older. > > have a good weekend. > > best /r mike |
|
From: Brian D. <do...@co...> - 2011-10-12 20:40:03
|
Hi Gary, what you report is of concern and should be fixed. You are correct that the old rationale for the way it works no longer applies. The map and font files that come with grads are quite small by today's standards. Even the hires data is only 1.7MB. If you have your own map files they may be bigger. Looking at the map code in gxwmap.c, the entry point is function gxdmap. My approach to caching the map files would be to replace the calls to fopen, fclose, fread, and fseek that appear in gxdmap with calls to some new routines that would be put in gxwmap.c. The new open routine would read the file in its entirety and put it in some dynamically-allocated memory buffer -- if it is smaller than some threshold, say. Then the read and seek routines would just pull data out of the memory buffer, or do the I/O if the memory buffer wasn't allocated. The open routine would also need to detect if the file that is being opened is already in memory. So the memory buffer should be a link list of structures where each structure contains the file name and the pointer to the data cache. It is probably sufficient to just test on the full path name; it seems unlikely people would point to different locations of the same file within one grads session. There probably needs to also be a routine to free up all the memory and reset; perhaps on "reinit" this should be done? For the fonts -- don't these get read in once? Well it's been 20 years so I'll take a look... in gxchpl.c the string plotting all goes through the gxchpl entry point. That calls an internal routine gxchgc, which checks some pointer and reads in the font data for that font if the pointer is NULL. Looks like the font data is read once, the first time the font is used. Your stats are showing a ton of reads to these files? That is odd and may be worth looking into further... Brian On Oct 12, 2011, at 3:16 PM, Love, Mr. Gary, Contractor, Code 7542 wrote: > Hi Brian, Jennifer, > > We use GrADS extensively in a production mode and have found when > profiling GrADS runs that 2/3 rds of the data reads are the > coastlines and font files. This may not sound like much, but at > FNMOC when running 100's of charts for each of 32 multi-grid > projects, the coastline and font reads amount to terabytes for each > 12 hour set of runs. > > The re-reads occur for each and every display of a variable from the > same file after it is opened. We are going to try to store these > files in memory after the first read. > > My first question is whether anyone else has looked at this and has > a solution? My next questions, do you have any ideas or guidance to > help us do this? > > I know that GrADS was designed to operate well on small platforms > with little memory, so our goal is to implement a memory caching > capability that would be controlled by a 'set' command and would be > moderated by the amount of memory available. > > Thanks, > Gary |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X