pas-dev — Development mailing list.

[Pas-dev] pas CVS commit

[Kyle R. B. <mo...@us...>]

Update of /cvsroot/pas/pas
In directory sc8-pr-cvs1:/tmp/cvs-serv32198

Modified Files:
	ChangeLog 
Log Message:
proxy shell enahncements for non-interactive use

[Pas-dev] pas/src/Org/Bgw/HTTP/Proxy CVS commit

[Kyle R. B. <mo...@us...>]

Update of /cvsroot/pas/pas/src/Org/Bgw/HTTP/Proxy
In directory sc8-pr-cvs1:/tmp/cvs-serv32198/src/Org/Bgw/HTTP/Proxy

Modified Files:
	Shell.pm 
Log Message:
proxy shell enahncements for non-interactive use

[Pas-dev] pas/src/Org/Bgw/Pas/Psp CVS commit

[Kyle R. B. <mo...@us...>]

Update of /cvsroot/pas/pas/src/Org/Bgw/Pas/Psp
In directory sc8-pr-cvs1:/tmp/cvs-serv2567/src/Org/Bgw/Pas/Psp

Modified Files:
	Compiler.pm 
Log Message:
oops, parent class wasn't being required in the generated file...

[Pas-dev] What's next?

[Kyle R . B. <mo...@vo...>]

I've used Pas now in a few projects where I work over the past two years
or so.  I've found it to perform well, and to be stable.  The feature set 
has been a good fit for the applicaiton domains in which I've used it.
It's worked well enough, there just doesn't seem to be much spark in the
project.

We've tossed around ideas for features, additions and fixes that we'd like
to explore or implement across this mailing list.  Some of them have been 
implemented because we either needed the feature or they were implemented 
as the opportunity arose, effectivly a bonus, when some bug got fixed.  
Though largely they've fallen aside and haven't received much attention.

There hasn't been a whole lot of need for new features or approaches.  There
hasn't been much vision for the project.  I suppose a large part of that
is that I'm the defacto visionary and I haven't been doing a very good
job of driving the project.

Towards the goals of where Pas should go from here, I'd like to solicit
your feedback on Pas in the past, what we have in the present, and what
you would like to see going forward.  Not just the parts that irk you,
or the new minor features you'd lke to have, but also what might excite
you, what you'd like to work on or see worked on.  I'm looking for
your personal opinions and not just your technical desires.



Kyle


-- 

------------------------------------------------------------------------------
Wisdom and Compassion are inseparable.
        -- Christmas Humphreys
mo...@vo...                            http://www.voicenet.com/~mortis
------------------------------------------------------------------------------

[Pas-dev] pas CVS commit

[Kyle R. B. <mo...@us...>]

Update of /cvsroot/pas/pas
In directory sc8-pr-cvs1:/tmp/cvs-serv30442

Modified Files:
	ChangeLog TODO 
Log Message:
added notes to changelog and removed some of the implemented features from the todo file...

[Pas-dev] New Compiler

[Kyle R . B. <mo...@vo...>]

I just checked in a new PSP compiler.  This should fix the previously
reported bug with newlines being dropped under specific circumstances.

For the willing, please check out the latest CVS and give the new compiler
a whirl.

The old compiler used a search/replace regex approach (complicated,
fragile, hard to extend, etc).  The new approach uses split on PSP
identifiers /(<%|%>)/ and then uses a state based approach on the resulting
tokens.

Some immediate benefits are that the compiler throws an exception if
you nest PSP tags or if you leave a tag open to the end of a PSP file.

The new compiler should be largely api compatible with the previous
compiler, so code that called it should continue to work (the request
handler and compile utilities both worked for me).  The new implementation
should also be much easier to add new features into - it is much easier
to read and understand.  In fact, instead of implementing includes as
a special case, the new compiler now uses itself.  It creates a compiler, 
uses it to compile the include file and then inserts the results into 
the including file.


Thanks in advance for any help with testing.

Kyle

-- 

------------------------------------------------------------------------------
Wisdom and Compassion are inseparable.
        -- Christmas Humphreys
mo...@vo...                            http://www.voicenet.com/~mortis
------------------------------------------------------------------------------

[Pas-dev] pas/src/Org/Bgw/Pas/Psp CVS commit

[Kyle R. B. <mo...@us...>]

Update of /cvsroot/pas/pas/src/Org/Bgw/Pas/Psp
In directory sc8-pr-cvs1:/tmp/cvs-serv5538/src/Org/Bgw/Pas/Psp

Modified Files:
	Compiler.pm 
Log Message:
new compiler

[Pas-dev] pas/src/Org/Bgw/Pas/Psp CVS commit

[Kyle R. B. <mo...@us...>]

Update of /cvsroot/pas/pas/src/Org/Bgw/Pas/Psp
In directory sc8-pr-cvs1:/tmp/cvs-serv3756/src/Org/Bgw/Pas/Psp

Modified Files:
	Compiler.pm 
Log Message:
new compiler

[Pas-dev] pas CVS commit

[Kyle R. B. <mo...@us...>]

Update of /cvsroot/pas/pas
In directory sc8-pr-cvs1:/tmp/cvs-serv3756

Modified Files:
	ChangeLog 
Log Message:
new compiler

[Pas-dev] pas/sandbox/newCompiler CVS commit

[Kyle R. B. <mo...@us...>]

Update of /cvsroot/pas/pas/sandbox/newCompiler
In directory sc8-pr-cvs1:/tmp/cvs-serv27225

Modified Files:
	test.psp 
Added Files:
	_footer.psp _header.psp 
Log Message:
working on new compiler

[Pas-dev] REMOTE_ADDR/HTTP_X_FORWARDED_FOR

[Kyle R . B. <mo...@vo...>]

We've got a Pas installation that is sitting behind a proxy (to make
SSL and mog_gzip work togeather).  Apache sets REMOTE_ADDR to the adress
of the proxy that's doing the forwarding (for us this is typically
127.0.0.1), and HTTP_X_FORWARDED_FOR to the address that the request
is actually for.

Our set up looks like this:

  PROXY => MOD_PERL/Pas

Where the proxy does SSL, while the mod_perl/Pas instance uses mod_gzip
for content compression.

Using the REMOTE_ADDR as part of the seesion id is a useful feature (in 
case you want to do ip validation/verification or use it for tracking 
purposes).

It would be nice if Pas could switch on which env var to use to obtain
the remote ip address.  I'm proposing to just make this a configuration
setting, though I'm not sure what other environmetns are like or how
else we could support this.


Feedback?

Kyle


-- 

------------------------------------------------------------------------------
Wisdom and Compassion are inseparable.
        -- Christmas Humphreys
mo...@vo...                            http://www.voicenet.com/~mortis
------------------------------------------------------------------------------

[Pas-dev] please discuss

[Kyle R . B. <mo...@vo...>]

Ok.  When developing our web applications with Pas, we nearly univerally
break the pages up into parts and use a standard layout to bring
those chunks in.  This saves us maintenence, disk-space for the PSP pages
and just generaly make the presentation layer easier to work with.

For example:

  <%@ include file = "_header.psp" %>
  <%-- page content goes here --%>
  <%@ include file = "_footer.psp" %>

This is pretty much the standard approach.  It makes it easy for a 
standard look and feel across all the pages, logos, standard navigation 
elements, etc.

I was just looking at the generated output of one of the pages.  Now
if you think about it, the implicaitons of this type of inclusion are
apparent.  I had never though about it.  

The implicaiton that I just realized is that, provided the included
file is static (no active psp code int it) you end up with the same
header data and same footer data in each and every generated file.
This is a waste of memory and space.  It could be refactored so that
pages with static portions that are the same all share the actual same
static region(s).  This should, for some sites, save memory.  Possibly
significant amounts of memory.

Thinking about this took me back to the concetps of content methods and 
taglibs.  And to the optmization strategies we were throwing around
before about marking up pages or sections as 'resolve at compile time',
'request-static', and 'static-only'.

Another, probably not quite correct, way to think about this is
like inlined functions.  It's the difference between referencing 
something shared vs having it expanded everwhere in the code.

There are a few ways I can initially think of we can approach 

  1. shared/inherited methods
  2. shared variables/constants

We could move all these shared content areas into their own package and
make them available as a method name or as a variable name.  Then
a static include could look like:

  <%@ static include file = "_header.psp" %>
  <%-- content goes here %-->
  <%@ static include file = "_footer.psp" %>

And the generated code could look something like:

  sub execute
  {
    my($self) = @_;
    ...
    $self->response->print(
      Some::Pkg::Name::Based::On::The::Include::Name->_staticContent
    );
    ...
  }

Or, maybe, we could just have that method do the printing internally,
you pass in the response object and it appends it's content to it..

  sub execute
  {
    my($self) = @_;
    ...
    Some::Pkg::Name::Based::On::The::Include::Name->_staticContent(
      $self->response
    );
    ...
  }

Using variables instead of methods it might look like:

  sub execute
  {
    my($self) = @_;
    ...
    $self->response->print(
      $Some::Pkg::Name::Based::On::The::Include::Name::_staticContent
    );
    ...
  }

Variables might be more efficient than methods/constants.  Though I'm
not certin which one is a better solution - which one will lead to better
design and maintenence later on.

As a related aside, the static include could easily be implemented in the
current compiler.  The behavior could be for the compiler to attempt to
comple/resolve/evaluate the included file as PSP code, so you could use
it as a simple first generation macro system for expanding out the
static code.  A few points to keep in mind are that you could put Perl
code in these static includes, but there would be no request object,
and no session available to you (or there would be but they'd be empty
and/or invalid).  They would be invalid because the code is being
executed at compile time, not in response to a request.


That's the latest round of ideas for now.  What do you think?  Is doing
something like this worth-while?  My hunch is that it would be.

Kyle

-- 

------------------------------------------------------------------------------
Wisdom and Compassion are inseparable.
        -- Christmas Humphreys
mo...@vo...                            http://www.voicenet.com/~mortis
------------------------------------------------------------------------------

[Pas-dev] compiler bug

[Kyle R . B. <mo...@vo...>]

The following PSP code:

  <%
    my $aplicaitonName = $self->config->get('com.yoyodine.app.name');
  %>
  Regards,
  <%= $applicationName %> Support Team
  Yoyodine, Inc.

Resulted in the following code in the generated Page object:

  #line X mail.psp
  $self->response->print( $applicationName );
  #line X mail.psp
  $self->response->print( q{ Support TeamYoyodine, Inc.
});


Which is patently wrong.  The newline between 'Team' and 'Yoyodine'
silently went away.  In standard HTML where tags often delimit content
it might often go unnoticed.  I certinly haven't noticed this bug 
until now (in a plain text email, all the whitespace is importaint).

The bug manifests itself when a line in a PSP file starts with an
expression (<%= %>) tag with two lines of content following it.
If you put so little as a single space in front of the expression
tags, everything is ok.

At some point in the near future we'll need to fix this...


Kyle

-- 

------------------------------------------------------------------------------
Wisdom and Compassion are inseparable.
        -- Christmas Humphreys
mo...@vo...                            http://www.voicenet.com/~mortis
------------------------------------------------------------------------------

[Pas-dev] pas/src/Org/Bgw/Pas CVS commit

[Kyle R. B. <mo...@us...>]

Update of /cvsroot/pas/pas/src/Org/Bgw/Pas
In directory sc8-pr-cvs1:/tmp/cvs-serv29386/src/Org/Bgw/Pas

Modified Files:
	Session.pm 
Log Message:
removed logging message about compression stats, noisy and unnecessary

[Pas-dev] pas/bin CVS commit

[Kyle R. B. <mo...@us...>]

Update of /cvsroot/pas/pas/bin
In directory sc8-pr-cvs1:/tmp/cvs-serv26060/bin

Modified Files:
	viewSession 
Log Message:
updated viewSession to attempt to parse the sid out of the file and just use the session object instead of trying to thaw the session...

[Pas-dev] pas/src/Org/Bgw/Pas/Psp CVS commit

[Kyle R. B. <mo...@us...>]

Update of /cvsroot/pas/pas/src/Org/Bgw/Pas/Psp
In directory sc8-pr-cvs1:/tmp/cvs-serv10190/src/Org/Bgw/Pas/Psp

Modified Files:
	Compiler.pm 
Log Message:
refactored conditional recompile checks

[Pas-dev] pas/src/Org/Bgw/Pas CVS commit

[Kyle R. B. <mo...@us...>]

Update of /cvsroot/pas/pas/src/Org/Bgw/Pas
In directory sc8-pr-cvs1:/tmp/cvs-serv10190/src/Org/Bgw/Pas

Modified Files:
	RequestHandler.pm 
Log Message:
refactored conditional recompile checks

[Pas-dev] pas/src CVS commit

[Kyle R. B. <mo...@us...>]

Update of /cvsroot/pas/pas/src
In directory sc8-pr-cvs1:/tmp/cvs-serv10190/src

Modified Files:
	build-startup.pl 
Log Message:
refactored conditional recompile checks

Re: [Pas-dev] Apache, mod_perl and Preloading?

[Kyle R . B. <mo...@vo...>]

Some notes on the new build-startup.pl:

  - Top reports SIZE of 50482, and SHARED of 32244, this represents 63%
    (after hitting most of the pages in the site)
  - The page generation/load times seems faster.  I haven't benchmarked
    this, but the responsiveness is better.

Some of this is because dynamicrecompile is turned off so those checks
are not being done at runtime.

Kyle

-- 

------------------------------------------------------------------------------
Wisdom and Compassion are inseparable.
        -- Christmas Humphreys
mo...@vo...                            http://www.voicenet.com/~mortis
------------------------------------------------------------------------------

Re: [Pas-dev] Apache, mod_perl and Preloading?

[Kyle R . B. <mo...@vo...>]

> > Are not PSP files mostly content?  Or at least commonly mostly content?
> 
> Well, yes. But most of the psp's I've written are small. So the memory
> usage was reasonable. A psp with a bunch of huge tables with a ton of
> content could get out of hand quickly. I can see why you're going down the
> preload path. 

Even with smaller psps, the code they compile into should still be shared.

> > Ah.  So pre-compilation and disabling dynamic-recompliation and StatInc
> > should be _good_ things?
> 
> Yes. Very good things. 

Good.

> > Wouldn't the pre-compile/pre-load provide both a performance boost and
> > a memory reduction (for the child processes)?  There is an obvious 
> > overhead associated with pre-loading infrequently hit parts of the 
> > site/codebase, but unless the bulk of your site is mostly infrequently 
> > hit stuff, then pre-loading should be a win (both cpu and memory) right?
> 
> THis is where it gets interresting. There's really one way to tell.
> Benchmarks :)

Yes.  I don't have the time or resources to do proper benchmarks right
now though.

> > Hrm...that's an interesting possibility.  Combiend with earlier
> > proposals of compile-time (psp compile time) expansion of psp code
> > into static content, that could provide more winnage...
> 
> Well, yes. Sort of. Depending.... I have a funny feeling we're about to
> reinvent SSI :)

Nope.  This has nothing to do with SSI or it's aproach IMO.



> Cool, let me know how it goes. 

I just checked in a build-startup.pl that uses File::Find on the
pas.documentRoot in the configuration file, and compiles all psp files
that don't start with an underscore (that's just a convention that
we've been using for included psp files, which might not result
in a valid .pm file after compilaion).

If your codebase compiles properly, that code compmiles every psp
file at Apache startup and pre-loads it.  I need to move the 
conditional recompile check logic out of the request handler and
into the psp compiler...so it can get reused by the build-startup.pl


Other than that, it looks like it works.

One thing is that the .pm files generated from the .psp files end 
up being owned by root.

Kyle

-- 

------------------------------------------------------------------------------
Wisdom and Compassion are inseparable.
        -- Christmas Humphreys
mo...@vo...                            http://www.voicenet.com/~mortis
------------------------------------------------------------------------------

[Pas-dev] pas/src CVS commit

[Kyle R. B. <mo...@us...>]

Update of /cvsroot/pas/pas/src
In directory sc8-pr-cvs1:/tmp/cvs-serv29303/src

Added Files:
	build-startup.pl 
Log Message:
trying out new startup that pre-compiles and preloads...

Re: [Pas-dev] Apache, mod_perl and Preloading?

[Kyle R . B. <mo...@vo...>]

> Exactly.... so if you have a ton of content.. this is a win.

Are not PSP files mostly content?  Or at least commonly mostly content?

> This is also why Embperl didnt cache content. It only caches the code so
> that lots of html doesnt mean lots of memory. Serving static content of
> the disk is still fast..... 

Embperl didn't cache the template file?  It only cached the code?  What
did it do with the content break it up into multiple little files and
serve them from disk or something?

Pas/PSP creates code out of the template file, so the content becomes
intertwined with the code...

> Sorry. Thats what happens when you word salad between other tings....
> what I meant was that turning off StatINC fairly significant perfomance
> boost in and of itself. 

Ah.  So pre-compilation and disabling dynamic-recompliation and StatInc
should be _good_ things?

> > I want the memory usage of our site to be lower.  For our dev envronment
> > we've got 3 apache instances set up so we can run 3 applications and
> > keep them seperate from each other.  All those children and all that
> > memory adds up to swapping (the same box runs the 3 databases as well).
> > Our production environment will soon be just like this.  Memory reduciton
> > allows us to do more with less hardware.
> > 
> 
> So are you tuning for speed or memory size? Would it be unreasonable to
> just cut down max requests per child? Keep httpd's from caching a ton of
> content in the first place? There's also Apache::SizeLimit. You can
> control how much shared/unshared memory each httpd can have. THat'll
> help ya too.

Wouldn't the pre-compile/pre-load provide both a performance boost and
a memory reduction (for the child processes)?  There is an obvious 
overhead associated with pre-loading infrequently hit parts of the 
site/codebase, but unless the bulk of your site is mostly infrequently 
hit stuff, then pre-loading should be a win (both cpu and memory) right?

For sites that are either huge, or have vast expanses of infrequently
used PSPs/Page objects, then preloading might be worse - but if that
is the case, then maybe there are other issues at work.


> > My conclusions from our conversation and the page you referenced are:
> >   - at a minimum pre-compilation and pre-loading reduces the time 
> >     it takes a newly forked child to service requests for a given page.
> >     In the non-pre-load scenario, the child may have to perform 
> >     PSP compliation, and it may have to perform Perl code compilation.
> >   - at a minimum pre-loading helps share at least some memory, the
> >     benefits are maximized if you keep the MaxRequestsPerChild down
> >     to a reasonable level to flush memory that resulted in copy-on-write
> >     because of member variables and per-request handling duties.
> > 
> > So, overall, pre-compliation and pre-loading at Apache startup time are
> > worth the effort of writing a new startup.pl - right?
> 
> Yup. I'd say load the most common pages, set a semi-restrictive
> Apache::SizeLimit ( think I forgot to mention it before, but
> Apache::SizeLimit is nice) in your startup.pl, and lower max requests.
> Hopefully that'll get you out of swap in the short term. In the long
> term it might be worht it to figure out a hybrid psp/html page. Where
> large static content gets served off the disk but the dynamic bits are
> in a module..... 

Hrm...that's an interesting possibility.  Combiend with earlier
proposals of compile-time (psp compile time) expansion of psp code
into static content, that could provide more winnage...

Thanks for the discussion, I'm in the process of writing an alternate
startup.pl that I'll check in later today.

Kyle



-- 

------------------------------------------------------------------------------
Wisdom and Compassion are inseparable.
        -- Christmas Humphreys
mo...@vo...                            http://www.voicenet.com/~mortis
------------------------------------------------------------------------------

Re: [Pas-dev] Apache, mod_perl and Preloading?

[Mental P. <me...@ne...>]

On Fri, 2003-05-02 at 11:03, Kyle R . Burton wrote:
> > True, but there are some details you're forgetting. :) Even get/sets
> > arent going to avoid the whole copy on write overhead. At least I dont
> > think so.... and I dont have time right now to check.
>=20
> The only benefit would be that the shared (compiled) code (the compiled
> Perl code, the op trees) would be shared.  Per-child data segments (membe=
r
> variables as you point out) would be subject to copy-on-write and diverge
> once any 'set' had been invoked.  That's unavoidable and expected.
>=20

Exactly.... so if you have a ton of content.. this is a win.

This is also why Embperl didnt cache content. It only caches the code so
that lots of html doesnt mean lots of memory. Serving static content of
the disk is still fast.....=20

> > My real question is... are you SURE that this will help? Wont execute()
> > be different as far as variables go on a per process basis? There reall=
y
> > isnt a way to force constants to use one page and variables to use
> > another..... Even if the code starts out shared, most page objects will
> > wind up being unshared simply by their nature. The lower, core modules
> > are more likely to stay shared longer. So preloading and lowering
> > maxrequests per child may help if done in tandem.
>=20
> This should work for the same reason preloading core modules works --
> code is not modified so it stays shared (no write, no copy-on-write)
> data that never changes (like all those strings that represent the
> bulk of the HTML in the psp files) will stay shared as well since
> those stirngs are effectivly constant across the lifetime of the=20
> appilcation.

> > Still, I havent spent too much time looking at the guts of apache memor=
y
> > management. I've found that simply by following the tuning guide and
> > loading the common core modules performance is fine.
>=20
> I don't think it's 100% Apache we're talking about here, the whole
> copy-on-write feature is part of the process model on most unixes
> for children that are the result of a fork (unless there is a subsequnt
> exec call used to overwrite the process).
>=20
> > I do concur we would benefit from a build spider to precompile all the
> > psp's, but honestly StatINC is more overhead.=20
>=20
> I'm not sure I understandt his - StatINC is more overhead than what?
>=20

Sorry. Thats what happens when you word salad between other tings....
what I meant was that turning off StatINC fairly significant perfomance
boost in and of itself.=20


> > Before you kill yourself doing this, check out this link:
> > http://perl.apache.org/docs/1.0/guide/performance.html#Know_Your_Operat=
ing_System
> >=20
> > I could just plagerize it, but I'd probably just explain it wrong ;)
>=20
> Quoting that page:
>=20
>   Do not forget that if you preload most of your code at server=20
>   startup, the newly forked child gets ready very fast, because=20
>   it inherits most of the preloaded code and the perl interpreter=20
>   from the parent process.
>=20
> And additaionly that page says that even though children will grow
> their memory usage as the number of requests they service climbs,
> it generaly says that it's still worth preloading as much of the
> codebase as possible.
>=20
> So even though it's not a be-all end-all win, it's still a big win
> to preload.
>=20
> > Granted its not written for pas, but optimizing at the level you're
> > talking about (OS tuning) is certainly addressed.=20
> >=20
> > Whats the performance issue you're seeing thats prompting you to go dow=
n
> > this path to begin with? I'm just asking so I can understand where
> > you're coming from.
>=20
> I want the memory usage of our site to be lower.  For our dev envronment
> we've got 3 apache instances set up so we can run 3 applications and
> keep them seperate from each other.  All those children and all that
> memory adds up to swapping (the same box runs the 3 databases as well).
> Our production environment will soon be just like this.  Memory reduciton
> allows us to do more with less hardware.
>=20

So are you tuning for speed or memory size? Would it be unreasonable to
just cut down max requests per child? Keep httpd's from caching a ton of
content in the first place? There's also Apache::SizeLimit. You can
control how much shared/unshared memory each httpd can have. THat'll
help ya too.


> > I wish I had more time to help with this, its an interesting problem an=
d
> > there's some great tools to gather data......=20
>=20
>=20
> My conclusions from our conversation and the page you referenced are:
>   - at a minimum pre-compilation and pre-loading reduces the time=20
>     it takes a newly forked child to service requests for a given page.
>     In the non-pre-load scenario, the child may have to perform=20
>     PSP compliation, and it may have to perform Perl code compilation.
>   - at a minimum pre-loading helps share at least some memory, the
>     benefits are maximized if you keep the MaxRequestsPerChild down
>     to a reasonable level to flush memory that resulted in copy-on-write
>     because of member variables and per-request handling duties.
>=20
> So, overall, pre-compliation and pre-loading at Apache startup time are
> worth the effort of writing a new startup.pl - right?
>=20

Yup. I'd say load the most common pages, set a semi-restrictive
Apache::SizeLimit ( think I forgot to mention it before, but
Apache::SizeLimit is nice) in your startup.pl, and lower max requests.
Hopefully that'll get you out of swap in the short term. In the long
term it might be worht it to figure out a hybrid psp/html page. Where
large static content gets served off the disk but the dynamic bits are
in a module.....=20



--=20
Mental (Me...@Ne...)

"Shouting at people who, for one reason or another, cannot hear you=20
is mentally-ill behavior -- or evidence of idiots in command."

--George Smith, a virus researcher and columnist for SecurityFocus.=20

CARPE NOCTEM, QUAM MINIMUM CREDULA POSTERO.

GPG public key: http://www.neverlight.com/pas/Mental.asc

Re: [Pas-dev] Apache, mod_perl and Preloading?

[Kyle R . B. <mo...@vo...>]

> True, but there are some details you're forgetting. :) Even get/sets
> arent going to avoid the whole copy on write overhead. At least I dont
> think so.... and I dont have time right now to check.

The only benefit would be that the shared (compiled) code (the compiled
Perl code, the op trees) would be shared.  Per-child data segments (member
variables as you point out) would be subject to copy-on-write and diverge
once any 'set' had been invoked.  That's unavoidable and expected.

> My real question is... are you SURE that this will help? Wont execute()
> be different as far as variables go on a per process basis? There really
> isnt a way to force constants to use one page and variables to use
> another..... Even if the code starts out shared, most page objects will
> wind up being unshared simply by their nature. The lower, core modules
> are more likely to stay shared longer. So preloading and lowering
> maxrequests per child may help if done in tandem.

This should work for the same reason preloading core modules works --
code is not modified so it stays shared (no write, no copy-on-write)
data that never changes (like all those strings that represent the
bulk of the HTML in the psp files) will stay shared as well since
those stirngs are effectivly constant across the lifetime of the 
appilcation.

> Still, I havent spent too much time looking at the guts of apache memory
> management. I've found that simply by following the tuning guide and
> loading the common core modules performance is fine.

I don't think it's 100% Apache we're talking about here, the whole
copy-on-write feature is part of the process model on most unixes
for children that are the result of a fork (unless there is a subsequnt
exec call used to overwrite the process).

> I do concur we would benefit from a build spider to precompile all the
> psp's, but honestly StatINC is more overhead. 

I'm not sure I understandt his - StatINC is more overhead than what?

> Before you kill yourself doing this, check out this link:
> http://perl.apache.org/docs/1.0/guide/performance.html#Know_Your_Operating_System
> 
> I could just plagerize it, but I'd probably just explain it wrong ;)

Quoting that page:

  Do not forget that if you preload most of your code at server 
  startup, the newly forked child gets ready very fast, because 
  it inherits most of the preloaded code and the perl interpreter 
  from the parent process.

And additaionly that page says that even though children will grow
their memory usage as the number of requests they service climbs,
it generaly says that it's still worth preloading as much of the
codebase as possible.

So even though it's not a be-all end-all win, it's still a big win
to preload.

> Granted its not written for pas, but optimizing at the level you're
> talking about (OS tuning) is certainly addressed. 
> 
> Whats the performance issue you're seeing thats prompting you to go down
> this path to begin with? I'm just asking so I can understand where
> you're coming from.

I want the memory usage of our site to be lower.  For our dev envronment
we've got 3 apache instances set up so we can run 3 applications and
keep them seperate from each other.  All those children and all that
memory adds up to swapping (the same box runs the 3 databases as well).
Our production environment will soon be just like this.  Memory reduciton
allows us to do more with less hardware.

> I wish I had more time to help with this, its an interesting problem and
> there's some great tools to gather data...... 


My conclusions from our conversation and the page you referenced are:
  - at a minimum pre-compilation and pre-loading reduces the time 
    it takes a newly forked child to service requests for a given page.
    In the non-pre-load scenario, the child may have to perform 
    PSP compliation, and it may have to perform Perl code compilation.
  - at a minimum pre-loading helps share at least some memory, the
    benefits are maximized if you keep the MaxRequestsPerChild down
    to a reasonable level to flush memory that resulted in copy-on-write
    because of member variables and per-request handling duties.

So, overall, pre-compliation and pre-loading at Apache startup time are
worth the effort of writing a new startup.pl - right?


Kyle

-- 

------------------------------------------------------------------------------
Wisdom and Compassion are inseparable.
        -- Christmas Humphreys
mo...@vo...                            http://www.voicenet.com/~mortis
------------------------------------------------------------------------------

Re: [Pas-dev] Apache, mod_perl and Preloading?

[Mental P. <me...@ne...>]

On Fri, 2003-05-02 at 09:59, Kyle R . Burton wrote:
> For Apache/mod_perl tuning purposes, it's a good idea to add all of your=20
> modules to the startup.pl that gets loaded when Apache starts.  The usual=
=20
> benefits of this are speed (the first request into the server is just as=20
> fast as the second) and memory utilizaiton (anything loaded by Apache
> before the children fork is effectivly shared between the parent and the
> child processes resulting in a savings on the order of the number of
> Apache processes you have configured).

True, but there are some details you're forgetting. :) Even get/sets
arent going to avoid the whole copy on write overhead. At least I dont
think so.... and I dont have time right now to check.

> For page objects and other modules that you write by hand, this is an
> easy and straight forward thing to do, you just add the modules to
> the end of the startup.pl and off you go.
>=20

My real question is... are you SURE that this will help? Wont execute()
be different as far as variables go on a per process basis? There really
isnt a way to force constants to use one page and variables to use
another..... Even if the code starts out shared, most page objects will
wind up being unshared simply by their nature. The lower, core modules
are more likely to stay shared longer. So preloading and lowering
maxrequests per child may help if done in tandem.

Still, I havent spent too much time looking at the guts of apache memory
management. I've found that simply by following the tuning guide and
loading the common core modules performance is fine.

I do concur we would benefit from a build spider to precompile all the
psp's, but honestly StatINC is more overhead.=20

Before you kill yourself doing this, check out this link:
http://perl.apache.org/docs/1.0/guide/performance.html#Know_Your_Operating_=
System

I could just plagerize it, but I'd probably just explain it wrong ;)

Granted its not written for pas, but optimizing at the level you're
talking about (OS tuning) is certainly addressed.=20

Whats the performance issue you're seeing thats prompting you to go down
this path to begin with? I'm just asking so I can understand where
you're coming from.

I wish I had more time to help with this, its an interesting problem and
there's some great tools to gather data......=20



--=20
Mental (Me...@Ne...)

"Shouting at people who, for one reason or another, cannot hear you=20
is mentally-ill behavior -- or evidence of idiots in command."

--George Smith, a virus researcher and columnist for SecurityFocus.=20

CARPE NOCTEM, QUAM MINIMUM CREDULA POSTERO.

GPG public key: http://www.neverlight.com/pas/Mental.asc