pas-dev

[Pas-dev] Apache, mod_perl and Preloading?

[Kyle R . B. <mo...@vo...>]

For Apache/mod_perl tuning purposes, it's a good idea to add all of your 
modules to the startup.pl that gets loaded when Apache starts.  The usual 
benefits of this are speed (the first request into the server is just as 
fast as the second) and memory utilizaiton (anything loaded by Apache
before the children fork is effectivly shared between the parent and the
child processes resulting in a savings on the order of the number of
Apache processes you have configured).

For page objects and other modules that you write by hand, this is an
easy and straight forward thing to do, you just add the modules to
the end of the startup.pl and off you go.

There are also modules generated from the psp files for your web-site.
Usualy these are generated on the fly in response to HTTP requests to
pages that are either not already compiled, or those that need 
recompliation.

If the compiled psp modules were pre-built (or built by the startup.pl), 
they could be loaded at startup time as well.  Unless I'm interpreting 
things incorrectly, that means that the entierity of your site's dynamic 
content and logic modules could be loaded at Apache startup.

Again, unless I'm mistaken, that type of situaiton should result in a
few benefits, 1: much better initial performance, 2: _significantly_ 
better memory usage by the web/application server.  This should still
fit in even if dynamic re-compliation were disabled (which should be 
the case for production environments).



Does this sound right?  Should we (I) look at implementing something
that can be invoked from startup.pl to do the pre-compliaiton and
pre-loading?  It'd slow down the web-server startup, but if you're
codebase is largely built already it shouldn't be that much of a
hit.  The implicaitons (performance and memory usage) for the final
run-time seem like they'd be more than worth-while.


Feedback?


Kyle

    


-- 

------------------------------------------------------------------------------
Wisdom and Compassion are inseparable.
        -- Christmas Humphreys
mo...@vo...                            http://www.voicenet.com/~mortis
------------------------------------------------------------------------------

Re: [Pas-dev] Apache, mod_perl and Preloading?

[Mental P. <me...@ne...>]

On Fri, 2003-05-02 at 09:59, Kyle R . Burton wrote:
> For Apache/mod_perl tuning purposes, it's a good idea to add all of your=20
> modules to the startup.pl that gets loaded when Apache starts.  The usual=
=20
> benefits of this are speed (the first request into the server is just as=20
> fast as the second) and memory utilizaiton (anything loaded by Apache
> before the children fork is effectivly shared between the parent and the
> child processes resulting in a savings on the order of the number of
> Apache processes you have configured).

True, but there are some details you're forgetting. :) Even get/sets
arent going to avoid the whole copy on write overhead. At least I dont
think so.... and I dont have time right now to check.

> For page objects and other modules that you write by hand, this is an
> easy and straight forward thing to do, you just add the modules to
> the end of the startup.pl and off you go.
>=20

My real question is... are you SURE that this will help? Wont execute()
be different as far as variables go on a per process basis? There really
isnt a way to force constants to use one page and variables to use
another..... Even if the code starts out shared, most page objects will
wind up being unshared simply by their nature. The lower, core modules
are more likely to stay shared longer. So preloading and lowering
maxrequests per child may help if done in tandem.

Still, I havent spent too much time looking at the guts of apache memory
management. I've found that simply by following the tuning guide and
loading the common core modules performance is fine.

I do concur we would benefit from a build spider to precompile all the
psp's, but honestly StatINC is more overhead.=20

Before you kill yourself doing this, check out this link:
http://perl.apache.org/docs/1.0/guide/performance.html#Know_Your_Operating_=
System

I could just plagerize it, but I'd probably just explain it wrong ;)

Granted its not written for pas, but optimizing at the level you're
talking about (OS tuning) is certainly addressed.=20

Whats the performance issue you're seeing thats prompting you to go down
this path to begin with? I'm just asking so I can understand where
you're coming from.

I wish I had more time to help with this, its an interesting problem and
there's some great tools to gather data......=20



--=20
Mental (Me...@Ne...)

"Shouting at people who, for one reason or another, cannot hear you=20
is mentally-ill behavior -- or evidence of idiots in command."

--George Smith, a virus researcher and columnist for SecurityFocus.=20

CARPE NOCTEM, QUAM MINIMUM CREDULA POSTERO.

GPG public key: http://www.neverlight.com/pas/Mental.asc

Re: [Pas-dev] Apache, mod_perl and Preloading?

[Kyle R . B. <mo...@vo...>]

> True, but there are some details you're forgetting. :) Even get/sets
> arent going to avoid the whole copy on write overhead. At least I dont
> think so.... and I dont have time right now to check.

The only benefit would be that the shared (compiled) code (the compiled
Perl code, the op trees) would be shared.  Per-child data segments (member
variables as you point out) would be subject to copy-on-write and diverge
once any 'set' had been invoked.  That's unavoidable and expected.

> My real question is... are you SURE that this will help? Wont execute()
> be different as far as variables go on a per process basis? There really
> isnt a way to force constants to use one page and variables to use
> another..... Even if the code starts out shared, most page objects will
> wind up being unshared simply by their nature. The lower, core modules
> are more likely to stay shared longer. So preloading and lowering
> maxrequests per child may help if done in tandem.

This should work for the same reason preloading core modules works --
code is not modified so it stays shared (no write, no copy-on-write)
data that never changes (like all those strings that represent the
bulk of the HTML in the psp files) will stay shared as well since
those stirngs are effectivly constant across the lifetime of the 
appilcation.

> Still, I havent spent too much time looking at the guts of apache memory
> management. I've found that simply by following the tuning guide and
> loading the common core modules performance is fine.

I don't think it's 100% Apache we're talking about here, the whole
copy-on-write feature is part of the process model on most unixes
for children that are the result of a fork (unless there is a subsequnt
exec call used to overwrite the process).

> I do concur we would benefit from a build spider to precompile all the
> psp's, but honestly StatINC is more overhead. 

I'm not sure I understandt his - StatINC is more overhead than what?

> Before you kill yourself doing this, check out this link:
> http://perl.apache.org/docs/1.0/guide/performance.html#Know_Your_Operating_System
> 
> I could just plagerize it, but I'd probably just explain it wrong ;)

Quoting that page:

  Do not forget that if you preload most of your code at server 
  startup, the newly forked child gets ready very fast, because 
  it inherits most of the preloaded code and the perl interpreter 
  from the parent process.

And additaionly that page says that even though children will grow
their memory usage as the number of requests they service climbs,
it generaly says that it's still worth preloading as much of the
codebase as possible.

So even though it's not a be-all end-all win, it's still a big win
to preload.

> Granted its not written for pas, but optimizing at the level you're
> talking about (OS tuning) is certainly addressed. 
> 
> Whats the performance issue you're seeing thats prompting you to go down
> this path to begin with? I'm just asking so I can understand where
> you're coming from.

I want the memory usage of our site to be lower.  For our dev envronment
we've got 3 apache instances set up so we can run 3 applications and
keep them seperate from each other.  All those children and all that
memory adds up to swapping (the same box runs the 3 databases as well).
Our production environment will soon be just like this.  Memory reduciton
allows us to do more with less hardware.

> I wish I had more time to help with this, its an interesting problem and
> there's some great tools to gather data...... 


My conclusions from our conversation and the page you referenced are:
  - at a minimum pre-compilation and pre-loading reduces the time 
    it takes a newly forked child to service requests for a given page.
    In the non-pre-load scenario, the child may have to perform 
    PSP compliation, and it may have to perform Perl code compilation.
  - at a minimum pre-loading helps share at least some memory, the
    benefits are maximized if you keep the MaxRequestsPerChild down
    to a reasonable level to flush memory that resulted in copy-on-write
    because of member variables and per-request handling duties.

So, overall, pre-compliation and pre-loading at Apache startup time are
worth the effort of writing a new startup.pl - right?


Kyle

-- 

------------------------------------------------------------------------------
Wisdom and Compassion are inseparable.
        -- Christmas Humphreys
mo...@vo...                            http://www.voicenet.com/~mortis
------------------------------------------------------------------------------

Re: [Pas-dev] Apache, mod_perl and Preloading?

[Mental P. <me...@ne...>]

On Fri, 2003-05-02 at 11:03, Kyle R . Burton wrote:
> > True, but there are some details you're forgetting. :) Even get/sets
> > arent going to avoid the whole copy on write overhead. At least I dont
> > think so.... and I dont have time right now to check.
>=20
> The only benefit would be that the shared (compiled) code (the compiled
> Perl code, the op trees) would be shared.  Per-child data segments (membe=
r
> variables as you point out) would be subject to copy-on-write and diverge
> once any 'set' had been invoked.  That's unavoidable and expected.
>=20

Exactly.... so if you have a ton of content.. this is a win.

This is also why Embperl didnt cache content. It only caches the code so
that lots of html doesnt mean lots of memory. Serving static content of
the disk is still fast.....=20

> > My real question is... are you SURE that this will help? Wont execute()
> > be different as far as variables go on a per process basis? There reall=
y
> > isnt a way to force constants to use one page and variables to use
> > another..... Even if the code starts out shared, most page objects will
> > wind up being unshared simply by their nature. The lower, core modules
> > are more likely to stay shared longer. So preloading and lowering
> > maxrequests per child may help if done in tandem.
>=20
> This should work for the same reason preloading core modules works --
> code is not modified so it stays shared (no write, no copy-on-write)
> data that never changes (like all those strings that represent the
> bulk of the HTML in the psp files) will stay shared as well since
> those stirngs are effectivly constant across the lifetime of the=20
> appilcation.

> > Still, I havent spent too much time looking at the guts of apache memor=
y
> > management. I've found that simply by following the tuning guide and
> > loading the common core modules performance is fine.
>=20
> I don't think it's 100% Apache we're talking about here, the whole
> copy-on-write feature is part of the process model on most unixes
> for children that are the result of a fork (unless there is a subsequnt
> exec call used to overwrite the process).
>=20
> > I do concur we would benefit from a build spider to precompile all the
> > psp's, but honestly StatINC is more overhead.=20
>=20
> I'm not sure I understandt his - StatINC is more overhead than what?
>=20

Sorry. Thats what happens when you word salad between other tings....
what I meant was that turning off StatINC fairly significant perfomance
boost in and of itself.=20


> > Before you kill yourself doing this, check out this link:
> > http://perl.apache.org/docs/1.0/guide/performance.html#Know_Your_Operat=
ing_System
> >=20
> > I could just plagerize it, but I'd probably just explain it wrong ;)
>=20
> Quoting that page:
>=20
>   Do not forget that if you preload most of your code at server=20
>   startup, the newly forked child gets ready very fast, because=20
>   it inherits most of the preloaded code and the perl interpreter=20
>   from the parent process.
>=20
> And additaionly that page says that even though children will grow
> their memory usage as the number of requests they service climbs,
> it generaly says that it's still worth preloading as much of the
> codebase as possible.
>=20
> So even though it's not a be-all end-all win, it's still a big win
> to preload.
>=20
> > Granted its not written for pas, but optimizing at the level you're
> > talking about (OS tuning) is certainly addressed.=20
> >=20
> > Whats the performance issue you're seeing thats prompting you to go dow=
n
> > this path to begin with? I'm just asking so I can understand where
> > you're coming from.
>=20
> I want the memory usage of our site to be lower.  For our dev envronment
> we've got 3 apache instances set up so we can run 3 applications and
> keep them seperate from each other.  All those children and all that
> memory adds up to swapping (the same box runs the 3 databases as well).
> Our production environment will soon be just like this.  Memory reduciton
> allows us to do more with less hardware.
>=20

So are you tuning for speed or memory size? Would it be unreasonable to
just cut down max requests per child? Keep httpd's from caching a ton of
content in the first place? There's also Apache::SizeLimit. You can
control how much shared/unshared memory each httpd can have. THat'll
help ya too.


> > I wish I had more time to help with this, its an interesting problem an=
d
> > there's some great tools to gather data......=20
>=20
>=20
> My conclusions from our conversation and the page you referenced are:
>   - at a minimum pre-compilation and pre-loading reduces the time=20
>     it takes a newly forked child to service requests for a given page.
>     In the non-pre-load scenario, the child may have to perform=20
>     PSP compliation, and it may have to perform Perl code compilation.
>   - at a minimum pre-loading helps share at least some memory, the
>     benefits are maximized if you keep the MaxRequestsPerChild down
>     to a reasonable level to flush memory that resulted in copy-on-write
>     because of member variables and per-request handling duties.
>=20
> So, overall, pre-compliation and pre-loading at Apache startup time are
> worth the effort of writing a new startup.pl - right?
>=20

Yup. I'd say load the most common pages, set a semi-restrictive
Apache::SizeLimit ( think I forgot to mention it before, but
Apache::SizeLimit is nice) in your startup.pl, and lower max requests.
Hopefully that'll get you out of swap in the short term. In the long
term it might be worht it to figure out a hybrid psp/html page. Where
large static content gets served off the disk but the dynamic bits are
in a module.....=20



--=20
Mental (Me...@Ne...)

"Shouting at people who, for one reason or another, cannot hear you=20
is mentally-ill behavior -- or evidence of idiots in command."

--George Smith, a virus researcher and columnist for SecurityFocus.=20

CARPE NOCTEM, QUAM MINIMUM CREDULA POSTERO.

GPG public key: http://www.neverlight.com/pas/Mental.asc

Re: [Pas-dev] Apache, mod_perl and Preloading?

[Kyle R . B. <mo...@vo...>]

> Exactly.... so if you have a ton of content.. this is a win.

Are not PSP files mostly content?  Or at least commonly mostly content?

> This is also why Embperl didnt cache content. It only caches the code so
> that lots of html doesnt mean lots of memory. Serving static content of
> the disk is still fast..... 

Embperl didn't cache the template file?  It only cached the code?  What
did it do with the content break it up into multiple little files and
serve them from disk or something?

Pas/PSP creates code out of the template file, so the content becomes
intertwined with the code...

> Sorry. Thats what happens when you word salad between other tings....
> what I meant was that turning off StatINC fairly significant perfomance
> boost in and of itself. 

Ah.  So pre-compilation and disabling dynamic-recompliation and StatInc
should be _good_ things?

> > I want the memory usage of our site to be lower.  For our dev envronment
> > we've got 3 apache instances set up so we can run 3 applications and
> > keep them seperate from each other.  All those children and all that
> > memory adds up to swapping (the same box runs the 3 databases as well).
> > Our production environment will soon be just like this.  Memory reduciton
> > allows us to do more with less hardware.
> > 
> 
> So are you tuning for speed or memory size? Would it be unreasonable to
> just cut down max requests per child? Keep httpd's from caching a ton of
> content in the first place? There's also Apache::SizeLimit. You can
> control how much shared/unshared memory each httpd can have. THat'll
> help ya too.

Wouldn't the pre-compile/pre-load provide both a performance boost and
a memory reduction (for the child processes)?  There is an obvious 
overhead associated with pre-loading infrequently hit parts of the 
site/codebase, but unless the bulk of your site is mostly infrequently 
hit stuff, then pre-loading should be a win (both cpu and memory) right?

For sites that are either huge, or have vast expanses of infrequently
used PSPs/Page objects, then preloading might be worse - but if that
is the case, then maybe there are other issues at work.


> > My conclusions from our conversation and the page you referenced are:
> >   - at a minimum pre-compilation and pre-loading reduces the time 
> >     it takes a newly forked child to service requests for a given page.
> >     In the non-pre-load scenario, the child may have to perform 
> >     PSP compliation, and it may have to perform Perl code compilation.
> >   - at a minimum pre-loading helps share at least some memory, the
> >     benefits are maximized if you keep the MaxRequestsPerChild down
> >     to a reasonable level to flush memory that resulted in copy-on-write
> >     because of member variables and per-request handling duties.
> > 
> > So, overall, pre-compliation and pre-loading at Apache startup time are
> > worth the effort of writing a new startup.pl - right?
> 
> Yup. I'd say load the most common pages, set a semi-restrictive
> Apache::SizeLimit ( think I forgot to mention it before, but
> Apache::SizeLimit is nice) in your startup.pl, and lower max requests.
> Hopefully that'll get you out of swap in the short term. In the long
> term it might be worht it to figure out a hybrid psp/html page. Where
> large static content gets served off the disk but the dynamic bits are
> in a module..... 

Hrm...that's an interesting possibility.  Combiend with earlier
proposals of compile-time (psp compile time) expansion of psp code
into static content, that could provide more winnage...

Thanks for the discussion, I'm in the process of writing an alternate
startup.pl that I'll check in later today.

Kyle



-- 

------------------------------------------------------------------------------
Wisdom and Compassion are inseparable.
        -- Christmas Humphreys
mo...@vo...                            http://www.voicenet.com/~mortis
------------------------------------------------------------------------------

Re: [Pas-dev] Apache, mod_perl and Preloading?

[Kyle R . B. <mo...@vo...>]

> > Are not PSP files mostly content?  Or at least commonly mostly content?
> 
> Well, yes. But most of the psp's I've written are small. So the memory
> usage was reasonable. A psp with a bunch of huge tables with a ton of
> content could get out of hand quickly. I can see why you're going down the
> preload path. 

Even with smaller psps, the code they compile into should still be shared.

> > Ah.  So pre-compilation and disabling dynamic-recompliation and StatInc
> > should be _good_ things?
> 
> Yes. Very good things. 

Good.

> > Wouldn't the pre-compile/pre-load provide both a performance boost and
> > a memory reduction (for the child processes)?  There is an obvious 
> > overhead associated with pre-loading infrequently hit parts of the 
> > site/codebase, but unless the bulk of your site is mostly infrequently 
> > hit stuff, then pre-loading should be a win (both cpu and memory) right?
> 
> THis is where it gets interresting. There's really one way to tell.
> Benchmarks :)

Yes.  I don't have the time or resources to do proper benchmarks right
now though.

> > Hrm...that's an interesting possibility.  Combiend with earlier
> > proposals of compile-time (psp compile time) expansion of psp code
> > into static content, that could provide more winnage...
> 
> Well, yes. Sort of. Depending.... I have a funny feeling we're about to
> reinvent SSI :)

Nope.  This has nothing to do with SSI or it's aproach IMO.



> Cool, let me know how it goes. 

I just checked in a build-startup.pl that uses File::Find on the
pas.documentRoot in the configuration file, and compiles all psp files
that don't start with an underscore (that's just a convention that
we've been using for included psp files, which might not result
in a valid .pm file after compilaion).

If your codebase compiles properly, that code compmiles every psp
file at Apache startup and pre-loads it.  I need to move the 
conditional recompile check logic out of the request handler and
into the psp compiler...so it can get reused by the build-startup.pl


Other than that, it looks like it works.

One thing is that the .pm files generated from the .psp files end 
up being owned by root.

Kyle

-- 

------------------------------------------------------------------------------
Wisdom and Compassion are inseparable.
        -- Christmas Humphreys
mo...@vo...                            http://www.voicenet.com/~mortis
------------------------------------------------------------------------------

Re: [Pas-dev] Apache, mod_perl and Preloading?

[Kyle R . B. <mo...@vo...>]

Some notes on the new build-startup.pl:

  - Top reports SIZE of 50482, and SHARED of 32244, this represents 63%
    (after hitting most of the pages in the site)
  - The page generation/load times seems faster.  I haven't benchmarked
    this, but the responsiveness is better.

Some of this is because dynamicrecompile is turned off so those checks
are not being done at runtime.

Kyle

-- 

------------------------------------------------------------------------------
Wisdom and Compassion are inseparable.
        -- Christmas Humphreys
mo...@vo...                            http://www.voicenet.com/~mortis
------------------------------------------------------------------------------