openinteract-dev

[Openinteract-dev] lookup_action error

[<an...@io...>]

Hi!

CTX->lookup_action not just dies (which is ok) but logs an OpenInteract2 
error if the action is not found. I personally would not like it to log 
an error and not even give a warning - I would like it to be a debug or 
info message, but i might be misusing OI2. What do you think?

I have action xxyy which returns a datastructure in our package Y but 
package Y is not required for functional system.  Other packages which 
would use action xxyy lookup for that action and if it does not exist 
they just don't use the functionality, which is ok since the 
functionality was not required.

I have understood that actions are an abstraction layer provided for 
functionality which can be defined by any package, so i don't know what 
other mechanism i should use to detect if a functionality exists or not.

I did not make this a bug report since i'm not quite sure if this is the 
intended way of using OI2 actions.

- Antti

PS. It would probably be nice to have a "null" or "none" controller by 
default which would not let the action to execute when called directly 
from a request.. If this is a decent way to handle things..

Re: [Openinteract-dev] lookup_action error

[<an...@io...>]

Oh. I just noticed that there was a url_none which can be used to 
prevent action from being executed directly by a request. No need for a 
none-controller then :)

But I'm still not sure about the error on lookup failure..

- Antti

[Openinteract-dev] About the Action class

[<an...@io...>]

I have filed a improvement request in JIRA on some improvements but 
after some more concideration I'd like to have some discussion on the 
Action class in general..

Maybe my needs are a bit odd. I want to manually create an action so 
that I personally specify things like language and additional parameters 
in addition to action and task. I would want this action to respect the 
environment I give to it and not use anything found in CTX->request.

A quick look in the Action class seems to indicate that relatively few 
places in the action depend on CTX->request but they are vital parts 
like additional assigning, language handle (and thus $action->_msg), 
message passing from request, checking cache (because of an admin check 
and cache params in request) and fetching parameters from request.

I think that is would be good if as much of this dependency would be 
cleared out so that one can determine how an action is executed without 
having a tweaked CTX->request handy. I think that most (if not all) of 
the data needed could be assigned to various keys in the action upon 
it's creation.

Also after this, could it be possible to find out the task and 
additional parameters before execute?

Now they are all being gathered in execute and it is thus pretty hard to 
do any security checks, or anything else, before the determined task is 
executed. So when you create an Action, it is pretty much a black box 
and you can just fire execute off and see what happens, since you can't 
know which task it chooses to execute and thus which additional 
parameters it will have set..

So what do you think?

- Antti

[Openinteract-dev] Preloading of everything

[<an...@io...>]

Hello everyone :) It's been pretty quiet lately, but at least our 
project with OI2 has not stopped. I personally have grown to like OI2 
more and more as I use it and would like to start using it also on some 
smaller projects than Dicole. For smaller projects it is not however 
very easy to justify the amount OI2 requires memory and I would like to 
touch the issue of preloading everything at startup. There are at least 
two problems with preloading everything:

1. At the moment, OI2 is completely unusable as CGI because it takes 
several seconds on a fast machine to initialize Context fully. This is a 
shame since otherwise OI2 works with CGI as well as it does on mod_perl, 
FastCGI and the daemon.

2. Multiple installations of OI2 require that they are running on 
different Apaches, which means completely different perl interpreters, 
which means multiplication of all the code in memory. This could be 
acceptable if the memory footprint was not so huge and huge amounts of 
it were not polluted during the first request.

On single, heavily resourced installations the current approach works - 
and works very nicely. But with smaller resources, limited environment 
or multiple installations this might not always be the best option.

Thus I think that it would be nice to have an option not to fully 
initialize the Context but still maintain it fully functional on 
request. On many occasions trading speed for lower memory usage or 
faster startup time could be welcome.

It is at the moment already possible to easily leave some portions of 
Context uninitialized because this was needed to speed up oi2_manage, 
but at the moment this means also that some of the functionality is lost.

One major time & space hog is preloading of all actions. This is done 
for most actions already when the action table is assigned to CTX since 
we need to gather actions dispatch url data (and thus requiring them 
separately afterwards seems to be useless?). Could we gather this 
dispatch url data (or the whole action table?) separately before Context 
initialization and place it in a "cached" repository? Normal mod_perl 
server start could invoke this generation before startup, but for CGI 
this could be done by hand after each setup/override modifications or 
package installs. Action's init_at_startup() would probably also need to 
be rethought.

Other noteworthy task is preloading of SPOPS objects. In fact it seems 
that at the moment you can already skip InitializeSPOPS and things "just 
work" :) (I haven't tested this yet). This just leaves the gathering of 
all spops config data which could be handled similarily as the action table.

Also one could leave some languages unintialized even though this isn't 
_currently_ a major timesaver. It will be one in the future since I'm 
pretty sure we are going to have way over 10 different localizations 
eventually. This I have not investigated much but could guess that it 
would be possible to actually pregenerate the classes that OI2 now 
generates and evaluates at startup and require them only when needed.

So is there some fundamental problem that I have missed that makes some 
of this impossible? To me it seems that the OI2 infrastructure is so 
well built that this could be done with relatively small changes, but 
since I can't say that I see the real picture perfectly, I would welcome 
some input from Chris - at least in theoretical level ;)

When it comes to managing different types on initializations I think we 
could use the recently added dependency builder. Instead of running all 
setup actions by default, one could specify what setup actions are run 
and then everything that those packages are dependent on would also run. 
One could then craft different metapackages with dependencies for 
different types of startups.

What do you all think about this? Am I the only one who would like a 
"light weight" OI2?

- Antti

[Openinteract-dev] More on: Preloading of everything

[<an...@io...>]

After a good night sleep I came to the conclusion that maybe providing a 
way not to preload everything is not very urgent and it might be good to 
delay it for a 2.x release.

But to give you some idea of what kind of benefits you might get from 
not preloading everything, I did some poking around with a stick on Dicole.

By default the main apache thread takes up 61 Mb on startup.
Leaving languages (2 full and one paritial) uninitialized frees ~1 Mb.
Leaving SPOPS objects uninitialized frees ~3 Mb.
Leaving Actions uninitialized frees ~27 Mb.

If all of these are left uninitialized, the main apache thread takes up 
28,5 Mb. If the whole Context is left uninitialized, the main thread 
takes up 23,5 Mb, so only about 5 Mb of Context is not in these three parts.

I also noticed that building the Setup hierarchy does a require on all 
of the Setup classes and this, with their dependencies, seems to use 
something like 6 Mb of space - much of which might be possible to avoid 
and which is probably not used after startup.

I'm not really sure if OI2 will ever be really usable with CGI though. 
Even if the required initialization could be dropped somewhere close to 
25 Mb, after leaving the 7 mb of mod_perl apache, it still leaves 18 Mb 
of modules to be initialized on each request. But it would still be nice 
to have a bit smaller apache threads when having multiple OI2 
installations on the same machine and use CGI when developing, so that 
one would not have to restart apache after each change.

- Antti

Re: [Openinteract-dev] More on: Preloading of everything

[Chris W. <ch...@cw...>]

On Jun 19, 2005, at 11:01 AM, Antti M V=E4h=E4kotam=E4ki wrote:
> After a good night sleep I came to the conclusion that maybe =20
> providing a way not to preload everything is not very urgent and it =20=

> might be good to delay it for a 2.x release.

This has been my general thought as well. However... your points =20
about loading/creating actions at startup might be solved fairly =20
elegantly with a lazy loading scheme. For instance, in OI2 action we =20
have a name -> prototype mapping. Whenever we get a request for a new=20
() action we grab  the prototype, clone it, and assign all the =20
inbound parameters to it. (So new() actually acts like a factory, but =20=

it's hidden.)

There's nothing stopping us from replacing the prototype in that map =20
with a lazy loading routine. On the first access it will go through =20
the same functions currently in OI2::Setup::InitializeActions (which =20
is fairly minimal).

Doing this for SPOPS is much trickier (and probably undoable) because =20=

of the dependencies -- if an object contains other objects we need to =20=

make sure the contained is loaded, and if the contained contains =20
other objects we need to ensure THAT is loaded.... yuck.

> ...
> I'm not really sure if OI2 will ever be really usable with CGI =20
> though. Even if the required initialization could be dropped =20
> somewhere close to 25 Mb, after leaving the 7 mb of mod_perl =20
> apache, it still leaves 18 Mb of modules to be initialized on each =20
> request. But it would still be nice to have a bit smaller apache =20
> threads when having multiple OI2 installations on the same machine =20
> and use CGI when developing, so that one would not have to restart =20
> apache after each change.

I agree -- CGI compatibility is interesting for development (you =20
don't need to restart Apache with every change) and for small-traffic =20=

internal/individual sites can be usable. But not much more than that.

Regarding restarting Apache, this is very difficult due to SPOPS and =20
the fact that it generates code at runtime. There's been a =20
longstanding request to store the generated code into modules on the =20
filesystem and I think that would solve the difficult reload issue.

...then again, we do runtime initialization of other parts of the =20
system as well. For instance, the above action-creation stuff =20
wouldn't do the right thing if you modified the action class with =20
Apache::Reload because we'd still have the prototype stored...

...then again, since this is a development vs. production issue, =20
maybe we can make flaggable the ability to initialize actions at =20
startup, or ever. So every time you get a new() action it does all =20
the initialization stuff every time.  Mulling over it a bit more I =20
think this is a good option. Time to create a JIRA task...

Chris

--
Chris Winters (ch...@cw...)
Building enterprise-capable snack solutions since 1988.

Re: [Openinteract-dev] More on: Preloading of everything

[<an...@io...>]

Chris Winters wrote:
> There's nothing stopping us from replacing the prototype in that map  
> with a lazy loading routine. On the first access it will go through  the 
> same functions currently in OI2::Setup::InitializeActions (which  is 
> fairly minimal).

Could we instead follow the same route which is planned for SPOPS 
objects and generate each action in advance to a file? We would generate 
OpenInteract2/Action/Xxx/action_name.pm (and possibly others) which 
would inherit OpenInteract2::Action::Xxx and set all the properties and 
params in new().

I think this would make it easier to detect code changes without 
restarting the server - maybe as easily as by setting Apache::Reload as 
PerlInitHandler.

> Doing this for SPOPS is much trickier (and probably undoable) because  
> of the dependencies -- if an object contains other objects we need to  
> make sure the contained is loaded, and if the contained contains  other 
> objects we need to ensure THAT is loaded.... yuck.

Would this be easier if the SPOPS objects were generated to the 
filesystem and use/require all the objects it relates to?

> Regarding restarting Apache, this is very difficult due to SPOPS and  
> the fact that it generates code at runtime. There's been a  longstanding 
> request to store the generated code into modules on the  filesystem and 
> I think that would solve the difficult reload issue.

I couldn't find a jira issue for this.. Should this be scheduled for 
some releasse? What about the same for actions? And translation files?

If those can be stored in the filesystem, are we left with any code 
generation at startup? Does template handling do any?

Would it be a reasonable goal to get rid of _all_ code generation at 
startup and generate all the code to files? This generation could 
ofcourse be initiated also at the beginning of startup when in 
production, but we would have the option to do it beforehand and only 
paritially.

I know that on the fly code generation is not black magic ;) but it 
would also be easier to understand and inspect (and control) the flow of 
execution if %INC would be populated properly.

- Antti

[Openinteract-dev] More on: Preloading of everything

[<ant...@he...>]

After a good night sleep I came to the conclusion that maybe providing a 
way not to preload everything is not very urgent and it might be good to 
delay it for a 2.x release.

But to give you some idea of what kind of benefits you might get from 
not preloading everything, I did some poking around with a stick on Dicole.

By default the main apache thread takes up 61 Mb on startup.
Leaving languages (2 full and one paritial) uninitialized frees ~1 Mb.
Leaving SPOPS objects uninitialized frees ~3 Mb.
Leaving Actions uninitialized frees ~27 Mb.

If all of these are left uninitialized, the main apache thread takes up 
28,5 Mb. If the whole Context is left uninitialized, the main thread 
takes up 23,5 Mb, so only about 5 Mb of Context is not in these three parts.

I also noticed that building the Setup hierarchy does a require on all 
of the Setup classes and this, with their dependencies, seems to use 
something like 6 Mb of space - much of which might be possible to avoid 
and which is probably not used after startup.

I'm not really sure if OI2 will ever be really usable with CGI though. 
Even if the required initialization could be dropped somewhere close to 
25 Mb, after leaving the 7 mb of mod_perl apache, it still leaves 18 Mb 
of modules to be initialized on each request. But it would still be nice 
to have a bit smaller apache threads when having multiple OI2 
installations on the same machine and use CGI when developing, so that 
one would not have to restart apache after each change.

- Antti

Re: [Openinteract-dev] Preloading of everything

[Chris W. <ch...@cw...>]

On Jun 18, 2005, at 11:04 PM, Antti M V=E4h=E4kotam=E4ki wrote:
> Hello everyone :) It's been pretty quiet lately,

Sorry about that. I've been kind of hibernating -- lots of RealLife =20
stuff in the way, plus I have an interesting job now which eats up =20
more brain cycles.

I'll get into the preloading stuff in your follow-up...

Chris

--
Chris Winters (ch...@cw...)
Building enterprise-capable snack solutions since 1988.

[Openinteract-dev] CGI::Fast vs FCGI

[<an...@io...>]

Is there a reason why OI uses FCGI instead of CGI::Fast?

I started experimenting with FastCGI again and experienced some odd 
things like CGI not reseting its POST parameters. These problems seem to 
have disappeared when I switched to CGI::Fast.

My fastcgi loop is
   while ( my $c = CGI::Fast->new() ) {
and I create request:
   my $request  = OpenInteract2::Request->new( { cgi => $c } );

Interestingly OpenInteract2::Request::CGI already contains the code for 
this to work as it should.. So... What's going on? =)

- Antti

Re: [Openinteract-dev] CGI::Fast vs FCGI

[Chris W. <ch...@cw...>]

On Jul 3, 2005, at 8:03 PM, Antti M V=E4h=E4kotam=E4ki wrote:
> Is there a reason why OI uses FCGI instead of CGI::Fast?

I don't think there's a reason; I just wanted to get something =20
working quickly and figured if someone knew better they'd speak up.

I guess I was right :-)

> I started experimenting with FastCGI again and experienced some odd =20=

> things like CGI not reseting its POST parameters. These problems =20
> seem to have disappeared when I switched to CGI::Fast.
>
> My fastcgi loop is
>   while ( my $c =3D CGI::Fast->new() ) {
> and I create request:
>   my $request  =3D OpenInteract2::Request->new( { cgi =3D> $c } );

So the CGI::Fast object is a subclass of CGI.pm? Cool.

> Interestingly OpenInteract2::Request::CGI already contains the code =20=

> for this to work as it should.. So... What's going on? =3D)

Hey, if FastCGI works better with CGI::Fast I'm all for switching. =20
Are there any other side effects? And is CGI::Fast the more commonly =20
used interface? (I've never used FastCGI in production, so it's all a =20=

shot in the dark for me...)

Chris

--
Chris Winters (ch...@cw...)
Building enterprise-capable snack solutions since 1988.

Re: [Openinteract-dev] CGI::Fast vs FCGI

[<an...@io...>]

Chris Winters wrote:
> Hey, if FastCGI works better with CGI::Fast I'm all for switching.  Are 
> there any other side effects? And is CGI::Fast the more commonly  used 
> interface? (I've never used FastCGI in production, so it's all a  shot 
> in the dark for me...)

I haven't used it in production either.. I'm just running my development 
box under FastCGI now since I'm experiencing somekind of a memory 
consuming loop when I stop my apache when it has served an OI2 request 
under mod_perl (Which then causes wild swapping of _everything_).

CGI::Fast seems to be older than FCGI and not updated in a while, but 
that doesn't necessarily make it outdated. Maybe the motivation behind 
FCGI has been jut to provide a version of FastCGI gateway which is 
independent of the CGI.pm module, which some people concider bloated.

So pretty blind shots for me also, but for my use it seems that 
CGI::Fast works better. I haven't seen any side effects yet except that 
is fixes the post parameter problem.

.. By the way I think CGI request doesn't mix url parameters with post 
parameters like Apache request does. They can be acquired from CGI 
object with url_param() just like post params are acquired using 
param(). It might be nice to add some kind of a separation for post and 
url params in OI2 also (I think this is already in JIRA by Teemu?) and 
make the behaviour consistent across different Requests. Maybe so that 
all the params are stuffed to $request->param but they are also 
ackquirable intependently with $request->url_param and 
$request->post_param ?

- Antti

[Openinteract-dev] Using FastCGI as development plaftorm

[<an...@io...>]

I will post this also here since not everybody pays so close attention 
to JIRA ;)

--

After few days I can recommend to to use FastCGI instead of mod_perl as 
your development platform. The benefits:

* Initialization is not done twice as with (at least DSO) mod_perl.

* Stopping apache is instantaneous and doesn't cause a memory consuming 
loop as sometimes with mod_perl.

* All initialization is done on the first request after apache restart. 
This makes it possible to restart apache when you stop testing, so that 
you don't need to restart apache after you modify the code (just as with 
CGI.)

* After the first request everything works slightly faster than with 
mod_perl since only one process is running and thus there is no need to 
pollute huge amounts of memory on each request.

- Antti

PS. Doesn't this list accept CC'd posts or do they require 
administration approval? I think at least two mails are missing, one 
from me and one from Salve.

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[Perrin H. <pe...@el...>]

On Tue, 2005-07-05 at 17:06 +0300, Antti M V=C3=A4h=C3=A4kotam=C3=A4ki wrot=
e:
> * Initialization is not done twice as with (at least DSO) mod_perl.

I don't know anyone who compiles mod_perl as DSO.  There are clear
warnings not to do that.

> * All initialization is done on the first request after apache restart.=20
> This makes it possible to restart apache when you stop testing, so that=20
> you don't need to restart apache after you modify the code (just as with=20
> CGI.)

What does this mean?  That it picks up module changes?  Apache::Reload
does that too.

> * After the first request everything works slightly faster than with=20
> mod_perl since only one process is running and thus there is no need to=20
> pollute huge amounts of memory on each request.

You can easily tell apache how many processes it should run on your
development server.  A FastCGI server intended for production will
certainly run more than one process.

- Perrin

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[<an...@io...>]

Perrin Harkins wrote:
> I don't know anyone who compiles mod_perl as DSO.  There are clear
> warnings not to do that.

Oh that is probably true. I just checked and the Debian (in fact sadly 
Ubuntu at the moment) apache-perl package has Perl statically compiled 
instead of a DSO - and it _still_ did all of the initializations twice 
with OI2. You can verify if this applies to you by looking at the OI2 
log. The memory consuming loop in shutdown is also happening on the same 
server if OI2 has served at least one request.

> What does this mean?  That it picks up module changes?  Apache::Reload
> does that too.

I'm not sure if Apache::Reload works with OI2 very well since at least 
the Actions are cached & cloned by OI2. Also OI2 uses lots of 
configuration files which determine behaviour and autogenerate modules, 
which are naturally not reloaded by Apache::Reload. Anyway this is just 
speculation but I think I'm not far from the truth..

> You can easily tell apache how many processes it should run on your
> development server.  A FastCGI server intended for production will
> certainly run more than one process.

Well by development I did mean really just development, not production. 
We run OI2 with mod_perl on our production servers and continue to do so 
also in the future, but the beauty of OI2 is that it really doesn't 
matter which one of the options you use for development. At least for me 
FastCGI seems to speed up the development process quite a lot.. Or at 
least it feels like that ;)

Anyway this is not really an issue when developing since you barely even 
notice it (even though I have noticed almost doubled execution times on 
the first request of a child, but this might be affected by many other 
things also) - the other ones weight a lot more when choosing the 
platform for development.

But to clarify the situation I'm talking about: It doesn't matter how 
many child processes you use with Apache, they all are forked from the 
main process and share almost all of their static memory with the parent 
process until the memory becomes dirty and has to be copied to the 
child's own pool. So even if you would run only one child process, you 
would still have to copy lots of memory on the first request. Memory 
copying itself is not such an expensive task, but with large codebases 
and long execution paths like with OI2, the copying is done quite many 
separate times.

By the way if somebody wants to inspect the OI2 execution trail and how 
long each of the stages take, you can change conf/log4perl.conf to use 
DEBUG for log4perl.logger.OI2 and add SSS to the ConversionPattern in 
the end ( to %d{HH:mm:ss:SSS} ) to see millisecond timestamps on all log 
entries. All of the logging adds something like 20% to the execution 
time but you get a nice picture of how the execution is laid out in 
terms of time.

- Antti

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[Perrin H. <pe...@el...>]

On Tue, 2005-07-05 at 23:30 +0300, Antti M V=C3=A4h=C3=A4kotam=C3=A4ki wr=
ote:
> it _still_ did all of the initializations twice=20
> with OI2.

I haven't looked at how OI2's startup is organized, but this is probably
fixable.  Things called from PerlModule or PerlRequire will be run
twice, but modules loaded inside of that will not.

> I'm not sure if Apache::Reload works with OI2 very well since at least=20
> the Actions are cached & cloned by OI2.

It might not if there is a lot of stuff happening that %INC doesn't know
about.

I still don't understand what you're saying FastCGI does though.
Doesn't it also keep the interpreter alive in memory?

> But to clarify the situation I'm talking about: It doesn't matter how=20
> many child processes you use with Apache, they all are forked from the=20
> main process and share almost all of their static memory with the paren=
t=20
> process until the memory becomes dirty and has to be copied to the=20
> child's own pool. So even if you would run only one child process, you=20
> would still have to copy lots of memory on the first request.

No, it's just a fork.  Forking is very fast on linux and none of the
memory gets copied until the pages are dirty.  FastCGI forks too.

- Perrin

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[<ant...@he...>]

Perrin Harkins wrote:
>>it _still_ did all of the initializations twice 
>>with OI2.
> I haven't looked at how OI2's startup is organized, but this is probably
> fixable.  Things called from PerlModule or PerlRequire will be run
> twice, but modules loaded inside of that will not.

OI2 initializes a singleton Context object on startup (through 
PerlRequiring a startup.pl) which contains quite a lot of various data. 
I'm not quite sure how to prevent the Context from initializing twice 
but I suppose it could be fixed.

Hmm.. Could we just check if the singleton object already exists in the 
memory and not initialize it twice? Why didn't I think of this before? 
But well I'm not bugged by this anymore so those who are can try if it 
works ;)

> I still don't understand what you're saying FastCGI does though.
> Doesn't it also keep the interpreter alive in memory?

It doesn't do anything special. If I restart apache, the FastCGI server 
stops and it starts again only when I issue the next request for it. 
This way it gets fully initialized with the current data.

Since we can't use Apache::Reload (at least atm) this is what you would 
want it to do when developing for OI2.

At least for me it is a huge difference that I don't have to watch the 
mod_perl apache to start (which takes quite some time) and issue my 
request after it has finished - I can just issue the request and it is 
executed instantly when the initialization is completed.

> No, it's just a fork.  Forking is very fast on linux and none of the
> memory gets copied until the pages are dirty.

Yes, it is just a fork and the forking is really cheap but there is no 
way to prevent mod_perl from polluting also much of the static data 
shared with the parent process according to this:

http://perl.apache.org/docs/1.0/guide/performance.html#toc_Sharing_Memory

Since the execution path of OI2 is long, large amounts of the data gets 
dirty along the way.

> FastCGI forks too.

By default, no it doesn't. It's just a single Perl interpreter serving 
requests one by one.

- Antti

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[Perrin H. <pe...@el...>]

On Wed, 2005-07-06 at 00:41 +0300, Antti M V=C3=A4h=C3=A4kotam=C3=A4ki wrot=
e:
> Hmm.. Could we just check if the singleton object already exists in the=20
> memory and not initialize it twice?

Sounds reasonable to me.  Or you can put this stuff in a module that
gets used by startup.pl.

> At least for me it is a huge difference that I don't have to watch the=20
> mod_perl apache to start (which takes quite some time) and issue my=20
> request after it has finished - I can just issue the request and it is=20
> executed instantly when the initialization is completed.

But... they do the same thing.  I don't see how the FastCGI one can
actually be ready more quickly, unless it's just the bug you referred to
with reloading during startup.

> Yes, it is just a fork and the forking is really cheap but there is no=20
> way to prevent mod_perl from polluting also much of the static data=20
> shared with the parent process according to this:
>=20
> http://perl.apache.org/docs/1.0/guide/performance.html#toc_Sharing_Memory

I'm one of the maintainers of that page. :)

A noticeable hit on the first request is more likely to be due to making
a database connection or reopening log files than overhead from copying
dirty pages.

> By default, no it doesn't. It's just a single Perl interpreter serving=20
> requests one by one.

Then you can get the same effect in mod_perl by setting the process
controls in httpd.conf to run a single server, or by starting apache
with the -X flag which tells it not to fork.

If it's just for dev though, the oi2_daemon server would probably work
just as well.

- Perrin

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[<an...@io...>]

Perrin Harkins wrote:
> But... they do the same thing.  I don't see how the FastCGI one can
> actually be ready more quickly, unless it's just the bug you referred to
> with reloading during startup.

Well there is the bug of initializing multiple times on startup.

Then there is the bug of causing an infinite loop when Apache is stopped 
probably because of undefined subroutine calls. This on the other hand 
causes a lot of swapping, setting swappiness doesn't cut it, and if I 
turn of the swap, it sometimes causes my Linux to crash ;)

Then there is the problem (which is probably caused by the previous one) 
that you can't start apache right after stopping it without producing a 
failure, probably because some of the processes are still busy trying to 
consume a lot of memory.

And then there is the thing I _tried_ to describe ;) At least with the 
above problems and the large codebase of OI2, stopping and starting 
Apache takes a long time. You just have to stare at the prompt until it 
gets done. When it is done you change the workspace and refresh the 
page. The time which elapses between server start and a refreshed page 
varies, but is usually more than nothing. With FastCGI there is no lag 
between these things caused by a lazy coder. Also apache restart never 
fails when using FastCGI so I can bind it to a key, press it and presume 
that apache is restarted.

The last ones are small things, but believe me, they relieve you of a 
lot of frustration when you go through the procedure of reloading for 
testing _many_ times a day ;)

And yes - I think that all of the above are bugs in OI2 but I have no 
clue how hard they are to fix, and since they don't affect productio use 
that much and can be circumvented in development by usin FastCGI, I have 
more urgent tasks to tackle (like spending the whole day writing to 
oi-dev ;-))

> I'm one of the maintainers of that page. :)

Thanks for the great pages :)

> A noticeable hit on the first request is more likely to be due to making
> a database connection or reopening log files than overhead from copying
> dirty pages.

This is what I thought of first, since copying dirty memory doesn't seem 
like such a huge deal, but looking at the timed logs of OI requests got 
me thinking otherwise, since the slowdowns seemed to occur throughout 
the execution.. But I'm seriously beginning to doubt myself here (it was 
quite late then and it's 5.15 AM now ;)) so I'll check this again later.

> Then you can get the same effect in mod_perl by setting the process
> controls in httpd.conf to run a single server, or by starting apache
> with the -X flag which tells it not to fork.

By using FastCGI I can use the same apache to serve all the static 
requests (which are usually plenty when reloading a normal page), so 
just reducing the amount of Apache processes does not give the same 
result as easily. But this discussion is not going anywhere ;)

> If it's just for dev though, the oi2_daemon server would probably work
> just as well.

That is true, but it also does initialization on startup (and forks, if 
it matters ;-)). It might have also been a good choice to overcome the 
problems, but FastCGI seems to be even slightly better because of the 
initialization on request.

- Antti

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[Perrin H. <pe...@el...>]

Antti M Vähäkotamäki wrote:
> Then there is the bug of causing an infinite loop when Apache is stopped 
> probably because of undefined subroutine calls.

There's something wrong with your mod_perl build or some module you are 
using.  That isn't normal behavior, and it doesn't happen for me.

> This on the other hand 
> causes a lot of swapping, setting swappiness doesn't cut it, and if I 
> turn of the swap, it sometimes causes my Linux to crash ;)

None of that has ever happened for me when using mod_perl.

> Then there is the problem (which is probably caused by the previous one) 
> that you can't start apache right after stopping it without producing a 
> failure, probably because some of the processes are still busy trying to 
> consume a lot of memory.

Apache tries to do a graceful shutdown, meaning it tries to handle any 
connections that are currently operating.  If you don't want it to do 
that, you can kill it more harshly with a -9/-KILL.  Not a problem on a 
dev box where you are the only client.

> And then there is the thing I _tried_ to describe ;) At least with the 
> above problems and the large codebase of OI2, stopping and starting 
> Apache takes a long time. You just have to stare at the prompt until it 
> gets done.

Why do you have to stare at it?  Are you sitting and waiting for the 
stop before you run start?  I always do something like this on a dev 
machine:

apachectl stop && apachectl start

or use 'killall -KILL httpd' instead of apachectl stop.  You can put 
that in a file and bind it to a key, just like you did with FastCGI.

Since the code to be compiled and the setup to be created is identical 
with any of these server options, the total time for restarting should 
be the same.

> By using FastCGI I can use the same apache to serve all the static 
> requests (which are usually plenty when reloading a normal page)

That's true, your images would end up waiting if you used apache in -X 
mode.  I don't think avoiding forking will make things start noticeably 
faster anyway.

>> If it's just for dev though, the oi2_daemon server would probably work
>> just as well.
> 
> 
> That is true, but it also does initialization on startup (and forks, if 
> it matters ;-)). It might have also been a good choice to overcome the 
> problems, but FastCGI seems to be even slightly better because of the 
> initialization on request.

They all load the same perl code, do the same initialization, etc. so 
the total restart time should be about the same.

Some people (amazon.com, for example) prefer to use FastCGI over 
mod_perl, and that's fine.  I'm just objecting to the idea that it's 
somehow faster at compiling the same code than mod_perl is.

- Perrin

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[<an...@io...>]

Perrin Harkins wrote:
>> Then there is the bug of causing an infinite loop when Apache is 
>> stopped probably because of undefined subroutine calls.
> There's something wrong with your mod_perl build or some module you are 
> using.  That isn't normal behavior, and it doesn't happen for me.

I'm using debian's apache-perl. I think I't got to do with OI2 code and 
the behaviour reported wih undefined subroutine calls. It's documented 
here among other places:

http://www.axint.net/apache/perl/mod_perl-1.29/faq/mod_perl_faq.pod 
(find "Joel Wagner reports")

> Apache tries to do a graceful shutdown, meaning it tries to handle any 
> connections that are currently operating.  If you don't want it to do 
> that, you can kill it more harshly with a -9/-KILL.  Not a problem on a 
> dev box where you are the only client.

This is true. After mulling about this thing yesterday I came to the 
same conclusion that this would have helped.

> Why do you have to stare at it?

Because I need to know when apache has correctly started and initialized 
OI2 since before that I can't do my request. I need to stare it to know 
when I can continue. With FastCGI I can do the restart at any time after 
testing and it takes virtually no time to complete and never fails, so I 
don't need to continue my actions the same instant it is ready. You are 
correct that wil -9 kill I could bind apache stop & start also, but I 
would somehow have to be ready to act when it's ready, since the only 
time I can do the start part is after I have done all the editing and 
I'm ready to test.

So the bonus for me here is just the fact that with FastCGI I can press 
CTRL-R on my browser, which causes OI2 to initialize and does the 
request without any user intervention between them. I still have to 
stare at the browser window while OI2 is initializing ;)

This is the same thing why you do 'stop && start' and not issue 'stop', 
wait for it to complete, and then issue 'start' yourself ;)

> Since the code to be compiled and the setup to be created is identical 
> with any of these server options, the total time for restarting should 
> be the same.

If there weren't for the bugs mentioned, this would probably be very 
close to the reality.

> That's true, your images would end up waiting if you used apache in -X 
> mode.  I don't think avoiding forking will make things start noticeably 
> faster anyway.

Things starting faster was not the point. The point was the amount of 
time a forked child takes to execute it's first request, when compared 
to a single process serving the result. But this might not be a really 
noticeable issue anyway.

- Antti

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[<an...@io...>]

Antti M Vähäkotamäki wrote:
> Things starting faster was not the point. The point was the amount of 
> time a forked child takes to execute it's first request, when compared 
> to a single process serving the result. But this might not be a really 
> noticeable issue anyway.

I tested this again quickly and did not notice any notable difference, 
so it was probably just a side effect of all that swapping ;)

- Antti

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[Perrin H. <pe...@el...>]

On Wed, 2005-07-06 at 14:01 +0300, Antti M V=C3=A4h=C3=A4kotam=C3=A4ki wr=
ote:
> I'm using debian's apache-perl.

Not a good idea.  I recommend you build your own.  The apache builds
that come with Linux distros are notorious for causing problems.

> I think I't got to do with OI2 code and=20
> the behaviour reported wih undefined subroutine calls.

I've never seen that behavior, and it doesn't sound like what you
described (i.e. it doesn't happen at shutdown).  More likely you have
some bad XS code in some module you are using.

- Perrin

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[Chris W. <ch...@cw...>]

On Jul 5, 2005, at 5:41 PM, Antti M V=E4h=E4kotam=E4ki wrote:
> ...
> Hmm.. Could we just check if the singleton object already exists in =20=

> the memory and not initialize it twice? Why didn't I think of this =20
> before? But well I'm not bugged by this anymore so those who are =20
> can try if it works ;)

We already do this -- OI2::Context->create() will return an existing =20
context if it already exists.

Chris

--
Chris Winters (ch...@cw...)
Building enterprise-capable snack solutions since 1988.

Re: [Openinteract-dev] Double initialization

[<an...@io...>]

Chris Winters wrote:
> We already do this -- OI2::Context->create() will return an existing  
> context if it already exists.

Hmm.. Weird.. Are you experiencing double initialization using mod_perl 
like me and Salve? (It's discussed in JIRA under OIN-162)

- Antti