openinteract-dev — For OI developers, includes project coordination and technical discussion.

Re: [Openinteract-dev] More on: Preloading of everything

[<an...@io...>]

Chris Winters wrote:
> There's nothing stopping us from replacing the prototype in that map  
> with a lazy loading routine. On the first access it will go through  the 
> same functions currently in OI2::Setup::InitializeActions (which  is 
> fairly minimal).

Could we instead follow the same route which is planned for SPOPS 
objects and generate each action in advance to a file? We would generate 
OpenInteract2/Action/Xxx/action_name.pm (and possibly others) which 
would inherit OpenInteract2::Action::Xxx and set all the properties and 
params in new().

I think this would make it easier to detect code changes without 
restarting the server - maybe as easily as by setting Apache::Reload as 
PerlInitHandler.

> Doing this for SPOPS is much trickier (and probably undoable) because  
> of the dependencies -- if an object contains other objects we need to  
> make sure the contained is loaded, and if the contained contains  other 
> objects we need to ensure THAT is loaded.... yuck.

Would this be easier if the SPOPS objects were generated to the 
filesystem and use/require all the objects it relates to?

> Regarding restarting Apache, this is very difficult due to SPOPS and  
> the fact that it generates code at runtime. There's been a  longstanding 
> request to store the generated code into modules on the  filesystem and 
> I think that would solve the difficult reload issue.

I couldn't find a jira issue for this.. Should this be scheduled for 
some releasse? What about the same for actions? And translation files?

If those can be stored in the filesystem, are we left with any code 
generation at startup? Does template handling do any?

Would it be a reasonable goal to get rid of _all_ code generation at 
startup and generate all the code to files? This generation could 
ofcourse be initiated also at the beginning of startup when in 
production, but we would have the option to do it beforehand and only 
paritially.

I know that on the fly code generation is not black magic ;) but it 
would also be easier to understand and inspect (and control) the flow of 
execution if %INC would be populated properly.

- Antti

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[Perrin H. <pe...@el...>]

On Thu, 2005-07-07 at 16:43 +1200, Sam Vilain wrote:
> In any case, a statement like that is pretty much handwaving.  Doing a
> custom build will merely give you a different set of problems to deal
> with.  I wouldn't knock the binary packages so much just on principle.

It's handwaving from experience.  The Bricolage project tried pretty
hard to make the Red Hat binaries of mod_perl work, and finally had to
give up because users kept on having mysterious problems that couldn't
be reproduced by developers.  We get the same thing on the mod_perl list
-- strange problem, gone when the user switches to their own build.

The issues probably stem from the fact that packagers for distributions
have to try to make a server that works for running every conceivable
project all at once.  They build in a lot more, or use DSO, or both.
It's not so different from the situation with the perl packages most
distros provide.  The one from Red Hat has threading and debugging
compiled in, which makes it about 15% slower than a perl built on the
same OS with all the defaults.

- Perrin

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[Sam V. <sa...@vi...>]

Perrin Harkins wrote:
>>I'm using debian's apache-perl.
> Not a good idea.  I recommend you build your own.  The apache builds
> that come with Linux distros are notorious for causing problems.

Of course, much better if you're using FastCGI mode is to scrap Apache
entirely, and just use a nice little web server like lighttpd.

In any case, a statement like that is pretty much handwaving.  Doing a
custom build will merely give you a different set of problems to deal
with.  I wouldn't knock the binary packages so much just on principle.

Sam.

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[Perrin H. <pe...@el...>]

On Wed, 2005-07-06 at 14:01 +0300, Antti M V=C3=A4h=C3=A4kotam=C3=A4ki wr=
ote:
> I'm using debian's apache-perl.

Not a good idea.  I recommend you build your own.  The apache builds
that come with Linux distros are notorious for causing problems.

> I think I't got to do with OI2 code and=20
> the behaviour reported wih undefined subroutine calls.

I've never seen that behavior, and it doesn't sound like what you
described (i.e. it doesn't happen at shutdown).  More likely you have
some bad XS code in some module you are using.

- Perrin

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[<an...@io...>]

Antti M Vähäkotamäki wrote:
> Things starting faster was not the point. The point was the amount of 
> time a forked child takes to execute it's first request, when compared 
> to a single process serving the result. But this might not be a really 
> noticeable issue anyway.

I tested this again quickly and did not notice any notable difference, 
so it was probably just a side effect of all that swapping ;)

- Antti

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[<an...@io...>]

Perrin Harkins wrote:
>> Then there is the bug of causing an infinite loop when Apache is 
>> stopped probably because of undefined subroutine calls.
> There's something wrong with your mod_perl build or some module you are 
> using.  That isn't normal behavior, and it doesn't happen for me.

I'm using debian's apache-perl. I think I't got to do with OI2 code and 
the behaviour reported wih undefined subroutine calls. It's documented 
here among other places:

http://www.axint.net/apache/perl/mod_perl-1.29/faq/mod_perl_faq.pod 
(find "Joel Wagner reports")

> Apache tries to do a graceful shutdown, meaning it tries to handle any 
> connections that are currently operating.  If you don't want it to do 
> that, you can kill it more harshly with a -9/-KILL.  Not a problem on a 
> dev box where you are the only client.

This is true. After mulling about this thing yesterday I came to the 
same conclusion that this would have helped.

> Why do you have to stare at it?

Because I need to know when apache has correctly started and initialized 
OI2 since before that I can't do my request. I need to stare it to know 
when I can continue. With FastCGI I can do the restart at any time after 
testing and it takes virtually no time to complete and never fails, so I 
don't need to continue my actions the same instant it is ready. You are 
correct that wil -9 kill I could bind apache stop & start also, but I 
would somehow have to be ready to act when it's ready, since the only 
time I can do the start part is after I have done all the editing and 
I'm ready to test.

So the bonus for me here is just the fact that with FastCGI I can press 
CTRL-R on my browser, which causes OI2 to initialize and does the 
request without any user intervention between them. I still have to 
stare at the browser window while OI2 is initializing ;)

This is the same thing why you do 'stop && start' and not issue 'stop', 
wait for it to complete, and then issue 'start' yourself ;)

> Since the code to be compiled and the setup to be created is identical 
> with any of these server options, the total time for restarting should 
> be the same.

If there weren't for the bugs mentioned, this would probably be very 
close to the reality.

> That's true, your images would end up waiting if you used apache in -X 
> mode.  I don't think avoiding forking will make things start noticeably 
> faster anyway.

Things starting faster was not the point. The point was the amount of 
time a forked child takes to execute it's first request, when compared 
to a single process serving the result. But this might not be a really 
noticeable issue anyway.

- Antti

Re: [Openinteract-dev] OI2 status from Perrin's talk

[Chris W. <ch...@cw...>]

On Jul 5, 2005, at 11:32 PM, Perrin Harkins wrote:
> ...
> Although it sounds like a silly difference to highlight, it's  
> somewhat important becaise of the strong anti-config file entiment  
> of the Catalyst crew.

I don't think it's silly at all. (Sorry if I gave that impression.)  
It's one of those things people care deeply about and is therefore a  
pretty useful discriminator, even if it's something you can get used  
to if you need to -- kind of like odd programming language syntax.

Chris

--
Chris Winters (ch...@cw...)
Building enterprise-capable snack solutions since 1988.

Re: [Openinteract-dev] OI2 status from Perrin's talk

[Perrin H. <pe...@el...>]

Chris Winters wrote:
> -- one of his comments was: "If you don't like  configuration files, you 
> won't like OpenInteract2". Tough to argue  with that...

Although it sounds like a silly difference to highlight, it's somewhat 
important becaise of the strong anti-config file entiment of the 
Catalyst crew.

- Perrin

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[Perrin H. <pe...@el...>]

Antti M Vähäkotamäki wrote:
> Then there is the bug of causing an infinite loop when Apache is stopped 
> probably because of undefined subroutine calls.

There's something wrong with your mod_perl build or some module you are 
using.  That isn't normal behavior, and it doesn't happen for me.

> This on the other hand 
> causes a lot of swapping, setting swappiness doesn't cut it, and if I 
> turn of the swap, it sometimes causes my Linux to crash ;)

None of that has ever happened for me when using mod_perl.

> Then there is the problem (which is probably caused by the previous one) 
> that you can't start apache right after stopping it without producing a 
> failure, probably because some of the processes are still busy trying to 
> consume a lot of memory.

Apache tries to do a graceful shutdown, meaning it tries to handle any 
connections that are currently operating.  If you don't want it to do 
that, you can kill it more harshly with a -9/-KILL.  Not a problem on a 
dev box where you are the only client.

> And then there is the thing I _tried_ to describe ;) At least with the 
> above problems and the large codebase of OI2, stopping and starting 
> Apache takes a long time. You just have to stare at the prompt until it 
> gets done.

Why do you have to stare at it?  Are you sitting and waiting for the 
stop before you run start?  I always do something like this on a dev 
machine:

apachectl stop && apachectl start

or use 'killall -KILL httpd' instead of apachectl stop.  You can put 
that in a file and bind it to a key, just like you did with FastCGI.

Since the code to be compiled and the setup to be created is identical 
with any of these server options, the total time for restarting should 
be the same.

> By using FastCGI I can use the same apache to serve all the static 
> requests (which are usually plenty when reloading a normal page)

That's true, your images would end up waiting if you used apache in -X 
mode.  I don't think avoiding forking will make things start noticeably 
faster anyway.

>> If it's just for dev though, the oi2_daemon server would probably work
>> just as well.
> 
> 
> That is true, but it also does initialization on startup (and forks, if 
> it matters ;-)). It might have also been a good choice to overcome the 
> problems, but FastCGI seems to be even slightly better because of the 
> initialization on request.

They all load the same perl code, do the same initialization, etc. so 
the total restart time should be about the same.

Some people (amazon.com, for example) prefer to use FastCGI over 
mod_perl, and that's fine.  I'm just objecting to the idea that it's 
somehow faster at compiling the same code than mod_perl is.

- Perrin

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[<an...@io...>]

Perrin Harkins wrote:
> But... they do the same thing.  I don't see how the FastCGI one can
> actually be ready more quickly, unless it's just the bug you referred to
> with reloading during startup.

Well there is the bug of initializing multiple times on startup.

Then there is the bug of causing an infinite loop when Apache is stopped 
probably because of undefined subroutine calls. This on the other hand 
causes a lot of swapping, setting swappiness doesn't cut it, and if I 
turn of the swap, it sometimes causes my Linux to crash ;)

Then there is the problem (which is probably caused by the previous one) 
that you can't start apache right after stopping it without producing a 
failure, probably because some of the processes are still busy trying to 
consume a lot of memory.

And then there is the thing I _tried_ to describe ;) At least with the 
above problems and the large codebase of OI2, stopping and starting 
Apache takes a long time. You just have to stare at the prompt until it 
gets done. When it is done you change the workspace and refresh the 
page. The time which elapses between server start and a refreshed page 
varies, but is usually more than nothing. With FastCGI there is no lag 
between these things caused by a lazy coder. Also apache restart never 
fails when using FastCGI so I can bind it to a key, press it and presume 
that apache is restarted.

The last ones are small things, but believe me, they relieve you of a 
lot of frustration when you go through the procedure of reloading for 
testing _many_ times a day ;)

And yes - I think that all of the above are bugs in OI2 but I have no 
clue how hard they are to fix, and since they don't affect productio use 
that much and can be circumvented in development by usin FastCGI, I have 
more urgent tasks to tackle (like spending the whole day writing to 
oi-dev ;-))

> I'm one of the maintainers of that page. :)

Thanks for the great pages :)

> A noticeable hit on the first request is more likely to be due to making
> a database connection or reopening log files than overhead from copying
> dirty pages.

This is what I thought of first, since copying dirty memory doesn't seem 
like such a huge deal, but looking at the timed logs of OI requests got 
me thinking otherwise, since the slowdowns seemed to occur throughout 
the execution.. But I'm seriously beginning to doubt myself here (it was 
quite late then and it's 5.15 AM now ;)) so I'll check this again later.

> Then you can get the same effect in mod_perl by setting the process
> controls in httpd.conf to run a single server, or by starting apache
> with the -X flag which tells it not to fork.

By using FastCGI I can use the same apache to serve all the static 
requests (which are usually plenty when reloading a normal page), so 
just reducing the amount of Apache processes does not give the same 
result as easily. But this discussion is not going anywhere ;)

> If it's just for dev though, the oi2_daemon server would probably work
> just as well.

That is true, but it also does initialization on startup (and forks, if 
it matters ;-)). It might have also been a good choice to overcome the 
problems, but FastCGI seems to be even slightly better because of the 
initialization on request.

- Antti

Re: [Openinteract-dev] Double initialization

[<an...@io...>]

Chris Winters wrote:
> We already do this -- OI2::Context->create() will return an existing  
> context if it already exists.

Hmm.. Weird.. Are you experiencing double initialization using mod_perl 
like me and Salve? (It's discussed in JIRA under OIN-162)

- Antti

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[Chris W. <ch...@cw...>]

On Jul 5, 2005, at 5:41 PM, Antti M V=E4h=E4kotam=E4ki wrote:
> ...
> Hmm.. Could we just check if the singleton object already exists in =20=

> the memory and not initialize it twice? Why didn't I think of this =20
> before? But well I'm not bugged by this anymore so those who are =20
> can try if it works ;)

We already do this -- OI2::Context->create() will return an existing =20
context if it already exists.

Chris

--
Chris Winters (ch...@cw...)
Building enterprise-capable snack solutions since 1988.

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[Perrin H. <pe...@el...>]

On Wed, 2005-07-06 at 00:41 +0300, Antti M V=C3=A4h=C3=A4kotam=C3=A4ki wrot=
e:
> Hmm.. Could we just check if the singleton object already exists in the=20
> memory and not initialize it twice?

Sounds reasonable to me.  Or you can put this stuff in a module that
gets used by startup.pl.

> At least for me it is a huge difference that I don't have to watch the=20
> mod_perl apache to start (which takes quite some time) and issue my=20
> request after it has finished - I can just issue the request and it is=20
> executed instantly when the initialization is completed.

But... they do the same thing.  I don't see how the FastCGI one can
actually be ready more quickly, unless it's just the bug you referred to
with reloading during startup.

> Yes, it is just a fork and the forking is really cheap but there is no=20
> way to prevent mod_perl from polluting also much of the static data=20
> shared with the parent process according to this:
>=20
> http://perl.apache.org/docs/1.0/guide/performance.html#toc_Sharing_Memory

I'm one of the maintainers of that page. :)

A noticeable hit on the first request is more likely to be due to making
a database connection or reopening log files than overhead from copying
dirty pages.

> By default, no it doesn't. It's just a single Perl interpreter serving=20
> requests one by one.

Then you can get the same effect in mod_perl by setting the process
controls in httpd.conf to run a single server, or by starting apache
with the -X flag which tells it not to fork.

If it's just for dev though, the oi2_daemon server would probably work
just as well.

- Perrin

Re: [Openinteract-dev] OI2 status from Perrin's talk

[Perrin H. <pe...@el...>]

On Wed, 2005-07-06 at 03:04 +0300, Antti M V=C3=A4h=C3=A4kotam=C3=A4ki wrot=
e:
> So Perrin, might it be possible to get your OSCON slides/material on the=20
> web after you are done with the presentation, and preferably a small=20
> notification on this mailing list?

The YAPC slides and samples are up:
http://wiki.yapctoronto.org/index.cgi?PresentationSlides

I will be redoing them for OSCON, so I can make sure to fit all three in
more evenly.

- Perrin

Re: [Openinteract-dev] OI2 status from Perrin's talk

[<an...@io...>]

Since I've been writing mails to OI-dev all day, I might as well continue ;)

Chris Winters wrote:
> He'll be giving this talk again at OSCON in early August, so I'd like  
> to get the next beta out before then if possible, if only so the  
> Class::DBI interface can be released.

It would be really nice to have some kind of a comparsion publicly 
available in the web so taht one could see where things stand, since 
most of the people in the world will not be attending OSCON.

I personally haven't had much time to glance around after we started 
with OI, so it would be nice if somebody else would do the job for me.. 
And I probably couldn't be very objective since I've gradually started 
to think more and more the way OI works ;)

So Perrin, might it be possible to get your OSCON slides/material on the 
web after you are done with the presentation, and preferably a small 
notification on this mailing list?

- Antti

[Openinteract-dev] OI2 status from Perrin's talk

[Chris W. <ch...@cw...>]

You might be interested: I went to Perrin's YAPC::NA talk [1] on MVC  
web frameworks; the frameworks under discussion were OI2, Catalyst  
and CGI::Application. OI2 came out looking decent, if a little  
complicated -- one of his comments was: "If you don't like  
configuration files, you won't like OpenInteract2". Tough to argue  
with that...

We did get props for tackling messy problems like i18n, having lots  
of documentation, and for having a decent auto-generated CRUD  
application. Fortunately, time ran out before Perrin could get to OI2  
negatives :-)

He'll be giving this talk again at OSCON in early August, so I'd like  
to get the next beta out before then if possible, if only so the  
Class::DBI interface can be released.

Chris

[1] http://yapc.org/America/schedule-2005/summary.html#49

--
Chris Winters (ch...@cw...)
Building enterprise-capable snack solutions since 1988.

Re: [Openinteract-dev] About the Action class

[<an...@io...>]

A new try - last versio of this mail disappeared :(

Chris Winters wrote:
> That's probably a good idea. I moved the code to assign these from  the 
> request into a separate subroutine as well, so you can do:
> 
>   # assign the additional parameters from the Request object
>   $action->url_additional_param_from_request;
>   my @values = url_additional_param;

What if all of the assigning would be done already in the actionresolver?

This way you wouldn't need to store tha url params in request at all. 
You would pass _only_ the relative url to an actionresolver and the 
actionresolver would return you an action which has task, language, 
params from additional and params from request.

Language and request params could be fetched using action's interface so 
that if they are not set for the action, they will be fetched from 
CTX->request, but if they are set, the set values would be used instead.

I think this would not only add new possibilities by making the action 
truly independent fron the request, but it would also simplify the 
execution process.


> It's not a bad idea. (That Action class just keeps growing... :-)

Well if you can move task validation and additional parameter populating 
  from execute to actionresolver, it wouldn't be a huge addition for the 
Action to add the possibility to make action run independently from the 
request but still using similiar ways to control actions behaviour.

This would make it possible to do a gateways that would make it possible 
to call normal requests but get the results in a filtered form 
(returning pages in ajax-readable form for example).

Also this would make it possible to build a public gateway so that if 
requests are called through this gateway, they would send correct HTTP 
headers so that for example squid could timecache them, but the requests 
could be called by others normally getting the latest data.

Like this:

/public/request/myaction/mytask/myadditional/

sub request {
   my ( $self ) = @_;

   my $sub_action = MyActionResolver->resolve(
     join '/', $self->param('other_additional')
   );

   # maybe set some extra params
   # maybe override language or request params
   # issue correct cache headers

   return $sub_action->execute;
}


> Creating a shortcut to fill the URL_PARAMS with default values, or to  
> be smarter to associate some additional parameters with named values  
> (like your 'area_id' above) seems application-specific, and may be  more 
> appropriate a base Action class in your app.

Well generally you are correct - if you don't know how the urls are 
parsed (since any of the actionresolvers might get it), you don't know 
how they are generated either. This is probably only for us then.

- Antti

Re: [Openinteract-dev] What is OpenInteract2 ?

[Teemu A. <te...@di...>]

>> Well what I was thinking was more in the lines of OI2 not being even partly a CMS.
>
> I'm sure you can mold OI2 into whatever you want it to be... :-)

I believe that for a project to become _the_ option for a certain
target audience, it should state it's mission and roadmap pretty clearly. This builds up
confidence in the platform and provides an answer to the most important question: ok if we start
to use it now, how is it going to improve and in which direction?

Answering, that it can be anything you want it to be is pretty confusing.

Let's revisit openinteract.org first paragraph. Let's call it the mission statement:

"OpenInteract is a web application server written in Perl. It features integrated data
persistence, security, user and group management, plus an easy way to create and distribute fully
database-independent applications."

I might also emphasize that it cuts the development time for implementing most of the things you
need when writing web applications. Less code, more consistency, extensibility, modularity and
easier tracking of code revisions.

So if it's intended as a base for web application development, then why on earth does it require
you to install an CMS you might have no use for? (never mind a news, comments and a whats new
package). If you want to get rid of these, skipping the installation of the SYSTEM package bundle
excludes not only the CMS function (page) but also all the necessary modules that are required for
OI2 to function.

I understand the importance of functional examples like an CMS. Developers should have something
to play with to come up with ideas how to use the platform and how to improve it.

As such, I believe these should be optional packages, serving as one of the examples how OI2 can
be used.

As OI2 consumes so much memory and dependencies already, I believe there is no reason to "pollute"
the base installation with higher level functionality you might never need when developing a web
application, say an address book. It's simply just useless if you are not writing a CMS but
something totally different.

This is certainly a limit in the application framework if it's considered to be an application
framework. It forces you to run code in the application stack you might never need.

In my opinion, for the system to function mainly as a web application framework, it should invest
development time mainly on that statement. In my ideal scenario, when you install the OI2 base, it
would greet you with a welcome message, the documentation package and clear instructions on how to
install example functionality like the CMS, news and comments packages to play with.

All the other functionality, like security, user/group management, UI widgets etc. are very
important for web application development. Higher or more specific implementations should be
optional.

Although the lower level functionality is important, sometimes these also do not serve the
purposes. We replaced/extended most of the user/group/security stuff in our OI2 application called
Dicole (demo at http://dicole.net) because doing so served our purposes. For example, in
securities we needed more detailed access rights other than action/task level security and SPOPS
object security so we replaced the security framework and disabled the OI2 implementation. We
could have extended it if given more thought, but afterall, I'm all thumbs up for such low level
built-in functionality in the base packages.

> They're there so you can get a notion of what's possible to do OI2, and when you've seen _that_,
the _real_ big questions show up: How can I use OI2? How can I improve OI2? What should be
improved? How easy is it to improve OI2? How can I make it easier to improve OI2? What should be
improved so OI2 becomes easier to improve?

These questions can be answered if the base installation greets you as a web application framework
and not as a generic CMS.

Despite some things I'm not happy with, OI2 is still the best decision we have made when we looked
for a new base to build upon. We have done more during this time for our customers than we could
have done without. The design of the web application framework is brilliant and I'm happy to help
it to be even better.

Regards,

Teemu Arina
Dicole

Re: [Openinteract-dev] What is OpenInteract2 ?

[Teemu A. <tee...@di...>]

>> Well what I was thinking was more in the lines of OI2 not being even
>> partly a CMS.
>
> I'm sure you can mold OI2 into whatever you want it to be... :-)

I believe that for a project to become _the_ option for a certain
target audience, it should state it's mission and roadmap pretty clearly.
This builds up confidence in the platform and provides an answer to the most
important question: ok if we start to use it now, how is it going to improve
and in which direction?

Answering, that it can be anything you want it to be is pretty confusing.

Let's revisit openinteract.org first paragraph. Let's call it the mission
statement:

"OpenInteract is a web application server written in Perl. It features integrated data
persistence, security, user and group management, plus an easy way to create and distribute fully
database-independent applications."

I might also emphasize that it cuts the development time for implementing most of the things you
need when writing web applications. Less code, more consistency, extensibility, modularity and
easier tracking of code revisions.

So if it's intended as a base for web application development, then why on earth
does it require you to install an CMS you might have no use for? (never mind a news, comments
and a whats new package). If you want to get rid of these, skipping the installation of the SYSTEM
package bundle excludes not only the CMS function (page) but also all the necessary modules that are
required for OI2 to function.

I understand the importance of functional examples like an CMS. Developers
should have something to play with to come up with ideas how to use the platform and
how to improve it.

As such, I believe these should be optional packages, serving as one of the examples
how OI2 can be used.

As OI2 consumes so much memory and dependencies already, I believe there is no reason
to "pollute" the base installation with higher level functionality you might
never need when developing a web application, say an address book. It's simply
just useless if you are not writing a CMS but something totally different.

This is certainly a limit in the application framework if it's considered to be an
application framework. It forces you to run code in the application stack you might never need.

In my opinion, for the system to function mainly as a web application framework, it should
invest development time mainly on that statement. In my ideal scenario, when you install
the OI2 base, it would greet you with a welcome message, the documentation package and clear
instructions on how to install example functionality like the CMS, news and comments packages
to play with.

All the other functionality, like security, user/group management, UI widgets etc.
are very important for web application development. Higher or more specific implementations
should be optional.

Although the lower level functionality is important, sometimes these also do not serve
the purposes. We replaced/extended most of the user/group/security stuff in our OI2 application
called Dicole (demo at http://dicole.net) because doing so served our purposes. For example, in
securities we needed more detailed access rights other than action/task level security and SPOPS
object security so we replaced the security framework and disabled the OI2 implementation. We could
have extended it if given more thought, but afterall, I'm all thumbs up for such low level built-in
functionality in the base packages.

> They're there so you can get a notion of what's possible to do OI2, and when
> you've seen _that_, the _real_ big questions show up: How can I use OI2? How
> can I improve OI2? What should be improved? How easy is it to improve OI2? How
> can I make it easier to improve OI2? What should be improved so OI2 becomes
> easier to improve?

These questions can be answered if the base installation greets you as a web application
framework and not as a generic CMS.

Despite some things I'm not happy with, OI2 is still the best decision we have made when
we looked for a new base to build upon. We have done more during this time for our customers
than we could have done without. The design of the web application framework
is brilliant and I'm happy to help it to be even better.

Regards,

Teemu Arina
Dicole

Re: [Openinteract-dev] What is OpenInteract2 ?

[<an...@io...>]

I'm sorry to be quoting this quite extensively since the former posts 
did never make it to this list.

Salve J Nilsen wrote:
> Antti M Vähäkotamäki wrote:
>> Well to make this clear from the beginning - I'm not in any way 
>> against you building new templates. They will be needed for sure. I'm 
>> just trying to look at the bigger picture.
>>
>> Salve J Nilsen wrote:
>>
>>> Urgh... I'm sorry about that. OI2 is definitly a application framwork 
>>> (where of "CMS" is only a minor part.) I didn't mean to cause this 
>>> confusion. :-\
>>
>> Well what I was thinking was more in the lines of OI2 not being even 
>> partly a CMS.
> 
> I'm sure you can mold OI2 into whatever you want it to be... :-)

Well I just don't see that the products being developed on OI2 as a part 
of OI2. They run on OI2, they can be used to promote OI2, but they are 
not part of OI2.


>> Sure I would heartly welcome a package which would implement all kinds 
>> of CMS functionality on top of OI2, and an another package which would 
>> provide a bucketfull of general TT2 templates which are easily 
>> themable by pure CSS, but I'm not sure if either of these should be 
>> included in the base packages IF they are to provide a general base 
>> for applications.
> 
> I think having a functional website with CMS, news and security features as
> part of the "base" is not only good, but the Sensible Thing to Do. OI2's 
> CMS
> features (e.g. page management and publishing, user/group management, 
> interface
> consistency features, templates, logging framework, etc.) may be a bit 
> lacking
> (in consistency, standards compliance, visual beauty and so on.) but having
> them available from a basic install quite important. Who do you think 
> would be
> impressed by a application framwork one can't test and experiment with?

Again, I'm not saying that developing this stuff is bad and it shouldn't 
be done. As you state, it's vital that there exists some ready code to 
experiment with and to reuse when needed, BUT making this code a part of 
OI2 base is not in any way vital for making it experimentable or reusable.

One could argue that it probably actually hinders the reuse value of 
base packages that they are distributed as bulk and thus their 
interfaces never need to be polished for reusability.

When you use OI2 in production, you probably do not want those OI2 base 
modules present which you don't use since they take lot's of space in 
memory when the memory consumption is alrealy huge. And since it is hard 
to separate them or use them selectively, you are usually better off 
with just replacing the needed parts alltogether.


>> So probably the bigger question here is the meaning of base packages - 
>> what are they for?
> 
> They're there so you can get a notion of what's possible to do OI2, and 
> when
> you've seen _that_, the _real_ big questions show up: How can I use OI2? 
> How
> can I improve OI2? What should be improved? How easy is it to improve 
> OI2? How
> can I make it easier to improve OI2? What should be improved so OI2 becomes
> easier to improve?

I don't think that every user who writes his application on OI2 should 
be an OI2 developer. In fact I think that OI2 developers' main goal 
should be that none of OI2's users should need to be OI2 developers.

I do agree with you that at the moment we aren't quite there yet ;-) but 
it doesn't change the goal.


>> Also currently the building block TT2 templates for OI2 website aren't 
>> even in a base package but they are directly in the website's template 
>> directory. This makes them impossible to be substituted or modified 
>> through the management system with packages. I'm not sure if the 
>> current theme implementation can be used to override these (we are 
>> using it only to load a single theme which causes a constant 50ms 
>> addition to each request ;-) ), but if we are getting rid of the 
>> current theming for a CSS only theming, we can't do this anymore and 
>> the whole thing should be rethought.
> 
> I'm sure you can find or formulate some bug reports that describe what your
> problems are. <http://jira.openinteract.org/> ;-)
> 
> (And yes, the theme implementation sucks :)

Well in fact I'm trying to formulate it by doing this discussion. I have 
already filed a JIRA issue for a possibility to create a clean OI2 
website wihout the base packages. Chris asked me if this was necessary 
and I'm trying to figure it here also myself by discussing the issue ;)

- Antti

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[<ant...@he...>]

Perrin Harkins wrote:
>>it _still_ did all of the initializations twice 
>>with OI2.
> I haven't looked at how OI2's startup is organized, but this is probably
> fixable.  Things called from PerlModule or PerlRequire will be run
> twice, but modules loaded inside of that will not.

OI2 initializes a singleton Context object on startup (through 
PerlRequiring a startup.pl) which contains quite a lot of various data. 
I'm not quite sure how to prevent the Context from initializing twice 
but I suppose it could be fixed.

Hmm.. Could we just check if the singleton object already exists in the 
memory and not initialize it twice? Why didn't I think of this before? 
But well I'm not bugged by this anymore so those who are can try if it 
works ;)

> I still don't understand what you're saying FastCGI does though.
> Doesn't it also keep the interpreter alive in memory?

It doesn't do anything special. If I restart apache, the FastCGI server 
stops and it starts again only when I issue the next request for it. 
This way it gets fully initialized with the current data.

Since we can't use Apache::Reload (at least atm) this is what you would 
want it to do when developing for OI2.

At least for me it is a huge difference that I don't have to watch the 
mod_perl apache to start (which takes quite some time) and issue my 
request after it has finished - I can just issue the request and it is 
executed instantly when the initialization is completed.

> No, it's just a fork.  Forking is very fast on linux and none of the
> memory gets copied until the pages are dirty.

Yes, it is just a fork and the forking is really cheap but there is no 
way to prevent mod_perl from polluting also much of the static data 
shared with the parent process according to this:

http://perl.apache.org/docs/1.0/guide/performance.html#toc_Sharing_Memory

Since the execution path of OI2 is long, large amounts of the data gets 
dirty along the way.

> FastCGI forks too.

By default, no it doesn't. It's just a single Perl interpreter serving 
requests one by one.

- Antti

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[Perrin H. <pe...@el...>]

On Tue, 2005-07-05 at 23:30 +0300, Antti M V=C3=A4h=C3=A4kotam=C3=A4ki wr=
ote:
> it _still_ did all of the initializations twice=20
> with OI2.

I haven't looked at how OI2's startup is organized, but this is probably
fixable.  Things called from PerlModule or PerlRequire will be run
twice, but modules loaded inside of that will not.

> I'm not sure if Apache::Reload works with OI2 very well since at least=20
> the Actions are cached & cloned by OI2.

It might not if there is a lot of stuff happening that %INC doesn't know
about.

I still don't understand what you're saying FastCGI does though.
Doesn't it also keep the interpreter alive in memory?

> But to clarify the situation I'm talking about: It doesn't matter how=20
> many child processes you use with Apache, they all are forked from the=20
> main process and share almost all of their static memory with the paren=
t=20
> process until the memory becomes dirty and has to be copied to the=20
> child's own pool. So even if you would run only one child process, you=20
> would still have to copy lots of memory on the first request.

No, it's just a fork.  Forking is very fast on linux and none of the
memory gets copied until the pages are dirty.  FastCGI forks too.

- Perrin

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[<an...@io...>]

Perrin Harkins wrote:
> I don't know anyone who compiles mod_perl as DSO.  There are clear
> warnings not to do that.

Oh that is probably true. I just checked and the Debian (in fact sadly 
Ubuntu at the moment) apache-perl package has Perl statically compiled 
instead of a DSO - and it _still_ did all of the initializations twice 
with OI2. You can verify if this applies to you by looking at the OI2 
log. The memory consuming loop in shutdown is also happening on the same 
server if OI2 has served at least one request.

> What does this mean?  That it picks up module changes?  Apache::Reload
> does that too.

I'm not sure if Apache::Reload works with OI2 very well since at least 
the Actions are cached & cloned by OI2. Also OI2 uses lots of 
configuration files which determine behaviour and autogenerate modules, 
which are naturally not reloaded by Apache::Reload. Anyway this is just 
speculation but I think I'm not far from the truth..

> You can easily tell apache how many processes it should run on your
> development server.  A FastCGI server intended for production will
> certainly run more than one process.

Well by development I did mean really just development, not production. 
We run OI2 with mod_perl on our production servers and continue to do so 
also in the future, but the beauty of OI2 is that it really doesn't 
matter which one of the options you use for development. At least for me 
FastCGI seems to speed up the development process quite a lot.. Or at 
least it feels like that ;)

Anyway this is not really an issue when developing since you barely even 
notice it (even though I have noticed almost doubled execution times on 
the first request of a child, but this might be affected by many other 
things also) - the other ones weight a lot more when choosing the 
platform for development.

But to clarify the situation I'm talking about: It doesn't matter how 
many child processes you use with Apache, they all are forked from the 
main process and share almost all of their static memory with the parent 
process until the memory becomes dirty and has to be copied to the 
child's own pool. So even if you would run only one child process, you 
would still have to copy lots of memory on the first request. Memory 
copying itself is not such an expensive task, but with large codebases 
and long execution paths like with OI2, the copying is done quite many 
separate times.

By the way if somebody wants to inspect the OI2 execution trail and how 
long each of the stages take, you can change conf/log4perl.conf to use 
DEBUG for log4perl.logger.OI2 and add SSS to the ConversionPattern in 
the end ( to %d{HH:mm:ss:SSS} ) to see millisecond timestamps on all log 
entries. All of the logging adds something like 20% to the execution 
time but you get a nice picture of how the execution is laid out in 
terms of time.

- Antti

Re: [Openinteract-dev] Using FastCGI as development plaftorm

[Perrin H. <pe...@el...>]

On Tue, 2005-07-05 at 17:06 +0300, Antti M V=C3=A4h=C3=A4kotam=C3=A4ki wrot=
e:
> * Initialization is not done twice as with (at least DSO) mod_perl.

I don't know anyone who compiles mod_perl as DSO.  There are clear
warnings not to do that.

> * All initialization is done on the first request after apache restart.=20
> This makes it possible to restart apache when you stop testing, so that=20
> you don't need to restart apache after you modify the code (just as with=20
> CGI.)

What does this mean?  That it picks up module changes?  Apache::Reload
does that too.

> * After the first request everything works slightly faster than with=20
> mod_perl since only one process is running and thus there is no need to=20
> pollute huge amounts of memory on each request.

You can easily tell apache how many processes it should run on your
development server.  A FastCGI server intended for production will
certainly run more than one process.

- Perrin

[Openinteract-dev] Using FastCGI as development plaftorm

[<an...@io...>]

I will post this also here since not everybody pays so close attention 
to JIRA ;)

--

After few days I can recommend to to use FastCGI instead of mod_perl as 
your development platform. The benefits:

* Initialization is not done twice as with (at least DSO) mod_perl.

* Stopping apache is instantaneous and doesn't cause a memory consuming 
loop as sometimes with mod_perl.

* All initialization is done on the first request after apache restart. 
This makes it possible to restart apache when you stop testing, so that 
you don't need to restart apache after you modify the code (just as with 
CGI.)

* After the first request everything works slightly faster than with 
mod_perl since only one process is running and thus there is no need to 
pollute huge amounts of memory on each request.

- Antti

PS. Doesn't this list accept CC'd posts or do they require 
administration approval? I think at least two mails are missing, one 
from me and one from Salve.