Webware for Python

On Fri, 2002-04-26 at 12:43, Magnus Lie Hetland wrote:
> Terrel Shumway <tshumway@...>:
> >
> > The code in CGIWrapper/CGIWrapper.py hasn't been touched in 10 months
> > (and then only to fix a minor bug). It seems to be implementing
> > MiscUtils.Configurable from before it was factored out into a separate
> > class. It may not be broken, but my feeling is that bit rot has
> > definitely set in.
> > 
> > Is anyone using it? Does anyone care?
> 
> This is the one that lets you use the standard CGI module through
> webware, for increased efficiency -- right? I've been thinking about
> using it -- I think it would be a shame if it went away... 
No, what you are thinking of is CGIAdapter, which is used by WebKit.cgi
to connect with AppServer. I would guess that *most* people use that,
until they figure out mod_webkit. It will definitely not go away.

> (Although I guess fcgi etc. are alternatives.)
mod_webkit works well. I am still trying to figure out fcgi.


CGIWrapper is actually a precursor to what we know as WebKit. AFAICT it
is totally obsolete. (The reason I even asked is because I didn't
clearly understand what it was. If I had understood, I would have just
sent a private message to Chuck and not bothered the list?) 

Am I right Chuck? Should it be removed from the distribution to avoid
silly questions like mine?

Terrel Shumway <tshumway@...>:
[snip]
> > This is the one that lets you use the standard CGI module through
> > webware, for increased efficiency -- right? I've been thinking about
> > using it -- I think it would be a shame if it went away... 
> No, what you are thinking of is CGIAdapter, which is used by WebKit.cgi
> to connect with AppServer.

Am I? What I'm thinking about is the thing that lets you run
completely ordinary CGI scripts with webware... (Not servlet stuff.)

[snip]
> CGIWrapper is actually a precursor to what we know as WebKit. AFAICT it
> is totally obsolete. (The reason I even asked is because I didn't
> clearly understand what it was. If I had understood, I would have just
> sent a private message to Chuck and not bothered the list?) 

:)

> Am I right Chuck? Should it be removed from the distribution to avoid
> silly questions like mine?

--
Magnus Lie Hetland                                  The Anygui Project
http://hetland.org                                  http://anygui.org

On Friday April 26, 2002 06:24 pm, Magnus Lie Hetland wrote:
> Terrel Shumway <tshumway@...>:
> [snip]
>
> > > This is the one that lets you use the standard CGI module through
> > > webware, for increased efficiency -- right? I've been thinking about
> > > using it -- I think it would be a shame if it went away...
> >
> > No, what you are thinking of is CGIAdapter, which is used by WebKit.cgi
> > to connect with AppServer.
>
> Am I? What I'm thinking about is the thing that lets you run
> completely ordinary CGI scripts with webware... (Not servlet stuff.)

I don't think such a thing exists.  Maybe you're thinking of mod_python, 
which has a mode which is intended to accelerate CGI scripts.

>
> [snip]
>
> > CGIWrapper is actually a precursor to what we know as WebKit. AFAICT it
> > is totally obsolete. (The reason I even asked is because I didn't
> > clearly understand what it was. If I had understood, I would have just
> > sent a private message to Chuck and not bothered the list?)
> >
> :)
> :
> > Am I right Chuck? Should it be removed from the distribution to avoid
> > silly questions like mine?

CGIWrapper serves a different purpose than WebKit.  Its goal is to make it 
easier to write a CGI application.  Some people might find it useful if they 
are writing an app that has to be deployed using CGI for whatever reason.  I 
personally don't see the harm of leaving it in Webware and occasionally 
getting silly questions :-)

- Geoff

Geoffrey Talvola <gtalvola@...>:
[snip]
> I don't think such a thing exists.  Maybe you're thinking of mod_python, 
> which has a mode which is intended to accelerate CGI scripts.

Hm. I guess I must have misread the docs. I just seem to remember some
mention of a component that would let you run your own CGI scripts.
(I'm not thinking of mod_python, no ;)

--
Magnus Lie Hetland                                  The Anygui Project
http://hetland.org                                  http://anygui.org

>>>>> "geoffrey" == Geoffrey Talvola <gtalvola@...> writes:

  geoffrey> Terrel Shumway wrote:
  >> > I agree completely that the way an app server like Webware
  >> manages URI
  >> > space should make no assumptions about filesystems. That's, in
  >> > my view, the major Webware wart.
  >>
  >> I hadn't thought of it that way. But yes, that is exactly right.

  geoffrey> I'm happy to see more flexibility in Webware's URI
  geoffrey> handling as long as the current behavior remains the
  geoffrey> default behavior out of the box.

I suppose that's reasonable, if for least-surprise reasons, if no
other, though least-surprise has a limited shelf life, IMO.

  geoffrey> More flexible URI handling is a bonus for
  geoffrey> content-management types of applications, but it doesn't
  geoffrey> help at all for the type of application I'm developing
  geoffrey> (and for many other apps too).

I think it's almost entirely orthogonal to the *kind* of application
one is building; that is, I just don't see any positive reason to make
assumptions about filesystem entities in the code that resolves a URI
into some resource on the server that is responsible for that URI.

Amazon is a pretty good example; there's a rather complex Web app that
runs the site, and I can't ever remember seeing a ?query-string URL;
it appears to pass all arguments as hierarchical elements separated by
"/".

That doesn't strike me as application *type*-specific at all. In some
cases, it's merely a matter of style, preference, and other
considerations (like those in the W3C's URI style recommendations).

Best,
Kendall Clark

> Amazon is a pretty good example; there's a rather complex
> Web app that
> runs the site, and I can't ever remember seeing a
> ?query-string URL;
> it appears to pass all arguments as hierarchical elements
> separated by
> "/".

We do things like this now, using Webware and mod_rewrite.
Not that I'm particularily enamored with having to create
page-specific regexps for mod_rewrite, but I think it will
be specific and hard to generalize for webware to
automatically handle.

For instance, old-style URL for us:

  http://www.somewhere.com/products.psp?id=502

is viewed/linked for the real world:

  http://www.somewhere.com/products/502

and could just as easily be

  http://www.somewhere.com/products/view/502

as opposed to perhaps

  http://www.somewhere.com/products/502/order

which could map to the real webware URL of

  http:://www.somewhere.com/order.psp?productid=502

The main reason we switched to this was actually for search
engines, as they tend to ignore query strings, and our
customers would prefer that every product gets hit. I
imagine Amazon's reasons would be similar.

I'm not sure at all how I would tell Webware to do this
stuff. In my mind the purpose is to remove restraints of
the underlying technical solution (what Webware pages do
what processing), which mod_rewrite is perfectly suited
for.

Luke

=====
------------------
Reference Counting Garbage Collection:
Look out philosophy majors, things really DO
cease to exist when no one is looking at them!
------------------

__________________________________________________
Do You Yahoo!?
Yahoo! Games - play chess, backgammon, pool and more
http://games.yahoo.com/

On Fri, Apr 26, 2002 at 02:46:00PM -0700, Luke Opperman wrote:
>   http://www.somewhere.com/products.psp?id=502

Are you able to use mod_rewrite with query strings?  I've got a
(non-Webware) application that stupidly funnels articles 
through 
	/article.php?article_id=1234
and I would like to redirect each article to a new URL
	/article/1234
or (in other cases) redirect a few articles while letting all
the others pass thorugh.

But whenever I try to use mod_rewrite, it strips the query string
off the source URI, so I can't match according to query string.

-- 
-Mike (Iron) Orr, iron@...  (if mail problems: mso@...)
   http://iron.cx/     English * Esperanto * Russkiy * Deutsch * Espan~ol

On Fri, Apr 26, 2002 at 02:46:00PM -0700, Luke Opperman wrote:
>  
> > Amazon is a pretty good example; there's a rather complex
> > Web app that
> > runs the site, and I can't ever remember seeing a
> > ?query-string URL;
> > it appears to pass all arguments as hierarchical elements
> > separated by
> > "/".
> 
> We do things like this now, using Webware and mod_rewrite.
> Not that I'm particularily enamored with having to create
> page-specific regexps for mod_rewrite, but I think it will
> be specific and hard to generalize for webware to
> automatically handle.

Why?

Imagine a configuration file that did nothing but this:

/URI-pattern    /filesystem-pointer

So,

/products        /foo/bar/webware/ProductServlet.py

And ProductServlet.py gets "/502" as an argument and does the right thing
with it. Seems pretty straightforward.

Now, I don't suggest this is how the mapping should go, since it's
simplistic and laborious, but there's no reason it couldn't go this way.
There's no reason you can't define a simple XML format to model URI space
(in fact, it's been done) and use explicit containment relations and XPath
queries to map URI space to Webware source resources.

> For instance, old-style URL for us:
> 
>   http://www.somewhere.com/products.psp?id=502
> 
> is viewed/linked for the real world:
> 
>   http://www.somewhere.com/products/502
> 
> and could just as easily be
> 
>   http://www.somewhere.com/products/view/502
> 
> as opposed to perhaps
> 
>   http://www.somewhere.com/products/502/order
> 
> which could map to the real webware URL of
> 
>   http:://www.somewhere.com/order.psp?productid=502
> 
> The main reason we switched to this was actually for search
> engines, as they tend to ignore query strings, and our
> customers would prefer that every product gets hit. I
> imagine Amazon's reasons would be similar.

Sure, that's one very good reason, actually.

> I'm not sure at all how I would tell Webware to do this
> stuff. 

I think the point is that presently it's not particularly easy, if doable at
all, 

  In my mind the purpose is to remove restraints of
> the underlying technical solution (what Webware pages do
> what processing), which mod_rewrite is perfectly suited
> for.

I use mod_rewrite heavily, but it's not terribly ideal:

1) It's grotty black magic voodoo and I usually have to ritually slaughter
many Perl hackers to get it to work, and that's messy

2) I want to do these mappings in something like Webware (or XPath), i.e.,
something that's expression-based; I don't like regex much at all

3) Though mod_rewrite is C code, I think there's more overhead doing it in
Apache than in Python as part of the algorithm of mapping URIs to source
resources, but that's more of a guess than a solid opinion.

There are several web and app servers that offere total flexibility to do
this mapping; I can't think of any reasons why in principle Webware can't do
the same thing.

Best,
Kendall Clark

Actually, I lied a bit. We're not using mod_rewrite for
these, but a little C++ ISAPI filter for IIS using
boost.org's Regex++.

However, you can do query strings from mod_rewrite as well.
Using RewriteCond for something like your example:

RewriteCond %{QUERY_STRING} id=([^&;]*)
RewriteRule (.*)articles\.psp $1articles/%1? [L]

Or something close to that. Relevant parts: RewriteCond
backreferences use %1,%2 etc as opposed to regular
backreferences $1,$2. The trailing question mark on the
RewriteRule tells it to remove the query string from the
request (i suppose this would be optional..?)

Now, for my purposes (where the incoming URL is already
correct sans-querystring, and I actually want to reinsert a
querystring for the servlets) it's very easy. If you
include a ? in the substitution string, mod_rewrite's docs
say it will automatically modify QUERY_STRING. 

RewriteRule (.*)articles/([0-9]+) $1articles?id=$2 [L]

Again, not tested but it's close.

Luke

--- Mike Orr <iron@...> wrote:
> On Fri, Apr 26, 2002 at 02:46:00PM -0700, Luke Opperman
> wrote:
> >   http://www.somewhere.com/products.psp?id=502
> 
> Are you able to use mod_rewrite with query strings?  I've
> got a
> (non-Webware) application that stupidly funnels articles 
> through 
> 	/article.php?article_id=1234
> and I would like to redirect each article to a new URL
> 	/article/1234
> or (in other cases) redirect a few articles while letting
> all
> the others pass thorugh.
> 
> But whenever I try to use mod_rewrite, it strips the
> query string
> off the source URI, so I can't match according to query
> string.


=====
------------------
Reference Counting Garbage Collection:
Look out philosophy majors, things really DO
cease to exist when no one is looking at them!
------------------

__________________________________________________
Do You Yahoo!?
Yahoo! Games - play chess, backgammon, pool and more
http://games.yahoo.com/

On Fri, Apr 26, 2002 at 03:00:48PM -0700, Mike Orr wrote:
> On Fri, Apr 26, 2002 at 02:46:00PM -0700, Luke Opperman wrote:
> >   http://www.somewhere.com/products.psp?id=502
> 
> Are you able to use mod_rewrite with query strings?  I've got a
> (non-Webware) application that stupidly funnels articles 
> through 
> 	/article.php?article_id=1234
> and I would like to redirect each article to a new URL
> 	/article/1234
> or (in other cases) redirect a few articles while letting all
> the others pass thorugh.

Hmm, here's the relevant parts of my mod_rewrite setup:

RewriteRule ^/+articles/+$ / [PT,QSA]
RewriteRule ^/+articles$ / [PT,QSA]
RewriteRule ^/+articles/+([0-9]+)/+plain/? /articles/plain.py?id=$1 [PT,QSA]
RewriteRule ^/+articles/+([0-9]+)/+email/? /articles/emtp.py?id=$1 [PT,QSA]
RewriteRule ^/+articles/+([0-9]+)/? /articles/index.py?id=$1 [PT,QSA]

So, if I understand you right, you want something like:

RewriteRule ^/article.php?article_id=([0-9]+) /article/$1 [PT,QSA]

and so on.

Feel free to ask questions if that's not clear.

Best,
Kendall Clark

On April 26, 2002 03:22 pm, Kendall Clark wrote:
> I use mod_rewrite heavily, but it's not terribly ideal:

I agree.  Pushing this stuff into WebKit allows much greater flexibility =
and=20
integration.  However, mod_rewrite rules and WebKit-based-URL-decoding ar=
e=20
not mutually exclusive.  Some uses of mod_rewrite, such as direct serving=
 of=20
static content, have no place in WebKit.

> 1) It's grotty black magic voodoo and I usually have to ritually slaugh=
ter
> many Perl hackers to get it to work, and that's messy
>
> 2) I want to do these mappings in something like Webware (or XPath), i.=
e.,
> something that's expression-based; I don't like regex much at all
>
> 3) Though mod_rewrite is C code, I think there's more overhead doing it=
 in
> Apache than in Python as part of the algorithm of mapping URIs to sourc=
e
> resources, but that's more of a guess than a solid opinion.

I'd guess the other way around!

-----

As an aside, the current WebKit URL-decoding process relies on each URL c=
hunk=20
being a valid Python module/package name:

http://example.com/WK/thisIsValid/_SoIsThis/00ButThisIsNot/Servlet.py

You can work around this with extraPathInfo, but then I assume you can't =
drop=20
the extension:

http://example.com/WK/Servlet.py/thisIsValid/_SoIsThis/00NowThisValid/ [w=
orks]

http://example.com/WK/Servlet/thisIsValid/_SoIsThis/00NowThisValid/ [does=
 this=20
work??]

Tavis

--- Kendall Clark <kendall@...> wrote:
> Imagine a configuration file that did nothing but this:
> 
> /URI-pattern    /filesystem-pointer
> 
> So,
> 
> /products        /foo/bar/webware/ProductServlet.py
> 
> And ProductServlet.py gets "/502" as an argument and does
> the right thing
> with it. Seems pretty straightforward.

Agreed, easy to manually define mappings. But it's not
automatic, I think my point was that tools already exist
for this (mod_rewrite in particular), and if you're going
to write a config file to map each one I don't see why that
functionality needs to be part of WebWare.

Presumably the motivation for it to be in Webware is that
there is knowledge we don't have at mod_rewrite's time.
perhaps you'd allow mappings based on Session info? Fair
enough, if there's actually a justifiable use.

Luke

=====
------------------
Reference Counting Garbage Collection:
Look out philosophy majors, things really DO
cease to exist when no one is looking at them!
------------------

__________________________________________________
Do You Yahoo!?
Yahoo! Games - play chess, backgammon, pool and more
http://games.yahoo.com/

On Friday April 26, 2002 04:43 pm, Kendall Clark wrote:
> >>>>> "geoffrey" == Geoffrey Talvola <gtalvola@...> writes:
>
>   geoffrey> Terrel Shumway wrote:
>   >> > I agree completely that the way an app server like Webware
>   >>
>   >> manages URI
>   >>
>   >> > space should make no assumptions about filesystems. That's, in
>   >> > my view, the major Webware wart.
>   >>
>   >> I hadn't thought of it that way. But yes, that is exactly right.
>
>   geoffrey> I'm happy to see more flexibility in Webware's URI
>   geoffrey> handling as long as the current behavior remains the
>   geoffrey> default behavior out of the box.
>
> I suppose that's reasonable, if for least-surprise reasons, if no
> other, though least-surprise has a limited shelf life, IMO.
>
>   geoffrey> More flexible URI handling is a bonus for
>   geoffrey> content-management types of applications, but it doesn't
>   geoffrey> help at all for the type of application I'm developing
>   geoffrey> (and for many other apps too).
>
> I think it's almost entirely orthogonal to the *kind* of application
> one is building; that is, I just don't see any positive reason to make
> assumptions about filesystem entities in the code that resolves a URI
> into some resource on the server that is responsible for that URI.

For the sake of keeping things simple.  Having a simple URI-servlet mapping 
helps with ease of understanding for the developers building the web 
application.

>
> Amazon is a pretty good example; there's a rather complex Web app that
> runs the site, and I can't ever remember seeing a ?query-string URL;
> it appears to pass all arguments as hierarchical elements separated by
> "/".

If I'm writing forms with action=get, I'm going to get ?=style query-string 
parameters regardless of whether I'd prefer a different format.  I may as 
well use that format in other places where I'm constructing the URI too -- 
why make my job more difficult by having to handle a different parameter 
format?

>
> That doesn't strike me as application *type*-specific at all. In some
> cases, it's merely a matter of style, preference, and other
> considerations (like those in the W3C's URI style recommendations).
>

Point taken, but I'll still prefer to keep things simple, and for me that 
means sticking with the way Webware works now.

- Geoff

On April 26, 2002 06:06 pm, Terrel Shumway wrote:
> http://webware.colorstudy.net/twiki/bin/view/Webware/UriMapping is for =
a
> concrete implementation that can go into Webware RSN that can replace
> the current filesystem-centric mapping strategy.

Tangent 1
-----------

This URI topic is somewhat related to how contexts/packages are loaded in=
to=20
WebKit.  The current implementation of ServletFactory relies on the 'imp'=
=20
module and thus can't take any advantage of custom import hooks.  As a=20
result, you have to create a new servlet factory for .tmpl files, .psp fi=
les,=20
etc., in addition to any import hooks you've written. =20

The code in ServletFactory.importAsPackage() and=20
ServletFactory._importModuleFromDirectory could be refactored to allow it=
 to=20
work with import hooks and remove the need for a separate ServletFactory =
for=20
each extension.  It should be possible to use a single ServletFactory for=
=20
=2Epy, .psp, and .tmpl servlets once the refactoring has been done and an=
=20
import hook has been written for .psp files. =20

Tangent 2
-----------

The current implementation of psp doesn't allow one psp servlet to seemle=
ssly=20
inherit from another psp servlet.  An import hook would make this possibl=
e.=20
Ditto for .tmpl files.  I've written an import hook for .tmpl files (usin=
g=20
Gordon McMillan's ui.py rather than ihooks or imputils, which are both ba=
dly=20
broken).  See ImportManager.py and ImportHooks.py in Cheetah's CVS.  It=20
should be fairly simple to extend this to work with PSP as well.  Anyone=20
interested?

Tavis

On Fri, 2002-04-26 at 09:39, Kendall Clark wrote:
> >>>>> "terrel" == Terrel Shumway <tshumway@...> writes:
> 
> Hmm, once more: GET is supposed to be idempotent *and* safe.
> 
>   terrel> I agree. GET should be idempotent unless there is a clear
>   terrel> indication otherwise such as
>   terrel> "/?action=this+will+change+your+document+forever" or
>   terrel> "/?bill=charge+my+visa&amount=5+zillion+dollars"
> 
> Those are idempotent, afaict. :>
> 
> Idempotent doesn't mean 'side-effect free'; it means "repeated
> applications have the same effect as the first one" (for some pretty
> vaguely undefined time span, though).

idempotent adj. Acting as if used only once, even if used multiple
times.  

F:V->V  is idempotent iff  F(F(v))=F(v) for all v in V

e.g. F(x)=x*1, F(x)=x*0, F(x)=c, F(x)=x+0, are idempotent, but F(x)=x+1
is not.

DELETE is idempotent (once it is gone, it is gone), but charging a
credit card is not.

Unfortunately,
http://www.w3.org/MarkUp/html-spec/html-spec_8.html#SEC8.2.2 
does claim that idempotent means "side-effect free" when describing when
to use POST vs GET in html forms. This MIGHT be why RFC 2068 even
mentions "Idempotent Methods"-- to clear up the confusion.

RFC 2068 (section 9.1) is correct, in the use of idempotent vs safe
methods, and the emphasis that GET and HEAD should always be safe.

   Naturally, it is not possible to ensure that the server does not
   generate side-effects as a result of performing a GET request; in
   fact, some dynamic resources consider that a feature. The important
   distinction here is that the user did not request the side-effects,
   so therefore cannot be held accountable for them.
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

9.1.2 Idempotent Methods

   Methods may also have the property of "idempotence" in that (aside
   from error or expiration issues) the side-effects of  N > 0 identical
   requests is the same as for a single request. The methods GET, HEAD,
   PUT and DELETE share this property.


Thank you, Kendall, for helping me to clarify this in my mind.
Idempotence is not the issue, safety is.

Therefore, a servlet that would otherwise perform an unsafe action
SHOULD NOT do so if the HTTP request method is GET or HEAD.


--
Note: GET side-effect as a feature: Counters are the best example.
Counters don't work if the UA considers GET idempotent and optimizes
away repeated requests for the same resource.

On Fri, Apr 26, 2002 at 02:48:17PM -0700, Terrel Shumway wrote:
> On Fri, 2002-04-26 at 09:39, Kendall Clark wrote:

> DELETE is idempotent (once it is gone, it is gone), but charging a
> credit card is not.

Yeah, it really is. Each GET has the same effect: to charge the same amount
on a credit card. That over the course of N GETs the *total* charge is N *
amount is no different than that, over the course of N GETs, a hit counter
is incremented N times.

> Thank you, Kendall, for helping me to clarify this in my mind.
> Idempotence is not the issue, safety is.

They are both issues, but, yes, GET and DELETE are both supposed to be
idempotent, but GET is supposed to be safe, too.

And yr welcome.

Kendall

Kendall Clark <kclark@...>:
>
> On Fri, Apr 26, 2002 at 02:48:17PM -0700, Terrel Shumway wrote:
> > On Fri, 2002-04-26 at 09:39, Kendall Clark wrote:
> 
> > DELETE is idempotent (once it is gone, it is gone), but charging a
> > credit card is not.
> 
> Yeah, it really is.

I guess this would depend on how you translate an "operation" to a
"function" (which is what "idempotent" describes). IMO the most useful
definition here would be as a function from one state to another, so
that applying the same operation repeatedly will have the same effect
as applying it once. (Not a perfect analogy to the function stuff,
since this would really entail that no change was possible I guess ;)

Anyway, charging a credit card repeatedly is not the same as deleting
a file repeatedly. After the first deletion, no more change will occur
by reapplying the operation. The same can hardly be said about the
credit card stuff.

> Each GET has the same effect: to charge the same amount
> on a credit card.

Sure... And each call to the function sin() has the same effect --
calculating the sine of a number. But sin(3) and sin(sin(3)) are not
equal, and thus sin() is not idempotent.

> That over the course of N GETs the *total* charge is N *
> amount is no different than that, over the course of N GETs, a hit counter
> is incremented N times.

Sure.

> > Thank you, Kendall, for helping me to clarify this in my mind.
> > Idempotence is not the issue, safety is.
> 
> They are both issues, but, yes, GET and DELETE are both supposed to be
> idempotent, but GET is supposed to be safe, too.

I'm not sure the term idempotent is very useful here, since it seems
to be used in two different ways (and since it's really only valid for
functions, or operators).

> And yr welcome.
> 
> Kendall

--
Magnus Lie Hetland                                  The Anygui Project
http://hetland.org                                  http://anygui.org