Webware for Python

Is there any good reason why we implement urlEncode and htmlEncode on 
our own, when cgi.escape and urllib.quote exist?

I like renaming these functions, but I don't see any reason we should 
reimplement them.  (I encountered a particular problem with urlEncode, 
which always uses + for space, even when it's not appropriate -- 
urllib.quote has a more complete interface)

If no one objects, I'll change WebUtils.Funcs, and add optional 
arguments for the complete behavior.

   Ian

I am trying to nock off some of the bugs as can.

I just fixed the bug in HTTPResponse setCookie().  And I also found some 
other problems with the expiration date handling which I fixed.

It seems like the expiration date parsing might best be placed on the 
Cookie object itself.  I didn't get that agressive with the fix at this 
time. Anyone see a reason not to do this?

Also, there was a missing allMatches() function referenced in 
DateInterval.  I am not a Python RE module expert.  Does anyone see 
anything wrong with this code?  It seems to work for me, and I just
added it to CVS. Although I think it should go in a more general purpose 
location.

def allMatches(source, regex):
     """Return a list of matches for regex in source
     """
     pos = 0
     end = len(source)
     rv = []
     match = regex.search(source, pos)
     while match:
         rv.append(match)
         match = regex.search(source, match.end() )
     return rv

There are a couple of other patches from Edmund Lian regarding deleting 
cookies.  They look reasonable to me at first glance, if I don't hear 
any comments to the contrary in the next day or so, I'll apply them as well.

So,

Bugs item #655562, was opened at 2002-12-18 00:38
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=104866&aid=655562&group_id=4866

Category: WebKit
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Edmund Lian (elian)
Assigned to: Nobody/Anonymous (nobody)
Summary: Error in setCookie method of HTTPRespons

Initial Comment:
Last line of setCookie method reads:

self.response().addCookie(cookie)

It should be:

self.addCookie(cookie)

Diff attached.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=104866&aid=655562&group_id=4866

So before I start getting too comfortable with my knowledge of Webware, 
I wanted to check on a couple of things that look like possible problems.

In the case of the psp-handler, the mod_webkit sets WK_ABSOLUTE=1 and 
then HTTPRequest.py urlPath()/fsPath() will return a path constructed 
from the REQUEST_URI and the DOCUMENT_ROOT.  This breaks  when 
referencing files under user directories, because it does not properly 
deal with the /~user/ expansion on Apache/Unix systems.

However, it appears in the limited testing I have done, that the 
SCRIPT_FILENAME always refers to the absolute server side path for the 
file, including any ~expansion for the username.  (note: we can't do our 
own ~ expansion, without adding code to check where Apache has the user 
public_html directories located, since that is a configuration option in 
Apache.)

Furthermore, in both with direct URL's (ie: no ~user expansion) and with 
the ~user URL's, the extra URLPath information is not properly parsed 
off of the .psp pages.  However the PATH_INFO environment variable does 
contain the extra URL information.

So in the case of the psp-handler / WK_ABSOLUTE=1 condition, it appears 
(using Apache 1.3.26 under Linux) that the urlPath() should return the 
SCRIPT_FILENAME and the extraURLPath should be set to PATH_INFO, rather 
than trying to parse it.

Under the normal webkit-handler it does not appear to be a problem, as 
the PATH_INFO contains the URL after parsing off the location where the 
webware handler resides.  And the PATH_INFO must then be parsed manually 
to determine the context and other information.

Has anyone else tried the psp-handler and either extraURL information, 
or the ~user expansion?

-Stuart-

If the ExtraPathInfo is set, serverSideInfoForRequest uses 
serverSideInfoForRequestNewAlgorithm.

However, this is currently incompatible with the psp-handler approach 
where the environment WK_ABSOLUTE is set to 1.  The NewAlgorithm version 
fails to check the absolute setting, and tries to process the absolute 
URL as a context.  That fails and the default context is used.

Any reason not to just put a similar check for absolutepath in the 
serverSideInfoForRequestNewAlgorithm?

Is the intention to have the serverSideInfoForRequestNewAlgorithm 
replace serverSideInfoForRequest?  If so, does that mean that the 
ExtraPathInfo application setting will be deprecated sometime?

-Stuart-

Stuart Donaldson [mailto:stu@...] wrote:
> The shutdown code in ThreadedAppServer.shutDown() sends a None to all 
> threads to cause them to terminate, and then does a join to 
> each thread.
> 
> The problem is that if you have a thread that is busy, perhaps a 
> server-push thread that is just sending a continuous stream 
> of data, or 
> a thread bound up for some other reason that they will not terminate.
> 
> This causes the shutDown to lock up.
> 
> A couple of questions:
> 
> 1) Should there be a timeout when shutting down, after which it just 
> forcibly terminates the thread, or just ignores the thread 
> allowing the 
> shutDown operation to continue?

There's no way to forcibly terminate a thread in Python (that I know of).

> 
> 2) Should functions like flush() and write() check to see if 
> the server 
> is still running, or see if the server has been trying to 
> shutdown for 
> some time and then raise an error?
> 
> 3) Should we just go out of the way to document the behavior in the 
> description of how to stop an AppServer, along with shutDown, 
> initiateShutDown, and probably the server-push examples?

I vote for this.

> 
> 4) Something else?
> 
> 
> I think if we're going to leave the behavior, we should at least do 
> option 3 and document it.  (If it's there, I must have missed 
> it.)  The 
> recommended usage be that any potential compute bound or long running 
> server-push operation should check server().running.
> 
> On another point, shouldn't server().running parameter be 
> _running, and 
> use a running() function to access it?  

Yes, to be consistent we should be using a function, especially if we're
going to document that servlets can check it during long-running operations.

- Geoff

Patches item #655502, was opened at 2002-12-17 16:51
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=304866&aid=655502&group_id=4866

Category: WebKit
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Stuart Donaldson (stuartd)
Assigned to: Nobody/Anonymous (nobody)
Summary: allow subclassing of ThreadedAppServer

Initial Comment:
This patch allows for subclassing of ThreadedAppServer.
It updates bothThreadedAppServer.py, and the proposed
replacement of NewThreadedAppServer.py 

An overview of how this is accomplished is as follows:

Launch.py is updated to allow specification of a module
which does not need to reside under WebKit.  The
specified python module path can either be a module
(which must contain a function main) or can be a
reference to a function whch is used in place of main.

The function main is called with the commandline to
start the server.

Within ThreadedAppServer.py (and
NewThreadedAppServer.py) the function main() takes an
optional argument specifying the serverClass.  The
serverClass defaults to the ThreadedAppServer defined
within the current module.

To use this, derive a subClass of ThreadedAppServer
adding or altering functionality as you wish.

Then declare a function such as main() which at least
must take the command line arguments.  If you want to
parse these yourself, that's OK, but unless you are
adding functionality beyond what is provided in the
main from ThreadedAppServer, you can turn around and
call that main function, passing in your new class as
the serverClass.

New method added include:
 waitShutdown()  wait until the shutDown completes.

Note that much of the functionality of run() was moved
into a run() method on the ThreadedAppServer.  I also
chose not to pass arguments into mainloop() but rather
to make them properties on the object.  This is because
alternate mainloop() implementations (ie:
NewThreadedAppServer) may take different arguments. 
Matching the properties will be simpler and more
streightforward.

I should also note this includes a fix for the bug
630432 which fixed some monitor issues on NT.  Although
this has not been tested on NT.

There is a TestServer.py file which tests this.  I
didn't know exactly where to put it, I don't think it
belongs in WebKit, but perhaps in a WebKit/Tests.  But
that doesn't exist.



----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=304866&aid=655502&group_id=4866

Patches item #630432, was opened at 2002-10-29 04:56
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=304866&aid=630432&group_id=4866

Category: WebKit
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Oleg Noga (oleg_noga)
Assigned to: Jay Love (jslove)
Summary: Monitor bug fixes

Initial Comment:
This patch provides minimal fixes to make 
ThreadedAppServer work with monitor enabled under 
Windows NT and descendant OSes.
---
Patch created for checkout on 2002-10-29.
It also actual for Webware 0.7 release.


----------------------------------------------------------------------

Comment By: Stuart Donaldson (stuartd)
Date: 2002-12-18 17:07

Message:
Logged In: YES 
user_id=326269

If you're interested, take a look at the ThreadedAppServer
patch. I incorporated this patch into that because it
affected the same code

Reference patch 655502

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=304866&aid=630432&group_id=4866

Patches item #656064, was opened at 2002-12-18 17:04
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=304866&aid=656064&group_id=4866

Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Stuart Donaldson (stuartd)
Assigned to: Nobody/Anonymous (nobody)
Summary: Change timestamp in mod_webkit to float

Initial Comment:
The timestamp sent in mod_webkit to Webware is an int,
this is useful in measuring performance and
communication delays between the web server, and Webware.

This patch changes the value from an int to a float
which is in the same format used by Python's
time.time() function.  This allows much finer grained
accuracy when using the timestamp to measure
communication delays and such.

Note that the marshal module in Python does not support
a binary floating point, so it is converted to a string
to do this. 

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=304866&aid=656064&group_id=4866

The shutdown code in ThreadedAppServer.shutDown() sends a None to all 
threads to cause them to terminate, and then does a join to each thread.

The problem is that if you have a thread that is busy, perhaps a 
server-push thread that is just sending a continuous stream of data, or 
a thread bound up for some other reason that they will not terminate.

This causes the shutDown to lock up.

A couple of questions:

1) Should there be a timeout when shutting down, after which it just 
forcibly terminates the thread, or just ignores the thread allowing the 
shutDown operation to continue?

2) Should functions like flush() and write() check to see if the server 
is still running, or see if the server has been trying to shutdown for 
some time and then raise an error?

3) Should we just go out of the way to document the behavior in the 
description of how to stop an AppServer, along with shutDown, 
initiateShutDown, and probably the server-push examples?

4) Something else?


I think if we're going to leave the behavior, we should at least do 
option 3 and document it.  (If it's there, I must have missed it.)  The 
recommended usage be that any potential compute bound or long running 
server-push operation should check server().running.

On another point, shouldn't server().running parameter be _running, and 
use a running() function to access it?  

-Stuart-

Stuart Donaldson [mailto:stuartd@...] wrote:
> So is Monitor.py being used extensively?
> 
> It appears that it is incompatible with the AppServer shell 
> script, and
> Launch.py, in that it does not support starting, or 
> restarting the appserver
> with arguments specified to AppServer/Launch...
> 
> Is it intended as an alternate to AppServer/Launch with reduced
> functionality of not supporting command line arguments?
> 
> Seems like you could have Monitor.py replace Launch.py, and 
> have a mode
> where it sticks around to provide continual monitoring, or just does a
> one-shot monitor and exit.  Any time it detects the AppServer is not
> responding, it would do its thing of restarting the 
> appserver.  Perhaps the
> command-line arguments need to be saved in a file and 
> re-loaded to insure
> their re-use?  
> 
> What is AsyncThreadedAppServer?  Is this somethign new, or something
> obsoleted?  Monitor seems to default to this.  It also 
> doesn't look like it
> works with NewThreadedAppServer as-is.

I can't say that I've ever tried to use Monitor.py.  I haven't ever
experienced an appserver crash (even though I'm running on Windows:-) so it
wouldn't have ever helped me.  But we ought to fix it if it's currently
broken in CVS.

AsyncThreadedAppServer is no longer part of Webware -- it has been removed
in CVS.  It was an experimental version of the AppServer that was supposed
to be faster, but turned out to be buggy and slower, so it was removed.

- Geoff

Bugs item #655531, was opened at 2002-12-17 19:33
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=104866&aid=655531&group_id=4866

Category: PSP
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Fazal Majid (majid)
Assigned to: Nobody/Anonymous (nobody)
Summary: JavaScript string escapes are unescaped

Initial Comment:
If I take the following PSP:

<html>
<script language="JavaScript">
var foo = "hello, world\n";
...
</script>
...
</html>

The backslash in the JavaScript \n literal is not
escaped in the Python string embedded in the servlet
(unlike the double quote), and is expanded by Python
when the servlet is called, thus breaking the
JavaScript, with the following result:

<html>
<script language="JavaScript">
var foo = "hello, world
";
...
</script>
...
</html>


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=104866&aid=655531&group_id=4866

Bugs item #655427, was opened at 2002-12-17 13:34
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=104866&aid=655427&group_id=4866

Category: WebKit
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Peter Kropf (peterk)
Assigned to: Nobody/Anonymous (nobody)
Summary: NewThreadedAppServer & http address

Initial Comment:
In WebKit/NewThreadedAppServer.py if it's going to 
start an http server, it always uses 127.0.0.1 as the 
address. This causes grief when attempting to connect 
to the server from systems on the network, especially if 
there is more than 1 nic. Changing the '127.0.0.1' to 
server.address()[0] will bind to the address that's used 
for the AppServer. 

In addition, it would be good to add configuration 
information to AppServer.config to specify which 
address and port are to be used.

BTW if you add me as a developer to the project, I'll be 
glad to fix the code. Or I can send in a patch...


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=104866&aid=655427&group_id=4866

Stuart Donaldson [mailto:stuartd@...] wrote:
> Given that some things work a little different between NT/Unix,
> can you put a quick test in and see if the catch KeyboardInterrupt
> is ever getting called, and let me know?

I just tried it.  Looks like KeyboardInterrupt never gets raised (on
windows).  It's the signal that causes the shutdown to be called.

> I can incorporate some cleanup into my subclassable 
> ThreadedAppServer patch.
> Part of which will make it so experiments like Ian's 
> NewThreadedAppServer
> could be done through subclassing, rather than 
> re-implementation.  That way
> the NT/Unix differences can be kept in as few places as 
> possible. (I am
> aware that the NewThreadedAppServer also cleans up some stuff in
> ThreadedAppServer so it is a little more than just a new 
> paradigm for the
> socket handling...)
> 

Would it be better to make your modifications to NewThreadedAppServer rather
than to ThreadedAppServer?  It may save time down the road if
NewThreadedAppServer becomes the standard.

- Geoff

Given that some things work a little different between NT/Unix,
can you put a quick test in and see if the catch KeyboardInterrupt
is ever getting called, and let me know?

I can incorporate some cleanup into my subclassable ThreadedAppServer patch.
Part of which will make it so experiments like Ian's NewThreadedAppServer
could be done through subclassing, rather than re-implementation.  That way
the NT/Unix differences can be kept in as few places as possible. (I am
aware that the NewThreadedAppServer also cleans up some stuff in
ThreadedAppServer so it is a little more than just a new paradigm for the
socket handling...)

-Stuart-


> -----Original Message-----
> From: Geoffrey Talvola [mailto:gtalvola@...]
> Sent: Tuesday, December 17, 2002 9:31 AM
> To: Stuart Donaldson; Webware-Devel (E-mail)
> Subject: RE: [Webware-devel] ThreadedAppServer keybard interrupt and
> termi nation
> 
> 
> Stuart Donaldson [mailto:stuartd@...] wrote:
> > I am working on a patch to integrate much of the 
> functionality of the
> > ThreadedAppServer.run() function into a method on 
> > ThreadedAppServer so it
> > can be more easily subclassed.
> > 
> > In trying to not break any existing functionality, I have a 
> couple of
> > questions:
> > 
> > The run function currently catches KeyboardInterrupt 
> > exceptions.  However
> > signals for this are also caught and directed to the shutDown 
> > function.  Is
> > there a case where the KeyboardInterrupt exceptions would 
> get thrown?
> 
> Not sure.  This might be paranoia or leftovers from an 
> earlier version of
> the code that didn't use signals.
> 
> > Would it make more sense if, after returning from mainloop() 
> > we checked to
> > see if we were still running, and then called shufDown() if needed?
> 
> Sounds plausible to me.
> 
> > Also, the NT code never seems to call shutDown(), it just sets
> > server.running=0.  Is this a problem on NT?  I don't have 
> > much NT/Python
> > combined experience so I don't know why it might not want to 
> > call shutDown()
> > and cleanup the queues.
> 
> But the signal code does cause shutDown() to be called on 
> Windows, so it's
> not necessary to call it after returning from mainloop 
> because it's the call
> to shutDown() that _caused_ mainloop to exit.
> 
> You do have a point that if the mainloop exits for some other 
> reason than
> Ctrl-C being called, then it looks like shutDown() won't be 
> called.  This is
> where your proposed test to see if we're still running might help.  In
> practice, I can definitely vouch that the existing code for 
> Windows seems to
> work properly.
> 
> This code is all kind of messy.  If you can figure out how to 
> clean it up
> without breaking anything, more power to you :-)  It's 
> difficult to do a
> good job because things work a little differently on Windows vs. Unix.
> 
> - Geoff
>

So is Monitor.py being used extensively?

It appears that it is incompatible with the AppServer shell script, and
Launch.py, in that it does not support starting, or restarting the appserver
with arguments specified to AppServer/Launch...

Is it intended as an alternate to AppServer/Launch with reduced
functionality of not supporting command line arguments?

Seems like you could have Monitor.py replace Launch.py, and have a mode
where it sticks around to provide continual monitoring, or just does a
one-shot monitor and exit.  Any time it detects the AppServer is not
responding, it would do its thing of restarting the appserver.  Perhaps the
command-line arguments need to be saved in a file and re-loaded to insure
their re-use?  

What is AsyncThreadedAppServer?  Is this somethign new, or something
obsoleted?  Monitor seems to default to this.  It also doesn't look like it
works with NewThreadedAppServer as-is.

Stuart Donaldson
Alerton Technologies Inc.

Stuart Donaldson [mailto:stuartd@...] wrote:
> I am working on a patch to integrate much of the functionality of the
> ThreadedAppServer.run() function into a method on 
> ThreadedAppServer so it
> can be more easily subclassed.
> 
> In trying to not break any existing functionality, I have a couple of
> questions:
> 
> The run function currently catches KeyboardInterrupt 
> exceptions.  However
> signals for this are also caught and directed to the shutDown 
> function.  Is
> there a case where the KeyboardInterrupt exceptions would get thrown?

Not sure.  This might be paranoia or leftovers from an earlier version of
the code that didn't use signals.

> Would it make more sense if, after returning from mainloop() 
> we checked to
> see if we were still running, and then called shufDown() if needed?

Sounds plausible to me.

> Also, the NT code never seems to call shutDown(), it just sets
> server.running=0.  Is this a problem on NT?  I don't have 
> much NT/Python
> combined experience so I don't know why it might not want to 
> call shutDown()
> and cleanup the queues.

But the signal code does cause shutDown() to be called on Windows, so it's
not necessary to call it after returning from mainloop because it's the call
to shutDown() that _caused_ mainloop to exit.

You do have a point that if the mainloop exits for some other reason than
Ctrl-C being called, then it looks like shutDown() won't be called.  This is
where your proposed test to see if we're still running might help.  In
practice, I can definitely vouch that the existing code for Windows seems to
work properly.

This code is all kind of messy.  If you can figure out how to clean it up
without breaking anything, more power to you :-)  It's difficult to do a
good job because things work a little differently on Windows vs. Unix.

- Geoff

I am working on a patch to integrate much of the functionality of the
ThreadedAppServer.run() function into a method on ThreadedAppServer so it
can be more easily subclassed.

In trying to not break any existing functionality, I have a couple of
questions:

The run function currently catches KeyboardInterrupt exceptions.  However
signals for this are also caught and directed to the shutDown function.  Is
there a case where the KeyboardInterrupt exceptions would get thrown?

Would it make more sense if, after returning from mainloop() we checked to
see if we were still running, and then called shufDown() if needed?

Also, the NT code never seems to call shutDown(), it just sets
server.running=0.  Is this a problem on NT?  I don't have much NT/Python
combined experience so I don't know why it might not want to call shutDown()
and cleanup the queues.

Stuart Donaldson

I might also suggest moving the 'shutDown' function from ThreadedAppServer
into AppServer, since it wants the AppServer instance (retrieved from
AppServer) and calls the AppServer.initiateShutdown() method which is
defined within AppServer.

 I am not clear on where the signal handling should be defined, but suspect
they could remain within ThreadedAppServer, and in fact we may want to
retain a reference in ThreadedAppServer for shutDown, but this could be done
through: "from AppServer import shutDown" 

-Stuart-

> -----Original Message-----
> From: Chuck Esterbrook [mailto:ChuckEsterbrook@...]
> Sent: Monday, December 16, 2002 4:52 PM
> To: webware-devel@...
> Subject: Re: [Webware-devel] Accessing the app server instance from
> anywhere
> 
> 
> On Monday 16 December 2002 04:38 pm, Stuart Donaldson wrote:
> > Couldn't this be done by AppServer.__init__()?
> >
> > Then require that all AppServer's must initialize the base class
> > AppServer which they likely do anyway.
> 
> Er, good point.  :-)
> 
> I need more sugar...
> 
> 
> > Is there any reason to have multiple AppServer's running?  Would it
> > make sense that WebKit.globalServer be a function that returns the
> > server? Then if for any reason we end up with a more complex server
> > referencing system accesses would at least be going through a
> > function which could be modified to do the right thing.
> 
> I can't imagine that there will ever be more than one AppServer in a 
> single process. I'm willing to work off that assumption.
> 
> -Chuck
> 
> 
> 
> -------------------------------------------------------
> This sf.net email is sponsored by:
> With Great Power, Comes Great Responsibility 
> Learn to use your power at OSDN's High Performance Computing Channel
> http://hpc.devchannel.org/
> _______________________________________________
> Webware-devel mailing list
> Webware-devel@...
> https://lists.sourceforge.net/lists/listinfo/webware-devel
>

Zope and it's related code (like ZODB) are not supported on Python 2.2. 
  That's why downgrading to Python 2.1 worked for that guy.   I heard 
that 2.6 beta does work and that 2.7 will work but I have no idea as I 
only looked into this for like 10 minutes. (I too have OS X  Jaguar 
boxes).   I decided to roll my own object persistence.

Mike
On Monday, December 16, 2002, at 08:53  PM, Lee Phillips wrote:

>
> I'm interested in using the ZODB with Webware. I develop on Macs 
> running
> OSX 10.2.x (Jaguar). I compiled a couple of stable versions of the ZODB
> on a Mac running OS X 10.2.1 with the python 2.2 included by Apple. 
> They
> (Standalone 1.0 and ZODB3-3.1) compiled with numerous warnings, and
> running the test.py gave me a bus error for both. I found a mailinglist
> archive wherein someone said that reverting to python 2.1 worked for
> him, but I'm not interested in doing that. Can anyone suggest how to 
> get
> this working, or suggest a different object persistence tool?