Webware for Python

The memory leak is still there.

Chuck Esterbrook wrote:
> 
> I've checked in the KeyValueAccess:
> - It no longer creates circular refs to self.
> - It has a memory leaks test.
> - The \r problem is fixed.
> 
> With regards to my white space problems:
> - Tabs were not the cause.
> - My new editor truncates trailing whitespace which explained some diffs. I
> don't see this as a problem and will leave as is.
> - KeyValueAccess.py was unique with it's extra \r problem. The \r's made it
> to the UNIX side via a copy and then made it to the repository. This threw
> me off.
> - Just as a note: There shouldn't ever be \r's in the repository since CVS
> for Windows translates \r\n <--> \n.
> 
> -Chuck
> 
> _______________________________________________
> Webware-discuss mailing list
> Webware-discuss@...
> http://lists.sourceforge.net/mailman/listinfo/webware-discuss

Well, I never said it wasn't.  :-)

BTW You never answered my question about whether you had 'LogActivity' on 
or off.

-Chuck


At 06:13 PM 5/31/00 -0400, Jay Love wrote:
>The memory leak is still there.
>
>Chuck Esterbrook wrote:
> >
> > I've checked in the KeyValueAccess:
> > - It no longer creates circular refs to self.
> > - It has a memory leaks test.
> > - The \r problem is fixed.
> >
> > With regards to my white space problems:
> > - Tabs were not the cause.
> > - My new editor truncates trailing whitespace which explained some diffs. I
> > don't see this as a problem and will leave as is.
> > - KeyValueAccess.py was unique with it's extra \r problem. The \r's made it
> > to the UNIX side via a copy and then made it to the repository. This threw
> > me off.
> > - Just as a note: There shouldn't ever be \r's in the repository since CVS
> > for Windows translates \r\n <--> \n.
> >
> > -Chuck
> >
> > _______________________________________________
> > Webware-discuss mailing list
> > Webware-discuss@...
> > http://lists.sourceforge.net/mailman/listinfo/webware-discuss
>
>_______________________________________________
>Webware-discuss mailing list
>Webware-discuss@...
>http://lists.sourceforge.net/mailman/listinfo/webware-discuss

Smartass.

Logactivity is off.

All of the transaient objects are getting deleted, Request, REsponse,
Transaction, Servlet and Session.

And, the leak is BIG, like 16K per Servlet.  For reference, an obkect is
16 bytes plus a dictionary, which is 24 bytes plus a factor of what's
inside.  In any case, the leak is big.  It's got to be something like
the reponse contents.  When the reponse is deleted, the _contents has 2
references to it.  It should have one, as best I can tell.

Chuck Esterbrook wrote:
> 
> Well, I never said it wasn't.  :-)
> 
> BTW You never answered my question about whether you had 'LogActivity' on
> or off.
> 
> -Chuck
> 
> At 06:13 PM 5/31/00 -0400, Jay Love wrote:
> >The memory leak is still there.
> >
> >Chuck Esterbrook wrote:
> > >
> > > I've checked in the KeyValueAccess:
> > > - It no longer creates circular refs to self.
> > > - It has a memory leaks test.
> > > - The \r problem is fixed.
> > >
> > > With regards to my white space problems:
> > > - Tabs were not the cause.
> > > - My new editor truncates trailing whitespace which explained some diffs. I
> > > don't see this as a problem and will leave as is.
> > > - KeyValueAccess.py was unique with it's extra \r problem. The \r's made it
> > > to the UNIX side via a copy and then made it to the repository. This threw
> > > me off.
> > > - Just as a note: There shouldn't ever be \r's in the repository since CVS
> > > for Windows translates \r\n <--> \n.
> > >
> > > -Chuck
> > >
> > > _______________________________________________
> > > Webware-discuss mailing list
> > > Webware-discuss@...
> > > http://lists.sourceforge.net/mailman/listinfo/webware-discuss
> >
> >_______________________________________________
> >Webware-discuss mailing list
> >Webware-discuss@...
> >http://lists.sourceforge.net/mailman/listinfo/webware-discuss
> 
> _______________________________________________
> Webware-discuss mailing list
> Webware-discuss@...
> http://lists.sourceforge.net/mailman/listinfo/webware-discuss

At 06:42 PM 5/31/00 -0400, Jay Love wrote:

>Smartass.
>
>Logactivity is off.
>
>All of the transaient objects are getting deleted, Request, REsponse,
>Transaction, Servlet and Session.
>
>And, the leak is BIG, like 16K per Servlet.  For reference, an obkect is
>16 bytes plus a dictionary, which is 24 bytes plus a factor of what's
>inside.  In any case, the leak is big.  It's got to be something like
>the reponse contents.  When the reponse is deleted, the _contents has 2
>references to it.  It should have one, as best I can tell.

That's strange. If it were the contents, why wouldn't we have leaking when 
Servlets are not being deleted? After all, responses and requests do get 
deleted for each transaction. Seems like keeping the servlets around could 
only make things worse, not better.

-Chuck

I've checked in serveral changes today:

* stress.py: Made localhost fix for Windows.

* WebKit.cgi: Added WebKitDir variable that when sent, points the adaptor 
to the location of WebKit. If you want Webware to reside somewhere else 
other than your server's web directory, then this is useful for platforms 
that don't have real links (e.g., Windows) or if your server won't follow 
symbolic links and you can't change that.

* AppServer.py:  Has better exception handling for plug-ins. Version ^ 0.3

* ExceptionHandler.py now only invokes os.getXid() funcs that exist. So 
error reporting works on Windows now.

* Application gracefully handles activity log columns that can't be 
resolved. This was a simple fix but a BIG improvement.

* KeyValueAccess.py: Cleaned up test suite. Now uses assertions instead of 
print's that have to be validated.

* Removed all the .pyc files and a TestingXYZ.tgz file.

* Added Logs/ and ErrorMsgs/ dirs to WebKit which are needed at runtime.


I believe WebKit will work for Windows out of the box now except for 
changing the first line of WebKit.cgi to #!python. I haven't tested CGI 
Wrapper or the FastCGI adaptor yet.

-Chuck

Chuck Esterbrook wrote:

> I believe WebKit will work for Windows out of the box now except for
> changing the first line of WebKit.cgi to #!python. I haven't tested CGI
> Wrapper or the FastCGI adaptor yet.

Thanks for the fixes!

Does PSP on Windows work yet?

I've investigated FastCGI on Windows, and as far as I can tell, FastCGI is
only available as a commercial extension to IIS from FastEngines.  It's
possible that Apache for Windows supports FastCGI but I'm skeptical.  In any
case, don't expect it to be easy to get a FastCGI-capable server running on
Windows.

--


- Geoff Talvola
  Parlance Corporation
  gtalvola@...

At 07:59 PM 5/31/00 -0400, Geoff Talvola wrote:
>Chuck Esterbrook wrote:
>
> > I believe WebKit will work for Windows out of the box now except for
> > changing the first line of WebKit.cgi to #!python. I haven't tested CGI
> > Wrapper or the FastCGI adaptor yet.
>
>Thanks for the fixes!

No problem. Thanks for pointing out the problems.


>Does PSP on Windows work yet?

I did another update. I now get a different error. Jay, do you have Windows 
at home?


Traceback (innermost last):
   File "Application.py", line 220, in dispatchRequest
     self.createServletInTransaction(transaction)
   File "Application.py", line 536, in createServletInTransaction
     inst = self.getServlet(transaction,path,cache)
   File "Application.py", line 477, in getServlet
     inst = factory.servletForTransaction(transaction)
   File "..\PSP\PSPServletFactory.py", line 115, in servletForTransaction
     clc.compile()
   File "..\PSP\PSPCompiler.py", line 58, in compile
     parser.parse()
   File "..\PSP\Parser.py", line 345, in parse
     if checkfunc(self, handler, reader):
   File "..\PSP\Parser.py", line 151, in checkDirective
     raise 'Invalid Directive'
Invalid Directive


-Chuck

Chuck Esterbrook wrote:
> 
> At 07:59 PM 5/31/00 -0400, Geoff Talvola wrote:
> >Chuck Esterbrook wrote:
> >
> > > I believe WebKit will work for Windows out of the box now except for
> > > changing the first line of WebKit.cgi to #!python. I haven't tested CGI
> > > Wrapper or the FastCGI adaptor yet.
> >
> >Thanks for the fixes!
> 
> No problem. Thanks for pointing out the problems.
> 
> >Does PSP on Windows work yet?
> 
> I did another update. I now get a different error. Jay, do you have Windows
> at home?

Yes (ashamed).

And the command line psp program actually works on Unix.  I'll run it
from Windows and let you know.


> 
> Traceback (innermost last):
>    File "Application.py", line 220, in dispatchRequest
>      self.createServletInTransaction(transaction)
>    File "Application.py", line 536, in createServletInTransaction
>      inst = self.getServlet(transaction,path,cache)
>    File "Application.py", line 477, in getServlet
>      inst = factory.servletForTransaction(transaction)
>    File "..\PSP\PSPServletFactory.py", line 115, in servletForTransaction
>      clc.compile()
>    File "..\PSP\PSPCompiler.py", line 58, in compile
>      parser.parse()
>    File "..\PSP\Parser.py", line 345, in parse
>      if checkfunc(self, handler, reader):
>    File "..\PSP\Parser.py", line 151, in checkDirective
>      raise 'Invalid Directive'
> Invalid Directive
> 
> -Chuck
> 
> _______________________________________________
> Webware-discuss mailing list
> Webware-discuss@...
> http://lists.sourceforge.net/mailman/listinfo/webware-discuss

>
> > I did another update. I now get a different error. Jay, do you have Windows
> > at home?
>
>Yes (ashamed).
>
>And the command line psp program actually works on Unix.  I'll run it
>from Windows and let you know.


I'm not testing the command line. I'm running the app server and running 
the Hello.psp program in Examples, like so:

http://localhost/WebKit.cgi/Hello


My point about Windows is that you could quickly setup apache and try this 
yourself. It was save a lot of these round trips.

-Chuck

All I need is the command line.  No need for apache running on windows
inside vmware running on linux. ;)

The bug in PSP on Win is much uglier than I thought, by the way.  I'm
not gonna get it tonight.

Jay

Chuck Esterbrook wrote:
> 
> >
> > > I did another update. I now get a different error. Jay, do you have Windows
> > > at home?
> >
> >Yes (ashamed).
> >
> >And the command line psp program actually works on Unix.  I'll run it
> >from Windows and let you know.
> 
> I'm not testing the command line. I'm running the app server and running
> the Hello.psp program in Examples, like so:
> 
> http://localhost/WebKit.cgi/Hello
> 
> My point about Windows is that you could quickly setup apache and try this
> yourself. It was save a lot of these round trips.
> 
> -Chuck
> 
> _______________________________________________
> Webware-discuss mailing list
> Webware-discuss@...
> http://lists.sourceforge.net/mailman/listinfo/webware-discuss

At 11:15 PM 5/31/00 -0400, Jay Love wrote:

>All I need is the command line.  No need for apache running on windows
>inside vmware running on linux. ;)
>
>The bug in PSP on Win is much uglier than I thought, by the way.  I'm
>not gonna get it tonight.
>
>Jay

Well let's shoot for Friday, our supposed code freeze. I'm kinda surprised 
that it doesn't work on Windows. The other code that broke was basically 
calls to functions that didn't exist.


I fixed HTTPServlet's map to use unbound references so no more circular 
refs. I think took out the line in Application that set the map to None.

I can get the app server to leak by changing Servlet.canBeReused() to 0 and 
running stress.py 200 3.

-Chuck

I haven't tried Apache on Windows but I'd definitely recommend Xitami.  It it free,
very small and fast, takes all of about 2 minutes to install, and works great
without any tweaking.  http://www.xitami.com

Chuck Esterbrook wrote:

> My point about Windows is that you could quickly setup apache and try this
> yourself. It was save a lot of these round trips.
>
> -Chuck

--


- Geoff Talvola
  Parlance Corporation
  gtalvola@...

At 10:15 AM 6/1/00 -0400, Geoff Talvola wrote:
>I haven't tried Apache on Windows but I'd definitely recommend Xitami.  It 
>it free,
>very small and fast, takes all of about 2 minutes to install, and works great
>without any tweaking.  http://www.xitami.com

I'll take a note of it, although I've already got apache working so my 
motivation is low.

Since this is the server you use and write, would you have any interest in 
developing a Xitami specific adaptor for it to improve performance?

-Chuck

At 11:13 AM 6/1/00 -0400, Chuck Esterbrook wrote:
>At 10:15 AM 6/1/00 -0400, Geoff Talvola wrote:
>>I haven't tried Apache on Windows but I'd definitely recommend 
>>Xitami.  It it free,
>>very small and fast, takes all of about 2 minutes to install, and works great
>>without any tweaking.  http://www.xitami.com
>
>I'll take a note of it, although I've already got apache working so my 
>motivation is low.
>
>Since this is the server you use and write, would you have any interest in

Er,                           use and _like_

Chuck Esterbrook wrote:

> I'll take a note of it, although I've already got apache working so my
> motivation is low.
>
> Since this is the server you use and write, would you have any interest in
> developing a Xitami specific adaptor for it to improve performance?
>
> -Chuck

I'd be much more interested in a general performance solution, not one linked to a
particular web server.  I have no great attachment toward Xitami other than its
ease of use and the fact that it's free.

Along those lines, I went back and checked again and noticed that as of May 8,
there's a beta version of mod_fastcgi for Win32 ( http://www.fastcgi.com/ ).  I'll
guess it's time to install Apache and fastcgi and see if I can get them to work.
That would be the ideal solution, I think, as long as Apache and fastcgi on Win32
are stable enough.

--


- Geoff Talvola
  Parlance Corporation
  gtalvola@...

At 11:30 AM 6/1/00 -0400, Geoff Talvola wrote:
>Chuck Esterbrook wrote:
>
> > I'll take a note of it, although I've already got apache working so my
> > motivation is low.
> >
> > Since this is the server you use and write, would you have any interest in
> > developing a Xitami specific adaptor for it to improve performance?
> >
> > -Chuck
>
>I'd be much more interested in a general performance solution, not one 
>linked to a
>particular web server.  I have no great attachment toward Xitami other 
>than its
>ease of use and the fact that it's free.

You might change your mind if you saw the actual performance difference. 
I'm expecting at least 2 X improvement.

Certainly, we also concentrate on generic performance improvements like 
better caching and more efficient APIs.

However, eventually, I hope to have server-specific high-performance 
adaptors for all the major servers.


>Along those lines, I went back and checked again and noticed that as of May 8,
>there's a beta version of mod_fastcgi for Win32 ( http://www.fastcgi.com/ 
>).  I'll
>guess it's time to install Apache and fastcgi and see if I can get them to 
>work.
>That would be the ideal solution, I think, as long as Apache and fastcgi 
>on Win32
>are stable enough.

Apache is noted by the apache group as not being ready for production on 
Windows. I don't know what your needs are, however. I use it for 
development and my production environment is on FreeBSD so no issue there.

-Chuck

Geoff Talvola wrote:

> Along those lines, I went back and checked again and noticed that as of May 8,
> there's a beta version of mod_fastcgi for Win32 ( http://www.fastcgi.com/ ).  I'll
> guess it's time to install Apache and fastcgi and see if I can get them to work.
> That would be the ideal solution, I think, as long as Apache and fastcgi on Win32
> are stable enough.

Even more interesting would be a mod_python adapter. Unfortunately, I am some
distance from understanding either mod_python or WebKit well enough to do this. I did
some experimentation, but I could only get the mod_python's CGI emulator to work
sporatically with WebKit.cgi

Dave.

At 11:39 AM 6/1/00 -0400, Dave Wallace wrote:
>Geoff Talvola wrote:
>
> > Along those lines, I went back and checked again and noticed that as of 
> May 8,
> > there's a beta version of mod_fastcgi for Win32 ( 
> http://www.fastcgi.com/ ).  I'll
> > guess it's time to install Apache and fastcgi and see if I can get them 
> to work.
> > That would be the ideal solution, I think, as long as Apache and 
> fastcgi on Win32
> > are stable enough.
>
>Even more interesting would be a mod_python adapter. Unfortunately, I am some
>distance from understanding either mod_python or WebKit well enough to do 
>this. I did
>some experimentation, but I could only get the mod_python's CGI emulator 
>to work
>sporatically with WebKit.cgi
>
>Dave.

Agreed.  I don't know much about mod_python, but I can certainly try to 
answer any questions you have about WebKit.cgi/CGIAdaptor.py.

-Chuck

Chuck Esterbrook wrote:

> Apache is noted by the apache group as not being ready for production on
> Windows. I don't know what your needs are, however. I use it for
> development and my production environment is on FreeBSD so no issue there.

The stability issue is of course important.  But conventional wisdom says that
nothing on Windows is as stable as Apache on Unix.  In particular, I'm seen claims
that IIS is inherently less stable because of its multithreaded architecture as
opposed to Apache on Unix's multiprocess architecture.  I also found the following
note at http://www.opensa.org/opensa/ about the Win32 Apache:

     The current release 0.20 has been tested for use in a production system,
     using the SSL-enabled Apache together with Allaire`s ColdFusion based
     Content Management System Sepctra.  During several tests Apache proved to
     be even more stable than IIS4 with Spectra.  The Test system was a
     PII-400Mhz based NT4 (SP5) with 256MB SDRAMs and two 8GB SCSI Harddisks.

Could be that the Apache people hold themselves to a much higher standard than
Microsoft?  Nah...

--


- Geoff Talvola
  Parlance Corporation
  gtalvola@...

Also, see this note from http://www.opensa.org/opensa/ :

     The latest release 0.20 was released on March 15th, which is be stable
     enough for use in a production system.

It sounds like they've made nice improvements on Win32.

--


- Geoff Talvola
  Parlance Corporation
  gtalvola@...

I've checked in some source to CVS:

* I updated the Configs/ dir in WebKit so that there are full blown config 
files for Application and AppServer as well as built-in backups whose names 
are appended by '.default' so users can always see what these were 
originally (I took that idea from apache).

* The admin script now shows:
     * the size of the access and error logs on the menu (in KB).
     * when the app server has been up since
     * how many requests have come in.
     - recall it's name is _admin as in http://localhost/WebKit.cgi/_admin

* The Activity Log and Error Log scripts now gracefully handle missing log 
files and report the number of rows.

* Application now prints a warning if it can't resolve the extension of a 
file for a URL because of multiple filenames. There is also an assertion 
that the path is found. This is slightly better than the previous behavior 
which caused an exception at a later point in time.



Special note:

* Sometimes I'm getting huge delays on requests. It seems to be mostly when 
I change a .py servlet file. Since I recently switched from Linux to 
Windows 98, I don't know if it's my new environment or something in the 
code. Please let me know if you experience this.

I've got Apache/NT installed and running.

When I try to use the Colors example served by Apache/NT, the WebKit.cgi
process hangs and the page never fully renders.  It's missing the last several
lines of HTML output.

Using Xitami as the web server, the Colors sample works just fine.  Every
other WebKit example works fine under Apache except Colors.  Sounds like
Apache handles the CGI IO differently, in a way that breaks WebKit.cgi.

Chuck, have you tried the Colors example from Apache on Windows?

--


- Geoff Talvola
  Parlance Corporation
  gtalvola@...

Are you using the Webware 0.2 .tar.gz download or the a snapshot from the 
repository?

0.2 has a simple but nasty bug where the CGIAdaptor includes some extra 
newlines in the output, although I experienced the symptons differently.

Also, if using the CVS, did you update recently?

-Chuck


At 04:27 PM 6/1/00 -0400, Geoff Talvola wrote:
>I've got Apache/NT installed and running.
>
>When I try to use the Colors example served by Apache/NT, the WebKit.cgi
>process hangs and the page never fully renders.  It's missing the last several
>lines of HTML output.
>
>Using Xitami as the web server, the Colors sample works just fine.  Every
>other WebKit example works fine under Apache except Colors.  Sounds like
>Apache handles the CGI IO differently, in a way that breaks WebKit.cgi.
>
>Chuck, have you tried the Colors example from Apache on Windows?
>
>--
>
>
>- Geoff Talvola
>   Parlance Corporation
>   gtalvola@...
>
>
>
>_______________________________________________
>Webware-discuss mailing list
>Webware-discuss@...
>http://lists.sourceforge.net/mailman/listinfo/webware-discuss

Chuck Esterbrook wrote:

> Are you using the Webware 0.2 .tar.gz download or the a snapshot from the
> repository?
>
> 0.2 has a simple but nasty bug where the CGIAdaptor includes some extra
> newlines in the output, although I experienced the symptons differently.
>
> Also, if using the CVS, did you update recently?
>
> -Chuck

I'm using a CVS snapshot, updated about an hour ago.

When I load the Colors page, the end of the page source looks like:

<td bgcolor=#ff66e5> <br> <font color=black>#ff66e5</font> </td>
<td bgcolor=#ff66ff> <br> <font color=black>#ff66ff</font> </td>
</tr>
<tr>

And that's it.  It has truncated the end of the page.  The WebKit.cgi python
process is still running.

--


- Geoff Talvola
  Parlance Corporation
  gtalvola@...

At 04:52 PM 6/1/00 -0400, Geoff Talvola wrote:
>Chuck Esterbrook wrote:
>
> > Are you using the Webware 0.2 .tar.gz download or the a snapshot from the
> > repository?
> >
> > 0.2 has a simple but nasty bug where the CGIAdaptor includes some extra
> > newlines in the output, although I experienced the symptons differently.
> >
> > Also, if using the CVS, did you update recently?
> >
> > -Chuck
>
>I'm using a CVS snapshot, updated about an hour ago.
>
>When I load the Colors page, the end of the page source looks like:
>
><td bgcolor=#ff66e5> <br> <font color=black>#ff66e5</font> </td>
><td bgcolor=#ff66ff> <br> <font color=black>#ff66ff</font> </td>
></tr>
><tr>
>
>And that's it.  It has truncated the end of the page.  The WebKit.cgi python
>process is still running.

It works me. I'm wondering if this is related to my "long delay" problem 
that I run into occasionally.

So I take it you're using the CGI adaptor and not the FastCGI adaptor.

Jay, does this work for you?

-Chuck

Chuck Esterbrook wrote:

> It works me. I'm wondering if this is related to my "long delay" problem
> that I run into occasionally.
>
> So I take it you're using the CGI adaptor and not the FastCGI adaptor.

CGI adaptor.

The version number on the Apache.exe file is 1, 3, 12, 0.  Are you using the same
version of Apache?  Did you modify anything in the configuration files?

You are using Windows 98, right?  I'm using WinNT 4.0, but I have a Win98 machine
at home I can test this on.  If I don't have a chance tonight, I'll try to test
it sometime this weekend.

--


- Geoff Talvola
  Parlance Corporation
  gtalvola@...

At 05:06 PM 6/1/00 -0400, Geoff Talvola wrote:
>Chuck Esterbrook wrote:
>
> > It works me. I'm wondering if this is related to my "long delay" problem
> > that I run into occasionally.
> >
> > So I take it you're using the CGI adaptor and not the FastCGI adaptor.
>
>CGI adaptor.
>
>The version number on the Apache.exe file is 1, 3, 12, 0.  Are you using 
>the same
>version of Apache?  Did you modify anything in the configuration files?
>
>You are using Windows 98, right?  I'm using WinNT 4.0, but I have a Win98 
>machine
>at home I can test this on.  If I don't have a chance tonight, I'll try to 
>test
>it sometime this weekend.


I have the same Apache version. I'm using Win98, 2nd ed, 4.10.2222 A

I tried reloading several times and it always worked. I also shut down the 
app server, started it and reloaded. I ran into my "long delay" problem 
again, so I shut down the server, started it and then the page came up.

I still don't know what that is. I'll probably put the server in 
'sequencing' tonight and see if I get the same problem.

Let me know how it goes with 98.

-Chuck

FYI,

fcgi.py uses socket.fromfd() which doesn't exist on Windows.

I don't know enough yet to determine if it can be rewritten to avoid use of that
function.

--


- Geoff Talvola
  Parlance Corporation
  gtalvola@...


> >
> >And that's it.  It has truncated the end of the page.  The WebKit.cgi python
> >process is still running.
> 
> It works me. I'm wondering if this is related to my "long delay" problem
> that I run into occasionally.
> 
> So I take it you're using the CGI adaptor and not the FastCGI adaptor.
> 
> Jay, does this work for you?
> 

Yeah, but I'm using a real OS. ;)

Colors is fine for me, and no delays.

Jay

I have updated the main documentation for Webware, e.g., 
Webware/Documentation/Webware.html.

Jay's latest PSP improvs that he sent me work for Windows.

We're still on track for release on Monday.

Three known bugs:
   - Sometimes there are long delays (approx. 1 min) when reloading 
servlets (1 user)
   - The Colors example doesn't produce all output (1 user)
   - WebKit leaks a little memory per transaction (all users)

I'll investigate the first two some more this weekend. (I can live with # 3 
for now.) And I'll update the WebKit docs including release notes and a new 
section for Windows users.

I assume Jay will update the PSP repository source and the PSP docs.

-Chuck

Chuck Esterbrook wrote:

>
> >Even more interesting would be a mod_python adapter. Unfortunately, I am some
> >distance from understanding either mod_python or WebKit well enough to do
> >this. I did
> >some experimentation, but I could only get the mod_python's CGI emulator
> >to work
> >sporatically with WebKit.cgi
> >
> >Dave.
>
> Agreed.  I don't know much about mod_python, but I can certainly try to
> answer any questions you have about WebKit.cgi/CGIAdaptor.py.

I played around with mod_python a little more last night, but it seems that the
content-type is getting occasionally scrambled (i.e. replaced with a random(!?)
python object instead of the string I put there) no matter if I use mod_python's
cgihandler or write my own handler. It looks like a problem with mod_python
itself, so hopefully I can come up with a minimal test case to verify that over
the weekend.

One interesting thing that I have come across is that apparently Apache and
therefore mod_python require headers to be sent separately from the subsequent
output stream. That is they must be set individually within an Apache table
object. Unfortunately, HTTPResponse combines them into a single string to be sent
all at once, the upshot being that my mod_python handler must parse the headers
out again. Perhaps this behavior could be based upon the value of the 'format'
key in the dictionary that is passed from the handler and the decision to combine
the headers moved into the Application or maybe a related object that can
encapsulate code for different handlers?

Also, just on the off-chance that this is helpful: in the mod_python cgiadapter
the os.environ is reset to a plain dictionary with a note claiming tha the
enviorn object leaks memory if it is manipulated too often (as in the case of
multiple cgi transactions). I haven't kept up with you leak tracking but maybe
this is a clue?

Dave.

At 08:46 AM 6/2/00 -0400, Dave Wallace wrote:
>Chuck Esterbrook wrote:
>
> >
> > >Even more interesting would be a mod_python adapter. Unfortunately, I 
> am some
> > >distance from understanding either mod_python or WebKit well enough to do
> > >this. I did
> > >some experimentation, but I could only get the mod_python's CGI emulator
> > >to work
> > >sporatically with WebKit.cgi
> > >
> > >Dave.
> >
> > Agreed.  I don't know much about mod_python, but I can certainly try to
> > answer any questions you have about WebKit.cgi/CGIAdaptor.py.
>
>I played around with mod_python a little more last night, but it seems 
>that the
>content-type is getting occasionally scrambled (i.e. replaced with a 
>random(!?)
>python object instead of the string I put there) no matter if I use 
>mod_python's
>cgihandler or write my own handler. It looks like a problem with mod_python
>itself, so hopefully I can come up with a minimal test case to verify that 
>over
>the weekend.

Sounds like a good plan.


>One interesting thing that I have come across is that apparently Apache and
>therefore mod_python require headers to be sent separately from the subsequent
>output stream. That is they must be set individually within an Apache table
>object. Unfortunately, HTTPResponse combines them into a single string to 
>be sent
>all at once, the upshot being that my mod_python handler must parse the 
>headers
>out again. Perhaps this behavior could be based upon the value of the 'format'
>key in the dictionary that is passed from the handler and the decision to 
>combine
>the headers moved into the Application or maybe a related object that can
>encapsulate code for different handlers?

Ah, juicy. Well the essence of an HTTPResponse is to have headers and a 
content, but the delivery needs to be the responsibility of the adaptor 
which should receive back a dictionary with headers, cookies & contents and 
then "do the right thing".

This was a design/implementation flaw on my part.

I can restructure this right after the 0.3 release on Monday. I don't want 
to muck with it before the release.

There is another flaw in WebKit's HTTPRequest where it assumes it has a 
CGI-like environment to pull info from. That will also get flushed out as 
we integrate WebKit in other ways.


>Also, just on the off-chance that this is helpful: in the mod_python 
>cgiadapter
>the os.environ is reset to a plain dictionary with a note claiming tha the
>enviorn object leaks memory if it is manipulated too often (as in the case of
>multiple cgi transactions). I haven't kept up with you leak tracking but maybe
>this is a clue?

I don't think we set the environment, I think we just read it. But that's 
interesting to keep in mind.

-Chuck

I added a new setting, ExtensionsToIgnore, whose default value is:

['.pyc', '.pyo', '.py~', '.bak']

I knew I wanted this eventually, but I didn't realize I would need it in 
this release. One of the admin pages is _dumpCVS.py which is used to show 
the access log, and another is _dumpError which inherits the former. 
Because it inherits the former and _dumpCSV.pyc gets created when it's used 
which resulted in confusion if you tried to use _dumpCSV directly because 
there was a .py and .pyc.

Another bug squashed for 0.3.

Let me know if you can think of other extensions that should go in there.

-Chuck

This update was supposed to go out Fri night, but a storm took out my local 
ISP. (I'm working off long distance this morning.)

On Friday night, I checked in a new PlugIn.py class, whose instances 
represent plug-ins. I also checked in chgs to AppServer to support this and 
a new _admin.py that displays the plug-in information.

You'll find actual documentation in PlugIn.py.

-Chuck

I've updated the Class Hierarchy section of the WebKit User's Guide. I also 
added to Webware/DocSupport a new script, classhier.py, that was used to do it.

We are still in code freeze for tomorrow's release, but classhier.py is not 
part of the Webware system. It gets run once by the dev team to update the 
docs and does not interact with WebKit, CGIWrapper, etc.

The new table is a nice improvement from the old table. In version 0.4, 
we'll shoot for being able to click on a class to get to it's info, docs 
and source.

Jay, depending on your time, you may want to run classhier.py for PSP. Let 
me know if you have questions.

-Chuck

CVS is updated now.
In my notes below, - means removed and + means added the method that follows.


* KeyValueAccess + hasValueForName() # return 1 if the object has a value 
for the given name

* Servlet + serverSideDir() serverSidePath() # if you need to know where 
the servlet lives in the file system

* Page + session()  # convenience method to get current session

* WebUtils.WebFuncs: HTMLEncode/Decode() now handle quotes,  " <--> &quot;

* HTTPResponse: - contents(), + rawResponse():

Returns the final contents of the response. Don't invoke this method until 
after deliver().

Returns a dictionary representing the response containing only strings, 
numbers, lists, tuples, etc. with no backreferences. That means you don't 
need any special imports to examine the contents and you can marshal it. 
Currently there are two keys. 'headers' is list of tuples each of which 
contains two strings: the header and it's value. 'contents' is a string 
(that may be binary (for example, if an image were being returned)).

* HTTPResponse: Added more assertions in various methods that the response 
is not already committed.

* AppServer: tweaked for raw response dictionaries

* CGIAdaptor: Handle raw response dictionaries. Also, shouldn't crap out on 
Windows anymore.

* I tested all WebKit examples and they work.

* FCGIAdaptor must be broken now. I can fix later today.


Geoff, Since you're the only one who experienced the Colors example problem 
could you update and try it out?

Dave, let me know if this works for you in mod_python.


-Chuck

Chuck Esterbrook wrote:

> Geoff, Since you're the only one who experienced the Colors example problem
> could you update and try it out?

Yup, the Colors example and all other examples work fine for me.

--


- Geoff Talvola
  Parlance Corporation
  gtalvola@...

CVS repository is updated:

* I "fixed" FCGIAdaptor.py in light of the new raw responses (which are 
dictionaries now instead of strings). However, I haven't tested as I don't 
have FastCGI working in my environment.

* I added OneShotAdapter.py and OneShot.cgi. During the development of my 
website, I found it too cumbersome to restart the AppServer as I was 
changing classes. The AppServer can detected and reload modified 
pages/servlets, but I found I had many base classes and other types of 
classes that are changing. Brief, but sufficient, docs are at the top of 
OneShotAdapter.py.

-Chuck

Chuck Esterbrook wrote:
> 
> CVS repository is updated:
> 
> * I "fixed" FCGIAdaptor.py in light of the new raw responses (which are
> dictionaries now instead of strings). However, I haven't tested as I don't
> have FastCGI working in my environment.

I tweaked Chuck's fixes to FCGIAdaptor, it works now.

Jay

(*) I overhauled Application.serverSidePathForRequest(). The biggest bonus 
is that files named 'index.*' or 'Main.*' are automatically picked up from 
directories. So for example, you can drop a Main.psp in a directory and 
that will be like a web server's "index.html".

The Application setting 'DirectoryFile', which defaults to ['index', 
'Main'], controls what names are looked for. Note that extensions are not 
included since WebKit discovers these on its own.

I added Testing/PathTesting.text. These tests are a requirement before 
checking in changes to Application.serverSidePathForRequest().


(*) Previously in Webware you accessed the admin scripts like so:

	http://localhost/WebKit.cgi/_admin

There was a special convention that URLs starting with underscores 
specified servlets that resided with the app server.

I have removed the special convention and put the admin scripts in a new 
subdir called Admin/. You now access them like so:

	http://localhost/WebKit.cgi/Admin/


(*) Put session timeout back to 1 hour and some other misc clean up items.

--

I am currently working on the redirect for directory requests. e.g.,

	http://localhost/WebKit.cgi/someDir

is redirected to:

	http://localhost/WebKit.cgi/someDir/

As we've discussed before, this is needed for relative links.

Rolled in with that (because it's around the same code) will be the 
handling of unknown URLs. Right now you get an ugly Python exception when 
that happens.


Admittedly, I broke the embedded servlet feature, which wasn't jiving with 
my overhaul of serverSidePathForRequest(). Also, I'm still having problems 
with accessing the PSP example page despite a recent patch to fix that.


FYI,

-Chuck

I've modified Application dispatchRequest() to gracefully handle bad URLs. 
That means returning a 404 Not Found status code with a basic HTML message. 
(Previously you got a Python exception on the console and a blank page in 
your browser.)

I took a stab at redirects to support directories, but didn't quite get 
there. However, I didn't break anything in the process either. :-)  I'll 
finish that up Tues or Wed.

Application also has some new accessor methods for contexts.

-Chuck

I updated the main Webware doc page and created release notes for 0.4 for 
Webware and WebKit. These are in CVS now.

Jay, if you could please review and see if I missed anything, which I'm 
sure I did, that would be great.

I still have to update the main WebKit documentation and I'd like to do a 
good job there. I also need to incorporate some changes from Geoff and look 
over his comments on session storage.

So we're further away than I thought, but a lot of good progress has been 
made recently. I'll continue with the work and the updates.

-Chuck

I added docs for MiscUtils.

I added hooks initThread() and delThread() per Geoff's need for 
Win32/COM/ADO integration. I repackaged Geoff's COM code as a Win32Kit/ 
plug-in for WebKit and sent to him for further work. The Win32 stuff 
probably won't make 0.4, but I want to know that the hooks added for it 
will work.

SessionMemoryStore was recently updated to save and load a single file 
containing all sessions upon shut down and start up of the app server, and 
also periodically as a means to backup.  This doesn't mesh well with the 
OneShot adapter and server which run in a CGI-like mode. Consider that 2 
OneShot's could be running at the same time. One of them would clobber the 
changes saved by the other, since all changes are saved in the same file. 
Also, it would be heinously inefficient for a site deployed in that fashion.

My solution was to add a new isPersistent() method to app server and 
family. SessionMemoryStore now reads and writes AllSessions.ses only if the 
server is persistent.

I have a great start on DocSupport/PySummary.py (as well as the install.py 
mod to invoke it on everything). It produces something that looks a little 
reminiscent of a C header file for a python file. The idea is to give you, 
at a glance, the names and arguments of all the methods, sans code and 
docs. I really want this as an enhancement to the documentation.


-Chuck

I finished off DocSupport/PySummary.py & .config and updated install.py to 
generate summaries, highlighted source code, ClassHier.html and 
ClassList.html for all Webware components. Hopefully, this will make the 
code and interfaces more accessible to people and get them up to speed faster.

For WebKit, PSP and MiscUtils, I added the appropriate links from the 
documentation to the generated files.

Outstanding:
   * Fix bug in Session pickling for ListBox example (Item class causes 
problem)
   * Review Geoff's comments on session timeouts
   * Update PSP docs (Jay)
   * Write new WebKit installation Guide
   * Run thorough test suite
   * Cut testing release
   * Update WebKit User's Guide
   * Cut final release


-Chuck

Geoff, who uses NT, IIS and Netscape, discovered a few problems with WebKit 
which I patched up this morning:

* Our use of the REQUEST_URI environment variable is not std CGI and not 
universally supported. I've fixed WebKit to accommodate. Admittedly, I 
don't have a web server that doesn't provide REQUEST_URI so we're in need 
of some testing.

* Using backslashes in URLs (like View?filename=Examples\Something.py) 
causes a problem on Netscape browsers. (The backslash can show up if you're 
running your server side software on Windows.) ExamplePage is now fixed re 
this.


I think we're still reviewing potential problems with 
AsyncThreadedAppServer, especially on NT.

-Chuck

I added some methods to the TableRecord class in MiscKit.DataTable.py to 
make it even more dictionary like (get(), items(), values(), etc.).

-Chuck

Regarding the upcoming 0.4:

I started writing the new WebKit Install Guide this morning (not in CVS 
yet). Not sure when I'll have it finished. Probably won't be done this week.

I dropped a note to Geoff (who is unavailable until mid-week) concerning 
WebKit's fairly recent changes to invalid session handling and how that 
interacts with his site.

I haven't decided whether I want to really try for the session pickling fix 
regarding instances, or leave that for the next version.


-Chuck

All in CVS:

* Fixes to socket calls like connect() and bind() to support API 
restrictions in Python 2.0.

* Fixes and refinements to the WebKit Install Guide.

* Updates to the Webware release procedures.


Sometime this week I plan on testing Webware with Python 2.0b2 and cutting 
release 0.4.1. Please refrain from non 0.4.1 changes in the repository 
since I have not been able to get CVS tagging working, and therefore 
branches. Ugg.

I don't want it this way, but I haven't cracked the "cvs tag" mystery (as 
described in an earlier msg)--yet.

For release 0.5, I am planning on supporting Python 1.5.2 and 2.0, by 
sticking to 1.5.2 features and testing with 2.0. I figure 1.6 will just pan 
out under this plan.

-Chuck

I upgraded to Py 2.0b2 (on Windows) and tested WebKit. No problems. 
(Remember 0.4 had a couple API issues like bind() and connect()).

In Application.__init__, moved the creation of sessions to come after 
contexts, since context initialization code might be necessary for 
unarchiving sessions. Also moved the creation of contexts after servlet 
factories, in case contexts need to access them.

This last change works me. Let me know if it works for you.

-Chuck

So far so good, it solves the problem I was having.

Dave.


Chuck Esterbrook wrote:

> 
> I upgraded to Py 2.0b2 (on Windows) and tested WebKit. No problems.  
> (Remember 0.4 had a couple API issues like bind() and connect()).
> 
> In Application.__init__, moved the creation of sessions to come after  
> contexts, since context initialization code might be necessary for  
> unarchiving sessions. Also moved the creation of contexts after 
> servlet  factories, in case contexts need to access them.
> 
> This last change works me. Let me know if it works for you.
> 
> -Chuck

Committed to CVS:

I took the section in the main Webware docs about "Source Code Conventions" 
and expanded it into a new document called "StyleGuidelines.html". See 
Webware/Documentation/.

I needed a quick "macro" system for the new document, so I also created 
DocSupport/RawToHTML.py. This is invoked on StyleGuidelines.raw by 
install.py to produce StyleGuidelines.html. I don't know yet if my idea 
behind RawToHTML is something that will last.

In any case, just remember to run install.py to refresh the docs.

Please don't assume that if you see a typo or an error of any sort in a 
document, that I am aware of it. I am not and would like to hear about it.

Nothing in the new Style Guidelines is really new or surprising. Some of it 
was already discussed on this list (like using tabs for indentation) and 
others were already described in Webware.html. Also, this is mostly for 
those who develop on Webware, which imposes no restrictions on those who 
use it.

Feel free to give feedback of any sort.


I created a new Webware/bin directory to hold utilities of general use 
(mostly for those who develop Webware). The two scripts in the main 
directory, clean and lines, will go here after they're rewritten in Python.

I added bin/checksrc.py to check some of the source code conventions 
described in the style guidelines. Unfortunately it doesn't grok multiline 
strings yet, so it gives many false alarms that it shouldn't.

I fixed DocSupport/toc.py which generated improper <font> tags in the Table 
of Contents for various docs.


In other news, Webware has been consistently in the top 10 downloads list 
and among the top 5 most active projects at the Python foundry:

   http://sourceforge.net/foundry/python-foundry/


-Chuck

I added several tasks to the SourceForge project page. They are viewable by 
all and administrated by myself, Jay and Geoff. I took some guesses at 
deadlines, completion %'s and so on. Feel free to update or make suggestions.

http://sourceforge.net/pm/task.php?group_project_id=4861&group_id=4866&func=browse


CVS changes:

The UNIX script Webware/lines has been removed in favor of the new 
Webware/bin/pystats.py. You can:
   * Invoke it with -h for help on the usage.
   * Read the doc string for complete information.
   * Just run it and see what happens:
     > cd Webware
     > python bin/pystats.py

I axed the MiscUtils.CSV class which has been subsumed by the more powerful 
MiscUtils.DataTable class (see it's doc string for full info).

Some file and code clean up that would bore you to tears.


-Chuck

Code changes:
   * Added new Error class to MiscUtils. Has complete doc string.
   * Added MiscUtils.Funcs and Comma()
   * Minor bug fix in DataTable.

I made progress on FormKit this weekend, but not to the point of uploading 
it yet. Some MiddleKit items demanded my attention.

For now, I'm swamped with some other work so it may be another week.


I forget if I mentioned this or not: I submitted a paper, "Introduction to 
Webware", for the Python9 conference (http://www.python9.org). It's about 
15 pages or so. Let's hope it gets accepted.

But if I not, I'll just put it in the docs. :-)

-Chuck