Webware for Python

Looks good,
You could also suggest SitePage.py for pages in the site (including ones 
that do not need authentication) and then subclass it to SecurePage for 
all secure pages. It would be a great intro to using classes to manage a 
site.

-Aaron

Ian Bicking wrote:

> I added some documentation today that gives some advice on application 
> development with Webware, mostly MakeAppWorkDir with a little on CVS.  
> I'd be interested on feedback -- I'd like this to become the basis for 
> some conventions on distributing and handling Webware applications.  
> If there's other topics that seem appropriate, or whatever.  
> Documentation and enhancements can also go hand-in-hand -- some things 
> might not be described because there's no one good way to do them in 
> Webware; we might be able to fix those and document them at the same 
> time.
>
> The document is in CVS at WebKit/Docs/ApplicationDevelopment.html (the 
> .txt is the canonical source), or online at:
>   http://tinyurl.com/3tfl
>
>   Ian
>
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Webware-discuss mailing list
> Webware-discuss@...
> https://lists.sourceforge.net/lists/listinfo/webware-discuss

On Sunday, December 29, 2002, at 10:21 AM, Aaron Held wrote:
> Looks good,
> You could also suggest SitePage.py for pages in the site (including 
> ones that do not need authentication) and then subclass it to 
> SecurePage for all secure pages. It would be a great intro to using 
> classes to manage a site.
>
I prefer to make all pages potentially secure, and then use a method to 
indicate which pages require a password.  Some pages may only be secure 
in some situations but public in others, and subclassing doesn't make 
that possible.

   Ian

dru-webware@... wrote:


> Has anyone thought about using a Java/WAR like archive
> for Webware deployments?  This would be really nice.

Agreed.

> For those who don't know, a WAR file is a Web ARchive.
> Essentially, it is like a Java JAR file... a zip of
> a bunch of class files. However a WAR has a couple
> of extra directories and an XML file describing the
> context.

We can wait for Python 2.3 which would support zip imports.

-- 
Bst rgrds, M.A.X.: Mechanical Artificial Xenomorph.

I've been trying to make a working directory into a distributable 
application.  Which it really is -- the only real problem at this point 
is figuring out where Webware is installed, otherwise it could be pretty 
drop-in.

Well, the other real problem is dealing with links -- relative links 
work fine, but it's hard to create absolute links if you take into 
account mod_rewrite and other stuff -- sorting out the semantics of the 
various parts of the URL is a mess.

As far as descriptions, that's a pending issue for Python in general.  
I'd imagine we'd want to use the same descriptions and repositories that 
everyone else uses.  It might even be cool if the working directory 
could be set up with distutils, but I'm not even sure what that would 
mean (as an application and not a library).

On Sunday, December 29, 2002, at 01:50 AM, dru-webware@... 
wrote:
>
>
> Hi,
>
> I like webware, and I've used it for a small project
> or two. I have a quick question:
>
> Has anyone thought about using a Java/WAR like archive
> for Webware deployments?  This would be really nice.
>
> For those who don't know, a WAR file is a Web ARchive.
> Essentially, it is like a Java JAR file... a zip of
> a bunch of class files. However a WAR has a couple
> of extra directories and an XML file describing the
> context.
>
> When you run a Tomcat or something like that, all
> you have to do to deploy your app is put the
> WAR file in the proper directory. This also makes
> things easy for hosting providers and deployment
> at large sites.
>
> dru
>
>
>
>
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Webware-discuss mailing list
> Webware-discuss@...
> https://lists.sourceforge.net/lists/listinfo/webware-discuss

That's a good list -- it would probably belong more to the installation 
documentation, which is separate (though it needs work too).  The 
choices are mostly pragmatic.  But it's a vague line -- obviously, 
setting up various AppServers and such is an installation issue as well, 
and ApplicationDevelopment.txt could flow in either direction (towards 
setting Webware up, or towards developing Webware/Python web 
applications).  I'm making another document (WebKit/Docs/Tutorial.txt) 
which covers the web application portion -- though for an inexperienced 
user.

Actually, I guess it's that there's two kinds of documents -- one which 
describes how you can do something (reference), and one which suggests 
how you should do something (a guide).  ApplicationDevelopment is 
intended to be a guide, and as such could probably cover most of 
installation, but in a non-complete manner.  In my opinion, only 
mod_webkit and wkcgi are worth introducing (though I haven't used 
standalone PSP pages, so I don't know about how that should best work).  
Some mod_rewrite documentation should also be included (I believe 
Quixote has some).  Then we'd ignore thinks like WebKit.cgi, 
OneShot.cgi, ModPythonAdapter.py, etc.  Those are options, but for 95%+ 
of people mod_webkit or wkcgi are the right solution, and so I think 
they should be emphasized.

Complete (and structured) reference documentation is also very 
important, but that's another issue...


On Thursday, December 26, 2002, at 01:49 PM, Stuart Donaldson wrote:
> Ian, Looks good, here are some comments.
>
> There are a couple of significantly different ways to setup and run 
> Webware.
>
> 	1) Use of global access to Webware with mod_webkit and httpd.conf
> 	referring a path such as /WK to Webware.
>
> 	2) Use of global access to Webware/PSP with mod_webkit and
> 	httpd.conf redirecting .psp pages to Webware.  However this
> 	does not currently work with PSP documents under ~user
> 	locations.
>
> 	3) Localized access where the .htaccess file redirects references
> 	through the webkit-handler. Requires mod_webkit be installed
> 	globally in httpd.conf but allows .htaccess references to
> 	unique Webware AppServer by specifying the port to use. Note
> 	that this may currently have problems with ~user expansion too.
>
> 	4) Localized access to PSP through psp-handler references
> 	in .htaccess file.  Requires mod_webkit be installed globally
> 	but allows .htaccess to reference a unique AppServer instance
> 	by specifying the port to use.  Note that this is broken
> 	currently when using ~user expansion, but a fix is in the works.
>
> 	5) Access through WebKit.cgi. I am not clear of the issues here
> 	but believe there may be some problems with ~user expansion
> 	here as well.
>
> 	6) Access through OneShot.cgi
>
> The options 1 and 2 above seem to be common.  However I believe that the
> other options also have value, especially if an ISP wants to offer 
> Webware,
> and allow users to reference their own instance of Webware.
> The problems I pointed out with the ~user expansion tell me that  
> currently,
> not many people are trying to use it in that type of an environment.  
> But I
> think the potential for large scale use there is high.
>
> I think it would be useful for the descriptions of setting up the
> environment to outline the various ways in which Webware can be 
> invoked, and
> provide a brief description.
>
> Also, given the discussions around referencing python vs python2, and 
> the
> requirement for python2, I think this should be clearly addressed.
> Unfortunately some distro's (RedHat) have "python" invoking python 1.5.2
> still.  That will cause problems for someone unless we automagically 
> fix it
> to reference python2.
>
> -Stuart-

Hi,
Ian Bicking wrote:
> installation, but in a non-complete manner.  In my opinion, only 
> mod_webkit and wkcgi are worth introducing (though I haven't used 

Please forgive my ignorance, but what is "wkcgi"? I cannot find it in the
InstallGuide. Or is it shorthand for WebKit.cgi?

ciao
-- 
Frank Barknecht

On Friday, December 27, 2002, at 07:21 AM, Frank Barknecht wrote:
> Hi,
> Ian Bicking wrote:
>> installation, but in a non-complete manner.  In my opinion, only
>> mod_webkit and wkcgi are worth introducing (though I haven't used
>
> Please forgive my ignorance, but what is "wkcgi"? I cannot find it in 
> the
> InstallGuide. Or is it shorthand for WebKit.cgi?
>
It's the C version of the CGI adapter.  It's equivalent to WebKit.cgi, 
but because it's written in C it is far faster.  In casual testing a 
long time ago, it was as fast as the ModPythonAdapter, but far easier to 
install because it's just a CGI script.  It's in 
WebKit/Adapters/wkcgi -- someone has also compiled it for Windows, which 
is easier than getting Python scripts to run in that environment 
(especially under IIS).

   Ian

Hi,
Ian Bicking schrieb:
> It's the C version of the CGI adapter.  It's equivalent to WebKit.cgi, 
> but because it's written in C it is far faster.  In casual testing a 
> long time ago, it was as fast as the ModPythonAdapter, but far easier to 
> install because it's just a CGI script.  It's in 
> WebKit/Adapters/wkcgi -- someone has also compiled it for Windows, which 
> is easier than getting Python scripts to run in that environment 

Cool. I just replaced all WebKit.cgi on my sites, and it works fine and
fast ;) 

I think, it should be made more prominent in the InstallGuide. I skipped
the WebKit.exe part, because I thought, that was only valid for
W32-systems.

ciao
-- 
Frank Barknecht

Thanks for bringing that up.  I do use an ORDER BY but for some reason 
forgot to mention it.

mike
On Thursday, December 26, 2002, at 12:42  PM, Tim Roberts wrote:

> If you will allow me to slide into SQL pedantics for a moment, you 
> should
> always use an ORDER BY clause when using LIMIT and OFFSET like that.  
> Remember
> that SQL records are, by definition, unordered.  Without an ORDER BY 
> clause,
> there is no guarantee that the record order will be the same from 
> query to
> query, so you could get unexpected gaps and repeats when fetching by 
> pages like
> this.
>
> Now, in practical terms, Postgres and MySQL all DO have an internal 
> ordering,
> so unless there are deletions and insertions in the interim, this 
> isn't a real
> concern, but since the standard allows it, someday there will be an
> implementation that takes advantage of it.  Forewarned is forearmed.

On Fri, 20 Dec 2002 16:55:20 -0500, Michael Engelhart 
<mengelhart@...> wrote:
>
>Actually it's on OS X which is BSD Unix underneath but I have noticed 
>that for example using a "cd" in a terminal window is not case 
>sensitive.  I guess BSD variants aren't necessarily case sensitive?

File name completion is usually case-insensitive, so that typing:
   cd web<tab>
will correctly expand to
   cd WebKit
but if you type "cd webkit", you should get "No such file or directory".

What's the default shell for OS X?

--
- Tim Roberts, timr@...
  Providenza & Boekelheide, Inc.

>> ve.  I guess BSD variants aren't necessarily case sensitive?
>
> File name completion is usually case-insensitive, so that typing:
>    cd web<tab>
> will correctly expand to
>    cd WebKit
> but if you type "cd webkit", you should get "No such file or 
> directory".
>
> What's the default shell for OS X?

The default shell on Mac OS X is tcsh.  The file name tab completion is 
case sensitive, but file names passed to applications are case 
_insensitive_.

[mark:~] mark% echo $shell
/bin/tcsh
[mark:~] mark% mkdir Test
[mark:~] mark% cd test
[mark:~/test] mark% pwd
/Users/mark/Test
[mark:~/test] mark%

I'm rather sure that this depends on the case sensitivity of the file 
system.  Mac OS X can use either HFS+ or UFS.  HFS+, the default and 
preferred file system, is case insensitive but case preserving.  UFS is 
case sensitive.  In general, Unix systems tend to use case sensitive 
file systems, and most "user friendly" systems tend to use case 
insensitive file systems.


-- Mark Rowe
bdash@...

I built a extranet application that pulled up to 5000 records (up to 30 
* 50char text fields and a few dates)
This was generated by a long SQL query that could take up to 12 seconds 
and here was my plan:

1) spawn a extra thread to pull the sql
2) when the sql was done update a flag in the users session
3) When Securepage.py saw that flag it added a "Report Done / View 
Report" link to the users menu
4) The thread took the entire result set and stuck into the session as a 
list of lists (Python strings and mxDateTime objects only)
5) Then the user could browser that list, resort it by any col, whatever 
they wanted.
6)  For rendering to HTML I used Cheetah Templates in a nested loop and 
only render the section of the HTML that the user is viewing

I use Dynamic Sessions but have only about 50 users at a time on a big 
server (4 Gig ram, SCSI RAIS , Dial PIII)

-Aaron

Max Ischenko wrote:
> Hi.
> 
> I have a servlet which prompts user for some values, then
> search the database using entered keys and the display the result back
> to the user. I want to break results into pieces so it would load
> more quickly and reduce server loading (generating HTML from 3000
> records take quite a few time).
> 
> Currently, it's done as this:
> - select servlet accepts input keys, runs a SQL SELECT, stores returned
>   SQL cursor to the session and redirects to browser servlet.
> - browser servlet uses c.fetchmany() to get next chunk of data and
>   display it. The 'Next' button does another c.fetchmany() and redirects
>   to itself.
> 
> These scheme is ugly and has drawbacks. Does someone know a better
> solution?
> 
> Main problems are:
> - can't reliably determine when cursor is no longer needed to close it
> - can't browse backward (also user can press Back button)
> 
> 
> 
> PS: Btw, how can I put a link into HTML page to get back (like
> pressing Back button)? I vaguelly recall that it should be some magic js
> spell.
>

Hi,
Aaron Held hat gesagt: // Aaron Held wrote:

> I built a extranet application that pulled up to 5000 records (up to 30 
> * 50char text fields and a few dates)
> This was generated by a long SQL query that could take up to 12 seconds 
> and here was my plan:
> [...]
Wow, this looks like heavy work. I'm currently working on something
similar, a record shop, where there are about 40.000 records in the
database (MySQL, flat, non-normalized schema at the moment, I'll need
to fix that, too)

I went another route in my prototype and I'd like to hear some
comments: 

I have a class "Product" that represents a Product:

> start py code
class Product:          
        def __init__(self, id):
                db = datapool.getConnection()
                c = db.cursor()
                q = '''select * from NORMAL where (id = %d)''' % id
                c.execute(q)
                res = c.fetchone()
                description = c.description
                self.data = {}
                if res: 
                        for i in range(len(description)):
                                self.data[description[i][0]] = res[i]
                db.close()
> end py code
                
The Product has methods for pretty-printing and such. Then there is a class 
for a ProductSelection, which basically is a list of IDs, *but* it
also limits the results and does the stepping:

> start py code
class ProductSelection:
        def __init__(self, clause=''):
                self.productIds = self.selectByClause(clause)
		
	def selectByClause(self, clause='', start=0, count=10):
		if clause:
			db = datapool.getConnection()
			c = db.cursor()
			q = '''select id from normal where (%s) order by artist''' % clause
			c.execute(q)
			# where do we start?
			startcount = start * count
			# where's the end, add 1, to see if there are
			# more results after this set:
			endcount   = startcount + count + 1
			res = c.fetchall()[startcount:endcount]
			db.close()
			return res
		else:
			return []
> end py code

Now in the Page code, this is easy to use:

> start py code
	stepsize = 10	# global stepsize variable
	startWhere = int(self.request().value("startProducts"))
	ps = ProductSelection(someClause, start=startWhere, count=stepsize )
	for id in ps.productIds[:-1]:
		p = Product(id)
		p.show()
		
	if len(ps.productIds) > stepsize:
		stpplus = stp + 1
		self.writeln('''
		<a href="?startProducts=%s"><b>To the Next %s results</b></a>
		''' % (stpplus, stepsize))

> end py code


Now my hope is (I didn't profile it, yet), that this is fast enough,
because the IDs are int fields and in the worst case ("select * from
NORMAL") I have a c.fetchall() that returns 40.000 ints. But this is
sliced to be in a range [startcount:endcount], so it actually is only
used for a very short time, if at all, isn't it?

OTOH I have another select for every Product and there might be a lot
of Products in memory. At least, we hope that people look at the
Products a lot... :)

ciao
-- 
 Frank Barknecht                               _ ______footils.org__

I would tend to let the database handle as much of this as possible.   
There are many SQL additions that can help here.

I use PostgreSQL and you can do something like:

SELECT * FROM my_table LIMIT 25 OFFSET 25

which would return 25 rows starting with the 25th row in the result set.

mySQL has something similar I believe which is something like:
SELECT * ...  LIMIT 5,10

where 5 and 10 are an upper and lower bounds.

Using these database features means you only need to keep track of 
where a particular user is in the cursor as 2 ints.

Like
cursor_start = 10
cursor_end = cursor_start + 25

Then you could have forward and backward links that just call the 
servlet and add or subtract from those cursor integers.

Much easier and a lot less memory then trying to have 40,000 rows in 
memory for each user.  That can easily get out of hand if you have more 
than a couple of users.

Also, you using LIMIT you can have a drop down menu on your search form 
that sets the limit so they can retrieve 10, 20, 50, 100, 1000 or 
whatever terms you want to show the user.

hope this helps

Mike

On Monday, December 23, 2002, at 03:09  AM, Max Ischenko wrote:

> Hi.
>
> I have a servlet which prompts user for some values, then
> search the database using entered keys and the display the result back
> to the user. I want to break results into pieces so it would load
> more quickly and reduce server loading (generating HTML from 3000
> records take quite a few time).
>
> Currently, it's done as this:
> - select servlet accepts input keys, runs a SQL SELECT, stores returned
>   SQL cursor to the session and redirects to browser servlet.
> - browser servlet uses c.fetchmany() to get next chunk of data and
>   display it. The 'Next' button does another c.fetchmany() and 
> redirects
>   to itself.
>
> These scheme is ugly and has drawbacks. Does someone know a better
> solution?
>
> Main problems are:
> - can't reliably determine when cursor is no longer needed to close it
> - can't browse backward (also user can press Back button)
>
>
>
> PS: Btw, how can I put a link into HTML page to get back (like
> pressing Back button)? I vaguelly recall that it should be some magic 
> js
> spell.
>
> -- 
> Bst rgrds, M.A.X.: Mechanical Artificial Xenomorph.
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Webware-discuss mailing list
> Webware-discuss@...
> https://lists.sourceforge.net/lists/listinfo/webware-discuss
>

<snip>
> I went another route in my prototype and I'd like to hear some
> comments:
>

No shortage of opinions here!

> I have a class "Product" that represents a Product:
>
>> start py code
> class Product:
>        def __init__(self, id):
>                db = datapool.getConnection()
>                c = db.cursor()
>                q = '''select * from NORMAL where (id = %d)''' % id
>                c.execute(q)
>                res = c.fetchone()
>                description = c.description
>                self.data = {}
>                if res:
>                        for i in range(len(description)):
>                                self.data[description[i][0]] = res[i]
>                db.close()
>> end py code
>
> The Product has methods for pretty-printing and such. Then there is a
> class  for a ProductSelection, which basically is a list of IDs, *but*
> it also limits the results and does the stepping:
>
>> start py code
> class ProductSelection:
>        def __init__(self, clause=''):
>                self.productIds = self.selectByClause(clause)
>
> 	def selectByClause(self, clause='', start=0, count=10):
> 		if clause:
> 			db = datapool.getConnection()
> 			c = db.cursor()
> 			q = '''select id from normal where (%s) order by artist''' % clause
> 			c.execute(q)
> 			# where do we start?
> 			startcount = start * count
> 			# where's the end, add 1, to see if there are
> 			# more results after this set:
> 			endcount   = startcount + count + 1
> 			res = c.fetchall()[startcount:endcount]
> 			db.close()
> 			return res
> 		else:
> 			return []
>> end py code
>
> Now in the Page code, this is easy to use:
>
>> start py code
> 	stepsize = 10	# global stepsize variable
> 	startWhere = int(self.request().value("startProducts"))
> 	ps = ProductSelection(someClause, start=startWhere, count=stepsize )
> 	for id in ps.productIds[:-1]:
> 		p = Product(id)
> 		p.show()
>
> 	if len(ps.productIds) > stepsize:
> 		stpplus = stp + 1
> 		self.writeln('''
> 		<a href="?startProducts=%s"><b>To the Next %s results</b></a>
> 		''' % (stpplus, stepsize))
>
>> end py code
>
>
> Now my hope is (I didn't profile it, yet), that this is fast enough,
> because the IDs are int fields and in the worst case ("select * from
> NORMAL") I have a c.fetchall() that returns 40.000 ints. But this is
> sliced to be in a range [startcount:endcount], so it actually is only
> used for a very short time, if at all, isn't it?
>
> OTOH I have another select for every Product and there might be a lot
> of Products in memory. At least, we hope that people look at the
> Products a lot... :)
>
> ciao
> --
> Frank Barknecht                               _ ______footils.org__
>

This is one of the areas where Webware/Python really shines and you can do a
lot of optimizations here with relatively little effort.

The product class looks good, but I would take a look at the selectot class.
 It looks to me that for each page you are:
1) creating a new instance of a python
2) Pulling lots of data from a sql db
3) Pulling all of the records from the native sql format into python
4) Slicing that array.

A couple of ideas
1) Cache the instance of ProductSelection in the users session and only go
back to the database if the 'clause' changes. (Stick the full resultset in
self._results and slice on that.
... Updates to the database will only be seen by new users, since old ones
will be looking at a snapshot
2) Follow version 1, but store in an application level cache.  There are
some good caching routines out there. You can set it up to store a result
set for 30 min or something like that.
3) Mixin ProductSelection to SitePage, and then pass the start and stop
right to the SQL and pull only the interesting records. (lose the init and
call self.selectByClause)

Finally: replace all of the self.writeln's w/ a few good cheetah template
fragments.  It will make life easier when batching.

If you are using mysql w/ simple clauses then you should have no problem
going to sql for each page.  I make extensive use of caching when the SQL
calls take a long time (seconds) or there are complex transformations (xml)
involved.

-Aaron Held

Hi,
Michael Engelhart hat gesagt: // Michael Engelhart wrote:

> I use PostgreSQL and you can do something like:
> 
> SELECT * FROM my_table LIMIT 25 OFFSET 25
> 
> which would return 25 rows starting with the 25th row in the result set.
> 
> mySQL has something similar I believe which is something like:
> SELECT * ...  LIMIT 5,10

Yep, you made my day. I was always using Postgres but the shop must
run on MySQL and this is the offset syntax I needed. 

ciao
-- 
 Frank Barknecht                               _ ______footils.org__