Webware for Python

Bugs item #696027, was opened at 2003-03-02 08:48
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=104866&aid=696027&group_id=4866

Category: WebKit
Group: None
>Status: Closed
>Resolution: Fixed
Priority: 5
Submitted By: Dietmar Schwertberger (dschwertberger)
Assigned to: Nobody/Anonymous (nobody)
Summary: filenamesForBaseNameNew

Initial Comment:
In Application.filenamesForBaseNameNew()
regex matches for self._filesToServeRegexes
are ignored because continue only continues the
inner for loop:

            for regex in self._filesToHideRegexes:
                if regex.match(shortFilename):
                    continue

Could be fixed with:

            shouldServe = 1
            for regex in self._filesToHideRegexes:
                if regex.match(shortFilename):
                    shouldServe = 0
                    break
            if not shouldServe:
                 continue


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=104866&aid=696027&group_id=4866

Hallo,
Chuck Esterbrook hat gesagt: // Chuck Esterbrook wrote:

> I disagree that UserKit hinders further development. It could be 
> extended by *anyone*, or looked at for ideas prior to writing a 
> "UserKit" from scratch or completely ignored. No one's hands are 
> bound...

I didn't mean to imply, that UserKit hinders development by bad design
or such, I didn't want judge this at all. In fact, I like the design
in general. I absoluetly agree that UserKit could be made to work.

I just wanted to say, that obviously several Webware developers aren't
using UserKit for whatever reasons. Instead they built a custom
solution.  They never published this because "Webware already has
UserKit". So UserKit remained unfinished, at least in regard to
alternative storage backends. I admit, that I never tried UserKit with
MiddleKit, because I didn't have access to MySQL on my server and also
because I didn't want to be forced to use MK. 

So as open source development goes, I tried for some time to fix what
I missed in UserKit, but of course this raised a lot of questions.
When I asked these questions here regarding UserKit with file storage
several months ago I simply got no useful answers. I really don't
blame anyone for this, but that's just the way it was.

ciao
-- 
 Frank Barknecht                               _ ______footils.org__

I'll have to profile the whole app, but I think that the poor machine is
plainly overworked. I'll ask the local Oracle guru on Monday. Is it me
or Oracle _needs_ a guru by its side? Sybase and MSSQL weren't so picky.

I am giving up on expensive things.

El jue, 27-02-2003 a las 22:12, Edmund Lian escribi=F3:
>=20
> On 02/27/2003 03:55:13 PM webware-devel-admin wrote:
>=20
> >I am stress-testing a Webware 0.7 app on Solaris (old) Oracle 7 and
> >apache-ssl 1.3. The results are very poor.
>=20
> Poor performance is more likely due to the way Oracle has been set up, or
> the way the DB is used than Webware itself. E.g., are your Oracle
> transaction logs on a different physical disk from the DB? How much memor=
y
> has Oracle been given? What kinds of queries are you making, are the tabl=
es
> you are using indexed appropriately, etc., etc.
>=20
> ...Edmund.
>=20

On 02/27/2003 04:36:02 PM Marcos S=E1nchez Provencio wrote:

>I'll have to profile the whole app, but I think that the poor machine =
is
>plainly overworked. I'll ask the local Oracle guru on Monday. Is it me=

>or Oracle _needs_ a guru by its side? Sybase and MSSQL weren't so pick=
y.

It definitely isn't you. Oracle is a pain in the ass to install/manage.=

It's a resource hog, and wants lots of memory, lots of semaphores, etc.=
,
etc. And it isn't self-tuning. Sybase and its cousing MS SQL are defini=
tely
on a different plane when it comes to these things. Have you looked at
PostgreSQL? It's very Oracle-like in its architecture, but is a snap to=

install/manage. You have to get the memory buffer sizing, etc. correct =
too,
like Oracle.

...Edmund.=

I'd laugh if I could. This is an application that was begun for a
mainframe maaany years ago and has been translated with the minimum
change (effort I would say) through different platforms since then. I
would dump the whole thing If I could.

So, my next suggestion will be leaving that dinosaur alone and have a
Linux box by its side to deal with the app server and the apache-ssl
business. Or better, two. They will cost as much as the memory upgrade
that the Sun is going to need.

Thank you again
--=20
Marcos, trying to get some free software into the Spanish Public
Administration

El vie, 28-02-2003 a las 00:02, Edmund Lian escribi=F3:
>=20
> On 02/27/2003 04:36:02 PM Marcos S=E1nchez Provencio wrote:
>=20
> >I'll have to profile the whole app, but I think that the poor machine is
> >plainly overworked. I'll ask the local Oracle guru on Monday. Is it me
> >or Oracle _needs_ a guru by its side? Sybase and MSSQL weren't so picky.
>=20
> It definitely isn't you. Oracle is a pain in the ass to install/manage.
> It's a resource hog, and wants lots of memory, lots of semaphores, etc.,
> etc. And it isn't self-tuning. Sybase and its cousing MS SQL are definite=
ly
> on a different plane when it comes to these things. Have you looked at
> PostgreSQL? It's very Oracle-like in its architecture, but is a snap to
> install/manage. You have to get the memory buffer sizing, etc. correct to=
o,
> like Oracle.
>=20
> ...Edmund.
>=20

Survey addition:

---
If you would like to provide a testimonial for Webware that we can use 
on the web site, please write it here including how you want it signed 
(full name vs. initials, email address or not).
---

-Chuck


On Wednesday 26 February 2003 01:04 pm, Chuck Esterbrook wrote:
> PLEASE NOTE: DO NOT ANSWER THE INCLUDED SURVEY!
>
> With the PyCon Webware Sprint coming up and given that we've never
> done a survey, I think one is in order.
>
> Here is what I have come up with. I'm looking for feedback from you
> guys (add/subtract/alter questions) before I post on -discuss.
>
> So take a look and make suggestions if you have any.
>
> Also, if someone wants to whip up a Webware page to collect this
> info, or make use of some canned survey-web solution, speak up.
> Otherwise, I'll just use email and collate by hand.
>
> Thanks - Chuck
>
> ---
>
> What version(s) of Python do you use?
>
> What op sys's do you use?
>
> Put an X after the Webware packages that you directly use:
> 	CGIWrapper
> 	MiddleKit
> 	MiscUtils
> 	PSP
> 	TaskKit
> 	UserKit
> 	WebKit
> 	WebUtils
>
> What third party Webware-related/specific packages do you use?
> Examples: Cheetah, FunFormKit
>
> Do you use PostgreSQL?
>   If so, what version?
>   What DB API adapter and version do you use?
>   Do you use MiddleKit with Postgres?
>   If you have an MK/Postgres patch, please attach your latest version
>   if you're willing to donate it).
>
> Besides MySQL (which is already well supported) and PostgreSQL (which
> was just inquired about) are there other databases that you use?
> State version number and applicable Python modules.
>
> Are you going to PyCon? (http://www.python.org/pycon/)
>   If so, are you going to the PyCon Sprint?
>   (http://www.python.org/cgi-bin/moinmoin/WebwareSprint)
>
> Put an X after the places that you use Webware:
>   workplace in permanent position
>   workplace as contractor
>   workplace as self/co-owned business (outside of contracting)
>   school as teacher
>   school as student
>   learning/hobby/tinkering
>   other: specify: ____
>
> Everyone always wants better docs. What specific doc improvements
> would you like to see in Webware?
>
> What else would you like to see in future versions of Webware?
> (list in order of preference)
>
>
> Thanks for completing the survey!
>
> P.S. Don't forget to read and share at the Wiki:
>      http://webware.sf.net/wiki/
>
> ---
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Scholarships for Techies!
> Can't afford IT training? All 2003 ictp students receive
> scholarships. Get hands-on training in Microsoft, Cisco, Sun,
> Linux/UNIX, and more. http://www.ictp.com/training/sourceforge.asp
> _______________________________________________
> Webware-devel mailing list
> Webware-devel@...
> https://lists.sourceforge.net/lists/listinfo/webware-devel

-- 
Chuck
http://ChuckEsterbrook.com

Matt,

That's a good point, but my advice is to not wait for any surveys or 
other events to pull together particular efforts. If you have an 
interest, and especially the code to go with it, feel free to start 
something.

At any time, feel free to post to -discuss and say:
  "I've got the beginnings of FooKit. Anyone care to join me in
   turning this into a reusable, public product?"

You can use the lists and the wiki as you need to. The kit in question 
could be adopted into Webware (like TaskKit) or stand as a separate 
project (like Cheetah).

-- 
Chuck
http://ChuckEsterbrook.com


On Wednesday 26 February 2003 02:42 pm, Matt Feifarek wrote:
> Chuck Esterbrook wrote:
> >Here is what I have come up with. I'm looking for feedback from you
> > guys (add/subtract/alter questions) before I post on -discuss.
>
> What about something like:
>
> I have my own librar(ies) to solve the following problems:
> - User Validation
> - Persistence
> - ORM
> - (etc.)
> - Webdav
>
> I'm willing to share this code?
>
> I get the sense that lots of us have "rolled our own" for common
> things; it would be nice to start scraping stuff together into new
> kits.
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Scholarships for Techies!
> Can't afford IT training? All 2003 ictp students receive
> scholarships. Get hands-on training in Microsoft, Cisco, Sun,
> Linux/UNIX, and more. http://www.ictp.com/training/sourceforge.asp
> _______________________________________________
> Webware-devel mailing list
> Webware-devel@...
> https://lists.sourceforge.net/lists/listinfo/webware-devel

On Wed, 26 Feb 2003, Chuck Esterbrook wrote:

> Matt,
>
> That's a good point, but my advice is to not wait for any surveys or
> other events to pull together particular efforts. If you have an
> interest, and especially the code to go with it, feel free to start
> something.

I'm going to put words into Matt's mouth, or at least give you my reading
of what he said :) I think there are plenty of people out there who have
code to do various bits, but don't feel it's "release quality", and
therefore put off doing anything with it outside of their project.

For instance, I don't see any releases from me, you, or Geoff concerning
Object{Renderer,Viewer,Editor} and whatever alternative to that you rolled
for StockAlerts :)

Putting that question on the survey, and using the survey responses as a
catalyst and / or guide for the sprints, I think we might dig up a bunch
of useful code that just needs an hour or three of polish to be releasable
enough to get people started thinking about it, using it, tweaking it,
etc.

In other words, it might get people over the "just do something" hump.

Tripp Lilley wrote:

>Putting that question on the survey, and using the survey responses as a
>catalyst and / or guide for the sprints, I think we might dig up a bunch
>of useful code that just needs an hour or three of polish to be releasable
>enough to get people started thinking about it, using it, tweaking it,
>etc.
>
That's a fair approximation of my intent. Thanks.

But Chuck's point is good, too. We don't need any sanction to release 
things. But lots of us probably have code that we don't feel "is ready" 
for things. The survey question might help us find what projects have a 
critical mass.

Tripp Lilley wrote:

>On Wed, 26 Feb 2003, Chuck Esterbrook wrote:
>
>  
>
>>Matt,
>>
>>That's a good point, but my advice is to not wait for any surveys or
>>other events to pull together particular efforts. If you have an
>>interest, and especially the code to go with it, feel free to start
>>something.
>>    
>>
>
>I'm going to put words into Matt's mouth, or at least give you my reading
>of what he said :) I think there are plenty of people out there who have
>code to do various bits, but don't feel it's "release quality", and
>therefore put off doing anything with it outside of their project.
>
>For instance, I don't see any releases from me, you, or Geoff concerning
>Object{Renderer,Viewer,Editor} and whatever alternative to that you rolled
>for StockAlerts :)
>
>Putting that question on the survey, and using the survey responses as a
>catalyst and / or guide for the sprints, I think we might dig up a bunch
>of useful code that just needs an hour or three of polish to be releasable
>enough to get people started thinking about it, using it, tweaking it,
>etc.\
>
If there are several people out there with partial solutions that are 
not really polished for release, maybe we need a repository for such 
solutions.  Some place that was admittedly weak on documentation, but 
that people could post their solutions. They could certainly be 
educational for some people, and might spur someone else on to help add 
the requisite polish to make it ready for release?

What about a loosely structured "contrib" directory in CVS?  The down 
side to that would be it would be difficult to manage and cleanup after 
a while.  Perhaps just a section on the Wiki?  What other forums might 
work for such a thing?

-Stuart-

On Thursday 27 February 2003 08:54 pm, Stuart Donaldson wrote:
> If there are several people out there with partial solutions that are
> not really polished for release, maybe we need a repository for such
> solutions.  Some place that was admittedly weak on documentation, but
> that people could post their solutions. They could certainly be
> educational for some people, and might spur someone else on to help
> add the requisite polish to make it ready for release?
>
> What about a loosely structured "contrib" directory in CVS?  The down
> side to that would be it would be difficult to manage and cleanup
> after a while.  Perhaps just a section on the Wiki?  What other
> forums might work for such a thing?
>
> -Stuart-

This is exactly what I was going to write back. So thanks! :-)

But let me add that I think a Webware-driven site could be written for 
this! This could fulfill the recent call for an interesting and useful 
app to be developed and maintained by the Webware community as a public 
example. *And* this site could be for the entire Python community, 
thereby giving Webware more exposure ("powered by Webware" on every 
footer).

My ideas for such a site:

- Name: CodeHeap
- Purpose: Browse, search and download Python code that you _may_ just 
find useful.
- Functions:
  - Upload code, anonymous or not.
  - See recent uploads.
  - Search by name, author, date, filename, keywords, description,
    *code contents*.

Basically if you had some potentially reusable Python code that wasn't 
baked enough or that you didn't want to support, you could toss it in 
the CodeHeap.

This reminds me of "Useless Python" but on a grander scale and without 
the "Useless".   :-)

Example use: If I'm about to write some DAV code, I first search 
CodeHeap for "DAV" and get to look through several hits ranging from 
individual modules to entire packages. I can read the code right in my 
browser, syntax highlighted, even if the original package was a tar 
ball.


Regarding the survey, I don't disagree with Tripp et al, but I don't 
really want to collate all the responses I'd get for a question in this 
area. On the bright side, shaking this code out of the bushes can be a 
project/effort all its own.

-- 
Chuck
http://ChuckEsterbrook.com

On Thu, 2003-02-27 at 22:54, Stuart Donaldson wrote:
> What about a loosely structured "contrib" directory in CVS?  The down
> side to that would be it would be difficult to manage and cleanup
> after a while.  Perhaps just a section on the Wiki?  What other forums
> might work for such a thing?

Well, docutils has a sandbox module for this purpose, with user
directories under that.  Each person does whatever they want in their
directory, so there's no particular effort at cleanliness.

It's a place where you can put files and modules that you don't feel
like packaging up, but still might be of use.  It's also a place to put
stuff that's in development, or may not be directly related to Webware. 
But it's still easy to share.

  Ian

On Thursday 27 February 2003 10:34 pm, Ian Bicking wrote:
> On Thu, 2003-02-27 at 22:54, Stuart Donaldson wrote:
> > What about a loosely structured "contrib" directory in CVS?  The
> > down side to that would be it would be difficult to manage and
> > cleanup after a while.  Perhaps just a section on the Wiki?  What
> > other forums might work for such a thing?
>
> Well, docutils has a sandbox module for this purpose, with user
> directories under that.  Each person does whatever they want in their
> directory, so there's no particular effort at cleanliness.
>
> It's a place where you can put files and modules that you don't feel
> like packaging up, but still might be of use.  It's also a place to
> put stuff that's in development, or may not be directly related to
> Webware. But it's still easy to share.
>
>   Ian

That's fine with me. Shall we call it "sandbox" and ask users to make 
directories with their SF user name?

And should this really be in Webware, or should we make another SF 
project so we can loosen up the cvs-write requirements and let 
everybody in?

--
Chuck
http://ChuckEsterbrook.com

On Fri, 2003-02-28 at 00:42, Chuck Esterbrook wrote:
> That's fine with me. Shall we call it "sandbox" and ask users to make 
> directories with their SF user name?
> 
> And should this really be in Webware, or should we make another SF 
> project so we can loosen up the cvs-write requirements and let 
> everybody in?

I like the idea of making it a separate project, because then we can
really give access to anybody at all without any care.  I don't think
it's too hard to ask that people stay in the sandbox module for a while,
but we'd at least have to know who the person is and... I dunno.  We
could give sandbox access to absolutely anyone who wanted it.

webware-sandbox.sf.net ?

  Ian

On Thursday 27 February 2003 11:24 pm, Ian Bicking wrote:
> On Fri, 2003-02-28 at 00:42, Chuck Esterbrook wrote:
> > That's fine with me. Shall we call it "sandbox" and ask users to
> > make directories with their SF user name?
> >
> > And should this really be in Webware, or should we make another SF
> > project so we can loosen up the cvs-write requirements and let
> > everybody in?
>
> I like the idea of making it a separate project, because then we can
> really give access to anybody at all without any care.  I don't think
> it's too hard to ask that people stay in the sandbox module for a
> while, but we'd at least have to know who the person is and... I
> dunno.  We could give sandbox access to absolutely anyone who wanted
> it.
>
> webware-sandbox.sf.net ?

Great. I just submitted the SF project. Hopefully they won't balk at the 
idea.

I guess if they do, we'll just make a cvs repository on somebody's 
server. But let's go for SF first.

-- 
Chuck
http://ChuckEsterbrook.com

On Thursday 27 February 2003 09:50 pm, Edmund Lian wrote:
> On 02/28/2003 12:35:47 AM Chuck wrote:
> >My ideas for such a site:
> >
> >- Name: CodeHeap
> >- Purpose: Browse, search and download Python code that you _may_
> > just find useful.
> >- Functions:
> >  - Upload code, anonymous or not.
> >  - See recent uploads.
> >  - Search by name, author, date, filename, keywords, description,
> >    *code contents*.
>
> Doesn't this sound like an extended Vaults of Parnassus?
>
> ...Edmund.

There is certainly crossover, but I think it depends on what kind of 
focus you took. Are you building a pristine catalog of Python 
resources, or a junkyard of Python parts? IMO those two would lead to 
different feature sets.

As one example, the pristine catalog should have the option to scrape 
your project info from SourceForge or FreshMeat. It should probably 
also email contributors every 6 months if they haven't updated their 
project records. The junkyard wouldn't do either of those. But it might 
provide the storage and downloads cuz when you throw your spare parts 
in the 'yard, you don't want any burdens afterwards (like maintaining 
your own site to vend the parts).

That's my view anyway. You could try to merge these two visions or even 
provide two different "areas" at one major site.

-- 
Chuck
http://ChuckEsterbrook.com

On Thu, 2003-02-27 at 14:55, Marcos S=E1nchez Provencio wrote:
> I am stress-testing a Webware 0.7 app on Solaris (old) Oracle 7 and
> apache-ssl 1.3. The results are very poor.
>=20
> Are there any performance reasons to go 0.8?
> Is there any reason for performance to be poor on Solaris?

There aren't any big performance improvements in 0.8.  I can't see why
Solaris would make a difference -- even if thread creation is slow (not
sure if that's the case on Solaris or not), the threads aren't created
that often (unless perhaps you are doing a ton of concurrent requests --
you could turn up the number of pre-allocated threads in
AppServer.config).

Are you pooling connections to the database (via MiscUtils.DBPool)?=20
That's an important performance measure.

What does top look like when you are stress testing it?  Is Python
(i.e., the AppServer) taking up most of the CPU?

  Ian

Thanks for answering (see between lines)

El jue, 27-02-2003 a las 22:06, Ian Bicking escribi=F3:
> On Thu, 2003-02-27 at 14:55, Marcos S=E1nchez Provencio wrote:
> > I am stress-testing a Webware 0.7 app on Solaris (old) Oracle 7 and
> > apache-ssl 1.3. The results are very poor.
> >=20
> > Are there any performance reasons to go 0.8?
> > Is there any reason for performance to be poor on Solaris?
>=20
> There aren't any big performance improvements in 0.8.  I can't see why
> Solaris would make a difference -- even if thread creation is slow (not
> sure if that's the case on Solaris or not), the threads aren't created
> that often (unless perhaps you are doing a ton of concurrent requests --
> you could turn up the number of pre-allocated threads in
> AppServer.config).

I'll have some threads/users/performance charts soon. Is there any magic
number ceiling in Webware, or only what common sense suggests?

>=20
> Are you pooling connections to the database (via MiscUtils.DBPool)?=20
> That's an important performance measure.
I have a modified version, since every user uses his own login. Funny
though, Oracle connection is very fast in this platform.

>=20
> What does top look like when you are stress testing it?  Is Python
> (i.e., the AppServer) taking up most of the CPU?

Actually not, it is Oracle, plus it is iowaiting 21% of the time. I
don't think I am going to find an easy solution...