From: Chris P. <cp...@fo...> - 2002-06-04 01:00:55
|
All, I just started using WebWare. I'm in the process of converting an application that I wrote for Python on IIS using the Python/ASP integration. I like what I see in Webkit so far, and the PSP implementation is making it pretty quick to move my asp pages over, since it's so similar. The bulk of my code was not in ASP, it's in plain python modules, so that helped (but I did briefly try moving this over to Zope, and that was going nowhere fast). There are a couple of things I have a question about however. In HTTPRequest.__init__, there's a bunch of code that set's up the request data, in particular the initialization of the fields from self._input. The problem is that the code assumes the incoming Content-type is application/x-www-form-urlencoded, although it doesn't explicity check that is the case. If the incoming data has a different content type, the data is lost once the self._input stream is passed to FieldStorage.FieldStorage becase it consumes the stream. You can see two special cases for 'text/xml' and 'text/python/pickled/dict' that do something with self._input before FieldStorage gets to it, but it's not a very extensible method of doing things. There aren't any places in that __init__ method to dispatch different handlers based on content-type, so even sublcassing is going to duplicate a lot of code. It looks like there might have been some thoughts about giving access to the original request through rawRequest, but I don't see where HTTPRequest._rawRequest is actually set. BTW, the reason for all this is that my application is a client and/or server for some web services type operations that are not form encoded (or in xml... else I could have used xmlInput). I need to be able to access the POSTed data because it might be MIME encoded, newline delimeted name-value (like java properties file), etc. In IIS/ASP land, I was just using the Request.Form() (no parameters) to grab the data. Another wrinkle is that in early stages of this, I wasn't even able to rely on the content-type being set, so I had to read in the POST data and figure out the format being used through various rules. It's easy enough for me to add some code to HTTPRequest for now, but seems like there should be a more general solution. Regards, Chris Prinos |
From: Ian B. <ia...@co...> - 2002-06-04 05:47:26
|
On Mon, 2002-06-03 at 19:58, Chris Prinos wrote: > In HTTPRequest.__init__, there's a bunch of code that set's up the request > data, in particular the initialization of the fields from self._input. The > problem is that the code assumes the incoming Content-type is > application/x-www-form-urlencoded, although it doesn't explicity check that > is the case. If the incoming data has a different content type, the data is > lost once the self._input stream is passed to FieldStorage.FieldStorage > becase it consumes the stream. You can see two special cases for 'text/xml' > and 'text/python/pickled/dict' that do something with self._input before > FieldStorage gets to it, but it's not a very extensible method of doing > things. There aren't any places in that __init__ method to dispatch > different handlers based on content-type, so even sublcassing is going to > duplicate a lot of code. Yes, this does appear to be a problem. There is some testing in FieldStorage as to the content-type, but if no content-type is given it assumes application/x-www-form-urlencoded. I assume this is because some clients do not set the content-type, and that this behavior should maintained for compatibility. I was looking at the FieldStorage code, and I couldn't quite figure out what happens when it gets a different content-type -- it seems to keep it somewhere (directly in .value of the FieldStorage instance?) If it does, I don't believe the Webware code will give access to it -- changing this would make xmlInput/dictInput unnecessary and probably be better all around. However, this doesn't solve your problem as you were talking about non-urlencoded input with no content-type to indicate that... if at all possible, I'd change these clients. text/xml will work (though be inaccurate), while text/plain seems more appropriate. To capture the input with no content-type, there'd have to be some setting to copy the input before it was sent to FieldStorage (it probably shouldn't do this by default, as it would consume more memory and web services *should* pass proper content-types, though that won't help you yet). I feel like any dispatching can probably best be done in the servlet itself. Ian |
From: <ir...@ms...> - 2002-06-04 14:20:51
|
On Tue, Jun 04, 2002 at 12:47:07AM -0500, Ian Bicking wrote: > However, this doesn't solve your problem as you were talking about > non-urlencoded input with no content-type to indicate that... if at all > possible, I'd change these clients. text/xml will work (though be > inaccurate), while text/plain seems more appropriate. To capture the > input with no content-type, there'd have to be some setting to copy the > input before it was sent to FieldStorage (it probably shouldn't do this > by default, as it would consume more memory and web services *should* > pass proper content-types, though that won't help you yet). Having a setting sounds like a good route. If it's off, it won't cause any overhead. If it's on, it will allow the maximum flexibility both for servlets using POST variables and for servlets using stdin for some other purpose, even if both servlets exist in the same application, and even if some browsers forget to set the Content-type. We'd just have to make sure FieldStorage doesn't blow up if given random input. -- -Mike (Iron) Orr, ir...@ms... (if mail problems: ms...@oz...) http://iron.cx/ English * Esperanto * Russkiy * Deutsch * Espan~ol |
From: Aaron H. <aa...@me...> - 2002-06-04 15:18:28
|
Hi all, I've been meaning to put up a webware blog for a while now, and there is a preliminary version up at http://webware.metrony.com/portal/ I am currently writing some search code to let you search the main webware site, WIKI, mailing lists, as well as this site from one page. (Google is great thing) Anyway, please take a look and tell me what you think. If anyone would be willing to help update it I'll pass out admin rights. I want to focus on getting the tutorials done. Currently there are no web links and only a few downloads listed. Thanks, -Aaron Held |
From: Frank B. <bar...@ph...> - 2002-06-04 17:54:15
|
Hi, Aaron Held hat gesagt: // Aaron Held wrote: > I've been meaning to put up a webware blog for a while now, and there is a preliminary > version up at http://webware.metrony.com/portal/ "Life's too short to compile" - I like that. > I am currently writing some search code to let you search the main webware site, WIKI, > mailing lists, as well as this site from one page. (Google is great thing) > > Anyway, please take a look and tell me what you think. I see, this is done with PostNuke, which of course is very capable, but I wonder, if such a site shouldn't be running something in webware and would thus function as a kind of "Eat your own stuff"-Dogbowl like Zope did with the CMF-site. ciao, -- Frank Barknecht _ ______footils.org__ |
From: Aaron H. <aa...@me...> - 2002-06-04 18:58:50
|
> I see, this is done with PostNuke, which of course is very capable, > but I wonder, if such a site shouldn't be running something in webware > and would thus function as a kind of "Eat your own stuff"-Dogbowl like > Zope did with the CMF-site. > > ciao, > -- > Frank Barknecht _ ______footils.org__ Yes, I agree completely. I worked with another Java app server a while ago and also complained that they used php for thier site rather then thier own product. PostNuke has a very capable admin system and it does the job. It is, however, a mess to extend. A neat, clean python based system would be a huge benefit to people using this software as an Intranet. I have been working on a webware portal, but it is no where near ready to use. I am following the same database schema and module naming conventions that postnuke uses, so the idea is a drop-in replacement for nuke sites. My goal is to slowly replace some php functionality with webware code, staring with the "client" facing html and slowly working my way to the admin side. Of course my goal was also to write a tutorial, and that is only about 25% done as well. But, PostNuke runs on my server, the webware version runs only in my head (but its less buggy!) Thanks, -Aaron Held |
From: Chris P. <cp...@fo...> - 2002-06-04 21:34:02
|
> Yes, this does appear to be a problem. There is some testing in > FieldStorage as to the content-type, but if no content-type > is given it > assumes application/x-www-form-urlencoded. I assume this is because > some clients do not set the content-type, and that this > behavior should > maintained for compatibility. I was looking at the FieldStorage code, > and I couldn't quite figure out what happens when it gets a different > content-type -- it seems to keep it somewhere (directly in > .value of the > FieldStorage instance?) If it does, I don't believe the Webware code > will give access to it -- changing this would make xmlInput/dictInput > unnecessary and probably be better all around. > I think I side tracked things a bit when I mentioned the part about clients not setting the content-type, though is still relavant. The real issue is that HTTPRequest.__init__ doesn't give you a chance to inspect the orginal POST data. I thought this was odd at first, becuase cgi.FieldStorage lets you get to the POST data (in fact it does it's own content type handling and even tries to handle multi part) I looked ath things some more, and about 2/3 of the way down the code in HTTPRequest.__init__ (line 113 in the .7 release), there's the killer line: self._fields = dict This takes dict (the environment dictionary) and overwrites self._fields, which to this point had been a FieldStorage.FieldStorage instance. The intent is to make request().fields() just return a simple dictionary, but once this is done you lose all the info that the FieldStorage object kept. So if you replace line 113 with, for instance: self._fieldStorage = self._fields self._fields = dict and to keep thing consistent add the method def fieldStorage(self): return self._fieldStorage you can now access the POST data by doing the following from a servlet: postData = self.request().fieldStorage().file.read() similarly HTTPRequest's xmlInput() and dictInput() method could have just returned self._fieldStorage.file without any of the current special case code. Also unlike the limited file like object you get back from socket.make_file, the fieldStorage().file object can be rewound with seek(0). > However, this doesn't solve your problem as you were talking about > non-urlencoded input with no content-type to indicate that... > if at all > possible, I'd change these clients. text/xml will work (though be > inaccurate), while text/plain seems more appropriate. To capture the > input with no content-type, there'd have to be some setting > to copy the > input before it was sent to FieldStorage (it probably > shouldn't do this > by default, as it would consume more memory and web services *should* > pass proper content-types, though that won't help you yet). I think with the above changes, you get the benefits without the downside. No copy is made (the FieldStorage instance has already been created), the only difference is that you keep the reference to it around so you can use it later I'll play with this a bit and put in a bug report at the webware site. Chris |
From: <ir...@ms...> - 2002-06-04 18:44:49
|
On Tue, Jun 04, 2002 at 07:40:41PM +0200, Frank Barknecht wrote: > I see, this is done with PostNuke, which of course is very capable, > but I wonder, if such a site shouldn't be running something in webware > and would thus function as a kind of "Eat your own stuff"-Dogbowl like > Zope did with the CMF-site. I think we're all hoping for that someday, but first somebody has to write the content management system. Specifications are starting to collect on the wiki http://webware.colorstudy.net/twiki/bin/view/Webware/AppIdeas but nothing has solidified yet. I'd like to see PHP Nuke ported wholesale to Webware, because there are a lot of sites using it, and they will need to do something when PHP 4.1 is no longer supported. However, I hope that we can duplicate just the user interface, admin interface and themability, NOT the implementation. We have no need for input coming in on global variables (a security risk and not supported by PHP 4.2), overuse of "include", passing "parameters" to includes via global variables (can you say GOSUB?), using the default database connection rather than a specific one, etc. Perhaps this can be built on top of our wonderful future CMS spec someday, but I'd kind of like to see something out the door sooner than that. However, I haven't been active in this because I don't use the PHP Nuke interface or admin interface directly, so I'm not really sure which features are needed or which could be designed better. I just program around it to make my applications compatible with its site framework. -- -Mike (Iron) Orr, ir...@ms... (if mail problems: ms...@oz...) http://iron.cx/ English * Esperanto * Russkiy * Deutsch * Espan~ol |
From: Aaron H. <aa...@me...> - 2002-06-04 19:10:33
|
> I'd like to see PHP Nuke ported wholesale to Webware, because there are > a lot of sites using it, and they will need to do something when PHP 4.1 > is no longer supported. However, I hope that we can duplicate just the > user interface, admin interface and themability, NOT the implementation. I agree, I have been studying the jetspeed (jetspeed.jakarta.org) portlet concept and trying to adapt a pythony version. Where jetspeed has portlets I have contextlets (hey its better they pyPortlets). The idea is a to build a framework that can juggle these contextlets. In order to get content out the framework simply calls the apropriate method with common params - such as getBlock(user, site) for a content block or getPage(user,site) for a full page view. Each contextlet is responsible for telling the framework what content types is supports, block , rss, page, txt, python. Since there is some interest here I'll put up some docs that I have on this, and try to get a working demo. -Aaron Held |
From: Ian B. <ia...@co...> - 2002-06-04 19:27:35
|
On Tue, 2002-06-04 at 13:53, Mike Orr wrote: > On Tue, Jun 04, 2002 at 07:40:41PM +0200, Frank Barknecht wrote: > > I see, this is done with PostNuke, which of course is very capable, > > but I wonder, if such a site shouldn't be running something in webware > > and would thus function as a kind of "Eat your own stuff"-Dogbowl like > > Zope did with the CMF-site. > > I think we're all hoping for that someday, but first somebody has to > write the content management system. Specifications are starting to > collect on the wiki > http://webware.colorstudy.net/twiki/bin/view/Webware/AppIdeas > but nothing has solidified yet. There's also Python Community Server: http://pycs.sourceforge.net/ It appears to be written for Medusa, and is trying to close Radio UserLand (kind of a personal-weblog system, I believe). Webware is similar to Medusa, but a whole lot better, so a port might be possible. Ian |
From: Chris P. <cp...@fo...> - 2002-06-08 12:56:25
|
All, I ran into another issue with HTTPRequest for POSTs with Content-type other than 'application/x-www-form-urlencoded'. If a POST whose content-type is not 'application/x-www-form-urlencoded' comes in with parameters on the query string you'll get: Traceback (most recent call last): File "WebKit\ThreadedAppServer.py", line 287, in threadloop rh.handleRequest() File "WebKit\ThreadedAppServer.py", line 526, in handleRequest transaction = self.server._app.dispatchRawRequest(dict, strmOut) File "WebKit\Application.py", line 348, in dispatchRawRequest return self.dispatchRequest(self.createRequestForDict(newRequestDict), strmOut) File "WebKit\Application.py", line 871, in createRequestForDict return self._requestClass(dict=newRequestDict) File "WebKit\HTTPRequest.py", line 43, in __init__ self._fields.parse_qs() File "WebUtils\FieldStorage.py", line 53, in parse_qs keys = self.keys() File "C:\Python\lib\cgi.py", line 592, in keys raise TypeError, "not indexable" TypeError: not indexable The problem is at the end of WebUtils.FieldStorage.parse_qs(), where the code does: # Only append values that aren't already in the FieldStorage's keys; # This makes POSTed vars override vars on the query string keys = self.keys() for key, values in dict.items(): if key not in keys: for value in values: self.list.append(cgi.MiniFieldStorage(key,value)) The line keys = self.keys() won't' work in this case becuse cgi.FieldStorage doesn't populate self.list if the content type is not 'application/x-www-form-urlencoded'. Seems to me like cgi.FieldStorage should be setting self.list = [] instead of None, then this wouldn't be a problem, but I don't know what the other implications of that would be. To fix it in the WebUtils.FieldStorage code, however, the above code can be changed to: # Only append values that aren't already in the FieldStorage's keys; # This makes POSTed vars override vars on the query string if not self.list: self.list = [] keys = self.keys() for key, values in dict.items(): if key not in keys: for value in values: self.list.append(cgi.MiniFieldStorage(key,value)) I'm also not sure about the fact that (as the comment indicates) POSTed parameters hide query string parameters when the request has both. Seems like you'd just want to add them along with the POSTED parameters. I guess you could always go back to queryString() and parse them out yourself. Regards, Chris |