From: Jay L. <js...@js...> - 2001-01-24 23:36:10
|
I would like to bring back up an old topic, which is the communication format between that appservers and the adapters. Specifically, the content section of the response. Right now, we wrap everything in one dictionary. The dictionary has 2 entries, headers and content. This means that the entire response must be sent from the appserver to the adapter before anything can be sent to the client. I'd like to change this format and send the content portion on its own, outside of a dictionary. So the new "protocol" would be to first send the headers dictionary, then wait for an acknowledgement from the adapter, then send the content. The benefit here is that the content can be sent to the client as it comes in, instead of waiting for the entire dictionary to be received, then demarshalled. It will be a trivial change to the code, and should result in at least a modest speed increase. It will also simplify the design of the apache module I've been working on, on and off, for forever. Anybody see a problem with this? For now, we wouldn't start sending data from the appserver to the adapter until the entire content section is complete, but in the future, this would lay the groundwork for sending content as it is produced. Jay |
From: Chuck E. <ec...@mi...> - 2001-01-24 23:51:27
|
At 06:45 PM 1/24/2001 -0500, Jay Love wrote: >I would like to bring back up an old topic, which is the communication >format between that appservers and the adapters. Specifically, the >content section of the response. > >Right now, we wrap everything in one dictionary. The dictionary has 2 >entries, headers and content. This means that the entire response must be >sent from the appserver to the adapter before anything can be sent to the >client. I'd like to change this format and send the content portion on >its own, outside of a dictionary. So the new "protocol" would be to first >send the headers dictionary, then wait for an acknowledgement from the >adapter, then send the content. Yes, we discussed this before and decided to go back to the original format, which was just a bona fide HTTP response. In retrospect, it just makes sense for the app server to do it this way. But that would mean sending the headers HTTP-style immediately following the content. I don't see any reason to wait for an acknowledgement from the adapter in between the headers and body. >The benefit here is that the content can be sent to the client as it comes >in, instead of waiting for the entire dictionary to be received, then >demarshalled. It will be a trivial change to the code, and should result >in at least a modest speed increase. It will also simplify the design of >the apache module I've been working on, on and off, for forever. Agreed. >Anybody see a problem with this? No. >For now, we wouldn't start sending data from the appserver to the adapter >until the entire content section is complete, but in the future, this >would lay the groundwork for sending content as it is produced. Agreed again. But I think a classic HTTP response is the way to go, unless anyone sees a problem with that. One person previously mentioned that under a certain circumstance, they would have to parse these, but my counterpoint is that if we marshal them, they are being parsed on the receiving end every time for every adapter. You just don't see it because marshal.dumps() does it for you (or is it loads()?). -Chuck |
From: Jay L. <js...@js...> - 2001-01-25 00:04:12
|
Chuck Esterbrook wrote: > At 06:45 PM 1/24/2001 -0500, Jay Love wrote: > >> I would like to bring back up an old topic, which is the >> communication format between that appservers and the adapters. >> Specifically, the content section of the response. >> >> Right now, we wrap everything in one dictionary. The dictionary has >> 2 entries, headers and content. This means that the entire response >> must be sent from the appserver to the adapter before anything can be >> sent to the client. I'd like to change this format and send the >> content portion on its own, outside of a dictionary. So the new >> "protocol" would be to first send the headers dictionary, then wait >> for an acknowledgement from the adapter, then send the content. > > > Yes, we discussed this before and decided to go back to the original > format, which was just a bona fide HTTP response. In retrospect, it > just makes sense for the app server to do it this way. > > But that would mean sending the headers HTTP-style immediately > following the content. > > I don't see any reason to wait for an acknowledgement from the adapter > in between the headers and body. I suggested the acknowlegment because I was assuming we would send the headers as a dictionary. I'm not sure doing a full HTTP response from the appserver to the adapter is the best approach. It elimnates the possibility for any communication between the appserver and the adapter. Or it at least means that the adapter has to parse the headers from the appserver to figure out if there are any problems. I'd like to suggest that we try leaving the headers in a dictionary for now. This would require an acknowledgement in between the headers and the content, so that we know what to unmarshall, and what to just send on out to the client. > > > But I think a classic HTTP response is the way to go, unless anyone > sees a problem with that. > > One person previously mentioned that under a certain circumstance, > they would have to parse these, but my counterpoint is that if we > marshal them, they are being parsed on the receiving end every time > for every adapter. You just don't see it because marshal.dumps() does > it for you (or is it loads()?). > See above. Jay > |
From: Chuck E. <ec...@mi...> - 2001-01-25 00:14:46
|
At 07:13 PM 1/24/2001 -0500, Jay Love wrote: >I suggested the acknowlegment because I was assuming we would send the >headers as a dictionary. > >I'm not sure doing a full HTTP response from the appserver to the adapter >is the best approach. It elimnates the possibility for any communication >between the appserver and the adapter. Or it at least means that the >adapter has to parse the headers from the appserver to figure out if there >are any problems. > >I'd like to suggest that we try leaving the headers in a dictionary for >now. This would require an acknowledgement in between the headers and the >content, so that we know what to unmarshall, and what to just send on out >to the client. Before we take this approach, we should back it up with some examples of why we would need it/how we would use it. I can't think of any so far. (Note that 404s and redirections are all done via HTTP headers.) But another thought is that if the app server ever needed to communicate something special to the adapter, the first HTTP header would be the indicator: WebKit-Instruction: SelfDestruct After which further HTTP headers would have additional key value pairs if needed. In fact, another thought is that HTTP is already a format which goes: dictionary content But the dictionary is cobbled together in a different syntax than marshal and pickle. What we're really debating is whether the dictionary should be encoded with HTTP headers or marshal. I vote for HTTP since I can think of a utility for that and it would enabled the FastCGIAppServer to get straight to the point. -Chuck |
From: Vladimir K. <vk...@so...> - 2001-01-26 13:49:58
|
Chuck Esterbrook wrote: > > At 07:13 PM 1/24/2001 -0500, Jay Love wrote: > >I suggested the acknowlegment because I was assuming we would send the > >headers as a dictionary. > > > >I'm not sure doing a full HTTP response from the appserver to the adapter > >is the best approach. It elimnates the possibility for any communication > >between the appserver and the adapter. Or it at least means that the > >adapter has to parse the headers from the appserver to figure out if there > >are any problems. > > > >I'd like to suggest that we try leaving the headers in a dictionary for > >now. This would require an acknowledgement in between the headers and the > >content, so that we know what to unmarshall, and what to just send on out > >to the client. > > Before we take this approach, we should back it up with some examples of > why we would need it/how we would use it. I need it for large request/response. Now all request/response is stored as string in memory in adapter and also in AppServer. I work on this in my third patch ( https://sourceforge.net/patch/?func=detailpatch&patch_id=102597&group_id=4866 ) and I want rewrite it for last version in CVS. Headers are send as dictionary and body as stream. Example from modpHadler in Webware 0.4.1 after patch was applied # response dict = { 'format': 'CGI', 'time': timestamp, 'environ': env } s = socket(AF_INET, SOCK_STREAM) s.connect((host, port)) s.send(dumps(dict,1)) inp = req.read(bufsize) while inp: s.send(inp) inp = req.read(bufsize) s.shutdown(1) # request # make file from socket file=s.makefile("r") headers = load(file) for pair in headers: req.headers_out[pair[0]] = str(pair[1]) req.content_type = req.headers_out['content-type'] if req.headers_out.has_key('status'): req.status = int(req.headers_out['status']) req.send_http_header() response=file.read(bufsize) while response: req.write(response) response=file.read(bufsize) I had problem with marshal, so I changed it to cPickle, but now I don't know why :-(. Maybe marshall on Python 1.5.2 can't read/write from/to StringIO. Why do you use marshal ? Can I rewrite this patch to last version of Webware or wait on protocol change ? > In fact, another thought is that HTTP is already a format which goes: > > dictionary > content > > But the dictionary is cobbled together in a different syntax than marshal > and pickle. Yes, is it, but it works :-). vlk |
From: Chuck E. <ec...@mi...> - 2001-01-26 16:39:08
|
At 02:48 PM 1/26/2001 +0100, Vladimir Kralik wrote: >I had problem with marshal, so I changed it to cPickle, but now I don't >know why :-(. Maybe marshall on Python 1.5.2 can't read/write from/to >StringIO. > >Why do you use marshal ? >Can I rewrite this patch to last version of Webware or wait on protocol >change ? My understanding when talking with Jay about the protocol is that we were only talking about the _response_, not the request. Since the request comes from things like os.environ, it has to be packed up anyway and marshal seems like the fastest way to do that. The response on the other hand eventually has to be packed up as HTTP, so we might as well do that from the start and be done with it. I chose marshal over cPickle, because my benchmarks showed it to be a lot faster. That makes sense given that marshal is more primitive. Here is more info: http://www.python.org/doc/current/lib/module-marshal.html If Jay verifies that the request protocol is NOT changing, then it seems like you could proceed however you wanted. However, I would really like to know why pickle isn't working for you before switching to it from marshal. -Chuck |
From: Vladimir K. <vk...@so...> - 2001-01-26 18:26:10
|
Chuck Esterbrook wrote: > My understanding when talking with Jay about the protocol is that we were > only talking about the _response_, not the request. But I think that this is one protocol, because both use one socket :-(. > Since the request comes > from things like os.environ, it has to be packed up anyway and marshal > seems like the fastest way to do that. From environment comes only header of request, body comes from stream and is reading into string in Adapter. Then string is marshaled to HTTPRequest, where from request body string is making file ( StringIO ) and this file is send to module cgi. This work, if request is small, but this cann't be used for file upload. My idea was : marshal as dictionary only data from os.environ, and input stream simly copy after marshaling dictionary to socket over which communication with AppServer comes. In HTTPRequest, instead of making file from string, simply make file from socket. Now request body isn't stored in memory as string. > The response on the other hand eventually has to be packed up as HTTP, so > we might as well do that from the start and be done with it. It has low importance for me, if header is packed up as dictionary or HTTP-string, but I can't store in memory 10-40MB of data, which user need download from my database :-(. My idea was : Request data ( headers and body ) is buffered in memory until HTTPRespone.deliver() was not called. Before deliver was call, all headers and body can be changed ( for example : Error page display ) After deliver headers and body are sent to adapter, but socket doesn't close, and following call of HTTPResponse.write() send data directly to socket. Disadvantage is that headers and sending part of body can not be changed after deliver. ( no error page :-(). > If Jay verifies that the request protocol is NOT changing, then it seems > like you could proceed however you wanted. However, I would really like to > know why pickle isn't working for you before switching to it from marshal. I found it, when I made patch AsyncoreThreadedAppServer, I need own implementation of file-like access. I take idea from StringIO, but this doesn't work with marshal( see code below ). import marshal x={1:1} import StringIO a=StringIO.StringIO() marshal.dump(x,a) Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: marshal.dump() 2nd arg must be file |
From: Chuck E. <ec...@mi...> - 2001-01-26 18:50:19
|
At 07:24 PM 1/26/2001 +0100, Vladimir Kralik wrote: >Chuck Esterbrook wrote: > > My understanding when talking with Jay about the protocol is that we were > > only talking about the _response_, not the request. >But I think that this is one protocol, because both use one socket :-(. I don't think so. TCP/IP is the underlying protocol for moving the raw data, but whether we arrange that data as HTTP, marshal, pickle or create our own format is our decision. We don't even have to do the same thing going in each direction. > > Since the request comes > > from things like os.environ, it has to be packed up anyway and marshal > > seems like the fastest way to do that. > From environment comes only header of request, body comes from stream >and is reading into string in Adapter. Then string is marshaled to >HTTPRequest, where from request body string is making file ( StringIO ) >and this file is send to module cgi. This work, if request is small, but >this cann't be used for file upload. Why? >My idea was : >marshal as dictionary only data from os.environ, and input stream simly >copy after marshaling dictionary to socket over which communication with >AppServer comes. In HTTPRequest, instead of making file from string, >simply make file from socket. Now request body isn't stored in memory as >string. Sounds reasonable. Whatever approach we come up with, should also work for OneShot.* > > The response on the other hand eventually has to be packed up as HTTP, so > > we might as well do that from the start and be done with it. >It has low importance for me, if header is packed up as dictionary or >HTTP-string, but I can't store in memory 10-40MB of data, which user >need download from my database :-(. So the issue here is when a person wants to download a huge amount of data. This is where our current approach of collecting the entire response before sending it, breaks down. >My idea was : >Request data ( headers and body ) is buffered in memory until >HTTPRespone.deliver() was not called. >Before deliver was call, all headers and body can be changed ( for >example : Error page display ) >After deliver headers and body are sent to adapter, but socket doesn't >close, and following call of HTTPResponse.write() send data directly to >socket. Disadvantage is that headers and sending part of body can not be >changed after deliver. ( no error page :-(). Sounds like the right approach to me. I assume that after deliver() is invoked, various methods such as setHeader() will raise an exception. > > If Jay verifies that the request protocol is NOT changing, then it seems > > like you could proceed however you wanted. However, I would really like to > > know why pickle isn't working for you before switching to it from marshal. >I found it, when I made patch AsyncoreThreadedAppServer, I need own >implementation of file-like access. I take idea from StringIO, but this >doesn't work with marshal( see code below ). > >import marshal >x={1:1} >import StringIO >a=StringIO.StringIO() >marshal.dump(x,a) >Traceback (most recent call last): > File "<stdin>", line 1, in ? >TypeError: marshal.dump() 2nd arg must be >file I haven't eaten yet, so maybe my brain is low on calories: why did you need the StringIO? -Chuck |
From: Vladimir K. <vk...@so...> - 2001-01-26 20:11:12
|
Chuck Esterbrook wrote: > I don't think so. TCP/IP is the underlying protocol for moving the raw > data, but whether we arrange that data as HTTP, marshal, pickle or create > our own format is our decision. We don't even have to do the same thing > going in each direction. Yes, now I understand it :-). > > From environment comes only header of request, body comes from stream > >and is reading into string in Adapter. Then string is marshaled to > >HTTPRequest, where from request body string is making file ( StringIO ) > >and this file is send to module cgi. This work, if request is small, but > >this cann't be used for file upload. > > Why? What happends if if user send 1 GB of data as file to Webware. This cann't be stored as string in memory :-(, but I need this feature. > > >My idea was : > >marshal as dictionary only data from os.environ, and input stream simly > >copy after marshaling dictionary to socket over which communication with > >AppServer comes. In HTTPRequest, instead of making file from string, > >simply make file from socket. Now request body isn't stored in memory as > >string. > > Sounds reasonable. Whatever approach we come up with, should also work for > OneShot.* What is problem ? OneShotAdapter used CGI-protocol, this mean request body comes from sys.stdin. In HTTPRequest needn't to be socket. This can be anything which has method of file ( exactly must see in cgi-module ). For OneShots we can simply send sys.stdin into HTTPRequest. Do you know, what I want to say ? ( Oh, english ....) > > > > The response on the other hand eventually has to be packed up as HTTP, so > > > we might as well do that from the start and be done with it. > >It has low importance for me, if header is packed up as dictionary or > >HTTP-string, but I can't store in memory 10-40MB of data, which user > >need download from my database :-(. > > So the issue here is when a person wants to download a huge amount of data. > This is where our current approach of collecting the entire response before > sending it, breaks down. In my case, I have collected all data in database, so I need only send output of "select * from table where ..." to browser. > > >My idea was : > >Request data ( headers and body ) is buffered in memory until > >HTTPRespone.deliver() was not called. > >Before deliver was call, all headers and body can be changed ( for > >example : Error page display ) > >After deliver headers and body are sent to adapter, but socket doesn't > >close, and following call of HTTPResponse.write() send data directly to > >socket. Disadvantage is that headers and sending part of body can not be > >changed after deliver. ( no error page :-(). > > Sounds like the right approach to me. I assume that after deliver() is > invoked, various methods such as setHeader() will raise an exception. Yes, you're right. > I haven't eaten yet, so maybe my brain is low on calories: why did you need > the StringIO? Not, exactly StringIO. In AsyncThreadedAppServer with my patch I had ( in 0.4.1 ) buffer with constant size ( for example 1 MB ). Request-data (headers and body) was buffered before for it was made Transaction. When amount of data is less than 1 MB, all data is buffered. If it is more than 1 MB, I need join buffer with data from stream, and on this I write own class. This has methods like file/StringIO. Constuctor gets buffer and stream. When read-method is called, first return data from buffer, and second, after all data from buffer was readed, simply copy data from stream. Approximate same solution I have for response. Until ammount of data is less then 1 MB, it's buffered. Otherwise, producer ( thread for HTTPResponse ) is stopped, until length of buffer is smaller than 1 MB. Can you understand me ? |
From: Jay L. <js...@js...> - 2001-01-27 14:40:38
|
I agree with this change to the protocol. The request passed to the Appserver from the adapter will be the same as it is now, with the exception that post data, the input section, will not be marshalled. The only problem is how to break up the dictionary portion from the input portion. Currently, the AppServer just reads from the adapter until the socket closes, and then we unmarshall all of the data that was read. We will need to add some communication to this structure, an acknowledgemnt, to make the proposed system work. We'd have to have the adapter send the dictionary, wait for a reponse from the AppServer, I again propose a 1 for go ahead, a 0 from stop, and then the adapter would send the post data. Chuck, you fought this type of communication for the other side of the protocol, what do you think here? BTW, this has no effect on OneShot. Jay Vladimir Kralik wrote: > Chuck Esterbrook wrote: > >> I don't think so. TCP/IP is the underlying protocol for moving the raw >> data, but whether we arrange that data as HTTP, marshal, pickle or create >> our own format is our decision. We don't even have to do the same thing >> going in each direction. > > Yes, now I understand it :-). > >>> From environment comes only header of request, body comes from stream >>> and is reading into string in Adapter. Then string is marshaled to >>> HTTPRequest, where from request body string is making file ( StringIO ) >>> and this file is send to module cgi. This work, if request is small, but >>> this cann't be used for file upload. >> >> Why? > > What happends if if user send 1 GB of data as file to Webware. This > cann't be stored as string in memory :-(, but I need this feature. > >>> My idea was : >>> marshal as dictionary only data from os.environ, and input stream simly >>> copy after marshaling dictionary to socket over which communication with >>> AppServer comes. In HTTPRequest, instead of making file from string, >>> simply make file from socket. Now request body isn't stored in memory as >>> string. >> >> Sounds reasonable. Whatever approach we come up with, should also work for >> OneShot.* > > What is problem ? OneShotAdapter used CGI-protocol, this mean request > body comes from sys.stdin. In HTTPRequest needn't to be socket. This can > be anything which has method of file ( exactly must see in cgi-module ). > For OneShots we can simply send sys.stdin into HTTPRequest. Do you know, > what I want to say ? ( Oh, english ....) > > >>>> The response on the other hand eventually has to be packed up as HTTP, so >>>> we might as well do that from the start and be done with it. >>> >>> It has low importance for me, if header is packed up as dictionary or >>> HTTP-string, but I can't store in memory 10-40MB of data, which user >>> need download from my database :-(. >> >> So the issue here is when a person wants to download a huge amount of data. >> This is where our current approach of collecting the entire response before >> sending it, breaks down. > > In my case, I have collected all data in database, so I need only send > output of "select * from table where ..." to browser. > >>> My idea was : >>> Request data ( headers and body ) is buffered in memory until >>> HTTPRespone.deliver() was not called. >>> Before deliver was call, all headers and body can be changed ( for >>> example : Error page display ) >>> After deliver headers and body are sent to adapter, but socket doesn't >>> close, and following call of HTTPResponse.write() send data directly to >>> socket. Disadvantage is that headers and sending part of body can not be >>> changed after deliver. ( no error page :-(). >> >> Sounds like the right approach to me. I assume that after deliver() is >> invoked, various methods such as setHeader() will raise an exception. > > Yes, you're right. > >> I haven't eaten yet, so maybe my brain is low on calories: why did you need >> the StringIO? > > Not, exactly StringIO. In AsyncThreadedAppServer with my patch I had ( > in 0.4.1 ) buffer with constant size ( for example 1 MB ). Request-data > (headers and body) was buffered before for it was made Transaction. When > amount of data is less than 1 MB, all data is buffered. If it is more > than 1 MB, I need join buffer with data from stream, and on this I write > own class. This has methods like file/StringIO. Constuctor gets buffer > and stream. When read-method is called, first return data from buffer, > and second, after all data from buffer was readed, simply copy data from > stream. > > Approximate same solution I have for response. Until ammount of data is > less then 1 MB, it's buffered. Otherwise, producer ( thread for > HTTPResponse ) is stopped, until length of buffer is smaller than 1 MB. > Can you understand me ? > > _______________________________________________ > Webware-discuss mailing list > Web...@li... > http://lists.sourceforge.net/lists/listinfo/webware-discuss |
From: Chuck E. <ec...@mi...> - 2001-01-27 20:31:16
|
At 09:49 AM 1/27/2001 -0500, Jay Love wrote: >I agree with this change to the protocol. > >The request passed to the Appserver from the adapter will be the same as >it is now, with the exception that post data, the input section, will not >be marshalled. > >The only problem is how to break up the dictionary portion from the input >portion. Currently, the AppServer just reads from the adapter until the >socket closes, and then we unmarshall all of the data that was read. We >will need to add some communication to this structure, an acknowledgemnt, >to make the proposed system work. We'd have to have the adapter send the >dictionary, wait for a reponse from the AppServer, I again propose a 1 for >go ahead, a 0 from stop, and then the adapter would send the post data. > >Chuck, you fought this type of communication for the other side of the >protocol, what do you think here? > >BTW, this has no effect on OneShot. > >Jay My thought is that it would be far more efficient to send the length of the marshalled dictionary as 32-bit int, and the app server assumes that everything past that is input as opposed to marshalled. That cuts the extra communication loop. -Chuck |
From: Jay L. <js...@js...> - 2001-01-27 14:51:19
|
Whoops. I misssed one piece on my earlier post on this. We'd need one more piece of communication. Before the dictionary is sent, we'd need to send the size of the dictionary. IF this sounds reasonable, I'll get it working and then we'll fine tune it. Jay Vladimir Kralik wrote: > Chuck Esterbrook wrote: > >> I don't think so. TCP/IP is the underlying protocol for moving the raw >> data, but whether we arrange that data as HTTP, marshal, pickle or create >> our own format is our decision. We don't even have to do the same thing >> going in each direction. > > Yes, now I understand it :-). > >>> From environment comes only header of request, body comes from stream >>> and is reading into string in Adapter. Then string is marshaled to >>> HTTPRequest, where from request body string is making file ( StringIO ) >>> and this file is send to module cgi. This work, if request is small, but >>> this cann't be used for file upload. >> >> Why? > > What happends if if user send 1 GB of data as file to Webware. This > cann't be stored as string in memory :-(, but I need this feature. > >>> My idea was : >>> marshal as dictionary only data from os.environ, and input stream simly >>> copy after marshaling dictionary to socket over which communication with >>> AppServer comes. In HTTPRequest, instead of making file from string, >>> simply make file from socket. Now request body isn't stored in memory as >>> string. >> >> Sounds reasonable. Whatever approach we come up with, should also work for >> OneShot.* > > What is problem ? OneShotAdapter used CGI-protocol, this mean request > body comes from sys.stdin. In HTTPRequest needn't to be socket. This can > be anything which has method of file ( exactly must see in cgi-module ). > For OneShots we can simply send sys.stdin into HTTPRequest. Do you know, > what I want to say ? ( Oh, english ....) > > >>>> The response on the other hand eventually has to be packed up as HTTP, so >>>> we might as well do that from the start and be done with it. >>> >>> It has low importance for me, if header is packed up as dictionary or >>> HTTP-string, but I can't store in memory 10-40MB of data, which user >>> need download from my database :-(. >> >> So the issue here is when a person wants to download a huge amount of data. >> This is where our current approach of collecting the entire response before >> sending it, breaks down. > > In my case, I have collected all data in database, so I need only send > output of "select * from table where ..." to browser. > >>> My idea was : >>> Request data ( headers and body ) is buffered in memory until >>> HTTPRespone.deliver() was not called. >>> Before deliver was call, all headers and body can be changed ( for >>> example : Error page display ) >>> After deliver headers and body are sent to adapter, but socket doesn't >>> close, and following call of HTTPResponse.write() send data directly to >>> socket. Disadvantage is that headers and sending part of body can not be >>> changed after deliver. ( no error page :-(). >> >> Sounds like the right approach to me. I assume that after deliver() is >> invoked, various methods such as setHeader() will raise an exception. > > Yes, you're right. > >> I haven't eaten yet, so maybe my brain is low on calories: why did you need >> the StringIO? > > Not, exactly StringIO. In AsyncThreadedAppServer with my patch I had ( > in 0.4.1 ) buffer with constant size ( for example 1 MB ). Request-data > (headers and body) was buffered before for it was made Transaction. When > amount of data is less than 1 MB, all data is buffered. If it is more > than 1 MB, I need join buffer with data from stream, and on this I write > own class. This has methods like file/StringIO. Constuctor gets buffer > and stream. When read-method is called, first return data from buffer, > and second, after all data from buffer was readed, simply copy data from > stream. > > Approximate same solution I have for response. Until ammount of data is > less then 1 MB, it's buffered. Otherwise, producer ( thread for > HTTPResponse ) is stopped, until length of buffer is smaller than 1 MB. > Can you understand me ? > > _______________________________________________ > Webware-discuss mailing list > Web...@li... > http://lists.sourceforge.net/lists/listinfo/webware-discuss |
From: Chuck E. <ec...@mi...> - 2001-01-27 20:32:18
|
If you send the size of the dictionary, then couldn't you immediately send the input? [size][marshalled dictionary][input] Then on the app server side it: - reads the size - reads that many bytes - unmarshals the dictionary - reads the input until eof. That was basically my earlier suggestion. -Chuck At 10:00 AM 1/27/2001 -0500, Jay Love wrote: >Whoops. I misssed one piece on my earlier post on this. We'd need one >more piece of communication. Before the dictionary is sent, we'd need to >send the size of the dictionary. > >IF this sounds reasonable, I'll get it working and then we'll fine tune it. > >Jay > > >Vladimir Kralik wrote: > >>Chuck Esterbrook wrote: >> >>>I don't think so. TCP/IP is the underlying protocol for moving the raw >>>data, but whether we arrange that data as HTTP, marshal, pickle or create >>>our own format is our decision. We don't even have to do the same thing >>>going in each direction. >>Yes, now I understand it :-). >> >>>> From environment comes only header of request, body comes from stream >>>>and is reading into string in Adapter. Then string is marshaled to >>>>HTTPRequest, where from request body string is making file ( StringIO ) >>>>and this file is send to module cgi. This work, if request is small, but >>>>this cann't be used for file upload. >>>Why? >>What happends if if user send 1 GB of data as file to Webware. This >>cann't be stored as string in memory :-(, but I need this feature. >>>>My idea was : >>>>marshal as dictionary only data from os.environ, and input stream simly >>>>copy after marshaling dictionary to socket over which communication with >>>>AppServer comes. In HTTPRequest, instead of making file from string, >>>>simply make file from socket. Now request body isn't stored in memory as >>>>string. >>>Sounds reasonable. Whatever approach we come up with, should also work for >>>OneShot.* >>What is problem ? OneShotAdapter used CGI-protocol, this mean request >>body comes from sys.stdin. In HTTPRequest needn't to be socket. This can >>be anything which has method of file ( exactly must see in cgi-module ). >>For OneShots we can simply send sys.stdin into HTTPRequest. Do you know, >>what I want to say ? ( Oh, english ....) >> >> >>>>>The response on the other hand eventually has to be packed up as HTTP, so >>>>>we might as well do that from the start and be done with it. >>>>It has low importance for me, if header is packed up as dictionary or >>>>HTTP-string, but I can't store in memory 10-40MB of data, which user >>>>need download from my database :-(. >>>So the issue here is when a person wants to download a huge amount of data. >>>This is where our current approach of collecting the entire response before >>>sending it, breaks down. >>In my case, I have collected all data in database, so I need only send >>output of "select * from table where ..." to browser. >>>>My idea was : >>>>Request data ( headers and body ) is buffered in memory until >>>>HTTPRespone.deliver() was not called. >>>>Before deliver was call, all headers and body can be changed ( for >>>>example : Error page display ) >>>>After deliver headers and body are sent to adapter, but socket doesn't >>>>close, and following call of HTTPResponse.write() send data directly to >>>>socket. Disadvantage is that headers and sending part of body can not be >>>>changed after deliver. ( no error page :-(). >>>Sounds like the right approach to me. I assume that after deliver() is >>>invoked, various methods such as setHeader() will raise an exception. >>Yes, you're right. >> >>>I haven't eaten yet, so maybe my brain is low on calories: why did you need >>>the StringIO? >>Not, exactly StringIO. In AsyncThreadedAppServer with my patch I had ( >>in 0.4.1 ) buffer with constant size ( for example 1 MB ). Request-data >>(headers and body) was buffered before for it was made Transaction. When >>amount of data is less than 1 MB, all data is buffered. If it is more >>than 1 MB, I need join buffer with data from stream, and on this I write >>own class. This has methods like file/StringIO. Constuctor gets buffer >>and stream. When read-method is called, first return data from buffer, >>and second, after all data from buffer was readed, simply copy data from >>stream. >>Approximate same solution I have for response. Until ammount of data is >>less then 1 MB, it's buffered. Otherwise, producer ( thread for >>HTTPResponse ) is stopped, until length of buffer is smaller than 1 MB. >>Can you understand me ? >>_______________________________________________ >>Webware-discuss mailing list >>Web...@li... >>http://lists.sourceforge.net/lists/listinfo/webware-discuss > > |
From: Jay L. <js...@js...> - 2001-01-27 20:46:29
|
Chuck Esterbrook wrote: > If you send the size of the dictionary, then couldn't you immediately > send the input? > > [size][marshalled dictionary][input] > > Then on the app server side it: > - reads the size > - reads that many bytes > - unmarshals the dictionary > - reads the input until eof. > > That was basically my earlier suggestion. > > -Chuck Yes, that's what I backed into as well. Are we OK on this? Jay |
From: Chuck E. <ec...@mi...> - 2001-01-27 21:25:16
|
At 03:55 PM 1/27/2001 -0500, Jay Love wrote: >Chuck Esterbrook wrote: > >>If you send the size of the dictionary, then couldn't you immediately >>send the input? >>[size][marshalled dictionary][input] >>Then on the app server side it: >> - reads the size >> - reads that many bytes >> - unmarshals the dictionary >> - reads the input until eof. >>That was basically my earlier suggestion. >>-Chuck > > > >Yes, that's what I backed into as well. > >Are we OK on this? > >Jay Sounds good to me. So it's still just a simple round trip: - adapter sends entire request to app server via socket - app server sends entire response back to adapter via same socket -Chuck |
From: Vladimir K. <vk...@so...> - 2001-01-29 08:58:33
|
Chuck Esterbrook wrote: > > At 03:55 PM 1/27/2001 -0500, Jay Love wrote: > >Chuck Esterbrook wrote: > > > >>If you send the size of the dictionary, then couldn't you immediately > >>send the input? Really need size of dictionary ? My test say : not ( tested on Python 2.0, Red Hat 6.2 ) import marshal #import cPickle import pprint dict={'Dictionary':1, 'bbb':[1,2,3,4]} string="Testing string" f=open("xxx","w") marshal.dump(dict,f) f.write(string) f.close() r=open("xxx","r") newdict=marshal.load(r) print "Dictionary :", pprint.pprint(newdict) print "*"*70 print "String :", st=r.read() print st r.close() So protocol can be simply : [marshalled dictionary][input] and unmarshalling can be done directly from stream ( socket ). > Sounds good to me. So it's still just a simple round trip: > - adapter sends entire request to app server via socket > - app server sends entire response back to adapter via same socket Need anybody additional communication between adapter and app-server ? I'm not, and donn't know for which reason can be used. Jay Love wrote: > IF this sounds reasonable, I'll get it working and then we'll fine tune it. Do you work only on Request-protocol, or on both Request and Response ? For Response the idea is the same, headers as marshalled dictionary and body as stream. vlk -- Mgr. Vladimir Kralik Softip a.s. Zvolenska cesta 19 974 01 Banska Bystrica tel. +421 88 4358 176 |
From: Chuck E. <ec...@mi...> - 2001-01-29 15:49:47
|
Sounds good. As long as Pickle has the same property, then I think we can eliminate the size. I prefer marshal over pickle for speed, but I recognize that it would be good to be able to switch if we ever had to. -Chuck At 09:56 AM 1/29/2001 +0100, Vladimir Kralik wrote: >Chuck Esterbrook wrote: > > > > At 03:55 PM 1/27/2001 -0500, Jay Love wrote: > > >Chuck Esterbrook wrote: > > > > > >>If you send the size of the dictionary, then couldn't you immediately > > >>send the input? >Really need size of dictionary ? My test say : not ( tested on Python >2.0, Red Hat 6.2 ) > import marshal > #import cPickle > import pprint > > dict={'Dictionary':1, 'bbb':[1,2,3,4]} > string="Testing string" > > f=open("xxx","w") > marshal.dump(dict,f) > f.write(string) > f.close() > > r=open("xxx","r") > newdict=marshal.load(r) > print "Dictionary :", > pprint.pprint(newdict) > print "*"*70 > print "String :", > st=r.read() > print st > r.close() > >So protocol can be simply : > [marshalled dictionary][input] > >and unmarshalling can be done directly from stream ( socket ). > > > Sounds good to me. So it's still just a simple round trip: > > - adapter sends entire request to app server via socket > > - app server sends entire response back to adapter via same socket >Need anybody additional communication between adapter and app-server ? >I'm not, and donn't know for which reason can be used. > >Jay Love wrote: > > IF this sounds reasonable, I'll get it working and then we'll fine tune it. >Do you work only on Request-protocol, or on both Request and Response ? >For Response the idea is the same, headers as marshalled dictionary and >body as stream. > > vlk > >-- >Mgr. Vladimir Kralik >Softip a.s. >Zvolenska cesta 19 >974 01 Banska Bystrica > >tel. +421 88 4358 176 > >_______________________________________________ >Webware-discuss mailing list >Web...@li... >http://lists.sourceforge.net/lists/listinfo/webware-discuss |
From: Jay L. <js...@js...> - 2001-01-30 00:05:33
|
This won't work. The fundamental flaw is reading directly from the socket. You can't assume that the entire dictionary will have arrived and be buffered at the socket. Then, as soon as you start reading from a string, you have to know the size of the dictionary. Jay Chuck Esterbrook wrote: > Sounds good. As long as Pickle has the same property, then I think we > can eliminate the size. I prefer marshal over pickle for speed, but I > recognize that it would be good to be able to switch if we ever had to. > > -Chuck > > > At 09:56 AM 1/29/2001 +0100, Vladimir Kralik wrote: > >> Chuck Esterbrook wrote: >> > >> > At 03:55 PM 1/27/2001 -0500, Jay Love wrote: >> > >Chuck Esterbrook wrote: >> > > >> > >>If you send the size of the dictionary, then couldn't you >> immediately >> > >>send the input? >> Really need size of dictionary ? My test say : not ( tested on Python >> 2.0, Red Hat 6.2 ) >> import marshal >> #import cPickle >> import pprint >> >> dict={'Dictionary':1, 'bbb':[1,2,3,4]} >> string="Testing string" >> >> f=open("xxx","w") >> marshal.dump(dict,f) >> f.write(string) >> f.close() >> >> r=open("xxx","r") >> newdict=marshal.load(r) >> print "Dictionary :", >> pprint.pprint(newdict) >> print "*"*70 >> print "String :", >> st=r.read() >> print st >> r.close() >> >> So protocol can be simply : >> [marshalled dictionary][input] >> >> and unmarshalling can be done directly from stream ( socket ). >> >> > Sounds good to me. So it's still just a simple round trip: >> > - adapter sends entire request to app server via socket >> > - app server sends entire response back to adapter via same socket >> Need anybody additional communication between adapter and app-server ? >> I'm not, and donn't know for which reason can be used. >> >> Jay Love wrote: >> > IF this sounds reasonable, I'll get it working and then we'll fine >> tune it. >> Do you work only on Request-protocol, or on both Request and Response ? >> For Response the idea is the same, headers as marshalled dictionary and >> body as stream. >> >> vlk >> >> -- >> Mgr. Vladimir Kralik >> Softip a.s. >> Zvolenska cesta 19 >> 974 01 Banska Bystrica >> >> tel. +421 88 4358 176 >> >> _______________________________________________ >> Webware-discuss mailing list >> Web...@li... >> http://lists.sourceforge.net/lists/listinfo/webware-discuss > > > > _______________________________________________ > Webware-discuss mailing list > Web...@li... > http://lists.sourceforge.net/lists/listinfo/webware-discuss |
From: Vladimir K. <vk...@so...> - 2001-01-30 07:26:38
|
Jay Love wrote: > > This won't work. The fundamental flaw is reading directly from the > socket. You can't assume that the entire dictionary will have arrived > and be buffered at the socket. Then, as soon as you start reading from > a string, you have to know the size of the dictionary. I don't how this is in M$ Window$, but in UNIX, anything is file, so when I read something from socket, and entire data ( dictionary ) doesn't buffer, reading thread wait until has whole amount of data ( if socket is in blocking mode ) or EOF. The same property has os.pipe(). Before change of protocol you aren't afraid, if all data dictionary is readed in buffer of socket. vlk |
From: Geoff T. <gta...@na...> - 2001-01-30 15:06:40
|
I think Jay is right. If the socket were a Python file object, you could use marshal.load(sock) and it would read just enough bytes to reconstruct the marshalled dictionary. But a socket is NOT a file object in Python, so you can't un-marshal directly from it -- you have to know how long the string is that represents the marshalled dictionary, read exactly that many bytes out of the socket, and use marshal.loads() on that string. And sock.makefile() isn't a solution because it doesn't return a true Python file object either, at least on Windows. So the count of bytes is needed. (In Windows, a socket is most definitely NOT a file; maybe this would work on Unix.) Vladimir Kralik wrote: > Jay Love wrote: > > > > This won't work. The fundamental flaw is reading directly from the > > socket. You can't assume that the entire dictionary will have arrived > > and be buffered at the socket. Then, as soon as you start reading from > > a string, you have to know the size of the dictionary. > I don't how this is in M$ Window$, but in UNIX, anything is file, so > when I read something from socket, and entire data ( dictionary ) > doesn't buffer, reading thread wait until has whole amount of data ( if > socket is in blocking mode ) or EOF. The same property has os.pipe(). > > Before change of protocol you aren't afraid, if all data dictionary is > readed in buffer of socket. > > vlk > > _______________________________________________ > Webware-discuss mailing list > Web...@li... > http://lists.sourceforge.net/lists/listinfo/webware-discuss -- - Geoff Talvola Parlance Corporation gtalvola@NameConnector.com |
From: Vladimir K. <vk...@so...> - 2001-01-30 17:18:33
|
Geoff Talvola wrote: > I think Jay is right. If the socket were a Python file object, you could > use marshal.load(sock) and it would read just enough bytes to reconstruct > the marshalled dictionary. But a socket is NOT a file object in Python, I agree, I think on socket.makefile(). > so you can't un-marshal directly from it -- you have to know how long > the string is that represents the marshalled dictionary, read exactly > that many bytes out of the socket, and use marshal.loads() on that string. > And sock.makefile() isn't a solution because it doesn't return a true > Python file object either, at least on Windows. So the count of bytes is Python documentation says : % 2.1.7.9 File Objects % % File objects are implemented using C's stdio package and can be created % with the built-in function open() described in section 2.3, ``Built-in % Functions.'' They are also returned by some other built-in functions % and methods, e.g., os.popen() and os.fdopen() and the makefile() method % of socket objects. ( see : http://www.python.org/doc/current/lib/bltin-file-objects.html ) This is not true on M$ Window$ ? > needed. (In Windows, a socket is most definitely NOT a file; maybe this > would work on Unix.) Can anybody test this on M$ Window$ or other systems ( Mac, OS/2, .... ). vlk |
From: Geoff T. <gta...@na...> - 2001-01-30 18:34:43
|
Vladimir Kralik wrote: > Python documentation says : > % 2.1.7.9 File Objects > % > % File objects are implemented using C's stdio package and can be > created > % with the built-in function open() described in section 2.3, ``Built-in > % Functions.'' They are also returned by some other built-in functions > % and methods, e.g., os.popen() and os.fdopen() and the makefile() > method > % of socket objects. > ( see : http://www.python.org/doc/current/lib/bltin-file-objects.html ) > This is not true on M$ Window$ ? On Windows: >>> import socket >>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) >>> s.makefile() <socket._fileobject instance at 10210e0> >>> open('foo.txt', 'w') <open file 'foo.txt', mode 'w' at 1020020> >>> import marshal >>> marshal.load(s.makefile()) Traceback (innermost last): File "<interactive input>", line 1, in ? TypeError: marshal.load() arg must be file A socket is not a file on Windows. -- - Geoff Talvola Parlance Corporation gtalvola@NameConnector.com |
From: Vladimir K. <vk...@so...> - 2001-01-30 18:51:10
|
Geoff Talvola wrote: > On Windows: > > >>> import socket > >>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > >>> s.makefile() > <socket._fileobject instance at 10210e0> > >>> open('foo.txt', 'w') > <open file 'foo.txt', mode 'w' at 1020020> > >>> import marshal > >>> marshal.load(s.makefile()) > Traceback (innermost last): > File "<interactive input>", line 1, in ? > TypeError: marshal.load() arg must be file > > A socket is not a file on Windows. This is the same exception as for StringIO ( marshal cannot read/write from/into StringIO ). File for marshall must be file in OS and not object which has method of file :-(. vlk |
From: Jay L. <js...@js...> - 2001-01-25 00:29:18
|
> Chuck said: > > But another thought is that if the app server ever needed to > communicate something special to the adapter, the first HTTP header > would be the indicator: > > WebKit-Instruction: SelfDestruct > > > After which further HTTP headers would have additional key value pairs > if needed. > Well, then we do have to parse at least the first header. And to do that, I guess we'd read a minimum of, say 24 bytes, and scan that input for the first newline, then parse that line. I don't know. Believe me, I don't want it to be marshalled. If it is, I have to write the demarshalling code for apache. But it just seems like we shouldn't just dump everything out to the adpter in one long stream of text. How about a compromise. We have the first byte sent be a status byte, either 1 or 0, If it's 1, we dump the rest of the response to the client. If it's 0, there's an issue, and we start parsing the rest of the stream for key value pairs separated by something, maybe null bytes. We don't need to worry about that now, just the status byte. Jay |
From: Chuck E. <ec...@mi...> - 2001-01-25 00:46:00
|
At 07:38 PM 1/24/2001 -0500, Jay Love wrote: >Well, then we do have to parse at least the first header. And to do that, >I guess we'd read a minimum of, say 24 bytes, and scan that input for the >first newline, then parse that line. >I don't know. Believe me, I don't want it to be marshalled. If it is, I >have to write the demarshalling code for apache. But it just seems like >we shouldn't just dump everything out to the adpter in one long stream of text. > >How about a compromise. We have the first byte sent be a status byte, >either 1 or 0, If it's 1, we dump the rest of the response to the >client. If it's 0, there's an issue, and we start parsing the rest of the >stream for key value pairs separated by something, maybe null bytes. We >don't need to worry about that now, just the status byte. > >Jay Two things I don't like: * 0 and 1 feel arcane. * The first char still has to be stripped before passing the response on. Assuming no one will ever create an HTTP header that begins with "WebKit", we can check if the response starts with "WebKit" (instead of if it starts with "0"). If it doesn't, then just pass the data on. Sound good? -Chuck |