I agree with this change to the protocol.
The request passed to the Appserver from the adapter will be the same as
it is now, with the exception that post data, the input section, will
not be marshalled.
The only problem is how to break up the dictionary portion from the
input portion. Currently, the AppServer just reads from the adapter
until the socket closes, and then we unmarshall all of the data that was
read. We will need to add some communication to this structure, an
acknowledgemnt, to make the proposed system work. We'd have to have the
adapter send the dictionary, wait for a reponse from the AppServer, I
again propose a 1 for go ahead, a 0 from stop, and then the adapter
would send the post data.
Chuck, you fought this type of communication for the other side of the
protocol, what do you think here?
BTW, this has no effect on OneShot.
Vladimir Kralik wrote:
> Chuck Esterbrook wrote:
>> I don't think so. TCP/IP is the underlying protocol for moving the raw
>> data, but whether we arrange that data as HTTP, marshal, pickle or create
>> our own format is our decision. We don't even have to do the same thing
>> going in each direction.
> Yes, now I understand it :-).
>>> From environment comes only header of request, body comes from stream
>>> and is reading into string in Adapter. Then string is marshaled to
>>> HTTPRequest, where from request body string is making file ( StringIO )
>>> and this file is send to module cgi. This work, if request is small, but
>>> this cann't be used for file upload.
> What happends if if user send 1 GB of data as file to Webware. This
> cann't be stored as string in memory :-(, but I need this feature.
>>> My idea was :
>>> marshal as dictionary only data from os.environ, and input stream simly
>>> copy after marshaling dictionary to socket over which communication with
>>> AppServer comes. In HTTPRequest, instead of making file from string,
>>> simply make file from socket. Now request body isn't stored in memory as
>> Sounds reasonable. Whatever approach we come up with, should also work for
> What is problem ? OneShotAdapter used CGI-protocol, this mean request
> body comes from sys.stdin. In HTTPRequest needn't to be socket. This can
> be anything which has method of file ( exactly must see in cgi-module ).
> For OneShots we can simply send sys.stdin into HTTPRequest. Do you know,
> what I want to say ? ( Oh, english ....)
>>>> The response on the other hand eventually has to be packed up as HTTP, so
>>>> we might as well do that from the start and be done with it.
>>> It has low importance for me, if header is packed up as dictionary or
>>> HTTP-string, but I can't store in memory 10-40MB of data, which user
>>> need download from my database :-(.
>> So the issue here is when a person wants to download a huge amount of data.
>> This is where our current approach of collecting the entire response before
>> sending it, breaks down.
> In my case, I have collected all data in database, so I need only send
> output of "select * from table where ..." to browser.
>>> My idea was :
>>> Request data ( headers and body ) is buffered in memory until
>>> HTTPRespone.deliver() was not called.
>>> Before deliver was call, all headers and body can be changed ( for
>>> example : Error page display )
>>> After deliver headers and body are sent to adapter, but socket doesn't
>>> close, and following call of HTTPResponse.write() send data directly to
>>> socket. Disadvantage is that headers and sending part of body can not be
>>> changed after deliver. ( no error page :-().
>> Sounds like the right approach to me. I assume that after deliver() is
>> invoked, various methods such as setHeader() will raise an exception.
> Yes, you're right.
>> I haven't eaten yet, so maybe my brain is low on calories: why did you need
>> the StringIO?
> Not, exactly StringIO. In AsyncThreadedAppServer with my patch I had (
> in 0.4.1 ) buffer with constant size ( for example 1 MB ). Request-data
> (headers and body) was buffered before for it was made Transaction. When
> amount of data is less than 1 MB, all data is buffered. If it is more
> than 1 MB, I need join buffer with data from stream, and on this I write
> own class. This has methods like file/StringIO. Constuctor gets buffer
> and stream. When read-method is called, first return data from buffer,
> and second, after all data from buffer was readed, simply copy data from
> Approximate same solution I have for response. Until ammount of data is
> less then 1 MB, it's buffered. Otherwise, producer ( thread for
> HTTPResponse ) is stopped, until length of buffer is smaller than 1 MB.
> Can you understand me ?
> Webware-discuss mailing list