[Radmind-devel] design of compression

[Maarten T. <mth...@ci...>]

Hi,

1. Changing what functions?

Several snet functions exist to read/write the "200 blabla" lines  
sent between client and server.

* The reads go through snet_readread

* Writes go through snet_write

So the code will need to be changed there.


2. Allow unset_compression?

If we want the ability to switch off the compression mid-stream then  
we're likely to have read some data into the uncompress buffer that's  
not compressed at all. So we'll need to save that data in a new  
buffer for safekeeping until the next snet_readread(). This means  
implementing a new buffer such as those used in snet_getline_multi().  
Similarly, if data is in that buffer and we switch on compression  
again we'll need to take the data from that buffer first before we  
start reading from the socket again.

The complexity is reduced a lot by not allowing the compression to be  
turned off. Is this design limitation OK with you?


3. Flushing

Data that's sent through the snet_gzipwrite() or whatever is sent  
through deflate(), a zlib function. This function tends to 'eat' data  
without producing output until it has enough data to initialize the  
huffman compression algorithm.

So it's easy to end up in a situation where we're sending a small  
file followed by a call to stor_response(). But the data we want a  
stor_response on is still in sender's buffer!

So we need to flush data before waiting for a response. 3 methods:

a) new function snet_flush() is called by stor_response before it  
does any reads (flush before read).

b) the same function is declared static and is called from  
snet_getline_multi() (flush before read).

c) same as b) but the code is called from snet_writef() (flush after  
write).

Using b) seems best because it introduces no interface changes and  
because there are less calls to the flush algorithm. Note that  
flushing also decreases the effectiveness of the compression algorithms.

Method a) allows the program to control the flushing behaviour -  
which may be a good thing: if the program doesn't use  
snet_getline_multi() the send buffer won't get flushed in scenario  
b). Without a flush() exposed in the interface programs will also  
hang at the end: the last command gets sent by the client but the  
client doesn't flush. The client exits, leaving data in the buffer  
and the server hangs waiting for the data.

I suggest doing both a) and b): force a flush before a read in  
snet_getline_multi() and allow programs that never call  
snet_getline_multi() control flushing themselves.

--
maarten