#2 client sends overly large size in "set" command

[dpeckham]

Because the memcached client uses the MemoryStream's GetBuffer() method instead of the ToArray() method, the client sends more bytes than necessary to the server.

A serialized 1000 byte array is 1028 bytes long. So, the server should send the command:

<8 new client connection
<8 set foo 8 0 1028
>8 STORED

However, when you serialize the 1000 bytes to the MemoryStream the internal buffer grows to 2048 bytes, even though only the first 1028 bytes contain the serialized object. The call to GetBuffer() returns a byte[2054] and the client sends this command:

<8 new client connection
<8 set foo 8 0 2054
>8 STORED

The problem here is that when you get to >512K sized objects, the size sent to the server in the set command is over 1024K, which causes the client to return the "SERVER_ERROR object too large for cache error" prematurely.

The following patch fixes the problem. The ToArray() methods only returns the actual bytes written to the stream.

$ diff -c MemCachedClient.cs MemCachedClientOrig.cs
*** MemCachedClient.cs Mon Mar 26 10:13:34 2007
--- MemCachedClientOrig.cs Mon Mar 26 10:13:09 2007
***************
*** 692,698 ****
{
MemoryStream memStream = new MemoryStream();
new BinaryFormatter().Serialize(memStream, obj);
! val = memStream.ToArray();
flags |= F_SERIALIZED;
}
catch(IOException e)
--- 692,698 ----
{
MemoryStream memStream = new MemoryStream();
new BinaryFormatter().Serialize(memStream, obj);
! val = memStream.GetBuffer();
flags |= F_SERIALIZED;
}
catch(IOException e)

The following test code repro's the problem if you run memcached with the "-vv" switch.

public void TestSetCommand()
{
MemcachedClient client = new MemcachedClient();
byte[] data = new byte[1000];
for (int i = 0; i < data.Length; i++)
{
data[i] = 5;
}
client.Set("foo", data);
}

[dpeckham]

Logged In: YES
user_id=1520768
Originator: YES

After more thinking on this, I have a better solution. GetBuffer() is efficient in terms of memory usage--ToArray() creates a copy. But we can fix this better on the SockIO class. If we overload Write(byte[] bytes) to Write(byte[] bytes, int offset, int count) then the method can pass the correct (read "used") length of the buffer on to the network stream:

public void Write(byte[] bytes)
{
this.Write(bytes, 0, bytes.Length);
}

public void Write(byte[] bytes, int offset, int count)
{
...
_networkStream.Write(bytes, offset, count);
}

Then the MemcachedClient will call sock.Write(val,0,realLength) at line 760.

Comments?