On 10/31/11 Oct 31 -8:28 AM, Rudolf Schlatte wrote:
>
> On Oct 31, 2011, at 14:15, Robert Goldman wrote:
>
>> On 10/30/11 Oct 30 -11:24 AM, Faré wrote:
>>> On Sun, Oct 30, 2011 at 11:56, Robert Goldman <rpgoldman@...> wrote:
>>>>>> I have been working on a large system that reads data off a socket. The
>>>>>> data is in Google Protocol Buffer format (thanks to Robert Brown's CL
>>>>>> protobuf library), and the socket is being filled by a Java program.
>>>>>
>>>>>> We have been testing this code, and when we set up a simple program that
>>>>>> just reads the messages off the socket (on linux, Java server on the
>>>>>> same machine), the messages come in disastrously slowly, two orders of
>>>>>> magnitude slower than reading the same protocol buffer data out of a file.
>>>>>>
>>>>>> Colleagues have programs in Java and Erlang that read from the same
>>>>>> socket source at quite acceptable speeds.
>>>>>>
>>>>>> Colleagues have also replicated these results on other linux machines,
>>>>>> so I don't believe this is an OS problem.
>>>>>>
>>>>>> Although I haven't figured out how to measure this carefully, when I
>>>>>> look at top, the simple Java and SBCL programs seem to be spending most
>>>>>> of their time idle, whereas I believe they should just be burning
>>>>>> through the data as fast as possible.
>>>>>
>>> I don't know what version of protobuf client you're using, but
>>> it is definitely possible to write vastly inefficient versions of protobuf
>>> by interpreting all the structure at runtime
>>> instead of compiling it as intended.
>>> Where did you get what version of your protobuf implementation?
>>> At a cursory glance, is it compiling or interpreting
>>> the protobuf structures?
>>
>> We are using Bob Brown's August version of CL-protobuf. It has
>> specially compiled methods for the different data structures.
>>
>> Also, I'm inclined to think the fact that we can read these data
>> structures off a file with a 2000-fold speedup suggests it's not the
>> protobuf code that's hurting us. It seems to be socket-specific.
>
> You are not by any chance using cl:read-sequence to read data from a socket?
> The CL standard forbids read-sequence to return with a half-filled buffer if
> the stream didn't EOF, which makes read-sequence unusable for sockets in
> practice.
We /are/ using READ-SEQUENCE, and this may well be the problem. I
ripped the protocol-buffer parsing code out of the loop, and now just
READ-SEQUENCE and just throw the bytes on the floor. Still slow.
What's the right replacement for READ-SEQUENCE? A hasty test seems to
show READ-BYTE to be equally slow.
Thanks,
r
|