#11 ByteBuffer limit isn't respected when serializing DNA

[Jeremy Lewi]

DNA is serialized using the record CompressedSequence. The DNA is represented using an array of bytes which Avro maps to a ByteBuffer. In a bytebuffer the capacity of the buffer can be larger than the actual number of bytes stored which is indicated by the limit of the bytebuffer.

When serializing data, we only want to store the minimum number of required bytes, so the number of bytes written should depend on the limit of the bytebuffer not the capacity.

This doesn't appear to be the case. To surface the problem, modify Sequence.readPackedBytes so that the function throws an exception if the input byte buffer is larger then the minimum size needed.

Its unclear where in the code we start getting byte buffers that are larger then necessary.

Here's my hypothesis:
GraphNode.setSequence(Sequence seq) sets the CompressedSequence to a new byte buffer. The capacity of the bytebuffer is a multiple of 4 because thats what Sequence.toPackedBytes returns. However the limit of the byte buffer is Sequence.numPackedBytes. So the question is whether AVRO respects the limit and automatically compacts the array when serializing it or not.

From my tests, it looks like the data outputted by BuildGraph are correct. The problem appears to start in the reducer in QuickMerge. In particular, after merging nodes the merged nodes will be the ones likely to store sequences which aren't as compressed as they could be.