[Bigdata-developers] commit: lexicon refactor branch

[Bryan T. <br...@sy...>]

All,

Per below, I've put together a fast forward iterator for the index segments based on multi-block IOs.  This was relatively straightforward and we will be able to exploit the new iterator immediately in compacting merges (note, however, that the implementation is in the LEXICON_REFACTOR_BRANCH). 

Once we finish the scale-out quads query issue, it should be a simple matter to integrate an alternative shard-local join strategy for scale-out which uses the multi-block iterator for unseletive access paths.  This should provide a drammatic performance improvement for closure (since many of the RDFS rules are unselective) and will support high-volume aggregation queries.

Thanks,
Bryan

Implemented a multi-block iterator for forward traversal of the tuples in an index segment.  Unlike the index segment cursor, this class will read 10M blocks from the index segment at a time.  This results in a much higher data transfer rate from the disk since we can read 100s of leaves in a single IO request.

The multi-block iterator will be integrated into query evaluation shortly under a different issue.

The multi-block iterator is in the LEXICON_REFACTOR_BRANCH since that is where we are doing the work on scale-out quads query and where we will handle high volume queries.