[Bigdata-developers] lexicon refactor - closing out the branch (breaks binary compatibility).

[Bryan T. <br...@sy...>]

All,

Once we close out the few remaining issues in the lexicon refactor branch [1,2], I would like to merge from the trunk to the branch to catch this branch up on evolution in the trunk.  We can then do the merge to the trunk and close out this branch.  We are planning a release which incorporates the lexicon refactor change set late this month.  For people who have not been tracking this change set, the lexicon refactor will allow us to inline datatype literals.  This should be a substantial advantage for numberic heavy data sets and aggregation style queries as we will can materialize the RDF Value objects directly from inlined datatype literals.  This advantage is even larger in scale-out than in the standalone Journals.

Please note that the lexicon refactor branch will break binary compatibility.  The change set includes a significant change in the physical schema of the RDF database to permit the inlining of xsd datatype literals into the statement indices.  In addition, there were a number of changes to the internal APIs as we moved from the assumption that the statement indices used long[3] or long[4] keys to variable length coding of a mixture of inlined RDF values and term identifiers as assigned by the lexicon.  We are also going to move long RDF literals out of the ID2TERM index into raw records on the journal which will be migrated automatically to the index segments during overflow processing.

The lexicon refactor change set has been validated againt the Journal implementations, but it will also need to be validated against a cluster.  To that end, I suggest that we tag the trunk before we merge the lexicon refactor branch in to provide a checkpoint that people can use if we have any issues to resolve with the lexicon refactor on the cluster.

Due to the changes in the physical schema, we were unable to avoid a binary compatibility break this time.  However, we should be better positioned in the future for forward compatible changes.  Given that we have a binary compatibility break with this change set, I plan to roll in a few other changes at the same time which would otherwise have caused problems with backward compatibility [3,4].

I will be mostly unavailable W/Th/F.  Therefore, I would like to merge the lexicon branch back to the trunk either today or early next week.

Thanks,
Bryan

[1] https://sourceforge.net/apps/trac/bigdata/ticket/59
[2] https://sourceforge.net/apps/trac/bigdata/ticket/109
[3] https://sourceforge.net/apps/trac/bigdata/ticket/107
[4] https://sourceforge.net/apps/trac/bigdata/ticket/41