[Bigdata-developers] property path optimizations

[Bryan T. <br...@sy...>]

Jeremy (and others who might have a concern), Michael has implemented some
optimizations for the arbitrary length property path operator that will be
in the 1.5.1 release.  The basic change in the code is:

- The query plan now includes a distinct projection (through a hash index
build) before the property path operator.  The hash index build is a
blocking operation, so if the upstream query plan generates a large number
of solutions with high latency, then the plan will block until that hash
index build is complete.  This is the same for all sub-plans (sub-join
groups, sub-optionals, sub-selects, etc.).
- The property path operator now incrementally evicts solutions;
- There is a streamed hash index join after the property path operator.

So a query such as the following should be quite fast now since the
incoming solution set is empty. It then begins to read on the POS index
with p:=rdf:type.  Solutions are incrementally added to an array buffer.
When that buffer overflows (the first 100 solutions), it will be flushed to
the hash index join.  The hash index join is streamed as well, so it will
output solutions as they are produced.

SELECT * WHERE {
?s rdf:type* ?o
} LIMIT 1

Perhaps in the future we could improve on this by pushing down the limit
onto the property path operator in this case since we know that each
solution generated by the operator will join with the hash index for the
incoming projection.

Thanks,
Bryan