Menu
▾
▴
bigdata-commit
|
From: <tho...@us...> - 2014-01-03 19:18:28
|
Revision: 7712
http://bigdata.svn.sourceforge.net/bigdata/?rev=7712&view=rev
Author: thompsonbry
Date: 2014-01-03 19:18:16 +0000 (Fri, 03 Jan 2014)
Log Message:
-----------
Changes to PipelineJoin, IBindingSetAccessPath, IHashJoinUtility,
JVMHashJoinUtility, HTreeHashJoinUtility, ServiceCallOp, etc. intended
to reduce re-chunking during vectored query evaluation. This change
provides correct accounting for chunksIn and unitsIn for the solution
set hash join and improved memory utilization through elimination of
some unnecessary dechunking and rechunking in the hash join API.
Changes were also made to the query hint infrastructure in order to
ensure that hint:chunkSize was correctly applied to HTree
operators. This is important now that the HTreeHashJoinUtility no
longer rechunks to a hard-coded chunkSize of 1000.
While working on this, I noticed that hint:chunkSize was not making it
onto the SliceOp, onto the Predicate associated with a PipelineJoin
(where it controls the vectoring for reading on the access path),
etc. This was due to both the types of nodes to which the IQueryHint
implementations were willing to apply themselves and the types of
nodes to which the ASTQueryHintOptimizer was willing to apply query
hints. I have made both more expansive.
To help analyze the issue with query hints (which was necessary to
assess the performance impact of my change to vectoring in the
HTreeHashJoinUtility) and to help analyze the problem with the
stochastic behavior of the HTree, I also added significantly more
information into the predSummary and added columns (in the detailed
explain mode) for the PipelineOp and Predicate annotations.
I also touched a lot of classes, adding @Override and final
annotations. A number of this were in the striterator package (when I
added a CloseableChunkedIteratorWrapperConverter to address the
rechunking pattern in HTreeHashJoinUtility, IBindingSetAccessPath, and
BOpContext#solutions()) and in the ast package. Several of join
operators were also touched, either to support the fix of the
rechunking pattern or to eliminate some old (and commented out) code.
I have added more unit tests of the query hints mechanisms. I found
and fixed several places where query hints were not being applied when
generating pipeline operators (AST2BOpUtility) and where query hints
were not being copied from one AST node to another when making
structural changes to the AST, e.g., ASTBottomUpOptimizer and
ASTSparql11SubqueryOptimizer both create a NamedSubqueryRoot and a
NamedSubqueryInclude. However, they were not copying across the query
hints from the parent join group (ASTBottomUpOptimizer) and the
original SubqueryRoot (ASTSparql11SubqueryOptimizer).
com.bigdata.rdf.sparql.ast::
- ASTBase: javadoc on Annotations.QUERY_HINTS. final and @Override annotations.
- AssignmentNode: final and @Override annotations.
- GraphPatternGroup: final and @Override annotations.
- GraphNodeGroup: license header, final and @Override annotations, toString(int) changes.
- JoinGroupNode: license header. Made the OPTIMIZER property explicit. Added getQueryOptimizer() method. final and @Override annotations.
- QueryBase: added query hints into toString(int).
- QueryHints: referenced ticket #791 (clean up query hints).
- QueryOptimizerEnum: removed dead code.
- QueryRoot: final, @Override, and javadoc correction.
- SliceNode: javadoc on annotations, toString(int) now shows the query hints. this was done to have visibility into vectoring.
- StatementPatternNode: license header, final and @Override, javadoc, dead code elimination.
- ValueExpressionListBaseNode: @Override
com.bigdata.striterator::
- CloseableChunkedIteratorWrapperConverter: new class converts from IChunkedIterator<E> to ICloseableIterator visiting E[].
- TestCloseableChunkedIteratorWrapperConverter: new test suite.
- TestAll: include new test class.
- AbstractChunkedResolverator: final, @Override.
- ChunkedArrayIterator: final, @Override.
- ChunkedArraysIterator: javadoc, final, @Override.
- ChunkedConvertingIterator: final, @Override.
- ChunkedResolvingIterator: javadoc, final, @Override.
- ChunkedWrappedIterator: final, @Override.
- Chunkerator: close() now tests for ICloseable rather than ICloseableIterator.
- CloseableIteratorWrapper: final, @Override.
- Dechunkerator: javadoc, final, @Override, close() now tests for ICloseable rather than ICloseableIterator.
- DelegateChunkedIterator: final, @Override.
- GenericChunkedStriterator: removed unnecessary @SuppressWarning.
- IChunkedIterator: @Override for methods declared by Iterator.
- IChunkedStriterator: @Override for methods in base interface.
- MergeFilter: final, @Override.
- PushbackIterator: final, @Override.
- Resolver: final, @Override.
- Striterator: final, @Override.
com.bigdata.bop.join::
- TestPipelineJoin: @Override and super.setUp() / super.tearDown().
- AbstractHashJoinUtilityTestCase: Modified how we execute the hash join for the IHashJoinUtility API change.
- HashIndexOp: @Override
- HashJoinOp: @Override, javadoc, API change for IHashJoinUtility.hashJoin().
- HTreeHashIndexOp: @Override, final, dead code eliminated.
- HTreeHashJoinUtility: removed static chunkSize field. Vectoring is now controlled by hint:chunkSize. Pushed down the logic to track chunskIn and unitsIn for hashJoin2 (they were not being tracked).
- IHashJoinUtility: API change to remove rechunking pattern.
- JoinVariableNotBoundException: final annotations.
- JVMHashIndex: Slight efficiency change in makeKey(). @Override
- JVMHashIndexOp: final annotation.
- JVMHashJoinUtility: API change for hashJoin2(): now accepts ICloseableIteratoe<IBindingSet[]> and BOpStats and tracks unitsIn and chunksIn.
- PipelineJoin: IBindingSetAccessPath now returns an ICloseableIterator<IBindingSet[]>. Modified the AccessPathTask to handle the IBindingSet[] chunks. Used to be just IBindingSets.
- SolutionSetHashJoin: Modified to pass context.getSource() and BOpStats into the IHashJoinUtility rather than dechunking.
com.bigdata.bop.controller::
- HTreeNamedSubqueryOp, JVMNamedSubqueryOp, and INamedSubqueryOp: Added INamedSubqueryOp as a marker interface for the two implementation classes so we can identify those operators when they appear in a query plan.
- ServiceCallJoin: Modified to pass ICloseableIterator<IBindingSet[]> into IHashJoinUtility. This should be pushed down further. There is a TODO to do that when we address vectoring in/out of the SERVICE operator.
com.bigdata.bop.QueryEngine:
- QueryLog: Significantly expanded and improved performance counter reporting for Explain, especially in the "detail" mode.
com.bigdata.bop::
- AbstractAccessPathOp: removed unused methods. This is part of the query hints cleanup.
- BOpContext#solutions() was modified to return an ICloseableIterator visiting IBindingSet[]s. This is part of the rechunking change for IHashJoinUtility.
- BOpUtility: Added getOnly() method used to obtain the only instance of a BOp from a query plan or AST. This is used by unit tests.
com.bigdata.relation.accesspath::
- IBindingSetAccessPath: solutions() was modified to return an ICloseableIterator visiting IBindingSet[]s.
- AccessPath: solutions() was modified to return an ICloseableIterator visiting IBindingSet[]s.
com.bigdata.rdf.sparql.hints::
- Extensive changes to clean up query hints. Many query hints are now apply to IQueryNode rather than IJoinNode. javadoc. Test cases have been expanded.
com.bigdata.rdf.sparql.ast::
- ASTBottomUpOptimizer: modified to pass through query hints from the parent join group to the lifted out named subquery and the INCLUDE.
- ASTSparql11SubqueryOptimizer: modified to pass through query hints from the original subquery when it is lifted out into a named subquery. The hints are applied to both the new named subquery and the new named subquery include.
- ASTStaticJoinOptimizer: changes to isStaticOptimizer() to use JoinGroup.getQueryOptimizer();
- ASTQueryHintOptimizer: extensive changes. modified how the optimizer identifies the nodes for which it will delegate to a query hint - it used to only do this for QueryNodeBase, which was too restrictive. It now does this for everthing but value expressions. javadoc clarifying the intention of the class. Removed some dead code.
Note: ASTQueryHintOptimizer No longer adds all query hints with Scope:=Query to AST2BOpContext.queryHints. I have clarified this in documentation in both classes.
com.bigdata.rdf.sparql.eval::
- AST2BOpBase: Changed the pattern for applyQueryHints() to use both the AST node's query hints and the AST2BOpUtility global defaults whenever possible.
- AST2BOpUtility: Modified to more systematically pass along query hints from the AST node to the constructed PipelineOps. Modified to pass through BufferAnnotations and IPredicate annotations to the Predicate created from a StatementPatternNode. This allows us to control vectoring for AccessPath reads.
- AST2BOpFilters: Partially addressed pass through of query hints, but only those in the global scope. We need to change the method interfaces to pass through the bounding AST node or the queryHints for that AST node.
- AST2BOpContext: license header, javadoc.
I have run through all of the bop, AST, SPARQL, and NSS test suites
and everything is green.
TODO:
- (*) I have not yet resolved the HTree stochastic behavior. I will
continue to look at that once this checkpoint is committed. See
#763.
- (*) Query hints for the materialization pipeline
(ChunkedMaterializationOp, ConditionalRoutingOp) are not being
applied correctly because the caller's AST node (or its query
hints) are not being passed down. I am going to defer fixing this
for a moment while I look at the RTO integration. (The RTO needs
to be able to use AST2BOpJoins#join(), which is the main entry
point into the materialization pipeline code.) See #791.
- AST2BOpBase: Once we fix this, we really do not need to pass in
the AST2BOpContext's query hints into applyQueryHints() any
more. The impact will be achieved by passing down the query
hints from the appropriate bounding AST node. The
ASTQueryHintOptimizer is responsible for making sure that the
query hints are applied to those AST nodes.
- (*) Check query performance on LUBM U50, BSBM 100M, and govtrack.
The changes in the HTree vectoring could hit govtrack, but we
should be able to override hint:chunkSize if necessary to correct
for this (or automatically increase the chunkSize for the analytic
query mode, or use dynamic rechunking, etc). The changes in the
ASTQueryHintsOptimizer could hit all queries, but I was pretty
careful and do not expect to see any performance regressions.
See http://sourceforge.net/apps/trac/bigdata/ticket/483 (Eliminate unnecessary dechunking and rechunking)
See http://sourceforge.net/apps/trac/bigdata/ticket/791 (Clean up query hints)
See http://sourceforge.net/apps/trac/bigdata/ticket/763 (Stochastic results with Analytic Query Mode)
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/AbstractAccessPathOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpContext.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpUtility.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/HTreeNamedSubqueryOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/JVMNamedSubqueryOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/ServiceCallJoin.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashIndexOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HashIndexOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HashJoinOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/IHashJoinUtility.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/JVMHashIndex.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/JVMHashIndexOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/JoinVariableNotBoundException.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/SolutionSetHashJoinOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/ProjectionOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/relation/accesspath/AccessPath.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/relation/accesspath/IBindingSetAccessPath.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/AbstractChunkedResolverator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/ChunkedArrayIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/ChunkedArraysIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/ChunkedConvertingIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/ChunkedOrderedStriterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/ChunkedResolvingIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/ChunkedWrappedIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/Chunkerator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/CloseableIteratorWrapper.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/Dechunkerator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/DelegateChunkedIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/GenericChunkedStriterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/IChunkedIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/IChunkedStriterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/MergeFilter.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/PushbackIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/Resolver.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/Striterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/join/AbstractHashJoinUtilityTestCase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/join/TestPipelineJoin.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/striterator/TestAll.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/ASTBase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/AssignmentNode.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/GraphPatternGroup.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/GroupNodeBase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/JoinGroupNode.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/QueryBase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/QueryHints.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/QueryOptimizerEnum.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/QueryRoot.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/SliceNode.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/StatementPatternNode.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/ValueExpressionListBaseNode.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpBase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpContext.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpFilters.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AbstractChunkSizeHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AbstractQueryHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AnalyticQueryHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AtOnceHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/BufferChunkCapacityHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/BufferChunkOfChunksCapacityHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/ChunkSizeHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/IQueryHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/OptimizerQueryHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/PipelineMaxMessagesPerTaskHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/PipelineMaxParallelHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/PipelineQueueCapacityHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/QueryHintRegistry.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/RTOSampleTypeQueryHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTBottomUpOptimizer.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTQueryHintOptimizer.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTSparql11SubqueryOptimizer.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTStaticJoinOptimizer.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/service/ServiceCall.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/TestQueryHints.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/query-hints-01.rq
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/query-hints-06.rq
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/INamedSubqueryOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/CloseableChunkedIteratorWrapperConverter.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/striterator/TestCloseableChunkedIteratorWrapperConverter.java
Property Changed:
----------------
branches/BIGDATA_RELEASE_1_3_0/
Property changes on: branches/BIGDATA_RELEASE_1_3_0
___________________________________________________________________
Modified: svn:ignore
- ant-build
src
bin
bigdata*.jar
ant-release
standalone
test*
countersfinal.xml
events.jnl
.settings
*.jnl
TestInsertRate.out
SYSTAP-BBT-result.txt
U10load+query
*.hprof
com.bigdata.cache.TestHardReferenceQueueWithBatchingUpdates.exp.csv
commit-log.txt
eventLog
dist
bigdata-test
com.bigdata.rdf.stress.LoadClosureAndQueryTest.*.csv
DIST.bigdata-*.tgz
REL.bigdata-*.tgz
queryLog*
queryRunState*
sparql.txt
benchmark
CI
+ ant-build
src
bin
bigdata*.jar
ant-release
standalone
test*
countersfinal.xml
events.jnl
.settings
*.jnl
TestInsertRate.out
SYSTAP-BBT-result.txt
U10load+query
*.hprof
com.bigdata.cache.TestHardReferenceQueueWithBatchingUpdates.exp.csv
commit-log.txt
eventLog
dist
bigdata-test
com.bigdata.rdf.stress.LoadClosureAndQueryTest.*.csv
DIST.bigdata-*.tgz
REL.bigdata-*.tgz
queryLog*
queryRunState*
sparql.txt
benchmark
CI
bsbm10-dataset.nt.gz
bsbm10-dataset.nt.zip
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/AbstractAccessPathOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/AbstractAccessPathOp.java 2013-12-31 19:48:13 UTC (rev 7711)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/AbstractAccessPathOp.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -76,26 +76,26 @@
super(op);
}
- /**
- * @see BufferAnnotations#CHUNK_CAPACITY
- */
- protected int getChunkCapacity() {
-
- return getProperty(Annotations.CHUNK_CAPACITY,
- Annotations.DEFAULT_CHUNK_CAPACITY);
+// /**
+// * @see BufferAnnotations#CHUNK_CAPACITY
+// */
+// protected int getChunkCapacity() {
+//
+// return getProperty(Annotations.CHUNK_CAPACITY,
+// Annotations.DEFAULT_CHUNK_CAPACITY);
+//
+// }
+//
+// /**
+// * @see BufferAnnotations#CHUNK_OF_CHUNKS_CAPACITY
+// */
+// protected int getChunkOfChunksCapacity() {
+//
+// return getProperty(Annotations.CHUNK_OF_CHUNKS_CAPACITY,
+// Annotations.DEFAULT_CHUNK_OF_CHUNKS_CAPACITY);
+//
+// }
- }
-
- /**
- * @see BufferAnnotations#CHUNK_OF_CHUNKS_CAPACITY
- */
- protected int getChunkOfChunksCapacity() {
-
- return getProperty(Annotations.CHUNK_OF_CHUNKS_CAPACITY,
- Annotations.DEFAULT_CHUNK_OF_CHUNKS_CAPACITY);
-
- }
-
// protected int getFullyBufferedReadThreshold() {
//
// return getProperty(Annotations.FULLY_BUFFERED_READ_THRESHOLD,
@@ -103,14 +103,14 @@
//
// }
- /**
- * @see BufferAnnotations#CHUNK_TIMEOUT
- */
- protected long getChunkTimeout() {
-
- return getProperty(Annotations.CHUNK_TIMEOUT,
- Annotations.DEFAULT_CHUNK_TIMEOUT);
-
- }
+// /**
+// * @see BufferAnnotations#CHUNK_TIMEOUT
+// */
+// protected long getChunkTimeout() {
+//
+// return getProperty(Annotations.CHUNK_TIMEOUT,
+// Annotations.DEFAULT_CHUNK_TIMEOUT);
+//
+// }
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpContext.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpContext.java 2013-12-31 19:48:13 UTC (rev 7711)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpContext.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -59,8 +59,9 @@
import com.bigdata.rwstore.sector.IMemoryManager;
import com.bigdata.striterator.ChunkedFilter;
import com.bigdata.striterator.Chunkerator;
-import com.bigdata.striterator.CloseableIteratorWrapper;
+import com.bigdata.striterator.CloseableChunkedIteratorWrapperConverter;
import com.bigdata.striterator.IChunkedIterator;
+import com.bigdata.striterator.IChunkedStriterator;
import cutthecrap.utils.striterators.ICloseableIterator;
@@ -1078,8 +1079,8 @@
}
/**
- * Convert an {@link IAccessPath#iterator()} into a stream of
- * {@link IBindingSet}s.
+ * Convert an {@link IAccessPath#iterator()} into a stream of chunks of
+ * {@link IBindingSet}.
*
* @param src
* The iterator draining the {@link IAccessPath}. This will visit
@@ -1090,7 +1091,7 @@
* Statistics to be updated as elements and chunks are consumed
* (optional).
*
- * @return The dechunked iterator visiting the solutions. The order of the
+ * @return An iterator visiting chunks of solutions. The order of the
* original {@link IElement}s is preserved.
*
* @see https://sourceforge.net/apps/trac/bigdata/ticket/209 (AccessPath
@@ -1105,14 +1106,15 @@
// * The array of distinct variables (no duplicates) to be
// * extracted from the visited {@link IElement}s.
@SuppressWarnings({ "rawtypes", "unchecked" })
- static public ICloseableIterator<IBindingSet> solutions(
+ static public ICloseableIterator<IBindingSet[]> solutions(
final IChunkedIterator<?> src, //
final IPredicate<?> pred,//
// final IVariable<?>[] varsx,
final BaseJoinStats stats//
) {
- return new CloseableIteratorWrapper(
+ //return new CloseableIteratorWrapper(
+ final IChunkedStriterator itr1 =
new com.bigdata.striterator.ChunkedStriterator(src).addFilter(
// new ChunkedFilter() {
new ChunkedFilter<IChunkedIterator<Object>, Object, Object>() {
@@ -1160,18 +1162,28 @@
}
- })) {
+ });
+ //) {
+//
+// /**
+// * Close the real source if the caller closes the returned iterator.
+// */
+// @Override
+// public void close() {
+// super.close();
+// src.close();
+// }
+// };
- /**
- * Close the real source if the caller closes the returned iterator.
- */
- @Override
- public void close() {
- super.close();
- src.close();
- }
- };
+ /*
+ * Convert from IChunkedIterator<IBindingSet> to
+ * ICloseableIterator<IBindingSet[]>. This is a fly weight conversion.
+ */
+ final ICloseableIterator<IBindingSet[]> itr2 = new CloseableChunkedIteratorWrapperConverter<IBindingSet>(
+ itr1);
+ return itr2;
+
}
/*
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpUtility.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpUtility.java 2013-12-31 19:48:13 UTC (rev 7711)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpUtility.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -35,6 +35,7 @@
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
+import java.util.NoSuchElementException;
import java.util.Set;
import org.apache.log4j.Logger;
@@ -72,7 +73,7 @@
* Pre-order recursive visitation of the operator tree (arguments only, no
* annotations).
*/
- @SuppressWarnings("unchecked")
+ @SuppressWarnings({ "unchecked", "rawtypes" })
public static Iterator<BOp> preOrderIterator(final BOp op) {
return new Striterator(new SingleValueIterator(op))
@@ -466,6 +467,8 @@
* The type of the node to be extracted.
*
* @return A list containing those references.
+ *
+ * @see #visitAll(BOp, Class)
*/
public static <C> List<C> toList(final BOp op, final Class<C> clas) {
@@ -483,6 +486,44 @@
}
+ /**
+ * Return the sole instance of the specified class.
+ *
+ * @param op
+ * The root of the traversal.
+ * @param class1
+ * The class to look for.
+ * @return The sole instance of that class.
+ * @throws NoSuchElementException
+ * if there is no such instance.
+ * @throws RuntimeException
+ * if there is more than one such instance.
+ */
+ public static <C> C getOnly(final BOp op, final Class<C> class1) {
+ final Iterator<C> it = visitAll(op, class1);
+ if (!it.hasNext())
+ throw new NoSuchElementException("No instance found: class="
+ + class1);
+ final C ret = it.next();
+ if (it.hasNext())
+ throw new RuntimeException("More than one instance exists: class="
+ + class1);
+ return ret;
+ }
+
+ /**
+ * Return an iterator visiting references to all nodes of the given type
+ * (recursive, including annotations).
+ *
+ * @param op
+ * The root of the operator tree.
+ * @param clas
+ * The type of the node to be extracted.
+ *
+ * @return A iterator visiting those references.
+ *
+ * @see #toList(BOp, Class)
+ */
@SuppressWarnings("unchecked")
public static <C> Iterator<C> visitAll(final BOp op, final Class<C> clas) {
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/HTreeNamedSubqueryOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/HTreeNamedSubqueryOp.java 2013-12-31 19:48:13 UTC (rev 7711)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/HTreeNamedSubqueryOp.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -73,7 +73,7 @@
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
*/
-public class HTreeNamedSubqueryOp extends PipelineOp {
+public class HTreeNamedSubqueryOp extends PipelineOp implements INamedSubqueryOp {
static private final transient Logger log = Logger
.getLogger(HTreeNamedSubqueryOp.class);
@@ -151,7 +151,7 @@
}
- public HTreeNamedSubqueryOp(final BOp[] args, NV... annotations) {
+ public HTreeNamedSubqueryOp(final BOp[] args, final NV... annotations) {
this(args, NV.asMap(annotations));
@@ -164,6 +164,7 @@
}
+ @Override
public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
return new FutureTask<Void>(new ControllerTask(this, context));
@@ -266,6 +267,7 @@
/**
* Evaluate.
*/
+ @Override
public Void call() throws Exception {
try {
@@ -356,6 +358,7 @@
}
+ @Override
public Void call() throws Exception {
// The subquery
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/INamedSubqueryOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/INamedSubqueryOp.java (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/INamedSubqueryOp.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -0,0 +1,42 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+package com.bigdata.bop.controller;
+
+import com.bigdata.bop.join.SolutionSetHashJoinOp;
+
+/**
+ * Marker interface for named subquery evaluation. Solutions from the pipeline
+ * flow through this operator without modification. The subquery is evaluated
+ * exactly once, the first time this operator is invoked, and the solutions for
+ * the subquery are written onto a hash index. Those solutions are then joined
+ * back within the query at latter points in the query plan using a solution set
+ * hash join.
+ *
+ * @see SolutionSetHashJoinOp
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ */
+public interface INamedSubqueryOp {
+
+}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/JVMNamedSubqueryOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/JVMNamedSubqueryOp.java 2013-12-31 19:48:13 UTC (rev 7711)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/JVMNamedSubqueryOp.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -73,7 +73,7 @@
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
*/
-public class JVMNamedSubqueryOp extends PipelineOp {
+public class JVMNamedSubqueryOp extends PipelineOp implements INamedSubqueryOp {
static private final transient Logger log = Logger
.getLogger(JVMNamedSubqueryOp.class);
@@ -140,7 +140,7 @@
}
- public JVMNamedSubqueryOp(final BOp[] args, NV... annotations) {
+ public JVMNamedSubqueryOp(final BOp[] args, final NV... annotations) {
this(args, NV.asMap(annotations));
@@ -153,6 +153,7 @@
}
+ @Override
public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
return new FutureTask<Void>(new ControllerTask(this, context));
@@ -254,6 +255,7 @@
/**
* Evaluate.
*/
+ @Override
public Void call() throws Exception {
try {
@@ -344,6 +346,7 @@
}
+ @Override
public Void call() throws Exception {
// The subquery
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/ServiceCallJoin.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/ServiceCallJoin.java 2013-12-31 19:48:13 UTC (rev 7711)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/ServiceCallJoin.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -69,6 +69,7 @@
import com.bigdata.relation.accesspath.IBuffer;
import com.bigdata.relation.accesspath.UnsyncLocalOutputBuffer;
import com.bigdata.striterator.ChunkedArrayIterator;
+import com.bigdata.striterator.Chunkerator;
import com.bigdata.util.InnerCause;
import com.bigdata.util.concurrent.LatchedExecutor;
@@ -571,6 +572,7 @@
}
+ @Override
public Void call() throws Exception {
final UnsyncLocalOutputBuffer<IBindingSet> unsyncBuffer = new UnsyncLocalOutputBuffer<IBindingSet>(
@@ -592,7 +594,7 @@
chunk), null/* stats */);
// The iterator draining the subquery
- ICloseableIterator<IBindingSet> serviceSolutionItr = null;
+ ICloseableIterator<IBindingSet[]> serviceSolutionItr = null;
try {
/*
@@ -609,10 +611,13 @@
* Do a hash join of the source solutions with the
* solutions from the service, outputting any solutions
* which join.
+ *
+ * Note:
*/
- state.hashJoin(serviceSolutionItr, unsyncBuffer);
-
+ state.hashJoin(serviceSolutionItr, null/* stats */,
+ unsyncBuffer);
+
}
} finally {
@@ -671,26 +676,35 @@
* SILENT is <code>true</code>.
*
* @throws Exception
+ *
+ * TODO RECHUNKING Push down the
+ * ICloseableIterator<IBindingSet[]> return type into
+ * the {@link ServiceCall} interface and the various
+ * ways in which we can execute a service call. Do this
+ * as part of vectoring solutions in and out of service
+ * calls?
*/
- private ICloseableIterator<IBindingSet> doServiceCall(
+ private ICloseableIterator<IBindingSet[]> doServiceCall(
final ServiceCall<? extends Object> serviceCall,
final IBindingSet[] left) throws Exception {
try {
+ final ICloseableIterator<IBindingSet> itr;
+
if (serviceCall instanceof BigdataServiceCall) {
- return doBigdataServiceCall(
+ itr = doBigdataServiceCall(
(BigdataServiceCall) serviceCall, left);
} else if (serviceCall instanceof ExternalServiceCall) {
- return doExternalServiceCall(
+ itr = doExternalServiceCall(
(ExternalServiceCall) serviceCall, left);
} else if (serviceCall instanceof RemoteServiceCall) {
- return doRemoteServiceCall(
+ itr = doRemoteServiceCall(
(RemoteServiceCall) serviceCall, left);
} else {
@@ -698,7 +712,13 @@
throw new AssertionError();
}
+
+ final ICloseableIterator<IBindingSet[]> itr2 = new Chunkerator<IBindingSet>(
+ itr, op.getChunkCapacity(), IBindingSet.class);
+
+ return itr2;
+
} catch (Throwable t) {
if (silent
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java 2013-12-31 19:48:13 UTC (rev 7711)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -61,6 +61,8 @@
import com.bigdata.bop.solutions.DropOp;
import com.bigdata.bop.solutions.GroupByOp;
import com.bigdata.bop.solutions.ProjectionOp;
+import com.bigdata.bop.solutions.SliceOp;
+import com.bigdata.btree.Tuple;
import com.bigdata.counters.render.XHTMLRenderer;
import com.bigdata.rawstore.Bytes;
import com.bigdata.rdf.sparql.ast.eval.AST2BOpJoins;
@@ -826,6 +828,10 @@
}
w.write("<th>bopSummary</th>");
w.write("<th>predSummary</th>");
+ if (detailedStats) {
+ w.write("<th>bopAnnotations</th>");
+ w.write("<th>predAnnotations</th>");
+ }
// metadata considered by the static optimizer.
if(detailedStats) {
w.write("<th>staticBestKeyOrder</th>"); // original key order assigned
@@ -1119,25 +1125,16 @@
w.write(TDx);
}
+ // bopSummary
w.write(TD);
- if(summary) {
+ if (summary) {
w.write("total");
} else {
w.write(cdata(bop.getClass().getSimpleName()));
w.write(cdata("[" + bopId + "]"));
- final Integer defaultSink = (Integer) bop
- .getProperty(PipelineOp.Annotations.SINK_REF);
- final Integer altSink = (Integer) bop
- .getProperty(PipelineOp.Annotations.ALT_SINK_REF);
- if (defaultSink != null) {
- w.write(cdata(", sink=" + defaultSink));
- }
- if (altSink != null) {
- w.write(cdata(", altSink=" + altSink));
- }
}
w.write(TDx);
-
+
/*
* Pperator summary (not shown for the "total" line).
*
@@ -1264,9 +1261,29 @@
w.write(cdata(Arrays.toString(((ProjectionOp) bop)
.getVariables())));
}
+ if (bop instanceof SliceOp) {
+ w.write(cdata("offset=" + ((SliceOp) bop).getOffset()));
+ w.write(cdata(", limit=" + ((SliceOp) bop).getLimit()));
+ }
}
- w.write(TDx); // end summary
+ w.write(TDx); // end predSummary
+ if (detailedStats) {
+ // bopAnnotations
+ w.write(TD);
+ showAnnotations(w, bop.annotations());
+ w.write(TDx);
+ }
+
+ if (detailedStats) {
+ // predAnnotations
+ w.write(TD);
+ if (pred != null) {
+ showAnnotations(w, pred.annotations());
+ }
+ w.write(TDx);
+ }
+
/*
* Static optimizer metadata.
*
@@ -1505,6 +1522,41 @@
}
/**
+ * Shows annotations on a {@link BOp}.
+ *
+ * @param w
+ * Where to write the XHTML data.
+ * @param anns
+ * The annotations (optional).
+ * @throws IOException
+ */
+ static private void showAnnotations(final Writer w,
+ final Map<String, Object> anns) throws IOException {
+ if (anns != null && !anns.isEmpty()) {
+ w.write("<dl>");
+ for (Map.Entry<String, Object> e : anns.entrySet()) {
+ w.write("<dt>");
+ final String key = e.getKey();
+ w.write(cdata(key));
+ w.write("</dt><dd>");
+ final Object val = e.getValue();
+ // See CoreBaseBop for this pattern.
+ if (val != null && val.getClass().isArray()) {
+ w.write(cdata(Arrays.toString((Object[]) val)));
+ } else if (key.equals(IPredicate.Annotations.FLAGS)) {
+ w.write(cdata(Tuple.flagString((Integer) val)));
+ } else if (val instanceof BOp) {
+ w.write(cdata(((BOp) val).toShortString()));
+ } else {
+ w.write(cdata("" + val));
+ }
+ w.write("</dd>");
+ }
+ w.write("</dl>");
+ }
+ }
+
+ /**
* Write a summary row for the query. The table element, header, and footer
* must be written separately.
*
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashIndexOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashIndexOp.java 2013-12-31 19:48:13 UTC (rev 7711)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashIndexOp.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -72,222 +72,22 @@
super(args, annotations);
-// if (getEvaluationContext() != BOpEvaluationContext.CONTROLLER) {
-// throw new IllegalArgumentException(
-// BOp.Annotations.EVALUATION_CONTEXT + "="
-// + getEvaluationContext());
-// }
-//
-// if (getMaxParallel() != 1) {
-// /*
-// * Parallel evaluation is not allowed. This operator writes on an
-// * object that is not thread-safe for mutation.
-// */
-// throw new IllegalArgumentException(
-// PipelineOp.Annotations.MAX_PARALLEL + "="
-// + getMaxParallel());
-// }
-//
-// if (!isLastPassRequested()) {
-// /*
-// * Last pass evaluation must be requested. This operator will not
-// * produce any outputs until all source solutions have been
-// * buffered.
-// */
-// throw new IllegalArgumentException(PipelineOp.Annotations.LAST_PASS
-// + "=" + isLastPassRequested());
-// }
-//
-// getRequiredProperty(Annotations.NAMED_SET_REF);
-//
-// @SuppressWarnings("unused")
-// final JoinTypeEnum joinType = (JoinTypeEnum) getRequiredProperty(Annotations.JOIN_TYPE);
-//
-// // Join variables must be specified.
-// final IVariable<?>[] joinVars = (IVariable[]) getRequiredProperty(Annotations.JOIN_VARS);
-//
-//// if (joinVars.length == 0)
-//// throw new IllegalArgumentException(Annotations.JOIN_VARS);
-//
-// for (IVariable<?> var : joinVars) {
-//
-// if (var == null)
-// throw new IllegalArgumentException(Annotations.JOIN_VARS);
-//
-// }
-
}
- public HTreeHashIndexOp(final BOp[] args, NV... annotations) {
+ public HTreeHashIndexOp(final BOp[] args, final NV... annotations) {
this(args, NV.asMap(annotations));
-
+
}
-// @Override
-// public BOpStats newStats() {
-//
-// return new NamedSolutionSetStats();
-//
-// }
-
@Override
protected HTreeHashJoinUtility newState(
final BOpContext<IBindingSet> context,
final INamedSolutionSetRef namedSetRef, final JoinTypeEnum joinType) {
- return new HTreeHashJoinUtility(
- context.getMemoryManager(namedSetRef.getQueryId()), this, joinType);
+ return new HTreeHashJoinUtility(context.getMemoryManager(namedSetRef
+ .getQueryId()), this, joinType);
}
-
-// public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
-//
-// return new FutureTask<Void>(new ControllerTask(this, context));
-//
-// }
-
-// /**
-// * Evaluates the subquery for each source binding set. If the controller
-// * operator is interrupted, then the subqueries are cancelled. If a subquery
-// * fails, then all subqueries are cancelled.
-// */
-// private static class ControllerTask implements Callable<Void> {
-//
-// private final BOpContext<IBindingSet> context;
-//
-// private final HTreeHashIndexOp op;
-//
-// private final NamedSolutionSetStats stats;
-//
-// private final IHashJoinUtility state;
-//
-// public ControllerTask(final HTreeHashIndexOp op,
-// final BOpContext<IBindingSet> context) {
-//
-// if (op == null)
-// throw new IllegalArgumentException();
-//
-// if (context == null)
-// throw new IllegalArgumentException();
-//
-// this.context = context;
-//
-// this.op = op;
-//
-// this.stats = ((NamedSolutionSetStats) context.getStats());
-//
-// // Metadata to identify the named solution set.
-// final NamedSolutionSetRef namedSetRef = (NamedSolutionSetRef) op
-// .getRequiredProperty(Annotations.NAMED_SET_REF);
-//
-// {
-//
-// /*
-// * First, see if the map already exists.
-// *
-// * Note: Since the operator is not thread-safe, we do not need
-// * to use a putIfAbsent pattern here.
-// */
-//
-// // Lookup the attributes for the query on which we will hang the
-// // solution set.
-// final IQueryAttributes attrs = context
-// .getQueryAttributes(namedSetRef.queryId);
-//
-// HTreeHashJoinUtility state = (HTreeHashJoinUtility) attrs
-// .get(namedSetRef);
-//
-// if (state == null) {
-//
-// final JoinTypeEnum joinType = (JoinTypeEnum) op
-// .getRequiredProperty(Annotations.JOIN_TYPE);
-//
-// state = new HTreeHashJoinUtility(
-// context.getMemoryManager(namedSetRef.queryId), op,
-// joinType);
-//
-// if (attrs.putIfAbsent(namedSetRef, state) != null)
-// throw new AssertionError();
-//
-// }
-//
-// this.state = state;
-//
-// }
-//
-// }
-//
-// /**
-// * Evaluate.
-// */
-// public Void call() throws Exception {
-//
-// try {
-//
-// // Buffer all source solutions.
-// acceptSolutions();
-//
-// if(context.isLastInvocation()) {
-//
-// // Checkpoint the solution set.
-// checkpointSolutionSet();
-//
-// // Output the buffered solutions.
-// outputSolutions();
-//
-// }
-//
-// // Done.
-// return null;
-//
-// } finally {
-//
-// context.getSource().close();
-//
-// context.getSink().close();
-//
-// }
-//
-// }
-//
-// /**
-// * Buffer intermediate resources.
-// */
-// private void acceptSolutions() {
-//
-// state.acceptSolutions(context.getSource(), stats);
-//
-// }
-//
-// /**
-// * Checkpoint and save the solution set.
-// */
-// private void checkpointSolutionSet() {
-//
-// state.saveSolutionSet();
-//
-// }
-//
-// /**
-// * Output the buffered solutions.
-// */
-// private void outputSolutions() {
-//
-// // default sink
-// final IBlockingBuffer<IBindingSet[]> sink = context.getSink();
-//
-// final UnsyncLocalOutputBuffer<IBindingSet> unsyncBuffer = new UnsyncLocalOutputBuffer<IBindingSet>(
-// op.getChunkCapacity(), sink);
-//
-// state.outputSolutions(unsyncBuffer);
-//
-// unsyncBuffer.flush();
-//
-// sink.flush();
-//
-// }
-//
-// } // ControllerTask
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java 2013-12-31 19:48:13 UTC (rev 7711)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -75,8 +75,6 @@
import com.bigdata.relation.accesspath.IBuffer;
import com.bigdata.rwstore.sector.IMemoryManager;
import com.bigdata.rwstore.sector.MemStore;
-import com.bigdata.striterator.Chunkerator;
-import com.bigdata.striterator.Dechunkerator;
import com.bigdata.util.InnerCause;
import cutthecrap.utils.striterators.Expander;
@@ -178,7 +176,12 @@
// Works Ok.
h = 31 * h + c.hashCode();
-// // Martyn's version. Also works Ok.
+ /*
+ * TODO Martyn's version. Also works Ok. Compare rate of hash
+ * collisions and impact on join performance. Also compare use of
+ * 64-bit hash codes and impact on join performance (there should be
+ * fewer hash collisions).
+ */
// @see http://burtleburtle.net/bob/hash/integer.html
//
// final int hc = c.hashCode();
@@ -210,15 +213,10 @@
*/
private final PipelineOp op;
- /**
- * This basically controls the vectoring of the hash join.
- *
- * TODO parameter from operator annotations. Note that 10k tends to put too
- * much heap pressure on the system if the source chunks happen to be
- * smallish. 1000k or 100 is probably the right value until we improve
- * vectoring of the query engine.
- */
- private final int chunkSize = 1000;//ChunkedWrappedIterator.DEFAULT_CHUNK_SIZE;
+// /**
+// * This basically controls the vectoring of the hash join.
+// */
+// private final int chunkSize = 1000;//ChunkedWrappedIterator.DEFAULT_CHUNK_SIZE;
/**
* Utility class for compactly and efficiently encoding and decoding
@@ -305,7 +303,7 @@
* The maximum #of (left,right) solution joins that will be considered
* before failing the join. This is used IFF there are no join variables.
*
- * FIXME Annotation and query hint for this. Probably on
+ * TODO HINTS: Annotation and query hint for this. Probably on
* {@link HashJoinAnnotations}.
*/
private final long noJoinVarsLimit = HashJoinAnnotations.DEFAULT_NO_JOIN_VARS_LIMIT;
@@ -357,8 +355,8 @@
sb.append(getClass().getSimpleName());
sb.append("{open=" + open);
- sb.append(",joinType="+joinType);
- sb.append(",chunkSize=" + chunkSize);
+ sb.append(",joinType=" + joinType);
+// sb.append(",chunkSize=" + chunkSize);
// sb.append(",optional=" + optional);
// sb.append(",filter=" + filter);
if (askVar != null)
@@ -707,30 +705,8 @@
final IKeyBuilder keyBuilder = htree.getIndexMetadata()
.getKeyBuilder();
- /*
- * Rechunk in order to have a nice fat vector size for ordered
- * inserts.
- *
- * TODO This should probably be eliminated in favor of the existing
- * chunk size. That allows us to control the vectoring directly from
- * the pipeline annotations for the query engine. If 1000 (the
- * current [chunkSize] hard wired into this class) makes a
- * significant difference over 100 (the current default pipeline
- * chunk capacity) then we should simply override the default chunk
- * capacity for the htree hash join operators (i.e., analytic
- * operators always imply a larger default chunk capacity, as could
- * operators running on a cluster). This change should be verified
- * against the GOVTRACK dataset and also by using BSBM with JVM and
- * HTree hash joins and measuring the change in the performance
- * delta when the HTree hash join vector size is changed.
- *
- * @see https://sourceforge.net/apps/trac/bigdata/ticket/483
- * (Eliminate unnecessary dechunking and rechunking)
- */
-
- final ICloseableIterator<IBindingSet[]> it = new Chunkerator<IBindingSet>(
- new Dechunkerator<IBindingSet>(itr), chunkSize,
- IBindingSet.class);
+ // Note: We no longer re-chunk here.
+ final ICloseableIterator<IBindingSet[]> it = itr;
try {
@@ -805,16 +781,13 @@
final IKeyBuilder keyBuilder = htree.getIndexMetadata().getKeyBuilder();
- /*
- * Rechunk in order to have a nice fat vector size for ordered inserts.
- */
- final Iterator<IBindingSet[]> it = new Chunkerator<IBindingSet>(
- new Dechunkerator<IBindingSet>(itr), chunkSize,
- IBindingSet.class);
+ // Note: We no longer rechunk here.
+ final Iterator<IBindingSet[]> it = itr;
final AtomicInteger vectorSize = new AtomicInteger();
while (it.hasNext()) {
+ // Vector a chunk of solutions.
final BS[] a = vector(it.next(), joinVars, selectVars,
true/* ignoreUnboundVariables */, vectorSize);
@@ -943,6 +916,7 @@
return 0;
}
+ @Override
public String toString() {
return getClass().getName() + "{hashCode=" + hashCode + ",bset="
+ bset + "}";
@@ -974,6 +948,7 @@
return 0;
}
+ @Override
public String toString() {
return getClass().getName() + "{hashCode=" + hashCode + ",value="
+ BytesUtil.toString(value) + "}";
@@ -983,21 +958,26 @@
@Override
public void hashJoin(//
- final ICloseableIterator<IBindingSet> leftItr,//
+ final ICloseableIterator<IBindingSet[]> leftItr,//
+ final BOpStats stats,
final IBuffer<IBindingSet> outputBuffer//
) {
- hashJoin2(leftItr, outputBuffer, constraints);
+ hashJoin2(leftItr, stats, outputBuffer, constraints);
}
@Override
public void hashJoin2(//
- final ICloseableIterator<IBindingSet> leftItr,//
+ final ICloseableIterator<IBindingSet[]> leftItr,//
+ final BOpStats stats,//
final IBuffer<IBindingSet> outputBuffer,//
final IConstraint[] constraints//
) {
+ // Note: We no longer rechunk in this method.
+ final Iterator<IBindingSet[]> it = leftItr;
+
try {
final HTree rightSolutions = this.getRightSolutions();
@@ -1012,22 +992,37 @@
final IKeyBuilder keyBuilder = rightSolutions.getIndexMetadata()
.getKeyBuilder();
- final Iterator<IBindingSet[]> it = new Chunkerator<IBindingSet>(
- leftItr, chunkSize, IBindingSet.class);
-
// true iff there are no join variables.
final boolean noJoinVars = joinVars.length == 0;
final AtomicInteger vectorSize = new AtomicInteger();
while (it.hasNext()) {
- final BS[] a = vector(it.next(), joinVars,
- null/* selectVars */,
- false/* ignoreUnboundVariables */, vectorSize);
-
- final int n = vectorSize.get();
+ final BS[] a; // vectored solutions.
+ final int n; // #of valid elements in a[].
+ {
+ // Next chunk of solutions from left.
+ final IBindingSet[] b = it.next();
+ if (stats != null) {
+ stats.chunksIn.increment();
+ stats.unitsIn.add(b.length);
+ }
- nleftConsidered.add(n);
+ // Vector a chunk of solutions, ordering by hashCode.
+ a = vector(b, joinVars, null/* selectVars */,
+ false/* ignoreUnboundVariables */, vectorSize);
+
+ // The size of that vector.
+ n = vectorSize.get();
+
+ nleftConsidered.add(n);
+
+ if (log.isTraceEnabled())
+ log.trace("Vectoring chunk for HTree locality: " + n
+ + " out of " + a.length
+ + " solutions are preserved.");
+
+ }
int fromIndex = 0;
@@ -1056,7 +1051,8 @@
if (log.isTraceEnabled())
log.trace("hashCode=" + hashCode + ": #left="
- + bucketSize + ", firstLeft=" + a[fromIndex]);
+ + bucketSize + ", vectorSize=" + n
+ + ", firstLeft=" + a[fromIndex]);
/*
* Note: all source solutions in [fromIndex:toIndex) have
@@ -1080,7 +1076,8 @@
int nrejected = 0;
{
- final byte[] key = keyBuilder.reset().append(hashCode).getKey();
+ final byte[] key = keyBuilder.reset().append(hashCode)
+ .getKey();
// vi...
[truncated message content] |
|
From: <tho...@us...> - 2014-01-04 17:43:56
|
Revision: 7715
http://bigdata.svn.sourceforge.net/bigdata/?rev=7715&view=rev
Author: thompsonbry
Date: 2014-01-04 17:43:49 +0000 (Sat, 04 Jan 2014)
Log Message:
-----------
@Override annotations.
Javadoc on AtOnceHint (it incorrectly stated that the source chunks would be merged before the operator was executed).
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/LocalChunkMessage.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/EmptyChunkMessage.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/NIOChunkMessage.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/ThickChunkMessage.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/relation/accesspath/MultiSourceSequentialCloseableIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/fed/TestNIOChunkMessage.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AtOnceHint.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/LocalChunkMessage.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/LocalChunkMessage.java 2014-01-04 17:15:04 UTC (rev 7714)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/LocalChunkMessage.java 2014-01-04 17:43:49 UTC (rev 7715)
@@ -55,34 +55,42 @@
*/
private IBindingSet[][] bindingSetChunks;
+ @Override
public IQueryClient getQueryController() {
return queryController;
}
+ @Override
public UUID getQueryControllerId() {
return queryControllerId;
}
+ @Override
public UUID getQueryId() {
return queryId;
}
+ @Override
public int getBOpId() {
return bopId;
}
+ @Override
public int getPartitionId() {
return partitionId;
}
+ @Override
public boolean isLastInvocation() {
return false; // Never.
}
+ @Override
public boolean isMaterialized() {
return true;
}
+ @Override
public int getSolutionCount() {
return solutionCount;
}
@@ -153,6 +161,7 @@
}
+ @Override
public String toString() {
return getClass().getName() + "{queryId=" + queryId + ",bopId=" + bopId
@@ -161,10 +170,12 @@
}
+ @Override
public void materialize(FederatedRunningQuery runningQuery) {
// NOP
}
+ @Override
public void release() {
final ChunkAccessor tmp = chunkAccessor;
if (tmp != null) {
@@ -177,6 +188,7 @@
}
}
+ @Override
public IChunkAccessor<IBindingSet> getChunkAccessor() {
if (chunkAccessor == null) {
chunkAccessor = new ChunkAccessor();
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/EmptyChunkMessage.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/EmptyChunkMessage.java 2014-01-04 17:15:04 UTC (rev 7714)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/EmptyChunkMessage.java 2014-01-04 17:43:49 UTC (rev 7715)
@@ -65,34 +65,42 @@
final private boolean lastInvocation;
+ @Override
public IQueryClient getQueryController() {
return queryController;
}
+ @Override
public UUID getQueryControllerId() {
return queryControllerId;
}
+ @Override
public UUID getQueryId() {
return queryId;
}
+ @Override
public int getBOpId() {
return bopId;
}
+ @Override
public int getPartitionId() {
return partitionId;
}
+ @Override
public boolean isLastInvocation() {
return true; // Always.
}
+ @Override
public boolean isMaterialized() {
return true;
}
+ @Override
public int getSolutionCount() {
return 0;
}
@@ -101,6 +109,7 @@
return 0;
}
+ @Override
public String toString() {
return getClass().getName() + "{queryId=" + queryId + ",bopId=" + bopId
@@ -143,14 +152,17 @@
}
+ @Override
public void materialize(FederatedRunningQuery runningQuery) {
// NOP
}
+ @Override
public void release() {
// NOP
}
+ @Override
public IChunkAccessor<E> getChunkAccessor() {
return new IChunkAccessor<E>() {
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/NIOChunkMessage.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/NIOChunkMessage.java 2014-01-04 17:15:04 UTC (rev 7714)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/NIOChunkMessage.java 2014-01-04 17:43:49 UTC (rev 7715)
@@ -103,26 +103,32 @@
*/
final private InetSocketAddress addr;
+ @Override
public IQueryClient getQueryController() {
return queryController;
}
+ @Override
public UUID getQueryControllerId() {
return queryControllerId;
}
+ @Override
public UUID getQueryId() {
return queryId;
}
+ @Override
public int getBOpId() {
return bopId;
}
+ @Override
public int getPartitionId() {
return partitionId;
}
+ @Override
public boolean isLastInvocation() {
return false; // Never.
}
@@ -132,6 +138,7 @@
*
* @todo we could track this in total and in {@link A} on a per-slice basis.
*/
+ @Override
public int getSolutionCount() {
return solutionCount;
}
@@ -149,6 +156,7 @@
return addr;
}
+ @Override
public String toString() {
return getClass().getName() + "{queryId=" + queryId + ",bopId=" + bopId
@@ -327,6 +335,7 @@
}
+ @Override
public boolean isMaterialized() {
return materialized != null;
@@ -335,6 +344,7 @@
private volatile List<IAllocation> materialized = null;
+ @Override
public void materialize(final FederatedRunningQuery runningQuery) {
final AllocationContextKey key = new ShardContext(queryId, bopId,
@@ -353,6 +363,7 @@
/**
* Discard the materialized data.
*/
+ @Override
public void release() {
if (chunkAccessor != null) {
@@ -441,6 +452,7 @@
}
+ @Override
public IChunkAccessor<E> getChunkAccessor() {
if (chunkAccessor == null) {
@@ -511,6 +523,7 @@
}
+ @Override
public void close() {
if(open) {
@@ -523,6 +536,7 @@
}
+ @Override
public boolean hasNext() {
if(open && src.hasNext())
@@ -534,6 +548,7 @@
}
+ @Override
@SuppressWarnings("unchecked")
public E[] next() {
@@ -556,6 +571,7 @@
}
+ @Override
public void remove() {
throw new UnsupportedOperationException();
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/ThickChunkMessage.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/ThickChunkMessage.java 2014-01-04 17:15:04 UTC (rev 7714)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/ThickChunkMessage.java 2014-01-04 17:43:49 UTC (rev 7715)
@@ -84,34 +84,42 @@
private byte[] data;
+ @Override
public IQueryClient getQueryController() {
return queryController;
}
+ @Override
public UUID getQueryControllerId() {
return queryControllerId;
}
+ @Override
public UUID getQueryId() {
return queryId;
}
+ @Override
public int getBOpId() {
return bopId;
}
+ @Override
public int getPartitionId() {
return partitionId;
}
+ @Override
public boolean isLastInvocation() {
return false; // Never.
}
+ @Override
public boolean isMaterialized() {
return true;
}
+ @Override
public int getSolutionCount() {
return solutionCount;
}
@@ -120,6 +128,7 @@
return data.length;
}
+ @Override
public String toString() {
return getClass().getName() + "{queryId=" + queryId + ",bopId=" + bopId
@@ -197,10 +206,12 @@
}
+ @Override
public void materialize(final FederatedRunningQuery runningQuery) {
// NOP
}
+ @Override
public void release() {
if (chunkAccessor != null)
chunkAccessor.close();
@@ -208,6 +219,7 @@
private transient volatile ChunkAccessor chunkAccessor = null;
+ @Override
public IChunkAccessor<E> getChunkAccessor() {
return new ChunkAccessor();
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/relation/accesspath/MultiSourceSequentialCloseableIterator.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/relation/accesspath/MultiSourceSequentialCloseableIterator.java 2014-01-04 17:15:04 UTC (rev 7714)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/relation/accesspath/MultiSourceSequentialCloseableIterator.java 2014-01-04 17:43:49 UTC (rev 7715)
@@ -71,6 +71,7 @@
current = src;
}
+ @Override
public void close() {
lock.lock();
try {
@@ -98,6 +99,7 @@
}
}
+ @Override
public boolean add(final ICloseableIterator<E> src) {
if (src == null)
throw new IllegalArgumentException();
@@ -164,6 +166,7 @@
}
}
+ @Override
public boolean hasNext() {
while (true) {
final ICloseableIterator<E> tmp = nextSource();
@@ -182,6 +185,7 @@
* {@link #next()} to throw {@link NoSuchElementException} if the
* iterator has been concurrently closed.
*/
+ @Override
public E next() {
while (true) {
final ICloseableIterator<E> tmp = nextSource();
@@ -192,6 +196,7 @@
}
}
+ @Override
public void remove() {
throw new UnsupportedOperationException();
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/fed/TestNIOChunkMessage.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/fed/TestNIOChunkMessage.java 2014-01-04 17:15:04 UTC (rev 7714)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/fed/TestNIOChunkMessage.java 2014-01-04 17:43:49 UTC (rev 7715)
@@ -298,6 +298,7 @@
/**
* Overridden to expose to the unit test.
*/
+ @Override
protected void materialize(
final ManagedResourceService resourceService,
final IAllocationContext allocationContext) {
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AtOnceHint.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AtOnceHint.java 2014-01-04 17:15:04 UTC (rev 7714)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AtOnceHint.java 2014-01-04 17:43:49 UTC (rev 7715)
@@ -28,6 +28,7 @@
package com.bigdata.rdf.sparql.ast.hints;
import com.bigdata.bop.PipelineOp;
+import com.bigdata.bop.engine.QueryEngine;
import com.bigdata.rdf.sparql.ast.ASTBase;
import com.bigdata.rdf.sparql.ast.IQueryNode;
import com.bigdata.rdf.sparql.ast.QueryHints;
@@ -35,9 +36,11 @@
/**
* Query hint marks the operator as requiring "at-once" evaluation. All
- * solutions will be buffered by the query engine before the operator is
+ * solutions will be buffered by the {@link QueryEngine} before the operator is
* evaluated. When it is evaluated, it will receive all solutions in a single
- * "chunk".
+ * invocation of that operator. However, the solutions MAY appear in multiple
+ * chunks since the {@link QueryEngine} does not guarantee that the chunk will
+ * be merged before the operator is invoked.
* <p>
* Note: The "at-once" hint is basically turned into <code>NOT(PIPELINED)</code>.
*
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-04 21:46:08
|
Revision: 7724
http://bigdata.svn.sourceforge.net/bigdata/?rev=7724&view=rev
Author: thompsonbry
Date: 2014-01-04 21:46:02 +0000 (Sat, 04 Jan 2014)
Log Message:
-----------
@Override annotations.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/NamedSolutionSetStats.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/bop/rdf/update/ParserStats.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/NamedSolutionSetStats.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/NamedSolutionSetStats.java 2014-01-04 21:07:51 UTC (rev 7723)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/NamedSolutionSetStats.java 2014-01-04 21:46:02 UTC (rev 7724)
@@ -42,6 +42,7 @@
public final CAT solutionSetSize = new CAT();
+ @Override
public void add(final BOpStats o) {
super.add(o);
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java 2014-01-04 21:07:51 UTC (rev 7723)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java 2014-01-04 21:46:02 UTC (rev 7724)
@@ -241,6 +241,7 @@
}
+ @Override
public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
return new FutureTask<Void>(new SliceTask(this, context));
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/bop/rdf/update/ParserStats.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/bop/rdf/update/ParserStats.java 2014-01-04 21:07:51 UTC (rev 7723)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/bop/rdf/update/ParserStats.java 2014-01-04 21:46:02 UTC (rev 7724)
@@ -43,6 +43,7 @@
public final CAT toldTriples = new CAT();
+ @Override
public void add(final BOpStats o) {
super.add(o);
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-06 14:07:47
|
Revision: 7732
http://bigdata.svn.sourceforge.net/bigdata/?rev=7732&view=rev
Author: thompsonbry
Date: 2014-01-06 14:07:38 +0000 (Mon, 06 Jan 2014)
Log Message:
-----------
Refactored AST2BOpUtility, AST2BOpFilters, and AST2BOpJoins to pass down the query hints as a Properties object from the dominating AST node. This causes query hints to be propagated to conditional routing operators, chunked materialization operators, etc. It also prepares the code for reuse by the RTO.
I relabeled the "addMaterialization()" methods as 1, 2, and 3. This makes it significantly easier to identify the recursion patterns in the call hierarchy.
I reorganized the method signatures to be more consistent in terms of where the AST2BOpContext and query hints appear in the list of arguments.
Change to QueryLog to set pred==null for the summary line to get rid of unwanted details.
The AST2BOpContext.queryHints field is now correctly '''ignored''' by AST2BOpBase.applyQueryHints(). The semantics of the AST2BOpContext.queryHints have already been applied to the AST nodes by the ASTQueryHintOptimizer. They do not need to be reapplied in applyQueryHints().
The test suites for the AST and SPARQL are green.
I have confirmed that the atOnce and chunkSize query hints are still be propagated correctly.
TODO: The only remaining issue that I am aware of for query hints is that there are a number of places where we lift out named subqueries. These all need to be reviewed and the query hints correctly brought across for both the NamedSubqueryRoot? and the NamedSubqueryInclude?. Failure to address this prevents hints such as hint:atOnce from correctly being applied to all operators in the query plan.
@see #791 (Clean up query hints)
@see #64 (RTO)
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpBase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpFilters.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpJoins.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java 2014-01-06 12:52:53 UTC (rev 7731)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java 2014-01-06 14:07:38 UTC (rev 7732)
@@ -1108,7 +1108,7 @@
}
@SuppressWarnings("rawtypes")
- final IPredicate pred = (IPredicate<?>) bop
+ final IPredicate pred = summary ? null : (IPredicate<?>) bop
.getProperty(PipelineJoin.Annotations.PREDICATE);
final Integer predId = pred == null ? null : (Integer) pred
.getProperty(BOp.Annotations.BOP_ID);
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpBase.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpBase.java 2014-01-06 12:52:53 UTC (rev 7731)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpBase.java 2014-01-06 14:07:38 UTC (rev 7732)
@@ -41,7 +41,6 @@
import com.bigdata.bop.cost.SubqueryCostReport;
import com.bigdata.rdf.sparql.ast.ASTBase;
import com.bigdata.rdf.sparql.ast.StatementPatternNode;
-import com.bigdata.rdf.sparql.ast.optimizers.ASTQueryHintOptimizer;
/**
* Base class provides support for triples, sids, and quads mode joins which
@@ -164,36 +163,37 @@
}
- /**
- * Apply any query hints to the operator as annotations of that operator.
- * <p>
- * Note: This method is responsible for transferring query hints from
- * {@link ASTBase#getQueryHints()} onto a generated {@link PipelineOp}.
- *
- * @param op
- * The operator.
- * @param queryHints
- * The query hints (from {@link ASTBase#getQueryHints()}).
- *
- * @return A copy of that operator to which the query hints (if any) have
- * been applied. If there are no query hints then the original
- * operator is returned.
- *
- * @deprecated by
- * {@link #applyQueryHints(PipelineOp, ASTBase, AST2BOpContext)}
- * which allows by global and AST node specific query hints to
- * be applied.
- *
- * @see <a href="http://sourceforge.net/apps/trac/bigdata/ticket/791" >
- * Clean up query hints </a>
- */
- protected static PipelineOp applyQueryHints(PipelineOp op,
- final Properties queryHints) {
+// /**
+// * Apply any query hints to the operator as annotations of that operator.
+// * <p>
+// * Note: This method is responsible for transferring query hints from
+// * {@link ASTBase#getQueryHints()} onto a generated {@link PipelineOp}.
+// *
+// * @param op
+// * The operator.
+// * @param queryHints
+// * The query hints (from {@link ASTBase#getQueryHints()}).
+// *
+// * @return A copy of that operator to which the query hints (if any) have
+// * been applied. If there are no query hints then the original
+// * operator is returned.
+// *
+// * @deprecated by
+// * {@link #applyQueryHints(PipelineOp, ASTBase, AST2BOpContext)}
+// * which allows by global and AST node specific query hints to
+// * be applied.
+// *
+// * @see <a href="http://sourceforge.net/apps/trac/bigdata/ticket/791" >
+// * Clean up query hints </a>
+// */
+// @Deprecated
+// protected static PipelineOp applyQueryHints(PipelineOp op,
+// final Properties queryHints) {
+//
+// return _applyQueryHints(op, queryHints);
+//
+// }
- return _applyQueryHints(op, queryHints);
-
- }
-
/**
* Apply any query hints to the operator as annotations of that operator.
* <p>
@@ -203,11 +203,12 @@
* @param op
* The pipeline operator generated from some AST node.
* @param node
- * The AST node from which the pipeline operator was generated.
- * The query hints (from {@link ASTBase#getQueryHints()}) will be
- * applied to that pipeline operator.
+ * The AST node from which the pipeline operator was generated
+ * (required). The query hints (from
+ * {@link ASTBase#getQueryHints()}) will be applied to that
+ * pipeline operator.
* @param ctx
- * The evaluation context (required). Global query hints declared
+ * The evaluation context (ignored). Global query hints declared
* here will be applied to the generated pipeline operator.
* Global hints are applied <strong>first</strong> so they can be
* override by AST node specific hints.
@@ -218,24 +219,22 @@
*
* @see <a href="http://sourceforge.net/apps/trac/bigdata/ticket/791" >
* Clean up query hints </a>
- *
- * FIXME HINTS: {@link AST2BOpContext#queryHints} is applied by
- * {@link ASTQueryHintOptimizer} to annotate the AST. Therefore its
- * impact is <strong>already</strong> accounted for in the
- * <code>node</code>'s query hints. It really DOES NOT need to be
- * passed in here. What is important are the query hints as applied to
- * the AST nodes, not the global defaults.
*/
protected static PipelineOp applyQueryHints(PipelineOp op,
final ASTBase node, final AST2BOpContext ctx) {
- // Apply global query hints from ASTContext.
- op = _applyQueryHints(op, ctx.queryHints);
+ /*
+ * Note: The global query hints are transferred onto the AST nodes by
+ * the ASTQueryHintOptimizer and the registered IQueryHint classes. They
+ * do NOT need to be reapplied here.
+ */
+// // Apply global query hints from ASTContext.
+// op = _applyQueryHints(op, ctx.queryHints);
- if (node != null) {
+// if (node != null) {
// Apply ASTBase node specific query hints.
op = _applyQueryHints(op, node.getQueryHints());
- }
+// }
return op;
@@ -256,9 +255,10 @@
* The pipeline operator generated from some AST node (required).
* @param nodeQueryHints
* The query hints for the AST node from which the pipeline
- * operator was generated (optional).
+ * operator was generated or its dominating operator context
+ * since not all operators have query hints applied (required).
* @param ctx
- * The evaluation context (required). Global query hints declared
+ * The evaluation context (ignored). Global query hints declared
* here will be applied to the generated pipeline operator.
* Global hints are applied <strong>first</strong> so they can be
* override by AST node specific hints.
@@ -273,13 +273,18 @@
protected static PipelineOp applyQueryHints(PipelineOp op,
final Properties nodeQueryHints, final AST2BOpContext ctx) {
- // Apply global query hints from ASTContext.
- op = _applyQueryHints(op, ctx.queryHints);
+ /*
+ * Note: The global query hints are transferred onto the AST nodes by
+ * the ASTQueryHintOptimizer and the registered IQueryHint classes. They
+ * do NOT need to be reapplied here.
+ */
+// // Apply global query hints from ASTContext.
+// op = _applyQueryHints(op, ctx.queryHints);
- if (nodeQueryHints != null) {
+// if (nodeQueryHints != null) {
// Apply ASTBase node specific query hints.
op = _applyQueryHints(op, nodeQueryHints);
- }
+// }
return op;
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpFilters.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpFilters.java 2014-01-06 12:52:53 UTC (rev 7731)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpFilters.java 2014-01-06 14:07:38 UTC (rev 7732)
@@ -70,7 +70,8 @@
import com.bigdata.rdf.sparql.ast.StaticAnalysis;
/**
- * Class handles the materialization pattern for filters.
+ * Class handles the materialization pattern for filters by adding a series of
+ * materialization steps to materialize terms needed downstream.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
* @version $Id$
@@ -85,86 +86,130 @@
protected AST2BOpFilters() {
}
+ /*
+ * Note: There was only one caller left for this method and it was from
+ * within this same class, so I rewrote the calling context very slightly
+ * and removed this method. The documentation on the general approach has
+ * been moved to the delegate method (immediate below). bbt.
+ */
+// /**
+// * Adds a series of materialization steps to materialize terms needed
+// * downstream.
+// *
+// * To materialize the variable ?term, the pipeline looks as follows:
+// *
+// * <pre>
+// * left
+// * ->
+// * ConditionalRoutingOp1 (condition=!IsMaterialized(?term), alt=right)
+// * ->
+// * ConditionalRoutingOp2 (condition=IsInline(?term), alt=PipelineJoin)
+// * ->
+// * InlineMaterializeOp (predicate=LexPredicate(?term), sink=right)
+// * ->
+// * PipelineJoin (predicate=LexPredicate(?term))
+// * ->
+// * right
+// * </pre>
+// *
+// * @param context
+// * @param left
+// * the left (upstream) operator that immediately proceeds the
+// * materialization steps
+// * @param right
+// * the right (downstream) operator that immediately follows the
+// * materialization steps
+// * @param c
+// * the constraint to run on the IsMaterialized op to see if the
+// * materialization pipeline can be bypassed (bypass if true and
+// * no {@link NotMaterializedException} is thrown).
+// * @param varsToMaterialize
+// * the terms to materialize
+// * @param queryHints
+// * the query hints
+// * @return the final bop added to the pipeline by this method
+// *
+// * @see AST2BOpUtility#addMaterializationSteps(PipelineOp, int,
+// * IValueExpression, Collection, AST2BOpContext)
+// */
+// @SuppressWarnings({ "rawtypes", "unchecked" })
+// protected static PipelineOp addMaterializationSteps(
+// final AST2BOpContext context,//
+// PipelineOp left, //
+// final int right,//
+// final IConstraint c,//
+// final Collection<IVariable<IV>> varsToMaterialize,//
+//// final AtomicInteger idFactory,
+// final Properties queryHints) {
+//
+//// final AST2BOpContext context = new AST2BOpContext(
+//// null/* astContainer */, idFactory, db, queryEngine, queryHints);
+//
+// final IValueExpression<IV> ve = (IValueExpression) c.get(0);
+//
+// return addMaterializationSteps(left, right, ve, varsToMaterialize,
+// context);
+//
+// }
+
/**
- * Adds a series of materialization steps to materialize terms needed
- * downstream.
+ * If the value expression that needs the materialized variables can run
+ * without a {@link NotMaterializedException} then just route to the
+ * <i>rightId</i> (around the rest of the materialization pipeline steps).
+ * This happens in the case of a value expression that only "sometimes"
+ * needs materialized values, but not always (i.e. materialization
+ * requirement depends on the data flowing through). A good example of this
+ * is {@link CompareBOp}, which can sometimes work on internal values and
+ * sometimes can't.
*
- * To materialize the variable ?term, the pipeline looks as follows:
+ * To materialize the variable <code>?term</code>, the pipeline looks as
+ * follows:
*
* <pre>
* left
* ->
- * ConditionalRoutingOp1 (condition=!IsMaterialized(?term), alt=right)
+ * ConditionalRoutingOp1 (condition=!IsMaterialized(?term), alt=rightId)
* ->
* ConditionalRoutingOp2 (condition=IsInline(?term), alt=PipelineJoin)
* ->
- * InlineMaterializeOp (predicate=LexPredicate(?term), sink=right)
+ * InlineMaterializeOp (predicate=LexPredicate(?term), sink=rightId)
* ->
* PipelineJoin (predicate=LexPredicate(?term))
* ->
- * right
+ * rightId
* </pre>
*
- * @param db
- * the database
- * @param queryEngine
- * the query engine
* @param left
- * the left (upstream) operator that immediately proceeds the
- * materialization steps
- * @param right
- * the right (downstream) operator that immediately follows the
- * materialization steps
- * @param c
- * the constraint to run on the IsMaterialized op to see if the
- * materialization pipeline can be bypassed (bypass if true and
- * no {@link NotMaterializedException} is thrown).
+ * The left (upstream) operator that immediately proceeds the
+ * materialization steps.
+ * @param rightId
+ * The id for the right (downstream) operator that immediately
+ * follows the materialization steps. This needs to be
+ * pre-reserved by the caller.
+ * @param ve
+ * The {@link IValueExpression} for the {@link SPARQLConstraint}.
* @param varsToMaterialize
- * the terms to materialize
+ * The variables to be materialize.
* @param queryHints
- * the query hints
- * @return the final bop added to the pipeline by this method
+ * The query hints to be applied from the dominating operator
+ * context.
+ * @param ctx
+ * The evaluation context.
*
- * @see AST2BOpUtility#addMaterializationSteps(PipelineOp, int,
- * IValueExpression, Collection, AST2BOpContext)
- */
- @SuppressWarnings({ "rawtypes", "unchecked" })
- public static PipelineOp addMaterializationSteps(
- final AST2BOpContext context,
-// final AbstractTripleStore db,
-// final QueryEngine queryEngine,
- PipelineOp left, final int right,
- final IConstraint c,
- final Collection<IVariable<IV>> varsToMaterialize,
-// final AtomicInteger idFactory,
- final Properties queryHints) {
-
-// final AST2BOpContext context = new AST2BOpContext(
-// null/* astContainer */, idFactory, db, queryEngine, queryHints);
-
- final IValueExpression<IV> ve = (IValueExpression) c.get(0);
-
- return addMaterializationSteps(left, right, ve, varsToMaterialize,
- context);
-
- }
-
- /**
- * If the value expression that needs the materialized variables can run
- * without a {@link NotMaterializedException} then just route to the
- * <i>rightId</i> (around the rest of the materialization pipeline steps).
- * This happens in the case of a value expression that only "sometimes"
- * needs materialized values, but not always (i.e. materialization
- * requirement depends on the data flowing through). A good example of this
- * is {@link CompareBOp}, which can sometimes work on internal values and
- * sometimes can't.
+ * @return The final bop added to the pipeline by this method
*
* @see TryBeforeMaterializationConstraint
+ * @see AST2BOpUtility#addMaterializationSteps1(PipelineOp, int,
+ * IValueExpression, Collection, AST2BOpContext)
*/
@SuppressWarnings("rawtypes")
- protected static PipelineOp addMaterializationSteps(PipelineOp left,
- final int rightId, final IValueExpression<IV> ve,
- final Collection<IVariable<IV>> vars, final AST2BOpContext ctx) {
+ protected static PipelineOp addMaterializationSteps1(//
+ PipelineOp left,//
+ final int rightId, //
+ final IValueExpression<IV> ve,//
+ final Collection<IVariable<IV>> vars, //
+ final Properties queryHints,
+ final AST2BOpContext ctx) {
/*
* If the constraint "c" can run without a NotMaterializedException then
@@ -177,9 +222,9 @@
new NV(BOp.Annotations.BOP_ID, ctx.nextId()),//
new NV(ConditionalRoutingOp.Annotations.CONDITION, c2),//
new NV(PipelineOp.Annotations.ALT_SINK_REF, rightId)//
- ), ctx.queryHints);
+ ), queryHints, ctx);
- return addMaterializationSteps(left, rightId, vars, ctx);
+ return addMaterializationSteps2(left, rightId, vars, queryHints, ctx);
}
@@ -200,35 +245,42 @@
* <pre>
* left
* ->
- * ConditionalRoutingOp1 (condition=!IsMaterialized(?term), alt=right)
+ * ConditionalRoutingOp1 (condition=!IsMaterialized(?term), alt=rightId)
* ->
* ConditionalRoutingOp2 (condition=IsInline(?term), alt=PipelineJoin)
* ->
- * InlineMaterializeOp (predicate=LexPredicate(?term), sink=right)
+ * InlineMaterializeOp (predicate=LexPredicate(?term), sink=rightId)
* ->
* PipelineJoin (predicate=LexPredicate(?term))
* ->
- * right
+ * rightId
* </pre>
*
* @param left
- * the left (upstream) operator that immediately proceeds the
- * materialization steps
+ * The left (upstream) operator that immediately proceeds the
+ * materialization steps.
* @param rightId
- * the id of the right (downstream) operator that immediately
- * follows the materialization steps
+ * The id of the right (downstream) operator that immediately
+ * follows the materialization steps.
* @param vars
- * the terms to materialize
+ * The terms to materialize
+ * @param queryHints
+ * The query hints from the dominating AST node.
+ * @param ctx
+ * The evaluation context.
*
- * @return the final bop added to the pipeline by this method
+ * @return The final bop added to the pipeline by this method
*
* @see TryBeforeMaterializationConstraint
*
- * TODO make [vars] a Set.
+ * TODO make [vars] a Set.
*/
@SuppressWarnings("rawtypes")
- protected static PipelineOp addMaterializationSteps(PipelineOp left,
- final int rightId, final Collection<IVariable<IV>> vars,
+ protected static PipelineOp addMaterializationSteps2(//
+ PipelineOp left,//
+ final int rightId, //
+ final Collection<IVariable<IV>> vars,//
+ final Properties queryHints, //
final AST2BOpContext ctx) {
final int nvars = vars.size();
@@ -257,8 +309,7 @@
new NV(ChunkedMaterializationOp.Annotations.TIMESTAMP, timestamp), //
new NV(PipelineOp.Annotations.SHARED_STATE, !ctx.isCluster()),// live stats, but not on the cluster.
new NV(BOp.Annotations.BOP_ID, ctx.nextId())//
- ), (Properties) null/*nodeQueryHints*/, ctx // FIXME HINTS: Pass in the caller's AST node query hints.
- );
+ ), queryHints, ctx);
// vars.toArray(new IVariable[nvars]), ns, timestamp)
// .setProperty(BOp.Annotations.BOP_ID, ctx.nextId());
}
@@ -309,7 +360,7 @@
c1),//
new NV(PipelineOp.Annotations.SINK_REF, condId2),//
new NV(PipelineOp.Annotations.ALT_SINK_REF, endId)//
- ), ctx.queryHints);
+ ), queryHints, ctx);
if (log.isDebugEnabled()) {
log.debug("adding 1st conditional routing op: " + condOp1);
@@ -327,7 +378,7 @@
inlineMaterializeId), //
new NV(PipelineOp.Annotations.ALT_SINK_REF,
lexJoinId)//
- ), ctx.queryHints);
+ ), queryHints, ctx);
if (log.isDebugEnabled()) {
log.debug("adding 2nd conditional routing op: " + condOp2);
@@ -361,7 +412,7 @@
new NV(InlineMaterializeOp.Annotations.PREDICATE,
lexPred.clone()),//
new NV(PipelineOp.Annotations.SINK_REF, endId)//
- ), ctx.queryHints);
+ ), queryHints, ctx);
if (log.isDebugEnabled()) {
log.debug("adding inline materialization op: "
@@ -393,7 +444,7 @@
final PipelineOp lexJoinOp = applyQueryHints(//
new PipelineJoin(leftOrEmpty(inlineMaterializeOp), //
anns.toArray(new NV[anns.size()])),
- ctx.queryHints);
+ queryHints, ctx);
// final PipelineOp lexJoinOp = newJoin(inlineMaterializeOp, anns,
// ctx.queryHints);
@@ -444,20 +495,19 @@
* @param needsMaterialization
* A map of constraints and their variable materialization
* requirements.
- * @param context
* @param queryHints
+ * Query hints from the dominating AST node.
+ * @param ctx
+ * The evaluation context.
*/
@SuppressWarnings("rawtypes")
- protected static PipelineOp addMaterializationSteps(
- final AST2BOpContext ctx,
- PipelineOp left,
- final Set<IVariable<?>> doneSet,
+ protected static PipelineOp addMaterializationSteps3(//
+ PipelineOp left,//
+ final Set<IVariable<?>> doneSet,//
final Map<IConstraint, Set<IVariable<IV>>> needsMaterialization,
-// final AbstractTripleStore db,//
-// final QueryEngine queryEngine,//
-// final AtomicInteger idFactory,//
-// final BOpContextBase context,//
- final Properties queryHints) {
+ final Properties queryHints,//
+ final AST2BOpContext ctx//
+ ) {
if (!needsMaterialization.isEmpty()) {
@@ -486,22 +536,35 @@
if (!terms.isEmpty()) {
// Add materialization steps for remaining variables.
- left = addMaterializationSteps(
- ctx,
-// db, queryEngine,
- left,
- condId, c, terms,
- // idFactory,
- queryHints
- );
+ @SuppressWarnings("unchecked")
+ final IValueExpression<IV> ve = (IValueExpression) c.get(0);
+
+ left = addMaterializationSteps1(//
+ left, //
+ condId, // right
+ ve, // value expression
+ terms,// varsToMaterialize,
+ queryHints,//
+ ctx);
+
+// left = addMaterializationSteps(//
+// ctx,//
+// left,//
+// condId,// rightId
+// c, // eval c.get(0)
+// terms, // varsToMaterialize
+// // idFactory,
+// queryHints//
+// );
+
}
left = applyQueryHints(//
new ConditionalRoutingOp(leftOrEmpty(left),//
new NV(BOp.Annotations.BOP_ID, condId),//
new NV(ConditionalRoutingOp.Annotations.CONDITION,c)//
- ), queryHints);
+ ), queryHints, ctx);
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpJoins.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpJoins.java 2014-01-06 12:52:53 UTC (rev 7731)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpJoins.java 2014-01-06 14:07:38 UTC (rev 7732)
@@ -97,7 +97,6 @@
* named-graph and default graph join patterns whether on a single machine
* or on a cluster.
*
- * @param ctx
* @param left
* @param pred
* The predicate describing the statement pattern.
@@ -107,16 +106,16 @@
* Constraints on that join (optional).
* @param queryHints
* Query hints associated with that {@link StatementPatternNode}.
- * @return
+ * @param ctx The evaluation context.
*/
@SuppressWarnings("rawtypes")
public static PipelineOp join(//
- final AST2BOpContext ctx,//
PipelineOp left,//
Predicate pred,//
final Set<IVariable<?>> doneSet,// variables known to be materialized.
final Collection<IConstraint> constraints,//
- final Properties queryHints//
+ final Properties queryHints,//
+ final AST2BOpContext ctx//
) {
final int joinId = ctx.nextId();
@@ -189,8 +188,8 @@
* pipeline.
*/
- left = addMaterializationSteps(ctx, left, doneSet,
- needsMaterialization, queryHints);
+ left = addMaterializationSteps3(left, doneSet, needsMaterialization,
+ queryHints, ctx);
return left;
@@ -233,8 +232,8 @@
anns.add(new NV(PipelineJoin.Annotations.PREDICATE, pred));
- return newJoin(ctx, left, anns, queryHints,
- false/* defaultGraphFilter */, null/* summary */);
+ return newJoin(left, anns, false/* defaultGraphFilter */,
+ null/* summary */, queryHints, ctx);
}
@@ -284,8 +283,8 @@
anns.add(new NV(PipelineJoin.Annotations.PREDICATE,pred));
- return newJoin(ctx, left, anns, queryHints,
- false/* defaultGraphFilter */, null/* summary */);
+ return newJoin(left, anns, false/* defaultGraphFilter */,
+ null/* summary */, queryHints, ctx);
}
@@ -297,8 +296,8 @@
anns.add(new NV(PipelineJoin.Annotations.PREDICATE, pred));
- return newJoin(ctx, left, anns, queryHints,
- false/* defaultGraphFilter */, null/* summary */);
+ return newJoin(left, anns, false/* defaultGraphFilter */,
+ null/* summary */, queryHints, ctx);
}
/*
@@ -325,8 +324,8 @@
anns.add(new NV(PipelineJoin.Annotations.PREDICATE,pred));
- return newJoin(ctx, left, anns, queryHints,
- false/* defaultGraphFilter */, summary);
+ return newJoin(left, anns, false/* defaultGraphFilter */, summary,
+ queryHints, ctx);
}
@@ -349,8 +348,8 @@
anns.add(new NV(PipelineJoin.Annotations.PREDICATE,pred));
- return newJoin(ctx, left, anns, queryHints,
- false/* defaultGraphFilter */, summary);
+ return newJoin(left, anns, false/* defaultGraphFilter */, summary,
+ queryHints, ctx);
}
@@ -425,8 +424,8 @@
anns.add(new NV(PipelineJoin.Annotations.PREDICATE,pred));
- return newJoin(ctx, left, anns, queryHints,
- false/* defaultGraphFilter */, summary);
+ return newJoin(left, anns, false/* defaultGraphFilter */, summary,
+ queryHints, ctx);
} else {
@@ -456,8 +455,8 @@
anns.add(new NV(PipelineJoin.Annotations.PREDICATE,pred));
- return newJoin(ctx, left, anns, queryHints,
- false/* defaultGraphFilter */, summary);
+ return newJoin(left, anns, false/* defaultGraphFilter */, summary,
+ queryHints, ctx);
}
@@ -495,8 +494,9 @@
anns.add(new NV(PipelineJoin.Annotations.PREDICATE, pred));
- return newJoin(ctx, left, anns, queryHints, true/* defaultGraphFilter */,
- summary);
+ return newJoin(left, anns, true/* defaultGraphFilter */, summary,
+ queryHints, ctx);
+
}
if (summary != null && summary.nknown == 0) {
@@ -513,8 +513,8 @@
anns.add(new NV(PipelineJoin.Annotations.PREDICATE,pred));
- return newJoin(ctx, left, anns, queryHints,
- false/* defaultGraphFilter */, summary);
+ return newJoin(left, anns, false/* defaultGraphFilter */, summary,
+ queryHints, ctx);
}
@@ -542,8 +542,8 @@
anns.add(new NV(PipelineJoin.Annotations.PREDICATE, pred));
- return newJoin(ctx, left, anns, queryHints,
- false/* defaultGraphFilter */, summary);
+ return newJoin(left, anns, false/* defaultGraphFilter */, summary,
+ queryHints, ctx);
}
@@ -754,8 +754,8 @@
anns.add(new NV(PipelineJoin.Annotations.PREDICATE, pred));
- return newJoin(ctx, left, anns, queryHints,
- true/* defaultGraphFilter */, summary);
+ return newJoin(left, anns, true/* defaultGraphFilter */, summary,
+ queryHints, ctx);
} else {
@@ -842,9 +842,9 @@
anns.add(new NV(PipelineJoin.Annotations.PREDICATE,pred));
- return newJoin(ctx, left, anns, queryHints,
- true/* defaultGraphFilter */, summary);
-
+ return newJoin(left, anns, true/* defaultGraphFilter */, summary,
+ queryHints, ctx);
+
}
}
@@ -930,7 +930,6 @@
*
* @param left
* @param anns
- * @param queryHints
* @param defaultGraphFilter
* <code>true</code> iff a DISTINCT filter must be imposed on the
* SPOs. This is never done for a named graph query. It is
@@ -939,6 +938,10 @@
* need to bother.
* @param summary
* The {@link DataSetSummary} (when available).
+ * @param queryHints
+ * The query hints from the dominating operator context.
+ * @param ctx
+ * The evaluation context.
* @return
*
* @see Annotations#HASH_JOIN
@@ -946,11 +949,13 @@
* @see Annotations#ESTIMATED_CARDINALITY
*/
@SuppressWarnings({ "rawtypes", "unchecked" })
- static private PipelineOp newJoin(final AST2BOpContext ctx,
- PipelineOp left, final List<NV> anns,
- final Properties queryHints,
- final boolean defaultGraphFilter,
- final DataSetSummary summary) {
+ static private PipelineOp newJoin(//
+ PipelineOp left, //
+ final List<NV> anns,//
+ final boolean defaultGraphFilter,//
+ final DataSetSummary summary,//
+ final Properties queryHints, //
+ final AST2BOpContext ctx) {
final Map<String, Object> map = NV.asMap(anns.toArray(new NV[anns
.size()]));
@@ -1073,7 +1078,7 @@
}
- left = applyQueryHints(left, queryHints);
+ left = applyQueryHints(left, queryHints, ctx);
return left;
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java 2014-01-06 12:52:53 UTC (rev 7731)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java 2014-01-06 14:07:38 UTC (rev 7732)
@@ -103,7 +103,7 @@
if (ctx.isQuads()) {
- // FIXME The RTO does not handle quads yet.
+ // FIXME RTO: The RTO does not handle quads yet.
return left;
}
@@ -173,7 +173,7 @@
// Something the RTO can handle.
sps.add(sp);
/*
- * FIXME Handle Triples vs Quads, Default vs Named Graph, and
+ * FIXME RTO: Handle Triples vs Quads, Default vs Named Graph, and
* DataSet. This probably means pushing more logic down into
* the RTO from AST2BOpJoins.
*/
@@ -231,7 +231,7 @@
}
/*
- * FIXME When running the RTO as anything other than the top-level join
+ * FIXME RTO: When running the RTO as anything other than the top-level join
* group in the query plan and for the *FIRST* joins in the query plan,
* we need to flow in any solutions that are already in the pipeline
* (unless we are going to run the RTO "bottom up") and build a hash
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java 2014-01-06 12:52:53 UTC (rev 7731)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java 2014-01-06 14:07:38 UTC (rev 7732)
@@ -244,7 +244,7 @@
* output.
*/
- left = addStartOp(ctx);
+ left = addStartOp(optimizedQuery, ctx);
}
@@ -363,7 +363,7 @@
if (left == null) {
- left = addStartOpOnCluster(ctx);
+ left = addStartOpOnCluster(queryBase, ctx);
}
@@ -443,8 +443,8 @@
for (AssignmentNode assignmentNode : projection
.getAssignmentProjections()) {
- left = addAssignment(left, assignmentNode, doneSet, ctx,
- true/* projection */);
+ left = addAssignment(left, assignmentNode, doneSet,
+ projection.getQueryHints(), ctx, true/* projection */);
}
@@ -482,7 +482,7 @@
preserveOrder = true;
- left = addOrderBy(left, orderBy, ctx);
+ left = addOrderBy(left, queryBase, orderBy, ctx);
} else {
@@ -504,7 +504,7 @@
if (orderBy != null && !orderBy.isEmpty()) {
- left = addOrderBy(left, orderBy, ctx);
+ left = addOrderBy(left, queryBase, orderBy, ctx);
}
@@ -591,7 +591,7 @@
if(queryBase.hasSlice()) {
- left = addSlice(left, queryBase.getSlice(), ctx);
+ left = addSlice(left, queryBase, queryBase.getSlice(), ctx);
}
@@ -839,6 +839,9 @@
final ServiceNode serviceNode, final Set<IVariable<?>> doneSet,
final AST2BOpContext ctx) {
+ // The query hints are taken from the SERVICE node.
+ final Properties queryHints = serviceNode.getQueryHints();
+
@SuppressWarnings("rawtypes")
final Map<IConstraint, Set<IVariable<IV>>> needsMaterialization = new LinkedHashMap<IConstraint, Set<IVariable<IV>>>();
@@ -994,7 +997,8 @@
if (!vars.isEmpty()) {
// Add the materialization step.
- left = addMaterializationSteps(left, rightId, (Collection) vars, ctx);
+ left = addMaterializationSteps2(left, rightId,
+ (Collection) vars, queryHints, ctx);
// These variables have now been materialized.
doneSet.addAll(vars);
@@ -1046,15 +1050,15 @@
anns.put(JoinAnnotations.CONSTRAINTS, joinConstraints);
left = applyQueryHints(new ServiceCallJoin(leftOrEmpty(left), anns),
- serviceNode.getQueryHints());
+ queryHints, ctx);
/*
* For each filter which requires materialization steps, add the
* materializations steps to the pipeline and then add the filter to the
* pipeline.
*/
- left = addMaterializationSteps(ctx, left, doneSet, needsMaterialization,
- serviceNode.getQueryHints());
+ left = addMaterializationSteps3(left, doneSet, needsMaterialization,
+ queryHints, ctx);
return left;
@@ -1455,8 +1459,8 @@
* materializations steps to the pipeline and then add the filter to the
* pipeline.
*/
- left = addMaterializationSteps(ctx, left, doneSet, needsMaterialization,
- nsi.getQueryHints());
+ left = addMaterializationSteps3(left, doneSet, needsMaterialization,
+ nsi.getQueryHints(), ctx);
return left;
@@ -1575,8 +1579,8 @@
* materializations steps to the pipeline and then add the filter to the
* pipeline.
*/
- left = addMaterializationSteps(ctx, left, doneSet,
- needsMaterialization, nsi.getQueryHints());
+ left = addMaterializationSteps3(left, doneSet, needsMaterialization,
+ nsi.getQueryHints(), ctx);
return left;
@@ -1762,8 +1766,8 @@
* materializations steps to the pipeline and then add the filter to the
* pipeline.
*/
- left = addMaterializationSteps(ctx, left, doneSet, needsMaterialization,
- subqueryRoot.getQueryHints());
+ left = addMaterializationSteps3(left, doneSet, needsMaterialization,
+ subqueryRoot.getQueryHints(), ctx);
return left;
@@ -2018,8 +2022,8 @@
* materializations steps to the pipeline and then add the filter to the
* pipeline.
*/
- left = addMaterializationSteps(ctx, left, doneSet, needsMaterialization,
- subqueryRoot.getQueryHints());
+ left = addMaterializationSteps3(left, doneSet, needsMaterialization,
+ subqueryRoot.getQueryHints(), ctx);
return left;
@@ -2220,9 +2224,10 @@
}
- left = applyQueryHints(new Tee(leftOrEmpty(left),
- NV.asMap(anns.toArray(new NV[anns.size()]))), ctx.queryHints);
-
+ left = applyQueryHints(
+ new Tee(leftOrEmpty(left), NV.asMap(anns
+ .toArray(new NV[anns.size()]))), unionNode, ctx);
+
}
/*
@@ -2286,7 +2291,7 @@
new NV(Predicate.Annotations.BOP_ID, downstreamId),//
new NV(BOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER)//
- ), ctx.queryHints);
+ ), unionNode, ctx);
// Add in anything which was known materialized for all child groups.
doneSet.addAll(doneSetsIntersection);
@@ -2302,7 +2307,7 @@
final ArbitraryLengthPathNode alpNode, final Set<IVariable<?>> doneSet,
final AST2BOpContext ctx) {
- final JoinGroupNode subgroup = (JoinGroupNode) alpNode.subgroup();
+ final JoinGroupNode subgroup = (JoinGroupNode) alpNode.subgroup();
// Convert the child join group.
final PipelineOp subquery = convertJoinGroup(null,
@@ -2324,10 +2329,10 @@
* We need to drop the internal variables bound by the subquery (in the
* case where the subquery is a nested path).
*/
- final Set<IVariable<?>> varsToDrop = new LinkedHashSet<IVariable<?>>();
- ctx.sa.getDefinitelyProducedBindings(subgroup, varsToDrop, true);
- varsToDrop.remove(tVarLeft);
- varsToDrop.remove(tVarRight);
+ final Set<IVariable<?>> varsToDrop = new LinkedHashSet<IVariable<?>>();
+ ctx.sa.getDefinitelyProducedBindings(subgroup, varsToDrop, true);
+ varsToDrop.remove(tVarLeft);
+ varsToDrop.remove(tVarRight);
left = applyQueryHints(new ArbitraryLengthPathOp(leftOrEmpty(left),//
new NV(ArbitraryLengthPathOp.Annotations.SUBQUERY, subquery),
@@ -2342,7 +2347,7 @@
new NV(Predicate.Annotations.BOP_ID, ctx.nextId()),//
new NV(BOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER)//
- ), ctx.queryHints);
+ ), alpNode, ctx);
return left;
@@ -2367,7 +2372,7 @@
new NV(Predicate.Annotations.BOP_ID, ctx.nextId()),//
new NV(BOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER)//
- ), ctx.queryHints);
+ ), zlpNode, ctx);
return left;
@@ -2549,11 +2554,11 @@
*/
final Predicate<?> pred = toPredicate(sp, ctx);
final boolean optional = sp.isOptional();
- left = join(ctx, left, pred,
+ left = join(left, pred,
optional ? new LinkedHashSet<IVariable<?>>(doneSet)
: doneSet,
getJoinConstraints(sp),
- sp.getQueryHints());
+ sp.getQueryHints(), ctx);
continue;
} else if (child instanceof ArbitraryLengthPathNode) {
final ArbitraryLengthPathNode alpNode = (ArbitraryLengthPathNode) child;
@@ -2641,12 +2646,12 @@
continue;
}
// FILTER
- left = addConditional(left, filter, doneSet, ctx);
+ left = addConditional(left, joinGroup, filter, doneSet, ctx);
continue;
} else if (child instanceof AssignmentNode) {
// LET / BIND
left = addAssignment(left, (AssignmentNode) child, doneSet,
- ctx, false/* projection */);
+ joinGroup.getQueryHints(), ctx, false/* projection */);
continue;
} else {
throw new UnsupportedOperationException("child: " + child);
@@ -2656,8 +2661,8 @@
if (!dropVars.isEmpty()) {
final IVariable<?>[] a = dropVars.toArray(new IVariable[dropVars
.size()]);
- left = applyQueryHints(new DropOp(leftOrEmpty(left), new NV(BOp.Annotations.BOP_ID,
- ctx.nextId()), //
+ left = applyQueryHints(new DropOp(leftOrEmpty(left), //
+ new NV(BOp.Annotations.BOP_ID, ctx.nextId()), //
new NV(DropOp.Annotations.DROP_VARS, a)//
), joinGroup, ctx);
}
@@ -3060,17 +3065,21 @@
}
/**
- * Conditionally add a {@link StartOp} iff the query will rin on a cluster.
- *
- * @param ctx
- *
- * @return The {@link StartOp} iff this query will run on a cluster and
- * otherwise <code>null</code>.
- *
- * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/478">
- * Cluster does not map input solution(s) across shards</a>
- */
- private static final PipelineOp addStartOpOnCluster(final AST2BOpContext ctx) {
+ * Conditionally add a {@link StartOp} iff the query will rin on a cluster.
+ *
+ * @param queryBase
+ * The {@link QueryBase} for which a {@link StartOp} might be
+ * required.
+ * @param ctx
+ *
+ * @return The {@link StartOp} iff this query will run on a cluster and
+ * otherwise <code>null</code>.
+ *
+ * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/478">
+ * Cluster does not map input solution(s) across shards</a>
+ */
+ private static final PipelineOp addStartOpOnCluster(
+ final QueryBase queryBase, final AST2BOpContext ctx) {
if (ctx.isCluster()) {
@@ -3086,7 +3095,7 @@
* @see https://sourceforge.net/apps/trac/bigdata/ticket/478
*/
- return addStartOp(ctx);
+ return addStartOp(queryBase, ctx);
}
@@ -3106,7 +3115,8 @@
*
* @return The {@link StartOp}.
*/
- private static final PipelineOp addStartOp(final AST2BOpContext ctx) {
+ private static final PipelineOp addStartOp(final QueryBase queryBase,
+ final AST2BOpContext ctx) {
final PipelineOp start = applyQueryHints(
new StartOp(BOp.NOARGS, NV.asMap(new NV[] {//
@@ -3114,7 +3124,7 @@
.nextId()),
new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER), })),
- ctx.queryHints);
+ queryBase, ctx);
return start;
@@ -3125,15 +3135,23 @@
*
* @param left
* @param assignmentNode
+ * The {@link AssignmentNode} (LET() or BIND()).
* @param doneSet
+ * @param queryHints
+ * The query hints from the AST node that dominates the
+ * assignment and from which we will take any query hints. E.g.,
+ * a PROJECTION or a JOIN GROUP.
* @param ctx
* @param projection
* @return
*/
@SuppressWarnings({ "unchecked", "rawtypes" })
private static final PipelineOp addAssignment(PipelineOp left,
- final AssignmentNode assignmentNode,
- final Set<IVariable<?>> doneSet, final AST2BOpContext ctx,
+// final ASTBase dominatingASTNode,//
+ final AssignmentNode assignmentNode,//
+ final Set<IVariable<?>> doneSet, //
+ final Properties queryHints,//
+ final AST2BOpContext ctx,//
final boolean projection) {
final IValueExpression ve = assignmentNode.getValueExpression();
@@ -3167,7 +3185,8 @@
*/
if (vars.size() > 0) {
- left = addMaterializationSteps(left, bopId, ve, vars, ctx);
+ left = addMaterializationSteps1(left, bopId, ve, vars,
+ queryHints, ctx);
if(req.getRequirement()==Requirement.ALWAYS) {
@@ -3186,9 +3205,9 @@
left = applyQueryHints(//
new ConditionalRoutingOp(leftOrEmpty(left), //
- new NV(BOp.Annotations.BOP_ID, bopId), //
- new NV(ConditionalRoutingOp.Annotations.CONDITION, c)//
- ), ctx.queryHints);
+ new NV(BOp.Annotations.BOP_ID, bopId), //
+ new NV(ConditionalRoutingOp.Annotations.CONDITION, c)//
+ ), queryHints, ctx);
return left;
@@ -3199,6 +3218,8 @@
* pipeline.
*
* @param left
+ * @param joinGroup
+ * The parent join group.
* @param filter
* The filter.
* @param doneSet
@@ -3208,9 +3229,12 @@
* @return
*/
@SuppressWarnings("rawtypes")
- private static final PipelineOp addConditional(PipelineOp left,
- final FilterNode filter,
- final Set<IVariable<?>> doneSet, final AST2BOpContext ctx) {
+ private static final PipelineOp addConditional(//
+ PipelineOp left,//
+ final JoinGroupNode joinGroup,//
+ final FilterNode filter,//
+ final Set<IVariable<?>> doneSet, //
+ final AST2BOpContext ctx) {
@SuppressWarnings("unchecked")
final IValueExpression<IV> ve = (IValueExpression<IV>) filter
@@ -3240,7 +3264,8 @@
if (!vars.isEmpty()) {
// Add materialization steps for those variables.
- left = addMaterializationSteps(left, bopId, ve, vars, ctx);
+ left = addMaterializationSteps1(left, bopId, ve, vars,
+ joinGroup.getQueryHints(), ctx);
if (req.getRequirement() == Requirement.ALWAYS) {
@@ -3261,7 +3286,7 @@
left = applyQueryHints(new ConditionalRoutingOp(leftOrEmpty(left),//
new NV(BOp.Annotations.BOP_ID, bopId), //
new NV(ConditionalRoutingOp.Annotations.CONDITION, c)//
- ), ctx.queryHints);
+ ), joinGroup, ctx);
return left;
@@ -3277,6 +3302,9 @@
* keep it, we could use a {@link JVMSolutionSetHashJoinOp}. Just push
* the data into the hash index and then just that operator to join it
* into the pipeline. Then we could ditch the {@link DataSetJoin}.
+ * <p>
+ * See {@link JoinGroupNode#getInFilters()} for how and where this is
+ * currently disabled.
*/
@SuppressWarnings("rawtypes")
private static final PipelineOp addKnownInConditional(PipelineOp left,
@@ -3518,8 +3546,8 @@
* materializations steps to the pipeline and then add the filter to the
* pipeline.
*/
- left = addMaterializationSteps(ctx, left, doneSet, needsMaterialization,
- subgroup.getQueryHints());
+ left = addMaterializationSteps3(left, doneSet, needsMaterialization,
+ subgroup.getQueryHints(), ctx);
return left;
@@ -3663,10 +3691,11 @@
}
/**
- * Add an aggregation operator. It will handle the grouping (if any),
- * optional having filter, and projected select expressions. A generalized
- * aggregation operator will be used unless the aggregation corresponds to
- * some special case.
+ * Add an aggregation operator. It will handle the <code>GROUP BY</code> (if
+ * any), the optional <code>HAVING</code> filter, and projected
+ * <code>SELECT</code> expressions. A generalized aggregation operator will
+ * be used unless the aggregation corresponds to some special case, e.g.,
+ * pipelined aggregation.
*
* @param left
* The previous operator in the pipeline.
@@ -3701,6 +3730,9 @@
final IGroupByRewriteState groupByRewrite = new GroupByRewriter(
groupByState);
+ // The query hints are taken from the PROJECTION.
+ final Properties queryHints = projection.getQueryHints();
+
final int bopId = ctx.nextId();
final GroupByOp op;
@@ -3784,7 +3816,7 @@
}
- left = addMaterializationSteps(left, bopId, vars, ctx);
+ left = addMaterializationSteps2(left, bopId, vars, queryHints, ctx);
if (!groupByState.isAnyDistinct() && !groupByState.isSelectDependency()
&& !groupByState.isNestedAggregates()) {
@@ -3847,7 +3879,7 @@
}
- left = applyQueryHints(op, ctx.queryHints);
+ left = applyQueryHints(op, queryHints, ctx);
return left;
@@ -3858,8 +3890,12 @@
*/
@SuppressWarnings({ "unchecked", "rawtypes" })
private static final PipelineOp addOrderBy(PipelineOp left,
- final OrderByNode orderBy, final AST2BOpContext ctx) {
+ final QueryBase queryBase, final OrderByNode orderBy,
+ final AST2BOpContext ctx) {
+ // The query hints are taken from the QueryBase
+ final Properties queryHints = queryBase.getQueryHints();
+
final Set<IVariable<IV>> vars = new LinkedHashSet<IVariable<IV>>();
final ISortOrder<IV>[] sortOrders = new ISortOrder[orderBy.size()];
@@ -3899,7 +3935,7 @@
final int sortId = ctx.nextId();
- left = addMaterializationSteps(left, sortId, vars, ctx);
+ left = addMaterializationSteps2(left, sortId, vars, queryHints, ctx);
left = applyQueryHints(
new MemorySortOp(
@@ -3919,7 +3955,7 @@
// new NV(MemorySortOp.Annotations.SHARED_STATE,
// true),//
new NV(MemorySortOp.Annotations.LAST_PASS, true),//
- })), ctx.queryHints);
+ })), queryHints, ctx);
return left;
@@ -3929,7 +3965,8 @@
* Impose an OFFSET and/or LIMIT on a query.
*/
private static final PipelineOp addSlice(PipelineOp left,
- final SliceNode slice, final AST2BOpContext ctx) {
+ final QueryBase queryBase, final SliceNode slice,
+ final AST2BOpContext ctx) {
final int bopId = ctx.nextId();
@@ -3942,7 +3979,7 @@
new NV(SliceOp.Annotations.PIPELINED, true),//
new NV(SliceOp.Annotations.MAX_PARALLEL, 1),//
new NV(MemorySortOp.Annotations.SHARED_STATE, true)//
- ), slice, ctx);
+ ), queryBase, ctx);
return left;
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-06 15:04:22
|
Revision: 7735
http://bigdata.svn.sourceforge.net/bigdata/?rev=7735&view=rev
Author: thompsonbry
Date: 2014-01-06 15:04:15 +0000 (Mon, 06 Jan 2014)
Log Message:
-----------
javadoc update indicating that these two classes are dead code and could be removed.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/paths/ZeroLengthPathOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/ZeroLengthPathNode.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/paths/ZeroLengthPathOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/paths/ZeroLengthPathOp.java 2014-01-06 14:12:44 UTC (rev 7734)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/paths/ZeroLengthPathOp.java 2014-01-06 15:04:15 UTC (rev 7735)
@@ -46,7 +46,9 @@
/**
* An attempt to solve the zero length path problem with its own operator.
*
- * @deprecated Does not work. Leads to cardinality problems.
+ * @deprecated Does not work. Leads to cardinality problems and can be removed.
+ * Zero Length Paths are integrated into the ALP node /
+ * ArbitraryLengthPathOp now.
*/
public class ZeroLengthPathOp extends PipelineOp {
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/ZeroLengthPathNode.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/ZeroLengthPathNode.java 2014-01-06 14:12:44 UTC (rev 7734)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/ZeroLengthPathNode.java 2014-01-06 15:04:15 UTC (rev 7735)
@@ -14,7 +14,9 @@
* A special kind of AST node that represents the SPARQL 1.1 zero length path
* operator.
*
- * @deprecated Does not work - leads to cardinality problems.
+ * @deprecated Does not work - leads to cardinality problems and can be removed.
+ * Zero Length Paths are integrated into the ALP node /
+ * ArbitraryLengthPathOp now.
*/
public class ZeroLengthPathNode
extends GroupMemberNodeBase<ZeroLengthPathNode>
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-06 19:10:55
|
Revision: 7739
http://bigdata.svn.sourceforge.net/bigdata/?rev=7739&view=rev
Author: thompsonbry
Date: 2014-01-06 19:10:46 +0000 (Mon, 06 Jan 2014)
Log Message:
-----------
Checkpoint on the RTO integration - See #64 (RTO).
I have reworked the AST2BOpUtility, AST2BOpJoins, and AST2BOpFilters
classes to support reuse of AST2BOpJoins#join() within the RTO and I
have verified (for one query using FILTER (?x != ?y)) that the RTO
will now generate a final query plan which includes conditional
routing operators to materialize variables for filters. However, the
sampling logic for cutoff joins has not yet been modified and will
ignore any filters that require materialization (or perhaps just die
on them). The old versus new integration point is configurable in
AST2BOpRTO so the old behavior can be trivially restored. I have also
added several static booleans into that class to allow us to gradually
enable more capabilities in the RTO integration.
The join graph test suite was not running. It is now hooked into CI.
However, there is not much in there and at least part of it might be
going away (for "fast join graphs").
The bop, AST, SPARQL evaluation test suites are all green.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpIdFactory.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/IdFactory.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/PartitionedJoinGroup.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/TestAll.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/joinGraph/rto/TestAll.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/joinGraph/rto/TestJGraph.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/joinGraph/rto/TestJoinGraph.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpContext.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpFilters.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpJoins.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/SimpleIdFactory.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpIdFactory.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpIdFactory.java 2014-01-06 17:19:27 UTC (rev 7738)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpIdFactory.java 2014-01-06 19:10:46 UTC (rev 7739)
@@ -1,5 +1,29 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2011. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
package com.bigdata.bop;
+import java.util.Iterator;
import java.util.LinkedHashSet;
/**
@@ -7,23 +31,102 @@
*/
public class BOpIdFactory implements IdFactory {
- private final LinkedHashSet<Integer> ids = new LinkedHashSet<Integer>();
-
- private int nextId = 0;
-
- public void reserve(int id) {
- ids.add(id);
- }
+ /** The set of reserved bop identifiers. */
+ private LinkedHashSet<Integer> ids;
- public int nextId() {
+ private int nextId = 0;
- while (ids.contains(nextId)) {
+ /**
+ * Reserve a bop id by adding it to a set of known identifiers that will not
+ * be issued by {@link #nextId()}.
+ *
+ * @param id
+ * The identifier.
+ */
+ public void reserve(final int id) {
+
+ synchronized (this) {
+
+ if (ids == null) {
- nextId++;
-
- }
+ // Lazily allocated.
+ ids = new LinkedHashSet<Integer>();
- return nextId++;
- }
-
+ ids.add(id);
+
+ }
+
+ }
+
+ }
+
+ @Override
+ public int nextId() {
+
+ synchronized (this) {
+
+ if (ids != null) {
+
+ while (ids.contains(nextId)) {
+
+ nextId++;
+
+ }
+
+ }
+
+ return nextId++;
+
+ }
+
+ }
+
+ /**
+ * Reserve ids used by the predicates or constraints associated with some
+ * join graph.
+ *
+ * @param preds
+ * The vertices of the join graph.
+ * @param constraints
+ * The constraints of the join graph (optional).
+ */
+ public void reserveIds(final IPredicate<?>[] preds,
+ final IConstraint[] constraints) {
+
+ if (preds == null)
+ throw new IllegalArgumentException();
+
+ final BOpIdFactory idFactory = this;
+
+ for (IPredicate<?> p : preds) {
+
+ idFactory.reserve(p.getId());
+
+ }
+
+ if (constraints != null) {
+
+ for (IConstraint c : constraints) {
+
+ final Iterator<BOp> itr = BOpUtility
+ .preOrderIteratorWithAnnotations(c);
+
+ while (itr.hasNext()) {
+
+ final BOp y = itr.next();
+
+ final Integer anId = (Integer) y
+ .getProperty(BOp.Annotations.BOP_ID);
+
+ if (anId != null)
+ idFactory.reserve(anId.intValue());
+
+ }
+
+ }
+
+ }
+
+ }
+
}
\ No newline at end of file
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/IdFactory.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/IdFactory.java 2014-01-06 17:19:27 UTC (rev 7738)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/IdFactory.java 2014-01-06 19:10:46 UTC (rev 7739)
@@ -1,3 +1,26 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2011. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
package com.bigdata.bop;
/**
@@ -5,6 +28,9 @@
*/
public interface IdFactory {
+ /**
+ * Issue the next bop identifier.
+ */
public int nextId();
}
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/SimpleIdFactory.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/SimpleIdFactory.java (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/SimpleIdFactory.java 2014-01-06 19:10:46 UTC (rev 7739)
@@ -0,0 +1,47 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2011. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+package com.bigdata.bop;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class SimpleIdFactory implements IdFactory {
+
+ /**
+ * Note: The ids are assigned using {@link AtomicInteger#incrementAndGet()}
+ * so ONE (1) is the first id that will be assigned when we pass in ZERO (0)
+ * as the initial state of the {@link AtomicInteger}.
+ */
+ private final AtomicInteger nextId = new AtomicInteger(0);
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public int nextId() {
+
+ return nextId.incrementAndGet();
+
+ }
+
+}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/PartitionedJoinGroup.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/PartitionedJoinGroup.java 2014-01-06 17:19:27 UTC (rev 7738)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/PartitionedJoinGroup.java 2014-01-06 19:10:46 UTC (rev 7739)
@@ -1018,24 +1018,25 @@
/*
* Reserve ids used by the join graph or its constraints.
*/
- {
- for (IPredicate<?> p : preds) {
- idFactory.reserve(p.getId());
- }
- if (constraints != null) {
- for (IConstraint c : constraints) {
- final Iterator<BOp> itr = BOpUtility
- .preOrderIteratorWithAnnotations(c);
- while (itr.hasNext()) {
- final BOp y = itr.next();
- final Integer anId = (Integer) y
- .getProperty(BOp.Annotations.BOP_ID);
- if (anId != null)
- idFactory.reserve(anId.intValue());
- }
- }
- }
- }
+ idFactory.reserveIds(preds, constraints);
+// {
+// for (IPredicate<?> p : preds) {
+// idFactory.reserve(p.getId());
+// }
+// if (constraints != null) {
+// for (IConstraint c : constraints) {
+// final Iterator<BOp> itr = BOpUtility
+// .preOrderIteratorWithAnnotations(c);
+// while (itr.hasNext()) {
+// final BOp y = itr.next();
+// final Integer anId = (Integer) y
+// .getProperty(BOp.Annotations.BOP_ID);
+// if (anId != null)
+// idFactory.reserve(anId.intValue());
+// }
+// }
+// }
+// }
// figure out which constraints are attached to which predicates.
final IConstraint[][] assignedConstraints = PartitionedJoinGroup
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java 2014-01-06 17:19:27 UTC (rev 7738)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java 2014-01-06 19:10:46 UTC (rev 7739)
@@ -29,6 +29,7 @@
import java.util.LinkedHashMap;
import java.util.Map;
+import java.util.Set;
import java.util.concurrent.Callable;
import java.util.concurrent.FutureTask;
import java.util.concurrent.TimeUnit;
@@ -37,11 +38,11 @@
import com.bigdata.bop.BOp;
import com.bigdata.bop.BOpContext;
-import com.bigdata.bop.BOpIdFactory;
import com.bigdata.bop.BOpUtility;
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.IConstraint;
import com.bigdata.bop.IPredicate;
+import com.bigdata.bop.IQueryAttributes;
import com.bigdata.bop.IVariable;
import com.bigdata.bop.NV;
import com.bigdata.bop.PipelineOp;
@@ -51,7 +52,12 @@
import com.bigdata.bop.engine.AbstractRunningQuery;
import com.bigdata.bop.engine.IRunningQuery;
import com.bigdata.bop.engine.QueryEngine;
-import com.bigdata.bop.joinGraph.PartitionedJoinGroup;
+import com.bigdata.rdf.sparql.ast.IJoinNode;
+import com.bigdata.rdf.sparql.ast.JoinGroupNode;
+import com.bigdata.rdf.sparql.ast.eval.AST2BOpContext;
+import com.bigdata.rdf.sparql.ast.eval.AST2BOpRTO;
+import com.bigdata.rdf.sparql.ast.optimizers.IASTOptimizer;
+import com.bigdata.util.NT;
import com.bigdata.util.concurrent.Haltable;
import cutthecrap.utils.striterators.ICloseableIterator;
@@ -88,10 +94,11 @@
*/
public interface Annotations extends PipelineOp.Annotations {
- /**
- * The variables which are projected out of the join graph.
- */
- String SELECTED = JoinGraph.class.getName() + ".selected";
+ /**
+ * The variables to be projected out of the join graph (optional). When
+ * <code>null</code>, all variables will be projected out.
+ */
+ String SELECTED = JoinGraph.class.getName() + ".selected";
/**
* The vertices of the join graph, expressed an an {@link IPredicate}[]
@@ -136,12 +143,65 @@
String DEFAULT_SAMPLE_TYPE = SampleType.RANDOM.name();
+ /**
+ * The set of variables that are known to have already been materialized
+ * in the context in which the RTO was invoked.
+ *
+ * FIXME In order to support left-to-right evaluation fully, the
+ * {@link JGraph} needs to accept this, track it as it binds variables,
+ * and pass it through when doing cutoff joins to avoid pipeline
+ * materialization steps for variables that are already known to be
+ * materialized. Otherwise the RTO will assume that it needs to
+ * materialize everything that needs to be materialized for a FILTER and
+ * thus do too much work (which is basically the assumption of bottom-up
+ * evaluation, or if you prefer that it is executing in its own little
+ * world).
+ */
+ String DONE_SET = JoinGraph.class.getName() + ".doneSet";
+
+// /**
+// * The query hints from the dominating AST node (if any). These query
+// * hints will be passed through and made available when we compile the
+// * query plan once the RTO has decided on the join ordering. While the
+// * RTO is running, it needs to override many of the query hints for the
+// * {@link IPredicate}s, {@link PipelineJoin}s, etc. in order to ensure
+// * that the cutoff evaluation semantics are correctly applied while it
+// * is exploring the plan state space for the join graph.
+// */
+// String AST_QUERY_HINTS = JoinGraph.class.getName() + ".astQueryHints";
+
+ /**
+ * The AST {@link JoinGroupNode} for the joins and filters that we are
+ * running through the RTO (required).
+ *
+ * FIXME This should be set by an ASTRTOOptimizer. That class should
+ * rewrite the original join group, replacing some set of joins with a
+ * JoinGraphNode which implements {@link IJoinNode} and gets hooked into
+ * AST2BOpUtility#convertJoinGroup() normally rather than through
+ * expectional processing. This will simplify the code and adhere to the
+ * general {@link IASTOptimizer} pattern and avoid problems with cloning
+ * children out of the {@link JoinGroupNode} when we set it up to run
+ * the RTO. [Eventually, we will need to pass this in rather than the
+ * {@link IPredicate}[] in order to handle JOINs that are not SPs, e.g.,
+ * sub-selects, etc.]
+ */
+ String JOIN_GROUP = JoinGraph.class.getName() + ".joinGroup";
+
+ /**
+ * An {@link NT} object specifying the namespace and timestamp of the KB
+ * view against which the RTO is running. This is necessary in order to
+ * reconstruct the {@link AST2BOpContext} when it comes time to evaluate
+ * either a cutoff join involving filters that need materialization or
+ * the selected join path.
+ */
+ String NT = JoinGraph.class.getName() + ".nt";
+
}
/**
- * Query attribute names for the {@link JoinGraph}. The fully qualified name
- * of the attribute is formed by appending the attribute name to the
- * "bopId-", where <code>bopId</code> is the value returned by
+ * {@link IQueryAttributes} names for the {@link JoinGraph}. The fully
+ * qualified name of the attribute is formed by appending the attribute name
+ * to the "bopId-", where <code>bopId</code> is the value returned by
* {@link BOp#getId()}
*
* @author <a href="mailto:tho...@us...">Bryan
@@ -168,6 +228,10 @@
}
+ /*
+ * JoinGraph operator annotations.
+ */
+
/**
* @see Annotations#SELECTED
*/
@@ -223,6 +287,24 @@
}
+ /**
+ * Return the set of variables that are known to have already been
+ * materialized at the point in the overall query plan where the RTO is
+ * being executed.
+ *
+ * @see Annotations#DONE_SET
+ */
+ @SuppressWarnings("unchecked")
+ public Set<IVariable<?>> getDoneSet() {
+
+ return (Set<IVariable<?>>) getRequiredProperty(Annotations.DONE_SET);
+
+ }
+
+ /*
+ * IQueryAttributes
+ */
+
/**
* Return the computed join path.
*
@@ -301,15 +383,15 @@
super(args, anns);
- // required property.
- final IVariable<?>[] selected = (IVariable[]) getProperty(Annotations.SELECTED);
+ // optional property.
+// final IVariable<?>[] selected = (IVariable[]) getProperty(Annotations.SELECTED);
+//
+// if (selected == null)
+// throw new IllegalArgumentException(Annotations.SELECTED);
+//
+// if (selected.length == 0)
+// throw new IllegalArgumentException(Annotations.SELECTED);
- if (selected == null)
- throw new IllegalArgumentException(Annotations.SELECTED);
-
- if (selected.length == 0)
- throw new IllegalArgumentException(Annotations.SELECTED);
-
// required property.
final IPredicate<?>[] vertices = (IPredicate[]) getProperty(Annotations.VERTICES);
@@ -325,6 +407,18 @@
if (getNEdges() <= 0)
throw new IllegalArgumentException(Annotations.NEDGES);
+ /*
+ * TODO Check DONE_SET, NT, JOIN_NODES. These annotations are required
+ * for the new code path. We should check for their presence. However,
+ * the old code path is used by some unit tests which have not yet been
+ * updated and do not supply these annotations.
+ */
+// // Required.
+// getDoneSet();
+//
+// // Required.
+// getRequiredProperty(Annotations.NT);
+
if (!isController())
throw new IllegalArgumentException();
@@ -345,7 +439,6 @@
}
-
/**
* Evaluation of a {@link JoinGraph}.
*
@@ -400,11 +493,11 @@
final Map<PathIds, EdgeSample> edgeSamples = new LinkedHashMap<PathIds, EdgeSample>();
// Find the best join path.
- final Path p = g.runtimeOptimizer(context.getRunningQuery()
+ final Path path = g.runtimeOptimizer(context.getRunningQuery()
.getQueryEngine(), getLimit(), getNEdges(), edgeSamples);
// Set attribute for the join path result.
- setPath(context.getRunningQuery(), p);
+ setPath(context.getRunningQuery(), path);
// Set attribute for the join path samples.
setSamples(context.getRunningQuery(), edgeSamples);
@@ -413,20 +506,11 @@
final long elapsed_queryOptimizer = mark - begin;
- // Factory avoids reuse of bopIds assigned to the predicates.
- final BOpIdFactory idFactory = new BOpIdFactory();
-
/*
- * Generate the query from the join path.
- *
- * FIXME Update this using StaticAnalysis logic. Also, both this and
- * the JGraph need to handle triples versus named graph versus
- * default graph APs. Further, JGraph should handle filters that
- * require conditional materialization.
+ * Generate the query from the selected join path.
*/
- final PipelineOp queryOp = PartitionedJoinGroup.getQuery(idFactory,
- false/* distinct */, getSelected(), p.getPredicates(),
- getConstraints());
+ final PipelineOp queryOp = AST2BOpRTO.compileJoinGraph(context
+ .getRunningQuery().getQueryEngine(), JoinGraph.this, path);
// Set attribute for the join path samples.
setQueryPlan(context.getRunningQuery(), queryOp);
@@ -482,8 +566,9 @@
/*
* Run the query.
*
- * @todo pass in the source binding sets here and also when sampling the
- * vertices.
+ * TODO Pass in the source binding sets here and also when sampling the
+ * vertices? Otherwise it is as if we are doing bottom-up evaluation (in
+ * which case the doneSet should be empty on entry).
*/
ICloseableIterator<IBindingSet[]> subquerySolutionItr = null;
@@ -505,13 +590,9 @@
null/* mergeSolution */, null/* selectVars */,
null/* constraints */, null/* stats */);
-// System.out.println("nout=" + nout);
-
// verify no problems.
runningSubquery.get();
-// System.out.println("Future Ok");
-
} catch (Throwable t) {
if (Haltable.isTerminationByInterrupt(t)) {
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/TestAll.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/TestAll.java 2014-01-06 17:19:27 UTC (rev 7738)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/TestAll.java 2014-01-06 19:10:46 UTC (rev 7739)
@@ -47,7 +47,7 @@
/**
* @param arg0
*/
- public TestAll(String arg0) {
+ public TestAll(final String arg0) {
super(arg0);
@@ -114,6 +114,9 @@
// high level query optimization and evaluation.
suite.addTest(com.bigdata.bop.controller.TestAll.suite());
+ // join graph processing (RTO, etc).
+ suite.addTest(com.bigdata.bop.joinGraph.TestAll.suite());
+
/*
* Note: This is tested later once we have gone through the core unit
* tests for the services.
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/joinGraph/rto/TestAll.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/joinGraph/rto/TestAll.java 2014-01-06 17:19:27 UTC (rev 7738)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/joinGraph/rto/TestAll.java 2014-01-06 19:10:46 UTC (rev 7739)
@@ -45,7 +45,7 @@
/**
* @param arg0
*/
- public TestAll(String arg0) {
+ public TestAll(final String arg0) {
super(arg0);
@@ -64,7 +64,8 @@
suite.addTestSuite(TestJoinGraph.class);
// runtime query optimizer behavior.
- suite.addTestSuite(TestJGraph.class);
+ // FIXME This test suite is empty. Either test at the AST eval level or add tests here.
+// suite.addTestSuite(TestJGraph.class);
return suite;
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/joinGraph/rto/TestJGraph.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/joinGraph/rto/TestJGraph.java 2014-01-06 17:19:27 UTC (rev 7738)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/joinGraph/rto/TestJGraph.java 2014-01-06 19:10:46 UTC (rev 7739)
@@ -60,15 +60,15 @@
/**
* @param name
*/
- public TestJGraph(String name) {
+ public TestJGraph(final String name) {
super(name);
}
- public void test_something() {
-
- fail("write tests");
-
- }
+// public void test_something() {
+//
+// fail("write tests");
+//
+// }
// /**
// * Test ability to recognize when there is a predicate without any shared
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/joinGraph/rto/TestJoinGraph.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/joinGraph/rto/TestJoinGraph.java 2014-01-06 17:19:27 UTC (rev 7738)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/joinGraph/rto/TestJoinGraph.java 2014-01-06 19:10:46 UTC (rev 7739)
@@ -34,6 +34,7 @@
import com.bigdata.bop.Constant;
import com.bigdata.bop.IConstraint;
import com.bigdata.bop.IPredicate;
+import com.bigdata.bop.IVariable;
import com.bigdata.bop.NV;
import com.bigdata.bop.Var;
import com.bigdata.bop.ap.Predicate;
@@ -75,6 +76,7 @@
};
final IConstraint[] constraints = null;
final JoinGraph joinGraph = new JoinGraph(new BOp[0],//
+ new NV(JoinGraph.Annotations.SELECTED, new IVariable[]{}),//
new NV(JoinGraph.Annotations.VERTICES, vertices),//
new NV(JoinGraph.Annotations.CONTROLLER, true), //
new NV(JoinGraph.Annotations.EVALUATION_CONTEXT,
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpContext.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpContext.java 2014-01-06 17:19:27 UTC (rev 7738)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpContext.java 2014-01-06 19:10:46 UTC (rev 7739)
@@ -28,7 +28,6 @@
import java.util.Map;
import java.util.Properties;
import java.util.UUID;
-import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
import com.bigdata.bop.BOp;
@@ -38,6 +37,7 @@
import com.bigdata.bop.IdFactory;
import com.bigdata.bop.NamedSolutionSetRefUtility;
import com.bigdata.bop.PipelineOp;
+import com.bigdata.bop.SimpleIdFactory;
import com.bigdata.bop.engine.IRunningQuery;
import com.bigdata.bop.engine.QueryEngine;
import com.bigdata.bop.fed.QueryEngineFactory;
@@ -79,7 +79,7 @@
public class AST2BOpContext implements IdFactory, IEvaluationContext {
/**
- * The {@link ASTContainer}
+ * The {@link ASTContainer} and never <code>null</code>.
*/
public final ASTContainer astContainer;
@@ -88,7 +88,7 @@
*
* @see #nextId()
*/
- private final AtomicInteger idFactory;
+ private final IdFactory idFactory;
/**
* The KB instance.
@@ -368,8 +368,9 @@
/**
*
- * @param queryRoot
- * The root of the query.
+ * @param astContainer
+ * The top-level {@link ASTContainer} for the query or update
+ * request to be evaluated (required).
* @param db
* The KB instance.
*
@@ -382,26 +383,32 @@
* {@link FunctionRegistry}.
*/
public AST2BOpContext(final ASTContainer astContainer,
- final AbstractTripleStore db) {
+ final AbstractTripleStore db) {
+ this(astContainer, db, new SimpleIdFactory());
+
+ }
+
+ // Note: Exposed to AST2BOpRTO
+ AST2BOpContext(final ASTContainer astContainer,
+ final AbstractTripleStore db, final IdFactory idFactory) {
+
if (astContainer == null)
throw new IllegalArgumentException();
if (db == null)
throw new IllegalArgumentException();
+ if (idFactory == null)
+ throw new IllegalArgumentException();
+
this.astContainer = astContainer;
this.db = db;
this.optimizers = new DefaultOptimizerList();
- /*
- * Note: The ids are assigned using incrementAndGet() so ONE (1) is the
- * first id that will be assigned when we pass in ZERO (0) as the
- * initial state of the AtomicInteger.
- */
- this.idFactory = new AtomicInteger(0);
+ this.idFactory = idFactory;
this.queryEngine = QueryEngineFactory.getQueryController(db
.getIndexManager());
@@ -496,7 +503,8 @@
@Override
public int nextId() {
- return idFactory.incrementAndGet();
+ return idFactory.nextId();
+// return idFactory.incrementAndGet();
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpFilters.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpFilters.java 2014-01-06 17:19:27 UTC (rev 7738)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpFilters.java 2014-01-06 19:10:46 UTC (rev 7739)
@@ -523,10 +523,11 @@
// remove any terms already materialized
terms.removeAll(alreadyMaterialized);
- if (c instanceof INeedsMaterialization && ((INeedsMaterialization) c).getRequirement() == Requirement.ALWAYS) {
-
- // add any new terms to the list of already materialized
- alreadyMaterialized.addAll(terms);
+ if (c instanceof INeedsMaterialization
+ && ((INeedsMaterialization) c).getRequirement() == Requirement.ALWAYS) {
+
+ // add any new terms to the list of already materialized
+ alreadyMaterialized.addAll(terms);
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpJoins.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpJoins.java 2014-01-06 17:19:27 UTC (rev 7738)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpJoins.java 2014-01-06 19:10:46 UTC (rev 7739)
@@ -220,7 +220,7 @@
anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.SHARDED));
- pred = (Predicate) pred.setProperty(
+ pred = (Predicate<?>) pred.setProperty(
Predicate.Annotations.REMOTE_ACCESS_PATH, false);
} else {
@@ -267,7 +267,7 @@
*/
anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.SHARDED));
- pred = (Predicate) pred.setProperty(
+ pred = (Predicate<?>) pred.setProperty(
Predicate.Annotations.REMOTE_ACCESS_PATH, false);
} else {
anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java 2014-01-06 17:19:27 UTC (rev 7738)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java 2014-01-06 19:10:46 UTC (rev 7739)
@@ -26,6 +26,8 @@
*/
package com.bigdata.rdf.sparql.ast.eval;
+import java.util.Arrays;
+import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.LinkedList;
@@ -36,21 +38,33 @@
import com.bigdata.bop.BOp;
import com.bigdata.bop.BOpEvaluationContext;
+import com.bigdata.bop.BOpIdFactory;
import com.bigdata.bop.IConstraint;
+import com.bigdata.bop.IPredicate;
import com.bigdata.bop.IValueExpression;
import com.bigdata.bop.IVariable;
import com.bigdata.bop.NV;
import com.bigdata.bop.PipelineOp;
import com.bigdata.bop.ap.Predicate;
import com.bigdata.bop.ap.SampleIndex.SampleType;
+import com.bigdata.bop.engine.QueryEngine;
+import com.bigdata.bop.joinGraph.PartitionedJoinGroup;
import com.bigdata.bop.joinGraph.rto.JGraph;
import com.bigdata.bop.joinGraph.rto.JoinGraph;
+import com.bigdata.bop.joinGraph.rto.Path;
+import com.bigdata.bop.solutions.ProjectionOp;
+import com.bigdata.journal.IIndexManager;
import com.bigdata.rdf.internal.IV;
import com.bigdata.rdf.internal.constraints.INeedsMaterialization;
+import com.bigdata.rdf.sparql.ast.ASTBase;
+import com.bigdata.rdf.sparql.ast.ASTContainer;
+import com.bigdata.rdf.sparql.ast.IBindingProducerNode;
import com.bigdata.rdf.sparql.ast.IGroupMemberNode;
import com.bigdata.rdf.sparql.ast.JoinGroupNode;
import com.bigdata.rdf.sparql.ast.QueryHints;
import com.bigdata.rdf.sparql.ast.StatementPatternNode;
+import com.bigdata.rdf.store.AbstractTripleStore;
+import com.bigdata.util.NT;
/**
* Integration with the Runtime Optimizer (RTO).
@@ -84,6 +98,70 @@
public class AST2BOpRTO extends AST2BOpJoins {
/**
+ * When <code>true</code>, the RTO will only accept simple joins into the
+ * join graph. Simple joins includes triples-mode joins and filters that do
+ * not have materialization requirements. Non-simple joins includes
+ * quads-mode joins and filters that have materialization contexts.
+ *
+ * TODO Eventually we can drop this. It is being used while we refactor the
+ * RTO to support more complex joins and variable materialization for
+ * filters.
+ */
+ static private final boolean onlySimpleJoins = false;
+
+ /**
+ * When <code>true</code>, the RTO will not accept OPTIONAL joins into the
+ * join graph. Optional joins can not decrease the intermediate cardinality
+ * since they can not eliminate solutions, but they can increase the
+ * intermediate cardinality by finding multiple bindings in the OPTIONAL
+ * join. Therefore, it makes sense to execute optional in an order that
+ * defers as long as possible any increase in the intermediate cardinality.
+ * This is very much the same as ordering required joins, but the OPTIONAL
+ * joins need to be handled AFTER the required joins.
+ *
+ * TODO RTO OPTIONALS: Handle optional SPs in joinGraph by ordering them in
+ * the tail so as to minimize the cost function. Once implemented, we can
+ * drop this field.
+ */
+ static private final boolean onlyRequiredJoins = true;
+
+ /**
+ * When <code>true</code>, the RTO will only accept statement patterns into
+ * the join graph. When <code>false</code>, it will attempt to handle non-SP
+ * joins, e.g., sub-query, exists, property paths, etc.
+ *
+ * TODO Eventually we can drop this.
+ */
+ static private final boolean onlySPs = true;
+
+ /**
+ * When <code>true</code>, the RTO will be applied as in we were doing
+ * bottom-up query optimization. In this case, it WILL NOT receive any
+ * solutions from the upstream operators in the pipeline when it performs
+ * its runtime sampling and it will ignore the <code>doneSet</code> for the
+ * context in which it is invoked. When run in this manner, the RTO *could*
+ * be run before the main query is executed. The only way to facilitate this
+ * at this time would be to lift out the joins on which the RTO would be run
+ * into a named subquery and then optimize that named subquery before the
+ * rest of the query.
+ * <p>
+ * When <code>false</code>, the RTO solutions from upstream operators will
+ * flow into the RTO.
+ *
+ * TODO We could still pass in exogenous solutions for bottom up evaluation.
+ * This would help constraint the RTOs exploration.
+ *
+ * TODO The RTO is not operating 100% in either an left-to-right or a
+ * bottom-up fashion, primarily because we are not passing in either
+ * exogenous bindings or meaningfully using the bindings from the upstream
+ * operator when exploring the join graph. In fact, the RTO could accept a
+ * new sample from the upstream operator in each iteration drawing from
+ * amoung those solutions which had already been materialized by the
+ * upstream operator.
+ */
+ static private final boolean bottomUp = true;
+
+ /**
* Inspect the remainder of the join group. If we can isolate a join graph
* and filters, then we will push them down into an RTO JoinGroup. Since the
* joins have already been ordered by the static optimizer, we can accept
@@ -101,117 +179,190 @@
final JoinGroupNode joinGroup, final Set<IVariable<?>> doneSet,
final AST2BOpContext ctx, final AtomicInteger start) {
- if (ctx.isQuads()) {
+ /*
+ * Snapshot of the doneSet on entry. This gets passed into the RTO.
+ */
+ final Set<IVariable<?>> doneSetIn = Collections
+ .unmodifiableSet(doneSet);
- // FIXME RTO: The RTO does not handle quads yet.
+ if (onlySimpleJoins && ctx.isQuads()) {
+
return left;
}
-
+
+ /*
+ * Consider the join group. See if it is complex enough to warrant
+ * running the RTO.
+ *
+ * TODO Can we also make a decision based on whether there is uncertain,
+ * e.g., a swing in stakes, about the cardinality estimates for the
+ * predicates in the join graph, etc.? This could give us a means to
+ * avoid using the RTO if the join graph is known to run quickly or the
+ * ordering of the joins generated by the static query optimizer is
+ * known to be good.
+ *
+ * TODO The static optimizer could simply annotation join groups for
+ * which it recognizes that it could have a bad join plan.
+ */
final int arity = joinGroup.arity();
- // The predicates for the RTO join group.
- final Set<StatementPatternNode> sps = new LinkedHashSet<StatementPatternNode>();
+ /*
+ * Create a JoinGroup just for the pieces that the RTO will handle.
+ *
+ * FIXME These joins are CLONED right now since they also exist in the
+ * caller's joinGroup. Eventually, this code should be moved into an AST
+ * rewrite and we can then destructively move the children from the
+ * original JoinGroupNode into a JoinGraphNode that will be the AST side
+ * of the RTO.
+ */
+ final JoinGroupNode rtoJoinGroup = new JoinGroupNode();
+ rtoJoinGroup.setQueryHints(joinGroup.getQueryHints());
+
+// final Set<StatementPatternNode> sps = new LinkedHashSet<StatementPatternNode>();
+ // The predicates for the join graph.
@SuppressWarnings("rawtypes")
final Set<Predicate> preds = new LinkedHashSet<Predicate>();
+ // The constraints for the join graph.
final List<IConstraint> constraints = new LinkedList<IConstraint>();
-
- // Examine the remaining joins, stopping at the first non-SP.
- for (int i = start.get(); i < arity; i++) {
-
- final IGroupMemberNode child = (IGroupMemberNode) joinGroup
- .get(i);
-
- if (child instanceof StatementPatternNode) {
- // SP
- final StatementPatternNode sp = (StatementPatternNode) child;
- final boolean optional = sp.isOptional();
- if(optional) {
+ // The #of JOINs accepted into the RTO's join group.
+ int naccepted = 0;
+ {
+ /*
+ * The [doneSet] will be modified iff we actually accept JOINs into the
+ * RTO. Therefore, we also make a temporary copy that we will use to
+ * avoid side-effects if this join group is not complex enough to run
+ * the RTO.
+ */
+ final Set<IVariable<?>> doneSetTmp = new LinkedHashSet<IVariable<?>>(
+ doneSet);
+
+ // Examine the remaining joins, stopping at the first non-SP.
+ for (int i = start.get(); i < arity; i++) {
+
+ final IGroupMemberNode child = (IGroupMemberNode) joinGroup
+ .get(i);
+
+ if (child instanceof StatementPatternNode) {
+ // SP
+ StatementPatternNode sp = (StatementPatternNode) child;
+ final boolean optional = sp.isOptional();
+ if (onlyRequiredJoins && optional) {
+ /*
+ * TODO The RTO does not order optional joins yet so we
+ * can not accept this join into the join graph.
+ */
+ break;
+ }
+
+ final List<IConstraint> attachedConstraints = getJoinConstraints(sp);
+
+ @SuppressWarnings("rawtypes")
+ final Map<IConstraint, Set<IVariable<IV>>> needsMaterialization =
+ new LinkedHashMap<IConstraint, Set<IVariable<IV>>>();
+
+ getJoinConstraints(attachedConstraints, needsMaterialization);
+
+ if (onlySimpleJoins && !needsMaterialization.isEmpty()) {
+ /*
+ * At least one variable requires (or might require)
+ * materialization. This is not currently handled by the RTO
+ * so we break out of the loop.
+ *
+ * TODO Handle materialization patterns within the RTO, in
+ * which case we need to collect up the doneSet here, but
+ * only conditionally since we might not actually execute the
+ * RTO depending on the number of SPs that we find.
+ */
+ break;
+ }
+
+ // // Add constraints to the join for that predicate.
+ // anns.add(new NV(JoinAnnotations.CONSTRAINTS, getJoinConstraints(
+ // constraints, needsMaterialization)));
+
+ // /*
+ // * Pull off annotations before we clear them from the predicate.
+ // */
+ // final Scope scope = (Scope) pred.getProperty(Annotations.SCOPE);
+ //
+ // // true iff this is a quads access path.
+ // final boolean quads = pred.getProperty(Annotations.QUADS,
+ // Annotations.DEFAULT_QUADS);
+ //
+ // // pull of the Sesame dataset before we strip the annotations.
+ // final DatasetNode dataset = (DatasetNode) pred
+ // .getProperty(Annotations.DATASET);
+
+ // Something the RTO can handle.
+ sp = (StatementPatternNode) sp.clone();// TODO Use destructive move.
+// sp.setId(ctx.nextId()); // assign id so we can reference back later.
+ rtoJoinGroup.addChild(sp); // add to group.
+ naccepted++;
/*
- * TODO Handle optional SPs in joinGraph (by ordering them
- * in the tail so as to minimize the cost function).
+ * FIXME RTO: Handle Triples vs Quads, Default vs Named
+ * Graph, and DataSet. This probably means pushing more
+ * logic down into the RTO from AST2BOpJoins.
+ * Path.cutoffJoin() will need to be call through to logic
+ * on this class that does the right thing with named graph
+ * joins, default graph joins, triples mode joins, remote AP
+ * joins, etc. This is the same code that we call through to
+ * when we take the selected join path from the RTO and
+ * compile it into a query plan to fully execute the join
+ * group.
*/
- break;
- }
-
- final List<IConstraint> attachedConstraints = getJoinConstraints(sp);
-
- @SuppressWarnings("rawtypes")
- final Map<IConstraint, Set<IVariable<IV>>> needsMaterialization =
- new LinkedHashMap<IConstraint, Set<IVariable<IV>>>();
-
- getJoinConstraints(attachedConstraints, needsMaterialization);
-
- if (!needsMaterialization.isEmpty()) {
+ final Predicate<?> pred = AST2BOpUtility.toPredicate(sp, ctx);
+ // final int joinId = ctx.nextId();
+ //
+ // // annotations for this join.
+ // final List<NV> anns = new LinkedList<NV>();
+ //
+ // anns.add(new NV(BOp.Annotations.BOP_ID, joinId));
+ preds.add(pred);
+ if (attachedConstraints != null) {
+ // RTO will figure out where to attach these constraints.
+ constraints.addAll(attachedConstraints);
+ }
+
+ } else {
+ // Non-SP.
+ if (onlySPs)
+ break;
/*
- * At least one variable requires (or might require)
- * materialization. This is not currently handled by
- * the RTO so we break out of the loop.
- *
- * TODO Handle materialization patterns within the RTO.
+ * TODO Handle non-SPs in the RTO. See convertJoinGroup()
+ * for how we handle non-SPs during normal query plan
+ * conversion. All of that would also have to be handled
+ * here for the RTO to allow in non-SPs.
*/
- break;
+ throw new UnsupportedOperationException();
}
-
-// // Add constraints to the join for that predicate.
-// anns.add(new NV(JoinAnnotations.CONSTRAINTS, getJoinConstraints(
-// constraints, needsMaterialization)));
+
+ }
-// /*
-// * Pull off annotations before we clear them from the predicate.
-// */
-// final Scope scope = (Scope) pred.getProperty(Annotations.SCOPE);
-//
-// // true iff this is a quads access path.
-// final boolean quads = pred.getProperty(Annotations.QUADS,
-// Annotations.DEFAULT_QUADS);
-//
-// // pull of the Sesame dataset before we strip the annotations.
-// final DatasetNode dataset = (DatasetNode) pred
-// .getProperty(Annotations.DATASET);
+ if (naccepted < 3) {
- // Something the RTO can handle.
- sps.add(sp);
/*
- * FIXME RTO: Handle Triples vs Quads, Default vs Named Graph, and
- * DataSet. This probably means pushing more logic down into
- * the RTO from AST2BOpJoins.
+ * There are not enough joins for the RTO.
+ *
+ * TODO For incremental query construction UIs, it would be
+ * useful to run just the RTO and to run it with even a single
+ * join. This will give us sample values as well as estimates
+ * cardinalities. If the UI has triple patterns that do not join
+ * (yet), then those should be grouped.
*/
- final Predicate<?> pred = AST2BOpUtility.toPredicate(sp, ctx);
-// final int joinId = ctx.nextId();
-//
-// // annotations for this join.
-// final List<NV> anns = new LinkedList<NV>();
-//
-// anns.add(new NV(BOp.Annotations.BOP_ID, joinId));
- preds.add(pred);
- if (attachedConstraints != null) {
- // RTO will figure out where to attach these constraints.
- constraints.addAll(attachedConstraints);
- }
+ return left;
- } else {
- // Non-SP.
- break;
}
- }
-
- if (sps.size() < 3) {
-
/*
- * There are not enough joins for the RTO.
- *
- * TODO For incremental query construction UIs, it would be useful
- * to run just the RTO and to run it with even a single join. This
- * will give us sample values as well as estimates cardinalities. If
- * the UI has triple patterns that do not join (yet), then those
- * should be grouped.
+ * Since we will run the RTO, we now record any variables that are
+ * known to be materialized in order to support the FILTERs
+ * associated with the join group that we feed into the RTO.
*/
- return left;
+ doneSet.addAll(doneSetTmp);
}
-
+
/*
* Figure out which variables are projected out of the RTO.
*
@@ -221,11 +372,15 @@
final Set<IVariable<?>> selectVars = new LinkedHashSet<IVariable<?>>();
{
- for (StatementPatternNode sp : sps) {
+ for (IGroupMemberNode child : rtoJoinGroup.getChildren()) {
+ if (!(child instanceof IBindingProducerNode))
+ continue;
+
// Note: recursive only matters for complex nodes, not SPs.
- ctx.sa.getDefinitelyProducedBindings(sp, selectVars, true/* recursive */);
-
+ ctx.sa.getDefinitelyProducedBindings(
+ (IBindingProducerNode) child, selectVars, true/* recursive */);
+
}
}
@@ -237,12 +392,16 @@
* (unless we are going to run the RTO "bottom up") and build a hash
* index. When the hash index is ready, we can execute the join group.
*/
+
final SampleType sampleType = joinGroup.getProperty(
QueryHints.RTO_SAMPLE_TYPE, QueryHints.DEFAULT_RTO_SAMPLE_TYPE);
+
final int limit = joinGroup.getProperty(QueryHints.RTO_LIMIT,
QueryHints.DEFAULT_RTO_LIMIT);
+
final int nedges = joinGroup.getProperty(QueryHints.RTO_NEDGES,
QueryHints.DEFAULT_RTO_NEDGES);
+
left = new JoinGraph(leftOrEmpty(left),//
new NV(BOp.Annotations.BOP_ID, ctx.nextId()),//
new NV(BOp.Annotations.EVALUATION_CONTEXT,
@@ -256,16 +415,262 @@
preds.toArray(new Predicate[preds.size()])),//
new NV(JoinGraph.Annotations.CONSTRAINTS, constraints
.toArray(new IConstraint[constraints.size()])),//
+ new NV(JoinGraph.Annotations.JOIN_GROUP, rtoJoinGroup),//
new NV(JoinGraph.Annotations.LIMIT, limit),//
new NV(JoinGraph.Annotations.NEDGES, nedges),//
- new NV(JoinGraph.Annotations.SAMPLE_TYPE, sampleType.name())//
+ new NV(JoinGraph.Annotations.SAMPLE_TYPE, sampleType.name()),//
+ new NV(JoinGraph.Annotations.DONE_SET, doneSetIn),//
+ new NV(JoinGraph.Annotations.NT, new NT(ctx.getNamespace(),
+ ctx.getTimestamp()))//
);
// These joins were consumed.
- start.addAndGet(sps.size());
+ start.addAndGet(naccepted);
return left;
}
-
+
+ /**
+ * Compile a join graph into a query plan.
+ *
+ * @param queryEngine
+ * The {@link QueryEngine} on which the RTO has been executing
+ * and on which the returned query plan may be executed.
+ * @param joinGraph
+ * The operator that executed the RTO.
+ * @param path
+ * The join path that was selected for full execution by the RTO
+ * based on deep sampling of the join graph.
+ *
+ * @return The query plan to fully execute that join graph.
+ */
+ public static PipelineOp compileJoinGraph(final QueryEngine queryEngine,
+ final JoinGraph joinGraph, final Path path) {
+
+ if (queryEngine == null)
+ throw new IllegalArgumentException();
+
+ if (joinGraph == null)
+ throw new IllegalArgumentException();
+
+ if (path == null)
+ throw new IllegalArgumentException();
+
+ final IVariable<?>[] selected = joinGraph.getSelected();
+
+ final IPredicate<?>[] predicates = path.getPredicates();
+
+ final IConstraint[] constraints = joinGraph.getConstraints();
+
+ if (onlySimpleJoins) {
+
+ /*
+ * This is the old code. It does not handle variable materialization
+ * for filters.
+ */
+
+ // Factory avoids reuse of bopIds assigned to the predicates.
+ final BOpIdFactory idFactory = new BOpIdFactory();
+
+ return PartitionedJoinGroup.getQuery(idFactory,
+ false/* distinct */, selected, predicates, constraints);
+
+ }
+
+ /*
+ * TODO The RTO is ignoring the doneSet so it always runs all
+ * materialization steps even if some variable is known to be
+ * materialized on entry.
+ */
+ final Set<IVariable<?>> doneSet = joinGraph.getDoneSet();
+
+ /*
+ * The AST JoinGroupNode for the joins and filters that we are running
+ * through the RTO.
+ */
+ final JoinGroupNode rtoJoinGroup = (JoinGroupNode) joinGraph
+ .getRequiredProperty(JoinGraph.Annotations.JOIN_GROUP);
+
+// // Build an index over the bopIds in that JoinGroupNode.
+// final Map<Integer, BOp> index = getIndex(rtoJoinGroup);
+
+ // Factory avoids reuse of bopIds assigned to the predicates.
+ final BOpIdFactory idFactory = new BOpIdFactory();
+
+ /*
+ * Reserve ids used by the join graph or its constraints.
+ */
+ idFactory.reserveIds(predicates, constraints);
+
+ /*
+ * Figure out which constraints are attached to which predicates.
+ *
+ * TODO Can we reconcile this code with the filter assignment code in
+ * AST2BOpFilter? If so, then we can get rid of the
+ * PartitionedJoinGroup.
+ */
+ final IConstraint[][] assignedConstraints = PartitionedJoinGroup
+ .getJoinGraphConstraints(predicates, constraints,
+ null/* knownBound */, true/* pathIsComplete */);
+
+ // Create an execution context for the query.
+ final AST2BOpContext ctx = getExecutionContext(queryEngine,
+ // Identifies the KB instance (namespace and timestamp).
+ (NT) joinGraph.getRequiredProperty(JoinGraph.Annotations.NT));
+
+ // Start with an empty plan.
+ PipelineOp left = null;
+
+ for (int i = 0; i < predicates.length; i++) {
+
+ final Predicate<?> pred = (Predicate<?>) predicates[i];
+
+ final IConstraint[] attachedJoinConstraints = assignedConstraints[i];
+
+ final boolean optional = pred.isOptional();
+
+ /*
+ * Lookup the AST node for that predicate.
+ *
+ * Note: The predicates are assigned bopIds by the RTO starting with
+ * ONE (1). Therefore we substract out ONE from the predicate's id
+ * to find its index into the join group.
+ *
+ * TODO This assumes that the join group does not contain anything
+ * other than the SPs for the predicates that we are using.,
+ *
+ * TODO HINTS: The Predicate's query hints should the hints for that
+ * specific join (aks the SP or other type of IJoinNode), not the
+ * hints for the JoinGroupNode or the JoinGraph operator. We could
+ * just pass the AST nodes themselves from the JoinGroupNode. That
+ * might make things easier, even if it make the query serialization
+ * fatter on a cluster.
+ */
+// final ASTBase astNode = (ASTBase) index.get(pred.getId());
+
+ final ASTBase astNode = (ASTBase) rtoJoinGroup.get(pred.getId() - 1);
+
+ left = join(left, //
+ pred, //
+ optional ? new LinkedHashSet<IVariable<?>>(doneSet)
+ : doneSet, //
+ attachedJoinConstraints == null ? null : Arrays
+ .asList(attachedJoinConstraints),//
+ astNode.getQueryHints(),//
+ ctx);
+
+ }
+
+ if (selected != null && selected.length != 0) {
+
+ // Drop variables that are not projected out.
+ left = applyQueryHints(new ProjectionOp(//
+ leftOrEmpty(left), //
+ new NV(ProjectionOp.Annotations.BOP_ID, idFactory.nextId()),//
+ new NV(ProjectionOp.Annotations.SELECT, selected)//
+ ), rtoJoinGroup, ctx);
+
+ }
+
+ return left;
+
+ }
+
+ /**
+ * Return an execution context that may be used to execute a cutoff join
+ * during sampling or the entire join path once it has been identified.
+ *
+ * @param queryEngine
+ * The query engine on which the RTO is executing.
+ * @param nt
+ * The namespace and timestamp of the KB view against which the
+ * RTO is running.
+ *
+ * @throws RuntimeException
+ * if there is no such KB instance.
+ */
+ private static AST2BOpContext getExecutionContext(
+ final QueryEngine queryEngine, final NT nt) {
+
+ // The index manager that can be used to resolve the KB view.
+ final IIndexManager indexManager = queryEngine.getFederation() == null ? queryEngine
+ .getIndexManager() : queryEngine.getFederation();
+
+ // Resolve the KB instance.
+ final AbstractTripleStore db = (AbstractTripleStore) indexManager
+ .getResourceLocator().locate(nt.getName(), nt.getTimestamp());
+
+ if (db == null)
+ throw new RuntimeException("No such KB? " + nt);
+
+ /*
+ * An empty container. You can not use any of the top-level SPARQL query
+ * conversion routines with this container, but it is enough for the
+ * low-level things that we need to run the RTO.
+ */
+ final ASTContainer astContainer = new ASTContainer(BOp.NOARGS,
+ BOp.NOANNS);
+
+ return...
[truncated message content] |
|
From: <tho...@us...> - 2014-01-07 18:34:13
|
Revision: 7745
http://bigdata.svn.sourceforge.net/bigdata/?rev=7745&view=rev
Author: thompsonbry
Date: 2014-01-07 18:34:06 +0000 (Tue, 07 Jan 2014)
Log Message:
-----------
Modified the QueryEngine to support a listener interface that allows unit tests to hook the IRunningQuery object in order to observe details about the manner in which the query was actually executed (BOpStats, IQueryAttributes), check for proper release of native memory, etc.
More work on the RTO integration.
See #64 (RTO)
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/AbstractDataAndSPARQLTestCase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/AbstractDataDrivenSPARQLTestCase.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java 2014-01-07 18:30:50 UTC (rev 7744)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java 2014-01-07 18:34:06 UTC (rev 7745)
@@ -38,6 +38,7 @@
import java.util.UUID;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.CopyOnWriteArraySet;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
@@ -74,6 +75,7 @@
import com.bigdata.resources.IndexManager;
import com.bigdata.service.IBigdataFederation;
import com.bigdata.service.IDataService;
+import com.bigdata.util.InnerCause;
import com.bigdata.util.concurrent.DaemonThreadFactory;
import com.bigdata.util.concurrent.IHaltable;
@@ -1886,10 +1888,21 @@
*/
protected void halt(final AbstractRunningQuery q) {
+ boolean interrupted = false;
lock.lock();
try {
+ // notify listener(s)
+ try {
+ fireEvent(q);
+ } catch (Throwable t) {
+ if (InnerCause.isInnerCause(t, InterruptedException.class)) {
+ // Defer impact until outside of this critical section.
+ interrupted = true;
+ }
+ }
+
// insert/touch the LRU of recently finished queries.
doneQueries.put(q.getQueryId(), q.getFuture());
@@ -1909,6 +1922,9 @@
}
+ if (interrupted)
+ Thread.currentThread().interrupt();
+
}
/**
@@ -1923,7 +1939,8 @@
* @throws RuntimeException
* if the query halted with an error.
*/
- private void handleDoneQuery(final UUID queryId,final Future<Void> doneQueryFuture) {
+ private void handleDoneQuery(final UUID queryId,
+ final Future<Void> doneQueryFuture) {
try {
// Check the Future.
doneQueryFuture.get();
@@ -1945,6 +1962,90 @@
}
}
+ /**
+ * Listener API for {@link IRunningQuery} life cycle events (start/halt).
+ * <p>
+ * Note: While this interface makes it possible to catch the start and halt
+ * of an {@link IRunningQuery}, it imposes an overhead on the query engine
+ * and the potential for significant latency and other problems depending on
+ * the behavior of the {@link IRunningQueryListener}. This interface was
+ * added to facilitate certain test suites which could not otherwise be
+ * written. It should not be used for protection code.
+ */
+ public interface IRunningQueryListener {
+
+ void notify(IRunningQuery q);
+
+ }
+
+ /** Registered listeners. */
+ private final CopyOnWriteArraySet<IRunningQueryListener> listeners = new CopyOnWriteArraySet<IRunningQueryListener>();
+
+ /** Add a query listener. */
+ public void addListener(final IRunningQueryListener l) {
+
+ if (l == null)
+ throw new IllegalArgumentException();
+
+ listeners.add(l);
+
+ }
+
+ /** Remove a query listener. */
+ public void removeListener(final IRunningQueryListener l) {
+
+ if (l == null)
+ throw new IllegalArgumentException();
+
+ listeners.remove(l);
+
+ }
+
+ /**
+ * Send an event to all registered listeners.
+ */
+ private void fireEvent(final IRunningQuery q) {
+
+ if (q == null)
+ throw new IllegalArgumentException();
+
+ if(listeners.isEmpty()) {
+
+ // NOP
+ return;
+
+ }
+
+ final IRunningQueryListener[] a = listeners
+ .toArray(new IRunningQueryListener[0]);
+
+ for (IRunningQueryListener l : a) {
+
+ final IRunningQueryListener listener = l;
+
+ try {
+
+ // send event.
+ listener.notify(q);
+
+ } catch (Throwable t) {
+
+ if (InnerCause.isInnerCause(t, InterruptedException.class)) {
+
+ // Propagate interrupt.
+ throw new RuntimeException(t);
+
+ }
+
+ // Log and ignore.
+ log.error(t, t);
+
+ }
+
+ }
+
+ }
+
/*
* RunningQuery factory.
*/
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java 2014-01-07 18:30:50 UTC (rev 7744)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java 2014-01-07 18:34:06 UTC (rev 7745)
@@ -28,6 +28,7 @@
import java.util.Arrays;
import java.util.Collections;
+import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.LinkedList;
@@ -39,11 +40,14 @@
import com.bigdata.bop.BOp;
import com.bigdata.bop.BOpEvaluationContext;
import com.bigdata.bop.BOpIdFactory;
+import com.bigdata.bop.BadBOpIdTypeException;
+import com.bigdata.bop.DuplicateBOpIdException;
import com.bigdata.bop.IConstraint;
import com.bigdata.bop.IPredicate;
import com.bigdata.bop.IValueExpression;
import com.bigdata.bop.IVariable;
import com.bigdata.bop.NV;
+import com.bigdata.bop.NoBOpIdException;
import com.bigdata.bop.PipelineOp;
import com.bigdata.bop.ap.Predicate;
import com.bigdata.bop.ap.SampleIndex.SampleType;
@@ -97,6 +101,19 @@
*/
public class AST2BOpRTO extends AST2BOpJoins {
+ public interface Annotations extends AST2BOpJoins.Annotations {
+
+ /**
+ * Annotation is used to tag the {@link StatementPatternNode}s in a
+ * {@link JoinGroupNode} with the identified assigned to the
+ * corresponding {@link IPredicate}. This makes it possible to lookup
+ * the SPO from the {@link IPredicate} when the RTO hands us back an
+ * ordered join path.
+ */
+ String PREDICATE_ID = "PredicateId";
+
+ }
+
/**
* When <code>true</code>, the RTO will only accept simple joins into the
* join graph. Simple joins includes triples-mode joins and filters that do
@@ -222,7 +239,7 @@
// final Set<StatementPatternNode> sps = new LinkedHashSet<StatementPatternNode>();
// The predicates for the join graph.
@SuppressWarnings("rawtypes")
- final Set<Predicate> preds = new LinkedHashSet<Predicate>();
+ final LinkedList<Predicate> preds = new LinkedList<Predicate>();
// The constraints for the join graph.
final List<IConstraint> constraints = new LinkedList<IConstraint>();
// The #of JOINs accepted into the RTO's join group.
@@ -296,7 +313,6 @@
// Something the RTO can handle.
sp = (StatementPatternNode) sp.clone();// TODO Use destructive move.
-// sp.setId(ctx.nextId()); // assign id so we can reference back later.
rtoJoinGroup.addChild(sp); // add to group.
naccepted++;
/*
@@ -310,17 +326,24 @@
* when we take the selected join path from the RTO and
* compile it into a query plan to fully execute the join
* group.
+ *
+ * Note: This assigns ids to the predicates as a
+ * side-effect. Those ids are assigned by the
+ * AST2BOpContext's BOpIdFactory. You can not rely on
+ * specific ID values being assigned, so you need to build a
+ * map and track the correspondence between the SPs and the
+ * predicates.
*/
- final Predicate<?> pred = AST2BOpUtility.toPredicate(sp, ctx);
- // final int joinId = ctx.nextId();
- //
- // // annotations for this join.
- // final List<NV> anns = new LinkedList<NV>();
- //
- // anns.add(new NV(BOp.Annotations.BOP_ID, joinId));
+ final Predicate<?> pred = AST2BOpUtility.toPredicate(sp,
+ ctx);
preds.add(pred);
+ // tag the SP with predicate's ID.
+ sp.setProperty(Annotations.PREDICATE_ID, pred.getId());
if (attachedConstraints != null) {
- // RTO will figure out where to attach these constraints.
+ /*
+ * The RTO will figure out where to attach these
+ * constraints.
+ */
constraints.addAll(attachedConstraints);
}
@@ -492,8 +515,8 @@
final JoinGroupNode rtoJoinGroup = (JoinGroupNode) joinGraph
.getRequiredProperty(JoinGraph.Annotations.JOIN_GROUP);
-// // Build an index over the bopIds in that JoinGroupNode.
-// final Map<Integer, BOp> index = getIndex(rtoJoinGroup);
+ // Build an index over the bopIds in that JoinGroupNode.
+ final Map<Integer, StatementPatternNode> index = getIndex(rtoJoinGroup);
// Factory avoids reuse of bopIds assigned to the predicates.
final BOpIdFactory idFactory = new BOpIdFactory();
@@ -530,34 +553,16 @@
final boolean optional = pred.isOptional();
- /*
- * Lookup the AST node for that predicate.
- *
- * Note: The predicates are assigned bopIds by the RTO starting with
- * ONE (1). Therefore we substract out ONE from the predicate's id
- * to find its index into the join group.
- *
- * TODO This assumes that the join group does not contain anything
- * other than the SPs for the predicates that we are using.,
- *
- * TODO HINTS: The Predicate's query hints should the hints for that
- * specific join (aks the SP or other type of IJoinNode), not the
- * hints for the JoinGroupNode or the JoinGraph operator. We could
- * just pass the AST nodes themselves from the JoinGroupNode. That
- * might make things easier, even if it make the query serialization
- * fatter on a cluster.
- */
-// final ASTBase astNode = (ASTBase) index.get(pred.getId());
+ // Lookup the AST node for that predicate.
+ final StatementPatternNode sp = index.get(pred.getId());
- final ASTBase astNode = (ASTBase) rtoJoinGroup.get(pred.getId() - 1);
-
left = join(left, //
pred, //
optional ? new LinkedHashSet<IVariable<?>>(doneSet)
: doneSet, //
attachedJoinConstraints == null ? null : Arrays
.asList(attachedJoinConstraints),//
- astNode.getQueryHints(),//
+ sp.getQueryHints(),//
ctx);
}
@@ -616,61 +621,60 @@
}
-// /**
-// * Return an index from the {@link BOp.Annotations#BOP_ID} to the
-// * {@link BOp}.
-// * <p>
-// * {@link BOp}s should form directed acyclic graphs, but this is not
-// * strictly enforced. The recursive traversal iterators declared by this
-// * class do not protect against loops in the operator tree. However,
-// * {@link #getIndex(BOp)} detects and report loops based on duplicate
-// * {@link Annotations#BOP_ID}s -or- duplicate {@link BOp} references.
-// *
-// * @param op
-// * A {@link BOp}.
-// *
-// * @return The index, which is immutable and thread-safe.
-// *
-// * @throws DuplicateBOpIdException
-// * if there are two or more {@link BOp}s having the same
-// * {@link Annotations#BOP_ID}.
-// * @throws BadBOpIdTypeException
-// * if the {@link Annotations#BOP_ID} is not an {@link Integer}.
-// * @throws NoBOpIdException
-// * if a {@link PipelineOp} does not have a
-// * {@link Annotations#BOP_ID}.
-// */
-// static private Map<Integer,BOp> getIndex(final JoinGroupNode op) {
-// if(op == null)
-// throw new IllegalArgumentException();
-// final LinkedHashMap<Integer, BOp> map = new LinkedHashMap<Integer, BOp>();
-// final Iterator<BOp> itr = op.argIterator();
-// while (itr.hasNext()) {
-// final BOp t = itr.next();
-//// if(!(t instanceof PipelineOp))
-//// throw new NotPipelineOpException(t.toString());
-// final Object x = t.getProperty(BOp.Annotations.BOP_ID);
-// if (x == null) {
-// throw new NoBOpIdException(t.toString());
-// }
-// if (!(x instanceof Integer)) {
-// throw new BadBOpIdTypeException("Must be Integer, not: "
-// + x.getClass() + ": " + BOp.Annotations.BOP_ID);
-// }
-// final Integer id = (Integer) t.getProperty(BOp.Annotations.BOP_ID);
-// final BOp conflict = map.put(id, t);
-// if (conflict != null) {
-// /*
-// * BOp appears more than once. This is not allowed for
-// * pipeline operators. If you are getting this exception for
-// * a non-pipeline operator, you should remove the bopId.
-// */
-// throw new DuplicateBOpIdException("duplicate id=" + id
-// + " for " + conflict + " and " + t);
-// }
-// }
-// // wrap to ensure immutable and thread-safe.
-// return Collections.unmodifiableMap(map);
-// }
+ /**
+ * Return a map from the {@link Annotations#PREDICATE_ID} to the
+ * corresponding {@link StatementPatternNode}.
+ *
+ * @param op
+ * The join group.
+ *
+ * @return The index, which is immutable and thread-safe.
+ *
+ * @throws DuplicateBOpIdException
+ * if there are two or more {@link BOp}s having the same
+ * {@link Annotations#PREDICATE_ID}.
+ * @throws BadBOpIdTypeException
+ * if the {@link Annotations#PREDICATE_ID} is not an
+ * {@link Integer}.
+ * @throws NoBOpIdException
+ * if a {@link StatementPatternNode} does not have a
+ * {@link Annotations#PREDICATE_ID}.
+ */
+ static private Map<Integer, StatementPatternNode> getIndex(
+ final JoinGroupNode op) {
+ if (op == null)
+ throw new IllegalArgumentException();
+ final LinkedHashMap<Integer, StatementPatternNode> map = new LinkedHashMap<Integer, StatementPatternNode>();
+ final Iterator<IGroupMemberNode> itr = op.iterator();
+ while (itr.hasNext()) {
+ final BOp t = itr.next();
+ if(!(t instanceof StatementPatternNode)) {
+ // Skip non-SP nodes.
+ continue;
+ }
+ final StatementPatternNode sp = (StatementPatternNode) t;
+ final Object x = t.getProperty(Annotations.PREDICATE_ID);
+ if (x == null) {
+ throw new NoBOpIdException(t.toString());
+ }
+ if (!(x instanceof Integer)) {
+ throw new BadBOpIdTypeException("Must be Integer, not: "
+ + x.getClass() + ": " + Annotations.PREDICATE_ID);
+ }
+ final Integer id = (Integer) x;
+ final BOp conflict = map.put(id, sp);
+ if (conflict != null) {
+ /*
+ * BOp appears more than once. This is not allowed for
+ * pipeline operators. If you are getting this exception for
+ * a non-pipeline operator, you should remove the bopId.
+ */
+ throw new DuplicateBOpIdException("duplicate id=" + id
+ + " for " + conflict + " and " + t);
+ }
+ }
+ // wrap to ensure immutable and thread-safe.
+ return Collections.unmodifiableMap(map);
+ }
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/AbstractDataAndSPARQLTestCase.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/AbstractDataAndSPARQLTestCase.java 2014-01-07 18:30:50 UTC (rev 7744)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/AbstractDataAndSPARQLTestCase.java 2014-01-07 18:34:06 UTC (rev 7745)
@@ -83,16 +83,26 @@
public abstract class AbstractDataAndSPARQLTestCase extends AbstractASTEvaluationTestCase {
+ public AbstractDataAndSPARQLTestCase() {
+ }
+
+ public AbstractDataAndSPARQLTestCase(final String name) {
+ super(name);
+ }
+
public class AbsHelper {
protected final String queryStr;
+
/**
* This is the astContainer of the last query executed.
*/
protected ASTContainer astContainer;
- public AbsHelper(String queryStr) {
- this.queryStr = queryStr;
+ public AbsHelper(final String queryStr) {
+
+ this.queryStr = queryStr;
+
}
protected AbstractTripleStore getTripleStore() {
@@ -101,17 +111,34 @@
}
- protected void compareTupleQueryResults(final TupleQueryResult queryResult, final TupleQueryResult expectedResult, final boolean checkOrder)
- throws QueryEvaluationException {
- AbstractQueryEngineTestCase.compareTupleQueryResults(getName(),
- "", store, astContainer, queryResult, expectedResult,
- false, checkOrder);
- }
+ protected void compareTupleQueryResults(
+ final TupleQueryResult queryResult,
+ final TupleQueryResult expectedResult, final boolean checkOrder)
+ throws QueryEvaluationException {
+ AbstractQueryEngineTestCase.compareTupleQueryResults(getName(), "",
+ store, astContainer, queryResult, expectedResult, false,
+ checkOrder);
+
+ }
- long loadData(final InputStream is, RDFFormat format, String uri) {
- final RDFParser rdfParser = RDFParserRegistry.getInstance().get(format).getParser();
+ /**
+ * Load data from an input stream.
+ *
+ * @param is
+ * The stream (required).
+ * @param format
+ * The format (required).
+ * @param uri
+ * The baseURL (required).
+ * @return The #of triples read from the stream.
+ */
+ long loadData(final InputStream is, final RDFFormat format,
+ final String uri) {
+ final RDFParser rdfParser = RDFParserRegistry.getInstance()
+ .get(format).getParser();
+
rdfParser.setValueFactory(store.getValueFactory());
rdfParser.setVerifyData(true);
@@ -122,7 +149,12 @@
final AddStatementHandler handler = new AddStatementHandler();
- handler.setContext(new URIImpl(uri));
+ if (getTripleStore().isQuads()) {
+
+ // Set the default context.
+ handler.setContext(new URIImpl(uri));
+
+ }
rdfParser.setRDFHandler(handler);
@@ -170,7 +202,7 @@
public AddStatementHandler() {
- buffer = new StatementBuffer<Statement>(store, 100/* capacity */);
+ buffer = new StatementBuffer<Statement>(store, 1000/* capacity */);
}
@@ -180,6 +212,7 @@
}
+ @Override
public void handleStatement(final Statement stmt)
throws RDFHandlerException {
@@ -214,11 +247,4 @@
}
- public AbstractDataAndSPARQLTestCase() {
- }
-
- public AbstractDataAndSPARQLTestCase(String name) {
- super(name);
- }
-
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/AbstractDataDrivenSPARQLTestCase.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/AbstractDataDrivenSPARQLTestCase.java 2014-01-07 18:30:50 UTC (rev 7744)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/AbstractDataDrivenSPARQLTestCase.java 2014-01-07 18:34:06 UTC (rev 7745)
@@ -71,13 +71,11 @@
import java.net.URL;
import java.util.LinkedHashSet;
import java.util.Set;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.ZipInputStream;
import org.apache.log4j.Logger;
-import org.openrdf.model.Resource;
import org.openrdf.model.Statement;
-import org.openrdf.model.URI;
-import org.openrdf.model.Value;
-import org.openrdf.model.impl.URIImpl;
import org.openrdf.query.GraphQueryResult;
import org.openrdf.query.QueryEvaluationException;
import org.openrdf.query.TupleQueryResult;
@@ -90,23 +88,15 @@
import org.openrdf.query.resultio.TupleQueryResultFormat;
import org.openrdf.query.resultio.TupleQueryResultParser;
import org.openrdf.rio.RDFFormat;
-import org.openrdf.rio.RDFHandlerException;
import org.openrdf.rio.RDFParser;
import org.openrdf.rio.RDFParser.DatatypeHandling;
-import org.openrdf.rio.RDFParserFactory;
-import org.openrdf.rio.RDFParserRegistry;
import org.openrdf.rio.Rio;
-import org.openrdf.rio.helpers.RDFHandlerBase;
import org.openrdf.rio.helpers.StatementCollector;
import com.bigdata.bop.engine.AbstractQueryEngineTestCase;
-import com.bigdata.rdf.model.StatementEnum;
-import com.bigdata.rdf.rio.StatementBuffer;
import com.bigdata.rdf.sail.sparql.Bigdata2ASTSPARQLParser;
import com.bigdata.rdf.sparql.ast.ASTContainer;
-import com.bigdata.rdf.sparql.ast.AbstractASTEvaluationTestCase;
import com.bigdata.rdf.sparql.ast.QueryRoot;
-import com.bigdata.rdf.sparql.ast.eval.AbstractDataAndSPARQLTestCase.AbsHelper;
import com.bigdata.rdf.store.AbstractTripleStore;
/**
@@ -231,8 +221,9 @@
final String[] dataFileURLs, final String resultFileURL,
final boolean checkOrder)
throws Exception {
- super(getResourceAsString(queryFileURL));
+ super(getResourceAsString(queryFileURL));
+
if (log.isInfoEnabled())
log.info("\ntestURI:\n" + testURI);
@@ -527,12 +518,64 @@
*/
protected long loadData(final String resource) {
- return loadData(getResourceAsStream(resource), RDFFormat.forFileName(resource), new File(resource).toURI().toString());
+ if (log.isInfoEnabled())
+ log.info("Loading " + resource);
+ final String baseURL = new File(resource).toURI().toString();
+
+ InputStream is = null;
+ try {
+
+ is = getResourceAsStream(resource);
+
+ final RDFFormat rdfFormat = RDFFormat.forFileName(resource);
+
+ if (rdfFormat == null)
+ throw new RuntimeException("Unknown format: resource="
+ + resource);
+
+ // final RDFFormat rdfFormat = guessFormat(new File(resource),
+ // null/* default */);
+
+ return loadData(is, rdfFormat, baseURL);
+
+ } finally {
+ if (is != null) {
+ try {
+ is.close();
+ } catch (IOException e) {
+ log.error("Could not close: resource=" + resource, e);
+ }
+ is = null;
+ }
+ }
+
+
}
}
+// private static RDFFormat guessFormat(final File file,
+// final RDFFormat defaultFormat) {
+//
+// final String n = file.getName();
+//
+// RDFFormat fmt = RDFFormat.forFileName(n);
+//
+// if (fmt == null && n.endsWith(".zip")) {
+// fmt = RDFFormat.forFileName(n.substring(0, n.length() - 4));
+// }
+//
+// if (fmt == null && n.endsWith(".gz")) {
+// fmt = RDFFormat.forFileName(n.substring(0, n.length() - 3));
+// }
+//
+// if (fmt == null) // fallback
+// fmt = defaultFormat;
+//
+// return fmt;
+//
+// }
private static InputStream getResourceAsStream(final String resource) {
@@ -595,6 +638,20 @@
if (is == null)
throw new RuntimeException("Not found: " + resource);
+ if (resource.toLowerCase().endsWith(".gz")) {
+
+ try {
+ is = new GZIPInputStream(is);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ } else if (resource.toLowerCase().endsWith(".zip")) {
+
+ is = new ZipInputStream(is);
+
+ }
+
return is;
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <jer...@us...> - 2014-01-07 22:10:41
|
Revision: 7746
http://bigdata.svn.sourceforge.net/bigdata/?rev=7746&view=rev
Author: jeremy_carroll
Date: 2014-01-07 22:10:34 +0000 (Tue, 07 Jan 2014)
Log Message:
-----------
Added tests (both runtime and static optimizer) for trac 794 to do with SERVICE and BIND
Also modified comment in ASTJoinOrderByTypeOptimizer which is doing the wrong thing.
Tests currently disabled since the fix is not yet in. (Note the fix is non-trivial, but not conceptually hard either)
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTBindingAssigner.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTJoinOrderByTypeOptimizer.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/optimizers/AbstractOptimizerTestCase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/AbstractProtocolTest.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/TestNanoSparqlServerWithProxyIndexManager.java
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/optimizers/TestASTMassagedServiceNodeOptimizer.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/TestService794.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTBindingAssigner.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTBindingAssigner.java 2014-01-07 18:34:06 UTC (rev 7745)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTBindingAssigner.java 2014-01-07 22:10:34 UTC (rev 7746)
@@ -73,7 +73,7 @@
public IQueryNode optimize(final AST2BOpContext context,
final IQueryNode queryNode, final IBindingSet[] bindingSet) {
- if (bindingSet == null || bindingSet.length > 1) {
+ if (bindingSet == null || bindingSet.length != 1) {
/*
* Used iff there is only one input solution.
*
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTJoinOrderByTypeOptimizer.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTJoinOrderByTypeOptimizer.java 2014-01-07 18:34:06 UTC (rev 7745)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTJoinOrderByTypeOptimizer.java 2014-01-07 22:10:34 UTC (rev 7746)
@@ -183,17 +183,20 @@
*
* Required joins:
*
- * 3. Service calls (Bigdata SEARCH)
+ * 3. Some Service calls (e.g. Bigdata SEARCH)
* 4. Subquery-includes
* 5. Statement patterns
* 7. Sparql11 subqueries
* 8. Non-optional subgroups
+ * 9. Other service calls
*
+ * TODO: the placement of OPTIONALS should really be more complicated than this.
+ * e.g. consider interaction with SERVICE calls etc.
* Optional joins:
- * 9. Simple optionals & optional subgroups
+ * 10. Simple optionals & optional subgroups
*
- * 10. Assignments
- * 11. Post-conditionals
+ * 11. Assignments
+ * 12. Post-conditionals
* </pre>
* Most of this logic was lifted out of {@link AST2BOpUtility}.
* <p>
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/optimizers/AbstractOptimizerTestCase.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/optimizers/AbstractOptimizerTestCase.java 2014-01-07 18:34:06 UTC (rev 7745)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/optimizers/AbstractOptimizerTestCase.java 2014-01-07 22:10:34 UTC (rev 7746)
@@ -64,6 +64,7 @@
import com.bigdata.rdf.sparql.ast.UnionNode;
import com.bigdata.rdf.sparql.ast.VarNode;
import com.bigdata.rdf.sparql.ast.eval.AST2BOpContext;
+import com.bigdata.rdf.sparql.ast.service.ServiceNode;
public abstract class AbstractOptimizerTestCase extends AbstractASTEvaluationTestCase {
@@ -426,6 +427,18 @@
protected FilterNode filter(IValueExpressionNode f) {
return new FilterNode(f);
}
+
+ protected IValueExpressionNode functionNode(String uri, ValueExpressionNode ... args) {
+ return new FunctionNode(new URIImpl(uri), null, args);
+ }
+
+ protected ServiceNode service(TermNode serviceRef, GraphPatternGroup<IGroupMemberNode> groupNode) {
+ return new ServiceNode(serviceRef, groupNode);
+ }
+
+ protected AssignmentNode bind(IValueExpressionNode valueNode, VarNode varNode) {
+ return new AssignmentNode(varNode, valueNode);
+ }
}
public AbstractOptimizerTestCase(String name) {
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/optimizers/TestASTMassagedServiceNodeOptimizer.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/optimizers/TestASTMassagedServiceNodeOptimizer.java (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/optimizers/TestASTMassagedServiceNodeOptimizer.java 2014-01-07 22:10:34 UTC (rev 7746)
@@ -0,0 +1,104 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2014. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+
+/**
+ * Trac 794 concerns interactions between BIND and SERVICE going to a remote
+ * SPARQL end-point. The service call must be done last.
+ */
+package com.bigdata.rdf.sparql.ast.optimizers;
+
+
+
+
+public class TestASTMassagedServiceNodeOptimizer extends AbstractOptimizerTestCase {
+
+ public TestASTMassagedServiceNodeOptimizer(String name) {
+ super(name);
+ }
+
+ public TestASTMassagedServiceNodeOptimizer() {
+ }
+ @Override
+ IASTOptimizer newOptimizer() {
+ return new ASTJoinOrderByTypeOptimizer();
+ }
+
+ public void testLeaveBindBeforeService() {
+
+ new Helper(){{
+
+ given = select( varNode(z),
+ where (
+ joinGroupNode(
+ statementPatternNode(varNode(x), constantNode(c), constantNode(d)),
+ bind(functionNode("eg:foo", varNode(x)), varNode(y) ),
+ service( constantNode(a),
+ joinGroupNode( statementPatternNode(varNode(z), constantNode(f), varNode(y)) ) )
+ )
+ ) );
+
+
+ expected = select( varNode(z),
+ where (
+ joinGroupNode(
+ statementPatternNode(varNode(x), constantNode(c), constantNode(d)),
+ bind(functionNode("eg:foo", varNode(x)), varNode(y) ),
+ service( constantNode(a),
+ joinGroupNode( statementPatternNode(varNode(z), constantNode(f), varNode(y)) ) )
+ )
+ ) );
+
+
+ }}.test();
+ }
+ public void testPutBindBeforeService() {
+
+ new Helper(){{
+
+ given = select( varNode(z),
+ where (
+ joinGroupNode(
+ service( constantNode(a),
+ joinGroupNode( statementPatternNode(varNode(z), constantNode(f), varNode(y)) ) ),
+ statementPatternNode(varNode(x), constantNode(c), constantNode(d)),
+ bind(functionNode("eg:foo", varNode(x)), varNode(y) )
+ )
+ ) );
+
+
+ expected = select( varNode(z),
+ where (
+ joinGroupNode(
+ statementPatternNode(varNode(x), constantNode(c), constantNode(d)),
+ bind(functionNode("eg:foo", varNode(x)), varNode(y) ),
+ service( constantNode(a),
+ joinGroupNode( statementPatternNode(varNode(z), constantNode(f), varNode(y)) ) )
+ )
+ ) );
+
+
+ }}.test();
+ }
+
+}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/AbstractProtocolTest.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/AbstractProtocolTest.java 2014-01-07 18:34:06 UTC (rev 7745)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/AbstractProtocolTest.java 2014-01-07 22:10:34 UTC (rev 7746)
@@ -114,6 +114,10 @@
};
private RequestFactory requestFactory = GET;
+
+ protected RequestFactory getRequestFactory() {
+ return requestFactory;
+ }
@Override
public void setUp() throws Exception {
super.setUp();
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/TestNanoSparqlServerWithProxyIndexManager.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/TestNanoSparqlServerWithProxyIndexManager.java 2014-01-07 18:34:06 UTC (rev 7745)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/TestNanoSparqlServerWithProxyIndexManager.java 2014-01-07 22:10:34 UTC (rev 7746)
@@ -233,6 +233,8 @@
suite.addTestSuite(TestInsertFilterFalse727.class);
suite.addTestSuite(TestCBD731.class);
+ suite.addTestSuite(TestService794.class);
+
// SPARQL UPDATE test suite.
switch(testMode) {
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/TestService794.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/TestService794.java (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/TestService794.java 2014-01-07 22:10:34 UTC (rev 7746)
@@ -0,0 +1,180 @@
+/**
+Copyright (C) SYSTAP, LLC 2014. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
+package com.bigdata.rdf.sail.webapp;
+
+import java.io.IOException;
+
+import junit.framework.Test;
+
+
+/**
+ * While writing this comment, early January 2014,
+ * the status is that the two interesting tests
+ * both fail and are disabled
+ * {@link #xtestMassageServiceCall()} and {@link #xtestMassageServiceNested1Call()}
+ *
+ * Also {@link #xtestServiceSyntaxError()} shows some bug some where in that
+ * we take legal SPARQL and make it illegal before the service call ....
+ *
+ * Some of the other tests show how to use a subselect as a workaround.
+ * @author jeremycarroll
+ *
+ */
+public class TestService794 extends AbstractProtocolTest {
+
+ public TestService794(String name) {
+ super(name);
+ }
+
+
+ static public Test suite() {
+ return ProxySuiteHelper.suiteWhenStandalone(TestService794.class,"test.*",
+ TestMode.quads,TestMode.sids,
+ TestMode.triples);
+ }
+
+ /**
+ * Execute an ASK query including a SERVICE keyword which gets sent back to this server.
+ * The test succeeeds if the query returns true, and fails otherwise
+ * @param args
+ * @throws IOException
+ */
+ private void abstactAskService(String ... args) throws IOException {
+
+ setMethodisPostUrlEncodedData();
+ serviceRequest("update","PREFIX eg: <http://example.com/a#> INSERT { eg:a eg:p \"rs123\" ; eg:q 123, 100 } WHERE {}");
+
+ StringBuilder bld = new StringBuilder();
+ // Set the base URI to be our sparql end point, for re-entrant queries,
+ // using the idiom SERVICE <>
+ bld.append("base <");
+ bld.append(m_serviceURL);
+ bld.append("/sparql>");
+ for (String arg:args) {
+ bld.append('\n');
+ bld.append(arg);
+ }
+
+ System.err.println(bld.toString());
+ String result = serviceRequest("query",bld.toString());
+ System.err.println(result);
+ assertTrue(result.contains("true"));
+
+ }
+
+ /**
+ * @throws IOException
+ */
+ public void testSimpleServiceCall() throws IOException {
+ abstactAskService("PREFIX eg: <http://example.com/a#>",
+ "ASK {",
+ "?x eg:p ?y ",
+ " SERVICE <> {",
+ " FILTER ( true )",
+ "{ SELECT ?x ?y {",
+ "?x eg:p ?y ",
+ "} ORDER BY ?y LIMIT 1 }",
+ "} }");
+ }
+
+ /**
+ * This one is currently broken, see trac794
+ *
+ * Note also there is something unintersting with syntax
+ * going wrong with some expressions like
+ * SERVICE <> {
+ * { SELECT * {
+ * ?x eg:q ?y
+ * }
+ * }
+ * }
+ */
+ public void xtestMassageServiceCall() throws IOException {
+ abstactAskService("PREFIX eg: <http://example.com/a#>",
+ "prefix xsd: <http://www.w3.org/2001/XMLSchema#>",
+ "ASK {",
+ "?x eg:p ?y ",
+ "BIND (xsd:integer(substr(?y,3)) as ?yy )",
+ " SERVICE <> {",
+ " FILTER (true )",
+ "{ SELECT ?x ?yy {",
+ "?x eg:q ?yy ",
+ "} ORDER BY ?yy LIMIT 1 }",
+ "} }");
+ }
+ /**
+ * @throws IOException
+ */
+ public void xtestMassageServiceNested1Call() throws IOException {
+ abstactAskService("PREFIX eg: <http://example.com/a#>",
+ "prefix xsd: <http://www.w3.org/2001/XMLSchema#>",
+ "ASK {",
+ "{ ?x eg:p ?y ",
+ "BIND (xsd:integer(substr(?y,3)) as ?yy ) }",
+ " SERVICE <> {",
+ "{ SELECT ?x ?yy {",
+ "?x eg:q ?yy ",
+ "} ORDER BY ?yy LIMIT 1 }",
+ "} }");
+ }
+ /**
+ * @throws IOException
+ */
+ public void testMassageServiceNested2Call() throws IOException {
+ abstactAskService("PREFIX eg: <http://example.com/a#>",
+ "prefix xsd: <http://www.w3.org/2001/XMLSchema#>",
+ "ASK {",
+ "{ SELECT ?x ?yy ",
+ " { ?x eg:p ?y ",
+ " BIND (xsd:integer(substr(?y,3)) as ?yy ) } }",
+ " SERVICE <> {",
+ "{ SELECT ?x ?yy {",
+ "?x eg:q ?yy ",
+ "} ORDER BY ?yy LIMIT 1 }",
+ "} }");
+ }
+ public void testMassageServiceNested3Call() throws IOException {
+ abstactAskService("PREFIX eg: <http://example.com/a#>",
+ "prefix xsd: <http://www.w3.org/2001/XMLSchema#>",
+ "ASK {",
+ "{ SELECT ?x (xsd:integer(substr(?y,3)) as ?yy ) ",
+ " { ?x eg:p ?y } }",
+ " SERVICE <> {",
+ "{ SELECT ?x ?yy {",
+ "?x eg:q ?yy ",
+ "} ORDER BY ?yy LIMIT 1 }",
+ "} }");
+ }
+ public void xtestServiceSyntaxError() throws IOException {
+ abstactAskService("PREFIX eg: <http://example.com/a#>",
+ "prefix xsd: <http://www.w3.org/2001/XMLSchema#>",
+ "ASK {",
+ "{ SELECT ?x (xsd:integer(substr(?y,3)) as ?yy ) ",
+ " { ?x eg:p ?y } }",
+ " SERVICE <> {",
+ "{ SELECT * {",
+ "?x eg:q ?yy ",
+ "} ORDER BY ?yy LIMIT 1 }",
+ "} }");
+ }
+}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-08 17:58:09
|
Revision: 7751
http://bigdata.svn.sourceforge.net/bigdata/?rev=7751&view=rev
Author: thompsonbry
Date: 2014-01-08 17:58:02 +0000 (Wed, 08 Jan 2014)
Log Message:
-----------
Added a hidden option to enable the RTO.
Changed the way in which the analytic option recognition works. Verified correctly works in sample queries.
See #64 (RTO.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/BigdataRDFContext.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/index.html
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/BigdataRDFContext.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/BigdataRDFContext.java 2014-01-08 17:56:54 UTC (rev 7750)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/BigdataRDFContext.java 2014-01-08 17:58:02 UTC (rev 7751)
@@ -101,6 +101,7 @@
import com.bigdata.rdf.sail.sparql.Bigdata2ASTSPARQLParser;
import com.bigdata.rdf.sparql.ast.ASTContainer;
import com.bigdata.rdf.sparql.ast.QueryHints;
+import com.bigdata.rdf.sparql.ast.QueryOptimizerEnum;
import com.bigdata.rdf.sparql.ast.QueryRoot;
import com.bigdata.rdf.sparql.ast.QueryType;
import com.bigdata.rdf.sparql.ast.Update;
@@ -145,6 +146,12 @@
protected static final String ANALYTIC = "analytic";
/**
+ * URL Query parameter used to request the use of the Runtime Query
+ * Optimizer.
+ */
+ protected static final String RTO = "RTO";
+
+ /**
* URL Query parameter used to request an XHTML response for SPARQL
* QUERY or SPARQL UPDATE. For SPARQL QUERY, this provides an XHTML
* table view of the solutions. For SPARQL UPDATE, this provides an
@@ -612,9 +619,14 @@
/**
* When <code>true</code>, enable the "analytic" query hints.
*/
- final Boolean analytic;
+ final boolean analytic;
/**
+ * When <code>true</code>, enable the Runtime Query Optimizer.
+ */
+ final boolean rto;
+
+ /**
* When <code>true</code>, provide an view of the XHTML representation
* of the solutions or graph result (SPARQL QUERY)
*
@@ -751,6 +763,9 @@
this.explainDetails = explain && isExplainDetails(req);
this.analytic = getEffectiveBooleanValue(
req.getParameter(ANALYTIC), QueryHints.DEFAULT_ANALYTIC);
+ this.rto = getEffectiveBooleanValue(req.getParameter(RTO),
+ QueryHints.DEFAULT_OPTIMIZER
+ .equals(QueryOptimizerEnum.Runtime));
this.xhtml = getEffectiveBooleanValue(req.getParameter(XHTML),
false);
this.monitor = getEffectiveBooleanValue(req.getParameter(MONITOR),
@@ -822,6 +837,9 @@
this.explainDetails = explain && isExplainDetails(req);
this.analytic = getEffectiveBooleanValue(
req.getParameter(ANALYTIC), QueryHints.DEFAULT_ANALYTIC);
+ this.rto = getEffectiveBooleanValue(req.getParameter(RTO),
+ QueryHints.DEFAULT_OPTIMIZER
+ .equals(QueryOptimizerEnum.Runtime));
this.xhtml = getEffectiveBooleanValue(req.getParameter(XHTML),
false);
this.monitor = getEffectiveBooleanValue(req.getParameter(MONITOR),
@@ -898,13 +916,18 @@
// Override query if data set protocol parameters were used.
overrideDataset(query);
- if (analytic != null) {
+ if (analytic) {
// Turn analytic query on/off as requested.
-// astContainer.getOriginalAST().setQueryHint(QueryHints.ANALYTIC,
-// analytic.toString());
- astContainer.setQueryHint(QueryHints.ANALYTIC,
- analytic.toString());
+ astContainer.setQueryHint(QueryHints.ANALYTIC, "true");
+
+ }
+
+ if (rto) {
+
+ // Turn analytic query on/off as requested.
+ astContainer.setQueryHint(QueryHints.OPTIMIZER,
+ QueryOptimizerEnum.Runtime.toString());
}
@@ -951,13 +974,18 @@
// Override query if data set protocol parameters were used.
overrideDataset(update);
- if (analytic != null) {
+ if (analytic) {
// Turn analytic query on/off as requested.
-// astContainer.getOriginalAST().setQueryHint(QueryHints.ANALYTIC,
-// analytic.toString());
- astContainer.setQueryHint(QueryHints.ANALYTIC,
- analytic.toString());
+ astContainer.setQueryHint(QueryHints.ANALYTIC, "true");
+
+ }
+
+ if (rto) {
+
+ // Turn analytic query on/off as requested.
+ astContainer.setQueryHint(QueryHints.OPTIMIZER,
+ QueryOptimizerEnum.Runtime.toString());
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/index.html
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/index.html 2014-01-08 17:56:54 UTC (rev 7750)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/index.html 2014-01-08 17:58:02 UTC (rev 7751)
@@ -59,7 +59,12 @@
<INPUT type="checkbox" name="analytic" value="true"
title="Enable the analytic query package."
> Analytic
- <INPUT type="checkbox" name="xhtml" value="true"
+<!-- TODO Uncomment to reveal the RTO option.
+ <INPUT type="checkbox" name="RTO" value="true"
+ title="Enable the Runtime Query Optimizer (RTO)."
+ > RTO
+-->
+ <INPUT type="checkbox" name="xhtml" value="true"
title="Request XHTML response (results formatted as table)."
checked="checked"
> XHTML
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-09 19:24:41
|
Revision: 7753
http://bigdata.svn.sourceforge.net/bigdata/?rev=7753&view=rev
Author: thompsonbry
Date: 2014-01-09 19:24:27 +0000 (Thu, 09 Jan 2014)
Log Message:
-----------
This issue is closed.
Branch MGC_1_3_0 was created from r7608 of branches/BIGDATA_RELEASE_1_3_0.
This branch was used to make HA robust in the face of sudden process kills.
Merging MGC_1_3_0 back to the main development branch (branches/BIGDATA_RELEASE_1_3_0). Work on MGC_1_3_0 is closed out.
{{{
merge -r7608:HEAD https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/MGC_1_3_0 /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-jini/src/test/com/bigdata/quorum/zk/MockQuorumMember.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestAll.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-jini/src/test/com/bigdata/journal/jini/ha/AbstractHA3JournalServerTestCase.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-jini/src/test/com/bigdata/journal/jini/ha/HAJournalTest.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHAJournalServer.java
A /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA3JustKills.java
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-jini/src/java/com/bigdata/disco
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-jini/src/java/com/bigdata/disco
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-jini/src/java/com/bigdata/quorum/zk/ZKQuorumImpl.java
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-jini/src/java/com/bigdata/attr
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-jini/src/java/com/bigdata/attr
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournalServer.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournal.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-jini/src/java/com/bigdata/journal/jini/ha/AbstractServer.java
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-jini/src/java/com/bigdata/util/config
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-jini/src/java/com/bigdata/util/config
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-perf/lubm
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-perf/lubm
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-perf/uniprot/src
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-perf/uniprot/src
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-perf/uniprot
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-perf/uniprot
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-perf/btc/src/resources
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-perf/btc/src/resources
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-perf/btc
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-perf/btc
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-perf
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-perf
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/src/resources/bin/config
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/src/resources/bin/config
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/build.properties
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/build.xml
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-compatibility
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-compatibility
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils/LEGAL
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils/LEGAL
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils/lib
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils/lib
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils/src/test/it/unimi/dsi
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils/src/test/it/unimi/dsi
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils/src/test/it/unimi
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils/src/test/it/unimi
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils/src/test
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils/src/test
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils/src/java/it/unimi
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils/src/java/it/unimi
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils/src/java/it
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils/src/java/it
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils/src/java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils/src/java
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils/src
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils/src
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/dsi-utils
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-war/src/html/index.html
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/lib/jetty
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/lib/jetty
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/bop/joinGraph
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/bop/joinGraph/rto/TestJoinGraph.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/bop/joinGraph/rto/TestAll.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/bop/joinGraph/rto/TestJGraph.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/bop/joinGraph
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/bop/TestAll.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/bop/join/TestPipelineJoin.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/bop/join/AbstractHashJoinUtilityTestCase.java
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/bop/util
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/bop/util
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN
C /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/striterator/TestCloseableChunkedIteratorWrapperConverter.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/quorum/MockQuorumFixture.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/TestAll.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/ha/pipeline/TestHASendAndReceive3Nodes.java
A /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/ha/pipeline/TestSocketsDirect.java
A /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/ha/pipeline/AbstractHASendAndReceiveTestCase.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/ha/pipeline/TestHASendAndReceive.java
A /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/ha/msg/TestHASendState.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/ha/msg/TestAll.java
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/jsr166
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/jsr166
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/io/writecache/TestWORMWriteCacheService.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/journal/ha/HABranch.txt
D /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/journal/ha/TestHAWORMStrategy.java
D /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/journal/ha/AbstractHAJournalTestCase.java
D /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/journal/ha/TestJournalHA.java
D /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/journal/ha/TestAll.java
D /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/journal/ha/TestHAWritePipeline.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/journal/TestWORMStrategyNoCache.java
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/util/httpd
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/test/com/bigdata/util/httpd
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/striterator/IChunkedIterator.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/striterator/CloseableIteratorWrapper.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/striterator/MergeFilter.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/striterator/AbstractChunkedResolverator.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/striterator/ChunkedWrappedIterator.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/striterator/Striterator.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/striterator/ChunkedResolvingIterator.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/striterator/IChunkedStriterator.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/striterator/Resolver.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/striterator/PushbackIterator.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/striterator/GenericChunkedStriterator.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/striterator/Chunkerator.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/striterator/DelegateChunkedIterator.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/striterator/ChunkedOrderedStriterator.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/striterator/ChunkedArrayIterator.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/striterator/ChunkedConvertingIterator.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/striterator/Dechunkerator.java
C /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/striterator/CloseableChunkedIteratorWrapperConverter.java
C /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/quorum/AbstractQuorum.java
C /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/quorum/QuorumActor.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/quorum/QuorumClient.java
A /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/quorum/ServiceLookup.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/QuorumPipelineImpl.java
A /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/HAPipelineResetResponse.java
A /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/IHAPipelineResetRequest.java
A /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/IHAPipelineResetResponse.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/QuorumPipeline.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/pipeline/PipelineDownstreamChange.java
A /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/pipeline/PipelineImmediateDownstreamReplicationException.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/pipeline/HASendService.java
A /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/pipeline/NestedPipelineException.java
A /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/pipeline/AbstractPipelineException.java
A /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/pipeline/ImmediateDownstreamReplicationException.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/pipeline/HAReceiveService.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/pipeline/PipelineUpstreamChange.java
A /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/pipeline/AbstractPipelineChangeException.java
A /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/HAPipelineResetRequest.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/HAPipelineGlue.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/QuorumServiceBase.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/msg/HAWriteMessageBase.java
A /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/msg/IHASendState.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/msg/HAWriteMessage.java
A /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/msg/HASendState.java
A /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/msg/IHAMessageWrapper.java
A /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/msg/HAMessageWrapper.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/QuorumCommitImpl.java
A /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/ha/AbstractMessageTask.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/AbstractAccessPathOp.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/solutions/DropOp.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/solutions/ProjectionOp.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/solutions/GroupByRewriter.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/solutions/GroupByState.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/CoreBaseBOp.java
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/aggregate
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/aggregate
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/engine/BOpStats.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/engine/RunState.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/engine/AbstractRunningQuery.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/util
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/util
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/joinGraph
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/joinGraph/rto/PathIds.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/joinGraph/rto/SampleBase.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/joinGraph
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN
C /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/controller/INamedSubqueryOp.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/BOpBase.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/fed/ThickChunkMessage.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/fed/NIOChunkMessage.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/fed/EmptyChunkMessage.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/join/IHashJoinUtility.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/join/HashJoinOp.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/join/BaseJoinStats.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/join/NamedSolutionSetStats.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/join/JoinVariableNotBoundException.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/BOpUtility.java
C /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/bop/SimpleIdFactory.java
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/jsr166
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/jsr166
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/btree/data/DefaultNodeCoder.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/btree/data/DefaultLeafCoder.java
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/htree/raba
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/htree/raba
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/journal/Options.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/journal/Name2Addr.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/journal/AbstractJournal.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/relation/accesspath/AccessPath.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/relation/accesspath/MultiSourceSequentialCloseableIterator.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/relation/accesspath/IBindingSetAccessPath.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/util/NT.java
C /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/resources/deployment
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/lgpl-utils/src/test/it/unimi/dsi/fastutil/bytes/custom
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/lgpl-utils/src/test/it/unimi/dsi/fastutil/bytes/custom
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/lgpl-utils/src/java/it/unimi/dsi/fastutil/bytes/custom
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/lgpl-utils/src/java/it/unimi/dsi/fastutil/bytes/custom
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/osgi
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/osgi
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-sails/src/test/com/bigdata/rdf/sail/bench
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-sails/src/test/com/bigdata/rdf/sail/bench
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp
C /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/TestService794.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-sails/src/java/com/bigdata/rdf/sail/bench
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-sails/src/java/com/bigdata/rdf/sail/bench
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-sails/src/java/com/bigdata/rdf/sail/changesets
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-sails/src/java/com/bigdata/rdf/sail/changesets
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/test/com/bigdata/bop/rdf/aggregate
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/test/com/bigdata/bop/rdf/aggregate
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN
C /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/TestQueryHints.java
C /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/optimizers/TestASTMassagedServiceNodeOptimizer.java
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/test/com/bigdata/rdf/internal
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/test/com/bigdata/rdf/internal
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/test/com/bigdata/rdf/relation
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/test/com/bigdata/rdf/relation
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/bop/rdf/update/ParserStats.java
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/bop/rdf/aggregate
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/bop/rdf/aggregate
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/changesets
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/changesets
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/error
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/error
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/IQueryHint.java
C /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/RTONEdgesQueryHint.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AbstractChunkSizeHint.java
C /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/RTOLimitQueryHint.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/QueryHintRegistry.java
C /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/RTOSampleTypeQueryHint.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/GraphPatternGroup.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/NamedSubqueryInclude.java
C /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpFilters.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpBase.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpJoins.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/QueryHints.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/ASTBase.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTBindingAssigner.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/ValueExpressionListBaseNode.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/SliceNode.java
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/QueryOptimizerEnum.java
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/internal
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/internal
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/relation
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/relation
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/util
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/java/com/bigdata/rdf/util
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/samples
U /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/samples
--- Merging r7608 through r7752 into /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN
C /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata-rdf/src/resources/data/lehigh/LUBM-U1.rdf.gz
C /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN
Merge complete.
===== File Statistics: =====
Deleted: 5
Added: 19
Updated: 108
==== Property Statistics: =====
Updated: 49
==== Conflict Statistics: =====
File conflicts: 2
Property conflicts: 1
Tree conflicts: 13
}}}
There were TWO files with conflicts. In both cases, the incoming
version was accepted. Those files are:
{{{
C /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/quorum/AbstractQuorum.java
C /Users/bryan/Documents/workspace/BIGDATA_RELEASE_1_3_0_CLEAN/bigdata/src/java/com/bigdata/quorum/QuorumActor.java
}}}
There was also an SVN property conflict on svn:ignore. This was resolved by using the version from the main branch.
I also needed to go into the tree conflict view and "mark resolved" a number of files. Each of this was reported as "local add" and "incoming and upon merge". In all cases these were simply recently added files from the main branch.
See #779 (pipeline resynchronization protocol)
See #724 (HA wire pulls and sure kill tests)
Revision Links:
--------------
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7608&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7752&view=rev
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/HAPipelineGlue.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/QuorumCommitImpl.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/QuorumPipeline.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/QuorumPipelineImpl.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/QuorumServiceBase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/msg/HAWriteMessage.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/msg/HAWriteMessageBase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/pipeline/HAReceiveService.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/pipeline/HASendService.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/pipeline/PipelineDownstreamChange.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/pipeline/PipelineUpstreamChange.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/journal/AbstractJournal.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/quorum/AbstractQuorum.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/quorum/QuorumActor.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/quorum/QuorumClient.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/TestAll.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/ha/msg/TestAll.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/ha/pipeline/TestHASendAndReceive.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/ha/pipeline/TestHASendAndReceive3Nodes.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/io/writecache/TestWORMWriteCacheService.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/journal/TestWORMStrategyNoCache.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/journal/ha/HABranch.txt
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/quorum/MockQuorumFixture.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-jini/src/java/com/bigdata/journal/jini/ha/AbstractServer.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournal.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournalServer.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-jini/src/test/com/bigdata/journal/jini/ha/AbstractHA3JournalServerTestCase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-jini/src/test/com/bigdata/journal/jini/ha/HAJournalTest.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestAll.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHAJournalServer.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-jini/src/test/com/bigdata/quorum/zk/MockQuorumMember.java
branches/BIGDATA_RELEASE_1_3_0/build.xml
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/AbstractMessageTask.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/HAPipelineResetRequest.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/HAPipelineResetResponse.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/IHAPipelineResetRequest.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/IHAPipelineResetResponse.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/msg/HAMessageWrapper.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/msg/HASendState.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/msg/IHAMessageWrapper.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/msg/IHASendState.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/pipeline/AbstractPipelineChangeException.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/pipeline/AbstractPipelineException.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/pipeline/ImmediateDownstreamReplicationException.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/pipeline/NestedPipelineException.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/ha/pipeline/PipelineImmediateDownstreamReplicationException.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/quorum/ServiceLookup.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/ha/msg/TestHASendState.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/ha/pipeline/AbstractHASendAndReceiveTestCase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/ha/pipeline/TestSocketsDirect.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA3JustKills.java
Removed Paths:
-------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/journal/ha/AbstractHAJournalTestCase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/journal/ha/TestAll.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/journal/ha/TestHAWORMStrategy.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/journal/ha/TestHAWritePipeline.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/journal/ha/TestJournalHA.java
Property Changed:
----------------
branches/BIGDATA_RELEASE_1_3_0/
branches/BIGDATA_RELEASE_1_3_0/bigdata/lib/jetty/
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/aggregate/
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/util/
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/htree/raba/
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/jsr166/
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/joinGraph/
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/util/
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/jsr166/
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/util/httpd/
branches/BIGDATA_RELEASE_1_3_0/bigdata-compatibility/
branches/BIGDATA_RELEASE_1_3_0/bigdata-jini/src/java/com/bigdata/attr/
branches/BIGDATA_RELEASE_1_3_0/bigdata-jini/src/java/com/bigdata/disco/
branches/BIGDATA_RELEASE_1_3_0/bigdata-jini/src/java/com/bigdata/util/config/
branches/BIGDATA_RELEASE_1_3_0/bigdata-perf/
branches/BIGDATA_RELEASE_1_3_0/bigdata-perf/btc/
branches/BIGDATA_RELEASE_1_3_0/bigdata-perf/btc/src/resources/
branches/BIGDATA_RELEASE_1_3_0/bigdata-perf/lubm/
branches/BIGDATA_RELEASE_1_3_0/bigdata-perf/uniprot/
branches/BIGDATA_RELEASE_1_3_0/bigdata-perf/uniprot/src/
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/bop/rdf/aggregate/
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/changesets/
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/error/
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/relation/
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/util/
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/samples/
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/bop/rdf/aggregate/
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/internal/
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/relation/
branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/java/com/bigdata/rdf/sail/bench/
branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/java/com/bigdata/rdf/sail/changesets/
branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/
branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/bench/
branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/
branches/BIGDATA_RELEASE_1_3_0/dsi-utils/
branches/BIGDATA_RELEASE_1_3_0/dsi-utils/LEGAL/
branches/BIGDATA_RELEASE_1_3_0/dsi-utils/lib/
branches/BIGDATA_RELEASE_1_3_0/dsi-utils/src/
branches/BIGDATA_RELEASE_1_3_0/dsi-utils/src/java/
branches/BIGDATA_RELEASE_1_3_0/dsi-utils/src/java/it/
branches/BIGDATA_RELEASE_1_3_0/dsi-utils/src/java/it/unimi/
branches/BIGDATA_RELEASE_1_3_0/dsi-utils/src/test/
branches/BIGDATA_RELEASE_1_3_0/dsi-utils/src/test/it/unimi/
branches/BIGDATA_RELEASE_1_3_0/dsi-utils/src/test/it/unimi/dsi/
branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/src/java/it/unimi/dsi/fastutil/bytes/custom/
branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/src/test/it/unimi/dsi/fastutil/bytes/custom/
branches/BIGDATA_RELEASE_1_3_0/osgi/
branches/BIGDATA_RELEASE_1_3_0/src/resources/bin/config/
Property changes on: branches/BIGDATA_RELEASE_1_3_0
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE:6769-6785
/branches/BIGDATA_RELEASE_1_2_0:6766-7380
/branches/BTREE_BUFFER_BRANCH:2004-2045
/branches/DEV_BRANCH_27_OCT_2009:2270-2546,2548-2782
/branches/INT64_BRANCH:4486-4522
/branches/JOURNAL_HA_BRANCH:2596-4066
/branches/LARGE_LITERALS_REFACTOR:4175-4387
/branches/LEXICON_REFACTOR_BRANCH:2633-3304
/branches/QUADS_QUERY_BRANCH:4525-4531,4550-4584,4586-4609,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/READ_CACHE:7215-7271
/branches/RWSTORE_1_1_0_DEBUG:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH:4814-4836
/branches/ZK_DISCONNECT_HANDLING:7465-7484
/branches/bugfix-btm:2594-3237
/branches/dev-btm:2574-2730
/branches/fko:3150-3194
/trunk:3392-3437,3656-4061
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE:6769-6785
/branches/BIGDATA_RELEASE_1_2_0:6766-7380
/branches/BTREE_BUFFER_BRANCH:2004-2045
/branches/DEV_BRANCH_27_OCT_2009:2270-2546,2548-2782
/branches/INT64_BRANCH:4486-4522
/branches/JOURNAL_HA_BRANCH:2596-4066
/branches/LARGE_LITERALS_REFACTOR:4175-4387
/branches/LEXICON_REFACTOR_BRANCH:2633-3304
/branches/MGC_1_3_0:7609-7752
/branches/QUADS_QUERY_BRANCH:4525-4531,4550-4584,4586-4609,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/READ_CACHE:7215-7271
/branches/RWSTORE_1_1_0_DEBUG:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH:4814-4836
/branches/ZK_DISCONNECT_HANDLING:7465-7484
/branches/bugfix-btm:2594-3237
/branches/dev-btm:2574-2730
/branches/fko:3150-3194
/trunk:3392-3437,3656-4061
Property changes on: branches/BIGDATA_RELEASE_1_3_0/bigdata/lib/jetty
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/lib/jetty:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/lib/jetty:6766-7380
/branches/INT64_BRANCH/bigdata/lib/jetty:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/lib/jetty:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/READ_CACHE/bigdata/lib/jetty:7215-7271
/branches/RWSTORE_1_1_0_DEBUG/bigdata/lib/jetty:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/lib/jetty:4814-4836
/branches/ZK_DISCONNECT_HANDLING/bigdata/lib/jetty:7465-7484
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/lib/jetty:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/lib/jetty:6766-7380
/branches/INT64_BRANCH/bigdata/lib/jetty:4486-4522
/branches/MGC_1_3_0/bigdata/lib/jetty:7609-7752
/branches/QUADS_QUERY_BRANCH/bigdata/lib/jetty:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/READ_CACHE/bigdata/lib/jetty:7215-7271
/branches/RWSTORE_1_1_0_DEBUG/bigdata/lib/jetty:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/lib/jetty:4814-4836
/branches/ZK_DISCONNECT_HANDLING/bigdata/lib/jetty:7465-7484
Property changes on: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/aggregate
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/java/com/bigdata/bop/aggregate:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/aggregate:6766-7380
/branches/INT64_BRANCH/bigdata/src/java/com/bigdata/bop/aggregate:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregate:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/READ_CACHE/bigdata/src/java/com/bigdata/bop/aggregate:7215-7271
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/java/com/bigdata/bop/aggregate:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/java/com/bigdata/bop/aggregate:4814-4836
/branches/ZK_DISCONNECT_HANDLING/bigdata/src/java/com/bigdata/bop/aggregate:7465-7484
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/java/com/bigdata/bop/aggregate:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/aggregate:6766-7380
/branches/INT64_BRANCH/bigdata/src/java/com/bigdata/bop/aggregate:4486-4522
/branches/MGC_1_3_0/bigdata/src/java/com/bigdata/bop/aggregate:7609-7752
/branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregate:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/READ_CACHE/bigdata/src/java/com/bigdata/bop/aggregate:7215-7271
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/java/com/bigdata/bop/aggregate:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/java/com/bigdata/bop/aggregate:4814-4836
/branches/ZK_DISCONNECT_HANDLING/bigdata/src/java/com/bigdata/bop/aggregate:7465-7484
Property changes on: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/java/com/bigdata/bop/joinGraph:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/joinGraph:6766-7380
/branches/INT64_BRANCH/bigdata/src/java/com/bigdata/bop/joinGraph:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/joinGraph:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/READ_CACHE/bigdata/src/java/com/bigdata/bop/joinGraph:7215-7271
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/java/com/bigdata/bop/joinGraph:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/java/com/bigdata/bop/joinGraph:4814-4836
/branches/ZK_DISCONNECT_HANDLING/bigdata/src/java/com/bigdata/bop/joinGraph:7465-7484
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/java/com/bigdata/bop/joinGraph:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/joinGraph:6766-7380
/branches/INT64_BRANCH/bigdata/src/java/com/bigdata/bop/joinGraph:4486-4522
/branches/MGC_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph:7609-7752
/branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/joinGraph:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/READ_CACHE/bigdata/src/java/com/bigdata/bop/joinGraph:7215-7271
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/java/com/bigdata/bop/joinGraph:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/java/com/bigdata/bop/joinGraph:4814-4836
/branches/ZK_DISCONNECT_HANDLING/bigdata/src/java/com/bigdata/bop/joinGraph:7465-7484
Property changes on: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/util
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/java/com/bigdata/bop/util:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/util:6766-7380
/branches/INT64_BRANCH/bigdata/src/java/com/bigdata/bop/util:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/util:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/READ_CACHE/bigdata/src/java/com/bigdata/bop/util:7215-7271
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/java/com/bigdata/bop/util:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/java/com/bigdata/bop/util:4814-4836
/branches/ZK_DISCONNECT_HANDLING/bigdata/src/java/com/bigdata/bop/util:7465-7484
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/java/com/bigdata/bop/util:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/util:6766-7380
/branches/INT64_BRANCH/bigdata/src/java/com/bigdata/bop/util:4486-4522
/branches/MGC_1_3_0/bigdata/src/ja...
[truncated message content] |
|
From: <tho...@us...> - 2014-01-09 21:55:18
|
Revision: 7755
http://bigdata.svn.sourceforge.net/bigdata/?rev=7755&view=rev
Author: thompsonbry
Date: 2014-01-09 21:55:07 +0000 (Thu, 09 Jan 2014)
Log Message:
-----------
The RTO now supports FILTERs on required joins when those filters have materialization requirements. This is the subject of #257, which is now closed. Over time we will probably refactor the RTO to operate directly on the AST nodes (e.g., StatementPatternNode versus SPOPredicate), but it currently handles conditional materialization pipelines just fine.
There are currently two different ways in which a cutoff join can be evaluated. One uses a single pipeline join and carefully controls the execution of that join to obtain an estimated cardinality. This is the historical method.
The other can handle a sequence of operators. A pipeline join is generated followed by whatever operators are required to materialize variable bindings and evaluate filters. Finally, a SLICE is appended to the query plan to limit the output. When the query is executed, a rowid column is injected into the source solutions. This is used to correctly correlate the #of input solutions required to produce a given #of output solutions and hence obtain the estimated cardinality of the join through cutoff evaluation.
Some slight differences in the resulting plans and runtime behavior have been observed when all queries
See #257 (Support BOP fragments in the RTO)
See #64 (Runtime Query Optimizer)
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpIdFactory.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/PartitionedJoinGroup.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/EdgeSample.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JGraph.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/Path.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/SampleBase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpBase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpFilters.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpJoins.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/AbstractRTOTestCase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/BAR-Q1.rq
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/BSBM-Q1.rq
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/BSBM-Q1.srx
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/LUBM-Q2.rq
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/LUBM-Q9.rq
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/TestRTO_BSBM.java
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/OutOfOrderEvaluationException.java
Removed Paths:
-------------
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/AbstractJoinGraphTestCase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestAll.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphOnBSBMData.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphOnBarData.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphOnLubm.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpIdFactory.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpIdFactory.java 2014-01-09 20:47:55 UTC (rev 7754)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpIdFactory.java 2014-01-09 21:55:07 UTC (rev 7755)
@@ -82,51 +82,54 @@
}
/**
- * Reserve ids used by the predicates or constraints associated with some
- * join graph.
+ * Reserve ids used by the predicates in some join graph.
*
* @param preds
* The vertices of the join graph.
- * @param constraints
- * The constraints of the join graph (optional).
*/
- public void reserveIds(final IPredicate<?>[] preds,
- final IConstraint[] constraints) {
+ public void reserveIds(final IPredicate<?>[] preds) {
if (preds == null)
throw new IllegalArgumentException();
- final BOpIdFactory idFactory = this;
-
for (IPredicate<?> p : preds) {
-
- idFactory.reserve(p.getId());
-
+
+ reserve(p.getId());
+
}
- if (constraints != null) {
-
- for (IConstraint c : constraints) {
-
- final Iterator<BOp> itr = BOpUtility
- .preOrderIteratorWithAnnotations(c);
+ }
- while (itr.hasNext()) {
-
- final BOp y = itr.next();
-
- final Integer anId = (Integer) y
- .getProperty(BOp.Annotations.BOP_ID);
-
- if (anId != null)
- idFactory.reserve(anId.intValue());
-
- }
-
+ /**
+ * Reserve ids used by the constraints for some predicate or join graph.
+ *
+ * @param constraints
+ * The constraints that attach to some predicate (optional).
+ */
+ public void reserveIds(final IConstraint[] constraints) {
+
+ if (constraints == null)
+ return;
+
+ for (IConstraint c : constraints) {
+
+ final Iterator<BOp> itr = BOpUtility
+ .preOrderIteratorWithAnnotations(c);
+
+ while (itr.hasNext()) {
+
+ final BOp y = itr.next();
+
+ final Integer anId = (Integer) y
+ .getProperty(BOp.Annotations.BOP_ID);
+
+ if (anId != null)
+ reserve(anId.intValue());
+
}
}
-
+
}
}
\ No newline at end of file
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java 2014-01-09 20:47:55 UTC (rev 7754)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java 2014-01-09 21:55:07 UTC (rev 7755)
@@ -204,8 +204,23 @@
String COALESCE_DUPLICATE_ACCESS_PATHS = (PipelineJoin.class.getName()
+ ".coalesceDuplicateAccessPaths").intern();
- boolean DEFAULT_COALESCE_DUPLICATE_ACCESS_PATHS = true;
+ boolean DEFAULT_COALESCE_DUPLICATE_ACCESS_PATHS = true;
+ /**
+ * When <code>true</code>, access paths will be reordered to maximize
+ * locality.
+ * <p>
+ * Note: This needs to be turned off when the RTO uses row identifiers
+ * to correlate the input and output solutions for complex joins (those
+ * which required materialization of RDF Values for FILTER evaluation).
+ *
+ * @todo unit tests when (en|dis)abled.
+ */
+ String REORDER_ACCESS_PATHS = (PipelineJoin.class.getName() + ".reorderAccessPaths")
+ .intern();
+
+ boolean DEFAULT_REORDER_ACCESS_PATHS = true;
+
}
/**
@@ -223,7 +238,7 @@
* @param args
* @param annotations
*/
- public PipelineJoin(final BOp[] args, NV... annotations) {
+ public PipelineJoin(final BOp[] args, final NV... annotations) {
this(args, NV.asMap(annotations));
@@ -239,6 +254,11 @@
super(args, annotations);
+ /*
+ * TODO We should be checking this operator's required properties,
+ * especially when used for the RTO.
+ */
+
// if (arity() != 1)
// throw new IllegalArgumentException();
@@ -247,15 +267,6 @@
}
- // /**
- // * The sole operand, which is the previous join in the pipeline join path.
- // */
- // public PipelineOp left() {
- //
- // return (PipelineOp) get(0);
- //
- // }
-
/**
* {@inheritDoc}
*
@@ -420,8 +431,16 @@
*
* @see Annotations#COALESCE_DUPLICATE_ACCESS_PATHS
*/
- final boolean coalesceAccessPaths;
+ final private boolean coalesceAccessPaths;
+ /**
+ * When <code>true</code>, access paths will be reordered to maximize
+ * locality.
+ *
+ * @see Annotations#REORDER_ACCESS_PATHS
+ */
+ final private boolean reorderAccessPaths;
+
/**
* Used to enforce the {@link Annotations#LIMIT} iff one is specified.
*/
@@ -523,6 +542,9 @@
this.coalesceAccessPaths = joinOp.getProperty(
Annotations.COALESCE_DUPLICATE_ACCESS_PATHS,
Annotations.DEFAULT_COALESCE_DUPLICATE_ACCESS_PATHS);
+ this.reorderAccessPaths = joinOp.getProperty(
+ Annotations.REORDER_ACCESS_PATHS,
+ Annotations.DEFAULT_REORDER_ACCESS_PATHS);
this.threadLocalBufferFactory = new TLBFactory(sink);
@@ -942,10 +964,11 @@
*/
final AccessPathTask[] tasks = generateAccessPaths(chunk);
- /*
- * Reorder those tasks for better index read performance.
- */
- reorderTasks(tasks);
+ /*
+ * Reorder those tasks for better index read performance.
+ */
+ if (reorderAccessPaths)
+ reorderTasks(tasks);
/*
* Execute the tasks (either in the caller's thread or on
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/PartitionedJoinGroup.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/PartitionedJoinGroup.java 2014-01-09 20:47:55 UTC (rev 7754)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/PartitionedJoinGroup.java 2014-01-09 21:55:07 UTC (rev 7755)
@@ -12,18 +12,12 @@
import org.apache.log4j.Logger;
import com.bigdata.bop.BOp;
-import com.bigdata.bop.BOpEvaluationContext;
-import com.bigdata.bop.BOpIdFactory;
import com.bigdata.bop.BOpUtility;
import com.bigdata.bop.Bind;
import com.bigdata.bop.IConstraint;
import com.bigdata.bop.IPredicate;
import com.bigdata.bop.IVariable;
-import com.bigdata.bop.NV;
-import com.bigdata.bop.PipelineOp;
-import com.bigdata.bop.join.PipelineJoin;
import com.bigdata.bop.joinGraph.rto.JoinGraph;
-import com.bigdata.bop.solutions.JVMDistinctBindingSetsOp;
import com.bigdata.rdf.sparql.ast.StaticAnalysis_CanJoin;
/**
@@ -962,180 +956,180 @@
}
- /**
- * Generate a query plan from an ordered collection of predicates.
- *
- * @param distinct
- * <code>true</code> iff only the distinct solutions are desired.
- * @param selected
- * The variable(s) to be projected out of the join graph.
- * @param preds
- * The join path which will be used to execute the join graph.
- * @param constraints
- * The constraints on the join graph.
- *
- * @return The query plan.
- *
- * FIXME Select only those variables required by downstream
- * processing or explicitly specified by the caller (in the case
- * when this is a subquery, the caller has to declare which
- * variables are selected and will be returned out of the subquery).
- *
- * FIXME For scale-out, we need to either mark the join's evaluation
- * context based on whether or not the access path is local or
- * remote (and whether the index is key-range distributed or hash
- * partitioned).
- *
- * FIXME Add a method to generate a runnable query plan from the
- * collection of predicates and constraints on the
- * {@link PartitionedJoinGroup} together with an ordering over the
- * join graph. This is a bit different for the join graph and the
- * optionals in the tail plan. The join graph itself should either
- * be a {@link JoinGraph} operator which gets evaluated at run time
- * or reordered by whichever optimizer is selected for the query
- * (query hints).
- *
- * @todo The order of the {@link IPredicate}s in the tail plan is currently
- * unchanged from their given order (optional joins without
- * constraints can not reduce the selectivity of the query). However,
- * it could be worthwhile to run optionals with constraints before
- * those without constraints since the constraints can reduce the
- * selectivity of the query. If we do this, then we need to reorder
- * the optionals based on the partial order imposed what variables
- * they MIGHT bind (which are not bound by the join graph).
- *
- * @todo multiple runFirst predicates can be evaluated in parallel unless
- * they have shared variables. When there are no shared variables,
- * construct a TEE pattern such that evaluation proceeds in parallel.
- * When there are shared variables, the runFirst predicates must be
- * ordered based on those shared variables (at which point, it is
- * probably an error to flag them as runFirst).
- */
- static public PipelineOp getQuery(final BOpIdFactory idFactory,
- final boolean distinct, final IVariable<?>[] selected,
- final IPredicate<?>[] preds, final IConstraint[] constraints) {
-
- /*
- * Reserve ids used by the join graph or its constraints.
- */
- idFactory.reserveIds(preds, constraints);
-// {
-// for (IPredicate<?> p : preds) {
-// idFactory.reserve(p.getId());
+// /**
+// * Generate a query plan from an ordered collection of predicates.
+// *
+// * @param distinct
+// * <code>true</code> iff only the distinct solutions are desired.
+// * @param selected
+// * The variable(s) to be projected out of the join graph.
+// * @param preds
+// * The join path which will be used to execute the join graph.
+// * @param constraints
+// * The constraints on the join graph.
+// *
+// * @return The query plan.
+// *
+// * FIXME Select only those variables required by downstream
+// * processing or explicitly specified by the caller (in the case
+// * when this is a subquery, the caller has to declare which
+// * variables are selected and will be returned out of the subquery).
+// *
+// * FIXME For scale-out, we need to either mark the join's evaluation
+// * context based on whether or not the access path is local or
+// * remote (and whether the index is key-range distributed or hash
+// * partitioned).
+// *
+// * FIXME Add a method to generate a runnable query plan from the
+// * collection of predicates and constraints on the
+// * {@link PartitionedJoinGroup} together with an ordering over the
+// * join graph. This is a bit different for the join graph and the
+// * optionals in the tail plan. The join graph itself should either
+// * be a {@link JoinGraph} operator which gets evaluated at run time
+// * or reordered by whichever optimizer is selected for the query
+// * (query hints).
+// *
+// * @todo The order of the {@link IPredicate}s in the tail plan is currently
+// * unchanged from their given order (optional joins without
+// * constraints can not reduce the selectivity of the query). However,
+// * it could be worthwhile to run optionals with constraints before
+// * those without constraints since the constraints can reduce the
+// * selectivity of the query. If we do this, then we need to reorder
+// * the optionals based on the partial order imposed what variables
+// * they MIGHT bind (which are not bound by the join graph).
+// *
+// * @todo multiple runFirst predicates can be evaluated in parallel unless
+// * they have shared variables. When there are no shared variables,
+// * construct a TEE pattern such that evaluation proceeds in parallel.
+// * When there are shared variables, the runFirst predicates must be
+// * ordered based on those shared variables (at which point, it is
+// * probably an error to flag them as runFirst).
+// */
+// static public PipelineOp getQuery(final BOpIdFactory idFactory,
+// final boolean distinct, final IVariable<?>[] selected,
+// final IPredicate<?>[] preds, final IConstraint[] constraints) {
+//
+// /*
+// * Reserve ids used by the join graph or its constraints.
+// */
+// idFactory.reserveIds(preds, constraints);
+//// {
+//// for (IPredicate<?> p : preds) {
+//// idFactory.reserve(p.getId());
+//// }
+//// if (constraints != null) {
+//// for (IConstraint c : constraints) {
+//// final Iterator<BOp> itr = BOpUtility
+//// .preOrderIteratorWithAnnotations(c);
+//// while (itr.hasNext()) {
+//// final BOp y = itr.next();
+//// final Integer anId = (Integer) y
+//// .getProperty(BOp.Annotations.BOP_ID);
+//// if (anId != null)
+//// idFactory.reserve(anId.intValue());
+//// }
+//// }
+//// }
+//// }
+//
+// // figure out which constraints are attached to which predicates.
+// final IConstraint[][] assignedConstraints = PartitionedJoinGroup
+// .getJoinGraphConstraints(preds, constraints, null/*knownBound*/,
+// true/*pathIsComplete*/);
+//
+//// final PipelineJoin<?>[] joins = new PipelineJoin[preds.length];
+//
+// PipelineOp lastOp = null;
+//
+// final Set<IVariable<?>> knownBound = new LinkedHashSet<IVariable<?>>();
+//
+// for (int i = 0; i < preds.length; i++) {
+//
+// // The next vertex in the selected join order.
+// final IPredicate<?> p = preds[i];
+//
+// // Annotations for this join.
+// final List<NV> anns = new LinkedList<NV>();
+//
+// anns.add(new NV(PipelineJoin.Annotations.PREDICATE, p));
+//
+// anns.add(new NV(PipelineJoin.Annotations.BOP_ID, idFactory
+// .nextId()));
+//
+//// anns.add(new NV(PipelineJoin.Annotations.EVALUATION_CONTEXT, BOpEvaluationContext.ANY));
+////
+//// anns.add(new NV(PipelineJoin.Annotations.SELECT, vars.toArray(new IVariable[vars.size()])));
+//
+// if (assignedConstraints[i] != null
+// && assignedConstraints[i].length > 0) {
+// // attach constraints to this join.
+// anns.add(new NV(PipelineJoin.Annotations.CONSTRAINTS,
+// assignedConstraints[i]));
// }
-// if (constraints != null) {
-// for (IConstraint c : constraints) {
-// final Iterator<BOp> itr = BOpUtility
-// .preOrderIteratorWithAnnotations(c);
-// while (itr.hasNext()) {
-// final BOp y = itr.next();
-// final Integer anId = (Integer) y
-// .getProperty(BOp.Annotations.BOP_ID);
-// if (anId != null)
-// idFactory.reserve(anId.intValue());
+//
+// // collect variables used as arguments by this predicate.
+// final Set<IVariable<?>> pvars = new LinkedHashSet<IVariable<?>>();
+// {
+// final Iterator<IVariable<?>> vitr = BOpUtility
+// .getArgumentVariables(p);
+// while (vitr.hasNext()) {
+// pvars.add(vitr.next());
+// }
+// }
+//
+// // figure out if there are ANY shared variables.
+// boolean shared = false;
+// {
+// for(IVariable<?> v : pvars) {
+// if(knownBound.contains(v)) {
+// shared = true;
+// break;
// }
// }
// }
+//
+// /*
+// * FIXME Explore the merit of this optimization with MikeP,
+// * including consideration of the PIPELINE_QUEUE_CAPACITY and
+// * whether or not to request an analytic join (hash join).
+// */
+// if (false && !shared) {
+// System.err.println("Full cross product join: " + p);
+// /*
+// * Force at-once evaluation to ensure that we evaluate the AP
+// * for [p] exactly once.
+// */
+// anns.add(new NV(PipelineOp.Annotations.PIPELINED, false));
+// }
+//
+// final PipelineJoin<?> joinOp = new PipelineJoin(//
+// lastOp == null ? new BOp[0] : new BOp[] { lastOp }, //
+// anns.toArray(new NV[anns.size()])//
+// );
+//
+// // Add predicate argument variables to [knownBound].
+// knownBound.addAll(pvars);
+//
+// lastOp = joinOp;
+//
+// }
+//
+// if (distinct) {
+// lastOp = new JVMDistinctBindingSetsOp(new BOp[] { lastOp }, NV
+// .asMap(new NV[] {
+// new NV(PipelineOp.Annotations.BOP_ID, idFactory
+// .nextId()), //
+// new NV(PipelineOp.Annotations.EVALUATION_CONTEXT,
+// BOpEvaluationContext.CONTROLLER),//
+// new NV(PipelineOp.Annotations.SHARED_STATE, true),//
+// new NV(JVMDistinctBindingSetsOp.Annotations.VARIABLES,
+// selected),//
+// })//
+// );
// }
-
- // figure out which constraints are attached to which predicates.
- final IConstraint[][] assignedConstraints = PartitionedJoinGroup
- .getJoinGraphConstraints(preds, constraints, null/*knownBound*/,
- true/*pathIsComplete*/);
-
-// final PipelineJoin<?>[] joins = new PipelineJoin[preds.length];
-
- PipelineOp lastOp = null;
-
- final Set<IVariable<?>> knownBound = new LinkedHashSet<IVariable<?>>();
-
- for (int i = 0; i < preds.length; i++) {
-
- // The next vertex in the selected join order.
- final IPredicate<?> p = preds[i];
-
- // Annotations for this join.
- final List<NV> anns = new LinkedList<NV>();
-
- anns.add(new NV(PipelineJoin.Annotations.PREDICATE, p));
-
- anns.add(new NV(PipelineJoin.Annotations.BOP_ID, idFactory
- .nextId()));
-
-// anns.add(new NV(PipelineJoin.Annotations.EVALUATION_CONTEXT, BOpEvaluationContext.ANY));
//
-// anns.add(new NV(PipelineJoin.Annotations.SELECT, vars.toArray(new IVariable[vars.size()])));
+// return lastOp;
+//
+// }
- if (assignedConstraints[i] != null
- && assignedConstraints[i].length > 0) {
- // attach constraints to this join.
- anns.add(new NV(PipelineJoin.Annotations.CONSTRAINTS,
- assignedConstraints[i]));
- }
-
- // collect variables used as arguments by this predicate.
- final Set<IVariable<?>> pvars = new LinkedHashSet<IVariable<?>>();
- {
- final Iterator<IVariable<?>> vitr = BOpUtility
- .getArgumentVariables(p);
- while (vitr.hasNext()) {
- pvars.add(vitr.next());
- }
- }
-
- // figure out if there are ANY shared variables.
- boolean shared = false;
- {
- for(IVariable<?> v : pvars) {
- if(knownBound.contains(v)) {
- shared = true;
- break;
- }
- }
- }
-
- /*
- * FIXME Explore the merit of this optimization with MikeP,
- * including consideration of the PIPELINE_QUEUE_CAPACITY and
- * whether or not to request an analytic join (hash join).
- */
- if (false && !shared) {
- System.err.println("Full cross product join: " + p);
- /*
- * Force at-once evaluation to ensure that we evaluate the AP
- * for [p] exactly once.
- */
- anns.add(new NV(PipelineOp.Annotations.PIPELINED, false));
- }
-
- final PipelineJoin<?> joinOp = new PipelineJoin(//
- lastOp == null ? new BOp[0] : new BOp[] { lastOp }, //
- anns.toArray(new NV[anns.size()])//
- );
-
- // Add predicate argument variables to [knownBound].
- knownBound.addAll(pvars);
-
- lastOp = joinOp;
-
- }
-
- if (distinct) {
- lastOp = new JVMDistinctBindingSetsOp(new BOp[] { lastOp }, NV
- .asMap(new NV[] {
- new NV(PipelineOp.Annotations.BOP_ID, idFactory
- .nextId()), //
- new NV(PipelineOp.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.CONTROLLER),//
- new NV(PipelineOp.Annotations.SHARED_STATE, true),//
- new NV(JVMDistinctBindingSetsOp.Annotations.VARIABLES,
- selected),//
- })//
- );
- }
-
- return lastOp;
-
- }
-
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/EdgeSample.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/EdgeSample.java 2014-01-09 20:47:55 UTC (rev 7754)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/EdgeSample.java 2014-01-09 21:55:07 UTC (rev 7755)
@@ -119,7 +119,7 @@
* <i>outputCount</i> as adjusted for a variety of edge
* conditions).
*/
- EdgeSample(final SampleBase sourceSample,//
+ public EdgeSample(final SampleBase sourceSample,//
final int inputCount, //
final long tuplesRead,//
final long sumRangeCount,//
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JGraph.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JGraph.java 2014-01-09 20:47:55 UTC (rev 7754)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JGraph.java 2014-01-09 21:55:07 UTC (rev 7755)
@@ -53,6 +53,7 @@
import com.bigdata.bop.joinGraph.NoSolutionsException;
import com.bigdata.bop.joinGraph.PartitionedJoinGroup;
import com.bigdata.bop.rdf.join.DataSetJoin;
+import com.bigdata.rdf.sparql.ast.eval.AST2BOpRTO;
import com.bigdata.util.concurrent.ExecutionExceptions;
/**
@@ -223,6 +224,14 @@
private static final transient Logger log = Logger.getLogger(JGraph.class);
/**
+ * The pipeline operator for executing the RTO. This provides additional
+ * context from the AST model that is necessary to handle some kinds of
+ * FILTERs (e.g., those which require conditional routing patterns for
+ * chunked materialization).
+ */
+ private final JoinGraph joinGraph;
+
+ /**
* Vertices of the join graph.
*/
private final Vertex[] V;
@@ -258,40 +267,49 @@
return sb.toString();
}
- /**
- *
- * @param v
- * The vertices of the join graph. These are {@link IPredicate}s
- * associated with required joins.
- * @param constraints
- * The constraints of the join graph (optional). Since all joins
- * in the join graph are required, constraints are dynamically
- * attached to the first join in which all of their variables are
- * bound.
- *
- * @throws IllegalArgumentException
- * if the vertices is <code>null</code>.
- * @throws IllegalArgumentException
- * if the vertices is an empty array.
- * @throws IllegalArgumentException
- * if any element of the vertices is <code>null</code>.
- * @throws IllegalArgumentException
- * if any constraint uses a variable which is never bound by the
- * given predicates.
- * @throws IllegalArgumentException
- * if <i>sampleType</i> is <code>null</code>.
- *
- * @todo unit test for a constraint using a variable which is never bound.
- * the constraint should be attached at the last vertex in the join
- * path. this will cause the query to fail unless the variable was
- * already bound, e.g., by a parent query or in the solutions pumped
- * into the {@link JoinGraph} operator.
- *
- * @todo unit test when the join graph has a single vertex.
- */
- public JGraph(final IPredicate<?>[] v, final IConstraint[] constraints,
- final SampleType sampleType) {
+ /**
+ * @param joinGraph
+ * The pipeline operator that is executing the RTO. This defines
+ * the join graph (vertices, edges, and constraints) and also
+ * provides access to the AST and related metadata required to
+ * execute the join graph.
+ *
+ * @throws IllegalArgumentException
+ * if the joinGraph is <code>null</code>.
+ * @throws IllegalArgumentException
+ * if the {@link JoinGraph#getVertices()} is <code>null</code>.
+ * @throws IllegalArgumentException
+ * if the {@link JoinGraph#getVertices()} is an empty array.
+ * @throws IllegalArgumentException
+ * if any element of the {@link JoinGraph#getVertices()}is
+ * <code>null</code>.
+ * @throws IllegalArgumentException
+ * if any constraint uses a variable which is never bound by the
+ * given predicates.
+ * @throws IllegalArgumentException
+ * if <i>sampleType</i> is <code>null</code>.
+ *
+ * @todo unit test for a constraint using a variable which is never bound.
+ * the constraint should be attached at the last vertex in the join
+ * path. this will cause the query to fail unless the variable was
+ * already bound, e.g., by a parent query or in the solutions pumped
+ * into the {@link JoinGraph} operator.
+ *
+ * @todo unit test when the join graph has a single vertex (we never invoke
+ * the RTO for less than 3 vertices since with one vertex you just run
+ * it and with two vertices you run the lower cardinality vertex first
+ * (though there might be cases where filters require materialization
+ * where running for two vertices could make sense)).
+ */
+ public JGraph(final JoinGraph joinGraph) {
+ if (joinGraph == null)
+ throw new IllegalArgumentException();
+
+ this.joinGraph = joinGraph;
+
+ final IPredicate<?>[] v = joinGraph.getVertices();
+
if (v == null)
throw new IllegalArgumentException();
@@ -309,6 +327,8 @@
}
+ final IConstraint[] constraints = joinGraph.getConstraints();
+
if (constraints != null) {
C = new IConstraint[constraints.length];
for (int i = 0; i < constraints.length; i++) {
@@ -321,6 +341,8 @@
C = null;
}
+ final SampleType sampleType = joinGraph.getSampleType();
+
if (sampleType == null)
throw new IllegalArgumentException();
@@ -519,9 +541,9 @@
}
// Should be one winner.
- if (paths.length != 1) {
- throw new AssertionError("Expected one path but have "
- + paths.length + " paths.");
+ if (paths.length != 1) {
+ throw new AssertionError("Expected one path but have "
+ + paths.length + " paths.");
}
if (log.isInfoEnabled()) {
@@ -651,18 +673,19 @@
*/
sampleAllVertices(queryEngine, limit);
- if (log.isInfoEnabled()) {
- final StringBuilder sb = new StringBuilder();
- sb.append("Sampled vertices:\n");
- for (Vertex v : V) {
- if (v.sample != null) {
- sb.append("id="+v.pred.getId()+" : ");
- sb.append(v.sample.toString());
- sb.append("\n");
- }
- }
- log.info(sb.toString());
- }
+ if (log.isInfoEnabled()) {
+ final StringBuilder sb = new StringBuilder();
+ sb.append("limit=" + limit + ", nedges=" + nedges);
+ sb.append(", sampled vertices::\n");
+ for (Vertex v : V) {
+ if (v.sample != null) {
+ sb.append("id=" + v.pred.getId() + " : ");
+ sb.append(v.sample.toString());
+ sb.append("\n");
+ }
+ }
+ log.info(sb.toString());
+ }
/*
* Estimate the cardinality for each edge.
@@ -940,8 +963,10 @@
* cardinality vertex.
*/
- edgeSample = Path.cutoffJoin(//
- queryEngine, limit,//
+ edgeSample = AST2BOpRTO.cutoffJoin(//
+ queryEngine, //
+ joinGraph, //
+ limit,//
x.getPathSegment(2),// 1st edge.
C,// constraints
V.length == 2,// pathIsComplete
@@ -978,7 +1003,9 @@
* edge of the path.
*/
- edgeSample = Path.cutoffJoin(queryEngine,//
+ edgeSample = AST2BOpRTO.cutoffJoin(//
+ queryEngine,//
+ joinGraph,//
limit,//
x.getPathSegment(ids.length()),//
C, // constraints
@@ -1245,9 +1272,14 @@
used.add(tVertex);
// Extend the path to the new vertex.
- final Path p = x
- .addEdge(queryEngine, limit, tVertex, /* dynamicEdge, */
- C, x.getVertexCount() + 1 == V.length/* pathIsComplete */);
+ final Path p = x.addEdge(//
+ queryEngine, //
+ joinGraph, //
+ limit,//
+ tVertex,//
+ C, //
+ x.getVertexCount() + 1 == V.length// pathIsComplete
+ );
// Add to the set of paths for this round.
tmp.add(p);
@@ -1284,9 +1316,14 @@
final Vertex tVertex = nothingShared.iterator().next();
// Extend the path to the new vertex.
- final Path p = x
- .addEdge(queryEngine, limit, tVertex,/* dynamicEdge */
- C, x.getVertexCount() + 1 == V.length/* pathIsComplete */);
+ final Path p = x.addEdge(//
+ queryEngine, //
+ joinGraph,//
+ limit, //
+ tVertex, //
+ C,//
+ x.getVertexCount() + 1 == V.length// pathIsComplete
+ );
// Add to the set of paths for this round.
tmp.add(p);
@@ -1640,7 +1677,9 @@
final IPredicate<?>[] preds = new IPredicate[] { v.pred, vp.pred };
// cutoff join of the edge (v,vp)
- final EdgeSample edgeSample = Path.cutoffJoin(queryEngine,//
+ final EdgeSample edgeSample = AST2BOpRTO.cutoffJoin(//
+ queryEngine,//
+ joinGraph,//
limit, // sample limit
preds, // ordered path segment.
C, // constraints
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java 2014-01-09 20:47:55 UTC (rev 7754)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java 2014-01-09 21:55:07 UTC (rev 7755)
@@ -475,8 +475,7 @@
final long begin = System.nanoTime();
// Create the join graph.
- final JGraph g = new JGraph(getVertices(), getConstraints(),
- getSampleType());
+ final JGraph g = new JGraph(JoinGraph.this);
/*
* This map is used to associate join path segments (expressed as an
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/Path.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/Path.java 2014-01-09 20:47:55 UTC (rev 7754)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/Path.java 2014-01-09 21:55:07 UTC (rev 7755)
@@ -25,30 +25,16 @@
import java.util.Arrays;
import java.util.Collections;
-import java.util.Iterator;
-import java.util.LinkedList;
import java.util.List;
-import java.util.Map;
-import java.util.UUID;
import org.apache.log4j.Logger;
import com.bigdata.bop.BOp;
-import com.bigdata.bop.BOpEvaluationContext;
-import com.bigdata.bop.BOpIdFactory;
import com.bigdata.bop.BOpUtility;
-import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.IConstraint;
import com.bigdata.bop.IPredicate;
-import com.bigdata.bop.NV;
-import com.bigdata.bop.PipelineOp;
-import com.bigdata.bop.engine.IRunningQuery;
-import com.bigdata.bop.engine.LocalChunkMessage;
import com.bigdata.bop.engine.QueryEngine;
-import com.bigdata.bop.join.PipelineJoin;
-import com.bigdata.bop.join.PipelineJoinStats;
-import com.bigdata.bop.joinGraph.PartitionedJoinGroup;
-import com.bigdata.striterator.Dechunkerator;
+import com.bigdata.rdf.sparql.ast.eval.AST2BOpRTO;
/**
* A join path is an ordered sequence of N {@link Vertex vertices} and
@@ -182,18 +168,20 @@
*/
final public long sumEstCost;
- /**
- * Combine the cumulative expected cardinality and the cumulative expected
- * tuples read to produce an overall measure of the expected cost of the
- * join path if it were to be fully executed.
- *
- * @return The cumulative estimated cost of the join path.
- *
- * TODO Compute this incrementally as estCost using estRead and
- * estCard and then take the running sum as sumEstCost and update
- * the JGraph trace appropriately.
- */
- private static long getCost(final long sumEstRead, final long sumEstCard) {
+ /**
+ * Combine the cumulative expected cardinality and the cumulative expected
+ * tuples read to produce an overall measure of the expected cost of the
+ * join path if it were to be fully executed.
+ *
+ * @return The cumulative estimated cost of the join path.
+ *
+ * TODO Compute this incrementally as estCost using estRead and
+ * estCard and then take the running sum as sumEstCost and update
+ * the JGraph trace appropriately. [Refactor into an IPathCost
+ * interface. It should have visibility into the full path and also
+ * allow visibility into the vertex cost for generality.]
+ */
+ private static long getCost(final long sumEstRead, final long sumEstCard) {
final long total;
// total = sumEstCard + sumEstRead; // intermediate results + IO.
@@ -631,9 +619,9 @@
*
* @throws Exception
*/
- public Path addEdge(final QueryEngine queryEngine, final int limit,
- final Vertex vnew, final IConstraint[] constraints,
- final boolean pathIsComplete)
+ public Path addEdge(final QueryEngine queryEngine,
+ final JoinGraph joinGraph, final int limit, final Vertex vnew,
+ final IConstraint[] constraints, final boolean pathIsComplete)
throws Exception {
if (vnew == null)
@@ -692,8 +680,9 @@
}
- final EdgeSample edgeSample2 = cutoffJoin(//
+ final EdgeSample edgeSample2 = AST2BOpRTO.cutoffJoin(//
queryEngine,//
+ joinGraph,//
limit, //
preds2,//
constraints,//
@@ -715,41 +704,47 @@
}
- /**
- * Cutoff join of the last vertex in the join path.
- * <p>
- * <strong>The caller is responsible for protecting against needless
- * re-sampling.</strong> This includes cases where a sample already exists
- * at the desired sample limit and cases where the sample is already exact.
- *
- * @param queryEngine
- * The query engine.
- * @param limit
- * The limit for the cutoff join.
- * @param path
- * The path segment, which must include the target vertex as the
- * last component of the path segment.
- * @param constraints
- * The constraints declared for the join graph (if any). The
- * appropriate constraints will be applied based on the variables
- * which are known to be bound as of the cutoff join for the last
- * vertex in the path segment.
- * @param pathIsComplete
- * <code>true</code> iff all vertices in the join graph are
- * incorporated into this path.
- * @param sourceSample
- * The input sample for the cutoff join. When this is a one-step
- * estimation of the cardinality of the edge, then this sample is
- * taken from the {@link VertexSample}. When the edge (vSource,
- * vTarget) extends some {@link Path}, then this is taken from
- * the {@link EdgeSample} for that {@link Path}.
- *
- * @return The result of sampling that edge.
- *
- * @throws Exception
- */
+ /**
+ * Cutoff join of the last vertex in the join path.
+ * <p>
+ * <strong>The caller is responsible for protecting against needless
+ * re-sampling.</strong> This includes cases where a sample already exists
+ * at the desired sample limit and cases where the sample is already exact.
+ *
+ * @param queryEngine
+ * The query engine.
+ * @param joinGraph
+ * The pipeline operator that is executing the RTO. This defines
+ * the join graph (vertices, edges, and constraints) and also
+ * provides access to the AST and related metadata required to
+ * execute the join graph.
+ * @param limit
+ * The limit for the cutoff join.
+ * @param path
+ * The path segment, which must include the target vertex as the
+ * last component of the path segment.
+ * @param constraints
+ * The constraints declared for the join graph (if any). The
+ * appropriate constraints will be applied based on the variables
+ * which are known to be bound as of the cutoff join for the last
+ * vertex in the path segment.
+ * @param pathIsComplete
+ * <code>true</code> iff all vertices in the join graph are
+ * incorporated into this path.
+ * @param sourceSample
+ * The input sample for the cutoff join. When this is a one-step
+ * estimation of the cardinality of the edge, then this sample is
+ * taken from the {@link VertexSample}. When the edge (vSource,
+ * vTarget) extends some {@link Path}, then this is taken from
+ * the {@link EdgeSample} for that {@link Path}.
+ *
+ * @return The result of sampling that edge.
+ *
+ * @throws Exception
+ */
static public EdgeSample cutoffJoin(//
final QueryEngine queryEngine,//
+ final JoinGraph joinGraph,//
final int limit,//
final IPredicate<?>[] path,//
final IConstraint[] constraints,//
@@ -757,283 +752,10 @@
final SampleBase sourceSample//
) throws Exception {
- if (path == null)
- throw new IllegalArgumentException();
+ // Note: Delegated to the AST/RTO integration class.
+ return AST2BOpRTO.cutoffJoin(queryEngine, joinGraph, limit, path,
+ constraints, pathIsComplete, sourceSample);
- if (limit <= 0)
- throw new IllegalArgumentException();
-
- // The access path on which the cutoff join will read.
- final IPredicate<?> pred = path[path.length - 1];
-
- if (pred == null)
- throw new IllegalArgumentException();
-
- if (sourceSample == null)
- throw new IllegalArgumentException();
-
- if (sourceSample.getSample() == null)
- throw new IllegalArgumentException();
-
- // Figure out which constraints attach to each predicate. FIXME RTO Replace with StaticAnalysis.
- final IConstraint[][] constraintAttachmentArray = PartitionedJoinGroup
- .getJoinGraphConstraints(path, constraints, null/*knownBound*/,
- pathIsComplete);
-
- // The constraint(s) (if any) for this join.
- final IConstraint[] c = constraintAttachmentArray[path.length - 1];
-
- /*
- * Setup factory for bopIds with reservations for ones already in use.
- */
- final BOpIdFactory idFactory = new BOpIdFactory();
-
- // Reservation for the bopId used by the predicate.
- idFactory.reserve(pred.getId());
-
- // Reservations for the bopIds used by the constraints.
- if (c != null) {
- for (IConstraint x : c) {
- if (log.isTraceEnabled())
- log.trace(Arrays.toString(BOpUtility.getPredIds(path))
- + ": constraint: " + x);
- final Iterator<BOp> itr = BOpUtility
- .preOrderIteratorWithAnnotations(x);
- while (itr.hasNext()) {
- final BOp y = itr.next();
- final Integer anId = (Integer) y
- .getProperty(BOp.Annotations.BOP_ID);
- if (anId != null)
- idFactory.reserve(anId.intValue());
- }
- }
- }
-
- /*
- * Set up a cutoff pipeline join operator which makes an accurate
- * estimate of the #of input solutions consumed and the #of output
- * solutions generated. From that, we can directly compute the join hit
- * ratio.
- *
- * Note: This approach is preferred to injecting a "RowId" column as the
- * estimates are taken based on internal counters in the join operator
- * and the join operator knows how to cutoff evaluation as soon as the
- * limit is satisfied, thus avoiding unnecessary effort.
- */
-
- final int joinId = idFactory.nextId();
- final Map<String, Object> anns = NV.asMap(//
- new NV(BOp.Annotations.BOP_ID, joinId),//
- new NV(PipelineJoin.Annotations.PREDICATE, pred),//
- // Note: does not matter since not executed by the query
- // controller.
- // // disallow parallel evaluation of tasks
- // new NV(PipelineOp.Annotations.MAX_PARALLEL,1),
- // disallow parallel evaluation of chunks.
- new NV(PipelineJoin.Annotations.MAX_PARALLEL_CHUNKS, 0),
- // disable access path coalescing
- new NV( PipelineJoin.Annotations.COALESCE_DUPLICATE_ACCESS_PATHS, false), //
- // pass in constraints on this join.
- new NV(PipelineJoin.Annotations.CONSTRAINTS, c),//
- // cutoff join.
- new NV(PipelineJoin.Annotations.LIMIT, (long) limit),
- /*
- * Note: In order to have an accurate estimate of the
- * join hit ratio we need to make sure that the join
- * operator runs using a single PipelineJoinStats
- * instance which will be visible to us when the query
- * is cutoff. In turn, this implies that the join must
- * be evaluated on the query controller.
- *
- * @todo This implies that sampling of scale-out joins
- * must be done using remote access paths.
- */
- new NV(PipelineJoin.Annotations.SHARED_STATE, true),//
- new NV(PipelineJoin.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.CONTROLLER)//
- );
-
- @SuppressWarnings("unchecked")
- final PipelineJoin<?> joinOp = new PipelineJoin(new BOp[] {}, anns);
-
- final PipelineOp queryOp = joinOp;
-
- // run the cutoff sampling of the edge.
- final UUID queryId = UUID.randomUUID();
- final IRunningQuery runningQuery = queryEngine.eval(//
- queryId,//
- queryOp,//
- null,// attributes
- new LocalChunkMessage(queryEngine, queryId, joinOp
- .getId()/* startId */, -1 /* partitionId */,
- sourceSample.getSample()));
-
- final List<IBindingSet> result = new LinkedList<IBindingSet>();
- try {
- int nresults = 0;
- try {
- IBindingSet bset = null;
- // Figure out the #of source samples consumed.
- final Iterator<IBindingSet> itr = new Dechunkerator<IBindingSet>(
- runningQuery.iterator());
- while (itr.hasNext()) {
- bset = itr.next();
- result.add(bset);
- if (nresults++ >= limit) {
- // Break out if cutoff join over produces!
- break;
- }
- }
- } finally {
- // ensure terminated regardless.
- runningQuery.cancel(true/* mayInterruptIfRunning */);
- }
- } finally {
- // verify no problems.
- if (runningQuery.getCause() != null) {
- // wrap throwable from abnormal termination.
- throw new RuntimeException(runningQuery.getCause());
- }
- }
-
- // The join hit ratio can be computed directly from these stats.
- final PipelineJoinStats joinStats = (PipelineJoinStats) runningQuery
- .getStats().get(joinId);
-
- if (log.isTraceEnabled())
- log.trace(Arrays.toString(BOpUtility.getPredIds(path)) + ": "
- + joinStats.toString());
-
- // #of solutions in.
- final int inputCount = (int) joinStats.inputSolutions.get();
-
- // #of solutions out.
- final long outputCount = joinStats.outputSolutions.get();
-
- // #of solutions out as adjusted for various edge conditions.
- final long adjustedCard;
-
- // cumulative range count of the sampled access paths.
- final long sumRangeCount = joinStats.accessPathRangeCount.get();
-
- final EstimateEnum estimateEnum;
- if (sourceSample.estimateEnum == EstimateEnum.Exact
- && outputCount < limit) {
- /*
- * Note: If the entire source vertex is being fed into the cutoff
- * join and the cutoff join outputCount is LT the limit, then the
- * sample is the actual result of the join. That is, feeding all
- * source solutions into the join gives fewer than the desired
- * number of output solutions.
- */
- estimateEnum = EstimateEnum.Exact;
- adjustedCard = outputCount;
- } else if (inputCount == 1 && outputCount == limit) {
- /*
- * If the inputCount is ONE (1) and the outputCount is the limit,
- * then the estimated cardinality is a lower bound as more than
- * outputCount solutions might be produced by the join when
- * presented with a single input solution.
- *
- * However, this condition suggests that the sum of the sampled
- * range counts is a much better estimate of the cardinality of this
- * join.
- *
- * For example, consider a join feeding a rangeCount of 16 into a
- * rangeCount of 175000. With a limit of 100, we estimated the
- * cardinality at 1600L (lower bound). In fact, the cardinality is
- * 16*175000. This falsely low estimate can cause solutions which
- * are really better to be dropped.
- */
- // replace outputCount with the sum of the sampled range counts.
- adjustedCard = sumRangeCount;
- estimateEnum = EstimateEnum.LowerBound;
- } else if ((sourceSample.estimateEnum != EstimateEnum.Exact)
- /*&& inputCount == Math.min(sourceSample.limit,
- sourceSample.estimatedCardinality) */ && outputCount == 0) {
- /*
- * When the source sample was not exact, the inputCount is EQ to the
- * lesser of the source range count and the source sample limit, and
- * the outputCount is ZERO (0), then feeding in all source solutions
- * is not sufficient to generate any output solutions. In this case,
- * the estimated join hit ratio appears to be zero. However, the
- * estimation of the join hit ratio actually underflowed and the
- * real join hit ratio might be a small non-negative value. A real
- * zero can only be identified by executing the full join.
- *
- * Note: An apparent join hit ratio of zero does NOT imply that the
- * join will be empty (unless the source vertex sample is actually
- * the fully materialized access path - this case is covered above).
- *
- * path sourceCard * f ( in read out limit adjCard) = estCard : sumEstCard joinPath
- * 15 4800L * 0.00 ( 200 200 0 300 0) = 0 : 3633 [ 3 1 6 5 ]
-
- */
- estimateEnum = EstimateEnum.Underflow;
- adjustedCard = outputCount;
- } else {
- estimateEnum = EstimateEnum.Normal;
- adjustedCard = outputCount;
- }
-
- /*
- * The #of tuples read from the sampled access paths. This is part of
- * the cost of the join path, even though it is not part of the expected
- * cardinality of the cutoff join.
- *
- * Note: While IOs is a better predictor of latency, it is possible to
- * choose a pipelined join versus a hash join once the query plan has
- * been decided. Their IO provides are both correlated to the #of tuples
- * read.
- */
- final long tuplesRead = joinStats.accessPathUnitsIn.get();
-
- /*
- * Compute the hit-join ratio based on the adjusted cardinality
- * estimate.
- */
- final double f = adjustedCard == 0 ? 0
- : (adjustedCard / (double) inputCount);
-// final double f = outputCount == 0 ? 0
-// : (outputCount / (double) inputCount);
-
- // estimated output cardinality of fully executed operator.
- final long estCard = (long) (sourceSample.estCard * f);
-
- /*
- * estimated tuples read for fully executed operator
- *
- * TODO The actual IOs depend on the join type (hash join versus
- * pipeline join) and whether or not the file has index order (segment
- * versus journal). A hash join will read once on the AP. A pipeline
- * join will read once per input solution. A key-range read on a segment
- * uses multi-block IO while a key-range read on a journal uses random
- * IO. Also, remote access path reads are more expensive than sharded
- * or hash partitioned access path reads in scale-out.
- */
- final long estRead = (long) (sumRangeCount * f);
-
- final EdgeSample edgeSample = new EdgeSample(//
- sourceSample,//
- inputCount,//
- tuplesRead,//
- sumRangeCount,//
- outputCount, //
- adjustedCard,//
- f, //
- // args to SampleBase
- estCard, // estimated output cardinality if fully executed.
- estRead, // estimated tuples read if fully executed.
- limit, //
- estimateEnum,//
- result.toArray(new IBindingSet[result.size()]));
-
- if (log.isDebugEnabled())
- log.debug(Arrays.toString(BOpUtility.getPredIds(path))
- + ": newSample=" + edgeSample);
-
- return edgeSample;
-
}
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/SampleBase.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/SampleBase.java 2014-01-09 20:47:55 UTC (rev 7754)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/SampleBase.java 2014-01-09 21:55:07 UTC (rev 7755)
@@ -32,6 +32,7 @@
import org.apache.log4j.Logger;
import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.engine.IChunkMessage;
import com.bigdata.rwstore.sector.IMemoryManager;
/**
@@ -108,8 +109,11 @@
*
* @return The sampled solution set -or- <code>null</code> if it has been
* released.
+ *
+ * TODO Wrap up as an {@link IChunkMessage} so we can store this on
+ * the native heap?
*/
- IBindingSet[] getSample() {
+ public IBindingSet[] getSample() {
return sampleRef.get();
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpBase.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpBase.java 2014-01-09 20:47:55 UTC (rev 7754)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpBase.java 2014-01-09 21:55:07 UTC (rev 7755)
@@ -94,6 +94,16 @@
*/
String SCOPE = AST2BOpBase.class.getName() + ".scope";
+ /**
+ * Boolean annotation indicates whether the generated JOIN is simple (a
+ * single JOIN operator with optional constraints but without any
+ * variable materialization requirements) or complex (a JOIN operator
+ * associated with at least one constraint which requires the
+ * materialization of variables that are not already known to be
+ * materialized).
+ */
+ String SIMPLE_JOIN = AST2BOpBase.class.getName() + ".simpleJoin";
+
/*
* Query planner and cost estimates.
*/
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpFilters.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpFilters.java 2014-01-09 20:47:55 UTC (rev 7754)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpFilters.java 2014-01-09 21:55:07 UTC (rev 7755)
@@ -593,10 +593,13 @@
* @param needsMaterialization
* A map providing for each constraint the set of variables which
* must be materialized before that constraint can be evaluated.
- * This map is populated as a side-effect.
+ * This map is populated as a side-effect. It will be empty iff
+ * there are no constraints that might or must require variable
+ * materialization.
*
* @return Constraints which can (or might) be able to run attached to that
- * join.
+ * join -or- <code>null</code> iff there are no constraints that can
+ * be attached to the join.
*/
static protected IConstraint[] getJoinConstraints(
final Collection<IConstraint> constraints,
Modified: branch...
[truncated message content] |
|
From: <tho...@us...> - 2014-01-10 21:10:54
|
Revision: 7761
http://bigdata.svn.sourceforge.net/bigdata/?rev=7761&view=rev
Author: thompsonbry
Date: 2014-01-10 21:10:45 +0000 (Fri, 10 Jan 2014)
Log Message:
-----------
Checkpoint on the RTO development.
See #64.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JGraph.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/Path.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/joinGraph/rto/TestJoinGraph.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/AbstractRTOTestCase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/BSBM-Q4.rq
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/TestAll.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/TestRTO_BSBM.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java 2014-01-10 17:25:56 UTC (rev 7760)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java 2014-01-10 21:10:45 UTC (rev 7761)
@@ -42,6 +42,8 @@
import java.util.concurrent.RejectedExecutionException;
import java.util.concurrent.atomic.AtomicInteger;
+import junit.framework.AssertionFailedError;
+
import org.apache.log4j.Logger;
import com.bigdata.bop.BOp;
@@ -1169,7 +1171,9 @@
stats = op.newStats();
// log.warn("bopId=" + bopId + ", stats=" + stats);
}
- assert stats != null : "No stats: op=" + op;
+ if (stats == null) {
+ throw new AssertionError("No stats: op=" + op);
+ }
// // The groupId (if any) for this operator.
// final Integer fromGroupId = (Integer) op
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JGraph.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JGraph.java 2014-01-10 17:25:56 UTC (rev 7760)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JGraph.java 2014-01-10 21:10:45 UTC (rev 7761)
@@ -1387,7 +1387,7 @@
* those get applied when we evaluate the cutoff joins from one
* vertex to another.
*/
- public void sampleAllVertices(final QueryEngine queryEngine, final int limit) {
+ private void sampleAllVertices(final QueryEngine queryEngine, final int limit) {
final Map<Vertex, AtomicInteger> vertexLimit = new LinkedHashMap<Vertex, AtomicInteger>();
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java 2014-01-10 17:25:56 UTC (rev 7760)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java 2014-01-10 21:10:45 UTC (rev 7761)
@@ -32,10 +32,7 @@
import java.util.Set;
import java.util.concurrent.Callable;
import java.util.concurrent.FutureTask;
-import java.util.concurrent.TimeUnit;
-import org.apache.log4j.Logger;
-
import com.bigdata.bop.BOp;
import com.bigdata.bop.BOpContext;
import com.bigdata.bop.BOpUtility;
@@ -50,14 +47,12 @@
import com.bigdata.bop.ap.SampleIndex.SampleType;
import com.bigdata.bop.controller.AbstractSubqueryOp;
import com.bigdata.bop.engine.AbstractRunningQuery;
-import com.bigdata.bop.engine.BOpStats;
import com.bigdata.bop.engine.IRunningQuery;
import com.bigdata.bop.engine.QueryEngine;
-import com.bigdata.rdf.sparql.ast.IJoinNode;
import com.bigdata.rdf.sparql.ast.JoinGroupNode;
import com.bigdata.rdf.sparql.ast.eval.AST2BOpContext;
import com.bigdata.rdf.sparql.ast.eval.AST2BOpRTO;
-import com.bigdata.rdf.sparql.ast.optimizers.IASTOptimizer;
+import com.bigdata.rdf.sparql.ast.eval.AST2BOpUtility;
import com.bigdata.util.NT;
import com.bigdata.util.concurrent.Haltable;
@@ -65,13 +60,33 @@
/**
* A join graph with annotations for estimated cardinality and other details in
- * support of runtime query optimization. A join graph is a collection of
- * relations and joins which connect those relations. This boils down to a
- * collection of {@link IPredicate}s (selects on relations), shared variables
- * (which identify joins), and {@link IConstraint}s (which limit solutions).
- * Operators other than standard joins (including optional joins, sort, order
- * by, etc.) must be handled downstream from the join graph in a "tail plan".
+ * support of runtime query optimization. A join graph is a collection of access
+ * paths reading on relations (the vertices of the join graph) and joins which
+ * connect those relations (the edges of the join graph). This boils down to a
+ * collection of {@link IPredicate}s (access paths reading on on relations),
+ * shared variables (which identify joins), and {@link IConstraint}s (which may
+ * reject some solutions for those joins). Operators other than standard joins
+ * (including optional joins, sort, order by, etc.) must be handled downstream
+ * from the join graph in a "tail plan".
+ * <p>
+ * The {@link JoinGraph} operator works in two phases. On its first invocation,
+ * it constructs a {@link JGraph join graph} and identifies a join path having a
+ * low cost join ordering. This join path is converted into a query plan and set
+ * as the {@link Attributes#QUERY_PLAN} attribute on the {@link IRunningQuery}.
+ * The upstream solutions are then flooded into sub-query that executes the
+ * chosen query plan. The solutions from the sub-query are simply copied to the
+ * output sink of the {@link JoinGraph} operator. Once the query plan has been
+ * identified by the first invocation, subsequent invocations of this operator
+ * simply push more data into the sub-query using the pre-identified query plan.
*
+ * TODO This approach amounts to bottom-up evaluation of the {@link JGraph}.
+ * Thus, the RTO is not using information from the upstream query when it
+ * decides on a query plan. Therefore, we could lift-out the RTO sections of the
+ * query into named subqueries, run them first in parallel, and then INCLUDE
+ * their results into the main query. This would require an AST optimizer to
+ * modify the AST. (Currently the RTO is integrated when the query plan is
+ * generated in {@link AST2BOpUtility} rather than as an AST optimizer.)
+ *
* @see http://arxiv.org/PS_cache/arxiv/pdf/0810/0810.4809v1.pdf, XQuery Join
* Graph Isolation.
*
@@ -87,19 +102,19 @@
private static final long serialVersionUID = 1L;
- private static final transient Logger log = Logger
- .getLogger(JoinGraph.class);
+// private static final transient Logger log = Logger
+// .getLogger(JoinGraph.class);
/**
* Known annotations.
*/
public interface Annotations extends PipelineOp.Annotations {
- /**
- * The variables to be projected out of the join graph (optional). When
- * <code>null</code>, all variables will be projected out.
- */
- String SELECTED = JoinGraph.class.getName() + ".selected";
+// /**
+// * The variables to be projected out of the join graph (optional). When
+// * <code>null</code>, all variables will be projected out.
+// */
+// String SELECTED = JoinGraph.class.getName() + ".selected";
/**
* The vertices of the join graph, expressed an an {@link IPredicate}[]
@@ -147,44 +162,12 @@
/**
* The set of variables that are known to have already been materialized
* in the context in which the RTO was invoked.
- *
- * FIXME In order to support left-to-right evaluation fully, the
- * {@link JGraph} needs to accept this, track it as it binds variables,
- * and pass it through when doing cutoff joins to avoid pipeline
- * materialization steps for variables that are already known to be
- * materialized. Otherwise the RTO will assume that it needs to
- * materialize everything that needs to be materialized for a FILTER and
- * thus do too much work (which is basically the assumption of bottom-up
- * evaluation, or if you prefer that it is executing in its own little
- * world).
*/
String DONE_SET = JoinGraph.class.getName() + ".doneSet";
-// /**
-// * The query hints from the dominating AST node (if any). These query
-// * hints will be passed through and made available when we compile the
-// * query plan once the RTO has decided on the join ordering. While the
-// * RTO is running, it needs to override many of the query hints for the
-// * {@link IPredicate}s, {@link PipelineJoin}s, etc. in order to ensure
-// * that the cutoff evaluation semantics are correctly applied while it
-// * is exploring the plan state space for the join graph.
-// */
-// String AST_QUERY_HINTS = JoinGraph.class.getName() + ".astQueryHints";
-
/**
* The AST {@link JoinGroupNode} for the joins and filters that we are
* running through the RTO (required).
- *
- * FIXME This should be set by an ASTRTOOptimizer. That class should
- * rewrite the original join group, replacing some set of joins with a
- * JoinGraphNode which implements {@link IJoinNode} and gets hooked into
- * AST2BOpUtility#convertJoinGroup() normally rather than through
- * expectional processing. This will simplify the code and adhere to the
- * general {@link IASTOptimizer} pattern and avoid problems with cloning
- * children out of the {@link JoinGroupNode} when we set it up to run
- * the RTO. [Eventually, we will need to pass this in rather than the
- * {@link IPredicate}[] in order to handle JOINs that are not SPs, e.g.,
- * sub-selects, etc.]
*/
String JOIN_GROUP = JoinGraph.class.getName() + ".joinGroup";
@@ -207,9 +190,6 @@
*
* @author <a href="mailto:tho...@us...">Bryan
* Thompson</a>
- *
- * TODO This could also be put on a {@link BOpStats} interface,
- * which is the other way for accessing shared state.
*/
public interface Attributes {
@@ -236,15 +216,15 @@
* JoinGraph operator annotations.
*/
- /**
- * @see Annotations#SELECTED
- */
- public IVariable<?>[] getSelected() {
+// /**
+// * @see Annotations#SELECTED
+// */
+// public IVariable<?>[] getSelected() {
+//
+// return (IVariable[]) getRequiredProperty(Annotations.SELECTED);
+//
+// }
- return (IVariable[]) getRequiredProperty(Annotations.SELECTED);
-
- }
-
/**
* @see Annotations#VERTICES
*/
@@ -462,72 +442,99 @@
}
- /**
- * {@inheritDoc}
- *
- * FIXME When run as sub-query, we need to fix point the upstream
- * solutions and then flood them into the join graph. Samples of the
- * known bound variables can be pulled from those initial solutions.
- */
@Override
public Void call() throws Exception {
- final long begin = System.nanoTime();
-
- // Create the join graph.
- final JGraph g = new JGraph(JoinGraph.this);
+ if (getQueryPlan(context.getRunningQuery()) == null) {
+
+ /*
+ * Use the RTO to generate a query plan.
+ *
+ * TODO Make sure that the JoinGraph can not be triggered
+ * concurrently, e.g., that the CONTROLLER attribute prevents
+ * concurrent evaluation, just like MAX_PARALLEL.
+ */
+
+ // final long begin = System.nanoTime();
- /*
- * This map is used to associate join path segments (expressed as an
- * ordered array of bopIds) with edge sample to avoid redundant effort.
- *
- * FIXME RTO: HEAP MANAGMENT : This map holds references to the cutoff
- * join samples. To ensure that the map has the minimum heap footprint,
- * it must be scanned each time we prune the set of active paths and any
- * entry which is not a prefix of an active path should be removed.
- *
- * TODO RTO: MEMORY MANAGER : When an entry is cleared from this map,
- * the corresponding allocation in the memory manager (if any) must be
- * released. The life cycle of the map needs to be bracketed by a
- * try/finally in order to ensure that all allocations associated with
- * the map are released no later than when we leave the lexicon scope of
- * that clause.
- */
- final Map<PathIds, EdgeSample> edgeSamples = new LinkedHashMap<PathIds, EdgeSample>();
+ // Create the join graph.
+ final JGraph g = new JGraph(JoinGraph.this);
- // Find the best join path.
- final Path path = g.runtimeOptimizer(context.getRunningQuery()
- .getQueryEngine(), getLimit(), getNEdges(), edgeSamples);
+ /*
+ * This map is used to associate join path segments (expressed
+ * as an ordered array of bopIds) with edge sample to avoid
+ * redundant effort.
+ */
+ final Map<PathIds, EdgeSample> edgeSamples = new LinkedHashMap<PathIds, EdgeSample>();
- // Set attribute for the join path result.
- setPath(context.getRunningQuery(), path);
+ // Find the best join path.
+ final Path path = g
+ .runtimeOptimizer(context.getRunningQuery()
+ .getQueryEngine(), getLimit(), getNEdges(),
+ edgeSamples);
- // Set attribute for the join path samples.
- setSamples(context.getRunningQuery(), edgeSamples);
+ /*
+ * Release samples.
+ *
+ * TODO If we have fully sampled some vertices or edges, then we
+ * could replace the JOIN with the sample. For this to work, we
+ * would need to access path that could read the sample and we
+ * would have to NOT release the samples until the RTO was done
+ * executing sub-queries against the generated query plan. Since
+ * we can flow multiple chunks into the sub-query, this amounts
+ * to having a LAST_PASS annotation.
+ */
+
+ for (EdgeSample s : edgeSamples.values()) {
- final long mark = System.nanoTime();
-
- final long elapsed_queryOptimizer = mark - begin;
-
- /*
- * Generate the query from the selected join path.
- */
- final PipelineOp queryOp = AST2BOpRTO.compileJoinGraph(context
- .getRunningQuery().getQueryEngine(), JoinGraph.this, path);
+ s.releaseSample();
+
+ }
+
+ for (Vertex v : g.getVertices()) {
- // Set attribute for the join path samples.
- setQueryPlan(context.getRunningQuery(), queryOp);
+ if (v.sample != null) {
+ v.sample.releaseSample();
+
+ }
+
+ }
+
+ // Set attribute for the join path result.
+ setPath(context.getRunningQuery(), path);
+ // Set attribute for the join path samples.
+ setSamples(context.getRunningQuery(), edgeSamples);
+
+ // final long mark = System.nanoTime();
+ //
+ // final long elapsed_queryOptimizer = mark - begin;
+
+ /*
+ * Generate the query from the selected join path.
+ */
+ final PipelineOp queryOp = AST2BOpRTO.compileJoinGraph(context
+ .getRunningQuery().getQueryEngine(), JoinGraph.this,
+ path);
+
+ // Set attribute for the join path samples.
+ setQueryPlan(context.getRunningQuery(), queryOp);
+
+ }
+
+ // The query plan.
+ final PipelineOp queryOp = getQueryPlan(context.getRunningQuery());
+
// Run the query, blocking until it is done.
JoinGraph.runSubquery(context, queryOp);
- final long elapsed_queryExecution = System.nanoTime() - mark;
-
- if (log.isInfoEnabled())
- log.info("RTO: queryOptimizer="
- + TimeUnit.NANOSECONDS.toMillis(elapsed_queryOptimizer)
- + ", queryExecution="
- + TimeUnit.NANOSECONDS.toMillis(elapsed_queryExecution));
+// final long elapsed_queryExecution = System.nanoTime() - mark;
+//
+// if (log.isInfoEnabled())
+// log.info("RTO: queryOptimizer="
+// + TimeUnit.NANOSECONDS.toMillis(elapsed_queryOptimizer)
+// + ", queryExecution="
+// + TimeUnit.NANOSECONDS.toMillis(elapsed_queryExecution));
return null;
@@ -542,42 +549,48 @@
* subquery. Therefore we have to take appropriate care to ensure that the
* results are copied out of the subquery and into the parent query. See
* {@link AbstractSubqueryOp} for how this is done.
- *
- * @throws Exception
- *
- * @todo When we execute the query, we should clear the references to the
- * samples (unless they are exact, in which case they can be used as
- * is) in order to release memory associated with those samples if the
- * query is long running. Samples must be held until we have
- * identified the final join path since each vertex will be used by
- * each maximum length join path and we use the samples from the
- * vertices to re-sample the surviving join paths in each round. [In
- * fact, the samples are not being provided to this evaluation context
- * right now.]
- *
- * @todo If there are source binding sets then they need to be applied above
- * (when we are sampling) and below (when we evaluate the selected
- * join path).
*/
static private void runSubquery(
final BOpContext<IBindingSet> parentContext,
final PipelineOp queryOp) throws Exception {
+ if(parentContext==null)
+ throw new IllegalArgumentException();
+
+ if(queryOp==null)
+ throw new IllegalArgumentException();
+
final QueryEngine queryEngine = parentContext.getRunningQuery()
.getQueryEngine();
/*
- * Run the query.
- *
- * TODO Pass in the source binding sets here and also when sampling the
- * vertices? Otherwise it is as if we are doing bottom-up evaluation (in
- * which case the doneSet should be empty on entry).
+ * Run the sub-query.
*/
ICloseableIterator<IBindingSet[]> subquerySolutionItr = null;
- final IRunningQuery runningSubquery = queryEngine.eval(queryOp);
+ // Fully materialize the upstream solutions.
+ final IBindingSet[] bindingSets = BOpUtility.toArray(
+ parentContext.getSource(), parentContext.getStats());
+ /*
+ * Run on all available upstream solutions.
+ *
+ * Note: The subquery will run for each chunk of upstream solutions, so
+ * it could make sense to increase the vector size or to collect all
+ * upstream solutions into a SolutionSet and then flood them into the
+ * sub-query.
+ *
+ * Note: We do not need to do a hash join with the output of the
+ * sub-query. This amounts to pipelined evaluation. Solutions flow into
+ * a subquery and then back out. The only reason for a hash join would
+ * be if we project in only a subset of the variables that were in scope
+ * in the parent context and then needed to pick up the correlated
+ * variables after running the query plan generated by the RTO.
+ */
+ final IRunningQuery runningSubquery = queryEngine.eval(queryOp,
+ bindingSets);
+
try {
// Declare the child query to the parent.
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/Path.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/Path.java 2014-01-10 17:25:56 UTC (rev 7760)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/Path.java 2014-01-10 21:10:45 UTC (rev 7761)
@@ -27,8 +27,6 @@
import java.util.Collections;
import java.util.List;
-import org.apache.log4j.Logger;
-
import com.bigdata.bop.BOp;
import com.bigdata.bop.BOpUtility;
import com.bigdata.bop.IConstraint;
@@ -87,7 +85,7 @@
*/
public class Path {
- private static final transient Logger log = Logger.getLogger(Path.class);
+// private static final transient Logger log = Logger.getLogger(Path.class);
/**
* An ordered list of the vertices in the {@link Path}.
@@ -180,6 +178,13 @@
* the JGraph trace appropriately. [Refactor into an IPathCost
* interface. It should have visibility into the full path and also
* allow visibility into the vertex cost for generality.]
+ *
+ * TODO Add a cost function API, e.g., IPathCost. This gets passed
+ * into Path to compute a score. We also compute a score for a
+ * vertex. Add query hints for both so we can control the behavior.
+ * The default should be estCard, but estRead or a weighted
+ * combination of estCard and estRead are also possible cost
+ * functions.
*/
private static long getCost(final long sumEstRead, final long sumEstCard) {
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/joinGraph/rto/TestJoinGraph.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/joinGraph/rto/TestJoinGraph.java 2014-01-10 17:25:56 UTC (rev 7760)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/joinGraph/rto/TestJoinGraph.java 2014-01-10 21:10:45 UTC (rev 7761)
@@ -76,7 +76,7 @@
};
final IConstraint[] constraints = null;
final JoinGraph joinGraph = new JoinGraph(new BOp[0],//
- new NV(JoinGraph.Annotations.SELECTED, new IVariable[]{}),//
+// new NV(JoinGraph.Annotations.SELECTED, new IVariable[]{}),//
new NV(JoinGraph.Annotations.VERTICES, vertices),//
new NV(JoinGraph.Annotations.CONTROLLER, true), //
new NV(JoinGraph.Annotations.EVALUATION_CONTEXT,
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java 2014-01-10 17:25:56 UTC (rev 7760)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java 2014-01-10 21:10:45 UTC (rev 7761)
@@ -75,13 +75,12 @@
import com.bigdata.bop.joinGraph.rto.SampleBase;
import com.bigdata.bop.joinGraph.rto.VertexSample;
import com.bigdata.bop.rdf.join.ChunkedMaterializationOp;
+import com.bigdata.bop.rdf.join.DataSetJoin;
import com.bigdata.bop.solutions.MemorySortOp;
-import com.bigdata.bop.solutions.ProjectionOp;
import com.bigdata.bop.solutions.SliceOp;
import com.bigdata.journal.IIndexManager;
import com.bigdata.rdf.internal.IV;
import com.bigdata.rdf.sparql.ast.ASTContainer;
-import com.bigdata.rdf.sparql.ast.IBindingProducerNode;
import com.bigdata.rdf.sparql.ast.IGroupMemberNode;
import com.bigdata.rdf.sparql.ast.JoinGroupNode;
import com.bigdata.rdf.sparql.ast.QueryHints;
@@ -92,6 +91,11 @@
/**
* Integration with the Runtime Optimizer (RTO).
+ * <p>
+ * Note: The RTO currently uses bottom-up evaluation to solve the join graph and
+ * generate a sub-query plan with an optimized join ordering. It uses
+ * left-to-right evaluation to pass pipeline solutions through the optimized
+ * subquery.
*
* @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/64">Runtime
* Query Optimization</a>
@@ -105,11 +109,6 @@
* @see JGraph
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- *
- * TODO Can the RTO give us information about the locality of joins that
- * we could use to decide on vectoring (chunkSize) for those joins or to
- * decide when to use a hash join against an access path as opposed to a
- * nested index join?
*/
public class AST2BOpRTO extends AST2BOpJoins {
@@ -180,33 +179,6 @@
static private final boolean onlySPs = true;
/**
- * When <code>true</code>, the RTO will be applied as in we were doing
- * bottom-up query optimization. In this case, it WILL NOT receive any
- * solutions from the upstream operators in the pipeline when it performs
- * its runtime sampling and it will ignore the <code>doneSet</code> for the
- * context in which it is invoked. When run in this manner, the RTO *could*
- * be run before the main query is executed. The only way to facilitate this
- * at this time would be to lift out the joins on which the RTO would be run
- * into a named subquery and then optimize that named subquery before the
- * rest of the query.
- * <p>
- * When <code>false</code>, the RTO solutions from upstream operators will
- * flow into the RTO.
- *
- * TODO We could still pass in exogenous solutions for bottom up evaluation.
- * This would help constraint the RTOs exploration.
- *
- * TODO The RTO is not operating 100% in either an left-to-right or a
- * bottom-up fashion, primarily because we are not passing in either
- * exogenous bindings or meaningfully using the bindings from the upstream
- * operator when exploring the join graph. In fact, the RTO could accept a
- * new sample from the upstream operator in each iteration drawing from
- * amoung those solutions which had already been materialized by the
- * upstream operator.
- */
- static private final boolean bottomUp = true;
-
- /**
* When <code>true</code>, even simple JOINs will run through the code path
* for the evaluation of complex joins.
* <p>
@@ -225,7 +197,7 @@
* @see #runSimpleJoin(QueryEngine, SampleBase, int, PipelineJoin)
* @see #runComplexJoin(QueryEngine, SampleBase, int, PipelineOp)
*
- * FIXME RTO: Measure performance when using the complex join code path
+ * TODO RTO: Measure performance when using the complex join code path
* as opposed to the simple join code path. If there is a performance
* win for the simple join code path, then set this to false in
* committed code. If not, then we might as well run everything in the
@@ -235,7 +207,7 @@
* somewhat in the end result that they produce. This can be observed
* in BSBM Q1 on the pc100 data set and the BAR query.
*/
- static final boolean runAllJoinsAsComplexJoins = false;
+ static final boolean runAllJoinsAsComplexJoins = false;
/**
* Inspect the remainder of the join group. If we can isolate a join graph
@@ -400,37 +372,6 @@
}
- /*
- * Figure out which variables are projected out of the RTO.
- *
- * TODO This should only include things that are not reused later in the
- * query.
- */
- final Set<IVariable<?>> selectVars = new LinkedHashSet<IVariable<?>>();
- {
-
- for (IGroupMemberNode child : rtoJoinGroup.getChildren()) {
-
- if (!(child instanceof IBindingProducerNode))
- continue;
-
- // Note: recursive only matters for complex nodes, not SPs.
- ctx.sa.getDefinitelyProducedBindings(
- (IBindingProducerNode) child, selectVars, true/* recursive */);
-
- }
-
- }
-
- /*
- * FIXME RTO: Sub-Groups: When running the RTO as anything other than
- * the top-level join group in the query plan and for the *FIRST* joins
- * in the query plan, we need to flow in any solutions that are already
- * in the pipeline (unless we are going to run the RTO "bottom up") and
- * build a hash index. When the hash index is ready, we can execute the
- * join group.
- */
-
final SampleType sampleType = joinGroup.getProperty(
QueryHints.RTO_SAMPLE_TYPE, QueryHints.DEFAULT_RTO_SAMPLE_TYPE);
@@ -447,8 +388,8 @@
new NV(BOp.Annotations.CONTROLLER, true),// Drop "CONTROLLER" annotation?
// new NV(PipelineOp.Annotations.MAX_PARALLEL, 1),//
// new NV(PipelineOp.Annotations.LAST_PASS, true),// required
- new NV(JoinGraph.Annotations.SELECTED, selectVars
- .toArray(new IVariable[selectVars.size()])),//
+// new NV(JoinGraph.Annotations.SELECTED, selectVars
+// .toArray(new IVariable[selectVars.size()])),//
new NV(JoinGraph.Annotations.VERTICES,
preds.toArray(new Predicate[preds.size()])),//
new NV(JoinGraph.Annotations.CONSTRAINTS, constraints
@@ -495,32 +436,12 @@
if (path == null)
throw new IllegalArgumentException();
- final IVariable<?>[] selected = joinGraph.getSelected();
+// final IVariable<?>[] selected = joinGraph.getSelected();
final IPredicate<?>[] predicates = path.getPredicates();
final IConstraint[] constraints = joinGraph.getConstraints();
-
-// if (onlySimpleJoins) {
-//
-// /*
-// * This is the old code. It does not handle variable materialization
-// * for filters.
-// */
-//
-// // Factory avoids reuse of bopIds assigned to the predicates.
-// final BOpIdFactory idFactory = new BOpIdFactory();
-//
-// return PartitionedJoinGroup.getQuery(idFactory,
-// false/* distinct */, selected, predicates, constraints);
-//
-// }
- /*
- * FIXME RTO: doneSet: The RTO is ignoring the doneSet so it always runs
- * all materialization steps even if some variable is known to be
- * materialized on entry.
- */
final Set<IVariable<?>> doneSet = joinGraph.getDoneSet();
/*
@@ -584,16 +505,16 @@
}
- if (selected != null && selected.length != 0) {
-
- // Drop variables that are not projected out.
- left = applyQueryHints(new ProjectionOp(//
- leftOrEmpty(left), //
- new NV(ProjectionOp.Annotations.BOP_ID, idFactory.nextId()),//
- new NV(ProjectionOp.Annotations.SELECT, selected)//
- ), rtoJoinGroup, ctx);
-
- }
+// if (selected != null && selected.length != 0) {
+//
+// // Drop variables that are not projected out.
+// left = applyQueryHints(new ProjectionOp(//
+// leftOrEmpty(left), //
+// new NV(ProjectionOp.Annotations.BOP_ID, idFactory.nextId()),//
+// new NV(ProjectionOp.Annotations.SELECT, selected)//
+// ), rtoJoinGroup, ctx);
+//
+// }
return left;
@@ -729,27 +650,6 @@
* the {@link EdgeSample} for that {@link Path}.
*
* @return The result of sampling that edge.
- *
- * TODO TESTS: Provide test coverage for running queries with
- * complex FILTERs (triples mode is Ok).
- *
- * TODO TESTS: Test with FILTERs that can not run until after all
- * joins. Such filters are only attached when the [pathIsComplete]
- * flag is set. This might only occur when we have FILTERs that
- * depend on variables that are only "maybe" bound by an OPTIONAL
- * join.
- *
- * TODO TESTS: Quads mode tests. We need to look in depth at how the
- * quads mode access paths are evaluated. There are several
- * different conditions. We need to look at each condition and at
- * whether and how it can be made compatible with cutoff evaluation.
- * (This is somewhat similar to the old scan+filter versus nested
- * query debate on quads mode joins.)
- *
- * TODO TESTS: Scale-out tests. For scale-out, we need to either
- * mark the join's evaluation context based on whether or not the
- * access path is local or remote (and whether the index is
- * key-range distributed or hash partitioned).
*/
static public EdgeSample cutoffJoin(//
final QueryEngine queryEngine,//
@@ -900,7 +800,11 @@
* that is known to have been materialized based on an analysis of the
* join path (as executed) up to this point in the path. This will let
* us potentially do less work. This will require tracking the doneSet
- * in the Path and passing the Path into cutoffJoin().
+ * in the Path and passing the Path into cutoffJoin(). The simplest way
+ * to manage this is to just annotate the Path as we go, which means
+ * making the Path more AST aware - or at least doneSet aware. Or we can
+ * just apply the analysis to each step in the path to figure out what
+ * is done each time we setup cutoff evaluation of an operator.
*/
final Set<IVariable<?>> doneSet = new LinkedHashSet<IVariable<?>>(
joinGraph.getDoneSet());
@@ -1101,6 +1005,19 @@
* the manner in which the query plan is constructed and the
* parallelism in the query plan. Any parallelism or reordering
* will trip this error.
+ *
+ * TODO If we hit the {@link OutOfOrderEvaluationException} for
+ * some kinds of access paths quads mode access paths, then we
+ * might need to look at the {@link DataSetJoin} in more depth
+ * and the way in which named graph and default graph joins are
+ * being executed for both local and scale-out deployments. One
+ * fall back position is to feed the input solutions in one at a
+ * time in different running queries. This will give us the
+ * exact output cardinality for a given source solution while
+ * preserving parallel evaluation over some chunk of source
+ * solutions. However, this approach can do too much work and
+ * will incur more overhead than injecting a rowid column into
+ * the source solutions.
*/
private static EdgeSample runComplexJoin(//
final QueryEngine queryEngine,//
@@ -1212,14 +1129,6 @@
final PipelineJoinStats joinStats = (PipelineJoinStats) runningQuery
.getStats().get(joinOp.getId());
- /*
- * TODO It would be interesting to see the stats on each operator in the
- * plan for each join sampled. We would be able to observe that in the
- * EXPLAIN view if we attached the IRunningQuery for the cutoff
- * evaluation to the parent query. However, this should only be enabled
- * in a mode for gruesome detail since we evaluate a LOT of cutoff joins
- * when running the RTO on a single join graph.
- */
if (log.isTraceEnabled())
log.trace(//Arrays.toString(BOpUtility.getPredIds(predicates)) + ": "+
"join::" + joinStats);
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/AbstractRTOTestCase.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/AbstractRTOTestCase.java 2014-01-10 17:25:56 UTC (rev 7760)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/AbstractRTOTestCase.java 2014-01-10 21:10:45 UTC (rev 7761)
@@ -91,6 +91,25 @@
*
* TODO Add some govtrack queries. Those queries use quads mode and have a
* lot of interesting query constructions.
+ *
+ * TODO TESTS: Provide test coverage for running queries with complex
+ * FILTERs (triples mode is Ok).
+ *
+ * TODO TESTS: Test with FILTERs that can not run until after all joins.
+ * Such filters are only attached when the [pathIsComplete] flag is set.
+ * This might only occur when we have FILTERs that depend on variables that
+ * are only "maybe" bound by an OPTIONAL join.
+ *
+ * TODO TESTS: Quads mode tests. We need to look in depth at how the quads
+ * mode access paths are evaluated. There are several different conditions.
+ * We need to look at each condition and at whether and how it can be made
+ * compatible with cutoff evaluation. (This is somewhat similar to the old
+ * scan+filter versus nested query debate on quads mode joins.)
+ *
+ * TODO TESTS: Scale-out tests. For scale-out, we need to either mark the
+ * join's evaluation context based on whether or not the access path is
+ * local or remote (and whether the index is key-range distributed or hash
+ * partitioned).
*/
public class AbstractRTOTestCase extends AbstractDataDrivenSPARQLTestCase {
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/BSBM-Q4.rq
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/BSBM-Q4.rq 2014-01-10 17:25:56 UTC (rev 7760)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/BSBM-Q4.rq 2014-01-10 21:10:45 UTC (rev 7761)
@@ -8,13 +8,17 @@
SELECT DISTINCT ?product ?label ?propertyTextual
WHERE {
+ {
+
+ # Note: The RTO is only applied to one of the join groups to make the
+ # test code simpler.
+ #
# Control all RTO parameters for repeatable behavior.
- hint:Query hint:optimizer "Runtime".
- hint:Query hint:RTO-sampleType "DENSE".
- hint:Query hint:RTO-limit "100".
- hint:Query hint:RTO-nedges "1".
+ hint:Group hint:optimizer "Runtime".
+ hint:Group hint:RTO-sampleType "DENSE".
+ hint:Group hint:RTO-limit "100".
+ hint:Group hint:RTO-nedges "1".
- {
?product rdfs:label ?label .
?product rdf:type <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/ProductType19> .
?product bsbm:productFeature <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/ProductFeature158> .
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/TestAll.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/TestAll.java 2014-01-10 17:25:56 UTC (rev 7760)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/TestAll.java 2014-01-10 21:10:45 UTC (rev 7761)
@@ -64,12 +64,20 @@
// LUBM test suite.
suite.addTestSuite(TestRTO_LUBM.class);
- // BSBM test suite.
+ // BSBM test suite: TODO Add BSBM BI tests.
suite.addTestSuite(TestRTO_BSBM.class);
- // 'barData' test suite.
+ // 'barData' test suite (quads mode).
suite.addTestSuite(TestRTO_BAR.class);
+ /*
+ * FOAF test suite (quads mode).
+ *
+ * TODO This test suite is disabled since queries are not complex enough
+ * to run the RTO (we need at least required joins).
+ */
+// suite.addTestSuite(TestRTO_FOAF.class);
+
return suite;
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/TestRTO_BSBM.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/TestRTO_BSBM.java 2014-01-10 17:25:56 UTC (rev 7760)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/TestRTO_BSBM.java 2014-01-10 21:10:45 UTC (rev 7761)
@@ -180,9 +180,16 @@
/**
* BSBM Q4 against pc100.
+ * <p>
+ * Note: This query has TWO join groups that are sufficiently complex to run
+ * the RTO. However, only one of the join groups is marked for RTO
+ * optimization in order to keep the test harness simple. The test harness
+ * assumes that there is a single JOIN group that is optimized by the RTO
+ * and then verifies the join ordering within that join group. The test
+ * harness breaks if there is more than one join group optimized by the RTO.
*/
public void test_BSBM_Q4_pc100() throws Exception {
-
+
final TestHelper helper = new TestHelper(//
"rto/BSBM-Q4", // testURI,
"rto/BSBM-Q4.rq",// queryFileURL
@@ -193,15 +200,8 @@
/*
* Verify that the runtime optimizer produced the expected join path.
*/
- final int[] expected = new int[] { 3, 4, 5, 1, 2, 6, 7, 8, 9, 10, 11, 12 };
+ final int[] expected = new int[] { 9, 6, 7, 8, 10, 11 };
- /*
- * FIXME This fails because there are actually TWO JoinGraph instances
- * and we are using getOnly() to extract just ONE. It looks like one of
- * those instances might not even run based on conditional routing in
- * the query plan. This is probably because the query is a UNION of two
- * complex join groups and one of them is probably failing the FILTER.
- */
assertSameJoinOrder(expected, helper);
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-10 22:55:24
|
Revision: 7768
http://bigdata.svn.sourceforge.net/bigdata/?rev=7768&view=rev
Author: thompsonbry
Date: 2014-01-10 22:55:17 +0000 (Fri, 10 Jan 2014)
Log Message:
-----------
Bug fix for complex OPTIONAL groups and the RTO. The root cause was a failure to visit the children of the JoinGraph node when assigning BOpStats objects to the top-level query. The fix was to AbstractRunningQuery#populateStatsMap().
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/AbstractRunningQuery.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/AbstractRTOTestCase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/BSBM-Q7.rq
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/TestRTO_BSBM.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/AbstractRunningQuery.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/AbstractRunningQuery.java 2014-01-10 22:38:54 UTC (rev 7767)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/AbstractRunningQuery.java 2014-01-10 22:55:17 UTC (rev 7768)
@@ -684,25 +684,25 @@
statsMap.put(bopId, stats);
// log.warn("bopId=" + bopId + ", stats=" + stats);
- if (!op.getProperty(BOp.Annotations.CONTROLLER,
- BOp.Annotations.DEFAULT_CONTROLLER)) {
- /*
- * Visit children, but not if this is a CONTROLLER operator since
- * its children belong to a subquery.
- */
- final Iterator<BOp> itr = op.argIterator();
+ /*
+ * Visit children.
+ *
+ * Note: The CONTROLLER concept has its subquery expressed through an
+ * annotation, not through its arguments. We always want to visit the
+ * child arguments of a pipeline operator. We just do not want to visit
+ * the operators in its sub-query plan.
+ */
+ final Iterator<BOp> itr = op.argIterator();
- while(itr.hasNext()) {
+ while (itr.hasNext()) {
+
+ final BOp t = itr.next();
+
+ // visit children (recursion)
+ populateStatsMap(t);
+
+ }
- final BOp t = itr.next();
-
- // visit children (recursion)
- populateStatsMap(t);
-
- }
-
- }
-
}
/**
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java 2014-01-10 22:38:54 UTC (rev 7767)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java 2014-01-10 22:55:17 UTC (rev 7768)
@@ -261,19 +261,19 @@
}
- if (joinGroup.isMinus() || joinGroup.isOptional()) {
+// if (false && (joinGroup.isMinus() || joinGroup.isOptional())) {
+//
+// /*
+// * FIXME At least an OPTIONAL join group causes a "No stats"
+// * assertion error during query evaluation. When this is fixed, take
+// * out this code block.
+// *
+// * See TestRTO_BSBM.test_BSBM_Q7b_pc100().
+// */
+// return left;
+//
+// }
- /*
- * FIXME At least an OPTIONAL join group causes a "No stats"
- * assertion error during query evaluation. When this is fixed, take
- * out this code block.
- *
- * See TestRTO_BSBM.test_BSBM_Q7b_pc100().
- */
- return left;
-
- }
-
/*
* Consider the join group. See if it is complex enough to warrant
* running the RTO.
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/AbstractRTOTestCase.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/AbstractRTOTestCase.java 2014-01-10 22:38:54 UTC (rev 7767)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/AbstractRTOTestCase.java 2014-01-10 22:55:17 UTC (rev 7768)
@@ -231,6 +231,9 @@
// The join path selected by the RTO.
final Path path = joinGraph.getPath(l.getRunningQuery());
+ // Verify that a path was attached to the query.
+ assertNotNull(path);
+
if (log.isInfoEnabled())
log.info("path=" + path);
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/BSBM-Q7.rq
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/BSBM-Q7.rq 2014-01-10 22:38:54 UTC (rev 7767)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/BSBM-Q7.rq 2014-01-10 22:55:17 UTC (rev 7768)
@@ -12,13 +12,16 @@
WHERE {
# Control all RTO parameters for repeatable behavior.
- hint:Query hint:optimizer "Runtime".
hint:Query hint:RTO-sampleType "DENSE".
hint:Query hint:RTO-limit "100".
hint:Query hint:RTO-nedges "1".
<http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/dataFromProducer1/Product7> rdfs:label ?productLabel .
OPTIONAL {
+
+ # Note: The RTO is only enabled in one join group to make the unit test easier to write.
+ hint:Group hint:optimizer "Runtime".
+
?offer bsbm:product <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/dataFromProducer1/Product7> .
?offer bsbm:price ?price .
?offer bsbm:vendor ?vendor .
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/TestRTO_BSBM.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/TestRTO_BSBM.java 2014-01-10 22:38:54 UTC (rev 7767)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/TestRTO_BSBM.java 2014-01-10 22:55:17 UTC (rev 7768)
@@ -29,6 +29,8 @@
import java.util.Properties;
+import com.bigdata.bop.engine.IRunningQuery;
+import com.bigdata.bop.joinGraph.rto.Path;
import com.bigdata.rdf.axioms.NoAxioms;
import com.bigdata.rdf.sail.BigdataSail;
import com.bigdata.rdf.sparql.ast.eval.OutOfOrderEvaluationException;
@@ -235,97 +237,10 @@
/**
* BSBM Q7 on the pc100 data set.
*
- * FIXME This fails in the RTO:
- *
- * <pre>
- * org.openrdf.query.QueryEvaluationException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.AssertionError: No stats: op=com.bigdata.bop.join.JVMSolutionSetHashJoinOp[7]()[ com.bigdata.bop.BOp.bopId=7, com.bigdata.bop.BOp.evaluationContext=CONTROLLER, com.bigdata.bop.PipelineOp.sharedState=true, namedSetRef=NamedSolutionSetRef{localName=--nsr-1,queryId=6690c373-8ff2-44b7-826c-f80d8e24eec2,joinVars=[]}, com.bigdata.bop.join.JoinAnnotations.constraints=null, class com.bigdata.bop.join.SolutionSetHashJoinOp.release=false]
- * at com.bigdata.rdf.sail.Bigdata2Sesame2BindingSetIterator.hasNext(Bigdata2Sesame2BindingSetIterator.java:188)
- * at org.openrdf.query.impl.TupleQueryResultImpl.hasNext(TupleQueryResultImpl.java:90)
- * at info.aduna.iteration.Iterations.addAll(Iterations.java:71)
- * at org.openrdf.query.impl.MutableTupleQueryResult.<init>(MutableTupleQueryResult.java:86)
- * at org.openrdf.query.impl.MutableTupleQueryResult.<init>(MutableTupleQueryResult.java:92)
- * at com.bigdata.bop.engine.AbstractQueryEngineTestCase.compareTupleQueryResults(AbstractQueryEngineTestCase.java:738)
- * at com.bigdata.rdf.sparql.ast.eval.AbstractDataAndSPARQLTestCase$AbsHelper.compareTupleQueryResults(AbstractDataAndSPARQLTestCase.java:119)
- * at com.bigdata.rdf.sparql.ast.eval.AbstractDataDrivenSPARQLTestCase$TestHelper.compareTupleQueryResults(AbstractDataDrivenSPARQLTestCase.java:498)
- * at com.bigdata.rdf.sparql.ast.eval.AbstractDataDrivenSPARQLTestCase$TestHelper.runTest(AbstractDataDrivenSPARQLTestCase.java:320)
- * at com.bigdata.rdf.sparql.ast.eval.rto.AbstractRTOTestCase.assertSameJoinOrder(AbstractRTOTestCase.java:181)
- * at com.bigdata.rdf.sparql.ast.eval.rto.TestRTO_BSBM.test_BSBM_Q6_pc100(TestRTO_BSBM.java:198)
- * at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
- * at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
- * at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
- * at java.lang.reflect.Method.invoke(Method.java:601)
- * at junit.framework.TestCase.runTest(TestCase.java:154)
- * at junit.framework.TestCase.runBare(TestCase.java:127)
- * at junit.framework.TestResult$1.protect(TestResult.java:106)
- * at junit.framework.TestResult.runProtected(TestResult.java:124)
- * at junit.framework.TestResult.run(TestResult.java:109)
- * at junit.framework.TestCase.run(TestCase.java:118)
- * at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130)
- * at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
- * at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
- * at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
- * at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
- * at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
- * Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.AssertionError: No stats: op=com.bigdata.bop.join.JVMSolutionSetHashJoinOp[7]()[ com.bigdata.bop.BOp.bopId=7, com.bigdata.bop.BOp.evaluationContext=CONTROLLER, com.bigdata.bop.PipelineOp.sharedState=true, namedSetRef=NamedSolutionSetRef{localName=--nsr-1,queryId=6690c373-8ff2-44b7-826c-f80d8e24eec2,joinVars=[]}, com.bigdata.bop.join.JoinAnnotations.constraints=null, class com.bigdata.bop.join.SolutionSetHashJoinOp.release=false]
- * at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:1523)
- * at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator._hasNext(BlockingBuffer.java:1710)
- * at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.hasNext(BlockingBuffer.java:1563)
- * at com.bigdata.striterator.AbstractChunkedResolverator._hasNext(AbstractChunkedResolverator.java:357)
- * at com.bigdata.striterator.AbstractChunkedResolverator.hasNext(AbstractChunkedResolverator.java:333)
- * at com.bigdata.rdf.sail.Bigdata2Sesame2BindingSetIterator.hasNext(Bigdata2Sesame2BindingSetIterator.java:134)
- * ... 26 more
- * Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.AssertionError: No stats: op=com.bigdata.bop.join.JVMSolutionSetHashJoinOp[7]()[ com.bigdata.bop.BOp.bopId=7, com.bigdata.bop.BOp.evaluationContext=CONTROLLER, com.bigdata.bop.PipelineOp.sharedState=true, namedSetRef=NamedSolutionSetRef{localName=--nsr-1,queryId=6690c373-8ff2-44b7-826c-f80d8e24eec2,joinVars=[]}, com.bigdata.bop.join.JoinAnnotations.constraints=null, class com.bigdata.bop.join.SolutionSetHashJoinOp.release=false]
- * at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
- * at java.util.concurrent.FutureTask.get(FutureTask.java:111)
- * at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:1454)
- * ... 31 more
- * Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.AssertionError: No stats: op=com.bigdata.bop.join.JVMSolutionSetHashJoinOp[7]()[ com.bigdata.bop.BOp.bopId=7, com.bigdata.bop.BOp.evaluationContext=CONTROLLER, com.bigdata.bop.PipelineOp.sharedState=true, namedSetRef=NamedSolutionSetRef{localName=--nsr-1,queryId=6690c373-8ff2-44b7-826c-f80d8e24eec2,joinVars=[]}, com.bigdata.bop.join.JoinAnnotations.constraints=null, class com.bigdata.bop.join.SolutionSetHashJoinOp.release=false]
- * at com.bigdata.rdf.sail.RunningQueryCloseableIterator.checkFuture(RunningQueryCloseableIterator.java:59)
- * at com.bigdata.rdf.sail.RunningQueryCloseableIterator.close(RunningQueryCloseableIterator.java:73)
- * at com.bigdata.rdf.sail.RunningQueryCloseableIterator.hasNext(RunningQueryCloseableIterator.java:82)
- * at com.bigdata.striterator.ChunkedWrappedIterator.hasNext(ChunkedWrappedIterator.java:197)
- * at com.bigdata.striterator.AbstractChunkedResolverator$ChunkConsumerTask.call(AbstractChunkedResolverator.java:222)
- * at com.bigdata.striterator.AbstractChunkedResolverator$ChunkConsumerTask.call(AbstractChunkedResolverator.java:1)
- * at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
- * at java.util.concurrent.FutureTask.run(FutureTask.java:166)
- * at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
- * at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
- * at java.lang.Thread.run(Thread.java:722)
- * Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.AssertionError: No stats: op=com.bigdata.bop.join.JVMSolutionSetHashJoinOp[7]()[ com.bigdata.bop.BOp.bopId=7, com.bigdata.bop.BOp.evaluationContext=CONTROLLER, com.bigdata.bop.PipelineOp.sharedState=true, namedSetRef=NamedSolutionSetRef{localName=--nsr-1,queryId=6690c373-8ff2-44b7-826c-f80d8e24eec2,joinVars=[]}, com.bigdata.bop.join.JoinAnnotations.constraints=null, class com.bigdata.bop.join.SolutionSetHashJoinOp.release=false]
- * at com.bigdata.util.concurrent.Haltable.get(Haltable.java:273)
- * at com.bigdata.bop.engine.AbstractRunningQuery.get(AbstractRunningQuery.java:1474)
- * at com.bigdata.bop.engine.AbstractRunningQuery.get(AbstractRunningQuery.java:1)
- * at com.bigdata.rdf.sail.RunningQueryCloseableIterator.checkFuture(RunningQueryCloseableIterator.java:46)
- * ... 10 more
- * Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.AssertionError: No stats: op=com.bigdata.bop.join.JVMSolutionSetHashJoinOp[7]()[ com.bigdata.bop.BOp.bopId=7, com.bigdata.bop.BOp.evaluationContext=CONTROLLER, com.bigdata.bop.PipelineOp.sharedState=true, namedSetRef=NamedSolutionSetRef{localName=--nsr-1,queryId=6690c373-8ff2-44b7-826c-f80d8e24eec2,joinVars=[]}, com.bigdata.bop.join.JoinAnnotations.constraints=null, class com.bigdata.bop.join.SolutionSetHashJoinOp.release=false]
- * at com.bigdata.bop.engine.ChunkedRunningQuery.scheduleNext(ChunkedRunningQuery.java:678)
- * at com.bigdata.bop.engine.ChunkedRunningQuery.acceptChunk(ChunkedRunningQuery.java:290)
- * at com.bigdata.bop.engine.QueryEngine.acceptChunk(QueryEngine.java:1031)
- * at com.bigdata.bop.engine.QueryEngine.startEval(QueryEngine.java:1697)
- * at com.bigdata.bop.engine.QueryEngine.eval(QueryEngine.java:1564)
- * at com.bigdata.bop.engine.QueryEngine.eval(QueryEngine.java:1470)
- * at com.bigdata.bop.engine.QueryEngine.eval(QueryEngine.java:1447)
- * at com.bigdata.bop.controller.JVMNamedSubqueryOp$ControllerTask$SubqueryTask.call(JVMNamedSubqueryOp.java:361)
- * at com.bigdata.bop.controller.JVMNamedSubqueryOp$ControllerTask.call(JVMNamedSubqueryOp.java:278)
- * at com.bigdata.bop.controller.JVMNamedSubqueryOp$ControllerTask.call(JVMNamedSubqueryOp.java:1)
- * at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
- * at java.util.concurrent.FutureTask.run(FutureTask.java:166)
- * at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1301)
- * at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTaskWrapper.run(ChunkedRunningQuery.java:856)
- * at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
- * at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
- * at java.util.concurrent.FutureTask.run(FutureTask.java:166)
- * at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63)
- * at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkFutureTask.run(ChunkedRunningQuery.java:751)
- * ... 3 more
- * Caused by: java.lang.RuntimeException: java.lang.AssertionError: No stats: op=com.bigdata.bop.join.JVMSolutionSetHashJoinOp[7]()[ com.bigdata.bop.BOp.bopId=7, com.bigdata.bop.BOp.evaluationContext=CONTROLLER, com.bigdata.bop.PipelineOp.sharedState=true, namedSetRef=NamedSolutionSetRef{localName=--nsr-1,queryId=6690c373-8ff2-44b7-826c-f80d8e24eec2,joinVars=[]}, com.bigdata.bop.join.JoinAnnotations.constraints=null, class com.bigdata.bop.join.SolutionSetHashJoinOp.release=false]
- * at com.bigdata.bop.engine.ChunkedRunningQuery.scheduleNext(ChunkedRunningQuery.java:648)
- * ... 21 more
- * Caused by: java.lang.AssertionError: No stats: op=com.bigdata.bop.join.JVMSolutionSetHashJoinOp[7]()[ com.bigdata.bop.BOp.bopId=7, com.bigdata.bop.BOp.evaluationContext=CONTROLLER, com.bigdata.bop.PipelineOp.sharedState=true, namedSetRef=NamedSolutionSetRef{localName=--nsr-1,queryId=6690c373-8ff2-44b7-826c-f80d8e24eec2,joinVars=[]}, com.bigdata.bop.join.JoinAnnotations.constraints=null, class com.bigdata.bop.join.SolutionSetHashJoinOp.release=false]
- * at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.<init>(ChunkedRunningQuery.java:1172)
- * at com.bigdata.bop.engine.ChunkedRunningQuery.scheduleNext(ChunkedRunningQuery.java:640)
- * ... 21 more
- * </pre>
+ * FIXME This fails because the RTO is running in a named subquery. The test
+ * harness is looking in the wrong place (it is looking on the wrong
+ * {@link IRunningQuery}) and therefore it fails to find the {@link Path}
+ * computed by the RTO.
*/
public void test_BSBM_Q7_pc100() throws Exception {
@@ -340,7 +255,7 @@
* Verify that the runtime optimizer produced the expected join path.
*/
- // FIXME The join order is unknown. This query does not run through the RTO yet.
+ // FIXME The join order is unknown.
final int[] expected = new int[] { 1, 3, 2, 5, 4, 7, 6 };
assertSameJoinOrder(expected, helper);
@@ -385,18 +300,17 @@
public void test_BSBM_Q7b_pc100() throws Exception {
final TestHelper helper = new TestHelper(//
- "rto/BSBM-Q7", // testURI,
- "rto/BSBM-Q7.rq",// queryFileURL
+ "rto/BSBM-Q7b", // testURI,
+ "rto/BSBM-Q7b.rq",// queryFileURL
"bigdata-rdf/src/resources/data/bsbm/dataset_pc100.nt",// dataFileURL
- "rto/BSBM-Q7.srx"// resultFileURL
+ "rto/BSBM-Q7b.srx"// resultFileURL
);
/*
* Verify that the runtime optimizer produced the expected join path.
*/
- // FIXME The join order is unknown. This query does not run through the RTO yet.
- final int[] expected = new int[] { 1, 3, 2, 5, 4, 7, 6 };
+ final int[] expected = new int[] { 5, 6, 7, 8 };
assertSameJoinOrder(expected, helper);
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-13 19:58:36
|
Revision: 7786
http://bigdata.svn.sourceforge.net/bigdata/?rev=7786&view=rev
Author: thompsonbry
Date: 2014-01-13 19:58:29 +0000 (Mon, 13 Jan 2014)
Log Message:
-----------
Modified JGraph by removing an unused public constructor.
Added an escape hatch if one or more join paths continues to have a cardinality underflow. This issue (cardinality underflow along some paths) doubtless needs to be examined further. This is to address an endless loop observed for one of the govtrac queries.
Added a test where there are no solutions. The RTO should be jumping out as soon as it recognizes that the join graph can not produce a solution. Instead, it is solving the join graph, which is pointless (this is noted, but not fixed).
See #64.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JGraph.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/BSBM-Q1.rq
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/TestRTO_BSBM.java
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/BSBM-Q1-noSolutions.srx
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JGraph.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JGraph.java 2014-01-13 16:32:25 UTC (rev 7785)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JGraph.java 2014-01-13 19:58:29 UTC (rev 7786)
@@ -350,68 +350,52 @@
}
- /**
- * Find a good join path in the data given the join graph. The join path is
- * not guaranteed to be the best join path (the search performed by the
- * runtime optimizer is not exhaustive) but it should always be a "good"
- * join path and may often be the "best" join path.
- *
- * @param queryEngine
- * The query engine.
- * @param limit
- * The limit for sampling a vertex and the initial limit for
- * cutoff join evaluation.
- * @param nedges
- * The edges in the join graph are sorted in order of increasing
- * cardinality and up to <i>nedges</i> of the edges having the
- * lowest cardinality are used to form the initial set of join
- * paths. For each edge selected to form a join path, the
- * starting vertex will be the vertex of that edge having the
- * lower cardinality.
- * @param sampleType
- * Type safe enumeration indicating the algorithm which will be
- * used to sample the initial vertices.
- *
- * @return The join path identified by the runtime query optimizer as the
- * best path given the join graph and the data.
- *
- * @throws NoSolutionsException
- * If there are no solutions for the join graph in the data (the
- * query does not have any results).
- * @throws IllegalArgumentException
- * if <i>queryEngine</i> is <code>null</code>.
- * @throws IllegalArgumentException
- * if <i>limit</i> is non-positive.
- * @throws IllegalArgumentException
- * if <i>nedges</i> is non-positive.
- * @throws Exception
- */
- public Path runtimeOptimizer(final QueryEngine queryEngine,
- final int limit, final int nedges) throws NoSolutionsException,
- Exception {
+// /**
+// * Find a good join path in the data given the join graph. The join path is
+// * not guaranteed to be the best join path (the search performed by the
+// * runtime optimizer is not exhaustive) but it should always be a "good"
+// * join path and may often be the "best" join path.
+// *
+// * @param queryEngine
+// * The query engine.
+// * @param limit
+// * The limit for sampling a vertex and the initial limit for
+// * cutoff join evaluation.
+// * @param nedges
+// * The edges in the join graph are sorted in order of increasing
+// * cardinality and up to <i>nedges</i> of the edges having the
+// * lowest cardinality are used to form the initial set of join
+// * paths. For each edge selected to form a join path, the
+// * starting vertex will be the vertex of that edge having the
+// * lower cardinality.
+// * @param sampleType
+// * Type safe enumeration indicating the algorithm which will be
+// * used to sample the initial vertices.
+// *
+// * @return The join path identified by the runtime query optimizer as the
+// * best path given the join graph and the data.
+// *
+// * @throws NoSolutionsException
+// * If there are no solutions for the join graph in the data (the
+// * query does not have any results).
+// * @throws IllegalArgumentException
+// * if <i>queryEngine</i> is <code>null</code>.
+// * @throws IllegalArgumentException
+// * if <i>limit</i> is non-positive.
+// * @throws IllegalArgumentException
+// * if <i>nedges</i> is non-positive.
+// * @throws Exception
+// */
+// public Path runtimeOptimizer(final QueryEngine queryEngine,
+// final int limit, final int nedges) throws NoSolutionsException,
+// Exception {
+//
+// final Map<PathIds, EdgeSample> edgeSamples = new LinkedHashMap<PathIds, EdgeSample>();
+//
+// return runtimeOptimizer(queryEngine, limit, nedges, edgeSamples);
+//
+// }
- /*
- * This map is used to associate join path segments (expressed as an
- * ordered array of bopIds) with edge sample to avoid redundant effort.
- *
- * FIXME RTO: HEAP MANAGMENT : This map holds references to the cutoff
- * join samples. To ensure that the map has the minimum heap footprint,
- * it must be scanned each time we prune the set of active paths and any
- * entry which is not a prefix of an active path should be removed.
- *
- * TODO RTO: MEMORY MANAGER : When an entry is cleared from this map,
- * the corresponding allocation in the memory manager (if any) must be
- * released. The life cycle of the map needs to be bracketed by a
- * try/finally in order to ensure that all allocations associated with
- * the map are released no later than when we leave the lexicon scope of
- * that clause.
- */
- final Map<PathIds, EdgeSample> edgeSamples = new LinkedHashMap<PathIds, EdgeSample>();
-
- return runtimeOptimizer(queryEngine, limit, nedges, edgeSamples);
-
- }
-
/**
* Find a good join path in the data given the join graph. The join path is
* not guaranteed to be the best join path (the search performed by the
@@ -420,19 +404,6 @@
*
* @param queryEngine
* The query engine.
- * @param limit
- * The limit for sampling a vertex and the initial limit for
- * cutoff join evaluation.
- * @param nedges
- * The edges in the join graph are sorted in order of increasing
- * cardinality and up to <i>nedges</i> of the edges having the
- * lowest cardinality are used to form the initial set of join
- * paths. For each edge selected to form a join path, the
- * starting vertex will be the vertex of that edge having the
- * lower cardinality.
- * @param sampleType
- * Type safe enumeration indicating the algorithm which will be
- * used to sample the initial vertices.
* @param edgeSamples
* A map that will be populated with the samples associated with
* each non-pruned join path. This map is used to associate join
@@ -463,18 +434,49 @@
* TODO We need to automatically increase the depth of search
* for queries where we have cardinality estimation underflows
* or punt to another method to decide the join order.
+ *
+ * TODO RTO: HEAP MANAGMENT : The edgeSamples map holds
+ * references to the cutoff join samples. To ensure that the map
+ * has the minimum heap footprint, it must be scanned each time
+ * we prune the set of active paths and any entry which is not a
+ * prefix of an active path should be removed.
+ *
+ * TODO RTO: MEMORY MANAGER : When an entry is cleared from this
+ * map, the corresponding allocation in the memory manager (if
+ * any) must be released. The life cycle of the map needs to be
+ * bracketed by a try/finally in order to ensure that all
+ * allocations associated with the map are released no later
+ * than when we leave the lexicon scope of that clause.
*/
- public Path runtimeOptimizer(final QueryEngine queryEngine,
- final int limit, final int nedges,
- final Map<PathIds, EdgeSample> edgeSamples)
- throws Exception, NoSolutionsException {
+ public Path runtimeOptimizer(//
+ final QueryEngine queryEngine,//
+ final Map<PathIds, EdgeSample> edgeSamples//
+ ) throws Exception, NoSolutionsException {
if (queryEngine == null)
throw new IllegalArgumentException();
+
+ /*
+ * The limit for sampling a vertex and the initial limit for cutoff join
+ * evaluation.
+ */
+ final int limit = joinGraph.getLimit();
+
if (limit <= 0)
throw new IllegalArgumentException();
+
+ /*
+ * The edges in the join graph are sorted in order of increasing
+ * cardinality and up to <i>nedges</i> of the edges having the lowest
+ * cardinality are used to form the initial set of join paths. For each
+ * edge selected to form a join path, the starting vertex will be the
+ * vertex of that edge having the lower cardinality.
+ */
+ final int nedges = joinGraph.getNEdges();
+
if (nedges <= 0)
throw new IllegalArgumentException();
+
if (edgeSamples == null)
throw new IllegalArgumentException();
@@ -500,32 +502,58 @@
while (paths.length > 0 && round < nvertices - 1) {
- /*
- * Resample the paths.
- *
- * Note: Since the vertex samples are random, it is possible for the
- * #of paths with cardinality estimate underflow to jump up and down
- * due to the sample which is making its way through each path in
- * each round.
- *
- * TODO The RTO needs an escape hatch here. FOr example, if the sum
- * of the expected IOs for some path(s) strongly dominates all other
- * paths sharing the same vertices, then we should prune those paths
- * even if there is a cardinality estimate underflow in those paths.
- * This will allow us to focus our efforts on those paths having
- * less IO cost while we seek cardinality estimates which do not
- * underflow.
- */
- int nunderflow;
+ /*
+ * Resample the paths.
+ *
+ * Note: Since the vertex samples are random, it is possible for the
+ * #of paths with cardinality estimate underflow to jump up and down
+ * due to the sample which is making its way through each path in
+ * each round.
+ *
+ * Note: The RTO needs an escape hatch here. Otherwise, it is
+ * possible for it to spin in a loop while resampling.
+ *
+ * TODO For example, if the sum of the expected IOs for some path(s)
+ * strongly dominates all other paths sharing the same vertices,
+ * then we should prune those paths even if there is a cardinality
+ * estimate underflow in those paths. This will allow us to focus
+ * our efforts on those paths having less IO cost while we seek
+ * cardinality estimates which do not underflow.
+ *
+ * TODO We should be examining the actual sampling limit that is
+ * currently in place on each vertex and for each path. This is
+ * available by inspection of the VertexSamples and EdgeSamples, but
+ * it is not passed back out of the resamplePaths() method as a
+ * side-effect. We should limit how much we are willing to raise the
+ * limit, e.g., by specifying a MAX_LIMIT annotation on the
+ * JoinGraph operator.
+ */
+ int nunderflow = 0;
- while ((nunderflow = resamplePaths(queryEngine, limit, round,
- paths, edgeSamples)) > 0) {
-
- log.warn("resampling in round=" + round + " : " + nunderflow
- + " paths have cardinality estimate underflow.");
-
- }
+ for (int i = 0; i < 3; i++) {
+ nunderflow = resamplePaths(queryEngine, limit, round, paths,
+ edgeSamples);
+
+ if (nunderflow == 0) {
+
+ // No paths have cardinality estimate underflow.
+ break;
+
+ }
+
+ log.warn("Cardinality estimate underflow - resampling: round="
+ + round + ", npaths=" + paths.length + ", nunderflow="
+ + nunderflow + ", limit=" + limit);
+
+ }
+
+ if (nunderflow > 0) {
+
+ log.warn("Continuing: some paths have cardinality underflow!");
+
+ }
+
/*
* Extend the paths by one vertex.
*/
@@ -542,8 +570,10 @@
// Should be one winner.
if (paths.length != 1) {
+
throw new AssertionError("Expected one path but have "
+ paths.length + " paths.");
+
}
if (log.isInfoEnabled()) {
@@ -770,8 +800,8 @@
*
* @throws Exception
*/
- protected int resamplePaths(final QueryEngine queryEngine, int limitIn,
- final int round, final Path[] a,
+ protected int resamplePaths(final QueryEngine queryEngine,
+ final int limitIn, final int round, final Path[] a,
final Map<PathIds, EdgeSample> edgeSamples) throws Exception {
if (queryEngine == null)
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java 2014-01-13 16:32:25 UTC (rev 7785)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java 2014-01-13 19:58:29 UTC (rev 7786)
@@ -468,10 +468,8 @@
final Map<PathIds, EdgeSample> edgeSamples = new LinkedHashMap<PathIds, EdgeSample>();
// Find the best join path.
- final Path path = g
- .runtimeOptimizer(context.getRunningQuery()
- .getQueryEngine(), getLimit(), getNEdges(),
- edgeSamples);
+ final Path path = g.runtimeOptimizer(context.getRunningQuery()
+ .getQueryEngine(), edgeSamples);
/*
* Release samples.
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/BSBM-Q1-noSolutions.srx
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/BSBM-Q1-noSolutions.srx (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/BSBM-Q1-noSolutions.srx 2014-01-13 19:58:29 UTC (rev 7786)
@@ -0,0 +1,10 @@
+<?xml version="1.0"?>
+<sparql xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema#" xmlns="http://www.w3.org/2005/sparql-results#">
+ <head>
+ <variable name="product" />
+ <variable name="label" />
+ </head>
+ <results>
+ </results>
+</sparql>
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/BSBM-Q1.rq
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/BSBM-Q1.rq 2014-01-13 16:32:25 UTC (rev 7785)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/BSBM-Q1.rq 2014-01-13 19:58:29 UTC (rev 7786)
@@ -5,7 +5,6 @@
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
-#SELECT (COUNT(DISTINCT *) as ?count)
SELECT DISTINCT ?product ?label
WHERE {
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/TestRTO_BSBM.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/TestRTO_BSBM.java 2014-01-13 16:32:25 UTC (rev 7785)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/rto/TestRTO_BSBM.java 2014-01-13 19:58:29 UTC (rev 7786)
@@ -31,6 +31,7 @@
import junit.framework.AssertionFailedError;
+import com.bigdata.bop.joinGraph.NoSolutionsException;
import com.bigdata.rdf.axioms.NoAxioms;
import com.bigdata.rdf.sail.BigdataSail;
@@ -116,6 +117,30 @@
}
/**
+ * Test of BSBM Q1 against an empty data set. There are no solutions in the
+ * data.
+ */
+ public void test_BSBM_Q1_noSolutions() throws Exception {
+
+ final TestHelper helper = new TestHelper(//
+ "rto/BSBM-Q1", // testURI,
+ "rto/BSBM-Q1.rq",// queryFileURL
+ new String[]{},// data files.
+ "rto/BSBM-Q1-noSolutions.srx"// resultFileURL
+ );
+
+ /*
+ * TODO In fact, the RTO should not be running for a group of required
+ * joins in which some vertex has a zero cardinality or when any join
+ * can provably produce ZERO results when fed solutions from a fully
+ * materialized vertex.
+ */
+
+ assertSameJoinOrder(new int[] { 2, 1, 3, 4, 5 }, helper);
+
+ }
+
+ /**
* BSBM Q1 against pc100.
*/
public void test_BSBM_Q1_pc100() throws Exception {
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-13 21:55:41
|
Revision: 7787
http://bigdata.svn.sourceforge.net/bigdata/?rev=7787&view=rev
Author: thompsonbry
Date: 2014-01-13 21:55:30 +0000 (Mon, 13 Jan 2014)
Log Message:
-----------
The root cause of the OutOfOrderEvaluationException was the ChunkedRunningQuery.HandleChunkBuffer class. That class was allowing reordering of chunks in order to output full chunks immediately and gather smaller chunks together until they can be combined as a single full chunk. The class has been rewritten and now has two distinct behaviors. If reordering is allowed, then the old behavior is preserved (except that it can output chunks of up to 150% of the target chunk size). If reordering is disallowed, then it will still combine chunks as much as possible, but not if that would violate an order preserving guarantee.
The HandleChunkBuffer class breaks the semantics for ORDER BY and SLICE, both of which should preserve order. It also breaks the semantics for ORDER_BY + DISTINCT. This problem would only appear in cases where the size of the output chunks was such that a (sufficiently) full chunk would be output ahead of smaller chunks in an internal buffer.
I have added a new PipelineOp annotation named REORDER_SOLUTIONS. This defaults to true, which is the historical throughput-oriented behavior.
The MemorySortOp and SliceOp constructors now check for and require REORDER_SOLUTIONS := false.
AST2BOpUtility has been modified to turn off REORDER_SOLUTIONS for the ORDER_BY_DISTINCT case (see #563 (ORDER BY + DISTINCT)).
The AST2BOpRTO integration now runs clean when
failOutOfOrderEvaluation := true.
PipelineOp - addded REORDER_SOLUTIONS annotation. Defaults to true (the historical throughput oriented behavior).
MemorySortOp - requires REORDER_SOLUTIONS:=false.
SliceOp - requires REORDER_SOLUTIONS:=false.
Reordered solutions are no longer observed during cutoff join evaluation. The failure to disable the REORDER_SOLUTIONS annotation is now detected if checking of cutoff query plans is enabled.
AST2BOpUtility - now adds the REORDER_SOLUTIONS:=false annotation as necessary for SLICE, ORDER_BY and DISTINCT (when paired with ORDER_BY).
See #64 (RTO)
See #798 (Solution order not always preserved)
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/PipelineOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/MemorySortOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine_Slice.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine_SortOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/solutions/TestMemorySortOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/solutions/TestSliceOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpFilters.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpJoins.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/PipelineOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/PipelineOp.java 2014-01-13 19:58:29 UTC (rev 7786)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/PipelineOp.java 2014-01-13 21:55:30 UTC (rev 7787)
@@ -93,6 +93,20 @@
boolean DEFAULT_SHARED_STATE = false;
+ /**
+ * When <code>true</code>, the {@link QueryEngine} MAY reorder the
+ * solutions as they flow through the query plan (this is done as a
+ * throughput optimization). When <code>false</code>, the
+ * {@link QueryEngine} MUST NOT reorder solutions.
+ *
+ * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/798" >
+ * Solution order not always preserved. </a>
+ */
+ String REORDER_SOLUTIONS = PipelineOp.class.getName()
+ + ".reorderSolutions";
+
+ boolean DEFAULT_REORDER_SOLUTIONS = true;
+
/**
* This option may be used to place an optional limit on the #of
* concurrent tasks which may run for the same (bopId,shardId) for a
@@ -295,6 +309,16 @@
}
/**
+ * @see Annotations#REORDER_SOLUTIONS
+ */
+ final public boolean isReorderSolutions() {
+
+ return getProperty(Annotations.REORDER_SOLUTIONS,
+ Annotations.DEFAULT_REORDER_SOLUTIONS);
+
+ }
+
+ /**
* The maximum amount of memory which may be used to buffered inputs for
* this operator on the native heap. When ZERO (0), the inputs will be
* buffered on the JVM heap. When {@link Long#MAX_VALUE}, an essentially
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java 2014-01-13 19:58:29 UTC (rev 7786)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java 2014-01-13 21:55:30 UTC (rev 7787)
@@ -1292,42 +1292,15 @@
* target that sink.
*/
private IBlockingBuffer<IBindingSet[]> newBuffer(final PipelineOp op,
- final int sinkId,
-// final SinkTransitionMetadata sinkTransitionMetadata,
- final AtomicInteger sinkMessagesOut, final BOpStats stats) {
+ final int sinkId,//
+ final AtomicInteger sinkMessagesOut, //
+ final BOpStats stats//
+ ) {
-// final MultiplexBlockingBuffer<IBindingSet[]> factory = inputBufferMap == null ? null
-// : inputBufferMap.get(sinkId);
-//
-// if (factory != null) {
-//
-// return factory.newInstance();
-//
-// }
+ return new HandleChunkBuffer(ChunkedRunningQuery.this, bopId,
+ partitionId, sinkId, op.getChunkCapacity(),
+ op.isReorderSolutions(), sinkMessagesOut, stats);
-// return new HandleChunkBuffer(sinkId, sinkMessagesOut, op
-// .newBuffer(stats));
-
- /*
- * FIXME The buffer allocated here is useless unless we play games
- * in HandleChunkBuffer to combine chunks or run a thread which
- * drains chunks from all operator tasks (but the task can not
- * complete until it is fully drained).
- */
-// final IBlockingBuffer<IBindingSet[]> b = new BlockingBuffer<IBindingSet[]>(
-// op.getChunkOfChunksCapacity(), op.getChunkCapacity(), op
-// .getChunkTimeout(),
-// BufferAnnotations.chunkTimeoutUnit);
-
- return
-// new SinkTransitionBuffer(
- new HandleChunkBuffer(
- ChunkedRunningQuery.this, bopId, partitionId, sinkId, op
- .getChunkCapacity(), sinkMessagesOut, stats)
-// ,
-// sinkTransitionMetadata)
- ;
-
}
/**
@@ -1409,8 +1382,6 @@
public NoCloseBuffer(final UUID queryId, final BOp bop, final int bopId,
final int partitionId, final IBlockingBuffer<E> delegate) {
-// public NoCloseBuffer(final IBlockingBuffer<E> delegate) {
-
super(delegate);
this.queryId = queryId;
@@ -1429,39 +1400,23 @@
}
}
- // public void add(E e) {
-// log.error(Arrays.toString((Object[])e));
-// super.add(e);
-// }
-//
-// public void reset() {
-// log.error("");
-// super.reset();
-// }
-//
-// public long flush() {
-// log.error("");
-// return super.flush();
-// }
-
@Override
public void close() {
- // NOP
-// log.error("");
+ // NOP - This makes sure that the query buffer is not closed.
}
- }
+ } // class NoCloseBuffer
/**
- * Class traps {@link #add(IBindingSet[])} to handle the IBindingSet[]
- * chunks as they are generated by the running operator task, invoking
- * {@link ChunkedRunningQuery#handleOutputChunk(BOp, int, IBlockingBuffer)} for
- * each generated chunk to synchronously emit {@link IChunkMessage}s.
+ * Class traps {@link #add(IBindingSet[])} to handle the {@link IBindingSet}
+ * [] chunks as they are generated by the running operator task, invoking
+ * {@link ChunkedRunningQuery#handleOutputChunk(BOp, int, IBlockingBuffer)}
+ * for each generated chunk to synchronously emit {@link IChunkMessage}s.
* <p>
* This use of this class significantly increases the parallelism and
- * throughput of selective queries. If output chunks are not "handled"
- * until the {@link ChunkTask} is complete then the total latency of
- * selective queries is increased dramatically.
+ * throughput of selective queries. If output chunks are not "handled" until
+ * the {@link ChunkTask} is complete then the total latency of selective
+ * queries is increased dramatically.
*/
static private class HandleChunkBuffer implements
IBlockingBuffer<IBindingSet[]> {
@@ -1474,28 +1429,39 @@
private final int sinkId;
+// /**
+// * The desired chunk size.
+// */
+// private final int chunkCapacity;
+
+ /** The minimum desired chunk size (50% of the {@link #chunkCapacity}). */
+ private final int minChunkSize;
+
+ /** The maximum desired chunk size (150% of the {@link #chunkCapacity}) */
+ private final int maxChunkSize;
/**
- * The target chunk size. When ZERO (0) chunks are output immediately as
- * they are received (the internal buffer is not used).
+ * When <code>true</code>, the buffer MAY reorder solutions. When
+ * <code>false</code>, it MUST NOT.
*/
- private final int chunkCapacity;
+ private final boolean reorderSolutions;
-// private final SinkTransitionMetadata sinkTransitionMetadata;
-
private final AtomicInteger sinkMessagesOut;
private final BOpStats stats;
private volatile boolean open = true;
-// /**
-// * An internal buffer which is used if chunkCapacity != ZERO.
-// */
-// private IBindingSet[] chunk = null;
+ /**
+ * A list of small chunks that will be combined into a single chunk. The
+ * solutions in this list are always evicted by {@link #flush()}.
+ */
private List<IBindingSet[]> smallChunks = null;
/**
- * The #of elements in the internal {@link #chunk} buffer.
+ * The #of elements in the {@link #smallChunks} buffer. Each element is
+ * an {@link IBindingSet}, so this is the number of solutions that have
+ * not yet been flushed through because we have not yet made up a single
+ * decent sized chunk.
*/
private int chunkSize = 0;
@@ -1505,22 +1471,28 @@
* @param bopId
* @param sinkId
* @param chunkCapacity
+ * The target capacity for each chunk.
+ * @param reorderSolutions
+ * When <code>true</code>, the buffer MAY reorder solutions.
+ * When <code>false</code>, it MUST NOT.
* @param sinkMessagesOut
* @param stats
*/
public HandleChunkBuffer(final ChunkedRunningQuery q, final int bopId,
final int partitionId,
final int sinkId, final int chunkCapacity,
-// final SinkTransitionMetadata sinkTransitionMetadata,
+ final boolean reorderSolutions,
final AtomicInteger sinkMessagesOut, final BOpStats stats) {
this.q = q;
this.bopId = bopId;
this.partitionId = partitionId;
this.sinkId = sinkId;
- this.chunkCapacity = chunkCapacity;
-// this.sinkTransitionMetadata = sinkTransitionMetadata;
+// this.chunkCapacity = chunkCapacity;
+ this.reorderSolutions = reorderSolutions;
this.sinkMessagesOut = sinkMessagesOut;
this.stats = stats;
+ this.minChunkSize = (chunkCapacity >> 1); // 50%
+ this.maxChunkSize = chunkCapacity + (chunkCapacity >> 1); // 150%
}
/**
@@ -1531,7 +1503,11 @@
* <p>
* Note: This must be synchronized in case the caller is multi-threaded
* since it has a possible side effect on the internal buffer.
+ *
+ * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/798" >
+ * Solution order not always preserved. </a>
*/
+ @Override
public void add(final IBindingSet[] e) {
if(!open)
@@ -1542,75 +1518,138 @@
partitionId, (IBindingSet[]) e);
}
-// for (IBindingSet bset : e) {
-// sinkTransitionMetadata.handleBindingSet(bset);
-// }
+ if (false) {
+
+ /*
+ * Note: Do this INSTEAD if you want to complete disable both
+ * reordering and chunk combination. This should ONLY be used
+ * for debugging. Chunk combination is an important throughput
+ * enhancer.
+ */
+
+ // outputChunk(e);
+
+ } else {
+
+ if (reorderSolutions) {
+
+ // Solutions MAY be reordered.
+ addReorderAllowed(e);
+
+ } else {
+
+ // Solutions MUST NOT be reordered.
+ addReorderNotAllowed(e);
+
+ }
+
+ }
-// if (chunkCapacity != 0 && e.length < (chunkCapacity >> 1)) {
-// /*
-// * The caller's array is significantly smaller than the target
-// * chunk size. Append the caller's array to the internal buffer
-// * and return immediately. The internal buffer will be copied
-// * through either in a subsequent add() or in flush().
-// */
-// synchronized (this) {
-//
-// if (chunk == null)
-// chunk = new IBindingSet[chunkCapacity];
-//
-// if (chunkSize + e.length > chunkCapacity) {
-//
-// // flush the buffer first.
-// outputBufferedChunk();
-//
-// }
-//
-// // copy the chunk into the buffer.
-// System.arraycopy(e/* src */, 0/* srcPos */,
-// chunk/* dest */, chunkSize/* destPos */,
-// e.length/* length */);
-//
-// chunkSize += e.length;
-//
-// return;
-//
-// }
-//
-// }
+ } // add()
- if (chunkCapacity != 0 && e.length < (chunkCapacity >> 1)) {
+ /**
+ * We are allowed to reorder the solutions.
+ * <p>
+ * This will reorder solutions by outputting the current chunk
+ * immediately if it is GTE 50% of the target chunkCapacity. This is
+ * also a non-blocking code path (no lock is taken in this method).
+ * <p>
+ * Otherwise, the chunk is added {@link #smallChunks} list. If the #of
+ * solutions on the {@link #smallChunks} reaches a threshold, then the
+ * {@link #smallChunks} list is converted into a single chunk an
+ * evicted.
+ */
+ private void addReorderAllowed(final IBindingSet[] e) {
+
+ if (e.length < minChunkSize) {
+
/*
* The caller's array is significantly smaller than the target
* chunk size. Append the caller's array to the internal list
* and return immediately. The buffered chunks will be copied
* through either in a subsequent add() or in flush().
*/
+
synchronized (this) {
- if (chunkSize + e.length > chunkCapacity) {
+ if (chunkSize + e.length > maxChunkSize) {
// flush the buffer first.
outputBufferedChunk();
}
-
+
if (smallChunks == null)
smallChunks = new LinkedList<IBindingSet[]>();
+ // Add to the buffer.
smallChunks.add(e);
-
chunkSize += e.length;
return;
}
+
}
// output the caller's chunk immediately.
outputChunk(e);
}
+
+ /**
+ * We are not allowed to reorder the solutions.
+ * <p>
+ * This always outputs solutions in the same order that they are added.
+ * In order to avoid pushing through small chunks, it allows the output
+ * chunk to be over the target capacity (by 50%).
+ *
+ * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/798" >
+ * Solution order not always preserved. </a>
+ */
+ private void addReorderNotAllowed(final IBindingSet[] e) {
+ synchronized (this) {
+
+ if (chunkSize + e.length > maxChunkSize) {
+
+ /*
+ * The combined chunk would be too large for the buffer.
+ */
+
+ // Flush the buffer.
+ outputBufferedChunk();
+
+ if (e.length > minChunkSize) {
+
+ /*
+ * The internal buffer is empty. The chunk is big
+ * enough. Sent it through immediately.
+ */
+
+ outputChunk(e);
+
+ return;
+
+ }
+
+ }
+
+ /*
+ * Add the chunk to the internal buffer.
+ */
+
+ if (smallChunks == null)
+ smallChunks = new LinkedList<IBindingSet[]>();
+
+ // Add to the buffer.
+ smallChunks.add(e);
+ chunkSize += e.length;
+
+ } // synchronized(this)
+
+ }
+
/**
* Output a chunk, updating the counters.
*
@@ -1630,12 +1669,6 @@
sinkMessagesOut.addAndGet(messagesOut);
-// try {
-// q.outstandingMessageSemaphore.acquire();
-// } catch (InterruptedException e1) {
-// throw new RuntimeException(e1);
-// }
-
}
/**
@@ -1643,15 +1676,6 @@
*/
synchronized // Note: has side-effect on internal buffer.
private void outputBufferedChunk() {
-// if (chunk == null || chunkSize == 0)
-// return;
-// if (chunkSize != chunk.length) {
-// // truncate the array.
-// chunk = Arrays.copyOf(chunk, chunkSize);
-// }
-// outputChunk(chunk);
-// chunkSize = 0;
-// chunk = null;
if (smallChunks == null || chunkSize == 0) {
return;
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/MemorySortOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/MemorySortOp.java 2014-01-13 19:58:29 UTC (rev 7786)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/MemorySortOp.java 2014-01-13 21:55:30 UTC (rev 7787)
@@ -18,6 +18,7 @@
import com.bigdata.bop.IVariableOrConstant;
import com.bigdata.bop.engine.BOpStats;
import com.bigdata.bop.engine.IRunningQuery;
+import com.bigdata.bop.solutions.SliceOp.Annotations;
import com.bigdata.rdf.error.SparqlTypeErrorException;
import com.bigdata.rdf.internal.IV;
import com.bigdata.relation.accesspath.IBlockingBuffer;
@@ -102,6 +103,11 @@
+ "=" + isLastPassRequested());
}
+ // ORDER_BY must preserve order.
+ if (isReorderSolutions())
+ throw new UnsupportedOperationException(
+ Annotations.REORDER_SOLUTIONS + "=" + isReorderSolutions());
+
// required parameter.
getValueComparator();
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java 2014-01-13 19:58:29 UTC (rev 7786)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java 2014-01-13 21:55:30 UTC (rev 7787)
@@ -150,7 +150,12 @@
if (!isSharedState())
throw new UnsupportedOperationException(Annotations.SHARED_STATE
+ "=" + isSharedState());
-
+
+ // SLICE must preserve order.
+ if (isReorderSolutions())
+ throw new UnsupportedOperationException(
+ Annotations.REORDER_SOLUTIONS + "=" + isReorderSolutions());
+
}
/**
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java 2014-01-13 19:58:29 UTC (rev 7786)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java 2014-01-13 21:55:30 UTC (rev 7787)
@@ -489,6 +489,7 @@
new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER),//
new NV(PipelineOp.Annotations.SHARED_STATE,true),//
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,false),//
// new NV(
// QueryEngineTestAnnotations.COMBINE_RECEIVED_CHUNKS,
// false),//
@@ -784,6 +785,7 @@
new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER),//
new NV(PipelineOp.Annotations.SHARED_STATE,true),//
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,false),//
// // Require the chunked running query impl.
// new NV(QueryEngine.Annotations.RUNNING_QUERY_CLASS,
// ChunkedRunningQuery.class.getName()),//
@@ -950,6 +952,7 @@
new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER),//
new NV(PipelineOp.Annotations.SHARED_STATE,true),//
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,false),//
})//
);
@@ -1102,6 +1105,7 @@
new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER),//
new NV(PipelineOp.Annotations.SHARED_STATE,true),//
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,false),//
})//
);
@@ -1580,6 +1584,7 @@
new NV(BOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER),//
new NV(PipelineOp.Annotations.SHARED_STATE,true),//
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,false),//
}));
final PipelineOp query = sliceOp;
@@ -1952,6 +1957,7 @@
new NV(BOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER),//
new NV(PipelineOp.Annotations.SHARED_STATE,true),//
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,false),//
}));
final PipelineOp query = sliceOp;
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine_Slice.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine_Slice.java 2014-01-13 19:58:29 UTC (rev 7786)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine_Slice.java 2014-01-13 21:55:30 UTC (rev 7787)
@@ -200,6 +200,7 @@
new NV(SliceOp.Annotations.LIMIT, limit),//
new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER),//
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,false),//
}));
final UUID queryId = UUID.randomUUID();
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine_SortOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine_SortOp.java 2014-01-13 19:58:29 UTC (rev 7786)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine_SortOp.java 2014-01-13 21:55:30 UTC (rev 7787)
@@ -72,6 +72,7 @@
*/
public class TestQueryEngine_SortOp extends TestCase2 {
+ @Override
public Properties getProperties() {
final Properties p = new Properties(super.getProperties());
@@ -240,6 +241,7 @@
new NV(MemorySortOp.Annotations.MAX_PARALLEL, 1),//
// new NV(MemorySortOp.Annotations.SHARED_STATE, true),//
new NV(MemorySortOp.Annotations.LAST_PASS, true),//
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,false),//
}));
final UUID queryId = UUID.randomUUID();
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/solutions/TestMemorySortOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/solutions/TestMemorySortOp.java 2014-01-13 19:58:29 UTC (rev 7786)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/solutions/TestMemorySortOp.java 2014-01-13 21:55:30 UTC (rev 7787)
@@ -42,6 +42,7 @@
import com.bigdata.bop.IQueryContext;
import com.bigdata.bop.IVariable;
import com.bigdata.bop.NV;
+import com.bigdata.bop.PipelineOp;
import com.bigdata.bop.Var;
import com.bigdata.bop.bindingSet.ListBindingSet;
import com.bigdata.bop.engine.AbstractQueryEngineTestCase;
@@ -132,6 +133,7 @@
new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER),//
new NV(MemorySortOp.Annotations.MAX_PARALLEL, 1),//
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,false),//
// new NV(MemorySortOp.Annotations.SHARED_STATE, true),//
new NV(MemorySortOp.Annotations.LAST_PASS, true),//
}));
@@ -244,6 +246,7 @@
new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER),//
new NV(MemorySortOp.Annotations.MAX_PARALLEL, 1),//
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,false),//
// new NV(MemorySortOp.Annotations.SHARED_STATE, true),//
new NV(MemorySortOp.Annotations.LAST_PASS, true),//
}));
@@ -360,6 +363,7 @@
new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER),//
new NV(MemorySortOp.Annotations.MAX_PARALLEL, 1),//
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,false),//
// new NV(MemorySortOp.Annotations.SHARED_STATE, true),//
new NV(MemorySortOp.Annotations.LAST_PASS, true),//
}));
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/solutions/TestSliceOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/solutions/TestSliceOp.java 2014-01-13 19:58:29 UTC (rev 7786)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/solutions/TestSliceOp.java 2014-01-13 21:55:30 UTC (rev 7787)
@@ -177,6 +177,7 @@
new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER),//
new NV(PipelineOp.Annotations.SHARED_STATE,true),//
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,false),//
}));
assertEquals("offset", offset, query.getOffset());
@@ -288,6 +289,7 @@
new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER),//
new NV(PipelineOp.Annotations.SHARED_STATE,true),//
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,false),//
}));
assertEquals("offset", offset, query.getOffset());
@@ -367,6 +369,7 @@
new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER),//
new NV(PipelineOp.Annotations.SHARED_STATE,true),//
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,false),//
}));
assertEquals("offset", offset, query.getOffset());
@@ -442,6 +445,7 @@
new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER),//
new NV(PipelineOp.Annotations.SHARED_STATE,true),//
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,false),//
}));
assertEquals("offset", offset, query.getOffset());
@@ -530,6 +534,7 @@
new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER),//
new NV(PipelineOp.Annotations.SHARED_STATE,true),//
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,false),//
}));
assertEquals("offset", 0L, query.getOffset());
@@ -591,6 +596,7 @@
new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER),//
new NV(PipelineOp.Annotations.SHARED_STATE,true),//
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,false),//
}));
assertEquals("offset", -1L, query.getOffset());
@@ -632,6 +638,7 @@
new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER),//
new NV(PipelineOp.Annotations.SHARED_STATE,true),//
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,false),//
}));
assertEquals("offset", 1L, query.getOffset());
@@ -709,6 +716,7 @@
new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
BOpEvaluationContext.CONTROLLER),//
new NV(PipelineOp.Annotations.SHARED_STATE,true),//
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,false),//
}));
final SliceStats stats = query.newStats();
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpFilters.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpFilters.java 2014-01-13 19:58:29 UTC (rev 7786)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpFilters.java 2014-01-13 21:55:30 UTC (rev 7787)
@@ -521,6 +521,11 @@
* "no reordering" guarantee.
*/
+ // disable reordering of solutions for cutoff joins.
+ final boolean reorderSolutions = (cutoffLimit != null) ? false
+ : PipelineJoin.Annotations.DEFAULT_REORDER_SOLUTIONS;
+
+ // disable operator parallelism for cutoff joins.
final int maxParallel = cutoffLimit != null ? 1
: PipelineOp.Annotations.DEFAULT_MAX_PARALLEL;
@@ -530,6 +535,7 @@
new NV(ChunkedMaterializationOp.Annotations.TIMESTAMP, timestamp), //
new NV(ChunkedMaterializationOp.Annotations.MATERIALIZE_INLINE_IVS, materializeInlineIvs), //
new NV(PipelineOp.Annotations.SHARED_STATE, !ctx.isCluster()),// live stats, but not on the cluster.
+ new NV(PipelineOp.Annotations.REORDER_SOLUTIONS,reorderSolutions),//
new NV(PipelineOp.Annotations.MAX_PARALLEL,maxParallel),//
new NV(BOp.Annotations.BOP_ID, ctx.nextId())//
), queryHints, ctx);
@@ -729,6 +735,8 @@
new ConditionalRoutingOp(leftOrEmpty(left),//
new NV(BOp.Annotations.BOP_ID, ctx.nextId()),//
new NV(PipelineOp.Annotations.MAX_PARALLEL, 1),//
+ // disallow reordering of solutions by the query engine.
+ new NV(PipelineJoin.Annotations.REORDER_SOLUTIONS, Boolean.FALSE),//
new NV(ConditionalRoutingOp.Annotations.CONDITION, c)//
), queryHints, ctx);
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpJoins.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpJoins.java 2014-01-13 19:58:29 UTC (rev 7786)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpJoins.java 2014-01-13 21:55:30 UTC (rev 7787)
@@ -1050,6 +1050,9 @@
.get(Annotations.SIMPLE_JOIN)).booleanValue()
&& !AST2BOpRTO.runAllJoinsAsComplexJoins;
+ // disallow reordering of solutions by the query engine.
+ map.put(PipelineJoin.Annotations.REORDER_SOLUTIONS, Boolean.FALSE);
+
// disallow parallel evaluation of tasks
map.put(PipelineOp.Annotations.MAX_PARALLEL, Integer.valueOf(1));
@@ -1060,7 +1063,7 @@
// disable access path coalescing
map.put(PipelineJoin.Annotations.COALESCE_DUPLICATE_ACCESS_PATHS,
Boolean.FALSE);
-
+
/*
* Disable access path reordering.
*
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java 2014-01-13 19:58:29 UTC (rev 7786)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java 2014-01-13 21:55:30 UTC (rev 7787)
@@ -61,7 +61,6 @@
import com.bigdata.bop.Var;
import com.bigdata.bop.ap.Predicate;
import com.bigdata.bop.ap.SampleIndex.SampleType;
-import com.bigdata.bop.bset.ConditionalRoutingOp;
import com.bigdata.bop.engine.IRunningQuery;
import com.bigdata.bop.engine.QueryEngine;
import com.bigdata.bop.join.JoinAnnotations;
@@ -77,7 +76,6 @@
import com.bigdata.bop.joinGraph.rto.VertexSample;
import com.bigdata.bop.rdf.join.ChunkedMaterializationOp;
import com.bigdata.bop.rdf.join.DataSetJoin;
-import com.bigdata.bop.solutions.MemorySortOp;
import com.bigdata.bop.solutions.SliceOp;
import com.bigdata.journal.IIndexManager;
import com.bigdata.rdf.internal.IV;
@@ -219,22 +217,23 @@
* estimated cardinality of the join since we can not compute the join hit
* ratio without knowing the #of solutions in required to produce a given
* #of solutions out.
+ * <p>
+ * There are several possible root causes for out of order evaluation. One
+ * is reordering of access paths in the pipeline join. Another is reordering
+ * of solutions when they are output from an operator. Another is an
+ * operator which uses the altSink and thus does not always route solutions
+ * along a single path.
*
- * FIXME Make this <code>true</code>. There is a known problem where a
- * {@link ConditionalRoutingOp} can cause out of order evaluation if some
- * solutions flow along the default sink and some along the alt sink. I
- * think that the fix for this is to make the materialization step
- * non-conditional when performing cutoff evaluation of the join. I need to
- * run this past MikeP, so this allows out-of-order evaluation for the
- * moment. See BSBM Q5 for a query that currently fails if out of order
- * evaluation is disallowed.
+ * @see #checkQueryPlans
*/
- static final private boolean failOutOfOrderEvaluation = false;
+ static final private boolean failOutOfOrderEvaluation = true;
/**
* When <code>true</code>, the generated query plans for cutoff evaluation
* will be checked to verify that the query plans do not permit reordering
* of solutions.
+ *
+ * @see #failOutOfOrderEvaluation
*/
static final private boolean checkQueryPlans = false; // Note: Make [false] in committed code!
@@ -904,7 +903,8 @@
BOpEvaluationContext.CONTROLLER),//
new NV(SliceOp.Annotations.PIPELINED, true),//
new NV(SliceOp.Annotations.MAX_PARALLEL, 1),//
- new NV(MemorySortOp.Annotations.SHARED_STATE, true)//
+ new NV(SliceOp.Annotations.REORDER_SOLUTIONS,false),//
+ new NV(SliceOp.Annotations.SHARED_STATE, true)//
), rtoJoinGroup, ctx);
}
@@ -975,6 +975,18 @@
}
+
+ if (tmp.isReorderSolutions()) {
+
+ // reordering of solutions is disallowed.
+ throw new RuntimeException("RTO "
+ + PipelineOp.Annotations.REORDER_SOLUTIONS
+ + ": expected=false, actual="
+ + tmp.isReorderSolutions() + ", op="
+ + tmp.toShortString());
+
+ }
+
if (tmp instanceof PipelineJoin) {
final PipelineJoin<?> t = (PipelineJoin<?>) tmp;
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java 2014-01-13 19:58:29 UTC (rev 7786)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java 2014-01-13 21:55:30 UTC (rev 7787)
@@ -534,7 +534,7 @@
new NV(ProjectionOp.Annotations.SELECT, projectedVars)//
), queryBase, ctx);
- if(materializeProjection) {
+ if (materializeProjection) {
/*
* Note: Materialization done from within the query plan needs
@@ -3620,7 +3620,7 @@
if (projection.isWildcard())
throw new AssertionError("Wildcard projection was not rewritten.");
-
+
final IVariable<?>[] vars = projection.getProjectionVars();
final PipelineOp op;
@@ -3643,6 +3643,7 @@
* BY + DISTINCT)
*/
anns.add(new NV(PipelineOp.Annotations.MAX_PARALLEL, 1));
+ anns.add(new NV(SliceOp.Annotations.REORDER_SOLUTIONS, false));
}
op = new JVMDistinctBindingSetsOp(leftOrEmpty(left),//
anns.toArray(new NV[anns.size()])//
@@ -3959,6 +3960,7 @@
BOpEvaluationContext.CONTROLLER),//
new NV(MemorySortOp.Annotations.PIPELINED, true),//
new NV(MemorySortOp.Annotations.MAX_PARALLEL, 1),//
+ new NV(MemorySortOp.Annotations.REORDER_SOLUTIONS, false),//
// new NV(MemorySortOp.Annotations.SHARED_STATE,
// true),//
new NV(MemorySortOp.Annotations.LAST_PASS, true),//
@@ -3985,7 +3987,8 @@
BOpEvaluationContext.CONTROLLER),//
new NV(SliceOp.Annotations.PIPELINED, true),//
new NV(SliceOp.Annotations.MAX_PARALLEL, 1),//
- new NV(MemorySortOp.Annotations.SHARED_STATE, true)//
+ new NV(SliceOp.Annotations.REORDER_SOLUTIONS,false),//
+ new NV(SliceOp.Annotations.SHARED_STATE, true)//
), queryBase, ctx);
return left;
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-14 16:06:29
|
Revision: 7795
http://bigdata.svn.sourceforge.net/bigdata/?rev=7795&view=rev
Author: thompsonbry
Date: 2014-01-14 16:06:20 +0000 (Tue, 14 Jan 2014)
Log Message:
-----------
Commit provides partial fix for #763. Martyn has modified the CustomByteArrayFrontCodedList to support search against a front-coded list with duplicate keys.
However, the tests developed for the HTree show a problem recovering all keys when duplicates are inserted. I have extended the tests that he developed to cover more dups and a variety of valued for the #of address bits in the HTree. The dups test is now integrated into CI and will show 2 new test failures.
I have generated a new lgpl-utils dependency, published it to our maven repo, and updated the top-level build.properties, .classpath, and pom.xml files.
mvn deploy:deploy-file \
-DgroupId=com.bigdata \
-DartifactId=lgpl-utils \
-Dversion=1.0.7-011414 \
-Dpackaging=jar \
-DrepositoryId=bigdata.releases \
-Durl=scpexe://www.systap.com/srv/www/htdocs/systap.com/maven/releases/ \
-Dfile=bigdata/lib/lgpl-utils-1.0.7-011414.jar
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/.classpath
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/htree/TestAll_HTree.java
branches/BIGDATA_RELEASE_1_3_0/build.properties
branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/build.properties
branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/build.xml
branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/src/java/it/unimi/dsi/fastutil/bytes/custom/CustomByteArrayFrontCodedList.java
branches/BIGDATA_RELEASE_1_3_0/pom.xml
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_3_0/bigdata/lib/lgpl-utils-1.0.7-140114.jar
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/htree/TestDuplicates.java
Removed Paths:
-------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/lib/lgpl-utils-1.0.6-020610.jar
Modified: branches/BIGDATA_RELEASE_1_3_0/.classpath
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/.classpath 2014-01-14 13:57:27 UTC (rev 7794)
+++ branches/BIGDATA_RELEASE_1_3_0/.classpath 2014-01-14 16:06:20 UTC (rev 7795)
@@ -32,7 +32,7 @@
<classpathentry kind="src" path="bigdata-gas/src/java"/>
<classpathentry kind="src" path="bigdata-gas/src/test"/>
<classpathentry exported="true" kind="lib" path="bigdata/lib/dsi-utils-1.0.6-020610.jar"/>
- <classpathentry exported="true" kind="lib" path="bigdata/lib/lgpl-utils-1.0.6-020610.jar"/>
+ <classpathentry exported="true" kind="lib" path="bigdata/lib/lgpl-utils-1.0.7-140114.jar"/>
<classpathentry kind="lib" path="bigdata-jini/lib/apache/zookeeper-3.3.3.jar"/>
<classpathentry exported="true" kind="lib" path="bigdata/lib/jetty/jetty-continuation-7.2.2.v20101205.jar"/>
<classpathentry exported="true" kind="lib" path="bigdata/lib/jetty/jetty-http-7.2.2.v20101205.jar"/>
Deleted: branches/BIGDATA_RELEASE_1_3_0/bigdata/lib/lgpl-utils-1.0.6-020610.jar
===================================================================
(Binary files differ)
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata/lib/lgpl-utils-1.0.7-140114.jar
===================================================================
(Binary files differ)
Property changes on: branches/BIGDATA_RELEASE_1_3_0/bigdata/lib/lgpl-utils-1.0.7-140114.jar
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/htree/TestAll_HTree.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/htree/TestAll_HTree.java 2014-01-14 13:57:27 UTC (rev 7794)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/htree/TestAll_HTree.java 2014-01-14 16:06:20 UTC (rev 7795)
@@ -92,6 +92,8 @@
suite.addTestSuite(TestReopen.class);
// test of storing null values under a key with persistence.
suite.addTestSuite(TestNullValues.class);
+ // test duplicate keys (with index checkpoint).
+ suite.addTestSuite(TestDuplicates.class);
/*
* test of transient HTree's (no backing store).
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/htree/TestDuplicates.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/htree/TestDuplicates.java (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/htree/TestDuplicates.java 2014-01-14 16:06:20 UTC (rev 7795)
@@ -0,0 +1,297 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2011. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+package com.bigdata.htree;
+
+import java.io.IOException;
+
+import com.bigdata.btree.ITupleIterator;
+import com.bigdata.rawstore.IRawStore;
+import com.bigdata.rawstore.SimpleMemoryRawStore;
+
+/**
+ * Test {@link HTree} with duplicate Keys.
+ *
+ * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/763" >
+ * Stochastic Results With Analytic Query Mode </a>
+ *
+ * @author Martyn Cutcher
+ */
+public class TestDuplicates extends AbstractHTreeTestCase {
+
+ /**
+ *
+ */
+ public TestDuplicates() {
+
+ }
+
+ /**
+ * @param name
+ */
+ public TestDuplicates(String name) {
+ super(name);
+ }
+
+// private static final boolean bufferNodes = true;
+
+ /**
+ * Tests the ability to store values against duplicate
+ * keys.
+ *
+ * @throws IOException
+ * @throws Exception
+ */
+ public void test_duplicateKeys() throws IOException, Exception {
+
+ final IRawStore store = new SimpleMemoryRawStore();
+
+ try {
+
+ HTree htree = getHTree(store, 3/* addressBits */,
+ false/* rawRecords */, true/* persistent */);
+
+ final byte[] k1 = new byte[] { 1 };
+ final byte[] v2 = new byte[] { 2 };
+ final byte[] v3 = new byte[] { 3 };
+
+ assertNull(htree.lookupFirst(k1));
+ assertFalse(htree.contains(k1));
+
+ assertNull(htree.insert(k1, v2));
+
+ assertEquals(htree.lookupFirst(k1), v2);
+ assertTrue(htree.contains(k1));
+
+ assertNull(htree.insert(k1, v3));
+
+ // test before checkpoint.
+ assertEquals(2, getCount(htree, k1));
+
+ final long addrCheckpoint1 = htree.writeCheckpoint();
+
+ htree = HTree.load(store, addrCheckpoint1, true/* readOnly */);
+
+ assertEquals(htree.lookupFirst(k1), v3);
+ assertTrue(htree.contains(k1));
+
+ // test after checkpoint.
+ assertEquals(2, getCount(htree, k1));
+
+ } finally {
+
+ store.destroy();
+
+ }
+
+ }
+
+ public void test_duplicateKeyRangeScans_3bits_500dups() {
+
+ doTest(3, 500);
+
+ }
+
+ public void test_duplicateKeyRangeScans_4bits_500dups() {
+
+ doTest(4, 500);
+
+ }
+
+ public void test_duplicateKeyRangeScans_10bits_500dups() {
+
+ doTest(10, 500);
+
+ }
+
+ public void test_duplicateKeyRangeScans_3bits_5000dups() {
+
+ doTest(3, 5000);
+
+ }
+
+ public void test_duplicateKeyRangeScans_4bits_5000dups() {
+
+ doTest(4, 5000);
+
+ }
+
+ public void test_duplicateKeyRangeScans_10bits_5000dups() {
+
+ doTest(10, 5000);
+
+ }
+
+ /**
+ * Test helper for a test based on duplicate keys. The test is designed to
+ * verify that the keys are stored correctly and that a scan of the records
+ * having that key returns all entries stored under that key.
+ *
+ * @param addressBits
+ * The #of address bits.
+ * @param ndups
+ * The #of duplicate entries for a given key.
+ *
+ * FIXME If the addressBits is 10 then this test can fail for
+ * specific dups values - eg 500 or 2000. This appears to be an
+ * independent failure at the {@link HTree} layer rather than in
+ * the key buffer search layer.
+ */
+ private void doTest(final int addressBits, final int ndups) {
+
+ final IRawStore store = new SimpleMemoryRawStore();
+ // final IRawStore store = getRWStore();
+
+ try {
+ HTree htree = getHTree(store, addressBits, false/* rawRecords */,
+ true/* persistent */);
+
+ final byte[] k1 = new byte[] { 1, 2, 3, 4 };
+ final byte[] k2 = new byte[] { 2, 3, 4 };
+ final byte[] k3 = new byte[] { 3, 4 };
+ final byte[] v1 = new byte[] { 1 };
+ final byte[] v2 = new byte[] { 2 };
+ final byte[] v3 = new byte[] { 3 };
+
+ for (int i = 0; i < ndups; i++) {
+ assertNull(htree.insert(k1, v1));
+ assertNull(htree.insert(k2, v2));
+ assertNull(htree.insert(k3, v3));
+ }
+
+ assertTrue(htree.contains(k1)); // 2000 dups fails here with 10 bit depth
+ assertTrue(htree.contains(k2));
+ assertTrue(htree.contains(k3));
+
+ // Check first with MutableBuckets
+ assertEquals(ndups, getCount(htree, k1));
+ assertEquals(ndups, getCount(htree, k2));
+ assertEquals(ndups, getCount(htree, k3));
+
+ final long addrCheckpoint1 = htree.writeCheckpoint();
+
+ htree = HTree.load(store, addrCheckpoint1, true/* readOnly */);
+
+ assertTrue(htree.contains(k1));
+ assertTrue(htree.contains(k2));
+ assertTrue(htree.contains(k3));
+
+ // Test after checkpoint.
+ assertEquals(ndups, getCount(htree, k1));
+ assertEquals(ndups, getCount(htree, k2));
+ assertEquals(ndups, getCount(htree, k3));// 500 dups fails here with 10bit depth
+
+ } finally {
+
+ store.destroy();
+
+ }
+
+ }
+
+// private IRawStore getRWStore() {
+//
+// final Properties properties = getProperties();
+//
+// properties.setProperty(Options.CREATE_TEMP_FILE, "true");
+//
+// properties.setProperty(Options.DELETE_ON_EXIT, "true");
+//
+// properties.setProperty(Options.BUFFER_MODE, BufferMode.DiskRW.toString());
+//
+// properties.setProperty(Options.WRITE_CACHE_ENABLED, "" + true);
+//
+// return new Journal(properties);//.getBufferStrategy();
+//
+// }
+
+ /**
+ * Return the #of entries having the specified key.
+ *
+ * @param htree
+ * The index.
+ * @param k1
+ * The key.
+ * @return
+ */
+ private int getCount(final HTree htree, final byte[] k1) {
+ final ITupleIterator<?> iter = htree.lookupAll(k1);
+ int count = 0;
+ while (iter.hasNext()) {
+ iter.next();
+ count++;
+ }
+
+ return count;
+ }
+
+ /**
+ * Simple test where the keys and the values are dups.
+ *
+ * @throws IOException
+ * @throws Exception
+ */
+ public void test_duplicateKeyValues() throws IOException, Exception {
+
+ final IRawStore store = new SimpleMemoryRawStore();
+
+ try {
+
+ HTree htree = getHTree(store, 3/* addressBits */,
+ false/* rawRecord */, true/* persistent */);
+
+ final byte[] k1 = new byte[] { 1 };
+ final byte[] v2 = new byte[] { 2 };
+
+ assertNull(htree.lookupFirst(k1));
+ assertFalse(htree.contains(k1));
+
+ assertNull(htree.insert(k1, v2));
+
+ assertEquals(htree.lookupFirst(k1), v2);
+ assertTrue(htree.contains(k1));
+
+ assertNull(htree.insert(k1, v2));
+
+ // before checkpoint.
+ assertEquals(2, getCount(htree, k1));
+
+ final long addrCheckpoint1 = htree.writeCheckpoint();
+
+ htree = HTree.load(store, addrCheckpoint1, true/* readOnly */);
+
+ assertEquals(htree.lookupFirst(k1), v2);
+ assertTrue(htree.contains(k1));
+
+ // after checkpoint.
+ assertEquals(2, getCount(htree, k1));
+
+ } finally {
+
+ store.destroy();
+
+ }
+
+ }
+
+}
Modified: branches/BIGDATA_RELEASE_1_3_0/build.properties
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/build.properties 2014-01-14 13:57:27 UTC (rev 7794)
+++ branches/BIGDATA_RELEASE_1_3_0/build.properties 2014-01-14 16:06:20 UTC (rev 7795)
@@ -65,7 +65,7 @@
log4j.version=1.2.17
fastutil.version=5.1.5
dsiutils.version=1.0.6-020610
-lgplutils.version=1.0.6-020610
+lgplutils.version=1.0.7-040114
ganglia-version=1.0.1
gas-version=0.1.0
Modified: branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/build.properties
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/build.properties 2014-01-14 13:57:27 UTC (rev 7794)
+++ branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/build.properties 2014-01-14 16:06:20 UTC (rev 7795)
@@ -39,7 +39,7 @@
release.dir=ant-release
# The build version.
-build.ver=1.0.6
+build.ver=1.0.7
# Set true to do a snapshot build. This changes the value of ${version} to
# include the date.
Modified: branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/build.xml
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/build.xml 2014-01-14 13:57:27 UTC (rev 7794)
+++ branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/build.xml 2014-01-14 16:06:20 UTC (rev 7795)
@@ -124,7 +124,7 @@
<bottom>
<![CDATA[
<i>
-Portions copyright © 2006-2009 SYSTAP, LLC. All Rights Reserved.<br>
+Portions copyright © 2006-2014 SYSTAP, LLC. All Rights Reserved.<br>
Portions copyright © 2005-2009 Sebastiano Vigna
</i>]]></bottom>
<tag name="todo" scope="all" description="TODO:" />
Modified: branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/src/java/it/unimi/dsi/fastutil/bytes/custom/CustomByteArrayFrontCodedList.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/src/java/it/unimi/dsi/fastutil/bytes/custom/CustomByteArrayFrontCodedList.java 2014-01-14 13:57:27 UTC (rev 7794)
+++ branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/src/java/it/unimi/dsi/fastutil/bytes/custom/CustomByteArrayFrontCodedList.java 2014-01-14 16:06:20 UTC (rev 7795)
@@ -1373,7 +1373,7 @@
// final int base = 0;
- final BackingBuffer bb = this.bb;
+// final BackingBuffer bb = this.bb;
/*
* We will test each entry having an index that is an even multiple of
@@ -1396,23 +1396,8 @@
/*
* Compare the probe with the full length byte[] at index [mid].
*/
- final int tmp;
- {
+ final int tmp = comparePos(mid, key);
- // The index into the backing buffer of index [mid].
- int pos = p[mid];
-
- // The #of bytes in the full length byte[] at index [mid].
- final int blen = bb.readInt(pos);
-
- // Skip the #of bytes required to code that length.
- pos += count(blen);
-
- // Compare key vs actual (in buffer).
- tmp = compareBytes(key, 0, key.length, bb, pos, blen);
-
- }
-
if (tmp > 0) {
// Actual GT probe, restrict lower bound and try again.
@@ -1425,10 +1410,19 @@
} else {
- // Found: return offset.
+ // duplicate check to see if previous is also a match
+ if (mid > 0 && comparePos(mid - 1, key) == 0) {
- return offset;
+ // in which case set it as the highest
+ high = mid - 1;
+ } else {
+
+ // Found: return offset.
+ return offset;
+
+ }
+
}
}
@@ -1440,7 +1434,35 @@
return -(offset + 1);
}
+
+ /**
+ * Compares the caller's key to a full length key at a specific offset
+ * in the {@link BackingBuffer}.
+ *
+ * @param index
+ * The index into the full length keys.
+ * @param key
+ * The probe key.
+ *
+ * @return A value which indicates whether the key at that offset into the
+ * backing buffer is LT, GT, or EQ to the caller's key.
+ */
+ private int comparePos(final int index, final byte[] key) {
+ // The index into the backing buffer of index [index].
+ int pos = p[index];
+
+ // The #of bytes in the full length byte[] at index [index].
+ final int blen = bb.readInt(pos);
+
+ // Skip the #of bytes required to code that length.
+ pos += count(blen);
+
+ // Compare key vs actual (in buffer).
+ return compareBytes(key, 0, key.length, bb, pos, blen);
+
+ }
+
/**
* Compare up to <i>len</i> bytes in <i>a</i> interpreted as unsigned bytes
* against the bytes in the {@link BackingBuffer} starting at offset
Modified: branches/BIGDATA_RELEASE_1_3_0/pom.xml
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/pom.xml 2014-01-14 13:57:27 UTC (rev 7794)
+++ branches/BIGDATA_RELEASE_1_3_0/pom.xml 2014-01-14 16:06:20 UTC (rev 7795)
@@ -94,7 +94,7 @@
<log4j.version>1.2.17</log4j.version>
<fastutil.version>5.1.5</fastutil.version>
<dsiutils.version>1.0.6-020610</dsiutils.version>
- <lgplutils.version>1.0.6-020610</lgplutils.version>
+ <lgplutils.version>1.0.7-140114</lgplutils.version>
<bigdata.ganglia.version>1.0.1</bigdata.ganglia.version>
</properties>
<!-- TODO Can we declare the versions of the dependencies here as
@@ -387,11 +387,11 @@
mvn deploy:deploy-file \
-DgroupId=com.bigdata \
-DartifactId=lgpl-utils \
- -Dversion=1.0.6-020610 \
+ -Dversion=1.0.7-140114 \
-Dpackaging=jar \
-DrepositoryId=bigdata.releases \
-Durl=scpexe://www.systap.com/srv/www/htdocs/systap.com/maven/releases/ \
- -Dfile=bigdata/lib/lgpl-utils-1.0.6-020610.jar
+ -Dfile=bigdata/lib/lgpl-utils-1.0.7-140114.jar
-->
<groupId>com.bigdata</groupId>
<artifactId>lgpl-utils</artifactId>
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-14 16:51:52
|
Revision: 7797
http://bigdata.svn.sourceforge.net/bigdata/?rev=7797&view=rev
Author: thompsonbry
Date: 2014-01-14 16:51:46 +0000 (Tue, 14 Jan 2014)
Log Message:
-----------
Commit of version of new lgpl-utils jar compiled with 1.6 compatability.
Modified lgpl-utils to build for 1.6 compatibility.
See #764 (Stochastic results in analytic query mode).
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/build.properties
branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/build.xml
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_3_0/bigdata/lib/lgpl-utils-1.0.7-140114.jar
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata/lib/lgpl-utils-1.0.7-140114.jar
===================================================================
(Binary files differ)
Property changes on: branches/BIGDATA_RELEASE_1_3_0/bigdata/lib/lgpl-utils-1.0.7-140114.jar
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Modified: branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/build.properties
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/build.properties 2014-01-14 16:50:34 UTC (rev 7796)
+++ branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/build.properties 2014-01-14 16:51:46 UTC (rev 7797)
@@ -20,8 +20,8 @@
# debuglevel=lines,vars,source (or any combination thereof).
javac.debuglevel=lines,vars,source
javac.verbose=off
-#javac.target=1.6
-#javac.source=1.6
+javac.target=1.6
+javac.source=1.6
javac.encoding=Cp1252
# where to find the unimi dependencies.
Modified: branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/build.xml
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/build.xml 2014-01-14 16:50:34 UTC (rev 7796)
+++ branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/build.xml 2014-01-14 16:51:46 UTC (rev 7797)
@@ -85,6 +85,8 @@
<javac destdir="${build.dir}/classes" classpathref="build.classpath"
debug="${javac.debug}" debuglevel="${javac.debuglevel}" verbose="${javac.verbose}"
encoding="${javac.encoding}"
+ source="${javac.source}"
+ target="${javac.target}"
>
<!-- note: must also specify -bootclasspath and -extdirs when cross-compiling -->
<!-- target="${javac.target}" source="${javac.source}" -->
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-15 12:34:04
|
Revision: 7799
http://bigdata.svn.sourceforge.net/bigdata/?rev=7799&view=rev
Author: thompsonbry
Date: 2014-01-15 12:33:55 +0000 (Wed, 15 Jan 2014)
Log Message:
-----------
javadoc related to query hints.
Modified the javadoc ant task to generate documentation for package private as well as public and protected.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/QueryHints.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AtOnceHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/ChunkSizeHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/PipelineMaxParallelHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/RunFirstHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/RunLastHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/RunOnceHint.java
branches/BIGDATA_RELEASE_1_3_0/build.xml
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/QueryHints.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/QueryHints.java 2014-01-14 16:59:02 UTC (rev 7798)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/QueryHints.java 2014-01-15 12:33:55 UTC (rev 7799)
@@ -341,32 +341,37 @@
*
* @see https://sourceforge.net/apps/trac/bigdata/ticket/283
*/
- String QUERYID = "queryId";//QueryHints.class.getName() + ".queryId";
+ String QUERYID = "queryId";
/**
* This query hint may be applied to any {@link IJoinNode} and marks a
* particular join to be run first among in a particular group. Only one
* "run first" join is permitted in a given group. This query hint is not
- * permitted on optional joins.
+ * permitted on optional joins. This hint must be used with
+ * {@link QueryHintScope#Prior}.
*/
- String RUN_FIRST = "runFirst";//QueryHints.class.getName() + ".runFirst";
+ String RUN_FIRST = "runFirst";
/**
* This query hint may be applied to any {@link IJoinNode} and marks a
* particular join to be run last among in a particular group. Only one
- * "run last" join is permitted in a given group.
+ * "run last" join is permitted in a given group. This hint must be used
+ * with {@link QueryHintScope#Prior}.
*/
- String RUN_LAST = "runLast";//QueryHints.class.getName() + ".runLast";
+ String RUN_LAST = "runLast";
/**
* Query hint indicating whether or not a Sub-Select should be transformed
* into a <em>named subquery</em>, lifting its evaluation out of the main
- * body of the query and replacing the subquery with an INCLUDE. This is
- * similar to {@link #AT_ONCE 'at-once'} evaluation, but creates a different
- * query plan by lifting out a named subquery. It is also only supported for
- * a Sub-Select. The {@link #AT_ONCE} query hint can be applied to other
- * things as well.
+ * body of the query and replacing the subquery with an INCLUDE. This hint
+ * must be used with {@link QueryHintScope#SubQuery}.
* <p>
+ * This is similar to {@link #AT_ONCE 'atOnce'} evaluation, but creates a
+ * different query plan by lifting out a named subquery. The
+ * {@link #RUN_ONCE} query hint is only supported for
+ * {@link QueryHintScope#SubQuery} while {@link #AT_ONCE} query hint can be
+ * applied to other things as well.
+ * <p>
* When <code>true</code>, the subquery will be lifted out. When
* <code>false</code>, the subquery will not be lifted unless other
* semantics require that it be lifted out regardless.
@@ -381,20 +386,23 @@
*
* @see #AT_ONCE
*/
- String RUN_ONCE = "runOnce";//QueryHints.class.getName() + ".runOnce";
+ String RUN_ONCE = "runOnce";
/**
* Query hint indicating whether or not a JOIN (including SERVICE,
- * SUB-SELECT, etc) should be run as an "at-once" operator. All solutions
- * for an "at-once" operator are materialized before the operator is
- * evaluated. It is then evaluated against those materialized solutions
- * exactly once.
+ * SUB-SELECT, etc) should be run as an "atOnce" operator. All solutions for
+ * an "atOnce" operator are materialized before the operator is evaluated.
+ * It is then evaluated against those materialized solutions exactly once.
* <p>
- * Note: "At-once" evaluation is a general property of the query engine.
- * This query hint does not change the structure of the query plan, but
- * simply serves as a directive to the query engine that it should buffer
- * all source solutions before running the operator. This is more general
+ * Note: "atOnce" evaluation is a general property of the query engine. This
+ * query hint does not change the structure of the query plan, but simply
+ * serves as a directive to the query engine that it should buffer all
+ * source solutions before running the operator. This is more general
* purpose than the {@link #RUN_ONCE} query hint.
+ * <p>
+ * This query hint is allowed in any scope. The hint is transferred as an
+ * annotation onto all query plan operators generated from the annotated
+ * scope.
*
* @see #RUN_ONCE
*
@@ -409,11 +417,14 @@
String AT_ONCE = "atOnce";
/**
- * The target chunk (aka vector size) for the operator.
+ * Sets the target chunk size (aka vector size) for the output buffer of the operator.
* <p>
- * Note: The vectored query engine will buffer multiple chunks for an
- * operator before the producer(s) (the operator(s) feeding into the
- * annotated operator) must block.
+ * This query hint does not change the structure of the query plan, but
+ * simply serves as a directive to the query engine that it should allocate
+ * an output buffer for the operator that will emit chunks of the indicated
+ * target capacity. This query hint is allowed in any scope, but is
+ * generally used to effect the behavior of a join group, a subquery, or the
+ * entire query.
*
* @see BufferAnnotations#CHUNK_CAPACITY
*/
@@ -421,6 +432,14 @@
/**
* The maximum parallelism for the operator within the query.
+ * <p>
+ * Note: "maxParallel" evaluation is a general property of the query engine.
+ * This query hint does not change the structure of the query plan, but
+ * simply serves as a directive to the query engine that it should not allow
+ * more than the indicated number of parallel instances of the operator to
+ * execute concurrently. This query hint is allowed in any scope. The hint is
+ * transferred as an annotation onto all query plan operators generated from
+ * the annotated scope.
*
* @see PipelineOp.Annotations#MAX_PARALLEL
*/
@@ -528,7 +547,7 @@
* elements (maximum) should be read from its access path. This
* effectively limits the input into the join.
*
- * @see {@link Annotations#CUTOFF_LIMIT}.
+ * @see Annotations#CUTOFF_LIMIT
*/
String CUTOFF_LIMIT = "cutoffLimit";
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AtOnceHint.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AtOnceHint.java 2014-01-14 16:59:02 UTC (rev 7798)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AtOnceHint.java 2014-01-15 12:33:55 UTC (rev 7799)
@@ -35,14 +35,16 @@
import com.bigdata.rdf.sparql.ast.eval.AST2BOpContext;
/**
- * Query hint marks the operator as requiring "at-once" evaluation. All
- * solutions will be buffered by the {@link QueryEngine} before the operator is
- * evaluated. When it is evaluated, it will receive all solutions in a single
- * invocation of that operator. However, the solutions MAY appear in multiple
- * chunks since the {@link QueryEngine} does not guarantee that the chunk will
- * be merged before the operator is invoked.
+ * Query hint marks the operator as requiring "atOnce" evaluation. All solutions
+ * will be buffered by the {@link QueryEngine} before the operator is evaluated.
+ * When it is evaluated, it will receive all solutions in a single invocation of
+ * that operator. However, the solutions MAY appear in multiple chunks since the
+ * {@link QueryEngine} does not guarantee that the chunk will be merged before
+ * the operator is invoked. This query hint is allowed in any scope. The hint is
+ * transferred as an annotation onto all query plan operators generated from the
+ * annotated scope.
* <p>
- * Note: The "at-once" hint is basically turned into <code>NOT(PIPELINED)</code>.
+ * Note: The "atOnce" hint is basically turned into <code>NOT(PIPELINED)</code>.
*
* @see PipelineOp.Annotations#PIPELINED
*/
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/ChunkSizeHint.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/ChunkSizeHint.java 2014-01-14 16:59:02 UTC (rev 7798)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/ChunkSizeHint.java 2014-01-15 12:33:55 UTC (rev 7799)
@@ -28,17 +28,24 @@
package com.bigdata.rdf.sparql.ast.hints;
import com.bigdata.bop.BufferAnnotations;
-import com.bigdata.bop.IBindingSet;
import com.bigdata.rdf.sparql.ast.QueryHints;
/**
* This is identical to the {@link BufferChunkCapacityHint}, but it is accessed
* through the well known name {@link QueryHints#CHUNK_SIZE}.
* <p>
- * Sets the capacity of the {@link IBindingSet}[]s used to accumulate a chunk of
- * {@link IBindingSet}s (default
- * {@value BufferAnnotations#DEFAULT_CHUNK_CAPACITY}). Partial chunks may be
- * automatically combined into full chunks.
+ * Sets the capacity of the output buffer that used to accumulate chunks of
+ * solutions (default {@value BufferAnnotations#DEFAULT_CHUNK_CAPACITY}).
+ * Partial chunks may be automatically combined into full chunks.
+ * <p>
+ * Note: The "chunkSize" is a general property of the query engine. This query
+ * hint does not change the structure of the query plan, but simply serves as a
+ * directive to the query engine that it should allocate an output buffer for
+ * the operator that will emit chunks of the indicated target capacity. This
+ * query hint is allowed in any scope, but is generally used to effect the
+ * behavior of a join group, a subquery, or the entire query. The hint is
+ * transferred as an annotation onto all query plan operators generated from the
+ * annotated scope.
*
* @see BufferAnnotations#CHUNK_CAPACITY
*/
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/PipelineMaxParallelHint.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/PipelineMaxParallelHint.java 2014-01-14 16:59:02 UTC (rev 7798)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/PipelineMaxParallelHint.java 2014-01-15 12:33:55 UTC (rev 7799)
@@ -36,6 +36,14 @@
/**
* Sets the maximum #of operator evaluation tasks which can execute
* concurrently.
+ * <p>
+ * Note: "maxParallel" is a general property of the query engine. This query
+ * hint does not change the structure of the query plan, but simply serves as a
+ * directive to the query engine that it should not allow more than the
+ * indicated number of parallel instances of the operator to execute
+ * concurrently. This query hint is allowed in any scope. The hint is
+ * transferred as an annotation onto all query plan operators generated from the
+ * annotated scope.
*
* @see PipelineOp.Annotations#MAX_PARALLEL
*/
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/RunFirstHint.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/RunFirstHint.java 2014-01-14 16:59:02 UTC (rev 7798)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/RunFirstHint.java 2014-01-15 12:33:55 UTC (rev 7799)
@@ -35,7 +35,8 @@
import com.bigdata.rdf.sparql.ast.optimizers.ASTStaticJoinOptimizer;
/**
- * Query hint to run a join first in a join group.
+ * Query hint to run a join first in a join group. This hint must be used
+ * with {@link QueryHintScope#Prior}.
* <p>
* Note: This sets an AST annotation which is interpreted by the
* {@link ASTRunFirstRunLastOptimizer} and {@link ASTStaticJoinOptimizer}.
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/RunLastHint.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/RunLastHint.java 2014-01-14 16:59:02 UTC (rev 7798)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/RunLastHint.java 2014-01-15 12:33:55 UTC (rev 7799)
@@ -35,7 +35,8 @@
import com.bigdata.rdf.sparql.ast.optimizers.ASTStaticJoinOptimizer;
/**
- * Query hint to run a join last in a join group.
+ * Query hint to run a join last in a join group. This hint must be used with
+ * {@link QueryHintScope#Prior}.
* <p>
* Note: This sets an AST annotation which is interpreted by the
* {@link ASTRunFirstRunLastOptimizer} and {@link ASTStaticJoinOptimizer}.
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/RunOnceHint.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/RunOnceHint.java 2014-01-14 16:59:02 UTC (rev 7798)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/RunOnceHint.java 2014-01-15 12:33:55 UTC (rev 7799)
@@ -40,6 +40,7 @@
* and replacing the subquery with an INCLUDE. When <code>true</code>, the
* subquery will be lifted out. When <code>false</code>, the subquery will not
* be lifted unless other semantics require that it be lifted out regardless.
+ * This hint must be used with {@link QueryHintScope#SubQuery}.
* <p>
* For example, the following may be used to lift out the sub-select in which it
* appears into a {@link NamedSubqueryRoot}. The lifted expression will be
Modified: branches/BIGDATA_RELEASE_1_3_0/build.xml
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/build.xml 2014-01-14 16:59:02 UTC (rev 7798)
+++ branches/BIGDATA_RELEASE_1_3_0/build.xml 2014-01-15 12:33:55 UTC (rev 7799)
@@ -385,6 +385,10 @@
overview="${bigdata.dir}/overview.html"
windowtitle="bigdata® v${build.ver}"
classpathref="build.classpath"
+ package="true"
+ protected="true"
+ public="true"
+ private="false"
>
<arg value="-J-Xmx1000m" />
<arg value="-quiet" />
@@ -401,7 +405,7 @@
<doctitle>
<![CDATA[<h1>bigdata® v${build.ver}</h1>]]></doctitle>
<bottom>
- <![CDATA[<i>Copyright © 2006-2012 SYSTAP, LLC. All Rights Reserved.</i>]]></bottom>
+ <![CDATA[<i>Copyright © 2006-2014 SYSTAP, LLC. All Rights Reserved.</i>]]></bottom>
<tag name="todo" scope="all" description="TODO:" />
<tag name="issue" scope="all" description="ISSUE:" />
<!--tag name="FIXME" scope="all" description="FIXME:"/-->
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-15 17:11:35
|
Revision: 7807
http://bigdata.svn.sourceforge.net/bigdata/?rev=7807&view=rev
Author: thompsonbry
Date: 2014-01-15 17:11:28 +0000 (Wed, 15 Jan 2014)
Log Message:
-----------
@Override and final annotations.
toString() for some types of filters that appear in named and default graph APs.
License headers.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/relation/accesspath/ElementFilter.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/spo/InGraphBinarySearchFilter.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/spo/InGraphHashSetFilter.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPOFilter.java
branches/BIGDATA_RELEASE_1_3_0/ctc-striterators/src/java/cutthecrap/utils/striterators/FilterBase.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/relation/accesspath/ElementFilter.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/relation/accesspath/ElementFilter.java 2014-01-15 16:36:55 UTC (rev 7806)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/relation/accesspath/ElementFilter.java 2014-01-15 17:11:28 UTC (rev 7807)
@@ -138,4 +138,17 @@
}
+ /**
+ * {@inheritDoc}
+ * <p>
+ * Extended to show a human readable representation of the test.
+ */
+ @Override
+ public String toString() {
+ final StringBuilder sb = new StringBuilder();
+ sb.append(super.toString());
+ sb.append("{test=" + test);
+ sb.append("}");
+ return sb.toString();
+ }
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/spo/InGraphBinarySearchFilter.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/spo/InGraphBinarySearchFilter.java 2014-01-15 16:36:55 UTC (rev 7806)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/spo/InGraphBinarySearchFilter.java 2014-01-15 17:11:28 UTC (rev 7807)
@@ -24,6 +24,7 @@
*
* @see InGraphHashSetFilter
*/
+@SuppressWarnings("rawtypes")
public final class InGraphBinarySearchFilter<E extends ISPO> extends SPOFilter<E>
implements Externalizable {
@@ -76,7 +77,8 @@
}
- public boolean isValid(Object o) {
+ @Override
+ public boolean isValid(final Object o) {
if (!canAccept(o)) {
@@ -96,7 +98,14 @@
}
- /**
+ @Override
+ public String toString() {
+
+ return getClass().getName() + "{size=" + a.length + "}";
+
+ }
+
+ /**
* The initial version.
*/
private static final transient short VERSION0 = 0;
@@ -106,6 +115,7 @@
*/
private static final transient short VERSION = VERSION0;
+ @Override
public void readExternal(final ObjectInput in) throws IOException,
ClassNotFoundException {
@@ -131,6 +141,7 @@
}
+ @Override
public void writeExternal(final ObjectOutput out) throws IOException {
out.writeShort(VERSION);
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/spo/InGraphHashSetFilter.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/spo/InGraphHashSetFilter.java 2014-01-15 16:36:55 UTC (rev 7806)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/spo/InGraphHashSetFilter.java 2014-01-15 17:11:28 UTC (rev 7807)
@@ -1,3 +1,18 @@
+/**
+ Copyright (C) SYSTAP, LLC 2006-2012. All rights reserved.
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+*/
package com.bigdata.rdf.spo;
import java.util.HashSet;
@@ -2,6 +17,3 @@
-import org.openrdf.model.URI;
-
import com.bigdata.rdf.internal.IV;
-import com.bigdata.rdf.model.BigdataURI;
@@ -19,6 +31,7 @@
*
* @see InGraphBinarySearchFilter
*/
+@SuppressWarnings("rawtypes")
public final class InGraphHashSetFilter<E extends ISPO> extends SPOFilter<E> {
/**
@@ -55,7 +68,8 @@
}
- public boolean isValid(Object o) {
+ @Override
+ public boolean isValid(final Object o) {
if (!canAccept(o)) {
@@ -75,4 +89,11 @@
}
+ @Override
+ public String toString() {
+
+ return getClass().getName() + "{size=" + contextSet.size() + "}";
+
+ }
+
}
\ No newline at end of file
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPOFilter.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPOFilter.java 2014-01-15 16:36:55 UTC (rev 7806)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPOFilter.java 2014-01-15 17:11:28 UTC (rev 7807)
@@ -1,3 +1,18 @@
+/**
+ Copyright (C) SYSTAP, LLC 2006-2012. All rights reserved.
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+*/
package com.bigdata.rdf.spo;
import com.bigdata.relation.accesspath.IElementFilter;
@@ -9,6 +24,7 @@
*/
private static final long serialVersionUID = 1L;
+ @Override
public boolean canAccept(final Object o) {
return o instanceof ISPO;
Modified: branches/BIGDATA_RELEASE_1_3_0/ctc-striterators/src/java/cutthecrap/utils/striterators/FilterBase.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/ctc-striterators/src/java/cutthecrap/utils/striterators/FilterBase.java 2014-01-15 16:36:55 UTC (rev 7806)
+++ branches/BIGDATA_RELEASE_1_3_0/ctc-striterators/src/java/cutthecrap/utils/striterators/FilterBase.java 2014-01-15 17:11:28 UTC (rev 7807)
@@ -171,6 +171,7 @@
/**
* Human readable representation of the filter chain.
*/
+ @Override
public String toString() {
final StringBuilder sb = new StringBuilder();
sb.append(super.toString());
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-15 17:20:11
|
Revision: 7808
http://bigdata.svn.sourceforge.net/bigdata/?rev=7808&view=rev
Author: thompsonbry
Date: 2014-01-15 17:20:04 +0000 (Wed, 15 Jan 2014)
Log Message:
-----------
javadoc on issues with named graph, default graph, and scale-out joins.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/Vertex.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpJoins.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/Vertex.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/Vertex.java 2014-01-15 17:11:28 UTC (rev 7807)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/Vertex.java 2014-01-15 17:20:04 UTC (rev 7808)
@@ -151,6 +151,41 @@
}
+ /*
+ * FIXME RTO: AST2BOpJoins is responsible for constructing the
+ * appropriate access path. Under some cases it can emit a DataSetJoin
+ * followed by a join against the access path. Under other cases, it
+ * will use a SCAN+FILTER pattern and attach a filter. The code below
+ * does not benefit from any of this because the vertex created from the
+ * [pred] before we invoke AST2BOpJoin#join() and hence lacks all of
+ * these interesting and critical annotations. When generating the join
+ * graph, the RTO needs to emit a set of vertices and filters that is
+ * sufficient for joins for named graphs and default graphs. It also
+ * needs to emit a set of predicates and filters that is sufficient for
+ * triples mode joins.
+ *
+ * Some possible approaches:
+ *
+ * - For the RTO, always do a DataSetJoin + SP. We would need to support
+ * the DataSetJoin as a Predicate (it does not get modeled that way
+ * right now). The SP would need to have the DISTINCT SPO filter
+ * attached for a default graph join. This might even be a DISTINCT
+ * FILTER that gets into the plan and winds up attached to either the
+ * DataSetJoin or the SP, depending on which runs first. This would give
+ * us two APs plus a visible FILTER rather than ONE AP with some hidden
+ * filters. The DataSetJoin would need to be associated with an AP that
+ * binds the (hidden) graph variable. This could be an opporunity to
+ * generalize for storing those data on the native heap / htree / etc. /
+ * named solution set as well.
+ *
+ * Basically, this amounts to saying that we will sample both the set of
+ * graphs that are in the named graphs or default graphs data set and
+ * the unconstrained triple pattern AP.
+ *
+ * - If C is bound, then we should just wind up with a FILTER that is
+ * imposing the DISTINCT SPO (for default graph APs) and do not need to
+ * do anything (for named graph AP)s.
+ */
final BOpContextBase context = new BOpContextBase(queryEngine);
final IRelation r = context.getRelation(pred);
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpJoins.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpJoins.java 2014-01-15 17:11:28 UTC (rev 7807)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpJoins.java 2014-01-15 17:20:04 UTC (rev 7808)
@@ -698,7 +698,6 @@
final int accessPathSampleLimit = pred.getProperty(
QueryHints.ACCESS_PATH_SAMPLE_LIMIT, ctx.accessPathSampleLimit);
final boolean estimateCosts = accessPathSampleLimit >= 0;
- final IRelation r = ctx.context.getRelation(pred);
final ScanCostReport scanCostReport;
final SubqueryCostReport subqueryCostReport;
final boolean scanAndFilter;
@@ -736,6 +735,7 @@
* the cost of the scan regardless of whether the query runs with
* partitioned or global index views when it is evaluated.
*/
+ final IRelation r = ctx.context.getRelation(pred);
scanCostReport = ((AccessPath) ctx.context.getAccessPath(r,
(Predicate<?>) pred.setProperty(
IPredicate.Annotations.REMOTE_ACCESS_PATH, true)))
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-23 11:50:46
|
Revision: 7820
http://bigdata.svn.sourceforge.net/bigdata/?rev=7820&view=rev
Author: thompsonbry
Date: 2014-01-23 11:50:35 +0000 (Thu, 23 Jan 2014)
Log Message:
-----------
Removed the stale overview.html. This has been replaced by a 25 page whitepaper linked from the blog [1].
Modified the README to point people to the release notes. The release notes have pointers for getting started with the platform.
Moved the JINI README into bigdata-jini/src/resources.
[1] http://www.bigdata.com/whitepapers/bigdata_architecture_whitepaper.pdf
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/README
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_3_0/bigdata-jini/src/resources/README-JINI
Removed Paths:
-------------
branches/BIGDATA_RELEASE_1_3_0/overview.html
Modified: branches/BIGDATA_RELEASE_1_3_0/README
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/README 2014-01-23 11:33:50 UTC (rev 7819)
+++ branches/BIGDATA_RELEASE_1_3_0/README 2014-01-23 11:50:35 UTC (rev 7820)
@@ -0,0 +1,4 @@
+Please see the release notes in bigdata/src/releases for getting started
+links. This will point you to the installation instructions for the
+different deployment modes, the online documentation, the wiki, etc. It
+will also point you to resources for support, subscriptions, and licensing.
Copied: branches/BIGDATA_RELEASE_1_3_0/bigdata-jini/src/resources/README-JINI (from rev 7775, branches/BIGDATA_RELEASE_1_3_0/README-JINI)
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-jini/src/resources/README-JINI (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-jini/src/resources/README-JINI 2014-01-23 11:50:35 UTC (rev 7820)
@@ -0,0 +1,160 @@
+Some notes on installation and use follow:
+
+JINI
+
+- jini is used as a service fabric for bigdata. <start up jini and
+ then configure your data and metadata services; clients then
+ discover those services>
+
+- jini 2.1 may report errors locating the shared libraries from awk,
+ dirname, basename and grep when installing under un*x. The problem
+ is an assumption about the kernel version. This problem is resolved
+ by editing the installer and the launchall script.
+ See http://www.jini.org/wiki/Category:Getting_Started and
+ http://www.linuxquestions.org/questions/showthread.php?t=370056 for
+ a resolution. Here is is in case that link goes away:
+
+ Open the bin installer file in an editor. Look for the line
+
+ export LD_ASSUME_KERNEL=2.2.5
+
+ and replace it with
+
+ #xport LD_ASSUME_KERNEL=2.2.5
+
+ Save the file and launch.
+
+ Once jini is installed, you need to do exactly the same thing for the
+ Launch-All script in the installverify directory - this is the script
+ that you use the start jini.
+
+- Here are some notes on getting things working on a Fedora Core 6 platform.
+ The symptom was that ifconfig was reporting MULTICAST for the interface
+ but the jini install was complaining that multicase was not enabled for
+ that interface.
+
+ Here's what I did:
+
+ First, verify that multicast is enabled on eth0 by typing ifconfig and looking for multicast
+
+ if it is not enabled type ifconfig eth0 multicast
+
+ after that add a default route for multicast broadcasts and bind it to eth0
+
+ route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0
+
+ additionally, I disabled the firewall and I disabled SELinux (but I think the firewall was the big culprit here).
+
+- Downloadable code is NOT required for deployments, but MAY be useful
+ for the following purposes:
+
+ (a) exposing services to the Jini service browser;
+
+ (b) running procedures against services which were deployed before
+ the procedures were written;
+
+ If you have a complete classpath when running the various services
+ then jini will not seek to transfer code from the client as the code
+ will be resolved locally by the service.
+
+ In order to support downloadable code you need to have an HTTP
+ server that will serve class files to jini, including the interfaces
+ for all remote services. You can use any HTTP server for this
+ purpose, and the server can be located on any machine accessible
+ from the host(s) on which you are running jini. As a convenience,
+ jini bundles a simple HTTP server that you can start using a command
+ like the following:
+
+ java -jar ${JINI_HOME}/lib/classserver.jar -port 8080 -dir classes -trees -verbose&
+
+ The javadoc describes the logging and command line options for this
+ HTTP server.
+
+ https://java.sun.com/products/jini/2.1/doc/api/com/sun/jini/tool/ClassServer.html
+
+ The directory from which downloadable code will be served should
+ contain at least the bigdata jar(s) plus any remote application code
+ that you have defined (i.e., code that will run in the server
+ process).
+
+ The recommended approach to downloadable code is to extract the
+ relevant classes into a directory that will be named to the HTTP
+ server as follows. Assuming that bigdata.jar is located in the
+ current directory:
+
+ mkdir classes
+ cd classes
+ jar xfz ../bigdata.jar
+
+ If you deploy a new version of any JAR, then you SHOULD delete the
+ classes directory and redeploy all relevant JARs to make sure that
+ old class files are not left lying around.
+
+- You can enable NIO support with JERI using TCP by specifying the
+ following property to the JVM. Note that JRMP does NOT allow for
+ the possibility of NIO.
+
+ -Dcom.sun.jini.jeri.tcp.useNIO=true
+
+ More information on JERI and NIO is available using the following links.
+
+ http://archives.java.sun.com/cgi-bin/wa?A2=ind0504&L=jini-users&P=33490
+ http://archives.java.sun.com/cgi-bin/wa?A2=ind0506&L=jini-users&P=9626
+ http://archives.java.sun.com/cgi-bin/wa?A2=ind0504&L=jini-users&D=0&P=26542
+ http://java.sun.com/products/jini/2.0.1/doc/api/net/jini/jeri/tcp/package-summary.html
+
+ Note that one server thread will still be required per concurrent RPC request
+ owing to the semantics of RPC (call and wait for response) and the definition
+ of JERI.
+
+- Clients downloadable code that will be run on the bigdata services MUST set:
+
+ -Djava.rmi.server.codebase=http://.../
+
+ where "..." is your host and path
+
+ in order for the correct codebase property to be communicated to clients that
+ will then download code from that HTTP server. Note: the trailing '/' is
+ REQUIRED in your codebase or the generated URLs will NOT resolve correctly.
+
+ There is an example of how to do this with the "ant lubm-install" target and
+ the "lubmMaster.sh" script.
+
+- Debugging with jini.
+
+ See http://java.sun.com/j2se/1.4.2/docs/guide/rmi/javarmiproperties.html for
+ some guidance. Among other things, it suggests:
+
+ -Djava.rmi.server.logCalls=true
+
+ as an aid to debugging. Also try setting
+
+ -Dcom.sun.jini.reggie.proxy.debug=1
+
+ for the client, e.g., the service browser. Also see:
+
+ http://www.adtmag.com/java/articleold.aspx?id=1159
+
+ for some (very good) guidance in debugging jini services.
+
+ Note: You may have to restart jini locally in order to force download of
+ updated classes from the codebase!
+
+ See http://archives.java.sun.com/cgi-bin/wa?A2=ind0512&L=jini-users&P=R391&I=-3
+ for instructions on setting up an "download jar" (dljar) ANT task that can make
+ life much simpler (one supposes).
+
+ See http://archives.java.sun.com/cgi-bin/wa?A2=ind0311&L=jini-users&F=&S=&P=7182
+ for a description of policy files and
+ http://www.dancres.org/cottage/jini-start-examples-2_1.zip for the
+ policy files described.
+
+ See http://jan.newmarch.name/java/jini/tutorial/Ant.xml for a description of
+ one (simple) approach to using ant for jini projects (it does not use the
+ dljar ant task but explicitly enumerates what goes where).
+
+ See http://jan.newmarch.name/java/jini/tutorial/TroubleShooting.xml#RMI%20Stubs
+ for common errors when using RMI stubs.
+
+ See https://java.sun.com/products/jini/2.1/doc/api/com/sun/jini/example/browser/package-summary.html
+ for the dirty on the jini Service Browser.
Deleted: branches/BIGDATA_RELEASE_1_3_0/overview.html
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/overview.html 2014-01-23 11:33:50 UTC (rev 7819)
+++ branches/BIGDATA_RELEASE_1_3_0/overview.html 2014-01-23 11:50:35 UTC (rev 7820)
@@ -1,422 +0,0 @@
-<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
-"http://www.w3.org/TR/html4/loose.dtd">
-<html>
-<head>
-<meta http-equiv="Content-Type" content="text/html;charset=utf-8" >
-<title>bigdata®</title>
-<!-- $Id$ -->
-</head>
-<body>
-
- <p>
-
- <em>bigdata®</em> is a scale-out data and computing
- fabric designed for commodity hardware. The bigdata
- architecture provides named scale-out indices that are
- transparently and dynamically key-range partitioned and
- distributed across a cluster or grid of commodity server
- platforms. The scale-out indices are B+Trees and remain
- balanced under insert and removal operations. Keys and
- values for btrees are variable length byte[]s (the keys are
- interpreted as unsigned byte[]s).
-
- Atomic "row" operations are supported for very high
- concurrency. However, full transactions are also available
- for applications needing less concurrency but requiring
- atomic operations that read or write on more than one row,
- index partition, or index.
-
- Writes are absorbed on mutable btree instances in append
- only "journals" of ~200M capacity. On overflow, data in a
- journal is evicted onto read-optimized, immutable "index
- segments". The metadata service manages the index
- definitions, the index partitions, and the assignment of
- index partitions to data services. A data service
- encapsulates a journal and zero or more index partitions
- assigned to that data service, including the logic to handle
- overflow.
-
-</p><p>
-
- A deployment is made up of a transaction service, a load
- balancer service, and a metadata service (with failover
- redundancy) and many data service instances. bigdata can
- provide data redundancy internally (by pipelining writes
- bound for a partition across primary, secondary, ... data
- service instances for that partition) or it can be deployed
- over NAS/SAN. bigdata itself is 100% Java and requires a
- JDK 1.6. There are additional dependencies for the
- installer (un*x) and for collecting performance counters
- from the OS (<a
- href="http://pagesperso-orange.fr/sebastien.godard/">sysstat</a>).
-
- </p>
-
- <h2>Architecture</h2>
-
- <p>
-
- The bigdata SOA defines three essential services and some
- additional services. The essential services are the
- metadata service (provides a locator service for index
- partitions on data services), the data service (provides
- read, write, and concurrency control for index partitions),
- and the transaction service (provides consistent timestamps
- for commits, facilitates the release of data associated with
- older commit points, read locks, and transactions). Full
- transactions are NOT required, so you can use bigdata as a
- scale-out row store. The load balancer service guides the
- dynamic redistribution of data across a bigdata federation.
- There are also client services, which are containers for
- executing distributed jobs.
-
- </p>
-
- <p>
-
- While other service fabric architectures are contemplated,
- bigdata services today use JINI 2. to advertise themselves
- and do service discovery. This means that you must be
- running a JINI registrar in order for services to be able to
- register themselves or discover other services. The JINI
- integration is bundled and installed automatically by
- default.
-
- </p>
-
- <p>
-
- Zookeeper handles master election, configuration management
- and global synchronous locks. Zookeeper was developed by
- Yahoo! as a distributed lock and configuration management
- service and is now an Apache subproject (part of
- Hadoop). Among other things, it gets master election
- protocols right. Zookeeper is bundled and installed
- automatically by default.
-
- </p>
-
- <p>
-
- The main building blocks for the bigdata architecture are
- the journal (both an append-only persistence store and a recently
- introduced read/write store with the ability to recycle historical
- commit points), the mutable
- B+Tree (used to absorb writes), and the read-optimized
- immutable B+Tree (aka the index segment). Highly efficient
- bulk index builds are used to transfer data absorbed by a
- mutable B+Tree on a journal into index segment files. Each
- for index segment contains data for a single partition of a
- scale-out index. In order to read from an index partition,
- a consistent view is created by dynamically fusing data for
- that index partition, including any recent writes on the
- current journal, any historical writes that are in the
- process of being transferred onto index segments, and any
- historical index segments that also contain data for that
- view. Periodically, index segments are merged together, at
- which point deleted tuples are purged from the view.
-
-</p>
-
-<p>
-
- Bigdata periodically releases data associated with older
- commit points, freeing up disk resources. The transaction
- service is configured with a minimum release age in
- milliseconds. This can be ZERO (0L) milliseconds, in which
- case historical views may be released if there are no read
- locks for that commit point. The minimum release age can
- also be hours or days if you want to keep historical states
- around for a while. When a data service overflows, it will
- consult the transaction service to determine the effective
- release time and release any old journals or index segments
- no longer required to maintain views GT that release time.
-</p><p>
- An immortal or temporal database can be realized by
- specifying Long#MAX_VALUE for the minimum release age. In
- this case, the old journals and index segments will all be
- retained and you can query any historical commit point of
- the database at any time.
-
- </p><p>
-
- An detailed architecture whitepaper for bigdata is posted only and
- linked from our <a href="http://www.bigdata.com/blog">blog</a>.
-
- </p>
-
- <h2>Sparse row store</h2>
-
- <p>
-
- The <em>SparseRowStore</em> provides a column flexible row
- store similar to Google's bigtable or HBase, including very
- high read/write concurrency and ACID operations on logical
- "rows". Internally, a "global row store" instance is used
- to maintain metadata on relations declared within a bigdata
- federation. You can use this instance to store your own
- data, or you can create your own named row store instances.
- However, there is no REST api for the row store at this time
- (trivial to be sure, but not something that we have gotten
- around to yet).
-
-</p><p>
-
- In fact, it is trivial to realize bigtable semantics with
- bigdata - all you need to do is exercise a specific protocol
- when forming the keys for your scale-out indices and then
- you simply choose to NOT use transactions. A bigtable style
- key-value is formed as:
-
- </p>
-
- <pre>
-
- [columnFamily][primaryKey][columnName][timestamp} : [value]
-
- </pre>
-
- <p>
-
- By placing the column family identifier up front, all data
- in the same column family will be clustered together by the
- index. The next component is the "row" identifier, what you
- would think of as the primary key in a relational table.
- The column name comes next - only column names for non-null
- columns are written into the index. Finally, there is a
- timestamp column that is used either to record a timestamp
- specified by the application or a datum write time. The
- value associated with the key is simply the datum for that
- column in that row. The use of nul byte separators makes it
- possible to parse the key, which is required for various
- operations including index partition splits and filtering
- key scans based on column names or timestamps. See the
- <em>KeyBuilder</em> class in
- <code>com.bigdata.btree.keys</code> for utilities that may
- be used to construct keys for variety of data types.
-
- </p>
-
- <h2>Map/reduce, Asynchronous Write API, and Query</h2>
-
- <p>
-
- Google's map/reduce architecture has received a lot of
- attention, along with its bigtable architecture. Map/reduce
- provides a means to transparently decompose processing
- across a cluster. The "map" process examines a series of
- key-value pairs, emitting a set of intermediate key-value
- pairs for each input. Those intermediate key-values are
- then hashed (module R) onto R reduce processes. The inputs
- for the reduce processes are pre-sorted. The reduce process
- then runs some arbitrary operation on the sorted data, such
- as computing an inverted index file or loading the data into
- a scale-out index.
-
- </p>
-
- <p>
-
- bigdata® supports an <em>asynchronous index write
- API</em>, which delivers extremely high throughput for
- scattered writes. While map/reduce is tremendously
- effective when there is good locality in the data, it is not
- the right tool for processing ordered data. Instead, you
- execute a master job, which spawns client(s) running in
- <em>client service</em>(s) associated with the bigdata
- federation. Those clients process data, writing onto
- blocking buffers. The writes are automatically split and
- buffered for each key-range shard. This maximizes the chunk
- size for ordered writes and provides a tremendous throughput
- boost. Bigdata can work well in combination with
- map/reduce. The basic paradigm is you use map/reduce jobs
- to generate data, which is then bulk loaded into a bigdata
- federation using bigdata jobs and the asynchronous write
- API.
-
- </p><p>
-
- bigdata® has built in support for distributed rule
- execution. This can be used for high-level query or for
- materializing derived views, including maintaining the RDFS+
- closure of a semantic web database. The implementation is
- highly efficient and propagates binding sets to the data
- service for each key-range shard touched by the query, so
- the JOINs happen right up against the data. Unlike using
- map/reduce for join processing, bigdata query processing is
- very low latency. Distributed query execution can be
- substantially faster than local query execution, even for
- low-latency queries.
-
- </p>
-
- <h2>Standalone Journals</h2>
-
- <p>
-
- While bigdata® is targeted at scale-out federations,
- it can also be deployed as simple persistence store using
- just the <code>com.bigdata.journal.Journal</code> API.
-
- </p><p>
-
- The read-write (RWStore) version of the journal can scale up to 50 billion
- triples or quads and is
- capable of reclaiming storage by releasing historical commit points, aging
- them out of the backing file in a manner very similar to how the scale-out
- database releases historical commit points. The read/write store is good
- for standalone database instances, especially when the data have a lot of
- skew and when the new data are arriving continually while older data should
- be periodically released (for example, in a monitoring application where
- historical events may be released after 30 days).
-
- </p><p>
-
- The read/write store is
- also used in the scale-out architecture for the transaction manager and the
- aggregated performance counters. However, the data services use a WORM
- store to buffer writes, asynchronously migrate the buffered writes onto
- read-optimized B+Tree files using index segments builds and compacting.
- One an index partition (aka shard) reaches ~ 200MB on the disk (dynamic
- sharding). Index partitions are moved from time to time to load balance
- the cluster.
-
- </p>
-
- <h2>Status</h2>
-
- <p>
-
- bigdata® is a petabyte scale database architecture. It has been
- tested on clusters of up to 16 nodes. We have loaded data sets of 10B+
- rows, at rates of over 300,000 rows per second. Overall, 100s of billions
- of rows have been put down safely on disk.
-
- </p>
-
- <ul>
-
- <li>Hadoop integration points, including a REST API for the sparse row
- store and scanners for HDFS files making easier to target bigdata
- distributed jobs from Hadoop map/reduce jobs.</li>
-
- <li>Online backup and point in time recovery.</li>
-
- <li>Full distributed read/write transaction support (read-only transaction
- support is done, but we still have some work to do on the commit protocol
- for read-write transactions).</li>
-
- <li>OODBMS. We will be introducing an OODBM layer based on the
- Generic Object Model shortly. This will be layered over the RDF
- database and will have bindings and clients for Java, JSON, and
- other client environments.</li>
-
- </ul>
-
- <h2>Getting Started</h2>
-
- <p>
-
- See the wiki for <a
- href="http://bigdata.wiki.sourceforge.net/GettingStarted">Getting
- Started</a> and our <a
- href="http://www.bigdata.com/bigdata/blog/">blog</a> for
- what's new. The javadoc is <a
- href="http://www.bigdata.com/bigdata/docs/api/">online</a>
- and you can also build it with the ant script. If you have a
- question, please post it on the blog or the forum.
-
- </p>
-
-<h2>Getting Involved</h2>
-
-<p>
-
- bigdata® is an open source project. Contributors and
- contributions are welcome. Like most open source project,
- contributions must be submitted under a contributor
- agreement, which must be signed by someone with the
- appropriate authority. This is necessary to ensure that the
- code base remains open.
-
- </p><p>
-
- If you want to help out, please check out what is going on
- our <a href="http://www.bigdata.com/bigdata/blog/">blog</a>
- and on the <a
- href="https://sourceforge.net/projects/bigdata/">main project
- site</a>. Post your questions and we will help you figure
- out where you can contribute or how to create that new
- feature that you need.
-
- </p>
-
- <h2>Licenses and Services</h2>
-
- <p>
-
- bigdata® is distributed under GPL(v2). SYSTAP, LLC
- offers commercial licenses for customers who either want the
- value add (warranty, technical support, additional
- regression testing), who want to redistribute bigdata with
- their own commercial products, or who are not "comfortable"
- with the GPL license. For inquiries or further information,
- please write <a
- href="mailto:lic...@bi...">lic...@bi...</a>.
-
- </p><p>
-
- Please let us know if you need specific feature development
- or help in applying bigdata® to your problem. We are
- especially interested in working directly with people who
- are trying to handle massive data, especially for the
- semantic web. Please <a
- href="http://www.systap.com/contact.htm">contact us</a>
- directly.
-
- </p>
-
- <h2>Related links</h2>
-
- <dl>
-
- <dt>CouchDB</dt>
- <dd>http://couchdb.org/CouchDB/CouchDBWeb.nsf/Home?OpenForm</dd>
-
- <dt>bigtable</dt>
- <dd>http://labs.google.com/papers/bigtable.html, http://www.techcrunch.com/2008/04/04/source-google-to-launch-bigtable-as-web-service/</dd>
-
- <dt>map/reduce</dt>
- <dd>http://labs.google.com/papers/mapreduce.html</dd>
-
- <dt>Hadoop</dt>
- <dd>http://lucene.apache.org/hadoop/</dd>
-
- <dt>Zookeeper</dt>
- <dd>http://hadoop.apache.org/zookeeper/</dd>
-
- <dt>Jini/River</dt>
- <dd>http://www.jini.org/wiki/Main_Page, http://incubator.apache.org/river/RIVER/index.html</dd>
-
- <dt>Pig</dt>
- <dd>http://research.yahoo.com/node/90</dd>
-
- <dt>Sawzall</dt>
- <dd>http://labs.google.com/papers/sawzall.html</dd>
-
- <dt>Boxwood</dt>
- <dd>http://research.microsoft.com/research/sv/Boxwood/</dd>
-
- <dt>Blue Cloud</dt>
- <dd>http://www.techcrunch.com/2007/11/15/ibms-blue-cloud-is-web-computng-by-another-name/</dd>
-
- <dt>SimpleDB</dt>
- <dd>http://www.techcrunch.com/2007/12/14/amazon-takes-on-oracle-and-ibm-with-simple-db-beta/</dd>
-
- <dt>mg4j</dt>
- <dd>http://mg4j.dsi.unimi.it/</dd>
-
- </dl>
-
-</body>
-</html>
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <jer...@us...> - 2014-01-23 22:38:34
|
Revision: 7825
http://bigdata.svn.sourceforge.net/bigdata/?rev=7825&view=rev
Author: jeremy_carroll
Date: 2014-01-23 22:38:27 +0000 (Thu, 23 Jan 2014)
Log Message:
-----------
Fix for trac 804: Update: Insert/Delete now sets quads mode on construct, which may use quad based equality rather than triple based equality for filtering
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/ConstructNode.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUpdate.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/ASTConstructIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/NamedGraphUpdateTest.java
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataQuadWrapper.java
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataQuadWrapper.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataQuadWrapper.java (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataQuadWrapper.java 2014-01-23 22:38:27 UTC (rev 7825)
@@ -0,0 +1,75 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2014. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created onJan 24, 2014
+ */
+
+package com.bigdata.rdf.model;
+
+import com.bigdata.rdf.internal.IV;
+
+/**
+ * This class wraps a {@link BigdataStatement}
+ * and provides {@link #hashCode()} and {@link #equals(Object)}
+ * respecting all four fields rather than SPO as per the {@link org.openrdf.model.Statement} contract
+ * @author jeremycarroll
+ *
+ */
+public class BigdataQuadWrapper {
+
+ private final BigdataStatement delegate;
+ public BigdataQuadWrapper(BigdataStatement cspo) {
+ delegate = cspo;
+ }
+ @Override
+ public int hashCode() {
+ if (hash == 0) {
+
+ if ( delegate.getContext() == null ) {
+ hash = delegate.hashCode();
+ } else {
+ hash = delegate.getContext().hashCode() + 31 * delegate.hashCode();
+ }
+ }
+
+ return hash;
+ }
+ private int hash = 0;
+ @Override
+ public boolean equals(Object o) {
+ if (! (o instanceof BigdataQuadWrapper)) {
+ return false;
+ }
+ final BigdataStatement oo = ((BigdataQuadWrapper)o).delegate;
+ return delegate.equals(oo) && equals(delegate.getContext(),oo.getContext());
+ }
+
+ private boolean equals(BigdataResource a, BigdataResource b) {
+ return a == b || (a != null && a.equals(b));
+ }
+ public BigdataStatement statement() {
+ return delegate;
+ }
+
+}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/ConstructNode.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/ConstructNode.java 2014-01-23 15:06:01 UTC (rev 7824)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/ConstructNode.java 2014-01-23 22:38:27 UTC (rev 7825)
@@ -52,7 +52,7 @@
public interface Annotations extends AbstractStatementContainer.Annotations {
/**
- * Boolean property (default {@value #DEFAULT_ALL_GRAPHS}) which is
+ * Boolean property (default {@value #DEFAULT_NATIVE_DISTINCT}) which is
* <code>true</code> iff a native DISTINCT {@link ISPO} filter should be
* applied (large cardinality is expected for the constructed graph).
* When <code>false</code>, a JVM based DISTINCT {@link ISPO} filter
@@ -66,7 +66,23 @@
*/
String NATIVE_DISTINCT = "nativeDistinct";
+ /**
+ * Internal boolean property (default {@value #DEFAULT_DISTINCT_QUDS}) which is
+ * <code>true</code> when used internally for constructing sets of quads rather
+ * than sets of triples.
+ * <p>
+ * Note: This can only be set programmatically using {@link ConstructNode#setDistinctQuads(boolean)}
+ * <p>
+ * Note: When this property is set and mixed quads are detected or suspected then the {@link #NATIVE_DISTINCT} property is ignored.
+ *
+ * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/579">
+ * CONSTRUCT should apply DISTINCT (s,p,o) filter </a>
+ */
+ String DISTINCT_QUADS = "quadsDistinct";
+
boolean DEFAULT_NATIVE_DISTINCT = false;
+
+ boolean DEFAULT_DISTINCT_QUADS = false;
}
@@ -143,6 +159,17 @@
}
+ public boolean isDistinctQuads() {
+
+ return getProperty(Annotations.DISTINCT_QUADS,
+ Annotations.DEFAULT_DISTINCT_QUADS);
+
+ }
+ public void setDistinctQuads(final boolean quads) {
+
+ setProperty(Annotations.DISTINCT_QUADS, quads);
+ }
+
@Override
public String toString(final int indent) {
@@ -167,4 +194,5 @@
}
+
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUpdate.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUpdate.java 2014-01-23 15:06:01 UTC (rev 7824)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUpdate.java 2014-01-23 22:38:27 UTC (rev 7825)
@@ -743,6 +743,8 @@
final ConstructNode template = op.getDeleteClause()
.getQuadData().flatten(new ConstructNode(context));
+
+ template.setDistinctQuads(true);
final ASTConstructIterator itr = new ASTConstructIterator(
context.conn.getTripleStore(), template,
@@ -813,6 +815,8 @@
final ConstructNode template = op.getInsertClause()
.getQuadData().flatten(new ConstructNode(context));
+ template.setDistinctQuads(true);
+
final ASTConstructIterator itr = new ASTConstructIterator(
context.conn.getTripleStore(), template,
op.getWhereClause(), null/* bnodesMap */,
@@ -923,6 +927,8 @@
final ConstructNode template = quadData
.flatten(new ConstructNode(context));
+ template.setDistinctQuads(true);
+
// Set the CONSTRUCT template (quads patterns).
queryRoot.setConstruct(template);
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/ASTConstructIterator.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/ASTConstructIterator.java 2014-01-23 15:06:01 UTC (rev 7824)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/ASTConstructIterator.java 2014-01-23 22:38:27 UTC (rev 7825)
@@ -54,6 +54,7 @@
import com.bigdata.bop.rdf.filter.NativeDistinctFilter;
import com.bigdata.rdf.internal.IV;
import com.bigdata.rdf.model.BigdataBNode;
+import com.bigdata.rdf.model.BigdataQuadWrapper;
import com.bigdata.rdf.model.BigdataStatement;
import com.bigdata.rdf.model.BigdataValue;
import com.bigdata.rdf.model.BigdataValueFactory;
@@ -278,7 +279,13 @@
this.src = src;
- /*
+ filter = createDistinctFilter(tripleStore, construct, whereClause);
+
+ }
+
+ IFilterTest createDistinctFilter(final AbstractTripleStore tripleStore, final ConstructNode construct,
+ final GraphPatternGroup<?> whereClause) {
+ /*
* Setup the DISTINCT SPO filter.
*
* Note: CONSTRUCT is sometimes used to materialize all triples for some
@@ -289,79 +296,124 @@
* graph is large.
*/
- final boolean isObviouslyDistinct = isObviouslyDistinct(
- tripleStore.isQuads(), templates, whereClause);
-
- if (isObviouslyDistinct) {
+ final boolean isObviouslyDistinct = isObviouslyDistinct(tripleStore.isQuads(), templates, whereClause);
+ if (isObviouslyDistinct) {
+
// Do not impose a DISTINCT filter.
- filter = null;
+ return null;
- } else {
-
- final boolean nativeDistinct = construct.isNativeDistinct();
-
- if (nativeDistinct) {
+ }
+
- /*
- * Construct a predicate for the first triple template. We will
- * use that as the bias for the scalable DISTINCT SPO filter.
- */
- @SuppressWarnings("rawtypes")
- final IPredicate pred;
- {
+
+ final boolean distinctQuads = construct.isDistinctQuads() && tripleStore.isQuads() && hasMixedQuadData(templates);
+
+ if (distinctQuads) {
+
+ return createDistinctQuadsFilter(construct);
+ }
+
+ return createDistinctTriplesFilter(construct);
+
+ }
- final StatementPatternNode sp = templates.get(0/* index */);
+ @SuppressWarnings("serial")
+ private IFilterTest createDistinctQuadsFilter(final ConstructNode construct) {
+ return new DistinctFilter.DistinctFilterImpl(construct){
+ @Override
+ public boolean isValid(Object o){
+ return super.isValid(new BigdataQuadWrapper((BigdataStatement)o));
+ }
+ };
+ }
- @SuppressWarnings("rawtypes")
- final IVariableOrConstant<IV> s = sp.s()
- .getValueExpression();
+ private boolean hasMixedQuadData(List<StatementPatternNode> templates) {
+ if (templates.size() == 0) {
+ return false;
+ }
+ TermNode singleValue = templates.get(0).c();
+ if (singleValue instanceof VarNode) {
+ return true;
+ }
+ for (StatementPatternNode spn:templates) {
+ TermNode tn = spn.c();
+ if (!equals(singleValue ,tn )) {
+ // this is a little too strong, but it is merely inefficient not incorrect
+ // to return true when false would be correct.
+ return true;
+ }
+ }
+ return false;
+ }
- @SuppressWarnings("rawtypes")
- final IVariableOrConstant<IV> p = sp.p()
- .getValueExpression();
-
- @SuppressWarnings("rawtypes")
- final IVariableOrConstant<IV> o = sp.o()
- .getValueExpression();
+ private boolean equals(TermNode a, TermNode b) {
+ return a == b || ( a != null && a.equals(b));
+ }
- // // The graph term/variable iff specified by the query.
- // final TermNode cvar = sp.c();
- // final IVariableOrConstant<IV> c = cvar == null ? null :
- // cvar
- // .getValueExpression();
+ IFilterTest createDistinctTriplesFilter(final ConstructNode construct) {
+
+ final boolean nativeDistinct = construct.isNativeDistinct();
- final BOp[] vars = new BOp[] { s, p, o /* , c */};
+ if (nativeDistinct) {
- pred = new SPOPredicate(vars, BOp.NOANNS);
+ /*
+ * Construct a predicate for the first triple template. We will
+ * use that as the bias for the scalable DISTINCT SPO filter.
+ */
+ @SuppressWarnings("rawtypes")
+ final IPredicate pred;
+ {
- }
+ final StatementPatternNode sp = templates.get(0/* index */);
- /*
- * The index that will be used to read on the B+Tree access
- * path.
- */
- @SuppressWarnings({ "unchecked", "rawtypes" })
- final SPOKeyOrder indexKeyOrder = SPOKeyOrder.getKeyOrder(
- (IPredicate) pred, 3/* keyArity */);
+ @SuppressWarnings("rawtypes")
+ final IVariableOrConstant<IV> s = sp.s()
+ .getValueExpression();
- construct.setProperty(
- NativeDistinctFilter.Annotations.KEY_ORDER,
- indexKeyOrder);
+ @SuppressWarnings("rawtypes")
+ final IVariableOrConstant<IV> p = sp.p()
+ .getValueExpression();
+
+ @SuppressWarnings("rawtypes")
+ final IVariableOrConstant<IV> o = sp.o()
+ .getValueExpression();
- // Native memory based DISTINCT filter.
- filter = new NativeDistinctFilter.DistinctFilterImpl(construct);
+ // // The graph term/variable iff specified by the query.
+ // final TermNode cvar = sp.c();
+ // final IVariableOrConstant<IV> c = cvar == null ? null :
+ // cvar
+ // .getValueExpression();
- } else {
+ final BOp[] vars = new BOp[] { s, p, o /* , c */};
- // JVM Based DISTINCT filter.
- filter = new DistinctFilter.DistinctFilterImpl(construct);
+ pred = new SPOPredicate(vars, BOp.NOANNS);
}
+
+ /*
+ * The index that will be used to read on the B+Tree access
+ * path.
+ */
+ @SuppressWarnings({ "unchecked", "rawtypes" })
+ final SPOKeyOrder indexKeyOrder = SPOKeyOrder.getKeyOrder(
+ (IPredicate) pred, 3/* keyArity */);
+
+ construct.setProperty(
+ NativeDistinctFilter.Annotations.KEY_ORDER,
+ indexKeyOrder);
+
+ // Native memory based DISTINCT filter.
+ return new NativeDistinctFilter.DistinctFilterImpl(construct);
+
+ } else {
+
+ // JVM Based DISTINCT filter.
+ return new DistinctFilter.DistinctFilterImpl(construct);
+
}
-
- }
+ }
/**
* Return <code>true</code> iff this CONSTRUCT template will obviously
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/NamedGraphUpdateTest.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/NamedGraphUpdateTest.java 2014-01-23 15:06:01 UTC (rev 7824)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/NamedGraphUpdateTest.java 2014-01-23 22:38:27 UTC (rev 7825)
@@ -107,7 +107,8 @@
assertTrue(result.contains(expected));
}
- private void updateAFewTimes(boolean useHint, int numberOfTimes, int numberOfUpdatesPerTime) throws IOException {
+ private void updateAFewTimes(boolean useHint, int numberOfUpdatesPerTime) throws IOException {
+ final int numberOfTimes = 5;
for (int i=0; i<numberOfTimes; i++) {
for (int j=0; j<numberOfUpdatesPerTime;j++) {
makeUpdate(insertData);
@@ -117,29 +118,29 @@
}
}
- public void test_t_20_1() throws IOException {
- updateAFewTimes(true, 20, 1);
+ public void test_t_1() throws IOException {
+ updateAFewTimes(true, 1);
}
- public void test_t_20_2() throws IOException {
- updateAFewTimes(true, 20, 2);
+ public void test_t_2() throws IOException {
+ updateAFewTimes(true, 2);
}
- public void test_t_20_3() throws IOException {
- updateAFewTimes(true, 20, 3);
+ public void test_t_3() throws IOException {
+ updateAFewTimes(true, 3);
}
- public void test_t_20_5() throws IOException {
- updateAFewTimes(true, 20, 5);
+ public void test_t_5() throws IOException {
+ updateAFewTimes(true, 5);
}
- public void test_f_20_1() throws IOException {
- updateAFewTimes(false, 20, 1);
+ public void test_f_1() throws IOException {
+ updateAFewTimes(false, 1);
}
- public void test_f_20_2() throws IOException {
- updateAFewTimes(false, 20, 2);
+ public void test_f_2() throws IOException {
+ updateAFewTimes(false, 2);
}
- public void test_f_20_3() throws IOException {
- updateAFewTimes(false, 20, 3);
+ public void test_f_3() throws IOException {
+ updateAFewTimes(false, 3);
}
- public void test_f_20_5() throws IOException {
- updateAFewTimes(false, 20, 5);
+ public void test_f_5() throws IOException {
+ updateAFewTimes(false, 5);
}
public void test_double_triple_delete() throws IOException {
setMethodisPostUrlEncodedData();
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <jer...@us...> - 2014-01-24 23:39:46
|
Revision: 7828
http://bigdata.svn.sourceforge.net/bigdata/?rev=7828&view=rev
Author: jeremy_carroll
Date: 2014-01-24 23:39:36 +0000 (Fri, 24 Jan 2014)
Log Message:
-----------
Tests and preparation for fixing of trac 807 concerning native distinct filter of insert/delete quads
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/QueryBase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUpdate.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/ASTConstructIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTConstructOptimizer.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/TestNanoSparqlServerWithProxyIndexManager.java
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/AbstractNamedGraphUpdateTest.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/HashDistinctNamedGraphUpdateTest.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/NativeDistinctNamedGraphUpdateTest.java
Removed Paths:
-------------
branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/NamedGraphUpdateTest.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/QueryBase.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/QueryBase.java 2014-01-24 18:42:02 UTC (rev 7827)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/QueryBase.java 2014-01-24 23:39:36 UTC (rev 7828)
@@ -178,6 +178,7 @@
public void setConstruct(final ConstructNode construct) {
setProperty(Annotations.CONSTRUCT, construct);
+ setQueryType(QueryType.CONSTRUCT);
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUpdate.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUpdate.java 2014-01-24 18:42:02 UTC (rev 7827)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUpdate.java 2014-01-24 23:39:36 UTC (rev 7828)
@@ -595,10 +595,18 @@
* associated with the SailConnection in case the view is
* isolated by a transaction.
*/
+
+ // If the query contains a nativeDistinctSPO query hint then
+ // the line below unfortunately isolates the query so that the hint does
+ // not impact any other execution, this is hacked by putting a property on the query root.
+
final MutableTupleQueryResult result = new MutableTupleQueryResult(
ASTEvalHelper.evaluateTupleQuery(
context.conn.getTripleStore(), astContainer,
null/* bindingSets */));
+
+ boolean nativeDistinct = astContainer.getOptimizedAST().getProperty(ConstructNode.Annotations.NATIVE_DISTINCT,
+ ConstructNode.Annotations.DEFAULT_NATIVE_DISTINCT);
try {
@@ -745,6 +753,10 @@
.getQuadData().flatten(new ConstructNode(context));
template.setDistinctQuads(true);
+
+ if (nativeDistinct) {
+ template.setNativeDistinct(true);
+ }
final ASTConstructIterator itr = new ASTConstructIterator(
context.conn.getTripleStore(), template,
@@ -816,6 +828,10 @@
.getQuadData().flatten(new ConstructNode(context));
template.setDistinctQuads(true);
+
+ if (nativeDistinct) {
+ template.setNativeDistinct(true);
+ }
final ASTConstructIterator itr = new ASTConstructIterator(
context.conn.getTripleStore(), template,
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/ASTConstructIterator.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/ASTConstructIterator.java 2014-01-24 18:42:02 UTC (rev 7827)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/ASTConstructIterator.java 2014-01-24 23:39:36 UTC (rev 7828)
@@ -188,13 +188,15 @@
private boolean open = true;
/**
- * <code>true</code> until we get the first solution for the WHERE clause.
+ * <code>false</code> until we get the first solution for the WHERE clause.
* We do not output ground triples in the template until we see that first
* solution. It is Ok if it is empty, but we need to know that the WHERE
* clause succeeded before we can output the ground triples.
*/
- private boolean haveFirstSolution = false;
+ private boolean haveFirstSolution = false;
+ public static boolean flagToCheckNativeDistinctQuadsInvocationForJUnitTesting = false;
+
/**
*
* @param tripleStore
@@ -283,7 +285,7 @@
}
- IFilterTest createDistinctFilter(final AbstractTripleStore tripleStore, final ConstructNode construct,
+ private IFilterTest createDistinctFilter(final AbstractTripleStore tripleStore, final ConstructNode construct,
final GraphPatternGroup<?> whereClause) {
/*
* Setup the DISTINCT SPO filter.
@@ -296,6 +298,13 @@
* graph is large.
*/
+ final boolean distinctQuads = construct.isDistinctQuads() && tripleStore.isQuads() && hasMixedQuadData(templates);
+ final boolean nativeDistinct = construct.isNativeDistinct();
+
+ if (nativeDistinct && construct.isDistinctQuads()) {
+ flagToCheckNativeDistinctQuadsInvocationForJUnitTesting = true;
+ }
+
final boolean isObviouslyDistinct = isObviouslyDistinct(tripleStore.isQuads(), templates, whereClause);
if (isObviouslyDistinct) {
@@ -307,20 +316,33 @@
}
-
- final boolean distinctQuads = construct.isDistinctQuads() && tripleStore.isQuads() && hasMixedQuadData(templates);
-
if (distinctQuads) {
- return createDistinctQuadsFilter(construct);
+ if (nativeDistinct) {
+ return createNativeDistinctQuadsFilter(construct);
+ } else {
+ return createHashDistinctQuadsFilter(construct);
+ }
+
+ } else {
+
+ if (nativeDistinct) {
+ return createNativeDistinctTripleFilter(construct);
+ } else {
+ // JVM Based DISTINCT filter.
+ return new DistinctFilter.DistinctFilterImpl(construct);
+ }
}
-
- return createDistinctTriplesFilter(construct);
}
+ private IFilterTest createNativeDistinctQuadsFilter(ConstructNode construct) {
+ // TODO This method needs to be written. It should scale, whereas the current implementation does not.
+ return createHashDistinctQuadsFilter(construct);
+ }
+
@SuppressWarnings("serial")
- private IFilterTest createDistinctQuadsFilter(final ConstructNode construct) {
+ private IFilterTest createHashDistinctQuadsFilter(final ConstructNode construct) {
return new DistinctFilter.DistinctFilterImpl(construct){
@Override
public boolean isValid(Object o){
@@ -352,67 +374,55 @@
return a == b || ( a != null && a.equals(b));
}
- IFilterTest createDistinctTriplesFilter(final ConstructNode construct) {
-
- final boolean nativeDistinct = construct.isNativeDistinct();
+ private IFilterTest createNativeDistinctTripleFilter(final ConstructNode construct) {
+ /*
+ * Construct a predicate for the first triple template. We will
+ * use that as the bias for the scalable DISTINCT SPO filter.
+ */
+ @SuppressWarnings("rawtypes")
+ final IPredicate pred;
+ {
- if (nativeDistinct) {
+ final StatementPatternNode sp = templates.get(0/* index */);
- /*
- * Construct a predicate for the first triple template. We will
- * use that as the bias for the scalable DISTINCT SPO filter.
- */
- @SuppressWarnings("rawtypes")
- final IPredicate pred;
- {
+ @SuppressWarnings("rawtypes")
+ final IVariableOrConstant<IV> s = sp.s()
+ .getValueExpression();
- final StatementPatternNode sp = templates.get(0/* index */);
+ @SuppressWarnings("rawtypes")
+ final IVariableOrConstant<IV> p = sp.p()
+ .getValueExpression();
+
+ @SuppressWarnings("rawtypes")
+ final IVariableOrConstant<IV> o = sp.o()
+ .getValueExpression();
- @SuppressWarnings("rawtypes")
- final IVariableOrConstant<IV> s = sp.s()
- .getValueExpression();
+ // // The graph term/variable iff specified by the query.
+ // final TermNode cvar = sp.c();
+ // final IVariableOrConstant<IV> c = cvar == null ? null :
+ // cvar
+ // .getValueExpression();
- @SuppressWarnings("rawtypes")
- final IVariableOrConstant<IV> p = sp.p()
- .getValueExpression();
-
- @SuppressWarnings("rawtypes")
- final IVariableOrConstant<IV> o = sp.o()
- .getValueExpression();
+ final BOp[] vars = new BOp[] { s, p, o /* , c */};
- // // The graph term/variable iff specified by the query.
- // final TermNode cvar = sp.c();
- // final IVariableOrConstant<IV> c = cvar == null ? null :
- // cvar
- // .getValueExpression();
+ pred = new SPOPredicate(vars, BOp.NOANNS);
- final BOp[] vars = new BOp[] { s, p, o /* , c */};
+ }
- pred = new SPOPredicate(vars, BOp.NOANNS);
+ /*
+ * The index that will be used to read on the B+Tree access
+ * path.
+ */
+ @SuppressWarnings({ "unchecked", "rawtypes" })
+ final SPOKeyOrder indexKeyOrder = SPOKeyOrder.getKeyOrder(
+ (IPredicate) pred, 3/* keyArity */);
- }
+ construct.setProperty(
+ NativeDistinctFilter.Annotations.KEY_ORDER,
+ indexKeyOrder);
- /*
- * The index that will be used to read on the B+Tree access
- * path.
- */
- @SuppressWarnings({ "unchecked", "rawtypes" })
- final SPOKeyOrder indexKeyOrder = SPOKeyOrder.getKeyOrder(
- (IPredicate) pred, 3/* keyArity */);
-
- construct.setProperty(
- NativeDistinctFilter.Annotations.KEY_ORDER,
- indexKeyOrder);
-
- // Native memory based DISTINCT filter.
- return new NativeDistinctFilter.DistinctFilterImpl(construct);
-
- } else {
-
- // JVM Based DISTINCT filter.
- return new DistinctFilter.DistinctFilterImpl(construct);
-
- }
+ // Native memory based DISTINCT filter.
+ return new NativeDistinctFilter.DistinctFilterImpl(construct);
}
/**
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTConstructOptimizer.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTConstructOptimizer.java 2014-01-24 18:42:02 UTC (rev 7827)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTConstructOptimizer.java 2014-01-24 23:39:36 UTC (rev 7828)
@@ -42,6 +42,7 @@
import com.bigdata.rdf.sparql.ast.SliceNode;
import com.bigdata.rdf.sparql.ast.SubqueryRoot;
import com.bigdata.rdf.sparql.ast.VarNode;
+import com.bigdata.rdf.sparql.ast.ConstructNode.Annotations;
import com.bigdata.rdf.sparql.ast.eval.AST2BOpContext;
/**
@@ -73,6 +74,9 @@
case CONSTRUCT:
break;
default:
+ if (context.nativeDistinctSPO) {
+ queryRoot.setProperty(Annotations.NATIVE_DISTINCT, true);
+ }
return queryRoot;
}
Copied: branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/AbstractNamedGraphUpdateTest.java (from rev 7827, branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/NamedGraphUpdateTest.java)
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/AbstractNamedGraphUpdateTest.java (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/AbstractNamedGraphUpdateTest.java 2014-01-24 23:39:36 UTC (rev 7828)
@@ -0,0 +1,296 @@
+/**
+Copyright (C) SYSTAP, LLC 2014. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+
+package com.bigdata.rdf.sail.webapp;
+
+import java.io.IOException;
+
+import com.bigdata.rdf.sparql.ast.eval.ASTConstructIterator;
+
+
+
+/**
+ * This class is concernign the issues raised in trac 804
+ * https://sourceforge.net/apps/trac/bigdata/ticket/804
+ * @author jeremycarroll
+ *
+ */
+public class AbstractNamedGraphUpdateTest extends AbstractProtocolTest {
+
+ private static String distinctHintFalse = " hint:Query hint:nativeDistinctSPO false . \n";
+ private static String distinctHintTrue = " hint:Query hint:nativeDistinctSPO true . \n";
+ private final boolean nativeDistinct;
+
+
+
+ public AbstractNamedGraphUpdateTest(boolean nativeDistinct, String name) {
+ super(name);
+ this.nativeDistinct = nativeDistinct;
+ }
+
+
+ private String atomicMoveNamedGraph() {
+ // Atomic update of uploaded graph - moving eg:tmp to eg:a (deleting old contents of eg:a)
+ return
+ "DELETE {\n" +
+ " GRAPH <eg:a> {\n" +
+ " ?olds ?oldp ?oldo\n" +
+ " }\n" +
+ " GRAPH <eg:tmp> {\n" +
+ " ?news ?newp ?newo\n" +
+ " }\n" +
+ "}\n" +
+ "INSERT {\n" +
+ " GRAPH <eg:a> {\n" +
+ " ?news ?newp ?newo\n" +
+ " }\n" +
+ "}\n" +
+ "WHERE {\n" +
+ distinctHintFalse +
+ " {\n" +
+ " GRAPH <eg:a> {\n" +
+ " ?olds ?oldp ?oldo\n" +
+ " }\n" +
+ " } UNION {\n" +
+ " GRAPH <eg:tmp> {\n" +
+ " ?news ?newp ?newo\n" +
+ " }\n" +
+ " }\n" +
+ "}";
+ }
+
+ private String insertData =
+ "prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n" +
+ "INSERT DATA\n" +
+ "{ \n" +
+ " GRAPH <eg:a> {\n" +
+ " [ a \"Blank\" ] .\n" +
+ " <eg:b> rdf:type <eg:c> ; rdf:value [] .\n" +
+ " [ rdf:value [] ]\n" +
+ " }\n" +
+ " GRAPH <eg:tmp> {\n" +
+ " [ a \"Blankx\" ] .\n" +
+ " <eg:B> rdf:type <eg:C> ; rdf:value [] .\n" +
+ " [ rdf:value [] ]\n" +
+ " }\n" +
+ "}\n";
+
+
+ private void makeUpdate(String update) throws IOException {
+ boolean hasHint = update.contains(distinctHintFalse);
+ if (hasHint) {
+ ASTConstructIterator.flagToCheckNativeDistinctQuadsInvocationForJUnitTesting = false;
+ if (nativeDistinct) {
+ update = update.replace(distinctHintFalse, distinctHintTrue);
+ }
+ }
+ setMethodisPostUrlEncodedData();
+ serviceRequest("update", update);
+ if (hasHint) {
+ assertEquals(nativeDistinct, ASTConstructIterator.flagToCheckNativeDistinctQuadsInvocationForJUnitTesting );
+ }
+ }
+
+ private void assertQuad(String graph, String triple) throws IOException {
+ assertQuad("true", graph, triple);
+ }
+
+ private void assertNotQuad(String graph, String triple) throws IOException {
+ assertQuad("false", graph, triple);
+ }
+
+ void assertQuad(String expected, String graph, String triple) throws IOException {
+ String result = serviceRequest("query", "ASK { GRAPH " + graph + " { " + triple + "} }" );
+ assertTrue(result.contains(expected));
+ }
+
+ private void updateAFewTimes(int numberOfUpdatesPerTime) throws IOException {
+ final int numberOfTimes = 5;
+ for (int i=0; i<numberOfTimes; i++) {
+ for (int j=0; j<numberOfUpdatesPerTime;j++) {
+ makeUpdate(insertData);
+ }
+ makeUpdate( atomicMoveNamedGraph() );
+ assertNotQuad("<eg:tmp>", " ?s ?p ?o ");
+ }
+ }
+
+ public void test_t_1() throws IOException {
+ updateAFewTimes(1);
+ }
+ public void test_t_2() throws IOException {
+ updateAFewTimes(2);
+ }
+ public void test_t_3() throws IOException {
+ updateAFewTimes(3);
+ }
+ public void test_t_5() throws IOException {
+ updateAFewTimes(5);
+ }
+ public void test_double_triple_delete() throws IOException {
+ setMethodisPostUrlEncodedData();
+ makeUpdate("prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n" +
+ "INSERT DATA\n" +
+ "{ \n" +
+ " GRAPH <eg:a> {\n" +
+ " <eg:b> rdf:type <eg:c> \n" +
+ " }\n" +
+ " GRAPH <eg:tmp> {\n" +
+ " <eg:b> rdf:type <eg:c> \n" +
+ " }\n" +
+ "}\n");
+ makeUpdate( "DELETE {\n" +
+ " GRAPH <eg:a> {\n" +
+ " ?olds ?oldp ?oldo\n" +
+ " }\n" +
+ " GRAPH <eg:tmp> {\n" +
+ " ?olds ?oldp ?oldo\n" +
+ " }\n" +
+ "}\n" +
+ "WHERE {\n" +
+ distinctHintFalse +
+ " GRAPH <eg:a> {\n" +
+ " ?olds ?oldp ?oldo\n" +
+ " }\n" +
+ "}");
+ assertNotQuad("?g","?s ?p ?o");
+
+ }
+
+ public void test_double_triple_insert() throws IOException {
+ makeUpdate( "prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n" +
+ "INSERT DATA\n" +
+ "{ \n" +
+ " GRAPH <eg:tmp> {\n" +
+ " <eg:b> rdf:type <eg:c> .\n" +
+ " <eg:x> rdf:type _:foo \n" +
+ " }\n" +
+ "}\n");
+ makeUpdate( "INSERT {\n" +
+ " GRAPH <eg:A> {\n" +
+ " ?olds ?oldp ?oldo\n" +
+ " }\n" +
+ " GRAPH <eg:B> {\n" +
+ " ?olds ?oldp ?oldo\n" +
+ " }\n" +
+ "}\n" +
+ "WHERE {\n" +
+ distinctHintFalse +
+ " GRAPH <eg:tmp> {\n" +
+ " ?olds ?oldp ?oldo\n" +
+ " }\n" +
+ "}");
+ assertQuad("<eg:A>","<eg:b> rdf:type <eg:c> ");
+ assertQuad("<eg:B>","<eg:b> rdf:type <eg:c> ");
+ assertQuad("<eg:A>","<eg:x> rdf:type ?x");
+ assertQuad("<eg:B>","<eg:x> rdf:type ?x ");
+ }
+ public void test_double_triple_delete_insert() throws IOException {
+ makeUpdate( "prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n" +
+ "INSERT DATA\n" +
+ "{ \n" +
+ " GRAPH <eg:tmp> {\n" +
+ " <eg:A> <eg:moveTo> <eg:AA> .\n" +
+ " <eg:B> <eg:moveTo> <eg:BB> \n" +
+ " }\n" +
+ "}\n");
+ makeUpdate( "INSERT {\n" +
+ " GRAPH <eg:A> {\n" +
+ " ?olds ?oldp ?oldo\n" +
+ " }\n" +
+ "}\n" +
+ "WHERE {\n" +
+ " GRAPH <eg:tmp> {\n" +
+ " ?olds ?oldp ?oldo\n" +
+ " }\n" +
+ "}");
+ makeUpdate( "INSERT {\n" +
+ " GRAPH <eg:B> {\n" +
+ " ?olds ?oldp ?oldo\n" +
+ " }\n" +
+ "}\n" +
+ "WHERE {\n" +
+ " GRAPH <eg:tmp> {\n" +
+ " ?olds ?oldp ?oldo\n" +
+ " }\n" +
+ "}");
+ assertQuad("<eg:A>","<eg:A> <eg:moveTo> <eg:AA> ");
+ assertQuad("<eg:B>","<eg:A> <eg:moveTo> <eg:AA> ");
+ assertQuad("<eg:A>","<eg:B> <eg:moveTo> <eg:BB>");
+ assertQuad("<eg:B>","<eg:B> <eg:moveTo> <eg:BB> ");
+ makeUpdate(
+ "DELETE {\n" +
+ " GRAPH ?oldg {\n" +
+ " ?olds ?oldp ?oldo\n" +
+ " }\n" +
+ "}\n" +
+ "INSERT {\n" +
+ " GRAPH ?newg {\n" +
+ " ?olds ?oldp ?oldo\n" +
+ " }\n" +
+ "}\n" +
+ "WHERE {\n" +
+ distinctHintFalse +
+ " GRAPH <eg:tmp> {\n" +
+ " ?oldg <eg:moveTo> ?newg\n" +
+ " }\n" +
+ " GRAPH ?oldg {\n" +
+ " ?olds ?oldp ?oldo\n" +
+ " }\n" +
+ "}");
+ assertNotQuad("<eg:A>","<eg:A> <eg:moveTo> <eg:AA> ");
+ assertNotQuad("<eg:B>","<eg:A> <eg:moveTo> <eg:AA> ");
+ assertNotQuad("<eg:A>","<eg:B> <eg:moveTo> <eg:BB>");
+ assertNotQuad("<eg:B>","<eg:B> <eg:moveTo> <eg:BB> ");
+ assertQuad("<eg:AA>","<eg:A> <eg:moveTo> <eg:AA> ");
+ assertQuad("<eg:BB>","<eg:A> <eg:moveTo> <eg:AA> ");
+ assertQuad("<eg:AA>","<eg:B> <eg:moveTo> <eg:BB>");
+ assertQuad("<eg:BB>","<eg:B> <eg:moveTo> <eg:BB> ");
+ }
+ public void test_triple_template_and_fixed_insert() throws IOException {
+ makeUpdate( "prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n" +
+ "INSERT DATA\n" +
+ "{ \n" +
+ " GRAPH <eg:tmp> {\n" +
+ " <eg:b> rdf:type <eg:c> .\n" +
+ " }\n" +
+ "}\n");
+ makeUpdate( "INSERT {\n" +
+ " GRAPH <eg:A> {\n" +
+ " ?olds ?oldp ?oldo\n" +
+ " }\n" +
+ " GRAPH <eg:B> {\n" +
+ " <eg:b> rdf:type <eg:c> .\n" +
+ " }\n" +
+ "}\n" +
+ "WHERE {\n" +
+ distinctHintFalse +
+ " GRAPH <eg:tmp> {\n" +
+ " ?olds ?oldp ?oldo\n" +
+ " }\n" +
+ "}");
+ assertQuad("<eg:A>","<eg:b> rdf:type <eg:c> ");
+ assertQuad("<eg:B>","<eg:b> rdf:type <eg:c> ");
+ }
+
+}
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/HashDistinctNamedGraphUpdateTest.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/HashDistinctNamedGraphUpdateTest.java (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/HashDistinctNamedGraphUpdateTest.java 2014-01-24 23:39:36 UTC (rev 7828)
@@ -0,0 +1,36 @@
+/**
+Copyright (C) SYSTAP, LLC 2014. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+package com.bigdata.rdf.sail.webapp;
+
+import junit.framework.Test;
+
+public class HashDistinctNamedGraphUpdateTest extends AbstractNamedGraphUpdateTest {
+
+ static public Test suite() {
+ return ProxySuiteHelper.suiteWhenStandalone(HashDistinctNamedGraphUpdateTest.class,"test.*", TestMode.quads);
+ }
+ public HashDistinctNamedGraphUpdateTest(String name) {
+ super(false, name);
+ }
+
+}
Deleted: branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/NamedGraphUpdateTest.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/NamedGraphUpdateTest.java 2014-01-24 18:42:02 UTC (rev 7827)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/NamedGraphUpdateTest.java 2014-01-24 23:39:36 UTC (rev 7828)
@@ -1,289 +0,0 @@
-/**
-Copyright (C) SYSTAP, LLC 2014. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
-
-package com.bigdata.rdf.sail.webapp;
-
-import java.io.IOException;
-
-import junit.framework.Test;
-
-
-/**
- * This class is concernign the issues raised in trac 804
- * https://sourceforge.net/apps/trac/bigdata/ticket/804
- * @author jeremycarroll
- *
- */
-public class NamedGraphUpdateTest extends AbstractProtocolTest {
-
-
- public NamedGraphUpdateTest(String name) {
- super(name);
- }
-
-
- private String atomicMoveNamedGraph(boolean useHint) {
- // Atomic update of uploaded graph - moving eg:tmp to eg:a (deleting old contents of eg:a)
- return
- "DELETE {\n" +
- " GRAPH <eg:a> {\n" +
- " ?olds ?oldp ?oldo\n" +
- " }\n" +
- " GRAPH <eg:tmp> {\n" +
- " ?news ?newp ?newo\n" +
- " }\n" +
- "}\n" +
- "INSERT {\n" +
- " GRAPH <eg:a> {\n" +
- " ?news ?newp ?newo\n" +
- " }\n" +
- "}\n" +
- "WHERE {\n" +
- (useHint?" hint:Query hint:chunkSize 2 .\n":"") +
- " {\n" +
- " GRAPH <eg:a> {\n" +
- " ?olds ?oldp ?oldo\n" +
- " }\n" +
- " } UNION {\n" +
- " GRAPH <eg:tmp> {\n" +
- " ?news ?newp ?newo\n" +
- " }\n" +
- " }\n" +
- "}";
- }
-
- private String insertData =
- "prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n" +
- "INSERT DATA\n" +
- "{ \n" +
- " GRAPH <eg:a> {\n" +
- " [ a \"Blank\" ] .\n" +
- " <eg:b> rdf:type <eg:c> ; rdf:value [] .\n" +
- " [ rdf:value [] ]\n" +
- " }\n" +
- " GRAPH <eg:tmp> {\n" +
- " [ a \"Blankx\" ] .\n" +
- " <eg:B> rdf:type <eg:C> ; rdf:value [] .\n" +
- " [ rdf:value [] ]\n" +
- " }\n" +
- "}\n";
-
-
- private void makeUpdate(String update) throws IOException {
- setMethodisPostUrlEncodedData();
- serviceRequest("update", update);
- }
-
- private void assertQuad(String graph, String triple) throws IOException {
- assertQuad("true", graph, triple);
- }
-
- private void assertNotQuad(String graph, String triple) throws IOException {
- assertQuad("false", graph, triple);
- }
-
- void assertQuad(String expected, String graph, String triple) throws IOException {
- String result = serviceRequest("query", "ASK { GRAPH " + graph + " { " + triple + "} }" );
- assertTrue(result.contains(expected));
- }
-
- private void updateAFewTimes(boolean useHint, int numberOfUpdatesPerTime) throws IOException {
- final int numberOfTimes = 5;
- for (int i=0; i<numberOfTimes; i++) {
- for (int j=0; j<numberOfUpdatesPerTime;j++) {
- makeUpdate(insertData);
- }
- makeUpdate( atomicMoveNamedGraph(useHint) );
- assertNotQuad("<eg:tmp>", " ?s ?p ?o ");
- }
- }
-
- public void test_t_1() throws IOException {
- updateAFewTimes(true, 1);
- }
- public void test_t_2() throws IOException {
- updateAFewTimes(true, 2);
- }
- public void test_t_3() throws IOException {
- updateAFewTimes(true, 3);
- }
- public void test_t_5() throws IOException {
- updateAFewTimes(true, 5);
- }
- public void test_f_1() throws IOException {
- updateAFewTimes(false, 1);
- }
- public void test_f_2() throws IOException {
- updateAFewTimes(false, 2);
- }
- public void test_f_3() throws IOException {
- updateAFewTimes(false, 3);
- }
- public void test_f_5() throws IOException {
- updateAFewTimes(false, 5);
- }
- public void test_double_triple_delete() throws IOException {
- setMethodisPostUrlEncodedData();
- makeUpdate("prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n" +
- "INSERT DATA\n" +
- "{ \n" +
- " GRAPH <eg:a> {\n" +
- " <eg:b> rdf:type <eg:c> \n" +
- " }\n" +
- " GRAPH <eg:tmp> {\n" +
- " <eg:b> rdf:type <eg:c> \n" +
- " }\n" +
- "}\n");
- makeUpdate( "DELETE {\n" +
- " GRAPH <eg:a> {\n" +
- " ?olds ?oldp ?oldo\n" +
- " }\n" +
- " GRAPH <eg:tmp> {\n" +
- " ?olds ?oldp ?oldo\n" +
- " }\n" +
- "}\n" +
- "WHERE {\n" +
- " GRAPH <eg:a> {\n" +
- " ?olds ?oldp ?oldo\n" +
- " }\n" +
- "}");
- assertNotQuad("?g","?s ?p ?o");
-
- }
-
- public void test_double_triple_insert() throws IOException {
- makeUpdate( "prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n" +
- "INSERT DATA\n" +
- "{ \n" +
- " GRAPH <eg:tmp> {\n" +
- " <eg:b> rdf:type <eg:c> .\n" +
- " <eg:x> rdf:type _:foo \n" +
- " }\n" +
- "}\n");
- makeUpdate( "INSERT {\n" +
- " GRAPH <eg:A> {\n" +
- " ?olds ?oldp ?oldo\n" +
- " }\n" +
- " GRAPH <eg:B> {\n" +
- " ?olds ?oldp ?oldo\n" +
- " }\n" +
- "}\n" +
- "WHERE {\n" +
- " GRAPH <eg:tmp> {\n" +
- " ?olds ?oldp ?oldo\n" +
- " }\n" +
- "}");
- assertQuad("<eg:A>","<eg:b> rdf:type <eg:c> ");
- assertQuad("<eg:B>","<eg:b> rdf:type <eg:c> ");
- assertQuad("<eg:A>","<eg:x> rdf:type ?x");
- assertQuad("<eg:B>","<eg:x> rdf:type ?x ");
- }
- public void test_double_triple_delete_insert() throws IOException {
- makeUpdate( "prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n" +
- "INSERT DATA\n" +
- "{ \n" +
- " GRAPH <eg:tmp> {\n" +
- " <eg:A> <eg:moveTo> <eg:AA> .\n" +
- " <eg:B> <eg:moveTo> <eg:BB> \n" +
- " }\n" +
- "}\n");
- makeUpdate( "INSERT {\n" +
- " GRAPH <eg:A> {\n" +
- " ?olds ?oldp ?oldo\n" +
- " }\n" +
- "}\n" +
- "WHERE {\n" +
- " GRAPH <eg:tmp> {\n" +
- " ?olds ?oldp ?oldo\n" +
- " }\n" +
- "}");
- makeUpdate( "INSERT {\n" +
- " GRAPH <eg:B> {\n" +
- " ?olds ?oldp ?oldo\n" +
- " }\n" +
- "}\n" +
- "WHERE {\n" +
- " GRAPH <eg:tmp> {\n" +
- " ?olds ?oldp ?oldo\n" +
- " }\n" +
- "}");
- assertQuad("<eg:A>","<eg:A> <eg:moveTo> <eg:AA> ");
- assertQuad("<eg:B>","<eg:A> <eg:moveTo> <eg:AA> ");
- assertQuad("<eg:A>","<eg:B> <eg:moveTo> <eg:BB>");
- assertQuad("<eg:B>","<eg:B> <eg:moveTo> <eg:BB> ");
- makeUpdate( "DELETE {\n" +
- " GRAPH ?oldg {\n" +
- " ?olds ?oldp ?oldo\n" +
- " }\n" +
- "}\n" +
- "INSERT {\n" +
- " GRAPH ?newg {\n" +
- " ?olds ?oldp ?oldo\n" +
- " }\n" +
- "}\n" +
- "WHERE {\n" +
- " GRAPH <eg:tmp> {\n" +
- " ?oldg <eg:moveTo> ?newg\n" +
- " }\n" +
- " GRAPH ?oldg {\n" +
- " ?olds ?oldp ?oldo\n" +
- " }\n" +
- "}");
- assertNotQuad("<eg:A>","<eg:A> <eg:moveTo> <eg:AA> ");
- assertNotQuad("<eg:B>","<eg:A> <eg:moveTo> <eg:AA> ");
- assertNotQuad("<eg:A>","<eg:B> <eg:moveTo> <eg:BB>");
- assertNotQuad("<eg:B>","<eg:B> <eg:moveTo> <eg:BB> ");
- assertQuad("<eg:AA>","<eg:A> <eg:moveTo> <eg:AA> ");
- assertQuad("<eg:BB>","<eg:A> <eg:moveTo> <eg:AA> ");
- assertQuad("<eg:AA>","<eg:B> <eg:moveTo> <eg:BB>");
- assertQuad("<eg:BB>","<eg:B> <eg:moveTo> <eg:BB> ");
- }
- public void test_triple_template_and_fixed_insert() throws IOException {
- makeUpdate( "prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n" +
- "INSERT DATA\n" +
- "{ \n" +
- " GRAPH <eg:tmp> {\n" +
- " <eg:b> rdf:type <eg:c> .\n" +
- " }\n" +
- "}\n");
- makeUpdate( "INSERT {\n" +
- " GRAPH <eg:A> {\n" +
- " ?olds ?oldp ?oldo\n" +
- " }\n" +
- " GRAPH <eg:B> {\n" +
- " <eg:b> rdf:type <eg:c> .\n" +
- " }\n" +
- "}\n" +
- "WHERE {\n" +
- " GRAPH <eg:tmp> {\n" +
- " ?olds ?oldp ?oldo\n" +
- " }\n" +
- "}");
- assertQuad("<eg:A>","<eg:b> rdf:type <eg:c> ");
- assertQuad("<eg:B>","<eg:b> rdf:type <eg:c> ");
- }
- static public Test suite() {
- return ProxySuiteHelper.suiteWhenStandalone(NamedGraphUpdateTest.class,"test.*", TestMode.quads);
- }
-
-}
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/NativeDistinctNamedGraphUpdateTest.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/NativeDistinctNamedGraphUpdateTest.java (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/NativeDistinctNamedGraphUpdateTest.java 2014-01-24 23:39:36 UTC (rev 7828)
@@ -0,0 +1,36 @@
+/**
+Copyright (C) SYSTAP, LLC 2014. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+package com.bigdata.rdf.sail.webapp;
+
+import junit.framework.Test;
+
+public class NativeDistinctNamedGraphUpdateTest extends AbstractNamedGraphUpdateTest {
+
+ static public Test suite() {
+ return ProxySuiteHelper.suiteWhenStandalone(NativeDistinctNamedGraphUpdateTest.class,"test.*", TestMode.quads);
+ }
+ public NativeDistinctNamedGraphUpdateTest(String name) {
+ super(true, name);
+ }
+
+}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/TestNanoSparqlServerWithProxyIndexManager.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/TestNanoSparqlServerWithProxyIndexManager.java 2014-01-24 18:42:02 UTC (rev 7827)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/TestNanoSparqlServerWithProxyIndexManager.java 2014-01-24 23:39:36 UTC (rev 7828)
@@ -247,6 +247,8 @@
case quads:
// QUADS mode UPDATE test suite.
suite.addTestSuite(TestSparqlUpdate.class);
+ suite.addTestSuite( NativeDistinctNamedGraphUpdateTest.class );
+ suite.addTestSuite( HashDistinctNamedGraphUpdateTest.class );
break;
default: throw new UnsupportedOperationException();
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-27 16:09:39
|
Revision: 7830
http://bigdata.svn.sourceforge.net/bigdata/?rev=7830&view=rev
Author: thompsonbry
Date: 2014-01-27 16:09:26 +0000 (Mon, 27 Jan 2014)
Log Message:
-----------
Fix for [1]. This incorporates a fix for the stochastic behavior of the analytic query mode whose root cause was the failure of the leaf keys coder for the HTree to support duplicate keys. Committed to CI for feedback. I will also re-run the govtrack queries to establish a new baseline and assess the performance impact relative to the last released code.
[1] https://sourceforge.net/apps/trac/bigdata/ticket/763 (Stochastic results in Analytic query mode).
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/.classpath
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/IndexMetadata.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/AbstractKeyBuffer.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/AbstractRaba.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/ConditionalRabaCoder.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/EmptyRaba.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/IRaba.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/MutableKeysRaba.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/MutableValueBuffer.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/AbstractCodedRaba.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/CanonicalHuffmanRabaCoder.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/EmptyRabaValueCoder.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/FixedLengthValueRabaCoder.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/FrontCodedRabaCoder.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/IRabaCoder.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/SimpleRabaCoder.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/htree/DirectoryPage.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/htree/NodeSerializer.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/htree/raba/MutableKeyBuffer.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/htree/raba/MutableValueBuffer.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/btree/raba/codec/MutableRabaCoder.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/htree/AbstractHTreeTestCase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/htree/TestDuplicates.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/htree/TestHTreeWithMemStore.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/htree/TestIncrementalWrite.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/spo/FastRDFValueCoder2.java
branches/BIGDATA_RELEASE_1_3_0/build.properties
branches/BIGDATA_RELEASE_1_3_0/lgpl-utils/src/java/it/unimi/dsi/fastutil/bytes/custom/CustomByteArrayFrontCodedList.java
branches/BIGDATA_RELEASE_1_3_0/pom.xml
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_3_0/bigdata/lib/lgpl-utils-1.0.7-270114.jar
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/FrontCodedRabaCoderDupKeys.java
Removed Paths:
-------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/lib/lgpl-utils-1.0.7-140114.jar
Modified: branches/BIGDATA_RELEASE_1_3_0/.classpath
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/.classpath 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/.classpath 2014-01-27 16:09:26 UTC (rev 7830)
@@ -32,7 +32,7 @@
<classpathentry kind="src" path="bigdata-gas/src/java"/>
<classpathentry kind="src" path="bigdata-gas/src/test"/>
<classpathentry exported="true" kind="lib" path="bigdata/lib/dsi-utils-1.0.6-020610.jar"/>
- <classpathentry exported="true" kind="lib" path="bigdata/lib/lgpl-utils-1.0.7-140114.jar"/>
+ <classpathentry kind="lib" path="bigdata/lib/lgpl-utils-1.0.7-270114.jar"/>
<classpathentry kind="lib" path="bigdata-jini/lib/apache/zookeeper-3.3.3.jar"/>
<classpathentry exported="true" kind="lib" path="bigdata/lib/jetty/jetty-continuation-7.2.2.v20101205.jar"/>
<classpathentry exported="true" kind="lib" path="bigdata/lib/jetty/jetty-http-7.2.2.v20101205.jar"/>
Deleted: branches/BIGDATA_RELEASE_1_3_0/bigdata/lib/lgpl-utils-1.0.7-140114.jar
===================================================================
(Binary files differ)
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata/lib/lgpl-utils-1.0.7-270114.jar
===================================================================
(Binary files differ)
Property changes on: branches/BIGDATA_RELEASE_1_3_0/bigdata/lib/lgpl-utils-1.0.7-270114.jar
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -59,6 +59,7 @@
import com.bigdata.btree.keys.ASCIIKeyBuilderFactory;
import com.bigdata.btree.keys.IKeyBuilder;
import com.bigdata.btree.raba.codec.FrontCodedRabaCoder;
+import com.bigdata.btree.raba.codec.FrontCodedRabaCoderDupKeys;
import com.bigdata.btree.raba.codec.SimpleRabaCoder;
import com.bigdata.counters.CAT;
import com.bigdata.htree.HTree;
@@ -486,7 +487,8 @@
@SuppressWarnings("rawtypes")
final ITupleSerializer<?, ?> tupleSer = new DefaultTupleSerializer(
new ASCIIKeyBuilderFactory(Bytes.SIZEOF_INT),
- new FrontCodedRabaCoder(ratio),// keys : TODO Optimize for int32!
+ // new FrontCodedRabaCoder(ratio),// keys : TODO Optimize for int32!
+ new FrontCodedRabaCoderDupKeys(ratio),// keys : TODO Optimize for int32!
new SimpleRabaCoder() // vals
);
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/IndexMetadata.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/IndexMetadata.java 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/IndexMetadata.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -48,6 +48,7 @@
import com.bigdata.btree.raba.codec.CanonicalHuffmanRabaCoder;
import com.bigdata.btree.raba.codec.FrontCodedRabaCoder;
import com.bigdata.btree.raba.codec.FrontCodedRabaCoder.DefaultFrontCodedRabaCoder;
+import com.bigdata.btree.raba.codec.FrontCodedRabaCoderDupKeys;
import com.bigdata.btree.raba.codec.IRabaCoder;
import com.bigdata.btree.view.FusedView;
import com.bigdata.config.Configuration;
@@ -2096,9 +2097,16 @@
// this.addrSer = AddressSerializer.INSTANCE;
// this.nodeKeySer = PrefixSerializer.INSTANCE;
+ final Class keyRabaCoder;
+ if (this instanceof HTreeIndexMetadata) {
+ keyRabaCoder = FrontCodedRabaCoderDupKeys.class;
+ } else {
+ keyRabaCoder = DefaultFrontCodedRabaCoder.class;
+ }
+
this.nodeKeysCoder = newInstance(getProperty(indexManager, properties,
namespace, Options.NODE_KEYS_CODER,
- DefaultFrontCodedRabaCoder.class.getName()), IRabaCoder.class);
+ keyRabaCoder.getName()), IRabaCoder.class);
// this.tupleSer = DefaultTupleSerializer.newInstance();
{
@@ -2116,7 +2124,7 @@
final IRabaCoder leafKeysCoder = newInstance(getProperty(
indexManager, properties, namespace,
- Options.LEAF_KEYS_CODER, DefaultFrontCodedRabaCoder.class
+ Options.LEAF_KEYS_CODER, keyRabaCoder
.getName()), IRabaCoder.class);
final IRabaCoder valuesCoder = newInstance(getProperty(
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/AbstractKeyBuffer.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/AbstractKeyBuffer.java 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/AbstractKeyBuffer.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -76,20 +76,24 @@
*/
abstract protected int _binarySearch(final int searchKeyOffset, final byte[] searchKey);
+ @Override
public Iterator<byte[]> iterator() {
return new Iterator<byte[]>() {
int i = 0;
+ @Override
public boolean hasNext() {
return i < size();
}
+ @Override
public byte[] next() {
return get(i++);
}
+ @Override
public void remove() {
throw new UnsupportedOperationException();
}
@@ -107,5 +111,5 @@
* The length of the leading prefix shared by all keys.
*/
abstract public int getPrefixLength();
-
+
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/AbstractRaba.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/AbstractRaba.java 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/AbstractRaba.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -124,25 +124,29 @@
this.a = a;
}
-
+
+ @Override
final public int size() {
return (toIndex - fromIndex);
}
+ @Override
final public boolean isEmpty() {
return toIndex == fromIndex;
}
+ @Override
final public boolean isFull() {
return size() == capacity();
}
+ @Override
final public int capacity() {
return capacity;
@@ -163,6 +167,7 @@
}
+ @Override
final public byte[] get(final int index) {
assert rangeCheck(index);
@@ -171,6 +176,7 @@
}
+ @Override
final public int length(final int index) {
assert rangeCheck(index);
@@ -184,6 +190,7 @@
}
+ @Override
final public boolean isNull(final int index) {
assert rangeCheck(index);
@@ -192,6 +199,7 @@
}
+ @Override
final public int copy(final int index, final OutputStream out) {
assert rangeCheck(index);
@@ -215,18 +223,21 @@
}
+ @Override
final public Iterator<byte[]> iterator() {
return new Iterator<byte[]>() {
int i = fromIndex;
+ @Override
public boolean hasNext() {
return i < toIndex;
}
+ @Override
public byte[] next() {
if (!hasNext())
@@ -236,6 +247,7 @@
}
+ @Override
public void remove() {
if (isReadOnly())
@@ -294,6 +306,7 @@
}
+ @Override
public void set(final int index, final byte[] key) {
assertNotReadOnly();
@@ -306,6 +319,7 @@
}
+ @Override
public int add(final byte[] key) {
assertNotReadOnly();
@@ -322,6 +336,7 @@
}
+ @Override
public int add(final byte[] key, final int off, final int len) {
assertNotReadOnly();
@@ -346,6 +361,7 @@
}
+ @Override
public int add(final DataInput in, final int len) throws IOException {
assertNotReadOnly();
@@ -362,6 +378,7 @@
}
+ @Override
public int search(final byte[] searchKey) {
if (!isKeys()) {
@@ -375,6 +392,7 @@
}
+ @Override
public String toString() {
return toString(this);
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/ConditionalRabaCoder.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/ConditionalRabaCoder.java 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/ConditionalRabaCoder.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -74,18 +74,27 @@
}
+ @Override
final public boolean isKeyCoder() {
return smallCoder.isKeyCoder() && bigCoder.isKeyCoder();
}
+ @Override
final public boolean isValueCoder() {
return smallCoder.isValueCoder() && bigCoder.isValueCoder();
}
+ @Override
+ public boolean isDuplicateKeys() {
+
+ return smallCoder.isDuplicateKeys() && bigCoder.isDuplicateKeys();
+
+ }
+
/**
* De-serialization ctor.
*/
@@ -123,6 +132,7 @@
}
+ @Override
public ICodedRaba decode(final AbstractFixedByteArrayBuffer data) {
final boolean isSmall = data.getByte(0) == 1 ? true : false;
@@ -146,6 +156,7 @@
}
+ @Override
public ICodedRaba encodeLive(final IRaba raba, final DataOutputBuffer buf) {
final int size = raba.size();
@@ -175,6 +186,7 @@
}
+ @Override
public AbstractFixedByteArrayBuffer encode(final IRaba raba,
final DataOutputBuffer buf) {
@@ -201,6 +213,7 @@
}
+ @Override
public void readExternal(final ObjectInput in) throws IOException,
ClassNotFoundException {
@@ -220,6 +233,7 @@
}
+ @Override
public void writeExternal(final ObjectOutput out) throws IOException {
out.writeByte(VERSION0);
@@ -260,72 +274,89 @@
}
+ @Override
public AbstractFixedByteArrayBuffer data() {
return data;
}
+ @Override
public int add(byte[] value, int off, int len) {
return delegate.add(value, off, len);
}
+ @Override
public int add(byte[] a) {
return delegate.add(a);
}
+ @Override
public int add(DataInput in, int len) throws IOException {
return delegate.add(in, len);
}
+ @Override
public int capacity() {
return delegate.capacity();
}
- public int copy(int index, OutputStream os) {
+ @Override
+ public int copy(final int index, final OutputStream os) {
return delegate.copy(index, os);
}
- public byte[] get(int index) {
+ @Override
+ public byte[] get(final int index) {
return delegate.get(index);
}
+ @Override
public boolean isEmpty() {
return delegate.isEmpty();
}
+ @Override
public boolean isFull() {
return delegate.isFull();
}
+ @Override
public boolean isKeys() {
return delegate.isKeys();
}
- public boolean isNull(int index) {
+ @Override
+ public boolean isNull(final int index) {
return delegate.isNull(index);
}
+ @Override
public boolean isReadOnly() {
return delegate.isReadOnly();
}
+ @Override
public Iterator<byte[]> iterator() {
return delegate.iterator();
}
- public int length(int index) {
+ @Override
+ public int length(final int index) {
return delegate.length(index);
}
- public int search(byte[] searchKey) {
+ @Override
+ public int search(final byte[] searchKey) {
return delegate.search(searchKey);
}
- public void set(int index, byte[] a) {
+ @Override
+ public void set(final int index, final byte[] a) {
delegate.set(index, a);
}
+ @Override
public int size() {
return delegate.size();
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/EmptyRaba.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/EmptyRaba.java 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/EmptyRaba.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -50,6 +50,7 @@
}
+ @Override
final public boolean isKeys() {
return true;
@@ -78,6 +79,7 @@
}
+ @Override
final public boolean isKeys() {
return false;
@@ -92,75 +94,92 @@
public EmptyRaba() {
}
-
+
+ @Override
final public int capacity() {
return 0;
}
+ @Override
final public boolean isEmpty() {
return true;
}
+ @Override
final public boolean isFull() {
return true;
}
+ @Override
final public int size() {
return 0;
}
+ @Override
final public boolean isReadOnly() {
return true;
}
+ @Override
final public boolean isNull(int index) {
throw new IndexOutOfBoundsException();
}
+ @Override
final public int length(int index) {
throw new IndexOutOfBoundsException();
}
+ @Override
final public byte[] get(int index) {
throw new IndexOutOfBoundsException();
}
+ @Override
final public int copy(int index, OutputStream os) {
throw new IndexOutOfBoundsException();
}
+ @Override
@SuppressWarnings("unchecked")
final public Iterator<byte[]> iterator() {
return EmptyIterator.DEFAULT;
}
+ @Override
final public int search(byte[] searchKey) {
if (isKeys())
return -1;
throw new UnsupportedOperationException();
}
+ @Override
final public void set(int index, byte[] a) {
throw new UnsupportedOperationException();
}
+ @Override
final public int add(byte[] a) {
throw new UnsupportedOperationException();
}
+ @Override
final public int add(byte[] value, int off, int len) {
throw new UnsupportedOperationException();
}
+ @Override
final public int add(DataInput in, int len) throws IOException {
throw new UnsupportedOperationException();
}
+ @Override
public void readExternal(ObjectInput in) throws IOException,
ClassNotFoundException {
// NOP
}
+ @Override
public void writeExternal(ObjectOutput out) throws IOException {
// NOP
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/IRaba.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/IRaba.java 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/IRaba.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -64,7 +64,6 @@
* support {@link #search(byte[])} and <code>null</code>s are allowed.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
*
* @todo consider discarding all of the {@link #add(byte[])} methods. It is
* typically easier and more convenient to directly manage the existing
@@ -78,7 +77,7 @@
/**
* Return <code>true</code> if this implementation is read-only.
*/
- public boolean isReadOnly();
+ boolean isReadOnly();
/**
* When <code>true</code> the {@link IRaba} supports search and elements are
@@ -96,6 +95,19 @@
*/
boolean isKeys();
+ /*
+ * TODO This could be added to differentiate between IRaba implementations
+ * that do / do not support duplicate keys. The ones used with the HTree do.
+ * The rest do not.
+ */
+// /**
+// * When <code>true</code>, then {@link IRaba} supports duplicate keys.
+// *
+// * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/763" >
+// * Stochastic Results With Analytic Query Mode </a>
+// */
+// boolean isDuplicateKeys();
+
/**
* The capacity of the logical byte[][].
*
@@ -194,6 +206,7 @@
* <code>null</code>, then the iterator will report a <code>null</code> for
* that element.
*/
+ @Override
public Iterator<byte[]> iterator();
/*
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/MutableKeysRaba.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/MutableKeysRaba.java 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/MutableKeysRaba.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -42,6 +42,7 @@
/**
* No - this view is mutable.
*/
+ @Override
public boolean isReadOnly() {
return false;
@@ -51,12 +52,13 @@
/**
* Yes.
*/
+ @Override
final public boolean isKeys() {
return true;
}
-
+
/**
* Create a view of a byte[][]. All elements in the array are visible in the
* view.
@@ -113,6 +115,7 @@
}
+ @Override
public MutableKeysRaba resize(final int n) {
return (MutableKeysRaba) super.resize(n);
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/MutableValueBuffer.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/MutableValueBuffer.java 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/MutableValueBuffer.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -63,6 +63,7 @@
/**
* For B+Tree values.
*/
+ @Override
final public boolean isKeys() {
return false;
@@ -142,24 +143,28 @@
}
+ @Override
final public int size() {
return nvalues;
}
+ @Override
final public boolean isEmpty() {
return nvalues == 0;
}
+ @Override
final public boolean isFull() {
return nvalues == values.length;
}
+ @Override
final public int capacity() {
return values.length;
@@ -188,6 +193,7 @@
}
+ @Override
final public int length(final int index) {
assert rangeCheck(index);
@@ -201,6 +207,7 @@
}
+ @Override
final public boolean isNull(final int index) {
assert rangeCheck(index);
@@ -209,6 +216,7 @@
}
+ @Override
final public int copy(final int index, final OutputStream out) {
assert rangeCheck(index);
@@ -232,18 +240,21 @@
}
+ @Override
final public Iterator<byte[]> iterator() {
return new Iterator<byte[]>() {
int i = 0;
+ @Override
public boolean hasNext() {
return i < nvalues;
}
+ @Override
public byte[] next() {
if (!hasNext())
@@ -253,6 +264,7 @@
}
+ @Override
public void remove() {
if (isReadOnly())
@@ -281,6 +293,7 @@
}
+ @Override
public void set(final int index, final byte[] key) {
assert rangeCheck(index);
@@ -289,6 +302,7 @@
}
+ @Override
public int add(final byte[] key) {
assertNotFull();
@@ -299,6 +313,7 @@
}
+ @Override
public int add(final byte[] key, final int off, final int len) {
assertNotFull();
@@ -313,6 +328,7 @@
}
+ @Override
public int add(final DataInput in, final int len) throws IOException {
assertNotFull();
@@ -327,12 +343,14 @@
}
+ @Override
final public int search(final byte[] searchKey) {
throw new UnsupportedOperationException();
}
+ @Override
public String toString() {
return AbstractRaba.toString(this);
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/AbstractCodedRaba.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/AbstractCodedRaba.java 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/AbstractCodedRaba.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -39,7 +39,6 @@
* mutation operations.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
*/
abstract public class AbstractCodedRaba implements ICodedRaba {
@@ -50,28 +49,34 @@
/**
* Implementation is read-only.
*/
+ @Override
final public boolean isReadOnly() {
return true;
}
+ @Override
final public int add(byte[] a) {
throw new UnsupportedOperationException();
}
+ @Override
final public int add(byte[] value, int off, int len) {
throw new UnsupportedOperationException();
}
+ @Override
final public int add(DataInput in, int len) throws IOException {
throw new UnsupportedOperationException();
}
+ @Override
final public void set(int index, byte[] a) {
throw new UnsupportedOperationException();
}
+ @Override
final public String toString() {
return AbstractRaba.toString(this);
@@ -82,18 +87,21 @@
* Basic implementation may be overridden if a faster implementation is
* available.
*/
+ @Override
public Iterator<byte[]> iterator() {
return new Iterator<byte[]>() {
int i = 0;
+ @Override
public boolean hasNext() {
return i < size();
}
+ @Override
public byte[] next() {
if (!hasNext())
@@ -103,6 +111,7 @@
}
+ @Override
public void remove() {
throw new UnsupportedOperationException();
@@ -112,5 +121,5 @@
};
}
-
+
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/CanonicalHuffmanRabaCoder.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/CanonicalHuffmanRabaCoder.java 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/CanonicalHuffmanRabaCoder.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -202,7 +202,6 @@
* the codec and the decoder, which must process 8 bytes at a time.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
*/
public class CanonicalHuffmanRabaCoder implements IRabaCoder, Externalizable {
@@ -218,21 +217,25 @@
*/
final protected static transient byte VERSION0 = 0x00;
+ @Override
public void readExternal(ObjectInput in) throws IOException,
ClassNotFoundException {
// No state.
}
+ @Override
public void writeExternal(ObjectOutput out) throws IOException {
// No state.
}
+ @Override
final public boolean isKeyCoder() {
return true;
}
+ @Override
final public boolean isValueCoder() {
return true;
@@ -666,6 +669,7 @@
/**
* Return the #of distinct symbols used to generate the code.
*/
+ @Override
abstract public int getSymbolCount();
/**
@@ -930,6 +934,7 @@
private final DecoderInputs decoderInputs;
+ @Override
public DecoderInputs decoderInputs() {
return decoderInputs;
@@ -956,12 +961,14 @@
}
+ @Override
final public int byte2symbol(final byte b) {
return byte2symbol.get(b);
}
+ @Override
final public byte symbol2byte(final int symbol) {
return (byte) symbol2byte[symbol];
@@ -1104,6 +1111,7 @@
}
+ @Override
public AbstractFixedByteArrayBuffer encode(final IRaba raba,
final DataOutputBuffer buf) {
@@ -1111,6 +1119,7 @@
}
+ @Override
public ICodedRaba encodeLive(final IRaba raba, final DataOutputBuffer buf) {
final AbstractCodingSetup setup = new RabaCodingSetup(raba);
@@ -1334,6 +1343,7 @@
}
+ @Override
public ICodedRaba decode(final AbstractFixedByteArrayBuffer data) {
return new CodedRabaImpl(data);
@@ -1668,6 +1678,7 @@
}
+ @Override
final public int size() {
return size;
@@ -1677,12 +1688,14 @@
/**
* The capacity is equal to the size (the data are immutable).
*/
+ @Override
final public int capacity() {
return size;
}
+ @Override
final public boolean isEmpty() {
return size == 0;
@@ -1693,18 +1706,21 @@
* Always returns <code>true</code> since {@link #size()} ==
* {@link #capacity()} by definition for this class.
*/
+ @Override
final public boolean isFull() {
return true;
}
+ @Override
final public boolean isKeys() {
return isKeys;
}
+ @Override
public AbstractFixedByteArrayBuffer data() {
return data;
@@ -1750,6 +1766,7 @@
//
// }
+ @Override
public boolean isNull(final int index) {
if (index < 0 || index >= size)
@@ -1774,6 +1791,7 @@
* This computes the length of the decoded byte[] by counting the code
* words for the coded value.
*/
+ @Override
public int length(final int index) {
if (index < 0 || index >= size)
@@ -1876,6 +1894,7 @@
* allocates the byte[]. The second pass decodes into the allocated
* byte[].
*/
+ @Override
public byte[] get(final int index) {
if (index < 0 || index >= size)
@@ -2048,6 +2067,7 @@
* This decodes the value at the specified index in a single pass onto
* the caller's stream.
*/
+ @Override
public int copy(final int index, final OutputStream os) {
if (index < 0 || index >= size)
@@ -2213,6 +2233,7 @@
* Basic implementation may be overridden if a faster implementation is
* available.
*/
+ @Override
public Iterator<byte[]> iterator() {
/**
@@ -2232,12 +2253,14 @@
int i = 0;
+ @Override
public boolean hasNext() {
return i < size();
}
+ @Override
public byte[] next() {
if (!hasNext())
@@ -2280,6 +2303,7 @@
}
+ @Override
public void remove() {
throw new UnsupportedOperationException();
@@ -2294,6 +2318,7 @@
* This is an efficient binary search performed without materializing the
* coded byte[][].
*/
+ @Override
public int search(final byte[] probe) {
if (probe == null)
@@ -2520,4 +2545,11 @@
}
+ @Override
+ public boolean isDuplicateKeys() {
+
+ return false;
+
+ }
+
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/EmptyRabaValueCoder.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/EmptyRabaValueCoder.java 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/EmptyRabaValueCoder.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -46,7 +46,6 @@
* B+Tree will be <strong>discarded</strong> by this {@link IRabaCoder}.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
*/
public class EmptyRabaValueCoder implements IRabaCoder, Externalizable {
@@ -55,6 +54,7 @@
*/
private static final long serialVersionUID = -8011456562258609162L;
+ @Override
public void readExternal(ObjectInput in) throws IOException,
ClassNotFoundException {
@@ -62,6 +62,7 @@
}
+ @Override
public void writeExternal(ObjectOutput out) throws IOException {
// NOP
@@ -77,6 +78,7 @@
/**
* No. Keys can not be constrained to be empty.
*/
+ @Override
final public boolean isKeyCoder() {
return false;
@@ -86,12 +88,21 @@
/**
* Yes.
*/
+ @Override
final public boolean isValueCoder() {
return true;
}
+ @Override
+ public boolean isDuplicateKeys() {
+
+ return false;
+
+ }
+
+ @Override
public ICodedRaba encodeLive(final IRaba raba, final DataOutputBuffer buf) {
if (raba == null)
@@ -119,6 +130,7 @@
* <strong>Any data in the {@link IRaba} will be discarded!</strong> Only
* the {@link IRaba#size()} is maintained.
*/
+ @Override
public AbstractFixedByteArrayBuffer encode(final IRaba raba,
final DataOutputBuffer buf) {
@@ -140,6 +152,7 @@
}
+ @Override
public ICodedRaba decode(final AbstractFixedByteArrayBuffer data) {
return new EmptyCodedRaba(data);
@@ -182,6 +195,7 @@
}
+ @Override
final public AbstractFixedByteArrayBuffer data() {
return data;
@@ -191,43 +205,50 @@
/**
* Yes.
*/
+ @Override
final public boolean isReadOnly() {
return true;
}
+ @Override
public boolean isKeys() {
return false;
}
+ @Override
final public int capacity() {
return size;
}
+ @Override
final public int size() {
return size;
}
+ @Override
final public boolean isEmpty() {
return size == 0;
}
+ @Override
final public boolean isFull() {
return true;
}
- final public boolean isNull(int index) {
+ @Override
+ final public boolean isNull(final int index) {
if (index < 0 || index >= size)
throw new IndexOutOfBoundsException();
@@ -236,7 +257,8 @@
}
- final public int length(int index) {
+ @Override
+ final public int length(final int index) {
if (index < 0 || index >= size)
throw new IndexOutOfBoundsException();
@@ -245,7 +267,8 @@
}
- final public byte[] get(int index) {
+ @Override
+ final public byte[] get(final int index) {
if (index < 0 || index >= size)
throw new IndexOutOfBoundsException();
@@ -254,7 +277,8 @@
}
- final public int copy(int index, OutputStream os) {
+ @Override
+ final public int copy(final int index, final OutputStream os) {
if (index < 0 || index >= size)
throw new IndexOutOfBoundsException();
@@ -263,18 +287,21 @@
}
+ @Override
final public Iterator<byte[]> iterator() {
return new Iterator<byte[]>() {
int i = 0;
+ @Override
public boolean hasNext() {
return i < size;
}
+ @Override
public byte[] next() {
i++;
@@ -283,6 +310,7 @@
}
+ @Override
public void remove() {
throw new UnsupportedOperationException();
@@ -299,6 +327,7 @@
* @throws UnsupportedOperationException
* unless the {@link IRaba} represents B+Tree keys.
*/
+ @Override
final public int search(final byte[] searchKey) {
if (isKeys())
@@ -312,22 +341,27 @@
* Mutation API is not supported.
*/
+ @Override
final public int add(byte[] a) {
throw new UnsupportedOperationException();
}
+ @Override
final public int add(byte[] value, int off, int len) {
throw new UnsupportedOperationException();
}
+ @Override
final public int add(DataInput in, int len) throws IOException {
throw new UnsupportedOperationException();
}
+ @Override
final public void set(int index, byte[] a) {
throw new UnsupportedOperationException();
}
+ @Override
final public String toString() {
return AbstractRaba.toString(this);
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/FixedLengthValueRabaCoder.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/FixedLengthValueRabaCoder.java 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/FixedLengthValueRabaCoder.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -73,7 +73,6 @@
* </dl>
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
*/
public class FixedLengthValueRabaCoder implements IRabaCoder, Externalizable {
@@ -87,6 +86,7 @@
/**
* No.
*/
+ @Override
final public boolean isKeyCoder() {
return false;
@@ -96,12 +96,20 @@
/**
* Yes.
*/
+ @Override
final public boolean isValueCoder() {
return true;
}
+ @Override
+ public boolean isDuplicateKeys() {
+
+ return false;
+
+ }
+
/**
* The required length for all non-<code>null</code> values.
*/
@@ -132,12 +140,14 @@
}
+ @Override
public void writeExternal(final ObjectOutput out) throws IOException {
out.writeInt(len);
}
+ @Override
public void readExternal(final ObjectInput in) throws IOException,
ClassNotFoundException {
@@ -168,6 +178,7 @@
* if the {@link IRaba} has a non-<code>null</code> value with a
* length other than the length specified to the constructor.
*/
+ @Override
public ICodedRaba encodeLive(final IRaba raba, final DataOutputBuffer buf) {
if (raba == null)
@@ -262,6 +273,7 @@
}
+ @Override
public AbstractFixedByteArrayBuffer encode(final IRaba raba,
final DataOutputBuffer buf) {
@@ -269,6 +281,7 @@
}
+ @Override
public ICodedRaba decode(final AbstractFixedByteArrayBuffer data) {
return new CodedRabaImpl(len, data);
@@ -355,36 +368,42 @@
*/
private final int O_values;
+ @Override
final public AbstractFixedByteArrayBuffer data() {
return data;
}
+ @Override
public boolean isKeys() {
return false;
}
+ @Override
final public int capacity() {
return size;
}
+ @Override
final public int size() {
return size;
}
+ @Override
final public boolean isEmpty() {
return size == 0;
}
+ @Override
final public boolean isFull() {
return true;
@@ -398,6 +417,7 @@
}
+ @Override
public boolean isNull(final int index) {
rangeCheck(index);
@@ -406,6 +426,7 @@
}
+ @Override
public int length(final int index) {
if (isNull(index))
@@ -415,6 +436,7 @@
}
+ @Override
public byte[] get(final int index) {
if (isNull(index))
@@ -431,6 +453,7 @@
}
+ @Override
public int copy(final int index, final OutputStream os) {
if (isNull(index))
@@ -456,6 +479,7 @@
* Search
*/
+ @Override
public int search(final byte[] key) {
throw new UnsupportedOperationException();
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/FrontCodedRabaCoder.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/FrontCodedRabaCoder.java 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/FrontCodedRabaCoder.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -45,7 +45,6 @@
* The data MUST be ordered. <code>null</code> values are not allowed.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
*/
public class FrontCodedRabaCoder implements IRabaCoder, Externalizable {
@@ -59,6 +58,7 @@
private int ratio;
+ @Override
public String toString() {
return super.toString() + "{ratio=" + ratio + "}";
@@ -72,7 +72,6 @@
*
* @author <a href="mailto:tho...@us...">Bryan
* Thompson</a>
- * @version $Id$
*/
public static class DefaultFrontCodedRabaCoder extends FrontCodedRabaCoder {
@@ -91,11 +90,13 @@
}
+ @Override
public void readExternal(ObjectInput in) throws IOException,
ClassNotFoundException {
// NOP
}
+ @Override
public void writeExternal(ObjectOutput out) throws IOException {
// NOP
}
@@ -142,18 +143,28 @@
}
+ @Override
final public boolean isKeyCoder() {
return true;
}
+ @Override
final public boolean isValueCoder() {
return false;
}
+ @Override
+ public boolean isDuplicateKeys() {
+
+ return false;
+
+ }
+
+ @Override
public void readExternal(ObjectInput in) throws IOException,
ClassNotFoundException {
@@ -161,6 +172,7 @@
}
+ @Override
public void writeExternal(ObjectOutput out) throws IOException {
out.writeInt(ratio);
@@ -190,6 +202,7 @@
/** The byte offset of the start of the front-coded representation. */
private static final int O_DATA = O_RATIO + SIZEOF_RATIO;
+ @Override
public ICodedRaba encodeLive(final IRaba raba, final DataOutputBuffer buf) {
if (raba == null)
@@ -212,7 +225,7 @@
// front-code the byte[][].
final CustomByteArrayFrontCodedList decoder = new CustomByteArrayFrontCodedList(
- raba.iterator(), ratio);
+ raba.iterator(), ratio, isDuplicateKeys());
try {
@@ -242,6 +255,7 @@
}
+ @Override
public AbstractFixedByteArrayBuffer encode(final IRaba raba,
final DataOutputBuffer buf) {
@@ -255,9 +269,10 @@
}
+ @Override
public ICodedRaba decode(final AbstractFixedByteArrayBuffer data) {
- return new CodedRabaImpl(data);
+ return new CodedRabaImpl(data, isDuplicateKeys());
}
@@ -278,8 +293,12 @@
*
* @param data
* The record containing the coded data.
+ * @param hasDups
+ * <code>true</code> iff the {@link IRabaCoder} supports
+ * duplicate keys.
*/
- public CodedRabaImpl(final AbstractFixedByteArrayBuffer data) {
+ public CodedRabaImpl(final AbstractFixedByteArrayBuffer data,
+ final boolean hasDups) {
final byte version = data.getByte(O_VERSION);
@@ -297,7 +316,7 @@
// wrap slice with decoder.
this.decoder = new CustomByteArrayFrontCodedList(size, ratio, data
- .array(), data.off() + O_DATA, data.len());
+ .array(), data.off() + O_DATA, data.len(), hasDups);
this.data = data;
@@ -321,6 +340,7 @@
}
+ @Override
public AbstractFixedByteArrayBuffer data() {
return data;
@@ -330,24 +350,28 @@
/**
* Represents B+Tree keys.
*/
+ @Override
final public boolean isKeys() {
return true;
}
+ @Override
final public int size() {
return decoder.size();
}
+ @Override
final public int capacity() {
return decoder.size();
}
+ @Override
final public boolean isEmpty() {
return size() == 0;
@@ -358,6 +382,7 @@
* Always returns <code>true</code> since the front-coded representation
* is dense.
*/
+ @Override
final public boolean isFull() {
return true;
@@ -368,24 +393,28 @@
* Always returns <code>false</code> (<code>null</code>s are not
* allowed).
*/
+ @Override
final public boolean isNull(final int index) {
return false;
}
+ @Override
final public byte[] get(final int index) {
return decoder.get(index);
}
+ @Override
final public int length(final int index) {
return decoder.arrayLength(index);
}
+ @Override
public int copy(final int index, final OutputStream os) {
try {
@@ -400,12 +429,14 @@
}
+ @Override
public Iterator<byte[]> iterator() {
return decoder.iterator();
}
+ @Override
public int search(final byte[] searchKey) {
// optimization: always keys.
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/FrontCodedRabaCoderDupKeys.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/FrontCodedRabaCoderDupKeys.java (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/FrontCodedRabaCoderDupKeys.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -0,0 +1,59 @@
+/*
+
+Copyright (C) SYSTAP, LLC 2006-2008. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+package com.bigdata.btree.raba.codec;
+
+/**
+ * Variant of the {@link FrontCodedRabaCoder} that supports duplicate keys.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ */
+public class FrontCodedRabaCoderDupKeys extends FrontCodedRabaCoder {
+
+ /**
+ * A default instance.
+ */
+ public static final transient FrontCodedRabaCoderDupKeys INSTANCE = new FrontCodedRabaCoderDupKeys();
+
+ protected transient static final int DEFAULT_RATIO = 8;
+
+ public FrontCodedRabaCoderDupKeys(final int ratio) {
+
+ super(ratio);
+
+ }
+
+ public FrontCodedRabaCoderDupKeys() {
+
+ super(DEFAULT_RATIO);
+
+ }
+
+ @Override
+ public boolean isDuplicateKeys() {
+
+ return true;
+
+ }
+
+}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/IRabaCoder.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/IRabaCoder.java 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/IRabaCoder.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -36,7 +36,6 @@
* @see IRaba
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
*/
public interface IRabaCoder extends Serializable {
@@ -55,6 +54,14 @@
boolean isValueCoder();
/**
+ * Return true iff this {@link IRabaCoder} supports duplicate keys.
+ *
+ * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/763" >
+ * Stochastic Results With Analytic Query Mode </a>
+ */
+ boolean isDuplicateKeys();
+
+ /**
* Encode the data, returning an {@link ICodedRaba}. Implementations of this
* method should be optimized for the very common use case where the caller
* requires immediate access to the coded data record. In that case, many of
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/SimpleRabaCoder.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/SimpleRabaCoder.java 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/btree/raba/codec/SimpleRabaCoder.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -47,7 +47,6 @@
* and B+Tree values.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
*/
public class SimpleRabaCoder implements IRabaCoder, Externalizable {
@@ -74,6 +73,7 @@
/**
* Yes.
*/
+ @Override
final public boolean isKeyCoder() {
return true;
@@ -83,25 +83,35 @@
/**
* Yes.
*/
+ @Override
final public boolean isValueCoder() {
return true;
}
- /**
+ @Override
+ public boolean isDuplicateKeys() {
+
+ return false;
+
+ }
+
+ /**
* De-serialization ctor. Use {@link #INSTANCE} otherwise.
*/
public SimpleRabaCoder() {
}
+ @Override
public void writeExternal(ObjectOutput out) throws IOException {
// NOP
}
+ @Override
public void readExternal(ObjectInput in) throws IOException,
ClassNotFoundException {
@@ -136,6 +146,7 @@
return O_SIZE + SIZEOF_SIZE + SIZEOF_CAPACITY;
}
+ @Override
public ICodedRaba encodeLive(final IRaba raba, final DataOutputBuffer buf) {
if (raba == null)
@@ -244,6 +255,7 @@
}
+ @Override
public AbstractFixedByteArrayBuffer encode(final IRaba raba,
final DataOutputBuffer buf) {
@@ -256,6 +268,7 @@
}
+ @Override
public ICodedRaba decode(final AbstractFixedByteArrayBuffer data) {
return new CodedRabaImpl(data);
@@ -366,36 +379,42 @@
*/
private final int O_offsets;
+ @Override
final public AbstractFixedByteArrayBuffer data() {
return data;
}
+ @Override
public boolean isKeys() {
return isKeys;
}
+ @Override
final public int capacity() {
return capacity;
}
+ @Override
final public int size() {
return size;
}
+ @Override
final public boolean isEmpty() {
return size == 0;
}
+ @Override
final public boolean isFull() {
return true;
@@ -409,6 +428,7 @@
}
+ @Override
public boolean isNull(final int index) {
if (index >= size && index < capacity) {
@@ -425,6 +445,7 @@
}
+ @Override
public int length(final int index) {
if (isNull(index))
@@ -443,6 +464,7 @@
}
+ @Override
public byte[] get(final int index) {
if (isNull(index))
@@ -477,6 +499,7 @@
}
+ @Override
public int copy(final int index, final OutputStream os) {
if (isNull(index))
@@ -509,6 +532,7 @@
* Search
*/
+ @Override
public int search(final byte[] key) {
if (!isKeys())
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/htree/DirectoryPage.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/htree/DirectoryPage.java 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/htree/DirectoryPage.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -38,7 +38,6 @@
import com.bigdata.btree.ITuple;
import com.bigdata.btree.ITupleIterator;
import com.bigdata.btree.Node;
-import com.bigdata.btree.PageStats;
import com.bigdata.htree.AbstractHTree.ChildMemoizer;
import com.bigdata.htree.AbstractHTree.LoadChildRequest;
import com.bigdata.htree.data.IDirectoryData;
@@ -2058,6 +2057,23 @@
}
/**
+ * If this is an overflow directory then the depth-based hashCode is irrelevant
+ * since it is used as a blob container for BucketPage references.
+ */
+ public int getLocalHashCode(final byte[] key, final int prefixLength) {
+ if (isOverflowDirectory()) {
+ /*
+ * Shouldn't need to check the key, this will be handled when
+ * the BucketPage is checked for a precise match
+ */
+ return 0;
+ }
+
+ return super.getLocalHashCode(key, prefixLength);
+
+ }
+
+ /**
* This method is never called at present since DirectoryPages are
* always created at maximum depth. Whether there is any advantage
* in supporting pages of lesser depths is yet to be determined.
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/htree/NodeSerializer.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/htree/NodeSerializer.java 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/htree/NodeSerializer.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -253,6 +253,19 @@
*/
this.nodeCoder = new DefaultDirectoryPageCoder();
+ if (!indexMetadata.getTupleSerializer().getLeafKeysCoder().isDuplicateKeys()) {
+
+ /*
+ * This constraint *could* be relaxed, but the HTree API presumes
+ * that we can have duplicate keys and this check verifies tha the
+ * keys coder supports duplicate keys.
+ */
+
+ throw new IllegalArgumentException(
+ "The leaf keys coder for HTree should allow duplicate keys.");
+
+ }
+
/*
* Note: We are using the same leaf coder class as the BTree.
*/
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/htree/raba/MutableKeyBuffer.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/htree/raba/MutableKeyBuffer.java 2014-01-26 20:37:25 UTC (rev 7829)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/htree/raba/MutableKeyBuffer.java 2014-01-27 16:09:26 UTC (rev 7830)
@@ -187,6 +187,7 @@
}
+ @Override
public String toString() {
return AbstractRaba.toString(this);
@@ -196,12 +197,14 @@
/**
* Returns a reference to the key at that index.
*/
+ @Override
final public byte[] get(final int index) {
return keys[index];
}
+ @Override
final public int length(final int index) {
final byte[] tmp = keys[index];
@@ -213,6 +216,7 @@
}
+ @Override
final public int copy(final int index, final OutputStream out) {
final byte[] tmp = keys[index];
@@ -236,12 +240,14 @@
*
* @return <code>true</code> iff the key at that index is <code>null</code>.
*/
+ @Override
final public boolean isNull(final int index) {
return keys[index] == null;
}
+ @Override
final public boolean isEmpty() {
return nkeys == 0;
@@ -256,18 +262,21 @@
* MUST explicitly scan a buddy bucket to determine the #of keys in a buddy
* bucket on the page.
*/
+ @Override
final public int size() {
return nkeys;
}
+ @Override
final public int capacity() {
return keys.length;
}
+ @Override
final public boolean isFull() {
return nkeys == keys.length;
@@ -277,6 +286,7 @@
/**
* Mutable.
*/
+ @Override
final public boolean isReadOnly() {
return false;
@@ -284,11 +294,11 @@
}
/**
- * Instances are NOT searchable. Duplicates and <code>null</code>s ARE
- * permitted.
+ * Instances are searchable and support duplicate keys.
*
- * @returns <code>false</code>
+ * @returns <code>true</code>
*/
+ @Override
final public boolean isKeys() {
return true;
@@ -301,20 +311,24 @@
* This iterator visits all keys on the bucket page, including
* <code>null</code>s.
*/
+ @Override
public Iterator<byte[]> iterator() {
return new Iterator<byte[]>() {
int i = 0;
+ @Override
public boolean hasNext() {
return i < size();
}
+ @Override
public byte[] next() {
return get(i++);
}
+ @Override
public void remove() {
throw new UnsupportedOperationException();
}
@@ -32...
[truncated message content] |
|
From: <tho...@us...> - 2014-01-27 21:23:44
|
Revision: 7834
http://bigdata.svn.sourceforge.net/bigdata/?rev=7834&view=rev
Author: thompsonbry
Date: 2014-01-27 21:23:38 +0000 (Mon, 27 Jan 2014)
Log Message:
-----------
Bug fix for [1]. The root cause was an incorrect AST generated from the SPARQL parser. I have added test cases for this to TestSubqueryPatterns. With this fix, the test case in TestTickets for #806 now runs correctly. The entire AST SPARQL test suite is green. Committed to CI.
See #806 (Incorrect AST generated for OPTIONAL { SELECT })
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/TestTickets.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/java/com/bigdata/rdf/sail/sparql/GroupGraphPatternBuilder.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/sparql/TestGroupGraphPatternBuilder.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/sparql/TestSubqueryPatterns.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/TestTickets.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/TestTickets.java 2014-01-27 16:55:56 UTC (rev 7833)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/TestTickets.java 2014-01-27 21:23:38 UTC (rev 7834)
@@ -320,9 +320,8 @@
}
/**
- * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/806" >
- * Incorrect computation of shared variables when lifting out named
- * subqueries </a>
+ * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/806>
+ * Incorrect AST generated for OPTIONAL { SELECT }</a>
*/
public void test_ticket_806() throws Exception {
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/java/com/bigdata/rdf/sail/sparql/GroupGraphPatternBuilder.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/java/com/bigdata/rdf/sail/sparql/GroupGraphPatternBuilder.java 2014-01-27 16:55:56 UTC (rev 7833)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/java/com/bigdata/rdf/sail/sparql/GroupGraphPatternBuilder.java 2014-01-27 21:23:38 UTC (rev 7834)
@@ -273,20 +273,41 @@
graphPattern = new GroupGraphPattern(parentGP);
// visit the children.
- super.visit(node, null);
+ final Object tmp = super.visit(node, null);
final JoinGroupNode joinGroup = new JoinGroupNode();
joinGroup.setOptional(true);
-
- @SuppressWarnings("rawtypes")
- final GroupNodeBase group = graphPattern.buildGroup(joinGroup);
- parentGP.add(group);
+ if (tmp instanceof SubqueryRoot) {
+
+ /**
+ * Sub-Select
+ *
+ * @see <a
+ * href="https://sourceforge.net/apps/trac/bigdata/ticket/806>
+ * Incorrect computation of shared variables when lifting out named
+ * subqueries </a>
+ */
+ joinGroup.addChild((SubqueryRoot) tmp);
+ } else {
+
+ // GraphPattern
+
+ @SuppressWarnings("rawtypes")
+ final GroupNodeBase group = graphPattern.buildGroup(joinGroup);
+
+ assert group == joinGroup;// should be the same reference.
+
+ }
+
+ parentGP.add(joinGroup);
+
graphPattern = parentGP;
return null;
+
}
/**
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/sparql/TestGroupGraphPatternBuilder.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/sparql/TestGroupGraphPatternBuilder.java 2014-01-27 16:55:56 UTC (rev 7833)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/sparql/TestGroupGraphPatternBuilder.java 2014-01-27 21:23:38 UTC (rev 7834)
@@ -1298,4 +1298,53 @@
}
+ /**
+ * A unit test for an OPTIONAL wrapping a SERVICE.
+ */
+ public void test_optional_SERVICE() throws MalformedQueryException,
+ TokenMgrError, ParseException {
+
+ final String serviceExpr = "service ?s { ?s ?p ?o }";
+
+ final String sparql = "select ?s where { optional { " + serviceExpr
+ + " } }";
+
+ final QueryRoot expected = new QueryRoot(QueryType.SELECT);
+ final ServiceNode service;
+ {
+
+ {
+ final Map<String, String> prefixDecls = new LinkedHashMap<String, String>();
+ expected.setPrefixDecls(prefixDecls);
+ }
+
+ {
+ final ProjectionNode projection = new ProjectionNode();
+ projection.addProjectionVar(new VarNode("s"));
+ expected.setProjection(projection);
+
+ final JoinGroupNode whereClause = new JoinGroupNode();
+ expected.setWhereClause(whereClause);
+
+ final JoinGroupNode serviceGraph = new JoinGroupNode();
+ serviceGraph.addChild(new StatementPatternNode(
+ new VarNode("s"), new VarNode("p"), new VarNode("o"),
+ null/* c */, Scope.DEFAULT_CONTEXTS));
+
+ service = new ServiceNode(new VarNode("s"), serviceGraph);
+ service.setExprImage(serviceExpr);
+
+ final JoinGroupNode wrapperGroup = new JoinGroupNode(true/* optional */);
+ whereClause.addChild(wrapperGroup);
+ wrapperGroup.addChild(service);
+ }
+
+ }
+
+ final QueryRoot actual = parse(sparql, baseURI);
+
+ assertSameAST(sparql, expected, actual);
+
+ }
+
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/sparql/TestSubqueryPatterns.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/sparql/TestSubqueryPatterns.java 2014-01-27 16:55:56 UTC (rev 7833)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-sails/src/test/com/bigdata/rdf/sail/sparql/TestSubqueryPatterns.java 2014-01-27 21:23:38 UTC (rev 7834)
@@ -146,6 +146,71 @@
}
/**
+ * Unit test for simple optional subquery without anything else in the outer
+ * join group.
+ *
+ * <pre>
+ * SELECT ?s where { OPTIONAL {SELECT ?s where {?s ?p ?o}}}
+ * </pre>
+ *
+ * Note: This requires recursion back in through the
+ * {@link BigdataExprBuilder}.
+ *
+ * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/806>
+ * Incorrect AST generated for OPTIONAL { SELECT }</a>
+ */
+ public void test_optional_subSelect() throws MalformedQueryException,
+ TokenMgrError, ParseException {
+
+ final String sparql = "select ?s where { optional {select ?s where { ?s ?p ?o } } }";
+
+ final QueryRoot expected = new QueryRoot(QueryType.SELECT);
+ final SubqueryRoot subSelect;
+ {
+
+ {
+ final Map<String, String> prefixDecls = new LinkedHashMap<String, String>();
+ expected.setPrefixDecls(prefixDecls);
+ }
+
+ {
+ final ProjectionNode projection = new ProjectionNode();
+ projection.addProjectionVar(new VarNode("s"));
+ expected.setProjection(projection);
+
+ final JoinGroupNode whereClause = new JoinGroupNode();
+ expected.setWhereClause(whereClause);
+
+ subSelect = new SubqueryRoot(QueryType.SELECT);
+// whereClause.addChild(subSelect);
+
+ final JoinGroupNode wrapperGroup = new JoinGroupNode(true/* optional */);
+ whereClause.addChild(wrapperGroup);
+ wrapperGroup.addChild(subSelect);
+ }
+ {
+
+ final ProjectionNode projection2 = new ProjectionNode();
+ projection2.addProjectionVar(new VarNode("s"));
+ subSelect.setProjection(projection2);
+
+ final JoinGroupNode whereClause2 = new JoinGroupNode();
+ subSelect.setWhereClause(whereClause2);
+
+ whereClause2.addChild(new StatementPatternNode(
+ new VarNode("s"), new VarNode("p"), new VarNode("o"),
+ null/* c */, Scope.DEFAULT_CONTEXTS));
+
+ }
+ }
+
+ final QueryRoot actual = parse(sparql, baseURI);
+
+ assertSameAST(sparql, expected, actual);
+
+ }
+
+ /**
* Unit test for simple subquery joined with a triple pattern in the outer
* join group.
*
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X