Menu
▾
▴
[Bigdata-commit] SF.net SVN: bigdata:[3730] branches/QUADS_QUERY_BRANCH
|
From: <tho...@us...> - 2010-10-05 19:57:35
|
Revision: 3730
http://bigdata.svn.sourceforge.net/bigdata/?rev=3730&view=rev
Author: thompsonbry
Date: 2010-10-05 19:57:25 +0000 (Tue, 05 Oct 2010)
Log Message:
-----------
UNION is still not implemented. It is the only remaining feature to be
code complete for scale-out quads query. The decision tree rewrites
in Rule2BOpUtility are still disabled by default. However, if you
enable it and then run TestBigdataSailWithQuads, only one additional
test fails ("Basic - Var 2") due to a "duplicate bop exception". The
problem goes back to the hashCode() and equals() implementations of
Predicate. I will address those in a second commit.
I also plan to look at some optimizations, including: deferring some
aspects of default graph and named graph rewrites to run time;
improved pipelining in the QueryEngine (chaining BlockingBuffers
together in pipes).
- Constants are no longer cloned during deep copy of operator trees.
Also, both Constant and Var now extend a common base class which
disallows copy-on-write mutations to their annotations.
- Hooked in cost model for SCALE-OUT. This requires a mixture of
models to approximate the shard views. Changed from a scalar 'cost'
into a ScanCostReport and a SubqueryCostReport. The cost reports
are attached as annotations to the query plan by Rule2BOpUtility.
- Added a shard split handler for the xxxC indices. Added a test suite
for that split handler. Refactored the existing split handler test
suites to have one test suite per split handler implementation.
- Fixed a bug in DataSetJoin. It was not constraining the output
solutions correctly.
- Removed the "NESTED_SUBQUERY" option, the implementation class
(NestedSubqueryWithJoinThreadsTask), and all of the logic to test
that join throughout the various test suites.
- Cleaned up logic to obtain an AccessPath (all code paths). There was
a broken assumption built into the refactor that the IKeyOrder could
be specified on the IPredicate as an annotation. This does not work
because we select the IKeyOrder, and hence the IAccessPath, based on
the asBound predicate during joins in order to always read on the
most efficient access path. This is also true in scale-out where we
map each binding set across the predicate for the target join in
order to obtain the asBound state for that predicate and then push
the binding set to the shard on which that asBound predicate will
read (or write).
- Fixed mapping over shards. It assumed that the target IKeyOrder was
fixed in advance. In fact, it should be determined at runtime for
normal pipeline joins.
- Moved the Rule2BOpUtility and associated classes to com.bigdata.sail
since they are SPARQL specific.
- Renamed ISolutionExpander to IAccessPathExpander
- An occasional test failure appears in
TestQueryEngine#test_queryJoin2_concurrentStressTest() where it
reports one failure ("nerror=1, ncancel=0, ntimeout=0, ninterrupt=0,
nsuccess=999"). I have not tracked this down, but I have noticed
this off and on for several days.
- TestFederatedQueryEngine is hanging / having controller identity
issues. This shows up on
test_query_join_2shards_nothingBoundOnAccessPath(). Once it happens
the query engine does not tear down correctly and CI will hang.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/architecture/query-cost-model.xls
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContextBase.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BufferAnnotations.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Constant.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPredicate.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Var.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedRunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/shards/Algorithm_FullyBoundPredicate.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/shards/Algorithm_NestedLocatorScan.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/shards/Bundle.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/shards/MapBindingSetsOverShardsBuffer.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/AbstractBTree.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/AbstractTuple.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/BTree.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/BTreeStatistics.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/BTreeUtilizationReport.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/ILocalBTreeView.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/IndexSegment.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/UnisolatedReadWriteIndex.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/AbstractRelation.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/AbstractResource.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/IRelation.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/RelationFusedView.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/accesspath/AccessPath.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/accesspath/BlockingBuffer.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/EmptyAccessPathExpander.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/AbstractJoinNexus.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/DefaultEvaluationPlan2.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/DefaultRangeCountFactory.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/DefaultRuleTaskFactory.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/IJoinNexus.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/NoReorderEvaluationPlan.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/pipeline/DistributedJoinMasterTask.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/search/FullTextIndex.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/AbstractScaleOutFederation.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/IBigdataClient.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/striterator/ChunkedWrappedIterator.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/striterator/IChunkedIterator.java
branches/QUADS_QUERY_BRANCH/bigdata/src/resources/logging/log4j.properties
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/TestBOpUtility.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/ap/TestPredicate.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestFederatedQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/shards/TestMapBindingSetsOverShards.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/relation/locator/TestDefaultResourceLocator.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/relation/rule/TestRule.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/resources/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/resources/TestSegSplitter.java
branches/QUADS_QUERY_BRANCH/bigdata-jini/src/test/com/bigdata/bop/fed/jini/TestJiniFederatedQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/bop/rdf/join/DataSetJoin.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/inf/OwlSameAsPropertiesExpandingIterator.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/lexicon/LexiconRelation.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/load/MappedRDFDataLoadMaster.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/magic/MagicRelation.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/rules/RDFJoinNexus.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/DefaultGraphSolutionExpander.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/NamedGraphSolutionExpander.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPOAccessPath.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPOPredicate.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPORelation.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/join/TestDataSetJoin.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestContextAdvancer.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestDatabaseAtOnceClosure.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestOptionals.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestRuleExpansion.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestSlice.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/spo/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/spo/TestSPORelation.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/store/StressTestCentos.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/FreeTextSearchExpander.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNamedGraphs.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestOptionals.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataConnectionTest.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataSparqlTest.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataStoreTest.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/stress/LoadClosureAndQueryTest.java
branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/NOPFilter.java
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ImmutableBOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/ScanCostReport.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/SubqueryCostReport.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/IAccessPathExpander.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/resources/AbstractTestSegSplitter.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/resources/TestFixedLengthPrefixShardSplits.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/resources/TestSparseRowStoreSplitHandler.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/XXXCShardSplitHandler.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/spo/TestXXXCShardSplitHandler.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/DGExpander.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/DataSetSummary.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/Rule2BOpUtility.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithQuads.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithQuadsWithoutInlining.java
Removed Paths:
-------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/ISolutionExpander.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/NestedSubqueryWithJoinThreadsTask.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithQuads.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithQuadsAndPipelineJoins.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithQuadsAndPipelineJoinsWithoutInlining.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/architecture/query-cost-model.xls
===================================================================
(Binary files differ)
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java 2010-10-05 02:33:11 UTC (rev 3729)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java 2010-10-05 19:57:25 UTC (rev 3730)
@@ -37,6 +37,7 @@
import java.util.Map;
import com.bigdata.bop.constraint.EQ;
+import com.bigdata.btree.Tuple;
/**
* Abstract base class for {@link BOp}s.
@@ -411,10 +412,12 @@
* The name.
* @param value
* The value.
+ *
+ * @return The old value.
*/
- protected void _setProperty(final String name, final Object value) {
+ protected Object _setProperty(final String name, final Object value) {
- annotations.put(name,value);
+ return annotations.put(name,value);
}
@@ -472,7 +475,7 @@
final BOpBase tmp = this.clone();
- if (tmp.annotations.put(name, value) != null)
+ if (tmp._setProperty(name, value) != null)
throw new IllegalStateException("Already set: name=" + name
+ ", value=" + value);
@@ -480,6 +483,49 @@
}
+ /**
+ * Clear the named annotation.
+ *
+ * @param name
+ * The annotation.
+ * @return A copy of this {@link BOp} in which the named annotation has been
+ * removed.
+ */
+ public BOpBase clearProperty(final String name) {
+
+ if (name == null)
+ throw new IllegalArgumentException();
+
+ final BOpBase tmp = this.clone();
+
+ tmp._clearProperty(name);
+
+ return tmp;
+
+ }
+
+ /**
+ * Strips off the named annotations.
+ *
+ * @param names
+ * The annotations to be removed.
+ *
+ * @return A copy of this {@link BOp} in which the specified annotations do not appear.
+ */
+ public BOp clearAnnotations(final String[] names) {
+
+ final BOpBase tmp = this.clone();
+
+ for(String name : names) {
+
+ tmp._clearProperty(name);
+
+ }
+
+ return tmp;
+
+ }
+
public int getId() {
return (Integer) getRequiredProperty(Annotations.BOP_ID);
@@ -489,14 +535,23 @@
public String toString() {
final StringBuilder sb = new StringBuilder();
-// sb.append(getClass().getName());
- sb.append(super.toString());
+ sb.append(getClass().getName());
+// sb.append(super.toString());
+ final Integer bopId = (Integer) getProperty(Annotations.BOP_ID);
+ if (bopId != null) {
+ sb.append("[" + bopId + "]");
+ }
sb.append("(");
- for (int i = 0; i < args.length; i++) {
- final BOp t = args[i];
- if (i > 0)
+ int nwritten = 0;
+ for (BOp t : args) {
+ if (nwritten > 0)
sb.append(',');
sb.append(t.getClass().getSimpleName());
+ final Integer tid = (Integer) t.getProperty(Annotations.BOP_ID);
+ if (tid != null) {
+ sb.append("[" + tid + "]");
+ }
+ nwritten++;
}
sb.append(")");
annotationsToString(sb);
@@ -510,11 +565,16 @@
sb.append("[");
boolean first = true;
for (Map.Entry<String, Object> e : annotations.entrySet()) {
- if (!first)
+ if (first)
+ sb.append(" ");
+ else
sb.append(", ");
if (e.getValue() != null && e.getValue().getClass().isArray()) {
sb.append(e.getKey() + "="
+ Arrays.toString((Object[]) e.getValue()));
+ } else if (e.getKey() == IPredicate.Annotations.FLAGS) {
+ sb.append(e.getKey() + "="
+ + Tuple.flagString((Integer) e.getValue()));
} else {
sb.append(e.getKey() + "=" + e.getValue());
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContextBase.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContextBase.java 2010-10-05 02:33:11 UTC (rev 3729)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContextBase.java 2010-10-05 19:57:25 UTC (rev 3730)
@@ -30,25 +30,15 @@
import java.util.concurrent.Executor;
import java.util.concurrent.ExecutorService;
-import org.apache.log4j.Logger;
-
import com.bigdata.bop.engine.QueryEngine;
-import com.bigdata.btree.IIndex;
import com.bigdata.btree.ILocalBTreeView;
-import com.bigdata.btree.IRangeQuery;
import com.bigdata.journal.IIndexManager;
-import com.bigdata.journal.TimestampUtility;
-import com.bigdata.relation.AbstractRelation;
import com.bigdata.relation.IRelation;
-import com.bigdata.relation.accesspath.AccessPath;
import com.bigdata.relation.accesspath.IAccessPath;
import com.bigdata.relation.locator.IResourceLocator;
import com.bigdata.relation.rule.IRule;
-import com.bigdata.relation.rule.ISolutionExpander;
import com.bigdata.relation.rule.eval.IJoinNexus;
-import com.bigdata.service.DataService;
import com.bigdata.service.IBigdataFederation;
-import com.bigdata.striterator.IKeyOrder;
/**
* Base class for the bigdata operation evaluation context (NOT serializable).
@@ -58,11 +48,16 @@
*/
public class BOpContextBase {
- static private final transient Logger log = Logger.getLogger(BOpContextBase.class);
+// static private final transient Logger log = Logger.getLogger(BOpContextBase.class);
-// private final QueryEngine queryEngine;
-
+ /**
+ * The federation iff running in scale-out.
+ */
private final IBigdataFederation<?> fed;
+
+ /**
+ * The <strong>local</strong> index manager.
+ */
private final IIndexManager indexManager;
/**
@@ -104,8 +99,11 @@
/**
* Core constructor.
+ *
* @param fed
+ * The federation iff running in scale-out.
* @param indexManager
+ * The <strong>local</strong> index manager.
*/
public BOpContextBase(final IBigdataFederation<?> fed,
final IIndexManager indexManager) {
@@ -232,7 +230,7 @@
* conversion is done. It has much of the same logic (this also
* handles remote access paths now).
*
- * @todo Support mutable relation views.
+ * @todo Support mutable relation views (no - just fix truth maintenance).
*/
// @SuppressWarnings("unchecked")
public <E> IAccessPath<E> getAccessPath(final IRelation<E> relation,
@@ -244,147 +242,149 @@
if (predicate == null)
throw new IllegalArgumentException();
- /*
- * FIXME This should be as assigned by the query planner so the query is
- * fully declarative.
- */
- final IKeyOrder<E> keyOrder;
- {
- final IKeyOrder<E> tmp = predicate.getKeyOrder();
- if (tmp != null) {
- // use the specified index.
- keyOrder = tmp;
- } else {
- // ask the relation for the best index.
- keyOrder = relation.getKeyOrder(predicate);
- }
- }
+ return relation.getAccessPath(indexManager/* localIndexManager */,
+ relation.getKeyOrder(predicate), predicate);
- if (keyOrder == null)
- throw new RuntimeException("No access path: " + predicate);
+// /*
+// * Note: ALWAYS use the "perfect" index.
+// */
+// final IKeyOrder<E> keyOrder = relation.getKeyOrder(predicate);
+//// {
+//// final IKeyOrder<E> tmp = predicate.getKeyOrder();
+//// if (tmp != null) {
+//// // use the specified index.
+//// keyOrder = tmp;
+//// } else {
+//// // ask the relation for the best index.
+//// keyOrder = relation.getKeyOrder(predicate);
+//// }
+//// }
+////
+//// if (keyOrder == null)
+//// throw new RuntimeException("No access path: " + predicate);
+//
+// final int partitionId = predicate.getPartitionId();
+//
+// final long timestamp = (Long) predicate
+// .getRequiredProperty(BOp.Annotations.TIMESTAMP);
+//
+// final int flags = predicate.getProperty(
+// IPredicate.Annotations.FLAGS,
+// IPredicate.Annotations.DEFAULT_FLAGS)
+// | (TimestampUtility.isReadOnly(timestamp) ? IRangeQuery.READONLY
+// : 0);
+//
+// final int chunkOfChunksCapacity = predicate.getProperty(
+// BufferAnnotations.CHUNK_OF_CHUNKS_CAPACITY,
+// BufferAnnotations.DEFAULT_CHUNK_OF_CHUNKS_CAPACITY);
+//
+// final int chunkCapacity = predicate.getProperty(
+// BufferAnnotations.CHUNK_CAPACITY,
+// BufferAnnotations.DEFAULT_CHUNK_CAPACITY);
+//
+// final int fullyBufferedReadThreshold = predicate.getProperty(
+// IPredicate.Annotations.FULLY_BUFFERED_READ_THRESHOLD,
+// IPredicate.Annotations.DEFAULT_FULLY_BUFFERED_READ_THRESHOLD);
+//
+// if (partitionId != -1) {
+//
+// /*
+// * Note: This handles a read against a local index partition. For
+// * scale-out, the [indexManager] will be the data service's local
+// * index manager.
+// *
+// * Note: Expanders ARE NOT applied in this code path. Expanders
+// * require a total view of the relation, which is not available
+// * during scale-out pipeline joins. Likewise, the [backchain]
+// * property will be ignored since it is handled by an expander.
+// *
+// * @todo Replace this with IRelation#getAccessPathForIndexPartition()
+// */
+//// return ((AbstractRelation<?>) relation)
+//// .getAccessPathForIndexPartition(indexManager,
+//// (IPredicate) predicate);
+//
+// /*
+// * @todo This is an error since expanders are currently ignored on
+// * shard-wise access paths. While it is possible to enable expanders
+// * for shard-wise access paths.
+// */
+// if (predicate.getSolutionExpander() != null)
+// throw new IllegalArgumentException();
+//
+// final String namespace = relation.getNamespace();//predicate.getOnlyRelationName();
+//
+// // The name of the desired index partition.
+// final String name = DataService.getIndexPartitionName(namespace
+// + "." + keyOrder.getIndexName(), partitionId);
+//
+// // MUST be a local index view.
+// final ILocalBTreeView ndx = (ILocalBTreeView) indexManager
+// .getIndex(name, timestamp);
+//
+// return new AccessPath<E>(relation, indexManager, timestamp,
+// predicate, keyOrder, ndx, flags, chunkOfChunksCapacity,
+// chunkCapacity, fullyBufferedReadThreshold).init();
+//
+// }
+//
+//// accessPath = relation.getAccessPath((IPredicate) predicate);
+//
+// // Decide on a local or remote view of the index.
+// final IIndexManager indexManager;
+// if (predicate.isRemoteAccessPath()) {
+// // use federation in scale-out for a remote access path.
+// indexManager = fed != null ? fed : this.indexManager;
+// } else {
+// indexManager = this.indexManager;
+// }
+//
+// // Obtain the index.
+// final String fqn = AbstractRelation.getFQN(relation, keyOrder);
+// final IIndex ndx = AbstractRelation.getIndex(indexManager, fqn, timestamp);
+//
+// if (ndx == null) {
+//
+// throw new IllegalArgumentException("no index? relation="
+// + relation.getNamespace() + ", timestamp=" + timestamp
+// + ", keyOrder=" + keyOrder + ", pred=" + predicate
+// + ", indexManager=" + getIndexManager());
+//
+// }
+//
+// // Obtain the access path for that relation and index.
+// final IAccessPath<E> accessPath = ((AbstractRelation<E>) relation)
+// .newAccessPath(relation, indexManager, timestamp, predicate,
+// keyOrder, ndx, flags, chunkOfChunksCapacity,
+// chunkCapacity, fullyBufferedReadThreshold);
+//
+// // optionally wrap with an expander pattern.
+// return expander(predicate, accessPath);
- final int partitionId = predicate.getPartitionId();
-
- final long timestamp = (Long) predicate
- .getRequiredProperty(BOp.Annotations.TIMESTAMP);
-
- final int flags = predicate.getProperty(
- IPredicate.Annotations.FLAGS,
- IPredicate.Annotations.DEFAULT_FLAGS)
- | (TimestampUtility.isReadOnly(timestamp) ? IRangeQuery.READONLY
- : 0);
-
- final int chunkOfChunksCapacity = predicate.getProperty(
- BufferAnnotations.CHUNK_OF_CHUNKS_CAPACITY,
- BufferAnnotations.DEFAULT_CHUNK_OF_CHUNKS_CAPACITY);
-
- final int chunkCapacity = predicate.getProperty(
- BufferAnnotations.CHUNK_CAPACITY,
- BufferAnnotations.DEFAULT_CHUNK_CAPACITY);
-
- final int fullyBufferedReadThreshold = predicate.getProperty(
- IPredicate.Annotations.FULLY_BUFFERED_READ_THRESHOLD,
- IPredicate.Annotations.DEFAULT_FULLY_BUFFERED_READ_THRESHOLD);
-
- if (partitionId != -1) {
-
- /*
- * Note: This handles a read against a local index partition. For
- * scale-out, the [indexManager] will be the data service's local
- * index manager.
- *
- * Note: Expanders ARE NOT applied in this code path. Expanders
- * require a total view of the relation, which is not available
- * during scale-out pipeline joins. Likewise, the [backchain]
- * property will be ignored since it is handled by an expander.
- *
- * @todo Replace this with IRelation#getAccessPathForIndexPartition()
- */
-// return ((AbstractRelation<?>) relation)
-// .getAccessPathForIndexPartition(indexManager,
-// (IPredicate) predicate);
-
- /*
- * @todo This is an error since expanders are currently ignored on
- * shard-wise access paths. While it is possible to enable expanders
- * for shard-wise access paths.
- */
- if (predicate.getSolutionExpander() != null)
- throw new IllegalArgumentException();
-
- final String namespace = relation.getNamespace();//predicate.getOnlyRelationName();
-
- // The name of the desired index partition.
- final String name = DataService.getIndexPartitionName(namespace
- + "." + keyOrder.getIndexName(), partitionId);
-
- // MUST be a local index view.
- final ILocalBTreeView ndx = (ILocalBTreeView) indexManager
- .getIndex(name, timestamp);
-
- return new AccessPath<E>(relation, indexManager, timestamp,
- predicate, keyOrder, ndx, flags, chunkOfChunksCapacity,
- chunkCapacity, fullyBufferedReadThreshold).init();
-
- }
-
-// accessPath = relation.getAccessPath((IPredicate) predicate);
-
- // Decide on a local or remote view of the index.
- final IIndexManager indexManager;
- if (predicate.isRemoteAccessPath()) {
- // use federation in scale-out for a remote access path.
- indexManager = fed != null ? fed : this.indexManager;
- } else {
- indexManager = this.indexManager;
- }
-
- // Obtain the index.
- final String fqn = AbstractRelation.getFQN(relation, keyOrder);
- final IIndex ndx = AbstractRelation.getIndex(indexManager, fqn, timestamp);
-
- if (ndx == null) {
-
- throw new IllegalArgumentException("no index? relation="
- + relation.getNamespace() + ", timestamp=" + timestamp
- + ", keyOrder=" + keyOrder + ", pred=" + predicate
- + ", indexManager=" + getIndexManager());
-
- }
-
- // Obtain the access path for that relation and index.
- final IAccessPath<E> accessPath = ((AbstractRelation<E>) relation)
- .newAccessPath(relation, indexManager, timestamp, predicate,
- keyOrder, ndx, flags, chunkOfChunksCapacity,
- chunkCapacity, fullyBufferedReadThreshold);
-
- // optionally wrap with an expander pattern.
- return expander(predicate, accessPath);
-
}
- /**
- * Optionally wrap with an expander pattern.
- *
- * @param predicate
- * @param accessPath
- * @return
- * @param <E>
- */
- private <E> IAccessPath<E> expander(final IPredicate<E> predicate,
- final IAccessPath<E> accessPath) {
+// /**
+// * Optionally wrap with an expander pattern.
+// *
+// * @param predicate
+// * @param accessPath
+// * @return
+// * @param <E>
+// */
+// private <E> IAccessPath<E> expander(final IPredicate<E> predicate,
+// final IAccessPath<E> accessPath) {
+//
+// final ISolutionExpander<E> expander = predicate.getSolutionExpander();
+//
+// if (expander != null) {
+//
+// // allow the predicate to wrap the access path
+// return expander.getAccessPath(accessPath);
+//
+// }
+//
+// return accessPath;
+//
+// }
- final ISolutionExpander<E> expander = predicate.getSolutionExpander();
-
- if (expander != null) {
-
- // allow the predicate to wrap the access path
- return expander.getAccessPath(accessPath);
-
- }
-
- return accessPath;
-
- }
-
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java 2010-10-05 02:33:11 UTC (rev 3729)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java 2010-10-05 19:57:25 UTC (rev 3730)
@@ -413,7 +413,8 @@
* trees whose sinks target a descendant, which is another way
* to create a loop.
*/
- throw new DuplicateBOpException(t.toString());
+ throw new DuplicateBOpException("id=" + t.getId() + ", root="
+ + toString(op));
}
}
// wrap to ensure immutable and thread-safe.
@@ -528,4 +529,93 @@
} // toArray()
+ /**
+ * Pretty print a bop.
+ *
+ * @param bop
+ * The bop.
+ *
+ * @return The formatted representation.
+ */
+ public static String toString(final BOp bop) {
+
+ final StringBuilder sb = new StringBuilder();
+
+ toString(bop, sb, 0);
+
+ // chop off the last \n
+ sb.setLength(sb.length() - 1);
+
+ return sb.toString();
+
+ }
+
+ private static void toString(final BOp bop, final StringBuilder sb,
+ final int indent) {
+
+ sb.append(indent(indent)).append(
+ bop == null ? "<null>" : bop.toString()).append('\n');
+
+ if (bop == null)
+ return;
+
+ for (BOp arg : bop.args()) {
+
+ if (arg.arity() > 0) {
+
+ toString(arg, sb, indent+1);
+
+ }
+
+ }
+
+ }
+
+ /**
+ * Returns a string that may be used to indent a dump of the nodes in
+ * the tree.
+ *
+ * @param height
+ * The height.
+ *
+ * @return A string suitable for indent at that height.
+ */
+ private static String indent(final int height) {
+
+ if( height == -1 ) {
+
+ // The height is not defined.
+
+ return "";
+
+ }
+
+ return ws.substring(0, height * 4);
+
+ }
+
+ private static final transient String ws = " ";
+
+// /**
+// * Verify that all bops from the identified <i>startId</i> to the root are
+// * {@link PipelineOp}s and have an assigned {@link BOp.Annotations#BOP_ID}.
+// * This is required in order for us to be able to target messages to those
+// * operators.
+// *
+// * @param startId
+// * The {@link BOp.Annotations#BOP_ID} at which the query will
+// * start. This is typically the left-most descendant.
+// * @param root
+// * The root of the operator tree.
+// *
+// * @throws RuntimeException
+// * if the operator tree does not meet any of the criteria for
+// * pipeline evaluation.
+// */
+// public static void verifyPipline(final int startId, final BOp root) {
+//
+// throw new UnsupportedOperationException();
+//
+// }
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BufferAnnotations.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BufferAnnotations.java 2010-10-05 02:33:11 UTC (rev 3729)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BufferAnnotations.java 2010-10-05 19:57:25 UTC (rev 3730)
@@ -55,7 +55,7 @@
int DEFAULT_CHUNK_OF_CHUNKS_CAPACITY = 100;
/**
- * Sets the capacity of the {@link IBuffer}s used to accumulate a chunk of
+ * Sets the capacity of the {@link IBuffer}[]s used to accumulate a chunk of
* {@link IBindingSet}s (default {@value #CHUNK_CAPACITY}). Partial chunks
* may be automatically combined into full chunks.
*
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Constant.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Constant.java 2010-10-05 02:33:11 UTC (rev 3729)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Constant.java 2010-10-05 19:57:25 UTC (rev 3730)
@@ -29,7 +29,7 @@
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
* @version $Id$
*/
-final public class Constant<E> extends BOpBase implements IConstant<E> {
+final public class Constant<E> extends ImmutableBOp implements IConstant<E> {
/**
*
@@ -84,6 +84,15 @@
}
+ /**
+ * Clone is overridden to reduce heap churn.
+ */
+ final public Constant<E> clone() {
+
+ return this;
+
+ }
+
public String toString() {
return value.toString();
@@ -150,5 +159,5 @@
throw new UnsupportedOperationException();
}
-
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPredicate.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPredicate.java 2010-10-05 02:33:11 UTC (rev 3729)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPredicate.java 2010-10-05 19:57:25 UTC (rev 3730)
@@ -45,7 +45,7 @@
import com.bigdata.relation.accesspath.ElementFilter;
import com.bigdata.relation.accesspath.IAccessPath;
import com.bigdata.relation.rule.IRule;
-import com.bigdata.relation.rule.ISolutionExpander;
+import com.bigdata.relation.rule.IAccessPathExpander;
import com.bigdata.relation.rule.eval.IEvaluationPlan;
import com.bigdata.relation.rule.eval.pipeline.JoinMasterTask;
import com.bigdata.striterator.IKeyOrder;
@@ -81,15 +81,15 @@
*/
String RELATION_NAME = "relationName";
- /**
- * The {@link IKeyOrder} which will be used to read on the relation.
- * <p>
- * Note: This is generally assigned by the query optimizer. The query
- * optimizer considers the possible indices for each {@link IPredicate}
- * in a query and the possible join orders and then specifies the order
- * of the joins and the index to use for each join.
- */
- String KEY_ORDER = "keyOrder";
+// /**
+// * The {@link IKeyOrder} which will be used to read on the relation.
+// * <p>
+// * Note: This is generally assigned by the query optimizer. The query
+// * optimizer considers the possible indices for each {@link IPredicate}
+// * in a query and the possible join orders and then specifies the order
+// * of the joins and the index to use for each join.
+// */
+// String KEY_ORDER = "keyOrder";
/**
* <code>true</code> iff the predicate is optional (the right operand of
@@ -160,17 +160,28 @@
* Access path expander pattern. This allows you to wrap or replace the
* {@link IAccessPath}.
* <p>
- * Note: You MUST be extremely careful when using this feature in
- * scale-out. Access path expanders in scale-out are logically
- * consistent with used with a {@link #REMOTE_ACCESS_PATH}, but remote
- * access paths often lack the performance of a local access path.
+ * Note: Access path expanders in scale-out are logically consistent
+ * when used with a {@link #REMOTE_ACCESS_PATH}. However, remote access
+ * paths often lack the performance of a local access path. In order for
+ * the expander to be consistent with a local access path it MUST NOT
+ * rewrite the predicate in such a manner as to read on data onto found
+ * on the shard onto which the predicate was mapped during query
+ * evaluation.
* <p>
- * In order for the expander to be consistent with a local access path
- * it MUST NOT rewrite the predicate in such a manner as to read on data
- * onto found on the shard onto which the predicate was mapped during
- * query evaluation.
+ * In general, scale-out query depends on binding sets being mapped onto
+ * the shards against which they will read or write. When an expander
+ * changes the bindings on an {@link IPredicate}, it typically changes
+ * the access path which will be used. This is only supported when the
+ * access path is {@link #REMOTE_ACCESS_PATH}.
+ * <p>
+ * Note: If an expander generates nested {@link IAccessPath}s then it
+ * typically must strip off both the {@link #ACCESS_PATH_EXPANDER} and
+ * the {@link #ACCESS_PATH_EXPANDER} from the {@link IPredicate} before
+ * generating the inner {@link IAccessPath} and then layer the
+ * {@link #ACCESS_PATH_FILTER} back onto the expanded visitation
+ * pattern.
*
- * @see ISolutionExpander
+ * @see IAccessPathExpander
*/
String ACCESS_PATH_EXPANDER = "accessPathExpander";
@@ -335,7 +346,7 @@
* enforced using {@link IConstraint}s on the {@link IRule}.
* <p>
* More control over the behavior of optionals may be gained through the use
- * of an {@link ISolutionExpander} pattern.
+ * of an {@link IAccessPathExpander} pattern.
*
* @return <code>true</code> iff this predicate is optional when evaluating
* a JOIN.
@@ -348,14 +359,14 @@
* Returns the object that may be used to selectively override the
* evaluation of the predicate.
*
- * @return The {@link ISolutionExpander}.
+ * @return The {@link IAccessPathExpander}.
*
* @see Annotations#ACCESS_PATH_EXPANDER
*
- * @todo replace with {@link ISolutionExpander#getAccessPath(IAccessPath)},
- * which is the only method declared by {@link ISolutionExpander}.
+ * @todo Replace with {@link IAccessPathExpander#getAccessPath(IAccessPath)}
+ * , which is the only method declared by {@link IAccessPathExpander}?
*/
- public ISolutionExpander<E> getSolutionExpander();
+ public IAccessPathExpander<E> getAccessPathExpander();
// /**
// * An optional constraint on the visitable elements.
@@ -386,32 +397,32 @@
*/
public IFilter getAccessPathFilter();
- /**
- * Return the {@link IKeyOrder} assigned to this {@link IPredicate} by the
- * query optimizer.
- *
- * @return The assigned {@link IKeyOrder} or <code>null</code> if the query
- * optimizer has not assigned the {@link IKeyOrder} yet.
- *
- * @see Annotations#KEY_ORDER
- */
- public IKeyOrder<E> getKeyOrder();
+// /**
+// * Return the {@link IKeyOrder} assigned to this {@link IPredicate} by the
+// * query optimizer.
+// *
+// * @return The assigned {@link IKeyOrder} or <code>null</code> if the query
+// * optimizer has not assigned the {@link IKeyOrder} yet.
+// *
+// * @see Annotations#KEY_ORDER
+// */
+// public IKeyOrder<E> getKeyOrder();
+//
+// /**
+// * Set the {@link IKeyOrder} annotation on the {@link IPredicate}, returning
+// * a new {@link IPredicate} in which the annotation takes on the given
+// * binding.
+// *
+// * @param keyOrder
+// * The {@link IKeyOrder}.
+// *
+// * @return The new {@link IPredicate}.
+// *
+// * @see Annotations#KEY_ORDER
+// */
+// public IPredicate<E> setKeyOrder(final IKeyOrder<E> keyOrder);
/**
- * Set the {@link IKeyOrder} annotation on the {@link IPredicate}, returning
- * a new {@link IPredicate} in which the annotation takes on the given
- * binding.
- *
- * @param keyOrder
- * The {@link IKeyOrder}.
- *
- * @return The new {@link IPredicate}.
- *
- * @see Annotations#KEY_ORDER
- */
- public IPredicate<E> setKeyOrder(final IKeyOrder<E> keyOrder);
-
- /**
* Figure out if all positions in the predicate which are required to form
* the key for this access path are bound in the predicate.
*/
@@ -487,6 +498,9 @@
/**
* Return a new instance in which all occurrences of the given variable have
* been replaced by the specified constant.
+ * <p>
+ * Note: The optimal {@link IKeyOrder} often changes when binding a variable
+ * on a predicate.
*
* @param var
* The variable.
@@ -504,6 +518,9 @@
/**
* Return a new instance in which all occurrences of the variable appearing
* in the binding set have been replaced by their bound values.
+ * <p>
+ * Note: The optimal {@link IKeyOrder} often changes when binding a variable
+ * on a predicate.
*
* @param bindingSet
* The binding set.
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ImmutableBOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ImmutableBOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ImmutableBOp.java 2010-10-05 19:57:25 UTC (rev 3730)
@@ -0,0 +1,76 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Oct 5, 2010
+ */
+
+package com.bigdata.bop;
+
+import java.util.Map;
+
+/**
+ * Base class for immutable operators such as {@link Var} and {@link Constant}.
+ * These operators do not deep copy their data and do not permit decoration with
+ * annotations.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+abstract public class ImmutableBOp extends BOpBase {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * @param op
+ */
+ public ImmutableBOp(BOpBase op) {
+ super(op);
+ }
+
+ /**
+ * @param args
+ * @param annotations
+ */
+ public ImmutableBOp(BOp[] args, Map<String, Object> annotations) {
+ super(args, annotations);
+ }
+
+ /*
+ * Overrides for the copy-on-write mutation API.
+ */
+
+ @Override
+ protected Object _setProperty(String name,Object value) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ protected void _clearProperty(String name) {
+ throw new UnsupportedOperationException();
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ImmutableBOp.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Var.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Var.java 2010-10-05 02:33:11 UTC (rev 3729)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Var.java 2010-10-05 19:57:25 UTC (rev 3730)
@@ -32,7 +32,7 @@
* limit the binding patterns across the rule. (In fact, variables can be
* "named" by their index into the binding set for most purposes.)
*/
-final public class Var<E> extends BOpBase implements IVariable<E>,
+final public class Var<E> extends ImmutableBOp implements IVariable<E>,
Comparable<IVariable<E>> {
private static final long serialVersionUID = -7100443208125002485L;
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java 2010-10-05 02:33:11 UTC (rev 3729)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java 2010-10-05 19:57:25 UTC (rev 3730)
@@ -28,7 +28,6 @@
package com.bigdata.bop.ap;
-import java.util.Iterator;
import java.util.Map;
import com.bigdata.bop.AbstractAccessPathOp;
@@ -44,11 +43,11 @@
import com.bigdata.bop.NV;
import com.bigdata.relation.accesspath.ElementFilter;
import com.bigdata.relation.accesspath.IElementFilter;
-import com.bigdata.relation.rule.ISolutionExpander;
+import com.bigdata.relation.rule.IAccessPathExpander;
import com.bigdata.striterator.IKeyOrder;
-import cutthecrap.utils.striterators.FilterBase;
import cutthecrap.utils.striterators.IFilter;
+import cutthecrap.utils.striterators.NOPFilter;
/**
* A generic implementation of an immutable {@link IPredicate}.
@@ -140,7 +139,7 @@
public Predicate(final IVariableOrConstant<?>[] values,
final String relationName, final int partitionId,
final boolean optional, final IElementFilter<E> constraint,
- final ISolutionExpander<E> expander, final long timestamp) {
+ final IAccessPathExpander<E> expander, final long timestamp) {
this(values, NV.asMap(new NV[] {//
new NV(Annotations.RELATION_NAME,new String[]{relationName}),//
@@ -258,9 +257,9 @@
}
@SuppressWarnings("unchecked")
- final public ISolutionExpander<E> getSolutionExpander() {
+ final public IAccessPathExpander<E> getAccessPathExpander() {
- return (ISolutionExpander<E>) getProperty(Annotations.ACCESS_PATH_EXPANDER);
+ return (IAccessPathExpander<E>) getProperty(Annotations.ACCESS_PATH_EXPANDER);
}
@@ -325,18 +324,12 @@
final int arity = arity();
+// boolean modified = false;
+
for (int i = 0; i < arity; i++) {
final IVariableOrConstant<?> t = (IVariableOrConstant<?>) get(i);
-// if (t == null) {
-// /*
-// * Note: t != null handles the case where the [c] position of an
-// * SPO is allowed to be null.
-// */
-// continue;
-// }
-
if (t.isConstant())
continue;
@@ -350,6 +343,8 @@
}
tmp.set(i, val.clone());
+
+// modified = true;
}
@@ -363,22 +358,22 @@
}
- @SuppressWarnings("unchecked")
- public IKeyOrder<E> getKeyOrder() {
-
- return (IKeyOrder<E>) getProperty(Annotations.KEY_ORDER);
-
- }
-
- public Predicate<E> setKeyOrder(final IKeyOrder<E> keyOrder) {
-
- final Predicate<E> tmp = this.clone();
-
- tmp._setProperty(Annotations.KEY_ORDER, keyOrder);
-
- return tmp;
-
- }
+// @SuppressWarnings("unchecked")
+// public IKeyOrder<E> getKeyOrder() {
+//
+// return (IKeyOrder<E>) getProperty(Annotations.KEY_ORDER);
+//
+// }
+//
+// public Predicate<E> setKeyOrder(final IKeyOrder<E> keyOrder) {
+//
+// final Predicate<E> tmp = this.clone();
+//
+// tmp._setProperty(Annotations.KEY_ORDER, keyOrder);
+//
+// return tmp;
+//
+// }
@SuppressWarnings("unchecked")
public Predicate<E> clone() {
@@ -486,38 +481,23 @@
/*
* Wrap the filter.
*/
- _setProperty(name, new FilterBase() {
-
- @Override
- protected Iterator filterOnce(Iterator src, Object context) {
- return src;
- }
-
- }.addFilter(current).addFilter(filter));
+ _setProperty(name, new NOPFilter().addFilter(current)
+ .addFilter(filter));
}
}
-
+
/**
- * Strips off the named annotations.
- *
- * @param names
- * The annotations to be removed.
- *
- * @return A new predicate in which the specified annotations do not appear.
+ * Strengthened return type.
+ * <p>
+ * {@inheritDoc}
*/
+ @SuppressWarnings("unchecked")
+ @Override
public Predicate<E> clearAnnotations(final String[] names) {
-
- final Predicate<E> tmp = this.clone();
-
- for(String name : names) {
-
- tmp._clearProperty(name);
-
- }
-
- return tmp;
+ return (Predicate<E>) super.clearAnnotations(names);
+
}
public String toString() {
@@ -533,7 +513,15 @@
final StringBuilder sb = new StringBuilder();
sb.append(getClass().getName());
+
+ final Integer bopId = (Integer) getProperty(Annotations.BOP_ID);
+ if (bopId != null) {
+
+ sb.append("[" + bopId + "]");
+
+ }
+
sb.append("(");
for (int i = 0; i < arity; i++) {
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/ScanCostReport.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/ScanCostReport.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/ScanCostReport.java 2010-10-05 19:57:25 UTC (rev 3730)
@@ -0,0 +1,76 @@
+package com.bigdata.bop.cost;
+
+
+/**
+ * A report on the expected cost of an index key range scan.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan
+ * Thompson</a>
+ * @version $Id$
+ */
+public class ScanCostReport {
+
+ /**
+ * The fast range count.
+ */
+ public final long rangeCount;
+
+ /**
+ * The #of index partitions in the scan.
+ */
+ public final long shardCount;
+
+ /**
+ * The expected cost of the scan (milliseconds).
+ */
+ public final double cost;
+
+ /**
+ *
+ * @param rangeCount
+ * The fast range count.
+ * @param cost
+ * The expected cost for the scan (milliseconds).
+ */
+ public ScanCostReport(final long rangeCount, final double cost) {
+
+ this.rangeCount = rangeCount;
+
+ this.shardCount = 1;
+
+ this.cost = cost;
+
+ }
+
+ /**
+ *
+ * @param rangeCount
+ * The fast range count.
+ * @param shardCount
+ * The #of index partitions in the scan.
+ * @param cost
+ * The expected cost for the scan (milliseconds).
+ */
+ public ScanCostReport(final long rangeCount, final long shardCount,
+ final double cost) {
+
+ this.rangeCount = rangeCount;
+
+ this.shardCount = shardCount;
+
+ this.cost = cost;
+
+ }
+
+ /**
+ * Human readable representation.
+ */
+ public String toString() {
+ return super.toString() + //
+ "{rangeCount=" + rangeCount + //
+ ",shardCount=" + shardCount + //
+ ",cost=" + cost + //
+ "}";
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/ScanCostReport.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/SubqueryCostReport.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/SubqueryCostReport.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/SubqueryCostReport.java 2010-10-05 19:57:25 UTC (rev 3730)
@@ -0,0 +1,74 @@
+package com.bigdata.bop.cost;
+
+import java.io.Serializable;
+
+/**
+ * Subquery cost report.
+ */
+public class SubqueryCostReport implements Serializable {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ /** The #of graphs against which subqueries will be issued. */
+ public final int ngraphs;
+
+ /** The #of samples to be taken. */
+ public final int limit;
+
+ /** The #of samples taken. */
+ public final int nsamples;
+
+ /**
+ * An estimated range count based on the samples and adjusted for the
+ * #of graphs.
+ */
+ public final long rangeCount;
+
+ /**
+ * An estimated cost (latency in milliseconds) based on the samples and
+ * adjusted for the #of graphs.
+ */
+ public final double subqueryCost;
+
+ /**
+ *
+ * @param ngraphs
+ * The #of graphs against which subqueries will be issued.
+ * @param limit
+ * The #of samples to be taken.
+ * @param nsamples
+ * The #of samples taken.
+ * @param rangeCount
+ * An estimated range count based on the samples and adjusted
+ * for the #of graphs.
+ * @param subqueryCost
+ * An estimated cost (latency in milliseconds) based on the
+ * samples and adjusted for the #of graphs.
+ */
+ public SubqueryCostReport(final int ngraphs, final int limit,
+ final int nsamples, final long rangeCount,
+ final double subqueryCost) {
+ this.ngraphs = ngraphs;
+ this.limit = limit;
+ this.nsamples = nsamples;
+ this.rangeCount = rangeCount;
+ this.subqueryCost = subqueryCost;
+ }
+
+ /**
+ * Human readable representation.
+ */
+ public String toString() {
+ return super.toString() + //
+ "{ngraphs=" + ngraphs + //
+ ",limit=" + limit + //
+ ",nsamples=" + nsamples + //
+ ",rangeCount=" + rangeCount + //
+ ",subqueryCost=" + subqueryCost + //
+ "}";
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/SubqueryCostReport.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java 2010-10-05 02:33:11 UTC (rev 3729)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java 2010-10-05 19:57:25 UTC (rev 3730)
@@ -46,8 +46,8 @@
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.PipelineOp;
import com.bigdata.bop.ap.Predicate;
-import com.bigdata.bop.cost.BTreeCostModel;
import com.bigdata.bop.cost.DiskCostModel;
+import com.bigdata.bop.cost.ScanCostReport;
import com.bigdata.btree.AbstractBTree;
import com.bigdata.btree.BTree;
import com.bigdata.btree.IBTreeStatistics;
@@ -443,6 +443,10 @@
// pool).
log.warn("Dropping chunk: queryId=" + queryId);
continue;
+ } catch (Throwable ex) {
+ // halt that query.
+ q.halt(ex);
+ continue;
}
} catch (InterruptedException e) {
...
[truncated message content] |
