Menu
▾
▴
[Bigdata-commit] SF.net SVN: bigdata:[3508] branches/QUADS_QUERY_BRANCH/bigdata
|
From: <tho...@us...> - 2010-09-05 17:02:44
|
Revision: 3508
http://bigdata.svn.sourceforge.net/bigdata/?rev=3508&view=rev
Author: thompsonbry
Date: 2010-09-05 17:02:34 +0000 (Sun, 05 Sep 2010)
Log Message:
-----------
Refactored the code to map binding sets over shards and made it more efficient (it reuses the same logic that is used by the scale-out index views).
Added IRelation#getKeyOrders() which brings us another step towards DDL support.
Added an in-memory binding set sort operator and an implementation for ordering binding sets for SPARQL which leverages the openrdf ValueCompator.
Added an "INSERT" operator to write on an index.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Constant.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/DistinctBindingSetOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/package.html
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/AbstractRelation.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/IMutableRelation.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/IRelation.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/RelationFusedView.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/accesspath/AbstractUnsynchronizedArrayBuffer.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/accesspath/ThreadLocalBufferFactory.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/IQueryOptions.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/QueryOptions.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/NestedSubqueryWithJoinThreadsTask.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/RuleStats.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/pipeline/JoinMasterTask.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/search/FullTextIndex.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/AbstractScaleOutFederation.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/ndx/AbstractSplitter.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/striterator/AbstractKeyOrder.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/striterator/IKeyOrder.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/aggregation/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/aggregation/TestDistinctBindingSets.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/ap/R.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/bset/TestConditionalRoutingOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/BOpStats.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/RunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestMapBindingSetsOverShards.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/join/TestPipelineJoin.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/relation/locator/TestDefaultResourceLocator.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/relation/rule/TestSlice.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/service/AbstractEmbeddedFederationTestCase.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/lexicon/LexiconRelation.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/magic/MagicIndexWriteProc.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/magic/MagicIndexWriter.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/magic/MagicKeyOrder.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/magic/MagicRelation.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/magic/MagicTupleSerializer.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPORelation.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestSlice.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl.java
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ComparatorOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISlice.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISortOrder.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/MemorySortOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/Slice.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOrder.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SparqlBindingSetComparatorOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/InsertOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/package.html
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/aggregation/TestMemorySortOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/aggregation/TestSPARQLBindingSetComparatorOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/FederatedQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/MapBindingSetsOverShardsBuffer.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestFederatedQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/mutation/
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/mutation/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/mutation/TestDelete.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/mutation/TestInsert.java
Removed Paths:
-------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ConstantEval.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/ISlice.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/ISortOrder.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/Slice.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/SortOrder.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/aggregation/TestSortBindingSets.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/MapBindingSetsOverShards.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOp.java 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOp.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -114,9 +114,22 @@
* @return The value of the annotation and <code>null</code> if the
* annotation is not bound.
*/
- Object getProperty(final String name);
+ <T> T getProperty(final String name);
/**
+ * Return the value of the named annotation.
+ *
+ * @param name
+ * The name of the annotation.
+ *
+ * @return The value of the annotation.
+ *
+ * @throws IllegalArgumentException
+ * if the named annotation is not bound.
+ */
+ <T> T getRequiredProperty(final String name);
+
+ /**
* Deep copy clone of the operator.
*/
BOp clone();
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -299,12 +299,23 @@
}
- public Object getProperty(final String name) {
+ public <T> T getProperty(final String name) {
- return annotations.get(name);
+ return (T) annotations.get(name);
}
+ public <T> T getRequiredProperty(final String name) {
+
+ final T tmp = (T) annotations.get(name);
+
+ if (tmp == null)
+ throw new IllegalArgumentException("Required property: " + name);
+
+ return tmp;
+
+ }
+
public String toString() {
final StringBuilder sb = new StringBuilder();
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -27,6 +27,8 @@
*/
package com.bigdata.bop;
+import java.util.Iterator;
+
import org.apache.log4j.Logger;
import com.bigdata.bop.engine.BOpStats;
@@ -36,6 +38,7 @@
import com.bigdata.journal.IIndexManager;
import com.bigdata.journal.ITx;
import com.bigdata.journal.TimestampUtility;
+import com.bigdata.mdi.PartitionLocator;
import com.bigdata.relation.IRelation;
import com.bigdata.relation.accesspath.AccessPath;
import com.bigdata.relation.accesspath.IAccessPath;
@@ -44,8 +47,10 @@
import com.bigdata.relation.locator.IResourceLocator;
import com.bigdata.relation.rule.IRule;
import com.bigdata.relation.rule.eval.IJoinNexus;
+import com.bigdata.service.AbstractScaleOutFederation;
import com.bigdata.service.DataService;
import com.bigdata.service.IBigdataFederation;
+import com.bigdata.service.ndx.IClientIndex;
import com.bigdata.striterator.IKeyOrder;
/**
@@ -270,12 +275,100 @@
*/
public IRelation getReadRelation(final IPredicate<?> pred) {
+ /*
+ * @todo Cache the resource locator?
+ *
+ * @todo This should be using the federation as the index manager when
+ * locating a resource for scale-out, right? But s/o reads must use the
+ * local index manager when actually obtaining the index view for the
+ * relation.
+ */
return (IRelation) getIndexManager().getResourceLocator().locate(
pred.getOnlyRelationName(), getReadTimestamp());
}
/**
+ * Return a writable view of the relation.
+ *
+ * @param namespace
+ * The namespace of the relation.
+ *
+ * @return A writable view of the relation.
+ */
+ public IRelation getWriteRelation(final String namespace) {
+
+ /*
+ * @todo Cache the resource locator?
+ *
+ * @todo This should be using the federation as the index manager when
+ * locating a resource for scale-out, right? But s/o writes must use
+ * the local index manager when actually obtaining the index view for
+ * the relation.
+ */
+ return (IRelation) getIndexManager().getResourceLocator().locate(
+ namespace, getWriteTimestamp());
+
+ }
+
+ /**
+ * Return an mutable view of the specified index.
+ *
+ * @param <T>
+ * The generic type of the elements in the relation.
+ * @param relation
+ * The relation.
+ * @param keyOrder
+ * The key order for that index.
+ * @param partitionId
+ * The partition identifier and <code>-1</code> unless running
+ * against an {@link IBigdataFederation}.
+ *
+ * @return The mutable view of the index.
+ *
+ * @throws UnsupportedOperationException
+ * if there is an attempt to read on an index partition when the
+ * database is not an {@link IBigdataFederation} or when the
+ * database is an {@link IBigdataFederation} unless the index
+ * partition was specified.
+ */
+ public <T> ILocalBTreeView getMutableLocalIndexView(
+ final IRelation<T> relation, final IKeyOrder<T> keyOrder,
+ final int partitionId) {
+
+ final String namespace = relation.getNamespace();
+
+ final ILocalBTreeView ndx;
+
+ if (partitionId == -1) {
+
+ if(indexManager instanceof IBigdataFederation<?>)
+ throw new UnsupportedOperationException();
+
+ // The index is not partitioned.
+ ndx = (ILocalBTreeView) indexManager.getIndex(namespace + "."
+ + keyOrder.getIndexName(), getWriteTimestamp());
+
+ } else {
+
+ if(!(indexManager instanceof IBigdataFederation<?>))
+ throw new UnsupportedOperationException();
+
+ // The name of the desired index partition.
+ final String name = DataService.getIndexPartitionName(namespace
+ + "." + keyOrder.getIndexName(), partitionId);
+
+ // MUST be a local index view.
+ ndx = (ILocalBTreeView) indexManager.getIndex(name,
+ getWriteTimestamp());
+
+ }
+
+ return ndx;
+
+ }
+
+ /**
* Obtain an access path reading from relation for the specified predicate
* (from the tail of some rule).
* <p>
@@ -558,5 +651,56 @@
return true;
}
-
+
+/*
+ * I've replaced this with AbstractSplitter for the moment.
+ */
+// /**
+// * Return an iterator visiting the {@link PartitionLocator} for the index
+// * partitions from which an {@link IAccessPath} must read in order to
+// * materialize all elements which would be visited for that predicate.
+// *
+// * @param predicate
+// * The predicate on which the next stage in the pipeline must
+// * read, with whatever bindings already applied. This is used to
+// * discover the shard(s) which span the key range against which
+// * the access path must read.
+// *
+// * @return The iterator.
+// */
+// public Iterator<PartitionLocator> locatorScan(final IPredicate<?> predicate) {
+//
+// final long timestamp = getReadTimestamp();
+//
+// // Note: assumes that we are NOT using a view of two relations.
+// final IRelation<?> relation = (IRelation<?>) fed.getResourceLocator()
+// .locate(predicate.getOnlyRelationName(), timestamp);
+//
+// /*
+// * Find the best access path for the predicate for that relation.
+// *
+// * Note: All we really want is the [fromKey] and [toKey] for that
+// * predicate and index. This MUST NOT layer on expanders since the
+// * layering also hides the [fromKey] and [toKey].
+// */
+// @SuppressWarnings("unchecked")
+// final AccessPath<?> accessPath = (AccessPath<?>) relation
+// .getAccessPath((IPredicate) predicate);
+//
+// // Note: assumes scale-out (EDS or JDS).
+// final IClientIndex ndx = (IClientIndex) accessPath.getIndex();
+//
+// /*
+// * Note: could also be formed from relationName + "." +
+// * keyOrder.getIndexName(), which is cheaper unless the index metadata
+// * is cached.
+// */
+// final String name = ndx.getIndexMetadata().getName();
+//
+// return ((AbstractScaleOutFederation<?>) fed).locatorScan(name,
+// timestamp, accessPath.getFromKey(), accessPath.getToKey(),
+// false/* reverse */);
+//
+// }
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -30,12 +30,15 @@
import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
+import java.util.LinkedList;
+import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.log4j.Logger;
import com.bigdata.bop.BOp.Annotations;
+import com.bigdata.bop.engine.BOpStats;
import com.bigdata.btree.AbstractNode;
import cutthecrap.utils.striterators.Expander;
@@ -462,4 +465,65 @@
}
+
+ /**
+ * Combine chunks drawn from an iterator into a single chunk.
+ *
+ * @param itr
+ * The iterator
+ * @param stats
+ * {@link BOpStats#chunksIn} and {@link BOpStats#unitsIn} are
+ * updated.
+ *
+ * @return A single chunk containing all of the chunks visited by the
+ * iterator.
+ *
+ * @todo unit tests.
+ */
+ static public IBindingSet[] toArray(final Iterator<IBindingSet[]> itr,
+ final BOpStats stats) {
+
+ final List<IBindingSet[]> list = new LinkedList<IBindingSet[]>();
+
+ int nchunks = 0, nelements = 0;
+ {
+
+ while (itr.hasNext()) {
+
+ final IBindingSet[] a = itr.next();
+
+ list.add(a);
+
+ nchunks++;
+
+ nelements += a.length;
+
+ list.add(a);
+
+ }
+
+ stats.chunksIn.add(nchunks);
+ stats.unitsIn.add(nelements);
+
+ }
+
+ if (nchunks == 0) {
+ return new IBindingSet[0];
+ } else if (nchunks == 1) {
+ return list.get(0);
+ } else {
+ int n = 0;
+ final IBindingSet[] a = new IBindingSet[nelements];
+ final Iterator<IBindingSet[]> itr2 = list.iterator();
+ while (itr2.hasNext()) {
+ final IBindingSet[] t = itr2.next();
+ System.arraycopy(t/* src */, 0/* srcPos */, a/* dest */,
+ n/* destPos */, t.length/* length */);
+ n += t.length;
+ }
+ return a;
+ }
+
+ } // toArray()
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Constant.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Constant.java 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Constant.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -23,8 +23,6 @@
*/
package com.bigdata.bop;
-
-
/**
* A constant.
*
Deleted: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ConstantEval.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ConstantEval.java 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ConstantEval.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -1,19 +0,0 @@
-package com.bigdata.bop;
-
-/**
- * Evaluate a constant
- *
- * @author <a href="mailto:tho...@us...">Bryan
- * Thompson</a>
- * @param <E>
- */
-public interface ConstantEval<E> extends BOp {
-
- /**
- * Evaluate a constant.
- *
- * @return The value.
- */
- E eval();
-
-}
\ No newline at end of file
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ComparatorOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ComparatorOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ComparatorOp.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,77 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Sep 4, 2010
+ */
+
+package com.bigdata.bop.aggregation;
+
+import java.util.Comparator;
+import java.util.Map;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpBase;
+import com.bigdata.bop.IBindingSet;
+
+/**
+ * Base class for operators which impose a sort order on binding sets.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+abstract public class ComparatorOp extends BOpBase implements
+ Comparator<IBindingSet> {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ public interface Annotations extends BOp.Annotations {
+
+ /**
+ * An {@link ISortOrder}[] specifying the variables on which the sort
+ * will be imposed and the order (ascending or descending) for each
+ * variable.
+ */
+ String ORDER = ComparatorOp.class.getName() + ".order";
+
+ }
+
+ /**
+ * @param op
+ */
+ public ComparatorOp(BOpBase op) {
+ super(op);
+ }
+
+ /**
+ * @param args
+ * @param annotations
+ */
+ public ComparatorOp(BOp[] args, Map<String, Object> annotations) {
+ super(args, annotations);
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ComparatorOp.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/DistinctBindingSetOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/DistinctBindingSetOp.java 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/DistinctBindingSetOp.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -88,14 +88,6 @@
super(args, annotations);
- final IVariable<?>[] vars = getVariables();
-
- if (vars == null)
- throw new IllegalArgumentException();
-
- if (vars.length == 0)
- throw new IllegalArgumentException();
-
}
/**
@@ -133,7 +125,7 @@
*/
public IVariable<?>[] getVariables() {
- return (IVariable<?>[]) annotations.get(Annotations.VARIABLES);
+ return getRequiredProperty(Annotations.VARIABLES);
}
@@ -208,6 +200,12 @@
this.vars = op.getVariables();
+ if (vars == null)
+ throw new IllegalArgumentException();
+
+ if (vars.length == 0)
+ throw new IllegalArgumentException();
+
this.map = new ConcurrentHashMap<Solution, Solution>(
op.getInitialCapacity(), op.getLoadFactor(),
op.getConcurrencyLevel());
@@ -305,12 +303,13 @@
}
+ sink.flush();
+
// done.
return null;
} finally {
- sink.flush();
sink.close();
// discard the map.
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISlice.java (from rev 3423, branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/ISlice.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISlice.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISlice.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,69 @@
+/*
+
+Copyright (C) SYSTAP, LLC 2006-2008. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+*/
+/*
+ * Created on Sep 24, 2008
+ */
+
+package com.bigdata.bop.aggregation;
+
+import java.io.Serializable;
+
+import com.bigdata.relation.accesspath.IAccessPath;
+
+/**
+ * Indicates the first solution to be returned to the caller (offset) and the
+ * #of solutions to be returned (limit).
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public interface ISlice extends Serializable {
+
+ /**
+ * The first solution to be returned to the caller. A value of ZERO (0)
+ * indicates that all solutions should be returned.
+ */
+ public long getOffset();
+
+ /**
+ * The maximum #of solutions to be returned to the caller. A value of
+ * {@link Long#MAX_VALUE} indicates that there is no limit.
+ *
+ * @todo modify to be consistent with
+ * {@link IAccessPath#iterator(long, long, int)} where a limit of ZERO
+ * (0L) is interpreted as NO limit and a limit of
+ * {@link Long#MAX_VALUE} is interpreted as ZERO (0L) (that is, also
+ * no limit).
+ */
+ public long getLimit();
+
+ /**
+ * The index of the last solution that we will generate (OFFSET + LIMIT). If
+ * OFFSET + LIMIT would be greater than {@link Long#MAX_VALUE}, then use
+ * {@link Long#MAX_VALUE} instead.
+ */
+ public long getLast();
+
+}
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISortOrder.java (from rev 3448, branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/ISortOrder.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISortOrder.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISortOrder.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,55 @@
+/*
+
+Copyright (C) SYSTAP, LLC 2006-2008. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+*/
+/*
+ * Created on Sep 24, 2008
+ */
+
+package com.bigdata.bop.aggregation;
+
+import java.io.Serializable;
+
+import com.bigdata.bop.IVariable;
+
+/**
+ * A variable and an order that will be imposed on the values for that variable.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public interface ISortOrder<E> extends Serializable {
+
+ /**
+ * The variable whose values will be sorted.
+ */
+ IVariable<E> getVariable();
+
+ /**
+ * <code>true</code> iff the values will be placed into an ascending sort
+ * and <code>false</code> if the values will be placed into a descending
+ * sort.
+ */
+ boolean isAscending();
+
+}
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/MemorySortOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/MemorySortOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/MemorySortOp.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,112 @@
+package com.bigdata.bop.aggregation;
+
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.FutureTask;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpContext;
+import com.bigdata.bop.BOpUtility;
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.relation.accesspath.IBlockingBuffer;
+
+/**
+ * An in-memory merge sort for binding sets.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id: DistinctElementFilter.java 3466 2010-08-27 14:28:04Z
+ * thompsonbry $
+ *
+ * @todo unit tests.
+ * @todo do an external merge sort operator.
+ */
+public class MemorySortOp extends SortOp {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * Required deep copy constructor.
+ */
+ public MemorySortOp(final MemorySortOp op) {
+ super(op);
+ }
+
+ /**
+ * Required shallow copy constructor.
+ */
+ public MemorySortOp(final BOp[] args,
+ final Map<String, Object> annotations) {
+
+ super(args, annotations);
+
+ }
+
+ public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
+
+ return new FutureTask<Void>(new SortTask(this, context));
+
+ }
+
+ /**
+ * Task executing on the node.
+ */
+ static private class SortTask implements Callable<Void> {
+
+ private final BOpContext<IBindingSet> context;
+
+ /**
+ * The binding set comparator.
+ */
+ private final Comparator<IBindingSet> comparator;
+
+ SortTask(final MemorySortOp op,
+ final BOpContext<IBindingSet> context) {
+
+ this.context = context;
+
+ this.comparator = op.getComparator();
+
+ }
+
+ public Void call() throws Exception {
+
+ final BOpStats stats = context.getStats();
+
+ final IBlockingBuffer<IBindingSet[]> sink = context.getSink();
+
+ try {
+
+ final IBindingSet[] all = BOpUtility.toArray(context
+ .getSource(), stats);
+
+ // sort.
+ Arrays.sort(all, comparator);
+
+ // update counters.
+ stats.unitsOut.add(all.length);
+ stats.chunksOut.increment();
+
+ // write output and flush.
+ sink.add(all);
+ sink.flush();
+
+ // done.
+ return null;
+
+ } finally {
+
+ sink.close();
+
+ }
+
+ }
+
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/MemorySortOp.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/Slice.java (from rev 3423, branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/Slice.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/Slice.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/Slice.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,129 @@
+/*
+
+Copyright (C) SYSTAP, LLC 2006-2008. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+*/
+/*
+ * Created on Sep 24, 2008
+ */
+
+package com.bigdata.bop.aggregation;
+
+import java.io.Externalizable;
+import java.io.IOException;
+import java.io.ObjectInput;
+import java.io.ObjectOutput;
+import java.math.BigInteger;
+
+/**
+ * Default implementation.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class Slice implements ISlice, Externalizable {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 5396509164843609197L;
+
+ private long offset;
+ private long limit;
+ private long last;
+
+ /**
+ * A slice corresponding to all results (offset is zero, limit is
+ * {@link Long#MAX_VALUE}).
+ */
+ public static final transient ISlice ALL = new Slice(0, Long.MAX_VALUE);
+
+ /**
+ *
+ * @param offset
+ * @param limit
+ *
+ * @throws IllegalArgumentException
+ * if offset is negative.
+ * @throws IllegalArgumentException
+ * if limit is non-positive.
+ */
+ public Slice(final long offset, final long limit) {
+
+ if (offset < 0)
+ throw new IllegalArgumentException();
+
+ if (limit <= 0)
+ throw new IllegalArgumentException();
+
+ this.offset = offset;
+
+ this.limit = limit;
+
+ // @todo what is a cheaper way to do this?
+ this.last = BigInteger.valueOf(offset).add(BigInteger.valueOf(limit))
+ .min(BigInteger.valueOf(Long.MAX_VALUE)).longValue();
+
+ }
+
+ public long getOffset() {
+
+ return offset;
+
+ }
+
+ public long getLimit() {
+
+ return limit;
+
+ }
+
+ public long getLast() {
+
+ return last;
+
+ }
+
+ public String toString() {
+
+ return "Slice{offset="+offset+", limit="+limit+", last="+last+"}";
+
+ }
+
+ public void readExternal(ObjectInput in) throws IOException,
+ ClassNotFoundException {
+
+ offset = in.readLong();
+ limit = in.readLong();
+ last = in.readLong();
+
+ }
+
+ public void writeExternal(ObjectOutput out) throws IOException {
+
+ out.writeLong(offset);
+ out.writeLong(limit);
+ out.writeLong(last);
+
+ }
+
+}
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOp.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,86 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Sep 4, 2010
+ */
+
+package com.bigdata.bop.aggregation;
+
+import java.util.Map;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.BindingSetPipelineOp;
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.PipelineOp;
+
+/**
+ * Base class for operators which sort binding sets.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+abstract public class SortOp extends BindingSetPipelineOp {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ public interface Annotations extends BindingSetPipelineOp.Annotations {
+
+ /**
+ * The {@link ComparatorOp} which will impose the ordering on the
+ * binding sets.
+ *
+ * @see ComparatorOp
+ */
+ String COMPARATOR = MemorySortOp.class.getName() + ".comparator";
+
+ }
+
+ /**
+ * @param op
+ */
+ public SortOp(PipelineOp<IBindingSet> op) {
+ super(op);
+ }
+
+ /**
+ * @param args
+ * @param annotations
+ */
+ public SortOp(BOp[] args, Map<String, Object> annotations) {
+ super(args, annotations);
+ }
+
+ /**
+ * @see Annotations#COMPARATOR
+ */
+ public ComparatorOp getComparator() {
+
+ return getRequiredProperty(Annotations.COMPARATOR);
+
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOp.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOrder.java (from rev 3448, branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/SortOrder.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOrder.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOrder.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,80 @@
+/*
+
+Copyright (C) SYSTAP, LLC 2006-2008. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+*/
+/*
+ * Created on Sep 24, 2008
+ */
+
+package com.bigdata.bop.aggregation;
+
+import com.bigdata.bop.IVariable;
+
+/**
+ * Default impl.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class SortOrder<E> implements ISortOrder<E> {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = -669873421670514139L;
+
+ private final IVariable<E> var;
+ private final boolean asc;
+
+ /**
+ *
+ * @param var
+ * The variable.
+ * @param asc
+ * <code>true</code> for an ascending sort and
+ * <code>false</code> for a descending sort.
+ */
+ public SortOrder(final IVariable<E> var, final boolean asc) {
+
+ if (var == null)
+ throw new IllegalArgumentException();
+
+ this.var = var;
+
+ this.asc = asc;
+
+ }
+
+ public IVariable<E> getVariable() {
+
+ return var;
+
+ }
+
+ public boolean isAscending() {
+
+ return asc;
+
+ }
+
+}
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SparqlBindingSetComparatorOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SparqlBindingSetComparatorOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SparqlBindingSetComparatorOp.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,101 @@
+package com.bigdata.bop.aggregation;
+
+import java.util.Comparator;
+import java.util.Map;
+
+import org.openrdf.query.algebra.evaluation.util.ValueComparator;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.IVariable;
+
+/**
+ * A comparator for SPARQL binding sets.
+ *
+ * @see http://www.w3.org/TR/rdf-sparql-query/#modOrderBy
+ * @see ValueComparator
+ *
+ * @todo unit tests.
+ */
+public class SparqlBindingSetComparatorOp extends ComparatorOp {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * Required deep copy constructor.
+ */
+ public SparqlBindingSetComparatorOp(final SparqlBindingSetComparatorOp op) {
+ super(op);
+ }
+
+ /**
+ * Required shallow copy constructor.
+ */
+ public SparqlBindingSetComparatorOp(final BOp[] args,
+ final Map<String, Object> annotations) {
+
+ super(args, annotations);
+
+ }
+
+ /**
+ * @see Annotations#ORDER
+ */
+ public ISortOrder<?>[] getOrder() {
+
+ return getRequiredProperty(Annotations.ORDER);
+
+ }
+
+ /**
+ * The sort order to be imposed.
+ */
+ private transient ISortOrder<?>[] order;
+
+ private transient Comparator vc;
+
+ public int compare(final IBindingSet bs1, final IBindingSet bs2) {
+
+ if (order == null) {
+
+ // lazy initialization.
+ order = getOrder();
+
+ if (order == null)
+ throw new IllegalArgumentException();
+
+ if (order.length == 0)
+ throw new IllegalArgumentException();
+
+ // comparator for RDF Value objects.
+ vc = new ValueComparator();
+
+ }
+
+ for (int i = 0; i < order.length; i++) {
+
+ final ISortOrder<?> o = order[i];
+
+ final IVariable v = o.getVariable();
+
+ int ret = vc.compare(bs1.get(v).get(), bs2.get(v).get());
+
+ if (!o.isAscending())
+ ret = -ret;
+
+ if (ret != 0) {
+ // not equal for this variable.
+ return ret;
+ }
+
+ }
+
+ // equal for all variables.
+ return 0;
+
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SparqlBindingSetComparatorOp.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/package.html
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/package.html 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/package.html 2010-09-05 17:02:34 UTC (rev 3508)
@@ -1,6 +1,6 @@
<html>
<head>
-<title>distinct, sort, and aggregation operators</title>
+<title>solution modifier operators (distinct, sort, slice, and aggregation)</title>
</head>
<body>
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/InsertOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/InsertOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/InsertOp.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,271 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Aug 25, 2010
+ */
+
+package com.bigdata.bop.mutation;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.FutureTask;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpContext;
+import com.bigdata.bop.BindingSetPipelineOp;
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.IVariableOrConstant;
+import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.btree.ILocalBTreeView;
+import com.bigdata.btree.ITupleSerializer;
+import com.bigdata.btree.keys.IKeyBuilder;
+import com.bigdata.relation.IRelation;
+import com.bigdata.relation.accesspath.IAsynchronousIterator;
+import com.bigdata.relation.accesspath.IBlockingBuffer;
+import com.bigdata.striterator.IKeyOrder;
+
+/**
+ * This operator writes elements constructed from binding sets and an orders
+ * list of variables and constants on an index.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ *
+ * @param <E>
+ * The generic type of the elements written onto the index.
+ */
+public class InsertOp<E> extends BindingSetPipelineOp {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ public interface Annotations extends BindingSetPipelineOp.Annotations {
+
+ /**
+ * An ordered {@link IVariableOrConstant}[]. Elements will be created
+ * using the binding sets which flow through the operator and
+ * {@link IRelation#newElement(java.util.List, IBindingSet)}.
+ */
+ String SELECTED = InsertOp.class.getName() + ".selected";
+
+ /**
+ * The namespace of the relation to which the index belongs.
+ */
+ String RELATION = InsertOp.class.getName() + ".relation";
+
+ /**
+ * The {@link IKeyOrder} for the index.
+ */
+ String KEY_ORDER = InsertOp.class.getName() + ".keyOrder";
+
+ }
+
+ /**
+ * Deep copy constructor.
+ *
+ * @param op
+ */
+ public InsertOp(InsertOp<E> op) {
+ super(op);
+ }
+
+ /**
+ * Shallow copy constructor.
+ *
+ * @param args
+ * @param annotations
+ */
+ public InsertOp(BOp[] args, Map<String, Object> annotations) {
+
+ super(args, annotations);
+
+ getRequiredProperty(Annotations.SELECTED);
+
+ }
+
+ /**
+ * @see Annotations#SELECTED
+ */
+ public IVariableOrConstant<?>[] getSelected() {
+
+// return (IVariableOrConstant<?>[]) getProperty(Annotations.SELECTED);
+ return getRequiredProperty(Annotations.SELECTED);
+
+ }
+
+ /**
+ * @see Annotations#RELATION
+ */
+ public String getRelation() {
+
+ return getRequiredProperty(Annotations.RELATION);
+
+ }
+
+ /**
+ * @see Annotations#KEY_ORDER
+ */
+ public IKeyOrder<E> getKeyOrder() {
+
+ return getRequiredProperty(Annotations.KEY_ORDER);
+
+ }
+
+ public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
+
+ return new FutureTask<Void>(new InsertTask<E>(this, context));
+
+ }
+
+ /**
+ * Create elements from the selected bindings and insert them onto the named
+ * index.
+ */
+ static private class InsertTask<E> implements Callable<Void> {
+
+ private final BOpStats stats;
+
+ private final BOpContext<IBindingSet> context;
+
+ private final IAsynchronousIterator<IBindingSet[]> source;
+
+ /**
+ * Only used to close the sink when we are done.
+ */
+ private final IBlockingBuffer<IBindingSet[]> sink;
+
+ private List<IVariableOrConstant<?>> selected;
+
+ private final IRelation<E> relation;
+
+ private final IKeyOrder<E> keyOrder;
+
+ @SuppressWarnings("unchecked")
+ InsertTask(final InsertOp<E> op, final BOpContext<IBindingSet> context) {
+
+ this.context = context;
+
+ stats = context.getStats();
+
+ source = context.getSource();
+
+ sink = context.getSink();
+
+ selected = Arrays.asList(op.getSelected());
+
+ relation = context.getWriteRelation(op.getRelation());
+
+ keyOrder = op.getKeyOrder();
+
+ }
+
+ /**
+ *
+ * @todo This does not order the tuples before writing on the local
+ * index. I am not sure that it should. I think that order is
+ * generally obtained from how we organize the tuples when mapping
+ * them across shards. However, for standalone databases it may
+ * make sense to insert a SORT on the selected attributes before
+ * the INSERT.
+ */
+ public Void call() throws Exception {
+
+ /*
+ * @todo validate for s/o. Since this goes through a common code
+ * path, what we really need to test is getMutableLocalIndexView().
+ * The rest of the insert operation can be tested against a local
+ * Journal.
+ */
+ final ILocalBTreeView ndx = context.getMutableLocalIndexView(
+ relation, keyOrder, context.getPartitionId());
+
+ final IKeyBuilder keyBuilder = ndx.getIndexMetadata()
+ .getKeyBuilder();
+
+ final ITupleSerializer tupleSer = ndx.getIndexMetadata()
+ .getTupleSerializer();
+
+ try {
+
+ while (source.hasNext()) {
+
+ final IBindingSet[] chunk = source.next();
+
+ stats.chunksIn.increment();
+ stats.unitsIn.add(chunk.length);
+
+ int nwritten = 0;
+ for (int i = 0; i < chunk.length; i++) {
+
+ final IBindingSet bset = chunk[i];
+
+ final E e = relation.newElement(selected, bset);
+
+ final byte[] key = keyOrder.getKey(keyBuilder, e);
+
+ if (!ndx.contains(key)) {
+
+ final byte[] val = tupleSer.serializeVal(e);
+
+ ndx.insert(key, val);
+
+ nwritten++;
+
+ }
+
+ }
+
+ if (nwritten > 0) {
+ stats.unitsOut.add(nwritten);
+ stats.chunksOut.increment();
+ }
+
+ }
+
+ return null;
+
+ } finally {
+
+ sink.close();
+
+ }
+
+ }
+
+ }
+
+
+ // E[] a = null;
+ // Note: useful if we will sort before writing on the index.
+// if (i == 0)
+// a = (E[]) java.lang.reflect.Array.newInstance(e
+// .getClass());
+//
+// a[i] = e;
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/InsertOp.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/package.html
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/package.html (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/package.html 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,14 @@
+<html>
+<head>
+<title>mutation operators</title>
+</head>
+<body>
+
+<p>
+
+
+
+</p>
+
+</body>
+</html>
\ No newline at end of file
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/package.html
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/AbstractRelation.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/AbstractRelation.java 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/AbstractRelation.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -28,8 +28,14 @@
package com.bigdata.relation;
+import java.util.ArrayList;
+import java.util.List;
import java.util.Properties;
import java.util.UUID;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicLong;
import com.bigdata.bop.IPredicate;
import com.bigdata.btree.IIndex;
@@ -43,10 +49,25 @@
import com.bigdata.journal.Journal;
import com.bigdata.journal.TemporaryRawStore;
import com.bigdata.journal.TemporaryStore;
+import com.bigdata.rdf.lexicon.LexiconRelation;
+import com.bigdata.rdf.model.StatementEnum;
+import com.bigdata.rdf.spo.ISPO;
+import com.bigdata.rdf.spo.JustificationRemover;
+import com.bigdata.rdf.spo.SPO;
+import com.bigdata.rdf.spo.SPOAccessPath;
+import com.bigdata.rdf.spo.SPOIndexRemover;
+import com.bigdata.rdf.spo.SPOIndexWriter;
+import com.bigdata.rdf.spo.SPOKeyOrder;
+import com.bigdata.rdf.spo.SPORelation;
+import com.bigdata.rdf.store.AbstractTripleStore;
import com.bigdata.relation.accesspath.AccessPath;
import com.bigdata.relation.accesspath.IAccessPath;
+import com.bigdata.relation.accesspath.IElementFilter;
+import com.bigdata.relation.rule.eval.ISolution;
+import com.bigdata.relation.rule.eval.AbstractSolutionBuffer.InsertSolutionBuffer;
import com.bigdata.service.DataService;
import com.bigdata.service.IBigdataFederation;
+import com.bigdata.striterator.IChunkedOrderedIterator;
import com.bigdata.striterator.IKeyOrder;
/**
@@ -56,11 +77,6 @@
* @version $Id$
* @param <E>
* The generic type of the [E]lements of the relation.
- *
- * @todo It would be interesting to do a GOM relation with its secondary index
- * support and the addition of clustered indices. We would then get
- * efficient JOINs via the rules layer for free and a high-level query
- * language could be mapped onto those JOINs.
*/
abstract public class AbstractRelation<E> extends AbstractResource<IRelation<E>> implements
IMutableRelation<E> {
@@ -323,5 +339,5 @@
getChunkCapacity(), getFullyBufferedReadThreshold()).init();
}
-
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/IMutableRelation.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/IMutableRelation.java 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/IMutableRelation.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -80,71 +80,4 @@
*/
public long delete(IChunkedOrderedIterator<E> itr);
- /*
- * @todo update is notional. it has not been implemented yet (you can use
- * delete+insert). i suspect that the implementation will eventually involve
- * the "transform" being specified as an extension to the rule, e.g.,
- *
- * update [relation] set z=foo from [relation] (x,y,bar)
- *
- * if the update causes a change in the key for either the primary index or
- * any secondary index for the relation then it must be realized as a
- * delete+insert since the changes to the underlying tuples might not be
- * local (e.g., they could be in an different index partition, on a
- * different data service, on even on a different host). such non-local
- * changes will not be atomic unless you use a full transaction or
- * read-behind from a last known consistent commit point.
- *
- * @todo the RDF DB faces exactly this problem with truth maintenance (the
- * problem is somewhat more severe since inference against an unstable KB
- * state can magnify any inconsistencies).
- */
-
-// /**
-// * Update elements on the relation.
-// * <p>
-// * The implemention must locate each element in the relation and, if found,
-// * update its state using the <i>transform</i>. It is an error if the
-// * transformed element has a different "primary key" than the visited
-// * element.
-// * <p>
-// * Note: While UPDATE is often realized as "DELETE + INSERT" within the same
-// * transaction, the advantage of this formulation is that is one-half of the
-// * cost since each element in the relation is visited only once. However, if
-// * you need to update parts of the primary key then "DELETE + INSERT" is the
-// * way to go since each the "delete" and the "insert" may operate on
-// * different tuples located in different parts of the index. For scale-out
-// * indices, those tuples can even lie on different machines.
-// *
-// * @param itr
-// * An iterator visiting the elements selected for update.
-// * @param transform
-// * A transform that produces the new state for each visited
-// * element.
-// *
-// * @return The #of elements that were actually modified in the relation.
-// */
-// public long update(IChunkedOrderedIterator...
[truncated message content] |
