Menu
▾
▴
bigdata-commit — Commit traffic.
This list is closed, nobody may subscribe to it.
| 2010 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(139) |
Aug
(94) |
Sep
(232) |
Oct
(143) |
Nov
(138) |
Dec
(55) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2011 |
Jan
(127) |
Feb
(90) |
Mar
(101) |
Apr
(74) |
May
(148) |
Jun
(241) |
Jul
(169) |
Aug
(121) |
Sep
(157) |
Oct
(199) |
Nov
(281) |
Dec
(75) |
| 2012 |
Jan
(107) |
Feb
(122) |
Mar
(184) |
Apr
(73) |
May
(14) |
Jun
(49) |
Jul
(26) |
Aug
(103) |
Sep
(133) |
Oct
(61) |
Nov
(51) |
Dec
(55) |
| 2013 |
Jan
(59) |
Feb
(72) |
Mar
(99) |
Apr
(62) |
May
(92) |
Jun
(19) |
Jul
(31) |
Aug
(138) |
Sep
(47) |
Oct
(83) |
Nov
(95) |
Dec
(111) |
| 2014 |
Jan
(125) |
Feb
(60) |
Mar
(119) |
Apr
(136) |
May
(270) |
Jun
(83) |
Jul
(88) |
Aug
(30) |
Sep
(47) |
Oct
(27) |
Nov
(23) |
Dec
|
| 2015 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(3) |
Oct
|
Nov
|
Dec
|
| 2016 |
Jan
|
Feb
|
Mar
(4) |
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: <tho...@us...> - 2010-09-08 20:52:16
|
Revision: 3524
http://bigdata.svn.sourceforge.net/bigdata/?rev=3524&view=rev
Author: thompsonbry
Date: 2010-09-08 20:52:07 +0000 (Wed, 08 Sep 2010)
Log Message:
-----------
Added a class (DirectBufferPoolAllocator) to manage allocations of direct ByteBuffer slices against the DirectBufferPool for use with the FederatedQueryEngine. There is a limited test suite for this class.
Partly integrated the DirectBufferPoolAllocator with the FederatedQueryEngine. The generated intermediate results are now written onto direct ByteBuffer slices allocated on the DirectBufferPool and notification messages are sent to the receiving services. I have not yet modified the receiving FederatedQueryEngine to demand the data from the remote query engine.
Harmonized InsertOp and PipelineJoin as IShardwiseBindingSetOps. This makes it possible to find the predicate for the access path on which we need to read/write and map the results to the appropriate shards.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IQueryPeer.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedRunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/InsertOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/IRelation.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/RelationFusedView.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/resources/StoreManager.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/DataService.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/IDataService.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/ManagedResourceService.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/ResourceService.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/ap/R.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestFederatedQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/join/TestPipelineJoin.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/io/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/resources/AbstractResourceManagerTestCase.java
branches/QUADS_QUERY_BRANCH/bigdata-jini/src/java/com/bigdata/service/jini/DataServer.java
branches/QUADS_QUERY_BRANCH/bigdata-jini/src/java/com/bigdata/service/jini/JiniFederation.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/lexicon/LexiconRelation.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/magic/MagicRelation.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPORelation.java
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IShardwisePipelineOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/io/DirectBufferPoolAllocator.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/io/TestDirectBufferPoolAllocator.java
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IShardwisePipelineOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IShardwisePipelineOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IShardwisePipelineOp.java 2010-09-08 20:52:07 UTC (rev 3524)
@@ -0,0 +1,48 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Sep 8, 2010
+ */
+
+package com.bigdata.bop;
+
+/**
+ * An interface for {@link BindingSetPipelineOp}s which are mapped across
+ * shards.
+ *
+ * @param <E>
+ * The generic type of the elements in the relation on which the
+ * predicate will read or write.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public interface IShardwisePipelineOp<E> {
+
+ /**
+ * The predicate which reads or writes on the shard.
+ */
+ IPredicate<E> getPredicate();
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IShardwisePipelineOp.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IQueryPeer.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IQueryPeer.java 2010-09-08 20:11:49 UTC (rev 3523)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IQueryPeer.java 2010-09-08 20:52:07 UTC (rev 3524)
@@ -3,8 +3,10 @@
import java.net.InetSocketAddress;
import java.rmi.Remote;
import java.rmi.RemoteException;
+import java.util.UUID;
import com.bigdata.bop.BOp;
+import com.bigdata.service.IService;
/**
* Interface for a node participating in the exchange of NIO buffers to
@@ -13,6 +15,14 @@
public interface IQueryPeer extends Remote {
/**
+ * The {@link UUID} of the service within which the {@link IQueryPeer} is
+ * running.
+ *
+ * @see IService#getServiceUUID()
+ */
+ UUID getServiceUUID() throws RemoteException;
+
+ /**
* Notify a service that a buffer having data for some {@link BOp} in some
* running query is available. The receiver may request the data when they
* are ready. If the query is cancelled, then the sender will drop the
@@ -30,7 +40,9 @@
* @return <code>true</code> unless the receiver knows that the query has
* already been cancelled.
*/
+// * @param nbytes
+// * The #of bytes of data which are available for that operator.
void bufferReady(IQueryClient clientProxy, InetSocketAddress serviceAddr,
- long queryId, int bopId) throws RemoteException;
+ long queryId, int bopId/*, int nbytes*/) throws RemoteException;
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java 2010-09-08 20:11:49 UTC (rev 3523)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java 2010-09-08 20:52:07 UTC (rev 3524)
@@ -372,7 +372,7 @@
* is running -or- <code>null</code> if the {@link QueryEngine} is
* not running against an {@link IBigdataFederation}.
*/
- protected UUID getServiceId() {
+ public UUID getServiceUUID() {
return null;
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-09-08 20:11:49 UTC (rev 3523)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-09-08 20:52:07 UTC (rev 3524)
@@ -59,7 +59,6 @@
import com.bigdata.relation.accesspath.BlockingBuffer;
import com.bigdata.relation.accesspath.IAsynchronousIterator;
import com.bigdata.relation.accesspath.IBlockingBuffer;
-import com.bigdata.resources.ResourceManager;
import com.bigdata.service.IBigdataFederation;
import com.bigdata.striterator.ICloseableIterator;
import com.bigdata.util.concurrent.Haltable;
@@ -209,40 +208,6 @@
private final Set<Integer/*bopId*/> startedSet = new LinkedHashSet<Integer>();
/**
- * A map associating resources with running queries. When a query halts, the
- * resources listed in its resource map are released. Resources can include
- * {@link ByteBuffer}s backing either incoming or outgoing
- * {@link BindingSetChunk}s, temporary files associated with the query, hash
- * tables, etc.
- *
- * @todo Cache any resources materialized for the query on this node (e.g.,
- * temporary graphs materialized from a peer or the client). A bop
- * should be able to demand those data from the cache and otherwise
- * have them be materialized.
- *
- * @todo only use the values in the map for transient objects, such as a
- * hash table which is not backed by the disk. For {@link ByteBuffer}s
- * we want to make the references go through the {@link BufferService}
- * . For files, through the {@link ResourceManager}.
- *
- * @todo We need to track the resources in use by the query so they can be
- * released when the query terminates. This includes: buffers; joins
- * for which there is a chunk of binding sets that are currently being
- * executed; downstream joins (they depend on the source joins to
- * notify them when they are complete in order to decide their own
- * termination condition); local hash tables which are part of a DHT
- * (especially when they are persistent); buffers and disk resources
- * allocated to N-way merge sorts, etc.
- *
- * @todo The set of buffers having data which has been accepted for this
- * query.
- *
- * @todo The set of buffers having data which has been generated for this
- * query.
- */
- private final ConcurrentHashMap<UUID, Object> resourceMap = new ConcurrentHashMap<UUID, Object>();
-
- /**
* The chunks available for immediate processing.
* <p>
* Note: This is package private so it will be visible to the
@@ -261,7 +226,7 @@
* combined before we execute the operator. For unselective operators,
* we are going to run over all the data anyway.
*/
- final BlockingQueue<BindingSetChunk> chunksIn = new LinkedBlockingDeque<BindingSetChunk>();
+ final /*private*/ BlockingQueue<BindingSetChunk> chunksIn = new LinkedBlockingDeque<BindingSetChunk>();
/**
* The class executing the query on this node.
@@ -601,7 +566,7 @@
final long elapsed = System.currentTimeMillis() - begin;
if (log.isTraceEnabled())
log.trace("bopId=" + msg.bopId + ",partitionId=" + msg.partitionId
- + ",serviceId=" + queryEngine.getServiceId()
+ + ",serviceId=" + queryEngine.getServiceUUID()
+ ", nchunks=" + fanOut + " : runningTaskCount="
+ runningTaskCount + ", availableChunkCount="
+ availableChunkCount + ", elapsed=" + elapsed);
@@ -742,7 +707,7 @@
.getProperty(BindingSetPipelineOp.Annotations.BOP_ID);
final IBlockingBuffer<IBindingSet[]> sink = (p == null ? queryBuffer
: op.newBuffer());
- // altSink [@todo altSink=null or sink when not specified?]
+ // altSink (null when not specified).
final Integer altSinkId = (Integer) op
.getProperty(BindingSetPipelineOp.Annotations.ALT_SINK_REF);
if (altSinkId != null && !bopIndex.containsKey(altSinkId)) {
@@ -758,7 +723,7 @@
// Hook the FutureTask.
final Runnable r = new Runnable() {
public void run() {
- final UUID serviceId = queryEngine.getServiceId();
+ final UUID serviceId = queryEngine.getServiceUUID();
int fanIn = 1;
int sinkChunksOut = 0;
int altSinkChunksOut = 0;
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java 2010-09-08 20:11:49 UTC (rev 3523)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java 2010-09-08 20:52:07 UTC (rev 3524)
@@ -27,17 +27,18 @@
package com.bigdata.bop.fed;
+import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
import com.bigdata.bop.BindingSetPipelineOp;
import com.bigdata.bop.IBindingSet;
-import com.bigdata.bop.bset.ConditionalRoutingOp;
import com.bigdata.bop.engine.IQueryClient;
+import com.bigdata.bop.engine.IQueryPeer;
import com.bigdata.bop.engine.QueryEngine;
import com.bigdata.bop.engine.RunningQuery;
-import com.bigdata.bop.join.PipelineJoin;
import com.bigdata.bop.solutions.SliceOp;
import com.bigdata.journal.IIndexManager;
import com.bigdata.relation.accesspath.IAsynchronousIterator;
@@ -45,6 +46,7 @@
import com.bigdata.relation.accesspath.IBuffer;
import com.bigdata.service.DataService;
import com.bigdata.service.IBigdataFederation;
+import com.bigdata.service.IDataService;
import com.bigdata.service.ManagedResourceService;
import com.bigdata.service.ResourceService;
@@ -55,50 +57,14 @@
* @version $Id: FederatedQueryEngine.java 3508 2010-09-05 17:02:34Z thompsonbry
* $
*
- * @todo Modify the {@link FederatedQueryEngine} to actually run a distributed
- * query. Since we are in the same JVM, the {@link IBindingSet} chunks can
- * be used directly without being marshalled onto {@link ByteBuffer}s and
- * transferred over the network.
- * <p>
- * Distributed query will fail until each {@link FederatedQueryEngine} is
- * receiving chunks and running operators against its local
- * {@link IIndexManager}. This requires that we map the output chunks for
- * an operator over the shards for the next operator, that we send the
- * appropriate messages to the query engine peers, that they demand the
- * necessary data from their peers, etc.
- * <p>
- * Once distributed query is running, begin to marshall the chunks onto
- * buffers [this might have to be done immediately to get the notification
- * protocol working].
+ * @todo buffer management for s/o including bindingSet[] movement for the
+ * pipeline and element[] movement for DHT on access path.
*
- * @todo buffer management for s/o, including binding sets movement, element
- * chunk movement for DHT on access path, and on demand materialization of
- * large query resources for large data sets, parallel closure, etc.;
- * grouping operators which will run locally (such as a pipeline join plus
- * a conditional routing operator) so we do not marshall binding sets
- * between operators when they will not cross a network boundary. Also,
- * handle mutation, programs and closure operators.
- *
- * @todo I have not yet figured out how to mark operators to indicate when their
- * output should be mapped across shards or handled locally. It would
- * appear that this is a concern of their parent in the operator tree. For
- * example, the {@link ConditionalRoutingOp} would be applied to transform
- * the output of a {@link PipelineJoin} before mapping the output over the
- * shards.
- * <p>
- * The operator themselves could carry this information either as a Java
- * method or as an annotation.
- * <p>
- * This could interact with how we combine {@link RunningQuery#chunksIn}.
- *
* @todo Override to release buffers associated with chunks buffered for a query
* when it terminates (buffers may be for received chunks or chunks which
* are awaiting transfer to another node). [This might be handled by a
* {@link RunningQuery} override.]
*
- * @todo Override protocol hooks for moving data around among the
- * {@link QueryEngine}s
- *
* @todo Compressed representations of binding sets with the ability to read
* them in place or materialize them onto the java heap. The
* representation should be ammenable to processing in C since we want to
@@ -167,7 +133,7 @@
}
@Override
- protected UUID getServiceId() {
+ public UUID getServiceUUID() {
return fed.getServiceUUID();
@@ -190,13 +156,18 @@
return resourceService;
}
-
+
+ /**
+ * FIXME Once buffers are ready their data needs to be materialized on this
+ * node and the chunks queued for processing.
+ *
+ * @todo What is the cost of passing the proxy around like this? Should it
+ * be discovered instead from a registrar?
+ */
@Override
public void bufferReady(IQueryClient clientProxy,
InetSocketAddress serviceAddr, long queryId, int bopId) {
- // @todo notify peer when a buffer is ready.
-
}
/**
@@ -246,4 +217,47 @@
}
+ /**
+ * Resolve an {@link IQueryPeer}.
+ * <p>
+ * Note: This only resolves the peers running on the {@link IDataService}s.
+ * It will not resolve a query controller unless an {@link IDataService} is
+ * being used as the query controller.
+ *
+ * @param serviceUUID
+ * The service {@link UUID}.
+ *
+ * @return The proxy for the query peer.
+ */
+ protected IQueryPeer getQueryPeer(final UUID serviceUUID) {
+
+ IQueryPeer proxy = proxyMap.get(serviceUUID);
+
+ if (proxy == null) {
+
+ final IDataService dataService = getFederation().getDataService(
+ serviceUUID);
+
+ if (dataService == null)
+ throw new RuntimeException("No such service: " + serviceUUID);
+
+ try {
+ proxy = dataService.getQueryEngine();
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ proxyMap.put(serviceUUID, proxy);
+
+ }
+
+ return proxy;
+
+ }
+
+ /**
+ * Cache for {@link #getQueryPeer(UUID)}.
+ */
+ private final ConcurrentHashMap<UUID, IQueryPeer> proxyMap = new ConcurrentHashMap<UUID, IQueryPeer>();
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedRunningQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedRunningQuery.java 2010-09-08 20:11:49 UTC (rev 3523)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedRunningQuery.java 2010-09-08 20:52:07 UTC (rev 3524)
@@ -27,19 +27,39 @@
package com.bigdata.bop.fed;
+import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
+import java.rmi.RemoteException;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpEvaluationContext;
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.IPredicate;
+import com.bigdata.bop.IShardwisePipelineOp;
+import com.bigdata.bop.engine.BindingSetChunk;
import com.bigdata.bop.engine.IQueryClient;
+import com.bigdata.bop.engine.IQueryPeer;
+import com.bigdata.bop.engine.QueryEngine;
import com.bigdata.bop.engine.RunningQuery;
+import com.bigdata.io.DirectBufferPoolAllocator;
+import com.bigdata.io.SerializerUtil;
+import com.bigdata.io.DirectBufferPoolAllocator.IAllocation;
+import com.bigdata.io.DirectBufferPoolAllocator.IAllocationContext;
import com.bigdata.mdi.PartitionLocator;
+import com.bigdata.relation.accesspath.BlockingBuffer;
import com.bigdata.relation.accesspath.IAsynchronousIterator;
import com.bigdata.relation.accesspath.IBlockingBuffer;
import com.bigdata.relation.accesspath.IBuffer;
+import com.bigdata.resources.ResourceManager;
import com.bigdata.service.IBigdataFederation;
-import com.bigdata.service.jini.master.IAsynchronousClientTask;
+import com.bigdata.service.ManagedResourceService;
+import com.bigdata.service.ResourceService;
import com.bigdata.striterator.IKeyOrder;
/**
@@ -69,6 +89,59 @@
* */
public class FederatedRunningQuery extends RunningQuery {
+ /**
+ * The {@link UUID} of the service which is the {@link IQueryClient} running
+ * this query.
+ */
+ private final UUID queryControllerUUID;
+
+ /**
+ * A map associating resources with running queries. When a query halts, the
+ * resources listed in its resource map are released. Resources can include
+ * {@link ByteBuffer}s backing either incoming or outgoing
+ * {@link BindingSetChunk}s, temporary files associated with the query, hash
+ * tables, etc.
+ *
+ * @todo This map will eventually need to be moved into {@link RunningQuery}
+ * in order to support temporary graphs or other disk-backed resources
+ * associated with the evaluation of a query against a standalone
+ * database. However, the main use case are the resources associated
+ * with query against an {@link IBigdataFederation} which it why it is
+ * being developed in the {@link FederatedRunningQuery} class.
+ *
+ * @todo Cache any resources materialized for the query on this node (e.g.,
+ * temporary graphs materialized from a peer or the client). A bop
+ * should be able to demand those data from the cache and otherwise
+ * have them be materialized.
+ *
+ * @todo Only use the values in the map for transient objects, such as a
+ * hash table which is not backed by the disk. For {@link ByteBuffer}s
+ * we want to make the references go through the {@link ResourceService}
+ * . For files, through the {@link ResourceManager}.
+ *
+ * @todo We need to track the resources in use by the query so they can be
+ * released when the query terminates. This includes: buffers; joins
+ * for which there is a chunk of binding sets that are currently being
+ * executed; downstream joins (they depend on the source joins to
+ * notify them when they are complete in order to decide their own
+ * termination condition); local hash tables which are part of a DHT
+ * (especially when they are persistent); buffers and disk resources
+ * allocated to N-way merge sorts, etc.
+ *
+ * @todo The set of buffers having data which has been accepted for this
+ * query.
+ *
+ * @todo The set of buffers having data which has been generated for this
+ * query.
+ */
+ private final ConcurrentHashMap<UUID, Object> resourceMap = new ConcurrentHashMap<UUID, Object>();
+
+ /**
+ * @todo Maintain multiple allocation contexts. Some can be query wide.
+ * Others might be specific to a serviceId and/or sinkId.
+ */
+ private final ConcurrentHashMap<Object/* key */, IAllocationContext> allocationContexts = new ConcurrentHashMap<Object, IAllocationContext>();
+
public FederatedRunningQuery(FederatedQueryEngine queryEngine,
long queryId, long readTimestamp, long writeTimestamp, long begin,
long timeout, boolean controller, IQueryClient clientProxy,
@@ -76,6 +149,17 @@
super(queryEngine, queryId, readTimestamp, writeTimestamp, begin,
timeout, controller, clientProxy, query, queryBuffer);
+
+ /*
+ * Note: getServiceUUID() should be a smart proxy method and thus not
+ * actually do RMI here. However, it is resolved eagerly and cached
+ * anyway.
+ */
+ try {
+ this.queryControllerUUID = getQueryController().getServiceUUID();
+ } catch (RemoteException e) {
+ throw new RuntimeException(e);
+ }
}
@@ -87,38 +171,148 @@
}
/**
+ * The allocation context key groups together allocations onto the same
+ * direct {@link ByteBuffer}s. There are different implementations depending
+ * on how it makes sense to group data data for a given query.
+ */
+ static abstract private class AllocationContextKey {
+
+ /**
+ * Must be overridden. The queryId must be a part of each hashCode() in
+ * order to ensure that the hash codes are well distributed across
+ * different queries on the same node.
+ */
+ @Override
+ abstract public int hashCode();
+
+ /**
+ * Must be overridden.
+ */
+ @Override
+ abstract public boolean equals(Object o);
+
+ }
+
+ /**
+ * An allocation context which is shared by all operators running in the
+ * same query.
+ */
+ static private class QueryContext extends AllocationContextKey {
+ private final Long queryId;
+
+ QueryContext(final Long queryId) {
+ this.queryId = Long.valueOf(queryId);
+ }
+
+ public int hashCode() {
+ return queryId.hashCode();
+ }
+
+ public boolean equals(final Object o) {
+ if (this == o)
+ return true;
+ if (!(o instanceof QueryContext))
+ return false;
+ if (!queryId.equals(((QueryContext) o).queryId))
+ return false;
+ return true;
+ }
+ }
+
+ /**
+ * An allocation context which is shared by all operators running in the
+ * same query which target the same service.
+ */
+ static private class ServiceContext extends AllocationContextKey {
+ private final Long queryId;
+
+ private final UUID serviceUUID;
+
+ ServiceContext(final Long queryId, final UUID serviceUUID) {
+ this.queryId = queryId;
+ this.serviceUUID = serviceUUID;
+ }
+
+ public int hashCode() {
+ return queryId.hashCode() * 31 + serviceUUID.hashCode();
+ }
+
+ public boolean equals(final Object o) {
+ if (this == o)
+ return true;
+ if (!(o instanceof ServiceContext))
+ return false;
+ if (!queryId.equals(((ServiceContext) o).queryId))
+ return false;
+ if (!serviceUUID.equals(((ServiceContext) o).serviceUUID))
+ return false;
+ return true;
+ }
+ }
+
+ /**
+ * An allocation context which is shared by all operators running in the
+ * same query which target the same shard (the same shard implies the same
+ * service, at least until we have HA with shard affinity).
+ */
+ static private class ShardContext extends AllocationContextKey {
+
+ private final Long queryId;
+
+ private final int partitionId;
+
+ ShardContext(final Long queryId, final int partitionId) {
+ this.queryId = queryId;
+ this.partitionId = partitionId;
+ }
+
+ public int hashCode() {
+ return queryId.hashCode() * 31 + partitionId;
+ }
+
+ public boolean equals(final Object o) {
+ if (this == o)
+ return true;
+ if (!(o instanceof ShardContext))
+ return false;
+ if (!queryId.equals(((ShardContext) o).queryId))
+ return false;
+ if (partitionId != partitionId)
+ return false;
+ return true;
+ }
+ }
+
+ /**
+ * Return the {@link IAllocationContext} for the given key.
+ *
+ * @param key
+ * The key.
+ *
+ * @return The allocation context.
+ */
+ private IAllocationContext getAllocationContext(
+ final AllocationContextKey key) {
+
+ return getQueryEngine().getResourceService().getAllocator()
+ .getAllocationContext(key);
+
+ }
+
+ /**
* {@inheritDoc}
- *
- * @return The #of chunks made available for consumption by the sink. This
- * will always be ONE (1) for scale-up. For scale-out, there will be
- * one chunk per index partition over which the intermediate results
- * were mapped.
- *
- * FIXME SCALEOUT: This is where we need to map the binding sets
- * over the shards for the target operator. Once they are mapped,
- * write the binding sets onto an NIO buffer for the target node and
- * then send an RMI message to the node telling it that there is a
- * chunk available for the given (queryId,bopId,partitionId).
- * <p>
- * For selective queries in s/o, first format the data onto a list
- * of byte[]s, one per target shard/node. Then, using a lock, obtain
- * a ByteBuffer if there is none associated with the query yet.
- * Otherwise, using the same lock, obtain a slice onto that
- * ByteBuffer and put as much of the byte[] as will fit, continuing
- * onto a newly recruited ByteBuffer if necessary. Release the lock
- * and notify the target of the ByteBuffer slice (buffer#, off,
- * len). Consider pushing the data proactively for selective
- * queries.
- * <p>
- * For unselective queries in s/o, proceed as above but we need to
- * get the data off the heap and onto the {@link ByteBuffer}s
- * quickly (incrementally) and we want the consumers to impose flow
- * control on the producers to bound the memory demand (this needs
- * to be coordinated carefully to avoid deadlocks). Typically, large
- * result sets should result in multiple passes over the consumer's
- * shard rather than writing the intermediate results onto the disk.
- *
- * */
+ * <p>
+ * This method is overridden to organize the output from one operator so in
+ * order to make it available to another operator running on a different
+ * node. There are several cases which have to be handled and which are
+ * identified by the {@link BOp#getEvaluationContext()}. In addition, we
+ * need to handle low latency and high data volume queries somewhat
+ * differently. Except for {@link BOpEvaluationContext#ANY}, all of these
+ * cases wind up writing the intermediate results onto a direct
+ * {@link ByteBuffer} and notifying the receiving service that there are
+ * intermediate results which it can pull when it is ready to process them.
+ * This pattern allows the receiver to impose flow control on the producer.
+ */
@Override
protected <E> int add(final int sinkId,
final IBlockingBuffer<IBindingSet[]> sink) {
@@ -132,27 +326,31 @@
throw new IllegalArgumentException();
switch (bop.getEvaluationContext()) {
- case ANY:
+ case ANY: {
return super.add(sinkId, sink);
+ }
case HASHED: {
/*
- * FIXME The sink self describes the nodes over which the
- * binding sets will be mapped and the hash function to be applied
- * so we look up those metadata and apply them to distributed the
- * binding sets across the nodes.
+ * @todo The sink must use annotations to describe the nodes over
+ * which the binding sets will be mapped and the hash function to be
+ * applied. Look up those annotations and apply them to distribute
+ * the binding sets across the nodes.
*/
throw new UnsupportedOperationException();
}
case SHARDED: {
/*
- * FIXME The sink must read or write on a shard so we map the
- * binding sets across the access path for the sink.
+ * The sink must read or write on a shard so we map the binding sets
+ * across the access path for the sink.
*
* @todo For a pipeline join, the predicate is the right hand
* operator of the sink. This might be true for INSERT and DELETE
- * operators as well.
+ * operators as well. [It is not, but make it so and document this
+ * pattern or have a common interface method which returns the
+ * IPredicate regardless of whether it is an operand or an
+ * annotation.]
*
- * @todo IKeyOrder tells us which index will be used and should be
+ * Note: IKeyOrder tells us which index will be used and should be
* set on the predicate by the join optimizer.
*
* @todo Use the read or write timestamp depending on whether the
@@ -163,28 +361,21 @@
* data contained in the sink (in fact, we should just process the
* sink data in place).
*/
- final IPredicate<E> pred = null; // @todo
- final IKeyOrder<E> keyOrder = null; // @todo
- final long timestamp = getReadTimestamp(); // @todo
+ @SuppressWarnings("unchecked")
+ final IPredicate<E> pred = ((IShardwisePipelineOp) bop).getPredicate();
+ final IKeyOrder<E> keyOrder = pred.getKeyOrder();
+ final long timestamp = getReadTimestamp(); // @todo read vs write timestamp.
final int capacity = 1000;// @todo
+ final int capacity2 = 1000;// @todo
final MapBindingSetsOverShardsBuffer<IBindingSet, E> mapper = new MapBindingSetsOverShardsBuffer<IBindingSet, E>(
getFederation(), pred, keyOrder, timestamp, capacity) {
-
- @Override
- IBuffer<IBindingSet> newBuffer(PartitionLocator locator) {
- // TODO Auto-generated method stub
- return null;
- }
-
+ @Override
+ IBuffer<IBindingSet> newBuffer(PartitionLocator locator) {
+ return new BlockingBuffer<IBindingSet>(capacity2);
+ }
};
/*
* Map the binding sets over shards.
- *
- * FIXME The buffers created above need to become associated with
- * this query as resources of the query. Once we are done mapping
- * the binding sets over the shards, the target node for each buffer
- * needs to be set an RMI message to let it know that there is a
- * chunk available for it for the target operator.
*/
{
final IAsynchronousIterator<IBindingSet[]> itr = sink
@@ -201,18 +392,52 @@
sink.close();
}
}
+ /*
+ * The allocation context.
+ *
+ * @todo use (queryId, serviceId, sinkId) when the target bop is
+ * high volume operator (this requires annotation by the query
+ * planner of the operator tree).
+ */
+ final IAllocationContext allocationContext = getAllocationContext(new QueryContext(
+ getQueryId()));
+
+ /*
+ * Generate the output chunks and notify the receivers.
+ *
+ * @todo This stage should probably be integrated with the stage
+ * which maps the binding sets over the shards (immediately above)
+ * to minimize copying or visiting in the data.
+ */
+ for (Map.Entry<PartitionLocator, IBuffer<IBindingSet>> e : mapper
+ .getSinks().entrySet()) {
+
+ final PartitionLocator locator = e.getKey();
+
+ final IBuffer<IBindingSet> shardSink = e.getValue();
+
+ // FIXME harmonize IBuffer<IBindingSet> vs IBuffer<IBindingSet[]>
+// sendOutputChunkReadyMessage(newOutputChunk(locator
+// .getDataServiceUUID(), sinkId, allocationContext,
+// shardSink));
+ throw new UnsupportedOperationException();
+ }
- throw new UnsupportedOperationException();
}
case CONTROLLER: {
- final IQueryClient clientProxy = getQueryController();
+ /*
+ * Format the binding sets onto a ByteBuffer and publish that
+ * ByteBuffer as a manager resource for the query and notify the
+ * query controller that data is available for it.
+ */
-// getQueryEngine().getResourceService().port;
-//
-// clientProxy.bufferReady(clientProxy, serviceAddr, getQueryId(), sinkId);
+ final IAllocationContext allocationContext = getAllocationContext(new QueryContext(
+ getQueryId()));
- throw new UnsupportedOperationException();
+ sendOutputChunkReadyMessage(newOutputChunk(queryControllerUUID,
+ sinkId, allocationContext, sink));
+
}
default:
throw new AssertionError(bop.getEvaluationContext());
@@ -220,4 +445,193 @@
}
+ /**
+ * Create an {@link OutputChunk} from some intermediate results.
+ *
+ * @param serviceUUID
+ * The {@link UUID} of the {@link IQueryPeer} who is the
+ * recipient.
+ * @param sinkId
+ * The identifier of the target {@link BOp}.
+ * @param allocationContext
+ * The allocation context within which the {@link ByteBuffer}s
+ * will be managed for this {@link OutputChunk}.
+ * @param source
+ * The binding sets to be formatted onto a buffer.
+ *
+ * @return The {@link OutputChunk}.
+ */
+ protected OutputChunk newOutputChunk(
+ final UUID serviceUUID,
+ final int sinkId,
+ final IAllocationContext allocationContext,
+ final IBlockingBuffer<IBindingSet[]> source) {
+
+ if (serviceUUID == null)
+ throw new IllegalArgumentException();
+
+ if (allocationContext == null)
+ throw new IllegalArgumentException();
+
+ if (source == null)
+ throw new IllegalArgumentException();
+
+ int nbytes = 0;
+
+ final List<IAllocation> allocations = new LinkedList<IAllocation>();
+
+ final IAsynchronousIterator<IBindingSet[]> itr = source.iterator();
+
+ try {
+
+ while (itr.hasNext()) {
+
+ // Next chunk to be serialized.
+ final IBindingSet[] chunk = itr.next();
+
+ // serialize the chunk of binding sets.
+ final byte[] data = SerializerUtil.serialize(chunk);
+
+ // track size of the allocations.
+ nbytes += data.length;
+
+ // allocate enough space for those data.
+ final IAllocation[] tmp;
+ try {
+ tmp = allocationContext.alloc(data.length);
+ } catch (InterruptedException ex) {
+ throw new RuntimeException(ex);
+ }
+
+ // copy the data into the allocations.
+ DirectBufferPoolAllocator.put(data, tmp);
+
+ // append the new allocations.
+ allocations.addAll(Arrays.asList(tmp));
+
+ }
+
+ } finally {
+
+ itr.close();
+
+ }
+
+ return new OutputChunk(getQueryId(), serviceUUID, sinkId, nbytes,
+ allocations);
+
+ }
+
+ protected IQueryPeer getQueryPeer(final UUID serviceUUID) {
+
+ if (serviceUUID == null)
+ throw new IllegalArgumentException();
+
+ final IQueryPeer queryPeer;
+
+ if (serviceUUID.equals(queryControllerUUID)) {
+
+ // The target is the query controller.
+ queryPeer = getQueryController();
+
+ } else {
+
+ // The target is some data service.
+ queryPeer = getQueryEngine().getQueryPeer(serviceUUID);
+
+ }
+
+ return queryPeer;
+
+ }
+
+ /**
+ * Notify a remote {@link IQueryPeer} that data is available for it.
+ *
+ * @todo If the target for the {@link OutputChunk} is this node then just
+ * drop it onto the {@link QueryEngine}.
+ *
+ * @todo Report the #of bytes available with this message. However, first
+ * figure out if that if the #of bytes in this {@link OutputChunk} or
+ * across all {@link OutputChunk}s available for the target service
+ * and sink.
+ *
+ * @todo Consider a fast path with inline RMI based transfer for small sets
+ * of data. We might just serialize to a byte[] and send that directly
+ * using a different message to notify the {@link IQueryPeer}.
+ */
+ protected void sendOutputChunkReadyMessage(final OutputChunk outputChunk) {
+
+ try {
+
+ // The peer to be notified.
+ final IQueryPeer peerProxy = getQueryPeer(outputChunk.serviceId);
+
+ // The Internet address and port where the peer can read the data
+ // from this node.
+ final InetSocketAddress serviceAddr = getQueryEngine()
+ .getResourceService().getAddr();
+
+ peerProxy.bufferReady(getQueryController(), serviceAddr,
+ getQueryId(), outputChunk.sinkId);
+
+ } catch (RemoteException e) {
+
+ throw new RuntimeException(e);
+
+ }
+
+ }
+
+ /**
+ * A chunk of outputs.
+ *
+ * @todo We probably need to use the {@link DirectBufferPoolAllocator} to
+ * receive the chunks within the {@link ManagedResourceService} as
+ * well.
+ *
+ * @todo Release the allocations associated with each output chunk once it
+ * is received by the remote service.
+ * <p>
+ * When the query terminates all output chunks targeting any node
+ * EXCEPT the query controller should be immediately dropped.
+ * <p>
+ * If there is an error during query evaluation, then the output
+ * chunks for the query controller should be immediately dropped.
+ * <p>
+ * If the iterator draining the results on the query controller is
+ * closed, then the output chunks for the query controller should be
+ * immediately dropped.
+ *
+ * @todo There are a few things where the resource must be made available to
+ * more than one operator evaluation phase. The best examples are
+ * temporary graphs for parallel closure and large collections of
+ * graphIds for SPARQL "NAMED FROM DATA SET" extensions.
+ */
+ private static class OutputChunk {
+
+ final long queryId;
+
+ final UUID serviceId;
+
+ final int sinkId;
+
+ final int nbytes;
+
+ final List<IAllocation> allocations;
+
+ public OutputChunk(final long queryId, final UUID serviceId,
+ final int sinkId, final int nbytes,
+ final List<IAllocation> allocations) {
+
+ this.queryId = queryId;
+ this.serviceId = serviceId;
+ this.sinkId = sinkId;
+ this.nbytes = nbytes;
+ this.allocations = allocations;
+
+ }
+
+ }
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java 2010-09-08 20:11:49 UTC (rev 3523)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java 2010-09-08 20:52:07 UTC (rev 3524)
@@ -50,6 +50,7 @@
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.IConstraint;
import com.bigdata.bop.IPredicate;
+import com.bigdata.bop.IShardwisePipelineOp;
import com.bigdata.bop.IVariable;
import com.bigdata.bop.engine.BOpStats;
import com.bigdata.btree.BytesUtil;
@@ -93,7 +94,8 @@
* @todo Break the star join logic out into its own join operator and test
* suite.
*/
-public class PipelineJoin extends BindingSetPipelineOp {
+public class PipelineJoin<E> extends BindingSetPipelineOp implements
+ IShardwisePipelineOp<E> {
static private final Logger log = Logger.getLogger(PipelineJoin.class);
@@ -256,22 +258,36 @@
}
- protected BindingSetPipelineOp left() {
+ /**
+ * The left hand operator, which is the previous join in the pipeline join
+ * path.
+ */
+ public BindingSetPipelineOp left() {
return (BindingSetPipelineOp) get(0);
}
- protected IPredicate<?> right() {
+ /**
+ * The right hand operator, which is the {@link IPredicate}.
+ */
+ @SuppressWarnings("unchecked")
+ public IPredicate<E> right() {
- return (IPredicate<?>) get(1);
+ return (IPredicate<E>) get(1);
}
+
+ public IPredicate<E> getPredicate() {
+
+ return right();
+
+ }
/**
* @see Annotations#CONSTRAINTS
*/
- protected IConstraint[] constraints() {
+ public IConstraint[] constraints() {
return getProperty(Annotations.CONSTRAINTS, null/* defaultValue */);
@@ -280,7 +296,7 @@
/**
* @see Annotations#OPTIONAL
*/
- protected boolean isOptional() {
+ public boolean isOptional() {
return getProperty(Annotations.OPTIONAL, Annotations.DEFAULT_OPTIONAL);
@@ -289,7 +305,7 @@
/**
* @see Annotations#MAX_PARALLEL
*/
- protected int getMaxParallel() {
+ public int getMaxParallel() {
return getProperty(Annotations.MAX_PARALLEL, Annotations.DEFAULT_MAX_PARALLEL);
@@ -298,7 +314,7 @@
/**
* @see Annotations#SELECT
*/
- protected IVariable<?>[] variablesToKeep() {
+ public IVariable<?>[] variablesToKeep() {
return getProperty(Annotations.SELECT, null/* defaultValue */);
@@ -325,7 +341,7 @@
/**
* The join that is being executed.
*/
- final private PipelineJoin joinOp;
+ final private PipelineJoin<?> joinOp;
/**
* The constraint (if any) specified for the join operator.
@@ -450,7 +466,7 @@
* @param context
*/
public JoinTask(//
- final PipelineJoin joinOp,//
+ final PipelineJoin<?> joinOp,//
final BOpContext<IBindingSet> context
) {
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/InsertOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/InsertOp.java 2010-09-08 20:11:49 UTC (rev 3523)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/InsertOp.java 2010-09-08 20:52:07 UTC (rev 3524)
@@ -27,8 +27,6 @@
package com.bigdata.bop.mutation;
-import java.util.Arrays;
-import java.util.List;
import java.util.Map;
import java.util.concurrent.Callable;
import java.util.concurrent.FutureTask;
@@ -39,7 +37,7 @@
import com.bigdata.bop.BindingSetPipelineOp;
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.IPredicate;
-import com.bigdata.bop.IVariableOrConstant;
+import com.bigdata.bop.IShardwisePipelineOp;
import com.bigdata.bop.engine.BOpStats;
import com.bigdata.btree.ILocalBTreeView;
import com.bigdata.btree.ITupleSerializer;
@@ -59,7 +57,8 @@
* @param <E>
* The generic type of the elements written onto the index.
*/
-public class InsertOp<E> extends BindingSetPipelineOp {
+public class InsertOp<E> extends BindingSetPipelineOp implements
+ IShardwisePipelineOp<E> {
/**
*
@@ -69,12 +68,12 @@
public interface Annotations extends BindingSetPipelineOp.Annotations {
/**
- * An ordered {@link IVariableOrConstant}[]. Elements will be created
- * using the binding sets which flow through the operator and
- * {@link IRelation#newElement(java.util.List, IBindingSet)}.
+ * An {@link IPredicate}. The {@link IPredicate#asBound(IBindingSet)}
+ * predicate will be used to create the elements to be inserted into
+ * the relation.
*
- * @todo This should be an {@link IPredicate} and should be the right
- * hand operand just like for a JOIN.
+ * @see IPredicate#asBound(IBindingSet)
+ * @see IRelation#newElement(java.util.List, IBindingSet)
*/
String SELECTED = InsertOp.class.getName() + ".selected";
@@ -116,9 +115,8 @@
/**
* @see Annotations#SELECTED
*/
- public IVariableOrConstant<?>[] getSelected() {
+ public IPredicate<E> getPredicate() {
-// return (IVariableOrConstant<?>[]) getProperty(Annotations.SELECTED);
return getRequiredProperty(Annotations.SELECTED);
}
@@ -164,7 +162,7 @@
*/
private final IBlockingBuffer<IBindingSet[]> sink;
- private List<IVariableOrConstant<?>> selected;
+ private IPredicate<E> predicate;
private final IRelation<E> relation;
@@ -181,7 +179,7 @@
sink = context.getSink();
- selected = Arrays.asList(op.getSelected());
+ predicate = op.getPredicate();
relation = context.getWriteRelation(op.getRelation());
@@ -229,7 +227,7 @@
final IBindingSet bset = chunk[i];
- final E e = relation.newElement(selected, bset);
+ final E e = relation.newElement(predicate.args(), bset);
final byte[] key = keyOrder.getKey(keyBuilder, e);
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/io/DirectBufferPoolAllocator.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/io/DirectBufferPoolAllocator.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/io/DirectBufferPoolAllocator.java 2010-09-08 20:52:07 UTC (rev 3524)
@@ -0,0 +1,653 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Sep 8, 2010
+ */
+
+package com.bigdata.io;
+
+import java.nio.BufferOverflowException;
+import java.nio.ByteBuffer;
+import java.util.LinkedList;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.ReentrantLock;
+
+import com.bigdata.service.ResourceService;
+import com.bigdata.util.concurrent.Haltable;
+
+/**
+ * An allocator for {@link ByteBuffer} slices backed by direct
+ * {@link ByteBuffer}s allocated against a {@link DirectBufferPool}.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ *
+ * @todo Make the type of the identifier for the {@link IAllocation} generic
+ * using a factory pattern (we need {@link UUID} for scale-out, but the
+ * class could be reused for other purposes as well). [The allocation
+ * context identifier should continue to be an application specified
+ * object.]
+ */
+public class DirectBufferPoolAllocator {
+
+ /**
+ * The pool from which the direct {@link ByteBuffer}s are allocated.
+ */
+ private final DirectBufferPool directBufferPool;
+
+ /**
+ * The set of allocation contexts.
+ */
+ private final ConcurrentHashMap<Object/* key */, AllocationContext> allocationContexts = new ConcurrentHashMap<Object, AllocationContext>();
+
+ /**
+ * The set of {@link IAllocation} outstanding against the
+ * {@link #directBufferPool}.
+ */
+ private final ConcurrentHashMap<UUID, Allocation> allocations = new ConcurrentHashMap<UUID, Allocation>();
+
+ /**
+ * @todo Maybe replace this with a private {@link Haltable} (or extend
+ * {@link Haltable}) so we can test {@link Haltable#halted()} in
+ * critical methods? If we expose the {@link Haltable} then the
+ * {@link ResourceService} can also check it to see whether all
+ * allocations have been invalidated. However, that will not help us
+ * to invalidate a specific {@link IAllocationContext}. For that
+ * purpose we would need to do pretty much the same thing recursively.
+ */
+ private final AtomicBoolean open = new AtomicBoolean(true);
+
+ /**
+ *
+ * @param pool
+ * The pool from which the direct {@link ByteBuffer}s are
+ * allocated.
+ */
+ public DirectBufferPoolAllocator(final DirectBufferPool pool) {
+
+ this.directBufferPool = pool;
+
+ }
+
+ /**
+ * Extended to {@link #close()} the allocator.
+ */
+ @Override
+ protected void finalize() throws Throwable {
+
+ close();
+
+ super.finalize();
+
+ }
+
+ /**
+ * Releases all {@link AllocationContext}s and all direct {@link ByteBuffer}
+ * s which they are using.
+ */
+ public void close() {
+
+ if (open.compareAndSet(true/* expect */, false/* update */)) {
+
+ for (AllocationContext c : allocationContexts.values()) {
+
+ c.release();
+
+ }
+
+ }
+
+ }
+
+ /**
+ * The maximum #of bytes in a single {@link IAllocation}.
+ */
+ public int getMaxSlotSize() {
+
+ return directBufferPool.getBufferCapacity();
+
+ }
+
+ /**
+ * Return an allocation context for the key. If none exists for that key,
+ * then one is atomically created and returned.
+ *
+ * @param key
+ * A key which uniquely identifies that context. The key will be
+ * inserted into a hash table and therefore must have appropriate
+ * hashCode() and equals() methods.
+ *
+ * @return The allocation context.
+ */
+ public IAllocationContext getAllocationContext(final Object key) {
+
+ AllocationContext c = allocationContexts.get(key);
+
+ if (c == null) {
+
+ final AllocationContext t = allocationContexts.putIfAbsent(key,
+ c = new AllocationContext(key));
+
+ if (t != null) {
+
+ // lost the race to another thread.
+ c = t;
+
+ }
+
+ }
+
+ return c;
+
+ }
+
+ /**
+ * Return the allocation associated with that id.
+ *
+ * @param id
+ * The allocation identifier.
+ *
+ * @return The allocation -or- <code>null</code> if there is no such
+ * allocation.
+ */
+ public IAllocation getAllocation(final UUID id) {
+
+ return allocations.get(id);
+
+ }
+
+// /**
+// * A direct {@link ByteBuffer} allocated from the {@link #directBufferPool}
+// * together with the identifier assigned to that {@link ByteBuffer} (we can
+// * not directly insert {@link ByteBuffer}s into the keys of a hash map since
+// * their hash code is a function of their content).
+// */
+// private class DirectBufferAllocation {
+//
+// private final Long id;
+//
+// private final ByteBuffer directBuffer;
+//
+// public DirectBufferAllocation(final Long id,
+// final ByteBuffer directBuffer) {
+//
+// if (id == null)
+// throw new IllegalArgumentException();
+//
+// if (directBuffer == null)
+// throw new IllegalArgumentException();
+//
+// this.id = id;
+//
+// this.directBuffer = directBuffer;
+//
+// }
+//
+// }
+
+ /**
+ * An allocation context links some application specified key with a list
+ * of direct {@link ByteBuffer}s on which allocations have been made by
+ * the application.
+ */
+ public interface IAllocationContext {
+
+ /**
+ * Allocate a series of {@link ByteBuffer} slices on which the
+ * application may write data. The application is encouraged to maintain
+ * the order of the allocations in the array in order to preserve the
+ * ordering of data written onto those allocation.
+ *
+ * @param nbytes
+ * The #of bytes required.
+ *
+ * @return The {@link UUID}s of those allocations.
+ *
+ * @throws InterruptedException
+ */
+ IAllocation[] alloc(int nbytes) throws InterruptedException;
+
+ /**
+ * Release all allocations made against this allocation context.
+ */
+ void release();
+
+ }
+
+ /**
+ * An allocation against a direct {@link ByteBuffer}.
+ */
+ public interface IAllocation {
+
+ /** The allocation identifier. */
+ public UUID getId();
+
+ /**
+ * The allocated {@link ByteBuffer#slice()}.
+ */
+ public ByteBuffer getSlice();
+
+ /**
+ * Release this allocation.
+ * <p>
+ * Note: The implementation is encouraged to release the associated
+ * direct {@link ByteBuffer} if there are no remaining allocations
+ * against it and MAY made the slice of the buffer available for
+ * reallocation.
+ * <p>
+ * Note: An {@link InterruptedException} MAY be thrown. This allows us
+ * to handle cases where a concurrent process (such as a query) was
+ * halted and its component threads were interrupted. By looking for the
+ * interrupt, we can avoid attempts to release an allocation in some
+ * thread where the entire {@link IAllocationContext} has already been
+ * released by another thread.
+ *
+ * @throws InterruptedException
+ */
+ public void release() throws InterruptedException;
+
+ }
+
+ /**
+ * An allocation against a direct {@link ByteBuffer}.
+ */
+ // Note: package private for the unit tests.
+ /*private*/ class Allocation implements IAll...
[truncated message content] |
|
From: <ble...@us...> - 2010-09-08 20:11:55
|
Revision: 3523
http://bigdata.svn.sourceforge.net/bigdata/?rev=3523&view=rev
Author: blevine218
Date: 2010-09-08 20:11:49 +0000 (Wed, 08 Sep 2010)
Log Message:
-----------
Added dependency on distribution tarball
Added support for starting/stopping services needed for integration tests
Modified Paths:
--------------
branches/maven_scaleout/pom.xml
Modified: branches/maven_scaleout/pom.xml
===================================================================
--- branches/maven_scaleout/pom.xml 2010-09-08 20:10:34 UTC (rev 3522)
+++ branches/maven_scaleout/pom.xml 2010-09-08 20:11:49 UTC (rev 3523)
@@ -1,5 +1,5 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.bigdata</groupId>
@@ -28,7 +28,11 @@
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
- <configuration><!-- Use Java 1.5 keywords, target running on a 1.5 JVM -->
+ <configuration>
+ <!--
+ Use Java 1.5 keywords, target running on a
+ 1.5 JVM
+ -->
<source>1.5</source>
<target>1.5</target>
<encoding>${project.build.sourceEncoding}</encoding>
@@ -64,6 +68,13 @@
<version>2.2-beta-5</version>
</plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-antrun-plugin</artifactId>
+ <version>1.4</version>
+ </plugin>
+
+
</plugins>
</pluginManagement>
@@ -72,32 +83,32 @@
<dependencyManagement>
<dependencies>
<dependency>
- <groupId>junit</groupId>
- <artifactId>junit</artifactId>
- <version>3.8.1</version>
- <scope>test</scope>
- </dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>3.8.1</version>
+ <scope>test</scope>
+ </dependency>
- <!-- slf4j -->
- <dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-api</artifactId>
- <version>1.6.1</version>
- </dependency>
+ <!-- slf4j -->
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ <version>1.6.1</version>
+ </dependency>
</dependencies>
</dependencyManagement>
<reporting>
<plugins>
-
+
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-report-plugin</artifactId>
<version>2.5</version>
</plugin>
-
+
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
@@ -118,24 +129,27 @@
<repositories>
- <!-- TODO: There may not be enough sub-modules that use Sesame/Aduna to warrant putting this here. -->
+ <!--
+ TODO: There may not be enough sub-modules that use
+ Sesame/Aduna to warrant putting this here.
+ -->
<repository>
- <releases>
- <enabled>true</enabled>
- </releases>
- <snapshots>
- <enabled>false</enabled>
- </snapshots>
- <id>aduna-opensource.releases</id>
- <name>Aduna Open Source - Maven releases</name>
- <url>http://repo.aduna-software.org/maven2/releases</url>
+ <releases>
+ <enabled>true</enabled>
+ </releases>
+ <snapshots>
+ <enabled>false</enabled>
+ </snapshots>
+ <id>aduna-opensource.releases</id>
+ <name>Aduna Open Source - Maven releases</name>
+ <url>http://repo.aduna-software.org/maven2/releases</url>
</repository>
</repositories>
<profiles>
-
+
<profile>
<id>bigdata-clover</id>
<activation>
@@ -146,29 +160,25 @@
<plugin>
<groupId>com.atlassian.maven.plugins</groupId>
<artifactId>maven-clover2-plugin</artifactId>
- <version>3.0.1</version>
+ <version>3.0.1</version>
<configuration>
<jdk>1.5</jdk>
<!--
- To use clover with bigdata, you should add the following to your maven settings file, .m2/settings.xml
-
- <profile>
- <id>bigdata-clover</id>
- <activation>
- <activeByDefault>true</activeByDefault>
- </activation>
- <properties>
- <maven.clover.licenseLocation>/your/path/to/clover.license</maven.clover.licenseLocation>
- </properties>
- </profile>
-
- -->
+ To use clover with bigdata, you should
+ add the following to your maven settings
+ file, .m2/settings.xml <profile>
+ <id>bigdata-clover</id> <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation> <properties>
+ <maven.clover.licenseLocation>/your/path/to/clover.license</maven.clover.licenseLocation>
+ </properties> </profile>
+ -->
</configuration>
</plugin>
</plugins>
- </build>
+ </build>
</profile>
-
+
</profiles>
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <ble...@us...> - 2010-09-08 20:10:41
|
Revision: 3522
http://bigdata.svn.sourceforge.net/bigdata/?rev=3522&view=rev
Author: blevine218
Date: 2010-09-08 20:10:34 +0000 (Wed, 08 Sep 2010)
Log Message:
-----------
Added dependency on distribution tarball
Added support for starting/stopping services needed for integration tests
Modified Paths:
--------------
branches/maven_scaleout/bigdata-integ/pom.xml
Modified: branches/maven_scaleout/bigdata-integ/pom.xml
===================================================================
--- branches/maven_scaleout/bigdata-integ/pom.xml 2010-09-08 20:09:11 UTC (rev 3521)
+++ branches/maven_scaleout/bigdata-integ/pom.xml 2010-09-08 20:10:34 UTC (rev 3522)
@@ -13,15 +13,63 @@
<artifactId>bigdata-integration-test</artifactId>
<name>bigdata Integration Tests</name>
+ <properties>
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+ <project.build.deployDirectory>${project.build.directory}/deploy</project.build.deployDirectory>
+ </properties>
+
+
<build>
<plugins>
<plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>unpack</id>
+ <goals>
+ <goal>unpack</goal>
+ </goals>
+ <configuration>
+ <artifactItems>
+ <artifactItem>
+ <groupId>com.nokia.dataos.rds.bigdata</groupId>
+ <artifactId>bigdata-dist</artifactId>
+ <version>0.83.2-SNAPSHOT</version>
+ <type>tgz</type>
+ <outputDirectory>${project.build.deployDirectory}</outputDirectory>
+ </artifactItem>
+ </artifactItems>
+ <overWriteSnapshots>true</overWriteSnapshots>
+ <overWriteReleases>true</overWriteReleases>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <configuration>
+
+ <!--
+ Make sure no tests are run as unit tests by
+ surefire. We want everything to run via failsafe
+ -->
+
+ <excludes>
+ <exclude>**/**</exclude>
+ </excludes>
+ </configuration>
+ </plugin>
+
+ <plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>failsafe-maven-plugin</artifactId>
<version>2.4.3-alpha-1</version>
<configuration>
<includes>
- <include>**/*Test.java</include>
+ <include>**/*.java</include>
</includes>
</configuration>
<executions>
@@ -33,6 +81,44 @@
</execution>
</executions>
</plugin>
+
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-antrun-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>start-test-services</id>
+ <phase>pre-integration-test</phase>
+ <configuration>
+ <tasks>
+ <ant
+ antfile="${project.build.testOutputDirectory}/test.xml"
+ target="hello" />
+ </tasks>
+ </configuration>
+ <goals>
+ <goal>run</goal>
+ </goals>
+ </execution>
+
+ <execution>
+ <id>stop-test-services</id>
+ <phase>post-integration-test</phase>
+ <configuration>
+ <tasks>
+ <ant
+ antfile="${project.build.testOutputDirectory}/test.xml"
+ target="hello" />
+ </tasks>
+ </configuration>
+ <goals>
+ <goal>run</goal>
+ </goals>
+ </execution>
+ </executions>
+
+ </plugin>
+
</plugins>
</build>
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <fko...@us...> - 2010-09-08 20:09:19
|
Revision: 3521
http://bigdata.svn.sourceforge.net/bigdata/?rev=3521&view=rev
Author: fkoliver
Date: 2010-09-08 20:09:11 +0000 (Wed, 08 Sep 2010)
Log Message:
-----------
Remove dependency on cweb-junit-*.jar by copying the sources into this
project. This is a temporary step: the extensions should be eventually
be removed as we move to junit 4.x.
Modified Paths:
--------------
branches/maven_scaleout/bigdata-core/pom.xml
branches/maven_scaleout/bigdata-core/src/test/deploy/testing/conf/standalone/ServerStarter.config
branches/maven_scaleout/bigdata-core/thirdparty/maven.xml
Added Paths:
-----------
branches/maven_scaleout/bigdata-core/src/test/java/junit/
branches/maven_scaleout/bigdata-core/src/test/java/junit/extensions/
branches/maven_scaleout/bigdata-core/src/test/java/junit/extensions/proxy/
branches/maven_scaleout/bigdata-core/src/test/java/junit/extensions/proxy/IProxyTest.java
branches/maven_scaleout/bigdata-core/src/test/java/junit/extensions/proxy/ProxyTestSuite.java
branches/maven_scaleout/bigdata-core/src/test/java/junit/extensions/proxy/package.html
branches/maven_scaleout/bigdata-core/src/test/java/junit/framework/
branches/maven_scaleout/bigdata-core/src/test/java/junit/framework/TestCase2.java
branches/maven_scaleout/bigdata-core/src/test/java/junit/framework/package.html
Removed Paths:
-------------
branches/maven_scaleout/bigdata-core/thirdparty/lib/cweb-junit-ext-1.1-b3-dev.jar
Property Changed:
----------------
branches/maven_scaleout/bigdata-core/
Property changes on: branches/maven_scaleout/bigdata-core
___________________________________________________________________
Added: svn:ignore
+ target
Modified: branches/maven_scaleout/bigdata-core/pom.xml
===================================================================
--- branches/maven_scaleout/bigdata-core/pom.xml 2010-09-08 14:52:29 UTC (rev 3520)
+++ branches/maven_scaleout/bigdata-core/pom.xml 2010-09-08 20:09:11 UTC (rev 3521)
@@ -216,12 +216,6 @@
</dependency>
<dependency>
<groupId>${thirdParty.groupId}</groupId>
- <artifactId>cweb-junit-ext</artifactId>
- <version>1.1.0-b3-dev</version>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>${thirdParty.groupId}</groupId>
<artifactId>cweb-commons</artifactId>
<version>1.1.0-b2-dev</version>
</dependency>
Modified: branches/maven_scaleout/bigdata-core/src/test/deploy/testing/conf/standalone/ServerStarter.config
===================================================================
--- branches/maven_scaleout/bigdata-core/src/test/deploy/testing/conf/standalone/ServerStarter.config 2010-09-08 14:52:29 UTC (rev 3520)
+++ branches/maven_scaleout/bigdata-core/src/test/deploy/testing/conf/standalone/ServerStarter.config 2010-09-08 20:09:11 UTC (rev 3521)
@@ -59,7 +59,6 @@
libdir+"icu4j-3_6.jar"+File.pathSeparator+
// test suites only!
libdir+"junit-3.8.1.jar"+File.pathSeparator+
- libdir+"cweb-junit-ext-1.1-b2-dev.jar"+File.pathSeparator+
// main bigdata JAR.
//libdir+
"bigdata-core.jar"
Added: branches/maven_scaleout/bigdata-core/src/test/java/junit/extensions/proxy/IProxyTest.java
===================================================================
--- branches/maven_scaleout/bigdata-core/src/test/java/junit/extensions/proxy/IProxyTest.java (rev 0)
+++ branches/maven_scaleout/bigdata-core/src/test/java/junit/extensions/proxy/IProxyTest.java 2010-09-08 20:09:11 UTC (rev 3521)
@@ -0,0 +1,77 @@
+/**
+
+The Notice below must appear in each file of the Source Code of any
+copy you distribute of the Licensed Product. Contributors to any
+Modifications may add their own copyright notices to identify their
+own contributions.
+
+License:
+
+The contents of this file are subject to the CognitiveWeb Open Source
+License Version 1.1 (the License). You may not copy or use this file,
+in either source code or executable form, except in compliance with
+the License. You may obtain a copy of the License from
+
+ http://www.CognitiveWeb.org/legal/license/
+
+Software distributed under the License is distributed on an AS IS
+basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
+the License for the specific language governing rights and limitations
+under the License.
+
+Copyrights:
+
+Portions created by or assigned to CognitiveWeb are Copyright
+(c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact
+information for CognitiveWeb is available at
+
+ http://www.CognitiveWeb.org
+
+Portions Copyright (c) 2002-2003 Bryan Thompson.
+
+Acknowledgements:
+
+Special thanks to the developers of the Jabber Open Source License 1.0
+(JOSL), from which this License was derived. This License contains
+terms that differ from JOSL.
+
+Special thanks to the CognitiveWeb Open Source Contributors for their
+suggestions and support of the Cognitive Web.
+
+Modifications:
+
+*/
+
+package junit.extensions.proxy;
+
+import junit.framework.*;
+
+/**
+ * A {@link Test} that holds a reference to a delegate. Normally the
+ * delegate will extend {@link TestCase} to provide, possibly
+ * persistent, implementation specific logic for establishing test
+ * fixtures. The delegate is normally an instance of your concrete
+ * implementation specific test harness with access to any required
+ * configuration data, e.g., a properties file.
+ */
+
+public interface IProxyTest
+ extends Test
+{
+
+ /**
+ * Sets the delegate. {@link ProxyTestSuite} uses this method to
+ * set the delegate on each test class instance that it creates
+ * that implements the {@link IProxyTest} interface.
+ */
+
+ public void setDelegate( Test delegate );
+
+ /**
+ * Returns the reference to the delegate or <code>null</code> if
+ * the delegate was not established.
+ */
+
+ public Test getDelegate();
+
+}
Property changes on: branches/maven_scaleout/bigdata-core/src/test/java/junit/extensions/proxy/IProxyTest.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: branches/maven_scaleout/bigdata-core/src/test/java/junit/extensions/proxy/ProxyTestSuite.java
===================================================================
--- branches/maven_scaleout/bigdata-core/src/test/java/junit/extensions/proxy/ProxyTestSuite.java (rev 0)
+++ branches/maven_scaleout/bigdata-core/src/test/java/junit/extensions/proxy/ProxyTestSuite.java 2010-09-08 20:09:11 UTC (rev 3521)
@@ -0,0 +1,395 @@
+/**
+
+The Notice below must appear in each file of the Source Code of any
+copy you distribute of the Licensed Product. Contributors to any
+Modifications may add their own copyright notices to identify their
+own contributions.
+
+License:
+
+The contents of this file are subject to the CognitiveWeb Open Source
+License Version 1.1 (the License). You may not copy or use this file,
+in either source code or executable form, except in compliance with
+the License. You may obtain a copy of the License from
+
+ http://www.CognitiveWeb.org/legal/license/
+
+Software distributed under the License is distributed on an AS IS
+basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
+the License for the specific language governing rights and limitations
+under the License.
+
+Copyrights:
+
+Portions created by or assigned to CognitiveWeb are Copyright
+(c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact
+information for CognitiveWeb is available at
+
+ http://www.CognitiveWeb.org
+
+Portions Copyright (c) 2002-2003 Bryan Thompson.
+
+Acknowledgements:
+
+Special thanks to the developers of the Jabber Open Source License 1.0
+(JOSL), from which this License was derived. This License contains
+terms that differ from JOSL.
+
+Special thanks to the CognitiveWeb Open Source Contributors for their
+suggestions and support of the Cognitive Web.
+
+Modifications:
+
+*/
+
+package junit.extensions.proxy;
+
+import junit.framework.Test;
+import junit.framework.TestCase;
+import junit.framework.TestSuite;
+
+import org.apache.log4j.Logger;
+
+/**
+ * <p>
+ * A simple wrapper around {@link TestSuite} that permits the caller to specify
+ * the delegate {@link Test} for either directly or recursively contained
+ * {@link IProxyTest}s added to a {@link ProxyTestSuite}. There are three cases
+ * for junit:
+ * <ol>
+ * <li>An instantiated test. This is an instance of some class that extends
+ * {@link TestCase}. If the class implements {@link IProxyTest} then the
+ * <i>delegate </i> will be set on the test instance and will be available when
+ * that test runs.</li>
+ * <li>A test case. This is a class named to
+ * {@link TestSuite#addTestSuite(java.lang.Class)}. That method scans the test
+ * case class and generates an instantiated test for each conforming test method
+ * identified in the test case. The delegate is then set per above.</li>
+ * <li>A test suite. The delegate declared in a {@link ProxyTestSuite}
+ * constructor will be flowed down to each test added either directly or
+ * indirectly to that {@link ProxyTestSuite}. The various constructors all
+ * invoke this method to flow down the delegate. In addition, the
+ * {@link #addTest(Test)} and {@link #addTestSuite(Class)} methods also
+ * invoke this method to flow down the delegate to instantiated tests that
+ * implement {@link IProxyTest}.</li>
+ * </ol>
+ * </p>
+ * <p>
+ * The only case when flow down does not occur is when you have created a
+ * standard test suite, addeded it to a proxy test suite, and <em>then</em>
+ * you add additional tests to the standard test suite. There is no event
+ * notification model in junit and this action is not noticed by the {@link
+ * ProxyTestSuite}.
+ * </p>
+ */
+
+public class ProxyTestSuite
+ extends TestSuite
+{
+
+ /**
+ * The {@link Logger} is named for this class.
+ */
+
+ protected static final Logger log = Logger.getLogger
+ ( ProxyTestSuite.class
+ );
+
+ /**
+ * The delegate that will be used by each {@link IProxyTest}added to this
+ * {@link ProxyTestSuite}.
+ */
+
+ private final Test m_delegate;
+
+ /**
+ * <p>
+ * Invoked automatically by the constructors.
+ * </p>
+ *
+ * @exception IllegalArgumentException
+ * if the <i>delegate </i> is <code>null</code>.
+ */
+
+ final private void checkDelegate( Test delegate )
+ {
+
+ if (delegate == null) {
+
+ throw new IllegalArgumentException("The delegate is null.");
+
+ }
+
+// if (m_delegate != null) {
+//
+// throw new IllegalStateException("The delegate is already set.");
+//
+// }
+
+ log.debug
+ ( "Delegate is "+delegate.getClass().getName()
+ );
+
+ }
+
+ /**
+ * Returns the <i>delegate </i> supplied to the constructor.
+ */
+
+ public Test getDelegate() {
+
+ return m_delegate;
+
+ }
+
+ /**
+ * Creates an empty unnamed test suite. The <i>delegate </i> will be
+ * assigned to tests added to this test suite that implement
+ * {@link IProxyTest}.
+ *
+ * @param delegate
+ * The delegate (non-null).
+ */
+
+ public ProxyTestSuite( Test delegate )
+ {
+
+ super();
+
+ checkDelegate( delegate );
+
+ m_delegate = delegate;
+
+ }
+
+ /**
+ * Creates a named test suite and populates it with test instances
+ * identified by scanning the <i>testClass </i>. The <i>delegate </i> will
+ * be assigned to tests added to this test suite that implement
+ * {@link IProxyTest}, including those created from <i>testClass </i>.
+ *
+ * @param delegate
+ * The delegate (non-null).
+ * @param testClass
+ * A class containing one or more tests.
+ * @param name
+ * The name of the test suite (optional).
+ */
+
+ public ProxyTestSuite( Test delegate, Class testClass, String name )
+ {
+
+ /*
+ * Let the super class do all the heavy lifting. Note that it can not set
+ * the delegate for us since we can not set our private fields until after
+ * we have invoked the super class constructor.
+ */
+
+ super( testClass, name );
+
+ checkDelegate( delegate );
+
+ m_delegate = delegate;
+
+ /*
+ * Apply delegate to all instantiated tests that were created by
+ * junit from testClass and which implement IProxyTest.
+ */
+ flowDown(this);
+
+ }
+
+ /**
+ * Creates an unnamed test suite and populates it with test instances
+ * identified by scanning the <i>testClass </i>. The <i>delegate </i> will
+ * be assigned to tests added to this test suite that implement
+ * {@link IProxyTest}, including those created from <i>testClass </i>.
+ *
+ * @param delegate
+ * The delegate (non-null).
+ * @param testClass
+ * A class containing one or more tests.
+ */
+
+ public ProxyTestSuite( Test delegate, Class testClass )
+ {
+
+ /*
+ * Let the super class do all the heavy lifting. Note that it can not set
+ * the delegate for us since we can not set our private fields until after
+ * we have invoked the super class constructor.
+ */
+
+ super( testClass );
+
+ // Remember the delegate.
+
+ checkDelegate( delegate );
+
+ m_delegate = delegate;
+
+ /*
+ * Apply delegate to all instantiated tests that were created by
+ * junit from testClass and which implement IProxyTest.
+ */
+ flowDown(this);
+
+ }
+
+ /**
+ * Creates an empty named test suite. The declared will be assigned to tests
+ * added to this test suite that implement {@link IProxyTest}.
+ *
+ * @param delegate
+ * The delegate (non-null).
+ * @param name
+ * The test suite name (optional).
+ */
+ public ProxyTestSuite( Test delegate, String name )
+ {
+
+ super( name );
+
+ checkDelegate( delegate );
+
+ m_delegate = delegate;
+
+ }
+
+ /**
+ * We override the implementation of {@link
+ * TestSuite#addTestSuite( Class theClass )} to wrap the
+ * <i>testClass</i> in another instance of this {@link
+ * ProxyTestSuite} class using the same delegate that was provided
+ * to our constructor. This causes the delegation to be inherited
+ * by any recursively contained <i>testClass</i> which implements
+ * {@link IProxyTest}.
+ */
+
+ public void addTestSuite( Class testClass )
+ {
+
+ if( m_delegate == null ) {
+
+ /*
+ * The delegate will be null if this method gets invoked from the
+ * super class constructor since that will happen before we have a
+ * chance to set the [m_delegate] field. The constructors handle
+ * this situation by explictly flowing down the delegate after the
+ * super class constructor has been invoked.
+ */
+
+ super.addTestSuite( testClass );
+
+ return;
+
+ }
+
+ // This is the default implementation from TestSuite:
+ //
+ // addTest(new TestSuite(testClass));
+ //
+ // Our implementation just substitutes new ProxyTestSuite(
+ // delegate, testClass ) for new TestSuite( testClass ).
+ //
+
+ ProxyTestSuite proxyTestSuite = new ProxyTestSuite
+ ( m_delegate,
+ testClass
+ );
+
+ addTest( proxyTestSuite );
+
+ }
+
+ /**
+ * If the suite is not a {@link ProxyTestSuite}, then the tests in the
+ * suite are recursively enumerated and a proxy test suite is created with
+ * the same name and tests. This ensures that the common delegate flows down
+ * through all tests even when using the traditional
+ * <code>static Test suite() {...}</code> construction.
+ *
+ * @param test
+ * A test.
+ */
+ public void addTest(Test test) {
+
+ if( m_delegate == null ) {
+
+ /*
+ * The delegate will be null if this method gets invoked from the
+ * super class constructor since that will happen before we have a
+ * chance to set the [m_delegate] field. The constructors handle
+ * this situation by explictly flowing down the delegate after the
+ * super class constructor has been invoked.
+ */
+
+ super.addTest( test );
+
+ return;
+
+ }
+
+ if ( ! (test instanceof ProxyTestSuite)) {
+
+ /*
+ * Flow down the delegate to any container IProxyTest instances.
+ */
+
+ flowDown( test );
+
+ }
+
+ super.addTest( test );
+
+ }
+
+ protected void flowDown(Test t) {
+
+ if (m_delegate == null) {
+
+ throw new AssertionError("delegate is not set.");
+
+ }
+
+ if ( t instanceof TestSuite ) {
+
+ flowDown( (TestSuite) t );
+
+ } else if (t instanceof IProxyTest) {
+
+ log.debug("Setting delegate on " + t.getClass() + " to "
+ + m_delegate.getClass());
+
+ ((IProxyTest) t).setDelegate(m_delegate);
+
+ }
+
+ }
+
+ /**
+ * <p>
+ * Sets the delegate on each instantiated {@link Test} that implements
+ * {@link IProxyTest}.
+ * </p>
+ */
+
+ protected void flowDown( TestSuite suite ) {
+
+ if( m_delegate == null ) {
+
+ throw new AssertionError("delegate is not set.");
+
+ }
+
+ for( java.util.Enumeration e= suite.tests(); e.hasMoreElements(); ) {
+
+ Test t = (Test)e.nextElement();
+
+ flowDown( t );
+
+ }
+
+ }
+
+}
Property changes on: branches/maven_scaleout/bigdata-core/src/test/java/junit/extensions/proxy/ProxyTestSuite.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: branches/maven_scaleout/bigdata-core/src/test/java/junit/extensions/proxy/package.html
===================================================================
--- branches/maven_scaleout/bigdata-core/src/test/java/junit/extensions/proxy/package.html (rev 0)
+++ branches/maven_scaleout/bigdata-core/src/test/java/junit/extensions/proxy/package.html 2010-09-08 20:09:11 UTC (rev 3521)
@@ -0,0 +1,146 @@
+<HTML>
+<HEAD>
+<TITLE>Proxy test suites</TITLE>
+</HEAD>
+<BODY>
+<H1>Integration testing with proxy test suites.</H1>
+<P>This package provides support for suites of <EM>proxy</EM> tests that
+share a common test <EM>delegate</EM>. Normally the proxy tests and the
+delegate will implement a common interface. The implementation of the
+interface on the proxy test base class should route the behaviors to the
+delegate test. This mechanism can be used to run an entire test suite
+against a specific configuration for integration testing. Since the same
+delegate instance is used for all tests in the suite, you can do things
+such as establish a shared database connection that is used for the
+entire test suite, i.e., you can do <EM>integration testing</EM>.</P>
+<H2>Defining a configuration using getProperties()</H2>
+<P>The pattern that I have found to be most useful for integration
+testing of different configurations is to define a <CODE>public
+Properties getProperties()</CODE> method that is implemented by the <EM>delegate</EM>
+and accessed by various integration test set up and tear down methods
+implemented by the delegate. For example:</P>
+<PRE>
+ public Properties getProperties() {
+ if( m_properties == null ) {
+ /*
+ * Read and cache System properties.
+ */
+ m_properties = System.getProperties();
+ }
+ /*
+ * Wrap up the cached properties so that they are not modifable by the
+ * caller (no side effects between calls).
+ */
+ return new Properties( m_properties );
+ }
+</PRE>
+<p>This approach reads properties from the environement, so you can
+define property values using JVM command line arguments <CODE>-Dname=value</CODE>.
+You can also extend the {@link junit.framework.TestCase2} class and use
+its <CODE>getProperties()</CODE> method, in which case properties are
+read from a hierarchy of sources, including a file with a <CODE>.properties</CODE>
+extension and the same base name as the delegate test class.</p>
+<H2>Sharing a connection</H2>
+<P>In order to setup variables, such as a JDBC connection, that are
+reused by each test in the suite, you need to declare those variables as
+fields on the <EM>delegate</EM> implementation class and wrap the test
+suite in a {@link junit.framework.TestDecorator} as follows.</P>
+<pre>
+ public static Test suite() {
+
+ /*
+ * Create the delegate instance that will be reused by all proxy tests in
+ * the test suite. The variable is declared as 'final' so that we can
+ * access it from the anonymous TestDecorator class, below.
+ */
+ final DefaultTestCase delegate = new DefaultTestCase(); // !!!! THIS CLASS !!!!
+
+ /*
+ * Use a proxy test suite and specify the delegate.
+ */
+
+ ProxyTestSuite suite = new ProxyTestSuite(delegate);
+
+ /*
+ * Add some tests to the test suite. The setDelegate() method will be
+ * called on each test that implements IProxyTest and passed the value
+ * provided to the ProxyTestSuite constructor above.
+ */
+ suite.addTestCase( MyTestCase1.class );
+
+ // ... more tests.
+
+ /*
+ * Wrap up in decorator that connects to the database before the test
+ * suite is run and then disconnects from the database afterwards.
+ */
+ return new TestDecorator(suite) {
+
+ private boolean disconnectAfter = false;
+
+ public void basicRun(TestResult result) {
+ if (!delegate.isConnected()) {
+ delegate.connect();
+ disconnectAfter = true;
+ }
+ super.basicRun(result);
+ if (disconnectAfter) {
+ delegate.disconnect();
+ }
+ }
+
+ };
+</pre>
+<P>While the life span of the <EM>delegate</EM> spans that of the test
+suite, the <CODE>setUp</CODE> and <CODE>tearDown</CODE> methods of the
+delegate are invoked before and after each test <EM>implemented by the
+delegate</EM>. Since the delegate does not generally implement tests
+directly, those set up and tear down methods can not be used to perform
+integration testing.</P>
+<H2>Running or debugging a single test</H2>
+<P>By its nature, a test that implements {@link IProxyTest} requires a <em>delegate</em>
+in order to run. This works fine when you are running an entire test
+suite defined by the delegate, but you have to take an extra step in
+order to be able to run or debug a single test. The following code gives
+an example of how you can deal with this problem. It defines a property
+<em>testClass</em> whose value is the name of the delegate class. If the
+delegate was not set (as will be the case if you try to run a single
+test directly), then this property must be specified as a JVM argument
+using <code>-DtestClass=....</code> and an instance of your delegate
+will be created and used to run the proxy test.</P>
+<P>If you test relies on a shared variable such as a JDBC connection,
+then you still have to provide for that. One way to handle this is to
+delegate the <code>setUp</code> and <code>tearDown</code> methods on the
+proxy test. The delegate implementation of those methods would then
+establish and close the Connection iff the Connection was not already
+established.</P>
+
+<pre>
+ public Test getDelegate() {
+
+ if (m_delegate == null) {
+
+ /*
+ * This property gives the class name of the concrete instance of
+ * testClass that we need to instantiate so that we can run or debug
+ * a single test at a time.
+ */
+ String testClass = getProperties().getProperty("testClass");
+ if (testClass == null) {
+
+ throw new IllegalStateException(
+ "testClass: property not defined, could not configure delegate.");
+
+ }
+ try {
+ Class cl = Class.forName(testClass);
+ m_delegate = (Test) cl.newInstance();
+ } catch (Exception ex) {
+ throw new RuntimeException(ex);
+ }
+
+ }
+ }
+</pre>
+</BODY>
+</HTML>
Property changes on: branches/maven_scaleout/bigdata-core/src/test/java/junit/extensions/proxy/package.html
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: branches/maven_scaleout/bigdata-core/src/test/java/junit/framework/TestCase2.java
===================================================================
--- branches/maven_scaleout/bigdata-core/src/test/java/junit/framework/TestCase2.java (rev 0)
+++ branches/maven_scaleout/bigdata-core/src/test/java/junit/framework/TestCase2.java 2010-09-08 20:09:11 UTC (rev 3521)
@@ -0,0 +1,2230 @@
+/**
+
+The Notice below must appear in each file of the Source Code of any
+copy you distribute of the Licensed Product. Contributors to any
+Modifications may add their own copyright notices to identify their
+own contributions.
+
+License:
+
+The contents of this file are subject to the CognitiveWeb Open Source
+License Version 1.1 (the License). You may not copy or use this file,
+in either source code or executable form, except in compliance with
+the License. You may obtain a copy of the License from
+
+ http://www.CognitiveWeb.org/legal/license/
+
+Software distributed under the License is distributed on an AS IS
+basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
+the License for the specific language governing rights and limitations
+under the License.
+
+Copyrights:
+
+Portions created by or assigned to CognitiveWeb are Copyright
+(c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact
+information for CognitiveWeb is available at
+
+ http://www.CognitiveWeb.org
+
+Portions Copyright (c) 2002-2003 Bryan Thompson.
+
+Acknowledgements:
+
+Special thanks to the developers of the Jabber Open Source License 1.0
+(JOSL), from which this License was derived. This License contains
+terms that differ from JOSL.
+
+Special thanks to the CognitiveWeb Open Source Contributors for their
+suggestions and support of the Cognitive Web.
+
+Modifications:
+
+*/
+
+package junit.framework;
+
+// Test resource loader.
+import java.io.Externalizable;
+import java.io.Reader;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.IOException;
+import java.io.Serializable;
+
+// Properties hierarchy loader.
+import java.io.File;
+import java.io.FileInputStream;
+import java.util.Properties;
+import java.util.Random;
+import java.util.TreeMap;
+import java.util.Map;
+import java.util.Iterator;
+
+// Logger.
+import org.CognitiveWeb.util.PropertyUtil;
+import org.apache.log4j.Logger;
+import org.apache.log4j.Level;
+
+/**
+ * Extension of {@link TestCase} that supports logging, loading test
+ * resources, and hierarchical properties.<p>
+ *
+ * Note: When using Maven (or Ant), you need to be aware that that
+ * your <code>project.xml</code> has an impact on which files are
+ * recognized as tests to be submitted to a test runner. For example,
+ * it is common practice that all files beginning or ending with
+ * "Test" will be submitted to JUnit, though I suggest that the
+ * practice of naming things "FooTestCase" will serve you better.
+ * Also note that the maven test plugin explictly skips files whose
+ * name matches the pattern "*AbstractTestSuite".<p>
+ */
+
+abstract public class TestCase2
+ extends TestCase
+{
+
+ public TestCase2() {
+ super();
+ }
+
+ public TestCase2(String name) {
+ super(name);
+ }
+
+ /**
+ * The default {@link Logger} for this class, which is named
+ * "junit.framework.Test".
+ */
+
+ protected static final Logger log = Logger.getLogger
+ ( junit.framework.Test.class
+ );
+
+ /**
+ * This convenience method provides a version of {@link #fail(
+ * String Message )} that also excepts a {@link Throwable} cause
+ * for the test failure.<p>
+ */
+
+ static public void fail
+ ( String message,
+ Throwable t
+ )
+ {
+
+ AssertionFailedError ex = new AssertionFailedError
+ ( message
+ );
+
+ ex.initCause( t );
+
+ throw ex;
+
+ }
+
+ //************************************************************
+ //************************************************************
+ //************************************************************
+
+ static public void assertEquals( boolean[] expected, boolean[] actual )
+ {
+ assertEquals( null, expected, actual );
+ }
+
+ /**
+ * Compares arrays of primitive values.
+ */
+
+ static public void assertEquals( String msg, boolean[] expected, boolean[] actual )
+ {
+
+ if( msg == null ) {
+ msg = "";
+ } else {
+ msg = msg + " : ";
+ }
+
+ if( expected == null && actual == null ) {
+
+ return;
+
+ }
+
+ if( expected == null && actual != null ) {
+
+ fail( msg+"Expected a null array." );
+
+ }
+
+ if( expected != null && actual == null ) {
+
+ fail( msg+"Not expecting a null array." );
+
+ }
+
+ if (expected.length != actual.length) {
+ /*
+ * Only do message construction if we know that the assert will
+ * fail.
+ */
+ assertEquals(msg + "length differs.", expected.length,
+ actual.length);
+ }
+
+ for( int i=0; i<expected.length; i++ ) {
+
+ if (expected[i] != actual[i]) {
+ /*
+ * Only do message construction if we know that the assert will
+ * fail.
+ */
+ assertEquals(msg + "values differ: index=" + i, expected[i],
+ actual[i]);
+ }
+
+ }
+
+ }
+
+ /**
+ * <p>
+ * Compares byte[]s by value (not reference).
+ * </p>
+ * <p>
+ * Note: This method will only be invoked if both arguments can be typed as
+ * byte[] by the compiler. If either argument is not strongly typed, you
+ * MUST case it to a byte[] or {@link #assertEquals(Object, Object)} will be
+ * invoked instead.
+ * </p>
+ *
+ * @param expected
+ * @param actual
+ */
+ static public void assertEquals( byte[] expected, byte[] actual )
+ {
+
+ assertEquals( null, expected, actual );
+
+ }
+
+ /**
+ * <p>
+ * Compares byte[]s by value (not reference).
+ * </p>
+ * <p>
+ * Note: This method will only be invoked if both arguments can be typed as
+ * byte[] by the compiler. If either argument is not strongly typed, you
+ * MUST case it to a byte[] or {@link #assertEquals(Object, Object)} will be
+ * invoked instead.
+ * </p>
+ *
+ * @param msg
+ * @param expected
+ * @param actual
+ */
+ static public void assertEquals( String msg, byte[] expected, byte[] actual )
+ {
+
+ if( msg == null ) {
+ msg = "";
+ } else {
+ msg = msg + " : ";
+ }
+
+ if( expected == null && actual == null ) {
+
+ return;
+
+ }
+
+ if( expected == null && actual != null ) {
+
+ fail( msg+"Expected a null array." );
+
+ }
+
+ if( expected != null && actual == null ) {
+
+ fail( msg+"Not expecting a null array." );
+
+ }
+
+ if (expected.length != actual.length) {
+ /*
+ * Only do message construction if we know that the assert will
+ * fail.
+ */
+ assertEquals(msg + "length differs.", expected.length,
+ actual.length);
+ }
+
+ for( int i=0; i<expected.length; i++ ) {
+
+ if (expected[i] != actual[i]) {
+ /*
+ * Only do the message construction if we know that the assert
+ * will fail.
+ */
+ assertEquals(msg + "values differ: index=" + i, expected[i],
+ actual[i]);
+ }
+
+ }
+
+ }
+
+ static public void assertEquals( char[] expected, char[] actual )
+ {
+ assertEquals( null, expected, actual );
+ }
+
+ static public void assertEquals( String msg, char[] expected, char[] actual )
+ {
+
+ if( msg == null ) {
+ msg = "";
+ } else {
+ msg = msg + " : ";
+ }
+
+ if( expected == null && actual == null ) {
+
+ return;
+
+ }
+
+ if( expected == null && actual != null ) {
+
+ fail( msg+"Expected a null array." );
+
+ }
+
+ if( expected != null && actual == null ) {
+
+ fail( msg+"Not expecting a null array." );
+
+ }
+
+ if (expected.length != actual.length) {
+ /*
+ * Only do message construction if we know that the assert will
+ * fail.
+ */
+ assertEquals(msg + "length differs.", expected.length,
+ actual.length);
+ }
+
+ for( int i=0; i<expected.length; i++ ) {
+
+ if (expected[i] != actual[i]) {
+ /*
+ * Only do the message construction if we know that the assert
+ * will fail.
+ */
+ assertEquals(msg + "values differ: index=" + i, expected[i],
+ actual[i]);
+ }
+
+ }
+
+ }
+
+ static public void assertEquals( short[] expected, short[] actual )
+ {
+ assertEquals( null, expected, actual );
+ }
+
+ static public void assertEquals( String msg, short[] expected, short[] actual )
+ {
+
+ if( msg == null ) {
+ msg = "";
+ } else {
+ msg = msg + " : ";
+ }
+
+ if( expected == null && actual == null ) {
+
+ return;
+
+ }
+
+ if( expected == null && actual != null ) {
+
+ fail( msg+"Expected a null array." );
+
+ }
+
+ if( expected != null && actual == null ) {
+
+ fail( msg+"Not expecting a null array." );
+
+ }
+
+ if (expected.length != actual.length) {
+ /*
+ * Only do message construction if we know that the assert will
+ * fail.
+ */
+ assertEquals(msg + "length differs.", expected.length,
+ actual.length);
+ }
+
+ for( int i=0; i<expected.length; i++ ) {
+
+ if (expected[i] != actual[i]) {
+ /*
+ * Only do the message construction if we know that the assert
+ * will fail.
+ */
+ assertEquals(msg + "values differ: index=" + i, expected[i],
+ actual[i]);
+ }
+
+ }
+
+ }
+
+ static public void assertEquals( int[] expected, int[] actual )
+ {
+
+ assertEquals( null, expected, actual );
+
+ }
+
+ static public void assertEquals( String msg, int[] expected, int[] actual )
+ {
+
+ if( msg == null ) {
+ msg = "";
+ } else {
+ msg = msg + " : ";
+ }
+
+ if( expected == null && actual == null ) {
+
+ return;
+
+ }
+
+ if( expected == null && actual != null ) {
+
+ fail( msg+"Expected a null array." );
+
+ }
+
+ if( expected != null && actual == null ) {
+
+ fail( msg+"Not expecting a null array." );
+
+ }
+
+ if (expected.length != actual.length) {
+ /*
+ * Only do message construction if we know that the assert will
+ * fail.
+ */
+ assertEquals(msg + "length differs.", expected.length,
+ actual.length);
+ }
+
+ for( int i=0; i<expected.length; i++ ) {
+
+ if (expected[i] != actual[i]) {
+ /*
+ * Only do the message construction if we know that the assert
+ * will fail.
+ */
+ assertEquals(msg + "values differ: index=" + i, expected[i],
+ actual[i]);
+ }
+
+ }
+
+ }
+
+ static public void assertEquals( long[] expected, long[] actual )
+ {
+ assertEquals( null, expected, actual );
+ }
+
+ static public void assertEquals( String msg, long[] expected, long[] actual )
+ {
+
+ if( msg == null ) {
+ msg = "";
+ } else {
+ msg = msg + " : ";
+ }
+
+ if( expected == null && actual == null ) {
+
+ return;
+
+ }
+
+ if( expected == null && actual != null ) {
+
+ fail( msg+"Expected a null array." );
+
+ }
+
+ if( expected != null && actual == null ) {
+
+ fail( msg+"Not expecting a null array." );
+
+ }
+
+ if (expected.length != actual.length) {
+ /*
+ * Only do message construction if we know that the assert will
+ * fail.
+ */
+ assertEquals(msg + "length differs.", expected.length,
+ actual.length);
+ }
+
+ for( int i=0; i<expected.length; i++ ) {
+
+ if (expected[i] != actual[i]) {
+ /*
+ * Only do the message construction if we know that the assert
+ * will fail.
+ */
+ assertEquals(msg + "values differ: index=" + i, expected[i],
+ actual[i]);
+
+ }
+
+ }
+
+ }
+
+ static public void assertEquals( float[] expected, float[] actual )
+ {
+ assertEquals( null, expected, actual );
+ }
+
+ /**
+ * @todo Use smarter floating point comparison and offer parameter
+ * for controlling the floating point comparison.
+ */
+ static public void assertEquals( String msg, float[] expected, float[] actual )
+ {
+
+ if( msg == null ) {
+ msg = "";
+ } else {
+ msg = msg + " : ";
+ }
+
+ if( expected == null && actual == null ) {
+
+ return;
+
+ }
+
+ if( expected == null && actual != null ) {
+
+ fail( msg+"Expected a null array." );
+
+ }
+
+ if( expected != null && actual == null ) {
+
+ fail( msg+"Not expecting a null array." );
+
+ }
+
+ if (expected.length != actual.length) {
+ /*
+ * Only do message construction if we know that the assert will
+ * fail.
+ */
+ assertEquals(msg + "length differs.", expected.length,
+ actual.length);
+ }
+
+ for( int i=0; i<expected.length; i++ ) {
+
+ float delta = 0f;
+
+ try {
+
+ assertEquals(expected[i], actual[i], delta);
+
+ } catch (AssertionFailedError ex) {
+
+ /*
+ * Only do the message construction once the assertion is known
+ * to fail.
+ */
+ fail(msg + "values differ: index=" + i, ex);
+
+ }
+
+ }
+
+ }
+
+ static public void assertEquals( double[] expected, double[] actual )
+ {
+ assertEquals( null, expected, actual );
+ }
+
+ /**
+ * @todo Use smarter floating point comparison and offer parameter
+ * for controlling the floating point comparison.
+ */
+ static public void assertEquals( String msg, double[] expected, double[] actual )
+ {
+
+ if( msg == null ) {
+ msg = "";
+ } else {
+ msg = msg + " : ";
+ }
+
+ if( expected == null && actual == null ) {
+
+ return;
+
+ }
+
+ if( expected == null && actual != null ) {
+
+ fail( msg+"Expected a null array." );
+
+ }
+
+ if( expected != null && actual == null ) {
+
+ fail( msg+"Not expecting a null array." );
+
+ }
+
+ if (expected.length != actual.length) {
+ /*
+ * Only do message construction if we know that the assert will
+ * fail.
+ */
+ assertEquals(msg + "length differs.", expected.length,
+ actual.length);
+ }
+
+ for( int i=0; i<expected.length; i++ ) {
+
+ double delta = 0.0d;
+
+ try {
+
+ assertEquals(expected[i], actual[i], delta);
+
+ } catch (AssertionFailedError ex) {
+
+ /*
+ * Only do the message construction once the assertion is known
+ * to fail.
+ */
+ fail(msg + "values differ: index=" + i, ex);
+
+ }
+
+ }
+
+ }
+
+ static public void assertEquals( Object[] expected, Object[] actual )
+ {
+ assertEquals( null, expected, actual );
+ }
+
+ /**
+ * Compares arrays of {@link Object}s. The length of the arrays
+ * must agree, and each array element must agree. However the
+ * class of the arrays does NOT need to agree, e.g., an Object[]
+ * MAY compare as equals with a String[].
+ */
+
+ static public void assertEquals( String msg, Object[] expected, Object[] actual )
+ {
+
+ if( msg == null ) {
+ msg = "";
+ } else {
+ msg = msg + " : ";
+ }
+
+ if( expected == null && actual == null ) {
+
+ return;
+
+ }
+
+ if( expected == null && actual != null ) {
+
+ fail( msg+"Expected a null array." );
+
+ }
+
+ if( expected != null && actual == null ) {
+
+ fail( msg+"Not expecting a null array." );
+
+ }
+
+ if (expected.length != actual.length) {
+ /*
+ * Only do message construction if we know that the assert will
+ * fail.
+ */
+ assertEquals(msg + "length differs.", expected.length,
+ actual.length);
+ }
+
+ for( int i=0; i<expected.length; i++ ) {
+
+ try {
+
+ assertEquals(expected[i], actual[i]);
+
+ } catch (AssertionFailedError ex) {
+
+ /*
+ * Only do the message construction once the assertion is known
+ * to fail.
+ */
+
+ fail(msg + "values differ: index=" + i, ex);
+
+ }
+
+ }
+
+ }
+
+ //************************************************************
+ //************************************************************
+ //************************************************************
+
+ /**
+ * Test helper that can correctly compare arrays of primitives and
+ * arrays of objects as well as primitives and objects.
+ */
+ static public void assertSameValue( Object expected, Object actual )
+ {
+ assertSameValue( null, expected, actual );
+ }
+
+ /**
+ * Test helper that can correctly compare arrays of primitives and
+ * arrays of objects as well as primitives and objects.
+ *
+ * @todo This will throw a {@link ClassCastException} if
+ * <i>actual</i> is of a different primitive array type. Change
+ * the code to throw {@link AssertionFailedError} instead.
+ */
+ static public void assertSameValue( String msg, Object expected, Object actual )
+ {
+
+ if( expected != null && expected.getClass().isArray() ) {
+
+ if( expected.getClass().getComponentType().isPrimitive() ) {
+
+ Class componentType = expected.getClass().getComponentType();
+
+ if( componentType.equals( Boolean.TYPE ) ) {
+
+ assertEquals
+ ( msg,
+ (boolean[]) expected,
+ (boolean[]) actual
+ );
+
+ } else if( componentType.equals( Byte.TYPE ) ) {
+
+ assertEquals
+ ( msg,
+ (byte[]) expected,
+ (byte[]) actual
+ );
+
+ } else if( componentType.equals( Character.TYPE ) ) {
+
+ assertEquals
+ ( msg,
+ (char[]) expected,
+ (char[]) actual
+ );
+
+ } else if( componentType.equals( Short.TYPE ) ) {
+
+ assertEquals
+ ( msg,
+ (short[]) expected,
+ (short[]) actual
+ );
+
+ } else if( componentType.equals( Integer.TYPE ) ) {
+
+ assertEquals
+ ( msg,
+ (int[]) expected,
+ (int[]) actual
+ );
+
+ } else if( componentType.equals( Long.TYPE ) ) {
+
+ assertEquals
+ ( msg,
+ (long[]) expected,
+ (long[]) actual
+ );
+
+ } else if( componentType.equals( Float.TYPE ) ) {
+
+ assertEquals
+ ( msg,
+ (float[]) expected,
+ (float[]) actual
+ );
+
+ } else if( componentType.equals( Double.TYPE ) ) {
+
+ assertEquals
+ ( msg,
+ (double[]) expected,
+ (double[]) actual
+ );
+
+ } else {
+
+ throw new AssertionError();
+
+ }
+
+ } else {
+
+ assertTrue
+ ( msg,
+ java.util.Arrays.equals
+ ( (Object[]) expected,
+ (Object[]) actual
+ )
+ );
+
+ }
+
+ } else {
+
+ assertEquals
+ ( msg,
+ expected,
+ actual
+ );
+
+ }
+
+ }
+
+ //************************************************************
+ //************************************************************
+ //************************************************************
+
+ /**
+ * Method verifies that the <i>actual</i> {@link Iterator}
+ * produces the expected objects in the expected order. Objects
+ * are compared using {@link Object#equals( Object other )}. Errors
+ * are reported if too few or too many objects are produced, etc.
+ */
+
+ static public void assertSameIterator
+ ( Object[] expected,
+ Iterator actual
+ )
+ {
+
+ assertSameIterator
+ ( "",
+ expected,
+ actual
+ );
+
+ }
+
+ /**
+ * Method verifies that the <i>actual</i> {@link Iterator}
+ * produces the expected objects in the expected order. Objects
+ * are compared using {@link Object#equals( Object other )}. Errors
+ * are reported if too few or too many objects are produced, etc.
+ */
+
+ static public void assertSameIterator
+ ( String msg,
+ Object[] expected,
+ Iterator actual
+ )
+ {
+
+ int i = 0;
+
+ while( actual.hasNext() ) {
+
+ if( i >= expected.length ) {
+
+ fail( msg+": The iterator is willing to visit more than "+
+ expected.length+
+ " objects."
+ );
+
+ }
+
+ Object g = actual.next();
+
+// if (!expected[i].equals(g)) {
+ try {
+ assertSameValue(expected[i],g);
+ } catch(AssertionFailedError ex) {
+ /*
+ * Only do message construction if we know that the assert will
+ * fail.
+ */
+ fail(msg + ": Different objects at index=" + i + ": expected="
+ + expected[i] + ", actual=" + g);
+ }
+
+ i++;
+
+ }
+
+ if( i < expected.length ) {
+
+ fail( msg+": The iterator SHOULD have visited "+expected.length+
+ " objects, but only visited "+i+
+ " objects."
+ );
+
+ }
+
+ }
+
+ /**
+ * Verifies that the iterator visits the specified objects in some
+ * arbitrary ordering and that the iterator is exhausted once all
+ * expected objects have been visited. The implementation uses a
+ * selection without replacement "pattern".
+ */
+
+ static public void assertSameIteratorAnyOrder
+ ( Comparable[] expected,
+ Iterator actual
+ )
+ {
+
+ assertSameIteratorAnyOrder
+ ( "",
+ expected,
+ actual
+ );
+
+ }
+
+ /**
+ * Verifies that the iterator visits the specified objects in some
+ * arbitrary ordering and that the iterator is exhausted once all
+ * expected objects have been visited. The implementation uses a
+ * selection without replacement "pattern".
+ *
+ * FIXME Write test cases for this one.
+ *
+ * FIXME Can we we this without requiring the objects to implement
+ * {@link Comparable}?
+ */
+
+ static public void assertSameIteratorAnyOrder
+ ( String msg,
+ Comparable[] expected,
+ Iterator actual
+ )
+ {
+
+ // Populate a map that we will use to realize the match and
+ // selection without replacement logic.
+
+ final int nrange = expected.length;
+
+ java.util.Map range = new java.util.TreeMap();
+
+ for( int j=0; j<nrange; j++ ) {
+
+ range.put
+ ( expected[ j ],
+ expected[ j ]
+ );
+
+ }
+
+ // Do selection without replacement for the objects visited by
+ // iterator.
+
+ for( int j=0; j<nrange; j++ ) {
+
+ if( ! actual.hasNext() ) {
+
+ fail( msg+": Index exhausted while expecting more object(s)"+
+ ": index="+j
+ );
+
+ }
+
+ Object actualObject = actual.next();
+
+ if( range.remove( actualObject ) == null ) {
+
+ fail( "Object not expected"+
+ ": index="+j+
+ ", object="+actualObject
+ );
+
+ }
+
+ }
+
+ if( actual.hasNext() ) {
+
+ fail( "Iterator will deliver too many objects."
+ );
+
+ }
+
+ }
+
+ //************************************************************
+ //************************************************************
+ //************************************************************
+
+ /**
+ * Test helper produces a random sequence of indices in the range [0:n-1]
+ * suitable for visiting the elements of an array of n elements in a random
+ * order. This is useful when you want to randomize the presentation of
+ * elements from two or more arrays. For example, known keys and values can
+ * be generated and their presentation order randomized by indexing with the
+ * returned array.
+ */
+
+ public static int[] getRandomOrder( final int n )
+ {
+
+ final class Pair
+ implements Comparable
+ {
+ public double r = Math.random();
+ public int val;
+ public Pair( int val ) {this.val = val;}
+ public int compareTo(Object other)
+ {
+ if( this == other ) return 0;
+ if( this.r < ((Pair)other).r ) return -1;
+ else return 1;
+ }
+
+ }
+
+ Pair[] pairs = new Pair[ n ];
+
+ for( int i=0; i<n; i++ ) {
+
+ pairs[ i ] = new Pair( i );
+
+ }
+
+ java.util.Arrays.sort( pairs );
+
+ int order[] = new int[ n ];
+
+ for( int i=0; i<n; i++ ) {
+
+ order[ i ] = pairs[ i ].val;
+
+ }
+
+ return order;
+
+ }
+
+ /**
+ * Random number generator used by {@link #getNormalInt( int range
+ * )}.
+ */
+
+ private Random m_random = new Random();
+
+ /**
+ * Returns a random integer normally distributed in [0:range].
+ */
+
+ public int getNormalInt( int range )
+ {
+
+ final double bound = 3d;
+
+ double rand = m_random.nextGaussian();
+
+ if( rand < -bound ) rand = -bound;
+ else if( rand > bound ) rand = bound;
+
+ rand = ( rand + bound ) / ( 2 * bound );
+ // normal distribution in [0:1].
+
+ if( rand < 0d ) rand = 0d;
+ // make sure it is not slightly
+ // negative
+
+ int r = (int)( rand * range );
+
+ if( r > range ) {
+
+ throw new AssertionError();
+
+ }
+
+ return r;
+
+ // @todo develop a test case for this method based on the code
+ // in comments below and migrate into TestCase2 (junit-ext
+ // package).
+
+// int[] r = new int[10000];
+
+// for( int i=0; i<r.length; i++ ) {
+
+// r[ i ] = getNormalInt( len );
+
+// }
+
+// java.util.Arrays.sort( r );
+
+
+ }
+
+ /**
+ * Returns a random but unique string of Unicode characters with a
+ * maximum length of len and a minimum length.
+ *
+ * @param len The maximum length of the string. Each generated
+ * literal will have a mean length of <code>len/2</code> and the
+ * lengths will be distributed using a normal distribution (bell
+ * curve).
+ *
+ * @param id A unique index used to obtain a unique string.
+ * Typically this is a one up identifier.
+ */
+
+ public String getRandomString( int len, int id )
+ {
+
+// final String data = "0123456789!@#$%^&*()`~-_=+[{]}\\|;:'\",<.>/?QWERTYUIOPASDFGHJKLZXCVBNMqwertyuiopasdfghjklzxcvbnm";
+ final String data = "0123456789QWERTYUIOPASDFGHJKLZXCVBNMqwertyuiopasdfghjklzxcvbnm";
+
+ int[] order = getRandomOrder( data.length() );
+
+ int n = getNormalInt( len );
+
+ StringBuffer sb = new StringBuffer( n );
+
+ int index = 0;
+
+ for( int i=0; i<n; i++ ) {
+
+ sb.append( data.charAt( order[ index++ ] ) );
+
+ if( index == order.length ) {
+
+ index = 0;
+
+ }
+
+ }
+
+ sb.append( id );
+
+ return sb.toString();
+
+ }
+
+ /**
+ * Used to create objects of random type.
+ *
+ * @todo support generation of random array types and dates.
+ *
+ * @author thompsonbry
+ */
+ protected class RandomType
+ {
+
+ /**
+ * An explicit NULL property value.
+ */
+
+ final public static short NULL = 0;
+
+// /**
+// * A directed link to another generic object in the same store.
+// *
+// * @see LinkValue
+// */
+//
+// final public static short LINK = 1;
+//
+// /**
+// * A BLOB (metadata that supports access to very large binary
+// * objects using a stream oriented API).
+// */
+//
+// final public static short BLOB = 2;
+
+ /**
+ * The Java primitive type <em>boolean</em>.
+ *
+ * Note: Support for primitives recognizes the corresponding
+ * objects, e.g., {@link Boolean}, and special cases their
+ * serialization. This significantly reduces their the storage
+ * requirements.
+ */
+
+ final public static short BOOLEAN = 20;
+ final public static short BYTE = 21;
+ final public static short CHAR = 22;
+ final public static short SHORT = 23;
+ final public static short INT = 24;
+ final public static short LONG = 25;
+ final public static short FLOAT = 26;
+ final public static short DOUBLE = 27;
+
+ /**
+ * Java Object (must implement {@link Serializable} or {@link
+ * Externalizable}).
+ */
+
+ final public static short OBJECT = 30;
+
+ /**
+ * Array of Java objects (but not Java primitives).
+ */
+
+ final public static short OBJECT_ARRAY = 31;
+
+ /**
+ * Array of Java primitives.
+ */
+
+ final public static short BOOLEAN_ARRAY = 40;
+ final public static short BYTE_ARRAY = 41;
+ final public static short CHAR_ARRAY = 42;
+ final public static short SHORT_ARRAY = 43;
+ final public static short INT_ARRAY = 44;
+ final public static short LONG_ARRAY = 45;
+ final public static short FLOAT_ARRAY = 46;
+ final public static short DOUBLE_ARRAY = 47;
+
+ final public int[] randomType = new int[] {
+// LINK,
+// BLOB,
+ BOOLEAN, BYTE, CHAR, SHORT, INT, LONG, FLOAT, DOUBLE,
+ OBJECT,
+// OBJECT_ARRAY,
+// BOOLEAN_ARRAY,
+// BYTE_ARRAY,
+// CHAR_ARRAY,
+// SHORT_ARRAY,
+// INT_ARRAY,
+// LONG_ARRAY,
+// FLOAT_ARRAY,
+// DOUBLE_ARRAY,
+ NULL // Note: null at end to make easy to exclude.
+ };
+
+ /**
+ * For random characters.
+ */
+ static final private String alphabet = "0123456789QWERTYUIOPASDFGHJKLZXCVBNMqwertyuiopasdfghjklzxcvbnm";
+
+ /**
+ * Returns an object with a random type and random value.
+ *
+ * @param rnd The random# generator to use.
+ *
+ * @param allowNull When true, a null object reference may be
+ * returned.
+ *
+ * @return The random object.
+ */
+
+ public Object nextObject( Random rnd, boolean allowNull )
+ {
+
+ int[] types = randomType;
+
+ int range = types.length;
+
+ if( ! allowNull ) range--; // exclude null from types.
+
+ int type = types[ rnd.nextInt( range ) ];
+
+ Object obj = null;
+
+ switch( type ) {
+
+ case NULL: {
+ return null;
+ }
+ case BOOLEAN: {
+ obj = ( rnd.nextBoolean()
+ ? Boolean.TRUE
+ : Boolean.FALSE
+ );
+ break;
+ }
+ case BYTE: {
+ obj = new Byte( (byte) ( rnd.nextInt( 255 ) - 127 ) );
+ break;
+ }
+ case CHAR: {
+ obj = new Character( alphabet.charAt( rnd.nextInt( alphabet.length() ) ) );
+ break;
+ }
+ case SHORT: {
+ obj = new Short( (short) rnd.nextInt() );
+ break;
+ }
+ case INT: {
+ obj = new Integer( rnd.nextInt() );
+ break;
+ }
+ case LONG: {
+ obj = new Long( rnd.nextLong() );
+ break;
+ }
+ case FLOAT: {
+ obj = new Float( rnd.nextFloat() );
+ break;
+ }
+ case DOUBLE: {
+ obj = new Double( rnd.nextDouble() );
+ break;
+ }
+ case OBJECT: {
+ obj = getRandomString( 40, rnd.nextInt() );
+ break;
+ }
+// OBJECT_ARRAY,
+// BOOLEAN_ARRAY,
+// BYTE_ARRAY,
+// CHAR_ARRAY,
+// SHORT_ARRAY,
+// INT_ARRAY,
+// LONG_ARRAY,
+// FLOAT_ARRAY,
+// DOUBLE_ARRAY
+ default: {
+ throw new AssertionError( "unknown type="+type );
+ }
+
+ }
+
+ return obj;
+
+ }
+
+ }
+
+ private RandomType _randomType = new RandomType();
+
+ /**
+ * Returns an object with a random type and random value.
+ *
+ * @param allowNull When true, a null object reference may be
+ * returned.
+ *
+ * @return The random object.
+ */
+
+ public Object getRandomObject( boolean allowNull )
+ {
+
+ return _randomType.nextObject( m_random, allowNull );
+
+ }
+
+ /**
+ * Returns an object with a random type and random value.
+ *
+ * @param rnd The random generator.
...
[truncated message content] |
|
From: <mar...@us...> - 2010-09-08 14:52:35
|
Revision: 3520
http://bigdata.svn.sourceforge.net/bigdata/?rev=3520&view=rev
Author: martyncutcher
Date: 2010-09-08 14:52:29 +0000 (Wed, 08 Sep 2010)
Log Message:
-----------
reduce parameters for test to avoid timeout
Modified Paths:
--------------
branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/journal/StressTestConcurrentUnisolatedIndices.java
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/journal/StressTestConcurrentUnisolatedIndices.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/journal/StressTestConcurrentUnisolatedIndices.java 2010-09-08 14:51:48 UTC (rev 3519)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/journal/StressTestConcurrentUnisolatedIndices.java 2010-09-08 14:52:29 UTC (rev 3520)
@@ -119,11 +119,11 @@
// }
doConcurrentClientTest(journal,//
- 80,// timeout
+ 10,// timeout
20,// nresources
1, // minLocks
3, // maxLocks
- 1000, // ntrials
+ 100, // ntrials
3, // keyLen
1000, // nops
0.02d // failureRate
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mar...@us...> - 2010-09-08 14:51:54
|
Revision: 3519
http://bigdata.svn.sourceforge.net/bigdata/?rev=3519&view=rev
Author: martyncutcher
Date: 2010-09-08 14:51:48 +0000 (Wed, 08 Sep 2010)
Log Message:
-----------
make reopen public for call from tests
Modified Paths:
--------------
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/RWStrategy.java
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/RWStrategy.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/RWStrategy.java 2010-09-08 14:49:01 UTC (rev 3518)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/RWStrategy.java 2010-09-08 14:51:48 UTC (rev 3519)
@@ -563,7 +563,7 @@
* TODO: Modify so that current WriteCacheService is reset and re-used by new
* store.
*/
- protected void reopen() {
+ public void reopen() {
try {
log.warn("Request to reopen store after interrupt");
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mar...@us...> - 2010-09-08 14:49:07
|
Revision: 3518
http://bigdata.svn.sourceforge.net/bigdata/?rev=3518&view=rev
Author: martyncutcher
Date: 2010-09-08 14:49:01 +0000 (Wed, 08 Sep 2010)
Log Message:
-----------
add stressAlloc and pureAlloc tests to test allocation structures for large stores
Modified Paths:
--------------
branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/rwstore/TestRWJournal.java
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/rwstore/TestRWJournal.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/rwstore/TestRWJournal.java 2010-09-08 14:46:40 UTC (rev 3517)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/rwstore/TestRWJournal.java 2010-09-08 14:49:01 UTC (rev 3518)
@@ -432,6 +432,11 @@
System.out.println("Final allocation: " + faddr
+ ", allocations: " + (rw.getTotalAllocations() - numAllocs)
+ ", allocated bytes: " + (rw.getTotalAllocationsSize() - startAllocations));
+
+ store.commit();
+
+ // Confirm that we can re-open the journal after commit
+ bufferStrategy.reopen();
} finally {
store.destroy();
@@ -1018,7 +1023,7 @@
long realAddr = 0;
try {
// allocBatch(store, 1, 32, 650, 100000000);
- allocBatch(store, 1, 32, 650, 100000);
+ allocBatch(store, 1, 32, 650, 5000000);
store.commit();
System.out.println("Final allocations: " + rw.getTotalAllocations()
+ ", allocated bytes: " + rw.getTotalAllocationsSize() + ", file length: "
@@ -1033,23 +1038,80 @@
}
}
- private long allocBatch(Journal store, int tsts, int min, int sze, int grp) {
-
- RWStrategy bs = (RWStrategy) store
- .getBufferStrategy();
+ /**
+ * The pureAlloc test is to test the allocation aspect of the memory
+ * management rather than worrying about writing the data
+ */
+ public void test_pureAlloc() {
+
+ Journal store = (Journal) getStore();
- byte[] buf = new byte[sze+4]; // extra for checksum
- r.nextBytes(buf);
-
-
- for (int i = 0; i < grp; i++) {
- int alloc = min + r.nextInt(sze-min);
- ByteBuffer bb = ByteBuffer.wrap(buf, 0, alloc);
- bs.write(bb);
- }
+ RWStrategy bs = (RWStrategy) store.getBufferStrategy();
- return 0L;
- }
+ RWStore rw = bs.getRWStore();
+ long realAddr = 0;
+ try {
+ // allocBatch(store, 1, 32, 650, 100000000);
+ pureAllocBatch(store, 1, 32, 3075, 300000); // cover wider range of blocks
+ store.commit();
+ System.out.println("Final allocations: " + rw.getTotalAllocations()
+ + ", allocated bytes: " + rw.getTotalAllocationsSize() + ", file length: "
+ + rw.getStoreFile().length());
+ store.close();
+ System.out.println("Re-open Journal");
+ store = (Journal) getStore();
+
+ showStore(store);
+ } finally {
+ store.destroy();
+ }
+ }
+
+ private long allocBatch(Journal store, int tsts, int min, int sze, int grp) {
+
+ RWStrategy bs = (RWStrategy) store
+ .getBufferStrategy();
+
+ byte[] buf = new byte[sze+4]; // extra for checksum
+ r.nextBytes(buf);
+
+
+ for (int i = 0; i < grp; i++) {
+ int alloc = min + r.nextInt(sze-min);
+ ByteBuffer bb = ByteBuffer.wrap(buf, 0, alloc);
+ bs.write(bb);
+ }
+
+ return 0L;
+ }
+ /*
+ * Allocate tests but save 50% to re-alloc
+ */
+ private long pureAllocBatch(Journal store, int tsts, int min, int sze, int grp) {
+
+ RWStrategy bs = (RWStrategy) store
+ .getBufferStrategy();
+
+ RWStore rw = bs.getRWStore();
+ int freeAddr[] = new int[2048];
+ int freeCurs = 0;
+ for (int i = 0; i < grp; i++) {
+ int alloc = min + r.nextInt(sze-min);
+ int addr = rw.alloc(alloc, null);
+
+ if (i % 3 != 0) { //make avail 2 out of 3 for realloc
+ freeAddr[freeCurs++] = addr;
+ if (freeCurs == freeAddr.length) {
+ for (int f = 0; f < freeAddr.length; f++) {
+ rw.free(freeAddr[f], 0);
+ }
+ freeCurs = 0;
+ }
+ }
+ }
+
+ return 0L;
+ }
}
/**
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mar...@us...> - 2010-09-08 14:46:46
|
Revision: 3517
http://bigdata.svn.sourceforge.net/bigdata/?rev=3517&view=rev
Author: martyncutcher
Date: 2010-09-08 14:46:40 +0000 (Wed, 08 Sep 2010)
Log Message:
-----------
add allocation stats output for debug
Modified Paths:
--------------
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/AllocBlock.java
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/FixedAllocator.java
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/AllocBlock.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/AllocBlock.java 2010-09-08 13:27:18 UTC (rev 3516)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/AllocBlock.java 2010-09-08 14:46:40 UTC (rev 3517)
@@ -193,7 +193,7 @@
final int total = m_ints * 32;
final int allocBits = getAllocBits();
- return "Addr : " + m_addr + " [" + allocBits + "::" + total + "]";
+ return " - start addr : " + RWStore.convertAddr(m_addr) + " [" + allocBits + "::" + total + "]";
}
public void addAddresses(final ArrayList addrs, final int rootAddr) {
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/FixedAllocator.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/FixedAllocator.java 2010-09-08 13:27:18 UTC (rev 3516)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/FixedAllocator.java 2010-09-08 14:46:40 UTC (rev 3517)
@@ -490,7 +490,18 @@
}
public void appendShortStats(StringBuffer str) {
- str.append("Index: " + m_index + ", address: " + getStartAddr() + ", " + m_size + "\n");
+ str.append("Index: " + m_index + ", " + m_size);
+
+ Iterator<AllocBlock> blocks = m_allocBlocks.iterator();
+ while (blocks.hasNext()) {
+ AllocBlock block = blocks.next();
+ if (block.m_addr != 0) {
+ str.append(block.getStats());
+ } else {
+ break;
+ }
+ }
+ str.append("\n");
}
public int getAllocatedBlocks() {
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2010-09-08 13:27:18 UTC (rev 3516)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2010-09-08 14:46:40 UTC (rev 3517)
@@ -713,6 +713,9 @@
protected void readAllocationBlocks() throws IOException {
assert m_allocs.size() == 0;
+
+ System.out.println("readAllocationBlocks, m_metaBits.length: "
+ + m_metaBits.length);
/**
* Allocators are sorted in StartAddress order (which MUST be the order
@@ -721,7 +724,8 @@
* the metaAllocation if two allocation blocks were loaded for the same
* address (must be two version of same Allocator).
*
- * Meta-Allocations stored as {int address; int[8] bits}
+ * Meta-Allocations stored as {int address; int[8] bits}, so each block
+ * holds 8*32=256 allocation slots of 1K totalling 256K.
*/
for (int b = 0; b < m_metaBits.length; b += 9) {
long blockStart = convertAddr(m_metaBits[b]);
@@ -772,6 +776,13 @@
for (int index = 0; index < m_allocs.size(); index++) {
((Allocator) m_allocs.get(index)).setIndex(index);
}
+
+ if (false) {
+ StringBuffer tmp = new StringBuffer();
+ showAllocators(tmp);
+
+ System.out.println("Allocators: " + tmp.toString());
+ }
}
/**
@@ -1997,9 +2008,10 @@
static boolean tstBit(int[] bits, int bitnum) {
int index = bitnum / 32;
int bit = bitnum % 32;
-
+
if (index >= bits.length)
- throw new IllegalArgumentException();
+ throw new IllegalArgumentException("Accessing bit index: " + index
+ + " of array length: " + bits.length);
return (bits[(int) index] & 1 << bit) != 0;
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-09-08 13:27:24
|
Revision: 3516
http://bigdata.svn.sourceforge.net/bigdata/?rev=3516&view=rev
Author: thompsonbry
Date: 2010-09-08 13:27:18 +0000 (Wed, 08 Sep 2010)
Log Message:
-----------
commented out two properties which are no longer used in the configuration file.
Modified Paths:
--------------
trunk/src/resources/config/bigdataCluster16.config
Modified: trunk/src/resources/config/bigdataCluster16.config
===================================================================
--- trunk/src/resources/config/bigdataCluster16.config 2010-09-08 13:16:30 UTC (rev 3515)
+++ trunk/src/resources/config/bigdataCluster16.config 2010-09-08 13:27:18 UTC (rev 3516)
@@ -1298,11 +1298,11 @@
static private namespace = "U"+univNum+"";
// minimum #of data services to run.
- static private minDataServices = bigdata.dataServiceCount;
+// static private minDataServices = bigdata.dataServiceCount; // unused
// How long the master will wait to discover the minimum #of data
// services that you specified (ms).
- static private awaitDataServicesTimeout = 8000;
+// static private awaitDataServicesTimeout = 8000; // unused.
/* Multiplier for the scatter effect.
*/
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-09-08 13:16:42
|
Revision: 3515
http://bigdata.svn.sourceforge.net/bigdata/?rev=3515&view=rev
Author: thompsonbry
Date: 2010-09-08 13:16:30 +0000 (Wed, 08 Sep 2010)
Log Message:
-----------
Modified the ResourceService to use InetSocketAddress and NicUtil for configuration purposes. This change also touched the StoreManager and the MoveTask.
Added a BOpEvaluationContext which specifies whether an operator must be mapped across shards, nodes, evaluated on the query controller, or if it may be evaluated without sending its inputs anywhere. This is only for scale-out.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPipelineOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IQueryClient.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/eval/JoinGraph.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedRunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/MapBindingSetsOverShardsBuffer.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/InsertOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ndx/AbstractSampleIndex.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/SortOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/resources/MoveTask.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/resources/StoreManager.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/ManagedResourceService.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/ResourceService.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/service/TestReceiveBuffer.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/service/TestReceiveFile.java
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpEvaluationContext.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOp.java 2010-09-07 20:14:45 UTC (rev 3514)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOp.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -135,6 +135,14 @@
BOp clone();
/**
+ * Return the evaluation context for the operator. The default is
+ * {@link BOpEvaluationContext#ANY}. Operators which must be mapped against
+ * shards, mapped against nodes, or evaluated on the query controller must
+ * override this method.
+ */
+ public BOpEvaluationContext getEvaluationContext();
+
+ /**
* Interface declaring well known annotations.
*/
public interface Annotations {
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java 2010-09-07 20:14:45 UTC (rev 3514)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -299,6 +299,7 @@
}
+ @SuppressWarnings("unchecked")
public <T> T getProperty(final String name) {
return (T) annotations.get(name);
@@ -307,6 +308,7 @@
public <T> T getRequiredProperty(final String name) {
+ @SuppressWarnings("unchecked")
final T tmp = (T) annotations.get(name);
if (tmp == null)
@@ -336,4 +338,10 @@
}
+ public BOpEvaluationContext getEvaluationContext() {
+
+ return BOpEvaluationContext.ANY;
+
+ }
+
}
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpEvaluationContext.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpEvaluationContext.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpEvaluationContext.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -0,0 +1,48 @@
+package com.bigdata.bop;
+
+import com.bigdata.bop.bset.ConditionalRoutingOp;
+import com.bigdata.bop.join.PipelineJoin;
+import com.bigdata.bop.solutions.DistinctBindingSetOp;
+import com.bigdata.bop.solutions.SliceOp;
+
+/**
+ * Type safe enumeration indicates where an operator may be evaluated. Operators
+ * fall into several distinct categories based on whether or not their inputs
+ * need to be made available on specific nodes ({@link #HASHED} or
+ * {@link #SHARDED}), whether they can be evaluated anywhere their inputs may
+ * exist ({@link #ANY}), or whether they must be evaluated at the query
+ * controller ({@link #CONTROLLER}).
+ * <p>
+ * Note: All operators are evaluated locally when running against a standalone
+ * database.
+ */
+public enum BOpEvaluationContext {
+
+ /**
+ * The operator may be evaluated anywhere, including piecewise evaluation on
+ * any node of the cluster where its inputs are available. This is used for
+ * operators which do not need to concentrate or coordinate their inputs
+ * such as {@link ConditionalRoutingOp}.
+ */
+ ANY,
+ /**
+ * The input to the operator must be mapped across nodes using a hash
+ * partitioning schema and the operator must be evaluated on each hash
+ * partition. This is used for operators such as
+ * {@link DistinctBindingSetOp}.
+ */
+ HASHED,
+ /**
+ * The input to the operator must be mapped across the shards on which the
+ * operator must read or write and the operator must be evaluated shard wise
+ * on the services having access to each shard. For example,
+ * {@link PipelineJoin}.
+ */
+ SHARDED,
+ /**
+ * The operator must be evaluated on the query controller. For example,
+ * {@link SliceOp} may not be evaluated piecewise.
+ */
+ CONTROLLER;
+
+}
\ No newline at end of file
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpEvaluationContext.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPipelineOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPipelineOp.java 2010-09-07 20:14:45 UTC (rev 3514)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPipelineOp.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -50,10 +50,14 @@
* class depending on the operator).
*/
BOpStats newStats();
-
+
/**
* Instantiate a buffer suitable as a sink for this operator. The buffer
* will be provisioned based on the operator annotations.
+ * <p>
+ * Note: if the operation swallows binding sets from the pipeline (such as
+ * operators which write on the database) then the operator MAY return an
+ * immutable empty buffer.
*
* @return The buffer.
*/
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IQueryClient.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IQueryClient.java 2010-09-07 20:14:45 UTC (rev 3514)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IQueryClient.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -6,9 +6,9 @@
import com.bigdata.bop.BOp;
/**
- * Interface for a client executing queries.
+ * Interface for a client executing queries (the query controller).
*/
-public interface IQueryClient extends IQueryPeer, Remote {
+public interface IQueryClient extends IQueryPeer {
/*
* @todo Could return a data structure which encapsulates the query results
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java 2010-09-07 20:14:45 UTC (rev 3514)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -28,7 +28,6 @@
package com.bigdata.bop.engine;
import java.net.InetSocketAddress;
-import java.nio.ByteBuffer;
import java.rmi.RemoteException;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
@@ -410,6 +409,11 @@
/**
* The currently executing queries.
+ *
+ * @todo DEADLINE: There should be a data structure representing
+ * {@link RunningQuery} having deadlines so we can
+ * {@link RunningQuery#cancel(boolean)} queries when their deadline
+ * expires.
*/
final ConcurrentHashMap<Long/* queryId */, RunningQuery> runningQueries = new ConcurrentHashMap<Long, RunningQuery>();
@@ -480,20 +484,31 @@
* <p>
* Handle priority for unselective queries based on the order in which
* they are submitted?
- *
+ *
* @todo The approach taken by the {@link QueryEngine} executes one task per
* pipeline bop per chunk. Outside of how the tasks are scheduled,
* this corresponds closely to the historical pipeline query
- * evaluation. The other difference is that there is less opportunity
- * for concatenation of chunks. However, chunk concatenation could be
- * performed here if we (a) mark the BindingSetChunk with a flag to
- * indicate when it has been accepted; and (b) rip through the
- * incoming chunks for the query for the target bop and combine them
- * to feed the task. Chunks which have already been assigned would be
- * dropped when take() discovers them above. [The chunk combination
- * could also be done when we output the chunk if the sink has not
- * been taken, e.g., by combining the chunk into the same target
- * ByteBuffer, or when we add the chunk to the RunningQuery.]
+ * evaluation.
+ * <p>
+ * Chunk concatenation could be performed here if we (a) mark the
+ * {@link BindingSetChunk} with a flag to indicate when it has been
+ * accepted; and (b) rip through the incoming chunks for the query for
+ * the target bop and combine them to feed the task. Chunks which have
+ * already been assigned would be dropped when take() discovers them.
+ * [The chunk combination could also be done when we output the chunk
+ * if the sink has not been taken, e.g., by combining the chunk into
+ * the same target ByteBuffer, or when we add the chunk to the
+ * RunningQuery.]
+ *
+ * @todo SCALEOUT: High volume query operators must demand that their inputs
+ * are materialized before they can begin evaluation. Scaleout
+ * therefore requires a separate queue which looks at the metadata
+ * concerning chunks available on remote nodes for an operator which
+ * will run on this node and then demands the data either when the
+ * predecessors in the pipeline are done (operator at once evaluation)
+ * or when sufficient data are available to run the operator (mega
+ * chunk pipelining). Once the data are locally materialized, the
+ * operator may be queued for evaluation.
*/
private class QueryEngineTask implements Runnable {
public void run() {
@@ -778,17 +793,7 @@
}
/**
- *
- * @todo if the top bop is an operation which writes on the database then it
- * should swallow the binding sets from the pipeline and we should be
- * able to pass along a <code>null</code> query buffer.
- *
- * @todo SCALEOUT: Return a proxy for the query buffer either here or when
- * the query is sent along to another node for evaluation?
- * <p>
- * Actually, it would be nice if we could reuse the same NIO transfer
- * of {@link ByteBuffer}s to move the final results back to the client
- * rather than using a proxy object for the query buffer.
+ * Return a buffer onto which the solutions will be written.
*/
protected IBlockingBuffer<IBindingSet[]> newQueryBuffer(
final BindingSetPipelineOp query) {
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-09-07 20:14:45 UTC (rev 3514)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -55,7 +55,6 @@
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.NoSuchBOpException;
import com.bigdata.bop.ap.Predicate;
-import com.bigdata.bop.solutions.SliceOp;
import com.bigdata.journal.IIndexManager;
import com.bigdata.relation.accesspath.BlockingBuffer;
import com.bigdata.relation.accesspath.IAsynchronousIterator;
@@ -123,7 +122,11 @@
final private boolean controller;
/**
- * The client executing this query.
+ * The client executing this query (aka the query controller).
+ * <p>
+ * Note: The proxy is primarily for light weight RMI messages used to
+ * coordinate the distributed query evaluation. Ideally, all large objects
+ * will be transfered among the nodes of the cluster using NIO buffers.
*/
final private IQueryClient clientProxy;
@@ -142,11 +145,11 @@
/**
* An index from the {@link BOp.Annotations#BOP_ID} to the {@link BOp}.
*/
- private final Map<Integer, BOp> bopIndex;
+ protected final Map<Integer, BOp> bopIndex;
/**
- * A collection of the currently executing future for operators for this
- * query.
+ * A collection of {@link Future}s for currently executing operators for
+ * this query.
*/
private final ConcurrentHashMap<BSBundle, Future<?>> operatorFutures = new ConcurrentHashMap<BSBundle, Future<?>>();
@@ -245,11 +248,6 @@
* Note: This is package private so it will be visible to the
* {@link QueryEngine}.
*
- * @todo SCALEOUT: We need to model the chunks available before they are
- * materialized locally such that (a) they can be materialized on
- * demand (flow control); and (b) we can run the operator when there
- * are sufficient chunks available without taking on too much data.
- *
* @todo It is likely that we can convert to the use of
* {@link BlockingQueue} instead of {@link BlockingBuffer} in the
* operators and then handle the logic for combining chunks inside of
@@ -273,6 +271,19 @@
return queryEngine;
}
+
+ /**
+ * The client executing this query (aka the query controller).
+ * <p>
+ * Note: The proxy is primarily for light weight RMI messages used to
+ * coordinate the distributed query evaluation. Ideally, all large objects
+ * will be transfered among the nodes of the cluster using NIO buffers.
+ */
+ public IQueryClient getQueryController() {
+
+ return clientProxy;
+
+ }
/**
* The unique identifier for this query.
@@ -296,7 +307,7 @@
synchronized (queryRef) {
- if (queryRef == null) {
+ if (queryRef.get() == null) {
try {
@@ -378,14 +389,16 @@
* rather than formatting it onto a {@link ByteBuffer}.
*
* @param sinkId
+ * The identifier of the target operator.
* @param sink
+ * The intermediate results to be passed to that target operator.
*
* @return The #of chunks made available for consumption by the sink. This
* will always be ONE (1) for scale-up. For scale-out, there will be
* one chunk per index partition over which the intermediate results
* were mapped.
*/
- protected int add(final int sinkId,
+ protected <E> int add(final int sinkId,
final IBlockingBuffer<IBindingSet[]> sink) {
/*
@@ -394,23 +407,11 @@
final BindingSetChunk chunk = new BindingSetChunk(queryId, sinkId,
-1/* partitionId */, sink.iterator());
- addChunkToQueryEngine(chunk);
+ queryEngine.add(chunk);
return 1;
}
-
- /**
- * Adds a chunk to the local {@link QueryEngine}.
- *
- * @param chunk
- * The chunk.
- */
- protected void addChunkToQueryEngine(final BindingSetChunk chunk) {
-
- queryEngine.add(chunk);
-
- }
/**
* Make a chunk of binding sets available for consumption by the query.
@@ -541,16 +542,6 @@
*
* @throws UnsupportedOperationException
* If this node is not the query coordinator.
- *
- * @todo Clone the {@link BOpStats} before reporting to avoid concurrent
- * modification?
- *
- * @todo SCALEOUT: Do not release buffers backing the binding set chunks
- * generated by an operator or the outputs of the final operator (the
- * query results) until the sink has accepted those outputs. This
- * means that we must not release the output buffers when the bop
- * finishes but when its consumer finishes draining the {@link BOp}s
- * outputs.
*/
public void haltOp(final HaltOpMessage msg) {
if (!controller)
@@ -768,11 +759,6 @@
final Runnable r = new Runnable() {
public void run() {
final UUID serviceId = queryEngine.getServiceId();
- /*
- * @todo SCALEOUT: Combine chunks available on the queue for the
- * current bop. This is not exactly "fan in" since multiple
- * chunks could be available in scaleup as well.
- */
int fanIn = 1;
int sinkChunksOut = 0;
int altSinkChunksOut = 0;
@@ -822,17 +808,6 @@
* Return an iterator which will drain the solutions from the query. The
* query will be cancelled if the iterator is
* {@link ICloseableIterator#close() closed}.
- *
- * @return
- *
- * @todo Not all queries produce binding sets. For example, mutation
- * operations. We could return the mutation count for mutation
- * operators, which could be reported by {@link BOpStats} for that
- * operator (unitsOut).
- *
- * @todo SCALEOUT: Track chunks consumed by the client so we do not release
- * the backing {@link ByteBuffer} before the client is done draining
- * the iterator.
*/
public IAsynchronousIterator<IBindingSet[]> iterator() {
@@ -854,14 +829,17 @@
}
/**
- * @todo Cancelled queries must reject or drop new chunks, etc.
- * <p>
- * Queries must release all of their resources when they are done().
- * <p>
- * Queries MUST NOT cause the solutions to be discarded before the
- * client can consume them. This means that we have to carefully
- * integrate {@link SliceOp} or just wrap the query buffer to impose
- * the slice (simpler).
+ * {@inheritDoc}
+ * <p>
+ * Cancelled queries :
+ * <ul>
+ * <li>must reject new chunks</li>
+ * <li>must cancel any running operators</li>
+ * <li>must not begin to evaluate operators</li>
+ * <li>must release all of their resources</li>
+ * <li>must not cause the solutions to be discarded before the client can
+ * consume them.</li>
+ * </ul>
*/
final public boolean cancel(final boolean mayInterruptIfRunning) {
// halt the query.
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/eval/JoinGraph.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/eval/JoinGraph.java 2010-09-07 20:14:45 UTC (rev 3514)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/eval/JoinGraph.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -36,6 +36,7 @@
import com.bigdata.bop.BOp;
import com.bigdata.bop.BOpContext;
+import com.bigdata.bop.BOpEvaluationContext;
import com.bigdata.bop.BindingSetPipelineOp;
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.IPredicate;
@@ -262,4 +263,14 @@
}
+ /**
+ * This operator must be evaluated on the query controller.
+ */
+ @Override
+ public BOpEvaluationContext getEvaluationContext() {
+
+ return BOpEvaluationContext.CONTROLLER;
+
+ }
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java 2010-09-07 20:14:45 UTC (rev 3514)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -38,7 +38,11 @@
import com.bigdata.bop.engine.QueryEngine;
import com.bigdata.bop.engine.RunningQuery;
import com.bigdata.bop.join.PipelineJoin;
+import com.bigdata.bop.solutions.SliceOp;
import com.bigdata.journal.IIndexManager;
+import com.bigdata.relation.accesspath.IAsynchronousIterator;
+import com.bigdata.relation.accesspath.IBlockingBuffer;
+import com.bigdata.relation.accesspath.IBuffer;
import com.bigdata.service.DataService;
import com.bigdata.service.IBigdataFederation;
import com.bigdata.service.ManagedResourceService;
@@ -110,10 +114,10 @@
* join in a query plan.
*/
private final IBigdataFederation<?> fed;
-
+
/**
- * A service used to expose {@link ByteBuffer}s and managed index resources
- * for transfer to remote services in support of distributed query
+ * The service used to expose {@link ByteBuffer}s and managed index
+ * resources for transfer to remote services in support of distributed query
* evaluation.
*/
private final ManagedResourceService resourceService;
@@ -176,6 +180,17 @@
}
+ /**
+ * The service used to expose {@link ByteBuffer}s and managed index
+ * resources for transfer to remote services in support of distributed query
+ * evaluation.
+ */
+ public ManagedResourceService getResourceService() {
+
+ return resourceService;
+
+ }
+
@Override
public void bufferReady(IQueryClient clientProxy,
InetSocketAddress serviceAddr, long queryId, int bopId) {
@@ -201,4 +216,34 @@
}
+ /**
+ * {@inheritDoc}
+ *
+ * @todo Historically, this has been a proxy object for an {@link IBuffer}
+ * on the {@link IQueryClient query controller}. However, it would be
+ * nice if we could reuse the same NIO transfer of {@link ByteBuffer}s
+ * to move the final results back to the client rather than using a
+ * proxy object for the query buffer.
+ * <p>
+ * In scale-out we must track chunks consumed by the client so we do
+ * not release the backing {@link ByteBuffer} on which the solutions
+ * are marshalled before the client is done draining the iterator. If
+ * the solutions are generated on the peers, then the peers must
+ * retain the data until the client has consumed them or have
+ * transferred the solutions to itself.
+ * <p>
+ * The places where this can show up as a problem are {@link SliceOp},
+ * when a query deadline is reached, and when a query terminates
+ * normally. Also pay attention when the client closes the
+ * {@link IAsynchronousIterator} from which it is draining solutions
+ * early.
+ */
+ @Override
+ protected IBlockingBuffer<IBindingSet[]> newQueryBuffer(
+ final BindingSetPipelineOp query) {
+
+ return query.newBuffer();
+
+ }
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedRunningQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedRunningQuery.java 2010-09-07 20:14:45 UTC (rev 3514)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedRunningQuery.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -31,11 +31,16 @@
import com.bigdata.bop.BOp;
import com.bigdata.bop.IBindingSet;
-import com.bigdata.bop.engine.BindingSetChunk;
+import com.bigdata.bop.IPredicate;
import com.bigdata.bop.engine.IQueryClient;
import com.bigdata.bop.engine.RunningQuery;
+import com.bigdata.mdi.PartitionLocator;
+import com.bigdata.relation.accesspath.IAsynchronousIterator;
import com.bigdata.relation.accesspath.IBlockingBuffer;
+import com.bigdata.relation.accesspath.IBuffer;
import com.bigdata.service.IBigdataFederation;
+import com.bigdata.service.jini.master.IAsynchronousClientTask;
+import com.bigdata.striterator.IKeyOrder;
/**
* Extends {@link RunningQuery} to provide additional state and logic required
@@ -43,8 +48,14 @@
* .
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
+ * @version $Id: FederatedRunningQuery.java 3511 2010-09-06 20:45:37Z
+ * thompsonbry $
*
+ * @todo SCALEOUT: We need to model the chunks available before they are
+ * materialized locally such that (a) they can be materialized on demand
+ * (flow control); and (b) we can run the operator when there are
+ * sufficient chunks available without taking on too much data.
+ *
* @todo SCALEOUT: Life cycle management of the operators and the query implies
* both a per-query bop:NodeList map on the query coordinator identifying
* the nodes on which the query has been executed and a per-query
@@ -76,57 +87,137 @@
}
/**
- * Create a {@link BindingSetChunk} from a sink and add it to the queue.
- * <p>
- * Note: If we are running standalone, then we leave the data on the heap
- * rather than formatting it onto a {@link ByteBuffer}.
+ * {@inheritDoc}
*
- * @param sinkId
- * @param sink
- *
* @return The #of chunks made available for consumption by the sink. This
* will always be ONE (1) for scale-up. For scale-out, there will be
* one chunk per index partition over which the intermediate results
* were mapped.
*
- * @todo <p>
- * For selective queries in s/o, first format the data onto a list of
- * byte[]s, one per target shard/node. Then, using a lock, obtain a
- * ByteBuffer if there is none associated with the query yet.
- * Otherwise, using the same lock, obtain a slice onto that ByteBuffer
- * and put as much of the byte[] as will fit, continuing onto a newly
- * recruited ByteBuffer if necessary. Release the lock and notify the
- * target of the ByteBuffer slice (buffer#, off, len). Consider
- * pushing the data proactively for selective queries.
- * <p>
- * For unselective queries in s/o, proceed as above but we need to get
- * the data off the heap and onto the {@link ByteBuffer}s quickly
- * (incrementally) and we want the consumers to impose flow control on
- * the producers to bound the memory demand (this needs to be
- * coordinated carefully to avoid deadlocks). Typically, large result
- * sets should result in multiple passes over the consumer's shard
- * rather than writing the intermediate results onto the disk.
+ * FIXME SCALEOUT: This is where we need to map the binding sets
+ * over the shards for the target operator. Once they are mapped,
+ * write the binding sets onto an NIO buffer for the target node and
+ * then send an RMI message to the node telling it that there is a
+ * chunk available for the given (queryId,bopId,partitionId).
+ * <p>
+ * For selective queries in s/o, first format the data onto a list
+ * of byte[]s, one per target shard/node. Then, using a lock, obtain
+ * a ByteBuffer if there is none associated with the query yet.
+ * Otherwise, using the same lock, obtain a slice onto that
+ * ByteBuffer and put as much of the byte[] as will fit, continuing
+ * onto a newly recruited ByteBuffer if necessary. Release the lock
+ * and notify the target of the ByteBuffer slice (buffer#, off,
+ * len). Consider pushing the data proactively for selective
+ * queries.
+ * <p>
+ * For unselective queries in s/o, proceed as above but we need to
+ * get the data off the heap and onto the {@link ByteBuffer}s
+ * quickly (incrementally) and we want the consumers to impose flow
+ * control on the producers to bound the memory demand (this needs
+ * to be coordinated carefully to avoid deadlocks). Typically, large
+ * result sets should result in multiple passes over the consumer's
+ * shard rather than writing the intermediate results onto the disk.
*
- * FIXME SCALEOUT: This is where we need to map the binding sets over
- * the shards for the target operator. Once they are mapped, write the
- * binding sets onto an NIO buffer for the target node and then send
- * an RMI message to the node telling it that there is a chunk
- * available for the given (queryId,bopId,partitionId).
- */
+ * */
@Override
- protected int add(final int sinkId,
+ protected <E> int add(final int sinkId,
final IBlockingBuffer<IBindingSet[]> sink) {
- /*
- * Note: The partitionId will always be -1 in scale-up.
- */
- final BindingSetChunk chunk = new BindingSetChunk(getQueryId(), sinkId,
- -1/* partitionId */, sink.iterator());
+ if (sink == null)
+ throw new IllegalArgumentException();
- addChunkToQueryEngine(chunk);
+ final BOp bop = bopIndex.get(sinkId);
- return 1;
+ if (bop == null)
+ throw new IllegalArgumentException();
+ switch (bop.getEvaluationContext()) {
+ case ANY:
+ return super.add(sinkId, sink);
+ case HASHED: {
+ /*
+ * FIXME The sink self describes the nodes over which the
+ * binding sets will be mapped and the hash function to be applied
+ * so we look up those metadata and apply them to distributed the
+ * binding sets across the nodes.
+ */
+ throw new UnsupportedOperationException();
+ }
+ case SHARDED: {
+ /*
+ * FIXME The sink must read or write on a shard so we map the
+ * binding sets across the access path for the sink.
+ *
+ * @todo For a pipeline join, the predicate is the right hand
+ * operator of the sink. This might be true for INSERT and DELETE
+ * operators as well.
+ *
+ * @todo IKeyOrder tells us which index will be used and should be
+ * set on the predicate by the join optimizer.
+ *
+ * @todo Use the read or write timestamp depending on whether the
+ * operator performs mutation [this must be part of the operator
+ * metadata.]
+ *
+ * @todo Set the capacity of the the "map" buffer to the size of the
+ * data contained in the sink (in fact, we should just process the
+ * sink data in place).
+ */
+ final IPredicate<E> pred = null; // @todo
+ final IKeyOrder<E> keyOrder = null; // @todo
+ final long timestamp = getReadTimestamp(); // @todo
+ final int capacity = 1000;// @todo
+ final MapBindingSetsOverShardsBuffer<IBindingSet, E> mapper = new MapBindingSetsOverShardsBuffer<IBindingSet, E>(
+ getFederation(), pred, keyOrder, timestamp, capacity) {
+
+ @Override
+ IBuffer<IBindingSet> newBuffer(PartitionLocator locator) {
+ // TODO Auto-generated method stub
+ return null;
+ }
+
+ };
+ /*
+ * Map the binding sets over shards.
+ *
+ * FIXME The buffers created above need to become associated with
+ * this query as resources of the query. Once we are done mapping
+ * the binding sets over the shards, the target node for each buffer
+ * needs to be set an RMI message to let it know that there is a
+ * chunk available for it for the target operator.
+ */
+ {
+ final IAsynchronousIterator<IBindingSet[]> itr = sink
+ .iterator();
+ try {
+ while (itr.hasNext()) {
+ final IBindingSet[] chunk = itr.next();
+ for (IBindingSet bset : chunk) {
+ mapper.add(bset);
+ }
+ }
+ } finally {
+ itr.close();
+ sink.close();
+ }
+ }
+
+ throw new UnsupportedOperationException();
+ }
+ case CONTROLLER: {
+
+ final IQueryClient clientProxy = getQueryController();
+
+// getQueryEngine().getResourceService().port;
+//
+// clientProxy.bufferReady(clientProxy, serviceAddr, getQueryId(), sinkId);
+
+ throw new UnsupportedOperationException();
+ }
+ default:
+ throw new AssertionError(bop.getEvaluationContext());
+ }
+
}
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/MapBindingSetsOverShardsBuffer.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/MapBindingSetsOverShardsBuffer.java 2010-09-07 20:14:45 UTC (rev 3514)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/MapBindingSetsOverShardsBuffer.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -84,9 +84,9 @@
extends AbstractUnsynchronizedArrayBuffer<E> {
/**
- * The predicate from which we generate the asBound predicates. This
+ * The predicate from which we generate the asBound binding sets. This
* predicate and the {@link IKeyOrder} together determine the required
- * access path.
+ * access path.
*/
private final IPredicate<F> pred;
@@ -123,7 +123,12 @@
* @param fed
* The federation.
* @param pred
- * The target predicate.
+ * The predicate associated with the target operator. The
+ * predicate identifies which variables and/or constants form the
+ * key for the access path and hence selects the shards on which
+ * the target operator must read or write. For example, when the
+ * target operator is a JOIN, this is the {@link IPredicate}
+ * associated with the right hand operator of the join.
* @param keyOrder
* Identifies the access path for the target predicate.
* @param timestamp
@@ -136,9 +141,10 @@
* The capacity of this buffer.
*/
public MapBindingSetsOverShardsBuffer(
- final IBigdataFederation<?> fed,
- final IPredicate<F> pred, final IKeyOrder<F> keyOrder,
- final long timestamp,
+ final IBigdataFederation<?> fed,//
+ final IPredicate<F> pred, //
+ final IKeyOrder<F> keyOrder,//
+ final long timestamp,//
final int capacity) {
super(capacity);
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java 2010-09-07 20:14:45 UTC (rev 3514)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -45,6 +45,7 @@
import com.bigdata.bop.BOp;
import com.bigdata.bop.BOpContext;
+import com.bigdata.bop.BOpEvaluationContext;
import com.bigdata.bop.BindingSetPipelineOp;
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.IConstraint;
@@ -1774,4 +1775,14 @@
}// class JoinTask
+ /**
+ * This is a shard wise operator.
+ */
+ @Override
+ public BOpEvaluationContext getEvaluationContext() {
+
+ return BOpEvaluationContext.SHARDED;
+
+ }
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/InsertOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/InsertOp.java 2010-09-07 20:14:45 UTC (rev 3514)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/InsertOp.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -35,8 +35,10 @@
import com.bigdata.bop.BOp;
import com.bigdata.bop.BOpContext;
+import com.bigdata.bop.BOpEvaluationContext;
import com.bigdata.bop.BindingSetPipelineOp;
import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.IPredicate;
import com.bigdata.bop.IVariableOrConstant;
import com.bigdata.bop.engine.BOpStats;
import com.bigdata.btree.ILocalBTreeView;
@@ -70,6 +72,9 @@
* An ordered {@link IVariableOrConstant}[]. Elements will be created
* using the binding sets which flow through the operator and
* {@link IRelation#newElement(java.util.List, IBindingSet)}.
+ *
+ * @todo This should be an {@link IPredicate} and should be the right
+ * hand operand just like for a JOIN.
*/
String SELECTED = InsertOp.class.getName() + ".selected";
@@ -267,5 +272,15 @@
// .getClass());
//
// a[i] = e;
-
+
+ /**
+ * This is a shard wise operator.
+ */
+ @Override
+ public BOpEvaluationContext getEvaluationContext() {
+
+ return BOpEvaluationContext.SHARDED;
+
+ }
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ndx/AbstractSampleIndex.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ndx/AbstractSampleIndex.java 2010-09-07 20:14:45 UTC (rev 3514)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ndx/AbstractSampleIndex.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -28,6 +28,7 @@
package com.bigdata.bop.ndx;
+import com.bigdata.bop.BOpEvaluationContext;
import com.bigdata.bop.PipelineOp;
import com.bigdata.bop.BOp;
import com.bigdata.bop.IBindingSet;
@@ -103,4 +104,14 @@
}
+ /**
+ * This is a shard wise operator.
+ */
+ @Override
+ public BOpEvaluationContext getEvaluationContext() {
+
+ return BOpEvaluationContext.SHARDED;
+
+ }
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java 2010-09-07 20:14:45 UTC (rev 3514)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -33,6 +33,7 @@
import com.bigdata.bop.BOp;
import com.bigdata.bop.BOpContext;
+import com.bigdata.bop.BOpEvaluationContext;
import com.bigdata.bop.BindingSetPipelineOp;
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.PipelineOp;
@@ -272,4 +273,14 @@
}
+ /**
+ * This operator must be evaluated on the query controller.
+ */
+ @Override
+ public BOpEvaluationContext getEvaluationContext() {
+
+ return BOpEvaluationContext.CONTROLLER;
+
+ }
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/SortOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/SortOp.java 2010-09-07 20:14:45 UTC (rev 3514)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/SortOp.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -30,6 +30,7 @@
import java.util.Map;
import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpEvaluationContext;
import com.bigdata.bop.BindingSetPipelineOp;
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.PipelineOp;
@@ -83,4 +84,16 @@
}
+ /**
+ * This operator must be evaluated on the query controller.
+ *
+ * @todo Define a distributed (external) merge sort operator.
+ */
+ @Override
+ public BOpEvaluationContext getEvaluationContext() {
+
+ return BOpEvaluationContext.CONTROLLER;
+
+ }
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/resources/MoveTask.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/resources/MoveTask.java 2010-09-07 20:14:45 UTC (rev 3514)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/resources/MoveTask.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -31,7 +31,7 @@
import java.io.File;
import java.io.IOException;
import java.io.Serializable;
-import java.net.InetAddress;
+import java.net.InetSocketAddress;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.ExecutionException;
@@ -63,7 +63,6 @@
import com.bigdata.service.IMetadataService;
import com.bigdata.service.MetadataService;
import com.bigdata.service.ResourceService;
-import com.bigdata.util.config.NicUtil;
/**
* Task moves an index partition to another {@link IDataService}.
@@ -433,7 +432,7 @@
private final UUID targetDataServiceUUID;
private final int targetIndexPartitionId;
private final Event parentEvent;
- private final InetAddress thisInetAddr;
+// private final InetAddress thisInetAddr;
/**
*
@@ -481,11 +480,11 @@
this.targetIndexPartitionId = targetIndexPartitionId;
this.parentEvent = parentEvent;
- try {
- this.thisInetAddr = InetAddress.getByName(NicUtil.getIpAddress("default.nic", "default", false));
- } catch(Throwable t) {
- throw new IllegalArgumentException(t.getMessage(), t);
- }
+// try {
+// this.thisInetAddr = InetAddress.getByName(NicUtil.getIpAddress("default.nic", "default", false));
+// } catch(Throwable t) {
+// throw new IllegalArgumentException(t.getMessage(), t);
+// }
}
/**
@@ -596,9 +595,8 @@
targetIndexPartitionId,//
historicalWritesBuildResult.segmentMetadata,//
bufferedWritesBuildResult.segmentMetadata,//
- thisInetAddr,
resourceManager
- .getResourceServicePort()//
+ .getResourceService().getAddr()//
)).get();
} catch (ExecutionException ex) {
@@ -913,8 +911,8 @@
final private int targetIndexPartitionId;
final private SegmentMetadata historyIndexSegmentMetadata;
final private SegmentMetadata bufferedWritesIndexSegmentMetadata;
- final private InetAddress addr;
- final private int port;
+ final private InetSocketAddress addr;
+// final private int port;
/**
* @param sourceIndexMetadata
@@ -927,14 +925,14 @@
* Describes the {@link IndexSegmentStore} containing the
* historical data for the source index partition.
* @param bufferedWritesIndexSegmentMetadata
- * Desribes the {@link IndexSegmentStore} containing the
+ * Describes the {@link IndexSegmentStore} containing the
* buffered writes from the live journal for the source index
* partition.
* @param addr
- * The {@link InetAddress} of the source data service.
- * @param port
- * The port at which the source data service has exposed its
- * {@link ResourceService}
+ * The {@link InetSocketAddress} of the
+ * {@link ResourceService} running on the source data service
+ * (the one from which the resources will be copied during
+ * the move).
*/
ReceiveIndexPartitionTask(//
final IndexMetadata sourceIndexMetadata,//
@@ -942,8 +940,7 @@
final int targetIndexPartitionId,//
final SegmentMetadata historyIndexSegmentMetadata,//
final SegmentMetadata bufferedWritesIndexSegmentMetadata,//
- final InetAddress addr,
- final int port
+ final InetSocketAddress addr
) {
this.sourceIndexMetadata = sourceIndexMetadata;
@@ -952,7 +949,6 @@
this.historyIndexSegmentMetadata = historyIndexSegmentMetadata;
this.bufferedWritesIndexSegmentMetadata = bufferedWritesIndexSegmentMetadata;
this.addr = addr;
- this.port = port;
}
// private transient DataService dataService;
@@ -1000,8 +996,7 @@
targetIndexPartitionId,//
historyIndexSegmentMetadata,//
bufferedWritesIndexSegmentMetadata,//
- addr,//
- port//
+ addr//
)).get();
// update the index partition receive counter.
@@ -1058,8 +1053,8 @@
final private SegmentMetadata sourceBufferedWritesSegmentMetadata;
final private Event parentEvent;
final private String summary;
- final InetAddress addr;
- final int port;
+ final InetSocketAddress addr;
+// final int port;
/**
* @param resourceManager
@@ -1080,10 +1075,9 @@
* buffered writes from the live journal for the source index
* partition.
* @param addr
- * The {@link InetAddress} of the source data service.
- * @param port
- * The port at which the source data service has exposed its
- * {@link ResourceService}
+ * The {@link InetSocketAddress} of the
+ * {@link ResourceService} of the source data service (the
+ * one from which the resources will be copied).
*/
InnerReceiveIndexPartitionTask(final ResourceManager resourceManager,
final String targetIndexName,
@@ -1092,8 +1086,7 @@
final int targetIndexPartitionId,
final SegmentMetadata historyIndexSegmentMetadata,
final SegmentMetadata bufferedWritesIndexSegmentMetadata,
- final InetAddress addr,
- final int port
+ final InetSocketAddress addr
) {
super(resourceManager.getConcurrencyManager(), ITx.UNISOLATED,
@@ -1124,7 +1117,6 @@
this.sourceHistorySegmentMetadata = historyIndexSegmentMetadata;
this.sourceBufferedWritesSegmentMetadata = bufferedWritesIndexSegmentMetadata;
this.addr = addr;
- this.port = port;
this.summary = OverflowActionEnum.Move + "(" + sourceIndexName
+ "->" + targetIndexName + ")";
@@ -1267,7 +1259,7 @@
try {
// read the resource, writing onto that file.
- new ResourceService.ReadResourceTask(addr, port,
+ new ResourceService.ReadResourceTask(addr,
sourceSegmentMetadata.getUUID(), file).call();
} catch (Throwable t) {
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/resources/StoreManager.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/resources/StoreManager.java 2010-09-07 20:14:45 UTC (rev 3514)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/resources/StoreManager.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -104,7 +104,6 @@
import com.bigdata.service.IBigdataFederation;
import com.bigdata.service.ManagedResourceService;
import com.bigdata.service.MetadataService;
-import com.bigdata.service.ResourceService;
import com.bigdata.sparse.SparseRowStore;
import com.bigdata.util.concurrent.DaemonThreadFactory;
import com.ibm.icu.impl.ByteBuffer;
@@ -657,24 +656,24 @@
}
- /**
- * The port at which you can connect to the {@link ResourceService}. This
- * service provides remote access to resources hosted by the owning
- * {@link DataService}. This is used for moving resources to other data
- * services in the federation, including supporting service failover.
- *
- * @return The port used to connect to that service.
- *
- * @todo this could also be used for remote backup. however, note that you
- * can not read the live journal using this object.
- */
- public int getResourceServicePort() {
-
- assertRunning();
-
- return resourceService.port;
-
- }
+// /**
+// * The port at which you can connect to the {@link ResourceService}. This
+// * service provides remote access to resources hosted by the owning
+// * {@link DataService}. This is used for moving resources to other data
+// * services in the federation, including supporting service failover.
+// *
+// * @return The port used to connect to that service.
+// *
+// * @todo this could also be used for remote backup. however, note that you
+// * can not read the live journal using this object.
+// */
+// public int getResourceServicePort() {
+//
+// assertRunning();
+//
+// return resourceService.port;
+//
+// }
/**
* @see Options#IGNORE_BAD_FILES
@@ -1569,7 +1568,7 @@
resourceService = new ManagedResourceService() {
@Override
- protected File getResource(UUID uuid) throws Exception {
+ protected File getResource(final UUID uuid) throws Exception {
if (!isRunning()) {
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/ManagedResourceService.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/ManagedResourceService.java 2010-09-07 20:14:45 UTC (rev 3514)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/ManagedResourceService.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -64,12 +64,17 @@
private final ConcurrentHashMap<UUID, ByteBuffer> buffers = new ConcurrentHashMap<UUID, ByteBuffer>();
/**
+ * Constructor uses the default nic, any free port, and the default request
+ * service pool size.
+ *
* @throws IOException
*/
public ManagedResourceService() throws IOException {
+ super();
}
/**
+ *
* @param port
* @throws IOException
*/
@@ -82,8 +87,8 @@
* @param requestServicePoolSize
* @throws IOException
*/
- public ManagedResourceService(int port, int requestServicePoolSize)
- throws IOException {
+ public ManagedResourceService(final int port,
+ final int requestServicePoolSize) throws IOException {
super(port, requestServicePoolSize);
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/ResourceService.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/ResourceService.java 2010-09-07 20:14:45 UTC (rev 3514)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/ResourceService.java 2010-09-08 13:16:30 UTC (rev 3515)
@@ -39,6 +39,7 @@
import java.io.InputStream;
import java.io.OutputStream;
import java.net.InetAddress;
+import java.net.InetSocketAddress;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.ByteBuffer;
@@ -70,11 +71,12 @@
import com.bigdata.rawstore.Bytes;
import com.bigdata.util.concurrent.DaemonThreadFactory;
import com.bigdata.util.concurrent.ShutdownHelper;
+import com.bigdata.util.config.NicUtil;
/**
- * A class which permits buffers identified by a {@link UUID} to be read by a
- * remote service. This class runs one thread to accept connections and thread
- * pool to send data.
+ * A service which permits resources (managed files or buffers) identified by a
+ * {@link UUID} to be read by a remote service. This class runs one thread to
+ * accept connections and thread pool to send data.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
* @version $Id$
@@ -92,6 +94,12 @@
* overhead with establishing those connections and problems with
* immediate reconnect under heavy load.
*
+ * @todo Should have start() method to defer initialization of thread pools and
+ * the {@link ServerSocket} until outside of the constructor. Either
+ * handle visibility appropriately for those fields or move them and the
+ * shutdown protocol into an inner class which is only initialized when
+ * the service is running.
+ *
* @todo Verify that multiple transfers may proceed in parallel to the from the
* same source to the same receiver. This is important in order for low
* latency queries to remain lively when we are moving a shard from one
@@ -135,9 +143,15 @@
protected static final Logger log = Logger.getLogger(ResourceService.class);
/**
- * The port on which the service is accepting connections.
+ * The Internet socket address at which the service is accepting
+ * connections.
*/
- public final int port;
+ private final InetSocketAddress addr;
+
+// /**
+// * The port on which the service is accepting connections.
+// */
+// public final int port;
/**
* The server socket.
@@ -150,163 +164,22 @@
private volatile boolean open = false;
/**
- * Performance counters for the {@link BufferService}.
- *
- * @todo could also monitor the accept and request thread pools. The latter
- * is the more interesting from a workload perspective.
+ * Performance counters for this service.
*/
- static public class Counters {
+ public final Counters counters = new Counters();
- /**
- * #of requests.
- */
- public final CAT requestCount = new CAT();
-
- /**
- * #of requests which are denied.
- */
- public final CAT denyCount = new CAT();
-
- /**
- * #of requests for which the resource was not found.
- */
- public final CAT notFoundCount = new CAT();
-
- /**
- * #of requests which end in an internal error.
- */
- public final CAT internalErrorCount = new CAT();
-
- /**
- * #of errors for responses where we attempt to write the requested data
- * on the socket.
- */
- public final CAT writeErrorCount = new CAT();
-
- /**
- * #of responses where we attempt to write the data on the socket.
- */
- public final CAT nwrites = new CAT();
-
- /**
- * #of data bytes sent.
- */
- public final CAT bytesWritten = new CAT();
-
- /**
- * The largest response written so far.
- */
- public long maxWriteSize;
-
- /**
- * A lock used to make updates to {@link #maxWriteSize} atomic.
- */
- final private Object maxWriteSizeLock = new Object();
-
- /**
- * #of nanoseconds sending data (this will double count time for data
- * that are served concurrently to different receivers).
- */
- public final CAT elapsedWriteNanos = new CAT();
-
- synchronized public CounterSet getCounters() {
-
- if (root == null) {
-
- root = new CounterSet();
-
- /*
- * #of requests and their status outcome counters.
- */
- {
- final CounterSet tmp = root.makePath("status");
-
- tmp.addCounter("Request Count", new Instrument<Long>() {
- public void sample() {
- setValue(requestCount.get());
- }
- });
-
- tmp.addCounter("Deny Count", new Instrument<Long>() {
- public void sample() {
- setValue(denyCount.get());
- }
- });
-
- tmp.addCounter("Not Found Count", new Instrument<Long>() {
- public void sample() {
- setValue(notFoundCount.get());
- }
- });
-
- tmp.addCounter("Internal Error Count", new Instrument<Long>() {
- public void sample() {
- setValue(internalErrorCount.get());
- }
- });
-
- }
-
- /*
- * writes (A write is a response where we try to write the file
- * on the socket).
- */
- {
-
- final CounterSet tmp = root.makePath("writes");
- tmp.addCounter("nwrites", new Instrument<Long>() {
- public void sample() {
- setValue(nwrites.get());
- }
- });
-
- tmp.addCounter("bytesWritten", new Instrument<Long>() {
- public void sample() {
- ...
[truncated message content] |
|
From: <fko...@us...> - 2010-09-07 20:14:52
|
Revision: 3514
http://bigdata.svn.sourceforge.net/bigdata/?rev=3514&view=rev
Author: fkoliver
Date: 2010-09-07 20:14:45 +0000 (Tue, 07 Sep 2010)
Log Message:
-----------
Remove additional references to SocketAppender, every process should have
its own log.
Modified Paths:
--------------
branches/maven_scaleout/bigdata-core/src/main/deploy/legacy/scripts/bigdata
branches/maven_scaleout/bigdata-core/src/main/deploy/var/config/logging/log4jStandalone.properties
Removed Paths:
-------------
branches/maven_scaleout/bigdata-core/src/main/deploy/var/config/logging/log4jServer.properties
Modified: branches/maven_scaleout/bigdata-core/src/main/deploy/legacy/scripts/bigdata
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/deploy/legacy/scripts/bigdata 2010-09-07 19:08:05 UTC (rev 3513)
+++ branches/maven_scaleout/bigdata-core/src/main/deploy/legacy/scripts/bigdata 2010-09-07 20:14:45 UTC (rev 3514)
@@ -68,8 +68,7 @@
#
# Note: Convert DOS cr-lf to unix style in emacs: C-x RET f then unix
#
-# @todo rollover the stateLog file. The log4j log files are rolled over by
-# the log4jServer.properties configuration.
+# @todo rollover the stateLog file.
#
# @todo could use nc (netcat) to read the argument file a url and thus define
# a 'url' as well the 'file' option for this script.
Deleted: branches/maven_scaleout/bigdata-core/src/main/deploy/var/config/logging/log4jServer.properties
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/deploy/var/config/logging/log4jServer.properties 2010-09-07 19:08:05 UTC (rev 3513)
+++ branches/maven_scaleout/bigdata-core/src/main/deploy/var/config/logging/log4jServer.properties 2010-09-07 20:14:45 UTC (rev 3514)
@@ -1,72 +0,0 @@
-# A log4j configuration for a SimpleSocketLogger (a logger daemon). The
-# clients and services are configured in [log4j.properties]. In this file
-# you specify how log events directed to the logger daemon are to be handled.
-
-log4j.rootCategory=ALL, errorLog, detailLog
-
-##
-# log ERROR+ onto the errorLog.
-log4j.appender.errorLog=org.apache.log4j.DailyRollingFileAppender
-log4j.appender.errorLog.Threshold=ERROR
-log4j.appender.errorLog.File=@ERROR_LOG@
-log4j.appender.errorLog.DatePattern=@LOG4J_DATE_PATTERN@
-log4j.appender.errorLog.Append=true
-# error messages show up immediately.
-log4j.appender.errorLog.BufferedIO=false
-log4j.appender.errorLog.layout=org.apache.log4j.PatternLayout
-log4j.appender.errorLog.layout.ConversionPattern=%5p [%t] %l %d - %m%n
-
-##
-# log everything (ALL) onto the detailLog.
-#
-# Note: The threshold is low here since we want to log anything that is
-# relayed from the clients/services onto the detail log. You should control
-# the verbosity from the log4j.properties file rather than here.
-log4j.appender.detailLog=org.apache.log4j.DailyRollingFileAppender
-log4j.appender.detailLog.Threshold=ALL
-log4j.appender.detailLog.File=@DETAIL_LOG@
-log4j.appender.detailLog.DatePattern=@LOG4J_DATE_PATTERN@
-log4j.appender.detailLog.Append=true
-# This is higher volume and typically less critical so you MAY allow buffered
-# IO but I find that it is nicer to have this unbuffered, especially for startup
-# since some problems have to be diagnosed from this log.
-log4j.appender.detailLog.BufferedIO=false
-log4j.appender.detailLog.layout=org.apache.log4j.PatternLayout
-log4j.appender.detailLog.layout.ConversionPattern=%5p [%t] %l %d - %m%n
-
-##
-# log bigdata events onto the eventLog.
-#
-# This is the sink for bigdata events, which describe things such as index
-# partition operations. The events are written using a tab-delimited format.
-# The logger layout should not add anything or it can cause problems when
-# reading the events back into a worksheet or the like.
-log4j.logger.com.bigdata.service.EventReceiver=INFO,eventLog
-log4j.additivity.com.bigdata.service.EventReceiver=false
-log4j.appender.eventLog=org.apache.log4j.DailyRollingFileAppender
-log4j.appender.eventLog.Threshold=INFO
-log4j.appender.eventLog.File=@EVENT_LOG@
-log4j.appender.eventLog.DatePattern=@LOG4J_DATE_PATTERN@
-log4j.appender.eventLog.Append=true
-# While this is less critical, I still like to have each event flushed
-# through to the log file so you don't wind up with partial events in the
-# file.
-log4j.appender.eventLog.BufferedIO=false
-log4j.appender.eventLog.layout=org.apache.log4j.PatternLayout
-log4j.appender.eventLog.layout.ConversionPattern=%m
-
-##
-# Rule execution log. This is a formatted log file (comma delimited).
-log4j.logger.com.bigdata.relation.rule.eval.RuleLog=INFO,ruleLog
-log4j.additivity.com.bigdata.relation.rule.eval.RuleLog=false
-log4j.appender.ruleLog=org.apache.log4j.DailyRollingFileAppender
-log4j.appender.ruleLog.Threshold=ALL
-log4j.appender.ruleLog.File=@RULE_LOG@
-log4j.appender.ruleLog.DatePattern=@LOG4J_DATE_PATTERN@
-log4j.appender.ruleLog.Append=true
-# I find that it is nicer to have this unbuffered since you can see what
-# is going on and to make sure that I have complete rule evaluation logs
-# on shutdown.
-log4j.appender.ruleLog.BufferedIO=false
-log4j.appender.ruleLog.layout=org.apache.log4j.PatternLayout
-log4j.appender.ruleLog.layout.ConversionPattern=%m
Modified: branches/maven_scaleout/bigdata-core/src/main/deploy/var/config/logging/log4jStandalone.properties
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/deploy/var/config/logging/log4jStandalone.properties 2010-09-07 19:08:05 UTC (rev 3513)
+++ branches/maven_scaleout/bigdata-core/src/main/deploy/var/config/logging/log4jStandalone.properties 2010-09-07 20:14:45 UTC (rev 3514)
@@ -1,24 +1,6 @@
# log4j client configuration.
#
-# Note: This is the configuration that the application should use. It
-# sends log events to a log4j server using the [socketLogger]. This
-# lets you aggregate log events across a cluster. The log4j server
-# handles things like rolling over the log files.
-#
-# Note: If the network link to the log4j server goes down, then the
-# application will block. This makes that link a single point of
-# failure.
-#
-# Note: If the log4j server goes down, then a warning will be logged
-# on stderr for the application.
-#
-# Note: The application will proceed no faster than the TCP packets
-# containing the log events that are being sent to the server. This
-# should not be a problem unless very detailed logging is turned on.
-#
-# Note: The layout is NOT used by the SocketAppender. The layout is
-# decided on the other end by the log4j server's configuration file.
-#
+# Note: This is the configuration that the application should use.
# You can also enable local logging. You will have to choose a well
# known location for the log file since the same log4j configuration
# is used by all of the services (at least, by all of the same service
@@ -29,19 +11,10 @@
# Configure appenders to be used.
##
-# Setup relays log messages to a remote socket logger. The remote socket
-# logger is then configured in [log4jServer.properties]. It will log ALL
-# on the detailLog, ERROR+ on the errorLog and events on the eventLog.
-# However, it will ONLY log those messages which are logged here onto the
-# [socketLogger].
-#
-# Note: ERROR+ is also logged onto a local console so you can see any problems
-# during startup when the socket logger might not be running.
-#
# Note: You can control many of the logger levels (on a service-by-service
# bases) using remote JMX MBeans.
#
-log4j.rootCategory=WARN, socketLogger, errorConsole
+log4j.rootCategory=INFO, errorConsole
# Loggers.
log4j.logger.com.bigdata=WARN
@@ -117,38 +90,13 @@
# Setup the possible appenders (log destinations and layouts).
##
-# errorConsole : logs ERROR+ onto the console. This is redundant
-# with the socketLogger, but it is nice to see the errors locally
-# as well during startup or if the socketLogger is down.
+# errorConsole : logs INFO+ onto the console.
#
log4j.appender.errorConsole=org.apache.log4j.ConsoleAppender
-log4j.appender.errorConsole.Threshold=ERROR
+log4j.appender.errorConsole.Threshold=INFO
log4j.appender.errorConsole.layout=org.apache.log4j.PatternLayout
log4j.appender.errorConsole.layout.ConversionPattern=%-5p: %r %X{clientname} %X{hostname} %X{serviceName} %t %l: %m%n
-# socketLogger : logs onto a remote logging service.
-#
-# Note: This is a good way to aggregate your log across a cluster.
-# The remote service is configured to log messages onto the different
-# files (eventLog, errorLog, and detailLog). Since there is only one
-# process writing on those log files, there is no risk of log messages
-# overwrites.
-#
-# Note: if you don't have anything configured to accept the events
-# from the socketLogger appender, you'll see an exception on program
-# startup (to console), and occasional status messages (to console)
-# on if the log4j system has managed to connect to the specified
-# socket.
-#
-# Note: One annoying feature of using this logger is that it DOES NOT
-# use a layout. This means that the MDC variables such as the hostname
-# and serviceName ARE NOT available when the messages get logged.
-#
-log4j.appender.socketLogger=org.apache.log4j.net.SocketAppender
-log4j.appender.socketLogger.RemoteHost=@LOG4J_SOCKET_LOGGER_HOST@
-log4j.appender.socketLogger.Port=@LOG4J_SOCKET_LOGGER_PORT@
-log4j.appender.socketLogger.LocationInfo=true
-
# SYSLOG : logs onto a remote syslogd service.
#
# Personally, I've never setup SYSLOG based logging, so you will
@@ -220,4 +168,4 @@
#log4j.appender.EventAppender.layout=org.apache.log4j.PatternLayout
#log4j.appender.EventAppender.layout.ConversionPattern=%m
-#log4j.logger.com.bigdata.loadbalancer=DEBUG
\ No newline at end of file
+#log4j.logger.com.bigdata.loadbalancer=DEBUG
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <ble...@us...> - 2010-09-07 19:08:11
|
Revision: 3513
http://bigdata.svn.sourceforge.net/bigdata/?rev=3513&view=rev
Author: blevine218
Date: 2010-09-07 19:08:05 +0000 (Tue, 07 Sep 2010)
Log Message:
-----------
removed Eclipse-specific files
Removed Paths:
-------------
branches/maven_scaleout/bigdata-integ/.classpath
branches/maven_scaleout/bigdata-integ/.project
branches/maven_scaleout/bigdata-integ/.settings/
Property Changed:
----------------
branches/maven_scaleout/bigdata-integ/
Property changes on: branches/maven_scaleout/bigdata-integ
___________________________________________________________________
Added: svn:ignore
+ target
Deleted: branches/maven_scaleout/bigdata-integ/.classpath
===================================================================
--- branches/maven_scaleout/bigdata-integ/.classpath 2010-09-07 19:04:36 UTC (rev 3512)
+++ branches/maven_scaleout/bigdata-integ/.classpath 2010-09-07 19:08:05 UTC (rev 3513)
@@ -1,16 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<classpath>
- <classpathentry kind="src" output="target/classes" path="src/main/java"/>
- <classpathentry excluding="**" kind="src" output="target/classes" path="src/main/resources"/>
- <classpathentry kind="src" output="target/test-classes" path="src/test/java"/>
- <classpathentry excluding="**" kind="src" output="target/test-classes" path="src/test/resources"/>
- <classpathentry kind="con" path="org.eclipse.jst.j2ee.internal.web.container"/>
- <classpathentry kind="con" path="org.eclipse.jst.j2ee.internal.module.container"/>
- <classpathentry kind="con" path="org.maven.ide.eclipse.MAVEN2_CLASSPATH_CONTAINER"/>
- <classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER/org.eclipse.jdt.internal.debug.ui.launcher.StandardVMType/java-6-sun-1.6.0.20">
- <attributes>
- <attribute name="owner.project.facets" value="jst.java"/>
- </attributes>
- </classpathentry>
- <classpathentry kind="output" path="target/classes"/>
-</classpath>
Deleted: branches/maven_scaleout/bigdata-integ/.project
===================================================================
--- branches/maven_scaleout/bigdata-integ/.project 2010-09-07 19:04:36 UTC (rev 3512)
+++ branches/maven_scaleout/bigdata-integ/.project 2010-09-07 19:08:05 UTC (rev 3513)
@@ -1,42 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<projectDescription>
- <name>rds-integration-test</name>
- <comment></comment>
- <projects>
- </projects>
- <buildSpec>
- <buildCommand>
- <name>org.eclipse.wst.jsdt.core.javascriptValidator</name>
- <arguments>
- </arguments>
- </buildCommand>
- <buildCommand>
- <name>org.eclipse.wst.common.project.facet.core.builder</name>
- <arguments>
- </arguments>
- </buildCommand>
- <buildCommand>
- <name>org.eclipse.jdt.core.javabuilder</name>
- <arguments>
- </arguments>
- </buildCommand>
- <buildCommand>
- <name>org.eclipse.wst.validation.validationbuilder</name>
- <arguments>
- </arguments>
- </buildCommand>
- <buildCommand>
- <name>org.maven.ide.eclipse.maven2Builder</name>
- <arguments>
- </arguments>
- </buildCommand>
- </buildSpec>
- <natures>
- <nature>org.eclipse.jem.workbench.JavaEMFNature</nature>
- <nature>org.eclipse.wst.common.modulecore.ModuleCoreNature</nature>
- <nature>org.eclipse.jdt.core.javanature</nature>
- <nature>org.maven.ide.eclipse.maven2Nature</nature>
- <nature>org.eclipse.wst.common.project.facet.core.nature</nature>
- <nature>org.eclipse.wst.jsdt.core.jsNature</nature>
- </natures>
-</projectDescription>
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <ble...@us...> - 2010-09-07 19:04:44
|
Revision: 3512
http://bigdata.svn.sourceforge.net/bigdata/?rev=3512&view=rev
Author: blevine218
Date: 2010-09-07 19:04:36 +0000 (Tue, 07 Sep 2010)
Log Message:
-----------
first checkin of framework for bigdata integration tests
Added Paths:
-----------
branches/maven_scaleout/bigdata-integ/
branches/maven_scaleout/bigdata-integ/.classpath
branches/maven_scaleout/bigdata-integ/.project
branches/maven_scaleout/bigdata-integ/.settings/
branches/maven_scaleout/bigdata-integ/.settings/.jsdtscope
branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.jdt.core.prefs
branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.wst.common.component
branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.wst.common.project.facet.core.xml
branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.wst.jsdt.ui.superType.container
branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.wst.jsdt.ui.superType.name
branches/maven_scaleout/bigdata-integ/.settings/org.maven.ide.eclipse.prefs
branches/maven_scaleout/bigdata-integ/pom.xml
branches/maven_scaleout/bigdata-integ/src/
branches/maven_scaleout/bigdata-integ/src/main/
branches/maven_scaleout/bigdata-integ/src/main/java/
branches/maven_scaleout/bigdata-integ/src/main/resources/
branches/maven_scaleout/bigdata-integ/src/test/
branches/maven_scaleout/bigdata-integ/src/test/java/
branches/maven_scaleout/bigdata-integ/src/test/java/com/
branches/maven_scaleout/bigdata-integ/src/test/java/com/bigdata/
branches/maven_scaleout/bigdata-integ/src/test/java/com/bigdata/test/
branches/maven_scaleout/bigdata-integ/src/test/java/com/bigdata/test/FakeTest.java
branches/maven_scaleout/bigdata-integ/src/test/resources/
Added: branches/maven_scaleout/bigdata-integ/.classpath
===================================================================
--- branches/maven_scaleout/bigdata-integ/.classpath (rev 0)
+++ branches/maven_scaleout/bigdata-integ/.classpath 2010-09-07 19:04:36 UTC (rev 3512)
@@ -0,0 +1,16 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<classpath>
+ <classpathentry kind="src" output="target/classes" path="src/main/java"/>
+ <classpathentry excluding="**" kind="src" output="target/classes" path="src/main/resources"/>
+ <classpathentry kind="src" output="target/test-classes" path="src/test/java"/>
+ <classpathentry excluding="**" kind="src" output="target/test-classes" path="src/test/resources"/>
+ <classpathentry kind="con" path="org.eclipse.jst.j2ee.internal.web.container"/>
+ <classpathentry kind="con" path="org.eclipse.jst.j2ee.internal.module.container"/>
+ <classpathentry kind="con" path="org.maven.ide.eclipse.MAVEN2_CLASSPATH_CONTAINER"/>
+ <classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER/org.eclipse.jdt.internal.debug.ui.launcher.StandardVMType/java-6-sun-1.6.0.20">
+ <attributes>
+ <attribute name="owner.project.facets" value="jst.java"/>
+ </attributes>
+ </classpathentry>
+ <classpathentry kind="output" path="target/classes"/>
+</classpath>
Added: branches/maven_scaleout/bigdata-integ/.project
===================================================================
--- branches/maven_scaleout/bigdata-integ/.project (rev 0)
+++ branches/maven_scaleout/bigdata-integ/.project 2010-09-07 19:04:36 UTC (rev 3512)
@@ -0,0 +1,42 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<projectDescription>
+ <name>rds-integration-test</name>
+ <comment></comment>
+ <projects>
+ </projects>
+ <buildSpec>
+ <buildCommand>
+ <name>org.eclipse.wst.jsdt.core.javascriptValidator</name>
+ <arguments>
+ </arguments>
+ </buildCommand>
+ <buildCommand>
+ <name>org.eclipse.wst.common.project.facet.core.builder</name>
+ <arguments>
+ </arguments>
+ </buildCommand>
+ <buildCommand>
+ <name>org.eclipse.jdt.core.javabuilder</name>
+ <arguments>
+ </arguments>
+ </buildCommand>
+ <buildCommand>
+ <name>org.eclipse.wst.validation.validationbuilder</name>
+ <arguments>
+ </arguments>
+ </buildCommand>
+ <buildCommand>
+ <name>org.maven.ide.eclipse.maven2Builder</name>
+ <arguments>
+ </arguments>
+ </buildCommand>
+ </buildSpec>
+ <natures>
+ <nature>org.eclipse.jem.workbench.JavaEMFNature</nature>
+ <nature>org.eclipse.wst.common.modulecore.ModuleCoreNature</nature>
+ <nature>org.eclipse.jdt.core.javanature</nature>
+ <nature>org.maven.ide.eclipse.maven2Nature</nature>
+ <nature>org.eclipse.wst.common.project.facet.core.nature</nature>
+ <nature>org.eclipse.wst.jsdt.core.jsNature</nature>
+ </natures>
+</projectDescription>
Added: branches/maven_scaleout/bigdata-integ/.settings/.jsdtscope
===================================================================
--- branches/maven_scaleout/bigdata-integ/.settings/.jsdtscope (rev 0)
+++ branches/maven_scaleout/bigdata-integ/.settings/.jsdtscope 2010-09-07 19:04:36 UTC (rev 3512)
@@ -0,0 +1,11 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<classpath>
+ <classpathentry kind="con" path="org.eclipse.wst.jsdt.launching.JRE_CONTAINER"/>
+ <classpathentry kind="con" path="org.eclipse.wst.jsdt.launching.WebProject">
+ <attributes>
+ <attribute name="hide" value="true"/>
+ </attributes>
+ </classpathentry>
+ <classpathentry kind="con" path="org.eclipse.wst.jsdt.launching.baseBrowserLibrary"/>
+ <classpathentry kind="output" path=""/>
+</classpath>
Added: branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.jdt.core.prefs
===================================================================
--- branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.jdt.core.prefs (rev 0)
+++ branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.jdt.core.prefs 2010-09-07 19:04:36 UTC (rev 3512)
@@ -0,0 +1,7 @@
+#Tue Sep 07 12:06:36 EDT 2010
+eclipse.preferences.version=1
+org.eclipse.jdt.core.compiler.codegen.targetPlatform=1.5
+org.eclipse.jdt.core.compiler.compliance=1.5
+org.eclipse.jdt.core.compiler.problem.assertIdentifier=error
+org.eclipse.jdt.core.compiler.problem.enumIdentifier=error
+org.eclipse.jdt.core.compiler.source=1.5
Added: branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.wst.common.component
===================================================================
--- branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.wst.common.component (rev 0)
+++ branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.wst.common.component 2010-09-07 19:04:36 UTC (rev 3512)
@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project-modules id="moduleCoreId" project-version="1.5.0">
+ <wb-module deploy-name="rds-integration-test">
+ <wb-resource deploy-path="/" source-path="/src/main/webapp"/>
+ <wb-resource deploy-path="/WEB-INF/classes" source-path="/src/main/java"/>
+ <wb-resource deploy-path="/WEB-INF/classes" source-path="/src/main/resources"/>
+ <property name="context-root" value="rds-integration-test"/>
+ <property name="java-output-path"/>
+ </wb-module>
+</project-modules>
Added: branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.wst.common.project.facet.core.xml
===================================================================
--- branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.wst.common.project.facet.core.xml (rev 0)
+++ branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.wst.common.project.facet.core.xml 2010-09-07 19:04:36 UTC (rev 3512)
@@ -0,0 +1,5 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<faceted-project>
+ <installed facet="jst.web" version="2.5"/>
+ <installed facet="jst.java" version="5.0"/>
+</faceted-project>
Added: branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.wst.jsdt.ui.superType.container
===================================================================
--- branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.wst.jsdt.ui.superType.container (rev 0)
+++ branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.wst.jsdt.ui.superType.container 2010-09-07 19:04:36 UTC (rev 3512)
@@ -0,0 +1 @@
+org.eclipse.wst.jsdt.launching.baseBrowserLibrary
\ No newline at end of file
Added: branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.wst.jsdt.ui.superType.name
===================================================================
--- branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.wst.jsdt.ui.superType.name (rev 0)
+++ branches/maven_scaleout/bigdata-integ/.settings/org.eclipse.wst.jsdt.ui.superType.name 2010-09-07 19:04:36 UTC (rev 3512)
@@ -0,0 +1 @@
+Window
\ No newline at end of file
Added: branches/maven_scaleout/bigdata-integ/.settings/org.maven.ide.eclipse.prefs
===================================================================
--- branches/maven_scaleout/bigdata-integ/.settings/org.maven.ide.eclipse.prefs (rev 0)
+++ branches/maven_scaleout/bigdata-integ/.settings/org.maven.ide.eclipse.prefs 2010-09-07 19:04:36 UTC (rev 3512)
@@ -0,0 +1,9 @@
+#Wed May 12 12:42:51 EDT 2010
+activeProfiles=
+eclipse.preferences.version=1
+fullBuildGoals=process-test-resources
+includeModules=false
+resolveWorkspaceProjects=true
+resourceFilterGoals=process-resources resources\:testResources
+skipCompilerPlugin=true
+version=1
Added: branches/maven_scaleout/bigdata-integ/pom.xml
===================================================================
--- branches/maven_scaleout/bigdata-integ/pom.xml (rev 0)
+++ branches/maven_scaleout/bigdata-integ/pom.xml 2010-09-07 19:04:36 UTC (rev 3512)
@@ -0,0 +1,56 @@
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+
+
+ <parent>
+ <groupId>com.bigdata</groupId>
+ <artifactId>bigdata</artifactId>
+ <version>0.83.2-SNAPSHOT</version>
+ </parent>
+
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>com.bigdata</groupId>
+ <artifactId>bigdata-integration-test</artifactId>
+ <name>bigdata Integration Tests</name>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>failsafe-maven-plugin</artifactId>
+ <version>2.4.3-alpha-1</version>
+ <configuration>
+ <includes>
+ <include>**/*Test.java</include>
+ </includes>
+ </configuration>
+ <executions>
+ <execution>
+ <goals>
+ <goal>integration-test</goal>
+ <goal>verify</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+
+ <dependencies>
+ <dependency>
+ <groupId>com.bigdata</groupId>
+ <artifactId>bigdata-core</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>4.7</version>
+ <scope>test</scope>
+ </dependency>
+
+ </dependencies>
+
+</project>
Added: branches/maven_scaleout/bigdata-integ/src/test/java/com/bigdata/test/FakeTest.java
===================================================================
--- branches/maven_scaleout/bigdata-integ/src/test/java/com/bigdata/test/FakeTest.java (rev 0)
+++ branches/maven_scaleout/bigdata-integ/src/test/java/com/bigdata/test/FakeTest.java 2010-09-07 19:04:36 UTC (rev 3512)
@@ -0,0 +1,69 @@
+/***********************************************************************
+ *
+ * $Id$
+ *
+ * Copyright (c) 2000-2010 Nokia Corporation.
+ *
+ * This material, including documentation and any related computer
+ * programs, is protected by copyright controlled by Nokia Corporation.
+ * All rights are reserved. Copying, including reproducing, storing,
+ * adapting or translating, any or all of this material requires the prior
+ * written consent of Nokia Corporation. This material also contains
+ * confidential information which may not be disclosed to others without
+ * the prior written consent of Nokia Corporation.
+ **********************************************************************/
+package com.bigdata.test;
+
+
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * @author blevine
+ *
+ */
+public class FakeTest
+{
+
+ /**
+ * @throws java.lang.Exception
+ */
+ @BeforeClass
+ public static void setUpBeforeClass() throws Exception
+ {
+ }
+
+ /**
+ * @throws java.lang.Exception
+ */
+ @AfterClass
+ public static void tearDownAfterClass() throws Exception
+ {
+ }
+
+ /**
+ * @throws java.lang.Exception
+ */
+ @Before
+ public void setUp() throws Exception
+ {
+ }
+
+ /**
+ * @throws java.lang.Exception
+ */
+ @After
+ public void tearDown() throws Exception
+ {
+ }
+
+ @Test
+ public void theTest()
+ {
+ System.out.println("HELLO WORLD!!!!");
+ }
+
+}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-09-06 20:45:47
|
Revision: 3511
http://bigdata.svn.sourceforge.net/bigdata/?rev=3511&view=rev
Author: thompsonbry
Date: 2010-09-06 20:45:37 +0000 (Mon, 06 Sep 2010)
Log Message:
-----------
Modified the IBindingSet API to define the contract for hashCode(). This was necessary in order for DISTINCT on binding sets to work correctly.
Reorganized the QueryEngine classes out of the test suite.
Added the basic framework for the FederatedQueryEngine test suite.
Added a "de-chunk" pattern to unchunk iterators.
Reconciled the BufferService with the ResourceService, including their test suites.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ArrayBindingSet.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/EmptyBindingSet.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/HashBindingSet.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IBindingSet.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPipelineOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/AbstractStepTask.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/ProgramTask.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/resources/ResourceManager.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/resources/StoreManager.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/DataService.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/ResourceService.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/striterator/GenericStriterator.java
branches/QUADS_QUERY_BRANCH/bigdata/src/resources/logging/log4j.properties
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/TestBindingSet.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestFederatedQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/service/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/striterator/TestAll.java
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/BOpStats.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/BSBundle.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/BindingSetChunk.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/HaltOpMessage.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IQueryClient.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IQueryPeer.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IRunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/PipelineUtility.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/StartOpMessage.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/DelegateIndexManager.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedRunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/MapBindingSetsOverShardsBuffer.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/ManagedResourceService.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/striterator/Dechunkerator.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/service/TestAll_DynamicSharding.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/service/TestAll_ResourceService.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/service/TestCase3.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/service/TestReceiveBuffer.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/service/TestReceiveFile.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/striterator/TestDechunkerator.java
Removed Paths:
-------------
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/BOpShard.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/BOpStats.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/BindingSetChunk.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/BufferService.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/FederatedQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/HaltOpMessage.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/IQueryClient.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/IQueryPeer.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/IRunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/ManagedBufferService.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/PipelineUtility.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/QueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/QueryEngine.properties
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/ReceiveBindingSets.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/Rule2BOpUtility.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/RunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/StartOpMessage.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestCase3.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestReceiveBuffer.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestReceiveFile.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/MapBindingSetsOverShardsBuffer.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestSendReceiveBuffers.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/service/TestResourceService.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ArrayBindingSet.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ArrayBindingSet.java 2010-09-05 20:17:44 UTC (rev 3510)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ArrayBindingSet.java 2010-09-06 20:45:37 UTC (rev 3511)
@@ -225,6 +225,9 @@
}
+ // clear the hash code.
+ hash = 0;
+
assert nbound == 0;
}
@@ -261,6 +264,9 @@
}
+ // clear the hash code.
+ hash = 0;
+
nbound--;
break;
@@ -316,7 +322,10 @@
if (vars[i] == var) {
vals[i] = val;
-
+
+ // clear the hash code.
+ hash = 0;
+
return;
}
@@ -327,6 +336,9 @@
vals[nbound] = val;
+ // clear the hash code.
+ hash = 0;
+
nbound++;
}
@@ -408,11 +420,16 @@
}
- public boolean equals(final IBindingSet o) {
+ public boolean equals(final Object t) {
- if (o == this)
+ if (this == t)
return true;
+ if(!(t instanceof IBindingSet))
+ return false;
+
+ final IBindingSet o = (IBindingSet)t;
+
if (nbound != o.size())
return false;
@@ -430,4 +447,27 @@
}
+ public int hashCode() {
+
+ if (hash == 0) {
+
+ int result = 0;
+
+ for (int i = 0; i < nbound; i++) {
+
+ if (vals[i] == null)
+ continue;
+
+ result ^= vals[i].hashCode();
+
+ }
+
+ hash = result;
+
+ }
+ return hash;
+
+ }
+ private int hash;
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/EmptyBindingSet.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/EmptyBindingSet.java 2010-09-05 20:17:44 UTC (rev 3510)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/EmptyBindingSet.java 2010-09-06 20:45:37 UTC (rev 3511)
@@ -96,18 +96,32 @@
return 0;
}
- public boolean equals(IBindingSet o) {
+ public boolean equals(final Object t) {
- if (this == o)
+ if (this == t)
return true;
-
+
+ if (!(t instanceof IBindingSet))
+ return false;
+
+ final IBindingSet o = (IBindingSet) t;
+
if (o.size() == 0)
return true;
-
+
return false;
}
+ /**
+ * The hash code of an empty binding set is always zero.
+ */
+ public int hashCode() {
+
+ return 0;
+
+ }
+
public IConstant get(IVariable var) {
if (var == null)
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/HashBindingSet.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/HashBindingSet.java 2010-09-05 20:17:44 UTC (rev 3510)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/HashBindingSet.java 2010-09-06 20:45:37 UTC (rev 3511)
@@ -146,6 +146,9 @@
throw new IllegalArgumentException();
map.put(var,val);
+
+ // clear the hash code.
+ hash = 0;
}
@@ -156,12 +159,18 @@
map.remove(var);
+ // clear the hash code.
+ hash = 0;
+
}
public void clearAll() {
map.clear();
-
+
+ // clear the hash code.
+ hash = 0;
+
}
public String toString() {
@@ -233,11 +242,14 @@
final HashBindingSet bs = new HashBindingSet();
- for (IVariable var : variablesToKeep) {
+ for (IVariable<?> var : variablesToKeep) {
- IConstant val = map.get(var);
+ final IConstant<?> val = map.get(var);
+
if (val != null) {
+
bs.map.put(var, val);
+
}
}
@@ -246,11 +258,16 @@
}
- public boolean equals(final IBindingSet o) {
+ public boolean equals(final Object t) {
- if (o == this)
+ if (this == t)
return true;
+ if(!(t instanceof IBindingSet))
+ return false;
+
+ final IBindingSet o = (IBindingSet) t;
+
if (size() != o.size())
return false;
@@ -260,9 +277,9 @@
final Map.Entry<IVariable,IConstant> entry = itr.next();
- final IVariable var = entry.getKey();
+ final IVariable<?> var = entry.getKey();
- final IConstant val = entry.getValue();
+ final IConstant<?> val = entry.getValue();
// if (!o.isBound(vars[i]))
// return false;
@@ -276,4 +293,27 @@
}
+ public int hashCode() {
+
+ if (hash == 0) {
+
+ int result = 0;
+
+ for(IConstant<?> c : map.values()) {
+
+ if (c == null)
+ continue;
+
+ result ^= c.hashCode();
+
+ }
+
+ hash = result;
+
+ }
+ return hash;
+
+ }
+ private int hash;
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IBindingSet.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IBindingSet.java 2010-09-05 20:17:44 UTC (rev 3510)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IBindingSet.java 2010-09-06 20:45:37 UTC (rev 3511)
@@ -167,6 +167,20 @@
* @param o
* Another binding set.
*/
- public boolean equals(IBindingSet o);
+ public boolean equals(Object o);
+
+ /**
+ * The hash code of a binding is defined as the bit-wise XOR of the hash
+ * codes of the {@link IConstant}s for its bound variables. Unbound
+ * variables are ignored when computing the hash code. Binding sets are
+ * unordered collections, therefore the calculated hash code intentionally
+ * does not dependent on the order in which the bindings are iterated over.
+ * The hash code reflects the current state of the bindings and must be
+ * recomputed if the bindings are changed.
+ *
+ * @todo the test suites should be enhanced to verify the contract for
+ * {@link IBindingSet#hashCode()}
+ */
+ public int hashCode();
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPipelineOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPipelineOp.java 2010-09-05 20:17:44 UTC (rev 3510)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPipelineOp.java 2010-09-06 20:45:37 UTC (rev 3511)
@@ -27,6 +27,7 @@
package com.bigdata.bop;
+import java.util.concurrent.Callable;
import java.util.concurrent.FutureTask;
import com.bigdata.bop.engine.BOpStats;
@@ -69,6 +70,11 @@
*
* @return The {@link FutureTask} which will compute the operator's
* evaluation.
+ *
+ * @todo Modify to return a {@link Callable}s for now since we must run each
+ * task in its own thread until Java7 gives us fork/join pools and
+ * asynchronous file I/O. For the fork/join model we will probably
+ * return the ForkJoinTask.
*/
FutureTask<Void> eval(BOpContext<E> context);
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/BOpStats.java (from rev 3508, branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/BOpStats.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/BOpStats.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/BOpStats.java 2010-09-06 20:45:37 UTC (rev 3511)
@@ -0,0 +1,134 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Aug 26, 2010
+ */
+
+package com.bigdata.bop.engine;
+
+import java.io.Serializable;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.counters.CAT;
+
+/**
+ * Statistics associated with the evaluation of a {@link BOp}. These statistics
+ * are per {@link BOp}. The top-level {@link BOp} will reflect the throughput
+ * for the entire pipeline.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ *
+ * @todo Add time per bop. This can not be directly aggregated into wall time
+ * since there are concurrent processes. However, this will be useful
+ * since we tend to process materialized chunks with the new
+ * {@link QueryEngine} such that the operator evaluation time now more or
+ * less directly corresponds to the time it takes to act on local data,
+ * producing local outputs. The {@link QueryEngine} itself now handles the
+ * transportation of data between the nodes so that time can be factored
+ * out of the local aspects of query execution.
+ */
+public class BOpStats implements Serializable {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+// /**
+// * The timestamp (milliseconds) associated with the start of execution for
+// * the join dimension. This is not aggregated. It should only be used to
+// * compute the elapsed time for the operator.
+// */
+// private final long startTime;
+
+// /**
+// * The index partition for which these statistics were collected or -1
+// * if the statistics are aggregated across index partitions.
+// */
+// public final int partitionId;
+
+ /**
+ * #of chunks in.
+ */
+ final public CAT chunksIn = new CAT();
+
+ /**
+ * #of units sets in (tuples, elements, binding sets, etc).
+ */
+ final public CAT unitsIn = new CAT();
+
+ /**
+ * #of chunks out.
+ */
+ final public CAT chunksOut = new CAT();
+
+ /**
+ * #of units sets in (tuples, elements, binding sets, etc).
+ */
+ final public CAT unitsOut = new CAT();
+
+ /**
+ * Constructor.
+ */
+ public BOpStats() {
+
+ }
+
+ /**
+ * Combine the statistics (addition).
+ *
+ * @param o
+ * Another statistics object.
+ */
+ public void add(final BOpStats o) {
+ chunksIn.add(o.chunksIn.get());
+ unitsIn.add(o.unitsIn.get());
+ unitsOut.add(o.unitsOut.get());
+ chunksOut.add(o.chunksIn.get());
+ }
+
+ public String toString() {
+ final StringBuilder sb = new StringBuilder();
+ sb.append(getClass().getName());
+ sb.append("{chunksIn=" + chunksIn.estimate_get());
+ sb.append(",unitsIn=" + unitsIn.estimate_get());
+ sb.append(",chunksOut=" + chunksOut.estimate_get());
+ sb.append(",unitsOut=" + unitsOut.estimate_get());
+ toString(sb);
+ sb.append("}");
+ return sb.toString();
+ }
+
+ /**
+ * Extension hook for {@link #toString()}.
+ *
+ * @param sb
+ * Where to write the additional state.
+ */
+ protected void toString(final StringBuilder sb) {
+
+ }
+
+}
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/BSBundle.java (from rev 3489, branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/BOpShard.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/BSBundle.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/BSBundle.java 2010-09-06 20:45:37 UTC (rev 3511)
@@ -0,0 +1,75 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Sep 1, 2010
+ */
+
+package com.bigdata.bop.engine;
+
+/**
+ * An immutable class capturing the evaluation context of an operator against a
+ * shard.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class BSBundle {
+
+ public final int bopId;
+
+ public final int shardId;
+
+ public BSBundle(final int bopId, final int shardId) {
+
+ this.bopId = bopId;
+
+ this.shardId = shardId;
+
+ }
+
+ /**
+ * {@inheritDoc}
+ *
+ * @todo verify that this is a decent hash function.
+ */
+ public int hashCode() {
+
+ return (bopId * 31) + shardId;
+
+ }
+
+ public boolean equals(final Object o) {
+
+ if (this == o)
+ return true;
+
+ if (!(o instanceof BSBundle))
+ return false;
+
+ return bopId == ((BSBundle) o).bopId
+ && shardId == ((BSBundle) o).shardId;
+
+ }
+
+}
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/BindingSetChunk.java (from rev 3472, branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/BindingSetChunk.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/BindingSetChunk.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/BindingSetChunk.java 2010-09-06 20:45:37 UTC (rev 3511)
@@ -0,0 +1,51 @@
+package com.bigdata.bop.engine;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.relation.accesspath.IAsynchronousIterator;
+
+/**
+ * A chunk of intermediate results which are ready to be consumed by some
+ * {@link BOp} in a specific query.
+ */
+public class BindingSetChunk {
+
+ /**
+ * The query identifier.
+ */
+ final long queryId;
+
+ /**
+ * The target {@link BOp}.
+ */
+ final int bopId;
+
+ /**
+ * The index partition which is being targeted for that {@link BOp}.
+ */
+ final int partitionId;
+
+ /**
+ * The binding sets to be consumed by that {@link BOp}.
+ */
+ final IAsynchronousIterator<IBindingSet[]> source;
+
+ public BindingSetChunk(final long queryId, final int bopId,
+ final int partitionId,
+ final IAsynchronousIterator<IBindingSet[]> source) {
+ if (source == null)
+ throw new IllegalArgumentException();
+ this.queryId = queryId;
+ this.bopId = bopId;
+ this.partitionId = partitionId;
+ this.source = source;
+ }
+
+ public String toString() {
+
+ return getClass().getName() + "{queryId=" + queryId + ",bopId=" + bopId
+ + ",partitionId=" + partitionId + "}";
+
+ }
+
+}
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/HaltOpMessage.java (from rev 3489, branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/HaltOpMessage.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/HaltOpMessage.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/HaltOpMessage.java 2010-09-06 20:45:37 UTC (rev 3511)
@@ -0,0 +1,135 @@
+package com.bigdata.bop.engine;
+
+import java.io.Serializable;
+import java.util.UUID;
+
+/**
+ * A message sent to the {@link IQueryClient} when an operator is done executing
+ * for some chunk of inputs.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class HaltOpMessage implements Serializable {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ /** The identifier of the query. */
+ final long queryId;
+
+ /** The identifier of the operator. */
+ final int bopId;
+
+ /**
+ * The index partition identifier against which the operator was
+ * executing.
+ */
+ final int partitionId;
+
+ /**
+ * The identifier of the service on which the operator was executing.
+ */
+ final UUID serviceId;
+
+ /**
+ * * The cause and <code>null</code> if the operator halted normally.
+ */
+ final Throwable cause;
+
+ /**
+ * The operator identifier for the primary sink -or- <code>null</code>
+ * if there is no primary sink (for example, if this is the last
+ * operator in the pipeline).
+ */
+ final Integer sinkId;
+
+ /**
+ * The number of the {@link BindingSetChunk}s that were output for the
+ * primary sink. (This information is used for the atomic termination
+ * decision.)
+ * <p>
+ * For a given downstream operator this is ONE (1) for scale-up. For
+ * scale-out, this is one per index partition over which the
+ * intermediate results were mapped.
+ */
+ final int sinkChunksOut;
+
+ /**
+ * The operator identifier for the alternative sink -or-
+ * <code>null</code> if there is no alternative sink.
+ */
+ final Integer altSinkId;
+
+ /**
+ * The number of the {@link BindingSetChunk}s that were output for the
+ * alternative sink. (This information is used for the atomic
+ * termination decision.)
+ * <p>
+ * For a given downstream operator this is ONE (1) for scale-up. For
+ * scale-out, this is one per index partition over which the
+ * intermediate results were mapped. It is zero if there was no
+ * alternative sink for the operator.
+ */
+ final int altSinkChunksOut;
+
+ /**
+ * The statistics for the execution of the bop against the partition on
+ * the service.
+ */
+ final BOpStats taskStats;
+
+ /**
+ * @param queryId
+ * The query identifier.
+ * @param bopId
+ * The operator whose execution phase has terminated for a
+ * specific index partition and input chunk.
+ * @param partitionId
+ * The index partition against which the operator was
+ * executed.
+ * @param serviceId
+ * The node which executed the operator.
+ * @param cause
+ * <code>null</code> unless execution halted abnormally.
+ * @param chunksOut
+ * A map reporting the #of binding set chunks which were
+ * output for each downstream operator for which at least one
+ * chunk of output was produced.
+ * @param taskStats
+ * The statistics for the execution of that bop on that shard
+ * and service.
+ */
+ public HaltOpMessage(
+ //
+ final long queryId, final int bopId, final int partitionId,
+ final UUID serviceId, Throwable cause, //
+ final Integer sinkId, final int sinkChunksOut,//
+ final Integer altSinkId, final int altSinkChunksOut,//
+ final BOpStats taskStats) {
+
+ if (altSinkId != null && sinkId == null) {
+ // The primary sink must be defined if the altSink is defined.
+ throw new IllegalArgumentException();
+ }
+
+ if (sinkId != null && altSinkId != null
+ && sinkId.intValue() == altSinkId.intValue()) {
+ // The primary and alternative sink may not be the same operator.
+ throw new IllegalArgumentException();
+ }
+
+ this.queryId = queryId;
+ this.bopId = bopId;
+ this.partitionId = partitionId;
+ this.serviceId = serviceId;
+ this.cause = cause;
+ this.sinkId = sinkId;
+ this.sinkChunksOut = sinkChunksOut;
+ this.altSinkId = altSinkId;
+ this.altSinkChunksOut = altSinkChunksOut;
+ this.taskStats = taskStats;
+ }
+}
\ No newline at end of file
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IQueryClient.java (from rev 3489, branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/IQueryClient.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IQueryClient.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IQueryClient.java 2010-09-06 20:45:37 UTC (rev 3511)
@@ -0,0 +1,105 @@
+package com.bigdata.bop.engine;
+
+import java.rmi.Remote;
+import java.rmi.RemoteException;
+
+import com.bigdata.bop.BOp;
+
+/**
+ * Interface for a client executing queries.
+ */
+public interface IQueryClient extends IQueryPeer, Remote {
+
+ /*
+ * @todo Could return a data structure which encapsulates the query results
+ * and could allow multiple results from a query, e.g., one per step in a
+ * program.
+ */
+
+// /**
+// * Evaluate a query which materializes elements, such as an
+// * {@link IPredicate}.
+// *
+// * @param queryId
+// * The unique identifier for the query.
+// * @param timestamp
+// * The timestamp or transaction against which the query will run.
+// * @param query
+// * The query to evaluate.
+// * @param source
+// * The initial binding sets to get the query going (this is
+// * typically an iterator visiting a single empty binding set).
+// *
+// * @return An iterator visiting the elements materialized by the query.
+// *
+// * @throws Exception
+// */
+// public IChunkedIterator<?> eval(long queryId, long timestamp, BOp query)
+// throws Exception;
+
+// /**
+// * Evaluate a query which visits {@link IBindingSet}s, such as a join.
+// *
+// * @param queryId
+// * The unique identifier for the query.
+// * @param timestamp
+// * The timestamp or transaction against which the query will run.
+// * @param query
+// * The query to evaluate.
+// * @param source
+// * The initial binding sets to get the query going (this is
+// * typically an iterator visiting a single empty binding set).
+// *
+// * @return An iterator visiting {@link IBindingSet}s which result from
+// * evaluating the query.
+// *
+// * @throws Exception
+// */
+// public IChunkedIterator<IBindingSet> eval(long queryId, long timestamp,
+// BOp query, IAsynchronousIterator<IBindingSet[]> source)
+// throws Exception;
+
+ /**
+ * Return the query.
+ *
+ * @param queryId
+ * The query identifier.
+ * @return The query.
+ *
+ * @throws RemoteException
+ */
+ public BOp getQuery(long queryId) throws RemoteException;
+
+ /**
+ * Notify the client that execution has started for some query, operator,
+ * node, and index partition.
+ */
+ public void startOp(StartOpMessage msg)
+ throws RemoteException;
+
+ /**
+ * Notify the client that execution has halted for some query, operator,
+ * node, shard, and source binding set chunk(s). If execution halted
+ * abnormally, then the cause is sent as well.
+ */
+ public void haltOp(HaltOpMessage msg) throws RemoteException;
+
+// /**
+// * Notify the query controller that a chunk of intermediate results is
+// * available for the query.
+// *
+// * @param queryId
+// * The query identifier.
+// */
+// public void addChunk(long queryId) throws RemoteException;
+//
+// /**
+// * Notify the query controller that a chunk of intermediate results was
+// * taken for processing by the query.
+// *
+// * @param queryId
+// * The query identifier.
+// */
+// public void takeChunk(long queryId) throws RemoteException;
+
+}
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IQueryPeer.java (from rev 3466, branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/IQueryPeer.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IQueryPeer.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IQueryPeer.java 2010-09-06 20:45:37 UTC (rev 3511)
@@ -0,0 +1,36 @@
+package com.bigdata.bop.engine;
+
+import java.net.InetSocketAddress;
+import java.rmi.Remote;
+import java.rmi.RemoteException;
+
+import com.bigdata.bop.BOp;
+
+/**
+ * Interface for a node participating in the exchange of NIO buffers to
+ * support query execution.
+ */
+public interface IQueryPeer extends Remote {
+
+ /**
+ * Notify a service that a buffer having data for some {@link BOp} in some
+ * running query is available. The receiver may request the data when they
+ * are ready. If the query is cancelled, then the sender will drop the
+ * buffer.
+ *
+ * @param clientProxy
+ * proxy used to communicate with the client running the query.
+ * @param serviceAddr
+ * address which may be used to demand the data.
+ * @param queryId
+ * the unique query identifier.
+ * @param bopId
+ * the identifier for the target {@link BOp}.
+ *
+ * @return <code>true</code> unless the receiver knows that the query has
+ * already been cancelled.
+ */
+ void bufferReady(IQueryClient clientProxy, InetSocketAddress serviceAddr,
+ long queryId, int bopId) throws RemoteException;
+
+}
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IRunningQuery.java (from rev 3510, branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/IRunningQuery.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IRunningQuery.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/IRunningQuery.java 2010-09-06 20:45:37 UTC (rev 3511)
@@ -0,0 +1,75 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
+/*
+ * Created on Sep 5, 2010
+ */
+
+package com.bigdata.bop.engine;
+
+import com.bigdata.btree.ILocalBTreeView;
+import com.bigdata.journal.IIndexManager;
+import com.bigdata.service.IBigdataFederation;
+
+/**
+ * Interface exposing a limited set of the state of an executing query.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public interface IRunningQuery {
+
+ /**
+ * The {@link IBigdataFederation} IFF the operator is being evaluated on an
+ * {@link IBigdataFederation}. When evaluating operations against an
+ * {@link IBigdataFederation}, this reference provides access to the
+ * scale-out view of the indices and to other bigdata services.
+ */
+ IBigdataFederation<?> getFederation();
+
+ /**
+ * The <strong>local</strong> {@link IIndexManager}. Query evaluation occurs
+ * against the local indices. In scale-out, query evaluation proceeds shard
+ * wise and this {@link IIndexManager} MUST be able to read on the
+ * {@link ILocalBTreeView}.
+ */
+ IIndexManager getIndexManager();
+
+ /**
+ * The timestamp or transaction identifier against which the query is
+ * reading.
+ */
+ long getReadTimestamp();
+
+ /**
+ * The timestamp or transaction identifier against which the query is
+ * writing.
+ */
+ long getWriteTimestamp();
+
+ /**
+ * Terminate query evaluation
+ */
+ void halt();
+
+}
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/PipelineUtility.java (from rev 3489, branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/PipelineUtility.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/PipelineUtility.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/PipelineUtility.java 2010-09-06 20:45:37 UTC (rev 3511)
@@ -0,0 +1,156 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Sep 1, 2010
+ */
+
+package com.bigdata.bop.engine;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.log4j.Logger;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpUtility;
+import com.bigdata.bop.NoSuchBOpException;
+
+/**
+ * Utility methods relevant to pipelined operator evaluation.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class PipelineUtility {
+
+ private static final Logger log = Logger.getLogger(PipelineUtility.class);
+
+ /**
+ * Return <code>true</code> iff the <i>runningCountMap</i> AND
+ * <i>availableChunkMap</i> map are ZERO (0) for both the given operator and
+ * for all operators which proceed the given operator in the tree structure
+ * of its operands.
+ * <p>
+ * Note: The movement of the intermediate binding set chunks forms an
+ * acyclic directed graph. We can decide whether or not a {@link BOp} in the
+ * query plan can be triggered by the current activity pattern by inspecting
+ * the {@link BOp} and its operands recursively. If neither the {@link BOp}
+ * nor any of its operands (recursively) has non-zero activity then the
+ * {@link BOp} can not be triggered and this method will return
+ * <code>true</code>.
+ *
+ * @param bopId
+ * The identifier for an operator which appears in the query
+ * plan.
+ * @param queryPlan
+ * The query plan.
+ * @param queryIndex
+ * An index for the query plan as constructed by
+ * {@link BOpUtility#getIndex(BOp)}.
+ * @param runningCountMap
+ * A map reporting the #of instances of each operator which are
+ * currently being evaluated (distinct evaluations are performed
+ * for each chunk and shard).
+ * @param availableChunkCountMap
+ * A map reporting the #of chunks available for each operator in
+ * the pipeline (we only report chunks for pipeline operators).
+ *
+ * @return <code>true</code> iff the {@link BOp} can not be triggered given
+ * the query plan and the activity map.
+ *
+ * @throws IllegalArgumentException
+ * if any argument is <code>null</code>.
+ * @throws NoSuchBOpException
+ * if <i>bopId</i> is not found in the query index.
+ */
+ static public boolean isDone(final int bopId, final BOp queryPlan,
+ final Map<Integer, BOp> queryIndex,
+ final Map<Integer, AtomicLong> runningCountMap,
+ final Map<Integer, AtomicLong> availableChunkCountMap) {
+
+ if (queryPlan == null)
+ throw new IllegalArgumentException();
+ if (queryIndex == null)
+ throw new IllegalArgumentException();
+ if (availableChunkCountMap == null)
+ throw new IllegalArgumentException();
+
+ final BOp op = queryIndex.get(bopId);
+
+ if (op == null)
+ throw new NoSuchBOpException(bopId);
+
+ final Iterator<BOp> itr = BOpUtility.preOrderIterator(op);
+
+ while (itr.hasNext()) {
+
+ final BOp t = itr.next();
+
+ final Integer id = (Integer) t.getProperty(BOp.Annotations.BOP_ID);
+
+ if (id == null)
+ continue;
+ {
+
+ final AtomicLong runningCount = runningCountMap.get(id);
+
+ if (runningCount != null && runningCount.get() != 0) {
+
+ if (log.isInfoEnabled())
+ log.info("Operator can be triggered: op=" + op
+ + ", possible trigger=" + t + " is running.");
+
+ return false;
+
+ }
+
+ }
+
+ {
+
+ final AtomicLong availableChunkCount = availableChunkCountMap
+ .get(id);
+
+ if (availableChunkCount != null
+ && availableChunkCount.get() != 0) {
+
+ if (log.isInfoEnabled())
+ log.info("Operator can be triggered: op=" + op
+ + ", possible trigger=" + t + " has "
+ + availableChunkCount + " chunks available.");
+
+ return false;
+
+ }
+
+ }
+
+ }
+
+ return true;
+
+ }
+
+}
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java (from rev 3510, branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/QueryEngine.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java 2010-09-06 20:45:37 UTC (rev 3511)
@@ -0,0 +1,800 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Aug 21, 2010
+ */
+
+package com.bigdata.bop.engine;
+
+import java.net.InetSocketAddress;
+import java.nio.ByteBuffer;
+import java.rmi.RemoteException;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Future;
+import java.util.concurrent.FutureTask;
+import java.util.concurrent.PriorityBlockingQueue;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.apache.log4j.Logger;
+
+import alice.tuprolog.Prolog;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.BindingSetPipelineOp;
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.IPredicate;
+import com.bigdata.bop.bset.Union;
+import com.bigdata.btree.BTree;
+import com.bigdata.btree.IndexSegment;
+import com.bigdata.btree.view.FusedView;
+import com.bigdata.journal.IIndexManager;
+import com.bigdata.journal.ITx;
+import com.bigdata.journal.TimestampUtility;
+import com.bigdata.rdf.internal.IV;
+import com.bigdata.rdf.spo.SPORelation;
+import com.bigdata.relation.IMutableRelation;
+import com.bigdata.relation.IRelation;
+import com.bigdata.relation.accesspath.IBlockingBuffer;
+import com.bigdata.relation.accesspath.IElementFilter;
+import com.bigdata.relation.rule.IRule;
+import com.bigdata.relation.rule.Program;
+import com.bigdata.relation.rule.eval.pipeline.DistributedJoinTask;
+import com.bigdata.resources.IndexManager;
+import com.bigdata.service.IBigdataFederation;
+import com.bigdata.service.IDataService;
+import com.bigdata.service.ndx.IAsynchronousWriteBufferFactory;
+import com.bigdata.striterator.ChunkedArrayIterator;
+import com.bigdata.striterator.IChunkedOrderedIterator;
+
+/**
+ * A class managing execution of concurrent queries against a local
+ * {@link IIndexManager}.
+ * <p>
+ * <h2>Design notes</h2>
+ * <p>
+ * Much of the complexity of the current approach owes itself to having to run a
+ * separate task for each join for each shard in order to have the appropriate
+ * lock when running against the unisolated shard view. This also means that the
+ * join task is running inside of the concurrency manager and hence has the
+ * local view of the shard.
+ * <p>
+ * The main, and perhaps the only, reason why we run unisolated rules is during
+ * closure, when we query against the unisolated indices and then write the
+ * entailments back on the unisolated indices.
+ * <p>
+ * Supporting closure has always been complicated. This complexity is mostly
+ * handled by ProgramTask#executeMutation() and
+ * AbstractTripleStore#newJoinNexusFactory() which play games with the
+ * timestamps used to read and write on the database, with commit points
+ * designed to create visibility for tuples written by a mutation rule, and with
+ * the automated advance of the read timestamp for the query in each closure
+ * pass in order to make newly committed tuples visible to subsequent rounds of
+ * closure. For scale-out, we do shard-wise auto commits so we always have a
+ * commit point which makes each write visible and the read timestamp is
+ * actually a read-only transaction which prevents the historical data we need
+ * during a closure round from being released as we are driving updates onto the
+ * federation. For the RWStore, we are having a similar problem (in the HA
+ * branch since that is where we are working on the RWStore) where historically
+ * allocated records were being released as writes drove updates on the indices.
+ * Again, we "solved" the problem for the RWStore using a commit point followed
+ * by a read-only transaction reading on that commit point to hold onto the view
+ * on which the next closure round needs to read (this uncovered a problem with
+ * the RWStore and transaction service interaction which Martyn is currently
+ * working to resolve through a combination of shadow allocators and deferred
+ * deletes which are processed once the release time is advanced by the
+ * transaction service).
+ * <p>
+ * The WORM does not have some of these problems with closure because we never
+ * delete history, so we do not need to create a commit point and a read-behind
+ * transaction. However, the WORM would have problems with concurrent access to
+ * the unisolated indices except that we hack that problem through the
+ * transparent use of the UnisolatedReadWriteIndex, which allows multiple
+ * threads to access the same unisolated index view using a read/write lock
+ * pattern (concurrent readers are allowed, but there is only one writer and it
+ * has exclusive access when it is running). This works out because we never run
+ * closure operations against the WORM through the concurrency manager. If we
+ * did, we would have to create a commit point after each mutation and use a
+ * read-behind transaction to prevent concurrent access to the unisolated index.
+ * <p>
+ * The main advantage that I can see of the current complexity is that it allows
+ * us to do load+closure as a single operation on the WORM, resulting in a
+ * single commit point. This makes that operation ACID without having to use
+ * full read/write transactions. This is how we gain the ACID contract for the
+ * standalone Journal in the SAIL for the WORM. Of course, the SAIL does not
+ * have that contract for the RWStore because we have to do the commit and
+ * read-behind transaction in order to have visibility and avoid concurrent
+ * access to the unisolated index (by reading behind on the last commit point).
+ * <p>
+ * I think that the reality is even one step more complicated. When doing truth
+ * maintenance (incremental closure), we bring the temporary graph to a fixed
+ * point (the rules write on the temp store) and then apply the delta in a
+ * single write to the database. That suggests that incremental truth
+ * maintenance would continue to be ACID, but that database-at-once-closure
+ * would be round-wise ACID.
+ * <p>
+ * So, I would like to suggest that we break ACID for database-at-once-closure
+ * and always follow the pattern of (1) do a commit before each round of
+ * closure; and (2) create a read-behind transaction to prevent the release of
+ * that commit point as we drive writes onto the indices. If we follow this
+ * pattern then we can write on the unisolated indices without conflict and read
+ * on the historical views without conflict. Since there will be a commit point
+ * before each mutation rule runs (which corresponds to a closure round),
+ * database-at-once-closure will be atomic within a round, but will not be a
+ * single atomic operation. Per above, I think that we would retain the ACID
+ * property for incremental truth maintenance against a WORM or RW mode Journal.
+ *
+ * <p>
+ * ----
+ * </p>
+ *
+ * The advantage of this proposal (commit before each mutation rule and run
+ * query against a read-behind transaction) is that this could enormously
+ * simplify how we execute joins.
+ * <p>
+ * Right now, we use a factory pattern to create a join task on each node for
+ * each shard for which that node receives binding sets for a query. The main
+ * reason for doing this is to gain the appropriate lock for the unisolated
+ * index. If we never run a query against the unisolated index then we can go
+ * around the concurrency manager and run a single "query manager" task for all
+ * joins for all shards for all queries. This has some great benefits which I
+ * will go into below.
+ * <p>
+ * That "query manager" task would be responsible for accepting buffers
+ * containing elements or binding sets from other nodes and scheduling
+ * consumption of those data based on various criteria (order of arrival,
+ * priority, buffer resource requirements, timeout, etc.). This manager task
+ * could use a fork join pool to execute light weight operations (NIO,
+ * formulation of access paths from binding sets, mapping of binding sets onto
+ * shards, joining a chunk already read from an access path against a binding
+ * set, etc). Operations which touch the disk need to run in their own thread
+ * (until we get Java 7 async file IO, which is already available in a preview
+ * library). We could handle that by queuing those operations against a fixed
+ * size thread pool for reads.
+ * <p>
+ * This is a radical change in how we handle distributed query execution, but I
+ * think that it could have a huge payoff by reducing the complexity of the join
+ * logic, making it significantly easier to execute different kinds of join
+ * operations, reducing the overhead for acquiring locks for the unisolated
+ * index views, reducing the #of threads consumed by joins (from one per shard
+ * per join per query to a fixed pool of N threads for reads), etc. It would
+ * centralize the management of resources on each node and make it possible for
+ * us to handle things like join termination by simply purging data from the
+ * query manager task for the terminated join.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ *
+ *
+ * FIXME Unit tests for non-distinct {@link IElementFilter}s on an
+ * {@link IPredicate}, unit tests for distinct element filter on an
+ * {@link IPredicate} which is capable of distributed operations. Do not use
+ * distinct where not required (SPOC, only one graph, etc).
+ * <p>
+ * It seems like the right way to approach this is by unifying the stackable CTC
+ * striterator pattern with the chunked iterator pattern and passing the query
+ * engine (or the bop context) into the iterator construction process (or simply
+ * requesting that the query engine construct the iterator stack).
+ * <p>
+ * In terms of harmonization, it is difficult to say which way would work
+ * better. In the short term we could simply allow both and mask the differences
+ * in how we construct the filters, but the conversion to/from striterators and
+ * chunked iterators seems to waste a bit of effort.
+ * <p>
+ * The trickiest part of all of this is to allow a distributed filter pattern
+ * where the filter gets created on a set of nodes identified by the operator
+ * and the elements move among those nodes using the query engine's buffers.
+ * <p>
+ * To actually implement the distributed distinct filter we need to stack the
+ * following:
+ *
+ * <pre>
+ * - ITupleIterator
+ * - Resolve ITuple to Element (e.g., SPOC).
+ * - Layer on optional IElementFilter associated with the IPredicate.
+ * - Layer on SameVariableConstraint iff required (done by AccessPath)
+ * - Resolve SPO to SPO, stripping off the context position.
+ * - Chunk SPOs (SPO[], IKeyOrder), where the key order is from the access path.
+ * - Filter SPO[] using DHT constructed on specified nodes of the cluster.
+ * The SPO[] chunks should be packaged into NIO buffers and shipped to those
+ * nodes. The results should be shipped back as a bit vectors packaged into
+ * a NIO buffers.
+ * - Dechunk SPO[] to SPO since that is the current expectation for the filter
+ * stack.
+ * - The result then gets wrapped as a {@link IChunkedOrderedIterator} by
+ * the AccessPath using a {@link ChunkedArrayIterator}.
+ * </pre>
+ *
+ * This stack is a bit complex(!). But it is certainly easy enough to generate
+ * the necessary bits programmatically.
+ *
+ * FIXME Handling the {@link Union} of binding sets. Consider whether the chunk
+ * combiner logic from the {@link DistributedJoinTask} could be reused.
+ *
+ * FIXME INSERT and DELETE which will construct elements using
+ * {@link IRelation#newElement(java.util.List, IBindingSet)} from a binding set
+ * and then use {@link IMutableRelation#insert(IChunkedOrderedIterator)} and
+ * {@link IMutableRelation#delete(IChunkedOrderedIterator)}. For s/o, we first
+ * need to move the bits into the right places so it makes sense to unpack the
+ * processing of the loop over the elements and move the data around, writing on
+ * each index as necessary. There could be eventually consistent approaches to
+ * this as well. For justifications we need to update some additional indices,
+ * in which case we are stuck going through {@link IRelation} rather than
+ * routing data directly or using the {@link IAsynchronousWriteBufferFactory}.
+ * For example, we could handle routing and writing in s/o as follows:
+ *
+ * <pre>
+ * INSERT(relation,bindingSets)
+ *
+ * expands to
+ *
+ * SEQUENCE(
+ * SELECT(s,p,o), // drop bindings that we do not need
+ * PARALLEL(
+ * INSERT_INDEX(spo), // construct (s,p,o) elements and insert
+ * INSERT_INDEX(pos), // construct (p,o,s) elements and insert
+ * INSERT_INDEX(osp), // construct (o,s,p) elements and insert
+ * ))
+ *
+ * </pre>
+ *
+ * The output of the SELECT operator would be automatically mapped against the
+ * shards on which the next operators need to write. Since there is a nested
+ * PARALLEL operator, the mapping will be against the shards of each of the
+ * given indices. (A simpler operator would invoke
+ * {@link SPORelation#insert(IChunkedOrderedIterator)}. Handling justifications
+ * requires that we also formulate the justification chain from the pattern of
+ * variable bindings in the rule).
+ *
+ * FIXME Handle {@link Program}s. There are three flavors, which should probably
+ * be broken into three operators: sequence(ops), set(ops), and closure(op). The
+ * 'set' version would be parallelized, or at least have an annotation for
+ * parallel evaluation. These things belong in the same broad category as the
+ * join graph since they are operators which control the evaluation of other
+ * operators (the current pipeline join also has that characteristic which it
+ * uses to do the nested index subqueries).
+ *
+ * FIXME SPARQL to BOP translation
+ * <p>
+ * The initial pass should translate from {@link IRule} to {@link BOp}s so we
+ * can immediately begin running SPARQL queries against the {@link QueryEngine}.
+ * A second pass should explore a rules base translation from the openrdf SPARQL
+ * operator tree into {@link BOp}s, perhaps using an embedded {@link Prolog}
+ * engine. What follows is a partial list of special considerations for that
+ * translation:
+ * <ul>
+ * <li>Distinct can be trivially enforced for default graph queries against the
+ * SPOC index.</li>
+ * <li>Local distinct should wait until there is more than one tuple from the
+ * index since a single tuple does not need to be made distinct using a hash
+ * map.</li>
+ * <li>Low volume distributed queries should use solution modifiers which
+ * evaluate on the query controller node rather than using distributed sort,
+ * distinct, slice, or aggregation operators.</li>
+ * <li></li>
+ * <li></li>
+ * <li></li>
+ * <li>High volume queries should use special operators (different
+ * implementations of joins, use an external merge sort, etc).</li>
+ * </ul>
+ *
+ * FIXME SPARQL Coverage: Add native support for all SPARQL operators. A lot of
+ * this can be picked up from Sesame. Some things, such as isIRI() can be done
+ * natively against the {@link IV}. Likewise, there is already a set of
+ * comparison methods for {@link IV}s which are inlined values. Add support for
+ * <ul>
+ * <li></li>
+ * <li></li>
+ * <li></li>
+ * <li></li>
+ * <li></li>
+ * <li></li>
+ * </ul>
+ *
+ * @todo Expander patterns will continue to exist until we handle the standalone
+ * backchainers in a different manner for scale-out so add support for
+ * those for now.
+ *
+ * @todo There is a going to be a relationship to recycling of intermediates
+ * (for individual {@link BOp}s or {@link BOp} tree fragments) and a
+ * distributed query cache which handles invalidation (for updates) and
+ * {@link BOp} aware reuse of result sets available in the cache. This
+ * sort of thing will have to be coordinated among the cache nodes.
+ */
+public class QueryEngine implements IQueryPeer, IQueryClient {
+
+ pr...
[truncated message content] |
|
From: <tho...@us...> - 2010-09-05 20:17:51
|
Revision: 3510
http://bigdata.svn.sourceforge.net/bigdata/?rev=3510&view=rev
Author: thompsonbry
Date: 2010-09-05 20:17:44 +0000 (Sun, 05 Sep 2010)
Log Message:
-----------
Introduced an interface for running queries and refactored the BOpContext to use that interface. This makes it possible to test more things with mock objects and also provides a hook from the BOpContext back to the RunningQuery. That hook is necessary when we need to halt a running query (for an offset/limit slice).
Added Rule2BOpUtility which will handle the initial conversion from a Rule or Program into a bop tree.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/bset/TestConditionalRoutingOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/QueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/RunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/join/TestPipelineJoin.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/solutions/TestDistinctBindingSets.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/solutions/TestSliceOp.java
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/IRunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/MockRunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/Rule2BOpUtility.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java 2010-09-05 18:16:01 UTC (rev 3509)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java 2010-09-05 20:17:44 UTC (rev 3510)
@@ -27,20 +27,18 @@
*/
package com.bigdata.bop;
-import java.util.Iterator;
-
import org.apache.log4j.Logger;
import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.bop.engine.IRunningQuery;
import com.bigdata.bop.engine.QueryEngine;
+import com.bigdata.bop.engine.RunningQuery;
import com.bigdata.bop.solutions.SliceOp;
import com.bigdata.btree.IIndex;
import com.bigdata.btree.ILocalBTreeView;
import com.bigdata.btree.IRangeQuery;
import com.bigdata.journal.IIndexManager;
-import com.bigdata.journal.ITx;
import com.bigdata.journal.TimestampUtility;
-import com.bigdata.mdi.PartitionLocator;
import com.bigdata.relation.IRelation;
import com.bigdata.relation.accesspath.AccessPath;
import com.bigdata.relation.accesspath.IAccessPath;
@@ -49,11 +47,10 @@
import com.bigdata.relation.locator.IResourceLocator;
import com.bigdata.relation.rule.IRule;
import com.bigdata.relation.rule.eval.IJoinNexus;
-import com.bigdata.service.AbstractScaleOutFederation;
import com.bigdata.service.DataService;
import com.bigdata.service.IBigdataFederation;
-import com.bigdata.service.ndx.IClientIndex;
import com.bigdata.striterator.IKeyOrder;
+import com.ibm.icu.impl.ByteBuffer;
/**
* The evaluation context for the operator (NOT serializable).
@@ -65,14 +62,16 @@
static private final Logger log = Logger.getLogger(BOpContext.class);
- private final IBigdataFederation<?> fed;
+ private final IRunningQuery runningQuery;
+
+// private final IBigdataFederation<?> fed;
+//
+// private final IIndexManager indexManager;
+//
+// private final long readTimestamp;
+//
+// private final long writeTimestamp;
- private final IIndexManager indexManager;
-
- private final long readTimestamp;
-
- private final long writeTimestamp;
-
private final int partitionId;
private final BOpStats stats;
@@ -84,13 +83,25 @@
private final IBlockingBuffer<E[]> sink2;
/**
+ * The interface for a running query.
+ * <p>
+ * Note: In scale-out each node will have a distinct {@link IRunningQuery}
+ * object and the query controller will have access to additional state,
+ * such as the aggregation of the {@link BOpStats} for the query on all
+ * nodes.
+ */
+ public IRunningQuery getRunningQuery() {
+ return runningQuery;
+ }
+
+ /**
* The {@link IBigdataFederation} IFF the operator is being evaluated on an
* {@link IBigdataFederation}. When evaluating operations against an
* {@link IBigdataFederation}, this reference provides access to the
* scale-out view of the indices and to other bigdata services.
*/
public IBigdataFederation<?> getFederation() {
- return fed;
+ return runningQuery.getFederation();
}
/**
@@ -100,7 +111,7 @@
* {@link ILocalBTreeView}.
*/
public final IIndexManager getIndexManager() {
- return indexManager;
+ return runningQuery.getIndexManager();
}
/**
@@ -108,7 +119,7 @@
* reading.
*/
public final long getReadTimestamp() {
- return readTimestamp;
+ return runningQuery.getReadTimestamp();
}
/**
@@ -116,7 +127,7 @@
* writing.
*/
public final long getWriteTimestamp() {
- return writeTimestamp;
+ return runningQuery.getWriteTimestamp();
}
/**
@@ -137,6 +148,22 @@
/**
* Where to read the data to be consumed by the operator.
+ *
+ * @todo Since joins now run from locally materialized data in all cases the
+ * API could be simplified somewhat given that we know that there will
+ * be a single "source" chunk of binding sets. Also, the reason for
+ * the {@link IAsynchronousIterator} here is that a downstream join
+ * could error (or satisfy a slice) and halt the upstream joins. That
+ * is being coordinated through the {@link RunningQuery} now.
+ * <p>
+ * It is not yet clear what the right API is for the source. The
+ * iterator model might be just fine, but might not need to be
+ * asynchronous and does not need to be closeable.
+ * <p>
+ * Perhaps the right thing is to expose an object with a richer API
+ * for obtaining various kinds of iterators or even access to the
+ * direct {@link ByteBuffer}s backing the data (for high volume joins,
+ * exernal merge sorts, etc).
*/
public final IAsynchronousIterator<E[]> getSource() {
return source;
@@ -194,18 +221,6 @@
* failed joins outside of the join group.
*
* @throws IllegalArgumentException
- * if the <i>indexManager</i> is <code>null</code>
- * @throws IllegalArgumentException
- * if the <i>indexManager</i> is is not a <em>local</em> index
- * manager.
- * @throws IllegalArgumentException
- * if the <i>readTimestamp</i> is {@link ITx#UNISOLATED}
- * (queries may not read on the unisolated indices).
- * @throws IllegalArgumentException
- * if the <i>writeTimestamp</i> is neither
- * {@link ITx#UNISOLATED} nor a read-write transaction
- * identifier.
- * @throws IllegalArgumentException
* if the <i>stats</i> is <code>null</code>
* @throws IllegalArgumentException
* if the <i>source</i> is <code>null</code> (use an empty
@@ -213,37 +228,47 @@
* @throws IllegalArgumentException
* if the <i>sink</i> is <code>null</code>
*/
- public BOpContext(final IBigdataFederation<?> fed,
- final IIndexManager indexManager, final long readTimestamp,
- final long writeTimestamp, final int partitionId,
+// * @throws IllegalArgumentException
+// * if the <i>indexManager</i> is <code>null</code>
+// * @throws IllegalArgumentException
+// * if the <i>indexManager</i> is is not a <em>local</em> index
+// * manager.
+// * @throws IllegalArgumentException
+// * if the <i>readTimestamp</i> is {@link ITx#UNISOLATED}
+// * (queries may not read on the unisolated indices).
+// * @throws IllegalArgumentException
+// * if the <i>writeTimestamp</i> is neither
+// * {@link ITx#UNISOLATED} nor a read-write transaction
+// * identifier.
+ public BOpContext(final IRunningQuery runningQuery,final int partitionId,
final BOpStats stats, final IAsynchronousIterator<E[]> source,
final IBlockingBuffer<E[]> sink, final IBlockingBuffer<E[]> sink2) {
- if (indexManager == null)
- throw new IllegalArgumentException();
- if (indexManager instanceof IBigdataFederation<?>) {
- /*
- * This is disallowed because the predicate specifies an index
- * partition and expects to have access to the local index objects
- * for that index partition.
- */
- throw new IllegalArgumentException(
- "Expecting a local index manager, not: "
- + indexManager.getClass().toString());
- }
- if (readTimestamp == ITx.UNISOLATED)
- throw new IllegalArgumentException();
- if (TimestampUtility.isReadOnly(writeTimestamp))
- throw new IllegalArgumentException();
+ this.runningQuery = runningQuery;
+// if (indexManager == null)
+// throw new IllegalArgumentException();
+// if (indexManager instanceof IBigdataFederation<?>) {
+// /*
+// * This is disallowed because predicates always read on local index
+// * objects, even in scale-out.
+// */
+// throw new IllegalArgumentException(
+// "Expecting a local index manager, not: "
+// + indexManager.getClass().toString());
+// }
+// if (readTimestamp == ITx.UNISOLATED)
+// throw new IllegalArgumentException();
+// if (TimestampUtility.isReadOnly(writeTimestamp))
+// throw new IllegalArgumentException();
if (stats == null)
throw new IllegalArgumentException();
if (source == null)
throw new IllegalArgumentException();
if (sink == null)
throw new IllegalArgumentException();
- this.fed = fed; // may be null
- this.indexManager = indexManager;
- this.readTimestamp = readTimestamp;
- this.writeTimestamp = writeTimestamp;
+// this.fed = fed; // may be null
+// this.indexManager = indexManager;
+// this.readTimestamp = readTimestamp;
+// this.writeTimestamp = writeTimestamp;
this.partitionId = partitionId;
this.stats = stats;
this.source = source;
@@ -344,24 +369,28 @@
if (partitionId == -1) {
- if(indexManager instanceof IBigdataFederation<?>)
+ if (getFederation() != null) {
+ // This is scale-out so the partition identifier is required.
throw new UnsupportedOperationException();
+ }
// The index is not partitioned.
- ndx = (ILocalBTreeView) indexManager.getIndex(namespace + "."
+ ndx = (ILocalBTreeView) getIndexManager().getIndex(namespace + "."
+ keyOrder.getIndexName(), getWriteTimestamp());
} else {
- if(!(indexManager instanceof IBigdataFederation<?>))
+ if (getFederation() == null) {
+ // This is not scale-out so index partitions are not supported.
throw new UnsupportedOperationException();
+ }
// The name of the desired index partition.
final String name = DataService.getIndexPartitionName(namespace
+ "." + keyOrder.getIndexName(), partitionId);
// MUST be a local index view.
- ndx = (ILocalBTreeView) indexManager.getIndex(name,
+ ndx = (ILocalBTreeView) getIndexManager().getIndex(name,
getWriteTimestamp());
}
@@ -427,6 +456,10 @@
PipelineOp.Annotations.FULLY_BUFFERED_READ_THRESHOLD,
PipelineOp.Annotations.DEFAULT_FULLY_BUFFERED_READ_THRESHOLD);
+ final IIndexManager indexManager = getIndexManager();
+
+ final long readTimestamp = getReadTimestamp();
+
if (predicate.getPartitionId() != -1) {
/*
@@ -669,6 +702,8 @@
* with that thrown cause.
*/
public void halt() {
+
+ runningQuery.halt();
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java 2010-09-05 18:16:01 UTC (rev 3509)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java 2010-09-05 20:17:44 UTC (rev 3510)
@@ -101,9 +101,6 @@
*/
private static final long serialVersionUID = 1L;
- /**
- * @todo Declare SLICE annotation and support SLICE in the {@link JoinTask}.
- */
public interface Annotations extends BindingSetPipelineOp.Annotations {
/**
@@ -478,11 +475,6 @@
this.optional = joinOp.isOptional();
this.variablesToKeep = joinOp.variablesToKeep();
this.context = context;
- /*
- * FIXME Carefully review which index manager (local versus fed) is
- * being used to resolve the relation. Also note that we used to
- * cache the resourceLocator.
- */
this.relation = context.getReadRelation(right);
this.source = context.getSource();
this.sink = context.getSink();
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/bset/TestConditionalRoutingOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/bset/TestConditionalRoutingOp.java 2010-09-05 18:16:01 UTC (rev 3509)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/bset/TestConditionalRoutingOp.java 2010-09-05 20:17:44 UTC (rev 3510)
@@ -29,7 +29,6 @@
import java.util.LinkedList;
import java.util.List;
-import java.util.Properties;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.FutureTask;
@@ -47,11 +46,10 @@
import com.bigdata.bop.Var;
import com.bigdata.bop.constraint.EQConstant;
import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.bop.engine.MockRunningQuery;
import com.bigdata.bop.engine.TestQueryEngine;
import com.bigdata.bop.solutions.DistinctBindingSetOp;
-import com.bigdata.journal.BufferMode;
import com.bigdata.journal.ITx;
-import com.bigdata.journal.Journal;
import com.bigdata.relation.accesspath.IAsynchronousIterator;
import com.bigdata.relation.accesspath.IBlockingBuffer;
import com.bigdata.relation.accesspath.ThickAsynchronousIterator;
@@ -79,25 +77,25 @@
super(name);
}
- @Override
- public Properties getProperties() {
+// @Override
+// public Properties getProperties() {
+//
+// final Properties p = new Properties(super.getProperties());
+//
+// p.setProperty(Journal.Options.BUFFER_MODE, BufferMode.Transient
+// .toString());
+//
+// return p;
+//
+// }
- final Properties p = new Properties(super.getProperties());
+// Journal jnl = null;
- p.setProperty(Journal.Options.BUFFER_MODE, BufferMode.Transient
- .toString());
-
- return p;
-
- }
-
- Journal jnl = null;
-
List<IBindingSet> data = null;
public void setUp() throws Exception {
- jnl = new Journal(getProperties());
+// jnl = new Journal(getProperties());
setUpData();
@@ -147,10 +145,10 @@
public void tearDown() throws Exception {
- if (jnl != null) {
- jnl.destroy();
- jnl = null;
- }
+// if (jnl != null) {
+// jnl.destroy();
+// jnl = null;
+// }
// clear reference.
data = null;
@@ -212,16 +210,18 @@
final IBlockingBuffer<IBindingSet[]> sink2 = query.newBuffer();
final BOpContext<IBindingSet> context = new BOpContext<IBindingSet>(
- null/* fed */, jnl/* indexManager */,
+ new MockRunningQuery(
+ null/* fed */, null/* indexManager */,
ITx.READ_COMMITTED/* readTimestamp */,
- ITx.UNISOLATED/* writeTimestamp */, -1/* partitionId */, stats,
+ ITx.UNISOLATED/* writeTimestamp */), -1/* partitionId */, stats,
source, sink, sink2);
// get task.
final FutureTask<Void> ft = query.eval(context);
// execute task.
- jnl.getExecutorService().execute(ft);
+// jnl.getExecutorService().execute(ft);
+ ft.run();
TestQueryEngine.assertSameSolutions(expected, sink.iterator());
TestQueryEngine.assertSameSolutions(expected2, sink2.iterator());
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/IRunningQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/IRunningQuery.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/IRunningQuery.java 2010-09-05 20:17:44 UTC (rev 3510)
@@ -0,0 +1,75 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
+/*
+ * Created on Sep 5, 2010
+ */
+
+package com.bigdata.bop.engine;
+
+import com.bigdata.btree.ILocalBTreeView;
+import com.bigdata.journal.IIndexManager;
+import com.bigdata.service.IBigdataFederation;
+
+/**
+ * Interface exposing a limited set of the state of an executing query.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public interface IRunningQuery {
+
+ /**
+ * The {@link IBigdataFederation} IFF the operator is being evaluated on an
+ * {@link IBigdataFederation}. When evaluating operations against an
+ * {@link IBigdataFederation}, this reference provides access to the
+ * scale-out view of the indices and to other bigdata services.
+ */
+ IBigdataFederation<?> getFederation();
+
+ /**
+ * The <strong>local</strong> {@link IIndexManager}. Query evaluation occurs
+ * against the local indices. In scale-out, query evaluation proceeds shard
+ * wise and this {@link IIndexManager} MUST be able to read on the
+ * {@link ILocalBTreeView}.
+ */
+ IIndexManager getIndexManager();
+
+ /**
+ * The timestamp or transaction identifier against which the query is
+ * reading.
+ */
+ long getReadTimestamp();
+
+ /**
+ * The timestamp or transaction identifier against which the query is
+ * writing.
+ */
+ long getWriteTimestamp();
+
+ /**
+ * Terminate query evaluation
+ */
+ void halt();
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/IRunningQuery.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/MockRunningQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/MockRunningQuery.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/MockRunningQuery.java 2010-09-05 20:17:44 UTC (rev 3510)
@@ -0,0 +1,96 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Sep 5, 2010
+ */
+
+package com.bigdata.bop.engine;
+
+import org.apache.log4j.Logger;
+
+import com.bigdata.journal.IIndexManager;
+import com.bigdata.service.IBigdataFederation;
+
+/**
+ * Mock object.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class MockRunningQuery implements IRunningQuery {
+
+ private static final Logger log = Logger.getLogger(MockRunningQuery.class);
+
+ private final IBigdataFederation<?> fed;
+
+ private final IIndexManager indexManager;
+
+ private final long readTimestamp;
+
+ private final long writeTimestamp;
+
+ /**
+ * Note: This constructor DOES NOT check its arguments so unit tests may be
+ * written with the minimum dependencies
+ *
+ * @param fed
+ * @param indexManager
+ * @param readTimestamp
+ * @param writeTimestamp
+ */
+ public MockRunningQuery(final IBigdataFederation<?> fed,
+ final IIndexManager indexManager, final long readTimestamp,
+ final long writeTimestamp) {
+
+ this.fed = fed;
+ this.indexManager = indexManager;
+ this.readTimestamp = readTimestamp;
+ this.writeTimestamp = writeTimestamp;
+
+ }
+
+ public IBigdataFederation<?> getFederation() {
+ return fed;
+ }
+
+ public IIndexManager getIndexManager() {
+ return indexManager;
+ }
+
+ public long getReadTimestamp() {
+ return readTimestamp;
+ }
+
+ public long getWriteTimestamp() {
+ return writeTimestamp;
+ }
+
+ /**
+ * NOP (you have to test things like slices with a full integration).
+ */
+ public void halt() {
+ log.warn("Mock object does not implement halt()");
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/MockRunningQuery.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/QueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/QueryEngine.java 2010-09-05 18:16:01 UTC (rev 3509)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/QueryEngine.java 2010-09-05 20:17:44 UTC (rev 3510)
@@ -41,10 +41,12 @@
import org.apache.log4j.Logger;
+import com.bigdata.bop.BOp;
import com.bigdata.bop.BindingSetPipelineOp;
-import com.bigdata.bop.BOp;
import com.bigdata.bop.IBindingSet;
import com.bigdata.journal.IIndexManager;
+import com.bigdata.journal.ITx;
+import com.bigdata.journal.TimestampUtility;
import com.bigdata.relation.accesspath.IBlockingBuffer;
import com.bigdata.service.IBigdataFederation;
@@ -580,6 +582,13 @@
* evaluating the query.
*
* @throws Exception
+ * @throws IllegalArgumentException
+ * if the <i>readTimestamp</i> is {@link ITx#UNISOLATED}
+ * (queries may not read on the unisolated indices).
+ * @throws IllegalArgumentException
+ * if the <i>writeTimestamp</i> is neither
+ * {@link ITx#UNISOLATED} nor a read-write transaction
+ * identifier.
*
* @todo Consider elevating the read/write timestamps into the query plan as
* annotations. Closure would then rewrite the query plan for each
@@ -597,6 +606,10 @@
if (query == null)
throw new IllegalArgumentException();
+ if (readTimestamp == ITx.UNISOLATED)
+ throw new IllegalArgumentException();
+ if (TimestampUtility.isReadOnly(writeTimestamp))
+ throw new IllegalArgumentException();
final long timeout = query.getProperty(BOp.Annotations.TIMEOUT,
BOp.Annotations.DEFAULT_TIMEOUT);
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/Rule2BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/Rule2BOpUtility.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/Rule2BOpUtility.java 2010-09-05 20:17:44 UTC (rev 3510)
@@ -0,0 +1,93 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Sep 5, 2010
+ */
+
+package com.bigdata.bop.engine;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.rdf.sail.BigdataSail;
+import com.bigdata.relation.rule.IProgram;
+import com.bigdata.relation.rule.IRule;
+import com.bigdata.relation.rule.IStep;
+import com.bigdata.relation.rule.Program;
+import com.bigdata.relation.rule.Rule;
+
+/**
+ * Utility class converts {@link IRule}s to {@link BOp}s.
+ * <p>
+ * Note: This is a stopgap measure designed to allow us to evaluate SPARQL
+ * queries and verify the standalone {@link QueryEngine} while we develop a
+ * direct translation from Sesame's SPARQL operator tree onto {@link BOp}s and
+ * work on the scale-out query buffer transfer mechanisms.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class Rule2BOpUtility {
+
+ /**
+ * Convert an {@link IStep} into an operator tree. This should handle
+ * {@link IRule}s and {@link IProgram}s as they are currently implemented
+ * and used by the {@link BigdataSail}.
+ *
+ * @param step
+ * The step.
+ *
+ * @return
+ */
+ public static BOp convert(final IStep step) {
+
+ throw new UnsupportedOperationException();
+
+ }
+
+ /**
+ * Convert a rule into an operator tree.
+ *
+ * @param rule
+ *
+ * @return
+ */
+ public static BOp convert(final Rule rule) {
+
+ throw new UnsupportedOperationException();
+
+ }
+
+ /**
+ * Convert a program into an operator tree.
+ *
+ * @param program
+ *
+ * @return
+ */
+ public static BOp convert(final Program program) {
+
+ throw new UnsupportedOperationException();
+
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/Rule2BOpUtility.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/RunningQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/RunningQuery.java 2010-09-05 18:16:01 UTC (rev 3509)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/RunningQuery.java 2010-09-05 20:17:44 UTC (rev 3510)
@@ -59,6 +59,8 @@
import com.bigdata.bop.NoSuchBOpException;
import com.bigdata.bop.ap.Predicate;
import com.bigdata.bop.bset.Union;
+import com.bigdata.bop.solutions.SliceOp;
+import com.bigdata.journal.IIndexManager;
import com.bigdata.rdf.internal.IV;
import com.bigdata.rdf.spo.SPORelation;
import com.bigdata.relation.IMutableRelation;
@@ -69,8 +71,9 @@
import com.bigdata.relation.accesspath.IElementFilter;
import com.bigdata.relation.rule.IRule;
import com.bigdata.relation.rule.Program;
-import com.bigdata.relation.rule.eval.pipeline.DistributedJoinMasterTask;
+import com.bigdata.relation.rule.eval.pipeline.DistributedJoinTask;
import com.bigdata.resources.ResourceManager;
+import com.bigdata.service.IBigdataFederation;
import com.bigdata.service.ndx.IAsynchronousWriteBufferFactory;
import com.bigdata.striterator.ChunkedArrayIterator;
import com.bigdata.striterator.IChunkedOrderedIterator;
@@ -82,7 +85,7 @@
*
* @todo HA aspects of running queries? Checkpoints for long running queries?
*/
-public class RunningQuery implements Future<Map<Integer,BOpStats>> {
+public class RunningQuery implements Future<Map<Integer,BOpStats>>, IRunningQuery {
private final static transient Logger log = Logger
.getLogger(RunningQuery.class);
@@ -887,10 +890,8 @@
final IBlockingBuffer<IBindingSet[]> altSink = altSinkId == null ? null
: op.newBuffer();
// context
- final BOpContext context = new BOpContext(queryEngine.getFederation(),
- queryEngine.getLocalIndexManager(), readTimestamp,
- writeTimestamp, chunk.partitionId, op.newStats(), chunk.source,
- sink, altSink);
+ final BOpContext context = new BOpContext(this, chunk.partitionId, op
+ .newStats(), chunk.source, sink, altSink);
// FutureTask for operator execution (not running yet).
final FutureTask<Void> f = op.eval(context);
// Hook the FutureTask.
@@ -976,9 +977,21 @@
* various methods in order to clean up the state of a completed query.
*/
+ public void halt() {
+
+ cancel(true/* mayInterruptIfRunning */);
+
+ }
+
/**
- * @todo Cancelled queries must reject or drop new chunks, etc. Queries must
- * release all of their resources when they are done().
+ * @todo Cancelled queries must reject or drop new chunks, etc.
+ * <p>
+ * Queries must release all of their resources when they are done().
+ * <p>
+ * Queries MUST NOT cause the solutions to be discarded before the
+ * client can consume them. This means that we have to carefully
+ * integrate/product {@link SliceOp} or just wrap the query buffer to
+ * impose the slice (simpler).
*/
final public boolean cancel(final boolean mayInterruptIfRunning) {
// halt the query.
@@ -1028,4 +1041,20 @@
}
+ public IBigdataFederation<?> getFederation() {
+ return queryEngine.getFederation();
+ }
+
+ public IIndexManager getIndexManager() {
+ return queryEngine.getLocalIndexManager();
+ }
+
+ public long getReadTimestamp() {
+ return readTimestamp;
+ }
+
+ public long getWriteTimestamp() {
+ return writeTimestamp;
+ }
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java 2010-09-05 18:16:01 UTC (rev 3509)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java 2010-09-05 20:17:44 UTC (rev 3510)
@@ -33,10 +33,10 @@
import junit.framework.TestCase2;
-import com.bigdata.bop.BindingSetPipelineOp;
import com.bigdata.bop.ArrayBindingSet;
import com.bigdata.bop.BOp;
import com.bigdata.bop.BOpContext;
+import com.bigdata.bop.BindingSetPipelineOp;
import com.bigdata.bop.Constant;
import com.bigdata.bop.HashBindingSet;
import com.bigdata.bop.IBindingSet;
@@ -59,6 +59,7 @@
import com.bigdata.service.jini.JiniFederation;
import com.bigdata.striterator.ChunkedArrayIterator;
import com.bigdata.striterator.ICloseableIterator;
+import com.ibm.icu.impl.ByteBuffer;
/**
* Test suite for the {@link QueryEngine} against a local database instance.
@@ -389,6 +390,12 @@
* the {@link PipelineDelayOp} can be used to impose sufficient
* latency on the pipeline that the test can close the query buffer
* iterator first].
+ * <p>
+ * This must also be tested in scale-out to make sure that the data
+ * backing the solutions is not discarded before the caller can use
+ * those data. [This could be handled by materializing binding set
+ * objects out of a {@link ByteBuffer} rather than using a live decode
+ * of the data in that {@link ByteBuffer}.]
*/
public void test_query_closeIterator() {
@@ -397,6 +404,26 @@
}
/**
+ * @todo Test ability to impose a limit/offset slice on a query.
+ * <p>
+ * Note: While the logic for visiting only the solutions selected by
+ * the slice can be tested against a mock object, the integration by
+ * which a slice halts a query when it is satisfied has to be tested
+ * against a {@link QueryEngine}.
+ * <p>
+ * This must also be tested in scale-out to make sure that the data
+ * backing the solutions is not discarded before the caller can use
+ * those data. [This could be handled by materializing binding set
+ * objects out of a {@link ByteBuffer} rather than using a live decode
+ * of the data in that {@link ByteBuffer}.]
+ */
+ public void test_query_slice() {
+
+ fail("write test");
+
+ }
+
+ /**
* @todo Test the ability run a query reading on an access path using a
* element filter (other than DISTINCT).
*/
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/join/TestPipelineJoin.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/join/TestPipelineJoin.java 2010-09-05 18:16:01 UTC (rev 3509)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/join/TestPipelineJoin.java 2010-09-05 20:17:44 UTC (rev 3510)
@@ -51,6 +51,7 @@
import com.bigdata.bop.ap.R;
import com.bigdata.bop.bset.CopyBindingSetOp;
import com.bigdata.bop.constraint.INBinarySearch;
+import com.bigdata.bop.engine.MockRunningQuery;
import com.bigdata.bop.engine.TestQueryEngine;
import com.bigdata.bop.join.PipelineJoin.PipelineJoinStats;
import com.bigdata.journal.BufferMode;
@@ -215,9 +216,9 @@
final IBlockingBuffer<IBindingSet[]> sink = query.newBuffer();
final BOpContext<IBindingSet> context = new BOpContext<IBindingSet>(
- null/* fed */, jnl/* indexManager */,
+ new MockRunningQuery(null/* fed */, jnl/* indexManager */,
ITx.READ_COMMITTED/* readTimestamp */,
- ITx.UNISOLATED/* writeTimestamp */, -1/* partitionId */, stats,
+ ITx.UNISOLATED/* writeTimestamp */), -1/* partitionId */, stats,
source, sink, null/* sink2 */);
// get task.
@@ -312,9 +313,9 @@
final IBlockingBuffer<IBindingSet[]> sink = query.newBuffer();
final BOpContext<IBindingSet> context = new BOpContext<IBindingSet>(
- null/* fed */, jnl/* indexManager */,
+ new MockRunningQuery(null/* fed */, jnl/* indexManager */,
ITx.READ_COMMITTED/* readTimestamp */,
- ITx.UNISOLATED/* writeTimestamp */, -1/* partitionId */, stats,
+ ITx.UNISOLATED/* writeTimestamp */), -1/* partitionId */, stats,
source, sink, null/* sink2 */);
// get task.
@@ -435,9 +436,9 @@
final IBlockingBuffer<IBindingSet[]> sink = query.newBuffer();
final BOpContext<IBindingSet> context = new BOpContext<IBindingSet>(
- null/* fed */, jnl/* indexManager */,
+ new MockRunningQuery(null/* fed */, jnl/* indexManager */,
ITx.READ_COMMITTED/* readTimestamp */,
- ITx.UNISOLATED/* writeTimestamp */, -1/* partitionId */, stats,
+ ITx.UNISOLATED/* writeTimestamp */), -1/* partitionId */, stats,
source, sink, null/* sink2 */);
// get task.
@@ -553,9 +554,9 @@
final PipelineJoinStats stats = query.newStats();
final BOpContext<IBindingSet> context = new BOpContext<IBindingSet>(
- null/* fed */, jnl/* indexManager */,
+ new MockRunningQuery(null/* fed */, jnl/* indexManager */,
ITx.READ_COMMITTED/* readTimestamp */,
- ITx.UNISOLATED/* writeTimestamp */, -1/* partitionId */, stats,
+ ITx.UNISOLATED/* writeTimestamp */), -1/* partitionId */, stats,
source, sink, null/* sink2 */);
// get task.
@@ -675,9 +676,9 @@
final PipelineJoinStats stats = query.newStats();
final BOpContext<IBindingSet> context = new BOpContext<IBindingSet>(
- null/* fed */, jnl/* indexManager */,
+ new MockRunningQuery(null/* fed */, jnl/* indexManager */,
ITx.READ_COMMITTED/* readTimestamp */,
- ITx.UNISOLATED/* writeTimestamp */, -1/* partitionId */, stats,
+ ITx.UNISOLATED/* writeTimestamp */), -1/* partitionId */, stats,
source, sink, sink2);
// get task.
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/solutions/TestDistinctBindingSets.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/solutions/TestDistinctBindingSets.java 2010-09-05 18:16:01 UTC (rev 3509)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/solutions/TestDistinctBindingSets.java 2010-09-05 20:17:44 UTC (rev 3510)
@@ -29,7 +29,6 @@
import java.util.LinkedList;
import java.util.List;
-import java.util.Properties;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.FutureTask;
@@ -46,11 +45,9 @@
import com.bigdata.bop.NV;
import com.bigdata.bop.Var;
import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.bop.engine.MockRunningQuery;
import com.bigdata.bop.engine.TestQueryEngine;
-import com.bigdata.bop.solutions.DistinctBindingSetOp;
-import com.bigdata.journal.BufferMode;
import com.bigdata.journal.ITx;
-import com.bigdata.journal.Journal;
import com.bigdata.relation.accesspath.IAsynchronousIterator;
import com.bigdata.relation.accesspath.IBlockingBuffer;
import com.bigdata.relation.accesspath.ThickAsynchronousIterator;
@@ -78,25 +75,25 @@
super(name);
}
- @Override
- public Properties getProperties() {
+// @Override
+// public Properties getProperties() {
+//
+// final Properties p = new Properties(super.getProperties());
+//
+// p.setProperty(Journal.Options.BUFFER_MODE, BufferMode.Transient
+// .toString());
+//
+// return p;
+//
+// }
- final Properties p = new Properties(super.getProperties());
+// Journal jnl = null;
- p.setProperty(Journal.Options.BUFFER_MODE, BufferMode.Transient
- .toString());
-
- return p;
-
- }
-
- Journal jnl = null;
-
List<IBindingSet> data = null;
public void setUp() throws Exception {
- jnl = new Journal(getProperties());
+// jnl = new Journal(getProperties());
setUpData();
@@ -153,11 +150,11 @@
public void tearDown() throws Exception {
- if (jnl != null) {
- jnl.destroy();
- jnl = null;
- }
-
+// if (jnl != null) {
+// jnl.destroy();
+// jnl = null;
+// }
+//
// clear reference.
data = null;
@@ -208,16 +205,17 @@
final IBlockingBuffer<IBindingSet[]> sink = query.newBuffer();
final BOpContext<IBindingSet> context = new BOpContext<IBindingSet>(
- null/* fed */, jnl/* indexManager */,
+ new MockRunningQuery(null/* fed */, null/* indexManager */,
ITx.READ_COMMITTED/* readTimestamp */,
- ITx.UNISOLATED/* writeTimestamp */, -1/* partitionId */, stats,
+ ITx.UNISOLATED/* writeTimestamp */), -1/* partitionId */, stats,
source, sink, null/* sink2 */);
// get task.
final FutureTask<Void> ft = query.eval(context);
// execute task.
- jnl.getExecutorService().execute(ft);
+// jnl.getExecutorService().execute(ft);
+ ft.run();
TestQueryEngine.assertSameSolutions(expected, sink.iterator());
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/solutions/TestSliceOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/solutions/TestSliceOp.java 2010-09-05 18:16:01 UTC (rev 3509)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/solutions/TestSliceOp.java 2010-09-05 20:17:44 UTC (rev 3510)
@@ -28,7 +28,6 @@
package com.bigdata.bop.solutions;
import java.util.ArrayList;
-import java.util.Properties;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.FutureTask;
@@ -45,11 +44,9 @@
import com.bigdata.bop.NV;
import com.bigdata.bop.Var;
import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.bop.engine.MockRunningQuery;
import com.bigdata.bop.engine.TestQueryEngine;
-import com.bigdata.bop.solutions.SliceOp;
-import com.bigdata.journal.BufferMode;
import com.bigdata.journal.ITx;
-import com.bigdata.journal.Journal;
import com.bigdata.relation.accesspath.IAsynchronousIterator;
import com.bigdata.relation.accesspath.IBlockingBuffer;
import com.bigdata.relation.accesspath.ThickAsynchronousIterator;
@@ -75,25 +72,25 @@
super(name);
}
- @Override
- public Properties getProperties() {
+// @Override
+// public Properties getProperties() {
+//
+// final Properties p = new Properties(super.getProperties());
+//
+// p.setProperty(Journal.Options.BUFFER_MODE, BufferMode.Transient
+// .toString());
+//
+// return p;
+//
+// }
+//
+// Journal jnl = null;
- final Properties p = new Properties(super.getProperties());
-
- p.setProperty(Journal.Options.BUFFER_MODE, BufferMode.Transient
- .toString());
-
- return p;
-
- }
-
- Journal jnl = null;
-
ArrayList<IBindingSet> data;
public void setUp() throws Exception {
- jnl = new Journal(getProperties());
+// jnl = new Journal(getProperties());
setUpData();
@@ -150,10 +147,10 @@
public void tearDown() throws Exception {
- if (jnl != null) {
- jnl.destroy();
- jnl = null;
- }
+// if (jnl != null) {
+// jnl.destroy();
+// jnl = null;
+// }
// clear reference.
data = null;
@@ -208,16 +205,17 @@
final IBlockingBuffer<IBindingSet[]> sink = query.newBuffer();
final BOpContext<IBindingSet> context = new BOpContext<IBindingSet>(
- null/* fed */, jnl/* indexManager */,
+ new MockRunningQuery(null/* fed */, null/* indexManager */,
ITx.READ_COMMITTED/* readTimestamp */,
- ITx.UNISOLATED/* writeTimestamp */, -1/* partitionId */, stats,
+ ITx.UNISOLATED/* writeTimestamp */), -1/* partitionId */, stats,
source, sink, null/* sink2 */);
// get task.
final FutureTask<Void> ft = query.eval(context);
// execute task.
- jnl.getExecutorService().execute(ft);
+// jnl.getExecutorService().execute(ft);
+ ft.run();
TestQueryEngine.assertSameSolutions(expected, sink.iterator());
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-09-05 18:16:10
|
Revision: 3509
http://bigdata.svn.sourceforge.net/bigdata/?rev=3509&view=rev
Author: thompsonbry
Date: 2010-09-05 18:16:01 +0000 (Sun, 05 Sep 2010)
Log Message:
-----------
Added a "slice" operator and a basic unit test for that operator.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/IQueryOptions.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/QueryOptions.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/NestedSubqueryWithJoinThreadsTask.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/RuleStats.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/pipeline/JoinMasterTask.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/bset/TestConditionalRoutingOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/RunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/join/TestPipelineJoin.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/relation/rule/TestSlice.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestSlice.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl.java
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/ComparatorOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/DistinctBindingSetOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/ISortOrder.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/MemorySortOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/SortOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/SortOrder.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/SparqlBindingSetComparatorOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/package.html
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/ISlice.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/Slice.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/solutions/
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/solutions/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/solutions/TestDistinctBindingSets.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/solutions/TestMemorySortOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/solutions/TestSPARQLBindingSetComparatorOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/solutions/TestSliceOp.java
Removed Paths:
-------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ComparatorOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/DistinctBindingSetOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISlice.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISortOrder.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/MemorySortOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/Slice.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOrder.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SparqlBindingSetComparatorOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/package.html
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/package.html
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/aggregation/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/aggregation/TestDistinctBindingSets.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/aggregation/TestMemorySortOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/aggregation/TestSPARQLBindingSetComparatorOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/solutions/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/solutions/TestDistinctBindingSets.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/solutions/TestSortBindingSets.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java 2010-09-05 17:02:34 UTC (rev 3508)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java 2010-09-05 18:16:01 UTC (rev 3509)
@@ -32,6 +32,8 @@
import org.apache.log4j.Logger;
import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.bop.engine.QueryEngine;
+import com.bigdata.bop.solutions.SliceOp;
import com.bigdata.btree.IIndex;
import com.bigdata.btree.ILocalBTreeView;
import com.bigdata.btree.IRangeQuery;
@@ -652,6 +654,24 @@
}
+ /**
+ * Cancel the running query (normal termination).
+ * <p>
+ * Note: This method provides a means for an operator to indicate that the
+ * query should halt immediately. It used used by {@link SliceOp}, which
+ * needs to terminate the entire query once the slice has been satisfied.
+ * (If {@link SliceOp} just jumped out of its own evaluation loop then the
+ * query would not produce more results, but it would continue to run and
+ * the over produced results would just be thrown away.)
+ * <p>
+ * Note: When an individual {@link BOp} evaluation throws an exception, the
+ * {@link QueryEngine} will catch that exception and halt query evaluation
+ * with that thrown cause.
+ */
+ public void halt() {
+
+ }
+
/*
* I've replaced this with AbstractSplitter for the moment.
*/
Deleted: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ComparatorOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ComparatorOp.java 2010-09-05 17:02:34 UTC (rev 3508)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ComparatorOp.java 2010-09-05 18:16:01 UTC (rev 3509)
@@ -1,77 +0,0 @@
-/**
-
-Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
-/*
- * Created on Sep 4, 2010
- */
-
-package com.bigdata.bop.aggregation;
-
-import java.util.Comparator;
-import java.util.Map;
-
-import com.bigdata.bop.BOp;
-import com.bigdata.bop.BOpBase;
-import com.bigdata.bop.IBindingSet;
-
-/**
- * Base class for operators which impose a sort order on binding sets.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
- */
-abstract public class ComparatorOp extends BOpBase implements
- Comparator<IBindingSet> {
-
- /**
- *
- */
- private static final long serialVersionUID = 1L;
-
- public interface Annotations extends BOp.Annotations {
-
- /**
- * An {@link ISortOrder}[] specifying the variables on which the sort
- * will be imposed and the order (ascending or descending) for each
- * variable.
- */
- String ORDER = ComparatorOp.class.getName() + ".order";
-
- }
-
- /**
- * @param op
- */
- public ComparatorOp(BOpBase op) {
- super(op);
- }
-
- /**
- * @param args
- * @param annotations
- */
- public ComparatorOp(BOp[] args, Map<String, Object> annotations) {
- super(args, annotations);
- }
-
-}
Deleted: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/DistinctBindingSetOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/DistinctBindingSetOp.java 2010-09-05 17:02:34 UTC (rev 3508)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/DistinctBindingSetOp.java 2010-09-05 18:16:01 UTC (rev 3509)
@@ -1,324 +0,0 @@
-package com.bigdata.bop.aggregation;
-
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
-import java.util.concurrent.Callable;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.FutureTask;
-
-import com.bigdata.bop.BOp;
-import com.bigdata.bop.BOpContext;
-import com.bigdata.bop.BindingSetPipelineOp;
-import com.bigdata.bop.HashBindingSet;
-import com.bigdata.bop.IBindingSet;
-import com.bigdata.bop.IConstant;
-import com.bigdata.bop.IVariable;
-import com.bigdata.bop.engine.BOpStats;
-import com.bigdata.relation.accesspath.IAsynchronousIterator;
-import com.bigdata.relation.accesspath.IBlockingBuffer;
-
-/**
- * A pipelined DISTINCT operator based on a hash table.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id: DistinctElementFilter.java 3466 2010-08-27 14:28:04Z
- * thompsonbry $
- */
-public class DistinctBindingSetOp extends BindingSetPipelineOp {
-
- /**
- *
- */
- private static final long serialVersionUID = 1L;
-
- public interface Annotations extends BindingSetPipelineOp.Annotations {
-
- /**
- * The initial capacity of the {@link ConcurrentHashMap} used to impose
- * the distinct constraint.
- *
- * @see #DEFAULT_INITIAL_CAPACITY
- */
- String INITIAL_CAPACITY = DistinctBindingSetOp.class.getName()+".initialCapacity";
-
- int DEFAULT_INITIAL_CAPACITY = 16;
-
- /**
- * The load factor of the {@link ConcurrentHashMap} used to impose
- * the distinct constraint.
- *
- * @see #DEFAULT_LOAD_FACTOR
- */
- String LOAD_FACTOR = DistinctBindingSetOp.class.getName()+".loadFactor";
-
- float DEFAULT_LOAD_FACTOR = .75f;
-
- /**
- * The concurrency level of the {@link ConcurrentHashMap} used to impose
- * the distinct constraint.
- *
- * @see #DEFAULT_CONCURRENCY_LEVEL
- */
- String CONCURRENCY_LEVEL = DistinctBindingSetOp.class.getName()+".concurrencyLevel";
-
- int DEFAULT_CONCURRENCY_LEVEL = 16;
-
- /**
- * The variables on which the distinct constraint will be imposed.
- * Binding sets with distinct values for the specified variables will be
- * passed on.
- */
- String VARIABLES = DistinctBindingSetOp.class.getName() + ".variables";
-
- }
-
- /**
- * Required deep copy constructor.
- */
- public DistinctBindingSetOp(final DistinctBindingSetOp op) {
- super(op);
- }
-
- /**
- * Required shallow copy constructor.
- */
- public DistinctBindingSetOp(final BOp[] args,
- final Map<String, Object> annotations) {
-
- super(args, annotations);
-
- }
-
- /**
- * @see Annotations#INITIAL_CAPACITY
- */
- public int getInitialCapacity() {
-
- return getProperty(Annotations.INITIAL_CAPACITY,
- Annotations.DEFAULT_INITIAL_CAPACITY);
-
- }
-
- /**
- * @see Annotations#LOAD_FACTOR
- */
- public float getLoadFactor() {
-
- return getProperty(Annotations.LOAD_FACTOR,
- Annotations.DEFAULT_LOAD_FACTOR);
-
- }
-
- /**
- * @see Annotations#CONCURRENCY_LEVEL
- */
- public int getConcurrencyLevel() {
-
- return getProperty(Annotations.CONCURRENCY_LEVEL,
- Annotations.DEFAULT_CONCURRENCY_LEVEL);
-
- }
-
- /**
- * @see Annotations#VARIABLES
- */
- public IVariable<?>[] getVariables() {
-
- return getRequiredProperty(Annotations.VARIABLES);
-
- }
-
- public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
-
- return new FutureTask<Void>(new DistinctTask(this, context));
-
- }
-
- /**
- * Wrapper used for the as bound solutions in the {@link ConcurrentHashMap}.
- */
- private static class Solution {
- private final int hash;
-
- private final IConstant<?>[] vals;
-
- public Solution(final IConstant<?>[] vals) {
- this.vals = vals;
- this.hash = java.util.Arrays.hashCode(vals);
- }
-
- public int hashCode() {
- return hash;
- }
-
- public boolean equals(final Object o) {
- if (this == o)
- return true;
- if (!(o instanceof Solution)) {
- return false;
- }
- final Solution t = (Solution) o;
- if (vals.length != t.vals.length)
- return false;
- for (int i = 0; i < vals.length; i++) {
- // @todo verify that this allows for nulls with a unit test.
- if (vals[i] == t.vals[i])
- continue;
- if (vals[i] == null)
- return false;
- if (!vals[i].equals(t.vals[i]))
- return false;
- }
- return true;
- }
- }
-
- /**
- * Task executing on the node.
- */
- static private class DistinctTask implements Callable<Void> {
-
- private final BOpContext<IBindingSet> context;
-
- /**
- * A concurrent map whose keys are the bindings on the specified
- * variables (the keys and the values are the same since the map
- * implementation does not allow <code>null</code> values).
- */
- private /*final*/ ConcurrentHashMap<Solution, Solution> map;
-
- /**
- * The variables used to impose a distinct constraint.
- */
- private final IVariable<?>[] vars;
-
- DistinctTask(final DistinctBindingSetOp op,
- final BOpContext<IBindingSet> context) {
-
- this.context = context;
-
- this.vars = op.getVariables();
-
- if (vars == null)
- throw new IllegalArgumentException();
-
- if (vars.length == 0)
- throw new IllegalArgumentException();
-
- this.map = new ConcurrentHashMap<Solution, Solution>(
- op.getInitialCapacity(), op.getLoadFactor(),
- op.getConcurrencyLevel());
-
- }
-
- /**
- * If the bindings are distinct for the configured variables then return
- * those bindings.
- *
- * @param bset
- * The binding set to be filtered.
- *
- * @return The distinct as bound values -or- <code>null</code> if the
- * binding set duplicates a solution which was already accepted.
- */
- private IConstant<?>[] accept(final IBindingSet bset) {
-
- final IConstant<?>[] r = new IConstant<?>[vars.length];
-
- for (int i = 0; i < vars.length; i++) {
-
- /*
- * Note: This allows null's.
- *
- * @todo write a unit test when some variables are not bound.
- */
- r[i] = bset.get(vars[i]);
-
- }
-
- final Solution s = new Solution(r);
-
- final boolean distinct = map.putIfAbsent(s, s) == null;
-
- return distinct ? r : null;
-
- }
-
- public Void call() throws Exception {
-
- final BOpStats stats = context.getStats();
-
- final IAsynchronousIterator<IBindingSet[]> itr = context
- .getSource();
-
- final IBlockingBuffer<IBindingSet[]> sink = context.getSink();
-
- try {
-
- while (itr.hasNext()) {
-
- final IBindingSet[] a = itr.next();
-
- stats.chunksIn.increment();
- stats.unitsIn.add(a.length);
-
- final List<IBindingSet> accepted = new LinkedList<IBindingSet>();
-
- int naccepted = 0;
-
- for (IBindingSet bset : a) {
-
-// System.err.println("considering: " + bset);
-
- final IConstant<?>[] vals = accept(bset);
-
- if (vals != null) {
-
-// System.err.println("accepted: "
-// + Arrays.toString(vals));
-
- accepted.add(new HashBindingSet(vars, vals));
-
- naccepted++;
-
- }
-
- }
-
- if (naccepted > 0) {
-
- final IBindingSet[] b = accepted
- .toArray(new IBindingSet[naccepted]);
-
-// System.err.println("output: "
-// + Arrays.toString(b));
-
- sink.add(b);
-
- stats.unitsOut.add(naccepted);
- stats.chunksOut.increment();
-
- }
-
- }
-
- sink.flush();
-
- // done.
- return null;
-
- } finally {
-
- sink.close();
-
- // discard the map.
- map = null;
-
- }
-
- }
-
- }
-
-}
Deleted: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISlice.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISlice.java 2010-09-05 17:02:34 UTC (rev 3508)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISlice.java 2010-09-05 18:16:01 UTC (rev 3509)
@@ -1,69 +0,0 @@
-/*
-
-Copyright (C) SYSTAP, LLC 2006-2008. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-
-*/
-/*
- * Created on Sep 24, 2008
- */
-
-package com.bigdata.bop.aggregation;
-
-import java.io.Serializable;
-
-import com.bigdata.relation.accesspath.IAccessPath;
-
-/**
- * Indicates the first solution to be returned to the caller (offset) and the
- * #of solutions to be returned (limit).
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
- */
-public interface ISlice extends Serializable {
-
- /**
- * The first solution to be returned to the caller. A value of ZERO (0)
- * indicates that all solutions should be returned.
- */
- public long getOffset();
-
- /**
- * The maximum #of solutions to be returned to the caller. A value of
- * {@link Long#MAX_VALUE} indicates that there is no limit.
- *
- * @todo modify to be consistent with
- * {@link IAccessPath#iterator(long, long, int)} where a limit of ZERO
- * (0L) is interpreted as NO limit and a limit of
- * {@link Long#MAX_VALUE} is interpreted as ZERO (0L) (that is, also
- * no limit).
- */
- public long getLimit();
-
- /**
- * The index of the last solution that we will generate (OFFSET + LIMIT). If
- * OFFSET + LIMIT would be greater than {@link Long#MAX_VALUE}, then use
- * {@link Long#MAX_VALUE} instead.
- */
- public long getLast();
-
-}
Deleted: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISortOrder.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISortOrder.java 2010-09-05 17:02:34 UTC (rev 3508)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISortOrder.java 2010-09-05 18:16:01 UTC (rev 3509)
@@ -1,55 +0,0 @@
-/*
-
-Copyright (C) SYSTAP, LLC 2006-2008. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-
-*/
-/*
- * Created on Sep 24, 2008
- */
-
-package com.bigdata.bop.aggregation;
-
-import java.io.Serializable;
-
-import com.bigdata.bop.IVariable;
-
-/**
- * A variable and an order that will be imposed on the values for that variable.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
- */
-public interface ISortOrder<E> extends Serializable {
-
- /**
- * The variable whose values will be sorted.
- */
- IVariable<E> getVariable();
-
- /**
- * <code>true</code> iff the values will be placed into an ascending sort
- * and <code>false</code> if the values will be placed into a descending
- * sort.
- */
- boolean isAscending();
-
-}
Deleted: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/MemorySortOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/MemorySortOp.java 2010-09-05 17:02:34 UTC (rev 3508)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/MemorySortOp.java 2010-09-05 18:16:01 UTC (rev 3509)
@@ -1,112 +0,0 @@
-package com.bigdata.bop.aggregation;
-
-import java.util.Arrays;
-import java.util.Comparator;
-import java.util.Map;
-import java.util.concurrent.Callable;
-import java.util.concurrent.FutureTask;
-
-import com.bigdata.bop.BOp;
-import com.bigdata.bop.BOpContext;
-import com.bigdata.bop.BOpUtility;
-import com.bigdata.bop.IBindingSet;
-import com.bigdata.bop.engine.BOpStats;
-import com.bigdata.relation.accesspath.IBlockingBuffer;
-
-/**
- * An in-memory merge sort for binding sets.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id: DistinctElementFilter.java 3466 2010-08-27 14:28:04Z
- * thompsonbry $
- *
- * @todo unit tests.
- * @todo do an external merge sort operator.
- */
-public class MemorySortOp extends SortOp {
-
- /**
- *
- */
- private static final long serialVersionUID = 1L;
-
- /**
- * Required deep copy constructor.
- */
- public MemorySortOp(final MemorySortOp op) {
- super(op);
- }
-
- /**
- * Required shallow copy constructor.
- */
- public MemorySortOp(final BOp[] args,
- final Map<String, Object> annotations) {
-
- super(args, annotations);
-
- }
-
- public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
-
- return new FutureTask<Void>(new SortTask(this, context));
-
- }
-
- /**
- * Task executing on the node.
- */
- static private class SortTask implements Callable<Void> {
-
- private final BOpContext<IBindingSet> context;
-
- /**
- * The binding set comparator.
- */
- private final Comparator<IBindingSet> comparator;
-
- SortTask(final MemorySortOp op,
- final BOpContext<IBindingSet> context) {
-
- this.context = context;
-
- this.comparator = op.getComparator();
-
- }
-
- public Void call() throws Exception {
-
- final BOpStats stats = context.getStats();
-
- final IBlockingBuffer<IBindingSet[]> sink = context.getSink();
-
- try {
-
- final IBindingSet[] all = BOpUtility.toArray(context
- .getSource(), stats);
-
- // sort.
- Arrays.sort(all, comparator);
-
- // update counters.
- stats.unitsOut.add(all.length);
- stats.chunksOut.increment();
-
- // write output and flush.
- sink.add(all);
- sink.flush();
-
- // done.
- return null;
-
- } finally {
-
- sink.close();
-
- }
-
- }
-
- }
-
-}
Deleted: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/Slice.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/Slice.java 2010-09-05 17:02:34 UTC (rev 3508)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/Slice.java 2010-09-05 18:16:01 UTC (rev 3509)
@@ -1,129 +0,0 @@
-/*
-
-Copyright (C) SYSTAP, LLC 2006-2008. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-
-*/
-/*
- * Created on Sep 24, 2008
- */
-
-package com.bigdata.bop.aggregation;
-
-import java.io.Externalizable;
-import java.io.IOException;
-import java.io.ObjectInput;
-import java.io.ObjectOutput;
-import java.math.BigInteger;
-
-/**
- * Default implementation.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
- */
-public class Slice implements ISlice, Externalizable {
-
- /**
- *
- */
- private static final long serialVersionUID = 5396509164843609197L;
-
- private long offset;
- private long limit;
- private long last;
-
- /**
- * A slice corresponding to all results (offset is zero, limit is
- * {@link Long#MAX_VALUE}).
- */
- public static final transient ISlice ALL = new Slice(0, Long.MAX_VALUE);
-
- /**
- *
- * @param offset
- * @param limit
- *
- * @throws IllegalArgumentException
- * if offset is negative.
- * @throws IllegalArgumentException
- * if limit is non-positive.
- */
- public Slice(final long offset, final long limit) {
-
- if (offset < 0)
- throw new IllegalArgumentException();
-
- if (limit <= 0)
- throw new IllegalArgumentException();
-
- this.offset = offset;
-
- this.limit = limit;
-
- // @todo what is a cheaper way to do this?
- this.last = BigInteger.valueOf(offset).add(BigInteger.valueOf(limit))
- .min(BigInteger.valueOf(Long.MAX_VALUE)).longValue();
-
- }
-
- public long getOffset() {
-
- return offset;
-
- }
-
- public long getLimit() {
-
- return limit;
-
- }
-
- public long getLast() {
-
- return last;
-
- }
-
- public String toString() {
-
- return "Slice{offset="+offset+", limit="+limit+", last="+last+"}";
-
- }
-
- public void readExternal(ObjectInput in) throws IOException,
- ClassNotFoundException {
-
- offset = in.readLong();
- limit = in.readLong();
- last = in.readLong();
-
- }
-
- public void writeExternal(ObjectOutput out) throws IOException {
-
- out.writeLong(offset);
- out.writeLong(limit);
- out.writeLong(last);
-
- }
-
-}
Deleted: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOp.java 2010-09-05 17:02:34 UTC (rev 3508)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOp.java 2010-09-05 18:16:01 UTC (rev 3509)
@@ -1,86 +0,0 @@
-/**
-
-Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
-/*
- * Created on Sep 4, 2010
- */
-
-package com.bigdata.bop.aggregation;
-
-import java.util.Map;
-
-import com.bigdata.bop.BOp;
-import com.bigdata.bop.BindingSetPipelineOp;
-import com.bigdata.bop.IBindingSet;
-import com.bigdata.bop.PipelineOp;
-
-/**
- * Base class for operators which sort binding sets.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
- */
-abstract public class SortOp extends BindingSetPipelineOp {
-
- /**
- *
- */
- private static final long serialVersionUID = 1L;
-
- public interface Annotations extends BindingSetPipelineOp.Annotations {
-
- /**
- * The {@link ComparatorOp} which will impose the ordering on the
- * binding sets.
- *
- * @see ComparatorOp
- */
- String COMPARATOR = MemorySortOp.class.getName() + ".comparator";
-
- }
-
- /**
- * @param op
- */
- public SortOp(PipelineOp<IBindingSet> op) {
- super(op);
- }
-
- /**
- * @param args
- * @param annotations
- */
- public SortOp(BOp[] args, Map<String, Object> annotations) {
- super(args, annotations);
- }
-
- /**
- * @see Annotations#COMPARATOR
- */
- public ComparatorOp getComparator() {
-
- return getRequiredProperty(Annotations.COMPARATOR);
-
- }
-
-}
Deleted: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOrder.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOrder.java 2010-09-05 17:02:34 UTC (rev 3508)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOrder.java 2010-09-05 18:16:01 UTC (rev 3509)
@@ -1,80 +0,0 @@
-/*
-
-Copyright (C) SYSTAP, LLC 2006-2008. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-
-*/
-/*
- * Created on Sep 24, 2008
- */
-
-package com.bigdata.bop.aggregation;
-
-import com.bigdata.bop.IVariable;
-
-/**
- * Default impl.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
- */
-public class SortOrder<E> implements ISortOrder<E> {
-
- /**
- *
- */
- private static final long serialVersionUID = -669873421670514139L;
-
- private final IVariable<E> var;
- private final boolean asc;
-
- /**
- *
- * @param var
- * The variable.
- * @param asc
- * <code>true</code> for an ascending sort and
- * <code>false</code> for a descending sort.
- */
- public SortOrder(final IVariable<E> var, final boolean asc) {
-
- if (var == null)
- throw new IllegalArgumentException();
-
- this.var = var;
-
- this.asc = asc;
-
- }
-
- public IVariable<E> getVariable() {
-
- return var;
-
- }
-
- public boolean isAscending() {
-
- return asc;
-
- }
-
-}
Deleted: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SparqlBindingSetComparatorOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SparqlBindingSetComparatorOp.java 2010-09-05 17:02:34 UTC (rev 3508)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SparqlBindingSetComparatorOp.java 2010-09-05 18:16:01 UTC (rev 3509)
@@ -1,101 +0,0 @@
-package com.bigdata.bop.aggregation;
-
-import java.util.Comparator;
-import java.util.Map;
-
-import org.openrdf.query.algebra.evaluation.util.ValueComparator;
-
-import com.bigdata.bop.BOp;
-import com.bigdata.bop.IBindingSet;
-import com.bigdata.bop.IVariable;
-
-/**
- * A comparator for SPARQL binding sets.
- *
- * @see http://www.w3.org/TR/rdf-sparql-query/#modOrderBy
- * @see ValueComparator
- *
- * @todo unit tests.
- */
-public class SparqlBindingSetComparatorOp extends ComparatorOp {
-
- /**
- *
- */
- private static final long serialVersionUID = 1L;
-
- /**
- * Required deep copy constructor.
- */
- public SparqlBindingSetComparatorOp(final SparqlBindingSetComparatorOp op) {
- super(op);
- }
-
- /**
- * Required shallow copy constructor.
- */
- public SparqlBindingSetComparatorOp(final BOp[] args,
- final Map<String, Object> annotations) {
-
- super(args, annotations);
-
- }
-
- /**
- * @see Annotations#ORDER
- */
- public ISortOrder<?>[] getOrder() {
-
- return getRequiredProperty(Annotations.ORDER);
-
- }
-
- /**
- * The sort order to be imposed.
- */
- private transient ISortOrder<?>[] order;
-
- private transient Comparator vc;
-
- public int compare(final IBindingSet bs1, final IBindingSet bs2) {
-
- if (order == null) {
-
- // lazy initialization.
- order = getOrder();
-
- if (order == null)
- throw new IllegalArgumentException();
-
- if (order.length == 0)
- throw new IllegalArgumentException();
-
- // comparator for RDF Value objects.
- vc = new ValueComparator();
-
- }
-
- for (int i = 0; i < order.length; i++) {
-
- final ISortOrder<?> o = order[i];
-
- final IVariable v = o.getVariable();
-
- int ret = vc.compare(bs1.get(v).get(), bs2.get(v).get());
-
- if (!o.isAscending())
- ret = -ret;
-
- if (ret != 0) {
- // not equal for this variable.
- return ret;
- }
-
- }
-
- // equal for all variables.
- return 0;
-
- }
-
-}
Deleted: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/package.html
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/package.html 2010-09-05 17:02:34 UTC (rev 3508)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/package.html 2010-09-05 18:16:01 UTC (rev 3509)
@@ -1,17 +0,0 @@
-<html>
-<head>
-<title>solution modifier operators (distinct, sort, slice, and aggregation)</title>
-</head>
-<body>
-
-<p>
-
- This package provides distinct, sort, and aggregation operators. All of
- these are potentially high volume hash partitioned operations against a
- clustered database. Both in memory and disk based versions of the each
- operator should be implemented.
-
-</p>
-
-</body>
-</html>
\ No newline at end of file
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/ComparatorOp.java (from rev 3508, branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ComparatorOp.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/ComparatorOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/ComparatorOp.java 2010-09-05 18:16:01 UTC (rev 3509)
@@ -0,0 +1,77 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Sep 4, 2010
+ */
+
+package com.bigdata.bop.solutions;
+
+import java.util.Comparator;
+import java.util.Map;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpBase;
+import com.bigdata.bop.IBindingSet;
+
+/**
+ * Base class for operators which impose a sort order on binding sets.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+abstract public class ComparatorOp extends BOpBase implements
+ Comparator<IBindingSet> {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ public interface Annotations extends BOp.Annotations {
+
+ /**
+ * An {@link ISortOrder}[] specifying the variables on which the sort
+ * will be imposed and the order (ascending or descending) for each
+ * variable.
+ */
+ String ORDER = ComparatorOp.class.getName() + ".order";
+
+ }
+
+ /**
+ * @param op
+ */
+ public ComparatorOp(BOpBase op) {
+ super(op);
+ }
+
+ /**
+ * @param args
+ * @param annotations
+ */
+ public ComparatorOp(BOp[] args, Map<String, Object> annotations) {
+ super(args, annotations);
+ }
+
+}
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/DistinctBindingSetOp.java (from rev 3508, branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/DistinctBindingSetOp.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/DistinctBindingSetOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/DistinctBindingSetOp.java 2010-09-05 18:16:01 UTC (rev 3509)
@@ -0,0 +1,324 @@
+package com.bigdata.bop.solutions;
+
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.FutureTask;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpContext;
+import com.bigdata.bop.BindingSetPipelineOp;
+import com.bigdata.bop.HashBindingSet;
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.IConstant;
+import com.bigdata.bop.IVariable;
+import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.relation.accesspath.IAsynchronousIterator;
+import com.bigdata.relation.accesspath.IBlockingBuffer;
+
+/**
+ * A pipelined DISTINCT operator based on a hash table.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id: DistinctElementFilter.java 3466 2010-08-27 14:28:04Z
+ * thompsonbry $
+ */
+public class DistinctBindingSetOp extends BindingSetPipelineOp {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ public interface Annotations extends BindingSetPipelineOp.Annotations {
+
+ /**
+ * The initial capacity of the {@link ConcurrentHashMap} used to impose
+ * the distinct constraint.
+ *
+ * @see #DEFAULT_INITIAL_CAPACITY
+ */
+ String INITIAL_CAPACITY = DistinctBindingSetOp.class.getName()+".initialCapacity";
+
+ int DEFAULT_INITIAL_CAPACITY = 16;
+
+ /**
+ * The load factor of the {@link ConcurrentHashMap} used to impose
+ * the distinct constraint.
+ *
+ * @see #DEFAULT_LOAD_FACTOR
+ */
+ String LOAD_FACTOR = DistinctBindingSetOp.class.getName()+".loadFactor";
+
+ float DEFAULT_LOAD_FACTOR = .75f;
+
+ /**
+ * The concurrency level of the {@link ConcurrentHashMap} used to impose
+ * the distinct constraint.
+ *
+ * @see #DEFAULT_CONCURRENCY_LEVEL
+ */
+ String CONCURRENCY_LEVEL = DistinctBindingSetOp.class.getName()+".concurrencyLevel";
+
+ int DEFAULT_CONCURRENCY_LEVEL = 16;
+
+ /**
+ * The variables on which the distinct constraint will be imposed.
+ * Binding sets with distinct values for the specified variables will be
+ * passed on.
+ */
+ String VARIABLES = DistinctBindingSetOp.class.getName() + ".variables";
+
+ }
+
+ /**
+ * Required deep copy constructor.
+ */
+ public DistinctBindingSetOp(final DistinctBindingSetOp op) {
+ super(op);
+ }
+
+ /**
+ * Required shallow copy constructor.
+ */
+ public DistinctBindingSetOp(final BOp[] args,
+ final Map<String, Object> annotations) {
+
+ super(args, annotations);
+
+ }
+
+ /**
+ * @see Annotations#INITIAL_CAPACITY
+ */
+ public int getInitialCapacity() {
+
+ return getProperty(Annotations.INITIAL_CAPACITY,
+ Annotations.DEFAULT_INITIAL_CAPACITY);
+
+ }
+
+ /**
+ * @see Annotations#LOAD_FACTOR
+ */
+ public float getLoadFactor() {
+
+ return getProperty(Annotations.LOAD_FACTOR,
+ Annotations.DEFAULT_LOAD_FACTOR);
+
+ }
+
+ /**
+ * @see Annotations#CONCURRENCY_LEVEL
+ */
+ public int getConcurrencyLevel() {
+
+ return getProperty(Annotations.CONCURRENCY_LEVEL,
+ Annotations.DEFAULT_CONCURRENCY_LEVEL);
+
+ }
+
+ /**
+ * @see Annotations#VARIABLES
+ */
+ public IVariable<?>[] getVariables() {
+
+ return getRequiredProperty(Annotations.VARIABLES);
+
+ }
+
+ public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
+
+ return new FutureTask<Void>(new DistinctTask(this, context));
+
+ }
+
+ /**
+ * Wrapper used for the as bound solutions in the {@link ConcurrentHashMap}.
+ */
+ private static class Solution {
+ private final int hash;
+
+ private final IConstant<?>[] vals;
+
+ public Solution(final IConstant<?>[] vals) {
+ this.vals = vals;
+ this.hash = java.util.Arrays.hashCode(vals);
+ }
+
+ public int hashCode() {
+ return hash;
+ }
+
+ public boolean equals(final Object o) {
+ if (this == o)
+ return true;
+ if (!(o instanceof Solution)) {
+ return false;
+ }
+ final Solution t = (Solution) o;
+ if (vals.length != t.vals.length)
+ return false;
+ for (int i = 0; i < vals.length; i++) {
+ // @todo verify that this allows for nulls with a unit test.
+ if (vals[i] == t.vals[i])
+ continue;
+ if (vals[i] == null)
+ return false;
+ if (!vals[i].equals(t.vals[i]))
+ return false;
+ }
+ return true;
+ }
+ }
+
+ /**
+ * Task executing on the node.
+ */
+ static private class DistinctTask implements Callable<Void> {
+
+ private final BOpContext<IBindingSet> context;
+
+ /**
+ * A concurrent map whose keys are the bindings on the specified
+ * variables (the keys and the values are the same since the map
+ * implementation does not allow <code>null</code> values).
+ */
+ private /*final*/ ConcurrentHashMap<Solution, Solution> map;
+
+ /**
+ * The variables used to impose a distinct constraint.
+ */
+ private final IVariable<?>[] vars;
+
+ DistinctTask(final DistinctBindingSetOp op,
+ final BOpContext<IBindingSet> context) {
+
+ this.context = context;
+
+ this.vars = op.getVariables();
+
+ if (vars == null)
+ throw new IllegalArgumentException();
+
+ if (vars.length == 0)
+ throw new IllegalArgumentException();
+
+ this.map = new ConcurrentHashMap<Solution, Solution>(
+ op.getInitialCapacity(), op.getLoadFactor(),
+ op.getConcurrencyLevel());
+
+ }
+
+ /**
+ * If the bindings are distinct for the configured variables then return
+ * those bindings.
+ *
+ * @param bset
+ * The binding set to be filtered.
+ *
+ * @return The distinct as bound values -or- <code>null</code> if the
+ * binding set duplicates a solution which was already accepted.
+ */
+ private IConstant<?>[] accept(final IBindingSet bset) {
+
+ final IConstant<?>[] r = new IConstant<?>[vars.length];
+
+ for (int i = 0; i < vars.length; i++) {
+
+ /*
+ * Note: This allows null's.
+ *
+ * @todo write a unit test when some variables are not bound.
+ */
+ r[i] = bset.get(vars[i]);
+
+ }
+
+ final Solution s = new Solution(r);
+
+ final boolean distinct = map.putIfAbsent(s, s) == null;
+
+ return distinct ? r : null;
+
+ }
+
+ public Void call() throws Exception {
+
+ final BOpStats stats = context.getStats();
+
+ final IAsynchronousIterator<IBindingSet[]> itr = context
+ .getSource();
+
+ final IBlockingBuffer<IBindingSet[]> sink = context.getSink();
+
+ try {
+
+ while (itr.hasNext()) {
+
+ final IBindingSet[] a = itr.next();
+
+ stats.chunksIn.increment();
+ stats.unitsIn.add(a.length);
+
+ final List<IBindingSet> accepted = new LinkedList<IBindingSet>();
+
+ int naccepted = 0;
+
+ for (IBindingSet bset : a) {
+
+// System.err.println("considering: " + bset);
+
+ final IConstant<?>[] vals = accept(bset);
+
+ if (vals != null) {
+
+// System.err.println("accepted: "
+// + Arrays.toString(vals));
+
+ accepted.add(new HashBindingSet(vars, vals));
+
+ naccepted++;
+
+ }
+
+ }
+
+ if (naccepted > 0) {
+
+ final IBindingSet[] b = accepted
+ .toArray(new IBindingSet[naccepted]);
+
+// System.err.println("output: "
+// + Arrays.toString(b));
+
+ sink.add(b);
+
+ stats.unitsOut.add(naccepted);
+ stats.chunksOut.increment();
+
+ }
+
+ }
+
+ sink.flush();
+
+ // done.
+ return null;
+
+ } finally {
+
+ sink.close();
+
+ // discard the map.
+ map = null;
+
+ }
+
+ }
+
+ }
+
+}
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/ISortOrder.java (from rev 3508, branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISortOrder.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/ISortOrder.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/ISortOrder.java 2010-09-05 18:16:01 UTC (rev 3509)
@@ -0,0 +1,55 @@
+/*
+
+Copyright (C) SYSTAP, LLC 2006-2008. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+*/
+/*
+ * Created on Sep 24, 2008
+ */
+
+package com.bigdata.bop.solutions;
+
+import java.io.Serializable;
+
+import com.bigdata.bop.IVariable;
+
+/**
+ * A variable and an order that will be imposed on the values for that variable.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public interface ISortOrder<E> extends Serializable {
+
+ /**
+ * The variable whose values will be sorted.
+ */
+ IVariable<E> getVariable();
+
+ /**
+ * <code>true</code> iff the values will be placed into an ascending sort
+ * and <code>false</code> if the values will be placed into a descending
+ * sort.
+ */
+ boolean isAscending();
+
+}
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/MemorySortOp.java (from rev 3508, branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/MemorySortOp.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/MemorySortOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/MemorySortOp.java 2010-09-05 18:16:01 UTC (rev 3509)
@@ -0,0 +1,112 @@
+package com.bigdata.bop.solutions;
+
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.FutureTask;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpContext;
+import com.bigdata.bop.BOpUtility;
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.relation.accesspath.IBlockingBuffer;
+
+/**
+ * An in-memory merge sort for binding sets.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id: DistinctElementFilter.java 3466 2010-08-27 14:28:04Z
+ * thompsonbry $
+ *
+ * @todo unit tests.
+ * @todo do an external merge sort operator.
+ */
+public class MemorySortOp extends SortOp {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * Required deep copy constructor.
+ */
+ public MemorySortOp(final MemorySortOp op) {
+ super(op);
+ }
+
+ /**
+ * Required shallow copy constructor.
+ */
+ public MemorySortOp(final BOp[] args,
+ final Map<String, Object> annotations) {
+
+ super(args, annotations);
+
+ }
+
+ public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
+
+ return new FutureTask<Void>(new SortTask(this, context));
+
+ }
+
+ /**
+ * Task executing on the node.
+ */
+ static private class SortTask implements Callable<Void> {
+
+ private final BOpContext<IBindingSet> context;
+
+ /**
+ * The binding set comparator.
+ */
+ private final Comparator<IBindingSet> comparator;
+
+ SortTask(final MemorySortOp op,
+ final BOpContext<IBindingSet> context) {
+
+ this.context = context;
+
+ this.comparator = op.getComparator();
+
+ }
+
+ public Void call() throws Exception {
+
+ final BOpStats stats = context.getStats();
+
+ final IBlockingBuffer<IBindingSet[]> sink = context.getSink();
+
+ try {
+
+ final IBindingSet[] all = BOpUtility.toArray(context
+ .getSource(), stats);
+
+ // sort.
+ Arrays.sort(all, comparator);
+
+ // update counters.
+ stats.unitsOut.add(all.length);
+ stats.chunksOut.increment();
+
+ // write output and flush.
+ sink.add(all);
+ sink.flush();
+
+ // done.
+ return null;
+
+ } finally {
+
+ sink.close();
+
+ }
+
+ }
+
+ }
+
+}
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java 2010-09-05 18:16:01 UTC (rev 3509)
@@ -0,0 +1,275 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Sep 5, 2010
+ */
+
+package com.bigdata.bop.solutions;
+
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.FutureTask;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpContext;
+import com.bigdata.bop.BindingSetPipelineOp;
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.PipelineOp;
+import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.bop.engine.RunningQuery;
+import com.bigdata.relation.accesspath.IAsynchronousIterator;
+import com.bigdata.relation.accesspath.IBlockingBuffer;
+import com.bigdata.relation.accesspath.UnsynchronizedArrayBuffer;
+import com.bigdata.service.IBigdataFederation;
+
+/**
+ * An operator which imposes an offset/limit on a binding set pipeline.
+ * <p>
+ * Note: join processing typically involves concurrent processes, hence the
+ * order of the results will not be stable unless the results are sorted before
+ * applying the slice. When a slice is applied without a sort, the same query
+ * may return different results each time it is evaluated.
+ * <p>
+ * Note: When running on an {@link IBigdataFederation}, this operator must be
+ * imposed on the query controller so it can count the solutions as they flow
+ * through.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ *
+ * @todo If this operator is invoked for each chunk output by a query onto the
+ * pipeline then it will over produce unless (A) it is given the same
+ * {@link BOpStats} each time; and (B) it is not invoked for two chunks
+ * concurrently.
+ * <p>
+ * A safer way to impose the slice constraint is by wrapping the query
+ * buffer on the query controller. Once the slice is satisfied, it can
+ * just cancel the query. The only drawback of this approach is that the
+ * wrapping a buffer is not really the same as applying a {@link BOp} to
+ * the pipeline so it falls outside of the standard operator evaluation
+ * logic.
+ *
+ * @todo If we allow complex operator trees in which "subqueries" can also use a
+ * slice then either then need to run as their own query with their own
+ * {@link RunningQuery} state or the API for cancelling a running query as
+ * used here needs to only cancel evaluation of the child operators.
+ * Otherwise we could cancel all operator evaluation for the query,
+ * including operators which are ancestors of the {@link SliceOp}.
+ */
+public class SliceOp extends BindingSetPipelineOp {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID ...
[truncated message content] |
|
From: <tho...@us...> - 2010-09-05 17:02:44
|
Revision: 3508
http://bigdata.svn.sourceforge.net/bigdata/?rev=3508&view=rev
Author: thompsonbry
Date: 2010-09-05 17:02:34 +0000 (Sun, 05 Sep 2010)
Log Message:
-----------
Refactored the code to map binding sets over shards and made it more efficient (it reuses the same logic that is used by the scale-out index views).
Added IRelation#getKeyOrders() which brings us another step towards DDL support.
Added an in-memory binding set sort operator and an implementation for ordering binding sets for SPARQL which leverages the openrdf ValueCompator.
Added an "INSERT" operator to write on an index.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Constant.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/DistinctBindingSetOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/package.html
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/AbstractRelation.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/IMutableRelation.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/IRelation.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/RelationFusedView.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/accesspath/AbstractUnsynchronizedArrayBuffer.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/accesspath/ThreadLocalBufferFactory.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/IQueryOptions.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/QueryOptions.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/NestedSubqueryWithJoinThreadsTask.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/RuleStats.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/pipeline/JoinMasterTask.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/search/FullTextIndex.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/AbstractScaleOutFederation.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/ndx/AbstractSplitter.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/striterator/AbstractKeyOrder.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/striterator/IKeyOrder.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/aggregation/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/aggregation/TestDistinctBindingSets.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/ap/R.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/bset/TestConditionalRoutingOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/BOpStats.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/RunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestMapBindingSetsOverShards.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/join/TestPipelineJoin.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/relation/locator/TestDefaultResourceLocator.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/relation/rule/TestSlice.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/service/AbstractEmbeddedFederationTestCase.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/lexicon/LexiconRelation.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/magic/MagicIndexWriteProc.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/magic/MagicIndexWriter.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/magic/MagicKeyOrder.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/magic/MagicRelation.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/magic/MagicTupleSerializer.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPORelation.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestSlice.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl.java
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ComparatorOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISlice.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISortOrder.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/MemorySortOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/Slice.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOrder.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SparqlBindingSetComparatorOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/InsertOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/package.html
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/aggregation/TestMemorySortOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/aggregation/TestSPARQLBindingSetComparatorOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/FederatedQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/MapBindingSetsOverShardsBuffer.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestFederatedQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/mutation/
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/mutation/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/mutation/TestDelete.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/mutation/TestInsert.java
Removed Paths:
-------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ConstantEval.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/ISlice.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/ISortOrder.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/Slice.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/SortOrder.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/aggregation/TestSortBindingSets.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/MapBindingSetsOverShards.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOp.java 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOp.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -114,9 +114,22 @@
* @return The value of the annotation and <code>null</code> if the
* annotation is not bound.
*/
- Object getProperty(final String name);
+ <T> T getProperty(final String name);
/**
+ * Return the value of the named annotation.
+ *
+ * @param name
+ * The name of the annotation.
+ *
+ * @return The value of the annotation.
+ *
+ * @throws IllegalArgumentException
+ * if the named annotation is not bound.
+ */
+ <T> T getRequiredProperty(final String name);
+
+ /**
* Deep copy clone of the operator.
*/
BOp clone();
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -299,12 +299,23 @@
}
- public Object getProperty(final String name) {
+ public <T> T getProperty(final String name) {
- return annotations.get(name);
+ return (T) annotations.get(name);
}
+ public <T> T getRequiredProperty(final String name) {
+
+ final T tmp = (T) annotations.get(name);
+
+ if (tmp == null)
+ throw new IllegalArgumentException("Required property: " + name);
+
+ return tmp;
+
+ }
+
public String toString() {
final StringBuilder sb = new StringBuilder();
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -27,6 +27,8 @@
*/
package com.bigdata.bop;
+import java.util.Iterator;
+
import org.apache.log4j.Logger;
import com.bigdata.bop.engine.BOpStats;
@@ -36,6 +38,7 @@
import com.bigdata.journal.IIndexManager;
import com.bigdata.journal.ITx;
import com.bigdata.journal.TimestampUtility;
+import com.bigdata.mdi.PartitionLocator;
import com.bigdata.relation.IRelation;
import com.bigdata.relation.accesspath.AccessPath;
import com.bigdata.relation.accesspath.IAccessPath;
@@ -44,8 +47,10 @@
import com.bigdata.relation.locator.IResourceLocator;
import com.bigdata.relation.rule.IRule;
import com.bigdata.relation.rule.eval.IJoinNexus;
+import com.bigdata.service.AbstractScaleOutFederation;
import com.bigdata.service.DataService;
import com.bigdata.service.IBigdataFederation;
+import com.bigdata.service.ndx.IClientIndex;
import com.bigdata.striterator.IKeyOrder;
/**
@@ -270,12 +275,100 @@
*/
public IRelation getReadRelation(final IPredicate<?> pred) {
+ /*
+ * @todo Cache the resource locator?
+ *
+ * @todo This should be using the federation as the index manager when
+ * locating a resource for scale-out, right? But s/o reads must use the
+ * local index manager when actually obtaining the index view for the
+ * relation.
+ */
return (IRelation) getIndexManager().getResourceLocator().locate(
pred.getOnlyRelationName(), getReadTimestamp());
}
/**
+ * Return a writable view of the relation.
+ *
+ * @param namespace
+ * The namespace of the relation.
+ *
+ * @return A writable view of the relation.
+ */
+ public IRelation getWriteRelation(final String namespace) {
+
+ /*
+ * @todo Cache the resource locator?
+ *
+ * @todo This should be using the federation as the index manager when
+ * locating a resource for scale-out, right? But s/o writes must use
+ * the local index manager when actually obtaining the index view for
+ * the relation.
+ */
+ return (IRelation) getIndexManager().getResourceLocator().locate(
+ namespace, getWriteTimestamp());
+
+ }
+
+ /**
+ * Return an mutable view of the specified index.
+ *
+ * @param <T>
+ * The generic type of the elements in the relation.
+ * @param relation
+ * The relation.
+ * @param keyOrder
+ * The key order for that index.
+ * @param partitionId
+ * The partition identifier and <code>-1</code> unless running
+ * against an {@link IBigdataFederation}.
+ *
+ * @return The mutable view of the index.
+ *
+ * @throws UnsupportedOperationException
+ * if there is an attempt to read on an index partition when the
+ * database is not an {@link IBigdataFederation} or when the
+ * database is an {@link IBigdataFederation} unless the index
+ * partition was specified.
+ */
+ public <T> ILocalBTreeView getMutableLocalIndexView(
+ final IRelation<T> relation, final IKeyOrder<T> keyOrder,
+ final int partitionId) {
+
+ final String namespace = relation.getNamespace();
+
+ final ILocalBTreeView ndx;
+
+ if (partitionId == -1) {
+
+ if(indexManager instanceof IBigdataFederation<?>)
+ throw new UnsupportedOperationException();
+
+ // The index is not partitioned.
+ ndx = (ILocalBTreeView) indexManager.getIndex(namespace + "."
+ + keyOrder.getIndexName(), getWriteTimestamp());
+
+ } else {
+
+ if(!(indexManager instanceof IBigdataFederation<?>))
+ throw new UnsupportedOperationException();
+
+ // The name of the desired index partition.
+ final String name = DataService.getIndexPartitionName(namespace
+ + "." + keyOrder.getIndexName(), partitionId);
+
+ // MUST be a local index view.
+ ndx = (ILocalBTreeView) indexManager.getIndex(name,
+ getWriteTimestamp());
+
+ }
+
+ return ndx;
+
+ }
+
+ /**
* Obtain an access path reading from relation for the specified predicate
* (from the tail of some rule).
* <p>
@@ -558,5 +651,56 @@
return true;
}
-
+
+/*
+ * I've replaced this with AbstractSplitter for the moment.
+ */
+// /**
+// * Return an iterator visiting the {@link PartitionLocator} for the index
+// * partitions from which an {@link IAccessPath} must read in order to
+// * materialize all elements which would be visited for that predicate.
+// *
+// * @param predicate
+// * The predicate on which the next stage in the pipeline must
+// * read, with whatever bindings already applied. This is used to
+// * discover the shard(s) which span the key range against which
+// * the access path must read.
+// *
+// * @return The iterator.
+// */
+// public Iterator<PartitionLocator> locatorScan(final IPredicate<?> predicate) {
+//
+// final long timestamp = getReadTimestamp();
+//
+// // Note: assumes that we are NOT using a view of two relations.
+// final IRelation<?> relation = (IRelation<?>) fed.getResourceLocator()
+// .locate(predicate.getOnlyRelationName(), timestamp);
+//
+// /*
+// * Find the best access path for the predicate for that relation.
+// *
+// * Note: All we really want is the [fromKey] and [toKey] for that
+// * predicate and index. This MUST NOT layer on expanders since the
+// * layering also hides the [fromKey] and [toKey].
+// */
+// @SuppressWarnings("unchecked")
+// final AccessPath<?> accessPath = (AccessPath<?>) relation
+// .getAccessPath((IPredicate) predicate);
+//
+// // Note: assumes scale-out (EDS or JDS).
+// final IClientIndex ndx = (IClientIndex) accessPath.getIndex();
+//
+// /*
+// * Note: could also be formed from relationName + "." +
+// * keyOrder.getIndexName(), which is cheaper unless the index metadata
+// * is cached.
+// */
+// final String name = ndx.getIndexMetadata().getName();
+//
+// return ((AbstractScaleOutFederation<?>) fed).locatorScan(name,
+// timestamp, accessPath.getFromKey(), accessPath.getToKey(),
+// false/* reverse */);
+//
+// }
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -30,12 +30,15 @@
import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
+import java.util.LinkedList;
+import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.log4j.Logger;
import com.bigdata.bop.BOp.Annotations;
+import com.bigdata.bop.engine.BOpStats;
import com.bigdata.btree.AbstractNode;
import cutthecrap.utils.striterators.Expander;
@@ -462,4 +465,65 @@
}
+
+ /**
+ * Combine chunks drawn from an iterator into a single chunk.
+ *
+ * @param itr
+ * The iterator
+ * @param stats
+ * {@link BOpStats#chunksIn} and {@link BOpStats#unitsIn} are
+ * updated.
+ *
+ * @return A single chunk containing all of the chunks visited by the
+ * iterator.
+ *
+ * @todo unit tests.
+ */
+ static public IBindingSet[] toArray(final Iterator<IBindingSet[]> itr,
+ final BOpStats stats) {
+
+ final List<IBindingSet[]> list = new LinkedList<IBindingSet[]>();
+
+ int nchunks = 0, nelements = 0;
+ {
+
+ while (itr.hasNext()) {
+
+ final IBindingSet[] a = itr.next();
+
+ list.add(a);
+
+ nchunks++;
+
+ nelements += a.length;
+
+ list.add(a);
+
+ }
+
+ stats.chunksIn.add(nchunks);
+ stats.unitsIn.add(nelements);
+
+ }
+
+ if (nchunks == 0) {
+ return new IBindingSet[0];
+ } else if (nchunks == 1) {
+ return list.get(0);
+ } else {
+ int n = 0;
+ final IBindingSet[] a = new IBindingSet[nelements];
+ final Iterator<IBindingSet[]> itr2 = list.iterator();
+ while (itr2.hasNext()) {
+ final IBindingSet[] t = itr2.next();
+ System.arraycopy(t/* src */, 0/* srcPos */, a/* dest */,
+ n/* destPos */, t.length/* length */);
+ n += t.length;
+ }
+ return a;
+ }
+
+ } // toArray()
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Constant.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Constant.java 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Constant.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -23,8 +23,6 @@
*/
package com.bigdata.bop;
-
-
/**
* A constant.
*
Deleted: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ConstantEval.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ConstantEval.java 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ConstantEval.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -1,19 +0,0 @@
-package com.bigdata.bop;
-
-/**
- * Evaluate a constant
- *
- * @author <a href="mailto:tho...@us...">Bryan
- * Thompson</a>
- * @param <E>
- */
-public interface ConstantEval<E> extends BOp {
-
- /**
- * Evaluate a constant.
- *
- * @return The value.
- */
- E eval();
-
-}
\ No newline at end of file
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ComparatorOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ComparatorOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ComparatorOp.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,77 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Sep 4, 2010
+ */
+
+package com.bigdata.bop.aggregation;
+
+import java.util.Comparator;
+import java.util.Map;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpBase;
+import com.bigdata.bop.IBindingSet;
+
+/**
+ * Base class for operators which impose a sort order on binding sets.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+abstract public class ComparatorOp extends BOpBase implements
+ Comparator<IBindingSet> {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ public interface Annotations extends BOp.Annotations {
+
+ /**
+ * An {@link ISortOrder}[] specifying the variables on which the sort
+ * will be imposed and the order (ascending or descending) for each
+ * variable.
+ */
+ String ORDER = ComparatorOp.class.getName() + ".order";
+
+ }
+
+ /**
+ * @param op
+ */
+ public ComparatorOp(BOpBase op) {
+ super(op);
+ }
+
+ /**
+ * @param args
+ * @param annotations
+ */
+ public ComparatorOp(BOp[] args, Map<String, Object> annotations) {
+ super(args, annotations);
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ComparatorOp.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/DistinctBindingSetOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/DistinctBindingSetOp.java 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/DistinctBindingSetOp.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -88,14 +88,6 @@
super(args, annotations);
- final IVariable<?>[] vars = getVariables();
-
- if (vars == null)
- throw new IllegalArgumentException();
-
- if (vars.length == 0)
- throw new IllegalArgumentException();
-
}
/**
@@ -133,7 +125,7 @@
*/
public IVariable<?>[] getVariables() {
- return (IVariable<?>[]) annotations.get(Annotations.VARIABLES);
+ return getRequiredProperty(Annotations.VARIABLES);
}
@@ -208,6 +200,12 @@
this.vars = op.getVariables();
+ if (vars == null)
+ throw new IllegalArgumentException();
+
+ if (vars.length == 0)
+ throw new IllegalArgumentException();
+
this.map = new ConcurrentHashMap<Solution, Solution>(
op.getInitialCapacity(), op.getLoadFactor(),
op.getConcurrencyLevel());
@@ -305,12 +303,13 @@
}
+ sink.flush();
+
// done.
return null;
} finally {
- sink.flush();
sink.close();
// discard the map.
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISlice.java (from rev 3423, branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/ISlice.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISlice.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISlice.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,69 @@
+/*
+
+Copyright (C) SYSTAP, LLC 2006-2008. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+*/
+/*
+ * Created on Sep 24, 2008
+ */
+
+package com.bigdata.bop.aggregation;
+
+import java.io.Serializable;
+
+import com.bigdata.relation.accesspath.IAccessPath;
+
+/**
+ * Indicates the first solution to be returned to the caller (offset) and the
+ * #of solutions to be returned (limit).
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public interface ISlice extends Serializable {
+
+ /**
+ * The first solution to be returned to the caller. A value of ZERO (0)
+ * indicates that all solutions should be returned.
+ */
+ public long getOffset();
+
+ /**
+ * The maximum #of solutions to be returned to the caller. A value of
+ * {@link Long#MAX_VALUE} indicates that there is no limit.
+ *
+ * @todo modify to be consistent with
+ * {@link IAccessPath#iterator(long, long, int)} where a limit of ZERO
+ * (0L) is interpreted as NO limit and a limit of
+ * {@link Long#MAX_VALUE} is interpreted as ZERO (0L) (that is, also
+ * no limit).
+ */
+ public long getLimit();
+
+ /**
+ * The index of the last solution that we will generate (OFFSET + LIMIT). If
+ * OFFSET + LIMIT would be greater than {@link Long#MAX_VALUE}, then use
+ * {@link Long#MAX_VALUE} instead.
+ */
+ public long getLast();
+
+}
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISortOrder.java (from rev 3448, branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/ISortOrder.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISortOrder.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/ISortOrder.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,55 @@
+/*
+
+Copyright (C) SYSTAP, LLC 2006-2008. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+*/
+/*
+ * Created on Sep 24, 2008
+ */
+
+package com.bigdata.bop.aggregation;
+
+import java.io.Serializable;
+
+import com.bigdata.bop.IVariable;
+
+/**
+ * A variable and an order that will be imposed on the values for that variable.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public interface ISortOrder<E> extends Serializable {
+
+ /**
+ * The variable whose values will be sorted.
+ */
+ IVariable<E> getVariable();
+
+ /**
+ * <code>true</code> iff the values will be placed into an ascending sort
+ * and <code>false</code> if the values will be placed into a descending
+ * sort.
+ */
+ boolean isAscending();
+
+}
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/MemorySortOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/MemorySortOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/MemorySortOp.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,112 @@
+package com.bigdata.bop.aggregation;
+
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.FutureTask;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpContext;
+import com.bigdata.bop.BOpUtility;
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.relation.accesspath.IBlockingBuffer;
+
+/**
+ * An in-memory merge sort for binding sets.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id: DistinctElementFilter.java 3466 2010-08-27 14:28:04Z
+ * thompsonbry $
+ *
+ * @todo unit tests.
+ * @todo do an external merge sort operator.
+ */
+public class MemorySortOp extends SortOp {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * Required deep copy constructor.
+ */
+ public MemorySortOp(final MemorySortOp op) {
+ super(op);
+ }
+
+ /**
+ * Required shallow copy constructor.
+ */
+ public MemorySortOp(final BOp[] args,
+ final Map<String, Object> annotations) {
+
+ super(args, annotations);
+
+ }
+
+ public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
+
+ return new FutureTask<Void>(new SortTask(this, context));
+
+ }
+
+ /**
+ * Task executing on the node.
+ */
+ static private class SortTask implements Callable<Void> {
+
+ private final BOpContext<IBindingSet> context;
+
+ /**
+ * The binding set comparator.
+ */
+ private final Comparator<IBindingSet> comparator;
+
+ SortTask(final MemorySortOp op,
+ final BOpContext<IBindingSet> context) {
+
+ this.context = context;
+
+ this.comparator = op.getComparator();
+
+ }
+
+ public Void call() throws Exception {
+
+ final BOpStats stats = context.getStats();
+
+ final IBlockingBuffer<IBindingSet[]> sink = context.getSink();
+
+ try {
+
+ final IBindingSet[] all = BOpUtility.toArray(context
+ .getSource(), stats);
+
+ // sort.
+ Arrays.sort(all, comparator);
+
+ // update counters.
+ stats.unitsOut.add(all.length);
+ stats.chunksOut.increment();
+
+ // write output and flush.
+ sink.add(all);
+ sink.flush();
+
+ // done.
+ return null;
+
+ } finally {
+
+ sink.close();
+
+ }
+
+ }
+
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/MemorySortOp.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/Slice.java (from rev 3423, branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/Slice.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/Slice.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/Slice.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,129 @@
+/*
+
+Copyright (C) SYSTAP, LLC 2006-2008. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+*/
+/*
+ * Created on Sep 24, 2008
+ */
+
+package com.bigdata.bop.aggregation;
+
+import java.io.Externalizable;
+import java.io.IOException;
+import java.io.ObjectInput;
+import java.io.ObjectOutput;
+import java.math.BigInteger;
+
+/**
+ * Default implementation.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class Slice implements ISlice, Externalizable {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 5396509164843609197L;
+
+ private long offset;
+ private long limit;
+ private long last;
+
+ /**
+ * A slice corresponding to all results (offset is zero, limit is
+ * {@link Long#MAX_VALUE}).
+ */
+ public static final transient ISlice ALL = new Slice(0, Long.MAX_VALUE);
+
+ /**
+ *
+ * @param offset
+ * @param limit
+ *
+ * @throws IllegalArgumentException
+ * if offset is negative.
+ * @throws IllegalArgumentException
+ * if limit is non-positive.
+ */
+ public Slice(final long offset, final long limit) {
+
+ if (offset < 0)
+ throw new IllegalArgumentException();
+
+ if (limit <= 0)
+ throw new IllegalArgumentException();
+
+ this.offset = offset;
+
+ this.limit = limit;
+
+ // @todo what is a cheaper way to do this?
+ this.last = BigInteger.valueOf(offset).add(BigInteger.valueOf(limit))
+ .min(BigInteger.valueOf(Long.MAX_VALUE)).longValue();
+
+ }
+
+ public long getOffset() {
+
+ return offset;
+
+ }
+
+ public long getLimit() {
+
+ return limit;
+
+ }
+
+ public long getLast() {
+
+ return last;
+
+ }
+
+ public String toString() {
+
+ return "Slice{offset="+offset+", limit="+limit+", last="+last+"}";
+
+ }
+
+ public void readExternal(ObjectInput in) throws IOException,
+ ClassNotFoundException {
+
+ offset = in.readLong();
+ limit = in.readLong();
+ last = in.readLong();
+
+ }
+
+ public void writeExternal(ObjectOutput out) throws IOException {
+
+ out.writeLong(offset);
+ out.writeLong(limit);
+ out.writeLong(last);
+
+ }
+
+}
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOp.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,86 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Sep 4, 2010
+ */
+
+package com.bigdata.bop.aggregation;
+
+import java.util.Map;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.BindingSetPipelineOp;
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.PipelineOp;
+
+/**
+ * Base class for operators which sort binding sets.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+abstract public class SortOp extends BindingSetPipelineOp {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ public interface Annotations extends BindingSetPipelineOp.Annotations {
+
+ /**
+ * The {@link ComparatorOp} which will impose the ordering on the
+ * binding sets.
+ *
+ * @see ComparatorOp
+ */
+ String COMPARATOR = MemorySortOp.class.getName() + ".comparator";
+
+ }
+
+ /**
+ * @param op
+ */
+ public SortOp(PipelineOp<IBindingSet> op) {
+ super(op);
+ }
+
+ /**
+ * @param args
+ * @param annotations
+ */
+ public SortOp(BOp[] args, Map<String, Object> annotations) {
+ super(args, annotations);
+ }
+
+ /**
+ * @see Annotations#COMPARATOR
+ */
+ public ComparatorOp getComparator() {
+
+ return getRequiredProperty(Annotations.COMPARATOR);
+
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOp.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOrder.java (from rev 3448, branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/SortOrder.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOrder.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SortOrder.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,80 @@
+/*
+
+Copyright (C) SYSTAP, LLC 2006-2008. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+*/
+/*
+ * Created on Sep 24, 2008
+ */
+
+package com.bigdata.bop.aggregation;
+
+import com.bigdata.bop.IVariable;
+
+/**
+ * Default impl.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class SortOrder<E> implements ISortOrder<E> {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = -669873421670514139L;
+
+ private final IVariable<E> var;
+ private final boolean asc;
+
+ /**
+ *
+ * @param var
+ * The variable.
+ * @param asc
+ * <code>true</code> for an ascending sort and
+ * <code>false</code> for a descending sort.
+ */
+ public SortOrder(final IVariable<E> var, final boolean asc) {
+
+ if (var == null)
+ throw new IllegalArgumentException();
+
+ this.var = var;
+
+ this.asc = asc;
+
+ }
+
+ public IVariable<E> getVariable() {
+
+ return var;
+
+ }
+
+ public boolean isAscending() {
+
+ return asc;
+
+ }
+
+}
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SparqlBindingSetComparatorOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SparqlBindingSetComparatorOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SparqlBindingSetComparatorOp.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,101 @@
+package com.bigdata.bop.aggregation;
+
+import java.util.Comparator;
+import java.util.Map;
+
+import org.openrdf.query.algebra.evaluation.util.ValueComparator;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.IVariable;
+
+/**
+ * A comparator for SPARQL binding sets.
+ *
+ * @see http://www.w3.org/TR/rdf-sparql-query/#modOrderBy
+ * @see ValueComparator
+ *
+ * @todo unit tests.
+ */
+public class SparqlBindingSetComparatorOp extends ComparatorOp {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * Required deep copy constructor.
+ */
+ public SparqlBindingSetComparatorOp(final SparqlBindingSetComparatorOp op) {
+ super(op);
+ }
+
+ /**
+ * Required shallow copy constructor.
+ */
+ public SparqlBindingSetComparatorOp(final BOp[] args,
+ final Map<String, Object> annotations) {
+
+ super(args, annotations);
+
+ }
+
+ /**
+ * @see Annotations#ORDER
+ */
+ public ISortOrder<?>[] getOrder() {
+
+ return getRequiredProperty(Annotations.ORDER);
+
+ }
+
+ /**
+ * The sort order to be imposed.
+ */
+ private transient ISortOrder<?>[] order;
+
+ private transient Comparator vc;
+
+ public int compare(final IBindingSet bs1, final IBindingSet bs2) {
+
+ if (order == null) {
+
+ // lazy initialization.
+ order = getOrder();
+
+ if (order == null)
+ throw new IllegalArgumentException();
+
+ if (order.length == 0)
+ throw new IllegalArgumentException();
+
+ // comparator for RDF Value objects.
+ vc = new ValueComparator();
+
+ }
+
+ for (int i = 0; i < order.length; i++) {
+
+ final ISortOrder<?> o = order[i];
+
+ final IVariable v = o.getVariable();
+
+ int ret = vc.compare(bs1.get(v).get(), bs2.get(v).get());
+
+ if (!o.isAscending())
+ ret = -ret;
+
+ if (ret != 0) {
+ // not equal for this variable.
+ return ret;
+ }
+
+ }
+
+ // equal for all variables.
+ return 0;
+
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/SparqlBindingSetComparatorOp.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/package.html
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/package.html 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/package.html 2010-09-05 17:02:34 UTC (rev 3508)
@@ -1,6 +1,6 @@
<html>
<head>
-<title>distinct, sort, and aggregation operators</title>
+<title>solution modifier operators (distinct, sort, slice, and aggregation)</title>
</head>
<body>
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/InsertOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/InsertOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/InsertOp.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,271 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Aug 25, 2010
+ */
+
+package com.bigdata.bop.mutation;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.FutureTask;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpContext;
+import com.bigdata.bop.BindingSetPipelineOp;
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.IVariableOrConstant;
+import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.btree.ILocalBTreeView;
+import com.bigdata.btree.ITupleSerializer;
+import com.bigdata.btree.keys.IKeyBuilder;
+import com.bigdata.relation.IRelation;
+import com.bigdata.relation.accesspath.IAsynchronousIterator;
+import com.bigdata.relation.accesspath.IBlockingBuffer;
+import com.bigdata.striterator.IKeyOrder;
+
+/**
+ * This operator writes elements constructed from binding sets and an orders
+ * list of variables and constants on an index.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ *
+ * @param <E>
+ * The generic type of the elements written onto the index.
+ */
+public class InsertOp<E> extends BindingSetPipelineOp {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ public interface Annotations extends BindingSetPipelineOp.Annotations {
+
+ /**
+ * An ordered {@link IVariableOrConstant}[]. Elements will be created
+ * using the binding sets which flow through the operator and
+ * {@link IRelation#newElement(java.util.List, IBindingSet)}.
+ */
+ String SELECTED = InsertOp.class.getName() + ".selected";
+
+ /**
+ * The namespace of the relation to which the index belongs.
+ */
+ String RELATION = InsertOp.class.getName() + ".relation";
+
+ /**
+ * The {@link IKeyOrder} for the index.
+ */
+ String KEY_ORDER = InsertOp.class.getName() + ".keyOrder";
+
+ }
+
+ /**
+ * Deep copy constructor.
+ *
+ * @param op
+ */
+ public InsertOp(InsertOp<E> op) {
+ super(op);
+ }
+
+ /**
+ * Shallow copy constructor.
+ *
+ * @param args
+ * @param annotations
+ */
+ public InsertOp(BOp[] args, Map<String, Object> annotations) {
+
+ super(args, annotations);
+
+ getRequiredProperty(Annotations.SELECTED);
+
+ }
+
+ /**
+ * @see Annotations#SELECTED
+ */
+ public IVariableOrConstant<?>[] getSelected() {
+
+// return (IVariableOrConstant<?>[]) getProperty(Annotations.SELECTED);
+ return getRequiredProperty(Annotations.SELECTED);
+
+ }
+
+ /**
+ * @see Annotations#RELATION
+ */
+ public String getRelation() {
+
+ return getRequiredProperty(Annotations.RELATION);
+
+ }
+
+ /**
+ * @see Annotations#KEY_ORDER
+ */
+ public IKeyOrder<E> getKeyOrder() {
+
+ return getRequiredProperty(Annotations.KEY_ORDER);
+
+ }
+
+ public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
+
+ return new FutureTask<Void>(new InsertTask<E>(this, context));
+
+ }
+
+ /**
+ * Create elements from the selected bindings and insert them onto the named
+ * index.
+ */
+ static private class InsertTask<E> implements Callable<Void> {
+
+ private final BOpStats stats;
+
+ private final BOpContext<IBindingSet> context;
+
+ private final IAsynchronousIterator<IBindingSet[]> source;
+
+ /**
+ * Only used to close the sink when we are done.
+ */
+ private final IBlockingBuffer<IBindingSet[]> sink;
+
+ private List<IVariableOrConstant<?>> selected;
+
+ private final IRelation<E> relation;
+
+ private final IKeyOrder<E> keyOrder;
+
+ @SuppressWarnings("unchecked")
+ InsertTask(final InsertOp<E> op, final BOpContext<IBindingSet> context) {
+
+ this.context = context;
+
+ stats = context.getStats();
+
+ source = context.getSource();
+
+ sink = context.getSink();
+
+ selected = Arrays.asList(op.getSelected());
+
+ relation = context.getWriteRelation(op.getRelation());
+
+ keyOrder = op.getKeyOrder();
+
+ }
+
+ /**
+ *
+ * @todo This does not order the tuples before writing on the local
+ * index. I am not sure that it should. I think that order is
+ * generally obtained from how we organize the tuples when mapping
+ * them across shards. However, for standalone databases it may
+ * make sense to insert a SORT on the selected attributes before
+ * the INSERT.
+ */
+ public Void call() throws Exception {
+
+ /*
+ * @todo validate for s/o. Since this goes through a common code
+ * path, what we really need to test is getMutableLocalIndexView().
+ * The rest of the insert operation can be tested against a local
+ * Journal.
+ */
+ final ILocalBTreeView ndx = context.getMutableLocalIndexView(
+ relation, keyOrder, context.getPartitionId());
+
+ final IKeyBuilder keyBuilder = ndx.getIndexMetadata()
+ .getKeyBuilder();
+
+ final ITupleSerializer tupleSer = ndx.getIndexMetadata()
+ .getTupleSerializer();
+
+ try {
+
+ while (source.hasNext()) {
+
+ final IBindingSet[] chunk = source.next();
+
+ stats.chunksIn.increment();
+ stats.unitsIn.add(chunk.length);
+
+ int nwritten = 0;
+ for (int i = 0; i < chunk.length; i++) {
+
+ final IBindingSet bset = chunk[i];
+
+ final E e = relation.newElement(selected, bset);
+
+ final byte[] key = keyOrder.getKey(keyBuilder, e);
+
+ if (!ndx.contains(key)) {
+
+ final byte[] val = tupleSer.serializeVal(e);
+
+ ndx.insert(key, val);
+
+ nwritten++;
+
+ }
+
+ }
+
+ if (nwritten > 0) {
+ stats.unitsOut.add(nwritten);
+ stats.chunksOut.increment();
+ }
+
+ }
+
+ return null;
+
+ } finally {
+
+ sink.close();
+
+ }
+
+ }
+
+ }
+
+
+ // E[] a = null;
+ // Note: useful if we will sort before writing on the index.
+// if (i == 0)
+// a = (E[]) java.lang.reflect.Array.newInstance(e
+// .getClass());
+//
+// a[i] = e;
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/InsertOp.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/package.html
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/package.html (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/package.html 2010-09-05 17:02:34 UTC (rev 3508)
@@ -0,0 +1,14 @@
+<html>
+<head>
+<title>mutation operators</title>
+</head>
+<body>
+
+<p>
+
+
+
+</p>
+
+</body>
+</html>
\ No newline at end of file
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/mutation/package.html
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/AbstractRelation.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/AbstractRelation.java 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/AbstractRelation.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -28,8 +28,14 @@
package com.bigdata.relation;
+import java.util.ArrayList;
+import java.util.List;
import java.util.Properties;
import java.util.UUID;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicLong;
import com.bigdata.bop.IPredicate;
import com.bigdata.btree.IIndex;
@@ -43,10 +49,25 @@
import com.bigdata.journal.Journal;
import com.bigdata.journal.TemporaryRawStore;
import com.bigdata.journal.TemporaryStore;
+import com.bigdata.rdf.lexicon.LexiconRelation;
+import com.bigdata.rdf.model.StatementEnum;
+import com.bigdata.rdf.spo.ISPO;
+import com.bigdata.rdf.spo.JustificationRemover;
+import com.bigdata.rdf.spo.SPO;
+import com.bigdata.rdf.spo.SPOAccessPath;
+import com.bigdata.rdf.spo.SPOIndexRemover;
+import com.bigdata.rdf.spo.SPOIndexWriter;
+import com.bigdata.rdf.spo.SPOKeyOrder;
+import com.bigdata.rdf.spo.SPORelation;
+import com.bigdata.rdf.store.AbstractTripleStore;
import com.bigdata.relation.accesspath.AccessPath;
import com.bigdata.relation.accesspath.IAccessPath;
+import com.bigdata.relation.accesspath.IElementFilter;
+import com.bigdata.relation.rule.eval.ISolution;
+import com.bigdata.relation.rule.eval.AbstractSolutionBuffer.InsertSolutionBuffer;
import com.bigdata.service.DataService;
import com.bigdata.service.IBigdataFederation;
+import com.bigdata.striterator.IChunkedOrderedIterator;
import com.bigdata.striterator.IKeyOrder;
/**
@@ -56,11 +77,6 @@
* @version $Id$
* @param <E>
* The generic type of the [E]lements of the relation.
- *
- * @todo It would be interesting to do a GOM relation with its secondary index
- * support and the addition of clustered indices. We would then get
- * efficient JOINs via the rules layer for free and a high-level query
- * language could be mapped onto those JOINs.
*/
abstract public class AbstractRelation<E> extends AbstractResource<IRelation<E>> implements
IMutableRelation<E> {
@@ -323,5 +339,5 @@
getChunkCapacity(), getFullyBufferedReadThreshold()).init();
}
-
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/IMutableRelation.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/IMutableRelation.java 2010-09-03 19:25:46 UTC (rev 3507)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/IMutableRelation.java 2010-09-05 17:02:34 UTC (rev 3508)
@@ -80,71 +80,4 @@
*/
public long delete(IChunkedOrderedIterator<E> itr);
- /*
- * @todo update is notional. it has not been implemented yet (you can use
- * delete+insert). i suspect that the implementation will eventually involve
- * the "transform" being specified as an extension to the rule, e.g.,
- *
- * update [relation] set z=foo from [relation] (x,y,bar)
- *
- * if the update causes a change in the key for either the primary index or
- * any secondary index for the relation then it must be realized as a
- * delete+insert since the changes to the underlying tuples might not be
- * local (e.g., they could be in an different index partition, on a
- * different data service, on even on a different host). such non-local
- * changes will not be atomic unless you use a full transaction or
- * read-behind from a last known consistent commit point.
- *
- * @todo the RDF DB faces exactly this problem with truth maintenance (the
- * problem is somewhat more severe since inference against an unstable KB
- * state can magnify any inconsistencies).
- */
-
-// /**
-// * Update elements on the relation.
-// * <p>
-// * The implemention must locate each element in the relation and, if found,
-// * update its state using the <i>transform</i>. It is an error if the
-// * transformed element has a different "primary key" than the visited
-// * element.
-// * <p>
-// * Note: While UPDATE is often realized as "DELETE + INSERT" within the same
-// * transaction, the advantage of this formulation is that is one-half of the
-// * cost since each element in the relation is visited only once. However, if
-// * you need to update parts of the primary key then "DELETE + INSERT" is the
-// * way to go since each the "delete" and the "insert" may operate on
-// * different tuples located in different parts of the index. For scale-out
-// * indices, those tuples can even lie on different machines.
-// *
-// * @param itr
-// * An iterator visiting the elements selected for update.
-// * @param transform
-// * A transform that produces the new state for each visited
-// * element.
-// *
-// * @return The #of elements that were actually modified in the relation.
-// */
-// public long update(IChunkedOrderedIterator...
[truncated message content] |
|
From: <tho...@us...> - 2010-09-03 19:25:52
|
Revision: 3507
http://bigdata.svn.sourceforge.net/bigdata/?rev=3507&view=rev
Author: thompsonbry
Date: 2010-09-03 19:25:46 +0000 (Fri, 03 Sep 2010)
Log Message:
-----------
Removed an unused public method which had introduced a dependency on jini into the bigdata core module.
Modified Paths:
--------------
trunk/bigdata/src/java/com/bigdata/util/config/NicUtil.java
Modified: trunk/bigdata/src/java/com/bigdata/util/config/NicUtil.java
===================================================================
--- trunk/bigdata/src/java/com/bigdata/util/config/NicUtil.java 2010-09-03 18:33:10 UTC (rev 3506)
+++ trunk/bigdata/src/java/com/bigdata/util/config/NicUtil.java 2010-09-03 19:25:46 UTC (rev 3507)
@@ -26,28 +26,20 @@
package com.bigdata.util.config;
import java.io.IOException;
-import java.net.InetAddress;
import java.net.Inet4Address;
+import java.net.InetAddress;
import java.net.InterfaceAddress;
-import java.net.MalformedURLException;
import java.net.NetworkInterface;
import java.net.SocketException;
import java.net.UnknownHostException;
+import java.util.Collections;
+import java.util.Enumeration;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import java.util.Enumeration;
-import java.util.Collections;
-import java.util.logging.LogRecord;
import org.apache.log4j.Level;
-import org.apache.log4j.Logger;
-import net.jini.config.Configuration;
-import net.jini.config.ConfigurationException;
-import com.sun.jini.config.Config;
-import com.sun.jini.logging.Levels;
-
/**
* Utility class that provides a set of static convenience methods
* related to processing information about the current node's Network
@@ -400,34 +392,34 @@
return macAddr;
}
- /**
- * Three-argument version of <code>getInetAddress</code> that retrieves
- * the desired interface name from the given <code>Configuration</code>
- * parameter.
- */
- public static InetAddress getInetAddress(Configuration config,
- String componentName,
- String nicNameEntry)
- {
- String nicName = "NoNetworkInterfaceName";
- try {
- nicName = (String)Config.getNonNullEntry(config,
- componentName,
- nicNameEntry,
- String.class,
- "eth0");
- } catch(ConfigurationException e) {
- jiniConfigLogger.log(WARNING, e
- +" - [componentName="+componentName
- +", nicNameEntry="+nicNameEntry+"]");
- utilLogger.log(Level.WARN, e
- +" - [componentName="+componentName
- +", nicNameEntry="+nicNameEntry+"]");
- e.printStackTrace();
- return null;
- }
- return ( getInetAddress(nicName, 0, null, false) );
- }
+// /**
+// * Three-argument version of <code>getInetAddress</code> that retrieves
+// * the desired interface name from the given <code>Configuration</code>
+// * parameter.
+// */
+// public static InetAddress getInetAddress(Configuration config,
+// String componentName,
+// String nicNameEntry)
+// {
+// String nicName = "NoNetworkInterfaceName";
+// try {
+// nicName = (String)Config.getNonNullEntry(config,
+// componentName,
+// nicNameEntry,
+// String.class,
+// "eth0");
+// } catch(ConfigurationException e) {
+// jiniConfigLogger.log(WARNING, e
+// +" - [componentName="+componentName
+// +", nicNameEntry="+nicNameEntry+"]");
+// utilLogger.log(Level.WARN, e
+// +" - [componentName="+componentName
+// +", nicNameEntry="+nicNameEntry+"]");
+// e.printStackTrace();
+// return null;
+// }
+// return ( getInetAddress(nicName, 0, null, false) );
+// }
// What follows are a number of versions of the getIpAddress method
// provided for convenience.
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-09-03 18:33:19
|
Revision: 3506
http://bigdata.svn.sourceforge.net/bigdata/?rev=3506&view=rev
Author: thompsonbry
Date: 2010-09-03 18:33:10 +0000 (Fri, 03 Sep 2010)
Log Message:
-----------
Changed IRelation#newElement() to be compatible with construction of new elements (SPOs, etc) from a binding set and an ordered list of variables and constants. The old method signature was basically reusing the IPredicate for this purpose.
Added a method to IVariableOrConstant to permit the retrieval of the "as bound" value without testing to see if the object is a variable or a constant. This should clear up an pattern of conditional logic based on the nature of the object (variable or constant).
Some more clean up of the operator hierarchy.
More notes on how to handle different operator constructs.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BindingSetPipelineOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Constant.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPredicate.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IVariableOrConstant.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Var.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bset/ConditionalRoutingOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ndx/AbstractSampleIndex.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/IRelation.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/RelationFusedView.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/Solution.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/search/FullTextIndex.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/ap/R.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/RunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/relation/locator/TestDefaultResourceLocator.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/lexicon/LexiconRelation.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/magic/MagicRelation.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPORelation.java
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPipelineOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/PipelineOp.java
Removed Paths:
-------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/AbstractPipelineOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/PipelineOp.java
Deleted: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/AbstractPipelineOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/AbstractPipelineOp.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/AbstractPipelineOp.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -1,119 +0,0 @@
-/**
-
-Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
-/*
- * Created on Aug 18, 2010
- */
-
-package com.bigdata.bop;
-
-import java.util.Map;
-import java.util.concurrent.TimeUnit;
-
-import com.bigdata.bop.engine.BOpStats;
-import com.bigdata.relation.accesspath.BlockingBuffer;
-import com.bigdata.relation.accesspath.IBlockingBuffer;
-
-/**
- * Abstract base class for pipelined operators regardless of the type of data
- * moving along the pipeline.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
- */
-abstract public class AbstractPipelineOp<E> extends BOpBase implements
- PipelineOp<E> {
-
- /**
- *
- */
- private static final long serialVersionUID = 1L;
-
- public interface Annotations extends PipelineOp.Annotations {
-
- }
-
- /**
- * Deep copy constructor.
- *
- * @param op
- */
- protected AbstractPipelineOp(final AbstractPipelineOp<E> op) {
-
- super(op);
-
- }
-
- /**
- * Shallow copy constructor.
- *
- * @param args
- * @param annotations
- */
- protected AbstractPipelineOp(final BOp[] args,
- final Map<String, Object> annotations) {
-
- super(args, annotations);
-
- }
-
- public int getChunkCapacity() {
-
- return getProperty(Annotations.CHUNK_CAPACITY,
- Annotations.DEFAULT_CHUNK_CAPACITY);
-
- }
-
- public int getChunkOfChunksCapacity() {
-
- return getProperty(Annotations.CHUNK_OF_CHUNKS_CAPACITY,
- Annotations.DEFAULT_CHUNK_OF_CHUNKS_CAPACITY);
-
- }
-
- public long getChunkTimeout() {
-
- return getProperty(Annotations.CHUNK_TIMEOUT,
- Annotations.DEFAULT_CHUNK_TIMEOUT);
-
- }
-
- /**
- * The {@link TimeUnit}s in which the {@link #chunkTimeout} is measured.
- */
- protected static transient final TimeUnit chunkTimeoutUnit = TimeUnit.MILLISECONDS;
-
- public BOpStats newStats() {
-
- return new BOpStats();
-
- }
-
- public IBlockingBuffer<E[]> newBuffer() {
-
- return new BlockingBuffer<E[]>(getChunkOfChunksCapacity(),
- getChunkCapacity(), getChunkTimeout(), chunkTimeoutUnit);
-
- }
-
-}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpContext.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -465,6 +465,66 @@
}
/**
+ * Copy the values for variables in the predicate from the element, applying
+ * them to the caller's {@link IBindingSet}.
+ *
+ * @param e
+ * The element.
+ * @param pred
+ * The predicate.
+ * @param bindingSet
+ * The binding set, which is modified as a side-effect.
+ */
+ @SuppressWarnings("unchecked")
+ final private void copyValues(final IElement e, final IPredicate<?> pred,
+ final IBindingSet bindingSet) {
+
+ for (int i = 0; i < pred.arity(); i++) {
+
+ final IVariableOrConstant<?> t = pred.get(i);
+
+ if (t.isVar()) {
+
+ final IVariable<?> var = (IVariable<?>) t;
+
+ final Constant<?> newval = new Constant(e.get(i));
+
+ bindingSet.set(var, newval);
+
+ }
+
+ }
+
+ }
+
+ /**
+ * Copy the bound values from the element into a binding set using the
+ * caller's variable names.
+ *
+ * @param vars
+ * The ordered list of variables.
+ * @param e
+ * The element.
+ * @param bindingSet
+ * The binding set, which is modified as a side-effect.
+ */
+ final public void bind(final IVariable<?>[] vars, final IElement e,
+ final IBindingSet bindingSet) {
+
+ for (int i = 0; i < vars.length; i++) {
+
+ final IVariable<?> var = vars[i];
+
+ @SuppressWarnings("unchecked")
+ final Constant<?> newval = new Constant(e.get(i));
+
+ bindingSet.set(var, newval);
+
+ }
+
+ }
+
+ /**
* Check constraints.
*
* @param constraints
@@ -499,26 +559,4 @@
}
- @SuppressWarnings("unchecked")
- final private void copyValues(final IElement e, final IPredicate<?> pred,
- final IBindingSet bindingSet) {
-
- for (int i = 0; i < pred.arity(); i++) {
-
- final IVariableOrConstant<?> t = pred.get(i);
-
- if (t.isVar()) {
-
- final IVariable<?> var = (IVariable<?>) t;
-
- final Constant<?> newval = new Constant(e.get(i));
-
- bindingSet.set(var, newval);
-
- }
-
- }
-
- }
-
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BindingSetPipelineOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BindingSetPipelineOp.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BindingSetPipelineOp.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -37,14 +37,14 @@
* @version $Id$
*/
abstract public class BindingSetPipelineOp extends
- AbstractPipelineOp<IBindingSet> {
+ PipelineOp<IBindingSet> {
/**
*
*/
private static final long serialVersionUID = 1L;
- public interface Annotations extends AbstractPipelineOp.Annotations {
+ public interface Annotations extends PipelineOp.Annotations {
/**
* The value of the annotation is the {@link BOp.Annotations#BOP_ID} of
@@ -61,7 +61,7 @@
*
* @param op
*/
- protected BindingSetPipelineOp(AbstractPipelineOp<IBindingSet> op) {
+ protected BindingSetPipelineOp(PipelineOp<IBindingSet> op) {
super(op);
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Constant.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Constant.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Constant.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -23,7 +23,6 @@
*/
package com.bigdata.bop;
-import java.util.Map;
/**
@@ -153,13 +152,19 @@
//
// }
- public E get() {
+ final public E get() {
return value;
}
- public String getName() {
+ final public E get(final IBindingSet bindingSet) {
+
+ return value;
+
+ }
+
+ final public String getName() {
throw new UnsupportedOperationException();
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPipelineOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPipelineOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPipelineOp.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -0,0 +1,75 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Sep 3, 2010
+ */
+
+package com.bigdata.bop;
+
+import java.util.concurrent.FutureTask;
+
+import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.relation.accesspath.IBlockingBuffer;
+
+/**
+ * An pipeline operator reads from a source and writes on a sink.
+ *
+ * @param <E>
+ * The generic type of the objects processed by the operator.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public interface IPipelineOp<E> extends BOp {
+
+ /**
+ * Return a new object which can be used to collect statistics on the
+ * operator evaluation (this may be overridden to return a more specific
+ * class depending on the operator).
+ */
+ BOpStats newStats();
+
+ /**
+ * Instantiate a buffer suitable as a sink for this operator. The buffer
+ * will be provisioned based on the operator annotations.
+ *
+ * @return The buffer.
+ */
+ IBlockingBuffer<E[]> newBuffer();
+
+ /**
+ * Return a {@link FutureTask} which computes the operator against the
+ * evaluation context. The caller is responsible for executing the
+ * {@link FutureTask} (this gives them the ability to hook the completion of
+ * the computation).
+ *
+ * @param context
+ * The evaluation context.
+ *
+ * @return The {@link FutureTask} which will compute the operator's
+ * evaluation.
+ */
+ FutureTask<Void> eval(BOpContext<E> context);
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPipelineOp.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPredicate.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPredicate.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPredicate.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -30,7 +30,6 @@
import java.io.Serializable;
-import com.bigdata.bop.join.PipelineJoin;
import com.bigdata.mdi.PartitionLocator;
import com.bigdata.relation.IMutableRelation;
import com.bigdata.relation.IRelation;
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IVariableOrConstant.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IVariableOrConstant.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IVariableOrConstant.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -64,6 +64,24 @@
E get();
/**
+ * Return the <i>as bound</i> value of the variable or constant. The <i>as
+ * bound</i> value of an {@link IConstant} is the contant's value. The <i>as
+ * bound</i> value of an {@link IVariable} is the bound value in the given
+ * {@link IBindingSet} -or- <code>null</code> if the variable is not bound
+ * in the {@link IBindingSet}.
+ *
+ * @param bindingSet
+ * The binding set.
+ *
+ * @return The as bound value of the constant or variable.
+ *
+ * @throws IllegalArgumentException
+ * if this is an {@link IVariable} and the <i>bindingSet</i> is
+ * <code>null</code>.
+ */
+ E get(IBindingSet bindingSet);
+
+ /**
* Return the name of a variable.
*
* @throws UnsupportedOperationException
Deleted: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/PipelineOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/PipelineOp.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/PipelineOp.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -1,182 +0,0 @@
-/**
-
-Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
-/*
- * Created on Aug 18, 2010
- */
-
-package com.bigdata.bop;
-
-import java.util.concurrent.FutureTask;
-
-import com.bigdata.bop.engine.BOpStats;
-import com.bigdata.btree.IRangeQuery;
-import com.bigdata.relation.accesspath.AccessPath;
-import com.bigdata.relation.accesspath.BlockingBuffer;
-import com.bigdata.relation.accesspath.IBlockingBuffer;
-import com.bigdata.relation.accesspath.IBuffer;
-
-/**
- * An pipeline operator reads from a source and writes on a sink.
- *
- * @param <E>
- * The generic type of the objects processed by the operator.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
- *
- * @todo It is too confusion to have an interface hierarchy which is separate
- * from the class hierarchy for the operators. Therefore roll this
- * interface into {@link AbstractPipelineOp} and then rename that class to
- * {@link PipelineOp}
- */
-public interface PipelineOp<E> extends BOp {
-
- /**
- * Well known annotations pertaining to the binding set pipeline.
- */
- public interface Annotations extends BOp.Annotations {
-
- /**
- * The maximum #of chunks that can be buffered before an the producer
- * would block (default {@value #DEFAULT_CHUNK_OF_CHUNKS_CAPACITY}).
- * Note that partial chunks may be combined into full chunks whose
- * nominal capacity is specified by {@link #CHUNK_CAPACITY}.
- *
- * @see #DEFAULT_CHUNK_OF_CHUNKS_CAPACITY
- */
- String CHUNK_OF_CHUNKS_CAPACITY = PipelineOp.class.getName()
- + ".chunkOfChunksCapacity";
-
- /**
- * Default for {@link #CHUNK_OF_CHUNKS_CAPACITY}
- *
- * @todo was 100. dialed down to reduce heap consumption for arrays.
- * test performance @ 100 and 1000.
- */
- int DEFAULT_CHUNK_OF_CHUNKS_CAPACITY = 100;
-
- /**
- * Sets the capacity of the {@link IBuffer}s used to accumulate a chunk
- * of {@link IBindingSet}s (default {@value #CHUNK_CAPACITY}). Partial
- * chunks may be automatically combined into full chunks.
- *
- * @see #DEFAULT_CHUNK_CAPACITY
- * @see #CHUNK_OF_CHUNKS_CAPACITY
- */
- String CHUNK_CAPACITY = PipelineOp.class.getName() + ".chunkCapacity";
-
- /**
- * Default for {@link #CHUNK_CAPACITY}
- */
- int DEFAULT_CHUNK_CAPACITY = 100;
-
- /**
- * The timeout in milliseconds that the {@link BlockingBuffer} will wait
- * for another chunk to combine with the current chunk before returning
- * the current chunk (default {@value #DEFAULT_CHUNK_TIMEOUT}). This may
- * be ZERO (0) to disable the chunk combiner.
- *
- * @see #DEFAULT_CHUNK_TIMEOUT
- */
- String CHUNK_TIMEOUT = PipelineOp.class.getName() + ".chunkTimeout";
-
- /**
- * The default for {@link #CHUNK_TIMEOUT}.
- *
- * @todo this is probably much larger than we want. Try 10ms.
- */
- int DEFAULT_CHUNK_TIMEOUT = 1000;
-
- /**
- * If the estimated rangeCount for an {@link AccessPath#iterator()} is
- * LTE this threshold then use a fully buffered (synchronous) iterator.
- * Otherwise use an asynchronous iterator whose capacity is governed by
- * {@link #CHUNK_OF_CHUNKS_CAPACITY}.
- *
- * @see #DEFAULT_FULLY_BUFFERED_READ_THRESHOLD
- */
- String FULLY_BUFFERED_READ_THRESHOLD = PipelineOp.class.getName()
- + ".fullyBufferedReadThreshold";
-
- /**
- * Default for {@link #FULLY_BUFFERED_READ_THRESHOLD}.
- *
- * @todo try something closer to the branching factor, e.g., 100.
- */
- int DEFAULT_FULLY_BUFFERED_READ_THRESHOLD = 1000;
-
- /**
- * Flags for the iterator ({@link IRangeQuery#KEYS},
- * {@link IRangeQuery#VALS}, {@link IRangeQuery#PARALLEL}).
- * <p>
- * Note: The {@link IRangeQuery#PARALLEL} flag here is an indication
- * that the iterator may run in parallel across the index partitions.
- * This only effects scale-out and only for simple triple patterns since
- * the pipeline join does something different (it runs inside the index
- * partition using the local index, not the client's view of a
- * distributed index).
- *
- * @see #DEFAULT_FLAGS
- */
- String FLAGS = PipelineOp.class.getName() + ".flags";
-
- /**
- * The default flags will visit the keys and values of the non-deleted
- * tuples and allows parallelism in the iterator (when supported).
- */
- final int DEFAULT_FLAGS = IRangeQuery.KEYS | IRangeQuery.VALS
- | IRangeQuery.PARALLEL;
-
- }
-
- /**
- * Return a new object which can be used to collect statistics on the
- * operator evaluation (this may be overridden to return a more specific
- * class depending on the operator).
- */
- BOpStats newStats();
-
- /**
- * Instantiate a buffer suitable as a sink for this operator. The buffer
- * will be provisioned based on the operator annotations.
- *
- * @return The buffer.
- */
- IBlockingBuffer<E[]> newBuffer();
-
- /**
- * Return a {@link FutureTask} which computes the operator against the
- * evaluation context. The caller is responsible for executing the
- * {@link FutureTask} (this gives them the ability to hook the completion of
- * the computation).
- *
- * @param context
- * The evaluation context.
- *
- * @return The {@link FutureTask} which will compute the operator's
- * evaluation.
- */
- FutureTask<Void> eval(BOpContext<E> context);
-
-}
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/PipelineOp.java (from rev 3495, branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/AbstractPipelineOp.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/PipelineOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/PipelineOp.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -0,0 +1,219 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Aug 18, 2010
+ */
+
+package com.bigdata.bop;
+
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.btree.IRangeQuery;
+import com.bigdata.relation.accesspath.AccessPath;
+import com.bigdata.relation.accesspath.BlockingBuffer;
+import com.bigdata.relation.accesspath.IBlockingBuffer;
+import com.bigdata.relation.accesspath.IBuffer;
+
+/**
+ * An pipeline operator reads from a source and writes on a sink. This is an
+ * abstract base class for pipelined operators regardless of the type of data
+ * moving along the pipeline.
+ *
+ * @param <E>
+ * The generic type of the objects processed by the operator.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+abstract public class PipelineOp<E> extends BOpBase implements IPipelineOp<E> {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * Well known annotations pertaining to the binding set pipeline.
+ */
+ public interface Annotations extends BOp.Annotations {
+
+ /**
+ * The maximum #of chunks that can be buffered before an the producer
+ * would block (default {@value #DEFAULT_CHUNK_OF_CHUNKS_CAPACITY}).
+ * Note that partial chunks may be combined into full chunks whose
+ * nominal capacity is specified by {@link #CHUNK_CAPACITY}.
+ *
+ * @see #DEFAULT_CHUNK_OF_CHUNKS_CAPACITY
+ */
+ String CHUNK_OF_CHUNKS_CAPACITY = PipelineOp.class.getName()
+ + ".chunkOfChunksCapacity";
+
+ /**
+ * Default for {@link #CHUNK_OF_CHUNKS_CAPACITY}
+ *
+ * @todo was 100. dialed down to reduce heap consumption for arrays.
+ * test performance @ 100 and 1000.
+ */
+ int DEFAULT_CHUNK_OF_CHUNKS_CAPACITY = 100;
+
+ /**
+ * Sets the capacity of the {@link IBuffer}s used to accumulate a chunk
+ * of {@link IBindingSet}s (default {@value #CHUNK_CAPACITY}). Partial
+ * chunks may be automatically combined into full chunks.
+ *
+ * @see #DEFAULT_CHUNK_CAPACITY
+ * @see #CHUNK_OF_CHUNKS_CAPACITY
+ */
+ String CHUNK_CAPACITY = PipelineOp.class.getName() + ".chunkCapacity";
+
+ /**
+ * Default for {@link #CHUNK_CAPACITY}
+ */
+ int DEFAULT_CHUNK_CAPACITY = 100;
+
+ /**
+ * The timeout in milliseconds that the {@link BlockingBuffer} will wait
+ * for another chunk to combine with the current chunk before returning
+ * the current chunk (default {@value #DEFAULT_CHUNK_TIMEOUT}). This may
+ * be ZERO (0) to disable the chunk combiner.
+ *
+ * @see #DEFAULT_CHUNK_TIMEOUT
+ */
+ String CHUNK_TIMEOUT = PipelineOp.class.getName() + ".chunkTimeout";
+
+ /**
+ * The default for {@link #CHUNK_TIMEOUT}.
+ *
+ * @todo this is probably much larger than we want. Try 10ms.
+ */
+ int DEFAULT_CHUNK_TIMEOUT = 1000;
+
+ /**
+ * If the estimated rangeCount for an {@link AccessPath#iterator()} is
+ * LTE this threshold then use a fully buffered (synchronous) iterator.
+ * Otherwise use an asynchronous iterator whose capacity is governed by
+ * {@link #CHUNK_OF_CHUNKS_CAPACITY}.
+ *
+ * @see #DEFAULT_FULLY_BUFFERED_READ_THRESHOLD
+ */
+ String FULLY_BUFFERED_READ_THRESHOLD = PipelineOp.class.getName()
+ + ".fullyBufferedReadThreshold";
+
+ /**
+ * Default for {@link #FULLY_BUFFERED_READ_THRESHOLD}.
+ *
+ * @todo try something closer to the branching factor, e.g., 100.
+ */
+ int DEFAULT_FULLY_BUFFERED_READ_THRESHOLD = 1000;
+
+ /**
+ * Flags for the iterator ({@link IRangeQuery#KEYS},
+ * {@link IRangeQuery#VALS}, {@link IRangeQuery#PARALLEL}).
+ * <p>
+ * Note: The {@link IRangeQuery#PARALLEL} flag here is an indication
+ * that the iterator may run in parallel across the index partitions.
+ * This only effects scale-out and only for simple triple patterns since
+ * the pipeline join does something different (it runs inside the index
+ * partition using the local index, not the client's view of a
+ * distributed index).
+ *
+ * @see #DEFAULT_FLAGS
+ */
+ String FLAGS = PipelineOp.class.getName() + ".flags";
+
+ /**
+ * The default flags will visit the keys and values of the non-deleted
+ * tuples and allows parallelism in the iterator (when supported).
+ */
+ final int DEFAULT_FLAGS = IRangeQuery.KEYS | IRangeQuery.VALS
+ | IRangeQuery.PARALLEL;
+
+ }
+
+ /**
+ * Deep copy constructor.
+ *
+ * @param op
+ */
+ protected PipelineOp(final PipelineOp<E> op) {
+
+ super(op);
+
+ }
+
+ /**
+ * Shallow copy constructor.
+ *
+ * @param args
+ * @param annotations
+ */
+ protected PipelineOp(final BOp[] args,
+ final Map<String, Object> annotations) {
+
+ super(args, annotations);
+
+ }
+
+ public int getChunkCapacity() {
+
+ return getProperty(Annotations.CHUNK_CAPACITY,
+ Annotations.DEFAULT_CHUNK_CAPACITY);
+
+ }
+
+ public int getChunkOfChunksCapacity() {
+
+ return getProperty(Annotations.CHUNK_OF_CHUNKS_CAPACITY,
+ Annotations.DEFAULT_CHUNK_OF_CHUNKS_CAPACITY);
+
+ }
+
+ public long getChunkTimeout() {
+
+ return getProperty(Annotations.CHUNK_TIMEOUT,
+ Annotations.DEFAULT_CHUNK_TIMEOUT);
+
+ }
+
+ /**
+ * The {@link TimeUnit}s in which the {@link #chunkTimeout} is measured.
+ */
+ protected static transient final TimeUnit chunkTimeoutUnit = TimeUnit.MILLISECONDS;
+
+ public BOpStats newStats() {
+
+ return new BOpStats();
+
+ }
+
+ public IBlockingBuffer<E[]> newBuffer() {
+
+ return new BlockingBuffer<E[]>(getChunkOfChunksCapacity(),
+ getChunkCapacity(), getChunkTimeout(), chunkTimeoutUnit);
+
+ }
+
+}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Var.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Var.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/Var.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -139,6 +139,18 @@
}
+ public E get(final IBindingSet bindingSet) {
+
+ if (bindingSet == null)
+ throw new IllegalArgumentException();
+
+ @SuppressWarnings("unchecked")
+ final IConstant<E> c = bindingSet.get(this);
+
+ return c == null ? null : c.get();
+
+ }
+
public String getName() {
return name;
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -309,24 +309,26 @@
public Object asBound(final int index, final IBindingSet bindingSet) {
- if (bindingSet == null)
- throw new IllegalArgumentException();
+ return get(index).get(bindingSet);
- final IVariableOrConstant<?> t = get(index);
+// if (bindingSet == null)
+// throw new IllegalArgumentException();
+//
+// final IVariableOrConstant<?> t = get(index);
+//
+// final IConstant<?> c;
+// if (t.isVar()) {
+//
+// c = bindingSet.get((IVariable<?>) t);
+//
+// } else {
+//
+// c = (IConstant<?>) t;
+//
+// }
+//
+// return c == null ? null : c.get();
- final IConstant<?> c;
- if (t.isVar()) {
-
- c = bindingSet.get((IVariable<?>) t);
-
- } else {
-
- c = (IConstant<?>) t;
-
- }
-
- return c == null ? null : c.get();
-
}
@SuppressWarnings("unchecked")
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bset/ConditionalRoutingOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bset/ConditionalRoutingOp.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bset/ConditionalRoutingOp.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -32,7 +32,7 @@
import java.util.concurrent.Callable;
import java.util.concurrent.FutureTask;
-import com.bigdata.bop.AbstractPipelineOp;
+import com.bigdata.bop.PipelineOp;
import com.bigdata.bop.BOp;
import com.bigdata.bop.BOpContext;
import com.bigdata.bop.BindingSetPipelineOp;
@@ -62,7 +62,7 @@
*/
private static final long serialVersionUID = 1L;
- public interface Annotations extends AbstractPipelineOp.Annotations {
+ public interface Annotations extends PipelineOp.Annotations {
/**
* An {@link IConstraint} which specifies the condition. When the
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ndx/AbstractSampleIndex.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ndx/AbstractSampleIndex.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ndx/AbstractSampleIndex.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -28,7 +28,7 @@
package com.bigdata.bop.ndx;
-import com.bigdata.bop.AbstractPipelineOp;
+import com.bigdata.bop.PipelineOp;
import com.bigdata.bop.BOp;
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.IPredicate;
@@ -59,7 +59,7 @@
* @todo This needs to operation on element chunks, not {@link IBindingSet}
* chunks. It also may not require pipelining.
*/
-abstract public class AbstractSampleIndex<E> extends AbstractPipelineOp<E> {
+abstract public class AbstractSampleIndex<E> extends PipelineOp<E> {
/**
*
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/IRelation.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/IRelation.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/IRelation.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -28,11 +28,13 @@
package com.bigdata.relation;
+import java.util.List;
import java.util.Set;
import java.util.concurrent.ExecutorService;
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.IPredicate;
+import com.bigdata.bop.IVariableOrConstant;
import com.bigdata.btree.IIndex;
import com.bigdata.btree.ITuple;
import com.bigdata.btree.ITupleIterator;
@@ -40,8 +42,6 @@
import com.bigdata.relation.accesspath.IAccessPath;
import com.bigdata.relation.accesspath.IElementFilter;
import com.bigdata.relation.locator.ILocatableResource;
-import com.bigdata.relation.rule.IRule;
-import com.bigdata.relation.rule.eval.ISolution;
import com.bigdata.striterator.IKeyOrder;
/**
@@ -75,25 +75,45 @@
*/
Class<E> getElementClass();
+// /**
+// * Create and return a new element. The element is constructed from the
+// * predicate given the bindings. Typically, this is used when generating an
+// * {@link ISolution} for an {@link IRule} during either a query or mutation
+// * operations. The element is NOT inserted into the relation.
+// *
+// * @param predicate
+// * The predicate that is the head of some {@link IRule}.
+// * @param bindingSet
+// * A set of bindings for that {@link IRule}.
+// *
+// * @return The new element.
+// *
+// * @throws IllegalArgumentException
+// * if any parameter is <code>null</code>.
+// * @throws IllegalStateException
+// * if the predicate is not fully bound given those bindings.
+// */
+// E newElement(IPredicate<E> predicate, IBindingSet bindingSet);
+
/**
* Create and return a new element. The element is constructed from the
- * predicate given the bindings. Typically, this is used when generating an
- * {@link ISolution} for an {@link IRule} during either a query or mutation
- * operations. The element is NOT inserted into the relation.
+ * ordered list of variables and constants. Variables are replaced using the
+ * given the bindings. The element is NOT inserted into the relation.
*
- * @param predicate
- * The predicate that is the head of some {@link IRule}.
+ * @param a
+ * An ordered list of variables and/or constants.
* @param bindingSet
- * A set of bindings for that {@link IRule}.
+ * A set of bindings.
*
* @return The new element.
*
* @throws IllegalArgumentException
* if any parameter is <code>null</code>.
* @throws IllegalStateException
- * if the predicate is not fully bound given those bindings.
+ * if there exists a variable which is not bound given those
+ * bindings.
*/
- E newElement(IPredicate<E> predicate, IBindingSet bindingSet);
+ E newElement(List<IVariableOrConstant<?>> a, IBindingSet bindingSet);
/**
* Return the {@link IKeyOrder} for the primary index for the relation.
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/RelationFusedView.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/RelationFusedView.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/RelationFusedView.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -1,12 +1,14 @@
package com.bigdata.relation;
import java.util.HashSet;
+import java.util.List;
import java.util.Set;
import java.util.concurrent.ExecutorService;
import com.bigdata.bop.BOp;
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.IPredicate;
+import com.bigdata.bop.IVariableOrConstant;
import com.bigdata.btree.IIndex;
import com.bigdata.journal.IIndexManager;
import com.bigdata.journal.TemporaryStore;
@@ -122,10 +124,17 @@
}
- public E newElement(final IPredicate<E> predicate,
+// public E newElement(final IPredicate<E> predicate,
+// final IBindingSet bindingSet) {
+//
+// return relation1.newElement(predicate, bindingSet);
+//
+// }
+
+ public E newElement(final List<IVariableOrConstant<?>> a,
final IBindingSet bindingSet) {
- return relation1.newElement(predicate, bindingSet);
+ return relation1.newElement(a, bindingSet);
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/Solution.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/Solution.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/eval/Solution.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -101,9 +101,12 @@
// the relation for the head of the rule.
final IRelation relation = joinNexus.getHeadRelationView(head);
+
+ // use the relation's element factory.
+ this.e = (E) relation.newElement(head.args(), bindingSet);
- // use the relation's element factory.
- this.e = (E) relation.newElement(head, bindingSet);
+// // use the relation's element factory.
+// this.e = (E) relation.newElement(head, bindingSet);
} else {
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/search/FullTextIndex.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/search/FullTextIndex.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/search/FullTextIndex.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -66,6 +66,7 @@
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.IPredicate;
+import com.bigdata.bop.IVariableOrConstant;
import com.bigdata.btree.BytesUtil;
import com.bigdata.btree.IIndex;
import com.bigdata.btree.ISimpleSplitHandler;
@@ -1354,7 +1355,7 @@
}
@SuppressWarnings("unchecked")
- public Object newElement(IPredicate predicate,
+ public Object newElement(List a,
IBindingSet bindingSet) {
throw new UnsupportedOperationException();
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/ap/R.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/ap/R.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/ap/R.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -29,19 +29,20 @@
import java.util.Comparator;
import java.util.HashSet;
+import java.util.List;
import java.util.Properties;
import java.util.Set;
import java.util.UUID;
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.IPredicate;
+import com.bigdata.bop.IVariableOrConstant;
import com.bigdata.btree.BytesUtil;
import com.bigdata.btree.IIndex;
import com.bigdata.btree.IndexMetadata;
import com.bigdata.btree.keys.IKeyBuilder;
import com.bigdata.journal.IIndexManager;
import com.bigdata.relation.AbstractRelation;
-import com.bigdata.relation.IMutableRelation;
import com.bigdata.relation.locator.ILocatableResource;
import com.bigdata.striterator.AbstractKeyOrder;
import com.bigdata.striterator.IChunkedOrderedIterator;
@@ -52,7 +53,7 @@
* <p>
* Note: This has to be public in order to be an {@link ILocatableResource}.
*/
-public class R extends AbstractRelation<E> implements IMutableRelation<E> {
+public class R extends AbstractRelation<E> {
/**
* Metadata about the index orders for this relation.
@@ -134,16 +135,27 @@
}
- public E newElement(final IPredicate<E> predicate,
+// public E newElement(final IPredicate<E> predicate,
+// final IBindingSet bindingSet) {
+//
+// final String name = (String) predicate.asBound(0, bindingSet);
+//
+// final String value = (String) predicate.asBound(1, bindingSet);
+//
+// return new E(name, value);
+// }
+
+ public E newElement(final List<IVariableOrConstant<?>> a,
final IBindingSet bindingSet) {
- final String name = (String) predicate.asBound(0, bindingSet);
+ final String name = (String) a.get(0).get(bindingSet);
- final String value = (String) predicate.asBound(1, bindingSet);
-
- return new E(name, value);
+ final String value = (String) a.get(1).get(bindingSet);
+
+ return new E(name,value);
+
}
-
+
public Set<String> getIndexNames() {
final Set<String> tmp = new HashSet<String>();
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/RunningQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/RunningQuery.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/RunningQuery.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -52,17 +52,25 @@
import com.bigdata.bop.BOpContext;
import com.bigdata.bop.BOpUtility;
import com.bigdata.bop.BindingSetPipelineOp;
+import com.bigdata.bop.ChunkedOrderedIteratorOp;
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.IConstraint;
import com.bigdata.bop.IPredicate;
import com.bigdata.bop.NoSuchBOpException;
import com.bigdata.bop.ap.Predicate;
import com.bigdata.bop.bset.Union;
+import com.bigdata.relation.IMutableRelation;
+import com.bigdata.relation.IRelation;
import com.bigdata.relation.accesspath.BlockingBuffer;
import com.bigdata.relation.accesspath.IAsynchronousIterator;
import com.bigdata.relation.accesspath.IBlockingBuffer;
import com.bigdata.relation.accesspath.IElementFilter;
+import com.bigdata.relation.rule.Program;
import com.bigdata.resources.ResourceManager;
+import com.bigdata.service.ndx.IAsynchronousWriteBufferFactory;
+import com.bigdata.striterator.ChunkedArrayIterator;
+import com.bigdata.striterator.ChunkedOrderedStriterator;
+import com.bigdata.striterator.IChunkedOrderedIterator;
import com.bigdata.striterator.ICloseableIterator;
import com.bigdata.util.concurrent.Haltable;
@@ -193,15 +201,70 @@
*
* FIXME Unit tests for non-distinct {@link IElementFilter}s on an
* {@link IPredicate}, unit tests for distinct element filter on an
- * {@link IPredicate} which is capable of distributed operations
+ * {@link IPredicate} which is capable of distributed operations. Do not use
+ * distinct where not required (SPOC, only one graph, etc).
+ * <p>
+ * It seems like the right way to approach this is by unifying the stackable
+ * CTC striterator pattern with the chunked iterator pattern and passing the
+ * query engine (or the bop context) into the iterator construction process
+ * (or simply requesting that the query engine construct the iterator
+ * stack).
+ * <p>
+ * In terms of harmonization, it is difficult to say which way would work
+ * better. In the short term we could simply allow both and mask the
+ * differences in how we construct the filters, but the conversion to/from
+ * striterators and chunked iterators seems to waste a bit of effort.
+ * <p>
+ * The trickiest part of all of this is to allow a distributed filter
+ * pattern where the filter gets created on a set of nodes identified by the
+ * operator and the elements move among those nodes using the query engine's
+ * buffers.
+ * <p>
+ * To actually implement the distributed distinct filter we need to stack
+ * the following:
*
+ * <pre>
+ * - ITupleIterator
+ * - Resolve ITuple to Element (e.g., SPOC).
+ * - Layer on optional IElementFilter associated with the IPredicate.
+ * - Layer on SameVariableConstraint iff required (done by AccessPath)
+ * - Resolve SPO to SPO, stripping off the context position.
+ * - Chunk SPOs (SPO[], IKeyOrder), where the key order is from the access path.
+ * - Filter SPO[] using DHT constructed on specified nodes of the cluster.
+ * The SPO[] chunks should be packaged into NIO buffers and shipped to those
+ * nodes. The results should be shipped back as a bit vectors packaged into
+ * a NIO buffers.
+ * - Dechunk SPO[] to SPO since that is the current expectation for the filter
+ * stack.
+ * - The result then gets wrapped as a {@link IChunkedOrderedIterator} by
+ * the AccessPath using a {@link ChunkedArrayIterator}.
+ * </pre>
+ *
+ * This stack is a bit complex(!). But it is certainly easy enough to
+ * generate the necessary bits programmatically.
+ *
* FIXME Handling the {@link Union} of binding sets.
*
- * FIXME conditional routing for binding sets in the pipeline (to route
- * around an optional join group based on an {@link IConstraint}). This
- * should probably wrap the {@link BindingSetPipelineOp} such that we simply
- * stream the grouped operator.
+ * FIXME INSERT and DELETE which will construct elements using
+ * {@link IRelation#newElement(java.util.List, IBindingSet)} from a binding
+ * set and then use {@link IMutableRelation#insert(IChunkedOrderedIterator)}
+ * and {@link IMutableRelation#delete(IChunkedOrderedIterator)}. For s/o, we
+ * first need to move the bits into the right places so it makes sense to
+ * unpack the processing of the loop over the elements and move the data
+ * around, writing on each index as necessary. There could be eventually
+ * consistent approaches to this as well. For justifications we need to
+ * update some additional indices, in which case we are stuck going through
+ * {@link IRelation} rather than routing data directly or using the
+ * {@link IAsynchronousWriteBufferFactory}.
*
+ * FIXME Handle {@link Program}s. There are three flavors, which should
+ * probably be broken into three operators: sequence(ops), set(ops), and
+ * closure(op). The 'set' version would be parallelized, or at least have an
+ * annotation for parallel evaluation. These things belong in the same broad
+ * category as the join graph since they are operators which control the
+ * evaluation of other operators (the current pipeline join also has that
+ * characteristic which it uses to do the nested index subqueries).
+ *
* FIXME SPARQL to BOP translation
*
* FIXME buffer management for s/o, including binding sets movement, element
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/relation/locator/TestDefaultResourceLocator.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/relation/locator/TestDefaultResourceLocator.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/relation/locator/TestDefaultResourceLocator.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -28,6 +28,7 @@
package com.bigdata.relation.locator;
+import java.util.List;
import java.util.Properties;
import java.util.Set;
import java.util.UUID;
@@ -38,6 +39,7 @@
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.IPredicate;
+import com.bigdata.bop.IVariableOrConstant;
import com.bigdata.btree.IIndex;
import com.bigdata.btree.IndexMetadata;
import com.bigdata.journal.BufferMode;
@@ -227,7 +229,8 @@
private static class MockRelation extends AbstractRelation {
- final private String indexName = "foo";
+ static final private String indexName = "foo";
+
private IIndex ndx;
/**
@@ -324,11 +327,16 @@
return null;
}
- public Object newElement(IPredicate predicate, IBindingSet bindingSet) {
+// public Object newElement(IPredicate predicate, IBindingSet bindingSet) {
+// // TODO Auto-generated method stub
+// return null;
+// }
+
+ public Object newElement(List a, IBindingSet bindingSet) {
// TODO Auto-generated method stub
return null;
}
-
+
public Class<ISPO> getElementClass() {
return null;
@@ -344,7 +352,7 @@
// TODO Auto-generated method stub
return null;
}
-
+
}
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/lexicon/LexiconRelation.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/lexicon/LexiconRelation.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/lexicon/LexiconRelation.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -56,6 +56,7 @@
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.IPredicate;
+import com.bigdata.bop.IVariableOrConstant;
import com.bigdata.btree.BytesUtil;
import com.bigdata.btree.IIndex;
import com.bigdata.btree.IRangeQuery;
@@ -804,13 +805,17 @@
*
* @throws UnsupportedOperationException
*/
- public BigdataValue newElement(IPredicate<BigdataValue> predicate,
+// public BigdataValue newElement(IPredicate<BigdataValue> predicate,
+// IBindingSet bindingSet) {
+//
+// throw new UnsupportedOperationException();
+//
+// }
+ public BigdataValue newElement(List<IVariableOrConstant<?>> a,
IBindingSet bindingSet) {
-
throw new UnsupportedOperationException();
-
}
-
+
public Class<BigdataValue> getElementClass() {
return BigdataValue.class;
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/magic/MagicRelation.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/magic/MagicRelation.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/magic/MagicRelation.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -2,6 +2,7 @@
import java.util.ArrayList;
import java.util.HashSet;
+import java.util.Iterator;
import java.util.List;
import java.util.Properties;
import java.util.Set;
@@ -13,9 +14,7 @@
import org.apache.log4j.Logger;
import com.bigdata.bop.IBindingSet;
-import com.bigdata.bop.IConstant;
import com.bigdata.bop.IPredicate;
-import com.bigdata.bop.IVariable;
import com.bigdata.bop.IVariableOrConstant;
import com.bigdata.bop.Var;
import com.bigdata.btree.BloomFilterFactory;
@@ -473,18 +472,44 @@
return indexNames;
}
- public IMagicTuple newElement(IPredicate<IMagicTuple> predicate,
- IBindingSet bindingSet) {
+// public IMagicTuple newElement(IPredicate<IMagicTuple> predicate,
+// IBindingSet bindingSet) {
+//
+// if (predicate == null)
+// throw new IllegalArgumentException();
+//
+// if (bindingSet == null)
+// throw new IllegalArgumentException();
+//
+// final IV[] terms = new IV[arity];
+// for (int i = 0; i < arity; i++) {
+// terms[i] = asBound(predicate, i, bindingSet);
+// }
+//
+// final MagicTuple magicTuple = new MagicTuple(terms);
+//
+// return magicTuple;
+//
+// }
- if (predicate == null)
+ @SuppressWarnings("unchecked")
+ public IMagicTuple newElement(final List<IVariableOrConstant<?>> a,
+ final IBindingSet bindingSet) {
+
+ if (a == null)
throw new IllegalArgumentException();
if (bindingSet == null)
throw new IllegalArgumentException();
final IV[] terms = new IV[arity];
+
+ final Iterator<IVariableOrConstant<?>> itr = a.iterator();
+
for (int i = 0; i < arity; i++) {
- terms[i] = asBound(predicate, i, bindingSet);
+
+ terms[i] = (IV) itr.next().get(bindingSet);
+
}
final MagicTuple magicTuple = new MagicTuple(terms);
@@ -492,39 +517,39 @@
return magicTuple;
}
-
+
public Class<IMagicTuple> getElementClass() {
return IMagicTuple.class;
}
- /**
- * Extract the bound value from the predicate. When the predicate is not
- * bound at that index, then extract its binding from the binding set.
- *
- * @param pred
- * The predicate.
- * @param index
- * The index into that predicate.
- * @param bindingSet
- * The binding set.
- *
- * @return The bound value.
- */
- private IV asBound(final IPredicate<IMagicTuple> predicate,
- final int index, final IBindingSet bindingSet) {
+// /**
+// * Extract the bound value from the predicate. When the predicate is not
+// * bound at that index, then extract its binding from the binding set.
+// *
+// * @param pred
+// * The predicate.
+// * @param index
+// * The index into that predicate.
+// * @param bindingSet
+// * The binding set.
+// *
+// * @return The bound value.
+// */
+// private IV asBound(final IPredicate<IMagicTuple> predicate,
+// final int index, final IBindingSet bindingSet) {
+//
+// final IVariableOrConstant<IV> t = predicate.get(index);
+// final IConstant<IV> c;
+// if (t.isVar()) {
+// c = bindingSet.get((IVariable) t);
+// } else {
+// c = (IConstant<IV>) t;
+// }
+//
+// return c.get();
+//
+// }
- final IVariableOrConstant<IV> t = predicate.get(index);
- final IConstant<IV> c;
- if (t.isVar()) {
- c = bindingSet.get((IVariable) t);
- } else {
- c = (IConstant<IV>) t;
- }
-
- return c.get();
-
- }
-
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPORelation.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPORelation.java 2010-09-03 13:40:22 UTC (rev 3505)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPORelation.java 2010-09-03 18:33:10 UTC (rev 3506)
@@ -1494,32 +1494,69 @@
}
+// /**
+// * @todo This implementation was written early on and works for creating new
+// * SPOs licensed by inference against a triple store. It does not
+// * allow us to specify the statement type, which is always set to
+// * [inferred]. It also does not capture the context if one exists, but
+// * that could be done by inspection of the arity of the predicate. It
+// * might be better to have an explicit "CONSTRUCT" operator rather
+// * than having this implicit relationship between the head of a rule
+// * and the element created from that rule. For example, that might let
+// * us capture the distinction of inferred versus explicit within the
+// * CONSTRUCT operator.
+// */
+// public SPO newElement(final IPredicate<ISPO> predicate,
+// final IBindingSet bindingSet) {
+//
+// if (predicate == null)
+// throw new IllegalArgumentException();
+//
+// if (bindingSet == null)
+// throw new IllegalArgumentException();
+//
+// final IV s = (IV) predicate.asBound(0, bindingSet);
+//
+// final IV p = (IV) predicate.asBound(1, bindingSet);
+//
+// final IV o = (IV) predicate.asBound(2, bindingSet);
+//
+// final SPO spo = new SPO(s, p, o, StatementEnum.Inferred);
+//
+// if(log.isDebugEnabled())
+// log.debug(spo.toString());
+//
+// return spo;
+//
+// }
+
/**
- * @todo This implementation was written early on and works for creating new
- * SPOs licensed by inference against a triple store. It does not
- * allow us to specify the statement type, which is always set to
- * [inferred]. It also does not capture the context if one exists, but
- * that could be done by inspection of the arity of the predicate. It
- * might be better to have an explicit "CONSTRUCT" operator rather
- * than having this implicit relationship between the head of a rule
- * and the element created from that rule. For example, that might let
- * us capture the distinction of inferred versus explicit within the
- * CONSTRUCT operator.
+ * @todo This works for creating new SPOs licensed by inference against a
+ * triple store. However, it does not allow us to specify the
+ * statement type, which is always set to [inferred]. It also does not
+ * capture the context if one exists, but that could be done by
+ * inspection of the arity of the predicate. It might be better to
+ * have an explicit "CONSTRUCT" operator rather than havin...
[truncated message content] |
|
From: <sgo...@us...> - 2010-09-03 13:40:32
|
Revision: 3505
http://bigdata.svn.sourceforge.net/bigdata/?rev=3505&view=rev
Author: sgossard
Date: 2010-09-03 13:40:22 +0000 (Fri, 03 Sep 2010)
Log Message:
-----------
[maven_scaleout] : Initial framework for support maven sub-module aggregation. Previous artifact moved into subdirectory and renamed to bigdata-core.
Modified Paths:
--------------
branches/maven_scaleout/bigdata-core/src/main/deploy/bin/config/serviceStarter.config
branches/maven_scaleout/bigdata-core/src/main/deploy/bin/pstart
branches/maven_scaleout/bigdata-core/src/main/deploy/var/config/jini/boot/boot-processes.xml
branches/maven_scaleout/bigdata-core/src/main/deploy/var/config/jini/boot/boot-properties.xml
branches/maven_scaleout/bigdata-core/src/main/deploy/var/config/policy/launcher.policy
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/package.html
branches/maven_scaleout/bigdata-core/src/test/deploy/testing/conf/standalone/ServerStarter.config
branches/maven_scaleout/bigdata-core/src/test/deploy/testing/data/com/bigdata/boot/config/BootComponentTest_boot-processes.xml
branches/maven_scaleout/bigdata-core/src/test/deploy/testing/data/com/bigdata/jini/start/testjini.config
branches/maven_scaleout/bigdata-core/src/test/deploy/testing/test.xml
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/boot/BootComponentTest.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/rdf/store/AbstractServerTestCase.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/service/jini/AbstractServerTestCase.java
Added Paths:
-----------
branches/maven_scaleout/bigdata-core/
branches/maven_scaleout/bigdata-core/LEGAL/
branches/maven_scaleout/bigdata-core/LICENSE.txt
branches/maven_scaleout/bigdata-core/README
branches/maven_scaleout/bigdata-core/README-JINI
branches/maven_scaleout/bigdata-core/bigdata-perf/
branches/maven_scaleout/bigdata-core/bigdata-uml/
branches/maven_scaleout/bigdata-core/contrib/
branches/maven_scaleout/bigdata-core/dsi-utils/
branches/maven_scaleout/bigdata-core/lgpl-utils/
branches/maven_scaleout/bigdata-core/osgi/
branches/maven_scaleout/bigdata-core/overview.html
branches/maven_scaleout/bigdata-core/pom.xml
branches/maven_scaleout/bigdata-core/src/
branches/maven_scaleout/bigdata-core/thirdparty/
branches/maven_scaleout/pom.xml
Removed Paths:
-------------
branches/maven_scaleout/LEGAL/
branches/maven_scaleout/LICENSE.txt
branches/maven_scaleout/README
branches/maven_scaleout/README-JINI
branches/maven_scaleout/bigdata-perf/
branches/maven_scaleout/bigdata-uml/
branches/maven_scaleout/contrib/
branches/maven_scaleout/dsi-utils/
branches/maven_scaleout/lgpl-utils/
branches/maven_scaleout/osgi/
branches/maven_scaleout/overview.html
branches/maven_scaleout/pom.xml
branches/maven_scaleout/src/
branches/maven_scaleout/thirdparty/
Deleted: branches/maven_scaleout/LICENSE.txt
===================================================================
--- branches/maven_scaleout/LICENSE.txt 2010-09-03 12:32:22 UTC (rev 3504)
+++ branches/maven_scaleout/LICENSE.txt 2010-09-03 13:40:22 UTC (rev 3505)
@@ -1,354 +0,0 @@
-The GNU General Public License (GPL)
-Version 2, June 1991
-
-Copyright (C) 1989, 1991 Free Software Foundation, Inc.
-
-59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-
-Everyone is permitted to copy and distribute verbatim copies
-
-of this license document, but changing it is not allowed.
-
-
-Preamble
-
-
-The licenses for most software are designed to take away your
-freedom to share and change it. By contrast, the GNU General Public
-License is intended to guarantee your freedom to share and change free
-software--to make sure the software is free for all its users. This
-General Public License applies to most of the Free Software
-Foundation's software and to any other program whose authors commit to
-using it. (Some other Free Software Foundation software is covered by
-the GNU Library General Public License instead.) You can apply it to
-your programs, too.
-
-When we speak of free software, we are referring to freedom, not
-price. Our General Public Licenses are designed to make sure that you
-have the freedom to distribute copies of free software (and charge for
-this service if you wish), that you receive source code or can get it
-if you want it, that you can change the software or use pieces of it
-in new free programs; and that you know you can do these things.
-
-To protect your rights, we need to make restrictions that forbid
-anyone to deny you these rights or to ask you to surrender the rights.
-These restrictions translate to certain responsibilities for you if you
-distribute copies of the software, or if you modify it.
-
-For example, if you distribute copies of such a program, whether
-gratis or for a fee, you must give the recipients all the rights that
-you have. You must make sure that they, too, receive or can get the
-source code. And you must show them these terms so they know their
-rights.
-
-We protect your rights with two steps: (1) copyright the software, and
-(2) offer you this license which gives you legal permission to copy,
-distribute and/or modify the software.
-
-Also, for each author's protection and ours, we want to make certain
-that everyone understands that there is no warranty for this free
-software. If the software is modified by someone else and passed on, we
-want its recipients to know that what they have is not the original, so
-that any problems introduced by others will not reflect on the original
-authors' reputations.
-
-Finally, any free program is threatened constantly by software
-patents. We wish to avoid the danger that redistributors of a free
-program will individually obtain patent licenses, in effect making the
-program proprietary. To prevent this, we have made it clear that any
-patent must be licensed for everyone's free use or not licensed at all.
-
-The precise terms and conditions for copying, distribution and
-modification follow.
-
-
-TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
-
-
-0. This License applies to any program or other work which contains
-a notice placed by the copyright holder saying it may be distributed
-under the terms of this General Public License. The "Program", below,
-refers to any such program or work, and a "work based on the Program"
-means either the Program or any derivative work under copyright law:
-that is to say, a work containing the Program or a portion of it,
-either verbatim or with modifications and/or translated into another
-language. (Hereinafter, translation is included without limitation in
-the term "modification".) Each licensee is addressed as "you".
-
-Activities other than copying, distribution and modification are not
-covered by this License; they are outside its scope. The act of
-running the Program is not restricted, and the output from the Program
-is covered only if its contents constitute a work based on the
-Program (independent of having been made by running the Program).
-Whether that is true depends on what the Program does.
-
-1. You may copy and distribute verbatim copies of the Program's
-source code as you receive it, in any medium, provided that you
-conspicuously and appropriately publish on each copy an appropriate
-copyright notice and disclaimer of warranty; keep intact all the
-notices that refer to this License and to the absence of any warranty;
-and give any other recipients of the Program a copy of this License
-along with the Program.
-
-You may charge a fee for the physical act of transferring a copy, and
-you may at your option offer warranty protection in exchange for a fee.
-
-2. You may modify your copy or copies of the Program or any portion
-of it, thus forming a work based on the Program, and copy and
-distribute such modifications or work under the terms of Section 1
-above, provided that you also meet all of these conditions:
-
-a) You must cause the modified files to carry prominent notices
-stating that you changed the files and the date of any change.
-
-b) You must cause any work that you distribute or publish, that in
-whole or in part contains or is derived from the Program or any
-part thereof, to be licensed as a whole at no charge to all third
-parties under the terms of this License.
-
-c) If the modified program normally reads commands interactively
-when run, you must cause it, when started running for such
-interactive use in the most ordinary way, to print or display an
-announcement including an appropriate copyright notice and a
-notice that there is no warranty (or else, saying that you provide
-a warranty) and that users may redistribute the program under
-these conditions, and telling the user how to view a copy of this
-License. (Exception: if the Program itself is interactive but
-does not normally print such an announcement, your work based on
-the Program is not required to print an announcement.)
-
-
-These requirements apply to the modified work as a whole. If
-identifiable sections of that work are not derived from the Program,
-and can be reasonably considered independent and separate works in
-themselves, then this License, and its terms, do not apply to those
-sections when you distribute them as separate works. But when you
-distribute the same sections as part of a whole which is a work based
-on the Program, the distribution of the whole must be on the terms of
-this License, whose permissions for other licensees extend to the
-entire whole, and thus to each and every part regardless of who wrote it.
-
-Thus, it is not the intent of this section to claim rights or contest
-your rights to work written entirely by you; rather, the intent is to
-exercise the right to control the distribution of derivative or
-collective works based on the Program.
-
-In addition, mere aggregation of another work not based on the Program
-with the Program (or with a work based on the Program) on a volume of
-a storage or distribution medium does not bring the other work under
-the scope of this License.
-
-3. You may copy and distribute the Program (or a work based on it,
-under Section 2) in object code or executable form under the terms of
-Sections 1 and 2 above provided that you also do one of the following:
-
-a) Accompany it with the complete corresponding machine-readable
-source code, which must be distributed under the terms of Sections
-1 and 2 above on a medium customarily used for software interchange; or,
-
-b) Accompany it with a written offer, valid for at least three
-years, to give any third party, for a charge no more than your
-cost of physically performing source distribution, a complete
-machine-readable copy of the corresponding source code, to be
-distributed under the terms of Sections 1 and 2 above on a medium
-customarily used for software interchange; or,
-
-c) Accompany it with the information you received as to the offer
-to distribute corresponding source code. (This alternative is
-allowed only for noncommercial distribution and only if you
-received the program in object code or executable form with such
-an offer, in accord with Subsection b above.)
-
-
-The source code for a work means the preferred form of the work for
-making modifications to it. For an executable work, complete source
-code means all the source code for all modules it contains, plus any
-associated interface definition files, plus the scripts used to
-control compilation and installation of the executable. However, as a
-special exception, the source code distributed need not include
-anything that is normally distributed (in either source or binary
-form) with the major components (compiler, kernel, and so on) of the
-operating system on which the executable runs, unless that component
-itself accompanies the executable.
-
-If distribution of executable or object code is made by offering
-access to copy from a designated place, then offering equivalent
-access to copy the source code from the same place counts as
-distribution of the source code, even though third parties are not
-compelled to copy the source along with the object code.
-
-4. You may not copy, modify, sublicense, or distribute the Program
-except as expressly provided under this License. Any attempt
-otherwise to copy, modify, sublicense or distribute the Program is
-void, and will automatically terminate your rights under this License.
-However, parties who have received copies, or rights, from you under
-this License will not have their licenses terminated so long as such
-parties remain in full compliance.
-
-5. You are not required to accept this License, since you have not
-signed it. However, nothing else grants you permission to modify or
-distribute the Program or its derivative works. These actions are
-prohibited by law if you do not accept this License. Therefore, by
-modifying or distributing the Program (or any work based on the
-Program), you indicate your acceptance of this License to do so, and
-all its terms and conditions for copying, distributing or modifying
-the Program or works based on it.
-
-6. Each time you redistribute the Program (or any work based on the
-Program), the recipient automatically receives a license from the
-original licensor to copy, distribute or modify the Program subject to
-these terms and conditions. You may not impose any further
-restrictions on the recipients' exercise of the rights granted herein.
-You are not responsible for enforcing compliance by third parties to
-this License.
-
-7. If, as a consequence of a court judgment or allegation of patent
-infringement or for any other reason (not limited to patent issues),
-conditions are imposed on you (whether by court order, agreement or
-otherwise) that contradict the conditions of this License, they do not
-excuse you from the conditions of this License. If you cannot
-distribute so as to satisfy simultaneously your obligations under this
-License and any other pertinent obligations, then as a consequence you
-may not distribute the Program at all. For example, if a patent
-license would not permit royalty-free redistribution of the Program by
-all those who receive copies directly or indirectly through you, then
-the only way you could satisfy both it and this License would be to
-refrain entirely from distribution of the Program.
-
-If any portion of this section is held invalid or unenforceable under
-any particular circumstance, the balance of the section is intended to
-apply and the section as a whole is intended to apply in other
-circumstances.
-
-It is not the purpose of this section to induce you to infringe any
-patents or other property right claims or to contest validity of any
-such claims; this section has the sole purpose of protecting the
-integrity of the free software distribution system, which is
-implemented by public license practices. Many people have made
-generous contributions to the wide range of software distributed
-through that system in reliance on consistent application of that
-system; it is up to the author/donor to decide if he or she is willing
-to distribute software through any other system and a licensee cannot
-impose that choice.
-
-This section is intended to make thoroughly clear what is believed to
-be a consequence of the rest of this License.
-
-8. If the distribution and/or use of the Program is restricted in
-certain countries either by patents or by copyrighted interfaces, the
-original copyright holder who places the Program under this License
-may add an explicit geographical distribution limitation excluding
-those countries, so that distribution is permitted only in or among
-countries not thus excluded. In such case, this License incorporates
-the limitation as if written in the body of this License.
-
-9. The Free Software Foundation may publish revised and/or new versions
-of the General Public License from time to time. Such new versions will
-be similar in spirit to the present version, but may differ in detail to
-address new problems or concerns.
-
-Each version is given a distinguishing version number. If the Program
-specifies a version number of this License which applies to it and "any
-later version", you have the option of following the terms and conditions
-either of that version or of any later version published by the Free
-Software Foundation. If the Program does not specify a version number of
-this License, you may choose any version ever published by the Free Software
-Foundation.
-
-10. If you wish to incorporate parts of the Program into other free
-programs whose distribution conditions are different, write to the author
-to ask for permission. For software which is copyrighted by the Free
-Software Foundation, write to the Free Software Foundation; we sometimes
-make exceptions for this. Our decision will be guided by the two goals
-of preserving the free status of all derivatives of our free software and
-of promoting the sharing and reuse of software generally.
-
-NO WARRANTY
-
-11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
-FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
-OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
-PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
-OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
-MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
-TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
-PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
-REPAIR OR CORRECTION.
-
-12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
-WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
-REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
-INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
-OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
-TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
-YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
-PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
-POSSIBILITY OF SUCH DAMAGES.
-
-END OF TERMS AND CONDITIONS
-
-
-How to Apply These Terms to Your New Programs
-
-If you develop a new program, and you want it to be of the greatest
-possible use to the public, the best way to achieve this is to make it
-free software which everyone can redistribute and change under these terms.
-
-To do so, attach the following notices to the program. It is safest
-to attach them to the start of each source file to most effectively
-convey the exclusion of warranty; and each file should have at least
-the "copyright" line and a pointer to where the full notice is found.
-
-One line to give the program's name and a brief idea of what it does.
-
-Copyright (C) <year> <name of author>
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; either version 2 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-
-
-Also add information on how to contact you by electronic and paper mail.
-
-If the program is interactive, make it output a short notice like this
-when it starts in an interactive mode:
-
-Gnomovision version 69, Copyright (C) year name of author
-Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
-This is free software, and you are welcome to redistribute it
-under certain conditions; type `show c' for details.
-
-
-The hypothetical commands `show w' and `show c' should show the appropriate
-parts of the General Public License. Of course, the commands you use may
-be called something other than `show w' and `show c'; they could even be
-mouse-clicks or menu items--whatever suits your program.
-
-You should also get your employer (if you work as a programmer) or your
-school, if any, to sign a "copyright disclaimer" for the program, if
-necessary. Here is a sample; alter the names:
-
-Yoyodyne, Inc., hereby disclaims all copyright interest in the program
-`Gnomovision' (which makes passes at compilers) written by James Hacker.
-
-signature of Ty Coon, 1 April 1989
-
-Ty Coon, President of Vice
-
-
-This General Public License does not permit incorporating your program into
-proprietary programs. If your program is a subroutine library, you may
-consider it more useful to permit linking proprietary applications with the
-library. If this is what you want to do, use the GNU Library General
-Public License instead of this License.
-
Deleted: branches/maven_scaleout/README
===================================================================
Deleted: branches/maven_scaleout/README-JINI
===================================================================
--- branches/maven_scaleout/README-JINI 2010-09-03 12:32:22 UTC (rev 3504)
+++ branches/maven_scaleout/README-JINI 2010-09-03 13:40:22 UTC (rev 3505)
@@ -1,160 +0,0 @@
-Some notes on installation and use follow:
-
-JINI
-
-- jini is used as a service fabric for bigdata. <start up jini and
- then configure your data and metadata services; clients then
- discover those services>
-
-- jini 2.1 may report errors locating the shared libraries from awk,
- dirname, basename and grep when installing under un*x. The problem
- is an assumption about the kernel version. This problem is resolved
- by editing the installer and the launchall script.
- See http://www.jini.org/wiki/Category:Getting_Started and
- http://www.linuxquestions.org/questions/showthread.php?t=370056 for
- a resolution. Here is is in case that link goes away:
-
- Open the bin installer file in an editor. Look for the line
-
- export LD_ASSUME_KERNEL=2.2.5
-
- and replace it with
-
- #xport LD_ASSUME_KERNEL=2.2.5
-
- Save the file and launch.
-
- Once jini is installed, you need to do exactly the same thing for the
- Launch-All script in the installverify directory - this is the script
- that you use the start jini.
-
-- Here are some notes on getting things working on a Fedora Core 6 platform.
- The symptom was that ifconfig was reporting MULTICAST for the interface
- but the jini install was complaining that multicase was not enabled for
- that interface.
-
- Here's what I did:
-
- First, verify that multicast is enabled on eth0 by typing ifconfig and looking for multicast
-
- if it is not enabled type ifconfig eth0 multicast
-
- after that add a default route for multicast broadcasts and bind it to eth0
-
- route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0
-
- additionally, I disabled the firewall and I disabled SELinux (but I think the firewall was the big culprit here).
-
-- Downloadable code is NOT required for deployments, but MAY be useful
- for the following purposes:
-
- (a) exposing services to the Jini service browser;
-
- (b) running procedures against services which were deployed before
- the procedures were written;
-
- If you have a complete classpath when running the various services
- then jini will not seek to transfer code from the client as the code
- will be resolved locally by the service.
-
- In order to support downloadable code you need to have an HTTP
- server that will serve class files to jini, including the interfaces
- for all remote services. You can use any HTTP server for this
- purpose, and the server can be located on any machine accessible
- from the host(s) on which you are running jini. As a convenience,
- jini bundles a simple HTTP server that you can start using a command
- like the following:
-
- java -jar ${JINI_HOME}/lib/classserver.jar -port 8080 -dir classes -trees -verbose&
-
- The javadoc describes the logging and command line options for this
- HTTP server.
-
- https://java.sun.com/products/jini/2.1/doc/api/com/sun/jini/tool/ClassServer.html
-
- The directory from which downloadable code will be served should
- contain at least the bigdata jar(s) plus any remote application code
- that you have defined (i.e., code that will run in the server
- process).
-
- The recommended approach to downloadable code is to extract the
- relevant classes into a directory that will be named to the HTTP
- server as follows. Assuming that bigdata.jar is located in the
- current directory:
-
- mkdir classes
- cd classes
- jar xfz ../bigdata.jar
-
- If you deploy a new version of any JAR, then you SHOULD delete the
- classes directory and redeploy all relevant JARs to make sure that
- old class files are not left lying around.
-
-- You can enable NIO support with JERI using TCP by specifying the
- following property to the JVM. Note that JRMP does NOT allow for
- the possibility of NIO.
-
- -Dcom.sun.jini.jeri.tcp.useNIO=true
-
- More information on JERI and NIO is available using the following links.
-
- http://archives.java.sun.com/cgi-bin/wa?A2=ind0504&L=jini-users&P=33490
- http://archives.java.sun.com/cgi-bin/wa?A2=ind0506&L=jini-users&P=9626
- http://archives.java.sun.com/cgi-bin/wa?A2=ind0504&L=jini-users&D=0&P=26542
- http://java.sun.com/products/jini/2.0.1/doc/api/net/jini/jeri/tcp/package-summary.html
-
- Note that one server thread will still be required per concurrent RPC request
- owing to the semantics of RPC (call and wait for response) and the definition
- of JERI.
-
-- Clients downloadable code that will be run on the bigdata services MUST set:
-
- -Djava.rmi.server.codebase=http://.../
-
- where "..." is your host and path
-
- in order for the correct codebase property to be communicated to clients that
- will then download code from that HTTP server. Note: the trailing '/' is
- REQUIRED in your codebase or the generated URLs will NOT resolve correctly.
-
- There is an example of how to do this with the "ant lubm-install" target and
- the "lubmMaster.sh" script.
-
-- Debugging with jini.
-
- See http://java.sun.com/j2se/1.4.2/docs/guide/rmi/javarmiproperties.html for
- some guidance. Among other things, it suggests:
-
- -Djava.rmi.server.logCalls=true
-
- as an aid to debugging. Also try setting
-
- -Dcom.sun.jini.reggie.proxy.debug=1
-
- for the client, e.g., the service browser. Also see:
-
- http://www.adtmag.com/java/articleold.aspx?id=1159
-
- for some (very good) guidance in debugging jini services.
-
- Note: You may have to restart jini locally in order to force download of
- updated classes from the codebase!
-
- See http://archives.java.sun.com/cgi-bin/wa?A2=ind0512&L=jini-users&P=R391&I=-3
- for instructions on setting up an "download jar" (dljar) ANT task that can make
- life much simpler (one supposes).
-
- See http://archives.java.sun.com/cgi-bin/wa?A2=ind0311&L=jini-users&F=&S=&P=7182
- for a description of policy files and
- http://www.dancres.org/cottage/jini-start-examples-2_1.zip for the
- policy files described.
-
- See http://jan.newmarch.name/java/jini/tutorial/Ant.xml for a description of
- one (simple) approach to using ant for jini projects (it does not use the
- dljar ant task but explicitly enumerates what goes where).
-
- See http://jan.newmarch.name/java/jini/tutorial/TroubleShooting.xml#RMI%20Stubs
- for common errors when using RMI stubs.
-
- See https://java.sun.com/products/jini/2.1/doc/api/com/sun/jini/example/browser/package-summary.html
- for the dirty on the jini Service Browser.
Copied: branches/maven_scaleout/bigdata-core/LICENSE.txt (from rev 3496, branches/maven_scaleout/LICENSE.txt)
===================================================================
--- branches/maven_scaleout/bigdata-core/LICENSE.txt (rev 0)
+++ branches/maven_scaleout/bigdata-core/LICENSE.txt 2010-09-03 13:40:22 UTC (rev 3505)
@@ -0,0 +1,354 @@
+The GNU General Public License (GPL)
+Version 2, June 1991
+
+Copyright (C) 1989, 1991 Free Software Foundation, Inc.
+
+59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+Everyone is permitted to copy and distribute verbatim copies
+
+of this license document, but changing it is not allowed.
+
+
+Preamble
+
+
+The licenses for most software are designed to take away your
+freedom to share and change it. By contrast, the GNU General Public
+License is intended to guarantee your freedom to share and change free
+software--to make sure the software is free for all its users. This
+General Public License applies to most of the Free Software
+Foundation's software and to any other program whose authors commit to
+using it. (Some other Free Software Foundation software is covered by
+the GNU Library General Public License instead.) You can apply it to
+your programs, too.
+
+When we speak of free software, we are referring to freedom, not
+price. Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+this service if you wish), that you receive source code or can get it
+if you want it, that you can change the software or use pieces of it
+in new free programs; and that you know you can do these things.
+
+To protect your rights, we need to make restrictions that forbid
+anyone to deny you these rights or to ask you to surrender the rights.
+These restrictions translate to certain responsibilities for you if you
+distribute copies of the software, or if you modify it.
+
+For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must give the recipients all the rights that
+you have. You must make sure that they, too, receive or can get the
+source code. And you must show them these terms so they know their
+rights.
+
+We protect your rights with two steps: (1) copyright the software, and
+(2) offer you this license which gives you legal permission to copy,
+distribute and/or modify the software.
+
+Also, for each author's protection and ours, we want to make certain
+that everyone understands that there is no warranty for this free
+software. If the software is modified by someone else and passed on, we
+want its recipients to know that what they have is not the original, so
+that any problems introduced by others will not reflect on the original
+authors' reputations.
+
+Finally, any free program is threatened constantly by software
+patents. We wish to avoid the danger that redistributors of a free
+program will individually obtain patent licenses, in effect making the
+program proprietary. To prevent this, we have made it clear that any
+patent must be licensed for everyone's free use or not licensed at all.
+
+The precise terms and conditions for copying, distribution and
+modification follow.
+
+
+TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+
+0. This License applies to any program or other work which contains
+a notice placed by the copyright holder saying it may be distributed
+under the terms of this General Public License. The "Program", below,
+refers to any such program or work, and a "work based on the Program"
+means either the Program or any derivative work under copyright law:
+that is to say, a work containing the Program or a portion of it,
+either verbatim or with modifications and/or translated into another
+language. (Hereinafter, translation is included without limitation in
+the term "modification".) Each licensee is addressed as "you".
+
+Activities other than copying, distribution and modification are not
+covered by this License; they are outside its scope. The act of
+running the Program is not restricted, and the output from the Program
+is covered only if its contents constitute a work based on the
+Program (independent of having been made by running the Program).
+Whether that is true depends on what the Program does.
+
+1. You may copy and distribute verbatim copies of the Program's
+source code as you receive it, in any medium, provided that you
+conspicuously and appropriately publish on each copy an appropriate
+copyright notice and disclaimer of warranty; keep intact all the
+notices that refer to this License and to the absence of any warranty;
+and give any other recipients of the Program a copy of this License
+along with the Program.
+
+You may charge a fee for the physical act of transferring a copy, and
+you may at your option offer warranty protection in exchange for a fee.
+
+2. You may modify your copy or copies of the Program or any portion
+of it, thus forming a work based on the Program, and copy and
+distribute such modifications or work under the terms of Section 1
+above, provided that you also meet all of these conditions:
+
+a) You must cause the modified files to carry prominent notices
+stating that you changed the files and the date of any change.
+
+b) You must cause any work that you distribute or publish, that in
+whole or in part contains or is derived from the Program or any
+part thereof, to be licensed as a whole at no charge to all third
+parties under the terms of this License.
+
+c) If the modified program normally reads commands interactively
+when run, you must cause it, when started running for such
+interactive use in the most ordinary way, to print or display an
+announcement including an appropriate copyright notice and a
+notice that there is no warranty (or else, saying that you provide
+a warranty) and that users may redistribute the program under
+these conditions, and telling the user how to view a copy of this
+License. (Exception: if the Program itself is interactive but
+does not normally print such an announcement, your work based on
+the Program is not required to print an announcement.)
+
+
+These requirements apply to the modified work as a whole. If
+identifiable sections of that work are not derived from the Program,
+and can be reasonably considered independent and separate works in
+themselves, then this License, and its terms, do not apply to those
+sections when you distribute them as separate works. But when you
+distribute the same sections as part of a whole which is a work based
+on the Program, the distribution of the whole must be on the terms of
+this License, whose permissions for other licensees extend to the
+entire whole, and thus to each and every part regardless of who wrote it.
+
+Thus, it is not the intent of this section to claim rights or contest
+your rights to work written entirely by you; rather, the intent is to
+exercise the right to control the distribution of derivative or
+collective works based on the Program.
+
+In addition, mere aggregation of another work not based on the Program
+with the Program (or with a work based on the Program) on a volume of
+a storage or distribution medium does not bring the other work under
+the scope of this License.
+
+3. You may copy and distribute the Program (or a work based on it,
+under Section 2) in object code or executable form under the terms of
+Sections 1 and 2 above provided that you also do one of the following:
+
+a) Accompany it with the complete corresponding machine-readable
+source code, which must be distributed under the terms of Sections
+1 and 2 above on a medium customarily used for software interchange; or,
+
+b) Accompany it with a written offer, valid for at least three
+years, to give any third party, for a charge no more than your
+cost of physically performing source distribution, a complete
+machine-readable copy of the corresponding source code, to be
+distributed under the terms of Sections 1 and 2 above on a medium
+customarily used for software interchange; or,
+
+c) Accompany it with the information you received as to the offer
+to distribute corresponding source code. (This alternative is
+allowed only for noncommercial distribution and only if you
+received the program in object code or executable form with such
+an offer, in accord with Subsection b above.)
+
+
+The source code for a work means the preferred form of the work for
+making modifications to it. For an executable work, complete source
+code means all the source code for all modules it contains, plus any
+associated interface definition files, plus the scripts used to
+control compilation and installation of the executable. However, as a
+special exception, the source code distributed need not include
+anything that is normally distributed (in either source or binary
+form) with the major components (compiler, kernel, and so on) of the
+operating system on which the executable runs, unless that component
+itself accompanies the executable.
+
+If distribution of executable or object code is made by offering
+access to copy from a designated place, then offering equivalent
+access to copy the source code from the same place counts as
+distribution of the source code, even though third parties are not
+compelled to copy the source along with the object code.
+
+4. You may not copy, modify, sublicense, or distribute the Program
+except as expressly provided under this License. Any attempt
+otherwise to copy, modify, sublicense or distribute the Program is
+void, and will automatically terminate your rights under this License.
+However, parties who have received copies, or rights, from you under
+this License will not have their licenses terminated so long as such
+parties remain in full compliance.
+
+5. You are not required to accept this License, since you have not
+signed it. However, nothing else grants you permission to modify or
+distribute the Program or its derivative works. These actions are
+prohibited by law if you do not accept this License. Therefore, by
+modifying or distributing the Program (or any work based on the
+Program), you indicate your acceptance of this License to do so, and
+all its terms and conditions for copying, distributing or modifying
+the Program or works based on it.
+
+6. Each time you redistribute the Program (or any work based on the
+Program), the recipient automatically receives a license from the
+original licensor to copy, distribute or modify the Program subject to
+these terms and conditions. You may not impose any further
+restrictions on the recipients' exercise of the rights granted herein.
+You are not responsible for enforcing compliance by third parties to
+this License.
+
+7. If, as a consequence of a court judgment or allegation of patent
+infringement or for any other reason (not limited to patent issues),
+conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License. If you cannot
+distribute so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you
+may not distribute the Program at all. For example, if a patent
+license would not permit royalty-free redistribution of the Program by
+all those who receive copies directly or indirectly through you, then
+the only way you could satisfy both it and this License would be to
+refrain entirely from distribution of the Program.
+
+If any portion of this section is held invalid or unenforceable under
+any particular circumstance, the balance of the section is intended to
+apply and the section as a whole is intended to apply in other
+circumstances.
+
+It is not the purpose of this section to induce you to infringe any
+patents or other property right claims or to contest validity of any
+such claims; this section has the sole purpose of protecting the
+integrity of the free software distribution system, which is
+implemented by public license practices. Many people have made
+generous contributions to the wide range of software distributed
+through that system in reliance on consistent application of that
+system; it is up to the author/donor to decide if he or she is willing
+to distribute software through any other system and a licensee cannot
+impose that choice.
+
+This section is intended to make thoroughly clear what is believed to
+be a consequence of the rest of this License.
+
+8. If the distribution and/or use of the Program is restricted in
+certain countries either by patents or by copyrighted interfaces, the
+original copyright holder who places the Program under this License
+may add an explicit geographical distribution limitation excluding
+those countries, so that distribution is permitted only in or among
+countries not thus excluded. In such case, this License incorporates
+the limitation as if written in the body of this License.
+
+9. The Free Software Foundation may publish revised and/or new versions
+of the General Public License from time to time. Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+Each version is given a distinguishing version number. If the Program
+specifies a version number of this License which applies to it and "any
+later version", you have the option of following the terms and conditions
+either of that version or of any later version published by the Free
+Software Foundation. If the Program does not specify a version number of
+this License, you may choose any version ever published by the Free Software
+Foundation.
+
+10. If you wish to incorporate parts of the Program into other free
+programs whose distribution conditions are different, write to the author
+to ask for permission. For software which is copyrighted by the Free
+Software Foundation, write to the Free Software Foundation; we sometimes
+make exceptions for this. Our decision will be guided by the two goals
+of preserving the free status of all derivatives of our free software and
+of promoting the sharing and reuse of software generally.
+
+NO WARRANTY
+
+11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
+FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
+OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
+PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
+OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
+TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
+PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
+REPAIR OR CORRECTION.
+
+12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
+REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
+INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
+OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
+TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
+YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
+PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGES.
+
+END OF TERMS AND CONDITIONS
+
+
+How to Apply These Terms to Your New Programs
+
+If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+To do so, attach the following notices to the program. It is safest
+to attach them to the start of each source file to most effectively
+convey the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+One line to give the program's name and a brief idea of what it does.
+
+Copyright (C) <year> <name of author>
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+
+Also add information on how to contact you by electronic and paper mail.
+
+If the program is interactive, make it output a short notice like this
+when it starts in an interactive mode:
+
+Gnomovision version 69, Copyright (C) year name of author
+Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+This is free software, and you are welcome to redistribute it
+under certain conditions; type `show c' for details.
+
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License. Of course, the commands you use may
+be called something other than `show w' and `show c'; they could even be
+mouse-clicks or menu items--whatever suits your program.
+
+You should also get your employer (if you work as a programmer) or your
+school, if any, to sign a "copyright disclaimer" for the program, if
+necessary. Here is a sample; alter the names:
+
+Yoyodyne, Inc., hereby disclaims all copyright interest in the program
+`Gnomovision' (which makes passes at compilers) written by James Hacker.
+
+signature of Ty Coon, 1 April 1989
+
+Ty Coon, President of Vice
+
+
+This General Public License does not permit incorporating your program into
+proprietary programs. If your program is a subroutine library, you may
+consider it more useful to permit linking proprietary applications with the
+library. If this is what you want to do, use the GNU Library General
+Public License instead of this License.
+
Copied: branches/maven_scaleout/bigdata-core/README (from rev 3496, branches/maven_scaleout/README)
===================================================================
Copied: branches/maven_scaleout/bigdata-core/README-JINI (from rev 3496, branches/maven_scaleout/README-JINI)
===================================================================
--- branches/maven_scaleout/bigdata-core/README-JINI (rev 0)
+++ branches/maven_scaleout/bigdata-core/README-JINI 2010-09-03 13:40:22 UTC (rev 3505)
@@ -0,0 +1,160 @@
+Some notes on installation and use follow:
+
+JINI
+
+- jini is used as a service fabric for bigdata. <start up jini and
+ then configure your data and metadata services; clients then
+ discover those services>
+
+- jini 2.1 may report errors locating the shared libraries from awk,
+ dirname, basename and grep when installing under un*x. The problem
+ is an assumption about the kernel version. This problem is resolved
+ by editing the installer and the launchall script.
+ See http://www.jini.org/wiki/Category:Getting_Started and
+ http://www.linuxquestions.org/questions/showthread.php?t=370056 for
+ a resolution. Here is is in case that link goes away:
+
+ Open the bin installer file in an editor. Look for the line
+
+ export LD_ASSUME_KERNEL=2.2.5
+
+ and replace it with
+
+ #xport LD_ASSUME_KERNEL=2.2.5
+
+ Save the file and launch.
+
+ Once jini is installed, you need to do exactly the same thing for the
+ Launch-All script in the installverify directory - this is the script
+ that you use the start jini.
+
+- Here are some notes on getting things working on a Fedora Core 6 platform.
+ The symptom was that ifconfig was reporting MULTICAST for the interface
+ but the jini install was complaining that multicase was not enabled for
+ that interface.
+
+ Here's what I did:
+
+ First, verify that multicast is enabled on eth0 by typing ifconfig and looking for multicast
+
+ if it is not enabled type ifconfig eth0 multicast
+
+ after that add a default route for multicast broadcasts and bind it to eth0
+
+ route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0
+
+ additionally, I disabled the firewall and I disabled SELinux (but I think the firewall was the big culprit here).
+
+- Downloadable code is NOT required for deployments, but MAY be useful
+ for the following purposes:
+
+ (a) exposing services to the Jini service browser;
+
+ (b) running procedures against services which were deployed before
+ the procedures were written;
+
+ If you have a complete classpath when running the various services
+ then jini will not seek to transfer code from the client as the code
+ will be resolved locally by the service.
+
+ In order to support downloadable code you need to have an HTTP
+ server that will serve class files to jini, including the interfaces
+ for all remote services. You can use any HTTP server for this
+ purpose, and the server can be located on any machine accessible
+ from the host(s) on which you are running jini. As a convenience,
+ jini bundles a simple HTTP server that you can start using a command
+ like the following:
+
+ java -jar ${JINI_HOME}/lib/classserver.jar -port 8080 -dir classes -trees -verbose&
+
+ The javadoc describes the logging and command line options for this
+ HTTP server.
+
+ https://java.sun.com/products/jini/2.1/doc/api/com/sun/jini/tool/ClassServer.html
+
+ The directory from which downloadable code will be served should
+ contain at least the bigdata jar(s) plus any remote application code
+ that you have defined (i.e., code that will run in the server
+ process).
+
+ The recommended approach to downloadable code is to extract the
+ relevant classes into a directory that will be named to the HTTP
+ server as follows. Assuming that bigdata-core.jar is located in the
+ current directory:
+
+ mkdir classes
+ cd classes
+ jar xfz ../bigdata-core.jar
+
+ If you deploy a new version of any JAR, then you SHOULD delete the
+ classes directory and redeploy all relevant JARs to make sure that
+ old class files are not left lying around.
+
+- You can enable NIO support with JERI using TCP by specifying the
+ following property to the JVM. Note that JRMP does NOT allow for
+ the possibility of NIO.
+
+ -Dcom.sun.jini.jeri.tcp.useNIO=true
+
+ More information on JERI and NIO is available using the following links.
+
+ http://archives.java.sun.com/cgi-bin/wa?A2=ind0504&L=jini-users&P=33490
+ http://archives.java.sun.com/cgi-bin/wa?A2=ind0506&L=jini-users&P=9626
+ http://archives.java.sun.com/cgi-bin/wa?A2=ind0504&L=jini-users&D=0&P=26542
+ http://java.sun.com/products/jini/2.0.1/doc/api/net/jini/jeri/tcp/package-summary.html
+
+ Note that one server thread will still be required per concurrent RPC request
+ owing to the semantics of RPC (call and wait for response) and the definition
+ of JERI.
+
+- Clients downloadable code that will be run on the bigdata services MUST set:
+
+ -Djava.rmi.server.codebase=http://.../
+
+ where "..." is your host and path
+
+ in order for the correct codebase property to be communicated to clients that
+ will then download code from that HTTP server. Note: the trailing '/' is
+ REQUIRED in your codebase or the generated URLs will NOT resolve correctly.
+
+ There is an example of how to do this with the "ant lubm-install" target and
+ the "lubmMaster.sh" script.
+
+- Debugging with jini.
+
+ See http://java.sun.com/j2se/1.4.2/docs/guide/rmi/javarmiproperties.html for
+ some guidance. Among other things, it suggests:
+
+ -Djava.rmi.server.logCalls=true
+
+ as an aid to debugging. Also try setting
+
+ -Dcom.sun.jini.reggie.proxy.debug=1
+
+ for the client, e.g., the service browser. Also see:
+
+ http://www.adtmag.com/java/articleold.aspx?id=1159
+
+ for some (very good) guidance in debugging jini services.
+
+ Note: You may have to restart jini locally in order to force download of
+ updated classes from the codebase!
+
+ See http://archives.java.sun.com/cgi-bin/wa?A2=ind0512&L=jini-users&P=R391&I=-3
+ for instructions on setting up an "download jar" (dljar) ANT task that can make
+ life much simpler (one supposes).
+
+ See http://archives.java.sun.com/cgi-bin/wa?A2=ind0311&L=jini-users&F=&S=&P=7182
+ for a description of policy files and
+ http://www.dancres.org/cottage/jini-start-examples-2_1.zip for the
+ policy files described.
+
+ See http://jan.newmarch.name/java/jini/tutorial/Ant.xml for a description of
+ one (simple) approach to using ant for jini projects (it does not use the
+ dljar ant task but explicitly enumerates what goes where).
+
+ See http://jan.newmarch.name/java/jini/tutorial/TroubleShooting.xml#RMI%20Stubs
+ for common errors when using RMI stubs.
+
+ See https://java.sun.com/products/jini/2.1/doc/api/com/sun/jini/example/browser/package-summary.html
+ for the dirty on the jini Service Browser.
Copied: branches/maven_scaleout/bigdata-core/overview.html (from rev 3496, branches/maven_scaleout/overview.html)
===================================================================
--- branches/maven_scaleout/bigdata-core/overview.html (rev 0)
+++ branches/maven_scaleout/bigdata-core/overview.html 2010-09-03 13:40:22 UTC (rev 3505)
@@ -0,0 +1,480 @@
+<html>
+<head>
+<title>bigdata®</title>
+<!-- $Id$ -->
+</head>
+<body>
+
+ <p>
+
+ <em>bigdata®</em> is a scale-out data and computing
+ fabric designed for commodity hardware. The bigdata
+ architecture provides named scale-out indices that are
+ transparently and dynamically key-range partitioned and
+ distributed across a cluster or grid of commodity server
+ platforms. The scale-out indices are B+Trees and remain
+ balanced under insert and removal operations. Keys and
+ values for btrees are variable length byte[]s (the keys are
+ interpreted as unsigned byte[]s).
+
+ Atomic "row" operations are supported for very high
+ concurrency. However, full transactions are also available
+ for applications needing less concurrency but requiring
+ atomic operations that read or write on more than one row,
+ index partition, or index.
+
+ Writes are absorbed on mutable btree instances in append
+ only "journals" of ~200M capacity. On overflow, data in a
+ journal is evicted onto read-optimized, immutable "index
+ segments". The metadata service manages the index
+ definitions, the index partitions, and the assignment of
+ index partitions to data services. A data service
+ encapsulates a journal and zero or more index partitions
+ assigned to that data service, including the logic to handle
+ overflow.
+
+</p><p>
+
+ A deployment is made up of a transaction service, a load
+ balancer service, and a metadata service (with failover
+ redundancy) and many data service instances. bigdata can
+ provide data redundancy internally (by pipelining writes
+ bound for a partition across primary, secondary, ... data
+ service instances for that partition) or it can be deployed
+ over NAS/SAN. bigdata itself is 100% Java and requires a
+ JDK 1.6. There are additional dependencies for the
+ installer (un*x) and for collecting performance counters
+ from the OS (<a
+ href="http://pagesperso-orange.fr/sebastien.godard/">sysstat</a>).
+
+ </p>
+
+ <h2>Architecture</h2>
+
+ <p>
+
+ The bigdata SOA defines three essential services and some
+ additional services. The essential services are the
+ metadata service (provides a locator service for index
+ partitions on data services), the data service (provides
+ read, write, and concurrency control for index partitions),
+ and the transaction service (provides consistent timestamps
+ for commits, facilitates the release of data associated with
+ older commit points, read locks, and transactions). Full
+ transactions are NOT required, so you can use bigdata as a
+ scale-out row store. The load balancer service guides the
+ dynamic redistribution of data across a bigdata federation.
+ There are also client services, which are containers for
+ executing distributed jobs.
+
+ </p>
+
+ <p>
+
+ While other service fabric architectures are contemplated,
+ bigdata services today use JINI 2. to advertise themselves
+ and do service discovery. This means that you must be
+ running a JINI registrar in order for services to be able to
+ register themselves or discover other services. The JINI
+ integration is bundled and installed automatically by
+ default.
+
+ </p>
+
+ <p>
+
+ Zookeeper handles master election, configuration management
+ and global synchronous locks. Zookeeper was developed by
+ Yahoo! as a distributed lock and configuration management
+ service and is now an Apache subproject (part of
+ Hadoop). Among other things, it gets master election
+ protocols right. Zookeeper is bundled and installed
+ automatically by default.
+
+ </p>
+
+ <p>
+
+ The main building blocks for the bigdata architecture are
+ the journal (both an append-only persistence store and a recently
+ introduced read/write store with the ability to recycle historical
+ commit points), the mutable
+ B+Tree (used to absorb writes), and the read-optimized
+ immutable B+Tree (aka the index segment). Highly efficient
+ bulk index builds are used to transfer data absorbed by a
+ mutable B+Tree on a journal into index segment files. Each
+ for index segment contains data for a single partition of a
+ scale-out index. In order to read from an index partition,
+ a consistent view is created by dynamically fusing data for
+ that index partition, including any recent writes on the
+ current journal, any historical writes that are in the
+ process of being transferred onto index segments, and any
+ historical index segments that also contain data for that
+ view. Periodically, index segments are merged together, at
+ which point deleted tuples are purged from the view.
+
+</p>
+
+<p>
+
+ Bigdata periodically releases data associated with older
+ commit points, freeing up disk resources. The transaction
+ service is configured with a minimum release age in
+ milliseconds. This can be ZERO (0L) milliseconds, in which
+ case historical views may be released if there are no read
+ locks for that commit point. The minimum release age can
+ also be hours or days if you want to keep historical states
+ around for a while. When a data service overflows, it will
+ consult the transaction service to determine the effective
+ release time and release any old journals or index segments
+ no longer required to maintain views GT that release time.
+</p><p>
+ An immortal or temporal database can be realized by
+ specifying Long#MAX_VALUE for the minimum release age. In
+ this case, the old journals and index segments will all be
+ retained and you can query any historical commit point of
+ the database at any time.
+
+ </p><p>
+
+ An detailed architecture whitepaper for bigdata is posted only and
+ linked from our <a href="www.bigdata.com/blog">blog</a>.
+
+ </p>
+
+ <h2>Sparse row store</h2>
+
+ <p>
+
+ The <em>SparseRowStore</em> provides a column flexible row
+ store similar to Google's bigtable or HBase, including very
+ high read/write concurrency and ACID operations on logical
+ "rows". Internally, a "global row store" instance is used
+ to maintain metadata on relations declared within a bigdata
+ federation. You can use this instance to store your own
+ data, or you can create your own named row store instances.
+ However, there is no REST api for the row store at this time
+ (trivial to be sure, but not something that we have gotten
+ around to yet).
+
+</p><p>
+
+ In fact, it is trivial to realize bigtable semantics with
+ bigdata - all you need to do is exercise a specific protocol
+ when forming the keys for your scale-out indices and then
+ you simply choose to NOT use transact...
[truncated message content] |
|
From: <tho...@us...> - 2010-09-03 12:32:28
|
Revision: 3504
http://bigdata.svn.sourceforge.net/bigdata/?rev=3504&view=rev
Author: thompsonbry
Date: 2010-09-03 12:32:22 +0000 (Fri, 03 Sep 2010)
Log Message:
-----------
Updated the javadoc on the WriteCache to clarify the conditions which require additional synchronization to protect the ByteBuffer's position() from concurrency related side effects.
Modified Paths:
--------------
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/io/writecache/WriteCache.java
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/io/writecache/WriteCache.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/io/writecache/WriteCache.java 2010-09-03 11:48:08 UTC (rev 3503)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/io/writecache/WriteCache.java 2010-09-03 12:32:22 UTC (rev 3504)
@@ -134,20 +134,28 @@
*/
final private ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
- /**
- * Return the backing {@link ByteBuffer}. The caller may read or write on
- * the buffer. Once they are done, the caller MUST call {@link #release()}.
- * This uses the read lock to allow concurrent read/write operations on the
- * backing buffer. Note that at most one write operation may execute
- * concurrently in order to avoid side effects on the buffers position when
- * copying data onto the buffer.
- *
- * @return The {@link ByteBuffer}.
- *
- * @throws InterruptedException
- * @throws IllegalStateException
- * if the {@link WriteCache} is closed.
- */
+ /**
+ * Return the backing {@link ByteBuffer}. The caller may read or write on
+ * the buffer, but MUST NOT have a side effect on the
+ * {@link ByteBuffer#position()} without first synchronizing on the
+ * {@link ByteBuffer}. Once they are done, the caller MUST call
+ * {@link #release()}.
+ * <p>
+ * Note: This uses the read lock to allow concurrent read/write operations
+ * on the backing buffer.
+ * <p>
+ * Note: <strong>At most one write operation may execute concurrently in
+ * order to avoid side effects on the buffers position when copying data
+ * onto the buffer. This constraint must be imposed by the caller using a
+ * <code>synchronized(buf){}</code> block during the critical sections where
+ * the buffer position will be updated by a write. </strong>
+ *
+ * @return The {@link ByteBuffer}.
+ *
+ * @throws InterruptedException
+ * @throws IllegalStateException
+ * if the {@link WriteCache} is closed.
+ */
private ByteBuffer acquire() throws InterruptedException, IllegalStateException {
final Lock readLock = lock.readLock();
@@ -668,12 +676,15 @@
if (remaining == 0)
throw new IllegalArgumentException(AbstractBufferStrategy.ERR_BUFFER_EMPTY);
- /*
- * Note: We need to be synchronized on the ByteBuffer here since
- * this operation relies on the position() being stable.
- *
- * Note: No other code adjust touch the buffer's position() !!!
- */
+ /*
+ * Note: We need to be synchronized on the ByteBuffer here since
+ * this operation relies on the position() being stable.
+ *
+ * Note: Also see clearAddrMap(long) which is synchronized on the
+ * acquired ByteBuffer in the same manner to protect it during
+ * critical sections which have a side effect on the buffer
+ * position.
+ */
final int pos;
synchronized (tmp) {
@@ -730,7 +741,7 @@
counters.naccept++;
counters.bytesAccepted += nwrite;
- }
+ } // synchronized(tmp)
/*
* Add metadata for the record so it can be read back from the
@@ -1737,21 +1748,26 @@
// }
final ByteBuffer tmp = acquire();
try {
- if (tmp.remaining() >= 12) {
- // We must synchronize
- synchronized (tmp) {
- int spos = tmp.position();
- tmp.putLong(addr);
- tmp.putInt(0);
- if (checker != null) {
- // update the checksum (no side-effects on [data])
- ByteBuffer chkBuf = tmp.asReadOnlyBuffer();
- chkBuf.position(spos);
- chkBuf.limit(tmp.position());
- checker.update(chkBuf);
- }
- }
- }
+ if (tmp.remaining() >= 12) {
+ /*
+ * Note: We must synchronize before having a side effect on
+ * position. Also see write(...) which is synchronized on
+ * the buffer during critical sections which have a side
+ * effect on the buffer position.
+ */
+ synchronized (tmp) {
+ final int spos = tmp.position();
+ tmp.putLong(addr);
+ tmp.putInt(0);
+ if (checker != null) {
+ // update the checksum (no side-effects on [data])
+ final ByteBuffer chkBuf = tmp.asReadOnlyBuffer();
+ chkBuf.position(spos);
+ chkBuf.limit(tmp.position());
+ checker.update(chkBuf);
+ }
+ } // synchronized(tmp)
+ }
} finally {
release();
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-09-03 11:48:14
|
Revision: 3503
http://bigdata.svn.sourceforge.net/bigdata/?rev=3503&view=rev
Author: thompsonbry
Date: 2010-09-03 11:48:08 +0000 (Fri, 03 Sep 2010)
Log Message:
-----------
toned down the message in fastload.properties.
Modified Paths:
--------------
trunk/bigdata-sails/src/samples/com/bigdata/samples/fastload.properties
Modified: trunk/bigdata-sails/src/samples/com/bigdata/samples/fastload.properties
===================================================================
--- trunk/bigdata-sails/src/samples/com/bigdata/samples/fastload.properties 2010-09-03 11:16:49 UTC (rev 3502)
+++ trunk/bigdata-sails/src/samples/com/bigdata/samples/fastload.properties 2010-09-03 11:48:08 UTC (rev 3503)
@@ -1,7 +1,8 @@
-# Be very careful when you use this configuration! This turns off incremental
-# inference for load and retract, so you must explicitly force these operations,
-# which requires punching through the SAIL layer. Of course, if you are not
-# using inference then this is just the ticket and quite fast.
+# This configuration turns off incremental inference for load and retract, so
+# you must explicitly force these operations if you want to compute the closure
+# of the knowledge base. Forcing the closure requires punching through the SAIL
+# layer. Of course, if you are not using inference then this configuration is
+# just the ticket and is quite fast.
# set the initial and maximum extent of the journal
com.bigdata.journal.AbstractJournal.initialExtent=209715200
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mar...@us...> - 2010-09-03 11:16:55
|
Revision: 3502
http://bigdata.svn.sourceforge.net/bigdata/?rev=3502&view=rev
Author: martyncutcher
Date: 2010-09-03 11:16:49 +0000 (Fri, 03 Sep 2010)
Log Message:
-----------
freeCurrentDeferrals should not update lastReleaseTime
Modified Paths:
--------------
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2010-09-03 11:14:17 UTC (rev 3501)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2010-09-03 11:16:49 UTC (rev 3502)
@@ -60,6 +60,7 @@
import com.bigdata.journal.Options;
import com.bigdata.journal.RWStrategy.FileMetadataView;
import com.bigdata.quorum.Quorum;
+import com.bigdata.util.ChecksumError;
import com.bigdata.util.ChecksumUtility;
/**
@@ -1119,7 +1120,9 @@
}
+ private long immediateFreeCount = 0;
private void immediateFree(final int addr, final int sze) {
+
switch (addr) {
case 0:
case -1:
@@ -2629,7 +2632,7 @@
}, 5L, TimeUnit.MILLISECONDS);
- if (false) freeAllDeferrals(freeTime);
+ freeCurrentDeferrals(freeTime);
} catch (RuntimeException e) {
// fine, will try again later
} catch (Exception e) {
@@ -2644,12 +2647,12 @@
*
* @param txnRelease - the max release time
*/
- protected void freeAllDeferrals(long txnRelease) {
- System.out.println("freeAllDeferrals");
+ protected void freeCurrentDeferrals(long txnRelease) {
m_deferFreeLock.lock();
try {
- if (m_lastDeferredReleaseTime <= txnRelease) {
+ if (m_rb.getLastCommitTime() <= txnRelease) {
+// System.out.println("freeCurrentDeferrals");
final Iterator<Integer> curdefers = m_currentTxnFreeList.iterator();
while (curdefers.hasNext()) {
final int rwaddr = curdefers.next();
@@ -2664,8 +2667,6 @@
}
}
m_currentTxnFreeList.clear();
-
- m_lastDeferredReleaseTime = txnRelease;
}
} finally {
m_deferFreeLock.unlock();
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mar...@us...> - 2010-09-03 11:14:23
|
Revision: 3501
http://bigdata.svn.sourceforge.net/bigdata/?rev=3501&view=rev
Author: martyncutcher
Date: 2010-09-03 11:14:17 +0000 (Fri, 03 Sep 2010)
Log Message:
-----------
Fix writeCache corruption on clearAddrMap
Modified Paths:
--------------
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/io/writecache/WriteCache.java
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/io/writecache/WriteCacheService.java
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/io/writecache/WriteCache.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/io/writecache/WriteCache.java 2010-09-03 00:27:45 UTC (rev 3500)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/io/writecache/WriteCache.java 2010-09-03 11:14:17 UTC (rev 3501)
@@ -821,9 +821,9 @@
if (useChecksum) {
- final int chk = tmp.getInt(pos + md.recordLength - 4);
+ final int chk = tmp.getInt(pos + reclen);
- if (chk != ChecksumUtility.threadChk.get().checksum(b, 0/* offset */, b.length)) {
+ if (chk != ChecksumUtility.threadChk.get().checksum(b, 0/* offset */, reclen)) {
// Note: [offset] is a (possibly relative) file offset.
throw new ChecksumError("offset=" + offset);
@@ -1713,6 +1713,9 @@
* write also and the buffer will be flushed either on commit or a
* subsequent write.
*
+ * A problem previously existed with unsynchronized access to the ByteBuffer.
+ * Resulting in a conflict over the position() and buffer corruption.
+ *
* @param addr
* The address of a cache entry.
*
@@ -1735,15 +1738,18 @@
final ByteBuffer tmp = acquire();
try {
if (tmp.remaining() >= 12) {
- int spos = tmp.position();
- tmp.putLong(addr);
- tmp.putInt(0);
- if (checker != null) {
- // update the checksum (no side-effects on [data])
- ByteBuffer chkBuf = tmp.asReadOnlyBuffer();
- chkBuf.position(spos);
- chkBuf.limit(tmp.position());
- checker.update(chkBuf);
+ // We must synchronize
+ synchronized (tmp) {
+ int spos = tmp.position();
+ tmp.putLong(addr);
+ tmp.putInt(0);
+ if (checker != null) {
+ // update the checksum (no side-effects on [data])
+ ByteBuffer chkBuf = tmp.asReadOnlyBuffer();
+ chkBuf.position(spos);
+ chkBuf.limit(tmp.position());
+ checker.update(chkBuf);
+ }
}
}
} finally {
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/io/writecache/WriteCacheService.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/io/writecache/WriteCacheService.java 2010-09-03 00:27:45 UTC (rev 3500)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/io/writecache/WriteCacheService.java 2010-09-03 11:14:17 UTC (rev 3501)
@@ -1944,7 +1944,7 @@
final WriteCache cache = recordMap.remove(offset);
if (cache == null)
return;
- acquireForWriter(); // in case current
+ final WriteCache cur = acquireForWriter(); // in case current
debugAddrs(offset, 0, 'F');
try {
cache.clearAddrMap(offset);
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-09-03 00:27:56
|
Revision: 3500
http://bigdata.svn.sourceforge.net/bigdata/?rev=3500&view=rev
Author: thompsonbry
Date: 2010-09-03 00:27:45 +0000 (Fri, 03 Sep 2010)
Log Message:
-----------
Reorganized the operators and some utility classes a bit.
Added an a conditional routing operator and a test suite for it.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/PipelineOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/QuoteOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/aggregation/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/RunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestPipelineUtility.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/join/TestPipelineJoin.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/util/concurrent/TestAll.java
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bset/
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bset/ConditionalRoutingOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bset/CopyBindingSetOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bset/Union.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/accesspath/ThreadLocalBufferFactory.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/accesspath/UnsyncLocalOutputBuffer.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/util/concurrent/Haltable.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/bset/
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/bset/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/bset/TestConditionalRoutingOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/bset/TestCopyBindingSets.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/bset/TestUnionBindingSets.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/util/concurrent/TestHaltable.java
Removed Paths:
-------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/PipelineStartOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/Union.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bop-notes.txt
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/ThreadLocalBufferFactory.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/UnsyncLocalOutputBuffer.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/aggregation/TestUnionBindingSets.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/Haltable.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestHaltable.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java 2010-09-02 22:24:22 UTC (rev 3499)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java 2010-09-03 00:27:45 UTC (rev 3500)
@@ -162,7 +162,7 @@
* @throws NullPointerException
* if the argument is <code>null</code>.
*/
- protected BOpBase(final BOpBase op) {
+ public BOpBase(final BOpBase op) {
// deep copy the arguments.
args = deepCopy(op.args);
// deep copy the annotations.
@@ -173,7 +173,7 @@
* @param args
* The arguments to the operator.
*/
- protected BOpBase(final BOp[] args) {
+ public BOpBase(final BOp[] args) {
this(args, null/* annotations */);
@@ -185,7 +185,7 @@
* @param annotations
* The annotations for the operator (optional).
*/
- protected BOpBase(final BOp[] args,
+ public BOpBase(final BOp[] args,
final Map<String, Object> annotations) {
if (args == null)
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/PipelineOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/PipelineOp.java 2010-09-02 22:24:22 UTC (rev 3499)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/PipelineOp.java 2010-09-03 00:27:45 UTC (rev 3500)
@@ -41,9 +41,14 @@
*
* @param <E>
* The generic type of the objects processed by the operator.
- *
+ *
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
* @version $Id$
+ *
+ * @todo It is too confusion to have an interface hierarchy which is separate
+ * from the class hierarchy for the operators. Therefore roll this
+ * interface into {@link AbstractPipelineOp} and then rename that class to
+ * {@link PipelineOp}
*/
public interface PipelineOp<E> extends BOp {
Deleted: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/PipelineStartOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/PipelineStartOp.java 2010-09-02 22:24:22 UTC (rev 3499)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/PipelineStartOp.java 2010-09-03 00:27:45 UTC (rev 3500)
@@ -1,117 +0,0 @@
-/**
-
-Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
-/*
- * Created on Aug 25, 2010
- */
-
-package com.bigdata.bop;
-
-import java.util.Map;
-import java.util.concurrent.Callable;
-import java.util.concurrent.FutureTask;
-
-import com.bigdata.bop.engine.BOpStats;
-import com.bigdata.relation.accesspath.IAsynchronousIterator;
-import com.bigdata.relation.accesspath.IBlockingBuffer;
-
-/**
- * This operator copies its source to its sink. It is used to feed the first
- * join in the pipeline. The operator should have no children but may be
- * decorated with annotations as necessary.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
- */
-public class PipelineStartOp extends BindingSetPipelineOp {
-
- /**
- *
- */
- private static final long serialVersionUID = 1L;
-
- /**
- * Deep copy constructor.
- *
- * @param op
- */
- public PipelineStartOp(PipelineStartOp op) {
- super(op);
- }
-
- /**
- * Shallow copy constructor.
- *
- * @param args
- * @param annotations
- */
- public PipelineStartOp(BOp[] args, Map<String, Object> annotations) {
- super(args, annotations);
- }
-
- public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
-
- return new FutureTask<Void>(new CopyTask(context));
-
- }
-
- /**
- * Copy the source to the sink.
- */
- static private class CopyTask implements Callable<Void> {
-
- private final BOpStats stats;
-
- private final IAsynchronousIterator<IBindingSet[]> source;
-
- private final IBlockingBuffer<IBindingSet[]> sink;
-
- CopyTask(final BOpContext<IBindingSet> context) {
-
- stats = context.getStats();
-
- this.source = context.getSource();
-
- this.sink = context.getSink();
-
- }
-
- public Void call() throws Exception {
- try {
- while (source.hasNext()) {
- final IBindingSet[] chunk = source.next();
- stats.chunksIn.increment();
- stats.unitsIn.add(chunk.length);
- sink.add(chunk);
- stats.chunksOut.increment();
- stats.unitsOut.add(chunk.length);
- }
- return null;
- } finally {
- sink.close();
- }
- }
-
- }
-
-}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/QuoteOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/QuoteOp.java 2010-09-02 22:24:22 UTC (rev 3499)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/QuoteOp.java 2010-09-03 00:27:45 UTC (rev 3500)
@@ -38,10 +38,10 @@
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
* @version $Id$
*
- * @todo If we do nothing then {@link QuoteOp} will already prevent the
- * evaluation of its child operand by the expediency of not defining its
- * own evaluation semantics. Alternatively, we could add
- * <code>eval():Op</code> using an appropriate evaluation interface.
+ * @todo I think that we can avoid quoting operators by using annotations (for
+ * some cases) and through explicit interaction between operators for
+ * others (such as between a join and a predicate). If that proves to be
+ * true then this class will be dropped.
*/
public class QuoteOp extends BOpBase {
Deleted: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/Union.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/Union.java 2010-09-02 22:24:22 UTC (rev 3499)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/Union.java 2010-09-03 00:27:45 UTC (rev 3500)
@@ -1,135 +0,0 @@
-/**
-
-Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
-/*
- * Created on Aug 18, 2010
- */
-
-package com.bigdata.bop.aggregation;
-
-import java.util.Map;
-import java.util.concurrent.Callable;
-import java.util.concurrent.FutureTask;
-
-import com.bigdata.bop.BOpContext;
-import com.bigdata.bop.BindingSetPipelineOp;
-import com.bigdata.bop.IBindingSet;
-import com.bigdata.bop.engine.Haltable;
-import com.bigdata.bop.join.PipelineJoin;
-import com.bigdata.rdf.rules.TMUtility;
-import com.bigdata.relation.RelationFusedView;
-
-/**
- * The union of two or more {@link BindingSetPipelineOp} operators.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
- *
- * @todo I have some basic questions about the ability to use a UNION of two
- * predicates in scale-out. I think that this might be more accurately
- * modeled as the UNION of two joins. That is, rather than:
- *
- * <pre>
- * JOIN( ...,
- * UNION( foo.spo(A,loves,B),
- * bar.spo(A,loves,B) )
- * )
- * </pre>
- *
- * using
- *
- * <pre>
- * UNION( JOIN( ..., foo.spo(A,loves,B) ),
- * JOIN( ..., bar.spo(A,loves,B) )
- * )
- * </pre>
- *
- * which would be a binding set union rather than an element union.
- *
- * @todo The union of access paths was historically handled by
- * {@link RelationFusedView}. That class should be removed once queries
- * are rewritten to use the union of joins.
- *
- * @todo The {@link TMUtility} will have to be updated to use this operator
- * rather than specifying multiple source "names" for the relation of the
- * predicate.
- *
- * @todo The FastClosureRuleTask will also need to be updated to use a
- * {@link Union} over the joins rather than a {@link RelationFusedView}.
- */
-public class Union extends BindingSetPipelineOp {
-
- /**
- *
- */
- private static final long serialVersionUID = 1L;
-
- /**
- * @param args
- * Two or more operators whose union is desired.
- * @param annotations
- */
- public Union(final BindingSetPipelineOp[] args,
- final Map<String, Object> annotations) {
-
- super(args, annotations);
-
- if (args.length < 2)
- throw new IllegalArgumentException();
-
- }
-
- public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
-
- return new FutureTask<Void>(new UnionTask(this, context));
-
- }
-
- /**
- * Pipeline union impl.
- *
- * FIXME All this does is copy its inputs to its outputs. Since we only run
- * one chunk of input at a time, it seems that the easiest way to implement
- * a union is to have the operators in the union just target the same sink.
- */
- private static class UnionTask extends Haltable<Void> implements Callable<Void> {
-
- public UnionTask(//
- final Union op,//
- final BOpContext<IBindingSet> context
- ) {
-
- if (op == null)
- throw new IllegalArgumentException();
- if (context == null)
- throw new IllegalArgumentException();
- }
-
- public Void call() throws Exception {
- // TODO Auto-generated method stub
- throw new UnsupportedOperationException();
- }
-
- }
-
-}
Deleted: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bop-notes.txt
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bop-notes.txt 2010-09-02 22:24:22 UTC (rev 3499)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bop-notes.txt 2010-09-03 00:27:45 UTC (rev 3500)
@@ -1,540 +0,0 @@
-- Add IElement interface with Object:get(int index) to pull off the
- fields from the element by index position. Use this to make
- RDFJoinNexus#bind() and copyValues() generic. We can just do a cast
- to IElement, but we could also change the generic type constraint on
- IRelation from <E> to IRelation<E extends IElement>. But we can
- just force the cast for now and not have to update all of those
- generic constraints.
-
-- Review annotation names and defaults. Make sure that the annotation
- names are all in appropriate namespaces. The namespaces should
- probably be the interface or class of the operator which defines
- that annotation.
-
-- RejectAnythingSameAsSelf appears to be assuming termIds rather than
- IVs.
-
-- Get rid of the concept of a relation view (two or more relations
- named by a given predicate) in favor of the UNION of the predicates,
- which is basically a UNION of their access paths.
-
-- Expanders will eventually need to be replaced by either shard-wise
- expanders (where possible) or query time materialization of various
- inferences, e.g., using magic sets or other query rewrite
- techniques.
-
-- IRelation#getAccessPath(IIndexManager,IPredicate). Raise this onto
- onto IRelation. It is the shard aware version. There is also a
- version getAccessPath(IPredicate) without the IIndexManager
- parameter which is used for local indices and for RMI based access
- to a scale-out index.
-
-- IRelation: DDL Support
- - Iterator<IKeyOrder<E>> getKeyOrders();
- - IKeyOrder<E> getKeyOrder(IPredicate<E> p);
- - IKeyOrder<E> getPrimaryKeyOrder();
- - IIndex getIndex(IKeyOrder)
-
-- IKeyOrder: DDL Support (reconcile with ITupleSerializer)
- - byte[] getFromKey(IKeyBuilder keyBuilder, IPredicate<ISPO> predicate);
- - byte[] getToKey(IKeyBuilder keyBuilder, IPredicate<ISPO> predicate);
- - byte[] encodeKey(IKeyBuilder keyBuilder, E e);
- - E decodeKey(byte[] key);
-
-- Elements: Add self-description of data to "elements". Harmonize with
- binding sets.
- - Iterator<Column> getFields();
-
-- Column:
- - getIndex();
- - getName();
- - getDatatype();
- - getForeignKeys();
- ...
-
-- Implement lexicon joins.
-
-- Implement spatial joins.
-
-- Nested optionals (conditional routing of joins).
-
-- Make the free text index a "resource" rather than a relation?
-
-- Use blocking queues with poison objects and chunking or modify
- BlockingBuffer to raise up the LinkedBlockingDeque into our code
- base.
-
-- Support tree rewrites using a pattern language.
-
- - http://functionaljava.org/, Functional Java.
-
- - tuprolog: http://alice.unibo.it/xwiki/bin/view/Tuprolog/, LGPL, Java.
-
- - http://scala.sygneca.com/code/mini-kanren, Example of constraint
- based programming in Scala. http://scala.sygneca.com/. BSD
- license. http://scala.sygneca.com/code/compressed-executable-jar.
-
- - PrologCafe: http://kaminari.istc.kobe-u.ac.jp/PrologCafe/, Java.
- License is not specified. Authors are in Japan. Appears to be
- two people.
-
- - YAP : http://www.dcc.fc.up.pt/~vsc/Yap/, Perl Artistic license, C.
-
- - XSB : http://xsb.sourceforge.net/, LGPL, C.
-
-BOp
- - Serializable
- - Cloneable
- - Unmodifiable
- - arity():int
- - args():List<BOp>
- - annotations():Map<String,Object>
-.BOpList: Used to represent lists of operators. E.g., IN(var,{graphIds}).
- - values():List<BOp>
-.IConstantOrVariable
- - isConstant()
- - isVariable()
- - get() // iff isConstant()
- - getName() // iff isVariable()
-..IConstant
-..IVariable
-.IOpN
-..IPredicate(name,arg...)[shardId:int;optional:boolean;constraint:IConstraint[],expander]
-
-- Distributed query execution pattern:
-
- The historical pipeline join propagated evaluation from left to
- right. This needs to be revisited in now that we are dealing with
- operator trees. Operator trees lend themselves naturally to top-down
- evaluation. While I think that we can do top-down evaluation of the
- operator tree for scaleup, the distributed query execution logic is
- more complicated and top-down evaluation is not compatible with
- distributed evaluation because joins must run for different shards
- based on the partition identifier associated with each bindingSet[]
- they receive.
-
- What we been doing is pushing binding sets along propagating joins
- at specific shards onto nodes together with those binding sets.
- This was a left-to-right evaluation strategy when the IRule was just
- an ordered array of tails on which we needed to read. However, now
- that we have an operator tree it would correspond to a bottom up
- evaluation of a left-deep tree where the right operands were always
- access path reads. That makes my head hurt just a bit when I
- consider that the bottom up evaluation would also be "partial" as
- binding sets appear.
-
- For a given incoming IBindingSet[] chunk we will continue to do
- exactly what we have been doing, but the surrounding logic needs to
- be replaced. Starting a "join" (at least for piplined scale-out)
- needs to merely register a join task factory that will handle
- binding sets as they arrive.
-
- I am thinking that the way to handle this is to send the query
- identifier, join operation identifier, and partition identifier
- along with the client's proxy object, the senders operator
- identifier, and the senders's service identifier. If the node (a
- data service) which receives that RMI message has not seen the query
- yet it uses RMI against the client's proxy to fetch the query and
- then "starts" the query on that node. Starting a query would merely
- allow execution of the operators described in the query plan on the
- node once they had sufficient data to run. The current pipeline
- join is a chunk-wise nested index join. It runs one pass each time
- it has a chunk of binding sets for some shard. Query termination
- would be controlled by the client. It would instruct all nodes
- known to be running the query to halt execution for that query.
- Even if a node misses that message, it will find out when it tries
- to send intermediate results to another node that the query was
- cancelled.
-
- Per shard locks, tasks, and threads.
-
- When running within a single Journal, the query plan is executed by
- one task which holds all of the necessary locks. Those locks are
- acquired up front by an inspection of the query plan to determine
- which indices are needed [actually, it may just be using the
- unisolated read/write index and otherwise historical views w/o
- locks.]
-
- There are issues relating to execution of the joins under the
- concurrency manager, both in order to have access to the correct
- IIndexManager and in order to managing reads and writes against the
- unisolated indices by acquiring the appropriate locks. The way
- things work right now the upstream join tests a cache for the
- downstream join task for a given shard. If there is a cache miss,
- it sends a factory task which uses a singleton pattern to start a
- join task executing with the appropriate locks under the concurrency
- manager and then returns the proxy for that join task to the caller.
- This guarantees that each join task has the correct locks, but it
- does so at the expense of running one thread per join task. It will
- be difficult to get around this one task per shard per join
- constraint without going directly to the lock manager with the shard
- lock requests. [Or just submitting a per binding set chunk task to
- the ConcurrencyManager, which might not be that bad if the chunks
- are chunky.]
-
- ==> Given this, maybe it would be easiest to adapt the current join
- execution to allow optional gotos by paying close attention to the
- termination condition for the query? We could then refactor to
- support BOPs within the same general control logic. A DISTINCT
- filter could be yet another custom RMI thing layered directly into
- the join logic.
-
- Rather than defining an eval() method for each operator, we have
- standoff interpretation of the pipeline operators (whether for
- binding sets, elements, or solutions). The query plan could
- encapsulate the local versus distributed execution with annotations
- on the operators rather than interposing operators and those
- annotations would be used to wrap the sink with one which marshells
- the outputs onto NIO buffers.
-
- - Pipelined chunk-wise nested index join. This is the existing
- join algorithm. For each binding set chunk received on a node to
- be joined with a given shard, we execute that chunk wise join and
- emit the intermediate results. [When the join is optional, we
- have an optional target and we send the binding sets which do not
- join to that optional target.]
-
- - multi-block io pipelined join. This is a version of the piplined
- chunk-wise nested index join which accumulates much larger chunks
- (mega chunks) of binding sets (potentially all intermediate
- results) and then computes the join of that using the
- operator-at-a-time approach for that mega chunk. The tradeoff
- between this join and the pure operator at a time join is that we
- can keep the intermediate results off the disk using this
- approach but we may have to read the shard multiple times.
-
- - operator-at-a-time shard wise multi-block-io join. This join
- proceeds an operator at a time. Once the producer is done, it
- computes the entire join using the intermediate results from the
- prior join and a single multi-block IO pass over the shard view.
-
- A tuple read from the shard joins if there exists a binding set
- which is consistent with that tuple. For example, given:
-
- :- ..., POS(A loves B), SPO(B loves C).
-
- and the following intermediate results from the POS shard:
-
- B0:[A=John, B=Mary, ...]
- B1:[A=Mary, B=Paul, ...]
- B2:[A=Paul, B=Leon, ...]
- B3:[A=Leon, B=Paul, ...]
-
- and the following tuples read from the SPO shard:
-
- T0:(John loves Mary)
- T1:(Mary loves Paul)
- T2:(Paul loves Leon)
- T3:(Leon loves Paul)
-
- then we have the following joins:
-
- (T2, B3) // T2:(Paul loves Leon) with B3:[A=Leon, B=Paul, ...].
- (T3, B2) // T3:(Leon loves Leon) with T2:[A=Paul, B=Leon, ...].
-
- There are several ways to manipulate the intermediate results to
- setup the join:
-
- Merge join: Merge sort the binding sets based on the relevant
- bound values (A,B) and do an ordered scan of the binding sets and
- the shard, emitting results which join.
-
- Hash join: Create a temporary hash table (backed by disk) whose
- keys are the relevant bound values for (A,B) and whose values are
- either: (a) the associated binding sets (there can be more than
- one per (A,B) and all must be preserved; or (b) the #of offsets
- and the offsets into an array or a persistent store of the
- associated binding sets.
-
- Based on our need for an N-way merge sort, I would think that the
- merge join will offer the most benefit initially and is likely to
- be better organized for execution in memory (it has locality at
- more memory tiers).
-
-- Scaleup query execution pattern:
-
- Queries which are not distributed can use top-down evaluation of the
- left deep pipeline joins.
-
-
-- BOp execution.
-
- (***) Implement all BOps for which there is existing functionality.
-
- PipelineJoin: Join binding set producer with local access path
- using asBound Predicate. This should be **TRIVIAL** and can
- be unit tested against mock objects for the left and right
- hand operands. (The vast majority of the logic in JoinTask
- dealt with coordinating the pipeline operations. It should
- even be trivial to put in "gotos" for failed optional joins.
- Defer the logic for mapShards until the core joins are
- running.)
-
- Predicate: read on as bound access path (IChunked(Ordered)Iterator).
-
- Work through evaluation for BOps, perhaps using BOpUtility,
- and definately using unit tests. We should be able to unit
- test correct termination, abnormal termination, etc. against
- the EDS [move to test suite.]
-
- How are the rule execution statistics going to be passed
- around? How will we get visibility into the running rules and
- their current execution statistics (especially for long
- running rules)? How can an operator cancel long running
- rules?
-
- Implement optional / conditional binding set routing to
- ancestors in the BOp tree (parent and eventual parent).
-
- Implement map shards with RMI then convert to buffers. Figure
- out how to label BOps as senders/receivers.
-
- Implement map nodes. The ordered list of nodes should be
- reused for each MapNodes operator.
-
- All pipeline operators can specify the pipeline annotations
- (chunkSize, chunksOfChunks, etc).
-
- (***) Harmonize IElementFilter, IConstraint, BOp, IChunkConverter,
- PipelineEval.
-
- Work through how to moving the binding sets and related stuff
- around, including when running it into a native heap buffer
- and when just draining a queue, blocking buffer's iterator, a
- bulk filter (DISTINCT, EXISTS, !EXISTS), bulk completion, etc.
-
- Asynchronous production of binding sets for vectored pipeline
- evaluation. Evaluation should also sets future on buffer as
- side effect or collects Future's of the operator tree.
-
- - Future<Void> eval(IJoinNexus,buffer<BindingSet[]>);
-
- Evaluation of a predicate, obtaining an access path. The
- caller can then wrap the result and do range counts, high
- level iterators with filters, low level tuple iterators,
- sample the index / view, etc.
-
- IPredicate<E>:
-
- - Future<IAccessPath<E>> eval(IJoinNexus,IBindingSet);
-
- IConstant<T>:
-
- - T eval(); // return the constant's value.
-
- IVariable<T>:
-
- - T eval(IBindingSet); // return the bound value (vs variable name?)
-
- Striterator patterns return the wrapped iterator. The wrapper
- iterator is then applied an (eventual) parent bop. This
- pattern can be used with the CTC Striterator,
- IChunked(Ordered)Iterator, and probably ITupleIterator (with
- the FilterConstructor, which could just be some bops).
-
- - wrapperItr eval(itr)
-
- - IElementFilter is element-at-a-time filtering of materialized tuples.
-
- - IConstraint is IBindingSet-at-a-time filtering.
-
- - BOp.PipelineEval is IBindingSet at a time evaluation, but it is
- designed for chunked pipelineing of the binding sets.
-
- - IChunkConverter is chunk at a time evaluation and has the sense of
- returning a chunk for each chunk consumed. That could be used to
- return the bit map from the DISTINCT operator, which is something
- that is otherwise missing from the BOp.PipelineEval interface.
-
- - We need a "batch-oriented" constraint for IN (due to compilation
- of the set into an efficient representation prior to execution)
- and DISTINCT (it has to batch binding sets in order to amortize
- the cost of generating the sort keys and/or chunking up the
- results).
-
- - Reconcile Distinct, DistinctSPOIterator, etc.
-
- - The IN filters should probably be turned into JOINs against an in
- memory IBindingSet[] source except in those cases where we will
- have locality in the index for the variable in the IN constraint.
- At present, IN is only used by the MatchRule. However, it could
- also be used for named graphs and default graphs. There are
- specialized filters for those purposes SPARQL data sets right now
- {InGraphBinarySearchFilter, InGraphHashSetFilter}, but I think
- that this is all pretty general purpose stuff.
-
-IElementFilter: element at a time filtering (does not scale-out).
-.ElementFilter: Unused (discard).
-.SameVariableConstraint: precompiles some private state (ok iff immutable).
-.SolutionFilter: applies filter to the visited elements; related to rule's head...
-.SPOFilter: returns false if not an ISPO.
- - canAccept(Object):boolean
-..DoNotAddFilter
-..ExplicitSPOFilter
-..InferredSPOFilter
-..InGraphBinarySearchFilter: duplicates IN filter?
-..InGraphHashSetFilter: duplicates IN filter?
-
-IChunkConverter<E,F>: bulk conversion in/out (scales-out since RMI can be chunky).
-.BulkCompleteConverter
-.BulkFilterConverter
-.DistinctFilter: Reconcile with Distinct, ISortKeyBuilder, IN, InGraphHashSetFilter, etc.
-.HitConverter
-
-- Evaluation types:
-
- IRelation(aka namespace)
- IDatabase(aka namespace / AbstractTripleStore)
- IIndex, ILocalBTreeView, BTree, IndexSegment, FusedView,
- IMap? (aka hash map, DHT)
- ITable?, IBat?
- File (flat file in/out),
-
- Bloomfilter
-
- E[], BlockingBuffer<E[]>.iterator()
-
- IBindingSet[],
- IChunkedIterator<IBindingSet[]>,
- BlockingBuffer<IBindingSet[]>.iterator(),
-
- ISolution[], etc.
-
- - Life cycle management of resources local to the operator
- execution. In some cases, resources must be eventually released,
- much like "finally{}". This has to be managed in a distributed
- environment where there is no "stack" to be unwound.
-
- - Explicit management of query priority, buffers, timeout, etc.
-
- - Visibility into long running queries.
-
- - Left-deep operator trees for pipelined execution.
-
- - newInstance(IJoinNexus) : Future<T>. Tasks will run on
- ForkJoinPools vs Executors depending on whether they support light
- weight asynchronous operations all the way down or need to use a
- thread per disk IO. However, if the disk IO thread pool is global
- to a DataService and we use queues for disk IO requests, then we
- can always use the ForkJoinPool for operators (queuing a request
- is consistent with fork/join, right?).
-
-- Add IOp library. Some operators will be specific to standalone (a
- ConcurrentHashMap based distinct) or scale-out (e.g., a DHT based
- distinct).
-
- - RuntimeQueryOptimization(JoinGraph) - Execute the join graph using
- interleaved query optimization and query execution.
-
- Evaluate for both selective and unselective joins. Note that
- sampling can result in all relevant tuples being materialized, at
- which point chain sampling will perform the actual join and
- materialize the real intermediate result set.
-
- - Predicate. This corresponds to an access path with the current
- bindings.
-
- - SCAN(fromKey,toKey) (local is trivial)
-
- - SCAN(partitionId,fromKey,toKey) (distributed requires the
- partitionId and the local IIndexManager must be set on the
- DataService where the operation will execute).
-
- - Sample(Predicate,limit):(E[]). Sample tuples from a relation which
- satisify the predicate. Returns the sampled elements.
-
- - SampleOp(IOp,limit):(E[],est). Sample output from an operation,
- halting if the limit is satisfied. Returns the sample and the
- estimated cardinality of the operation.
-
- - MapNodes(f,BS[],DS[]). Hash partition mapping of some binding sets
- using a function across a set of nodes (typically data service
- nodes, but that is not required).
-
- - MapShards(BS[],tail[i+1]). Maps binding sets across shards on
- which we need to read for the next join predicate. The operator
- is associated with a logical port and a maximum buffer allocation.
-
- @todo If we want to assign indices to variables for binding sets
- then that needs to be done with reference to a total order over
- the "rule". With the generalization to an operator tree, the fact
- that we strip out variables from the binding set when they are no
- longer in use, and the possibility of permutations over the as yet
- unevaluated parts of the operator, that mapping needs to be quite
- explicit. Perhaps it could be part of the {@link IJoinNexus} or
- raised into the root node of the operator tree.
-
- - Receive()[queryId, port]. Receive binding sets a logical port for
- a given query. The receiver is given a maximum buffer allocation.
- If the buffers are full, it uses flow control to halt the sender
- (NACKs the request to send a buffer to the receiver). Can be used
- with MapNodes or MapShards. [Can be tested within a single JVM.]
-
- - IN(var,E[]). Pipeline join operator binds the variable to each of
- the elements in the array in turn for each binding set presented
- to the operator. [This is useful for RDF dataset constructs.]
-
- - JOIN(IPredicate). Pipeline join operator. It accepts binding
- sets on one side and joins then against the (local) access path
- for the specified predicate.
-
- - StarJoin(...).
-
- - ? HashJoin? Join two arguments, each of which evaluates to a
- collection of binding sets. This probably can't be pipelined
- unless one of the arguments is relatively small and hence can be
- fully materialized. Look at how to handle cases where both
- arguments have large result sets.
-
- - ? Execute a correlated subquery and join of the results from that
- subquery against with the each binding set presented to the
- subquery?
-
- - DISTINCT (local, concurrent hash map).
-
- - DISTINCT (distributed hash table).
-
- - SORT (local, Arrays.sort(), radix sort).
-
- - SORT (distributed, N-way merge sort).
-
- - CONSTRUCT (create an element from a binding set).
-
- - INSERT. Insert elements into a relation (local, sharded uses
- buffers to move the constructed elements; behavior can use either
- unisolated writes, eventually consistent unisolated writes, or
- full transaction isolation).
-
- - REMOVE. Removes elements from a relation (sharded uses buffers to
- move the constructed elements).
-
- Note: Handle remove of elements matching a predicate elements by
- first executing the predicate to select the elements and then
- removing the elements using the same kinds of mechansims which are
- used for insert.
-
-============================================================
-
-Much of the complexity of the current approach owes itself to having to run a separate task for each join for each shard in order to have the appropriate lock when running against the unisolated shard view. This also means that the join task is running inside of the concurrency manager and hence has the local view of the shard.
-
-The main, and perhaps the only, reason why we run unisolated rules is during closure, when we query against the unisolated indices and then write the entailments back on the unisolated indices.
-
-Supporting closure has always been complicated. This complexity is mostly handled by ProgramTask#executeMutation() and AbstractTripleStore#newJoinNexusFactory() which play games with the timestamps used to read and write on the database, with commit points designed to create visibility for tuples written by a mutation rule, and with the automated advance of the read timestamp for the query in each closure pass in order to make newly committed tuples visible to subsequent rounds of closure. For scale-out, we do shard-wise auto commits so we always have a commit point which makes each write visible and the read timestamp is actually a read-only transaction which prevents the historical data we need during a closure round from being released as we are driving updates onto the federation. For the RWStore, we are having a similar problem (in the HA branch since that is where we are working on the RWStore) where historically allocated records were being released as writes drove updates on the indices. Again, we "solved" the problem for the RWStore using a commit point followed by a read-only transaction reading on that commit point to hold onto the view on which the next closure round needs to read (this uncovered a problem with the RWStore and transaction service interaction which Martyn is currently working to resolve through a combination of shadow allocators and deferred deletes which are processed once the release time is advanced by the transaction service).
-
-The WORM does not have some of these problems with closure because we never delete history, so we do not need to create a commit point and a read-behind transaction. However, the WORM would have problems with concurrent access to the unisolated indices except that we hack that problem through the transparent use of the UnisolatedReadWriteIndex, which allows multiple threads to access the same unisolated index view using a read/write lock pattern (concurrent readers are allowed, but there is only one writer and it has exclusive access when it is running). This works out because we never run closure operations against the WORM through the concurrency manager. If we did, we would have to create a commit point after each mutation and use a read-behind transaction to prevent concurrent access to the unisolated index.
-
-The main advantage that I can see of the current complexity is that it allows us to do load+closure as a single operation on the WORM, resulting in a single commit point. This makes that operation ACID without having to use full read/write transactions. This is how we gain the ACID contract for the standalone Journal in the SAIL for the WORM. Of course, the SAIL does not have that contract for the RWStore because we have to do the commit and read-behind transaction in order to have visibility and avoid concurrent access to the unisolated index (by reading behind on the last commit point).
-
-I think that the reality is even one step more complicated. When doing truth maintenance (incremental closure), we bring the temporary graph to a fixed point (the rules write on the temp store) and then apply the delta in a single write to the database. That suggests that incremental truth maintenance would continue to be ACID, but that database-at-once-closure would be round-wise ACID.
-
-So, I would like to suggest that we break ACID for database-at-once-closure and always follow the pattern of (1) do a commit before each round of closure; and (2) create a read-behind transaction to prevent the release of that commit point as we drive writes onto the indices. If we follow this pattern then we can write on the unisolated indices without conflict and read on the historical views without conflict. Since there will be a commit point before each mutation rule runs (which corresponds to a closure round), database-at-once-closure will be atomic within a round, but will not be a single atomic operation. Per above, I think that we would retain the ACID property for incremental truth maintenance against a WORM or RW mode Journal.
-
-----
-
-The advantage of this proposal (commit before each mutation rule and run query against a read-behind transaction) is that this could enormously simplify how we execute joins.
-
-Right now, we use a factory pattern to create a join task on each node for each shard for which that node receives binding sets for a query. The main reason for doing this is to gain the appropriate lock for the unisolated index. If we never run a query against the unisolated index then we can go around the concurrency manager and run a single "query manager" task for all joins for all shards for all queries. This has some great benefits which I will go into below.
-
-That "query manager" task would be responsible for accepting buffers containing elements or binding sets from other nodes and scheduling consumption of those data based on various critieria (order of arrival, priority, buffer resource requirements, timeout, etc.). This manager task could use a fork join pool to execute light weight operations (NIO, formulation of access paths from binding sets, mapping of binding sets onto shards, joining a chunk already read from an access path against a binding set, etc). Operations which touch the disk need to run in their own thread (until we get Java 7 async file IO, which is already available in a preview library). We could handle that by queuing those operations against a fixed size thread pool for reads.
-
-This is a radical change in how we handle distributed query execution, but I think that it could have a huge payoff by reducing the complexity of the join logic, making it significantly easier to execute different kinds of join operations, reducing the overhead for acquiring locks for the unisolated index views, reducing the #of threads consumed by joins (from one per shard per join per query to a fixed pool of N threads for reads), etc. It would centralize the management of resources on each node and make it possible for us to handle things like join termination by simply purging data from the query manager task for the terminated join.
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bset/ConditionalRoutingOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bset/ConditionalRoutingOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bset/ConditionalRoutingOp.java 2010-09-03 00:27:45 UTC (rev 3500)
@@ -0,0 +1,217 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Aug 25, 2010
+ */
+
+package com.bigdata.bop.bset;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.FutureTask;
+
+import com.bigdata.bop.AbstractPipelineOp;
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpContext;
+import com.bigdata.bop.BindingSetPipelineOp;
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.IConstraint;
+import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.relation.accesspath.IAsynchronousIterator;
+import com.bigdata.relation.accesspath.IBlockingBuffer;
+
+/**
+ * An operator for conditional routing of binding sets in a pipeline. The
+ * operator will copy binding sets either to the default sink (if a condition is
+ * satisfied) and to the alternate sink otherwise.
+ * <p>
+ * Conditional routing can be useful where a different data flow is required
+ * based on the type of an object (for example a term identifier versus an
+ * inline term in the RDF database) or where there is a need to jump around a
+ * join group based on some condition.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class ConditionalRoutingOp extends BindingSetPipelineOp {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ public interface Annotations extends AbstractPipelineOp.Annotations {
+
+ /**
+ * An {@link IConstraint} which specifies the condition. When the
+ * condition is satisfied the binding set is routed to the default sink.
+ * When the condition is not satisfied, the binding set is routed to the
+ * alternative sink.
+ */
+ String CONDITION = ConditionalRoutingOp.class.getName() + ".condition";
+
+ }
+
+ /**
+ * Deep copy constructor.
+ *
+ * @param op
+ */
+ public ConditionalRoutingOp(ConditionalRoutingOp op) {
+ super(op);
+ }
+
+ /**
+ * Shallow copy constructor.
+ *
+ * @param args
+ * @param annotations
+ */
+ public ConditionalRoutingOp(BOp[] args, Map<String, Object> annotations) {
+ super(args, annotations);
+ }
+
+ /**
+ * @see Annotations#CONDITION
+ */
+ public IConstraint getCondition() {
+
+ return (IConstraint) getProperty(Annotations.CONDITION);
+
+ }
+
+ public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
+
+ return new FutureTask<Void>(new ConditionalRouteTask(this, context));
+
+ }
+
+ /**
+ * Copy the source to the sink or the alternative sink depending on the
+ * condition.
+ */
+ static private class ConditionalRouteTask implements Callable<Void> {
+
+ private final BOpStats stats;
+
+ private final IConstraint condition;
+
+ private final IAsynchronousIterator<IBindingSet[]> source;
+
+ private final IBlockingBuffer<IBindingSet[]> sink;
+
+ private final IBlockingBuffer<IBindingSet[]> sink2;
+
+ ConditionalRouteTask(final ConditionalRoutingOp op,
+ final BOpContext<IBindingSet> context) {
+
+ this.stats = context.getStats();
+
+ this.condition = op.getCondition();
+
+ if (condition == null)
+ throw new IllegalArgumentException();
+
+ this.source = context.getSource();
+
+ this.sink = context.getSink();
+
+ this.sink2 = context.getSink2();
+
+ if (sink2 == null)
+ throw new IllegalArgumentException();
+
+ if (sink == sink2)
+ throw new IllegalArgumentException();
+
+ }
+
+ public Void call() throws Exception {
+ try {
+ while (source.hasNext()) {
+
+ final IBindingSet[] chunk = source.next();
+
+ stats.chunksIn.increment();
+ stats.unitsIn.add(chunk.length);
+
+ final IBindingSet[] def = new IBindingSet[chunk.length];
+ final IBindingSet[] alt = new IBindingSet[chunk.length];
+
+ int ndef = 0, nalt = 0;
+
+ for(int i=0; i<chunk.length; i++) {
+
+ final IBindingSet bset = chunk[i];
+
+ if (condition.accept(bset)) {
+
+ def[ndef++] = bset;
+
+ } else {
+
+ alt[nalt++] = bset;
+
+ }
+
+ }
+
+ if (ndef > 0) {
+ if (ndef == def.length)
+ sink.add(def);
+ else
+ sink.add(Arrays.copyOf(def, ndef));
+ stats.chunksOut.increment();
+ stats.unitsOut.add(ndef);
+ }
+
+ if (nalt > 0) {
+ if (nalt == alt.length)
+ sink2.add(alt);
+ else
+ sink2.add(Arrays.copyOf(alt, nalt));
+ stats.chunksOut.increment();
+ stats.unitsOut.add(nalt);
+ }
+
+ }
+
+ sink.flush();
+ sink2.flush();
+
+ return null;
+
+ } finally {
+
+ sink.close();
+ sink2.close();
+
+ }
+
+ }
+
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bset/ConditionalRoutingOp.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bset/CopyBindingSetOp.java (from rev 3495, branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/PipelineStartOp.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bset/CopyBindingSetOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bset/CopyBindingSetOp.java 2010-09-03 00:27:45 UTC (rev 3500)
@@ -0,0 +1,124 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Aug 25, 2010
+ */
+
+package com.bigdata.bop.bset;
+
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.FutureTask;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpContext;
+import com.bigdata.bop.BindingSetPipelineOp;
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.relation.accesspath.IAsynchronousIterator;
+import com.bigdata.relation.accesspath.IBlockingBuffer;
+
+/**
+ * This operator copies its source to its sink. It is used to feed the first
+ * join in the pipeline. The operator should have no children but may be
+ * decorated with annotations as necessary.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ *
+ * @todo unit tests.
+ */
+public class CopyBindingSetOp extends BindingSetPipelineOp {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * Deep copy constructor.
+ *
+ * @param op
+ */
+ public CopyBindingSetOp(CopyBindingSetOp op) {
+ super(op);
+ }
+
+ /**
+ * Shallow copy constructor.
+ *
+ * @param args
+ * @param annotations
+ */
+ public CopyBindingSetOp(BOp[] args, Map<String, Object> annotations) {
+ super(args, annotations);
+ }
+
+ public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
+
+ return new FutureTask<Void>(new CopyTask(context));
+
+ }
+
+ /**
+ * Copy the source to the sink.
+ */
+ static private class CopyTask implements Callable<Void> {
+
+ private final BOpStats stats;
+
+ private final IAsynchronousIterator<IBindingSet[]> source;
+
+ private final IBlockingBuffer<IBindingSet[]> sink;
+
+ CopyTask(final BOpContext<IBindingSet> context) {
+
+ stats = context.getStats();
+
+ this.source = context.getSource();
+
+ this.sink = context.getSink();
+
+ }
+
+ public Void call() throws Exception {
+ try {
+ while (source.hasNext()) {
+ final IBindingSet[] chunk = source.next();
+ stats.chunksIn.increment();
+ stats.unitsIn.add(chunk.length);
+ sink.add(chunk);
+ stats.chunksOut.increment();
+ stats.unitsOut.add(chunk.length);
+ }
+ sink.flush();
+ return null;
+ } finally {
+ sink.close();
+ }
+ }
+
+ }
+
+}
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bset/Union.java (from rev 3495, branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregation/Union.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bset/Union.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/bset/Union.java 2010-09-03 00:27:45 UTC (rev 3500)
@@ -0,0 +1,135 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Aug 18, 2010
+ */
+
+package com.bigdata.bop.bset;
+
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.FutureTask;
+
+import com.bigdata.bop.BOpContext;
+import com.bigdata.bop.BindingSetPipelineOp;
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.join.PipelineJoin;
+import com.bigdata.rdf.rules.TMUtility;
+import com.bigdata.relation.RelationFusedView;
+import com.bigdata.util.concurrent.Haltable;
+
+/**
+ * The union of two or more {@link BindingSetPipelineOp} operators.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ *
+ * @todo I have some basic questions about the ability to use a UNION of two
+ * predicates in scale-out. I think that this might be more accurately
+ * modeled as the UNION of two joins. That is, rather than:
+ *
+ * <pre>
+ * JOIN( ...,
+ * UNION( foo.spo(A,loves,B),
+ * bar.spo(A,loves,B) )
+ * )
+ * </pre>
+ *
+ * using
+ *
+ * <pre>
+ * UNION( JOIN( ..., foo.spo(A,loves,B) ),
+ * JOIN( ..., bar.spo(A,loves,B) )
+ * )
+ * </pre>
+ *
+ * which would be a binding set union rather than an element union.
+ *
+ * @todo The union of access paths was historically handled by
+ * {@link RelationFusedView}. That class should be removed once queries
+ * are rewritten to use the union of joins.
+ *
+ * @todo The {@link TMUtility} will have to be updated to use this operator
+ * rather than specifying multiple source "names" for the relation of the
+ * predicate.
+ *
+ * @todo The FastClosureRuleTask will also need to be updated to use a
+ * {@link Union} over the joins rather than a {@link RelationFusedView}.
+ */
+public class Union extends BindingSetPipelineOp {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ /**
+ * @param args
+ * Two or more operators whose union is desired.
+ * @param annotations
+ */
+ public Union(final BindingSetPipelineOp[] args,
+ final Map<String, Object> annotations) {
+
+ super(args, annotations);
+
+ if (args.length < 2)
+ throw new IllegalArgumentException();
+
+ }
+
+ public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
+
+ return new FutureTask<Void>(new UnionTask(this, context));
+
+ }
+
+ /**
+ * Pipeline union impl.
+ *
+ * FIXME All this does is copy its inputs to its outputs. Since we only run
+ * one chunk of input at a time, it seems that the easiest way to implement
+ * a union is to have the operators in the union just target the same sink.
+ */
+ private static class UnionTask extends Haltable<Void> implements Callable<Void> {
+
+ public UnionTask(//
+ final Union op,//
+ final BOpContext<IBindingSet> context
+ ) {
+
+ if (op == null)
+ throw new IllegalArgumentException();
+ if (context == null)
+ throw new IllegalArgumentException();
+ }
+
+ public Void call() throws Exception {
+ // TODO Auto-generated method stub
+ throw new UnsupportedOperationException();
+ }
+
+ }
+
+}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java 2010-09-02 22:24:22 UTC (rev 3499)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java 20...
[truncated message content] |
4 messages has been excluded from this view by a project administrator.
