Menu
▾
▴
bigdata-commit — Commit traffic.
This list is closed, nobody may subscribe to it.
| 2010 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(139) |
Aug
(94) |
Sep
(232) |
Oct
(143) |
Nov
(138) |
Dec
(55) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2011 |
Jan
(127) |
Feb
(90) |
Mar
(101) |
Apr
(74) |
May
(148) |
Jun
(241) |
Jul
(169) |
Aug
(121) |
Sep
(157) |
Oct
(199) |
Nov
(281) |
Dec
(75) |
| 2012 |
Jan
(107) |
Feb
(122) |
Mar
(184) |
Apr
(73) |
May
(14) |
Jun
(49) |
Jul
(26) |
Aug
(103) |
Sep
(133) |
Oct
(61) |
Nov
(51) |
Dec
(55) |
| 2013 |
Jan
(59) |
Feb
(72) |
Mar
(99) |
Apr
(62) |
May
(92) |
Jun
(19) |
Jul
(31) |
Aug
(138) |
Sep
(47) |
Oct
(83) |
Nov
(95) |
Dec
(111) |
| 2014 |
Jan
(125) |
Feb
(60) |
Mar
(119) |
Apr
(136) |
May
(270) |
Jun
(83) |
Jul
(88) |
Aug
(30) |
Sep
(47) |
Oct
(27) |
Nov
(23) |
Dec
|
| 2015 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(3) |
Oct
|
Nov
|
Dec
|
| 2016 |
Jan
|
Feb
|
Mar
(4) |
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: <tho...@us...> - 2014-01-05 17:35:51
|
Revision: 7728
http://bigdata.svn.sourceforge.net/bigdata/?rev=7728&view=rev
Author: thompsonbry
Date: 2014-01-05 17:35:43 +0000 (Sun, 05 Jan 2014)
Log Message:
-----------
javadoc update on WRITE_CACHE_COMPACTION_THRESHOLD.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/journal/Options.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/journal/Options.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/journal/Options.java 2014-01-04 23:19:59 UTC (rev 7727)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/journal/Options.java 2014-01-05 17:35:43 UTC (rev 7728)
@@ -256,12 +256,15 @@
String WRITE_CACHE_MIN_CLEAN_LIST_SIZE = AbstractJournal.class.getName()+".writeCacheMinCleanListSize";
/**
- * Option may be used to control whether the {@link WriteCacheService} will
- * compact the {@link WriteCache} buffers in order to reduce the number of
- * writes to the disk (default
- * {@value #DEFAULT_WRITE_CACHE_COMPACTION_THRESHOLD}) by specifying the
- * minimum percentage of the {@link WriteCache} buffer that could be
- * reclaimed.
+ * Option specifies the minimum percentage of empty space in a
+ * {@link WriteCache} buffer that could be recovered before we will attempt
+ * to compact the buffer (in [0:100], default
+ * {@value #DEFAULT_WRITE_CACHE_COMPACTION_THRESHOLD}). Free space is
+ * created in a dirty buffer when allocation slots written on that buffer
+ * are recycled before the buffer is evicted to the disk. Such recycling is
+ * common in large transactions. Compaction is disabled when the buffers are
+ * flushed during the commit protocol since all data must be written through
+ * to the disk.
* <p>
* Note: This option has no effect for a WORM mode journal.
* <p>
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-04 21:59:15
|
Revision: 7726
http://bigdata.svn.sourceforge.net/bigdata/?rev=7726&view=rev
Author: thompsonbry
Date: 2014-01-04 21:59:08 +0000 (Sat, 04 Jan 2014)
Log Message:
-----------
The best way to analyze this is to use hint:atOnce to force each operator in the query plan to execute only once. You can then look at the opCount column in the detailed Explain view and see whether the correct number of operator invocations was reported. It should be ONE (1) for each operator where the atOnce hint was correctly applied. If the atOnce hint is applied to an operator, then it sets pipelined:=false as follows:
com.bigdata.bop.PipelineOp.pipelined = false
I have modified BOPStats to NOT pre-increment opCount. I have modified AbstractRunningQuery.haltOp() to post-increment opCount
One consequence is that the opCount will not update until the operator has halted. Thus, it is a "how many times did this operator run successfully counter" rather than a "how many times did this operator start counter". However, it is now reporting the correct values. I cross validated this by logging out the following in ChunkedRunningQuery:
{{{
if (log.isInfoEnabled())
log.info("Running task: bop=" + bundle.bopId
+ (pipelined?"":", atOnceReady=" + atOnceReady) + ", bop="
+ bop.toShortString() + ", messages=" + naccepted
+ ", solutions=" + solutionsAccepted
+ (log.isDebugEnabled()?", runState=" + runStateString():""));
getQueryEngine().execute(cft);
}}}
This is the code that actually runs the operator. I then verified that the data on operator invocations was correct. I also cross correlated where pipelined:=false in the query plan with opCount:=1, i.e., if atOnce evaluation was imposed on the operator, then we had only one invocation reported for that operator.
See #793.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/AbstractRunningQuery.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/BOpStats.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/AbstractRunningQuery.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/AbstractRunningQuery.java 2014-01-04 21:46:36 UTC (rev 7725)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/AbstractRunningQuery.java 2014-01-04 21:59:08 UTC (rev 7726)
@@ -825,18 +825,38 @@
log.trace(msg.toString());
// update per-operator statistics.
- final BOpStats tmp = statsMap.putIfAbsent(msg.getBOpId(), msg.getStats());
+ {
+ // Data race on insert into CHM.
+ BOpStats tmp = statsMap.putIfAbsent(msg.getBOpId(),
+ msg.getStats());
- /*
- * Combine stats, but do not combine a stats object with itself.
- *
- * @see https://sourceforge.net/apps/trac/bigdata/ticket/464 (Query
- * Statistics do not update correctly on cluster)
- */
- if (tmp != null && tmp != msg.getStats()) {
- tmp.add(msg.getStats());
+ /**
+ * Combine stats, but do not combine a stats object with itself.
+ *
+ * @see <a
+ * href="https://sourceforge.net/apps/trac/bigdata/ticket/464">
+ * Query Statistics do not update correctly on cluster</a>
+ */
+ if (tmp == null) {
+ // won the data race.
+ tmp = msg.getStats();
+ } else {
+ // lost the data race.
+ if (tmp != msg.getStats()) {
+ tmp.add(msg.getStats());
+ }
+ }
+ /**
+ * Post-increment now that we know who one the data race.
+ *
+ * @see <a
+ * href="https://sourceforge.net/apps/trac/bigdata/ticket/793">
+ * Explain reports incorrect value for opCount</a>
+ */
+ tmp.opCount.increment();
+ // log.warn("bop=" + getBOp(msg.getBOpId()).toShortString()
+ // + " : stats=" + tmp);
}
-// log.warn(msg.toString() + " : stats=" + tmp);
switch (runState.haltOp(msg)) {
case Running:
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/BOpStats.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/BOpStats.java 2014-01-04 21:46:36 UTC (rev 7725)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/BOpStats.java 2014-01-04 21:59:08 UTC (rev 7726)
@@ -30,7 +30,6 @@
import java.io.Serializable;
import com.bigdata.bop.BOp;
-import com.bigdata.bop.PipelineOp;
import com.bigdata.counters.CAT;
/**
@@ -55,18 +54,15 @@
*/
final public CAT elapsed = new CAT();
- /**
- * The #of instances of a given operator which have been created for a given
- * query. This provides interesting information about the #of task instances
- * for each operator which were required to execute a query.
- *
- * TODO Due to the way this is incremented, this is always ONE (1) if
- * {@link PipelineOp.Annotations#SHARED_STATE} is <code>true</code> (it
- * reflects the #of times {@link #add(BOpStats)} was invoked plus one for
- * the ctor rather than the #of times the operator task was invoked). This
- * should be changed to reflect the #of operator task instances created
- * instead.
- */
+ /**
+ * The #of instances of a given operator which have been started (and
+ * successully terminated) for a given query. This provides interesting
+ * information about the #of task instances for each operator which were
+ * required to execute a query.
+ *
+ * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/793">
+ * Explain reports incorrect value for opCount</a>
+ */
final public CAT opCount = new CAT();
/**
@@ -127,10 +123,16 @@
/**
* Constructor.
+ * <p>
+ * Note: Do not pre-increment {@link #opCount}. See {@link #add(BOpStats)}
+ * and {@link AbstractRunningQuery#haltOp(IHaltOpMessage)}.
+ *
+ * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/793">
+ * Explain reports incorrect value for opCount</a>
*/
public BOpStats() {
- opCount.increment();
+// opCount.increment();
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-04 21:46:42
|
Revision: 7725
http://bigdata.svn.sourceforge.net/bigdata/?rev=7725&view=rev
Author: thompsonbry
Date: 2014-01-04 21:46:36 +0000 (Sat, 04 Jan 2014)
Log Message:
-----------
Modified to NOT show the bopAnnotations for the "total" link of the Explain.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java 2014-01-04 21:46:02 UTC (rev 7724)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java 2014-01-04 21:46:36 UTC (rev 7725)
@@ -1271,7 +1271,9 @@
if (detailedStats) {
// bopAnnotations
w.write(TD);
- showAnnotations(w, bop.annotations());
+ if (!summary) {
+ showAnnotations(w, bop.annotations());
+ }
w.write(TDx);
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-04 21:46:08
|
Revision: 7724
http://bigdata.svn.sourceforge.net/bigdata/?rev=7724&view=rev
Author: thompsonbry
Date: 2014-01-04 21:46:02 +0000 (Sat, 04 Jan 2014)
Log Message:
-----------
@Override annotations.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/NamedSolutionSetStats.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/bop/rdf/update/ParserStats.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/NamedSolutionSetStats.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/NamedSolutionSetStats.java 2014-01-04 21:07:51 UTC (rev 7723)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/NamedSolutionSetStats.java 2014-01-04 21:46:02 UTC (rev 7724)
@@ -42,6 +42,7 @@
public final CAT solutionSetSize = new CAT();
+ @Override
public void add(final BOpStats o) {
super.add(o);
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java 2014-01-04 21:07:51 UTC (rev 7723)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java 2014-01-04 21:46:02 UTC (rev 7724)
@@ -241,6 +241,7 @@
}
+ @Override
public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
return new FutureTask<Void>(new SliceTask(this, context));
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/bop/rdf/update/ParserStats.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/bop/rdf/update/ParserStats.java 2014-01-04 21:07:51 UTC (rev 7723)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/bop/rdf/update/ParserStats.java 2014-01-04 21:46:02 UTC (rev 7724)
@@ -43,6 +43,7 @@
public final CAT toldTriples = new CAT();
+ @Override
public void add(final BOpStats o) {
super.add(o);
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-04 21:07:58
|
Revision: 7723
http://bigdata.svn.sourceforge.net/bigdata/?rev=7723&view=rev
Author: thompsonbry
Date: 2014-01-04 21:07:51 +0000 (Sat, 04 Jan 2014)
Log Message:
-----------
@Override
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/AbstractRunningQuery.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/AbstractRunningQuery.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/AbstractRunningQuery.java 2014-01-04 20:53:51 UTC (rev 7722)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/AbstractRunningQuery.java 2014-01-04 21:07:51 UTC (rev 7723)
@@ -493,6 +493,7 @@
}
+ @Override
final public Map<Integer, BOp> getBOpIndex() {
return bopIndex;
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-04 20:53:58
|
Revision: 7722
http://bigdata.svn.sourceforge.net/bigdata/?rev=7722&view=rev
Author: thompsonbry
Date: 2014-01-04 20:53:51 +0000 (Sat, 04 Jan 2014)
Log Message:
-----------
Some additional log statements, inline documentation on the analytic mode hash join algorithm, and inline documentation on the root cause for #763 (stochastic results with the analytic query mode).
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java 2014-01-04 20:03:56 UTC (rev 7721)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java 2014-01-04 20:53:51 UTC (rev 7722)
@@ -600,6 +600,7 @@
* This exposes a view of the {@link HTree} which is safe for concurrent
* readers.
*/
+ @Override
public void saveSolutionSet() {
if (!open.get())
@@ -642,6 +643,9 @@
// Checkpoint the HTree.
final Checkpoint checkpoint = tmp.writeCheckpoint2();
+ if (log.isInfoEnabled())
+ log.info(checkpoint.toString());
+
final HTree readOnly = HTree.load(store,
checkpoint.getCheckpointAddr(), true/* readOnly */);
@@ -735,7 +739,7 @@
// Encode the solution.
final byte[] val = encoder.encodeSolution(tmp.bset);
-
+//log.warn("insert: key="+BytesUtil.toString(key));
// Insert binding set under hash code for that key.
htree.insert(key, val);
@@ -755,6 +759,10 @@
}
+ if (log.isInfoEnabled())
+ log.info("naccepted=" + naccepted + ", nright="
+ + htree.getEntryCount());
+
return naccepted;
} catch(Throwable t) {
@@ -813,13 +821,20 @@
/*
* Search the hash index for a match.
+ *
+ * TODO VECTOR: This does not take explicit advantage of the
+ * fact that different source solutions will fall into the
+ * same hash bucket in the HTree. The solutions are ordered
+ * by hashCode by vector() above, but we are using one lookupAll()
+ * invocation per source solution here rather than recognizing that
+ * multiple source solutions will hit the same hash bucket.
*/
boolean found = false;
-
+
final ITupleIterator<?> titr = htree.lookupAll(key);
- while(titr.hasNext()) {
-
+ while (titr.hasNext()) {
+
final ITuple<?> t = titr.next();
final ByteArrayBuffer tb = t.getValueBuffer();
@@ -967,6 +982,14 @@
}
+ /*
+ * The hash join is vectored. We compute the hashCode for each source
+ * solution from the leftItr and then sort those left solutions. This gives
+ * us an ordered progression through the hash buckets for the HTree.
+ * Further, since we know that any left solution having the same hash code
+ * will read on the same hash bucket, we probe that hash bucket once for all
+ * left solutions that hash into the same bucket.
+ */
@Override
public void hashJoin2(//
final ICloseableIterator<IBindingSet[]> leftItr,//
@@ -976,7 +999,13 @@
) {
// Note: We no longer rechunk in this method.
- final Iterator<IBindingSet[]> it = leftItr;
+ final Iterator<IBindingSet[]> it;
+ it = leftItr;// incremental.
+ /*
+ * Note: This forces all source chunks into a single chunk. This could
+ * improve vectoring, but was really added for debugging.
+ */
+// it = new SingletonIterator<IBindingSet[]>(BOpUtility.toArray(leftItr, null/*stats*/));
try {
@@ -996,6 +1025,7 @@
final boolean noJoinVars = joinVars.length == 0;
final AtomicInteger vectorSize = new AtomicInteger();
+
while (it.hasNext()) {
final BS[] a; // vectored solutions.
@@ -1017,11 +1047,6 @@
nleftConsidered.add(n);
- if (log.isTraceEnabled())
- log.trace("Vectoring chunk for HTree locality: " + n
- + " out of " + a.length
- + " solutions are preserved.");
-
}
int fromIndex = 0;
@@ -1079,9 +1104,37 @@
final byte[] key = keyBuilder.reset().append(hashCode)
.getKey();
- // visit all source solutions having the same hash code
- final ITupleIterator<?> titr = rightSolutions
- .lookupAll(key);
+ /**
+ * Visit all source solutions having the same hash code.
+ *
+ * @see <a
+ * href="https://sourceforge.net/apps/trac/bigdata/ticket/763#comment:19">
+ * Stochastic results with Analytic Query Mode)
+ * </a>
+ *
+ * FIXME This appears to be the crux of the problem
+ * for #764. If you replace lookupAll(key) with
+ * rangeIterator() then the hash join is correct.
+ * Of course, it is also scanning all tuples each
+ * time so it is very inefficient. The root cause
+ * is the FrontCodedRabaCoder. It is doing a binary
+ * search on the BucketPage. However, the
+ * FrontCodedRabaCoder was not developed to deal
+ * with duplicates on the page. Therefore it is
+ * returning an offset into the middle of a run of
+ * duplicate keys when it does its binary search.
+ * We will either need to modify this IRabaCoder to
+ * handle this case (where duplicate keys are
+ * allowed) or write a new IRabaCoder that is smart
+ * about duplicates.
+ */
+ final ITupleIterator<?> titr;
+ if (true) {// scan just the hash bucket for that key.
+//log.warn(" probe: key="+BytesUtil.toString(key));
+ titr = rightSolutions.lookupAll(key);
+ } else { // do a full scan on the HTree.
+ titr = rightSolutions.rangeIterator();
+ }
long sameHashCodeCount = 0;
@@ -1167,7 +1220,6 @@
// Join failed.
continue;
}
-
// Resolve against ivCache.
encoder.resolveCachedValues(outSolution);
@@ -1247,6 +1299,9 @@
} // while(itr.hasNext()
+ if (log.isInfoEnabled())
+ log.info("done: " + toString());
+
} catch(Throwable t) {
throw launderThrowable(t);
@@ -1292,7 +1347,8 @@
final BS[] a = new BS[leftSolutions.length];
int n = 0; // The #of non-dropped source solutions.
-
+ int ndropped = 0; // The #of dropped solutions.
+
for (int i = 0; i < a.length; i++) {
/*
@@ -1317,6 +1373,8 @@
if (log.isTraceEnabled())
log.trace(ex);
+
+ ndropped++;
continue;
@@ -1338,7 +1396,11 @@
// Indicate the actual vector size to the caller via a side-effect.
vectorSize.set(n);
-
+
+ if (log.isTraceEnabled())
+ log.trace("Vectoring chunk for HTree locality: naccepted=" + n
+ + ", ndropped=" + ndropped);
+
return a;
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-04 20:04:03
|
Revision: 7721
http://bigdata.svn.sourceforge.net/bigdata/?rev=7721&view=rev
Author: thompsonbry
Date: 2014-01-04 20:03:56 +0000 (Sat, 04 Jan 2014)
Log Message:
-----------
logging change.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java 2014-01-04 19:48:22 UTC (rev 7720)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java 2014-01-04 20:03:56 UTC (rev 7721)
@@ -663,13 +663,12 @@
/*
* Submit task for execution (asynchronous).
*/
- if (log.isDebugEnabled()) {
- log.debug("Running task: bop=" + bundle.bopId
- + ", atOnceReady=" + atOnceReady + ", bop="
+ if (log.isInfoEnabled())
+ log.info("Running task: bop=" + bundle.bopId
+ + (pipelined?"":", atOnceReady=" + atOnceReady) + ", bop="
+ bop.toShortString() + ", messages=" + naccepted
+ ", solutions=" + solutionsAccepted
- + ", runState=" + runStateString());
- }
+ + (log.isDebugEnabled()?", runState=" + runStateString():""));
getQueryEngine().execute(cft);
return true;
} catch(Throwable t) {
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-04 19:48:28
|
Revision: 7720
http://bigdata.svn.sourceforge.net/bigdata/?rev=7720&view=rev
Author: thompsonbry
Date: 2014-01-04 19:48:22 +0000 (Sat, 04 Jan 2014)
Log Message:
-----------
final and @Override annotations.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/NamedSubqueryInclude.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/NamedSubqueryInclude.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/NamedSubqueryInclude.java 2014-01-04 19:47:59 UTC (rev 7719)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/NamedSubqueryInclude.java 2014-01-04 19:48:22 UTC (rev 7720)
@@ -104,7 +104,7 @@
/**
* Constructor required for {@link com.bigdata.bop.BOpUtility#deepCopy(FilterNode)}.
*/
- public NamedSubqueryInclude(NamedSubqueryInclude op) {
+ public NamedSubqueryInclude(final NamedSubqueryInclude op) {
super(op);
@@ -113,7 +113,7 @@
/**
* Required shallow copy constructor.
*/
- public NamedSubqueryInclude(BOp[] args, Map<String, Object> anns) {
+ public NamedSubqueryInclude(final BOp[] args, final Map<String, Object> anns) {
super(args, anns);
@@ -129,12 +129,14 @@
}
+ @Override
public String getName() {
return (String) getProperty(Annotations.NAMED_SET);
}
+ @Override
public void setName(final String name) {
if (name == null)
@@ -195,6 +197,7 @@
/**
* Returns <code>false</code>.
*/
+ @Override
final public boolean isOptional() {
return false;
@@ -204,12 +207,14 @@
/**
* Returns <code>false</code>.
*/
+ @Override
final public boolean isMinus() {
return false;
}
+ @Override
final public List<FilterNode> getAttachedJoinFilters() {
@SuppressWarnings("unchecked")
@@ -225,6 +230,7 @@
}
+ @Override
final public void setAttachedJoinFilters(final List<FilterNode> filters) {
setProperty(Annotations.FILTERS, filters);
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-04 19:48:06
|
Revision: 7719
http://bigdata.svn.sourceforge.net/bigdata/?rev=7719&view=rev
Author: thompsonbry
Date: 2014-01-04 19:47:59 +0000 (Sat, 04 Jan 2014)
Log Message:
-----------
Replaced test for ICloseableIterator with test for ICloseable.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpUtility.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpUtility.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpUtility.java 2014-01-04 18:30:29 UTC (rev 7718)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpUtility.java 2014-01-04 19:47:59 UTC (rev 7719)
@@ -54,6 +54,7 @@
import cutthecrap.utils.striterators.EmptyIterator;
import cutthecrap.utils.striterators.Expander;
import cutthecrap.utils.striterators.Filter;
+import cutthecrap.utils.striterators.ICloseable;
import cutthecrap.utils.striterators.ICloseableIterator;
import cutthecrap.utils.striterators.SingleValueIterator;
import cutthecrap.utils.striterators.Striterator;
@@ -988,9 +989,9 @@
} finally {
- if (itr instanceof ICloseableIterator<?>) {
+ if (itr instanceof ICloseable) {
- ((ICloseableIterator<?>) itr).close();
+ ((ICloseable) itr).close();
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-04 18:30:36
|
Revision: 7718
http://bigdata.svn.sourceforge.net/bigdata/?rev=7718&view=rev
Author: thompsonbry
Date: 2014-01-04 18:30:29 +0000 (Sat, 04 Jan 2014)
Log Message:
-----------
ChunkedRunningQuery:: Slight re-ordering of acceptChunk(). Now tests for an empty queue before testing for atOnce evaluation. Also computes the number of solutions across the messages and then conditionally logs that information.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java 2014-01-04 18:29:11 UTC (rev 7717)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java 2014-01-04 18:30:29 UTC (rev 7718)
@@ -521,22 +521,6 @@
// }
//
// }
- if (!pipelined && !queue.isEmpty() && !isAtOnceReady(bundle.bopId)) {
- /*
- * This operator is not pipelined, so we need to wait until all
- * of its input solutions have been materialized (no prior
- * operator in the pipeline is running or has inputs available
- * which could cause it to run).
- *
- * TODO This is where we should examine MAX_MEMORY and the
- * buffered data to see whether or not to trigger an evaluation
- * pass for the operator based on the data already materialized
- * for that operator.
- */
- if (log.isDebugEnabled())
- log.debug("Waiting on producer(s): bopId=" + bundle.bopId);
- return false;
- }
if (queue.isEmpty()) {
// No work, so remove work queue for (bopId,partitionId).
if(removeMapOperatorQueueEntries)
@@ -544,6 +528,31 @@
return false;
}
/*
+ * true iff operator requires at once evaluation and all solutions
+ * are now available for that operator.
+ */
+ boolean atOnceReady = false;
+ if (!pipelined) {
+ if (!isAtOnceReady(bundle.bopId)) {
+ /*
+ * This operator is not pipelined, so we need to wait until
+ * all of its input solutions have been materialized (no
+ * prior operator in the pipeline is running or has inputs
+ * available which could cause it to run).
+ *
+ * TODO This is where we should examine MAX_MEMORY and the
+ * buffered data to see whether or not to trigger an
+ * evaluation pass for the operator based on the data
+ * already materialized for that operator.
+ */
+ if (log.isDebugEnabled())
+ log.debug("Waiting on producer(s): bopId="
+ + bundle.bopId);
+ return false;
+ }
+ atOnceReady = true;
+ }
+ /*
* Drain the work queue for that (bopId,partitionId).
*
* Note: If the operator is pipelined, then we do not drain more
@@ -553,104 +562,116 @@
*/
final List<IChunkMessage<IBindingSet>> accepted = new LinkedList<IChunkMessage<IBindingSet>>();
try {
- /*
- * Note: Once we drain these messages from the work queue we are
- * responsible for calling release() on them.
- */
- queue.drainTo(accepted, pipelined ? maxMessagesPerTask
- : Integer.MAX_VALUE);
- // #of messages accepted from the work queue.
- final int naccepted = accepted.size();
- // #of messages remaining on the work queue.
- final int nremaining = queue.size();
- if(nremaining == 0) {
- // Remove the work queue for that (bopId,partitionId).
- if(removeMapOperatorQueueEntries)
- if(queue != operatorQueues.remove(bundle)) throw new AssertionError();
- } else if(pipelined) {
/*
- * After removing the maximum amount from a pipelined operator,
- * the work queue is still not empty.
+ * Note: Once we drain these messages from the work queue we are
+ * responsible for calling release() on them.
*/
- if (log.isInfoEnabled())
- log.info("Work queue is over capacity: bundle=" + bundle
- + ", naccepted=" + naccepted + ", nremaining="
- + nremaining + ", maxMessagesPerTask="
- + maxMessagesPerTask + ", runState="
- + runStateString());
- }
- /*
- * Combine the messages into a single source to be consumed by a
- * task.
- */
- int nassigned = 1;
- final Iterator<IChunkMessage<IBindingSet>> mitr = accepted.iterator();
- final IChunkMessage<IBindingSet> firstChunk = mitr.next();
- // See BOpContext#isLastInvocation()
- final boolean isLastInvocation = pipelined
-// && nremaining == 0
-// && maxParallel == 1
-// && isOperatorDone(bundle.bopId)
- && firstChunk.isLastInvocation()
- ;
- /*
- * Note: There is no longer any reliance on the IAsynchronous
- * Iterator API here. It is perfectly sufficient to only
- * implement ICloseableIterator. Query operator and chunk
- * message implementations should be revisited with this
- * simplifying assumption in mind.
- *
- * @see https://sourceforge.net/apps/trac/bigdata/ticket/475
- */
- final IMultiSourceCloseableIterator<IBindingSet[]> source = new MultiSourceSequentialCloseableIterator<IBindingSet[]>(//
-// accepted.remove(0).getChunkAccessor().iterator()//
- firstChunk.getChunkAccessor().iterator()//
- );
-// for (IChunkMessage<IBindingSet> msg : accepted) {
-// source.add(msg.getChunkAccessor().iterator());
- while(mitr.hasNext()) {
- source.add(mitr.next().getChunkAccessor().iterator());
- nassigned++;
- }
- if (nassigned != naccepted)
- throw new AssertionError();
- /*
- * Create task to consume that source.
- */
- final ChunkFutureTask cft;
- try {
- cft = new ChunkFutureTask(
- new ChunkTask(bundle.bopId, bundle.shardId,
- naccepted, isLastInvocation, source));
- } catch (Throwable t2) {
- // Ensure accepted messages are released();
- safeRelease(accepted);
- halt(t2); // ensure query halts.
- if (getCause() != null) {
- // Abnormal termination - wrap and rethrow.
- throw new RuntimeException(t2);
+ queue.drainTo(accepted, pipelined ? maxMessagesPerTask
+ : Integer.MAX_VALUE);
+ // #of messages accepted from the work queue.
+ final int naccepted = accepted.size();
+ // #of messages remaining on the work queue.
+ final int nremaining = queue.size();
+ if (nremaining == 0) {
+ // Remove the work queue for that (bopId,partitionId).
+ if(removeMapOperatorQueueEntries)
+ if(queue != operatorQueues.remove(bundle)) throw new AssertionError();
+ } else if (pipelined) {
+ /*
+ * After removing the maximum amount from a pipelined operator,
+ * the work queue is still not empty.
+ */
+ if (log.isInfoEnabled())
+ log.info("Work queue is over capacity: bundle=" + bundle
+ + ", naccepted=" + naccepted + ", nremaining="
+ + nremaining + ", maxMessagesPerTask="
+ + maxMessagesPerTask + ", runState="
+ + runStateString());
}
- // normal termination - swallow the exception.
- return false;
- }
- /*
- * Save the Future for this task. Together with the logic above this
- * may be used to limit the #of concurrent tasks per (bopId,shardId)
- * to one for a given query.
- */
- if (map == null) {
- map = new ConcurrentHashMap<ChunkFutureTask, ChunkFutureTask>();
- operatorFutures.put(bundle, map);
- }
- map.put(cft, cft);
- /*
- * Submit task for execution (asynchronous).
- */
- if (log.isDebugEnabled())
- log.debug("Running task: bop=" + bundle.bopId + ", naccepted="
- + naccepted+", runState="+runStateString());
- getQueryEngine().execute(cft);
- return true;
+ /*
+ * Combine the messages into a single source to be consumed by a
+ * task.
+ */
+ int nassigned = 1;
+ final Iterator<IChunkMessage<IBindingSet>> mitr = accepted.iterator();
+ final IChunkMessage<IBindingSet> firstChunk = mitr.next();
+ // See BOpContext#isLastInvocation()
+ final boolean isLastInvocation = pipelined
+ // && nremaining == 0
+ // && maxParallel == 1
+ // && isOperatorDone(bundle.bopId)
+ && firstChunk.isLastInvocation()
+ ;
+ /*
+ * Note: There is no longer any reliance on the IAsynchronous
+ * Iterator API here. It is perfectly sufficient to only
+ * implement ICloseableIterator. Query operator and chunk
+ * message implementations should be revisited with this
+ * simplifying assumption in mind.
+ *
+ * @see https://sourceforge.net/apps/trac/bigdata/ticket/475
+ */
+ final IMultiSourceCloseableIterator<IBindingSet[]> source = new MultiSourceSequentialCloseableIterator<IBindingSet[]>(//
+ // accepted.remove(0).getChunkAccessor().iterator()//
+ firstChunk.getChunkAccessor().iterator()//
+ );
+ // for (IChunkMessage<IBindingSet> msg : accepted) {
+ // source.add(msg.getChunkAccessor().iterator());
+ // #of solutions accepted across those chunk messages.
+ final long solutionsAccepted;
+ {
+ long na = firstChunk.getSolutionCount();
+ while (mitr.hasNext()) {
+ final IChunkMessage<IBindingSet> msg = mitr.next();
+ na += msg.getSolutionCount();
+ source.add(msg.getChunkAccessor().iterator());
+ nassigned++;
+ }
+ solutionsAccepted = na;
+ }
+ if (nassigned != naccepted)
+ throw new AssertionError();
+ /*
+ * Create task to consume that source.
+ */
+ final ChunkFutureTask cft;
+ try {
+ cft = new ChunkFutureTask(
+ new ChunkTask(bundle.bopId, bundle.shardId,
+ naccepted, isLastInvocation, source));
+ } catch (Throwable t2) {
+ // Ensure accepted messages are released();
+ safeRelease(accepted);
+ halt(t2); // ensure query halts.
+ if (getCause() != null) {
+ // Abnormal termination - wrap and rethrow.
+ throw new RuntimeException(t2);
+ }
+ // normal termination - swallow the exception.
+ return false;
+ }
+ /*
+ * Save the Future for this task. Together with the logic above this
+ * may be used to limit the #of concurrent tasks per (bopId,shardId)
+ * to one for a given query.
+ */
+ if (map == null) {
+ map = new ConcurrentHashMap<ChunkFutureTask, ChunkFutureTask>();
+ operatorFutures.put(bundle, map);
+ }
+ map.put(cft, cft);
+ /*
+ * Submit task for execution (asynchronous).
+ */
+ if (log.isDebugEnabled()) {
+ log.debug("Running task: bop=" + bundle.bopId
+ + ", atOnceReady=" + atOnceReady + ", bop="
+ + bop.toShortString() + ", messages=" + naccepted
+ + ", solutions=" + solutionsAccepted
+ + ", runState=" + runStateString());
+ }
+ getQueryEngine().execute(cft);
+ return true;
} catch(Throwable t) {
// Ensure accepted messages are released();
safeRelease(accepted);
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-04 18:29:19
|
Revision: 7717
http://bigdata.svn.sourceforge.net/bigdata/?rev=7717&view=rev
Author: thompsonbry
Date: 2014-01-04 18:29:11 +0000 (Sat, 04 Jan 2014)
Log Message:
-----------
@Override
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/RunState.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/RunState.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/RunState.java 2014-01-04 17:51:55 UTC (rev 7716)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/RunState.java 2014-01-04 18:29:11 UTC (rev 7717)
@@ -329,7 +329,8 @@
* @see #startOp(IStartOpMessage, boolean)
*/
final Set<Integer/* bopId */> lastPassRequested = new LinkedHashSet<Integer>();
-
+
+ @Override
public String toString() {
return toString(new StringBuilder()).toString();
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-04 17:43:56
|
Revision: 7715
http://bigdata.svn.sourceforge.net/bigdata/?rev=7715&view=rev
Author: thompsonbry
Date: 2014-01-04 17:43:49 +0000 (Sat, 04 Jan 2014)
Log Message:
-----------
@Override annotations.
Javadoc on AtOnceHint (it incorrectly stated that the source chunks would be merged before the operator was executed).
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/LocalChunkMessage.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/EmptyChunkMessage.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/NIOChunkMessage.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/ThickChunkMessage.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/relation/accesspath/MultiSourceSequentialCloseableIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/fed/TestNIOChunkMessage.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AtOnceHint.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/LocalChunkMessage.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/LocalChunkMessage.java 2014-01-04 17:15:04 UTC (rev 7714)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/LocalChunkMessage.java 2014-01-04 17:43:49 UTC (rev 7715)
@@ -55,34 +55,42 @@
*/
private IBindingSet[][] bindingSetChunks;
+ @Override
public IQueryClient getQueryController() {
return queryController;
}
+ @Override
public UUID getQueryControllerId() {
return queryControllerId;
}
+ @Override
public UUID getQueryId() {
return queryId;
}
+ @Override
public int getBOpId() {
return bopId;
}
+ @Override
public int getPartitionId() {
return partitionId;
}
+ @Override
public boolean isLastInvocation() {
return false; // Never.
}
+ @Override
public boolean isMaterialized() {
return true;
}
+ @Override
public int getSolutionCount() {
return solutionCount;
}
@@ -153,6 +161,7 @@
}
+ @Override
public String toString() {
return getClass().getName() + "{queryId=" + queryId + ",bopId=" + bopId
@@ -161,10 +170,12 @@
}
+ @Override
public void materialize(FederatedRunningQuery runningQuery) {
// NOP
}
+ @Override
public void release() {
final ChunkAccessor tmp = chunkAccessor;
if (tmp != null) {
@@ -177,6 +188,7 @@
}
}
+ @Override
public IChunkAccessor<IBindingSet> getChunkAccessor() {
if (chunkAccessor == null) {
chunkAccessor = new ChunkAccessor();
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/EmptyChunkMessage.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/EmptyChunkMessage.java 2014-01-04 17:15:04 UTC (rev 7714)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/EmptyChunkMessage.java 2014-01-04 17:43:49 UTC (rev 7715)
@@ -65,34 +65,42 @@
final private boolean lastInvocation;
+ @Override
public IQueryClient getQueryController() {
return queryController;
}
+ @Override
public UUID getQueryControllerId() {
return queryControllerId;
}
+ @Override
public UUID getQueryId() {
return queryId;
}
+ @Override
public int getBOpId() {
return bopId;
}
+ @Override
public int getPartitionId() {
return partitionId;
}
+ @Override
public boolean isLastInvocation() {
return true; // Always.
}
+ @Override
public boolean isMaterialized() {
return true;
}
+ @Override
public int getSolutionCount() {
return 0;
}
@@ -101,6 +109,7 @@
return 0;
}
+ @Override
public String toString() {
return getClass().getName() + "{queryId=" + queryId + ",bopId=" + bopId
@@ -143,14 +152,17 @@
}
+ @Override
public void materialize(FederatedRunningQuery runningQuery) {
// NOP
}
+ @Override
public void release() {
// NOP
}
+ @Override
public IChunkAccessor<E> getChunkAccessor() {
return new IChunkAccessor<E>() {
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/NIOChunkMessage.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/NIOChunkMessage.java 2014-01-04 17:15:04 UTC (rev 7714)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/NIOChunkMessage.java 2014-01-04 17:43:49 UTC (rev 7715)
@@ -103,26 +103,32 @@
*/
final private InetSocketAddress addr;
+ @Override
public IQueryClient getQueryController() {
return queryController;
}
+ @Override
public UUID getQueryControllerId() {
return queryControllerId;
}
+ @Override
public UUID getQueryId() {
return queryId;
}
+ @Override
public int getBOpId() {
return bopId;
}
+ @Override
public int getPartitionId() {
return partitionId;
}
+ @Override
public boolean isLastInvocation() {
return false; // Never.
}
@@ -132,6 +138,7 @@
*
* @todo we could track this in total and in {@link A} on a per-slice basis.
*/
+ @Override
public int getSolutionCount() {
return solutionCount;
}
@@ -149,6 +156,7 @@
return addr;
}
+ @Override
public String toString() {
return getClass().getName() + "{queryId=" + queryId + ",bopId=" + bopId
@@ -327,6 +335,7 @@
}
+ @Override
public boolean isMaterialized() {
return materialized != null;
@@ -335,6 +344,7 @@
private volatile List<IAllocation> materialized = null;
+ @Override
public void materialize(final FederatedRunningQuery runningQuery) {
final AllocationContextKey key = new ShardContext(queryId, bopId,
@@ -353,6 +363,7 @@
/**
* Discard the materialized data.
*/
+ @Override
public void release() {
if (chunkAccessor != null) {
@@ -441,6 +452,7 @@
}
+ @Override
public IChunkAccessor<E> getChunkAccessor() {
if (chunkAccessor == null) {
@@ -511,6 +523,7 @@
}
+ @Override
public void close() {
if(open) {
@@ -523,6 +536,7 @@
}
+ @Override
public boolean hasNext() {
if(open && src.hasNext())
@@ -534,6 +548,7 @@
}
+ @Override
@SuppressWarnings("unchecked")
public E[] next() {
@@ -556,6 +571,7 @@
}
+ @Override
public void remove() {
throw new UnsupportedOperationException();
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/ThickChunkMessage.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/ThickChunkMessage.java 2014-01-04 17:15:04 UTC (rev 7714)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/fed/ThickChunkMessage.java 2014-01-04 17:43:49 UTC (rev 7715)
@@ -84,34 +84,42 @@
private byte[] data;
+ @Override
public IQueryClient getQueryController() {
return queryController;
}
+ @Override
public UUID getQueryControllerId() {
return queryControllerId;
}
+ @Override
public UUID getQueryId() {
return queryId;
}
+ @Override
public int getBOpId() {
return bopId;
}
+ @Override
public int getPartitionId() {
return partitionId;
}
+ @Override
public boolean isLastInvocation() {
return false; // Never.
}
+ @Override
public boolean isMaterialized() {
return true;
}
+ @Override
public int getSolutionCount() {
return solutionCount;
}
@@ -120,6 +128,7 @@
return data.length;
}
+ @Override
public String toString() {
return getClass().getName() + "{queryId=" + queryId + ",bopId=" + bopId
@@ -197,10 +206,12 @@
}
+ @Override
public void materialize(final FederatedRunningQuery runningQuery) {
// NOP
}
+ @Override
public void release() {
if (chunkAccessor != null)
chunkAccessor.close();
@@ -208,6 +219,7 @@
private transient volatile ChunkAccessor chunkAccessor = null;
+ @Override
public IChunkAccessor<E> getChunkAccessor() {
return new ChunkAccessor();
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/relation/accesspath/MultiSourceSequentialCloseableIterator.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/relation/accesspath/MultiSourceSequentialCloseableIterator.java 2014-01-04 17:15:04 UTC (rev 7714)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/relation/accesspath/MultiSourceSequentialCloseableIterator.java 2014-01-04 17:43:49 UTC (rev 7715)
@@ -71,6 +71,7 @@
current = src;
}
+ @Override
public void close() {
lock.lock();
try {
@@ -98,6 +99,7 @@
}
}
+ @Override
public boolean add(final ICloseableIterator<E> src) {
if (src == null)
throw new IllegalArgumentException();
@@ -164,6 +166,7 @@
}
}
+ @Override
public boolean hasNext() {
while (true) {
final ICloseableIterator<E> tmp = nextSource();
@@ -182,6 +185,7 @@
* {@link #next()} to throw {@link NoSuchElementException} if the
* iterator has been concurrently closed.
*/
+ @Override
public E next() {
while (true) {
final ICloseableIterator<E> tmp = nextSource();
@@ -192,6 +196,7 @@
}
}
+ @Override
public void remove() {
throw new UnsupportedOperationException();
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/fed/TestNIOChunkMessage.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/fed/TestNIOChunkMessage.java 2014-01-04 17:15:04 UTC (rev 7714)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/fed/TestNIOChunkMessage.java 2014-01-04 17:43:49 UTC (rev 7715)
@@ -298,6 +298,7 @@
/**
* Overridden to expose to the unit test.
*/
+ @Override
protected void materialize(
final ManagedResourceService resourceService,
final IAllocationContext allocationContext) {
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AtOnceHint.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AtOnceHint.java 2014-01-04 17:15:04 UTC (rev 7714)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AtOnceHint.java 2014-01-04 17:43:49 UTC (rev 7715)
@@ -28,6 +28,7 @@
package com.bigdata.rdf.sparql.ast.hints;
import com.bigdata.bop.PipelineOp;
+import com.bigdata.bop.engine.QueryEngine;
import com.bigdata.rdf.sparql.ast.ASTBase;
import com.bigdata.rdf.sparql.ast.IQueryNode;
import com.bigdata.rdf.sparql.ast.QueryHints;
@@ -35,9 +36,11 @@
/**
* Query hint marks the operator as requiring "at-once" evaluation. All
- * solutions will be buffered by the query engine before the operator is
+ * solutions will be buffered by the {@link QueryEngine} before the operator is
* evaluated. When it is evaluated, it will receive all solutions in a single
- * "chunk".
+ * invocation of that operator. However, the solutions MAY appear in multiple
+ * chunks since the {@link QueryEngine} does not guarantee that the chunk will
+ * be merged before the operator is invoked.
* <p>
* Note: The "at-once" hint is basically turned into <code>NOT(PIPELINED)</code>.
*
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-04 17:15:10
|
Revision: 7714
http://bigdata.svn.sourceforge.net/bigdata/?rev=7714&view=rev
Author: thompsonbry
Date: 2014-01-04 17:15:04 +0000 (Sat, 04 Jan 2014)
Log Message:
-----------
organize imports.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AtOnceHint.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AtOnceHint.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AtOnceHint.java 2014-01-04 02:38:30 UTC (rev 7713)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AtOnceHint.java 2014-01-04 17:15:04 UTC (rev 7714)
@@ -27,7 +27,6 @@
package com.bigdata.rdf.sparql.ast.hints;
-import com.bigdata.bop.BufferAnnotations;
import com.bigdata.bop.PipelineOp;
import com.bigdata.rdf.sparql.ast.ASTBase;
import com.bigdata.rdf.sparql.ast.IQueryNode;
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <dme...@us...> - 2014-01-04 02:38:38
|
Revision: 7713
http://bigdata.svn.sourceforge.net/bigdata/?rev=7713&view=rev
Author: dmekonnen
Date: 2014-01-04 02:38:30 +0000 (Sat, 04 Jan 2014)
Log Message:
-----------
initial commit of a vagrant provisioner.
Added Paths:
-----------
trunk/src/resources/deployment/
trunk/src/resources/deployment/vagrant/
trunk/src/resources/deployment/vagrant/systap-bigdata/
trunk/src/resources/deployment/vagrant/systap-bigdata/Berksfile
trunk/src/resources/deployment/vagrant/systap-bigdata/Berksfile.lock
trunk/src/resources/deployment/vagrant/systap-bigdata/CHANGELOG.md
trunk/src/resources/deployment/vagrant/systap-bigdata/Gemfile
trunk/src/resources/deployment/vagrant/systap-bigdata/README.md
trunk/src/resources/deployment/vagrant/systap-bigdata/Thorfile
trunk/src/resources/deployment/vagrant/systap-bigdata/Vagrantfile
trunk/src/resources/deployment/vagrant/systap-bigdata/attributes/
trunk/src/resources/deployment/vagrant/systap-bigdata/attributes/default.rb
trunk/src/resources/deployment/vagrant/systap-bigdata/chefignore
trunk/src/resources/deployment/vagrant/systap-bigdata/definitions/
trunk/src/resources/deployment/vagrant/systap-bigdata/files/
trunk/src/resources/deployment/vagrant/systap-bigdata/files/default/
trunk/src/resources/deployment/vagrant/systap-bigdata/files/default/test/
trunk/src/resources/deployment/vagrant/systap-bigdata/files/default/test/default_test.rb
trunk/src/resources/deployment/vagrant/systap-bigdata/libraries/
trunk/src/resources/deployment/vagrant/systap-bigdata/metadata.rb
trunk/src/resources/deployment/vagrant/systap-bigdata/providers/
trunk/src/resources/deployment/vagrant/systap-bigdata/recipes/
trunk/src/resources/deployment/vagrant/systap-bigdata/recipes/configure.rb
trunk/src/resources/deployment/vagrant/systap-bigdata/recipes/default.rb
trunk/src/resources/deployment/vagrant/systap-bigdata/resources/
trunk/src/resources/deployment/vagrant/systap-bigdata/templates/
trunk/src/resources/deployment/vagrant/systap-bigdata/templates/default/
Added: trunk/src/resources/deployment/vagrant/systap-bigdata/Berksfile
===================================================================
--- trunk/src/resources/deployment/vagrant/systap-bigdata/Berksfile (rev 0)
+++ trunk/src/resources/deployment/vagrant/systap-bigdata/Berksfile 2014-01-04 02:38:30 UTC (rev 7713)
@@ -0,0 +1,7 @@
+site :opscode
+
+cookbook "apt"
+cookbook "java", "~> 1.14.0"
+cookbook "tomcat"
+
+metadata
Added: trunk/src/resources/deployment/vagrant/systap-bigdata/Berksfile.lock
===================================================================
--- trunk/src/resources/deployment/vagrant/systap-bigdata/Berksfile.lock (rev 0)
+++ trunk/src/resources/deployment/vagrant/systap-bigdata/Berksfile.lock 2014-01-04 02:38:30 UTC (rev 7713)
@@ -0,0 +1,28 @@
+{
+ "sources": {
+ "apt": {
+ "locked_version": "2.2.0"
+ },
+ "java": {
+ "locked_version": "1.14.0"
+ },
+ "tomcat": {
+ "locked_version": "0.15.2"
+ },
+ "systap-bigdata": {
+ "path": "."
+ },
+ "windows": {
+ "locked_version": "1.10.0"
+ },
+ "chef_handler": {
+ "locked_version": "1.1.4"
+ },
+ "aws": {
+ "locked_version": "1.0.0"
+ },
+ "openssl": {
+ "locked_version": "1.1.0"
+ }
+ }
+}
Added: trunk/src/resources/deployment/vagrant/systap-bigdata/CHANGELOG.md
===================================================================
--- trunk/src/resources/deployment/vagrant/systap-bigdata/CHANGELOG.md (rev 0)
+++ trunk/src/resources/deployment/vagrant/systap-bigdata/CHANGELOG.md 2014-01-04 02:38:30 UTC (rev 7713)
@@ -0,0 +1,8 @@
+# CHANGELOG for systap-bigdata
+
+This file is used to list changes made in each version of systap-bigdata.
+
+## 0.1.0:
+
+* Initial release of systap-bigdata
+
Added: trunk/src/resources/deployment/vagrant/systap-bigdata/Gemfile
===================================================================
--- trunk/src/resources/deployment/vagrant/systap-bigdata/Gemfile (rev 0)
+++ trunk/src/resources/deployment/vagrant/systap-bigdata/Gemfile 2014-01-04 02:38:30 UTC (rev 7713)
@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+
+gem 'berkshelf'
Added: trunk/src/resources/deployment/vagrant/systap-bigdata/README.md
===================================================================
--- trunk/src/resources/deployment/vagrant/systap-bigdata/README.md (rev 0)
+++ trunk/src/resources/deployment/vagrant/systap-bigdata/README.md 2014-01-04 02:38:30 UTC (rev 7713)
@@ -0,0 +1,114 @@
+systap-bigdata Cookbook
+======================
+This cookbook provides [http://www.bigdata.com/bigdata/blog/](bigdata v1.3.0) under Tomcat 7 (latest), with Oracle JDK 7 (latest) within an Ubuntu 12.0.4 VM.
+
+Typical synopsis:
+
+ % vagrant up
+
+ The bigdata service is then available at: http://33.33.33.10:8080/bigdata/
+
+
+Requirements
+------------
+
+#### packages
+In a stand alone context, this cookbook assumes the following resources have been installed:
+
+* `VirtualBox` - Virtual machine provider [http://virtualbox.org/](http://virtualbox.org/)
+* `Vagrant` - Environment assembler [http://vagrantup.com/](http://vagrantup.com/)
+* `Berkshelf` - The Berkshelf cookbook manager [http://berkshelf.com/](http://berkshelf.com/).
+
+
+#### cookbook dependencies
+Chef 10.14.2 or higher - has not been tested with previous versions.
+
+The following Opscode cookbooks are dependencies (automatically retrieved by `Berkshelf`):
+
+* apt
+* java
+* tomcat
+
+
+
+Attributes
+----------
+
+
+#### systap-bigdata::default
+<table>
+ <tr>
+ <th>Key</th>
+ <th>Type</th>
+ <th>Description</th>
+ <th>Default</th>
+ </tr>
+ <tr>
+ <td><tt>url</tt></td>
+ <td>String</td>
+ <td>where to download the bigdata.war file form</td>
+ <td><tt>http://sourceforge.net/projects/bigdata/
+ files/bigdata/1.3.0/bigdata.war/download</tt></td>
+ </tr>
+ <tr>
+ <td><tt>home</tt></td>
+ <td>String</td>
+ <td>where the extracted bigdata.war contents are found</td>
+ <td>Default: <tt>/var/lib/tomcat7/webapps/bigdata</tt></td>
+ </tr>
+ <tr>
+ <td><tt>etc</tt></td>
+ <td>String</td>
+ <td>Where under the tomcat root the log files and the bigdata.jni should reside. Discussed in <a href="http://sourceforge.net/apps/mediawiki/bigdata/index.php?title=NanoSparqlServer#Common_Startup_Problems">"Common Startup Problmems</a></td>
+ <td>Default: <tt>/var/lib/tomcat7/webapps/bigdata/etc</tt></td>
+ </tr>
+</table>
+
+
+Usage
+-----
+### Stand Alone Context
+To bring the VM up the first time, or any future time after a `halt`, invoke from the cookbook directory:
+
+ % vagrant up
+
+The cookbbok will retrieve the Ubuntu 12.04 VM, Oracle's JDK 7, Apahce's Tomcat 7 and the Bigdata WAR file. These downloads may take a significant amount of time to complete. Should a download be interupted or some other error occur, continue with:
+
+ % vagrant provision
+
+Once complete, the bigdata server will be available under:
+ [http://33.33.33.10:8080/bigdata/](http://33.33.33.10:8080/bigdata/)
+
+To halt the VM:
+
+ % vagrant halt
+
+To delete the VM and from VirtualBox:
+
+ % vagrant destroy
+
+To login into the VM:
+
+ % vagrant ssh
+
+
+### Cookbook Context
+
+To use as a reciple in new cookbook, just include `systap-bigdata` in your node's `run_list` in the standard way:
+
+```
+ "run_list": [
+ "recipe[systap-bigdata::default]",
+ ...
+ ]
+
+```
+
+
+License and Authors
+-------------------
+Author:: Daniel Mekonnen [daniel<no-spam-at>systap.com]
+
+```
+This pakcage may be resiributed under the same terms and conditions as the Bigdata project that it is a part of.
+```
Added: trunk/src/resources/deployment/vagrant/systap-bigdata/Thorfile
===================================================================
--- trunk/src/resources/deployment/vagrant/systap-bigdata/Thorfile (rev 0)
+++ trunk/src/resources/deployment/vagrant/systap-bigdata/Thorfile 2014-01-04 02:38:30 UTC (rev 7713)
@@ -0,0 +1,5 @@
+# encoding: utf-8
+
+require 'bundler'
+require 'bundler/setup'
+require 'berkshelf/thor'
Added: trunk/src/resources/deployment/vagrant/systap-bigdata/Vagrantfile
===================================================================
--- trunk/src/resources/deployment/vagrant/systap-bigdata/Vagrantfile (rev 0)
+++ trunk/src/resources/deployment/vagrant/systap-bigdata/Vagrantfile 2014-01-04 02:38:30 UTC (rev 7713)
@@ -0,0 +1,94 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+Vagrant.require_plugin "vagrant-berkshelf"
+
+Vagrant.configure("2") do |config|
+ # All Vagrant configuration is done here. The most common configuration
+ # options are documented and commented below. For a complete reference,
+ # please see the online documentation at vagrantup.com.
+
+ config.vm.hostname = "systap-bigdata"
+
+ # Every Vagrant virtual environment requires a box to build off of.
+ config.vm.box = "precise64"
+
+ # The url from where the 'config.vm.box' box will be fetched if it
+ # doesn't already exist on the user's system.
+ # config.vm.box_url = "http://repo.px.net/vagrant/#{config.vm.box}.box"
+ config.vm.box_url = "http://files.vagrantup.com/precise64.box"
+
+ # Assign this VM to a host-only network IP, allowing you to access it
+ # via the IP. Host-only networks can talk to the host machine as well as
+ # any other machines on the same network, but cannot be accessed (through this
+ # network interface) by any external networks.
+ config.vm.network :private_network, ip: "33.33.33.10"
+
+ # Create a public network, which generally matched to bridged network.
+ # Bridged networks make the machine appear as another physical device on
+ # your network.
+
+ # config.vm.network :public_network
+
+ # Create a forwarded port mapping which allows access to a specific port
+ # within the machine from a port on the host machine. In the example below,
+ # accessing "localhost:8080" will access port 80 on the guest machine.
+
+ # Share an additional folder to the guest VM. The first argument is
+ # the path on the host to the actual folder. The second argument is
+ # the path on the guest to mount the folder. And the optional third
+ # argument is a set of non-required options.
+ # config.vm.synced_folder "../data", "/vagrant_data"
+
+ # Provider-specific configuration so you can fine-tune various
+ # backing providers for Vagrant. These expose provider-specific options.
+ # Example for VirtualBox:
+ #
+ # config.vm.provider :virtualbox do |vb|
+ # # Don't boot with headless mode
+ # vb.gui = true
+ #
+ # # Use VBoxManage to customize the VM. For example to change memory:
+ # vb.customize ["modifyvm", :id, "--memory", "1024"]
+ # end
+ #
+ # View the documentation for the provider you're using for more
+ # information on available options.
+
+ # The path to the Berksfile to use with Vagrant Berkshelf
+ # config.berkshelf.berksfile_path = "./Berksfile"
+
+ # Enabling the Berkshelf plugin. To enable this globally, add this configuration
+ # option to your ~/.vagrant.d/Vagrantfile file
+ config.berkshelf.enabled = true
+
+ # An array of symbols representing groups of cookbook described in the Vagrantfile
+ # to exclusively install and copy to Vagrant's shelf.
+ # config.berkshelf.only = []
+
+ # An array of symbols representing groups of cookbook described in the Vagrantfile
+ # to skip installing and copying to Vagrant's shelf.
+ # config.berkshelf.except = []
+
+ config.vm.provision :chef_solo do |chef|
+ chef.json = {
+ "systap-bigdata" => {
+ # "1.3.0" => { :version => "1.3.5", :url => "http://repo.px.net/software/bigdata-1.3.0.war"
+ },
+ :tomcat => {
+ :base_version => "7"
+ },
+ :java => {
+ "install_flavor" => "oracle",
+ :jdk_version => "7",
+ :oracle => { 'accept_oracle_download_terms' => true }
+ }
+ }
+
+ chef.run_list = [
+ "recipe[apt]",
+ "recipe[java]",
+ "recipe[tomcat::default]",
+ "recipe[systap-bigdata::default]"
+ ]
+ end
+end
Added: trunk/src/resources/deployment/vagrant/systap-bigdata/attributes/default.rb
===================================================================
--- trunk/src/resources/deployment/vagrant/systap-bigdata/attributes/default.rb (rev 0)
+++ trunk/src/resources/deployment/vagrant/systap-bigdata/attributes/default.rb 2014-01-04 02:38:30 UTC (rev 7713)
@@ -0,0 +1,6 @@
+default['tomcat']['base_version'] = 7
+default['systap-bigdata'][:url] = "http://sourceforge.net/projects/bigdata/files/bigdata/1.3.0/bigdata.war/download"
+
+webapp_dir = node['tomcat']['webapp_dir']
+default['systap-bigdata'][:home] = webapp_dir + "/bigdata"
+default['systap-bigdata'][:etc] = webapp_dir + "/bigdata/etc"
Added: trunk/src/resources/deployment/vagrant/systap-bigdata/chefignore
===================================================================
--- trunk/src/resources/deployment/vagrant/systap-bigdata/chefignore (rev 0)
+++ trunk/src/resources/deployment/vagrant/systap-bigdata/chefignore 2014-01-04 02:38:30 UTC (rev 7713)
@@ -0,0 +1,96 @@
+# Put files/directories that should be ignored in this file when uploading
+# or sharing to the community site.
+# Lines that start with '# ' are comments.
+
+# OS generated files #
+######################
+.DS_Store
+Icon?
+nohup.out
+ehthumbs.db
+Thumbs.db
+
+# SASS #
+########
+.sass-cache
+
+# EDITORS #
+###########
+\#*
+.#*
+*~
+*.sw[a-z]
+*.bak
+REVISION
+TAGS*
+tmtags
+*_flymake.*
+*_flymake
+*.tmproj
+.project
+.settings
+mkmf.log
+
+## COMPILED ##
+##############
+a.out
+*.o
+*.pyc
+*.so
+*.com
+*.class
+*.dll
+*.exe
+*/rdoc/
+
+# Testing #
+###########
+.watchr
+.rspec
+spec/*
+spec/fixtures/*
+test/*
+features/*
+Guardfile
+Procfile
+
+# SCM #
+#######
+.git
+*/.git
+.gitignore
+.gitmodules
+.gitconfig
+.gitattributes
+.svn
+*/.bzr/*
+*/.hg/*
+*/.svn/*
+
+# Berkshelf #
+#############
+Berksfile
+Berksfile.lock
+cookbooks/*
+tmp
+
+# Cookbooks #
+#############
+CONTRIBUTING
+CHANGELOG*
+
+# Strainer #
+############
+Colanderfile
+Strainerfile
+.colander
+.strainer
+
+# Vagrant #
+###########
+.vagrant
+Vagrantfile
+
+# Travis #
+##########
+.travis.yml
Added: trunk/src/resources/deployment/vagrant/systap-bigdata/files/default/test/default_test.rb
===================================================================
--- trunk/src/resources/deployment/vagrant/systap-bigdata/files/default/test/default_test.rb (rev 0)
+++ trunk/src/resources/deployment/vagrant/systap-bigdata/files/default/test/default_test.rb 2014-01-04 02:38:30 UTC (rev 7713)
@@ -0,0 +1,6 @@
+require 'minitest/spec'
+describe_recipe 'systap-bigdata::test' do
+ it "is running the tomcat server" do
+ service('tomcat').must_be_running
+ end
+end
Added: trunk/src/resources/deployment/vagrant/systap-bigdata/metadata.rb
===================================================================
--- trunk/src/resources/deployment/vagrant/systap-bigdata/metadata.rb (rev 0)
+++ trunk/src/resources/deployment/vagrant/systap-bigdata/metadata.rb 2014-01-04 02:38:30 UTC (rev 7713)
@@ -0,0 +1,10 @@
+name 'systap-bigdata'
+maintainer 'Daniel Mekonnen'
+maintainer_email 'daniel<no-spam-at>systap.com'
+license 'All rights reserved'
+description 'Installs/Configures systap-bigdata'
+long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
+version '0.1.0'
+depends 'apt'
+depends 'java'
+depends 'tomcat'
Added: trunk/src/resources/deployment/vagrant/systap-bigdata/recipes/configure.rb
===================================================================
--- trunk/src/resources/deployment/vagrant/systap-bigdata/recipes/configure.rb (rev 0)
+++ trunk/src/resources/deployment/vagrant/systap-bigdata/recipes/configure.rb 2014-01-04 02:38:30 UTC (rev 7713)
@@ -0,0 +1,48 @@
+#
+# Cookbook Name:: systap-bigdata
+# Recipe:: configure
+#
+# Copyright 2013, Systap
+#
+#
+include_recipe "tomcat"
+
+directory node['systap-bigdata'][:etc] do
+ owner node['tomcat']['user']
+ group node['tomcat']['group']
+ mode 00755
+ action :create
+ #
+ # This is a little hackish. We need to wait for tomcat to extract the bigdata.war file before we can modify
+ # resources within the bigdata folder. We'll attempt to update this to use the chef notification system later.
+ #
+ retries 3
+ retry_delay 10
+end
+
+execute "set absolute path for RWStore.properties" do
+ cwd "#{node['systap-bigdata'][:home]}/WEB-INF"
+ command "sed -i 's|<param-value>../webapps/bigdata/RWStore.properties|<param-value>#{node['systap-bigdata'][:home]}/RWStore.properties|' web.xml"
+end
+
+execute "set path for bigdata.jnl file" do
+ cwd "#{node['systap-bigdata'][:home]}"
+ command "sed -i 's|=bigdata.jnl|=#{node['systap-bigdata'][:etc]}/bigdata.jnl|' RWStore.properties"
+end
+
+
+execute "set ruleLog in log4j.properties " do
+ cwd "#{node['systap-bigdata'][:home]}/WEB-INF/classes"
+ command "sed -i 's|log4j.appender.queryLog.File=queryLog.csv|log4j.appender.queryLog.File=#{node['systap-bigdata'][:etc]}/queryLog.csv|' log4j.properties"
+end
+
+execute "set ruleLog in log4j.properties " do
+ cwd "#{node['systap-bigdata'][:home]}/WEB-INF/classes"
+ command "sed -i 's|log4j.appender.ruleLog.File=rules.log|log4j.appender.ruleLog.File=#{node['systap-bigdata'][:etc]}/rules.log|' log4j.properties"
+end
+
+execute "set ruleLog in log4j.properties " do
+ cwd "#{node['systap-bigdata'][:home]}/WEB-INF/classes"
+ command "sed -i 's|log4j.appender.queryRunStateLog.File=queryRunStateLog.csv|log4j.appender.queryRunStateLog.File=#{node['systap-bigdata'][:etc]}/queryRunStateLog.csv|' log4j.properties"
+end
+
Added: trunk/src/resources/deployment/vagrant/systap-bigdata/recipes/default.rb
===================================================================
--- trunk/src/resources/deployment/vagrant/systap-bigdata/recipes/default.rb (rev 0)
+++ trunk/src/resources/deployment/vagrant/systap-bigdata/recipes/default.rb 2014-01-04 02:38:30 UTC (rev 7713)
@@ -0,0 +1,60 @@
+#
+# Cookbook Name:: systap-bigdata
+# Recipe:: default
+#
+# Copyright 2013, Systap
+#
+#
+include_recipe "tomcat"
+
+#
+# Install the WAR file:
+#
+remote_file "#{node['tomcat']['webapp_dir']}/bigdata.war" do
+ source node['systap-bigdata'][:url]
+ owner node['tomcat']['user']
+ group node['tomcat']['group']
+end
+
+
+#
+# Configuration for Tomcat on Ubuntu:
+#
+directory node['systap-bigdata'][:etc] do
+ owner node['tomcat']['user']
+ group node['tomcat']['group']
+ mode 00755
+ action :create
+ #
+ # This is a little hackish. We need to wait for tomcat to extract the bigdata.war file before we can modify
+ # resources within the bigdata folder. We'll attempt to update this to use the chef notification system later.
+ #
+ retries 3
+ retry_delay 10
+end
+
+execute "set absolute path for RWStore.properties" do
+ cwd "#{node['systap-bigdata'][:home]}/WEB-INF"
+ command "sed -i 's|<param-value>../webapps/bigdata/RWStore.properties|<param-value>#{node['systap-bigdata'][:home]}/RWStore.properties|' web.xml"
+end
+
+execute "set path for bigdata.jnl file" do
+ cwd "#{node['systap-bigdata'][:home]}"
+ command "sed -i 's|=bigdata.jnl|=#{node['systap-bigdata'][:etc]}/bigdata.jnl|' RWStore.properties"
+end
+
+
+execute "set ruleLog in log4j.properties " do
+ cwd "#{node['systap-bigdata'][:home]}/WEB-INF/classes"
+ command "sed -i 's|log4j.appender.queryLog.File=queryLog.csv|log4j.appender.queryLog.File=#{node['systap-bigdata'][:etc]}/queryLog.csv|' log4j.properties"
+end
+
+execute "set ruleLog in log4j.properties " do
+ cwd "#{node['systap-bigdata'][:home]}/WEB-INF/classes"
+ command "sed -i 's|log4j.appender.ruleLog.File=rules.log|log4j.appender.ruleLog.File=#{node['systap-bigdata'][:etc]}/rules.log|' log4j.properties"
+end
+
+execute "set ruleLog in log4j.properties " do
+ cwd "#{node['systap-bigdata'][:home]}/WEB-INF/classes"
+ command "sed -i 's|log4j.appender.queryRunStateLog.File=queryRunStateLog.csv|log4j.appender.queryRunStateLog.File=#{node['systap-bigdata'][:etc]}/queryRunStateLog.csv|' log4j.properties"
+end
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2014-01-03 19:18:28
|
Revision: 7712
http://bigdata.svn.sourceforge.net/bigdata/?rev=7712&view=rev
Author: thompsonbry
Date: 2014-01-03 19:18:16 +0000 (Fri, 03 Jan 2014)
Log Message:
-----------
Changes to PipelineJoin, IBindingSetAccessPath, IHashJoinUtility,
JVMHashJoinUtility, HTreeHashJoinUtility, ServiceCallOp, etc. intended
to reduce re-chunking during vectored query evaluation. This change
provides correct accounting for chunksIn and unitsIn for the solution
set hash join and improved memory utilization through elimination of
some unnecessary dechunking and rechunking in the hash join API.
Changes were also made to the query hint infrastructure in order to
ensure that hint:chunkSize was correctly applied to HTree
operators. This is important now that the HTreeHashJoinUtility no
longer rechunks to a hard-coded chunkSize of 1000.
While working on this, I noticed that hint:chunkSize was not making it
onto the SliceOp, onto the Predicate associated with a PipelineJoin
(where it controls the vectoring for reading on the access path),
etc. This was due to both the types of nodes to which the IQueryHint
implementations were willing to apply themselves and the types of
nodes to which the ASTQueryHintOptimizer was willing to apply query
hints. I have made both more expansive.
To help analyze the issue with query hints (which was necessary to
assess the performance impact of my change to vectoring in the
HTreeHashJoinUtility) and to help analyze the problem with the
stochastic behavior of the HTree, I also added significantly more
information into the predSummary and added columns (in the detailed
explain mode) for the PipelineOp and Predicate annotations.
I also touched a lot of classes, adding @Override and final
annotations. A number of this were in the striterator package (when I
added a CloseableChunkedIteratorWrapperConverter to address the
rechunking pattern in HTreeHashJoinUtility, IBindingSetAccessPath, and
BOpContext#solutions()) and in the ast package. Several of join
operators were also touched, either to support the fix of the
rechunking pattern or to eliminate some old (and commented out) code.
I have added more unit tests of the query hints mechanisms. I found
and fixed several places where query hints were not being applied when
generating pipeline operators (AST2BOpUtility) and where query hints
were not being copied from one AST node to another when making
structural changes to the AST, e.g., ASTBottomUpOptimizer and
ASTSparql11SubqueryOptimizer both create a NamedSubqueryRoot and a
NamedSubqueryInclude. However, they were not copying across the query
hints from the parent join group (ASTBottomUpOptimizer) and the
original SubqueryRoot (ASTSparql11SubqueryOptimizer).
com.bigdata.rdf.sparql.ast::
- ASTBase: javadoc on Annotations.QUERY_HINTS. final and @Override annotations.
- AssignmentNode: final and @Override annotations.
- GraphPatternGroup: final and @Override annotations.
- GraphNodeGroup: license header, final and @Override annotations, toString(int) changes.
- JoinGroupNode: license header. Made the OPTIMIZER property explicit. Added getQueryOptimizer() method. final and @Override annotations.
- QueryBase: added query hints into toString(int).
- QueryHints: referenced ticket #791 (clean up query hints).
- QueryOptimizerEnum: removed dead code.
- QueryRoot: final, @Override, and javadoc correction.
- SliceNode: javadoc on annotations, toString(int) now shows the query hints. this was done to have visibility into vectoring.
- StatementPatternNode: license header, final and @Override, javadoc, dead code elimination.
- ValueExpressionListBaseNode: @Override
com.bigdata.striterator::
- CloseableChunkedIteratorWrapperConverter: new class converts from IChunkedIterator<E> to ICloseableIterator visiting E[].
- TestCloseableChunkedIteratorWrapperConverter: new test suite.
- TestAll: include new test class.
- AbstractChunkedResolverator: final, @Override.
- ChunkedArrayIterator: final, @Override.
- ChunkedArraysIterator: javadoc, final, @Override.
- ChunkedConvertingIterator: final, @Override.
- ChunkedResolvingIterator: javadoc, final, @Override.
- ChunkedWrappedIterator: final, @Override.
- Chunkerator: close() now tests for ICloseable rather than ICloseableIterator.
- CloseableIteratorWrapper: final, @Override.
- Dechunkerator: javadoc, final, @Override, close() now tests for ICloseable rather than ICloseableIterator.
- DelegateChunkedIterator: final, @Override.
- GenericChunkedStriterator: removed unnecessary @SuppressWarning.
- IChunkedIterator: @Override for methods declared by Iterator.
- IChunkedStriterator: @Override for methods in base interface.
- MergeFilter: final, @Override.
- PushbackIterator: final, @Override.
- Resolver: final, @Override.
- Striterator: final, @Override.
com.bigdata.bop.join::
- TestPipelineJoin: @Override and super.setUp() / super.tearDown().
- AbstractHashJoinUtilityTestCase: Modified how we execute the hash join for the IHashJoinUtility API change.
- HashIndexOp: @Override
- HashJoinOp: @Override, javadoc, API change for IHashJoinUtility.hashJoin().
- HTreeHashIndexOp: @Override, final, dead code eliminated.
- HTreeHashJoinUtility: removed static chunkSize field. Vectoring is now controlled by hint:chunkSize. Pushed down the logic to track chunskIn and unitsIn for hashJoin2 (they were not being tracked).
- IHashJoinUtility: API change to remove rechunking pattern.
- JoinVariableNotBoundException: final annotations.
- JVMHashIndex: Slight efficiency change in makeKey(). @Override
- JVMHashIndexOp: final annotation.
- JVMHashJoinUtility: API change for hashJoin2(): now accepts ICloseableIteratoe<IBindingSet[]> and BOpStats and tracks unitsIn and chunksIn.
- PipelineJoin: IBindingSetAccessPath now returns an ICloseableIterator<IBindingSet[]>. Modified the AccessPathTask to handle the IBindingSet[] chunks. Used to be just IBindingSets.
- SolutionSetHashJoin: Modified to pass context.getSource() and BOpStats into the IHashJoinUtility rather than dechunking.
com.bigdata.bop.controller::
- HTreeNamedSubqueryOp, JVMNamedSubqueryOp, and INamedSubqueryOp: Added INamedSubqueryOp as a marker interface for the two implementation classes so we can identify those operators when they appear in a query plan.
- ServiceCallJoin: Modified to pass ICloseableIterator<IBindingSet[]> into IHashJoinUtility. This should be pushed down further. There is a TODO to do that when we address vectoring in/out of the SERVICE operator.
com.bigdata.bop.QueryEngine:
- QueryLog: Significantly expanded and improved performance counter reporting for Explain, especially in the "detail" mode.
com.bigdata.bop::
- AbstractAccessPathOp: removed unused methods. This is part of the query hints cleanup.
- BOpContext#solutions() was modified to return an ICloseableIterator visiting IBindingSet[]s. This is part of the rechunking change for IHashJoinUtility.
- BOpUtility: Added getOnly() method used to obtain the only instance of a BOp from a query plan or AST. This is used by unit tests.
com.bigdata.relation.accesspath::
- IBindingSetAccessPath: solutions() was modified to return an ICloseableIterator visiting IBindingSet[]s.
- AccessPath: solutions() was modified to return an ICloseableIterator visiting IBindingSet[]s.
com.bigdata.rdf.sparql.hints::
- Extensive changes to clean up query hints. Many query hints are now apply to IQueryNode rather than IJoinNode. javadoc. Test cases have been expanded.
com.bigdata.rdf.sparql.ast::
- ASTBottomUpOptimizer: modified to pass through query hints from the parent join group to the lifted out named subquery and the INCLUDE.
- ASTSparql11SubqueryOptimizer: modified to pass through query hints from the original subquery when it is lifted out into a named subquery. The hints are applied to both the new named subquery and the new named subquery include.
- ASTStaticJoinOptimizer: changes to isStaticOptimizer() to use JoinGroup.getQueryOptimizer();
- ASTQueryHintOptimizer: extensive changes. modified how the optimizer identifies the nodes for which it will delegate to a query hint - it used to only do this for QueryNodeBase, which was too restrictive. It now does this for everthing but value expressions. javadoc clarifying the intention of the class. Removed some dead code.
Note: ASTQueryHintOptimizer No longer adds all query hints with Scope:=Query to AST2BOpContext.queryHints. I have clarified this in documentation in both classes.
com.bigdata.rdf.sparql.eval::
- AST2BOpBase: Changed the pattern for applyQueryHints() to use both the AST node's query hints and the AST2BOpUtility global defaults whenever possible.
- AST2BOpUtility: Modified to more systematically pass along query hints from the AST node to the constructed PipelineOps. Modified to pass through BufferAnnotations and IPredicate annotations to the Predicate created from a StatementPatternNode. This allows us to control vectoring for AccessPath reads.
- AST2BOpFilters: Partially addressed pass through of query hints, but only those in the global scope. We need to change the method interfaces to pass through the bounding AST node or the queryHints for that AST node.
- AST2BOpContext: license header, javadoc.
I have run through all of the bop, AST, SPARQL, and NSS test suites
and everything is green.
TODO:
- (*) I have not yet resolved the HTree stochastic behavior. I will
continue to look at that once this checkpoint is committed. See
#763.
- (*) Query hints for the materialization pipeline
(ChunkedMaterializationOp, ConditionalRoutingOp) are not being
applied correctly because the caller's AST node (or its query
hints) are not being passed down. I am going to defer fixing this
for a moment while I look at the RTO integration. (The RTO needs
to be able to use AST2BOpJoins#join(), which is the main entry
point into the materialization pipeline code.) See #791.
- AST2BOpBase: Once we fix this, we really do not need to pass in
the AST2BOpContext's query hints into applyQueryHints() any
more. The impact will be achieved by passing down the query
hints from the appropriate bounding AST node. The
ASTQueryHintOptimizer is responsible for making sure that the
query hints are applied to those AST nodes.
- (*) Check query performance on LUBM U50, BSBM 100M, and govtrack.
The changes in the HTree vectoring could hit govtrack, but we
should be able to override hint:chunkSize if necessary to correct
for this (or automatically increase the chunkSize for the analytic
query mode, or use dynamic rechunking, etc). The changes in the
ASTQueryHintsOptimizer could hit all queries, but I was pretty
careful and do not expect to see any performance regressions.
See http://sourceforge.net/apps/trac/bigdata/ticket/483 (Eliminate unnecessary dechunking and rechunking)
See http://sourceforge.net/apps/trac/bigdata/ticket/791 (Clean up query hints)
See http://sourceforge.net/apps/trac/bigdata/ticket/763 (Stochastic results with Analytic Query Mode)
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/AbstractAccessPathOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpContext.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpUtility.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/HTreeNamedSubqueryOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/JVMNamedSubqueryOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/ServiceCallJoin.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashIndexOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HashIndexOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HashJoinOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/IHashJoinUtility.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/JVMHashIndex.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/JVMHashIndexOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/JoinVariableNotBoundException.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/SolutionSetHashJoinOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/ProjectionOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/SliceOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/relation/accesspath/AccessPath.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/relation/accesspath/IBindingSetAccessPath.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/AbstractChunkedResolverator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/ChunkedArrayIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/ChunkedArraysIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/ChunkedConvertingIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/ChunkedOrderedStriterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/ChunkedResolvingIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/ChunkedWrappedIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/Chunkerator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/CloseableIteratorWrapper.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/Dechunkerator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/DelegateChunkedIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/GenericChunkedStriterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/IChunkedIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/IChunkedStriterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/MergeFilter.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/PushbackIterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/Resolver.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/Striterator.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/join/AbstractHashJoinUtilityTestCase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/bop/join/TestPipelineJoin.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/striterator/TestAll.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/ASTBase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/AssignmentNode.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/GraphPatternGroup.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/GroupNodeBase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/JoinGroupNode.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/QueryBase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/QueryHints.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/QueryOptimizerEnum.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/QueryRoot.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/SliceNode.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/StatementPatternNode.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/ValueExpressionListBaseNode.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpBase.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpContext.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpFilters.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AbstractChunkSizeHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AbstractQueryHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AnalyticQueryHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/AtOnceHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/BufferChunkCapacityHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/BufferChunkOfChunksCapacityHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/ChunkSizeHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/IQueryHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/OptimizerQueryHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/PipelineMaxMessagesPerTaskHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/PipelineMaxParallelHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/PipelineQueueCapacityHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/QueryHintRegistry.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/hints/RTOSampleTypeQueryHint.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTBottomUpOptimizer.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTQueryHintOptimizer.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTSparql11SubqueryOptimizer.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTStaticJoinOptimizer.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/service/ServiceCall.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/TestQueryHints.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/query-hints-01.rq
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/query-hints-06.rq
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/INamedSubqueryOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/striterator/CloseableChunkedIteratorWrapperConverter.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/striterator/TestCloseableChunkedIteratorWrapperConverter.java
Property Changed:
----------------
branches/BIGDATA_RELEASE_1_3_0/
Property changes on: branches/BIGDATA_RELEASE_1_3_0
___________________________________________________________________
Modified: svn:ignore
- ant-build
src
bin
bigdata*.jar
ant-release
standalone
test*
countersfinal.xml
events.jnl
.settings
*.jnl
TestInsertRate.out
SYSTAP-BBT-result.txt
U10load+query
*.hprof
com.bigdata.cache.TestHardReferenceQueueWithBatchingUpdates.exp.csv
commit-log.txt
eventLog
dist
bigdata-test
com.bigdata.rdf.stress.LoadClosureAndQueryTest.*.csv
DIST.bigdata-*.tgz
REL.bigdata-*.tgz
queryLog*
queryRunState*
sparql.txt
benchmark
CI
+ ant-build
src
bin
bigdata*.jar
ant-release
standalone
test*
countersfinal.xml
events.jnl
.settings
*.jnl
TestInsertRate.out
SYSTAP-BBT-result.txt
U10load+query
*.hprof
com.bigdata.cache.TestHardReferenceQueueWithBatchingUpdates.exp.csv
commit-log.txt
eventLog
dist
bigdata-test
com.bigdata.rdf.stress.LoadClosureAndQueryTest.*.csv
DIST.bigdata-*.tgz
REL.bigdata-*.tgz
queryLog*
queryRunState*
sparql.txt
benchmark
CI
bsbm10-dataset.nt.gz
bsbm10-dataset.nt.zip
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/AbstractAccessPathOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/AbstractAccessPathOp.java 2013-12-31 19:48:13 UTC (rev 7711)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/AbstractAccessPathOp.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -76,26 +76,26 @@
super(op);
}
- /**
- * @see BufferAnnotations#CHUNK_CAPACITY
- */
- protected int getChunkCapacity() {
-
- return getProperty(Annotations.CHUNK_CAPACITY,
- Annotations.DEFAULT_CHUNK_CAPACITY);
+// /**
+// * @see BufferAnnotations#CHUNK_CAPACITY
+// */
+// protected int getChunkCapacity() {
+//
+// return getProperty(Annotations.CHUNK_CAPACITY,
+// Annotations.DEFAULT_CHUNK_CAPACITY);
+//
+// }
+//
+// /**
+// * @see BufferAnnotations#CHUNK_OF_CHUNKS_CAPACITY
+// */
+// protected int getChunkOfChunksCapacity() {
+//
+// return getProperty(Annotations.CHUNK_OF_CHUNKS_CAPACITY,
+// Annotations.DEFAULT_CHUNK_OF_CHUNKS_CAPACITY);
+//
+// }
- }
-
- /**
- * @see BufferAnnotations#CHUNK_OF_CHUNKS_CAPACITY
- */
- protected int getChunkOfChunksCapacity() {
-
- return getProperty(Annotations.CHUNK_OF_CHUNKS_CAPACITY,
- Annotations.DEFAULT_CHUNK_OF_CHUNKS_CAPACITY);
-
- }
-
// protected int getFullyBufferedReadThreshold() {
//
// return getProperty(Annotations.FULLY_BUFFERED_READ_THRESHOLD,
@@ -103,14 +103,14 @@
//
// }
- /**
- * @see BufferAnnotations#CHUNK_TIMEOUT
- */
- protected long getChunkTimeout() {
-
- return getProperty(Annotations.CHUNK_TIMEOUT,
- Annotations.DEFAULT_CHUNK_TIMEOUT);
-
- }
+// /**
+// * @see BufferAnnotations#CHUNK_TIMEOUT
+// */
+// protected long getChunkTimeout() {
+//
+// return getProperty(Annotations.CHUNK_TIMEOUT,
+// Annotations.DEFAULT_CHUNK_TIMEOUT);
+//
+// }
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpContext.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpContext.java 2013-12-31 19:48:13 UTC (rev 7711)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpContext.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -59,8 +59,9 @@
import com.bigdata.rwstore.sector.IMemoryManager;
import com.bigdata.striterator.ChunkedFilter;
import com.bigdata.striterator.Chunkerator;
-import com.bigdata.striterator.CloseableIteratorWrapper;
+import com.bigdata.striterator.CloseableChunkedIteratorWrapperConverter;
import com.bigdata.striterator.IChunkedIterator;
+import com.bigdata.striterator.IChunkedStriterator;
import cutthecrap.utils.striterators.ICloseableIterator;
@@ -1078,8 +1079,8 @@
}
/**
- * Convert an {@link IAccessPath#iterator()} into a stream of
- * {@link IBindingSet}s.
+ * Convert an {@link IAccessPath#iterator()} into a stream of chunks of
+ * {@link IBindingSet}.
*
* @param src
* The iterator draining the {@link IAccessPath}. This will visit
@@ -1090,7 +1091,7 @@
* Statistics to be updated as elements and chunks are consumed
* (optional).
*
- * @return The dechunked iterator visiting the solutions. The order of the
+ * @return An iterator visiting chunks of solutions. The order of the
* original {@link IElement}s is preserved.
*
* @see https://sourceforge.net/apps/trac/bigdata/ticket/209 (AccessPath
@@ -1105,14 +1106,15 @@
// * The array of distinct variables (no duplicates) to be
// * extracted from the visited {@link IElement}s.
@SuppressWarnings({ "rawtypes", "unchecked" })
- static public ICloseableIterator<IBindingSet> solutions(
+ static public ICloseableIterator<IBindingSet[]> solutions(
final IChunkedIterator<?> src, //
final IPredicate<?> pred,//
// final IVariable<?>[] varsx,
final BaseJoinStats stats//
) {
- return new CloseableIteratorWrapper(
+ //return new CloseableIteratorWrapper(
+ final IChunkedStriterator itr1 =
new com.bigdata.striterator.ChunkedStriterator(src).addFilter(
// new ChunkedFilter() {
new ChunkedFilter<IChunkedIterator<Object>, Object, Object>() {
@@ -1160,18 +1162,28 @@
}
- })) {
+ });
+ //) {
+//
+// /**
+// * Close the real source if the caller closes the returned iterator.
+// */
+// @Override
+// public void close() {
+// super.close();
+// src.close();
+// }
+// };
- /**
- * Close the real source if the caller closes the returned iterator.
- */
- @Override
- public void close() {
- super.close();
- src.close();
- }
- };
+ /*
+ * Convert from IChunkedIterator<IBindingSet> to
+ * ICloseableIterator<IBindingSet[]>. This is a fly weight conversion.
+ */
+ final ICloseableIterator<IBindingSet[]> itr2 = new CloseableChunkedIteratorWrapperConverter<IBindingSet>(
+ itr1);
+ return itr2;
+
}
/*
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpUtility.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpUtility.java 2013-12-31 19:48:13 UTC (rev 7711)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/BOpUtility.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -35,6 +35,7 @@
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
+import java.util.NoSuchElementException;
import java.util.Set;
import org.apache.log4j.Logger;
@@ -72,7 +73,7 @@
* Pre-order recursive visitation of the operator tree (arguments only, no
* annotations).
*/
- @SuppressWarnings("unchecked")
+ @SuppressWarnings({ "unchecked", "rawtypes" })
public static Iterator<BOp> preOrderIterator(final BOp op) {
return new Striterator(new SingleValueIterator(op))
@@ -466,6 +467,8 @@
* The type of the node to be extracted.
*
* @return A list containing those references.
+ *
+ * @see #visitAll(BOp, Class)
*/
public static <C> List<C> toList(final BOp op, final Class<C> clas) {
@@ -483,6 +486,44 @@
}
+ /**
+ * Return the sole instance of the specified class.
+ *
+ * @param op
+ * The root of the traversal.
+ * @param class1
+ * The class to look for.
+ * @return The sole instance of that class.
+ * @throws NoSuchElementException
+ * if there is no such instance.
+ * @throws RuntimeException
+ * if there is more than one such instance.
+ */
+ public static <C> C getOnly(final BOp op, final Class<C> class1) {
+ final Iterator<C> it = visitAll(op, class1);
+ if (!it.hasNext())
+ throw new NoSuchElementException("No instance found: class="
+ + class1);
+ final C ret = it.next();
+ if (it.hasNext())
+ throw new RuntimeException("More than one instance exists: class="
+ + class1);
+ return ret;
+ }
+
+ /**
+ * Return an iterator visiting references to all nodes of the given type
+ * (recursive, including annotations).
+ *
+ * @param op
+ * The root of the operator tree.
+ * @param clas
+ * The type of the node to be extracted.
+ *
+ * @return A iterator visiting those references.
+ *
+ * @see #toList(BOp, Class)
+ */
@SuppressWarnings("unchecked")
public static <C> Iterator<C> visitAll(final BOp op, final Class<C> clas) {
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/HTreeNamedSubqueryOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/HTreeNamedSubqueryOp.java 2013-12-31 19:48:13 UTC (rev 7711)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/HTreeNamedSubqueryOp.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -73,7 +73,7 @@
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
*/
-public class HTreeNamedSubqueryOp extends PipelineOp {
+public class HTreeNamedSubqueryOp extends PipelineOp implements INamedSubqueryOp {
static private final transient Logger log = Logger
.getLogger(HTreeNamedSubqueryOp.class);
@@ -151,7 +151,7 @@
}
- public HTreeNamedSubqueryOp(final BOp[] args, NV... annotations) {
+ public HTreeNamedSubqueryOp(final BOp[] args, final NV... annotations) {
this(args, NV.asMap(annotations));
@@ -164,6 +164,7 @@
}
+ @Override
public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
return new FutureTask<Void>(new ControllerTask(this, context));
@@ -266,6 +267,7 @@
/**
* Evaluate.
*/
+ @Override
public Void call() throws Exception {
try {
@@ -356,6 +358,7 @@
}
+ @Override
public Void call() throws Exception {
// The subquery
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/INamedSubqueryOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/INamedSubqueryOp.java (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/INamedSubqueryOp.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -0,0 +1,42 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+package com.bigdata.bop.controller;
+
+import com.bigdata.bop.join.SolutionSetHashJoinOp;
+
+/**
+ * Marker interface for named subquery evaluation. Solutions from the pipeline
+ * flow through this operator without modification. The subquery is evaluated
+ * exactly once, the first time this operator is invoked, and the solutions for
+ * the subquery are written onto a hash index. Those solutions are then joined
+ * back within the query at latter points in the query plan using a solution set
+ * hash join.
+ *
+ * @see SolutionSetHashJoinOp
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ */
+public interface INamedSubqueryOp {
+
+}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/JVMNamedSubqueryOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/JVMNamedSubqueryOp.java 2013-12-31 19:48:13 UTC (rev 7711)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/JVMNamedSubqueryOp.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -73,7 +73,7 @@
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
*/
-public class JVMNamedSubqueryOp extends PipelineOp {
+public class JVMNamedSubqueryOp extends PipelineOp implements INamedSubqueryOp {
static private final transient Logger log = Logger
.getLogger(JVMNamedSubqueryOp.class);
@@ -140,7 +140,7 @@
}
- public JVMNamedSubqueryOp(final BOp[] args, NV... annotations) {
+ public JVMNamedSubqueryOp(final BOp[] args, final NV... annotations) {
this(args, NV.asMap(annotations));
@@ -153,6 +153,7 @@
}
+ @Override
public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
return new FutureTask<Void>(new ControllerTask(this, context));
@@ -254,6 +255,7 @@
/**
* Evaluate.
*/
+ @Override
public Void call() throws Exception {
try {
@@ -344,6 +346,7 @@
}
+ @Override
public Void call() throws Exception {
// The subquery
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/ServiceCallJoin.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/ServiceCallJoin.java 2013-12-31 19:48:13 UTC (rev 7711)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/controller/ServiceCallJoin.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -69,6 +69,7 @@
import com.bigdata.relation.accesspath.IBuffer;
import com.bigdata.relation.accesspath.UnsyncLocalOutputBuffer;
import com.bigdata.striterator.ChunkedArrayIterator;
+import com.bigdata.striterator.Chunkerator;
import com.bigdata.util.InnerCause;
import com.bigdata.util.concurrent.LatchedExecutor;
@@ -571,6 +572,7 @@
}
+ @Override
public Void call() throws Exception {
final UnsyncLocalOutputBuffer<IBindingSet> unsyncBuffer = new UnsyncLocalOutputBuffer<IBindingSet>(
@@ -592,7 +594,7 @@
chunk), null/* stats */);
// The iterator draining the subquery
- ICloseableIterator<IBindingSet> serviceSolutionItr = null;
+ ICloseableIterator<IBindingSet[]> serviceSolutionItr = null;
try {
/*
@@ -609,10 +611,13 @@
* Do a hash join of the source solutions with the
* solutions from the service, outputting any solutions
* which join.
+ *
+ * Note:
*/
- state.hashJoin(serviceSolutionItr, unsyncBuffer);
-
+ state.hashJoin(serviceSolutionItr, null/* stats */,
+ unsyncBuffer);
+
}
} finally {
@@ -671,26 +676,35 @@
* SILENT is <code>true</code>.
*
* @throws Exception
+ *
+ * TODO RECHUNKING Push down the
+ * ICloseableIterator<IBindingSet[]> return type into
+ * the {@link ServiceCall} interface and the various
+ * ways in which we can execute a service call. Do this
+ * as part of vectoring solutions in and out of service
+ * calls?
*/
- private ICloseableIterator<IBindingSet> doServiceCall(
+ private ICloseableIterator<IBindingSet[]> doServiceCall(
final ServiceCall<? extends Object> serviceCall,
final IBindingSet[] left) throws Exception {
try {
+ final ICloseableIterator<IBindingSet> itr;
+
if (serviceCall instanceof BigdataServiceCall) {
- return doBigdataServiceCall(
+ itr = doBigdataServiceCall(
(BigdataServiceCall) serviceCall, left);
} else if (serviceCall instanceof ExternalServiceCall) {
- return doExternalServiceCall(
+ itr = doExternalServiceCall(
(ExternalServiceCall) serviceCall, left);
} else if (serviceCall instanceof RemoteServiceCall) {
- return doRemoteServiceCall(
+ itr = doRemoteServiceCall(
(RemoteServiceCall) serviceCall, left);
} else {
@@ -698,7 +712,13 @@
throw new AssertionError();
}
+
+ final ICloseableIterator<IBindingSet[]> itr2 = new Chunkerator<IBindingSet>(
+ itr, op.getChunkCapacity(), IBindingSet.class);
+
+ return itr2;
+
} catch (Throwable t) {
if (silent
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java 2013-12-31 19:48:13 UTC (rev 7711)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -61,6 +61,8 @@
import com.bigdata.bop.solutions.DropOp;
import com.bigdata.bop.solutions.GroupByOp;
import com.bigdata.bop.solutions.ProjectionOp;
+import com.bigdata.bop.solutions.SliceOp;
+import com.bigdata.btree.Tuple;
import com.bigdata.counters.render.XHTMLRenderer;
import com.bigdata.rawstore.Bytes;
import com.bigdata.rdf.sparql.ast.eval.AST2BOpJoins;
@@ -826,6 +828,10 @@
}
w.write("<th>bopSummary</th>");
w.write("<th>predSummary</th>");
+ if (detailedStats) {
+ w.write("<th>bopAnnotations</th>");
+ w.write("<th>predAnnotations</th>");
+ }
// metadata considered by the static optimizer.
if(detailedStats) {
w.write("<th>staticBestKeyOrder</th>"); // original key order assigned
@@ -1119,25 +1125,16 @@
w.write(TDx);
}
+ // bopSummary
w.write(TD);
- if(summary) {
+ if (summary) {
w.write("total");
} else {
w.write(cdata(bop.getClass().getSimpleName()));
w.write(cdata("[" + bopId + "]"));
- final Integer defaultSink = (Integer) bop
- .getProperty(PipelineOp.Annotations.SINK_REF);
- final Integer altSink = (Integer) bop
- .getProperty(PipelineOp.Annotations.ALT_SINK_REF);
- if (defaultSink != null) {
- w.write(cdata(", sink=" + defaultSink));
- }
- if (altSink != null) {
- w.write(cdata(", altSink=" + altSink));
- }
}
w.write(TDx);
-
+
/*
* Pperator summary (not shown for the "total" line).
*
@@ -1264,9 +1261,29 @@
w.write(cdata(Arrays.toString(((ProjectionOp) bop)
.getVariables())));
}
+ if (bop instanceof SliceOp) {
+ w.write(cdata("offset=" + ((SliceOp) bop).getOffset()));
+ w.write(cdata(", limit=" + ((SliceOp) bop).getLimit()));
+ }
}
- w.write(TDx); // end summary
+ w.write(TDx); // end predSummary
+ if (detailedStats) {
+ // bopAnnotations
+ w.write(TD);
+ showAnnotations(w, bop.annotations());
+ w.write(TDx);
+ }
+
+ if (detailedStats) {
+ // predAnnotations
+ w.write(TD);
+ if (pred != null) {
+ showAnnotations(w, pred.annotations());
+ }
+ w.write(TDx);
+ }
+
/*
* Static optimizer metadata.
*
@@ -1505,6 +1522,41 @@
}
/**
+ * Shows annotations on a {@link BOp}.
+ *
+ * @param w
+ * Where to write the XHTML data.
+ * @param anns
+ * The annotations (optional).
+ * @throws IOException
+ */
+ static private void showAnnotations(final Writer w,
+ final Map<String, Object> anns) throws IOException {
+ if (anns != null && !anns.isEmpty()) {
+ w.write("<dl>");
+ for (Map.Entry<String, Object> e : anns.entrySet()) {
+ w.write("<dt>");
+ final String key = e.getKey();
+ w.write(cdata(key));
+ w.write("</dt><dd>");
+ final Object val = e.getValue();
+ // See CoreBaseBop for this pattern.
+ if (val != null && val.getClass().isArray()) {
+ w.write(cdata(Arrays.toString((Object[]) val)));
+ } else if (key.equals(IPredicate.Annotations.FLAGS)) {
+ w.write(cdata(Tuple.flagString((Integer) val)));
+ } else if (val instanceof BOp) {
+ w.write(cdata(((BOp) val).toShortString()));
+ } else {
+ w.write(cdata("" + val));
+ }
+ w.write("</dd>");
+ }
+ w.write("</dl>");
+ }
+ }
+
+ /**
* Write a summary row for the query. The table element, header, and footer
* must be written separately.
*
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashIndexOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashIndexOp.java 2013-12-31 19:48:13 UTC (rev 7711)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashIndexOp.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -72,222 +72,22 @@
super(args, annotations);
-// if (getEvaluationContext() != BOpEvaluationContext.CONTROLLER) {
-// throw new IllegalArgumentException(
-// BOp.Annotations.EVALUATION_CONTEXT + "="
-// + getEvaluationContext());
-// }
-//
-// if (getMaxParallel() != 1) {
-// /*
-// * Parallel evaluation is not allowed. This operator writes on an
-// * object that is not thread-safe for mutation.
-// */
-// throw new IllegalArgumentException(
-// PipelineOp.Annotations.MAX_PARALLEL + "="
-// + getMaxParallel());
-// }
-//
-// if (!isLastPassRequested()) {
-// /*
-// * Last pass evaluation must be requested. This operator will not
-// * produce any outputs until all source solutions have been
-// * buffered.
-// */
-// throw new IllegalArgumentException(PipelineOp.Annotations.LAST_PASS
-// + "=" + isLastPassRequested());
-// }
-//
-// getRequiredProperty(Annotations.NAMED_SET_REF);
-//
-// @SuppressWarnings("unused")
-// final JoinTypeEnum joinType = (JoinTypeEnum) getRequiredProperty(Annotations.JOIN_TYPE);
-//
-// // Join variables must be specified.
-// final IVariable<?>[] joinVars = (IVariable[]) getRequiredProperty(Annotations.JOIN_VARS);
-//
-//// if (joinVars.length == 0)
-//// throw new IllegalArgumentException(Annotations.JOIN_VARS);
-//
-// for (IVariable<?> var : joinVars) {
-//
-// if (var == null)
-// throw new IllegalArgumentException(Annotations.JOIN_VARS);
-//
-// }
-
}
- public HTreeHashIndexOp(final BOp[] args, NV... annotations) {
+ public HTreeHashIndexOp(final BOp[] args, final NV... annotations) {
this(args, NV.asMap(annotations));
-
+
}
-// @Override
-// public BOpStats newStats() {
-//
-// return new NamedSolutionSetStats();
-//
-// }
-
@Override
protected HTreeHashJoinUtility newState(
final BOpContext<IBindingSet> context,
final INamedSolutionSetRef namedSetRef, final JoinTypeEnum joinType) {
- return new HTreeHashJoinUtility(
- context.getMemoryManager(namedSetRef.getQueryId()), this, joinType);
+ return new HTreeHashJoinUtility(context.getMemoryManager(namedSetRef
+ .getQueryId()), this, joinType);
}
-
-// public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
-//
-// return new FutureTask<Void>(new ControllerTask(this, context));
-//
-// }
-
-// /**
-// * Evaluates the subquery for each source binding set. If the controller
-// * operator is interrupted, then the subqueries are cancelled. If a subquery
-// * fails, then all subqueries are cancelled.
-// */
-// private static class ControllerTask implements Callable<Void> {
-//
-// private final BOpContext<IBindingSet> context;
-//
-// private final HTreeHashIndexOp op;
-//
-// private final NamedSolutionSetStats stats;
-//
-// private final IHashJoinUtility state;
-//
-// public ControllerTask(final HTreeHashIndexOp op,
-// final BOpContext<IBindingSet> context) {
-//
-// if (op == null)
-// throw new IllegalArgumentException();
-//
-// if (context == null)
-// throw new IllegalArgumentException();
-//
-// this.context = context;
-//
-// this.op = op;
-//
-// this.stats = ((NamedSolutionSetStats) context.getStats());
-//
-// // Metadata to identify the named solution set.
-// final NamedSolutionSetRef namedSetRef = (NamedSolutionSetRef) op
-// .getRequiredProperty(Annotations.NAMED_SET_REF);
-//
-// {
-//
-// /*
-// * First, see if the map already exists.
-// *
-// * Note: Since the operator is not thread-safe, we do not need
-// * to use a putIfAbsent pattern here.
-// */
-//
-// // Lookup the attributes for the query on which we will hang the
-// // solution set.
-// final IQueryAttributes attrs = context
-// .getQueryAttributes(namedSetRef.queryId);
-//
-// HTreeHashJoinUtility state = (HTreeHashJoinUtility) attrs
-// .get(namedSetRef);
-//
-// if (state == null) {
-//
-// final JoinTypeEnum joinType = (JoinTypeEnum) op
-// .getRequiredProperty(Annotations.JOIN_TYPE);
-//
-// state = new HTreeHashJoinUtility(
-// context.getMemoryManager(namedSetRef.queryId), op,
-// joinType);
-//
-// if (attrs.putIfAbsent(namedSetRef, state) != null)
-// throw new AssertionError();
-//
-// }
-//
-// this.state = state;
-//
-// }
-//
-// }
-//
-// /**
-// * Evaluate.
-// */
-// public Void call() throws Exception {
-//
-// try {
-//
-// // Buffer all source solutions.
-// acceptSolutions();
-//
-// if(context.isLastInvocation()) {
-//
-// // Checkpoint the solution set.
-// checkpointSolutionSet();
-//
-// // Output the buffered solutions.
-// outputSolutions();
-//
-// }
-//
-// // Done.
-// return null;
-//
-// } finally {
-//
-// context.getSource().close();
-//
-// context.getSink().close();
-//
-// }
-//
-// }
-//
-// /**
-// * Buffer intermediate resources.
-// */
-// private void acceptSolutions() {
-//
-// state.acceptSolutions(context.getSource(), stats);
-//
-// }
-//
-// /**
-// * Checkpoint and save the solution set.
-// */
-// private void checkpointSolutionSet() {
-//
-// state.saveSolutionSet();
-//
-// }
-//
-// /**
-// * Output the buffered solutions.
-// */
-// private void outputSolutions() {
-//
-// // default sink
-// final IBlockingBuffer<IBindingSet[]> sink = context.getSink();
-//
-// final UnsyncLocalOutputBuffer<IBindingSet> unsyncBuffer = new UnsyncLocalOutputBuffer<IBindingSet>(
-// op.getChunkCapacity(), sink);
-//
-// state.outputSolutions(unsyncBuffer);
-//
-// unsyncBuffer.flush();
-//
-// sink.flush();
-//
-// }
-//
-// } // ControllerTask
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java 2013-12-31 19:48:13 UTC (rev 7711)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java 2014-01-03 19:18:16 UTC (rev 7712)
@@ -75,8 +75,6 @@
import com.bigdata.relation.accesspath.IBuffer;
import com.bigdata.rwstore.sector.IMemoryManager;
import com.bigdata.rwstore.sector.MemStore;
-import com.bigdata.striterator.Chunkerator;
-import com.bigdata.striterator.Dechunkerator;
import com.bigdata.util.InnerCause;
import cutthecrap.utils.striterators.Expander;
@@ -178,7 +176,12 @@
// Works Ok.
h = 31 * h + c.hashCode();
-// // Martyn's version. Also works Ok.
+ /*
+ * TODO Martyn's version. Also works Ok. Compare rate of hash
+ * collisions and impact on join performance. Also compare use of
+ * 64-bit hash codes and impact on join performance (there should be
+ * fewer hash collisions).
+ */
// @see http://burtleburtle.net/bob/hash/integer.html
//
// final int hc = c.hashCode();
@@ -210,15 +213,10 @@
*/
private final PipelineOp op;
- /**
- * This basically controls the vectoring of the hash join.
- *
- * TODO parameter from operator annotations. Note that 10k tends to put too
- * much heap pressure on the system if the source chunks happen to be
- * smallish. 1000k or 100 is probably the right value until we improve
- * vectoring of the query engine.
- */
- private final int chunkSize = 1000;//ChunkedWrappedIterator.DEFAULT_CHUNK_SIZE;
+// /**
+// * This basically controls the vectoring of the hash join.
+// */
+// private final int chunkSize = 1000;//ChunkedWrappedIterator.DEFAULT_CHUNK_SIZE;
/**
* Utility class for compactly and efficiently encoding and decoding
@@ -305,7 +303,7 @@
* The maximum #of (left,right) solution joins that will be considered
* before failing the join. This is used IFF there are no join variables.
*
- * FIXME Annotation and query hint for this. Probably on
+ * TODO HINTS: Annotation and query hint for this. Probably on
* {@link HashJoinAnnotations}.
*/
private final long noJoinVarsLimit = HashJoinAnnotations.DEFAULT_NO_JOIN_VARS_LIMIT;
@@ -357,8 +355,8 @@
sb.append(getClass().getSimpleName());
sb.append("{open=" + open);
- sb.append(",joinType="+joinType);
- sb.append(",chunkSize=" + chunkSize);
+ sb.append(",joinType=" + joinType);
+// sb.append(",chunkSize=" + chunkSize);
// sb.append(",optional=" + optional);
// sb.append(",filter=" + filter);
if (askVar != null)
@@ -707,30 +705,8 @@
final IKeyBuilder keyBuilder = htree.getIndexMetadata()
.getKeyBuilder();
- /*
- * Rechunk in order to have a nice fat vector size for ordered
- * inserts.
- *
- * TODO This should probably be eliminated in favor of the existing
- * chunk size. That allows us to control the vectoring directly from
- * the pipeline annotations for the query engine. If 1000 (the
- * current [chunkSize] hard wired into this class) makes a
- * significant difference over 100 (the current default pipeline
- * chunk capacity) then we should simply override the default chunk
- * capacity for the htree hash join operators (i.e., analytic
- * operators always imply a larger default chunk capacity, as could
- * operators running on a cluster). This change should be verified
- * against the GOVTRACK dataset and also by using BSBM with JVM and
- * HTree hash joins and measuring the change in the performance
- * delta when the HTree hash join vector size is changed.
- *
- * @see https://sourceforge.net/apps/trac/bigdata/ticket/483
- * (Eliminate unnecessary dechunking and rechunking)
- */
-
- final ICloseableIterator<IBindingSet[]> it = new Chunkerator<IBindingSet>(
- new Dechunkerator<IBindingSet>(itr), chunkSize,
- IBindingSet.class);
+ // Note: We no longer re-chunk here.
+ final ICloseableIterator<IBindingSet[]> it = itr;
try {
@@ -805,16 +781,13 @@
final IKeyBuilder keyBuilder = htree.getIndexMetadata().getKeyBuilder();
- /*
- * Rechunk in order to have a nice fat vector size for ordered inserts.
- */
- final Iterator<IBindingSet[]> it = new Chunkerator<IBindingSet>(
- new Dechunkerator<IBindingSet>(itr), chunkSize,
- IBindingSet.class);
+ // Note: We no longer rechunk here.
+ final Iterator<IBindingSet[]> it = itr;
final AtomicInteger vectorSize = new AtomicInteger();
while (it.hasNext()) {
+ // Vector a chunk of solutions.
final BS[] a = vector(it.next(), joinVars, selectVars,
true/* ignoreUnboundVariables */, vectorSize);
@@ -943,6 +916,7 @@
return 0;
}
+ @Override
public String toString() {
return getClass().getName() + "{hashCode=" + hashCode + ",bset="
+ bset + "}";
@@ -974,6 +948,7 @@
return 0;
}
+ @Override
public String toString() {
return getClass().getName() + "{hashCode=" + hashCode + ",value="
+ BytesUtil.toString(value) + "}";
@@ -983,21 +958,26 @@
@Override
public void hashJoin(//
- final ICloseableIterator<IBindingSet> leftItr,//
+ final ICloseableIterator<IBindingSet[]> leftItr,//
+ final BOpStats stats,
final IBuffer<IBindingSet> outputBuffer//
) {
- hashJoin2(leftItr, outputBuffer, constraints);
+ hashJoin2(leftItr, stats, outputBuffer, constraints);
}
@Override
public void hashJoin2(//
- final ICloseableIterator<IBindingSet> leftItr,//
+ final ICloseableIterator<IBindingSet[]> leftItr,//
+ final BOpStats stats,//
final IBuffer<IBindingSet> outputBuffer,//
final IConstraint[] constraints//
) {
+ // Note: We no longer rechunk in this method.
+ final Iterator<IBindingSet[]> it = leftItr;
+
try {
final HTree rightSolutions = this.getRightSolutions();
@@ -1012,22 +992,37 @@
final IKeyBuilder keyBuilder = rightSolutions.getIndexMetadata()
.getKeyBuilder();
- final Iterator<IBindingSet[]> it = new Chunkerator<IBindingSet>(
- leftItr, chunkSize, IBindingSet.class);
-
// true iff there are no join variables.
final boolean noJoinVars = joinVars.length == 0;
final AtomicInteger vectorSize = new AtomicInteger();
while (it.hasNext()) {
- final BS[] a = vector(it.next(), joinVars,
- null/* selectVars */,
- false/* ignoreUnboundVariables */, vectorSize);
-
- final int n = vectorSize.get();
+ final BS[] a; // vectored solutions.
+ final int n; // #of valid elements in a[].
+ {
+ // Next chunk of solutions from left.
+ final IBindingSet[] b = it.next();
+ if (stats != null) {
+ stats.chunksIn.increment();
+ stats.unitsIn.add(b.length);
+ }
- nleftConsidered.add(n);
+ // Vector a chunk of solutions, ordering by hashCode.
+ a = vector(b, joinVars, null/* selectVars */,
+ false/* ignoreUnboundVariables */, vectorSize);
+
+ // The size of that vector.
+ n = vectorSize.get();
+
+ nleftConsidered.add(n);
+
+ if (log.isTraceEnabled())
+ log.trace("Vectoring chunk for HTree locality: " + n
+ + " out of " + a.length
+ + " solutions are preserved.");
+
+ }
int fromIndex = 0;
@@ -1056,7 +1051,8 @@
if (log.isTraceEnabled())
log.trace("hashCode=" + hashCode + ": #left="
- + bucketSize + ", firstLeft=" + a[fromIndex]);
+ + bucketSize + ", vectorSize=" + n
+ + ", firstLeft=" + a[fromIndex]);
/*
* Note: all source solutions in [fromIndex:toIndex) have
@@ -1080,7 +1076,8 @@
int nrejected = 0;
{
- final byte[] key = keyBuilder.reset().append(hashCode).getKey();
+ final byte[] key = keyBuilder.reset().append(hashCode)
+ .getKey();
// vi...
[truncated message content] |
|
From: <tho...@us...> - 2013-12-31 19:48:20
|
Revision: 7711
http://bigdata.svn.sourceforge.net/bigdata/?rev=7711&view=rev
Author: thompsonbry
Date: 2013-12-31 19:48:13 +0000 (Tue, 31 Dec 2013)
Log Message:
-----------
Improved reporting in the Explain view.
Override and final annotations.
Refactored ChunkedMaterializationOp and DropOp to expose their VARS annotation through getVars() and to use that method rather than getProperty() when accessing the variables.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/DropOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/bop/rdf/join/ChunkedMaterializationOp.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java 2013-12-31 19:43:51 UTC (rev 7710)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java 2013-12-31 19:48:13 UTC (rev 7711)
@@ -43,8 +43,9 @@
import com.bigdata.bop.BOpUtility;
import com.bigdata.bop.IPredicate;
import com.bigdata.bop.IQueryAttributes;
-import com.bigdata.bop.IVariable;
import com.bigdata.bop.IVariableOrConstant;
+import com.bigdata.bop.PipelineOp;
+import com.bigdata.bop.bset.ConditionalRoutingOp;
import com.bigdata.bop.controller.INamedSolutionSetRef;
import com.bigdata.bop.controller.NamedSetAnnotations;
import com.bigdata.bop.engine.RunState.RunStateEnum;
@@ -57,6 +58,8 @@
import com.bigdata.bop.joinGraph.rto.Path;
import com.bigdata.bop.joinGraph.rto.PathIds;
import com.bigdata.bop.rdf.join.ChunkedMaterializationOp;
+import com.bigdata.bop.solutions.DropOp;
+import com.bigdata.bop.solutions.GroupByOp;
import com.bigdata.bop.solutions.ProjectionOp;
import com.bigdata.counters.render.XHTMLRenderer;
import com.bigdata.rawstore.Bytes;
@@ -401,6 +404,16 @@
} else {
sb.append(bop.getClass().getSimpleName());
sb.append("[" + bopId + "]");
+ final Integer defaultSink = (Integer) bop
+ .getProperty(PipelineOp.Annotations.SINK_REF);
+ final Integer altSink = (Integer) bop
+ .getProperty(PipelineOp.Annotations.ALT_SINK_REF);
+ if (defaultSink != null) {
+ sb.append(", sink=" + defaultSink);
+ }
+ if (altSink != null) {
+ sb.append(", altSink=" + altSink);
+ }
}
sb.append('\t');
if (pred != null) {
@@ -476,10 +489,41 @@
}
}
if (bop instanceof ChunkedMaterializationOp) {
- final IVariable<?>[] vars = (IVariable<?>[]) bop
- .getProperty(ChunkedMaterializationOp.Annotations.VARS);
- sb.append(Arrays.toString(vars));
+ sb.append(cdata("vars="
+ + Arrays.toString(((ChunkedMaterializationOp) bop)
+ .getVars()) + ",materializeInlineIVs="
+ + ((ChunkedMaterializationOp) bop).materializeInlineIVs()));
}
+ if (bop instanceof GroupByOp) {
+ sb.append(cdata(((GroupByOp) bop).getGroupByState().toString()));
+ sb.append(cdata(" "));// whitespace to break the line.
+ sb.append(cdata(((GroupByOp) bop).getGroupByRewrite().toString()));
+ }
+ if (bop instanceof DropOp) {
+ sb.append(cdata(Arrays.toString(((DropOp)bop).getDropVars())));
+ }
+ if (bop instanceof ConditionalRoutingOp) {
+ sb.append(cdata(((ConditionalRoutingOp) bop).getCondition()
+ .toString()));
+ }
+ if (bop instanceof JoinGraph) {
+ final JoinGraph t = ((JoinGraph) bop);
+// final Path p = t.getPath(q);
+// final Map<PathIds, EdgeSample> samples = t
+// .getSamples(q);
+ sb.append(cdata("sampleType=" + t.getSampleType()));
+ sb.append(cdata(", limit=" + t.getLimit()));
+ sb.append(cdata(", nedges=" + t.getNEdges()));
+// if (p != null && samples != null) { // Note: breaks table formatting.
+// // Show the RTO discovered join path.
+// w.write("<pre>");
+// w.write(cdata(JGraph.showPath(p, samples)));
+// w.write("</pre>");
+// }
+ }
+ if (bop instanceof ProjectionOp) {
+ sb.append(cdata(Arrays.toString(((ProjectionOp) bop).getVariables())));
+ }
/*
* Static optimizer metadata.
@@ -1081,10 +1125,28 @@
} else {
w.write(cdata(bop.getClass().getSimpleName()));
w.write(cdata("[" + bopId + "]"));
+ final Integer defaultSink = (Integer) bop
+ .getProperty(PipelineOp.Annotations.SINK_REF);
+ final Integer altSink = (Integer) bop
+ .getProperty(PipelineOp.Annotations.ALT_SINK_REF);
+ if (defaultSink != null) {
+ w.write(cdata(", sink=" + defaultSink));
+ }
+ if (altSink != null) {
+ w.write(cdata(", altSink=" + altSink));
+ }
}
w.write(TDx);
- // operator summary (not shown for the "total" line).
+ /*
+ * Pperator summary (not shown for the "total" line).
+ *
+ * TODO We should have self-reporting of the summary for each operator,
+ * potentially as XHTML. Also, the parser should pass along the SPARQL
+ * snip that corresponds to the operator so we can display it here. We
+ * already handle this for the SERVICE call's inner graph pattern. It
+ * could be handled in general.
+ */
w.write(TD);
if(!summary) {
if (pred != null) {
@@ -1161,10 +1223,28 @@
}
}
if (bop instanceof ChunkedMaterializationOp) {
- final IVariable<?>[] vars = (IVariable<?>[]) bop
- .getProperty(ChunkedMaterializationOp.Annotations.VARS);
- w.write(cdata(Arrays.toString(vars)));
+ w.write(cdata("vars="
+ + Arrays.toString(((ChunkedMaterializationOp) bop)
+ .getVars())
+ + ",materializeInlineIVs="
+ + ((ChunkedMaterializationOp) bop)
+ .materializeInlineIVs()));
}
+ if (bop instanceof GroupByOp) {
+ w.write(cdata(((GroupByOp) bop).getGroupByState().toString()));
+ if (detailedStats) {
+ w.write(cdata(" "));// whitespace to break the line.
+ w.write(cdata(((GroupByOp) bop).getGroupByRewrite()
+ .toString()));
+ }
+ }
+ if (bop instanceof DropOp) {
+ w.write(cdata(Arrays.toString(((DropOp) bop).getDropVars())));
+ }
+ if (bop instanceof ConditionalRoutingOp) {
+ w.write(cdata(((ConditionalRoutingOp) bop).getCondition()
+ .toString()));
+ }
if (bop instanceof JoinGraph) {
final JoinGraph t = ((JoinGraph) bop);
final Path p = t.getPath(q);
@@ -1181,9 +1261,8 @@
}
}
if (bop instanceof ProjectionOp) {
- final IVariable<?>[] vars = (IVariable<?>[]) bop
- .getProperty(ProjectionOp.Annotations.SELECT);
- w.write(cdata(Arrays.toString(vars)));
+ w.write(cdata(Arrays.toString(((ProjectionOp) bop)
+ .getVariables())));
}
}
w.write(TDx); // end summary
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/DropOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/DropOp.java 2013-12-31 19:43:51 UTC (rev 7710)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/DropOp.java 2013-12-31 19:48:13 UTC (rev 7711)
@@ -61,7 +61,8 @@
/**
* An {@link IVariable}[] identifying the variables to be DROPPED in the
- * {@link IBindingSet}s written out by the operator.
+ * {@link IBindingSet}s written out by the operator (required, must be a
+ * non-empty array).
*/
String DROP_VARS = DropOp.class.getName() + ".dropVars";
@@ -78,35 +79,33 @@
* @param args
* @param annotations
*/
- public DropOp(BOp[] args, Map<String, Object> annotations) {
+ public DropOp(final BOp[] args, final Map<String, Object> annotations) {
super(args, annotations);
- final IVariable<?>[] dropVars = (IVariable<?>[]) getRequiredProperty(Annotations.DROP_VARS);
+ final IVariable<?>[] dropVars = getDropVars();
- if (dropVars == null)
- throw new IllegalArgumentException();
-
if (dropVars.length == 0)
throw new IllegalArgumentException();
}
- public DropOp(final BOp[] args, NV... annotations) {
+ public DropOp(final BOp[] args, final NV... annotations) {
this(args, NV.asMap(annotations));
}
-// /**
-// * @see Annotations#DROP
-// */
-// public IVariable<?>[] getVariables() {
-//
-// return (IVariable<?>[]) getRequiredProperty(Annotations.DROP);
-//
-// }
+ /**
+ * @see Annotations#DROP_VARS
+ */
+ public IVariable<?>[] getDropVars() {
+ return (IVariable<?>[]) getRequiredProperty(Annotations.DROP_VARS);
+
+ }
+
+ @Override
public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
return new FutureTask<Void>(new ChunkTask(this, context));
@@ -129,11 +128,11 @@
this.context = context;
- this.vars = (IVariable<?>[]) op
- .getRequiredProperty(Annotations.DROP_VARS);
+ this.vars = op.getDropVars();
}
+ @Override
public Void call() throws Exception {
final BOpStats stats = context.getStats();
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/bop/rdf/join/ChunkedMaterializationOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/bop/rdf/join/ChunkedMaterializationOp.java 2013-12-31 19:43:51 UTC (rev 7710)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/bop/rdf/join/ChunkedMaterializationOp.java 2013-12-31 19:48:13 UTC (rev 7711)
@@ -102,11 +102,12 @@
* @param args
* @param annotations
*/
- public ChunkedMaterializationOp(BOp[] args, Map<String, Object> annotations) {
+ public ChunkedMaterializationOp(final BOp[] args,
+ final Map<String, Object> annotations) {
super(args, annotations);
- final IVariable<?>[] vars = (IVariable<?>[]) getProperty(Annotations.VARS);
+ final IVariable<?>[] vars = getVars();
if (vars != null && vars.length == 0)
throw new IllegalArgumentException();
@@ -120,13 +121,13 @@
/**
* @param op
*/
- public ChunkedMaterializationOp(ChunkedMaterializationOp op) {
+ public ChunkedMaterializationOp(final ChunkedMaterializationOp op) {
super(op);
}
- public ChunkedMaterializationOp(final BOp[] args, NV... annotations) {
+ public ChunkedMaterializationOp(final BOp[] args, final NV... annotations) {
this(args, NV.asMap(annotations));
@@ -154,6 +155,20 @@
}
/**
+ * Return the variables to be materialized.
+ *
+ * @return The variables to be materialized -or- <code>null</code> iff all
+ * variables should be materialized.
+ *
+ * @see Annotations#VARS
+ */
+ public IVariable<?>[] getVars() {
+
+ return (IVariable<?>[]) getProperty(Annotations.VARS);
+
+ }
+
+ /**
* When <code>true</code>, inline {@link IV}s are also materialized.
*
* @see Annotations#MATERIALIZE_INLINE_IVS
@@ -165,6 +180,7 @@
}
+ @Override
public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
return new FutureTask<Void>(new ChunkTask(this, context));
@@ -195,11 +211,8 @@
this.context = context;
- this.vars = (IVariable<?>[]) op.getProperty(Annotations.VARS);
+ this.vars = op.getVars();
- if (vars != null && vars.length == 0)
- throw new IllegalArgumentException();
-
namespace = ((String[]) op.getProperty(Annotations.RELATION_NAME))[0];
timestamp = (Long) op.getProperty(Annotations.TIMESTAMP);
@@ -208,6 +221,7 @@
}
+ @Override
public Void call() throws Exception {
final BOpStats stats = context.getStats();
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-12-31 19:43:57
|
Revision: 7710
http://bigdata.svn.sourceforge.net/bigdata/?rev=7710&view=rev
Author: thompsonbry
Date: 2013-12-31 19:43:51 +0000 (Tue, 31 Dec 2013)
Log Message:
-----------
final, @Override, and rawtypes/unchecked attributes.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/SPARQLConstraint.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/SPARQLConstraint.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/SPARQLConstraint.java 2013-12-31 19:43:09 UTC (rev 7709)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/SPARQLConstraint.java 2013-12-31 19:43:51 UTC (rev 7710)
@@ -68,7 +68,8 @@
* The value expression will be automatically wrapped inside an
* {@link EBVBOp} if it does not itself evaluate to a boolean.
*/
- public SPARQLConstraint(final IValueExpression<? extends IV> x) {
+ @SuppressWarnings("rawtypes")
+ public SPARQLConstraint(final IValueExpression<? extends IV> x) {
this(new BOp[] { wrap(x) }, null/*annocations*/);
@@ -77,38 +78,44 @@
/**
* Required shallow copy constructor.
*/
- public SPARQLConstraint(final BOp[] args,
- final Map<String, Object> anns) {
-
+ public SPARQLConstraint(final BOp[] args, final Map<String, Object> anns) {
+
super(args, anns);
-
+
if (args.length != 1 || args[0] == null)
- throw new IllegalArgumentException();
+ throw new IllegalArgumentException();
}
/**
* Constructor required for {@link com.bigdata.bop.BOpUtility#deepCopy(FilterNode)}.
*/
- public SPARQLConstraint(final SPARQLConstraint op) {
+ public SPARQLConstraint(final SPARQLConstraint<X> op) {
+
super(op);
+
}
+ @SuppressWarnings({ "rawtypes", "unchecked" })
@Override
public IValueExpression<? extends XSDBooleanIV> get(final int i) {
- return (IValueExpression<? extends XSDBooleanIV>) super.get(i);
+
+ return (IValueExpression<? extends XSDBooleanIV>) super.get(i);
+
}
// public IValueExpression<X> getValueExpression() {
// return (IValueExpression<X>) get(0);
// }
+ @Override
public boolean accept(final IBindingSet bs) {
try {
// evaluate the EBV operator
- final XSDBooleanIV iv = get(0).get(bs);
+ @SuppressWarnings("rawtypes")
+ final XSDBooleanIV iv = get(0).get(bs);
return iv.booleanValue();
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-12-31 19:43:16
|
Revision: 7709
http://bigdata.svn.sourceforge.net/bigdata/?rev=7709&view=rev
Author: thompsonbry
Date: 2013-12-31 19:43:09 +0000 (Tue, 31 Dec 2013)
Log Message:
-----------
final and @Override attributes
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/BOpStats.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/BaseJoinStats.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/PipelineJoinStats.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/BOpStats.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/BOpStats.java 2013-12-31 17:11:39 UTC (rev 7708)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/BOpStats.java 2013-12-31 19:43:09 UTC (rev 7709)
@@ -155,6 +155,7 @@
mutationCount.add(o.mutationCount.get());
}
+ @Override
public String toString() {
final StringBuilder sb = new StringBuilder();
sb.append(super.toString());
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/BaseJoinStats.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/BaseJoinStats.java 2013-12-31 17:11:39 UTC (rev 7708)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/BaseJoinStats.java 2013-12-31 19:43:09 UTC (rev 7709)
@@ -84,6 +84,7 @@
*/
public final CAT accessPathUnitsIn = new CAT();
+ @Override
public void add(final BOpStats o) {
super.add(o);
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/PipelineJoinStats.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/PipelineJoinStats.java 2013-12-31 17:11:39 UTC (rev 7708)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/join/PipelineJoinStats.java 2013-12-31 19:43:09 UTC (rev 7709)
@@ -115,6 +115,7 @@
}
+ @Override
public void add(final BOpStats o) {
super.add(o);
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-12-31 17:11:45
|
Revision: 7708
http://bigdata.svn.sourceforge.net/bigdata/?rev=7708&view=rev
Author: thompsonbry
Date: 2013-12-31 17:11:39 +0000 (Tue, 31 Dec 2013)
Log Message:
-----------
Removed TODO related to the RTO.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java 2013-12-31 17:08:36 UTC (rev 7707)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java 2013-12-31 17:11:39 UTC (rev 7708)
@@ -2545,9 +2545,6 @@
*
* Note: This winds up handling materialization steps as well
* (it calls through to Rule2BOpUtility).
- *
- * TODO The RTO will need to assign the ids to joins so it can
- * correlate them with the {@link IJoinNode}s.
*/
final Predicate<?> pred = toPredicate(sp, ctx);
final boolean optional = sp.isOptional();
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-12-31 17:08:42
|
Revision: 7707
http://bigdata.svn.sourceforge.net/bigdata/?rev=7707&view=rev
Author: thompsonbry
Date: 2013-12-31 17:08:36 +0000 (Tue, 31 Dec 2013)
Log Message:
-----------
@Override, final.
Added public methods to access some annotations.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/GroupByOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/GroupByRewriter.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/GroupByState.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/MemoryGroupByOp.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/PipelinedAggregationOp.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/GroupByOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/GroupByOp.java 2013-12-31 17:06:32 UTC (rev 7706)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/GroupByOp.java 2013-12-31 17:08:36 UTC (rev 7707)
@@ -29,7 +29,6 @@
import java.util.Map;
-
import com.bigdata.bop.BOp;
import com.bigdata.bop.PipelineOp;
import com.bigdata.bop.aggregate.IAggregate;
@@ -136,4 +135,22 @@
*/
abstract public boolean isPipelinedAggregationOp();
+ /**
+ * @see Annotations#GROUP_BY_STATE
+ */
+ public IGroupByState getGroupByState() {
+
+ return (IGroupByState) getRequiredProperty(Annotations.GROUP_BY_STATE);
+
+ }
+
+ /**
+ * @see Annotations#GROUP_BY_REWRITE
+ */
+ public IGroupByRewriteState getGroupByRewrite() {
+
+ return (IGroupByRewriteState) getRequiredProperty(Annotations.GROUP_BY_REWRITE);
+
+ }
+
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/GroupByRewriter.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/GroupByRewriter.java 2013-12-31 17:06:32 UTC (rev 7706)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/GroupByRewriter.java 2013-12-31 17:08:36 UTC (rev 7707)
@@ -76,6 +76,7 @@
private final IValueExpression<?>[] select2;
private final IConstraint[] having2;
+ @Override
public String toString() {
final StringBuilder sb = new StringBuilder();
sb.append(getClass().getSimpleName());
@@ -101,6 +102,7 @@
* {@link IAggregate} functions. All {@link IAggregate} functions have been
* lifted out into {@link #aggExpr}.
*/
+ @Override
public IValueExpression<?>[] getSelect2() {
return select2;
}
@@ -112,6 +114,7 @@
* {@link IAggregate} functions. All {@link IAggregate} functions have been
* lifted out into {@link #aggExpr}.
*/
+ @Override
public IConstraint[] getHaving2() {
return having2;
}
@@ -460,6 +463,7 @@
* Return a new anonymous variable (this is overridden by some unit tests in
* order to have predictable variable names).
*/
+ @Override
public IVariable<?> var() {
return Var.var();
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/GroupByState.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/GroupByState.java 2013-12-31 17:06:32 UTC (rev 7706)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/GroupByState.java 2013-12-31 17:08:36 UTC (rev 7707)
@@ -28,6 +28,7 @@
package com.bigdata.bop.solutions;
import java.io.Serializable;
+import java.util.Arrays;
import java.util.Iterator;
import java.util.LinkedHashSet;
import java.util.concurrent.atomic.AtomicBoolean;
@@ -71,26 +72,43 @@
final private boolean nestedAggregates;
final private boolean simpleHaving;
+ @Override
+ public String toString() {
+ final StringBuilder sb = new StringBuilder();
+ sb.append(getClass().getSimpleName());
+ sb.append("{select=" + Arrays.toString(select));
+ sb.append(",groupBy=" + Arrays.toString(groupBy));
+ sb.append(",having=" + Arrays.toString(having));
+ sb.append("}");
+ return sb.toString();
+ }
+
+ @Override
public IValueExpression<?>[] getGroupByClause() {
return groupBy;
}
+ @Override
public LinkedHashSet<IVariable<?>> getGroupByVars() {
return groupByVars;
}
+ @Override
public IValueExpression<?>[] getSelectClause() {
return select;
}
+ @Override
public LinkedHashSet<IVariable<?>> getSelectVars() {
return selectVars;
}
+ @Override
public IConstraint[] getHavingClause() {
return having;
}
+ @Override
public LinkedHashSet<IVariable<?>> getColumnVars() {
return columnVars;
}
@@ -99,18 +117,22 @@
// return distinctColumnVars;
// }
+ @Override
public boolean isAnyDistinct() {
return anyDistinct;
}
+ @Override
public boolean isSelectDependency() {
return selectDependency;
}
+ @Override
public boolean isNestedAggregates() {
return nestedAggregates;
}
+ @Override
public boolean isSimpleHaving() {
return simpleHaving;
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/MemoryGroupByOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/MemoryGroupByOp.java 2013-12-31 17:06:32 UTC (rev 7706)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/MemoryGroupByOp.java 2013-12-31 17:08:36 UTC (rev 7707)
@@ -87,6 +87,7 @@
* Returns <code>false</code>. This is a generalized aggregation operator
* and may be used to evaluate any aggregation request.
*/
+ @Override
final public boolean isPipelinedAggregationOp() {
return false;
@@ -145,6 +146,7 @@
}
+ @Override
public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
return new FutureTask<Void>(new GroupByTask(this, context));
@@ -165,6 +167,7 @@
*/
private final IConstant<?>[] vals;
+ @Override
public String toString() {
return super.toString() + //
"{group=" + Arrays.toString(vals) + //
@@ -234,10 +237,12 @@
this.hash = java.util.Arrays.hashCode(vals);
}
+ @Override
public int hashCode() {
return hash;
}
+ @Override
public boolean equals(final Object o) {
if (this == o)
return true;
@@ -368,6 +373,7 @@
}
+ @Override
public Void call() throws Exception {
final ICloseableIterator<IBindingSet[]> itr = context
@@ -914,12 +920,14 @@
}
+ @Override
public int hashCode() {
return hash;
}
+ @Override
public boolean equals(final Object o) {
if (this == o)
return true;
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/PipelinedAggregationOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/PipelinedAggregationOp.java 2013-12-31 17:06:32 UTC (rev 7706)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/solutions/PipelinedAggregationOp.java 2013-12-31 17:08:36 UTC (rev 7707)
@@ -83,6 +83,7 @@
* MAY NOT be used to evaluate aggregation requests which use DISTINCT or
* which nest {@link IAggregate}s in other {@link IAggregate}s.
*/
+ @Override
public boolean isPipelinedAggregationOp() {
return true;
@@ -173,6 +174,7 @@
}
+ @Override
public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
return new FutureTask<Void>(new ChunkTask(this, context));
@@ -193,6 +195,7 @@
*/
private final IConstant<?>[] vals;
+ @Override
public String toString() {
return super.toString() + //
"{group=" + Arrays.toString(vals) + //
@@ -252,10 +255,12 @@
this.hash = java.util.Arrays.hashCode(vals);
}
+ @Override
public int hashCode() {
return hash;
}
+ @Override
public boolean equals(final Object o) {
if (this == o)
return true;
@@ -622,6 +627,7 @@
}
+ @Override
public Void call() throws Exception {
final ICloseableIterator<IBindingSet[]> itr = context
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-12-31 17:06:39
|
Revision: 7706
http://bigdata.svn.sourceforge.net/bigdata/?rev=7706&view=rev
Author: thompsonbry
Date: 2013-12-31 17:06:32 +0000 (Tue, 31 Dec 2013)
Log Message:
-----------
Override, final, and javadoc edits.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/constraint/Constraint.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/constraint/Constraint.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/constraint/Constraint.java 2013-12-31 17:06:10 UTC (rev 7705)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/constraint/Constraint.java 2013-12-31 17:06:32 UTC (rev 7706)
@@ -34,7 +34,7 @@
import com.bigdata.bop.IValueExpressionConstraint;
/**
- * BOpConstraint that wraps a {@link BooleanValueExpression}.
+ * {@link Constraint} wraps a {@link BooleanValueExpression}.
*/
public class Constraint<X> extends BOpBase implements
IValueExpressionConstraint<X> {
@@ -66,9 +66,8 @@
/**
* Required shallow copy constructor.
*/
- public Constraint(final BOp[] args,
- final Map<String, Object> anns) {
-
+ public Constraint(final BOp[] args, final Map<String, Object> anns) {
+
super(args, anns);
if (args.length != 1 || args[0] == null)
@@ -83,6 +82,7 @@
super(op);
}
+ @Override
@SuppressWarnings("unchecked")
public IValueExpression<X> getValueExpression() {
@@ -90,6 +90,7 @@
}
+ @Override
public boolean accept(final IBindingSet bs) {
// try {
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-12-31 17:06:17
|
Revision: 7705
http://bigdata.svn.sourceforge.net/bigdata/?rev=7705&view=rev
Author: thompsonbry
Date: 2013-12-31 17:06:10 +0000 (Tue, 31 Dec 2013)
Log Message:
-----------
inline documentation correction
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/CoreBaseBOp.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/CoreBaseBOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/CoreBaseBOp.java 2013-12-30 19:52:04 UTC (rev 7704)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/CoreBaseBOp.java 2013-12-31 17:06:10 UTC (rev 7705)
@@ -105,9 +105,9 @@
if (t instanceof IValueExpression<?>
|| t instanceof IValueExpressionNode) {
/*
- * Note: toString() is intercepted for a few bops, mainly those with
- * a pretty simple structure. This delegates to toString() in those
- * cases.
+ * Note: toShortString() is intercepted for a few bops, mainly those
+ * with a pretty simple structure. This delegates to toString() in
+ * those cases.
*/
return t.toString();
} else {
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-12-30 19:52:11
|
Revision: 7704
http://bigdata.svn.sourceforge.net/bigdata/?rev=7704&view=rev
Author: thompsonbry
Date: 2013-12-30 19:52:04 +0000 (Mon, 30 Dec 2013)
Log Message:
-----------
ChunkedRunningQuery - added more information for an assert that is being tripped by the RTO for plans involving an OPTIONAL join group.
QueryLog - added conditional logic for a cause with a null message.
PartitionedJoinGroup - removed some dead code and organized imports.
JoinGraph - added some logic to attach the generated query plan to the IRunningQuery's attributes. This is forward looking to an integration where the RTO runs in parallel with the head of the query and then floods all solutions into a subquery that runs the RTO determined join ordering. We need to get the integration going first with the different join modes (triples versus quads), conditional materialization for filters, etc.
@see #64 (RTO).
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/PartitionedJoinGroup.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java 2013-12-30 19:07:46 UTC (rev 7703)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/ChunkedRunningQuery.java 2013-12-30 19:52:04 UTC (rev 7704)
@@ -1149,7 +1149,7 @@
stats = op.newStats();
// log.warn("bopId=" + bopId + ", stats=" + stats);
}
- assert stats != null;
+ assert stats != null : "No stats: op=" + op;
// // The groupId (if any) for this operator.
// final Integer fromGroupId = (Integer) op
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java 2013-12-30 19:07:46 UTC (rev 7703)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java 2013-12-30 19:52:04 UTC (rev 7704)
@@ -964,8 +964,12 @@
w.write(TDx);
}
w.write(TD);
- if (cause != null)
- w.write(cause.getLocalizedMessage());
+ if (cause != null) {
+ String msg = cause.getLocalizedMessage();
+ if (msg == null)
+ msg = cause.toString();
+ w.write(cdata(msg));
+ }
w.write(TDx);
final Map<Integer, BOp> bopIndex = q.getBOpIndex();
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/PartitionedJoinGroup.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/PartitionedJoinGroup.java 2013-12-30 19:07:46 UTC (rev 7703)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/PartitionedJoinGroup.java 2013-12-30 19:52:04 UTC (rev 7704)
@@ -21,7 +21,6 @@
import com.bigdata.bop.IVariable;
import com.bigdata.bop.NV;
import com.bigdata.bop.PipelineOp;
-import com.bigdata.bop.bset.EndOp;
import com.bigdata.bop.join.PipelineJoin;
import com.bigdata.bop.joinGraph.rto.JoinGraph;
import com.bigdata.bop.solutions.JVMDistinctBindingSetsOp;
@@ -1134,24 +1133,6 @@
);
}
- /*
- * FIXME Why does wrapping with this slice appear to be
- * necessary? (It is causing runtime errors when not wrapped).
- * Is this a bopId collision which is not being detected?
- *
- * @see https://sourceforge.net/apps/trac/bigdata/ticket/227
- *
- * [This should perhaps be moved into the caller.]
- */
-// lastOp = new EndOp(new BOp[] { lastOp }, NV
-// .asMap(new NV[] {
-// new NV(JoinGraph.Annotations.BOP_ID, idFactory.nextId()), //
-// new NV(JoinGraph.Annotations.EVALUATION_CONTEXT,
-// BOpEvaluationContext.CONTROLLER)//
-//// new NV(PipelineOp.Annotations.SHARED_STATE,true),//
-// }) //
-// );
-
return lastOp;
}
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java 2013-12-30 19:07:46 UTC (rev 7703)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/java/com/bigdata/bop/joinGraph/rto/JoinGraph.java 2013-12-30 19:52:04 UTC (rev 7704)
@@ -52,7 +52,6 @@
import com.bigdata.bop.engine.IRunningQuery;
import com.bigdata.bop.engine.QueryEngine;
import com.bigdata.bop.joinGraph.PartitionedJoinGroup;
-import com.bigdata.relation.accesspath.BufferClosedException;
import com.bigdata.util.concurrent.Haltable;
import cutthecrap.utils.striterators.ICloseableIterator;
@@ -81,7 +80,10 @@
private static final long serialVersionUID = 1L;
- /**
+ private static final transient Logger log = Logger
+ .getLogger(JoinGraph.class);
+
+ /**
* Known annotations.
*/
public interface Annotations extends PipelineOp.Annotations {
@@ -157,6 +159,13 @@
*/
String SAMPLES = JoinGraph.class.getName() + ".samples";
+ /**
+ * The physical query plan generated from the RTO determined best join
+ * ordering (output). This is used to specify the query plan to be
+ * executed by a downstream operator.
+ */
+ String QUERY_PLAN = JoinGraph.class.getName() + ".queryPlan";
+
}
/**
@@ -252,6 +261,26 @@
}
/**
+ * Return the query plan to be executed based on the RTO determined join
+ * ordering.
+ *
+ * @see Attributes#QUERY_PLAN
+ */
+ public PipelineOp getQueryPlan(final IRunningQuery q) {
+
+ return (PipelineOp) q.getAttributes().get(
+ getId() + "-" + Attributes.QUERY_PLAN);
+
+ }
+
+ private void setQueryPlan(final IRunningQuery q,
+ final PipelineOp queryPlan) {
+
+ q.getAttributes().put(getId() + "-" + Attributes.QUERY_PLAN, queryPlan);
+
+ }
+
+ /**
* Deep copy constructor.
*
* @param op
@@ -327,14 +356,6 @@
private final BOpContext<IBindingSet> context;
- // private final JGraph g;
-
-// final private int limit;
-//
-// final private int nedges;
-//
-// final private SampleType sampleType;
-
JoinGraphTask(final BOpContext<IBindingSet> context) {
if (context == null)
@@ -342,30 +363,11 @@
this.context = context;
-// // The initial cutoff sampling limit.
-// limit = getLimit();
-//
-// // The initial number of edges (1 step paths) to explore.
-// nedges = getNEdges();
-//
-// sampleType = getSampleType();
-
-// if (limit <= 0)
-// throw new IllegalArgumentException();
- //
-// if (nedges <= 0)
-// throw new IllegalArgumentException();
-
-// g = new JGraph(getVertices(), getConstraints());
-
}
/**
* {@inheritDoc}
*
- *
- * TODO where to handle DISTINCT, ORDER BY, GROUP BY for join graph?
- *
* FIXME When run as sub-query, we need to fix point the upstream
* solutions and then flood them into the join graph. Samples of the
* known bound variables can be pulled from those initial solutions.
@@ -414,13 +416,22 @@
// Factory avoids reuse of bopIds assigned to the predicates.
final BOpIdFactory idFactory = new BOpIdFactory();
- // Generate the query from the join path.
- // FIXME Update this using StaticAnalysis logic.
+ /*
+ * Generate the query from the join path.
+ *
+ * FIXME Update this using StaticAnalysis logic. Also, both this and
+ * the JGraph need to handle triples versus named graph versus
+ * default graph APs. Further, JGraph should handle filters that
+ * require conditional materialization.
+ */
final PipelineOp queryOp = PartitionedJoinGroup.getQuery(idFactory,
false/* distinct */, getSelected(), p.getPredicates(),
getConstraints());
- // Run the query, blocking until it is done.
+ // Set attribute for the join path samples.
+ setQueryPlan(context.getRunningQuery(), queryOp);
+
+ // Run the query, blocking until it is done.
JoinGraph.runSubquery(context, queryOp);
final long elapsed_queryExecution = System.nanoTime() - mark;
@@ -436,7 +447,6 @@
}
} // class JoinGraphTask
- private static final transient Logger log = Logger.getLogger(JGraph.class);
/**
* Execute the selected join path.
@@ -461,10 +471,6 @@
* @todo If there are source binding sets then they need to be applied above
* (when we are sampling) and below (when we evaluate the selected
* join path).
- *
- * FIXME runQuery() is not working correctly. The query is being
- * halted by a {@link BufferClosedException} which appears before it
- * has materialized the necessary results.
*/
static private void runSubquery(
final BOpContext<IBindingSet> parentContext,
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-12-30 19:07:53
|
Revision: 7703
http://bigdata.svn.sourceforge.net/bigdata/?rev=7703&view=rev
Author: thompsonbry
Date: 2013-12-30 19:07:46 +0000 (Mon, 30 Dec 2013)
Log Message:
-----------
Explicitly disable the RTO for QUADS mode query.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java 2013-12-30 16:25:31 UTC (rev 7702)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpRTO.java 2013-12-30 19:07:46 UTC (rev 7703)
@@ -101,6 +101,13 @@
final JoinGroupNode joinGroup, final Set<IVariable<?>> doneSet,
final AST2BOpContext ctx, final AtomicInteger start) {
+ if (ctx.isQuads()) {
+
+ // FIXME The RTO does not handle quads yet.
+ return left;
+
+ }
+
final int arity = joinGroup.arity();
// The predicates for the RTO join group.
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-12-30 16:25:38
|
Revision: 7702
http://bigdata.svn.sourceforge.net/bigdata/?rev=7702&view=rev
Author: thompsonbry
Date: 2013-12-30 16:25:31 +0000 (Mon, 30 Dec 2013)
Log Message:
-----------
Modified ASTRangeCountOptimizer to obtain the range counts in parallel threads for the statement pattern nodes.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTRangeCountOptimizer.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTRangeCountOptimizer.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTRangeCountOptimizer.java 2013-12-30 16:02:18 UTC (rev 7701)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTRangeCountOptimizer.java 2013-12-30 16:25:31 UTC (rev 7702)
@@ -24,8 +24,14 @@
package com.bigdata.rdf.sparql.ast.optimizers;
+import java.util.LinkedList;
import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import org.apache.log4j.Logger;
+
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.IConstant;
import com.bigdata.bop.IVariable;
@@ -38,8 +44,10 @@
import com.bigdata.rdf.sparql.ast.TermNode;
import com.bigdata.rdf.sparql.ast.eval.AST2BOpContext;
import com.bigdata.rdf.sparql.ast.optimizers.ASTStaticJoinOptimizer.Annotations;
+import com.bigdata.rdf.spo.SPORelation;
import com.bigdata.rdf.store.AbstractTripleStore;
import com.bigdata.relation.accesspath.IAccessPath;
+import com.bigdata.util.concurrent.ExecutionExceptions;
/**
* Attach range counts to all statement patterns in the query.
@@ -50,11 +58,13 @@
public class ASTRangeCountOptimizer extends AbstractJoinGroupOptimizer
implements IASTOptimizer {
-// private static final transient Logger log = Logger.getLogger(ASTPropertyPathOptimizer.class);
+ private static final transient Logger log = Logger
+ .getLogger(ASTRangeCountOptimizer.class);
/**
* Optimize the join group.
*/
+ @Override
protected void optimizeJoinGroup(final AST2BOpContext ctx,
final StaticAnalysis sa, final IBindingSet[] bSets,
final JoinGroupNode group) {
@@ -62,40 +72,110 @@
final List<StatementPatternNode> spNodes = group.getStatementPatterns();
if (!spNodes.isEmpty()) {
-
+
// Always attach the range counts.
- attachRangeCounts(ctx, spNodes, getExogenousBindings(bSets));
+ attachRangeCounts(ctx, spNodes, getExogenousBindings(bSets));
}
}
/**
- * Use the SPORelation from the database to grab the appropriate range
- * counts for the {@link StatementPatternNode}s. Only tries to attach them
- * if the annotation {@link Annotations#ESTIMATED_CARDINALITY} is not
- * already attached to the node. This makes it possible to write unit
- * tests without real data.
+ * Use the {@link SPORelation} from the database to grab the appropriate
+ * range counts for the {@link StatementPatternNode}s. Only tries to attach
+ * them if the annotation {@link Annotations#ESTIMATED_CARDINALITY} is not
+ * already attached to the node. This makes it possible to write unit tests
+ * without real data.
*/
private final void attachRangeCounts(final AST2BOpContext ctx,
final List<StatementPatternNode> spNodes,
final IBindingSet exogenousBindings) {
final AbstractTripleStore db = ctx.getAbstractTripleStore();
-
- for (StatementPatternNode sp : spNodes) {
-
- if (sp.getProperty(Annotations.ESTIMATED_CARDINALITY) == null) {
-
- estimateCardinality(sp, db, exogenousBindings);
- }
-
- }
-
+ // Setup tasks to obtain estimated range counts.
+ final List<Callable<Void>> tasks = new LinkedList<Callable<Void>>();
+ for (StatementPatternNode sp : spNodes) {
+
+ if (sp.getProperty(Annotations.ESTIMATED_CARDINALITY) == null) {
+
+ tasks.add(new RangeCountTask(sp, db, exogenousBindings));
+
+ }
+
+ }
+
+ // Obtain range counts in parallel.
+ final List<Future<Void>> futures;
+ try {
+
+ futures = db.getExecutorService().invokeAll(tasks);
+
+ } catch (InterruptedException e) {
+ // propagate interrupt.
+ Thread.currentThread().interrupt();
+ return;
+ }
+
+ // Check futures for errors.
+ final List<Throwable> causes = new LinkedList<Throwable>();
+ for (Future<Void> f : futures) {
+ try {
+ f.get();
+ } catch (InterruptedException e) {
+ log.error(e);
+ causes.add(e);
+ } catch (ExecutionException e) {
+ log.error(e);
+ causes.add(e);
+ }
+ }
+
+ /*
+ * If there were any errors, then throw an exception listing them.
+ */
+ if (!causes.isEmpty()) {
+ // Throw exception back to the leader.
+ if (causes.size() == 1)
+ throw new RuntimeException(causes.get(0));
+ throw new RuntimeException("nerrors=" + causes.size(),
+ new ExecutionExceptions(causes));
+ }
+
}
/**
+ * Task unconditionally obtains the range count for the
+ * {@link StatementPatternNode}.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan
+ * Thompson</a>
+ */
+ private class RangeCountTask implements Callable<Void> {
+
+ private final StatementPatternNode sp;
+ private final AbstractTripleStore db;
+ private final IBindingSet exogenousBindings;
+
+ public RangeCountTask(final StatementPatternNode sp,
+ final AbstractTripleStore db,
+ final IBindingSet exogenousBindings) {
+ this.sp = sp;
+ this.db = db;
+ this.exogenousBindings = exogenousBindings;
+ }
+
+ @Override
+ public Void call() throws Exception {
+
+ estimateCardinality(sp, db, exogenousBindings);
+
+ return null;
+ }
+
+ }
+
+ /**
* For testing purposes we can override this method.
* @param sp
* @param db
@@ -222,6 +302,4 @@
}
-
-
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
4 messages has been excluded from this view by a project administrator.
