Menu
▾
▴
bigdata-commit
|
From: <tho...@us...> - 2010-09-17 23:28:18
|
Revision: 3587
http://bigdata.svn.sourceforge.net/bigdata/?rev=3587&view=rev
Author: thompsonbry
Date: 2010-09-17 23:28:12 +0000 (Fri, 17 Sep 2010)
Log Message:
-----------
Added notes on operator at a time queries, e.g., sort.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-09-17 19:17:52 UTC (rev 3586)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-09-17 23:28:12 UTC (rev 3587)
@@ -414,10 +414,16 @@
* Note: The partitionId will always be -1 in scale-up.
*/
final int partitionId = -1;
-
+
/*
* FIXME Raise this into an annotation that we can tweak from the unit
* tests and then debug the problem.
+ *
+ * FIXME Add an annotation or method to mark operators which must be
+ * evaluated using operator-at-a-time evaluation. SORT is the main
+ * example here (it must be operator at a time of necessity) but other
+ * operators may implemented with operator at a time assumptions. This
+ * might be on PipelineOp and could be trinary {Chunked,Blocked,All}.
*/
final boolean oneMessagePerChunk = false;
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-10-07 19:52:41
|
Revision: 3746
http://bigdata.svn.sourceforge.net/bigdata/?rev=3746&view=rev
Author: thompsonbry
Date: 2010-10-07 19:52:35 +0000 (Thu, 07 Oct 2010)
Log Message:
-----------
Kicking off another CI build in response to a weird error from Hudson.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-10-07 19:05:15 UTC (rev 3745)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-10-07 19:52:35 UTC (rev 3746)
@@ -376,7 +376,6 @@
statsMap = null;
}
- // runStateLock = controller ? new ReentrantLock() : null;
lock = new ReentrantLock();
runState = controller ? new RunState(this) : null;
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-10-07 20:00:22
|
Revision: 3747
http://bigdata.svn.sourceforge.net/bigdata/?rev=3747&view=rev
Author: thompsonbry
Date: 2010-10-07 20:00:16 +0000 (Thu, 07 Oct 2010)
Log Message:
-----------
Reorganized the code in an attempt to get around a weird CI problem.
compile:
[javac] /var/lib/hudson/jobs/Bigdata Quads Query Branch/workspace/QUADS_QUERY_BRANCH/build.xml:76: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 1398 source files to /var/lib/hudson/jobs/Bigdata Quads Query Branch/workspace/QUADS_QUERY_BRANCH/ant-build/classes
[javac] javac 1.6.0_17
[javac] /var/lib/hudson/jobs/Bigdata Quads Query Branch/workspace/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java:379: variable lock might already have been assigned
[javac] lock = new ReentrantLock();
[javac] ^
[javac] /var/lib/hudson/jobs/Bigdata Quads Query Branch/workspace/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java:381: variable runState might already have been assigned
[javac] runState = controller ? new RunState(this) : null;
[javac] ^
[javac] /var/lib/hudson/jobs/Bigdata Quads Query Branch/workspace/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java:392: variable queryBuffer might already have been assigned
[javac] queryBuffer = query.newBuffer(queryStats);
[javac] ^
[javac] /var/lib/hudson/jobs/Bigdata Quads Query Branch/workspace/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java:397: variable queryBuffer might already have been assigned
[javac] queryBuffer = null;
[javac] ^
[javac] /var/lib/hudson/jobs/Bigdata Quads Query Branch/workspace/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java:404: variable queryBuffer might already have been assigned
[javac] queryBuffer = null;
[javac] ^
[javac] /var/lib/hudson/jobs/Bigdata Quads Query Branch/workspace/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java:410: variable queryIterator might already have been assigned
[javac] queryIterator = null;
[javac] ^
[javac] /var/lib/hudson/jobs/Bigdata Quads Query Branch/workspace/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java:414: variable queryIterator might already have been assigned
[javac] queryIterator = new QueryResultIterator<IBindingSet[]>(this,
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 7 errors
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-10-07 19:52:35 UTC (rev 3746)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-10-07 20:00:16 UTC (rev 3747)
@@ -357,63 +357,64 @@
this.query = query;
- bopIndex = BOpUtility.getIndex(query);
+ this.lock = new ReentrantLock();
+ this.runState = controller ? new RunState(this) : null;
+
+ this.bopIndex = BOpUtility.getIndex(query);
+
/*
* Setup the BOpStats object for each pipeline operator in the query.
*/
if (controller) {
+
statsMap = new ConcurrentHashMap<Integer, BOpStats>();
+
for (Map.Entry<Integer, BOp> e : bopIndex.entrySet()) {
+
final int bopId = e.getKey();
+
final BOp tmp = e.getValue();
+
if ((tmp instanceof PipelineOp)) {
+
final PipelineOp bop = (PipelineOp) tmp;
+
statsMap.put(bopId, bop.newStats());
+
}
}
- } else {
- statsMap = null;
- }
- lock = new ReentrantLock();
+// final BOpStats queryStats = query.newStats();
+//
+// statsMap.put((Integer) query
+// .getRequiredProperty(BOp.Annotations.BOP_ID), queryStats);
- runState = controller ? new RunState(this) : null;
+ if (!query.isMutation()) {
- if (controller) {
+ final BOpStats queryStats = statsMap.get(query.getId());
- final BOpStats queryStats = query.newStats();
+ queryBuffer = query.newBuffer(queryStats);
- statsMap.put((Integer) query
- .getRequiredProperty(BOp.Annotations.BOP_ID), queryStats);
+ queryIterator = new QueryResultIterator<IBindingSet[]>(this,
+ queryBuffer.iterator());
- if (!query.isMutation()) {
-
- queryBuffer = query.newBuffer(queryStats);
-
} else {
// Note: Not used for mutation queries.
queryBuffer = null;
+ queryIterator = null;
}
} else {
+ statsMap = null;
+
// Note: only exists on the query controller.
queryBuffer = null;
-
- }
-
- if (queryBuffer == null) {
-
queryIterator = null;
-
- } else {
-
- queryIterator = new QueryResultIterator<IBindingSet[]>(this,
- queryBuffer.iterator());
-
+
}
// System.err
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-10-07 20:04:28
|
Revision: 3748
http://bigdata.svn.sourceforge.net/bigdata/?rev=3748&view=rev
Author: thompsonbry
Date: 2010-10-07 20:04:21 +0000 (Thu, 07 Oct 2010)
Log Message:
-----------
Another code reorganization.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-10-07 20:00:16 UTC (rev 3747)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-10-07 20:04:21 UTC (rev 3748)
@@ -166,7 +166,7 @@
*
* @see RunState
*/
- private final ReentrantLock lock;
+ private final ReentrantLock lock = new ReentrantLock();
/**
* The run state of this query and <code>null</code> unless this is the
@@ -357,10 +357,6 @@
this.query = query;
- this.lock = new ReentrantLock();
-
- this.runState = controller ? new RunState(this) : null;
-
this.bopIndex = BOpUtility.getIndex(query);
/*
@@ -368,6 +364,8 @@
*/
if (controller) {
+ runState = new RunState(this);
+
statsMap = new ConcurrentHashMap<Integer, BOpStats>();
for (Map.Entry<Integer, BOp> e : bopIndex.entrySet()) {
@@ -409,20 +407,13 @@
} else {
- statsMap = null;
+ runState = null; // Note: only on the query controller.
+ statsMap = null; // Note: only on the query controller.
+ queryBuffer = null; // Note: only on the query controller.
+ queryIterator = null; // Note: only when queryBuffer is defined.
- // Note: only exists on the query controller.
- queryBuffer = null;
- queryIterator = null;
-
}
- // System.err
- // .println("new RunningQuery:: queryId=" + queryId
- // + ", isController=" + controller + ", queryController="
- // + clientProxy + ", queryEngine="
- // + queryEngine.getServiceUUID());
-
}
/**
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-10-07 20:23:11
|
Revision: 3749
http://bigdata.svn.sourceforge.net/bigdata/?rev=3749&view=rev
Author: thompsonbry
Date: 2010-10-07 20:23:05 +0000 (Thu, 07 Oct 2010)
Log Message:
-----------
Trying another code change.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-10-07 20:04:21 UTC (rev 3748)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-10-07 20:23:05 UTC (rev 3749)
@@ -366,28 +366,8 @@
runState = new RunState(this);
- statsMap = new ConcurrentHashMap<Integer, BOpStats>();
-
- for (Map.Entry<Integer, BOp> e : bopIndex.entrySet()) {
-
- final int bopId = e.getKey();
-
- final BOp tmp = e.getValue();
-
- if ((tmp instanceof PipelineOp)) {
-
- final PipelineOp bop = (PipelineOp) tmp;
-
- statsMap.put(bopId, bop.newStats());
-
- }
- }
+ statsMap = createStatsMap(bopIndex);
-// final BOpStats queryStats = query.newStats();
-//
-// statsMap.put((Integer) query
-// .getRequiredProperty(BOp.Annotations.BOP_ID), queryStats);
-
if (!query.isMutation()) {
final BOpStats queryStats = statsMap.get(query.getId());
@@ -417,6 +397,41 @@
}
/**
+ * Pre-populate a map with {@link BOpStats} objects for a query.
+ *
+ * @param bopIndex
+ * A map of the operators in the query which have assigned
+ * bopIds.
+ *
+ * @return A new map with an entry for each operator with a bopId which
+ * associates that operator with its {@link BOpStats} object.
+ */
+ static private ConcurrentHashMap<Integer, BOpStats> createStatsMap(
+ final Map<Integer, BOp> bopIndex) {
+
+ ConcurrentHashMap<Integer, BOpStats> statsMap = new ConcurrentHashMap<Integer, BOpStats>();
+
+ for (Map.Entry<Integer, BOp> e : bopIndex.entrySet()) {
+
+ final int bopId = e.getKey();
+
+ final BOp tmp = e.getValue();
+
+ if ((tmp instanceof PipelineOp)) {
+
+ final PipelineOp bop = (PipelineOp) tmp;
+
+ statsMap.put(bopId, bop.newStats());
+
+ }
+
+ }
+
+ return statsMap;
+
+ }
+
+ /**
* Take a chunk generated by some pass over an operator and make it
* available to the target operator. How this is done depends on whether the
* query is running against a standalone database or the scale-out database.
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-10-22 20:08:54
|
Revision: 3841
http://bigdata.svn.sourceforge.net/bigdata/?rev=3841&view=rev
Author: thompsonbry
Date: 2010-10-22 20:08:48 +0000 (Fri, 22 Oct 2010)
Log Message:
-----------
Replaced use of LinkedHashMap with ConcurrentHashMap to avoid problems with concurrent modification arising from an iterator over a set whose contents are being changed from within a subroutine in the same thread (versus a second thread).
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-10-22 19:58:41 UTC (rev 3840)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-10-22 20:08:48 UTC (rev 3841)
@@ -452,7 +452,7 @@
this.operatorFutures = new ConcurrentHashMap<BSBundle, ChunkFutureTask>();
- this.operatorQueues = new LinkedHashMap<BSBundle, BlockingQueue<IChunkMessage<IBindingSet>>>();
+ this.operatorQueues = new ConcurrentHashMap<BSBundle, BlockingQueue<IChunkMessage<IBindingSet>>>();
/*
* Setup the BOpStats object for each pipeline operator in the query.
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X