bigdata-commit

[Bigdata-commit] SF.net SVN: bigdata:[8161] branches/BIGDATA_RELEASE_1_3_0/bigdata-gas/src/ java/com/bigdata/rdf/graph/impl/GASState.java

[<tho...@us...>]

Revision: 8161
          http://sourceforge.net/p/bigdata/code/8161
Author:   thompsonbry
Date:     2014-05-01 17:08:29 +0000 (Thu, 01 May 2014)
Log Message:
-----------
Elevated priority of the parallel IReducer evaluation issue for the GAS code. This is the main bottleneck in our testing performance tests of the GAS platform.

Modified Paths:
--------------
    branches/BIGDATA_RELEASE_1_3_0/bigdata-gas/src/java/com/bigdata/rdf/graph/impl/GASState.java

Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-gas/src/java/com/bigdata/rdf/graph/impl/GASState.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-gas/src/java/com/bigdata/rdf/graph/impl/GASState.java	2014-05-01 16:21:25 UTC (rev 8160)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-gas/src/java/com/bigdata/rdf/graph/impl/GASState.java	2014-05-01 17:08:29 UTC (rev 8161)
@@ -344,7 +344,7 @@
     /**
      * {@inheritDoc}
      * 
-     * TODO REDUCE : parallelize with nthreads. The reduce operations are often
+     * FIXME REDUCE : parallelize with nthreads. The reduce operations are often
      * lightweight, so maybe a fork/join pool would work better?
      * <p>
      * Note: We can not do a parallel reduction right now because the backing

This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[Bigdata-commit] SF.net SVN: bigdata:[8162] branches/BIGDATA_RELEASE_1_3_0/bigdata-gas/src/ java/com/bigdata/rdf/graph/impl/GASState.java

[<tho...@us...>]

Revision: 8162
          http://sourceforge.net/p/bigdata/code/8162
Author:   thompsonbry
Date:     2014-05-01 17:09:21 +0000 (Thu, 01 May 2014)
Log Message:
-----------
Elevated priority of the parallel IReducer evaluation issue for the GAS code. This is the main bottleneck in our testing performance tests of the GAS platform.

Modified Paths:
--------------
    branches/BIGDATA_RELEASE_1_3_0/bigdata-gas/src/java/com/bigdata/rdf/graph/impl/GASState.java

Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-gas/src/java/com/bigdata/rdf/graph/impl/GASState.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-gas/src/java/com/bigdata/rdf/graph/impl/GASState.java	2014-05-01 17:08:29 UTC (rev 8161)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-gas/src/java/com/bigdata/rdf/graph/impl/GASState.java	2014-05-01 17:09:21 UTC (rev 8162)
@@ -356,6 +356,16 @@
      * solution would be to drop them onto stripped lists at the same time that
      * they are first inserted into the CHM. I could then read over those
      * striped lists in parallel during the reduction.
+     * <p>
+     * The IReducer should run with parallel threads. This is a huge serial
+     * bottleneck right now. Fixing this will require a data structure with a
+     * parallel iterator, not the CHM. See
+     * http://stackoverflow.com/questions/20164690
+     * /using-scalas-parhashmap-in-javas-project-instead-of-concurrenthashmap
+     * <p>
+     * It seems like the short term solution would be to drop them onto striped
+     * lists at the same time that they are first inserted into the CHM. I could
+     * then read over those striped lists in parallel during the reduction.
      */
     @Override
     public <T> T reduce(final IReducer<VS, ES, ST, T> op) {

This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.