Thread: [Bigdata-commit] SF.net SVN: bigdata:[7209] branches/READ_CACHE (Page 4)

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Revision: 7209
          http://bigdata.svn.sourceforge.net/bigdata/?rev=7209&view=rev
Author:   thompsonbry
Date:     2013-07-01 16:27:45 +0000 (Mon, 01 Jul 2013)
Log Message:
-----------
Changes to the GATHER/PREPARE protocol in order to close some gaps
introduced by the recent refactoring to handle a service that joins
after the GATHER but before the PREPARE.
{{{
- Modified the GatherTask to return the consensusReleaseTime. This
  makes it possible to further validate things in Prepare2Phase.

- Modified the PREPARE message to include (a) whether the service
  participated in the GATHER; and (b) the consensusReleaseTime from
  the leader. This change is NOT backwards compatible. Migration
  requires that commits are suspended on the HA replication cluster
  during a rolling upgrade.  The decision to not provide backwards
  compatibility was made on the basis that we do not yet have a GA
  release with support for the HA replication cluster.

- Modified commitNow() to collect and pass along the set of joined
  services in the GATHER as well as in the PREPARE.

- Modified QuorumCommitImpl to pass along whether or not a service was
  joined in the GATHER as well as in the PREPARE and the consensus
  release time for the GATHER.

- Modified Prepare2Phase to check additional conditions based on
  whether the service participated in the GATHER and to verify the
  release time on the followers (must be consisent with the GATHER
  even if the follower joined after the GATHER).

- Modified HAJournal.config to use the FEDNAME environment variable.

- Updated comments in HAJournal.env.

- Isolated the FEDNAME as a static constant in the HA CI test suite.  

  ---- test results ----

- 2 unexpected failures. both are stochastic.

- testQuorumBreaksABC_failLeader: Failed once. Failed twice (uncurable
  service joined problem in seek consensus on C). 3rd time passed. 4th
  time passed.

java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.TimeoutException
	at com.bigdata.io.TestCase3.assertCondition(TestCase3.java:233)
	at com.bigdata.journal.jini.ha.AbstractHA3JournalServerTestCase.awaitNextQuorumMeet(AbstractHA3JournalServerTestCase.java:2019)
	at com.bigdata.journal.jini.ha.TestHA3JournalServer.testQuorumBreaksABC_failLeader(TestHA3JournalServer.java:1329)
Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException
	at com.bigdata.journal.jini.ha.AbstractHA3JournalServerTestCase$9.run(AbstractHA3JournalServerTestCase.java:2031)
	at com.bigdata.io.TestCase3.assertCondition(TestCase3.java:223)
	... 22 more
Caused by: java.util.concurrent.TimeoutException
	at com.bigdata.quorum.AbstractQuorum.awaitQuorum(AbstractQuorum.java:1159)
	at com.bigdata.journal.jini.ha.AbstractHA3JournalServerTestCase$9.run(AbstractHA3JournalServerTestCase.java:2022)
	... 23 more

- testABCMultiLoadFollowerReadsLargeLoad: Failed once (trace
  below). 2nd time passed.

java.util.concurrent.ExecutionException: com.bigdata.quorum.QuorumException: Quorum not met on token: expected 0, actual=1
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
	at java.util.concurrent.FutureTask.get(FutureTask.java:111)
	at com.bigdata.journal.jini.ha.TestHA3JournalServer.doABCMultiLoadFollowerReads2(TestHA3JournalServer.java:2015)
	at com.bigdata.journal.jini.ha.TestHA3JournalServer.testABCMultiLoadFollowerReadsLargeLoad(TestHA3JournalServer.java:1901)
Caused by: com.bigdata.quorum.QuorumException: Quorum not met on token: expected 0, actual=1
	at com.bigdata.quorum.AbstractQuorum.assertQuorum(AbstractQuorum.java:1065)
	at com.bigdata.quorum.AbstractQuorumClient.getLeader(AbstractQuorumClient.java:106)
	at com.bigdata.journal.jini.ha.AbstractHA3JournalServerTestCase$LargeLoadTask.call(AbstractHA3JournalServerTestCase.java:2312)
	at com.bigdata.journal.jini.ha.AbstractHA3JournalServerTestCase$LargeLoadTask.call(AbstractHA3JournalServerTestCase.java:1)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)
}}}

@see https://sourceforge.net/apps/trac/bigdata/ticket/530 (Journal HA)

Modified Paths:
--------------
    branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/PrepareRequest.java
    branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/QuorumCommitImpl.java
    branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/msg/HA2PhasePrepareMessage.java
    branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/msg/IHA2PhasePrepareMessage.java
    branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/AbstractJournal.java
    branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/Journal.java
    branches/READ_CACHE/bigdata/src/java/com/bigdata/service/AbstractHATransactionService.java
    branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/AbstractHA3JournalServerTestCase.java
    branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/AbstractHAJournalServerTestCase.java
    branches/READ_CACHE/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataSPARQLUpdateTest2.java
    branches/READ_CACHE/src/resources/HAJournal/HAJournal.config
    branches/READ_CACHE/src/resources/HAJournal/HAJournal.env

Added Paths:
-----------
    branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/IJoinedAndNonJoinedServices.java
    branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/JoinedAndNonJoinedServices.java

Added: branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/IJoinedAndNonJoinedServices.java
===================================================================

--- branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/IJoinedAndNonJoinedServices.java	                        (rev 0)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/IJoinedAndNonJoinedServices.java	2013-07-01 16:27:45 UTC (rev 7209)
@@ -0,0 +1,44 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010.  All rights reserved.
+
+Contact:
+     SYSTAP, LLC
+     4501 Tower Road
+     Greensboro, NC 27410
+     lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+*/
+package com.bigdata.ha;
+
+import java.util.Set;
+import java.util.UUID;
+
+/**
+ * Interface providing an atomic snapshot of the services that are joined with a
+ * met quorum (and the services that are not joined with a met quorum) as of
+ * some point in the GATHER or PREPARE protocol.
+ * 
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ */
+public interface IJoinedAndNonJoinedServices {
+
+    /** The services joined with the met quorum, in their join order. */
+    public UUID[] getJoinedServiceIds();
+    
+    /** The services in the write pipeline (in any order). */
+    public Set<UUID> getNonJoinedPipelineServiceIds();
+
+}

Added: branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/JoinedAndNonJoinedServices.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/JoinedAndNonJoinedServices.java	                        (rev 0)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/JoinedAndNonJoinedServices.java	2013-07-01 16:27:45 UTC (rev 7209)
@@ -0,0 +1,110 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010.  All rights reserved.
+
+Contact:
+     SYSTAP, LLC
+     4501 Tower Road
+     Greensboro, NC 27410
+     lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+*/
+package com.bigdata.ha;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.LinkedHashSet;
+import java.util.Set;
+import java.util.UUID;
+
+import com.bigdata.quorum.Quorum;
+
+/**
+ * Helper class finds all joined and non-joined services for the quorum
+ * client.
+ * 
+ * @author <a href="mailto:tho...@us...">Bryan
+ *         Thompson</a>
+ */
+public class JoinedAndNonJoinedServices implements Serializable,
+        IJoinedAndNonJoinedServices {
+
+    /**
+     * 
+     */
+    private static final long serialVersionUID = 1L;
+
+    // The services joined with the met quorum, in their join order.
+    private final UUID[] joinedServiceIds;
+    
+    // The services in the write pipeline (in any order).
+    private final Set<UUID> nonJoinedPipelineServiceIds;
+
+    public JoinedAndNonJoinedServices(
+            final Quorum<HAGlue, QuorumService<HAGlue>> quorum) {
+
+        // The services joined with the met quorum, in their join order.
+        joinedServiceIds = quorum.getJoined();
+
+        // The UUID for this service.
+        final UUID serviceId = quorum.getClient().getServiceId();
+
+        if (joinedServiceIds.length == 0
+                || !joinedServiceIds[0].equals(serviceId)) {
+
+            /*
+             * Sanity check. Verify that the first service in the join order
+             * is *this* service. This is a precondition for the service to
+             * be the leader.
+             */
+
+            throw new RuntimeException("Not leader: serviceId=" + serviceId
+                    + ", joinedServiceIds="
+                    + Arrays.toString(joinedServiceIds));
+
+        }
+
+        // The services in the write pipeline (in any order).
+        nonJoinedPipelineServiceIds = new LinkedHashSet<UUID>(
+                Arrays.asList(quorum.getPipeline()));
+
+        // Remove all services that are joined from this collection.
+        for (UUID joinedServiceId : joinedServiceIds) {
+
+            nonJoinedPipelineServiceIds.remove(joinedServiceId);
+
+        }
+
+    }
+
+    @Override
+    public UUID[] getJoinedServiceIds() {
+        return joinedServiceIds;
+    }
+
+    @Override
+    public Set<UUID> getNonJoinedPipelineServiceIds() {
+        return nonJoinedPipelineServiceIds;
+    }
+
+    @Override
+    public String toString() {
+        return super.toString() + "{#joined=" + joinedServiceIds.length
+                + ", #nonJoined=" + nonJoinedPipelineServiceIds.size()
+                + ", joinedServices=" + Arrays.toString(joinedServiceIds)
+                + ", nonJoined=" + nonJoinedPipelineServiceIds + "}";
+    }
+
+}

Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/PrepareRequest.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/PrepareRequest.java	2013-07-01 16:18:22 UTC (rev 7208)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/PrepareRequest.java	2013-07-01 16:27:45 UTC (rev 7209)
@@ -26,11 +26,9 @@
  */
 package com.bigdata.ha;
 
-import java.util.Arrays;
-import java.util.Set;
-import java.util.UUID;
 import java.util.concurrent.TimeUnit;
 
+import com.bigdata.ha.msg.IHANotifyReleaseTimeResponse;
 import com.bigdata.journal.IRootBlockView;
 
 /**
@@ -40,38 +38,79 @@
  */
 public class PrepareRequest {
 
-    private final UUID[] joinedServiceIds;
-    private final Set<UUID> nonJoinedPipelineServiceIds;
+    /** The consensus release time from the GATHER. */
+    private final IHANotifyReleaseTimeResponse consensusReleaseTime;
+    
+    /**
+     * The services joined and non-joined with the met quorum as of the atomic
+     * decision point for the GATHER.
+     */
+    private final IJoinedAndNonJoinedServices gatherJoinedAndNonJoinedServices;
+    /**
+     * The services joined and non-joined with the met quorum as of the atomic
+     * decision point for the PREPARE.
+     */
+    private final IJoinedAndNonJoinedServices prepareJoinedAndNonJoinedServices;
+//    private final UUID[] joinedServiceIds;
+//    private final Set<UUID> nonJoinedPipelineServiceIds;
     private final IRootBlockView rootBlock;
     private final long timeout;
     private final TimeUnit unit;
 
-    public UUID[] getJoinedServiceIds() {
-        return joinedServiceIds;
+    /** The consensus release time from the GATHER. */
+    public IHANotifyReleaseTimeResponse getConsensusReleaseTime() {
+    
+        return consensusReleaseTime;
+        
     }
+    
+    /**
+     * The services joined and non-joined with the met quorum as of the atomic
+     * decision point for the GATHER.
+     */
+    public IJoinedAndNonJoinedServices getGatherJoinedAndNonJoinedServices() {
 
-    public Set<UUID> getNonJoinedPipelineServiceIds() {
-        return nonJoinedPipelineServiceIds;
+        return gatherJoinedAndNonJoinedServices;
+        
     }
 
+    /**
+     * The services joined and non-joined with the met quorum as of the atomic
+     * decision point for the PREPARE.
+     */
+    public IJoinedAndNonJoinedServices getPrepareAndNonJoinedServices() {
+        
+        return prepareJoinedAndNonJoinedServices;
+        
+    }
+
     public IRootBlockView getRootBlock() {
+
         return rootBlock;
+        
     }
 
     public long getTimeout() {
+
         return timeout;
+        
     }
 
     public TimeUnit getUnit() {
+
         return unit;
+        
     }
 
     /**
-     * 
-     * @param joinedServiceIds
-     *            The services joined with the met quorum, in their join order.
-     * @param nonJoinedPipelineServiceIds
-     *            The non-joined services in the write pipeline (in any order).
+     * @param consensusReleaseTime
+     *            The consensus release time from the GATHER.
+     * @param gatherJoinedAndNonJoinedServices
+     *            The services joined and non-joined with the met quorum as of
+     *            the atomic decision point for the GATHER.
+     * @param prepareJoinedAndNonJoinedServices
+     *            The services joined and non-joined with the met quorum as of
+     *            the atomic decision point for the PREPARE.
      * @param isRootBlock0
      *            if this is rootBlock0.
      * @param rootBlock
@@ -82,19 +121,34 @@
      *            The unit for the timeout.
      */
     public PrepareRequest(
-            final UUID[] joinedServiceIds, //
-            final Set<UUID> nonJoinedPipelineServiceIds,//
+            final IHANotifyReleaseTimeResponse consensusReleaseTime,
+            final IJoinedAndNonJoinedServices gatherJoinedAndNonJoinedServices,
+            final IJoinedAndNonJoinedServices prepareJoinedAndNonJoinedServices,
+//            final UUID[] joinedServiceIds, //
+//            final Set<UUID> nonJoinedPipelineServiceIds,//
             final IRootBlockView rootBlock, final long timeout,
             final TimeUnit unit) {
         
+        if (consensusReleaseTime == null)
+            throw new IllegalArgumentException();
+        
+        if (gatherJoinedAndNonJoinedServices == null)
+            throw new IllegalArgumentException();
+
+        if (prepareJoinedAndNonJoinedServices == null)
+            throw new IllegalArgumentException();
+        
         if (rootBlock == null)
             throw new IllegalArgumentException();
 
         if (unit == null)
             throw new IllegalArgumentException();
 
-        this.joinedServiceIds = joinedServiceIds;
-        this.nonJoinedPipelineServiceIds = nonJoinedPipelineServiceIds;
+        this.consensusReleaseTime = consensusReleaseTime;
+        this.gatherJoinedAndNonJoinedServices = gatherJoinedAndNonJoinedServices;
+        this.prepareJoinedAndNonJoinedServices = prepareJoinedAndNonJoinedServices;
+//        this.joinedServiceIds = joinedServiceIds;
+//        this.nonJoinedPipelineServiceIds = nonJoinedPipelineServiceIds;
         this.rootBlock = rootBlock;
         this.timeout = timeout;
         this.unit = unit;
@@ -102,13 +156,21 @@
 
     @Override
     public String toString() {
-        return super.toString() + "{isRootBlock0=" + rootBlock.isRootBlock0()
-                + ", rootBlock=" + rootBlock + ", #joined="
-                + joinedServiceIds.length + ", #nonJoined="
-                + nonJoinedPipelineServiceIds.size() + ", joinedServices="
-                + Arrays.toString(joinedServiceIds) + ", nonJoined="
-                + nonJoinedPipelineServiceIds + ", timeout=" + timeout
-                + ", unit=" + unit + "}";
+        return super.toString()
+                + "{"//
+                + "isRootBlock0="
+                + rootBlock.isRootBlock0()//
+                + ", rootBlock="
+                + rootBlock//
+                + ", consensusReleaseTime="
+                + consensusReleaseTime//
+                + ", gatherJoinedAndNonJoinedServices="
+                + gatherJoinedAndNonJoinedServices//
+                + ", prepareJoinedAndNonJoinedServices="
+                + prepareJoinedAndNonJoinedServices //
+                + ", timeout=" + timeout//
+                + ", unit=" + unit //
+                + "}";
     }
 
 }

Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/QuorumCommitImpl.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/QuorumCommitImpl.java	2013-07-01 16:18:22 UTC (rev 7208)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/QuorumCommitImpl.java	2013-07-01 16:27:45 UTC (rev 7209)
@@ -147,7 +147,8 @@
 
         final IRootBlockView rootBlock = req.getRootBlock();
 
-        final UUID[] joinedServiceIds = req.getJoinedServiceIds();
+        final UUID[] joinedServiceIds = req.getPrepareAndNonJoinedServices()
+                .getJoinedServiceIds();
         
 //        final Set<UUID> nonJoinedPipelineServiceIds = req
 //                .getNonJoinedPipelineServiceIds();
@@ -205,10 +206,6 @@
 
             {
 
-                // The message used for the services that are joined.
-                final IHA2PhasePrepareMessage msgForJoinedService = new HA2PhasePrepareMessage(
-                        true/* isJoinedService */, rootBlock, timeout, unit);
-
                 // First, message the joined services (met with the quorum).
                 int i = 1;
                 {
@@ -218,6 +215,31 @@
                         final UUID serviceId = joinedServiceIds[i];
 
                         /*
+                         * Figure out if this service participated in the
+                         * GATHER.
+                         */
+                        final boolean isGatherService;
+                        {
+                            boolean found = false;
+                            for (UUID x : req
+                                    .getGatherJoinedAndNonJoinedServices()
+                                    .getJoinedServiceIds()) {
+                                if (serviceId.equals(x)) {
+                                    found = true;
+                                    break;
+                                }
+                            }
+                            isGatherService = found;
+                        }
+
+                        // The message used for the services that are joined.
+                        final IHA2PhasePrepareMessage msgForJoinedService = new HA2PhasePrepareMessage(
+                                req.getConsensusReleaseTime(),//
+                                isGatherService,//
+                                true, // isJoinedService
+                                rootBlock, timeout, unit);
+
+                        /*
                          * Runnable which will execute this message on the
                          * remote service.
                          * 
@@ -292,6 +314,13 @@
 
                     final S leader = member.getService();
                     
+                    // The message used for the leader.
+                    final IHA2PhasePrepareMessage msgForJoinedService = new HA2PhasePrepareMessage(
+                            req.getConsensusReleaseTime(),//
+                            true, // isGatherService (always true for leader)
+                            true, // isJoinedService (always true for leader)
+                            rootBlock, timeout, unit);
+
                     final Future<Boolean> f = leader
                             .prepare2Phase(msgForJoinedService);
                     
@@ -433,7 +462,8 @@
         
         final PrepareRequest preq = req.getPrepareRequest();
 
-        final UUID[] joinedServiceIds = preq.getJoinedServiceIds();
+        final UUID[] joinedServiceIds = preq.getPrepareAndNonJoinedServices()
+                .getJoinedServiceIds();
 
 //        final Set<UUID> nonJoinedPipelineServiceIds = preq
 //                .getNonJoinedPipelineServiceIds();

Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/msg/HA2PhasePrepareMessage.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/msg/HA2PhasePrepareMessage.java	2013-07-01 16:18:22 UTC (rev 7208)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/msg/HA2PhasePrepareMessage.java	2013-07-01 16:27:45 UTC (rev 7209)
@@ -34,18 +34,33 @@
 
 public class HA2PhasePrepareMessage implements IHA2PhasePrepareMessage, Serializable {
 
-    private static final long serialVersionUID = 1L;
-    
+    /**
+     * Note: The original {@link #serialVersionUID} was <code>1L</code> - this
+     * version was never release. The {@link #serialVersionUID} was changed to
+     * <code>2L</code> when adding the {@link #consensusReleaseTime} and
+     * {@link #isGatherService} fields. It is not possible to roll forward from
+     * the non-released version without shutting down each service before
+     * allowing another commit.
+     */
+    private static final long serialVersionUID = 2L;
+
+    private final IHANotifyReleaseTimeResponse consensusReleaseTime;
+    private final boolean isGatherService;
     private final boolean isJoinedService;
     private final boolean isRootBlock0;
     private final byte[] rootBlock;
     private final long timeout;
     private final TimeUnit unit;
 
-    public HA2PhasePrepareMessage(final boolean isJoinedService,
+    public HA2PhasePrepareMessage(
+            final IHANotifyReleaseTimeResponse consensusReleaseTime,
+            final boolean isGatherService, final boolean isJoinedService,
             final IRootBlockView rootBlock, final long timeout,
             final TimeUnit unit) {
         
+        if (consensusReleaseTime == null)
+            throw new IllegalArgumentException();
+
         if (rootBlock == null)
             throw new IllegalArgumentException();
 
@@ -55,6 +70,10 @@
         if (unit == null)
             throw new IllegalArgumentException();
 
+        this.consensusReleaseTime = consensusReleaseTime;
+        
+        this.isGatherService = isGatherService;
+        
         this.isJoinedService = isJoinedService;
         
         this.isRootBlock0 = rootBlock.isRootBlock0();
@@ -72,7 +91,18 @@
         
     }
 
+
     @Override
+    public IHANotifyReleaseTimeResponse getConsensusReleaseTime() {
+        return consensusReleaseTime;
+    }
+
+    @Override
+    public boolean isGatherService() {
+        return isGatherService;
+    }
+
+    @Override
     public boolean isJoinedService() {
         return isJoinedService;
     }
@@ -100,4 +130,17 @@
         return unit;
     }
 
+    @Override
+    public String toString() {
+        return super.toString()+"{"//
+                +"consensusReleaseTime="+getConsensusReleaseTime()//
+                +",isGatherService="+isGatherService()//
+                +",isPrepareService="+isJoinedService()//
+                +",isRootBlock0="+isRootBlock0()//
+                +",rootBlock()="+getRootBlock()//
+                +",timeout="+getTimeout()//
+                +",unit="+getUnit()//
+                +"}";
+    }
+    
 }

Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/msg/IHA2PhasePrepareMessage.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/msg/IHA2PhasePrepareMessage.java	2013-07-01 16:18:22 UTC (rev 7208)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/msg/IHA2PhasePrepareMessage.java	2013-07-01 16:27:45 UTC (rev 7209)
@@ -39,14 +39,33 @@
 public interface IHA2PhasePrepareMessage extends IHAMessage {
 
     /**
+     * The consensus release time from the GATHER.
+     */
+    IHANotifyReleaseTimeResponse getConsensusReleaseTime();
+    
+    /**
      * <code>true</code> iff the service was recognized as being joined with the
-     * met quorum at the time that the prepare message was prepared.
+     * met quorum at the time that the GATHER message was prepared.
      * <p>
      * Note: This is used to support atomic decisions about whether or not a
      * service was joined with the met quorum at the time that the leader
+     * decided to update the consensus release time. Services that are in the
+     * pipeline and resynchronizing were either joined or not for the purposes
+     * of a given 2-phase GATHER operation based on this flag.
+     */
+    boolean isGatherService();
+    
+    /**
+     * <code>true</code> iff the service was recognized as being joined with the
+     * met quorum at the time that the PREPARE message was prepared.
+     * <p>
+     * Note: This is used to support atomic decisions about whether or not a
+     * service was joined with the met quorum at the time that the leader
      * decided to commit. Services that are in the pipeline and resynchronizing
      * will either be joined or not for the purposes of a given 2-phase commit
      * based on this flag.
+     * <p>
+     * Note: A better name for this flag would be <code>isPrepareService()</code>.
      */
     boolean isJoinedService();
     

Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/AbstractJournal.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/AbstractJournal.java	2013-07-01 16:18:22 UTC (rev 7208)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/AbstractJournal.java	2013-07-01 16:27:45 UTC (rev 7209)
@@ -38,11 +38,9 @@
 import java.rmi.RemoteException;
 import java.security.DigestException;
 import java.security.NoSuchAlgorithmException;
-import java.util.Arrays;
 import java.util.HashMap;
 import java.util.HashSet;
 import java.util.Iterator;
-import java.util.LinkedHashSet;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
@@ -101,6 +99,8 @@
 import com.bigdata.ha.HAGlue;
 import com.bigdata.ha.HAStatusEnum;
 import com.bigdata.ha.HATXSGlue;
+import com.bigdata.ha.IJoinedAndNonJoinedServices;
+import com.bigdata.ha.JoinedAndNonJoinedServices;
 import com.bigdata.ha.PrepareRequest;
 import com.bigdata.ha.PrepareResponse;
 import com.bigdata.ha.QuorumService;
@@ -2907,60 +2907,6 @@
     }
 
     /**
-     * Helper class finds all joined and non-joined services for the quorum
-     * client.
-     * 
-     * @author <a href="mailto:tho...@us...">Bryan
-     *         Thompson</a>
-     */
-    private static class JoinedAndNonJoinedServices {
-
-        // The services joined with the met quorum, in their join order.
-        final UUID[] joinedServiceIds;
-        
-        // The services in the write pipeline (in any order).
-        final Set<UUID> nonJoinedPipelineServiceIds;
-
-        public JoinedAndNonJoinedServices(
-                final Quorum<HAGlue, QuorumService<HAGlue>> quorum) {
-
-            // The services joined with the met quorum, in their join order.
-            joinedServiceIds = quorum.getJoined();
-
-            // The UUID for this service.
-            final UUID serviceId = quorum.getClient().getServiceId();
-
-            if (joinedServiceIds.length == 0
-                    || !joinedServiceIds[0].equals(serviceId)) {
-
-                /*
-                 * Sanity check. Verify that the first service in the join order
-                 * is *this* service. This is a precondition for the service to
-                 * be the leader.
-                 */
-
-                throw new RuntimeException("Not leader: serviceId=" + serviceId
-                        + ", joinedServiceIds="
-                        + Arrays.toString(joinedServiceIds));
-
-            }
-
-            // The services in the write pipeline (in any order).
-            nonJoinedPipelineServiceIds = new LinkedHashSet<UUID>(
-                    Arrays.asList(quorum.getPipeline()));
-
-            // Remove all services that are joined from this collection.
-            for (UUID joinedServiceId : joinedServiceIds) {
-
-                nonJoinedPipelineServiceIds.remove(joinedServiceId);
-
-            }
-
-        }
-        
-    }
-        
-    /**
      * Get timestamp that will be assigned to this commit point.
      * <P>
      * Note: This will spin until commit time advances over
@@ -3151,6 +3097,8 @@
             final QuorumService<HAGlue> quorumService = quorum == null ? null
                     : quorum.getClient();
 
+            final IJoinedAndNonJoinedServices gatherJoinedAndNonJoinedServices;
+            final IHANotifyReleaseTimeResponse consensusReleaseTime;
             if ((_bufferStrategy instanceof IHABufferStrategy)
                     && quorum != null && quorum.isHighlyAvailable()) {
 
@@ -3182,13 +3130,14 @@
                 try {
                 
                     // Atomic decision point for GATHER re joined services.
-                    final JoinedAndNonJoinedServices tmp = new JoinedAndNonJoinedServices(
+                    gatherJoinedAndNonJoinedServices = new JoinedAndNonJoinedServices(
                             quorum);
-
+                    
                     // Run the GATHER protocol.
-                    ((AbstractHATransactionService) getLocalTransactionManager()
+                    consensusReleaseTime = ((AbstractHATransactionService) getLocalTransactionManager()
                             .getTransactionService())
-                            .updateReleaseTimeConsensus(tmp.joinedServiceIds,
+                            .updateReleaseTimeConsensus(
+                                    gatherJoinedAndNonJoinedServices.getJoinedServiceIds(),
                                     getHAReleaseTimeConsensusTimeout(),
                                     TimeUnit.MILLISECONDS);
 
@@ -3205,7 +3154,16 @@
                     
                 }
                 
-            } // if HA
+            } else {
+                
+                /*
+                 * Not HA. Did not do GATHER.
+                 */
+                
+                gatherJoinedAndNonJoinedServices = null;
+                consensusReleaseTime = null;
+                
+            } // if (HA) do GATHER
 
             /*
              * Before flushing the commitRecordIndex we need to check for
@@ -3455,13 +3413,13 @@
                 try {
 
                     // Atomic decision point for joined vs non-joined services.
-                    final JoinedAndNonJoinedServices tmp = new JoinedAndNonJoinedServices(
+                    final IJoinedAndNonJoinedServices prepareJoinedAndNonJoinedServices = new JoinedAndNonJoinedServices(
                             quorum);
 
-                    final PrepareRequest req = new PrepareRequest(
-                            tmp.joinedServiceIds,//
-                            tmp.nonJoinedPipelineServiceIds,//
-//                            !old.isRootBlock0(),//
+                    final PrepareRequest req = new PrepareRequest(//
+                            consensusReleaseTime,//
+                            gatherJoinedAndNonJoinedServices,//
+                            prepareJoinedAndNonJoinedServices,//
                             newRootBlock,//
                             quorumService.getPrepareTimeout(), // timeout
                             TimeUnit.MILLISECONDS//
@@ -6076,7 +6034,7 @@
      * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/673" >
      *      Native thread leak in HAJournalServer process </a>
      */
-    private final AtomicReference<Future<Void>> gatherFuture = new AtomicReference<Future<Void>>();
+    private final AtomicReference<Future<IHANotifyReleaseTimeResponse>> gatherFuture = new AtomicReference<Future<IHANotifyReleaseTimeResponse>>();
     
     /**
      * The {@link Quorum} for this service -or- <code>null</code> if the service
@@ -6585,174 +6543,196 @@
              */
             public Boolean call() throws Exception {
 
-                /*
-                 * Get and clear the [gatherFuture]. A service which was joined
-                 * at the atomic decision point for the GATHER will have a
-                 * non-null Future here. A service which is newly joined and
-                 * which joined *after* the GATHER will have a [null] Future
-                 * here. If the service participated in the gather, then we will
-                 * use this Future to decide if it should vote NO. If the
-                 * service joined *after* the GATHER, then the Future will be
-                 * [null] and we will ignore it.
-                 * 
-                 * FIXME GATHER/PREPARE: This does not verify that the service
-                 * joined after the GATHER when [oldFuture] is [null]. This
-                 * condition is simply assumed to be true. We need to
-                 * distinguish between the two cases described above That could
-                 * be done by returning the set of joined services as of the
-                 * atomic decision point for the GATHER.
-                 */
-                final Future<Void> oldFuture = gatherFuture
-                        .getAndSet(null/* newValue */);
-
                 try {
-                
-                if (haLog.isInfoEnabled())
-                    haLog.info("gatherFuture=" + oldFuture);
                     
-                final IRootBlockView rootBlock = prepareMessage.getRootBlock();
+                    /*
+                     * Get and clear the [gatherFuture]. A service which was
+                     * joined at the atomic decision point for the GATHER will
+                     * have a non-null Future here. A service which is newly
+                     * joined and which joined *after* the GATHER will have a
+                     * [null] Future here. If the service participated in the
+                     * gather, then we will use this Future to decide if it
+                     * should vote NO. If the service joined *after* the GATHER,
+                     * then the Future will be [null] and we will ignore it.
+                     * 
+                     * Note: This is checked below.
+                     */
+                    final Future<IHANotifyReleaseTimeResponse> oldFuture = gatherFuture
+                            .getAndSet(null/* newValue */);
 
-                if (haLog.isInfoEnabled())
-                    haLog.info("preparedRequest=" + rootBlock + ", isLeader: " + isLeader);
+                    if (haLog.isInfoEnabled())
+                        haLog.info("gatherFuture=" + oldFuture);
+                        
+                    final IRootBlockView rootBlock = prepareMessage.getRootBlock();
+    
+                    if (haLog.isInfoEnabled())
+                        haLog.info("preparedRequest=" + rootBlock + ", isLeader: " + isLeader);
+    
+                    if (rootBlock == null)
+                        throw new IllegalStateException();
+    
+                    // Validate the new root block against the current root block.
+                    validateNewRootBlock(/*isJoined,*/ isLeader, AbstractJournal.this._rootBlock, rootBlock);
+    
+                    if (haLog.isInfoEnabled())
+                        haLog.info("validated=" + rootBlock);
+    
+                    /*
+                     * Verify that the local release time is consisent with the
+                     * GATHER.
+                     */
+                    final IHANotifyReleaseTimeResponse consensusReleaseTime = prepareMessage
+                            .getConsensusReleaseTime();
+                    
+                    {
+                        
+                        final long localReleaseTime = getLocalTransactionManager()
+                                .getTransactionService().getReleaseTime();
+    
+                        // Note: Per the GatherTask (in Journal.java).
+                        final long expectedReleaseTime = Math.max(0L,
+                                consensusReleaseTime.getCommitTime() - 1);
 
-                if (rootBlock == null)
-                    throw new IllegalStateException();
+                        if (localReleaseTime != expectedReleaseTime) {
 
-                // Validate the new root block against the current root block.
-                validateNewRootBlock(/*isJoined,*/ isLeader, AbstractJournal.this._rootBlock, rootBlock);
+                            throw new AssertionError(
+                                    "Local service does not agree with consensusReleaseTime: localReleaseTime="
+                                            + localReleaseTime
+                                            + ", expectedReleaseTime="
+                                            + expectedReleaseTime
+                                            + ", consensusReleaseTime="
+                                            + consensusReleaseTime);
 
-                if (haLog.isInfoEnabled())
-                    haLog.info("validated=" + rootBlock);
-
-                /*
-                 * if(follower) {...}
-                 */
-                if (/*isJoined &&*/ !isLeader) {
-
-                    /**
-                     * This is a follower.
-                     * 
-                     * Validate the release time consensus protocol was
-                     * completed successfully on the follower.
-                     * 
-                     * Note: We need to block here (on oldFuture.get()) in case
-                     * the follower has not finished applying the updated
-                     * release time.
-                     * 
-                     * @see <a
-                     *      href="https://sourceforge.net/apps/trac/bigdata/ticket/673"
-                     *      > Native thread leak in HAJournalServer process </a>
+                        }
+    
+                    }
+    
+                    /*
+                     * if(follower) {...}
                      */
-
-                    if (oldFuture == null) {
-
-                        /*
-                         * Ok not to be part of consensus, could have just
-                         * joined.
+                    if (/*isJoined &&*/ !isLeader) {
+    
+                        /**
+                         * This is a follower.
                          * 
-                         * The GATHER is determines the earliest visible
-                         * commit point for new transaction starts. It is
-                         * not specifically about the commit itself.
+                         * Validate the release time consensus protocol was
+                         * completed successfully on the follower.
                          * 
-                         * Therefore the only restriction is to control when
-                         * historical transactions are permitted. We should
-                         * only fail the PREPARE if we have an earliest
-                         * transaction point prior to that reported in the
-                         * prepared rootblock.
-                         * 
-                         * Note: There should be no active transactions if
-                         * we joined on a live write and have not taken part
-                         * in a gather.
-                         * 
-                         * FIXME GATHER/PREPARE: Can we validate that this is a
-                         * newly joined service and hence that we can vote
-                         * YES unconditionally? Can we validate that there
-                         * are no active transactions? Can we validate that
-                         * the follower's releaseTime is consisent with (the
-                         * same as) the consensus release time for the
-                         * leader?
+                         * @see <a
+                         *      href="https://sourceforge.net/apps/trac/bigdata/ticket/673"
+                         *      > Native thread leak in HAJournalServer process </a>
                          */
-                        
-                        vote.set(true);
+    
+                        if (!prepareMessage.isGatherService()) {
+    
+                            /*
+                             * This service did not participate in the GATHER.
+                             * Instead, it joined after the GATHER but before
+                             * the PREPARE.
+                             */
 
-                        // Done.
-                        return vote.get();
-                        
-                    }
+                            // [gatherFuture] should have been [null].
+                            assert oldFuture == null;
+                            
+                            vote.set(true);
+    
+                            // Done.
+                            return vote.get();
+                            
+                        }
 
-                    try {
-                        oldFuture.get();
-                        // Gather was successful - fall through.
-                    } catch (InterruptedException e) {
-                        // Note: Future isDone(). Caller should not block.
-                        throw new AssertionError();
-                    } catch (CancellationException e) {
-                        // Gather cancelled on the follower (e.g., immediately above).
-                        haLog.error("Gather cancelled on follower: serviceId="
-                                + getServiceId() + " : " + e, e);
-                        return vote.get();
-                    } catch (ExecutionException e) {
-                        // Gather failed on the follower.
-                        haLog.error("Gather failed on follower: serviceId="
-                                + getServiceId() + " : " + e, e);
-                        return vote.get();
-                    }
+                        /**
+                         * Note: We need to block here (on oldFuture.get()) in
+                         * case the follower has not finished applying the
+                         * updated release time.
+                         */
+                        try {
 
-                }
-                
-                /*
-                 * Call to ensure strategy does everything required for itself
-                 * before final root block commit. At a minimum it must flush
-                 * its write cache to the backing file (issue the writes).
-                 */
-                // _bufferStrategy.commit(); // lifted to before we
-                // retrieve
-                // RootBlock in commitNow
-                /*
-                 * Force application data to stable storage _before_ we update
-                 * the root blocks. This option guarantees that the application
-                 * data is stable on the disk before the atomic commit. Some
-                 * operating systems and/or file systems may otherwise choose an
-                 * ordered write with the consequence that the root blocks are
-                 * laid down on the disk before the application data and a hard
-                 * failure could result in the loss of application data
-                 * addressed by the new root blocks (data loss on restart).
-                 * 
-                 * Note: We do not force the file metadata to disk. If that is
-                 * done, it will be done by a force() after we write the root
-                 * block on the disk.
-                 */
-                if (doubleSync) {
+                            // Note: [oldFuture] MUST be non-null!
+                            final IHANotifyReleaseTimeResponse tmp = oldFuture.get();
+                            
+                            if ((tmp.getCommitCounter() != consensusReleaseTime
+                                    .getCommitCounter())
+                                    || (tmp.getCommitTime() != consensusReleaseTime
+                                            .getCommitTime())) {
+                            
+                                throw new AssertionError(
+                                        "GatherTask reports different consensus: GatherTask="
+                                                + tmp
+                                                + ", consensusReleaseTime="
+                                                + consensusReleaseTime);
+                            }
 
-                    _bufferStrategy.force(false/* metadata */);
-
-                }
-
-                // Vote YES.
-                vote.set(true);
-
-                return vote.get();
-
-                } finally {
-                
-                    if(!vote.get()) {
-                        if (oldFuture != null) {
                             /*
-                             * Did GATHER and voted NO.
+                             * Gather was successful - fall through.
                              */
-                            doRejectedCommit();
-                        } else {
+                            
+                        } catch (InterruptedException e) {
                             /*
-                             * FIXME GATHER/PREPARE : This is *assuming* that we
-                             * have a newly joined service. That should have
-                             * been verified above.
+                             * Note: Future isDone(). Caller should not block.
                              */
-                            haLog.info("Did not do GATHER : Presumed newly joined service.");
+                            throw new AssertionError();
+                        } catch (CancellationException e) {
+                            /*
+                             * Gather cancelled on the follower (e.g.,
+                             * immediately above).
+                             */
+                            haLog.error("Gather cancelled on follower: serviceId="
+                                    + getServiceId() + " : " + e, e);
+                            return vote.get();
+                        } catch (ExecutionException e) {
+                            // Gather failed on the follower.
+                            haLog.error("Gather failed on follower: serviceId="
+                                    + getServiceId() + " : " + e, e);
+                            return vote.get();
                         }
+    
                     }
                     
+                    /*
+                     * Call to ensure strategy does everything required for itself
+                     * before final root block commit. At a minimum it must flush
+                     * its write cache to the backing file (issue the writes).
+                     */
+                    // _bufferStrategy.commit(); // lifted to before we
+                    // retrieve
+                    // RootBlock in commitNow
+                    /*
+                     * Force application data to stable storage _before_ we update
+                     * the root blocks. This option guarantees that the application
+                     * data is stable on the disk before the atomic commit. Some
+                     * operating systems and/or file systems may otherwise choose an
+                     * ordered write with the consequence that the root blocks are
+                     * laid down on the disk before the application data and a hard
+                     * failure could result in the loss of application data
+                     * addressed by the new root blocks (data loss on restart).
+                     * 
+                     * Note: We do not force the file metadata to disk. If that is
+                     * done, it will be done by a force() after we write the root
+                     * block on the disk.
+                     */
+                    if (doubleSync) {
+    
+                        _bufferStrategy.force(false/* metadata */);
+    
+                    }
+    
+                    // Vote YES.
+                    vote.set(true);
+    
+                    return vote.get();
+
+                } finally {
+
+                    if (!vote.get()) {
+                        /*
+                         * Throw away our local write set.
+                         */
+                        doRejectedCommit();
+                    }
+
                 }
-                
+
             }
 
         }
@@ -7439,18 +7419,26 @@
             if (haLog.isInfoEnabled())
                 haLog.info("req=" + req);
 
-            // Clear the old outcome. Reference SHOULD be null. Ensure not running.
-            final Future<Void> oldFuture = gatherFuture.getAndSet(null);
+            {
+
+                /*
+                 * Clear the old outcome. Reference SHOULD be null. Ensure not
+                 * running.
+                 */
+                final Future<IHANotifyReleaseTimeResponse> oldFuture = gatherFuture
+                        .getAndSet(null);
+
+                if (oldFuture != null && !oldFuture.isDone())
+                    oldFuture.cancel(true/* mayInterruptIfRunning */);
+
+            }
             
-            if (oldFuture != null && !oldFuture.isDone())
-                oldFuture.cancel(true/* mayInterruptIfRunning */);
-            
-            final Callable<Void> task = ((AbstractHATransactionService) AbstractJournal.this
+            final Callable<IHANotifyReleaseTimeResponse> task = ((AbstractHATransactionService) AbstractJournal.this
                     .getLocalTransactionManager()
                     .getTransactionService())
                     .newGatherMinimumVisibleCommitTimeTask(req);
 
-            final FutureTask<Void> ft = new FutureTask<Void>(task);
+            final FutureTask<IHANotifyReleaseTimeResponse> ft = new FutureTask<IHANotifyReleaseTimeResponse>(task);
 
             // Save reference to the gather Future.
             gatherFuture.set(ft);

Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/Journal.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/Journal.java	2013-07-01 16:18:22 UTC (rev 7208)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/Journal.java	2013-07-01 16:27:45 UTC (rev 7209)
@@ -1113,7 +1113,7 @@
          */
         // Note: Executed on the leader.
         @Override
-        public void updateReleaseTimeConsensus(final UUID[] joinedServiceIds,
+        public IHANotifyReleaseTimeResponse updateReleaseTimeConsensus(final UUID[] joinedServiceIds,
                 final long timeout, final TimeUnit units) throws IOException,
                 InterruptedException, TimeoutException, BrokenBarrierException {
 
@@ -1208,6 +1208,8 @@
 
                 setReleaseTime(newReleaseTime);
 
+                return consensus;
+                
             } finally {
 
                 barrierLock.unlock();
@@ -1372,11 +1374,8 @@
 
         }
 
-        /**
-         * Return the {@link GatherTask} that will be executed by the follower.
-         */
         @Override
-        public Callable<Void> newGatherMinimumVisibleCommitTimeTask(
+        public Callable<IHANotifyReleaseTimeResponse> newGatherMinimumVisibleCommitTimeTask(
                 final IHAGatherReleaseTimeRequest req) {
 
             return new GatherTask(req);
@@ -1414,7 +1413,7 @@
          * 
          * @see #newTx(long)
          */
-        private class GatherTask implements Callable<Void> {
+        private class GatherTask implements Callable<IHANotifyReleaseTimeResponse> {
 
             private final IHAGatherReleaseTimeRequest req;
 
@@ -1434,7 +1433,7 @@
              * commitNow(), then that change will be detected by the leader and
              * it will break the {@link CyclicBarrier}.
              */
-            public Void call() throws Exception {
+            public IHANotifyReleaseTimeResponse call() throws Exception {
 
                 if (log.isInfoEnabled())
                     log.info("Running gather on follower");
@@ -1580,7 +1579,7 @@
                     }
 
                     // Done.
-                    return null;
+                    return consensusReleaseTime;
                     
                 } catch (Throwable t) {
 

Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/service/AbstractHATransactionService.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/service/AbstractHATransactionService.java	2013-07-01 16:18:22 UTC (rev 7208)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/service/AbstractHATransactionService.java	2013-07-01 16:27:45 UTC (rev 7209)
@@ -36,6 +36,7 @@
 
 import com.bigdata.ha.HATXSGlue;
 import com.bigdata.ha.msg.IHAGatherReleaseTimeRequest;
+import com.bigdata.ha.msg.IHANotifyReleaseTimeResponse;
 import com.bigdata.journal.AbstractJournal;
 import com.bigdata.journal.ITransactionService;
 
@@ -54,7 +55,7 @@
     }
     
     /**
-     * Factory for the Gather task on the follower.
+     * Factory for the Gather task that will be executed by the follower.
      * 
      * @param req
      *            The request.
@@ -64,7 +65,7 @@
      * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/673" >
      *      Native thread leak in HAJournalServer process </a>
      */
-    abstract public Callable<Void> newGatherMinimumVisibleCommitTimeTask(
+    abstract public Callable<IHANotifyReleaseTimeResponse> newGatherMinimumVisibleCommitTimeTask(
             final IHAGatherReleaseTimeRequest req);
 
     /**
@@ -79,7 +80,7 @@
      * @param units
      *            The units for that timeout.
      */
-    abstract public void updateReleaseTimeConsensus(
+    abstract public IHANotifyReleaseTimeResponse updateReleaseTimeConsensus(
             final UUID[] joinedServiceIds, final long timeout,
             final TimeUnit units) throws IOException, TimeoutException,
             InterruptedException, Exception;

Modified: branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/AbstractHA3JournalServerTestCase.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/AbstractHA3JournalServerTestCase.java	2013-07-01 16:18:22 UTC (rev 7208)
+++ branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/AbstractHA3JournalServerTestCase.java	2013-07-01 16:27:45 UTC (rev 7209)
@@ -536,9 +536,7 @@
      * individual service directory.
      */
     protected File getTestDir() {
-
-        return new File("benchmark/CI-HAJournal-1");
-        
+        return new File(FEDNAME + "/CI-HAJournal-1");
     }
 
     protected File getServiceDirA() {

Modified: branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/AbstractHAJournalServerTestCase.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/AbstractHAJournalServerTestCase.java	2013-07-01 16:18:22 UTC (rev 7208)
+++ branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/AbstractHAJournalServerTestCase.java	2013-07-01 16:27:45 UTC (rev 7209)
@@ -99,6 +99,16 @@
     }
 
     /**
+     * The name of the federation. This is also used for the JINI locator
+     * GROUPS.
+     * <p>
+     * Note: If you want to change this, then you also need to change the
+     * HAJournal-{ABC}.config, jiniClient.config, zkClient.config, and how
+     * you run the LookupStarter class to use the new federation name.
+     */
+    static final protected String FEDNAME = "benchmark";
+    
+    /**
      * Path to the config files.
      */
     static final protected String SRC_PATH = "bigdata-jini/src/test/com/bigdata/journal/jini/ha/";
@@ -107,8 +117,8 @@
      * Path to the directory in which the service directories exist. The
      * individual service directories are formed by adding the service name.
      */
-    // static final protected String TGT_PATH = "/Volumes/SSDData/bigdata/benchmark/CI-HAJournal-1/";
-    static final protected String TGT_PATH = "benchmark/CI-HAJournal-1/";
+    // static final protected String TGT_PATH = "/Volumes/SSDData/bigdata/"+FEDNAME+"/CI-HAJournal-1/";
+    static final protected String TGT_PATH = FEDNAME + "/CI-HAJournal-1/";
 
     /**
      * The timeout used to await quorum meet or break.

Modified: branches/READ_CACHE/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataSPARQLUpdateTest2.java
===================================================================
--- branches/READ_CACHE/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataSPARQLUpdateTest2.java	2013-07-01 16:18:22 UTC (rev 7208)
+++ branches/READ_CACHE/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataSPARQLUpdateTest2.java	2013-07-01 16:27:45 UTC (rev 7209)
@@ -466,17 +466,29 @@
         {
 
             final StringBuilder sb = new StringBuilder();
-            
+
+            /*
+             * FIXME test variants w/ and w/o embedded sub-select and verify the
+             * *order* is preserved when using the embedded subselect w/ its
+             * order by. Also, verify that we translate this by lifting out the
+             * sub-select since the top-level query is empty at thast point.
+             * 
+             * Also, document this on the wiki. The sub-select is necessary because
+             * SPARQL does not allow solution modifiers on the top-level WHERE clause
+             * for INSERT/DELETE+WHERE.
+             */
             sb.append("PREFIX rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n");
             sb.append("PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n");
             sb.append("PREFIX foaf: <http://xmlns.com/foaf/0.1/>\n");
             sb.append("INSERT INTO %namedSet1\n");
             sb.append("SELECT ?x ?name\n");
+            sb.append("WHERE { SELECT ?x ?name\n");
             sb.append("WHERE {\n");
             sb.append("  ?x rdf:type foaf:Person .\n");
             sb.append("  ?x rdfs:label ?name .\n");
             sb.append("}\n");
-//            sb.append("ORDER BY ?name");
+            sb.append("ORDER BY ?name\n");
+            sb.append("}");
             
             con.prepareUpdate(QueryLanguage.SPARQL, sb.toString()).execute();
 

Modified: branches/READ_CACHE/src/resources/HAJournal/HAJournal.config
===================================================================
--- branches/READ_CACHE/src/resources/HAJournal/HAJournal.config	2013...
 
[truncated message content]

Thread: [Bigdata-commit] SF.net SVN: bigdata:[7209] branches/READ_CACHE (Page 4)

Fast, scalable, robust graph database platform

bigdata-commit