cweb-cvs — CVS commit messages are posted to this list.

[cweb-CVS] bigdata/src/java/com/bigdata/scaleup PartitionedIndexView.java, NONE, 1.1 MasterJournal.java, NONE, 1.1 IsolatablePartitionedIndexView.java, NONE, 1.1 Name2MetadataAddr.java, 1.3, 1.4 MetadataIndex.java, 1.5, 1.6 SlaveJournal.java, 1.3, 1.4 AbstractPartitionTask.java, 1.1, 1.2 PartitionedIndex.java, 1.8, NONE PartitionedJournal.java, 1.13, NONE

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv5433/src/java/com/bigdata/scaleup

Modified Files:
	Name2MetadataAddr.java MetadataIndex.java SlaveJournal.java 
	AbstractPartitionTask.java 
Added Files:
	PartitionedIndexView.java MasterJournal.java 
	IsolatablePartitionedIndexView.java 
Removed Files:
	PartitionedIndex.java PartitionedJournal.java 
Log Message:
Continued minor refactoring in line with model updates.

--- NEW FILE: IsolatablePartitionedIndexView.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Feb 12, 2007
 */

package com.bigdata.scaleup;

import com.bigdata.isolation.IIsolatableIndex;
import com.bigdata.isolation.UnisolatedBTree;
import com.bigdata.objndx.IEntryIterator;

/**
 * A {@link PartitionedIndexView} that supports transactions and deletion
 * markers.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 * 
 * FIXME implement; support processing of delete markers. They can exist in the mutable
 * btree and in index segments that are not either a clean first eviction or a
 * full compacting merge (e.g., they can still exist in a compacting merge if
 * there are other index segments or btrees that are part of a partition but are
 * not partitipating in the compacting merge).
 */
public class IsolatablePartitionedIndexView extends PartitionedIndexView implements IIsolatableIndex {

    /**
     * @param btree
     * @param mdi
     */
    public IsolatablePartitionedIndexView(UnisolatedBTree btree, MetadataIndex mdi) {
        super(btree, mdi);
        throw new UnsupportedOperationException();
    }

    public boolean contains(byte[] key) {
        // TODO Auto-generated method stub
        return false;
    }

    public Object insert(Object key, Object value) {
        // TODO Auto-generated method stub
        return null;
    }

    public Object lookup(Object key) {
        // TODO Auto-generated method stub
        return null;
    }

    public int rangeCount(byte[] fromKey, byte[] toKey) {
        // TODO Auto-generated method stub
        return 0;
    }

    public IEntryIterator rangeIterator(byte[] fromKey, byte[] toKey) {
        // TODO Auto-generated method stub
        return null;
    }

    public Object remove(Object key) {
        // TODO Auto-generated method stub
        return null;
    }

    public void contains(int ntuples, byte[][] keys, boolean[] contains) {
        // TODO Auto-generated method stub

    }

    public void insert(int ntuples, byte[][] keys, Object[] values) {
        // TODO Auto-generated method stub

    }

    public void lookup(int ntuples, byte[][] keys, Object[] values) {
        // TODO Auto-generated method stub

    }

    public void remove(int ntuples, byte[][] keys, Object[] values) {
        // TODO Auto-generated method stub

    }

}

Index: MetadataIndex.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/MetadataIndex.java,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** MetadataIndex.java	6 Mar 2007 20:38:06 -0000	1.5
--- MetadataIndex.java	11 Mar 2007 11:42:42 -0000	1.6
***************
*** 44,50 ****
  package com.bigdata.scaleup;
  
! 
  import com.bigdata.objndx.BTree;
  import com.bigdata.objndx.BTreeMetadata;
  import com.bigdata.rawstore.IRawStore;
  
--- 44,53 ----
  package com.bigdata.scaleup;
  
! import com.bigdata.isolation.IsolatedBTree;
! import com.bigdata.journal.Journal;
! import com.bigdata.journal.Tx;
  import com.bigdata.objndx.BTree;
  import com.bigdata.objndx.BTreeMetadata;
+ import com.bigdata.objndx.IndexSegment;
  import com.bigdata.rawstore.IRawStore;
  
***************
*** 64,67 ****
--- 67,76 ----
   * @todo mutation operations need to be synchronized.
   * 
+  * @todo Track which {@link IndexSegment}s and {@link Journal}s are required
+  *       to support the {@link IsolatedBTree}s in use by a {@link Tx}. Deletes
+  *       of old journals and index segments MUST be deferred until no
+  *       transaction remains which can read those data. This metadata must be
+  *       restart-safe so that resources are eventually deleted.
+  *       
   * @todo define a UUID so that is at least possible to rename a partitioned
   *       index? the uuid would be store in the metadata record for the metadata
***************
*** 78,82 ****
      /**
       * The name of the metadata index, which is the always the same as the name
!      * under which the corresponding {@link PartitionedIndex} was registered.
       */
      private final String name;
--- 87,91 ----
      /**
       * The name of the metadata index, which is the always the same as the name
!      * under which the corresponding {@link PartitionedIndexView} was registered.
       */
      private final String name;
***************
*** 84,88 ****
      /**
       * The name of the metadata index, which is the always the same as the name
!      * under which the corresponding {@link PartitionedIndex} was registered.
       */
      final public String getName() {
--- 93,97 ----
      /**
       * The name of the metadata index, which is the always the same as the name
!      * under which the corresponding {@link PartitionedIndexView} was registered.
       */
      final public String getName() {
***************
*** 101,105 ****
       * @param name
       *            The name of the metadata index - this MUST be the name under
!      *            which the corresponding {@link PartitionedIndex} was
       *            registered.
       */
--- 110,114 ----
       * @param name
       *            The name of the metadata index - this MUST be the name under
!      *            which the corresponding {@link PartitionedIndexView} was
       *            registered.
       */
***************
*** 139,143 ****
          /**
           * The name of the metadata index, which is the always the same as the name
!          * under which the corresponding {@link PartitionedIndex} was registered.
           */
          public final String name;
--- 148,152 ----
          /**
           * The name of the metadata index, which is the always the same as the name
!          * under which the corresponding {@link PartitionedIndexView} was registered.
           */
          public final String name;

--- PartitionedJournal.java DELETED ---

--- NEW FILE: MasterJournal.java ---
/**

 The Notice below must appear in each file of the Source Code of any
 copy you distribute of the Licensed Product.  Contributors to any
 Modifications may add their own copyright notices to identify their
 own contributions.

 License:

 The contents of this file are subject to the CognitiveWeb Open Source
 License Version 1.1 (the License).  You may not copy or use this file,
 in either source code or executable form, except in compliance with
 the License.  You may obtain a copy of the License from

 http://www.CognitiveWeb.org/legal/license/

 Software distributed under the License is distributed on an AS IS
 basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
 the License for the specific language governing rights and limitations
[...1306 lines suppressed...]
            
        }

        return seg;

    }

    /**
     * The maximum #of index segments that will be held open without a hard
     * reference existing for that index segment in the application.
     */
    final int INDEX_SEGMENT_LRU_CAPACITY = 5;
    
    /**
     * A cache for recently used index segments designed to prevent their being
     * swept by the VM between uses.
     */
    private final WeakValueCache<String/*filename*/, IndexSegment> resourceCache;
    
}

Index: AbstractPartitionTask.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/AbstractPartitionTask.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** AbstractPartitionTask.java	8 Mar 2007 18:14:06 -0000	1.1
--- AbstractPartitionTask.java	11 Mar 2007 11:42:42 -0000	1.2
***************
*** 100,104 ****
          IPartitionTask {
  
!     protected final PartitionedJournal master;
      /**
       * Branching factor used for generated {@link IndexSegment}(s). 
--- 100,104 ----
          IPartitionTask {
  
!     protected final MasterJournal master;
      /**
       * Branching factor used for generated {@link IndexSegment}(s). 
***************
*** 171,175 ****
       *       correct separator keys must be scheduled.
       */
!     public AbstractPartitionTask(PartitionedJournal master, String name,
              int branchingFactor, double errorRate, int partId,
              byte[] fromKey, byte[] toKey) {
--- 171,175 ----
       *       correct separator keys must be scheduled.
       */
!     public AbstractPartitionTask(MasterJournal master, String name,
              int branchingFactor, double errorRate, int partId,
              byte[] fromKey, byte[] toKey) {
***************
*** 227,231 ****
           *            The output segment identifier.
           */
!         public BuildTask(PartitionedJournal master, String name,
                  int branchingFactor, double errorRate, int partId,
                  byte[] fromKey, byte[] toKey, IResourceMetadata src, int segId) {
--- 227,231 ----
           *            The output segment identifier.
           */
!         public BuildTask(MasterJournal master, String name,
                  int branchingFactor, double errorRate, int partId,
                  byte[] fromKey, byte[] toKey, IResourceMetadata src, int segId) {
***************
*** 290,294 ****
           *            True iff this will be a full compacting merge.
           */
!         protected AbstractMergeTask(PartitionedJournal master, String name,
                  int branchingFactor, double errorRate, int partId,
                  byte[] fromKey, byte[] toKey, int segId,
--- 290,294 ----
           *            True iff this will be a full compacting merge.
           */
!         protected AbstractMergeTask(MasterJournal master, String name,
                  int branchingFactor, double errorRate, int partId,
                  byte[] fromKey, byte[] toKey, int segId,
***************
*** 429,433 ****
           *            The output segment identifier.
           */
!         public MergeTask(PartitionedJournal master, String name,
                  int branchingFactor, double errorRate, int partId,
                  byte[] fromKey, byte[] toKey, IResourceMetadata[] resources,
--- 429,433 ----
           *            The output segment identifier.
           */
!         public MergeTask(MasterJournal master, String name,
                  int branchingFactor, double errorRate, int partId,
                  byte[] fromKey, byte[] toKey, IResourceMetadata[] resources,
***************
*** 483,487 ****
           *            merge operation.
           */
!         public FullMergeTask(PartitionedJournal master, String name,
                  int branchingFactor, double errorRate, int partId,
                  byte[] fromKey, byte[] toKey, long commitTime, int segId) {
--- 483,487 ----
           *            merge operation.
           */
!         public FullMergeTask(MasterJournal master, String name,
                  int branchingFactor, double errorRate, int partId,
                  byte[] fromKey, byte[] toKey, long commitTime, int segId) {
***************
*** 496,500 ****
          protected IResourceMetadata[] getResources() {
              
!             final PartitionedIndex oldIndex = ((PartitionedIndex) master
                      .getIndex(name, commitTime));
              
--- 496,500 ----
          protected IResourceMetadata[] getResources() {
              
!             final PartitionedIndexView oldIndex = ((PartitionedIndexView) master
                      .getIndex(name, commitTime));
              

Index: SlaveJournal.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/SlaveJournal.java,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** SlaveJournal.java	8 Mar 2007 18:14:06 -0000	1.3
--- SlaveJournal.java	11 Mar 2007 11:42:42 -0000	1.4
***************
*** 47,50 ****
--- 47,51 ----
  import java.util.Properties;
  
+ import com.bigdata.isolation.IIsolatableIndex;
  import com.bigdata.isolation.UnisolatedBTree;
  import com.bigdata.journal.IJournal;
***************
*** 52,56 ****
  import com.bigdata.journal.Name2Addr;
  import com.bigdata.objndx.BTree;
- import com.bigdata.objndx.BTreeMetadata;
  import com.bigdata.objndx.IEntryIterator;
  import com.bigdata.objndx.IIndex;
--- 53,56 ----
***************
*** 82,88 ****
      protected Name2MetadataAddr name2MetadataAddr;
  
!     private final PartitionedJournal master;
      
!     protected PartitionedJournal getMaster() {
  
          return master;
--- 82,88 ----
      protected Name2MetadataAddr name2MetadataAddr;
  
!     private final MasterJournal master;
      
!     protected MasterJournal getMaster() {
  
          return master;
***************
*** 90,94 ****
      }
      
!     public SlaveJournal(PartitionedJournal master,Properties properties) {
          
          super(properties);
--- 90,94 ----
      }
      
!     public SlaveJournal(MasterJournal master,Properties properties) {
          
          super(properties);
***************
*** 155,159 ****
               */
  
!             name2MetadataAddr = (Name2MetadataAddr)BTreeMetadata
                      .load(this, addr);
  
--- 155,159 ----
               */
  
!             name2MetadataAddr = (Name2MetadataAddr)BTree
                      .load(this, addr);
  
***************
*** 166,172 ****
  
      /**
!      * Registers and returns a {@link PartitionedIndex} under the given name and
       * assigns an {@link UnisolatedBTree} to absorb writes for that
!      * {@link PartitionedIndex}. The resulting index will support transactional
       * isolation.
       * <p>
--- 166,172 ----
  
      /**
!      * Registers and returns a {@link PartitionedIndexView} under the given name and
       * assigns an {@link UnisolatedBTree} to absorb writes for that
!      * {@link PartitionedIndexView}. The resulting index will support transactional
       * isolation.
       * <p>
***************
*** 176,181 ****
       * zero {@link IndexSegment}s.
       * <p>
!      * Note: The returned object is invalid once the {@link PartitionedJournal}
!      * {@link PartitionedJournal#overflow()}s.
       * <p>
       * Note: You MUST {@link #commit()} before the registered index will be
--- 176,181 ----
       * zero {@link IndexSegment}s.
       * <p>
!      * Note: The returned object is invalid once the {@link MasterJournal}
!      * {@link MasterJournal#overflow()}s.
       * <p>
       * Note: You MUST {@link #commit()} before the registered index will be
***************
*** 189,195 ****
      
      /**
!      * Registers and returns a {@link PartitionedIndex} under the given name and
       * assigns the supplied {@link IIndex} to absorb writes for that
!      * {@link PartitionedIndex}.
       * <p>
       * A {@link MetadataIndex} is also registered under the given name and an
--- 189,195 ----
      
      /**
!      * Registers and returns a {@link PartitionedIndexView} under the given name and
       * assigns the supplied {@link IIndex} to absorb writes for that
!      * {@link PartitionedIndexView}.
       * <p>
       * A {@link MetadataIndex} is also registered under the given name and an
***************
*** 198,203 ****
       * zero {@link IndexSegment}s.
       * <p>
!      * Note: The returned object is invalid once the {@link PartitionedJournal}
!      * {@link PartitionedJournal#overflow()}s.
       * <p>
       * Note: You MUST {@link #commit()} before the registered index will be
--- 198,203 ----
       * zero {@link IndexSegment}s.
       * <p>
!      * Note: The returned object is invalid once the {@link MasterJournal}
!      * {@link MasterJournal#overflow()}s.
       * <p>
       * Note: You MUST {@link #commit()} before the registered index will be
***************
*** 243,272 ****
  
          /*
!          * Now register the partitioned index on the super class.
           */
!         return super.registerIndex(name, new PartitionedIndex(
!                 (BTree) btree, mdi));
          
      }
      
      public IIndex getIndex(String name) {
  
          IIndex index = super.getIndex(name);
!         
!         if(index instanceof BTree) {
              
!             index = new PartitionedIndex((BTree)index,getMetadataIndex(name));
              
          }
  
          return index;
!         
      }
  
      /**
!      * Return the {@link MetadataIndex} for the named {@link PartitionedIndex}.
!      * This object is used to maintain the definitions of the partitions for
!      * that index, including where each file is located that contains live data
!      * for the index.
       */
      public MetadataIndex getMetadataIndex(String name) {
--- 243,314 ----
  
          /*
!          * Now register the view on the super class. The view will delegate
!          * writes and the commit protocol to the btree specified by the caller,
!          * but will support index partitions and will read from a fused view of
!          * the resources for each index partition.
           */
!         return super.registerIndex(name, getView((BTree) btree, mdi));
          
      }
      
+     /**
+      * Returns a {@link PartitionedIndexView} or
+      * {@link IsolatablePartitionedIndexView} depending on whether or not the
+      * named index supports isolation.
+      */
      public IIndex getIndex(String name) {
  
+         /*
+          * Obtain the persistence capable index. If this is a cache hit, then it
+          * will be the view. Otherwise this will be either a BTree or an
+          * UnisolatedBTree that was just loaded from the store and we will have
+          * to wrap it up as a view.
+          */
          IIndex index = super.getIndex(name);
! 
!         if (index instanceof BTree) {
! 
!             /*
!              * Wrap up the mutable B+-Tree responsible for absorbing writes as
!              * a view.
!              */
              
!             MetadataIndex mdi = getMetadataIndex(name);
              
+             /*
+              * Choose the type of view based on whether or not the registered index
+              * supports isolation.
+              */
+             index = getView((BTree)index,mdi);
+ 
          }
  
          return index;
! 
      }
  
      /**
!      * Choose the type of view based on whether or not the registered index
!      * supports isolation.
!      */
!     private PartitionedIndexView getView(BTree btree, MetadataIndex mdi) {
!         
!         if (btree instanceof IIsolatableIndex) {
!             
!             return new IsolatablePartitionedIndexView((UnisolatedBTree) btree,
!                     mdi);
!         } else {
!             
!             return new PartitionedIndexView((BTree) btree, mdi);
!     
!         }
! 
!     }
!     
!     /**
!      * Return the {@link MetadataIndex} for the named
!      * {@link PartitionedIndexView}. This object is used to maintain the
!      * definitions of the partitions for that index, including where each
!      * resource is located that contains data for each partition of the index.
       */
      public MetadataIndex getMetadataIndex(String name) {
***************
*** 279,292 ****
  
      /**
!      * FIXME write tests. use to clean up the test suites. note that we can
!      * not drop files until we have done an atomic commit. also note that
       * restart-safe removal of files is not going to happen without some
!      * additional sophistication.  one solution is to periodically compare
!      * the named indices and the segments directory, either deleting or
       * flagging for an operator those things which do not belong.
       */
      public void dropIndex(String name) {
          
!         PartitionedIndex ndx = (PartitionedIndex) getIndex(name);
          
          if (ndx == null)
--- 321,335 ----
  
      /**
!      * FIXME write tests. use to clean up the test suites. note that we can not
!      * drop files until we have done an atomic commit. also note that
       * restart-safe removal of files is not going to happen without some
!      * additional sophistication, eg, placing a task to schedule deletion of
!      * resources onto the restart safe schedule. one solution is to periodically
!      * compare the named indices and the segments directory, either deleting or
       * flagging for an operator those things which do not belong.
       */
      public void dropIndex(String name) {
          
!         PartitionedIndexView ndx = (PartitionedIndexView) getIndex(name);
          
          if (ndx == null)
***************
*** 354,356 ****
      }
  
! }
\ No newline at end of file
--- 397,399 ----
      }
  
! }

--- NEW FILE: PartitionedIndexView.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Feb 7, 2007
 */

package com.bigdata.scaleup;

import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.NoSuchElementException;

import com.bigdata.journal.ICommitter;
import com.bigdata.journal.Journal;
import com.bigdata.objndx.AbstractBTree;
import com.bigdata.objndx.BTree;
import com.bigdata.objndx.BatchContains;
import com.bigdata.objndx.BatchInsert;
import com.bigdata.objndx.BatchLookup;
import com.bigdata.objndx.BatchRemove;
import com.bigdata.objndx.EmptyEntryIterator;
import com.bigdata.objndx.ReadOnlyFusedView;
import com.bigdata.objndx.IEntryIterator;
import com.bigdata.objndx.IFusedView;
import com.bigdata.objndx.IIndex;
import com.bigdata.objndx.IndexSegment;

/**
 * A mutable B+-Tree that is dynamically partitioned into one or more key
 * ranges. Each key range is a <i>partition</i>. A partition is defined by the
 * lowest value key that can enter that partition (the separator key). A
 * {@link MetadataIndex} contains the definitions of the partitions. Each
 * partition has a mutable {@link BTree} that will absorb writes for a
 * partition. The same {@link BTree} is used to absorb writes distined for any
 * partitions of the same index on the same {@link Journal}. This class
 * internally tracks whether or not a write has occurred on each partition. That
 * information is used to guide eviction decisions when the
 * {@link Journal#overflow()}s and buffered writes are migrated onto one or
 * more {@link IndexSegment}s each of which contains historical read-only data
 * for a single index partition.
 * <p>
 * Writes on a {@link PartitionIndex} simply write through to the {@link BTree}
 * corresponding to that index (having the same name) on the
 * {@link Journal journal} (they are effectively multiplexed on the journal).
 * <p>
 * Read operations process the spanned partitions sequence so that the
 * aggregated results are in index order. For each partition, the read request
 * is constructed using ah {@link IFusedView} of the mutable {@link BTree} used
 * to absorb writes and the live {@link IndexSegment}s for that partition.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 * 
 * @todo add a restart safe data structure tracking each partition for which
 *       writes have been absorbed on the {@link Journal}. this could be just
 *       an ordered set of partition identifiers for which writes have been
 *       absorbed that is maintained by an insertion sort, an ordered array of
 *       <partId:writeCounter>, or a btree mapping partition identifiers to
 *       write counters. The data need to be associated as part of the metadata
 *       for the btree that is absorbing writes for that partition. The simplest
 *       way to do this is to subclass BTree or UnisolatedBTree so that they are
 *       partition aware and automatically track the #of writes destined for
 *       each partition.
 * 
 * @todo For a scale-out solution the clients are responsible for talking with
 *       the metadata service and locating the data service for each partition.
 *       The client is responsible for directing the writes to the correct data
 *       service. The data service simply fronts for the journal and directs the
 *       writes onto the {@link BTree} absorbing writes for the named index. It
 *       is a (client) error for a write to be directed to a {@link Journal} not
 *       tasked with buffering data for the partition into which that write
 *       should go.
 */
public class PartitionedIndexView implements IIndex, ICommitter {

    /**
     * The mutable {@link BTree} used to absorb all writes for the
     * {@link PartitionedIndexView}.
     */
    protected final BTree btree;

    /**
     * The metadata index used to locate the partitions relevant to a given read
     * operation.
     */
    private final MetadataIndex mdi;

    /**
     * A cache of the fused views for in use partitions.
     * 
     * @todo reconcile this with
     *       {@link MasterJournal#getIndex(String, IResourceMetadata)}
     *       which provides a weak value cache for index segments.
     */
    private final Map<Integer,ReadOnlyFusedView> views = new HashMap<Integer,ReadOnlyFusedView>();

    public MasterJournal getMaster() {
        
        return getSlave().getMaster();
        
    }
    
    public SlaveJournal getSlave() {
        
        return (SlaveJournal)getBTree().getStore();
        
    }
    
    /**
     * The mutable {@link BTree} used to absorb writes.
     */
    public BTree getBTree() {
        
        return btree;
        
    }
    
    /**
     * Opens and returns the live {@link IndexSegment}s for the partition.
     * These {@link IndexSegments} MUST be explicitly closed in order to reclaim
     * resources and release the lock on the backing files.
     * 
     * @param pmd
     *            The partition.
     * 
     * @return The live {@link IndexSegment}s for that partition.
     */
    protected IndexSegment[] openIndexSegments(PartitionMetadata pmd) {
        
        final int liveCount = pmd.getLiveCount();
        
        IndexSegment[] segs = new IndexSegment[liveCount];
        
        MasterJournal master = getMaster();
        
        int n = 0;
        
        for(int i=0; i<pmd.segs.length; i++) {
            
            if(pmd.segs[i].state != ResourceState.Live) continue;
            
            segs[n++] = (IndexSegment) master.getIndex(getName(), pmd.segs[i]);
            
        }
        
        assert n == liveCount;
        
        return segs;

    }

    /**
     * Close all open views, including any backing index segments.
     */
    protected void closeViews() {

        Iterator<Map.Entry<Integer,ReadOnlyFusedView>> itr = views.entrySet().iterator();
        
        while(itr.hasNext()) {
            
            ReadOnlyFusedView view = itr.next().getValue();
            
            // @todo assumes one open segment per view.
            IndexSegment seg = (IndexSegment)view.srcs[1];
            
            seg.close();
            
        }

        views.clear();
        
    }
    
    /**
     * Return a fused view on the partition in which the key would be found. The
     * view is cached and will be reused on subsequent requests. This operation
     * can have high latency if the view for the partition is not in the cache.
     * 
     * @param key
     *            The search key.
     * 
     * @return The fused view.
     * 
     * @todo use an weak value LRU policy to close out partitions that are not
     *       being actively used. note that we still need to know which
     *       partitions have been touched on a {@link PartitionedIndexView} when we
     *       export its data in response to a journal overflow event. This
     *       suggests a hard reference queue to hold onto recently used views
     *       and a persistent bitmap or flag in the metadata index to identify
     *       those partitions for which we need to export data. Note that we
     *       MUST hold onto any view that is currently being used to report data
     *       to the application, e.g., by a striterator. This means that we need
     *       to use a hash map with weak or soft reference values.
     *       <p>
     *       Alternatively, perhaps we can write a custom iterator over the
     *       mutable btree for the index that automatically exports or performs
     *       a compacting merge for each partition for which it encounters data
     *       on an entry scan.
     */
    protected IIndex getView(byte[] key) {
        
        PartitionMetadata pmd = mdi.find(key);
        
        ReadOnlyFusedView view = views.get(pmd.partId);
        
        if(view==null) {
            
            if(pmd.getLiveCount()==0) {
                
                // the btree is the view.
                return btree;
                
            }
            
            /*
             * Open the live index segments for this partition (high latency).
             */
            
            IndexSegment[] segs = openIndexSegments(pmd);
            
            AbstractBTree[] sources = new AbstractBTree[segs.length+1];
            
            // the mutable btree.
            sources[0] = btree;
            
            // the immutable historical segments.
            System.arraycopy(segs, 0, sources, 1, segs.length);
            
            // create the fused view.
            view = new ReadOnlyFusedView(sources);
            
            // place the view in the cache.
            views.put(pmd.partId, view);
            
        }
        
        return view;
        
    }

    /**
     * Return the resources required to provide a coherent view of the partition
     * in which the key is found.
     * 
     * @param key
     *            The key.
     * 
     * @return The resources required to read on the partition containing that
     *         key. The resources are arranged in reverse timestamp order
     *         (increasing age).
     * 
     * @todo reconcile with {@link #getView(byte[])}?
     */
    protected IResourceMetadata[] getResources(byte[] key) {

        JournalMetadata journalResource = new JournalMetadata((Journal) btree
                .getStore(), ResourceState.Live);

        PartitionMetadata pmd = mdi.find(key);

        final int liveCount = pmd.getLiveCount();

        IResourceMetadata[] resources = new IResourceMetadata[liveCount + 1];

        int n = 0;

        resources[n++] = journalResource;

        for (int i = 0; i < resources.length; i++) {

            SegmentMetadata seg = pmd.segs[i];

            if (seg.state != ResourceState.Live)
                continue;

            resources[n++] = seg;

        }

        assert n == resources.length;
        
        return resources;

    }
    
    public String getName() {
        
        return mdi.getName();
        
    }
    
    /**
     * @param btree
     *            The mutable {@link BTree} used to absorb all writes for the
     *            {@link PartitionedIndexView}.
     * @param mdi
     *            The metadata index used to locate the partitions relevant to a
     *            given read operation.
     */
    public PartitionedIndexView(BTree btree, MetadataIndex mdi) {
        
        this.btree = btree;
        
        this.mdi = mdi;
        
    }
    
    /*
     * Non-batch API.
     * 
     * The write operations are trivial since we always direct them to the named
     * btree on the journal.
     * 
     * The point read operations are also trivial conceptually. The are always
     * directed to the fused view of the mutable btree on the journal and the
     * live index segments for the partition that spans the search key.
     * 
     * The rangeCount and rangeIterator operations are more complex. The need to
     * be processed by each partition spanned by the from/to key range. We need
     * to present the operation to each partition in turn and append the results
     * together.
     */

    public Object insert(Object key, Object value) {

        return btree.insert(key, value);
        
    }

    public Object remove(Object key) {

        return btree.remove(key);
        
    }

    public boolean contains(byte[] key) {

        return getView(key).contains(key);
        
    }

    public Object lookup(Object key) {

        return getView((byte[])key).lookup(key);
        
    }

    /**
     * For each partition spanned by the key range, report the sum of the counts
     * for each source in the view for that partition (since this is based on 
     * {@link ReadOnlyFusedView}s for each partition, it will may overcount the #of
     * entries actually in the range on each partition).
     * <p>
     * If the count would exceed {@link Integer#MAX_VALUE} then the result is
     * {@link Integer#MAX_VALUE}.
     */
    public int rangeCount(byte[] fromKey, byte[] toKey) {

        // index of the first partition to check.
        final int fromIndex = (fromKey == null ? 0 : mdi.findIndexOf(fromKey));

        // index of the last partition to check.
        final int toIndex = (toKey == null ? 0 : mdi.findIndexOf(toKey));

        // per javadoc, keys out of order returns zero(0).
        if(toIndex<fromIndex) return 0;

        // use to counters so that we can look for overflow.
        int count = 0;
        int lastCount = 0;
        
        for( int index = fromIndex; index<=toIndex; index++) {

            // The first key that would enter the nth partition.
            byte[] separatorKey = mdi.keyAt(index);

            // Add in the count from that partition.
            count += getView(separatorKey).rangeCount(fromKey, toKey);
            
            if(count<lastCount) {
                
                return Integer.MAX_VALUE;
                
            }
            
            lastCount = count;
            
        }
        
        return count;
        
    }

    /**
     * Return an iterator that visits all values in the key range in key order.
     * The iterator will visit each partition spanned by the key range in turn.
     */
    public IEntryIterator rangeIterator(byte[] fromKey, byte[] toKey) {

        // index of the first partition to check.
        final int fromIndex = (fromKey == null ? 0 : mdi.findIndexOf(fromKey));

        // index of the last partition to check.
        final int toIndex = (toKey == null ? 0 : mdi.findIndexOf(toKey));

        // keys are out of order.
        if(toIndex<fromIndex) return EmptyEntryIterator.INSTANCE;

        // iterator that will visit all key/vals in that range.
        return new PartitionedRangeIterator(fromKey, toKey, fromIndex, toIndex);

    }
    
    /*
     * Batch API.
     * 
     * The write operations are trivial since we always direct them to the named
     * btree on the journal.
     * 
     * The read operations are more complex. We need to partition the set of
     * keys based on the partitions to which those keys would be directed. This
     * is basically a join against the {@link MetadataIndex}. This operation is
     * also required on the rangeCount and rangeIterator methods on the
     * non-batch api.
     */

    public void insert(BatchInsert op) {

        btree.insert(op);

    }

    public void remove(BatchRemove op) {

        btree.remove(op);

    }

    // FIXME contains(batch)
    public void contains(BatchContains op) {

        throw new UnsupportedOperationException();

    }

    // FIXME lookup(batch)
    public void lookup(BatchLookup op) {

        throw new UnsupportedOperationException();

    }

    /**
     * An iterator that visits all key/value entries in a specified key range
     * across one or more partitions.
     * 
     * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
     * @version $Id$
     */
    private class PartitionedRangeIterator implements IEntryIterator {

        /**
         * The first key to visit or null iff there is no lower bound.
         */
        private final byte[] fromKey;

        /**
         * The first key NOT to visit or null iff there is no upper bound.
         */
        private final byte[] toKey;

        /**
         * The index of the first partition spanned by the key range.
         */
        private final int fromIndex;

        /**
         * The index of the last partition spanned by the key range.
         */
        private final int toIndex;

        /**
         * The index of the partition whose entries are currently being visited.
         */
        private int index;
        
        /**
         * The iterator for the current partition.
         */
        private IEntryIterator src;

        /**
         * The last visited key and null iff we have not visited anything yet.
         */
        private byte[] key = null;

        /**
         * The last visited value.
         */
        private Object val = null;

        /**
         * 
         * @param fromKey
         *            The first key to visit or null iff there is no lower
         *            bound.
         * @param toKey
         *            The first key NOT to visit or null iff there is no upper
         *            bound.
         * @param fromIndex
         *            The index of the first partition spanned by the key range.
         * @param toIndex
         *            The index of the last partition spanned by the key range.
         */
        public PartitionedRangeIterator(byte[] fromKey, byte[] toKey, int fromIndex,int toIndex) {

            assert fromIndex >= 0;
            assert toIndex >= fromIndex;
            
            this.fromKey = fromKey;
            
            this.toKey = toKey;
            
            this.fromIndex = fromIndex;
            
            this.toIndex = toIndex;

            // the first partition to visit.
            this.index = fromIndex;
            
            // The first key that would enter that partition.
            byte[] separatorKey = mdi.keyAt(index);

            // The rangeIterator for that partition.
            src = getView(separatorKey).rangeIterator(fromKey, toKey);
            
        }

        public boolean hasNext() {

            if(src.hasNext()) return true;

            // The current partition has been exhausted.
            if(index < toIndex) {
                
                // the next partition to visit.
                index++;
                
                // The first key that would enter that partition.
                byte[] separatorKey = mdi.keyAt(index);

                // The rangeIterator for that partition.
                src = getView(separatorKey).rangeIterator(fromKey, toKey);

                return src.hasNext();

            } else {

                // All partitions have been exhausted.
                return false;
            
            }
            
        }

        public Object next() {

            if (!hasNext()) {

                throw new NoSuchElementException();

            }

            /*
             * eagerly fetch the key and value so that we can report them using
             * getKey() and getValue()
             */
            val = src.next();
            key = src.getKey();

            return val;

        }

        public Object getValue() {

            if (key == null) {
                
                // nothing has been visited yet.
                throw new IllegalStateException();
                
            }

            return val;

        }

        public byte[] getKey() {

            if (key == null) {
                
                // nothing has been visited yet.
                throw new IllegalStateException();
                
            }
            
            return key;

        }

        /**
         * Not supported.
         * 
         * @exception UnsupportedOperationException
         *                Always.
         */
        public void remove() {

            throw new UnsupportedOperationException();

        }

    }

    /**
     * The commit is delegated to the mutable B+-Tree since that is where all
     * the writes are absorbed.
     */
    public long handleCommit() {

        return btree.handleCommit();
        
    }

}

Index: Name2MetadataAddr.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/Name2MetadataAddr.java,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** Name2MetadataAddr.java	17 Feb 2007 21:34:21 -0000	1.3
--- Name2MetadataAddr.java	11 Mar 2007 11:42:41 -0000	1.4
***************
*** 50,57 ****
  import com.bigdata.journal.Name2Addr;
  import com.bigdata.objndx.BTreeMetadata;
- import com.bigdata.objndx.IIndex;
  import com.bigdata.rawstore.IRawStore;
  
  /**
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
   * @version $Id$
--- 50,59 ----
  import com.bigdata.journal.Name2Addr;
  import com.bigdata.objndx.BTreeMetadata;
  import com.bigdata.rawstore.IRawStore;
  
  /**
+  * Extension of {@link Name2Addr} for locating the {@link MetadataIndex}
+  * associated with a named {@link PartitionedIndexView}.
+  * 
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
   * @version $Id$
***************
*** 70,79 ****
          
      }
-     
-     protected IIndex loadBTree(IRawStore store, String name, long addr) {
-         
-         return (MetadataIndex)BTreeMetadata.load(this.store, addr);
- 
-     }
  
  }
--- 72,75 ----

--- PartitionedIndex.java DELETED ---

[cweb-CVS] bigdata/src/java/com/bigdata/istore JournalStore.java, 1.5, 1.6 IOM.java, 1.3, 1.4

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/istore
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv5433/src/java/com/bigdata/istore

Modified Files:
	JournalStore.java IOM.java 
Log Message:
Continued minor refactoring in line with model updates.

Index: IOM.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/istore/IOM.java,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** IOM.java	11 Dec 2006 16:33:59 -0000	1.3
--- IOM.java	11 Mar 2007 11:42:43 -0000	1.4
***************
*** 79,83 ****
   * @todo Add stream-based operations. Note that very large objects are going to
   *       scale out using an unisolated stream-based API. That API probably
!  *       belongs to the {@link IStore} rather than the {@link IOM} since large
   *       streams can not be made transactional (well, they can we a journal
   *       designed for large writes and using a disk-only strategy, but do you
--- 79,83 ----
   * @todo Add stream-based operations. Note that very large objects are going to
   *       scale out using an unisolated stream-based API. That API probably
!  *       belongs to the {@link IIndexStore} rather than the {@link IOM} since large
   *       streams can not be made transactional (well, they can we a journal
   *       designed for large writes and using a disk-only strategy, but do you

Index: JournalStore.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/istore/JournalStore.java,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** JournalStore.java	8 Feb 2007 21:32:11 -0000	1.5
--- JournalStore.java	11 Mar 2007 11:42:43 -0000	1.6
***************
*** 137,141 ****
  //    static class OM implements IOM {
  //        
! //        private final IStore store;
  //        private final Journal journal;
  //
--- 137,141 ----
  //    static class OM implements IOM {
  //        
! //        private final IIndexStore store;
  //        private final Journal journal;
  //
***************
*** 149,153 ****
  //        final private OMExtensibleSerializer _extSer;
  //        
! //        OM(IStore store,Journal journal) {
  //        
  //            assert store != null;
--- 149,153 ----
  //        final private OMExtensibleSerializer _extSer;
  //        
! //        OM(IIndexStore store,Journal journal) {
  //        
  //            assert store != null;
***************
*** 291,295 ****
  //    static class OMTx implements ITx {
  //
! //        private final IStore store;
  //
  //        private final Tx tx;
--- 291,295 ----
  //    static class OMTx implements ITx {
  //
! //        private final IIndexStore store;
  //
  //        private final Tx tx;
***************
*** 297,301 ****
  //        private final TxExtensibleSerializer extser;
  //
! //        OMTx(IStore store, Tx tx) {
  //
  //            assert store != null;
--- 297,301 ----
  //        private final TxExtensibleSerializer extser;
  //
! //        OMTx(IIndexStore store, Tx tx) {
  //
  //            assert store != null;

[cweb-CVS] bigdata/src/test/com/bigdata/journal TestCommitRecordIndex.java, 1.1, 1.2 TestTx.java, 1.19, 1.20 TestNamedIndices.java, 1.2, 1.3

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/test/com/bigdata/journal
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv5433/src/test/com/bigdata/journal

Modified Files:
	TestCommitRecordIndex.java TestTx.java TestNamedIndices.java 
Log Message:
Continued minor refactoring in line with model updates.

Index: TestCommitRecordIndex.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/journal/TestCommitRecordIndex.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** TestCommitRecordIndex.java	17 Feb 2007 21:34:12 -0000	1.1
--- TestCommitRecordIndex.java	11 Mar 2007 11:42:34 -0000	1.2
***************
*** 50,54 ****
  import java.nio.ByteBuffer;
  
! import com.bigdata.objndx.BTreeMetadata;
  import com.bigdata.rawstore.IRawStore;
  import com.bigdata.rawstore.SimpleMemoryRawStore;
--- 50,54 ----
  import java.nio.ByteBuffer;
  
! import com.bigdata.objndx.BTree;
  import com.bigdata.rawstore.IRawStore;
  import com.bigdata.rawstore.SimpleMemoryRawStore;
***************
*** 136,140 ****
              
              // reload the index from the store.
!             ndx = (CommitRecordIndex)BTreeMetadata.load(store, ndx_addr);
  
              // verify existence test.
--- 136,140 ----
              
              // reload the index from the store.
!             ndx = (CommitRecordIndex)BTree.load(store, ndx_addr);
  
              // verify existence test.

Index: TestTx.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/journal/TestTx.java,v
retrieving revision 1.19
retrieving revision 1.20
diff -C2 -d -r1.19 -r1.20
*** TestTx.java	22 Feb 2007 16:59:34 -0000	1.19
--- TestTx.java	11 Mar 2007 11:42:34 -0000	1.20
***************
*** 733,737 ****
  //
  //            // Transaction begins before the write.
! //            IStore tx0 = new Tx(journal,0);
  //
  //            // Write a random data version for id 0.
--- 733,737 ----
  //
  //            // Transaction begins before the write.
! //            IIndexStore tx0 = new Tx(journal,0);
  //
  //            // Write a random data version for id 0.
***************
*** 752,756 ****
  //
  //            // Transaction begins after the write.
! //            IStore tx1 = new Tx(journal,1);
  //
  //            /*
--- 752,756 ----
  //
  //            // Transaction begins after the write.
! //            IIndexStore tx1 = new Tx(journal,1);
  //
  //            /*

Index: TestNamedIndices.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/journal/TestNamedIndices.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** TestNamedIndices.java	22 Feb 2007 16:59:34 -0000	1.2
--- TestNamedIndices.java	11 Mar 2007 11:42:34 -0000	1.3
***************
*** 50,54 ****
  import com.bigdata.objndx.BTree;
  import com.bigdata.objndx.SimpleEntry;
! import com.bigdata.scaleup.PartitionedJournal;
  
  /**
--- 50,54 ----
  import com.bigdata.objndx.BTree;
  import com.bigdata.objndx.SimpleEntry;
! import com.bigdata.scaleup.MasterJournal;
  
  /**
***************
*** 68,72 ****
   * 
   * @todo reuse this test suite to test the basic features of a
!  *       {@link PartitionedJournal}.
   * 
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
--- 68,72 ----
   * 
   * @todo reuse this test suite to test the basic features of a
!  *       {@link MasterJournal}.
   * 
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>

[cweb-CVS] bigdata/src/java/com/bigdata/objndx ReadOnlyFusedView.java, NONE, 1.1 IndexSegmentMerger.java, 1.10, 1.11 BTreeMetadata.java, 1.12, 1.13 BTree.java, 1.36, 1.37 IFusedView.java, 1.1, 1.2 FusedView.java, 1.6, NONE

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv5433/src/java/com/bigdata/objndx

Modified Files:
	IndexSegmentMerger.java BTreeMetadata.java BTree.java 
	IFusedView.java 
Added Files:
	ReadOnlyFusedView.java 
Removed Files:
	FusedView.java 
Log Message:
Continued minor refactoring in line with model updates.

Index: BTree.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/BTree.java,v
retrieving revision 1.36
retrieving revision 1.37
diff -C2 -d -r1.36 -r1.37
*** BTree.java	6 Mar 2007 20:38:05 -0000	1.36
--- BTree.java	11 Mar 2007 11:41:45 -0000	1.37
***************
*** 47,50 ****
--- 47,52 ----
  package com.bigdata.objndx;
  
+ import java.lang.reflect.Constructor;
+ 
  import com.bigdata.cache.HardReferenceQueue;
  import com.bigdata.journal.ICommitter;
***************
*** 645,648 ****
--- 647,695 ----
  
      /**
+      * Re-load the {@link BTree} or derived class from the store. The
+      * {@link BTree} or derived class MUST provide a public constructor with the
+      * following signature: <code>
+      * 
+      * <i>className</i>(IRawStore store, BTreeMetadata metadata)
+      * 
+      * </code>
+      * 
+      * @param store
+      *            The store.
+      * @param addr
+      *            The address of the {@link BTreeMetadata} record for that
+      *            class.
+      * 
+      * @return The {@link BTree} or derived class loaded from that
+      *         {@link BTreeMetadata} record.
+      * 
+      * @see BTree#newMetadata(), which MUST be overloaded if you subclass
+      *      {@link BTreeMetadata} in order to store extended metadata.
+      */
+     public static BTree load(IRawStore store, long addr) {
+         
+         BTreeMetadata metadata = BTreeMetadata.read(store, addr);
+         
+         try {
+             
+             Class cl = Class.forName(metadata.className);
+             
+             Constructor ctor = cl.getConstructor(new Class[] {
+                     IRawStore.class, BTreeMetadata.class });
+             
+             BTree btree = (BTree) ctor.newInstance(new Object[] { store,
+                     metadata });
+             
+             return btree;
+             
+         } catch(Exception ex) {
+             
+             throw new RuntimeException(ex);
+             
+         }
+         
+     }
+ 
+     /**
       * Factory for mutable nodes and leaves used by the {@link NodeSerializer}.
       */

Index: BTreeMetadata.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/BTreeMetadata.java,v
retrieving revision 1.12
retrieving revision 1.13
diff -C2 -d -r1.12 -r1.13
*** BTreeMetadata.java	8 Mar 2007 18:14:05 -0000	1.12
--- BTreeMetadata.java	11 Mar 2007 11:41:44 -0000	1.13
***************
*** 3,7 ****
  import java.io.Externalizable;
  import java.io.Serializable;
- import java.lang.reflect.Constructor;
  import java.nio.ByteBuffer;
  
--- 3,6 ----
***************
*** 140,148 ****
       * @return the metadata record.
       * 
!      * @see #load(IRawStore, long), which will load the btree not just the
!      *      {@link BTreeMetadata}.
!      * 
!      * @todo review remaining uses of this method vs
!      *       {@link #load(IRawStore, long)}.
       */
      public static BTreeMetadata read(IRawStore store, long addr) {
--- 139,144 ----
       * @return the metadata record.
       * 
!      * @see #load(IRawStore, long), which will load the {@link BTree} or derived
!      *      subclass not just the {@link BTreeMetadata}.
       */
      public static BTreeMetadata read(IRawStore store, long addr) {
***************
*** 153,267 ****
  
      /**
-      * Re-load the {@link BTree} or derived class from the store. The
-      * {@link BTree} or derived class MUST provide a public constructor with the
-      * following signature: <code>
-      * 
-      * <i>className</i>(IRawStore store, BTreeMetadata metadata)
-      * 
-      * </code>
-      * 
-      * @param store
-      *            The store.
-      * @param addr
-      *            The address of the {@link BTreeMetadata} record for that
-      *            class.
-      * 
-      * @return The {@link BTree} or derived class loaded from that
-      *         {@link BTreeMetadata} record.
-      * 
-      * @see BTree#newMetadata(), which MUST be overloaded if you subclass
-      *      {@link BTreeMetadata}.
-      */
-     public static BTree load(IRawStore store, long addr) {
-         
-         BTreeMetadata metadata = read(store, addr);
-         
-         try {
-             
-             Class cl = Class.forName(metadata.className);
-             
-             Constructor ctor = cl.getConstructor(new Class[] {
-                     IRawStore.class, BTreeMetadata.class });
-             
-             BTree btree = (BTree) ctor.newInstance(new Object[] { store,
-                     metadata });
-             
-             return btree;
-             
-         } catch(Exception ex) {
-             
-             throw new RuntimeException(ex);
-             
-         }
-         
-     }
-     
- //    /**
- //     * Write out the metadata record for the btree on the store and return the
- //     * {@link Addr address}.
- //     * 
- //     * @return The address of the metadata record.
- //     */
- //    protected long write(IRawStore store) {
- //
- //        ByteBuffer buf = ByteBuffer.allocate(SIZEOF_METADATA);
- //
- //        buf.putLong(addrRoot);
- //        buf.putInt(branchingFactor);
- //        buf.putInt(height);
- //        buf.putInt(nnodes);
- //        buf.putInt(nleaves);
- //        buf.putInt(nentries);
- //        buf.putInt(keyType.intValue());
- //        
- //        buf.flip(); // prepare for writing.
- //
- //    }
- 
- //    /**
- //     * Read the persistent metadata record for the btree.
- //     * 
- //     * @param addrMetadta
- //     *            The address from which the metadata record will be read.
- //     * 
- //     * @return The persistent identifier of the root of the btree.
- //     */
- //    public BTreeMetadata(IRawStore store,long addrMetadata) {
- //
- //        assert store != null;
- //        
- //        assert addrMetadata != 0L;
- //        
- //        this.addrMetadata = addrMetadata;
- //        
- //        ByteBuffer buf = store.read(addrMetadata,null);
- //        assert buf.position() == 0;
- //        assert buf.limit() == Addr.getByteCount(addrMetadata);
- //        
- //        addrRoot = buf.getLong();
- //        
- //        branchingFactor = buf.getInt();
- //        
- //        height = buf.getInt();
- //
- //        nnodes = buf.getInt();
- //        
- //        nleaves = buf.getInt();
- //        
- //        nentries = buf.getInt();
- //        
- //        keyType = ArrayType.parseInt( buf.getInt() );
- //        
- //        assert branchingFactor >= BTree.MIN_BRANCHING_FACTOR;
- //        assert height >= 0;
- //        assert nnodes >= 0;
- //        assert nleaves >= 0;
- //        assert nentries >= 0;
- //
- //        BTree.log.info(toString());
- //
- //    }
- 
-     /**
       * A human readable representation of the metadata record.
       */
--- 149,152 ----
***************
*** 277,280 ****
--- 162,170 ----
          sb.append(", nentries=" + nentries);
          sb.append(", addrMetadata=" + Addr.toString(addrMetadata));
+         sb.append(", valueSerializer=" + valueSer.getClass().getName());
+         sb.append(", recordCompressor="
+                 + (recordCompressor == null ? null : recordCompressor
+                         .getClass().getName()));
+         sb.append(", useChecksum=" + useChecksum);
          
          return sb.toString();

--- FusedView.java DELETED ---

--- NEW FILE: ReadOnlyFusedView.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Feb 1, 2007
 */

package com.bigdata.objndx;

import java.util.Arrays;
import java.util.NoSuchElementException;

/**
 * <p>
 * A fused view providing read-only operations on multiple B+-Trees mapping
 * variable length unsigned byte[] keys to arbitrary values.
 * </p>
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 * 
 * @todo support N sources for a {@link ReadOnlyFusedView} by chaining together multiple
 *       {@link ReadOnlyFusedView} instances if not in a more efficient manner.
 */
public class ReadOnlyFusedView implements IIndex, IFusedView {

    /**
     * Holds the various btrees that are the sources for the view.
     */
    public final AbstractBTree[] srcs;
    
    public AbstractBTree[] getSources() {
        
        return srcs;
        
    }
    
    public ReadOnlyFusedView(AbstractBTree src1, AbstractBTree src2) {
        
        this(new AbstractBTree[] { src1, src2 });
        
    }
    
    /**
     * 
     * @param srcs
     *            The ordered sources for the fused view. The order of the
     *            elements in this array determines which value will be selected
     *            for a given key by lookup() and which value is retained by
     *            rangeQuery().
     */
    public ReadOnlyFusedView(final AbstractBTree[] srcs) {
        
        if (srcs == null)
            throw new IllegalArgumentException("sources is null");

        if (srcs.length < 2) {
            throw new IllegalArgumentException(
                    "at least two sources are required");
        }

        if(srcs.length>2) {
            // @todo generalize to N>2 sources.
            throw new UnsupportedOperationException(
                    "Only two sources are supported.");
        }
        
        for( int i=0; i<srcs.length; i++) {
            
            if (srcs[i] == null)
                throw new IllegalArgumentException("a source is null");
            
            for(int j=0; j<i; j++) {
                
                if (srcs[i] == srcs[j])
                    throw new IllegalArgumentException(
                            "source used more than once");
                
            }
            
        }

        this.srcs = srcs.clone();
        
    }
    
    /**
     * Write operations are not supported on the view.
     */
    public void insert(BatchInsert op) {
        
        throw new UnsupportedOperationException();
        
    }

    /**
     * Write operations are not supported on the view.
     */
    public void remove(BatchRemove op) {

        throw new UnsupportedOperationException();
        
    }
    
    public Object insert(Object key, Object value) {

        throw new UnsupportedOperationException();

    }

    public Object remove(Object key) {

        throw new UnsupportedOperationException();

    }

    /**
     * Return the first value for the key in an ordered search of the trees in
     * the view.
     */
    public Object lookup(Object key) {
        
        for( int i=0; i<srcs.length; i++) {
            
            Object ret = srcs[i].lookup(key);
            
            if (ret != null)
                return ret;
            
        }

        return null;
        
    }

    /**
     * Returns true if any tree in the view has an entry for the key.
     */
    public boolean contains(byte[] key) {
        
        for( int i=0; i<srcs.length; i++) {
            
            if (srcs[i].contains(key))
                return true;
            
        }

        return false;
        
    }
    
    /**
     * @todo implement and write test of chained lookup operations. the
     *       challenge here is that we only want the first value for a
     *       given key.  this seems to require that we mark tuples to
     *       be ignored on subsequent indices, which in turn needs to
     *       get into the batch api.  the contains() method already has
     *       been modified to finesse this.
     */
    public void lookup(BatchLookup op) {
        
        throw new UnsupportedOperationException();
        
    }

    public void contains( BatchContains op ) {

        for( int i=0; i<srcs.length; i++) {

            AbstractBTree src = srcs[i];
            
            // reset the first tuple index for each pass.
            op.tupleIndex = 0;
            
            src.contains(op);
            
        }

    }

    /**
     * Returns the sum of the range count on each index in the view. This is the
     * maximum #of entries that could lie within that key range. However, the
     * actual number could be less if there are entries for the same key in more
     * than one source index.
     * 
     * @todo this could be done using concurrent threads.
     */
    public int rangeCount(byte[] fromKey, byte[] toKey) {
        
        int count = 0;
        
        for(int i=0; i<srcs.length; i++) {
            
            count += srcs[i].rangeCount(fromKey, toKey);
            
        }
        
        return count;
        
    }

    /**
     * Returns an iterator that visits the distinct entries. When an entry
     * appears in more than one index, the entry is choosen based on the order
     * in which the indices were declared to the constructor.
     */
    public IEntryIterator rangeIterator(byte[] fromKey, byte[] toKey) {
        
        return new FusedEntryIterator(srcs, fromKey, toKey);
        
    }

    /**
     * Helper class merges entries from the sources in the view.
     * 
     * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
     * @version $Id$
     */
    protected static class FusedEntryIterator implements IEntryIterator {

        private final AbstractBTree[] srcs;
//        private final byte[] fromKey;
//        private final byte[] toKey;
        
        private final IEntryIterator[] itrs;
//        private final boolean[] available;
        private final boolean[] exhausted;

        /**
         * The current key from each source and null if we need to get the next
         * key from that source.
         */
        private final byte[][] keys;
        
        /**
         * Index of the iterator that returned the last value and -1 if no
         * iterator has returned a value yet.
         */
        private int current = -1;
        
        public FusedEntryIterator(AbstractBTree[] srcs,byte[] fromKey, byte[] toKey) {
        
            assert srcs != null;
            
            assert srcs.length > 0;
            
            this.srcs = srcs;
            
//            this.fromKey = fromKey;
//            
//            this.toKey = toKey;
            
            itrs = new IEntryIterator[srcs.length];

            for( int i=0; i<itrs.length; i++) {
                
                itrs[i] = srcs[i].rangeIterator(fromKey, toKey);
                
            }
            
//            available = new boolean[srcs.length];
//            
//            Arrays.fill(available,true);

            keys = new byte[itrs.length][];
            
            exhausted = new boolean[srcs.length];
            
            Arrays.fill(exhausted,false);
            
        }

        public byte[] getKey() {
            
            if(current == -1) throw new IllegalStateException();
            
            return itrs[current].getKey();
            
        }

        public Object getValue() {
            
            if(current == -1) throw new IllegalStateException();
            
            return itrs[current].getValue();
            
        }

        public boolean hasNext() {

            // @todo this could use advanceKeyStreams() instead.
            for( int i=0; i<itrs.length; i++ ) {
                
                if (!exhausted[i] && keys[i]!= null || itrs[i].hasNext()) {

                    return true;
                    
                }
                
            }
            
            return false;
            
        }

        /**
         * Make sure that we have the current key for each key stream. If we
         * already have a key for that stream then we use it.
         * 
         * @return The #of key streams with an available key in {@link #keys}.
         *         When zero(0), all key streams are exhausted.
         */
        private int advanceKeyStreams() {
            
            // #of key streams with a key for us to examine.
            int navailable = 0;
            
            for(int i=0; i<itrs.length; i++) {
                
                if (exhausted[i])
                    continue;
                
                if (keys[i] == null) {

                    if (itrs[i].hasNext()) {

                        itrs[i].next();

                        keys[i] = itrs[i].getKey();

                        navailable++;

                    } else {
                        
                        exhausted[i] = true;
                        
                    }
                    
                } else {
                    
                    navailable++;
                    
                }
                
            }

            return navailable;
            
        }
        
        /**
         * We are presented with an ordered set of key streams. Each key stream
         * delivers its keys in order. For a given key, we always choose the
         * first stream having that key. Once a key is found, all subsequent key
         * streams are then advanced until their next key is greater than the
         * current key (this can only cause the same key to be skipped).
         * <p>
         * 
         * Each invocation of this method advances one key in the union of the
         * key streams. We test the current key for each stream on each pass and
         * choose the key that orders first across all key streams.
         * <p>
         * 
         * In the simple case with two streams we examine the current
         * {@link #keys key} on each stream, fetching the next key iff there is
         * no key available on that stream and otherwise using the current key
         * from that stream. If the keys are the same, then we choose the first
         * stream and also clear the current {@link #keys key} for the other
         * stream so that we will skip the current entry on that key stream. If
         * the keys differ, then we choose the stream with the lessor key and
         * clear the {@link #keys key} for that stream to indicate that it has
         * been consumed. In any case we set the index of the choosen stream on
         * {@link #current} so that the other methods on this api will use the
         * corresponding key and value from that stream and return the current
         * value for the choosen stream.
         * 
         * @todo generalize to N>2 key streams.
         */
        public Object next() {

            assert srcs.length == 2;

            int navailable = advanceKeyStreams();

            if(navailable == 0) {
                
                throw new NoSuchElementException();
                
            }
            
            /*
             * Generalization to N streams might sort {key,order,itr} tuples.
             * The choice of the stream with the lessor key is then the first
             * entry in sorted tuples if the comparator pays attention to the
             * stream [order] in addition to the keys.
             * 
             * if a stream is exhausted then we no longer consider it as a key
             * source.
             */
            
            final int cmp = (keys[0] == null ? 1 : keys[1] == null ? -1
                    : BytesUtil.compareBytes(keys[0], keys[1]));
            
            if( cmp == 0 ) {

                // Choose the first stream in a tie.
                
                current = 0;
                
                // The current key on each stream tied in 1st place is consumed.
                
                keys[0] = keys[1] = null;  
                
            } else {
                
                // Choose the stream with the lessor key.
                
                current = cmp < 0 ? 0 : 1;

                keys[current] = null; // this key was consumed.

            }

            // Return the current object on the choosen stream.
            Object value = itrs[current].getValue();  return value;
            
        }

        public void remove() {
            
            throw new UnsupportedOperationException();
            
        }
        
    }
    
}

Index: IFusedView.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/IFusedView.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** IFusedView.java	6 Mar 2007 20:38:05 -0000	1.1
--- IFusedView.java	11 Mar 2007 11:42:34 -0000	1.2
***************
*** 56,60 ****
   * @version $Id$
   */
! public interface IFusedView {
  
  }
--- 56,65 ----
   * @version $Id$
   */
! public interface IFusedView extends IIndex {
  
+     /**
+      * Return the ordered array of sources from which the fused view is reading.
+      */
+     public AbstractBTree[] getSources();
+     
  }

Index: IndexSegmentMerger.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/IndexSegmentMerger.java,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** IndexSegmentMerger.java	6 Mar 2007 20:38:05 -0000	1.10
--- IndexSegmentMerger.java	11 Mar 2007 11:41:44 -0000	1.11
***************
*** 85,89 ****
   *       well.
   * 
!  * @see {@link FusedView}, which provides a dynamic view of two or more btrees.
   *      However, this class is more efficient when we are going to do a bulk
   *      merge operation since it performs the merge and computes the #of output
--- 85,89 ----
   *       well.
   * 
!  * @see {@link ReadOnlyFusedView}, which provides a dynamic view of two or more btrees.
   *      However, this class is more efficient when we are going to do a bulk
   *      merge operation since it performs the merge and computes the #of output

[cweb-CVS] bigdata/src/test/com/bigdata/objndx TestFusedView.java, 1.1, 1.2 TestRestartSafe.java, 1.5, 1.6 TestCommit.java, 1.12, 1.13

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv5433/src/test/com/bigdata/objndx

Modified Files:
	TestFusedView.java TestRestartSafe.java TestCommit.java 
Log Message:
Continued minor refactoring in line with model updates.

Index: TestRestartSafe.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx/TestRestartSafe.java,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** TestRestartSafe.java	22 Feb 2007 16:59:35 -0000	1.5
--- TestRestartSafe.java	11 Mar 2007 11:42:49 -0000	1.6
***************
*** 204,209 ****
              journal = reopenStore(journal);
              
!             final BTree btree = new BTree(journal, BTreeMetadata.read(journal,
!                     addr1));
              
              assertTrue(btree.dump(Level.DEBUG,System.err));
--- 204,208 ----
              journal = reopenStore(journal);
              
!             final BTree btree = BTree.load(journal, addr1);
              
              assertTrue(btree.dump(Level.DEBUG,System.err));

Index: TestCommit.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx/TestCommit.java,v
retrieving revision 1.12
retrieving revision 1.13
diff -C2 -d -r1.12 -r1.13
*** TestCommit.java	9 Feb 2007 16:13:18 -0000	1.12
--- TestCommit.java	11 Mar 2007 11:42:50 -0000	1.13
***************
*** 123,137 ****
  
              // Load the tree.
!             BTree btree = new BTree(store, BTreeMetadata.read(store, addrMetadata),
!                     new HardReferenceQueue<PO>(new DefaultEvictionListener(),
!                             BTree.DEFAULT_HARD_REF_QUEUE_CAPACITY,
!                             BTree.DEFAULT_HARD_REF_QUEUE_SCAN)
!                             );
! //            Integer.valueOf(0),
! //                            null, // no comparator for primitive key type.
! //                            Int32OIdKeySerializer.INSTANCE,
! //                            SimpleEntry.Serializer.INSTANCE,
! //                            null // no record compressor
! //                            );
  
              // verify addrRoot.
--- 123,127 ----
  
              // Load the tree.
!             BTree btree = BTree.load(store, addrMetadata);
  
              // verify addrRoot.
***************
*** 156,164 ****
  
              // Load the tree.
!             BTree btree = new BTree(store, BTreeMetadata.read(store,
!                     addrMetadata), new HardReferenceQueue<PO>(
!                     new DefaultEvictionListener(),
!                     BTree.DEFAULT_HARD_REF_QUEUE_CAPACITY,
!                     BTree.DEFAULT_HARD_REF_QUEUE_SCAN));
  
              // verify addrRoot.
--- 146,150 ----
  
              // Load the tree.
!             BTree btree = BTree.load(store, addrMetadata);
  
              // verify addrRoot.

Index: TestFusedView.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx/TestFusedView.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** TestFusedView.java	1 Feb 2007 21:23:29 -0000	1.1
--- TestFusedView.java	11 Mar 2007 11:42:49 -0000	1.2
***************
*** 49,53 ****
  
  /**
!  * Test suite for {@link FusedView}.
   * 
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
--- 49,53 ----
  
  /**
!  * Test suite for {@link ReadOnlyFusedView}.
   * 
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
***************
*** 75,79 ****
          
          try {
!             new FusedView(null);
              fail("Expecting: "+IllegalArgumentException.class);
          } catch(IllegalArgumentException ex) {
--- 75,79 ----
          
          try {
!             new ReadOnlyFusedView(null);
              fail("Expecting: "+IllegalArgumentException.class);
          } catch(IllegalArgumentException ex) {
***************
*** 82,86 ****
          
          try {
!             new FusedView(new AbstractBTree[]{});
              fail("Expecting: "+IllegalArgumentException.class);
          } catch(IllegalArgumentException ex) {
--- 82,86 ----
          
          try {
!             new ReadOnlyFusedView(new AbstractBTree[]{});
              fail("Expecting: "+IllegalArgumentException.class);
          } catch(IllegalArgumentException ex) {
***************
*** 89,93 ****
          
          try {
!             new FusedView(new AbstractBTree[]{btree1});
              fail("Expecting: "+IllegalArgumentException.class);
          } catch(IllegalArgumentException ex) {
--- 89,93 ----
          
          try {
!             new ReadOnlyFusedView(new AbstractBTree[]{btree1});
              fail("Expecting: "+IllegalArgumentException.class);
          } catch(IllegalArgumentException ex) {
***************
*** 96,100 ****
                  
          try {
!             new FusedView(new AbstractBTree[]{btree1,null});
              fail("Expecting: "+IllegalArgumentException.class);
          } catch(IllegalArgumentException ex) {
--- 96,100 ----
                  
          try {
!             new ReadOnlyFusedView(new AbstractBTree[]{btree1,null});
              fail("Expecting: "+IllegalArgumentException.class);
          } catch(IllegalArgumentException ex) {
***************
*** 103,107 ****
                  
          try {
!             new FusedView(new AbstractBTree[]{btree1,btree1});
              fail("Expecting: "+IllegalArgumentException.class);
          } catch(IllegalArgumentException ex) {
--- 103,107 ----
                  
          try {
!             new ReadOnlyFusedView(new AbstractBTree[]{btree1,btree1});
              fail("Expecting: "+IllegalArgumentException.class);
          } catch(IllegalArgumentException ex) {
***************
*** 109,113 ****
          }
                  
!         new FusedView(new AbstractBTree[]{btree1,btree2});
                  
      }
--- 109,113 ----
          }
                  
!         new ReadOnlyFusedView(new AbstractBTree[]{btree1,btree2});
                  
      }
***************
*** 143,147 ****
          BTree btree2 = getBTree(3);
  
!         FusedView view = new FusedView(new AbstractBTree[] { btree1, btree2 });
          
          assertEquals(0, btree1.rangeCount(null, null));
--- 143,147 ----
          BTree btree2 = getBTree(3);
  
!         ReadOnlyFusedView view = new ReadOnlyFusedView(new AbstractBTree[] { btree1, btree2 });
          
          assertEquals(0, btree1.rangeCount(null, null));

[cweb-CVS] bigdata/src/test/com/bigdata/isolation TestIsolatedBTree.java, 1.2, 1.3 TestUnisolatedBTree.java, 1.5, 1.6

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/test/com/bigdata/isolation
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv5433/src/test/com/bigdata/isolation

Modified Files:
	TestIsolatedBTree.java TestUnisolatedBTree.java 
Log Message:
Continued minor refactoring in line with model updates.

Index: TestUnisolatedBTree.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/isolation/TestUnisolatedBTree.java,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** TestUnisolatedBTree.java	17 Feb 2007 21:34:21 -0000	1.5
--- TestUnisolatedBTree.java	11 Mar 2007 11:42:49 -0000	1.6
***************
*** 50,54 ****
  import com.bigdata.objndx.AbstractBTree;
  import com.bigdata.objndx.AbstractBTreeTestCase;
! import com.bigdata.objndx.BTreeMetadata;
  import com.bigdata.objndx.IBatchOp;
  import com.bigdata.objndx.IRangeQuery;
--- 50,54 ----
  import com.bigdata.objndx.AbstractBTree;
  import com.bigdata.objndx.AbstractBTreeTestCase;
! import com.bigdata.objndx.BTree;
  import com.bigdata.objndx.IBatchOp;
  import com.bigdata.objndx.IRangeQuery;
***************
*** 106,110 ****
              final long addr = btree.write();
              
!             btree = (UnisolatedBTree)BTreeMetadata.load(store, addr);
  
              assertTrue(store==btree.getStore());
--- 106,110 ----
              final long addr = btree.write();
              
!             btree = (UnisolatedBTree)BTree.load(store, addr);
  
              assertTrue(store==btree.getStore());
***************
*** 124,128 ****
              final long addr = btree.write();
  
!             btree = (UnisolatedBTree)BTreeMetadata.load(store, addr);
  
              assertTrue(store == btree.getStore());
--- 124,128 ----
              final long addr = btree.write();
  
!             btree = (UnisolatedBTree)BTree.load(store, addr);
  
              assertTrue(store == btree.getStore());
***************
*** 412,416 ****
          final long addr1 = btree.write();
          
!         btree = (UnisolatedBTree)BTreeMetadata.load(store, addr1);
          
          assertSameIterator(new Object[]{v3,v5,v7},btree.entryIterator());
--- 412,416 ----
          final long addr1 = btree.write();
          
!         btree = (UnisolatedBTree)BTree.load(store, addr1);
          
          assertSameIterator(new Object[]{v3,v5,v7},btree.entryIterator());
***************
*** 426,430 ****
          final long addr2 = btree.write();
          
!         btree = (UnisolatedBTree)BTreeMetadata.load(store, addr2);
          
          assertSameIterator(new Object[]{v3,v7},btree.entryIterator());
--- 426,430 ----
          final long addr2 = btree.write();
          
!         btree = (UnisolatedBTree)BTree.load(store, addr2);
          
          assertSameIterator(new Object[]{v3,v7},btree.entryIterator());
***************
*** 440,444 ****
          final long addr3 = btree.write();
          
!         btree = (UnisolatedBTree)BTreeMetadata.load(store, addr3);
          
          assertSameIterator(new Object[]{v3,v7a},btree.entryIterator());
--- 440,444 ----
          final long addr3 = btree.write();
          
!         btree = (UnisolatedBTree)BTree.load(store, addr3);
          
          assertSameIterator(new Object[]{v3,v7a},btree.entryIterator());
***************
*** 455,459 ****
          final long addr4 = btree.write();
          
!         btree = (UnisolatedBTree)BTreeMetadata.load(store, addr4);
          
          assertSameIterator(new Object[]{v3,v5a,v7a},btree.entryIterator());
--- 455,459 ----
          final long addr4 = btree.write();
          
!         btree = (UnisolatedBTree)BTree.load(store, addr4);
          
          assertSameIterator(new Object[]{v3,v5a,v7a},btree.entryIterator());

Index: TestIsolatedBTree.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/isolation/TestIsolatedBTree.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** TestIsolatedBTree.java	19 Feb 2007 19:00:23 -0000	1.2
--- TestIsolatedBTree.java	11 Mar 2007 11:42:49 -0000	1.3
***************
*** 50,53 ****
--- 50,54 ----
  import com.bigdata.journal.TestTx;
  import com.bigdata.objndx.AbstractBTreeTestCase;
+ import com.bigdata.objndx.BTree;
  import com.bigdata.objndx.BTreeMetadata;
  import com.bigdata.objndx.IBatchOp;
***************
*** 134,138 ****
              final long addr = btree.write();
              
!             btree = new IsolatedBTree(store,BTreeMetadata.read(store, addr), src);
  
              assertTrue(store==btree.getStore());
--- 135,139 ----
              final long addr = btree.write();
              
!             btree = (IsolatedBTree)BTree.load(store,addr);
  
              assertTrue(store==btree.getStore());
***************
*** 154,158 ****
              final long addr = btree.write();
  
!             btree = new IsolatedBTree(store, BTreeMetadata.read(store, addr), src);
  
              assertTrue(store == btree.getStore());
--- 155,159 ----
              final long addr = btree.write();
  
!             btree = (IsolatedBTree)BTree.load(store,addr);
  
              assertTrue(store == btree.getStore());

[cweb-CVS] bigdata/src/java/com/bigdata/scaleup ResourceState.java, NONE, 1.1 IPartitionTask.java, NONE, 1.1 JournalMetadata.java, NONE, 1.1 AbstractPartitionTask.java, NONE, 1.1 IResourceMetadata.java, NONE, 1.1 PartitionedIndex.java, 1.7, 1.8 SegmentMetadata.java, 1.2, 1.3 SlaveJournal.java, 1.2, 1.3 PartitionMetadata.java, 1.2, 1.3 PartitionedJournal.java, 1.12, 1.13 IndexSegmentLifeCycleEnum.java, 1.1, NONE

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv15000/src/java/com/bigdata/scaleup

Modified Files:
	PartitionedIndex.java SegmentMetadata.java SlaveJournal.java 
	PartitionMetadata.java PartitionedJournal.java 
Added Files:
	ResourceState.java IPartitionTask.java JournalMetadata.java 
	AbstractPartitionTask.java IResourceMetadata.java 
Removed Files:
	IndexSegmentLifeCycleEnum.java 
Log Message:
Updated the UML model and added a ZIP containing an HTML presentation of the model.
Working on partitioned index support.

--- NEW FILE: IPartitionTask.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
package com.bigdata.scaleup;

import com.bigdata.objndx.IndexSegment;

/**
 * Interface for a scheduleable task that produces one or more
 * {@link IndexSegment}s, updates the {@link MetadataIndex} to reflect the
 * existence of the new {@link IndexSegment}s and notifies existing views
 * with a depedency on the source(s) that they must switch over to the new
 * {@link IndexSegment}s.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public interface IPartitionTask extends
        java.util.concurrent.Callable<Object> {
    
    /**
     * Run the task.
     * 
     * @return No return semantics are defined.
     * 
     * @throws Exception
     *             The exception thrown by the task.
     */
    public Object call() throws Exception;

}
--- IndexSegmentLifeCycleEnum.java DELETED ---

--- NEW FILE: JournalMetadata.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
package com.bigdata.scaleup;

import java.io.File;

import com.bigdata.journal.Journal;

/**
 * Metadata required to locate a {@link Journal} resource.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 * 
 * @todo make this persistence capable by modifying the value serializer to
 * use the {@link IResourceMetadata} interface.
 */
public class JournalMetadata implements IResourceMetadata {

    protected final String filename;
    protected final ResourceState state;
    
    public File getFile() {
        return new File(filename);
    }

    /**
     * Always returns ZERO (0L) since we can not accurately estimate the #of
     * bytes on the journal dedicated to a given partition of a named index.
     */
    public long size() {
        return 0L;
    }

    public ResourceState state() {
        return state;
    }

    public JournalMetadata(Journal journal, ResourceState state) {
        
        if(journal.getFile()==null) {
            
            throw new IllegalArgumentException("Journal is not persistent.");
            
        }
        
        this.filename = journal.getFile().toString();

        this.state = state;
        
    }
    
}
--- NEW FILE: IResourceMetadata.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
package com.bigdata.scaleup;

import java.io.File;

import com.bigdata.journal.Journal;
import com.bigdata.objndx.IndexSegment;

/**
 * Interface for metadata about a {@link Journal} or {@link IndexSegment}.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public interface IResourceMetadata {

    /**
     * The store file.
     */
    public File getFile();
    
    /**
     * The #of bytes in the store file.
     */
    public long size();

    /**
     * The life cycle state of that store file.
     */
    public ResourceState state();

//    public int hashCode();
//    
//    public boolean equals(IResourceMetadata o);

}

Index: SlaveJournal.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/SlaveJournal.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** SlaveJournal.java	17 Feb 2007 21:34:21 -0000	1.2
--- SlaveJournal.java	8 Mar 2007 18:14:06 -0000	1.3
***************
*** 47,50 ****
--- 47,51 ----
  import java.util.Properties;
  
+ import com.bigdata.isolation.UnisolatedBTree;
  import com.bigdata.journal.IJournal;
  import com.bigdata.journal.Journal;
***************
*** 83,86 ****
--- 84,93 ----
      private final PartitionedJournal master;
      
+     protected PartitionedJournal getMaster() {
+ 
+         return master;
+         
+     }
+     
      public SlaveJournal(PartitionedJournal master,Properties properties) {
          
***************
*** 157,179 ****
  
      }
      
      /**
!      * Registers and returns {@link PartitionedIndex} under the given name
!      * and assigns the supplied {@link BTree} to absorb writes for that
       * {@link PartitionedIndex}.
       * <p>
!      * A {@link MetadataIndex} is also registered under the given name and
!      * an initial partition for that index is created using the separator
!      * key <code>new byte[]{}</code>. The partition will initially
!      * consist of zero {@link IndexSegment}s.
       * <p>
!      * Note: The returned object is invalid once the
!      * {@link PartitionedJournal} {@link PartitionedJournal#overflow()}s.
       * 
       * @todo use a prototype model so that the registered btree type is
!      *       preserved? (Only the metadata extensions are preserved right
!      *       now).  One way to do this is by putting the constructor on the
!      *       metadata object.  Another is to make the btree Serializable and
!      *       then just declare everything else as transient.
       */
      public IIndex registerIndex(String name, IIndex btree) {
--- 164,212 ----
  
      }
+ 
+     /**
+      * Registers and returns a {@link PartitionedIndex} under the given name and
+      * assigns an {@link UnisolatedBTree} to absorb writes for that
+      * {@link PartitionedIndex}. The resulting index will support transactional
+      * isolation.
+      * <p>
+      * A {@link MetadataIndex} is also registered under the given name and an
+      * initial partition for that index is created using the separator key
+      * <code>new byte[]{}</code>. The partition will initially consist of
+      * zero {@link IndexSegment}s.
+      * <p>
+      * Note: The returned object is invalid once the {@link PartitionedJournal}
+      * {@link PartitionedJournal#overflow()}s.
+      * <p>
+      * Note: You MUST {@link #commit()} before the registered index will be
+      * either restart-safe or visible to new transactions.
+      */
+     public IIndex registerIndex(String name) {
+         
+         return registerIndex(name, new UnisolatedBTree(this));
+         
+     }
      
      /**
!      * Registers and returns a {@link PartitionedIndex} under the given name and
!      * assigns the supplied {@link IIndex} to absorb writes for that
       * {@link PartitionedIndex}.
       * <p>
!      * A {@link MetadataIndex} is also registered under the given name and an
!      * initial partition for that index is created using the separator key
!      * <code>new byte[]{}</code>. The partition will initially consist of
!      * zero {@link IndexSegment}s.
       * <p>
!      * Note: The returned object is invalid once the {@link PartitionedJournal}
!      * {@link PartitionedJournal#overflow()}s.
!      * <p>
!      * Note: You MUST {@link #commit()} before the registered index will be
!      * either restart-safe or visible to new transactions.
       * 
       * @todo use a prototype model so that the registered btree type is
!      *       preserved? (Only the metadata extensions are preserved right now).
!      *       One way to do this is by putting the constructor on the metadata
!      *       object. Another is to make the btree Serializable and then just
!      *       declare everything else as transient.
       */
      public IIndex registerIndex(String name, IIndex btree) {

Index: SegmentMetadata.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/SegmentMetadata.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** SegmentMetadata.java	6 Mar 2007 20:38:06 -0000	1.2
--- SegmentMetadata.java	8 Mar 2007 18:14:05 -0000	1.3
***************
*** 1,45 ****
  /**
  
!  The Notice below must appear in each file of the Source Code of any
!  copy you distribute of the Licensed Product.  Contributors to any
!  Modifications may add their own copyright notices to identify their
!  own contributions.
  
!  License:
  
!  The contents of this file are subject to the CognitiveWeb Open Source
!  License Version 1.1 (the License).  You may not copy or use this file,
!  in either source code or executable form, except in compliance with
!  the License.  You may obtain a copy of the License from
  
!  http://www.CognitiveWeb.org/legal/license/
  
!  Software distributed under the License is distributed on an AS IS
!  basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
!  the License for the specific language governing rights and limitations
!  under the License.
  
!  Copyrights:
  
!  Portions created by or assigned to CognitiveWeb are Copyright
!  (c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
!  information for CognitiveWeb is available at
  
!  http://www.CognitiveWeb.org
  
!  Portions Copyright (c) 2002-2003 Bryan Thompson.
  
!  Acknowledgements:
  
!  Special thanks to the developers of the Jabber Open Source License 1.0
!  (JOSL), from which this License was derived.  This License contains
!  terms that differ from JOSL.
  
!  Special thanks to the CognitiveWeb Open Source Contributors for their
!  suggestions and support of the Cognitive Web.
  
!  Modifications:
  
!  */
  package com.bigdata.scaleup;
  
--- 1,45 ----
  /**
  
! The Notice below must appear in each file of the Source Code of any
! copy you distribute of the Licensed Product.  Contributors to any
! Modifications may add their own copyright notices to identify their
! own contributions.
  
! License:
  
! The contents of this file are subject to the CognitiveWeb Open Source
! License Version 1.1 (the License).  You may not copy or use this file,
! in either source code or executable form, except in compliance with
! the License.  You may obtain a copy of the License from
  
!   http://www.CognitiveWeb.org/legal/license/
  
! Software distributed under the License is distributed on an AS IS
! basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
! the License for the specific language governing rights and limitations
! under the License.
  
! Copyrights:
  
! Portions created by or assigned to CognitiveWeb are Copyright
! (c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
! information for CognitiveWeb is available at
  
!   http://www.CognitiveWeb.org
  
! Portions Copyright (c) 2002-2003 Bryan Thompson.
  
! Acknowledgements:
  
! Special thanks to the developers of the Jabber Open Source License 1.0
! (JOSL), from which this License was derived.  This License contains
! terms that differ from JOSL.
  
! Special thanks to the CognitiveWeb Open Source Contributors for their
! suggestions and support of the Cognitive Web.
  
! Modifications:
  
! */
  package com.bigdata.scaleup;
  
***************
*** 47,51 ****
  
  import com.bigdata.objndx.IndexSegment;
- import com.bigdata.scaleup.PartitionedJournal.IViewMetadata;
  
  /**
--- 47,50 ----
***************
*** 55,59 ****
   * @version $Id$
   */
! public class SegmentMetadata implements IViewMetadata {
  
      /**
--- 54,58 ----
   * @version $Id$
   */
! public class SegmentMetadata implements IResourceMetadata {
  
      /**
***************
*** 70,76 ****
       * The life-cycle state for that {@link IndexSegment}.
       */
!     final public IndexSegmentLifeCycleEnum state;
      
!     public SegmentMetadata(String filename,long nbytes,IndexSegmentLifeCycleEnum state) {
  
          this.filename = filename;
--- 69,75 ----
       * The life-cycle state for that {@link IndexSegment}.
       */
!     final public ResourceState state;
      
!     public SegmentMetadata(String filename,long nbytes,ResourceState state) {
  
          this.filename = filename;
***************
*** 103,107 ****
      }
  
!     public IndexSegmentLifeCycleEnum status() {
          return state;
      }
--- 102,106 ----
      }
  
!     public ResourceState state() {
          return state;
      }

Index: PartitionedJournal.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/PartitionedJournal.java,v
retrieving revision 1.12
retrieving revision 1.13
diff -C2 -d -r1.12 -r1.13
*** PartitionedJournal.java	6 Mar 2007 20:38:06 -0000	1.12
--- PartitionedJournal.java	8 Mar 2007 18:14:06 -0000	1.13
***************
*** 53,59 ****
  import java.util.Iterator;
  import java.util.Properties;
- import java.util.concurrent.Executors;
  
! import com.bigdata.isolation.IIsolatableIndex;
  import com.bigdata.isolation.UnisolatedBTree;
  import com.bigdata.isolation.Value;
--- 53,59 ----
  import java.util.Iterator;
  import java.util.Properties;
[...967 lines suppressed...]
+             
+         }
+ 
+         return seg;
+ 
      }
+ 
+     /**
+      * The maximum #of index segments that will be held open without a hard
+      * reference existing for that index segment in the application.
+      */
+     final int INDEX_SEGMENT_LRU_CAPACITY = 5;
+     
+     /**
+      * A cache for recently used index segments designed to prevent their being
+      * swept by the VM between uses.
+      */
+     private final WeakValueCache<String/*filename*/, IndexSegment> resourceCache;
      
  }

Index: PartitionMetadata.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/PartitionMetadata.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** PartitionMetadata.java	9 Feb 2007 18:56:59 -0000	1.2
--- PartitionMetadata.java	8 Mar 2007 18:14:06 -0000	1.3
***************
*** 116,120 ****
          for (int i = 0; i < segs.length; i++) {
  
!             if (segs[i].state == IndexSegmentLifeCycleEnum.LIVE)
                  count++;
  
--- 116,120 ----
          for (int i = 0; i < segs.length; i++) {
  
!             if (segs[i].state == ResourceState.Live)
                  count++;
  
***************
*** 138,142 ****
          for (int i = 0; i < segs.length; i++) {
  
!             if (segs[i].state == IndexSegmentLifeCycleEnum.LIVE) {
  
                  files[k++] = segs[i].filename;
--- 138,142 ----
          for (int i = 0; i < segs.length; i++) {
  
!             if (segs[i].state == ResourceState.Live) {
  
                  files[k++] = segs[i].filename;
***************
*** 333,337 ****
                      long nbytes = is.readLong();
  
!                     IndexSegmentLifeCycleEnum state = IndexSegmentLifeCycleEnum
                              .valueOf(is.readInt());
  
--- 333,337 ----
                      long nbytes = is.readLong();
  
!                     ResourceState state = ResourceState
                              .valueOf(is.readInt());
  

--- NEW FILE: AbstractPartitionTask.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
package com.bigdata.scaleup;

import java.io.File;
import java.util.concurrent.Executors;

import com.bigdata.isolation.IIsolatableIndex;
import com.bigdata.isolation.UnisolatedBTree;
import com.bigdata.isolation.Value;
import com.bigdata.objndx.AbstractBTree;
import com.bigdata.objndx.IValueSerializer;
import com.bigdata.objndx.IndexSegment;
import com.bigdata.objndx.IndexSegmentBuilder;
import com.bigdata.objndx.IndexSegmentMerger;
import com.bigdata.objndx.IndexSegmentMetadata;
import com.bigdata.objndx.RecordCompressor;
import com.bigdata.objndx.IndexSegmentMerger.MergedEntryIterator;
import com.bigdata.objndx.IndexSegmentMerger.MergedLeafIterator;
import com.bigdata.rawstore.Bytes;

/**
 * Abstract base class for tasks that build {@link IndexSegment}(s).
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 * 
 * @todo write test suite for executing partition task schedules.
 * 
 * @todo add a
 *       {@link Executors#newSingleThreadExecutor(java.util.concurrent.ThreadFactory)}
 *       that will be used to run these tasks and modify the shutdown logic
 *       for the master to also shutdown that thread.
 * 
 * @todo add result to persistent schedule outcome so that this is restart
 *       safe.
 * 
 * @todo once the {@link IndexSegment} is ready the metadata index needs to
 *       be updated to reflect that the indexsegment is live and the views
 *       that rely on the partition on the old journal need to be
 *       invalidated so new views utilize the new indexsegment rather than
 *       the data on the old journal.
 * 
 * @todo the old journal is not available for release until all partitions
 *       for all indices have been evicted. we need to track that in a
 *       restart safe manner.
 * 
 * @todo parameterize useChecksum, recordCompressor.
 * 
 * @todo assiging sequential segment identifiers may impose an unnecessary
 *       message overhead since we can just use the temporary file mechanism
 *       and the inspect the {@link IndexSegmentMetadata} to learn more
 *       about a given store file.
 * 
 * @todo try performance with and without checksums and with and without
 *       record compression.
 */
abstract public class AbstractPartitionTask implements
        IPartitionTask {

    protected final PartitionedJournal master;
    /**
     * Branching factor used for generated {@link IndexSegment}(s). 
     */
    protected final int branchingFactor;
    protected final double errorRate;
    protected final String name;
    protected final int partId;
    protected final byte[] fromKey;
    protected final byte[] toKey;

    /**
     * Branching factor used for temporary file(s).
     */
    protected final int tmpFileBranchingFactor = Bytes.kilobyte32*4;

    /**
     * When true, pre-record checksum are generated for the output
     * {@link IndexSegment}.
     */
    protected final boolean useChecksum = false;
    
    /**
     * When non-null, a {@link RecordCompressor} will be applied to the
     * output {@link IndexSegment}.
     */
    protected final RecordCompressor recordCompressor = null;
    
    /**
     * The serializer used by all {@link IIsolatableIndex}s.
     */
    static protected final IValueSerializer valSer = Value.Serializer.INSTANCE;
    
    /**
     * 
     * @param master
     *            The master.
     * @param name
     *            The index name.
     * @param branchingFactor
     *            The branching factor to be used on the new
     *            {@link IndexSegment}(s).
     * @param errorRate
     *            The error rate for the bloom filter for the new
     *            {@link IndexSegment}(s) -or- zero(0d) if no bloom filter
     *            is desired.
     * @param partId
     *            The unique partition identifier for the partition.
     * @param fromKey
     *            The first key that would be accepted into that partition
     *            (aka the separator key for that partition).<br>
     *            Note: Failure to use the actual separatorKeys for the
     *            partition will result in changing the separator key in a
     *            manner that is incoherent! The change arises since the
     *            updates are stored in the metadata index based on the
     *            supplied <i>fromKey</i>.
     * @param toKey
     *            The first key that would NOT be accepted into that
     *            partition (aka the separator key for the right sibling
     *            partition and <code>null</code> iff there is no right
     *            sibling).
     * 
     * @todo It is an error to supply a <i>fromKey</i> that is not also the
     *       separatorKey for the partition, however the code does not
     *       detect this error.
     * 
     * @todo It is possible for the partition to be redefined (joined with a
     *       sibling or split into two partitions). In this case any already
     *       scheduled operations MUST abort and new operations with the
     *       correct separator keys must be scheduled.
     */
    public AbstractPartitionTask(PartitionedJournal master, String name,
            int branchingFactor, double errorRate, int partId,
            byte[] fromKey, byte[] toKey) {

        this.master = master;
        this.branchingFactor = branchingFactor;
        this.errorRate = errorRate;
        this.name = name;
        this.partId = partId;
        this.fromKey = fromKey;
        this.toKey = toKey;

    }

    /**
     * 
     * @todo update the metadata index (synchronized) and invalidate
     *       existing partitioned index views that depend on the source
     *       index.
     *       
     * @param resources
     */
    protected void updatePartition(IResourceMetadata[] resources) {

        throw new UnsupportedOperationException();
        
    }
    
    /**
     * Task builds an {@link IndexSegment} for a partition from data on a
     * historical read-only {@link SlaveJournal}. When the {@link IndexSegment}
     * is ready the metadata index is updated to make the segment "live" and
     * existing views are notified that the source data on the partition must be
     * invalidated in favor of the new {@link IndexSegment}.
     * 
     * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
     * @version $Id$
     * 
     * @todo this must use a merge rule that knows about deletion markers and is
     *       only usable when the input is an {@link UnisolatedBTree}. The
     *       output {@link IndexSegment} will contain timestamps and deletion
     *       markers and support isolation.
     */
    public static class BuildTask extends AbstractPartitionTask {

        private final IResourceMetadata src;
        private final int segId;
        
        /**
         * 
         * @param src
         *            The source for the build operation. Only those entries in
         *            the described key range will be used.
         * @param segId
         *            The output segment identifier.
         */
        public BuildTask(PartitionedJournal master, String name,
                int branchingFactor, double errorRate, int partId,
                byte[] fromKey, byte[] toKey, IResourceMetadata src, int segId) {
            
            super(master,name,branchingFactor,errorRate,partId,fromKey,toKey);
            
            this.src = src;
            
            this.segId = segId;
            
        }
        
        /**
         * Build an {@link IndexSegment} from a key range corresponding to an
         * index partition.
         * 
         * @todo this needs to use the rangeIterator on the root of the btree so
         *       that the {@link Value}s will be visible, including both
         *       version counters and deletion markers.
         */
        public Object call() throws Exception {

            AbstractBTree src = master.getIndex(name,this.src);
            
            File outFile = master.getSegmentFile(name, partId, segId);

            new IndexSegmentBuilder(outFile, master.tmpDir, src.rangeCount(
                    fromKey, toKey), src.rangeIterator(fromKey, toKey),
                    branchingFactor, valSer, useChecksum, recordCompressor,
                    errorRate);

            IResourceMetadata[] resources = new SegmentMetadata[] { new SegmentMetadata(
                    "" + outFile, outFile.length(),
                    ResourceState.New) };

            updatePartition(resources);
            
            return null;
            
        }
        
    }

    /**
     * Abstract base class for compacting merge tasks.
     * 
     * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
     * @version $Id$
     */
    abstract static class AbstractMergeTask extends AbstractPartitionTask {

        protected final int segId;
        protected final boolean fullCompactingMerge;

        /**
         * A compacting merge of two or more resources.
         * 
         * @param segId
         *            The output segment identifier.
         * 
         * @param fullCompactingMerge
         *            True iff this will be a full compacting merge.
         */
        protected AbstractMergeTask(PartitionedJournal master, String name,
                int branchingFactor, double errorRate, int partId,
                byte[] fromKey, byte[] toKey, int segId,
                boolean fullCompactingMerge) {
            
            super(master, name, branchingFactor, errorRate, partId, fromKey,
                    toKey);
            
            this.segId = segId;
            
            this.fullCompactingMerge = fullCompactingMerge;
            
        }
        
        /**
         * Compacting merge of two or more resources - deletion markers are
         * preserved iff {@link #fullCompactingMerge} is <code>false</code>.
         * 
         * @todo make sure that deletion markers get removed from the view as
         *       part of the index segment merger logic (choose the most recent
         *       write for each key, but if it is a delete then remove the entry
         *       from the merge so that it does not appear in the temporary file
         *       that is the input to the {@link IndexSegmentBuilder}).
         */
        public Object call() throws Exception {
            
            // tmp file for the merge process.
            File tmpFile = File.createTempFile("merge", ".tmp", master.tmpDir);

            tmpFile.deleteOnExit();

            // output file for the merged segment.
            File outFile = master.getSegmentFile(name, partId, segId);

            IResourceMetadata[] resources = getResources();
            
            AbstractBTree[] srcs = new AbstractBTree[resources.length];
            
            for(int i=0; i<srcs.length; i++) {

                // open the index - closed by the weak value cache.
                srcs[i] = master.getIndex(name, resources[i]);
                
            }
            
            // merge the data from the btree on the slave and the index
            // segment.
            MergedLeafIterator mergeItr = new IndexSegmentMerger(
                    tmpFileBranchingFactor, srcs).merge();

            // build the merged index segment.
            new IndexSegmentBuilder(outFile, null, mergeItr.nentries,
                    new MergedEntryIterator(mergeItr), branchingFactor, valSer,
                    useChecksum, recordCompressor, errorRate);

            // close the merged leaf iterator (and release its buffer/file).
            // @todo this should be automatic when the iterator is exhausted but
            // I am not seeing that.
            mergeItr.close();

            /*
             * @todo Update the metadata index for this partition. This needs to
             * be an atomic operation so we have to synchronize on the metadata
             * index or simply move the operation into the MetadataIndex API and
             * let it encapsulate the problem.
             * 
             * We mark the earlier index segment as "Dead" for this partition
             * since it has been replaced by the merged result.
             * 
             * Note: it is a good idea to wait until you have opened the merged
             * index segment, and even until it has begun to serve data, before
             * deleting the old index segment that was an input to that merge!
             * The code currently waits until the next time a compacting merge
             * is performed for the partition and then deletes the Dead index
             * segment.
             */

            final MetadataIndex mdi = master.getSlave().getMetadataIndex(name);
            
            final PartitionMetadata pmd = mdi.get(fromKey);

            // #of live segments for this partition.
            final int liveCount = pmd.getLiveCount();
            
            // @todo assuming compacting merge each time.
            if(liveCount!=1) throw new UnsupportedOperationException();
            
            // new segment definitions.
            final SegmentMetadata[] newSegs = new SegmentMetadata[2];

            // assume only the last segment is live.
            final SegmentMetadata oldSeg = pmd.segs[pmd.segs.length-1];
            
            newSegs[0] = new SegmentMetadata(oldSeg.filename, oldSeg.nbytes,
                    ResourceState.Dead);

            newSegs[1] = new SegmentMetadata(outFile.toString(), outFile.length(),
                    ResourceState.Live);
            
            mdi.put(fromKey, new PartitionMetadata(0, segId + 1, newSegs));
            
            return null;
            
        }
    
        /**
         * The resources that comprise the view in reverse timestamp order
         * (increasing age).
         */
        abstract protected IResourceMetadata[] getResources();
        
    }
    
    /**
     * Task builds an {@link IndexSegment} using a compacting merge of two
     * resources having data for the same partition. Common use cases are a
     * journal and an index segment or two index segments. Only the most recent
     * writes will be retained for any given key (duplicate suppression). Since
     * this task does NOT provide a full compacting merge (it may be applied to
     * a subset of the resources required to materialize a view for a commit
     * time), deletion markers MAY be present in the resulting
     * {@link IndexSegment}.
     * 
     * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
     * @version $Id$
     */
    public static class MergeTask extends AbstractMergeTask {

        private final IResourceMetadata[] resources;

        /**
         * A compacting merge of two or more resources.
         * 
         * @param srcs
         *            The source resources, which must be given in reverse
         *            timestamp order (increasing age).
         * @param segId
         *            The output segment identifier.
         */
        public MergeTask(PartitionedJournal master, String name,
                int branchingFactor, double errorRate, int partId,
                byte[] fromKey, byte[] toKey, IResourceMetadata[] resources,
                int segId) {

            super(master, name, branchingFactor, errorRate, partId, fromKey,
                    toKey, segId, false);
            
            this.resources = resources;
            
        }

        protected IResourceMetadata[] getResources() {
            
            return resources;
            
        }
        
    }
    
    /**
     * Task builds an {@link IndexSegment} using a full compacting merge of all
     * resources having data for the same partition as of a specific commit
     * time. Only the most recent writes will be retained for any given key
     * (duplicate suppression). Deletion markers wil NOT be present in the
     * resulting {@link IndexSegment}.
     * <p>
     * Note: A full compacting merge does not necessarily result in only a
     * single {@link IndexSegment} for a partition since (a) it may be requested
     * for a historical commit time; and (b) subsequent writes may have
     * occurred, resulting in additional data on either the journal and/or new
     * {@link IndexSegment} that do not participate in the merge.
     * <p>
     * Note: in order to perform a full compacting merge the task MUST read from
     * all resources required to provide a consistent view for a partition
     * otherwise lost deletes may result.
     * 
     * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
     * @version $Id$
     */
    public static class FullMergeTask extends AbstractMergeTask {

        private final long commitTime;

        /**
         * A full compacting merge of an index partition.
         * 
         * @param segId
         *            The output segment identifier.
         * 
         * @param commitTime
         *            The commit time for the view that is the input to the
         *            merge operation.
         */
        public FullMergeTask(PartitionedJournal master, String name,
                int branchingFactor, double errorRate, int partId,
                byte[] fromKey, byte[] toKey, long commitTime, int segId) {

            super(master, name, branchingFactor, errorRate, partId, fromKey,
                    toKey, segId, true);

            this.commitTime = commitTime;

        }

        protected IResourceMetadata[] getResources() {
            
            final PartitionedIndex oldIndex = ((PartitionedIndex) master
                    .getIndex(name, commitTime));
            
            final IResourceMetadata[] resources = oldIndex.getResources(fromKey);
            
            return resources;

        }
        
    }

//    /**
//     * Task builds an {@link IndexSegment} by joining an existing
//     * {@link IndexSegment} for a partition with an existing
//     * {@link IndexSegment} for either the left or right sibling of that
//     * partition.
//     * 
//     * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
//     * @version $Id$
//     */
//    public static class JoinTask implements IPartitionTask {
//        
//    }
//    /**
//     * Task splits an {@link IndexSegment} into two new {@link IndexSegment}s.
//     * This task is executed when a partition is split in order to breakdown the
//     * {@link IndexSegment} for that partition into data for the partition and
//     * its new left/right sibling.
//     * 
//     * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
//     * @version $Id$
//     */
//    public static class SplitTask implements IPartitionTask {
//        
//    }
    
//    /**
//     * Update the metadata index to reflect the split of one index segment
//     * into two index segments.
//     * 
//     * @param separatorKey
//     *            Requests greater than or equal to the separatorKey (and
//     *            less than the next largest separatorKey in the metadata
//     *            index) are directed into seg2. Requests less than the
//     *            separatorKey (and greated than any proceeding separatorKey
//     *            in the metadata index) are directed into seg1.
//     * @param seg1
//     *            The metadata for the index segment that was split.
//     * @param seg2
//     *            The metadata for the right sibling of the split index
//     *            segment in terms of the key range of the distributed
//     *            index.
//     */
//    public void split(Object separatorKey, PartitionMetadata md1, PartitionMetadata md2) {
//        
//    }
//
//    /**
//     * @todo join of index segment with left or right sibling. unlike the
//     *       nodes of a btree we merge nodes whenever a segment goes under
//     *       capacity rather than trying to redistribute part of the key
//     *       range from one index segment to another.
//     */
//    public void join() {
//        
//    }

}

Index: PartitionedIndex.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/PartitionedIndex.java,v
retrieving revision 1.7
retrieving revision 1.8
diff -C2 -d -r1.7 -r1.8
*** PartitionedIndex.java	6 Mar 2007 20:38:06 -0000	1.7
--- PartitionedIndex.java	8 Mar 2007 18:14:05 -0000	1.8
***************
*** 48,52 ****
  package com.bigdata.scaleup;
  
- import java.io.File;
  import java.util.HashMap;
  import java.util.Iterator;
--- 48,51 ----
***************
*** 67,73 ****
  import com.bigdata.objndx.IIndex;
  import com.bigdata.objndx.IndexSegment;
- import com.bigdata.objndx.IndexSegmentFileStore;
- import com.bigdata.scaleup.PartitionedJournal.IViewMetadata;
- import com.bigdata.scaleup.PartitionedJournal.JournalMetadata;
  
  /**
--- 66,69 ----
***************
*** 114,120 ****
--- 110,132 ----
      /**
       * A cache of the fused views for in use partitions.
+      * 
+      * @todo reconcile this with
+      *       {@link PartitionedJournal#getIndex(String, IResourceMetadata)}
+      *       which provides a weak value cache for index segments.
       */
      private final Map<Integer,FusedView> views = new HashMap<Integer,FusedView>();
  
+     public PartitionedJournal getMaster() {
+         
+         return getSlave().getMaster();
+         
+     }
+     
+     public SlaveJournal getSlave() {
+         
+         return (SlaveJournal)getBTree().getStore();
+         
+     }
+     
      /**
       * The mutable {@link BTree} used to absorb writes.
***************
*** 135,160 ****
       * 
       * @return The live {@link IndexSegment}s for that partition.
-      * 
-      * @todo there needs to be synchronization with the
-      *       {@link PartitionedJournal} so that index segments may be closed out
-      *       as required but it would be very nice (or perhaps necessary) to be
-      *       able to perform a compacting merge while an {@link IndexSegment}
-      *       was open for reading.
       */
      protected IndexSegment[] openIndexSegments(PartitionMetadata pmd) {
          
!         String[] files = pmd.getLiveSegmentFiles();
          
!         IndexSegment[] segs = new IndexSegment[files.length];
          
!         for(int i=0; i<segs.length; i++) {
              
!             File file = new File(files[i]);
              
!             segs[i] = new IndexSegment(new IndexSegmentFileStore(file), btree
!                     .getNodeSerializer().getValueSerializer());
              
          }
          
          return segs;
  
--- 147,171 ----
       * 
       * @return The live {@link IndexSegment}s for that partition.
       */
      protected IndexSegment[] openIndexSegments(PartitionMetadata pmd) {
          
!         final int liveCount = pmd.getLiveCount();
          
!         IndexSegment[] segs = new IndexSegment[liveCount];
          
!         PartitionedJournal master = getMaster();
!         
!         int n = 0;
!         
!         for(int i=0; i<pmd.segs.length; i++) {
              
!             if(pmd.segs[i].state != ResourceState.Live) continue;
              
!             segs[n++] = (IndexSegment) master.getIndex(getName(), pmd.segs[i]);
              
          }
          
+         assert n == liveCount;
+         
          return segs;
  
***************
*** 263,270 ****
       * @todo reconcile with {@link #getView(byte[])}?
       */
!     protected IViewMetadata[] getResources(byte[] key) {
  
          JournalMetadata journalResource = new JournalMetadata((Journal) btree
!                 .getStore(), IndexSegmentLifeCycleEnum.LIVE);
  
          PartitionMetadata pmd = mdi.find(key);
--- 274,281 ----
       * @todo reconcile with {@link #getView(byte[])}?
       */
!     protected IResourceMetadata[] getResources(byte[] key) {
  
          JournalMetadata journalResource = new JournalMetadata((Journal) btree
!                 .getStore(), ResourceState.Live);
  
          PartitionMetadata pmd = mdi.find(key);
***************
*** 272,276 ****
          final int liveCount = pmd.getLiveCount();
  
!         IViewMetadata[] resources = new IViewMetadata[liveCount + 1];
  
          int n = 0;
--- 283,287 ----
          final int liveCount = pmd.getLiveCount();
  
!         IResourceMetadata[] resources = new IResourceMetadata[liveCount + 1];
  
          int n = 0;
***************
*** 282,286 ****
              SegmentMetadata seg = pmd.segs[i];
  
!             if (seg.state != IndexSegmentLifeCycleEnum.LIVE)
                  continue;
  
--- 293,297 ----
              SegmentMetadata seg = pmd.segs[i];
  
!             if (seg.state != ResourceState.Live)
                  continue;
  

--- NEW FILE: ResourceState.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
package com.bigdata.scaleup;

import com.bigdata.objndx.IndexSegment;

/**
 * Enumeration of life-cycle states for {@link IndexSegment}s in a
 * partition.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public enum ResourceState {

    New("New",0),
    Live("Live",1),
    Dead("Dead",2);
    
    final private String name;
    final private int id;
    
    ResourceState(String name,int id) {this.name = name;this.id = id;}
    
    public String toString() {return name;}
    
    public int valueOf() {return id;}
    
    static public ResourceState valueOf(int id) {
        switch(id) {
        case 0: return New;
        case 1: return Live;
        case 2: return Dead;
        default: throw new IllegalArgumentException("Unknown: code="+id);
        }
    }
    
}

[cweb-CVS] bigdata/src/java/com/bigdata/isolation UnisolatedIndexSegment.java, NONE, 1.1 IsolatableFusedView.java, 1.2, 1.3 IsolatedBTree.java, 1.6, 1.7 IIsolatableIndex.java, 1.1, 1.2 UnisolatedBTree.java, 1.7, 1.8

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/isolation
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv15000/src/java/com/bigdata/isolation

Modified Files:
	IsolatableFusedView.java IsolatedBTree.java 
	IIsolatableIndex.java UnisolatedBTree.java 
Added Files:
	UnisolatedIndexSegment.java 
Log Message:
Updated the UML model and added a ZIP containing an HTML presentation of the model.
Working on partitioned index support.

--- NEW FILE: UnisolatedIndexSegment.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Mar 7, 2007
 */

package com.bigdata.isolation;

import com.bigdata.isolation.UnisolatedBTree.DeletedEntryFilter;
import com.bigdata.objndx.AbstractBTree;
import com.bigdata.objndx.BatchContains;
import com.bigdata.objndx.BatchLookup;
import com.bigdata.objndx.IEntryIterator;
import com.bigdata.objndx.IndexSegment;
import com.bigdata.objndx.IndexSegmentExtensionMetadata;
import com.bigdata.objndx.IndexSegmentFileStore;

/**
 * <p>
 * A scalable read-only B+-Tree mapping variable length unsigned byte[] keys to
 * byte[] values that is capable of being isolated by a transaction (it
 * maintains version counters) and supports deletion markers. Application data
 * are transparently encapsulated in {@link IValue} objects which keep track of
 * version counters (in support of transactions) and deletion markers (in
 * support of both transactions and partitioned indices). Users of this class
 * will only see application values, not {@link IValue} objects.
 * </p>
 * 
 * @see UnisolatedBTree, which provides a mutable implementation with a similar
 *      contract.
 * 
 * @todo This class should either share code or tests cases with
 *       {@link UnisolatedBTree} (I just copied over the logic for non-mutation
 *       operations). There are no direct tests of this class at this time.
 * 
 * @todo define extension that stores the index name and uuid for the named
 *       index to which the segment belongs (add method to {@link AbstractBTree}
 *       to allow subclassing {@link IndexSegmentExtensionMetadata}).
 * 
 * @todo add a boolean flag to mark index segments that are the final result of
 *       a compacting merge. This will make it possible to reconstruct from the
 *       file system which index segments are part of the consistent state for a
 *       given restart time.
 * 
 * @todo consider caching the first/last key in support of both correct
 *       rejection of queries directed to the wrong index segment and managing
 *       the metadataMap for a distributed index.
 * 
 * @todo examine the format of the uuid. can we use part of it as the unique
 *       basis for one up identifiers within a parition?
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public class UnisolatedIndexSegment extends IndexSegment implements IIsolatableIndex {

    /**
     * 
     */
    public UnisolatedIndexSegment(IndexSegmentFileStore store) {
        super(store);
    }

//    /**
//     * This method breaks isolation to return the {@link Value} for a key.
//     * It is used by {@link IsolatedBTree#validate(UnisolatedBTree)} to test
//     * version counters when a key already exists in the global scope.
//     * 
//     * @todo make protected and refactor tests so that we do not need public
//     *       access to this method. there should be tests in this package
//     *       that examine the specific version counters that are assigned
//     *       such that we do not need to expose this method as public.
//     */
//    final public Value getValue(byte[] key) {
//
//        return (Value) super.lookup(key);
//
//    }

    /**
     * True iff the key does not exist or if it exists but is marked as
     * {@link IValue#isDeleted()}.
     * 
     * @param key
     *            The search key.
     * 
     * @return True iff there is an non-deleted entry for the search key.
     */
    public boolean contains(byte[] key) {

        if (key == null)
            throw new IllegalArgumentException();

        Value value = (Value) super.lookup(key);

        if (value == null || value.deleted)
            return false;

        return true;

    }

    /**
     * Return the {@link IValue#getValue()} associated with the key or
     * <code>null</code> if the key is not found or if the key was found
     * by the entry is flagged as {@link IValue#isDeleted()}.
     * 
     * @param key
     *            The search key.
     * 
     * @return The application value stored under that search key (may be
     *         null) or null if the key was not found or if they entry was
     *         marked as deleted.
     */
    public Object lookup(Object key) {

        if (key == null)
            throw new IllegalArgumentException();

        Value value = (Value) super.lookup(key);

        if (value == null || value.deleted)
            return null;

        return value.datum;

    }

    /**
     * Overriden to return <code>null</code> if the entry at that index is
     * deleted.
     */
    public Object valueAt(int index) {

        Value value = (Value) super.valueAt(index);

        if (value == null || value.deleted)
            return null;

        return value.datum;

    }

    /**
     * This method will include deleted entries in the key range in the
     * returned count.
     */
    public int rangeCount(byte[] fromKey, byte[] toKey) {

        return super.rangeCount(fromKey, toKey);

    }

    /**
     * Visits only the non-deleted entries in the key range.
     */
    public IEntryIterator rangeIterator(byte[] fromKey, byte[] toKey) {

        return root.rangeIterator(fromKey, toKey, DeletedEntryFilter.INSTANCE);

    }

    public IEntryIterator entryIterator() {

        return root.rangeIterator(null, null, DeletedEntryFilter.INSTANCE);

    }

    public void contains(BatchContains op) {

        op.apply(this);

    }

    public void lookup(BatchLookup op) {

        op.apply(this);

    }

}

Index: UnisolatedBTree.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/isolation/UnisolatedBTree.java,v
retrieving revision 1.7
retrieving revision 1.8
diff -C2 -d -r1.7 -r1.8
*** UnisolatedBTree.java	21 Feb 2007 20:17:22 -0000	1.7
--- UnisolatedBTree.java	8 Mar 2007 18:14:06 -0000	1.8
***************
*** 57,60 ****
--- 57,61 ----
  import com.bigdata.objndx.IEntryIterator;
  import com.bigdata.objndx.ISimpleBTree;
+ import com.bigdata.objndx.IndexSegment;
  import com.bigdata.objndx.EntryIterator.EntryFilter;
  import com.bigdata.rawstore.IRawStore;
***************
*** 82,112 ****
   * 
   * @see IsolatedBTree, a {@link BTree} that has been isolated by a transaction.
!  *
   * The following is an ad-hoc summary of the behavior of some methods exposed by
   * this class:
! <pre>
! 
! contains() - done.
! insert() - done.
! remove() - done.
! lookup() - done.
! 
! addAll() - ok as implemented.  values will be wrapped in {@link IValue} objects
! as they are inserted.  if the source is also an {@link UnisolatedBTree} then the
! application values will be inserted into this tree, not the {@link IValue} objects.
! 
! indexOf() - ok as implemented (counts deleted entries).
! keyAt() - ok as implemented, but will return keys for deleted entries.
! valueAt() - overriden to return null for a deleted entry.
! 
! rangeCount - ok as implemented (counts deleted entries).
! rangeIterator - must filter out deleted entries.
! 
! entryIterator() - only non-deleted entries.
! 
! IBatchBTree - all methods are overriden to use {@link IBatchOp#apply(ISimpleBTree)}
! so that they will correctly apply the semantics of the {@link UnisolatedBTree}.
! 
! </pre>
   */
  public class UnisolatedBTree extends BTree implements IIsolatableIndex {
--- 83,120 ----
   * 
   * @see IsolatedBTree, a {@link BTree} that has been isolated by a transaction.
!  * 
!  * @see UnisolatedIndexSegment, a read-only {@link IndexSegment} suitable for
!  *      storing data from an {@link UnisolatedBTree}.
!  * 
   * The following is an ad-hoc summary of the behavior of some methods exposed by
   * this class:
!  * 
!  * <pre>
!  *  
!  *  contains() - done.
!  *  insert() - done.
!  *  remove() - done.
!  *  lookup() - done.
!  *  
!  *  addAll() - ok as implemented.  values will be wrapped in {@link IValue} objects
!  *  as they are inserted.  if the source is also an {@link UnisolatedBTree} then the
!  *  application values will be inserted into this tree, not the {@link IValue} objects.
!  *  
!  *  indexOf() - ok as implemented (counts deleted entries).
!  *  keyAt() - ok as implemented, but will return keys for deleted entries.
!  *  valueAt() - overriden to return null for a deleted entry.
!  *  
!  *  rangeCount - ok as implemented (counts deleted entries).
!  *  rangeIterator - must filter out deleted entries.
!  *  
!  *  entryIterator() - only non-deleted entries.
!  *  
!  *  IBatchBTree - all methods are overriden to use {@link IBatchOp#apply(ISimpleBTree)}
!  *  so that they will correctly apply the semantics of the {@link UnisolatedBTree}.
!  *  
!  * </pre>
!  * 
!  * @todo Any changes to the non-mutation operations on this class MUST also be
!  *       reflected in {@link UnisolatedIndexSegment}.
   */
  public class UnisolatedBTree extends BTree implements IIsolatableIndex {
***************
*** 257,263 ****
      
      /**
!      * This method breaks isolatation to return the {@link Value} for a key. It
!      * is used by {@link IsolatedBTree#validate(UnisolatedBTree)} to test
!      * version counters when a key already exists in the global scope.
       * 
       * @todo make protected and refactor tests so that we do not need public
--- 265,271 ----
      
      /**
!      * This method breaks isolation to return the {@link Value} for a key. It is
!      * used by {@link IsolatedBTree#validate(UnisolatedBTree)} to test version
!      * counters when a key already exists in the global scope.
       * 
       * @todo make protected and refactor tests so that we do not need public
***************
*** 437,441 ****
      
      /**
!      * A filter that hides deleted entries.
       * 
       * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
--- 445,450 ----
      
      /**
!      * A filter that hides deleted entries and resolves {@link Value}s to the
!      * corresponding application datum.
       * 
       * @author <a href="mailto:tho...@us...">Bryan Thompson</a>

Index: IsolatableFusedView.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/isolation/IsolatableFusedView.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** IsolatableFusedView.java	6 Mar 2007 20:38:05 -0000	1.2
--- IsolatableFusedView.java	8 Mar 2007 18:14:06 -0000	1.3
***************
*** 50,54 ****
  import com.bigdata.objndx.AbstractBTree;
  import com.bigdata.objndx.FusedView;
- import com.bigdata.scaleup.PartitionedJournal.IViewMetadata;
  
  /**
--- 50,53 ----

Index: IIsolatableIndex.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/isolation/IIsolatableIndex.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** IIsolatableIndex.java	13 Feb 2007 23:01:02 -0000	1.1
--- IIsolatableIndex.java	8 Mar 2007 18:14:06 -0000	1.2
***************
*** 61,71 ****
  /**
   * <p>
!  * Interface for transactional isolation of an index.
!  * </p>
!  * <p>
!  * This is a marker interface for an index that supports deletion markers and
!  * transactions. Both unisolated and isolated indicies MUST use this interface
!  * in order to support transactions since version counters MUST be maintained in
!  * the unisolated indices as well as the isolated indices.
   * </p>
   * <p>
--- 61,70 ----
  /**
   * <p>
!  * This is a marker interface for an index that can be isolated by a
!  * transaction. Implementations of this interface understand and maintain both
!  * version counters and deletion markers and constrain application data to
!  * variable length byte[]s. Both unisolated and isolated indicies MUST use this
!  * interface in order to support transactions since version counters MUST be
!  * maintained in the unisolated indices as well as the isolated indices.
   * </p>
   * <p>
***************
*** 168,172 ****
   *       should it be silently ignored?
   * 
-  * 
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
   * @version $Id$
--- 167,170 ----

Index: IsolatedBTree.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/isolation/IsolatedBTree.java,v
retrieving revision 1.6
retrieving revision 1.7
diff -C2 -d -r1.6 -r1.7
*** IsolatedBTree.java	19 Feb 2007 19:00:20 -0000	1.6
--- IsolatedBTree.java	8 Mar 2007 18:14:06 -0000	1.7
***************
*** 119,124 ****
   * class.
   */
! public class IsolatedBTree extends UnisolatedBTree implements IIsolatableIndex,
!         IIsolatedIndex {
  
      /**
--- 119,123 ----
   * class.
   */
! public class IsolatedBTree extends UnisolatedBTree implements IIsolatedIndex {
  
      /**

[cweb-CVS] bigdata/src/java/com/bigdata/objndx IndexSegmentExtensionMetadata.java, NONE, 1.1 IndexSegmentBuilder.java, 1.26, 1.27 IndexSegmentFileStore.java, 1.9, 1.10 AbstractBTree.java, 1.17, 1.18 BTreeMetadata.java, 1.11, 1.12 IndexSegment.java, 1.16, 1.17 IndexSegmentMetadata.java, 1.10, 1.11

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv15000/src/java/com/bigdata/objndx

Modified Files:
	IndexSegmentBuilder.java IndexSegmentFileStore.java 
	AbstractBTree.java BTreeMetadata.java IndexSegment.java 
	IndexSegmentMetadata.java 
Added Files:
	IndexSegmentExtensionMetadata.java 
Log Message:
Updated the UML model and added a ZIP containing an HTML presentation of the model.
Working on partitioned index support.

Index: IndexSegmentFileStore.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/IndexSegmentFileStore.java,v
retrieving revision 1.9
retrieving revision 1.10
diff -C2 -d -r1.9 -r1.10
*** IndexSegmentFileStore.java	6 Mar 2007 20:38:05 -0000	1.9
--- IndexSegmentFileStore.java	8 Mar 2007 18:14:05 -0000	1.10
***************
*** 1,9 ****
--- 1,15 ----
  package com.bigdata.objndx;
  
+ import it.unimi.dsi.mg4j.util.BloomFilter;
+ 
  import java.io.File;
  import java.io.IOException;
  import java.io.RandomAccessFile;
+ import java.lang.reflect.Constructor;
  import java.nio.ByteBuffer;
  
+ import org.apache.log4j.Logger;
+ 
+ import com.bigdata.io.SerializerUtil;
  import com.bigdata.rawstore.Addr;
  import com.bigdata.rawstore.IRawStore;
***************
*** 24,27 ****
--- 30,39 ----
  
      /**
+      * Logger.
+      */
+     protected static final Logger log = Logger
+             .getLogger(IndexSegmentFileStore.class);
+ 
+     /**
       * A buffer containing the disk image of the nodes in the index segment.
       * While some nodes will be held in memory by the hard reference queue
***************
*** 47,50 ****
--- 59,67 ----
  
      /**
+      * A read-only view of the extension metadata record for the index segment.
+      */
+     protected final IndexSegmentExtensionMetadata extensionMetadata;
+     
+     /**
       * True iff the store is open.
       */
***************
*** 59,63 ****
       * 
       * @todo make it optional to fully buffer the index nodes?
!      * @todo make it optional to fully buffer the entire file.
       */
      public IndexSegmentFileStore(File file) {
--- 76,82 ----
       * 
       * @todo make it optional to fully buffer the index nodes?
!      * @todo make it optional to fully buffer the leaves as well as the nodes?
!      * 
!      * @see #load()
       */
      public IndexSegmentFileStore(File file) {
***************
*** 86,89 ****
--- 105,113 ----
  
              /*
+              * Read in the extension metadata record.
+              */
+             this.extensionMetadata = readExtensionMetadata();
+ 
+             /*
               * Read the index nodes from the file into a buffer. If there are no
               * index nodes then we skip this step. Note that we always read in
***************
*** 95,98 ****
--- 119,126 ----
                      : null);
  
+             /*
+              * Mark as open so that we can use read(long addr) to read other
+              * data (the root node/leaf).
+              */
              this.open = true;
  
***************
*** 105,108 ****
--- 133,175 ----
      }
  
+     /**
+      * Load the {@link IndexSegment} or derived class from the store. The
+      * {@link IndexSegment} or derived class MUST provide a public constructor
+      * with the following signature: <code>
+      * 
+      * <i>className</i>(IndexSegmentFileStore store)
+      * 
+      * </code>
+      * 
+      * @param store
+      *            The store.
+      * 
+      * @return The {@link IndexSegment} or derived class loaded from that store.
+      * 
+      * @see IndexSegmentExtensionMetadata, which provides a metadata extension
+      *      protocol for the {@link IndexSegment}.
+      */
+     public IndexSegment load() {
+         
+         try {
+             
+             Class cl = Class.forName(extensionMetadata.className);
+             
+             Constructor ctor = cl
+                     .getConstructor(new Class[] { IndexSegmentFileStore.class });
+ 
+             IndexSegment seg = (IndexSegment) ctor
+                     .newInstance(new Object[] { this });
+             
+             return seg;
+             
+         } catch(Exception ex) {
+             
+             throw new RuntimeException(ex);
+             
+         }
+         
+     }
+     
      public boolean isOpen() {
          
***************
*** 284,286 ****
--- 351,486 ----
      }
  
+     /**
+      * Reads the bloom filter directly from the file.
+      * 
+      * @return The bloom filter -or- <code>null</code> if the bloom filter was
+      *         not constructed when the {@link IndexSegment} was built.
+      */
+     protected BloomFilter readBloomFilter() throws IOException {
+ 
+         final long addr = metadata.addrBloom;
+         
+         if(addr == 0L) {
+             
+             return null;
+             
+         }
+         
+         log.info("reading bloom filter: "+Addr.toString(addr));
+         
+         final int off = Addr.getOffset(addr);
+         
+         final int len = Addr.getByteCount(addr);
+         
+         ByteBuffer buf = ByteBuffer.allocate(len);
+ 
+         buf.limit(len);
+ 
+         buf.position(0);
+ 
+         try {
+ 
+             // read into [dst] - does not modify the channel's position().
+             final int nread = raf.getChannel().read(buf, off);
+             
+             assert nread == len;
+             
+             buf.flip(); // Flip buffer for reading.
+             
+         } catch (IOException ex) {
+ 
+             throw new RuntimeException(ex);
+ 
+         }
+ 
+         assert buf.position() == 0;
+         assert buf.limit() == len;
+ 
+ //        ByteBufferInputStream bais = new ByteBufferInputStream(buf);
+ //        
+ ////        ByteArrayInputStream bais = new ByteArrayInputStream(buf.array());
+ //        
+ //        ObjectInputStream ois = new ObjectInputStream(bais);
+ //        
+ //        try {
+ //
+ //            BloomFilter bloomFilter = (BloomFilter) ois.readObject();
+ //            
+ //            log.info("Read bloom filter: minKeys=" + bloomFilter.size()
+ //                    + ", entryCount=" + metadata.nentries + ", bytesOnDisk="
+ //                    + len + ", errorRate=" + metadata.errorRate);
+ //            
+ //            return bloomFilter;
+ //            
+ //        }
+ //        
+ //        catch(Exception ex) {
+ //            
+ //            IOException ex2 = new IOException("Could not read bloom filter: "+ex);
+ //            
+ //            ex2.initCause(ex);
+ //            
+ //            throw ex2;
+ //            
+ //        }
+ 
+       BloomFilter bloomFilter = (BloomFilter) SerializerUtil.deserialize(buf);
+       
+       log.info("Read bloom filter: minKeys=" + bloomFilter.size()
+               + ", entryCount=" + metadata.nentries + ", bytesOnDisk="
+               + len + ", errorRate=" + metadata.errorRate);
+       
+       return bloomFilter;
+ 
+     }
+     
+     /**
+      * Reads the {@link IndexSegmentExtensionMetadata} record directly from the
+      * file.
+      */
+     protected IndexSegmentExtensionMetadata readExtensionMetadata() throws IOException {
+ 
+         final long addr = metadata.addrExtensionMetadata;
+         
+         assert addr != 0L;
+         
+         log.info("reading extension metadata record: "+Addr.toString(addr));
+         
+         final int off = Addr.getOffset(addr);
+         
+         final int len = Addr.getByteCount(addr);
+         
+         ByteBuffer buf = ByteBuffer.allocate(len);
+ 
+         buf.limit(len);
+ 
+         buf.position(0);
+ 
+         try {
+ 
+             // read into [dst] - does not modify the channel's position().
+             final int nread = raf.getChannel().read(buf, off);
+             
+             assert nread == len;
+             
+             buf.flip(); // Flip buffer for reading.
+             
+         } catch (IOException ex) {
+ 
+             throw new RuntimeException(ex);
+ 
+         }
+ 
+         assert buf.position() == 0;
+         assert buf.limit() == len;
+ 
+         IndexSegmentExtensionMetadata extensionMetadata = (IndexSegmentExtensionMetadata) SerializerUtil
+                 .deserialize(buf);
+ 
+         log.info("Read extension metadata: " + extensionMetadata);
+ 
+         return extensionMetadata;
+ 
+     }
+     
  }

--- NEW FILE: IndexSegmentExtensionMetadata.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Mar 7, 2007
 */

package com.bigdata.objndx;

import java.io.Serializable;

import com.bigdata.io.SerializerUtil;

/**
 * The base class for variable length metadataMap and extension metadataMap for an
 * {@link IndexSegment} as persisted on an {@link IndexSegmentFileStore}. The
 * {@link IndexSegmentMetadata} class is NOT extensible and is used solely for
 * fixed length metadataMap common to all {@link IndexSegment}s, including the
 * root addresses required to bootstrap the load of an {@link IndexSegment} from
 * a file. In contrast, this class provides for both required variable length
 * metadataMap and arbitrary extension metadataMap for an {@link IndexSegment}.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public class IndexSegmentExtensionMetadata implements Serializable {

    private static final long serialVersionUID = 4846316492768402991L;

    /**
     * Either {@link IndexSegment} or a derived class that will be instantiated
     * when the index segment is loaded using
     * {@link IndexSegmentFileStore#load()}
     */
    public final String className;
    
    /**
     * The serializer used for the values in the leaves of the index.
     */
    public final IValueSerializer valSer;
    
    /**
     * When non-null, a {@link RecordCompressor} that was used to write the
     * nodes and leaves of the {@link IndexSegment}.
     * 
     * @todo modify to use an interface.
     */
    final public RecordCompressor recordCompressor;

//    /**
//     * When non-null, a map containing extension metadata.
//     * 
//     * @see #getMetadata(String name)
//     */
//    final private Map<String, Serializable> metadataMap;
//
//    /**
//     * Return the metadata object stored under the key.
//     * 
//     * @param name
//     *            The key.
//     * 
//     * @return The metadata object or <code>null</code> if there is nothing
//     *         stored under that key.
//     */
//    public Serializable getMetadata(String name) {
//        
//        if(metadataMap==null) return null;
//        
//        return metadataMap.get(name);
//        
//    }
    
    /**
     * 
     * @param cl
     *            The name of the {@link IndexSegment} class that will be
     *            instantiated when the {@link IndexSegment} is loaded from the
     *            file.
     * 
     * @param valSer
     *            The object responsible for (de-)serializing the values in the
     *            leaves of the B+-Tree.
     * 
     * @param recordCompressor
     *            When non-null, a {@link RecordCompressor} that was used to
     *            write the nodes and leaves of the {@link IndexSegment}.
     */
//    * 
//    * @param metadataMap
//    *            An optional serializable map containing application defined
//    *            extension metadata. The map will be serialized with the
//    *            {@link IndexSegmentExtensionMetadata} object as part of the
//    *            {@link IndexSegmentFileStore}.
    public IndexSegmentExtensionMetadata(Class cl, IValueSerializer valSer,
            RecordCompressor recordCompressor) {
//            Map<String, Serializable> metadataMap) {

        if( cl == null ) throw new IllegalArgumentException();
        
        if( ! IndexSegment.class.isAssignableFrom(cl) ) {
            
            throw new IllegalArgumentException("Does not extend: "
                    + IndexSegment.class);
            
        }

        if( valSer == null ) throw new IllegalArgumentException();

        this.className = cl.getName();
        
        this.valSer = valSer;
        
        this.recordCompressor = recordCompressor;
        
//        this.metadataMap = metadataMap;
        
    }

    /**
     * Read the extension metadataMap record from the store.
     * 
     * @param store
     *            the store.
     * @param addr
     *            the address of the extension metadataMap record.
     * 
     * @return the extension metadataMap record.
     * 
     * @see IndexSegmentFileStore#load(), which will return an
     *      {@link IndexSegment} that is ready for use.
     */
    public static IndexSegmentExtensionMetadata read(IndexSegmentFileStore store, long addr) {
        
        return (IndexSegmentExtensionMetadata) SerializerUtil.deserialize(store.read(addr));
        
    }

}

Index: IndexSegmentBuilder.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/IndexSegmentBuilder.java,v
retrieving revision 1.26
retrieving revision 1.27
diff -C2 -d -r1.26 -r1.27
*** IndexSegmentBuilder.java	6 Mar 2007 20:38:05 -0000	1.26
--- IndexSegmentBuilder.java	8 Mar 2007 18:14:05 -0000	1.27
***************
*** 63,66 ****
--- 63,69 ----
  import org.apache.log4j.Logger;
  
+ import com.bigdata.io.SerializerUtil;
+ import com.bigdata.isolation.UnisolatedIndexSegment;
+ import com.bigdata.isolation.Value;
  import com.bigdata.journal.Journal;
  import com.bigdata.journal.TemporaryRawStore;
***************
*** 74,80 ****
--- 77,85 ----
   * factor. There are two main use cases:
   * <ol>
+  * 
   * <li>Evicting a key range of an index into an optimized on-disk index. In
   * this case, the input is a {@link BTree} that is ideally backed by a fully
   * buffered {@link IRawStore} so that no random reads are required.</li>
+  * 
   * <li>Merging index segments. In this case, the input is typically records
   * emerging from a merge-sort. There are two distinct cases here. In one, we
***************
*** 88,91 ****
--- 93,97 ----
   * history policy is defined, then it must be applied here to cause key-value
   * whose retention is no longer required by that policy to be dropped.</li>
+  * 
   * </ol>
   * 
***************
*** 109,116 ****
--- 115,129 ----
   *      less efficient on first glance.
   * 
+  * FIXME support build from a key range rather than just an entire source tree.
+  * this is required to support partitioned indices.
+  * 
+  * FIXME support efficient prior/next leaf scans.
+  * 
   * FIXME use the shortest separator key.
   * 
   * @see IndexSegment
   * @see IndexSegmentFile
+  * @see IndexSegmentMetadata
+  * @see IndexSegmentExtensionMetadata
   * @see IndexSegmentMerger
   */
***************
*** 200,203 ****
--- 213,222 ----
      final BloomFilter bloomFilter;
      
+ //    /**
+ //     * When non-null, a map containing extension metadata.  This is set by the
+ //     * constructor.
+ //     */
+ //    final Map<String, Serializable> metadataMap;
+     
      /**
       * The offset in the output file of the last leaf written onto that file.
***************
*** 263,266 ****
--- 282,301 ----
       */
      final public IndexSegmentPlan plan;
+    
+ //    /**
+ //     * The address at which each leaf is written on the file.  This information
+ //     * is stored in the {@link IndexSegmentExtensionMetadata} and may be used to
+ //     * perform fast forward or reverse leaf scans by first looking up the leaf
+ //     * address based on its ordinal position within the file and then scanning
+ //     * forward or backward through the addresses in this array.  The array is
+ //     * buffered when the {@link IndexSegment} is loaded.
+ //     *
+ //     * @todo We need to be able to extend the leaf data structure for this
+ //     * purpose.  We could write the addr of the prior leaf on the leaf, but
+ //     * not of the next leaf.  For that we need to ordinal position of the leaf.
+ //     * 
+ //     * @see ... 
+ //     */
+ //    final long[] leaveAddrs;
      
      /**
***************
*** 412,424 ****
       *            option should only be enabled if you know that point access
       *            tests are a hotspot for an index.
-      * 
       * @throws IOException
-      * 
-      * FIXME support efficient prior/next leaf scans.
       */
      public IndexSegmentBuilder(File outFile, File tmpDir, final int entryCount,
              IEntryIterator entryIterator, final int m,
              IValueSerializer valueSerializer, boolean useChecksum,
!             RecordCompressor recordCompressor, final double errorRate)
              throws IOException {
  
--- 447,463 ----
       *            option should only be enabled if you know that point access
       *            tests are a hotspot for an index.
       * @throws IOException
       */
+ //    * @param metadataMap
+ //    *            An optional serializable map containing application defined
+ //    *            extension metadataMap. The map will be serialized with the
+ //    *            {@link IndexSegmentExtensionMetadata} object as part of the
+ //    *            {@link IndexSegmentFileStore}.
      public IndexSegmentBuilder(File outFile, File tmpDir, final int entryCount,
              IEntryIterator entryIterator, final int m,
              IValueSerializer valueSerializer, boolean useChecksum,
!             RecordCompressor recordCompressor, final double errorRate
! //          , final Map<String, Serializable> metadataMap
!             )
              throws IOException {
  
***************
*** 501,504 ****
--- 540,545 ----
              }
  
+ //            this.metadataMap = metadataMap;
+             
              // Used to serialize the nodes and leaves for the output tree.
              nodeSer = new NodeSerializer(NOPNodeFactory.INSTANCE,
***************
*** 1055,1076 ****
          ByteBuffer buf = nodeSer.putLeaf(leaf);
  
- //        /*
- //         * Write leaf on the channel.
- //         */
- //        
- //        FileChannel outChannel = out.getChannel();
- //
- //        // position on the channel before the write.
- //        final long offset = outChannel.position();
- //        
- //        if(offset>Integer.MAX_VALUE) {
- //            
- //            throw new IOException("Index segment exceeds int32 bytes.");
- //            
- //        }
- //        
- //        // write on the channel.
- //        final int nbytes = outChannel.write(buf);
- 
          final long addr1 = leafBuffer.write(buf);
          
--- 1096,1099 ----
***************
*** 1199,1210 ****
       * 
       * <pre>
!      * metadata record
!      * leaves
!      * nodes (may be empty)
!      * extension metadata records, including the optional bloom filter.
       * </pre>
       * 
!      * Each of these regions has a variable length and some may be missing
!      * entirely.
       * <p>
       * Once all nodes and leaves have been buffered we are ready to start
--- 1222,1241 ----
       * 
       * <pre>
!      *  fixed length metadata record (required)
!      *  leaves (required)
!      *  nodes (may be empty)
!      *  the bloom filter (optional).
!      *  the extension metadata record (required, but extensible).
       * </pre>
       * 
!      * <p>
!      * The index segment metadata is divided into a base
!      * {@link IndexSegmentMetadata} record with a fixed format containing only
!      * essential data and additional metadata records written at the end of the
!      * file including the optional bloom filter and the required
!      * {@link IndexSegmentExtensionMetadata} record. the latter is where we
!      * write variable length metadata including the _name_ of the index, or
!      * additional metadata defined by a specific class of index.
!      * 
       * <p>
       * Once all nodes and leaves have been buffered we are ready to start
***************
*** 1225,1246 ****
       * @throws IOException
       * 
-      * @todo the metadata record can be divided into a base record with a fixed
-      *       format containing only essential data and an extensible record to
-      *       be written at the end of the file. the latter section is where we
-      *       would write the bloom filters and other variable length metadata
-      *       including the _name_ of the index, or additional metadata defined
-      *       by a specific class of index.
-      *       <p>
-      *       the commit point for the index segment file should be a metadata
-      *       record at the head of the file having identical timestamps at the
-      *       start and end of its data section. since the file format is
-      *       immutable it is ok to have what is essentially only a single root
-      *       block. if the timestamps do not agree then the build was no
-      *       successfully completed.
-      *       <p>
-      *       the checksum of the index segment file should be stored in the
-      *       partitioned index so that it can be validated after being moved
-      *       around, etc.
-      * 
       * @todo it would be nice to have the same file format and addresses as the
       *       journal. in order to do this we need to (a) write two root blocks
--- 1256,1259 ----
***************
*** 1376,1379 ****
--- 1389,1438 ----
          
          /*
+          * Write out the extensible metadata record at the end of the file.
+          */
+         final long addrExtensionMetadata;
+         {
+ 
+             /*
+              * Choose the implementation class based on whether or not the index
+              * is isolatable.
+              * 
+              * @todo this test may be too fragile and is not extensible to other
+              * implementation classes.
+              * 
+              * FIXME at a minimum, the BTree class should be accessible and
+              * provided to the extension metadata constructor so that any
+              * interesting metadata may also be stored in the index segment.
+              */
+             final Class cl = (nodeSer.valueSerializer instanceof Value.Serializer ? UnisolatedIndexSegment.class
+                     : IndexSegment.class);
+ 
+             /*
+              * Setup and serialize the extension metadata.
+              */
+             IndexSegmentExtensionMetadata extensionMetadata = new IndexSegmentExtensionMetadata(
+                     cl, nodeSer.valueSerializer, nodeSer.recordCompressor);
+ //                    metadataMap);
+ 
+             final byte[] extensionMetadataBytes = SerializerUtil
+                     .serialize(extensionMetadata);
+ 
+             assert out.length() < Integer.MAX_VALUE;
+             
+             final int offset = (int)out.length(); 
+ 
+             // seek to the end of the file.
+             out.seek(offset);
+             
+             // write the serialized extension metadata.
+             out.write(extensionMetadataBytes, 0, extensionMetadataBytes.length);
+ 
+             // note its address.
+             addrExtensionMetadata = Addr.toLong(extensionMetadataBytes.length,
+                     offset);
+             
+         }
+         
+         /*
           * Seek to the start of the file and write out the metadata record.
           */
***************
*** 1383,1396 ****
              final long now = System.currentTimeMillis();
              
-             // @todo name of the index segment - drop this field? add uuids?
-             final String name = "<no name>";
-             
              outChannel.position(0);
              
              IndexSegmentMetadata md = new IndexSegmentMetadata(plan.m,
!                     plan.height, useChecksum, recordCompressor != null,
!                     plan.nleaves, nnodesWritten, plan.nentries,
!                     maxNodeOrLeafLength, addrLeaves, addrNodes, addrRoot,
!                     errorRate, addrBloom, out.length(), now, name);
  
              md.write(out);
--- 1442,1452 ----
              final long now = System.currentTimeMillis();
              
              outChannel.position(0);
              
              IndexSegmentMetadata md = new IndexSegmentMetadata(plan.m,
!                     plan.height, useChecksum, plan.nleaves, nnodesWritten,
!                     plan.nentries, maxNodeOrLeafLength, addrLeaves, addrNodes,
!                     addrRoot, addrExtensionMetadata, addrBloom, errorRate, out
!                             .length(), now);
  
              md.write(out);
***************
*** 1480,1483 ****
--- 1536,1548 ----
  
          // mutable.
+         
+ //        /**
+ //         * The ordinal position of this leaf in the {@link IndexSegment}.
+ //         */
+ //        int leafIndex;
+         
+         /**
+          * The values stored in the leaf.
+          */
          final Object[] vals;
          

Index: IndexSegmentMetadata.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/IndexSegmentMetadata.java,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** IndexSegmentMetadata.java	5 Feb 2007 18:17:39 -0000	1.10
--- IndexSegmentMetadata.java	8 Mar 2007 18:14:05 -0000	1.11
***************
*** 4,7 ****
--- 4,8 ----
  import java.io.RandomAccessFile;
  import java.util.Date;
+ import java.util.UUID;
  
  import com.bigdata.rawstore.Addr;
***************
*** 10,38 ****
  /**
   * The metadata record for an {@link IndexSegment}.
   * 
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
   * @version $Id$
   * 
!  * @todo consider recording the min/max key or just making it easy to determine
!  *       that for an {@link IndexSegment}. This has do to with both correct
!  *       rejection of queries directed to the wrong index segment and managing
!  *       the metadata for a distributed index.
!  * 
!  * @todo add a uuid for each index segment and a uuid for the index to which the
!  *       segments belong? examine the format of the uuid. can we use part of it
!  *       as the unique basis for one up identifiers within a parition?
!  * 
!  * FIXME We need a general mechanism for persisting metadata including the
!  * valSer, record compressor, and user-defined objects for indices. These data
!  * can go into a series of extensible metadata records located at the end of the
!  * file. The bloom filter itself could be an example of such a metadata record.
!  * Such metadata should survive conversions from a btree to an index segment,
!  * mergers of index segments or btrees, and conversion from an index segment to
!  * a btree.
!  * 
!  * FIXME introduce two timestamps in the metadata record. the record is valid
!  * iff both timestamps agree and are non-zero.
   */
  public class IndexSegmentMetadata {
      
      /**
--- 11,66 ----
  /**
   * The metadata record for an {@link IndexSegment}.
+  * <p>
+  * The commit point for the index segment file should be a metadata record at
+  * the head of the file having identical timestamps at the start and end of its
+  * data section. Since the file format is immutable it is ok to have what is
+  * essentially only a single root block. If the timestamps do not agree then the
+  * build was not successfully completed.
   * 
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
   * @version $Id$
   * 
!  * @todo the checksum of the index segment file should be stored in the
!  *       partitioned index so that it can be validated after being moved around,
!  *       etc. it would also be good to checksum the {@link IndexSegmentMetadata}
!  *       record.
   */
  public class IndexSegmentMetadata {
+ 
+     static final int SIZEOF_MAGIC = Bytes.SIZEOF_INT;
+     static final int SIZEOF_VERSION = Bytes.SIZEOF_INT;
+     static final int SIZEOF_BRANCHING_FACTOR = Bytes.SIZEOF_INT;
+     static final int SIZEOF_COUNTS = Bytes.SIZEOF_INT;
+     static final int SIZEOF_NBYTES = Bytes.SIZEOF_INT;
+     static final int SIZEOF_ADDR = Bytes.SIZEOF_LONG;
+     static final int SIZEOF_ERROR_RATE = Bytes.SIZEOF_DOUBLE;
+     static final int SIZEOF_TIMESTAMP = Bytes.SIZEOF_LONG;
+ 
+     /**
+      * The #of unused bytes in the metadata record format. Note that the unused
+      * space occurs between the file size and the final timestamp in the record.
+      * As the unused bytes are allocated in new versions the value in this field
+      * MUST be adjusted down from its original value of 256.
+      */
+     static final int SIZEOF_UNUSED = 256;
+ 
+     /**
+      * The #of bytes required by the current metadata record format.
+      */
+     static final int SIZE = //
+             SIZEOF_MAGIC + //
+             SIZEOF_VERSION + //
+             Bytes.SIZEOF_LONG + // timestamp0
+             Bytes.SIZEOF_UUID + // index segment UUID.
+             SIZEOF_BRANCHING_FACTOR + // branchingFactor
+             SIZEOF_COUNTS * 4 + // height, #leaves, #nodes, #entries
+             SIZEOF_NBYTES + // max record length
+             Bytes.SIZEOF_BYTE + // useChecksum
+             SIZEOF_ADDR * 5 + // leaves, nodes, root, ext metadata, bloomFilter
+             Bytes.SIZEOF_DOUBLE + // errorRate
+             Bytes.SIZEOF_LONG + // file size
+             SIZEOF_UNUSED + // available bytes for future versions.
+             Bytes.SIZEOF_LONG // timestamp1
+     ;
      
      /**
***************
*** 45,49 ****
       */
      static transient final public int VERSION0 = 0x0;
!     
      /**
       * Branching factor for the index segment.
--- 73,82 ----
       */
      static transient final public int VERSION0 = 0x0;
!    
!     /**
!      * UUID for this {@link IndexSegment}.
!      */
!     final public UUID uuid;
! 
      /**
       * Branching factor for the index segment.
***************
*** 58,76 ****
      
      /**
-      * When true, the checksum was computed and stored for the nodes and leaves
-      * in the file and will be verified on de-serialization.
-      */
-     final public boolean useChecksum;
-     
-     /**
-      * When true, a {@link RecordCompressor} was used to write the nodes and
-      * leaves of the {@link IndexSegment}.
-      * 
-      * @todo modify to specify the implementation of a record compressor
-      *       interface.
-      */
-     final public boolean useRecordCompressor;
-     
-     /**
       * The #of leaves serialized in the file.
       */
--- 91,94 ----
***************
*** 99,102 ****
--- 117,126 ----
      
      /**
+      * When true, the checksum was computed and stored for the nodes and leaves
+      * in the file and will be verified on de-serialization.
+      */
+     final public boolean useChecksum;
+     
+     /**
       * The {@link Addr address} of the contiguous region containing the
       * serialized leaves in the file.
***************
*** 123,130 ****
  
      /**
!      * The target error rate for the optional bloom filter and 0.0 iff
!      * the bloom filter was not constructed.
       */
!     final public double errorRate;
      
      /**
--- 147,153 ----
  
      /**
!      * The address of the {@link IndexSegmentExtensionMetadata} record.
       */
!     final public long addrExtensionMetadata;
      
      /**
***************
*** 137,140 ****
--- 160,169 ----
      
      /**
+      * The target error rate for the optional bloom filter and 0.0 iff
+      * the bloom filter was not constructed.
+      */
+     final public double errorRate;
+     
+     /**
       * Length of the file in bytes.
       */
***************
*** 145,165 ****
       */
      final public long timestamp;
-     
-     /**
-      * @todo Name of the index?  or uuid? or drop?
-      */
-     final public String name;
- 
-     /**
-      * The #of bytes in the metadata record.
-      * 
-      * @todo This is oversized in order to allow some slop for future entries
-      *       and in order to permit the variable length index name to be
-      *       recorded in the index segment file.  The size needs to be reviewed
-      *       once the design is crisper.
-      */
-     public static final int SIZE = Bytes.kilobyte32 * 4;
-     
-     public static final int MAX_NAME_LENGTH = Bytes.kilobyte32 * 2;
  
      /**
--- 174,177 ----
***************
*** 192,195 ****
--- 204,211 ----
              
          }
+ 
+         final long timestamp0 = raf.readLong();
+         
+         uuid = new UUID(raf.readLong()/*MSB*/, raf.readLong()/*LSB*/);
          
          branchingFactor = raf.readInt();
***************
*** 197,204 ****
          height = raf.readInt();
          
-         useChecksum = raf.readBoolean();
-         
-         useRecordCompressor = raf.readBoolean();
-         
          nleaves = raf.readInt();
          
--- 213,216 ----
***************
*** 209,212 ****
--- 221,226 ----
          maxNodeOrLeafLength = raf.readInt();
          
+         useChecksum = raf.readBoolean();
+                 
          addrLeaves = raf.readLong();
  
***************
*** 215,222 ****
          addrRoot = raf.readLong();
  
!         errorRate = raf.readDouble();
!         
          addrBloom = raf.readLong();
          
          length = raf.readLong();
          
--- 229,238 ----
          addrRoot = raf.readLong();
  
!         addrExtensionMetadata = raf.readLong();
! 
          addrBloom = raf.readLong();
          
+         errorRate = raf.readDouble();
+         
          length = raf.readLong();
          
***************
*** 226,234 ****
          }
          
!         timestamp = raf.readLong();
! 
!         name = raf.readUTF();
          
!         assert name.length() <= MAX_NAME_LENGTH;
          
      }
--- 242,256 ----
          }
          
!         raf.skipBytes(SIZEOF_UNUSED);
          
!         final long timestamp1 = raf.readLong();
!         
!         if(timestamp0 != timestamp1) {
!             
!             throw new RuntimeException("Timestamps do not agree - file is not useable.");
!             
!         }
!         
!         this.timestamp = timestamp0;
          
      }
***************
*** 241,249 ****
       */
      public IndexSegmentMetadata(int branchingFactor, int height,
!             boolean useChecksum, boolean useRecordCompressor, int nleaves,
!             int nnodes, int nentries, int maxNodeOrLeafLength,
!             long addrLeaves, long addrNodes, long addrRoot,
!             double errorRate, long addrBloom, long length, long timestamp,
!             String name) {
          
          assert branchingFactor >= BTree.MIN_BRANCHING_FACTOR;
--- 263,270 ----
       */
      public IndexSegmentMetadata(int branchingFactor, int height,
!             boolean useChecksum, int nleaves, int nnodes, int nentries,
!             int maxNodeOrLeafLength, long addrLeaves, long addrNodes,
!             long addrRoot, long addrExtensionMetadata, long addrBloom,
!             double errorRate, long length, long timestamp) {
          
          assert branchingFactor >= BTree.MIN_BRANCHING_FACTOR;
***************
*** 286,292 ****
          assert timestamp != 0L;
          
!         assert name != null;
!         
!         assert name.length() <= MAX_NAME_LENGTH;
  
          this.branchingFactor = branchingFactor;
--- 307,311 ----
          assert timestamp != 0L;
          
!         this.uuid = UUID.randomUUID();
  
          this.branchingFactor = branchingFactor;
***************
*** 294,301 ****
          this.height = height;
  
-         this.useChecksum = useChecksum;
- 
-         this.useRecordCompressor = useRecordCompressor;
-         
          this.nleaves = nleaves;
          
--- 313,316 ----
***************
*** 306,309 ****
--- 321,326 ----
          this.maxNodeOrLeafLength = maxNodeOrLeafLength;
          
+         this.useChecksum = useChecksum;
+         
          this.addrLeaves = addrLeaves;
          
***************
*** 312,324 ****
          this.addrRoot = addrRoot;
  
!         this.errorRate = errorRate;
!         
          this.addrBloom = addrBloom;
          
          this.length = length;
          
          this.timestamp = timestamp;
- 
-         this.name = name;
          
      }
--- 329,341 ----
          this.addrRoot = addrRoot;
  
!         this.addrExtensionMetadata = addrExtensionMetadata;
! 
          this.addrBloom = addrBloom;
          
+         this.errorRate = errorRate;
+         
          this.length = length;
          
          this.timestamp = timestamp;
          
      }
***************
*** 339,351 ****
  
          raf.writeInt(VERSION0);
          
          raf.writeInt(branchingFactor);
                          
          raf.writeInt(height);
          
-         raf.writeBoolean(useChecksum);
-                 
-         raf.writeBoolean(useRecordCompressor);
-         
          raf.writeInt(nleaves);
  
--- 356,370 ----
  
          raf.writeInt(VERSION0);
+ 
+         raf.writeLong(timestamp);
          
+         raf.writeLong(uuid.getMostSignificantBits());
+ 
+         raf.writeLong(uuid.getLeastSignificantBits());
+ 
          raf.writeInt(branchingFactor);
                          
          raf.writeInt(height);
          
          raf.writeInt(nleaves);
  
***************
*** 356,359 ****
--- 375,380 ----
          raf.writeInt(maxNodeOrLeafLength);
          
+         raf.writeBoolean(useChecksum);
+         
          raf.writeLong(addrLeaves);
          
***************
*** 362,375 ****
          raf.writeLong(addrRoot);
  
!         raf.writeDouble(errorRate);
          
          raf.writeLong(addrBloom);
          
          raf.writeLong(length);
          
          raf.writeLong(timestamp);
          
-         raf.writeUTF(name);
-         
      }
      
--- 383,398 ----
          raf.writeLong(addrRoot);
  
!         raf.writeLong(addrExtensionMetadata);
          
          raf.writeLong(addrBloom);
          
+         raf.writeDouble(errorRate);
+         
          raf.writeLong(length);
+ 
+         raf.skipBytes(SIZEOF_UNUSED);
          
          raf.writeLong(timestamp);
          
      }
      
***************
*** 382,401 ****
          
          sb.append("magic="+Integer.toHexString(MAGIC));
          sb.append(", branchingFactor="+branchingFactor);
          sb.append(", height=" + height);
-         sb.append(", useChecksum=" + useChecksum);
-         sb.append(", useRecordCompressor=" + useRecordCompressor);
          sb.append(", nleaves=" + nleaves);
          sb.append(", nnodes=" + nnodes);
          sb.append(", nentries=" + nentries);
          sb.append(", maxNodeOrLeafLength=" + maxNodeOrLeafLength);
!         sb.append(", addrLeaves=" + addrLeaves);
!         sb.append(", addrNodes=" + addrNodes);
          sb.append(", addrRoot=" + Addr.toString(addrRoot));
!         sb.append(", errorRate=" + errorRate);
          sb.append(", addrBloom=" + Addr.toString(addrBloom));
          sb.append(", length=" + length);
          sb.append(", timestamp=" + new Date(timestamp));
-         sb.append(", name="+name);
  
          return sb.toString();
--- 405,424 ----
          
          sb.append("magic="+Integer.toHexString(MAGIC));
+         sb.append(", uuid="+uuid);
          sb.append(", branchingFactor="+branchingFactor);
          sb.append(", height=" + height);
          sb.append(", nleaves=" + nleaves);
          sb.append(", nnodes=" + nnodes);
          sb.append(", nentries=" + nentries);
          sb.append(", maxNodeOrLeafLength=" + maxNodeOrLeafLength);
!         sb.append(", useChecksum=" + useChecksum);
!         sb.append(", addrLeaves=" + Addr.toString(addrLeaves));
!         sb.append(", addrNodes=" + Addr.toString(addrNodes));
          sb.append(", addrRoot=" + Addr.toString(addrRoot));
!         sb.append(", addrExtensionMetadata=" + Addr.toString(addrExtensionMetadata));
          sb.append(", addrBloom=" + Addr.toString(addrBloom));
+         sb.append(", errorRate=" + errorRate);
          sb.append(", length=" + length);
          sb.append(", timestamp=" + new Date(timestamp));
  
          return sb.toString();

Index: BTreeMetadata.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/BTreeMetadata.java,v
retrieving revision 1.11
retrieving revision 1.12
diff -C2 -d -r1.11 -r1.12
*** BTreeMetadata.java	21 Feb 2007 20:17:21 -0000	1.11
--- BTreeMetadata.java	8 Mar 2007 18:14:05 -0000	1.12
***************
*** 7,11 ****
  
  import com.bigdata.io.SerializerUtil;
- import com.bigdata.isolation.IConflictResolver;
  import com.bigdata.rawstore.Addr;
  import com.bigdata.rawstore.IRawStore;
--- 7,10 ----
***************
*** 155,159 ****
      /**
       * Re-load the {@link BTree} or derived class from the store. The
!      * {@link BTree} or derived class MUST provide a public construct with the
       * following signature: <code>
       * 
--- 154,158 ----
      /**
       * Re-load the {@link BTree} or derived class from the store. The
!      * {@link BTree} or derived class MUST provide a public constructor with the
       * following signature: <code>
       * 
***************
*** 171,175 ****
       *         {@link BTreeMetadata} record.
       * 
!      * @see BTree#newMetadata(), which MUST be overloaded if you subclass extend
       *      {@link BTreeMetadata}.
       */
--- 170,174 ----
       *         {@link BTreeMetadata} record.
       * 
!      * @see BTree#newMetadata(), which MUST be overloaded if you subclass
       *      {@link BTreeMetadata}.
       */

Index: IndexSegment.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/IndexSegment.java,v
retrieving revision 1.16
retrieving revision 1.17
diff -C2 -d -r1.16 -r1.17
*** IndexSegment.java	12 Feb 2007 21:51:07 -0000	1.16
--- IndexSegment.java	8 Mar 2007 18:14:05 -0000	1.17
***************
*** 1,16 ****
  package com.bigdata.objndx;
  
- import it.unimi.dsi.mg4j.util.BloomFilter;
- 
  import java.io.DataInputStream;
  import java.io.DataOutputStream;
  import java.io.IOException;
- import java.io.ObjectInputStream;
- import java.nio.ByteBuffer;
  
  import org.CognitiveWeb.extser.LongPacker;
  
  import com.bigdata.cache.HardReferenceQueue;
- import com.bigdata.io.ByteBufferInputStream;
  import com.bigdata.rawstore.Addr;
  import com.bigdata.rawstore.Bytes;
--- 1,11 ----
***************
*** 36,40 ****
   * @version $Id$
   */
! public class IndexSegment extends AbstractBTree implements IIndex {
  
      /**
--- 31,35 ----
   * @version $Id$
   */
! public class IndexSegment extends AbstractBTree {
  
      /**
***************
*** 43,52 ****
      final protected IndexSegmentFileStore fileStore;
      
- //    /**
- //     * The root of the btree. Since this is a read-only index the root can never
- //     * be replaced.
- //     */
- //    final protected AbstractNode root;
- 
      /**
       * An optional bloom filter that will be used to filter point tests. Since
--- 38,41 ----
***************
*** 87,96 ****
      }
  
- //    public AbstractNode getRoot() {
- //        
- //        return root;
- //        
- //    }
-     
      public int getEntryCount() {
  
--- 76,79 ----
***************
*** 99,108 ****
      }
  
!     public IndexSegment(IndexSegmentFileStore fileStore, IValueSerializer valSer) {
! 
          this(fileStore, new HardReferenceQueue<PO>(
                  new DefaultEvictionListener(),
                  BTree.DEFAULT_HARD_REF_QUEUE_CAPACITY,
!                 BTree.DEFAULT_HARD_REF_QUEUE_SCAN), valSer);
          
      }
--- 82,91 ----
      }
  
!     public IndexSegment(IndexSegmentFileStore fileStore ) {
!         
          this(fileStore, new HardReferenceQueue<PO>(
                  new DefaultEvictionListener(),
                  BTree.DEFAULT_HARD_REF_QUEUE_CAPACITY,
!                 BTree.DEFAULT_HARD_REF_QUEUE_SCAN));
          
      }
***************
*** 121,147 ****
       *            large, e.g., try with 100 and 20 respectively.
       * @param valSer
       * @throws IOException
       * 
       * @todo explore good defaults for the hard reference queue, which should
       *       probably be much smaller as the branching factor grows larger.
-      * 
-      * FIXME move the value serializer into the metadata record.
-      * 
-      * FIXME add a boolean flag to mark index segments that are the final result
-      * of a compacting merge.  This will make it possible to reconstruct from the
-      * file system which index segments are part of the consistent state for a
-      * given restart time.
       */
!     public IndexSegment(IndexSegmentFileStore fileStore,
!             HardReferenceQueue<PO> hardReferenceQueue,
!             IValueSerializer valSer) {
  
          super(fileStore, fileStore.metadata.branchingFactor,
                  fileStore.metadata.maxNodeOrLeafLength, hardReferenceQueue,
                  new CustomAddressSerializer(Addr
!                         .getOffset(fileStore.metadata.addrNodes)), valSer,
                  ImmutableNodeFactory.INSTANCE,
!                 fileStore.metadata.useRecordCompressor ? new RecordCompressor()
!                         : null, fileStore.metadata.useChecksum);
  
          // Type-safe reference to the backing store.
--- 104,124 ----
       *            large, e.g., try with 100 and 20 respectively.
       * @param valSer
+      * 
       * @throws IOException
       * 
       * @todo explore good defaults for the hard reference queue, which should
       *       probably be much smaller as the branching factor grows larger.
       */
!     protected IndexSegment(IndexSegmentFileStore fileStore,
!             HardReferenceQueue<PO> hardReferenceQueue) {
  
          super(fileStore, fileStore.metadata.branchingFactor,
                  fileStore.metadata.maxNodeOrLeafLength, hardReferenceQueue,
                  new CustomAddressSerializer(Addr
!                         .getOffset(fileStore.metadata.addrNodes)),
!                 fileStore.extensionMetadata.valSer,
                  ImmutableNodeFactory.INSTANCE,
!                 fileStore.extensionMetadata.recordCompressor,
!                 fileStore.metadata.useChecksum);
  
          // Type-safe reference to the backing store.
***************
*** 167,171 ****
              try {
  
!                 this.bloomFilter = readBloomFilter(fileStore.metadata.addrBloom);
  
              } catch (IOException ex) {
--- 144,148 ----
              try {
  
!                 this.bloomFilter = fileStore.readBloomFilter();
  
              } catch (IOException ex) {
***************
*** 179,258 ****
      }
  
-     /**
-      * Reads the bloom filter from the file.
-      * 
-      * Note: this goes around the {@link IndexSegmentFileStore} API since the bloom filter
-      * is not (currently) written as a compressed record and since the size of
-      * the largest compressed record does not pay attention to the serialized
-      * size of the optional bloom filter.
-      */
-     protected BloomFilter readBloomFilter(long addr) throws IOException {
- 
-         assert addr != 0L;
-         
-         System.err.println("reading bloom filter: "+Addr.toString(addr));
-         
-         final int off = Addr.getOffset(addr);
-         
-         final int len = Addr.getByteCount(addr);
-         
-         ByteBuffer buf = ByteBuffer.allocate(len);
- 
-         buf.limit(len);
- 
-         buf.position(0);
- 
-         try {
- 
-             // read into [dst] - does not modify the channel's position().
-             final int nread = fileStore.raf.getChannel().read(buf, off);
-             
-             assert nread == len;
-             
-             buf.flip(); // Flip buffer for reading.
-             
-         } catch (IOException ex) {
- 
-             throw new RuntimeException(ex);
- 
-         }
- 
-         assert buf.position() == 0;
-         assert buf.limit() == len;
- 
-         ByteBufferInputStream bais = new ByteBufferInputStream(buf);
-         
- //        ByteArrayInputStream bais = new ByteArrayInputStream(buf.array());
-         
-         ObjectInputStream ois = new ObjectInputStream(bais);
-         
-         try {
- 
-             BloomFilter bloomFilter = (BloomFilter) ois.readObject();
-             
-             log.info("Read bloom filter: minKeys=" + bloomFilter.size()
-                     + ", entryCount=" + getEntryCount() + ", bytesOnDisk="
-                     + len + ", errorRate=" + fileStore.metadata.errorRate);
-             
-             return bloomFilter;
-             
-         }
-         
-         catch(Exception ex) {
-             
-             IOException ex2 = new IOException("Could not read bloom filter: "+ex);
-             
-             ex2.initCause(ex);
-             
-             throw ex2;
-             
-         }
- 
-     }
-     
-     /**
-      * @todo move to parent class and have various methods test to validate that
-      *       the index is open (lookup, insert, remove, scan).
-      */
      public void close() {
          
--- 156,159 ----

Index: AbstractBTree.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/AbstractBTree.java,v
retrieving revision 1.17
retrieving revision 1.18
diff -C2 -d -r1.17 -r1.18
*** AbstractBTree.java	21 Feb 2007 20:17:21 -0000	1.17
--- AbstractBTree.java	8 Mar 2007 18:14:05 -0000	1.18
***************
*** 106,113 ****
      /**
       * Log for btree opeations.
-      * 
-      * @todo consider renaming the logger.
       */
!     protected static final Logger log = Logger.getLogger(BTree.class);
  
      /**
--- 106,111 ----
      /**
       * Log for btree opeations.
       */
!     protected static final Logger log = Logger.getLogger(AbstractBTree.class);
  
      /**

[cweb-CVS] bigdata/src/java/com/bigdata/rawstore package.html, 1.1, 1.2 Bytes.java, 1.1, 1.2 SimpleMemoryRawStore.java, 1.5, NONE SimpleFileRawStore.java, 1.6, NONE

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/rawstore
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv15000/src/java/com/bigdata/rawstore

Modified Files:
	package.html Bytes.java 
Removed Files:
	SimpleMemoryRawStore.java SimpleFileRawStore.java 
Log Message:
Updated the UML model and added a ZIP containing an HTML presentation of the model.
Working on partitioned index support.

Index: package.html
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/rawstore/package.html,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** package.html	13 Feb 2007 23:01:05 -0000	1.1
--- package.html	8 Mar 2007 18:14:05 -0000	1.2
***************
*** 11,14 ****
--- 11,20 ----
  IRawStore} interface that supports atomic commit and is restart-safe.
  </p>
+ <p>
+ There are some trivial implementations in the test suite that are used
+ for bootstrapping classes with a dependency on {@link IRawStore}, such
+ as the {@link com.bigdata.objndx.BTree}, without introducing a dependency
+ on the {@link com.bigdata.journal.Journal}.
+ </p>
  
  </body>

--- SimpleFileRawStore.java DELETED ---

--- SimpleMemoryRawStore.java DELETED ---

Index: Bytes.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/rawstore/Bytes.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** Bytes.java	5 Feb 2007 18:17:42 -0000	1.1
--- Bytes.java	8 Mar 2007 18:14:05 -0000	1.2
***************
*** 56,63 ****
  public class Bytes {
  
!     public static final int SIZEOF_BYTE  = 1;
!     public static final int SIZEOF_SHORT = 2;
!     public static final int SIZEOF_INT   = 4;
!     public static final int SIZEOF_LONG  = 8;
      
      /*
--- 56,66 ----
  public class Bytes {
  
!     public static final int SIZEOF_BYTE   = 1;
!     public static final int SIZEOF_SHORT  = 2;
!     public static final int SIZEOF_INT    = 4;
!     public static final int SIZEOF_LONG   = 8;
!     public static final int SIZEOF_FLOAT  = 4;
!     public static final int SIZEOF_DOUBLE = 8;
!     public static final int SIZEOF_UUID   = 16;
      
      /*

[cweb-CVS] bigdata-rdf/src/java/com/bigdata/rdf/rio BulkRioLoader.java, 1.10, 1.11

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/rio
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv15836/src/java/com/bigdata/rdf/rio

Modified Files:
	BulkRioLoader.java 
Log Message:
Updated the UML model and added a ZIP containing an HTML presentation of the model.
Working on partitioned index support.

Index: BulkRioLoader.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/rio/BulkRioLoader.java,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** BulkRioLoader.java	9 Feb 2007 20:18:57 -0000	1.10
--- BulkRioLoader.java	8 Mar 2007 18:14:40 -0000	1.11
***************
*** 390,395 ****
              if (seg == null) {
  
!                 seg = new IndexSegment(new IndexSegmentFileStore(file),
!                         com.bigdata.rdf.serializers.TermIdSerializer.INSTANCE);
              }
              
--- 390,395 ----
              if (seg == null) {
  
!                 seg = new IndexSegmentFileStore(file).load();
!                 // com.bigdata.rdf.serializers.TermIdSerializer.INSTANCE)
              }
              
***************
*** 405,410 ****
              if (seg == null) {
  
!                 seg = new IndexSegment(new IndexSegmentFileStore(file),
!                         com.bigdata.rdf.serializers.RdfValueSerializer.INSTANCE);
  
              }
--- 405,410 ----
              if (seg == null) {
  
!                 seg = new IndexSegmentFileStore(file).load();
! //                        com.bigdata.rdf.serializers.RdfValueSerializer.INSTANCE);
  
              }
***************
*** 420,425 ****
              if (seg == null) {
  
!                 seg = new IndexSegment(new IndexSegmentFileStore(file),
!                         com.bigdata.rdf.serializers.StatementSerializer.INSTANCE);
  
              }
--- 420,425 ----
              if (seg == null) {
  
!                 seg = new IndexSegmentFileStore(file).load();
! //                com.bigdata.rdf.serializers.StatementSerializer.INSTANCE);
  
              }

[cweb-CVS] bigdata/src/test/com/bigdata/objndx TestIndexSegmentFastLeafScan.java, NONE, 1.1 TestIndexSegmentWithBloomFilter.java, 1.8, 1.9 TestIndexSegmentBuilderWithSmallTree.java, 1.9, 1.10 TestIndexSegmentBuilderWithLargeTrees.java, 1.10, 1.11 AbstractBTreeTestCase.java, 1.30, 1.31 TestAll.java, 1.30, 1.31 TestIndexSegmentMerger.java, 1.4, 1.5

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv15000/src/test/com/bigdata/objndx

Modified Files:
	TestIndexSegmentWithBloomFilter.java 
	TestIndexSegmentBuilderWithSmallTree.java 
	TestIndexSegmentBuilderWithLargeTrees.java 
	AbstractBTreeTestCase.java TestAll.java 
	TestIndexSegmentMerger.java 
Added Files:
	TestIndexSegmentFastLeafScan.java 
Log Message:
Updated the UML model and added a ZIP containing an HTML presentation of the model.
Working on partitioned index support.

Index: TestIndexSegmentBuilderWithLargeTrees.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx/TestIndexSegmentBuilderWithLargeTrees.java,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** TestIndexSegmentBuilderWithLargeTrees.java	22 Feb 2007 16:59:35 -0000	1.10
--- TestIndexSegmentBuilderWithLargeTrees.java	8 Mar 2007 18:14:05 -0000	1.11
***************
*** 278,286 ****
              
              IndexSegmentBuilder builder = new IndexSegmentBuilder(outFile,
!                     tmpDir, btree, m, 0.);
  
-             // @todo verify post-condition for the temporary file.
- //            assertFalse(builder.tmp)
-             
              /*
               * Verify can load the index file and that the metadata
--- 278,287 ----
              
              IndexSegmentBuilder builder = new IndexSegmentBuilder(outFile,
!                     tmpDir, btree, m, 0.
!                     /*
!                      * @todo pass in btree.nodeSer.valueSerializer
!                      */
!                     );
  
              /*
               * Verify can load the index file and that the metadata
***************
*** 291,300 ****
               */
              System.err.println("Opening index segment.");
!             final IndexSegment seg = new IndexSegment(new IndexSegmentFileStore(outFile),
!                     // setup reference queue.
!                     new HardReferenceQueue<PO>(new DefaultEvictionListener(),
!                             100, 20),
!                     // take the other parameters from the btree.
!                     btree.nodeSer.valueSerializer);
              /*
               * Verify the total index order.
--- 292,296 ----
               */
              System.err.println("Opening index segment.");
!             final IndexSegment seg = new IndexSegmentFileStore(outFile).load();
              /*
               * Verify the total index order.

Index: TestIndexSegmentWithBloomFilter.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx/TestIndexSegmentWithBloomFilter.java,v
retrieving revision 1.8
retrieving revision 1.9
diff -C2 -d -r1.8 -r1.9
*** TestIndexSegmentWithBloomFilter.java	22 Feb 2007 16:59:35 -0000	1.8
--- TestIndexSegmentWithBloomFilter.java	8 Mar 2007 18:14:05 -0000	1.9
***************
*** 283,291 ****
           */
          System.err.println("Opening index segment w/ bloom filter.");
!         final IndexSegment seg2 = new IndexSegment(new IndexSegmentFileStore(outFile2),
!                 // setup reference queue.
!                 new HardReferenceQueue<PO>(new DefaultEvictionListener(),
!                         100, 20),
!                 btree.nodeSer.valueSerializer);
  
          /*
--- 283,291 ----
           */
          System.err.println("Opening index segment w/ bloom filter.");
!         final IndexSegment seg2 = new IndexSegmentFileStore(outFile2).load();
! //                // setup reference queue.
! //                new HardReferenceQueue<PO>(new DefaultEvictionListener(),
! //                        100, 20),
! //                @todo btree.nodeSer.valueSerializer);
  
          /*
***************
*** 378,386 ****
               */
              System.err.println("Opening index segment w/o bloom filter.");
!             final IndexSegment seg = new IndexSegment(new IndexSegmentFileStore(outFile),
!                     // setup reference queue.
!                     new HardReferenceQueue<PO>(new DefaultEvictionListener(),
!                             100, 20),
!                             btree.nodeSer.valueSerializer);
  
              /*
--- 378,386 ----
               */
              System.err.println("Opening index segment w/o bloom filter.");
!             final IndexSegment seg = new IndexSegmentFileStore(outFile).load();
! //                    // setup reference queue.
! //                    new HardReferenceQueue<PO>(new DefaultEvictionListener(),
! //                            100, 20),
! //                            @todo btree.nodeSer.valueSerializer);
  
              /*
***************
*** 392,400 ****
               */
              System.err.println("Opening index segment w/ bloom filter.");
!             final IndexSegment seg2 = new IndexSegment(new IndexSegmentFileStore(outFile2),
!                     // setup reference queue.
!                     new HardReferenceQueue<PO>(new DefaultEvictionListener(),
!                             100, 20),
!                             btree.nodeSer.valueSerializer);
  
              /*
--- 392,400 ----
               */
              System.err.println("Opening index segment w/ bloom filter.");
!             final IndexSegment seg2 = new IndexSegmentFileStore(outFile2).load();
! //                    // setup reference queue.
! //                    new HardReferenceQueue<PO>(new DefaultEvictionListener(),
! //                            100, 20),
! //                            @todo btree.nodeSer.valueSerializer);
  
              /*

Index: TestIndexSegmentBuilderWithSmallTree.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx/TestIndexSegmentBuilderWithSmallTree.java,v
retrieving revision 1.9
retrieving revision 1.10
diff -C2 -d -r1.9 -r1.10
*** TestIndexSegmentBuilderWithSmallTree.java	29 Jan 2007 20:37:38 -0000	1.9
--- TestIndexSegmentBuilderWithSmallTree.java	8 Mar 2007 18:14:05 -0000	1.10
***************
*** 7,12 ****
  import org.apache.log4j.Level;
  
- import com.bigdata.cache.HardReferenceQueue;
- 
  /**
   * Test suite based on a small btree with known keys and values.
--- 7,10 ----
***************
*** 124,133 ****
            * specifics of the length of serialized nodes or leaves).
            */
!         final IndexSegment seg = new IndexSegment(new IndexSegmentFileStore(outFile),
!                 // setup reference queue to hold all leaves and nodes.
!                 new HardReferenceQueue<PO>(new DefaultEvictionListener(),
!                         7, 7),
!                 // take the other parameters from the btree.
!                 btree.nodeSer.valueSerializer);
          TestIndexSegmentPlan.assertEquals(3,seg.getBranchingFactor());
          TestIndexSegmentPlan.assertEquals(2,seg.getHeight());
--- 122,131 ----
            * specifics of the length of serialized nodes or leaves).
            */
!         final IndexSegment seg = new IndexSegmentFileStore(outFile).load();
! //                // setup reference queue to hold all leaves and nodes.
! //                new HardReferenceQueue<PO>(new DefaultEvictionListener(),
! //                        7, 7),
! //                // take the other parameters from the btree.
! //                @todo btree.nodeSer.valueSerializer);
          TestIndexSegmentPlan.assertEquals(3,seg.getBranchingFactor());
          TestIndexSegmentPlan.assertEquals(2,seg.getHeight());
***************
*** 198,208 ****
            * specifics of the length of serialized nodes or leaves).
            */
!         final IndexSegment seg = new IndexSegment(new IndexSegmentFileStore(outFile),
!                 // setup reference queue to hold all leaves and nodes.
!                 new HardReferenceQueue<PO>(new DefaultEvictionListener(),
!                         3, 3),
!                 // take the other parameters from the btree.
!                 btree.nodeSer.valueSerializer
!                 );
          TestIndexSegmentPlan.assertEquals(9,seg.getBranchingFactor());
          TestIndexSegmentPlan.assertEquals(1,seg.getHeight());
--- 196,206 ----
            * specifics of the length of serialized nodes or leaves).
            */
!         final IndexSegment seg = new IndexSegmentFileStore(outFile).load();
! //                // setup reference queue to hold all leaves and nodes.
! //                new HardReferenceQueue<PO>(new DefaultEvictionListener(),
! //                        3, 3),
! //                // take the other parameters from the btree.
! //                @todo btree.nodeSer.valueSerializer
! //                );
          TestIndexSegmentPlan.assertEquals(9,seg.getBranchingFactor());
          TestIndexSegmentPlan.assertEquals(1,seg.getHeight());
***************
*** 258,268 ****
               * of serialized nodes or leaves).
               */
!         final IndexSegment seg = new IndexSegment(new IndexSegmentFileStore(outFile),
!                 // setup reference queue to hold all leaves and nodes.
!                 new HardReferenceQueue<PO>(new DefaultEvictionListener(),
!                         1, 1),
!                 // take the other parameters from the btree.
!                         btree.nodeSer.valueSerializer
!                 );
          TestIndexSegmentPlan.assertEquals(10,seg.getBranchingFactor());
          TestIndexSegmentPlan.assertEquals(0,seg.getHeight());
--- 256,266 ----
               * of serialized nodes or leaves).
               */
!         final IndexSegment seg = new IndexSegmentFileStore(outFile).load();
! //                // setup reference queue to hold all leaves and nodes.
! //                new HardReferenceQueue<PO>(new DefaultEvictionListener(),
! //                        1, 1),
! //                // take the other parameters from the btree.
! //                        @todo btree.nodeSer.valueSerializer
! //                );
          TestIndexSegmentPlan.assertEquals(10,seg.getBranchingFactor());
          TestIndexSegmentPlan.assertEquals(0,seg.getHeight());
***************
*** 339,348 ****
            * specifics of the length of serialized nodes or leaves).
            */
!         final IndexSegment seg = new IndexSegment(new IndexSegmentFileStore(outFile),
!                 // setup reference queue to hold all leaves and nodes.
!                 new HardReferenceQueue<PO>(new DefaultEvictionListener(),
!                         3, 3),
!                         btree.nodeSer.valueSerializer
!                 );
          TestIndexSegmentPlan.assertEquals(3,seg.getBranchingFactor());
          TestIndexSegmentPlan.assertEquals(1,seg.getHeight());
--- 337,346 ----
            * specifics of the length of serialized nodes or leaves).
            */
!         final IndexSegment seg = new IndexSegmentFileStore(outFile).load();
! //                // setup reference queue to hold all leaves and nodes.
! //                new HardReferenceQueue<PO>(new DefaultEvictionListener(),
! //                        3, 3),
! //                        @todo btree.nodeSer.valueSerializer
! //                );
          TestIndexSegmentPlan.assertEquals(3,seg.getBranchingFactor());
          TestIndexSegmentPlan.assertEquals(1,seg.getHeight());
***************
*** 432,441 ****
            * specifics of the length of serialized nodes or leaves).
            */
!         final IndexSegment seg = new IndexSegment(new IndexSegmentFileStore(outFile),
!                 // setup reference queue to hold all leaves and nodes.
!                 new HardReferenceQueue<PO>(new DefaultEvictionListener(),
!                         11, 11),
!                         btree.nodeSer.valueSerializer
!                 );
          TestIndexSegmentPlan.assertEquals(3,seg.getBranchingFactor());
          TestIndexSegmentPlan.assertEquals(2,seg.getHeight());
--- 430,439 ----
            * specifics of the length of serialized nodes or leaves).
            */
!         final IndexSegment seg = new IndexSegmentFileStore(outFile).load();
! //                // setup reference queue to hold all leaves and nodes.
! //                new HardReferenceQueue<PO>(new DefaultEvictionListener(),
! //                        11, 11),
! //                        @todo btree.nodeSer.valueSerializer
! //                );
          TestIndexSegmentPlan.assertEquals(3,seg.getBranchingFactor());
          TestIndexSegmentPlan.assertEquals(2,seg.getHeight());

Index: TestAll.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx/TestAll.java,v
retrieving revision 1.30
retrieving revision 1.31
diff -C2 -d -r1.30 -r1.31
*** TestAll.java	21 Feb 2007 20:17:22 -0000	1.30
--- TestAll.java	8 Mar 2007 18:14:05 -0000	1.31
***************
*** 126,129 ****
--- 126,131 ----
          // test of the bloom filter integration.
          suite.addTestSuite( TestIndexSegmentWithBloomFilter.class );
+         // test of the fast forward and reverse leaf scans.
+         suite.addTestSuite( TestIndexSegmentFastLeafScan.class );
          // test compacting merge of two index segments.
          suite.addTestSuite( TestIndexSegmentMerger.class );

Index: AbstractBTreeTestCase.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx/AbstractBTreeTestCase.java,v
retrieving revision 1.30
retrieving revision 1.31
diff -C2 -d -r1.30 -r1.31
*** AbstractBTreeTestCase.java	22 Feb 2007 16:59:35 -0000	1.30
--- AbstractBTreeTestCase.java	8 Mar 2007 18:14:05 -0000	1.31
***************
*** 1660,1664 ****
              try {
  
!                 assertEquals(expectedVal, actualVal);
                  
              } catch (AssertionFailedError ex) {
--- 1660,1664 ----
              try {
  
!                 assertSameValue(expectedVal, actualVal);
                  
              } catch (AssertionFailedError ex) {
***************
*** 1666,1672 ****
                   * Lazily generate message.
                   */
!                 fail("Values differ: index=" + index + ", key="
!                         + BytesUtil.toString(expectedKey) + ", expected="
!                         + expectedVal + ", actual=" + actualVal, ex);
                  
              }
--- 1666,1679 ----
                   * Lazily generate message.
                   */
!                 fail("Values differ: index="
!                         + index
!                         + ", key="
!                         + BytesUtil.toString(expectedKey)
!                         + ", expected="
!                         + (expectedVal instanceof byte[] ? BytesUtil
!                                 .toString((byte[]) expectedVal) : expectedVal)
!                         + ", actual="
!                         + (actualVal instanceof byte[] ? BytesUtil
!                                 .toString((byte[]) actualVal) : actualVal), ex);
                  
              }

--- NEW FILE: TestIndexSegmentFastLeafScan.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Mar 7, 2007
 */

package com.bigdata.objndx;

/**
 * Test suite for verification of fast forward and reverse leaf scans for
 * {@link IndexSegment}.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public class TestIndexSegmentFastLeafScan extends AbstractBTreeTestCase {

    /**
     * 
     */
    public TestIndexSegmentFastLeafScan() {
    }

    /**
     * @param name
     */
    public TestIndexSegmentFastLeafScan(String name) {
        super(name);
    }

    public void test_fastForwardScan() {
        
        fail("write test");

    }
    
    public void test_fastReverseScan() {

        fail("write test");
        
    }
    
}

Index: TestIndexSegmentMerger.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx/TestIndexSegmentMerger.java,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** TestIndexSegmentMerger.java	6 Mar 2007 20:38:06 -0000	1.4
--- TestIndexSegmentMerger.java	8 Mar 2007 18:14:05 -0000	1.5
***************
*** 48,52 ****
  package com.bigdata.objndx;
  
- import java.io.File;
  import java.io.IOException;
  
--- 48,51 ----
***************
*** 56,59 ****
--- 55,63 ----
   * Test suite for compacting merge of B+-Trees.
   * 
+  * @todo write tests of a key range merge (feature not supported yet, but
+  *       required for partitioned index segment builds).
+  * 
+  * @todo test N-way merge.
+  * 
   * @todo write tests where the keys do not overlap (done).
   * 
***************
*** 103,119 ****
          super(name);
      }
- 
-     /**
-      * Return the temporary directory.
-      */
-     static synchronized protected File getTempDir() {
-         
-         File tmpDir = new File(System.getProperty("java.io.tmpdir"));
- 
-         assertTrue(!tmpDir.exists() || tmpDir.mkdirs());
- 
-         return tmpDir;
-         
-     }
      
      /**
--- 107,110 ----

[cweb-CVS] bigdata perf.txt,NONE,1.1 .cvsignore,1.10,1.11

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv15000

Modified Files:
	.cvsignore 
Added Files:
	perf.txt 
Log Message:
Updated the UML model and added a ZIP containing an HTML presentation of the model.
Working on partitioned index support.

Index: .cvsignore
===================================================================
RCS file: /cvsroot/cweb/bigdata/.cvsignore,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** .cvsignore	9 Feb 2007 18:56:59 -0000	1.10
--- .cvsignore	8 Mar 2007 18:14:05 -0000	1.11
***************
*** 10,11 ****
--- 10,12 ----
  *.tmp
  test_*
+ com.bigdata.journal.StressTestConcurrent.comparison.csv

--- NEW FILE: perf.txt ---

1. Transactions per second target of 8k.  Implement TCP A and C.
   Implement group commit.  See
   http://blogs.oracle.com/charleslamb/2006/09/22#a14.  Is the disk
   write cache enabled or disabled?  Are transactions really durable?
   http://www.soliddata.com/pdf/TimesTen.pdf discusses write-through
   RAID vs write-behind RAID vs solid state for log files.  Since
   journals are size limited, a very small (.5 - 1G) solid state drive
   would sufficient.  Old journals are just transferred to normal
   disk.  Note that solid state drives do eventually degrade with use
   and the journal will cause repeated overwrites of data on the solid
   state disk.
   
   Consider and minimize startup costs for an index.  When the index
   is isolated by a transaction and we are seeking high TPS rates,
   then those startup and run costs need to be optimized to reduce
   startup latency and also run latency for short transactions.

MROW performance test: 2/20/07.

Note that AIO support is not implemented.  The performance is CPU
bound for all cases except Disk only.  However, note that this is with
concurrent clients.  When using a single client, direct mode
performance might be IO bound with a single client since the Direct
mode does not use AIO.  The disk queue length is short except on
sustained writes (pre-writes before the readers start).

Significantly less performance is obtained when using a single client.

Transient(heap)  : #clients=20, ntrials=100000, nok=39691, ncancelled=60309, nerrors=0 in 10094ms (3932 reads per second); nwritten=2728
Transient(direct): #clients=20, ntrials=100000, nok=41027, ncancelled=58973, nerrors=0 in 10297ms (3984 reads per second); nwritten=2739
Direct(direct)   : #clients=20, ntrials=100000, nok=38776, ncancelled=61224, nerrors=0 in 10047ms (3859 reads per second); nwritten=2716
Direct(heap)     : #clients=20, ntrials=100000, nok=39030, ncancelled=60970, nerrors=0 in 10063ms (3878 reads per second); nwritten=2739
Mapped           : #clients=20, ntrials=100000, nok=38801, ncancelled=61199, nerrors=0 in 10094ms (3843 reads per second); nwritten=2731
Disk		 : #clients=20, ntrials=100000, nok=12671, ncancelled=87329, nerrors=0 in 10047ms (1261 reads per second); nwritten=2784

Note: Default VM args using JRockit Jrockit R26.4.0 (jdk1.5.0_06)

--------------------

Concurrent client transaction stress test: 2/21/07

JVM: Jrockit R26.4.0 (jdk1.5.0_06)

VM ARGS: -server -Xms1g -Xmx1g -XX:MaxDirectMemorySize=256M

Test at clients in { 1, 2, 10, 20, 100, 200 }.

mode=Transient, #clients=1, nops=100, ntx=10000, ncomitted=3989, naborted=444, nfailed=0, nuncommitted=6011 in 30219ms (132 tps)
mode=Transient, #clients=2, nops=100, ntx=10000, ncomitted=4244, naborted=457, nfailed=12, nuncommitted=5744 in 30000ms (141 tps)
mode=Transient, #clients=10, nops=100, ntx=10000, ncomitted=4316, naborted=483, nfailed=11, nuncommitted=5673 in 30032ms (143 tps)
mode=Transient, #clients=20, nops=100, ntx=10000, ncomitted=4344, naborted=506, nfailed=11, nuncommitted=5645 in 30015ms (144 tps)
mode=Transient, #clients=100, nops=100, ntx=10000, ncomitted=4060, naborted=486, nfailed=11, nuncommitted=5929 in 30016ms (135 tps)
mode=Transient, #clients=200, nops=100, ntx=10000, ncomitted=4044, naborted=494, nfailed=13, nuncommitted=5943 in 33390ms (121 tps)

mode=Direct, #clients=1, nops=100, ntx=10000, ncomitted=601, naborted=70, nfailed=0, nuncommitted=9399 in 30000ms (20 tps)
mode=Direct, #clients=2, nops=100, ntx=10000, ncomitted=628, naborted=64, nfailed=0, nuncommitted=9372 in 30000ms (20 tps)
mode=Direct, #clients=10, nops=100, ntx=10000, ncomitted=633, naborted=81, nfailed=1, nuncommitted=9366 in 30000ms (21 tps)
mode=Direct, #clients=20, nops=100, ntx=10000, ncomitted=633, naborted=77, nfailed=1, nuncommitted=9366 in 30031ms (21 tps)
mode=Direct, #clients=100, nops=100, ntx=10000, ncomitted=639, naborted=91, nfailed=0, nuncommitted=9361 in 30016ms (21 tps)
mode=Direct, #clients=200, nops=100, ntx=10000, ncomitted=643, naborted=103, nfailed=0, nuncommitted=9357 in 30015ms (21 tps)

mode=Disk, #clients=1, nops=100, ntx=10000, ncomitted=604, naborted=59, nfailed=0, nuncommitted=9396 in 30015ms (20 tps)
mode=Disk, #clients=2, nops=100, ntx=10000, ncomitted=586, naborted=62, nfailed=0, nuncommitted=9414 in 30000ms (19 tps)
mode=Disk, #clients=10, nops=100, ntx=10000, ncomitted=660, naborted=89, nfailed=0, nuncommitted=9340 in 30000ms (22 tps)
mode=Disk, #clients=20, nops=100, ntx=10000, ncomitted=629, naborted=67, nfailed=0, nuncommitted=9371 in 30016ms (20 tps)
mode=Disk, #clients=100, nops=100, ntx=10000, ncomitted=665, naborted=106, nfailed=0, nuncommitted=9335 in 30016ms (22 tps)
mode=Disk, #clients=200, nops=100, ntx=10000, ncomitted=623, naborted=83, nfailed=0, nuncommitted=9377 in 30000ms (20 tps)

w/ Options.FORCE := No

mode=Direct, #clients=10, nops=100, ntx=10000, ncomitted=3337, naborted=350, nfailed=3, nuncommitted=6660 in 30062ms (111 tps)
mode=Disk, #clients=200, nops=100, ntx=10000, ncomitted=3242, naborted=416, nfailed=7, nuncommitted=6751 in 30015ms (108 tps)

w/ Options.FORCE := Force (data, but not metadata)

mode=Disk, #clients=200, nops=100, ntx=10000, ncomitted=635, naborted=92, nfailed=1, nuncommitted=9364 in 30015ms (21 tps)


Note: Page faults per second is quite high for this stress test and
that needs to be looked at futher.  I was running MS Outlook as well
at the time that these measures were collected, so there might be
memory contention on the host (Dell laptop; Windows XP/Pro. 2G ram.)

Note: TPS is basically constant for a given combination of the buffer
mode and whether or not commits are forced to disk.  This means that
the #of clients is not a strong influence on performance.  The big
wins are Transient and Force := No since neither conditions synchs to
disk.  This suggests that the big win for TPS throughput is going to
be group commit.

[cweb-CVS] bigdata/src/test/com/bigdata/rawstore SimpleMemoryRawStore.java, NONE, 1.1 SimpleFileRawStore.java, NONE, 1.1

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/test/com/bigdata/rawstore
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv15000/src/test/com/bigdata/rawstore

Added Files:
	SimpleMemoryRawStore.java SimpleFileRawStore.java 
Log Message:
Updated the UML model and added a ZIP containing an HTML presentation of the model.
Working on partitioned index support.

--- NEW FILE: SimpleFileRawStore.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Jan 31, 2007
 */

package com.bigdata.rawstore;

import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.ByteBuffer;
import java.util.HashSet;
import java.util.Set;

import com.bigdata.journal.TemporaryRawStore;


/**
 * A simple persistent unbuffered implementation backed by a file. The maximum
 * size of the file is restricted to {@link Integer#MAX_VALUE} bytes since we
 * must code the offset into the file using {@link Addr#toLong(int, int)}.
 * 
 * @see {@link TemporaryRawStore}, which provides a more solution for temporary
 *      data that begins with the benefits of a memory-resident buffer and then
 *      converts to a disk-based store on overflow.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public class SimpleFileRawStore implements IRawStore {

    private boolean open = true;
    
    public final File file;
    protected final RandomAccessFile raf;
    
    /**
     * This provides a purely transient means to identify deleted records.  This
     * data does NOT survive restart of the store. 
     */
    private final Set<Long> deleted = new HashSet<Long>();

    /**
     * Open a store. The file will be created if it does not exist and it is
     * opened for writing. If the file is opened for writing, then an exception
     * will be thrown unless an exclusive lock can be obtained on the file.
     * 
     * @param file
     *            The name of the file to use as the backing store.
     * @param mode
     *            The file open mode for
     *            {@link RandomAccessFile#RandomAccessFile(File, String)()}.
     */
    public SimpleFileRawStore(File file, String mode) throws IOException {
        
        if (file == null)
            throw new IllegalArgumentException("file is null");
        
        this.file = file;
        
        raf = new RandomAccessFile(file,mode);

        if( mode.indexOf("w") != -1 ) {

            if (raf.getChannel().tryLock() == null) {

                throw new IOException("Could not lock file: "
                        + file.getAbsoluteFile());

            }
            
        }
        
    }
    
    public boolean isOpen() {

        return open;
        
    }
    
    public boolean isStable() {
        
        return true;
        
    }

    public boolean isFullyBuffered() {
        
        return false;
        
    }
    
    public File getFile() {
        
        return file;
        
    }
    
    /**
     * This also releases the lock if any obtained by the constructor.
     */
    public void close() {
        
        if( !open ) throw new IllegalStateException();

        open = false;

        try {

            raf.close();
            
        } catch(IOException ex) { 
            
            throw new RuntimeException(ex);
            
        }
        
    }

    public void closeAndDelete() {
        
        close();
        
        if(!file.delete()) {
            
            System.err.println("WARN: Could not delete: "+file.getAbsolutePath());
            
        }
        
    }

    public ByteBuffer read(long addr) {

        if (addr == 0L)
            throw new IllegalArgumentException("Address is 0L");

        final int offset = Addr.getOffset(addr);

        final int nbytes = Addr.getByteCount(addr);

        if (nbytes == 0) {

            throw new IllegalArgumentException(
                    "Address encodes record length of zero");

        }

        if (deleted.contains(addr)) {

            throw new IllegalArgumentException(
                    "Address was deleted in this session");

        }

        try {

            if (offset + nbytes > raf.length()) {

                throw new IllegalArgumentException("Address never written.");

            }

            // allocate a new buffer of the exact capacity.

            ByteBuffer dst = ByteBuffer.allocate(nbytes);

            // copy the data into the buffer.

            raf.getChannel().read(dst, (long) offset);

            // flip for reading.

            dst.flip();

            // return the buffer.

            return dst;

        } catch (IOException ex) {

            throw new RuntimeException(ex);

        }
        
    }

    public long write(ByteBuffer data) {

        if (data == null)
            throw new IllegalArgumentException("Buffer is null");

        // #of bytes to store.
        final int nbytes = data.remaining();

        if (nbytes == 0)
            throw new IllegalArgumentException("No bytes remaining in buffer");

        try {

            // the new record will be appended to the end of the file.
            long pos = raf.length();

            if (pos + nbytes > Integer.MAX_VALUE) {

                throw new IOException("Would exceed int32 bytes in file.");

            }

            // the offset into the file at which the record will be written.
            final int offset = (int) pos;

            // // extend the file to have sufficient space for this record.
            // raf.setLength(pos + nbytes);

            // write the data onto the end of the file.
            raf.getChannel().write(data, pos);

            // formulate the address that can be used to recover that record.
            return Addr.toLong(nbytes, offset);

        } catch (IOException ex) {

            throw new RuntimeException(ex);

        }

    }

//    /**
//     * Note: the delete implementation checks its arguments and makes a
//     * <em>transient</em> note that the record has been deleted but that
//     * information does NOT survive restart of the store.
//     */
//    public void delete(long addr) {
//
//        if(addr==0L) throw new IllegalArgumentException("Address is 0L");
//        
//        final int offset = Addr.getOffset(addr);
//        
//        final int nbytes = Addr.getByteCount(addr);
//
//        if(nbytes==0) {
//            
//            throw new IllegalArgumentException(
//                    "Address encodes record length of zero");
//            
//        }
//        
//        try {
//
//            if (offset + nbytes > raf.length()) {
//
//                throw new IllegalArgumentException("Address never written.");
//
//            }
//
//        } catch (IOException ex) {
//
//            throw new RuntimeException(ex);
//
//        }
//
//        Long l = Long.valueOf(addr);
//        
//        if(deleted.contains(l)) {
//        
//            throw new IllegalArgumentException("Address was deleted in this session");
//            
//        }
//        
//        deleted.add(l);
//        
//    }

    public void force(boolean metadata) {

        try {
            
            raf.getChannel().force(metadata);
            
        } catch( IOException ex) {
            
            throw new RuntimeException(ex);
            
        }
                
    }

    public long size() {
        
        try {
        
            return raf.length();
            
        } catch(IOException ex) {
            
            throw new RuntimeException(ex);
            
        }
        
    }
    
}

--- NEW FILE: SimpleMemoryRawStore.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Jan 31, 2007
 */

package com.bigdata.rawstore;

import java.io.File;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Map;

import com.bigdata.journal.TemporaryRawStore;

/**
 * A purely transient append-only implementation useful when data need to be
 * buffered in memory. The writes are stored in an {@link ArrayList}.
 * 
 * @see {@link TemporaryRawStore}, which provides a more scalable solution for temporary
 *      data.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public class SimpleMemoryRawStore implements IRawStore {

    private boolean open = true;
    
    /**
     * The #of bytes written so far. This is used to generate the
     * {@link Addr address} values returned by {@link #write(ByteBuffer)}. This
     * is necessary in order for this implementation to assign addresses in the
     * same manner as they would be assigned by an implementation using an
     * append only byte[] or file.
     */
    protected int nextOffset = 0;
    
    /**
     * Maps an {@link Addr} onto the index in {@link #records} at which the
     * data for that address was written.
     */
    private final Map<Long,Integer> addrs;
    
    /**
     * The buffered records in the order written. If a record is deleted then
     * that element in the list will be a [null] value.
     */
    protected final ArrayList<byte[]> records;
    
    /**
     * Uses an initial capacity of 1000 records.
     */
    public SimpleMemoryRawStore() {

        this(1000);
        
    }
    
    /**
     * 
     * @param capacity
     *            The #of records that you expect to store (non-negative). If
     *            the capacity is exceeded then the internal {@link ArrayList}
     *            will be grown as necessary.
     */
    public SimpleMemoryRawStore(int capacity) {
        
        if (capacity < 0)
            throw new IllegalArgumentException("capacity is negative");
        
        records = new ArrayList<byte[]>(capacity);
        
        // estimate hash table capacity to avoid resizing.
        addrs = new HashMap<Long,Integer>((int)(capacity*1.25));

    }
    
    public boolean isOpen() {

        return open;
        
    }

    public boolean isStable() {
        
        return false;
        
    }

    public boolean isFullyBuffered() {
        
        return true;
        
    }
    
    /**
     * This always returns <code>null</code>.
     */
    public File getFile() {
        
        return null;
        
    }
    
    public void close() {
        
        if( !open ) throw new IllegalStateException();

        open = false;
        
        // discard all the records.
        records.clear();
        
    }

    public void closeAndDelete() {
        
        close();
        
    }

    public ByteBuffer read(long addr) {

        if (addr == 0L)
            throw new IllegalArgumentException("Address is 0L");
        
//        final int offset = Addr.getOffset(addr);
        
        final int nbytes = Addr.getByteCount(addr);

        if(nbytes==0) {
            
            throw new IllegalArgumentException(
                    "Address encodes record length of zero");
            
        }
        
        Integer index = addrs.get(addr);
        
        if(index==null) {
            
            throw new IllegalArgumentException("Address never written.");

        }

        final byte[] b = records.get(index);

        if(b == null) {
            
            throw new IllegalArgumentException("Record was deleted");
            
        }
        
        if(b.length != nbytes) {
            
            throw new RuntimeException("Bad address / data");
            
        }
        
        // return a read-only view onto the data in the store.
            
        return ByteBuffer.wrap(b).asReadOnlyBuffer();
        
    }

    public long write(ByteBuffer data) {

        if (data == null)
            throw new IllegalArgumentException("Buffer is null");

        // #of bytes to store.
        final int nbytes = data.remaining();

        if (nbytes == 0)
            throw new IllegalArgumentException("No bytes remaining in buffer");

        // allocate a new array that is an exact fit for the data.
        final byte[] b = new byte[nbytes];

        // copy the data into the array.
        data.get(b);
        
        // the next offset.
        final int offset = nextOffset;
        
        // increment by the #of bytes written.
        nextOffset += nbytes;

        // the position in the records[] where this record is stored.
        final int index = records.size();
        
        // add the record to the records array.
        records.add(b);
        
        // formulate the address that can be used to recover that record.
        long addr = Addr.toLong(nbytes, offset);
        
        // save the mapping from the addr to the records[].
        addrs.put(addr,index);
        
        return addr;
        
    }

//    public void delete(long addr) {
//
//        if(addr==0L) throw new IllegalArgumentException("Address is 0L");
//        
////        final int offset = Addr.getOffset(addr);
//        
//        final int nbytes = Addr.getByteCount(addr);
//
//        if(nbytes==0) {
//            
//            throw new IllegalArgumentException(
//                    "Address encodes record length of zero");
//            
//        }
//        
//        Integer index = addrs.get(addr);
//        
//        if(index==null) {
//         
//            throw new IllegalArgumentException("Address never written.");
//
//        }
//        
//        byte[] b = records.get(index);
//
//        if(b == null) {
//            
//            throw new IllegalArgumentException("Record was deleted");
//            
//        }
//
//        if(b.length != nbytes) {
//            
//            throw new RuntimeException("Bad address / data");
//            
//        }
//        
//        // release that record.
//        records.set(index, null);
//        
//    }

    public void force(boolean metadata) {
        
        // NOP.
        
    }
    
    public long size() {
        
        return nextOffset;
        
    }
    
}

[cweb-CVS] bigdata/src/java/com/bigdata/journal RunState.java, 1.1, 1.2 TemporaryStore.java, 1.6, 1.7 IJournal.java, 1.7, 1.8 IStore.java, 1.7, 1.8 CommitRecord.java, 1.2, 1.3 TransactionServer.java, 1.5, 1.6 IMROW.java, 1.1, 1.2 IAtomicStore.java, 1.4, 1.5 AbstractTx.java, 1.1, 1.2 Journal.java, 1.58, 1.59 IIndexManager.java, 1.1, 1.2

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/journal
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv15000/src/java/com/bigdata/journal

Modified Files:
	RunState.java TemporaryStore.java IJournal.java IStore.java 
	CommitRecord.java TransactionServer.java IMROW.java 
	IAtomicStore.java AbstractTx.java Journal.java 
	IIndexManager.java 
Log Message:
Updated the UML model and added a ZIP containing an HTML presentation of the model.
Working on partitioned index support.

Index: IJournal.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/journal/IJournal.java,v
retrieving revision 1.7
retrieving revision 1.8
diff -C2 -d -r1.7 -r1.8
*** IJournal.java	22 Feb 2007 16:59:34 -0000	1.7
--- IJournal.java	8 Mar 2007 18:14:06 -0000	1.8
***************
*** 51,55 ****
  
  import com.bigdata.objndx.IIndex;
- import com.bigdata.rawstore.IRawStore;
  
  /**
--- 51,54 ----
***************
*** 63,68 ****
   * @version $Id$
   */
! public interface IJournal extends IMROW, IRawStore, IAtomicStore,
!         IIndexManager, ITransactionManager {
  
      /**
--- 62,67 ----
   * @version $Id$
   */
! public interface IJournal extends IMROW, IAtomicStore, IIndexManager,
!         ITransactionManager {
  
      /**
***************
*** 102,127 ****
      public IIndex getIndex(String name, long startTime);
  
-     /**
-      * Return a read-only view of the current root block.
-      * 
-      * @return The current root block.
-      */
-     public IRootBlockView getRootBlockView();
- 
-     /**
-      * Return the {@link ICommitRecord} for the most recent committed state
-      * whose commit timestamp is less than or equal to <i>timestamp</i>. This
-      * is used by a {@link Tx transaction} to locate the committed state that is
-      * the basis for its operations.
-      * 
-      * @param timestamp
-      *            Typically, the timestamp assigned to a transaction.
-      * 
-      * @return The {@link ICommitRecord} for the most recent committed state
-      *         whose commit timestamp is less than or equal to <i>timestamp</i>
-      *         -or- <code>null</code> iff there are no {@link ICommitRecord}s
-      *         that satisify the probe.
-      */
-     public ICommitRecord getCommitRecord(long commitTime);
- 
  }
--- 101,103 ----

Index: CommitRecord.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/journal/CommitRecord.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** CommitRecord.java	21 Feb 2007 20:17:21 -0000	1.2
--- CommitRecord.java	8 Mar 2007 18:14:06 -0000	1.3
***************
*** 53,59 ****
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
   * @version $Id$
-  * 
-  * @todo can I make the roots named in some less scalable manner? E.g., an array
-  *       list or hash table?
   */
  public class CommitRecord implements ICommitRecord {
--- 53,56 ----

Index: IAtomicStore.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/journal/IAtomicStore.java,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** IAtomicStore.java	17 Feb 2007 21:34:17 -0000	1.4
--- IAtomicStore.java	8 Mar 2007 18:14:07 -0000	1.5
***************
*** 97,101 ****
       * Note: the committers must be reset after restart or whenever 
       * 
!      * @param rootSlot
       *            The slot in the root block where the {@link Addr address} of
       *            the {@link ICommitter} will be recorded.
--- 97,101 ----
       * Note: the committers must be reset after restart or whenever 
       * 
!      * @param index
       *            The slot in the root block where the {@link Addr address} of
       *            the {@link ICommitter} will be recorded.
***************
*** 104,108 ****
       *            The commiter.
       */
!     public void setCommitter(int rootSlot, ICommitter committer);
  
      /**
--- 104,108 ----
       *            The commiter.
       */
!     public void setCommitter(int index, ICommitter committer);
  
      /**
***************
*** 120,122 ****
--- 120,145 ----
      public long getRootAddr(int index);
  
+     /**
+      * Return a read-only view of the current root block.
+      * 
+      * @return The current root block.
+      */
+     public IRootBlockView getRootBlockView();
+ 
+     /**
+      * Return the {@link ICommitRecord} for the most recent committed state
+      * whose commit timestamp is less than or equal to <i>timestamp</i>. This
+      * is used by a {@link Tx transaction} to locate the committed state that is
+      * the basis for its operations.
+      * 
+      * @param timestamp
+      *            Typically, the timestamp assigned to a transaction.
+      * 
+      * @return The {@link ICommitRecord} for the most recent committed state
+      *         whose commit timestamp is less than or equal to <i>timestamp</i>
+      *         -or- <code>null</code> iff there are no {@link ICommitRecord}s
+      *         that satisify the probe.
+      */
+     public ICommitRecord getCommitRecord(long commitTime);
+ 
  }

Index: TransactionServer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/journal/TransactionServer.java,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** TransactionServer.java	28 Feb 2007 13:59:10 -0000	1.5
--- TransactionServer.java	8 Mar 2007 18:14:06 -0000	1.6
***************
*** 203,207 ****
              this.readCommitted = isolationLevel == IsolationEnum.ReadCommitted;
              
! //            runState = RunState.ACTIVE;
              
              heartbeat = System.nanoTime();
--- 203,207 ----
              this.readCommitted = isolationLevel == IsolationEnum.ReadCommitted;
              
! //            runState = RunState.Active;
              
              heartbeat = System.nanoTime();
***************
*** 567,571 ****
  // * The criteria for deallocating historical versions is that (a) there is
  // a
! // * more recent version; and (b) there is no ACTIVE (vs PENDING or
  // COMPLETED)
  // * transaction which could read from that historical version. The journal
--- 567,571 ----
  // * The criteria for deallocating historical versions is that (a) there is
  // a
! // * more recent version; and (b) there is no Active (vs PENDING or
  // COMPLETED)
  // * transaction which could read from that historical version. The journal
***************
*** 574,579 ****
  // come
  // * from a transaction service which has global knowledge of which
! // * transactions have PREPARED or ABORTED and can generate notices when all
! // * transactions before a given timestamp have been PREPARED or ABORTED.
  // For
  // * example, a long running transaction can cause notice to be delayed for
--- 574,579 ----
  // come
  // * from a transaction service which has global knowledge of which
! // * transactions have Prepared or Aborted and can generate notices when all
! // * transactions before a given timestamp have been Prepared or Aborted.
  // For
  // * example, a long running transaction can cause notice to be delayed for

Index: Journal.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/journal/Journal.java,v
retrieving revision 1.58
retrieving revision 1.59
diff -C2 -d -r1.58 -r1.59
*** Journal.java	6 Mar 2007 20:38:06 -0000	1.58
--- Journal.java	8 Mar 2007 18:14:07 -0000	1.59
***************
*** 62,65 ****
--- 62,66 ----
  import org.apache.log4j.Logger;
  
+ import com.bigdata.isolation.UnisolatedBTree;
  import com.bigdata.objndx.BTree;
  import com.bigdata.objndx.BTreeMetadata;
***************
*** 1284,1287 ****
--- 1285,1301 ----
  
      /**
+      * Registers an {@link UnisolatedBTree} that will support transactional
+      * isolation.
+      * <p>
+      * Note: You MUST {@link #commit()} before the registered index will be
+      * either restart-safe or visible to new transactions.
+      */
+     public IIndex registerIndex(String name) {
+         
+         return registerIndex( name, new UnisolatedBTree(this));
+         
+     }
+     
+     /**
       * Note: You MUST {@link #commit()} before the registered index will be
       * either restart-safe or visible to new transactions.

Index: TemporaryStore.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/journal/TemporaryStore.java,v
retrieving revision 1.6
retrieving revision 1.7
diff -C2 -d -r1.6 -r1.7
*** TemporaryStore.java	21 Feb 2007 20:17:21 -0000	1.6
--- TemporaryStore.java	8 Mar 2007 18:14:06 -0000	1.7
***************
*** 48,51 ****
--- 48,54 ----
  package com.bigdata.journal;
  
+ import com.bigdata.objndx.AbstractBTree;
+ import com.bigdata.objndx.BTree;
+ import com.bigdata.objndx.ByteArrayValueSerializer;
  import com.bigdata.objndx.IIndex;
  import com.bigdata.rawstore.Addr;
***************
*** 57,61 ****
   * @version $Id$
   */
! public class TemporaryStore extends TemporaryRawStore {
  
      /**
--- 60,64 ----
   * @version $Id$
   */
! public class TemporaryStore extends TemporaryRawStore implements IIndexManager {
  
      /**
***************
*** 100,103 ****
--- 103,117 ----
  
      }
+ 
+     /**
+      * Registers a {@link BTree} whose values are variable length byte[]s.
+      */
+     public IIndex registerIndex(String name) {
+     
+         return registerIndex(name, new BTree(this,
+                 BTree.DEFAULT_BRANCHING_FACTOR,
+                 ByteArrayValueSerializer.INSTANCE));
+         
+     }
      
      public IIndex registerIndex(String name, IIndex btree) {

Index: AbstractTx.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/journal/AbstractTx.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** AbstractTx.java	28 Feb 2007 13:59:10 -0000	1.1
--- AbstractTx.java	8 Mar 2007 18:14:07 -0000	1.2
***************
*** 117,121 ****
          journal.activateTx(this);
  
!         this.runState = RunState.ACTIVE;
  
      }
--- 117,121 ----
          journal.activateTx(this);
  
!         this.runState = RunState.Active;
  
      }
***************
*** 157,165 ****
          
          switch(runState) {
!         case ACTIVE:
!         case ABORTED:
              throw new IllegalStateException();
!         case PREPARED:
!         case COMMITTED:
              if(commitTime == 0L) throw new AssertionError();
              return commitTime;
--- 157,165 ----
          
          switch(runState) {
!         case Active:
!         case Aborted:
              throw new IllegalStateException();
!         case Prepared:
!         case Committed:
              if(commitTime == 0L) throw new AssertionError();
              return commitTime;
***************
*** 187,191 ****
      final public boolean isActive() {
          
!         return runState == RunState.ACTIVE;
          
      }
--- 187,191 ----
      final public boolean isActive() {
          
!         return runState == RunState.Active;
          
      }
***************
*** 193,197 ****
      final public boolean isPrepared() {
          
!         return runState == RunState.PREPARED;
          
      }
--- 193,197 ----
      final public boolean isPrepared() {
          
!         return runState == RunState.Prepared;
          
      }
***************
*** 199,203 ****
      final public boolean isComplete() {
          
!         return runState == RunState.COMMITTED || runState == RunState.ABORTED;
          
      }
--- 199,203 ----
      final public boolean isComplete() {
          
!         return runState == RunState.Committed || runState == RunState.Aborted;
          
      }
***************
*** 205,209 ****
      final public boolean isCommitted() {
          
!         return runState == RunState.COMMITTED;
          
      }
--- 205,209 ----
      final public boolean isCommitted() {
          
!         return runState == RunState.Committed;
          
      }
***************
*** 211,215 ****
      final public boolean isAborted() {
          
!         return runState == RunState.ABORTED;
          
      }
--- 211,215 ----
      final public boolean isAborted() {
          
!         return runState == RunState.Aborted;
          
      }
***************
*** 222,226 ****
          try {
  
!             runState = RunState.ABORTED;
  
              journal.completedTx(this);
--- 222,226 ----
          try {
  
!             runState = RunState.Aborted;
  
              journal.completedTx(this);
***************
*** 291,295 ****
          journal.prepared(this);
  
!         runState = RunState.PREPARED;
          
      }
--- 291,295 ----
          journal.prepared(this);
  
!         runState = RunState.Prepared;
          
      }
***************
*** 340,344 ****
              }
              
!             runState = RunState.COMMITTED;
  
              journal.completedTx(this);
--- 340,344 ----
              }
              
!             runState = RunState.Committed;
  
              journal.completedTx(this);

Index: RunState.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/journal/RunState.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** RunState.java	8 Nov 2006 15:18:13 -0000	1.1
--- RunState.java	8 Mar 2007 18:14:06 -0000	1.2
***************
*** 9,16 ****
  public enum RunState {
      
!     ACTIVE("active"),
!     PREPARED("prepared"),
!     COMMITTED("committed"),
!     ABORTED("aborted");
      
      private final String name;
--- 9,16 ----
  public enum RunState {
      
!     Active("Active"),
!     Prepared("Prepared"),
!     Committed("Committed"),
!     Aborted("Aborted");
      
      private final String name;

Index: IStore.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/journal/IStore.java,v
retrieving revision 1.7
retrieving revision 1.8
diff -C2 -d -r1.7 -r1.8
*** IStore.java	9 Feb 2007 16:13:18 -0000	1.7
--- IStore.java	8 Mar 2007 18:14:06 -0000	1.8
***************
*** 72,76 ****
      public IIndex getIndex(String name);
      
- //    public ITx startTx(long txId);
-     
  }
--- 72,74 ----

Index: IMROW.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/journal/IMROW.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** IMROW.java	21 Feb 2007 20:17:21 -0000	1.1
--- IMROW.java	8 Mar 2007 18:14:06 -0000	1.2
***************
*** 56,60 ****
   * @version $Id$
   */
! public interface IMROW extends IRawStore {
  
  }
--- 56,60 ----
   * @version $Id$
   */
! public interface IMROW /*extends IRawStore*/ {
  
  }

Index: IIndexManager.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/journal/IIndexManager.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** IIndexManager.java	17 Feb 2007 21:34:18 -0000	1.1
--- IIndexManager.java	8 Mar 2007 18:14:07 -0000	1.2
***************
*** 65,68 ****
--- 65,82 ----
       * Note: A named index must be registered outside of any transaction before
       * it may be used inside of a transaction.
+      * 
+      * @param name
+      *            The name that can be used to recover the index.
+      * 
+      * @return The object that would be returned by {@link #getIndex(String)}.
+      */
+     public IIndex registerIndex(String name);
+ 
+     /**
+      * Register a named index. Once registered the index will participate in
+      * atomic commits.
+      * <p>
+      * Note: A named index must be registered outside of any transaction before
+      * it may be used inside of a transaction.
       * <p>
       * Note: The return object MAY differ from the supplied {@link BTree}. For
***************
*** 82,86 ****
       */
      public IIndex registerIndex(String name, IIndex btree);
!     
      /**
       * Drops the named index (unisolated). The index is removed as a
--- 96,100 ----
       */
      public IIndex registerIndex(String name, IIndex btree);
! 
      /**
       * Drops the named index (unisolated). The index is removed as a

[cweb-CVS] bigdata/src/architecture bigdata-published-model.zip, NONE, 1.1 segment math.xls, 1.13, 1.14

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/architecture
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv15000/src/architecture

Modified Files:
	segment math.xls 
Added Files:
	bigdata-published-model.zip 
Log Message:
Updated the UML model and added a ZIP containing an HTML presentation of the model.
Working on partitioned index support.

--- NEW FILE: bigdata-published-model.zip ---
(This appears to be a binary file; contents omitted.)

Index: segment math.xls
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/architecture/segment math.xls,v
retrieving revision 1.13
retrieving revision 1.14
diff -C2 -d -r1.13 -r1.14
Binary files /tmp/cvs9zapVj and /tmp/cvsWOx1h4 differ

[cweb-CVS] bigdata/src/test/com/bigdata/scaleup TestPartitionedIndex.java, 1.1, 1.2 TestMetadataIndex.java, 1.8, 1.9 TestPartitionedJournal.java, 1.4, 1.5

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/test/com/bigdata/scaleup
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv15000/src/test/com/bigdata/scaleup

Modified Files:
	TestPartitionedIndex.java TestMetadataIndex.java 
	TestPartitionedJournal.java 
Log Message:
Updated the UML model and added a ZIP containing an HTML presentation of the model.
Working on partitioned index support.

Index: TestMetadataIndex.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/scaleup/TestMetadataIndex.java,v
retrieving revision 1.8
retrieving revision 1.9
diff -C2 -d -r1.8 -r1.9
*** TestMetadataIndex.java	6 Mar 2007 20:38:06 -0000	1.8
--- TestMetadataIndex.java	8 Mar 2007 18:14:06 -0000	1.9
***************
*** 162,166 ****
          
          PartitionMetadata part2 = new PartitionMetadata(partId0,1,
!                 new SegmentMetadata[] { new SegmentMetadata("a", 10L,IndexSegmentLifeCycleEnum.LIVE) });
  
          assertEquals(part1,md.put(key0, part2));
--- 162,166 ----
          
          PartitionMetadata part2 = new PartitionMetadata(partId0,1,
!                 new SegmentMetadata[] { new SegmentMetadata("a", 10L,ResourceState.Live) });
  
          assertEquals(part1,md.put(key0, part2));
***************
*** 169,174 ****
          
          PartitionMetadata part3 = new PartitionMetadata(partId0,2,
!                 new SegmentMetadata[] { new SegmentMetadata("a", 10L,IndexSegmentLifeCycleEnum.LIVE),
!                         new SegmentMetadata("b", 20L,IndexSegmentLifeCycleEnum.LIVE) });
  
          assertEquals(part2,md.put(key0, part3));
--- 169,174 ----
          
          PartitionMetadata part3 = new PartitionMetadata(partId0,2,
!                 new SegmentMetadata[] { new SegmentMetadata("a", 10L,ResourceState.Live),
!                         new SegmentMetadata("b", 20L,ResourceState.Live) });
  
          assertEquals(part2,md.put(key0, part3));
***************
*** 242,246 ****
          final int partId1 = 1;
          PartitionMetadata part1 = new PartitionMetadata(partId1,1,
!                 new SegmentMetadata[] { new SegmentMetadata("a", 10L,IndexSegmentLifeCycleEnum.LIVE) });
          assertEquals(null,md.put(key1, part1));
          assertEquals(part1,md.get(key1));
--- 242,246 ----
          final int partId1 = 1;
          PartitionMetadata part1 = new PartitionMetadata(partId1,1,
!                 new SegmentMetadata[] { new SegmentMetadata("a", 10L,ResourceState.Live) });
          assertEquals(null,md.put(key1, part1));
          assertEquals(part1,md.get(key1));
***************
*** 248,253 ****
          final int partId2 = 2;
          PartitionMetadata part2 = new PartitionMetadata(partId2,2,
!                 new SegmentMetadata[] { new SegmentMetadata("a", 10L,IndexSegmentLifeCycleEnum.LIVE),
!                         new SegmentMetadata("b", 20L,IndexSegmentLifeCycleEnum.LIVE) });
          assertEquals(null, md.put(key2, part2));
          assertEquals(part2, md.get(key2));
--- 248,253 ----
          final int partId2 = 2;
          PartitionMetadata part2 = new PartitionMetadata(partId2,2,
!                 new SegmentMetadata[] { new SegmentMetadata("a", 10L,ResourceState.Live),
!                         new SegmentMetadata("b", 20L,ResourceState.Live) });
          assertEquals(null, md.put(key2, part2));
          assertEquals(part2, md.get(key2));
***************
*** 330,334 ****
          final int partId1 = 1;
          PartitionMetadata part1 = new PartitionMetadata(partId1,1,
!                 new SegmentMetadata[] { new SegmentMetadata("a", 10L,IndexSegmentLifeCycleEnum.LIVE) });
          assertEquals(null,md.put(key1, part1));
          assertEquals(part1,md.get(key1));
--- 330,334 ----
          final int partId1 = 1;
          PartitionMetadata part1 = new PartitionMetadata(partId1,1,
!                 new SegmentMetadata[] { new SegmentMetadata("a", 10L,ResourceState.Live) });
          assertEquals(null,md.put(key1, part1));
          assertEquals(part1,md.get(key1));
***************
*** 336,341 ****
          final int partId2 = 2;
          PartitionMetadata part2 = new PartitionMetadata(partId2,2,
!                 new SegmentMetadata[] { new SegmentMetadata("a", 10L,IndexSegmentLifeCycleEnum.LIVE),
!                         new SegmentMetadata("b", 20L,IndexSegmentLifeCycleEnum.LIVE) });
          assertEquals(null, md.put(key2, part2));
          assertEquals(part2,md.get(key2));
--- 336,341 ----
          final int partId2 = 2;
          PartitionMetadata part2 = new PartitionMetadata(partId2,2,
!                 new SegmentMetadata[] { new SegmentMetadata("a", 10L,ResourceState.Live),
!                         new SegmentMetadata("b", 20L,ResourceState.Live) });
          assertEquals(null, md.put(key2, part2));
          assertEquals(part2,md.get(key2));
***************
*** 455,465 ****
          md.put(new byte[] {}, new PartitionMetadata(0,1,
                  new SegmentMetadata[] { new SegmentMetadata("" + outFile,
!                         outFile.length(),IndexSegmentLifeCycleEnum.LIVE) }));
  
          /*
           * open and verify the index segment against the btree data.
           */
!         IndexSegment seg = new IndexSegment(new IndexSegmentFileStore(outFile),
!                 btree.getNodeSerializer().getValueSerializer());
  
          assertSameBTree(btree,seg);
--- 455,465 ----
          md.put(new byte[] {}, new PartitionMetadata(0,1,
                  new SegmentMetadata[] { new SegmentMetadata("" + outFile,
!                         outFile.length(),ResourceState.Live) }));
  
          /*
           * open and verify the index segment against the btree data.
           */
!         IndexSegment seg = new IndexSegmentFileStore(outFile).load();
! //                @todo btree.getNodeSerializer().getValueSerializer());
  
          assertSameBTree(btree,seg);
***************
*** 560,570 ****
          md.put(new byte[] {}, new PartitionMetadata(0,2,
                  new SegmentMetadata[] { new SegmentMetadata("" + outFile01,
!                         outFile01.length(),IndexSegmentLifeCycleEnum.LIVE) }));
  
          /*
           * open and verify the index segment against the btree data.
           */
!         IndexSegment seg01 = new IndexSegment(new IndexSegmentFileStore(outFile01),
!                 btree.getNodeSerializer().getValueSerializer());
  
          assertSameBTree(btree,seg01);
--- 560,570 ----
          md.put(new byte[] {}, new PartitionMetadata(0,2,
                  new SegmentMetadata[] { new SegmentMetadata("" + outFile01,
!                         outFile01.length(),ResourceState.Live) }));
  
          /*
           * open and verify the index segment against the btree data.
           */
!         IndexSegment seg01 = new IndexSegmentFileStore(outFile01).load();
!         // @todo btree.getNodeSerializer().getValueSerializer());
  
          assertSameBTree(btree,seg01);
***************
*** 609,618 ****
           * update the metadata index for this partition.
           * 
!          * Note: We mark index segment 01 as "DEAD" for this partition since it
           * has been replaced by the merged result (index segment 02).
           */
          md.put(new byte[] {}, new PartitionMetadata(0, 3, new SegmentMetadata[] {
!                 new SegmentMetadata("" + outFile01, outFile01.length(),IndexSegmentLifeCycleEnum.DEAD),
!                 new SegmentMetadata("" + outFile02, outFile02.length(),IndexSegmentLifeCycleEnum.LIVE) }));
  
          /*
--- 609,618 ----
           * update the metadata index for this partition.
           * 
!          * Note: We mark index segment 01 as "Dead" for this partition since it
           * has been replaced by the merged result (index segment 02).
           */
          md.put(new byte[] {}, new PartitionMetadata(0, 3, new SegmentMetadata[] {
!                 new SegmentMetadata("" + outFile01, outFile01.length(),ResourceState.Dead),
!                 new SegmentMetadata("" + outFile02, outFile02.length(),ResourceState.Live) }));
  
          /*
***************
*** 620,625 ****
           * data.
           */
!         IndexSegment seg02 = new IndexSegment(new IndexSegmentFileStore(
!                 outFile02), btree.getNodeSerializer().getValueSerializer());
  
          assertSameIterator(new Object[] { v1, v2, v3, v4, v5, v6, v7, v8 },
--- 620,625 ----
           * data.
           */
!         IndexSegment seg02 = new IndexSegmentFileStore(outFile02).load();
!         // @todo btree.getNodeSerializer().getValueSerializer());
  
          assertSameIterator(new Object[] { v1, v2, v3, v4, v5, v6, v7, v8 },
***************
*** 829,839 ****
                                  new SegmentMetadata[] { new SegmentMetadata(""
                                          + outFile01, outFile01.length(),
!                                         IndexSegmentLifeCycleEnum.LIVE) }));
  
                  /*
                   * open and verify the index segment against the btree data.
                   */
!                 seg = new IndexSegment(new IndexSegmentFileStore(outFile01),
!                         testData.getNodeSerializer().getValueSerializer());
  
                  if(validateEachTrial) assertSameBTree(testData, seg);
--- 829,839 ----
                                  new SegmentMetadata[] { new SegmentMetadata(""
                                          + outFile01, outFile01.length(),
!                                         ResourceState.Live) }));
  
                  /*
                   * open and verify the index segment against the btree data.
                   */
!                 seg = new IndexSegmentFileStore(outFile01).load();
! //                        @todo testData.getNodeSerializer().getValueSerializer());
  
                  if(validateEachTrial) assertSameBTree(testData, seg);
***************
*** 885,891 ****
                   * update the metadata index for this partition.
                   * 
!                  * @todo We could mark the earlier index segment as "DEAD" for
                   * this partition since it has been replaced by the merged
!                  * result.  We could then delete the files for "DEAD" index
                   * segments after a suitable grace period. 
                   */
--- 885,891 ----
                   * update the metadata index for this partition.
                   * 
!                  * @todo We could mark the earlier index segment as "Dead" for
                   * this partition since it has been replaced by the merged
!                  * result.  We could then delete the files for "Dead" index
                   * segments after a suitable grace period. 
                   */
***************
*** 894,898 ****
                          new SegmentMetadata[] {
                                  new SegmentMetadata("" + outFile02, outFile02
!                                         .length(), IndexSegmentLifeCycleEnum.LIVE) }));
  
                  /*
--- 894,898 ----
                          new SegmentMetadata[] {
                                  new SegmentMetadata("" + outFile02, outFile02
!                                         .length(), ResourceState.Live) }));
  
                  /*
***************
*** 900,906 ****
                   * expected data.
                   */
!                 IndexSegment seg02 = new IndexSegment(
!                         new IndexSegmentFileStore(outFile02),
!                         testData.getNodeSerializer().getValueSerializer());
  
                  if(validateEachTrial) assertSameBTree(groundTruth,seg02);
--- 900,905 ----
                   * expected data.
                   */
!                 IndexSegment seg02 = new IndexSegmentFileStore(outFile02).load();
! //                 @todo testData.getNodeSerializer().getValueSerializer());
  
                  if(validateEachTrial) assertSameBTree(groundTruth,seg02);
***************
*** 1029,1039 ****
          md.put(new byte[] {}, new PartitionMetadata(0,1,
                  new SegmentMetadata[] { new SegmentMetadata("" + outFile01,
!                         outFile01.length(),IndexSegmentLifeCycleEnum.LIVE) }));
  
          /*
           * open and verify the index segment against the btree data.
           */
!         IndexSegment seg01 = new IndexSegment(new IndexSegmentFileStore(outFile01),
!                 btree.getNodeSerializer().getValueSerializer());
  
          assertSameBTree(btree,seg01);
--- 1028,1038 ----
          md.put(new byte[] {}, new PartitionMetadata(0,1,
                  new SegmentMetadata[] { new SegmentMetadata("" + outFile01,
!                         outFile01.length(),ResourceState.Live) }));
  
          /*
           * open and verify the index segment against the btree data.
           */
!         IndexSegment seg01 = new IndexSegmentFileStore(outFile01).load();
!         // @todo btree.getNodeSerializer().getValueSerializer());
  
          assertSameBTree(btree,seg01);
***************
*** 1072,1083 ****
           */
          md.put(new byte[] {}, new PartitionMetadata(0, 1, new SegmentMetadata[] {
!                 new SegmentMetadata("" + outFile01, outFile01.length(),IndexSegmentLifeCycleEnum.LIVE),
!                 new SegmentMetadata("" + outFile02, outFile02.length(),IndexSegmentLifeCycleEnum.LIVE) }));
  
          /*
           * open and verify the index segment against the btree data.
           */
!         IndexSegment seg02 = new IndexSegment(new IndexSegmentFileStore(
!                 outFile02), btree.getNodeSerializer().getValueSerializer());
  
          assertSameBTree(btree, seg02);
--- 1071,1082 ----
           */
          md.put(new byte[] {}, new PartitionMetadata(0, 1, new SegmentMetadata[] {
!                 new SegmentMetadata("" + outFile01, outFile01.length(),ResourceState.Live),
!                 new SegmentMetadata("" + outFile02, outFile02.length(),ResourceState.Live) }));
  
          /*
           * open and verify the index segment against the btree data.
           */
!         IndexSegment seg02 = new IndexSegmentFileStore(outFile02).load();
!         // @todo btree.getNodeSerializer().getValueSerializer());
  
          assertSameBTree(btree, seg02);

Index: TestPartitionedIndex.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/scaleup/TestPartitionedIndex.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** TestPartitionedIndex.java	8 Feb 2007 21:32:15 -0000	1.1
--- TestPartitionedIndex.java	8 Mar 2007 18:14:06 -0000	1.2
***************
*** 48,56 ****
--- 48,66 ----
  package com.bigdata.scaleup;
  
+ import com.bigdata.isolation.UnisolatedBTree;
  import com.bigdata.objndx.AbstractBTreeTestCase;
+ import com.bigdata.objndx.IndexSegmentBuilder;
+ import com.bigdata.objndx.IndexSegmentMerger;
  
  /**
   * Test suite for {@link PartitionedIndex}.
   * 
+  * @todo This should be refactored to test when using {@link BTree} as well as
+  *       {@link UnisolatedBTree}. The latter is more important for database
+  *       operations as it supports isolation and column stores.  Changing over
+  *       from {@link BTree} to {@link UnisolatedBTree} highlights many assumptions
+  *       in {@link IndexSegmentBuilder}, {@link IndexSegmentMerger}, and the
+  *       code that handles {@link PartitionedJournal#overflow()}.
+  * 
   * @todo where possible, it would be nice to leverage the unit tests for the
   *       mutable btree. note that rangeCount does not have the same behavior for

Index: TestPartitionedJournal.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/scaleup/TestPartitionedJournal.java,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** TestPartitionedJournal.java	22 Feb 2007 16:59:35 -0000	1.4
--- TestPartitionedJournal.java	8 Mar 2007 18:14:06 -0000	1.5
***************
*** 54,64 ****
  import junit.framework.TestCase2;
  
  import com.bigdata.journal.Journal;
  import com.bigdata.objndx.AbstractBTreeTestCase;
- import com.bigdata.objndx.BTree;
  import com.bigdata.objndx.BatchInsert;
  import com.bigdata.objndx.IIndex;
  import com.bigdata.objndx.KeyBuilder;
- import com.bigdata.objndx.SimpleEntry;
  import com.bigdata.rawstore.SimpleMemoryRawStore;
  import com.bigdata.scaleup.PartitionedJournal.MergePolicy;
--- 54,63 ----
  import junit.framework.TestCase2;
  
+ import com.bigdata.isolation.UnisolatedBTree;
  import com.bigdata.journal.Journal;
  import com.bigdata.objndx.AbstractBTreeTestCase;
  import com.bigdata.objndx.BatchInsert;
  import com.bigdata.objndx.IIndex;
  import com.bigdata.objndx.KeyBuilder;
  import com.bigdata.rawstore.SimpleMemoryRawStore;
  import com.bigdata.scaleup.PartitionedJournal.MergePolicy;
***************
*** 144,148 ****
          final String name = "abc";
          
!         IIndex index = new BTree(journal, 3, SimpleEntry.Serializer.INSTANCE);
          
          assertNull(journal.getIndex(name));
--- 143,147 ----
          final String name = "abc";
          
!         IIndex index = new UnisolatedBTree(journal);
          
          assertNull(journal.getIndex(name));
***************
*** 160,164 ****
          
          final byte[] k0 = new byte[]{0};
!         final Object v0 = new SimpleEntry(0);
          
          index.insert( k0, v0);
--- 159,163 ----
          
          final byte[] k0 = new byte[]{0};
!         final byte[] v0 = new byte[]{0};
          
          index.insert( k0, v0);
***************
*** 182,186 ****
              assertNotNull("btree", index);
              assertEquals("entryCount", 1, ((PartitionedIndex)index).getBTree().getEntryCount());
!             assertEquals(v0, index.lookup(k0));
  
              journal.dropIndex(name);
--- 181,185 ----
              assertNotNull("btree", index);
              assertEquals("entryCount", 1, ((PartitionedIndex)index).getBTree().getEntryCount());
!             assertEquals(v0, (byte[])index.lookup(k0));
  
              journal.dropIndex(name);
***************
*** 210,215 ****
          final String name = "abc";
          
!         assertNotNull(journal.registerIndex(name, new BTree(journal, 3,
!                 SimpleEntry.Serializer.INSTANCE)));
          
          journal.overflow();
--- 209,213 ----
          final String name = "abc";
          
!         assertNotNull(journal.registerIndex(name, new UnisolatedBTree(journal)));
          
          journal.overflow();
***************
*** 245,250 ****
          final String name = "abc";
          
!         assertNotNull(journal.registerIndex(name, new BTree(journal, 3,
!                 SimpleEntry.Serializer.INSTANCE)));
  
          final TestData data = new TestData(journal.migrationThreshold-1);
--- 243,247 ----
          final String name = "abc";
          
!         assertNotNull(journal.registerIndex(name, new UnisolatedBTree(journal, 3)));
  
          final TestData data = new TestData(journal.migrationThreshold-1);
***************
*** 296,301 ****
          final String name = "abc";
          
!         assertNotNull(journal.registerIndex(name, new BTree(journal, 3,
!                 SimpleEntry.Serializer.INSTANCE)));
  
          final TestData data = new TestData(journal.migrationThreshold);
--- 293,297 ----
          final String name = "abc";
          
!         assertNotNull(journal.registerIndex(name, new UnisolatedBTree(journal)));
  
          final TestData data = new TestData(journal.migrationThreshold);
***************
*** 362,370 ****
          final String name = "abc";
          
!         assertNotNull(journal.registerIndex(name, new BTree(journal, 3,
!                 SimpleEntry.Serializer.INSTANCE)));
  
!         final BTree groundTruth = new BTree(new SimpleMemoryRawStore(),
!                 BTree.DEFAULT_BRANCHING_FACTOR, SimpleEntry.Serializer.INSTANCE);
  
          final int ntrials = 10;
--- 358,365 ----
          final String name = "abc";
          
!         assertNotNull(journal.registerIndex(name, new UnisolatedBTree(journal)));
  
!         final UnisolatedBTree groundTruth = new UnisolatedBTree(
!                 new SimpleMemoryRawStore());
  
          final int ntrials = 10;
***************
*** 404,411 ****
                  assertEquals("#segments",2,pmd.segs.length);
                  
!                 assertEquals("state", IndexSegmentLifeCycleEnum.DEAD,
                          pmd.segs[0].state);
                  
!                 assertEquals("state", IndexSegmentLifeCycleEnum.LIVE,
                          pmd.segs[1].state);
                  
--- 399,406 ----
                  assertEquals("#segments",2,pmd.segs.length);
                  
!                 assertEquals("state", ResourceState.Dead,
                          pmd.segs[0].state);
                  
!                 assertEquals("state", ResourceState.Live,
                          pmd.segs[1].state);
                  
***************
*** 461,465 ****
          
          public final byte[][] keys;
!         public final SimpleEntry[] vals;
          
          /**
--- 456,460 ----
          
          public final byte[][] keys;
!         public final byte[][] vals;
          
          /**
***************
*** 477,481 ****
              keys = new byte[nrecords][];
              
!             vals = new SimpleEntry[nrecords];
              
              int key = 0;
--- 472,476 ----
              keys = new byte[nrecords][];
              
!             vals = new byte[nrecords][];
              
              int key = 0;
***************
*** 490,495 ****
                  
                  keys[i] = keyBuilder.reset().append(key).getKey();
!                 
!                 vals[i] = new SimpleEntry(key);
                  
              }
--- 485,490 ----
                  
                  keys[i] = keyBuilder.reset().append(key).getKey();
! 
!                 vals[i] = keys[i].clone();
                  
              }
***************
*** 508,512 ****
              for(int i=0; i<nrecords; i++) {
  
!                 SimpleEntry actual = (SimpleEntry)ndx.lookup(keys[i]);
                  
                  assertEquals(vals[i],actual);
--- 503,507 ----
              for(int i=0; i<nrecords; i++) {
  
!                 byte[] actual = (byte[])ndx.lookup(keys[i]);
                  
                  assertEquals(vals[i],actual);

[cweb-CVS] bigdata-rdf/src/test/com/bigdata/rdf/rio TestRioIntegration.java, 1.15, 1.16

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata-rdf/src/test/com/bigdata/rdf/rio
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv24380/src/test/com/bigdata/rdf/rio

Modified Files:
	TestRioIntegration.java 
Log Message:
Refactoring to introduce asynchronous handling of overflow events in support of a scale-up/scale-out design.

Index: TestRioIntegration.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/test/com/bigdata/rdf/rio/TestRioIntegration.java,v
retrieving revision 1.15
retrieving revision 1.16
diff -C2 -d -r1.15 -r1.16
*** TestRioIntegration.java	22 Feb 2007 16:58:59 -0000	1.15
--- TestRioIntegration.java	6 Mar 2007 20:38:13 -0000	1.16
***************
*** 120,125 ****
          TestRioIntegration test = new TestRioIntegration("TestInsertRateStore");
          test.setUp();
! //        test.doTest( new PresortRioLoader(test.store), args );
!         test.doTest( new BulkRioLoader(test.store), args );
          test.tearDown();
              
--- 120,125 ----
          TestRioIntegration test = new TestRioIntegration("TestInsertRateStore");
          test.setUp();
!         test.doTest( new PresortRioLoader(test.store), args );
! //        test.doTest( new BulkRioLoader(test.store), args );
          test.tearDown();
              
***************
*** 280,283 ****
--- 280,284 ----
              "data/nciOncology.owl" // nterms := 289844
  //            "data/wordnet_nouns-20010201.rdf"
+ //            "data/taxonomy.rdf"
              };
      

[cweb-CVS] bigdata/src/java/com/bigdata/objndx IFusedView.java, NONE, 1.1 IndexSegmentMerger.java, 1.9, 1.10 NodeSerializer.java, 1.32, 1.33 IndexSegmentBuilder.java, 1.25, 1.26 BTree.java, 1.35, 1.36 IndexSegmentFileStore.java, 1.8, 1.9 Node.java, 1.24, 1.25 MutableKeyBuffer.java, 1.1, 1.2 FusedView.java, 1.5, 1.6 RangeIterator.java, 1.5, NONE IRangeIterator.java, 1.4, NONE

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv23960/src/java/com/bigdata/objndx

Modified Files:
	IndexSegmentMerger.java NodeSerializer.java 
	IndexSegmentBuilder.java BTree.java IndexSegmentFileStore.java 
	Node.java MutableKeyBuffer.java FusedView.java 
Added Files:
	IFusedView.java 
Removed Files:
	RangeIterator.java IRangeIterator.java 
Log Message:
Refactoring to introduce asynchronous handling of overflow events in support of a scale-up/scale-out design.

Index: IndexSegmentFileStore.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/IndexSegmentFileStore.java,v
retrieving revision 1.8
retrieving revision 1.9
diff -C2 -d -r1.8 -r1.9
*** IndexSegmentFileStore.java	22 Feb 2007 16:59:35 -0000	1.8
--- IndexSegmentFileStore.java	6 Mar 2007 20:38:05 -0000	1.9
***************
*** 171,174 ****
--- 171,180 ----
          
      }
+     
+     public long size() {
+         
+         return metadata.length;
+         
+     }
  
      /**

Index: Node.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/Node.java,v
retrieving revision 1.24
retrieving revision 1.25
diff -C2 -d -r1.24 -r1.25
*** Node.java	13 Feb 2007 23:01:02 -0000	1.24
--- Node.java	6 Mar 2007 20:38:05 -0000	1.25
***************
*** 1895,1898 ****
--- 1895,1907 ----
           * index of the children before modifying their keys.
           * 
+          * @todo 85% of the use of this method is updateEntryCount(). Since that
+          * method is only called on update, we would do well to buffer hard
+          * references during descent and test the buffer in this method before
+          * performing a full search. Since concurrent writers are not allowed,
+          * we only need a single buffer whose height is the height of the tree.
+          * This should prove especially beneficial for larger branching factors.
+          * For smaller branching factors the cost might be so small as to be
+          * ignorable.
+          * 
           * @see Leaf#merge(Leaf sibling,boolean isRightSibling)
           */

Index: MutableKeyBuffer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/MutableKeyBuffer.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** MutableKeyBuffer.java	26 Jan 2007 02:39:23 -0000	1.1
--- MutableKeyBuffer.java	6 Mar 2007 20:38:05 -0000	1.2
***************
*** 1,8 ****
  package com.bigdata.objndx;
  
- 
  /**
   * A mutable implementation of {@link IKeyBuffer}.
   * 
   * @todo track prefix length for mutable keys (update when first/last key are
   *       updated).  at present the node/leaf logic directly manipulates the
--- 1,8 ----
  package com.bigdata.objndx;
  
  /**
   * A mutable implementation of {@link IKeyBuffer}.
   * 
+  * @todo 27% of the search cost is dealing with the prefix.
   * @todo track prefix length for mutable keys (update when first/last key are
   *       updated).  at present the node/leaf logic directly manipulates the

Index: FusedView.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/FusedView.java,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** FusedView.java	17 Feb 2007 21:34:21 -0000	1.5
--- FusedView.java	6 Mar 2007 20:38:05 -0000	1.6
***************
*** 63,67 ****
   *       {@link FusedView} instances if not in a more efficient manner.
   */
! public class FusedView implements IIndex {
  
      /**
--- 63,67 ----
   *       {@link FusedView} instances if not in a more efficient manner.
   */
! public class FusedView implements IIndex, IFusedView {
  
      /**

Index: IndexSegmentBuilder.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/IndexSegmentBuilder.java,v
retrieving revision 1.25
retrieving revision 1.26
diff -C2 -d -r1.25 -r1.26
*** IndexSegmentBuilder.java	21 Feb 2007 20:17:21 -0000	1.25
--- IndexSegmentBuilder.java	6 Mar 2007 20:38:05 -0000	1.26
***************
*** 93,99 ****
   * have to buffer them in memory or have to write them onto a temporary file and
   * then copy them into place after the last leaf has been processed. The code
!  * currently uses a temporary file for this purpose. This space demand was not
!  * present in West's algorithm because it did not attempt to place the leaves
!  * contiguously onto the store.
   * 
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
--- 93,99 ----
   * have to buffer them in memory or have to write them onto a temporary file and
   * then copy them into place after the last leaf has been processed. The code
!  * abstracts this decision using a {@link TemporaryRawStore} for this purpose.
!  * This space demand was not present in West's algorithm because it did not
!  * attempt to place the leaves contiguously onto the store.
   * 
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
***************
*** 110,114 ****
   * 
   * FIXME use the shortest separator key.
!  *
   * @see IndexSegment
   * @see IndexSegmentFile
--- 110,114 ----
   * 
   * FIXME use the shortest separator key.
!  * 
   * @see IndexSegment
   * @see IndexSegmentFile
***************
*** 288,291 ****
--- 288,296 ----
      
      /**
+      * The data throughput rate in megabytes per second.
+      */
+     final float mbPerSec;
+     
+     /**
       * <p>
       * Builds an index segment on the disk from a {@link BTree}. The index
***************
*** 714,717 ****
--- 719,726 ----
              elapsed = System.currentTimeMillis() - begin;
  
+             // data rate in MB/sec.
+             mbPerSec = (elapsed == 0 ? 0 : md.length / Bytes.megabyte32
+                     / (elapsed / 1000f));
+             
              NumberFormat cf = NumberFormat.getNumberInstance();
              
***************
*** 726,733 ****
              System.err.println("index segment build: total=" + elapsed
                      + "ms := setup(" + elapsed_setup + "ms) + build("
!                     + elapsed_build + "ms) +  write(" + elapsed_write
!                     + "ms); " + cf.format(plan.nentries) + " entries, "
                      + fpf.format(((double) md.length / Bytes.megabyte32))
!                     + "MB");
  
              log.info("finished: total=" + elapsed + "ms := setup("
--- 735,742 ----
              System.err.println("index segment build: total=" + elapsed
                      + "ms := setup(" + elapsed_setup + "ms) + build("
!                     + elapsed_build + "ms) +  write(" + elapsed_write + "ms); "
!                     + cf.format(plan.nentries) + " entries, "
                      + fpf.format(((double) md.length / Bytes.megabyte32))
!                     + "MB" + ", rate=" + fpf.format(mbPerSec) + "MB/sec");
  
              log.info("finished: total=" + elapsed + "ms := setup("
***************
*** 735,739 ****
                      + "ms) +  write(" + elapsed_write + "ms); nentries="
                      + plan.nentries + ", branchingFactor=" + m + ", nnodes="
!                     + nnodesWritten + ", nleaves=" + nleavesWritten);
  
          } catch (Throwable ex) {
--- 744,751 ----
                      + "ms) +  write(" + elapsed_write + "ms); nentries="
                      + plan.nentries + ", branchingFactor=" + m + ", nnodes="
!                     + nnodesWritten + ", nleaves=" + nleavesWritten+
!                     fpf.format(((double) md.length / Bytes.megabyte32))
!                     + "MB"+", rate="+fpf.format(mbPerSec)+"MB/sec");
! 
  
          } catch (Throwable ex) {

--- IRangeIterator.java DELETED ---

Index: BTree.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/BTree.java,v
retrieving revision 1.35
retrieving revision 1.36
diff -C2 -d -r1.35 -r1.36
*** BTree.java	21 Feb 2007 20:17:21 -0000	1.35
--- BTree.java	6 Mar 2007 20:38:05 -0000	1.36
***************
*** 411,416 ****
                  NodeFactory.INSTANCE, //
                  recordCompressor, //
!                 // FIXME only use checksum for stores that are not fully buffered.
!                 true || !store.isFullyBuffered()/* useChecksum */
                  );
  
--- 411,423 ----
                  NodeFactory.INSTANCE, //
                  recordCompressor, //
!                 /*
!                  * Note: there is less need to use checksum for stores that are
!                  * not fully buffered since the data are always read from memory
!                  * which we presume is already parity checked. While a checksum
!                  * on a fully buffered store could detect an overwrite, the
!                  * journal architecture makes that extremely unlikely and one
!                  * has never been observed.
!                  */
!                 !store.isFullyBuffered()/* useChecksum */
                  );
  

--- NEW FILE: IFusedView.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Mar 6, 2007
 */

package com.bigdata.objndx;

/**
 * A marker interface indicating fused view providing read-only operations on
 * multiple B+-Trees mapping variable length unsigned byte[] keys to arbitrary
 * values.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public interface IFusedView {

}

--- RangeIterator.java DELETED ---

Index: NodeSerializer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/NodeSerializer.java,v
retrieving revision 1.32
retrieving revision 1.33
diff -C2 -d -r1.32 -r1.33
*** NodeSerializer.java	21 Feb 2007 20:17:21 -0000	1.32
--- NodeSerializer.java	6 Mar 2007 20:38:05 -0000	1.33
***************
*** 191,195 ****
       * {@link #branchingFactor}.
       */
!     public static final transient int DEFAULT_BUFFER_CAPACITY_PER_ENTRY = Bytes.kilobyte32 * 1;
      
      /**
--- 191,195 ----
       * {@link #branchingFactor}.
       */
!     public static final transient int DEFAULT_BUFFER_CAPACITY_PER_ENTRY = Bytes.kilobyte32 / 4;
      
      /**

Index: IndexSegmentMerger.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/IndexSegmentMerger.java,v
retrieving revision 1.9
retrieving revision 1.10
diff -C2 -d -r1.9 -r1.10
*** IndexSegmentMerger.java	21 Feb 2007 20:17:21 -0000	1.9
--- IndexSegmentMerger.java	6 Mar 2007 20:38:05 -0000	1.10
***************
*** 48,68 ****
  package com.bigdata.objndx;
  
- import java.io.File;
  import java.io.IOException;
- import java.io.RandomAccessFile;
  import java.nio.ByteBuffer;
- import java.nio.channels.FileChannel;
  import java.text.NumberFormat;
  import java.util.Iterator;
  import java.util.NoSuchElementException;
  
  import org.apache.log4j.Level;
  import org.apache.log4j.Logger;
  
  import com.bigdata.objndx.IndexSegmentBuilder.NOPNodeFactory;
  import com.bigdata.objndx.IndexSegmentBuilder.SimpleLeafData;
- import com.bigdata.rawstore.Addr;
  import com.bigdata.rawstore.Bytes;
- import com.bigdata.rawstore.IRawStore;
  
  import cutthecrap.utils.striterators.Expander;
--- 48,65 ----
  package com.bigdata.objndx;
  
  import java.io.IOException;
  import java.nio.ByteBuffer;
  import java.text.NumberFormat;
  import java.util.Iterator;
  import java.util.NoSuchElementException;
+ import java.util.Vector;
  
  import org.apache.log4j.Level;
  import org.apache.log4j.Logger;
  
+ import com.bigdata.journal.TemporaryRawStore;
  import com.bigdata.objndx.IndexSegmentBuilder.NOPNodeFactory;
  import com.bigdata.objndx.IndexSegmentBuilder.SimpleLeafData;
  import com.bigdata.rawstore.Bytes;
  
  import cutthecrap.utils.striterators.Expander;
***************
*** 71,87 ****
  /**
   * Class supporting a compacting merge of two btrees into a series of ordered
!  * leaves on a temporary file in support of a compacting merge of mutable
!  * {@link BTree}s and/or immutable {@link IndexSegment}s.
   * 
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
   * @version $Id$
   * 
!  * @todo write tests: merging a tree with itself, merging trees w/o deletion
!  *       markers, merging trees w/ deletion markers, merging trees w/ age-based
!  *       version expiration, merging trees with count-based version expiration.
!  * 
!  * @todo Support delete during merge to support transactions (a TimestampValue
!  *       having a greater timestamp and a null value is interpreted as a delete
!  *       marker).
   * 
   * @todo Support deletion based on history policy (requires timestamps in the
--- 68,80 ----
  /**
   * Class supporting a compacting merge of two btrees into a series of ordered
!  * leaves written on a {@link TemporaryRawStore} in support of a compacting
!  * merge of mutable {@link BTree}s and/or immutable {@link IndexSegment}s.
   * 
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
   * @version $Id$
   * 
!  * @todo Support merge rule that knows how to process timestamped entries and
!  *       deletion markers and that removes deletion markers iff a full
!  *       compacting merge is requested.
   * 
   * @todo Support deletion based on history policy (requires timestamps in the
***************
*** 92,103 ****
   *       well.
   * 
-  * FIXME rewrite to use {@link IRawStore} objects as buffers ala the
-  * {@link IndexSegmentBuilder} and see if we are better off using memory or disk
-  * to buffer the merge.
-  * 
   * @see {@link FusedView}, which provides a dynamic view of two or more btrees.
   *      However, this class is more efficient when we are going to do a bulk
   *      merge operation since it performs the merge and computes the #of output
   *      entries in one pass.
   */
  public class IndexSegmentMerger {
--- 85,94 ----
   *       well.
   * 
   * @see {@link FusedView}, which provides a dynamic view of two or more btrees.
   *      However, this class is more efficient when we are going to do a bulk
   *      merge operation since it performs the merge and computes the #of output
   *      entries in one pass.
+  * 
+  * @todo parameterize recordCompressor.
   */
  public class IndexSegmentMerger {
***************
*** 122,125 ****
--- 113,121 ----
  
      /**
+      * @todo parameterize useChecksum.
+      */
+     final boolean useChecksum = false;
+     
+     /**
       * Compacting merge of two btrees, writing the results onto a file. The file
       * data format is simply a sequence of leaves using the specified branching
***************
*** 129,152 ****
       * leaves.
       * 
-      * @param raf
-      *            The file on which the results are written as a series of
-      *            leaves. Typically this file will be created in a temporary
-      *            directory and the file will be removed when the results of the
-      *            compacting merge are no longer required.
       * @param m
!      *            The branching factor used by the leaves written on that file.
       * @param in1
       *            A btree.
       * @param in2
       *            Another btree.
-      * 
-      * @todo exclusive lock on the output file.
       */
!     public IndexSegmentMerger(File outFile, int m, AbstractBTree in1,
!             AbstractBTree in2) throws IOException {
          
-         if (outFile == null)
-             throw new IllegalArgumentException();
- 
          if (m < AbstractBTree.MIN_BRANCHING_FACTOR)
              throw new IllegalArgumentException();
--- 125,138 ----
       * leaves.
       * 
       * @param m
!      *            The branching factor used by the leaves written on the
!      *            temporary store.
       * @param in1
       *            A btree.
       * @param in2
       *            Another btree.
       */
!     public IndexSegmentMerger(int m, AbstractBTree in1, AbstractBTree in2) throws IOException {
          
          if (m < AbstractBTree.MIN_BRANCHING_FACTOR)
              throw new IllegalArgumentException();
***************
*** 158,182 ****
              throw new IllegalArgumentException();
          
!         // @todo verify that we are compacting trees for the same index
!         
!         this.outFile = outFile;
!         
!         if( outFile.exists() && outFile.length() > 0) {
!             
!             throw new IOException("output file exists: "
!                     + outFile.getAbsoluteFile());
!             
!         }
!         
!         // this is a temporary file so make sure that it will disappear.
!         outFile.deleteOnExit();
! 
!         out = new RandomAccessFile(outFile,"rw");
          
          // reads leaves from the 1st btree.
!         itr1 = in1.leafIterator();
          
          // reads leaves from the 2nd btree.
!         itr2 = in2.leafIterator();
  
          // output leaf - reused for each leaf written.
--- 144,156 ----
              throw new IllegalArgumentException();
          
!         tmpStore = new TemporaryRawStore();
          
          // reads leaves from the 1st btree.
! //        itr1 = in1.leafIterator();
!         itr1 = new SourceLeafIterator(in1);
          
          // reads leaves from the 2nd btree.
! //        itr2 = in2.leafIterator();
!         itr2 = new SourceLeafIterator(in2);
  
          // output leaf - reused for each leaf written.
***************
*** 184,195 ****
          leaf.reset(m);
          
!         // @todo should we always use checksums for the temporary file?
!         final boolean useChecksum = true;
!         
!         // Used to serialize the stack and leaves for the output tree.
!         int initialBufferCapacity = 0; // will be estimated.
          nodeSer = new NodeSerializer(NOPNodeFactory.INSTANCE,
                  m,
!                 initialBufferCapacity,
                  new IndexSegment.CustomAddressSerializer(),
                  in1.nodeSer.keySerializer,
--- 158,165 ----
          leaf.reset(m);
          
!         // Used to serialize the leaves.
          nodeSer = new NodeSerializer(NOPNodeFactory.INSTANCE,
                  m,
!                 0 /*initialBufferCapacity will be estimated*/,
                  new IndexSegment.CustomAddressSerializer(),
                  in1.nodeSer.keySerializer,
***************
*** 201,227 ****
      }
  
!     final File outFile;
      
      /**
!      * Used to write the merged output file.
       */
!     final RandomAccessFile out;
      
      final NodeSerializer nodeSer;
      
!     final Iterator itr1; // reads leaves from the 1st btree.
!     Leaf leaf1 = null; // current leaf in 1st btree.
!     int index1 = 0; // current entry index in current leaf of 1st btree.
!     boolean exhausted1 = false; // true iff the 1st iterator is exhausted.
  
!     final Iterator itr2; // reads leaves from the 2nd btree.
!     Leaf leaf2 = null; // current leaf in 2nd btree.
!     int index2 = 0; // current entry index in current leaf of 2nd btree.
!     boolean exhausted2 = false; // true iff the 2nd iterator is exhausted.
  
      /**
!      * the output leaf. we reuse this for each leaf that we write. when the
!      * leaf is full we write it onto the file and then reset it so that it
!      * is ready to accept more keys.
       */
      final SimpleLeafData leaf;
--- 171,273 ----
      }
  
!     /**
!      * FIXME Support n-way merge.
!      * 
!      * @param m
!      *            The output branching factor.
!      * @param srcs
!      *            The source indices in reverse timestamp order (by increasing
!      *            age).
!      *            
!      * @throws IOException
!      */
!     public IndexSegmentMerger(int m, AbstractBTree[] srcs) throws IOException {
!     
!         throw new UnsupportedOperationException();
!     
!     }
! 
!     /**
!      * Used to buffer the output of the merge process.
!      */
!     final TemporaryRawStore tmpStore;
      
      /**
!      * The address at which each leaf in written in the {@link #tmpStore}. The
!      * entries in this list are ordered. The first entry is the first leaf
!      * written, the second entry is the second leaf written, etc.
       */
!     final Vector<Long> addrs = new Vector<Long>();
      
      final NodeSerializer nodeSer;
+ 
+     final SourceLeafIterator itr1;
+     final SourceLeafIterator itr2;
      
! //    final Iterator itr1; // reads leaves from the 1st btree.
! //    Leaf leaf1 = null; // current leaf in 1st btree.
! //    int index1 = 0; // current entry index in current leaf of 1st btree.
! //    boolean exhausted1 = false; // true iff the 1st iterator is exhausted.
! //
! //    final Iterator itr2; // reads leaves from the 2nd btree.
! //    Leaf leaf2 = null; // current leaf in 2nd btree.
! //    int index2 = 0; // current entry index in current leaf of 2nd btree.
! //    boolean exhausted2 = false; // true iff the 2nd iterator is exhausted.
  
!     /**
!      * @todo this will need to be modified to use a key range so that we can
!      *       evict an index partition worth of data at a time from a btree on
!      *       the journal. Once we do that, is there any point to operating at
!      *       the leaf iterator level vs just using an {@link EntryIterator}?
!      * 
!      * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
!      * @version $Id$
!      */
!     public static class SourceLeafIterator {
! 
!         final Iterator itr; // reads leaves from the source btree.
!         Leaf leaf = null; // current leaf in source btree.
!         int index = 0; // current entry index in current leaf of source btree.
!         boolean exhausted = false; // true iff the source btree is exhausted.
  
+         public SourceLeafIterator(AbstractBTree src) {
+             
+             itr = src.leafIterator();
+             
+         }
+ 
+         public Leaf next() {
+             
+             return (Leaf)itr.next();
+             
+         }
+         
+         /**
+          * If the current leaf is not fully consumed then return immediately.
+          * Otherwise read the next leaf from the source btree into {@link #leaf}.
+          * If there are no more leaves available then {@link #exhausted} is set
+          * to false. {@link #index} is reset to zero(0) in either case.
+          * 
+          * @return true unless this source btree is exhausted.
+          */
+         protected boolean nextLeaf() {
+             if (index < leaf.nkeys)
+                 return !exhausted;
+             index = 0;
+             if (itr.hasNext()) {
+                 leaf = (Leaf) itr.next();
+             } else {
+                 leaf = null;
+                 exhausted = true;
+             }
+             return !exhausted;
+         }
+ 
+     }
+     
      /**
!      * The output leaf. We reuse this for each leaf that we write. When the
!      * output leaf is full we write it onto the {@link #tmpStore} and then reset
!      * it so that it is ready to accept more keys.
       */
      final SimpleLeafData leaf;
***************
*** 306,314 ****
           */
          
!         leaf1 = (Leaf)itr1.next();
          
!         leaf2 = (Leaf)itr2.next();
          
!         while( !exhausted1 && ! exhausted2 ) {
              
              applyMergeRule();
--- 352,360 ----
           */
          
!         itr1.leaf = itr1.next();
          
!         itr2.leaf = itr2.next();
          
!         while( !itr1.exhausted && ! itr2.exhausted ) {
              
              applyMergeRule();
***************
*** 317,334 ****
          
          // copy anything remaining in the 1st btree.
!         while(!exhausted1) {
  
!             outputKey(leaf1,index1++);
              
!             nextLeaf1();
              
          }
  
          // copy anything remaining in the 2nd btree.
!         while(!exhausted2) {
  
!             outputKey(leaf2,index2++);
              
!             nextLeaf2();
              
          }
--- 363,380 ----
          
          // copy anything remaining in the 1st btree.
!         while(!itr1.exhausted) {
  
!             outputKey(itr1.leaf,itr1.index++);
              
!             itr1.nextLeaf();
              
          }
  
          // copy anything remaining in the 2nd btree.
!         while(!itr2.exhausted) {
  
!             outputKey(itr2.leaf,itr2.index++);
              
!             itr2.nextLeaf();
              
          }
***************
*** 343,369 ****
          }
  
! //        // synch to disk (not necessary since file is not reused).
! //        out.getChannel().force(false);
  
!         final long elapsed = System.currentTimeMillis()-begin;
  
!         final long length = out.length();
!         
!         NumberFormat cf = NumberFormat.getNumberInstance();
!         
!         cf.setGroupingUsed(true);
!         
!         NumberFormat fpf = NumberFormat.getNumberInstance();
!         
!         fpf.setGroupingUsed(false);
!         
!         fpf.setMaximumFractionDigits(2);
  
!         System.err.println("merge: " + elapsed + "ms, " + cf.format(nentries)
!                 + " entries, "
!                 + fpf.format(((double) length / Bytes.megabyte32)) + "MB");
  
!         return new MergedLeafIterator(outFile, out, leaf.m, nentries, nleaves,
!                 maxLeafBytes, nodeSer);
          
      }
--- 389,419 ----
          }
  
!         /*
!          * reporting.
!          */
!         {
  
!             final long elapsed = System.currentTimeMillis() - begin;
  
!             final long length = tmpStore.size();
  
!             NumberFormat cf = NumberFormat.getNumberInstance();
  
!             cf.setGroupingUsed(true);
! 
!             NumberFormat fpf = NumberFormat.getNumberInstance();
! 
!             fpf.setGroupingUsed(false);
! 
!             fpf.setMaximumFractionDigits(2);
! 
!             System.err.println("merge: " + elapsed + "ms, "
!                     + cf.format(nentries) + " entries, "
!                     + fpf.format(((double) length / Bytes.megabyte32)) + "MB");
!             
!         }
! 
!         return new MergedLeafIterator(tmpStore, addrs, leaf.m, nentries,
!                 nleaves, maxLeafBytes, nodeSer);
          
      }
***************
*** 389,393 ****
      protected void applyMergeRule() throws IOException {
          
!         assert !exhausted1 && !exhausted2;
  
          /*
--- 439,443 ----
      protected void applyMergeRule() throws IOException {
          
!         assert !itr1.exhausted && !itr2.exhausted;
  
          /*
***************
*** 397,403 ****
           * do less allocation and less search for this case.
           */ 
!         byte[] key1 = leaf1.keys.getKey(index1);
          
!         byte[] key2 = leaf2.keys.getKey(index2);
          
          int ret = BytesUtil.compareBytes(key1,key2);
--- 447,453 ----
           * do less allocation and less search for this case.
           */ 
!         byte[] key1 = itr1.leaf.keys.getKey(itr1.index);
          
!         byte[] key2 = itr2.leaf.keys.getKey(itr2.index);
          
          int ret = BytesUtil.compareBytes(key1,key2);
***************
*** 409,430 ****
               */
              
!             outputKey(leaf1,index1++);
              
!             index2++;
              
!             nextLeaf1();
!             nextLeaf2();
  
          } else if(ret<0) {
  
!             outputKey(leaf1,index1++);
              
!             nextLeaf1();
              
          } else {
  
!             outputKey(leaf2,index2++);
              
!             nextLeaf2();
              
          }
--- 459,480 ----
               */
              
!             outputKey(itr1.leaf,itr1.index++);
              
!             itr2.index++;
              
!             itr1.nextLeaf();
!             itr2.nextLeaf();
  
          } else if(ret<0) {
  
!             outputKey(itr1.leaf,itr1.index++);
              
!             itr1.nextLeaf();
              
          } else {
  
!             outputKey(itr2.leaf,itr2.index++);
              
!             itr2.nextLeaf();
              
          }
***************
*** 432,476 ****
      }
      
!     /**
!      * If the current leaf is not fully consumed then return immediately.
!      * Otherwise read the next leaf from the 1nd source btree into
!      * {@link #leaf1}. If there are no more leaves available then
!      * {@link #exhausted1} is set to false. {@link #index1} is reset to zero(0)
!      * in either case.
!      * 
!      * @return true unless this source btree is exhausted.
!      */
!     protected boolean nextLeaf1() {
!         if(index1<leaf1.nkeys) return !exhausted1;
!         index1 = 0;
!         if (itr1.hasNext()) {
!             leaf1 = (Leaf) itr1.next();
!         } else {
!             leaf1 = null;
!             exhausted1 = true;
!         }
!         return !exhausted1;
!     }
! 
!     /**
!      * If the current leaf is not fully consumed then return immediately.
!      * Otherwise read the next leaf from the 2nd source btree into
!      * {@link #leaf2}. If there are no more leaves available then
!      * {@link #exhausted2} is set to false. {@link #index2} is reset to zero(0)
!      * in either case.
!      * 
!      * @return true unless this source btree is exhausted.
!      */
!     protected boolean nextLeaf2() {
!         if(index2<leaf2.nkeys) return !exhausted2;
!         index2 = 0;
!         if (itr2.hasNext()) {
!             leaf2 = (Leaf) itr2.next();
!         } else {
!             leaf2 = null;
!             exhausted2 = true;
!         }
!         return !exhausted2;
!     }
  
      /**
--- 482,526 ----
      }
      
! //    /**
! //     * If the current leaf is not fully consumed then return immediately.
! //     * Otherwise read the next leaf from the 1nd source btree into
! //     * {@link #leaf1}. If there are no more leaves available then
! //     * {@link #exhausted1} is set to false. {@link #index1} is reset to zero(0)
! //     * in either case.
! //     * 
! //     * @return true unless this source btree is exhausted.
! //     */
! //    protected boolean nextLeaf1() {
! //        if(index1<leaf1.nkeys) return !exhausted1;
! //        index1 = 0;
! //        if (itr1.hasNext()) {
! //            leaf1 = (Leaf) itr1.next();
! //        } else {
! //            leaf1 = null;
! //            exhausted1 = true;
! //        }
! //        return !exhausted1;
! //    }
! //
! //    /**
! //     * If the current leaf is not fully consumed then return immediately.
! //     * Otherwise read the next leaf from the 2nd source btree into
! //     * {@link #leaf2}. If there are no more leaves available then
! //     * {@link #exhausted2} is set to false. {@link #index2} is reset to zero(0)
! //     * in either case.
! //     * 
! //     * @return true unless this source btree is exhausted.
! //     */
! //    protected boolean nextLeaf2() {
! //        if(index2<leaf2.nkeys) return !exhausted2;
! //        index2 = 0;
! //        if (itr2.hasNext()) {
! //            leaf2 = (Leaf) itr2.next();
! //        } else {
! //            leaf2 = null;
! //            exhausted2 = true;
! //        }
! //        return !exhausted2;
! //    }
  
      /**
***************
*** 489,493 ****
  
          if(DEBUG) log.debug("#leavesWritten=" + nleaves + ", src="
!                 + (src == leaf1 ? "leaf1" : "leaf2") + ", srcpos=" + srcpos);
          
          MutableKeyBuffer keys = (MutableKeyBuffer) leaf.keys;
--- 539,543 ----
  
          if(DEBUG) log.debug("#leavesWritten=" + nleaves + ", src="
!                 + (src == itr1.leaf ? "leaf1" : "leaf2") + ", srcpos=" + srcpos);
          
          MutableKeyBuffer keys = (MutableKeyBuffer) leaf.keys;
***************
*** 530,561 ****
          ByteBuffer buf = nodeSer.putNodeOrLeaf( leaf );
  
!         FileChannel outChannel = out.getChannel();
!         
!         // position on the channel before the write.
!         final long offset = outChannel.position();
!         
!         if(offset>Integer.MAX_VALUE) {
!             
!             throw new IOException("Index segment exceeds int32 bytes.");
!             
!         }
          
          final int nbytes = buf.limit();
          
!         /*
!          * write header containing the #of bytes in the record.
!          * 
!          * @todo it is unelegant to have to read in this this 4 byte header for
!          * each leaf. perhaps it would have better performance to write a header
!          * block at the end of the merge file that indexed into the leaves?
!          */
!         out.writeInt(nbytes);
!         
!         // write the compressed record on the channel.
!         final int nbytes2 = outChannel.write(buf);
!         
!         assert nbytes2 == buf.limit();
          
!         System.err.print("."); // wrote a leaf.
  
          nleaves++;
--- 580,592 ----
          ByteBuffer buf = nodeSer.putNodeOrLeaf( leaf );
  
!         // write the record
          
          final int nbytes = buf.limit();
          
!         final long addr = tmpStore.write(buf);
! 
!         addrs.add(addr);
          
!         System.err.print(">"); // wrote a leaf.
  
          nleaves++;
***************
*** 577,582 ****
      public static class MergedLeafIterator implements Iterator<ILeafData> {
  
!         public final File file;
!         protected final RandomAccessFile raf;
          public final int m;
          public final int nentries;
--- 608,613 ----
      public static class MergedLeafIterator implements Iterator<ILeafData> {
  
!         public final TemporaryRawStore tmpStore;
!         public final Vector<Long> addrs;
          public final int m;
          public final int nentries;
***************
*** 586,599 ****
          
          private int leafIndex = 0;
- 
-         /**
-          * Offset of the last leaf read from the file.
-          */
-         private int offset;
-         
-         /**
-          * Used to read leaves from the file.
-          */
-         private final ByteBuffer buf;
          
          /**
--- 617,620 ----
***************
*** 604,609 ****
          
          /**
!          * @param file 
!          * @param raf
           * @param m
           * @param nentries
--- 625,632 ----
          
          /**
!          * @param tmpStore
!          * @param addrs
!          *            An ordered list of the addresses at which the leaves may
!          *            be found in <i>tmpStore</i>.
           * @param m
           * @param nentries
***************
*** 612,621 ****
           * @param nodeSer
           */
!         public MergedLeafIterator(File file, RandomAccessFile raf, int m,
!                 int nentries, int nleaves, int maxLeafBytes,
!                 NodeSerializer nodeSer) throws IOException {
              
!             this.file = file;
!             this.raf = raf;
              this.m = m;
              this.nentries = nentries;
--- 635,644 ----
           * @param nodeSer
           */
!         public MergedLeafIterator(TemporaryRawStore tmpStore,
!                 Vector<Long> addrs, int m, int nentries, int nleaves,
!                 int maxLeafBytes, NodeSerializer nodeSer) throws IOException {
              
!             this.tmpStore = tmpStore;
!             this.addrs = addrs;
              this.m = m;
              this.nentries = nentries;
***************
*** 626,640 ****
                      nodeSer.keySerializer, nodeSer.valueSerializer,
                      nodeSer.recordCompressor, nodeSer.useChecksum);
- 
-             // note: allocates direct buffer when size is large.
-             this.buf = NodeSerializer.alloc(maxLeafBytes);
-             
-             // rewind.
-             raf.seek(0);
              
          }
         
          /**
!          * Close the channel and delete the merge file.
           * <p>
           * This is invoked automatically when the iterator is exhausted. 
--- 649,657 ----
                      nodeSer.keySerializer, nodeSer.valueSerializer,
                      nodeSer.recordCompressor, nodeSer.useChecksum);
              
          }
         
          /**
!          * Closes and deletes the backing store.
           * <p>
           * This is invoked automatically when the iterator is exhausted. 
***************
*** 643,679 ****
           */
          public void close() {
-             
-             if (raf.getChannel().isOpen()) {
- 
-                 try {
-                     
-                     System.err.println("Closing MergedLeafIterator: file="+file);
  
!                     raf.close();
! 
!                 } catch (IOException ex) {
! 
!                     throw new RuntimeException(ex);
! 
!                 }
! 
!                 if (file.exists() && !file.delete()) {
! 
!                     log.warn("Could not delete file: " + file.getAbsoluteFile());
! 
!                 }
!                 
!             }
  
          }
  
          /**
!          * Test whether more leaves are available.
!          * <p>
!          * Automatically closes out the backing buffer when all leaves have been
!          * processed.
!          * 
!          * FIXME I am not seeing an automatic close of this iterator when
!          * invoked by {@link MergedEntryIterator}.
           */
          public boolean hasNext() {
--- 660,673 ----
           */
          public void close() {
  
!             log.info("Closing temporary store");
!             
!             tmpStore.close();
  
          }
  
          /**
!          * Test whether more leaves are available. Automatically closes out the
!          * backing buffer when all leaves have been processed.
           */
          public boolean hasNext() {
***************
*** 695,743 ****
          public ILeafData next() {
  
!             if(!hasNext()) throw new NoSuchElementException();
!             
!             try {
! 
!                 // #of bytes in the next leaf.
!                 int nbytes = raf.readInt();
! 
!                 offset += Bytes.SIZEOF_INT;
!                 
!                 if (DEBUG)
!                     log.debug("will read " + nbytes + " bytes at offset="
!                             + offset);
! 
!                 buf.limit(nbytes);
!                 buf.position(0);
! 
!                 int nread = raf.getChannel().read(buf);
! 
!                 assert nread == nbytes;
! 
!                 offset += nread;
  
!                 long addr = Addr.toLong(nbytes, offset);
  
!                 /*
!                  * Note: this is using a nodeSer whose node factory does not
!                  * require a non-null btree reference.
!                  */ 
!                 
!                 // @todo cleanup buffer logic here and elsewhere.
!                 buf.position(0);
!                 
!                 ILeafData leaf = nodeSer.getLeaf(null/*btree*/, addr, buf);
  
!                 leafIndex++;
!                 
!                 return leaf;
  
!             }
  
!             catch (IOException ex) {
  
!                 throw new RuntimeException(ex);
  
!             }
  
          }
--- 689,712 ----
          public ILeafData next() {
  
!             if (!hasNext())
!                 throw new NoSuchElementException();
  
!             // the address of the next leaf.
!             final long addr = addrs.get(leafIndex);
  
!             /*
!              * Note: this is using a nodeSer whose node factory does not require
!              * a non-null btree reference.
!              */
  
!             ByteBuffer buf = tmpStore.read(addr);
  
!             ILeafData leaf = nodeSer.getLeaf(null/* btree */, addr, buf);
  
!             leafIndex++;
  
!             System.err.print("<"); // read a leaf.
  
!             return leaf;
  
          }

[cweb-CVS] bigdata-rdf/src/java/com/bigdata/rdf BatchTermInsertOp.java, NONE, 1.1 TripleStore.java, 1.20, 1.21

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv24380/src/java/com/bigdata/rdf

Modified Files:
	TripleStore.java 
Added Files:
	BatchTermInsertOp.java 
Log Message:
Refactoring to introduce asynchronous handling of overflow events in support of a scale-up/scale-out design.

--- NEW FILE: BatchTermInsertOp.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Mar 2, 2007
 */

package com.bigdata.rdf;

import java.util.Arrays;

import com.bigdata.objndx.Errors;
import com.bigdata.objndx.IBatchOp;
import com.bigdata.objndx.IIndex;
import com.bigdata.objndx.ISimpleBTree;
import com.bigdata.rdf.model.OptimizedValueFactory.TermIdComparator;
import com.bigdata.rdf.model.OptimizedValueFactory._Value;
import com.bigdata.rdf.model.OptimizedValueFactory._ValueSortKeyComparator;

/**
 * A batch operation for loading data into the terms and ids indices. This
 * operation is designed to write on <em>unisolated</em> indices - that is,
 * outside of any transactional isolation. This strategy relies on the fact that
 * (a) the terms index is the sole source of term ids; and (b) the batch api
 * provides an atomic guarentee for operations on a single index partition.
 * Accordingly, this batch operation takes responsibility for breaking down the
 * terms according to the then current index partitions.
 * <p>
 * If a term is not found in the term index, then it is synchronously inserted.
 * The insert operation defines the identifier. This occurs during a
 * single-threaded write on the unisolated terms index. In practice, we lookup
 * and, if lookup fails, define, the term identifier for all terms that would be
 * found in a single index partition at once. Batch operations for terms that
 * would be found in different index partitions may be executed in parallel. If
 * the operation fails for any single partition, then it may proceed for all
 * remaining partitions or abort eagerly.
 * <p>
 * If a term identifier is NOT found, then the operation caused the term
 * identifier to be assigned and must record the id:term mapping in the ids
 * index. Since the writes on the ids index will be on different partitions and,
 * more likely than not, different hosts, we can not obtain a simple guarentee
 * of atomicity across writes on both the terms and the ids index. Therefore, in
 * order to guard against failures or concurrent inserts, we may be required to
 * conditionally insert terms in to the ids index regardless of whether they are
 * found on the terms index since we could not otherwise guarentee that the
 * write on the terms index partition and the corresponding (scattered) writes
 * on the ids index partition would be atomic as a unit.
 * <p>
 * We optimize the scattered writes on the ids index by sorting by the term
 * identifier and scattering ordered writes against the various partitions of
 * the ids index.
 * 
 * @todo If the batch term insert operation were done transactionally then we
 *       could guarentee atomicity across the writes on the terms and ids
 *       indices.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
abstract public class BatchTermInsertOp implements IBatchOp {

//    /**
//     * The #of tuples to be processed.
//     */
//    public final int ntuples;
//    
//    /**
//     * The keys for each tuple.
//     */
//    public final byte[][] keys;
//    
//    /**
//     * The value corresponding to each key.
//     */
//    public final boolean[] contains;
//    
//    /**
//     * The index of the tuple that is currently being processed.
//     */
//    public int tupleIndex = 0;
//    
//    /**
//     * Create a batch operation to lookup/define terms.
//     * 
//     * @param ntuples
//     *            The #of tuples in the operation (in).
//     * 
//     * @param keys
//     *            A series of keys paired to values (in). Each key is an
//     *            variable length unsigned byte[]. The keys must be presented in
//     *            sorted order in order to obtain maximum efficiency for the
//     *            batch operation.
//     * 
//     * @param contains
//     *            An array of boolean flags, one per tuple (in,out). On input,
//     *            the tuple will be tested iff the corresponding element is
//     *            <code>false</code> (this supports chaining of this operation
//     *            on a view over multiple btrees). On output, the array element
//     *            corresponding to a tuple will be true iff the key exists.
//     * 
//     * @exception IllegalArgumentException
//     *                if the dimensions of the arrays are not sufficient for the
//     *                #of tuples declared.
//     */
//    public BatchTermInsertOp(int ntuples, byte[][] keys, boolean[] contains) {
//        
//        if (ntuples <= 0)
//            throw new IllegalArgumentException(Errors.ERR_NTUPLES_NON_POSITIVE);
//
//        if (keys == null)
//            throw new IllegalArgumentException(Errors.ERR_KEYS_NULL);
//
//        if (keys.length < ntuples)
//            throw new IllegalArgumentException(Errors.ERR_NOT_ENOUGH_KEYS);
//
//        if (contains == null)
//            throw new IllegalArgumentException(Errors.ERR_VALS_NULL);
//
//        if (contains.length < ntuples)
//            throw new IllegalArgumentException(Errors.ERR_NOT_ENOUGH_VALS);
//
//        this.ntuples = ntuples;
//        this.keys = keys;
//        this.contains = contains;
//
//    }
//
//    /**
//     * Applies the operation.
//     * 
//     * @param btree
//     */
//    public void apply(ISimpleBTree btree) {
//        
//        while( tupleIndex < ntuples ) {
//
//            // skip tuples already marked as true.
//            if (contains[tupleIndex]) {
//
//                tupleIndex++;
//
//                continue;
//
//            }
//
//            contains[tupleIndex] = btree.contains(keys[tupleIndex]);
//
//            tupleIndex ++;
//
//        }
//
//    }
//    
//    /**
//     * Batch insert of terms into the database.
//     * 
//     * @param terms
//     *            An array whose elements [0:nterms-1] will be inserted.
//     * @param numTerms
//     *            The #of terms to insert.
//     * @param haveKeys
//     *            True if the terms already have their sort keys.
//     * @param sorted
//     *            True if the terms are already sorted by their sort keys (in
//     *            the correct order for a batch insert).
//     * 
//     * @exception IllegalArgumentException
//     *                if <code>!haveKeys && sorted</code>.
//     */
//    public BatchTermInsertOp( _Value[] terms, int numTerms, boolean haveKeys, boolean sorted,
//            RdfKeyBuilder keyBuilder ) {
//
//        if (numTerms == 0)
//            return;
//
//        if (!haveKeys && sorted)
//            throw new IllegalArgumentException("sorted requires keys");
//        
//        long begin = System.currentTimeMillis();
//        long keyGenTime = 0; // time to convert unicode terms to byte[] sort keys.
//        long sortTime = 0; // time to sort terms by assigned byte[] keys.
//        long insertTime = 0; // time to insert terms into the forward and reverse index.
//        
//        System.err.print("Writing "+numTerms+" terms ("+terms.getClass().getSimpleName()+")...");
//
//        {
//
//            /*
//             * First make sure that each term has an assigned sort key.
//             */
//            if(!haveKeys) {
//
//                long _begin = System.currentTimeMillis();
//                
//                generateSortKeys(keyBuilder, terms, numTerms);
//                
//                keyGenTime = System.currentTimeMillis() - _begin;
//
//            }
//            
//            /* 
//             * Sort terms by their assigned sort key.  This places them into
//             * the natural order for the term:id index.
//             */
//
//            if (!sorted ) {
//            
//                long _begin = System.currentTimeMillis();
//                
//                Arrays.sort(terms, 0, numTerms, _ValueSortKeyComparator.INSTANCE);
//
//                sortTime += System.currentTimeMillis() - _begin;
//                
//            }
//
//            {
//            
//                /*
//                 * Lookup the term in the term:id index. If it is there then
//                 * take its termId and mark it as 'known' so that we can avoid
//                 * testing the reverse index. Otherwise, insert the term into
//                 * the term:id index which gives us its termId.
//                 * 
//                 * @todo use batch api?
//                 */
//
//                IIndex termId = getTermIdIndex();
//                AutoIncCounter counter = getCounter();
//                
//                long _begin = System.currentTimeMillis();
//                
//                for (int i = 0; i < numTerms; i++) {
//
//                    _Value term = terms[i];
//
//                    if (!term.known) {
//
//                        //assert term.termId==0L; FIXME uncomment this and figure out why this assertion is failing.
//                        assert term.key != null;
//
//                        // Lookup in the forward index.
//                        Long tmp = (Long)termId.lookup(term.key);
//                        
//                        if(tmp == null) { // not found.
//
//                            // assign termId.
//                            term.termId = counter.nextId();
//                        
//                            // insert into index.
//                            if(termId.insert(term.key, Long.valueOf(term.termId))!=null) {
//                                
//                                throw new AssertionError();
//                                
//                            }
//                            
//                        } else { // found.
//                        
//                            term.termId = tmp.longValue();
//                            
//                            term.known = true;
//                        
//                        }
//                        
//                    } else assert term.termId != 0L;
//                    
//                }
//
//                insertTime += System.currentTimeMillis() - _begin;
//                
//            }
//            
//        }
//        
//        {
//            
//            /*
//             * Sort terms based on their assigned termId.
//             * 
//             * FIXME The termId should be converted to a byte[8] for this sort
//             * order since that is how the termId is actually encoded when we
//             * insert into the termId:term (reverse) index.  We could also 
//             * fake this with a comparator that compared the termIds as if they
//             * were _unsigned_ long integers.
//             */
//
//            long _begin = System.currentTimeMillis();
//
//            Arrays.sort(terms, 0, numTerms, TermIdComparator.INSTANCE);
//
//            sortTime += System.currentTimeMillis() - _begin;
//
//        }
//        
//        {
//        
//            /*
//             * Add unknown terms to the reverse index, which is the index that
//             * we use to lookup the RDF value by its termId to serialize some
//             * data as RDF/XML or the like.
//             * 
//             * Note: We only insert terms that were reported as "not found" when
//             * we inserted them into the forward mapping. This reduces the #of
//             * index tests that we perform.
//             * 
//             * @todo use batch api?
//             */
//            
//            IIndex idTerm = getIdTermIndex();
//            
//            long _begin = System.currentTimeMillis();
//            
//            for (int i = 0; i < numTerms; i++) {
//
//                _Value term = terms[i];
//                
//                assert term.termId != 0L;
//                
//                if (!term.known) {
//                    
//                    final byte[] idKey = keyBuilder.id2key(term.termId);
//
//                    if(idTerm.insert(idKey, term) != null) {
//
//                        // term was already in this index.
//                        
//                        throw new AssertionError();
//                        
//                    }
//                    
//                    term.known = true; // now in the fwd and rev indices.
//                    
//                }
//
//            }
//
//            insertTime += System.currentTimeMillis() - _begin;
//
//        }
//
//        long elapsed = System.currentTimeMillis() - begin;
//        
//        System.err.println("in " + elapsed + "ms; keygen=" + keyGenTime
//                + "ms, sort=" + sortTime + "ms, insert=" + insertTime + "ms");
//        
//    }

}

Index: TripleStore.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/TripleStore.java,v
retrieving revision 1.20
retrieving revision 1.21
diff -C2 -d -r1.20 -r1.21
*** TripleStore.java	22 Feb 2007 16:58:59 -0000	1.20
--- TripleStore.java	6 Mar 2007 20:38:13 -0000	1.21
***************
*** 69,73 ****
--- 69,76 ----
  import com.bigdata.journal.Tx;
  import com.bigdata.objndx.BTree;
+ import com.bigdata.objndx.Errors;
+ import com.bigdata.objndx.IBatchOp;
  import com.bigdata.objndx.IIndex;
+ import com.bigdata.objndx.ISimpleBTree;
  import com.bigdata.objndx.KeyBuilder;
  import com.bigdata.rawstore.Bytes;
***************
*** 576,583 ****
--- 579,592 ----
          long sortTime = 0; // time to sort terms by assigned byte[] keys.
          long insertTime = 0; // time to insert terms into the forward and reverse index.
+         
+         final long elapsed_SPO;
+         final long elapsed_POS;
+         final long elapsed_OSP;
  
          System.err.print("Writing " + numStmts + " statements...");
  
          { // SPO
+ 
+             final long beginIndex = System.currentTimeMillis();
              
              IIndex ndx_spo = getSPOIndex();
***************
*** 610,617 ****
--- 619,630 ----
              }
  
+             elapsed_SPO = System.currentTimeMillis() - beginIndex;
+             
          }
  
          { // POS
  
+             final long beginIndex = System.currentTimeMillis();
+             
              IIndex ndx_pos = getPOSIndex();
  
***************
*** 643,649 ****
--- 656,666 ----
              }
  
+             elapsed_POS = System.currentTimeMillis() - beginIndex;
+ 
          }
  
          { // OSP
+ 
+             final long beginIndex = System.currentTimeMillis();
              
              IIndex ndx_osp = getOSPIndex();
***************
*** 676,679 ****
--- 693,698 ----
              }
  
+             elapsed_OSP = System.currentTimeMillis() - beginIndex;
+ 
          }
  
***************
*** 681,685 ****
  
          System.err.println("in " + elapsed + "ms; sort=" + sortTime
!                 + "ms, keyGen+insert=" + insertTime + "ms");
          
      }
--- 700,705 ----
  
          System.err.println("in " + elapsed + "ms; sort=" + sortTime
!                 + "ms, keyGen+insert=" + insertTime + "ms; spo=" + elapsed_SPO
!                 + "ms, pos=" + elapsed_POS + "ms, osp=" + elapsed_OSP + "ms");
          
      }
***************
*** 906,910 ****
          
      }
!     
      /**
       * Add a term into the term:id index and the id:term index, returning the
--- 926,930 ----
          
      }
! 
      /**
       * Add a term into the term:id index and the id:term index, returning the

[cweb-CVS] bigdata/src/java/com/bigdata/rawstore SimpleMemoryRawStore.java, 1.4, 1.5 SimpleFileRawStore.java, 1.5, 1.6 IRawStore.java, 1.5, 1.6

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/rawstore
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv23960/src/java/com/bigdata/rawstore

Modified Files:
	SimpleMemoryRawStore.java SimpleFileRawStore.java 
	IRawStore.java 
Log Message:
Refactoring to introduce asynchronous handling of overflow events in support of a scale-up/scale-out design.

Index: SimpleFileRawStore.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/rawstore/SimpleFileRawStore.java,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** SimpleFileRawStore.java	22 Feb 2007 16:59:35 -0000	1.5
--- SimpleFileRawStore.java	6 Mar 2007 20:38:05 -0000	1.6
***************
*** 329,332 ****
--- 329,346 ----
                  
      }
+ 
+     public long size() {
+         
+         try {
+         
+             return raf.length();
+             
+         } catch(IOException ex) {
+             
+             throw new RuntimeException(ex);
+             
+         }
+         
+     }
      
  }

Index: IRawStore.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/rawstore/IRawStore.java,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** IRawStore.java	22 Feb 2007 16:59:35 -0000	1.5
--- IRawStore.java	6 Mar 2007 20:38:05 -0000	1.6
***************
*** 258,260 ****
--- 258,266 ----
      public void force(boolean metadata);
      
+     /**
+      * The #of application data bytes written on the store (does not count any
+      * headers or root blocks that may exist for the store).
+      */
+     public long size();
+     
  }

Index: SimpleMemoryRawStore.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/rawstore/SimpleMemoryRawStore.java,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** SimpleMemoryRawStore.java	22 Feb 2007 16:59:35 -0000	1.4
--- SimpleMemoryRawStore.java	6 Mar 2007 20:38:05 -0000	1.5
***************
*** 294,296 ****
--- 294,302 ----
      }
      
+     public long size() {
+         
+         return nextOffset;
+         
+     }
+     
  }

[cweb-CVS] bigdata/src/java/com/bigdata/scaleup PartitionedIndex.java, 1.6, 1.7 PartitionedJournal.java, 1.11, 1.12 SegmentMetadata.java, 1.1, 1.2 MetadataIndex.java, 1.4, 1.5

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv23960/src/java/com/bigdata/scaleup

Modified Files:
	PartitionedIndex.java PartitionedJournal.java 
	SegmentMetadata.java MetadataIndex.java 
Log Message:
Refactoring to introduce asynchronous handling of overflow events in support of a scale-up/scale-out design.

Index: MetadataIndex.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/MetadataIndex.java,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** MetadataIndex.java	9 Feb 2007 18:56:59 -0000	1.4
--- MetadataIndex.java	6 Mar 2007 20:38:06 -0000	1.5
***************
*** 62,65 ****
--- 62,67 ----
   * @version $Id$
   * 
+  * @todo mutation operations need to be synchronized.
+  * 
   * @todo define a UUID so that is at least possible to rename a partitioned
   *       index? the uuid would be store in the metadata record for the metadata

Index: PartitionedIndex.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/PartitionedIndex.java,v
retrieving revision 1.6
retrieving revision 1.7
diff -C2 -d -r1.6 -r1.7
*** PartitionedIndex.java	13 Feb 2007 23:01:04 -0000	1.6
--- PartitionedIndex.java	6 Mar 2007 20:38:06 -0000	1.7
***************
*** 68,71 ****
--- 68,73 ----
  import com.bigdata.objndx.IndexSegment;
  import com.bigdata.objndx.IndexSegmentFileStore;
+ import com.bigdata.scaleup.PartitionedJournal.IViewMetadata;
+ import com.bigdata.scaleup.PartitionedJournal.JournalMetadata;
  
  /**
***************
*** 247,250 ****
--- 249,297 ----
          
      }
+ 
+     /**
+      * Return the resources required to provide a coherent view of the partition
+      * in which the key is found.
+      * 
+      * @param key
+      *            The key.
+      * 
+      * @return The resources required to read on the partition containing that
+      *         key. The resources are arranged in reverse timestamp order
+      *         (increasing age).
+      * 
+      * @todo reconcile with {@link #getView(byte[])}?
+      */
+     protected IViewMetadata[] getResources(byte[] key) {
+ 
+         JournalMetadata journalResource = new JournalMetadata((Journal) btree
+                 .getStore(), IndexSegmentLifeCycleEnum.LIVE);
+ 
+         PartitionMetadata pmd = mdi.find(key);
+ 
+         final int liveCount = pmd.getLiveCount();
+ 
+         IViewMetadata[] resources = new IViewMetadata[liveCount + 1];
+ 
+         int n = 0;
+ 
+         resources[n++] = journalResource;
+ 
+         for (int i = 0; i < resources.length; i++) {
+ 
+             SegmentMetadata seg = pmd.segs[i];
+ 
+             if (seg.state != IndexSegmentLifeCycleEnum.LIVE)
+                 continue;
+ 
+             resources[n++] = seg;
+ 
+         }
+ 
+         assert n == resources.length;
+         
+         return resources;
+ 
+     }
      
      public String getName() {

Index: SegmentMetadata.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/SegmentMetadata.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** SegmentMetadata.java	8 Feb 2007 21:32:14 -0000	1.1
--- SegmentMetadata.java	6 Mar 2007 20:38:06 -0000	1.2
***************
*** 44,48 ****
--- 44,51 ----
  package com.bigdata.scaleup;
  
+ import java.io.File;
+ 
  import com.bigdata.objndx.IndexSegment;
+ import com.bigdata.scaleup.PartitionedJournal.IViewMetadata;
  
  /**
***************
*** 52,56 ****
   * @version $Id$
   */
! public class SegmentMetadata {
  
      /**
--- 55,59 ----
   * @version $Id$
   */
! public class SegmentMetadata implements IViewMetadata {
  
      /**
***************
*** 91,94 ****
          
      }
      
! }
\ No newline at end of file
--- 94,109 ----
          
      }
+ 
+     public File getFile() {
+         return new File(filename);
+     }
+ 
+     public long size() {
+         return nbytes;
+     }
+ 
+     public IndexSegmentLifeCycleEnum status() {
+         return state;
+     }
      
! }

Index: PartitionedJournal.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/PartitionedJournal.java,v
retrieving revision 1.11
retrieving revision 1.12
diff -C2 -d -r1.11 -r1.12
*** PartitionedJournal.java	28 Feb 2007 13:59:10 -0000	1.11
--- PartitionedJournal.java	6 Mar 2007 20:38:06 -0000	1.12
***************
*** 53,57 ****
--- 53,62 ----
  import java.util.Iterator;
  import java.util.Properties;
+ import java.util.concurrent.Executors;
  
+ import com.bigdata.isolation.IIsolatableIndex;
+ import com.bigdata.isolation.UnisolatedBTree;
+ import com.bigdata.isolation.Value;
+ import com.bigdata.journal.BufferMode;
  import com.bigdata.journal.CommitRecordIndex;
  import com.bigdata.journal.ICommitRecord;
***************
*** 62,71 ****
--- 67,80 ----
  import com.bigdata.journal.Journal;
  import com.bigdata.journal.Name2Addr.Entry;
+ import com.bigdata.objndx.AbstractBTree;
  import com.bigdata.objndx.BTree;
  import com.bigdata.objndx.FusedView;
  import com.bigdata.objndx.IIndex;
+ import com.bigdata.objndx.IValueSerializer;
  import com.bigdata.objndx.IndexSegment;
  import com.bigdata.objndx.IndexSegmentBuilder;
  import com.bigdata.objndx.IndexSegmentMerger;
+ import com.bigdata.objndx.IndexSegmentMetadata;
+ import com.bigdata.objndx.RecordCompressor;
  import com.bigdata.objndx.IndexSegmentMerger.MergedEntryIterator;
  import com.bigdata.objndx.IndexSegmentMerger.MergedLeafIterator;
***************
*** 620,629 ****
      public void overflow() {
                 
-         synchronousOverflow();
-         
-     }
- 
-     protected void synchronousOverflow() {
-         
          /* 
           * Create the new buffer.
--- 629,632 ----
***************
*** 649,652 ****
--- 652,737 ----
          final SlaveJournal oldJournal = slave;
          
+         synchronousOverflow(oldJournal, newJournal);
+         
+         /*
+          * replace the old buffer with the new buffer and delete the old buffer.
+          * 
+          * @todo consider scheduling the old buffer for deletion instead so that
+          * we have some time to verify that the new index segments are good
+          * before deleting the buffered writes.  We are already doing this with
+          * the old segments.
+          */
+         
+         // atomic commit makes these changes restart safe
+         newJournal.commit();
+ 
+         // Change to the new journal.
+         slave = newJournal;
+ 
+         // immediate shutdown of the old journal.
+         oldJournal.closeAndDelete();
+         
+ //        // delete the old backing file (if any).
+ //        oldJournal.getBufferStrategy().deleteFile();
+         
+     }
+ 
+     /**
+      * Does not queue indices for eviction, forcing the old journal to remain
+      * active. The old journal is re-opened using {@link BufferMode#Disk} in
+      * order to reduce the amount of RAM required to buffer the data. This
+      * option is conceptually the simplest. It has the advantage of never
+      * discarding historical commit states, and the corresponding disadvantage
+      * of never throwing away old data that is no longer reachable from the
+      * current committed state. Old journals can be distributed, just like index
+      * segments, but they are not optimized for read performance and they
+      * multiplex together multiple indices using smaller branching factors.
+      */
+     protected void noSegmentsOverflow() {
+         
+         throw new UnsupportedOperationException();
+         
+     }
+     
+     /**
+      * Creates an asynchronous task that will evict data onto index segments.
+      * For each named index for which the journal has absorbed writes, the task
+      * is responsible for identifying the index partitions that were written on
+      * and queuing up (in the index key order) tasks to build/merge the data on
+      * the journal for each partition onto a new index segment.
+      * <p>
+      * While these tasks can be run in parallel either for the same index or for
+      * different indices, parallism could impose a significant resource burden
+      * and a single worker thread is probably sufficient to keep pace with
+      * writes absorbed on the new journal.
+      * <p>
+      * The schedule of tasks to be executed needs to be restart safe so that all
+      * tasks will run eventually. This is most easily accomplished by
+      * representing the schedule on a btree that is migrated from
+      * {@link SlaveJournal} to {@link SlaveJournal} on {@link #overflow()}.
+      * Writes on that btree MUST be synchronized since concurrent worker threads
+      * could update the schedule state concurrently and an {@link #overflow()}
+      * event could be concurrent with the execution of a worker thread.
+      * 
+      * @todo consider making the touched (written on) partitions part of the
+      *       persistent state of the {@link UnisolatedBTree} so that we do not
+      *       need to explicitly search the key space of the btree and compare it
+      *       with the key space of the partitions.
+      */
+     protected void asynchronousOverflow() {
+ 
+         throw new UnsupportedOperationException();
+ 
+     }
+     
+     /**
+      * Synchronously evicts all data onto {@link IndexSegment}s.
+      * 
+      * @todo rewrite this in terms of {@link IPartitionTask}s that get executed
+      *       synchronously.
+      */
+     protected void synchronousOverflow(SlaveJournal oldJournal,
+             SlaveJournal newJournal) {
+         
          /*
           * We need to consider each named btree registered on this store. We can
***************
*** 740,744 ****
               * the still valid open index segments across an overflow. This is
               * important because opening an index segment has significant
!              * latency.
               */
              oldIndex.closeViews();
--- 825,829 ----
               * the still valid open index segments across an overflow. This is
               * important because opening an index segment has significant
!              * latency (fix this using a weak value cache for open indices).
               */
              oldIndex.closeViews();
***************
*** 757,781 ****
          }
          
-         /*
-          * replace the old buffer with the new buffer and delete the old buffer.
-          * 
-          * @todo consider scheduling the old buffer for deletion instead so that
-          * we have some time to verify that the new index segments are good
-          * before deleting the buffered writes.  We are already doing this with
-          * the old segments.
-          */
-         
-         // atomic commit makes these changes restart safe
-         newJournal.commit();
- 
-         // Change to the new journal.
-         slave = newJournal;
- 
-         // immediate shutdown of the old journal.
-         oldJournal.closeAndDelete();
-         
- //        // delete the old backing file (if any).
- //        oldJournal.getBufferStrategy().deleteFile();
-         
      }
      
--- 842,845 ----
***************
*** 878,886 ****
              final IndexSegment seg = (IndexSegment)view.srcs[1];
  
-             // tmp file for the merge process.
-             File tmpFile = File.createTempFile("merge", ".tmp", tmpDir);
- 
-             tmpFile.deleteOnExit();
- 
              // output file for the merged segment.
              File outFile = getSegmentFile(name, pmd.partId, segId);
--- 942,945 ----
***************
*** 888,893 ****
              // merge the data from the btree on the slave and the index
              // segment.
!             MergedLeafIterator mergeItr = new IndexSegmentMerger(tmpFile,
!                     mseg2, oldIndex.btree, seg).merge();
  
              // build the merged index segment.
--- 947,952 ----
              // merge the data from the btree on the slave and the index
              // segment.
!             MergedLeafIterator mergeItr = new IndexSegmentMerger(mseg2,
!                     oldIndex.btree, seg).merge();
  
              // build the merged index segment.
***************
*** 986,989 ****
--- 1045,1617 ----
  
      }
+ 
+     /**
+      * Interface for metadata about a {@link Journal} or {@link IndexSegment}.
+      * 
+      * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+      * @version $Id$
+      * 
+      * @todo rename IResourceMetadata
+      */
+     public static interface IViewMetadata {
+        
+         /**
+          * The store file.
+          */
+         public File getFile();
+         
+         /**
+          * The #of bytes in the store file.
+          */
+         public long size();
+     
+         /**
+          * The life cycle state of that store file.
+          * 
+          * @todo rename ResourceState/state()
+          */
+         public IndexSegmentLifeCycleEnum status();
+     
+     }
+     
+     /**
+      * Metadata required to locate a {@link Journal} resource.
+      * 
+      * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+      * @version $Id$
+      * 
+      * @todo make this persistence capable by modifying the value serializer to
+      * use the {@link IViewMetadata} interface.
+      */
+     public static class JournalMetadata implements IViewMetadata {
+ 
+         protected final String filename;
+         protected final IndexSegmentLifeCycleEnum state;
+         
+         public File getFile() {
+             return new File(filename);
+         }
+ 
+         /**
+          * Always returns ZERO (0L) since we can not accurately estimate the #of
+          * bytes on the journal dedicated to a given partition of a named index.
+          */
+         public long size() {
+             return 0L;
+         }
+ 
+         public IndexSegmentLifeCycleEnum status() {
+             return state;
+         }
+ 
+         public JournalMetadata(Journal journal, IndexSegmentLifeCycleEnum state) {
+             
+             if(journal.getFile()==null) {
+                 
+                 throw new IllegalArgumentException("Journal is not persistent.");
+                 
+             }
+             
+             this.filename = journal.getFile().toString();
+ 
+             this.state = state;
+             
+         }
+         
+     }
+ 
+     /**
+      * Interface for a scheduleable task that produces one or more
+      * {@link IndexSegment}s, updates the {@link MetadataIndex} to reflect the
+      * existence of the new {@link IndexSegment}s and notifies existing views
+      * with a depedency on the source(s) that they must switch over to the new
+      * {@link IndexSegment}s.
+      * 
+      * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+      * @version $Id$
+      */
+     public static interface IPartitionTask extends
+             java.util.concurrent.Callable<Object> {
+         
+         /**
+          * Run the task.
+          * 
+          * @return No return semantics are defined.
+          * 
+          * @throws Exception
+          *             The exception thrown by the task.
+          */
+         public Object call() throws Exception;
+ 
+     }
+ 
+     /**
+      * Abstract base class for tasks that build {@link IndexSegment}(s).
+      * 
+      * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+      * @version $Id$
+      * 
+      * @todo write test suite for executing partition task schedules.
+      * 
+      * @todo add a
+      *       {@link Executors#newSingleThreadExecutor(java.util.concurrent.ThreadFactory)}
+      *       that will be used to run these tasks and modify the shutdown logic
+      *       for the master to also shutdown that thread.
+      * 
+      * @todo add result to persistent schedule outcome so that this is restart
+      *       safe.
+      * 
+      * @todo once the {@link IndexSegment} is ready the metadata index needs to
+      *       be updated to reflect that the indexsegment is live and the views
+      *       that rely on the partition on the old journal need to be
+      *       invalidated so new views utilize the new indexsegment rather than
+      *       the data on the old journal.
+      * 
+      * @todo the old journal is not available for release until all partitions
+      *       for all indices have been evicted. we need to track that in a
+      *       restart safe manner.
+      * 
+      * @todo parameterize useChecksum, recordCompressor.
+      * 
+      * @todo assiging sequential segment identifiers may impose an unnecessary
+      *       message overhead since we can just use the temporary file mechanism
+      *       and the inspect the {@link IndexSegmentMetadata} to learn more
+      *       about a given store file.
+      * 
+      * @todo try performance with and without checksums and with and without
+      *       record compression.
+      */
+     abstract public static class AbstractPartitionTask implements
+             IPartitionTask {
+ 
+         protected final PartitionedJournal master;
+         /**
+          * Branching factor used for generated {@link IndexSegment}(s). 
+          */
+         protected final int branchingFactor;
+         protected final double errorRate;
+         protected final String name;
+         protected final int partId;
+         protected final byte[] fromKey;
+         protected final byte[] toKey;
+ 
+         /**
+          * Branching factor used for temporary file(s).
+          */
+         protected final int tmpFileBranchingFactor = Bytes.kilobyte32*4;
+ 
+         /**
+          * When true, pre-record checksum are generated for the output
+          * {@link IndexSegment}.
+          */
+         protected final boolean useChecksum = false;
+         
+         /**
+          * When non-null, a {@link RecordCompressor} will be applied to the
+          * output {@link IndexSegment}.
+          */
+         protected final RecordCompressor recordCompressor = null;
+         
+         /**
+          * The serializer used by all {@link IIsolatableIndex}s.
+          */
+         static protected final IValueSerializer valSer = Value.Serializer.INSTANCE;
+         
+         /**
+          * 
+          * @param master
+          *            The master.
+          * @param name
+          *            The index name.
+          * @param branchingFactor
+          *            The branching factor to be used on the new
+          *            {@link IndexSegment}(s).
+          * @param errorRate
+          *            The error rate for the bloom filter for the new
+          *            {@link IndexSegment}(s) -or- zero(0d) if no bloom filter
+          *            is desired.
+          * @param partId
+          *            The unique partition identifier for the partition.
+          * @param fromKey
+          *            The first key that would be accepted into that partition
+          *            (aka the separator key for that partition).<br>
+          *            Note: Failure to use the actual separatorKeys for the
+          *            partition will result in changing the separator key in a
+          *            manner that is incoherent! The change arises since the
+          *            updates are stored in the metadata index based on the
+          *            supplied <i>fromKey</i>.
+          * @param toKey
+          *            The first key that would NOT be accepted into that
+          *            partition (aka the separator key for the right sibling
+          *            partition and <code>null</code> iff there is no right
+          *            sibling).
+          * 
+          * @todo It is an error to supply a <i>fromKey</i> that is not also the
+          *       separatorKey for the partition, however the code does not
+          *       detect this error.
+          * 
+          * @todo It is possible for the partition to be redefined (joined with a
+          *       sibling or split into two partitions). In this case any already
+          *       scheduled operations MUST abort and new operations with the
+          *       correct separator keys must be scheduled.
+          */
+         public AbstractPartitionTask(PartitionedJournal master, String name,
+                 int branchingFactor, double errorRate, int partId,
+                 byte[] fromKey, byte[] toKey) {
+ 
+             this.master = master;
+             this.branchingFactor = branchingFactor;
+             this.errorRate = errorRate;
+             this.name = name;
+             this.partId = partId;
+             this.fromKey = fromKey;
+             this.toKey = toKey;
+ 
+         }
+ 
+         /**
+          * 
+          * @todo update the metadata index (synchronized) and invalidate
+          *       existing partitioned index views that depend on the source
+          *       index.
+          *       
+          * @param resources
+          */
+         protected void updatePartition(IViewMetadata[] resources) {
+ 
+             throw new UnsupportedOperationException();
+             
+         }
+         
+     }
+ 
+     /**
+      * Task builds an {@link IndexSegment} for a partition from data on a
+      * historical read-only {@link SlaveJournal}. When the {@link IndexSegment}
+      * is ready the metadata index is updated to make the segment "live" and
+      * existing views are notified that the source data on the partition must be
+      * invalidated in favor of the new {@link IndexSegment}.
+      * 
+      * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+      * @version $Id$
+      * 
+      * @todo this must use a merge rule that knows about deletion markers and is
+      *       only usable when the input is an {@link UnisolatedBTree}. The
+      *       output {@link IndexSegment} will contain timestamps and deletion
+      *       markers and support isolation.
+      */
+     public static class BuildTask extends AbstractPartitionTask {
+ 
+         private final IViewMetadata src;
+         private final int segId;
+         
+         /**
+          * 
+          * @param src
+          *            The source for the build operation. Only those entries in
+          *            the described key range will be used.
+          * @param segId
+          *            The output segment identifier.
+          */
+         public BuildTask(PartitionedJournal master, String name,
+                 int branchingFactor, double errorRate, int partId,
+                 byte[] fromKey, byte[] toKey, IViewMetadata src, int segId) {
+             
+             super(master,name,branchingFactor,errorRate,partId,fromKey,toKey);
+             
+             this.src = src;
+             
+             this.segId = segId;
+             
+         }
+         
+         /**
+          * Build an {@link IndexSegment} from a key range corresponding to an
+          * index partition.
+          */
+         public Object call() throws Exception {
+ 
+             AbstractBTree src = master.getIndex(name,this.src);
+             
+             File outFile = master.getSegmentFile(name, partId, segId);
+ 
+             new IndexSegmentBuilder(outFile, master.tmpDir, src.rangeCount(
+                     fromKey, toKey), src.rangeIterator(fromKey, toKey),
+                     branchingFactor, valSer, useChecksum, recordCompressor,
+                     errorRate);
+ 
+             IViewMetadata[] resources = new SegmentMetadata[] { new SegmentMetadata(
+                     "" + outFile, outFile.length(),
+                     IndexSegmentLifeCycleEnum.NEW) };
+ 
+             updatePartition(resources);
+             
+             return null;
+             
+         }
+         
+     }
+ 
+     /**
+      * Abstract base class for compacting merge tasks.
+      * 
+      * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+      * @version $Id$
+      */
+     abstract static class AbstractMergeTask extends AbstractPartitionTask {
+ 
+         protected final int segId;
+         protected final boolean fullCompactingMerge;
+ 
+         /**
+          * A compacting merge of two or more resources.
+          * 
+          * @param segId
+          *            The output segment identifier.
+          * 
+          * @param fullCompactingMerge
+          *            True iff this will be a full compacting merge.
+          */
+         protected AbstractMergeTask(PartitionedJournal master, String name,
+                 int branchingFactor, double errorRate, int partId,
+                 byte[] fromKey, byte[] toKey, int segId,
+                 boolean fullCompactingMerge) {
+             
+             super(master, name, branchingFactor, errorRate, partId, fromKey,
+                     toKey);
+             
+             this.segId = segId;
+             
+             this.fullCompactingMerge = fullCompactingMerge;
+             
+         }
+         
+         /**
+          * Compacting merge of two or more resources - deletion markers are
+          * preserved iff {@link #fullCompactingMerge} is <code>false</code>.
+          * 
+          * @todo make sure that deletion markers get removed from the view as
+          *       part of the index segment merger logic (choose the most recent
+          *       write for each key, but if it is a delete then remove the entry
+          *       from the merge so that it does not appear in the temporary file
+          *       that is the input to the {@link IndexSegmentBuilder}).
+          */
+         public Object call() throws Exception {
+             
+             // tmp file for the merge process.
+             File tmpFile = File.createTempFile("merge", ".tmp", master.tmpDir);
+ 
+             tmpFile.deleteOnExit();
+ 
+             // output file for the merged segment.
+             File outFile = master.getSegmentFile(name, partId, segId);
+ 
+             IViewMetadata[] resources = getResources();
+             
+             AbstractBTree[] srcs = new AbstractBTree[resources.length];
+             
+             for(int i=0; i<srcs.length; i++) {
+ 
+                 // open the index - closed by the weak value cache.
+                 srcs[i] = master.getIndex(name, resources[i]);
+                 
+             }
+             
+             // merge the data from the btree on the slave and the index
+             // segment.
+             MergedLeafIterator mergeItr = new IndexSegmentMerger(
+                     tmpFileBranchingFactor, srcs).merge();
+ 
+             // build the merged index segment.
+             new IndexSegmentBuilder(outFile, null, mergeItr.nentries,
+                     new MergedEntryIterator(mergeItr), branchingFactor, valSer,
+                     useChecksum, recordCompressor, errorRate);
+ 
+             // close the merged leaf iterator (and release its buffer/file).
+             // @todo this should be automatic when the iterator is exhausted but
+             // I am not seeing that.
+             mergeItr.close();
+ 
+             /*
+              * @todo Update the metadata index for this partition. This needs to
+              * be an atomic operation so we have to synchronize on the metadata
+              * index or simply move the operation into the MetadataIndex API and
+              * let it encapsulate the problem.
+              * 
+              * We mark the earlier index segment as "DEAD" for this partition
+              * since it has been replaced by the merged result.
+              * 
+              * Note: it is a good idea to wait until you have opened the merged
+              * index segment, and even until it has begun to serve data, before
+              * deleting the old index segment that was an input to that merge!
+              * The code currently waits until the next time a compacting merge
+              * is performed for the partition and then deletes the DEAD index
+              * segment.
+              */
+ 
+             final MetadataIndex mdi = master.getSlave().getMetadataIndex(name);
+             
+             final PartitionMetadata pmd = mdi.get(fromKey);
+ 
+             // #of live segments for this partition.
+             final int liveCount = pmd.getLiveCount();
+             
+             // @todo assuming compacting merge each time.
+             if(liveCount!=1) throw new UnsupportedOperationException();
+             
+             // new segment definitions.
+             final SegmentMetadata[] newSegs = new SegmentMetadata[2];
+ 
+             // assume only the last segment is live.
+             final SegmentMetadata oldSeg = pmd.segs[pmd.segs.length-1];
+             
+             newSegs[0] = new SegmentMetadata(oldSeg.filename, oldSeg.nbytes,
+                     IndexSegmentLifeCycleEnum.DEAD);
+ 
+             newSegs[1] = new SegmentMetadata(outFile.toString(), outFile.length(),
+                     IndexSegmentLifeCycleEnum.LIVE);
+             
+             mdi.put(fromKey, new PartitionMetadata(0, segId + 1, newSegs));
+             
+             return null;
+             
+         }
+     
+         /**
+          * The resources that comprise the view in reverse timestamp order
+          * (increasing age).
+          */
+         abstract protected IViewMetadata[] getResources();
+         
+     }
+     
+     /**
+      * Task builds an {@link IndexSegment} using a compacting merge of two
+      * resources having data for the same partition. Common use cases are a
+      * journal and an index segment or two index segments. Only the most recent
+      * writes will be retained for any given key (duplicate suppression). Since
+      * this task does NOT provide a full compacting merge (it may be applied to
+      * a subset of the resources required to materialize a view for a commit
+      * time), deletion markers MAY be present in the resulting
+      * {@link IndexSegment}.
+      * 
+      * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+      * @version $Id$
+      */
+     public static class MergeTask extends AbstractMergeTask {
+ 
+         private final IViewMetadata[] resources;
+ 
+         /**
+          * A compacting merge of two or more resources.
+          * 
+          * @param srcs
+          *            The source resources, which must be given in reverse
+          *            timestamp order (increasing age).
+          * @param segId
+          *            The output segment identifier.
+          */
+         public MergeTask(PartitionedJournal master, String name,
+                 int branchingFactor, double errorRate, int partId,
+                 byte[] fromKey, byte[] toKey, IViewMetadata[] resources,
+                 int segId) {
+ 
+             super(master, name, branchingFactor, errorRate, partId, fromKey,
+                     toKey, segId, false);
+             
+             this.resources = resources;
+             
+         }
+ 
+         protected IViewMetadata[] getResources() {
+             
+             return resources;
+             
+         }
+         
+     }
+     
+     /**
+      * Task builds an {@link IndexSegment} using a full compacting merge of all
+      * resources having data for the same partition as of a specific commit
+      * time. Only the most recent writes will be retained for any given key
+      * (duplicate suppression). Deletion markers wil NOT be present in the
+      * resulting {@link IndexSegment}.
+      * <p>
+      * Note: A full compacting merge does not necessarily result in only a
+      * single {@link IndexSegment} for a partition since (a) it may be requested
+      * for a historical commit time; and (b) subsequent writes may have
+      * occurred, resulting in additional data on either the journal and/or new
+      * {@link IndexSegment} that do not participate in the merge.
+      * <p>
+      * Note: in order to perform a full compacting merge the task MUST read from
+      * all resources required to provide a consistent view for a partition
+      * otherwise lost deletes may result.
+      * 
+      * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+      * @version $Id$
+      */
+     public static class FullMergeTask extends AbstractMergeTask {
+ 
+         private final long commitTime;
+ 
+         /**
+          * A full compacting merge of an index partition.
+          * 
+          * @param segId
+          *            The output segment identifier.
+          * 
+          * @param commitTime
+          *            The commit time for the view that is the input to the
+          *            merge operation.
+          */
+         public FullMergeTask(PartitionedJournal master, String name,
+                 int branchingFactor, double errorRate, int partId,
+                 byte[] fromKey, byte[] toKey, long commitTime, int segId) {
+ 
+             super(master, name, branchingFactor, errorRate, partId, fromKey,
+                     toKey, segId, true);
+ 
+             this.commitTime = commitTime;
+ 
+         }
+ 
+         protected IViewMetadata[] getResources() {
+             
+             final PartitionedIndex oldIndex = ((PartitionedIndex) master
+                     .getIndex(name, commitTime));
+             
+             final IViewMetadata[] resources = oldIndex.getResources(fromKey);
+             
+             return resources;
+ 
+         }
+         
+     }
+ 
+ //    /**
+ //     * Task builds an {@link IndexSegment} by joining an existing
+ //     * {@link IndexSegment} for a partition with an existing
+ //     * {@link IndexSegment} for either the left or right sibling of that
+ //     * partition.
+ //     * 
+ //     * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ //     * @version $Id$
+ //     */
+ //    public static class JoinTask implements IPartitionTask {
+ //        
+ //    }
+ //    /**
+ //     * Task splits an {@link IndexSegment} into two new {@link IndexSegment}s.
+ //     * This task is executed when a partition is split in order to breakdown the
+ //     * {@link IndexSegment} for that partition into data for the partition and
+ //     * its new left/right sibling.
+ //     * 
+ //     * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ //     * @version $Id$
+ //     */
+ //    public static class SplitTask implements IPartitionTask {
+ //        
+ //    }
      
  //    /**
***************
*** 1139,1142 ****
--- 1767,1774 ----
      }
  
+     public long size() {
+         return slave.size();
+     }
+     
      public boolean isOpen() {
          return slave.isOpen();
***************
*** 1235,1237 ****
--- 1867,1884 ----
      }
  
+     /**
+      * Return an index view located either on a historical slave journal or an
+      * index segment.
+      * 
+      * @todo instances must be stored in a weak value cache so that we do not
+      *       attempt to re-open the journal (or index segment) if it is already
+      *       open and so that the journal (or index segment) may be closed out
+      *       once it is no longer required by any active view.
+      */
+     protected AbstractBTree getIndex(String name,IViewMetadata resource) {
+         
+         throw new UnsupportedOperationException();
+         
+     }
+     
  }

[cweb-CVS] bigdata/src/test/com/bigdata/scaleup TestMetadataIndex.java, 1.7, 1.8

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/test/com/bigdata/scaleup
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv23960/src/test/com/bigdata/scaleup

Modified Files:
	TestMetadataIndex.java 
Log Message:
Refactoring to introduce asynchronous handling of overflow events in support of a scale-up/scale-out design.

Index: TestMetadataIndex.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/scaleup/TestMetadataIndex.java,v
retrieving revision 1.7
retrieving revision 1.8
diff -C2 -d -r1.7 -r1.8
*** TestMetadataIndex.java	22 Feb 2007 16:59:35 -0000	1.7
--- TestMetadataIndex.java	6 Mar 2007 20:38:06 -0000	1.8
***************
*** 123,127 ****
          
      }
! 
      /**
       * Test the ability to create and update the metadata for a partition.
--- 123,127 ----
          
      }
!     
      /**
       * Test the ability to create and update the metadata for a partition.
***************
*** 592,608 ****
           */
  
-         File outFile02_tmp = new File(getName() + "-part0.02.tmp");
- 
-         outFile02_tmp.deleteOnExit();
- 
          File outFile02 = new File(getName() + "-part0.02.seg");
  
          outFile02.deleteOnExit();
  
-         assertTrue(!outFile02_tmp.exists() || outFile02_tmp.delete());
          assertTrue(!outFile02.exists() || outFile02.delete());
  
!         MergedLeafIterator mergeItr = new IndexSegmentMerger(outFile02_tmp,
!                 100, btree, seg01).merge();
          
          new IndexSegmentBuilder(outFile02, null, mergeItr.nentries,
--- 592,603 ----
           */
  
          File outFile02 = new File(getName() + "-part0.02.seg");
  
          outFile02.deleteOnExit();
  
          assertTrue(!outFile02.exists() || outFile02.delete());
  
!         MergedLeafIterator mergeItr = new IndexSegmentMerger(100, btree, seg01)
!                 .merge();
          
          new IndexSegmentBuilder(outFile02, null, mergeItr.nentries,
***************
*** 855,865 ****
                  // The #of entries that we expected in the segment after the merge.
                  final int expectedEntryCount = groundTruth.getEntryCount();
-                 
-                 // tmp file for the merge process.
-                 File outFile02_tmp = new File(getName() + "-part0."+trial+".tmp");
- 
-                 outFile02_tmp.deleteOnExit();
- 
-                 assertTrue(!outFile02_tmp.exists() || outFile02_tmp.delete());
  
                  // output file for the merged segment.
--- 850,853 ----
***************
*** 870,876 ****
                  assertTrue(!outFile02.exists() || outFile02.delete());
  
!                 // merge the data from the btree on the journal and the index segment.
!                 MergedLeafIterator mergeItr = new IndexSegmentMerger(
!                         outFile02_tmp, mseg2, testData, seg).merge();
  
                  // verify #of entries that are passed on by the merge.
--- 858,865 ----
                  assertTrue(!outFile02.exists() || outFile02.delete());
  
!                 // merge the data from the btree on the journal and the index
!                 // segment.
!                 MergedLeafIterator mergeItr = new IndexSegmentMerger(mseg2,
!                         testData, seg).merge();
  
                  // verify #of entries that are passed on by the merge.

[cweb-CVS] bigdata/src/test/com/bigdata/objndx TestIndexSegmentMerger.java, 1.3, 1.4

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv23960/src/test/com/bigdata/objndx

Modified Files:
	TestIndexSegmentMerger.java 
Log Message:
Refactoring to introduce asynchronous handling of overflow events in support of a scale-up/scale-out design.

Index: TestIndexSegmentMerger.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx/TestIndexSegmentMerger.java,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** TestIndexSegmentMerger.java	7 Feb 2007 13:44:52 -0000	1.3
--- TestIndexSegmentMerger.java	6 Mar 2007 20:38:06 -0000	1.4
***************
*** 54,58 ****
  
  /**
!  * Test suite for compacting merge of index segments (really, of two B+-Trees).
   * 
   * @todo write tests where the keys do not overlap (done).
--- 54,58 ----
  
  /**
!  * Test suite for compacting merge of B+-Trees.
   * 
   * @todo write tests where the keys do not overlap (done).
***************
*** 60,63 ****
--- 60,67 ----
   * @todo write tests where the keys overlap but there are no conflicts.
   * 
+  * @todo write tests: merging a tree with itself, merging trees w/o deletion
+  *       markers, merging trees w/ deletion markers, merging trees w/ age-based
+  *       version expiration, merging trees with count-based version expiration.
+  * 
   * @todo write tests where there are keys that conflict. conflicts need to be
   *       resolved according to some policy, and there are a variety of policies
***************
*** 101,104 ****
--- 105,121 ----
  
      /**
+      * Return the temporary directory.
+      */
+     static synchronized protected File getTempDir() {
+         
+         File tmpDir = new File(System.getProperty("java.io.tmpdir"));
+ 
+         assertTrue(!tmpDir.exists() || tmpDir.mkdirs());
+ 
+         return tmpDir;
+         
+     }
+     
+     /**
       * Test builds two btrees, populates them with disjoint key ranges (1-10
       * and 11-20), and then merges their entries.
***************
*** 124,153 ****
          }
          
-         File outFile = new File( getName()+".merge");
-         
-         if(outFile.exists() && ! outFile.delete() ) {
-             
-             throw new AssertionError("Could not delete test file: "
-                     + outFile.getAbsoluteFile());
-             
-         }
-         
          // branching factor used by leaves emitted by the merge process.
          final int m = 3;
          
!         IndexSegmentMerger merger = new IndexSegmentMerger(outFile, m, btree1,
!                 btree2);
  
-         /*
-          * FIXME nothing is really defined until this operation so maybe it
-          * makes sense to have the "merger" be an iterator backed by a file
-          * whose contents are eagerly generated when the source btrees are large (>
-          * X MB) and buffered in a simple in-memory leaf data structure when
-          * they are small. The only real reason to go to an external file to
-          * buffer the contents is that we may have to merge index segments that
-          * are larger than we would like. Actually, another reason is that we
-          * could produce multiple merged file segments if there was a need to
-          * split the index during the merge operation.
-          */
          MergedLeafIterator itr = merger.merge();
          
--- 141,149 ----
          }
          
          // branching factor used by leaves emitted by the merge process.
          final int m = 3;
          
!         IndexSegmentMerger merger = new IndexSegmentMerger(m, btree1, btree2);
  
          MergedLeafIterator itr = merger.merge();
          
***************
*** 180,193 ****
          assertEquals(20,entryIndex);
          
-         // close the iterator and delete the backing file (if any).
-         itr.close();
- 
-         // @todo consider adding a rewind() method to the iterator.
-         
          /*
!          * @todo build an index segment from the merged entries and validate its
!          * contents.
           */
!         
      }
  
--- 176,185 ----
          assertEquals(20,entryIndex);
          
          /*
!          * Verify that the tmpStore was closed. The backing file (if any) is
!          * deleted when the tmpStore is closed.
           */
!         assertFalse(itr.tmpStore.isOpen());
! 
      }
  
***************
*** 224,241 ****
          assertEquals("btree2.nentries", 10, btree2.getEntryCount());
          
-         File outFile = new File( getName()+".merge");
-         
-         if(outFile.exists() && ! outFile.delete() ) {
-             
-             throw new AssertionError("Could not delete test file: "
-                     + outFile.getAbsoluteFile());
-             
-         }
-         
          // branching factor used by leaves emitted by the merge process.
          final int m = 3;
          
!         IndexSegmentMerger merger = new IndexSegmentMerger(outFile, m, btree1,
!                 btree2);
  
          MergedLeafIterator itr = merger.merge();
--- 216,223 ----
          assertEquals("btree2.nentries", 10, btree2.getEntryCount());
          
          // branching factor used by leaves emitted by the merge process.
          final int m = 3;
          
!         IndexSegmentMerger merger = new IndexSegmentMerger(m, btree1, btree2);
  
          MergedLeafIterator itr = merger.merge();
***************
*** 269,281 ****
          assertEquals(20,entryIndex);
          
-         // close the iterator and delete the backing file (if any).
-         itr.close();
- 
-         // @todo consider adding a rewind() method to the iterator.
-         
          /*
!          * @todo build an index segment from the merged entries and validate its
!          * contents.
           */
          
      }
--- 251,259 ----
          assertEquals(20,entryIndex);
          
          /*
!          * Verify that the tmpStore was closed. The backing file (if any) is
!          * deleted when the tmpStore is closed.
           */
+         assertFalse(itr.tmpStore.isOpen());
          
      }

[cweb-CVS] bigdata/src/java/com/bigdata/isolation Value.java, 1.2, 1.3 IsolatableFusedView.java, 1.1, 1.2

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/isolation
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv23960/src/java/com/bigdata/isolation

Modified Files:
	Value.java IsolatableFusedView.java 
Log Message:
Refactoring to introduce asynchronous handling of overflow events in support of a scale-up/scale-out design.

Index: IsolatableFusedView.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/isolation/IsolatableFusedView.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** IsolatableFusedView.java	13 Feb 2007 23:01:01 -0000	1.1
--- IsolatableFusedView.java	6 Mar 2007 20:38:05 -0000	1.2
***************
*** 50,53 ****
--- 50,54 ----
  import com.bigdata.objndx.AbstractBTree;
  import com.bigdata.objndx.FusedView;
+ import com.bigdata.scaleup.PartitionedJournal.IViewMetadata;
  
  /**

Index: Value.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/isolation/Value.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** Value.java	15 Feb 2007 01:34:22 -0000	1.2
--- Value.java	6 Mar 2007 20:38:05 -0000	1.3
***************
*** 56,60 ****
  
  /**
!  * A non-persistence capable implementation of {@link IObjectIndexEntry}.
   * 
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
--- 56,60 ----
  
  /**
!  * A persistence capable implementation of {@link IValue}.
   * 
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>