cweb-cvs — CVS commit messages are posted to this list.

[cweb-CVS] lgpl-utils/src/java/misc/friedland - New directory

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/lgpl-utils/src/java/misc/friedland
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv14487/src/java/misc/friedland

Log Message:
Directory /cvsroot/cweb/lgpl-utils/src/java/misc/friedland added to the repository

[cweb-CVS] lgpl-utils/src/java/misc/friedland/huffman - New directory

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/lgpl-utils/src/java/misc/friedland/huffman
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv14487/src/java/misc/friedland/huffman

Log Message:
Directory /cvsroot/cweb/lgpl-utils/src/java/misc/friedland/huffman added to the repository

[cweb-CVS] lgpl-utils/src/java/it/unimi/dsi/mg4j/compression - New directory

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/lgpl-utils/src/java/it/unimi/dsi/mg4j/compression
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv14487/src/java/it/unimi/dsi/mg4j/compression

Log Message:
Directory /cvsroot/cweb/lgpl-utils/src/java/it/unimi/dsi/mg4j/compression added to the repository

[cweb-CVS] lgpl-utils/src/java/cern/colt/bitvector - New directory

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/lgpl-utils/src/java/cern/colt/bitvector
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv14487/src/java/cern/colt/bitvector

Log Message:
Directory /cvsroot/cweb/lgpl-utils/src/java/cern/colt/bitvector added to the repository

[cweb-CVS] lgpl-utils/src/test/it/unimi/dsi/mg4j/compression - New directory

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/lgpl-utils/src/test/it/unimi/dsi/mg4j/compression
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv14487/src/test/it/unimi/dsi/mg4j/compression

Log Message:
Directory /cvsroot/cweb/lgpl-utils/src/test/it/unimi/dsi/mg4j/compression added to the repository

[cweb-CVS] lgpl-utils/src/java/cern/colt - New directory

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/lgpl-utils/src/java/cern/colt
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv14487/src/java/cern/colt

Log Message:
Directory /cvsroot/cweb/lgpl-utils/src/java/cern/colt added to the repository

[cweb-CVS] lgpl-utils/src/java/it/unimi/dsi/mg4j/io - New directory

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/lgpl-utils/src/java/it/unimi/dsi/mg4j/io
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv14487/src/java/it/unimi/dsi/mg4j/io

Log Message:
Directory /cvsroot/cweb/lgpl-utils/src/java/it/unimi/dsi/mg4j/io added to the repository

[cweb-CVS] lgpl-utils/src/java/misc - New directory

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/lgpl-utils/src/java/misc
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv14487/src/java/misc

Log Message:
Directory /cvsroot/cweb/lgpl-utils/src/java/misc added to the repository

[cweb-CVS] lgpl-utils/src/java/cern - New directory

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/lgpl-utils/src/java/cern
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv14487/src/java/cern

Log Message:
Directory /cvsroot/cweb/lgpl-utils/src/java/cern added to the repository

[cweb-CVS] lgpl-utils/src/java/cern/colt/function - New directory

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/lgpl-utils/src/java/cern/colt/function
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv14487/src/java/cern/colt/function

Log Message:
Directory /cvsroot/cweb/lgpl-utils/src/java/cern/colt/function added to the repository

[cweb-CVS] junit-ext project.properties,1.3,1.4

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/junit-ext
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv26284

Modified Files:
	project.properties 
Log Message:
Removed an unused maven repository from the project.properties file.

Index: project.properties
===================================================================
RCS file: /cvsroot/cweb/junit-ext/project.properties,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** project.properties	20 Mar 2007 13:52:31 -0000	1.3
--- project.properties	26 Apr 2007 11:59:03 -0000	1.4
***************
*** 7,11 ****
  # authoritative source.
  
! maven.repo.remote=http://www.ibiblio.org/maven/,http://www.cognitiveweb.org/maven-repository,http://proto.cognitiveweb.org/maven-repository
  
  # Links to external packages.  The package-list javadoc file for
--- 7,11 ----
  # authoritative source.
  
! maven.repo.remote=http://www.ibiblio.org/maven/,http://proto.cognitiveweb.org/maven-repository
  
  # Links to external packages.  The package-list javadoc file for

[cweb-CVS] bigdata/src/java/com/bigdata/scaleup MetadataIndex.java, 1.11, 1.12

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv11220/src/java/com/bigdata/scaleup

Modified Files:
	MetadataIndex.java 
Log Message:
Implemented range count for partitioned indices.

Index: MetadataIndex.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/MetadataIndex.java,v
retrieving revision 1.11
retrieving revision 1.12
diff -C2 -d -r1.11 -r1.12
*** MetadataIndex.java	25 Apr 2007 14:57:11 -0000	1.11
--- MetadataIndex.java	25 Apr 2007 16:05:37 -0000	1.12
***************
*** 327,335 ****
       * @return The partition spanning the given key or <code>null</code> if
       *         there are no partitions defined.
-      * 
-      * FIXME offer a variant that reports the index partitions spanned by a key
-      * range and write tests for that. Note that the remote API for that method
-      * should use a result-set data model to efficiently communicate the data
-      * when there are a large #of spanned partitions.
       */
      public PartitionMetadata find(byte[] key) {
--- 327,330 ----
***************
*** 339,343 ****
          if(index == -1) return null;
          
!         byte[] val = (byte[]) super.valueAt(index);
          
          return (PartitionMetadata) SerializerUtil.deserialize(val);
--- 334,338 ----
          if(index == -1) return null;
          
!         byte[] val = (byte[]) valueAt(index);
          
          return (PartitionMetadata) SerializerUtil.deserialize(val);
***************
*** 357,361 ****
      public PartitionMetadata get(byte[] key) {
          
!         byte[] val = (byte[]) super.lookup(key);
          
          if(val==null) return null;
--- 352,356 ----
      public PartitionMetadata get(byte[] key) {
          
!         byte[] val = (byte[]) lookup(key);
          
          if(val==null) return null;
***************
*** 392,396 ****
          byte[] newval = SerializerUtil.serialize(val);
          
!         byte[] oldval2 = (byte[])super.insert(key, newval);
  
          PartitionMetadata oldval = oldval2 == null ? null
--- 387,391 ----
          byte[] newval = SerializerUtil.serialize(val);
          
!         byte[] oldval2 = (byte[])insert(key, newval);
  
          PartitionMetadata oldval = oldval2 == null ? null

[cweb-CVS] bigdata/src/java/com/bigdata/service MetadataService.java, 1.9, 1.10 ClientIndexView.java, 1.3, 1.4 BigdataClient.java, 1.2, 1.3 IMetadataService.java, 1.7, 1.8

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/service
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv11220/src/java/com/bigdata/service

Modified Files:
	MetadataService.java ClientIndexView.java BigdataClient.java 
	IMetadataService.java 
Log Message:
Implemented range count for partitioned indices.

Index: BigdataClient.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/BigdataClient.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** BigdataClient.java	25 Apr 2007 14:57:11 -0000	1.2
--- BigdataClient.java	25 Apr 2007 16:05:37 -0000	1.3
***************
*** 51,55 ****
  import java.rmi.RemoteException;
  import java.util.UUID;
- import java.util.concurrent.ExecutionException;
  
  import net.jini.config.Configuration;
--- 51,54 ----
***************
*** 106,109 ****
--- 105,113 ----
   * transaction identifier for its read and write operations.
   * 
+  * @todo write tests where an index is static partitioned over multiple data
+  *       services and verify that the {@link ClientIndexView} is consistent.
+  *       <p>
+  *       Work towards the same guarentee when dynamic partitioning is enabled.
+  * 
   * @todo support transactions (there is no transaction manager service yet and
   *       the 2-/3-phase commit protocol has not been implemented on the
***************
*** 112,117 ****
   * @todo Write or refactor logic to map operations across multiple partitions.
   * 
!  * FIXME Support factory for indices, reuse cached information across
!  * transactional and non-transactional views of the same index.
   * 
   * @todo Use lambda expressions (and downloaded code) for server-side logic for
--- 116,121 ----
   * @todo Write or refactor logic to map operations across multiple partitions.
   * 
!  * @todo reuse cached information across transactional and non-transactional
!  *       views of the same index.
   * 
   * @todo Use lambda expressions (and downloaded code) for server-side logic for
***************
*** 598,625 ****
  
          /**
!          * @todo setup cache.  test cache and lookup on metadata service if a 
!          * cache miss.
           * 
           * @param name
           * @param key
           * @return
           */
!         public IPartitionMetadata getPartition(String name, byte[] key) {
!         
! //            synchronized(indexCache) {
! //            
! //                Map<Integer,IDataService> partitionCache = indexCache.get(name);
! //             
! //                if(partitionCache==null) {
! //                    
! //                    partitionCache = new ConcurrentHashMap<Integer, IDataService>();
! //                    
! //                    indexCache.put(name, partitionCache);
! //                    
! //                }
! //                
! //                IDataService dataService = 
! //                
! //            }
  
              /*
--- 602,649 ----
  
          /**
!          * @todo setup cache. test cache and lookup on metadata service if a
!          *       cache miss. the cache should be based on a lease and the data
!          *       service should know whether an index partition has been moved
!          *       and notify the client that it needs to re-discover the data
!          *       service for an index partition. the cache is basically a
!          *       partial copy of the metadata index that is local to the client.
!          *       The cache needs to have "fake" entries that are the
!          *       left-sibling of each real partition entry so that it can
!          *       correctly determine when there is a cache miss.
!          *       <p>
!          *       Note that index partition definitions will evolve slowly over
!          *       time through splits and joins of index segments. again, the
!          *       client should presume consistency of its information but the
!          *       data service should know when it no longer has information for
!          *       a key range in a partition (perhaps passing the partitionId and
!          *       a timestamp for the last partition update to the data service
!          *       with each request).
!          *       <p>
!          *       Provide a historical view of the index partition definitions
!          *       when transactional isolation is in use by the client. This
!          *       should make it possible for a client to not be perturbed by
!          *       split/joins of index partitions when executing with
!          *       transactional isolation.
!          *       <p>
!          *       Note that service failover is at least partly orthogonal to the
!          *       partition metadata in as much as the index partition
!          *       definitions themselves do not evolve (the same separator keys
!          *       are in place and the same resources have the consistent data
!          *       for a view of the index partition), but it is possible that the
!          *       data services have changed. It is an open question how to
!          *       maintain isolation with failover while supporting failover
!          *       without aborting the transaction. The most obvious thing is to
!          *       have a transationally isolated client persue the failover
!          *       services already defined in the historical transaction without
!          *       causing the partition metadata to be updated on the metadata
!          *       service. (Unisolated clients would begin to see updated
!          *       partition metadata more or immediately.)
           * 
+          * @param tx
           * @param name
           * @param key
           * @return
           */
!         public IPartitionMetadata getPartition(long tx, String name, byte[] key) {
  
              /*
***************
*** 632,636 ****
              try {
               
!                 pmd = getMetadataService().getPartition(name, key);
                  
              } catch(Exception ex) {
--- 656,664 ----
              try {
               
!                 byte[] val = getMetadataService().getPartition(name, key);
!                 
!                 if(val ==null) return null;
!                 
!                 pmd = (IPartitionMetadata) SerializerUtil.deserialize(val);
                  
              } catch(Exception ex) {
***************
*** 645,648 ****
--- 673,692 ----
  //        
  //        private Map<String, Map<Integer, IDataService>> indexCache = new ConcurrentHashMap<String, Map<Integer, IDataService>>(); 
+ //
+ //        synchronized(indexCache) {
+ //      
+ //          Map<Integer,IDataService> partitionCache = indexCache.get(name);
+ //       
+ //          if(partitionCache==null) {
+ //              
+ //              partitionCache = new ConcurrentHashMap<Integer, IDataService>();
+ //              
+ //              indexCache.put(name, partitionCache);
+ //              
+ //          }
+ //          
+ //          IDataService dataService = 
+ //          
+ //      }
  
          /**

Index: ClientIndexView.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/ClientIndexView.java,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** ClientIndexView.java	25 Apr 2007 14:57:10 -0000	1.3
--- ClientIndexView.java	25 Apr 2007 16:05:37 -0000	1.4
***************
*** 57,61 ****
--- 57,63 ----
  import com.bigdata.btree.IEntryIterator;
  import com.bigdata.btree.IIndex;
+ import com.bigdata.io.SerializerUtil;
  import com.bigdata.scaleup.IPartitionMetadata;
+ import com.bigdata.scaleup.MetadataIndex;
  import com.bigdata.scaleup.PartitionedIndexView;
  import com.bigdata.service.BigdataClient.BigdataFederation;
***************
*** 65,68 ****
--- 67,74 ----
   * A client-side view of an index.
   * 
+  * @todo consider writing a client interface to the {@link MetadataIndex} so
+  *       that this code can look identifical to the code that we would write if
+  *       the metdata index was local.
+  * 
   * @todo cache leased information about index partitions of interest to the
   *       client. The cache will be a little tricky since we need to know when
***************
*** 175,179 ****
      public boolean contains(byte[] key) {
          
!         IPartitionMetadata pmd = fed.getPartition(name, key);
  
          IDataService dataService = fed.getDataService(pmd);
--- 181,185 ----
      public boolean contains(byte[] key) {
          
!         IPartitionMetadata pmd = fed.getPartition(tx,name, key);
  
          IDataService dataService = fed.getDataService(pmd);
***************
*** 197,201 ****
      public Object insert(Object key, Object value) {
  
!         IPartitionMetadata pmd = fed.getPartition(name, (byte[])key);
          
          IDataService dataService = fed.getDataService(pmd);
--- 203,207 ----
      public Object insert(Object key, Object value) {
  
!         IPartitionMetadata pmd = fed.getPartition(tx,name, (byte[])key);
          
          IDataService dataService = fed.getDataService(pmd);
***************
*** 223,227 ****
      public Object lookup(Object key) {
  
!         IPartitionMetadata pmd = fed.getPartition(name, (byte[])key);
  
          IDataService dataService = fed.getDataService(pmd);
--- 229,233 ----
      public Object lookup(Object key) {
  
!         IPartitionMetadata pmd = fed.getPartition(tx,name, (byte[])key);
  
          IDataService dataService = fed.getDataService(pmd);
***************
*** 246,250 ****
      public Object remove(Object key) {
  
!         IPartitionMetadata pmd = fed.getPartition(name, (byte[])key);
  
          IDataService dataService = fed.getDataService(pmd);
--- 252,256 ----
      public Object remove(Object key) {
  
!         IPartitionMetadata pmd = fed.getPartition(tx,name, (byte[])key);
  
          IDataService dataService = fed.getDataService(pmd);
***************
*** 286,319 ****
      public int rangeCount(byte[] fromKey, byte[] toKey) {
  
!         IPartitionMetadata pmd1 = fed.getPartition(name, (byte[])fromKey);
!         
!         IPartitionMetadata pmd2 = fed.getPartition(name, (byte[])toKey);
          
!         if(pmd2.getPartitionId()!=pmd1.getPartitionId()) {
              
!             throw new UnsupportedOperationException(
!                     "Can not span partitions at this time");
              
          }
-         
-         IDataService dataService = fed.getDataService(pmd1);
  
!         int rangeCount = 0;
!         
!         try {
  
!             rangeCount += dataService.rangeCount(IDataService.UNISOLATED, name,
!                     fromKey, toKey);
!         
!         } catch(Exception ex) {
  
!             throw new RuntimeException(ex);
              
          }
!         
!         return rangeCount;
          
      }
  
      public IEntryIterator rangeIterator(byte[] fromKey, byte[] toKey) {
          // TODO Auto-generated method stub
--- 292,388 ----
      public int rangeCount(byte[] fromKey, byte[] toKey) {
  
!         IMetadataService metadataService = getMetadataService();
          
!         final int fromIndex;
!         final int toIndex;
!         try {
! 
!             // index of the first partition to check.
!             fromIndex = (fromKey == null ? 0 : metadataService
!                     .findIndexOfPartition(name, fromKey));
              
!             // index of the last partition to check.
!             toIndex = (toKey == null ? 0 : metadataService
!                     .findIndexOfPartition(name, toKey));
!             
!         } catch (IOException ex) {
!             
!             throw new RuntimeException(ex);
              
          }
  
!         // per javadoc, keys out of order returns zero(0).
!         if (toIndex < fromIndex)
!             return 0;
  
!         // use to counters so that we can look for overflow.
!         int count = 0;
!         int lastCount = 0;
  
!         for (int index = fromIndex; index <= toIndex; index++) {
! 
!             IPartitionMetadata pmd;
! 
!             try {
! 
!                 byte[] tmp = metadataService.getPartitionAtIndex(name,
!                         fromIndex);
! 
!                 if (tmp == null)
!                     throw new AssertionError();
! 
!                 pmd = (IPartitionMetadata) SerializerUtil.deserialize(tmp);
! 
!             } catch(IOException ex) {
!                 
!                 throw new RuntimeException(ex);
!                 
!             }
! 
!             // // The first key that would enter the nth partition.
!             // byte[] separatorKey = mdi.keyAt(index);
! 
!             IDataService dataService = fed.getDataService(pmd);
! 
!             try {
! 
!                 /*
!                  * Add in the count from that partition.
!                  * 
!                  * @todo modify to request only the range count that actually
!                  * lies within the partition so that the data service can check
!                  * the range and notify clients that appear to be requesting
!                  * data for index partitions that have been relocated.
!                  */
! 
!                 count += dataService.rangeCount(tx, name, fromKey, toKey);
!                 
!             } catch(Exception ex) {
! 
!                 throw new RuntimeException(ex);
!                 
!             }
! 
!             if(count<lastCount) {
!             
!                 // more than would fit in an Integer.
!                 return Integer.MAX_VALUE;
!                 
!             }
!             
!             lastCount = count;
              
          }
! 
!         return count;
          
      }
  
+     /**
+      * FIXME provide an {@link IEntryIterator} that kinds the use of a series of
+      * {@link ResultSet}s to produce the range iterator. We need an outer loop
+      * over the partitions spanned by the key range and then an inner loop until
+      * we have exhausted the key range overlapping with each partition.
+      */
      public IEntryIterator rangeIterator(byte[] fromKey, byte[] toKey) {
          // TODO Auto-generated method stub

Index: IMetadataService.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/IMetadataService.java,v
retrieving revision 1.7
retrieving revision 1.8
diff -C2 -d -r1.7 -r1.8
*** IMetadataService.java	25 Apr 2007 14:57:10 -0000	1.7
--- IMetadataService.java	25 Apr 2007 16:05:37 -0000	1.8
***************
*** 73,118 ****
   */
  public interface IMetadataService extends IDataService {
!     
! //    /**
! //     * Return the partition metadata for the index partition that includes the
! //     * specified key.
! //     * 
! //     * @param key
! //     *            The key.
! //     * 
! //     * @return The metadata index partition in which that key is or would be
! //     *         located.
! //     * 
! //     * @todo return lease for the index partition that would contain the key.
! //     * 
! //     * @todo abstract away from Jini so that we can support other fabrics
! //     *       (OSGi/SCA).
! //     * 
! //     * @todo Either the client or the metadata service should support
! //     *       pre-caching of some number of index partitions surrounding that
! //     *       partition.
! //     * 
! //     * @todo do a variant that supports a key range - this should really just be
! //     *       the same as
! //     *       {@link IDataService#rangeQuery(long, String, byte[], byte[], int, int)}
! //     *       with the client addressing the metadata index rather than the data
! //     *       index (likewise for this method as well).
! //     * 
! //     * @todo update the {@link PartitionMetadata} data model to reflect a single
! //     *       point of responsibility with a media replication chain for
! //     *       failover. Either this method or a variant method needs to return
! //     *       the partition metadata itself so that {@link DataService}s can
! //     *       configure their downstream media replication pipelines.
! //     */
! //    public PartitionMetadata getPartition(String name, byte[] key) throws IOException;
! //
!     
      /*
       * methods that require access to the metadata server for their
       * implementations.
       */
      
      /**
!      * Return the UUID of an under utilized data service.
       */
      public ServiceID getUnderUtilizedDataService() throws IOException;
--- 73,90 ----
   */
  public interface IMetadataService extends IDataService {
!         
      /*
       * methods that require access to the metadata server for their
       * implementations.
+      * 
+      * @todo the tx identifier will have to be pass in for clients that want to
+      * use transactional isolation to achieve a consistent and stable view of
+      * the metadata index as of the start time of their transaction.
       */
      
      /**
!      * Return the identifier of an under utilized data service.
!      * 
!      * @todo convert to return a UUID to kind Jini isolated from the core impl.
       */
      public ServiceID getUnderUtilizedDataService() throws IOException;
***************
*** 129,132 ****
--- 101,106 ----
       *         
       * @throws IOException
+      * 
+      * @todo convert serviceID to a UUID to keep Jini encapsulated.
       */
      public IDataService getDataServiceByID(ServiceID serviceID)
***************
*** 173,190 ****
       * @param key
       *            The key.
-      * @return The metadata for the index partition in which that key would be
-      *         found.
-      *         
-      * @throws IOException
       * 
!      * FIXME offer a variant that reports the index partitions spanned by a key
!      * range and write tests for that. Note that the remote API for that method
!      * should use a result-set data model to efficiently communicate the data
!      * when there are a large #of spanned partitions.
       * 
       * @see MetadataIndex#find(byte[])
       */
!     public IPartitionMetadata getPartition(String name, byte[] key)
              throws IOException;
      
  }
--- 147,196 ----
       * @param key
       *            The key.
       * 
!      * @return The serialized {@link IPartitionMetadata} spanning the given key
!      *         or <code>null</code> if there are no partitions defined.
!      * 
!      * @throws IOException
       * 
       * @see MetadataIndex#find(byte[])
       */
!     public byte[] getPartition(String name, byte[] key)
              throws IOException;
      
+     /**
+      * Find the index of the partition spanning the given key.
+      * 
+      * @return The index of the partition spanning the given key or
+      *         <code>-1</code> iff there are no partitions defined.
+      * 
+      * @exception IllegalStateException
+      *                if there are partitions defined but no partition spans the
+      *                key. In this case the {@link MetadataIndex} lacks an entry
+      *                for the key <code>new byte[]{}</code>.
+      */
+     public int findIndexOfPartition(String name,byte[] key) throws IOException;
+     
+     /**
+      * The partition at that index.
+      * 
+      * @param name
+      *            The name of the scale-out index.
+      * @param index
+      *            The entry index in the metadata index.
+      * 
+      * @return The serialized {@link IPartitionMetadata} for that the entry with
+      *         that index.
+      * 
+      * @throws IOException
+      * 
+      * @todo this is subject to concurrent modification of the metadata index
+      *       would can cause the index to identify a different partition. client
+      *       requests that use {@link #findIndexOfPartition(String, byte[])} and
+      *       {@link #getPartitionAtIndex(String, int)} really need to refer to
+      *       the same historical version of the metadata index (this effects
+      *       range count and range iterator requests and to some extent batch
+      *       operations that span multiple index partitions).
+      */
+     public byte[] getPartitionAtIndex(String name, int index ) throws IOException;
+     
  }

Index: MetadataService.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/MetadataService.java,v
retrieving revision 1.9
retrieving revision 1.10
diff -C2 -d -r1.9 -r1.10
*** MetadataService.java	25 Apr 2007 14:57:11 -0000	1.9
--- MetadataService.java	25 Apr 2007 16:05:37 -0000	1.10
***************
*** 73,76 ****
--- 73,80 ----
   *       later.
   * 
+  * @todo support transactionally isolated views onto the metadata index by
+  *       passing in the tx identifier and using the appropriate historical view
+  *       of the metadata index.
+  * 
   * @todo Provide a means to reconstruct the metadata index from the journal and
   *       index segment data files. We tag each journal and index segment with a
***************
*** 155,159 ****
      }
  
!     public IPartitionMetadata getPartition(String name,byte[] key) throws IOException {
          
          // the name of the metadata index itself.
--- 159,168 ----
      }
  
! 
!     /**
!      * Note: This is equivilent to {@link MetadataIndex#find(byte[])} except that
!      * it does not deserialize the {@link IPartitionMetadata}.
!      */
!     public byte[] getPartition(String name,byte[] key) throws IOException {
          
          // the name of the metadata index itself.
***************
*** 169,185 ****
          }
  
!         /*
!          * @todo this winds up deserializing the value into a PartitionMetadata
!          * object and then re-serializing it to return to the remote client.
!          */
!         IPartitionMetadata pmd = mdi.find(key);
          
!         if( pmd == null ) {
              
!             throw new IllegalStateException("No partitioned in index: "+name);
              
          }
          
!         return pmd;
          
      }
--- 178,233 ----
          }
  
!         final int index = mdi.findIndexOf(key);
          
!         if(index == -1) return null;
!         
!         byte[] val = (byte[]) mdi.valueAt(index);
!         
!         return val;
!         
!     }
! 
!     /**
!      * This is equivilent to {@link MetadataIndex#findIndexOf(byte[])}.
!      */
!     public int findIndexOfPartition(String name,byte[] key) throws IOException {
!         
!         // the name of the metadata index itself.
!         final String metadataName = getMetadataName(name);
!         
!         // make sure there is no metadata index for that btree.
!         MetadataIndex mdi = (MetadataIndex) journal.getIndex(metadataName);
!         
!         if(mdi == null) {
              
!             throw new IllegalArgumentException("Index not registered: " + name);
              
          }
+ 
+         final int index = mdi.findIndexOf(key);
+ 
+         return index;
          
!     }
!     
!     public byte[] getPartitionAtIndex(String name, int index ) throws IOException {
!         
!         // the name of the metadata index itself.
!         final String metadataName = getMetadataName(name);
!         
!         // make sure there is no metadata index for that btree.
!         MetadataIndex mdi = (MetadataIndex) journal.getIndex(metadataName);
!         
!         if(mdi == null) {
!             
!             throw new IllegalArgumentException("Index not registered: " + name);
!             
!         }
! 
!         if(index == -1) return null;
!         
!         byte[] val = (byte[]) mdi.valueAt(index);
!         
!         return val;
          
      }

[cweb-CVS] bigdata/src/test/com/bigdata/service TestBigdataClient.java, 1.2, 1.3 TestMetadataServer0.java, 1.5, 1.6

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/test/com/bigdata/service
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv11220/src/test/com/bigdata/service

Modified Files:
	TestBigdataClient.java TestMetadataServer0.java 
Log Message:
Implemented range count for partitioned indices.

Index: TestBigdataClient.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/service/TestBigdataClient.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** TestBigdataClient.java	25 Apr 2007 14:57:10 -0000	1.2
--- TestBigdataClient.java	25 Apr 2007 16:05:37 -0000	1.3
***************
*** 220,227 ****
  
          // verify some range counts.
-         // @todo test with rangeCount(null,null).
          assertEquals(0,ndx.rangeCount(new byte[]{}, new byte[]{1}));
          assertEquals(1,ndx.rangeCount(new byte[]{}, new byte[]{2}));
          assertEquals(1,ndx.rangeCount(new byte[]{1}, new byte[]{2}));
          
          // remove the index entry.
--- 220,230 ----
  
          // verify some range counts.
          assertEquals(0,ndx.rangeCount(new byte[]{}, new byte[]{1}));
          assertEquals(1,ndx.rangeCount(new byte[]{}, new byte[]{2}));
          assertEquals(1,ndx.rangeCount(new byte[]{1}, new byte[]{2}));
+         assertEquals(0,ndx.rangeCount(null, new byte[]{1}));
+         assertEquals(1,ndx.rangeCount(new byte[]{1},null));
+         assertEquals(1,ndx.rangeCount(null,new byte[]{2}));
+         assertEquals(1,ndx.rangeCount(null,null));
          
          // remove the index entry.
***************
*** 241,250 ****
           * relevant index partition(s) need to be compacted for those deletion
           * markers to be removed and the range counts adjusted to match.
-          * 
-          * @todo test with rangeCount(null,null).
           */
          assertEquals(0,ndx.rangeCount(new byte[]{}, new byte[]{1}));
          assertEquals(1,ndx.rangeCount(new byte[]{}, new byte[]{2}));
          assertEquals(1,ndx.rangeCount(new byte[]{1}, new byte[]{2}));
  
      }
--- 244,255 ----
           * relevant index partition(s) need to be compacted for those deletion
           * markers to be removed and the range counts adjusted to match.
           */
          assertEquals(0,ndx.rangeCount(new byte[]{}, new byte[]{1}));
          assertEquals(1,ndx.rangeCount(new byte[]{}, new byte[]{2}));
          assertEquals(1,ndx.rangeCount(new byte[]{1}, new byte[]{2}));
+         assertEquals(0,ndx.rangeCount(null, new byte[]{1}));
+         assertEquals(1,ndx.rangeCount(new byte[]{1},null));
+         assertEquals(1,ndx.rangeCount(null,new byte[]{2}));
+         assertEquals(1,ndx.rangeCount(null,null));
  
      }

Index: TestMetadataServer0.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/service/TestMetadataServer0.java,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** TestMetadataServer0.java	25 Apr 2007 14:57:10 -0000	1.5
--- TestMetadataServer0.java	25 Apr 2007 16:05:37 -0000	1.6
***************
*** 335,343 ****
           */
  
!         IPartitionMetadata pmd = metadataServiceProxy.getPartition(indexName,
!                 new byte[] {});
! 
!         assertNotNull(pmd);
          
          /*
           * Resolve the data service to which the initial index partition was
--- 335,351 ----
           */
  
!         IPartitionMetadata pmd;
          
+         {
+ 
+             byte[] val = metadataServiceProxy.getPartition(indexName,
+                     new byte[] {});
+ 
+             assertNotNull(val);
+ 
+             pmd = (IPartitionMetadata) SerializerUtil.deserialize(val);
+             
+         }
+ 
          /*
           * Resolve the data service to which the initial index partition was

[cweb-CVS] bigdata/src/java/com/bigdata/scaleup MasterJournal.java, 1.9, 1.10 MetadataIndex.java, 1.10, 1.11

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv16408/src/java/com/bigdata/scaleup

Modified Files:
	MasterJournal.java MetadataIndex.java 
Log Message:
Progress on the BigdataClient.

Index: MasterJournal.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/MasterJournal.java,v
retrieving revision 1.9
retrieving revision 1.10
diff -C2 -d -r1.9 -r1.10
*** MasterJournal.java	23 Apr 2007 18:58:37 -0000	1.9
--- MasterJournal.java	25 Apr 2007 14:57:11 -0000	1.10
***************
*** 1177,1180 ****
--- 1177,1187 ----
          return slave.getFile();
      }
+ 
+     /**
+      * Shutdown the journal politely.
+      */
+     public void shutdown() {
+         slave.shutdown();
+     }
      
      public void close() {

Index: MetadataIndex.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/MetadataIndex.java,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** MetadataIndex.java	23 Apr 2007 18:58:37 -0000	1.10
--- MetadataIndex.java	25 Apr 2007 14:57:11 -0000	1.11
***************
*** 327,330 ****
--- 327,335 ----
       * @return The partition spanning the given key or <code>null</code> if
       *         there are no partitions defined.
+      * 
+      * FIXME offer a variant that reports the index partitions spanned by a key
+      * range and write tests for that. Note that the remote API for that method
+      * should use a result-set data model to efficiently communicate the data
+      * when there are a large #of spanned partitions.
       */
      public PartitionMetadata find(byte[] key) {

[cweb-CVS] bigdata/src/java/com/bigdata/service DataServiceMap.java, NONE, 1.1 AbstractServer.java, 1.8, 1.9 DataServer.java, 1.7, 1.8 ClientIndexView.java, 1.2, 1.3 IMetadataService.java, 1.6, 1.7 JiniUtil.java, 1.1, 1.2 IDataService.java, 1.11, 1.12 MetadataService.java, 1.8, 1.9 DataService.java, 1.13, 1.14 MetadataServer.java, 1.6, 1.7 BigdataClient.java, 1.1, 1.2 EmbeddedDataService.java, 1.7, NONE DataServiceClient.java, 1.8, NONE

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/service
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv16408/src/java/com/bigdata/service

Modified Files:
	AbstractServer.java DataServer.java ClientIndexView.java 
	IMetadataService.java JiniUtil.java IDataService.java 
	MetadataService.java DataService.java MetadataServer.java 
	BigdataClient.java 
Added Files:
	DataServiceMap.java 
Removed Files:
	EmbeddedDataService.java DataServiceClient.java 
Log Message:
Progress on the BigdataClient.

Index: DataServer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/DataServer.java,v
retrieving revision 1.7
retrieving revision 1.8
diff -C2 -d -r1.7 -r1.8
*** DataServer.java	23 Apr 2007 13:09:29 -0000	1.7
--- DataServer.java	25 Apr 2007 14:57:10 -0000	1.8
***************
*** 51,54 ****
--- 51,55 ----
  import java.rmi.RemoteException;
  import java.util.Properties;
+ import java.util.UUID;
  
  import net.jini.config.Configuration;
***************
*** 160,163 ****
--- 161,165 ----
          
          protected AbstractServer server;
+         private UUID serviceUUID;
          
          public AdministrableDataService(AbstractServer server,Properties properties) {
***************
*** 205,208 ****
--- 207,222 ----
          }
  
+         protected UUID getDataServiceUUID() {
+ 
+             if(serviceUUID==null) {
+ 
+                 serviceUUID = JiniUtil.serviceID2UUID(server.getServiceID());
+                 
+             }
+             
+             return serviceUUID;
+             
+         }
+ 
  // /*
  // * JoinAdmin

--- NEW FILE: DataServiceMap.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Apr 25, 2007
 */

package com.bigdata.service;

import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

import org.apache.log4j.Logger;

import net.jini.core.lookup.ServiceID;
import net.jini.core.lookup.ServiceItem;
import net.jini.lookup.LookupCache;
import net.jini.lookup.ServiceDiscoveryEvent;
import net.jini.lookup.ServiceDiscoveryListener;
import net.jini.lookup.ServiceDiscoveryManager;

/**
 * A mapping from {@link ServiceID} to {@link ServiceItem} that is maintained by
 * a suitable {@link ServiceDiscoveryManager} manager.  In order to use this
 * class, you must register it as the {@link ServiceDiscoveryListener} with the
 * {@link ServiceDiscoveryManager}.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public class DataServiceMap implements ServiceDiscoveryListener {

    public static final transient Logger log = Logger
            .getLogger(DataServiceMap.class);
    
    private Map<ServiceID, ServiceItem> serviceIdMap = new ConcurrentHashMap<ServiceID, ServiceItem>();

    public DataServiceMap() {
        
    }

    /*
     * ServiceDiscoveryListener.
     */

    /**
     * Adds the {@link ServiceItem} to the internal map to support
     * {@link #getServiceByID()}
     * <p>
     * Note: This event is generated by the {@link LookupCache}. There is an
     * event for each {@link DataService} as it joins any registrar in the set
     * of registrars to which the {@link MetadataServer} is listening. The set
     * of distinct joined {@link DataService}s is accessible via the
     * {@link LookupCache}.
     */
    public void serviceAdded(ServiceDiscoveryEvent e) {
        
        log.info("" + e + ", class="
                + e.getPostEventServiceItem().toString());
        
        serviceIdMap.put(e.getPostEventServiceItem().serviceID, e
                .getPostEventServiceItem());
        
    }

    /**
     * NOP.
     */
    public void serviceChanged(ServiceDiscoveryEvent e) {

        log.info(""+e+", class="
                + e.getPostEventServiceItem().toString());
        
        serviceIdMap.put(e.getPostEventServiceItem().serviceID, e
                .getPostEventServiceItem());

    }

    /**
     * NOP.
     */
    public void serviceRemoved(ServiceDiscoveryEvent e) {

        log.info(""+e+", class="
                + e.getPreEventServiceItem().toString());

        serviceIdMap.remove(e.getPreEventServiceItem().serviceID);

    }

    /*
     * Our own API.
     */
    
    /**
     * Resolve the {@link ServiceID} for a {@link DataService} to the cached
     * {@link ServiceItem} for that {@link DataService}.
     * 
     * @param serviceID
     *            The {@link ServiceID} for the {@link DataService}.
     * 
     * @return The cache {@link ServiceItem} for that {@link DataService}.
     */
    public ServiceItem getDataServiceByID(ServiceID serviceID) {
        
        return serviceIdMap.get(serviceID);
        
    }
    
    /**
     * Return the #of {@link DataService}s known to this {@link MetadataServer}.
     * 
     * @return The #of {@link DataService}s in the {@link LookupCache}.
     */
    public int getDataServiceCount() {
        
        return serviceIdMap.size();
        
    }
    
}

Index: IDataService.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/IDataService.java,v
retrieving revision 1.11
retrieving revision 1.12
diff -C2 -d -r1.11 -r1.12
*** IDataService.java	23 Apr 2007 17:22:14 -0000	1.11
--- IDataService.java	25 Apr 2007 14:57:11 -0000	1.12
***************
*** 147,150 ****
--- 147,164 ----
  
      /**
+      * Return the unique index identifier for the named index (synchronous,
+      * unisolated).
+      * 
+      * @param name
+      *            The index name.
+      *            
+      * @return The index UUID -or- <code>null</code> if the index is not
+      *         registered on this {@link IDataService}.
+      *         
+      * @throws IOException
+      */
+     public UUID getIndexUUID(String name) throws IOException;
+ 
+     /**
       * Drops the named index (unisolated).
       * 
***************
*** 229,235 ****
       * @todo support extension operations (read or mutable).
       */
!     public void batchInsert(long tx, String name, int ntuples, byte[][] keys,
!             byte[][] values) throws InterruptedException, ExecutionException,
!             IOException;
  
      public boolean[] batchContains(long tx, String name, int ntuples,
--- 243,249 ----
       * @todo support extension operations (read or mutable).
       */
!     public byte[][] batchInsert(long tx, String name, int ntuples,
!             byte[][] keys, byte[][] values, boolean returnOldValues)
!             throws InterruptedException, ExecutionException, IOException;
  
      public boolean[] batchContains(long tx, String name, int ntuples,

Index: MetadataServer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/MetadataServer.java,v
retrieving revision 1.6
retrieving revision 1.7
diff -C2 -d -r1.6 -r1.7
*** MetadataServer.java	23 Apr 2007 23:44:42 -0000	1.6
--- MetadataServer.java	25 Apr 2007 14:57:11 -0000	1.7
***************
*** 51,57 ****
  import java.rmi.Remote;
  import java.rmi.RemoteException;
- import java.util.Map;
  import java.util.Properties;
! import java.util.concurrent.ConcurrentHashMap;
  
  import net.jini.core.lookup.ServiceID;
--- 51,56 ----
  import java.rmi.Remote;
  import java.rmi.RemoteException;
  import java.util.Properties;
! import java.util.UUID;
  
  import net.jini.core.lookup.ServiceID;
***************
*** 60,65 ****
  import net.jini.lease.LeaseRenewalManager;
  import net.jini.lookup.LookupCache;
- import net.jini.lookup.ServiceDiscoveryEvent;
- import net.jini.lookup.ServiceDiscoveryListener;
  import net.jini.lookup.ServiceDiscoveryManager;
  import net.jini.lookup.ServiceItemFilter;
--- 59,62 ----
***************
*** 91,99 ****
   * @version $Id$
   */
! public class MetadataServer extends DataServer implements ServiceDiscoveryListener {
  
      private ServiceDiscoveryManager serviceDiscoveryManager = null;
      private LookupCache dataServiceLookupCache = null;
!     private Map<ServiceID, ServiceItem> serviceIdMap = new ConcurrentHashMap<ServiceID, ServiceItem>();
  
      protected LookupCache getDataServiceLookupCache() {
--- 88,101 ----
   * @version $Id$
   */
! public class MetadataServer extends DataServer {
  
      private ServiceDiscoveryManager serviceDiscoveryManager = null;
      private LookupCache dataServiceLookupCache = null;
!     
!     /**
!      * Provides direct cached lookup of {@link DataService}s by their
!      * {@link ServiceID}.
!      */
!     public DataServiceMap dataServiceMap = new DataServiceMap();
  
      protected LookupCache getDataServiceLookupCache() {
***************
*** 139,143 ****
              dataServiceLookupCache = serviceDiscoveryManager
                      .createLookupCache(template,
!                             new DataServiceFilter() /* filter */, this /* ServiceDiscoveryListener */);
              
          } catch(RemoteException ex) {
--- 141,145 ----
              dataServiceLookupCache = serviceDiscoveryManager
                      .createLookupCache(template,
!                             new DataServiceFilter() /* filter */, dataServiceMap/* ServiceDiscoveryListener */);
              
          } catch(RemoteException ex) {
***************
*** 210,288 ****
          
      }
-         
-     /*
-      * ServiceDiscoveryListener.
-      */
- 
-     /**
-      * Adds the {@link ServiceItem} to the internal map to support
-      * {@link #getServiceByID()}
-      * <p>
-      * Note: This event is generated by the {@link LookupCache}. There is an
-      * event for each {@link DataService} as it joins any registrar in the set
-      * of registrars to which the {@link MetadataServer} is listening. The set
-      * of distinct joined {@link DataService}s is accessible via the
-      * {@link LookupCache}.
-      */
-     public void serviceAdded(ServiceDiscoveryEvent e) {
-         
-         log.info("" + e + ", class="
-                 + e.getPostEventServiceItem().toString());
-         
-         serviceIdMap.put(e.getPostEventServiceItem().serviceID, e
-                 .getPostEventServiceItem());
-         
-     }
- 
-     /**
-      * NOP.
-      */
-     public void serviceChanged(ServiceDiscoveryEvent e) {
- 
-         log.info(""+e+", class="
-                 + e.getPostEventServiceItem().toString());
-         
-         serviceIdMap.put(e.getPostEventServiceItem().serviceID, e
-                 .getPostEventServiceItem());
- 
-     }
- 
-     /**
-      * NOP.
-      */
-     public void serviceRemoved(ServiceDiscoveryEvent e) {
- 
-         log.info(""+e+", class="
-                 + e.getPreEventServiceItem().toString());
- 
-         serviceIdMap.remove(e.getPreEventServiceItem().serviceID);
- 
-     }
- 
-     /**
-      * Resolve the {@link ServiceID} for a {@link DataService} to the cached
-      * {@link ServiceItem} for that {@link DataService}.
-      * 
-      * @param serviceID
-      *            The {@link ServiceID} for the {@link DataService}.
-      * 
-      * @return The cache {@link ServiceItem} for that {@link DataService}.
-      */
-     public ServiceItem getDataServiceByID(ServiceID serviceID) {
-         
-         return serviceIdMap.get(serviceID);
-         
-     }
-     
-     /**
-      * Return the #of {@link DataService}s known to this {@link MetadataServer}.
-      * 
-      * @return The #of {@link DataService}s in the {@link LookupCache}.
-      */
-     public int getDataServiceCount() {
-         
-         return serviceIdMap.size();
-         
-     }
      
      /**
--- 212,215 ----
***************
*** 310,314 ****
          
          protected MetadataServer server;
!         
          /**
           * @param properties
--- 237,242 ----
          
          protected MetadataServer server;
!         private UUID serviceUUID;
! 
          /**
           * @param properties
***************
*** 330,333 ****
--- 258,273 ----
          }
  
+         protected UUID getDataServiceUUID() {
+ 
+             if (serviceUUID == null) {
+ 
+                 serviceUUID = JiniUtil.serviceID2UUID(server.getServiceID());
+                 
+             }
+             
+             return serviceUUID;
+             
+         }
+ 
          /**
           * Return the UUID of an under utilized data service.
***************
*** 348,352 ****
          public IDataService getDataServiceByID(ServiceID serviceID) throws IOException {
              
!             return (IDataService)server.getDataServiceByID(serviceID).service;
              
          }
--- 288,292 ----
          public IDataService getDataServiceByID(ServiceID serviceID) throws IOException {
              
!             return (IDataService)server.dataServiceMap.getDataServiceByID(serviceID).service;
              
          }

--- DataServiceClient.java DELETED ---

Index: ClientIndexView.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/ClientIndexView.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** ClientIndexView.java	23 Apr 2007 23:44:42 -0000	1.2
--- ClientIndexView.java	25 Apr 2007 14:57:10 -0000	1.3
***************
*** 48,51 ****
--- 48,52 ----
  package com.bigdata.service;
  
+ import java.io.IOException;
  import java.util.UUID;
  
***************
*** 56,60 ****
--- 57,64 ----
  import com.bigdata.btree.IEntryIterator;
  import com.bigdata.btree.IIndex;
+ import com.bigdata.scaleup.IPartitionMetadata;
  import com.bigdata.scaleup.PartitionedIndexView;
+ import com.bigdata.service.BigdataClient.BigdataFederation;
+ import com.bigdata.service.BigdataClient.IBigdataFederation;
  
  /**
***************
*** 81,84 ****
--- 85,96 ----
   *       and can be refactored for this purpose.
   * 
+  * @todo offer alternatives for the batch insert and remove methods that do not
+  *       return the old values to the client so that the client may opt to
+  *       minimize network traffic for data that it does not need.
+  * 
+  * @todo develop and offer policies for handling index partitions that are
+  *       unavailable at the time of the request (continued operation during
+  *       partial failure).
+  * 
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
   * @version $Id$
***************
*** 86,122 ****
  public class ClientIndexView implements IIndex {
  
!     protected final BigdataClient client;
!     protected final String name;
      
!     protected IMetadataService mdproxy;
!     protected UUID indexUUID; // @todo final.
      
!     public ClientIndexView(BigdataClient client, String name) {
          
!         if(client==null) throw new IllegalArgumentException();
          
          if(name==null) throw new IllegalArgumentException();
          
!         this.client = client;
  
!         this.name = name;
!         
!         /*
!          * obtain the proxy for a metadata service. if this instance fails, then
!          * we can always ask for a new instance for the same distributed
!          * database (failover).
!          */
! //        mdproxy = client.getMetadataService();
!         
!         /*
!          * obtain the indexUUID.  this is a means of verifying that the index
!          * exists.
!          */
! //        indexUUID = mdproxy.getIndexUUID(name);
          
      }
      
      public UUID getIndexUUID() {
          
          return indexUUID;
          
--- 98,172 ----
  public class ClientIndexView implements IIndex {
  
!     private final BigdataFederation fed;
!     private final long tx;
!     private final String name;
! 
!     /**
!      * The unique index identifier (initally null and cached once fetched).
!      */
!     private UUID indexUUID;
      
!     /**
!      * Obtain the proxy for a metadata service. if this instance fails, then we
!      * can always ask for a new instance for the same federation (failover).
!      */
!     protected IMetadataService getMetadataService() {
!         
!         return fed.getMetadataService();
!         
!     }
      
!     /**
!      * Create a view on a scale-out index.
!      * 
!      * @param fed
!      *            The federation containing the index.
!      * @param tx
!      *            The transaction identifier or zero(0L) iff the index view is
!      *            not isolated by a transaction.
!      * @param name
!      *            The index name.
!      */
!     public ClientIndexView(BigdataFederation fed, long tx, String name) {
          
!         if(fed ==null) throw new IllegalArgumentException();
          
          if(name==null) throw new IllegalArgumentException();
+     
+         if(tx != IBigdataFederation.UNISOLATED) {
+             
+             throw new UnsupportedOperationException(
+                     "Only unisolated views are supported at this time");
+             
+         }
          
!         this.fed = fed;
  
!         this.tx = tx;
          
+         this.name = name;
+        
      }
      
      public UUID getIndexUUID() {
          
+         if(indexUUID==null) {
+ 
+             /*
+              * obtain the UUID for the managed scale-out index.
+              */
+             
+             try {
+ 
+                 indexUUID = getMetadataService().getManagedIndexUUID(name);
+                 
+             } catch(IOException ex) {
+                 
+                 throw new RuntimeException(ex);
+                 
+             }
+ 
+         }
+         
          return indexUUID;
          
***************
*** 124,149 ****
  
      public boolean contains(byte[] key) {
!         // TODO Auto-generated method stub
!         return false;
      }
  
      public Object insert(Object key, Object value) {
!         // TODO Auto-generated method stub
!         return null;
      }
  
      public Object lookup(Object key) {
!         // TODO Auto-generated method stub
!         return null;
      }
  
      public Object remove(Object key) {
!         // TODO Auto-generated method stub
!         return null;
      }
  
      public int rangeCount(byte[] fromKey, byte[] toKey) {
!         // TODO Auto-generated method stub
!         return 0;
      }
  
--- 174,317 ----
  
      public boolean contains(byte[] key) {
!         
!         IPartitionMetadata pmd = fed.getPartition(name, key);
! 
!         IDataService dataService = fed.getDataService(pmd);
! 
!         final boolean[] ret;
!         
!         try {
! 
!             ret = dataService.batchContains(tx, name, 1, new byte[][]{key});
!             
!         } catch(Exception ex) {
!             
!             throw new RuntimeException(ex);
!             
!         }
!         
!         return ret[0];
!         
      }
  
      public Object insert(Object key, Object value) {
! 
!         IPartitionMetadata pmd = fed.getPartition(name, (byte[])key);
!         
!         IDataService dataService = fed.getDataService(pmd);
! 
!         final boolean returnOldValues = true;
!         
!         final byte[][] ret;
!         
!         try {
!             
!             ret = dataService.batchInsert(tx, name, 1,
!                     new byte[][] { (byte[]) key },
!                     new byte[][] { (byte[]) value }, returnOldValues);
! 
!         } catch (Exception ex) {
! 
!             throw new RuntimeException(ex);
! 
!         }
! 
!         return ret[0];
! 
      }
  
      public Object lookup(Object key) {
! 
!         IPartitionMetadata pmd = fed.getPartition(name, (byte[])key);
! 
!         IDataService dataService = fed.getDataService(pmd);
!         
!         final byte[][] ret;
!         
!         try {
!             
!             ret = dataService.batchLookup(tx, name, 1,
!                     new byte[][] { (byte[]) key });
! 
!         } catch (Exception ex) {
! 
!             throw new RuntimeException(ex);
! 
!         }
! 
!         return ret[0];
! 
      }
  
      public Object remove(Object key) {
! 
!         IPartitionMetadata pmd = fed.getPartition(name, (byte[])key);
! 
!         IDataService dataService = fed.getDataService(pmd);
!         
!         final byte[][] ret;
!         
!         final boolean returnOldValues = true;
!         
!         try {
!             
!             ret = dataService.batchRemove(tx, name, 1,
!                     new byte[][] { (byte[]) key }, returnOldValues );
! 
!         } catch (Exception ex) {
! 
!             throw new RuntimeException(ex);
! 
!         }
! 
!         return ret[0];
! 
      }
  
+     /**
+      * Returns the sum of the range count for each index partition spanned by
+      * the key range.
+      * 
+      * FIXME support range count that spans more than one partition.
+      * 
+      * FIXME Handle fromKey == null, toKey == null.
+      * 
+      * @todo note that it is possible (though unlikely) for an index partition
+      *       split or join to occur during this operation. Figure out how I want
+      *       to handle that, and how I want to handle that with transactional
+      *       isolation (presumably a read-only historical view of the metadata
+      *       index would be used - in which case we need to pass the tx into
+      *       the getPartition() method).
+      */
      public int rangeCount(byte[] fromKey, byte[] toKey) {
! 
!         IPartitionMetadata pmd1 = fed.getPartition(name, (byte[])fromKey);
!         
!         IPartitionMetadata pmd2 = fed.getPartition(name, (byte[])toKey);
!         
!         if(pmd2.getPartitionId()!=pmd1.getPartitionId()) {
!             
!             throw new UnsupportedOperationException(
!                     "Can not span partitions at this time");
!             
!         }
!         
!         IDataService dataService = fed.getDataService(pmd1);
! 
!         int rangeCount = 0;
!         
!         try {
! 
!             rangeCount += dataService.rangeCount(IDataService.UNISOLATED, name,
!                     fromKey, toKey);
!         
!         } catch(Exception ex) {
! 
!             throw new RuntimeException(ex);
!             
!         }
!         
!         return rangeCount;
!         
      }
  

Index: MetadataService.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/MetadataService.java,v
retrieving revision 1.8
retrieving revision 1.9
diff -C2 -d -r1.8 -r1.9
*** MetadataService.java	23 Apr 2007 23:44:42 -0000	1.8
--- MetadataService.java	25 Apr 2007 14:57:11 -0000	1.9
***************
*** 55,58 ****
--- 55,59 ----
  import net.jini.core.lookup.ServiceID;
  
+ import com.bigdata.scaleup.IPartitionMetadata;
  import com.bigdata.scaleup.MasterJournal;
  import com.bigdata.scaleup.MetadataIndex;
***************
*** 96,99 ****
--- 97,115 ----
          IMetadataService, IServiceShutdown {
  
+     /**
+      * Return the name of the metadata index.
+      * 
+      * @param indexName
+      *            The name of the scale-out index.
+      * 
+      * @return The name of the corresponding {@link MetadataIndex} that is used
+      *         to manage the partitions in the named scale-out index.
+      */
+     public static String getMetadataName(String indexName) {
+         
+         return "metadata-"+indexName;
+         
+     }
+     
      protected MetadataService(Properties properties) {
  
***************
*** 120,139 ****
                  
      }
!     
!     /**
!      * Return the name of the metadata index.
!      * 
!      * @param indexName
!      *            The name of the scale-out index.
!      * 
!      * @return The name of the corresponding {@link MetadataIndex} that is used
!      *         to manage the partitions in the named scale-out index.
!      */
!     public static String getMetadataName(String indexName) {
          
!         return "metadata-"+indexName;
          
      }
      
      /**
       * Registers a metadata index for a named scale-out index and creates the
--- 136,192 ----
                  
      }
! 
!     public UUID getManagedIndexUUID(String name) throws IOException {
          
!         // the name of the metadata index itself.
!         final String metadataName = getMetadataName(name);
!         
!         // make sure there is no metadata index for that btree.
!         MetadataIndex mdi = (MetadataIndex) journal.getIndex(metadataName);
!         
!         if(mdi == null) {
!             
!             return null;
!             
!         }
!         
!         return mdi.getManagedIndexUUID();
!         
!     }
! 
!     public IPartitionMetadata getPartition(String name,byte[] key) throws IOException {
!         
!         // the name of the metadata index itself.
!         final String metadataName = getMetadataName(name);
!         
!         // make sure there is no metadata index for that btree.
!         MetadataIndex mdi = (MetadataIndex) journal.getIndex(metadataName);
!         
!         if(mdi == null) {
!             
!             throw new IllegalArgumentException("Index not registered: " + name);
!             
!         }
! 
!         /*
!          * @todo this winds up deserializing the value into a PartitionMetadata
!          * object and then re-serializing it to return to the remote client.
!          */
!         IPartitionMetadata pmd = mdi.find(key);
!         
!         if( pmd == null ) {
!             
!             throw new IllegalStateException("No partitioned in index: "+name);
!             
!         }
!         
!         return pmd;
          
      }
      
+     /*
+      * Tasks.
+      */
+     
      /**
       * Registers a metadata index for a named scale-out index and creates the

--- EmbeddedDataService.java DELETED ---

Index: BigdataClient.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/BigdataClient.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** BigdataClient.java	23 Apr 2007 23:44:42 -0000	1.1
--- BigdataClient.java	25 Apr 2007 14:57:11 -0000	1.2
***************
*** 50,53 ****
--- 50,55 ----
  import java.io.IOException;
  import java.rmi.RemoteException;
+ import java.util.UUID;
+ import java.util.concurrent.ExecutionException;
  
  import net.jini.config.Configuration;
***************
*** 64,72 ****
--- 66,77 ----
  import net.jini.lookup.LookupCache;
  import net.jini.lookup.ServiceDiscoveryManager;
+ import net.jini.lookup.ServiceItemFilter;
  
  import org.apache.log4j.Logger;
  
  import com.bigdata.btree.IIndex;
+ import com.bigdata.io.SerializerUtil;
  import com.bigdata.journal.ITransactionManager;
+ import com.bigdata.journal.CommitRecordIndex.Entry;
  import com.bigdata.scaleup.IPartitionMetadata;
  
***************
*** 101,104 ****
--- 106,113 ----
   * transaction identifier for its read and write operations.
   * 
+  * @todo support transactions (there is no transaction manager service yet and
+  *       the 2-/3-phase commit protocol has not been implemented on the
+  *       journal).
+  * 
   * @todo Write or refactor logic to map operations across multiple partitions.
   * 
***************
*** 151,154 ****
--- 160,175 ----
      private ServiceDiscoveryManager serviceDiscoveryManager = null;
      private LookupCache metadataServiceLookupCache = null;
+     private LookupCache dataServiceLookupCache = null;
+     private DataServiceMap dataServiceMap = new DataServiceMap();
+ 
+     private final ServiceTemplate metadataServiceTemplate = new ServiceTemplate(
+             null, new Class[] { IMetadataService.class }, null);
+ 
+     private final ServiceTemplate dataServiceTemplate = new ServiceTemplate(
+             null, new Class[] { IDataService.class }, null);
+ 
+     private ServiceItemFilter metadataServiceFilter = null;
+ 
+     private ServiceItemFilter dataServiceFilter = new MetadataServer.DataServiceFilter();
  
      /**
***************
*** 170,198 ****
       * @todo parameter for the bigdata federation? (filter)
       */
!     public IMetadataService getMetadataService() throws InterruptedException {
!         
!         ServiceItem item = null;
!         
!         for(int i=0; i<10; i++) {
          
!             item = metadataServiceLookupCache.lookup(null);
              
!             if (item != null) {
  
!                 // found one.
                  
!                 break;
                  
              }
-         
-             Thread.sleep(200);
              
          }
          
!         if(item==null) {
!             
!             log.warn("No metadata service in cache");
              
!             return null;
              
          }
--- 191,299 ----
       * @todo parameter for the bigdata federation? (filter)
       */
!     public IMetadataService getMetadataService() {
          
!         ServiceItem item = metadataServiceLookupCache.lookup(null);
              
!         if (item == null) {
  
!             /*
!              * cache miss. do a remote query on the managed set of service
!              * registrars.
!              */
! 
!             log.info("Cache miss.");
!             
!             final long timeout = 1000L; // millis.
! 
!             try {
! 
!                 item = serviceDiscoveryManager.lookup(metadataServiceTemplate,
!                         metadataServiceFilter, timeout);
! 
!             } catch (RemoteException ex) {
                  
!                 log.error(ex);
!                 
!                 return null;
!                 
!             } catch(InterruptedException ex) {
!                 
!                 log.info("Interrupted - no match.");
!                 
!                 return null;
!                 
!             }
! 
!             if( item == null ) {
!                 
!                 // Could not discover a metadata service.
!                 
!                 log.warn("Could not discover metadata service");
!                 
!                 return null;
                  
              }
              
          }
          
!         log.info("Found: "+item);
!         
!         return (IMetadataService)item.service;
!         
!     }
! 
!     /**
!      * Resolve the {@link ServiceID} to an {@link IDataService} using a local
!      * cache.
!      * 
!      * @param serviceID
!      *            The identifier for a {@link DataService}.
!      *            
!      * @return The proxy for that {@link DataService} or <code>null</code> iff
!      *         the {@link DataService} could not be discovered.
!      */
!     public IDataService getDataService(ServiceID serviceID) {
!         
!         ServiceItem item = dataServiceMap.getDataServiceByID(serviceID);
!         
!         if (item == null) {
! 
!             /*
!              * cache miss. do a remote query on the managed set of service
!              * registrars.
!              */
! 
!             log.info("Cache miss.");
              
!             final long timeout = 1000L; // millis.
! 
!             try {
! 
!                 item = serviceDiscoveryManager.lookup(dataServiceTemplate,
!                         dataServiceFilter, timeout);
! 
!             } catch (RemoteException ex) {
!                 
!                 log.error(ex);
!                 
!                 return null;
!                 
!             } catch(InterruptedException ex) {
!                 
!                 log.info("Interrupted - no match.");
!                 
!                 return null;
!                 
!             }
! 
!             if( item == null ) {
!                 
!                 // Could not discover a data service.
!                 
!                 log.warn("Could not discover data service");
!                 
!                 return null;
!                 
!             }
              
          }
***************
*** 200,204 ****
          log.info("Found: "+item);
          
!         return (IMetadataService)item.service;
          
      }
--- 301,305 ----
          log.info("Found: "+item);
          
!         return (IDataService)item.service;
          
      }
***************
*** 273,277 ****
          /*
           * Setup a helper class that will be notified as services join or leave
!          * the various registrars to which the metadata server is listening.
           */
          try {
--- 374,378 ----
          /*
           * Setup a helper class that will be notified as services join or leave
!          * the various registrars to which the client is listening.
           */
          try {
***************
*** 301,313 ****
          try {
              
!             ServiceTemplate template = new ServiceTemplate(null,
!                     new Class[] { IMetadataService.class }, null);
  
!             metadataServiceLookupCache = serviceDiscoveryManager
!                     .createLookupCache(template, null /* filter */, null);
              
          } catch(RemoteException ex) {
              
!             log.error("Could not setup lookup metadataServiceLookupCache", ex);
  
              terminate();
--- 402,437 ----
          try {
              
!             metadataServiceLookupCache = serviceDiscoveryManager.createLookupCache(
!                     metadataServiceTemplate,
!                     null /* filter */, null);
!             
!         } catch(RemoteException ex) {
!             
!             log.error("Could not setup MetadataService LookupCache", ex);
  
!             terminate();
!             
!             System.exit(1);
!             
!         }
! 
!         /*
!          * Setup a LookupCache that is used to keep track of all data services
!          * registered with any service registrar to which the client is
!          * listening.
!          * 
!          * @todo provide filtering by attributes to select the primary vs
!          * failover metadata servers? by attributes identiying the bigdata
!          * federation?
!          */
!         try {
! 
!             dataServiceLookupCache = serviceDiscoveryManager.createLookupCache(
!                     dataServiceTemplate,
!                     dataServiceFilter /* filter */, dataServiceMap);
              
          } catch(RemoteException ex) {
              
!             log.error("Could not setup DataService LookupCache", ex);
  
              terminate();
***************
*** 332,335 ****
--- 456,467 ----
          }
  
+         if (dataServiceLookupCache != null) {
+ 
+             dataServiceLookupCache.terminate();
+             
+             dataServiceLookupCache = null;
+ 
+         }
+ 
          if (serviceDiscoveryManager != null) {
  
***************
*** 350,352 ****
--- 482,676 ----
      }
  
+     /**
+      * Connect to a bigdata federation.
+      * 
+      * @return
+      * 
+      * @todo determine how a federation will be identified, e.g., by a name that
+      *       is an {@link Entry} on the {@link MetadataServer} and
+      *       {@link DataServer} service descriptions and provide that name
+      *       attribute here. Note that a {@link MetadataService} can failover,
+      *       so the {@link ServiceID} for the {@link MetadataService} is not the
+      *       invariant, but rather the name attribute for the federation.
+      */
+     public IBigdataFederation connect() {
+ 
+         return new BigdataFederation(this);
+ 
+     }
+     
+     /**
+      * Interface to a bigdata federation.
+      *  
+      * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+      * @version $Id$
+      * 
+      * @todo declare public methods intended for application use.
+      */
+     public static interface IBigdataFederation {
+ 
+         /**
+          * A constant that may be used as the transaction identifier when the
+          * operation is <em>unisolated</em> (non-transactional).  The value of
+          * this constant is ZERO (0L).
+          */
+         public static final long UNISOLATED = 0L;
+ 
+         /**
+          * Register a scale-out index with the federation.
+          * 
+          * @param name
+          *            The index name.
+          *            
+          * @return The UUID that identifies the scale-out index.
+          */
+         public UUID registerIndex(String name);
+         
+         /**
+          * Obtain a view on a partitioned index.
+          * 
+          * @param tx
+          *            The transaction identifier or zero(0L) iff the index will
+          *            be unisolated.
+          * 
+          * @param name
+          *            The index name.
+          * 
+          * @return The index or <code>null</code> if the index is not
+          *         registered with the {@link MetadataService}.
+          */
+         public IIndex getIndex(long tx, String name);
+ 
+     }
+ 
+     /**
+      * This class encapsulates access to the metadata and data services for a
+      * bigdata federation.
+      * 
+      * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+      * @version $Id$
+      */
+     public static class BigdataFederation implements IBigdataFederation {
+        
+         private final BigdataClient client;
+ 
+         public IMetadataService getMetadataService() {
+             
+             return client.getMetadataService();
+             
+         }
+         
+         public BigdataFederation(BigdataClient client) {
+             
+             if(client==null) throw new IllegalArgumentException();
+             
+             this.client = client;
+             
+         }
+         
+         public UUID registerIndex(String name) {
+             
+             try {
+ 
+                 return getMetadataService().registerIndex(name);
+                 
+             } catch(Exception ex) {
+                 
+                 log.error(ex);
+                 
+                 throw new RuntimeException(ex);
+                 
+             }
+             
+         }
+         
+         /**
+          * @todo support isolated views, share cached data service information
+          *       between isolated and unisolated views.
+          */
+         public IIndex getIndex(long tx, String name) {
+ 
+             // @todo if(tx!=0l) validate exists?
+             
+             return new ClientIndexView(this,tx,name); 
+             
+         }
+ 
+         /**
+          * @todo setup cache.  test cache and lookup on metadata service if a 
+          * cache miss.
+          * 
+          * @param name
+          * @param key
+          * @return
+          */
+         public IPartitionMetadata getPartition(String name, byte[] key) {
+         
+ //            synchronized(indexCache) {
+ //            
+ //                Map<Integer,IDataService> partitionCache = indexCache.get(name);
+ //             
+ //                if(partitionCache==null) {
+ //                    
+ //                    partitionCache = new ConcurrentHashMap<Integer, IDataService>();
+ //                    
+ //                    indexCache.put(name, partitionCache);
+ //                    
+ //                }
+ //                
+ //                IDataService dataService = 
+ //                
+ //            }
+ 
+             /*
+              * Request the index partition metadata for the initial partition of the
+              * scale-out index.
+              */
+ 
+             IPartitionMetadata pmd;
+             
+             try {
+              
+                 pmd = getMetadataService().getPartition(name, key);
+                 
+             } catch(Exception ex) {
+                 
+                 throw new RuntimeException(ex);
+                 
+             }
+ 
+             return pmd;
+ 
+         }
+ //        
+ //        private Map<String, Map<Integer, IDataService>> indexCache = new ConcurrentHashMap<String, Map<Integer, IDataService>>(); 
+ 
+         /**
+          * Resolve the data service to which the index partition was mapped.
+          * 
+          * @todo use lookup cache in a real client.
+          */
+         public IDataService getDataService(IPartitionMetadata pmd) {
+ 
+             ServiceID serviceID = JiniUtil
+                     .uuid2ServiceID(pmd.getDataServices()[0]);
+ 
+             final IDataService dataService;
+             
+             try {
+ 
+                 dataService = client.getDataService(serviceID);
+                 
+             } catch(Exception ex) {
+                 
+                 throw new RuntimeException(ex);
+                 
+             }
+ 
+             return dataService;
+ 
+         }
+         
+     }
+     
  }

Index: DataService.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/DataService.java,v
retrieving revision 1.13
retrieving revision 1.14
diff -C2 -d -r1.13 -r1.14
*** DataService.java	23 Apr 2007 23:44:42 -0000	1.13
--- DataService.java	25 Apr 2007 14:57:11 -0000	1.14
***************
*** 197,201 ****
   *       gateways is also a common use case. Do we have to handle it specially?
   */
! public class DataService implements IDataService,
          IWritePipeline, IResourceTransfer {
  
--- 197,201 ----
   *       gateways is also a common use case. Do we have to handle it specially?
   */
! abstract public class DataService implements IDataService,
          IWritePipeline, IResourceTransfer {
  
***************
*** 348,351 ****
--- 348,359 ----
      }
  
+     /**
+      * The unique identifier for this data service - this is used mainly for log
+      * messages.
+      * 
+      * @return The unique data service identifier.
+      */
+     protected abstract UUID getDataServiceUUID();
+     
      /*
       * ITxCommitProtocol.
***************
*** 386,395 ****
      }
  
!     public void registerIndex(String name,UUID indexUUID) throws IOException, InterruptedException, ExecutionException{
          
!         journal.serialize(new RegisterIndexTask(name,indexUUID)).get();
          
      }
!     
      public void dropIndex(String name) throws IOException, InterruptedException, ExecutionException {
          
--- 394,418 ----
      }
  
!     public void registerIndex(String name, UUID indexUUID) throws IOException,
!             InterruptedException, ExecutionException {
! 
!         journal.serialize(new RegisterIndexTask(name, indexUUID)).get();
! 
!     }
!     
!     public UUID getIndexUUID(String name) throws IOException {
          
!         IIndex ndx = journal.getIndex(name);
!         
!         if(ndx == null) {
!             
!             return null;
!             
!         }
!         
!         return ndx.getIndexUUID();
          
      }
! 
      public void dropIndex(String name) throws IOException, InterruptedException, ExecutionException {
          
***************
*** 407,419 ****
      }
      
!     public void batchInsert(long tx, String name, int ntuples, byte[][] keys,
!             byte[][] vals) throws InterruptedException, ExecutionException {
!         
!         batchOp( tx, name, new BatchInsert(ntuples, keys, vals));
!         
      }
  
!     public boolean[] batchContains(long tx, String name, int ntuples, byte[][] keys
!             ) throws InterruptedException, ExecutionException {
          
          BatchContains op = new BatchContains(ntuples, keys, new boolean[ntuples]);
--- 430,447 ----
      }
      
!     public byte[][] batchInsert(long tx, String name, int ntuples,
!             byte[][] keys, byte[][] vals, boolean returnOldValues)
!             throws InterruptedException, ExecutionException {
! 
!         BatchInsert op = new BatchInsert(ntuples, keys, vals);
! 
!         batchOp(tx, name, op);
! 
!         return returnOldValues ? (byte[][]) op.values : null;
! 
      }
  
!     public boolean[] batchContains(long tx, String name, int ntuples,
!             byte[][] keys) throws InterruptedException, ExecutionException {
          
          BatchContains op = new BatchContains(ntuples, keys, new boolean[ntuples]);
***************
*** 706,710 ****
  
              journal.commit();
! 
              log.info("registeredIndex: "+name+", indexUUID="+indexUUID);
              
--- 734,738 ----
  
              journal.commit();
!             // @todo log the dataServiceID with each log message for this class.
              log.info("registeredIndex: "+name+", indexUUID="+indexUUID);
              

Index: AbstractServer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/AbstractServer.java,v
retrieving revision 1.8
retrieving revision 1.9
diff -C2 -d -r1.8 -r1.9
*** AbstractServer.java	23 Apr 2007 17:22:13 -0000	1.8
--- AbstractServer.java	25 Apr 2007 14:57:10 -0000	1.9
***************
*** 198,202 ****
      
      /**
!      * Return the assigned {@link ServiceID}.
       */
      public ServiceID getServiceID() {
--- 198,204 ----
      
      /**
!      * Return the assigned {@link ServiceID}. If this is a new service, then
!      * the {@link ServiceID} will be <code>null</code> until it has been
!      * assigned by a service registrar.
       */
      public ServiceID getServiceID() {

Index: IMetadataService.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/IMetadataService.java,v
retrieving revision 1.6
retrieving revision 1.7
diff -C2 -d -r1.6 -r1.7
*** IMetadataService.java	23 Apr 2007 17:22:14 -0000	1.6
--- IMetadataService.java	25 Apr 2007 14:57:10 -0000	1.7
***************
*** 55,58 ****
--- 55,61 ----
  import net.jini.core.lookup.ServiceItem;
  
+ import com.bigdata.scaleup.IPartitionMetadata;
+ import com.bigdata.scaleup.MetadataIndex;
+ 
  /**
   * A metadata service for a named index.
***************
*** 148,151 ****
--- 151,190 ----
      public UUID registerIndex(String name) throws IOException,
              InterruptedException, ExecutionException;
+     
+     /**
+      * Return the unique identifier for the managed index.
+      * 
+      * @param name
+      *            The name of the managed index.
+      * 
+      * @return The managed index UUID -or- <code>null</code> if there is no
+      *         managed scale-out index with that name.
+      *         
+      * @throws IOException
+      */
+     public UUID getManagedIndexUUID(String name) throws IOException;
  
+     /**
+      * Return the metadata for the index partition in which the key would be
+      * found.
+      * 
+      * @param name
+      *            The name of the scale-out index.
+      * @param key
+      *            The key.
+      * @return The metadata for the index partition in which that key would be
+      *         found.
+      *         
+      * @throws IOException
+      * 
+      * FIXME offer a variant that reports the index partitions spanned by a key
+      * range and write tests for that. Note that the remote API for that method
+      * should use a result-set data model to efficiently communicate the data
+      * when there are a large #of spanned partitions.
+      * 
+      * @see MetadataIndex#find(byte[])
+      */
+     public IPartitionMetadata getPartition(String name, byte[] key)
+             throws IOException;
+     
  }

Index: JiniUtil.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/JiniUtil.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** JiniUtil.java	23 Apr 2007 23:44:42 -0000	1.1
--- JiniUtil.java	25 Apr 2007 14:57:10 -0000	1.2
***************
*** 72,75 ****
--- 72,77 ----
      public static UUID serviceID2UUID(ServiceID serviceID) {
  
+         if(serviceID==null) return null;
+         
          return new UUID(serviceID.getMostSignificantBits(), serviceID
                  .getLeastSignificantBits());
***************
*** 88,91 ****
--- 90,95 ----
      public static ServiceID uuid2ServiceID(UUID uuid) {
  
+         if(uuid==null) return null;
+         
          return new ServiceID(uuid.getMostSignificantBits(), uuid
                  .getLeastSignificantBits());

[cweb-CVS] bigdata/src/test/com/bigdata/service TestBigdataClient.java, 1.1, 1.2 TestMetadataServer0.java, 1.4, 1.5 TestDataServer0.java, 1.4, 1.5

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/test/com/bigdata/service
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv16408/src/test/com/bigdata/service

Modified Files:
	TestBigdataClient.java TestMetadataServer0.java 
	TestDataServer0.java 
Log Message:
Progress on the BigdataClient.

Index: TestBigdataClient.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/service/TestBigdataClient.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** TestBigdataClient.java	23 Apr 2007 23:44:41 -0000	1.1
--- TestBigdataClient.java	25 Apr 2007 14:57:10 -0000	1.2
***************
*** 48,51 ****
--- 48,56 ----
  package com.bigdata.service;
  
+ import java.util.UUID;
+ 
+ import com.bigdata.btree.IIndex;
+ import com.bigdata.service.BigdataClient.IBigdataFederation;
+ 
  /**
   * Test suite for the {@link BigdataClient}.
***************
*** 184,188 ****
--- 189,251 ----
          
          assertNotNull("metadataService", client.getMetadataService());
+ 
+         IBigdataFederation fed = client.connect();
+         
+         final String name = "testIndex";
+         
+         UUID indexUUID = fed.registerIndex(name);
+         
+         IIndex ndx = fed.getIndex(IBigdataFederation.UNISOLATED,name);
+         
+         // fetches the index UUID from the metadata service.
+         assertEquals("indexUUID",indexUUID,ndx.getIndexUUID());
+ 
+         // uses the cached copy of the index UUID.
+         assertEquals("indexUUID",indexUUID,ndx.getIndexUUID());
+ 
+         // the index is empty.
+         assertFalse(ndx.contains(new byte[]{1}));
+      
+         // the key is not in the index.
+         assertEquals(null,(byte[])ndx.lookup(new byte[]{1}));
          
+         // insert a key-value pair.
+         assertNull(ndx.insert(new byte[]{1}, new byte[]{1}));
+ 
+         // verify index reports value exists for the key.
+         assertTrue(ndx.contains(new byte[]{1}));
+ 
+         // verify correct value in the index.
+         assertEquals(new byte[]{1},(byte[])ndx.lookup(new byte[]{1}));
+ 
+         // verify some range counts.
+         // @todo test with rangeCount(null,null).
+         assertEquals(0,ndx.rangeCount(new byte[]{}, new byte[]{1}));
+         assertEquals(1,ndx.rangeCount(new byte[]{}, new byte[]{2}));
+         assertEquals(1,ndx.rangeCount(new byte[]{1}, new byte[]{2}));
+         
+         // remove the index entry.
+         assertEquals(new byte[]{1},(byte[])ndx.remove(new byte[]{1}));
+ 
+         // the index is empty.
+         assertFalse(ndx.contains(new byte[]{1}));
+ 
+         // the key is not in the index.
+         assertEquals(null,(byte[])ndx.lookup(new byte[]{1}));
+         
+         /*
+          * verify some range counts.
+          * 
+          * Note: the range counts do NOT immediately adjust when keys are
+          * removed since deletion markers are written into those entries. The
+          * relevant index partition(s) need to be compacted for those deletion
+          * markers to be removed and the range counts adjusted to match.
+          * 
+          * @todo test with rangeCount(null,null).
+          */
+         assertEquals(0,ndx.rangeCount(new byte[]{}, new byte[]{1}));
+         assertEquals(1,ndx.rangeCount(new byte[]{}, new byte[]{2}));
+         assertEquals(1,ndx.rangeCount(new byte[]{1}, new byte[]{2}));
+ 
      }
      

Index: TestMetadataServer0.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/service/TestMetadataServer0.java,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** TestMetadataServer0.java	23 Apr 2007 23:44:41 -0000	1.4
--- TestMetadataServer0.java	25 Apr 2007 14:57:10 -0000	1.5
***************
*** 235,244 ****
              Thread.sleep(500);
  
!             assertNotNull(metadataServer0.getDataServiceByID(dataService0ID));
  
!             assertNotNull(metadataServer0.getDataServiceByID(dataService1ID));
  
!             assertEquals("#dataServices", 2, metadataServer0
!                     .getDataServiceCount());
  
          } finally {
--- 235,244 ----
              Thread.sleep(500);
  
!             assertNotNull(metadataServer0.dataServiceMap.getDataServiceByID(dataService0ID));
  
!             assertNotNull(metadataServer0.dataServiceMap.getDataServiceByID(dataService1ID));
  
!             assertEquals("#dataServices", 2, metadataServer0.
!                     dataServiceMap.getDataServiceCount());
  
          } finally {
***************
*** 257,266 ****
  
                  assertEquals("#dataServices", 1, metadataServer0
!                         .getDataServiceCount());
  
!                 assertNull(metadataServer0.getDataServiceByID(dataService0ID));
  
                  assertNotNull(metadataServer0
!                         .getDataServiceByID(dataService1ID));
  
              }
--- 257,266 ----
  
                  assertEquals("#dataServices", 1, metadataServer0
!                         .dataServiceMap.getDataServiceCount());
  
!                 assertNull(metadataServer0.dataServiceMap.getDataServiceByID(dataService0ID));
  
                  assertNotNull(metadataServer0
!                         .dataServiceMap.getDataServiceByID(dataService1ID));
  
              }
***************
*** 335,349 ****
           */
  
!         IPartitionMetadata pmd;
!         {
!          
!             byte[] val = metadataServiceProxy.lookup(IDataService.UNISOLATED,
!                     MetadataService.getMetadataName(indexName), new byte[] {});
!             
!             assertNotNull(val);
!             
!             pmd = (IPartitionMetadata) SerializerUtil.deserialize(val);
!             
!         }
          
          /*
--- 335,342 ----
           */
  
!         IPartitionMetadata pmd = metadataServiceProxy.getPartition(indexName,
!                 new byte[] {});
! 
!         assertNotNull(pmd);
          
          /*

Index: TestDataServer0.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/service/TestDataServer0.java,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** TestDataServer0.java	23 Apr 2007 13:09:30 -0000	1.4
--- TestDataServer0.java	25 Apr 2007 14:57:10 -0000	1.5
***************
*** 144,148 ****
          proxy.batchInsert(IDataService.UNISOLATED, name, 1,
                  new byte[][] { new byte[] { 1 } },
!                 new byte[][] { new byte[] { 1 } });
  
          // verify keys that exist/do not exist.
--- 144,149 ----
          proxy.batchInsert(IDataService.UNISOLATED, name, 1,
                  new byte[][] { new byte[] { 1 } },
!                 new byte[][] { new byte[] { 1 } },
!                 true /*returnOldValues*/ );
  
          // verify keys that exist/do not exist.

[cweb-CVS] bigdata/src/java/com/bigdata/journal IJournal.java, 1.13, 1.14 AbstractJournal.java, 1.12, 1.13

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/journal
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv16408/src/java/com/bigdata/journal

Modified Files:
	IJournal.java AbstractJournal.java 
Log Message:
Progress on the BigdataClient.

Index: IJournal.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/journal/IJournal.java,v
retrieving revision 1.13
retrieving revision 1.14
diff -C2 -d -r1.13 -r1.14
*** IJournal.java	16 Apr 2007 10:02:48 -0000	1.13
--- IJournal.java	25 Apr 2007 14:57:11 -0000	1.14
***************
*** 87,90 ****
--- 87,96 ----
      
      /**
+      * Shutdown the journal politely. Scheduled operations will run to
+      * completion, but no new operations will be scheduled.
+      */
+     public void shutdown();
+ 
+     /**
       * Return the named index which MAY may be invalidated by a
       * {@link IAtomicStore#commit()}.

Index: AbstractJournal.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/journal/AbstractJournal.java,v
retrieving revision 1.12
retrieving revision 1.13
diff -C2 -d -r1.12 -r1.13
*** AbstractJournal.java	13 Apr 2007 15:04:23 -0000	1.12
--- AbstractJournal.java	25 Apr 2007 14:57:11 -0000	1.13
***************
*** 775,781 ****
  
      /**
!      * Shutdown the journal politely. Active transactions and transactions
!      * pending commit will run to completion, but no new transactions will be
!      * accepted.
       */
      public void shutdown() {
--- 775,781 ----
  
      /**
!      * Shutdown the journal politely. Active writes, active transactions and
!      * transactions pending commit will run to completion, but no new
!      * transactions will be accepted.
       */
      public void shutdown() {

[cweb-CVS] bigdata/src/java/com/bigdata/service JiniUtil.java, NONE, 1.1 BigdataClient.java, NONE, 1.1 MetadataService.java, 1.7, 1.8 DataService.java, 1.12, 1.13 ClientIndexView.java, 1.1, 1.2 MetadataServer.java, 1.5, 1.6 AbstractClient.java, 1.2, NONE

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/service
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv12159/src/java/com/bigdata/service

Modified Files:
	MetadataService.java DataService.java ClientIndexView.java 
	MetadataServer.java 
Added Files:
	JiniUtil.java BigdataClient.java 
Removed Files:
	AbstractClient.java 
Log Message:
Expanding on the client and its test suite.

Index: ClientIndexView.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/ClientIndexView.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** ClientIndexView.java	23 Apr 2007 13:09:29 -0000	1.1
--- ClientIndexView.java	23 Apr 2007 23:44:42 -0000	1.2
***************
*** 86,90 ****
  public class ClientIndexView implements IIndex {
  
!     protected final AbstractClient client;
      protected final String name;
      
--- 86,90 ----
  public class ClientIndexView implements IIndex {
  
!     protected final BigdataClient client;
      protected final String name;
      
***************
*** 92,96 ****
      protected UUID indexUUID; // @todo final.
      
!     public ClientIndexView(AbstractClient client, String name) {
          
          if(client==null) throw new IllegalArgumentException();
--- 92,96 ----
      protected UUID indexUUID; // @todo final.
      
!     public ClientIndexView(BigdataClient client, String name) {
          
          if(client==null) throw new IllegalArgumentException();

Index: DataService.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/DataService.java,v
retrieving revision 1.12
retrieving revision 1.13
diff -C2 -d -r1.12 -r1.13
*** DataService.java	23 Apr 2007 18:58:38 -0000	1.12
--- DataService.java	23 Apr 2007 23:44:42 -0000	1.13
***************
*** 919,924 ****
      protected class RangeCountTask implements Callable<Object> {
  
!         // startTime or 0L iff unisolated.
!         private final long startTime;
          private final String name;
          private final byte[] fromKey;
--- 919,924 ----
      protected class RangeCountTask implements Callable<Object> {
  
! //        // startTime or 0L iff unisolated.
! //        private final long startTime;
          private final String name;
          private final byte[] fromKey;
***************
*** 930,934 ****
                  byte[] toKey) {
              
!             this.startTime = startTime;
              
              if(startTime != 0L) {
--- 930,934 ----
                  byte[] toKey) {
              
! //            this.startTime = startTime;
              
              if(startTime != 0L) {
***************
*** 1000,1005 ****
      protected class RangeQueryTask implements Callable<Object> {
  
!         // startTime or 0L iff unisolated.
!         private final long startTime;
          private final String name;
          private final byte[] fromKey;
--- 1000,1005 ----
      protected class RangeQueryTask implements Callable<Object> {
  
! //        // startTime or 0L iff unisolated.
! //        private final long startTime;
          private final String name;
          private final byte[] fromKey;
***************
*** 1013,1017 ****
                  byte[] toKey, int capacity, int flags) {
              
!             this.startTime = startTime;
              
              if(startTime != 0L) {
--- 1013,1017 ----
                  byte[] toKey, int capacity, int flags) {
              
! //            this.startTime = startTime;
              
              if(startTime != 0L) {

--- AbstractClient.java DELETED ---

Index: MetadataService.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/MetadataService.java,v
retrieving revision 1.7
retrieving revision 1.8
diff -C2 -d -r1.7 -r1.8
*** MetadataService.java	23 Apr 2007 18:58:38 -0000	1.7
--- MetadataService.java	23 Apr 2007 23:44:42 -0000	1.8
***************
*** 194,198 ****
              final UUID[] dataServices = new UUID[]{
                      
!                     MetadataServer.serviceID2UUID(dataServiceID)
                      
              };
--- 194,198 ----
              final UUID[] dataServices = new UUID[]{
                      
!                     JiniUtil.serviceID2UUID(dataServiceID)
                      
              };

Index: MetadataServer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/MetadataServer.java,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** MetadataServer.java	23 Apr 2007 18:58:38 -0000	1.5
--- MetadataServer.java	23 Apr 2007 23:44:42 -0000	1.6
***************
*** 53,57 ****
  import java.util.Map;
  import java.util.Properties;
- import java.util.UUID;
  import java.util.concurrent.ConcurrentHashMap;
  
--- 53,56 ----
***************
*** 105,140 ****
      
      /**
-      * Convert a Jini {@link ServiceID} to a {@link UUID} (this changes the kind
-      * of UUID implementation object but preserves the UUID data).
-      * 
-      * @param serviceID
-      *            The {@link ServiceID}.
-      * 
-      * @return The {@link UUID}.
-      */
-     public static UUID serviceID2UUID(ServiceID serviceID) {
- 
-         return new UUID(serviceID.getMostSignificantBits(), serviceID
-                 .getLeastSignificantBits());
- 
-     }
-     
-     /**
-      * Convert a {@link UUID} to a Jini {@link ServiceID} (this changes the kind
-      * of UUID implementation object but preserves the UUID data).
-      * 
-      * @param uuid
-      *            The {@link UUID}.
-      * 
-      * @return The Jini {@link ServiceID}.
-      */
-     public static ServiceID uuid2ServiceID(UUID uuid) {
- 
-         return new ServiceID(uuid.getMostSignificantBits(), uuid
-                 .getLeastSignificantBits());
- 
-     }
-     
-     /**
       * @param args
       */
--- 104,107 ----
***************
*** 242,253 ****
          }
          
!     };
          
      /*
       * ServiceDiscoveryListener.
       */
!     
      /**
!      * Adds the {@link ServiceItem} to the internal map to support {@link #getServiceByID()}
       * <p>
       * Note: This event is generated by the {@link LookupCache}. There is an
--- 209,221 ----
          }
          
!     }
          
      /*
       * ServiceDiscoveryListener.
       */
! 
      /**
!      * Adds the {@link ServiceItem} to the internal map to support
!      * {@link #getServiceByID()}
       * <p>
       * Note: This event is generated by the {@link LookupCache}. There is an

--- NEW FILE: BigdataClient.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Mar 24, 2007
 */

package com.bigdata.service;

import java.io.IOException;
import java.rmi.RemoteException;

import net.jini.config.Configuration;
import net.jini.config.ConfigurationException;
import net.jini.config.ConfigurationProvider;
import net.jini.core.discovery.LookupLocator;
import net.jini.core.lookup.ServiceID;
import net.jini.core.lookup.ServiceItem;
import net.jini.core.lookup.ServiceTemplate;
import net.jini.discovery.DiscoveryManagement;
import net.jini.discovery.LookupDiscovery;
import net.jini.discovery.LookupDiscoveryManager;
import net.jini.lease.LeaseRenewalManager;
import net.jini.lookup.LookupCache;
import net.jini.lookup.ServiceDiscoveryManager;

import org.apache.log4j.Logger;

import com.bigdata.btree.IIndex;
import com.bigdata.journal.ITransactionManager;
import com.bigdata.scaleup.IPartitionMetadata;

/**
 * Abstract base class for a bigdata client.
 * <p>
 * Clients are configured to perform service lookup with a jini group that
 * identifies the bigdata federation. Clients begin by discovering the
 * {@link IMetadataService}. Clients use the {@link IMetadataService} to manage
 * scale-out indices (add/drop) and to obtain the {@link IDataService} that it
 * must talk to for any given partition of a scale-out index. Once a client has
 * the {@link IDataService} for an index partition, it carries out read and
 * write operations using that {@link IDataService}. An {@link IIndex} factory
 * is provided that hides the {@link IMetadataService} and {@link IDataService}
 * communications from the application. The {@link IIndex} objects provided by
 * the factory are responsible for transparently discovering the
 * {@link IDataService}s on which the index partitions are located and
 * directing read and write operations appropriately.
 * <p>
 * A client may discover and use an {@link ITransactionManager} if needs to use
 * transactions as opposed to unisolated reads and writes. When the client
 * requests a transaction, the transaction manager responds with a long integer
 * containing the transaction identifier - this is simply the unique start time
 * assigned to that transaction by the transaction manager. The client then
 * provides that transaction identifier for operations that are isolated within
 * the transaction. When the client is done with the transaction, it must use
 * the transaction manager to either abort or commit the transaction.
 * (Transactions that fail to progress may be eventually aborted.)
 * <p>
 * When using unisolated operations, the client does not need to resolve or use
 * the transaction manager and it simply specifies <code>0L</code> as the
 * transaction identifier for its read and write operations.
 * 
 * @todo Write or refactor logic to map operations across multiple partitions.
 * 
 * FIXME Support factory for indices, reuse cached information across
 * transactional and non-transactional views of the same index.
 * 
 * @todo Use lambda expressions (and downloaded code) for server-side logic for
 *       batch operations.
 * 
 * @todo support failover metadata service discovery.
 * 
 * @todo document client configuration, the relationship between jini groups and
 *       a bigdata federation, and whether and how a single client could connect
 *       to more than one bigdata federation.
 * 
 * @todo Use a weak-ref cache with an LRU (or hard reference cache) to retain
 *       cached {@link IPartitionMetadata}.
 * 
 * @todo add a cache for {@link DataService}s. The client needs access by {
 *       indexName, key } to obtain a {@link ServiceID} for a
 *       {@link DataService} and then needs to translate the {@link ServiceID}
 *       to a data service proxy using a cache. The use case is not quite the
 *       same as the {@link MetadataServer} since a client is not interested in
 *       a cache of all {@link DataService}s but only in those to which its
 *       attention has been directed based on the index partitions on which it
 *       is operating. Also, the cache needs to be a weak-ref/LRU combination or
 *       hard reference cache so that we can discard cached services that we are
 *       not interested in tracking. There are other differences to, the client
 *       interest is really based on the index partition so if the index
 *       partition is moved, then the data service may no longer be of interest.
 * 
 * @see ClientIndexView
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public class BigdataClient {//implements DiscoveryListener {

    public static final transient Logger log = Logger
            .getLogger(BigdataClient.class);

    /**
     * The label in the {@link Configuration} file for the client configuration
     * data.
     */
    protected final static transient String CLIENT_LABEL = "ClientDescription";

    private Configuration config;
    private DiscoveryManagement discoveryManager = null;
    private ServiceDiscoveryManager serviceDiscoveryManager = null;
    private LookupCache metadataServiceLookupCache = null;

    /**
     * Return the {@link MetadataService} from the cache. If there is a cache
     * miss, then this will wait a bit for a {@link MetadataService} to show up
     * in the cache before reporting failure.
     * 
     * @return The {@link MetadataService} or <code>null</code> if none was
     *         found.
     * 
     * @throws InterruptedException
     * 
     * @todo force synchronous lookup on the {@link ServiceDiscoveryManager}
     *       when there is a cache miss?
     * 
     * @todo filter for primary vs failover service? maintain simple field for
     *       return?
     * 
     * @todo parameter for the bigdata federation? (filter)
     */
    public IMetadataService getMetadataService() throws InterruptedException {
        
        ServiceItem item = null;
        
        for(int i=0; i<10; i++) {
        
            item = metadataServiceLookupCache.lookup(null);
            
            if (item != null) {

                // found one.
                
                break;
                
            }
        
            Thread.sleep(200);
            
        }
        
        if(item==null) {
            
            log.warn("No metadata service in cache");
            
            return null;
            
        }
        
        log.info("Found: "+item);
        
        return (IMetadataService)item.service;
        
    }
    
    /**
     * Client startup reads {@link Configuration} data from the file(s) named by
     * <i>args</i>, starts the client, attempts to discover one or more
     * registrars and establishes a lookup cache for {@link MetadataService}s.
     * 
     * @param args
     *            The command line arguments.
     * 
     * @see #terminate()
     */
    public BigdataClient(String[] args) {

        // @todo verify that this belongs here.
        System.setSecurityManager(new SecurityManager());

        LookupLocator[] unicastLocators = null;
        String[] groups = null;

        try {

            config = ConfigurationProvider.getInstance(args);

            /*
             * Extract how the client will discover services from the
             * Configuration.
             */

            groups = (String[]) config.getEntry(CLIENT_LABEL, "groups",
                    String[].class, LookupDiscovery.ALL_GROUPS/* default */);

            unicastLocators = (LookupLocator[]) config
                    .getEntry(CLIENT_LABEL, "unicastLocators",
                            LookupLocator[].class, null/* default */);

        } catch (ConfigurationException ex) {

            log.fatal("Configuration error: " + ex, ex);

            System.exit(1);
            
        }
        
        try {

            /*
             * Note: This class will perform multicast discovery if ALL_GROUPS
             * is specified and otherwise requires you to specify one or more
             * unicast locators (URIs of hosts running discovery services). As
             * an alternative, you can use LookupDiscovery, which always does
             * multicast discovery.
             */
            discoveryManager = new LookupDiscoveryManager(
                    groups, unicastLocators, null /*DiscoveryListener*/
            );
            
//            discoveryManager = new LookupDiscovery(groups);

        } catch (IOException ex) {

            log.fatal("Lookup service discovery error: " + ex, ex);

            terminate();

            System.exit(1);

        }

        /*
         * Setup a helper class that will be notified as services join or leave
         * the various registrars to which the metadata server is listening.
         */
        try {

            serviceDiscoveryManager = new ServiceDiscoveryManager(
                    discoveryManager, new LeaseRenewalManager());
            
        } catch(IOException ex) {
            
            log.error("Could not initiate service discovery manager", ex);
            
            terminate();
            
            System.exit(1);
            
        }

        /*
         * Setup a LookupCache that is used to keep track of all metadata
         * services registered with any service registrar to which the client is
         * listening.
         * 
         * @todo provide filtering by attributes to select the primary vs
         * failover metadata servers? by attributes identiying the bigdata
         * federation?
         */
        try {
            
            ServiceTemplate template = new ServiceTemplate(null,
                    new Class[] { IMetadataService.class }, null);

            metadataServiceLookupCache = serviceDiscoveryManager
                    .createLookupCache(template, null /* filter */, null);
            
        } catch(RemoteException ex) {
            
            log.error("Could not setup lookup metadataServiceLookupCache", ex);

            terminate();
            
            System.exit(1);
            
        }

    }

    /**
     * Terminates various background processes.
     */
    public void terminate() {

        if (metadataServiceLookupCache != null) {

            metadataServiceLookupCache.terminate();
            
            metadataServiceLookupCache = null;

        }

        if (serviceDiscoveryManager != null) {

            serviceDiscoveryManager.terminate();

            serviceDiscoveryManager = null;
            
        }

        if (discoveryManager != null) {

            discoveryManager.terminate();

            discoveryManager = null;
            
        }

    }

}

--- NEW FILE: JiniUtil.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Apr 23, 2007
 */

package com.bigdata.service;

import java.util.UUID;

import net.jini.core.lookup.ServiceID;

/**
 * Some utility methods that attempt to isolate the aspects of the Jini
 * architecture that would otherwise bleed into the bigdata architecture.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public class JiniUtil {

    /**
     * Convert a Jini {@link ServiceID} to a {@link UUID} (this changes the kind
     * of UUID implementation object but preserves the UUID data).
     * 
     * @param serviceID
     *            The {@link ServiceID}.
     * 
     * @return The {@link UUID}.
     */
    public static UUID serviceID2UUID(ServiceID serviceID) {

        return new UUID(serviceID.getMostSignificantBits(), serviceID
                .getLeastSignificantBits());

    }
    
    /**
     * Convert a {@link UUID} to a Jini {@link ServiceID} (this changes the kind
     * of UUID implementation object but preserves the UUID data).
     * 
     * @param uuid
     *            The {@link UUID}.
     * 
     * @return The Jini {@link ServiceID}.
     */
    public static ServiceID uuid2ServiceID(UUID uuid) {

        return new ServiceID(uuid.getMostSignificantBits(), uuid
                .getLeastSignificantBits());

    }
    
}

[cweb-CVS] bigdata/src/resources/config/standalone Client.config, NONE, 1.1

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/resources/config/standalone
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv12159/src/resources/config/standalone

Added Files:
	Client.config 
Log Message:
Expanding on the client and its test suite.

--- NEW FILE: Client.config ---
import net.jini.discovery.LookupDiscovery;
import net.jini.core.discovery.LookupLocator;
import net.jini.core.entry.Entry;
import net.jini.lookup.entry.Name;
import net.jini.lookup.entry.Comment;
import net.jini.lookup.entry.Address;
import net.jini.lookup.entry.Location;
import net.jini.lookup.entry.ServiceInfo;

/*
 * Declares how the client will discover the Jini service registrar.  Once the
 * client has access to the registrar it will lookup the metadata service using
 * the provided group(s).  Each bigdata federation should use its own group for
 * ease of management and each client should limit its discovery to the group
 * corresponding to the federation(s) to which it will connect.
 */
ClientDescription {

	/*
	 * Note: multicast discovery is always used if LookupDiscovery.ALL_GROUPS is
	 * specified.
	 */
//    groups = LookupDiscovery.ALL_GROUPS;
    groups = new String[]{"bigdata"};

	/*
	 * One or more unicast URIs of the form jini://host/ or jini://host:port/.
	 * This MAY be an empty array if you want to use multicast discovery _and_
	 * you have specified LookupDiscovery.ALL_GROUPS above.
	 */
    unicastLocators = new LookupLocator[] { // empty
    	new LookupLocator("jini://localhost/")
                                          };    

}

[cweb-CVS] bigdata/src/test/com/bigdata/service TestBigdataClient.java, NONE, 1.1 TestMetadataServer0.java, 1.3, 1.4 TestAll.java, 1.3, 1.4

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/test/com/bigdata/service
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv12159/src/test/com/bigdata/service

Modified Files:
	TestMetadataServer0.java TestAll.java 
Added Files:
	TestBigdataClient.java 
Log Message:
Expanding on the client and its test suite.

--- NEW FILE: TestBigdataClient.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Apr 23, 2007
 */

package com.bigdata.service;

/**
 * Test suite for the {@link BigdataClient}.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public class TestBigdataClient extends AbstractServerTestCase {

    public TestBigdataClient() {
    }

    public TestBigdataClient(String name) {
        super(name);
    }

    /**
     * Starts in {@link #setUp()}.
     */
    MetadataServer metadataServer0;
    /**
     * Starts in {@link #setUp()}.
     */
    DataServer dataServer1;
    /**
     * Starts in {@link #setUp()}.
     */
    DataServer dataServer0;
    /**
     * Starts in {@link #setUp()}.
     */
    BigdataClient client;
    
    /**
     * Starts a {@link DataServer} ({@link #dataServer1}) and then a
     * {@link MetadataServer} ({@link #metadataServer0}). Each runs in its own
     * thread.
     */
    public void setUp() throws Exception {
        
        /*
         * Start up a data server before the metadata server so that we can make
         * sure that it is detected by the metadata server once it starts up.
         */
        dataServer1 = new DataServer(
                new String[] { "src/resources/config/standalone/DataServer1.config" });

        new Thread() {

            public void run() {
                
                dataServer1.run();
                
            }
            
        }.start();

        /*
         * Start the metadata server.
         */
        metadataServer0 = new MetadataServer(
                new String[] { "src/resources/config/standalone/MetadataServer0.config" });
        
        new Thread() {

            public void run() {
                
                metadataServer0.run();
                
            }
            
        }.start();

        /*
         * Start up a data server after the metadata server so that we can make
         * sure that it is detected by the metadata server once it starts up.
         */
        dataServer0 = new DataServer(
                new String[] { "src/resources/config/standalone/DataServer0.config" });

        new Thread() {

            public void run() {
                
                dataServer0.run();
                
            }
            
        }.start();

        client = new BigdataClient(
                new String[] { "src/resources/config/standalone/Client.config" });
        
    }
    
    /**
     * Destroy the test services.
     */
    public void tearDown() throws Exception {
        
        if(metadataServer0!=null) {

            metadataServer0.destroy();
        
            metadataServer0 = null;

        }

        if(dataServer0!=null) {

            dataServer0.destroy();
        
            dataServer0 = null;

        }
        
        if (dataServer1 != null) {
            
            dataServer1.destroy();

            dataServer1 = null;
            
        }
        
        if(client!=null) {

            client.terminate();

            client = null;
            
        }
        
    }
    
    public void test_federationRunning() throws Exception {
        
        assertNotNull("metadataService", client.getMetadataService());
        
    }
    
}

Index: TestMetadataServer0.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/service/TestMetadataServer0.java,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** TestMetadataServer0.java	23 Apr 2007 18:58:38 -0000	1.3
--- TestMetadataServer0.java	23 Apr 2007 23:44:41 -0000	1.4
***************
*** 355,359 ****
          {
  
!             ServiceID serviceID = MetadataServer.uuid2ServiceID(pmd.getDataServices()[0]);
  
              // @todo use lookup cache in a real client.
--- 355,359 ----
          {
  
!             ServiceID serviceID = JiniUtil.uuid2ServiceID(pmd.getDataServices()[0]);
  
              // @todo use lookup cache in a real client.

Index: TestAll.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/service/TestAll.java,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** TestAll.java	23 Apr 2007 13:09:30 -0000	1.3
--- TestAll.java	23 Apr 2007 23:44:41 -0000	1.4
***************
*** 82,85 ****
--- 82,90 ----
          suite.addTestSuite( TestMetadataServer0.class );
  
+         /*
+          * Test of a single client talking to a bigdata federation.
+          */
+         suite.addTestSuite( TestBigdataClient.class );
+         
          //        suite.addTestSuite( TestServer.class ); // Does not implement TestCase.
  

[cweb-CVS] bigdata/src/java/com/bigdata/scaleup IPartitionMetadata.java, NONE, 1.1 PartitionedIndexView.java, 1.5, 1.6 PartitionMetadata.java, 1.9, 1.10 MasterJournal.java, 1.8, 1.9 SegmentMetadata.java, 1.6, 1.7 MetadataIndex.java, 1.9, 1.10 ResourceState.java, 1.2, 1.3 SlaveJournal.java, 1.9, 1.10 AbstractPartitionTask.java, 1.6, 1.7 IResourceMetadata.java, 1.4, 1.5 JournalMetadata.java, 1.3, 1.4

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2382/src/java/com/bigdata/scaleup

Modified Files:
	PartitionedIndexView.java PartitionMetadata.java 
	MasterJournal.java SegmentMetadata.java MetadataIndex.java 
	ResourceState.java SlaveJournal.java 
	AbstractPartitionTask.java IResourceMetadata.java 
	JournalMetadata.java 
Added Files:
	IPartitionMetadata.java 
Log Message:
Refactored the MetadataIndex to extend UnisolatedBTree and put it to some use in the MetadataService and validated aspects of its use in the test case for the metadata service.

Index: MetadataIndex.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/MetadataIndex.java,v
retrieving revision 1.9
retrieving revision 1.10
diff -C2 -d -r1.9 -r1.10
*** MetadataIndex.java	16 Apr 2007 10:02:49 -0000	1.9
--- MetadataIndex.java	23 Apr 2007 18:58:37 -0000	1.10
***************
*** 52,60 ****
  import org.CognitiveWeb.extser.LongPacker;
  
- import com.bigdata.btree.BTree;
  import com.bigdata.btree.BTreeMetadata;
  import com.bigdata.btree.IIndex;
  import com.bigdata.btree.IndexSegment;
  import com.bigdata.isolation.IsolatedBTree;
  import com.bigdata.journal.Journal;
  import com.bigdata.journal.Tx;
--- 52,61 ----
  import org.CognitiveWeb.extser.LongPacker;
  
  import com.bigdata.btree.BTreeMetadata;
  import com.bigdata.btree.IIndex;
  import com.bigdata.btree.IndexSegment;
+ import com.bigdata.io.SerializerUtil;
  import com.bigdata.isolation.IsolatedBTree;
+ import com.bigdata.isolation.UnisolatedBTree;
  import com.bigdata.journal.Journal;
  import com.bigdata.journal.Tx;
***************
*** 79,99 ****
   *       restart-safe so that resources are eventually deleted.
   */
! public class MetadataIndex extends BTree {
  
      /**
!      * The name of the metadata index, which is the always the same as the name
!      * under which the corresponding {@link PartitionedIndexView} was registered.
!      * 
!      * @todo rename as managedIndexName (and the access method as well).
       */
!     private final String name;
      
      /**
!      * The name of the metadata index, which is the always the same as the name
!      * under which the corresponding {@link PartitionedIndexView} was registered.
       */
!     final public String getName() {
          
!         return name;
          
      }
--- 80,96 ----
   *       restart-safe so that resources are eventually deleted.
   */
! public class MetadataIndex extends UnisolatedBTree {
  
      /**
!      * The name of the managed index.
       */
!     private final String managedIndexName;
      
      /**
!      * The name of the managed index.
       */
!     final public String getManagedIndexName() {
          
!         return managedIndexName;
          
      }
***************
*** 138,143 ****
       * @param store
       *            The backing store.
-      * @param branchingFactor
-      *            The branching factor.
       * @param indexUUID
       *            The unique identifier for the metadata index.
--- 135,138 ----
***************
*** 145,157 ****
       *            The unique identifier for the managed scale-out index.
       * @param managedIndexName
!      *            The name of the managed scale out index.
       */
!     public MetadataIndex(IRawStore store, int branchingFactor, UUID indexUUID,
              UUID managedIndexUUID, String managedIndexName) {
  
!         super(store, branchingFactor, indexUUID,
!                 PartitionMetadata.Serializer.INSTANCE);
          
!         this.name = managedIndexName;
  
          //
--- 140,151 ----
       *            The unique identifier for the managed scale-out index.
       * @param managedIndexName
!      *            The managedIndexName of the managed scale out index.
       */
!     public MetadataIndex(IRawStore store, UUID indexUUID,
              UUID managedIndexUUID, String managedIndexName) {
  
!         super(store, indexUUID );
          
!         this.managedIndexName = managedIndexName;
  
          //
***************
*** 164,168 ****
          super(store, metadata);
          
!         name = ((MetadataIndexMetadata)metadata).getName();
  
          managedIndexUUID = ((MetadataIndexMetadata)metadata).getManagedIndexUUID();
--- 158,162 ----
          super(store, metadata);
          
!         managedIndexName = ((MetadataIndexMetadata)metadata).getName();
  
          managedIndexUUID = ((MetadataIndexMetadata)metadata).getManagedIndexUUID();
***************
*** 177,181 ****
  
      /**
!      * Extends the {@link BTreeMetadata} record to also hold the name of the
       * partitioned index.
       * 
--- 171,175 ----
  
      /**
!      * Extends the {@link BTreeMetadata} record to also hold the managedIndexName of the
       * partitioned index.
       * 
***************
*** 183,187 ****
       * @version $Id$
       */
!     public static class MetadataIndexMetadata extends BTreeMetadata implements Externalizable {
  
          private static final long serialVersionUID = -7309267778881420043L;
--- 177,181 ----
       * @version $Id$
       */
!     public static class MetadataIndexMetadata extends UnisolatedBTreeMetadata implements Externalizable {
  
          private static final long serialVersionUID = -7309267778881420043L;
***************
*** 191,195 ****
          
          /**
!          * The name of the metadata index, which is the always the same as the name
           * under which the corresponding {@link PartitionedIndexView} was registered.
           */
--- 185,189 ----
          
          /**
!          * The managedIndexName of the metadata index, which is the always the same as the managedIndexName
           * under which the corresponding {@link PartitionedIndexView} was registered.
           */
***************
*** 231,235 ****
              super(mdi);
              
!             this.name = mdi.getName();
              
              this.managedIndexUUID = mdi.getManagedIndexUUID();
--- 225,229 ----
              super(mdi);
              
!             this.name = mdi.getManagedIndexName();
              
              this.managedIndexUUID = mdi.getManagedIndexUUID();
***************
*** 340,344 ****
          if(index == -1) return null;
          
!         return (PartitionMetadata) super.valueAt( index );
          
      }
--- 334,340 ----
          if(index == -1) return null;
          
!         byte[] val = (byte[]) super.valueAt(index);
!         
!         return (PartitionMetadata) SerializerUtil.deserialize(val);
          
      }
***************
*** 356,360 ****
      public PartitionMetadata get(byte[] key) {
          
!         return (PartitionMetadata) super.lookup(key);
          
      }
--- 352,360 ----
      public PartitionMetadata get(byte[] key) {
          
!         byte[] val = (byte[]) super.lookup(key);
!         
!         if(val==null) return null;
!         
!         return (PartitionMetadata) SerializerUtil.deserialize(val);
          
      }
***************
*** 385,395 ****
          }
          
!         PartitionMetadata oldval = (PartitionMetadata) super.insert(key,
!                 val);
  
!         if (oldval != null && oldval.partId != val.partId) {
  
              throw new IllegalArgumentException("Expecting: partId="
!                     + oldval.partId + ", but have partId=" + val.partId);
  
          }
--- 385,400 ----
          }
          
!         byte[] newval = SerializerUtil.serialize(val);
!         
!         byte[] oldval2 = (byte[])super.insert(key, newval);
  
!         PartitionMetadata oldval = oldval2 == null ? null
!                 : (PartitionMetadata) SerializerUtil.deserialize(oldval2);
!         
!         if (oldval != null && oldval.getPartitionId() != val.getPartitionId()) {
  
              throw new IllegalArgumentException("Expecting: partId="
!                     + oldval.getPartitionId() + ", but have partId="
!                     + val.getPartitionId());
  
          }
***************
*** 411,415 ****
      public PartitionMetadata remove(byte[] key) {
          
!         return (PartitionMetadata) super.remove(key);
          
      }
--- 416,425 ----
      public PartitionMetadata remove(byte[] key) {
          
!         byte[] oldval2 = (byte[])super.remove(key);
! 
!         PartitionMetadata oldval = oldval2 == null ? null
!                 : (PartitionMetadata) SerializerUtil.deserialize(oldval2);
! 
!         return oldval;
          
      }

Index: JournalMetadata.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/JournalMetadata.java,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** JournalMetadata.java	29 Mar 2007 17:01:33 -0000	1.3
--- JournalMetadata.java	23 Apr 2007 18:58:37 -0000	1.4
***************
*** 53,66 ****
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
   * @version $Id$
-  * 
-  * @todo make this persistence capable by modifying the value serializer to
-  * use the {@link IResourceMetadata} interface.
   */
  public class JournalMetadata implements IResourceMetadata {
  
!     protected final String filename;
!     protected final long nbytes;
!     protected final ResourceState state;
!     protected final UUID uuid;
  
      public final boolean isIndexSegment() {
--- 53,63 ----
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
   * @version $Id$
   */
  public class JournalMetadata implements IResourceMetadata {
  
!     private final String filename;
!     private final long nbytes;
!     private final ResourceState state;
!     private final UUID uuid;
  
      public final boolean isIndexSegment() {

Index: IResourceMetadata.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/IResourceMetadata.java,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** IResourceMetadata.java	13 Apr 2007 15:04:24 -0000	1.4
--- IResourceMetadata.java	23 Apr 2007 18:58:37 -0000	1.5
***************
*** 44,48 ****
  package com.bigdata.scaleup;
  
- import java.io.File;
  import java.util.UUID;
  
--- 44,47 ----

Index: AbstractPartitionTask.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/AbstractPartitionTask.java,v
retrieving revision 1.6
retrieving revision 1.7
diff -C2 -d -r1.6 -r1.7
*** AbstractPartitionTask.java	13 Apr 2007 15:04:24 -0000	1.6
--- AbstractPartitionTask.java	23 Apr 2007 18:58:37 -0000	1.7
***************
*** 370,382 ****
  
              // assume only the last segment is live.
!             final SegmentMetadata oldSeg = (SegmentMetadata)pmd.resources[pmd.resources.length-1];
              
!             newSegs[0] = new SegmentMetadata(oldSeg.filename, oldSeg.nbytes,
!                     ResourceState.Dead, oldSeg.uuid);
  
              newSegs[1] = new SegmentMetadata(outFile.toString(), outFile
                      .length(), ResourceState.Live, builder.segmentUUID);
              
!             mdi.put(fromKey, new PartitionMetadata(0, pmd.dataServices, newSegs));
              
              return null;
--- 370,383 ----
  
              // assume only the last segment is live.
!             final SegmentMetadata oldSeg = (SegmentMetadata) pmd.getResources()[pmd
!                     .getResources().length - 1];
              
!             newSegs[0] = new SegmentMetadata(oldSeg.getFile(), oldSeg.size(),
!                     ResourceState.Dead, oldSeg.getUUID());
  
              newSegs[1] = new SegmentMetadata(outFile.toString(), outFile
                      .length(), ResourceState.Live, builder.segmentUUID);
              
!             mdi.put(fromKey, new PartitionMetadata(0, pmd.getDataServices(), newSegs));
              
              return null;

--- NEW FILE: IPartitionMetadata.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Apr 23, 2007
 */

package com.bigdata.scaleup;

import java.util.UUID;

import com.bigdata.btree.IndexSegment;
import com.bigdata.journal.Journal;

/**
 * A description of the metadata state for a partition of a scale-out index.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public interface IPartitionMetadata {

    /**
     * The unique partition identifier.
     */
    public int getPartitionId();

    /**
     * The ordered list of data services on which data for this partition will
     * be written and from which data for this partition may be read.
     * 
     * @todo refactor into a dataService UUID (required) and an array of zero or
     *       more media replication services for failover.
     */
    public UUID[] getDataServices();
    
    /**
     * Zero or more files containing {@link Journal}s or {@link IndexSegment}s
     * holding live data for this partition. The entries in the array reflect
     * the creation time of the index segments. The earliest resource is listed
     * first. The most recently created resource is listed last. Only the
     * {@link ResourceState#Live} resources must be read in order to provide a
     * consistent view of the data for the index partition.
     * {@link ResourceState#Dead} resources will eventually be scheduled for
     * restart-safe deletion.
     * 
     * @see ResourceState
     */
    public IResourceMetadata[] getResources();

}

Index: SlaveJournal.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/SlaveJournal.java,v
retrieving revision 1.9
retrieving revision 1.10
diff -C2 -d -r1.9 -r1.10
*** SlaveJournal.java	16 Apr 2007 10:02:49 -0000	1.9
--- SlaveJournal.java	23 Apr 2007 18:58:37 -0000	1.10
***************
*** 52,55 ****
--- 52,56 ----
  import com.bigdata.btree.IIndex;
  import com.bigdata.btree.IndexSegment;
+ import com.bigdata.io.SerializerUtil;
  import com.bigdata.isolation.IIsolatableIndex;
  import com.bigdata.isolation.UnisolatedBTree;
***************
*** 246,251 ****
          final UUID managedIndexUUID = btree.getIndexUUID();
          
!         MetadataIndex mdi = new MetadataIndex(this,
!                 BTree.DEFAULT_BRANCHING_FACTOR, metadataIndexUUID,
                  managedIndexUUID, name);
          
--- 247,251 ----
          final UUID managedIndexUUID = btree.getIndexUUID();
          
!         MetadataIndex mdi = new MetadataIndex(this, metadataIndexUUID,
                  managedIndexUUID, name);
          
***************
*** 254,258 ****
           * 
           * @todo specify the DataSerivce(s) that will accept writes for this
!          * index partition.  This should be done as part of refactoring the
           * metadata index into a first level service.
           */
--- 254,258 ----
           * 
           * @todo specify the DataSerivce(s) that will accept writes for this
!          * index partition. This should be done as part of refactoring the
           * metadata index into a first level service.
           */
***************
*** 378,386 ****
          while (itr.hasNext()) {
  
!             final PartitionMetadata pmd = (PartitionMetadata) itr.next();
  
!             for (int i = 0; i < pmd.resources.length; i++) {
  
!                 IResourceMetadata rmd = pmd.resources[i];
  
                  if (rmd.isIndexSegment()) {
--- 378,389 ----
          while (itr.hasNext()) {
  
!             final PartitionMetadata pmd = (PartitionMetadata) SerializerUtil
!                     .deserialize((byte[]) itr.next());
  
!             final IResourceMetadata[] resources = pmd.getResources();
!             
!             for (int i = 0; i < resources.length; i++) {
  
!                 IResourceMetadata rmd = resources[i];
  
                  if (rmd.isIndexSegment()) {
***************
*** 399,403 ****
              }
  
!             File dir = master.getPartitionDirectory(name, pmd.partId);
  
              if (!dir.delete()) {
--- 402,406 ----
              }
  
!             File dir = master.getPartitionDirectory(name, pmd.getPartitionId());
  
              if (!dir.delete()) {

Index: SegmentMetadata.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/SegmentMetadata.java,v
retrieving revision 1.6
retrieving revision 1.7
diff -C2 -d -r1.6 -r1.7
*** SegmentMetadata.java	13 Apr 2007 15:04:24 -0000	1.6
--- SegmentMetadata.java	23 Apr 2007 18:58:37 -0000	1.7
***************
*** 61,77 ****
       * The name of the file containing the {@link IndexSegment}.
       */
!     final public String filename;
      
      /**
       * The size of that file in bytes.
       */
!     final public long nbytes;
      
      /**
       * The life-cycle state for that {@link IndexSegment}.
       */
!     final public ResourceState state;
      
!     final public UUID uuid;
      
      public final boolean isIndexSegment() {
--- 61,77 ----
       * The name of the file containing the {@link IndexSegment}.
       */
!     private String filename;
      
      /**
       * The size of that file in bytes.
       */
!     private long nbytes;
      
      /**
       * The life-cycle state for that {@link IndexSegment}.
       */
!     private ResourceState state;
      
!     private UUID uuid;
      
      public final boolean isIndexSegment() {
***************
*** 114,131 ****
      }
  
!     public String getFile() {
          return filename;
      }
  
!     public long size() {
          return nbytes;
      }
  
!     public ResourceState state() {
          return state;
      }
  
!     public UUID getUUID() {
          return uuid;
      }
      
--- 114,138 ----
      }
  
!     final public String getFile() {
!         
          return filename;
+         
      }
  
!     final public long size() {
!         
          return nbytes;
      }
  
!     final public ResourceState state() {
!         
          return state;
+         
      }
  
!     final public UUID getUUID() {
!         
          return uuid;
+         
      }
      

Index: PartitionedIndexView.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/PartitionedIndexView.java,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** PartitionedIndexView.java	15 Apr 2007 18:17:05 -0000	1.5
--- PartitionedIndexView.java	23 Apr 2007 18:58:37 -0000	1.6
***************
*** 184,192 ****
          int n = 0;
          
!         for(int i=0; i<pmd.resources.length; i++) {
              
!             if(pmd.resources[i].state() != ResourceState.Live) continue;
              
!             segs[n++] = (IndexSegment) master.getIndex(getName(), pmd.resources[i]);
              
          }
--- 184,194 ----
          int n = 0;
          
!         IResourceMetadata[] resources = pmd.getResources();
!         
!         for(int i=0; i<resources.length; i++) {
              
!             if(resources[i].state() != ResourceState.Live) continue;
              
!             segs[n++] = (IndexSegment) master.getIndex(getName(), resources[i]);
              
          }
***************
*** 250,254 ****
          PartitionMetadata pmd = mdi.find(key);
          
!         ReadOnlyFusedView view = views.get(pmd.partId);
          
          if(view==null) {
--- 252,256 ----
          PartitionMetadata pmd = mdi.find(key);
          
!         ReadOnlyFusedView view = views.get(pmd.getPartitionId());
          
          if(view==null) {
***************
*** 279,283 ****
              
              // place the view in the cache.
!             views.put(pmd.partId, view);
              
          }
--- 281,285 ----
              
              // place the view in the cache.
!             views.put(pmd.getPartitionId(), view);
              
          }
***************
*** 309,332 ****
          final int liveCount = pmd.getLiveCount();
  
!         IResourceMetadata[] resources = new IResourceMetadata[liveCount + 1];
! 
          int n = 0;
  
!         resources[n++] = journalResource;
  
!         for (int i = 0; i < resources.length; i++) {
  
!             IResourceMetadata seg = pmd.resources[i];
  
              if (seg.state() != ResourceState.Live)
                  continue;
  
!             resources[n++] = seg;
  
          }
  
!         assert n == resources.length;
          
!         return resources;
  
      }
--- 311,336 ----
          final int liveCount = pmd.getLiveCount();
  
!         IResourceMetadata[] tmp = new IResourceMetadata[liveCount + 1];
!         
          int n = 0;
  
!         tmp[n++] = journalResource;
  
!         // @todo this should probably be a loop out to pmd.getResources().length
!         for (int i = 0; i < tmp.length; i++) {
  
!             // @todo refactor out of loop.
!             IResourceMetadata seg = pmd.getResources()[i];
  
              if (seg.state() != ResourceState.Live)
                  continue;
  
!             tmp[n++] = seg;
  
          }
  
!         assert n == tmp.length;
          
!         return tmp;
  
      }
***************
*** 334,338 ****
      public String getName() {
          
!         return mdi.getName();
          
      }
--- 338,342 ----
      public String getName() {
          
!         return mdi.getManagedIndexName();
          
      }

Index: PartitionMetadata.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/PartitionMetadata.java,v
retrieving revision 1.9
retrieving revision 1.10
diff -C2 -d -r1.9 -r1.10
*** PartitionMetadata.java	23 Apr 2007 17:22:16 -0000	1.9
--- PartitionMetadata.java	23 Apr 2007 18:58:37 -0000	1.10
***************
*** 44,61 ****
  package com.bigdata.scaleup;
  
- import java.io.DataInput;
  import java.io.Externalizable;
  import java.io.IOException;
  import java.util.UUID;
  
! import com.bigdata.btree.DataOutputBuffer;
! import com.bigdata.btree.IValueSerializer;
  import com.bigdata.btree.IndexSegment;
- import com.bigdata.isolation.UnisolatedBTree;
  import com.bigdata.journal.Journal;
  
  /**
-  * A description of the {@link IndexSegment}s containing the user data for a
-  * partition.
   * 
   * @todo provide a persistent event log or just integrate the state changes over
--- 44,60 ----
  package com.bigdata.scaleup;
  
  import java.io.Externalizable;
  import java.io.IOException;
+ import java.io.ObjectInput;
+ import java.io.ObjectOutput;
  import java.util.UUID;
  
! import org.CognitiveWeb.extser.LongPacker;
! import org.CognitiveWeb.extser.ShortPacker;
! 
  import com.bigdata.btree.IndexSegment;
  import com.bigdata.journal.Journal;
  
  /**
   * 
   * @todo provide a persistent event log or just integrate the state changes over
***************
*** 66,75 ****
   * @version $Id$
   */
! public class PartitionMetadata /*implements Externalizable*/ {
  
      /**
       * The unique partition identifier.
       */
!     final protected int partId;
  
      /**
--- 65,79 ----
   * @version $Id$
   */
! public class PartitionMetadata implements IPartitionMetadata, Externalizable {
! 
!     /**
!      * 
!      */
!     private static final long serialVersionUID = 5234405541356126104L;
  
      /**
       * The unique partition identifier.
       */
!     private int partId;
  
      /**
***************
*** 80,84 ****
       *       more media replication services for failover.
       */
!     final protected UUID[] dataServices;
      
      /**
--- 84,88 ----
       *       more media replication services for failover.
       */
!     private UUID[] dataServices;
      
      /**
***************
*** 94,98 ****
       * @see ResourceState
       */
!     final protected IResourceMetadata[] resources;
  
      public PartitionMetadata(int partId, UUID[] dataServices ) {
--- 98,109 ----
       * @see ResourceState
       */
!     private IResourceMetadata[] resources;
!     
!     /**
!      * De-serialization constructor.
!      */
!     public PartitionMetadata() {
!         
!     }
  
      public PartitionMetadata(int partId, UUID[] dataServices ) {
***************
*** 132,135 ****
--- 143,155 ----
      }
  
+ 
+     public int getPartitionId() {
+         return partId;
+     }
+ 
+     public IResourceMetadata[] getResources() {
+         return resources;
+     }
+ 
      /**
       * The #of data services on which the data for this partition will be
***************
*** 247,367 ****
      }
  
!     /**
!      * Serialization for an index segment metadata entry.
!      * 
!      * FIXME convert to use {@link UnisolatedBTree} (so byte[] values that we
!      * (de-)serialize one a one-by-one basis ourselves), implement
!      * {@link Externalizable} and use explicit versioning and packed integers.
!      * 
!      * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
!      * @version $Id$
!      */
!     public static class Serializer implements IValueSerializer {
! 
!         /**
!          * 
!          */
!         private static final long serialVersionUID = 4307076612127034103L;
! 
!         public transient static final PartitionMetadata.Serializer INSTANCE = new Serializer();
! 
! //        private static final transient int VERSION0 = 0x0;
          
!         public Serializer() {
!         }
! 
!         public void putValues(DataOutputBuffer os, Object[] values, int nvals)
!                 throws IOException {
              
!             for (int i = 0; i < nvals; i++) {
! 
!                 PartitionMetadata val = (PartitionMetadata) values[i];
! 
!                 final int nservices = val.dataServices.length;
!                 
!                 final int nresources = val.resources.length;
!                 
!                 os.writeInt(val.partId);
!                 
!                 os.writeInt(nservices);
! 
!                 os.writeInt(nresources);
  
!                 for( int j=0; j<nservices; j++) {
!                     
!                     final UUID serviceUUID = val.dataServices[j];
!                     
!                     os.writeLong(serviceUUID.getMostSignificantBits());
!                     
!                     os.writeLong(serviceUUID.getLeastSignificantBits());
!                     
!                 }
!                 
!                 for (int j = 0; j < nresources; j++) {
  
!                     IResourceMetadata rmd = val.resources[j];
  
!                     os.writeBoolean(rmd.isIndexSegment());
!                     
!                     os.writeUTF(rmd.getFile());
  
!                     os.writeLong(rmd.size());
  
!                     os.writeInt(rmd.state().valueOf());
  
!                     final UUID resourceUUID = rmd.getUUID();
!                     
!                     os.writeLong(resourceUUID.getMostSignificantBits());
!                     
!                     os.writeLong(resourceUUID.getLeastSignificantBits());
  
!                 }
  
!             }
  
          }
  
!         public void getValues(DataInput is, Object[] values, int nvals)
!                 throws IOException {
! 
!             for (int i = 0; i < nvals; i++) {
! 
!                 final int partId = is.readInt();
! 
!                 final int nservices = is.readInt();
!                 
!                 final int nresources = is.readInt();
!                 
!                 final UUID[] services = new UUID[nservices];
!                 
!                 final IResourceMetadata[] resources = new IResourceMetadata[nresources];
! 
!                 for (int j = 0; j < nservices; j++) {
  
!                     services[j] = new UUID(is.readLong()/*MSB*/,is.readLong()/*LSB*/);
!                     
!                 }
!                 
!                 for (int j = 0; j < nresources; j++) {
  
!                     boolean isIndexSegment = is.readBoolean();
!                     
!                     String filename = is.readUTF();
  
!                     long nbytes = is.readLong();
  
!                     ResourceState state = ResourceState.valueOf(is.readInt());
  
!                     UUID uuid = new UUID(is.readLong()/*MSB*/,is.readLong()/*LSB*/);
  
!                     resources[j] = (isIndexSegment ? new SegmentMetadata(
!                             filename, nbytes, state, uuid)
!                             : new JournalMetadata(filename, nbytes, state, uuid));
  
!                 }
  
!                 values[i] = new PartitionMetadata(partId, services, resources);
  
!             }
  
          }
--- 267,370 ----
      }
  
!     private static final transient short VERSION0 = 0x0;
!     
!     public void readExternal(ObjectInput in) throws IOException, ClassNotFoundException {
          
!         final short version = ShortPacker.unpackShort(in);
!         
!         if(version!=VERSION0) {
              
!             throw new IOException("Unknown version: "+version);
!             
!         }
!         
!         partId = (int) LongPacker.unpackLong(in);
  
!         final int nservices = ShortPacker.unpackShort(in);
!         
!         final int nresources = ShortPacker.unpackShort(in);
!         
!         dataServices = new UUID[nservices];
!         
!         resources = new IResourceMetadata[nresources];
  
!         for (int j = 0; j < nservices; j++) {
  
!             dataServices[j] = new UUID(in.readLong()/*MSB*/,in.readLong()/*LSB*/);
!             
!         }
!         
!         for (int j = 0; j < nresources; j++) {
  
!             boolean isIndexSegment = in.readBoolean();
!             
!             String filename = in.readUTF();
  
!             long nbytes = LongPacker.unpackLong(in);
  
!             ResourceState state = ResourceState.valueOf(ShortPacker.unpackShort(in));
  
!             UUID uuid = new UUID(in.readLong()/*MSB*/,in.readLong()/*LSB*/);
  
!             resources[j] = (isIndexSegment ? new SegmentMetadata(
!                     filename, nbytes, state, uuid)
!                     : new JournalMetadata(filename, nbytes, state, uuid));
  
          }
+         
+     }
  
!     public void writeExternal(ObjectOutput out) throws IOException {
  
!         ShortPacker.packShort(out, VERSION0);
!         
!         final int nservices = dataServices.length;
!         
!         final int nresources = resources.length;
!         
!         assert nservices < Short.MAX_VALUE;
  
!         assert nresources < Short.MAX_VALUE;
!         
!         LongPacker.packLong(out,partId);
!         
!         ShortPacker.packShort(out,(short)nservices);
  
!         ShortPacker.packShort(out,(short)nresources);
  
!         for( int j=0; j<nservices; j++) {
!             
!             final UUID serviceUUID = dataServices[j];
!             
!             out.writeLong(serviceUUID.getMostSignificantBits());
!             
!             out.writeLong(serviceUUID.getLeastSignificantBits());
!             
!         }
!         
!         /*
!          * Note: we serialize using the IResourceMetadata interface so that we
!          * can handle different subclasses and then special case the
!          * deserialization based on the boolean flag. This is significantly more
!          * compact than using an Externalizable for each ResourceMetadata object
!          * since we do not have to write the class names for those objects.
!          */
!         for (int j = 0; j < nresources; j++) {
  
!             IResourceMetadata rmd = resources[j];
  
!             out.writeBoolean(rmd.isIndexSegment());
!             
!             out.writeUTF(rmd.getFile());
  
!             LongPacker.packLong(out,rmd.size());
  
!             ShortPacker.packShort(out,rmd.state().valueOf());
  
!             final UUID resourceUUID = rmd.getUUID();
!             
!             out.writeLong(resourceUUID.getMostSignificantBits());
!             
!             out.writeLong(resourceUUID.getLeastSignificantBits());
  
          }

Index: MasterJournal.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/MasterJournal.java,v
retrieving revision 1.8
retrieving revision 1.9
diff -C2 -d -r1.8 -r1.9
*** MasterJournal.java	16 Apr 2007 10:02:49 -0000	1.8
--- MasterJournal.java	23 Apr 2007 18:58:37 -0000	1.9
***************
*** 922,926 ****
               */
  
!             File outFile = getSegmentFile(name,pmd.partId);
  
              IndexSegmentBuilder builder = new IndexSegmentBuilder(outFile,
--- 922,926 ----
               */
  
!             File outFile = getSegmentFile(name,pmd.getPartitionId());
  
              IndexSegmentBuilder builder = new IndexSegmentBuilder(outFile,
***************
*** 935,939 ****
               * update the metadata index for this partition.
               */
!             mdi.put(separatorKey, new PartitionMetadata(0, pmd.dataServices,
                      new SegmentMetadata[] { new SegmentMetadata("" + outFile,
                              outFile.length(), ResourceState.Live,
--- 935,939 ----
               * update the metadata index for this partition.
               */
!             mdi.put(separatorKey, new PartitionMetadata(0, pmd.getDataServices(),
                      new SegmentMetadata[] { new SegmentMetadata("" + outFile,
                              outFile.length(), ResourceState.Live,
***************
*** 961,965 ****
  
              // output file for the merged segment.
!             File outFile = getSegmentFile(name, pmd.partId);
  
              // merge the data from the btree on the slave and the index
--- 961,965 ----
  
              // output file for the merged segment.
!             File outFile = getSegmentFile(name, pmd.getPartitionId());
  
              // merge the data from the btree on the slave and the index
***************
*** 1005,1017 ****
  
              // assume only the last segment is live.
!             final SegmentMetadata oldSeg = (SegmentMetadata)pmd.resources[pmd.resources.length-1];
              
!             newSegs[0] = new SegmentMetadata(oldSeg.filename, oldSeg.nbytes,
!                     ResourceState.Dead, oldSeg.uuid);
  
              newSegs[1] = new SegmentMetadata(outFile.toString(), outFile
                      .length(), ResourceState.Live, builder.segmentUUID);
  
!             mdi.put(separatorKey, new PartitionMetadata(0, pmd.dataServices, newSegs));
  
  // /*
--- 1005,1018 ----
  
              // assume only the last segment is live.
!             final SegmentMetadata oldSeg = (SegmentMetadata) pmd.getResources()[pmd
!                     .getResources().length - 1];
              
!             newSegs[0] = new SegmentMetadata(oldSeg.getFile(), oldSeg.size(),
!                     ResourceState.Dead, oldSeg.getUUID());
  
              newSegs[1] = new SegmentMetadata(outFile.toString(), outFile
                      .length(), ResourceState.Live, builder.segmentUUID);
  
!             mdi.put(separatorKey, new PartitionMetadata(0, pmd.getDataServices(), newSegs));
  
  // /*
***************
*** 1032,1040 ****
  
              // assuming at most one dead and one live segment.
!             if(pmd.resources.length>1) {
                  
!                 final SegmentMetadata deadSeg = (SegmentMetadata)pmd.resources[0];
                  
!                 if(deadSeg.state!=ResourceState.Dead) {
                      
                      throw new AssertionError();
--- 1033,1041 ----
  
              // assuming at most one dead and one live segment.
!             if(pmd.getResources().length>1) {
                  
!                 final SegmentMetadata deadSeg = (SegmentMetadata)pmd.getResources()[0];
                  
!                 if(deadSeg.state()!=ResourceState.Dead) {
                      
                      throw new AssertionError();
***************
*** 1042,1046 ****
                  }
                  
!                 File deadSegFile = new File(deadSeg.filename);
                  
                  if(deadSegFile.exists() && !deadSegFile.delete() ) {
--- 1043,1047 ----
                  }
                  
!                 File deadSegFile = new File(deadSeg.getFile());
                  
                  if(deadSegFile.exists() && !deadSegFile.delete() ) {

Index: ResourceState.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/ResourceState.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** ResourceState.java	13 Apr 2007 15:04:24 -0000	1.2
--- ResourceState.java	23 Apr 2007 18:58:37 -0000	1.3
***************
*** 66,72 ****
      public String toString() {return name;}
      
!     public int valueOf() {return id;}
      
!     static public ResourceState valueOf(int id) {
          switch(id) {
          case 0: return New;
--- 66,72 ----
      public String toString() {return name;}
      
!     public short valueOf() {return (short)id;}
      
!     static public ResourceState valueOf(short id) {
          switch(id) {
          case 0: return New;

[cweb-CVS] bigdata/src/java/com/bigdata/service MetadataService.java, 1.6, 1.7 DataService.java, 1.11, 1.12 MetadataServer.java, 1.4, 1.5

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/service
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2382/src/java/com/bigdata/service

Modified Files:
	MetadataService.java DataService.java MetadataServer.java 
Log Message:
Refactored the MetadataIndex to extend UnisolatedBTree and put it to some use in the MetadataService and validated aspects of its use in the test case for the metadata service.

Index: DataService.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/DataService.java,v
retrieving revision 1.11
retrieving revision 1.12
diff -C2 -d -r1.11 -r1.12
*** DataService.java	23 Apr 2007 17:22:14 -0000	1.11
--- DataService.java	23 Apr 2007 18:58:38 -0000	1.12
***************
*** 105,108 ****
--- 105,115 ----
   *      an NIO interface to the data service.
   * 
+  * @todo should the data service monitor key ranges so that it can readily
+  *       reject requests when the index is registered but the key lies outside
+  *       of the range of an index partition mapped onto the data service? This
+  *       probably needs to happen in order for the data service to be able to
+  *       redirect clients if it sheds an index partition while the client has a
+  *       lease.
+  * 
   * @todo make sure that all service methods that create a {@link Future} do a
   *       get() so that the will block until the serialized task actually runs.

Index: MetadataService.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/MetadataService.java,v
retrieving revision 1.6
retrieving revision 1.7
diff -C2 -d -r1.6 -r1.7
*** MetadataService.java	23 Apr 2007 17:22:14 -0000	1.6
--- MetadataService.java	23 Apr 2007 18:58:38 -0000	1.7
***************
*** 55,60 ****
  import net.jini.core.lookup.ServiceID;
  
- import com.bigdata.btree.BTree;
- import com.bigdata.journal.Journal;
  import com.bigdata.scaleup.MasterJournal;
  import com.bigdata.scaleup.MetadataIndex;
--- 55,58 ----
***************
*** 67,70 ****
--- 65,75 ----
   * @version $Id$
   * 
+  * @todo Support creation and management of scale-out indices, including mapping
+  *       their index partitions to data services. Build out this functionality
+  *       with a series of test cases that invoke the basic operations
+  *       (registerIndex (done), getPartition (done), putPartition,
+  *       getPartitions, movePartition, etc.) and handle the load-balancing
+  *       later.
+  * 
   * @todo Provide a means to reconstruct the metadata index from the journal and
   *       index segment data files. We tag each journal and index segment with a
***************
*** 84,88 ****
   *       be able to access the historical state of the metadata index that
   *       corresponds to the commit time of interest for the database.)
!  *       
   * @todo support two-tier metadata index and reconcile with
   *       {@link MetadataIndex} and {@link MasterJournal}.
--- 89,93 ----
   *       be able to access the historical state of the metadata index that
   *       corresponds to the commit time of interest for the database.)
!  * 
   * @todo support two-tier metadata index and reconcile with
   *       {@link MetadataIndex} and {@link MasterJournal}.
***************
*** 97,123 ****
      }
      
-     /*
-      * @todo Support creation and management of scale-out indices, including
-      * mapping their index partitions to data services. Build out this
-      * functionality with a series of test cases that invoke the basic
-      * operations (registerIndex, getPartition, getPartitions, movePartition,
-      * etc.) and handle the load-balancing later.
-      */
- 
      /**
!      * @todo if if exits already?
       * 
       * @todo index metadata options (unicode support, per-partition counters,
!      *       etc.)  i had been passing in the BTree instance, but that does
!      *       not work as well in a distributed environment.
!      * 
!      * @todo refactor so that the {@link MetadataIndex} can be used with a
!      *       normal {@link Journal}
!      * 
!      * @todo create the initial partition and assign to the "least used" data
!      *       service (the data service impl needs to aggregate events and log
!      *       them in a manner that gets noticed by the metadata service).
       */
!     public UUID registerIndex(String name) throws IOException, InterruptedException, ExecutionException {
          
          MetadataIndex mdi = (MetadataIndex) journal.serialize(
--- 102,114 ----
      }
      
      /**
!      * @todo if if exits already? (and has consistent/inconsistent metadata)?
       * 
       * @todo index metadata options (unicode support, per-partition counters,
!      *       etc.) i had been passing in the BTree instance, but that does not
!      *       work as well in a distributed environment.
       */
!     public UUID registerIndex(String name) throws IOException,
!             InterruptedException, ExecutionException {
          
          MetadataIndex mdi = (MetadataIndex) journal.serialize(
***************
*** 187,195 ****
              final UUID managedIndexUUID = UUID.randomUUID();
              
!             MetadataIndex mdi = new MetadataIndex(journal,
!                     BTree.DEFAULT_BRANCHING_FACTOR, metadataIndexUUID,
                      managedIndexUUID, name);
              
              /*
               * Setup the initial partition which is able to accept any key.
               */
--- 178,190 ----
              final UUID managedIndexUUID = UUID.randomUUID();
              
!             MetadataIndex mdi = new MetadataIndex(journal, metadataIndexUUID,
                      managedIndexUUID, name);
              
              /*
+              * Register the metadata index.
+              */
+             journal.registerIndex(metadataName, mdi);
+             
+             /*
               * Setup the initial partition which is able to accept any key.
               */
***************
*** 197,206 ****
              ServiceID dataServiceID = getUnderUtilizedDataService();
              
-             UUID dataServiceUUID = new UUID(dataServiceID
-                     .getMostSignificantBits(), dataServiceID
-                     .getLeastSignificantBits());
-             
              final UUID[] dataServices = new UUID[]{
!                     dataServiceUUID
              };
              
--- 192,199 ----
              ServiceID dataServiceID = getUnderUtilizedDataService();
              
              final UUID[] dataServices = new UUID[]{
!                     
!                     MetadataServer.serviceID2UUID(dataServiceID)
!                     
              };
              
***************
*** 219,223 ****
               * MUST NOT be run inside of the serialized write on the metadata
               * index itself. It is a good question exactly when this operation
!              * should be run....
               */
              
--- 212,218 ----
               * MUST NOT be run inside of the serialized write on the metadata
               * index itself. It is a good question exactly when this operation
!              * should be run.... One option would be lazily by the data service
!              * when it receives a request for an index / index key range that
!              * was not known to be mapped to that data service.
               */
              

Index: MetadataServer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/MetadataServer.java,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** MetadataServer.java	23 Apr 2007 17:22:14 -0000	1.4
--- MetadataServer.java	23 Apr 2007 18:58:38 -0000	1.5
***************
*** 53,56 ****
--- 53,57 ----
  import java.util.Map;
  import java.util.Properties;
+ import java.util.UUID;
  import java.util.concurrent.ConcurrentHashMap;
  
***************
*** 104,107 ****
--- 105,140 ----
      
      /**
+      * Convert a Jini {@link ServiceID} to a {@link UUID} (this changes the kind
+      * of UUID implementation object but preserves the UUID data).
+      * 
+      * @param serviceID
+      *            The {@link ServiceID}.
+      * 
+      * @return The {@link UUID}.
+      */
+     public static UUID serviceID2UUID(ServiceID serviceID) {
+ 
+         return new UUID(serviceID.getMostSignificantBits(), serviceID
+                 .getLeastSignificantBits());
+ 
+     }
+     
+     /**
+      * Convert a {@link UUID} to a Jini {@link ServiceID} (this changes the kind
+      * of UUID implementation object but preserves the UUID data).
+      * 
+      * @param uuid
+      *            The {@link UUID}.
+      * 
+      * @return The Jini {@link ServiceID}.
+      */
+     public static ServiceID uuid2ServiceID(UUID uuid) {
+ 
+         return new ServiceID(uuid.getMostSignificantBits(), uuid
+                 .getLeastSignificantBits());
+ 
+     }
+     
+     /**
       * @param args
       */

[cweb-CVS] bigdata/src/test/com/bigdata/scaleup TestMetadataIndex.java, 1.16, 1.17 TestPartitionedJournal.java, 1.10, 1.11

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/test/com/bigdata/scaleup
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2382/src/test/com/bigdata/scaleup

Modified Files:
	TestMetadataIndex.java TestPartitionedJournal.java 
Log Message:
Refactored the MetadataIndex to extend UnisolatedBTree and put it to some use in the MetadataService and validated aspects of its use in the test case for the metadata service.

Index: TestMetadataIndex.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/scaleup/TestMetadataIndex.java,v
retrieving revision 1.16
retrieving revision 1.17
diff -C2 -d -r1.16 -r1.17
*** TestMetadataIndex.java	23 Apr 2007 17:22:17 -0000	1.16
--- TestMetadataIndex.java	23 Apr 2007 18:58:38 -0000	1.17
***************
*** 137,141 ****
          
          // create the metadata index.
!         MetadataIndex md = new MetadataIndex(store, 3, indexUUID,
                  managedIndexUUID, "abc");
  
--- 137,141 ----
          
          // create the metadata index.
!         MetadataIndex md = new MetadataIndex(store, indexUUID,
                  managedIndexUUID, "abc");
  
***************
*** 220,224 ****
          
          // create the metadata index.
!         MetadataIndex md = new MetadataIndex(store, 3, indexUUID,
                  managedIndexUUID, "abc");
  
--- 220,224 ----
          
          // create the metadata index.
!         MetadataIndex md = new MetadataIndex(store, indexUUID,
                  managedIndexUUID, "abc");
  
***************
*** 290,294 ****
          // re-load the index.
          md = (MetadataIndex)BTree.load(store, addr);
!         assertEquals("name","abc",md.getName());
          
          assertEquals("#entries",3,md.getEntryCount());
--- 290,294 ----
          // re-load the index.
          md = (MetadataIndex)BTree.load(store, addr);
!         assertEquals("name","abc",md.getManagedIndexName());
          
          assertEquals("#entries",3,md.getEntryCount());
***************
*** 320,324 ****
          
          // create the metadata index.
!         MetadataIndex md = new MetadataIndex(store, 3, indexUUID,
                  managedIndexUUID, "abc");
  
--- 320,324 ----
          
          // create the metadata index.
!         MetadataIndex md = new MetadataIndex(store, indexUUID,
                  managedIndexUUID, "abc");
  
***************
*** 440,444 ****
          
          // create the metadata index.
!         MetadataIndex md = new MetadataIndex(store, 3, indexUUID,
                  managedIndexUUID, "abc");
          
--- 440,444 ----
          
          // create the metadata index.
!         MetadataIndex md = new MetadataIndex(store, indexUUID,
                  managedIndexUUID, "abc");
          
***************
*** 550,554 ****
          
          // create the metadata index.
!         MetadataIndex md = new MetadataIndex(store, 3, indexUUID,
                  managedIndexUUID, "abc");
          
--- 550,554 ----
          
          // create the metadata index.
!         MetadataIndex md = new MetadataIndex(store, indexUUID,
                  managedIndexUUID, "abc");
          
***************
*** 734,738 ****
          
          // create the metadata index.
!         MetadataIndex md = new MetadataIndex(store, 3, indexUUID,
                  managedIndexUUID, "abc");
          
--- 734,738 ----
          
          // create the metadata index.
!         MetadataIndex md = new MetadataIndex(store, indexUUID,
                  managedIndexUUID, "abc");
          
***************
*** 968,972 ****
                  seg.close();
                  
!                 new File(oldpart.resources[0].getFile()).delete();
  
                  // this is now the current index segment.
--- 968,972 ----
                  seg.close();
                  
!                 new File(oldpart.getResources()[0].getFile()).delete();
  
                  // this is now the current index segment.
***************
*** 995,999 ****
          seg.close();
  
!         new File(md.get(new byte[]{}).resources[0].getFile()).delete();
  
          System.err.println("End of stress test: ntrial="+ntrials+", nops="+nops);
--- 995,999 ----
          seg.close();
  
!         new File(md.get(new byte[]{}).getResources()[0].getFile()).delete();
  
          System.err.println("End of stress test: ntrial="+ntrials+", nops="+nops);
***************
*** 1035,1039 ****
          
          // create the metadata index.
!         MetadataIndex md = new MetadataIndex(store, 3, indexUUID,
                  managedIndexUUID, "abc");
          
--- 1035,1039 ----
          
          // create the metadata index.
!         MetadataIndex md = new MetadataIndex(store, indexUUID,
                  managedIndexUUID, "abc");
          

Index: TestPartitionedJournal.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/scaleup/TestPartitionedJournal.java,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** TestPartitionedJournal.java	13 Apr 2007 15:04:24 -0000	1.10
--- TestPartitionedJournal.java	23 Apr 2007 18:58:38 -0000	1.11
***************
*** 327,331 ****
          assertEquals("#partitions",1,mdi.getEntryCount());
          
!         assertEquals("#segments",0,mdi.get(new byte[]{}).resources.length);
          
          /*
--- 327,331 ----
          assertEquals("#partitions",1,mdi.getEntryCount());
          
!         assertEquals("#segments",0,mdi.get(new byte[]{}).getResources().length);
          
          /*
***************
*** 376,380 ****
          assertEquals("#partitions",1,mdi.getEntryCount());
          
!         assertEquals("#segments",1,mdi.get(new byte[]{}).resources.length);
          
          /*
--- 376,380 ----
          assertEquals("#partitions",1,mdi.getEntryCount());
          
!         assertEquals("#segments",1,mdi.get(new byte[]{}).getResources().length);
          
          /*
***************
*** 453,469 ****
              PartitionMetadata pmd = mdi.get(separatorKey);
              
!             assertEquals("partId",0,pmd.partId);
              
              assertEquals("#segments", 1, pmd.getLiveCount());
              
!             if(pmd.resources.length>1) {
                  
!                 assertEquals("#segments",2,pmd.resources.length);
                  
                  assertEquals("state", ResourceState.Dead,
!                         pmd.resources[0].state());
                  
                  assertEquals("state", ResourceState.Live,
!                         pmd.resources[1].state());
                  
              }
--- 453,471 ----
              PartitionMetadata pmd = mdi.get(separatorKey);
              
!             assertEquals("partId",0,pmd.getPartitionId());
              
              assertEquals("#segments", 1, pmd.getLiveCount());
              
!             IResourceMetadata[] resources = pmd.getResources();
!             
!             if(resources.length>1) {
                  
!                 assertEquals("#segments",2,resources.length);
                  
                  assertEquals("state", ResourceState.Dead,
!                         resources[0].state());
                  
                  assertEquals("state", ResourceState.Live,
!                         resources[1].state());
                  
              }

[cweb-CVS] bigdata/src/java/com/bigdata/isolation UnisolatedBTree.java, 1.12, 1.13

[Bryan T. <tho...@us...>]

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/isolation
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2382/src/java/com/bigdata/isolation

Modified Files:
	UnisolatedBTree.java 
Log Message:
Refactored the MetadataIndex to extend UnisolatedBTree and put it to some use in the MetadataService and validated aspects of its use in the test case for the metadata service.

Index: UnisolatedBTree.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/isolation/UnisolatedBTree.java,v
retrieving revision 1.12
retrieving revision 1.13
diff -C2 -d -r1.12 -r1.13
*** UnisolatedBTree.java	18 Apr 2007 17:29:20 -0000	1.12
--- UnisolatedBTree.java	23 Apr 2007 18:58:38 -0000	1.13
***************
*** 249,253 ****
      }
  
!     public BTreeMetadata newMetadata() {
          
          return new UnisolatedBTreeMetadata(this);
--- 249,253 ----
      }
  
!     protected BTreeMetadata newMetadata() {
          
          return new UnisolatedBTreeMetadata(this);