cweb-cvs Mailing List for CognitiveWeb (Page 10)

Status: Beta

Brought to you by: beebs, guylukes, thompsonbry

cweb-cvs — CVS commit messages are posted to this list.

You can subscribe to this list here.

2006	_Jan	_Feb	_Mar (414)	_Apr (123)	_May (448)	_Jun (180)	_Jul (17)	_Aug (49)	_Sep (3)	_Oct (92)	_Nov (101)	_Dec (64)
2007	_Jan (132)	_Feb (230)	_Mar (146)	_Apr (146)	_May	_Jun	_Jul (34)	_Aug (4)	_Sep (3)	_Oct (10)	_Nov (12)	_Dec (24)
2008	_Jan (6)	_Feb	_Mar	_Apr	_May (1)	_Jun	_Jul	_Aug	_Sep	_Oct (11)	_Nov (4)	_Dec
2009	_Jan	_Feb	_Mar	_Apr	_May	_Jun	_Jul	_Aug	_Sep (1)	_Oct	_Nov	_Dec

Flat | Threaded

<< < 1 .. 8 9 10 11 12 .. 91 > >> (Page 10 of 91)

[cweb-CVS] bigdata-rdf/src/java/com/bigdata/rdf/serializers TermIdSerializer.java, 1.2, 1.3 StatementSerializer.java, 1.2, 1.3 RdfValueSerializer.java, 1.3, 1.4

From: Bryan T. <tho...@us...> - 2007-04-13 15:02:38

Update of /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/serializers
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv9212/src/java/com/bigdata/rdf/serializers

Modified Files:
	TermIdSerializer.java StatementSerializer.java 
	RdfValueSerializer.java 
Log Message:
Package rename from com.bigdata.objndx to com.bigdata.btree.  Since we are using CVS, this is going to break the access to prior versions in the source file history.

Index: TermIdSerializer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/serializers/TermIdSerializer.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** TermIdSerializer.java	12 Apr 2007 23:59:21 -0000	1.2
--- TermIdSerializer.java	13 Apr 2007 15:02:32 -0000	1.3
***************
*** 49,54 ****
  import org.CognitiveWeb.extser.LongPacker;
  
! import com.bigdata.objndx.DataOutputBuffer;
! import com.bigdata.objndx.IValueSerializer;
  
  /**
--- 49,54 ----
  import org.CognitiveWeb.extser.LongPacker;
  
! import com.bigdata.btree.DataOutputBuffer;
! import com.bigdata.btree.IValueSerializer;
  
  /**

Index: RdfValueSerializer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/serializers/RdfValueSerializer.java,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** RdfValueSerializer.java	12 Apr 2007 23:59:21 -0000	1.3
--- RdfValueSerializer.java	13 Apr 2007 15:02:33 -0000	1.4
***************
*** 51,57 ****
  import org.openrdf.model.Value;
  
! import com.bigdata.objndx.DataOutputBuffer;
! import com.bigdata.objndx.IIndex;
! import com.bigdata.objndx.IValueSerializer;
  import com.bigdata.rdf.RdfKeyBuilder;
  import com.bigdata.rdf.model.OptimizedValueFactory._BNode;
--- 51,57 ----
  import org.openrdf.model.Value;
  
! import com.bigdata.btree.DataOutputBuffer;
! import com.bigdata.btree.IIndex;
! import com.bigdata.btree.IValueSerializer;
  import com.bigdata.rdf.RdfKeyBuilder;
  import com.bigdata.rdf.model.OptimizedValueFactory._BNode;

Index: StatementSerializer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/serializers/StatementSerializer.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** StatementSerializer.java	12 Apr 2007 23:59:21 -0000	1.2
--- StatementSerializer.java	13 Apr 2007 15:02:33 -0000	1.3
***************
*** 47,52 ****
  import java.io.IOException;
  
! import com.bigdata.objndx.DataOutputBuffer;
! import com.bigdata.objndx.IValueSerializer;
  
  /**
--- 47,52 ----
  import java.io.IOException;
  
! import com.bigdata.btree.DataOutputBuffer;
! import com.bigdata.btree.IValueSerializer;
  
  /**

[cweb-CVS] bigdata/src/java/com/bigdata/objndx PackedAddressSerializer.java, 1.3, 1.4 NodeSerializer.java, 1.35, 1.36 DataOutputBuffer.java, 1.1, 1.2 AddressSerializer.java, 1.2, 1.3 ByteArrayValueSerializer.java, 1.1, 1.2 IValueSerializer.java, 1.6, 1.7 KeyBufferSerializer.java, 1.2, 1.3 IAddressSerializer.java, 1.1, 1.2 IndexSegment.java, 1.20, 1.21 IKeySerializer.java, 1.7, 1.8

From: Bryan T. <tho...@us...> - 2007-04-12 23:59:53

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2548/src/java/com/bigdata/objndx

Modified Files:
	PackedAddressSerializer.java NodeSerializer.java 
	DataOutputBuffer.java AddressSerializer.java 
	ByteArrayValueSerializer.java IValueSerializer.java 
	KeyBufferSerializer.java IAddressSerializer.java 
	IndexSegment.java IKeySerializer.java 
Log Message:
Added a Sesame 1.x SAIL implementation.  This is NOT intended for production use.  It is just being done to gain a high-level query language integration for the triple store.

Index: IAddressSerializer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/IAddressSerializer.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** IAddressSerializer.java	26 Dec 2006 16:16:14 -0000	1.1
--- IAddressSerializer.java	12 Apr 2007 23:59:35 -0000	1.2
***************
*** 48,53 ****
  package com.bigdata.objndx;
  
! import java.io.DataInputStream;
! import java.io.DataOutputStream;
  import java.io.IOException;
  
--- 48,53 ----
  package com.bigdata.objndx;
  
! import java.io.DataInput;
! import java.io.DataOutput;
  import java.io.IOException;
  
***************
*** 85,89 ****
       *            written on the array.
       */
!     public void getChildAddresses(DataInputStream is, long[] childAddr,
              int nchildren) throws IOException;
  
--- 85,89 ----
       *            written on the array.
       */
!     public void getChildAddresses(DataInput is, long[] childAddr,
              int nchildren) throws IOException;
  
***************
*** 99,103 ****
       *            [0:n-1] are defined and must be written.
       */
!     public void putChildAddresses(DataOutputStream os, long[] childAddr,
              int nchildren) throws IOException;
  
--- 99,103 ----
       *            [0:n-1] are defined and must be written.
       */
!     public void putChildAddresses(DataOutputBuffer os, long[] childAddr,
              int nchildren) throws IOException;
  

Index: IValueSerializer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/IValueSerializer.java,v
retrieving revision 1.6
retrieving revision 1.7
diff -C2 -d -r1.6 -r1.7
*** IValueSerializer.java	13 Feb 2007 23:01:02 -0000	1.6
--- IValueSerializer.java	12 Apr 2007 23:59:34 -0000	1.7
***************
*** 48,53 ****
  package com.bigdata.objndx;
  
! import java.io.DataInputStream;
! import java.io.DataOutputStream;
  import java.io.IOException;
  import java.io.Serializable;
--- 48,53 ----
  package com.bigdata.objndx;
  
! import java.io.DataInput;
! import java.io.DataOutput;
  import java.io.IOException;
  import java.io.Serializable;
***************
*** 74,78 ****
       *            written on the array.
       */
!     public void getValues(DataInputStream is, Object[] values, int nvals)
              throws IOException;
  
--- 74,78 ----
       *            written on the array.
       */
!     public void getValues(DataInput is, Object[] values, int nvals)
              throws IOException;
  
***************
*** 88,92 ****
       *            [0:n-1] are defined and must be written.
       */
!     public void putValues(DataOutputStream os, Object[] values, int nvals)
              throws IOException;
  
--- 88,92 ----
       *            [0:n-1] are defined and must be written.
       */
!     public void putValues(DataOutputBuffer os, Object[] values, int nvals)
              throws IOException;
  

Index: ByteArrayValueSerializer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/ByteArrayValueSerializer.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** ByteArrayValueSerializer.java	15 Feb 2007 01:34:22 -0000	1.1
--- ByteArrayValueSerializer.java	12 Apr 2007 23:59:34 -0000	1.2
***************
*** 48,52 ****
  
  import java.io.ByteArrayOutputStream;
! import java.io.DataInputStream;
  import java.io.DataOutputStream;
  import java.io.IOException;
--- 48,53 ----
  
  import java.io.ByteArrayOutputStream;
! import java.io.DataInput;
! import java.io.DataOutput;
  import java.io.DataOutputStream;
  import java.io.IOException;
***************
*** 74,78 ****
      public static final transient int VERSION0 = 0x0;
  
!     public void putValues(DataOutputStream os, Object[] values, int n) throws IOException {
  
          /*
--- 75,79 ----
      public static final transient int VERSION0 = 0x0;
  
!     public void putValues(DataOutputBuffer os, Object[] values, int n) throws IOException {
  
          /*
***************
*** 81,91 ****
          {
  
!             final int size = 2 + n * 2; // est of buffer capacity.
              
!             ByteArrayOutputStream baos = new ByteArrayOutputStream(size);
! 
!             DataOutputStream dbaos = new DataOutputStream(baos);
! 
!             LongPacker.packLong(dbaos,VERSION0);
              
              for (int i = 0; i < n; i++) {
--- 82,94 ----
          {
  
! //            final int size = 2 + n * 2; // est of buffer capacity.
! //            
! //            ByteArrayOutputStream baos = new ByteArrayOutputStream(size);
! //
! //            DataOutputStream dbaos = new DataOutputStream(baos);
! //
! //            LongPacker.packLong(dbaos,VERSION0);
              
!             os.packLong(VERSION0);
              
              for (int i = 0; i < n; i++) {
***************
*** 97,107 ****
                  // Note: we add (1) so that the length is always
                  // non-negative so that we can pack it.
!                 LongPacker.packLong(dbaos,len+1);
                  
              }
  
!             dbaos.flush();
!             
!             os.write(baos.toByteArray());
              
          }
--- 100,111 ----
                  // Note: we add (1) so that the length is always
                  // non-negative so that we can pack it.
! //                LongPacker.packLong(dbaos,len+1);
!                 os.packLong(len+1);
                  
              }
  
! //            dbaos.flush();
! //            
! //            os.write(baos.toByteArray());
              
          }
***************
*** 124,128 ****
      }
      
!     public void getValues(DataInputStream is, Object[] values, int n) throws IOException {
  
          final int version = (int)LongPacker.unpackLong(is);
--- 128,132 ----
      }
      
!     public void getValues(DataInput is, Object[] values, int n) throws IOException {
  
          final int version = (int)LongPacker.unpackLong(is);
***************
*** 159,163 ****
              if( value==null) continue;
              
!             is.read(value, 0, value.length);
              
          }
--- 163,167 ----
              if( value==null) continue;
              
!             is.readFully(value, 0, value.length);
              
          }

Index: DataOutputBuffer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/DataOutputBuffer.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** DataOutputBuffer.java	10 Apr 2007 18:33:31 -0000	1.1
--- DataOutputBuffer.java	12 Apr 2007 23:59:34 -0000	1.2
***************
*** 162,168 ****
  
      /**
!      * The #of bytes of data in the key.
       */
!     final public int getLength() {
          
          return len;
--- 162,168 ----
  
      /**
!      * The current position in the buffer.
       */
!     final public int position() {
          
          return len;
***************
*** 171,174 ****
--- 171,194 ----
  
      /**
+      * Set the position in the buffer.
+      * 
+      * @param pos
+      *            The new position, must be in [0:capacity).
+      * 
+      * @return The old position.
+      */
+     final public int position(int pos) {
+ 
+         if(pos<0 || pos>=buf.length) throw new IllegalArgumentException();
+         
+         int v = this.len;
+         
+         this.len = pos;
+         
+         return v;
+         
+     }
+     
+     /**
       * Ensure that at least <i>len</i> bytes are free in the buffer. The
       * {@link #buf buffer} may be grown by this operation but it will not be
***************
*** 234,238 ****
      protected int extend(int required) {
  
!         return Math.max(required, buf.length * 2);
          
      }
--- 254,263 ----
      protected int extend(int required) {
  
!         int capacity = Math.max(required, buf.length * 2);
! 
!         System.err.println("Extending buffer to capacity=" + capacity
!                 + " bytes.");
! 
!         return capacity;
          
      }
***************
*** 451,460 ****
      
      /**
!      * Note: This is not wildly efficient (it would be fine if
!      * DataOutputStream#writeUTF(String str, DataOutput out)} was public) but
!      * the use cases for serializing the nodes and leaves of a btree do not
!      * suggest any requirement for Unicode (if you assume that the application
!      * values are already being serialized as byte[]s - which is always true
!      * when there is a client-server divide).
       */
      public void writeUTF(String str) throws IOException {
--- 476,488 ----
      
      /**
!      * @todo This is not wildly efficient (it would be fine if
!      *       DataOutputStream#writeUTF(String str, DataOutput out)} was public)
!      *       but the use cases for serializing the nodes and leaves of a btree
!      *       do not suggest any requirement for Unicode (if you assume that the
!      *       application values are already being serialized as byte[]s - which
!      *       is always true when there is a client-server divide). The RDF value
!      *       serializer does use this method right now, but that will be client
!      *       side code as soon we as refactor to isolate the client and the
!      *       server.
       */
      public void writeUTF(String str) throws IOException {

Index: IndexSegment.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/IndexSegment.java,v
retrieving revision 1.20
retrieving revision 1.21
diff -C2 -d -r1.20 -r1.21
*** IndexSegment.java	10 Apr 2007 18:33:31 -0000	1.20
--- IndexSegment.java	12 Apr 2007 23:59:35 -0000	1.21
***************
*** 1,6 ****
  package com.bigdata.objndx;
  
! import java.io.DataInputStream;
! import java.io.DataOutputStream;
  import java.io.IOException;
  
--- 1,6 ----
  package com.bigdata.objndx;
  
! import java.io.DataInput;
! import java.io.DataOutput;
  import java.io.IOException;
  
***************
*** 606,610 ****
           * to the conventions of this class.
           */
!         public void putChildAddresses(DataOutputStream os, long[] childAddr,
                  int nchildren) throws IOException {
  
--- 606,610 ----
           * to the conventions of this class.
           */
!         public void putChildAddresses(DataOutputBuffer os, long[] childAddr,
                  int nchildren) throws IOException {
  
***************
*** 637,644 ****
  
                  // write the adjusted offset (requires decoding).
!                 LongPacker.packLong(os, adjustedOffset);
  
                  // write the #of bytes (does not require decoding).
!                 LongPacker.packLong(os, nbytes);
  
              }
--- 637,646 ----
  
                  // write the adjusted offset (requires decoding).
! //                LongPacker.packLong(os, adjustedOffset);
!                 os.packLong(adjustedOffset);
  
                  // write the #of bytes (does not require decoding).
! //                LongPacker.packLong(os, nbytes);
!                 os.packLong(nbytes);
  
              }
***************
*** 649,653 ****
           * Unpacks and decodes the addresses.
           */
!         public void getChildAddresses(DataInputStream is, long[] childAddr,
                  int nchildren) throws IOException {
  
--- 651,655 ----
           * Unpacks and decodes the addresses.
           */
!         public void getChildAddresses(DataInput is, long[] childAddr,
                  int nchildren) throws IOException {
  

Index: IKeySerializer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/IKeySerializer.java,v
retrieving revision 1.7
retrieving revision 1.8
diff -C2 -d -r1.7 -r1.8
*** IKeySerializer.java	26 Jan 2007 02:39:24 -0000	1.7
--- IKeySerializer.java	12 Apr 2007 23:59:35 -0000	1.8
***************
*** 48,53 ****
  package com.bigdata.objndx;
  
! import java.io.DataInputStream;
! import java.io.DataOutputStream;
  import java.io.IOException;
  import java.io.Serializable;
--- 48,53 ----
  package com.bigdata.objndx;
  
! import java.io.DataInput;
! import java.io.DataOutput;
  import java.io.IOException;
  import java.io.Serializable;
***************
*** 69,73 ****
       * @return The keys.
       */
!     public IKeyBuffer getKeys(DataInputStream is) throws IOException;
  
      /**
--- 69,73 ----
       * @return The keys.
       */
!     public IKeyBuffer getKeys(DataInput is) throws IOException;
  
      /**
***************
*** 80,84 ****
       *            The keys from a {@link Leaf} or {@link Node}.
       */
!     public void putKeys(DataOutputStream os, IKeyBuffer keys)
              throws IOException;
  
--- 80,84 ----
       *            The keys from a {@link Leaf} or {@link Node}.
       */
!     public void putKeys(DataOutputBuffer os, IKeyBuffer keys)
              throws IOException;
  

Index: AddressSerializer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/AddressSerializer.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** AddressSerializer.java	5 Feb 2007 18:17:39 -0000	1.2
--- AddressSerializer.java	12 Apr 2007 23:59:34 -0000	1.3
***************
*** 48,53 ****
  package com.bigdata.objndx;
  
! import java.io.DataInputStream;
! import java.io.DataOutputStream;
  import java.io.IOException;
  
--- 48,53 ----
  package com.bigdata.objndx;
  
! import java.io.DataInput;
! import java.io.DataOutput;
  import java.io.IOException;
  
***************
*** 57,60 ****
--- 57,62 ----
   * Serializes each address as an unpacked long integer.
   * 
+  * @see PackedAddressSerializer
+  * 
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
   * @version $Id$
***************
*** 75,79 ****
      }
  
!     public void putChildAddresses(DataOutputStream os, long[] childAddr,
              int nchildren) throws IOException {
  
--- 77,81 ----
      }
  
!     public void putChildAddresses(DataOutputBuffer os, long[] childAddr,
              int nchildren) throws IOException {
  
***************
*** 98,102 ****
      }
  
!     public void getChildAddresses(DataInputStream is, long[] childAddr,
              int nchildren) throws IOException {
  
--- 100,104 ----
      }
  
!     public void getChildAddresses(DataInput is, long[] childAddr,
              int nchildren) throws IOException {
  

Index: NodeSerializer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/NodeSerializer.java,v
retrieving revision 1.35
retrieving revision 1.36
diff -C2 -d -r1.35 -r1.36
*** NodeSerializer.java	10 Apr 2007 18:33:31 -0000	1.35
--- NodeSerializer.java	12 Apr 2007 23:59:34 -0000	1.36
***************
*** 47,51 ****
  package com.bigdata.objndx;
  
- import java.io.ByteArrayOutputStream;
  import java.io.DataInputStream;
  import java.io.DataOutput;
--- 47,50 ----
***************
*** 103,106 ****
--- 102,107 ----
   * @version $Id$
   * 
+  * @todo modify deserialization to use a fast DataInput wrapping a byte[]?
+  * 
   * @todo automatically resize the decompression buffers as required and start
   *       with a smaller buffer.
***************
*** 176,180 ****
       * retried.
       */
!     protected ByteBuffer _buf;
      
      /**
--- 177,181 ----
       * retried.
       */
!     protected DataOutputBuffer _buf;
      
      /**
***************
*** 472,476 ****
       * @return The buffer.
       */
!     static protected ByteBuffer alloc(int capacity) {
          
  //        return (true || capacity < Bytes.kilobyte32 * 8 )? ByteBuffer
--- 473,477 ----
       * @return The buffer.
       */
!     static protected DataOutputBuffer alloc(int capacity) {
          
  //        return (true || capacity < Bytes.kilobyte32 * 8 )? ByteBuffer
***************
*** 490,523 ****
           */
  
!         return ByteBuffer.allocate(capacity);
          
!     }
! 
!     /**
!      * Extends the internal buffer used to serialize nodes and leaves.
!      * <p>
!      * Note: large buffer requirements are not at all uncommon so we grow the
!      * buffer rapidly to avoid multiple resizing and the expense of a too large
!      * buffer.
!      * 
!      * FIXME We can encapsulate the extension of the buffer within a class
!      * derived from or using the {@link ByteBufferOutputStream} and simply copy
!      * the data when we need to extend the buffer rather than restarting
!      * serialization. This will make underestimates of the required buffer
!      * capacity much less costly.
!      */
!     protected void extendBuffer() {
! 
!         int capacity = _buf.capacity();
          
-         capacity *= 2;
- 
-         System.err.println("Extending buffer to capacity=" + capacity
-                 + " bytes.");
- 
-         _buf = alloc(capacity);
- 
      }
  
      /**
       * De-serialize a node or leaf. This method is used when the caller does not
--- 491,526 ----
           */
  
! //        return ByteBuffer.allocate(capacity);
          
!         return new DataOutputBuffer(capacity);
          
      }
  
+ //    /**
+ //     * Extends the internal buffer used to serialize nodes and leaves.
+ //     * <p>
+ //     * Note: large buffer requirements are not at all uncommon so we grow the
+ //     * buffer rapidly to avoid multiple resizing and the expense of a too large
+ //     * buffer.
+ //     * 
+ //     * FIXME We can encapsulate the extension of the buffer within a class
+ //     * derived from or using the {@link ByteBufferOutputStream} and simply copy
+ //     * the data when we need to extend the buffer rather than restarting
+ //     * serialization. This will make underestimates of the required buffer
+ //     * capacity much less costly.
+ //     */
+ //    protected void extendBuffer() {
+ //
+ //        int capacity = _buf.capacity();
+ //        
+ //        capacity *= 2;
+ //
+ //        System.err.println("Extending buffer to capacity=" + capacity
+ //                + " bytes.");
+ //
+ //        _buf = alloc(capacity);
+ //
+ //    }
+ 
      /**
       * De-serialize a node or leaf. This method is used when the caller does not
***************
*** 658,678 ****
          }
          
!         while (true) {
! 
              try {
  
                  return putNode(_buf, node);
! 
              } catch (BufferOverflowException ex) {
  
!                 extendBuffer();
  
              }
! 
!         }
  
      }
      
!     private ByteBuffer putNode(ByteBuffer buf, INodeData node) {
  
          assert buf != null;
--- 661,686 ----
          }
          
! //        while (true) {
! //
              try {
  
                  return putNode(_buf, node);
! //
!             } catch (IOException ex) {
!                 
!                 throw new RuntimeException(ex); // exception is not expected.
!                 
              } catch (BufferOverflowException ex) {
  
!                 throw ex; // exception is not expected.
! //                extendBuffer();
  
              }
! //
! //        }
  
      }
      
!     private ByteBuffer putNode(DataOutputBuffer buf, INodeData node) throws IOException {
  
          assert buf != null;
***************
*** 690,727 ****
           */
  
!         buf.clear();
          
          final int pos0 = buf.position();
  
          // checksum
!         buf.putInt(0); // will overwrite below with the checksum.
  
          // #bytes
!         buf.putInt(0); // will overwrite below with the actual value.
          
          // nodeType
!         buf.put(TYPE_NODE); // this is a non-leaf node.
  
          // version
!         buf.putShort(VERSION0);
          
!         /*
!          * Setup output stream over the buffer.
!          * 
!          * Note: I have tested the use of a {@link BufferedOutputStream} here
!          * and in putLeaf() and it actually slows things down a smidge.
!          */
!         DataOutputStream os = new DataOutputStream(//
!                 new ByteBufferOutputStream(buf)
! //              new BufferedOutputStream(new ByteBufferOutputStream(buf))
!                 );
          
          try {
  
              // branching factor.
!             LongPacker.packLong(os, branchingFactor);
  
              // #of spanned entries.
!             LongPacker.packLong(os, nentries);
              
  //            // #of keys
--- 698,735 ----
           */
  
!         buf.reset();
          
          final int pos0 = buf.position();
  
          // checksum
!         buf.writeInt(0); // will overwrite below with the checksum.
  
          // #bytes
!         buf.writeInt(0); // will overwrite below with the actual value.
          
          // nodeType
!         buf.writeByte(TYPE_NODE); // this is a non-leaf node.
  
          // version
!         buf.writeShort(VERSION0);
          
! //        /*
! //         * Setup output stream over the buffer.
! //         * 
! //         * Note: I have tested the use of a {@link BufferedOutputStream} here
! //         * and in putLeaf() and it actually slows things down a smidge.
! //         */
! //        DataOutputStream os = new DataOutputStream(//
! //                new ByteBufferOutputStream(buf)
! ////              new BufferedOutputStream(new ByteBufferOutputStream(buf))
! //                );
          
          try {
  
              // branching factor.
!             buf.packLong( branchingFactor);
  
              // #of spanned entries.
!             buf.packLong( nentries);
              
  //            // #of keys
***************
*** 729,742 ****
  
              // keys.
!             keySerializer.putKeys(os, keys);
  
              // addresses.
!             addrSerializer.putChildAddresses(os, childAddr, nkeys+1);
  
              // #of entries spanned per child.
!             putChildEntryCounts(os,childEntryCounts,nkeys+1);
              
!             // Done using the DataOutputStream so flush to the ByteBuffer.
!             os.flush();
  
          }
--- 737,750 ----
  
              // keys.
!             keySerializer.putKeys(buf, keys);
  
              // addresses.
!             addrSerializer.putChildAddresses(buf, childAddr, nkeys+1);
  
              // #of entries spanned per child.
!             putChildEntryCounts(buf,childEntryCounts,nkeys+1);
              
! //            // Done using the DataOutputStream so flush to the ByteBuffer.
! //            os.flush();
  
          }
***************
*** 746,750 ****
              /*
               * Masquerade the EOFException as a buffer overflow since that is
!              * what it really represents.
               */
              RuntimeException ex2 = new BufferOverflowException();
--- 754,760 ----
              /*
               * Masquerade the EOFException as a buffer overflow since that is
!              * what it really represents (@todo since ByteBuffer is not used
!              * anymore we do not need to masquerade this and the javadoc should
!              * be updated).
               */
              RuntimeException ex2 = new BufferOverflowException();
***************
*** 770,779 ****
          final int nbytes = buf.position() - pos0;
          assert nbytes > SIZEOF_NODE_HEADER;
  
          // patch #of bytes written on the record format.
!         buf.putInt(pos0 + OFFSET_NBYTES, nbytes);
  
          // compute checksum for data written.
!         final int checksum = useChecksum ? chk.checksum(buf, pos0
                  + SIZEOF_CHECKSUM, pos0 + nbytes) : 0;
  
--- 780,791 ----
          final int nbytes = buf.position() - pos0;
          assert nbytes > SIZEOF_NODE_HEADER;
+         
+         ByteBuffer buf2 = ByteBuffer.wrap(buf.buf,0,nbytes);
  
          // patch #of bytes written on the record format.
!         buf2.putInt(pos0 + OFFSET_NBYTES, nbytes);
  
          // compute checksum for data written.
!         final int checksum = useChecksum ? chk.checksum(buf2, pos0
                  + SIZEOF_CHECKSUM, pos0 + nbytes) : 0;
  
***************
*** 781,791 ****
  
          // write the checksum into the buffer.
!         buf.putInt(pos0, checksum);
  
!         // flip the buffer to prepare for reading.
!         buf.flip();
  
          // optionally compresses the record.
!         return compress( buf );
                  
      }
--- 793,808 ----
  
          // write the checksum into the buffer.
!         buf2.putInt(pos0, checksum);
  
!         /*
!          * Note: The position will be zero(0). The limit will be the #of bytes
!          * in the buffer.
!          */
!         
! //        // flip the buffer to prepare for reading.
! //        buf2.flip();
  
          // optionally compresses the record.
!         return compress( buf2 );
                  
      }
***************
*** 974,978 ****
          }
          
!         while (true) {
  
              try {
--- 991,995 ----
          }
          
! //        while (true) {
  
              try {
***************
*** 980,994 ****
                  return putLeaf(_buf,leaf);
  
              } catch (BufferOverflowException ex) {
  
!                 extendBuffer();
  
              }
  
!         }
          
      }
       
!     private ByteBuffer putLeaf(ByteBuffer buf, ILeafData leaf) {
  
          assert buf != null;
--- 997,1017 ----
                  return putLeaf(_buf,leaf);
  
+             } catch (IOException ex) {
+ 
+                 throw new RuntimeException(ex); // exception is not expected.
+                 
              } catch (BufferOverflowException ex) {
  
!                 throw ex; // exception is not expected.
!                 
! //                extendBuffer();
  
              }
  
! //        }
          
      }
       
!     private ByteBuffer putLeaf(DataOutputBuffer buf, ILeafData leaf) throws IOException {
  
          assert buf != null;
***************
*** 1000,1004 ****
          final Object[] vals = leaf.getValues();
          
!         buf.clear();
          
          /*
--- 1023,1027 ----
          final Object[] vals = leaf.getValues();
          
!         buf.reset();
          
          /*
***************
*** 1008,1021 ****
  
          // checksum
!         buf.putInt(0); // will overwrite below with the checksum.
          
          // nbytes
!         buf.putInt(0); // will overwrite below with the actual value.
          
          // nodeType
!         buf.put(TYPE_LEAF); // this is a leaf node.
  
          // version
!         buf.putShort(VERSION0);
          
          /*
--- 1031,1044 ----
  
          // checksum
!         buf.writeInt(0); // will overwrite below with the checksum.
          
          // nbytes
!         buf.writeInt(0); // will overwrite below with the actual value.
          
          // nodeType
!         buf.writeByte(TYPE_LEAF); // this is a leaf node.
  
          // version
!         buf.writeShort(VERSION0);
          
          /*
***************
*** 1024,1036 ****
           * Note: wrapping this with a BufferedOutputStream is slightly slower.
           */
!         DataOutputStream os = new DataOutputStream(//
!                 new ByteBufferOutputStream(buf)
! //                new BufferedOutputStream(new ByteBufferOutputStream(buf))
!                 );
  
          try {
              
              // branching factor.
!             LongPacker.packLong(os, branchingFactor);
  
  //            // #of keys
--- 1047,1059 ----
           * Note: wrapping this with a BufferedOutputStream is slightly slower.
           */
! //        DataOutputStream os = new DataOutputStream(//
! //                new ByteBufferOutputStream(buf)
! ////                new BufferedOutputStream(new ByteBufferOutputStream(buf))
! //                );
  
          try {
              
              // branching factor.
!             buf.packLong( branchingFactor);
  
  //            // #of keys
***************
*** 1038,1048 ****
  
              // keys.
!             keySerializer.putKeys(os, keys);
  
              // values.
!             valueSerializer.putValues(os, vals, nkeys);
  
!             // Done using the DataOutputStream so flush to the ByteBuffer.
!             os.flush();
              
          }
--- 1061,1071 ----
  
              // keys.
!             keySerializer.putKeys(buf, keys);
  
              // values.
!             valueSerializer.putValues(buf, vals, nkeys);
  
! //            // Done using the DataOutputStream so flush to the ByteBuffer.
! //            os.flush();
              
          }
***************
*** 1052,1056 ****
              /*
               * Masquerade the EOFException as a buffer overflow since that is
!              * what it really represents.
               */
              RuntimeException ex2 = new BufferOverflowException();
--- 1075,1080 ----
              /*
               * Masquerade the EOFException as a buffer overflow since that is
!              * what it really represents (@todo we do not need to masquerade
!              * this exception since we are not using ByteBuffer anymore).
               */
              RuntimeException ex2 = new BufferOverflowException();
***************
*** 1077,1099 ****
          assert nbytes > SIZEOF_LEAF_HEADER;
  
          // patch #of bytes written on the record format.
!         buf.putInt(pos0 + OFFSET_NBYTES, nbytes);
  
!         // compute checksum
!         final int checksum = (useChecksum ? chk.checksum(buf, pos0
                  + SIZEOF_CHECKSUM, pos0 + nbytes) : 0);
          // System.err.println("computed leaf checksum: "+checksum);
  
          // write checksum on buffer.
!         buf.putInt(pos0, checksum);
! 
          /*
!          * Flip the buffer to prepare it for reading. The position will be zero
!          * and the limit will be the #of bytes in the serialized record.
           */
!         buf.flip();
          
          // optionally compresses the record.
!         return compress( buf );
                  
      }
--- 1101,1130 ----
          assert nbytes > SIZEOF_LEAF_HEADER;
  
+         ByteBuffer buf2 = ByteBuffer.wrap(buf.buf,0,nbytes);
+         
          // patch #of bytes written on the record format.
!         buf2.putInt(pos0 + OFFSET_NBYTES, nbytes);
  
!         // compute checksum.
!         final int checksum = (useChecksum ? chk.checksum(buf2, pos0
                  + SIZEOF_CHECKSUM, pos0 + nbytes) : 0);
          // System.err.println("computed leaf checksum: "+checksum);
  
          // write checksum on buffer.
!         buf2.putInt(pos0, checksum);
!         
          /*
!          * Note: The position will be zero(0).  The limit will be the #of bytes
!          * in the buffer.
           */
! 
! //        /*
! //         * Flip the buffer to prepare it for reading. The position will be zero
! //         * and the limit will be the #of bytes in the serialized record.
! //         */
! //        buf2.flip();
          
          // optionally compresses the record.
!         return compress( buf2 );
                  
      }
***************
*** 1254,1257 ****
--- 1285,1289 ----
       * 
       * @param os
+      *            The output stream.
       * @param childEntryCounts
       *            The #of entries spanned by each direct child.
***************
*** 1260,1264 ****
       * @throws IOException
       */
!     protected void putChildEntryCounts(DataOutputStream os,
              int[] childEntryCounts, int nchildren) throws IOException {
  
--- 1292,1296 ----
       * @throws IOException
       */
!     protected void putChildEntryCounts(DataOutput os,
              int[] childEntryCounts, int nchildren) throws IOException {
  

Index: PackedAddressSerializer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/PackedAddressSerializer.java,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** PackedAddressSerializer.java	5 Feb 2007 18:17:38 -0000	1.3
--- PackedAddressSerializer.java	12 Apr 2007 23:59:34 -0000	1.4
***************
*** 48,53 ****
  package com.bigdata.objndx;
  
! import java.io.DataInputStream;
! import java.io.DataOutputStream;
  import java.io.IOException;
  
--- 48,53 ----
  package com.bigdata.objndx;
  
! import java.io.DataInput;
! import java.io.DataOutput;
  import java.io.IOException;
  
***************
*** 78,82 ****
      }
  
!     public void putChildAddresses(DataOutputStream os, long[] childAddr,
              int nchildren) throws IOException {
  
--- 78,82 ----
      }
  
!     public void putChildAddresses(DataOutputBuffer os, long[] childAddr,
              int nchildren) throws IOException {
  
***************
*** 101,105 ****
      }
  
!     public void getChildAddresses(DataInputStream is, long[] childAddr,
              int nchildren) throws IOException {
  
--- 101,105 ----
      }
  
!     public void getChildAddresses(DataInput is, long[] childAddr,
              int nchildren) throws IOException {
  

Index: KeyBufferSerializer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/KeyBufferSerializer.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** KeyBufferSerializer.java	1 Feb 2007 16:00:36 -0000	1.2
--- KeyBufferSerializer.java	12 Apr 2007 23:59:34 -0000	1.3
***************
*** 2,10 ****
  
  import java.io.ByteArrayOutputStream;
! import java.io.DataInputStream;
  import java.io.DataOutputStream;
  import java.io.IOException;
  
  import org.CognitiveWeb.extser.LongPacker;
  
  /**
--- 2,12 ----
  
  import java.io.ByteArrayOutputStream;
! import java.io.DataInput;
! import java.io.DataOutput;
  import java.io.DataOutputStream;
  import java.io.IOException;
  
  import org.CognitiveWeb.extser.LongPacker;
+ import org.CognitiveWeb.extser.ShortPacker;
  
  /**
***************
*** 25,35 ****
      private static final long serialVersionUID = 7361581167520945586L;
  
!     public static final transient int VERSION0 = 0x0;
      
      public static final transient IKeySerializer INSTANCE = new KeyBufferSerializer();
      
!     public IKeyBuffer getKeys(DataInputStream is) throws IOException {
  
!         final int version = is.readInt();
          
          if (version != VERSION0)
--- 27,37 ----
      private static final long serialVersionUID = 7361581167520945586L;
  
!     public static final transient short VERSION0 = 0x0;
      
      public static final transient IKeySerializer INSTANCE = new KeyBufferSerializer();
      
!     public IKeyBuffer getKeys(DataInput is) throws IOException {
  
!         final short version = ShortPacker.unpackShort(is);
          
          if (version != VERSION0)
***************
*** 72,78 ****
      }
  
!     public void putKeys(DataOutputStream os, IKeyBuffer keys) throws IOException {
  
!         os.writeInt(VERSION0);
          
          if(keys instanceof ImmutableKeyBuffer ) {
--- 74,80 ----
      }
  
!     public void putKeys(DataOutputBuffer os, IKeyBuffer keys) throws IOException {
  
!         os.packShort(VERSION0);
          
          if(keys instanceof ImmutableKeyBuffer ) {
***************
*** 88,92 ****
      }
      
!     protected void putKeys2(DataOutputStream os, ImmutableKeyBuffer keys) throws IOException {
          
          final int nkeys = keys.nkeys;
--- 90,94 ----
      }
      
!     protected void putKeys2(DataOutputBuffer os, ImmutableKeyBuffer keys) throws IOException {
          
          final int nkeys = keys.nkeys;
***************
*** 95,105 ****
  
          // #of keys in the node or leaf.
!         LongPacker.packLong(os, nkeys);
  
          // maximum #of keys allowed in the node or leaf.
!         LongPacker.packLong(os, keys.maxKeys);
  
          // length of the byte[] buffer containing the prefix and remainder for each key.
!         LongPacker.packLong(os, bufferLength);
          
          /*
--- 97,110 ----
  
          // #of keys in the node or leaf.
! //        LongPacker.packLong(os, nkeys);
!         os.packLong(nkeys);
  
          // maximum #of keys allowed in the node or leaf.
! //        LongPacker.packLong(os, keys.maxKeys);
!         os.packLong(keys.maxKeys);       
  
          // length of the byte[] buffer containing the prefix and remainder for each key.
! //        LongPacker.packLong(os, bufferLength);
!         os.packLong(bufferLength);
          
          /*
***************
*** 114,118 ****
              int delta = offset - lastOffset;
              
!             LongPacker.packLong(os, delta);
              
              lastOffset = offset;
--- 119,124 ----
              int delta = offset - lastOffset;
              
! //            LongPacker.packLong(os, delta);
!             os.packLong(delta);
              
              lastOffset = offset;
***************
*** 120,124 ****
          }
          
!         os.write(keys.buf);
          
      }
--- 126,131 ----
          }
          
! //        os.write(keys.buf);
!         os.write(keys.buf, 0, bufferLength);
          
      }
***************
*** 134,138 ****
       * @throws IOException
       */
!     protected void putKeys2(DataOutputStream os, MutableKeyBuffer keys) throws IOException {
          
          final int nkeys = keys.nkeys;
--- 141,145 ----
       * @throws IOException
       */
!     protected void putKeys2(DataOutputBuffer os, MutableKeyBuffer keys) throws IOException {
          
          final int nkeys = keys.nkeys;
***************
*** 160,175 ****
          
          // #of keys in the node or leaf.
!         LongPacker.packLong(os, nkeys);
  
          // maximum #of keys allowed in the node or leaf.
!         LongPacker.packLong(os, keys.getMaxKeys());
  
          // length of the byte[] buffer containing the prefix and remainder for each key.
!         LongPacker.packLong(os, bufferLength);
          
          /*
           * Write out deltas between offsets.
           * 
!          * FIXME this is 60% of the cost of this method. This is not pack long
           * so much as doing individual byte put operations on the output stream
           * (which is over a ByteBuffer).  Just using a BAOS here doubles the 
--- 167,185 ----
          
          // #of keys in the node or leaf.
! //        LongPacker.packLong(os, nkeys);
!         os.packLong(nkeys);
  
          // maximum #of keys allowed in the node or leaf.
! //        LongPacker.packLong(os, keys.getMaxKeys());
!         os.packLong(keys.getMaxKeys());
  
          // length of the byte[] buffer containing the prefix and remainder for each key.
! //        LongPacker.packLong(os, bufferLength);
!         os.packLong(bufferLength);
          
          /*
           * Write out deltas between offsets.
           * 
!          * Note: this is 60% of the cost of this method. This is not pack long
           * so much as doing individual byte put operations on the output stream
           * (which is over a ByteBuffer).  Just using a BAOS here doubles the 
***************
*** 177,182 ****
           */
          {
!             ByteArrayOutputStream baos = new ByteArrayOutputStream(nkeys*8);
!             DataOutputStream dbaos = new DataOutputStream(baos);
  
              int lastOffset = 0;
--- 187,192 ----
           */
          {
! //            ByteArrayOutputStream baos = new ByteArrayOutputStream(nkeys*8);
! //            DataOutputStream dbaos = new DataOutputStream(baos);
  
              int lastOffset = 0;
***************
*** 188,192 ****
              int delta = offset - lastOffset;
              
!             LongPacker.packLong(dbaos, delta);
              
              lastOffset = offset;
--- 198,203 ----
              int delta = offset - lastOffset;
              
! //            LongPacker.packLong(dbaos, delta);
!             os.packLong(delta);
              
              lastOffset = offset;
***************
*** 194,200 ****
          }
          
!         dbaos.flush();
!         
!         os.write(baos.toByteArray());
          }
          
--- 205,211 ----
          }
          
! //        dbaos.flush();
! //        
! //        os.write(baos.toByteArray());
          }

[cweb-CVS] bigdata/src/java/com/bigdata/scaleup PartitionMetadata.java, 1.5, 1.6

From: Bryan T. <tho...@us...> - 2007-04-12 23:59:52

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2548/src/java/com/bigdata/scaleup

Modified Files:
	PartitionMetadata.java 
Log Message:
Added a Sesame 1.x SAIL implementation.  This is NOT intended for production use.  It is just being done to gain a high-level query language integration for the triple store.

Index: PartitionMetadata.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/PartitionMetadata.java,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** PartitionMetadata.java	29 Mar 2007 17:01:33 -0000	1.5
--- PartitionMetadata.java	12 Apr 2007 23:59:35 -0000	1.6
***************
*** 44,49 ****
  package com.bigdata.scaleup;
  
! import java.io.DataInputStream;
! import java.io.DataOutputStream;
  import java.io.Externalizable;
  import java.io.IOException;
--- 44,49 ----
  package com.bigdata.scaleup;
  
! import java.io.DataInput;
! import java.io.DataOutput;
  import java.io.Externalizable;
  import java.io.IOException;
***************
*** 52,55 ****
--- 52,56 ----
  import com.bigdata.isolation.UnisolatedBTree;
  import com.bigdata.journal.Journal;
+ import com.bigdata.objndx.DataOutputBuffer;
  import com.bigdata.objndx.IValueSerializer;
  import com.bigdata.objndx.IndexSegment;
***************
*** 270,274 ****
          }
  
!         public void putValues(DataOutputStream os, Object[] values, int nvals)
                  throws IOException {
              
--- 271,275 ----
          }
  
!         public void putValues(DataOutputBuffer os, Object[] values, int nvals)
                  throws IOException {
              
***************
*** 321,325 ****
          }
  
!         public void getValues(DataInputStream is, Object[] values, int nvals)
                  throws IOException {
  
--- 322,326 ----
          }
  
!         public void getValues(DataInput is, Object[] values, int nvals)
                  throws IOException {

[cweb-CVS] bigdata/src/architecture performance.xls,1.2,1.3

From: Bryan T. <tho...@us...> - 2007-04-12 23:59:52

Update of /cvsroot/cweb/bigdata/src/architecture
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2548/src/architecture

Modified Files:
	performance.xls 
Log Message:
Added a Sesame 1.x SAIL implementation.  This is NOT intended for production use.  It is just being done to gain a high-level query language integration for the triple store.

Index: performance.xls
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/architecture/performance.xls,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
Binary files /tmp/cvsns4fgL and /tmp/cvsU71Oiz differ

[cweb-CVS] bigdata/src/test/com/bigdata/isolation TestValueSerializer.java, 1.2, 1.3

From: Bryan T. <tho...@us...> - 2007-04-12 23:59:52

Update of /cvsroot/cweb/bigdata/src/test/com/bigdata/isolation
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2548/src/test/com/bigdata/isolation

Modified Files:
	TestValueSerializer.java 
Log Message:
Added a Sesame 1.x SAIL implementation.  This is NOT intended for production use.  It is just being done to gain a high-level query language integration for the triple store.

Index: TestValueSerializer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/isolation/TestValueSerializer.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** TestValueSerializer.java	15 Feb 2007 01:34:23 -0000	1.2
--- TestValueSerializer.java	12 Apr 2007 23:59:35 -0000	1.3
***************
*** 49,55 ****
  
  import java.io.ByteArrayInputStream;
- import java.io.ByteArrayOutputStream;
  import java.io.DataInputStream;
- import java.io.DataOutputStream;
  import java.io.IOException;
  import java.util.Random;
--- 49,53 ----
***************
*** 58,61 ****
--- 56,61 ----
  import junit.framework.TestCase2;
  
+ import com.bigdata.objndx.DataOutputBuffer;
+ 
  /**
   * Test suite for {@link Value.Serializer}.
***************
*** 184,196 ****
              final byte[] data;
              {
!             ByteArrayOutputStream baos = new ByteArrayOutputStream();
! 
!             DataOutputStream dos = new DataOutputStream(baos);
  
              ser.putValues(dos, values, values.length);
              
!             dos.flush();
              
!             data = baos.toByteArray();
              }
  
--- 184,198 ----
              final byte[] data;
              {
! //            ByteArrayOutputStream baos = new ByteArrayOutputStream();
! //
! //            DataOutputStream dos = new DataOutputStream(baos);
!                 
!                 DataOutputBuffer dos = new DataOutputBuffer();
  
              ser.putValues(dos, values, values.length);
              
! //            dos.flush();
              
!             data = dos.toByteArray();
              }

[cweb-CVS] bigdata/src/java/com/bigdata/isolation Value.java, 1.3, 1.4

From: Bryan T. <tho...@us...> - 2007-04-12 23:59:51

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/isolation
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2548/src/java/com/bigdata/isolation

Modified Files:
	Value.java 
Log Message:
Added a Sesame 1.x SAIL implementation.  This is NOT intended for production use.  It is just being done to gain a high-level query language integration for the triple store.

Index: Value.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/isolation/Value.java,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** Value.java	6 Mar 2007 20:38:05 -0000	1.3
--- Value.java	12 Apr 2007 23:59:34 -0000	1.4
***************
*** 44,50 ****
  package com.bigdata.isolation;
  
! import java.io.ByteArrayOutputStream;
! import java.io.DataInputStream;
! import java.io.DataOutputStream;
  import java.io.IOException;
  import java.util.Arrays;
--- 44,48 ----
  package com.bigdata.isolation;
  
! import java.io.DataInput;
  import java.io.IOException;
  import java.util.Arrays;
***************
*** 53,56 ****
--- 51,55 ----
  import org.CognitiveWeb.extser.ShortPacker;
  
+ import com.bigdata.objndx.DataOutputBuffer;
  import com.bigdata.objndx.IValueSerializer;
  
***************
*** 160,164 ****
          public Serializer() {}
  
!         public void getValues(DataInputStream is, Object[] values, int n)
          throws IOException {
  
--- 159,163 ----
          public Serializer() {}
  
!         public void getValues(DataInput is, Object[] values, int n)
          throws IOException {
  
***************
*** 206,210 ****
                  if( value.deleted || value.datum==null) continue;
                  
!                 is.read(value.datum, 0, value.datum.length);
                  
              }
--- 205,209 ----
                  if( value.deleted || value.datum==null) continue;
                  
!                 is.readFully(value.datum, 0, value.datum.length);
                  
              }
***************
*** 212,216 ****
          }
  
!         public void putValues(DataOutputStream os, Object[] values, int n)
                  throws IOException {
  
--- 211,215 ----
          }
  
!         public void putValues(DataOutputBuffer os, Object[] values, int n)
                  throws IOException {
  
***************
*** 223,233 ****
              {
  
!                 final int size = 2 + n * 2 + n * 4; // est of buffer capacity.
                  
!                 ByteArrayOutputStream baos = new ByteArrayOutputStream(size);
! 
!                 DataOutputStream dbaos = new DataOutputStream(baos);
! 
!                 LongPacker.packLong(dbaos, VERSION0);
  
                  for (int i = 0; i < n; i++) {
--- 222,234 ----
              {
  
! //                final int size = 2 + n * 2 + n * 4; // est of buffer capacity.
! //                
! //                ByteArrayOutputStream baos = new ByteArrayOutputStream(size);
! //
! //                DataOutputStream dbaos = new DataOutputStream(baos);
! //
! //                LongPacker.packLong(dbaos, VERSION0);
                  
!                 os.packLong(VERSION0);
  
                  for (int i = 0; i < n; i++) {
***************
*** 235,239 ****
                      Value value = (Value) values[i];
  
!                     ShortPacker.packShort(dbaos,value.versionCounter);
                      
                      final long len;
--- 236,241 ----
                      Value value = (Value) values[i];
  
! //                    ShortPacker.packShort(dbaos,value.versionCounter);
!                     os.packShort(value.versionCounter);
                      
                      final long len;
***************
*** 253,263 ****
                      // Note: we add (2) so that the length is always
                      // non-negative so that we can pack it.
!                     LongPacker.packLong(dbaos,len+2);
                      
                  }
  
!                 dbaos.flush();
!                 
!                 os.write(baos.toByteArray());
                  
              }
--- 255,266 ----
                      // Note: we add (2) so that the length is always
                      // non-negative so that we can pack it.
! //                    LongPacker.packLong(dbaos,len+2);
!                     os.packLong(len+2);
                      
                  }
  
! //                dbaos.flush();
! //                
! //                os.write(baos.toByteArray());
                  
              }

[cweb-CVS] bigdata/src/java/com/bigdata/journal Name2Addr.java, 1.8, 1.9 CommitRecordIndex.java, 1.4, 1.5

From: Bryan T. <tho...@us...> - 2007-04-12 23:59:44

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/journal
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2548/src/java/com/bigdata/journal

Modified Files:
	Name2Addr.java CommitRecordIndex.java 
Log Message:
Added a Sesame 1.x SAIL implementation.  This is NOT intended for production use.  It is just being done to gain a high-level query language integration for the triple store.

Index: CommitRecordIndex.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/journal/CommitRecordIndex.java,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** CommitRecordIndex.java	27 Mar 2007 14:34:23 -0000	1.4
--- CommitRecordIndex.java	12 Apr 2007 23:59:34 -0000	1.5
***************
*** 1,10 ****
  package com.bigdata.journal;
  
! import java.io.DataInputStream;
! import java.io.DataOutputStream;
  import java.io.IOException;
  import java.util.UUID;
  
  import org.CognitiveWeb.extser.LongPacker;
  
  import com.bigdata.cache.LRUCache;
--- 1,10 ----
  package com.bigdata.journal;
  
! import java.io.DataInput;
  import java.io.IOException;
  import java.util.UUID;
  
  import org.CognitiveWeb.extser.LongPacker;
+ import org.CognitiveWeb.extser.ShortPacker;
  
  import com.bigdata.cache.LRUCache;
***************
*** 12,15 ****
--- 12,16 ----
  import com.bigdata.objndx.BTree;
  import com.bigdata.objndx.BTreeMetadata;
+ import com.bigdata.objndx.DataOutputBuffer;
  import com.bigdata.objndx.IValueSerializer;
  import com.bigdata.objndx.KeyBuilder;
***************
*** 365,377 ****
          public static transient final IValueSerializer INSTANCE = new ValueSerializer();
  
!         final public static transient int VERSION0 = 0x0;
  
          public ValueSerializer() {
          }
  
!         public void putValues(DataOutputStream os, Object[] values, int n)
                  throws IOException {
  
!             os.writeInt(VERSION0);
              
              for (int i = 0; i < n; i++) {
--- 366,378 ----
          public static transient final IValueSerializer INSTANCE = new ValueSerializer();
  
!         final public static transient short VERSION0 = 0x0;
  
          public ValueSerializer() {
          }
  
!         public void putValues(DataOutputBuffer os, Object[] values, int n)
                  throws IOException {
  
!             os.packShort(VERSION0);
              
              for (int i = 0; i < n; i++) {
***************
*** 379,384 ****
                  Entry entry = (Entry) values[i];
  
!                 LongPacker.packLong(os, entry.commitTime);
!                 LongPacker.packLong(os, entry.addr);
                  
              }
--- 380,387 ----
                  Entry entry = (Entry) values[i];
  
! //                LongPacker.packLong(os, entry.commitTime);
! //                LongPacker.packLong(os, entry.addr);
!                 os.packLong(entry.commitTime);
!                 os.packLong(entry.addr);
                  
              }
***************
*** 386,393 ****
          }
          
!         public void getValues(DataInputStream is, Object[] values, int n)
                  throws IOException {
  
!             final int version = is.readInt();
              
              if (version != VERSION0)
--- 389,396 ----
          }
          
!         public void getValues(DataInput is, Object[] values, int n)
                  throws IOException {
  
!             final short version = ShortPacker.unpackShort(is);
              
              if (version != VERSION0)

Index: Name2Addr.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/journal/Name2Addr.java,v
retrieving revision 1.8
retrieving revision 1.9
diff -C2 -d -r1.8 -r1.9
*** Name2Addr.java	27 Mar 2007 14:34:23 -0000	1.8
--- Name2Addr.java	12 Apr 2007 23:59:34 -0000	1.9
***************
*** 1,6 ****
  package com.bigdata.journal;
  
! import java.io.DataInputStream;
! import java.io.DataOutputStream;
  import java.io.IOException;
  import java.util.HashMap;
--- 1,6 ----
  package com.bigdata.journal;
  
! import java.io.DataInput;
! import java.io.DataOutput;
  import java.io.IOException;
  import java.util.HashMap;
***************
*** 11,17 ****
--- 11,19 ----
  
  import org.CognitiveWeb.extser.LongPacker;
+ import org.CognitiveWeb.extser.ShortPacker;
  
  import com.bigdata.objndx.BTree;
  import com.bigdata.objndx.BTreeMetadata;
+ import com.bigdata.objndx.DataOutputBuffer;
  import com.bigdata.objndx.IIndex;
  import com.bigdata.objndx.IValueSerializer;
***************
*** 359,363 ****
      /**
       * The values are {@link Entry}s.
!      * 
       * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
       * @version $Id$
--- 361,365 ----
      /**
       * The values are {@link Entry}s.
!      *
       * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
       * @version $Id$
***************
*** 369,384 ****
          public static transient final IValueSerializer INSTANCE = new ValueSerializer();
  
!         /**
!          * Note: It is faster to use packed longs, at least on write with
!          * test data (bulk load of wordnet nouns).
!          */
!         final static boolean packedLongs = true;
  
          public ValueSerializer() {
          }
  
!         public void putValues(DataOutputStream os, Object[] values, int n)
                  throws IOException {
  
              for (int i = 0; i < n; i++) {
  
--- 371,384 ----
          public static transient final IValueSerializer INSTANCE = new ValueSerializer();
  
!         final public static transient short VERSION0 = 0x0;
  
          public ValueSerializer() {
          }
  
!         public void putValues(DataOutputBuffer os, Object[] values, int n)
                  throws IOException {
  
+             os.packShort(VERSION0);
+ 
              for (int i = 0; i < n; i++) {
  
***************
*** 387,399 ****
                  os.writeUTF(entry.name);
  
!                 if (packedLongs) {
! 
!                     LongPacker.packLong(os, entry.addr);
! 
!                 } else {
! 
!                     os.writeLong(entry.addr);
  
!                 }
                  
              }
--- 387,401 ----
                  os.writeUTF(entry.name);
  
!                 Addr.pack(os, entry.addr);
  
! //                if (packedLongs) {
! //
! //                    LongPacker.packLong(os, entry.addr);
! //
! //                } else {
! //
! //                    os.writeLong(entry.addr);
! //
! //                }
                  
              }
***************
*** 401,407 ****
          }
          
!         public void getValues(DataInputStream is, Object[] values, int n)
                  throws IOException {
  
              for (int i = 0; i < n; i++) {
  
--- 403,414 ----
          }
          
!         public void getValues(DataInput is, Object[] values, int n)
                  throws IOException {
  
+             final short version = ShortPacker.unpackShort(is);
+             
+             if (version != VERSION0)
+                 throw new RuntimeException("Unknown version: " + version);
+ 
              for (int i = 0; i < n; i++) {
  
***************
*** 410,422 ****
                  final long addr;
                  
!                 if (packedLongs) {
! 
!                     addr = Long.valueOf(LongPacker.unpackLong(is));
! 
!                 } else {
! 
!                     addr = Long.valueOf(is.readLong());
! 
!                 }
                  
                  values[i] = new Entry(name,addr);
--- 417,431 ----
                  final long addr;
                  
! //                if (packedLongs) {
! //
! //                    addr = Long.valueOf(LongPacker.unpackLong(is));
! //
! //                } else {
! //
! //                    addr = Long.valueOf(is.readLong());
! //
! //                }
!                 
!                 addr = Addr.unpack(is);
                  
                  values[i] = new Entry(name,addr);

[cweb-CVS] bigdata/src/test/com/bigdata/objndx TestKeyBufferSerializer.java, 1.1, 1.2 TestByteArrayValueSerializer.java, 1.1, 1.2 SimpleEntry.java, 1.9, 1.10

From: Bryan T. <tho...@us...> - 2007-04-12 23:59:44

Update of /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2548/src/test/com/bigdata/objndx

Modified Files:
	TestKeyBufferSerializer.java TestByteArrayValueSerializer.java 
	SimpleEntry.java 
Log Message:
Added a Sesame 1.x SAIL implementation.  This is NOT intended for production use.  It is just being done to gain a high-level query language integration for the triple store.

Index: TestByteArrayValueSerializer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx/TestByteArrayValueSerializer.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** TestByteArrayValueSerializer.java	15 Feb 2007 01:34:23 -0000	1.1
--- TestByteArrayValueSerializer.java	12 Apr 2007 23:59:34 -0000	1.2
***************
*** 167,179 ****
              final byte[] data;
              {
!             ByteArrayOutputStream baos = new ByteArrayOutputStream();
! 
!             DataOutputStream dos = new DataOutputStream(baos);
  
              ser.putValues(dos, values, values.length);
              
!             dos.flush();
              
!             data = baos.toByteArray();
              }
  
--- 167,181 ----
              final byte[] data;
              {
! //            ByteArrayOutputStream baos = new ByteArrayOutputStream();
! //
! //            DataOutputStream dos = new DataOutputStream(baos);
!                 
!                 DataOutputBuffer dos = new DataOutputBuffer();
  
              ser.putValues(dos, values, values.length);
              
! //            dos.flush();
              
!             data = dos.buf;
              }
  

Index: SimpleEntry.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx/SimpleEntry.java,v
retrieving revision 1.9
retrieving revision 1.10
diff -C2 -d -r1.9 -r1.10
*** SimpleEntry.java	13 Feb 2007 23:01:10 -0000	1.9
--- SimpleEntry.java	12 Apr 2007 23:59:34 -0000	1.10
***************
*** 47,51 ****
--- 47,53 ----
  package com.bigdata.objndx;
  
+ import java.io.DataInput;
  import java.io.DataInputStream;
+ import java.io.DataOutput;
  import java.io.DataOutputStream;
  import java.io.IOException;
***************
*** 119,123 ****
          public Serializer() {}
  
!         public void putValues(DataOutputStream os, Object[] values, int n)
                  throws IOException {
  
--- 121,125 ----
          public Serializer() {}
  
!         public void putValues(DataOutputBuffer os, Object[] values, int n)
                  throws IOException {
  
***************
*** 130,134 ****
          }
  
!         public void getValues(DataInputStream is, Object[] values, int n)
                  throws IOException {
  
--- 132,136 ----
          }
  
!         public void getValues(DataInput is, Object[] values, int n)
                  throws IOException {
  
***************
*** 162,166 ****
          public NoSerializer() {}
          
!         public void getValues(DataInputStream is, Object[] values, int n) throws IOException {
  
              throw new UnsupportedOperationException();
--- 164,168 ----
          public NoSerializer() {}
          
!         public void getValues(DataInput is, Object[] values, int n) throws IOException {
  
              throw new UnsupportedOperationException();
***************
*** 168,172 ****
          }
  
!         public void putValues(DataOutputStream os, Object[] values, int n) throws IOException {
  
              throw new UnsupportedOperationException();
--- 170,174 ----
          }
  
!         public void putValues(DataOutputBuffer os, Object[] values, int n) throws IOException {
  
              throw new UnsupportedOperationException();

Index: TestKeyBufferSerializer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx/TestKeyBufferSerializer.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** TestKeyBufferSerializer.java	26 Jan 2007 02:39:25 -0000	1.1
--- TestKeyBufferSerializer.java	12 Apr 2007 23:59:34 -0000	1.2
***************
*** 123,137 ****
          // serialize.
          
!         ByteArrayOutputStream baos = new ByteArrayOutputStream();
!         
!         DataOutputStream dos = new DataOutputStream(baos);
          
          KeyBufferSerializer.INSTANCE.putKeys(dos, expected);
          
!         dos.flush();
  
          // de-serialize.
          
!         ByteArrayInputStream bais = new ByteArrayInputStream(baos.toByteArray());
  
          DataInputStream dis = new DataInputStream(bais);
--- 123,138 ----
          // serialize.
          
! //        ByteArrayOutputStream baos = new ByteArrayOutputStream();
! //        
! //        DataOutputStream dos = new DataOutputStream(baos);
!         DataOutputBuffer dos = new DataOutputBuffer();
          
          KeyBufferSerializer.INSTANCE.putKeys(dos, expected);
          
! //        dos.flush();
  
          // de-serialize.
          
!         ByteArrayInputStream bais = new ByteArrayInputStream(dos.buf);
  
          DataInputStream dis = new DataInputStream(bais);
***************
*** 152,166 ****
          // serialize the mutable key buffer.
          
!         baos = new ByteArrayOutputStream();
          
!         dos = new DataOutputStream(baos);
          
          KeyBufferSerializer.INSTANCE.putKeys(dos, expected2);
          
!         dos.flush();
  
          // de-serialize again.
          
!         bais = new ByteArrayInputStream(baos.toByteArray());
  
          dis = new DataInputStream(bais);
--- 153,167 ----
          // serialize the mutable key buffer.
          
! //        baos = new ByteArrayOutputStream();
          
!         dos = new DataOutputBuffer();
          
          KeyBufferSerializer.INSTANCE.putKeys(dos, expected2);
          
! //        dos.flush();
  
          // de-serialize again.
          
!         bais = new ByteArrayInputStream(dos.buf);
  
          dis = new DataInputStream(bais);

[cweb-CVS] bigdata-rdf/src/test/com/bigdata/rdf/sail TestSimpleRdfRepository.java, NONE, 1.1 TestAll.java, NONE, 1.1

From: Bryan T. <tho...@us...> - 2007-04-12 23:59:27

Update of /cvsroot/cweb/bigdata-rdf/src/test/com/bigdata/rdf/sail
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2483/src/test/com/bigdata/rdf/sail

Added Files:
	TestSimpleRdfRepository.java TestAll.java 
Log Message:
Added a Sesame 1.x SAIL implementation.  This is NOT intended for production use.  It is just being done to gain a high-level query language integration for the triple store.

--- NEW FILE: TestSimpleRdfRepository.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Apr 12, 2007
 */

package com.bigdata.rdf.sail;

import java.util.Properties;

import org.openrdf.model.Resource;
import org.openrdf.model.URI;
import org.openrdf.model.Value;
import org.openrdf.model.impl.LiteralImpl;
import org.openrdf.model.impl.URIImpl;
import org.openrdf.sesame.sail.RdfRepository;
import org.openrdf.sesame.sail.SailInitializationException;
import org.openrdf.sesame.sail.SailUpdateException;
import org.openrdf.vocabulary.XmlSchema;

import com.bigdata.journal.BufferMode;
import com.bigdata.journal.Options;

/**
 * Test suite for {@link SimpleRdfRepository}.
 * <p>
 * Note: This class has a dependency on the openrdf test suite.  If you are not
 * trying to run the test suites, then can simply ignore compile errors in this
 * class.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public class TestSimpleRdfRepository extends org.openrdf.sesame.sail.RdfRepositoryTest {

    SimpleRdfRepository repo;
    
    /**
     * @param arg0
     */
    public TestSimpleRdfRepository(String arg0) {
        super(arg0);
    }

    /**
     * Test using a literal for the value.
     */
    public void test_addStatement_001() throws SailUpdateException {

        repo.startTransaction();

        Resource s = new URIImpl("http://www.somewhere.org");

        URI p = new URIImpl("http://www.outthere.com");

        Value o = new LiteralImpl("in the sky");

        assertFalse(repo.hasStatement(s, p, o));

        repo.addStatement(s, p, o);

        assertTrue(repo.hasStatement(s, p, o));

        repo.commitTransaction();

    }

    /**
     * Test using a literal with a language tag.
     */
    public void test_addStatement_002() throws SailUpdateException {

        repo.startTransaction();

        Resource s = new URIImpl("http://www.somewhere.org");

        URI p = new URIImpl("http://www.outthere.com");

        Value o = new LiteralImpl("in the sky", "en");

        assertFalse(repo.hasStatement(s, p, o));

        repo.addStatement(s, p, o);

        assertTrue(repo.hasStatement(s, p, o));
        
        repo.commitTransaction();
        
    }

    /** Test using a literal with a datatype URI. */    
    public void test_addStatement_003()
        throws SailUpdateException
    {
    
        repo.startTransaction();
        
        Resource s = new URIImpl("http://www.somewhere.org");

        URI p = new URIImpl("http://www.outthere.com");

        Value o = new LiteralImpl("in the sky", new URIImpl(XmlSchema.STRING));

        assertFalse(repo.hasStatement(s, p, o));

        repo.addStatement(s, p, o);

        assertTrue(repo.hasStatement(s, p, o));
        
        repo.commitTransaction();
        
    }

    /**
     * Configures the test repository.
     * <p>
     * Note: This test suite requires that the RDFS closure is NOT maintained
     * since some of the tests require RDF-only semantics and will trip up on
     * the entailments inserted into the RDF model by closure, e.g., the axoims.
     * For example, an empty repository is not empty after closure.
     */    
    protected RdfRepository _getTestRepository()
       throws SailInitializationException
    {

        repo = new SimpleRdfRepository();
        
        Properties params = new Properties();
        
//        Map params = System.getProperties();
//
//        // Clone the system properties.
//        params = new java.util.HashMap( params );
//        
//        if(params.get(Options.BUFFER_MODE)==null) {
//            
//            /*
//             * Use the transient journal by default.
//             */
            params.put(Options.BUFFER_MODE, BufferMode.Transient.toString());
//            
//        }
        
        try {
            
            repo.initialize( params );
        
        }
        
        catch( SailInitializationException ex ) {
        
            /*
             * Note: Sesame is not provide the stack trace, so we do it
             * ourselves.
             */
            ex.printStackTrace();
            
            throw ex;
            
        }
        
        return repo;
        
    }
    
}

--- NEW FILE: TestAll.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
package com.bigdata.rdf.sail;

import junit.framework.Test;
import junit.framework.TestCase;
import junit.framework.TestSuite;

/**
 * Aggregates test suites into increasing dependency order.
 *
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public class TestAll extends TestCase {

    /**
     * 
     */
    public TestAll() {
    }

    /**
     * @param arg0
     */
    public TestAll(String arg0) {
        super(arg0);
    }

    /**
     * Returns a test that will run test suites in turn.
     */
    public static Test suite()
    {

        TestSuite suite = new TestSuite("SAIL");

        suite.addTestSuite( TestSimpleRdfRepository.class );
     
        return suite;
        
    }
    
}

[cweb-CVS] bigdata-rdf/src/java/com/bigdata/rdf/inf AbstractRuleRdf.java, 1.4, 1.5 InferenceEngine.java, 1.10, 1.11 Rule.java, 1.3, 1.4 AbstractRuleRdfs511.java, 1.4, 1.5 RuleRdf01.java, 1.4, 1.5 AbstractRuleRdfs2379.java, 1.3, 1.4 AbstractRuleRdfs68101213.java, 1.4, 1.5

From: Bryan T. <tho...@us...> - 2007-04-12 23:59:26

Update of /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2483/src/java/com/bigdata/rdf/inf

Modified Files:
	AbstractRuleRdf.java InferenceEngine.java Rule.java 
	AbstractRuleRdfs511.java RuleRdf01.java 
	AbstractRuleRdfs2379.java AbstractRuleRdfs68101213.java 
Log Message:
Added a Sesame 1.x SAIL implementation.  This is NOT intended for production use.  It is just being done to gain a high-level query language integration for the triple store.

Index: InferenceEngine.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/InferenceEngine.java,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** InferenceEngine.java	17 Feb 2007 23:15:26 -0000	1.10
--- InferenceEngine.java	12 Apr 2007 23:59:21 -0000	1.11
***************
*** 355,358 ****
--- 355,365 ----
      }
      
+     /**
+      * Copies the entailments from the temporary store into the main store.
+      * 
+      * @param entailments
+      * 
+      * @return The #of entailments inserted into the main store.
+      */
      private int transferBTrees( TempTripleStore entailments ) {
          
***************
*** 403,407 ****
          final int n = ndx.rangeCount(fromKey, toKey);
  
!         // bufferQueue for storing the extracted s:p:o data.
          SPO[] ids = new SPO[n];
  
--- 410,414 ----
          final int n = ndx.rangeCount(fromKey, toKey);
  
!         // buffer for storing the extracted s:p:o data.
          SPO[] ids = new SPO[n];
  
***************
*** 439,442 ****
--- 446,452 ----
       * @exception IllegalArgumentException
       *                if query is a fact (no variables).
+      * 
+      * FIXME Magic sets has NOT been implemented -- this method does NOT
+      * function.
       */
      public TripleStore query(Triple query, Rule[] rules) throws IOException {

Index: AbstractRuleRdfs511.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/AbstractRuleRdfs511.java,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** AbstractRuleRdfs511.java	17 Feb 2007 03:07:59 -0000	1.4
--- AbstractRuleRdfs511.java	12 Apr 2007 23:59:21 -0000	1.5
***************
*** 115,119 ****
              }
          }
!         dumpBuffer( stmts3.toArray( new SPO[stmts3.size()] ), entailments );
          
          stats.computeTime = System.currentTimeMillis() - computeStart;
--- 115,119 ----
              }
          }
!         if(debug)dumpBuffer( stmts3.toArray( new SPO[stmts3.size()] ), entailments );
          
          stats.computeTime = System.currentTimeMillis() - computeStart;

Index: AbstractRuleRdfs2379.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/AbstractRuleRdfs2379.java,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** AbstractRuleRdfs2379.java	17 Feb 2007 03:07:59 -0000	1.3
--- AbstractRuleRdfs2379.java	12 Apr 2007 23:59:21 -0000	1.4
***************
*** 87,91 ****
              }
          }
!         dumpBuffer( stmts3.toArray( new SPO[stmts3.size()] ), entailments );
          
          stats.computeTime = System.currentTimeMillis() - computeStart;
--- 87,91 ----
              }
          }
!         if(debug) dumpBuffer( stmts3.toArray( new SPO[stmts3.size()] ), entailments );
          
          stats.computeTime = System.currentTimeMillis() - computeStart;

Index: RuleRdf01.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/RuleRdf01.java,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** RuleRdf01.java	17 Feb 2007 03:07:59 -0000	1.4
--- RuleRdf01.java	12 Apr 2007 23:59:21 -0000	1.5
***************
*** 99,103 ****
              
          }
!         dumpBuffer( entailments.toArray( new SPO[entailments.size()] ), btree );
          
          stats.computeTime = System.currentTimeMillis() - computeStart;
--- 99,103 ----
              
          }
!         if(debug)dumpBuffer( entailments.toArray( new SPO[entailments.size()] ), btree );
          
          stats.computeTime = System.currentTimeMillis() - computeStart;

Index: Rule.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/Rule.java,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** Rule.java	17 Feb 2007 03:07:59 -0000	1.3
--- Rule.java	12 Apr 2007 23:59:21 -0000	1.4
***************
*** 69,72 ****
--- 69,74 ----
  abstract public class Rule {
  
+     static protected final boolean debug = false; 
+     
      /**
       * The inference engine.

Index: AbstractRuleRdf.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/AbstractRuleRdf.java,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** AbstractRuleRdf.java	17 Feb 2007 03:07:59 -0000	1.4
--- AbstractRuleRdf.java	12 Apr 2007 23:59:21 -0000	1.5
***************
*** 53,62 ****
  import com.bigdata.rdf.TempTripleStore;
  
- 
  public abstract class AbstractRuleRdf extends Rule {
  
      protected final int BUFFER_SIZE = 10*1024*1024;
      
-     
      public AbstractRuleRdf(InferenceEngine store, Triple head, Pred[] body) {
  
--- 53,60 ----

Index: AbstractRuleRdfs68101213.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/AbstractRuleRdfs68101213.java,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** AbstractRuleRdfs68101213.java	17 Feb 2007 03:07:59 -0000	1.4
--- AbstractRuleRdfs68101213.java	12 Apr 2007 23:59:21 -0000	1.5
***************
*** 102,106 ****
              
          }
!         dumpBuffer( entailments.toArray( new SPO[entailments.size()] ), btree );
          
          stats.computeTime = System.currentTimeMillis() - computeStart;
--- 102,106 ----
              
          }
!         if(debug)dumpBuffer( entailments.toArray( new SPO[entailments.size()] ), btree );
          
          stats.computeTime = System.currentTimeMillis() - computeStart;

[cweb-CVS] bigdata-rdf/src/java/com/bigdata/rdf/sail SimpleRdfRepository.java, NONE, 1.1 package.html, NONE, 1.1

From: Bryan T. <tho...@us...> - 2007-04-12 23:59:26

Update of /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/sail
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2483/src/java/com/bigdata/rdf/sail

Added Files:
	SimpleRdfRepository.java package.html 
Log Message:
Added a Sesame 1.x SAIL implementation.  This is NOT intended for production use.  It is just being done to gain a high-level query language integration for the triple store.

--- NEW FILE: SimpleRdfRepository.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Apr 12, 2007
 */

package com.bigdata.rdf.sail;

import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.Vector;

import org.CognitiveWeb.util.PropertyUtil;
import org.apache.log4j.Logger;
import org.openrdf.model.BNode;
import org.openrdf.model.Literal;
import org.openrdf.model.Resource;
import org.openrdf.model.Statement;
import org.openrdf.model.URI;
import org.openrdf.model.Value;
import org.openrdf.model.ValueFactory;
import org.openrdf.sesame.sail.NamespaceIterator;
import org.openrdf.sesame.sail.RdfRepository;
import org.openrdf.sesame.sail.SailChangedEvent;
import org.openrdf.sesame.sail.SailChangedListener;
import org.openrdf.sesame.sail.SailInitializationException;
import org.openrdf.sesame.sail.SailInternalException;
import org.openrdf.sesame.sail.SailUpdateException;
import org.openrdf.sesame.sail.StatementIterator;
import org.openrdf.sesame.sail.query.Query;
import org.openrdf.sesame.sail.query.QueryOptimizer;
import org.openrdf.sesame.sail.query.Var;
import org.openrdf.sesame.sail.util.EmptyStatementIterator;
import org.openrdf.sesame.sail.util.SailChangedEventImpl;
import org.openrdf.sesame.sail.util.SingleStatementIterator;

import com.bigdata.journal.Journal;
import com.bigdata.journal.Options;
import com.bigdata.objndx.IEntryIterator;
import com.bigdata.rdf.KeyOrder;
import com.bigdata.rdf.TripleStore;
import com.bigdata.rdf.inf.InferenceEngine;
import com.bigdata.rdf.inf.SPO;
import com.bigdata.rdf.model.OptimizedValueFactory;
import com.bigdata.rdf.model.OptimizedValueFactory._Resource;
import com.bigdata.rdf.model.OptimizedValueFactory._Statement;
import com.bigdata.rdf.model.OptimizedValueFactory._URI;
import com.bigdata.rdf.model.OptimizedValueFactory._Value;

/**
 * A simple Sesame 1.x SAIL integration.
 * <p>
 * This is very much a trial balloon. Sesame 1.x coupled the control logic and
 * data structures in such a way that you could not write your own JOIN
 * operators, which makes it very difficult to optimize performance. A very
 * basic and untuned mechanisms is provided to compute the
 * {@link #fullForwardClosure()} of the triple store (there is no truth
 * maintenance and you must re-compute the entire closure if you add more data
 * to the store) - use of this method will let you run high level queries using
 * the SAIL with entailments. Only a simple transaction model is supported (no
 * transactional isolation). The {@link TripleStore} uses an embedded write-only
 * {@link Journal} as its store (it does not support distributed scale-out). The
 * namespace management methods and the methods to clear the repository have not
 * been implemented.
 * <p>
 * A custom integration is provided for directly loading data into the triple
 * store with throughput of 20,000+ triples per second - see
 * {@link #loadData(java.io.File, String, boolean)}.
 * <p>
 * <em>THIS CLASS IS NOT THREAD SAFE</em>
 * <p>
 * <em>THIS CLASS IS NOT SAFE FOR TRAVERSAL UNDER CONCURRENT MODIFICATION</em>
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public class SimpleRdfRepository implements RdfRepository {

    /**
     * Logger.
     */
    public static final Logger log = Logger.getLogger
    ( SimpleRdfRepository.class
      );

    /**
     * The equivilent of a null identifier for an internal RDF Value.
     */
    protected static final long NULL = TripleStore.NULL;
    
    private OptimizedValueFactory valueFactory;
    private InferenceEngine tripleStore;
    private Properties properties;
    private boolean transactionStarted;
    
    /**
     * The implementation object.
     */
    public TripleStore getTripleStore() {
        
        return tripleStore;
        
    }
    
    /**
     * Create a repository.
     * 
     * @see #initialize(Map)
     */
    public SimpleRdfRepository() {
    }

    //
    // SailChangedListener support.
    //

    /**
     * Vector of transient {@link SailChangedListener}s registered with this
     * SAIL.
     */
    private Vector<SailChangedListener> m_listeners = null;
    private boolean m_stmtAdded = false;
    private boolean m_stmtRemoved = false;    

    public void addListener(SailChangedListener listener) {

        if( m_listeners == null ) {
            
            m_listeners = new Vector<SailChangedListener>();
            
            m_listeners.add( listener );
            
        } else {
            
            if( m_listeners.contains( listener ) ) {
                
                throw new IllegalStateException
                    ( "Already registered: listener="+listener
                      );
                
            }
            
            m_listeners.add( listener );
            
        }

    }

    public void removeListener(SailChangedListener listener) {

        if( m_listeners == null ) {
            
            throw new IllegalStateException
                ( "Not registered: listener="+listener
                  );
            
        }
        
        if( ! m_listeners.remove( listener ) ) {
            
            throw new IllegalStateException
                ( "Not registered: listener="+listener
                  );
            
        }

    }

    /**
     * Notifies {@link SailChangeListener}s if one or more statements
     * have been added to or removed from the repository using the SAIL
     * methods: <ul>
     * 
     * <li> {@link #addStatement(Resource, URI, Value)}
     * <li> {@link #removeStatements(Resource, URI, Value)}
     * <li> {@link #clearRepository()}
     * </ul>
     */
    synchronized protected void fireSailChangedEvents()
    {
        
        if( m_listeners == null ) return;
        
        if( ! m_stmtAdded && ! m_stmtRemoved ) return;

        SailChangedEvent e = new SailChangedEventImpl
            ( m_stmtAdded,
              m_stmtRemoved
              );
        
        SailChangedListener[] listeners = (SailChangedListener[]) 
            m_listeners.toArray( new SailChangedListener[]{} );

        for( int i=0; i<listeners.length; i++ ) {
            
            SailChangedListener l = listeners[ i ];
            
            l.sailChanged( e );
                
        }
        
    }

    public void addStatement(Resource s, URI  p, Value o)
            throws SailUpdateException {

        assertTransactionStarted();

        s = (Resource) valueFactory.toNativeValue(s);

        p = (URI) valueFactory.toNativeValue(p);

        o = (Value) valueFactory.toNativeValue(o);
        
        tripleStore.addStatement(s, p, o);
        
        m_stmtAdded = true;
        
    }

    public void changeNamespacePrefix(String arg0, String arg1)
            throws SailUpdateException {

        assertTransactionStarted();

        // TODO Auto-generated method stub

        throw new UnsupportedOperationException();

    }

    public NamespaceIterator getNamespaces() {
        
        // TODO Auto-generated method stub

        throw new UnsupportedOperationException();
    }

    /**
     * Note: This is not implemented. Since there is only one RdfRepository per
     * persistence store, the easiest way to achive this end is to delete the
     * persistence store and open/create a new one.
     */
    public void clearRepository() throws SailUpdateException {

        assertTransactionStarted();

        // TODO Auto-generated method stub
        
        m_stmtRemoved = true;
        
        throw new UnsupportedOperationException();
        
    }

    /**
     * Materializes the set of statements matching the triple pattern and then
     * deletes those statements from each index using an ordered batch delete
     * operation.
     */
    public int removeStatements(Resource s, URI p, Value o)
            throws SailUpdateException {
        
        assertTransactionStarted();
        
        /*
         * convert other Value object types to our object types.
         */
        if (s != null)
            s = (Resource) valueFactory.toNativeValue(s);

        if (p != null)
            p = (URI) valueFactory.toNativeValue(p);

        if (o != null)
            o = (Value) valueFactory.toNativeValue(o);
        
        int n = tripleStore.removeStatements(s, p, o);
        
        if (n > 0) {
            
            m_stmtRemoved = true;
            
        }
        
        return n;

    }

    public void startTransaction() {
        
        if(transactionStarted) {
            
            throw new SailInternalException(
                    "A transaction was already started.");

        }
        
        transactionStarted = true;
        
        m_stmtAdded = false;

        m_stmtRemoved = false;

    }

    public boolean transactionStarted() {

        return transactionStarted;
        
    }

    public void commitTransaction() {

        if( ! transactionStarted ) {
            
            throw new SailInternalException
                ( "No transaction has been started."
                  );

        }

        tripleStore.fullForwardClosure();
        
        tripleStore.commit();
        
        transactionStarted = false;
        
        fireSailChangedEvents();

    }

    protected void assertTransactionStarted() {

        if (!transactionStarted) {

            throw new SailInternalException("No transaction has been started");

        }
        
    }
    
    public StatementIterator getStatements(Resource s, URI p, Value o) {

        /*
         * convert other Value object types to our object types.
         */
        if (s != null)
            s = (Resource) valueFactory.toNativeValue(s);

        if (p != null)
            p = (URI) valueFactory.toNativeValue(p);

        if (o != null)
            o = (Value) valueFactory.toNativeValue(o);
        
        /*
         * convert our object types to internal identifiers.
         */
        long _s, _p, _o;

        _s = (s == null ? NULL : tripleStore.getTermId(s));
        _p = (p == null ? NULL : tripleStore.getTermId(p));
        _o = (o == null ? NULL : tripleStore.getTermId(o));

        /*
         * If a value was specified and it is not in the terms index then the
         * statement can not exist in the KB.
         */
        if (_s == NULL && s != null) {

            return new EmptyStatementIterator();
            
        }
        
        if (_p == NULL && p != null) {
        
            return new EmptyStatementIterator();
            
        }
        
        if (_o == NULL && o != null) {
            
            return new EmptyStatementIterator();
            
        }
        
        /*
         * if all bound, then a slight optimization.
         */
        if (_s != NULL && _p != NULL && _o != NULL) {

            if( tripleStore.getSPOIndex().contains(tripleStore.keyBuilder.statement2Key(_s, _p, _o))) {
                
                return new SingleStatementIterator(s,p,o);
                
            } else {
                
                return new EmptyStatementIterator();
                
            }
            
        }
        
        /*
         * Choose the access path and encapsulate the resulting range iterator
         * as a sesame statement iterator.
         */
        return new MyStatementIterator(KeyOrder.getKeyOrder(_s, _p, _o),
                tripleStore.rangeQuery(_s, _p, _o));
        
    }
    
    /**
     * Wraps the raw iterator that traverses a statement index and exposes each
     * visited statement as a {@link Statement} object.
     * 
     * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
     * @version $Id$
     */
    public class MyStatementIterator implements StatementIterator {

        private final KeyOrder keyOrder;
        private final IEntryIterator src;
        
        public MyStatementIterator(KeyOrder keyOrder,IEntryIterator src) {
            
            if(keyOrder == null) throw new IllegalArgumentException();

            if(src == null) throw new IllegalArgumentException();
            
            this.keyOrder = keyOrder;
            
            this.src = src;
            
        }
        
        public boolean hasNext() {
            
            return src.hasNext();
            
        }

        public Statement next() {
            
            // visit the next entry.
            src.next();

            // unpacks the term identifiers from the key.
            SPO spo = new SPO(keyOrder, tripleStore.keyBuilder, src.getKey());
            
            // resolves the term identifiers to the terms.
            return new _Statement((_Resource) tripleStore.getTerm(spo.s),
                    (_URI) tripleStore.getTerm(spo.p), (_Value) tripleStore
                            .getTerm(spo.o));
            
        }
        
        public void close() {
            // NOP.
        }

    }

    public ValueFactory getValueFactory() {
        
        return valueFactory;
        
    }

    public boolean hasStatement(Resource s, URI p, Value o) {

        if (s != null)
            s = (Resource) valueFactory.toNativeValue(s);

        if (p != null)
            p = (URI) valueFactory.toNativeValue(p);

        if (o != null)
            o = (Value) valueFactory.toNativeValue(o);
        
        return tripleStore.containsStatement(s, p, o);
        
    }

    /**
     * @todo rewrite the join order based on selectivity.
     */
    public Query optimizeQuery(Query query) {
        
        /*
         * This static method is provided by the Sesame framework and performs a
         * variety of default optimizations.
         */

        QueryOptimizer.optimizeQuery(query);

        /*
         * Replace all Value objects stored in variables with the corresponding
         * _Value objects.
         */

        replaceValuesInQuery
            ( query
              );

        return query;
        
    }
    
    /**
     * Replace all {@link Value} objects stored in variables with the
     * corresponding {@link _Value} objects.
     * <p>
     * 
     * Note: This can cause unknown terms to be inserted into store.
     */
    private void replaceValuesInQuery(Query query) {

        final List varList = new ArrayList();

        query.getVariables(varList);

        for (int i = 0; i < varList.size(); i++) {

            final Var var = (Var) varList.get(i);

            if (var.hasValue()) {

                final Value value = var.getValue();

                if (value instanceof URI) {

                    // URI substitution.

                    String uriString = ((URI) value).getURI();

                    var.setValue(valueFactory.createURI(uriString));

                } else if (value instanceof BNode) {

                    // BNode substitution.

                    final String id = ((BNode) value).getID();

                    if (id == null) {

                        var.setValue(valueFactory.createBNode());

                    } else {

                        var.setValue(valueFactory.createBNode(id));

                    }

                } else if (value instanceof Literal) {

                    // Literal substitution.

                    Literal lit = (Literal) value;

                    final String lexicalForm = lit.getLabel();

                    final String language = lit.getLanguage();

                    if (language != null) {

                        lit = valueFactory.createLiteral(lexicalForm, language);

                    } else {

                        URI datatype = lit.getDatatype();

                        if (datatype != null) {

                            lit = valueFactory.createLiteral(lexicalForm,
                                    datatype);

                        } else {

                            lit = valueFactory.createLiteral(lexicalForm);

                        }

                    }

                    var.setValue(lit);

                } // if( literal )

            } // if( hasValue )

        } // next variable.

    }

    /**
     * @param configParams
     *            See {@link Options} for the persistence store options.
     *            <p>
     *            The optional boolean option "eagerClosure" may be used to turn
     *            on a simple inference engine that will compute the eager
     *            closure of data loaded into the store via either the SAIL
     *            transaction mechanism or the batch load mechanism. The
     *            selection of this option is NOT restart safe (it is not saved
     *            in the store).
     * 
     * @see #loadData(File, String, boolean)
     */
    public void initialize(Map configParams) throws SailInitializationException {

        properties = PropertyUtil.flatCopy(PropertyUtil.convert(configParams));

        valueFactory = new OptimizedValueFactory();
        
        try {

             tripleStore = new InferenceEngine(properties);
            
        } catch(IOException ex) {
            
            throw new SailInitializationException(ex);
            
        }
        
    }

    public void shutDown() {

        tripleStore.shutdown();
        
    }

    /**
     * Uses a fast batch loader to load the data into the store.
     * <p>
     * This method lies outside of the SAIL and does not rely on the SAIL
     * "transaction" mechanisms.
     * 
     * @param file
     *            The file containing the data.
     * @param baseURI
     *            The base URI (optional).
     * @param commit
     *            When true, the store will be committed after the data load
     *            (you can use false here if you will lots of small files in
     *            sequence).
     */
    public void loadData(File file, String baseURI, boolean commit)
            throws IOException {

        tripleStore.loadData(file, baseURI, commit);
        
    }
    
    /**
     * Computes the closure of the triple store for RDFS entailments. This
     * implementation computes the full forward closure of the store and then
     * commits the store. Since incremental closure and truth maintenance are
     * NOT supported, you should load all your data into store using
     * {@link #loadData(File, String, boolean)} and then invoke this method to
     * compute the RDFS entailments.
     * <p>
     * This method lies outside of the SAIL and does not rely on the SAIL
     * "transaction" mechanisms.
     */
    public void fullForwardClosure() {
        
        tripleStore.fullForwardClosure();
        
        tripleStore.commit();
                
    }
    
}

--- NEW FILE: package.html ---
<html>
<head>
<title>Sesame 1.x SAIL integration</title>
</head>
<body>
<p>
This package provides a Sesame 1.x integration using the SAIL.  This is very
much a trial balloon.  Sesame 1.x coupled the control logic and data structures
in such a way that you could not write your own JOIN operators, which makes it
very difficult to optimize performance.  An option is provided to turn on a
simple (and not very well tested) eager closure solution without truth maintenance.
A custom integration is provided for directly loading data into the triple store
with throughput of 20,000+ triples per second - see the com.bigdata.rdf.rio
package and {@link com.bigdata.rdf.TripleStore}.
</p>
</body>
</html>

[cweb-CVS] bigdata-rdf/src/test/com/bigdata/rdf/metrics TaskATest.java, 1.1, 1.2

From: Bryan T. <tho...@us...> - 2007-04-12 23:59:25

Update of /cvsroot/cweb/bigdata-rdf/src/test/com/bigdata/rdf/metrics
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2483/src/test/com/bigdata/rdf/metrics

Modified Files:
	TaskATest.java 
Log Message:
Added a Sesame 1.x SAIL implementation.  This is NOT intended for production use.  It is just being done to gain a high-level query language integration for the triple store.

Index: TaskATest.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/test/com/bigdata/rdf/metrics/TaskATest.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** TaskATest.java	6 Feb 2007 23:06:44 -0000	1.1
--- TaskATest.java	12 Apr 2007 23:59:22 -0000	1.2
***************
*** 52,59 ****
--- 52,62 ----
  import java.io.IOException;
  import java.io.PrintWriter;
+ import java.nio.BufferOverflowException;
  
  import org.openrdf.sesame.admin.UpdateException;
  import org.openrdf.sesame.sail.RdfSchemaRepository;
  
+ import com.bigdata.rdf.TripleStore.LoadStats;
+ 
  /**
   * @todo update javadoc.
***************
*** 188,194 ****
  
          /*
!          * Load the data.
           */
!         long elapsedLoad = loadData();
          
      }
--- 191,197 ----
  
          /*
!          * Load the data. Sets some instance variables from various counters.
           */
!         loadData();
          
      }
***************
*** 265,269 ****
      
      /**
!      * Load each RDF source in turn and perform RDFS closure on the data.
       * 
       * @return The elasped time to load the data and perform closure.
--- 268,272 ----
      
      /**
!      * Load each RDF source in turn.
       * 
       * @return The elasped time to load the data and perform closure.
***************
*** 278,282 ****
          for(int i=0; i<filename.length; i++) {
           
!             store.loadData(new File(filename[i]), baseURI[i]);
              
          }
--- 281,285 ----
          for(int i=0; i<filename.length; i++) {
           
!             loadStats = store.loadData(new File(filename[i]), baseURI[i], true);
              
          }
***************
*** 289,292 ****
--- 292,300 ----
  
      }
+     
+     /**
+      * Set by {@link #loadData()}.
+      */
+     protected LoadStats loadStats;
  
      /**
***************
*** 418,421 ****
--- 426,431 ----
          }
  
+         LoadStats[] loadStats = new LoadStats[ all_sources.length ];
+ 
          for( int i=0, run=0; i<all_sources.length; i+=3, run++ ) {
              
***************
*** 435,438 ****
--- 445,449 ----
                      test.testFilesExist();
                      test.testOntology();
+                     loadStats[run] = test.loadStats;
  //                    test.testReadPerformance();
                      w.println( "SUCCESS" );
***************
*** 462,470 ****
          System.out.println( "\n\n\n"+nok+" out of "+nruns+" Ok.");
          
          for( int run=0; run<nruns; run++ ) {
              
!             System.out.println( all_sources[ run * 3 ]+ " : " +
                      ( errors[ run ] == null
!                       ? "Ok"
                        : errors[ run ].getMessage()
                        )
--- 473,486 ----
          System.out.println( "\n\n\n"+nok+" out of "+nruns+" Ok.");
          
+         System.out.println("name, status, triplesPerSecond, loadTime(s), commitTime(ms)");
+         
          for( int run=0; run<nruns; run++ ) {
              
!             System.out.println( all_sources[ run * 3 ]+ ", " +
                      ( errors[ run ] == null
!                       ? "Ok, "
!                             + loadStats[run].triplesPerSecond() + ", "
!                             + loadStats[run].loadTime / 1000 + ", "
!                             + loadStats[run].commitTime
                        : errors[ run ].getMessage()
                        )

[cweb-CVS] bigdata-rdf/src/java/com/bigdata/rdf KeyOrder.java, 1.1, 1.2 TripleStore.java, 1.26, 1.27

From: Bryan T. <tho...@us...> - 2007-04-12 23:59:25

Update of /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2483/src/java/com/bigdata/rdf

Modified Files:
	KeyOrder.java TripleStore.java 
Log Message:
Added a Sesame 1.x SAIL implementation.  This is NOT intended for production use.  It is just being done to gain a high-level query language integration for the triple store.

Index: KeyOrder.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/KeyOrder.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** KeyOrder.java	26 Jan 2007 20:51:28 -0000	1.1
--- KeyOrder.java	12 Apr 2007 23:59:21 -0000	1.2
***************
*** 69,71 ****
--- 69,124 ----
      }
      
+     private static final long NULL = TripleStore.NULL;
+     
+     /**
+      * Return the access path that should be used for the triple pattern.
+      * 
+      * @param s
+      *            The optional subject identifier or {@link TripleStore#NULL}.
+      * @param p
+      *            The optional subject identifier or {@link TripleStore#NULL}.
+      * @param o
+      *            The optional subject identifier or {@link TripleStore#NULL}.
+      * 
+      * @return The KeyOrder that identifies the index to use for that triple
+      *         pattern.
+      */
+     public static KeyOrder getKeyOrder(long s, long p, long o) {
+ 
+         if (s != NULL && p != NULL && o != NULL) {
+ 
+             return SPO;
+ 
+         } else if (s != NULL && p != NULL) {
+ 
+             return SPO;
+ 
+         } else if (s != NULL && o != NULL) {
+ 
+             return OSP;
+ 
+         } else if (p != NULL && o != NULL) {
+ 
+             return POS;
+ 
+         } else if (s != NULL) {
+ 
+             return SPO;
+ 
+         } else if (p != NULL) {
+ 
+             return POS;
+ 
+         } else if (o != NULL) {
+ 
+             return OSP;
+ 
+         } else {
+ 
+             return SPO;
+ 
+         }
+ 
+     }
+     
  }

Index: TripleStore.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/TripleStore.java,v
retrieving revision 1.26
retrieving revision 1.27
diff -C2 -d -r1.26 -r1.27
*** TripleStore.java	29 Mar 2007 17:01:47 -0000	1.26
--- TripleStore.java	12 Apr 2007 23:59:21 -0000	1.27
***************
*** 72,79 ****
--- 72,81 ----
  import com.bigdata.objndx.Errors;
  import com.bigdata.objndx.IBatchOp;
+ import com.bigdata.objndx.IEntryIterator;
  import com.bigdata.objndx.IIndex;
  import com.bigdata.objndx.ISimpleBTree;
  import com.bigdata.objndx.KeyBuilder;
  import com.bigdata.rawstore.Bytes;
+ import com.bigdata.rdf.inf.SPO;
  import com.bigdata.rdf.model.OptimizedValueFactory.OSPComparator;
  import com.bigdata.rdf.model.OptimizedValueFactory.POSComparator;
***************
*** 553,578 ****
      /**
       * Return true if the statement exists in the store (non-batch API).
       */
      public boolean containsStatement(Resource s, URI p, Value o) {
  
          long _s, _p, _o;
          
!         if( (_s = getTermId(s)) == 0L ) return false;
!         if( (_p = getTermId(p)) == 0L ) return false;
!         if( (_o = getTermId(o)) == 0L ) return false;
          
!         return getSPOIndex().contains(keyBuilder.statement2Key(_s, _p, _o));
          
      }
  
      /**
       * Adds the statements to each index (batch api).
!      * <p> 
       * Note: this is not sorting by the generated keys so the sort order may not
!      *       perfectly reflect the natural order of the index. however, i
!      *       suspect that it simply creates a few partitions out of the natural
!      *       index order based on the difference between signed and unsigned
!      *       interpretations of the termIds when logically combined into a
!      *       statement identifier.
       * 
       * @param stmts
--- 555,965 ----
      /**
       * Return true if the statement exists in the store (non-batch API).
+      * 
+      * @param s
+      *            Optional subject.
+      * @param p
+      *            Optional predicate.
+      * @param o
+      *            Optional object.
       */
      public boolean containsStatement(Resource s, URI p, Value o) {
  
          long _s, _p, _o;
+ 
+         _s = (s == null ? NULL : getTermId(s));
+         _p = (p == null ? NULL : getTermId(p));
+         _o = (o == null ? NULL : getTermId(o));
+ 
+         /*
+          * If a value was specified and it is not in the terms index then the
+          * statement can not exist in the KB.
+          */
+         if (_s == NULL && s != null)
+             return false;
+         if (_p == NULL && p != null)
+             return false;
+         if (_o == NULL && o != null)
+             return false;
          
!         /*
!          * if all bound, then a slight optimization.
!          */
!         if (_s != NULL && _p != NULL && _o != NULL) {
! 
!             return getSPOIndex().contains(keyBuilder.statement2Key(_s, _p, _o));
!             
!         }
          
!         /*
!          * Choose the access path and test to see if any statements would be
!          * visited for that triple pattern.
!          */
!         return rangeQuery(_s,_p,_o).hasNext();
!         
!     }
! 
!     /**
!      * Return a range query iterator that will visit the statements matching the
!      * triple pattern using the best access path given the triple pattern.
!      * 
!      * @param s
!      *            An optional term identifier for the subject role or
!      *            {@link #NULL}.
!      * @param p
!      *            An optional term identifier for the predicate role or
!      *            {@link #NULL}.
!      * @param o
!      *            An optional term identifier for the object role or
!      *            {@link #NULL}.
!      * 
!      * @return The range query iterator.
!      * 
!      * @todo write tests.
!      */
!     public IEntryIterator rangeQuery(long s, long p, long o) {
! 
!         if (s != NULL && p != NULL && o != NULL) {
! 
!             byte[] fromKey = keyBuilder.statement2Key(s, p, o);
! 
!             byte[] toKey = keyBuilder.statement2Key(s, p, o + 1);
! 
!             return getSPOIndex().rangeIterator(fromKey, toKey);
! 
!         } else if (s != NULL && p != NULL) {
! 
!             byte[] fromKey = keyBuilder.statement2Key(s, p, NULL);
! 
!             byte[] toKey = keyBuilder.statement2Key(s, p + 1, NULL);
! 
!             return getSPOIndex().rangeIterator(fromKey, toKey);
! 
!         } else if (s != NULL && o != NULL) {
! 
!             byte[] fromKey = keyBuilder.statement2Key(o, s, NULL);
! 
!             byte[] toKey = keyBuilder.statement2Key(o, s + 1, NULL);
! 
!             return getOSPIndex().rangeIterator(fromKey, toKey);
! 
!         } else if (p != NULL && o != NULL) {
! 
!             byte[] fromKey = keyBuilder.statement2Key(p, o, NULL);
! 
!             byte[] toKey = keyBuilder.statement2Key(p, o + 1, NULL);
! 
!             return getPOSIndex().rangeIterator(fromKey, toKey);
! 
!         } else if (s != NULL) {
! 
!             byte[] fromKey = keyBuilder.statement2Key(s, NULL, NULL);
! 
!             byte[] toKey = keyBuilder.statement2Key(s + 1, NULL, NULL);
! 
!             return getSPOIndex().rangeIterator(fromKey, toKey);
! 
!         } else if (p != NULL) {
! 
!             byte[] fromKey = keyBuilder.statement2Key(p, NULL, NULL);
! 
!             byte[] toKey = keyBuilder.statement2Key(p + 1, NULL, NULL);
! 
!             return getPOSIndex().rangeIterator(fromKey, toKey);
! 
!         } else if (o != NULL) {
! 
!             byte[] fromKey = keyBuilder.statement2Key(o, NULL, NULL);
! 
!             byte[] toKey = keyBuilder.statement2Key(o + 1, NULL, NULL);
! 
!             return getOSPIndex().rangeIterator(fromKey, toKey);
! 
!         } else {
! 
!             return getSPOIndex().rangeIterator(null, null);
! 
!         }
! 
!     }
!     
!     /**
!      * Return the #of statements matching the triple pattern using the best
!      * access path given the triple pattern (the count will be approximate if
!      * partitioned indices are being used).
!      * 
!      * @param s
!      *            An optional term identifier for the subject role or
!      *            {@link #NULL}.
!      * @param p
!      *            An optional term identifier for the predicate role or
!      *            {@link #NULL}.
!      * @param o
!      *            An optional term identifier for the object role or
!      *            {@link #NULL}.
!      * 
!      * @return The range count.
!      * 
!      * @todo write tests.
!      */
!     public int rangeCount(long s, long p, long o) {
! 
!         if (s != NULL && p != NULL && o != NULL) {
! 
!             byte[] fromKey = keyBuilder.statement2Key(s, p, o);
! 
!             byte[] toKey = keyBuilder.statement2Key(s, p, o + 1);
! 
!             return getSPOIndex().rangeCount(fromKey, toKey);
! 
!         } else if (s != NULL && p != NULL) {
! 
!             byte[] fromKey = keyBuilder.statement2Key(s, p, NULL);
! 
!             byte[] toKey = keyBuilder.statement2Key(s, p + 1, NULL);
! 
!             return getSPOIndex().rangeCount(fromKey, toKey);
! 
!         } else if (s != NULL && o != NULL) {
! 
!             byte[] fromKey = keyBuilder.statement2Key(o, s, NULL);
! 
!             byte[] toKey = keyBuilder.statement2Key(o, s + 1, NULL);
! 
!             return getOSPIndex().rangeCount(fromKey, toKey);
! 
!         } else if (p != NULL && o != NULL) {
! 
!             byte[] fromKey = keyBuilder.statement2Key(p, o, NULL);
! 
!             byte[] toKey = keyBuilder.statement2Key(p, o + 1, NULL);
! 
!             return getPOSIndex().rangeCount(fromKey, toKey);
! 
!         } else if (s != NULL) {
! 
!             byte[] fromKey = keyBuilder.statement2Key(s, NULL, NULL);
! 
!             byte[] toKey = keyBuilder.statement2Key(s + 1, NULL, NULL);
! 
!             return getSPOIndex().rangeCount(fromKey, toKey);
! 
!         } else if (p != NULL) {
! 
!             byte[] fromKey = keyBuilder.statement2Key(p, NULL, NULL);
! 
!             byte[] toKey = keyBuilder.statement2Key(p + 1, NULL, NULL);
! 
!             return getPOSIndex().rangeCount(fromKey, toKey);
! 
!         } else if (o != NULL) {
! 
!             byte[] fromKey = keyBuilder.statement2Key(o, NULL, NULL);
! 
!             byte[] toKey = keyBuilder.statement2Key(o + 1, NULL, NULL);
! 
!             return getOSPIndex().rangeCount(fromKey, toKey);
! 
!         } else {
! 
!             return getSPOIndex().rangeCount(null, null);
! 
!         }
! 
!     }
!     
!     /**
!      * Removes statements matching the triple pattern.
!      * 
!      * @param s
!      * @param p
!      * @param o
!      * 
!      * @return The #of statements removed.
!      * 
!      * @todo write tests.
!      */
!     public int removeStatements(Resource s,URI p,Value o) {
! 
!         /*
!          * convert our object types to internal identifiers.
!          */
!         long _s, _p, _o;
! 
!         _s = (s == null ? NULL : getTermId(s));
!         _p = (p == null ? NULL : getTermId(p));
!         _o = (o == null ? NULL : getTermId(o));
! 
!         /*
!          * If a value was specified and it is not in the terms index then the
!          * statement can not exist in the KB.
!          */
!         if (_s == NULL && s != null) {
! 
!             return 0;
!             
!         }
!         
!         if (_p == NULL && p != null) {
!         
!             return 0;
!             
!         }
!         
!         if (_o == NULL && o != null) {
!             
!             return 0;
!             
!         }
!     
!         return removeStatements(_s,_p,_o);
          
      }
+     
+     /**
+      * Remove statements matching the triple pattern.
+      * <p>
+      * Since the indices do not support modification with concurrent traversal
+      * the statements are materialized before they are deleted.
+      * 
+      * @param _s
+      * @param _p
+      * @param _o
+      * 
+      * @return The #of statements removed.
+      * 
+      * @todo the {@link #keyBuilder} is, which means that this is NOT thread
+      *       safe.
+      * 
+      * @todo this is not using the batch btree api.
+      * 
+      * @todo write tests.
+      */
+     public int removeStatements(long _s, long _p, long _o) {
+     
+         /*
+          * if all bound, then a slight optimization.
+          */
+         if (_s != NULL && _p != NULL && _o != NULL) {
+ 
+             byte[] key = keyBuilder.statement2Key(_s, _p, _o);
+ 
+             if (getSPOIndex().contains(key)) {
+ 
+                 getSPOIndex().remove(key);
+ 
+                 return 1;
+                 
+             } else {
+                 
+                 return 0;
+                 
+             }
+             
+         }
+         
+         /*
+          * Choose the access path, count the #of statements that match the
+          * triple pattern, and the materalize those statements (since traversal
+          * with concurrent modification is not supported).
+          */
+         
+         KeyOrder keyOrder = KeyOrder.getKeyOrder(_s, _p, _o);
+         
+         // #of matching statements.
+         int rangeCount = rangeCount(_s, _p, _o);
+         
+         SPO[] stmts = new SPO[rangeCount];
+ 
+         // materialize the matching statements.
+         {
+             IEntryIterator itr1 = rangeQuery(_s, _p, _o);
+ 
+             int i = 0;
+ 
+             while (itr1.hasNext()) {
+ 
+                 itr1.next();
+ 
+                 stmts[i++] = new SPO(keyOrder, keyBuilder, itr1.getKey());
+ 
+             }
+ 
+             assert i == rangeCount;
+         }
+ 
+         /*
+          * Remove the statements from each of the access paths.
+          */
+         {
+ 
+             {
+                 IIndex ndx = getSPOIndex();
+ 
+                 // Place statements in SPO order.
+                 Arrays.sort(stmts, com.bigdata.rdf.inf.SPOComparator.INSTANCE);
+ 
+                 // remove statements from SPO index.
+                 for (int i = 0; i < stmts.length; i++) {
+ 
+                     SPO spo = stmts[i];
+ 
+                     ndx.remove(keyBuilder.statement2Key(spo.s, spo.p, spo.o));
+ 
+                 }
+             }
+ 
+             {
+ 
+                 IIndex ndx = getPOSIndex();
+                 // Place statements in POS order.
+                 Arrays.sort(stmts, com.bigdata.rdf.inf.POSComparator.INSTANCE);
+ 
+                 // Remove statements from POS index.
+                 for (int i = 0; i < stmts.length; i++) {
+ 
+                     SPO spo = stmts[i];
+ 
+                     ndx.remove(keyBuilder.statement2Key(spo.p, spo.o, spo.s));
+ 
+                 }
+             }
+ 
+             {
+ 
+                 IIndex ndx = getOSPIndex();
+ 
+                 // Place statements in OSP order.
+                 Arrays.sort(stmts, com.bigdata.rdf.inf.OSPComparator.INSTANCE);
+ 
+                 // Remove statements from OSP index.
+                 for (int i = 0; i < stmts.length; i++) {
+ 
+                     SPO spo = stmts[i];
+ 
+                     ndx.remove(keyBuilder.statement2Key(spo.o, spo.s, spo.p));
+ 
+                 }
+ 
+             }
+             
+         }
+ 
+         return rangeCount;
+ 
+     }
+ 
+     /**
+      * Value used for a "NULL" term identifier.
+      */
+     public static final long NULL = 0L;
  
      /**
       * Adds the statements to each index (batch api).
!      * <p>
       * Note: this is not sorting by the generated keys so the sort order may not
!      * perfectly reflect the natural order of the index. however, i suspect that
!      * it simply creates a few partitions out of the natural index order based
!      * on the difference between signed and unsigned interpretations of the
!      * termIds when logically combined into a statement identifier.
       * 
       * @param stmts
***************
*** 1193,1196 ****
--- 1580,1604 ----
          public long commitTime;
          
+         public long triplesPerSecond() {
+             
+             return ((long)( ((double)toldTriples) / ((double)totalTime) * 1000d ));
+             
+         }
+         
+         /**
+          * Human readable representation.
+          */
+         public String toString() {
+ 
+             return toldTriples+" stmts added in " + 
+                     ((double)loadTime) / 1000d +
+                     " secs, rate= " + 
+                     triplesPerSecond()+
+                     ", commitLatency="+
+                     commitTime+"ms" 
+                     ;
+ 
+         }
+         
      }
      
***************
*** 1270,1274 ****
                      ((double)loader.getInsertTime()) / 1000d +
                      " secs, rate= " + 
!                     loader.getInsertRate() 
                      );
  
--- 1678,1682 ----
                      ((double)loader.getInsertTime()) / 1000d +
                      " secs, rate= " + 
!                     loader.getInsertRate()
                      );

[cweb-CVS] bigdata-rdf/src/java/com/bigdata/rdf/model OptimizedValueFactory.java, 1.5, 1.6

From: Bryan T. <tho...@us...> - 2007-04-12 23:59:25

Update of /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/model
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2483/src/java/com/bigdata/rdf/model

Modified Files:
	OptimizedValueFactory.java 
Log Message:
Added a Sesame 1.x SAIL implementation.  This is NOT intended for production use.  It is just being done to gain a high-level query language integration for the triple store.

Index: OptimizedValueFactory.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/model/OptimizedValueFactory.java,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** OptimizedValueFactory.java	9 Feb 2007 20:18:57 -0000	1.5
--- OptimizedValueFactory.java	12 Apr 2007 23:59:21 -0000	1.6
***************
*** 71,74 ****
--- 71,124 ----
  public class OptimizedValueFactory implements ValueFactory {
  
+     /**
+      * Converts a {@link Value} for a different {@link ValueFactory} into a
+      * {@link _Value}.
+      * 
+      * @param v
+      *            The value.
+      * 
+      * @return The value iff it is a {@link _Value} and otherwise a
+      *         {@link _Value} with the same data.
+      */
+     public Value toNativeValue( Value v ) {
+         
+         if( v == null ) throw new IllegalArgumentException();
+         
+         if( v instanceof URI && ! ( v instanceof _URI) ) {
+             
+             v = createURI(v.toString());
+             
+         } else if( v instanceof Literal && ! ( v instanceof _Literal )) {
+             
+             String label = ((Literal)v).getLabel();
+             
+             String language = ((Literal)v).getLanguage();
+             
+             URI datatype = ((Literal)v).getDatatype();
+             
+             if( language != null ) {
+ 
+                 v = createLiteral(label,language);
+                 
+             } else if( datatype != null ) {
+                 
+                 v = createLiteral(label,createURI(datatype.toString()));
+                 
+             } else {
+                 
+                 v = createLiteral(label);
+                 
+             }
+             
+         } else if( v instanceof BNode && ! ( v instanceof _BNode )) {
+ 
+             v = createBNode( ((BNode)v).getID() );
+             
+         }
+         
+         return v;
+         
+     }
+     
      public BNode createBNode() {

[cweb-CVS] bigdata-rdf/src/java/com/bigdata/rdf/serializers TermIdSerializer.java, 1.1, 1.2 StatementSerializer.java, 1.1, 1.2 RdfValueSerializer.java, 1.2, 1.3 DefaultRdfValueSerializer.java, 1.1, NONE

From: Bryan T. <tho...@us...> - 2007-04-12 23:59:24

Update of /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/serializers
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2483/src/java/com/bigdata/rdf/serializers

Modified Files:
	TermIdSerializer.java StatementSerializer.java 
	RdfValueSerializer.java 
Removed Files:
	DefaultRdfValueSerializer.java 
Log Message:
Added a Sesame 1.x SAIL implementation.  This is NOT intended for production use.  It is just being done to gain a high-level query language integration for the triple store.

Index: TermIdSerializer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/serializers/TermIdSerializer.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** TermIdSerializer.java	9 Feb 2007 20:18:56 -0000	1.1
--- TermIdSerializer.java	12 Apr 2007 23:59:21 -0000	1.2
***************
*** 44,53 ****
  package com.bigdata.rdf.serializers;
  
! import java.io.DataInputStream;
! import java.io.DataOutputStream;
  import java.io.IOException;
  
  import org.CognitiveWeb.extser.LongPacker;
  
  import com.bigdata.objndx.IValueSerializer;
  
--- 44,53 ----
  package com.bigdata.rdf.serializers;
  
! import java.io.DataInput;
  import java.io.IOException;
  
  import org.CognitiveWeb.extser.LongPacker;
  
+ import com.bigdata.objndx.DataOutputBuffer;
  import com.bigdata.objndx.IValueSerializer;
  
***************
*** 73,77 ****
      public TermIdSerializer() {}
      
!     public void getValues(DataInputStream is, Object[] values, int n)
              throws IOException {
  
--- 73,77 ----
      public TermIdSerializer() {}
      
!     public void getValues(DataInput is, Object[] values, int n)
              throws IOException {
  
***************
*** 92,96 ****
      }
  
!     public void putValues(DataOutputStream os, Object[] values, int n)
              throws IOException {
  
--- 92,96 ----
      }
  
!     public void putValues(DataOutputBuffer os, Object[] values, int n)
              throws IOException {
  
***************
*** 99,103 ****
              if(packedLongs) {
  
!                 LongPacker.packLong(os, ((Long) values[i]).longValue());
                  
              } else {
--- 99,104 ----
              if(packedLongs) {
  
! //                LongPacker.packLong(os, ((Long) values[i]).longValue());
!                 os.packLong(((Long) values[i]).longValue());
                  
              } else {

--- DefaultRdfValueSerializer.java DELETED ---

Index: RdfValueSerializer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/serializers/RdfValueSerializer.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** RdfValueSerializer.java	29 Mar 2007 17:01:47 -0000	1.2
--- RdfValueSerializer.java	12 Apr 2007 23:59:21 -0000	1.3
***************
*** 44,49 ****
  package com.bigdata.rdf.serializers;
  
! import java.io.DataInputStream;
! import java.io.DataOutputStream;
  import java.io.IOException;
  
--- 44,49 ----
  package com.bigdata.rdf.serializers;
  
! import java.io.DataInput;
! import java.io.DataOutput;
  import java.io.IOException;
  
***************
*** 51,54 ****
--- 51,55 ----
  import org.openrdf.model.Value;
  
+ import com.bigdata.objndx.DataOutputBuffer;
  import com.bigdata.objndx.IIndex;
  import com.bigdata.objndx.IValueSerializer;
***************
*** 89,93 ****
      public RdfValueSerializer() {}
      
!     protected void writeUTF(DataOutputStream os,String s) throws IOException {
          
          if (utfCompression) {
--- 90,94 ----
      public RdfValueSerializer() {}
      
!     protected void writeUTF(DataOutputBuffer os,String s) throws IOException {
          
          if (utfCompression) {
***************
*** 95,99 ****
              byte[] data = UnicodeCompressor.compress(s);
  
!             LongPacker.packLong(os, data.length);
  
              os.write(data);
--- 96,101 ----
              byte[] data = UnicodeCompressor.compress(s);
  
! //            LongPacker.packLong(os, data.length);
!             os.packLong(data.length);
  
              os.write(data);
***************
*** 107,111 ****
      }
      
!     protected String readUTF(DataInputStream is) throws IOException {
          
          if(utfCompression) {
--- 109,113 ----
      }
      
!     protected String readUTF(DataInput is) throws IOException {
          
          if(utfCompression) {
***************
*** 127,131 ****
      }
      
!     public void getValues(DataInputStream is, Object[] vals, int n)
              throws IOException {
  
--- 129,133 ----
      }
      
!     public void getValues(DataInput is, Object[] vals, int n)
              throws IOException {
  
***************
*** 168,172 ****
      }
  
!     public void putValues(DataOutputStream os, Object[] vals, int n)
              throws IOException {
  
--- 170,174 ----
      }
  
!     public void putValues(DataOutputBuffer os, Object[] vals, int n)
              throws IOException {
  

Index: StatementSerializer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/serializers/StatementSerializer.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** StatementSerializer.java	9 Feb 2007 20:18:56 -0000	1.1
--- StatementSerializer.java	12 Apr 2007 23:59:21 -0000	1.2
***************
*** 44,51 ****
  package com.bigdata.rdf.serializers;
  
! import java.io.DataInputStream;
! import java.io.DataOutputStream;
  import java.io.IOException;
  
  import com.bigdata.objndx.IValueSerializer;
  
--- 44,51 ----
  package com.bigdata.rdf.serializers;
  
! import java.io.DataInput;
  import java.io.IOException;
  
+ import com.bigdata.objndx.DataOutputBuffer;
  import com.bigdata.objndx.IValueSerializer;
  
***************
*** 67,76 ****
      }
  
!     public void getValues(DataInputStream is, Object[] values, int n)
              throws IOException {
          return;
      }
  
!     public void putValues(DataOutputStream os, Object[] values, int n)
              throws IOException {
          return;
--- 67,76 ----
      }
  
!     public void getValues(DataInput is, Object[] values, int n)
              throws IOException {
          return;
      }
  
!     public void putValues(DataOutputBuffer os, Object[] values, int n)
              throws IOException {
          return;

[cweb-CVS] bigdata-rdf/src/test/com/bigdata/rdf/inf TestFullForwardClosure.java, 1.4, 1.5

From: Bryan T. <tho...@us...> - 2007-04-12 23:59:24

Update of /cvsroot/cweb/bigdata-rdf/src/test/com/bigdata/rdf/inf
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2483/src/test/com/bigdata/rdf/inf

Modified Files:
	TestFullForwardClosure.java 
Log Message:
Added a Sesame 1.x SAIL implementation.  This is NOT intended for production use.  It is just being done to gain a high-level query language integration for the triple store.

Index: TestFullForwardClosure.java
===================================================================
RCS file: /cvsroot/cweb/bigdata-rdf/src/test/com/bigdata/rdf/inf/TestFullForwardClosure.java,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** TestFullForwardClosure.java	22 Feb 2007 16:58:58 -0000	1.4
--- TestFullForwardClosure.java	12 Apr 2007 23:59:21 -0000	1.5
***************
*** 80,90 ****
  
          /*
-          * @todo this is committing the data first we do not want to do if we
-          * know that we are closing the store.
-          * 
           * @todo use a dataset that we can add to CVS for a performance test and
           * hand-crafted data sets to test the rule implementations.
           */
!         store.loadData(new File("data/alibaba_v41.rdf"),"");
  //        store.loadData(new File("data/nciOncology.owl"),"");
  
--- 80,87 ----
  
          /*
           * @todo use a dataset that we can add to CVS for a performance test and
           * hand-crafted data sets to test the rule implementations.
           */
!         store.loadData(new File("data/alibaba_v41.rdf"),"",false);
  //        store.loadData(new File("data/nciOncology.owl"),"");

[cweb-CVS] bigdata-rdf/src/test/com/bigdata/rdf/sail - New directory

From: Bryan T. <tho...@us...> - 2007-04-12 23:59:13

Update of /cvsroot/cweb/bigdata-rdf/src/test/com/bigdata/rdf/sail
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2466/src/test/com/bigdata/rdf/sail

Log Message:
Directory /cvsroot/cweb/bigdata-rdf/src/test/com/bigdata/rdf/sail added to the repository

[cweb-CVS] bigdata-rdf/src/java/com/bigdata/rdf/sail - New directory

From: Bryan T. <tho...@us...> - 2007-04-12 23:59:13

Update of /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/sail
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv2466/src/java/com/bigdata/rdf/sail

Log Message:
Directory /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/sail added to the repository

[cweb-CVS] bigdata/src/java/com/bigdata/objndx DataInputBuffer.java, NONE, 1.1 DataOutputBuffer.java, NONE, 1.1 NodeSerializer.java, 1.34, 1.35 AbstractBTree.java, 1.22, 1.23 IndexSegmentBuilder.java, 1.30, 1.31 Leaf.java, 1.23, 1.24 IndexSegment.java, 1.19, 1.20

From: Bryan T. <tho...@us...> - 2007-04-10 18:34:22

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv24584/src/java/com/bigdata/objndx

Modified Files:
	NodeSerializer.java AbstractBTree.java 
	IndexSegmentBuilder.java Leaf.java IndexSegment.java 
Added Files:
	DataInputBuffer.java DataOutputBuffer.java 
Log Message:
Fixed a bug in UUID assignment to index segments.

Added some logic for fast data output for btree nodes and leaves.

Index: IndexSegmentBuilder.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/IndexSegmentBuilder.java,v
retrieving revision 1.30
retrieving revision 1.31
diff -C2 -d -r1.30 -r1.31
*** IndexSegmentBuilder.java	27 Mar 2007 17:11:41 -0000	1.30
--- IndexSegmentBuilder.java	10 Apr 2007 18:33:31 -0000	1.31
***************
*** 1470,1474 ****
                      plan.nentries, maxNodeOrLeafLength, addrLeaves, addrNodes,
                      addrRoot, addrExtensionMetadata, addrBloom, errorRate, out
!                             .length(), segmentUUID, indexUUID, now);
  
              md.write(out);
--- 1470,1474 ----
                      plan.nentries, maxNodeOrLeafLength, addrLeaves, addrNodes,
                      addrRoot, addrExtensionMetadata, addrBloom, errorRate, out
!                             .length(), indexUUID, segmentUUID, now);
  
              md.write(out);

--- NEW FILE: DataOutputBuffer.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Apr 7, 2007
 */

package com.bigdata.objndx;

import java.io.ByteArrayOutputStream;
import java.io.DataOutput;
import java.io.DataOutputStream;
import java.io.IOException;
import java.nio.ByteBuffer;

import com.bigdata.io.ByteBufferOutputStream;

/**
 * Fast special purpose serialization onto a managed byte[] buffer.
 * 
 * @todo IO costs might be reduced by special purpose routines using a counter
 *       and a pre-/re-sized byte[] with direct byte[] access. This code would
 *       be similar to the {@link KeyBuilder} in its handling of the byte[], but
 *       would use standard Java data types rather than their unsigned variants
 *       and would support packing of unsigned short, int, and long values. I
 *       estimate 29% of the costs of the index load performance benchmark is
 *       serialization with 13% of the total cost being the
 *       {@link DataOutputStream} over a {@link ByteArrayOutputStream} or
 *       {@link ByteBufferOutputStream}. The class could be a
 *       {@link DataOutput} implementation that had intimate knowledge of a
 *       backing byte[]. If we go this route, then
 *       {@link DataOutput#writeChars(String)} and
 *       {@link DataOutput#writeUTF(String)} might throw exceptions since they
 *       are not used when (de-)serializing nodes and leaves.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public class DataOutputBuffer implements DataOutput {

    /**
     * The default capacity of the buffer.
     */
    final public static int DEFAULT_INITIAL_CAPACITY = 1024;

    /**
     * A non-negative integer specifying the #of bytes of data in the buffer
     * that contain valid data starting from position zero(0).
     */
    protected int len;
    
    /**
     * The buffer. This is re-allocated whenever the capacity of the buffer
     * is too small and reused otherwise.
     */
    protected byte[] buf;
    
    /**
     * Throws exception unless the value is non-negative.
     * 
     * @param msg
     *            The exception message.
     * @param v
     *            The value.
     * 
     * @return The value.
     * 
     * @exception IllegalArgumentException
     *                unless the value is non-negative.
     */
    protected static int assertNonNegative(String msg,int v) {
       
        if(v<0) throw new IllegalArgumentException(msg);
        
        return v;
        
    }
    
    /**
     * Uses an initial buffer capacity of <code>1024</code> bytes.
     */
    public DataOutputBuffer() {
        
        this(DEFAULT_INITIAL_CAPACITY);
        
    }
    
    /**
     * @param initialCapacity
     *            The initial capacity of the internal byte[].
     */
    public DataOutputBuffer(int initialCapacity) {
        
        this(0, new byte[assertNonNegative("initialCapacity", initialCapacity)]);
        
    }

    /**
     * @param len
     *            The #of bytes of data in the provided buffer.
     * @param buf
     *            The buffer, with <i>len</i> pre-existing bytes of valid data.
     *            The buffer reference is used directly rather than making a
     *            copy of the data.
     */
    public DataOutputBuffer(int len, byte[] buf) {

        if (len < 0)
            throw new IllegalArgumentException("len");

        if (buf == null)
            throw new IllegalArgumentException("buf");

        if (len > buf.length)
            throw new IllegalArgumentException("len>buf.length");

        this.len = len;

        this.buf = buf;

    }

    /**
     * The #of bytes of data in the key.
     */
    final public int getLength() {
        
        return len;
        
    }

    /**
     * Ensure that at least <i>len</i> bytes are free in the buffer. The
     * {@link #buf buffer} may be grown by this operation but it will not be
     * truncated.
     * <p>
     * This operation is equivilent to
     * 
     * <pre>
     * ensureCapacity(this.len + len)
     * </pre>
     * 
     * and the latter is often used as an optimization.
     * 
     * @param len
     *            The minimum #of free bytes.
     */
    final public void ensureFree(int len) {
        
        ensureCapacity(this.len + len );
        
    }

    /**
     * Ensure that the buffer capacity is a least <i>capacity</i> total bytes.
     * The {@link #buf buffer} may be grown by this operation but it will not be
     * truncated.
     * 
     * @param capacity
     *            The minimum #of bytes in the buffer.
     */
    final public void ensureCapacity(int capacity) {
        
        if(capacity<0) throw new IllegalArgumentException();
//        assert capacity >= 0;
        
        final int overflow = capacity - buf.length;
        
        if(overflow>0) {
        
            /*
             * extend to at least the target capacity.
             */
            final byte[] tmp = new byte[extend(capacity)];
            
            // copy only the defined bytes.
            System.arraycopy(buf, 0, tmp, 0, this.len);
            
            buf = tmp;
            
        }

    }

    /**
     * Return the new capacity for the buffer (default is always large enough
     * and will normally double the buffer capacity each time it overflows).
     * 
     * @param required
     *            The minimum required capacity.
     * 
     * @return The new capacity.
     */
    protected int extend(int required) {

        return Math.max(required, buf.length * 2);
        
    }

    /**
     * Return a copy of the buffer.
     * 
     * @return A new array containing data in the buffer.
     * 
     * @see #wrap()
     */
    final public byte[] toByteArray() {
        
        byte[] tmp = new byte[this.len];
        
        System.arraycopy(buf, 0, tmp, 0, this.len);
        
        return tmp;
        
    }

    /**
     * Wraps up a reference to the data in a {@link ByteBuffer}
     * 
     * @return A {@link ByteBuffer} encapsulating a reference to the data in the
     *         current buffer. The data will be overwritten if {@link #reset()}
     *         is invoked followed by any operations that write on the buffer.
     */
    final public ByteBuffer wrap() {

        return ByteBuffer.wrap(buf, 0, len);
        
    }
    
//    /**
//     * Copy the data from the internal buffer into the supplied buffer.
//     * 
//     * @param b
//     *            A byte[].
//     * 
//     * @exception IndexOutOfBoundsException
//     *                if the supplied buffer is not large enough.
//     */
//    final public void copy(byte[] b) {
//    
//        System.arraycopy(this.buf, 0, b, 0, this.len);
//        
//    }
    
    /**
     * Prepares the buffer for new data by resetting the length to zero.
     * 
     * @return This buffer.
     */
    final public DataOutputBuffer reset() {
        
        len = 0;
        
        return this;
        
    }

    final public void write(int b) throws IOException {

        if (len + 1 > buf.length)
            ensureCapacity(len + 1);

        buf[len++] = (byte) (b & 0xff);

    }

    final public void write(byte[] b) throws IOException {

        write(b,0,b.length);

    }

    final public void write(byte[] b, int off, int len) throws IOException {

      ensureFree(len);
      
      System.arraycopy(b, off, buf, this.len, len);
      
      this.len += len;

    }

    final public void writeBoolean(boolean v) throws IOException {

        if (len + 1 > buf.length)
            ensureCapacity(len + 1);

        buf[len++] = v ? (byte)1 : (byte)0;

    }

    final public void writeByte(int v) throws IOException {

        if (len + 1 > buf.length)
            ensureCapacity(len + 1);

        buf[len++] = (byte) (v & 0xff);

    }

    final public void writeDouble(double d) throws IOException {

        if (len + 8 > buf.length)
            ensureCapacity(len + 8);

        long v = Double.doubleToLongBits(d);

        // big-endian.
        buf[len++] = (byte) (v >>> 56);
        buf[len++] = (byte) (v >>> 48);
        buf[len++] = (byte) (v >>> 40);
        buf[len++] = (byte) (v >>> 32);
        buf[len++] = (byte) (v >>> 24);
        buf[len++] = (byte) (v >>> 16);
        buf[len++] = (byte) (v >>> 8);
        buf[len++] = (byte) (v >>> 0);

    }

    final public void writeFloat(float f) throws IOException {

        if (len + 4 > buf.length)
            ensureCapacity(len + 4);

        int v = Float.floatToIntBits(f);

        buf[len++] = (byte) (v >>> 24);
        buf[len++] = (byte) (v >>> 16);
        buf[len++] = (byte) (v >>> 8);
        buf[len++] = (byte) (v >>> 0);

    }

    final public void writeInt(int v) throws IOException {

        if (len + 4 > buf.length)
            ensureCapacity(len + 4);

        buf[len++] = (byte) (v >>> 24);
        buf[len++] = (byte) (v >>> 16);
        buf[len++] = (byte) (v >>> 8);
        buf[len++] = (byte) (v >>> 0);

    }

    final public void writeLong(long v) throws IOException {

        if (len + 8 > buf.length)
            ensureCapacity(len + 8);

        // big-endian.
        buf[len++] = (byte) (v >>> 56);
        buf[len++] = (byte) (v >>> 48);
        buf[len++] = (byte) (v >>> 40);
        buf[len++] = (byte) (v >>> 32);
        buf[len++] = (byte) (v >>> 24);
        buf[len++] = (byte) (v >>> 16);
        buf[len++] = (byte) (v >>> 8);
        buf[len++] = (byte) (v >>> 0);

    }

    final public void writeShort(int v) throws IOException {

        if (len + 2 > buf.length)
            ensureCapacity(len + 2);

        // big-endian
        buf[len++] = (byte) (v >>> 8);
        buf[len++] = (byte) (v >>> 0);

    }

    final public void writeChar(int v) throws IOException {

        if (len + 2 > buf.length)
            ensureCapacity(len + 2);

        buf[len++] = (byte) (v >>> 8);
        buf[len++] = (byte) (v >>> 0);

    }

    public void writeBytes(String s) throws IOException {

        int len = s.length();
        
        for (int i = 0 ; i < len ; i++) {
            
            write((byte)s.charAt(i));
            
        }

    }

    public void writeChars(String s) throws IOException {

        int len = s.length();
        
        for (int i = 0 ; i < len ; i++) {
        
            int v = s.charAt(i);
            
            write((v >>> 8) & 0xFF); 
            
            write((v >>> 0) & 0xFF); 
        
        }
        
    }
    
    /**
     * Note: This is not wildly efficient (it would be fine if
     * DataOutputStream#writeUTF(String str, DataOutput out)} was public) but
     * the use cases for serializing the nodes and leaves of a btree do not
     * suggest any requirement for Unicode (if you assume that the application
     * values are already being serialized as byte[]s - which is always true
     * when there is a client-server divide).
     */
    public void writeUTF(String str) throws IOException {
        
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        
        DataOutputStream dos = new DataOutputStream(baos);
        
        dos.writeUTF(str);
        
        dos.flush();
        
        write(baos.toByteArray());

    }

    /*
     * Pack unsigned long integer.
     */
    
    /**
     * Packs a non-negative long value into the minimum #of bytes in which the
     * value can be represented and writes those bytes onto the output stream.
     * The first byte determines whether or not the long value was packed and,
     * if packed, how many bytes were required to represent the packed long
     * value. When the high bit of the first byte is a one (1), then the long
     * value could not be packed and the long value is found by clearing the
     * high bit and interpreting the first byte plus the next seven (7) bytes as
     * a long. Otherwise the next three (3) bits are interpreted as an unsigned
     * integer giving the #of bytes (nbytes) required to represent the packed
     * long value. To recover the long value the high nibble is cleared and the
     * first byte together with the next nbytes are interpeted as an unsigned
     * long value whose leading zero bytes were not written.
     * 
     * <pre>
     *    
     * [0|1|2|3|4|5|6|7]
     *  1 - - -   nbytes = 8, clear high bit and interpret this plus the next 7 bytes as a long.
     *  0 1 1 1   nbytes = 7, clear high nibble and interpret this plus the next 6 bytes as a long. 
     *  0 1 1 0   nbytes = 6, clear high nibble and interpret this plus the next 5 bytes as a long. 
     *  0 1 0 1   nbytes = 5, clear high nibble and interpret this plus the next 4 bytes as a long.
     *  0 1 0 0   nbytes = 4, clear high nibble and interpret this plus the next 3 bytes as a long.
     *  0 0 1 1   nbytes = 3, clear high nibble and interpret this plus the next 3 bytes as a long.
     *  0 0 1 0   nbytes = 2, clear high nibble and interpret this plus the next byte as a long.
     *  0 0 0 1   nbytes = 1, clear high nibble.  value is the low nibble.
     *  
     * </pre>
     * 
     * @param v The unsigned long value.
     * 
     * @return The #of bytes onto which the unsigned long value was packed.
     */
    final public int packLong( long v ) throws IOException {
        
        /*
         * You can only pack non-negative long values with this method.
         */
        
        if (v < 0) {
            
            throw new IllegalArgumentException("negative value: v=" + v);
            
        }
        
        /*
         * If the high byte is non-zero then we will write the value as a normal
         * long and return nbytes == 8. This case handles large positive long
         * values.
         */
        if( ( v >> 56 ) != 0 ) {
            pbuf[0] = ( (byte)((0xff & (v >> 56))|0x80) ); // note: set the high bit.
            pbuf[1] = ( (byte)(0xff & (v >> 48)) );
            pbuf[2] = ( (byte)(0xff & (v >> 40)) );
            pbuf[3] = ( (byte)(0xff & (v >> 32)) );
            pbuf[4] = ( (byte)(0xff & (v >> 24)) );
            pbuf[5] = ( (byte)(0xff & (v >> 16)) );
            pbuf[6] = ( (byte)(0xff & (v >>  8)) );
            pbuf[7] = ( (byte)(0xff & v) );
            write(pbuf, 0, 8);
            return 8;
        }
        
        // #of nibbles required to represent the long value.
        final int nnibbles = getNibbleLength( v );

        /*
         * Is [nnibbles] even? (If it is even then we need to pad out an extra
         * zero nibble in the first byte.)
         */
        final boolean evenNibbleCount = ( nnibbles == ( ( nnibbles >> 1 ) << 1 ) );
        
        // #of bytes required to represent the long value (plus the header nibble).
        final int nbytes = ( ( nnibbles +1 ) >> 1 ) + (evenNibbleCount?1:0);
        
        int nwritten = 0;
        
        if( evenNibbleCount ) {
            
            /*
             * An even nibble count requires that we pad the low nibble of the
             * first byte with zeros.
             */
        
            // header byte. low nibble is empty.
            byte b = (byte) ( nbytes << 4 );
            
            pbuf[nwritten++] = b;
            
            // remaining bytes containing the packed value.
            for( int i=(nnibbles-2)<<2; i>=0; i-=8 ) {
                
                b = (byte) (0xff & (v >> i));
                
                pbuf[nwritten++] = b;
                
            }
            
        } else {
            
            /*
             * An odd nibble count means that we pack the first nibble of the
             * long value into the low nibble of the header byte. In this case
             * the first nibble will always be the low nibble of the first
             * non-zero byte in the long value (the high nibble of that byte
             * must be zero since there is an odd nibble count).
             */
            
            byte highByte = (byte) (0xff & (v >> ((nbytes-1)*8) ));
            
            byte b = (byte) ( ( nbytes << 4 ) | highByte );
            
            pbuf[nwritten++] = b;
            
            for( int i=(nnibbles-3)<<2; i>=0; i-=8 ) {
            
                b = (byte) (0xff & (v >> i));
                
                pbuf[nwritten++] = b;
                
            }
            
        }
        
        write(pbuf,0,nwritten);
        
        return nwritten;
        
    }

    /**
     * Private buffer for packing long integers.
     */
    private byte[] pbuf = new byte[8];
    
    /**
     * Return the #of non-zero nibbles, counting from the first non-zero nibble
     * in the long value. A value of <code>0L</code> is considered to be one
     * nibble for our purposes.
     * 
     * @param v
     *            The long value.
     * 
     * @return The #of nibbles in [1:16].
     */
    static protected final int getNibbleLength( long v )
    {

        for( int i=56, j=16; i>=0; i-=8, j-=2 ) {
            
            if( (0xf0 & (v >> i)) != 0 ) return j;
            
            if( (0x0f & (v >> i)) != 0 ) return j-1;
            
        }
        
        if (v != 0)
            throw new AssertionError("v=" + v);
     
        /*
         * value is zero, which is considered to be one nibble for our purposes.
         */

        return 1;
        
    }
    
    /*
     * Pack unsigned short integer.
     */
    
    /**
     * Packs a non-negative short value into one or two bytes and writes them on
     * <i>os </i>. A short in [0:127] is packed into one byte. Larger values are
     * packed into two bytes. The high bit of the first byte is set if the value
     * was packed into two bytes. If the bit is set, clear the high bit, read
     * the next byte, and interpret the two bytes as a short value. Otherwise
     * interpret the byte as a short value.
     * 
     * @param v The unsigned short integer.
     * 
     * @return The #of bytes into which the value was packed.
     */ 
    final public int packShort( short v ) throws IOException
    {
    
        /*
         * You can only pack non-negative values with this method.
         */
        if( v < 0 ) {
            throw new IllegalArgumentException( "negative value: v="+v );
        }
        if( v > 127 ) {
            // the value requires two bytes.
            buf[len++] = ( (byte)((0xff & (v >> 8))|0x80) ); // note: set the high bit.
            buf[len++] = ( (byte)(0xff & v) );
            return 2;
        } else {
            // the value fits in one byte.
            buf[len++] = ( (byte)(0xff & v) );
            return 1;
        }
    }

}

Index: IndexSegment.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/IndexSegment.java,v
retrieving revision 1.19
retrieving revision 1.20
diff -C2 -d -r1.19 -r1.20
*** IndexSegment.java	27 Mar 2007 14:34:22 -0000	1.19
--- IndexSegment.java	10 Apr 2007 18:33:31 -0000	1.20
***************
*** 100,107 ****
                  BTree.DEFAULT_HARD_REF_QUEUE_SCAN));
  
-         // report on the event.
-         ResourceManager.openIndexSegment(null/* name */, fileStore.getFile()
-                 .toString(), fileStore.size());
- 
      }
  
--- 100,103 ----
***************
*** 142,145 ****
--- 138,145 ----
          _open();
  
+         // report on the event.
+         ResourceManager.openIndexSegment(null/* name */, fileStore.getFile()
+                 .toString(), fileStore.size());
+ 
      }
  
***************
*** 311,314 ****
--- 311,315 ----
  
              } else {
+                 
                  _key = unbox(key);
  

Index: Leaf.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/Leaf.java,v
retrieving revision 1.23
retrieving revision 1.24
diff -C2 -d -r1.23 -r1.24
*** Leaf.java	13 Feb 2007 23:01:02 -0000	1.23
--- Leaf.java	10 Apr 2007 18:33:31 -0000	1.24
***************
*** 48,51 ****
--- 48,52 ----
  
  import java.io.PrintStream;
+ import java.lang.ref.WeakReference;
  import java.util.Arrays;
  import java.util.Iterator;
***************
*** 775,779 ****
  
          btree.touch(this);
!         
          final int entryIndex = keys.search(searchKey);
  
--- 776,780 ----
  
          btree.touch(this);
! 
          final int entryIndex = keys.search(searchKey);
  
***************
*** 795,799 ****
       * Looks up one or more tuples and reports whether or not they exist.
       * 
! -     * @return The #of tuples processed.
       * 
       * @todo optimize batch lookup here (also when bloom filter is available on
--- 796,800 ----
       * Looks up one or more tuples and reports whether or not they exist.
       * 
!      * @return The #of tuples processed.
       * 
       * @todo optimize batch lookup here (also when bloom filter is available on

Index: AbstractBTree.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/AbstractBTree.java,v
retrieving revision 1.22
retrieving revision 1.23
diff -C2 -d -r1.22 -r1.23
*** AbstractBTree.java	4 Apr 2007 16:52:16 -0000	1.22
--- AbstractBTree.java	10 Apr 2007 18:33:31 -0000	1.23
***************
*** 49,52 ****
--- 49,53 ----
  
  import java.io.PrintStream;
+ import java.lang.ref.WeakReference;
  import java.nio.ByteBuffer;
  import java.util.Iterator;
***************
*** 132,141 ****
       * flag turns on some more expensive assertions.
       */
!     final protected boolean debug = DEBUG||true;
  
      /**
       * Counters tracking various aspects of the btree.
       */
!     /*protected*/ public final Counters counters = new Counters(this);
  
      /**
--- 133,142 ----
       * flag turns on some more expensive assertions.
       */
!     final protected boolean debug = DEBUG || true;
  
      /**
       * Counters tracking various aspects of the btree.
       */
!     /* protected */public final Counters counters = new Counters(this);
  
      /**
***************
*** 153,157 ****
       */
      final protected UUID indexUUID;
!     
      /**
       * The branching factor for the btree.
--- 154,158 ----
       */
      final protected UUID indexUUID;
! 
      /**
       * The branching factor for the btree.
***************
*** 175,179 ****
       */
      protected AbstractNode root;
!     
      /**
       * Used to serialize and de-serialize the nodes and leaves of the tree.
--- 176,205 ----
       */
      protected AbstractNode root;
! 
! //    /**
! //     * The finger is a trial feature. The purpose is to remember the last
! //     * leaf(s) in the tree that was visited by a search operation and to
! //     * pre-test that those leaf(s) on the next search operation.
! //     * <p>
! //     * Fingers can do wonders by reducing search in common cases where the same
! //     * leaf is visited by a sequence of operations, e.g., sequential insert or
! //     * conditional insert realized by lookup + insert. Multiple fingers may be
! //     * required if the tree is used by concurrent readers (and the finger might
! //     * be part of the reader state) while a single finger would do for a tree
! //     * being used by a writer (since concurrent modification is not supported).
! //     * <p>
! //     * The finger is set each time a leaf is found as the result of a search. It
! //     * is tested by each search operation before walking the tree. The finger is
! //     * a {@link WeakReference} so that we do not need to worry about having a
! //     * hard reference to an arbitrary part of the tree.
! //     * 
! //     * @todo Does the finger need to encapsulate the hard references for the
! //     *       parents in order to ensure that the parents remain strongly
! //     *       reachable? (Parents are only weakly reachable from their children.)
! //     */
! //    private WeakReference<Leaf> finger;
! //
! //    final private boolean useFinger = false;
! 
      /**
       * Used to serialize and de-serialize the nodes and leaves of the tree.
***************
*** 193,197 ****
       */
      public int referenceCount = 0;
!     
      /**
       * Leaves (and nodes) are added to a hard reference queue when they are
--- 219,223 ----
       */
      public int referenceCount = 0;
! 
      /**
       * Leaves (and nodes) are added to a hard reference queue when they are
***************
*** 343,348 ****
          assert nodeFactory != null;
  
!         if(indexUUID == null) throw new IllegalArgumentException("indexUUID");
!         
          this.store = store;
  
--- 369,375 ----
          assert nodeFactory != null;
  
!         if (indexUUID == null)
!             throw new IllegalArgumentException("indexUUID");
! 
          this.store = store;
  
***************
*** 356,360 ****
  
          this.indexUUID = indexUUID;
!         
      }
  
--- 383,387 ----
  
          this.indexUUID = indexUUID;
! 
      }
  
***************
*** 366,370 ****
       * store). The index reference remains valid after a {@link #close()}. A
       * closed index is transparently restored by either {@link #getRoot()} or
!      * {@link #reopen()}. 
       * <p>
       * This implementation clears the hard reference queue (releasing all node
--- 393,397 ----
       * store). The index reference remains valid after a {@link #close()}. A
       * closed index is transparently restored by either {@link #getRoot()} or
!      * {@link #reopen()}.
       * <p>
       * This implementation clears the hard reference queue (releasing all node
***************
*** 389,409 ****
      public void close() {
  
!         if(root==null) {
  
              throw new IllegalStateException("Already closed");
!             
          }
  
          if (root.dirty) {
!             
              throw new IllegalStateException("Root node is dirty");
!             
          }
!         
          /*
           * Release buffers.
           */
          nodeSer.close();
!         
          /*
           * Clear the hard reference queue (this will not trigger any writes
--- 416,436 ----
      public void close() {
  
!         if (root == null) {
  
              throw new IllegalStateException("Already closed");
! 
          }
  
          if (root.dirty) {
! 
              throw new IllegalStateException("Root node is dirty");
! 
          }
! 
          /*
           * Release buffers.
           */
          nodeSer.close();
! 
          /*
           * Clear the hard reference queue (this will not trigger any writes
***************
*** 411,422 ****
           */
          leafQueue.evictAll(true);
!         
          /*
           * Clear the reference to the root node (permits GC).
           */
          root = null;
!         
      }
!     
      /**
       * This is part of a {@link #close()}/{@link #reopen()} protocol that may
--- 438,449 ----
           */
          leafQueue.evictAll(true);
! 
          /*
           * Clear the reference to the root node (permits GC).
           */
          root = null;
! 
      }
! 
      /**
       * This is part of a {@link #close()}/{@link #reopen()} protocol that may
***************
*** 432,436 ****
       */
      abstract protected void reopen();
!     
      /**
       * An "open" index has its buffers and root node in place rather than having
--- 459,463 ----
       */
      abstract protected void reopen();
! 
      /**
       * An "open" index has its buffers and root node in place rather than having
***************
*** 444,452 ****
       */
      final public boolean isOpen() {
!         
          return root != null;
!         
      }
!     
      /**
       * The backing store.
--- 471,479 ----
       */
      final public boolean isOpen() {
! 
          return root != null;
! 
      }
! 
      /**
       * The backing store.
***************
*** 515,523 ****
       */
      final public UUID getIndexUUID() {
!         
          return indexUUID;
!         
      }
!     
      /**
       * The root of the btree. This is initially a leaf until the leaf is split,
--- 542,550 ----
       */
      final public UUID getIndexUUID() {
! 
          return indexUUID;
! 
      }
! 
      /**
       * The root of the btree. This is initially a leaf until the leaf is split,
***************
*** 532,537 ****
  
          // make sure that the root is defined.
!         if(root == null) reopen();
!         
          return root;
  
--- 559,565 ----
  
          // make sure that the root is defined.
!         if (root == null)
!             reopen();
! 
          return root;
  
***************
*** 550,572 ****
       */
      public void addAll(AbstractBTree src) {
!         
!         if(src==null) throw new IllegalArgumentException();
!         
!         if(src==this) throw new IllegalArgumentException();
!         
          IEntryIterator itr = src.entryIterator();
!         
!         while(itr.hasNext()) {
!         
              Object val = itr.next();
!             
              byte[] key = itr.getKey();
!             
!             insert(key,val);
!             
          }
!         
      }
!     
      public void insert(BatchInsert op) {
  
--- 578,602 ----
       */
      public void addAll(AbstractBTree src) {
! 
!         if (src == null)
!             throw new IllegalArgumentException();
! 
!         if (src == this)
!             throw new IllegalArgumentException();
! 
          IEntryIterator itr = src.entryIterator();
! 
!         while (itr.hasNext()) {
! 
              Object val = itr.next();
! 
              byte[] key = itr.getKey();
! 
!             insert(key, val);
! 
          }
! 
      }
! 
      public void insert(BatchInsert op) {
  
***************
*** 639,646 ****
  
      public void contains(BatchContains op) {
!         
          final int ntuples = op.ntuples;
!         
!         while( op.tupleIndex < ntuples ) {
  
              // skip tuples already marked as true.
--- 669,676 ----
  
      public void contains(BatchContains op) {
! 
          final int ntuples = op.ntuples;
! 
!         while (op.tupleIndex < ntuples) {
  
              // skip tuples already marked as true.
***************
*** 669,676 ****
  
      public void remove(BatchRemove op) {
!         
          final int ntuples = op.ntuples;
!         
!         while( op.tupleIndex < ntuples) {
  
              /*
--- 699,706 ----
  
      public void remove(BatchRemove op) {
! 
          final int ntuples = op.ntuples;
! 
!         while (op.tupleIndex < ntuples) {
  
              /*
***************
*** 710,716 ****
       */
      final protected byte[] unbox(Object key) {
!      
          return keyBuilder.reset().append(((Integer) key).intValue()).getKey();
!         
      }
  
--- 740,820 ----
       */
      final protected byte[] unbox(Object key) {
! 
          return keyBuilder.reset().append(((Integer) key).intValue()).getKey();
! 
!     }
! 
!     /**
!      * Returns the node or leaf to be used for search. This implementation is
!      * aware of the {@link #finger} and will return it if the key lies within
!      * the finger.
!      * 
!      * @param key
!      *            The key.
!      * 
!      * @return Either the root node of the tree or a recently touched leaf that
!      *         is known to span the key.
!      */
!     protected AbstractNode getRootOrFinger(byte[] key) {
! 
!         return getRoot();
! //        
! //        if (finger == null)
! //            return getRoot();
! //
! //        Leaf leaf = finger.get();
! //
! //        if (leaf == null || leaf.deleted) {
! //
! //            // @todo clear finger on delete/copyOnWrite
! //
! //            return getRoot();
! //
! //        }
! //
! //        int entryIndex = leaf.keys.search(key);
! //
! //        if (entryIndex >= 0) {
! //
! //            /*
! //             * There is an exact match on the finger so we know that this key
! //             * belongs in this leaf.
! //             */
! //
! //            return leaf;
! //
! //        }
! //
! //        if (entryIndex < 0) {
! //
! //            // Convert the position to obtain the insertion point.
! //            entryIndex = -entryIndex - 1;
! //
! //            if (entryIndex > 0 && entryIndex < leaf.nkeys - 1) {
! //
! //                /*
! //                 * The key belongs somewhere in between the first and the last
! //                 * key in the leaf. For all those positions, we are again
! //                 * guarenteed that this key belongs within this leaf.
! //                 */
! //
! //                return leaf;
! //
! //            }
! //
! //        }
! //
! //        /*
! //         * The key might belong in the leaf, but we are not sure since it lies
! //         * on either the low or the high edge of the leaf.
! //         * 
! //         * @todo We could disambiguate this using the actual value of the
! //         * separator keys on the parent. If the key is greater than or equal to
! //         * the separatorKey for this leaf and strictly less than the
! //         * separatorKey for the rightSibling, then it belongs in this leaf.
! //         */
! //
! //        return getRoot();
! 
      }
  
***************
*** 722,735 ****
          counters.ninserts++;
  
!         if (key instanceof byte[]) {
!         
!             return getRoot().insert((byte[]) key,value);
!             
!         } else {
!             
!             return getRoot().insert( unbox(key), value );
!             
          }
!         
      }
  
--- 826,837 ----
          counters.ninserts++;
  
!         if (!(key instanceof byte[])) {
! 
!             key = unbox(key);
! 
          }
! 
!         return getRootOrFinger((byte[]) key).insert((byte[]) key, value);
! 
      }
  
***************
*** 741,754 ****
          counters.nfinds++;
  
!         if (key instanceof byte[]) {
  
-             return getRoot().lookup((byte[])key);
-             
-         } else {
-             
-             return getRoot().lookup(unbox(key));
-             
          }
  
      }
  
--- 843,854 ----
          counters.nfinds++;
  
!         if (!(key instanceof byte[])) {
! 
!             key = unbox(key);
  
          }
  
+         return getRootOrFinger((byte[]) key).lookup((byte[]) key);
+ 
      }
  
***************
*** 759,764 ****
  
          counters.nfinds++;
!         
!         return getRoot().contains((byte[])key);
  
      }
--- 859,864 ----
  
          counters.nfinds++;
! 
!         return getRootOrFinger((byte[]) key).contains((byte[]) key);
  
      }
***************
*** 771,784 ****
          counters.nremoves++;
  
!         if (key instanceof byte[]) {
!         
!             return getRoot().remove((byte[])key);
!             
!         } else {
!             
!             return getRoot().remove(unbox(key));
!             
          }
  
      }
  
--- 871,882 ----
          counters.nremoves++;
  
!         if (!(key instanceof byte[])) {
! 
!             key = unbox(key);
! 
          }
  
+         return getRootOrFinger((byte[]) key).remove((byte[]) key);
+ 
      }
  
***************
*** 790,794 ****
          counters.nindexOf++;
  
!         int index = getRoot().indexOf(key);
  
          return index;
--- 888,892 ----
          counters.nindexOf++;
  
!         int index = getRootOrFinger((byte[]) key).indexOf(key);
  
          return index;
***************
*** 839,843 ****
  
          AbstractNode root = getRoot();
!         
          int fromIndex = (fromKey == null ? 0 : root.indexOf(fromKey));
  
--- 937,941 ----
  
          AbstractNode root = getRoot();
! 
          int fromIndex = (fromKey == null ? 0 : root.indexOf(fromKey));
  
***************
*** 983,990 ****
  
          if (root != null) {
!             
              return root.dump(level, out, 0, true);
!             
!         } else return true;
  
      }
--- 1081,1089 ----
  
          if (root != null) {
! 
              return root.dump(level, out, 0, true);
! 
!         } else
!             return true;
  
      }
***************
*** 1059,1062 ****
--- 1158,1162 ----
               * Also see {@link DefaultEvictionListener}.
               */
+ 
              if (node.referenceCount == 1) {
  
***************
*** 1067,1070 ****
--- 1167,1180 ----
          }
  
+ //        if (useFinger && node instanceof ILeafData) {
+ //
+ //            if (finger == null || finger.get() != node) {
+ //
+ //                finger = new WeakReference<Leaf>((Leaf) node);
+ //
+ //            }
+ //
+ //        }
+ 
      }
  
***************
*** 1267,1276 ****
      protected AbstractNode readNodeOrLeaf(long addr) {
  
! //        /*
! //         * offer the node serializer's buffer to the IRawStore. it will be used
! //         * iff it is large enough and the store does not prefer to return a
! //         * read-only slice.
! //         */
! //        ByteBuffer tmp = store.read(addr, nodeSer._buf);
          ByteBuffer tmp = store.read(addr);
          assert tmp.position() == 0;
--- 1377,1387 ----
      protected AbstractNode readNodeOrLeaf(long addr) {
  
!         // /*
!         // * offer the node serializer's buffer to the IRawStore. it will be
!         // used
!         // * iff it is large enough and the store does not prefer to return a
!         // * read-only slice.
!         // */
!         // ByteBuffer tmp = store.read(addr, nodeSer._buf);
          ByteBuffer tmp = store.read(addr);
          assert tmp.position() == 0;

--- NEW FILE: DataInputBuffer.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Apr 9, 2007
 */

package com.bigdata.objndx;

import java.io.DataInput;
import java.io.DataInputStream;
import java.io.EOFException;
import java.io.IOException;

/**
 * A fast implementation of DataInput designed to read from a byte[].
 * 
 * @see DataOutputBuffer
 * 
 * @see DataInputStream
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public class DataInputBuffer implements DataInput {

    /**
     * The buffer whose contents are being read.
     */
    final protected byte[] buf;
    
    /**
     * The current offset in the buffer.  This is incremented each time
     * any data is read from the buffer.
     */
    protected int off;
    
    /**
     * The index of the last byte in the buffer having valid data.
     */
    final protected int len;
    
    public DataInputBuffer(byte[] buf) {
        
        if(buf==null) throw new IllegalArgumentException();
        
        this.buf = buf;
        
        this.off = 0;
        
        this.len = buf.length;
        
    }
    
    /*
     * DataInput
     */
    
    public boolean readBoolean() throws IOException {

        if(off>=len) throw new EOFException();
        
        return buf[off++] == 1 ? true : false;
        
    }

    public byte readByte() throws IOException {

        if (off >= len)
            throw new EOFException();
        
        return buf[off++];
        
    }

    public char readChar() throws IOException {

        if (off + 2 > len)
            throw new EOFException();

        int ch1 = buf[off++];

        int ch2 = buf[off++];
        
        return (char) ((ch1 << 8) + (ch2 << 0));
        
    }

    public double readDouble() throws IOException {

        return Double.longBitsToDouble(readLong());

    }

    public float readFloat() throws IOException {

        return Float.intBitsToFloat(readInt());
        
    }

    final public void readFully(byte[] b) throws IOException {

        readFully(b, 0, b.length);
        
    }

    final public void readFully(byte[] b, final  int off, final int len) throws IOException {

        if (this.off + len > this.len)
            throw new EOFException();
        
        System.arraycopy(buf, this.off, b, off, len);
        
        this.off += len;
        
    }

    public int readInt() throws IOException {

        if(off+4>len) throw new EOFException();
        
        int v = 0;
        
        // big-endian.
        v += (0xffL & buf[off++]) << 24;
        v += (0xffL & buf[off++]) << 16;
        v += (0xffL & buf[off++]) <<  8;
        v += (0xffL & buf[off++]) <<  0;
        
        return v;

    }

    public String readLine() throws IOException {
        
        throw new UnsupportedOperationException();
        
    }

    public long readLong() throws IOException {

        if(off+8>len) throw new EOFException();
        
        long v = 0L;
        
        // big-endian.
        v += (0xffL & buf[off++]) << 56;
        v += (0xffL & buf[off++]) << 48;
        v += (0xffL & buf[off++]) << 40;
        v += (0xffL & buf[off++]) << 32;
        v += (0xffL & buf[off++]) << 24;
        v += (0xffL & buf[off++]) << 16;
        v += (0xffL & buf[off++]) <<  8;
        v += (0xffL & buf[off++]) <<  0;
        
        return v;
 
    }

    public short readShort() throws IOException {
        
        if (off + 2 > len)
            throw new EOFException();

        int ch1 = buf[off++];

        int ch2 = buf[off++];
        
        return (short) ((ch1 << 8) + (ch2 << 0));
        
    }

    public String readUTF() throws IOException {

        return DataInputStream.readUTF(this);
        
    }

    public int readUnsignedByte() throws IOException {
        
        if (off >= len)
            throw new EOFException();
        
        return buf[off++];
        
    }

    public int readUnsignedShort() throws IOException {

        if (off + 2 > len)
            throw new EOFException();

        int ch1 = buf[off++];

        int ch2 = buf[off++];
        
        return ((ch1 << 8) + (ch2 << 0));
        
    }

    public int skipBytes(int n) throws IOException {

        off += n;
        
        if(off>len) throw new IOException();
        
        return n;
        
    }

    /*
     * unpack unsigned long integer.
     */

    /**
     * Unpack a long value from the current buffer position.
     * 
     * @return The long value.
     * 
     * @throws IOException
     */
    final public long unpackLong() throws IOException {
        int b = buf[off++];
        int nbytes;
        long l;
        if ((b & 0x80) != 0) {
            // high bit is set.
            nbytes = 8; // use 8 bytes (this one plus the next 7).
            l = b & 0x7f; // clear the high bit - the rest of the byte is the
                            // start value.
        } else {
            // high bit is clear.
            nbytes = b >> 4; // nbytes is the upper nibble. (right shift one
                                // nibble).
            l = b & 0x0f; // starting value is lower nibble (clear the upper
                            // nibble).
        }
        for (int i = 1; i < nbytes; i++) {
            // Read the next byte.
            b = buf[off++];
            // Shift the existing value one byte left and add into the low
            // (unsigned) byte.
            l = (l << 8) + (0xff & b);
        }
        return l;
    }

    /*
     * unpack unsigned short integer.
     */
    
    /**
     * Unpack a non-negative short value from the input stream.
     * 
     * @param is
     *            The input stream.
     * 
     * @return The short value.
     * 
     * @throws IOException
     */
    final public short unpackShort() throws IOException {
        short b = (short) buf[off++];
        short v;
        if ((b & 0x80) != 0) { // high bit is set.
            /*
             * clear the high bit and shift over one byte.
             */
            v = (short) ((b & 0x7f) << 8);
            b = buf[off++]; // read the next byte.
            v |= (b & 0xff); // and combine it together with the high byte.
        } else {
            // high bit is clear.
            v = b; // interpret the byte as a short value.
        }
        return (short) v;
    }

}

Index: NodeSerializer.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/objndx/NodeSerializer.java,v
retrieving revision 1.34
retrieving revision 1.35
diff -C2 -d -r1.34 -r1.35
*** NodeSerializer.java	15 Mar 2007 16:11:08 -0000	1.34
--- NodeSerializer.java	10 Apr 2007 18:33:31 -0000	1.35
***************
*** 47,51 ****
--- 47,53 ----
  package com.bigdata.objndx;
  
+ import java.io.ByteArrayOutputStream;
  import java.io.DataInputStream;
+ import java.io.DataOutput;
  import java.io.DataOutputStream;
  import java.io.EOFException;
***************
*** 97,100 ****
--- 99,103 ----
   * interfaces is encouraged.
   * </p>
+  * 
   * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
   * @version $Id$

[cweb-CVS] bigdata/src/architecture performance.xls, 1.1, 1.2 segment math.xls, 1.15, 1.16

From: Bryan T. <tho...@us...> - 2007-04-10 18:34:20

Update of /cvsroot/cweb/bigdata/src/architecture
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv24584/src/architecture

Modified Files:
	performance.xls segment math.xls 
Log Message:
Fixed a bug in UUID assignment to index segments.

Added some logic for fast data output for btree nodes and leaves.

Index: performance.xls
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/architecture/performance.xls,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
Binary files /tmp/cvsp1r2Ac and /tmp/cvsMeOaPb differ

Index: segment math.xls
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/architecture/segment math.xls,v
retrieving revision 1.15
retrieving revision 1.16
diff -C2 -d -r1.15 -r1.16
Binary files /tmp/cvsH6VFBm and /tmp/cvs5F0R2l differ

[cweb-CVS] bigdata/src/java/com/bigdata/service IDataService.java, 1.4, 1.5

From: Bryan T. <tho...@us...> - 2007-04-10 18:33:39

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/service
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv24584/src/java/com/bigdata/service

Modified Files:
	IDataService.java 
Log Message:
Fixed a bug in UUID assignment to index segments.

Added some logic for fast data output for btree nodes and leaves.

Index: IDataService.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/IDataService.java,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** IDataService.java	27 Mar 2007 14:34:24 -0000	1.4
--- IDataService.java	10 Apr 2007 18:33:31 -0000	1.5
***************
*** 93,102 ****
--- 93,105 ----
  
      /**
+      * <p>
       * Used by the client to submit a batch operation on a named B+Tree
       * (synchronous).
+      * </p>
       * <p>
       * Unisolated operations SHOULD be used to achieve "auto-commit" semantics.
       * Fully isolated transactions are useful IFF multiple operations must be
       * composed into a ACID unit.
+      * </p>
       * <p>
       * While unisolated batch operations on a single data service are ACID,
***************
*** 111,114 ****
--- 114,118 ----
       * throughput SHOULD choose unisolated read operations in preference to a
       * read-committed transaction.
+      * </p>
       * <p>
       * This method is thread-safe. It will block for each operation. It should
***************
*** 118,121 ****
--- 122,126 ----
       * operations MUST be buffered by a thread pool with a FIFO policy so that
       * client requests may be decoupled from data service operations.
+      * </p>
       * 
       * @param tx
***************
*** 134,148 ****
       *                {@link ExecutionException#getCause()} for the underlying
       *                error.
       */
      public void batchOp(long tx, String name, IBatchOp op)
              throws InterruptedException, ExecutionException, IOException;
! 
      /**
-      * Submit a procedure.
       * <p>
       * <p>
       * Unisolated operations SHOULD be used to achieve "auto-commit" semantics.
       * Fully isolated transactions are useful IFF multiple operations must be
       * composed into a ACID unit.
       * <p>
       * While unisolated batch operations on a single data service are ACID,
--- 139,202 ----
       *                {@link ExecutionException#getCause()} for the underlying
       *                error.
+      * 
+      * @todo it is possible to have concurrent execution of batch operations for
+      *       distinct indices. In order to support this, the write thread would
+      *       have to become a pool of N worker threads fed from a queue of
+      *       operations. Concurrent writers can execute as long as they are
+      *       writing on different indices. (Concurrent readers can execute as
+      *       long as they are reading from a historical commit time.)
       */
      public void batchOp(long tx, String name, IBatchOp op)
              throws InterruptedException, ExecutionException, IOException;
!     
      /**
       * <p>
+      * Streaming traversal of keys and/or values in a given key range.
+      * </p>
+      * <p>
+      * Note: The rangeQuery operation is NOT allowed for read-committed
+      * transactions (the underlying constraint is that the {@link BTree} does
+      * NOT support traversal under concurrent modification so this operation is
+      * limited to read-only or fully isolated transactions or to unisolated
+      * reads against a historical commit time).
+      * </p>
+      * 
+      * @param tx
+      * @param name
+      * @param fromKey
+      * @param toKey
+      * @param flags
+      *            (@todo define flags: count yes/no, keys yes/no, values yes/no)
+      * 
+      * @exception InterruptedException
+      *                if the operation was interrupted (typically by
+      *                {@link #shutdownNow()}.
+      * @exception ExecutionException
+      *                If the operation caused an error. See
+      *                {@link ExecutionException#getCause()} for the underlying
+      *                error.
+      * @exception UnsupportedOperationException
+      *                If the tx is zero (0L) (indicating an unisolated
+      *                operation) -or- if the identifed transaction is
+      *                {@link IsolationEnum#ReadCommitted}.
+      * 
+      * FIXME support filters. there are a variety of use cases from clients that
+      * are aware of version counters and delete markers to clients that encode a
+      * column name and datum or write time into the key to those that will
+      * filter based on inspection of the value associated with the key, e.g.,
+      * only values having some attribute.
+      */
+     public RangeQueryResult rangeQuery(long tx, String name, byte[] fromKey,
+             byte[] toKey, int flags) throws InterruptedException,
+             ExecutionException, IOException, IOException;
+     /**
+      * <p>
+      * Submit a procedure.
+      * </p>
       * <p>
       * Unisolated operations SHOULD be used to achieve "auto-commit" semantics.
       * Fully isolated transactions are useful IFF multiple operations must be
       * composed into a ACID unit.
+      * </p>
       * <p>
       * While unisolated batch operations on a single data service are ACID,
***************
*** 157,160 ****
--- 211,215 ----
       * throughput SHOULD choose unisolated read operations in preference to a
       * read-committed transaction.
+      * </p>
       * 
       * @param tx
***************
*** 163,167 ****
       * @param proc
       *            The procedure to be executed.
!      *            
       * @throws InterruptedException
       * @throws ExecutionException
--- 218,222 ----
       * @param proc
       *            The procedure to be executed.
!      * 
       * @throws InterruptedException
       * @throws ExecutionException
***************
*** 169,202 ****
      public void submit(long tx, IProcedure proc) throws InterruptedException,
              ExecutionException, IOException;
-     
-     /**
-      * Streaming traversal of keys and/or values in a given key range.
-      * <p>
-      * Note: The rangeQuery operation is NOT allowed for either unisolated reads
-      * or read-committed transactions (the underlying constraint is that the
-      * {@link BTree} does NOT support traversal under concurrent modification
-      * this operation is limited to read-only or fully isolated transactions).
-      * 
-      * @param tx
-      * @param name
-      * @param fromKey
-      * @param toKey
-      * @param flags (@todo define flags: count yes/no, keys yes/no, values yes/no)
-      * 
-      * @exception InterruptedException
-      *                if the operation was interrupted (typically by
-      *                {@link #shutdownNow()}.
-      * @exception ExecutionException
-      *                If the operation caused an error. See
-      *                {@link ExecutionException#getCause()} for the underlying
-      *                error.
-      * @exception UnsupportedOperationException
-      *                If the tx is zero (0L) (indicating an unisolated
-      *                operation) -or- if the identifed transaction is
-      *                {@link IsolationEnum#ReadCommitted}.
-      */
-     public RangeQueryResult rangeQuery(long tx, String name, byte[] fromKey,
-             byte[] toKey, int flags) throws InterruptedException,
-             ExecutionException, IOException, IOException;
  
  //    /**
--- 224,227 ----

[cweb-CVS] bigdata/src/test/com/bigdata/objndx TestDataOutputBuffer.java, NONE, 1.1 TestLongPacker.java, NONE, 1.1 TestShortPacker.java, NONE, 1.1 TestAll.java, 1.32, 1.33 TestIndexSegmentBuilderWithSmallTree.java, 1.10, 1.11 AbstractBTreeTestCase.java, 1.32, 1.33

From: Bryan T. <tho...@us...> - 2007-04-10 18:33:39

Update of /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv24584/src/test/com/bigdata/objndx

Modified Files:
	TestAll.java TestIndexSegmentBuilderWithSmallTree.java 
	AbstractBTreeTestCase.java 
Added Files:
	TestDataOutputBuffer.java TestLongPacker.java 
	TestShortPacker.java 
Log Message:
Fixed a bug in UUID assignment to index segments.

Added some logic for fast data output for btree nodes and leaves.

Index: TestIndexSegmentBuilderWithSmallTree.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx/TestIndexSegmentBuilderWithSmallTree.java,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** TestIndexSegmentBuilderWithSmallTree.java	8 Mar 2007 18:14:05 -0000	1.10
--- TestIndexSegmentBuilderWithSmallTree.java	10 Apr 2007 18:33:31 -0000	1.11
***************
*** 417,421 ****
      public void test_problem3_buildOrder3() throws IOException {
  
!         BTree btree = getProblem3();
  
          btree.dump(Level.DEBUG,System.err);
--- 417,421 ----
      public void test_problem3_buildOrder3() throws IOException {
  
!         final BTree btree = getProblem3();
  
          btree.dump(Level.DEBUG,System.err);

Index: AbstractBTreeTestCase.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx/AbstractBTreeTestCase.java,v
retrieving revision 1.32
retrieving revision 1.33
diff -C2 -d -r1.32 -r1.33
*** AbstractBTreeTestCase.java	27 Mar 2007 14:34:21 -0000	1.32
--- AbstractBTreeTestCase.java	10 Apr 2007 18:33:31 -0000	1.33
***************
*** 1555,1562 ****
          assert actual != null;
          
!         assertEquals("indexUUID",expected.getIndexUUID(), actual.getIndexUUID());
          
          // The #of entries must agree.
!         assertEquals("entryCount",expected.getEntryCount(), actual.getEntryCount());
          
          // verify the entry iterator.
--- 1555,1565 ----
          assert actual != null;
          
!         // Must be the same "index".
!         assertEquals("indexUUID", expected.getIndexUUID(), actual
!                 .getIndexUUID());
          
          // The #of entries must agree.
!         assertEquals("entryCount", expected.getEntryCount(), actual
!                 .getEntryCount());
          
          // verify the entry iterator.

--- NEW FILE: TestLongPacker.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Oct 28, 2005
 */
package com.bigdata.objndx;

import java.io.IOException;
import java.util.Random;

import junit.framework.TestCase;

/**
 * Test suite for packing and unpacking unsigned long integers.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public class TestLongPacker extends TestCase {

    /**
     * 
     */
    public TestLongPacker() {
        super();
    }

    /**
     * @param name
     */
    public TestLongPacker(String name) {
        super(name);
    }

    /**
     * Unpacks a long.
     * 
     * @param expected The expected long value.
     * 
     * @param packed The packed byte[].
     * 
     * @throws IOException
     *             If there was not enough data.
     * 
     * @throws junit.framework.AssertionFailedError
     *             If there is too much data.
     */

    public void doUnpackTest( long expected, byte[] packed )
    	throws IOException
    {
    
        DataInputBuffer dib = new DataInputBuffer(packed);
        
        long actual = dib.unpackLong();
        
        assertEquals( "value", expected, actual );
        
//        assertTrue( "Expecting EOF", dib.read() == -1 );
        try {
            dib.readByte();
            fail("Expecting: "+IOException.class);
        }
        catch(IOException ex) {
            System.err.println("Ignoring expected exception: "+ex);
        }

    }

    /**
     * Given the first byte of a packed long value, return the #of bytes into which
     * that value was packed (including this one).
     * 
     * @param firstByte The first byte.
     * 
     * @return The #of bytes.  This is in the range [1:8] inclusive.
     */
    
    public int getNBytes( int firstByte ) {
        int nbytes;
        if( ( firstByte & 0x80 ) != 0 ) { // high bit is set.
            nbytes = 8;
        } else {
            nbytes = firstByte >> 4; // clear the high bit and right shift one nibble.
        }
        return nbytes;  
    }
    
    public void testNBytes()
    {
       
        // high bit is set - always 8 bytes.
        assertEquals( "nbytes", 8, getNBytes( 0x80 ) );
        assertEquals( "nbytes", 8, getNBytes( 0x81 ) );
        assertEquals( "nbytes", 8, getNBytes( 0x8e ) );
        assertEquals( "nbytes", 8, getNBytes( 0x8f ) );
        
        // high bit is NOT set. nbytes is the upper nibble.
        assertEquals( "nbytes", 1, getNBytes( 0x10 ) );
        assertEquals( "nbytes", 2, getNBytes( 0x20 ) );
        assertEquals( "nbytes", 3, getNBytes( 0x30 ) );
        assertEquals( "nbytes", 4, getNBytes( 0x40 ) );
        assertEquals( "nbytes", 5, getNBytes( 0x50 ) );
        assertEquals( "nbytes", 6, getNBytes( 0x60 ) );
        assertEquals( "nbytes", 7, getNBytes( 0x70 ) );
       
        // high bit is NOT set. nbytes is the upper nibble.
        assertEquals( "nbytes", 1, getNBytes( 0x11 ) );
        assertEquals( "nbytes", 2, getNBytes( 0x21 ) );
        assertEquals( "nbytes", 3, getNBytes( 0x31 ) );
        assertEquals( "nbytes", 4, getNBytes( 0x41 ) );
        assertEquals( "nbytes", 5, getNBytes( 0x51 ) );
        assertEquals( "nbytes", 6, getNBytes( 0x61 ) );
        assertEquals( "nbytes", 7, getNBytes( 0x71 ) );
       
        // high bit is NOT set. nbytes is the upper nibble.
        assertEquals( "nbytes", 1, getNBytes( 0x1f ) );
        assertEquals( "nbytes", 2, getNBytes( 0x2f ) );
        assertEquals( "nbytes", 3, getNBytes( 0x3f ) );
        assertEquals( "nbytes", 4, getNBytes( 0x4f ) );
        assertEquals( "nbytes", 5, getNBytes( 0x5f ) );
        assertEquals( "nbytes", 6, getNBytes( 0x6f ) );
        assertEquals( "nbytes", 7, getNBytes( 0x7f ) );
       
    }
    
    public void testUnpack()
    	throws IOException
    {

        // upper nibble is 1, so nbytes == 1 and the lower nibble is the value.
        doUnpackTest( 0x0, new byte[]{(byte)0x10} );
        doUnpackTest( 0x1, new byte[]{(byte)0x11} );
        doUnpackTest( 0x7, new byte[]{(byte)0x17} );
        doUnpackTest( 0xf, new byte[]{(byte)0x1f} );

        // upper nibble is 2, so nbytes == 2.
        doUnpackTest( 0xf00, new byte[]{(byte)0x2f, (byte)0x00} );
        doUnpackTest( 0xfa7, new byte[]{(byte)0x2f, (byte)0xa7} );
        doUnpackTest( 0xfa0, new byte[]{(byte)0x2f, (byte)0xa0} );
        doUnpackTest( 0xf07, new byte[]{(byte)0x2f, (byte)0x07} );
        
        // upper nibble is 3, so nbytes == 3.
        doUnpackTest( 0xcaa4d, new byte[]{(byte)0x3c, (byte)0xaa, (byte)0x4d} );
        
        // high bit only, lower nibble plus next seven bytes are the value.
        doUnpackTest( 0xaeede00a539271fL,
                	new byte[]{(byte)0x8a, (byte)0xee, (byte)0xde, (byte)0x00,
                	           (byte)0xa5, (byte)0x39, (byte)0x27, (byte)0x1f
        			  }
        	      );
        
    }

    public void doPackTest( long v, byte[] expected )
    	throws IOException
    {
        
        DataOutputBuffer dob = new DataOutputBuffer();
        
        final int nbytes = dob.packLong( v );
        
        final byte[] actual = dob.toByteArray();
        
        assertEquals( "nbytes", expected.length, nbytes );
        assertEquals( "nbytes", getNBytes(expected[0]), nbytes );
        assertEquals( "bytes", expected, actual );
        
    }

    public void test_getNibbleLength()
    {

        // Note: zero (0) is interpreted as being one nibble for our purposes.
        assertEquals( "nibbles", 1, DataOutputBuffer.getNibbleLength( 0x0 ) );
        
        assertEquals( "nibbles", 1, DataOutputBuffer.getNibbleLength( 0x1 ) );
        assertEquals( "nibbles", 1, DataOutputBuffer.getNibbleLength( 0x2 ) );
        assertEquals( "nibbles", 1, DataOutputBuffer.getNibbleLength( 0x7 ) );
        assertEquals( "nibbles", 1, DataOutputBuffer.getNibbleLength( 0x8 ) );
        assertEquals( "nibbles", 1, DataOutputBuffer.getNibbleLength( 0xe ) );
        assertEquals( "nibbles", 1, DataOutputBuffer.getNibbleLength( 0xf ) );
        
        assertEquals( "nibbles", 2, DataOutputBuffer.getNibbleLength( 0x10 ) );
        assertEquals( "nibbles", 2, DataOutputBuffer.getNibbleLength( 0x11 ) );
        assertEquals( "nibbles", 2, DataOutputBuffer.getNibbleLength( 0x12 ) );
        assertEquals( "nibbles", 2, DataOutputBuffer.getNibbleLength( 0x17 ) );
        assertEquals( "nibbles", 2, DataOutputBuffer.getNibbleLength( 0x18 ) );
        assertEquals( "nibbles", 2, DataOutputBuffer.getNibbleLength( 0x1e ) );
        assertEquals( "nibbles", 2, DataOutputBuffer.getNibbleLength( 0x1f ) );
        assertEquals( "nibbles", 2, DataOutputBuffer.getNibbleLength( 0x7f ) );
        assertEquals( "nibbles", 2, DataOutputBuffer.getNibbleLength( 0x8f ) );
        assertEquals( "nibbles", 2, DataOutputBuffer.getNibbleLength( 0xff ) );

        assertEquals( "nibbles", 3, DataOutputBuffer.getNibbleLength( 0x100 ) );
        assertEquals( "nibbles", 3, DataOutputBuffer.getNibbleLength( 0x101 ) );
        assertEquals( "nibbles", 3, DataOutputBuffer.getNibbleLength( 0x121 ) );
        assertEquals( "nibbles", 3, DataOutputBuffer.getNibbleLength( 0x1ee ) );
        assertEquals( "nibbles", 3, DataOutputBuffer.getNibbleLength( 0x1ff ) );
        assertEquals( "nibbles", 3, DataOutputBuffer.getNibbleLength( 0xfff ) );
        
        assertEquals( "nibbles", 4, DataOutputBuffer.getNibbleLength( 0x1ff0 ) );
        assertEquals( "nibbles", 4, DataOutputBuffer.getNibbleLength( 0x7ff0 ) );
        assertEquals( "nibbles", 4, DataOutputBuffer.getNibbleLength( 0xfff0 ) );
        assertEquals( "nibbles", 4, DataOutputBuffer.getNibbleLength( 0xfff1 ) );

        assertEquals( "nibbles", 5, DataOutputBuffer.getNibbleLength( 0x12345 ) );
        assertEquals( "nibbles", 5, DataOutputBuffer.getNibbleLength( 0x54321 ) );

        assertEquals( "nibbles", 6, DataOutputBuffer.getNibbleLength( 0x123456 ) );
        assertEquals( "nibbles", 6, DataOutputBuffer.getNibbleLength( 0x654321 ) );

        assertEquals( "nibbles", 7, DataOutputBuffer.getNibbleLength( 0x1234567 ) );
        assertEquals( "nibbles", 7, DataOutputBuffer.getNibbleLength( 0x7654321 ) );

        /*
         * Note: At 8 nibbles we have 32 bits. When the high bit is one, this
         * MUST be expressed as a long (trailing 'L', NOT cast to a long) or it
         * will be interpreted as a negative integer and sign extended to a
         * negative long.
         */
        assertEquals( "nibbles", 8, DataOutputBuffer.getNibbleLength( 0x12345678L ) );
        assertEquals( "nibbles", 8, DataOutputBuffer.getNibbleLength( 0x87654321L ) ); 

        assertEquals( "nibbles", 9, DataOutputBuffer.getNibbleLength( 0x123456789L ) );
        assertEquals( "nibbles", 9, DataOutputBuffer.getNibbleLength( 0x987654321L ) ); 

        assertEquals( "nibbles", 10, DataOutputBuffer.getNibbleLength( 0x123456789aL ) );
        assertEquals( "nibbles", 10, DataOutputBuffer.getNibbleLength( 0xa987654321L ) ); 

        assertEquals( "nibbles", 11, DataOutputBuffer.getNibbleLength( 0x123456789abL ) );
        assertEquals( "nibbles", 11, DataOutputBuffer.getNibbleLength( 0xba987654321L ) ); 

        assertEquals( "nibbles", 12, DataOutputBuffer.getNibbleLength( 0x123456789abcL ) );
        assertEquals( "nibbles", 12, DataOutputBuffer.getNibbleLength( 0xcba987654321L ) ); 

        assertEquals( "nibbles", 13, DataOutputBuffer.getNibbleLength( 0x123456789abcdL ) );
        assertEquals( "nibbles", 13, DataOutputBuffer.getNibbleLength( 0xdcba987654321L ) ); 

        assertEquals( "nibbles", 14, DataOutputBuffer.getNibbleLength( 0x123456789abcdeL ) );
        assertEquals( "nibbles", 14, DataOutputBuffer.getNibbleLength( 0xedcba987654321L ) ); 

        assertEquals( "nibbles", 15, DataOutputBuffer.getNibbleLength( 0x123456789abcdefL ) );
        assertEquals( "nibbles", 15, DataOutputBuffer.getNibbleLength( 0xfedcba987654321L ) ); 

        assertEquals( "nibbles", 16, DataOutputBuffer.getNibbleLength( 0x1234567890abcdefL ) );
        assertEquals( "nibbles", 16, DataOutputBuffer.getNibbleLength( 0xfedcba0987654321L ) ); 

    }
    
    public void testPack()
    	throws IOException
    {

        // [0:15] should be packed into one byte.
        doPackTest( 0x0, new byte[]{(byte)0x10} );
        doPackTest( 0x1, new byte[]{(byte)0x11} );
        doPackTest( 0x2, new byte[]{(byte)0x12} );
        doPackTest( 0xe, new byte[]{(byte)0x1e} );
        doPackTest( 0xf, new byte[]{(byte)0x1f} );

        /*
         * 0x10 through 0xfff overflow the lower nibble, so the value is packed
         * into two bytes. the first byte has the header and the next three
         * nibbles are the value. This case is good for up to 2^12, since there
         * are three full nibbles to encode the value.
         */

        doPackTest( 0x10,  new byte[]{(byte)0x20, (byte)0x10 });
        doPackTest( 0x11,  new byte[]{(byte)0x20, (byte)0x11 });
        doPackTest( 0x1f,  new byte[]{(byte)0x20, (byte)0x1f });
        doPackTest( 0x20,  new byte[]{(byte)0x20, (byte)0x20 });
        doPackTest( 0xff,  new byte[]{(byte)0x20, (byte)0xff });
        doPackTest( 0x100, new byte[]{(byte)0x21, (byte)0x00 });
        doPackTest( 0x101, new byte[]{(byte)0x21, (byte)0x01 });
        doPackTest( 0x121, new byte[]{(byte)0x21, (byte)0x21 });
        doPackTest( 0x1ee, new byte[]{(byte)0x21, (byte)0xee });
        doPackTest( 0x1ff, new byte[]{(byte)0x21, (byte)0xff });
        doPackTest( 0xfff, new byte[]{(byte)0x2f, (byte)0xff });

        /*
         * 0x1000 through 0xfffff fit into one more byte.
         */

        doPackTest( 0x1000,  new byte[]{(byte)0x30, (byte)0x10, (byte)0x00 });
        doPackTest( 0x1234,  new byte[]{(byte)0x30, (byte)0x12, (byte)0x34 });
        doPackTest( 0x1fff,  new byte[]{(byte)0x30, (byte)0x1f, (byte)0xff });
        doPackTest( 0x54321, new byte[]{(byte)0x35, (byte)0x43, (byte)0x21 });
        doPackTest( 0xfffff, new byte[]{(byte)0x3f, (byte)0xff, (byte)0xff });
        
    }
        
    public static final long SIGN_MASK = 1L<<63;
    
    public void testHighBit() {
        assertTrue( "sign bit", ( -1L & SIGN_MASK ) != 0 );
        assertFalse( "sign bit", ( 0L & SIGN_MASK ) != 0 );
    }
    
    private interface LongGenerator
    {
        public long nextLong();
    }
    
    /**
     * All long values in sequence starting from the given start value
     * and using the given increment.
     * @author thompsonbry
     */
    private static class Sequence implements LongGenerator
    {
        
        long _start, _inc, _next;
        
        public Sequence( long start, long inc ) {
            _start = start;
            _inc   = inc;
            _next  = start;
        }
        
        public long nextLong() {
            long v = _next;
            _next += _inc;
            return v;
//            double d = rnd.nextGaussian();
////          if( d < 0 ) d = -d;
//          final long expected = (long) ( d * Long.MAX_VALUE );
        }
        
    }
 
    /**
     * Random long values (32 bits of random long), including negatives,
     * with a uniform distribution.
     * 
     * @author thompsonbry
     */
    private static class RandomLong implements LongGenerator
    {
        
        Random _rnd;
        
        public RandomLong( Random rnd ) {
            _rnd = rnd;
        }

        public long nextLong() {
            return _rnd.nextLong();
        }
        
    }

    /**
     * Run a large #of pack/unpack operations on a sequence of long values to
     * demonstrate correctness in that sequence.  The sequence is the long
     * values from -1 to 1M by one (dense coverage).
     * 
     * @throws IOException
     */

    public void testStressSequence() throws IOException {

        // dense coverage of the first 1M values.
        doStressTest( 1000000, new Sequence( -1, 1 ) );
        
    }
    
    /**
     * Run a large #of random pack/unpack operations to sample the space while
     * showing correctness on those samples.  The amount of compression due to
     * packing for this test is <em>very</em> small since all bits are equally
     * likely to be non-zero, so the #of bytes required on average to pack a
     * long value is 8. 
     * 
     * @throws IOException
     */
    
    public void testStressRandom() throws IOException {

        // test on 1M random long values.
        doStressTest( 1000000, new RandomLong( new Random() ) );
        
    }

    /**
     * Run a stress test.  Writes some information of possible interest onto
     * System.err.
     * 
     * @param ntrials #of trials.
     * 
     * @param g Generator for the long values.
     * 
     * @throws IOException
     */
    public void doStressTest( int ntrials, LongGenerator g ) throws IOException {
        
        long nwritten = 0L;

        long packlen = 0L;
        
        long minv = Long.MAX_VALUE, maxv = Long.MIN_VALUE;
        
        for( int i=0; i<ntrials; i++ ) {
            
            long expected = g.nextLong();
            
            if( expected < 0L ) {
         
                DataOutputBuffer dos = new DataOutputBuffer();
                
                try {
                    
                    dos.packLong( expected );
                    
                    fail( "Expecting rejection of negative value: val="+expected );
                    
                }
                
                catch( IllegalArgumentException ex ) {
                    
//                    System.err.println( "Ingoring expected exception: "+ex );
                    
                }
    
                
            } else {

                if( expected > maxv ) maxv = expected;
                if( expected < minv ) minv = expected;
                    
                DataOutputBuffer dos = new DataOutputBuffer();
                
                final int actualByteLength1 = dos.packLong( expected );

                byte[] packed = dos.toByteArray();
                
                final int actualByteLength2 = getNBytes( packed[ 0 ] );
            
                DataInputBuffer dis = new DataInputBuffer( packed );
                
                final long actual = dis.unpackLong();

                assertEquals( "trial="+i, expected, actual );
                
                assertEquals( "trial="+i+", v="+expected+", nbytes", actualByteLength1, actualByteLength2 );

                packlen += packed.length; // total #of packed bytes.
                nwritten++; // count #of non-negative random values.
                
            }
            
        }

        System.err.println( "\nWrote "+nwritten+" non-negative long values." );
        System.err.println( "minv="+minv+", maxv="+maxv );
        System.err.println( "#packed bytes       ="+packlen );
        System.err.println( "#bytes if not packed="+(nwritten * 8));
        long nsaved = ( nwritten * 8 ) - packlen;
        System.err.println ("#bytes saved        ="+nsaved);
        System.err.println( "%saved by packing   ="+nsaved/(nwritten*8f)*100+"%");
        
    }
    
    public static void assertEquals( String msg, byte[] expected, byte[] actual )
    {
        assertEquals( msg+": length", expected.length, actual.length );
        for( int i=0; i<expected.length; i++ ) {
            assertEquals( msg+": byte[i="+i+"]", expected[i], actual[i] );
        }
    }
    
}

--- NEW FILE: TestDataOutputBuffer.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Apr 9, 2007
 */

package com.bigdata.objndx;

import java.io.IOException;
import java.util.Random;

import junit.framework.TestCase;
import junit.framework.TestCase2;

/**
 * Test suite for {@link DataOutputBuffer}.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public class TestDataOutputBuffer extends TestCase2
{

    /**
     * 
     */
    public TestDataOutputBuffer() {
    }

    /**
     * @param arg0
     */
    public TestDataOutputBuffer(String arg0) {
        super(arg0);
    }

    /**
     * ctor tests, including correct rejection.
     */
    public void test_ctor() {

        {
            DataOutputBuffer buf = new DataOutputBuffer();

            assertNotNull(buf.buf);
            assertEquals(DataOutputBuffer.DEFAULT_INITIAL_CAPACITY,
                    buf.buf.length);
            assertEquals(0, buf.len);

        }

        {
            DataOutputBuffer buf = new DataOutputBuffer(0);
            assertNotNull(buf.buf);
            assertEquals(0, buf.buf.length);
            assertEquals(0, buf.len);
        }

        {
            DataOutputBuffer buf = new DataOutputBuffer(20);
            assertNotNull(buf.buf);
            assertEquals(20, buf.buf.length);
            assertEquals(0, buf.len);
        }

        {
            final byte[] expected = new byte[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };
            DataOutputBuffer buf = new DataOutputBuffer(4, expected);
            assertNotNull(buf.buf);
            assertEquals(4, buf.len);
            assertEquals(10, buf.buf.length);
            assertTrue(expected == buf.buf);
        }

    }

    /*
     * correct rejection tests.
     */
    {
        try {
            new DataOutputBuffer(-1);
            fail("Expecting: " + IllegalArgumentException.class);
        } catch (IllegalArgumentException ex) {
            System.err.println("Ignoring expected exception: " + ex);
        }
    }

    {
        try {
            new DataOutputBuffer(20, new byte[10]);
            fail("Expecting: " + IllegalArgumentException.class);
        } catch (IllegalArgumentException ex) {
            System.err.println("Ignoring expected exception: " + ex);
        }
    }

    public void test_DataOutputBuffer_ensureCapacity() {

        DataOutputBuffer DataOutputBuffer = new DataOutputBuffer(0);

        assertEquals(0, DataOutputBuffer.len);
        assertNotNull(DataOutputBuffer.buf);
        assertEquals(0, DataOutputBuffer.buf.length);

        final byte[] originalBuffer = DataOutputBuffer.buf;

        // correct rejection.
        try {
            DataOutputBuffer.ensureCapacity(-1);
            fail("Expecting: " + IllegalArgumentException.class);
        } catch (IllegalArgumentException ex) {
            System.err.println("Ignoring expected exception: " + ex);
        }
        assertTrue(originalBuffer == DataOutputBuffer.buf); // same buffer.

        // no change.
        DataOutputBuffer.ensureCapacity(0);
        assertEquals(0, DataOutputBuffer.len);
        assertNotNull(DataOutputBuffer.buf);
        assertEquals(0, DataOutputBuffer.buf.length);
        assertTrue(originalBuffer == DataOutputBuffer.buf); // same buffer.
    }

    public void test_DataOutputBuffer_ensureCapacity02() {

        DataOutputBuffer DataOutputBuffer = new DataOutputBuffer(0);

        assertEquals(0, DataOutputBuffer.len);
        assertNotNull(DataOutputBuffer.buf);
        assertEquals(0, DataOutputBuffer.buf.length);

        final byte[] originalBuffer = DataOutputBuffer.buf;

        // extends buffer.
        DataOutputBuffer.ensureCapacity(100);
        assertEquals(0, DataOutputBuffer.len);
        assertNotNull(DataOutputBuffer.buf);
        assertEquals(100, DataOutputBuffer.buf.length);
        assertTrue(originalBuffer != DataOutputBuffer.buf); // same buffer.
    }

    /**
     * verify that existing data is preserved if the capacity is extended.
     */
    public void test_DataOutputBuffer_ensureCapacity03() {

        Random r = new Random();
        byte[] expected = new byte[20];
        r.nextBytes(expected);

        DataOutputBuffer DataOutputBuffer = new DataOutputBuffer(20, expected);

        assertEquals(20, DataOutputBuffer.len);
        assertNotNull(DataOutputBuffer.buf);
        assertTrue(expected == DataOutputBuffer.buf);

        DataOutputBuffer.ensureCapacity(30);
        assertEquals(20, DataOutputBuffer.len);
        assertTrue(DataOutputBuffer.buf.length >= 30);

        assertEquals(0, BytesUtil.compareBytesWithLenAndOffset(0,
                expected.length, expected, 0, expected.length,
                DataOutputBuffer.buf));

        for (int i = 21; i < 30; i++) {
            assertEquals(0, DataOutputBuffer.buf[i]);
        }

    }

    public void test_DataOutputBuffer_ensureFree() {

        DataOutputBuffer DataOutputBuffer = new DataOutputBuffer(0);

        assertEquals(0, DataOutputBuffer.len);
        assertNotNull(DataOutputBuffer.buf);
        assertEquals(0, DataOutputBuffer.buf.length);

        DataOutputBuffer.ensureFree(2);

        assertEquals(0, DataOutputBuffer.len);
        assertNotNull(DataOutputBuffer.buf);
        assertTrue(DataOutputBuffer.buf.length >= 2);

    }

    /**
     * Tests ability to append to the buffer, including with overflow of the
     * buffer capacity.
     */
    public void test_DataOutputBuffer_append_bytes() throws IOException {

        // setup buffer with some data and two(2) free bytes.
        DataOutputBuffer DataOutputBuffer = new DataOutputBuffer(5, new byte[] {
                1, 2, 3, 4, 5, 0, 0 });

        /*
         * fill to capacity by copying two bytes from the middle of another
         * array. since this does not overflow we know the exact capacity of the
         * internal buffer (it is not reallocated).
         */
        byte[] tmp = new byte[] { 4, 5, 6, 7, 8, 9 };
        DataOutputBuffer.write(tmp, 2, 2);
        assertEquals(7, DataOutputBuffer.len);
        assertEquals(new byte[] { 1, 2, 3, 4, 5, 6, 7 }, DataOutputBuffer.buf);
        assertEquals(0, BytesUtil.compareBytes(
                new byte[] { 1, 2, 3, 4, 5, 6, 7 }, DataOutputBuffer.buf));

        // overflow capacity (new capacity is not known in advance).
        tmp = new byte[] { 8, 9, 10 };
        byte[] expected = new byte[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };
        DataOutputBuffer.write(tmp);
        assertEquals(10, DataOutputBuffer.len);
        assertEquals(0, BytesUtil.compareBytesWithLenAndOffset(0,
                expected.length, expected, 0, DataOutputBuffer.len,
                DataOutputBuffer.buf));

        // possible overflow (old and new capacity are unknown).
        tmp = new byte[] { 11, 12 };
        expected = new byte[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 };
        DataOutputBuffer.write(tmp);
        assertEquals(12, DataOutputBuffer.len);
        assertEquals(0, BytesUtil.compareBytesWithLenAndOffset(0,
                expected.length, expected, 0, DataOutputBuffer.len,
                DataOutputBuffer.buf));

    }

    /**
     * Test ability to extract and return a key.
     */
    public void test_DataOutputBuffer_getKey() {

        DataOutputBuffer DataOutputBuffer = new DataOutputBuffer(5, new byte[] {
                1, 2, 3, 4, 5, 6, 7, 8, 9, 10 });

        byte[] key = DataOutputBuffer.toByteArray();

        assertEquals(5, key.length);
        assertEquals(new byte[] { 1, 2, 3, 4, 5 }, key);

    }

    /**
     * Verify returns zero length byte[] when the key has zero bytes.
     */
    public void test_DataOutputBuffer_getKey_len0() {

        DataOutputBuffer DataOutputBuffer = new DataOutputBuffer();

        byte[] key = DataOutputBuffer.toByteArray();

        assertEquals(0, key.length);

    }

    /**
     * Test ability to reset the key buffer (simply zeros the #of valid bytes in
     * the buffer without touching the buffer itself).
     */
    public void test_DataOutputBuffer_reset() {

        byte[] expected = new byte[10];

        DataOutputBuffer DataOutputBuffer = new DataOutputBuffer(5, expected);

        assertEquals(5, DataOutputBuffer.len);
        assertTrue(expected == DataOutputBuffer.buf);

        assertTrue(DataOutputBuffer == DataOutputBuffer.reset());

        assertEquals(0, DataOutputBuffer.len);
        assertTrue(expected == DataOutputBuffer.buf);

    }

    public void test_roundTrip() {
        fail("write tests");
    }

    public void test_fencePosts() {
        fail("write tests");
    }

}

--- NEW FILE: TestShortPacker.java ---
/**

The Notice below must appear in each file of the Source Code of any
copy you distribute of the Licensed Product.  Contributors to any
Modifications may add their own copyright notices to identify their
own contributions.

License:

The contents of this file are subject to the CognitiveWeb Open Source
License Version 1.1 (the License).  You may not copy or use this file,
in either source code or executable form, except in compliance with
the License.  You may obtain a copy of the License from

  http://www.CognitiveWeb.org/legal/license/

Software distributed under the License is distributed on an AS IS
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
the License for the specific language governing rights and limitations
under the License.

Copyrights:

Portions created by or assigned to CognitiveWeb are Copyright
(c) 2003-2003 CognitiveWeb.  All Rights Reserved.  Contact
information for CognitiveWeb is available at

  http://www.CognitiveWeb.org

Portions Copyright (c) 2002-2003 Bryan Thompson.

Acknowledgements:

Special thanks to the developers of the Jabber Open Source License 1.0
(JOSL), from which this License was derived.  This License contains
terms that differ from JOSL.

Special thanks to the CognitiveWeb Open Source Contributors for their
suggestions and support of the Cognitive Web.

Modifications:

*/
/*
 * Created on Oct 28, 2005
 */
package com.bigdata.objndx;

import java.io.IOException;
import java.util.Random;

import junit.framework.TestCase;

/**
 * Test suite for packing and unpacking unsigned short integers.
 * 
 * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
 * @version $Id$
 */
public class TestShortPacker extends TestCase {

    /**
     * 
     */
    public TestShortPacker() {
        super();
    }

    /**
     * @param name
     */
    public TestShortPacker(String name) {
        super(name);
    }

    /**
     * Unpacks a short value.
     * 
     * @param expected The expected long value.
     * 
     * @param packed The packed byte[].
     * 
     * @throws IOException
     *             If there was not enough data.
     * 
     * @throws junit.framework.AssertionFailedError
     *             If there is too much data.
     */
    public void doUnpackTest( short expected, byte[] packed )
    	throws IOException
    {

        DataInputBuffer dib = new DataInputBuffer(packed);
        
        short actual = dib.unpackShort();
        
        assertEquals( "value", expected, actual );
        
//        assertTrue( "Expecting EOF", dib.read() == -1 );
        try {
            dib.readByte();
            fail("Expecting: "+IOException.class);
        }
        catch(IOException ex) {
            System.err.println("Ignoring expected exception: "+ex);
        }
    
    }

    /*
     * A short in [0:127] is packed into one byte. Larger values are packed into
     * two bytes. The high bit of the first byte is set if the value was packed
     * into two bytes. If the bit is set, clear the high bit, read the next
     * byte, and interpret the two bytes as a short value. Otherwise interpret
     * the byte as a short value.
     */
    public void testUnpack()
    	throws IOException
    {

        doUnpackTest( (short) 0x0, new byte[]{(byte)0x0} );
        doUnpackTest( (short) 0x1, new byte[]{(byte)0x1} );
        doUnpackTest( (short) 0x7, new byte[]{(byte)0x7} );
        doUnpackTest( (short) 0xf, new byte[]{(byte)0xf} );

        doUnpackTest( (short) 0x70, new byte[]{(byte)0x70} );
        doUnpackTest( (short) 0x7f, new byte[]{(byte)0x7f} );

        doUnpackTest( (short) 0x80, new byte[]{(byte)0x80, (byte)0x80} );
        doUnpackTest( (short) 0x8e, new byte[]{(byte)0x80, (byte)0x8e} );
        
    }

    public void doPackTest( short v, byte[] expected )
    	throws IOException
    {

        DataOutputBuffer dob = new DataOutputBuffer();
        
        final int nbytes = dob.packShort( v );
        
        final byte[] actual = dob.toByteArray();
        
        assertEquals( "nbytes", expected.length, nbytes );
        assertEquals( "bytes", expected, actual );
        
    }
    
    public void testPack()
    	throws IOException
    {

        // [0:127] should be packed into one byte.
        doPackTest( (short) 0x00,  new byte[]{(byte)0x0} );
        doPackTest( (short) 0x01,  new byte[]{(byte)0x1} );
        doPackTest( (short) 0x02,  new byte[]{(byte)0x2} );
        doPackTest( (short) 0x0e,  new byte[]{(byte)0xe} );
        doPackTest( (short) 0x0f,  new byte[]{(byte)0xf} );
        doPackTest( (short) 0x10,  new byte[]{(byte)0x10 });
        doPackTest( (short) 0x11,  new byte[]{(byte)0x11 });
        doPackTest( (short) 0x1f,  new byte[]{(byte)0x1f });
        doPackTest( (short) 0x20,  new byte[]{(byte)0x20 });
        doPackTest( (short) 0x70,  new byte[]{(byte)0x70 });
        doPackTest( (short) 0x7f,  new byte[]{(byte)0x7f });

        // high nibble of the short value is zero.
        doPackTest( (short) 0x80,  new byte[]{(byte)0x80, (byte)0x80 });
        doPackTest( (short) 0xff,  new byte[]{(byte)0x80, (byte)0xff });
        doPackTest( (short) 0x100, new byte[]{(byte)0x81, (byte)0x00 });
        doPackTest( (short) 0x101, new byte[]{(byte)0x81, (byte)0x01 });
        doPackTest( (short) 0x121, new byte[]{(byte)0x81, (byte)0x21 });
        doPackTest( (short) 0x1ee, new byte[]{(byte)0x81, (byte)0xee });
        doPackTest( (short) 0x1ff, new byte[]{(byte)0x81, (byte)0xff });
        doPackTest( (short) 0xfff, new byte[]{(byte)0x8f, (byte)0xff });

    }

    public void test_rejectNegatives() throws IOException
    {

        try {
            doPackTest( (short) 0x8fff, new byte[]{});
            fail( "Expecting: "+IllegalArgumentException.class );
        }
        catch( IllegalArgumentException ex ) {
            System.err.println( "Ignoring expected exception: "+ex );
        }
        
        try {
            doPackTest( (short) 0xffff, new byte[]{});
            fail( "Expecting: "+IllegalArgumentException.class );
        }
        catch( IllegalArgumentException ex ) {
            System.err.println( "Ignoring expected exception: "+ex );
        }

    }
    
    public static final long SIGN_MASK = 11<<15;
    
    public void testHighBit() {
        assertTrue( "sign bit",  ( (short)-1 & SIGN_MASK ) != 0 );
        assertFalse( "sign bit", ( (short) 0 & SIGN_MASK ) != 0 );
    }
    
    private interface ShortGenerator
    {
        public short nextShort();
    }
    
    /**
     * All long values in sequence starting from the given start value
     * and using the given increment.
     * @author thompsonbry
     */
    private static class Sequence implements ShortGenerator
    {
        
        short _start, _inc, _next;
        
        public Sequence( short start, short inc ) {
            _start = start;
            _inc   = inc;
            _next  = start;
        }
        
        public short nextShort() {
            if( _next == Short.MAX_VALUE ) {
                throw new RuntimeException( "No more short values.");
            }
            short v = _next;
            _next += _inc;
            return v;
        }
        
    }
 
    /**
     * Random short values (16 bits of random short), including negatives,
     * with a uniform distribution.
     * 
     * @author thompsonbry
     */
    private static class RandomShort implements ShortGenerator
    {
        
        Random _rnd;
        
        public RandomShort( Random rnd ) {
            _rnd = rnd;
        }

        public short nextShort() {
            return (short)_rnd.nextInt(); // truncate.
        }
        
    }

    /**
     * Run a large #of pack/unpack operations on a sequence of short values to
     * demonstrate correctness in that sequence. The sequence is the short values
     * from -1 to {@link Short#MAX_VALUE} by one (dense coverage).
     * 
     * @throws IOException
     */

    public void testStressSequence() throws IOException {

        // dense coverage of the first 1M values.
        doStressTest( Short.MAX_VALUE+1, new Sequence( (short)-1, (short)1 ) );
        
    }
    
    /**
     * Run a large #of random pack/unpack operations to sample the space while
     * showing correctness on those samples.
     * 
     * @throws IOException
     */
    
    public void testStressRandom() throws IOException {

        // test on random long values.
        doStressTest( 0xffff, new RandomShort( new Random() ) );
        
    }

    /**
     * Returns the #of bytes into which a short value was packed based on the
     * first byte.
     * 
     * @param firstByte
     *            The first byte.
     * 
     * @return The #of bytes (either one (1) or two (2)).
     */
    static int getNBytes(byte firstByte) {

        if ((firstByte & 0x80) != 0) {
            
            return 2;
            
        } else {
            
            return 1;
            
        }
        
    }

    /**
     * Run a stress test.  Writes some information of possible interest onto
     * System.err.
     * 
     * @param ntrials #of trials.
     * 
     * @param g Generator for the long values.
     * 
     * @throws IOException
     */
    
    public void doStressTest( int ntrials, ShortGenerator g ) throws IOException {
        
        long nwritten = 0L;

        long packlen = 0L;
        
        long minv = Short.MAX_VALUE, maxv = Short.MIN_VALUE;
        
        for( int i=0; i<ntrials; i++ ) {
            
            short expected = g.nextShort();
            
            if( expected < 0L ) {
                
                DataOutputBuffer dos = new DataOutputBuffer();
                
                try {
                    
                    dos.packShort( expected );
                    
                    fail( "Expecting rejection of negative value: val="+expected );
                    
                }
                
                catch( IllegalArgumentException ex ) {
                    
//                    System.err.println( "Ingoring expected exception: "+ex );
                    
                }
                    
            } else {

                if( expected > maxv ) maxv = expected;
                if( expected < minv ) minv = expected;

                DataOutputBuffer dos = new DataOutputBuffer();

                int nbytesActual = dos.packShort( expected );
                
                byte[] packed = dos.toByteArray();
                
                final int nbytesExpected = getNBytes( packed[ 0 ] );
            
                DataInputBuffer dis = new DataInputBuffer( packed );
                
                final short actual = dis.unpackShort();

                assertEquals( "trial="+i, expected, actual );
                
                assertEquals( "trial="+i+", v="+expected+", nbytes", nbytesExpected, nbytesActual );

                assertEquals( "trial="+i+", v="+expected+", nbytes", nbytesExpected, packed.length );

                packlen += packed.length; // total #of packed bytes.
                nwritten++; // count #of non-negative random values.
                
            }
            
        }

        System.err.println( "\nWrote "+nwritten+" non-negative long values." );
        System.err.println( "minv="+minv+", maxv="+maxv );
        System.err.println( "#packed bytes       ="+packlen );
        System.err.println( "#bytes if not packed="+(nwritten * 8));
        long nsaved = ( nwritten * 8 ) - packlen;
        System.err.println ("#bytes saved        ="+nsaved);
        System.err.println( "%saved by packing   ="+nsaved/(nwritten*8f)*100+"%");
        
    }
    
    public static void assertEquals( String msg, byte[] expected, byte[] actual )
    {
        assertEquals( msg+": length", expected.length, actual.length );
        for( int i=0; i<expected.length; i++ ) {
            assertEquals( msg+": byte[i="+i+"]", expected[i], actual[i] );
        }
    }

}

Index: TestAll.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/objndx/TestAll.java,v
retrieving revision 1.32
retrieving revision 1.33
diff -C2 -d -r1.32 -r1.33
*** TestAll.java	15 Mar 2007 16:11:10 -0000	1.32
--- TestAll.java	10 Apr 2007 18:33:31 -0000	1.33
***************
*** 34,38 ****
  
          TestSuite suite = new TestSuite("BTree");
! 
          /*
           * test key encoding and comparison support.
--- 34,45 ----
  
          TestSuite suite = new TestSuite("BTree");
!         
!         /*
!          * test fast DataOutput and DataInput implementations.
!          */
!         suite.addTestSuite(TestDataOutputBuffer.class);
!         suite.addTestSuite(TestShortPacker.class);
!         suite.addTestSuite(TestLongPacker.class);
!         
          /*
           * test key encoding and comparison support.

[cweb-CVS] bigdata/src/java/com/bigdata/journal ResourceManager.java, 1.2, 1.3 AbstractJournal.java, 1.10, 1.11

From: Bryan T. <tho...@us...> - 2007-04-10 18:33:38

Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/journal
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv24584/src/java/com/bigdata/journal

Modified Files:
	ResourceManager.java AbstractJournal.java 
Log Message:
Fixed a bug in UUID assignment to index segments.

Added some logic for fast data output for btree nodes and leaves.

Index: ResourceManager.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/journal/ResourceManager.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** ResourceManager.java	29 Mar 2007 17:01:33 -0000	1.2
--- ResourceManager.java	10 Apr 2007 18:33:31 -0000	1.3
***************
*** 53,56 ****
--- 53,57 ----
  import org.apache.log4j.Logger;
  
+ import com.bigdata.objndx.AbstractBTree;
  import com.bigdata.objndx.IndexSegment;
  import com.bigdata.objndx.IndexSegmentBuilder;
***************
*** 179,182 ****
--- 180,185 ----
       *       finalized (unisolated vs isolated vs index segment can be
       *       identified based on their interfaces).
+      * 
+      * @todo add reporting for {@link AbstractBTree#reopen()}.
       */
      static public void closeUnisolatedBTree(String name) {

Index: AbstractJournal.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/journal/AbstractJournal.java,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** AbstractJournal.java	4 Apr 2007 16:52:16 -0000	1.10
--- AbstractJournal.java	10 Apr 2007 18:33:31 -0000	1.11
***************
*** 158,167 ****
   * to secondary journals on failover hosts. </li>
   * <li> Scale-up database (automatic re-partitioning of indices and processing
!  * of deletion markers).</li>
!  * <li> AIO for the Direct and Disk modes.</li>
   * <li> GOM integration, including: support for primary key (clustered) indices;
   * using queues from GOM to journal/database segment server supporting both
   * embedded and remote scenarios; and using state-based conflict resolution to
!  * obtain high concurrency for generic objects, link set metadata, and indices.</li>
   * <li> Scale-out database, including:
   * <ul>
--- 158,170 ----
   * to secondary journals on failover hosts. </li>
   * <li> Scale-up database (automatic re-partitioning of indices and processing
!  * of deletion markers). Note that the split point must be choosen with some
!  * awareness of the application keys in order to provide an atomic row update
!  * guarentee when using keys formed as { primaryKey, columnName, timestamp }.</li>
!  * <li> AIO for the Direct and Disk modes (low priority since not IO bound).</li>
   * <li> GOM integration, including: support for primary key (clustered) indices;
   * using queues from GOM to journal/database segment server supporting both
   * embedded and remote scenarios; and using state-based conflict resolution to
!  * obtain high concurrency for generic objects, link set metadata, indices, and
!  * distributed split cache and hot cache support.</li>
   * <li> Scale-out database, including:
   * <ul>
***************
*** 177,180 ****
--- 180,189 ----
   * </ol>
   * 
+  * @todo Move Addr methods onto IRawStore and store the bitsplit point in the
+  *       root block. This will let us provision the journal to have more
+  *       distinct offsets at which records can be written (e.g., 35 bits for
+  *       that vs 32) by accepting a maximum record length with fewer bytes (29
+  *       vs 32). The bitsplit point can be choosen differently for each store.
+  * 
   * @todo Define distributed transaction protocol. Pay attention to 2-phase or
   *       3-phase commits when necessary, but take advantage of locality when
***************
*** 190,198 ****
   *       failover.
   * 
!  * @todo I need to revisit the assumptions for very large objects in the face of
!  *       the recent / planned redesign. I expect that using an index with a key
!  *       formed as [URI][chuck#] would work just fine. Chunks would then be
!  *       limited to 32k or so. Writes on such indices should probably be
!  *       directed to a journal using a disk-only mode.
   * 
   * @todo Checksums and/or record compression are currently handled on a per-{@link BTree}
--- 199,206 ----
   *       failover.
   * 
!  * @todo I need to revisit the assumptions for very large objects. I expect that
!  *       using an index with a key formed as [URI][chuck#] would work just fine.
!  *       Chunks would then be limited to 32k or so. Writes on such indices
!  *       should probably be directed to a journal using a disk-only mode.
   * 
   * @todo Checksums and/or record compression are currently handled on a per-{@link BTree}

[cweb-CVS] bigdata/src/test/com/bigdata/journal BenchmarkJournalWriteRate.java, 1.16, 1.17

From: Bryan T. <tho...@us...> - 2007-04-10 18:33:38

Update of /cvsroot/cweb/bigdata/src/test/com/bigdata/journal
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv24584/src/test/com/bigdata/journal

Modified Files:
	BenchmarkJournalWriteRate.java 
Log Message:
Fixed a bug in UUID assignment to index segments.

Added some logic for fast data output for btree nodes and leaves.

Index: BenchmarkJournalWriteRate.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/journal/BenchmarkJournalWriteRate.java,v
retrieving revision 1.16
retrieving revision 1.17
diff -C2 -d -r1.16 -r1.17
*** BenchmarkJournalWriteRate.java	4 Apr 2007 16:53:25 -0000	1.16
--- BenchmarkJournalWriteRate.java	10 Apr 2007 18:33:31 -0000	1.17
***************
*** 60,65 ****
--- 60,68 ----
  import junit.framework.TestSuite;
  
+ import com.bigdata.isolation.IIsolatableIndex;
  import com.bigdata.isolation.UnisolatedBTree;
  import com.bigdata.objndx.BTree;
+ import com.bigdata.objndx.ByteArrayValueSerializer;
+ import com.bigdata.objndx.IIndex;
  import com.bigdata.objndx.KeyBuilder;
  import com.bigdata.rawstore.Bytes;
***************
*** 210,213 ****
--- 213,217 ----
      protected int getBranchingFactor() {
          
+ //        return 16;
          return 256;
          
***************
*** 286,291 ****
      
      /**
!      * Test the index write rate without transactional isolation using 32 bit
!      * integer keys and 128 byte values for the index entries.
       */
      public void testUnisolatedIndexWriteRate() throws IOException {
--- 290,319 ----
      
      /**
!      * Test the index write rate using an index that does NOT support
!      * transactional isolation using 32 bit integer keys and 128 byte values for
!      * the index entries.
!      */
!     public void testNonIsolatableIndexWriteRate() throws IOException {
!         
!         // register named index that does NOT support isolation.
!         String name = "abc";
! 
!         journal.registerIndex(name, new BTree(journal, getBranchingFactor(),
!                 UUID.randomUUID(), ByteArrayValueSerializer.INSTANCE));
! 
!         journal.commit();
! 
!         // NOT isolated.
!         long tx = 0L;
! 
!         // run test.
!         doIndexWriteRateTest(name, tx, 128);
!         
!     }
! 
!     /**
!      * Test the index write rate using an index that supports transactional
!      * isolation but without transactional isolation using 32 bit integer keys
!      * and 128 byte values for the index entries.
       */
      public void testUnisolatedIndexWriteRate() throws IOException {
***************
*** 418,428 ****
          KeyBuilder keyBuilder = new KeyBuilder();
          
          System.err.println("Begin: index write rate, isolated="
!                 + (tx == 0 ? "no" : "yes") + ", bufferMode="
                  + journal._bufferStrategy.getBufferMode());
  
-         UnisolatedBTree ndx = (UnisolatedBTree) (tx == 0 ? journal
-                 .getIndex(name) : journal.getTx(tx).getIndex(name));
-         
          // target percentage full to avoid journal overflow.
          final double percentFull = .90;
--- 446,457 ----
          KeyBuilder keyBuilder = new KeyBuilder();
          
+         IIndex ndx = (tx == 0 ? journal.getIndex(name)
+                 : journal.getTx(tx).getIndex(name));
+ 
          System.err.println("Begin: index write rate, isolated="
!                 + (tx == 0 ? "no" : "yes") + ", isolatable="
!                 + (ndx instanceof IIsolatableIndex) + ", bufferMode="
                  + journal._bufferStrategy.getBufferMode());
  
          // target percentage full to avoid journal overflow.
          final double percentFull = .90;
***************
*** 431,448 ****
          final int nwrites = (int) (journal._bufferStrategy.getExtent()
                  * percentFull / valueSize);
-                 
-         final long begin = System.currentTimeMillis();
  
!         for( int i=0; i<nwrites; i++ ) {
  
!             // key[] is new on each insert; keys are monotonically increasing.
!             final byte[] key = keyBuilder.reset().append(i).getKey();
!             
!             // value[] is new on each insert.
!             final byte[] value = new byte[valueSize];
!             
!             value[0] = (byte)i; // at least one non-zero byte.
!             
!             ndx.insert(key,value);
              
          }
--- 460,484 ----
          final int nwrites = (int) (journal._bufferStrategy.getExtent()
                  * percentFull / valueSize);
  
!         final long begin;
  
!         {
! 
!             begin = System.currentTimeMillis();
! 
!             for (int i = 0; i < nwrites; i++) {
! 
!                 // key[] is new on each insert; keys are monotonically
!                 // increasing.
!                 final byte[] key = keyBuilder.reset().append(i).getKey();
! 
!                 // value[] is new on each insert.
!                 final byte[] value = new byte[valueSize];
! 
!                 value[0] = (byte) i; // at least one non-zero byte.
! 
!                 ndx.insert(key, value);
! 
!             }
              
          }
***************
*** 502,506 ****
  
          // The unisolated btree on which the data were actually written.
!         final UnisolatedBTree btree = (UnisolatedBTree)journal.getIndex(name);
          
          final int nodesWritten = btree.counters.getNodesWritten();
--- 538,542 ----
  
          // The unisolated btree on which the data were actually written.
!         final BTree btree = (BTree)journal.getIndex(name);
          
          final int nodesWritten = btree.counters.getNodesWritten();
***************
*** 803,807 ****
      /**
       * Runs the tests that have not been commented out :-)
!      * 
       * Note: Running all benchmarks together can challange the VM by running low
       * on heap, native memory given over to direct buffers - and things can actually
--- 839,843 ----
      /**
       * Runs the tests that have not been commented out :-)
!      * <p>
       * Note: Running all benchmarks together can challange the VM by running low
       * on heap, native memory given over to direct buffers - and things can actually
***************
*** 824,826 ****
--- 860,897 ----
      }
      
+     /**
+      * Main routine can be used for running the test under a performance
+      * analyzer.
+      * 
+      * @param args
+      *            Not used.
+      * 
+      * @throws Exception
+      */
+     public static void main(String[] args) throws Exception {
+         
+         BenchmarkTransientJournal test = new BenchmarkTransientJournal();
+         
+         test.setUp();
+         
+         try {
+ 
+             /*
+              * Choose one test to run. (You must setUp/tearDown for each test).
+              */
+             test.testNonIsolatableIndexWriteRate();
+ 
+ //          test.testUnisolatedIndexWriteRate();
+ 
+ //            test.testIsolatedIndexWriteRate();
+ 
+         }
+         finally {
+             
+             test.tearDown();
+             
+         }
+         
+     }
+     
  }

[cweb-CVS] bigdata/src/test/com/bigdata/journal BenchmarkJournalWriteRate.java, 1.15, 1.16

From: Bryan T. <tho...@us...> - 2007-04-04 16:53:28

Update of /cvsroot/cweb/bigdata/src/test/com/bigdata/journal
In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv12336/src/test/com/bigdata/journal

Modified Files:
	BenchmarkJournalWriteRate.java 
Log Message:
Uncommented some tests that should be run.

Index: BenchmarkJournalWriteRate.java
===================================================================
RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/journal/BenchmarkJournalWriteRate.java,v
retrieving revision 1.15
retrieving revision 1.16
diff -C2 -d -r1.15 -r1.16
*** BenchmarkJournalWriteRate.java	4 Apr 2007 16:52:16 -0000	1.15
--- BenchmarkJournalWriteRate.java	4 Apr 2007 16:53:25 -0000	1.16
***************
*** 812,819 ****
          TestSuite suite = new TestSuite("Benchmark Journal Write Rates");
          
! //        suite.addTestSuite( BenchmarkTransientJournal.class );
! //        suite.addTestSuite( BenchmarkDirectJournal.class );
! ////        suite.addTestSuite( BenchmarkMappedJournal.class );
! //        suite.addTestSuite( BenchmarkDiskJournal.class );
          suite.addTestSuite( BenchmarkSmallRecordOptimium.class );
          suite.addTestSuite( BenchmarkBlockBasedOptimium.class );
--- 812,819 ----
          TestSuite suite = new TestSuite("Benchmark Journal Write Rates");
          
!         suite.addTestSuite( BenchmarkTransientJournal.class );
!         suite.addTestSuite( BenchmarkDirectJournal.class );
! //        suite.addTestSuite( BenchmarkMappedJournal.class );
!         suite.addTestSuite( BenchmarkDiskJournal.class );
          suite.addTestSuite( BenchmarkSmallRecordOptimium.class );
          suite.addTestSuite( BenchmarkBlockBasedOptimium.class );

Flat | Threaded

<< < 1 .. 8 9 10 11 12 .. 91 > >> (Page 10 of 91)

2006	Jan	Feb	Mar (414)	Apr (123)	May (448)	Jun (180)	Jul (17)	Aug (49)	Sep (3)	Oct (92)	Nov (101)	Dec (64)
2007	Jan (132)	Feb (230)	Mar (146)	Apr (146)	May	Jun	Jul (34)	Aug (4)	Sep (3)	Oct (10)	Nov (12)	Dec (24)
2008	Jan (6)	Feb	Mar	Apr	May (1)	Jun	Jul	Aug	Sep	Oct (11)	Nov (4)	Dec
2009	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep (1)	Oct	Nov	Dec