Menu
▾
▴
bigdata-commit — Commit traffic.
This list is closed, nobody may subscribe to it.
| 2010 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(139) |
Aug
(94) |
Sep
(232) |
Oct
(143) |
Nov
(138) |
Dec
(55) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2011 |
Jan
(127) |
Feb
(90) |
Mar
(101) |
Apr
(74) |
May
(148) |
Jun
(241) |
Jul
(169) |
Aug
(121) |
Sep
(157) |
Oct
(199) |
Nov
(281) |
Dec
(75) |
| 2012 |
Jan
(107) |
Feb
(122) |
Mar
(184) |
Apr
(73) |
May
(14) |
Jun
(49) |
Jul
(26) |
Aug
(103) |
Sep
(133) |
Oct
(61) |
Nov
(51) |
Dec
(55) |
| 2013 |
Jan
(59) |
Feb
(72) |
Mar
(99) |
Apr
(62) |
May
(92) |
Jun
(19) |
Jul
(31) |
Aug
(138) |
Sep
(47) |
Oct
(83) |
Nov
(95) |
Dec
(111) |
| 2014 |
Jan
(125) |
Feb
(60) |
Mar
(119) |
Apr
(136) |
May
(270) |
Jun
(83) |
Jul
(88) |
Aug
(30) |
Sep
(47) |
Oct
(27) |
Nov
(23) |
Dec
|
| 2015 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(3) |
Oct
|
Nov
|
Dec
|
| 2016 |
Jan
|
Feb
|
Mar
(4) |
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: <mrp...@us...> - 2013-05-03 18:00:22
|
Revision: 7101
http://bigdata.svn.sourceforge.net/bigdata/?rev=7101&view=rev
Author: mrpersonick
Date: 2013-05-03 18:00:16 +0000 (Fri, 03 May 2013)
Log Message:
-----------
added an indexOf method
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/ModifiableBOpBase.java
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/ModifiableBOpBase.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/ModifiableBOpBase.java 2013-05-03 16:35:41 UTC (rev 7100)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/ModifiableBOpBase.java 2013-05-03 18:00:16 UTC (rev 7101)
@@ -164,6 +164,16 @@
return args.get(index);
}
+
+ /**
+ * Return the index of the bop in the args. Returns -1 if bop is not
+ * present in the args.
+ */
+ public int indexOf(final BOp bop) {
+
+ return args.indexOf(bop);
+
+ }
/**
* Invoked automatically any time a mutation operation occurs. The default
@@ -254,7 +264,7 @@
* Add a new argument (core mutation method) at the specified index.
*
* @param index
- * The index of the child expression to be replaced.
+ * The index at which the child expression is to be inserted.
* @param newArg
* The argument.
*
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-05-03 16:35:52
|
Revision: 7100
http://bigdata.svn.sourceforge.net/bigdata/?rev=7100&view=rev
Author: thompsonbry
Date: 2013-05-03 16:35:41 +0000 (Fri, 03 May 2013)
Log Message:
-----------
- RWStore.postHACommit(): Lock ordering problem was causing a deadlock
in testABCMultiTransactionFollowerReads(). The allocation lock must
be taken before the extension lock. Javadoc update.
- RWStore.resetFromRootBlock(). The same lock ordering problem
existed here. Javadoc update.
- RWStore: Added / updated javadoc for the allocation and extension
locks and reviewed all locking patterns in RWStore.
- RWStore: getData() requires shared access to the allocators. That
access must be MUTEX with allocation mutation in postHACommit().
Modified getData() to use the m_allocationReadLock for shared access
(was using the m_extension.readLock() which is to make file IO MUTEX
with file extension).
Note: The m_allocationReadLock COULD be pushed down into
physicalAddress() since that is the enter point for readers to
translate. Should it?
- RWStore: freeDeferrals(). Modified the code to not release
historical commit points if the key range scan on the commit record
index would have a toKey LT the fromKey. This was done to support
the HA TXS use case.
- RWStore.readRootBlock(). Modified to take the
m_extensionLock.readLock() to protect against concurrent file
extension.
- RWStore.readFromLatchedAddress(). Modified to take the allocator
read lock since accesses the allocators. Modified to take the
m_extensionLock.readLock() to protect against concurrent file
extension. This method is used by DumpJournal. DumpJournal can now
be invoked from the NSS on a live Journal.
- RWStore.writeRaw(). Modified to take the ReadLock of the
extensionLock to protect against concurrent file extension.
- RWStore.writeRawBuffer(). Removed the code that was taking the
allocation lock. It is not required for file IO.
- WORMStrategy.writeRawBuffer(). Removed the code that was taking the
file extension lock since it is always taken by writeOnChannel.
- RWStore.physicalAddress(). Must take the ReadLock of the
allocationLock since this is a public method and it reads on the
allocators.
- RWStore.getFixedAllocatorCount(). Must take allocationReadLock.
- RWStore.getAllocatedBlocks(). Must take allocationReadLock.
- RWStore.getFileStorage(). Must take allocationReadLock.
- RWStore.getAllocatorSlots(). Must take allocationReadLock.
- RWStore.computeDigest(). Made it easier to change between the two
digest methods (Old and Alt).
- TestJournalRW is green.
- TestWORMStrategy is green.
- TestHA3JournalStrategy.testABCMultiTransactionFollowerReads() is
green locally (I have other changes locally pertaining to how and
when the releaseTime is updated).
- Bug fix to TestHA3SnapshotPolicy. It needed to enabled online disaster recovery.
- All HA test are passing.
@see https://sourceforge.net/apps/trac/bigdata/ticket/530 (Journal HA)
Modified Paths:
--------------
branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/WORMStrategy.java
branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java
branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA3SnapshotPolicy2.java
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/WORMStrategy.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/WORMStrategy.java 2013-05-03 12:11:49 UTC (rev 7099)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/WORMStrategy.java 2013-05-03 16:35:41 UTC (rev 7100)
@@ -1167,7 +1167,7 @@
}
- @Override
+ @Override
public long getBlockSequence() {
return lastBlockSequence;
@@ -1238,8 +1238,8 @@
writeCacheService.close();
writeCacheService = newWriteCacheService();
} else {
- writeCacheService.reset();
- writeCacheService.setExtent(extent);
+ writeCacheService.reset();
+ writeCacheService.setExtent(extent);
}
} catch (InterruptedException e) {
throw new RuntimeException(e);
@@ -1545,10 +1545,10 @@
* @return the physical address of the offset provided
*/
private long offset2PhysicalAddress(final long offset) {
- return offset + headerSize;
- }
+ return offset + headerSize;
+ }
- /**
+ /**
* Read on the backing file. {@link ByteBuffer#remaining()} bytes will be
* read into the caller's buffer, starting at the specified offset in the
* backing file.
@@ -1567,7 +1567,7 @@
final Lock readLock = extensionLock.readLock();
readLock.lock();
try {
- final int startPos = dst.position();
+ final int startPos = dst.position();
try {
// the offset into the disk file.
@@ -1635,23 +1635,23 @@
*/
private FileChannel reopenChannel() throws IOException {
- /*
- * Note: This is basically a double-checked locking pattern. It is
- * used to avoid synchronizing when the backing channel is already
- * open.
- */
- {
- final RandomAccessFile tmp = raf;
- if (tmp != null) {
- final FileChannel channel = tmp.getChannel();
- if (channel.isOpen()) {
- // The channel is still open.
- return channel;
- }
- }
- }
+ /*
+ * Note: This is basically a double-checked locking pattern. It is
+ * used to avoid synchronizing when the backing channel is already
+ * open.
+ */
+ {
+ final RandomAccessFile tmp = raf;
+ if (tmp != null) {
+ final FileChannel channel = tmp.getChannel();
+ if (channel.isOpen()) {
+ // The channel is still open.
+ return channel;
+ }
+ }
+ }
- synchronized (opener) {
+ synchronized (opener) {
assertOpen();
@@ -1862,9 +1862,9 @@
offset = getOffset(addr);
- final long paddr = offset2PhysicalAddress(offset);
+ final long paddr = offset2PhysicalAddress(offset);
- boolean wroteOnCache = false;
+ boolean wroteOnCache = false;
if (writeCacheService != null) {
if (!writeCacheService.write(paddr, data, chk))
throw new AssertionError();
@@ -1952,9 +1952,9 @@
*/
private final ByteBuffer _checkbuf;
-// private HARebuildRequest m_rebuildRequest;
+// private HARebuildRequest m_rebuildRequest;
//
-// private int m_rebuildSequence;
+// private int m_rebuildSequence;
/**
* Make sure that the file is large enough to accept a write of
@@ -2431,13 +2431,13 @@
super.closeForWrites();
// do not discard the write cache, just reset it to preserve
- // read cache
+ // read cache
// releaseWriteCache();
try {
- writeCacheService.reset();
- } catch (InterruptedException e) {
- throw new RuntimeException(e);
- }
+ writeCacheService.reset();
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
}
@@ -2466,7 +2466,7 @@
* of this method are ignored.
*/
@Override
- public void delete(final long addr) {
+ public void delete(final long addr) {
if (writeCacheService != null) {
@@ -2517,44 +2517,49 @@
final int limit = bb.limit();
bb.position(limit);
- // Flush the write in the write cache to the backing store.
- final Lock readLock = extensionLock.readLock();
- readLock.lock();
- try {
+ /* Flush the write in the write cache to the backing store.
+ *
+ * Note: writeOnChannel() takes the extensionLock for us.
+ */
+// final Lock readLock = extensionLock.readLock();
+// readLock.lock();
+// try {
+
writeCache.flush(false/* force */);
-
- // install reads into readCache (if any)
- writeCacheService.installReads(writeCache);
- } finally {
- readLock.unlock();
- }
+// } finally {
+// readLock.unlock();
+// }
+
+ // install reads into readCache (if any)
+ writeCacheService.installReads(writeCache);
+
}
@Override
public Future<Void> sendHALogBuffer(final IHALogRequest req,
- final IHAWriteMessage msg, final IBufferAccess b)
- throws IOException, InterruptedException {
+ final IHAWriteMessage msg, final IBufferAccess b)
+ throws IOException, InterruptedException {
- // read direct from store
- final ByteBuffer clientBuffer = b.buffer();
- final int nbytes = msg.getSize();
- clientBuffer.position(0);
- clientBuffer.limit(nbytes);
+ // read direct from store
+ final ByteBuffer clientBuffer = b.buffer();
+ final int nbytes = msg.getSize();
+ clientBuffer.position(0);
+ clientBuffer.limit(nbytes);
readRaw(/*nbytes, */msg.getFirstOffset(), clientBuffer);
-
- assert clientBuffer.remaining() > 0 : "Empty buffer: " + clientBuffer;
+
+ assert clientBuffer.remaining() > 0 : "Empty buffer: " + clientBuffer;
- @SuppressWarnings("unchecked")
- final QuorumPipeline<HAPipelineGlue> quorumMember = (QuorumPipeline<HAPipelineGlue>) quorum
- .getMember();
+ @SuppressWarnings("unchecked")
+ final QuorumPipeline<HAPipelineGlue> quorumMember = (QuorumPipeline<HAPipelineGlue>) quorum
+ .getMember();
- final Future<Void> remoteWriteFuture = quorumMember.replicate(req, msg,
- clientBuffer);
+ final Future<Void> remoteWriteFuture = quorumMember.replicate(req, msg,
+ clientBuffer);
- return remoteWriteFuture;
- }
+ return remoteWriteFuture;
+ }
@Override
public Future<Void> sendRawBuffer(final IHARebuildRequest req,
@@ -2863,38 +2868,38 @@
}
}
- @Override
- public void writeRawBuffer(HARebuildRequest req, IHAWriteMessage msg,
- ByteBuffer transfer) throws IOException {
-// if (m_rebuildRequest == null)
-// throw new IllegalStateException("Store is not in rebuild state");
-//
-// if (m_rebuildSequence != msg.getSequence())
-// throw new IllegalStateException("Invalid sequence number for rebuild, expected: " + m_rebuildSequence + ", actual: " + msg.getSequence());
+ @Override
+ public void writeRawBuffer(HARebuildRequest req, IHAWriteMessage msg,
+ ByteBuffer transfer) throws IOException {
+// if (m_rebuildRequest == null)
+// throw new IllegalStateException("Store is not in rebuild state");
+//
+// if (m_rebuildSequence != msg.getSequence())
+// throw new IllegalStateException("Invalid sequence number for rebuild, expected: " + m_rebuildSequence + ", actual: " + msg.getSequence());
- FileChannelUtility.writeAll(this.opener, transfer, msg.getFirstOffset());
-
-// m_rebuildSequence++;
- }
+ FileChannelUtility.writeAll(this.opener, transfer, msg.getFirstOffset());
+
+// m_rebuildSequence++;
+ }
-// @Override
-// public void prepareForRebuild(HARebuildRequest req) {
-// assert m_rebuildRequest == null;
-//
-// m_rebuildRequest = req;
-// m_rebuildSequence = 0;
-// }
+// @Override
+// public void prepareForRebuild(HARebuildRequest req) {
+// assert m_rebuildRequest == null;
+//
+// m_rebuildRequest = req;
+// m_rebuildSequence = 0;
+// }
//
-// @Override
-// public void completeRebuild(final HARebuildRequest req, final IRootBlockView rbv) {
-// assert m_rebuildRequest != null;
-//
-// assert m_rebuildRequest.equals(req);
-//
-// // TODO: reinit from file
-// this.resetFromHARootBlock(rbv);
-//
-// m_rebuildRequest = null;
-// }
-
+// @Override
+// public void completeRebuild(final HARebuildRequest req, final IRootBlockView rbv) {
+// assert m_rebuildRequest != null;
+//
+// assert m_rebuildRequest.equals(req);
+//
+// // TODO: reinit from file
+// this.resetFromHARootBlock(rbv);
+//
+// m_rebuildRequest = null;
+// }
+
}
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2013-05-03 12:11:49 UTC (rev 7099)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2013-05-03 16:35:41 UTC (rev 7100)
@@ -53,7 +53,6 @@
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
import java.util.concurrent.locks.Lock;
-import java.util.concurrent.locks.ReentrantLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock.ReadLock;
import java.util.concurrent.locks.ReentrantReadWriteLock.WriteLock;
@@ -267,27 +266,27 @@
*/
public interface Options {
- /**
- * Option defines the Allocation block sizes for the RWStore. The values
- * defined are multiplied by 64 to provide the actual allocations. The
- * list of allocations should be ',' delimited and in increasing order.
- * This array is written into the store so changing the values does not
- * break older stores. For example,
- *
- * <pre>
- * "1,2,4,8,116,32,64"
- * </pre>
- *
- * defines allocations from 64 to 4K in size. It is a good to define
- * block sizes on 4K boundaries as soon as possible to optimize IO. This
- * is particularly relevant for SSDs. A 1K boundary is expressed as
- * <code>16</code> in the allocation sizes, so a 4K boundary is
- * expressed as <code>64</code> and an 8k boundary as <code>128</code>.
- * <p>
- * The default allocations are {@value #DEFAULT_ALLOCATION_SIZES}.
- *
- * @see #DEFAULT_ALLOCATION_SIZES
- */
+ /**
+ * Option defines the Allocation block sizes for the RWStore. The values
+ * defined are multiplied by 64 to provide the actual allocations. The
+ * list of allocations should be ',' delimited and in increasing order.
+ * This array is written into the store so changing the values does not
+ * break older stores. For example,
+ *
+ * <pre>
+ * "1,2,4,8,116,32,64"
+ * </pre>
+ *
+ * defines allocations from 64 to 4K in size. It is a good to define
+ * block sizes on 4K boundaries as soon as possible to optimize IO. This
+ * is particularly relevant for SSDs. A 1K boundary is expressed as
+ * <code>16</code> in the allocation sizes, so a 4K boundary is
+ * expressed as <code>64</code> and an 8k boundary as <code>128</code>.
+ * <p>
+ * The default allocations are {@value #DEFAULT_ALLOCATION_SIZES}.
+ *
+ * @see #DEFAULT_ALLOCATION_SIZES
+ */
String ALLOCATION_SIZES = RWStore.class.getName() + ".allocationSizes";
/**
@@ -335,12 +334,12 @@
String DEFAULT_FREE_BITS_THRESHOLD = "300";
- /**
- * When <code>true</code>, scattered writes which are strictly ascending
- * will be coalesced within a buffer and written out as a single IO
- * (default {@value #DEFAULT_DOUBLE_BUFFER_WRITES}). This improves write
- * performance for SATA, SAS, and even SSD.
- */
+ /**
+ * When <code>true</code>, scattered writes which are strictly ascending
+ * will be coalesced within a buffer and written out as a single IO
+ * (default {@value #DEFAULT_DOUBLE_BUFFER_WRITES}). This improves write
+ * performance for SATA, SAS, and even SSD.
+ */
String DOUBLE_BUFFER_WRITES = RWStore.class.getName() + ".doubleBuffer";
String DEFAULT_DOUBLE_BUFFER_WRITES = "true";
@@ -370,19 +369,19 @@
private static final String ERR_WRITE_CACHE_CREATE = "Unable to create write cache service";
- /**
- * The fixed size of any allocator on the disk in bytes. The #of allocations
- * managed by an allocator is this value times 8 because each slot uses one
- * bit in the allocator. When an allocator is allocated, the space on the
- * persistent heap is reserved for all slots managed by that allocator.
- * However, the {@link FixedAllocator} only incrementally allocates the
- * {@link AllocBlock}s.
- */
- static private final int ALLOC_BLOCK_SIZE = 1024;
-
-// // from 32 bits, need 13 to hold max offset of 8 * 1024, leaving 19 for number of blocks: 256K
-// static final int BLOCK_INDEX_BITS = 19;
/**
+ * The fixed size of any allocator on the disk in bytes. The #of allocations
+ * managed by an allocator is this value times 8 because each slot uses one
+ * bit in the allocator. When an allocator is allocated, the space on the
+ * persistent heap is reserved for all slots managed by that allocator.
+ * However, the {@link FixedAllocator} only incrementally allocates the
+ * {@link AllocBlock}s.
+ */
+ static private final int ALLOC_BLOCK_SIZE = 1024;
+
+// // from 32 bits, need 13 to hold max offset of 8 * 1024, leaving 19 for number of blocks: 256K
+// static final int BLOCK_INDEX_BITS = 19;
+ /**
* The #of low bits in a latched address that encode the offset of the bit
* in a {@link FixedAllocator}. The {@link FixedAllocator} will map the bit
* onto an allocation slot.
@@ -392,107 +391,107 @@
* order in which it was created. This is used to index into
* {@link #m_allocs}, which are the {@link FixedAllocator}s.
*/
- static final int OFFSET_BITS = 13;
- static final int OFFSET_BITS_MASK = 0x1FFF; // was 0xFFFF
-
- static final int ALLOCATION_SCALEUP = 16; // multiplier to convert allocations based on minimum allocation of 32k
- static private final int META_ALLOCATION = 8; // 8 * 32K is size of meta Allocation
+ static final int OFFSET_BITS = 13;
+ static final int OFFSET_BITS_MASK = 0x1FFF; // was 0xFFFF
+
+ static final int ALLOCATION_SCALEUP = 16; // multiplier to convert allocations based on minimum allocation of 32k
+ static private final int META_ALLOCATION = 8; // 8 * 32K is size of meta Allocation
- // If required, then allocate 1M direct buffers
- private static final int cDirectBufferCapacity = 1024 * 1024;
+ // If required, then allocate 1M direct buffers
+ private static final int cDirectBufferCapacity = 1024 * 1024;
- private int cMaxDirectBuffers = 20; // 20M of direct buffers
- static final int cDirectAllocationOffset = 64 * 1024;
+ private int cMaxDirectBuffers = 20; // 20M of direct buffers
+ static final int cDirectAllocationOffset = 64 * 1024;
- // ///////////////////////////////////////////////////////////////////////////////////////
- // RWStore Data
- // ///////////////////////////////////////////////////////////////////////////////////////
+ // ///////////////////////////////////////////////////////////////////////////////////////
+ // RWStore Data
+ // ///////////////////////////////////////////////////////////////////////////////////////
- private final File m_fd;
-// private RandomAccessFile m_raf;
-// protected FileMetadata m_metadata;
-// protected int m_transactionCount;
-// private boolean m_committing;
+ private final File m_fd;
+// private RandomAccessFile m_raf;
+// protected FileMetadata m_metadata;
+// protected int m_transactionCount;
+// private boolean m_committing;
// /**
// * When <code>true</code> the allocations will not actually be recycled
// * until after a store restart. When <code>false</code>, the allocations are
// * recycled once they satisfy the history retention requirement.
// */
-// private boolean m_preserveSession = false;
-// private boolean m_readOnly;
+// private boolean m_preserveSession = false;
+// private boolean m_readOnly;
- /**
- * The UUID of the backing store.
- *
- * @see #initfromRootBlock(IRootBlockView)
- * @see IRawStore#getUUID()
- */
- private UUID m_storeUUID;
-
- /**
- * lists of total alloc blocks.
- *
- * @todo examine concurrency and lock usage for {@link #m_alloc} and the
- * rest of these lists.
- */
- private final ArrayList<FixedAllocator> m_allocs;
+ /**
+ * The UUID of the backing store.
+ *
+ * @see #initfromRootBlock(IRootBlockView)
+ * @see IRawStore#getUUID()
+ */
+ private UUID m_storeUUID;
+
+ /**
+ * lists of total alloc blocks.
+ *
+ * @todo examine concurrency and lock usage for {@link #m_alloc} and the
+ * rest of these lists.
+ */
+ private final ArrayList<FixedAllocator> m_allocs;
- /**
- * A fixed length array of lists of free {@link FixedAllocator}s with one
- * entry in the array for each configured allocator size. An allocator is
- * put onto this free list when it is initially created. When the store is
- * opened, it will be added to this list if {@link Allocator#hasFree()}
- * returns true. It will be removed when it has no free space remaining. It
- * will be added back to the free list when its free slots exceeds a
- * configured threshold.
- */
- private ArrayList<FixedAllocator> m_freeFixed[];
-
-// /** lists of free blob allocators. */
- // private final ArrayList<BlobAllocator> m_freeBlobs;
+ /**
+ * A fixed length array of lists of free {@link FixedAllocator}s with one
+ * entry in the array for each configured allocator size. An allocator is
+ * put onto this free list when it is initially created. When the store is
+ * opened, it will be added to this list if {@link Allocator#hasFree()}
+ * returns true. It will be removed when it has no free space remaining. It
+ * will be added back to the free list when its free slots exceeds a
+ * configured threshold.
+ */
+ private ArrayList<FixedAllocator> m_freeFixed[];
+
+// /** lists of free blob allocators. */
+ // private final ArrayList<BlobAllocator> m_freeBlobs;
- /** lists of blocks requiring commitment. */
- private final ArrayList<FixedAllocator> m_commitList;
+ /** lists of blocks requiring commitment. */
+ private final ArrayList<FixedAllocator> m_commitList;
-// private WriteBlock m_writes;
-
- private final Quorum<?,?> m_quorum;
-
- /**
- * The #of buffers that will be used by the {@link WriteCacheService}.
- *
- * @see com.bigdata.journal.Options#WRITE_CACHE_BUFFER_COUNT
- */
- private final int m_writeCacheBufferCount;
+// private WriteBlock m_writes;
+
+ private final Quorum<?,?> m_quorum;
+
+ /**
+ * The #of buffers that will be used by the {@link WriteCacheService}.
+ *
+ * @see com.bigdata.journal.Options#WRITE_CACHE_BUFFER_COUNT
+ */
+ private final int m_writeCacheBufferCount;
- /**
- * @see com.bigdata.journal.Options#WRITE_CACHE_MIN_CLEAN_LIST_SIZE
- */
+ /**
+ * @see com.bigdata.journal.Options#WRITE_CACHE_MIN_CLEAN_LIST_SIZE
+ */
private final int m_minCleanListSize;
- /**
- * The #of read buffers that will be used by the {@link WriteCacheService}.
- *
- * @see com.bigdata.journal.Options#READ_CACHE_BUFFER_COUNT
- */
- private final int m_readCacheBufferCount;
+ /**
+ * The #of read buffers that will be used by the {@link WriteCacheService}.
+ *
+ * @see com.bigdata.journal.Options#READ_CACHE_BUFFER_COUNT
+ */
+ private final int m_readCacheBufferCount;
/**
* @see com.bigdata.journal.Options#WRITE_CACHE_COMPACTION_THRESHOLD
*/
- private final int m_compactionThreshold;
-
+ private final int m_compactionThreshold;
+
/**
* @see com.bigdata.journal.Options#HOT_CACHE_THRESHOLD
*/
- private final int m_hotCacheThreshold;
-
+ private final int m_hotCacheThreshold;
+
/**
* @see com.bigdata.journal.Options#HOT_CACHE_SIZE
*/
- private final int m_hotCacheSize;
-
+ private final int m_hotCacheSize;
+
/**
* Note: This is not final because we replace the {@link WriteCacheService}
* during {@link #reset(long)} in order to propagate the then current quorum
@@ -500,79 +499,110 @@
*/
RWWriteCacheService m_writeCacheService;
- /**
- * The actual allocation sizes as read from the store.
- *
- * @see #DEFAULT_ALLOCATION_SIZES
- */
- private int[] m_allocSizes;
+ /**
+ * The actual allocation sizes as read from the store.
+ *
+ * @see #DEFAULT_ALLOCATION_SIZES
+ */
+ private int[] m_allocSizes;
- /**
- * The maximum allocation size (bytes).
- */
+ /**
+ * The maximum allocation size (bytes).
+ */
final int m_maxFixedAlloc;
/**
* The minimum allocation size (bytes).
*/
final int m_minFixedAlloc;
-
+
/**
* We allow blob headers so the maximum blob size is Integer.MAX_VALUE.
*/
final int m_maxBlobAllocSize = Integer.MAX_VALUE;
-
+
/**
- * This lock is used to exclude readers when the extent of the backing file
- * is about to be changed.
+ * This lock is used to exclude readers/writers performing IOs against the
+ * backing file when the extent of the backing file is about to be changed.
+ * Readers and writers take the {@link ReadLock}. The {@link WriteLock} is
+ * taken when the file extent must be changed. This is a workaround for an
+ * old (an unresolved as of February 2010) Sun bug.
* <p>
- * At present we use synchronized (this) for alloc/commitChanges and
- * getData, since only alloc and commitChanges can cause a file extend, and
- * only getData can read.
+ * Note: Any public method that ONLY takes the extensionLock MUST NOT make
+ * calls that could take the {@link #m_allocationLock}. This would cause a
+ * lock ordering problem. If both locks must be taken, then the
+ * {@link #m_allocationLock} MUST be taken first.
+ *
+ * @see http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6371642
+ * @see #m_allocationLock
+ */
+ final private ReentrantReadWriteLock m_extensionLock = new ReentrantReadWriteLock();
+
+ /**
+ * An explicit allocation lock supports exclusive access for allocator
+ * mutation and shared access for readers.
* <p>
- * By using an explicit extensionLock we can unsure that that the taking of
- * the lock is directly related to the functionality, plus we can support
- * concurrent reads.
+ * Note: You must hold the {@link #m_allocationReadLock} to read the
+ * allocators.
* <p>
- * You MUST hold the {@link #m_allocationLock} before acquiring the
- * {@link ReentrantReadWriteLock#writeLock()} of the
- * {@link #m_extensionLock}.
+ * Note: You must hold the {@link #m_allocationWriteLock} while allocating
+ * or clearing allocations.
+ * <p>
+ * Note: It is only when an allocation triggers a file extension that the
+ * {@link WriteLock} of the {@link #m_extensionLock} needs to be taken.
+ *
+ * TODO: There is scope to take advantage of the different allocator sizes
+ * and provide allocation locks on the fixed allocators. We will still need
+ * a store-wide allocation lock when creating new allocation areas, but
+ * significant contention may be avoided.
*/
- final private ReentrantReadWriteLock m_extensionLock = new ReentrantReadWriteLock();
-
- /**
- * An explicit allocation lock allows for reads concurrent with allocation
- * requests. You must hold the allocation lock while allocating or clearing
- * allocations. It is only when an allocation triggers a file extension that
- * the write extensionLock needs to be taken.
- *
- * TODO: There is scope to take advantage of the different allocator sizes
- * and provide allocation locks on the fixed allocators. We will still need
- * a store-wide allocation lock when creating new allocation areas, but
- * significant contention may be avoided.
- */
final private ReentrantReadWriteLock m_allocationLock = new ReentrantReadWriteLock();
+ /**
+ * Lock used for exclusive access to the allocators.
+ * <p>
+ * Note: Historically, this lock was only required for mutation and readers
+ * did not content for a lock.
+ */
final private WriteLock m_allocationWriteLock = m_allocationLock.writeLock();
+ /**
+ * Lock used for shared access to allocators.
+ * <p>
+ * Note: Historically the allocators were unprotected for shared acccess
+ * (readers) and protected by a single lock for mutation (writes). Shared
+ * access by readers was safe since (a) old allocators were never replaced;
+ * and (b) readers had access only to committed data.
+ * <p>
+ * This situation was changed when the {@link #postHACommit(IRootBlockView)}
+ * method was introduced since it could replace allocators in a manner that
+ * was not safe for shared access by readers. Methods that were historically
+ * using unprotected shared access now require protected shared access using
+ * this lock.
+ *
+ * @see #postHACommit(IRootBlockView)
+ * @see #getData(long, int)
+ * @see #getData(long, byte[])
+ * @see #getData(long, byte[], int, int)
+ */
final private ReadLock m_allocationReadLock = m_allocationLock.readLock();
- /**
- * The deferredFreeList is simply an array of releaseTime,freeListAddrs
- * stored at commit.
- * <p>
- * Note that when the deferredFreeList is saved, ONLY thefreeListAddrs
- * are stored, NOT the releaseTime. This is because on any open of
- * the store, all deferredFrees can be released immediately. This
- * mechanism may be changed in the future to enable explicit history
- * retention, but if so a different header structure would be used since
- * it would not be appropriate to retain a simple header linked to
- * thousands if not millions of commit points.
- */
+ /**
+ * The deferredFreeList is simply an array of releaseTime,freeListAddrs
+ * stored at commit.
+ * <p>
+ * Note that when the deferredFreeList is saved, ONLY thefreeListAddrs
+ * are stored, NOT the releaseTime. This is because on any open of
+ * the store, all deferredFrees can be released immediately. This
+ * mechanism may be changed in the future to enable explicit history
+ * retention, but if so a different header structure would be used since
+ * it would not be appropriate to retain a simple header linked to
+ * thousands if not millions of commit points.
+ */
// *
// * If the current txn list exceeds the MAX_DEFERRED_FREE then it is
// * incrementally saved and a new list begun. The master list itself
// * serves as a BLOB header when there is more than a single entry with
// * the same txReleaseTime.
-// private static final int MAX_DEFERRED_FREE = 4094; // fits in 16k block
+// private static final int MAX_DEFERRED_FREE = 4094; // fits in 16k block
private final long m_minReleaseAge;
/**
@@ -582,25 +612,25 @@
*/
private int m_activeTxCount = 0;
- private volatile long m_lastDeferredReleaseTime = 0L;
-// private final ArrayList<Integer> m_currentTxnFreeList = new ArrayList<Integer>();
- private final PSOutputStream m_deferredFreeOut;
+ private volatile long m_lastDeferredReleaseTime = 0L;
+// private final ArrayList<Integer> m_currentTxnFreeList = new ArrayList<Integer>();
+ private final PSOutputStream m_deferredFreeOut;
/**
* Used to transparently re-open the backing channel if it has been closed
* by an interrupt during an IO.
*/
- private final ReopenFileChannel m_reopener;
+ private final ReopenFileChannel m_reopener;
- private volatile BufferedWrite m_bufferedWrite;
-
- /**
- * Our StoreageStats objects
- */
- private StorageStats m_storageStats;
- private long m_storageStatsAddr = 0;
-
+ private volatile BufferedWrite m_bufferedWrite;
+
/**
+ * Our StoreageStats objects
+ */
+ private StorageStats m_storageStats;
+ private long m_storageStatsAddr = 0;
+
+ /**
* <code>true</code> iff the backing store is open.
*/
private volatile boolean m_open = true;
@@ -619,7 +649,7 @@
// private ConcurrentHashMap<Integer, String> m_blacklist = null;
private ConcurrentHashMap<Integer, Long> m_lockAddresses = null;
- class WriteCacheImpl extends WriteCache.FileChannelScatteredWriteCache {
+ class WriteCacheImpl extends WriteCache.FileChannelScatteredWriteCache {
public WriteCacheImpl(final IBufferAccess buf,
final boolean useChecksum,
final boolean bufferHasData,
@@ -661,23 +691,23 @@
// Added to enable debug of rare problem
// FIXME: disable by removal once solved
protected void registerWriteStatus(long offset, int length, char action) {
- m_writeCacheService.debugAddrs(offset, length, action);
+ m_writeCacheService.debugAddrs(offset, length, action);
}
-
+
@Override
protected void addAddress(int latchedAddr, int size) {
- // No longer valid
- // RWStore.this.addAddress(latchedAddr, size);
+ // No longer valid
+ // RWStore.this.addAddress(latchedAddr, size);
}
@Override
protected void removeAddress(int latchedAddr) {
- // No longer valid
- // RWStore.this.removeAddress(latchedAddr);
+ // No longer valid
+ // RWStore.this.removeAddress(latchedAddr);
}
- };
-
+ };
+
/**
* The ALLOC_SIZES must be initialized from either the file or the
* properties associated with the fileMetadataView
@@ -722,50 +752,50 @@
}
m_metaBits = new int[m_metaBitsSize];
-
- m_metaTransientBits = new int[m_metaBitsSize];
-
-
+
+ m_metaTransientBits = new int[m_metaBitsSize];
+
+
m_quorum = quorum;
-
- m_fd = fileMetadata.file;
-
- // initialize striped performance counters for this store.
+
+ m_fd = fileMetadata.file;
+
+ // initialize striped performance counters for this store.
this.storeCounters.set(new StoreCounters(10/* batchSize */));
- final IRootBlockView m_rb = fileMetadata.rootBlock;
+ final IRootBlockView m_rb = fileMetadata.rootBlock;
- m_commitList = new ArrayList<FixedAllocator>();
+ m_commitList = new ArrayList<FixedAllocator>();
- m_allocs = new ArrayList<FixedAllocator>();
-
- // m_freeBlobs = new ArrayList<BlobAllocator>();
+ m_allocs = new ArrayList<FixedAllocator>();
+
+ // m_freeBlobs = new ArrayList<BlobAllocator>();
- try {
- final RandomAccessFile m_raf = fileMetadata.getRandomAccessFile();
- m_reopener = new ReopenFileChannel(m_fd, m_raf, "rw");
- } catch (IOException e1) {
- throw new RuntimeException(e1);
- }
+ try {
+ final RandomAccessFile m_raf = fileMetadata.getRandomAccessFile();
+ m_reopener = new ReopenFileChannel(m_fd, m_raf, "rw");
+ } catch (IOException e1) {
+ throw new RuntimeException(e1);
+ }
- if (Boolean.valueOf(fileMetadata.getProperty(
- Options.DOUBLE_BUFFER_WRITES,
- Options.DEFAULT_DOUBLE_BUFFER_WRITES))) {
- try {
- m_bufferedWrite = new BufferedWrite(this);
- } catch (InterruptedException e1) {
- m_bufferedWrite = null;
- }
- } else {
- m_bufferedWrite = null;
- }
+ if (Boolean.valueOf(fileMetadata.getProperty(
+ Options.DOUBLE_BUFFER_WRITES,
+ Options.DEFAULT_DOUBLE_BUFFER_WRITES))) {
+ try {
+ m_bufferedWrite = new BufferedWrite(this);
+ } catch (InterruptedException e1) {
+ m_bufferedWrite = null;
+ }
+ } else {
+ m_bufferedWrite = null;
+ }
- m_writeCacheBufferCount = fileMetadata.writeCacheBufferCount;
-
- m_readCacheBufferCount = Integer.valueOf(fileMetadata.getProperty(
+ m_writeCacheBufferCount = fileMetadata.writeCacheBufferCount;
+
+ m_readCacheBufferCount = Integer.valueOf(fileMetadata.getProperty(
com.bigdata.journal.Options.READ_CACHE_BUFFER_COUNT,
com.bigdata.journal.Options.DEFAULT_READ_CACHE_BUFFER_COUNT));
-
+
if (log.isInfoEnabled())
log.info(com.bigdata.journal.Options.WRITE_CACHE_BUFFER_COUNT
+ "=" + m_writeCacheBufferCount);
@@ -804,87 +834,87 @@
// m_writeCache = newWriteCache();
- try {
+ try {
if (m_rb.getNextOffset() == 0) { // if zero then new file
- setAllocations(fileMetadata);
+ setAllocations(fileMetadata);
/*
* FIXME Martyn, the code paths here are crazy complicated.
* defaultInit() is also invoked from initFromRootBlock().
* Simplify this. BBT
*/
- m_storeUUID = m_rb.getUUID();
+ m_storeUUID = m_rb.getUUID();
- defaultInit();
-
- m_maxFixedAlloc = m_allocSizes[m_allocSizes.length-1]*64;
- m_minFixedAlloc = m_allocSizes[0]*64;
-
- m_storageStats = new StorageStats(m_allocSizes);
+ defaultInit();
+
+ m_maxFixedAlloc = m_allocSizes[m_allocSizes.length-1]*64;
+ m_minFixedAlloc = m_allocSizes[0]*64;
+
+ m_storageStats = new StorageStats(m_allocSizes);
-// // Check for overwrite option and set overwrite buffer if
-// // required
-// if (Boolean.valueOf(fileMetadata.getProperty(
-// Options.OVERWRITE_DELETE,
-// Options.DEFAULT_OVERWRITE_DELETE))) {
-// m_writeCache.setOverwriteBuffer(m_maxFixedAlloc);
-// }
- } else {
-
- initfromRootBlock(m_rb);
-
- m_maxFixedAlloc = m_allocSizes[m_allocSizes.length-1]*64;
- m_minFixedAlloc = m_allocSizes[0]*64;
+// // Check for overwrite option and set overwrite buffer if
+// // required
+// if (Boolean.valueOf(fileMetadata.getProperty(
+// Options.OVERWRITE_DELETE,
+// Options.DEFAULT_OVERWRITE_DELETE))) {
+// m_writeCache.setOverwriteBuffer(m_maxFixedAlloc);
+// }
+ } else {
+
+ initfromRootBlock(m_rb);
+
+ m_maxFixedAlloc = m_allocSizes[m_allocSizes.length-1]*64;
+ m_minFixedAlloc = m_allocSizes[0]*64;
- if (m_storageStatsAddr != 0) {
- final long statsAddr = m_storageStatsAddr >> 16;
- final int statsLen = ((int) m_storageStatsAddr) & 0xFFFF;
- final byte[] stats = new byte[statsLen + 4]; // allow for checksum
- getData(statsAddr, stats);
- final DataInputStream instr = new DataInputStream(new ByteArrayInputStream(stats));
- m_storageStats = new StorageStats(instr);
-
- for (FixedAllocator fa: m_allocs) {
- m_storageStats.register(fa);
- }
- } else {
- m_storageStats = new StorageStats(m_allocSizes);
- }
-
+ if (m_storageStatsAddr != 0) {
+ final long statsAddr = m_storageStatsAddr >> 16;
+ final int statsLen = ((int) m_storageStatsAddr) & 0xFFFF;
+ final byte[] stats = new byte[statsLen + 4]; // allow for checksum
+ getData(statsAddr, stats);
+ final DataInputStream instr = new DataInputStream(new ByteArrayInputStream(stats));
+ m_storageStats = new StorageStats(instr);
+
+ for (FixedAllocator fa: m_allocs) {
+ m_storageStats.register(fa);
+ }
+ } else {
+ m_storageStats = new StorageStats(m_allocSizes);
+ }
+
if (log.isTraceEnabled()) {
final StringBuilder str = new StringBuilder();
this.showAllocators(str);
log.trace(str);
}
- }
-
- // Maximum theoretically addressable file size is determined by the
- // maximum allocator slot size multiplied by Integer.MAX_VALUE
+ }
+
+ // Maximum theoretically addressable file size is determined by the
+ // maximum allocator slot size multiplied by Integer.MAX_VALUE
// FIXME: do we want to constrain this as a system property?
- m_maxFileSize = ((long) Integer.MAX_VALUE) * m_maxFixedAlloc;
+ m_maxFileSize = ((long) Integer.MAX_VALUE) * m_maxFixedAlloc;
- // setup write cache AFTER init to ensure filesize is correct!
+ // setup write cache AFTER init to ensure filesize is correct!
- m_writeCacheService = newWriteCache();
+ m_writeCacheService = newWriteCache();
- final int maxBlockLessChk = m_maxFixedAlloc-4;
+ final int maxBlockLessChk = m_maxFixedAlloc-4;
- assert m_maxFixedAlloc > 0;
-
- m_deferredFreeOut = PSOutputStream.getNew(this, m_maxFixedAlloc, null);
+ assert m_maxFixedAlloc > 0;
+
+ m_deferredFreeOut = PSOutputStream.getNew(this, m_maxFixedAlloc, null);
-// if (Boolean.valueOf(fileMetadata.getProperty(
-// Options.MAINTAIN_BLACKLIST,
-// Options.DEFAULT_MAINTAIN_BLACKLIST))) {
-// m_blacklist = new ConcurrentHashMap<Integer, String>();
-// m_lockAddresses = new ConcurrentHashMap<Integer, Long>();
-// }
+// if (Boolean.valueOf(fileMetadata.getProperty(
+// Options.MAINTAIN_BLACKLIST,
+// Options.DEFAULT_MAINTAIN_BLACKLIST))) {
+// m_blacklist = new ConcurrentHashMap<Integer, String>();
+// m_lockAddresses = new ConcurrentHashMap<Integer, Long>();
+// }
- } catch (IOException e) {
- throw new StorageTerminalError("Unable to initialize store", e);
- }
- }
+ } catch (IOException e) {
+ throw new StorageTerminalError("Unable to initialize store", e);
+ }
+ }
/**
* Called from WriteCache.resetRecordMapFromBuffer
@@ -901,59 +931,59 @@
* <i>latchedAddr</i> but the address itself should not yet be
* allocated.
*/
- void addAddress(final int latchedAddr, final int size) {
- // ignore zero address
- if (latchedAddr == 0)
- return;
+ void addAddress(final int latchedAddr, final int size) {
+ // ignore zero address
+ if (latchedAddr == 0)
+ return;
- m_allocationWriteLock.lock();
- try {
- FixedAllocator alloc = null;
- try {
- alloc = getBlock(latchedAddr);
- } catch (final PhysicalAddressResolutionException par) {
- // Must create new allocator
- }
- final int size2 = size < 0 ? -size : size;
- if (alloc == null) {
- final int i = fixedAllocatorIndex(size2);
- final int block = 64 * m_allocSizes[i];
- final ArrayList<FixedAllocator> list = m_freeFixed[i];
- if (log.isTraceEnabled())
- log.trace("Creating new Allocator for address: "
- + latchedAddr);
+ m_allocationWriteLock.lock();
+ try {
+ FixedAllocator alloc = null;
+ try {
+ alloc = getBlock(latchedAddr);
+ } catch (final PhysicalAddressResolutionException par) {
+ // Must create new allocator
+ }
+ final int size2 = size < 0 ? -size : size;
+ if (alloc == null) {
+ final int i = fixedAllocatorIndex(size2);
+ final int block = 64 * m_allocSizes[i];
+ final ArrayList<FixedAllocator> list = m_freeFixed[i];
+ if (log.isTraceEnabled())
+ log.trace("Creating new Allocator for address: "
+ + latchedAddr);
- final FixedAllocator allocator = new FixedAllocator(this, block);
+ final FixedAllocator allocator = new FixedAllocator(this, block);
- allocator.setFreeList(list);
- allocator.setIndex(m_allocs.size());
+ allocator.setFreeList(list);
+ allocator.setIndex(m_allocs.size());
- m_allocs.add(allocator);
+ m_allocs.add(allocator);
- // Check correctly synchronized creation
- assert allocator == getBlock(latchedAddr);
+ // Check correctly synchronized creation
+ assert allocator == getBlock(latchedAddr);
- alloc = allocator;
- }
+ alloc = allocator;
+ }
- assert size2 <= alloc.getSlotSize();
+ assert size2 <= alloc.getSlotSize();
- if (size > 0) {
+ if (size > 0) {
- /*
- * This is a real allocation.
- */
+ /*
+ * This is a real allocation.
+ */
- alloc.setAddressExternal(latchedAddr);
+ alloc.setAddressExternal(latchedAddr);
- }
+ }
- } finally {
+ } finally {
- m_allocationWriteLock.unlock();
+ m_allocationWriteLock.unlock();
- }
- }
+ }
+ }
/**
* Called from WriteCache.resetRecordMapFromBuffer
@@ -962,40 +992,40 @@
*
* @param latchedAddr
*/
- void removeAddress(final int latchedAddr) {
- // ignore zero address
- if (latchedAddr == 0)
- return;
+ void removeAddress(final int latchedAddr) {
+ // ignore zero address
+ if (latchedAddr == 0)
+ return;
- m_allocationWriteLock.lock();
- try {
- // assert m_commitList.size() == 0;
+ m_allocationWriteLock.lock();
+ try {
+ // assert m_commitList.size() == 0;
- final FixedAllocator alloc = getBlockByAddress(latchedAddr);
+ final FixedAllocator alloc = getBlockByAddress(latchedAddr);
- assert alloc != null;
+ assert alloc != null;
- final int addrOffset = getOffset(latchedAddr);
- if (alloc == null) {
- throw new IllegalArgumentException(
- "Invalid address provided to immediateFree: "
- + latchedAddr);
- }
- final long pa = alloc.getPhysicalAddress(addrOffset);
+ final int addrOffset = getOffset(latchedAddr);
+ if (alloc == null) {
+ throw new IllegalArgumentException(
+ "Invalid address provided to immediateFree: "
+ + latchedAddr);
+ }
+ final long pa = alloc.getPhysicalAddress(addrOffset);
- if (log.isTraceEnabled())
- log.trace("Freeing allocation at " + latchedAddr
- + ", physical address: " + pa);
+ if (log.isTraceEnabled())
+ log.trace("Freeing allocation at " + latchedAddr
+ + ", physical address: " + pa);
- alloc.free(latchedAddr, 0, false);
+ alloc.free(latchedAddr, 0, false);
- // assert m_commitList.size() == 0;
- } finally {
- m_allocationWriteLock.unlock();
- }
- }
-
- /**
+ // assert m_commitList.size() == 0;
+ } finally {
+ m_allocationWriteLock.unlock();
+ }
+ }
+
+ /**
* Create and return a new {@link RWWriteCacheService} instance. The caller
* is responsible for closing out the old one and must be holding the
* appropriate locks when it switches in the new instance.
@@ -1033,7 +1063,7 @@
}
}
- private void setAllocations(final FileMetadata fileMetadata)
+ private void setAllocations(final FileMetadata fileMetadata)
throws IOException {
final String buckets = fileMetadata.getProperty(
@@ -1052,30 +1082,30 @@
}
private void defaultInit() throws IOException {
- final int numFixed = m_allocSizes.length;
+ final int numFixed = m_allocSizes.length;
- m_freeFixed = new ArrayList[numFixed];
+ m_freeFixed = new ArrayList[numFixed];
- for (int i = 0; i < numFixed; i++) {
- m_freeFixed[i] = new ArrayList<FixedAllocator>();
- }
+ for (int i = 0; i < numFixed; i++) {
+ m_freeFixed[i] = new ArrayList<FixedAllocator>();
+ }
- m_fileSize = convertFromAddr(m_fd.length());
-
- // make space for meta-allocators
- m_metaBits[0] = -1;
- m_metaTransientBits[0] = -1;
- m_nextAllocation = -(1 + META_ALLOCATION); // keep on a minimum 8K boundary
- m_committedNextAllocation = m_nextAllocation;
-
- if (m_fileSize > m_nextAllocation) {
- m_fileSize = m_nextAllocation;
- }
-
- if (log.isInfoEnabled())
- log.info("Set default file extent " + convertAddr(m_fileSize));
-
- m_reopener.raf.setLength(convertAddr(m_fileSize));
+ m_fileSize = convertFromAddr(m_fd.length());
+
+ // make space for meta-allocators
+ m_metaBits[0] = -1;
+ m_metaTransientBits[0] = -1;
+ m_nextAllocation = -(1 + META_ALLOCATION); // keep on a minimum 8K boundary
+ m_committedNextAllocation = m_nextAllocation;
+
+ if (m_fileSize > m_nextAllocation) {
+ m_fileSize = m_nextAllocation;
+ }
+
+ if (log.isInfoEnabled())
+ log.info("Set default file extent " + convertAddr(m_fileSize));
+
+ m_reopener.raf.setLength(convertAddr(m_fileSize));
}
@@ -1104,71 +1134,71 @@
}
}
- /**
- * Basic check on key root block validity
- *
- * @param rbv
- */
- private void checkRootBlock(final IRootBlockView rbv) {
- final long nxtOffset = rbv.getNextOffset();
- final int nxtalloc = -(int) (nxtOffset >> 32);
+ /**
+ * Basic check on key root block validity
+ *
+ * @param rbv
+ */
+ private void checkRootBlock(final IRootBlockView rbv) {
+ final long nxtOffset = rbv.getNextOffset();
+ final int nxtalloc = -(int) (nxtOffset >> 32);
- final int metaBitsAddr = -(int) nxtOffset;
+ final int metaBitsAddr = -(int) nxtOffset;
- final long metaAddr = rbv.getMetaStartAddr();
- final long rawMetaBitsAddr = rbv.getMetaBitsAddr();
- if (metaAddr == 0 || rawMetaBitsAddr == 0) {
+ final long metaAddr = rbv.getMetaStartAddr();
+ final long rawMetaBitsAddr = rbv.getMetaBitsAddr();
+ if (metaAddr == 0 || rawMetaBitsAddr == 0) {
/*
* possible when rolling back to empty file.
*/
- log.warn("No meta allocation data included in root block for RWStore");
- }
-
- if (log.isTraceEnabled()) {
+ log.warn("No meta allocation data included in root block for RWStore");
+ }
+
+ if (log.isTraceEnabled()) {
final int commitRecordAddr = (int) (rbv.getCommitRecordAddr() >> 32);
log.trace("CommitRecord " + rbv.getCommitRecordAddr()
+ " at physical address: "
+ physicalAddress(commitRecordAddr));
}
-
- final long commitCounter = rbv.getCommitCounter();
+
+ final long commitCounter = rbv.getCommitCounter();
-// final int metaStartAddr = (int) -(metaAddr >> 32); // void
-// final int fileSize = (int) -(metaAddr & 0xFFFFFFFF);
+// final int metaStartAddr = (int) -(metaAddr >> 32); // void
+// final int fileSize = (int) -(metaAddr & 0xFFFFFFFF);
- if (log.isTraceEnabled())
- log.trace("m_allocation: " + nxtalloc + ", m_metaBitsAddr: "
- + metaBitsAddr + ", m_commitCounter: " + commitCounter);
-
- }
-
- /*
- * Utility to encapsulate RootBlock interpreation
- */
- static class RootBlockInfo {
-
- static int nextAllocation(final IRootBlockView rb) {
- final long nxtOffset = rb.getNextOffset();
+ if (log.isTraceEnabled())
+ log.trace("m_allocation: " + nxtalloc + ", m_metaBitsAddr: "
+ + metaBitsAddr + ", m_commitCounter: " + commitCounter);
+
+ }
+
+ /*
+ * Utility to encapsulate RootBlock interpreation
+ */
+ static class RootBlockInfo {
+
+ static int nextAllocation(final IRootBlockView rb) {
+ final long nxtOffset = rb.getNextOffset();
- // next allocation to be made (in -32K units).
- final int ret = -(int) (nxtOffset >> 32);
-
+ // next allocation to be made (in -32K units).
+ final int ret = -(int) (nxtOffset >> 32);
+
/*
* Skip the first 32K in the file. The root blocks live here but
* nothing else.
*/
- return ret == 0 ? -(1 + META_ALLOCATION) : ret;
- }
-
- /*
- * Meta-Allocations stored as {int address; int[8] bits}, so each block
- * holds 8*32=256 allocation slots of 1K totaling 256K.
- *
- * The returned int array is a flattened list of these int[9] blocks
- */
- static int[] metabits(final IRootBlockView rb, final ReopenFileChannel reopener) throws IOException {
- final long rawmbaddr = rb.getMetaBitsAddr();
-
+ return ret == 0 ? -(1 + META_ALLOCATION) : ret;
+ }
+
+ /*
+ * Meta-Allocations stored as {int address; int[8] bits}, so each block
+ * holds 8*32=256 allocation slots of 1K totaling 256K.
+ *
+ * The returned int array is a flattened list of these int[9] blocks
+ */
+ static int[] metabits(final IRootBlockView rb, final ReopenFileChannel reopener) throws IOException {
+ final long rawmbaddr = rb.getMetaBitsAddr();
+
/*
* The #of int32 values in the metabits region.
*
@@ -1176,86 +1206,86 @@
* gives the #of int32 values in the metabits regions (up to 64k
* int32 values).
*/
- final int metaBitsStore = (int) (rawmbaddr & 0xFFFF);
-
-
- // The byte offset of the metabits region in the file.
- final long pmaddr = rawmbaddr >> 16;
-
+ final int metaBitsStore = (int) (rawmbaddr & 0xFFFF);
+
+
+ // The byte offset of the metabits region in the file.
+ final long pmaddr = rawmbaddr >> 16;
+
/*
* Read the metabits block, including a header and the int32[]
* that encodes both startAddrs and bit vectors.
*/
final byte[] buf = new byte[metaBitsStore * 4];
- FileChannelUtility.readAll(reopener, ByteBuffer.wrap(buf), pmaddr);
-
- final DataInputStream strBuf = new DataInputStream(new ByteArrayInputStream(buf));
-
- // Can handle minor store version incompatibility
- strBuf.readInt(); // STORE VERSION
- strBuf.readLong(); // Last Deferred Release Time
- strBuf.readInt(); // cDefaultMetaBitsSize
-
- final int allocBlocks = strBuf.readInt();
- strBuf.readLong(); // m_storageStatsAddr
+ FileChannelUtility.readAll(reopener, ByteBuffer.wrap(buf), pmaddr);
+
+ final DataInputStream strBuf = new DataInputStream(new ByteArrayInputStream(buf));
+
+ // Can handle minor store version incompatibility
+ strBuf.readInt(); // STORE VERSION
+ strBuf.readLong(); // Last Deferred Release Time
+ strBuf.readInt(); // cDefaultMetaBitsSize
+
+ final int allocBlocks = strBuf.readInt();
+ strBuf.readLong(); // m_storageStatsAddr
- // step over those reserved ints
+ // step over those reserved ints
for (int i = 0; i < cReservedMetaBits; i++) {
strBuf.readInt();
}
// step over the allocSizes
- for (int i = 0; i < allocBlocks; i++) {
- strBuf.readInt();
- }
- final int metaBitsSize = metaBitsStore - allocBlocks - cMetaHdrFields; // allow for header fields
-
- // Must be multiple of 9
- assert metaBitsSize % 9 == 0;
-
- int[] ret = new int[metaBitsSize];
- for (int i = 0; i < metaBitsSize; i++) {
- ret[i] = strBuf.readInt();
- }
+ for (int i = 0; i < allocBlocks; i++) {
+ strBuf.readInt();
+ }
+ final int metaBitsSize = metaBitsStore - allocBlocks - cMetaHdrFields; // allow for header fields
+
+ // Must be multiple of 9
+ assert metaBitsSize % 9 == 0;
+
+ int[] ret = new int[metaBitsSize];
+ for (int i = 0; i < metaBitsSize; i++) {
+ ret[i] = strBuf.readInt();
+ }
- /*
- * Meta-Allocations stored as {int address; int[8] bits}, so each block
- * holds 8*32=256 allocation slots of 1K totaling 256K.
- */
- return ret;
- }
- }
-
- /**
- * Should be called where previously initFileSpec was used.
- *
- * Rather than reading from file, instead reads from the current root block.
- *
- * We use the rootBlock fields, nextOffset, metaStartAddr, metaBitsAddr.
- *
- * metaBitsAddr indicates where the meta allocation bits are.
- *
- * metaStartAddr is the offset in the file where the allocation blocks are
- * allocated the long value also indicates the size of the allocation, such
- * that the address plus the size is the "filesize".
- *
- * Note that metaBitsAddr must be an absolute address, with the low order 16
- * bits used to indicate the size.
- *
- * @throws IOException
- */
- private void initfromRootBlock(final IRootBlockView rb) throws IOException {
- // m_rb = m_fmv.getRootBlock();
- assert(rb != null);
+ /*
+ * Meta-Allocations stored as {int address; int[8] bits}, so each block
+ * holds 8*32=256 allocation slots of 1K totaling 256K.
+ */
+ return ret;
+ }
+ }
+
+ /**
+ * Should be called where previously initFileSpec was used.
+ *
+ * Rather than reading from file, instead reads from the current root block.
+ *
+ * We use the rootBlock fields, nextOffset, metaStartAddr, metaBitsAddr.
+ *
+ * metaBitsAddr indicates where the meta allocation bits are.
+ *
+ * metaStartAddr is the offset in the file where the allocation blocks are
+ * allocated the long value also indicates the size of the allocation, such
+ * that the address plus the size is the "filesize".
+ *
+ * Note that metaBitsAddr must be an absolute address, with the low order 16
+ * bits used to indicate the size.
+ *
+ * @throws IOException
+ */
+ private void initfromRootBlock(final IRootBlockView rb) throws IOException {
+ // m_rb = m_fmv.getRootBlock();
+ assert(rb != null);
- m_storeUUID = rb.getUUID();
-
- if (rb.getNextOffset() == 0) {
+ m_storeUUID = rb.getUUID();
+
+ if (rb.getNextOffset() == 0) {
- defaultInit();
-
- } else {
+ defaultInit();
+
+ } else {
/*
* The RWStore stores in IRootBlock.getNextOffset() two distinct
@@ -1271,52 +1301,52 @@
* FixedAllocators in order to turn it into a byte offset on the
* file.
*/
- final long nxtOffset = rb.getNextOffset();
+ final long nxtOffset = rb.getNextOffset();
- // next allocation to be made (in -32K units).
- m_nextAllocation = -(int) (nxtOffset >> 32);
-
- if (m_nextAllocation == 0) {
+ // next allocation to be made (in -32K units).
+ m_nextAllocation = -(int) (nxtOffset >> 32);
+
+ if (m_nextAllocation == 0) {
/*
* Skip the first 32K in the file. The root blocks live here but
* nothing else.
*/
-
- m_nextAllocation = -(1 + META_ALLOCATION);
-
- }
-
- m_committedNextAllocation = m_nextAllocation;
-
- // latched offset of the metabits region.
- m_metaBitsAddr = -(int) nxtOffset;
-
- if (log.isInfoEnabled()) {
- log.info("MetaBitsAddr: " + m_metaBitsAddr);
- }
+
+ m_nextAllocation = -(1 + META_ALLOCATION);
+
+ }
+
+ ...
[truncated message content] |
|
From: <tho...@us...> - 2013-05-03 12:11:55
|
Revision: 7099
http://bigdata.svn.sourceforge.net/bigdata/?rev=7099&view=rev
Author: thompsonbry
Date: 2013-05-03 12:11:49 +0000 (Fri, 03 May 2013)
Log Message:
-----------
javadoc on the extensionLock in the WORMStrategy.
Modified Paths:
--------------
branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/WORMStrategy.java
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/WORMStrategy.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/WORMStrategy.java 2013-05-02 20:56:56 UTC (rev 7098)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/WORMStrategy.java 2013-05-03 12:11:49 UTC (rev 7099)
@@ -39,6 +39,8 @@
import java.util.concurrent.atomic.AtomicReference;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock.ReadLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock.WriteLock;
import com.bigdata.LRUNexus;
import com.bigdata.btree.BTree.Counter;
@@ -192,9 +194,11 @@
// private final AtomicReference<Quorum<?,?>> quorumRef;
/**
- * This lock is used to exclude readers when the extent of the backing file
- * is about to be changed. This is a workaround for an old (an unresolved as
- * of February 2010) Sun bug.
+ * This lock is used to exclude readers/writers performing IOs against the
+ * backing file when the extent of the backing file is about to be changed.
+ * Readers and writers take the {@link ReadLock}. The {@link WriteLock} is
+ * taken when the file extent must be changed. This is a workaround for an
+ * old (an unresolved as of February 2010) Sun bug.
*
* @see http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6371642
*/
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mrp...@us...> - 2013-05-02 20:57:03
|
Revision: 7098
http://bigdata.svn.sourceforge.net/bigdata/?rev=7098&view=rev
Author: mrpersonick
Date: 2013-05-02 20:56:56 +0000 (Thu, 02 May 2013)
Log Message:
-----------
fixed ticket 669 - sparql11subquery optimizer was not recursing into nested sparql 11 subqueries, only into nested GraphPatternGroups.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTSparql11SubqueryOptimizer.java
branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithQuads.java
branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithSids.java
branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithoutSids.java
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/test/com/bigdata/rdf/sail/TestTicket669.java
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTSparql11SubqueryOptimizer.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTSparql11SubqueryOptimizer.java 2013-05-02 10:53:43 UTC (rev 7097)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTSparql11SubqueryOptimizer.java 2013-05-02 20:56:56 UTC (rev 7098)
@@ -165,7 +165,16 @@
liftSubqueries(context, sa,
((GraphPatternGroup<IGroupMemberNode>) child));
- } else if (child instanceof ServiceNode) {
+ } else if (child instanceof SubqueryRoot) {
+
+ // Recursion into subqueries.
+
+ final SubqueryRoot subqueryRoot = (SubqueryRoot) child;
+
+ liftSubqueries(context, sa,
+ subqueryRoot.getWhereClause());
+
+ }else if (child instanceof ServiceNode) {
// Do not rewrite things inside of a SERVICE node.
continue;
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithQuads.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithQuads.java 2013-05-02 10:53:43 UTC (rev 7097)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithQuads.java 2013-05-02 20:56:56 UTC (rev 7098)
@@ -130,6 +130,7 @@
suite.addTestSuite(com.bigdata.rdf.sail.TestLexJoinOps.class);
suite.addTestSuite(com.bigdata.rdf.sail.TestMaterialization.class);
suite.addTestSuite(com.bigdata.rdf.sail.TestTicket632.class);
+ suite.addTestSuite(com.bigdata.rdf.sail.TestTicket669.class);
// The Sesame TCK, including the SPARQL test suite.
{
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithSids.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithSids.java 2013-05-02 10:53:43 UTC (rev 7097)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithSids.java 2013-05-02 20:56:56 UTC (rev 7098)
@@ -108,6 +108,7 @@
suite.addTestSuite(com.bigdata.rdf.sail.TestMaterialization.class);
suite.addTestSuite(com.bigdata.rdf.sail.TestTicket610.class);
+ suite.addTestSuite(com.bigdata.rdf.sail.TestTicket669.class);
return suite;
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithoutSids.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithoutSids.java 2013-05-02 10:53:43 UTC (rev 7097)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithoutSids.java 2013-05-02 20:56:56 UTC (rev 7098)
@@ -103,6 +103,7 @@
suite.addTestSuite(com.bigdata.rdf.sail.TestMaterialization.class);
suite.addTestSuite(com.bigdata.rdf.sail.TestTicket610.class);
+ suite.addTestSuite(com.bigdata.rdf.sail.TestTicket669.class);
return suite;
Added: branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/test/com/bigdata/rdf/sail/TestTicket669.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/test/com/bigdata/rdf/sail/TestTicket669.java (rev 0)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/test/com/bigdata/rdf/sail/TestTicket669.java 2013-05-02 20:56:56 UTC (rev 7098)
@@ -0,0 +1,273 @@
+/**
+Copyright (C) SYSTAP, LLC 2011. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+
+package com.bigdata.rdf.sail;
+
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.Properties;
+
+import org.apache.log4j.Logger;
+import org.openrdf.model.Graph;
+import org.openrdf.model.URI;
+import org.openrdf.model.impl.GraphImpl;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.repository.Repository;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.sail.SailRepository;
+import org.openrdf.repository.sail.SailTupleQuery;
+import org.openrdf.sail.Sail;
+import org.openrdf.sail.memory.MemoryStore;
+
+import com.bigdata.rdf.axioms.NoAxioms;
+import com.bigdata.rdf.vocab.NoVocabulary;
+
+/**
+ * Unit test template for use in submission of bugs.
+ * <p>
+ * This test case will delegate to an underlying backing store. You can
+ * specify this store via a JVM property as follows:
+ * <code>-DtestClass=com.bigdata.rdf.sail.TestBigdataSailWithQuads</code>
+ * <p>
+ * There are three possible configurations for the testClass:
+ * <ul>
+ * <li>com.bigdata.rdf.sail.TestBigdataSailWithQuads (quads mode)</li>
+ * <li>com.bigdata.rdf.sail.TestBigdataSailWithoutSids (triples mode)</li>
+ * <li>com.bigdata.rdf.sail.TestBigdataSailWithSids (SIDs mode)</li>
+ * </ul>
+ * <p>
+ * The default for triples and SIDs mode is for inference with truth maintenance
+ * to be on. If you would like to turn off inference, make sure to do so in
+ * {@link #getProperties()}.
+ *
+ * @author <a href="mailto:mrp...@us...">Mike Personick</a>
+ * @version $Id$
+ */
+public class TestTicket669 extends QuadsTestCase {
+
+ protected static final Logger log = Logger.getLogger(TestTicket669.class);
+
+ /**
+ * Please set your database properties here, except for your journal file,
+ * please DO NOT SPECIFY A JOURNAL FILE.
+ */
+ @Override
+ public Properties getProperties() {
+
+ Properties props = super.getProperties();
+
+ /*
+ * For example, here is a set of five properties that turns off
+ * inference, truth maintenance, and the free text index.
+ */
+ props.setProperty(BigdataSail.Options.AXIOMS_CLASS, NoAxioms.class.getName());
+ props.setProperty(BigdataSail.Options.VOCABULARY_CLASS, NoVocabulary.class.getName());
+ props.setProperty(BigdataSail.Options.TRUTH_MAINTENANCE, "false");
+ props.setProperty(BigdataSail.Options.JUSTIFY, "false");
+ props.setProperty(BigdataSail.Options.TEXT_INDEX, "false");
+
+ return props;
+
+ }
+
+ public TestTicket669() {
+ }
+
+ public TestTicket669(String arg0) {
+ super(arg0);
+ }
+
+ public void testBug() throws Exception {
+
+ /*
+ * We use an in-memory Sesame store as our point of reference. This
+ * will supply the "correct" answer to the query (below).
+ */
+ final Sail sesameSail = new MemoryStore();
+
+ /*
+ * The bigdata store, backed by a temporary journal file.
+ */
+ final BigdataSail bigdataSail = getSail();
+
+ /*
+ * Data file containing the data demonstrating your bug.
+ */
+// final String data = "data.ttl";
+// final String baseURI = "";
+// final RDFFormat format = RDFFormat.TURTLE;
+
+ final String update =
+ "INSERT DATA " +
+ "{ " +
+ "<http://example.com/book1> a <http://example.com/Book> . " +
+ "<http://example.com/book2> a <http://example.com/Book> . " +
+ "<http://example.com/book3> a <http://example.com/Book> . " +
+ "}";
+
+ /*
+ * Query(ies) demonstrating your bug.
+ */
+ final String nested =
+ "SELECT ?s WHERE { " +
+ " SELECT ?s WHERE { ?s ?p ?o} LIMIT 1 " +
+ "}";
+
+ final String doubleNested =
+ "SELECT ?s WHERE { " +
+ " SELECT ?s WHERE { " +
+ " SELECT ?s WHERE { ?s ?p ?o} LIMIT 1 " +
+ " } " +
+ "}";
+
+ final String query = doubleNested;
+
+ try {
+
+ sesameSail.initialize();
+ bigdataSail.initialize();
+
+ final Repository sesameRepo = new SailRepository(sesameSail);
+ final BigdataSailRepository bigdataRepo = new BigdataSailRepository(bigdataSail);
+
+ final URI book1 = new URIImpl("http://example.com/book1");
+ final URI book2 = new URIImpl("http://example.com/book2");
+ final URI book3 = new URIImpl("http://example.com/book3");
+ final URI book = new URIImpl("http://example.com/book");
+ final Graph data = new GraphImpl();
+ data.add(book1, RDF.TYPE, book);
+ data.add(book2, RDF.TYPE, book);
+ data.add(book3, RDF.TYPE, book);
+
+ { // load the data into the Sesame store
+
+ final RepositoryConnection cxn = sesameRepo.getConnection();
+ try {
+ cxn.setAutoCommit(false);
+// cxn.add(getClass().getResourceAsStream(data), baseURI, format);
+ cxn.add(data);
+ cxn.commit();
+ } finally {
+ cxn.close();
+ }
+
+ }
+
+ { // load the data into the bigdata store
+
+ final RepositoryConnection cxn = bigdataRepo.getConnection();
+ try {
+ cxn.setAutoCommit(false);
+// cxn.add(getClass().getResourceAsStream(data), baseURI, format);
+ cxn.add(data);
+ cxn.commit();
+ } finally {
+ cxn.close();
+ }
+
+ }
+
+ final Collection<BindingSet> answer = new LinkedList<BindingSet>();
+
+ /*
+ * Here is how you manually build the answer set, but please make
+ * sure you answer truly is correct if you choose to do it this way.
+ */
+
+// answer.add(createBindingSet(
+// new BindingImpl("neType", vf.createURI("http://example/class/Location"))
+// ));
+// answer.add(createBindingSet(
+// new BindingImpl("neType", vf.createURI("http://example/class/Person"))
+// ));
+
+ /*
+ * Run the problem query using the Sesame store to gather the
+ * correct results.
+ */
+ {
+ final RepositoryConnection cxn = sesameRepo.getConnection();
+ try {
+ final SailTupleQuery tupleQuery = (SailTupleQuery)
+ cxn.prepareTupleQuery(QueryLanguage.SPARQL, query);
+ tupleQuery.setIncludeInferred(false /* includeInferred */);
+ final TupleQueryResult result = tupleQuery.evaluate();
+
+ if (log.isInfoEnabled()) {
+ log.info("sesame results:");
+ if (!result.hasNext()) {
+ log.info("no results.");
+ }
+ }
+
+ while (result.hasNext()) {
+ final BindingSet bs = result.next();
+ answer.add(bs);
+ if (log.isInfoEnabled())
+ log.info(bs);
+ }
+ } finally {
+ cxn.close();
+ }
+ }
+
+ /*
+ * Run the problem query using the bigdata store and then compare
+ * the answer.
+ */
+ final RepositoryConnection cxn = bigdataRepo.getReadOnlyConnection();
+ try {
+ final SailTupleQuery tupleQuery = (SailTupleQuery)
+ cxn.prepareTupleQuery(QueryLanguage.SPARQL, query);
+ tupleQuery.setIncludeInferred(false /* includeInferred */);
+
+ if (log.isInfoEnabled()) {
+ final TupleQueryResult result = tupleQuery.evaluate();
+ log.info("bigdata results:");
+ if (!result.hasNext()) {
+ log.info("no results.");
+ }
+ while (result.hasNext()) {
+ log.info(result.next());
+ }
+ }
+
+ final TupleQueryResult result = tupleQuery.evaluate();
+ compare(result, answer);
+
+ } finally {
+ cxn.close();
+ }
+
+ } finally {
+ bigdataSail.__tearDownUnitTest();
+ sesameSail.shutDown();
+ }
+
+ }
+
+}
Property changes on: branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/test/com/bigdata/rdf/sail/TestTicket669.java
___________________________________________________________________
Added: svn:mime-type
+ text/plain
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-05-02 10:53:50
|
Revision: 7097
http://bigdata.svn.sourceforge.net/bigdata/?rev=7097&view=rev
Author: thompsonbry
Date: 2013-05-02 10:53:43 +0000 (Thu, 02 May 2013)
Log Message:
-----------
Corrected timeout unit from MINUTES to MILLISECONDS.
Modified Paths:
--------------
branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA3JournalServer.java
Modified: branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA3JournalServer.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA3JournalServer.java 2013-05-01 20:41:23 UTC (rev 7096)
+++ branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA3JournalServer.java 2013-05-02 10:53:43 UTC (rev 7097)
@@ -1715,7 +1715,7 @@
executorService.submit(queryTaskFuture);
// Wait for QueryTask : ends once UPDATE task is done.
- queryTaskFuture.get(timeout, TimeUnit.MINUTES);
+ queryTaskFuture.get(timeout, TimeUnit.MILLISECONDS);
// Verify update task is done.
assertTrue(updateTaskFuture.isDone());
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-05-01 20:41:30
|
Revision: 7096
http://bigdata.svn.sourceforge.net/bigdata/?rev=7096&view=rev
Author: thompsonbry
Date: 2013-05-01 20:41:23 +0000 (Wed, 01 May 2013)
Log Message:
-----------
A problem has been identified in the HA replication cluster where a query on the followers with concurrent update(s) on the leader can result in a stack trace such as the following. I have modified the HA CI unit test TestHA3JournalServer.testABCMultiTransactionFollowerReads() such that we can now replicate this problem. The likely cause is a failure in the release time consensus protocol resulting in the following losing its protection of the commit point against which the query is reading.
This commit modifies the test so that it now fails.
See https://sourceforge.net/apps/trac/bigdata/ticket/530 (Journal HA)
Modified Paths:
--------------
branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/AbstractHA3JournalServerTestCase.java
branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA3JournalServer.java
Modified: branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/AbstractHA3JournalServerTestCase.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/AbstractHA3JournalServerTestCase.java 2013-04-30 17:44:39 UTC (rev 7095)
+++ branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/AbstractHA3JournalServerTestCase.java 2013-05-01 20:41:23 UTC (rev 7096)
@@ -2201,6 +2201,9 @@
// Verify quorum is still valid.
quorum.assertQuorum(token);
+ if(log.isInfoEnabled())
+ log.info("Done with load.");
+
// Done.
return null;
Modified: branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA3JournalServer.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA3JournalServer.java 2013-04-30 17:44:39 UTC (rev 7095)
+++ branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA3JournalServer.java 2013-05-01 20:41:23 UTC (rev 7096)
@@ -27,6 +27,7 @@
package com.bigdata.journal.jini.ha;
import java.io.File;
+import java.io.IOException;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.UUID;
@@ -1493,111 +1494,247 @@
}
public void testABCMultiTransactionFollowerReads() throws Exception {
- // doABCMultiTransactionFollowerReads(2000/*nTransactions*/, 20/*delay per transaction*/); // STRESS
- doABCMultiTransactionFollowerReads(200/*nTransactions*/, 20/*delay per transaction*/);
+ // doABCMultiTransactionFollowerReads(2000/*nTransactions*/, 20/*delay per transaction*/); // STRESS
+ doABCMultiTransactionFollowerReads(200/*nTransactions*/, 20/*delay per transaction*/);
}
/**
- * Tests multiple concurrent reads on followers in presence of multiple updates.
- * @throws Exception
+ * Tests multiple concurrent reads on followers in presence of multiple
+ * updates.
+ *
+ * @param nTransactions
+ * The #of UPDATE transactions to execute.
+ * @param transactionDelay
+ * The delay between the UPDATE transactions.
*/
- protected void doABCMultiTransactionFollowerReads(final int nTransactions, final long transactionDelay) throws Exception {
+ protected void doABCMultiTransactionFollowerReads(final int nTransactions,
+ final long transactionDelay) throws Exception {
final long timeout = TimeUnit.MINUTES.toMillis(4);
try {
-
- // Start all services.
+
+ // Start all services.
final ABC services = new ABC(true/* sequential */);
+ // Wait for a quorum meet.
+ final long token = quorum.awaitQuorum(awaitQuorumTimeout,
+ TimeUnit.MILLISECONDS);
- // Wait for a quorum meet.
- final long token = quorum.awaitQuorum(awaitQuorumTimeout,
- TimeUnit.MILLISECONDS);
+ assertEquals(token, awaitFullyMetQuorum());
- awaitFullyMetQuorum();
-
final HAGlue leader = quorum.getClient().getLeader(token);
// Verify assumption in this test.
assertEquals(leader, services.serverA);
- // Wait until leader is ready.
+ // Wait until all services are "HA" ready.
leader.awaitHAReady(awaitQuorumTimeout, TimeUnit.MILLISECONDS);
+ services.serverB.awaitHAReady(awaitQuorumTimeout, TimeUnit.MILLISECONDS);
+ services.serverC.awaitHAReady(awaitQuorumTimeout, TimeUnit.MILLISECONDS);
+
+ /*
+ * Start a long running long. This gives us enough data to make the
+ * query we run have a little latency. We need that query to cross
+ * a commit boundary when it is executed.
+ */
+ {
+
+ final FutureTask<Void> ft = new FutureTask<Void>(
+ new LargeLoadTask(token, true/* reallyLargeLoad */));
+
+ try {
+
+ // Start the LOAD task.
+ executorService.submit(ft);
+
+ // Wait for the LOAD task.
+ ft.get(timeout, TimeUnit.MILLISECONDS);
+
+ } finally {
+
+ ft.cancel(true/* mayInterruptIfRunning */);
+
+ }
+
+ }
- // start concurrent task to load for specified transactions
- final Callable<Void> task = new Callable<Void>() {
- public Void call() throws Exception {
- for (int n = 0; n < nTransactions; n++) {
+ /**
+ * Class will issue a series of UPDATE requests.
+ * <p>
+ * Note: We need to actually modify the KB for each UPDATE but not
+ * drop all the statemets each time.
+ */
+ class UpdateTask implements Callable<Void> {
- final StringBuilder sb = new StringBuilder();
- sb.append("PREFIX dc: <http://purl.org/dc/elements/1.1/>\n");
- sb.append("INSERT DATA {\n");
- sb.append(" <http://example/book" + n
- + "> dc:title \"A new book\" ;\n");
- sb.append(" dc:creator \"A.N.Other\" .\n");
- sb.append("}\n");
+ private final RemoteRepository remoteRepo;
- final String updateStr = sb.toString();
+ public UpdateTask() throws IOException {
- final HAGlue leader = quorum.getClient().getLeader(
- token);
+ remoteRepo = getRemoteRepository(leader);
- // Verify quorum is still valid.
- quorum.assertQuorum(token);
+ }
- getRemoteRepository(leader).prepareUpdate(
- updateStr).evaluate();
- log.warn("COMPLETED TRANSACTION " + n);
+ public Void call() throws Exception {
+
+ for (int n = 0; n < nTransactions; n++) {
- Thread.sleep(transactionDelay);
- }
- // done.
- return null;
- }
- };
- final FutureTask<Void> load = new FutureTask<Void>(task);
+ // Verify quorum is still valid.
+ quorum.assertQuorum(token);
- executorService.submit(load);
-
- // Now create a Callable for the final followes to repeatedly query against the current commit point
- final Callable<Void> query = new Callable<Void>() {
- public Void call() throws Exception {
- int queryCount = 0;
- SimpleDateFormat df = new SimpleDateFormat("hh:mm:ss,SSS");
- while (!load.isDone()) {
+ // Do a simple UPDATE transaction.
+ final StringBuilder sb = new StringBuilder();
+ sb.append("PREFIX dc: <http://purl.org/dc/elements/1.1/>\n");
+ sb.append("INSERT DATA {\n");
+ sb.append(" <http://example/book" + n // Note: distinct triple!
+ + "> dc:title \"A new book\" ;\n");
+ sb.append(" dc:creator \"A.N.Other\" .\n");
+ sb.append("}\n");
- final StringBuilder sb = new StringBuilder();
- sb.append("SELECT (COUNT(*) AS ?count) WHERE { ?s ?p ?o }\n");
+ final String updateStr = sb.toString();
- final String query = sb.toString();
+ remoteRepo.prepareUpdate(updateStr).evaluate();
+
+ if(log.isInfoEnabled())
+ log.info("COMPLETED TRANSACTION " + n);
- // final RemoteRepository follower = getRemoteRepository(services.serverA); // try with Leader to see difference! 6537 queries (less than for follower)
- final RemoteRepository follower = getRemoteRepository(services.serverC); // 10109 queries for 2000 transact ons
+ Thread.sleep(transactionDelay);
- // Verify quorum is still valid.
- quorum.assertQuorum(token);
+ }
+
+ // done.
+ return null;
+
+ }
- follower.prepareTupleQuery(query).evaluate();
-
- // add date time format to support comparison with HA logs
- log.info(df.format(new Date()) + " - completed query: " + ++queryCount);
- }
- // done.
- return null;
- }
- };
-
- final FutureTask<Void> queries = new FutureTask<Void>(query);
-
- executorService.submit(queries);
-
- // Now wait for query completion!
- queries.get();
-
- assertTrue(load.isDone());
-
- } finally {
- destroyAll();
+ };
+
+ /*
+ * Future for the task executing a series of UPDATES.
+ *
+ * Note: The task is not running yet, but we will pass the Future
+ * into the QueryTask so it can monitor the UpdateTask.
+ */
+ final FutureTask<Void> updateTaskFuture = new FutureTask<Void>(new UpdateTask());
+
+ /*
+ * Now create a Callable for the final followes to repeatedly query
+ * against the then current commit point. The task returns the #of
+ * queries that were executed. The task will run until we stop
+ * issuing UPDATE requests.
+ */
+ class QueryTask implements Callable<Long> {
+
+// /** The service to query. */
+// final HAGlue haGlue;
+
+ /**
+ * The SPARQL end point for that service.
+ */
+ final RemoteRepository remoteRepo;
+
+ /**
+ * Format for timestamps that may be used to correlate with the
+ * HA log messages.
+ */
+ final SimpleDateFormat df = new SimpleDateFormat("hh:mm:ss,SSS");
+
+ /**
+ * @param haGlue
+ * The service to query.
+ *
+ * @throws IOException
+ */
+ public QueryTask(final HAGlue haGlue) throws IOException {
+
+// this.haGlue = haGlue;
+
+ /*
+ * Run query against one of the followers.
+ *
+ * 6537 queries for 2000 transactions (leader)
+ *
+ * 10109 queries for 2000 transactions (follower)
+ */
+ remoteRepo = getRemoteRepository(haGlue);
+
+ }
+
+ public Long call() throws Exception {
+
+ long queryCount = 0;
+
+ while (!updateTaskFuture.isDone()) {
+
+ final String query;
+ if (queryCount % 2 == 0) {
+ // Shorter query. Counts all tuples.
+ query = "SELECT (COUNT(*) AS ?count) WHERE { ?s ?p ?o }";
+ } else {
+ // Longer query. Materializies up to N tuples.
+ query = "SELECT * WHERE { ?s ?p ?o } LIMIT 100000";
+ }
+
+ // Verify quorum is still valid.
+ quorum.assertQuorum(token);
+
+ // Run query.
+ final long nresults = countResults(remoteRepo
+ .prepareTupleQuery(query).evaluate());
+
+ queryCount++;
+
+ // add date time format for comparison with HA logs
+ if (log.isInfoEnabled())
+ log.info(df.format(new Date())
+ + " - completed query: " + queryCount
+ + ", nresults=" + nresults);
+
+ }
+
+ // done.
+ return queryCount;
+
+ }
+
+ };
+
+ final FutureTask<Long> queryTaskFuture = new FutureTask<Long>(
+ new QueryTask(services.serverC));
+
+ /*
+ * Start the tasks to run the UPDATES (against the leader) and the
+ * QUERIES (against a follower).
+ */
+
+ try {
+
+ // Start the UPDATE task. Runs many updates.
+ executorService.submit(updateTaskFuture);
+
+ // Start the QUERY task. Runs many queries.
+ executorService.submit(queryTaskFuture);
+
+ // Wait for QueryTask : ends once UPDATE task is done.
+ queryTaskFuture.get(timeout, TimeUnit.MINUTES);
+
+ // Verify update task is done.
+ assertTrue(updateTaskFuture.isDone());
+
+ // Check Future for errors (task is already done).
+ updateTaskFuture.get();
+
+ } finally {
+
+ updateTaskFuture.cancel(true/* mayInterruptIfRunning */);
+
+ queryTaskFuture.cancel(true/* mayInterruptIfRunning */);
+
+ }
+
+ } finally {
+
+ destroyAll();
+
}
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-04-30 17:44:49
|
Revision: 7095
http://bigdata.svn.sourceforge.net/bigdata/?rev=7095&view=rev
Author: thompsonbry
Date: 2013-04-30 17:44:39 +0000 (Tue, 30 Apr 2013)
Log Message:
-----------
Moved the bounceZK and enterErrorState RMI methods from HAGlue into a HAGlueTest interface that is only exposed by the HAJournalTest class. This class is accessible from the HA CI test suite, but is not deployed. This is a much nicer way of exposing hooks for the test suite.
The HA test suite is 100% green.
Modified Paths:
--------------
branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/HAGlue.java
branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/HAPipelineGlue.java
branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/AbstractJournal.java
branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournal.java
branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA2JournalServer.java
branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHAJournalServerOverride.java
Added Paths:
-----------
branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/HAJournalTest.java
Removed Paths:
-------------
branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/HAGlueDelegate.java
branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/MyHAJournal.java
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/HAGlue.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/HAGlue.java 2013-04-30 15:32:54 UTC (rev 7094)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/HAGlue.java 2013-04-30 17:44:39 UTC (rev 7095)
@@ -74,28 +74,6 @@
*/
/**
- * This method may be issued to force the service to close and then reopen
- * its zookeeper connection. This is a drastic action which will cause all
- * <i>ephemeral</i> tokens for that service to be retracted from zookeeper.
- * When the service reconnects, it will reestablish those connections.
- * <p>
- * Note: This method is intended primarily as an aid in writing various HA
- * unit tests.
- *
- * @see http://wiki.apache.org/hadoop/ZooKeeper/FAQ#A4
- */
- public Future<Void> bounceZookeeperConnection() throws IOException;
-
- /**
- * Force the end point to enter into an error state from which it will
- * naturally move back into a consistent state.
- * <p>
- * Note: This method is intended primarily as an aid in writing various HA
- * unit tests.
- */
- public Future<Void> enterErrorState() throws IOException;
-
- /**
* Await the service being ready to partitipate in an HA quorum. The
* preconditions include:
* <ol>
Deleted: branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/HAGlueDelegate.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/HAGlueDelegate.java 2013-04-30 15:32:54 UTC (rev 7094)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/HAGlueDelegate.java 2013-04-30 17:44:39 UTC (rev 7095)
@@ -1,309 +0,0 @@
-/**
-
-Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
-package com.bigdata.ha;
-
-import java.io.IOException;
-import java.net.InetSocketAddress;
-import java.rmi.RemoteException;
-import java.security.DigestException;
-import java.security.NoSuchAlgorithmException;
-import java.util.UUID;
-import java.util.concurrent.BrokenBarrierException;
-import java.util.concurrent.Future;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.TimeoutException;
-
-import com.bigdata.ha.msg.IHA2PhaseAbortMessage;
-import com.bigdata.ha.msg.IHA2PhaseCommitMessage;
-import com.bigdata.ha.msg.IHA2PhasePrepareMessage;
-import com.bigdata.ha.msg.IHADigestRequest;
-import com.bigdata.ha.msg.IHADigestResponse;
-import com.bigdata.ha.msg.IHAGatherReleaseTimeRequest;
-import com.bigdata.ha.msg.IHAGlobalWriteLockRequest;
-import com.bigdata.ha.msg.IHALogDigestRequest;
-import com.bigdata.ha.msg.IHALogDigestResponse;
-import com.bigdata.ha.msg.IHALogRequest;
-import com.bigdata.ha.msg.IHALogRootBlocksRequest;
-import com.bigdata.ha.msg.IHALogRootBlocksResponse;
-import com.bigdata.ha.msg.IHANotifyReleaseTimeRequest;
-import com.bigdata.ha.msg.IHANotifyReleaseTimeResponse;
-import com.bigdata.ha.msg.IHAReadRequest;
-import com.bigdata.ha.msg.IHAReadResponse;
-import com.bigdata.ha.msg.IHARebuildRequest;
-import com.bigdata.ha.msg.IHARootBlockRequest;
-import com.bigdata.ha.msg.IHARootBlockResponse;
-import com.bigdata.ha.msg.IHASendStoreResponse;
-import com.bigdata.ha.msg.IHASnapshotDigestRequest;
-import com.bigdata.ha.msg.IHASnapshotDigestResponse;
-import com.bigdata.ha.msg.IHASnapshotRequest;
-import com.bigdata.ha.msg.IHASnapshotResponse;
-import com.bigdata.ha.msg.IHASyncRequest;
-import com.bigdata.ha.msg.IHAWriteMessage;
-import com.bigdata.ha.msg.IHAWriteSetStateRequest;
-import com.bigdata.ha.msg.IHAWriteSetStateResponse;
-import com.bigdata.quorum.AsynchronousQuorumCloseException;
-import com.bigdata.quorum.QuorumException;
-
-/**
- * Delegation pattern.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- *
- * @deprecated No longer used.
- */
-public class HAGlueDelegate implements HAGlue {
-
- private final HAGlue delegate;
-
- public HAGlueDelegate(final HAGlue delegate) {
-
- if(delegate == null)
- throw new IllegalArgumentException();
-
- this.delegate = delegate;
-
- }
-
- @Override
- public Future<Void> bounceZookeeperConnection() throws IOException {
- return delegate.bounceZookeeperConnection();
- }
-
- @Override
- public Future<Void> enterErrorState() throws IOException {
- return delegate.enterErrorState();
- }
-
- @Override
- public UUID getServiceId() throws IOException {
- return delegate.getServiceId();
- }
-
- @Override
- public Future<Boolean> prepare2Phase(IHA2PhasePrepareMessage msg)
- throws IOException {
- return delegate.prepare2Phase(msg);
- }
-
- @Override
- public Future<IHAReadResponse> readFromDisk(
- IHAReadRequest readMessage) throws IOException {
- return delegate.readFromDisk(readMessage);
- }
-
- @Override
- public InetSocketAddress getWritePipelineAddr() throws IOException {
- return delegate.getWritePipelineAddr();
- }
-
- @Override
- public IHARootBlockResponse getRootBlock(IHARootBlockRequest msg) throws IOException {
- return delegate.getRootBlock(msg);
- }
-
- @Override
- public Future<Void> moveToEndOfPipeline() throws IOException {
- return delegate.moveToEndOfPipeline();
- }
-
- @Override
- public Future<Void> commit2Phase(IHA2PhaseCommitMessage commitMessage)
- throws IOException {
- return delegate.commit2Phase(commitMessage);
- }
-
- @Override
- public Future<Void> abort2Phase(IHA2PhaseAbortMessage abortMessage)
- throws IOException {
- return delegate.abort2Phase(abortMessage);
- }
-
- @Override
- public Future<Void> receiveAndReplicate(final IHASyncRequest req,
- final IHAWriteMessage msg) throws IOException {
- return delegate.receiveAndReplicate(req, msg);
- }
-
- @Override
- public UUID getServiceUUID() throws IOException {
- return delegate.getServiceUUID();
- }
-
- @Override
- public Class getServiceIface() throws IOException {
- return delegate.getServiceIface();
- }
-
- @Override
- public String getHostname() throws IOException {
- return delegate.getHostname();
- }
-
- @Override
- public String getServiceName() throws IOException {
- return delegate.getServiceName();
- }
-
- @Override
- public void destroy() throws RemoteException {
- delegate.destroy();
- }
-
- @Override
- public Future<Void> gatherMinimumVisibleCommitTime(
- final IHAGatherReleaseTimeRequest req) throws IOException {
- return delegate.gatherMinimumVisibleCommitTime(req);
- }
-
- @Override
- public IHANotifyReleaseTimeResponse notifyEarliestCommitTime(
- final IHANotifyReleaseTimeRequest req) throws IOException,
- InterruptedException, BrokenBarrierException {
- return delegate.notifyEarliestCommitTime(req);
- }
-
-// @Override
-// public Future<Void> getTXSCriticalSectionLockOnLeader(
-// final IHATXSLockRequest req) throws IOException {
-// return delegate.getTXSCriticalSectionLockOnLeader(req);
-// }
-
-// @Override
-// public long nextTimestamp() throws IOException {
-// return delegate.nextTimestamp();
-// }
-//
-// @Override
-// public long newTx(long timestamp) throws IOException {
-// return delegate.newTx(timestamp);
-// }
-//
-// @Override
-// public long commit(long tx) throws ValidationError, IOException {
-// return delegate.commit(tx);
-// }
-//
-// @Override
-// public void abort(long tx) throws IOException {
-// delegate.abort(tx);
-// }
-//
-// @Override
-// public void notifyCommit(long commitTime) throws IOException {
-// delegate.notifyCommit(commitTime);
-// }
-//
-// @Override
-// public long getLastCommitTime() throws IOException {
-// return delegate.getLastCommitTime();
-// }
-//
-// @Override
-// public long getReleaseTime() throws IOException {
-// return delegate.getReleaseTime();
-// }
-
- @Override
- public IHALogRootBlocksResponse getHALogRootBlocksForWriteSet(
- IHALogRootBlocksRequest msg) throws IOException {
- return delegate.getHALogRootBlocksForWriteSet(msg);
- }
-
- @Override
- public Future<Void> sendHALogForWriteSet(IHALogRequest msg)
- throws IOException {
- return delegate.sendHALogForWriteSet(msg);
- }
-
- @Override
- public int getNSSPort() throws IOException {
- return delegate.getNSSPort();
- }
-
- @Override
- public RunState getRunState() throws IOException {
- return delegate.getRunState();
- }
-
- @Override
- public String getExtendedRunState() throws IOException {
- return delegate.getExtendedRunState();
- }
-
- @Override
- public HAStatusEnum getHAStatus() throws IOException {
- return delegate.getHAStatus();
- }
-
- @Override
- public Future<IHASendStoreResponse> sendHAStore(IHARebuildRequest msg)
- throws IOException {
- return delegate.sendHAStore(msg);
- }
-
- @Override
- public IHADigestResponse computeDigest(final IHADigestRequest req)
- throws IOException, NoSuchAlgorithmException, DigestException {
- return delegate.computeDigest(req);
- }
-
- @Override
- public IHALogDigestResponse computeHALogDigest(final IHALogDigestRequest req)
- throws IOException, NoSuchAlgorithmException, DigestException {
- return delegate.computeHALogDigest(req);
- }
-
- @Override
- public IHASnapshotDigestResponse computeHASnapshotDigest(
- final IHASnapshotDigestRequest req) throws IOException,
- NoSuchAlgorithmException, DigestException {
- return delegate.computeHASnapshotDigest(req);
- }
-
- @Override
- public Future<Void> globalWriteLock(final IHAGlobalWriteLockRequest req)
- throws IOException, TimeoutException, InterruptedException {
- return delegate.globalWriteLock(req);
- }
-
- @Override
- public IHAWriteSetStateResponse getHAWriteSetState(
- final IHAWriteSetStateRequest req) throws IOException {
- return delegate.getHAWriteSetState(req);
- }
-
- @Override
- public long awaitHAReady(final long timeout, final TimeUnit unit)
- throws IOException, InterruptedException, QuorumException,
- AsynchronousQuorumCloseException, TimeoutException {
- return delegate.awaitHAReady(timeout, unit);
- }
-
- @Override
- public Future<IHASnapshotResponse> takeSnapshot(final IHASnapshotRequest req)
- throws IOException {
- return delegate.takeSnapshot(req);
- }
-
-}
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/HAPipelineGlue.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/HAPipelineGlue.java 2013-04-30 15:32:54 UTC (rev 7094)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/HAPipelineGlue.java 2013-04-30 17:44:39 UTC (rev 7095)
@@ -95,7 +95,7 @@
* leave without a service leave, if services which are joined always
* rejoin the write pipeline when the observe a pipeline leave, and if
* we can force a service leave (by instructing the service to
- * {@link #bounceZookeeperConnection()}) if the service fails to
+ * bounce the zookeeper connection) if the service fails to
* rejoin the pipeline, then it would be easier to reorganize the
* pipeline. [This would still make it possible for services to add
* themselves to the pipeline without being joined with the quorum but
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/AbstractJournal.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/AbstractJournal.java 2013-04-30 15:32:54 UTC (rev 7094)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/AbstractJournal.java 2013-04-30 17:44:39 UTC (rev 7095)
@@ -6558,28 +6558,28 @@
}
- @Override
- public Future<Void> bounceZookeeperConnection() {
- final FutureTask<Void> ft = new FutureTaskMon<Void>(new Runnable() {
- public void run() {
- // NOP (not implemented at this layer).
- }
- }, null);
- ft.run();
- return getProxy(ft);
- }
+// @Override
+// public Future<Void> bounceZookeeperConnection() {
+// final FutureTask<Void> ft = new FutureTaskMon<Void>(new Runnable() {
+// public void run() {
+// // NOP (not implemented at this layer).
+// }
+// }, null);
+// ft.run();
+// return getProxy(ft);
+// }
+//
+// @Override
+// public Future<Void> enterErrorState() {
+// final FutureTask<Void> ft = new FutureTaskMon<Void>(new Runnable() {
+// public void run() {
+// // NOP (not implemented at this layer).
+// }
+// }, null);
+// ft.run();
+// return getProxy(ft);
+// }
- @Override
- public Future<Void> enterErrorState() {
- final FutureTask<Void> ft = new FutureTaskMon<Void>(new Runnable() {
- public void run() {
- // NOP (not implemented at this layer).
- }
- }, null);
- ft.run();
- return getProxy(ft);
- }
-
/**
* {@inheritDoc}
* <p>
Modified: branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournal.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournal.java 2013-04-30 15:32:54 UTC (rev 7094)
+++ branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournal.java 2013-04-30 17:44:39 UTC (rev 7095)
@@ -103,7 +103,6 @@
import com.bigdata.journal.jini.ha.HAJournalServer.RunStateEnum;
import com.bigdata.quorum.AsynchronousQuorumCloseException;
import com.bigdata.quorum.Quorum;
-import com.bigdata.quorum.zk.ZKQuorumImpl;
import com.bigdata.service.AbstractTransactionService;
import com.bigdata.service.jini.JiniClient;
import com.bigdata.service.jini.RemoteAdministrable;
@@ -1375,84 +1374,6 @@
}
- @Override
- public Future<Void> bounceZookeeperConnection() {
-
- final FutureTask<Void> ft = new FutureTaskMon<Void>(
- new BounceZookeeperConnectionTask(), null/* result */);
-
- ft.run();
-
- return getProxy(ft);
-
- }
-
- private class BounceZookeeperConnectionTask implements Runnable {
-
- @SuppressWarnings("rawtypes")
- public void run() {
-
- if (getQuorum() instanceof ZKQuorumImpl) {
-
- // Note: Local method call on AbstractJournal.
- final UUID serviceId = getServiceId();
-
- try {
-
- haLog.warn("BOUNCING ZOOKEEPER CONNECTION: "
- + serviceId);
-
- // Close the current connection (if any).
- ((ZKQuorumImpl) getQuorum()).getZookeeper().close();
-
- // Obtain a new connection.
- ((ZKQuorumImpl) getQuorum()).getZookeeper();
-
- haLog.warn("RECONNECTED TO ZOOKEEPER: " + serviceId);
-
- } catch (InterruptedException e) {
-
- // Propagate the interrupt.
- Thread.currentThread().interrupt();
-
- }
-
- }
- }
-
- }
-
- @Override
- public Future<Void> enterErrorState() {
-
- final FutureTask<Void> ft = new FutureTaskMon<Void>(
- new EnterErrorStateTask(), null/* result */);
-
- ft.run();
-
- return getProxy(ft);
-
- }
-
- private class EnterErrorStateTask implements Runnable {
-
- public void run() {
-
- @SuppressWarnings("unchecked")
- final HAQuorumService<HAGlue, HAJournal> service = (HAQuorumService<HAGlue, HAJournal>) getQuorum()
- .getClient();
-
- // Note: Local method call on AbstractJournal.
- final UUID serviceId = getServiceId();
-
- haLog.warn("ENTERING ERROR STATE: " + serviceId);
-
- service.enterErrorState();
-
- }
-
- }
-
/**
* Note: The invocation layer factory is reused for each exported proxy (but
* the exporter itself is paired 1:1 with the exported proxy).
Copied: branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/HAJournalTest.java (from rev 7092, branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/MyHAJournal.java)
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/HAJournalTest.java (rev 0)
+++ branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/HAJournalTest.java 2013-04-30 17:44:39 UTC (rev 7095)
@@ -0,0 +1,209 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Oct 31, 2012
+ */
+package com.bigdata.journal.jini.ha;
+
+import java.io.IOException;
+import java.rmi.Remote;
+import java.util.UUID;
+import java.util.concurrent.Future;
+import java.util.concurrent.FutureTask;
+
+import net.jini.config.Configuration;
+import net.jini.config.ConfigurationException;
+
+import org.apache.log4j.Logger;
+
+import com.bigdata.concurrent.FutureTaskMon;
+import com.bigdata.ha.HAGlue;
+import com.bigdata.ha.QuorumService;
+import com.bigdata.journal.jini.ha.HAJournalServer.HAQuorumService;
+import com.bigdata.quorum.Quorum;
+import com.bigdata.quorum.zk.ZKQuorumImpl;
+
+/**
+ * Class extends {@link HAJournal} and allows the unit tests to play various
+ * games with the services, simulating a variety of different kinds of problems.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ */
+public class HAJournalTest extends HAJournal {
+
+ private static final Logger log = Logger.getLogger(HAJournal.class);
+
+ public HAJournalTest(final HAJournalServer server,
+ final Configuration config,
+ final Quorum<HAGlue, QuorumService<HAGlue>> quorum)
+ throws ConfigurationException, IOException {
+
+ super(server, config, quorum);
+
+ }
+
+ @Override
+ protected HAGlue newHAGlue(final UUID serviceId) {
+
+ return new HAGlueTestService(serviceId);
+
+ }
+
+ /**
+ * A {@link Remote} interface for new methods published by the service.
+ */
+ public static interface HAGlueTest extends HAGlue {
+
+ /**
+ * Logs a "hello world" message.
+ */
+ public void helloWorld() throws IOException;
+
+ /**
+ * Force the end point to enter into an error state from which it will
+ * naturally move back into a consistent state.
+ * <p>
+ * Note: This method is intended primarily as an aid in writing various HA
+ * unit tests.
+ */
+ public Future<Void> enterErrorState() throws IOException;
+
+ /**
+ * This method may be issued to force the service to close and then reopen
+ * its zookeeper connection. This is a drastic action which will cause all
+ * <i>ephemeral</i> tokens for that service to be retracted from zookeeper.
+ * When the service reconnects, it will reestablish those connections.
+ * <p>
+ * Note: This method is intended primarily as an aid in writing various HA
+ * unit tests.
+ *
+ * @see http://wiki.apache.org/hadoop/ZooKeeper/FAQ#A4
+ */
+ public Future<Void> bounceZookeeperConnection() throws IOException;
+
+ }
+
+ /**
+ * Class extends the public RMI interface of the {@link HAJournal}.
+ * <p>
+ * Note: Any new RMI methods must be (a) declared on an interface; and (b)
+ * must throw {@link IOException}.
+ */
+ protected class HAGlueTestService extends HAJournal.HAGlueService implements
+ HAGlueTest {
+
+ protected HAGlueTestService(final UUID serviceId) {
+
+ super(serviceId);
+
+ }
+
+ @Override
+ public void helloWorld() throws IOException {
+
+ log.warn("Hello world!");
+
+ }
+
+ @Override
+ public Future<Void> enterErrorState() {
+
+ final FutureTask<Void> ft = new FutureTaskMon<Void>(
+ new EnterErrorStateTask(), null/* result */);
+
+ ft.run();
+
+ return getProxy(ft);
+
+ }
+
+ private class EnterErrorStateTask implements Runnable {
+
+ public void run() {
+
+ @SuppressWarnings("unchecked")
+ final HAQuorumService<HAGlue, HAJournal> service = (HAQuorumService<HAGlue, HAJournal>) getQuorum()
+ .getClient();
+
+ // Note: Local method call on AbstractJournal.
+ final UUID serviceId = getServiceId();
+
+ haLog.warn("ENTERING ERROR STATE: " + serviceId);
+
+ service.enterErrorState();
+
+ }
+
+ }
+
+ @Override
+ public Future<Void> bounceZookeeperConnection() {
+
+ final FutureTask<Void> ft = new FutureTaskMon<Void>(
+ new BounceZookeeperConnectionTask(), null/* result */);
+
+ ft.run();
+
+ return getProxy(ft);
+
+ }
+
+ private class BounceZookeeperConnectionTask implements Runnable {
+
+ @SuppressWarnings("rawtypes")
+ public void run() {
+
+ if (getQuorum() instanceof ZKQuorumImpl) {
+
+ // Note: Local method call on AbstractJournal.
+ final UUID serviceId = getServiceId();
+
+ try {
+
+ haLog.warn("BOUNCING ZOOKEEPER CONNECTION: "
+ + serviceId);
+
+ // Close the current connection (if any).
+ ((ZKQuorumImpl) getQuorum()).getZookeeper().close();
+
+ // Obtain a new connection.
+ ((ZKQuorumImpl) getQuorum()).getZookeeper();
+
+ haLog.warn("RECONNECTED TO ZOOKEEPER: " + serviceId);
+
+ } catch (InterruptedException e) {
+
+ // Propagate the interrupt.
+ Thread.currentThread().interrupt();
+
+ }
+
+ }
+ }
+
+ }
+
+ } // HAGlueTestService
+
+}
Deleted: branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/MyHAJournal.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/MyHAJournal.java 2013-04-30 15:32:54 UTC (rev 7094)
+++ branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/MyHAJournal.java 2013-04-30 17:44:39 UTC (rev 7095)
@@ -1,101 +0,0 @@
-/**
-
-Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
-/*
- * Created on Oct 31, 2012
- */
-package com.bigdata.journal.jini.ha;
-
-import java.io.IOException;
-import java.rmi.Remote;
-import java.util.UUID;
-
-import org.apache.log4j.Logger;
-
-import net.jini.config.Configuration;
-import net.jini.config.ConfigurationException;
-
-import com.bigdata.ha.HAGlue;
-import com.bigdata.ha.QuorumService;
-import com.bigdata.quorum.Quorum;
-
-/**
- * Class extends {@link HAJournal}.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- */
-public class MyHAJournal extends HAJournal {
-
- private static final Logger log = Logger.getLogger(HAJournal.class);
-
- public MyHAJournal(final HAJournalServer server,
- final Configuration config,
- final Quorum<HAGlue, QuorumService<HAGlue>> quorum)
- throws ConfigurationException, IOException {
-
- super(server, config, quorum);
-
- }
-
- @Override
- protected HAGlue newHAGlue(final UUID serviceId) {
-
-// return super.newHAGlue(serviceId);
- return new MyHAGlueService(serviceId);
-
- }
-
- /**
- * A {@link Remote} interface for new methods published by the service.
- */
- public static interface MyHAGlue extends HAGlue {
-
- public void helloWorld() throws IOException;
-
- }
-
- /**
- * Class extends the public RMI interface of the {@link HAJournal}.
- * <p>
- * Note: Any new RMI methods must be (a) declared on an interface; and (b)
- * must throw {@link IOException}.
- */
- protected class MyHAGlueService extends HAJournal.HAGlueService implements
- MyHAGlue {
-
- protected MyHAGlueService(final UUID serviceId) {
-
- super(serviceId);
-
- }
-
- @Override
- public void helloWorld() throws IOException {
-
- log.warn("Hello world!");
-
- }
-
- }
-
-}
Modified: branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA2JournalServer.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA2JournalServer.java 2013-04-30 15:32:54 UTC (rev 7094)
+++ branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA2JournalServer.java 2013-04-30 17:44:39 UTC (rev 7095)
@@ -37,6 +37,7 @@
import com.bigdata.ha.HAStatusEnum;
import com.bigdata.ha.msg.HARootBlockRequest;
import com.bigdata.journal.IRootBlockView;
+import com.bigdata.journal.jini.ha.HAJournalTest.HAGlueTest;
import com.bigdata.quorum.Quorum;
import com.bigdata.rdf.sail.webapp.client.RemoteRepository;
@@ -66,7 +67,8 @@
return new String[]{
"com.bigdata.journal.jini.ha.HAJournalServer.restorePolicy=new com.bigdata.journal.jini.ha.DefaultRestorePolicy(0L,1,0)",
- "com.bigdata.journal.jini.ha.HAJournalServer.snapshotPolicy=new com.bigdata.journal.jini.ha.NoSnapshotPolicy()"
+ "com.bigdata.journal.jini.ha.HAJournalServer.snapshotPolicy=new com.bigdata.journal.jini.ha.NoSnapshotPolicy()",
+ "com.bigdata.journal.jini.ha.HAJournalServer.HAJournalClass=\""+HAJournalTest.class.getName()+"\""
};
}
@@ -74,7 +76,7 @@
public TestHA2JournalServer() {
}
- public TestHA2JournalServer(String name) {
+ public TestHA2JournalServer(final String name) {
super(name);
}
@@ -214,11 +216,11 @@
if (leader.equals(serverA)) {
- serverB.bounceZookeeperConnection().get();
+ ((HAGlueTest) serverB).bounceZookeeperConnection().get();
} else {
- serverA.bounceZookeeperConnection().get();
+ ((HAGlueTest) serverA).bounceZookeeperConnection().get();
}
@@ -317,7 +319,7 @@
// final UUID leaderId1 = leader.getServiceId();
- leader.bounceZookeeperConnection().get();
+ ((HAGlueTest)leader).bounceZookeeperConnection().get();
// Wait for the quorum to break and then meet again.
final long token2 = awaitNextQuorumMeet(token1);
Modified: branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHAJournalServerOverride.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHAJournalServerOverride.java 2013-04-30 15:32:54 UTC (rev 7094)
+++ branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHAJournalServerOverride.java 2013-04-30 17:44:39 UTC (rev 7095)
@@ -33,7 +33,7 @@
import com.bigdata.ha.HAGlue;
import com.bigdata.ha.HAStatusEnum;
-import com.bigdata.journal.jini.ha.MyHAJournal.MyHAGlue;
+import com.bigdata.journal.jini.ha.HAJournalTest.HAGlueTest;
/**
* Unit test of the ability to override the {@link HAJournal} implementation
@@ -63,7 +63,7 @@
return new String[]{
"com.bigdata.journal.jini.ha.HAJournalServer.restorePolicy=new com.bigdata.journal.jini.ha.DefaultRestorePolicy(0L,1,0)",
"com.bigdata.journal.jini.ha.HAJournalServer.snapshotPolicy=new com.bigdata.journal.jini.ha.NoSnapshotPolicy()",
- "com.bigdata.journal.jini.ha.HAJournalServer.HAJournalClass=\"com.bigdata.journal.jini.ha.MyHAJournal\""
+ "com.bigdata.journal.jini.ha.HAJournalServer.HAJournalClass=\""+HAJournalTest.class.getName()+"\""
};
}
@@ -115,7 +115,7 @@
assertTrue(getSnapshotDirA().exists());
// Verify that we can invoke extension methods on the service.
- ((MyHAGlue)serverA).helloWorld();
+ ((HAGlueTest)serverA).helloWorld();
// serverA.enterErrorState().get();
//
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-04-30 15:33:01
|
Revision: 7094
http://bigdata.svn.sourceforge.net/bigdata/?rev=7094&view=rev
Author: thompsonbry
Date: 2013-04-30 15:32:54 +0000 (Tue, 30 Apr 2013)
Log Message:
-----------
Added a concept for an "Operator" run state. Added some stubs for operator alerts. This is only displayed through the /status page for now (through getExtendedRunState()).
Added a boolean option that may be used to enable the online disaster recovery (REBUILD). It is disabled by default for now to avoid "accidental" rebuilds. We could give people a button to press to start a rebuild on the status page, but right now I want to make sure that we do not start rebuilds unnecessarily.
Added override in TestHA3JournalServer to enable the testing of REBUILD even if it is disabled by default.
Modified Paths:
--------------
branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournal.java
branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournalServer.java
branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA3JournalServer.java
Modified: branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournal.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournal.java 2013-04-30 15:21:45 UTC (rev 7093)
+++ branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournal.java 2013-04-30 15:32:54 UTC (rev 7094)
@@ -1664,11 +1664,26 @@
: quorumService.getRunStateEnum());
final HAJournal journal = HAJournal.this;
-
- final String innerRunStateStr = (innerRunState == null ? "N/A"
- : (innerRunState.name() + ((innerRunState == RunStateEnum.Resync && journal != null) ? (" @ " + journal
- .getRootBlockView().getCommitCounter()) : "")));
+ final StringBuilder innerRunStateStr = new StringBuilder();
+ if (innerRunState != null) {
+ innerRunStateStr.append(innerRunState.name());
+ switch (innerRunState) {
+ case Resync:
+ innerRunStateStr.append(" @ "
+ + journal.getRootBlockView().getCommitCounter());
+ break;
+ case Operator: {
+ final String msg = server.getOperatorAlert();
+ innerRunStateStr.append("msg=" + msg);
+ break;
+ }
+ default:
+ break;
+ }
+ } else {
+ innerRunStateStr.append("N/A");
+ }
return "{server=" + server.getRunState() + ", quorumService="
+ innerRunStateStr + "}";
Modified: branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournalServer.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournalServer.java 2013-04-30 15:21:45 UTC (rev 7093)
+++ branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournalServer.java 2013-04-30 15:32:54 UTC (rev 7094)
@@ -301,7 +301,28 @@
String HA_JOURNAL_CLASS = "HAJournalClass";
String DEFAULT_HA_JOURNAL_CLASS = HAJournal.class.getName();
+
+ /**
+ * When <code>true</code> the service will automatically perform online
+ * disaster recovery (REBUILD). When <code>false</code>, it will enter
+ * the OPERATOR state instead (human intervention required).
+ */
+ String ONLINE_DISASTER_RECOVERY = "onlineDisasterRecovery";
+ /**
+ * TODO This feature is disabled by default. We want to develop more
+ * experience with the online disaster recovery and snapshot / restore
+ * mechanisms before putting enabling it in a release. There are two
+ * possible downsides to enabling this feature: (1) REBUILD should not
+ * trigger unless it is necessary so we need to make sure that spurious
+ * and curable errors do not result in a REBUILD; (2) The current
+ * REBUILD implementation does not replicate the pinned HALogs from the
+ * leader. This means that the rebuilt service is not as robust since it
+ * can not replicate the missing HALogs to another service if that
+ * service should then require disaster recovery as well.
+ */
+ boolean DEFAULT_ONLINE_DISASTER_RECOVERY = false;
+
}
/**
@@ -334,6 +355,11 @@
private UUID serviceUUID;
+ /**
+ * @see ConfigurationOptions#ONLINE_DISASTER_RECOVERY
+ */
+ private boolean onelineDisasterRecovery;
+
private ZookeeperClientConfig zkClientConfig;
private ZooKeeperAccessor zka;
@@ -388,13 +414,48 @@
* state.
*/
static enum RunStateEnum {
- Restore, // apply local HALog files GT current commit point.
- SeekConsensus, // seek consensus.
- RunMet, // run while joined with met quorum.
- Resync, // only resynchronization
- Rebuild, // online disaster recovery
- Error, // error state.
- Shutdown; // TODO SHUTDOWN: We are not using this systematically (no ShutdownTask for this run state).
+ /**
+ * Roll forward the database by applying local HALog files GT current
+ * commit point.
+ */
+ Restore,
+ /**
+ * Seek a consensus with the other quorum members regarding the most
+ * recent commit point on the database. If a consensus is established
+ * then the quorum will meet. If the quorum is already met, then this
+ * service must {@link RunStateEnum#Resync}.
+ */
+ SeekConsensus,
+ /** Run while joined with met quorum. */
+ RunMet,
+ /**
+ * Resynchronize with the leader of a met quorum, replicating and
+ * applying HALogs and rolling forward the database until it catches up
+ * with the quorum and joins.
+ */
+ Resync,
+ /**
+ * Online disaster recovery. The backing store is replicated from the
+ * quorum leader and then the service enters {@link RunStateEnum#Resync}
+ * to catch up with any missed writes since the start of the replication
+ * procedure.
+ */
+ Rebuild,
+ /**
+ * Error state. This state should be self correcting.
+ */
+ Error,
+ /**
+ * Shutdown.
+ *
+ * TODO SHUTDOWN: We are not using this systematically (no ShutdownTask
+ * for this run state).
+ */
+ Shutdown,
+ /**
+ * Operator intervention required - service can not proceed.
+ */
+ Operator;
}
/**
@@ -411,7 +472,31 @@
super(args, lifeCycle);
}
+
+ /*
+ * Operator alerts.
+ *
+ * TODO If we keep this abstraction, then extract an interface for operator
+ * alerts. However, this could also be captured as Entry[] attributes that
+ * are published on the service registrar. Once those attributes are
+ * published, it is easy enough to write utilities to monitor those Entry[]
+ * attributes and then execute appropriate business logic.
+ */
+ private final AtomicReference<String> operatorAlert = new AtomicReference<String>();
+
+ protected void sendOperatorAlert(final String msg) {
+ operatorAlert.set(msg);
+ }
+
+ protected void clearOperatorAlert() {
+ operatorAlert.set(null);
+ }
+
+ protected String getOperatorAlert() {
+ return operatorAlert.get();
+ }
+
@Override
protected void terminate() {
@@ -477,8 +562,16 @@
// UUID variant of that ServiceID.
serviceUUID = JiniUtil.serviceID2UUID(serviceID);
-
+
/*
+ * Extract various configuration options.
+ */
+ onelineDisasterRecovery = (Boolean) config.getEntry(
+ ConfigurationOptions.COMPONENT,
+ ConfigurationOptions.ONLINE_DISASTER_RECOVERY, Boolean.TYPE,
+ ConfigurationOptions.DEFAULT_ONLINE_DISASTER_RECOVERY);
+
+ /*
* Setup the Quorum / HAJournal.
*/
@@ -1649,6 +1742,46 @@
} // class RunMetTask
/**
+ * While the quorum is met, accept replicated writes, laying them down
+ * on the HALog and the backing store, and participate in the 2-phase
+ * commit protocol.
+ */
+ private class OperatorTask extends RunStateCallable<Void> {
+
+ final String msg;
+
+ public OperatorTask(final String msg) {
+
+ super(RunStateEnum.Operator);
+
+ this.msg = msg;
+
+ }
+
+ @Override
+ public Void doRun() throws Exception {
+
+ try {
+
+ server.sendOperatorAlert(msg);
+
+ // Block until this run state gets interrupted.
+ blockInterruptably();
+
+ // Done.
+ return null;
+
+ } finally {
+
+ server.clearOperatorAlert();
+
+ }
+
+ } // call()
+
+ } // class OperatorTask
+
+ /**
* Run state responsible for replaying local HALog files during service
* startup. This allows an offline restore of the backing journal file
* (a full backup) and zero or more HALog files (each an incremental
@@ -2145,11 +2278,15 @@
log.error(msg);
- enterRunState(new RebuildTask(token));
+ if (server.onelineDisasterRecovery) {
+ enterRunState(new RebuildTask(token));
+ } else {
+ enterRunState(new OperatorTask(msg));
+ }
// Force immediate exit of the resync protocol.
throw new InterruptedException(msg);
-
+
}
// root block when the quorum started that write set.
Modified: branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA3JournalServer.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA3JournalServer.java 2013-04-30 15:21:45 UTC (rev 7093)
+++ branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA3JournalServer.java 2013-04-30 15:32:54 UTC (rev 7094)
@@ -89,7 +89,8 @@
return new String[]{
"com.bigdata.journal.jini.ha.HAJournalServer.restorePolicy=new com.bigdata.journal.jini.ha.DefaultRestorePolicy(0L,1,0)",
- "com.bigdata.journal.jini.ha.HAJournalServer.snapshotPolicy=new com.bigdata.journal.jini.ha.NoSnapshotPolicy()"
+ "com.bigdata.journal.jini.ha.HAJournalServer.snapshotPolicy=new com.bigdata.journal.jini.ha.NoSnapshotPolicy()",
+ "com.bigdata.journal.jini.ha.HAJournalServer.onlineDisasterRecovery=true"
};
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mrp...@us...> - 2013-04-30 15:21:53
|
Revision: 7093
http://bigdata.svn.sourceforge.net/bigdata/?rev=7093&view=rev
Author: mrpersonick
Date: 2013-04-30 15:21:45 +0000 (Tue, 30 Apr 2013)
Log Message:
-----------
added a recursion flag
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTComplexOptionalOptimizer.java
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTComplexOptionalOptimizer.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTComplexOptionalOptimizer.java 2013-04-30 13:04:12 UTC (rev 7092)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTComplexOptionalOptimizer.java 2013-04-30 15:21:45 UTC (rev 7093)
@@ -133,6 +133,11 @@
// private static final Logger log = Logger
// .getLogger(ASTComplexOptionalOptimizer.class);
+
+ /**
+ * Set this to false to only optimize the top level group.
+ */
+ private static boolean recurse = true;
@Override
public IQueryNode optimize(final AST2BOpContext context,
@@ -201,8 +206,12 @@
@SuppressWarnings("unchecked")
final GraphPatternGroup<IGroupMemberNode> childGroup = (GraphPatternGroup<IGroupMemberNode>) child;
- convertComplexOptionalGroups(context, sa, query, childGroup,
- exogenousVars);
+ if (recurse) {
+
+ convertComplexOptionalGroups(context, sa, query, childGroup,
+ exogenousVars);
+
+ }
if (childGroup.isOptional()
&& (!(childGroup.arity() == 1 && childGroup.get(0) instanceof NamedSubqueryInclude))) {
@@ -223,8 +232,12 @@
final SubqueryRoot subqueryRoot = (SubqueryRoot) child;
- convertComplexOptionalGroups(context, sa, query,
- subqueryRoot.getWhereClause(), exogenousVars);
+ if (recurse) {
+
+ convertComplexOptionalGroups(context, sa, query,
+ subqueryRoot.getWhereClause(), exogenousVars);
+
+ }
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-04-30 13:04:21
|
Revision: 7092
http://bigdata.svn.sourceforge.net/bigdata/?rev=7092&view=rev
Author: thompsonbry
Date: 2013-04-30 13:04:12 +0000 (Tue, 30 Apr 2013)
Log Message:
-----------
Added a Configuration option to HAJournalServer to allow the HAJournal to be subclassed. This is being done to support writing unit tests that cause failures in the 2-phase commit protocol. Wrote a unit test which verifies that the HAJournal may be subclassed, including the ability to publish new RMI methods that are only visible to the test suite.
Pushed down the Administrable RMI implementation onto HAJournal.HAGlueService and removed the use of the delegation pattern to wrap the HAGlueService with the Administrable methods.
HAGlueDelegate is no longer used and has been deprecated.
Modified Paths:
--------------
branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/HAGlueDelegate.java
branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/IRWStrategy.java
branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournal.java
branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournalServer.java
branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestAll.java
Added Paths:
-----------
branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/MyHAJournal.java
branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHAJournalServerOverride.java
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/HAGlueDelegate.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/HAGlueDelegate.java 2013-04-29 20:15:51 UTC (rev 7091)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/HAGlueDelegate.java 2013-04-30 13:04:12 UTC (rev 7092)
@@ -69,6 +69,8 @@
* Delegation pattern.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ *
+ * @deprecated No longer used.
*/
public class HAGlueDelegate implements HAGlue {
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/IRWStrategy.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/IRWStrategy.java 2013-04-29 20:15:51 UTC (rev 7091)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/IRWStrategy.java 2013-04-30 13:04:12 UTC (rev 7092)
@@ -38,7 +38,11 @@
/**
* Called post commit to dispose any transient commit state retained to
- * support reset/rollback
+ * support reset/rollback.
+ * <p>
+ * Note: It is the responsibility of the commit protocol layers to wind up
+ * calling {@link IBufferStrategy#abort()} if there is a failure during the
+ * commit protocol.
*/
public void postCommit();
Modified: branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournal.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournal.java 2013-04-29 20:15:51 UTC (rev 7091)
+++ branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournal.java 2013-04-30 13:04:12 UTC (rev 7092)
@@ -31,6 +31,7 @@
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.rmi.Remote;
+import java.rmi.RemoteException;
import java.rmi.server.ExportException;
import java.security.DigestException;
import java.security.MessageDigest;
@@ -62,6 +63,7 @@
import com.bigdata.ha.HAGlue;
import com.bigdata.ha.QuorumService;
import com.bigdata.ha.QuorumServiceBase;
+import com.bigdata.ha.RunState;
import com.bigdata.ha.halog.HALogReader;
import com.bigdata.ha.halog.HALogWriter;
import com.bigdata.ha.halog.IHALogReader;
@@ -97,11 +99,15 @@
import com.bigdata.journal.Journal;
import com.bigdata.journal.WriteExecutorService;
import com.bigdata.journal.jini.ha.HAJournalServer.HAQuorumService;
+import com.bigdata.journal.jini.ha.HAJournalServer.NSSConfigurationOptions;
+import com.bigdata.journal.jini.ha.HAJournalServer.RunStateEnum;
import com.bigdata.quorum.AsynchronousQuorumCloseException;
import com.bigdata.quorum.Quorum;
import com.bigdata.quorum.zk.ZKQuorumImpl;
import com.bigdata.service.AbstractTransactionService;
import com.bigdata.service.jini.JiniClient;
+import com.bigdata.service.jini.RemoteAdministrable;
+import com.bigdata.service.jini.RemoteDestroyAdmin;
import com.bigdata.service.proxy.ClientFuture;
import com.bigdata.service.proxy.RemoteFuture;
import com.bigdata.service.proxy.RemoteFutureImpl;
@@ -170,6 +176,11 @@
}
/**
+ * The {@link HAJournalServer} instance that is managing this {@link HAJournal}.
+ */
+ private final HAJournalServer server;
+
+ /**
* @see HAJournalServer.ConfigurationOptions#WRITE_PIPELINE_ADDR
*/
private final InetSocketAddress writePipelineAddr;
@@ -229,6 +240,16 @@
}
/**
+ * The {@link HAJournalServer} instance that is managing this
+ * {@link HAJournal}.
+ */
+ protected HAJournalServer getHAJournalServer() {
+
+ return server;
+
+ }
+
+ /**
* {@inheritDoc}
* <p>
* Overridden to strengthen the return type.
@@ -241,12 +262,15 @@
}
/**
- *
+ * @param server
+ * The {@link HAJournalServer} instance.
* @param config
+ * The {@link Configuration} object.
* @param quorum
+ * The {@link Quorum} implementation.
*
* @throws ConfigurationException
- * @throws IOException
+ * @throws IOException
*/
public HAJournal(final HAJournalServer server, final Configuration config,
final Quorum<HAGlue, QuorumService<HAGlue>> quorum)
@@ -278,6 +302,8 @@
super(checkProperties(properties), quorum);
+ this.server = server;
+
{
// The address at which this service exposes its write pipeline.
@@ -398,6 +424,10 @@
}
+ /*
+ * Note: The HAJournal and HAGlueService MAY be subclassed. Therefore, do
+ * not perform any initialization in this factory method.
+ */
@Override
protected HAGlue newHAGlue(final UUID serviceId) {
@@ -425,11 +455,15 @@
* observes a quorum break or similar event. This is just being
* proactive.
*
- * TODO Lift into HAJournalServer.quorumBreak() handler?
- *
* FIXME This will not be called if the quorum remains met but the
* local service leaves the quorum. However, we should still cancel
- * a running snapshot if that occurs.
+ * a running snapshot if that occurs (if we add a serviceLeave()
+ * handle then this will fix that).
+ *
+ * FIXME We MUST ensure that the snapshot is terminated based on the
+ * clear of the haReadyToken. Even if the quorumToken does not
+ * change, the haReadyToken can be cleared if a serviceLeave() is
+ * performed by this service.
*/
final Future<IHASnapshotResponse> ft = getSnapshotManager()
@@ -605,7 +639,8 @@
/**
* Extended implementation supports RMI.
*/
- protected class HAGlueService extends BasicHA {
+ protected class HAGlueService extends BasicHA implements
+ RemoteAdministrable, RemoteDestroyAdmin {
protected HAGlueService(final UUID serviceId) {
@@ -1442,7 +1477,7 @@
* @return The {@link Exporter}.
*/
protected Exporter getExporter(final boolean enableDGC) {
-
+ // TODO This should be based on the Configuration object (EXPORTER field). See AbstractServer.
return new BasicJeriExporter(TcpServerEndpoint
.getInstance(0/* port */), invocationLayerFactory, enableDGC,
false/* keepAlive */);
@@ -1523,6 +1558,122 @@
}
+ /*
+ *
+ */
+
+ /**
+ * Returns an object that implements whatever administration interfaces
+ * are appropriate for the particular service.
+ *
+ * @return an object that implements whatever administration interfaces
+ * are appropriate for the particular service.
+ */
+ public Object getAdmin() throws RemoteException {
+
+ if (log.isInfoEnabled())
+ log.info("serviceID=" + server.getServiceID());
+
+ return server.proxy;
+
+ }
+
+ /*
+ * DestroyAdmin
+ */
+
+ @Override
+ public void destroy() {
+
+ server.runShutdown(true/* destroy */);
+
+ }
+
+ @Override
+ public void shutdown() {
+
+ server.runShutdown(false/* destroy */);
+
+ }
+
+ @Override
+ public void shutdownNow() {
+
+ server.runShutdown(false/* destroy */);
+
+ }
+
+ /*
+ * Misc.
+ */
+
+ @Override
+ public int getNSSPort() {
+
+ final String COMPONENT = NSSConfigurationOptions.COMPONENT;
+
+ try {
+
+ final Integer port = (Integer) server.config.getEntry(
+ COMPONENT, NSSConfigurationOptions.PORT, Integer.TYPE,
+ NSSConfigurationOptions.DEFAULT_PORT);
+
+ return port;
+
+ } catch (ConfigurationException e) {
+
+ throw new RuntimeException(e);
+
+ }
+
+ }
+
+ @Override
+ public RunState getRunState() {
+
+ return server.getRunState();
+
+ }
+
+ /**
+ * Return this quorum member, appropriately cast.
+ *
+ * @return The quorum member -or- <code>null</code> if the quorum is not
+ * running.
+ */
+ protected HAQuorumService<HAGlue, HAJournal> getQuorumService() {
+
+ // This quorum member.
+ @SuppressWarnings("unchecked")
+ final HAQuorumService<HAGlue, HAJournal> quorumService = (HAQuorumService<HAGlue, HAJournal>) getQuorum()
+ .getClient();
+
+ return quorumService;
+
+ }
+
+ @Override
+ public String getExtendedRunState() {
+
+ final HAJournalServer server = getHAJournalServer();
+
+ // This quorum member.
+ final HAQuorumService<HAGlue, HAJournal> quorumService = getQuorumService();
+
+ final RunStateEnum innerRunState = (quorumService == null ? null
+ : quorumService.getRunStateEnum());
+
+ final HAJournal journal = HAJournal.this;
+
+ final String innerRunStateStr = (innerRunState == null ? "N/A"
+ : (innerRunState.name() + ((innerRunState == RunStateEnum.Resync && journal != null) ? (" @ " + journal
+ .getRootBlockView().getCommitCounter()) : "")));
+
+ return "{server=" + server.getRunState() + ", quorumService="
+ + innerRunStateStr + "}";
+
+ }
+
}
-
+
}
Modified: branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournalServer.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournalServer.java 2013-04-29 20:15:51 UTC (rev 7091)
+++ branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournalServer.java 2013-04-30 13:04:12 UTC (rev 7092)
@@ -28,11 +28,12 @@
import java.io.FileNotFoundException;
import java.io.FilenameFilter;
import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
import java.net.InetSocketAddress;
import java.net.URL;
import java.nio.ByteBuffer;
import java.rmi.Remote;
-import java.rmi.RemoteException;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
@@ -54,7 +55,6 @@
import net.jini.core.lookup.ServiceRegistrar;
import org.apache.log4j.Logger;
-import org.apache.log4j.MDC;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException.NodeExistsException;
import org.apache.zookeeper.data.ACL;
@@ -62,11 +62,9 @@
import com.bigdata.concurrent.FutureTaskMon;
import com.bigdata.ha.HAGlue;
-import com.bigdata.ha.HAGlueDelegate;
import com.bigdata.ha.HAPipelineGlue;
import com.bigdata.ha.QuorumService;
import com.bigdata.ha.QuorumServiceBase;
-import com.bigdata.ha.RunState;
import com.bigdata.ha.halog.HALogWriter;
import com.bigdata.ha.halog.IHALogReader;
import com.bigdata.ha.msg.HALogRequest;
@@ -102,8 +100,6 @@
import com.bigdata.rwstore.RWStore;
import com.bigdata.service.AbstractHATransactionService;
import com.bigdata.service.jini.FakeLifeCycle;
-import com.bigdata.service.jini.RemoteAdministrable;
-import com.bigdata.service.jini.RemoteDestroyAdmin;
import com.bigdata.util.InnerCause;
import com.bigdata.util.concurrent.LatchedExecutor;
import com.bigdata.util.concurrent.MonitoredFutureTask;
@@ -299,6 +295,13 @@
IRestorePolicy DEFAULT_RESTORE_POLICY = new DefaultRestorePolicy();
+ /**
+ * Permit override of the {@link HAJournal} implementation class.
+ */
+ String HA_JOURNAL_CLASS = "HAJournalClass";
+
+ String DEFAULT_HA_JOURNAL_CLASS = HAJournal.class.getName();
+
}
/**
@@ -384,7 +387,7 @@
* Enum of the run states. The states are labeled by the goal of the run
* state.
*/
- private enum RunStateEnum {
+ static enum RunStateEnum {
Restore, // apply local HALog files GT current commit point.
SeekConsensus, // seek consensus.
RunMet, // run while joined with met quorum.
@@ -553,7 +556,7 @@
}
// The HAJournal.
- this.journal = new HAJournal(this, config, quorum);
+ this.journal = newHAJournal(this, config, quorum);
}
@@ -564,16 +567,94 @@
// our external interface.
haGlueService = journal.newHAGlue(serviceUUID);
- // wrap the external interface, exposing administrative functions.
- final AdministrableHAGlueService administrableService = new AdministrableHAGlueService(
- this, haGlueService);
+// // wrap the external interface, exposing administrative functions.
+// final AdministrableHAGlueService administrableService = new AdministrableHAGlueService(
+// this, haGlueService);
+//
+// // return that wrapped interface.
+// return administrableService;
- // return that wrapped interface.
- return administrableService;
+ /*
+ * Return that object. This will get proxied. If we wrap it with a
+ * delegation pattern here, then RMI methods on a subclass of
+ * HAGlueService will not be visible on the exported proxy.
+ */
+ return haGlueService;
}
/**
+ * Permit override of the {@link HAJournal} implementation class.
+ *
+ * @throws ConfigurationException
+ */
+ private HAJournal newHAJournal(final HAJournalServer server,
+ final Configuration config,
+ final Quorum<HAGlue, QuorumService<HAGlue>> quorum)
+ throws ConfigurationException {
+
+ final String className = (String) config.getEntry(
+ ConfigurationOptions.COMPONENT,
+ ConfigurationOptions.HA_JOURNAL_CLASS, String.class,
+ ConfigurationOptions.DEFAULT_HA_JOURNAL_CLASS);
+
+ try {
+ @SuppressWarnings("unchecked")
+ final Class<HAJournal> cls = (Class<HAJournal>) Class
+ .forName(className);
+
+ if (!HAJournal.class.isAssignableFrom(cls)) {
+
+ throw new ConfigurationException("Invalid option: "
+ + ConfigurationOptions.HA_JOURNAL_CLASS + "="
+ + className + ":: Class does not extend "
+ + HAJournal.class);
+
+ }
+
+ final Constructor<HAJournal> ctor = cls.getConstructor(new Class[] {
+ HAJournalServer.class, Configuration.class, Quorum.class });
+
+ final HAJournal jnl = ctor.newInstance(new Object[] { server,
+ config, quorum });
+
+ return jnl;
+
+ } catch (ClassNotFoundException e) {
+
+ throw new ConfigurationException(
+ ConfigurationOptions.HA_JOURNAL_CLASS + "=" + className, e);
+
+ } catch (NoSuchMethodException e) {
+
+ throw new ConfigurationException(
+ ConfigurationOptions.HA_JOURNAL_CLASS + "=" + className, e);
+
+ } catch (InstantiationException e) {
+
+ throw new ConfigurationException(
+ ConfigurationOptions.HA_JOURNAL_CLASS + "=" + className, e);
+
+ } catch (IllegalAccessException e) {
+
+ throw new ConfigurationException(
+ ConfigurationOptions.HA_JOURNAL_CLASS + "=" + className, e);
+
+ } catch (IllegalArgumentException e) {
+
+ throw new ConfigurationException(
+ ConfigurationOptions.HA_JOURNAL_CLASS + "=" + className, e);
+
+ } catch (InvocationTargetException e) {
+
+ throw new ConfigurationException(
+ ConfigurationOptions.HA_JOURNAL_CLASS + "=" + className, e);
+
+ }
+
+ }
+
+ /**
* {@inheritDoc}
* <p>
* Overridden to handle initialization that must wait until the
@@ -798,6 +879,15 @@
private final AtomicReference<RunStateEnum> runStateRef = new AtomicReference<RunStateEnum>(
null/* none */);
+ /*
+ * Exposed to HAJournal.HAGlueService.
+ */
+ protected RunStateEnum getRunStateEnum() {
+
+ return runStateRef.get();
+
+ }
+
protected void setRunState(final RunStateEnum runState) {
if (runStateRef.get() == RunStateEnum.Shutdown) {
@@ -3712,219 +3802,154 @@
}
- /**
- * Adds jini administration interfaces to the basic {@link HAGlue} interface
- * exposed by the {@link HAJournal}.
- *
- * @see HAJournal.HAGlueService
- *
- * @author <a href="mailto:tho...@us...">Bryan
- * Thompson</a>
- */
- public static class AdministrableHAGlueService extends HAGlueDelegate
- implements RemoteAdministrable, RemoteDestroyAdmin {
-
- final protected HAJournalServer server;
-
- public AdministrableHAGlueService(final HAJournalServer server,
- final HAGlue service) {
-
- super(service);
-
- this.server = server;
-
- }
-
- public Object getAdmin() throws RemoteException {
-
- if (log.isInfoEnabled())
- log.info("serviceID=" + server.getServiceID());
-
- return server.proxy;
-
- }
-
-// /**
-// * Sets up the {@link MDC} logging context. You should do this on every
-// * client facing point of entry and then call
-// * {@link #clearLoggingContext()} in a <code>finally</code> clause. You
-// * can extend this method to add additional context.
-// * <p>
-// * This implementation adds the following parameters to the {@link MDC}.
-// * <dl>
-// * <dt>serviceName</dt>
-// * <dd>The serviceName is typically a configuration property for the
-// * service. This datum can be injected into log messages using
-// * <em>%X{serviceName}</em> in your log4j pattern layout.</dd>
-// * <dt>serviceUUID</dt>
-// * <dd>The serviceUUID is, in general, assigned asynchronously by the
-// * service registrar. Once the serviceUUID becomes available it will be
-// * added to the {@link MDC}. This datum can be injected into log
-// * messages using <em>%X{serviceUUID}</em> in your log4j pattern layout.
-// * </dd>
-// * <dt>hostname</dt>
-// * <dd>The hostname statically determined. This datum can be injected
-// * into log messages using <em>%X{hostname}</em> in your log4j pattern
-// * layout.</dd>
-// * <dt>clientname
-// * <dt>
-// * <dd>The hostname or IP address of the client making the request.</dd>
-// * </dl>
-// * Note: {@link InetAddress#getHostName()} is used. This method makes a
-// * one-time best effort attempt to resolve the host name from the
-// * {@link InetAddress}.
-// */
-// private void setupLoggingContext() {
+// /**
+// * Adds jini administration interfaces to the basic {@link HAGlue} interface
+// * exposed by the {@link HAJournal}.
+// *
+// * @see HAJournal.HAGlueService
+// *
+// * @author <a href="mailto:tho...@us...">Bryan
+// * Thompson</a>
+// */
+// public static class AdministrableHAGlueService extends HAGlueDelegate
+// implements RemoteAdministrable, RemoteDestroyAdmin {
//
-// try {
+// final protected HAJournalServer server;
//
-// // Note: This _is_ a local method call.
-// final ServiceID serviceUUID = server.getServiceID();
+// public AdministrableHAGlueService(final HAJournalServer server,
+// final HAGlue service) {
//
-// // Will be null until assigned by the service registrar.
+// super(service);
//
-// if (serviceUUID != null) {
-//
-// MDC.put("serviceUUID", serviceUUID);
-//
-// }
-//
-// MDC.put("serviceName", server.getServiceName());
-//
-// MDC.put("hostname", server.getHostName());
-//
-// try {
-//
-// final InetAddress clientAddr = ((ClientHost) ServerContext
-// .getServerContextElement(ClientHost.class))
-// .getClientHost();
-//
-// MDC.put("clientname", clientAddr.getHostName());
-//
-// } catch (ServerNotActiveException e) {
-//
-// /*
-// * This exception gets thrown if the client has made a
-// * direct (vs RMI) call so we just ignore it.
-// */
-//
-// }
-//
-// } catch (Throwable t) {
-//
-// /*
-// * Ignore.
-// */
-//
-// }
-//
+// this.server = server;
+//
// }
-
- /**
- * Clear the logging context.
- */
- protected void clearLoggingContext() {
-
- MDC.remove("serviceName");
-
- MDC.remove("serviceUUID");
-
- MDC.remove("hostname");
-
- MDC.remove("clientname");
-
- }
-
- /*
- * DestroyAdmin
- */
-
- @Override
- public void destroy() {
-
- server.runShutdown(true/* destroy */);
-
- }
-
- @Override
- public void shutdown() {
-
- server.runShutdown(false/* destroy */);
-
- }
-
- @Override
- public void shutdownNow() {
-
- server.runShutdown(false/* destroy */);
-
- }
-
-// /**
-// * Extends the base behavior to return a {@link Name} of the service
-// * from the {@link Configuration}. If no name was specified in the
-// * {@link Configuration} then the value returned by the base class is
-// * returned instead.
-// */
-// @Override
-// public String getServiceName() {
+//
+//// /**
+//// * Returns an object that implements whatever administration interfaces
+//// * are appropriate for the particular service.
+//// *
+//// * @return an object that implements whatever administration interfaces
+//// * are appropriate for the particular service.
+//// */
+//// public Object getAdmin() throws RemoteException {
+////
+//// if (log.isInfoEnabled())
+//// log.info("serviceID=" + server.getServiceID());
+////
+//// return server.proxy;
+////
+//// }
+//
+//// /**
+//// * Sets up the {@link MDC} logging context. You should do this on every
+//// * client facing point of entry and then call
+//// * {@link #clearLoggingContext()} in a <code>finally</code> clause. You
+//// * can extend this method to add additional context.
+//// * <p>
+//// * This implementation adds the following parameters to the {@link MDC}.
+//// * <dl>
+//// * <dt>serviceName</dt>
+//// * <dd>The serviceName is typically a configuration property for the
+//// * service. This datum can be injected into log messages using
+//// * <em>%X{serviceName}</em> in your log4j pattern layout.</dd>
+//// * <dt>serviceUUID</dt>
+//// * <dd>The serviceUUID is, in general, assigned asynchronously by the
+//// * service registrar. Once the serviceUUID becomes available it will be
+//// * added to the {@link MDC}. This datum can be injected into log
+//// * messages using <em>%X{serviceUUID}</em> in your log4j pattern layout.
+//// * </dd>
+//// * <dt>hostname</dt>
+//// * <dd>The hostname statically determined. This datum can be injected
+//// * into log messages using <em>%X{hostname}</em> in your log4j pattern
+//// * layout.</dd>
+//// * <dt>clientname
+//// * <dt>
+//// * <dd>The hostname or IP address of the client making the request.</dd>
+//// * </dl>
+//// * Note: {@link InetAddress#getHostName()} is used. This method makes a
+//// * one-time best effort attempt to resolve the host name from the
+//// * {@link InetAddress}.
+//// */
+//// private void setupLoggingContext() {
+////
+//// try {
+////
+//// // Note: This _is_ a local method call.
+//// final ServiceID serviceUUID = server.getServiceID();
+////
+//// // Will be null until assigned by the service registrar.
+////
+//// if (serviceUUID != null) {
+////
+//// MDC.put("serviceUUID", serviceUUID);
+////
+//// }
+////
+//// MDC.put("serviceName", server.getServiceName());
+////
+//// MDC.put("hostname", server.getHostName());
+////
+//// try {
+////
+//// final InetAddress clientAddr = ((ClientHost) ServerContext
+//// .getServerContextElement(ClientHost.class))
+//// .getClientHost();
+////
+//// MDC.put("clientname", clientAddr.getHostName());
+////
+//// } catch (ServerNotActiveException e) {
+////
+//// /*
+//// * This exception gets thrown if the client has made a
+//// * direct (vs RMI) call so we just ignore it.
+//// */
+////
+//// }
+////
+//// } catch (Throwable t) {
+////
+//// /*
+//// * Ignore.
+//// */
+////
+//// }
+////
+//// }
+////
+//// /**
+//// * Clear the logging context.
+//// */
+//// protected void clearLoggingContext() {
+////
+//// MDC.remove("serviceName");
+////
+//// MDC.remove("serviceUUID");
+////
+//// MDC.remove("hostname");
+////
+//// MDC.remove("clientname");
+////
+//// }
//
-// String s = server.getServiceName();
+//// /**
+//// * Extends the base behavior to return a {@link Name} of the service
+//// * from the {@link Configuration}. If no name was specified in the
+//// * {@link Configuration} then the value returned by the base class is
+//// * returned instead.
+//// */
+//// @Override
+//// public String getServiceName() {
+////
+//// String s = server.getServiceName();
+////
+//// if (s == null)
+//// s = super.getServiceName();
+////
+//// return s;
+////
+//// }
//
-// if (s == null)
-// s = super.getServiceName();
-//
-// return s;
-//
-// }
+// }
- @Override
- public int getNSSPort() {
-
- final String COMPONENT = NSSConfigurationOptions.COMPONENT;
-
- try {
-
- final Integer port = (Integer) server.config.getEntry(
- COMPONENT, NSSConfigurationOptions.PORT, Integer.TYPE,
- NSSConfigurationOptions.DEFAULT_PORT);
-
- return port;
-
- } catch (ConfigurationException e) {
-
- throw new RuntimeException(e);
-
- }
-
- }
-
- @Override
- public RunState getRunState() {
-
- return server.getRunState();
-
- }
-
- @Override
- public String getExtendedRunState() {
-
- final HAQuorumService<HAGlue, HAJournal> quorumService = server.quorumService;
-
- final RunStateEnum innerRunState = (quorumService == null ? null
- : quorumService.runStateRef.get());
-
- final HAJournal journal = server.journal;
-
- final String innerRunStateStr = (innerRunState == null ? "N/A"
- : (innerRunState.name() + ((innerRunState == RunStateEnum.Resync && journal != null) ? (" @ " + journal
- .getRootBlockView().getCommitCounter()) : "")));
-
- return "{server=" + server.getRunState() + ", quorumService="
- + innerRunStateStr + "}";
-
- }
-
- }
-
}
Added: branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/MyHAJournal.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/MyHAJournal.java (rev 0)
+++ branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/MyHAJournal.java 2013-04-30 13:04:12 UTC (rev 7092)
@@ -0,0 +1,101 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Oct 31, 2012
+ */
+package com.bigdata.journal.jini.ha;
+
+import java.io.IOException;
+import java.rmi.Remote;
+import java.util.UUID;
+
+import org.apache.log4j.Logger;
+
+import net.jini.config.Configuration;
+import net.jini.config.ConfigurationException;
+
+import com.bigdata.ha.HAGlue;
+import com.bigdata.ha.QuorumService;
+import com.bigdata.quorum.Quorum;
+
+/**
+ * Class extends {@link HAJournal}.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ */
+public class MyHAJournal extends HAJournal {
+
+ private static final Logger log = Logger.getLogger(HAJournal.class);
+
+ public MyHAJournal(final HAJournalServer server,
+ final Configuration config,
+ final Quorum<HAGlue, QuorumService<HAGlue>> quorum)
+ throws ConfigurationException, IOException {
+
+ super(server, config, quorum);
+
+ }
+
+ @Override
+ protected HAGlue newHAGlue(final UUID serviceId) {
+
+// return super.newHAGlue(serviceId);
+ return new MyHAGlueService(serviceId);
+
+ }
+
+ /**
+ * A {@link Remote} interface for new methods published by the service.
+ */
+ public static interface MyHAGlue extends HAGlue {
+
+ public void helloWorld() throws IOException;
+
+ }
+
+ /**
+ * Class extends the public RMI interface of the {@link HAJournal}.
+ * <p>
+ * Note: Any new RMI methods must be (a) declared on an interface; and (b)
+ * must throw {@link IOException}.
+ */
+ protected class MyHAGlueService extends HAJournal.HAGlueService implements
+ MyHAGlue {
+
+ protected MyHAGlueService(final UUID serviceId) {
+
+ super(serviceId);
+
+ }
+
+ @Override
+ public void helloWorld() throws IOException {
+
+ log.warn("Hello world!");
+
+ }
+
+ }
+
+}
Modified: branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestAll.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestAll.java 2013-04-29 20:15:51 UTC (rev 7091)
+++ branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestAll.java 2013-04-30 13:04:12 UTC (rev 7092)
@@ -84,6 +84,9 @@
// Basic tests for a single HAJournalServer (quorum does not meet)
suite.addTestSuite(TestHAJournalServer.class);
+ // Verify ability to override the HAJournal implementation class.
+ suite.addTestSuite(TestHAJournalServerOverride.class);
+
// HA2 test suite (k=3, but only 2 services are running).
suite.addTestSuite(TestHA2JournalServer.class);
Added: branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHAJournalServerOverride.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHAJournalServerOverride.java (rev 0)
+++ branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHAJournalServerOverride.java 2013-04-30 13:04:12 UTC (rev 7092)
@@ -0,0 +1,126 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Oct 31, 2012
+ */
+package com.bigdata.journal.jini.ha;
+
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import net.jini.config.Configuration;
+
+import com.bigdata.ha.HAGlue;
+import com.bigdata.ha.HAStatusEnum;
+import com.bigdata.journal.jini.ha.MyHAJournal.MyHAGlue;
+
+/**
+ * Unit test of the ability to override the {@link HAJournal} implementation
+ * class.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ */
+public class TestHAJournalServerOverride extends AbstractHA3JournalServerTestCase {
+
+ public TestHAJournalServerOverride() {
+ }
+
+ public TestHAJournalServerOverride(final String name) {
+ super(name);
+ }
+
+ /**
+ * {@inheritDoc}
+ * <p>
+ * Note: This overrides some {@link Configuration} values for the
+ * {@link HAJournalServer} in order to establish conditions suitable for
+ * testing the {@link ISnapshotPolicy} and {@link IRestorePolicy}.
+ */
+ @Override
+ protected String[] getOverrides() {
+
+ return new String[]{
+ "com.bigdata.journal.jini.ha.HAJournalServer.restorePolicy=new com.bigdata.journal.jini.ha.DefaultRestorePolicy(0L,1,0)",
+ "com.bigdata.journal.jini.ha.HAJournalServer.snapshotPolicy=new com.bigdata.journal.jini.ha.NoSnapshotPolicy()",
+ "com.bigdata.journal.jini.ha.HAJournalServer.HAJournalClass=\"com.bigdata.journal.jini.ha.MyHAJournal\""
+ };
+
+ }
+
+ /**
+ * One service starts, quorum does not meet (replication factor is 3). This
+ * also serves to verify the <code>HAJournal-A.config</code> file.
+ */
+ public void testStartA() throws Exception {
+
+ final HAGlue serverA = startA();
+
+ try {
+
+ quorum.awaitQuorum(awaitQuorumTimeout, TimeUnit.MILLISECONDS);
+
+ fail("Not expecting quorum meet");
+
+ } catch (TimeoutException ex) {
+
+ // ignore.
+
+ }
+
+ // Service is not met in any role around a quorum.
+ try {
+ serverA.awaitHAReady(awaitQuorumTimeout, TimeUnit.MILLISECONDS);
+ } catch (TimeoutException ex) {
+ // Ignore expected exception.
+ }
+
+ // Verify can access the REST API "status" page.
+ doNSSStatusRequest(serverA);
+
+ // Verify self-reports as NotReady.
+ awaitHAStatus(serverA, HAStatusEnum.NotReady);
+
+ // Verify that service self-reports as NotReady via the REST API.
+ assertEquals(HAStatusEnum.NotReady, getNSSHAStatus(serverA));
+
+ // Verify can not read on service.
+ assertReadRejected(serverA);
+
+ // Verify can not write on service.
+ assertWriteRejected(serverA);
+
+ assertTrue(getHAJournalFileA().exists());
+ assertTrue(getHALogDirA().exists());
+ assertTrue(getSnapshotDirA().exists());
+
+ // Verify that we can invoke extension methods on the service.
+ ((MyHAGlue)serverA).helloWorld();
+
+// serverA.enterErrorState().get();
+//
+// Thread.sleep(10000/*ms*/);
+
+ }
+
+}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-04-29 20:15:58
|
Revision: 7091
http://bigdata.svn.sourceforge.net/bigdata/?rev=7091&view=rev
Author: thompsonbry
Date: 2013-04-29 20:15:51 +0000 (Mon, 29 Apr 2013)
Log Message:
-----------
Bug fix. I had introduced a bug in r7089 that caused the following two tests to fail:
{{{
com.bigdata.bop.join.TestJVMHashJoinOp.test_optionalJoin_withAltSink
com.bigdata.bop.join.TestJVMHashJoinOp.test_optionalJoin_and_constraint
}}}
The bug was in the semantics of the indexSolutionsHavingUnboundJoinVars flag. It was negated in the previous commit, which broke optional joins.
{{{
rightSolutionsRef.set(//
new JVMHashIndex(//
keyVars,//
indexSolutionsHavingUnboundJoinVars,//
new LinkedHashMap<Key, Bucket>(op.getProperty(
}}}
In addition, JVMHashIndex.makeKey() was hitting an NPE due to a failure to "continue" if a given key variable was not bound.
{{{
if (c == null) {
if (!indexSolutionsHavingUnboundJoinVars) {
/*
* Drop solution having an unbound join variable.
*/
if (log.isDebugEnabled())
log.debug("Join variable is not bound: var=" + v
+ ", solution=" + bset);
return null;
}
continue; // <== This line was missing.
}
}}}
See https://sourceforge.net/apps/trac/bigdata/ticket/668 (JoinGroup optimization).
Revision Links:
--------------
http://bigdata.svn.sourceforge.net/bigdata/?rev=7089&view=rev
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashIndex.java
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashIndex.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashIndex.java 2013-04-29 18:11:01 UTC (rev 7090)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashIndex.java 2013-04-29 20:15:51 UTC (rev 7091)
@@ -169,6 +169,8 @@
}
+ continue;
+
}
h = 31 * h + c.hashCode();
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java 2013-04-29 18:11:01 UTC (rev 7090)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java 2013-04-29 20:15:51 UTC (rev 7091)
@@ -325,7 +325,7 @@
rightSolutionsRef.set(//
new JVMHashIndex(//
keyVars,//
- !indexSolutionsHavingUnboundJoinVars,//
+ indexSolutionsHavingUnboundJoinVars,//
new LinkedHashMap<Key, Bucket>(op.getProperty(
HashMapAnnotations.INITIAL_CAPACITY,
HashMapAnnotations.DEFAULT_INITIAL_CAPACITY),//
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-04-29 18:11:07
|
Revision: 7090
http://bigdata.svn.sourceforge.net/bigdata/?rev=7090&view=rev
Author: thompsonbry
Date: 2013-04-29 18:11:01 +0000 (Mon, 29 Apr 2013)
Log Message:
-----------
Accidentially enabled the DISTINCT filter on the solutions flowing into the sub-group in SVN. I want to run performance tests on this before we enable this change in the code.
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java 2013-04-29 17:28:42 UTC (rev 7089)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java 2013-04-29 18:11:01 UTC (rev 7090)
@@ -838,7 +838,7 @@
* @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/668" >
* JoinGroup optimizations </a>
*/
- final boolean distinct = true;
+ final boolean distinct = false;
final IDistinctFilter distinctFilter;
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-04-29 17:28:53
|
Revision: 7089
http://bigdata.svn.sourceforge.net/bigdata/?rev=7089&view=rev
Author: thompsonbry
Date: 2013-04-29 17:28:42 +0000 (Mon, 29 Apr 2013)
Log Message:
-----------
I have lifted out the JVMHashIndex. This was an inner class of the JVMHashJoinUtility. It is now a top-level class in the same package.
Refactored makeKey() by lifting in the hashCode computation and removing the thrown exception when a join variable was not bound. This should reduce overhead associated with exception throws that could be handled through other means. One side-effect is that the hashCode is no longer always ONE for a solution having an unbound variable. Instead, we simply ignore that variable when computing the hash code but continue to compute the hash code based on the variables that are bound. Solutions with unbound variables are discovered during a post-join scan in which we identify the solutions that did not join. Thus, this should even help with OPTIONAL, NOT EXISTS, or MINUS joins when there are a large number of solutions that did not join by distributing them throughout the buckets rather than always in the bucket associated with the hash code ONE.
Rationalize the semantics of the "optional" and "joinVars" parameters for the JVMHashIndex. The joinVars has been relabeled the keyVars and merely indicates which variables are used to form the keys for the hash index. The optional flag indicates whether or not the key variables MUST be bound. When optional:=false, an unbound key variable is simply ignored.
Modified GroupNodeBase to output additional information (joinVars and projectInVars).
Extracted IDistinctFilter interface with both accept() and filterSolutions() methods. accept() is for solution at a time processing. filterSolutions() supports vectored processing. Modified JVMDistinctBindingSets and JVMDistinctFilter to use and support the new interface. I have started work on HTree version of this interface. The HTree version will be used to impose a DISTINCT filter in the solutions projected into a sub-group. The HTree version is more complex because of the vectoring optimizations that we perform for the HTree. There are various notes on this in HTreeHashJoinUtility.
All SPARQL and BOP related tests pass. The DISTINCT filter option is disabled in the committed code. One of the next steps is to determine whether the ASTComplexOptionalOptimizer is still required. The specific customer issue which motivated this ticket is fixed primarily by disabling that optimizer (it reduces the run time to less than 1 second).
The DISTINCT filter on the solutions flowing into the subgroup has much less of an impact on overall performance for that query. However, the DISTINCT filter does provide a significant performance gain for some queries that we have extracted from the original query as illustrated below.
# With DISTINCT filter in outputSolutions():
#
# Note: About 1/2 this if we use count(*) versus select(*)
#
# solutions=180953, chunks=1810, subqueries=0, elapsed=8997ms.
# solutions=180953, chunks=1810, subqueries=0, elapsed=8702ms.
#
# Without DISTINCT filter in outputSolutions():
#
# solutions=180953, chunks=1813, subqueries=0, elapsed=210856ms
These various issues are clearly worth continued pursuit.
See https://sourceforge.net/apps/trac/bigdata/ticket/668 (JoinGroup optimizations).
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashIndexOp.java
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/HashIndexOp.java
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMDistinctFilter.java
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashIndexOp.java
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMSolutionSetHashJoinOp.java
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/SolutionSetHashJoinOp.java
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/solutions/JVMDistinctBindingSetsOp.java
branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/GroupNodeBase.java
branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPORelation.java
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/IDistinctFilter.java
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashIndex.java
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashIndexOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashIndexOp.java 2013-04-28 15:03:41 UTC (rev 7088)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashIndexOp.java 2013-04-29 17:28:42 UTC (rev 7089)
@@ -53,7 +53,7 @@
HTreeHashJoinAnnotations {
}
-
+
/**
* Deep copy constructor.
*/
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java 2013-04-28 15:03:41 UTC (rev 7088)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/HTreeHashJoinUtility.java 2013-04-29 17:28:42 UTC (rev 7089)
@@ -40,6 +40,7 @@
import com.bigdata.bop.BOpContext;
import com.bigdata.bop.Constant;
import com.bigdata.bop.HTreeAnnotations;
+import com.bigdata.bop.HashMapAnnotations;
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.IConstant;
import com.bigdata.bop.IConstraint;
@@ -202,6 +203,14 @@
private final AtomicBoolean open = new AtomicBoolean(true);
/**
+ * The operator whose annotations are used to initialize this object.
+ * <p>
+ * Note: This was added to support the DISTINCT FILTER in
+ * {@link #outputSolutions(IBuffer)}.
+ */
+ private final PipelineOp op;
+
+ /**
* This basically controls the vectoring of the hash join.
*
* TODO parameter from operator annotations. Note that 10k tends to put too
@@ -259,6 +268,16 @@
private final IVariable<?>[] selectVars;
/**
+ * The variables to be projected into a join group. When non-
+ * <code>null</code> variables that are NOT in this array are NOT flowed
+ * into the join group.
+ *
+ * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/668" >
+ * JoinGroup optimizations </a>
+ */
+ private final IVariable<?>[] projectedInVars;
+
+ /**
* The join constraints (optional).
*/
private final IConstraint[] constraints;
@@ -344,6 +363,8 @@
if (askVar != null)
sb.append(",askVar=" + askVar);
sb.append(",joinVars=" + Arrays.toString(joinVars));
+ if (projectedInVars != null)
+ sb.append(",projectedInVars=" + Arrays.toString(projectedInVars));
if (selectVars != null)
sb.append(",selectVars=" + Arrays.toString(selectVars));
if (constraints != null)
@@ -497,7 +518,7 @@
if(joinType == null)
throw new IllegalArgumentException();
-// this.op = op;
+ this.op = op;
this.joinType = joinType;
this.optional = joinType == JoinTypeEnum.Optional;
this.filter = joinType == JoinTypeEnum.Filter;
@@ -516,6 +537,12 @@
.getProperty(JoinAnnotations.SELECT);
/*
+ * The variables that are projected IN to the join group.
+ */
+ this.projectedInVars = (IVariable<?>[]) op
+ .getProperty(HashJoinAnnotations.PROJECT_IN_VARS);
+
+ /*
* This wraps an efficient raw store interface around a child memory
* manager created from the IMemoryManager which will back the named
* solution set.
@@ -1524,82 +1551,179 @@
}
+ /**
+ * DISTINCT solutions filter for
+ * {@link HTreeHashJoinUtility#outputSolutions(IBuffer)}
+ *
+ * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/668" >
+ * JoinGroup optimizations </a>
+ */
+ private class HTreeDistinctFilter implements IDistinctFilter {
+
+ /**
+ * The variables used to impose a distinct constraint.
+ */
+ private final IVariable<?>[] vars;
+
+ private final HTreeHashJoinUtility state;
+
+ public HTreeDistinctFilter(final IVariable<?>[] vars, final PipelineOp op) {
+
+ this.vars = vars;
+
+ this.state = new HTreeHashJoinUtility(
+ ((MemStore) store).getMemoryManager(), op,
+ JoinTypeEnum.Filter);
+
+ }
+
+ @Override
+ public IVariable<?>[] getProjectedVars() {
+
+ return vars;
+
+ }
+
+ @Override
+ public IBindingSet accept(final IBindingSet bset) {
+ // FIXME Auto-generated method stub
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public long filterSolutions(ICloseableIterator<IBindingSet[]> itr,
+ BOpStats stats, IBuffer<IBindingSet> sink) {
+ // FIXME Auto-generated method stub
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void release() {
+
+ state.release();
+
+ }
+
+ }
+
@Override
public void outputSolutions(final IBuffer<IBindingSet> out) {
try {
-// if (false) {
-//
-// /*
-// * Striterator pattern.
-// */
-//
-// final ICloseableIterator<IBindingSet> itr = indexScan();
-//
-// try {
-//
-// while(itr.hasNext()) {
-//
-// IBindingSet bset = itr.next();
-//
-// if (selectVars != null) {
-//
-// // Drop variables which are not projected.
-// bset = bset.copy(selectVars);
-//
-// }
-// out.add(bset);
-//
-// }
-//
-// } finally {
-//
-// itr.close();
-//
-// }
-//
-//
-// } else {
+ /*
+ * FIXME Set this to enable "DISTINCT" on the solutions flowing into the
+ * join group.
+ *
+ * Note: This should be set by the HashIndexOp (or passed in through the
+ * interface).
+ *
+ * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/668" >
+ * JoinGroup optimizations </a>
+ */
+ final boolean distinct = false;
+
+ /*
+ * FIXME Replace with an HTreeDistinctFilter and integrate to NOT
+ * flow duplicate solutions into the sub-group. The HTree
+ * filterSolutions() method needs to be vectored to be efficient.
+ * Therefore, this outputSolutions() method needs to be rewritten to
+ * be vectored as well. It is efficient in reading the solutions
+ * from the HTree, and the solutions are in the "natural" order of
+ * the HTree for the join vars. This order SHOULD be pretty
+ * efficient for the DISTINCT solutions set as well, but note that
+ * joinVars:=projectedInVars. To maximize the corrleation, both the
+ * joinVars[] and the projectedInVars[] should be sorted so the
+ * variables in the solutions will be correllated and any variables
+ * that are NOT in the projectedInVars should appear towards the end
+ * of the joinVars where they will cause the least perturbation in
+ * this scan + filter.
+ */
+ final IDistinctFilter distinctFilter;
+
+ if (distinct && projectedInVars != null && projectedInVars.length > 0) {
/*
- * Simple iterator pattern.
+ * Note: We are single threaded here so we can use a lower
+ * concurrencyLevel value.
+ *
+ * Note: If necessary, this could be replaced with JVMHashIndex so
+ * we get the #of occurrences of each distinct combination of
+ * bindings that is projected into the sub-group/-query.
*/
+ final int concurrencyLevel = 1;//ConcurrentHashMapAnnotations.DEFAULT_CONCURRENCY_LEVEL;
+
+ distinctFilter = new JVMDistinctFilter(projectedInVars, //
+ op.getProperty(HashMapAnnotations.INITIAL_CAPACITY,
+ HashMapAnnotations.DEFAULT_INITIAL_CAPACITY),//
+ op.getProperty(HashMapAnnotations.LOAD_FACTOR,
+ HashMapAnnotations.DEFAULT_LOAD_FACTOR),//
+ concurrencyLevel
+ );
- final HTree rightSolutions = getRightSolutions();
+ } else {
+
+ distinctFilter = null;
+
+ }
+
+ final HTree rightSolutions = getRightSolutions();
- if (log.isInfoEnabled()) {
- log.info("rightSolutions: #nnodes="
- + rightSolutions.getNodeCount() + ",#leaves="
- + rightSolutions.getLeafCount() + ",#entries="
- + rightSolutions.getEntryCount());
- }
+ if (log.isInfoEnabled()) {
+ log.info("rightSolutions: #nnodes="
+ + rightSolutions.getNodeCount() + ",#leaves="
+ + rightSolutions.getLeafCount() + ",#entries="
+ + rightSolutions.getEntryCount());
+ }
- // source.
- final ITupleIterator<?> solutionsIterator = rightSolutions
- .rangeIterator();
+ // source.
+ final ITupleIterator<?> solutionsIterator = rightSolutions
+ .rangeIterator();
- while (solutionsIterator.hasNext()) {
+ while (solutionsIterator.hasNext()) {
- final ITuple<?> t = solutionsIterator.next();
+ final ITuple<?> t = solutionsIterator.next();
- IBindingSet bset = decodeSolution(t);
+ IBindingSet bset = decodeSolution(t);
- if (selectVars != null) {
+ if (distinctFilter != null) {
- // Drop variables which are not projected.
- bset = bset.copy(selectVars);
+ /*
+ * Note: The DISTINCT filter is based on the variables
+ * that are projected INTO the child join group.
+ * However, those are NOT always the same as the
+ * variables that are projected OUT of the child join
+ * group, so we need to
+ */
+ if ((bset = distinctFilter.accept(bset)) == null) {
+
+ // Drop duplicate solutions.
+ continue;
+
}
- encoder.resolveCachedValues(bset);
+ } else if (selectVars != null) {
- out.add(bset);
+ /*
+ * FIXME We should be using projectedInVars here since
+ * outputSolutions() is used to stream solutions into
+ * the child join group (at least for some kinds of
+ * joins, but there might be exceptions for joining with
+ * a named solution set).
+ */
+ // Drop variables which are not projected.
+ bset = bset.copy(selectVars);
+
}
-// }
+ encoder.resolveCachedValues(bset);
+ out.add(bset);
+
+ }
+
} catch (Throwable t) {
throw launderThrowable(t);
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/HashIndexOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/HashIndexOp.java 2013-04-28 15:03:41 UTC (rev 7088)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/HashIndexOp.java 2013-04-29 17:28:42 UTC (rev 7089)
@@ -114,8 +114,7 @@
* @param args
* @param annotations
*/
- public HashIndexOp(final BOp[] args,
- final Map<String, Object> annotations) {
+ public HashIndexOp(final BOp[] args, final Map<String, Object> annotations) {
super(args, annotations);
Added: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/IDistinctFilter.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/IDistinctFilter.java (rev 0)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/IDistinctFilter.java 2013-04-29 17:28:42 UTC (rev 7089)
@@ -0,0 +1,81 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Apr 29, 2013
+ */
+package com.bigdata.bop.join;
+
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.IVariable;
+import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.relation.accesspath.IBuffer;
+import com.bigdata.striterator.ICloseableIterator;
+
+/**
+ * A "DISTINCT" filter for {@link IBindingSet}s.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ */
+public interface IDistinctFilter {
+
+ /**
+ * The variables that are being projected out of the DISTINCT filter. The
+ * solutions will be DISTINCT on this combination of variables. Bindings on
+ * other variables will be dropped.
+ */
+ IVariable<?>[] getProjectedVars();
+
+ /**
+ * If the bindings are distinct for the configured variables then return a
+ * new {@link IBindingSet} consisting of only the selected variables.
+ *
+ * @param bset
+ * The binding set to be filtered.
+ *
+ * @return A new {@link IBindingSet} containing only the distinct as bound
+ * values -or- <code>null</code> if the binding set duplicates a
+ * solution which was already accepted.
+ */
+ IBindingSet accept(final IBindingSet bset);
+
+ /**
+ * Vectored DISTINCT.
+ *
+ * @param itr
+ * The source solutions.
+ * @param stats
+ * Statistics object to be updated.
+ * @param sink
+ * The sink onto which the DISTINCT solutions will be written.
+ * @return The #of DISTINCT solutions.
+ */
+ long filterSolutions(final ICloseableIterator<IBindingSet[]> itr,
+ final BOpStats stats, final IBuffer<IBindingSet> sink);
+
+ /**
+ * Discard the map backing this filter.
+ */
+ void release();
+
+}
\ No newline at end of file
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMDistinctFilter.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMDistinctFilter.java 2013-04-28 15:03:41 UTC (rev 7088)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMDistinctFilter.java 2013-04-29 17:28:42 UTC (rev 7089)
@@ -35,6 +35,9 @@
import com.bigdata.bop.IConstant;
import com.bigdata.bop.IVariable;
import com.bigdata.bop.bindingSet.ListBindingSet;
+import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.relation.accesspath.IBuffer;
+import com.bigdata.striterator.ICloseableIterator;
/**
* Utility class for imposing a DISTINCT filter on {@link IBindingSet}. This
@@ -42,7 +45,7 @@
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
*/
-public class JVMDistinctFilter {
+public class JVMDistinctFilter implements IDistinctFilter {
private static final Logger log = Logger.getLogger(JVMDistinctFilter.class);
@@ -107,7 +110,7 @@
* The set of variables on which the DISTINCT filter will be
* imposed. Only these variables will be present in the
* "accepted" solutions. Any variable bindings not specified in
- * this array will be dropped).
+ * this array will be dropped.
* @param initialCapacity
* @param loadFactor
* @param concurrencyLevel
@@ -129,6 +132,23 @@
}
+ /* (non-Javadoc)
+ * @see com.bigdata.bop.join.IDistinctFilter#clear()
+ */
+ @Override
+ public void release() {
+
+ map.clear();
+
+ }
+
+ @Override
+ public IVariable<?>[] getProjectedVars() {
+
+ return vars;
+
+ }
+
/**
* If the bindings are distinct for the configured variables then return
* those bindings.
@@ -139,7 +159,7 @@
* @return The distinct as bound values -or- <code>null</code> if the
* binding set duplicates a solution which was already accepted.
*/
- public IConstant<?>[] accept(final IBindingSet bset) {
+ private IConstant<?>[] _accept(final IBindingSet bset) {
final IConstant<?>[] r = new IConstant<?>[vars.length];
@@ -168,20 +188,13 @@
}
- /**
- * If the bindings are distinct for the configured variables then return a
- * new {@link IBindingSet} consisting of only the selected variables.
- *
- * @param bset
- * The binding set to be filtered.
- *
- * @return A new {@link IBindingSet} containing only the distinct as bound
- * values -or- <code>null</code> if the binding set duplicates a
- * solution which was already accepted.
+ /* (non-Javadoc)
+ * @see com.bigdata.bop.join.IDistinctFilter#accept(com.bigdata.bop.IBindingSet)
*/
- public IBindingSet accept2(final IBindingSet bset) {
+ @Override
+ public IBindingSet accept(final IBindingSet bset) {
- final IConstant<?>[] vals = accept(bset);
+ final IConstant<?>[] vals = _accept(bset);
if (vals == null) {
@@ -212,13 +225,45 @@
}
- /**
- * Discard the map backing this filter.
- */
- public void clear() {
+ @Override
+ public long filterSolutions(final ICloseableIterator<IBindingSet[]> itr,
+ final BOpStats stats, final IBuffer<IBindingSet> sink) {
- map.clear();
+ long n = 0L;
+ while (itr.hasNext()) {
+
+ final IBindingSet[] a = itr.next();
+
+ stats.chunksIn.increment();
+ stats.unitsIn.add(a.length);
+
+ for (IBindingSet bset : a) {
+
+ /*
+ * Test to see if this solution is distinct from those already
+ * seen.
+ */
+ if ((bset = accept(bset)) == null) {
+
+ // Drop duplicate solution.
+ continue;
+ }
+
+ /*
+ * This is a distinct solution.
+ */
+
+ sink.add(bset);
+
+ n++;
+
+ }
+
+ } // next chunk.
+
+ return n;
+
}
}
Added: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashIndex.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashIndex.java (rev 0)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashIndex.java 2013-04-29 17:28:42 UTC (rev 7089)
@@ -0,0 +1,692 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Apr 29, 2013
+ */
+package com.bigdata.bop.join;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.log4j.Logger;
+
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.IConstant;
+import com.bigdata.bop.IVariable;
+import com.bigdata.bop.solutions.JVMDistinctBindingSetsOp;
+import com.bigdata.counters.CAT;
+
+/**
+ * A hash index for {@link IBindingSet}s that supports duplicate solutions and
+ * hit counts. The hit counts are used to detect {@link IBindingSet}s that do
+ * not join for OPTIONAL, MINUS, and related kinds of "negation" joins.
+ * <p>
+ * Note: The {@link JVMDistinctBindingSetsOp} does not use this class right now
+ * because it enjoys better concurrency than the {@link JVMHashIndex}. Also see
+ * {@link JVMDistinctFilter}, which is the backing implementation for the
+ * {@link JVMDistinctBindingSetsOp}.
+ *
+ * @see JVMDistinctFilter
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ */
+public class JVMHashIndex {
+
+ private static final Logger log = Logger.getLogger(JVMHashIndex.class);
+
+ /**
+ * Note: If joinVars is an empty array, then the solutions will all hash to
+ * ONE (1).
+ */
+ private static final int ONE = 1;
+
+// /**
+// * Return the hash code which will be used as the key given the ordered
+// * as-bound values for the join variables.
+// *
+// * @param joinVars
+// * The join variables.
+// * @param bset
+// * The bindings whose as-bound hash code for the join variables
+// * will be computed.
+// * @param ignoreUnboundVariables
+// * If a variable without a binding should be silently ignored.
+// *
+// * @return The hash code.
+// *
+// * @throws JoinVariableNotBoundException
+// * if there is no binding for a join variable.
+// */
+// private static int hashCode(final IVariable<?>[] joinVars,
+// final IBindingSet bset, final boolean ignoreUnboundVariables)
+// throws JoinVariableNotBoundException {
+//
+// int h = ONE;
+//
+// for (IVariable<?> v : joinVars) {
+//
+// final IConstant<?> c = bset.get(v);
+//
+// if (c == null) {
+//
+// if (ignoreUnboundVariables)
+// continue;
+//
+// // Reject any solution which does not have a binding for a join
+// // variable.
+//
+// throw new JoinVariableNotBoundException(v.getName());
+//
+// }
+//
+// h = 31 * h + c.hashCode();
+//
+// }
+//
+// if (log.isTraceEnabled())
+// log.trace("hashCode=" + h + ", joinVars="
+// + Arrays.toString(joinVars) + " : " + bset);
+//
+// return h;
+//
+// }
+
+ /**
+ * Return an array of constants corresponding to the as-bound values of the
+ * join variables for the given solution.
+ *
+ * @param bset
+ * The solution.
+ *
+ * @return The as-bound values for the {@link #keyVars} for that solution
+ * -or- <code>null</code> if one or more join variables is not bound
+ * by the solution and {@link #indexSolutionsHavingUnboundJoinVars}
+ * is <code>false</code>.
+ *
+ * @see #keyVars
+ * @see #indexSolutionsHavingUnboundJoinVars
+ */
+ private Key makeKey(//final IVariable<?>[] keyVars,
+ final IBindingSet bset
+// final boolean indexSolutionsHavingUnboundJoinVars
+ ) {
+
+ final IConstant<?>[] vals = new IConstant<?>[keyVars.length];
+
+ for (int i = 0; i < keyVars.length; i++) {
+
+ final IVariable<?> v = keyVars[i];
+
+ vals[i] = bset.get(v);
+
+ }
+
+ int h = ONE;
+
+ for (IVariable<?> v : keyVars) {
+
+ final IConstant<?> c = bset.get(v);
+
+ if (c == null) {
+
+ if (!indexSolutionsHavingUnboundJoinVars) {
+
+ /*
+ * Drop solution having an unbound join variable.
+ */
+
+ if (log.isDebugEnabled())
+ log.debug("Join variable is not bound: var=" + v
+ + ", solution=" + bset);
+
+ return null;
+
+ }
+
+ }
+
+ h = 31 * h + c.hashCode();
+
+ }
+
+ if (log.isTraceEnabled())
+ log.trace("hashCode=" + h + ", joinVars="
+ + Arrays.toString(keyVars) + " : " + bset);
+
+ return new Key(h, vals);
+
+ }
+
+ /**
+ * Wrapper for the keys in the hash table. This is necessary for the hash
+ * table to compare the keys as equal and also provides efficiencies in the
+ * hash code and equals() methods.
+ */
+ public static class Key {
+
+ private final int hash;
+
+ private final IConstant<?>[] vals;
+
+ private Key(final int hashCode, final IConstant<?>[] vals) {
+ this.vals = vals;
+ this.hash = hashCode;
+ }
+
+ public int hashCode() {
+ return hash;
+ }
+
+ public boolean equals(final Object o) {
+ if (this == o)
+ return true;
+ if (!(o instanceof Key)) {
+ return false;
+ }
+ final Key t = (Key) o;
+ if (vals.length != t.vals.length)
+ return false;
+ for (int i = 0; i < vals.length; i++) {
+ if (vals[i] == t.vals[i])
+ continue;
+ if (vals[i] == null)
+ return false;
+ if (!vals[i].equals(t.vals[i]))
+ return false;
+ }
+ return true;
+ }
+ }
+
+ /**
+ * An solution and a hit counter as stored in the {@link JVMHashIndex}.
+ */
+ public static class SolutionHit {
+
+ /**
+ * The input solution.
+ */
+ final public IBindingSet solution;
+
+ /**
+ * The #of hits on that solution. This may be used to detect solutions
+ * that did not join. E.g., by scanning and reporting out all solutions
+ * where {@link #nhits} is ZERO (0L).
+ */
+ public final CAT nhits = new CAT();
+
+ private SolutionHit(final IBindingSet solution) {
+
+ if (solution == null)
+ throw new IllegalArgumentException();
+
+ this.solution = solution;
+
+ }
+
+ public String toString() {
+
+ return getClass().getName() + "{nhits=" + nhits + ",solution="
+ + solution + "}";
+
+ }
+
+ } // class SolutionHit
+
+ /**
+ * A group of solutions having the same as-bound values for the join vars.
+ * Each solution is paired with a hit counter so we can support OPTIONAL
+ * semantics for the join.
+ */
+ public static class Bucket implements Iterable<SolutionHit>,
+ Comparable<Bucket> {
+
+ /** The hash code for this collision bucket. */
+ private final int hashCode;
+
+ /**
+ * A set of solutions (and their hit counters) which have the same
+ * as-bound values for the join variables.
+ */
+ private final List<SolutionHit> solutions = new LinkedList<SolutionHit>();
+
+ public String toString() {
+ return super.toString()
+ + //
+ "{hashCode=" + hashCode + ",#solutions=" + solutions.size()
+ + "}";
+ }
+
+ public Bucket(final int hashCode, final IBindingSet solution) {
+
+ this.hashCode = hashCode;
+
+ add(solution);
+
+ }
+
+ public void add(final IBindingSet solution) {
+
+ if (solution == null)
+ throw new IllegalArgumentException();
+
+ solutions.add(new SolutionHit(solution));
+
+ }
+
+ /**
+ * Add the solution to the bucket iff the solutions is not already
+ * present in the bucket.
+ * <p>
+ * Note: There is already a hash index in place on the join variables
+ * when we are doing a DISTINCT filter. Further, only the "join"
+ * variables are "selected" and participate in a DISTINCT filter.
+ * Therefore, if we have a hash collision such that two solutions would
+ * be directed into the same {@link Bucket} then we can not improve
+ * matters but must simply scan the solutions in the bucket to decide
+ * whether the new solution duplicates a solution which is already
+ * present.
+ *
+ * @param solution
+ * The solution.
+ *
+ * @return <code>true</code> iff the bucket was modified by this
+ * operation.
+ */
+ public boolean addDistinct(final IBindingSet solution) {
+
+ if (solutions.isEmpty()) {
+
+ // First solution.
+ solutions.add(new SolutionHit(solution));
+
+ return true;
+
+ }
+
+ final Iterator<SolutionHit> itr = solutions.iterator();
+
+ while (itr.hasNext()) {
+
+ final SolutionHit aSolution = itr.next();
+
+ if (aSolution.solution.equals(solution)) {
+
+ // Solution already in this bucket.
+ return false;
+
+ }
+
+ }
+
+ // This is a distinct solution.
+ solutions.add(new SolutionHit(solution));
+
+ return true;
+
+ }
+
+ final public Iterator<SolutionHit> iterator() {
+
+ // return Collections.unmodifiableList(solutions).iterator();
+ return solutions.iterator();
+
+ }
+
+ // @SuppressWarnings("unchecked")
+ // public Iterator<IBindingSet> bindingSetIterator() {
+ //
+ // return new Striterator(solutions.iterator()).addFilter(new Resolver()
+ // {
+ //
+ // @Override
+ // protected Object resolve(Object obj) {
+ // return ((SolutionHit)obj).solution;
+ // }
+ // });
+ //
+ // }
+
+ /**
+ * Orders the buckets based on their hash codes.
+ */
+ @Override
+ final public int compareTo(final Bucket o) {
+ if (hashCode > o.hashCode)
+ return 1;
+ if (hashCode < o.hashCode)
+ return -1;
+ return 0;
+ }
+
+ @Override
+ final public int hashCode() {
+
+ return hashCode;
+
+ }
+
+ /**
+ * Return <code>true</code> iff this {@link Bucket} is empty (if there
+ * are no solutions in the bucket).
+ */
+ final public boolean isEmpty() {
+
+ return solutions.isEmpty();
+
+ }
+
+ } // Bucket
+
+ /**
+ * The join variables (required, but may be empty). The order of the entries
+ * is used when forming the as-bound keys for the hash table. Duplicate
+ * elements and null elements are not permitted. If no join variables are
+ * specified, then the join will consider the N x M cross product, filtering
+ * for solutions which join. This is very expensive when compared to a hash
+ * join. Whenever possible you should identify one or more variables which
+ * must be bound for the join and specify those as the join variables.
+ */
+ private final IVariable<?>[] keyVars;
+
+ /**
+ * When <code>true</code>, we allow solutions to be stored in the hash index
+ * that have unbound variables for the {@link #keyVars}. When
+ * <code>false</code>, such solutions are dropped.
+ * <p>
+ * Note: This must be <code>true</code> for DISTINCT, OPTIONAL, and NOT
+ * EXISTS / MINUS since in each case we do not want to drop solutions
+ * lacking a binding for some {@link #keyVars}. For DISTINCT, this is
+ * because we want to project all solutions, regardless of unbound
+ * variables. For OPTIONAL and NOT EXISTS / MINUS, this is because we must
+ * index all solutions since we will report only those solutions that do not
+ * join. Once all solutions that do join have been identified, the solutions
+ * that do not join are identified by a scan of the hash index looking for
+ * {@link SolutionHit#nhits} equals ZERO (0L).
+ */
+ private final boolean indexSolutionsHavingUnboundJoinVars;
+
+ /**
+ * The backing map - this is NOT thread safe.
+ */
+ private final Map<Key, Bucket> map;
+
+ /**
+ * @param keyVars
+ * The variables that are used to form the keys in the hash index
+ * (required, but may be empty). The order of the entries is used
+ * when forming the as-bound keys for the hash table. Duplicate
+ * elements and null elements are not permitted. If no join
+ * variables are specified, then the join will consider the N x M
+ * cross product, filtering for solutions which join. This is
+ * very expensive when compared to a hash join. Whenever possible
+ * you should identify one or more variables which must be bound
+ * for the join and specify those as the join variables.
+ * @param indexSolutionsHavingUnboundJoinVars
+ * When <code>true</code>, we allow solutions to be stored in the
+ * hash index that have unbound variables for the
+ * {@link #keyVars}. When <code>false</code>, such solutions are
+ * dropped (they are not added to the index).
+ * @param map
+ * The backing map. A {@link HashMap} should be faster for insert
+ * and search. A {@link LinkedHashMap} should be faster for
+ * scans. Some join patterns do not require us to use scans, in
+ * which case {@link HashMap} is the clear winner. (For example,
+ * a non-optional hash join against an access path never uses the
+ * iterator over the hash index.)
+ */
+ public JVMHashIndex(final IVariable<?>[] keyVars,
+ final boolean indexSolutionsHavingUnboundJoinVars,
+ final Map<Key, Bucket> map) {
+
+ if (keyVars == null) {
+
+ /*
+ * A ZERO LENGTH joinVars[] means that all solutions will be in the
+ * same hash bucket. This can arise due to poor assignment of join
+ * variables or simply because there are no available join variables
+ * (full cross product join). Such joins are very expensive.
+ */
+
+ throw new IllegalArgumentException();
+
+ }
+
+ if (map == null) {
+
+ throw new IllegalArgumentException();
+
+ }
+
+ this.map = map;
+
+ this.indexSolutionsHavingUnboundJoinVars = indexSolutionsHavingUnboundJoinVars;
+
+ this.keyVars = keyVars;
+
+ }
+
+ /**
+ * Add the solution to the index.
+ *
+ * @param bset
+ * The {@link IBindingSet}.
+ *
+ * @return The {@link Key} iff the solution was added to the index and
+ * <code>null</code> iff the solution was not added (because a
+ * {@link Key} could not be formed for the solution given the
+ * specified {@link #keyVars}).
+ */
+ public Key add(final IBindingSet bset) {
+
+ final Key key = makeKey(bset);
+
+ if (key == null) {
+
+ // Drop solution.
+ return null;
+
+ }
+
+ /*
+ * TODO There is an opportunity for CONCURRENT hash map for at least the
+ * DISTINCT SOLUTIONS filter and perhaps for others as well. However, to
+ * do this with the DISTINCT SOLUTIONS filter we would have to make the
+ * mutation operations on a Bucket atomic. E.g., using the synchronized
+ * keyword. This would give us what amounts to per-hash code striped
+ * locks.
+ *
+ * Note: This pattern could be made thread safe. If the get() fails, use
+ * a putIfAbsent() in a data race to create and insert the new bucket.
+ * If the thread looses the data race, then it must use the other
+ * thread's bucket and add its solution to that bucket.
+ *
+ * The Bucket.addDistinct() could also be made thread safe by using the
+ * monitor for the Bucket (or its Solutions List). This is necessary for
+ * correctness, but note that we do not use addDistinct() and instead
+ * rely on the more efficient JVMDistinctFilter. The JVMDistinctFilter
+ * it is more efficient because it based on a ConcurrentHashMap does not
+ * require any explicit synchronization.
+ *
+ * TODO This change would allow us to execute the JVMHashIndexOp
+ * concurrently which could provide a substantial throughput gain.
+ * However, we still are faced with the requirement to decide atomically
+ * when the HashIndexOp is done (the isLastPass() test). It is possible
+ * to decide when no more solutions will be available. If the thread
+ * that executes the last pass awaits a latch to count down to ONE, then
+ * it will known that it is (a) the last invocation, and (b) that all
+ * other invocations are complete. This pattern would have to be
+ * supported in the QueryEngine and PipelineOp since the latch would
+ * have to be incremented by the QueryEngine in a critical section when
+ * the new ChunkTask is created and then decremented in a critical
+ * section when the ChunkTask ends. If the latch is then exposed to the
+ * BOpContext, the operator can decide that it is the last invocation
+ * and that no other task is running (or will run) for that operator and
+ * then execute the post-processing step (flooding the solutions in the
+ * hash index to the downstream operator in the query plan). [Actually,
+ * we might not have to do that for the JVMHashIndexOp since we do not
+ * have to checkpoint the JVMHashIndex and could incrementally pass
+ * along the indexed solutions to the downstream operator, but this
+ * would also mean that outputSolutions() would need to use sharedState
+ * for its DISTINCT FILTER on the solutions flowing into the sub-group.
+ * All of this could be done, but it might require us to specialize the
+ * JVMHashIndexOp. We would also have to update AST2BOpUtility to
+ * generate the appropriate annotations.]
+ */
+ Bucket b = map.get(key);
+
+ if (b == null) {
+
+ map.put(key, b = new Bucket(key.hash, bset));
+
+ } else {
+
+ b.add(bset);
+
+ }
+
+ return key;
+
+ }
+
+ /**
+ * Add the solution to the index iff the solution is not already present in
+ * the index.
+ *
+ * @param bset
+ * The solution.
+ *
+ * @return <code>true</code> iff the index was modified by this operation.
+ */
+ public boolean addDistinct(final IBindingSet bset) {
+
+ final Key key = makeKey(bset);
+
+ assert key != null;
+
+ Bucket b = map.get(key);
+
+ if (b == null) {
+
+ // New bucket holding just this solution.
+ map.put(key, b = new Bucket(key.hash, bset));
+
+ return true;
+
+ } else {
+
+ if (b.addDistinct(bset)) {
+
+ // Existing bucket not having this solution.
+ return true;
+
+ }
+
+ // Existing bucket with duplicate solution.
+ return false;
+
+ }
+
+ }
+
+ /**
+ * Return the hash {@link Bucket} into which the given solution is mapped.
+ * <p>
+ * Note: The caller must apply an appropriate join constraint in order to
+ * correctly reject solutions that (a) violate the join contract; and (b)
+ * that are present in the hash bucket due to a hash collection rather than
+ * because they have the same bindings for the join variables.
+ *
+ * @param left
+ * The probe.
+ *
+ * @return The hash {@link Bucket} into which the given solution is mapped
+ * -or- <code>null</code> if there is no such hash bucket.
+ */
+ public Bucket getBucket(final IBindingSet left) {
+
+ final Key key = makeKey(left);
+
+ if (key == null) {
+
+ return null;
+
+ }
+
+ // Probe the hash map : May return [null]!
+ return map.get(key);
+
+ }
+
+ /**
+ * Visit all buckets in the hash index.
+ */
+ public Iterator<Bucket> buckets() {
+
+ return map.values().iterator();
+
+ }
+
+ /**
+ * The #of buckets in the hash index. Each bucket has a distinct hash code.
+ * Hash collisions can cause solutions that are distinct in their
+ * {@link #keyVars} to nevertheless be mapped into the same hash bucket.
+ *
+ * @return The #of buckets in the hash index.
+ */
+ public int bucketCount() {
+
+ return map.size();
+
+ }
+
+ /**
+ * Export the {@link Bucket}s as an array.
+ */
+ public Bucket[] toArray() {
+
+ // source.
+ final Iterator<Bucket> bucketIterator = map.values()
+ .iterator();
+
+ final Bucket[] a = new Bucket[map.size()];
+
+ int i = 0;
+
+ while (bucketIterator.hasNext()) {
+
+ a[i++] = bucketIterator.next();
+
+ }
+
+ return a;
+
+ }
+
+}
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashIndexOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashIndexOp.java 2013-04-28 15:03:41 UTC (rev 7088)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashIndexOp.java 2013-04-29 17:28:42 UTC (rev 7089)
@@ -53,7 +53,7 @@
JVMHashJoinAnnotations {
}
-
+
/**
* Deep copy constructor.
*/
@@ -72,55 +72,6 @@
super(args, annotations);
-// switch (getEvaluationContext()) {
-// case CONTROLLER:
-// case SHARDED:
-// case HASHED:
-// break;
-// default:
-// throw new UnsupportedOperationException(
-// BOp.Annotations.EVALUATION_CONTEXT + "="
-// + getEvaluationContext());
-// }
-//
-// if (getMaxParallel() != 1) {
-// /*
-// * Parallel evaluation is not allowed. This operator writes on an
-// * object that is not thread-safe for mutation.
-// */
-// throw new IllegalArgumentException(
-// PipelineOp.Annotations.MAX_PARALLEL + "="
-// + getMaxParallel());
-// }
-//
-// if (!isLastPassRequested()) {
-// /*
-// * Last pass evaluation must be requested. This operator will not
-// * produce any outputs until all source solutions have been
-// * buffered.
-// */
-// throw new IllegalArgumentException(PipelineOp.Annotations.LAST_PASS
-// + "=" + isLastPassRequested());
-// }
-//
-// getRequiredProperty(Annotations.NAMED_SET_REF);
-//
-// @SuppressWarnings("unused")
-// final JoinTypeEnum joinType = (JoinTypeEnum) getRequiredProperty(Annotations.JOIN_TYPE);
-//
-// // Join variables must be specified.
-// final IVariable<?>[] joinVars = (IVariable[]) getRequiredProperty(Annotations.JOIN_VARS);
-//
-//// if (joinVars.length == 0)
-//// throw new IllegalArgumentException(Annotations.JOIN_VARS);
-//
-// for (IVariable<?> var : joinVars) {
-//
-// if (var == null)
-// throw new IllegalArgumentException(Annotations.JOIN_VARS);
-//
-// }
-
}
public JVMHashIndexOp(final BOp[] args, NV... annotations) {
@@ -128,13 +79,6 @@
this(args, NV.asMap(annotations));
}
-
-// @Override
-// public BOpStats newStats() {
-//
-// return new NamedSolutionSetStats();
-//
-// }
@Override
protected JVMHashJoinUtility newState(
@@ -145,151 +89,4 @@
}
-// public FutureTask<Void> eval(final BOpContext<IBindingSet> context) {
-//
-// return new FutureTask<Void>(new ControllerTask(this, context));
-//
-// }
-
-// /**
-// * Evaluates the subquery for each source binding set. If the controller
-// * operator is interrupted, then the subqueries are cancelled. If a subquery
-// * fails, then all subqueries are cancelled.
-// */
-// private static class ControllerTask implements Callable<Void> {
-//
-// private final BOpContext<IBindingSet> context;
-//
-// private final JVMHashIndexOp op;
-//
-// private final NamedSolutionSetStats stats;
-//
-// private final IHashJoinUtility state;
-//
-// public ControllerTask(final JVMHashIndexOp op,
-// final BOpContext<IBindingSet> context) {
-//
-// if (op == null)
-// throw new IllegalArgumentException();
-//
-// if (context == null)
-// throw new IllegalArgumentException();
-//
-// this.context = context;
-//
-// this.op = op;
-//
-// this.stats = ((NamedSolutionSetStats) context.getStats());
-//
-// // Metadata to identify the named solution set.
-// final NamedSolutionSetRef namedSetRef = (NamedSolutionSetRef) op
-// .getRequiredProperty(Annotations.NAMED_SET_REF);
-//
-// {
-//
-// /*
-// * First, see if the map already exists.
-// *
-// * Note: Since the operator is not thread-safe, we do not need
-// * to use a putIfAbsent pattern here.
-// */
-//
-// // Lookup the attributes for the query on which we will hang the
-// // solution set.
-// final IQueryAttributes attrs = context
-// .getQueryAttributes(namedSetRef.queryId);
-//
-// JVMHashJoinUtility state = (JVMHashJoinUtility) attrs
-// .get(namedSetRef);
-//
-// if (state == null) {
-//
-// final JoinTypeEnum joinType = (JoinTypeEnum) op
-// .getRequiredProperty(Annotations.JOIN_TYPE);
-//
-// state = new JVMHashJoinUtility(op, joinType);
-//
-// if (attrs.putIfAbsent(namedSetRef, state) != null)
-// throw new AssertionError();
-//
-// }
-//
-// this.state = state;
-//
-// }
-//
-// }
-//
-// /**
-// * Evaluate.
-// */
-// public Void call() throws Exception {
-//
-// try {
-//
-// // Buffer all source solutions.
-// acceptSolutions();
-//
-// if(context.isLastInvocation()) {
-//
-// // Checkpoint the solution set.
-// checkpointSolutionSet();
-//
-// // Output the buffered solutions.
-// outputSolutions();
-//
-// }
-//
-// // Done.
-// return null;
-//
-// } finally {
-//
-// context.getSource().close();
-//
-// context.getSink().close();
-//
-// }
-//
-// }
-//
-// /**
-// * Buffer intermediate resources.
-// */
-// private void acceptSolutions() {
-//
-// state.acceptSolutions(context.getSource(), stats);
-//
-// }
-//
-// /**
-// * Checkpoint and save the solution set.
-// */
-// private void checkpointSolutionSet() {
-//
-// state.saveSolutionSet();
-//
-// }
-//
-// /**
-// * Output the buffered solutions.
-// */
-// private void outputSolutions() {
-//
-// // default sink
-// final IBlockingBuffer<IBindingSet[]> sink = context.getSink();
-//
-// final UnsyncLocalOutputBuffer<IBindingSet> unsyncBuffer = new UnsyncLocalOutputBuffer<IBindingSet>(
-// op.getChunkCapacity(), sink);
-//
-// state.outputSolutions(unsyncBuffer);
-//
-// unsyncBuffer.flush();
-//
-// sink.flush();
-//
-// }
-//
-// } // ControllerTask
-
}
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java 2013-04-28 15:03:41 UTC (rev 7088)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java 2013-04-29 17:28:42 UTC (rev 7089)
@@ -32,7 +32,6 @@
import java.util.LinkedHashMap;
import java.util.LinkedList;
import java.util.List;
-import java.util.Map;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
@@ -44,13 +43,13 @@
import com.bigdata.bop.HTreeAnnotations;
import com.bigdata.bop.HashMapAnnotations;
import com.bigdata.bop.IBindingSet;
-import com.bigdata.bop.IConstant;
import com.bigdata.bop.IConstraint;
import com.bigdata.bop.IVariable;
import com.bigdata.bop.PipelineOp;
import com.bigdata.bop.engine.BOpStats;
-import com.bigdata.bop.join.JVMHashJoinUtility.JVMHashIndex.Bucket;
-import com.bigdata.bop.join.JVMHashJoinUtility.JVMHashIndex.SolutionHit;
+import com.bigdata.bop.join.JVMHashIndex.Bucket;
+import com.bigdata.bop.join.JVMHashIndex.Key;
+import com.bigdata.bop.join.JVMHashIndex.SolutionHit;
import com.bigdata.counters.CAT;
import com.bigdata.htree.HTree;
import com.bigdata.rdf.internal.impl.literal.XSDBooleanIV;
@@ -74,523 +73,7 @@
public class JVMHashJoinUtility implements IHashJoinUtility {
private static final Logger log = Logger.getLogger(JVMHashJoinUtility.class);
-
- public static class JVMHashIndex {
- /**
- * Note: If joinVars is an empty array, then the solutions will all hash to
- * ONE (1).
- */
- private static final int ONE = 1;
-
- /**
- * Return the hash code which will be used as the key given the ordered
- * as-bound values for the join variables.
- *
- * @param joinVars
- * The join variables.
- * @param bset
- * The bindings whose as-bound hash code for the join variables
- * will be computed.
- * @param ignoreUnboundVariables
- * If a variable without a binding should be silently ignored.
- *
- * @return The hash code.
- *
- * @throws JoinVariableNotBoundException
- * if there is no binding for a join variable.
- */
- private static int hashCode(final IVariable<?>[] joinVars,
- final IBindingSet bset, final boolean ignoreUnboundVariables)
- throws JoinVariableNotBoundException {
-
- int h = ONE;
-
- for (IVariable<?> v : joinVars) {
-
- final IConstant<?> c = bset.get(v);
-
- if (c == null) {
-
- if(ignoreUnboundVariables)
- continue;
-
- // Reject any solution which does not have a binding for a join
- // variable.
-
- throw new JoinVariableNotBoundException(v.getName());
-
- }
-
- h = 31 * h + c.hashCode();
-
- }
-
- if (log.isTraceEnabled())
- log.trace("hashCode=" + h + ", joinVars="
- + Arrays.toString(joinVars) + " : " + bset);
-
- return h;
-
- }
-
-
- /**
- * Return an array of constants corresponding to the as-bound values of the
- * join variables for the given solution.
- *
- * @param joinVars
- * The join variables.
- * @param bset
- * The solution.
- * @param optional
- * <code>true</code> iff the hash join is optional.
- *
- * @return The as-bound values for the join variables for that solution.
- */
- static private Key makeKey(final IVariable<?>[] joinVars,
- final IBindingSet bset, final boolean optional) {
-
- final IConstant<?>[] vals = new IConstant<?>[joinVars.length];
-
- for (int i = 0; i < joinVars.length; i++) {
-
- final IVariable<?> v = joinVars[i];
-
- vals[i] = bset.get(v);
-
- }
-
- int hashCode = ONE;
- try {
-
- /*
- * Note: The original version of this class always throws an
- * exception for an unbound join variable out of its hashCode() impl
- * and then handles that exception here.
- */
-
- hashCode = hashCode(joinVars, bset, false/* ignoreUnboundVariables */);
-
- } catch (JoinVariableNotBoundException ex) {
-
- if (!optional) {
-
- // Drop solution;
- ...
[truncated message content] |
|
From: <tho...@us...> - 2013-04-28 15:03:50
|
Revision: 7088
http://bigdata.svn.sourceforge.net/bigdata/?rev=7088&view=rev
Author: thompsonbry
Date: 2013-04-28 15:03:41 +0000 (Sun, 28 Apr 2013)
Log Message:
-----------
Refactored the JVMHashJoinUtility to extract a JVMHashIndex class that encapsulates the Map/Bucket/SolutionHit pattern for reuse. In particular, this pattern allows us to track the #of occurrences of a distinct solution (a histogram).
I identified and fixed 2 bugs in JVMHashJoinUtility.filterSolutions(). It appears that this method was not in use, which makes sense. We prefer the JVMDistinctBindingSetsOp which is based on a ConcurrentHashMap and has significantly better performance for DISTINCT.
I found a bug in my implementation of the "DISTINCT" filter on the solutions projected into the sub-group. I was failing to restrict the solutions passed into the sub-group to only the "projected-in-vars". This has been fixed, but there are still FIXMEs in the JVMHashJoinUtility.outputSolutions() method documenting things that need to be reviewed. The "distinct" filter is still disabled in the code at the head of the outputSolutions() method.
There are 5 places where we use a hashIndexBuild => subGroup => hashJoin pattern. These all need to be reviewed for correctness of the projectedIn versus projectedOut (aka "selected"). The bug mentioned directly above was related to the use of "selected" in outputSolutions(). It is assumed in the code that the projectedInVars and projectedOutVars are the same, but that is not true because we want to project in only those variables that are visible to the sub-group and we want to project out only those variables bound in the sub-group that are reused in the parent. We also need to carefully review the named solution set query patterns as the source is not always the pipeline and this might change the effective semantics of the outputSolutions() method (code review required).
- addExistsSubquery(PipelineOp, SubqueryRoot, Set<IVariable<?>>, AST2BOpContext)
- addNamedSubqueryInclude(PipelineOp, NamedSubqueryInclude, Set<IVariable<?>>, AST2BOpContext)
- addSparql11Subquery(PipelineOp, SubqueryRoot, Set<IVariable<?>>, AST2BOpContext)
- addSubgroup(PipelineOp, GraphPatternGroup<IGroupMemberNode>, Set<IVariable<?>>, AST2BOpContext)
- doMergeJoin(PipelineOp, JoinGroupNode, Set<IVariable<?>>, AtomicInteger, AST2BOpContext)
See https://sourceforge.net/apps/trac/bigdata/ticket/668 (JoinGroup Optimization)
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/solutions/JVMDistinctBindingSetsOp.java
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMDistinctFilter.java
Removed Paths:
-------------
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/solutions/JVMDistinctFilter.java
Copied: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMDistinctFilter.java (from rev 7087, branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/solutions/JVMDistinctFilter.java)
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMDistinctFilter.java (rev 0)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMDistinctFilter.java 2013-04-28 15:03:41 UTC (rev 7088)
@@ -0,0 +1,224 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Apr 26, 2013
+ */
+package com.bigdata.bop.join;
+
+import java.util.Arrays;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.log4j.Logger;
+
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.IConstant;
+import com.bigdata.bop.IVariable;
+import com.bigdata.bop.bindingSet.ListBindingSet;
+
+/**
+ * Utility class for imposing a DISTINCT filter on {@link IBindingSet}. This
+ * class is thread-safe. It is based on a {@link ConcurrentHashMap}.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ */
+public class JVMDistinctFilter {
+
+ private static final Logger log = Logger.getLogger(JVMDistinctFilter.class);
+
+ /**
+ * Wrapper used for the as bound solutions in the {@link ConcurrentHashMap}.
+ */
+ private static class Solution {
+
+ private final int hash;
+
+ private final IConstant<?>[] vals;
+
+ public Solution(final IConstant<?>[] vals) {
+ this.vals = vals;
+ this.hash = java.util.Arrays.hashCode(vals);
+ }
+
+ public int hashCode() {
+ return hash;
+ }
+
+ public boolean equals(final Object o) {
+ if (this == o)
+ return true;
+ if (!(o instanceof Solution)) {
+ return false;
+ }
+ final Solution t = (Solution) o;
+ if (vals.length != t.vals.length)
+ return false;
+ for (int i = 0; i < vals.length; i++) {
+ // @todo verify that this allows for nulls with a unit test.
+ if (vals[i] == t.vals[i])
+ continue;
+ if (vals[i] == null)
+ return false;
+ if (!vals[i].equals(t.vals[i]))
+ return false;
+ }
+ return true;
+ }
+ }
+
+ /**
+ * The variables used to impose a distinct constraint.
+ */
+ private final IVariable<?>[] vars;
+
+ /**
+ * A concurrent map whose keys are the bindings on the specified variables
+ * (the keys and the values are the same since the map implementation does
+ * not allow <code>null</code> values).
+ * <p>
+ * Note: The map is shared state and can not be discarded or cleared until
+ * the last invocation!!!
+ */
+ private final ConcurrentHashMap<Solution, Solution> map;
+
+ /**
+ *
+ * @param vars
+ * The set of variables on which the DISTINCT filter will be
+ * imposed. Only these variables will be present in the
+ * "accepted" solutions. Any variable bindings not specified in
+ * this array will be dropped).
+ * @param initialCapacity
+ * @param loadFactor
+ * @param concurrencyLevel
+ */
+ public JVMDistinctFilter(final IVariable<?>[] vars,
+ final int initialCapacity, final float loadFactor,
+ final int concurrencyLevel) {
+
+ if (vars == null)
+ throw new IllegalArgumentException();
+
+ if (vars.length == 0)
+ throw new IllegalArgumentException();
+
+ this.vars = vars;
+
+ this.map = new ConcurrentHashMap<Solution, Solution>(initialCapacity,
+ loadFactor, concurrencyLevel);
+
+ }
+
+ /**
+ * If the bindings are distinct for the configured variables then return
+ * those bindings.
+ *
+ * @param bset
+ * The binding set to be filtered.
+ *
+ * @return The distinct as bound values -or- <code>null</code> if the
+ * binding set duplicates a solution which was already accepted.
+ */
+ public IConstant<?>[] accept(final IBindingSet bset) {
+
+ final IConstant<?>[] r = new IConstant<?>[vars.length];
+
+ for (int i = 0; i < vars.length; i++) {
+
+ /*
+ * Note: This allows null's.
+ *
+ * @todo write a unit test when some variables are not bound.
+ */
+ r[i] = bset.get(vars[i]);
+
+ }
+
+ final Solution s = new Solution(r);
+
+ if (log.isTraceEnabled())
+ log.trace("considering: " + Arrays.toString(r));
+
+ final boolean distinct = map.putIfAbsent(s, s) == null;
+
+ if (distinct && log.isDebugEnabled())
+ log.debug("accepted: " + Arrays.toString(r));
+
+ return distinct ? r : null;
+
+ }
+
+ /**
+ * If the bindings are distinct for the configured variables then return a
+ * new {@link IBindingSet} consisting of only the selected variables.
+ *
+ * @param bset
+ * The binding set to be filtered.
+ *
+ * @return A new {@link IBindingSet} containing only the distinct as bound
+ * values -or- <code>null</code> if the binding set duplicates a
+ * solution which was already accepted.
+ */
+ public IBindingSet accept2(final IBindingSet bset) {
+
+ final IConstant<?>[] vals = accept(bset);
+
+ if (vals == null) {
+
+ /*
+ * This is a duplicate solution.
+ */
+
+ return null;
+
+ }
+
+ /*
+ * This is a distinct solution. Copy only the variables used to select
+ * distinct solutions into a new binding set and add that to the set of
+ * [accepted] binding sets which will be emitted by this operator.
+ */
+
+ final ListBindingSet tmp = new ListBindingSet();
+
+ for (int i = 0; i < vars.length; i++) {
+
+ if (vals[i] != null)
+ tmp.set(vars[i], vals[i]);
+
+ }
+
+ return tmp;
+
+ }
+
+ /**
+ * Discard the map backing this filter.
+ */
+ public void clear() {
+
+ map.clear();
+
+ }
+
+}
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java 2013-04-26 20:07:14 UTC (rev 7087)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java 2013-04-28 15:03:41 UTC (rev 7088)
@@ -49,7 +49,8 @@
import com.bigdata.bop.IVariable;
import com.bigdata.bop.PipelineOp;
import com.bigdata.bop.engine.BOpStats;
-import com.bigdata.bop.solutions.JVMDistinctFilter;
+import com.bigdata.bop.join.JVMHashJoinUtility.JVMHashIndex.Bucket;
+import com.bigdata.bop.join.JVMHashJoinUtility.JVMHashIndex.SolutionHit;
import com.bigdata.counters.CAT;
import com.bigdata.htree.HTree;
import com.bigdata.rdf.internal.impl.literal.XSDBooleanIV;
@@ -74,321 +75,522 @@
private static final Logger log = Logger.getLogger(JVMHashJoinUtility.class);
- /**
- * Note: If joinVars is an empty array, then the solutions will all hash to
- * ONE (1).
- */
- private static final int ONE = 1;
-
- /**
- * Return the hash code which will be used as the key given the ordered
- * as-bound values for the join variables.
- *
- * @param joinVars
- * The join variables.
- * @param bset
- * The bindings whose as-bound hash code for the join variables
- * will be computed.
- * @param ignoreUnboundVariables
- * If a variable without a binding should be silently ignored.
- *
- * @return The hash code.
- *
- * @throws JoinVariableNotBoundException
- * if there is no binding for a join variable.
- */
- private static int hashCode(final IVariable<?>[] joinVars,
- final IBindingSet bset, final boolean ignoreUnboundVariables)
- throws JoinVariableNotBoundException {
+ public static class JVMHashIndex {
- int h = ONE;
+ /**
+ * Note: If joinVars is an empty array, then the solutions will all hash to
+ * ONE (1).
+ */
+ private static final int ONE = 1;
+
+ /**
+ * Return the hash code which will be used as the key given the ordered
+ * as-bound values for the join variables.
+ *
+ * @param joinVars
+ * The join variables.
+ * @param bset
+ * The bindings whose as-bound hash code for the join variables
+ * will be computed.
+ * @param ignoreUnboundVariables
+ * If a variable without a binding should be silently ignored.
+ *
+ * @return The hash code.
+ *
+ * @throws JoinVariableNotBoundException
+ * if there is no binding for a join variable.
+ */
+ private static int hashCode(final IVariable<?>[] joinVars,
+ final IBindingSet bset, final boolean ignoreUnboundVariables)
+ throws JoinVariableNotBoundException {
- for (IVariable<?> v : joinVars) {
+ int h = ONE;
- final IConstant<?> c = bset.get(v);
+ for (IVariable<?> v : joinVars) {
- if (c == null) {
+ final IConstant<?> c = bset.get(v);
- if(ignoreUnboundVariables)
- continue;
+ if (c == null) {
- // Reject any solution which does not have a binding for a join
- // variable.
+ if(ignoreUnboundVariables)
+ continue;
- throw new JoinVariableNotBoundException(v.getName());
+ // Reject any solution which does not have a binding for a join
+ // variable.
+
+ throw new JoinVariableNotBoundException(v.getName());
+
+ }
+
+ h = 31 * h + c.hashCode();
}
-
- h = 31 * h + c.hashCode();
- }
-
- if (log.isTraceEnabled())
- log.trace("hashCode=" + h + ", joinVars="
- + Arrays.toString(joinVars) + " : " + bset);
+ if (log.isTraceEnabled())
+ log.trace("hashCode=" + h + ", joinVars="
+ + Arrays.toString(joinVars) + " : " + bset);
- return h;
+ return h;
- }
-
- /**
- * Return an array of constants corresponding to the as-bound values of the
- * join variables for the given solution.
- *
- * @param joinVars
- * The join variables.
- * @param bset
- * The solution.
- * @param optional
- * <code>true</code> iff the hash join is optional.
- *
- * @return The as-bound values for the join variables for that solution.
- */
- static private Key makeKey(final IVariable<?>[] joinVars,
- final IBindingSet bset, final boolean optional) {
+ }
- final IConstant<?>[] vals = new IConstant<?>[joinVars.length];
+
+ /**
+ * Return an array of constants corresponding to the as-bound values of the
+ * join variables for the given solution.
+ *
+ * @param joinVars
+ * The join variables.
+ * @param bset
+ * The solution.
+ * @param optional
+ * <code>true</code> iff the hash join is optional.
+ *
+ * @return The as-bound values for the join variables for that solution.
+ */
+ static private Key makeKey(final IVariable<?>[] joinVars,
+ final IBindingSet bset, final boolean optional) {
- for (int i = 0; i < joinVars.length; i++) {
+ final IConstant<?>[] vals = new IConstant<?>[joinVars.length];
- final IVariable<?> v = joinVars[i];
+ for (int i = 0; i < joinVars.length; i++) {
- vals[i] = bset.get(v);
+ final IVariable<?> v = joinVars[i];
- }
+ vals[i] = bset.get(v);
- int hashCode = ONE;
- try {
+ }
- /*
- * Note: The original version of this class always throws an
- * exception for an unbound join variable out of its hashCode() impl
- * and then handles that exception here.
- */
-
- hashCode = hashCode(joinVars, bset, false/* ignoreUnboundVariables */);
+ int hashCode = ONE;
+ try {
- } catch (JoinVariableNotBoundException ex) {
-
- if (!optional) {
+ /*
+ * Note: The original version of this class always throws an
+ * exception for an unbound join variable out of its hashCode() impl
+ * and then handles that exception here.
+ */
- // Drop solution;
+ hashCode = hashCode(joinVars, bset, false/* ignoreUnboundVariables */);
+
+ } catch (JoinVariableNotBoundException ex) {
- if (log.isDebugEnabled())
- log.debug(ex);
+ if (!optional) {
+
+ // Drop solution;
+
+ if (log.isDebugEnabled())
+ log.debug(ex);
- return null;
+ return null;
+ }
+
}
+
+ return new Key(hashCode, vals);
}
-
- return new Key(hashCode, vals);
- }
+ /**
+ * Wrapper for the keys in the hash table. This is necessary for the hash
+ * table to compare the keys as equal and also provides a efficiencies in
+ * the hash code and equals() methods.
+ */
+ static class Key {
+
+ private final int hash;
- /**
- * Wrapper for the keys in the hash table. This is necessary for the hash
- * table to compare the keys as equal and also provides a efficiencies in
- * the hash code and equals() methods.
- */
- private static class Key {
-
- private final int hash;
+ private final IConstant<?>[] vals;
- private final IConstant<?>[] vals;
+ private Key(final int hashCode, final IConstant<?>[] vals) {
+ this.vals = vals;
+ this.hash = hashCode;
+ }
- private Key(final int hashCode, final IConstant<?>[] vals) {
- this.vals = vals;
- this.hash = hashCode;
- }
+ public int hashCode() {
+ return hash;
+ }
- public int hashCode() {
- return hash;
- }
-
- public boolean equals(final Object o) {
- if (this == o)
- return true;
- if (!(o instanceof Key)) {
- return false;
- }
- final Key t = (Key) o;
- if (vals.length != t.vals.length)
- return false;
- for (int i = 0; i < vals.length; i++) {
- if (vals[i] == t.vals[i])
- continue;
- if (vals[i] == null)
+ public boolean equals(final Object o) {
+ if (this == o)
+ return true;
+ if (!(o instanceof Key)) {
return false;
- if (!vals[i].equals(t.vals[i]))
+ }
+ final Key t = (Key) o;
+ if (vals.length != t.vals.length)
return false;
+ for (int i = 0; i < vals.length; i++) {
+ if (vals[i] == t.vals[i])
+ continue;
+ if (vals[i] == null)
+ return false;
+ if (!vals[i].equals(t.vals[i]))
+ return false;
+ }
+ return true;
}
- return true;
}
- }
-
- /**
- * An input solution and a hit counter.
- */
- private static class SolutionHit {
-
+
/**
- * The input solution.
+ * An input solution and a hit counter.
*/
- final public IBindingSet solution;
+ public static class SolutionHit {
+ /**
+ * The input solution.
+ */
+ final public IBindingSet solution;
+
+ /**
+ * The #of hits on that input solution when processing the join against
+ * the subquery.
+ */
+ public final CAT nhits = new CAT();
+
+ private SolutionHit(final IBindingSet solution) {
+
+ if(solution == null)
+ throw new IllegalArgumentException();
+
+ this.solution = solution;
+
+ }
+
+ public String toString() {
+
+ return getClass().getName() + "{nhits=" + nhits + ",solution="
+ + solution + "}";
+
+ }
+
+ } // class SolutionHit
+
/**
- * The #of hits on that input solution when processing the join against
- * the subquery.
+ * A group of solutions having the same as-bound values for their join vars.
+ * Each solution is paired with a hit counter so we can support OPTIONAL
+ * semantics for the join.
*/
- public final CAT nhits = new CAT();
-
- private SolutionHit(final IBindingSet solution) {
+ public static class Bucket implements Iterable<SolutionHit>,
+ Comparable<Bucket> {
+
+ /** The hash code for this collision bucket. */
+ private final int hashCode;
- if(solution == null)
- throw new IllegalArgumentException();
+ /**
+ * A set of solutions (and their hit counters) which have the same
+ * as-bound values for the join variables.
+ */
+ private final List<SolutionHit> solutions = new LinkedList<SolutionHit>();
+
+ public String toString() {
+ return super.toString()
+ + //
+ "{hashCode=" + hashCode + ",#solutions=" + solutions.size()
+ + "}";
+ }
- this.solution = solution;
+ public Bucket(final int hashCode, final IBindingSet solution) {
+
+ this.hashCode = hashCode;
+
+ add(solution);
+
+ }
+
+ public void add(final IBindingSet solution) {
+
+ if (solution == null)
+ throw new IllegalArgumentException();
+
+ solutions.add(new SolutionHit(solution));
+
+ }
- }
-
- public String toString() {
+ /**
+ * Add the solution to the bucket iff the solutions is not already
+ * present in the bucket.
+ * <p>
+ * Note: There is already a hash index in place on the join variables
+ * when we are doing a DISTINCT filter. Further, only the "join"
+ * variables are "selected" and participate in a DISTINCT filter.
+ * Therefore, if we have a hash collision such that two solutions would
+ * be directed into the same {@link Bucket} then we can not improve
+ * matters but must simply scan the solutions in the bucket to decide
+ * whether the new solution duplicates a solution which is already
+ * present.
+ *
+ * @param solution
+ * The solution.
+ *
+ * @return <code>true</code> iff the bucket was modified by this
+ * operation.
+ */
+ public boolean addDistinct(final IBindingSet solution) {
- return getClass().getName() + "{nhits=" + nhits + ",solution="
- + solution + "}";
+ if(solutions.isEmpty()) {
- }
-
- } // class SolutionHit
+ // First solution.
+ solutions.add(new SolutionHit(solution));
+
+ return true;
+
+ }
- /**
- * A group of solutions having the same as-bound values for their join vars.
- * Each solution is paired with a hit counter so we can support OPTIONAL
- * semantics for the join.
- */
- private static class Bucket implements Iterable<SolutionHit>,
- Comparable<Bucket> {
+ final Iterator<SolutionHit> itr = solutions.iterator();
+
+ while(itr.hasNext()) {
+
+ final SolutionHit aSolution = itr.next();
+
+ if(aSolution.solution.equals(solution)) {
+
+ // Solution already in this bucket.
+ return false;
+
+ }
+
+ }
+
+ // This is a distinct solution.
+ solutions.add(new SolutionHit(solution));
+
+ return true;
+
+ }
+
+ public Iterator<SolutionHit> iterator() {
+
+// return Collections.unmodifiableList(solutions).iterator();
+ return solutions.iterator();
+
+ }
- /** The hash code for this collision bucket. */
- private final int hashCode;
-
+// @SuppressWarnings("unchecked")
+// public Iterator<IBindingSet> bindingSetIterator() {
+//
+// return new Striterator(solutions.iterator()).addFilter(new Resolver() {
+//
+// @Override
+// protected Object resolve(Object obj) {
+// return ((SolutionHit)obj).solution;
+// }
+// });
+//
+// }
+
+ /**
+ * Orders the buckets based on their hash codes.
+ */
+ @Override
+ public int compareTo(final Bucket o) {
+ if (hashCode > o.hashCode)
+ return 1;
+ if (hashCode < o.hashCode)
+ return -1;
+ return 0;
+ }
+
+ } // Bucket
+
/**
- * A set of solutions (and their hit counters) which have the same
- * as-bound values for the join variables.
+ * The backing map - this is NOT thread safe.
*/
- private final List<SolutionHit> solutions = new LinkedList<SolutionHit>();
+ private final Map<Key, Bucket> map;
+ private final IVariable<?>[] joinVars;
+// private final boolean optional;
- public String toString() {
- return super.toString()
- + //
- "{hashCode=" + hashCode + ",#solutions=" + solutions.size()
- + "}";
- }
-
- public Bucket(final int hashCode, final IBindingSet solution) {
+ public JVMHashIndex(final int initialCapacity, final float loadFactor,
+ final IVariable<?>[] joinVars) {//, final boolean optional) {
- this.hashCode = hashCode;
+ if (joinVars == null) {
+ /*
+ * A ZERO LENGTH joinVars[] means that all solutions will be in
+ * the same hash bucket. This can arise due to poor assignment
+ * of join variables or simply because there are no available
+ * join variables (full cross product join). Such joins are very
+ * expensive.
+ */
+ throw new IllegalArgumentException();
+ }
- add(solution);
-
+ this.map = new LinkedHashMap<Key, Bucket>(initialCapacity,
+ loadFactor);
+
+ this.joinVars = joinVars;
+
+ /*
+ * TOOD Can we pass this in and remove it from the API? But see
+ * filterDistinct().
+ */
+// this.optional = optional;
+
}
- public void add(final IBindingSet solution) {
-
- if (solution == null)
- throw new IllegalArgumentException();
+ /**
+ * Add the solution to the index.
+ *
+ * @param bset
+ * The {@link IBindingSet}.
+ * @param optional
+ *
+ * @return The {@link Key} iff the solution was added to the index and
+ * <code>null</code> iff the solution was not added (because a
+ * {@link Key} could not be formed for the solution given the
+ * specified {@link #joinVars}).
+ *
+ * TODO javadoc on OPTIONAL
+ */
+ public Key add(final IBindingSet bset,final boolean optional) {
+
+ final Key key = makeKey(joinVars, bset, optional);
+
+ if (key == null) {
+
+ // Drop solution.
+ return null;
+
+ }
+
+ Bucket b = map.get(key);
+
+ if (b == null) {
+
+ map.put(key, b = new Bucket(key.hash, bset));
+
+ } else {
+
+ b.add(bset);
+
+ }
+
+ return key;
- solutions.add(new SolutionHit(solution));
-
}
/**
- * Add the solution to the bucket iff the solutions is not already
- * present in the bucket.
- * <p>
- * Note: There is already a hash index in place on the join variables
- * when we are doing a DISTINCT filter. Further, only the "join"
- * variables are "selected" and participate in a DISTINCT filter.
- * Therefore, if we have a hash collision such that two solutions would
- * be directed into the same {@link Bucket} then we can not improve
- * matters but must simply scan the solutions in the bucket to decide
- * whether the new solution duplicates a solution which is already
- * present.
+ * Add the solution to the index iff the solution is not already present
+ * in the index.
*
* @param solution
* The solution.
*
- * @return <code>true</code> iff the bucket was modified by this
+ * @return <code>true</code> iff the index was modified by this
* operation.
*/
- public boolean addDistinct(final IBindingSet solution) {
+ public boolean addDistinct(final IBindingSet bset) {
- if(solutions.isEmpty()) {
+ // TODO Review why optional:=true here.
+ final Key key = makeKey(joinVars, bset, true/* optional */);
- // First solution.
- solutions.add(new SolutionHit(solution));
-
+ assert key != null;
+
+ Bucket b = map.get(key);
+
+ if (b == null) {
+
+ // New bucket holding just this solution.
+ map.put(key, b = new Bucket(key.hash, bset));
+
return true;
-
- }
- final Iterator<SolutionHit> itr = solutions.iterator();
-
- while(itr.hasNext()) {
-
- final SolutionHit aSolution = itr.next();
-
- if(aSolution.solution.equals(solution)) {
-
- // Solution already in this bucket.
- return false;
-
+ } else {
+
+ if (b.addDistinct(bset)) {
+
+ // Existing bucket not having this solution.
+ return true;
+
}
-
+
+ // Existing bucket with duplicate solution.
+ return false;
+
}
+
+ }
+
+ /**
+ * Return the hash {@link Bucket} into which the given solution is mapped.
+ * <p>
+ * Note: The caller must apply an appropriate join constraint in order
+ * to correctly reject solutions that (a) violate the join contract; and
+ * (b) that are present in the hash bucket due to a hash collection
+ * rather than because they have the same bindings for the join
+ * variables.
+ *
+ * @param left
+ * The probe.
+ * @param optional
+ *
+ * @return The hash {@link Bucket} into which the given solution is
+ * mapped -or- <code>null</code> if there is no such hash
+ * bucket.
+ *
+ * TODO javadoc [optional].
+ */
+ public Bucket getBucket(final IBindingSet left, final boolean optional) {
+
+ final Key key = makeKey(joinVars, left, optional);
+
+ if (key == null) {
+
+ return null;
+
+ }
+
+ // Probe the hash map : May return [null]!
+ return map.get(key);
+
+ }
+
+ /**
+ * Visit all buckets in the hash index.
+ */
+ public Iterator<Bucket> buckets() {
- // This is a distinct solution.
- solutions.add(new SolutionHit(solution));
+ return map.values().iterator();
- return true;
-
}
- public Iterator<SolutionHit> iterator() {
+ /**
+ * The #of buckets in the hash index. Each bucket has a distinct hash
+ * code. Hash collisions can cause solutions that are distinct in their
+ * {@link #joinVars} to nevertheless be mapped into the same hash
+ * bucket.
+ *
+ * @return The #of buckets in the hash index.
+ */
+ public int bucketCount() {
-// return Collections.unmodifiableList(solutions).iterator();
- return solutions.iterator();
+ return map.size();
}
+
+ /**
+ * Export the {@link Bucket}s as an array.
+ */
+ public Bucket[] toArray() {
-// @SuppressWarnings("unchecked")
-// public Iterator<IBindingSet> bindingSetIterator() {
-//
-// return new Striterator(solutions.iterator()).addFilter(new Resolver() {
-//
-// @Override
-// protected Object resolve(Object obj) {
-// return ((SolutionHit)obj).solution;
-// }
-// });
-//
-// }
+ // source.
+ final Iterator<Bucket> bucketIterator = map.values().iterator();
- /**
- * Orders the buckets based on their hash codes.
- */
- @Override
- public int compareTo(final Bucket o) {
- if (hashCode > o.hashCode)
- return 1;
- if (hashCode < o.hashCode)
- return -1;
- return 0;
+ final Bucket[] a = new Bucket[map.size()];
+
+ int i = 0;
+
+ while (bucketIterator.hasNext()) {
+
+ a[i++] = bucketIterator.next();
+
+ }
+
+ return a;
+
}
- } // Bucket
-
+ }
+
/**
* <code>true</code> until the state is discarded by {@link #release()}.
*/
@@ -454,7 +656,7 @@
* Note: There is no separate "joinSet". Instead, the {@link SolutionHit}
* class provides a join hit counter.
*/
- private final AtomicReference<Map<Key, Bucket>> rightSolutionsRef = new AtomicReference<Map<Key, Bucket>>();
+ private final AtomicReference<JVMHashIndex> rightSolutionsRef = new AtomicReference<JVMHashIndex>();
/**
* The #of solutions accepted into the hash index.
@@ -583,16 +785,20 @@
* do this with the DISTINCT SOLUTIONS filter we would have to make the
* mutation operations on a Bucket atomic. E.g., using the synchronized
* keyword. This would give us what amounts to per-hash code striped
- * locks. Note: the JVMDistinctBindingSetsOp does not use this class
- * right now because it enjoys better concurrency than the
- * JVMHashJoinUtility. Also see JVMDistinctFilter, which is the backing
- * implementation for the JVMDistinctBindingSetsOp.
+ * locks.
+ *
+ * Note: the JVMDistinctBindingSetsOp does not use this class right now
+ * because it enjoys better concurrency than the JVMHashJoinUtility.
+ * Also see JVMDistinctFilter, which is the backing implementation for
+ * the JVMDistinctBindingSetsOp.
*/
- rightSolutionsRef.set(new LinkedHashMap<Key, Bucket>(//
+ rightSolutionsRef.set(new JVMHashIndex(//
op.getProperty(HashMapAnnotations.INITIAL_CAPACITY,
HashMapAnnotations.DEFAULT_INITIAL_CAPACITY),//
op.getProperty(HashMapAnnotations.LOAD_FACTOR,
- HashMapAnnotations.DEFAULT_LOAD_FACTOR)//
+ HashMapAnnotations.DEFAULT_LOAD_FACTOR),//
+ joinVars//
+// optional//
));
}
@@ -624,7 +830,7 @@
}
- private Map<Key,Bucket> getRightSolutions() {
+ private JVMHashIndex getRightSolutions() {
return rightSolutionsRef.get();
@@ -659,123 +865,108 @@
final BOpStats stats) {
try {
-
- final Map<Key,Bucket> map = getRightSolutions();
-
- final IBindingSet[] all = BOpUtility.toArray(itr, stats);
- if (log.isDebugEnabled())
- log.debug("Materialized: " + all.length
- + " source solutions.");
+ final JVMHashIndex index = getRightSolutions();
- long naccepted = 0;
-
- for (IBindingSet bset : all) {
+ final IBindingSet[] all = BOpUtility.toArray(itr, stats);
- final Key key = makeKey(joinVars, bset, optional);
+ if (log.isDebugEnabled())
+ log.debug("Materialized: " + all.length + " source solutions.");
- if (key == null) {
- // Drop solution.
- continue;
+ long naccepted = 0;
+
+ for (IBindingSet bset : all) {
+
+ if (index.add(bset, optional) == null) {
+
+ continue;
+
+ }
+
+ naccepted++;
+
}
-
- Bucket b = map.get(key);
-
- if(b == null) {
-
- map.put(key, b = new Bucket(key.hash, bset));
-
- } else {
-
- b.add(bset);
-
- }
-
- naccepted++;
- }
+ if (log.isDebugEnabled())
+ log.debug("There are " + index.bucketCount()
+ + " hash buckets, joinVars="
+ + Arrays.toString(joinVars));
- if (log.isDebugEnabled())
- log.debug("There are : " + map.size()
- + " distinct combinations of the join vars: "
- + Arrays.toString(joinVars));
+ rightSolutionCount.add(naccepted);
- rightSolutionCount.add(naccepted);
-
- return naccepted;
+ return naccepted;
- } catch(Throwable t) {
+ } catch (Throwable t) {
+
throw launderThrowable(t);
+
}
}
+ /*
+ * FIXME I have observed two apparent bugs in this class. First, it was not
+ * assigning the output of [bset.copy(joinVars)] back to bset. Second, it
+ * was failing to output the first solution in a given bucket. I suspect
+ * that nobody is calling this code and that the JVMDistinctBindingSetOp is
+ * being used instead (which is a better choice since it allows full
+ * concurrency) - I have verified this. This method is not called. We might
+ * use the method by the same name on the HTreeHashJoinUtility, but not this
+ * version.
+ */
@Override
public long filterSolutions(final ICloseableIterator<IBindingSet[]> itr,
final BOpStats stats, final IBuffer<IBindingSet> sink) {
-
+
try {
- final Map<Key, Bucket> map = getRightSolutions();
-
- final IBindingSet[] all = BOpUtility.toArray(itr, stats);
+ final JVMHashIndex index = getRightSolutions();
- if (log.isDebugEnabled())
- log.debug("Materialized: " + all.length
- + " source solutions.");
+ final IBindingSet[] all = BOpUtility.toArray(itr, stats);
- for (IBindingSet bset : all) {
+ if (log.isDebugEnabled())
+ log.debug("Materialized: " + all.length + " source solutions.");
- /*
- * Note: For a DISTINCT SOLUTIONS filter, we only consider the
- * variables that are being projected. Further, all variables are
- * used when computing the hash code. Therefore "joinVars" ==
- * "selectedVars" for a DISTINCT SOLUTIONS filter.
- */
- bset.copy(joinVars); // only consider the selected variables.
+ for (IBindingSet bset : all) {
- /*
- * Note: Solutions are NOT dropped if a variable is not bound in a
- * given solution. The variable is simply not used when computing
- * the hash code. Specifying optional:=true here causes makeKey() to
- * have this behavior.
- */
- final Key key = makeKey(joinVars, bset, true/* optional */);
+ /*
+ * Note: For a DISTINCT SOLUTIONS filter, we only consider the
+ * variables that are being projected. Further, all variables
+ * are used when computing the hash code. Therefore "joinVars"
+ * == "selectedVars" for a DISTINCT SOLUTIONS filter.
+ */
+ bset = bset.copy(joinVars); // only consider the selected variables.
- assert key != null;
+ /*
+ * Note: Solutions are NOT dropped if a variable is not bound in
+ * a given solution. The variable is simply not used when
+ * computing the hash code. Specifying optional:=true here
+ * causes makeKey() to have this behavior.
+ */
+ if (index.addDistinct(bset)) {
- Bucket b = map.get(key);
-
- if(b == null) {
-
- map.put(key, b = new Bucket(key.hash, bset));
-
- } else {
-
- if(b.addDistinct(bset)) {
-
// Write on the output sink.
sink.add(bset);
-
+
}
-
+
}
- }
+ if (log.isDebugEnabled())
+ log.debug("There are " + index.bucketCount()
+ + " hash buckets, joinVars="
+ + Arrays.toString(joinVars));
- if (log.isDebugEnabled())
- log.debug("There are : " + map.size()
- + " distinct combinations of the join vars: "
- + Arrays.toString(joinVars));
+ final long naccepted = all.length;
- final long naccepted = all.length;
-
- rightSolutionCount.add(naccepted);
+ rightSolutionCount.add(naccepted);
- return naccepted;
+ return naccepted;
- } catch(Throwable t) {
+ } catch (Throwable t) {
+
throw launderThrowable(t);
+
}
}
@@ -805,10 +996,10 @@
final IConstraint[] constraints//
) {
- final Map<Key,Bucket> rightSolutions = getRightSolutions();
+ final JVMHashIndex rightSolutions = getRightSolutions();
if (log.isInfoEnabled()) {
- log.info("rightSolutions: #buckets=" + rightSolutions.size()
+ log.info("rightSolutions: #buckets=" + rightSolutions.bucketCount()
+ ",#solutions=" + getRightSolutionCount());
}
@@ -826,22 +1017,15 @@
if (log.isDebugEnabled())
log.debug("Considering " + left);
- final Key key = JVMHashJoinUtility.makeKey(joinVars, left,
- optional);
+ final Bucket bucket = rightSolutions.getBucket(
+ left, optional);
- if (key == null) {
- // Drop solution.
+ if (bucket == null)
continue;
- }
+
+ final Iterator<SolutionHit> ritr = bucket
+ .iterator();
- // Probe the hash map.
- final Bucket b = rightSolutions.get(key);
-
- if (b == null)
- continue;
-
- final Iterator<SolutionHit> ritr = b.iterator();
-
while (ritr.hasNext()) {
final SolutionHit right = ritr.next();
@@ -983,20 +1167,24 @@
final Constant f = askVar == null ? null : new Constant(
XSDBooleanIV.FALSE);
- final Map<Key, Bucket> rightSolutions = getRightSolutions();
+ final JVMHashIndex rightSolutions = getRightSolutions();
final IVariable<?>[] selected = getSelectVars();
if (log.isInfoEnabled())
- log.info("rightSolutions: #buckets=" + rightSolutions.size());
+ log.info("rightSolutions: #buckets="
+ + rightSolutions.bucketCount());
/*
* Note: when NO solutions joined for a given source binding set AND
* the join is OPTIONAL then we output the _original_ binding set to
* the sink join task(s) and DO NOT apply the CONSTRAINT(s).
*/
+ final Iterator<Bucket> bitr = rightSolutions.buckets();
+
+ while (bitr.hasNext()) {
- for (Bucket b : rightSolutions.values()) {
+ final Bucket b = bitr.next();
for (SolutionHit hit : b) {
@@ -1050,14 +1238,13 @@
// */
// final IVariable<?>[] selected = getSelectVars();
- final Map<Key, Bucket> rightSolutions = getRightSolutions();
+ final JVMHashIndex rightSolutions = getRightSolutions();
if (log.isInfoEnabled())
- log.info("rightSolutions: #buckets=" + rightSolutions.size());
+ log.info("rightSolutions: #buckets=" + rightSolutions.bucketCount());
// Visit the buckets.
- IStriterator itr = new Striterator(rightSolutions.values()
- .iterator());
+ IStriterator itr = new Striterator(rightSolutions.buckets());
itr = itr.addFilter(new Expander() {
@@ -1068,7 +1255,7 @@
*/
@SuppressWarnings("rawtypes")
@Override
- protected Iterator expand(Object obj) {
+ protected Iterator expand(final Object obj) {
final Bucket b = (Bucket) obj;
@@ -1085,7 +1272,7 @@
private static final long serialVersionUID = 1L;
@Override
- protected Object resolve(Object obj) {
+ protected Object resolve(final Object obj) {
final IBindingSet bs = ((SolutionHit) obj).solution;
@@ -1126,6 +1313,10 @@
* of them might also be related to a failure to produce the correct set
* of variables for [projectedInVars].
*
+ * TODO I have factored out the JVMHashIndex class. This class tracks
+ * the #of hits for each distinct solution. We can use this to correct
+ * the output cardinality.
+ *
* @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/668" >
* JoinGroup optimizations </a>
*/
@@ -1138,6 +1329,10 @@
/*
* Note: We are single threaded here so we can use a lower
* concurrencyLevel value.
+ *
+ * Note: If necessary, this could be replaced with JVMHashIndex so
+ * we get the #of occurrences of each distinct combination of
+ * bindings that is projected into the sub-group/-query.
*/
final int concurrencyLevel = 1;//ConcurrentHashMapAnnotations.DEFAULT_CONCURRENCY_LEVEL;
@@ -1201,17 +1396,17 @@
* Code works, uses nested iterators pattern.
*/
- final Map<Key, Bucket> rightSolutions = getRightSolutions();
+ final JVMHashIndex rightSolutions = getRightSolutions();
final IVariable<?>[] selected = getSelectVars();
if (log.isInfoEnabled())
log.info("rightSolutions: #buckets="
- + rightSolutions.size());
+ + rightSolutions.bucketCount());
// source.
- final Iterator<Bucket> bucketIterator = rightSolutions.values()
- .iterator();
+ final Iterator<Bucket> bucketIterator = rightSolutions.
+ buckets();
while (bucketIterator.hasNext()) {
@@ -1232,17 +1427,23 @@
*
*/
- if (distinctFilter.accept2(bs) == null) {
+ if ((bs = distinctFilter.accept2(bs)) == null) {
// Drop duplicate solutions.
continue;
}
-
- }
-
- if (selected != null) {
+ } else if (selected != null) {
+
+ /*
+ * FIXME We should be using projectedInVars here since
+ * outputSolutions() is used to stream solutions into
+ * the child join group (at least for some kinds of
+ * joins, but there might be exceptions for joining with
+ * a named solution set).
+ */
+
// Drop variables which are not projected.
bs = bs.copy(selected);
@@ -1276,15 +1477,19 @@
final Constant t = askVar == null ? null : new Constant(
XSDBooleanIV.TRUE);
- final Map<Key, Bucket> rightSolutions = getRightSolutions();
+ final JVMHashIndex rightSolutions = getRightSolutions();
final IVariable<?>[] selected = getSelectVars();
if (log.isInfoEnabled())
- log.info("rightSolutions: #buckets=" + rightSolutions.size());
+ log.info("rightSolutions: #buckets=" + rightSolutions.bucketCount());
- for (Bucket b : rightSolutions.values()) {
+ final Iterator<Bucket> bitr = rightSolutions.buckets();
+ while(bitr.hasNext()) {
+
+ final Bucket b = bitr.next();
+
for (SolutionHit hit : b) {
if (hit.nhits.get() == 0)
@@ -1367,29 +1572,6 @@
}
/**
- * Export the {@link Bucket}s as an array.
- */
- static private Bucket[] toArray(final Map<Key,Bucket> rightSolutions) {
-
- // source.
- final Iterator<Bucket> bucketIterator = rightSolutions.values()
- .iterator();
-
- final Bucket[] a = new Bucket[rightSolutions.size()];
-
- int i = 0;
-
- while (bucketIterator.hasNext()) {
-
- a[i++] = bucketIterator.next();
-
- }
-
- return a;
-
- }
-
- /**
* Advance each other source to the first hash code GTE the hashCode for the
* first source.
* <p>
@@ -1550,8 +1732,8 @@
for (int i = 0; i < all.length; i++) {
// Fully materialize the solution set as a Bucket[].
- final Bucket[] t = toArray(all[i].getRightSolutions());
-
+ final Bucket[] t = all[i].getRightSolutions().toArray();
+
/*
* Sort the array. It's natural sort order is by the hash code
* of the join variables.
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/solutions/JVMDistinctBindingSetsOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/solutions/JVMDistinctBindingSetsOp.java 2013-04-26 20:07:14 UTC (rev 7087)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/solutions/JVMDistinctBindingSetsOp.java 2013-04-28 15:03:41 UTC (rev 7088)
@@ -39,6 +39,7 @@
import com.bigdata.bop.NV;
import com.bigdata.bop.PipelineOp;
import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.bop.join.JVMDistinctFilter;
import com.bigdata.bop.join.JVMHashJoinUtility;
import com.bigdata.relation.accesspath.IBlockingBuffer;
import com.bigdata.striterator.ICloseableIterator;
Deleted: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/solutions/JVMDistinctFilter.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/solutions/JVMDistinctFilter.java 2013-04-26 20:07:14 UTC (rev 7087)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/solutions/JVMDistinctFilter.java 2013-04-28 15:03:41 UTC (rev 7088)
@@ -1,224 +0,0 @@
-/**
-
-Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
-/*
- * Created on Apr 26, 2013
- */
-package com.bigdata.bop.solutions;
-
-import java.util.Arrays;
-import java.util.concurrent.ConcurrentHashMap;
-
-import org.apache.log4j.Logger;
-
-import com.bigdata.bop.IBindingSet;
-import com.bigdata.bop.IConstant;
-import com.bigdata.bop.IVariable;
-import com.bigdata.bop.bindingSet.ListBindingSet;
-
-/**
- * Utility class for imposing a DISTINCT filter on {@link IBindingSet}. This
- * class is thread-safe. It is based on a {@link ConcurrentHashMap}.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- */
-public class JVMDistinctFilter {
-
- private static final Logger log = Logger.getLogger(JVMDistinctFilter.class);
-
- /**
- * Wrapper used for the as bound solutions in the {@link ConcurrentHashMap}.
- */
- private static class Solution {
-
- private final int hash;
-
- private final IConstant<?>[] vals;
-
- public Solution(final IConstant<?>[] vals) {
- this.vals = vals;
- this.hash = java.util.Arrays.hashCode(vals);
- }
-
- public int hashCode() {
- return hash;
- }
-
- public boolean equals(final Object o) {
- if (this == o)
- return true;
- if (!(o instanceof Solution)) {
- return false;
- }
- final Solution t = (Solution) o;
- if (vals.length != t.vals.length)
- return false;
- for (int i = 0; i < vals.length; i++) {
- // @todo verify that this allows for nulls with a unit test.
- if (vals[i] == t.vals[i])
- continue;
- if (vals[i] == null)
- return false;
- if (!vals[i].equals(t.vals[i]))
- return false;
- }
- return true;
- }
- }
-
- /**
- * The variables used to impose a distinct constraint.
- */
- private final IVariable<?>[] vars;
-
- /**
- * A concurrent map whose keys are the bindings on the specified variables
- * (the keys and the values are the same since the map implementation does
- * not allow <code>null</code> values).
- * <p>
- * Note: The map is shared state and can not be discarded or cleared until
- * the last invocation!!!
- */
- private final ConcurrentHashMap<Solution, Solution> map;
-
- /**
- *
- * @param vars
- * The set of variables on which the DISTINCT filter will be
- * imposed. Only these variables will be present in the
- * "accepted" solutions. Any variable bindings not specified in
- * this array will be dropped).
- * @param initialCapacity
- * @param loadFactor
- * @param concurrencyLevel
- */
- public JVMDistinctFilter(final IVariable<?>[] vars,
- final int initialCapacity, final float loadFactor,
- final int concurrencyLevel) {
-
- if (vars == null)
- throw new IllegalArgumentException();
-
- if (vars.length == 0)
- throw new IllegalArgumentException();
-
- this.vars = vars;
-
- this.map = new ConcurrentHashMap<Solution, Solution>(initialCapacity,
- loadFactor, concurrencyLevel);
-
- }
-
- /**
- * If the bindings are distinct for the configured variables then return
- * those bindings.
- *
- * @param bset
- * The binding set to be filtered.
- *
- * @return The distinct as bound values -or- <code>null</code> if the
- * binding set duplicates a solution which was already accepted.
- */
- public IConstant<?>[] accept(final IBindingSet bset) {
-
- final IConstant<?>[] r = new IConstant<?>[vars.length];
-
- for (int i = 0; i < vars.length; i++) {
-
- /*
- * Note: This allows null's.
- *
- * @todo write a unit test when some variables are not bound.
- */
- r[i] = bset.get(vars[i]);
-
- }
-
- final Solution s = new Solution(r);
-
- if (log.isTraceEnabled())
- log.trace("considering: " + Arrays.toString(r));
-
- final boolean distinct = map.putIfAbsent(s, s) == null;
-
- if (distinct && log.isDebugEnabled())
- log.debug("accepted: " + Arrays.toString(r));
-
- return distinct ? r : null;
-
- }
-
- /**
- * If the bindings are distinct for the configured variables then return a
- * new {@link IBin...
[truncated message content] |
|
From: <tho...@us...> - 2013-04-26 20:07:21
|
Revision: 7087
http://bigdata.svn.sourceforge.net/bigdata/?rev=7087&view=rev
Author: thompsonbry
Date: 2013-04-26 20:07:14 +0000 (Fri, 26 Apr 2013)
Log Message:
-----------
The commit above broke one unit test. If we retain this feature, then the test needs to be modified to predict the new annotation.
{{{
TestASTSubGroupJoinVarOptimizer.test_govtrack_21
}}}
----
I refactored the JVMDistinctBindingSets operator to extract its DISTINCT implementation as a JVMDistinctFilter.
I added support into the JVMHashJoinUtility implementation for this. The feature MUST be enabled by hand in JVMHashJoinUtility.outputSolutions(). set distinct:=true.
When this feature is NOT enabled the following tests fail (these are known failures and documented in the test case).
{{{
TestASTSparql11SubqueryOptimizer#test_subSelectWithNoJoinVars()
TestASTHashJoinOptimizer#test_hashJoinOptimizer_BSBM_Q5()
TestTCK#test_sparql11_order_02()
TestTCK#test_sparql11_order_03()
}}}
If you enable the DISTINCT on the solutions flowing into the child join group, then the following unit tests fail.
I suspect that these failures are related to underproducing solutions, but some of them might also be related to a failure to produce the correct set of variables for [projectedInVars].
{{{
TestNamedGraphs#test_default_graph_joins_01f()
TestOptionals#test_optionals_simplest()
TestOptionals#test_optionals_simplestWithFilter()
TestOptionals#test_double_optional_include()
TestNegation#test_sparql11_minus_01()
TestTCK#test_open_eq_12()
}}}
----
Note: We need to add unit tests (SPARQL and HashJoinUtility) for this feature.
Note: We need to add unit tests for DISTINCT when some of the variables on which the DISTINCT is imposed are null.
----
@see https://sourceforge.net/apps/trac/bigdata/ticket/668 (JoinGroup optimizations)
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/solutions/JVMDistinctBindingSetsOp.java
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/solutions/JVMDistinctFilter.java
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java 2013-04-26 18:36:07 UTC (rev 7086)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/JVMHashJoinUtility.java 2013-04-26 20:07:14 UTC (rev 7087)
@@ -49,6 +49,7 @@
import com.bigdata.bop.IVariable;
import com.bigdata.bop.PipelineOp;
import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.bop.solutions.JVMDistinctFilter;
import com.bigdata.counters.CAT;
import com.bigdata.htree.HTree;
import com.bigdata.rdf.internal.impl.literal.XSDBooleanIV;
@@ -394,6 +395,14 @@
private final AtomicBoolean open = new AtomicBoolean(true);
/**
+ * The operator whose annotations are used to initialize this object.
+ * <p>
+ * Note: This was added to support the DISTINCT FILTER in
+ * {@link #outputSolutions(IBuffer)}.
+ */
+ private final PipelineOp op;
+
+ /**
* The type of join to be performed.
*/
private final JoinTypeEnum joinType;
@@ -419,12 +428,22 @@
private final IVariable<?>[] joinVars;
/**
- * The variables to be retained (optional, all variables are retained if
- * not specified).
+ * The variables to be retained (aka projected out) (optional, all variables
+ * are retained if not specified).
*/
private final IVariable<?>[] selectVars;
/**
+ * The variables to be projected into a join group. When non-
+ * <code>null</code> variables that are NOT in this array are NOT flowed
+ * into the join group.
+ *
+ * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/668" >
+ * JoinGroup optimizations </a>
+ */
+ private final IVariable<?>[] projectedInVars;
+
+ /**
* The join constraints (optional).
*/
private final IConstraint[] constraints;
@@ -515,6 +534,7 @@
if(joinType == null)
throw new IllegalArgumentException();
+ this.op = op;
this.joinType = joinType;
this.optional = joinType == JoinTypeEnum.Optional;
this.filter = joinType == JoinTypeEnum.Filter;
@@ -527,11 +547,19 @@
this.joinVars = (IVariable<?>[]) op
.getRequiredProperty(HashJoinAnnotations.JOIN_VARS);
- // The projected variables (optional and equal to the join variables iff
- // this is a DISTINCT filter).
+ /*
+ * The projected OUT variables (optional and equal to the join variables
+ * iff this is a DISTINCT filter).
+ */
this.selectVars = filter ? joinVars : (IVariable<?>[]) op
.getProperty(JoinAnnotations.SELECT);
+ /*
+ * The variables that are projected IN to the join group.
+ */
+ this.projectedInVars = (IVariable<?>[]) op
+ .getProperty(HashJoinAnnotations.PROJECT_IN_VARS);
+
// The join constraints (optional).
this.constraints = (IConstraint[]) op
.getProperty(JoinAnnotations.CONSTRAINTS);
@@ -557,7 +585,8 @@
* keyword. This would give us what amounts to per-hash code striped
* locks. Note: the JVMDistinctBindingSetsOp does not use this class
* right now because it enjoys better concurrency than the
- * JVMHashJoinUtility.
+ * JVMHashJoinUtility. Also see JVMDistinctFilter, which is the backing
+ * implementation for the JVMDistinctBindingSetsOp.
*/
rightSolutionsRef.set(new LinkedHashMap<Key, Bucket>(//
op.getProperty(HashMapAnnotations.INITIAL_CAPACITY,
@@ -682,8 +711,8 @@
}
@Override
- public long filterSolutions(ICloseableIterator<IBindingSet[]> itr,
- BOpStats stats, IBuffer<IBindingSet> sink) {
+ public long filterSolutions(final ICloseableIterator<IBindingSet[]> itr,
+ final BOpStats stats, final IBuffer<IBindingSet> sink) {
try {
@@ -1085,6 +1114,47 @@
@Override
public void outputSolutions(final IBuffer<IBindingSet> out) {
+ /*
+ * FIXME Set this to enable "DISTINCT" on the solutions flowing into the
+ * join group.
+ *
+ * Note: This should be set by the HashIndexOp (or passed in through the
+ * interface).
+ *
+ * Note: Enabling this causes failures. See the ticket below. I suspect
+ * that these failures are related to underproducing solutions, but some
+ * of them might also be related to a failure to produce the correct set
+ * of variables for [projectedInVars].
+ *
+ * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/668" >
+ * JoinGroup optimizations </a>
+ */
+ final boolean distinct = false;
+
+ final JVMDistinctFilter distinctFilter;
+
+ if (distinct && projectedInVars != null && projectedInVars.length > 0) {
+
+ /*
+ * Note: We are single threaded here so we can use a lower
+ * concurrencyLevel value.
+ */
+ final int concurrencyLevel = 1;//ConcurrentHashMapAnnotations.DEFAULT_CONCURRENCY_LEVEL;
+
+ distinctFilter = new JVMDistinctFilter(projectedInVars, //
+ op.getProperty(HashMapAnnotations.INITIAL_CAPACITY,
+ HashMapAnnotations.DEFAULT_INITIAL_CAPACITY),//
+ op.getProperty(HashMapAnnotations.LOAD_FACTOR,
+ HashMapAnnotations.DEFAULT_LOAD_FACTOR),//
+ concurrencyLevel
+ );
+
+ } else {
+
+ distinctFilter = null;
+
+ }
+
try {
// if (true) {
@@ -1151,6 +1221,26 @@
IBindingSet bs = solutionHit.solution;
+ if (distinctFilter != null) {
+
+ /*
+ * Note: The DISTINCT filter is based on the
+ * variables that are projected INTO the child
+ * join group. However, those are NOT always
+ * the same as the variables that are projected
+ * OUT of the child join group, so we need to
+ *
+ */
+
+ if (distinctFilter.accept2(bs) == null) {
+
+ // Drop duplicate solutions.
+ continue;
+
+ }
+
+ }
+
if (selected != null) {
// Drop variables which are not projected.
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/solutions/JVMDistinctBindingSetsOp.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/solutions/JVMDistinctBindingSetsOp.java 2013-04-26 18:36:07 UTC (rev 7086)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/solutions/JVMDistinctBindingSetsOp.java 2013-04-26 20:07:14 UTC (rev 7087)
@@ -1,26 +1,43 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
package com.bigdata.bop.solutions;
-import java.util.Arrays;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.Callable;
-import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.FutureTask;
-import org.apache.log4j.Logger;
-
import com.bigdata.bop.BOp;
import com.bigdata.bop.BOpContext;
import com.bigdata.bop.ConcurrentHashMapAnnotations;
import com.bigdata.bop.IBindingSet;
-import com.bigdata.bop.IConstant;
import com.bigdata.bop.IQueryAttributes;
import com.bigdata.bop.IVariable;
import com.bigdata.bop.NV;
import com.bigdata.bop.PipelineOp;
-import com.bigdata.bop.bindingSet.ListBindingSet;
import com.bigdata.bop.engine.BOpStats;
import com.bigdata.bop.join.JVMHashJoinUtility;
import com.bigdata.relation.accesspath.IBlockingBuffer;
@@ -31,22 +48,18 @@
* <p>
* Note: This implementation is a pipelined operator which inspects each chunk
* of solutions as they arrive and those solutions which are distinct for each
- * chunk are passed on. It uses a {@link ConcurrentMap} and is thread-safe.
+ * chunk are passed on. It uses a {@link ConcurrentMap} and is thread-safe. It
+ * is significantly faster than the single-threaded hash index routines in the
+ * {@link JVMHashJoinUtility}.
*
- * TODO Look into reconciling this class with {@link JVMHashJoinUtility}.
- * However, note that this implementation is thread-safe and uses a
- * {@link ConcurrentMap}. It may be better to leave things as they are since
- * this implementation may be more efficient for the special case which it
- * handles.
- *
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
* @version $Id: DistinctElementFilter.java 3466 2010-08-27 14:28:04Z
* thompsonbry $
*/
public class JVMDistinctBindingSetsOp extends PipelineOp {
- private final static transient Logger log = Logger
- .getLogger(JVMDistinctBindingSetsOp.class);
+// private final static transient Logger log = Logger
+// .getLogger(JVMDistinctBindingSetsOp.class);
/**
*
@@ -148,81 +161,21 @@
}
/**
- * Wrapper used for the as bound solutions in the {@link ConcurrentHashMap}.
- */
- private static class Solution {
-
- private final int hash;
-
- private final IConstant<?>[] vals;
-
- public Solution(final IConstant<?>[] vals) {
- this.vals = vals;
- this.hash = java.util.Arrays.hashCode(vals);
- }
-
- public int hashCode() {
- return hash;
- }
-
- public boolean equals(final Object o) {
- if (this == o)
- return true;
- if (!(o instanceof Solution)) {
- return false;
- }
- final Solution t = (Solution) o;
- if (vals.length != t.vals.length)
- return false;
- for (int i = 0; i < vals.length; i++) {
- // @todo verify that this allows for nulls with a unit test.
- if (vals[i] == t.vals[i])
- continue;
- if (vals[i] == null)
- return false;
- if (!vals[i].equals(t.vals[i]))
- return false;
- }
- return true;
- }
- }
-
- /**
* Task executing on the node.
*/
static private class DistinctTask implements Callable<Void> {
private final BOpContext<IBindingSet> context;
- /**
- * A concurrent map whose keys are the bindings on the specified
- * variables (the keys and the values are the same since the map
- * implementation does not allow <code>null</code> values).
- * <p>
- * Note: The map is shared state and can not be discarded or cleared
- * until the last invocation!!!
- */
- private final ConcurrentHashMap<Solution, Solution> map;
-
- /**
- * The variables used to impose a distinct constraint.
- */
- private final IVariable<?>[] vars;
+ private final JVMDistinctFilter filter;
- @SuppressWarnings("unchecked")
DistinctTask(final JVMDistinctBindingSetsOp op,
final BOpContext<IBindingSet> context) {
this.context = context;
- this.vars = op.getVariables();
+ final IVariable<?>[] vars = op.getVariables();
- if (vars == null)
- throw new IllegalArgumentException();
-
- if (vars.length == 0)
- throw new IllegalArgumentException();
-
/*
* The map is shared state across invocations of this operator task.
*/
@@ -232,68 +185,28 @@
final IQueryAttributes attribs = context.getRunningQuery()
.getAttributes();
- ConcurrentHashMap<Solution, Solution> map = (ConcurrentHashMap<Solution, Solution>) attribs
- .get(key);
+ JVMDistinctFilter filter = (JVMDistinctFilter) attribs.get(key);
- if (map == null) {
+ if (filter == null) {
- map = new ConcurrentHashMap<Solution, Solution>(
+ filter = new JVMDistinctFilter(vars,
op.getInitialCapacity(), op.getLoadFactor(),
op.getConcurrencyLevel());
- final ConcurrentHashMap<Solution, Solution> tmp = (ConcurrentHashMap<Solution, Solution>) attribs
- .putIfAbsent(key, map);
+ final JVMDistinctFilter tmp = (JVMDistinctFilter) attribs
+ .putIfAbsent(key, filter);
if (tmp != null)
- map = tmp;
+ filter = tmp;
}
- this.map = map;
+ this.filter = filter;
}
}
- /**
- * If the bindings are distinct for the configured variables then return
- * those bindings.
- *
- * @param bset
- * The binding set to be filtered.
- *
- * @return The distinct as bound values -or- <code>null</code> if the
- * binding set duplicates a solution which was already accepted.
- */
- private IConstant<?>[] accept(final IBindingSet bset) {
-
- final IConstant<?>[] r = new IConstant<?>[vars.length];
-
- for (int i = 0; i < vars.length; i++) {
-
- /*
- * Note: This allows null's.
- *
- * @todo write a unit test when some variables are not bound.
- */
- r[i] = bset.get(vars[i]);
-
- }
-
- final Solution s = new Solution(r);
-
- if (log.isTraceEnabled())
- log.trace("considering: " + Arrays.toString(r));
-
- final boolean distinct = map.putIfAbsent(s, s) == null;
-
- if (distinct && log.isDebugEnabled())
- log.debug("accepted: " + Arrays.toString(r));
-
- return distinct ? r : null;
-
- }
-
public Void call() throws Exception {
final BOpStats stats = context.getStats();
@@ -323,27 +236,14 @@
* Test to see if this solution is distinct from those
* already seen.
*/
- final IConstant<?>[] vals = accept(bset);
+ final IBindingSet tmp = filter.accept2(bset);
- if (vals != null) {
+ if (tmp != null) {
- /*
- * This is a distinct solution. Copy only the
- * variables used to select distinct solutions into
- * a new binding set and add that to the set of
- * [accepted] binding sets which will be emitted by
- * this operator.
- */
-
- final ListBindingSet tmp = new ListBindingSet();
-
- for (int i = 0; i < vars.length; i++) {
-
- if (vals[i] != null)
- tmp.set(vars[i], vals[i]);
-
- }
-
+ /*
+ * This is a distinct solution.
+ */
+
accepted.add(tmp);
naccepted++;
@@ -391,7 +291,7 @@
* is not going to be cleared until the query goes out of
* scope and is swept by GC.
*/
- map.clear();
+ filter.clear();
}
Added: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/solutions/JVMDistinctFilter.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/solutions/JVMDistinctFilter.java (rev 0)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/solutions/JVMDistinctFilter.java 2013-04-26 20:07:14 UTC (rev 7087)
@@ -0,0 +1,224 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Apr 26, 2013
+ */
+package com.bigdata.bop.solutions;
+
+import java.util.Arrays;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.log4j.Logger;
+
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.IConstant;
+import com.bigdata.bop.IVariable;
+import com.bigdata.bop.bindingSet.ListBindingSet;
+
+/**
+ * Utility class for imposing a DISTINCT filter on {@link IBindingSet}. This
+ * class is thread-safe. It is based on a {@link ConcurrentHashMap}.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ */
+public class JVMDistinctFilter {
+
+ private static final Logger log = Logger.getLogger(JVMDistinctFilter.class);
+
+ /**
+ * Wrapper used for the as bound solutions in the {@link ConcurrentHashMap}.
+ */
+ private static class Solution {
+
+ private final int hash;
+
+ private final IConstant<?>[] vals;
+
+ public Solution(final IConstant<?>[] vals) {
+ this.vals = vals;
+ this.hash = java.util.Arrays.hashCode(vals);
+ }
+
+ public int hashCode() {
+ return hash;
+ }
+
+ public boolean equals(final Object o) {
+ if (this == o)
+ return true;
+ if (!(o instanceof Solution)) {
+ return false;
+ }
+ final Solution t = (Solution) o;
+ if (vals.length != t.vals.length)
+ return false;
+ for (int i = 0; i < vals.length; i++) {
+ // @todo verify that this allows for nulls with a unit test.
+ if (vals[i] == t.vals[i])
+ continue;
+ if (vals[i] == null)
+ return false;
+ if (!vals[i].equals(t.vals[i]))
+ return false;
+ }
+ return true;
+ }
+ }
+
+ /**
+ * The variables used to impose a distinct constraint.
+ */
+ private final IVariable<?>[] vars;
+
+ /**
+ * A concurrent map whose keys are the bindings on the specified variables
+ * (the keys and the values are the same since the map implementation does
+ * not allow <code>null</code> values).
+ * <p>
+ * Note: The map is shared state and can not be discarded or cleared until
+ * the last invocation!!!
+ */
+ private final ConcurrentHashMap<Solution, Solution> map;
+
+ /**
+ *
+ * @param vars
+ * The set of variables on which the DISTINCT filter will be
+ * imposed. Only these variables will be present in the
+ * "accepted" solutions. Any variable bindings not specified in
+ * this array will be dropped).
+ * @param initialCapacity
+ * @param loadFactor
+ * @param concurrencyLevel
+ */
+ public JVMDistinctFilter(final IVariable<?>[] vars,
+ final int initialCapacity, final float loadFactor,
+ final int concurrencyLevel) {
+
+ if (vars == null)
+ throw new IllegalArgumentException();
+
+ if (vars.length == 0)
+ throw new IllegalArgumentException();
+
+ this.vars = vars;
+
+ this.map = new ConcurrentHashMap<Solution, Solution>(initialCapacity,
+ loadFactor, concurrencyLevel);
+
+ }
+
+ /**
+ * If the bindings are distinct for the configured variables then return
+ * those bindings.
+ *
+ * @param bset
+ * The binding set to be filtered.
+ *
+ * @return The distinct as bound values -or- <code>null</code> if the
+ * binding set duplicates a solution which was already accepted.
+ */
+ public IConstant<?>[] accept(final IBindingSet bset) {
+
+ final IConstant<?>[] r = new IConstant<?>[vars.length];
+
+ for (int i = 0; i < vars.length; i++) {
+
+ /*
+ * Note: This allows null's.
+ *
+ * @todo write a unit test when some variables are not bound.
+ */
+ r[i] = bset.get(vars[i]);
+
+ }
+
+ final Solution s = new Solution(r);
+
+ if (log.isTraceEnabled())
+ log.trace("considering: " + Arrays.toString(r));
+
+ final boolean distinct = map.putIfAbsent(s, s) == null;
+
+ if (distinct && log.isDebugEnabled())
+ log.debug("accepted: " + Arrays.toString(r));
+
+ return distinct ? r : null;
+
+ }
+
+ /**
+ * If the bindings are distinct for the configured variables then return a
+ * new {@link IBindingSet} consisting of only the selected variables.
+ *
+ * @param bset
+ * The binding set to be filtered.
+ *
+ * @return A new {@link IBindingSet} containing only the distinct as bound
+ * values -or- <code>null</code> if the binding set duplicates a
+ * solution which was already accepted.
+ */
+ public IBindingSet accept2(final IBindingSet bset) {
+
+ final IConstant<?>[] vals = accept(bset);
+
+ if (vals == null) {
+
+ /*
+ * This is a duplicate solution.
+ */
+
+ return null;
+
+ }
+
+ /*
+ * This is a distinct solution. Copy only the variables used to select
+ * distinct solutions into a new binding set and add that to the set of
+ * [accepted] binding sets which will be emitted by this operator.
+ */
+
+ final ListBindingSet tmp = new ListBindingSet();
+
+ for (int i = 0; i < vars.length; i++) {
+
+ if (vals[i] != null)
+ tmp.set(vars[i], vals[i]);
+
+ }
+
+ return tmp;
+
+ }
+
+ /**
+ * Discard the map backing this filter.
+ */
+ public void clear() {
+
+ map.clear();
+
+ }
+
+}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-04-26 18:36:13
|
Revision: 7086
http://bigdata.svn.sourceforge.net/bigdata/?rev=7086&view=rev
Author: thompsonbry
Date: 2013-04-26 18:36:07 +0000 (Fri, 26 Apr 2013)
Log Message:
-----------
Increased the maximum log file size and #of retained logs to facilitate analysis when the log level is increased for debugging.
Modified Paths:
--------------
branches/READ_CACHE/src/resources/HAJournal/log4j.properties
Modified: branches/READ_CACHE/src/resources/HAJournal/log4j.properties
===================================================================
--- branches/READ_CACHE/src/resources/HAJournal/log4j.properties 2013-04-26 18:26:18 UTC (rev 7085)
+++ branches/READ_CACHE/src/resources/HAJournal/log4j.properties 2013-04-26 18:36:07 UTC (rev 7086)
@@ -39,8 +39,8 @@
#log4j.appender.dest2=org.apache.log4j.ConsoleAppender
log4j.appender.dest2=org.apache.log4j.RollingFileAppender
log4j.appender.dest2.File=HAJournalServer.log
-log4j.appender.dest2.MaxFileSize=500KB
-log4j.appender.dest2.MaxBackupIndex=10
+log4j.appender.dest2.MaxFileSize=500MB
+log4j.appender.dest2.MaxBackupIndex=20
log4j.appender.dest2.layout=org.apache.log4j.PatternLayout
log4j.appender.dest2.layout.ConversionPattern=%-5p: %r %d{ISO8601} %X{hostname} %X{serviceUUID} %X{taskname} %X{timestamp} %X{resources} %t %l: %m%n
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mrp...@us...> - 2013-04-26 18:26:26
|
Revision: 7085
http://bigdata.svn.sourceforge.net/bigdata/?rev=7085&view=rev
Author: mrpersonick
Date: 2013-04-26 18:26:18 +0000 (Fri, 26 Apr 2013)
Log Message:
-----------
added a "project in" var set to the hash index and graph pattern group so that we can decide what variables to project into a hash join
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/HashJoinAnnotations.java
branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/GraphPatternGroup.java
branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java
branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTSubGroupJoinVarOptimizer.java
branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/TestNegation.java
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/HashJoinAnnotations.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/HashJoinAnnotations.java 2013-04-26 16:53:48 UTC (rev 7084)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/join/HashJoinAnnotations.java 2013-04-26 18:26:18 UTC (rev 7085)
@@ -84,4 +84,13 @@
*/
long DEFAULT_NO_JOIN_VARS_LIMIT = Long.MAX_VALUE;
+
+ /**
+ * The {@link IVariable[]} specifying what variables need to flow into
+ * the right operator of the hash join (i.e. what visible variables inside
+ * the right operator have appeared previously in the query and may be
+ * bound).
+ */
+ String PROJECT_IN_VARS = HashJoinAnnotations.class.getName() + ".projectInVars";
+
}
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/GraphPatternGroup.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/GraphPatternGroup.java 2013-04-26 16:53:48 UTC (rev 7084)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/GraphPatternGroup.java 2013-04-26 18:26:18 UTC (rev 7085)
@@ -63,6 +63,15 @@
*/
String JOIN_VARS = "joinVars";
+ /**
+ * An {@link IVariable}[] of the variables that are used by the
+ * group and that have already appeared in the query up to this point
+ * (and thus may be bound and should be projected into the group).
+ *
+ * @see ASTSubGroupJoinVarOptimizer
+ */
+ String PROJECT_IN_VARS = "projectInVars";
+
}
/**
@@ -102,4 +111,17 @@
setProperty(Annotations.JOIN_VARS, joinVars);
}
+ /**
+ * The variables that should be projected into the group.
+ *
+ * @see Annotations#PROJECT_IN_VARS
+ */
+ public IVariable<?>[] getProjectInVars() {
+ return (IVariable[]) getProperty(Annotations.PROJECT_IN_VARS);
+ }
+
+ public void setProjectInVars(final IVariable<?>[] projectInVars) {
+ setProperty(Annotations.PROJECT_IN_VARS, projectInVars);
+ }
+
}
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java 2013-04-26 16:53:48 UTC (rev 7084)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java 2013-04-26 18:26:18 UTC (rev 7085)
@@ -3417,6 +3417,7 @@
new NV(HTreeHashIndexOp.Annotations.RELATION_NAME, new String[]{ctx.getLexiconNamespace()}),//
new NV(HTreeHashIndexOp.Annotations.JOIN_TYPE, joinType),//
new NV(HTreeHashIndexOp.Annotations.JOIN_VARS, joinVars),//
+ new NV(HTreeHashIndexOp.Annotations.PROJECT_IN_VARS, subgroup.getProjectInVars()),//
new NV(HTreeHashIndexOp.Annotations.SELECT, selectVars),//
new NV(HTreeHashIndexOp.Annotations.NAMED_SET_REF, namedSolutionSet)//
);
@@ -3430,6 +3431,7 @@
new NV(PipelineOp.Annotations.SHARED_STATE, true),// live stats.
new NV(JVMHashIndexOp.Annotations.JOIN_TYPE, joinType),//
new NV(JVMHashIndexOp.Annotations.JOIN_VARS, joinVars),//
+ new NV(HTreeHashIndexOp.Annotations.PROJECT_IN_VARS, subgroup.getProjectInVars()),//
new NV(JVMHashIndexOp.Annotations.SELECT, selectVars),//
new NV(JVMHashIndexOp.Annotations.NAMED_SET_REF, namedSolutionSet)//
);
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTSubGroupJoinVarOptimizer.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTSubGroupJoinVarOptimizer.java 2013-04-26 16:53:48 UTC (rev 7084)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTSubGroupJoinVarOptimizer.java 2013-04-26 18:26:18 UTC (rev 7085)
@@ -27,6 +27,7 @@
package com.bigdata.rdf.sparql.ast.optimizers;
+import java.util.Arrays;
import java.util.LinkedHashSet;
import java.util.Set;
@@ -126,7 +127,35 @@
final IVariable[] joinVars = boundByGroup.toArray(new IVariable[0]);
group.setJoinVars(joinVars);
+
+ /*
+ * The variables used by the group and its children, including
+ * filters.
+ */
+ final Set<IVariable<?>> usedByGroup = sa
+ .getSpannedVariables(group,
+ true /*filters*/, new LinkedHashSet<IVariable<?>>());
+ /*
+ * Find the set of variables which have appeared in the query and
+ * may be bound by the time the group is evaluated.
+ */
+ final Set<IVariable<?>> maybeIncomingBindings = sa
+ .getMaybeIncomingBindings(
+ (GraphPatternGroup<?>) group,
+ new LinkedHashSet<IVariable<?>>());
+
+ /*
+ * Retain the variables used by the group that have already
+ * appeared previously in the query up to this point.
+ */
+ usedByGroup.retainAll(maybeIncomingBindings);
+
+ @SuppressWarnings("rawtypes")
+ final IVariable[] projectInVars = usedByGroup.toArray(new IVariable[0]);
+
+ group.setProjectInVars(projectInVars);
+
}
/*
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/TestNegation.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/TestNegation.java 2013-04-26 16:53:48 UTC (rev 7084)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/TestNegation.java 2013-04-26 18:26:18 UTC (rev 7085)
@@ -33,6 +33,7 @@
import org.openrdf.model.vocabulary.RDF;
import com.bigdata.bop.IVariable;
+import com.bigdata.bop.Var;
import com.bigdata.rdf.model.BigdataURI;
import com.bigdata.rdf.model.BigdataValue;
import com.bigdata.rdf.model.BigdataValueFactory;
@@ -704,6 +705,7 @@
//
// group.setJoinVars(new IVariable[]{Var.var("ar")});
group.setJoinVars(new IVariable[]{});
+ group.setProjectInVars(new IVariable[]{});
} // end group
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mar...@us...> - 2013-04-26 16:53:54
|
Revision: 7084
http://bigdata.svn.sourceforge.net/bigdata/?rev=7084&view=rev
Author: martyncutcher
Date: 2013-04-26 16:53:48 +0000 (Fri, 26 Apr 2013)
Log Message:
-----------
moved summary logging for postHAComit from INFO to WARN log level
Modified Paths:
--------------
branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2013-04-26 16:50:36 UTC (rev 7083)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2013-04-26 16:53:48 UTC (rev 7084)
@@ -6046,8 +6046,8 @@
}
- if (log.isInfoEnabled())
- log.info("Released: " + totalFreed + " addresses from " + modCount + " modified Allocators");
+ // if (log.isInfoEnabled())
+ log.warn("Released: " + totalFreed + " addresses from " + modCount + " modified Allocators");
if (log.isTraceEnabled()) {
log.trace("OLD BITS: " + BytesUtil.toHexString(oldmetabits));
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mar...@us...> - 2013-04-26 16:50:43
|
Revision: 7083
http://bigdata.svn.sourceforge.net/bigdata/?rev=7083&view=rev
Author: martyncutcher
Date: 2013-04-26 16:50:36 +0000 (Fri, 26 Apr 2013)
Log Message:
-----------
Moved summary logging from Trace to Info level for postHACommit
Modified Paths:
--------------
branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2013-04-26 16:11:42 UTC (rev 7082)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2013-04-26 16:50:36 UTC (rev 7083)
@@ -6046,8 +6046,10 @@
}
- if (log.isTraceEnabled()) {
- log.trace("Checked freed addresses: " + totalFreed);
+ if (log.isInfoEnabled())
+ log.info("Released: " + totalFreed + " addresses from " + modCount + " modified Allocators");
+
+ if (log.isTraceEnabled()) {
log.trace("OLD BITS: " + BytesUtil.toHexString(oldmetabits));
log.trace("NEW BITS: " + BytesUtil.toHexString(m_metaBits));
log.trace("MODDED BITS: " + BytesUtil.toHexString(moddedBits));
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mar...@us...> - 2013-04-26 16:11:52
|
Revision: 7082
http://bigdata.svn.sourceforge.net/bigdata/?rev=7082&view=rev
Author: martyncutcher
Date: 2013-04-26 16:11:42 +0000 (Fri, 26 Apr 2013)
Log Message:
-----------
Complete implementation of postHACommit to incrementally remove entries from the historical index cache
Modified Paths:
--------------
branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/FixedAllocator.java
branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/FixedAllocator.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/FixedAllocator.java 2013-04-26 08:07:48 UTC (rev 7081)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/FixedAllocator.java 2013-04-26 16:11:42 UTC (rev 7082)
@@ -34,6 +34,8 @@
import org.apache.log4j.Logger;
import com.bigdata.btree.BytesUtil;
+import com.bigdata.cache.ConcurrentWeakValueCache;
+import com.bigdata.journal.ICommitter;
import com.bigdata.rawstore.IAllocationContext;
import com.bigdata.rwstore.RWStore.AllocationStats;
import com.bigdata.rwstore.StorageStats.Bucket;
@@ -1247,4 +1249,53 @@
}
+ /*
+ * Checks for allocations committed in xfa that are free in this allocator
+ * and should be removed from the historical external cache.
+ */
+ public int removeFreedWrites(final FixedAllocator xfa,
+ final ConcurrentWeakValueCache<Long, ICommitter> externalCache) {
+ // Compare the committed bits in each AllocBlock
+ int count = 0;
+ for (int i = 0; i < m_allocBlocks.size(); i++) {
+ final AllocBlock ab = m_allocBlocks.get(i);
+ final AllocBlock xab = xfa.m_allocBlocks.get(i);
+ // NOTE that absolute bit offsets are bumped by 3 for historical reasons
+ final int blockBitOffset = 3 + (i * xab.m_commit.length * 32);
+ for (int b = 0; b < xab.m_commit.length; b++) {
+ if (xab.m_commit[b] != ab.m_commit[b]) { // some difference
+ // compute those set in xfa not set in ab (removed)
+ final int removed = xab.m_commit[b] & ~ab.m_commit[b];
+ if (removed != 0) { // something to do
+ // need to test each of 32 bits
+ for (int bit = 0; bit < 32; bit++) {
+ if ((removed & (1 << bit)) != 0) {
+ // Test bit calculation
+ final int tstBit = blockBitOffset + (b * 32) + bit;
+ if (!(xfa.isCommitted(tstBit) && !isCommitted(tstBit))) {
+ log.error("Bit problem: " + tstBit);
+ }
+
+ final long paddr = xfa.getPhysicalAddress(tstBit);
+
+ if (log.isTraceEnabled()) {
+ log.trace("Checking address for removal: " + paddr);
+ }
+
+ count++;
+
+ externalCache.remove(paddr);
+ }
+ }
+ }
+ }
+ }
+ }
+
+ if (log.isTraceEnabled())
+ log.trace("FA index: " + m_index + ", freed: " + count);
+
+ return count;
+ }
+
}
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2013-04-26 08:07:48 UTC (rev 7081)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2013-04-26 16:11:42 UTC (rev 7082)
@@ -31,6 +31,7 @@
import java.io.InputStream;
import java.io.OutputStream;
import java.io.RandomAccessFile;
+import java.lang.ref.WeakReference;
import java.nio.ByteBuffer;
import java.nio.channels.Channel;
import java.nio.channels.ClosedByInterruptException;
@@ -44,6 +45,7 @@
import java.util.Iterator;
import java.util.LinkedHashSet;
import java.util.Map;
+import java.util.Map.Entry;
import java.util.Set;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
@@ -5943,11 +5945,14 @@
m_allocationWriteLock.lock();
try {
// Current FixedAllocators for sanity
- log.warn("POSTHACOMMIT START");
- for (int index = 0; index < m_allocs.size(); index++) {
- final FixedAllocator xfa = m_allocs.get(index);
- log.warn("Allocator " + index + ", startAddress: " + xfa.getStartAddr() + ", allocated: " + xfa.getAllocatedSlots());
- }
+ if (log.isTraceEnabled())
+ {
+ log.trace("POSTHACOMMIT START");
+ for (int index = 0; index < m_allocs.size(); index++) {
+ final FixedAllocator xfa = m_allocs.get(index);
+ log.trace("Allocator " + index + ", size: " + xfa.m_size + ", startAddress: " + xfa.getStartAddr() + ", allocated: " + (xfa.getAllocatedSlots()/xfa.m_size));
+ }
+ }
final ArrayList<FixedAllocator> nallocs = new ArrayList<FixedAllocator>();
@@ -5966,12 +5971,22 @@
}
}
+ if (log.isTraceEnabled()) {
+ StringBuffer sb = new StringBuffer();
+ Iterator<Entry<Long, WeakReference<ICommitter>>> entries = m_externalCache.entryIterator();
+ while (entries.hasNext()) {
+ sb.append(entries.next().getKey() + "|");
+ }
+
+ log.trace("External Cache Start Size: " + m_externalCache.size() + ", entries: " + sb.toString());
+ }
// Stage One: Count moddedBits
// Stage Two: Compute Address of modded bits
// Stage Three: Read Allocator from modded address
// Stage Four: Update Live Allocators
int modCount = 0;
+ int totalFreed = 0;
for (int i = 0; i < moddedBits.length; i+=9) {
final long startAddr = convertAddr(m_metaBits[i]);
for (int j = 1; j < 9; j++) {
@@ -5999,23 +6014,29 @@
boolean found = false;
if (log.isTraceEnabled())
log.trace("Checking allocator at " + nalloc.getStartAddr());
+
for (int index = 0; !found && index < m_allocs.size(); index++) {
final FixedAllocator xfa = m_allocs.get(index);
if (xfa.getStartAddr() == nalloc.getStartAddr()) {
if (log.isTraceEnabled())
log.trace("Found updated allocator at " + index
- + ", size: " + xfa.m_size + " vs " + nalloc.m_size + ", allocated slots: " + xfa.getAllocatedSlots() + " vs " + nalloc.getAllocatedSlots());
+ + ", size: " + xfa.m_size + " vs " + nalloc.m_size + ", allocated slots: " + (xfa.getAllocatedSlots()/xfa.m_size) + " vs " + (nalloc.getAllocatedSlots()/xfa.m_size));
// Compare allocators to see if same
found = true;
// Replace old with new
m_allocs.set(index, nalloc);
+ nalloc.setIndex(index);
// remove old from free list (if set)
xfa.removeFromFreeList();
+
+ // now clear any cached writes now freed
+ totalFreed +=nalloc.removeFreedWrites(xfa, m_externalCache);
}
}
+
if (!found) {
nallocs.add(nalloc);
}
@@ -6024,7 +6045,9 @@
}
}
+
if (log.isTraceEnabled()) {
+ log.trace("Checked freed addresses: " + totalFreed);
log.trace("OLD BITS: " + BytesUtil.toHexString(oldmetabits));
log.trace("NEW BITS: " + BytesUtil.toHexString(m_metaBits));
log.trace("MODDED BITS: " + BytesUtil.toHexString(moddedBits));
@@ -6076,9 +6099,13 @@
}
}
- // KICK external cache into touch - FIXME: handle with improved Allocator synchronization
- m_externalCache.clear();
+ if (log.isTraceEnabled())
+ log.trace("External Cache Pre Clear Size: " + m_externalCache.size());
+ // If FixedAllocator.removeFreedWrites does its job then we do not
+ // need to clear the external cache
+ // m_externalCache.clear();
+
assert m_nextAllocation != 0;
} finally {
m_allocationWriteLock.unlock();
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mar...@us...> - 2013-04-26 08:07:54
|
Revision: 7081
http://bigdata.svn.sourceforge.net/bigdata/?rev=7081&view=rev
Author: martyncutcher
Date: 2013-04-26 08:07:48 +0000 (Fri, 26 Apr 2013)
Log Message:
-----------
Fix array bounds exception in postHaCommit when metabits are extended.
Modified Paths:
--------------
branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2013-04-25 18:37:52 UTC (rev 7080)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2013-04-26 08:07:48 UTC (rev 7081)
@@ -5959,7 +5959,7 @@
// need to compute modded metabits, those newly written slots by ANDing
// new bits with compliment of current
int[] moddedBits = m_metaBits.clone();
- for (int b = 0; b < moddedBits.length; b+=9) {
+ for (int b = 0; b < oldmetabits.length; b+=9) {
// int[0] is startAddr, int[1:9] bits
for (int i = 1; i < 9; i++) {
moddedBits[b+i] &= ~oldmetabits[b+i];
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-04-25 18:37:58
|
Revision: 7080
http://bigdata.svn.sourceforge.net/bigdata/?rev=7080&view=rev
Author: thompsonbry
Date: 2013-04-25 18:37:52 +0000 (Thu, 25 Apr 2013)
Log Message:
-----------
Enabled the SERVLET_CONTEXT_LISTENER_CLASS parameter for the HAJournalServer in startNSS().
See https://sourceforge.net/apps/trac/bigdata/ticket/667 (Provide NanoSparqlServer initialization hook)
Modified Paths:
--------------
branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournalServer.java
Modified: branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournalServer.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournalServer.java 2013-04-25 17:06:50 UTC (rev 7079)
+++ branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournalServer.java 2013-04-25 18:37:52 UTC (rev 7080)
@@ -3600,9 +3600,9 @@
NSSConfigurationOptions.PORT, Integer.TYPE,
NSSConfigurationOptions.DEFAULT_PORT);
-// final String servletContextListenerClass = (String) config.getEntry(COMPONENT,
-// NSSConfigurationOptions.SERVLET_CONTEXT_LISTENER_CLASS, String.class,
-// NSSConfigurationOptions.DEFAULT_SERVLET_CONTEXT_LISTENER_CLASS);
+ final String servletContextListenerClass = (String) config.getEntry(COMPONENT,
+ NSSConfigurationOptions.SERVLET_CONTEXT_LISTENER_CLASS, String.class,
+ NSSConfigurationOptions.DEFAULT_SERVLET_CONTEXT_LISTENER_CLASS);
log.warn("Starting NSS: port=" + port);
@@ -3617,8 +3617,8 @@
// Note: Create will be handled by the QuorumListener (above).
initParams.put(ConfigParams.CREATE, Boolean.toString(create));
-// initParams.put(ConfigParams.SERVLET_CONTEXT_LISTENER_CLASS,
-// servletContextListenerClass);
+ initParams.put(ConfigParams.SERVLET_CONTEXT_LISTENER_CLASS,
+ servletContextListenerClass);
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2013-04-25 17:07:02
|
Revision: 7079
http://bigdata.svn.sourceforge.net/bigdata/?rev=7079&view=rev
Author: thompsonbry
Date: 2013-04-25 17:06:50 +0000 (Thu, 25 Apr 2013)
Log Message:
-----------
Checkpoint on refactor to support heirarchical directory structures for snapshots and HALog files. This is being done primarily for the HALog files. Since there is one file per commit point, there can be quite a few of these files.
- Modified the SnapshotIndex (was CommitTimeIndex) to use the
UnisolatedReadWriteIndex for thread safety and wrote a direct
test suite for the SnapshotIndex. Added the sizeOnDisk to a new
ISnapshotRecord interface whose instances are the values in the
SnapshotIndex. Added a CommitCounterUtility class to ecapsulate
some of the common routines for generating and parsing these
file names. Updated the test suite where we had hard-coded some
of the snapshot file names. Modified the SnapshotManager to use
a recursive scan.
- Extracted the recursiveDelete() methods from HAJournalServer,
HAJournal, and SnapshotManager. I have currently placed this
on CommitCounterUtility, but maybe a better location can be
found. I added a boolean flag indicating whether the method
would throw an IOException if a visited file could not be
removed so we can handle the two use cases (manditory cleanup
throws an exception while best effort cleanup does not).
- HAJournalServer.deleteSnapshots(token,earliestRestorableCommitTime)
was refactored to use the SnapshotIndex rather than scanning
the file system for better efficiency and scalability. The
method was moved into the SnapshotManager.
- I have not yet modified modified the SnapshotManager to
actually generate recursive directory structures for file
names. That would be done by changing the CommitCounterUtility.
I will do that once I get the HALog stuff refactored as well.
All HA tests are green.
Modified Paths:
--------------
branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/halog/IHALogReader.java
branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/Journal.java
branches/READ_CACHE/bigdata/src/java/com/bigdata/service/AbstractHATransactionService.java
branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/DefaultRestorePolicy.java
branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournal.java
branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournalServer.java
branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HARestore.java
branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/SnapshotManager.java
branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/AbstractHA3BackupTestCase.java
branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestAll.java
branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA3RestorePolicy.java
branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA3SnapshotPolicy.java
branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestHA3SnapshotPolicy2.java
branches/READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/HAStatusServletUtil.java
Added Paths:
-----------
branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/CommitCounterUtility.java
branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/SnapshotIndex.java
branches/READ_CACHE/bigdata-jini/src/test/com/bigdata/journal/jini/ha/TestSnapshotIndex.java
Removed Paths:
-------------
branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/CommitTimeIndex.java
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/halog/IHALogReader.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/halog/IHALogReader.java 2013-04-25 16:23:07 UTC (rev 7078)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/ha/halog/IHALogReader.java 2013-04-25 17:06:50 UTC (rev 7079)
@@ -23,6 +23,8 @@
*/
package com.bigdata.ha.halog;
+import java.io.File;
+import java.io.FileFilter;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.security.DigestException;
@@ -44,7 +46,30 @@
*/
public static final String HA_LOG_EXT = ".ha-log";
- /**
+ /**
+ * A {@link FileFilter} that visits all files ending with the
+ * {@link #HA_LOG_EXT} and the names of all direct child directories. This
+ * {@link FileFilter} may be used to establish recursive scans of the HALog
+ * directory.
+ */
+ static public final FileFilter HALOG_FILTER = new FileFilter() {
+
+ @Override
+ public boolean accept(File f) {
+
+ if (f.isDirectory()) {
+
+ return true;
+
+ }
+
+ return f.getName().endsWith(HA_LOG_EXT);
+
+ }
+
+ };
+
+ /**
* Closes the Reader.
*
* @throws IOException
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/Journal.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/Journal.java 2013-04-25 16:23:07 UTC (rev 7078)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/journal/Journal.java 2013-04-25 17:06:50 UTC (rev 7079)
@@ -716,7 +716,7 @@
final private AtomicReference<BarrierState> barrierRef = new AtomicReference<BarrierState>();
@Override
- public void executeWithBarrierLock(final Runnable r) {
+ public void runWithBarrierLock(final Runnable r) {
barrierLock.lock();
try {
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/service/AbstractHATransactionService.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/service/AbstractHATransactionService.java 2013-04-25 16:23:07 UTC (rev 7078)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/service/AbstractHATransactionService.java 2013-04-25 17:06:50 UTC (rev 7079)
@@ -58,6 +58,6 @@
/**
* Used to make a serviceJoin() MUTEX with the consensus protocol.
*/
- abstract public void executeWithBarrierLock(Runnable r);
+ abstract public void runWithBarrierLock(Runnable r);
}
Added: branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/CommitCounterUtility.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/CommitCounterUtility.java (rev 0)
+++ branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/CommitCounterUtility.java 2013-04-25 17:06:50 UTC (rev 7079)
@@ -0,0 +1,198 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+package com.bigdata.journal.jini.ha;
+
+import java.io.File;
+import java.io.FileFilter;
+import java.io.IOException;
+import java.util.Formatter;
+
+import org.apache.log4j.Logger;
+
+/**
+ * Utility class for operations on files that are named using a commit counter.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ */
+public class CommitCounterUtility {
+
+ private static final Logger log = Logger
+ .getLogger(CommitCounterUtility.class);
+
+ /**
+ * Return the name of the {@link File} associated with the commitCounter.
+ *
+ * @param dir
+ * The directory spanning all such files.
+ * @param commitCounter
+ * The commit counter for the current root block on the journal.
+ * @param ext
+ * The filename extension.
+ * @return The name of the corresponding snapshot file.
+ */
+ public static File getCommitCounterFile(final File dir,
+ final long commitCounter, final String ext) {
+
+ /*
+ * Format the name of the file.
+ *
+ * Note: The commit counter in the file name should be zero filled to 20
+ * digits so we have the files in lexical order in the file system (for
+ * convenience).
+ */
+ final String file;
+ {
+
+ final StringBuilder sb = new StringBuilder();
+
+ final Formatter f = new Formatter(sb);
+
+ f.format("%020d" + ext, commitCounter);
+ f.flush();
+ f.close();
+
+ file = sb.toString();
+
+ }
+
+ return new File(dir, file);
+
+ }
+
+ /**
+ * Parse out the commitCounter from the file name.
+ *
+ * @param name
+ * The file name
+ * @param ext
+ * The expected file extension.
+ *
+ * @return The commit counter from the file name.
+ *
+ * @throws IllegalArgumentException
+ * if either argument is <code>null</code>
+ * @throws NumberFormatException
+ * if the file name can not be interpreted as a commit counter.
+ */
+ public static long parseCommitCounterFile(final String name,
+ final String ext) throws NumberFormatException {
+
+ if (name == null)
+ throw new IllegalArgumentException();
+
+ if (ext == null)
+ throw new IllegalArgumentException();
+
+ // Strip off the filename extension.
+ final int len = name.length() - ext.length();
+
+ final String fileBaseName = name.substring(0, len);
+
+ // Closing commitCounter for snapshot file.
+ final long commitCounter = Long.parseLong(fileBaseName);
+
+ return commitCounter;
+
+ }
+
+ /**
+ * Return the basename of the file (strip off the extension).
+ *
+ * @param name
+ * The file name.
+ * @param ext
+ * The extension.
+ *
+ * @return The base name of the file without the extension.
+ */
+ public static String getBaseName(final String name, final String ext) {
+
+ final String basename = name.substring(0, name.length() - ext.length());
+
+ return basename;
+
+ }
+
+ /**
+ * Recursively removes any files and subdirectories and then removes the
+ * file (or directory) itself. Only files recognized by
+ * {@link #getFileFilter()} will be deleted.
+ * <p>
+ * Note: A dedicated version of this method exists here to thrown an
+ * {@link IOException} if we can not delete a file. This is deliberate. It
+ * is thrown to prevent a REBUILD from proceeding unless we can clear out
+ * the old snapshot and HALog files.
+ *
+ * @param errorIfDeleteFails
+ * When <code>true</code> and {@link IOException} is thrown if a
+ * file matching the filter or an empty directory matching the
+ * filter can not be removed. When <code>false</code>, that event
+ * is logged @ WARN instead.
+ * @param f
+ * A file or directory.
+ * @param fileFilter
+ * A filter matching the files and directories to be visited and
+ * removed. If directories are matched, then they will be removed
+ * iff they are empty. A depth first visitation is used, so the
+ * files and sub-directories will be cleared before we attempt to
+ * remove the parent directory.
+ * @throws IOException
+ * if any file or non-empty directory can not be deleted (iff
+ * <i>errorIfDeleteFails</i> is <code>true</code>).
+ */
+ public static void recursiveDelete(final boolean errorIfDeleteFails,
+ final File f, final FileFilter fileFilter) throws IOException {
+
+ if (f.isDirectory()) {
+
+ final File[] children = f.listFiles(fileFilter);
+
+ for (int i = 0; i < children.length; i++) {
+
+ recursiveDelete(errorIfDeleteFails, children[i], fileFilter);
+
+ }
+
+ }
+
+ if (log.isInfoEnabled())
+ log.info("Removing: " + f);
+
+ if (f.exists() && !f.delete()) {
+ if (f.isDirectory() && f.list().length != 0) {
+ // Ignore non-empty directory.
+ return;
+ }
+ final String msg = "Could not remove file: " + f;
+ if (errorIfDeleteFails) {
+ // Complete if we can not delete a file.
+ throw new IOException(msg);
+ } else {
+ log.warn(msg);
+ }
+ }
+
+ }
+
+}
Deleted: branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/CommitTimeIndex.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/CommitTimeIndex.java 2013-04-25 16:23:07 UTC (rev 7078)
+++ branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/CommitTimeIndex.java 2013-04-25 17:06:50 UTC (rev 7079)
@@ -1,548 +0,0 @@
-/**
-
-Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
-package com.bigdata.journal.jini.ha;
-
-import java.io.IOException;
-import java.io.ObjectInput;
-import java.io.ObjectOutput;
-import java.nio.ByteBuffer;
-import java.util.UUID;
-
-import com.bigdata.btree.BTree;
-import com.bigdata.btree.BytesUtil;
-import com.bigdata.btree.Checkpoint;
-import com.bigdata.btree.DefaultTupleSerializer;
-import com.bigdata.btree.IRangeQuery;
-import com.bigdata.btree.ITuple;
-import com.bigdata.btree.ITupleIterator;
-import com.bigdata.btree.IndexMetadata;
-import com.bigdata.btree.keys.ASCIIKeyBuilderFactory;
-import com.bigdata.btree.keys.IKeyBuilder;
-import com.bigdata.btree.keys.IKeyBuilderFactory;
-import com.bigdata.btree.keys.KeyBuilder;
-import com.bigdata.journal.ICommitRecord;
-import com.bigdata.journal.IRootBlockView;
-import com.bigdata.journal.RootBlockView;
-import com.bigdata.rawstore.Bytes;
-import com.bigdata.rawstore.IRawStore;
-import com.bigdata.util.ChecksumUtility;
-
-/**
- * {@link BTree} mapping <em>commitTime</em> (long integers) to
- * {@link IRootBlockView} records.
- * <p>
- * Note: Access to this object MUST be synchronized.
- * <p>
- * Note: This is used as a transient data structure that is populated from the
- * file system by the {@link HAJournalServer}. A separate instance is maintained
- * for the HALog files and the snapshot files.
- */
-public class CommitTimeIndex extends BTree {
-
- /**
- * Instance used to encode the timestamp into the key.
- */
- final private IKeyBuilder keyBuilder = new KeyBuilder(Bytes.SIZEOF_LONG);
-
- /**
- * Create a transient instance.
- *
- * @return The new instance.
- */
- static public CommitTimeIndex createTransient() {
-
- final IndexMetadata metadata = new IndexMetadata(UUID.randomUUID());
-
- metadata.setBTreeClassName(CommitTimeIndex.class.getName());
-
- metadata.setTupleSerializer(new TupleSerializer(
- new ASCIIKeyBuilderFactory(Bytes.SIZEOF_LONG)));
-
- return (CommitTimeIndex) BTree.createTransient(/*store, */metadata);
-
- }
-
- /**
- * Load from the store.
- *
- * @param store
- * The backing store.
- * @param checkpoint
- * The {@link Checkpoint} record.
- * @param metadata
- * The metadata record for the index.
- */
- public CommitTimeIndex(final IRawStore store, final Checkpoint checkpoint,
- final IndexMetadata metadata, final boolean readOnly) {
-
- super(store, checkpoint, metadata, readOnly);
-
- }
-
- /**
- * Encodes the commit time into a key.
- *
- * @param commitTime
- * The commit time.
- *
- * @return The corresponding key.
- */
- private byte[] getKey(final long commitTime) {
-
- return keyBuilder.reset().append(commitTime).getKey();
-
- }
-
- /**
- * Return the {@link IRootBlock} identifying the journal having the largest
- * commitTime that is less than or equal to the given timestamp. This is
- * used primarily to locate the commit record that will serve as the ground
- * state for a transaction having <i>timestamp</i> as its start time. In
- * this context the LTE search identifies the most recent commit state that
- * not later than the start time of the transaction.
- *
- * @param timestamp
- * The given timestamp.
- *
- * @return The description of the relevant journal resource -or-
- * <code>null</code> iff there are no journals in the index that
- * satisify the probe.
- *
- * @throws IllegalArgumentException
- * if <i>timestamp</i> is less than or equals to ZERO (0L).
- */
- synchronized public IRootBlockView find(final long timestamp) {
-
- if (timestamp <= 0L)
- throw new IllegalArgumentException();
-
- // find (first less than or equal to).
- final long index = findIndexOf(timestamp);
-
- if(index == -1) {
-
- // No match.
- return null;
-
- }
-
- return valueAtIndex(index);
-
- }
-
- /**
- * Retrieve the entry from the index.
- */
- private IRootBlockView valueAtIndex(final long index) {
-
- final byte[] val = super.valueAt(index);
-
- assert val != null : "Entry has null value: index=" + index;
-
- final IRootBlockView entry = new RootBlockView(false/* rootBlock0 */,
- ByteBuffer.wrap(val), ChecksumUtility.getCHK());
-
- return entry;
-
- }
-
- /**
- * Return the {@link IRootBlockView} identifying the first snapshot whose
- * <em>commitTime</em> is strictly greater than the timestamp.
- *
- * @param timestamp
- * The timestamp. A value of ZERO (0) may be used to find the
- * first snapshot.
- *
- * @return The root block of that snapshot -or- <code>null</code> if there
- * is no snapshot whose timestamp is strictly greater than
- * <i>timestamp</i>.
- */
- synchronized public IRootBlockView findNext(final long timestamp) {
-
- /*
- * Note: can also be written using rangeIterator().next().
- */
-
- if (timestamp < 0L)
- throw new IllegalArgumentException();
-
- // find first strictly greater than.
- final long index = findIndexOf(timestamp) + 1;
-
- if (index == nentries) {
-
- // No match.
-
- return null;
-
- }
-
- return valueAtIndex(index);
-
- }
-
- /**
- * Find the index of the {@link ICommitRecord} having the largest timestamp
- * that is less than or equal to the given timestamp.
- *
- * @return The index of the {@link ICommitRecord} having the largest
- * timestamp that is less than or equal to the given timestamp -or-
- * <code>-1</code> iff there are no {@link ICommitRecord}s
- * defined.
- */
- synchronized public long findIndexOf(final long timestamp) {
-
- long pos = super.indexOf(getKey(timestamp));
-
- if (pos < 0) {
-
- /*
- * the key lies between the entries in the index, or possible before
- * the first entry in the index. [pos] represents the insert
- * position. we convert it to an entry index and subtract one to get
- * the index of the first commit record less than the given
- * timestamp.
- */
-
- pos = -(pos+1);
-
- if (pos == 0) {
-
- // No entry is less than or equal to this timestamp.
- return -1;
-
- }
-
- pos--;
-
- return pos;
-
- } else {
-
- /*
- * exact hit on an entry.
- */
-
- return pos;
-
- }
-
- }
-
- /**
- * Add an entry under the commitTime associated with the
- * {@link IRootBlockView} record.
- *
- * @param rootBlock
- * The {@link IRootBlockView} record.
- *
- * @exception IllegalArgumentException
- * if <i>commitTime</i> is <code>0L</code>.
- * @exception IllegalArgumentException
- * if <i>rootBLock</i> is <code>null</code>.
- * @exception IllegalArgumentException
- * if there is already an entry registered under for the
- * given timestamp.
- */
- synchronized public void add(final IRootBlockView rootBlock) {
-
- if (rootBlock == null)
- throw new IllegalArgumentException();
-
- final long createTime = rootBlock.getLastCommitTime();
-
- if (createTime == 0L)
- throw new IllegalArgumentException();
-
- final byte[] key = getKey(createTime);
-
- if(super.contains(key)) {
-
- throw new IllegalArgumentException("entry exists: timestamp="
- + createTime);
-
- }
-
- // add a serialized entry to the persistent index.
- super.insert(key, BytesUtil.getBytes(rootBlock.asReadOnlyBuffer()));
-
- }
-
- /**
- * Find and return the {@link IRootBlockView} for the oldest snapshot (if
- * any).
- *
- * @return That {@link IRootBlockView} -or- <code>null</code> if there are
- * no snapshots.
- */
- public IRootBlockView getOldestSnapshot() {
-
- synchronized (this) {
-
- if (getEntryCount() == 0L) {
-
- // Empty index.
- return null;
-
- }
-
- // Lookup first tuple in index.
- @SuppressWarnings("unchecked")
- final ITuple<IRootBlockView> t = valueAt(0L, getLookupTuple());
-
- final IRootBlockView rb = t.getObject();
-
- return rb;
-
- }
-
- }
-
- /**
- * Find the {@link IRootBlockView} for the most recent snapshot (if any).
- *
- * @return That {@link IRootBlockView} -or- <code>null</code> if there are
- * no snapshots.
- */
- public IRootBlockView getNewestSnapshot() {
-
- /*
- * Note: This could also be written using valueAt(nentries).
- */
- synchronized (this) {
-
- @SuppressWarnings("unchecked")
- final ITupleIterator<IRootBlockView> itr = rangeIterator(
- null/* fromKey */, null/* toKey */, 1/* capacity */,
- IRangeQuery.DEFAULT | IRangeQuery.REVERSE/* flags */, null/* filter */);
-
- if (!itr.hasNext()) {
-
- return null;
-
- }
-
- final ITuple<IRootBlockView> t = itr.next();
-
- final IRootBlockView rb = t.getObject();
-
- return rb;
-
- }
-
- }
-
- /**
- * Find the oldest snapshot whose commit counter is LTE the specified commit
- * counter.
- *
- * @return The {@link IRootBlockView} for that snapshot -or-
- * <code>null</code> if there is no such snapshot.
- *
- * @throws IllegalArgumentException
- * if <code>commitCounter LT ZERO (0)</code>
- */
- public IRootBlockView findByCommitCounter(final long commitCounter) {
-
- if (commitCounter < 0L)
- throw new IllegalArgumentException();
-
- synchronized (this) {
-
- // Reverse scan.
- @SuppressWarnings("unchecked")
- final ITupleIterator<IRootBlockView> itr = rangeIterator(
- null/* fromKey */, null/* toKey */, 0/* capacity */,
- IRangeQuery.DEFAULT | IRangeQuery.REVERSE/* flags */, null/* filter */);
-
- while (itr.hasNext()) {
-
- final ITuple<IRootBlockView> t = itr.next();
-
- final IRootBlockView rb = t.getObject();
-
- if (rb.getCommitCounter() <= commitCounter) {
-
- // First snapshot LTE that commit counter.
- return rb;
-
- }
-
- }
-
- return null;
-
- }
-
- }
-
- /**
- * Return the snapshot that is associated with the specified ordinal index
- * (origin ZERO) counting backwards from the most recent snapshot (0)
- * towards the earliest snapshot (nsnapshots-1).
- * <p>
- * Note: The effective index is given by <code>(entryCount-1)-index</code>.
- * If the effective index is LT ZERO (0) then there is no such snapshot and
- * this method will return <code>null</code>.
- *
- * @param index
- * The index.
- *
- * @return The {@link IRootBlockView} for that snapshot -or-
- * <code>null</code> if there is no such snapshot.
- *
- * @throws IllegalArgumentException
- * if <code>index LT ZERO (0)</code>
- */
- public IRootBlockView getSnapshotByReverseIndex(final int index) {
-
- if (index < 0)
- throw new IllegalArgumentException();
-
- synchronized (this) {
-
- final long entryCount = getEntryCount();
-
- if (entryCount > Integer.MAX_VALUE)
- throw new AssertionError();
-
- final int effectiveIndex = ((int) entryCount - 1) - index;
-
- if (effectiveIndex < 0) {
-
- // No such snapshot.
- return null;
-
- }
-
- @SuppressWarnings("unchecked")
- final ITuple<IRootBlockView> t = valueAt(effectiveIndex,
- getLookupTuple());
-
- final IRootBlockView rb = t.getObject();
-
- return rb;
-
- }
-
- }
-
- /**
- * Encapsulates key and value formation.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id: JournalIndex.java 5892 2012-01-27 13:22:11Z thompsonbry $
- */
- static protected class TupleSerializer extends
- DefaultTupleSerializer<Long, IRootBlockView> {
-
- /**
- *
- */
- private static final long serialVersionUID = -2851852959439807542L;
-
- /**
- * De-serialization ctor.
- */
- public TupleSerializer() {
-
- super();
-
- }
-
- /**
- * Ctor when creating a new instance.
- *
- * @param keyBuilderFactory
- */
- public TupleSerializer(final IKeyBuilderFactory keyBuilderFactory) {
-
- super(keyBuilderFactory);
-
- }
-
- /**
- * Decodes the key as a commit time.
- */
- @Override
- public Long deserializeKey(final ITuple tuple) {
-
- return KeyBuilder
- .decodeLong(tuple.getKeyBuffer().array(), 0/* offset */);
-
- }
-
- /**
- * De-serializes an object from the {@link ITuple#getValue() value} stored
- * in the tuple (ignores the key stored in the tuple).
- */
- public IRootBlockView deserialize(final ITuple tuple) {
-
- if (tuple == null)
- throw new IllegalArgumentException();
-
- return (IRootBlockView) new RootBlockView(false/* rootBlock0 */,
- ByteBuffer.wrap(tuple.getValue()), ChecksumUtility.getCHK());
-
- }
-
- /**
- * The initial version (no additional persistent state).
- */
- private final static transient byte VERSION0 = 0;
-
- /**
- * The current version.
- */
- private final static transient byte VERSION = VERSION0;
-
- public void readExternal(final ObjectInput in) throws IOException,
- ClassNotFoundException {
-
- super.readExternal(in);
-
- final byte version = in.readByte();
-
- switch (version) {
- case VERSION0:
- break;
- default:
- throw new UnsupportedOperationException("Unknown version: "
- + version);
- }
-
- }
-
- public void writeExternal(final ObjectOutput out) throws IOException {
-
- super.writeExternal(out);
-
- out.writeByte(VERSION);
-
- }
-
- }
-
-}
Modified: branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/DefaultRestorePolicy.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/DefaultRestorePolicy.java 2013-04-25 16:23:07 UTC (rev 7078)
+++ branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/DefaultRestorePolicy.java 2013-04-25 17:06:50 UTC (rev 7079)
@@ -28,6 +28,7 @@
import org.apache.log4j.Logger;
import com.bigdata.journal.IRootBlockView;
+import com.bigdata.journal.jini.ha.SnapshotIndex.ISnapshotRecord;
/**
* The default restore policy. This policy supports three different criteria for
@@ -233,7 +234,8 @@
final long then = now - minRestoreAgeMillis;
// The root block for the snapshot with a commitTime LTE [then].
- final IRootBlockView rootBlock = jnl.getSnapshotManager().find(then);
+ final ISnapshotRecord sr = jnl.getSnapshotManager().find(then);
+ final IRootBlockView rootBlock = sr == null ? null : sr.getRootBlock();
if (rootBlock == null) {
@@ -258,17 +260,17 @@
private long getEarliestRestorableCommitCounterBySnapshots(
final HAJournal jnl, final long commitCounterOnJournal) {
- final IRootBlockView rootBlock = jnl.getSnapshotManager()
+ final ISnapshotRecord r = jnl.getSnapshotManager()
.getSnapshotByReverseIndex(minSnapshots - 1);
- if (rootBlock == null) {
+ if (r == null) {
// There are fewer than minSnapshots snapshots.
return 0L;
}
- return rootBlock.getCommitCounter();
+ return r.getRootBlock().getCommitCounter();
}
@@ -292,16 +294,16 @@
}
// Find the oldest snapshot LTE that commitCounter.
- final IRootBlockView rootBlock = jnl.getSnapshotManager()
- .findByCommitCounter(desiredCommitCounter);
+ final ISnapshotRecord r = jnl.getSnapshotManager().findByCommitCounter(
+ desiredCommitCounter);
- if (rootBlock == null) {
+ if (r == null) {
return commitCounterOnJournal;
}
- return rootBlock.getCommitCounter();
+ return r.getRootBlock().getCommitCounter();
}
Modified: branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournal.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournal.java 2013-04-25 16:23:07 UTC (rev 7078)
+++ branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournal.java 2013-04-25 17:06:50 UTC (rev 7079)
@@ -326,7 +326,9 @@
if (!haLogDir.exists()) {
// Create the directory.
- haLogDir.mkdirs();
+ if (!haLogDir.mkdirs())
+ throw new IOException("Could not create directory: "
+ + haLogDir);
}
@@ -519,52 +521,33 @@
});
recursiveDelete(getSnapshotManager().getSnapshotDir(),
- new FileFilter() {
-
- @Override
- public boolean accept(File f) {
+ SnapshotManager.SNAPSHOT_FILTER);
- if (f.isDirectory())
- return true;
-
- return f.getName().endsWith(SnapshotManager.SNAPSHOT_EXT);
- }
-
- });
-
}
/**
* Recursively removes any files and subdirectories and then removes the
- * file (or directory) itself. Only files recognized by
- * {@link #getFileFilter()} will be deleted.
+ * file (or directory) itself. Only files recognized by filter will be
+ * deleted.
*
* @param f
* A file or directory.
+ * @param fileFilter
+ * The filter.
*/
- private void recursiveDelete(final File f,final FileFilter fileFilter) {
+ private void recursiveDelete(final File f, final FileFilter fileFilter) {
- if (f.isDirectory()) {
-
- final File[] children = f.listFiles(fileFilter);
-
- for (int i = 0; i < children.length; i++) {
-
- recursiveDelete(children[i], fileFilter);
-
- }
-
+ try {
+ CommitCounterUtility.recursiveDelete(false/* errorIfDeleteFails */, f,
+ fileFilter);
+ } catch (IOException e) {
+ /*
+ * Note: IOException is not thrown here since
+ * errorIfDeleteFails:=false.
+ */
+ throw new RuntimeException(e);
}
- if (log.isInfoEnabled())
- log.info("Removing: " + f);
-
- if (f.exists() && !f.delete()) {
-
- log.warn("Could not remove: " + f);
-
- }
-
}
/**
@@ -645,7 +628,7 @@
// The commit counter of the desired closing root block.
final long commitCounter = msg.getCommitCounter();
- final File logFile = new File(haLogDir,
+ final File logFile = new File(getHALogDir(),
HALogWriter.getHALogFileName(commitCounter));
if (!logFile.exists()) {
Modified: branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournalServer.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournalServer.java 2013-04-25 16:23:07 UTC (rev 7078)
+++ branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HAJournalServer.java 2013-04-25 17:06:50 UTC (rev 7079)
@@ -24,6 +24,7 @@
package com.bigdata.journal.jini.ha;
import java.io.File;
+import java.io.FileFilter;
import java.io.FileNotFoundException;
import java.io.FilenameFilter;
import java.io.IOException;
@@ -2864,7 +2865,7 @@
* TODO What happens if we are blocked here?
*/
((AbstractHATransactionService) journal.getTransactionService())
- .executeWithBarrierLock(new Runnable() {
+ .runWithBarrierLock(new Runnable() {
public void run() {
@@ -3099,8 +3100,9 @@
// Delete snapshots, returning commit counter of the oldest
// retained snapshot.
- final long earliestRetainedSnapshotLastCommitCounter = deleteSnapshots(
- token, earliestRestorableCommitPoint);
+ final long earliestRetainedSnapshotLastCommitCounter = journal
+ .getSnapshotManager().deleteSnapshots(token,
+ earliestRestorableCommitPoint);
// Delete HALogs not retained by that snapshot.
deleteHALogs(token, earliestRetainedSnapshotLastCommitCounter);
@@ -3113,158 +3115,154 @@
}
- /**
- * Delete snapshots that are no longer required.
- * <p>
- * Note: If ZERO (0) is passed into this method, then no snapshots will
- * be deleted. This is because the first possible commit counter is ONE
- * (1).
- *
- * @param earliestRestorableCommitPoint
- * The earliest commit point that we need to be able to
- * restore from local backups.
- *
- * @return The commitCounter of the earliest retained snapshot.
- */
- private long deleteSnapshots(final long token,
- final long earliestRestorableCommitPoint) {
- /*
- * List the snapshot files for this service.
- */
- final File[] files;
- // #of snapshot files found. Set during scan.
- final AtomicLong nfound = new AtomicLong();
- // Set to the commit counter of the earliest retained snapshot.
- final AtomicLong earliestRetainedSnapshotCommitCounter = new AtomicLong(Long.MAX_VALUE);
- final SnapshotManager snapshotManager = journal
- .getSnapshotManager();
- {
-
- final File snapshotDir = snapshotManager.getSnapshotDir();
-
- files = snapshotDir.listFiles(new FilenameFilter() {
-
- /**
- * Return <code>true</code> iff the file is an snapshot file
- * that should be deleted.
- *
- * @param name
- * The name of that file (encodes the
- * commitCounter).
- */
- @Override
- public boolean accept(final File dir, final String name) {
-
- if (!name.endsWith(SnapshotManager.SNAPSHOT_EXT)) {
- // Not an snapshot file.
- return false;
- }
-
- // Strip off the filename extension.
- final int len = name.length()
- - SnapshotManager.SNAPSHOT_EXT.length();
- final String fileBaseName = name.substring(0, len);
-
- // Closing commitCounter for snapshot file.
- final long commitCounter = Long.parseLong(fileBaseName);
-
- // Count all snapshot files.
- nfound.incrementAndGet();
-
- // true iff we will delete this snapshot.
- final boolean deleteFile = commitCounter < earliestRestorableCommitPoint;
-
- if (haLog.isInfoEnabled())
- log.info("snapshotFile="
- + name//
- + ", deleteFile="
- + deleteFile//
- + ", commitCounter="
- + commitCounter//
- + ", earliestRestoreableCommitPoint="
- + earliestRestorableCommitPoint);
-
- if (!deleteFile
- && commitCounter < earliestRetainedSnapshotCommitCounter
- .get()) {
-
- /*
- * Update the earliest retained snapshot.
- */
-
- earliestRetainedSnapshotCommitCounter
- .set(commitCounter);
-
- }
-
- return deleteFile;
-
- }
- });
-
- }
-
- int ndeleted = 0;
- long totalBytes = 0L;
-
- /*
- * If people specify NoSnapshotPolicy then backup is in their hands.
- * HALogs will not be retained beyond a fully met commit unless
- * there is a snapshot against which they can be applied..
- */
-
-// if (files.length == 0) {
+// /**
+// * Delete snapshots that are no longer required.
+// * <p>
+// * Note: If ZERO (0) is passed into this method, then no snapshots will
+// * be deleted. This is because the first possible commit counter is ONE
+// * (1).
+// *
+// * @param earliestRestorableCommitPoint
+// * The earliest commit point that we need to be able to
+// * restore from local backups.
+// *
+// * @return The commitCounter of the earliest retained snapshot.
+// */
+// private long deleteSnapshots(final long token,
+// final long earliestRestorableCommitPoint) {
+// /*
+// * List the snapshot files for this service.
+// */
+// final File[] files;
+// // #of snapshot files found. Set during scan.
+// final AtomicLong nfound = new AtomicLong();
+// // Set to the commit counter of the earliest retained snapshot.
+// final AtomicLong earliestRetainedSnapshotCommitCounter = new AtomicLong(Long.MAX_VALUE);
+// final SnapshotManager snapshotManager = journal
+// .getSnapshotManager();
+// {
//
-// /*
-// * Note: If there are no snapshots then we MUST retain ALL HALog
-// * files.
-// */
-// earliestRetainedSnapshotCommitCounter.set(0L);
-//
-// } else {
-
- for (File file : files) {
-
- // #of bytes in that file.
- final long len = file.length();
-
- if (!getQuorum().isQuorumFullyMet(token)) {
- /*
- * Halt operation.
- *
- * Note: This is not an error, but we can not remove
- * snapshots or HALogs if this invariant is violated.
- */
- break;
- }
-
- if (!snapshotManager.removeSnapshot(file)) {
-
- haLog.warn("COULD NOT DELETE FILE: " + file);
-
- continue;
-
- }
-
- ndeleted++;
-
- totalBytes += len;
-
- }
-
+// final File snapshotDir = snapshotManager.getSnapshotDir();
+//
+// files = snapshotDir.listFiles(new FilenameFilter() {
+//
+// /**
+// * Return <code>true</code> iff the file is an snapshot file
+// * that should be deleted.
+// *
+// * @param name
+// * The name of that file (encodes the
+// * commitCounter).
+// */
+// @Override
+// public boolean accept(final File dir, final String name) {
+//
+// if (!name.endsWith(SnapshotManager.SNAPSHOT_EXT)) {
+// // Not an snapshot file.
+// return false;
+// }
+//
+// // Closing commitCounter for snapshot file.
+// final long commitCounter = SnapshotManager
+// .parseCommitCounterFile(name);
+//
+// // Count all snapshot files.
+// nfound.incrementAndGet();
+//
+// // true iff we will delete this snapshot.
+// final boolean deleteFile = commitCounter < earliestRestorableCommitPoint;
+//
+// if (haLog.isInfoEnabled())
+// log.info("snapshotFile="
+// + name//
+// + ", deleteFile="
+// + deleteFile//
+// + ", commitCounter="
+// + commitCounter//
+// + ", earliestRestoreableCommitPoint="
+// + earliestRestorableCommitPoint);
+//
+// if (!deleteFile
+// && commitCounter < earliestRetainedSnapshotCommitCounter
+// .get()) {
+//
+// /*
+// * Update the earliest retained snapshot.
+// */
+//
+// earliestRetainedSnapshotCommitCounter
+// .set(commitCounter);
+//
+// }
+//
+// return deleteFile;
+//
+// }
+// });
+//
// }
-
- if (haLog.isInfoEnabled())
- haLog.info("PURGED SNAPSHOTS: nfound=" + nfound + ", ndeleted="
- + ndeleted + ", totalBytes=" + totalBytes
- + ", earliestRestorableCommitPoint="
- + earliestRestorableCommitPoint
- + ", earliestRetainedSnapshotCommitCounter="
- + earliestRetainedSnapshotCommitCounter.get());
-
- return earliestRetainedSnapshotCommitCounter.get();
-
- }
+//
+// int ndeleted = 0;
+// long totalBytes = 0L;
+//
+// /*
+// * If people specify NoSnapshotPolicy then backup is in their hands.
+// * HALogs will not be retained beyond a fully met commit unless
+// * there is a snapshot against which they can be applied..
+// */
+//
+//// if (files.length == 0) {
+////
+//// /*
+//// * Note: If there are no snapshots then we MUST retain ALL HALog
+//// * files.
+//// */
+//// earliestRetainedSnapshotCommitCounter.set(0L);
+////
+//// } else {
+//
+// for (File file : files) {
+//
+// // #of bytes in that file.
+// final long len = file.length();
+//
+// if (!getQuorum().isQuorumFullyMet(token)) {
+// /*
+// * Halt operation.
+// *
+// * Note: This is not an error, but we can not remove
+// * snapshots or HALogs if this invariant is violated.
+// */
+// break;
+// }
+//
+// if (!snapshotManager.removeSnapshot(file)) {
+//
+// haLog.warn("COULD NOT DELETE FILE: " + file);
+//
+// continue;
+//
+// }
+//
+// ndeleted++;
+//
+// totalBytes += len;
+//
+// }
+//
+//// }
+//
+// if (haLog.isInfoEnabled())
+// haLog.info("PURGED SNAPSHOTS: nfound=" + nfound + ", ndeleted="
+// + ndeleted + ", totalBytes=" + totalBytes
+// + ", earliestRestorableCommitPoint="
+// + earliestRestorableCommitPoint
+// + ", earliestRetainedSnapshotCommitCounter="
+// + earliestRetainedSnapshotCommitCounter.get());
+//
+// return earliestRetainedSnapshotCommitCounter.get();
+//
+// }
/**
* Delete HALogs that are no longer required.
@@ -3415,42 +3413,45 @@
haLog.warn("Destroying local backups.");
// Delete all snapshots.
- {
-
- final File snapshotDir = journal.getSnapshotManager()
- .getSnapshotDir();
-
- final File[] files = snapshotDir
- .listFiles(new FilenameFilter() {
- @Override
- public boolean accept(final File dir,
- final String name) {
- return name
- .endsWith(SnapshotManager.SNAPSHOT_EXT);
- }
- });
- for (File file : files) {
- if (!file.delete())
- throw new IOException("COULD NOT DELETE FILE: "
- + file);
- }
+ journal.getSnapshotManager().deleteAllSnapshots();
- }
-
// Delete all HALogs (except the current one).
- {
+ deleteAllHALogsExceptCurrent();
+
+ } finally {
- final File currentLogFile = journal.getHALogWriter()
- .getFile();
+ logLock.unlock();
- final String currentLogFileName = currentLogFile == null ? null
- : currentLogFile.getName();
+ }
+
+ }
- final File logDir = journal.getHALogDir();
+ /**
+ * Delete all HALog files (except the current one).
+ *
+ * @throws IOException
+ */
+ private void deleteAllHALogsExceptCurrent() throws IOException {
+
+ final File currentLogFile = journal.getHALogWriter()
+ .getFile();
- final File[] files = logDir.listFiles(new FilenameFilter() {
+ final String currentLogFileName = currentLogFile == null ? null
+ : currentLogFile.getName();
+
+ final File logDir = journal.getHALogDir();
+
+ CommitCounterUtility.recursiveDelete(true/* errorIfDeleteFails */,
+ logDir, new FileFilter() {
+
@Override
- public boolean accept(final File dir, final String name) {
+ public boolean accept(final File f) {
+
+ if (f.isDirectory())
+ return true;
+
+ final String name = f.getName();
+
// filter out the current log file
if (currentLogFile != null
&& name.equals(currentLogFileName)) {
@@ -3459,22 +3460,12 @@
*/
return false;
}
+
return name.endsWith(IHALogReader.HA_LOG_EXT);
+
}
- });
- for (File file : files) {
- if (!file.delete())
- throw new IOException("COULD NOT DELETE FILE: "
- + file);
- }
-
- }
-
- } finally {
- logLock.unlock();
-
- }
+ });
}
@@ -3609,6 +3600,10 @@
NSSConfigurationOptions.PORT, Integer.TYPE,
NSSConfigurationOptions.DEFAULT_PORT);
+// final String servletContextListenerClass = (String) config.getEntry(COMPONENT,
+// NSSConfigurationOptions.SERVLET_CONTEXT_LISTENER_CLASS, String.class,
+// NSSConfigurationOptions.DEFAULT_SERVLET_CONTEXT_LISTENER_CLASS);
+
log.warn("Starting NSS: port=" + port);
final Map<String, String> initParams = new LinkedHashMap<String, String>();
@@ -3622,6 +3617,9 @@
// Note: Create will be handled by the QuorumListener (above).
initParams.put(ConfigParams.CREATE, Boolean.toString(create));
+// initParams.put(ConfigParams.SERVLET_CONTEXT_LISTENER_CLASS,
+// servletContextListenerClass);
+
}
if (jettyServer != null && jettyServer.isRunning()) {
Modified: branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HARestore.java
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HARestore.java 2013-04-25 16:23:07 UTC (rev 7078)
+++ branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/HARestore.java 2013-04-25 17:06:50 UTC (rev 7079)
@@ -457,13 +457,11 @@
// source is the snapshot.
final File in = journalFile;
- final String basename = journalFile.getName().substring(
- 0,
- journalFile.getName().length()
- - SnapshotManager.SNAPSHOT_EXT.length());
+ final long commitCounter = SnapshotManager
+ .parseCommitCounterFile(journalFile.getName());
// temporary file in the same directory as the snapshot.
- final File out = File.createTempFile(basename + "-",
+ final File out = File.createTempFile("" + commitCounter + "-",
Journal.Options.JNL, journalFile.getAbsoluteFile()
.getParentFile());
Copied: branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/SnapshotIndex.java (from rev 7048, branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/CommitTimeIndex.java)
===================================================================
--- branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/SnapshotIndex.java (rev 0)
+++ branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/journal/jini/ha/SnapshotIndex.java 2013-04-25 17:06:50 UTC (rev 7079)
@@ -0,0 +1,862 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+package com.bigdata.journal.jini.ha;
+
+import java.io.Externalizable;
+import java.io.IOException;
+import java.io.ObjectInput;
+import java.io.ObjectOutput;
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.UUID;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReadWriteLock;
+
+import com.bigdata.btree.BTree;
+import com.bigdata.btree.BytesUtil;
+import com.bigdata.btree.DefaultTupleSerializer;
+import com.bigdata.btree.DelegateIndex;
+import com.bigdata.btree.ILinearList;
+import com.bigdata.btree.IRangeQuery;
+import com.bigdata.btree.ITuple;
+import com.bigdata.btree.ITupleIterator;
+import com.bigdata.btree.IndexMetadata;
+import com.bigdata.btree.Tuple;
+import com.bigdata.btree.UnisolatedReadWriteIndex;
+import com.bigdata.btree.keys.ASCIIKeyBuilderFactory;
+import com.bigdata.btree.keys.IKeyBuilderFactory;
+import com.bigdata.btree.keys.KeyBuilder;
+import com.bigdata.journal.IRootBlockView;
+import com.bigdata.journal.RootBlockView;
+import com.bigdata.rawstore.Bytes;
+import com.bigdata.util.ChecksumUtility;
+
+/**
+ * {@link BTree} mapping <em>commitTime</em> (long integers) to
+ * {@link ISnapshotRecord} records.
+ * <p>
+ * This object is thread-safe for concurrent readers and writers.
+ * <p>
+ * Note: This is used as a transient data structure that is populated from the
+ * file system by the {@link HAJournalServer}.
+ */
+public class SnapshotIndex extends DelegateIndex implements ILinearList {
+
+ /**
+ * The underlying index. Access to this is NOT thread safe unless you take
+ * the appropriate lock on the {@link #readWriteLock}.
+ */
+ private final BTree btree;
+
+ /**
+ * The {@link ReadWriteLock} used by the {@link UnisolatedReadWriteIndex} to
+ * make operations on the underlying {@link #btree} thread-safe.
+ */
+ private final ReadWriteLock readWriteLock;
+
+ @SuppressWarnings("unchecked")
+ private Tuple<ISnapshotRecord> getLookupTuple() {
+
+ return btree.getLookupTuple();
+
+ }
+
+// /**
+// * Instance used to encode the timestamp into the key.
+// */
+// final private IKeyBuilder keyBuilder = new KeyBuilder(Bytes.SIZEOF_LONG);
+
+ /**
+ * Create a transient instance.
+ *
+ * @return The new instance.
+ */
+ static public SnapshotIndex createTransient() {
+
+ final IndexMetadata metadata = new IndexMetadata(UUID.randomUUID());
+
+// metadata.setBTreeClassName(SnapshotIndex.class.getName());
+
+ metadata.setTupleSerializer(new TupleSerializer(
+ new ASCIIKeyBuilderFactory(Bytes.SIZEOF_LONG)));
+
+ final BTree ndx = BTree.createTransient(/*store, */metadata);
+
+ return new SnapshotIndex(ndx);
+
+ }
+
+ private SnapshotIndex(final BTree ndx) {
+
+ // Wrap B+Tree for read/write thread safety.
+ super(new UnisolatedReadWriteIndex(ndx));
+
+ this.btree = ndx;
+
+// this.delegate = new UnisolatedReadWriteIndex(ndx);
+
+ // Save reference to lock for extended synchronization patterns.
+ this.readWriteLock = UnisolatedReadWriteIndex.getReadWriteLock(ndx);
+
+ }
+
+// /**
+// * Load from the store.
+// *
+// * @param store
+// * The backing store.
+// * @param checkpoint
+// * The {@link Checkpoint} record.
+// * @param metadata
+// * The metadata record for the index.
+// */
+// public SnapshotIndex(final IRawStore store, final Checkpoint checkpoint,
+// final IndexMetadata metadata, final boolean readOnly) {
+//
+// super(store, checkpoint, metadata, readOnly);
+//
+// }
+
+ /**
+ * Encodes the commit time into a key.
+ *
+ * @param commitTime
+ * The commit time.
+ *
+ * @return The corresponding key.
+ */
+ public byte[] getKey(final long commitTime) {
+
+ return getIndexMetadata().g...
[truncated message content] |
|
From: <tho...@us...> - 2013-04-25 16:23:19
|
Revision: 7078
http://bigdata.svn.sourceforge.net/bigdata/?rev=7078&view=rev
Author: thompsonbry
Date: 2013-04-25 16:23:07 +0000 (Thu, 25 Apr 2013)
Log Message:
-----------
@r7077.
{{{
merge https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0 /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE
--- Merging r7048 through r7077 into /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE
U /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE/bigdata/src/java/com/bigdata/search/FullTextIndex.java
U /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE/bigdata/src/java/com/bigdata/search/CountIndexTask.java
U /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE/bigdata/src/java/com/bigdata/search/AbstractIndexTask.java
U /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE/bigdata/src/java/com/bigdata/search/Hit.java
U /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE/bigdata/src/java/com/bigdata/search/ReadIndexTask.java
U /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/CreateKBTask.java
--- Merging r7048 through r7077 into /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp
U /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/NanoSparqlServer.java
U /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/BigdataRDFServletContextListener.java
U /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/ConfigParams.java
U /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/QueryServlet.java
U /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/RESTServlet.java
U /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/UpdateServlet.java
C /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/StatusServlet.java
U /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/BigdataRDFContext.java
Merge complete.
===== File Statistics: =====
Updated: 13
==== Conflict Statistics: =====
File conflicts: 1
}}}
The StatusServlet now has some conflicts. For future reference, the following have been added or modified in the READ_CACHE branch to support HA:
{{{
/**
* Request a snapshot of the journal (HA only). The snapshot will be written
* into the configured directory on the server. If a snapshot is already
* being taken then this is a NOP.
*/
static final String SNAPSHOT = "snapshot";
/**
* Request to generate the digest for the journals, HALog files, and
* snapshot files. This is only a debugging tool. In particular, the digests
* on the journal are only valid if there are no concurrent writes on the
* journal and the journal has been through either a commit or an abort
* protocol.
*/
static final String DIGESTS = "digests";
/**
* Special HA status request designed for clients that poll to determine the
* status of an HAJournalServer. This option is exclusive of other
* parameters.
*/
static final String HA = "HA";
}}}
{{{
protected void doGet(final HttpServletRequest req,
final HttpServletResponse resp) throws IOException {
if (req.getParameter(HA) != null
&& getIndexManager() instanceof AbstractJournal
&& ((AbstractJournal) getIndexManager()).isHighlyAvailable()) {
new HAStatusServletUtil(getIndexManager()).doHAStatus(req, resp);
return;
}
}}}
{{{
if (getIndexManager() instanceof AbstractJournal
&& ((AbstractJournal) getIndexManager())
.isHighlyAvailable()) {
new HAStatusServletUtil(getIndexManager()).
doGet(req, resp, current);
}
}}}
The rest of the changes have originated in the development branch. There is now a substantial delta for this class in the development branch to provide support for listing and canceling SPARQL UPDATE requests.
Revision Links:
--------------
http://bigdata.svn.sourceforge.net/bigdata/?rev=7077&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7048&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7077&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7048&view=rev
http://bigdata.svn.sourceforge.net/bigdata/?rev=7077&view=rev
Modified Paths:
--------------
branches/READ_CACHE/bigdata/src/java/com/bigdata/search/AbstractIndexTask.java
branches/READ_CACHE/bigdata/src/java/com/bigdata/search/CountIndexTask.java
branches/READ_CACHE/bigdata/src/java/com/bigdata/search/FullTextIndex.java
branches/READ_CACHE/bigdata/src/java/com/bigdata/search/Hit.java
branches/READ_CACHE/bigdata/src/java/com/bigdata/search/ReadIndexTask.java
branches/READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/CreateKBTask.java
branches/READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/BigdataRDFContext.java
branches/READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/BigdataRDFServletContextListener.java
branches/READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/ConfigParams.java
branches/READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/NanoSparqlServer.java
branches/READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/QueryServlet.java
branches/READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/RESTServlet.java
branches/READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/StatusServlet.java
branches/READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/UpdateServlet.java
Property Changed:
----------------
branches/READ_CACHE/
branches/READ_CACHE/bigdata/lib/jetty/
branches/READ_CACHE/bigdata/src/java/com/bigdata/bop/aggregate/
branches/READ_CACHE/bigdata/src/java/com/bigdata/bop/joinGraph/
branches/READ_CACHE/bigdata/src/java/com/bigdata/bop/util/
branches/READ_CACHE/bigdata/src/java/com/bigdata/htree/raba/
branches/READ_CACHE/bigdata/src/java/com/bigdata/jsr166/
branches/READ_CACHE/bigdata/src/test/com/bigdata/bop/joinGraph/
branches/READ_CACHE/bigdata/src/test/com/bigdata/bop/util/
branches/READ_CACHE/bigdata/src/test/com/bigdata/jsr166/
branches/READ_CACHE/bigdata/src/test/com/bigdata/util/httpd/
branches/READ_CACHE/bigdata-compatibility/
branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/attr/
branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/disco/
branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/util/config/
branches/READ_CACHE/bigdata-perf/
branches/READ_CACHE/bigdata-perf/btc/
branches/READ_CACHE/bigdata-perf/btc/src/resources/
branches/READ_CACHE/bigdata-perf/lubm/
branches/READ_CACHE/bigdata-perf/uniprot/
branches/READ_CACHE/bigdata-perf/uniprot/src/
branches/READ_CACHE/bigdata-rdf/src/java/com/bigdata/bop/rdf/aggregate/
branches/READ_CACHE/bigdata-rdf/src/java/com/bigdata/rdf/changesets/
branches/READ_CACHE/bigdata-rdf/src/java/com/bigdata/rdf/error/
branches/READ_CACHE/bigdata-rdf/src/java/com/bigdata/rdf/internal/
branches/READ_CACHE/bigdata-rdf/src/java/com/bigdata/rdf/relation/
branches/READ_CACHE/bigdata-rdf/src/java/com/bigdata/rdf/util/
branches/READ_CACHE/bigdata-rdf/src/samples/
branches/READ_CACHE/bigdata-rdf/src/test/com/bigdata/bop/rdf/aggregate/
branches/READ_CACHE/bigdata-rdf/src/test/com/bigdata/rdf/internal/
branches/READ_CACHE/bigdata-rdf/src/test/com/bigdata/rdf/relation/
branches/READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/bench/
branches/READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/changesets/
branches/READ_CACHE/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/
branches/READ_CACHE/bigdata-sails/src/test/com/bigdata/rdf/sail/bench/
branches/READ_CACHE/bigdata-sails/src/test/com/bigdata/rdf/sail/webapp/
branches/READ_CACHE/dsi-utils/
branches/READ_CACHE/dsi-utils/LEGAL/
branches/READ_CACHE/dsi-utils/lib/
branches/READ_CACHE/dsi-utils/src/
branches/READ_CACHE/dsi-utils/src/java/
branches/READ_CACHE/dsi-utils/src/java/it/
branches/READ_CACHE/dsi-utils/src/java/it/unimi/
branches/READ_CACHE/dsi-utils/src/test/
branches/READ_CACHE/dsi-utils/src/test/it/unimi/
branches/READ_CACHE/dsi-utils/src/test/it/unimi/dsi/
branches/READ_CACHE/lgpl-utils/src/java/it/unimi/dsi/fastutil/bytes/custom/
branches/READ_CACHE/lgpl-utils/src/test/it/unimi/dsi/fastutil/bytes/custom/
branches/READ_CACHE/osgi/
branches/READ_CACHE/src/resources/bin/config/
Property changes on: branches/READ_CACHE
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE:6769-6785
/branches/BIGDATA_RELEASE_1_2_0:6766-7048
/branches/BTREE_BUFFER_BRANCH:2004-2045
/branches/DEV_BRANCH_27_OCT_2009:2270-2546,2548-2782
/branches/INT64_BRANCH:4486-4522
/branches/JOURNAL_HA_BRANCH:2596-4066
/branches/LARGE_LITERALS_REFACTOR:4175-4387
/branches/LEXICON_REFACTOR_BRANCH:2633-3304
/branches/QUADS_QUERY_BRANCH:4525-4531,4550-4584,4586-4609,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH:4814-4836
/branches/bugfix-btm:2594-3237
/branches/dev-btm:2574-2730
/branches/fko:3150-3194
/trunk:3392-3437,3656-4061
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE:6769-6785
/branches/BIGDATA_RELEASE_1_2_0:6766-7077
/branches/BTREE_BUFFER_BRANCH:2004-2045
/branches/DEV_BRANCH_27_OCT_2009:2270-2546,2548-2782
/branches/INT64_BRANCH:4486-4522
/branches/JOURNAL_HA_BRANCH:2596-4066
/branches/LARGE_LITERALS_REFACTOR:4175-4387
/branches/LEXICON_REFACTOR_BRANCH:2633-3304
/branches/QUADS_QUERY_BRANCH:4525-4531,4550-4584,4586-4609,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH:4814-4836
/branches/bugfix-btm:2594-3237
/branches/dev-btm:2574-2730
/branches/fko:3150-3194
/trunk:3392-3437,3656-4061
Property changes on: branches/READ_CACHE/bigdata/lib/jetty
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/lib/jetty:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/lib/jetty:6766-7048
/branches/INT64_BRANCH/bigdata/lib/jetty:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/lib/jetty:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/lib/jetty:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/lib/jetty:4814-4836
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/lib/jetty:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/lib/jetty:6766-7077
/branches/INT64_BRANCH/bigdata/lib/jetty:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/lib/jetty:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/lib/jetty:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/lib/jetty:4814-4836
Property changes on: branches/READ_CACHE/bigdata/src/java/com/bigdata/bop/aggregate
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/java/com/bigdata/bop/aggregate:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/aggregate:6766-7048
/branches/INT64_BRANCH/bigdata/src/java/com/bigdata/bop/aggregate:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregate:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/java/com/bigdata/bop/aggregate:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/java/com/bigdata/bop/aggregate:4814-4836
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/java/com/bigdata/bop/aggregate:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/aggregate:6766-7077
/branches/INT64_BRANCH/bigdata/src/java/com/bigdata/bop/aggregate:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/aggregate:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/java/com/bigdata/bop/aggregate:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/java/com/bigdata/bop/aggregate:4814-4836
Property changes on: branches/READ_CACHE/bigdata/src/java/com/bigdata/bop/joinGraph
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/java/com/bigdata/bop/joinGraph:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/joinGraph:6766-7048
/branches/INT64_BRANCH/bigdata/src/java/com/bigdata/bop/joinGraph:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/joinGraph:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/java/com/bigdata/bop/joinGraph:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/java/com/bigdata/bop/joinGraph:4814-4836
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/java/com/bigdata/bop/joinGraph:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/joinGraph:6766-7077
/branches/INT64_BRANCH/bigdata/src/java/com/bigdata/bop/joinGraph:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/joinGraph:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/java/com/bigdata/bop/joinGraph:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/java/com/bigdata/bop/joinGraph:4814-4836
Property changes on: branches/READ_CACHE/bigdata/src/java/com/bigdata/bop/util
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/java/com/bigdata/bop/util:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/util:6766-7048
/branches/INT64_BRANCH/bigdata/src/java/com/bigdata/bop/util:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/util:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/java/com/bigdata/bop/util:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/java/com/bigdata/bop/util:4814-4836
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/java/com/bigdata/bop/util:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/bop/util:6766-7077
/branches/INT64_BRANCH/bigdata/src/java/com/bigdata/bop/util:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/util:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/java/com/bigdata/bop/util:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/java/com/bigdata/bop/util:4814-4836
Property changes on: branches/READ_CACHE/bigdata/src/java/com/bigdata/htree/raba
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/java/com/bigdata/htree/raba:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/htree/raba:6766-7048
/branches/INT64_BRANCH/bigdata/src/java/com/bigdata/htree/raba:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/htree/raba:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/java/com/bigdata/htree/raba:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/java/com/bigdata/htree/raba:4814-4836
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/java/com/bigdata/htree/raba:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/htree/raba:6766-7077
/branches/INT64_BRANCH/bigdata/src/java/com/bigdata/htree/raba:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/htree/raba:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/java/com/bigdata/htree/raba:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/java/com/bigdata/htree/raba:4814-4836
Property changes on: branches/READ_CACHE/bigdata/src/java/com/bigdata/jsr166
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/java/com/bigdata/jsr166:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/jsr166:6766-7048
/branches/INT64_BRANCH/bigdata/src/java/com/bigdata/jsr166:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/jsr166:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/java/com/bigdata/jsr166:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/java/com/bigdata/jsr166:4814-4836
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/java/com/bigdata/jsr166:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/java/com/bigdata/jsr166:6766-7077
/branches/INT64_BRANCH/bigdata/src/java/com/bigdata/jsr166:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/jsr166:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/java/com/bigdata/jsr166:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/java/com/bigdata/jsr166:4814-4836
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/search/AbstractIndexTask.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/search/AbstractIndexTask.java 2013-04-25 15:59:02 UTC (rev 7077)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/search/AbstractIndexTask.java 2013-04-25 16:23:07 UTC (rev 7078)
@@ -17,6 +17,8 @@
final private static Logger log = Logger.getLogger(AbstractIndexTask.class);
protected final String queryTerm;
+ protected final int queryTermNdx;
+ protected final int numQueryTerms;
protected final double queryTermWeight;
protected final byte[] fromKey;
protected final byte[] toKey;
@@ -27,6 +29,10 @@
*
* @param termText
* The term text for the search term.
+ * @param termNdx
+ * The index of this term within the overall search.
+ * @param numTerms
+ * The overall number of search terms.
* @param prefixMatch
* When <code>true</code> any term having <i>termText</i> as a
* prefix will be matched. Otherwise the term must be an exact
@@ -36,8 +42,10 @@
* @param searchEngine
* The search engine.
*/
- public AbstractIndexTask(final String termText, final boolean prefixMatch,
- final double queryTermWeight, final FullTextIndex<V> searchEngine) {
+ public AbstractIndexTask(final String termText,
+ final int termNdx, final int numTerms,
+ final boolean prefixMatch, final double queryTermWeight,
+ final FullTextIndex<V> searchEngine) {
if (termText == null)
throw new IllegalArgumentException();
@@ -46,40 +54,82 @@
throw new IllegalArgumentException();
this.queryTerm = termText;
+
+ this.queryTermNdx = termNdx;
+
+ this.numQueryTerms = numTerms;
this.queryTermWeight = queryTermWeight;
final IKeyBuilder keyBuilder = searchEngine.getIndex()
.getIndexMetadata().getKeyBuilder();
- {// = recordBuilder.getFromKey(keyBuilder, termText);
- keyBuilder.reset();
- keyBuilder
- .appendText(termText, true/* unicode */, false/* successor */);
- fromKey = keyBuilder.getKey();
+// {// = recordBuilder.getFromKey(keyBuilder, termText);
+// keyBuilder.reset();
+// keyBuilder
+// .appendText(termText, true/* unicode */, false/* successor */);
+// fromKey = keyBuilder.getKey();
+// }
+//
+// if (prefixMatch) {
+// /*
+// * Accepts anything starting with the search term. E.g., given
+// * "bro", it will match "broom" and "brown" but not "break".
+// *
+// * Note: This uses the successor of the Unicode sort key, so it will
+// * scan all keys starting with that prefix until the sucessor of
+// * that prefix.
+// */
+// keyBuilder.reset();
+// keyBuilder.appendText(termText, true/* unicode */, true/*successor*/);
+// toKey = keyBuilder.getKey();
+// } else {
+// /*
+// * Accepts only those entries that exactly match the search term.
+// *
+// * Note: This uses the fixed length successor of the fromKey. That
+// * gives us a key-range scan which only access keys having the same
+// * Unicode sort key.
+// */
+// toKey = SuccessorUtil.successor(fromKey.clone());
+// }
+
+ /*
+ * Changed this to lop off the last three bytes (the pad plus
+ * run-length) for prefix search. Adding this three byte suffix to the
+ * prefix was causing problems for searches whose prefix ended with a
+ * numeric less than 7, because this codes to a byte less than the pad
+ * byte.
+ *
+ * TODO We eventually need to change the pad byte to code to zero, but
+ * this will break binary compatibility.
+ */
+
+ { // from key
+ keyBuilder.reset();
+ keyBuilder
+ .appendText(termText, true/* unicode */, false/* successor */);
+ final byte[] tmp = keyBuilder.getKey();
+
+ if (prefixMatch) {
+ fromKey = new byte[tmp.length-3];
+ System.arraycopy(tmp, 0, fromKey, 0, fromKey.length);
+ }
+ else
+ {// = recordBuilder.getFromKey(keyBuilder, termText);
+ fromKey = tmp;
+ }
}
- if (prefixMatch) {
- /*
- * Accepts anything starting with the search term. E.g., given
- * "bro", it will match "broom" and "brown" but not "break".
- *
- * Note: This uses the successor of the Unicode sort key, so it will
- * scan all keys starting with that prefix until the sucessor of
- * that prefix.
- */
- keyBuilder.reset();
- keyBuilder.appendText(termText, true/* unicode */, true/*successor*/);
- toKey = keyBuilder.getKey();
- } else {
- /*
- * Accepts only those entries that exactly match the search term.
- *
- * Note: This uses the fixed length successor of the fromKey. That
- * gives us a key-range scan which only access keys having the same
- * Unicode sort key.
- */
- toKey = SuccessorUtil.successor(fromKey.clone());
+ { // to key
+ /*
+ * Accepts only those entries that exactly match the search term.
+ *
+ * Note: This uses the fixed length successor of the fromKey. That
+ * gives us a key-range scan which only access keys having the same
+ * Unicode sort key.
+ */
+ toKey = SuccessorUtil.successor(fromKey.clone());
}
}
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/search/CountIndexTask.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/search/CountIndexTask.java 2013-04-25 15:59:02 UTC (rev 7077)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/search/CountIndexTask.java 2013-04-25 16:23:07 UTC (rev 7078)
@@ -41,6 +41,10 @@
*
* @param termText
* The term text for the search term.
+ * @param termNdx
+ * The index of this term within the overall search.
+ * @param numTerms
+ * The overall number of search terms.
* @param prefixMatch
* When <code>true</code> any term having <i>termText</i> as a
* prefix will be matched. Otherwise the term must be an exact
@@ -50,10 +54,11 @@
* @param searchEngine
* The search engine.
*/
- public CountIndexTask(final String termText, final boolean prefixMatch,
+ public CountIndexTask(final String termText, final int termNdx, final int numTerms,
+ final boolean prefixMatch,
final double queryTermWeight, final FullTextIndex<V> searchEngine) {
- super(termText, prefixMatch, queryTermWeight, searchEngine);
+ super(termText, termNdx, numTerms, prefixMatch, queryTermWeight, searchEngine);
if (log.isDebugEnabled())
log.debug("termText=[" + termText + "], prefixMatch=" + prefixMatch
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/search/FullTextIndex.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/search/FullTextIndex.java 2013-04-25 15:59:02 UTC (rev 7077)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/search/FullTextIndex.java 2013-04-25 16:23:07 UTC (rev 7078)
@@ -1008,6 +1008,7 @@
+ ", minRank=" + minRank
+ ", maxRank=" + maxRank
+ ", matchAllTerms=" + matchAllTerms
+ + ", prefixMatch=" + prefixMatch
+ ", timeout=" + timeout + ", unit=" + unit);
if (timeout == 0L) {
@@ -1085,7 +1086,7 @@
final ITermMetadata md = e.getValue();
- final CountIndexTask<V> task1 = new CountIndexTask<V>(termText, prefixMatch, md
+ final CountIndexTask<V> task1 = new CountIndexTask<V>(termText, 0, 1, prefixMatch, md
.getLocalTermWeight(), this);
hits = new SingleTokenHitCollector<V>(task1);
@@ -1095,13 +1096,14 @@
final List<CountIndexTask<V>> tasks = new ArrayList<CountIndexTask<V>>(
qdata.distinctTermCount());
+ int i = 0;
for (Map.Entry<String, ITermMetadata> e : qdata.terms.entrySet()) {
final String termText = e.getKey();
final ITermMetadata md = e.getValue();
- tasks.add(new CountIndexTask<V>(termText, prefixMatch, md
+ tasks.add(new CountIndexTask<V>(termText, i++, qdata.terms.size(), prefixMatch, md
.getLocalTermWeight(), this));
}
@@ -1116,14 +1118,15 @@
final List<Callable<Object>> tasks = new ArrayList<Callable<Object>>(
qdata.distinctTermCount());
+ int i = 0;
for (Map.Entry<String, ITermMetadata> e : qdata.terms.entrySet()) {
final String termText = e.getKey();
final ITermMetadata md = e.getValue();
- tasks.add(new ReadIndexTask<V>(termText, prefixMatch, md
- .getLocalTermWeight(), this, hits));
+ tasks.add(new ReadIndexTask<V>(termText, i++, qdata.terms.size(),
+ prefixMatch, md.getLocalTermWeight(), this, hits));
}
@@ -1160,7 +1163,7 @@
if (a.length == 0) {
- log.warn("No hits: languageCode=[" + languageCode + "], query=["
+ log.info("No hits: languageCode=[" + languageCode + "], query=["
+ query + "]");
cache.put(cacheKey, a);
@@ -1178,8 +1181,10 @@
final int nterms = qdata.terms.size();
- if (log.isInfoEnabled())
+ if (log.isInfoEnabled()) {
log.info("matchAll=true, nterms=" + nterms);
+ log.info("size before: " + a.length);
+ }
final Hit<V>[] tmp = new Hit[a.length];
@@ -1192,6 +1197,10 @@
}
+ if (log.isDebugEnabled()) {
+ log.debug(i);
+ }
+
if (i < a.length) {
a = new Hit[i];
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/search/Hit.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/search/Hit.java 2013-04-25 15:59:02 UTC (rev 7077)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/search/Hit.java 2013-04-25 16:23:07 UTC (rev 7078)
@@ -1,9 +1,13 @@
package com.bigdata.search;
+import it.unimi.dsi.bits.LongArrayBitVector;
+
import java.util.concurrent.ConcurrentHashMap;
import org.apache.log4j.Logger;
+import test.it.unimi.dsi.bits.LongArrayBitVectorTest;
+
/**
* Metadata about a search result.
*
@@ -18,8 +22,12 @@
/** note: defaults to an illegal value. */
private V docId = null;
- /** #of terms reporting. */
- private int nterms;
+// /** #of terms reporting. */
+// private int nterms;
+
+ /** Array of whether each search term appears or does not appear in the hit. **/
+// private LongArrayBitVector searchTerms;
+ private final boolean[] searchTerms;
/** Net cosine for the reporting terms. */
private double cosine;
@@ -35,8 +43,11 @@
*
* @see ReadIndexTask2
*/
- Hit() {
+ Hit(final int numSearchTerms) {
+// this.searchTerms = LongArrayBitVector.ofLength(numSearchTerms);
+ this.searchTerms = new boolean[numSearchTerms];
+
}
synchronized void setDocId(final V docId) {
@@ -54,11 +65,26 @@
}
+// synchronized void setNumSearchTerms(final int numSearchTerms) {
+//
+//// this.searchTerms = LongArrayBitVector.ofLength(numSearchTerms);
+// this.searchTerms = new boolean[numSearchTerms];
+//
+// }
+
/**
* The #of terms for which a hit was reported for this document.
*/
synchronized public int getTermCount() {
+// if (searchTerms.size() == 0)
+ if (searchTerms.length == 0)
+ return 0;
+
+ int nterms = 0;
+ for (boolean b : searchTerms)
+ if (b) nterms++;
+
return nterms;
}
@@ -84,13 +110,16 @@
/**
* Adds another component to the cosine.
*/
- public void add(final String term, final double weight) {
+ public void add(final int termNdx, final double weight) {
synchronized (this) {
cosine += weight;
- nterms++;
+// nterms++;
+
+// searchTerms.set(termNdx, true);
+ searchTerms[termNdx] = true;
}
@@ -105,7 +134,7 @@
public String toString() {
- return "Hit{docId"+docId+",nterms="+nterms+",cosine="+cosine+"}";
+ return "Hit{docId"+docId+",nterms="+getTermCount()+",cosine="+cosine+"}";
}
Modified: branches/READ_CACHE/bigdata/src/java/com/bigdata/search/ReadIndexTask.java
===================================================================
--- branches/READ_CACHE/bigdata/src/java/com/bigdata/search/ReadIndexTask.java 2013-04-25 15:59:02 UTC (rev 7077)
+++ branches/READ_CACHE/bigdata/src/java/com/bigdata/search/ReadIndexTask.java 2013-04-25 16:23:07 UTC (rev 7078)
@@ -48,7 +48,7 @@
* inserted, the {@link Hit#setDocId(long) docId} is set on the {@link Hit}
* and a new instance is assigned to {@link #tmp}.
*/
- private Hit<V> tmp = new Hit<V>();
+ private Hit<V> tmp;// = new Hit<V>();
/**
* Setup a task that will perform a range scan for entries matching the
@@ -56,6 +56,10 @@
*
* @param termText
* The term text for the search term.
+ * @param termNdx
+ * The index of this term within the overall search.
+ * @param numTerms
+ * The overall number of search terms.
* @param prefixMatch
* When <code>true</code> any term having <i>termText</i> as a
* prefix will be matched. Otherwise the term must be an exact
@@ -67,11 +71,12 @@
* @param hits
* The map where the hits are being aggregated.
*/
- public ReadIndexTask(final String termText, final boolean prefixMatch,
- final double queryTermWeight, final FullTextIndex<V> searchEngine,
- final IHitCollector<V> hits) {
+ public ReadIndexTask(final String termText,
+ final int termNdx, final int numTerms,
+ final boolean prefixMatch, final double queryTermWeight,
+ final FullTextIndex<V> searchEngine, final IHitCollector<V> hits) {
- super(termText, prefixMatch, queryTermWeight, searchEngine);
+ super(termText, termNdx, numTerms, prefixMatch, queryTermWeight, searchEngine);
if (hits == null)
throw new IllegalArgumentException();
@@ -93,6 +98,8 @@
itr = searchEngine.getIndex()
.rangeIterator(fromKey, toKey, 0/* capacity */,
IRangeQuery.KEYS | IRangeQuery.VALS, null/* filter */);
+
+ tmp = new Hit<V>(numTerms);
}
@@ -175,13 +182,13 @@
if (oldValue == null) {
hit = tmp;
hit.setDocId(docId);
- tmp = new Hit<V>();
+ tmp = new Hit<V>(numQueryTerms);
} else {
hit = oldValue;
}
}
- hit.add( queryTerm, queryTermWeight * termWeight );
+ hit.add( queryTermNdx, queryTermWeight * termWeight );
nhits++;
Property changes on: branches/READ_CACHE/bigdata/src/test/com/bigdata/bop/joinGraph
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/test/com/bigdata/bop/joinGraph:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/test/com/bigdata/bop/joinGraph:6766-7048
/branches/INT64_BRANCH/bigdata/src/test/com/bigdata/bop/joinGraph:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/joinGraph:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/test/com/bigdata/bop/joinGraph:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/test/com/bigdata/bop/joinGraph:4814-4836
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/test/com/bigdata/bop/joinGraph:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/test/com/bigdata/bop/joinGraph:6766-7077
/branches/INT64_BRANCH/bigdata/src/test/com/bigdata/bop/joinGraph:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/joinGraph:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/test/com/bigdata/bop/joinGraph:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/test/com/bigdata/bop/joinGraph:4814-4836
Property changes on: branches/READ_CACHE/bigdata/src/test/com/bigdata/bop/util
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/test/com/bigdata/bop/util:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/test/com/bigdata/bop/util:6766-7048
/branches/INT64_BRANCH/bigdata/src/test/com/bigdata/bop/util:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/util:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/test/com/bigdata/bop/util:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/test/com/bigdata/bop/util:4814-4836
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/test/com/bigdata/bop/util:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/test/com/bigdata/bop/util:6766-7077
/branches/INT64_BRANCH/bigdata/src/test/com/bigdata/bop/util:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/util:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/test/com/bigdata/bop/util:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/test/com/bigdata/bop/util:4814-4836
Property changes on: branches/READ_CACHE/bigdata/src/test/com/bigdata/jsr166
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/test/com/bigdata/jsr166:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/test/com/bigdata/jsr166:6766-7048
/branches/INT64_BRANCH/bigdata/src/test/com/bigdata/jsr166:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/jsr166:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/test/com/bigdata/jsr166:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/test/com/bigdata/jsr166:4814-4836
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/test/com/bigdata/jsr166:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/test/com/bigdata/jsr166:6766-7077
/branches/INT64_BRANCH/bigdata/src/test/com/bigdata/jsr166:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/jsr166:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/test/com/bigdata/jsr166:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/test/com/bigdata/jsr166:4814-4836
Property changes on: branches/READ_CACHE/bigdata/src/test/com/bigdata/util/httpd
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/test/com/bigdata/util/httpd:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/test/com/bigdata/util/httpd:6766-7048
/branches/INT64_BRANCH/bigdata/src/test/com/bigdata/util/httpd:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/util/httpd:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/test/com/bigdata/util/httpd:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/test/com/bigdata/util/httpd:4814-4836
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata/src/test/com/bigdata/util/httpd:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata/src/test/com/bigdata/util/httpd:6766-7077
/branches/INT64_BRANCH/bigdata/src/test/com/bigdata/util/httpd:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/util/httpd:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata/src/test/com/bigdata/util/httpd:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata/src/test/com/bigdata/util/httpd:4814-4836
Property changes on: branches/READ_CACHE/bigdata-compatibility
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-compatibility:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-compatibility:6766-7048
/branches/INT64_BRANCH/bigdata-compatibility:4486-4522
/branches/LARGE_LITERALS_REFACTOR/bigdata-compatibility:4175-4387
/branches/QUADS_QUERY_BRANCH/bigdata-compatibility:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-compatibility:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-compatibility:4814-4836
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-compatibility:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-compatibility:6766-7077
/branches/INT64_BRANCH/bigdata-compatibility:4486-4522
/branches/LARGE_LITERALS_REFACTOR/bigdata-compatibility:4175-4387
/branches/QUADS_QUERY_BRANCH/bigdata-compatibility:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-compatibility:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-compatibility:4814-4836
Property changes on: branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/attr
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-jini/src/java/com/bigdata/attr:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-jini/src/java/com/bigdata/attr:6766-7048
/branches/BTREE_BUFFER_BRANCH/bigdata-jini/src/java/com/bigdata/attr:2004-2045
/branches/DEV_BRANCH_27_OCT_2009/bigdata-jini/src/java/com/bigdata/attr:2270-2546,2548-2782
/branches/INT64_BRANCH/bigdata-jini/src/java/com/bigdata/attr:4486-4522
/branches/JOURNAL_HA_BRANCH/bigdata-jini/src/java/com/bigdata/attr:2596-4066
/branches/LARGE_LITERALS_REFACTOR/bigdata-jini/src/java/com/bigdata/attr:4175-4387
/branches/LEXICON_REFACTOR_BRANCH/bigdata-jini/src/java/com/bigdata/attr:2633-3304
/branches/QUADS_QUERY_BRANCH/bigdata-jini/src/java/com/bigdata/attr:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-jini/src/java/com/bigdata/attr:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-jini/src/java/com/bigdata/attr:4814-4836
/branches/bugfix-btm/bigdata-jini/src/java/com/bigdata/attr:2594-3237
/branches/dev-btm/bigdata-jini/src/java/com/bigdata/attr:2574-2730
/branches/fko/bigdata-jini/src/java/com/bigdata/attr:3150-3194
/trunk/bigdata-jini/src/java/com/bigdata/attr:2981-3282,3368-3437,3656-4061
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-jini/src/java/com/bigdata/attr:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-jini/src/java/com/bigdata/attr:6766-7077
/branches/BTREE_BUFFER_BRANCH/bigdata-jini/src/java/com/bigdata/attr:2004-2045
/branches/DEV_BRANCH_27_OCT_2009/bigdata-jini/src/java/com/bigdata/attr:2270-2546,2548-2782
/branches/INT64_BRANCH/bigdata-jini/src/java/com/bigdata/attr:4486-4522
/branches/JOURNAL_HA_BRANCH/bigdata-jini/src/java/com/bigdata/attr:2596-4066
/branches/LARGE_LITERALS_REFACTOR/bigdata-jini/src/java/com/bigdata/attr:4175-4387
/branches/LEXICON_REFACTOR_BRANCH/bigdata-jini/src/java/com/bigdata/attr:2633-3304
/branches/QUADS_QUERY_BRANCH/bigdata-jini/src/java/com/bigdata/attr:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-jini/src/java/com/bigdata/attr:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-jini/src/java/com/bigdata/attr:4814-4836
/branches/bugfix-btm/bigdata-jini/src/java/com/bigdata/attr:2594-3237
/branches/dev-btm/bigdata-jini/src/java/com/bigdata/attr:2574-2730
/branches/fko/bigdata-jini/src/java/com/bigdata/attr:3150-3194
/trunk/bigdata-jini/src/java/com/bigdata/attr:2981-3282,3368-3437,3656-4061
Property changes on: branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/disco
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-jini/src/java/com/bigdata/disco:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-jini/src/java/com/bigdata/disco:6766-7048
/branches/BTREE_BUFFER_BRANCH/bigdata-jini/src/java/com/bigdata/disco:2004-2045
/branches/DEV_BRANCH_27_OCT_2009/bigdata-jini/src/java/com/bigdata/disco:2270-2546,2548-2782
/branches/INT64_BRANCH/bigdata-jini/src/java/com/bigdata/disco:4486-4522
/branches/JOURNAL_HA_BRANCH/bigdata-jini/src/java/com/bigdata/disco:2596-4066
/branches/LARGE_LITERALS_REFACTOR/bigdata-jini/src/java/com/bigdata/disco:4175-4387
/branches/LEXICON_REFACTOR_BRANCH/bigdata-jini/src/java/com/bigdata/disco:2633-3304
/branches/QUADS_QUERY_BRANCH/bigdata-jini/src/java/com/bigdata/disco:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-jini/src/java/com/bigdata/disco:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-jini/src/java/com/bigdata/disco:4814-4836
/branches/bugfix-btm/bigdata-jini/src/java/com/bigdata/disco:2594-3237
/branches/dev-btm/bigdata-jini/src/java/com/bigdata/disco:2574-2730
/branches/fko/bigdata-jini/src/java/com/bigdata/disco:3150-3194
/trunk/bigdata-jini/src/java/com/bigdata/disco:2981-3282,3368-3437,3656-4061
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-jini/src/java/com/bigdata/disco:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-jini/src/java/com/bigdata/disco:6766-7077
/branches/BTREE_BUFFER_BRANCH/bigdata-jini/src/java/com/bigdata/disco:2004-2045
/branches/DEV_BRANCH_27_OCT_2009/bigdata-jini/src/java/com/bigdata/disco:2270-2546,2548-2782
/branches/INT64_BRANCH/bigdata-jini/src/java/com/bigdata/disco:4486-4522
/branches/JOURNAL_HA_BRANCH/bigdata-jini/src/java/com/bigdata/disco:2596-4066
/branches/LARGE_LITERALS_REFACTOR/bigdata-jini/src/java/com/bigdata/disco:4175-4387
/branches/LEXICON_REFACTOR_BRANCH/bigdata-jini/src/java/com/bigdata/disco:2633-3304
/branches/QUADS_QUERY_BRANCH/bigdata-jini/src/java/com/bigdata/disco:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-jini/src/java/com/bigdata/disco:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-jini/src/java/com/bigdata/disco:4814-4836
/branches/bugfix-btm/bigdata-jini/src/java/com/bigdata/disco:2594-3237
/branches/dev-btm/bigdata-jini/src/java/com/bigdata/disco:2574-2730
/branches/fko/bigdata-jini/src/java/com/bigdata/disco:3150-3194
/trunk/bigdata-jini/src/java/com/bigdata/disco:2981-3282,3368-3437,3656-4061
Property changes on: branches/READ_CACHE/bigdata-jini/src/java/com/bigdata/util/config
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-jini/src/java/com/bigdata/util/config:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-jini/src/java/com/bigdata/util/config:6766-7048
/branches/BTREE_BUFFER_BRANCH/bigdata-jini/src/java/com/bigdata/util/config:2004-2045
/branches/DEV_BRANCH_27_OCT_2009/bigdata-jini/src/java/com/bigdata/util/config:2270-2546,2548-2782
/branches/INT64_BRANCH/bigdata-jini/src/java/com/bigdata/util/config:4486-4522
/branches/JOURNAL_HA_BRANCH/bigdata-jini/src/java/com/bigdata/util/config:2596-4066
/branches/LARGE_LITERALS_REFACTOR/bigdata-jini/src/java/com/bigdata/util/config:4175-4387
/branches/LEXICON_REFACTOR_BRANCH/bigdata-jini/src/java/com/bigdata/util/config:2633-3304
/branches/QUADS_QUERY_BRANCH/bigdata-jini/src/java/com/bigdata/util/config:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-jini/src/java/com/bigdata/util/config:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-jini/src/java/com/bigdata/util/config:4814-4836
/branches/bugfix-btm/bigdata-jini/src/java/com/bigdata/util/config:2594-3237
/branches/dev-btm/bigdata-jini/src/java/com/bigdata/util/config:2574-2730
/branches/fko/bigdata-jini/src/java/com/bigdata/util/config:3150-3194
/trunk/bigdata-jini/src/java/com/bigdata/util/config:2981-3282,3368-3437,3656-4061
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-jini/src/java/com/bigdata/util/config:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-jini/src/java/com/bigdata/util/config:6766-7077
/branches/BTREE_BUFFER_BRANCH/bigdata-jini/src/java/com/bigdata/util/config:2004-2045
/branches/DEV_BRANCH_27_OCT_2009/bigdata-jini/src/java/com/bigdata/util/config:2270-2546,2548-2782
/branches/INT64_BRANCH/bigdata-jini/src/java/com/bigdata/util/config:4486-4522
/branches/JOURNAL_HA_BRANCH/bigdata-jini/src/java/com/bigdata/util/config:2596-4066
/branches/LARGE_LITERALS_REFACTOR/bigdata-jini/src/java/com/bigdata/util/config:4175-4387
/branches/LEXICON_REFACTOR_BRANCH/bigdata-jini/src/java/com/bigdata/util/config:2633-3304
/branches/QUADS_QUERY_BRANCH/bigdata-jini/src/java/com/bigdata/util/config:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-jini/src/java/com/bigdata/util/config:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-jini/src/java/com/bigdata/util/config:4814-4836
/branches/bugfix-btm/bigdata-jini/src/java/com/bigdata/util/config:2594-3237
/branches/dev-btm/bigdata-jini/src/java/com/bigdata/util/config:2574-2730
/branches/fko/bigdata-jini/src/java/com/bigdata/util/config:3150-3194
/trunk/bigdata-jini/src/java/com/bigdata/util/config:2981-3282,3368-3437,3656-4061
Property changes on: branches/READ_CACHE/bigdata-perf
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-perf:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-perf:6766-7048
/branches/BTREE_BUFFER_BRANCH/bigdata-perf:2004-2045
/branches/DEV_BRANCH_27_OCT_2009/bigdata-perf:2270-2546,2548-2782
/branches/INT64_BRANCH/bigdata-perf:4486-4522
/branches/JOURNAL_HA_BRANCH/bigdata-perf:2596-4066
/branches/LARGE_LITERALS_REFACTOR/bigdata-perf:4175-4387
/branches/LEXICON_REFACTOR_BRANCH/bigdata-perf:2633-3304
/branches/QUADS_QUERY_BRANCH/bigdata-perf:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-perf:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-perf:4814-4836
/branches/bugfix-btm/bigdata-perf:2594-3237
/branches/dev-btm/bigdata-perf:2574-2730
/branches/fko/bigdata-perf:3150-3194
/trunk/bigdata-perf:2981-3043,3368-3437,3656-4061
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-perf:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-perf:6766-7077
/branches/BTREE_BUFFER_BRANCH/bigdata-perf:2004-2045
/branches/DEV_BRANCH_27_OCT_2009/bigdata-perf:2270-2546,2548-2782
/branches/INT64_BRANCH/bigdata-perf:4486-4522
/branches/JOURNAL_HA_BRANCH/bigdata-perf:2596-4066
/branches/LARGE_LITERALS_REFACTOR/bigdata-perf:4175-4387
/branches/LEXICON_REFACTOR_BRANCH/bigdata-perf:2633-3304
/branches/QUADS_QUERY_BRANCH/bigdata-perf:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-perf:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-perf:4814-4836
/branches/bugfix-btm/bigdata-perf:2594-3237
/branches/dev-btm/bigdata-perf:2574-2730
/branches/fko/bigdata-perf:3150-3194
/trunk/bigdata-perf:2981-3043,3368-3437,3656-4061
Property changes on: branches/READ_CACHE/bigdata-perf/btc
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-perf/btc:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-perf/btc:6766-7048
/branches/INT64_BRANCH/bigdata-perf/btc:4486-4522
/branches/JOURNAL_HA_BRANCH/bigdata-perf/btc:2596-4066
/branches/LARGE_LITERALS_REFACTOR/bigdata-perf/btc:4175-4387
/branches/QUADS_QUERY_BRANCH/bigdata-perf/btc:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-perf/btc:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-perf/btc:4814-4836
/trunk/bigdata-perf/btc:2981-3043,3368-3437,3656-4061
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-perf/btc:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-perf/btc:6766-7077
/branches/INT64_BRANCH/bigdata-perf/btc:4486-4522
/branches/JOURNAL_HA_BRANCH/bigdata-perf/btc:2596-4066
/branches/LARGE_LITERALS_REFACTOR/bigdata-perf/btc:4175-4387
/branches/QUADS_QUERY_BRANCH/bigdata-perf/btc:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-perf/btc:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-perf/btc:4814-4836
/trunk/bigdata-perf/btc:2981-3043,3368-3437,3656-4061
Property changes on: branches/READ_CACHE/bigdata-perf/btc/src/resources
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-perf/btc/src/resources:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-perf/btc/src/resources:6766-7048
/branches/INT64_BRANCH/bigdata-perf/btc/src/resources:4486-4522
/branches/JOURNAL_HA_BRANCH/bigdata-perf/btc/src/resources:2596-4066
/branches/LARGE_LITERALS_REFACTOR/bigdata-perf/btc/src/resources:4175-4387
/branches/QUADS_QUERY_BRANCH/bigdata-perf/btc/src/resources:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-perf/btc/src/resources:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-perf/btc/src/resources:4814-4836
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-perf/btc/src/resources:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-perf/btc/src/resources:6766-7077
/branches/INT64_BRANCH/bigdata-perf/btc/src/resources:4486-4522
/branches/JOURNAL_HA_BRANCH/bigdata-perf/btc/src/resources:2596-4066
/branches/LARGE_LITERALS_REFACTOR/bigdata-perf/btc/src/resources:4175-4387
/branches/QUADS_QUERY_BRANCH/bigdata-perf/btc/src/resources:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-perf/btc/src/resources:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-perf/btc/src/resources:4814-4836
Property changes on: branches/READ_CACHE/bigdata-perf/lubm
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-perf/lubm:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-perf/lubm:6766-7048
/branches/INT64_BRANCH/bigdata-perf/lubm:4486-4522
/branches/LARGE_LITERALS_REFACTOR/bigdata-perf/lubm:4175-4387
/branches/QUADS_QUERY_BRANCH/bigdata-perf/lubm:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-perf/lubm:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-perf/lubm:4814-4836
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-perf/lubm:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-perf/lubm:6766-7077
/branches/INT64_BRANCH/bigdata-perf/lubm:4486-4522
/branches/LARGE_LITERALS_REFACTOR/bigdata-perf/lubm:4175-4387
/branches/QUADS_QUERY_BRANCH/bigdata-perf/lubm:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-perf/lubm:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-perf/lubm:4814-4836
Property changes on: branches/READ_CACHE/bigdata-perf/uniprot
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-perf/uniprot:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-perf/uniprot:6766-7048
/branches/INT64_BRANCH/bigdata-perf/uniprot:4486-4522
/branches/JOURNAL_HA_BRANCH/bigdata-perf/uniprot:2596-4066
/branches/LARGE_LITERALS_REFACTOR/bigdata-perf/uniprot:4175-4387
/branches/QUADS_QUERY_BRANCH/bigdata-perf/uniprot:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-perf/uniprot:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-perf/uniprot:4814-4836
/trunk/bigdata-perf/uniprot:2981-3043,3368-3437,3656-4061
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-perf/uniprot:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-perf/uniprot:6766-7077
/branches/INT64_BRANCH/bigdata-perf/uniprot:4486-4522
/branches/JOURNAL_HA_BRANCH/bigdata-perf/uniprot:2596-4066
/branches/LARGE_LITERALS_REFACTOR/bigdata-perf/uniprot:4175-4387
/branches/QUADS_QUERY_BRANCH/bigdata-perf/uniprot:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-perf/uniprot:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-perf/uniprot:4814-4836
/trunk/bigdata-perf/uniprot:2981-3043,3368-3437,3656-4061
Property changes on: branches/READ_CACHE/bigdata-perf/uniprot/src
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-perf/uniprot/src:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-perf/uniprot/src:6766-7048
/branches/INT64_BRANCH/bigdata-perf/uniprot/src:4486-4522
/branches/JOURNAL_HA_BRANCH/bigdata-perf/uniprot/src:2596-4066
/branches/LARGE_LITERALS_REFACTOR/bigdata-perf/uniprot/src:4175-4387
/branches/QUADS_QUERY_BRANCH/bigdata-perf/uniprot/src:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-perf/uniprot/src:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-perf/uniprot/src:4814-4836
+ /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-perf/uniprot/src:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-perf/uniprot/src:6766-7077
/branches/INT64_BRANCH/bigdata-perf/uniprot/src:4486-4522
/branches/JOURNAL_HA_BRANCH/bigdata-perf/uniprot/src:2596-4066
/branches/LARGE_LITERALS_REFACTOR/bigdata-perf/uniprot/src:4175-4387
/branches/QUADS_QUERY_BRANCH/bigdata-perf/uniprot/src:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-4792,4794-4796,4798-4801
/branches/RWSTORE_1_1_0_DEBUG/bigdata-perf/uniprot/src:5896-5935
/branches/TIDS_PLUS_BLOBS_BRANCH/bigdata-perf/uniprot/src:4814-4836
Property changes on: branches/READ_CACHE/bigdata-rdf/src/java/com/bigdata/bop/rdf/aggregate
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/BIGDATA_OPENRDF_2_6_9_UPDATE/bigdata-rdf/src/java/com/bigdata/bop/rdf/aggregate:6769-6785
/branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/bop/rdf/aggregate:6766-7048
/branches/INT64_BRANCH/bigdata-rdf/src/java/com/bigdata/bop/rdf/aggregate:4486-4522
/branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/bop/rdf/aggregate:4525-4531,4533-4548,4550-4584,4586-4609,4611-4632,4634-4643,4646-4672,4674-4685,4687-4693,4697-4735,4737-4782,4784-479...
[truncated message content] |
|
From: <tho...@us...> - 2013-04-25 15:59:09
|
Revision: 7077
http://bigdata.svn.sourceforge.net/bigdata/?rev=7077&view=rev
Author: thompsonbry
Date: 2013-04-25 15:59:02 +0000 (Thu, 25 Apr 2013)
Log Message:
-----------
You can now override the ServletContextListener implementation class using three methods. These methods depend on ConfigParams.SERVLET_CONTEXT_LISTENER_CLASS and the NanoSparqlServer.getContextHandler() method.
1. ConfigParams.SERVLET_CONTEXT_LISTENER_CLASS may be specified for the NanoSparqlServer.newInstance(..., initParams) as one of the initialization parameters. When specified, the given class will be used as the ServletContextListener.
init params and the NanoSparqlServer main() routine.
2. NanoSparqlServer.main() now accepts "-servletContextListenerClass".
3. If you are using a standard servlet engine, then also be accomplished using the standard web.xml semantics by modifying the following line.
{{{
?\194?\160<listener>
?\194?\160 <listener-class>com.bigdata.rdf.sail.webapp.BigdataRDFServletContextListener</listener-class>
</listener>
}}}
I will bring this change across to the READ_CACHE branch and then add support for this in the HAJournalServer as well.
See https://sourceforge.net/apps/trac/bigdata/ticket/667 (Provide NanoSparqlServer initialization hook).
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/BigdataRDFServletContextListener.java
branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/ConfigParams.java
branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/NanoSparqlServer.java
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/BigdataRDFServletContextListener.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/BigdataRDFServletContextListener.java 2013-04-25 15:34:18 UTC (rev 7076)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/BigdataRDFServletContextListener.java 2013-04-25 15:59:02 UTC (rev 7077)
@@ -90,12 +90,17 @@
*/
private boolean closeIndexManager;
+ public BigdataRDFServletContextListener() {
+ super();
+ }
+
protected BigdataRDFContext getBigdataRDFContext() {
return rdfContext;
}
+ @Override
public void contextInitialized(final ServletContextEvent e) {
if(log.isInfoEnabled())
@@ -382,6 +387,7 @@
}
+ @Override
public void contextDestroyed(final ServletContextEvent e) {
if(log.isInfoEnabled())
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/ConfigParams.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/ConfigParams.java 2013-04-25 15:34:18 UTC (rev 7076)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/ConfigParams.java 2013-04-25 15:59:02 UTC (rev 7077)
@@ -22,6 +22,9 @@
*/
package com.bigdata.rdf.sail.webapp;
+import javax.servlet.ServletContextEvent;
+import javax.servlet.ServletContextListener;
+
/**
* Interface declaring the <code>config-param</code>s understood by the
* {@link BigdataRDFServletContextListener}.
@@ -125,4 +128,23 @@
long DEFAULT_QUERY_TIMEOUT = 0L;
+ /**
+ * A class that extends {@link BigdataRDFServletContextListener}. This
+ * offers applications a means to hook the {@link ServletContextListener}
+ * methods.
+ * <p>
+ * Note:
+ *
+ * @see BigdataRDFServletContextListener#contextInitialized(ServletContextEvent)
+ * @see BigdataRDFServletContextListener#contextDestroyed(ServletContextEvent)
+ * @see #DEFAULT_SERVLET_CONTEXT_LISTENER_CLASS
+ *
+ * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/667" >
+ * Provide NanoSparqlServer initialization hook </a>
+ */
+ String SERVLET_CONTEXT_LISTENER_CLASS = "servletContextListenerClass";
+
+ String DEFAULT_SERVLET_CONTEXT_LISTENER_CLASS = BigdataRDFServletContextListener.class
+ .getName();
+
}
Modified: branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/NanoSparqlServer.java
===================================================================
--- branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/NanoSparqlServer.java 2013-04-25 15:34:18 UTC (rev 7076)
+++ branches/BIGDATA_RELEASE_1_2_0/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/NanoSparqlServer.java 2013-04-25 15:59:02 UTC (rev 7077)
@@ -27,6 +27,8 @@
import java.util.LinkedHashMap;
import java.util.Map;
+import javax.servlet.ServletContextListener;
+
import org.apache.log4j.Logger;
import org.eclipse.jetty.server.Handler;
import org.eclipse.jetty.server.Server;
@@ -140,6 +142,10 @@
* the database. Regardless, each query will be issued against a
* read-only transaction.</dt>
* </dl>
+ * <dt>servletContextListenerClass</dt>
+ * <dd>The name of a class that extends
+ * {@link BigdataRDFServletContextListener}. This allows you to
+ * hook the {@link ServletContextListener} events.</dd>
* </p>
*/
// * <dt>bufferCapacity [#bytes]</dt>
@@ -156,6 +162,7 @@
int queryThreadPoolSize = ConfigParams.DEFAULT_QUERY_THREAD_POOL_SIZE;
boolean forceOverflow = false;
Long readLock = null;
+ String servletContextListenerClass = ConfigParams.DEFAULT_SERVLET_CONTEXT_LISTENER_CLASS;
/*
* Handle all arguments starting with "-". These should appear before
@@ -184,6 +191,8 @@
"Read lock must be commit time or -1 (MINUS ONE) to assert a read lock on the last commit time: "
+ readLock);
}
+ } else if (arg.equals("-servletContextListenerClass")) {
+ servletContextListenerClass = args[++i];
} else {
usage(1/* status */, "Unknown argument: " + arg);
}
@@ -275,6 +284,9 @@
ConfigParams.READ_LOCK,
Long.toString(readLock));
}
+
+ initParams.put(ConfigParams.SERVLET_CONTEXT_LISTENER_CLASS,
+ servletContextListenerClass);
// Create the service.
final Server server = NanoSparqlServer.newInstance(port, propertyFile,
@@ -336,7 +348,7 @@
*/
static public Server newInstance(final int port,
final IIndexManager indexManager,
- final Map<String, String> initParams) {
+ final Map<String, String> initParams) throws Exception {
final Server server = new Server(port);
@@ -384,7 +396,7 @@
* Embedding Jetty </a>
*/
static public Server newInstance(final int port, final String propertyFile,
- final Map<String, String> initParams) {
+ final Map<String, String> initParams) throws Exception {
final Server server = new Server(port);
@@ -456,7 +468,7 @@
*/
static private ServletContextHandler getContextHandler(
final Server server,
- final Map<String, String> initParams) {
+ final Map<String, String> initParams) throws Exception {
if (initParams == null)
throw new IllegalArgumentException();
@@ -475,8 +487,32 @@
* Register a listener which will handle the life cycle events for the
* ServletContext.
*/
- context.addEventListener(new BigdataRDFServletContextListener());
+ {
+ String className = initParams
+ .get(ConfigParams.SERVLET_CONTEXT_LISTENER_CLASS);
+
+ if (className == null)
+ className = ConfigParams.DEFAULT_SERVLET_CONTEXT_LISTENER_CLASS;
+
+ final Class<BigdataRDFServletContextListener> cls = (Class<BigdataRDFServletContextListener>) Class
+ .forName(className);
+
+ if (!BigdataRDFServletContextListener.class.isAssignableFrom(cls)) {
+
+ throw new RuntimeException("Invalid option: "
+ + ConfigParams.SERVLET_CONTEXT_LISTENER_CLASS + "="
+ + className + ":: Class does not extend "
+ + BigdataRDFServletContextListener.class);
+
+ }
+
+ final BigdataRDFServletContextListener listener = cls.newInstance();
+
+ context.addEventListener(listener);
+
+ }
+
/*
* Set the servlet context properties.
*/
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
4 messages has been excluded from this view by a project administrator.
