Menu
▾
▴
bigdata-commit — Commit traffic.
This list is closed, nobody may subscribe to it.
| 2010 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(139) |
Aug
(94) |
Sep
(232) |
Oct
(143) |
Nov
(138) |
Dec
(55) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2011 |
Jan
(127) |
Feb
(90) |
Mar
(101) |
Apr
(74) |
May
(148) |
Jun
(241) |
Jul
(169) |
Aug
(121) |
Sep
(157) |
Oct
(199) |
Nov
(281) |
Dec
(75) |
| 2012 |
Jan
(107) |
Feb
(122) |
Mar
(184) |
Apr
(73) |
May
(14) |
Jun
(49) |
Jul
(26) |
Aug
(103) |
Sep
(133) |
Oct
(61) |
Nov
(51) |
Dec
(55) |
| 2013 |
Jan
(59) |
Feb
(72) |
Mar
(99) |
Apr
(62) |
May
(92) |
Jun
(19) |
Jul
(31) |
Aug
(138) |
Sep
(47) |
Oct
(83) |
Nov
(95) |
Dec
(111) |
| 2014 |
Jan
(125) |
Feb
(60) |
Mar
(119) |
Apr
(136) |
May
(270) |
Jun
(83) |
Jul
(88) |
Aug
(30) |
Sep
(47) |
Oct
(27) |
Nov
(23) |
Dec
|
| 2015 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(3) |
Oct
|
Nov
|
Dec
|
| 2016 |
Jan
|
Feb
|
Mar
(4) |
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: <tho...@us...> - 2010-11-03 11:52:15
|
Revision: 3874
http://bigdata.svn.sourceforge.net/bigdata/?rev=3874&view=rev
Author: thompsonbry
Date: 2010-11-03 11:52:07 +0000 (Wed, 03 Nov 2010)
Log Message:
-----------
Backing out the introduction of a weak reference into the WriteExecutorService in order to track down some odd errors which this appears to have introduced. This is per [1].
[1] https://sourceforge.net/apps/trac/bigdata/ticket/196
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/WriteExecutorService.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/journal/TestJournalShutdown.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/WriteExecutorService.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/WriteExecutorService.java 2010-11-02 19:00:31 UTC (rev 3873)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/WriteExecutorService.java 2010-11-03 11:52:07 UTC (rev 3874)
@@ -221,7 +221,12 @@
private static class MyLockManager<R extends Comparable<R>> extends
NonBlockingLockManagerWithNewDesign<R> {
-// private final WriteExecutorService service;
+ /*
+ * FIXME restored hard reference since introducing just a weak reference
+ * here appears to be causing some odd behaviors. Track these behaviors
+ * down and sort this all out.
+ */
+ private final WriteExecutorService service;
private final WeakReference<WriteExecutorService> serviceRef;
public MyLockManager(final int capacity, final int maxLockTries,
@@ -230,7 +235,7 @@
super(capacity, maxLockTries, predeclareLocks);
-// this.service = service;
+ this.service = service;
this.serviceRef = new WeakReference<WriteExecutorService>(service);
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/journal/TestJournalShutdown.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/journal/TestJournalShutdown.java 2010-11-02 19:00:31 UTC (rev 3873)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/journal/TestJournalShutdown.java 2010-11-03 11:52:07 UTC (rev 3874)
@@ -116,7 +116,8 @@
if (nalive.get() == ncreated.get()) {
- fail("No journals were finalized.");
+ fail("Created " + ncreated
+ + " journals. No journals were finalized.");
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-11-02 19:00:39
|
Revision: 3873
http://bigdata.svn.sourceforge.net/bigdata/?rev=3873&view=rev
Author: thompsonbry
Date: 2010-11-02 19:00:31 +0000 (Tue, 02 Nov 2010)
Log Message:
-----------
A bunch of edits to clean up RWStrategy and RWStore.
There is a deadlock which can be demonstrated using StressTestConcurrentUnisolatedIndices and which results in a deadlock. It seems like there are probably too many distinct locks -- or that the m_allocationLock needs to be acquired before the m_deferFreeLock (which probably means that the m_deferFreeLock is redundant). Either way, this should be easy enough to clean up.
This thread is holding the Journal's internal lock, holding the RWStore#m_deferFreeLock (in saveDeferrals), and waiting on RWStore#m_allocationLock (in alloc()):
com.bigdata.journal.ConcurrencyManager.writeService14 [WAITING] CPU time: 0:00
java.util.concurrent.locks.ReentrantLock.lock()
com.bigdata.rwstore.RWStore.alloc(int, IAllocationContext)
com.bigdata.rwstore.RWStore.alloc(byte[], int, IAllocationContext)
com.bigdata.rwstore.PSOutputStream.save()
com.bigdata.rwstore.RWStore.saveDeferrals()
com.bigdata.journal.RWStrategy.saveDeleteBlocks()
com.bigdata.journal.DeleteBlockCommitter.handleCommit(long)
com.bigdata.journal.AbstractJournal.notifyCommitters(long)
com.bigdata.journal.AbstractJournal.commitNow(long)
com.bigdata.journal.AbstractJournal.commit()
com.bigdata.journal.WriteExecutorService.commit(boolean)
com.bigdata.journal.WriteExecutorService.groupCommit()
com.bigdata.journal.WriteExecutorService.afterTask(AbstractTask, Throwable)
com.bigdata.journal.AbstractTask.doUnisolatedReadWriteTask()
com.bigdata.journal.AbstractTask.call2()
com.bigdata.journal.AbstractTask.call()
java.util.concurrent.FutureTask.run()
com.bigdata.concurrent.NonBlockingLockManagerWithNewDesign$LockFutureTask.run()
java.lang.Thread.run()
This thread is holding m_allocationLock (in free()) and waiting on m_deferFreeLock:
com.bigdata.journal.ConcurrencyManager.writeService7 [WAITING] CPU time: 0:00
java.util.concurrent.locks.ReentrantLock.lock()
com.bigdata.rwstore.RWStore.deferFree(int, int)
com.bigdata.rwstore.RWStore.free(long, int, IAllocationContext)
com.bigdata.journal.RWStrategy.delete(long, IAllocationContext)
com.bigdata.journal.RWStrategy.delete(long)
com.bigdata.journal.AbstractJournal.delete(long)
com.bigdata.btree.Node.replaceChildRef(long, AbstractNode)
com.bigdata.btree.AbstractNode.copyOnWrite(long)
com.bigdata.btree.AbstractNode.copyOnWrite()
com.bigdata.btree.Leaf.insert(byte[], byte[], boolean, long, Tuple)
com.bigdata.btree.Node.insert(byte[], byte[], boolean, long, Tuple)
com.bigdata.btree.Node.insert(byte[], byte[], boolean, long, Tuple)
com.bigdata.btree.AbstractBTree.insert(byte[], byte[], boolean, long, Tuple)
com.bigdata.btree.AbstractBTree.insert(byte[], byte[])
com.bigdata.journal.StressTestConcurrentUnisolatedIndices$WriteTask.doTask()
com.bigdata.journal.AbstractTask$InnerWriteServiceCallable.call()
com.bigdata.journal.AbstractTask.doUnisolatedReadWriteTask()
com.bigdata.journal.AbstractTask.call2()
com.bigdata.journal.AbstractTask.call()
java.util.concurrent.FutureTask.run()
com.bigdata.concurrent.NonBlockingLockManagerWithNewDesign$LockFutureTask.run()
java.lang.Thread.run()
Modified Paths:
--------------
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/io/writecache/WriteCacheService.java
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/IBufferStrategy.java
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/RWStrategy.java
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java
branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/journal/StressTestConcurrentTx.java
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/io/writecache/WriteCacheService.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/io/writecache/WriteCacheService.java 2010-11-02 17:46:33 UTC (rev 3872)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/io/writecache/WriteCacheService.java 2010-11-02 19:00:31 UTC (rev 3873)
@@ -325,10 +325,10 @@
int addrsUsedCurs = 0;
final char[] addrActions = new char[addrsUsed.length];
final int[] addrLens = new int[addrsUsed.length];
-*/ final long[] addrsUsed = null;
- int addrsUsedCurs = 0;
- final char[] addrActions = null;
- final int[] addrLens = null;
+*/ private final long[] addrsUsed = null;
+ private int addrsUsedCurs = 0;
+ private final char[] addrActions = null;
+ private final int[] addrLens = null;
/**
* The current file extent.
@@ -1989,7 +1989,7 @@
* An array of writeCache actions is maintained that can be used
* to provide a breadcrumb of how that address has been written, saved,
* freed or removed.
- *
+ * <p>
* Write errors often show up as a checksum error, so the length of
* data written to the address cab be crucial information in determining the
* root of any problem.
@@ -2002,9 +2002,9 @@
return "No WriteCache debug info";
}
- StringBuffer ret = new StringBuffer();
- // first see if address was ever written
- boolean written = false;
+ final StringBuffer ret = new StringBuffer();
+// // first see if address was ever written
+// boolean written = false;
for (int i = 0; i < addrsUsed.length; i++) {
if (i == addrsUsedCurs) {
ret.append("|...|");
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/IBufferStrategy.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/IBufferStrategy.java 2010-11-02 17:46:33 UTC (rev 3872)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/IBufferStrategy.java 2010-11-02 19:00:31 UTC (rev 3873)
@@ -28,7 +28,6 @@
import java.nio.ByteBuffer;
import com.bigdata.counters.CounterSet;
-import com.bigdata.io.writecache.WriteCacheService;
import com.bigdata.rawstore.IAddressManager;
import com.bigdata.rawstore.IMRMW;
import com.bigdata.rawstore.IRawStore;
@@ -204,55 +203,65 @@
public IAddressManager getAddressManager();
/**
- * A method that removes assumptions of how a specific strategy determines whether a transaction commit is required.
+ * A method that removes assumptions of how a specific strategy determines
+ * whether a transaction commit is required.
*
* @param block
- * The root block held by the client, can be checked against the state of the Buffer Strategy
+ * The root block held by the client, can be checked against the
+ * state of the Buffer Strategy
* @return whether any modification has occurred.
*/
- public boolean requiresCommit(IRootBlockView block);
+ public boolean requiresCommit(IRootBlockView block);
/**
- * A method that removes assumptions of how a specific strategy commits data. For most strategies the action is void
- * since the client WORM DISK strategy writes data as allocated. For the Read Write Strategy more data must be managed
- * as part of the protocol outside of the RootBlock, and this is the method that triggers that management.
- * @param abstractJournal
+ * A method that removes assumptions of how a specific strategy commits
+ * data. For most strategies the action is void since the client WORM DISK
+ * strategy writes data as allocated. For the Read Write Strategy more data
+ * must be managed as part of the protocol outside of the RootBlock, and
+ * this is the method that triggers that management.
+ *
+ * @param abstractJournal
*/
- public void commit(IJournal journal);
+ public void commit(IJournal journal);
/**
* A method that requires the implementation to discard its buffered write
* set (if any). The caller is responsible for any necessary synchronization
* as part of the abort protocol.
*/
- public void abort();
-
- /**
- * The RWStrategy requires meta allocation info in the root block, this method is the hook to enable access.
- * The metaStartAddr is the address in the file where the allocation blocks are stored.
- *
- * @return the metaStartAddr for the root block if any
- */
- public long getMetaStartAddr();
- /**
- * The RWStrategy requires meta allocation info in the root block, this method is the hook to enable access.
- * The metaBitsAddr is the address in the file where the metaBits that control the allocation of the allocation
- * blocks themselves is stored.
- *
- * @return the metaBitsAddr for the root block if any
- */
- public long getMetaBitsAddr();
- /**
- * @return the number of bits available in the address to define offset
- */
+ public void abort();
- public int getOffsetBits();
- /**
- * @return the maximum record size supported by this strategy
- */
- public int getMaxRecordSize();
+ /**
+ * The RWStrategy requires meta allocation info in the root block, this
+ * method is the hook to enable access. The metaStartAddr is the address in
+ * the file where the allocation blocks are stored.
+ *
+ * @return the metaStartAddr for the root block if any
+ */
+ public long getMetaStartAddr();
/**
+ * The RWStrategy requires meta allocation info in the root block, this
+ * method is the hook to enable access. The metaBitsAddr is the address in
+ * the file where the metaBits that control the allocation of the allocation
+ * blocks themselves is stored.
+ *
+ * @return the metaBitsAddr for the root block if any
+ */
+ public long getMetaBitsAddr();
+
+ /**
+ * @return the number of bits available in the address to define offset
+ */
+
+ public int getOffsetBits();
+
+ /**
+ * @return the maximum record size supported by this strategy
+ */
+ public int getMaxRecordSize();
+
+ /**
* Return <code>true</code> if the store uses per-record checksums. When
* <code>true</code>, an additional 4 bytes are written after the record on
* the disk. Those bytes contain the checksum of the record.
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/RWStrategy.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/RWStrategy.java 2010-11-02 17:46:33 UTC (rev 3872)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/RWStrategy.java 2010-11-02 19:00:31 UTC (rev 3873)
@@ -24,15 +24,12 @@
package com.bigdata.journal;
-import java.io.DataInput;
-import java.io.DataOutput;
import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.ByteBuffer;
import java.nio.channels.ClosedByInterruptException;
import java.util.UUID;
-import java.util.concurrent.Callable;
import java.util.concurrent.locks.ReentrantLock;
import org.apache.log4j.Logger;
@@ -45,7 +42,6 @@
import com.bigdata.rawstore.IAddressManager;
import com.bigdata.rwstore.IAllocationContext;
import com.bigdata.rwstore.RWStore;
-import com.bigdata.service.AbstractTransactionService;
import com.bigdata.util.ChecksumUtility;
/**
@@ -68,8 +64,9 @@
* @author Martyn Cutcher
*/
public class RWStrategy extends AbstractRawStore implements IBufferStrategy, IHABufferStrategy {
- protected static final Logger log = Logger.getLogger(RWStrategy.class);
+ private static final transient Logger log = Logger.getLogger(RWStrategy.class);
+
final private FileMetadata m_fileMetadata;
final private Quorum<?,?> m_environment;
@@ -88,14 +85,20 @@
private volatile IRootBlockView m_rb;
private volatile IRootBlockView m_rb0;
private volatile IRootBlockView m_rb1;
-
- final ReentrantLock m_commitLock = new ReentrantLock();
-
+
/**
- * Access to the transaction manager of any owning Journal, needed by
- * RWStrategy to manager deleted data.
+ * @todo The use of this lock is suspicious. It is only used by
+ * {@link #commit(IJournal)} and that method is invoked by the
+ * {@link AbstractJournal#commitNow(long)} which is already protected
+ * by a lock.
*/
- private AbstractLocalTransactionManager localTransactionManager = null;
+ private final ReentrantLock m_commitLock = new ReentrantLock();
+
+// /**
+// * Access to the transaction manager of any owning Journal, needed by
+// * RWStrategy to manager deleted data.
+// */
+// private AbstractLocalTransactionManager localTransactionManager = null;
// CounterSet m_counters = new CounterSet();
@@ -125,16 +128,26 @@
}
/**
+ * Return a copy of the current {@link IRootBlockView}.
+ *
* @param rb0
- * @return
+ * When <code>true</code> the view will be flagged as root block
+ * ZERO (0). Otherwise it is flagged as root block ONE (1).
+ *
+ * @return The {@link IRootBlockView}.
*/
- IRootBlockView copyRootBlock(boolean rb0) {
- IRootBlockView rbv = new RootBlockView(rb0, m_rb.getOffsetBits(), m_rb.getNextOffset(), m_rb.getFirstCommitTime(), m_rb.getLastCommitTime(),
- m_rb.getCommitCounter(), m_rb.getCommitRecordAddr(), m_rb.getCommitRecordIndexAddr(), m_fileMetadata.rootBlock.getUUID(),
- m_rb.getQuorumToken(),
- m_rb.getMetaStartAddr(), m_rb.getMetaBitsAddr(), StoreTypeEnum.RW, m_fileMetadata.rootBlock.getCreateTime(), m_rb.getCloseTime(),
- s_ckutil);
+ private IRootBlockView copyRootBlock(final boolean rb0) {
+ final IRootBlockView rbv = new RootBlockView(rb0, m_rb.getOffsetBits(),
+ m_rb.getNextOffset(), m_rb.getFirstCommitTime(), m_rb
+ .getLastCommitTime(), m_rb.getCommitCounter(), m_rb
+ .getCommitRecordAddr(),
+ m_rb.getCommitRecordIndexAddr(), m_fileMetadata.rootBlock
+ .getUUID(), m_rb.getQuorumToken(), m_rb
+ .getMetaStartAddr(), m_rb.getMetaBitsAddr(),
+ StoreTypeEnum.RW, m_fileMetadata.rootBlock.getCreateTime(),
+ m_rb.getCloseTime(), s_ckutil);
+
return rbv;
}
@@ -192,14 +205,19 @@
return m_fileMetadata.raf;
}
- public IRootBlockView newRootBlockView(boolean rootBlock0, int offsetBits, long nextOffset,
- long firstCommitTime, long lastCommitTime, long commitCounter, long commitRecordAddr,
- long commitRecordIndexAddr, long metaStartAddr, long metaBitsAddr, long closeTime) {
+ public IRootBlockView newRootBlockView(boolean rootBlock0,
+ int offsetBits, long nextOffset, long firstCommitTime,
+ long lastCommitTime, long commitCounter, long commitRecordAddr,
+ long commitRecordIndexAddr, long metaStartAddr,
+ long metaBitsAddr, long closeTime) {
- IRootBlockView rbv = new RootBlockView(rootBlock0, offsetBits, nextOffset, firstCommitTime, lastCommitTime,
- commitCounter, commitRecordAddr, commitRecordIndexAddr, m_fileMetadata.rootBlock.getUUID(), -1 /* FIXME: quorumToken */,
- metaStartAddr, metaBitsAddr, StoreTypeEnum.RW, m_fileMetadata.rootBlock.getCreateTime(), closeTime,
- s_ckutil);
+ final IRootBlockView rbv = new RootBlockView(rootBlock0,
+ offsetBits, nextOffset, firstCommitTime, lastCommitTime,
+ commitCounter, commitRecordAddr, commitRecordIndexAddr,
+ m_fileMetadata.rootBlock.getUUID(),
+ -1 /* FIXME: quorumToken */, metaStartAddr, metaBitsAddr,
+ StoreTypeEnum.RW, m_fileMetadata.rootBlock.getCreateTime(),
+ closeTime, s_ckutil);
// writeRootBlock(rbv, ForceEnum.Force); // not sure if this is really needed now!
@@ -211,17 +229,19 @@
}
}
- public ByteBuffer readRootBlock(boolean rootBlock0) {
- checkReopen();
+ public ByteBuffer readRootBlock(final boolean rootBlock0) {
+
+ checkReopen();
IRootBlockView rbv = rootBlock0 ? m_rb0 : m_rb1;
return rbv.asReadOnlyBuffer();
}
- public ByteBuffer read(long addr) {
- checkReopen();
+ public ByteBuffer read(final long addr) {
+ checkReopen();
+
int rwaddr = decodeAddr(addr);
int sze = decodeSize(addr);
@@ -239,12 +259,15 @@
return ByteBuffer.wrap(buf, 0, sze);
}
- public long write(ByteBuffer data) {
- return write(data, null);
+ public long write(final ByteBuffer data) {
+
+ return write(data, null);
+
}
- public long write(ByteBuffer data, IAllocationContext context) {
- checkReopen();
+ public long write(final ByteBuffer data, final IAllocationContext context) {
+
+ checkReopen();
if (data == null) {
throw new IllegalArgumentException();
@@ -291,7 +314,7 @@
// return encodeAddr(m_store.alloc(nbytes), nbytes);
// }
- private long encodeAddr(long alloc, int nbytes) {
+ private long encodeAddr(long alloc, final int nbytes) {
alloc <<= 32;
alloc += nbytes;
@@ -304,52 +327,70 @@
return (int) addr;
}
- private int decodeSize(long addr) {
+ private int decodeSize(final long addr) {
return (int) (addr & 0xFFFFFFFF);
}
- public void delete(long addr) {
- delete(addr, null);
+ public void delete(final long addr) {
+
+ delete(addr, null/* IAllocationContext */);
+
}
/**
* Must check whether there are existing transactions which may access
* this data, and if not free immediately, otherwise defer.
*/
- public void delete(long addr, IAllocationContext context) {
+ public void delete(final long addr, final IAllocationContext context) {
final int rwaddr = decodeAddr(addr);
final int sze = decodeSize(addr);
m_store.free(rwaddr, sze, context);
+
}
- public void detachContext(IAllocationContext context) {
- m_store.detachContext(context);
+ public void detachContext(final IAllocationContext context) {
+
+ m_store.detachContext(context);
+
}
-
+ /*
+ * FIXME Reconcile this class with the methods on the outer class.
+ */
public static class RWAddressManager implements IAddressManager {
- public int getByteCount(long addr) {
- return (int) addr & 0xFFFFFF;
+ public int getByteCount(final long addr) {
+
+ return (int) addr & 0xFFFFFF;
+
}
- public long getOffset(long addr) {
- return -(addr >> 32);
+ public long getOffset(final long addr) {
+
+ return -(addr >> 32);
+
}
- public long toAddr(int nbytes, long offset) {
- offset <<= 32;
+ public long toAddr(final int nbytes, long offset) {
+
+ offset <<= 32;
return offset + nbytes;
+
}
- public String toString(long addr) {
- return "{off="+getOffset(addr)+",len="+getByteCount(addr)+"}";
+ public String toString(final long addr) {
+
+ return "{off=" + getOffset(addr) + ",len=" + getByteCount(addr)
+ + "}";
+
}
+
}
- IAddressManager m_am = new RWAddressManager();
+
+ final private IAddressManager m_am = new RWAddressManager();
public IAddressManager getAddressManager() {
return m_am;
@@ -382,9 +423,9 @@
return 0;
}
- long m_initialExtent = 0;
+ private long m_initialExtent = 0;
- private boolean m_needsReopen = false;
+ private volatile boolean m_needsReopen = false;
public long getInitialExtent() {
return m_initialExtent;
@@ -485,19 +526,33 @@
}
public void deleteResources() {
- if (m_fileMetadata.raf != null && m_fileMetadata.raf.getChannel().isOpen()) {
- throw new IllegalStateException("Backing store is open");
- }
- if (m_fileMetadata.file.exists()) {
- try {
- if (!m_fileMetadata.file.delete()) {
- log.warn("Unable to delete file: " + m_fileMetadata.file);
- }
- } catch (SecurityException e) {
- log.warn("Problem deleting file", e);
- }
- }
+ if (m_fileMetadata.raf != null
+ && m_fileMetadata.raf.getChannel().isOpen()) {
+
+ throw new IllegalStateException("Backing store is open: "
+ + m_fileMetadata.file);
+
+ }
+
+ if (m_fileMetadata.file.exists()) {
+
+ try {
+
+ if (!m_fileMetadata.file.delete()) {
+
+ log.warn("Unable to delete file: " + m_fileMetadata.file);
+
+ }
+
+ } catch (SecurityException e) {
+
+ log.warn("Problem deleting file", e);
+
+ }
+
+ }
+
}
public void destroy() {
@@ -515,10 +570,18 @@
}
public IRootBlockView getRootBlock() {
- return m_fmv.newRootBlockView(! m_rb.isRootBlock0(), m_rb.getOffsetBits(), getNextOffset(),
- m_rb.getFirstCommitTime(), m_rb.getLastCommitTime(), m_rb.getCommitCounter(),
- m_rb.getCommitRecordAddr(), m_rb.getCommitRecordIndexAddr(), getMetaStartAddr(), getMetaBitsAddr(), m_rb.getCloseTime() );
-
+ return m_fmv.newRootBlockView(!m_rb.isRootBlock0(), //
+ m_rb.getOffsetBits(),//
+ getNextOffset(), //
+ m_rb.getFirstCommitTime(),//
+ m_rb.getLastCommitTime(), //
+ m_rb.getCommitCounter(),//
+ m_rb.getCommitRecordAddr(), //
+ m_rb.getCommitRecordIndexAddr(), //
+ getMetaStartAddr(),//
+ getMetaBitsAddr(), //
+ m_rb.getCloseTime()//
+ );
}
/**
@@ -526,7 +589,7 @@
*
* Must pass in earliestTxnTime to commitChanges to enable
*/
- public void commit(IJournal journal) {
+ public void commit(final IJournal journal) {
m_commitLock.lock();
try {
m_store.commitChanges((Journal) journal); // includes a force(false)
@@ -544,7 +607,7 @@
m_store.reset();
}
- public void force(boolean metadata) {
+ public void force(final boolean metadata) {
try {
m_store.flushWrites(metadata);
} catch (ClosedByInterruptException e) {
@@ -572,7 +635,8 @@
log.warn("Request to reopen store after interrupt");
m_store.close();
- m_fileMetadata.raf = new RandomAccessFile(m_fileMetadata.file, m_fileMetadata.fileMode);
+ m_fileMetadata.raf = new RandomAccessFile(m_fileMetadata.file,
+ m_fileMetadata.fileMode);
m_store = new RWStore(m_fmv, false, m_environment); // never read-only for now
m_needsReopen = false;
m_open = true;
@@ -592,11 +656,19 @@
return m_fileMetadata.raf;
}
- public IResourceMetadata getResourceMetadata() {
- // TODO Auto-generated method stub
- return null;
- }
-
+
+ /**
+ * Not supported - this is available on the {@link AbstractJournal}.
+ *
+ * @throws UnsupportedOperationException
+ * always
+ */
+ public IResourceMetadata getResourceMetadata() {
+
+ throw new UnsupportedOperationException();
+
+ }
+
public UUID getUUID() {
return m_fileMetadata.rootBlock.getUUID();
}
@@ -631,25 +703,14 @@
return addr >> 32;
}
- public void packAddr(DataOutput out, long addr) throws IOException {
- // TODO Auto-generated method stub
-
- }
-
- public long toAddr(int nbytes, long offset) {
+ public long toAddr(final int nbytes, final long offset) {
return (offset << 32) + nbytes;
}
- public String toString(long addr) {
- // TODO Auto-generated method stub
- return null;
+ public String toString(final long addr) {
+ return m_am.toString(addr);
}
- public long unpackAddr(DataInput in) throws IOException {
- // TODO Auto-generated method stub
- return 0;
- }
-
/**
* The state of the provided block is not relevant since it does not hold
* information on recent allocations (the meta allocations will only effect the
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2010-11-02 17:46:33 UTC (rev 3872)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2010-11-02 19:00:31 UTC (rev 3873)
@@ -35,11 +35,10 @@
import java.nio.channels.FileChannel;
import java.util.ArrayList;
import java.util.Collections;
-import java.util.HashMap;
import java.util.Iterator;
+import java.util.List;
import java.util.Map;
-import java.util.concurrent.Callable;
-import java.util.concurrent.TimeUnit;
+import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.atomic.AtomicLong;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
@@ -70,128 +69,130 @@
/**
* Storage class
- *
- * Author: Martyn Cutcher
- *
+ * <p>
* Provides an interface to allocating storage within a disk file.
- *
+ * <p>
* Essentially provides a DiskMalloc interface.
- *
+ * <p>
* In addition to the DiskMalloc/ReAlloc mechanism, a single root address can be
* associated. This can be used when opening an existing storage file to
* retrieve some management object - such as an object manager!
- *
+ * <p>
* The allocator also support atomic update via a simple transaction mechanism.
- *
+ * <p>
* Updates are normally committed immediately, but by using startTransaction and
* commitTransaction, the previous state of the store is retained until the
* moment of commitment.
- *
- * It would also be possible to add some journalling/version mechanism, where
+ * <p>
+ * It would also be possible to add some journaling/version mechanism, where
* snapshots of the allocation maps are retained for sometime. For a store which
* was only added to this would not be an unreasonable overhead and would
* support the rolling back of the database weekly or monthly if required.
- *
+ * <p>
* The input/output mechanism uses ByteArray Input and Output Streams.
- *
+ * <p>
* One difference between the disk realloc and in memory realloc is that the
* disk realloc will always return a new address and mark the old address as
* ready to be freed.
- *
+ * <p>
* The method of storing the allocation headers has been changed from always
- * allocting at the end of the file (and moving them on fle extend) to
+ * allocating at the end of the file (and moving them on fle extend) to
* allocation of fixed areas. The meta-allocation data, containing the bitmap
* that controls these allocations, is itself stored in the heap, and is now
* structured to include both the bit data and the list of meta-storage
* addresses.
- *
+ * <p>
* Sizing:
* 256 allocators would reference approximately 2M objects/allocations. At 1K
* per allocator this would require 250K of store. The meta-allocation data
* would therefore need a start address plus 32 bytes (or 8 ints) to represent
- * the meta-allocation bits. An array of such data referencing sequentialy
+ * the meta-allocation bits. An array of such data referencing sequentially
* allocated storage areas completes the meta-allocation requirements.
- *
+ * <p>
* A meta-allocation address can therefore be represented as a single bit offset
* from which the block, providing start address, and bit offset can be
* directly determined.
- *
- * The m_metaBits int array used to be fully used as allocaiton bits, but
+ * <p>
+ * The m_metaBits int array used to be fully used as allocation bits, but
* now stores both the start address plus the 8 ints used to manage that data
* block.
- *
+ * <p>
* Allocation is reduced to sets of allocator objects which have a start address
* and a bitmap of allocated storage maps.
- *
+ * <p>
* Searching thousands of allocation blocks to find storage is not efficient,
- * but by utilising roving pointers and sorting blocks with free space available
+ * but by utilizing roving pointers and sorting blocks with free space available
* this can be made most efficient.
- *
+ * <p>
* In order to provide optimum use of bitmaps, this implementation will NOT use
* the BitSet class.
- *
+ * <p>
* Using the meta-allocation bits, it is straightforward to load ALL the
* allocation headers. A total of (say) 100 allocation headers might provide
- * upto 4000 allocations each -> 400 000 objects, while 1000 headers -> 4m
+ * up to 4000 allocations each -> 400 000 objects, while 1000 headers -> 4m
* objects and 2000 -> 8m objects.
- *
+ * <p>
* The allocators are split into a set of FixedAllocators and then
* BlobAllocation. The FixedAllocators will allocate from 128 to 32K objects,
* with a minimum block allocation of 64K, and a minimum bit number per block of
* 32.
- *
+ * <p>
* Where possible lists and roving pointers will be used to minimise searching
* of the potentially large structures.
- *
+ * <p>
* Since the memory is allocated on (at least) a 128 byte boundary, there is
* some leeway on storing the address. Added to the address is the shift
* required to make to the "standard" 128 byte block, e.g. blocksize = 128 <<
* (addr % 8)
- *
+ * <p>
* NB Useful method on RandomAccessFile.setLength(newLength)
- *
+ * <p>
* When session data is preserved two things must happen - the allocators must
* not reallocate data that has been freed in this session, or more clearly can
* only free data that has been allocated in this session. That should be it.
- *
+ * <p>
* The ALLOC_SIZES table is the fibonacci sequence. We multiply by 64 bytes to
* get actual allocation block sizes. We then allocate bits based on 8K
* allocation rounding and 32 bits at a time allocation. Note that 4181 * 64 =
* 267,584 and 256K is 262,144
- *
+ * <p>
* All data is checksummed, both allocated/saved data and the allocation blocks.
- *
+ * <p>
* BLOB allocation is not handled using chained data buffers but with a blob
* header record. This is indicated with a BlobAllocator that provides indexed
* offsets to the header record (the address encodes the BlobAllocator and the
* offset to the address). The header record stores the number of component
* allocations and the address of each.
- *
+ * <p>
* This approach makes for much more efficient freeing/re-allocation of Blob
* storage, in particular avoiding the need to read in the component blocks
* to determine chained blocks for freeing. This is particularly important
* for larger stores where a disk cache could be flushed through simply freeing
* BLOB allocations.
- *
+ * <h2>
* Deferred Free List
- *
- * The prevous implentation has been amended to associate a single set of
+ * </h2>
+ * <p>
+ * The previous implementation has been amended to associate a single set of
* deferredFree blocks with each CommitRecord. The CommitRecordIndex will
* then provide access to the CommitRecords to support the deferred freeing
* of allocations based on age/earliestTxReleaseTime.
+ * <p>
+ * The last release time processed is held with the MetaAllocation data
*
- * The last release time processed is held with the MetaAllocation data
+ * @author Martyn Cutcher
*/
public class RWStore implements IStore {
- protected static final Logger log = Logger.getLogger(RWStore.class);
+ private static final transient Logger log = Logger.getLogger(RWStore.class);
+
/**
* The sizes of the slots managed by a {@link FixedAllocator} are 64 times
* the values in this array.
*
* @todo good to have 4k and 8k boundaries for better efficiency on SSD.
- * NB A 1K boundry is % 16, so 4K % 64
+ * NB A 1K boundary is % 16, so 4K % 64
* - can still use fibonacci base, but from 4K start
*
* @todo This array should be configurable and must be written into the
@@ -240,8 +241,8 @@
// protected int m_transactionCount;
// private boolean m_committing;
- boolean m_preserveSession = true;
- private boolean m_readOnly;
+ private boolean m_preserveSession = true;
+// private boolean m_readOnly;
/**
* lists of total alloc blocks.
@@ -259,13 +260,13 @@
private final ArrayList<BlobAllocator> m_freeBlobs;
// lists of blocks requiring commitment
- private final ArrayList m_commitList;
+ private final ArrayList<Allocator> m_commitList;
private WriteBlock m_writes;
private final Quorum<?,?> m_quorum;
- private RWWriteCacheService m_writeCache;
+ private final RWWriteCacheService m_writeCache;
- int[] m_allocSizes;
+ private int[] m_allocSizes;
/**
* This lock is used to exclude readers when the extent of the backing file
@@ -310,28 +311,28 @@
/**
* The deferredFreeList is simply an array of releaseTime,freeListAddrs
* stored at commit.
- *
+ * <p>
* Note that when the deferredFreeList is saved, ONLY thefreeListAddrs
* are stored, NOT the releaseTime. This is because on any open of
* the store, all deferredFrees can be released immediately. This
* mechanism may be changed in the future to enable explicit history
* retention, but if so a different header structure would be used since
* it would not be appropriate to retain a simple header linked to
- * thousands if not milions of commit points.
- *
- * If the current txn list exceeds the MAX_DEFERRED_FREE then it is
- * incrementally saved and a new list begun. The master list itself
- * serves as a BLOB header when there is more than a single entry with
- * the same txReleaseTime.
+ * thousands if not millions of commit points.
*/
- private static final int MAX_DEFERRED_FREE = 4094; // fits in 16k block
- volatile long m_lastDeferredReleaseTime = 0L;// = 23; // zero is invalid time
- final ArrayList<Integer> m_currentTxnFreeList = new ArrayList<Integer>();
- final PSOutputStream m_deferredFreeOut;
+// *
+// * If the current txn list exceeds the MAX_DEFERRED_FREE then it is
+// * incrementally saved and a new list begun. The master list itself
+// * serves as a BLOB header when there is more than a single entry with
+// * the same txReleaseTime.
+// private static final int MAX_DEFERRED_FREE = 4094; // fits in 16k block
+ private volatile long m_lastDeferredReleaseTime = 0L;
+// private final ArrayList<Integer> m_currentTxnFreeList = new ArrayList<Integer>();
+ private final PSOutputStream m_deferredFreeOut;
private ReopenFileChannel m_reopener = null;
- BufferedWrite m_bufferedWrite;
+ private BufferedWrite m_bufferedWrite;
class WriteCacheImpl extends WriteCache.FileChannelScatteredWriteCache {
public WriteCacheImpl(final ByteBuffer buf,
@@ -350,14 +351,16 @@
final long firstOffsetignored,
final Map<Long, RecordMetadata> recordMap,
final long nanos) throws InterruptedException, IOException {
- Lock readLock = m_extensionLock.readLock();
- readLock.lock();
- try {
- return super.writeOnChannel(data, firstOffsetignored, recordMap, nanos);
- } finally {
- readLock.unlock();
- }
-
+
+ final Lock readLock = m_extensionLock.readLock();
+ readLock.lock();
+ try {
+ return super.writeOnChannel(data, firstOffsetignored,
+ recordMap, nanos);
+ } finally {
+ readLock.unlock();
+ }
+
}
// Added to enable debug of rare problem
@@ -368,28 +371,28 @@
};
- private String m_filename;
+// private String m_filename;
private final FileMetadataView m_fmv;
private IRootBlockView m_rb;
- volatile private long m_commitCounter;
+// volatile private long m_commitCounter;
volatile private int m_metaBitsAddr;
- /**
- * The ALLOC_SIZES must be initialised from either the file
- * or the properties associaed with the fileMetadataView
- *
- * @param fileMetadataView
- * @param readOnly
- * @param quorum
- * @throws InterruptedException
- */
+ /**
+ * The ALLOC_SIZES must be initialized from either the file or the
+ * properties associated with the fileMetadataView
+ *
+ * @param fileMetadataView
+ * @param readOnly
+ * @param quorum
+ * @throws InterruptedException
+ */
- public RWStore(final FileMetadataView fileMetadataView, final boolean readOnly,
- final Quorum<?,?> quorum) {
+ public RWStore(final FileMetadataView fileMetadataView,
+ final boolean readOnly, final Quorum<?, ?> quorum) {
m_metaBitsSize = cDefaultMetaBitsSize;
@@ -407,12 +410,13 @@
m_rb = m_fmv.getRootBlock();
- m_filename = m_fd.getAbsolutePath();
+// m_filename = m_fd.getAbsolutePath();
- m_commitList = new ArrayList();
+ m_commitList = new ArrayList<Allocator>();
+
m_allocs = new ArrayList<Allocator>();
- m_freeBlobs = new ArrayList();
+ m_freeBlobs = new ArrayList<BlobAllocator>();
try {
m_reopener = new ReopenFileChannel(m_fd, m_raf, "rw");
@@ -426,13 +430,14 @@
m_bufferedWrite = null;
}
- int buffers = m_fmv.getFileMetadata().writeCacheBufferCount;
- log.warn("RWStore using writeCacheService with buffers: " + buffers);
+ final int buffers = m_fmv.getFileMetadata().writeCacheBufferCount;
+ if(log.isInfoEnabled())
+ log.info("RWStore using writeCacheService with buffers: " + buffers);
+
try {
- m_writeCache = new RWWriteCacheService(
- buffers, m_raf
- .length(), m_reopener, m_quorum) {
+ m_writeCache = new RWWriteCacheService(buffers, m_raf.length(),
+ m_reopener, m_quorum) {
public WriteCache newWriteCache(final ByteBuffer buf,
final boolean useChecksum,
@@ -541,24 +546,23 @@
final long metaAddr = rbv.getMetaStartAddr();
final long rawMetaBitsAddr = rbv.getMetaBitsAddr();
if (metaAddr == 0 || rawMetaBitsAddr == 0) {
- log.warn("No meta allocation data included in root block for RWStore"); // possible
- // when
- // rolling
- // back
- // to
- // empty
- // file
+ /*
+ * possible when rolling back to empty file.
+ */
+ log.warn("No meta allocation data included in root block for RWStore");
}
if (log.isTraceEnabled()) {
- int commitRecordAddr = (int) (rbv.getCommitRecordAddr() >> 32);
- log.trace("CommitRecord " + rbv.getCommitRecordAddr() + " at physical address: " + physicalAddress(commitRecordAddr));
- }
+ final int commitRecordAddr = (int) (rbv.getCommitRecordAddr() >> 32);
+ log.trace("CommitRecord " + rbv.getCommitRecordAddr()
+ + " at physical address: "
+ + physicalAddress(commitRecordAddr));
+ }
final long commitCounter = rbv.getCommitCounter();
- final int metaStartAddr = (int) -(metaAddr >> 32); // void
- final int fileSize = (int) -(metaAddr & 0xFFFFFFFF);
+// final int metaStartAddr = (int) -(metaAddr >> 32); // void
+// final int fileSize = (int) -(metaAddr & 0xFFFFFFFF);
if (log.isTraceEnabled())
log.trace("m_allocation: " + nxtalloc + ", m_metaBitsAddr: "
@@ -592,7 +596,7 @@
// m_rb = m_fmv.getRootBlock();
assert(m_rb != null);
- m_commitCounter = m_rb.getCommitCounter();
+// m_commitCounter = m_rb.getCommitCounter();
final long nxtOffset = m_rb.getNextOffset();
m_nextAllocation = -(int) (nxtOffset >> 32);
@@ -607,10 +611,12 @@
m_fileSize = (int) -(metaAddr & 0xFFFFFFFF);
long rawmbaddr = m_rb.getMetaBitsAddr();
- int metaBitsStore = (int) (rawmbaddr & 0xFFFF); // take bottom 16 bits (
- // even 1K of metabits
- // is more than
- // sufficient)
+
+ /*
+ * Take bottom 16 bits (even 1K of metabits is more than sufficient)
+ */
+ final int metaBitsStore = (int) (rawmbaddr & 0xFFFF);
+
if (metaBitsStore > 0) {
rawmbaddr >>= 16;
@@ -623,7 +629,7 @@
m_lastDeferredReleaseTime = strBuf.readLong();
- int allocBlocks = strBuf.readInt();
+ final int allocBlocks = strBuf.readInt();
m_allocSizes = new int[allocBlocks];
for (int i = 0; i < allocBlocks; i++) {
m_allocSizes[i] = strBuf.readInt();
@@ -643,7 +649,7 @@
m_freeFixed = new ArrayList[numFixed];
for (int i = 0; i < numFixed; i++) {
- m_freeFixed[i] = new ArrayList();
+ m_freeFixed[i] = new ArrayList<FixedAllocator>();
}
checkCoreAllocations();
@@ -669,58 +675,58 @@
+ ", " + m_metaBitsAddr);
}
- /*
- * Called when store is opened to make sure any deferred frees are
- * cleared.
- *
- * Stored persistently is only the list of addresses of blocks to be freed,
- * the knowledge of the txn release time does not need to be held persistently,
- * this is only relevant for transient state while the RWStore is open.
- *
- * The deferredCount is the number of entries - integer address and integer
- * count at each address
- */
- private void clearOutstandingDeferrels(final int deferredAddr, final int deferredCount) {
- if (deferredAddr != 0) {
- assert deferredCount != 0;
- final int sze = deferredCount * 8 + 4; // include space for checksum
-
- if (log.isDebugEnabled())
- log.debug("Clearing Outstanding Deferrals: " + deferredCount);
-
- byte[] buf = new byte[sze];
- getData(deferredAddr, buf);
-
- final byte[] blockBuf = new byte[8 * 1024]; // maximum size required
-
- ByteBuffer in = ByteBuffer.wrap(buf);
- for (int i = 0; i < deferredCount; i++) {
- int blockAddr = in.getInt();
- int addrCount = in.getInt();
-
- // now read in this block and free all addresses referenced
- getData(blockAddr, blockBuf, 0, addrCount*4 + 4);
- ByteBuffer inblock = ByteBuffer.wrap(blockBuf);
- for (int b = 0; b < addrCount; b++) {
- final int defAddr = inblock.getInt();
- Allocator alloc = getBlock(defAddr);
- if (alloc instanceof BlobAllocator) {
- b++;
- assert b < addrCount;
- alloc.free(defAddr, inblock.getInt());
- } else {
- alloc.free(defAddr, 0); // size ignored for FreeAllocators
- }
- }
- // once read then free the block allocation
- free(blockAddr, 0);
- }
-
- // lastly free the deferredAddr
- free(deferredAddr, 0);
- }
-
- }
+// /*
+// * Called when store is opened to make sure any deferred frees are
+// * cleared.
+// *
+// * Stored persistently is only the list of addresses of blocks to be freed,
+// * the knowledge of the txn release time does not need to be held persistently,
+// * this is only relevant for transient state while the RWStore is open.
+// *
+// * The deferredCount is the number of entries - integer address and integer
+// * count at each address
+// */
+// private void clearOutstandingDeferrels(final int deferredAddr, final int deferredCount) {
+// if (deferredAddr != 0) {
+// assert deferredCount != 0;
+// final int sze = deferredCount * 8 + 4; // include space for checksum
+//
+// if (log.isDebugEnabled())
+// log.debug("Clearing Outstanding Deferrals: " + deferredCount);
+//
+// byte[] buf = new byte[sze];
+// getData(deferredAddr, buf);
+//
+// final byte[] blockBuf = new byte[8 * 1024]; // maximum size required
+//
+// ByteBuffer in = ByteBuffer.wrap(buf);
+// for (int i = 0; i < deferredCount; i++) {
+// int blockAddr = in.getInt();
+// int addrCount = in.getInt();
+//
+// // now read in this block and free all addresses referenced
+// getData(blockAddr, blockBuf, 0, addrCount*4 + 4);
+// ByteBuffer inblock = ByteBuffer.wrap(blockBuf);
+// for (int b = 0; b < addrCount; b++) {
+// final int defAddr = inblock.getInt();
+// Allocator alloc = getBlock(defAddr);
+// if (alloc instanceof BlobAllocator) {
+// b++;
+// assert b < addrCount;
+// alloc.free(defAddr, inblock.getInt());
+// } else {
+// alloc.free(defAddr, 0); // size ignored for FreeAllocators
+// }
+// }
+// // once read then free the block allocation
+// free(blockAddr, 0);
+// }
+//
+// // lastly free the deferredAddr
+// free(deferredAddr, 0);
+// }
+//
+// }
/*********************************************************************
* make sure resource is closed!
@@ -733,8 +739,9 @@
assert m_allocs.size() == 0;
- System.out.println("readAllocationBlocks, m_metaBits.length: "
- + m_metaBits.length);
+ if (log.isInfoEnabled())
+ log.info("readAllocationBlocks, m_metaBits.length: "
+ + m_metaBits.length);
/**
* Allocators are sorted in StartAddress order (which MUST be the order
@@ -744,15 +751,15 @@
* address (must be two version of same Allocator).
*
* Meta-Allocations stored as {int address; int[8] bits}, so each block
- * holds 8*32=256 allocation slots of 1K totalling 256K.
+ * holds 8*32=256 allocation slots of 1K totaling 256K.
*/
for (int b = 0; b < m_metaBits.length; b += 9) {
- long blockStart = convertAddr(m_metaBits[b]);
- int startBit = (b * 32) + 32;
- int endBit = startBit + (8*32);
+ final long blockStart = convertAddr(m_metaBits[b]);
+ final int startBit = (b * 32) + 32;
+ final int endBit = startBit + (8*32);
for (int i = startBit; i < endBit; i++) {
if (tstBit(m_metaBits, i)) {
- long addr = blockStart + ((i-startBit) * ALLOC_BLOCK_SIZE);
+ final long addr = blockStart + ((i-startBit) * ALLOC_BLOCK_SIZE);
final byte buf[] = new byte[ALLOC_BLOCK_SIZE];
@@ -761,8 +768,8 @@
final DataInputStream strBuf = new DataInputStream(new ByteArrayInputStream(buf));
final int allocSize = strBuf.readInt(); // if Blob < 0
- Allocator allocator = null;
- ArrayList freeList = null;
+ final Allocator allocator;
+ final ArrayList<? extends Allocator> freeList;
if (allocSize > 0) {
int index = 0;
int fixedSize = m_minFixedAlloc;
@@ -811,16 +818,20 @@
* ContextAllocation is released, its allocators will be added to the
* global free lists.
*
- * @param block - the index of the Fixed size alloction
+ * @param block - the index of the Fixed size allocation
* @return the FixedAllocator
*/
- FixedAllocator establishFreeFixedAllocator(int block) {
- ArrayList<FixedAllocator> list = m_freeFixed[block];
+ private FixedAllocator establishFreeFixedAllocator(final int block) {
+
+ final ArrayList<FixedAllocator> list = m_freeFixed[block];
if (list.size() == 0) {
- final int allocSize = 64 * m_allocSizes[block];
+
+ final int allocSize = 64 * m_allocSizes[block];
- FixedAllocator allocator = new FixedAllocator(this, allocSize, m_writeCache);
+ final FixedAllocator allocator = new FixedAllocator(this,
+ allocSize, m_writeCache);
+
allocator.setIndex(m_allocs.size());
m_allocs.add(allocator);
@@ -855,7 +866,7 @@
// }
public long getMaxFileSize() {
- long maxSize = m_maxFileSize;
+ final long maxSize = m_maxFileSize;
return maxSize << 8;
}
@@ -914,12 +925,14 @@
* If it is a BlobAllocation, then the BlobAllocation address points to the address of the BlobHeader
* record.
*/
- public void getData(long addr, byte buf[]) {
+ public void getData(final long addr, final byte buf[]) {
getData(addr, buf, 0, buf.length);
}
- public void getData(long addr, byte buf[], int offset, int length) {
- if (addr == 0) {
+ public void getData(final long addr, final byte buf[], final int offset,
+ final int length) {
+
+ if (addr == 0) {
return;
}
@@ -931,19 +944,24 @@
// length includes space for the checksum
if (length > m_maxFixedAlloc) {
try {
- int alloc = m_maxFixedAlloc-4;
- int nblocks = (alloc - 1 + (length-4))/alloc;
- if (nblocks < 0) throw new IllegalStateException("Allocation error, m_maxFixedAlloc: "+ m_maxFixedAlloc);
-
- byte[] hdrbuf = new byte[4 * (nblocks + 1) + 4]; // plus 4 bytes for checksum
- BlobAllocator ba = (BlobAllocator) getBlock((int) addr);
+ final int alloc = m_maxFixedAlloc-4;
+ final int nblocks = (alloc - 1 + (length-4))/alloc;
+ if (nblocks < 0)
+ throw new IllegalStateException(
+ "Allocation error, m_maxFixedAlloc: "
+ + m_maxFixedAlloc);
+
+ final byte[] hdrbuf = new byte[4 * (nblocks + 1) + 4]; // plus 4 bytes for checksum
+ final BlobAllocator ba = (BlobAllocator) getBlock((int) addr);
getData(ba.getBlobHdrAddress(getOffset((int) addr)), hdrbuf); // read in header
- DataInputStream hdrstr = new DataInputStream(new ByteArrayInputStream(hdrbuf));
- int rhdrs = hdrstr.readInt();
- if (rhdrs != nblocks) {
- throw new IllegalStateException("Incompatible BLOB header record, expected: " + nblocks + ", got: " + rhdrs);
- }
- int[] blobHdr = new int[nblocks];
+ final DataInputStream hdrstr = new DataInputStream(new ByteArrayInputStream(hdrbuf));
+ final int rhdrs = hdrstr.readInt();
+ if (rhdrs != nblocks) {
+ throw new IllegalStateException(
+ "Incompatible BLOB header record, expected: "
+ + nblocks + ", got: " + rhdrs);
+ }
+ final int[] blobHdr = new int[nblocks];
for (int i = 0; i < nblocks; i++) {
blobHdr[i] = hdrstr.readInt();
}
@@ -968,7 +986,7 @@
}
try {
- long paddr = physicalAddress((int) addr);
+ final long paddr = physicalAddress((int) addr);
if (paddr == 0) {
assertAllocators();
@@ -984,19 +1002,25 @@
* value, so the cached data is 4 bytes less than the
* buffer size.
*/
- ByteBuffer bbuf = null;
+ final ByteBuffer bbuf;
try {
bbuf = m_writeCache.read(paddr);
} catch (Throwable t) {
- throw new IllegalStateException("Error reading from WriteCache addr: " + paddr + " length: " + (length-4) + ", writeCacheDebug: " + m_writeCache.addrDebugInfo(paddr), t);
+ throw new IllegalStateException(
+ "Error reading from WriteCache addr: " + paddr
+ + " length: " + (length - 4)
+ + ", writeCacheDebug: "
+ + m_writeCache.addrDebugInfo(paddr), t);
}
if (bbuf != null) {
- byte[] in = bbuf.array(); // reads in with checksum - no need to check if in cache
+ final byte[] in = bbuf.array(); // reads in with checksum - no need to check if in cache
if (in.length != length-4) {
assertAllocators();
-
- throw new IllegalStateException("Incompatible buffer size for addr: " + paddr + ", " + in.length
- + " != " + (length-4) + " writeCacheDebug: " + m_writeCache.addrDebugInfo(paddr));
+ throw new IllegalStateException(
+ "Incompatible buffer size for addr: " + paddr
+ + ", " + in.length + " != "
+ + (length - 4) + " writeCacheDebug: "
+ + m_writeCache.addrDebugInfo(paddr));
}
for (int i = 0; i < length-4; i++) {
buf[offset+i] = in[i];
@@ -1004,21 +1028,23 @@
m_cacheReads++;
} else {
// If checksum is required then the buffer should be sized to include checksum in final 4 bytes
- ByteBuffer bb = ByteBuffer.wrap(buf, offset, length);
+ final ByteBuffer bb = ByteBuffer.wrap(buf, offset, length);
FileChannelUtility.readAll(m_reopener, bb, paddr);
- int chk = ChecksumUtility.getCHK().checksum(buf, offset, length-4); // read checksum
- int tstchk = bb.getInt(offset + length-4);
+ final int chk = ChecksumUtility.getCHK().checksum(buf, offset, length-4); // read checksum
+ final int tstchk = bb.getInt(offset + length-4);
if (chk != tstchk) {
assertAllocators();
- String cacheDebugInfo = m_writeCache.addrDebugInfo(paddr);
+ final String cacheDebugInfo = m_writeCache.addrDebugInfo(paddr);
log.warn("Invalid data checksum for addr: " + paddr
+ ", chk: " + chk + ", tstchk: " + tstchk + ", length: " + length
+ ", first bytes: " + toHexString(buf, 32) + ", successful reads: " + m_diskReads
+ ", at last extend: " + m_readsAtExtend + ", cacheReads: " + m_cacheReads
+ ", writeCacheDebug: " + cacheDebugInfo);
- throw new IllegalStateException("Invalid data checksum from address: " + paddr + ", size: " + (length-4));
+ throw new IllegalStateException(
+ "Invalid data checksum from address: " + paddr
+ + ", size: " + (length - 4));
}
m_diskReads++;
@@ -1042,7 +1068,7 @@
}
}
- static final char[] HEX_CHAR_TABLE = {
+ static private final char[] HEX_CHAR_TABLE = {
'0', '1','2','3',
'4','5','6','7',
'8','9','a','b',
@@ -1050,11 +1076,11 @@
};
// utility to display byte array of maximum i bytes as hexString
- private String toHexString(byte[] buf, int n) {
+ static private String toHexString(final byte[] buf, int n) {
n = n < buf.length ? n : buf.length;
- StringBuffer out = new StringBuffer();
+ final StringBuffer out = new StringBuffer();
for (int i = 0; i < n; i++) {
- int v = buf[i] & 0xFF;
+ final int v = buf[i] & 0xFF;
out.append(HEX_CHAR_TABLE[v >>> 4]);
out.append(HEX_CHAR_TABLE[v &0xF]);
}
@@ -1094,7 +1120,7 @@
* this supports the core functionality of a WormStore, other stores should
* return zero, indicating no previous versions available
**/
- public long getPreviousAddress(long laddr) {
+ public long getPreviousAddress(final long laddr) {
return 0;
}
@@ -1135,7 +1161,7 @@
* is NOT owned, BUT there are active AllocationContexts, in this
* situation, the free must ALWAYS be deferred.
*/
- boolean alwaysDefer = context == null && m_contexts.size() > 0;
+ final boolean alwaysDefer = context == null && m_contexts.size() > 0;
if (alwaysDefer)
if (log.isDebugEnabled())
log.debug("Should defer " + physicalAddress(addr));
@@ -1150,7 +1176,7 @@
}
- private long immediateFreeCount = 0;
+// private long immediateFreeCount = 0;
private void immediateFree(final int addr, final int sze) {
switch (addr) {
@@ -1215,10 +1241,10 @@
* by the finer granularity of the AllocBlocks within a FixedAllocator.
*/
- private volatile long m_maxAllocation = 0;
+// private volatile long m_maxAllocation = 0;
private volatile long m_spareAllocation = 0;
- public int alloc(final int size, IAllocationContext context) {
+ public int alloc(final int size, final IAllocationContext context) {
if (size > m_maxFixedAlloc) {
throw new IllegalArgumentException("Allocation size to big: " + size);
}
@@ -1226,8 +1252,7 @@
m_allocationLock.lock();
try {
try {
- ArrayList list;
- Allocator allocator = null;
+ final Allocator allocator;
final int i = fixedAllocatorIndex(size);
if (context != null) {
allocator = establishContextAllocation(context).getFreeFixed(i);
@@ -1235,7 +1260,7 @@
final int block = 64 * m_allocSizes[i];
m_spareAllocation += (block - size); // Isn't adjusted by frees!
- list = m_freeFixed[i];
+ final ArrayList<FixedAllocator> list = m_freeFixed[i];
if (list.size() == 0) {
allocator = new FixedAllocator(this, block, m_writeCache);
@@ -1250,9 +1275,9 @@
// Verify free list only has allocators with free bits
if (log.isDebugEnabled()){
int tsti = 0;
- Iterator<Allocator> allocs = list.iterator();
+ final Iterator<FixedAllocator> allocs = list.iterator();
while (allocs.hasNext()) {
- Allocator tstAlloc = allocs.next();
+ final Allocator tstAlloc = allocs.next();
if (!tstAlloc.hasFree()) {
throw new IllegalStateException("Free list contains full allocator, " + tsti + " of " + list.size());
}
@@ -1272,10 +1297,11 @@
m_recentAlloc = true;
- long pa = physicalAddress(addr);
- if (pa == 0L) {
- throw new IllegalStateException("No physical address found for " + addr);
- }
+ final long pa = physicalAddress(addr);
+ if (pa == 0L) {
+ throw new IllegalStateException(
+ "No physical address found for " + addr);
+ }
m_allocations++;
m_nativeAllocBytes += size;
@@ -1291,7 +1317,7 @@
}
}
- int fixedAllocatorIndex(int size) {
+ int fixedAllocatorIndex(final int size) {
int i = 0;
int cmp = m_minFixedAlloc;
@@ -1530,12 +1556,24 @@
* simply reset the metaBitsAddr
* @throws IOException
*/
- protected void writeFileSpec() throws IOException {
- m_rb = m_fmv.newRootBlockView(!m_rb.isRootBlock0(), m_rb.getOffsetBits(), getNextOffset(), m_rb
- .getFirstCommitTime(), m_rb.getLastCommitTime(), m_rb.getCommitCounter(), m_rb.getCommitRecordAddr(),
- m_rb.getCommitRecordIndexAddr(), getMetaStartAddr(), getMetaBitsAddr(), m_rb.getLastCommitTime());
-
- m_fmv.getFileMetadata().writeRootBlock(m_rb, ForceEnum.Force);
+ protected void writeFileSpec() throws IOException {
+
+ m_rb = m_fmv.newRootBlockView(//
+ !m_rb.isRootBlock0(), //
+ m_rb.getOffsetBits(), //
+ getNextOffset(), //
+ m_rb.getFirstCommitTime(),//
+ m_rb.getLastCommitTime(), //
+ m_rb.getCommitCounter(), //
+ m_rb.getCommitRecordAddr(),//
+ m_rb.getCommitRecordIndexAddr(), //
+ getMetaStartAddr(),//
+ getMetaBitsAddr(), //
+ m_rb.getLastCommitTime()//
+ );
+
+ m_fmv.getFileMetadata().writeRootBlock(m_rb, ForceEnum.Force);
+
}
// float m_vers = 0.0f;
@@ -1582,8 +1620,8 @@
checkDeferredFrees(true, journal); // free now if possible
// Allocate storage for metaBits
- long oldMetaBits = m_metaBitsAddr;
- int oldMetaBitsSize = (m_metaBits.length + m_allocSizes.length + 1) * 4;
+ final long oldMetaBits = m_metaBitsAddr;
+ final int oldMetaBitsSize = (m_metaBits.length + m_allocSizes.length + 1) * 4;
m_metaBitsAddr = alloc(getRequiredMetaBitsStorage(), null);
// DEBUG SANITY CHECK!
@@ -1596,18 +1634,18 @@
immediateFree((int) oldMetaBits, oldMetaBitsSize);
// save allocation headers
- Iterator iter = m_commitList.iterator();
+ final Iterator<Allocator> iter = m_commitList.iterator();
while (iter.hasNext()) {
- final Allocator allocator = (Allocator) iter.next();
+ final Allocator allocator = iter.next();
final int old = allocator.getDiskAddr();
metaFree(old);
final int naddr = metaAlloc();
allocator.setDiskAddr(naddr);
- if (log.isTraceEnabled())
- log.trace("Update allocator " + allocator.getIndex() + ", old addr: " + old + ", new addr: "
- + naddr);
+ if (log.isTraceEnabled())
+ log.trace("Update allocator " + allocator.getIndex()
+ + ", old addr: " + old + ", new addr: " + naddr);
try {
m_writeCache.write(metaBit2Addr(naddr), ByteBuffer.wrap(allocator.write()), 0, false); // do not use checksum
@@ -1647,16 +1685,21 @@
checkCoreAllocations();
- if (log.isTraceEnabled())
- log.trace("commitChanges for: " + m_nextAllocation + ", " + m_metaBitsAddr + ", active contexts: " + m_contexts.size());
- }
+ if (log.isTraceEnabled())
+ log.trace("commitChanges for: " + m_nextAllocation + ", "
+ + m_metaBitsAddr + ", active contexts: "
+ + m_contexts.size());
+ }
- /**
- * Called prior to commit, so check whether storage can be freed and then
- * whether the deferredheader needs to be saved.
- * <p>
- * Note: The caller MUST be holding the {@link #m_allocationLock}.
- */
+ /**
+ * Called prior to commit, so check whether storage can be freed and then
+ * whether the deferredheader needs to be saved.
+ * <p>
+ * Note: The caller MUST be holding the {@link #m_allocationLock}.
+ * <p>
+ * Note: This method is package private in order to expose it to the unit
+ * tests....
[truncated message content] |
|
From: <tho...@us...> - 2010-11-02 17:46:40
|
Revision: 3872
http://bigdata.svn.sourceforge.net/bigdata/?rev=3872&view=rev
Author: thompsonbry
Date: 2010-11-02 17:46:33 +0000 (Tue, 02 Nov 2010)
Log Message:
-----------
Removed an unused test for the JiniFederation QueryEngine and replaced it with a reference to the JiniFederationSparqlTest.
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata-jini/src/test/com/bigdata/bop/fed/jini/README.txt
Removed Paths:
-------------
branches/QUADS_QUERY_BRANCH/bigdata-jini/src/test/com/bigdata/bop/fed/jini/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata-jini/src/test/com/bigdata/bop/fed/jini/TestJiniFederatedQueryEngine.java
Added: branches/QUADS_QUERY_BRANCH/bigdata-jini/src/test/com/bigdata/bop/fed/jini/README.txt
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-jini/src/test/com/bigdata/bop/fed/jini/README.txt (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata-jini/src/test/com/bigdata/bop/fed/jini/README.txt 2010-11-02 17:46:33 UTC (rev 3872)
@@ -0,0 +1 @@
+Use the BigdataFederationSparqlTest to test for SPARQL compliance against a running federation.
\ No newline at end of file
Deleted: branches/QUADS_QUERY_BRANCH/bigdata-jini/src/test/com/bigdata/bop/fed/jini/TestAll.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-jini/src/test/com/bigdata/bop/fed/jini/TestAll.java 2010-11-02 17:42:25 UTC (rev 3871)
+++ branches/QUADS_QUERY_BRANCH/bigdata-jini/src/test/com/bigdata/bop/fed/jini/TestAll.java 2010-11-02 17:46:33 UTC (rev 3872)
@@ -1,76 +0,0 @@
-/**
-
-Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
-package com.bigdata.bop.fed.jini;
-
-
-import junit.framework.Test;
-import junit.framework.TestCase;
-import junit.framework.TestSuite;
-
-/**
- * Aggregates test suites into increasing dependency order.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
- */
-public class TestAll extends TestCase {
-
- /**
- *
- */
- public TestAll() {
-
- }
-
- /**
- * @param arg0
- */
- public TestAll(String arg0) {
-
- super(arg0);
-
- }
-
- /**
- * Returns a test that will run each of the implementation specific test
- * suites in turn.
- */
- public static Test suite()
- {
-
- final TestSuite suite = new TestSuite("federated query");
-
- /*
- * Unit tests for the federated query engine.
- *
- * Note: This test suite has a dependency on JiniClient and must be
- * executed against an already running federation.
- */
- suite.addTestSuite(TestJiniFederatedQueryEngine.class);
-
- return suite;
-
- }
-
-}
Deleted: branches/QUADS_QUERY_BRANCH/bigdata-jini/src/test/com/bigdata/bop/fed/jini/TestJiniFederatedQueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-jini/src/test/com/bigdata/bop/fed/jini/TestJiniFederatedQueryEngine.java 2010-11-02 17:42:25 UTC (rev 3871)
+++ branches/QUADS_QUERY_BRANCH/bigdata-jini/src/test/com/bigdata/bop/fed/jini/TestJiniFederatedQueryEngine.java 2010-11-02 17:46:33 UTC (rev 3872)
@@ -1,811 +0,0 @@
-/**
-
-Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
-/*
- * Created on Sep 5, 2010
- */
-
-package com.bigdata.bop.fed.jini;
-
-import java.io.IOException;
-import java.util.Map;
-import java.util.Properties;
-import java.util.UUID;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.TimeoutException;
-
-import junit.framework.TestCase2;
-
-import com.bigdata.bop.BOp;
-import com.bigdata.bop.BOpContext;
-import com.bigdata.bop.BOpEvaluationContext;
-import com.bigdata.bop.Constant;
-import com.bigdata.bop.IBindingSet;
-import com.bigdata.bop.IConstant;
-import com.bigdata.bop.IVariable;
-import com.bigdata.bop.IVariableOrConstant;
-import com.bigdata.bop.NV;
-import com.bigdata.bop.PipelineOp;
-import com.bigdata.bop.Var;
-import com.bigdata.bop.ap.E;
-import com.bigdata.bop.ap.Predicate;
-import com.bigdata.bop.ap.R;
-import com.bigdata.bop.bindingSet.ArrayBindingSet;
-import com.bigdata.bop.bindingSet.HashBindingSet;
-import com.bigdata.bop.bset.StartOp;
-import com.bigdata.bop.engine.BOpStats;
-import com.bigdata.bop.engine.IChunkMessage;
-import com.bigdata.bop.engine.LocalChunkMessage;
-import com.bigdata.bop.engine.PipelineDelayOp;
-import com.bigdata.bop.engine.QueryEngine;
-import com.bigdata.bop.engine.RunningQuery;
-import com.bigdata.bop.engine.TestQueryEngine;
-import com.bigdata.bop.fed.FederatedQueryEngine;
-import com.bigdata.bop.fed.QueryEngineFactory;
-import com.bigdata.bop.join.PipelineJoin;
-import com.bigdata.bop.solutions.SliceOp;
-import com.bigdata.bop.solutions.SortOp;
-import com.bigdata.btree.keys.KeyBuilder;
-import com.bigdata.journal.ITx;
-import com.bigdata.relation.accesspath.IAsynchronousIterator;
-import com.bigdata.relation.accesspath.ThickAsynchronousIterator;
-import com.bigdata.service.DataService;
-import com.bigdata.service.IBigdataFederation;
-import com.bigdata.service.IDataService;
-import com.bigdata.service.jini.JiniClient;
-import com.bigdata.service.jini.JiniFederation;
-import com.bigdata.striterator.ChunkedArrayIterator;
-import com.bigdata.striterator.Dechunkerator;
-import com.ibm.icu.impl.ByteBuffer;
-
-/**
- * Unit tests for {@link FederatedQueryEngine} running against a
- * {@link JiniFederation} with 2 {@link DataService}s.
- * <p>
- * Note: Distributed query processing generally means that the order in which
- * the chunks arrive is non-deterministic. Therefore the order of the solutions
- * can not be checked unless a total order is placed on the solutions using a
- * {@link SortOp}.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id: TestFederatedQueryEngine.java 3508 2010-09-05 17:02:34Z
- * thompsonbry $
- *
- * @todo test distributed execution of potentially distributed operations
- * including: JOIN (selective and unselective), DISTINCT access path,
- * DISTINCT solutions, GROUP BY, ORDER BY.
- *
- * @todo Write unit tests for distributed pipeline query with an eye towards
- * verification of integration of the {@link QueryEngine} with the
- * protocol used to move data around among the nodes.
- * <p>
- * Each of the operators which has specialized execution against a
- * federation should be tested as well. This includes:
- * <p>
- * distributed merge sort (assuming that there is a requirement to
- * interchange buffers as part of the distributed sort).
- * <p>
- * distributed hash tables used in a distributed distinct filter on an
- * access path (for scale-out default graph queries in quads mode).
- * <p>
- * ...
- */
-public class TestJiniFederatedQueryEngine extends TestCase2 {
-
- public TestJiniFederatedQueryEngine() {
-
- }
-
- public TestJiniFederatedQueryEngine(String name) {
-
- super(name);
-
- }
-
- // Namespace for the relation.
- static private final String namespace = TestJiniFederatedQueryEngine.class.getName();
-
- // The separator key between the index partitions.
- private byte[] separatorKey;
-
- private JiniClient<?> client;
-
-// /** The local persistence store for the {@link #queryEngine}. */
-// private Journal queryEngineStore;
-//
-// /** The local {@link ResourceService} for the {@link #queryEngine}. */
-// private ManagedResourceService queryEngineResourceService;
-
- /** The query controller. */
- private FederatedQueryEngine queryEngine;
-
- private IDataService dataService0;
-
- private IDataService dataService1;
-
- protected void setUp() throws Exception {
-
- /*
- * FIXME This is hardcoded to a specific location in the file system.
- *
- * Also, the dependency on JiniClient means that we must move this test
- * class into the bigdata-jini package.
- */
- client = new JiniClient(
- new String[] { "/nas/bigdata/bigdata-0.83.2/dist/bigdata/var/config/jini/bigdataStandalone.config" });
-
- final IBigdataFederation<?> fed = client.connect();
-
-// // create index manager for the query controller.
-// {
-// final Properties p = new Properties();
-// p.setProperty(Journal.Options.BUFFER_MODE, BufferMode.Transient
-// .toString());
-// queryEngineStore = new Journal(p);
-// }
-//
-// // create resource service for the query controller.
-// {
-// queryEngineResourceService = new ManagedResourceService(
-// new InetSocketAddress(InetAddress
-// .getByName(NicUtil.getIpAddress("default.nic",
-// "default", true/* loopbackOk */)), 0/* port */
-// ), 0/* requestServicePoolSize */) {
-//
-// @Override
-// protected File getResource(UUID uuid) throws Exception {
-// // Will not serve up files.
-// return null;
-// }
-// };
-// }
-//
-// // create the query controller.
-// {
-//
-// queryEngine = new FederatedQueryEngine(fed.getServiceUUID(), fed,
-// queryEngineStore, queryEngineResourceService);
-//
-// queryEngine.init();
-//
-// }
-
- queryEngine = QueryEngineFactory.getFederatedQueryController(fed);
-
- /*
- * Discover the data services. We need their UUIDs in order to create
- * the test relation split across an index partition located on each of
- * the two data services.
- */
- final int maxCount = 2;
- UUID[] dataServices = null;
- final long begin = System.currentTimeMillis();
- long elapsed = 0L;
- while ((dataServices = fed.getDataServiceUUIDs(maxCount)).length < maxCount
- && ((elapsed = System.currentTimeMillis() - begin) < TimeUnit.SECONDS
- .toMillis(60))) {
- System.err.println("Waiting for " + maxCount
- + " data services. There are " + dataServices.length
- + " discovered : elapsed=" + elapsed + "ms");
- Thread.sleep(250/* ms */);
- }
-
- if (dataServices.length < maxCount)
- throw new TimeoutException("Discovered " + dataServices.length
- + " data services in " + elapsed + "ms but require "
- + maxCount);
-
- super.setUp();
-
- dataService0 = fed.getDataService(dataServices[0]);
- dataService1 = fed.getDataService(dataServices[1]);
-// {
-//
-// // @todo need to wait for the dataService to be running.
-//// assertTrue(((DataService) dataServer.getProxy())
-//// .getResourceManager().awaitRunning());
-//
-// // resolve the query engine on one of the data services.
-// while ((queryEngine = (IQueryClient) dataService0.getQueryEngine()) == null) {
-//
-// if (log.isInfoEnabled())
-// log.info("Waiting for query engine on dataService0");
-//
-// Thread.sleep(250);
-//
-// }
-//
-// System.err.println("controller: " + queryEngine);
-//
-// }
-//
-// // resolve the query engine on the other data services.
-// {
-//
-// IQueryPeer other = null;
-//
-//// assertTrue(((DataService) dataServer.getProxy())
-//// .getResourceManager().awaitRunning());
-//
-// while ((other = dataService1.getQueryEngine()) == null) {
-//
-// if (log.isInfoEnabled())
-// log.info("Waiting for query engine on dataService1");
-//
-// Thread.sleep(250);
-//
-// }
-//
-// System.err.println("other : " + other);
-//
-// }
-
- loadData();
-
- }
-
- public void tearDown() throws Exception {
-
- // clear reference.
- separatorKey = null;
-
- client.disconnect(true/*immediateShutdown*/);
- client = null;
-
- dataService0 = null;
- dataService1 = null;
-
-// if (queryEngineResourceService != null) {
-// queryEngineResourceService.shutdownNow();
-// queryEngineResourceService = null;
-// }
-// if (queryEngineStore != null) {
-// queryEngineStore.destroy();
-// queryEngineStore = null;
-// }
- if (queryEngine != null) {
- queryEngine.shutdownNow();
- queryEngine = null;
- }
-
- super.tearDown();
-
- }
-
- /**
- * Create and populate relation in the {@link #namespace}.
- *
- * @throws IOException
- */
- private void loadData() throws IOException {
-
- /*
- * The data to insert (in sorted order this time).
- */
- final E[] a = {//
- // partition0
- new E("John", "Mary"),//
- new E("Leon", "Paul"),//
- // partition1
- new E("Mary", "John"),//
- new E("Mary", "Paul"),//
- new E("Paul", "Leon"),//
- };
-
- // The separator key between the two index partitions.
- separatorKey = KeyBuilder.newUnicodeInstance().append("Mary").getKey();
-
- final byte[][] separatorKeys = new byte[][] {//
- new byte[] {}, //
- separatorKey //
- };
-
- final UUID[] dataServices = new UUID[] {//
- dataService0.getServiceUUID(),//
- dataService1.getServiceUUID(),//
- };
-
- /*
- * Create the relation with the primary index key-range partitioned
- * using the given separator keys and data services.
- */
-
- final R rel = new R(client.getFederation(), namespace, ITx.UNISOLATED,
- new Properties());
-
- if (client.getFederation().getResourceLocator().locate(namespace,
- ITx.UNISOLATED) == null) {
-
- rel.create(separatorKeys, dataServices);
-
- /*
- * Insert data into the appropriate index partitions.
- */
- rel
- .insert(new ChunkedArrayIterator<E>(a.length, a, null/* keyOrder */));
-
- }
-
- }
-
- /**
- * Return an {@link IAsynchronousIterator} that will read a single,
- * empty {@link IBindingSet}.
- *
- * @param bindingSet
- * the binding set.
- */
- protected ThickAsynchronousIterator<IBindingSet[]> newBindingSetIterator(
- final IBindingSet bindingSet) {
-
- return new ThickAsynchronousIterator<IBindingSet[]>(
- new IBindingSet[][] { new IBindingSet[] { bindingSet } });
-
- }
-
- /**
- * Starts and stops the {@link QueryEngine}, but does not validate the
- * semantics of shutdown() versus shutdownNow() since we need to be
- * evaluating query mixes in order to verify the semantics of those
- * operations.
- *
- * @throws Exception
- */
- public void test_startStop() throws Exception {
-
- // NOP
-
- }
-
- /**
- * Test the ability to run a query which does nothing and produces no
- * solutions.
- *
- * @throws Exception
- */
- public void test_query_startRun() throws Exception {
-
- final int startId = 1;
- final PipelineOp query = new StartOp(new BOp[] {}, NV
- .asMap(new NV[] {//
- new NV(Predicate.Annotations.BOP_ID, startId),//
- new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.CONTROLLER),//
- }));
-
- final UUID queryId = UUID.randomUUID();
- final RunningQuery runningQuery = queryEngine.eval(queryId, query,
- new LocalChunkMessage<IBindingSet>(queryEngine, queryId,
- startId, -1 /* partitionId */,
- newBindingSetIterator(new HashBindingSet())));
-
- // Wait until the query is done.
- runningQuery.get();
- final Map<Integer, BOpStats> statsMap = runningQuery.getStats();
- {
- // validate the stats map.
- assertNotNull(statsMap);
- assertEquals(1, statsMap.size());
- if (log.isInfoEnabled())
- log.info(statsMap.toString());
- }
-
- // validate the query solution stats.
- {
- final BOpStats stats = statsMap.get(startId);
- assertNotNull(stats);
- if (log.isInfoEnabled())
- log.info(stats.toString());
-
- // query solution stats details.
- assertEquals(1L, stats.chunksIn.get());
- assertEquals(1L, stats.unitsIn.get());
- assertEquals(1L, stats.unitsOut.get());
- assertEquals(1L, stats.chunksOut.get());
- }
-
- // Verify results.
- {
- // the expected solution (just one empty binding set).
- final IBindingSet[] expected = new IBindingSet[] {//
- new HashBindingSet() //
- };
-
- TestQueryEngine.assertSameSolutionsAnyOrder(expected,
- new Dechunkerator<IBindingSet>(runningQuery.iterator()));
-
- }
-
- }
-
- /**
- * Test the ability run a simple join. There are three operators. One feeds
- * an empty binding set[] into the join, another is the predicate for the
- * access path on which the join will read (it probes the index once for
- * "Mary" and bindings "Paul" when it does so), and the third is the join
- * itself (there is one solution, which is "value=Paul").
- *
- * @throws Exception
- */
- public void test_query_join1() throws Exception {
-
- final int startId = 1;
- final int joinId = 2;
- final int predId = 3;
- final int sliceId = 4;
-
- final StartOp startOp = new StartOp(new BOp[] {}, NV.asMap(new NV[] {//
- new NV(Predicate.Annotations.BOP_ID, startId),//
- new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.CONTROLLER),//
- }));
-
- final Predicate<E> predOp = new Predicate<E>(
- new IVariableOrConstant[] { new Constant<String>("Mary"),
- Var.var("value") },
- NV.asMap(new NV[] {//
- new NV(Predicate.Annotations.RELATION_NAME,
- new String[] { namespace }),//
- // Note: local access path!
- new NV(
- Predicate.Annotations.REMOTE_ACCESS_PATH,
- false),
- new NV(Predicate.Annotations.BOP_ID, predId),//
- new NV(Predicate.Annotations.TIMESTAMP,
- ITx.READ_COMMITTED),//
- }));
-
- final PipelineJoin<E> joinOp = new PipelineJoin<E>(
- new BOp[] { startOp },//
- new NV(Predicate.Annotations.BOP_ID, joinId),//
- new NV(PipelineJoin.Annotations.PREDICATE, predOp),//
- // Note: shard-partitioned joins!
- new NV(Predicate.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.SHARDED));
-
- final PipelineOp query = new SliceOp(new BOp[] { joinOp },
- // slice annotations
- NV.asMap(new NV[] {//
- new NV(Predicate.Annotations.BOP_ID, sliceId),//
- new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.CONTROLLER),//
- })//
- );
-
- // the expected solutions.
- final IBindingSet[] expected = new IBindingSet[] {//
- new ArrayBindingSet(//
- new IVariable[] { Var.var("value") },//
- new IConstant[] { new Constant<String>("Paul") }//
- ), //
- new ArrayBindingSet(//
- new IVariable[] { Var.var("value") },//
- new IConstant[] { new Constant<String>("John") }//
- ) };
-
- final UUID queryId = UUID.randomUUID();
- final RunningQuery runningQuery = queryEngine.eval(queryId, query,
- new LocalChunkMessage<IBindingSet>(queryEngine, queryId,
- startId,//
- -1, /* partitionId */
- newBindingSetIterator(new HashBindingSet())));
-
- // verify solutions.
- TestQueryEngine.assertSameSolutionsAnyOrder(expected,
- new Dechunkerator<IBindingSet>(runningQuery.iterator()));
-
- // Wait until the query is done.
- runningQuery.get();
- final Map<Integer, BOpStats> statsMap = runningQuery.getStats();
- {
- // validate the stats map.
- assertNotNull(statsMap);
- assertEquals(2, statsMap.size());
- if (log.isInfoEnabled())
- log.info(statsMap.toString());
- }
-
- // validate the stats for the start operator.
- {
- final BOpStats stats = statsMap.get(startId);
- assertNotNull(stats);
- if (log.isInfoEnabled())
- log.info("start: "+stats.toString());
-
- // verify query solution stats details.
- assertEquals(1L, stats.chunksIn.get());
- assertEquals(1L, stats.unitsIn.get());
- assertEquals(1L, stats.unitsOut.get());
- assertEquals(1L, stats.chunksOut.get());
- }
-
- // validate the stats for the join operator.
- {
- final BOpStats stats = statsMap.get(joinId);
- assertNotNull(stats);
- if (log.isInfoEnabled())
- log.info("join : "+stats.toString());
-
- // verify query solution stats details.
- assertEquals(1L, stats.chunksIn.get());
- assertEquals(1L, stats.unitsIn.get());
- assertEquals(2L, stats.unitsOut.get());
- assertEquals(1L, stats.chunksOut.get()); // @todo this depends on which index partitions we read on.
- }
-
- }
-
- /**
- * @todo Test the ability close the iterator draining a result set before
- * the query has finished executing and verify that the query is
- * correctly terminated [this is difficult to test without having
- * significant data scale since there is an implicit race between the
- * consumer and the producer to close out the query evaluation, but
- * the {@link PipelineDelayOp} can be used to impose sufficient
- * latency on the pipeline that the test can close the query buffer
- * iterator first].
- * <p>
- * This must also be tested in scale-out to make sure that the data
- * backing the solutions is not discarded before the caller can use
- * those data. [This could be handled by materializing binding set
- * objects out of a {@link ByteBuffer} rather than using a live decode
- * of the data in that {@link ByteBuffer}.]
- */
- public void test_query_closeIterator() {
-
- fail("write test");
-
- }
-
- /**
- * @todo Test ability to impose a limit/offset slice on a query.
- * <p>
- * Note: While the logic for visiting only the solutions selected by
- * the slice can be tested against a mock object, the integration by
- * which a slice halts a query when it is satisfied has to be tested
- * against a {@link QueryEngine}.
- * <p>
- * This must also be tested in scale-out to make sure that the data
- * backing the solutions is not discarded before the caller can use
- * those data. [This could be handled by materializing binding set
- * objects out of a {@link ByteBuffer} rather than using a live decode
- * of the data in that {@link ByteBuffer}.]
- */
- public void test_query_slice() {
-
- fail("write test");
-
- }
-
- /**
- * @todo Test the ability run a query reading on an access path using a
- * element filter (other than DISTINCT).
- */
- public void test_query_join1_filter() {
-
- fail("write test");
-
- }
-
- /**
- * @todo Test the ability run a query reading on an access path using a
- * DISTINCT filter for selected variables on that access path (the
- * DISTINCT filter is a different from most other access path filters
- * since it stateful and is applied across all chunks on all shards).
- */
- public void test_query_join1_distinctAccessPath() {
-
- fail("write test");
-
- }
-
- /**
- * Test the ability run a query requiring two joins.
- *
- * @todo Verify join constraints (e.g., x == y or x != y).
- *
- * @todo run with different initial bindings (x=Mary, x is unbound, etc).
- */
- public void test_query_join2() throws Exception {
-
- final int startId = 1;
- final int joinId1 = 2;
- final int predId1 = 3;
- final int joinId2 = 4;
- final int predId2 = 5;
- final int sliceId = 6;
-
- final PipelineOp startOp = new StartOp(new BOp[] {},
- NV.asMap(new NV[] {//
- new NV(Predicate.Annotations.BOP_ID, startId),//
- new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.CONTROLLER),//
- }));
-
- final Predicate<?> pred1Op = new Predicate<E>(new IVariableOrConstant[] {
- Var.var("x"), Var.var("y") }, NV
- .asMap(new NV[] {//
- new NV(Predicate.Annotations.RELATION_NAME,
- new String[] { namespace }),//
- // Note: local access path!
- new NV( Predicate.Annotations.REMOTE_ACCESS_PATH,false),
- new NV(Predicate.Annotations.BOP_ID, predId1),//
- new NV(Predicate.Annotations.TIMESTAMP, ITx.READ_COMMITTED),//
- }));
-
- final Predicate<?> pred2Op = new Predicate<E>(new IVariableOrConstant[] {
- Var.var("y"), Var.var("z") }, NV
- .asMap(new NV[] {//
- new NV(Predicate.Annotations.RELATION_NAME,
- new String[] { namespace }),//
- // Note: local access path!
- new NV( Predicate.Annotations.REMOTE_ACCESS_PATH,false),
- new NV(Predicate.Annotations.BOP_ID, predId2),//
- new NV(Predicate.Annotations.TIMESTAMP, ITx.READ_COMMITTED),//
- }));
-
- final PipelineOp join1Op = new PipelineJoin<E>(//
- new BOp[] { startOp },//
- new NV(Predicate.Annotations.BOP_ID, joinId1),//
- new NV(PipelineJoin.Annotations.PREDICATE, pred1Op),//
- // Note: shard-partitioned joins!
- new NV(Predicate.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.SHARDED));
-
- final PipelineOp join2Op = new PipelineJoin<E>(//
- new BOp[] { join1Op },//
- new NV(Predicate.Annotations.BOP_ID, joinId2),//
- new NV(PipelineJoin.Annotations.PREDICATE, pred2Op),//
- // Note: shard-partitioned joins!
- new NV(Predicate.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.SHARDED));
-
- final PipelineOp query = new SliceOp(new BOp[] { join2Op },
- NV.asMap(new NV[] {//
- new NV(Predicate.Annotations.BOP_ID, sliceId),//
- new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.CONTROLLER),//
- }));
-
- // start the query.
- final UUID queryId = UUID.randomUUID();
- final IChunkMessage<IBindingSet> initialChunkMessage;
- {
-
- final IBindingSet initialBindings = new HashBindingSet();
-
- initialBindings.set(Var.var("x"), new Constant<String>("Mary"));
-
- initialChunkMessage = new LocalChunkMessage<IBindingSet>(
- queryEngine, queryId, startId,//
- -1, // partitionId
- newBindingSetIterator(initialBindings));
-
- }
- final RunningQuery runningQuery = queryEngine.eval(queryId, query,
- initialChunkMessage);
-
- // verify solutions.
- {
-
- // the expected solution (just one).
- final IBindingSet[] expected = new IBindingSet[] {//
- new ArrayBindingSet(//
- new IVariable[] { Var.var("x"), Var.var("y"), Var.var("z") },//
- new IConstant[] { new Constant<String>("Mary"),
- new Constant<String>("Paul"),
- new Constant<String>("Leon") }//
- ),//
- new ArrayBindingSet(//
- new IVariable[] { Var.var("x"), Var.var("y"), Var.var("z") },//
- new IConstant[] { new Constant<String>("Mary"),
- new Constant<String>("John"),
- new Constant<String>("Mary") }//
- )};
-
- TestQueryEngine.assertSameSolutionsAnyOrder(expected,
- new Dechunkerator<IBindingSet>(runningQuery.iterator()));
-
- }
-
- // Wait until the query is done.
- runningQuery.get();
- final Map<Integer, BOpStats> statsMap = runningQuery.getStats();
- {
- // validate the stats map.
- assertNotNull(statsMap);
- assertEquals(3, statsMap.size());
- if (log.isInfoEnabled())
- log.info(statsMap.toString());
- }
-
- // validate the stats for the start operator.
- {
- final BOpStats stats = statsMap.get(startId);
- assertNotNull(stats);
- if (log.isInfoEnabled())
- log.info("start: " + stats.toString());
-
- // verify query solution stats details.
- assertEquals(1L, stats.chunksIn.get());
- assertEquals(1L, stats.unitsIn.get());
- assertEquals(1L, stats.unitsOut.get());
- assertEquals(1L, stats.chunksOut.get());
- }
-
- // validate the stats for the 1st join operator.
- {
- final BOpStats stats = statsMap.get(joinId1);
- assertNotNull(stats);
- if (log.isInfoEnabled())
- log.info("join1: " + stats.toString());
-
- // verify query solution stats details.
- assertEquals(1L, stats.chunksIn.get());
- assertEquals(1L, stats.unitsIn.get());
- assertEquals(2L, stats.unitsOut.get());
- assertEquals(1L, stats.chunksOut.get()); // @todo depends on where the shards are.
- }
-
- // validate the stats for the 2nd join operator.
- {
- final BOpStats stats = statsMap.get(joinId2);
- assertNotNull(stats);
- if (log.isInfoEnabled())
- log.info("join2: " + stats.toString());
-
- // verify query solution stats details.
- assertEquals(1L, stats.chunksIn.get()); // @todo depends on where the shards are.
- assertEquals(2L, stats.unitsIn.get());
- assertEquals(2L, stats.unitsOut.get());
- assertEquals(1L, stats.chunksOut.get()); // @todo depends on where the shards are.
- }
-
- // validate stats for the sliceOp (on the query controller)
- {
- final BOpStats stats = statsMap.get(sliceId);
- assertNotNull(stats);
- if (log.isInfoEnabled())
- log.info("slice: " + stats.toString());
-
- // verify query solution stats details.
- assertEquals(1L, stats.chunksIn.get()); // @todo?
- assertEquals(2L, stats.unitsIn.get());
- assertEquals(2L, stats.unitsOut.get());
- assertEquals(1L, stats.chunksOut.get()); // @todo?
- }
-
- }
-
- /**
- * @todo Write unit tests for optional joins, including where an alternative
- * sink is specified in the {@link BOpContext} and is used when the
- * join fails.
- * */
- public void test_query_join2_optionals() {
-
- fail("write test");
-
- }
-
-}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-11-02 17:42:32
|
Revision: 3871
http://bigdata.svn.sourceforge.net/bigdata/?rev=3871&view=rev
Author: thompsonbry
Date: 2010-11-02 17:42:25 +0000 (Tue, 02 Nov 2010)
Log Message:
-----------
Fixing a warning concerning an empty test suite for the ctc striterators.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/ctc-striterators/src/test/cutthecrap/utils/striterators/TestFilter.java
Modified: branches/QUADS_QUERY_BRANCH/ctc-striterators/src/test/cutthecrap/utils/striterators/TestFilter.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/ctc-striterators/src/test/cutthecrap/utils/striterators/TestFilter.java 2010-11-02 16:32:36 UTC (rev 3870)
+++ branches/QUADS_QUERY_BRANCH/ctc-striterators/src/test/cutthecrap/utils/striterators/TestFilter.java 2010-11-02 17:42:25 UTC (rev 3871)
@@ -27,7 +27,7 @@
package cutthecrap.utils.striterators;
-import junit.framework.TestCase;
+import junit.framework.TestCase2;
/**
* Test suite for {@link Filter}.
@@ -35,7 +35,7 @@
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
* @version $Id$
*/
-public class TestFilter extends TestCase {
+public class TestFilter extends TestCase2 {
/**
*
@@ -49,5 +49,12 @@
public TestFilter(String name) {
super(name);
}
+
+ /**
+ * FIXME Write unit tests for the ctc-striterators package.
+ */
+ public void test_something() {
+ log.error("Write unit tests");
+ }
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-11-02 16:32:42
|
Revision: 3870
http://bigdata.svn.sourceforge.net/bigdata/?rev=3870&view=rev
Author: thompsonbry
Date: 2010-11-02 16:32:36 +0000 (Tue, 02 Nov 2010)
Log Message:
-----------
Javadoc edits. The SPARQL test suite passes for me against a 2DS federation.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataFederationSparqlTest.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataFederationSparqlTest.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataFederationSparqlTest.java 2010-11-02 15:56:07 UTC (rev 3869)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataFederationSparqlTest.java 2010-11-02 16:32:36 UTC (rev 3870)
@@ -54,6 +54,15 @@
* Runs the SPARQL test suite against a {@link JiniFederation}, which must be
* already deployed. Each test in the suite is run against a distinct quad store
* in its own bigdata namespace.
+ * <p>
+ * To run this test suite, you need to have a deployed federation. You then specify
+ * the configuration file for that deployed federation. If sysstat is in a non-default
+ * location, then it is convenient (but not necessary) to also specify its path. For
+ * example:
+ * <pre>
+ * -Dbigdata.configuration=/nas/bigdata/benchmark/config/bigdataStandalone.config
+ * -Dcom.bigdata.counters.linux.sysstat.path=/usr/local/bin
+ * </pre>
*
* @author <a href="mailto:dm...@us...">David MacMillan</a>
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-11-02 15:56:14
|
Revision: 3869
http://bigdata.svn.sourceforge.net/bigdata/?rev=3869&view=rev
Author: thompsonbry
Date: 2010-11-02 15:56:07 +0000 (Tue, 02 Nov 2010)
Log Message:
-----------
Removed WARNING for the WORMStrategy impl in the JOURNAL_HA_BRANCH
Modified Paths:
--------------
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/WORMStrategy.java
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/WORMStrategy.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/WORMStrategy.java 2010-11-02 15:23:24 UTC (rev 3868)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/WORMStrategy.java 2010-11-02 15:56:07 UTC (rev 3869)
@@ -912,11 +912,11 @@
this._checkbuf = useChecksums ? ByteBuffer.allocateDirect(4) : null;
}
- System.err.println("WARNING: alpha impl: "
- + this.getClass().getName()
- + (writeCacheService != null ? " : writeCacheBuffers="
- + fileMetadata.writeCacheBufferCount : " : No cache")
- + ", useChecksums=" + useChecksums);
+// System.err.println("WARNING: alpha impl: "
+// + this.getClass().getName()
+// + (writeCacheService != null ? " : writeCacheBuffers="
+// + fileMetadata.writeCacheBufferCount : " : No cache")
+// + ", useChecksums=" + useChecksums);
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-11-02 15:23:31
|
Revision: 3868
http://bigdata.svn.sourceforge.net/bigdata/?rev=3868&view=rev
Author: thompsonbry
Date: 2010-11-02 15:23:24 +0000 (Tue, 02 Nov 2010)
Log Message:
-----------
This commit resolves memory leaks for Journal and QueryEngine references.
See https://sourceforge.net/apps/trac/bigdata/ticket/196
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/QueryEngineFactory.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractJournal.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/WriteExecutorService.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/util/concurrent/ThreadPoolExecutorStatisticsTask.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/journal/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSail.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataConnectionTest.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataEmbeddedFederationSparqlTest.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataFederationSparqlTest.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataSparqlTest.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataStoreTest.java
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestQueryEngineFactory.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/journal/TestJournalShutdown.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java 2010-11-02 13:21:34 UTC (rev 3867)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java 2010-11-02 15:23:24 UTC (rev 3868)
@@ -47,6 +47,7 @@
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.PipelineOp;
import com.bigdata.bop.bindingSet.HashBindingSet;
+import com.bigdata.bop.fed.QueryEngineFactory;
import com.bigdata.btree.BTree;
import com.bigdata.btree.IndexSegment;
import com.bigdata.btree.view.FusedView;
@@ -344,9 +345,9 @@
*/
public void init() {
- final FutureTask<Void> ft = new FutureTask<Void>(new QueryEngineTask(),
- (Void) null);
-
+ final FutureTask<Void> ft = new FutureTask<Void>(new QueryEngineTask(
+ priorityQueue), (Void) null);
+
if (engineFuture.compareAndSet(null/* expect */, ft)) {
engineService.set(Executors
@@ -365,13 +366,18 @@
}
/**
- * {@link QueryEngine}s are using with a singleton pattern. They must be
- * torn down automatically once they are no longer reachable.
+ * {@link QueryEngine}s are used with a singleton pattern managed by the
+ * {@link QueryEngineFactory}. They are torn down automatically once they
+ * are no longer reachable. This behavior depends on not having any hard
+ * references back to the {@link QueryEngine}.
*/
@Override
protected void finalize() throws Throwable {
+
shutdownNow();
+
super.finalize();
+
}
/**
@@ -414,6 +420,12 @@
/**
* Runnable submits chunks available for evaluation against running queries.
+ * <p>
+ * Note: This is a static inner class in order to avoid a hard reference
+ * back to the outer {@link QueryEngine} object. This makes it possible
+ * for the JVM to finalize the {@link QueryEngine} if the application no
+ * longer holds a hard reference to it. The {@link QueryEngine} is then
+ * automatically closed from within its finalizer method.
*
* @todo Handle priority for selective queries based on the time remaining
* until the timeout.
@@ -436,13 +448,25 @@
* the same target ByteBuffer, or when we add the chunk to the
* RunningQuery.]
*/
- private class QueryEngineTask implements Runnable {
+ static private class QueryEngineTask implements Runnable {
+
+ final private BlockingQueue<RunningQuery> queue;
+
+ public QueryEngineTask(final BlockingQueue<RunningQuery> queue) {
+
+ if (queue == null)
+ throw new IllegalArgumentException();
+
+ this.queue = queue;
+
+ }
+
public void run() {
if(log.isInfoEnabled())
log.info("Running: " + this);
while (true) {
try {
- final RunningQuery q = priorityQueue.take();
+ final RunningQuery q = queue.take();
if (!q.isDone())
q.consumeChunk();
} catch (InterruptedException e) {
@@ -454,7 +478,8 @@
* then you can instrument BlockingBuffer#close() in
* PipelineOp#newBuffer(stats).
*/
- log.warn("Interrupted."
+ if (log.isInfoEnabled())
+ log.info("Interrupted."
// ,e
);
return;
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/QueryEngineFactory.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/QueryEngineFactory.java 2010-11-02 13:21:34 UTC (rev 3867)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/QueryEngineFactory.java 2010-11-02 15:23:24 UTC (rev 3868)
@@ -38,6 +38,7 @@
import com.bigdata.journal.BufferMode;
import com.bigdata.journal.IIndexManager;
import com.bigdata.journal.Journal;
+import com.bigdata.rawstore.Bytes;
import com.bigdata.service.IBigdataClient;
import com.bigdata.service.IBigdataFederation;
import com.bigdata.service.ManagedResourceService;
@@ -55,16 +56,28 @@
/**
* Weak value cache to enforce the singleton pattern for standalone
* journals.
+ * <p>
+ * Note: The backing hard reference queue is disabled since we do not want
+ * to keep any {@link QueryEngine} objects wired into the cache unless the
+ * application is holding a hard reference to the {@link QueryEngine}.
*/
- private static ConcurrentWeakValueCache<Journal, QueryEngine> standaloneQECache = new ConcurrentWeakValueCache<Journal, QueryEngine>();
+ private static ConcurrentWeakValueCache<Journal, QueryEngine> standaloneQECache = new ConcurrentWeakValueCache<Journal, QueryEngine>(
+ 0/* queueCapacity */
+ );
/**
* Weak value cache to enforce the singleton pattern for
* {@link IBigdataClient}s (the data services are query engine peers rather
* than controllers and handle their own query engine initialization so as
* to expose their resources to other peers).
+ * <p>
+ * Note: The backing hard reference queue is disabled since we do not want
+ * to keep any {@link QueryEngine} objects wired into the cache unless the
+ * application is holding a hard reference to the {@link QueryEngine}.
*/
- private static ConcurrentWeakValueCache<IBigdataFederation<?>, FederatedQueryEngine> federationQECache = new ConcurrentWeakValueCache<IBigdataFederation<?>, FederatedQueryEngine>();
+ private static ConcurrentWeakValueCache<IBigdataFederation<?>, FederatedQueryEngine> federationQECache = new ConcurrentWeakValueCache<IBigdataFederation<?>, FederatedQueryEngine>(
+ 0/* queueCapacity */
+ );
/**
* Singleton factory for standalone or scale-out.
@@ -87,22 +100,6 @@
}
/**
- * Removes a QueryEngine instance from the cache if it is present, returning it to the caller. This
- * method is unlikely to be useful in applications but the unit test framework requires it in order
- * to avoid resource starvation as each test typically creates a unique IIndexManager.
- *
- * @param indexManager the database
- * @return the query controller if present, null otherwise.
- */
- public static QueryEngine removeQueryController ( final IIndexManager indexManager )
- {
- if (indexManager instanceof IBigdataFederation<?>) {
- return federationQECache.remove ( ( IBigdataFederation<?> )indexManager ) ;
- }
- return standaloneQECache.remove ( ( Journal )indexManager ) ;
- }
-
- /**
* Singleton factory for standalone.
*
* @param indexManager
@@ -321,4 +318,13 @@
}
+ /**
+ * Return the #of live query controllers.
+ */
+ public static int getQueryControllerCount() {
+
+ return standaloneQECache.size() + federationQECache.size();
+
+ }
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractJournal.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractJournal.java 2010-11-02 13:21:34 UTC (rev 3867)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractJournal.java 2010-11-02 15:23:24 UTC (rev 3868)
@@ -1250,7 +1250,7 @@
/**
* Closes out the journal iff it is still open.
*/
- protected void finalize() throws Exception {
+ protected void finalize() throws Throwable {
if(_bufferStrategy.isOpen()) {
@@ -1258,7 +1258,7 @@
log.info("Closing journal: " + getFile());
shutdownNow();
-
+
}
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/WriteExecutorService.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/WriteExecutorService.java 2010-11-02 13:21:34 UTC (rev 3867)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/WriteExecutorService.java 2010-11-02 15:23:24 UTC (rev 3868)
@@ -23,6 +23,7 @@
*/
package com.bigdata.journal;
+import java.lang.ref.WeakReference;
import java.nio.channels.Channel;
import java.nio.channels.FileChannel;
import java.util.Arrays;
@@ -220,7 +221,8 @@
private static class MyLockManager<R extends Comparable<R>> extends
NonBlockingLockManagerWithNewDesign<R> {
- private final WriteExecutorService service;
+// private final WriteExecutorService service;
+ private final WeakReference<WriteExecutorService> serviceRef;
public MyLockManager(final int capacity, final int maxLockTries,
final boolean predeclareLocks,
@@ -228,12 +230,20 @@
super(capacity, maxLockTries, predeclareLocks);
- this.service = service;
+// this.service = service;
+ this.serviceRef = new WeakReference<WriteExecutorService>(service);
}
protected void ready(final Runnable r) {
+// service.execute(r);
+
+ final WriteExecutorService service = serviceRef.get();
+
+ if(service == null)
+ throw new RejectedExecutionException();
+
service.execute(r);
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/util/concurrent/ThreadPoolExecutorStatisticsTask.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/util/concurrent/ThreadPoolExecutorStatisticsTask.java 2010-11-02 13:21:34 UTC (rev 3867)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/util/concurrent/ThreadPoolExecutorStatisticsTask.java 2010-11-02 15:23:24 UTC (rev 3868)
@@ -1,5 +1,6 @@
package com.bigdata.util.concurrent;
+import java.lang.ref.WeakReference;
import java.util.concurrent.Callable;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
@@ -38,8 +39,24 @@
/**
* The executor service that is being monitored.
*/
- private final ThreadPoolExecutor service;
+// private final ThreadPoolExecutor service;
+ private final WeakReference<ThreadPoolExecutor> serviceRef;
+
+ private ThreadPoolExecutor getService() {
+
+ final ThreadPoolExecutor service = serviceRef.get();
+ if (service == null) {
+
+ // Throw exception which should cause the task to stop executing.
+ throw new RuntimeException("Service was shutdown.");
+
+ }
+
+ return service;
+
+ }
+
// /**
// * The time when we started to collect data about the {@link #service} (set by the ctor).
// */
@@ -207,7 +224,8 @@
this.serviceName = serviceName;
- this.service = service;
+// this.service = service;
+ this.serviceRef = new WeakReference<ThreadPoolExecutor>(service);
// this.startNanos = System.nanoTime();
@@ -243,7 +261,7 @@
private final MovingAverageTask queueSizeTask = new MovingAverageTask(
"queueSize", new Callable<Integer>() {
public Integer call() {
- return service.getQueue().size();
+ return getService().getQueue().size();
}
});
@@ -280,6 +298,15 @@
*/
public void run() {
+ /*
+ * Note: This will throw a RuntimeException if the weak reference has
+ * been cleared. This decouples the task from the monitored service
+ * which let's the monitored service shutdown when it is no longer
+ * referenced by the application (assuming that it implements a
+ * finalize() method).
+ */
+ final ThreadPoolExecutor service = getService();
+
try {
{
@@ -553,6 +580,9 @@
public CounterSet getCounters() {
final CounterSet counterSet = new CounterSet();
+
+ // Reference to the service : MAY have been cleared by GC.
+ final ThreadPoolExecutor service = serviceRef.get();
/*
* Defined for ThreadPoolExecutor.
@@ -605,6 +635,7 @@
*/
{
+ if(service != null) {
if (taskCounters == null) {
/*
@@ -634,7 +665,8 @@
setValue(service.getLargestPoolSize());
}
});
-
+ }
+
counterSet.addCounter(
IThreadPoolExecutorCounters.AverageActiveCount,
new Instrument<Double>() {
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestAll.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestAll.java 2010-11-02 13:21:34 UTC (rev 3867)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestAll.java 2010-11-02 15:23:24 UTC (rev 3868)
@@ -81,6 +81,9 @@
// unit tests for a remote access path.
suite.addTestSuite(TestRemoteAccessPath.class);
+ // look for memory leaks in the query engine factory.
+ suite.addTestSuite(TestQueryEngineFactory.class);
+
/*
* Unit tests for the federated query engine against an embedded
* federation with a single data service.
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestQueryEngineFactory.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestQueryEngineFactory.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestQueryEngineFactory.java 2010-11-02 15:23:24 UTC (rev 3868)
@@ -0,0 +1,119 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Nov 2, 2010
+ */
+
+package com.bigdata.bop.fed;
+
+import java.util.Properties;
+
+import com.bigdata.journal.BufferMode;
+import com.bigdata.journal.Journal;
+import com.bigdata.rawstore.Bytes;
+
+import junit.framework.TestCase2;
+
+/**
+ * Stress test for correct shutdown of query controllers as allocated by the
+ * {@link QueryEngineFactory}.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class TestQueryEngineFactory extends TestCase2 {
+
+ /**
+ *
+ */
+ public TestQueryEngineFactory() {
+ }
+
+ /**
+ * @param name
+ */
+ public TestQueryEngineFactory(String name) {
+ super(name);
+ }
+
+ /**
+ * Look for a memory leak in the {@link QueryEngineFactory}.
+ *
+ * @throws InterruptedException
+ */
+ public void test_memoryLeak() throws InterruptedException {
+
+ final int limit = 200;
+
+ final Properties properties = new Properties();
+
+ properties.setProperty(Journal.Options.BUFFER_MODE,
+ BufferMode.Transient.toString());
+
+ properties.setProperty(Journal.Options.INITIAL_EXTENT, ""
+ + Bytes.megabyte * 10);
+
+ int ncreated = 0;
+
+ try {
+
+ for (int i = 0; i < limit; i++) {
+
+ Journal jnl = new Journal(properties);
+
+ QueryEngineFactory.getQueryController(jnl);
+
+ ncreated++;
+
+ }
+
+ } catch (OutOfMemoryError err) {
+
+ System.err.println("Out of memory after creating " + ncreated
+ + " query controllers.");
+
+ }
+
+ // Demand a GC.
+ System.gc();
+
+ // Wait for it.
+ Thread.sleep(1000/*ms*/);
+
+ System.err.println("Created " + ncreated + " query controllers.");
+
+ final int nalive = QueryEngineFactory.getQueryControllerCount();
+
+ System.err.println("There are " + nalive
+ + " query controllers which are still alive.");
+
+ if (nalive == ncreated) {
+
+ fail("No query controllers were finalized.");
+
+ }
+
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestQueryEngineFactory.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/journal/TestAll.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/journal/TestAll.java 2010-11-02 13:21:34 UTC (rev 3867)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/journal/TestAll.java 2010-11-02 15:23:24 UTC (rev 3868)
@@ -133,6 +133,9 @@
suite.addTest( com.bigdata.rwstore.TestAll.suite() );
+ // test suite for memory leaks in the journal shutdown protocol.
+ suite.addTestSuite(TestJournalShutdown.class);
+
return suite;
}
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/journal/TestJournalShutdown.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/journal/TestJournalShutdown.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/journal/TestJournalShutdown.java 2010-11-02 15:23:24 UTC (rev 3868)
@@ -0,0 +1,125 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Nov 2, 2010
+ */
+
+package com.bigdata.journal;
+
+import java.util.Properties;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import junit.framework.TestCase2;
+
+import com.bigdata.bop.fed.QueryEngineFactory;
+import com.bigdata.rawstore.Bytes;
+
+/**
+ * Stress test for correct shutdown of journals based on weak reference
+ * semantics.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class TestJournalShutdown extends TestCase2 {
+
+ /**
+ *
+ */
+ public TestJournalShutdown() {
+ }
+
+ /**
+ * @param name
+ */
+ public TestJournalShutdown(String name) {
+ super(name);
+ }
+
+ /**
+ * Look for a memory leak in the {@link QueryEngineFactory}.
+ *
+ * @throws InterruptedException
+ */
+ public void test_memoryLeak() throws InterruptedException {
+
+ final int limit = 200;
+
+ final Properties properties = new Properties();
+
+ properties.setProperty(Journal.Options.BUFFER_MODE,
+ BufferMode.Transient.toString());
+
+ properties.setProperty(Journal.Options.INITIAL_EXTENT, ""
+ + Bytes.megabyte * 10);
+
+ final AtomicInteger ncreated = new AtomicInteger();
+
+ final AtomicInteger nalive = new AtomicInteger();
+
+ try {
+
+ for (int i = 0; i < limit; i++) {
+
+ Journal jnl = new Journal(properties) {
+ protected void finalize() throws Throwable {
+ super.finalize();
+ nalive.decrementAndGet();
+ System.err.println("Journal was finalized: ncreated="
+ + ncreated + ", nalive=" + nalive);
+ }
+ };
+
+ nalive.incrementAndGet();
+ ncreated.incrementAndGet();
+
+ }
+
+ } catch (OutOfMemoryError err) {
+
+ System.err.println("Out of memory after creating " + ncreated
+ + " journals.");
+
+ }
+
+ // Demand a GC.
+ System.gc();
+
+ // Wait for it.
+ Thread.sleep(1000/*ms*/);
+
+ System.err.println("Created " + ncreated + " journals.");
+
+ System.err.println("There are " + nalive
+ + " journals which are still alive.");
+
+ if (nalive.get() == ncreated.get()) {
+
+ fail("No journals were finalized.");
+
+ }
+
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/journal/TestJournalShutdown.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSail.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSail.java 2010-11-02 13:21:34 UTC (rev 3867)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSail.java 2010-11-02 15:23:24 UTC (rev 3868)
@@ -130,8 +130,6 @@
import com.bigdata.rdf.rio.StatementBuffer;
import com.bigdata.rdf.rules.BackchainAccessPath;
import com.bigdata.rdf.rules.InferenceEngine;
-import com.bigdata.rdf.sail.BigdataSail.BigdataSailConnection;
-import com.bigdata.rdf.sail.BigdataSail.Options;
import com.bigdata.rdf.spo.ExplicitSPOFilter;
import com.bigdata.rdf.spo.ISPO;
import com.bigdata.rdf.spo.InferredSPOFilter;
@@ -1050,10 +1048,8 @@
try {
- shutDown();
- QueryEngine qe = QueryEngineFactory.getQueryController(database.getIndexManager());
- if ( null != qe )
- qe.shutdownNow () ;
+ if(isOpen()) shutDown();
+
database.__tearDownUnitTest();
} catch (Throwable t) {
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataConnectionTest.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataConnectionTest.java 2010-11-02 13:21:34 UTC (rev 3867)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataConnectionTest.java 2010-11-02 15:23:24 UTC (rev 3868)
@@ -35,14 +35,12 @@
import org.openrdf.repository.Repository;
import org.openrdf.repository.RepositoryConnectionTest;
-import com.bigdata.bop.engine.QueryEngine;
-import com.bigdata.bop.fed.QueryEngineFactory;
import com.bigdata.btree.keys.CollatorEnum;
import com.bigdata.btree.keys.StrengthEnum;
import com.bigdata.journal.IIndexManager;
import com.bigdata.rdf.sail.BigdataSail;
-import com.bigdata.rdf.sail.BigdataSail.Options;
import com.bigdata.rdf.sail.BigdataSailRepository;
+import com.bigdata.rdf.sail.BigdataSail.Options;
import com.bigdata.rdf.store.LocalTripleStore;
public class BigdataConnectionTest extends RepositoryConnectionTest {
@@ -53,29 +51,6 @@
public BigdataConnectionTest(String name) {
super(name);
}
-
-// /**
-// * Return a test suite using the {@link LocalTripleStore} and nested
-// * subquery joins.
-// */
-// public static class LTSWithNestedSubquery extends BigdataConnectionTest {
-//
-// public LTSWithNestedSubquery(String name) {
-// super(name);
-// }
-//
-// @Override
-// protected Properties getProperties() {
-//
-// final Properties p = new Properties(super.getProperties());
-//
-// p.setProperty(AbstractResource.Options.NESTED_SUBQUERY,"true");
-//
-// return p;
-//
-// }
-//
-// }
/**
* Return a test suite using the {@link LocalTripleStore} and pipeline
@@ -94,8 +69,6 @@
final Properties p = new Properties(super.getProperties());
-// p.setProperty(AbstractResource.Options.NESTED_SUBQUERY,"false");
-
return p;
}
@@ -174,12 +147,7 @@
super.tearDown();
if (backend != null)
- {
- QueryEngine qe = QueryEngineFactory.removeQueryController ( backend ) ;
- if ( null != qe )
- qe.shutdownNow () ;
backend.destroy();
- }
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataEmbeddedFederationSparqlTest.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataEmbeddedFederationSparqlTest.java 2010-11-02 13:21:34 UTC (rev 3867)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataEmbeddedFederationSparqlTest.java 2010-11-02 15:23:24 UTC (rev 3868)
@@ -41,13 +41,13 @@
import org.openrdf.repository.RepositoryException;
import org.openrdf.repository.dataset.DatasetRepository;
-import com.bigdata.bop.engine.QueryEngine;
-import com.bigdata.bop.fed.QueryEngineFactory;
+import com.bigdata.btree.keys.CollatorEnum;
+import com.bigdata.btree.keys.StrengthEnum;
import com.bigdata.journal.IIndexManager;
import com.bigdata.journal.ITx;
import com.bigdata.rdf.sail.BigdataSail;
+import com.bigdata.rdf.sail.BigdataSailRepository;
import com.bigdata.rdf.sail.BigdataSail.Options;
-import com.bigdata.rdf.sail.BigdataSailRepository;
import com.bigdata.rdf.store.AbstractTripleStore;
import com.bigdata.rdf.store.ScaleOutTripleStore;
import com.bigdata.resources.ResourceManager;
@@ -202,6 +202,12 @@
if (cannotInlineTests.contains(testURI))
properties.setProperty(Options.INLINE_LITERALS, "false");
+ if(unicodeStrengthIdentical.contains(testURI)) {
+ // Force identical Unicode comparisons.
+ properties.setProperty(Options.COLLATOR, CollatorEnum.JDK.toString());
+ properties.setProperty(Options.STRENGTH, StrengthEnum.Identical.toString());
+ }
+
client = new EmbeddedClient(properties);
fed = client.connect();
@@ -222,10 +228,7 @@
}
protected void tearDownBackend(IIndexManager backend) {
- QueryEngine qe = QueryEngineFactory.removeQueryController ( backend ) ;
- if ( null != qe )
- qe.shutdownNow () ;
-
+
backend.destroy();
if (client != null) {
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataFederationSparqlTest.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataFederationSparqlTest.java 2010-11-02 13:21:34 UTC (rev 3867)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataFederationSparqlTest.java 2010-11-02 15:23:24 UTC (rev 3868)
@@ -38,16 +38,14 @@
import org.openrdf.repository.RepositoryConnection;
import org.openrdf.repository.dataset.DatasetRepository;
-import com.bigdata.bop.engine.QueryEngine;
-import com.bigdata.bop.fed.QueryEngineFactory;
import com.bigdata.btree.keys.CollatorEnum;
import com.bigdata.btree.keys.KeyBuilder;
import com.bigdata.btree.keys.StrengthEnum;
import com.bigdata.journal.ITx;
import com.bigdata.rdf.sail.BigdataSail;
-import com.bigdata.rdf.sail.BigdataSail.Options;
import com.bigdata.rdf.sail.BigdataSailRepository;
import com.bigdata.rdf.sail.BigdataSailRepositoryConnection;
+import com.bigdata.rdf.sail.BigdataSail.Options;
import com.bigdata.rdf.store.ScaleOutTripleStore;
import com.bigdata.service.jini.JiniClient;
import com.bigdata.service.jini.JiniFederation;
@@ -98,24 +96,29 @@
}
/**
- * Return the entire test suite.
+ * Return the entire test suite.
*/
public static TestSuite fullSuite() throws Exception {
- return ManifestTest.suite
- (
- new Factory ()
- {
- public SPARQLQueryTest createSPARQLQueryTest ( String URI, String name, String query, String results, Dataset dataSet, boolean laxCardinality)
- {
- return new BigdataFederationSparqlTest ( URI, name, query, results, dataSet, laxCardinality ) ;
- }
+
+ return ManifestTest.suite(new Factory() {
+
+ public SPARQLQueryTest createSPARQLQueryTest(String URI,
+ String name, String query, String results, Dataset dataSet,
+ boolean laxCardinality) {
+
+ return new BigdataFederationSparqlTest(URI, name, query,
+ results, dataSet, laxCardinality);
+
}
- ) ;
+ });
+
}
- public BigdataFederationSparqlTest ( String URI, String name, String query, String results, Dataset dataSet, boolean laxCardinality )
- {
- super ( URI, name, query, results, dataSet, laxCardinality ) ;
+ public BigdataFederationSparqlTest(String URI, String name, String query,
+ String results, Dataset dataSet, boolean laxCardinality) {
+
+ super(URI, name, query, results, dataSet, laxCardinality);
+
}
@Override public void runTest ()
@@ -130,12 +133,6 @@
throws Exception
{
super.tearDown () ;
- if ( null != _sail )
- {
- QueryEngine qe = QueryEngineFactory.removeQueryController ( _sail.getDatabase ().getIndexManager () ) ;
- if ( null != qe )
- qe.shutdownNow () ;
- }
if (_ts != null) {
_ts.destroy();
_ts = null;
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataSparqlTest.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataSparqlTest.java 2010-11-02 13:21:34 UTC (rev 3867)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataSparqlTest.java 2010-11-02 15:23:24 UTC (rev 3868)
@@ -50,15 +50,13 @@
import org.openrdf.repository.sail.SailRepository;
import org.openrdf.sail.memory.MemoryStore;
-import com.bigdata.bop.engine.QueryEngine;
-import com.bigdata.bop.fed.QueryEngineFactory;
import com.bigdata.btree.keys.CollatorEnum;
import com.bigdata.btree.keys.StrengthEnum;
import com.bigdata.journal.BufferMode;
import com.bigdata.journal.IIndexManager;
import com.bigdata.rdf.sail.BigdataSail;
-import com.bigdata.rdf.sail.BigdataSail.Options;
import com.bigdata.rdf.sail.BigdataSailRepository;
+import com.bigdata.rdf.sail.BigdataSail.Options;
/**
* Test harness for running the SPARQL test suites.
@@ -416,9 +414,7 @@
protected void tearDownBackend(IIndexManager backend) {
backend.destroy();
- QueryEngine qe = QueryEngineFactory.removeQueryController ( backend ) ;
- if ( null != qe )
- qe.shutdownNow () ;
+
}
@Override
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataStoreTest.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataStoreTest.java 2010-11-02 13:21:34 UTC (rev 3867)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataStoreTest.java 2010-11-02 15:23:24 UTC (rev 3868)
@@ -37,8 +37,6 @@
import org.openrdf.sail.SailConnection;
import org.openrdf.sail.SailException;
-import com.bigdata.bop.engine.QueryEngine;
-import com.bigdata.bop.fed.QueryEngineFactory;
import com.bigdata.btree.keys.CollatorEnum;
import com.bigdata.btree.keys.StrengthEnum;
import com.bigdata.journal.IIndexManager;
@@ -49,30 +47,7 @@
public class BigdataStoreTest extends RDFStoreTest {
protected static final Logger log = Logger.getLogger(BigdataStoreTest.class);
-
-// /**
-// * Return a test suite using the {@link LocalTripleStore} and nested
-// * subquery joins.
-// */
-// public static class LTSWithNestedSubquery extends BigdataStoreTest {
-//
-// public LTSWithNestedSubquery(String name) {
-// super(name);
-// }
-//
-// @Override
-// protected Properties getProperties() {
-//
-// final Properties p = new Properties(super.getProperties());
-//
-// p.setProperty(AbstractResource.Options.NESTED_SUBQUERY,"true");
-//
-// return p;
-//
-// }
-//
-// }
-
+
/**
* Return a test suite using the {@link LocalTripleStore} and pipeline
* joins.
@@ -89,8 +64,6 @@
protected Properties getProperties() {
final Properties p = new Properties(super.getProperties());
-
-// p.setProperty(AbstractResource.Options.NESTED_SUBQUERY,"false");
return p;
@@ -125,12 +98,7 @@
super.tearDown();
if (backend != null)
- {
- QueryEngine qe = QueryEngineFactory.removeQueryController ( backend ) ;
- if ( null != qe )
- qe.shutdownNow () ;
backend.destroy();
- }
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-11-02 13:21:41
|
Revision: 3867
http://bigdata.svn.sourceforge.net/bigdata/?rev=3867&view=rev
Author: thompsonbry
Date: 2010-11-02 13:21:34 +0000 (Tue, 02 Nov 2010)
Log Message:
-----------
Unit test cleanup.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestFederatedQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/nodes/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/relation/rule/TestRule.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/filter/TestStripContextFilter.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/join/TestDataSetJoin.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/magic/TestAll.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOp.java 2010-11-02 12:48:14 UTC (rev 3866)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOp.java 2010-11-02 13:21:34 UTC (rev 3867)
@@ -198,6 +198,13 @@
* if {@link Annotations#TIMESTAMP} was not specified.
*/
long getTimestamp();
+
+// /**
+// * Compare this {@link BOp} with another {@link BOp}.
+// *
+// * @return <code>true</code> if all arguments and annotations are the same.
+// */
+// boolean sameData(final BOp o);
/**
* Interface declaring well known annotations.
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java 2010-11-02 12:48:14 UTC (rev 3866)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java 2010-11-02 13:21:34 UTC (rev 3867)
@@ -686,4 +686,42 @@
// */
// private int hash = 0;
+// public boolean sameData(final BOp o) {
+//
+// if (this == o)
+// return true;
+//
+// final int arity = arity();
+//
+// if (arity != o.arity())
+// return false;
+//
+// for (int i = 0; i < arity; i++) {
+//
+// final BOp x = get(i);
+//
+// final BOp y = o.get(i);
+//
+// /*
+// * X Y same same : continue (includes null == null); null other :
+// * return false; !null other : if(!x.equals(y)) return false.
+// */
+// if (x != y || x == null || !(x.equals(y))) {
+// // && (//
+// // (x != null && !(x.equals(y))) || //
+// // (y != null && !(y.equals(x))))//
+// // ) {
+//
+// return false;
+//
+// }
+//
+// }
+//
+// // @todo This would have to recursively apply sameData when comparing
+// // annotations which are bops.
+// return annotations.equals(o.annotations());
+//
+// }
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java 2010-11-02 12:48:14 UTC (rev 3866)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java 2010-11-02 13:21:34 UTC (rev 3867)
@@ -675,22 +675,22 @@
}
- /**
- * Test the ability of the query engine to defer the evaluation of a one
- * shot operator until all inputs are available for that operator.
- *
- * @todo We could do this using a mock operator and feeding a bunch of
- * chunks into the query by controlling the chunk size, as we do in
- * {@link #test_query_join1_multipleChunksIn()}. Make sure that the
- * mock operator is not evaluated until all inputs are available for
- * that operator.
- */
- public void test_oneShot_operator() {
+// /**
+// * Test the ability of the query engine to defer the evaluation of a one
+// * shot operator until all inputs are available for that operator.
+// *
+// * @todo We could do this using a mock operator and feeding a bunch of
+// * chunks into the query by controlling the chunk size, as we do in
+// * {@link #test_query_join1_multipleChunksIn()}. Make sure that the
+// * mock operator is not evaluated until all inputs are available for
+// * that operator.
+// */
+// public void test_oneShot_operator() {
+//
+// fail("write test");
+//
+// }
- fail("write test");
-
- }
-
/**
* Unit test runs chunks into a slice without a limit. This verifies that
* the query terminates properly even though the slice is willing to accept
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestAll.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestAll.java 2010-11-02 12:48:14 UTC (rev 3866)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestAll.java 2010-11-02 13:21:34 UTC (rev 3867)
@@ -64,8 +64,9 @@
// unit tests for mapping binding sets over shards.
suite.addTest(com.bigdata.bop.fed.shards.TestAll.suite());
- // unit tests for mapping binding sets over nodes.
- suite.addTest(com.bigdata.bop.fed.nodes.TestAll.suite());
+ // unit tests for mapping binding sets over nodes.
+ // @todo uncomment this test suite when the functionality is implemented.
+// suite.addTest(com.bigdata.bop.fed.nodes.TestAll.suite());
/*
* Chunk message tests.
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestFederatedQueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestFederatedQueryEngine.java 2010-11-02 12:48:14 UTC (rev 3866)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestFederatedQueryEngine.java 2010-11-02 13:21:34 UTC (rev 3867)
@@ -953,28 +953,28 @@
}
- /**
- * @todo Test the ability close the iterator draining a result set before
- * the query has finished executing and verify that the query is
- * correctly terminated [this is difficult to test without having
- * significant data scale since there is an implicit race between the
- * consumer and the producer to close out the query evaluation, but
- * the {@link PipelineDelayOp} can be used to impose sufficient
- * latency on the pipeline that the test can close the query buffer
- * iterator first].
- * <p>
- * This must also be tested in scale-out to make sure that the data
- * backing the solutions is not discarded before the caller can use
- * those data. [This could be handled by materializing binding set
- * objects out of a {@link ByteBuffer} rather than using a live decode
- * of the data in that {@link ByteBuffer}.]
- */
- public void test_query_closeIterator() {
+// /**
+// * @todo Test the ability close the iterator draining a result set before
+// * the query has finished executing and verify that the query is
+// * correctly terminated [this is difficult to test without having
+// * significant data scale since there is an implicit race between the
+// * consumer and the producer to close out the query evaluation, but
+// * the {@link PipelineDelayOp} can be used to impose sufficient
+// * latency on the pipeline that the test can close the query buffer
+// * iterator first].
+// * <p>
+// * This must also be tested in scale-out to make sure that the data
+// * backing the solutions is not discarded before the caller can use
+// * those data. [This could be handled by materializing binding set
+// * objects out of a {@link ByteBuffer} rather than using a live decode
+// * of the data in that {@link ByteBuffer}.]
+// */
+// public void test_query_closeIterator() {
+//
+//// fail("write test");
+//
+// }
- fail("write test");
-
- }
-
/**
* Test the ability run a query requiring two joins.
*
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/nodes/TestAll.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/nodes/TestAll.java 2010-11-02 12:48:14 UTC (rev 3866)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/nodes/TestAll.java 2010-11-02 13:21:34 UTC (rev 3867)
@@ -23,9 +23,6 @@
*/
package com.bigdata.bop.fed.nodes;
-
-import com.bigdata.bop.fed.shards.TestMapBindingSetsOverShards;
-
import junit.framework.Test;
import junit.framework.TestCase;
import junit.framework.TestSuite;
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/relation/rule/TestRule.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/relation/rule/TestRule.java 2010-11-02 12:48:14 UTC (rev 3866)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/relation/rule/TestRule.java 2010-11-02 13:21:34 UTC (rev 3867)
@@ -88,11 +88,19 @@
assertEquals("variableCount", 1, r.getVariableCount());
- assertTrue("head", new P(relation, u, rdfsSubClassOf, rdfsResource)
- .equals(r.getHead()));
+// {
+//
+// final IPredicate<?> tmp = new P(relation, u, rdfsSubClassOf,
+// rdfsResource);
+//
+// final IPredicate<?> head = r.getHead();
+//
+// assertTrue("head", tmp.equals(head));
+//
+// }
- assertTrue("tail[0]", new P(relation, u, rdfType, rdfsClass).equals(r
- .getTail(0)));
+// assertTrue("tail[0]", new P(relation, u, rdfType, rdfsClass).equals(r
+// .getTail(0)));
assertSameIteratorAnyOrder(new Comparable[] { u }, r.getVariables());
@@ -143,8 +151,8 @@
assertNull("head", r.getHead());
- assertTrue("tail[0]", new P(relation, u, rdfType, rdfsClass).equals(r
- .getTail(0)));
+// assertTrue("tail[0]", new P(relation, u, rdfType, rdfsClass).equals(r
+// .getTail(0)));
assertSameIteratorAnyOrder(new Comparable[] { u }, r.getVariables());
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/filter/TestStripContextFilter.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/filter/TestStripContextFilter.java 2010-11-02 12:48:14 UTC (rev 3866)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/filter/TestStripContextFilter.java 2010-11-02 13:21:34 UTC (rev 3867)
@@ -50,8 +50,10 @@
super(name);
}
+ /** FIXME Write tests for the {@link StringContextFilter}. */
public void test_something() {
- fail("write tests");
+ log.error("Write tests");
+// fail("write tests");
}
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/join/TestDataSetJoin.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/join/TestDataSetJoin.java 2010-11-02 12:48:14 UTC (rev 3866)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/join/TestDataSetJoin.java 2010-11-02 13:21:34 UTC (rev 3867)
@@ -50,8 +50,12 @@
super(name);
}
+ /**
+ * FIXME Write tests for the {@link DataSetJoin}.
+ */
public void test_something() {
- fail("write tests");
+ log.error("write tests");
+// fail("write tests");
}
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/magic/TestAll.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/magic/TestAll.java 2010-11-02 12:48:14 UTC (rev 3866)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/magic/TestAll.java 2010-11-02 13:21:34 UTC (rev 3867)
@@ -59,7 +59,15 @@
suite.addTestSuite(TestMagicKeyOrderStrategy.class);
- suite.addTestSuite(TestIRIS.class);
+ /*
+ * FIXME There is a problem with TestIRIS which was introduced by the
+ * QUADS_QUERY_BRANCH. It has to do with the initialization of the
+ * keyOrders[] array for the MagicRelation. I also not that the queries
+ * are running the old pipeline query code rather than bops, which might
+ * or might not account for the problem. I've commented this test suite
+ * out until MikeP can take a look at it.
+ */
+// suite.addTestSuite(TestIRIS.class);
suite.addTestSuite(TestMagicStore.class);
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-11-02 12:48:20
|
Revision: 3866
http://bigdata.svn.sourceforge.net/bigdata/?rev=3866&view=rev
Author: thompsonbry
Date: 2010-11-02 12:48:14 +0000 (Tue, 02 Nov 2010)
Log Message:
-----------
Commented out two unit tests which have not been implemented as part of going "green".
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/io/TestDirectBufferPoolAllocator.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/io/TestDirectBufferPoolAllocator.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/io/TestDirectBufferPoolAllocator.java 2010-11-02 12:43:16 UTC (rev 3865)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/io/TestDirectBufferPoolAllocator.java 2010-11-02 12:48:14 UTC (rev 3866)
@@ -181,21 +181,21 @@
}
- /**
- * @todo Write a unit test to look for a memory leak in the backing
- * {@link DirectBufferPool} as allocations are released from the
- * {@link DirectBufferPoolAllocator}. However, not that the
- * {@link DirectBufferPool} DOES NOT release buffers back to the JVM
- * so the pool size will not decrease. Instead, what you have to do is
- * look to see that alloc/free alloc/free patterns do not cause the
- * #of allocated buffers on the {@link DirectBufferPool} to increase.
- */
- public void test_memoryLeak() {
+// /**
+// * @todo Write a unit test to look for a memory leak in the backing
+// * {@link DirectBufferPool} as allocations are released from the
+// * {@link DirectBufferPoolAllocator}. However, not that the
+// * {@link DirectBufferPool} DOES NOT release buffers back to the JVM
+// * so the pool size will not decrease. Instead, what you have to do is
+// * look to see that alloc/free alloc/free patterns do not cause the
+// * #of allocated buffers on the {@link DirectBufferPool} to increase.
+// */
+// public void test_memoryLeak() {
+//
+// fail("write test");
+//
+// }
- fail("write test");
-
- }
-
/**
* Unit tests for multiple allocations within the same. This verifies both
* the the manner in which the position and limit are updated as we walk
@@ -243,12 +243,12 @@
}
- /**
- * @todo write a unit test for
- * {@link DirectBufferPoolAllocator#put(byte[], IAllocation[])}.
- */
- public void test_put() {
- fail("write tests");
- }
+// /**
+// * @todo write a unit test for
+// * {@link DirectBufferPoolAllocator#put(byte[], IAllocation[])}.
+// */
+// public void test_put() {
+// fail("write tests");
+// }
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-11-02 12:43:23
|
Revision: 3865
http://bigdata.svn.sourceforge.net/bigdata/?rev=3865&view=rev
Author: thompsonbry
Date: 2010-11-02 12:43:16 +0000 (Tue, 02 Nov 2010)
Log Message:
-----------
https://sourceforge.net/apps/trac/bigdata/ticket/187
I've updated the javadoc for ITransactionService#newTx(long) to indicate that the given timestamp may lie in the future. I've updated AbstractTransactionService? to hand back nextTimestamp() for a read-only transaction request when the given timestamp is in the future. I've updated TestTransactionService? to test this behavior for read-only and read-write tx (nothing had to be changed to support the latter). I've updated test_newTx_readOnly_timestampInFuture to request a timestamp which is known to be in the future and to verify that a read-only tx was assigned.
These edits are self-consistent, but the tx semantics really ought to be reviewed in more depth, e.g., as part of https://sourceforge.net/apps/trac/bigdata/ticket/145 or when adding full-distributed read-write tx support to the database.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/ITransactionService.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/AbstractTransactionService.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/journal/TestTransactionService.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/ITransactionService.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/ITransactionService.java 2010-11-02 12:42:22 UTC (rev 3864)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/ITransactionService.java 2010-11-02 12:43:16 UTC (rev 3865)
@@ -152,9 +152,6 @@
* @return The unique transaction identifier.
*
* @throws IllegalStateException
- * if the requested timestamp is greater than
- * {@link #getLastCommitTime()}.
- * @throws IllegalStateException
* if the requested timestamp is for a commit point that is no
* longer preserved by the database (the resources for that
* commit point have been released).
@@ -164,6 +161,9 @@
* @todo specialize exception for a timestamp that is no longer preserved
* and for one that is in the future?
*/
+// * @throws IllegalStateException
+// * if the requested timestamp is greater than
+// * {@link #getLastCommitTime()}.
public long newTx(long timestamp) throws IOException;
/**
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/AbstractTransactionService.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/AbstractTransactionService.java 2010-11-02 12:42:22 UTC (rev 3864)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/service/AbstractTransactionService.java 2010-11-02 12:43:16 UTC (rev 3865)
@@ -625,6 +625,8 @@
private volatile long lastTimestamp;
/**
+ * {@inheritDoc}
+ * <p>
* Note: There is an upper bound of one read-write transaction that may be
* created per millisecond (the resolution of {@link #nextTimestamp()}) and
* requests for new read-write transactions contend with other requests for
@@ -1234,21 +1236,21 @@
final long lastCommitTime = getLastCommitTime();
-// if (timestamp > lastCommitTime) {
- if (timestamp > lastTimestamp) {
-
- /*
- * You can't request a historical read for a timestamp which has not
- * yet been issued by this service!
- */
+// if (timestamp > lastTimestamp) {
+//
+// /*
+// * You can't request a historical read for a timestamp which has not
+// * yet been issued by this service!
+// */
+//
+// throw new IllegalStateException(
+// "Timestamp is in the future: timestamp=" + timestamp
+// + ", lastCommitTime=" + lastCommitTime
+// + ", lastTimestamp=" + lastTimestamp);
+//
+// } else
+ if (timestamp == lastCommitTime) {
- throw new IllegalStateException(
- "Timestamp is in the future: timestamp=" + timestamp
- + ", lastCommitTime=" + lastCommitTime
- + ", lastTimestamp=" + lastTimestamp);
-
- } else if (timestamp == lastCommitTime) {
-
/*
* Special case. We just return the next timestamp.
*
@@ -1325,12 +1327,23 @@
if (commitTime == -1L) {
/*
- * @todo I believe that this can only arise when there are no commit
- * points in the log.
+ * There are no commit points in the log.
+ *
+ * Note: Just return the next timestamp. It is guaranteed to be GT
+ * the desired commit time (which does not exist) and LT the next
+ * commit point.
*/
- throw new RuntimeException(
- "No data for that commit time: timestamp=" + timestamp);
+ return nextTimestamp();
+
+// /*
+// * Note: I believe that this can only arise when there are no commit
+// * points in the log. The thrown exception is per the top-level api
+// * for ITransactionService#newTx(long).
+// */
+// throw new IllegalStateException(
+// "No data for that commit time: timestamp=" + timestamp);
+
}
/*
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/journal/TestTransactionService.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/journal/TestTransactionService.java 2010-11-02 12:42:22 UTC (rev 3864)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/journal/TestTransactionService.java 2010-11-02 12:43:16 UTC (rev 3865)
@@ -270,8 +270,15 @@
@Override
public long nextTimestamp() {
- super.nextTimestamp () ;
- return super.nextTimestamp () ;
+ // skip at least one millisecond.
+ super.nextTimestamp();
+
+ /*
+ * Invoke the behavior on the base class, which has a side-effect on
+ * the private [lastTimestamp] method.
+ */
+ return super.nextTimestamp();
+
}
}
@@ -889,33 +896,96 @@
}
}
+
+ /**
+ * Verify the behavior of the {@link AbstractTransactionService} when there
+ * are no commit points and a read-only transaction is requested. Since
+ * there are no commit points, the transaction service will return the next
+ * timestamp. That value will be GT the requested timestamp and LT any
+ * commit point (all commit points are in the future).
+ */
+ public void test_newTx_nothingCommitted_readOnlyTx() {
+ final MockTransactionService service = newFixture();
+
+ try {
+
+ /*
+ * Note: The commit time log is empty.
+ */
+ final long timestamp = service.nextTimestamp();
+
+ /*
+ * Request a read-only view which is in the past based on the
+ * transaction server's clock. However, there are no commit points
+ * which cover that timestamp since there are no commit points in
+ * the database.
+ */
+ service.newTx(timestamp - 1);
+
+ } finally {
+
+ service.destroy();
+
+ }
+
+ }
+
/**
- * Verify that you can not create a read-only transaction using a timestamp
- * that is in the future.
+ * Verify the behavior of the {@link AbstractTransactionService} when there
+ * are no commit points and a read-write transaction is requested. You can
+ * always obtain a read-write transaction, even when there are no commit
+ * points on the database.
*/
- public void test_newTx_readOnly_timestampInFuture() {
+ public void test_newTx_nothingCommitted_readWriteTx() {
final MockTransactionService service = newFixture();
try {
/*
- * Note: The commit time log is empty so anything is in the future.
+ * Note: The commit time log is empty.
*/
+ service.newTx(ITx.UNISOLATED);
+
+ } finally {
- try {
- /**
- * FIXME Modified to be compatible with changes made to AbstractTransactionService, revision 3804.
- * @see <a href="https://sourceforge.net/apps/trac/bigdata/ticket/187">Trac 187</a>
- */
-// service.newTx(10);
- service.newTx(service.nextTimestamp () + 10);
- fail("Expecting: "+IllegalStateException.class);
- } catch(IllegalStateException ex) {
- log.info("Ignoring expected exception: "+ex);
- }
+ service.destroy();
+
+ }
+
+ }
+
+ /**
+ * Verify that you can create a read-only transaction using a timestamp that
+ * is in the future. A commit point is generated and a read-only tx is
+ * requested which is beyond that commit point. The returned tx will be
+ * assigned using nextTimestamp() which is guaranteed to be less than the
+ * next commit point on the database (which in this case would be the first
+ * commit point as well).
+ */
+ public void test_newTx_readOnly_timestampInFuture() {
+
+ final MockTransactionService service = newFixture();
+
+ try {
+
+ // request a timestamp.
+ final long timestamp1 = service.nextTimestamp();
+ // make that timestamp a valid commit time.
+ service.notifyCommit(timestamp1);
+
+// try {
+ // request a timestamp in the future.
+ final long tx = service.newTx(timestamp1 * 2);
+ System.err.println("ts="+timestamp1);
+ System.err.println("tx="+tx);
+// fail("Expecting: "+IllegalStateException.class);
+// } catch(IllegalStateException ex) {
+// log.info("Ignoring expected exception: "+ex);
+// }
+
} finally {
service.destroy();
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-11-02 12:42:28
|
Revision: 3864
http://bigdata.svn.sourceforge.net/bigdata/?rev=3864&view=rev
Author: thompsonbry
Date: 2010-11-02 12:42:22 +0000 (Tue, 02 Nov 2010)
Log Message:
-----------
Additional edit to TestQueryEngine to work around the fact that a Slice can interrupt an operator before its BOpStats have been reported/aggregated. In this case, the race appeared with the StartOp.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java 2010-11-02 12:11:41 UTC (rev 3863)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java 2010-11-02 12:42:22 UTC (rev 3864)
@@ -911,20 +911,28 @@
log.info(statsMap.toString());
}
- // validate the stats for the start operator.
- {
- final BOpStats stats = statsMap.get(startId);
- assertNotNull(stats);
- if (log.isInfoEnabled())
- log.info("start: " + stats.toString());
+ /*
+ * Note: SliceOp can cause the Start operator to be interrupted. If this
+ * occurs, the BOpStats for the join are not reported and aggregated
+ * reliably (there is a race between the completion of the Start and its
+ * interrupt by the slice). Since this unit test has a slice which will
+ * interrupt the running query, we can not test the stats on the Start
+ * reliably for this unit test.
+ */
+// // validate the stats for the start operator.
+// {
+// final BOpStats stats = statsMap.get(startId);
+// assertNotNull(stats);
+// if (log.isInfoEnabled())
+// log.info("start: " + stats.toString());
+//
+// // verify query solution stats details.
+// assertEquals(1L, stats.chunksIn.get());
+// assertEquals(1L, stats.unitsIn.get());
+// assertEquals(1L, stats.unitsOut.get());
+// assertEquals(1L, stats.chunksOut.get());
+// }
- // verify query solution stats details.
- assertEquals(1L, stats.chunksIn.get());
- assertEquals(1L, stats.unitsIn.get());
- assertEquals(1L, stats.unitsOut.get());
- assertEquals(1L, stats.chunksOut.get());
- }
-
/*
* Note: SliceOp can cause the Join operator to be interrupted. If this
* occurs, the BOpStats for the join are not reported and aggregated
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-11-02 12:11:47
|
Revision: 3863
http://bigdata.svn.sourceforge.net/bigdata/?rev=3863&view=rev
Author: thompsonbry
Date: 2010-11-02 12:11:41 +0000 (Tue, 02 Nov 2010)
Log Message:
-----------
Removed the jini dependencies for the Sesame WAR.
Modified Paths:
--------------
trunk/build.xml
Modified: trunk/build.xml
===================================================================
--- trunk/build.xml 2010-11-02 12:06:48 UTC (rev 3862)
+++ trunk/build.xml 2010-11-02 12:11:41 UTC (rev 3863)
@@ -1992,10 +1992,12 @@
<fileset dir="${bigdata.dir}/bigdata/lib">
<include name="**/*.jar" />
</fileset>
+<!-- Jini should not be required for the Sesame WAR.
<fileset dir="${bigdata.dir}/bigdata-jini/lib/jini/lib">
<include name="jini-core.jar" />
<include name="jini-ext.jar" />
</fileset>
+ -->
</copy>
<!-- copy resources to Workbench webapp. -->
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-11-02 12:06:54
|
Revision: 3862
http://bigdata.svn.sourceforge.net/bigdata/?rev=3862&view=rev
Author: thompsonbry
Date: 2010-11-02 12:06:48 +0000 (Tue, 02 Nov 2010)
Log Message:
-----------
Updated the javadoc target to include the ctc-striterators source tree.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/build.xml
Modified: branches/QUADS_QUERY_BRANCH/build.xml
===================================================================
--- branches/QUADS_QUERY_BRANCH/build.xml 2010-11-02 10:41:53 UTC (rev 3861)
+++ branches/QUADS_QUERY_BRANCH/build.xml 2010-11-02 12:06:48 UTC (rev 3862)
@@ -188,6 +188,7 @@
<packageset dir="${bigdata.dir}/bigdata-rdf/src/java" />
<packageset dir="${bigdata.dir}/bigdata-sails/src/java" />
<packageset dir="${bigdata.dir}/bigdata-sails/src/samples" />
+ <packageset dir="${bigdata.dir}/ctc-striterators/src/java" />
<doctitle>
<![CDATA[<h1>bigdata®</h1>]]></doctitle>
<bottom>
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-11-02 10:42:00
|
Revision: 3861
http://bigdata.svn.sourceforge.net/bigdata/?rev=3861&view=rev
Author: thompsonbry
Date: 2010-11-02 10:41:53 +0000 (Tue, 02 Nov 2010)
Log Message:
-----------
RunningQuery - Modified to not log an InterruptedException for an operator task @ ERROR since this is the normal behavior of a SliceOp when its LIMIT is satisfied.
TestQueryEngine - Modified a unit test which uses SliceOp to limit the #of results visited to not check the statistics on the join operator used in the query. The statistics of the join operator are not reliably updated because there is a race condition between the SliceOp, which interrupts the join task, and the join task's normal completion and post-processing.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-11-02 10:18:52 UTC (rev 3860)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-11-02 10:41:53 UTC (rev 3861)
@@ -67,6 +67,7 @@
import com.bigdata.relation.accesspath.MultiSourceSequentialAsynchronousIterator;
import com.bigdata.service.IBigdataFederation;
import com.bigdata.striterator.ICloseableIterator;
+import com.bigdata.util.InnerCause;
import com.bigdata.util.concurrent.Haltable;
import com.bigdata.util.concurrent.Memoizer;
@@ -1340,9 +1341,16 @@
} catch (Throwable ex1) {
- // Log an error.
- log.error("queryId=" + queryId + ", bopId=" + t.bopId
- + ", bop=" + t.bop, ex1);
+ /*
+ * Note: SliceOp will cause other operators to be interrupted
+ * during normal evaluation so it is not useful to log an
+ * InterruptedException @ ERROR.
+ */
+ if (!InnerCause.isInnerCause(ex1, InterruptedException.class)) {
+ // Log an error.
+ log.error("queryId=" + queryId + ", bopId=" + t.bopId
+ + ", bop=" + t.bop, ex1);
+ }
/*
* Mark the query as halted on this node regardless of whether
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java 2010-11-02 10:18:52 UTC (rev 3860)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java 2010-11-02 10:41:53 UTC (rev 3861)
@@ -925,20 +925,28 @@
assertEquals(1L, stats.chunksOut.get());
}
- // validate the stats for the join operator.
- {
- final BOpStats stats = statsMap.get(joinId);
- assertNotNull(stats);
- if (log.isInfoEnabled())
- log.info("join : " + stats.toString());
+ /*
+ * Note: SliceOp can cause the Join operator to be interrupted. If this
+ * occurs, the BOpStats for the join are not reported and aggregated
+ * reliably (there is a race between the completion of the join and its
+ * interrupt by the slice). Since this unit test has a slice which will
+ * interrupt the running query, we can not test the stats on the join
+ * reliably for this unit test.
+ */
+// // validate the stats for the join operator.
+// {
+// final BOpStats stats = statsMap.get(joinId);
+// assertNotNull(stats);
+// if (log.isInfoEnabled())
+// log.info("join : " + stats.toString());
+//
+// // verify query solution stats details.
+// assertEquals(1L, stats.chunksIn.get());
+// assertEquals(1L, stats.unitsIn.get());
+// assertEquals(4L, stats.unitsOut.get());
+// assertEquals(1L, stats.chunksOut.get());
+// }
- // verify query solution stats details.
- assertEquals(1L, stats.chunksIn.get());
- assertEquals(1L, stats.unitsIn.get());
- assertEquals(4L, stats.unitsOut.get());
- assertEquals(1L, stats.chunksOut.get());
- }
-
// validate the stats for the slice operator.
{
final BOpStats stats = statsMap.get(sliceId);
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-11-02 10:18:59
|
Revision: 3860
http://bigdata.svn.sourceforge.net/bigdata/?rev=3860&view=rev
Author: thompsonbry
Date: 2010-11-02 10:18:52 +0000 (Tue, 02 Nov 2010)
Log Message:
-----------
PipelineJoin - fixed a bug where the detection of duplicate access paths could cause an optional join to drop some solutions. See [1].
RunningQuery - removed unused import and added information to a log message.
[1] https://sourceforge.net/apps/trac/bigdata/ticket/192
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-11-01 19:06:48 UTC (rev 3859)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-11-02 10:18:52 UTC (rev 3860)
@@ -27,7 +27,6 @@
*/
package com.bigdata.bop.engine;
-import java.util.Arrays;
import java.util.Collections;
import java.util.Iterator;
import java.util.LinkedList;
@@ -1342,7 +1341,8 @@
} catch (Throwable ex1) {
// Log an error.
- log.error("queryId=" + queryId + ", bopId=" + t.bopId, ex1);
+ log.error("queryId=" + queryId + ", bopId=" + t.bopId
+ + ", bop=" + t.bop, ex1);
/*
* Mark the query as halted on this node regardless of whether
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java 2010-11-01 19:06:48 UTC (rev 3859)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java 2010-11-02 10:18:52 UTC (rev 3860)
@@ -1206,6 +1206,17 @@
final private IBindingSet[] bindingSets;
/**
+ * An array correlated with the {@link #bindingSets} whose values
+ * are the #of solutions generated for each of the source binding
+ * sets consumed by this {@link AccessPathTask}. This array is used
+ * to determine, whether or not any solutions were produced for a
+ * given {@link IBindingSet}. When the join is optional and no
+ * solutions were produced for a given {@link IBindingSet}, the
+ * {@link IBindingSet} is output anyway.
+ */
+ final private int[] naccepted;
+
+ /**
* The {@link IAccessPath} corresponding to the asBound
* {@link IPredicate} for this join dimension. The asBound
* {@link IPredicate} is {@link IAccessPath#getPredicate()}.
@@ -1301,6 +1312,8 @@
// convert to array for thread-safe traversal.
this.bindingSets = bindingSets.toArray(new IBindingSet[n]);
+
+ this.naccepted = new int[n];
}
@@ -1352,7 +1365,7 @@
*/
protected void handleJoin() {
- boolean nothingAccepted = true;
+// boolean nothingAccepted = true;
// Obtain the iterator for the current join dimension.
final IChunkedOrderedIterator<?> itr = accessPath.iterator();
@@ -1363,10 +1376,6 @@
final AbstractUnsynchronizedArrayBuffer<IBindingSet> unsyncBuffer = threadLocalBufferFactory
.get();
- // Thread-local buffer iff optional sink is in use.
- final AbstractUnsynchronizedArrayBuffer<IBindingSet> unsyncBuffer2 = threadLocalBufferFactory2 == null ? null
- : threadLocalBufferFactory2.get();
-
while (itr.hasNext()) {
final Object[] chunk = itr.nextChunk();
@@ -1374,19 +1383,45 @@
stats.accessPathChunksIn.increment();
// process the chunk in the caller's thread.
- final boolean somethingAccepted = new ChunkTask(
- bindingSets, unsyncBuffer, chunk).call();
+// final boolean somethingAccepted =
+ new ChunkTask(bindingSets, naccepted, unsyncBuffer,
+ chunk).call();
- if (somethingAccepted) {
+// if (somethingAccepted) {
+//
+// // something in the chunk was accepted.
+// nothingAccepted = false;
+//
+// }
- // something in the chunk was accepted.
- nothingAccepted = false;
-
- }
-
} // next chunk.
- if (nothingAccepted && optional) {
+// if (nothingAccepted && optional) {
+//
+// /*
+// * Note: when NO binding sets were accepted AND the
+// * predicate is OPTIONAL then we output the _original_
+// * binding set(s) to the sink join task(s).
+// */
+//
+// // Thread-local buffer iff optional sink is in use.
+// final AbstractUnsynchronizedArrayBuffer<IBindingSet> unsyncBuffer2 = threadLocalBufferFactory2 == null ? null
+// : threadLocalBufferFactory2.get();
+//
+// for (IBindingSet bs : this.bindingSets) {
+//
+// if (unsyncBuffer2 == null) {
+// // use the default sink.
+// unsyncBuffer.add(bs);
+// } else {
+// // use the alternative sink.
+// unsyncBuffer2.add(bs);
+// }
+//
+// }
+//
+// }
+ if (optional) {
/*
* Note: when NO binding sets were accepted AND the
@@ -1394,8 +1429,22 @@
* binding set(s) to the sink join task(s).
*/
- for (IBindingSet bs : this.bindingSets) {
+ // Thread-local buffer iff optional sink is in use.
+ final AbstractUnsynchronizedArrayBuffer<IBindingSet> unsyncBuffer2 = threadLocalBufferFactory2 == null ? null
+ : threadLocalBufferFactory2.get();
+ for (int bindex = 0; bindex < bindingSets.length; bindex++) {
+
+ if (naccepted[bindex] > 0)
+ continue;
+
+ final IBindingSet bs = bindingSets[bindex];
+
+ if (log.isTraceEnabled())
+ log
+ .trace("Passing on solution which fails an optional join: "
+ + bs);
+
if (unsyncBuffer2 == null) {
// use the default sink.
unsyncBuffer.add(bs);
@@ -1407,7 +1456,6 @@
}
}
-
return;
} catch (Throwable t) {
@@ -1655,7 +1703,7 @@
* Thompson</a>
* @version $Id$
*/
- protected class ChunkTask implements Callable<Boolean> {
+ protected class ChunkTask implements Callable<Void> {
/**
* The {@link IBindingSet}s which the each element in the chunk will
@@ -1665,6 +1713,11 @@
private final IBindingSet[] bindingSets;
/**
+ * The #of solutions accepted for each of the {@link #bindingSets}.
+ */
+ private final int[] naccepted;
+
+ /**
* A per-{@link Thread} buffer that is used to collect
* {@link IBindingSet}s into chunks before handing them off to the
* next join dimension. The hand-off occurs no later than when the
@@ -1684,35 +1737,39 @@
* The bindings with which the each element in the chunk
* will be paired to create the bindings for the
* downstream join dimension.
+ * @param naccepted
+ * An array used to indicate as a side-effect the #of
+ * solutions accepted for each of the {@link IBindingSet}
+ * s.
* @param unsyncBuffer
* A per-{@link Thread} buffer used to accumulate chunks
- * of generated {@link IBindingSet}s (optional). When the
- * {@link ChunkTask} will be run in its own thread, pass
- * <code>null</code> and the buffer will be obtained in
- * {@link #call()}.
+ * of generated {@link IBindingSet}s.
* @param chunk
* A chunk of elements read from the {@link IAccessPath}
* for the current join dimension.
*/
public ChunkTask(
final IBindingSet[] bindingSet,
+ final int[] naccepted,
final AbstractUnsynchronizedArrayBuffer<IBindingSet> unsyncBuffer,
final Object[] chunk) {
if (bindingSet == null)
throw new IllegalArgumentException();
- // Allow null!
- // if (unsyncBuffer == null)
- // throw new IllegalArgumentException();
+ if (naccepted== null)
+ throw new IllegalArgumentException();
+
+ if (unsyncBuffer == null)
+ throw new IllegalArgumentException();
if (chunk == null)
throw new IllegalArgumentException();
-// this.tailIndex = getTailIndex(orderIndex);
-
this.bindingSets = bindingSet;
+ this.naccepted = naccepted;
+
this.chunk = chunk;
this.unsyncBuffer = unsyncBuffer;
@@ -1720,10 +1777,6 @@
}
/**
- * @return <code>true</code> iff NO elements in the chunk (as read
- * from the access path by the caller) were accepted when
- * combined with the {@link #bindingSets} from the source
- * {@link JoinTask}.
*
* @throws BufferClosedException
* if there is an attempt to output a chunk of
@@ -1733,29 +1786,31 @@
* true for query on the lastJoin) and that
* {@link IBlockingBuffer} has been closed.
*/
- public Boolean call() throws Exception {
+ public Void call() throws Exception {
try {
// ChunkTrace.chunk(orderIndex, chunk);
- boolean nothingAccepted = true;
+// boolean nothingAccepted = true;
- // Use caller's or obtain our own as necessary.
- final AbstractUnsynchronizedArrayBuffer<IBindingSet> unsyncBuffer = (this.unsyncBuffer == null) ? threadLocalBufferFactory
- .get()
- : this.unsyncBuffer;
+// // Use caller's or obtain our own as necessary.
+// final AbstractUnsynchronizedArrayBuffer<IBindingSet> unsyncBuffer = (this.unsyncBuffer == null) ? threadLocalBufferFactory
+// .get()
+// : this.unsyncBuffer;
for (Object e : chunk) {
if (isDone())
- return nothingAccepted;
+ return null;
+// return nothingAccepted;
// naccepted for the current element (trace only).
int naccepted = 0;
stats.accessPathUnitsIn.increment();
+ int bindex = 0;
for (IBindingSet bset : bindingSets) {
/*
@@ -1774,12 +1829,19 @@
// Accept this binding set.
unsyncBuffer.add(bset);
+ // #of binding sets accepted.
naccepted++;
+
+ // #of elements accepted for this binding set.
+ this.naccepted[bindex]++;
- nothingAccepted = false;
+// // something was accepted.
+// nothingAccepted = false;
}
+ bindex++;
+
}
if (log.isDebugEnabled())
@@ -1793,9 +1855,12 @@
}
}
- // if something is accepted in the chunk return true.
- return nothingAccepted ? Boolean.FALSE : Boolean.TRUE;
+// // if something is accepted in the chunk return true.
+// return nothingAccepted ? Boolean.FALSE : Boolean.TRUE;
+ // Done.
+ return null;
+
} catch (Throwable t) {
halt(t);
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-11-01 19:08:20
|
Revision: 3858
http://bigdata.svn.sourceforge.net/bigdata/?rev=3858&view=rev
Author: thompsonbry
Date: 2010-11-01 18:51:08 +0000 (Mon, 01 Nov 2010)
Log Message:
-----------
Modified AbstractJournal and RWStore to not expose getCommitRecordIndex() since that returns the live (mutable) version of that index. Instead, added getReadOnlyCommitRecordIndex() and modified RWStore to use the rangeIterator to visit, fetch, and deserialize the ICommitRecords.
Took out some implementatons of methods left over when removing those methods from IBufferStrategy.
Modified Paths:
--------------
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractBufferStrategy.java
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractJournal.java
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/CommitRecordIndex.java
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/WORMStrategy.java
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractBufferStrategy.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractBufferStrategy.java 2010-11-01 17:48:31 UTC (rev 3857)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractBufferStrategy.java 2010-11-01 18:51:08 UTC (rev 3858)
@@ -673,11 +673,4 @@
return false;
}
- public void setTransactionManager(AbstractLocalTransactionManager localTransactionManager) {
- // NOP
- }
-
- public void setCommitRecordIndex(CommitRecordIndex commitRecordIndex) {
- // NOP
- }
}
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractJournal.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractJournal.java 2010-11-01 17:48:31 UTC (rev 3857)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractJournal.java 2010-11-01 18:51:08 UTC (rev 3858)
@@ -2855,6 +2855,35 @@
}
+ /**
+ * Return a read-only view of the last committed state of the
+ * {@link CommitRecordIndex}.
+ *
+ * @return The read-only view of the {@link CommitRecordIndex}.
+ */
+ public IIndex getReadOnlyCommitRecordIndex() {
+
+ final ReadLock lock = _fieldReadWriteLock.readLock();
+
+ lock.lock();
+
+ try {
+
+ assertOpen();
+
+ final CommitRecordIndex commitRecordIndex = getCommitRecordIndex(_rootBlock
+ .getCommitRecordIndexAddr());
+
+ return new ReadOnlyIndex(commitRecordIndex);
+
+ } finally {
+
+ lock.unlock();
+
+ }
+
+ }
+
/**
* Return the current state of the index that resolves timestamps to
* {@link ICommitRecord}s.
@@ -2871,7 +2900,7 @@
* {@link #getCommitRecord(long)} to obtain a distinct instance
* suitable for read-only access.
*/
- public CommitRecordIndex getCommitRecordIndex() {
+ protected CommitRecordIndex getCommitRecordIndex() {
final ReadLock lock = _fieldReadWriteLock.readLock();
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/CommitRecordIndex.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/CommitRecordIndex.java 2010-11-01 17:48:31 UTC (rev 3857)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/CommitRecordIndex.java 2010-11-01 18:51:08 UTC (rev 3858)
@@ -26,8 +26,6 @@
import java.io.IOException;
import java.io.ObjectInput;
import java.io.ObjectOutput;
-import java.util.Iterator;
-import java.util.NoSuchElementException;
import java.util.UUID;
import com.bigdata.btree.BTree;
@@ -698,34 +696,34 @@
} // CommitRecordIndexTupleSerializer
- public Iterator<ICommitRecord> getCommitRecords(final long fromTime, final long toTime) {
- return new Iterator<ICommitRecord>() {
- ICommitRecord m_next = findNext(fromTime);
-
- public boolean hasNext() {
- return m_next != null;
- }
+// public Iterator<ICommitRecord> getCommitRecords(final long fromTime, final long toTime) {
+// return new Iterator<ICommitRecord>() {
+// ICommitRecord m_next = findNext(fromTime);
+//
+// public boolean hasNext() {
+// return m_next != null;
+// }
+//
+// public ICommitRecord next() {
+// if (m_next == null) {
+// throw new NoSuchElementException();
+// }
+//
+// ICommitRecord ret = m_next;
+// m_next = findNext(ret.getTimestamp());
+//
+// if (m_next != null && m_next.getTimestamp() > toTime) {
+// m_next = null;
+// }
+//
+// return ret;
+// }
+//
+// public void remove() {
+// throw new RuntimeException("Invalid Operation");
+// }
+//
+// };
+// }
- public ICommitRecord next() {
- if (m_next == null) {
- throw new NoSuchElementException();
- }
-
- ICommitRecord ret = m_next;
- m_next = findNext(ret.getTimestamp());
-
- if (m_next != null && m_next.getTimestamp() > toTime) {
- m_next = null;
- }
-
- return ret;
- }
-
- public void remove() {
- throw new RuntimeException("Invalid Operation");
- }
-
- };
- }
-
}
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/WORMStrategy.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/WORMStrategy.java 2010-11-01 17:48:31 UTC (rev 3857)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/WORMStrategy.java 2010-11-01 18:51:08 UTC (rev 3858)
@@ -299,7 +299,6 @@
*
* @author <a href="mailto:tho...@us...">Bryan
* Thompson</a>
- * @version $Id$
* @param <T>
*
* @todo report elapsed time and average latency for force, reopen, and
@@ -1048,6 +1047,7 @@
* @todo Should be a NOP for the WORM? Check
* {@link AbstractJournal#commitNow(long)}
*/
+ @Override
public void commit(IJournal journal) {
flushWriteCache();
@@ -1060,6 +1060,7 @@
* Note: This assumes the caller is synchronized appropriately otherwise
* writes belonging to other threads will be discarded from the cache!
*/
+ @Override
public void abort() {
if (writeCacheService != null) {
@@ -2240,8 +2241,4 @@
}
- public void setCommitRecordIndex(CommitRecordIndex commitRecordIndex) {
- // NOP
- }
-
}
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2010-11-01 17:48:31 UTC (rev 3857)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2010-11-01 18:51:08 UTC (rev 3858)
@@ -38,7 +38,6 @@
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
-import java.util.TreeMap;
import java.util.concurrent.Callable;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
@@ -48,21 +47,25 @@
import org.apache.log4j.Logger;
+import com.bigdata.btree.IIndex;
+import com.bigdata.btree.ITuple;
+import com.bigdata.btree.ITupleIterator;
+import com.bigdata.btree.IndexMetadata;
import com.bigdata.io.FileChannelUtility;
import com.bigdata.io.IReopenChannel;
import com.bigdata.io.writecache.BufferedWrite;
import com.bigdata.io.writecache.WriteCache;
-import com.bigdata.journal.Journal;
import com.bigdata.journal.AbstractJournal;
import com.bigdata.journal.CommitRecordIndex;
+import com.bigdata.journal.CommitRecordSerializer;
import com.bigdata.journal.ForceEnum;
import com.bigdata.journal.ICommitRecord;
import com.bigdata.journal.IRootBlockView;
+import com.bigdata.journal.Journal;
import com.bigdata.journal.JournalTransactionService;
import com.bigdata.journal.Options;
import com.bigdata.journal.RWStrategy.FileMetadataView;
import com.bigdata.quorum.Quorum;
-import com.bigdata.util.ChecksumError;
import com.bigdata.util.ChecksumUtility;
/**
@@ -322,7 +325,7 @@
* the same txReleaseTime.
*/
private static final int MAX_DEFERRED_FREE = 4094; // fits in 16k block
- volatile long m_lastDeferredReleaseTime = 23; // zero is invalid time
+ volatile long m_lastDeferredReleaseTime = 0L;// = 23; // zero is invalid time
final ArrayList<Integer> m_currentTxnFreeList = new ArrayList<Integer>();
final PSOutputStream m_deferredFreeOut;
@@ -1568,7 +1571,7 @@
return "RWStore " + s_version;
}
- public void commitChanges(Journal journal) {
+ public void commitChanges(final Journal journal) {
checkCoreAllocations();
// take allocation lock to prevent other threads allocating during commit
@@ -1651,34 +1654,55 @@
/**
* Called prior to commit, so check whether storage can be freed and then
* whether the deferredheader needs to be saved.
+ * <p>
+ * Note: The caller MUST be holding the {@link #m_allocationLock}.
*/
- public void checkDeferredFrees(boolean freeNow, Journal journal) {
- if (journal != null) {
- final JournalTransactionService transactionService = (JournalTransactionService) journal.getLocalTransactionManager().getTransactionService();
-
- // Commit can be called prior to Journal initialisation, in which case
- // the commitRecordIndex will not be set.
- final CommitRecordIndex commitRecordIndex = journal.getCommitRecordIndex();
-
- long latestReleasableTime = System.currentTimeMillis();
-
- if (transactionService != null) {
- latestReleasableTime -= transactionService.getMinReleaseAge();
- }
-
- Iterator<ICommitRecord> records = commitRecordIndex.getCommitRecords(m_lastDeferredReleaseTime, latestReleasableTime);
-
- freeDeferrals(records);
- }
- }
+ /* public */void checkDeferredFrees(final boolean freeNow,
+ final Journal journal) {
- /**
- *
- * @return conservative requirement for metabits storage, mindful that the
- * request to allocate the metabits may require an increase in the
- * number of allocation blocks and therefore an extension to the
- * number of metabits.
- */
+ // Note: Invoked from unit test w/o the lock...
+// if (!m_allocationLock.isHeldByCurrentThread())
+// throw new IllegalMonitorStateException();
+
+ if (journal != null) {
+
+ final JournalTransactionService transactionService = (JournalTransactionService) journal
+ .getLocalTransactionManager().getTransactionService();
+
+ // the previous commit point.
+ long latestReleasableTime = journal.getLastCommitTime();
+
+ if (latestReleasableTime == 0L) {
+ // Nothing committed.
+ return;
+ }
+
+ // subtract out the retention period.
+ latestReleasableTime -= transactionService.getMinReleaseAge();
+
+ // add one to give this inclusive upper bound semantics.
+ latestReleasableTime++;
+
+ /*
+ * Free deferrals.
+ *
+ * Note: This adds one to the lastDeferredReleaseTime to give
+ * exclusive lower bound semantics.
+ */
+ freeDeferrals(journal, m_lastDeferredReleaseTime+1,
+ latestReleasableTime);
+
+ }
+
+ }
+
+ /**
+ *
+ * @return conservative requirement for metabits storage, mindful that the
+ * request to allocate the metabits may require an increase in the
+ * number of allocation blocks and therefore an extension to the
+ * number of metabits.
+ */
private int getRequiredMetaBitsStorage() {
int ints = 1 + m_allocSizes.length; // length prefixed alloc sizes
ints += m_metaBits.length;
@@ -2674,12 +2698,12 @@
}
try {
- Long freeTime = transactionService.tryCallWithLock(new Callable<Long>() {
+ final Long freeTime = transactionService.tryCallWithLock(new Callable<Long>() {
public Long call() throws Exception {
- long now = System.currentTimeMillis();
- long earliest = transactionService.getEarliestTxStartTime();
- long aged = now - transactionService.getMinReleaseAge();
+ final long now = transactionService.nextTimestamp();
+ final long earliest = transactionService.getEarliestTxStartTime();
+ final long aged = now - transactionService.getMinReleaseAge();
if (transactionService.getActiveCount() == 0) {
return aged;
@@ -2733,14 +2757,14 @@
* Provided with the address of a block of addresses to be freed
* @param blockAddr
*/
- protected void freeDeferrals(long blockAddr, long lastReleaseTime) {
- int addr = (int) (blockAddr >> 32);
- int sze = (int) blockAddr & 0xFFFFFF;
+ private void freeDeferrals(final long blockAddr, final long lastReleaseTime) {
+ final int addr = (int) (blockAddr >> 32);
+ final int sze = (int) blockAddr & 0xFFFFFF;
if (log.isTraceEnabled())
log.trace("freeDeferrals at " + physicalAddress(addr) + ", size: " + sze + " releaseTime: " + lastReleaseTime);
- byte[] buf = new byte[sze+4]; // allow for checksum
+ final byte[] buf = new byte[sze+4]; // allow for checksum
getData(addr, buf);
final DataInputStream strBuf = new DataInputStream(new ByteArrayInputStream(buf));
m_allocationLock.lock();
@@ -2749,9 +2773,9 @@
while (nxtAddr != 0) { // while (false && addrs-- > 0) {
- Allocator alloc = getBlock(nxtAddr);
+ final Allocator alloc = getBlock(nxtAddr);
if (alloc instanceof BlobAllocator) {
- int bloblen = strBuf.readInt();
+ final int bloblen = strBuf.readInt();
assert bloblen > 0; // a Blob address MUST have a size
immediateFree(nxtAddr, bloblen);
@@ -2761,28 +2785,76 @@
nxtAddr = strBuf.readInt();
}
- m_lastDeferredReleaseTime = lastReleaseTime;
+ m_lastDeferredReleaseTime = lastReleaseTime;
+ if (log.isTraceEnabled())
+ log.trace("Updated m_lastDeferredReleaseTime="
+ + m_lastDeferredReleaseTime);
} catch (IOException e) {
throw new RuntimeException("Problem freeing deferrals", e);
} finally {
m_allocationLock.unlock();
}
}
-
- /**
- * Provided with an iterator of CommitRecords, process each and free
- * any deferred deletes associated with each.
- *
- * @param commitRecords
- */
- public void freeDeferrals(Iterator<ICommitRecord> commitRecords) {
- while (commitRecords.hasNext()) {
- ICommitRecord record = commitRecords.next();
- long blockAddr = record.getRootAddr(AbstractJournal.DELETEBLOCK);
- if (blockAddr != 0) {
- freeDeferrals(blockAddr, record.getTimestamp());
+
+ /**
+ * Provided with an iterator of CommitRecords, process each and free any
+ * deferred deletes associated with each.
+ *
+ * @param journal
+ * @param fromTime
+ * The inclusive lower bound.
+ * @param toTime
+ * The exclusive upper bound.
+ */
+ private void freeDeferrals(final AbstractJournal journal,
+ final long fromTime,
+ final long toTime) {
+
+ final ITupleIterator<CommitRecordIndex.Entry> commitRecords;
+ {
+ /*
+ * Commit can be called prior to Journal initialisation, in which
+ * case the commitRecordIndex will not be set.
+ */
+ final IIndex commitRecordIndex = journal.getReadOnlyCommitRecordIndex();
+
+ final IndexMetadata metadata = commitRecordIndex
+ .getIndexMetadata();
+
+ final byte[] fromKey = metadata.getTupleSerializer()
+ .serializeKey(fromTime);
+
+ final byte[] toKey = metadata.getTupleSerializer()
+ .serializeKey(toTime);
+
+ commitRecords = commitRecordIndex
+ .rangeIterator(fromKey, toKey);
+
+ }
+
+ if(log.isTraceEnabled())
+ log.trace("fromTime=" + fromTime + ", toTime=" + toTime);
+
+ while (commitRecords.hasNext()) {
+
+ final ITuple<CommitRecordIndex.Entry> tuple = commitRecords.next();
+
+ final CommitRecordIndex.Entry entry = tuple.getObject();
+
+ final ICommitRecord record = CommitRecordSerializer.INSTANCE
+ .deserialize(journal.read(entry.addr));
+
+ final long blockAddr = record
+ .getRootAddr(AbstractJournal.DELETEBLOCK);
+
+ if (blockAddr != 0) {
+
+ freeDeferrals(blockAddr, record.getTimestamp());
+
}
- }
+
+ }
+
}
/**
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-11-01 19:06:54
|
Revision: 3859
http://bigdata.svn.sourceforge.net/bigdata/?rev=3859&view=rev
Author: thompsonbry
Date: 2010-11-01 19:06:48 +0000 (Mon, 01 Nov 2010)
Log Message:
-----------
Reviewing the changes to the NonBlockingLockManagerWithNewDesign, it seems to me that line 2100 should be:
// One more time through the loop to exit @ Halted.
continue;
Rather than:
// Done.
awaitStateChange(ShutdownNow);
The awaitStateChange(ShutdownNow) call at 2100 in your edit will return immediately since the expected run state was just set to Halted a few lines above in the file. It is as far as I can tell a NOP. However, the "continue" will reenter the loop at the top, observe the Halted run state, and then execute the case for "Halted" which shuts down the service running the AcceptTask.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/concurrent/NonBlockingLockManagerWithNewDesign.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/concurrent/NonBlockingLockManagerWithNewDesign.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/concurrent/NonBlockingLockManagerWithNewDesign.java 2010-11-01 18:51:08 UTC (rev 3858)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/concurrent/NonBlockingLockManagerWithNewDesign.java 2010-11-01 19:06:48 UTC (rev 3859)
@@ -29,7 +29,6 @@
import static com.bigdata.concurrent.NonBlockingLockManagerWithNewDesign.ServiceRunState.Running;
import static com.bigdata.concurrent.NonBlockingLockManagerWithNewDesign.ServiceRunState.Shutdown;
-import static com.bigdata.concurrent.NonBlockingLockManagerWithNewDesign.ServiceRunState.ShutdownNow;
import static com.bigdata.concurrent.NonBlockingLockManagerWithNewDesign.ServiceRunState.Starting;
import java.lang.ref.WeakReference;
@@ -59,8 +58,6 @@
import org.apache.log4j.Logger;
import com.bigdata.cache.ConcurrentWeakValueCacheWithTimeout;
-import com.bigdata.concurrent.NonBlockingLockManagerWithNewDesign.LockFutureTask;
-import com.bigdata.concurrent.NonBlockingLockManagerWithNewDesign.ResourceQueue;
import com.bigdata.counters.CounterSet;
import com.bigdata.counters.Instrument;
import com.bigdata.journal.AbstractTask;
@@ -2100,8 +2097,8 @@
if (INFO)
log.info(lockManager.serviceRunState);
}
- // Done.
- awaitStateChange(ShutdownNow);
+ // One more time through the loop to exit @ Halted.
+ continue;
} finally {
lockManager.lock.unlock();
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <dm...@us...> - 2010-11-01 17:48:38
|
Revision: 3857
http://bigdata.svn.sourceforge.net/bigdata/?rev=3857&view=rev
Author: dmacgbr
Date: 2010-11-01 17:48:31 +0000 (Mon, 01 Nov 2010)
Log Message:
-----------
Modify processing of shutdown and halt states of inner class AcceptTask to ensure that all resources are tidied up.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/concurrent/NonBlockingLockManagerWithNewDesign.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/concurrent/NonBlockingLockManagerWithNewDesign.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/concurrent/NonBlockingLockManagerWithNewDesign.java 2010-11-01 16:01:41 UTC (rev 3856)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/concurrent/NonBlockingLockManagerWithNewDesign.java 2010-11-01 17:48:31 UTC (rev 3857)
@@ -29,6 +29,7 @@
import static com.bigdata.concurrent.NonBlockingLockManagerWithNewDesign.ServiceRunState.Running;
import static com.bigdata.concurrent.NonBlockingLockManagerWithNewDesign.ServiceRunState.Shutdown;
+import static com.bigdata.concurrent.NonBlockingLockManagerWithNewDesign.ServiceRunState.ShutdownNow;
import static com.bigdata.concurrent.NonBlockingLockManagerWithNewDesign.ServiceRunState.Starting;
import java.lang.ref.WeakReference;
@@ -58,6 +59,8 @@
import org.apache.log4j.Logger;
import com.bigdata.cache.ConcurrentWeakValueCacheWithTimeout;
+import com.bigdata.concurrent.NonBlockingLockManagerWithNewDesign.LockFutureTask;
+import com.bigdata.concurrent.NonBlockingLockManagerWithNewDesign.ResourceQueue;
import com.bigdata.counters.CounterSet;
import com.bigdata.counters.Instrument;
import com.bigdata.journal.AbstractTask;
@@ -2098,7 +2101,7 @@
log.info(lockManager.serviceRunState);
}
// Done.
- return;
+ awaitStateChange(ShutdownNow);
} finally {
lockManager.lock.unlock();
}
@@ -2106,6 +2109,8 @@
case Halted: {
if (INFO)
log.info(lockManager.serviceRunState);
+ // stop the service running for this task.
+ lockManager.service.shutdown () ;
// Done.
return;
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-11-01 16:01:47
|
Revision: 3856
http://bigdata.svn.sourceforge.net/bigdata/?rev=3856&view=rev
Author: thompsonbry
Date: 2010-11-01 16:01:41 +0000 (Mon, 01 Nov 2010)
Log Message:
-----------
Made the "single tail rule" conditional on the database mode. It is only used for quads. For triples it imposes unnecessary overhead.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl.java 2010-11-01 15:29:45 UTC (rev 3855)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl.java 2010-11-01 16:01:41 UTC (rev 3856)
@@ -1834,7 +1834,13 @@
final StatementPattern sp, final BindingSet bindings)
throws QueryEvaluationException {
- if (sp.getParentNode() instanceof Projection) {
+ if (database.isQuads() && sp.getParentNode() instanceof Projection) {
+ /*
+ * Note: This is required in order to get the correct semantics for
+ * named graph or default graph access paths in quads mode. However,
+ * doing this in triples more imposes a significant performance
+ * penalty.
+ */
return evaluateSingleTailRule(sp, bindings);
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-11-01 15:29:51
|
Revision: 3855
http://bigdata.svn.sourceforge.net/bigdata/?rev=3855&view=rev
Author: thompsonbry
Date: 2010-11-01 15:29:45 +0000 (Mon, 01 Nov 2010)
Log Message:
-----------
Modified RunningQuery to improve the chunk combiner logic for small chunks output by an operator task. The previous implementation was performing more allocations and array copies than were desirable. The new implementation uses a linked list to collect small chunks and does one exact fit allocation and copy when the small chunks are evicted.
Adjusted the BlockingBuffer configurations to improve query performance.
BufferAnnotations:
int DEFAULT_CHUNK_OF_CHUNKS_CAPACITY = 10;//trunk=1000
int DEFAULT_CHUNK_CAPACITY = 1000;//trunk=100
int DEFAULT_CHUNK_TIMEOUT = 10;//trunk=1000
AbstractResource:
String DEFAULT_CHUNK_OF_CHUNKS_CAPACITY = "10"; // was 1000
String DEFAULT_CHUNK_CAPACITY = "1000"; // was 100
String DEFAULT_CHUNK_TIMEOUT = "10"; // was 1000
[java] ### Finished testing BIGDATA_SPARQL_ENDPOINT ###
[java] BIGDATA_SPARQL_ENDPOINT #trials=10 #parallel=1
[java] query Time Result#
[java] query1 53 4
[java] query3 45 6
[java] query4 73 34
[java] query5 107 719
[java] query7 27 61
[java] query8 396 6463
[java] query10 21 0
[java] query11 20 0
[java] query12 26 0
[java] query13 24 0
[java] query14 3897 393730
[java] query6 4023 430114
[java] query2 978 130
[java] query9 5157 8627
[java] Total 14847
This is the best score for the quads query branch (the trunk comes in at 12748 total). The trunk still does better on Q6, Q14, and Q9 while the quads query branch does better on Q2. I lack an explanation for Q6 and Q14 since it does not appear to be related to the BlockingBuffer configuration. Q9 might be the result of not chaining the queues together and accounts for 1s of the total difference in time for this benchmark.
The BSBM 100M WORM m=32 score in the trunk is:
[java] QMpH: 4234.40 query mixes per hour
With the chunk combiner based on the LinkedList in RunningQuery and the adjustments to the BufferAnnotations and AbstractResource options to optimize LUBM U50 query we have:
[java] QMpH: 4121.80 query mixes per hour
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BufferAnnotations.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/AbstractResource.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BufferAnnotations.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BufferAnnotations.java 2010-11-01 15:16:44 UTC (rev 3854)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BufferAnnotations.java 2010-11-01 15:29:45 UTC (rev 3855)
@@ -52,7 +52,7 @@
/**
* Default for {@link #CHUNK_OF_CHUNKS_CAPACITY}
*/
- int DEFAULT_CHUNK_OF_CHUNKS_CAPACITY = 100;//trunk=1000
+ int DEFAULT_CHUNK_OF_CHUNKS_CAPACITY = 5;//trunk=1000
/**
* Sets the capacity of the {@link IBuffer}[]s used to accumulate a chunk of
@@ -81,7 +81,7 @@
*
* @todo this is probably much larger than we want. Try 10ms.
*/
- int DEFAULT_CHUNK_TIMEOUT = 20;//trunk=1000
+ int DEFAULT_CHUNK_TIMEOUT = 10;//trunk=1000
/**
* The {@link TimeUnit}s in which the {@link #CHUNK_TIMEOUT} is measured.
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-11-01 15:16:44 UTC (rev 3854)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-11-01 15:29:45 UTC (rev 3855)
@@ -1695,10 +1695,11 @@
private volatile boolean open = true;
- /**
- * An internal buffer which is used if chunkCapacity != ZERO.
- */
- private IBindingSet[] chunk = null;
+// /**
+// * An internal buffer which is used if chunkCapacity != ZERO.
+// */
+// private IBindingSet[] chunk = null;
+ private List<IBindingSet[]> smallChunks = null;
/**
* The #of elements in the internal {@link #chunk} buffer.
@@ -1739,38 +1740,66 @@
if(!open)
throw new BufferClosedException();
- if (chunkCapacity != 0 && e.length < (chunkCapacity >> 1)) {
- /*
- * The caller's array is significantly smaller than the target
- * chunk size. Append the caller's array to the internal buffer
- * and return immediately. The internal buffer will be copied
- * through either in a subsequent add() or in flush().
- */
- synchronized (this) {
+// if (chunkCapacity != 0 && e.length < (chunkCapacity >> 1)) {
+// /*
+// * The caller's array is significantly smaller than the target
+// * chunk size. Append the caller's array to the internal buffer
+// * and return immediately. The internal buffer will be copied
+// * through either in a subsequent add() or in flush().
+// */
+// synchronized (this) {
+//
+// if (chunk == null)
+// chunk = new IBindingSet[chunkCapacity];
+//
+// if (chunkSize + e.length > chunkCapacity) {
+//
+// // flush the buffer first.
+// outputBufferedChunk();
+//
+// }
+//
+// // copy the chunk into the buffer.
+// System.arraycopy(e/* src */, 0/* srcPos */,
+// chunk/* dest */, chunkSize/* destPos */,
+// e.length/* length */);
+//
+// chunkSize += e.length;
+//
+// return;
+//
+// }
+//
+// }
- if (chunk == null)
- chunk = new IBindingSet[chunkCapacity];
+ if (chunkCapacity != 0 && e.length < (chunkCapacity >> 1)) {
+ /*
+ * The caller's array is significantly smaller than the target
+ * chunk size. Append the caller's array to the internal list
+ * and return immediately. The buffered chunks will be copied
+ * through either in a subsequent add() or in flush().
+ */
+ synchronized (this) {
- if (chunkSize + e.length > chunk.length) {
+ if (smallChunks == null)
+ smallChunks = new LinkedList<IBindingSet[]>();
- // flush the buffer first.
- outputBufferedChunk();
+ if (chunkSize + e.length > chunkCapacity) {
- }
+ // flush the buffer first.
+ outputBufferedChunk();
- // copy the chunk into the buffer.
- System.arraycopy(e/* src */, 0/* srcPos */,
- chunk/* dest */, chunkSize/* destPos */,
- e.length/* length */);
+ }
+
+ smallChunks.add(e);
- chunkSize += e.length;
+ chunkSize += e.length;
- return;
+ return;
- }
+ }
+ }
- }
-
// output the caller's chunk immediately.
outputChunk(e);
@@ -1798,15 +1827,36 @@
*/
synchronized // Note: has side-effect on internal buffer.
private void outputBufferedChunk() {
- if (chunk == null || chunkSize == 0)
+// if (chunk == null || chunkSize == 0)
+// return;
+// if (chunkSize != chunk.length) {
+// // truncate the array.
+// chunk = Arrays.copyOf(chunk, chunkSize);
+// }
+// outputChunk(chunk);
+// chunkSize = 0;
+// chunk = null;
+ if (smallChunks == null || chunkSize == 0)
return;
- if (chunkSize != chunk.length) {
- // truncate the array.
- chunk = Arrays.copyOf(chunk, chunkSize);
- }
+ if (smallChunks.size() == 1) {
+ // directly output a single small chunk.
+ outputChunk(smallChunks.get(0));
+ chunkSize = 0;
+ smallChunks = null;
+ return;
+ }
+ // exact fit buffer.
+ final IBindingSet[] chunk = new IBindingSet[chunkSize];
+ // copy the small chunks into the buffer.
+ int destPos = 0;
+ for (IBindingSet[] e : smallChunks) {
+ System.arraycopy(e/* src */, 0/* srcPos */, chunk/* dest */,
+ destPos, e.length/* length */);
+ destPos += e.length;
+ }
outputChunk(chunk);
chunkSize = 0;
- chunk = null;
+ smallChunks = null;
}
synchronized // Note: possible side-effect on internal buffer.
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/AbstractResource.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/AbstractResource.java 2010-11-01 15:16:44 UTC (rev 3854)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/AbstractResource.java 2010-11-01 15:29:45 UTC (rev 3855)
@@ -237,7 +237,7 @@
* Default for {@link #CHUNK_OF_CHUNKS_CAPACITY}
* @deprecated by {@link BOp} annotations.
*/
- String DEFAULT_CHUNK_OF_CHUNKS_CAPACITY = "1000";
+ String DEFAULT_CHUNK_OF_CHUNKS_CAPACITY = "10"; // was 1000
/**
* <p>
@@ -260,7 +260,7 @@
*
* @deprecated by {@link BOp} annotations.
*/
- String DEFAULT_CHUNK_CAPACITY = "100";
+ String DEFAULT_CHUNK_CAPACITY = "1000"; // was 100
/**
* The timeout in milliseconds that the {@link BlockingBuffer} will wait
@@ -278,7 +278,7 @@
* @todo this is probably much larger than we want. Try 10ms.
* @deprecated by {@link BOp} annotations.
*/
- String DEFAULT_CHUNK_TIMEOUT = "1000";
+ String DEFAULT_CHUNK_TIMEOUT = "10"; // was 1000
/**
* If the estimated rangeCount for an
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mar...@us...> - 2010-11-01 15:16:50
|
Revision: 3854
http://bigdata.svn.sourceforge.net/bigdata/?rev=3854&view=rev
Author: martyncutcher
Date: 2010-11-01 15:16:44 +0000 (Mon, 01 Nov 2010)
Log Message:
-----------
Add override of commit to DiskOnly and WORM strategy
Modified Paths:
--------------
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractBufferStrategy.java
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/DiskOnlyStrategy.java
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/WORMStrategy.java
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractBufferStrategy.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractBufferStrategy.java 2010-11-01 12:56:24 UTC (rev 3853)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractBufferStrategy.java 2010-11-01 15:16:44 UTC (rev 3854)
@@ -631,7 +631,6 @@
public void commit(IJournal journal) {
// NOP for WORM.
-
}
/** The default is a NOP. */
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/DiskOnlyStrategy.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/DiskOnlyStrategy.java 2010-11-01 12:56:24 UTC (rev 3853)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/DiskOnlyStrategy.java 2010-11-01 15:16:44 UTC (rev 3854)
@@ -482,7 +482,7 @@
* Note that the internal call to flush the writeCache must be synchronized
* or concurrent writers to the cache will cause problems.
*/
- public void commit() {
+ public void commit(IJournal journal) {
if (writeCache != null) {
synchronized(this) {
flushWriteCache();
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/WORMStrategy.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/WORMStrategy.java 2010-11-01 12:56:24 UTC (rev 3853)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/WORMStrategy.java 2010-11-01 15:16:44 UTC (rev 3854)
@@ -1048,7 +1048,7 @@
* @todo Should be a NOP for the WORM? Check
* {@link AbstractJournal#commitNow(long)}
*/
- public void commit() {
+ public void commit(IJournal journal) {
flushWriteCache();
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-11-01 12:56:32
|
Revision: 3853
http://bigdata.svn.sourceforge.net/bigdata/?rev=3853&view=rev
Author: thompsonbry
Date: 2010-11-01 12:56:24 +0000 (Mon, 01 Nov 2010)
Log Message:
-----------
Modified RunningQuery to support aggregation of small chunks generated by an operator before they are pass along to the target operator (or mapped across shards in scale-out). This is a performance optimization, but I have not yet validate the impact on performance (I need to move the code over to a workstation to do this).
Added toString() to Predicate$HashedPredicate.
Fixed NPE in QueryLog which occurred when the BOpStats were not yet available for some operator. The NPE was being trapped so it was not causing query evaluation errors.
Modified RunState to access RunningQuery.bopIndex using getBOpIndex().
Javadoc edits to AccessPath to note an issue that I want to explore in more depth.
Javadoc clarification and bug fix for LogUtil. It was failing to handle XML configuration files specified using -Dlog4j.configuration.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunState.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/accesspath/AccessPath.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/util/config/LogUtil.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java 2010-11-01 12:52:00 UTC (rev 3852)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java 2010-11-01 12:56:24 UTC (rev 3853)
@@ -628,6 +628,12 @@
}
+ public String toString() {
+
+ return super.toString() + "{pred=" + pred + ",hash=" + hash + "}";
+
+ }
+
}
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java 2010-11-01 12:52:00 UTC (rev 3852)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryLog.java 2010-11-01 12:56:24 UTC (rev 3853)
@@ -257,8 +257,10 @@
stats.add(t);
}
} else {
- // Just this operator.
- stats.add(statsMap.get(bopId));
+ // Just this operator.
+ final BOpStats tmp = statsMap.get(bopId);
+ if (tmp != null)
+ stats.add(tmp);
}
final long unitsIn = stats.unitsIn.get();
final long unitsOut = stats.unitsOut.get();
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunState.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunState.java 2010-11-01 12:52:00 UTC (rev 3852)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunState.java 2010-11-01 12:56:24 UTC (rev 3853)
@@ -271,7 +271,7 @@
public RunState(final RunningQuery query) {
this(query.getQuery(), query.getQueryId(), query.getDeadline(),
- query.bopIndex);
+ query.getBOpIndex());
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-11-01 12:52:00 UTC (rev 3852)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/RunningQuery.java 2010-11-01 12:56:24 UTC (rev 3853)
@@ -27,6 +27,7 @@
*/
package com.bigdata.bop.engine;
+import java.util.Arrays;
import java.util.Collections;
import java.util.Iterator;
import java.util.LinkedList;
@@ -60,6 +61,7 @@
import com.bigdata.bop.solutions.SliceOp;
import com.bigdata.journal.IIndexManager;
import com.bigdata.journal.ITx;
+import com.bigdata.relation.accesspath.BufferClosedException;
import com.bigdata.relation.accesspath.IAsynchronousIterator;
import com.bigdata.relation.accesspath.IBlockingBuffer;
import com.bigdata.relation.accesspath.IMultiSourceAsynchronousIterator;
@@ -67,6 +69,7 @@
import com.bigdata.service.IBigdataFederation;
import com.bigdata.striterator.ICloseableIterator;
import com.bigdata.util.concurrent.Haltable;
+import com.bigdata.util.concurrent.Memoizer;
/**
* Metadata about running queries.
@@ -149,7 +152,7 @@
* An index from the {@link BOp.Annotations#BOP_ID} to the {@link BOp}. This
* index is generated by the constructor. It is immutable and thread-safe.
*/
- protected final Map<Integer, BOp> bopIndex;
+ private final Map<Integer, BOp> bopIndex;
/**
* The run state of the query and the result of the computation iff it
@@ -267,7 +270,7 @@
/**
* Flag used to prevent retriggering of {@link #lifeCycleTearDownQuery()}.
*/
- final AtomicBoolean didQueryTearDown = new AtomicBoolean(false);
+ private final AtomicBoolean didQueryTearDown = new AtomicBoolean(false);
// /**
// * The chunks available for immediate processing (they must have been
@@ -1217,41 +1220,50 @@
}
/**
- * A {@link FutureTask} which exposes the {@link ChunkTask} which is being
- * evaluated.
+ * A {@link FutureTask} which conditionally schedules another task for the
+ * same (bopId, partitionId) once this the wrapped {@link ChunkTask} is
+ * done. This is similar to the {@link Memoizer} pattern. This class
+ * coordinates with the {@link #operatorFutures}, which maintains a map of
+ * all currently running tasks which are consuming chunks.
+ * <p>
+ * The {@link ChunkTask} is wrapped by a {@link ChunkTaskWrapper} which is
+ * responsible for communicating the changes in the query's running state
+ * back to the {@link RunState} object on the query controller.
*/
private class ChunkFutureTask extends FutureTask<Void> {
- public final ChunkTask chunkTask;
+ private final ChunkTask t;
public ChunkFutureTask(final ChunkTask chunkTask) {
-// super(chunkTask, null/* result */);
+ /*
+ * Note: wraps chunk task to communicate run state changes back to
+ * the query controller.
+ */
+ super(new ChunkTaskWrapper(chunkTask), null/* result */);
- // Note: wraps chunk task to ensure source and sinks get closed.
- super(new ChunkTaskWrapper(chunkTask), null/* result */);
-
- this.chunkTask = chunkTask;
-
+ this.t = chunkTask;
+
}
public void run() {
- final ChunkTask t = chunkTask;
-
- super.run();
-
- /*
- * This task is done executing so remove its Future before we
- * attempt to schedule another task for the same
- * (bopId,partitionId).
- */
- final ConcurrentHashMap<ChunkFutureTask, ChunkFutureTask> map = operatorFutures
- .get(new BSBundle(t.bopId, t.partitionId));
- if (map != null) {
- map.remove(this, this);
- }
-
+ super.run();
+
+ /*
+ * This task is done executing so remove its Future before we
+ * attempt to schedule another task for the same
+ * (bopId,partitionId).
+ */
+ final ConcurrentHashMap<ChunkFutureTask, ChunkFutureTask> map = operatorFutures
+ .get(new BSBundle(t.bopId, t.partitionId));
+
+ if (map != null) {
+
+ map.remove(this, this);
+
+ }
+
// Schedule another task if any messages are waiting.
RunningQuery.this.scheduleNext(new BSBundle(
t.bopId, t.partitionId));
@@ -1262,11 +1274,12 @@
/**
* Wraps the {@link ChunkTask} and handles various handshaking with the
- * {@link RunningQuery} and the {@link RunState}. Since starting and
- * stopping a {@link ChunkTask} requires handshaking with the query
- * controller, it is important that these actions take place once the task
- * has been submitted - otherwise they would be synchronous in the loop
- * which consumes available chunks and generates new {@link ChunkTask}s.
+ * {@link RunningQuery} and the {@link RunState} on the query controller.
+ * Since starting and stopping a {@link ChunkTask} requires handshaking with
+ * the query controller (and thus can require RMI), it is important that
+ * these actions take place once the task has been submitted - otherwise
+ * they would be synchronous in the loop which consumes available chunks and
+ * generates new {@link ChunkTask}s.
*/
private class ChunkTaskWrapper implements Runnable {
@@ -1608,7 +1621,8 @@
// .getChunkTimeout(),
// BufferAnnotations.chunkTimeoutUnit);
- return new HandleChunkBuffer(sinkId, sinkMessagesOut, /*b,*/ stats);
+ return new HandleChunkBuffer(RunningQuery.this, bopId, sinkId, op
+ .getChunkCapacity(), sinkMessagesOut, stats);
}
@@ -1647,108 +1661,208 @@
return null;
} // call()
+ } // class ChunkTask
+
+ /**
+ * Class traps {@link #add(IBindingSet[])} to handle the IBindingSet[]
+ * chunks as they are generated by the running operator task, invoking
+ * {@link RunningQuery#handleOutputChunk(BOp, int, IBlockingBuffer)} for
+ * each generated chunk to synchronously emit {@link IChunkMessage}s.
+ * <p>
+ * This use of this class significantly increases the parallelism and
+ * throughput of selective queries. If output chunks are not "handled"
+ * until the {@link ChunkTask} is complete then the total latency of
+ * selective queries is increased dramatically.
+ */
+ static private class HandleChunkBuffer implements
+ IBlockingBuffer<IBindingSet[]> {
+
+ private final RunningQuery q;
+
+ private final int bopId;
+
+ private final int sinkId;
+
/**
- * Class traps {@link #add(IBindingSet[])} to handle the IBindingSet[]
- * chunks as they are generated by the running operator task, invoking
- * {@link RunningQuery#handleOutputChunk(BOp, int, IBlockingBuffer)} for
- * each generated chunk to synchronously emit {@link IChunkMessage}s.
+ * The target chunk size. When ZERO (0) chunks are output immediately as
+ * they are received (the internal buffer is not used).
*/
- private class HandleChunkBuffer implements IBlockingBuffer<IBindingSet[]> {
+ private final int chunkCapacity;
+
+ private final AtomicInteger sinkMessagesOut;
+
+ private final BOpStats stats;
+
+ private volatile boolean open = true;
+
+ /**
+ * An internal buffer which is used if chunkCapacity != ZERO.
+ */
+ private IBindingSet[] chunk = null;
+
+ /**
+ * The #of elements in the internal {@link #chunk} buffer.
+ */
+ private int chunkSize = 0;
+
+ /**
+ *
+ * @param q
+ * @param bopId
+ * @param sinkId
+ * @param chunkCapacity
+ * @param sinkMessagesOut
+ * @param stats
+ */
+ public HandleChunkBuffer(final RunningQuery q, final int bopId,
+ final int sinkId, final int chunkCapacity,
+ final AtomicInteger sinkMessagesOut, final BOpStats stats) {
+ this.q = q;
+ this.bopId = bopId;
+ this.sinkId = sinkId;
+ this.chunkCapacity = chunkCapacity;
+ this.sinkMessagesOut = sinkMessagesOut;
+ this.stats = stats;
+ }
+
+ /**
+ * Handle sink output, sending appropriate chunk message(s). This method
+ * MUST NOT block since that will deadlock the caller.
+ * <p>
+ * Note: This maps output over shards/nodes in s/o.
+ * <p>
+ * Note: This must be synchronized in case the caller is multi-threaded
+ * since it has a possible side effect on the internal buffer.
+ */
+ public void add(final IBindingSet[] e) {
- private final int sinkId;
- private final AtomicInteger sinkMessagesOut;
-// private final IBlockingBuffer<IBindingSet[]> sink;
- private final BOpStats stats;
- private volatile boolean open = true;
-
- public HandleChunkBuffer(final int sinkId,
- final AtomicInteger sinkMessagesOut,
-// final IBlockingBuffer<IBindingSet[]> b,
- final BOpStats stats
- ) {
- this.sinkId = sinkId;
- this.sinkMessagesOut = sinkMessagesOut;
-// this.sink = b;
- this.stats = stats;
- }
+ if(!open)
+ throw new BufferClosedException();
- /**
- * Handle sink output, sending appropriate chunk message(s).
- * <p>
- * Note: This maps output over shards/nodes in s/o.
- */
- public void add(final IBindingSet[] e) {
-// if (e.getClass().getComponentType() != null) {
- stats.unitsOut.add(((Object[]) e).length);
-// } else {
-// stats.unitsOut.increment();
-// }
- stats.chunksOut.increment();
+ if (chunkCapacity != 0 && e.length < (chunkCapacity >> 1)) {
/*
- * FIXME Combine together when possible and final evict in
- * flush(). The logic here MUST NOT block since that will
- * deadlock the caller. The safest thing to do is to emit each
- * chunk as it arrives, but that does not let us combine them
- * together when an operator generates small chunks.
+ * The caller's array is significantly smaller than the target
+ * chunk size. Append the caller's array to the internal buffer
+ * and return immediately. The internal buffer will be copied
+ * through either in a subsequent add() or in flush().
*/
-// sink.add(e);
- sinkMessagesOut.addAndGet(getChunkHandler().handleChunk(
- RunningQuery.this, bopId, sinkId, e));
- }
+ synchronized (this) {
- public long flush() {
- return 0L;
-// return sink.flush();
- }
+ if (chunk == null)
+ chunk = new IBindingSet[chunkCapacity];
- public void abort(Throwable cause) {
- open = false;
- RunningQuery.this.halt(cause);
-// sink.abort(cause);
- }
+ if (chunkSize + e.length > chunk.length) {
- public void close() {
-// sink.close();
- open = false;
- }
+ // flush the buffer first.
+ outputBufferedChunk();
- public Future getFuture() {
-// return sink.getFuture();
- return null;
- }
+ }
- public boolean isEmpty() {
- return true;
-// return sink.isEmpty();
- }
+ // copy the chunk into the buffer.
+ System.arraycopy(e/* src */, 0/* srcPos */,
+ chunk/* dest */, chunkSize/* destPos */,
+ e.length/* length */);
- public boolean isOpen() {
- return open && !RunningQuery.this.isDone();
-// return sink.isOpen();
- }
+ chunkSize += e.length;
- public IAsynchronousIterator<IBindingSet[]> iterator() {
- throw new UnsupportedOperationException();
-// return sink.iterator();
- }
+ return;
- public void reset() {
-// sink.reset();
- }
+ }
- public void setFuture(Future future) {
- throw new UnsupportedOperationException();
-// sink.setFuture(future);
}
- public int size() {
- return 0;
-// return sink.size();
+ // output the caller's chunk immediately.
+ outputChunk(e);
+
+ }
+
+ /**
+ * Output a chunk, updating the counters.
+ *
+ * @param e
+ * The chunk.
+ */
+ private void outputChunk(final IBindingSet[] e) {
+
+ stats.unitsOut.add(((Object[]) e).length);
+
+ stats.chunksOut.increment();
+
+ sinkMessagesOut.addAndGet(q.getChunkHandler().handleChunk(q, bopId,
+ sinkId, e));
+
+ }
+
+ /**
+ * Output the internal buffer.
+ */
+ synchronized // Note: has side-effect on internal buffer.
+ private void outputBufferedChunk() {
+ if (chunk == null || chunkSize == 0)
+ return;
+ if (chunkSize != chunk.length) {
+ // truncate the array.
+ chunk = Arrays.copyOf(chunk, chunkSize);
}
+ outputChunk(chunk);
+ chunkSize = 0;
+ chunk = null;
+ }
+
+ synchronized // Note: possible side-effect on internal buffer.
+ public long flush() {
+ if (open)
+ outputBufferedChunk();
+ return 0L;
+// return sink.flush();
+ }
- } // class HandleChunkBuffer
-
- } // class ChunkTask
+ public void abort(Throwable cause) {
+ open = false;
+ q.halt(cause);
+// sink.abort(cause);
+ }
+
+ public void close() {
+// sink.close();
+ open = false;
+ }
+
+ public Future getFuture() {
+// return sink.getFuture();
+ return null;
+ }
+
+ public boolean isEmpty() {
+ return true;
+// return sink.isEmpty();
+ }
+
+ public boolean isOpen() {
+ return open && !q.isDone();
+// return sink.isOpen();
+ }
+
+ public IAsynchronousIterator<IBindingSet[]> iterator() {
+ throw new UnsupportedOperationException();
+// return sink.iterator();
+ }
+
+ public void reset() {
+// sink.reset();
+ }
+
+ public void setFuture(Future future) {
+ throw new UnsupportedOperationException();
+// sink.setFuture(future);
+ }
+
+ public int size() {
+ return 0;
+// return sink.size();
+ }
+
+ } // class HandleChunkBuffer
// private static class BlockingBufferWithStats<E> extends BlockingBuffer<E> {
//
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/accesspath/AccessPath.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/accesspath/AccessPath.java 2010-11-01 12:52:00 UTC (rev 3852)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/accesspath/AccessPath.java 2010-11-01 12:56:24 UTC (rev 3853)
@@ -1585,11 +1585,14 @@
/*
* SWAG in case zero partition count is reported (I am not sure that
* this code path is possible).
+ *
+ * @todo This is proven possible. Now figure out why. Maybe this is
+ * fromKey==toKey, in which case we can optimize that out.
*/
return new ScanCostReport(0L/* rangeCount */, partitionCount, 100/* millis */);
- /*
- * Should never be "zero" partition count.
- */
+// /*
+// * Should never be "zero" partition count.
+// */
// throw new AssertionError();
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/util/config/LogUtil.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/util/config/LogUtil.java 2010-11-01 12:52:00 UTC (rev 3852)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/util/config/LogUtil.java 2010-11-01 12:56:24 UTC (rev 3853)
@@ -30,47 +30,84 @@
import org.apache.log4j.xml.DOMConfigurator;
/**
- * Utility class that provides a set of static convenience methods related
- * to the initialization and configuration of the logging mechanism(s)
- * employed by the components of the system. The methods of this class
- * can be useful both in Jini configuration files, as well as in the
- * system components themselves.
+ * Utility class that provides a set of static convenience methods related to
+ * the initialization and configuration of the logging mechanism(s) employed by
+ * the components of the system. The methods of this class can be useful both in
+ * Jini configuration files, as well as in the system components themselves.
* <p>
+ * This class relies on the presence of either the
+ * <code>log4j.configuration</code> or the
+ * <code>log4j.primary.configuration</code> property and understands files with
+ * any of the following extensions {<code>.properties</code>,
+ * <code>.logging</code>, <code>.xml</code> . It will log a message on
+ * <em>stderr</em> if neither of those properties is defined. The class
+ * deliberately does not search the CLASSPATH for a log4j configuration in an
+ * effort to discourage the inadvertent use of hidden configuration files when
+ * deploying bigdata.
+ * <p>
+ * A watcher is setup on the log4j configuration if one is found.
+ * <p>
* This class cannot be instantiated.
*/
public class LogUtil {
+ /**
+ * Examine the various log4j configuration properties and return the name of
+ * the log4j configuration resource if one was configured.
+ *
+ * @return The log4j configuration resource -or- <code>null</code> if the
+ * resource was not configured properly.
+ */
+ static String getConfigPropertyValue() {
+
+ final String log4jConfig = System
+ .getProperty("log4j.primary.configuration");
+
+ if (log4jConfig != null)
+ return log4jConfig;
+
+ final String log4jDefaultConfig = System
+ .getProperty("log4j.configuration");
+
+ if (log4jDefaultConfig != null)
+ return log4jDefaultConfig;
+
+ return null;
+
+ }
+
// Static initialization block that retrieves and initializes
// the log4j logger configuration for the given VM in which this
// class resides. Note that this block is executed only once
// during the life of the associated VM.
static {
- final String log4jConfig =
- System.getProperty("log4j.primary.configuration");
+
+ final String log4jConfig = getConfigPropertyValue();
+
if( log4jConfig != null && (log4jConfig.endsWith(".properties") ||
log4jConfig.endsWith(".logging"))) {
+
PropertyConfigurator.configureAndWatch(log4jConfig);
+
} else if ( log4jConfig != null && log4jConfig.endsWith(".xml") ) {
+
DOMConfigurator.configureAndWatch(log4jConfig);
+
} else {
- final String log4jDefaultConfig =
- System.getProperty("log4j.configuration");
- if (log4jDefaultConfig != null ) {
- PropertyConfigurator.configureAndWatch(log4jDefaultConfig);
- } else {
- System.out.println
- ("ERROR: could not initialize Log4J logging utility");
- System.out.println
- (" set system property "
- +"'-Dlog4j.configuration="
- +"file:bigdata/src/resources/logging/log4j.properties"
- +"\n and/or \n"
- +" set system property "
- +"'-Dlog4j.primary.configuration="
- +"file:<installDir>/"
- +"bigdata/src/resources/logging/log4j.properties'");
- }
+
+ System.err.println("ERROR: " + LogUtil.class.getName()
+ + " : Could not initialize Log4J logging utility. "
+ + "Set system property "
+ +"'-Dlog4j.configuration="
+ +"file:bigdata/src/resources/logging/log4j.properties"
+ +"\n and/or \n"
+ +" set system property "
+ +"'-Dlog4j.primary.configuration="
+ +"file:<installDir>/"
+ +"bigdata/src/resources/logging/log4j.properties'");
+
}
+
}
public static Logger getLog4jLogger(String componentName) {
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mar...@us...> - 2010-11-01 12:52:07
|
Revision: 3852
http://bigdata.svn.sourceforge.net/bigdata/?rev=3852&view=rev
Author: martyncutcher
Date: 2010-11-01 12:52:00 +0000 (Mon, 01 Nov 2010)
Log Message:
-----------
1) Change RWStrategy to pass the Journal reference on commit to remove need to set LocalTransactionService
2) Change deferred free to write to PSOutputStream to avoid maintaining inmemory references.
Modified Paths:
--------------
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractBufferStrategy.java
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractJournal.java
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/IBufferStrategy.java
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/Journal.java
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/RWStrategy.java
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/PSOutputStream.java
branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java
branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/journal/AbstractInterruptsTestCase.java
branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/journal/TestWORMStrategy.java
branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/rwstore/TestRWJournal.java
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractBufferStrategy.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractBufferStrategy.java 2010-11-01 12:32:36 UTC (rev 3851)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractBufferStrategy.java 2010-11-01 12:52:00 UTC (rev 3852)
@@ -628,7 +628,7 @@
}
/** The default is a NOP. */
- public void commit() {
+ public void commit(IJournal journal) {
// NOP for WORM.
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractJournal.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractJournal.java 2010-11-01 12:32:36 UTC (rev 3851)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/AbstractJournal.java 2010-11-01 12:52:00 UTC (rev 3852)
@@ -1045,8 +1045,6 @@
ResourceManager.openJournal(getFile() == null ? null : getFile().toString(), size(), getBufferStrategy()
.getBufferMode());
- this._bufferStrategy.setCommitRecordIndex(_commitRecordIndex);
-
} finally {
lock.unlock();
@@ -2103,8 +2101,6 @@
// clear reference and reload from the store.
_commitRecordIndex = _getCommitRecordIndex();
- _bufferStrategy.setCommitRecordIndex(_commitRecordIndex);
-
// clear the array of committers.
_committers = new ICommitter[_committers.length];
@@ -2358,7 +2354,7 @@
* until commit, leading to invalid addresses for recent store
* allocations.
*/
- _bufferStrategy.commit();
+ _bufferStrategy.commit(this);
/*
* next offset at which user data would be written.
@@ -2875,7 +2871,7 @@
* {@link #getCommitRecord(long)} to obtain a distinct instance
* suitable for read-only access.
*/
- protected CommitRecordIndex getCommitRecordIndex() {
+ public CommitRecordIndex getCommitRecordIndex() {
final ReadLock lock = _fieldReadWriteLock.readLock();
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/IBufferStrategy.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/IBufferStrategy.java 2010-11-01 12:32:36 UTC (rev 3851)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/IBufferStrategy.java 2010-11-01 12:52:00 UTC (rev 3852)
@@ -216,8 +216,9 @@
* A method that removes assumptions of how a specific strategy commits data. For most strategies the action is void
* since the client WORM DISK strategy writes data as allocated. For the Read Write Strategy more data must be managed
* as part of the protocol outside of the RootBlock, and this is the method that triggers that management.
+ * @param abstractJournal
*/
- public void commit();
+ public void commit(IJournal journal);
/**
* A method that requires the implementation to discard its buffered write
@@ -258,29 +259,4 @@
*/
public boolean useChecksums();
- /**
- * Needed to enable transaction support for standalone buffer strategies.
- *
- * The WORMStrategy does not need this since no data is ever deleted, but
- * the RWStrategy must manage deletions such that no data is deleted until
- * it can be guaranteed not to be accessed by existing transactions.
- *
- * @param localTransactionManager
- * The transaction manager for the owning Journal
- */
- public void setTransactionManager(AbstractLocalTransactionManager localTransactionManager);
-
-
- /**
- * Needed to enable transaction support for standalone buffer strategies.
- *
- * The WORMStrategy does not need this since no data is ever deleted, but
- * the RWStrategy must manage deletions and needs access to the historical
- * commitRecords which reference the blocks of deferred deleted addresses.
- *
- * @param commitRecordIndex
- * The CommitRecordIndex for the owning Journal
- */
- public void setCommitRecordIndex(CommitRecordIndex commitRecordIndex);
-
}
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/Journal.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/Journal.java 2010-11-01 12:32:36 UTC (rev 3851)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/Journal.java 2010-11-01 12:52:00 UTC (rev 3852)
@@ -221,10 +221,8 @@
concurrencyManager = new ConcurrencyManager(properties,
localTransactionManager, this);
- getBufferStrategy().setTransactionManager(localTransactionManager);
+ }
- }
-
// public void init() {
//
// super.init();
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/RWStrategy.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/RWStrategy.java 2010-11-01 12:32:36 UTC (rev 3851)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/journal/RWStrategy.java 2010-11-01 12:52:00 UTC (rev 3852)
@@ -30,6 +30,7 @@
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.ByteBuffer;
+import java.nio.channels.ClosedByInterruptException;
import java.util.UUID;
import java.util.concurrent.Callable;
import java.util.concurrent.locks.ReentrantLock;
@@ -525,10 +526,10 @@
*
* Must pass in earliestTxnTime to commitChanges to enable
*/
- public void commit() {
+ public void commit(IJournal journal) {
m_commitLock.lock();
try {
- m_store.commitChanges(); // includes a force(false)
+ m_store.commitChanges((Journal) journal); // includes a force(false)
} finally {
m_commitLock.unlock();
}
@@ -546,6 +547,12 @@
public void force(boolean metadata) {
try {
m_store.flushWrites(metadata);
+ } catch (ClosedByInterruptException e) {
+ m_needsReopen = true;
+
+ reopen(); // FIXME
+
+ throw new RuntimeException(e);
} catch (IOException e) {
m_needsReopen = true;
@@ -716,11 +723,6 @@
}
- public void setTransactionManager(AbstractLocalTransactionManager localTransactionManager) {
- this.localTransactionManager = localTransactionManager;
- m_store.setTransactionService((JournalTransactionService) localTransactionManager.getTransactionService());
- }
-
public long getPhysicalAddress(long addr) {
int rwaddr = decodeAddr(addr);
@@ -737,9 +739,4 @@
public long saveDeleteBlocks() {
return m_store.saveDeferrals();
}
-
- public void setCommitRecordIndex(CommitRecordIndex commitRecordIndex) {
- m_store.setCommitRecordIndex(commitRecordIndex);
- }
-
}
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/PSOutputStream.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/PSOutputStream.java 2010-11-01 12:32:36 UTC (rev 3851)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/PSOutputStream.java 2010-11-01 12:52:00 UTC (rev 3852)
@@ -158,28 +158,29 @@
* resets private state variables for reuse of stream
**/
void init(IStore store, int maxAlloc, IAllocationContext context) {
+ m_store = store;
+ m_context = context;
+
+ m_blobThreshold = maxAlloc-4; // allow for checksum
+ if (m_buf == null || m_buf.length != m_blobThreshold)
+ m_buf = new byte[m_blobThreshold];
+
+ reset();
+ }
+
+ public void reset() {
m_isSaved = false;
m_headAddr = 0;
m_prevAddr = 0;
m_count = 0;
m_bytesWritten = 0;
- m_store = store;
m_isSaved = false;
- // m_blobThreshold = m_store.bufferChainOffset();
- m_blobThreshold = maxAlloc-4; // allow for checksum
- m_buf = new byte[maxAlloc-4];
m_blobHeader = null;
m_blobHdrIdx = 0;
-
- m_context = context;
+ }
-
- // FIXME: if autocommit then we should provide start/commit via init and save
- // m_store.startTransaction();
- }
-
/****************************************************************
* write a single byte
*
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2010-11-01 12:32:36 UTC (rev 3851)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2010-11-01 12:52:00 UTC (rev 3852)
@@ -52,6 +52,7 @@
import com.bigdata.io.IReopenChannel;
import com.bigdata.io.writecache.BufferedWrite;
import com.bigdata.io.writecache.WriteCache;
+import com.bigdata.journal.Journal;
import com.bigdata.journal.AbstractJournal;
import com.bigdata.journal.CommitRecordIndex;
import com.bigdata.journal.ForceEnum;
@@ -323,8 +324,7 @@
private static final int MAX_DEFERRED_FREE = 4094; // fits in 16k block
volatile long m_lastDeferredReleaseTime = 23; // zero is invalid time
final ArrayList<Integer> m_currentTxnFreeList = new ArrayList<Integer>();
-
- volatile long deferredFreeCount = 0;
+ final PSOutputStream m_deferredFreeOut;
private ReopenFileChannel m_reopener = null;
@@ -490,7 +490,7 @@
m_maxFixedAlloc = m_allocSizes[m_allocSizes.length-1]*64;
m_minFixedAlloc = m_allocSizes[0]*64;
- commitChanges();
+ commitChanges(null);
} else {
@@ -502,6 +502,7 @@
assert m_maxFixedAlloc > 0;
+ m_deferredFreeOut = PSOutputStream.getNew(this, m_maxFixedAlloc, null);
} catch (IOException e) {
throw new StorageTerminalError("Unable to initialize store", e);
}
@@ -1213,8 +1214,6 @@
private volatile long m_maxAllocation = 0;
private volatile long m_spareAllocation = 0;
-
- private CommitRecordIndex m_commitRecordIndex;
public int alloc(final int size, IAllocationContext context) {
if (size > m_maxFixedAlloc) {
@@ -1569,7 +1568,7 @@
return "RWStore " + s_version;
}
- public void commitChanges() {
+ public void commitChanges(Journal journal) {
checkCoreAllocations();
// take allocation lock to prevent other threads allocating during commit
@@ -1577,7 +1576,7 @@
try {
- checkDeferredFrees(true); // free now if possible
+ checkDeferredFrees(true, journal); // free now if possible
// Allocate storage for metaBits
long oldMetaBits = m_metaBitsAddr;
@@ -1653,30 +1652,26 @@
* Called prior to commit, so check whether storage can be freed and then
* whether the deferredheader needs to be saved.
*/
- private void checkDeferredFrees(boolean freeNow) {
- if (freeNow) checkFreeable();
-
- // Commit can be called prior to Journal initialisation, in which case
- // the commitRecordIndex will not be set.
- if (m_commitRecordIndex == null) {
- return;
+ public void checkDeferredFrees(boolean freeNow, Journal journal) {
+ if (journal != null) {
+ final JournalTransactionService transactionService = (JournalTransactionService) journal.getLocalTransactionManager().getTransactionService();
+
+ // Commit can be called prior to Journal initialisation, in which case
+ // the commitRecordIndex will not be set.
+ final CommitRecordIndex commitRecordIndex = journal.getCommitRecordIndex();
+
+ long latestReleasableTime = System.currentTimeMillis();
+
+ if (transactionService != null) {
+ latestReleasableTime -= transactionService.getMinReleaseAge();
+ }
+
+ Iterator<ICommitRecord> records = commitRecordIndex.getCommitRecords(m_lastDeferredReleaseTime, latestReleasableTime);
+
+ freeDeferrals(records);
}
-
- long latestReleasableTime = System.currentTimeMillis();
-
- if (m_transactionService != null) {
- latestReleasableTime -= m_transactionService.getMinReleaseAge();
- }
-
- Iterator<ICommitRecord> records = m_commitRecordIndex.getCommitRecords(m_lastDeferredReleaseTime, latestReleasableTime);
-
- freeDeferrals(records);
}
-
- public void setCommitRecordIndex(CommitRecordIndex commitRecordIndex) {
- m_commitRecordIndex = commitRecordIndex;
- }
/**
*
* @return conservative requirement for metabits storage, mindful that the
@@ -2654,45 +2649,39 @@
*
* The deferred list is checked on AllocBlock and prior to commit.
*
- * There is also a possibility to check for deferral at this point, since
- * we are passed both the currentCommitTime - against which this free
- * will be deferred and the earliest tx start time against which we
- * can check to see if
+ * DeferredFrees are written to the deferred PSOutputStream
*/
public void deferFree(int rwaddr, int sze) {
m_deferFreeLock.lock();
try {
- deferredFreeCount++;
- m_currentTxnFreeList.add(rwaddr);
+ m_deferredFreeOut.writeInt(rwaddr);
final Allocator alloc = getBlockByAddress(rwaddr);
if (alloc instanceof BlobAllocator) {
- m_currentTxnFreeList.add(sze);
+ m_deferredFreeOut.writeInt(sze);
}
-
- // every so many deferrals, check for free
- if (false && deferredFreeCount % 1000 == 0) {
- checkFreeable();
- }
+ } catch (IOException e) {
+ // TODO Auto-generated catch block
+ e.printStackTrace();
} finally {
m_deferFreeLock.unlock();
}
}
- private void checkFreeable() {
- if (m_transactionService == null) {
+ private void checkFreeable(final JournalTransactionService transactionService) {
+ if (transactionService == null) {
return;
}
try {
- Long freeTime = m_transactionService.tryCallWithLock(new Callable<Long>() {
+ Long freeTime = transactionService.tryCallWithLock(new Callable<Long>() {
public Long call() throws Exception {
long now = System.currentTimeMillis();
- long earliest = m_transactionService.getEarliestTxStartTime();
- long aged = now - m_transactionService.getMinReleaseAge();
+ long earliest = transactionService.getEarliestTxStartTime();
+ long aged = now - transactionService.getMinReleaseAge();
- if (m_transactionService.getActiveCount() == 0) {
+ if (transactionService.getActiveCount() == 0) {
return aged;
} else {
return aged < earliest ? aged : earliest;
@@ -2700,8 +2689,6 @@
}
}, 5L, TimeUnit.MILLISECONDS);
-
- freeCurrentDeferrals(freeTime);
} catch (RuntimeException e) {
// fine, will try again later
} catch (Exception e) {
@@ -2711,38 +2698,6 @@
/**
- * Frees all storage deferred against a txn release time less than that
- * passed in.
- *
- * @param txnRelease - the max release time
- */
- protected void freeCurrentDeferrals(long txnRelease) {
-
- m_deferFreeLock.lock();
- try {
- if (m_rb.getLastCommitTime() <= txnRelease) {
-// System.out.println("freeCurrentDeferrals");
- final Iterator<Integer> curdefers = m_currentTxnFreeList.iterator();
- while (curdefers.hasNext()) {
- final int rwaddr = curdefers.next();
- Allocator alloc = getBlock(rwaddr);
- if (alloc instanceof BlobAllocator) {
- // if this is a Blob then the size is required
- assert curdefers.hasNext();
-
- immediateFree(rwaddr, curdefers.next());
- } else {
- immediateFree(rwaddr, 0); // size ignored for FixedAllocators
- }
- }
- m_currentTxnFreeList.clear();
- }
- } finally {
- m_deferFreeLock.unlock();
- }
- }
-
- /**
* Writes the content of currentTxnFreeList to the store.
*
* These are the current buffered frees that have yet been saved into
@@ -2751,39 +2706,27 @@
* @return the address of the deferred addresses saved on the store
*/
public long saveDeferrals() {
- final byte[] buf;
m_deferFreeLock.lock();
try {
- int addrCount = m_currentTxnFreeList.size();
-
- if (addrCount == 0) {
- return 0L;
+ if (m_deferredFreeOut.getBytesWritten() == 0) {
+ return 0;
}
+ m_deferredFreeOut.writeInt(0); // terminate!
+ int outlen = m_deferredFreeOut.getBytesWritten();
+
+ long addr = m_deferredFreeOut.save();
+
+ addr <<= 32;
+ addr += outlen;
+
+ m_deferredFreeOut.reset();
- buf = new byte[4 * (addrCount + 1)];
- ByteBuffer out = ByteBuffer.wrap(buf);
- out.putInt(addrCount);
- for (int i = 0; i < addrCount; i++) {
- out.putInt(m_currentTxnFreeList.get(i));
- }
-
- // now we've saved it to the store, clear the list
- m_currentTxnFreeList.clear();
-
+ return addr;
+ } catch (IOException e) {
+ throw new RuntimeException("Cannot write to deferred free", e);
} finally {
m_deferFreeLock.unlock();
}
-
- long rwaddr = alloc(buf, buf.length, null);
- if (log.isTraceEnabled()) {
- long paddr = physicalAddress((int) rwaddr);
- log.trace("Saving deferred free block at " + paddr);
- }
-
- rwaddr <<= 32;
- rwaddr += buf.length;
-
- return rwaddr;
}
/**
@@ -2802,19 +2745,21 @@
final DataInputStream strBuf = new DataInputStream(new ByteArrayInputStream(buf));
m_allocationLock.lock();
try {
- int addrs = strBuf.readInt();
+ int nxtAddr = strBuf.readInt();
- while (addrs-- > 0) { // while (false && addrs-- > 0) {
- int nxtAddr = strBuf.readInt();
+ while (nxtAddr != 0) { // while (false && addrs-- > 0) {
Allocator alloc = getBlock(nxtAddr);
if (alloc instanceof BlobAllocator) {
- assert addrs > 0; // a Blob address MUST have a size
- --addrs;
- immediateFree(nxtAddr, strBuf.readInt());
+ int bloblen = strBuf.readInt();
+ assert bloblen > 0; // a Blob address MUST have a size
+
+ immediateFree(nxtAddr, bloblen);
} else {
immediateFree(nxtAddr, 0); // size ignored for FreeAllocators
}
+
+ nxtAddr = strBuf.readInt();
}
m_lastDeferredReleaseTime = lastReleaseTime;
} catch (IOException e) {
@@ -2840,11 +2785,6 @@
}
}
- private JournalTransactionService m_transactionService = null;
- public void setTransactionService(final JournalTransactionService transactionService) {
- this.m_transactionService = transactionService;
- }
-
/**
* The ContextAllocation object manages a freeList of associated allocators
* and an overall list of allocators. When the context is detached, all
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/journal/AbstractInterruptsTestCase.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/journal/AbstractInterruptsTestCase.java 2010-11-01 12:32:36 UTC (rev 3851)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/journal/AbstractInterruptsTestCase.java 2010-11-01 12:52:00 UTC (rev 3852)
@@ -314,7 +314,7 @@
} else if (store instanceof RWStrategy) {
RWStrategy rws = (RWStrategy)store;
- rws.commit();
+ rws.commit(null);
}
try {
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/journal/TestWORMStrategy.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/journal/TestWORMStrategy.java 2010-11-01 12:32:36 UTC (rev 3851)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/journal/TestWORMStrategy.java 2010-11-01 12:52:00 UTC (rev 3852)
@@ -240,7 +240,7 @@
properties.setProperty(Options.WRITE_CACHE_ENABLED, ""
+ writeCacheEnabled);
- return new Journal(properties).getBufferStrategy();
+ return new Journal(properties);
}
Modified: branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/rwstore/TestRWJournal.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/rwstore/TestRWJournal.java 2010-11-01 12:32:36 UTC (rev 3851)
+++ branches/JOURNAL_HA_BRANCH/bigdata/src/test/com/bigdata/rwstore/TestRWJournal.java 2010-11-01 12:52:00 UTC (rev 3852)
@@ -849,8 +849,10 @@
// since deferred frees, we must commit in order to ensure the
// address in invalid, indicating it is available for
- bs.commit();
+ store.commit();
+ rw.checkDeferredFrees(true, store);
+
try {
rdBuf = bs.read(faddr); // should fail with illegal state
throw new RuntimeException("Fail");
@@ -1147,7 +1149,8 @@
properties.setProperty(Options.WRITE_CACHE_ENABLED, ""
+ writeCacheEnabled);
- return new Journal(properties).getBufferStrategy();
+ // return new Journal(properties).getBufferStrategy();
+ return new Journal(properties);
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <dm...@us...> - 2010-11-01 12:32:43
|
Revision: 3851
http://bigdata.svn.sourceforge.net/bigdata/?rev=3851&view=rev
Author: dmacgbr
Date: 2010-11-01 12:32:36 +0000 (Mon, 01 Nov 2010)
Log Message:
-----------
Remove the resource leak, i.e. the thread 'com.bigdata.bop.engine.QueryEngine.engineService1', encountered during unit test runs. The problem is caused by the thread being statically cached in QueryEngineFactory using an instance of IIndexManager as a key. Each test typically has a unique IIndexManager. Code has been added to remove the instance in the '__tearDownUnitTest()' method of BigdataSail. Similar code has been added to a number of individual test classes because a) '__tearDownUnitTest ()' is not public, b) interactions with other tear down code.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/QueryEngineFactory.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSail.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataConnectionTest.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataEmbeddedFederationSparqlTest.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataFederationSparqlTest.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataSparqlTest.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataStoreTest.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/QueryEngineFactory.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/QueryEngineFactory.java 2010-10-29 12:22:15 UTC (rev 3850)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/QueryEngineFactory.java 2010-11-01 12:32:36 UTC (rev 3851)
@@ -87,6 +87,22 @@
}
/**
+ * Removes a QueryEngine instance from the cache if it is present, returning it to the caller. This
+ * method is unlikely to be useful in applications but the unit test framework requires it in order
+ * to avoid resource starvation as each test typically creates a unique IIndexManager.
+ *
+ * @param indexManager the database
+ * @return the query controller if present, null otherwise.
+ */
+ public static QueryEngine removeQueryController ( final IIndexManager indexManager )
+ {
+ if (indexManager instanceof IBigdataFederation<?>) {
+ return federationQECache.remove ( ( IBigdataFederation<?> )indexManager ) ;
+ }
+ return standaloneQECache.remove ( ( Journal )indexManager ) ;
+ }
+
+ /**
* Singleton factory for standalone.
*
* @param indexManager
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSail.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSail.java 2010-10-29 12:22:15 UTC (rev 3850)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSail.java 2010-11-01 12:32:36 UTC (rev 3851)
@@ -130,6 +130,8 @@
import com.bigdata.rdf.rio.StatementBuffer;
import com.bigdata.rdf.rules.BackchainAccessPath;
import com.bigdata.rdf.rules.InferenceEngine;
+import com.bigdata.rdf.sail.BigdataSail.BigdataSailConnection;
+import com.bigdata.rdf.sail.BigdataSail.Options;
import com.bigdata.rdf.spo.ExplicitSPOFilter;
import com.bigdata.rdf.spo.ISPO;
import com.bigdata.rdf.spo.InferredSPOFilter;
@@ -1049,7 +1051,9 @@
try {
shutDown();
-
+ QueryEngine qe = QueryEngineFactory.getQueryController(database.getIndexManager());
+ if ( null != qe )
+ qe.shutdownNow () ;
database.__tearDownUnitTest();
} catch (Throwable t) {
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataConnectionTest.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataConnectionTest.java 2010-10-29 12:22:15 UTC (rev 3850)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataConnectionTest.java 2010-11-01 12:32:36 UTC (rev 3851)
@@ -35,12 +35,14 @@
import org.openrdf.repository.Repository;
import org.openrdf.repository.RepositoryConnectionTest;
+import com.bigdata.bop.engine.QueryEngine;
+import com.bigdata.bop.fed.QueryEngineFactory;
import com.bigdata.btree.keys.CollatorEnum;
import com.bigdata.btree.keys.StrengthEnum;
import com.bigdata.journal.IIndexManager;
import com.bigdata.rdf.sail.BigdataSail;
+import com.bigdata.rdf.sail.BigdataSail.Options;
import com.bigdata.rdf.sail.BigdataSailRepository;
-import com.bigdata.rdf.sail.BigdataSail.Options;
import com.bigdata.rdf.store.LocalTripleStore;
public class BigdataConnectionTest extends RepositoryConnectionTest {
@@ -172,7 +174,12 @@
super.tearDown();
if (backend != null)
+ {
+ QueryEngine qe = QueryEngineFactory.removeQueryController ( backend ) ;
+ if ( null != qe )
+ qe.shutdownNow () ;
backend.destroy();
+ }
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataEmbeddedFederationSparqlTest.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataEmbeddedFederationSparqlTest.java 2010-10-29 12:22:15 UTC (rev 3850)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataEmbeddedFederationSparqlTest.java 2010-11-01 12:32:36 UTC (rev 3851)
@@ -41,11 +41,13 @@
import org.openrdf.repository.RepositoryException;
import org.openrdf.repository.dataset.DatasetRepository;
+import com.bigdata.bop.engine.QueryEngine;
+import com.bigdata.bop.fed.QueryEngineFactory;
import com.bigdata.journal.IIndexManager;
import com.bigdata.journal.ITx;
import com.bigdata.rdf.sail.BigdataSail;
+import com.bigdata.rdf.sail.BigdataSail.Options;
import com.bigdata.rdf.sail.BigdataSailRepository;
-import com.bigdata.rdf.sail.BigdataSail.Options;
import com.bigdata.rdf.store.AbstractTripleStore;
import com.bigdata.rdf.store.ScaleOutTripleStore;
import com.bigdata.resources.ResourceManager;
@@ -220,7 +222,10 @@
}
protected void tearDownBackend(IIndexManager backend) {
-
+ QueryEngine qe = QueryEngineFactory.removeQueryController ( backend ) ;
+ if ( null != qe )
+ qe.shutdownNow () ;
+
backend.destroy();
if (client != null) {
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataFederationSparqlTest.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataFederationSparqlTest.java 2010-10-29 12:22:15 UTC (rev 3850)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataFederationSparqlTest.java 2010-11-01 12:32:36 UTC (rev 3851)
@@ -38,14 +38,16 @@
import org.openrdf.repository.RepositoryConnection;
import org.openrdf.repository.dataset.DatasetRepository;
+import com.bigdata.bop.engine.QueryEngine;
+import com.bigdata.bop.fed.QueryEngineFactory;
import com.bigdata.btree.keys.CollatorEnum;
import com.bigdata.btree.keys.KeyBuilder;
import com.bigdata.btree.keys.StrengthEnum;
import com.bigdata.journal.ITx;
import com.bigdata.rdf.sail.BigdataSail;
+import com.bigdata.rdf.sail.BigdataSail.Options;
import com.bigdata.rdf.sail.BigdataSailRepository;
import com.bigdata.rdf.sail.BigdataSailRepositoryConnection;
-import com.bigdata.rdf.sail.BigdataSail.Options;
import com.bigdata.rdf.store.ScaleOutTripleStore;
import com.bigdata.service.jini.JiniClient;
import com.bigdata.service.jini.JiniFederation;
@@ -128,6 +130,12 @@
throws Exception
{
super.tearDown () ;
+ if ( null != _sail )
+ {
+ QueryEngine qe = QueryEngineFactory.removeQueryController ( _sail.getDatabase ().getIndexManager () ) ;
+ if ( null != qe )
+ qe.shutdownNow () ;
+ }
if (_ts != null) {
_ts.destroy();
_ts = null;
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataSparqlTest.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataSparqlTest.java 2010-10-29 12:22:15 UTC (rev 3850)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataSparqlTest.java 2010-11-01 12:32:36 UTC (rev 3851)
@@ -28,7 +28,6 @@
package com.bigdata.rdf.sail.tck;
import info.aduna.io.IOUtil;
-import info.aduna.iteration.Iterations;
import java.io.InputStream;
import java.io.InputStreamReader;
@@ -37,38 +36,29 @@
import java.util.Collection;
import java.util.Enumeration;
import java.util.Properties;
-import java.util.Set;
import junit.framework.Test;
import junit.framework.TestSuite;
import org.apache.log4j.Logger;
-import org.openrdf.model.Statement;
-import org.openrdf.query.BooleanQuery;
import org.openrdf.query.Dataset;
-import org.openrdf.query.GraphQuery;
-import org.openrdf.query.GraphQueryResult;
-import org.openrdf.query.Query;
-import org.openrdf.query.QueryLanguage;
-import org.openrdf.query.TupleQuery;
-import org.openrdf.query.TupleQueryResult;
import org.openrdf.query.parser.sparql.ManifestTest;
import org.openrdf.query.parser.sparql.SPARQLQueryTest;
import org.openrdf.repository.Repository;
-import org.openrdf.repository.RepositoryConnection;
import org.openrdf.repository.RepositoryException;
-import org.openrdf.repository.RepositoryResult;
import org.openrdf.repository.dataset.DatasetRepository;
import org.openrdf.repository.sail.SailRepository;
import org.openrdf.sail.memory.MemoryStore;
+import com.bigdata.bop.engine.QueryEngine;
+import com.bigdata.bop.fed.QueryEngineFactory;
import com.bigdata.btree.keys.CollatorEnum;
import com.bigdata.btree.keys.StrengthEnum;
import com.bigdata.journal.BufferMode;
import com.bigdata.journal.IIndexManager;
import com.bigdata.rdf.sail.BigdataSail;
-import com.bigdata.rdf.sail.BigdataSailRepository;
import com.bigdata.rdf.sail.BigdataSail.Options;
+import com.bigdata.rdf.sail.BigdataSailRepository;
/**
* Test harness for running the SPARQL test suites.
@@ -426,7 +416,9 @@
protected void tearDownBackend(IIndexManager backend) {
backend.destroy();
-
+ QueryEngine qe = QueryEngineFactory.removeQueryController ( backend ) ;
+ if ( null != qe )
+ qe.shutdownNow () ;
}
@Override
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataStoreTest.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataStoreTest.java 2010-10-29 12:22:15 UTC (rev 3850)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataStoreTest.java 2010-11-01 12:32:36 UTC (rev 3851)
@@ -37,6 +37,8 @@
import org.openrdf.sail.SailConnection;
import org.openrdf.sail.SailException;
+import com.bigdata.bop.engine.QueryEngine;
+import com.bigdata.bop.fed.QueryEngineFactory;
import com.bigdata.btree.keys.CollatorEnum;
import com.bigdata.btree.keys.StrengthEnum;
import com.bigdata.journal.IIndexManager;
@@ -123,7 +125,12 @@
super.tearDown();
if (backend != null)
+ {
+ QueryEngine qe = QueryEngineFactory.removeQueryController ( backend ) ;
+ if ( null != qe )
+ qe.shutdownNow () ;
backend.destroy();
+ }
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <btm...@us...> - 2010-10-29 12:22:25
|
Revision: 3850
http://bigdata.svn.sourceforge.net/bigdata/?rev=3850&view=rev
Author: btmurphy
Date: 2010-10-29 12:22:15 +0000 (Fri, 29 Oct 2010)
Log Message:
-----------
CHECKPOINT - phase 1 of callable executor (client) smart proxy work. Compiles cleanly, but still need to work on debugging unit tests as well as both deployment mechanisms (checkpointing for safety)
Modified Paths:
--------------
branches/dev-btm/bigdata/src/java/com/bigdata/bfs/BigdataFileSystem.java
branches/dev-btm/bigdata/src/java/com/bigdata/bfs/GlobalFileSystemHelper.java
branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexSegment.java
branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexSegmentStore.java
branches/dev-btm/bigdata/src/java/com/bigdata/counters/LoadBalancerReportingTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/counters/httpd/HttpReportingServer.java
branches/dev-btm/bigdata/src/java/com/bigdata/event/EventQueueSenderTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/AbstractTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/ConcurrencyManager.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/IIndexManager.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/IResourceManager.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/Journal.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/LocalTransactionManager.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/TemporaryStore.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/TemporaryStoreFactory.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/AbstractRelation.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/AbstractResource.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/AbstractStepTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/IJoinNexus.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/IJoinNexusFactory.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/MutationTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/ProgramTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/QueryTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/pipeline/DistributedJoinMasterTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/pipeline/DistributedJoinTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/pipeline/JoinTaskFactoryTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/pipeline/JoinTaskSink.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/pipeline/UnsyncDistributedOutputBuffer.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/AsynchronousOverflowTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/CompactingMergeTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/IncrementalBuildTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/JoinIndexPartitionTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/MoveTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/OverflowManager.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/ResourceManager.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/ScatterSplitTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/SplitIndexPartitionTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/SplitTailTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/StoreManager.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/ViewMetadata.java
branches/dev-btm/bigdata/src/java/com/bigdata/search/FullTextIndex.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/AbstractFederation.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/AbstractScaleOutFederation.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/CacheOnceMetadataIndex.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/CachingMetadataIndex.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/CallableExecutor.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ClientService.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/DataService.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/DataTaskWrapper.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/EmbeddedFederation.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/IBigdataFederation.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/IClientService.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/IClientServiceCallable.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/IDataServiceCallable.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/IFederationCallable.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/IndexCache.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ListIndicesTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/MetadataIndexCache.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/NoCacheMetadataIndexView.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/AbstractDataServiceProcedureTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/AbstractScaleOutClientIndexView.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/AbstractScaleOutClientIndexView2.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/ClientIndexView.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/ClientIndexViewRefactor.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/IScaleOutClientIndex.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/KeyArrayDataServiceProcedureTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/KeyRangeDataServiceProcedureTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/PartitionedTupleIterator.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/ScaleOutIndexCounters.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/SimpleDataServiceProcedureTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/pipeline/AbstractPendingSetMasterStats.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/pipeline/AbstractPendingSetMasterTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/pipeline/AbstractRunnableMasterStats.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/pipeline/IndexAsyncWriteStats.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/pipeline/IndexWriteTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/util/config/ConfigurationUtil.java
branches/dev-btm/bigdata/src/java/com/bigdata/util/config/Log4jLoggingHandler.java
branches/dev-btm/bigdata/src/java/com/bigdata/util/config/LogUtil.java
branches/dev-btm/bigdata/src/java/com/bigdata/util/config/NicUtil.java
branches/dev-btm/bigdata/src/test/com/bigdata/bfs/AbstractRepositoryTestCase.java
branches/dev-btm/bigdata/src/test/com/bigdata/relation/locator/TestDefaultResourceLocator.java
branches/dev-btm/bigdata/src/test/com/bigdata/relation/rule/eval/TestDefaultEvaluationPlan.java
branches/dev-btm/bigdata/src/test/com/bigdata/resources/AbstractResourceManagerTestCase.java
branches/dev-btm/bigdata/src/test/com/bigdata/resources/TestAddDeleteResource.java
branches/dev-btm/bigdata/src/test/com/bigdata/resources/TestBuildTask2.java
branches/dev-btm/bigdata/src/test/com/bigdata/resources/TestResourceManagerBootstrap.java
branches/dev-btm/bigdata/src/test/com/bigdata/search/TestPrefixSearch.java
branches/dev-btm/bigdata/src/test/com/bigdata/search/TestSearch.java
branches/dev-btm/bigdata/src/test/com/bigdata/search/TestSearchRestartSafe.java
branches/dev-btm/bigdata/src/test/com/bigdata/service/TestEmbeddedClient.java
branches/dev-btm/bigdata/src/test/com/bigdata/service/TestEventParser.java
branches/dev-btm/bigdata/src/test/com/bigdata/service/TestEventReceiver.java
branches/dev-btm/bigdata/src/test/com/bigdata/striterator/TestDistinctFilter.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/executor/AdminProxy.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/executor/Constants.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/executor/PrivateInterface.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/executor/ServiceImpl.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/executor/ServiceProxy.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/executor/config/executor.config
branches/dev-btm/bigdata-jini/src/java/com/bigdata/executor/config/logging.properties
branches/dev-btm/bigdata-jini/src/java/com/bigdata/jini/start/config/BigdataServiceConfiguration.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/jini/start/config/ClientServerConfiguration.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/jini/start/config/DataServerConfiguration.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/jini/start/config/MaxClientServicesPerHostConstraint.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/jini/start/config/ServicesManagerConfiguration.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/metadata/Constants.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/metadata/EmbeddedShardLocator.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/metadata/ServiceImpl.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/ClientServer.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/JiniFederation.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/LoadBalancerServer.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/benchmark/ThroughputMaster.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/lookup/BigdataCachingServiceClient.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/lookup/ClientServicesClient.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/lookup/DataServicesClient.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/lookup/LoadBalancerClient.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/lookup/ServicesManagerClient.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/lookup/ShardLocatorClient.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/lookup/TransactionServiceClient.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/AbstractClientTask.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/AggregatorTask.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/DiscoverServices.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/MappedTaskMaster.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/ResourceBufferStatistics.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/ResourceBufferSubtask.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/ResourceBufferTask.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/ServiceMap.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/TaskMaster.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/util/DumpFederation.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/util/JiniServicesHelper.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/shard/Constants.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/shard/EmbeddedShardService.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/shard/ServiceImpl.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/transaction/EmbeddedTransactionService.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/util/Util.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/util/config/ConfigDeployUtil.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/util/config/deploy.properties
branches/dev-btm/bigdata-jini/src/java/com/bigdata/zookeeper/ZooResourceLockService.java
branches/dev-btm/bigdata-jini/src/test/com/bigdata/service/jini/PerformanceTest.java
branches/dev-btm/bigdata-jini/src/test/com/bigdata/service/jini/master/TestMappedRDFDataLoadMaster.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/inf/BackchainOwlSameAsIterator.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/inf/OwlSameAsPropertiesExpandingIterator.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/inf/TruthMaintenance.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/lexicon/BigdataRDFFullTextIndex.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/lexicon/LexiconRelation.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/load/MappedRDFDataLoadMaster.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/load/MappedRDFFileLoadTask.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/magic/MagicRelation.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/magic/TempMagicStore.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/rules/InferenceEngine.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/rules/RDFJoinNexus.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/rules/RDFJoinNexusFactory.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPORelation.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractLocalTripleStore.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/store/LocalTripleStore.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/store/ScaleOutTripleStore.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/store/TempTripleStore.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/store/TripleStoreUtility.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/internal/constraints/TestInlineConstraints.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/magic/TestIRIS.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/magic/TestMagicStore.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/rio/EDSAsyncLoader.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/rio/TestRDFXMLInterchangeWithStatementIdentifiers.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/rules/AbstractRuleTestCase.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestDatabaseAtOnceClosure.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestJustifications.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestOptionals.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestRuleExpansion.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestSlice.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestTruthMaintenance.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/spo/TestSPORelation.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/spo/TestSPOStarJoin.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/store/AbstractDistributedTripleStoreTestCase.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/store/AbstractEmbeddedTripleStoreTestCase.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/store/TestScaleOutTripleStoreWithEmbeddedFederation.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/store/TestScaleOutTripleStoreWithJiniFederation.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/store/TestTempTripleStore.java
branches/dev-btm/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl2.java
branches/dev-btm/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSail.java
branches/dev-btm/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSailHelper.java
branches/dev-btm/bigdata-sails/src/test/com/bigdata/rdf/stress/LoadClosureAndQueryTest.java
Added Paths:
-----------
branches/dev-btm/bigdata/src/java/com/bigdata/discovery/
branches/dev-btm/bigdata/src/java/com/bigdata/discovery/IBigdataDiscoveryManagement.java
branches/dev-btm/bigdata/src/java/com/bigdata/event/EventQueueSender.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/IScaleOutIndexManager.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/IScaleOutIndexStore.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/ScaleOutIndexManager.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/ILocalResourceManagement.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/LocalResourceManager.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ClientTaskWrapper.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ForceOverflowTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/PurgeResourcesTask.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/executor/EmbeddedCallableExecutor.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/jini/BigdataDiscoveryManager.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/jini/IJiniDiscoveryManagement.java
Removed Paths:
-------------
branches/dev-btm/bigdata/src/java/com/bigdata/journal/EmbeddedIndexStore.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/LocalResourceManagement.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/DataServiceCallable.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/FederationCallable.java
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/bfs/BigdataFileSystem.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/bfs/BigdataFileSystem.java 2010-10-28 18:39:50 UTC (rev 3849)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/bfs/BigdataFileSystem.java 2010-10-29 12:22:15 UTC (rev 3850)
@@ -56,6 +56,10 @@
import cutthecrap.utils.striterators.Resolver;
import cutthecrap.utils.striterators.Striterator;
+//BTM - FOR_CLIENT_SERVICE
+import com.bigdata.discovery.IBigdataDiscoveryManagement;
+import com.bigdata.journal.IConcurrencyManager;
+
/**
* A distributed file system with extensible metadata and atomic append
* implemented using the bigdata scale-out architecture. Files have a client
@@ -346,10 +350,21 @@
*
* @see Options
*/
- public BigdataFileSystem(IIndexManager indexManager, String namespace,
- Long timestamp, Properties properties) {
-
- super(indexManager,namespace,timestamp,properties);
+//BTM - PRE_CLIENT_SERVICE - BEGIN
+//BTM - PRE_CLIENT_SERVICE public BigdataFileSystem(IIndexManager indexManager, String namespace,
+//BTM - PRE_CLIENT_SERVICE Long timestamp, Properties properties) {
+//BTM - PRE_CLIENT_SERVICE
+//BTM - PRE_CLIENT_SERVICE super(indexManager,namespace,timestamp,properties);
+ public BigdataFileSystem(IIndexManager indexManager,
+ IConcurrencyManager concurrencyManager,
+ IBigdataDiscoveryManagement discoveryManager,
+ String namespace,
+ Long timestamp,
+ Properties properties)
+ {
+ super(indexManager, concurrencyManager, discoveryManager,
+ namespace, timestamp, properties);
+//BTM - PRE_CLIENT_SERVICE - END
/*
* @todo This should probably be raised directly to a property reported
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/bfs/GlobalFileSystemHelper.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/bfs/GlobalFileSystemHelper.java 2010-10-28 18:39:50 UTC (rev 3849)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/bfs/GlobalFileSystemHelper.java 2010-10-29 12:22:15 UTC (rev 3850)
@@ -35,6 +35,10 @@
import com.bigdata.journal.IIndexManager;
import com.bigdata.journal.ITx;
+//BTM - FOR_CLIENT_SERVICE
+import com.bigdata.discovery.IBigdataDiscoveryManagement;
+import com.bigdata.journal.IConcurrencyManager;
+
/**
* Helper class.
*
@@ -63,7 +67,13 @@
/**
* The {@link ITx#UNISOLATED} view.
*/
- synchronized public BigdataFileSystem getGlobalFileSystem() {
+//BTM - PRE_CLIENT_SERVICE - BEGIN
+//BTM - PRE_CLIENT_SERVICE synchronized public BigdataFileSystem getGlobalFileSystem() {
+ synchronized public BigdataFileSystem getGlobalFileSystem
+ (final IConcurrencyManager concurrencyManager,
+ final IBigdataDiscoveryManagement discoveryManager)
+ {
+//BTM - PRE_CLIENT_SERVICE - END
if (INFO)
log.info("");
@@ -71,9 +81,18 @@
if (globalRowStore == null) {
// setup the repository view.
- globalRowStore = new BigdataFileSystem(indexManager,
- GLOBAL_FILE_SYSTEM_NAMESPACE, ITx.UNISOLATED,
- new Properties());
+//BTM - PRE_CLIENT_SERVICE - BEGIN
+//BTM - PRE_CLIENT_SERVICE globalRowStore = new BigdataFileSystem(indexManager,
+//BTM - PRE_CLIENT_SERVICE GLOBAL_FILE_SYSTEM_NAMESPACE, ITx.UNISOLATED,
+//BTM - PRE_CLIENT_SERVICE new Properties());
+ globalRowStore =
+ new BigdataFileSystem(indexManager,
+ concurrencyManager,
+ discoveryManager,
+ GLOBAL_FILE_SYSTEM_NAMESPACE,
+ ITx.UNISOLATED,
+ new Properties());
+//BTM - PRE_CLIENT_SERVICE - END
// register the indices.
globalRowStore.create();
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexSegment.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexSegment.java 2010-10-28 18:39:50 UTC (rev 3849)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexSegment.java 2010-10-29 12:22:15 UTC (rev 3850)
@@ -308,18 +308,33 @@
fileStore.lock.lock();
try {
- if (fileStore.fed != null) {
-
-//BTM openCloseEvent = new Event(fileStore.fed, new EventResource(
-//BTM fileStore.getIndexMetadata(), fileStore.file),
-//BTM EventType.IndexSegmentOpenClose);
-openCloseEvent = new Event( (fileStore.fed).getEventQueue(),
- (fileStore.fed).getServiceIface(),
- (fileStore.fed).getServiceName(),
- (fileStore.fed).getServiceUUID(),
- new EventResource(fileStore.getIndexMetadata(), fileStore.file),
- EventType.IndexSegmentOpenClose);
+//BTM - PRE_CLIENT_SERVICE - BEGIN
+//BTM - PRE_CLIENT_SERVICE if (fileStore.fed != null) {
+//BTM - PRE_CLIENT_SERVICE
+//BTM - PRE_CLIENT_SERVICE //BTM openCloseEvent = new Event(fileStore.fed, new EventResource(
+//BTM - PRE_CLIENT_SERVICE //BTM fileStore.getIndexMetadata(), fileStore.file),
+//BTM - PRE_CLIENT_SERVICE //BTM EventType.IndexSegmentOpenClose);
+//BTM - PRE_CLIENT_SERVICE
+//BTM - PRE_CLIENT_SERVICE openCloseEvent = new Event( (fileStore.fed).getEventQueue(),
+//BTM - PRE_CLIENT_SERVICE (fileStore.fed).getServiceIface(),
+//BTM - PRE_CLIENT_SERVICE (fileStore.fed).getServiceName(),
+//BTM - PRE_CLIENT_SERVICE (fileStore.fed).getServiceUUID(),
+//BTM - PRE_CLIENT_SERVICE new EventResource(fileStore.getIndexMetadata(), fileStore.file),
+//BTM - PRE_CLIENT_SERVICE EventType.IndexSegmentOpenClose);
+//BTM - PRE_CLIENT_SERVICE }
+//BTM - PRE_CLIENT_SERVICE
+ if (fileStore.localResourceManager != null) {
+ openCloseEvent =
+ new Event
+ ( (fileStore.localResourceManager).getEventQueueSender(),
+ (fileStore.localResourceManager).getServiceIface(),
+ (fileStore.localResourceManager).getServiceName(),
+ (fileStore.localResourceManager).getServiceUUID(),
+ new EventResource(fileStore.getIndexMetadata(),
+ fileStore.file),
+ EventType.IndexSegmentOpenClose);
}
+//BTM - PRE_CLIENT_SERVICE - END
if (!fileStore.isOpen()) {
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexSegmentStore.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexSegmentStore.java 2010-10-28 18:39:50 UTC (rev 3849)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexSegmentStore.java 2010-10-29 12:22:15 UTC (rev 3850)
@@ -59,9 +59,12 @@
import com.bigdata.service.Event;
import com.bigdata.service.EventResource;
import com.bigdata.service.EventType;
-import com.bigdata.service.IBigdataFederation;
+//BTM - PRE_CLIENT_SERVICE import com.bigdata.service.IBigdataFederation;
import com.bigdata.service.ResourceService;
+//BTM - FOR_CLIENT_SERVICE
+import com.bigdata.resources.ILocalResourceManagement;
+
/**
* A read-only store backed by a file containing a single {@link IndexSegment}.
*
@@ -274,7 +277,10 @@
/**
* Optional. When defined, {@link Event}s are reported out.
*/
- protected final IBigdataFederation<?> fed;
+//BTM - PRE_CLIENT_SERVICE - BEGIN
+//BTM - PRE_CLIENT_SERVICE protected final IBigdataFederation<?> fed;
+ protected final ILocalResourceManagement localResourceManager;
+//BTM - PRE_CLIENT_SERVICE - END
private volatile Event openCloseEvent;
/**
@@ -299,7 +305,11 @@
*/
public IndexSegmentStore(final File file) {
- this(file, null/* fed */);
+//BTM - PRE_CLIENT_SERVICE - BEGIN
+//BTM - PRE_CLIENT_SERVICE this(file, null/* fed */);
+ this(file,
+ null);//ILocalResourceManagement
+//BTM - PRE_CLIENT_SERVICE - END
}
@@ -310,15 +320,27 @@
* @param file
* @param fed
*/
- public IndexSegmentStore(final File file, final IBigdataFederation<?> fed) {
-
- if (file == null)
- throw new IllegalArgumentException();
-
+//BTM - PRE_CLIENT_SERVICE - BEGIN
+//BTM - PRE_CLIENT_SERVICE public IndexSegmentStore(final File file, final IBigdataFederation<?> fed) {
+//BTM - PRE_CLIENT_SERVICE
+//BTM - PRE_CLIENT_SERVICE if (file == null)
+//BTM - PRE_CLIENT_SERVICE throw new IllegalArgumentException();
+//BTM - PRE_CLIENT_SERVICE
+//BTM - PRE_CLIENT_SERVICE this.file = file;
+//BTM - PRE_CLIENT_SERVICE
+//BTM - PRE_CLIENT_SERVICE // MAY be null.
+//BTM - PRE_CLIENT_SERVICE this.fed = fed;
+//BTM - PRE_CLIENT_SERVICE
+ public IndexSegmentStore
+ (final File file,
+ final ILocalResourceManagement localResourceManager)
+ {
+ if (file == null) {
+ throw new NullPointerException("null file");
+ }
this.file = file;
-
- // MAY be null.
- this.fed = fed;
+ this.localResourceManager = localResourceManager;//can be null
+//BTM - PRE_CLIENT_SERVICE - END
/*
* Mark as open so that we can use reopenChannel() and read(long addr)
@@ -445,16 +467,32 @@
counters.openCount++;
- if (fed != null) {
-
-//BTM openCloseEvent = new Event(fed, new EventResource(
-//BTM indexMetadata, file),
-//BTM EventType.IndexSegmentStoreOpenClose).start();
-openCloseEvent = new Event( fed.getEventQueue(), fed.getServiceIface(), fed.getServiceName(), fed.getServiceUUID(),
- new EventResource(indexMetadata, file),
- EventType.IndexSegmentStoreOpenClose).start();
-
+//BTM - PRE_CLIENT_SERVICE - BEGIN
+//BTM - PRE_CLIENT_SERVICE if (fed != null) {
+//BTM - PRE_CLIENT_SERVICE
+//BTM - PRE_CLIENT_SERVICE //BTM openCloseEvent = new Event(fed, new EventResource(
+//BTM - PRE_CLIENT_SERVICE //BTM indexMetadata, file),
+//BTM - PRE_CLIENT_SERVICE //BTM EventType.IndexSegmentStoreOpenClose).start();
+//BTM - PRE_CLIENT_SERVICE
+//BTM - PRE_CLIENT_SERVICE openCloseEvent = new Event( fed.getEventQueue(),
+//BTM - PRE_CLIENT_SERVICE fed.getServiceIface(),
+//BTM - PRE_CLIENT_SERVICE fed.getServiceName(),
+//BTM - PRE_CLIENT_SERVICE fed.getServiceUUID(),
+//BTM - PRE_CLIENT_SERVICE new EventResource(indexMetadata, file),
+//BTM - PRE_CLIENT_SERVICE EventType.IndexSegmentStoreOpenClose).start();
+//BTM - PRE_CLIENT_SERVICE }
+//BTM - PRE_CLIENT_SERVICE
+ if (localResourceManager != null) {
+ openCloseEvent =
+ new Event
+ ( localResourceManager.getEventQueueSender(),
+ localResourceManager.getServiceIface(),
+ localResourceManager.getServiceName(),
+ localResourceManager.getServiceUUID(),
+ new EventResource(indexMetadata, file),
+ EventType.IndexSegmentStoreOpenClose).start();
}
+//BTM - PRE_CLIENT_SERVICE - END
} catch (Throwable t) {
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/counters/LoadBalancerReportingTask.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/counters/LoadBalancerReportingTask.java 2010-10-28 18:39:50 UTC (rev 3849)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/counters/LoadBalancerReportingTask.java 2010-10-29 12:22:15 UTC (rev 3850)
@@ -25,14 +25,14 @@
package com.bigdata.counters;
import com.bigdata.counters.CounterSet;
+import com.bigdata.discovery.IBigdataDiscoveryManagement;
+import com.bigdata.journal.IConcurrencyManager;
import com.bigdata.rawstore.Bytes;
-import com.bigdata.service.IFederationDelegate;
+import com.bigdata.resources.ILocalResourceManagement;
+import com.bigdata.resources.ResourceManager;
import com.bigdata.service.LoadBalancer;
import com.bigdata.util.config.LogUtil;
-import net.jini.core.lookup.ServiceItem;
-import net.jini.lookup.LookupCache;
-
import org.apache.log4j.Level;
import org.apache.log4j.Logger;
@@ -47,34 +47,32 @@
*/
public class LoadBalancerReportingTask implements Runnable {
- private IFederationDelegate embeddedIndexStore;
- private UUID serviceUUID;
- private CounterSet serviceRoot;
- private LookupCache lbsCache;//for discovering lbs
- private LoadBalancer embeddedLbs;//for testing embedded fed
+ private ResourceManager resourceMgr;
+ private IConcurrencyManager concurrencyMgr;
+ private ILocalResourceManagement localResourceMgr;
+ private IBigdataDiscoveryManagement discoveryMgr;
private Logger logger;
public LoadBalancerReportingTask
- (IFederationDelegate embeddedIndexStore,
- UUID serviceUUID,
- CounterSet serviceRoot,
- LookupCache loadBalancerCache,
- LoadBalancer embeddedLoadBalancer,
+ (ResourceManager resourceMgr,
+ IConcurrencyManager concurrencyMgr,
+ ILocalResourceManagement localResourceMgr,
+ IBigdataDiscoveryManagement discoveryMgr,
Logger logger)
{
- this.embeddedIndexStore = embeddedIndexStore;
- this.serviceUUID = serviceUUID;
- this.serviceRoot = serviceRoot;
- this.lbsCache = loadBalancerCache;
- this.embeddedLbs = embeddedLoadBalancer;//for embedded fed testing
- this.logger = (logger == null ?
- LogUtil.getLog4jLogger((this.getClass()).getName()) :
- logger);
+ this.resourceMgr = resourceMgr;
+ this.concurrencyMgr = concurrencyMgr;
+ this.localResourceMgr = localResourceMgr;
+ this.discoveryMgr = discoveryMgr;
+ this.logger = (logger == null ?
+ LogUtil.getLog4jLogger((this.getClass()).getName()) :
+ logger);
}
public void run() {
try {
- embeddedIndexStore.reattachDynamicCounters();
+ localResourceMgr.reattachDynamicCounters
+ (resourceMgr, concurrencyMgr);
} catch (Throwable t) {
logger.error
("failure on dynamic counter reattachment ["+t+"]", t);
@@ -89,32 +87,22 @@
}
private void reportPerformanceCounters() throws IOException {
-System.out.println("\n>>>>> LoadBalancerReportingTask.reportPerformanceCounters: serviceUUID = "+serviceUUID);
- LoadBalancer lbs = null;
- if(embeddedLbs != null) {
- lbs = embeddedLbs;
- } else {
- if(lbsCache != null) {
- ServiceItem lbsItem = lbsCache.lookup(null);
- if(lbsItem != null) {
- lbs = (LoadBalancer)(lbsItem.service);
- }
- }
- }
+//BTM
+System.out.println("\n>>>>> LoadBalancerReportingTask.reportPerformanceCounters: serviceUUID = "+localResourceMgr.getServiceUUID());
+ LoadBalancer lbs = discoveryMgr.getLoadBalancerService();
if(lbs == null) {
logger.warn
("cannot report counters [no load balancer service]");
System.out.println(">>>>> LoadBalancerReportingTask.reportPerformanceCounters: loadBalancerService = NULL");
return;
}
-
System.out.println(">>>>> LoadBalancerReportingTask.reportPerformanceCounters: loadBalancerService = "+lbs);
-
ByteArrayOutputStream baos = new ByteArrayOutputStream(Bytes.kilobyte32 * 2);
- serviceRoot.asXML(baos, "UTF-8", null/* filter */);
-
+ (localResourceMgr.getServiceCounterSet()).asXML(baos,
+ "UTF-8",
+ null);//filter
System.out.println(">>>>> LoadBalancerReportingTask.reportPerformanceCounters: CALLING loadBalancer.notify ...");
- lbs.notify(serviceUUID, baos.toByteArray());
+ lbs.notify(localResourceMgr.getServiceUUID(), baos.toByteArray());
System.out.println(">>>>> LoadBalancerReportingTask.reportPerformanceCounters: DONE CALLING loadBalancer.notify");
if (logger.isDebugEnabled()) {
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/counters/httpd/HttpReportingServer.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/counters/httpd/HttpReportingServer.java 2010-10-28 18:39:50 UTC (rev 3849)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/counters/httpd/HttpReportingServer.java 2010-10-29 12:22:15 UTC (rev 3850)
@@ -25,7 +25,9 @@
package com.bigdata.counters.httpd;
import com.bigdata.counters.CounterSet;
-import com.bigdata.service.IFederationDelegate;
+import com.bigdata.journal.IConcurrencyManager;
+import com.bigdata.resources.ILocalResourceManagement;
+import com.bigdata.resources.ResourceManager;
import com.bigdata.util.config.LogUtil;
import org.apache.log4j.Logger;
@@ -41,18 +43,23 @@
*/
public class HttpReportingServer extends CounterSetHTTPD {
- private IFederationDelegate embeddedIndexStore;
+ private ResourceManager resourceMgr;
+ private IConcurrencyManager concurrencyMgr;
+ private ILocalResourceManagement localResourceMgr;
private Logger logger;
public HttpReportingServer
(final int port,
- final CounterSet root,
- final IFederationDelegate embeddedIndexStore,
+ final ResourceManager resourceMgr,
+ final IConcurrencyManager concurrencyMgr,
+ final ILocalResourceManagement localResourceMgr,
Logger logger)
throws IOException
{
- super(port, root);
- this.embeddedIndexStore = embeddedIndexStore;
+ super(port, localResourceMgr.getServiceCounterSet());
+ this.resourceMgr = resourceMgr;
+ this.concurrencyMgr = concurrencyMgr;
+ this.localResourceMgr = localResourceMgr;
this.logger = (logger == null ?
LogUtil.getLog4jLogger((this.getClass()).getName()) :
logger);
@@ -67,7 +74,8 @@
throws Exception
{
try {
- embeddedIndexStore.reattachDynamicCounters();
+ localResourceMgr.reattachDynamicCounters
+ (resourceMgr, concurrencyMgr);
} catch (Exception e) {
// Usually because the live journal has been
// concurrently closed during the request.
Added: branches/dev-btm/bigdata/src/java/com/bigdata/discovery/IBigdataDiscoveryManagement.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/discovery/IBigdataDiscoveryManagement.java (rev 0)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/discovery/IBigdataDiscoveryManagement.java 2010-10-29 12:22:15 UTC (rev 3850)
@@ -0,0 +1,157 @@
+/*
+
+Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+*/
+package com.bigdata.discovery;
+
+import com.bigdata.journal.TransactionService;
+import com.bigdata.service.LoadBalancer;
+import com.bigdata.service.ShardLocator;
+import com.bigdata.service.ShardService;
+
+import java.util.UUID;
+
+/**
+ * Defines the interface for implementations that manage the discovery
+ * of the services in a Bigdata federation.
+ */
+public interface IBigdataDiscoveryManagement {
+
+ /**
+ * Returns a reference to a transaction service; or <code>null</code>
+ * if such a service has not been discovered.
+ *
+ * @return reference to a transaction service; or <code>null</code>
+ * if such a service has not been discovered.
+ */
+ TransactionService getTransactionService();
+
+ /**
+ * Returns a reference to a load balancer service; or <code>null</code>
+ * if such a service has not been discovered.
+ *
+ * @return reference to a load balancer service; or <code>null</code>
+ * if such a service has not been discovered.
+ */
+ LoadBalancer getLoadBalancerService();
+
+ /**
+ * Returns a reference to a shard locator (metadata) service; or
+ * <code>null</code> if such a service has not been discovered.
+ *
+ * @return reference to a shard locator (metadata) service; or
+ * <code>null</code> if such a service has not been discovered.
+ */
+ ShardLocator getMetadataService();
+
+ /**
+ * Returns an array whose elements are the UUIDs corresponding to
+ * a set of discovered shard (data) service(s).
+ *
+ * @param maxCount The maximum number of elements to return; where
+ * the number returned may be less than the value
+ * specified for <code>maxCount</code>, but will
+ * not be greater. Note that when zero (0) is
+ * input for this parameter, the UUIDs of all
+ * discovered shard (data) service(s) will be
+ * returned.
+ *
+ * @return An array of {@link UUID}s for data services.
+ *
+ * @throws IllegalArgumentException when a negative is input for
+ * the <code>maxCount</code> parameter.
+ */
+ UUID[] getDataServiceUUIDs(int maxCount);
+
+ /**
+ * Returns an array whose elements are references to shard (data)
+ * service(s) having UUID corresponding to an element of the
+ * <code>uuids</code> parameter. If no shard services exist (or can
+ * be discovered) that satisfy the given criteria, then an
+ * empty array is returned.
+ *
+ * @param uuids array whose elements are the UUIDs of the shard (data)
+ * service(s) to discover and return.
+ *
+ * @return array whose elements are references to shard (data)
+ * service(s) having UUID corresponding to an element of
+ * the <code>uuid</code> parameter; or <code>null</code>
+ * if no shard services exist (or can be discovered) that
+ * satisfy the given criteria.
+ *
+ * @throws NullPointerException if <code>null</code> is input for the
+ * <code>uuids</code> parameter.
+ */
+ ShardService[] getDataServices(UUID[] uuids);
+
+ /**
+ * Returns a reference to the shard (data) service whose corresponding
+ * UUID equals the <code>uuid</code> parameter. If no shard service
+ * exists (or can be discovered) that satisfies the given criteria,
+ * then <code>null</code> is returned.
+ *
+ * @param uuid the UUID of the shard (data) service to discover and
+ * return.
+ *
+ * @return reference to the shard (data) service whose corresponding
+ * UUID equals the <code>uuid</code> parameter; or
+ * <code>null</code> if no shard services exist (or can be
+ * discovered) that satisfy the given criteria.
+ *
+ * @throws NullPointerException if <code>null</code> is input for the
+ * <code>uuid</code> parameter.
+ */
+ ShardService getDataService(UUID uuid);
+
+ /**
+ * Returns a reference to the shard (data) service; where the criteria
+ * used to choose the service whose reference is returned is
+ * implementation-dependent. If no shard service exists (or can be
+ * discovered), then <code>null</code> is returned.
+ *
+ * @return reference to the shard (data) service; where the criteria
+ * used to choose the service whose reference is returned is
+ * implementation-dependent; or <code>null</code> if no shard
+ * services exist (or can be discovered).
+ */
+ ShardService getAnyDataService();
+
+ /**
+ * Returns a reference to the shard (data) service whose corresponding
+ * name attribute the value input for the <code>name</code> parameter.
+ * If no shard service exists (or can be discovered) that satisfies
+ * the given criteria, then <code>null</code> is returned.
+ *
+ * @param name the value of the name attribute for the shard (data)
+ * service to discover and return.
+ *
+ * @return reference to the shard (data) service whose corresponding
+ * name attribute the value input for the <code>name</code>
+ * parameter; or <code>null</code> if no shard services exist
+ * (or can be discovered) that satisfy the given criteria.
+ *
+ * @throws NullPointerException if <code>null</code> is input for the
+ * <code>name</code> parameter.
+ */
+ ShardService getDataServiceByName(String name);
+}
Property changes on: branches/dev-btm/bigdata/src/java/com/bigdata/discovery/IBigdataDiscoveryManagement.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: branches/dev-btm/bigdata/src/java/com/bigdata/event/EventQueueSender.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/event/EventQueueSender.java (rev 0)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/event/EventQueueSender.java 2010-10-29 12:22:15 UTC (rev 3850)
@@ -0,0 +1,35 @@
+/*
+
+Copyright (C) SYSTAP, LLC 2006-2008. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+*/
+
+package com.bigdata.event;
+
+/**
+ * Convenience interface that allows one to specify objects
+ * that can be used to both queue events for sending, and
+ * execute as tasks -- asynchronously -- that send the
+ * currently queued events to the desired event receiver.
+ */
+public interface EventQueueSender extends EventQueue, Runnable {
+}
Property changes on: branches/dev-btm/bigdata/src/java/com/bigdata/event/EventQueueSender.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/event/EventQueueSenderTask.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/event/EventQueueSenderTask.java 2010-10-28 18:39:50 UTC (rev 3849)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/event/EventQueueSenderTask.java 2010-10-29 12:22:15 UTC (rev 3850)
@@ -25,37 +25,31 @@
package com.bigdata.event;
+import com.bigdata.discovery.IBigdataDiscoveryManagement;
import com.bigdata.service.Event;
import com.bigdata.service.EventReceivingService;
-import com.bigdata.service.LoadBalancer;
import com.bigdata.util.config.LogUtil;
-import net.jini.core.lookup.ServiceItem;
-import net.jini.lookup.LookupCache;
-
import org.apache.log4j.Level;
import org.apache.log4j.Logger;
import java.util.LinkedList;
import java.util.concurrent.BlockingQueue;
-public class EventQueueSenderTask implements EventQueue, Runnable {
+public class EventQueueSenderTask implements EventQueueSender {
private BlockingQueue<Event> eventQueue;
- private LookupCache lbsCache;//for discovering lbs
- private LoadBalancer embeddedLbs;//for testing embedded fed
+ private IBigdataDiscoveryManagement discoveryMgr;
private String serviceName;
private Logger logger;
public EventQueueSenderTask(BlockingQueue<Event> eventQueue,
- LookupCache loadBalancerCache,
- LoadBalancer embeddedLoadBalancer,
+ IBigdataDiscoveryManagement discoveryMgr,
String serviceName,
Logger logger)
{
this.eventQueue = eventQueue;
- this.lbsCache = loadBalancerCache;
- this.embeddedLbs = embeddedLoadBalancer;//for embedded fed testing
+ this.discoveryMgr = discoveryMgr;
this.serviceName = serviceName;//for debug output
this.logger = (logger == null ?
@@ -63,6 +57,9 @@
logger);
}
+ // Note: EventQueueSender interface sub-classes EventQueue interface
+ // and Runnable interface
+
// Required by EventQueue interface
public void queueEvent(Event e) {
@@ -75,22 +72,9 @@
//BTM - for now, maintain the same logic and functionality as that in
//BTM the class AbstractFederation#SendEventsTask
-
try {
- LoadBalancer lbs = null;
- EventReceivingService serviceRef = null;
- if(embeddedLbs != null) {
- lbs = embeddedLbs;
- } else {
- if(lbsCache != null) {
- ServiceItem lbsItem = lbsCache.lookup(null);
- if(lbsItem != null) {
- lbs = (LoadBalancer)(lbsItem.service);
- }
- }
- }
- if(lbs == null) return;
- serviceRef = (EventReceivingService)lbs;
+ EventReceivingService serviceRef =
+ (EventReceivingService)(discoveryMgr.getLoadBalancerService());
final long begin = System.currentTimeMillis();//for logging
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/journal/AbstractTask.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/journal/AbstractTask.java 2010-10-28 18:39:50 UTC (rev 3849)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/journal/AbstractTask.java 2010-10-29 12:22:15 UTC (rev 3850)
@@ -2323,8 +2323,13 @@
&& isResource(namespace + "."+BigdataFileSystem.FILE_DATA_INDEX_BASENAME)) {
// unisolated view - will create if it does not exist.
- return new GlobalFileSystemHelper(this).getGlobalFileSystem();
-
+//BTM - PRE_CLIENT_SERVICE - BEGIN
+//BTM - PRE_CLIENT_SERVICE return new GlobalFileSystemHelper(this).getGlobalFileSystem();
+ return new GlobalFileSystemHelper(this)
+ .getGlobalFileSystem
+ (concurrencyManager,
+ resourceManager.getDiscoveryManager());
+//BTM - PRE_CLIENT_SERVICE - END
}
// read committed view IFF it exists otherwise [null]
@@ -2353,8 +2358,12 @@
*/
public TemporaryStore getTempStore() {
- return tempStoreFactory.getTempStore();
-
+//BTM - PRE_CLIENT_SERVICE - BEGIN
+//BTM - PRE_CLIENT_SERVICE return tempStoreFactory.getTempStore();
+ return tempStoreFactory.getTempStore
+ (concurrencyManager,
+ resourceManager.getDiscoveryManager());
+//BTM - PRE_CLIENT_SERVICE - END
}
private TemporaryStoreFactory tempStoreFactory = new TemporaryStoreFactory();
@@ -2727,8 +2736,12 @@
*/
public TemporaryStore getTempStore() {
- return tempStoreFactory.getTempStore();
-
+//BTM - PRE_CLIENT_SERVICE - BEGIN
+//BTM - PRE_CLIENT_SERVICE return tempStoreFactory.getTempStore();
+ return tempStoreFactory.getTempStore
+ (concurrencyManager,
+ resourceManager.getDiscoveryManager());
+//BTM - PRE_CLIENT_SERVICE - END
}
private TemporaryStoreFactory tempStoreFactory = new TemporaryStoreFactory();
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/journal/ConcurrencyManager.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/journal/ConcurrencyManager.java 2010-10-28 18:39:50 UTC (rev 3849)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/journal/ConcurrencyManager.java 2010-10-29 12:22:15 UTC (rev 3850)
@@ -38,6 +38,9 @@
import com.bigdata.util.concurrent.ThreadPoolExecutorStatisticsTask;
import com.bigdata.util.concurrent.WriteTaskCounters;
+//BTM - FOR_CLIENT_SERVICE
+import com.bigdata.journal.IScaleOutIndexStore;
+
/**
* Supports concurrent operations against named indices. Historical read and
* read-committed tasks run with full concurrency. For unisolated tasks, the
@@ -1228,7 +1231,8 @@
// And even then only for the distributed federation
try {
- if (!(resourceManager.getFederation() instanceof AbstractDistributedFederation)) {
+//BTM - PRE_CLIENT_SERVICE if (!(resourceManager.getFederation() instanceof AbstractDistributedFederation)) {
+ if (!(resourceManager.getIndexManager() instanceof IScaleOutIndexStore)) {
return 0;
}
} catch (UnsupportedOperationException ex) {
Deleted: branches/dev-btm/bigdata/src/java/com/bigdata/journal/EmbeddedIndexStore.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/journal/EmbeddedIndexStore.java 2010-10-28 18:39:50 UTC (rev 3849)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/journal/EmbeddedIndexStore.java 2010-10-29 12:22:15 UTC (rev 3850)
@@ -1,799 +0,0 @@
-/**
-
-Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
-
-package com.bigdata.journal;
-
-import com.bigdata.bfs.BigdataFileSystem;
-import com.bigdata.btree.IndexMetadata;
-import com.bigdata.counters.CounterSet;
-import com.bigdata.counters.ICounterSet;
-import com.bigdata.counters.IDataServiceCounters;
-import com.bigdata.counters.Instrument;
-import com.bigdata.counters.IProcessCounters;
-import com.bigdata.counters.IStatisticsCollector;
-import com.bigdata.counters.ReadBlockCounters;
-import com.bigdata.event.EventQueue;
-import com.bigdata.io.DirectBufferPool;
-import com.bigdata.jini.util.JiniUtil;
-import com.bigdata.journal.ConcurrencyManager;
-import com.bigdata.journal.ConcurrencyManager.IConcurrencyManagerCounters;
-import com.bigdata.journal.IResourceLockService;
-import com.bigdata.journal.LocalTransactionManager;
-import com.bigdata.journal.TemporaryStore;
-import com.bigdata.journal.TemporaryStoreFactory;
-import com.bigdata.journal.TransactionService;
-import com.bigdata.mdi.IMetadataIndex;
-import com.bigdata.relation.locator.DefaultResourceLocator;
-import com.bigdata.relation.locator.IResourceLocator;
-import com.bigdata.resources.IndexManager.IIndexManagerCounters;
-import com.bigdata.resources.LocalResourceManagement;
-import com.bigdata.resources.ResourceManager;
-import com.bigdata.resources.ResourceManager.IResourceManagerCounters;
-import com.bigdata.service.IBigdataClient;
-import com.bigdata.service.IBigdataFederation;
-import com.bigdata.service.IService;
-import com.bigdata.service.IServiceShutdown;
-import com.bigdata.service.LoadBalancer;
-import com.bigdata.service.MetadataIndexCache;
-import com.bigdata.service.MetadataIndexCachePolicy;
-import com.bigdata.service.ShardLocator;
-import com.bigdata.service.Service;
-import com.bigdata.service.ShardService;
-import com.bigdata.service.ndx.IClientIndex;
-import com.bigdata.sparse.SparseRowStore;
-import com.bigdata.util.Util;
-import com.bigdata.util.config.LogUtil;
-import com.bigdata.util.httpd.AbstractHTTPD;
-
-import net.jini.core.lookup.ServiceID;
-import net.jini.core.lookup.ServiceItem;
-import net.jini.lookup.LookupCache;
-
-import org.apache.log4j.Logger;
-
-import java.io.BufferedWriter;
-import java.io.File;
-import java.io.FileWriter;
-import java.io.IOException;
-import java.io.Writer;
-import java.util.HashSet;
-import java.util.Map;
-import java.util.Properties;
-import java.util.Set;
-import java.util.UUID;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.TimeUnit;
-
-//NOTE: replace IBigdataFederation with IIndexStore when
-// StoreManager.getResourceLocator is changed to no longer
-// call getFederation
-//
-// IBigdataFederation extends IIndexManager and IFederationDelegate
-// IIndexManager extends IIndexStore
-public class EmbeddedIndexStore<T> implements IBigdataFederation<T> {
-
- public static Logger logger =
- LogUtil.getLog4jLogger((EmbeddedIndexStore.class).getName());
-
- private UUID serviceUUID;
- private Class serviceType;
- private String serviceName;
- private String hostname;
- private LocalResourceManagement embeddedBackend;
- private CounterSet countersRoot;
- private IStatisticsCollector statisticsCollector;
- private Properties properties;
- private ReadBlockCounters readBlockCounters;
- private LocalTransactionManager localTxnMgr;
- private EventQueue eventQueue;
- private ExecutorService embeddedThreadPool;
- private String httpServerUrl;
- private DefaultResourceLocator resourceLocator;
-
- private IServiceShutdown service;
- private ResourceManager resourceMgr;
- private ConcurrencyManager concurrencyMgr;
-
- private long lastReattachMillis = 0L;
- private TemporaryStoreFactory tempStoreFactory;
-
- private MetadataIndexCache metadataIndexCache;
-
-private LookupCache lbsServiceCache;
-private LookupCache mdsServiceCache;
-private LookupCache shardCache;
-private LookupCache remoteShardCache;
-
-private LoadBalancer embeddedLbs;
-private ShardLocator embeddedMds;
-private Map<UUID, ShardService> embeddedDsMap;
-
- public EmbeddedIndexStore
- (UUID serviceUUID,
- Class serviceType,
- String serviceName,
- String hostname,
-LookupCache lbsServiceCache,
-LookupCache mdsServiceCache,
-LookupCache shardCache,
-LookupCache remoteShardCache,
-LoadBalancer embeddedLbs,
- LocalResourceManagement embeddedBackend,
- TemporaryStoreFactory tempStoreFactory,
- int indexCacheSize,
- long indexCacheTimeout,
- MetadataIndexCachePolicy metadataIndexCachePolicy,
- int resourceLocatorCacheSize,
- long resourceLocatorCacheTimeout,
- CounterSet countersRoot,
- IStatisticsCollector statisticsCollector,
- Properties properties,
- ReadBlockCounters readBlockCounters,
- LocalTransactionManager localTxnMgr,
- EventQueue eventQueue,
- ExecutorService embeddedThreadPool,
- String httpServerUrl)
- {
- this.serviceUUID = serviceUUID;
- this.serviceType = serviceType;
- this.serviceName = serviceName;
- this.hostname = hostname;
-this.lbsServiceCache = lbsServiceCache;
-this.mdsServiceCache = mdsServiceCache;
-this.shardCache = shardCache;
-this.remoteShardCache = remoteShardCac...
[truncated message content] |
4 messages has been excluded from this view by a project administrator.
