Menu
▾
▴
bigdata-commit — Commit traffic.
This list is closed, nobody may subscribe to it.
| 2010 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(139) |
Aug
(94) |
Sep
(232) |
Oct
(143) |
Nov
(138) |
Dec
(55) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2011 |
Jan
(127) |
Feb
(90) |
Mar
(101) |
Apr
(74) |
May
(148) |
Jun
(241) |
Jul
(169) |
Aug
(121) |
Sep
(157) |
Oct
(199) |
Nov
(281) |
Dec
(75) |
| 2012 |
Jan
(107) |
Feb
(122) |
Mar
(184) |
Apr
(73) |
May
(14) |
Jun
(49) |
Jul
(26) |
Aug
(103) |
Sep
(133) |
Oct
(61) |
Nov
(51) |
Dec
(55) |
| 2013 |
Jan
(59) |
Feb
(72) |
Mar
(99) |
Apr
(62) |
May
(92) |
Jun
(19) |
Jul
(31) |
Aug
(138) |
Sep
(47) |
Oct
(83) |
Nov
(95) |
Dec
(111) |
| 2014 |
Jan
(125) |
Feb
(60) |
Mar
(119) |
Apr
(136) |
May
(270) |
Jun
(83) |
Jul
(88) |
Aug
(30) |
Sep
(47) |
Oct
(27) |
Nov
(23) |
Dec
|
| 2015 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(3) |
Oct
|
Nov
|
Dec
|
| 2016 |
Jan
|
Feb
|
Mar
(4) |
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: <btm...@us...> - 2010-10-04 20:30:21
|
Revision: 3724
http://bigdata.svn.sourceforge.net/bigdata/?rev=3724&view=rev
Author: btmurphy
Date: 2010-10-04 20:30:13 +0000 (Mon, 04 Oct 2010)
Log Message:
-----------
[branch dev-btm]: hand-merged changeset 3476 from trunk --> dev-btm
Modified Paths:
--------------
branches/dev-btm/bigdata-perf/lubm/src/java/edu/lehigh/swat/bench/ubt/Test.java
Modified: branches/dev-btm/bigdata-perf/lubm/src/java/edu/lehigh/swat/bench/ubt/Test.java
===================================================================
--- branches/dev-btm/bigdata-perf/lubm/src/java/edu/lehigh/swat/bench/ubt/Test.java 2010-10-04 19:05:05 UTC (rev 3723)
+++ branches/dev-btm/bigdata-perf/lubm/src/java/edu/lehigh/swat/bench/ubt/Test.java 2010-10-04 20:30:13 UTC (rev 3724)
@@ -85,7 +85,7 @@
hostname = NicUtil.getIpAddress("default.nic", "default", false);
} catch(Throwable t) {//for now, maintain same failure logic as used previously
t.printStackTrace();
- s = NicUtil.getIpAddressByLocalHost();
+ hostname = NicUtil.getIpAddressByLocalHost();
}
QUERY_TEST_RESULT_FILE = hostname + "-result.txt";
} else {
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <btm...@us...> - 2010-10-04 19:05:11
|
Revision: 3723
http://bigdata.svn.sourceforge.net/bigdata/?rev=3723&view=rev
Author: btmurphy
Date: 2010-10-04 19:05:05 +0000 (Mon, 04 Oct 2010)
Log Message:
-----------
[branch dev-btm]: forgot to add new IDataServiceOnlyFilter class shared by the EmbeddedShardLocator and EmbeddedService classes
Added Paths:
-----------
branches/dev-btm/bigdata/src/java/com/bigdata/service/IDataServiceOnlyFilter.java
Added: branches/dev-btm/bigdata/src/java/com/bigdata/service/IDataServiceOnlyFilter.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/service/IDataServiceOnlyFilter.java (rev 0)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/service/IDataServiceOnlyFilter.java 2010-10-04 19:05:05 UTC (rev 3723)
@@ -0,0 +1,44 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+
+package com.bigdata.service;
+
+import net.jini.core.lookup.ServiceItem;
+import net.jini.lookup.ServiceItemFilter;
+
+public class IDataServiceOnlyFilter implements ServiceItemFilter {
+
+ public boolean check(ServiceItem item) {
+ if((item == null) || (item.service == null)) {
+ return false;
+ }
+ Class serviceType = (item.service).getClass();
+ boolean isIDataService =
+ (IDataService.class).isAssignableFrom(serviceType);
+ if( !isIDataService ) return false;
+ boolean isIMetadataService =
+ (IMetadataService.class).isAssignableFrom(serviceType);
+ return (isIDataService && !isIMetadataService);
+ }
+}
Property changes on: branches/dev-btm/bigdata/src/java/com/bigdata/service/IDataServiceOnlyFilter.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <btm...@us...> - 2010-10-04 18:57:20
|
Revision: 3722
http://bigdata.svn.sourceforge.net/bigdata/?rev=3722&view=rev
Author: btmurphy
Date: 2010-10-04 18:57:09 +0000 (Mon, 04 Oct 2010)
Log Message:
-----------
[branch dev-btm]: CHECKPOINT - phase 1 of the smart proxy implementation of the shard (data) service; which still includes numerous debug println's and BTM markers that will be cleaned up during phase 2. Also made appropriate additional changes to smart proxy implementations of the load balancer, the shard locator (metadata), and transaction services (from previous changesets), as required by the new smart proxy shard service. With these changes, either the ServicesManagerService mechanism or the BootManager mechanism can be used to start those service implementations. Note that this changeset also includes hand-merged changes from the trunk for the following changesets: 3481
Modified Paths:
--------------
branches/dev-btm/bigdata/src/java/com/bigdata/bfs/BigdataFileSystem.java
branches/dev-btm/bigdata/src/java/com/bigdata/btree/IRangeQuery.java
branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexMetadata.java
branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexSegment.java
branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexSegmentStore.java
branches/dev-btm/bigdata/src/java/com/bigdata/btree/ResultSet.java
branches/dev-btm/bigdata/src/java/com/bigdata/btree/proc/AbstractKeyArrayIndexProcedureConstructor.java
branches/dev-btm/bigdata/src/java/com/bigdata/btree/proc/IIndexProcedure.java
branches/dev-btm/bigdata/src/java/com/bigdata/config/Configuration.java
branches/dev-btm/bigdata/src/java/com/bigdata/counters/CounterSet.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/AbstractLocalTransactionManager.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/ConcurrencyManager.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/IBTreeManager.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/IResourceManager.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/ITransactionService.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/JournalTransactionService.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/RegisterIndexTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/Tx.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/WriteExecutorService.java
branches/dev-btm/bigdata/src/java/com/bigdata/mdi/IMetadataIndex.java
branches/dev-btm/bigdata/src/java/com/bigdata/mdi/IPartitionMetadata.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/accesspath/AbstractAccessPath.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/accesspath/IAccessPath.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/locator/DefaultResourceLocator.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/AbstractStepTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/IJoinNexus.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/MutationTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/ProgramTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/QueryTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/pipeline/DistributedJoinMasterTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/pipeline/DistributedJoinTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/pipeline/JoinTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/relation/rule/eval/pipeline/JoinTaskFactoryTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/AbstractResourceManagerTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/AsynchronousOverflowTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/CompactingMergeTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/IncrementalBuildTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/IndexManager.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/JoinIndexPartitionTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/MoveResult.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/MoveTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/OverflowManager.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/OverflowMetadata.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/ResourceManager.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/ScatterSplitTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/SplitIndexPartitionTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/SplitTailTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/StoreManager.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/AbstractFederation.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/AbstractIndexCache.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/AbstractRoundRobinServiceLoadHelper.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/AbstractScaleOutClient.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/AbstractScaleOutFederation.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/AbstractServiceLoadHelper.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/AbstractTransactionService.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/CacheOnceMetadataIndex.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/CachingMetadataIndex.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ClientService.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/DataService.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/DataServiceCallable.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/DistributedTransactionService.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/EmbeddedFederation.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/Event.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/IBigdataClient.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/IBigdataFederation.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/IClientService.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/IDataService.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/IDataServiceCallable.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/IRemoteExecutor.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/IServiceLoadHelper.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ITxCommitProtocol.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ListIndicesTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/LoadBalancerService.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/MetadataIndexCache.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/MetadataService.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/NoCacheMetadataIndexView.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ShardManagement.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ShardService.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/AbstractDataServiceProcedureTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/AbstractScaleOutClientIndexView.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/ClientIndexView.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/ClientIndexViewRefactor.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/DataServiceTupleIterator.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/IClientIndex.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/IScaleOutClientIndex.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/PartitionedTupleIterator.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/RawDataServiceTupleIterator.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/SimpleDataServiceProcedureTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/pipeline/AbstractPendingSetSubtask.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/pipeline/IndexPartitionWriteTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/ndx/pipeline/IndexWriteTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/util/config/NicUtil.java
branches/dev-btm/bigdata/src/resources/logging/log4j.properties
branches/dev-btm/bigdata/src/test/com/bigdata/journal/TestTransactionService.java
branches/dev-btm/bigdata/src/test/com/bigdata/resources/AbstractResourceManagerTestCase.java
branches/dev-btm/bigdata/src/test/com/bigdata/resources/MockTransactionService.java
branches/dev-btm/bigdata/src/test/com/bigdata/resources/TestAddDeleteResource.java
branches/dev-btm/bigdata/src/test/com/bigdata/resources/TestBuildTask2.java
branches/dev-btm/bigdata/src/test/com/bigdata/service/AbstractEmbeddedFederationTestCase.java
branches/dev-btm/bigdata/src/test/com/bigdata/service/StressTestConcurrent.java
branches/dev-btm/bigdata/src/test/com/bigdata/service/TestBasicIndexStuff.java
branches/dev-btm/bigdata/src/test/com/bigdata/service/TestDistributedTransactionService.java
branches/dev-btm/bigdata/src/test/com/bigdata/service/TestEmbeddedClient.java
branches/dev-btm/bigdata/src/test/com/bigdata/service/TestEventParser.java
branches/dev-btm/bigdata/src/test/com/bigdata/service/TestEventReceiver.java
branches/dev-btm/bigdata/src/test/com/bigdata/service/TestMove.java
branches/dev-btm/bigdata/src/test/com/bigdata/service/TestOverflow.java
branches/dev-btm/bigdata/src/test/com/bigdata/service/TestRangeQuery.java
branches/dev-btm/bigdata/src/test/com/bigdata/service/TestRangeQueryRemote.java
branches/dev-btm/bigdata/src/test/com/bigdata/service/TestRestartSafe.java
branches/dev-btm/bigdata/src/test/com/bigdata/service/TestScatterSplit.java
branches/dev-btm/bigdata/src/test/com/bigdata/service/TestSplitJoin.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/boot/config/boot-processes.xml
branches/dev-btm/bigdata-jini/src/java/com/bigdata/jini/start/BigdataZooDefs.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/jini/start/ManageLogicalServiceTask.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/jini/start/ServicesManagerServer.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/jini/start/config/BigdataServiceConfiguration.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/jini/start/config/DataServerConfiguration.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/jini/start/config/JiniServiceConfiguration.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/jini/start/config/MaxDataServicesPerHostConstraint.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/jini/start/config/ServicesManagerConfiguration.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/loadbalancer/EmbeddedLoadBalancer.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/loadbalancer/ServiceImpl.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/metadata/Constants.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/metadata/EmbeddedShardLocator.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/metadata/PrivateInterface.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/metadata/ServiceImpl.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/metadata/ServiceProxy.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/metadata/config/logging.properties
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/ClientServer.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/DataServer.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/JiniFederation.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/MetadataServer.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/benchmark/ThroughputMaster.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/lookup/BigdataCachingServiceClient.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/lookup/DataServicesClient.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/AbstractAsynchronousClientTask.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/AbstractClientTask.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/AbstractResourceScanner.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/DiscoverServices.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/IAsynchronousClientTask.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/IResourceScannerFactory.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/MappedTaskMaster.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/ResourceBufferTask.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/ServiceMap.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/TaskMaster.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/util/DumpFederation.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/util/JiniServicesHelper.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/shard/AdminProxy.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/shard/Constants.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/shard/PrivateInterface.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/shard/ServiceImpl.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/shard/ServiceProxy.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/shard/config/logging.properties
branches/dev-btm/bigdata-jini/src/java/com/bigdata/shard/config/shard.config
branches/dev-btm/bigdata-jini/src/java/com/bigdata/transaction/EmbeddedTransactionService.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/transaction/ServiceImpl.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/util/EntryUtil.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/util/Util.java
branches/dev-btm/bigdata-jini/src/resources/config/bigdataStandaloneTesting.config
branches/dev-btm/bigdata-jini/src/test/com/bigdata/service/jini/AbstractServerTestCase.java
branches/dev-btm/bigdata-jini/src/test/com/bigdata/service/jini/PerformanceTest.java
branches/dev-btm/bigdata-jini/src/test/com/bigdata/service/jini/TestBigdataClient.java
branches/dev-btm/bigdata-jini/src/test/com/bigdata/service/jini/master/TestMappedRDFDataLoadMaster.config
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/load/MappedRDFDataLoadMaster.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/load/MappedRDFFileLoadTask.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/rules/RDFJoinNexus.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPORelation.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java
branches/dev-btm/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java
branches/dev-btm/bigdata-rdf/src/test/com/bigdata/rdf/store/AbstractServerTestCase.java
branches/dev-btm/bigdata-sails/src/test/com/bigdata/rdf/stress/LoadClosureAndQueryTest.java
branches/dev-btm/src/resources/bin/config/browser.config
branches/dev-btm/src/resources/config/bigdataCluster.config
Added Paths:
-----------
branches/dev-btm/bigdata/src/java/com/bigdata/counters/IDataServiceCounters.java
branches/dev-btm/bigdata/src/java/com/bigdata/counters/LoadBalancerReportingTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/counters/ReadBlockCounters.java
branches/dev-btm/bigdata/src/java/com/bigdata/counters/httpd/HttpReportingServer.java
branches/dev-btm/bigdata/src/java/com/bigdata/event/
branches/dev-btm/bigdata/src/java/com/bigdata/event/EventQueue.java
branches/dev-btm/bigdata/src/java/com/bigdata/event/EventQueueSenderTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/DistributedCommitTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/EmbeddedIndexStore.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/GetIndexMetadataTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/LocalTransactionManager.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/RangeIteratorTask.java
branches/dev-btm/bigdata/src/java/com/bigdata/journal/TxState.java
branches/dev-btm/bigdata/src/java/com/bigdata/resources/LocalResourceManagement.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/DataTaskWrapper.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/IClientServiceCallable.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/MetadataIndexCachePolicy.java
branches/dev-btm/bigdata/src/java/com/bigdata/service/OverflowAdmin.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/service/jini/master/FileServer.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/shard/EmbeddedShardService.java
Removed Paths:
-------------
branches/dev-btm/bigdata/src/java/com/bigdata/service/TxState.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/metadata/EmbeddedIndexStore.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/metadata/TestAdmin.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/shard/TestAdmin.java
branches/dev-btm/bigdata-jini/src/java/com/bigdata/transaction/TestAdmin.java
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/bfs/BigdataFileSystem.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/bfs/BigdataFileSystem.java 2010-10-04 13:27:07 UTC (rev 3721)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/bfs/BigdataFileSystem.java 2010-10-04 18:57:09 UTC (rev 3722)
@@ -44,7 +44,6 @@
import com.bigdata.relation.locator.DefaultResourceLocator;
import com.bigdata.search.FullTextIndex;
import com.bigdata.service.IBigdataFederation;
-import com.bigdata.service.IDataService;
import com.bigdata.sparse.AutoIncIntegerCounter;
import com.bigdata.sparse.IRowStoreConstants;
import com.bigdata.sparse.ITPS;
@@ -259,12 +258,12 @@
* <p>
* Note: The {@link BigdataFileSystem} makes the <strong>assumption</strong>
* that the {@link com.bigdata.journal.Options#OFFSET_BITS} is the #of
- * offset bits configured for the {@link IDataService}s in the connected
+ * offset bits configured for the shard services in the connected
* {@link IBigdataFederation} and computes the
* {@link BigdataFileSystem#getBlockSize()} based on that assumption. It is
* NOT possible to write blocks on the {@link BigdataFileSystem} whose size
* is greater than the maximum block size actually configured for the
- * {@link IDataService}s in the connected {@link IBigdataFederation}.
+ * shard services in the connected {@link IBigdataFederation}.
*
* @see com.bigdata.journal.Options#OFFSET_BITS
* @see #getOffsetBits()
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/btree/IRangeQuery.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/btree/IRangeQuery.java 2010-10-04 13:27:07 UTC (rev 3721)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/btree/IRangeQuery.java 2010-10-04 18:57:09 UTC (rev 3722)
@@ -36,7 +36,6 @@
import com.bigdata.btree.filter.ITupleFilter;
import com.bigdata.btree.filter.TupleRemover;
import com.bigdata.service.IBigdataClient;
-import com.bigdata.service.IDataService;
import com.bigdata.service.ndx.IClientIndex;
import cutthecrap.utils.striterators.Striterator;
@@ -190,9 +189,9 @@
* <p>
* Note: This semantics of this flag require that the entries are atomically
* removed within the isolation level of the operation. In particular, if
- * the iterator is running against an {@link IDataService} using an
- * unisolated view then the entries MUST be buffered and removed as the
- * {@link ResultSet} is populated.
+ * the iterator is running against a shard service using an unisolated view
+ * then the entries MUST be buffered and removed as the {@link ResultSet}
+ * is populated.
* <p>
* Note: The {@link BigdataFileSystem#deleteHead(String, int)} relies on
* this atomic guarantee.
@@ -274,8 +273,7 @@
* processing which benefits from order effects can exploit the within chunk
* ordering of the tuples. The semantics of this flag are realized by the
* {@link IClientIndex} implementation. This flag has no effect on an
- * {@link ILocalBTreeView} and is not passed through to the
- * {@link IDataService}.
+ * {@link ILocalBTreeView} and is not passed through to the shard service.
*
* @todo This flag is not supported in combination with {@link #CURSOR}?
* @todo This flag is not supported in combination with {@link #REVERSE}?
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexMetadata.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexMetadata.java 2010-10-04 13:27:07 UTC (rev 3721)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexMetadata.java 2010-10-04 18:57:09 UTC (rev 3722)
@@ -65,12 +65,17 @@
import com.bigdata.resources.OverflowManager;
import com.bigdata.resources.StaleLocatorException;
import com.bigdata.service.AbstractFederation;
-import com.bigdata.service.DataService;
+//BTM import com.bigdata.service.DataService;
import com.bigdata.service.IBigdataFederation;
-import com.bigdata.service.IDataService;
+//BTM import com.bigdata.service.IDataService;
import com.bigdata.service.ndx.pipeline.AbstractSubtask;
import com.bigdata.sparse.SparseRowStore;
+//BTM
+import com.bigdata.service.IService;
+import com.bigdata.service.Service;
+import com.bigdata.service.ShardService;
+
/**
* <p>
* The persistent and mostly immutable metadata for a {@link AbstractBTree}.
@@ -904,7 +909,7 @@
* Boolean option indicates whether or not scatter splits are performed
* (default {@value #SCATTER_SPLIT_ENABLED}). Scatter splits only apply
* for scale-out indices where they "scatter" the initial index
- * partition across the {@link IDataService}s in the federation. This
+ * partition across the {@link ShardService}s in the federation. This
* is normally very useful.
* <P>
* Sometimes a scatter split is not the "right" thing for an index. An
@@ -1012,7 +1017,7 @@
}
/**
- * The {@link UUID} of the {@link DataService} on which the first partition
+ * The {@link UUID} of the {@link ShardService} on which the first partition
* of the scale-out index should be created. This is a purely transient
* property and will be <code>null</code> unless either explicitly set or
* set using {@value Options#INITIAL_DATA_SERVICE}. This property is only
@@ -1928,14 +1933,19 @@
final IBigdataFederation<?> fed = (IBigdataFederation<?>) indexManager;
- final IDataService dataService = fed
- .getDataServiceByName(val);
+//BTM final IDataService dataService = fed.getDataServiceByName(val);
+final ShardService dataService = fed.getDataServiceByName(val);
if (dataService != null) {
try {
- uuid = dataService.getServiceUUID();
+//BTM uuid = dataService.getServiceUUID();
+if(dataService instanceof IService) {
+ uuid = ((IService)dataService).getServiceUUID();
+} else {
+ uuid = ((Service)dataService).getServiceUUID();
+}
} catch (IOException ex) {
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexSegment.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexSegment.java 2010-10-04 13:27:07 UTC (rev 3721)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexSegment.java 2010-10-04 18:57:09 UTC (rev 3722)
@@ -310,10 +310,15 @@
if (fileStore.fed != null) {
- openCloseEvent = new Event(fileStore.fed, new EventResource(
- fileStore.getIndexMetadata(), fileStore.file),
- EventType.IndexSegmentOpenClose);
-
+//BTM openCloseEvent = new Event(fileStore.fed, new EventResource(
+//BTM fileStore.getIndexMetadata(), fileStore.file),
+//BTM EventType.IndexSegmentOpenClose);
+openCloseEvent = new Event( (fileStore.fed).getEventQueue(),
+ (fileStore.fed).getServiceIface(),
+ (fileStore.fed).getServiceName(),
+ (fileStore.fed).getServiceUUID(),
+ new EventResource(fileStore.getIndexMetadata(), fileStore.file),
+ EventType.IndexSegmentOpenClose);
}
if (!fileStore.isOpen()) {
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexSegmentStore.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexSegmentStore.java 2010-10-04 13:27:07 UTC (rev 3721)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/btree/IndexSegmentStore.java 2010-10-04 18:57:09 UTC (rev 3722)
@@ -447,8 +447,11 @@
if (fed != null) {
- openCloseEvent = new Event(fed, new EventResource(
- indexMetadata, file),
+//BTM openCloseEvent = new Event(fed, new EventResource(
+//BTM indexMetadata, file),
+//BTM EventType.IndexSegmentStoreOpenClose).start();
+openCloseEvent = new Event( fed.getEventQueue(), fed.getServiceIface(), fed.getServiceName(), fed.getServiceUUID(),
+ new EventResource(indexMetadata, file),
EventType.IndexSegmentStoreOpenClose).start();
}
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/btree/ResultSet.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/btree/ResultSet.java 2010-10-04 13:27:07 UTC (rev 3721)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/btree/ResultSet.java 2010-10-04 18:57:09 UTC (rev 3722)
@@ -55,7 +55,6 @@
import com.bigdata.io.FixedByteArrayBuffer;
import com.bigdata.journal.ITx;
import com.bigdata.mdi.IResourceMetadata;
-import com.bigdata.service.IDataService;
/**
* An object used to stream key scan results back to the client.
@@ -283,7 +282,7 @@
* The values returned by {@link ITuple#getSourceIndex()} may be used to
* identify the resource from which a given tuple was read. That information
* is used to direct {@link ITuple#readBlock(long)} requests to the correct
- * resource on the {@link IDataService}.
+ * resource on the shard service.
*/
final public IResourceMetadata[] getSources() {
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/btree/proc/AbstractKeyArrayIndexProcedureConstructor.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/btree/proc/AbstractKeyArrayIndexProcedureConstructor.java 2010-10-04 13:27:07 UTC (rev 3721)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/btree/proc/AbstractKeyArrayIndexProcedureConstructor.java 2010-10-04 18:57:09 UTC (rev 3722)
@@ -5,14 +5,12 @@
import com.bigdata.btree.ITupleSerializer;
import com.bigdata.btree.IndexMetadata;
import com.bigdata.btree.raba.codec.IRabaCoder;
-import com.bigdata.service.IDataService;
/**
* A factory for {@link IKeyArrayIndexProcedure}s so that their data may be key
* range partitions and mapped against each relevant index partition.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
*/
public abstract class AbstractKeyArrayIndexProcedureConstructor<T extends IKeyArrayIndexProcedure> {
@@ -29,7 +27,8 @@
*
* @param ndx
* The index - this is used to determine the serializers for the
- * keys and/or values to be sent to a remote {@link IDataService}.
+ * keys and/or values to be sent to the backend of the shard
+ * service.
* @param fromIndex
* The index of the first key to be used (inclusive).
* @param toIndex
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/btree/proc/IIndexProcedure.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/btree/proc/IIndexProcedure.java 2010-10-04 13:27:07 UTC (rev 3721)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/btree/proc/IIndexProcedure.java 2010-10-04 18:57:09 UTC (rev 3722)
@@ -34,8 +34,6 @@
import com.bigdata.btree.IIndex;
import com.bigdata.btree.IRangeQuery;
import com.bigdata.btree.ISimpleBTree;
-import com.bigdata.service.DataService;
-import com.bigdata.service.IDataService;
import com.bigdata.service.ndx.ClientIndexView;
import com.bigdata.sparse.SparseRowStore;
@@ -86,15 +84,15 @@
* </dl>
*
* Note: this interface extends {@link Serializable}, however that provides
- * only for communicating state to the {@link IDataService}. If an instance of
+ * only for communicating state to the shard service. If an instance of
* this procedure will cross a network interface, then the implementation class
- * MUST be available to the {@link IDataService} on which it will execute. This
+ * MUST be available to the shard service on which it will execute. This
* can be as simple as bundling the procedure into a JAR that is part of the
- * CLASSPATH used to start a {@link DataService} or you can use downloaded code
- * with the JINI codebase mechanism (<code>java.rmi.server.codebase</code>).
+ * CLASSPATH used to start the frontend data service or backend embedded
+ * shard service, or you can use downloaded code with the JINI codebase
+ * mechanism (<code>java.rmi.server.codebase</code>).
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
*
* @todo add generic type for {@link #apply(IIndex)} 's return value (much like
* {@link Callable}).
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/config/Configuration.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/config/Configuration.java 2010-10-04 13:27:07 UTC (rev 3721)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/config/Configuration.java 2010-10-04 18:57:09 UTC (rev 3722)
@@ -38,11 +38,16 @@
import com.bigdata.btree.IndexMetadata;
import com.bigdata.journal.IIndexManager;
import com.bigdata.relation.RelationSchema;
-import com.bigdata.service.DataService;
+//BTM import com.bigdata.service.DataService;
import com.bigdata.service.IBigdataFederation;
-import com.bigdata.service.IDataService;
+//BTM import com.bigdata.service.IDataService;
import com.bigdata.util.NV;
+//BTM
+import com.bigdata.service.IService;
+import com.bigdata.service.Service;
+import com.bigdata.service.ShardService;
+
/**
* Base class for managing the initial configuration metadata for indices and
* locatable resources.
@@ -317,7 +322,7 @@
// }
/**
- * Resolve the value to a {@link DataService} {@link UUID}.
+ * Resolve the value to a {@link ShardService} {@link UUID}.
*
* @param indexManager
* The index manager (optional).
@@ -365,14 +370,19 @@
*/
{
- final IDataService dataService = fed.getDataServiceByName(val);
+//BTM final IDataService dataService = fed.getDataServiceByName(val);
+final ShardService dataService = fed.getDataServiceByName(val);
if (dataService != null) {
try {
- return dataService.getServiceUUID();
-
+//BTM return dataService.getServiceUUID();
+if(dataService instanceof IService) {
+ return ((IService)dataService).getServiceUUID();
+} else {
+ return ((Service)dataService).getServiceUUID();
+}
} catch (IOException ex) {
throw new RuntimeException(ex);
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/counters/CounterSet.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/counters/CounterSet.java 2010-10-04 13:27:07 UTC (rev 3721)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/counters/CounterSet.java 2010-10-04 18:57:09 UTC (rev 3722)
@@ -785,6 +785,8 @@
// counter exists for that path.
log.error("Exists: path=" + getPath() + ", name=" + name);
+//BTM
+System.out.println("\n"+com.bigdata.util.Util.getCurrentStackTrace());
// return existing counter for path @todo vs replace.
return (ICounter)counter;
Added: branches/dev-btm/bigdata/src/java/com/bigdata/counters/IDataServiceCounters.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/counters/IDataServiceCounters.java (rev 0)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/counters/IDataServiceCounters.java 2010-10-04 18:57:09 UTC (rev 3722)
@@ -0,0 +1,53 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+
+package com.bigdata.counters;
+
+import com.bigdata.journal.ConcurrencyManager;
+import com.bigdata.resources.ResourceManager;
+
+/**
+ * Interface that defines and documents the counters and counter namespaces
+ * reported by the shard (data) service and the various services which it
+ * uses.
+ */
+public interface IDataServiceCounters
+ extends ConcurrencyManager.IConcurrencyManagerCounters,
+ ResourceManager.IResourceManagerCounters
+{
+ /**
+ * The namespace for the counters pertaining to the {@link ConcurrencyManager}.
+ */
+ String concurrencyManager = "Concurrency Manager";
+
+ /**
+ * The namespace for the counters pertaining to the {@link ILocalTransactionService}.
+ */
+ String transactionManager = "Transaction Manager";
+
+ /**
+ * The namespace for the counters pertaining to the {@link ResourceManager}.
+ */
+ String resourceManager = "Resource Manager";
+}
\ No newline at end of file
Property changes on: branches/dev-btm/bigdata/src/java/com/bigdata/counters/IDataServiceCounters.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: branches/dev-btm/bigdata/src/java/com/bigdata/counters/LoadBalancerReportingTask.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/counters/LoadBalancerReportingTask.java (rev 0)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/counters/LoadBalancerReportingTask.java 2010-10-04 18:57:09 UTC (rev 3722)
@@ -0,0 +1,124 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+
+package com.bigdata.counters;
+
+import com.bigdata.counters.CounterSet;
+import com.bigdata.rawstore.Bytes;
+import com.bigdata.service.IFederationDelegate;
+import com.bigdata.service.LoadBalancer;
+import com.bigdata.util.config.LogUtil;
+
+import net.jini.core.lookup.ServiceItem;
+import net.jini.lookup.LookupCache;
+
+import org.apache.log4j.Level;
+import org.apache.log4j.Logger;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.UUID;
+
+/**
+ * <code>Runnable</code> task class that periodically reports
+ * performance counter data to the load balancer service.
+ * (Note: see AbstractFederation.ReportTask).
+ */
+public class LoadBalancerReportingTask implements Runnable {
+
+ private IFederationDelegate embeddedIndexStore;
+ private UUID serviceUUID;
+ private CounterSet serviceRoot;
+ private LookupCache lbsCache;//for discovering lbs
+ private LoadBalancer embeddedLbs;//for testing embedded fed
+ private Logger logger;
+
+ public LoadBalancerReportingTask
+ (IFederationDelegate embeddedIndexStore,
+ UUID serviceUUID,
+ CounterSet serviceRoot,
+ LookupCache loadBalancerCache,
+ LoadBalancer embeddedLoadBalancer,
+ Logger logger)
+ {
+ this.embeddedIndexStore = embeddedIndexStore;
+ this.serviceUUID = serviceUUID;
+ this.serviceRoot = serviceRoot;
+ this.lbsCache = loadBalancerCache;
+ this.embeddedLbs = embeddedLoadBalancer;//for embedded fed testing
+ this.logger = (logger == null ?
+ LogUtil.getLog4jLogger((this.getClass()).getName()) :
+ logger);
+ }
+
+ public void run() {
+ try {
+ embeddedIndexStore.reattachDynamicCounters();
+ } catch (Throwable t) {
+ logger.error
+ ("failure on dynamic counter reattachment ["+t+"]", t);
+ }
+ try {
+ // Report the performance counters to the load balancer.
+ reportPerformanceCounters();
+ } catch (Throwable t) {
+ logger.error("failure while reporting performance "
+ +"counters to load balancer ["+t+"]", t);
+ }
+ }
+
+ private void reportPerformanceCounters() throws IOException {
+System.out.println("\n>>>>> LoadBalancerReportingTask.reportPerformanceCounters: serviceUUID = "+serviceUUID);
+ LoadBalancer lbs = null;
+ if(embeddedLbs != null) {
+ lbs = embeddedLbs;
+ } else {
+ if(lbsCache != null) {
+ ServiceItem lbsItem = lbsCache.lookup(null);
+ if(lbsItem != null) {
+ lbs = (LoadBalancer)(lbsItem.service);
+ }
+ }
+ }
+ if(lbs == null) {
+ logger.warn
+ ("cannot report counters [no load balancer service]");
+System.out.println(">>>>> LoadBalancerReportingTask.reportPerformanceCounters: loadBalancerService = NULL");
+ return;
+ }
+
+System.out.println(">>>>> LoadBalancerReportingTask.reportPerformanceCounters: loadBalancerService = "+lbs);
+
+ ByteArrayOutputStream baos = new ByteArrayOutputStream(Bytes.kilobyte32 * 2);
+ serviceRoot.asXML(baos, "UTF-8", null/* filter */);
+
+System.out.println(">>>>> LoadBalancerReportingTask.reportPerformanceCounters: CALLING loadBalancer.notify ...");
+ lbs.notify(serviceUUID, baos.toByteArray());
+System.out.println(">>>>> LoadBalancerReportingTask.reportPerformanceCounters: DONE CALLING loadBalancer.notify");
+
+ if (logger.isDebugEnabled()) {
+ logger.debug("report counters sent to load balancer");
+ }
+ }
+}
Property changes on: branches/dev-btm/bigdata/src/java/com/bigdata/counters/LoadBalancerReportingTask.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: branches/dev-btm/bigdata/src/java/com/bigdata/counters/ReadBlockCounters.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/counters/ReadBlockCounters.java (rev 0)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/counters/ReadBlockCounters.java 2010-10-04 18:57:09 UTC (rev 3722)
@@ -0,0 +1,45 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+
+package com.bigdata.counters;
+
+/**
+ * Class that can be used as a data structure that encapsulates
+ * the current state of the read block counters maintained by
+ * the shard (data) service.
+ *
+ * @todo improve reporting here and for block write as well
+ * (goes through unisolated tasks at the present).
+ */
+public class ReadBlockCounters {
+
+ /** #of block read requests. */
+ public long readBlockCount;
+ public long readBlockErrorCount;
+ public long readBlockBytes;
+ public long readBlockNanos;
+
+ public ReadBlockCounters() { }
+}
+
Property changes on: branches/dev-btm/bigdata/src/java/com/bigdata/counters/ReadBlockCounters.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: branches/dev-btm/bigdata/src/java/com/bigdata/counters/httpd/HttpReportingServer.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/counters/httpd/HttpReportingServer.java (rev 0)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/counters/httpd/HttpReportingServer.java 2010-10-04 18:57:09 UTC (rev 3722)
@@ -0,0 +1,79 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+
+package com.bigdata.counters.httpd;
+
+import com.bigdata.counters.CounterSet;
+import com.bigdata.service.IFederationDelegate;
+import com.bigdata.util.config.LogUtil;
+
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Properties;
+import java.util.LinkedHashMap;
+import java.util.Vector;
+
+/**
+ * Overrides the method <code>doGet</code> in
+ * <code>CounterSetHTTPD</code>.
+ */
+public class HttpReportingServer extends CounterSetHTTPD {
+
+ private IFederationDelegate embeddedIndexStore;
+ private Logger logger;
+
+ public HttpReportingServer
+ (final int port,
+ final CounterSet root,
+ final IFederationDelegate embeddedIndexStore,
+ Logger logger)
+ throws IOException
+ {
+ super(port, root);
+ this.embeddedIndexStore = embeddedIndexStore;
+ this.logger = (logger == null ?
+ LogUtil.getLog4jLogger((this.getClass()).getName()) :
+ logger);
+ }
+
+ @Override
+ public Response doGet(String uri,
+ String method,
+ Properties header,
+ LinkedHashMap<String,
+ Vector<String>> parms)
+ throws Exception
+ {
+ try {
+ embeddedIndexStore.reattachDynamicCounters();
+ } catch (Exception e) {
+ // Usually because the live journal has been
+ // concurrently closed during the request.
+ logger.error
+ ("failure on dynamic counter reattachment ["+e+"]", e);
+ }
+ return super.doGet(uri, method, header, parms);
+ }
+}
Property changes on: branches/dev-btm/bigdata/src/java/com/bigdata/counters/httpd/HttpReportingServer.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: branches/dev-btm/bigdata/src/java/com/bigdata/event/EventQueue.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/event/EventQueue.java (rev 0)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/event/EventQueue.java 2010-10-04 18:57:09 UTC (rev 3722)
@@ -0,0 +1,33 @@
+/*
+
+Copyright (C) SYSTAP, LLC 2006-2008. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+*/
+
+package com.bigdata.event;
+
+import com.bigdata.service.Event;
+
+public interface EventQueue {
+
+ void queueEvent(Event e);
+}
Property changes on: branches/dev-btm/bigdata/src/java/com/bigdata/event/EventQueue.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: branches/dev-btm/bigdata/src/java/com/bigdata/event/EventQueueSenderTask.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/event/EventQueueSenderTask.java (rev 0)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/event/EventQueueSenderTask.java 2010-10-04 18:57:09 UTC (rev 3722)
@@ -0,0 +1,118 @@
+/*
+
+Copyright (C) SYSTAP, LLC 2006-2008. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+*/
+
+package com.bigdata.event;
+
+import com.bigdata.service.Event;
+import com.bigdata.service.EventReceivingService;
+import com.bigdata.service.LoadBalancer;
+import com.bigdata.util.config.LogUtil;
+
+import net.jini.core.lookup.ServiceItem;
+import net.jini.lookup.LookupCache;
+
+import org.apache.log4j.Level;
+import org.apache.log4j.Logger;
+
+import java.util.LinkedList;
+import java.util.concurrent.BlockingQueue;
+
+public class EventQueueSenderTask implements EventQueue, Runnable {
+
+ private BlockingQueue<Event> eventQueue;
+ private LookupCache lbsCache;//for discovering lbs
+ private LoadBalancer embeddedLbs;//for testing embedded fed
+ private String serviceName;
+ private Logger logger;
+
+ public EventQueueSenderTask(BlockingQueue<Event> eventQueue,
+ LookupCache loadBalancerCache,
+ LoadBalancer embeddedLoadBalancer,
+ String serviceName,
+ Logger logger)
+ {
+ this.eventQueue = eventQueue;
+ this.lbsCache = loadBalancerCache;
+ this.embeddedLbs = embeddedLoadBalancer;//for embedded fed testing
+
+ this.serviceName = serviceName;//for debug output
+ this.logger = (logger == null ?
+ LogUtil.getLog4jLogger((this.getClass()).getName()) :
+ logger);
+ }
+
+ // Required by EventQueue interface
+
+ public void queueEvent(Event e) {
+ eventQueue.add(e);
+ }
+
+ // Required by Runnable interface
+
+ public void run() {
+
+//BTM - for now, maintain the same logic and functionality as that in
+//BTM the class AbstractFederation#SendEventsTask
+
+ try {
+ LoadBalancer lbs = null;
+ EventReceivingService serviceRef = null;
+ if(embeddedLbs != null) {
+ lbs = embeddedLbs;
+ } else {
+ if(lbsCache != null) {
+ ServiceItem lbsItem = lbsCache.lookup(null);
+ if(lbsItem != null) {
+ lbs = (LoadBalancer)(lbsItem.service);
+ }
+ }
+ }
+ if(lbs == null) return;
+ serviceRef = (EventReceivingService)lbs;
+
+ final long begin = System.currentTimeMillis();//for logging
+
+ final LinkedList<Event> queuedEvents = new LinkedList<Event>();
+ eventQueue.drainTo(queuedEvents);
+
+ for (Event event : queuedEvents) {
+ serviceRef.notifyEvent(event);//don't sync on remote call
+ }
+
+ if (logger.isDebugEnabled()) {
+ final int nevents = queuedEvents.size();
+ if (nevents > 0) {
+ logger.log
+ (Level.DEBUG, nevents+" event(s) sent to load "
+ +"balancer ["+(System.currentTimeMillis()-begin)
+ +" ms]");
+ }
+ }
+ } catch (Throwable t) {
+ logger.log(Level.WARN, "while sending events to load "
+ +"balancer from "+serviceName, t);
+ }
+ }
+}
Property changes on: branches/dev-btm/bigdata/src/java/com/bigdata/event/EventQueueSenderTask.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/journal/AbstractLocalTransactionManager.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/journal/AbstractLocalTransactionManager.java 2010-10-04 13:27:07 UTC (rev 3721)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/journal/AbstractLocalTransactionManager.java 2010-10-04 18:57:09 UTC (rev 3722)
@@ -9,15 +9,13 @@
import com.bigdata.counters.Instrument;
import com.bigdata.resources.StoreManager;
import com.bigdata.service.IBigdataFederation;
-import com.bigdata.service.IDataService;
/**
* Manages the client side of a transaction either for a standalone
- * {@link Journal} or for an {@link IDataService} in an
+ * {@link Journal} or for a shard service in an
* {@link IBigdataFederation}.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
*/
abstract public class AbstractLocalTransactionManager implements
ILocalTransactionManager {
Modified: branches/dev-btm/bigdata/src/java/com/bigdata/journal/ConcurrencyManager.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/journal/ConcurrencyManager.java 2010-10-04 13:27:07 UTC (rev 3721)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/journal/ConcurrencyManager.java 2010-10-04 18:57:09 UTC (rev 3722)
@@ -86,27 +86,25 @@
* </dd>
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
*/
public class ConcurrencyManager implements IConcurrencyManager {
- final protected static Logger log = Logger.getLogger(ConcurrencyManager.class);
+ final private static Logger log = Logger.getLogger(ConcurrencyManager.class);
// /**
// * True iff the {@link #log} level is INFO or less.
// */
-// final protected static boolean INFO = log.isInfoEnabled();
+// final private static boolean INFO = log.isInfoEnabled();
/**
* True iff the {@link #log} level is DEBUG or less.
*/
- final protected static boolean DEBUG = log.isDebugEnabled();
+ final private static boolean DEBUG = log.isDebugEnabled();
/**
* Options for the {@link ConcurrentManager}.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
*/
public static interface Options extends IServiceShutdown.Options {
@@ -340,7 +338,7 @@
* Once the transaction has acquired those writable indices it then runs its
* commit phrase as an unisolated operation on the {@link #writeService}.
*/
- final protected ThreadPoolExecutor txWriteService;
+ final private ThreadPoolExecutor txWriteService;
/**
* Pool of threads for handling concurrent unisolated read operations on
@@ -358,7 +356,7 @@
* historical commit records (which may span more than one logical
* journal) until the reader terminates.
*/
- final protected ThreadPoolExecutor readService;
+ final private ThreadPoolExecutor readService;
/**
* Pool of threads for handling concurrent unisolated write operations on
@@ -371,6 +369,7 @@
* Serialization of access to unisolated named indices is acomplished by
* gaining an exclusive lock on the unisolated named index.
*/
+ // protected for access by tests.
final protected WriteExecutorService writeService;
/**
@@ -400,7 +399,7 @@
}
- protected void assertOpen() {
+ private void assertOpen() {
if (!open)
throw new IllegalStateException();
@@ -466,7 +465,7 @@
* Long.MAX_VALUE.
*/
- final long shutdownTimeout = this.shutdownTimeout == 0L ? Long.MAX_VALUE
+ final long tmpShutdownTimeout = this.shutdownTimeout == 0L ? Long.MAX_VALUE
: this.shutdownTimeout;
final TimeUnit unit = TimeUnit.MILLISECONDS;
@@ -486,7 +485,7 @@
final long elapsed = System.currentTimeMillis() - begin;
- if(!txWriteService.awaitTermination(shutdownTimeout-elapsed, unit)) {
+ if(!txWriteService.awaitTermination(tmpShutdownTimeout-elapsed, unit)) {
log.warn("Transaction service termination: timeout");
@@ -505,7 +504,7 @@
final long elapsed = System.currentTimeMillis() - begin;
- if(!readService.awaitTermination(shutdownTimeout-elapsed, unit)) {
+ if(!readService.awaitTermination(tmpShutdownTimeout-elapsed, unit)) {
log.warn("Read service termination: timeout");
@@ -521,7 +520,7 @@
final long elapsed = System.currentTimeMillis() - begin;
- final long timeout = shutdownTimeout-elapsed;
+ final long timeout = tmpShutdownTimeout-elapsed;
if (log.isInfoEnabled())
log.info("Awaiting write service termination: will wait "
@@ -921,13 +920,13 @@
}
/** Counters for {@link #writeService}. */
- protected final WriteTaskCounters countersUN = new WriteTaskCounters();
+ final WriteTaskCounters countersUN = new WriteTaskCounters();
/** Counters for the {@link #txWriteService}. */
- protected final TaskCounters countersTX = new TaskCounters();
+ final TaskCounters countersTX = new TaskCounters();
/** Counters for the {@link #readService}. */
- protected final TaskCounters countersHR = new TaskCounters();
+ final TaskCounters countersHR = new TaskCounters();
/**
* Sampling instruments for the various queues giving us the moving average
@@ -942,7 +941,6 @@
* the {@link ConcurrencyManager}.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
*/
public static interface IConcurrencyManagerCounters {
@@ -977,7 +975,6 @@
* Reports the elapsed time since the service was started.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
*/
private static class ServiceElapsedTimeInstrument extends Instrument<Long> {
Added: branches/dev-btm/bigdata/src/java/com/bigdata/journal/DistributedCommitTask.java
===================================================================
--- branches/dev-btm/bigdata/src/java/com/bigdata/journal/DistributedCommitTask.java (rev 0)
+++ branches/dev-btm/bigdata/src/java/com/bigdata/journal/DistributedCommitTask.java 2010-10-04 18:57:09 UTC (rev 3722)
@@ -0,0 +1,211 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have rece...
[truncated message content] |
|
From: <res...@us...> - 2010-10-04 13:27:15
|
Revision: 3721
http://bigdata.svn.sourceforge.net/bigdata/?rev=3721&view=rev
Author: resendes
Date: 2010-10-04 13:27:07 +0000 (Mon, 04 Oct 2010)
Log Message:
-----------
Added tests and fixes for ConfigDeployUtil class.
Modified Paths:
--------------
branches/bbb_cleanup/bigdata-core/src/main/java/com/bigdata/util/config/ConfigDeployUtil.java
Added Paths:
-----------
branches/bbb_cleanup/bigdata-core/src/test/java/com/bigdata/util/config/
branches/bbb_cleanup/bigdata-core/src/test/java/com/bigdata/util/config/ConfigDeployUtilTest.java
branches/bbb_cleanup/bigdata-core/src/test/java/com/bigdata/util/config/ConfigDeployUtilTest2.java
Modified: branches/bbb_cleanup/bigdata-core/src/main/java/com/bigdata/util/config/ConfigDeployUtil.java
===================================================================
--- branches/bbb_cleanup/bigdata-core/src/main/java/com/bigdata/util/config/ConfigDeployUtil.java 2010-10-04 13:07:03 UTC (rev 3720)
+++ branches/bbb_cleanup/bigdata-core/src/main/java/com/bigdata/util/config/ConfigDeployUtil.java 2010-10-04 13:27:07 UTC (rev 3721)
@@ -34,6 +34,8 @@
import java.io.FileInputStream;
import java.io.IOException;
import java.net.SocketException;
+import java.text.NumberFormat;
+import java.text.ParseException;
import java.util.Arrays;
import java.util.List;
import java.util.Properties;
@@ -57,7 +59,9 @@
private static final String TYPE = ".type";
private static final String FALLBACK_FEDNAME_FORMAT = "bigdata.test.group-%s";
- private static final String TEMPLATE_TOKEN_PATTERN = "@.*@";
+
+ //use current locale
+ private static final NumberFormat numberFormat = NumberFormat.getInstance();
public static String getString(String parameter)
throws ConfigurationException
@@ -88,23 +92,24 @@
return value;
}
- public static boolean getBoolean(String parameter)
+ public static boolean getBoolean(String parameter) throws ConfigurationException
+ {
+ boolean value;
+ value = validateBoolean(parameter, get(parameter));
+ return value;
+ }
+
+ private static boolean validateBoolean(String parameter, String value)
throws ConfigurationException {
boolean boolValue = false;
- String value;
- try {
- value = get(parameter);
- } catch (Exception ex) {
- throw new ConfigurationException("parameter value ["+parameter+"] "
- +"neither 'true' nor 'false'");
- }
- if( value != null && value.equalsIgnoreCase("true")
- || value.equalsIgnoreCase("false") )
+
+ if( value != null && (value.equalsIgnoreCase("true")
+ || value.equalsIgnoreCase("false")) )
{
boolValue = Boolean.parseBoolean(value);
} else {
throw new ConfigurationException("parameter value ["+parameter+"] "
- +"neither 'true' nor 'false'");
+ +"is neither 'true' nor 'false'");
}
return boolValue;
}
@@ -113,11 +118,7 @@
throws ConfigurationException
{
String value;
- if(deploymentProps == null) {
- deploymentProps = new Properties();
- loadDeployProps(deploymentProps);
- }
- value = deploymentProps.getProperty(parameter + DESCRIPTION);
+ value = getDeploymentProperties().getProperty(parameter + DESCRIPTION);
return value;
}
@@ -125,11 +126,7 @@
throws ConfigurationException
{
String value;
- if (deploymentProps == null) {
- deploymentProps = new Properties();
- loadDeployProps(deploymentProps);
- }
- value = deploymentProps.getProperty(parameter + TYPE);
+ value = getDeploymentProperties().getProperty(parameter + TYPE);
return value;
}
@@ -137,11 +134,7 @@
throws ConfigurationException
{
String value;
- if (deploymentProps == null) {
- deploymentProps = new Properties();
- loadDeployProps(deploymentProps);
- }
- value = deploymentProps.getProperty(parameter + DEFAULT);
+ value = getDeploymentProperties().getProperty(parameter + DEFAULT);
if (value == null) {
throw new ConfigurationException
("deployment parameter not found ["+parameter+"]");
@@ -150,7 +143,7 @@
}
/**
- * Returns a <code>String</code> array whose elments represent the
+ * Returns a <code>String</code> array whose elements represent the
* lookup service groups to discover. If the system property named
* "federation.name" is set then that value be used; otherwise,
* the deployment properties files will be consulted.
@@ -164,7 +157,7 @@
}
/**
- * Retrieve the federation name (also used as Jini group name) via this pecking pecking order:
+ * Retrieve the federation name (also used as Jini group name) via this pecking order:
* <ol>
* <li>From the Java system property: <code>federation.name</code></li>
* <li>From the deployment properties file. Note that a value from the deployment
@@ -254,11 +247,7 @@
private static String get(String parameter) throws ConfigurationException {
String value;
- if (deploymentProps == null) {
- deploymentProps = new Properties();
- loadDeployProps(deploymentProps);
- }
- value = deploymentProps.getProperty(parameter);
+ value = getDeploymentProperties().getProperty(parameter);
if (value == null) value = getDefault(parameter);
return value;
}
@@ -267,7 +256,7 @@
throws ConfigurationException
{
String validValuesStr =
- (String) deploymentProps.get(parameter + STRINGVALS);
+ (String) getDeploymentProperties().get(parameter + STRINGVALS);
if (validValuesStr != null) {
String[] validValues = validValuesStr.split(",");
@@ -284,7 +273,7 @@
throws ConfigurationException
{
String validValuesStr =
- (String)(deploymentProps.get(parameter + STRINGVALS));
+ (String)(getDeploymentProperties().get(parameter + STRINGVALS));
String[] values = value.split(",");
if (validValuesStr != null) {
@@ -304,48 +293,77 @@
private static int validateInt(String parameter, String strvalue)
throws ConfigurationException
{
- String maxString = (String)(deploymentProps.get(parameter + MAX));
- String minString = (String)(deploymentProps.get(parameter + MIN));
+ String maxString = (String)(getDeploymentProperties().get(parameter + MAX));
+ String minString = (String)(getDeploymentProperties().get(parameter + MIN));
int value = str2int(strvalue);
if (maxString != null) {
- int max = Integer.parseInt(maxString);
- if (value > max) {
- throw new ConfigurationException("parameter ["+parameter+"] "
- +"exceeds maximum ["+max+"]");
+ try {
+ int max = numberFormat.parse(maxString).intValue();
+ if (value > max) {
+ throw new ConfigurationException("parameter ["+parameter+"] "
+ +"exceeds maximum ["+max+"]");
+ }
+ } catch (ParseException e) {
+ throw new NumberFormatException(
+ "Invalid maximum integer for parameter: " + parameter);
+
}
}
+
+ if (minString != null) {
+ try {
+ int min = numberFormat.parse(minString).intValue();
+ if (value < min) {
+ throw new ConfigurationException("parameter ["+parameter+"] "
+ + "is less than manimum ["+min+"]");
+ }
+ } catch (ParseException e) {
+ throw new NumberFormatException(
+ "Invalid minimum integer for parameter: " + parameter);
+ }
+ }
+
return value;
}
private static long validateLong(String parameter, String strvalue)
throws ConfigurationException
{
- String maxString = (String)(deploymentProps.get(parameter + MAX));
- String minString = (String)(deploymentProps.get(parameter + MIN));
+ String maxString = (String)(getDeploymentProperties().get(parameter + MAX));
+ String minString = (String)(getDeploymentProperties().get(parameter + MIN));
long value = str2long(strvalue);
if (maxString != null) {
- long max = Long.parseLong(maxString);
- if (value > max) {
- throw new ConfigurationException("parameter ["+parameter+"] "
- +"exceeds maximum ["+max+"]");
+ try {
+ long max = numberFormat.parse(maxString).longValue();
+ if (value > max) {
+ throw new ConfigurationException("parameter ["+parameter+"] "
+ +"exceeds maximum ["+max+"]");
+ }
+ } catch (ParseException e) {
+ throw new NumberFormatException(
+ "Invalid maximum long for parameter: " + parameter);
+
}
}
if (minString != null) {
- long min = Long.parseLong(minString);
- if (value < min) {
- throw new ConfigurationException("parameter ["+parameter+"] "
- +"is less than manimum "
- +"["+min+"]");
- }
+ try {
+ long min = numberFormat.parse(minString).longValue();
+ if (value < min) {
+ throw new ConfigurationException("parameter ["+parameter+"] "
+ + "is less than manimum ["+min+"]");
+ }
+ } catch (ParseException e) {
+ throw new NumberFormatException(
+ "Invalid minimum long for parameter: " + parameter);
+ }
}
return value;
}
-
private static File getPropertiesPath() {
File rootDir = new File("/opt/bigdata"); //real installation
String appHome = System.getProperty("appHome");//pstart
@@ -372,30 +390,23 @@
}
private static void loadDefaultProps(Properties deployProps) {
- FileInputStream fis = null;
- try {
- File flnm = new File(getPropertiesPath(), "default-deploy.properties");
- fis = new FileInputStream(flnm);
- deployProps.load(fis);
- } catch (Exception ex) {
- ex.printStackTrace();
- } finally {
- if (fis != null) {
- try {
- fis.close();
- } catch (IOException ioex) { /* swallow */ }
- }
- }
+ loadPropsInternal("default-deploy.properties", deployProps, true);
}
private static void loadOverrideProps(Properties deployProps) {
+ loadPropsInternal("deploy.properties", deployProps, false);
+ }
+
+ private static void loadPropsInternal(String propFileName,
+ Properties deployProps, boolean prtinTrace)
+ {
FileInputStream fis = null;
try {
- File flnm = new File(getPropertiesPath(), "default-deploy.properties");
+ File flnm = new File(getPropertiesPath(), propFileName);
fis = new FileInputStream(flnm);
deployProps.load(fis);
} catch (Exception ex) {
- // using all defaults
+ if (prtinTrace) ex.printStackTrace();
} finally {
if (fis != null) {
try {
@@ -407,214 +418,52 @@
private static int str2int(String argx) {
- long l;
-
- if( argx.trim().equals(Integer.MAX_VALUE) ) return Integer.MAX_VALUE;
- if( argx.trim().equals(Integer.MIN_VALUE) ) return Integer.MIN_VALUE;
-
- l = str2long(argx);
- if (l < Integer.MAX_VALUE && l > Integer.MIN_VALUE) {
- return (int) l;
- } else {
- throw new NumberFormatException("Invalid number:"+argx
- +" --number out of range");
- }
+ Number n = null;
+ try {
+ //TODO - truncation can occur -- check for overflow
+ n = numberFormat.parse(argx);
+ } catch (ParseException e) {
+ throw new NumberFormatException("Invalid integer: " + argx);
+ }
+ return n.intValue();
}
private static long str2long(String argx) {
-
- int minDigitNumBetwnComma = 3;
-
- String arg = argx.trim();
- arg = arg.replaceAll("\"", ""); // strip all quotes
- int sz = arg.length();
-
- if( arg.equals("Long.MAX_VALUE") ) return Long.MAX_VALUE;
-
- if( arg.equals("Long.MIN_VALUE") ) return Long.MIN_VALUE;
-
- int asterPos = -1;
- String arg1 = null;
- String arg2 = null;
- if( (asterPos = arg.indexOf("*")) != -1) {
- int dotPos = -1;
- arg1 = arg.substring(0, asterPos).trim();
- int denom1 = 1;
- if( (dotPos = arg1.indexOf(".")) != -1) {
- StringBuffer tmpBuf = new StringBuffer("1");
- int hitNumber = 0;
- for (int i = dotPos + 1; i < (arg1.length() - dotPos); i++) {
- if( Character.isDigit(arg1.charAt(i)) ) {
- tmpBuf.append("0");
- } else {
- break;
- }
- }
- denom1 = Integer.valueOf(tmpBuf.toString());
- arg1 = arg1.substring(0, dotPos) + arg1.substring(dotPos + 1);
- }
-
- arg2 = arg.substring(asterPos + 1).trim();
- int denom2 = 1;
- if( (dotPos = arg2.indexOf(".")) != -1) {
- StringBuffer tmpBuf = new StringBuffer("1");
- for(int i = dotPos + 1; i <= (arg2.length() - dotPos); i++) {
- tmpBuf.append("0");
- }
-
- denom2 = Integer.valueOf(tmpBuf.toString());
- arg2 = arg2.substring(0, dotPos) + arg2.substring(dotPos + 1);
- }
-
- long numerator = str2long(arg1) * str2long(arg2);
- long denom = (denom1 * denom2);
-
- if (numerator % denom != 0) {
- throw new NumberFormatException(" Bad value passed in:" +
- ((double) (numerator) /
- denom) +
- ", expecting a long");
- }
- return (numerator / denom);
- }
-
- char unit = arg.charAt(sz - 1);
-
- String valScalar = arg.substring(0, (sz - 1)).trim();
-
- long factor = 0l;
-
- switch (Character.toUpperCase(unit)) {
-
- case 'G':
- factor = 1000000000l;
- break;
- case 'M':
- factor = 1000000l;
- break;
- case 'K':
- factor = 1000l;
- break;
- case 'B':
- char unitPre = arg.charAt(sz - 2);
- if (Character.isDigit(unitPre)) {
- factor = -1l;
- } else {
- factor =
- (Character.toUpperCase(unitPre) ==
- 'G' ? 1000000000l : (Character.toUpperCase(unitPre) ==
- 'M' ? 1000000l : (Character.
- toUpperCase
- (unitPre) ==
- 'K' ? 1000l :
- -1l)));
- valScalar = arg.substring(0, (sz - 2)).trim();
- }
- break;
-
- default:
- if (Character.isDigit(unit)) {
- factor = 1l;
- valScalar = arg;
- }
- }
- if (factor == -1l) {
- throw new NumberFormatException("Invalid number:" + arg);
- }
-
- int comaPos = -1;
- if( (comaPos = valScalar.indexOf(',')) != -1) {
- if(valScalar.indexOf('.') != -1) {
- throw new NumberFormatException("Invalid number:"+arg
- +" both \",\" and decimal "
- +"point are not supported");
- }
- if( comaPos != 0 && comaPos != (valScalar.length() - 1) ) {
- String[]spltByComa = valScalar.split(",");
- valScalar = "";
- for (int i = spltByComa.length - 1; i >= 0; i--) {
- if(i > 0 && spltByComa[i].length() < minDigitNumBetwnComma)
- {
- throw new NumberFormatException("Invalid number:"+arg
- +" unexpected comma "
- +"format");
- }
- valScalar = spltByComa[i] + valScalar;
- }
- } else {
- throw new NumberFormatException("Invalid number:\"" +arg
- +"\" -unexpected comma in "
- +"position: "+comaPos);
- }
- }
-
- int decimalPos = -1;
- String valMultiplByFactor = null;
- int numZero = 0;
- try {
- if( (decimalPos = valScalar.indexOf('.')) != -1) {
- if (decimalPos != valScalar.lastIndexOf('.')) {
- throw new NumberFormatException("Invalid number:"
- +valScalar
- +" --invalid decimal "
- +"number, bad value");
- }
-
- String facStr = String.valueOf(factor);
- int numZeroFactor = facStr.length() - 1;
- int numDigitsAfterDecimal =
- valScalar.length() - decimalPos - 1;
- int countZero = 0;
- for(int i = valScalar.length() - 1; i > decimalPos; i--) {
-
- if (valScalar.charAt(i) != '0')
- break;
- --numDigitsAfterDecimal;
- countZero++;
- }
- numZero = numZeroFactor - numDigitsAfterDecimal;
- if (numZero == numDigitsAfterDecimal) {
- numZero = 0;
- }
- if(numZero < 0) {
- throw new NumberFormatException("Invalid number:"
- +valScalar
- +" --invalid decimal "
- +"number, numzero="
- + numZero);
- }
-
- if(numZero >= 0) {
- StringBuffer tmpStrNum =
- new StringBuffer(20).
- append(valScalar.substring(0, decimalPos)).
- append(valScalar.substring(decimalPos + 1,
- decimalPos + 1 +
- numDigitsAfterDecimal));
- for(int i=0; i<numZero; i++) {
- tmpStrNum.append('0');
- }
- valMultiplByFactor = tmpStrNum.toString();
- }
-
- }
- } catch(NumberFormatException nfe) {
- throw new NumberFormatException("Invalid number:" +valScalar
- +" --invalid decimal number, "
- +"numZero="+numZero);
- }
-
- long ret = -1l;
-
- Long ll = ((decimalPos != -1) ? Long.valueOf(valMultiplByFactor)
- : (Long.valueOf(valScalar) * factor));
- if( (ret = Long.valueOf(ll)) >= Long.MAX_VALUE
- || ret <= Long.MIN_VALUE)
- {
- throw new NumberFormatException("Invalid number:"+arg
- +" --absolute value of number "
- +"too big");
- }
- return ret;
+ Number n = null;
+ try {
+ //TODO - truncation can occur -- check for overflow
+ n = numberFormat.parse(argx);
+ } catch (ParseException e) {
+ throw new NumberFormatException("Invalid long: " + argx);
+ }
+ return n.longValue();
}
+
+ /**
+ * Returns reference to <code>deploymenyProps</code> field. If null, then the
+ * field is populated by looking for the default and override properties files
+ * (defined in <code>loadDefaultProps</code> and <code>loadOverrideProps</code>).
+ * This method is synchronized in order to ensure that the returned reference is
+ * a singleton instance.
+ * Note: This method should be private, but is needed by the unit test in order
+ * to access and modify the <code>Properties</code> method.
+ * @return Properties instance containing the default and user-defined overrides
+ * for configuration properties.
+ */
+ synchronized static Properties getDeploymentProperties() {
+ if(deploymentProps == null) {
+ deploymentProps = new Properties();
+ loadDeployProps(deploymentProps);
+ }
+ return deploymentProps;
+ }
+
+ /**
+ * Convenience method intended for use by unit tests only.
+ * @param properties Sets <code>Properties</code> object
+ */
+ synchronized static void setDeploymentProperties(Properties properties) {
+ deploymentProps = properties;
+ }
+
}
Added: branches/bbb_cleanup/bigdata-core/src/test/java/com/bigdata/util/config/ConfigDeployUtilTest.java
===================================================================
--- branches/bbb_cleanup/bigdata-core/src/test/java/com/bigdata/util/config/ConfigDeployUtilTest.java (rev 0)
+++ branches/bbb_cleanup/bigdata-core/src/test/java/com/bigdata/util/config/ConfigDeployUtilTest.java 2010-10-04 13:27:07 UTC (rev 3721)
@@ -0,0 +1,1128 @@
+/**
+ *
+ */
+package com.bigdata.util.config;
+
+import static org.junit.Assert.*;
+
+import java.io.IOException;
+import java.net.SocketException;
+import java.text.ParseException;
+import java.util.Arrays;
+import java.util.Properties;
+import java.util.UUID;
+
+import net.jini.config.ConfigurationException;
+import net.jini.core.discovery.LookupLocator;
+
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import com.bigdata.attr.ServiceInfo;
+import com.ibm.icu.text.NumberFormat;
+
+/**
+ * Note: These tests use a package protected method
+ * {@link com.bigdata.util.config.ConfigDeployUtil#getDeploymentProperties()}
+ * to set its underlying <code>Properties</code> object.
+ */
+public class ConfigDeployUtilTest {
+ /**
+ * Holds a reference to the default <code>Properties</code> object created by
+ * <code>ConfigDeployUtil</code>. The reference is reset on the
+ * <code>ConfigDeployUtil</code> object after these unit tests finish.
+ */
+ private static Properties _old_properties = null;
+
+ /**
+ * @throws java.lang.Exception
+ */
+ @BeforeClass
+ public static void setUpBeforeClass() throws Exception {
+ _old_properties = ConfigDeployUtil.getDeploymentProperties();
+ ConfigDeployUtil.setDeploymentProperties(new Properties());
+ }
+
+ /**
+ * @throws java.lang.Exception
+ */
+ @AfterClass
+ public static void tearDownAfterClass() throws Exception {
+ ConfigDeployUtil.setDeploymentProperties(_old_properties);
+ }
+
+ /**
+ * @throws java.lang.Exception
+ */
+ @Before
+ public void setUp() throws Exception {
+ ConfigDeployUtil.getDeploymentProperties().clear();
+ }
+
+ /**
+ * @throws java.lang.Exception
+ */
+ @After
+ public void tearDown() throws Exception {
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getString(java.lang.String)}.
+ */
+ @Test
+ public void testGetString_bogus() {
+ try {
+ String k = "testGetString_bogus";
+ String v = ConfigDeployUtil.getString(k);
+ fail("Successfully called getString with bogus parameter string: ["
+ + k + ":" + v + "]");
+ } catch (ConfigurationException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getString(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetString_empty_name() throws ConfigurationException {
+ String key = "testGetString_empty_name";
+ String value = "";
+ ConfigDeployUtil.getDeploymentProperties().setProperty(
+ key, value);
+ String actual = ConfigDeployUtil.getString(key);
+ String expected = value;
+ assertEquals(actual, expected);
+ }
+
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getString(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetString_valid_name() throws ConfigurationException {
+ String key = "testGetString_valid_name";
+ String value = key + "_value";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key, value);
+ p.setProperty(key+".stringvals", value + ",other");
+ String actual = ConfigDeployUtil.getString(key);
+ String expected = value;
+ assertEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getString(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetString_invalid_name() throws ConfigurationException {
+ String key = "testGetString_invalid_name";
+ String value = key + "_value";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key, value);
+ p.setProperty(key+".stringvals", "other1,other2");
+ try {
+ ConfigDeployUtil.getString(key);
+ fail("Successfully called getString with bogus parameter string: ["
+ + key + ":" + value + "]");
+ } catch (ConfigurationException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetStringArray_array_bogus() throws ConfigurationException {
+ String key = "testGetStringArray_array_bogus";
+ String value = null;
+ try {
+ ConfigDeployUtil.getStringArray(key);
+ fail("Successfully called getStringArray with bogus parameter string: ["
+ + key + ":" + value + "]");
+ } catch (ConfigurationException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetStringArray_array_default_singleton() throws ConfigurationException {
+ String key = "testGetStringArray_array_default_singleton";
+ String value = "infrastructure";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", value);
+ p.setProperty(key+".type", "string");
+ p.setProperty(key+".stringvals", value + ",non-infrastructure,other");
+ String[] actual = ConfigDeployUtil.getStringArray(key);
+ String[] expected = {value};
+ assertArrayEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetStringArray_valid_singleton() throws ConfigurationException {
+ String key = "testGetStringArray_valid_singleton";
+ String value = "infrastructure";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", value + "_default");
+ p.setProperty(key, value);
+ p.setProperty(key+".type", "string");
+ p.setProperty(key+".stringvals", value + ",non-infrastructure,other");
+ String[] actual = ConfigDeployUtil.getStringArray(key);
+ String[] expected = {value};
+ assertArrayEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetStringArray_invalid_singleton() throws ConfigurationException {
+ String key = "testGetStringArray_invalid_singleton";
+ String value = "infrastructure";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", value + "_default");
+ p.setProperty(key, value);
+ p.setProperty(key+".type", "string");
+ p.setProperty(key+".stringvals", "non-infrastructure,other");
+ try {
+ ConfigDeployUtil.getStringArray(key);
+ fail("Successfully called getStringArray with invalid parameter string: ["
+ + key + ":" + value + "]");
+ } catch (ConfigurationException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetStringArray_array_default_multiple() throws ConfigurationException {
+ String key = "testGetStringArray_array_default_multiple";
+ String value = "infrastructure1,infrastructure2";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", value);
+ p.setProperty(key+".type", "string");
+ p.setProperty(key+".stringvals", value + ",non-infrastructure,other");
+ String[] actual = ConfigDeployUtil.getStringArray(key);
+ String[] expected = value.split(",");
+ assertArrayEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetStringArray_array_valid_multiple() throws ConfigurationException {
+ String key = "testGetStringArray_array_valid_multiple";
+ String value = "infrastructure1,infrastructure2";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", value);
+ p.setProperty(key+".type", "string");
+ p.setProperty(key+".stringvals", value + ",non-infrastructure,other");
+ String[] actual = ConfigDeployUtil.getStringArray(key);
+ String[] expected = value.split(",");
+ assertArrayEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetStringArray_array_invalid_multiple1() throws ConfigurationException {
+ String key = "testGetStringArray_array_invalid_multiple1";
+ String value = "infrastructure1,infrastructure2";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", value);
+ p.setProperty(key+".type", "string");
+ p.setProperty(key+".stringvals", "non-infrastructure,other");
+ try {
+ ConfigDeployUtil.getStringArray(key);
+ fail("Successfully called getStringArray with invalid parameter string: ["
+ + key + ":" + value + "]");
+ } catch (ConfigurationException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetStringArray_array_invalid_multiple2() throws ConfigurationException {
+ String key = "testGetStringArray_array_invalid_multiple2";
+ String value = "infrastructure1,infrastructure2";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", value);
+ p.setProperty(key+".type", "string");
+ p.setProperty(key+".stringvals", "infrastructure1,non-infrastructure,other");
+ try {
+ ConfigDeployUtil.getStringArray(key);
+ fail("Successfully called getStringArray with invalid parameter string: ["
+ + key + ":" + value + "]");
+ } catch (ConfigurationException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getInt(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetInt_bogus() throws ConfigurationException {
+ String key = "testGetInt_bogus";
+ String value = null;
+ try {
+ ConfigDeployUtil.getInt(key);
+ fail("Successfully called getInt with invalid parameter string: ["
+ + key + ":" + value + "]");
+ } catch (ConfigurationException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetInt_valid1() throws ConfigurationException {
+ String key = "testGetInt_valid1";
+ String sValue = "2";
+ int expected = Integer.parseInt(sValue);
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", "-1");
+ p.setProperty(key+".max", "1000");
+ p.setProperty(key+".type", "int");
+ p.setProperty(key, sValue);
+ int actual = ConfigDeployUtil.getInt(key);
+ assertEquals(actual, expected);
+ }
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetInt_valid2() throws ConfigurationException {
+ String key = "testGetInt_valid2";
+ String sValue = "-1";
+ int expected = Integer.parseInt(sValue);
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", "-1");
+ p.setProperty(key+".max", "1000");
+ p.setProperty(key+".type", "int");
+ p.setProperty(key, sValue);
+ int actual = ConfigDeployUtil.getInt(key);
+ assertEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetInt_valid3() throws ConfigurationException {
+ String key = "testGetInt_valid3";
+ String sValue = "1000";
+ int expected = Integer.parseInt(sValue);
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", "-1");
+ p.setProperty(key+".max", "1000");
+ p.setProperty(key+".type", "int");
+ p.setProperty(key, sValue);
+ int actual = ConfigDeployUtil.getInt(key);
+ assertEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetInt_valid4() throws ConfigurationException {
+ String key = "testGetInt_valid4";
+ String sValue = Integer.toString(Integer.MAX_VALUE);
+ int expected = Integer.MAX_VALUE;
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", "-1");
+ p.setProperty(key+".max", sValue);
+ p.setProperty(key+".type", "int");
+ p.setProperty(key, sValue);
+ int actual = ConfigDeployUtil.getInt(key);
+ assertEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetInt_valid5() throws ConfigurationException {
+ String key = "testGetInt_valid5";
+ String sValue = Integer.toString(Integer.MIN_VALUE);
+ int expected = Integer.MIN_VALUE;
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", sValue);
+ p.setProperty(key+".max", "1000");
+ p.setProperty(key+".type", "int");
+ p.setProperty(key, sValue);
+ int actual = ConfigDeployUtil.getInt(key);
+ assertEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ * @throws ParseException
+ */
+ @Test
+ public void testGetInt_valid6() throws ConfigurationException, ParseException {
+ String key = "testGetInt_valid5";
+ String sValue = "1,000,000";//try string with separators
+ int expected = NumberFormat.getInstance().parse(sValue).intValue();
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "0");
+ p.setProperty(key+".min", "-1,000");
+ p.setProperty(key+".max", "10,000,000");
+ p.setProperty(key+".type", "int");
+ p.setProperty(key, sValue);
+ int actual = ConfigDeployUtil.getInt(key);
+ assertEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetInt_invaild_min() throws ConfigurationException {
+ String key = "testGetInt_invaild_min";
+ String sValue = "-2";
+ int expected = Integer.parseInt(sValue);
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", "-1");
+ p.setProperty(key+".max", "1000");
+ p.setProperty(key+".type", "int");
+ p.setProperty(key, sValue);
+ try {
+ ConfigDeployUtil.getInt(key);
+ fail("Successfully called getInt with invalid parameter string: ["
+ + key + ":" + sValue + "]");
+ } catch (ConfigurationException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetInt_invaild_max() throws ConfigurationException {
+ String key = "testGetInt_invaild_max";
+ String sValue = "2000";
+ int expected = Integer.parseInt(sValue);
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", "-1");
+ p.setProperty(key+".max", "1000");
+ p.setProperty(key+".type", "int");
+ p.setProperty(key, sValue);
+ try {
+ ConfigDeployUtil.getInt(key);
+ fail("Successfully called getInt with invalid parameter string: ["
+ + key + ":" + sValue + "]");
+ } catch (ConfigurationException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetInt_invaild_not_a_number() throws ConfigurationException {
+ String key = "testGetInt_invaild_not_a_number";
+ String sValue = "abcxyz";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", "-1");
+ p.setProperty(key+".max", "10000");
+ p.setProperty(key+".type", "int");
+ p.setProperty(key, sValue);
+ try {
+ ConfigDeployUtil.getInt(key);
+ fail("Successfully called getLong with getLong parameter string: ["
+ + key + ":" + sValue + "]");
+ } catch (NumberFormatException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetInt_invaild_max_not_a_number() throws ConfigurationException {
+ String key = "testGetInt_invaild_max_not_a_number";
+ String sValue = "10000";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", "-1");
+ p.setProperty(key+".max", "bogus_max_value");
+ p.setProperty(key+".type", "int");
+ p.setProperty(key, sValue);
+ try {
+ ConfigDeployUtil.getInt(key);
+ fail("Successfully called getLong with getLong parameter string: ["
+ + key + ":" + sValue + "]");
+ } catch (NumberFormatException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetInt_invaild_min_not_a_number() throws ConfigurationException {
+ String key = "testGetInt_invaild_min_not_a_number";
+ String sValue = "-1";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", "bogus_min_value");
+ p.setProperty(key+".max", "10000");
+ p.setProperty(key+".type", "int");
+ p.setProperty(key, sValue);
+ try {
+ ConfigDeployUtil.getInt(key);
+ fail("Successfully called getLong with getLong parameter string: ["
+ + key + ":" + sValue + "]");
+ } catch (NumberFormatException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getLong(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetLong_bogus() throws ConfigurationException {
+ String key = "testGetLong_bogus";
+ String sValue = "2000";
+ try {
+ ConfigDeployUtil.getLong(key);
+ fail("Successfully called getLong with bogus parameter string: ["
+ + key + ":" + sValue + "]");
+ } catch (ConfigurationException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetLong_valid1() throws ConfigurationException {
+ String key = "testGetLong_valid1";
+ String sValue = "2222";
+ long expected = Long.parseLong(sValue);
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", "-1");
+ p.setProperty(key+".max", "10000");
+ p.setProperty(key+".type", "long");
+ p.setProperty(key, sValue);
+ long actual = ConfigDeployUtil.getLong(key);
+ assertEquals(actual, expected);
+ }
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetLong_valid2() throws ConfigurationException {
+ String key = "testGetLong_valid2";
+ String sValue = "-1";
+ long expected = Long.parseLong(sValue);
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", "-1");
+ p.setProperty(key+".max", "10000");
+ p.setProperty(key+".type", "long");
+ p.setProperty(key, sValue);
+ long actual = ConfigDeployUtil.getLong(key);
+ assertEquals(actual, expected);
+ }
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetLong_valid3() throws ConfigurationException {
+ String key = "testGetLong_valid3";
+ String sValue = "10000";
+ long expected = Long.parseLong(sValue);
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", "-1");
+ p.setProperty(key+".max", "10000");
+ p.setProperty(key+".type", "long");
+ p.setProperty(key, sValue);
+ long actual = ConfigDeployUtil.getLong(key);
+ assertEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetLong_valid4() throws ConfigurationException {
+ String key = "testGetLong_valid4";
+ String sValue = Long.toString(Long.MAX_VALUE);
+ long expected = Long.parseLong(sValue);
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", "-1");
+ p.setProperty(key+".max", sValue);
+ p.setProperty(key+".type", "long");
+ p.setProperty(key, sValue);
+ long actual = ConfigDeployUtil.getLong(key);
+ assertEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetLong_valid5() throws ConfigurationException {
+ String key = "testGetLong_valid5";
+ String sValue = Long.toString(Long.MIN_VALUE);
+ long expected = Long.parseLong(sValue);
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", sValue);
+ p.setProperty(key+".max", "10000");
+ p.setProperty(key+".type", "long");
+ p.setProperty(key, sValue);
+ long actual = ConfigDeployUtil.getLong(key);
+ assertEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ * @throws ParseException
+ */
+ @Test
+ public void testGetLong_valid6() throws ConfigurationException, ParseException {
+ String key = "testGetInt_valid6";
+ String sValue = "1,000,000";//try string with separators
+ long expected = NumberFormat.getInstance().parse(sValue).longValue();
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "0");
+ p.setProperty(key+".min", "-1,000");
+ p.setProperty(key+".max", "10,000,000");
+ p.setProperty(key+".type", "long");
+ p.setProperty(key, sValue);
+ long actual = ConfigDeployUtil.getLong(key);
+ assertEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetLong_invaild_min() throws ConfigurationException {
+ String key = "testGetLong_invaild_min";
+ String sValue = "-20000";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", "-1");
+ p.setProperty(key+".max", "10000");
+ p.setProperty(key+".type", "int");
+ p.setProperty(key, sValue);
+ try {
+ ConfigDeployUtil.getLong(key);
+ fail("Successfully called getInt with getLong parameter string: ["
+ + key + ":" + sValue + "]");
+ } catch (ConfigurationException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetLong_invaild_max() throws ConfigurationException {
+ String key = "testGetLong_invaild_max";
+ String sValue = "20000";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", "-1");
+ p.setProperty(key+".max", "10000");
+ p.setProperty(key+".type", "int");
+ p.setProperty(key, sValue);
+ try {
+ ConfigDeployUtil.getLong(key);
+ fail("Successfully called getLong with getLong parameter string: ["
+ + key + ":" + sValue + "]");
+ } catch (ConfigurationException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetLong_invaild_not_a_number() throws ConfigurationException {
+ String key = "testGetLong_invaild_max";
+ String sValue = "abcxyz";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", "-1");
+ p.setProperty(key+".max", "10000");
+ p.setProperty(key+".type", "int");
+ p.setProperty(key, sValue);
+ try {
+ ConfigDeployUtil.getLong(key);
+ fail("Successfully called getLong with getLong parameter string: ["
+ + key + ":" + sValue + "]");
+ } catch (NumberFormatException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetLong_invaild_max_not_a_number() throws ConfigurationException {
+ String key = "testGetLong_invaild_max";
+ String sValue = "1000";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", "-1");
+ p.setProperty(key+".max", "bogus_max_value");
+ p.setProperty(key+".type", "int");
+ p.setProperty(key, sValue);
+ try {
+ ConfigDeployUtil.getLong(key);
+ fail("Successfully called getLong with getLong parameter string: ["
+ + key + ":" + sValue + "]");
+ } catch (NumberFormatException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getStringArray(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetLong_invaild_min_not_a_number() throws ConfigurationException {
+ String key = "testGetLong_invaild_max";
+ String sValue = "1000";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "-1");
+ p.setProperty(key+".min", "bogus_min_value");
+ p.setProperty(key+".max", "10000");
+ p.setProperty(key+".type", "int");
+ p.setProperty(key, sValue);
+ try {
+ ConfigDeployUtil.getLong(key);
+ fail("Successfully called getLong with getLong parameter string: ["
+ + key + ":" + sValue + "]");
+ } catch (NumberFormatException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getBoolean(java.lang.String)}.
+ */
+ @Test
+ public void testGetBoolean_bogus() {
+ try {
+ String k = "bogus.field.name";
+ boolean v = ConfigDeployUtil.getBoolean(k);
+ fail("Successfully called getBoolean with bogus parameter string: ["
+ + k + ":" + v + "]");
+ } catch (ConfigurationException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getBoolean(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetBoolean_valid_true1() throws ConfigurationException {
+ String key = "testGetBoolean_valid_true1";
+ String sValue = "true";
+ boolean expected = Boolean.parseBoolean(sValue);
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "false");
+ p.setProperty(key+".type", "boolean");
+ p.setProperty(key, sValue);
+ boolean actual = ConfigDeployUtil.getBoolean(key);
+ assertEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getBoolean(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetBoolean_valid_true2() throws ConfigurationException {
+ String key = "testGetBoolean_valid_true2";
+ String sValue = "TrUe";
+ boolean expected = Boolean.parseBoolean(sValue);
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "false");
+ p.setProperty(key+".type", "boolean");
+ p.setProperty(key, sValue);
+ boolean actual = ConfigDeployUtil.getBoolean(key);
+ assertEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getBoolean(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetBoolean_valid_false1() throws ConfigurationException {
+ String key = "testGetBoolean_valid_false1";
+ String sValue = "false";
+ boolean expected = Boolean.parseBoolean(sValue);
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "true");
+ p.setProperty(key+".type", "boolean");
+ p.setProperty(key, sValue);
+ boolean actual = ConfigDeployUtil.getBoolean(key);
+ assertEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getBoolean(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetBoolean_valid_false2() throws ConfigurationException {
+ String key = "testGetBoolean_valid_false2";
+ String sValue = "FaLsE";
+ boolean expected = Boolean.parseBoolean(sValue);
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "true");
+ p.setProperty(key+".type", "boolean");
+ p.setProperty(key, sValue);
+ boolean actual = ConfigDeployUtil.getBoolean(key);
+ assertEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getBoolean(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetBoolean_invalid() throws ConfigurationException {
+ String key = "testGetBoolean_invalid";
+ String sValue = "bogus";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", "Description");
+ p.setProperty(key+".default", "true");
+ p.setProperty(key+".type", "boolean");
+ p.setProperty(key, sValue);
+ try {
+ ConfigDeployUtil.getBoolean(key);
+ fail("Successfully called getBoolean with invalid parameter string: ["
+ + key + ":" + sValue + "]");
+ } catch (ConfigurationException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getDescription(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetDescription_bogus() throws ConfigurationException {
+ String key = "testGetDescription_bogus";
+ String expected = null;
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key, "");
+ String actual = ConfigDeployUtil.getDescription(key);
+ assertEquals(actual, expected);
+ }
+
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getDescription(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetDescription_valid() throws ConfigurationException {
+ String key = "testGetDescription_valid";
+ String expected = key + "_desc";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".description", expected);
+ String actual = ConfigDeployUtil.getDescription(key);
+ assertEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getType(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetType_bogus() throws ConfigurationException {
+ String k = "testGetType_bogus";
+ String actual = ConfigDeployUtil.getType(k);
+ String expected = null;
+ assertEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getType(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetType_valid() throws ConfigurationException {
+ String key = "testGetType_valid";
+ String expected = "float";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key+".type", expected);
+ String actual = ConfigDeployUtil.getType(key);
+ assertEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getDefault(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetDefault_invalid() throws ConfigurationException {
+ String key = "testGetDefault_invalid";
+ String sValue = null;
+ try {
+ ConfigDeployUtil.getDefault(key);
+ fail("Successfully called getDefault with bogus parameter string: ["
+ + key + ":" + sValue + "]");
+ } catch (ConfigurationException e) {
+ //ignore -- exception
+ }
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getDefault(java.lang.String)}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetDefault_valid() throws ConfigurationException {
+ String key = "testGetDefault_valid";
+ String expected = "defaultValue";
+ Properties p = ConfigDeployUtil.getDeploymentProperties();
+ p.setProperty(key + ".default", expected);
+ String actual = ConfigDeployUtil.getDefault(key);
+ assertEquals(actual, expected);
+ }
+
+ /**
+ * Test method for {@link com.bigdata.util.config.ConfigDeployUtil#getGroupsToDiscover()}.
+ * @throws ConfigurationException
+ */
+ @Test
+ public void testGetGroupsToDiscover_bogus() throws ConfigurationException {
+ try {
+ ConfigDeployUtil....
[truncated message content] |
|
From: <dm...@us...> - 2010-10-04 13:07:09
|
Revision: 3720
http://bigdata.svn.sourceforge.net/bigdata/?rev=3720&view=rev
Author: dmacgbr
Date: 2010-10-04 13:07:03 +0000 (Mon, 04 Oct 2010)
Log Message:
-----------
See Trac #183
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/src/resources/config/bigdataCluster.config
branches/QUADS_QUERY_BRANCH/src/resources/config/bigdataCluster16.config
branches/QUADS_QUERY_BRANCH/src/resources/config/bigdataStandalone.config
Modified: branches/QUADS_QUERY_BRANCH/src/resources/config/bigdataCluster.config
===================================================================
--- branches/QUADS_QUERY_BRANCH/src/resources/config/bigdataCluster.config 2010-10-02 16:10:07 UTC (rev 3719)
+++ branches/QUADS_QUERY_BRANCH/src/resources/config/bigdataCluster.config 2010-10-04 13:07:03 UTC (rev 3720)
@@ -401,6 +401,10 @@
*/
serviceCount = 1;
+ /**
+ * @see <a href="http://sourceforge.net/apps/trac/bigdata/ticket/183">trac 183</a>
+ */
+ timeout = 20000 ;
}
/**
Modified: branches/QUADS_QUERY_BRANCH/src/resources/config/bigdataCluster16.config
===================================================================
--- branches/QUADS_QUERY_BRANCH/src/resources/config/bigdataCluster16.config 2010-10-02 16:10:07 UTC (rev 3719)
+++ branches/QUADS_QUERY_BRANCH/src/resources/config/bigdataCluster16.config 2010-10-04 13:07:03 UTC (rev 3720)
@@ -447,6 +447,10 @@
*/
serviceCount = 1;
+ /**
+ * @see <a href="http://sourceforge.net/apps/trac/bigdata/ticket/183">trac 183</a>
+ */
+ timeout = 20000 ;
}
/**
Modified: branches/QUADS_QUERY_BRANCH/src/resources/config/bigdataStandalone.config
===================================================================
--- branches/QUADS_QUERY_BRANCH/src/resources/config/bigdataStandalone.config 2010-10-02 16:10:07 UTC (rev 3719)
+++ branches/QUADS_QUERY_BRANCH/src/resources/config/bigdataStandalone.config 2010-10-04 13:07:03 UTC (rev 3720)
@@ -423,6 +423,10 @@
*/
serviceCount = 1;
+ /**
+ * @see <a href="http://sourceforge.net/apps/trac/bigdata/ticket/183">trac 183</a>
+ */
+ timeout = 20000 ;
}
/**
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <ble...@us...> - 2010-10-02 16:10:14
|
Revision: 3719
http://bigdata.svn.sourceforge.net/bigdata/?rev=3719&view=rev
Author: blevine218
Date: 2010-10-02 16:10:07 +0000 (Sat, 02 Oct 2010)
Log Message:
-----------
part 1 of getting the unit tests to run in maven via the surefire plugin
- unit tests are run during the integration-test phase so that they can continue
to be based on resources included in the deployment tarball.
- surefire executes the top-level JUnit suite, com.bigdata.TestAll
- initial surefire infrastructure is in place, but unit test execution is disabled
by default pending additional testing of the tests.
- tests can be enabled by adding -DskipTests=false to the mvn command-line
- Execution of the unit tests via the tests.xml ANT script remains unchanged.
Modified Paths:
--------------
branches/maven_scaleout/bigdata-core/pom.xml
Modified: branches/maven_scaleout/bigdata-core/pom.xml
===================================================================
--- branches/maven_scaleout/bigdata-core/pom.xml 2010-10-02 01:00:49 UTC (rev 3718)
+++ branches/maven_scaleout/bigdata-core/pom.xml 2010-10-02 16:10:07 UTC (rev 3719)
@@ -15,21 +15,49 @@
<name>Bigdata Core</name>
<url>http://www.bigdata.com</url>
- <properties>
- <thirdParty.groupId>com.bigdata.thirdparty</thirdParty.groupId><!-- group ID for non-public bigdata dependencies. -->
+ <properties>
+ <!-- group ID for non-public bigdata dependencies. -->
+ <thirdParty.groupId>com.bigdata.thirdparty</thirdParty.groupId>
+
+ <!-- These properties are used to set system properties that are required by the unit
+ tests. See the surefire plugin configuration below. Any of these can be overridden on
+ the command-line by adding -D<property-name>=<propert-value>.
+ -->
+
+ <!-- Skip running unit tests by default until we get the kinks worked out. Can be overridden
+ by adding '-DskipTests=false' to the command-line -->
+ <skipTests>true</skipTests>
+
+ <!-- This directory is created when the deployment tarball is created
+ during the 'package' phase by the maven assembly plugin. If the naming
+ convention as defined in src/main/assembly/deploy.xml changes, this property
+ needs to change as well. -->
+ <deploy.root.dir>${project.build.directory}/${project.artifactId}-${project.version}-deploy</deploy.root.dir>
+
+ <deploy.dir>${deploy.root.dir}/${project.artifactId}-${project.version}</deploy.dir>
+ <app.home>${deploy.dir}</app.home>
+ <test.dir>${deploy.dir}/testing</test.dir>
+ <deploy.conf.dir>${test.dir}/conf</deploy.conf.dir>
+ <deploy.lib>${deploy.dir}/lib</deploy.lib>
+ <test.codebase.dir>${deploy.lib.dl}</test.codebase.dir>
+ <test.codebase.port>23333</test.codebase.port>
+ <java.security.policy>${deploy.conf.dir}/policy.all</java.security.policy>
+ <log4j.configuration>${deploy.dir}/var/config/logging/log4j.properties</log4j.configuration>
+ <java.net.preferIPv4Stack>true</java.net.preferIPv4Stack>
+ <default.nic>eth0</default.nic>
+
+ <!-- Set to empty string to indicate "unset." Application code will set a reasonable default -->
+ <federation.name></federation.name>
</properties>
-
<build>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<compilerArguments>
- <!--
- Apparently Javac may compile java source files inside jars put on the classpath. Weird. Zookeeper 3.2.1 jar contained classes and sources, and under some circumstances, the
- java files were getting recompiled and put into the bigdata jar. This setting forces javac to only look for source in the current maven source directory.
- -->
+ <!-- Apparently Javac may compile java source files inside jars put on the classpath. Weird. Zookeeper 3.2.1 jar contained classes and sources, and under some circumstances, the
+ java files were getting recompiled and put into the bigdata jar. This setting forces javac to only look for source in the current maven source directory. -->
<sourcepath>${project.build.sourceDirectory}</sourcepath>
</compilerArguments>
</configuration>
@@ -82,48 +110,65 @@
</executions>
</plugin>
+ <!-- Configure the surefire plugin to run in the 'integration-test' phase rather than the 'test' phase.
+ This allows us to run the unit tests after packaging has been done, but before the build artifacts
+ are installed during the 'install' phase -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
- <!--
- TODO: BLECH! All the tests are excluded from the regular unit test phase. TODO: Need to split out the unit tests and the heavier integration tests, plus TODO: get all the unit
- tests passing so regressions can properly fail the build.
- -->
- <testFailureIgnore>true</testFailureIgnore>
- <includes />
- <excludes>
- <exclude>**/*</exclude>
- </excludes>
+ <skip>true</skip>
</configuration>
- </plugin>
-
- <plugin>
- <!--
- These are where the heavier tests can be run. Right now failsafe looks for tests starting or ending with IT, aka FooIT.java or ITFoo.java, which don't exist yet, so nothing runs.
- -->
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
- <id>integration-test</id>
+ <id>unit-tests</id>
+ <phase>integration-test</phase>
<goals>
- <goal>integration-test</goal>
+ <goal>test</goal>
</goals>
+ <configuration>
+ <skip>false</skip>
+ <!-- Run only the top-level suite which in turn runs the other suites and tests -->
+ <test>com.bigdata.TestAll</test>
+
+ <!-- These system properties are required by the unit tests. -->
+ <systemPropertyVariables>
+ <java.security.policy>${java.security.policy}</java.security.policy>
+ <java.net.preferIPv4Stack>{java.net.preferIPv4Stack}"</java.net.preferIPv4Stack>
+ <log4j.configuration>${log4j.configuration}</log4j.configuration>
+
+ <app.home>${app.home}</app.home> <!-- This is the deployment directory, easily accessed by the DataFinder class. -->
+ <log4j.path>${log4j.configuration}</log4j.path>
+ <default.nic>${default.nic}</default.nic>
+ <federation.name>${federation.name}</federation.name>
+
+ <classserver.jar>${deploy.lib}/classserver.jar</classserver.jar>
+ <colt.jar>${deploy.lib}/colt.jar</colt.jar>
+ <ctc_utils.jar>${deploy.lib}/ctc_utils.jar</ctc_utils.jar>
+ <cweb-commons.jar>${deploy.lib}/cweb-commons.jar</cweb-commons.jar>
+ <cweb-extser.jar>${deploy.lib}/cweb-extser.jar</cweb-extser.jar>
+ <highscalelib.jar>${deploy.lib}/highscalelib.jar</highscalelib.jar>
+ <dsiutils.jar>${deploy.lib}/dsiutils.jar</dsiutils.jar>
+ <lgplutils.jar>${deploy.lib}/lgplutils.jar</lgplutils.jar>
+ <fastutil.jar>${deploy.lib}/fastutil.jar</fastutil.jar>
+ <icu4j.jar>${deploy.lib}/icu4j.jar</icu4j.jar>
+ <jsk-lib.jar>${deploy.lib}/jsk-lib.jar</jsk-lib.jar>
+ <jsk-platform.jar>${deploy.lib}jsk-platform.jar</jsk-platform.jar>
+ <log4j.jar>${deploy.lib}/log4j.jar</log4j.jar>
+ <iris.jar>${deploy.lib}/iris.jar</iris.jar>
+ <jgrapht.jar>${deploy.lib}/jgrapht.jar</jgrapht.jar>
+ <openrdf-sesame.jar>${deploy.lib}/openrdf-sesame.jar</openrdf-sesame.jar>
+ <slf4j.jar>${deploy.lib}/slf4j.jar</slf4j.jar>
+ <nxparser.jar>${deploy.lib}/nxparser.jar</nxparser.jar>
+ <zookeeper.jar>${deploy.lib}/zookeeper.jar</zookeeper.jar>
+ </systemPropertyVariables>
+ </configuration>
</execution>
- <execution>
- <id>verify</id>
- <goals>
- <goal>verify</goal>
- </goals>
- </execution>
</executions>
</plugin>
<plugin>
- <!-- Workaround assembly issue for nicely organized libs in the deployment tarball. For more info
- see src/main/assemblies/deploy.xml
- -->
+ <!-- Workaround assembly issue for nicely organized libs in the deployment tarball. For more info see src/main/assemblies/deploy.xml -->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
@@ -189,13 +234,9 @@
</execution>
</executions>
</plugin>
-
-
</plugins>
-
</build>
-
<dependencies>
@@ -204,7 +245,7 @@
<!-- ************************ Start of non-public dependencies ************************ -->
<!-- ************************ Start of non-public dependencies ************************ -->
- <!-- TODO: look at maven-bundle-plugin from felix to provide osgi support. bndlib version 0.0.357 in central. -->
+ <!-- TODO: look at maven-bundle-plugin from felix to provide osgi support. bndlib version 0.0.357 in central. -->
<dependency>
<groupId>${thirdParty.groupId}</groupId>
@@ -259,7 +300,7 @@
<version>[3.2.1,3.3)</version> <!-- TODO: This artifact is older, but the new 3.3.1 artifact in central is incompatible -->
</dependency>
- <!-- Apache River -->
+ <!-- Apache River -->
<dependency>
<groupId>org.apache.river</groupId>
<artifactId>tools</artifactId>
@@ -388,10 +429,8 @@
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.15</version>
- <!--
- These exclusions are to address the fact that 1.2.15 added new features that depends on Sun specific jars, but these jars cannot be made available due to Sun's click-through
- requirement on them. We aren't using the new features anyway, so they are safe to exclude. log4j should have made these optional in their POM.
- -->
+ <!-- These exclusions are to address the fact that 1.2.15 added new features that depends on Sun specific jars, but these jars cannot be made available due to Sun's click-through requirement
+ on them. We aren't using the new features anyway, so they are safe to exclude. log4j should have made these optional in their POM. -->
<exclusions>
<exclusion>
<groupId>javax.mail</groupId>
@@ -584,9 +623,7 @@
<groupId>com.atlassian.maven.plugins</groupId>
<artifactId>maven-clover2-plugin</artifactId>
<configuration>
- <!--
- Details on configuring clover are in the parent pom.
- -->
+ <!-- Details on configuring clover are in the parent pom. -->
<excludes><!-- These are stuffed into the lookup starter jar, which uses a stripped classpath without clover. -->
<exclude>**/LookupStarter*.java</exclude>
<exclude>**/LogUtil.java</exclude>
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-10-02 01:00:56
|
Revision: 3718
http://bigdata.svn.sourceforge.net/bigdata/?rev=3718&view=rev
Author: thompsonbry
Date: 2010-10-02 01:00:49 +0000 (Sat, 02 Oct 2010)
Log Message:
-----------
Worked through more of the default graph access path decision tree. Only the parallel subquery case is left. I've also simplified the decision tree in the worksheet.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/architecture/query-cost-model.xls
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/filter/DistinctFilter.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/bop/rdf/filter/StripContextFilter.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestContextAdvancer.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/architecture/query-cost-model.xls
===================================================================
(Binary files differ)
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java 2010-10-01 20:37:19 UTC (rev 3717)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java 2010-10-02 01:00:49 UTC (rev 3718)
@@ -408,7 +408,10 @@
if (!distinct.add(t) && !(t instanceof IVariableOrConstant<?>)) {
/*
* BOp appears more than once. This is only allowed for
- * constants and variables.
+ * constants and variables to reduce the likelihood of operator
+ * trees which describe loops. This will not detect operator
+ * trees whose sinks target a descendant, which is another way
+ * to create a loop.
*/
throw new DuplicateBOpException(t.toString());
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/filter/DistinctFilter.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/filter/DistinctFilter.java 2010-10-01 20:37:19 UTC (rev 3717)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/filter/DistinctFilter.java 2010-10-02 01:00:49 UTC (rev 3718)
@@ -21,6 +21,10 @@
* @version $Id: DistinctElementFilter.java 3466 2010-08-27 14:28:04Z
* thompsonbry $
*
+ * @todo Extract a common interface or metadata for all DISTINCT element filters
+ * (in memory hash map, persistence capable hash map, distributed hash
+ * map).
+ *
* @todo Reconcile with {@link IChunkConverter},
* {@link com.bigdata.striterator.DistinctFilter} (handles solutions) and
* {@link MergeFilter} (handles comparables),
@@ -39,6 +43,13 @@
}
/**
+ * A instance using the default configuration for the in memory hash map.
+ */
+ public static DistinctFilter newInstance() {
+ return new DistinctFilter(NOARGS, NOANNS);
+ }
+
+ /**
* Required deep copy constructor.
*/
public DistinctFilter(final DistinctFilter op) {
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java 2010-10-01 20:37:19 UTC (rev 3717)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java 2010-10-02 01:00:49 UTC (rev 3718)
@@ -59,7 +59,9 @@
import com.bigdata.bop.PipelineOp;
import com.bigdata.bop.Var;
import com.bigdata.bop.ap.Predicate;
+import com.bigdata.bop.ap.filter.DistinctFilter;
import com.bigdata.bop.bset.StartOp;
+import com.bigdata.bop.fed.FederatedQueryEngine;
import com.bigdata.bop.join.PipelineJoin;
import com.bigdata.bop.rdf.filter.StripContextFilter;
import com.bigdata.bop.rdf.join.DataSetJoin;
@@ -89,6 +91,7 @@
import com.bigdata.relation.rule.eval.DefaultEvaluationPlan2;
import com.bigdata.relation.rule.eval.IRangeCountFactory;
import com.bigdata.relation.rule.eval.RuleState;
+import com.bigdata.service.ResourceService;
import com.bigdata.striterator.IKeyOrder;
/**
@@ -119,6 +122,14 @@
* {@link DefaultGraphSolutionExpander}.
*/
private static boolean enableDecisionTree = false;
+
+ /**
+ * The #of samples to take when comparing the cost of a SCAN with an IN
+ * filter to subquery for each graph in the data set.
+ *
+ * @todo Add query hint to override this default.
+ */
+ private static final int SAMPLE_LIMIT = 100;
/**
* Annotations used by the {@link BigdataEvaluationStrategyImpl} to
@@ -248,8 +259,8 @@
public static PipelineOp convert(final IRule rule, final int startId,
final AbstractTripleStore db, final QueryEngine queryEngine) {
- // true iff the database is in quads mode.
- final boolean isQuadsQuery = db.isQuads();
+// // true iff the database is in quads mode.
+// final boolean isQuadsQuery = db.isQuads();
int bopId = startId;
@@ -508,15 +519,6 @@
final List<NV> anns, Predicate pred, final Dataset dataset,
final org.openrdf.query.algebra.Var cvar) {
- /*
- * @todo raise this into the caller and do one per rule rather than once
- * per access path.
- */
- final DataSetSummary summary = new DataSetSummary(dataset
- .getNamedGraphs());
-
- anns.add(new NV(Annotations.NKNOWN, summary.nknown));
-
final boolean scaleOut = queryEngine.isScaleOut();
if (scaleOut) {
anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
@@ -528,7 +530,19 @@
// true iff C is bound to a constant.
final boolean isCBound = cvar.getValue() != null;
-
+
+ if (dataset == null) {
+
+ /*
+ * The dataset is all graphs. C is left unbound and the unmodified
+ * access path is used.
+ */
+
+ return new PipelineJoin(new BOp[] { left, pred }, anns
+ .toArray(new NV[anns.size()]));
+
+ }
+
if (isCBound) {
/*
@@ -538,8 +552,19 @@
return new PipelineJoin(new BOp[] { left, pred }, anns
.toArray(new NV[anns.size()]));
- } else if (summary.nknown == 0) {
+ }
+
+ /*
+ * @todo raise this into the caller and do one per rule rather than once
+ * per access path.
+ */
+ final DataSetSummary summary = new DataSetSummary(dataset
+ .getNamedGraphs());
+ anns.add(new NV(Annotations.NKNOWN, summary.nknown));
+
+ if (summary.nknown == 0) {
+
/*
* The data set is empty (no graphs). Return a join backed by an
* empty access path.
@@ -553,133 +578,96 @@
return new PipelineJoin(new BOp[] { left, pred }, anns
.toArray(new NV[anns.size()]));
- } else if (summary.nknown == 1) {
+ }
+ if (summary.nknown == 1) {
+
/*
* The dataset contains exactly one graph. Bind C.
*/
-
+
pred = pred.asBound((IVariable) pred.get(3), new Constant(
summary.firstContext));
-
+
return new PipelineJoin(new BOp[] { left, pred }, anns
.toArray(new NV[anns.size()]));
- } else if (dataset == null) {
+ }
+ /*
+ * Estimate cost of SCAN with C unbound.
+ */
+ final double scanCost = queryEngine.estimateCost(context, pred);
+ anns.add(new NV(Annotations.COST_SCAN, scanCost));
+
+ // Estimate cost of SUBQUERY with C bound (sampling).
+ final double subqueryCost = summary.estimateSubqueryCost(queryEngine,
+ context, SAMPLE_LIMIT, pred, anns);
+
+ if (scanCost < subqueryCost) {
+
/*
- * The dataset is all graphs. C is left unbound and the unmodified
- * access path is used.
+ * Scan and filter. C is left unbound. We do a range scan on the
+ * index and filter using an IN constraint.
*/
+ // IN filter for the named graphs.
+ final IElementFilter<ISPO> test = new InGraphHashSetFilter<ISPO>(
+ summary.nknown, summary.graphs);
+
+ // layer filter onto the predicate.
+ pred = pred.addIndexLocalFilter(ElementFilter.newInstance(test));
+
return new PipelineJoin(new BOp[] { left, pred }, anns
.toArray(new NV[anns.size()]));
} else {
/*
- * Estimate cost of SCAN with C unbound.
+ * Parallel Subquery.
*/
- final double scanCost = queryEngine.estimateCost(context, pred);
- anns.add(new NV(Annotations.COST_SCAN, scanCost));
-
/*
- * Estimate cost of SUBQUERY with C bound (sampling).
+ * Setup the data set join.
*
- * @todo This should randomly sample in case there is bias in the
- * order in which the URIs are presented here. However, the only
- * thing which would be likely to create a strong bias is if someone
- * sorted them on the IVs or if the URIs were in the same ordering
- * in which their IVs were assigned AND the data were somehow
- * correlated with that order. I rate the latter as pretty unlikely
- * and the former is not true, so this sampling approach should be
- * pretty good.
+ * @todo When the #of named graphs is large we need to do something
+ * special to avoid sending huge graph sets around with the query.
+ * For example, we should create named data sets and join against
+ * them rather than having an in-memory DataSetJoin.
*
- * @todo parameter for the #of samples to take.
+ * @todo The historical approach performed parallel subquery using
+ * an expander pattern rather than a data set join. The data set
+ * join should have very much the same effect, but it may need to
+ * emit multiple chunks to have good parallelism.
*/
- double subqueryCost = 0d;
- final int limit = 100;
- int nsamples = 0;
- for (URI uri : summary.graphs) {
- if (nsamples == limit)
- break;
- final IV graph = ((BigdataURI) uri).getIV();
- subqueryCost += queryEngine.estimateCost(context, pred.asBound(
- (IVariable) pred.get(3), new Constant(graph)));
- nsamples++;
- }
- subqueryCost /= nsamples;
- anns.add(new NV(Annotations.COST_SUBQUERY, subqueryCost));
- anns.add(new NV(Annotations.COST_SUBQUERY_SAMPLE_COUNT, nsamples));
+ // The variable to be bound.
+ final IVariable var = (IVariable) pred.get(3);
- if (scanCost < subqueryCost * summary.nknown) {
+ // The data set join.
+ final DataSetJoin dataSetJoin = new DataSetJoin(new BOp[] { var },
+ NV.asMap(new NV[] {
+ new NV(DataSetJoin.Annotations.VAR, var),
+ new NV(DataSetJoin.Annotations.GRAPHS, summary
+ .getGraphs()) }));
- /*
- * Scan and filter. C is left unbound. We do a range scan on the
- * index and filter using an IN constraint.
- */
-
- // IN filter for the named graphs.
- final IElementFilter<ISPO> test = new InGraphHashSetFilter<ISPO>(
- summary.nknown, summary.graphs);
-
- // layer filter onto the predicate.
- pred = pred
- .addIndexLocalFilter(ElementFilter.newInstance(test));
-
- return new PipelineJoin(new BOp[] { left, pred }, anns
- .toArray(new NV[anns.size()]));
-
+ if (scaleOut) {
+ anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
+ BOpEvaluationContext.SHARDED));
+ anns
+ .add(new NV(Predicate.Annotations.REMOTE_ACCESS_PATH,
+ false));
} else {
+ anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
+ BOpEvaluationContext.ANY));
+ anns
+ .add(new NV(Predicate.Annotations.REMOTE_ACCESS_PATH,
+ false));
+ }
- /*
- * Parallel Subquery.
- */
+ return new PipelineJoin(new BOp[] { left, pred }, anns
+ .toArray(new NV[anns.size()]));
- /*
- * Setup the data set join.
- *
- * @todo When the #of named graphs is large we need to do
- * something special to avoid sending huge graph sets around
- * with the query. For example, we should create named data sets
- * and join against them rather than having an in-memory
- * DataSetJoin.
- *
- * @todo The historical approach performed parallel subquery
- * using an expander pattern rather than a data set join. The
- * data set join should have very much the same effect, but it
- * may need to emit multiple chunks to have good parallelism.
- */
-
- // The variable to be bound.
- final IVariable var = (IVariable) pred.get(3);
-
- // The data set join.
- final DataSetJoin dataSetJoin = new DataSetJoin(
- new BOp[] { var }, NV.asMap(new NV[] {
- new NV(DataSetJoin.Annotations.VAR, var),
- new NV(DataSetJoin.Annotations.GRAPHS, summary
- .getGraphs()) }));
-
- if (scaleOut) {
- anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.SHARDED));
- anns.add(new NV(Predicate.Annotations.REMOTE_ACCESS_PATH,
- false));
- } else {
- anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.ANY));
- anns.add(new NV(Predicate.Annotations.REMOTE_ACCESS_PATH,
- false));
- }
-
- return new PipelineJoin(new BOp[] { left, pred }, anns
- .toArray(new NV[anns.size()]));
-
- }
-
}
}
@@ -692,10 +680,6 @@
* @param anns
* @param pred
* @return
- *
- * @todo The default graph remote access path query estimates do not take
- * RMI costs into account. This is Ok since we are only comparing
- * remote access paths with other remote access paths.
*/
private static PipelineOp defaultGraphJoin(final QueryEngine queryEngine,
final BOpContextBase context, final PipelineOp left,
@@ -706,15 +690,12 @@
* @todo raise this into the caller and do one per rule rather than once
* per access path.
*/
- final DataSetSummary summary = new DataSetSummary(dataset
- .getDefaultGraphs());
+ final DataSetSummary summary = dataset == null ? null
+ : new DataSetSummary(dataset.getDefaultGraphs());
- // true iff C is bound to a constant.
- final boolean isCBound = cvar.getValue() != null;
-
final boolean scaleOut = queryEngine.isScaleOut();
- if (summary.nknown == 0) {
+ if (dataset != null && summary.nknown == 0) {
/*
* The data set is empty (no graphs). Return a join backed by an
@@ -729,7 +710,9 @@
return new PipelineJoin(new BOp[] { left, pred }, anns
.toArray(new NV[anns.size()]));
- } else if (summary.nknown == 1) {
+ }
+
+ if (dataset != null && summary.nknown == 1) {
/*
* The dataset contains exactly one graph. Bind C. Add a filter to
@@ -740,12 +723,14 @@
pred = pred.asBound((IVariable) pred.get(3), new Constant(
summary.firstContext));
- pred = pred.addAccessPathFilter(StripContextFilter.INSTANCE);
+ pred = pred.addAccessPathFilter(StripContextFilter.newInstance());
return new PipelineJoin(new BOp[] { left, pred }, anns
.toArray(new NV[anns.size()]));
- } else if (pred.getKeyOrder().getIndexName().endsWith("C")) {
+ }
+
+ if (pred.getKeyOrder().getIndexName().endsWith("C")) {
/*
* C is not bound. An advancer is imposed on the AP to skip to the
@@ -764,7 +749,7 @@
pred = pred.addIndexLocalFilter(new ContextAdvancer());
// Filter to strip off the context position.
- pred = pred.addAccessPathFilter(StripContextFilter.INSTANCE);
+ pred = pred.addAccessPathFilter(StripContextFilter.newInstance());
if(scaleOut) {
@@ -802,129 +787,115 @@
}
}
+
return new PipelineJoin(new BOp[] { left, pred }, anns
.toArray(new NV[anns.size()]));
+
+ }
+
+ // Estimate cost of SCAN with C unbound.
+ final double scanCost = queryEngine.estimateCost(context, pred);
+ anns.add(new NV(Annotations.COST_SCAN, scanCost));
+
+ /*
+ * Estimate cost of SUBQUERY with C bound (sampling). A large value is
+ * used if the dataset is null since the default graph query will run
+ * against all contexts and we are better off doing a SCAN.
+ */
+ final double subqueryCost = dataset == null ? Double.MAX_VALUE
+ : summary.estimateSubqueryCost(queryEngine, context,
+ SAMPLE_LIMIT, pred, anns);
+
+ if (scanCost < subqueryCost) {
+
+ /*
+ * SCAN AND FILTER. C is not bound. Unless all graphs are used,
+ * layer IN filter on the AP to select for the desired graphs. Layer
+ * a filter on the AP to strip off the context position. Layer a
+ * DISTINCT filter on top of that.
+ */
+
+ if (dataset != null) {
+
+ // IN filter for the named graphs.
+ final IElementFilter<ISPO> test = new InGraphHashSetFilter<ISPO>(
+ summary.nknown, summary.graphs);
+
+ // layer filter onto the predicate.
+ pred = pred
+ .addIndexLocalFilter(ElementFilter.newInstance(test));
+
+ }
+
+ // Filter to strip off the context position.
+ pred = pred.addAccessPathFilter(StripContextFilter.newInstance());
+
+ // Filter for distinct SPOs.
+ pred = pred.addAccessPathFilter(DistinctFilter.newInstance());
+
+ return new PipelineJoin(new BOp[] { left, pred }, anns
+ .toArray(new NV[anns.size()]));
+
+ } else {
+
+ /*
+ * PARALLEL SUBQUERY. Bind each value of C in turn, issuing parallel
+ * subqueries against the asBound access paths using an expander
+ * pattern and layer on a filter to strip off the context position.
+ * The asBound access paths write on a shared buffer. That shared
+ * buffer is read from by the expander.
+ *
+ * Scale-out: JOIN is ANY or HASHED. AP is REMOTE.
+ *
+ * FIXME This needs to be implemented based on an expander pattern
+ * which we can capture from DefaultGraphExpander.
+ */
+
+ throw new UnsupportedOperationException();
- }
- // FIXME Finish the default graph decision tree.
- throw new UnsupportedOperationException();
-// } else if (dataset == null) {
-//
// /*
-// * The dataset is all graphs. C is left unbound and the unmodified
-// * access path is used.
-// */
-//
-// return new PipelineJoin(new BOp[] { left, pred }, anns
-// .toArray(new NV[anns.size()]));
-//
-// } else {
-//
-// /*
-// * Estimate cost of SCAN with C unbound.
-// */
-// final double scanCost = queryEngine.estimateCost(context, pred);
-//
-// anns.add(new NV(Annotations.COST_SCAN, scanCost));
-//
-// /*
-// * Estimate cost of SUBQUERY with C bound (sampling).
+// * Setup the data set join.
// *
-// * @todo This should randomly sample in case there is bias in the
-// * order in which the URIs are presented here. However, the only
-// * thing which would be likely to create a strong bias is if someone
-// * sorted them on the IVs or if the URIs were in the same ordering
-// * in which their IVs were assigned AND the data were somehow
-// * correlated with that order. I rate the latter as pretty unlikely
-// * and the former is not true, so this sampling approach should be
-// * pretty good.
+// * @todo When the #of named graphs is large we need to do something
+// * special to avoid sending huge graph sets around with the query.
+// * For example, we should create named data sets and join against
+// * them rather than having an in-memory DataSetJoin.
// *
-// * @todo parameter for the #of samples to take.
+// * @todo The historical approach performed parallel subquery using
+// * an expander pattern rather than a data set join. The data set
+// * join should have very much the same effect, but it may need to
+// * emit multiple chunks to have good parallelism.
// */
-// double subqueryCost = 0d;
-// final int limit = 100;
-// int nsamples = 0;
-// for (URI uri : summary.graphs) {
-// if (nsamples == limit)
-// break;
-// final IV graph = ((BigdataURI) uri).getIV();
-// subqueryCost += queryEngine.estimateCost(context, pred.asBound(
-// (IVariable) pred.get(3), new Constant(graph)));
-// nsamples++;
-// }
-// subqueryCost /= nsamples;
//
-// anns.add(new NV(Annotations.COST_SUBQUERY, subqueryCost));
-// anns.add(new NV(Annotations.COST_SUBQUERY_SAMPLE_COUNT, nsamples));
+// // The variable to be bound.
+// final IVariable var = (IVariable) pred.get(3);
//
-// if (scanCost < subqueryCost * summary.nknown) {
+// // The data set join.
+// final DataSetJoin dataSetJoin = new DataSetJoin(new BOp[] { var },
+// NV.asMap(new NV[] {
+// new NV(DataSetJoin.Annotations.VAR, var),
+// new NV(DataSetJoin.Annotations.GRAPHS, summary
+// .getGraphs()) }));
//
-// /*
-// * Scan and filter. C is left unbound. We do a range scan on the
-// * index and filter using an IN constraint.
-// */
-//
-// // IN filter for the named graphs.
-// final IElementFilter<ISPO> test = new InGraphHashSetFilter<ISPO>(
-// summary.nknown, summary.graphs);
-//
-// // layer filter onto the predicate.
-// pred = pred
-// .addIndexLocalFilter(ElementFilter.newInstance(test));
-//
-// return new PipelineJoin(new BOp[] { left, pred }, anns
-// .toArray(new NV[anns.size()]));
-//
+// if (scaleOut) {
+// anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
+// BOpEvaluationContext.SHARDED));
+// anns
+// .add(new NV(Predicate.Annotations.REMOTE_ACCESS_PATH,
+// false));
// } else {
-//
-// /*
-// * Parallel Subquery.
-// */
-//
-// /*
-// * Setup the data set join.
-// *
-// * @todo When the #of named graphs is large we need to do
-// * something special to avoid sending huge graph sets around
-// * with the query. For example, we should create named data sets
-// * and join against them rather than having an in-memory
-// * DataSetJoin.
-// *
-// * @todo The historical approach performed parallel subquery
-// * using an expander pattern rather than a data set join. The
-// * data set join should have very much the same effect, but it
-// * may need to emit multiple chunks to have good parallelism.
-// */
-//
-// // The variable to be bound.
-// final IVariable var = (IVariable) pred.get(3);
-//
-// // The data set join.
-// final DataSetJoin dataSetJoin = new DataSetJoin(
-// new BOp[] { var }, NV.asMap(new NV[] {
-// new NV(DataSetJoin.Annotations.VAR, var),
-// new NV(DataSetJoin.Annotations.GRAPHS, summary
-// .getGraphs()) }));
-//
-// if (scaleOut) {
-// anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
-// BOpEvaluationContext.SHARDED));
-// anns.add(new NV(Predicate.Annotations.REMOTE_ACCESS_PATH,
-// false));
-// } else {
-// anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
-// BOpEvaluationContext.ANY));
-// anns.add(new NV(Predicate.Annotations.REMOTE_ACCESS_PATH,
-// false));
-// }
-//
-// return new PipelineJoin(new BOp[] { left, pred }, anns
-// .toArray(new NV[anns.size()]));
-//
+// anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
+// BOpEvaluationContext.ANY));
+// anns
+// .add(new NV(Predicate.Annotations.REMOTE_ACCESS_PATH,
+// false));
// }
//
-// }
+// return new PipelineJoin(new BOp[] { left, pred }, anns
+// .toArray(new NV[anns.size()]));
+ }
+
}
/**
@@ -1113,7 +1084,61 @@
return a;
}
-
+
+ /**
+ * Estimate cost of SUBQUERY with C bound (sampling).
+ *
+ * @param queryEngine
+ * @param context
+ * @param limit
+ * The maximum #of samples to take.
+ * @param pred
+ * The predicate.
+ * @param anns
+ * A list of annotations to which the cost estimate data will
+ * be added.
+ *
+ * @return The estimated cost. This is adjusted based on the sample size
+ * and the #of graphs against which the query was issued and
+ * represents the total expected cost of the subqueries against
+ * all of the graphs in the {@link Dataset}.
+ *
+ * @todo Subquery will be less efficient than a scan when the access
+ * path is remote since there will be remote requests. This model
+ * does not capture that additional overhead. We need to measure
+ * the overhead using appropriate data sets and queries and then
+ * build it into the model. The overhead itself could be changed
+ * dramatically by optimizations in the
+ * {@link FederatedQueryEngine} and the {@link ResourceService}.
+ *
+ * @todo This should randomly sample in case there is bias in the order
+ * in which the URIs are presented here. However, the only thing
+ * which would be likely to create a strong bias is if someone
+ * sorted them on the IVs or if the URIs were in the same ordering
+ * in which their IVs were assigned AND the data were somehow
+ * correlated with that order. I rate the latter as pretty
+ * unlikely and the former is not true, so this sampling approach
+ * should be pretty good.
+ */
+ public double estimateSubqueryCost(QueryEngine queryEngine,
+ BOpContextBase context, int limit, Predicate pred, List<NV> anns) {
+ double subqueryCost = 0d;
+ int nsamples = 0;
+ for (URI uri : graphs) {
+ if (nsamples == limit)
+ break;
+ final IV graph = ((BigdataURI) uri).getIV();
+ subqueryCost += queryEngine.estimateCost(context, pred.asBound(
+ (IVariable) pred.get(3), new Constant(graph)));
+ nsamples++;
+ }
+ subqueryCost = (subqueryCost * nknown) / nsamples;
+
+ anns.add(new NV(Annotations.COST_SUBQUERY, subqueryCost));
+ anns.add(new NV(Annotations.COST_SUBQUERY_SAMPLE_COUNT, nsamples));
+ return subqueryCost;
+ }
+
} // DataSetSummary
/**
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/bop/rdf/filter/StripContextFilter.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/bop/rdf/filter/StripContextFilter.java 2010-10-01 20:37:19 UTC (rev 3717)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/bop/rdf/filter/StripContextFilter.java 2010-10-02 01:00:49 UTC (rev 3718)
@@ -52,10 +52,11 @@
private static final long serialVersionUID = 1L;
/**
- * A global instance.
+ * A default instance.
*/
- public static final transient StripContextFilter INSTANCE = new StripContextFilter(
- BOpBase.NOARGS, BOpBase.NOANNS);
+ public static StripContextFilter newInstance() {
+ return new StripContextFilter(BOpBase.NOARGS, BOpBase.NOANNS);
+ }
/**
* @param op
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestContextAdvancer.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestContextAdvancer.java 2010-10-01 20:37:19 UTC (rev 3717)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestContextAdvancer.java 2010-10-02 01:00:49 UTC (rev 3718)
@@ -180,7 +180,8 @@
pred = pred.addIndexLocalFilter(new ContextAdvancer());
- pred = pred.addAccessPathFilter(StripContextFilter.INSTANCE);
+ pred = pred.addAccessPathFilter(StripContextFilter
+ .newInstance());
final IAccessPath<ISPO> ap = context.getAccessPath(db
.getSPORelation(), pred);
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-10-01 20:37:27
|
Revision: 3717
http://bigdata.svn.sourceforge.net/bigdata/?rev=3717&view=rev
Author: thompsonbry
Date: 2010-10-01 20:37:19 +0000 (Fri, 01 Oct 2010)
Log Message:
-----------
Added the ContextAdvancer for default graph access paths. There is a unit test for this. It also tests the StripContextFilter as a side-effect.
Provided coverage for some of the default graph decision tree.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/architecture/query-cost-model.xls
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/bop/rdf/filter/StripContextFilter.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/DistinctMultiTermAdvancer.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestDistinctTermScan.java
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/ContextAdvancer.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestContextAdvancer.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/architecture/query-cost-model.xls
===================================================================
(Binary files differ)
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java 2010-10-01 19:03:22 UTC (rev 3716)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java 2010-10-01 20:37:19 UTC (rev 3717)
@@ -479,19 +479,20 @@
/*
* Set the filter.
*/
- _setProperty(Annotations.INDEX_LOCAL_FILTER, filter);
+ _setProperty(name, filter);
} else {
/*
* Wrap the filter.
*/
- _setProperty(Annotations.INDEX_LOCAL_FILTER, new FilterBase() {
+ _setProperty(name, new FilterBase() {
@Override
protected Iterator filterOnce(Iterator src, Object context) {
return src;
}
+
}.addFilter(current).addFilter(filter));
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java 2010-10-01 19:03:22 UTC (rev 3716)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java 2010-10-01 20:37:19 UTC (rev 3717)
@@ -61,8 +61,10 @@
import com.bigdata.bop.ap.Predicate;
import com.bigdata.bop.bset.StartOp;
import com.bigdata.bop.join.PipelineJoin;
+import com.bigdata.bop.rdf.filter.StripContextFilter;
import com.bigdata.bop.rdf.join.DataSetJoin;
import com.bigdata.bop.solutions.SliceOp;
+import com.bigdata.btree.IRangeQuery;
import com.bigdata.rdf.internal.IV;
import com.bigdata.rdf.internal.TermId;
import com.bigdata.rdf.internal.VTE;
@@ -70,6 +72,7 @@
import com.bigdata.rdf.model.BigdataURI;
import com.bigdata.rdf.sail.BigdataEvaluationStrategyImpl;
import com.bigdata.rdf.sail.BigdataSail;
+import com.bigdata.rdf.spo.ContextAdvancer;
import com.bigdata.rdf.spo.DefaultGraphSolutionExpander;
import com.bigdata.rdf.spo.ISPO;
import com.bigdata.rdf.spo.InGraphHashSetFilter;
@@ -245,6 +248,9 @@
public static PipelineOp convert(final IRule rule, final int startId,
final AbstractTripleStore db, final QueryEngine queryEngine) {
+ // true iff the database is in quads mode.
+ final boolean isQuadsQuery = db.isQuads();
+
int bopId = startId;
final PipelineOp startOp = new StartOp(new BOp[] {},
@@ -502,6 +508,15 @@
final List<NV> anns, Predicate pred, final Dataset dataset,
final org.openrdf.query.algebra.Var cvar) {
+ /*
+ * @todo raise this into the caller and do one per rule rather than once
+ * per access path.
+ */
+ final DataSetSummary summary = new DataSetSummary(dataset
+ .getNamedGraphs());
+
+ anns.add(new NV(Annotations.NKNOWN, summary.nknown));
+
final boolean scaleOut = queryEngine.isScaleOut();
if (scaleOut) {
anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
@@ -511,10 +526,6 @@
BOpEvaluationContext.ANY));
}
- final DataSetSummary summary = new DataSetSummary(dataset.getNamedGraphs());
-
- anns.add(new NV(Annotations.NKNOWN, summary.nknown));
-
// true iff C is bound to a constant.
final boolean isCBound = cvar.getValue() != null;
@@ -688,23 +699,231 @@
*/
private static PipelineOp defaultGraphJoin(final QueryEngine queryEngine,
final BOpContextBase context, final PipelineOp left,
- final List<NV> anns, final Predicate pred, final Dataset dataset,
+ final List<NV> anns, Predicate pred, final Dataset dataset,
final org.openrdf.query.algebra.Var cvar) {
- // @todo decision of local vs remote ap.
+ /*
+ * @todo raise this into the caller and do one per rule rather than once
+ * per access path.
+ */
+ final DataSetSummary summary = new DataSetSummary(dataset
+ .getDefaultGraphs());
+
+ // true iff C is bound to a constant.
+ final boolean isCBound = cvar.getValue() != null;
+
final boolean scaleOut = queryEngine.isScaleOut();
- if (scaleOut) {
- anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.SHARDED));
- } else {
- anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.ANY));
+
+ if (summary.nknown == 0) {
+
+ /*
+ * The data set is empty (no graphs). Return a join backed by an
+ * empty access path.
+ */
+
+ // force an empty access path for this predicate.
+ pred = (Predicate) pred.setUnboundProperty(
+ IPredicate.Annotations.ACCESS_PATH_EXPANDER,
+ EmptyAccessPathExpander.INSTANCE);
+
+ return new PipelineJoin(new BOp[] { left, pred }, anns
+ .toArray(new NV[anns.size()]));
+
+ } else if (summary.nknown == 1) {
+
+ /*
+ * The dataset contains exactly one graph. Bind C. Add a filter to
+ * strip off the context position.
+ */
+
+ // bind C.
+ pred = pred.asBound((IVariable) pred.get(3), new Constant(
+ summary.firstContext));
+
+ pred = pred.addAccessPathFilter(StripContextFilter.INSTANCE);
+
+ return new PipelineJoin(new BOp[] { left, pred }, anns
+ .toArray(new NV[anns.size()]));
+
+ } else if (pred.getKeyOrder().getIndexName().endsWith("C")) {
+
+ /*
+ * C is not bound. An advancer is imposed on the AP to skip to the
+ * next possible triple after each match. Impose filter on AP to
+ * strip off the context position. Distinct filter is not required
+ * since the advancer pattern used will not report duplicates.
+ */
+
+ // Set the CURSOR flag.
+ pred = (Predicate) pred.setProperty(IPredicate.Annotations.FLAGS,
+ pred.getProperty(IPredicate.Annotations.FLAGS,
+ IPredicate.Annotations.DEFAULT_FLAGS)
+ | IRangeQuery.CURSOR);
+
+ // Set Advancer (runs at the index).
+ pred = pred.addIndexLocalFilter(new ContextAdvancer());
+
+ // Filter to strip off the context position.
+ pred = pred.addAccessPathFilter(StripContextFilter.INSTANCE);
+
+ if(scaleOut) {
+
+ /*
+ * When true, an ISimpleSplitHandler guarantees that no triple
+ * on that index spans more than one shard.
+ *
+ * @todo Implement the split handler and detect when it is being
+ * used.
+ */
+ final boolean shardTripleConstraint = false;
+
+ if (shardTripleConstraint) {
+
+ // JOIN is SHARDED.
+ pred = (Predicate) pred.setProperty(
+ BOp.Annotations.EVALUATION_CONTEXT,
+ BOpEvaluationContext.SHARDED);
+
+ // AP is LOCAL.
+ pred = (Predicate) pred.setProperty(
+ IPredicate.Annotations.REMOTE_ACCESS_PATH, false);
+
+ } else {
+
+ // JOIN is ANY.
+ pred = (Predicate) pred.setProperty(
+ BOp.Annotations.EVALUATION_CONTEXT,
+ BOpEvaluationContext.ANY);
+
+ // AP is REMOTE.
+ pred = (Predicate) pred.setProperty(
+ IPredicate.Annotations.REMOTE_ACCESS_PATH, true);
+
+ }
+
+ }
+ return new PipelineJoin(new BOp[] { left, pred }, anns
+ .toArray(new NV[anns.size()]));
+
}
-
- /*
- * FIXME implement the default graph decision tree.
- */
+ // FIXME Finish the default graph decision tree.
throw new UnsupportedOperationException();
+// } else if (dataset == null) {
+//
+// /*
+// * The dataset is all graphs. C is left unbound and the unmodified
+// * access path is used.
+// */
+//
+// return new PipelineJoin(new BOp[] { left, pred }, anns
+// .toArray(new NV[anns.size()]));
+//
+// } else {
+//
+// /*
+// * Estimate cost of SCAN with C unbound.
+// */
+// final double scanCost = queryEngine.estimateCost(context, pred);
+//
+// anns.add(new NV(Annotations.COST_SCAN, scanCost));
+//
+// /*
+// * Estimate cost of SUBQUERY with C bound (sampling).
+// *
+// * @todo This should randomly sample in case there is bias in the
+// * order in which the URIs are presented here. However, the only
+// * thing which would be likely to create a strong bias is if someone
+// * sorted them on the IVs or if the URIs were in the same ordering
+// * in which their IVs were assigned AND the data were somehow
+// * correlated with that order. I rate the latter as pretty unlikely
+// * and the former is not true, so this sampling approach should be
+// * pretty good.
+// *
+// * @todo parameter for the #of samples to take.
+// */
+// double subqueryCost = 0d;
+// final int limit = 100;
+// int nsamples = 0;
+// for (URI uri : summary.graphs) {
+// if (nsamples == limit)
+// break;
+// final IV graph = ((BigdataURI) uri).getIV();
+// subqueryCost += queryEngine.estimateCost(context, pred.asBound(
+// (IVariable) pred.get(3), new Constant(graph)));
+// nsamples++;
+// }
+// subqueryCost /= nsamples;
+//
+// anns.add(new NV(Annotations.COST_SUBQUERY, subqueryCost));
+// anns.add(new NV(Annotations.COST_SUBQUERY_SAMPLE_COUNT, nsamples));
+//
+// if (scanCost < subqueryCost * summary.nknown) {
+//
+// /*
+// * Scan and filter. C is left unbound. We do a range scan on the
+// * index and filter using an IN constraint.
+// */
+//
+// // IN filter for the named graphs.
+// final IElementFilter<ISPO> test = new InGraphHashSetFilter<ISPO>(
+// summary.nknown, summary.graphs);
+//
+// // layer filter onto the predicate.
+// pred = pred
+// .addIndexLocalFilter(ElementFilter.newInstance(test));
+//
+// return new PipelineJoin(new BOp[] { left, pred }, anns
+// .toArray(new NV[anns.size()]));
+//
+// } else {
+//
+// /*
+// * Parallel Subquery.
+// */
+//
+// /*
+// * Setup the data set join.
+// *
+// * @todo When the #of named graphs is large we need to do
+// * something special to avoid sending huge graph sets around
+// * with the query. For example, we should create named data sets
+// * and join against them rather than having an in-memory
+// * DataSetJoin.
+// *
+// * @todo The historical approach performed parallel subquery
+// * using an expander pattern rather than a data set join. The
+// * data set join should have very much the same effect, but it
+// * may need to emit multiple chunks to have good parallelism.
+// */
+//
+// // The variable to be bound.
+// final IVariable var = (IVariable) pred.get(3);
+//
+// // The data set join.
+// final DataSetJoin dataSetJoin = new DataSetJoin(
+// new BOp[] { var }, NV.asMap(new NV[] {
+// new NV(DataSetJoin.Annotations.VAR, var),
+// new NV(DataSetJoin.Annotations.GRAPHS, summary
+// .getGraphs()) }));
+//
+// if (scaleOut) {
+// anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
+// BOpEvaluationContext.SHARDED));
+// anns.add(new NV(Predicate.Annotations.REMOTE_ACCESS_PATH,
+// false));
+// } else {
+// anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
+// BOpEvaluationContext.ANY));
+// anns.add(new NV(Predicate.Annotations.REMOTE_ACCESS_PATH,
+// false));
+// }
+//
+// return new PipelineJoin(new BOp[] { left, pred }, anns
+// .toArray(new NV[anns.size()]));
+//
+// }
+//
+// }
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/bop/rdf/filter/StripContextFilter.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/bop/rdf/filter/StripContextFilter.java 2010-10-01 19:03:22 UTC (rev 3716)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/bop/rdf/filter/StripContextFilter.java 2010-10-01 20:37:19 UTC (rev 3717)
@@ -30,13 +30,16 @@
import java.util.Map;
import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpBase;
+import com.bigdata.bop.IPredicate;
import com.bigdata.bop.ap.filter.BOpResolver;
import com.bigdata.rdf.spo.ISPO;
import com.bigdata.rdf.spo.SPO;
/**
* Strips the context information from an {@link SPO}. This is used in default
- * graph access paths.
+ * graph access paths. It operators on {@link ISPO}s so it must be applied using
+ * {@link IPredicate.Annotations#ACCESS_PATH_FILTER}.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
* @version $Id$
@@ -48,6 +51,12 @@
*/
private static final long serialVersionUID = 1L;
+ /**
+ * A global instance.
+ */
+ public static final transient StripContextFilter INSTANCE = new StripContextFilter(
+ BOpBase.NOARGS, BOpBase.NOANNS);
+
/**
* @param op
*/
Added: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/ContextAdvancer.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/ContextAdvancer.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/ContextAdvancer.java 2010-10-01 20:37:19 UTC (rev 3717)
@@ -0,0 +1,104 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Oct 1, 2010
+ */
+
+package com.bigdata.rdf.spo;
+
+import com.bigdata.btree.IRangeQuery;
+import com.bigdata.btree.ITuple;
+import com.bigdata.btree.ITupleCursor;
+import com.bigdata.btree.filter.Advancer;
+import com.bigdata.btree.keys.IKeyBuilder;
+import com.bigdata.btree.keys.KeyBuilder;
+import com.bigdata.btree.keys.SuccessorUtil;
+import com.bigdata.rdf.internal.IV;
+import com.bigdata.rdf.internal.IVUtility;
+
+/**
+ * Advancer for a quads index whose last key component is the "context position
+ * (such as SPOC or SOPC). The advancer will skip to first possible key for the
+ * next distinct triple for each quad which it visits. This is a cheap way to
+ * impose a "DISTINCT" filter using an index scan and works well for both local
+ * and scale-out indices.
+ * <p>
+ * You have to use {@link IRangeQuery#CURSOR} to request an {@link ITupleCursor}
+ * when using an {@link Advancer} pattern.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class ContextAdvancer extends Advancer<SPO> {
+
+ private static final long serialVersionUID = 1L;
+
+ private transient IKeyBuilder keyBuilder;
+
+ public ContextAdvancer() {
+
+ }
+
+ @Override
+ protected void advance(final ITuple<SPO> tuple) {
+
+ if (keyBuilder == null) {
+
+ /*
+ * Note: It appears that you can not set this either implicitly or
+ * explicitly during ctor initialization if you want it to exist
+ * during de-serialization. Hence it is initialized lazily here.
+ * This is Ok since the iterator pattern is single threaded.
+ */
+
+ keyBuilder = KeyBuilder.newInstance();
+
+ }
+
+ // extract the key.
+ final byte[] key = tuple.getKey();
+
+ // decode the first three components of the key.
+ final IV[] terms = IVUtility.decode(key, 3/*nterms*/);
+
+ // reset the buffer.
+ keyBuilder.reset();
+
+ // encode the first three components of the key.
+ IVUtility.encode(keyBuilder,terms[0]);
+ IVUtility.encode(keyBuilder,terms[1]);
+ IVUtility.encode(keyBuilder,terms[2]);
+
+ // obtain the key.
+ final byte[] fromKey = keyBuilder.getKey();
+
+ // obtain the successor of the key.
+ final byte[] toKey = SuccessorUtil.successor(fromKey.clone());
+
+ // seek to that successor.
+ src.seek(toKey);
+
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/ContextAdvancer.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/DistinctMultiTermAdvancer.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/DistinctMultiTermAdvancer.java 2010-10-01 19:03:22 UTC (rev 3716)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/DistinctMultiTermAdvancer.java 2010-10-01 20:37:19 UTC (rev 3717)
@@ -47,7 +47,7 @@
* <p>
* Note: This class only offers additional functionality over the
* {@link DistinctTermAdvancer} for a quad store. For example, consider a triple
- * store with 2-bound on the {@link SPOKeyOrder#SPO} index. SInce you are only
+ * store with 2-bound on the {@link SPOKeyOrder#SPO} index. Since you are only
* going to visit the distinct Object values, the advancer will not "advance"
* over anything and you might as well use a normal {@link IAccessPath} or
* rangeIterator.
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestAll.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestAll.java 2010-10-01 19:03:22 UTC (rev 3716)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestAll.java 2010-10-01 20:37:19 UTC (rev 3717)
@@ -92,6 +92,9 @@
// test suite for distinct term scan
suite.addTestSuite(TestDistinctTermScan.class);
+ // test suite for the ContextAdvancer.
+ suite.addTestSuite(TestContextAdvancer.class);
+
// test suite for rdf1.
suite.addTestSuite(TestRuleRdf01.class);
Added: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestContextAdvancer.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestContextAdvancer.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestContextAdvancer.java 2010-10-01 20:37:19 UTC (rev 3717)
@@ -0,0 +1,203 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Oct 1, 2010
+ */
+
+package com.bigdata.rdf.rules;
+
+import java.util.Properties;
+
+import junit.framework.TestCase2;
+
+import org.openrdf.model.vocabulary.RDF;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpContextBase;
+import com.bigdata.bop.NV;
+import com.bigdata.bop.Var;
+import com.bigdata.bop.ap.Predicate;
+import com.bigdata.bop.rdf.filter.StripContextFilter;
+import com.bigdata.btree.IRangeQuery;
+import com.bigdata.journal.BufferMode;
+import com.bigdata.journal.ITx;
+import com.bigdata.journal.Journal;
+import com.bigdata.rdf.model.BigdataStatement;
+import com.bigdata.rdf.model.BigdataURI;
+import com.bigdata.rdf.model.BigdataValue;
+import com.bigdata.rdf.model.BigdataValueFactory;
+import com.bigdata.rdf.model.StatementEnum;
+import com.bigdata.rdf.spo.ContextAdvancer;
+import com.bigdata.rdf.spo.ISPO;
+import com.bigdata.rdf.spo.SPOKeyOrder;
+import com.bigdata.rdf.spo.SPOPredicate;
+import com.bigdata.rdf.store.AbstractTestCase;
+import com.bigdata.rdf.store.AbstractTripleStore;
+import com.bigdata.rdf.store.LocalTripleStore;
+import com.bigdata.relation.accesspath.IAccessPath;
+
+/**
+ * Test suite for the {@link ContextAdvancer}.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class TestContextAdvancer extends TestCase2 {
+
+ /**
+ *
+ */
+ public TestContextAdvancer() {
+ }
+
+ public TestContextAdvancer(String name) {
+ super(name);
+ }
+
+ /**
+ * Unit test verifies the {@link ContextAdvancer} against the
+ * {@link SPOKeyOrder#SPOC} index.
+ */
+ public void test_contextAdvancer() {
+
+ final Properties properties = new Properties();
+
+ properties.setProperty(AbstractTripleStore.Options.QUADS_MODE, "true");
+
+ properties.setProperty(Journal.Options.BUFFER_MODE,
+ BufferMode.Transient.toString());
+
+ final Journal store = new Journal(properties);
+
+ try {
+
+ final LocalTripleStore db = new LocalTripleStore(store, "test",
+ ITx.UNISOLATED, properties);
+
+ db.create();
+
+// final StatementBuffer<BigdataStatement> sb = new StatementBuffer<BigdataStatement>(
+// db, 100/* capacity */);
+
+ final BigdataValueFactory f = db.getValueFactory();
+
+ final BigdataURI u1 = f.createURI("http://www.bigdata.com/u1");
+ final BigdataURI u2 = f.createURI("http://www.bigdata.com/u2");
+ final BigdataURI v1 = f.createURI("http://www.bigdata.com/v1");
+ final BigdataURI v2 = f.createURI("http://www.bigdata.com/v2");
+ final BigdataURI c1 = f.createURI("http://www.bigdata.com/c1");
+ final BigdataURI c2 = f.createURI("http://www.bigdata.com/c2");
+ final BigdataURI rdfType = f.createURI(RDF.TYPE.stringValue());
+
+ final BigdataValue[] terms = new BigdataValue[] {
+ u1,u2,//
+ v1,v2,//
+ c1,c2,//
+ rdfType//
+ };
+
+ db.getLexiconRelation()
+ .addTerms(terms, terms.length, false/* readOnly */);
+
+ final StatementEnum explicit = StatementEnum.Explicit;
+ final BigdataStatement[] stmts = new BigdataStatement[]{
+ f.createStatement(u1, rdfType, v1, c1, explicit),
+ f.createStatement(u1, rdfType, v1, c2, explicit),
+ f.createStatement(u1, rdfType, v2, c1, explicit),
+ f.createStatement(u1, rdfType, v2, c2, explicit),
+ f.createStatement(u2, rdfType, v1, c1, explicit),
+ f.createStatement(u2, rdfType, v1, c2, explicit),
+ f.createStatement(u2, rdfType, v2, c1, explicit),
+ f.createStatement(u2, rdfType, v2, c2, explicit),
+ };
+
+ db.addStatements(stmts, stmts.length);
+
+ db.commit();
+
+ System.err.println(db.dumpStore());
+
+ // The expected distinct statements w/o their context info.
+ final BigdataStatement[] expectedDistinct = new BigdataStatement[]{
+ f.createStatement(u1, rdfType, v1),
+ f.createStatement(u1, rdfType, v2),
+ f.createStatement(u2, rdfType, v1),
+ f.createStatement(u2, rdfType, v2),
+ };
+
+ // predicate using the SPOC index.
+ Predicate<ISPO> pred = new SPOPredicate(new BOp[] { Var.var("s"),
+ Var.var("p"), Var.var("o"), Var.var("c") }, NV
+ .asMap(new NV[] {//
+ new NV(Predicate.Annotations.KEY_ORDER,
+ SPOKeyOrder.SPOC), //
+ new NV(Predicate.Annotations.TIMESTAMP,
+ ITx.UNISOLATED),//
+ }));
+
+ final BOpContextBase context = new BOpContextBase(null/* fed */,
+ store/* indexManager */);
+
+ // First verify assumptions without the advancer.
+ {
+
+ final IAccessPath<ISPO> ap = context.getAccessPath(db
+ .getSPORelation(), pred);
+
+ assertEquals(SPOKeyOrder.SPOC, ap.getKeyOrder());
+
+ assertEquals(stmts.length, ap.rangeCount(true/* exact */));
+
+ }
+
+ // Now verify assumptions with the advancer.
+ {
+
+ pred = (Predicate) pred.setProperty(
+ Predicate.Annotations.FLAGS, IRangeQuery.DEFAULT
+ | IRangeQuery.CURSOR);
+
+ pred = pred.addIndexLocalFilter(new ContextAdvancer());
+
+ pred = pred.addAccessPathFilter(StripContextFilter.INSTANCE);
+
+ final IAccessPath<ISPO> ap = context.getAccessPath(db
+ .getSPORelation(), pred);
+
+ assertEquals(4, ap.rangeCount(true/* exact */));
+
+ AbstractTestCase.assertSameSPOsAnyOrder(db, expectedDistinct,
+ ap.iterator());
+
+ }
+
+ } finally {
+
+ store.destroy();
+
+ }
+
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestContextAdvancer.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestDistinctTermScan.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestDistinctTermScan.java 2010-10-01 19:03:22 UTC (rev 3716)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestDistinctTermScan.java 2010-10-01 20:37:19 UTC (rev 3717)
@@ -55,10 +55,13 @@
import com.bigdata.rdf.internal.IV;
import com.bigdata.rdf.rio.IStatementBuffer;
import com.bigdata.rdf.rio.StatementBuffer;
+import com.bigdata.rdf.spo.DistinctTermAdvancer;
import com.bigdata.rdf.spo.SPOKeyOrder;
import com.bigdata.rdf.store.AbstractTripleStore;
/**
+ * Test suite for the {@link DistinctTermAdvancer}.
+ *
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
* @version $Id$
*/
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <sgo...@us...> - 2010-10-01 19:03:31
|
Revision: 3716
http://bigdata.svn.sourceforge.net/bigdata/?rev=3716&view=rev
Author: sgossard
Date: 2010-10-01 19:03:22 +0000 (Fri, 01 Oct 2010)
Log Message:
-----------
[maven_scaleout] : Broke all direct dependency cycles with package 'com.bigdata.concurrent'. Removed old NonBlockingLockManager class and moved logic from NonBlockingLockManagerWithNewDesign that updated journal related write counters into WriteExecutorService.MyLockManager subclass already in journal package.
Modified Paths:
--------------
branches/maven_scaleout/bigdata-core/bigdata-perf/bsbm/src/resources/logging/log4j.properties
branches/maven_scaleout/bigdata-core/bigdata-perf/btc/src/resources/logging/log4j.properties
branches/maven_scaleout/bigdata-core/bigdata-perf/lubm/src/resources/logging/log4j.properties
branches/maven_scaleout/bigdata-core/bigdata-perf/uniprot/src/resources/logging/log4j.properties
branches/maven_scaleout/bigdata-core/src/main/deploy/legacy/bigdata-resources/logging/log4j-perf-tests.properties
branches/maven_scaleout/bigdata-core/src/main/deploy/var/config/logging/log4j.properties
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/concurrent/NonBlockingLockManagerWithNewDesign.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/journal/AbstractTask.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/journal/ConcurrencyManager.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/journal/WriteExecutorService.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/concurrent/AbstractStressTestNonBlockingLockManager.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/concurrent/TestAll.java
Removed Paths:
-------------
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/concurrent/NonBlockingLockManager.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/concurrent/TestNonBlockingLockManager.java
Modified: branches/maven_scaleout/bigdata-core/bigdata-perf/bsbm/src/resources/logging/log4j.properties
===================================================================
--- branches/maven_scaleout/bigdata-core/bigdata-perf/bsbm/src/resources/logging/log4j.properties 2010-10-01 16:48:09 UTC (rev 3715)
+++ branches/maven_scaleout/bigdata-core/bigdata-perf/bsbm/src/resources/logging/log4j.properties 2010-10-01 19:03:22 UTC (rev 3716)
@@ -68,7 +68,6 @@
#log4j.logger.com.bigdata.service.AbstractTransactionService=INFO
#log4j.logger.com.bigdata.journal.AbstractLocalTransactionManager=INFO
#log4j.logger.com.bigdata.concurrent.TxDag=WARN
-#log4j.logger.com.bigdata.concurrent.NonBlockingLockManager=WARN
#log4j.logger.com.bigdata.concurrent.TestNonBlockingLockManager=INFO
#log4j.logger.com.bigdata.concurrent.AbstractStressTestNonBlockingLockManager=INFO
#log4j.logger.com.bigdata.concurrent.LockManager=INFO
Modified: branches/maven_scaleout/bigdata-core/bigdata-perf/btc/src/resources/logging/log4j.properties
===================================================================
--- branches/maven_scaleout/bigdata-core/bigdata-perf/btc/src/resources/logging/log4j.properties 2010-10-01 16:48:09 UTC (rev 3715)
+++ branches/maven_scaleout/bigdata-core/bigdata-perf/btc/src/resources/logging/log4j.properties 2010-10-01 19:03:22 UTC (rev 3716)
@@ -67,7 +67,6 @@
#log4j.logger.com.bigdata.service.AbstractTransactionService=INFO
#log4j.logger.com.bigdata.journal.AbstractLocalTransactionManager=INFO
#log4j.logger.com.bigdata.concurrent.TxDag=WARN
-#log4j.logger.com.bigdata.concurrent.NonBlockingLockManager=WARN
#log4j.logger.com.bigdata.concurrent.TestNonBlockingLockManager=INFO
#log4j.logger.com.bigdata.concurrent.AbstractStressTestNonBlockingLockManager=INFO
#log4j.logger.com.bigdata.concurrent.LockManager=INFO
Modified: branches/maven_scaleout/bigdata-core/bigdata-perf/lubm/src/resources/logging/log4j.properties
===================================================================
--- branches/maven_scaleout/bigdata-core/bigdata-perf/lubm/src/resources/logging/log4j.properties 2010-10-01 16:48:09 UTC (rev 3715)
+++ branches/maven_scaleout/bigdata-core/bigdata-perf/lubm/src/resources/logging/log4j.properties 2010-10-01 19:03:22 UTC (rev 3716)
@@ -57,7 +57,6 @@
#log4j.logger.com.bigdata.service.AbstractTransactionService=INFO
#log4j.logger.com.bigdata.journal.AbstractLocalTransactionManager=INFO
log4j.logger.com.bigdata.concurrent.TxDag=WARN
-log4j.logger.com.bigdata.concurrent.NonBlockingLockManager=WARN
log4j.logger.com.bigdata.concurrent.TestNonBlockingLockManager=INFO
log4j.logger.com.bigdata.concurrent.AbstractStressTestNonBlockingLockManager=INFO
#log4j.logger.com.bigdata.concurrent.LockManager=INFO
Modified: branches/maven_scaleout/bigdata-core/bigdata-perf/uniprot/src/resources/logging/log4j.properties
===================================================================
--- branches/maven_scaleout/bigdata-core/bigdata-perf/uniprot/src/resources/logging/log4j.properties 2010-10-01 16:48:09 UTC (rev 3715)
+++ branches/maven_scaleout/bigdata-core/bigdata-perf/uniprot/src/resources/logging/log4j.properties 2010-10-01 19:03:22 UTC (rev 3716)
@@ -67,7 +67,6 @@
#log4j.logger.com.bigdata.service.AbstractTransactionService=INFO
#log4j.logger.com.bigdata.journal.AbstractLocalTransactionManager=INFO
#log4j.logger.com.bigdata.concurrent.TxDag=WARN
-#log4j.logger.com.bigdata.concurrent.NonBlockingLockManager=WARN
#log4j.logger.com.bigdata.concurrent.TestNonBlockingLockManager=INFO
#log4j.logger.com.bigdata.concurrent.AbstractStressTestNonBlockingLockManager=INFO
#log4j.logger.com.bigdata.concurrent.LockManager=INFO
Modified: branches/maven_scaleout/bigdata-core/src/main/deploy/legacy/bigdata-resources/logging/log4j-perf-tests.properties
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/deploy/legacy/bigdata-resources/logging/log4j-perf-tests.properties 2010-10-01 16:48:09 UTC (rev 3715)
+++ branches/maven_scaleout/bigdata-core/src/main/deploy/legacy/bigdata-resources/logging/log4j-perf-tests.properties 2010-10-01 19:03:22 UTC (rev 3716)
@@ -52,7 +52,6 @@
#log4j.logger.com.bigdata.service.AbstractTransactionService=INFO
#log4j.logger.com.bigdata.journal.AbstractLocalTransactionManager=INFO
log4j.logger.com.bigdata.concurrent.TxDag=WARN
-log4j.logger.com.bigdata.concurrent.NonBlockingLockManager=WARN
log4j.logger.com.bigdata.concurrent.TestNonBlockingLockManager=INFO
log4j.logger.com.bigdata.concurrent.AbstractStressTestNonBlockingLockManager=INFO
#log4j.logger.com.bigdata.concurrent.LockManager=INFO
Modified: branches/maven_scaleout/bigdata-core/src/main/deploy/var/config/logging/log4j.properties
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/deploy/var/config/logging/log4j.properties 2010-10-01 16:48:09 UTC (rev 3715)
+++ branches/maven_scaleout/bigdata-core/src/main/deploy/var/config/logging/log4j.properties 2010-10-01 19:03:22 UTC (rev 3716)
@@ -56,7 +56,6 @@
#log4j.logger.com.bigdata.service.AbstractTransactionService=INFO
#log4j.logger.com.bigdata.journal.AbstractLocalTransactionManager=INFO
log4j.logger.com.bigdata.concurrent.TxDag=WARN
-log4j.logger.com.bigdata.concurrent.NonBlockingLockManager=WARN
log4j.logger.com.bigdata.concurrent.TestNonBlockingLockManager=INFO
log4j.logger.com.bigdata.concurrent.AbstractStressTestNonBlockingLockManager=INFO
#log4j.logger.com.bigdata.concurrent.LockManager=INFO
Deleted: branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/concurrent/NonBlockingLockManager.java
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/concurrent/NonBlockingLockManager.java 2010-10-01 16:48:09 UTC (rev 3715)
+++ branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/concurrent/NonBlockingLockManager.java 2010-10-01 19:03:22 UTC (rev 3716)
@@ -1,2472 +0,0 @@
-/**
-
- Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
-
- Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
- This program is free software; you can redistribute it and/or modify
- it under the terms of the GNU General Public License as published by
- the Free Software Foundation; version 2 of the License.
-
- This program is distributed in the hope that it will be useful,
- but WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- GNU General Public License for more details.
-
- You should have received a copy of the GNU General Public License
- along with this program; if not, write to the Free Software
- Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
- */
-/*
- * Created on Oct 3, 2007
- */
-
-package com.bigdata.concurrent;
-
-import static com.bigdata.concurrent.NonBlockingLockManager.RunState.Halted;
-import static com.bigdata.concurrent.NonBlockingLockManager.RunState.Running;
-import static com.bigdata.concurrent.NonBlockingLockManager.RunState.Shutdown;
-import static com.bigdata.concurrent.NonBlockingLockManager.RunState.ShutdownNow;
-import static com.bigdata.concurrent.NonBlockingLockManager.RunState.Starting;
-
-import java.lang.ref.WeakReference;
-import java.util.Arrays;
-import java.util.Iterator;
-import java.util.LinkedHashSet;
-import java.util.LinkedList;
-import java.util.Map;
-import java.util.concurrent.BlockingQueue;
-import java.util.concurrent.Callable;
-import java.util.concurrent.Executor;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.Future;
-import java.util.concurrent.FutureTask;
-import java.util.concurrent.LinkedBlockingQueue;
-import java.util.concurrent.RejectedExecutionException;
-import java.util.concurrent.SynchronousQueue;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.locks.Condition;
-import java.util.concurrent.locks.ReentrantLock;
-
-import org.apache.log4j.Logger;
-
-import com.bigdata.cache.ConcurrentWeakValueCacheWithTimeout;
-import com.bigdata.counters.CounterSet;
-import com.bigdata.counters.Instrument;
-import com.bigdata.journal.AbstractTask;
-import com.bigdata.util.concurrent.DaemonThreadFactory;
-import com.bigdata.util.concurrent.ThreadPoolExecutorStatisticsTask;
-import com.bigdata.util.concurrent.WriteTaskCounters;
-
-/**
- * This class coordinates a schedule among concurrent operations requiring
- * exclusive access to shared resources. Whenever possible, the result is a
- * concurrent schedule - that is, operations having non-overlapping lock
- * requirements run concurrently while operations that have lock contentions are
- * queued behind operations that currently have locks on the relevant resources.
- * A {@link ResourceQueue} is created for each resource and used to block
- * operations that are awaiting a lock. When locks are not being pre-declared, a
- * {@link TxDag WAITS_FOR} graph is additionally used to detect deadlocks.
- * <p>
- * This implementation uses a single {@link AcceptTask} thread to accept tasks,
- * update the requests in the {@link ResourceQueue}s, and in general perform
- * housekeeping for the internal state. Tasks submitted to this class ARE NOT
- * bound to a worker thread until they are executed by the delegate
- * {@link Executor}.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
- *
- * @param R
- * The type of the object that identifies a resource for the purposes
- * of the locking system. This is typically the name of an index.
- *
- * @todo a fair option? What constraints or freedoms would it provide?
- *
- * @todo a {@link SynchronousQueue} for the {@link #acceptedTasks}?
- *
- * @todo a {@link SynchronousQueue} for the writeService workQueue?
- *
- * FIXME In order to support 2PL we need to decouple the {@link LockFutureTask}
- * from the transaction with which it is associated. Otherwise each
- * {@link #submit(Comparable[], Callable)} will look like a new transaction (2PL
- * is impossible unless you can execute multiple tasks for the same
- * transaction).
- * <p>
- * Perhaps this would be easier if we did not delegate the task to an
- * {@link Executor} since that does not really support the 2PL pattern.
- *
- * @todo Support escalation of operation priority based on time and scheduling
- * of higher priority operations. the latter is done by queueing lock
- * requests in front of pending requests for each resource on which an
- * operation attempt to gain a lock. The former is just a dynamic
- * adjustment of the position of the operation in the resource queue where
- * it is awaiting a lock (an operation never awaits more than one lock at
- * a time). This facility could be used to give priority to distributed
- * transactions over local unisolated operations and to priviledge certain
- * operations that have low latency requirements. This is not quite a
- * "real-time" guarentee since the VM is not (normally) providing
- * real-time guarentees and since we are not otherwise attempting to
- * ensure anything except lower latency when compared to other operations
- * awaiting their own locks.
- */
-public class NonBlockingLockManager</* T, */R extends Comparable<R>> {
-
- final protected static Logger log = Logger
- .getLogger(NonBlockingLockManager.class);
-
- final protected static boolean INFO = log.isInfoEnabled();
- final protected static boolean DEBUG = log.isDebugEnabled();
-
- /**
- * Each resource that can be locked has an associated {@link ResourceQueue}.
- * <p>
- * Note: This is a concurrent collection since new resources may be added
- * while concurrent operations resolve resources to their queues. Stale
- * {@link ResourceQueue}s are purged after they become only weakly
- * reachable.
- *
- * @todo reconsider the timeout. it is set to one LBS period right now.
- */
- final private ConcurrentWeakValueCacheWithTimeout<R, ResourceQueue<LockFutureTask<? extends Object>>> resourceQueues = new ConcurrentWeakValueCacheWithTimeout<R, ResourceQueue<LockFutureTask<? extends Object>>>(
- 1000/* nresources */, TimeUnit.SECONDS.toNanos(60));
-
- /**
- * Release all locks held by the {@link LockFutureTask} currently holding a
- * lock on the specified resource.
- *
- * @param resource[]
- * The declared locks for the task.
- *
- * FIXME This is an integration hack for {@link AbstractTask}.
- * {@link AbstractTask} needs to be able to release the locks as soon as the
- * work of an unisolated task is done (when it is waiting for a group
- * commit) so that other tasks can gain access to the same indices and make
- * it into the same group commit. It is using this method to obtain the
- * {@link LockFutureTask} and then release its locks.
- * <p>
- * {@link AbstractTask} really needs a refactor.
- */
- public final void releaseLocksForTask(final R[] resource) {
-
- if (resource == null)
- throw new IllegalArgumentException();
-
- if(resource.length == 0) {
-
- // No declared locks.
- return;
-
- }
-
- lock.lock();
- try {
-
- LockFutureTask<? extends Object> task = null;
-
- for (R r : resource) {
-
- final ResourceQueue<LockFutureTask<? extends Object>> resourceQueue = resourceQueues
- .get(r);
-
- if (task == null) {
-
- /*
- * find the task by checking the resource queue for any of
- * its declared locks.
- */
- task = resourceQueue.queue.peek();
-
- if (task == null)
- throw new IllegalArgumentException(
- "Task does not hold declared lock: " + r);
-
- } else {
-
- /*
- * verify that the task holds the rest of its declared
- * locks.
- */
- if (task != resourceQueue.queue.peek()) {
-
- throw new IllegalArgumentException(
- "Task does not hold declared lock: " + r);
-
- }
-
- }
-
- }
-
- if(task == null) {
-
- throw new AssertionError();
-
- }
-
- /*
- * At this point we have verified that there is a task which holds
- * all of the locks specified by the caller.
- */
-
- /*
- * The task will be running since that is the presumption for the
- * context in which this method is invoked within AbstractTask.
- */
- final boolean waiting = false;
-
- // release the locks for the task.
- releaseLocks(task, waiting);
-
- } finally {
-
- lock.unlock();
-
- }
-
- }
-
- /**
- * True iff locks MUST be predeclared by the operation - this is a special
- * case of 2PL (two-phrase locking) that allows significant optimizations
- * and avoids the possibility of deadlock altogether.
- */
- final private boolean predeclareLocks;
-
- /**
- * When true, the resources in a lock request are sorted before the lock
- * requests are made to the various resource queues. This option is ONLY
- * turned off for testing purposes as it ALWAYS reduces the chance of
- * deadlocks and eliminates it entirely when locks are also predeclared.
- */
- final private boolean sortLockRequests;
-
- /**
- * Used to track dependencies among transactions.
- */
- final protected TxDag waitsFor;
-
- /**
- * Tasks holding their locks are submitted to this service for execution.
- */
- final protected Executor delegate;
-
- /**
- * Run states for the {@link NonBlockingLockManager}.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
- */
- static public enum RunState {
-
- /**
- * During startup. Tasks are NOT accepted.
- */
- Starting(0),
-
- /**
- * While running (aka open). Tasks are accepted and submitted for
- * execution once they hold their locks.
- */
- Running(1),
-
- /**
- * When shutting down normally. New tasks are not accepted but
- * {@link Future}s are still monitored for completion and waiting tasks
- * will eventually be granted their locks and execute on the delegate.
- */
- Shutdown(2),
-
- /**
- * When shutting down immediately. New tasks are not accepted, tasks
- * waiting for their locks are cancelled (they will not execute) and
- * {@link Future}s for running tasks are cancelled (they are
- * interrupted).
- */
- ShutdownNow(3),
-
- /**
- * When halted. New tasks are not accepted. No tasks are waiting. Any
- * {@link Future}s were cancelled.
- */
- Halted(4);
-
- private RunState(int val) {
-
- this.val = val;
-
- }
-
- final private int val;
-
- public int value() {
-
- return val;
-
- }
-
- public boolean isTransitionLegal(final RunState newval) {
-
- if (this == Starting) {
-
- if (newval == Running)
- return true;
-
- if (newval == Halted)
- return true;
-
- } else if (this == Running) {
-
- if (newval == Shutdown)
- return true;
-
- if (newval == ShutdownNow)
- return true;
-
- } else if (this == Shutdown) {
-
- if (newval == ShutdownNow)
- return true;
-
- if (newval == Halted)
- return true;
-
- } else if (this == ShutdownNow) {
-
- if (newval == Halted)
- return true;
-
- }
-
- return false;
-
- }
-
- }
-
- /*
- * counters
- */
-
- synchronized public CounterSet getCounters() {
-
- if (root == null) {
-
- root = new CounterSet();
-
- root.addCounter("naccepted", new Instrument<Long>() {
- public void sample() {
- setValue(counters.naccepted);
- }
- });
-
- root.addCounter("nrejected", new Instrument<Long>() {
- public void sample() {
- setValue(counters.nrejected);
- }
- });
-
- root.addCounter("nstarted", new Instrument<Long>() {
- public void sample() {
- setValue(counters.nstarted);
- }
- });
-
- root.addCounter("nended", new Instrument<Long>() {
- public void sample() {
- setValue(counters.nended);
- }
- });
-
- root.addCounter("ncancel", new Instrument<Long>() {
- public void sample() {
- setValue(counters.ncancel);
- }
- });
-
- root.addCounter("nerror", new Instrument<Long>() {
- public void sample() {
- setValue(counters.nerror);
- }
- });
-
- root.addCounter("ndeadlock", new Instrument<Long>() {
- public void sample() {
- setValue(counters.ndeadlock);
- }
- });
-
- root.addCounter("ntimeout", new Instrument<Long>() {
- public void sample() {
- setValue(counters.ntimeout);
- }
- });
-
- // Note: #that are waiting for their locks.
- root.addCounter("nwaiting", new Instrument<Integer>() {
- public void sample() {
- setValue(counters.nwaiting);
- }
- });
-
- // Note: #that have acquired locks are executing concurrently.
- root.addCounter("nrunning", new Instrument<Integer>() {
- public void sample() {
- setValue(counters.nrunning);
- }
- });
-
- // the maximum observed value for [nrunning].
- root.addCounter("maxRunning", new Instrument<Integer>() {
- public void sample() {
- setValue(counters.maxRunning);
- }
- });
-
- // #of resource queues
- root.addCounter("nresourceQueues", new Instrument<Integer>() {
- public void sample() {
- setValue(resourceQueues.size());
- }
- });
-
- root.addCounter("runState", new Instrument<String>() {
- public void sample() {
- setValue(runState.toString());
- }
- });
-
- /*
- * Displays the #of tasks waiting on each resource queue for the
- * lock (does not count the task at the head of the queue).
- *
- * @todo this could also be handled by dynamic reattachment of the
- * counters.
- */
- root.addCounter("queues", new Instrument<String>() {
- public void sample() {
- final Iterator<Map.Entry<R, WeakReference<ResourceQueue<LockFutureTask<? extends Object>>>>> itr = resourceQueues
- .entryIterator();
- final LinkedList<ResourceQueueSize> list = new LinkedList<ResourceQueueSize>();
- while (itr.hasNext()) {
- final Map.Entry<R, WeakReference<ResourceQueue<LockFutureTask<? extends Object>>>> entry = itr
- .next();
- final WeakReference<ResourceQueue<LockFutureTask<? extends Object>>> queueRef = entry
- .getValue();
- final ResourceQueue<LockFutureTask<? extends Object>> queue = queueRef
- .get();
- if (queue == null)
- continue;
- list.add(new ResourceQueueSize(queue));
- }
- final Object[] a = list.toArray();
- Arrays.sort(a);
- final StringBuilder sb = new StringBuilder();
- for (Object t : a) {
- sb.append(t.toString());
- sb.append(" ");
- }
- setValue(sb.toString());
- }
- });
-
- }
-
- return root;
-
- }
-
- private CounterSet root;
-
- /**
- * Helper class pairs up the resource with its sampled queue size and allows
- * ordering of the samples. E.g., by size or by resource.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
- */
- private class ResourceQueueSize implements Comparable<ResourceQueueSize> {
- final R resource;
- int size;
- public ResourceQueueSize(ResourceQueue<LockFutureTask<? extends Object>> queue){
- resource = queue.getResource();
- size = queue.getQueueSize();
- }
- public int compareTo(ResourceQueueSize arg0) {
- // resource name order.
-// return resource.compareTo(arg0.resource);
- // descending queue size order.
- return arg0.size - size;
- }
- public String toString() {
- return "(" + resource + "," + size + ")";
- }
- }
-
- /**
- * Counters for the {@link NonBlockingLockManager}.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
- */
- protected static class Counters {
-
- /**
- * The #of tasks that were accepted by the service (running total).
- */
- public long naccepted;
-
- /**
- * The #of tasks that were rejected by the service (running total).
- */
- public long nrejected;
-
- /**
- * The #of tasks that have been started on the delegate {@link Executor}
- * (running total).
- */
- public long nstarted;
-
- /**
- * The #of tasks that whose execution on the delegate {@link Executor}
- * is complete (either by normal completion or by error, but only for
- * tasks which were executed on the delegate) (running total).
- */
- public long nended;
-
- /**
- * The #of tasks that were cancelled (running total).
- */
- public long ncancel;
-
- /**
- * The #of tasks whose exception was set (running total).
- */
- public long nerror;
-
- /**
- * The #of tasks that deadlocked when they attempted to acquire their
- * locks (running total). Note that a task MAY retry lock acquisition
- * and this counter will be incremented each time it does so and then
- * deadlocks.
- */
- public long ndeadlock;
-
- /**
- * The #of tasks that timed out when they attempted to acquire their
- * locks (running total). Note that a task MAY retry lock acquisition
- * and this counter will be incremented each time it does so and then
- * times out.
- */
- public long ntimeout;
-
- /**
- * #of tasks that are currently waiting on locks. This is the effective
- * queue length of the {@link NonBlockingLockManager}. To get the
- * actual queue length you need to add this to the length of the queue
- * for the delegate {@link Executor}.
- *
- * FIXME This counter can be off since a task does not "know" when it is
- * placed onto the [waitingTasks] queue and therefore can not decrement
- * the counter conditionally if the task is cancelled or if an exception
- * is set. Right now the counter is only decremented if the task begins
- * to execute. This is a problem since this is the primary indication of
- * the total size of the {@link NonBlockingLockManager} as a queue
- * feeding its delegate {@link Executor}.
- */
- public int nwaiting;
-
- /**
- * #of tasks that have acquired their locks and are concurrently
- * executing. This is the true measure of concurrency.
- */
- public int nrunning;
-
- /**
- * The maximum observed value of {@link #nrunning}.
- */
- public int maxRunning;
-
- }
-
- /**
- * Counters for various things.
- */
- final Counters counters = new Counters();
-
- /**
- * Create a lock manager. No concurrency limit imposed when
- * <i>predeclareLocks</i> is <code>true</code> as deadlocks are
- * impossible and we do not maintain a WAITS_FOR graph.
- *
- * @param maxConcurrency
- * The maximum multi-programming level (ignored if
- * <i>predeclareLocks</i> is <code>true</code>).
- * @param predeclareLocks
- * When <code>true</code>, operations MUST declare all locks
- * before they begin to execute. This makes possible several
- * efficiencies and by sorting the resources in each lock request
- * into a common order we are able to avoid deadlocks entirely.
- * @param delegate
- * The service on which the tasks will be executed.
- * <p>
- * Note: The <i>delegate</i> MUST NOT use a bounded queue or
- * cause tasks to be run in the caller's thread. The use of a
- * {@link SynchronousQueue} or an unbounded
- * {@link LinkedBlockingQueue} for the <i>delegate</i>'s
- * workQueue are both acceptable.
- * <p>
- * Note: If {@link Executor#execute(Runnable)} blocks for the
- * <i>delegate</i> then the {@link AcceptTask} will also block
- * and this class will be non-responsive until the <i>delegate</i>
- * has accepted each [waitingTask] for execution. Some
- * {@link Executor}s can cause the task to be run in the
- * caller's thread, which would be the {@link AcceptTask} itself
- * and which also has the effect of causing this class to be
- * non-responsive until the task is complete.
- */
- public NonBlockingLockManager(final int maxConcurrency,
- final boolean predeclareLocks, final Executor delegate) {
-
- this(maxConcurrency, predeclareLocks, true/* sortLockRequests */,
- delegate);
-
- }
-
- /**
- * Create a lock manager. No concurrency limit imposed when
- * <i>predeclareLocks</i> is <code>true</code> as deadlocks are
- * impossible and we do not maintain a WAITS_FOR graph.
- *
- * @param maxConcurrency
- * The maximum multi-programming level (ignored if
- * <i>predeclareLocks</i> is <code>true</code>).
- * @param predeclareLocks
- * When <code>true</code>, operations MUST declare all locks
- * before they begin to execute. This makes possible several
- * efficiencies and by sorting the resources in each lock request
- * into a common order we are able to avoid deadlocks entirely.
- * @param sortLockRequests
- * This option indicates whether or not the resources in a lock
- * request will be sorted before attempting to acquire the locks
- * for those resources. Normally <code>true</code> this option
- * MAY be disabled for testing purposes. It is an error to
- * disable this option if <i>predeclareLocks</i> is
- * <code>false</code>.
- * @param delegate
- * The service on which the tasks will be executed.
- * <p>
- * Note: The <i>delegate</i> MUST NOT use a bounded queue or
- * cause tasks to be run in the caller's thread. The use of a
- * {@link SynchronousQueue} or an unbounded
- * {@link LinkedBlockingQueue} for the <i>delegate</i>'s
- * workQueue are both acceptable.
- * <p>
- * Note: If {@link Executor#execute(Runnable)} blocks for the
- * <i>delegate</i> then the {@link AcceptTask} will also block
- * and this class will be non-responsive until the <i>delegate</i>
- * has accepted each [waitingTask] for execution. Some
- * {@link Executor}s can cause the task to be run in the
- * caller's thread, which would be the {@link AcceptTask} itself
- * and which also has the effect of causing this class to be
- * non-responsive until the task is complete.
- */
- NonBlockingLockManager(final int maxConcurrency,
- final boolean predeclareLocks, final boolean sortLockRequests,
- final Executor delegate) {
-
- if (maxConcurrency < 2 && !predeclareLocks) {
-
- throw new IllegalArgumentException(
- "maxConcurrency: must be 2+ unless you are predeclaring locks, not "
- + maxConcurrency);
-
- }
-
- if (predeclareLocks && !sortLockRequests) {
-
- /*
- * This is required since we do not maintain TxDag when locks are
- * predeclare and therefore can not detect deadlocks. Sorting with
- * predeclared locks avoids the possibility of deadlocks so we do
- * not need the TxDag (effectively, it means that all locks that can
- * be requested by an operation are sorted since they are
- * predeclared and acquired in one go).
- */
-
- throw new IllegalArgumentException(
- "Sorting of lock requests MUST be enabled when locks are being predeclared.");
-
- }
-
- if (delegate == null)
- throw new IllegalArgumentException();
-
- this.predeclareLocks = predeclareLocks;
-
- this.sortLockRequests = sortLockRequests;
-
- if (predeclareLocks) {
-
- /*
- * Note: waitsFor is NOT required if we will acquire all locks at
- * once for a given operation since we can simply sort the lock
- * requests for each operation into a common order, thereby making
- * deadlock impossible!
- *
- * Note: waitsFor is also NOT required if we are using only a single
- * threaded system.
- *
- * Note: if you allocate waitsFor here anyway then you can measure
- * the cost of deadlock detection. As far as I can tell it is
- * essentially zero when locks are predeclared.
- */
-
- waitsFor = null;
-
- // waitsFor = new TxDag(maxConcurrency);
-
- } else {
-
- /*
- * Construct the directed graph used to detect deadlock cycles.
- */
-
- waitsFor = new TxDag(maxConcurrency);
-
- }
-
- this.delegate = delegate;
-
- // start service.
- service.submit(new AcceptTask());
-
- // change the run state.
- lock.lock();
- try {
-
- setRunState(RunState.Running);
-
- } finally {
-
- lock.unlock();
-
- }
-
- }
-
- /**
- * {@link FutureTask} which executes once it holds its locks.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
- *
- * @param <T>
- * The generic type of the outcome for the {@link Future}.
- */
- protected class LockFutureTask<T> extends FutureTask<T> {
-
- private final R[] resource;
-
- private final long lockTimeout;
-
- private final int maxLockTries;
-
- /**
- * Incremented each time a deadlock is detected. We will not retry if
- * {@link #maxLockTries} is exceeded.
- */
- private int ntries = 0;
-
- /**
- * The timestamp in nanoseconds when this task was accepted. This is
- * used to decide whether the {@link #lockTimeout} has expired.
- */
- final private long acceptTime = System.nanoTime();
-
- /**
- * The set of {@link ResourceQueue}s for which this task owns a lock
- * (is a member of the granted group) (NOT THREAD SAFE).
- * <p>
- * Note: This collection is required in order for the
- * {@link ResourceQueue}s for which the task has asserted a lock
- * request to remain strongly reachable. Without such hard references
- * the {@link ResourceQueue}s would be asynchronously cleared from the
- * {@link NonBlockingLockManager#resourceQueues} collection by the
- * garbage collector.
- */
- private final LinkedHashSet<ResourceQueue<LockFutureTask<? extends Object>>> lockedResources = new LinkedHashSet<ResourceQueue<LockFutureTask<? extends Object>>>();
-
- /**
- * True if the {@link #lockTimeout} has expired when measured against
- * <i>now</i>.
- */
- protected boolean isTimeout() {
-
- return (System.nanoTime() - acceptTime) >= lockTimeout;
-
- }
-
- private final Object task;
-
- public String toString() {
-
- return super.toString() + //
- "{resources=" + Arrays.toString(resource) + //
- ", done=" + isDone() + //
- ", cancelled=" + isCancelled() + //
- ", ntries=" + ntries +
- "}";
-
- }
-
- public LockFutureTask(final R[] resource, final Callable<T> task,
- final long timeout, final int maxLockTries) {
-
- super(task);
-
- this.resource = resource;
-
- this.lockTimeout = timeout;
-
- this.maxLockTries = maxLockTries;
-
- this.task = task;
-
- }
-
- public LockFutureTask(final R[] resources, final Runnable task,
- final T val, final long timeout, final int maxLockTries) {
-
- super(task, val);
-
- this.resource = resources;
-
- this.lockTimeout = timeout;
-
- this.maxLockTries = maxLockTries;
-
- this.task = task;
-
- }
-
- /**
- * The resource(s) that are pre-declared by the task. {@link #call()}
- * will ensure that the task as a lock on these resources before it
- * invokes {@link #run()} to execution the task.
- */
- public R[] getResource() {
-
- return resource;
-
- }
-
- /**
- * The elapsed nanoseconds the task waited to acquire its locks.
- */
- public long getLockLatency() {
-
- return nanoTime_lockLatency;
-
- }
-
- private long nanoTime_lockLatency;
-
- /**
- * The maximum #of times that the task will attempt to acquire its locks
- * (positive integer).
- */
- public int getMaxLockTries() {
-
- return maxLockTries;
-
- }
-
- /**
- * The timeout (milliseconds) or ZERO (0L) for an infinite timeout.
- */
- public long getLockTimeout() {
-
- return lockTimeout;
-
- }
-
- /**
- * Extended signal {@link NonBlockingLockManager#stateChanged} when the
- * task completes, to track counters, and also exposed to the outer
- * class.
- */
- @Override
- protected void setException(final Throwable t) {
-
- super.setException(t);
-
- lock.lock();
- try {
- if (DEBUG)
- log.debug("Exception: " + this + ", cause=" + t, t);
- counters.nerror++;
- /*
- * Note: Not known to be running, hence assume waiting (this
- * method is sometimes called before the task begins to
- * execute).
- */
- final boolean waiting = true;
- releaseLocks(this,waiting);
- stateChanged.signal();
- } finally {
- lock.unlock();
- }
-
- }
-
- /**
- * Extended signal {@link NonBlockingLockManager#stateChanged} when the
- * task completes and to track counters.
- */
- @Override
- public boolean cancel(final boolean mayInterruptIfRunning) {
-
- final boolean ret = super.cancel(mayInterruptIfRunning);
-
- lock.lock();
- try {
- if (DEBUG)
- log.debug("Cancelled: " + this);
- counters.ncancel++;
- /*
- * Note: Not known to be running, hence assume waiting (this
- * method is sometimes called before the task begins to
- * execute).
- */
- final boolean waiting = true;
- releaseLocks(this, waiting);
- stateChanged.signal();
- } finally {
- lock.unlock();
- }
-
- return ret;
-
- }
-
- /**
- * Extended signal {@link NonBlockingLockManager#stateChanged} when the
- * task completes and to track counters.
- */
- @Override
- public void run() {
-
- /*
- * Increment by the amount of time that the task was waiting to
- * acquire its lock(s).
- *
- * Note: This is being measured from the time when the task was
- * accepted by submit() on the outer class and counts all time until
- * the task begins to execute with its locks held.
- */
- if (task instanceof AbstractTask
- && ((AbstractTask) task).getTaskCounters() instanceof WriteTaskCounters) {
-
- final long lockWaitingTime = System.nanoTime() - acceptTime;
-
- ((WriteTaskCounters) ((AbstractTask) task).getTaskCounters()).lockWaitingNanoTime
- .addAndGet(lockWaitingTime);
-
- }
-
- synchronized (counters) {
-
- counters.nstarted++;
-
- counters.nwaiting--;
-
- counters.nrunning++;
-
- if (counters.nrunning > counters.maxRunning) {
-
- counters.maxRunning = counters.nrunning;
-
- }
-
- }
-
- try {
-
- if(DEBUG)
- log.debug("Running: "+this);
-
- super.run();
-
- } finally {
-
- lock.lock();
- try {
- if(DEBUG)
- log.debug("Did run: "+this);
- synchronized (counters) {
- counters.nended++;
- counters.nrunning--;
- }
- /*
- * The task is KNOWN to not be waiting since it was running.
- */
- final boolean waiting = false;
- releaseLocks(this, waiting);
- stateChanged.signal();
- } finally {
- lock.unlock();
- }
-
- }
-
- }
-
- }
-
- /**
- * Add if absent and return a {@link ResourceQueue} for the named resource.
- *
- * @param resource
- * The resource.
- *
- * @return The {@link ResourceQueue}.
- */
- private ResourceQueue<LockFutureTask<? extends Object>> declareResource(final R resource) {
-
- // test 1st to avoid creating a new ResourceQueue if it already exists.
- ResourceQueue<LockFutureTask<?extends Object>> resourceQueue = resourceQueues
- .get(resource);
-
- // not found, so create a new ResourceQueue for that resource.
- resourceQueue = new ResourceQueue<LockFutureTask<?extends Object>>(resource);
-
- // put if absent.
- final ResourceQueue<LockFutureTask<?extends Object>> oldval = resourceQueues
- .putIfAbsent(resource, resourceQueue);
-
- if (oldval != null) {
-
- // concurrent insert, so use the winner's resource queue.
- return oldval;
-
- }
-
- // we were the winner, so return the our new resource queue.
- return resourceQueue;
-
- }
-
- /**
- * Submit a task for execution. The task will wait until it holds the
- * declared locks. It will then execute. This method is non-blocking. The
- * caller must use {@link FutureTask#get()} to await the outcome.
- *
- * @param resource
- * An array of resources whose locks are required to execute the
- * <i>task</i>.
- * @param task
- * The task to be executed.
- *
- * @throws IllegalArgumentException
- * if <i>resource</i> is <code>null</code> or if any element
- * of that array is <code>null</code>.
- * @throws IllegalArgumentException
- * if the <i>task</i> is <code>null</code>.
- *
- * @throws RejectedExecutionException
- * if the task can not be queued for execution (including if the
- * service is not running or if a blocking queue was used and
- * the queue is at capacity).
- *
- * @todo add variant for Runnable target.
- *
- * @todo get rid [lockTimeout] since you can do get(timeout) on the Future
- * and the task will be cancelled unless it is complete by the
- * timeout.
- *
- * @todo get rid of [maxLockTries] since you can specify a timeout and that
- * will determine how much effort will be put into attempting to work
- * around a deadlock?
- */
- public <T> Future<T> submit(final R[] resource, final Callable<T> task) {
-
- return submit(resource, task, TimeUnit.SECONDS,
- Long.MAX_VALUE/* timeout */, 1/* maxLockTries */);
-
- }
-
-// public <T> Future<T> submit(final LockCallable<R, T> task) {
-//
-// return submit(task.getResource(), task);
-//
-// }
-
- public <T> Future<T> submit(final R[] resource, final Callable<T> task,
- final TimeUnit unit, final long lockTimeout, final int maxLockTries) {
-
- if (resource == null)
- throw new IllegalArgumentException();
-
- for (R r : resource) {
-
- if (r == null)
- throw new IllegalArgumentException();
-
- }
-
- if (task == null)
- throw new IllegalArgumentException();
-
- if (maxLockTries <= 0)
- throw new IllegalArgumentException();
-
- /*
- * Note: We clone the resources to avoid side-effects on the caller if
- * the resources are sorted (below) and also to prevent the caller from
- * changing the declared locks after they submit the task.
- */
- final R[] a = resource.clone();
-
- if (sortLockRequests) {
-
- /*
- * Sort the resources in the lock request.
- *
- * Note: Sorting the resources reduces the chance of a deadlock and
- * excludes it entirely when predeclaration of locks is also used.
- *
- * Note: This will throw an exception if the "resource" does not
- * implement Comparable.
- */
-
- Arrays.sort(a);
-
- }
-
- lock.lock();
- try {
-
- switch (runState) {
-
- case Running: {
-
- final LockFutureTask<T> future = new LockFutureTask<T>(a, task,
- lockTimeout, maxLockTries);
-
- try {
-
- acceptedTasks.add(future);
-
- counters.naccepted++;
-
- } catch (IllegalStateException ex) {
-
- counters.nrejected++;
-
- throw new RejectedExecutionException(ex);
-
- }
-
- stateChanged.signal();
-
- return future;
-
- }
-
- default:
-
- counters.nrejected++;
-
- throw new RejectedExecutionException("runState=" + runState);
-
- }
-
- } finally {
-
- lock.unlock();
-
- }
-
- }
-
- /**
- * Used to run the {@link AcceptTask} and the {@link MonitorTask}.
- *
- * FIXME Monitor this service using a {@link ThreadPoolExecutorStatisticsTask} to convert
- * {@link Counters#nrunning} and {@link Counters#nwaiting} into moving
- * averages.
- */
- private final ExecutorService service = Executors
- .newSingleThreadExecutor(new DaemonThreadFactory(getClass()
- .getName()));
-
- /**
- * The service run state.
- */
- private volatile RunState runState = RunState.Starting;
-
- /**
- * Lock used to protect changes various state changes, including change to
- * the {@link #runState}.
- */
- private final ReentrantLock lock = new ReentrantLock();
-
- /**
- * Condition is signaled whenever the {@link AcceptTask} needs to wake up.
- */
- private final Condition stateChanged = lock.newCondition();
-
- /**
- * Tasks accepted but not yet waiting on their locks. Tasks are moved from
- * here to the {@link #waitingTasks} asynchronously in order to de-couple
- * the caller from {@link DeadlockException}s or periods when the system is
- * at the maximum multi-programming level and can not accept another lock
- * request.
- */
- final private BlockingQueue<LockFutureTask<? extends Object>> acceptedTasks = new LinkedBlockingQueue<LockFutureTask<? extends Object>>();
-
- /**
- * Tasks whose lock requests are in the appropriate {@link ResourceQueue}s
- * but which are not yet executing.
- */
- private final BlockingQueue<LockFutureTask<? extends Object>> waitingTasks = new LinkedBlockingQueue<LockFutureTask<? extends Object>>();
-
- /**
- * {@link Runnable} drains the {@link #acceptedTasks} queue and manages
- * state changes in the {@link ResourceQueue}s. Once a task is holding all
- * necessary locks, the task is submitted to the delegate {@link Executor}
- * for execution. This thread is also responsible for monitoring the
- * {@link Future}s and releasing locks for a {@link Future} it is complete.
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
- */
- private class AcceptTask implements Runnable {
-
- public void run() {
- while (true) {
- switch (runState) {
- case Starting: {
- awaitStateChange(Starting);
- continue;
- }
- case Running: {
- while (processAcceptedTasks() || processWaitingTasks()) {
- // do work
- }
- awaitStateChange(Running);
- continue;
- }
- case Shutdown: {
- while (processAcceptedTasks() || processWaitingTasks()) {
- /*
- * Do work.
- *
- * Note: will run anything already accepted. That is
- * intentional. Once the lock manager is shutdown it will no
- * longer accept tasks, but it will process those tasks
- * which it has already accepted.
- */
- }
- lock.lock();
- try {
- if (acceptedTasks.isEmpty() && waitingTasks.isEmpty()) {
- /*
- * There is no more work to be performed so we can
- * change the runState.
- */
- if(INFO)
- log.info("No more work.");
- if (runState.val < RunState.ShutdownNow.val) {
- setRunState(RunState.ShutdownNow);
- break;
- }
- }
- } finally {
- lock.unlock();
- }
- awaitStateChange(Shutdown);
- continue;
- }
- case ShutdownNow: {
- /*
- * Cancel all tasks, clearing all queues. Note that only
- * tasks which are on [runningTasks] need to be interrupted
- * as tasks on the other queues are NOT running.
- */
- if(INFO)
- log.info(runState);
- cancelTasks(acceptedTasks.iterator(), false/* mayInterruptIfRunning */);
- cancelTasks(waitingTasks.iterator(), false/* mayInterruptIfRunning */);
- lock.lock();
- try {
- if (runState.val < RunState.Halted.val) {
- setRunState(RunState.Halted);
- }
- } finally {
- lock.unlock();
- }
- // fall through.
- }
- case Halted: {
- if (INFO)
- log.info(runState);
- // Done.
- return;
- }
- default:
- throw new AssertionError();
- } // switch(runState)
- } // while(true)
- } // run
-
- /**
- * IFF there is no work that could be performed and we are in the
- * expected {@link RunState} then this blocks until someone signals
- * {@link NonBlockingLockManager#stateChanged}. That signal can come
- * either from submitting a new task, from a running task that
- * completes, from a task being cancelled, or from a task whose
- * exception was set.
- *
- * @see LockFutureTask
- * @see NonBlockingLockManager#submit(Comparable[], Callable, TimeUnit, long, int)
- */
- private void awaitStateChange(final RunState expected) {
- lock.lock();
- try {
- /*
- * While we hold the lock we verify that there really is no work
- * to be done and that we are in the expected run state. Then
- * and only then do we wait on [stateChanged].
- */
- if (runState != expected) {
- // In a different run state.
- return;
- }
- if (!acceptedTasks.isEmpty() || !waitingTasks.isEmpty()) {
- // Some work can be done.
- return;
- }
- if (INFO)
- log.info("Waiting...");
- stateChanged.await();
- if (INFO)
- log.info("Woke up...");
- } catch (InterruptedException ex) {
- // someone woke us up.
- } finally {
- lock.unlock();
- }
- }
-
- /**
- * Cancel all tasks and remove them from the queue.
- *
- * @param tasks
- * The tasks.
- */
- private void cancelTasks(
- final Iterator<LockFutureTask<? extends Object>> itr,
- final boolean mayInterruptIfRunning) {
-
- while (itr.hasNext()) {
-
- final LockFutureTask<? extends Object> t = itr.next();
-
- t.cancel(mayInterruptIfRunning);
-
- itr.remove();
-
- }
-
- }
-
- /**
- * Processes accepted tasks, adding lock requests for each in turn:
- * <ul>
- * <li> If requesting locks for a task would exceed the configured
- * multi-programming level then we do not issue the request and return
- * immediately. The task is left on the accepted queue and will be
- * retried later.</li>
- * <li>If the lock requests for a task would cause a deadlock and the
- * #of retries has been exceeded, then set the {@link DeadlockException}
- * on the {@link Future} and drop the task. Otherwise ntries was
- * incremented and we just ignore the task for now and will retry it
- * again later.</li>
- * <li>If the timeout for the lock requests has already expired, then
- * set set {@link TimeoutException} on the {@link Future} and drop the
- * task.</li>
- * </ul>
- *
- * @return true iff any tasks were moved to the waiting queue.
- */
- private boolean processAcceptedTasks() {
-
- int nmoved = 0;
-
- final Iterator<LockFutureTask<? extends Object>> itr = acceptedTasks
- .iterator();
-
- while (itr.hasNext()) {
-
- final LockFutureTask<? extends Object> t = itr.next();
-
- if (t.isCancelled()) {
-
- // already cancelled, e.g., by the caller.
- itr.remove();
-
- continue;
-
- }
-
- if (t.isTimeout()) {
-
- // set exception on the task.
- t.setException(new java.util.concurrent.TimeoutException());
-
- counters.ntimeout++;
-
- itr.remove();
-
- continue;
-
- }
-
- int nvertices = -1;
- lock.lock();
- try {
-
- // #of vertices before we request the locks.
- if (waitsFor != null)
- nvertices = waitsFor.size();
-
- if (waitsFor != null && waitsFor.isFull()) {
-
- /*
- * Note: When TxDag is used we MUST NOT add the lock
- * requests if it would exceed the configured
- * multi-programming capacity (an exception would be
- * thrown by TxDag). Therefore we stop processing
- * accepted tasks and will wait until a running task
- * completes so we can start another one.
- *
- * Note: when tasks != transactions we need to wait
- * until a transaction completes. This could require
- * multiple tasks to complete if they are tasks for the
- * same transaction.
- */...
[truncated message content] |
|
From: <sgo...@us...> - 2010-10-01 16:48:15
|
Revision: 3715
http://bigdata.svn.sourceforge.net/bigdata/?rev=3715&view=rev
Author: sgossard
Date: 2010-10-01 16:48:09 +0000 (Fri, 01 Oct 2010)
Log Message:
-----------
[maven_scaleout] : removing weird character in file that makes my IDE cranky.
Modified Paths:
--------------
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/rdf/rules/BackchainAccessPath.java
Modified: branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/rdf/rules/BackchainAccessPath.java
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/rdf/rules/BackchainAccessPath.java 2010-10-01 16:01:57 UTC (rev 3714)
+++ branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/rdf/rules/BackchainAccessPath.java 2010-10-01 16:48:09 UTC (rev 3715)
@@ -92,7 +92,7 @@
* itself, but I wanted to run it past you anyway. -bryan
* <p>
* The only way that could happen is if there were a property that was a
- * subproperty of owl:sameAs and that subproperty was used in the data. I\x92ve
+ * subproperty of owl:sameAs and that subproperty was used in the data. I've
* never seen anything like that, but it is technically possible. -mike
* <p>
* Ok. But still, it is not a problem since we are not using the backchainer
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-10-01 16:02:04
|
Revision: 3714
http://bigdata.svn.sourceforge.net/bigdata/?rev=3714&view=rev
Author: thompsonbry
Date: 2010-10-01 16:01:57 +0000 (Fri, 01 Oct 2010)
Log Message:
-----------
Added BOpBase#setProperty() and #setUnboundProperty().
Added an expander pattern for an empty access path and hooked it into the named graph decision tree.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPredicate.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPOPredicate.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl.java
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/EmptyAccessPathExpander.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java 2010-10-01 15:36:09 UTC (rev 3713)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java 2010-10-01 16:01:57 UTC (rev 3714)
@@ -412,7 +412,7 @@
* @param value
* The value.
*/
- protected void setProperty(final String name, final Object value) {
+ protected void _setProperty(final String name, final Object value) {
annotations.put(name,value);
@@ -429,11 +429,56 @@
* @param name
* The name.
*/
- protected void clearProperty(final String name) {
+ protected void _clearProperty(final String name) {
annotations.remove(name);
}
+
+ /**
+ * Unconditionally sets the property.
+ *
+ * @param name
+ * The name.
+ * @param value
+ * The value.
+ *
+ * @return A copy of this {@link BOp} on which the property has been set.
+ */
+ public BOpBase setProperty(final String name, final Object value) {
+
+ final BOpBase tmp = this.clone();
+
+ tmp._setProperty(name, value);
+
+ return tmp;
+
+ }
+
+ /**
+ * Conditionally sets the property.
+ *
+ * @param name
+ * The name.
+ * @param value
+ * The value.
+ *
+ * @return A copy of this {@link BOp} on which the property has been set.
+ *
+ * @throws IllegalStateException
+ * if the property is already set.
+ */
+ public BOpBase setUnboundProperty(final String name, final Object value) {
+
+ final BOpBase tmp = this.clone();
+
+ if (tmp.annotations.put(name, value) != null)
+ throw new IllegalStateException("Already set: name=" + name
+ + ", value=" + value);
+
+ return tmp;
+
+ }
public int getId() {
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPredicate.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPredicate.java 2010-10-01 15:36:09 UTC (rev 3713)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPredicate.java 2010-10-01 16:01:57 UTC (rev 3714)
@@ -44,7 +44,6 @@
import com.bigdata.relation.accesspath.AccessPath;
import com.bigdata.relation.accesspath.ElementFilter;
import com.bigdata.relation.accesspath.IAccessPath;
-import com.bigdata.relation.accesspath.IElementFilter;
import com.bigdata.relation.rule.IRule;
import com.bigdata.relation.rule.ISolutionExpander;
import com.bigdata.relation.rule.eval.IEvaluationPlan;
@@ -76,6 +75,9 @@
*
* FIXME Change this to be a scalar value. It is currently an array for
* backwards compatibility.
+ *
+ * @see https://sourceforge.net/apps/trac/bigdata/ticket/180 (Migrate
+ * the RDFS inference and truth maintenance logic to BOPs)
*/
String RELATION_NAME = "relationName";
@@ -155,9 +157,22 @@
String ACCESS_PATH_FILTER = "accessPathFilter";
/**
- * Expander pattern.
+ * Access path expander pattern. This allows you to wrap or replace the
+ * {@link IAccessPath}.
+ * <p>
+ * Note: You MUST be extremely careful when using this feature in
+ * scale-out. Access path expanders in scale-out are logically
+ * consistent with used with a {@link #REMOTE_ACCESS_PATH}, but remote
+ * access paths often lack the performance of a local access path.
+ * <p>
+ * In order for the expander to be consistent with a local access path
+ * it MUST NOT rewrite the predicate in such a manner as to read on data
+ * onto found on the shard onto which the predicate was mapped during
+ * query evaluation.
+ *
+ * @see ISolutionExpander
*/
- String EXPANDER = "expander";
+ String ACCESS_PATH_EXPANDER = "accessPathExpander";
/**
* The partition identifier -or- <code>-1</code> if the predicate does
@@ -335,7 +350,7 @@
*
* @return The {@link ISolutionExpander}.
*
- * @see Annotations#EXPANDER
+ * @see Annotations#ACCESS_PATH_EXPANDER
*
* @todo replace with {@link ISolutionExpander#getAccessPath(IAccessPath)},
* which is the only method declared by {@link ISolutionExpander}.
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java 2010-10-01 15:36:09 UTC (rev 3713)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java 2010-10-01 16:01:57 UTC (rev 3714)
@@ -148,7 +148,7 @@
new NV(Annotations.OPTIONAL,optional),//
new NV(Annotations.INDEX_LOCAL_FILTER,
ElementFilter.newInstance(constraint)),//
- new NV(Annotations.EXPANDER,expander),//
+ new NV(Annotations.ACCESS_PATH_EXPANDER,expander),//
new NV(Annotations.TIMESTAMP, timestamp)
}));
@@ -197,7 +197,7 @@
// throw new UnsupportedOperationException();
final Predicate<E> tmp = this.clone();
- tmp.setProperty(Annotations.RELATION_NAME, relationName);
+ tmp._setProperty(Annotations.RELATION_NAME, relationName);
return tmp;
@@ -260,7 +260,7 @@
@SuppressWarnings("unchecked")
final public ISolutionExpander<E> getSolutionExpander() {
- return (ISolutionExpander<E>) getProperty(Annotations.EXPANDER);
+ return (ISolutionExpander<E>) getProperty(Annotations.ACCESS_PATH_EXPANDER);
}
@@ -374,7 +374,7 @@
final Predicate<E> tmp = this.clone();
- tmp.setProperty(Annotations.KEY_ORDER, keyOrder);
+ tmp._setProperty(Annotations.KEY_ORDER, keyOrder);
return tmp;
@@ -391,7 +391,7 @@
final Predicate<E> tmp = this.clone();
- tmp.setProperty(Annotations.PARTITION_ID, partitionId);
+ tmp._setProperty(Annotations.PARTITION_ID, partitionId);
return tmp;
@@ -401,7 +401,7 @@
final Predicate<E> tmp = this.clone();
- tmp.setProperty(Annotations.BOP_ID, bopId);
+ tmp._setProperty(Annotations.BOP_ID, bopId);
return tmp;
@@ -411,7 +411,7 @@
final Predicate<E> tmp = this.clone();
- tmp.setProperty(Annotations.TIMESTAMP, timestamp);
+ tmp._setProperty(Annotations.TIMESTAMP, timestamp);
return tmp;
@@ -479,14 +479,14 @@
/*
* Set the filter.
*/
- setProperty(Annotations.INDEX_LOCAL_FILTER, filter);
+ _setProperty(Annotations.INDEX_LOCAL_FILTER, filter);
} else {
/*
* Wrap the filter.
*/
- setProperty(Annotations.INDEX_LOCAL_FILTER, new FilterBase() {
+ _setProperty(Annotations.INDEX_LOCAL_FILTER, new FilterBase() {
@Override
protected Iterator filterOnce(Iterator src, Object context) {
@@ -511,7 +511,7 @@
for(String name : names) {
- tmp.clearProperty(name);
+ tmp._clearProperty(name);
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java 2010-10-01 15:36:09 UTC (rev 3713)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java 2010-10-01 16:01:57 UTC (rev 3714)
@@ -79,6 +79,7 @@
import com.bigdata.relation.IRelation;
import com.bigdata.relation.accesspath.ElementFilter;
import com.bigdata.relation.accesspath.IElementFilter;
+import com.bigdata.relation.rule.EmptyAccessPathExpander;
import com.bigdata.relation.rule.IProgram;
import com.bigdata.relation.rule.IRule;
import com.bigdata.relation.rule.IStep;
@@ -107,7 +108,7 @@
* <p>
* Note: When enabled, the {@link NamedGraphSolutionExpander} and
* {@link DefaultGraphSolutionExpander} must be stripped from the
- * {@link IPredicate.Annotations#EXPANDER}. In the long term, we will simply
+ * {@link IPredicate.Annotations#ACCESS_PATH_EXPANDER}. In the long term, we will simply
* no longer generate them in {@link BigdataEvaluationStrategyImpl}.
* <p>
* Note: If you want to test just the named graph stuff, then the default
@@ -349,12 +350,11 @@
final IVariable<?> v = (IVariable<?>) arg;
/*
* We do a remove because we don't ever need to run these
- * constraints again during subsequent joins once they
- * have been run once at the initial appearance of the
- * variable.
+ * constraints again during subsequent joins once they have
+ * been run once at the initial appearance of the variable.
*
- * FIXME revisit this when we dynamically re-order running
- * joins
+ * @todo revisit this when we dynamically re-order running
+ * joins
*/
if (constraintsByVar.containsKey(v))
constraints.addAll(constraintsByVar.remove(v));
@@ -406,7 +406,7 @@
* the long term it will simply not be generated.)
*/
pred = pred
- .clearAnnotations(new String[] { IPredicate.Annotations.EXPANDER });
+ .clearAnnotations(new String[] { IPredicate.Annotations.ACCESS_PATH_EXPANDER });
switch (scope) {
case NAMED_CONTEXTS:
@@ -532,21 +532,16 @@
/*
* The data set is empty (no graphs). Return a join backed by an
* empty access path.
- *
- * Note: Since the join could be optional or part of an optional
- * join group, we can not just drop it. Instead we need to return a
- * join against an empty access path. Since the join could also
- * "select" for some subset of variables, it seems that we really
- * need to modify PipelineJoin to recognize an annotation indicating
- * an empty access path. It can then substitute the empty access
- * path when processing the source binding sets. There should be
- * unit tests for this.
- *
- * FIXME Return PipelineJoin with an EMPTY ACCESS PATH.
*/
-
- throw new UnsupportedOperationException();
-
+
+ // force an empty access path for this predicate.
+ pred = (Predicate) pred.setUnboundProperty(
+ IPredicate.Annotations.ACCESS_PATH_EXPANDER,
+ EmptyAccessPathExpander.INSTANCE);
+
+ return new PipelineJoin(new BOp[] { left, pred }, anns
+ .toArray(new NV[anns.size()]));
+
} else if (summary.nknown == 1) {
/*
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/EmptyAccessPathExpander.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/EmptyAccessPathExpander.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/EmptyAccessPathExpander.java 2010-10-01 16:01:57 UTC (rev 3714)
@@ -0,0 +1,67 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Oct 1, 2010
+ */
+
+package com.bigdata.relation.rule;
+
+import com.bigdata.relation.accesspath.EmptyAccessPath;
+import com.bigdata.relation.accesspath.IAccessPath;
+
+/**
+ * An "expander" which replaces the access path with an {@link EmptyAccessPath}.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class EmptyAccessPathExpander<E> implements ISolutionExpander<E> {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ public static transient final EmptyAccessPathExpander INSTANCE = new EmptyAccessPathExpander();
+
+ public IAccessPath<E> getAccessPath(IAccessPath<E> accessPath) {
+
+ return new EmptyAccessPath<E>(accessPath.getPredicate(), accessPath
+ .getKeyOrder());
+
+ }
+
+ public boolean runFirst() {
+
+ return false;
+
+ }
+
+ public boolean backchain() {
+
+ return false;
+
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/rule/EmptyAccessPathExpander.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPOPredicate.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPOPredicate.java 2010-10-01 15:36:09 UTC (rev 3713)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPOPredicate.java 2010-10-01 16:01:57 UTC (rev 3714)
@@ -189,7 +189,7 @@
// expander);
super(new IVariableOrConstant[] { s, p, o }, //
new NV(Annotations.RELATION_NAME, new String[]{relationName}), //
- new NV(Annotations.EXPANDER, expander));
+ new NV(Annotations.ACCESS_PATH_EXPANDER, expander));
}
@@ -216,7 +216,7 @@
super(new IVariableOrConstant[] { s, p, o }, //
new NV(Annotations.RELATION_NAME, new String[]{relationName}), //
new NV(Annotations.OPTIONAL, optional), //
- new NV(Annotations.EXPANDER, expander));
+ new NV(Annotations.ACCESS_PATH_EXPANDER, expander));
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl.java 2010-10-01 15:36:09 UTC (rev 3713)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl.java 2010-10-01 16:01:57 UTC (rev 3714)
@@ -1522,7 +1522,7 @@
// free text search expander or named graphs expander
if (expander != null)
- anns.add(new NV(IPredicate.Annotations.EXPANDER, expander));
+ anns.add(new NV(IPredicate.Annotations.ACCESS_PATH_EXPANDER, expander));
// timestamp
anns.add(new NV(IPredicate.Annotations.TIMESTAMP, database
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-10-01 15:36:16
|
Revision: 3713
http://bigdata.svn.sourceforge.net/bigdata/?rev=3713&view=rev
Author: thompsonbry
Date: 2010-10-01 15:36:09 +0000 (Fri, 01 Oct 2010)
Log Message:
-----------
Added some interfaces for reporting on those aspects of a B+Tree which feed into cost estimates for query plans.
Integrated the named graph decision tree logic with the B+Tree cost model in QueryEngine. I've only done this for standalone as this point. We will have to override this for scale-out to use an approximation of the mixtures of cost models for a BTree on the Journal and ~3 index segment files. That can be overridden in FederatedQueryEngine.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/BTreeCostModel.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/DiskCostModel.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/IndexSegmentCostModel.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/AbstractBTree.java
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/BTreeStatistics.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/BTreeUtilizationReport.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/IBTreeStatistics.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/IBTreeUtilizationReport.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java 2010-10-01 15:14:05 UTC (rev 3712)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java 2010-10-01 15:36:09 UTC (rev 3713)
@@ -453,15 +453,32 @@
sb.append(',');
sb.append(t.getClass().getSimpleName());
}
- sb.append(")[");
- final Integer id = (Integer) annotations.get(Annotations.BOP_ID);
- if (id != null)
- sb.append("Annotations.BOP_ID=" + id);
- sb.append("]");
+ sb.append(")");
+ annotationsToString(sb);
return sb.toString();
}
+ protected void annotationsToString(final StringBuilder sb) {
+ final Map<String,Object> annotations = annotations();
+ if (!annotations.isEmpty()) {
+ sb.append("[");
+ boolean first = true;
+ for (Map.Entry<String, Object> e : annotations.entrySet()) {
+ if (!first)
+ sb.append(", ");
+ if (e.getValue() != null && e.getValue().getClass().isArray()) {
+ sb.append(e.getKey() + "="
+ + Arrays.toString((Object[]) e.getValue()));
+ } else {
+ sb.append(e.getKey() + "=" + e.getValue());
+ }
+ first = false;
+ }
+ sb.append("]");
+ }
+ }
+
final public BOpEvaluationContext getEvaluationContext() {
return getProperty(Annotations.EVALUATION_CONTEXT,
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java 2010-10-01 15:14:05 UTC (rev 3712)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java 2010-10-01 15:36:09 UTC (rev 3713)
@@ -31,8 +31,6 @@
import java.util.Iterator;
import java.util.Map;
-import cern.colt.Arrays;
-
import com.bigdata.bop.AbstractAccessPathOp;
import com.bigdata.bop.ArrayBindingSet;
import com.bigdata.bop.BOp;
@@ -44,8 +42,6 @@
import com.bigdata.bop.IVariable;
import com.bigdata.bop.IVariableOrConstant;
import com.bigdata.bop.NV;
-import com.bigdata.rdf.internal.IV;
-import com.bigdata.rdf.spo.SPOPredicate;
import com.bigdata.relation.accesspath.ElementFilter;
import com.bigdata.relation.accesspath.IElementFilter;
import com.bigdata.relation.rule.ISolutionExpander;
@@ -553,28 +549,7 @@
}
sb.append(")");
-
- final Map<String,Object> annotations = annotations();
- if (!annotations.isEmpty()) {
- sb.append("[");
- boolean first = true;
- for (Map.Entry<String, Object> e : annotations.entrySet()) {
- if (!first)
- sb.append(", ");
- // @todo remove relation name hack when making relation name a scalar.
- if (Annotations.RELATION_NAME.equals(e.getKey())
- && e.getValue() != null
- && e.getValue().getClass().isArray()) {
- sb.append(e.getKey() + "="
- + Arrays.toString((String[]) e.getValue()));
- } else {
- sb.append(e.getKey() + "=" + e.getValue());
- }
- first = false;
- }
- sb.append("]");
- }
-
+ annotationsToString(sb);
return sb.toString();
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/BTreeCostModel.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/BTreeCostModel.java 2010-10-01 15:14:05 UTC (rev 3712)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/BTreeCostModel.java 2010-10-01 15:36:09 UTC (rev 3713)
@@ -26,9 +26,13 @@
*/
package com.bigdata.bop.cost;
+import java.text.NumberFormat;
+
import com.bigdata.btree.AbstractBTree;
import com.bigdata.btree.BTree;
-import com.bigdata.journal.IIndexManager;
+import com.bigdata.btree.BTreeUtilizationReport;
+import com.bigdata.btree.IBTreeStatistics;
+import com.bigdata.btree.IBTreeUtilizationReport;
import com.bigdata.journal.Journal;
/**
@@ -52,43 +56,71 @@
*/
public class BTreeCostModel {
+ private final DiskCostModel diskCostModel;
+
/**
+ *
+ * @param diskCostModel
+ * The cost model for the disk on which the {@link Journal}
+ * backing the {@link BTree} is located.
+ */
+ public BTreeCostModel(final DiskCostModel diskCostModel) {
+
+ if (diskCostModel == null)
+ throw new IllegalArgumentException();
+
+ this.diskCostModel = diskCostModel;
+
+ }
+
+ /**
+ * Compute the height of the B+Tree from its entry count and branching
+ * factor (this can also be used to find the minimum height at which there
+ * could exist a given range count).
+ */
+ static public int estimateHeight(final int entryCount,
+ final int branchingFactor) {
+
+ if (entryCount < branchingFactor)
+ return 0;
+
+ final double logm = Math.log(branchingFactor);
+
+ final double logn = Math.log(entryCount);
+
+ final double h = (logm / logn) - 1;
+
+ return (int) Math.ceil(h);
+
+ }
+
+ /**
* Return the estimated cost of a range scan of the index.
*
- * @param diskCostModel
- * The cost model for the disk.
* @param rangeCount
* The range count for the scan.
- * @param btree
- * The index.
+ * @param m
+ * The B+Tree branching factor.
+ * @param h
+ * The B+Tree height.
+ * @param leafUtilization
+ * The leaf utilization percentage [0:100].
*
* @return The estimated cost (milliseconds).
- *
- * @todo how to get the right view onto the BTree without locking? or raise
- * the cost model into the {@link IIndexManager}?
*/
- public double rangeScan(final DiskCostModel diskCostModel,
- final int rangeCount, final BTree btree) {
+ public double rangeScan(final long rangeCount, final int m, final int h,
+ final int leafUtilization) {
if (rangeCount == 0)
return 0d;
- // double height = (Math.log(branchingFactor) / Math.log(entryCount)) -
- // 1;
-
- final int m = btree.getBranchingFactor();
-
- final int entryCount = btree.getEntryCount();
-
- final int height = btree.getHeight();
-
// average seek time to a leaf.
- final double averageSeekTime = Math.max(0, (height - 1))
+ final double averageSeekTime = Math.max(0, (h - 1))
* diskCostModel.seekTime;
// the percentage of the leaves which are full.
// final double leafFillRate = .70d;
- final double leafFillRate = ((double) btree.getUtilization()[1]) / 100;
+ final double leafFillRate = ((double) leafUtilization) / 100;
/*
* The expected #of leaves to visit for that range scan.
@@ -96,7 +128,7 @@
* Note: There is an edge condition when the root leaf is empty
* (fillRate is zero).
*/
- final double expectedLeafCount = Math.ceil((rangeCount / m)
+ final double expectedLeafCount = Math.ceil((((double) rangeCount) / m)
* Math.min(1, (1 / leafFillRate)));
/*
@@ -110,4 +142,140 @@
}
+ /**
+ * Prints out some tables based on different disk cost models, branching
+ * factors, and range scans. To validate this, you can do a scatter plot of
+ * the rangeCount and cost columns and observe the log linear curve of the
+ * B+Tree.
+ *
+ * @param args
+ * ignored.
+ *
+ * @see <a
+ * href="src/resources/architectures/query-cost-model.xls">query-cost-model.xls</a>
+ */
+ public static void main(String[] args) {
+
+ final DiskCostModel[] diskCostModels = new DiskCostModel[] { DiskCostModel.DEFAULT };
+
+ final int[] branchingFactors = new int[] { 32, 64, 128, 256, 512, 1024 };
+
+ final int[] heights = new int[] {0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
+
+ final int[] rangeCounts = new int[] { 1, 10, 100, 1000, 2000, 5000, 10000, 2000, 50000, 100000 };
+
+ final int leafUtilization = 65; // average percent "full" per leaf.
+
+ System.out.println("seekTime\txferRate\tleafUtil\tm\theight\trangeCount\tcost(ms)");
+
+ final NumberFormat millisFormat = NumberFormat.getIntegerInstance();
+ millisFormat.setGroupingUsed(true);
+
+ final NumberFormat percentFormat = NumberFormat.getPercentInstance();
+ percentFormat.setMinimumFractionDigits(0);
+
+ final StringBuilder sb = new StringBuilder();
+
+ for (DiskCostModel diskCostModel : diskCostModels) {
+
+ final BTreeCostModel btreeCostModel = new BTreeCostModel(
+ diskCostModel);
+
+ for (int m : branchingFactors) {
+
+ for (int h : heights) {
+
+ for (int rangeCount : rangeCounts) {
+
+ final int estimatedHeight = estimateHeight(rangeCount,
+ m);
+
+ if (estimatedHeight > h) {
+ /*
+ * Skip range counts which are too large for the
+ * current B+Tree height.
+ */
+ break;
+ }
+
+ final double cost = btreeCostModel.rangeScan(
+ rangeCount, m, h, leafUtilization);
+
+ sb.setLength(0); // reset.
+ sb.append(millisFormat.format(diskCostModel.seekTime));
+ sb.append('\t');
+ sb.append(millisFormat
+ .format(diskCostModel.transferRate));
+ sb.append('\t');
+ sb.append(percentFormat.format(leafUtilization / 100d));
+ sb.append('\t');
+ sb.append(m);
+ sb.append('\t');
+ sb.append(h);
+ sb.append('\t');
+ sb.append(rangeCount);
+ sb.append('\t');
+ sb.append(millisFormat.format(cost));
+ System.out.println(sb);
+
+ }
+
+ }
+
+ }
+
+ }
+
+ }
+
+ private static class MockBTreeStatistics implements IBTreeStatistics {
+
+ private final int m;
+
+ private final int entryCount;
+
+ private final int height;
+
+ private final int leafCount;
+
+ private final int nodeCount;
+
+ private final IBTreeUtilizationReport utilReport;
+
+ public MockBTreeStatistics(final int m, final int entryCount,
+ final int height, final int leafCount, final int nodeCount) {
+ this.m = m;
+ this.entryCount = entryCount;
+ this.height = height;
+ this.leafCount = leafCount;
+ this.nodeCount = nodeCount;
+ this.utilReport = new BTreeUtilizationReport(this);
+ }
+
+ public int getBranchingFactor() {
+ return m;
+ }
+
+ public int getEntryCount() {
+ return entryCount;
+ }
+
+ public int getHeight() {
+ return height;
+ }
+
+ public int getLeafCount() {
+ return leafCount;
+ }
+
+ public int getNodeCount() {
+ return nodeCount;
+ }
+
+ public IBTreeUtilizationReport getUtilization() {
+ return utilReport;
+ }
+
+ }
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/DiskCostModel.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/DiskCostModel.java 2010-10-01 15:14:05 UTC (rev 3712)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/DiskCostModel.java 2010-10-01 15:36:09 UTC (rev 3713)
@@ -47,16 +47,24 @@
*/
final public double seekTime;
+ /**
+ * The average disk transfer rate (megabytes per second).
+ */
final public double transferRate;
/**
*
* @param seekTime
+ * The average disk seek time (milliseconds).
* @param transferRate
+ * The average disk transfer rate (megabytes per second).
*/
public DiskCostModel(double seekTime, double transferRate) {
+
this.seekTime = seekTime;
+
this.transferRate = transferRate;
+
}
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/IndexSegmentCostModel.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/IndexSegmentCostModel.java 2010-10-01 15:14:05 UTC (rev 3712)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/IndexSegmentCostModel.java 2010-10-01 15:36:09 UTC (rev 3713)
@@ -37,15 +37,27 @@
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
* @version $Id$
- *
- * @todo
*/
public class IndexSegmentCostModel {
+ private final DiskCostModel diskCostModel;
+
/**
*
* @param diskCostModel
* The disk cost model.
+ */
+ public IndexSegmentCostModel(final DiskCostModel diskCostModel) {
+
+ if (diskCostModel == null)
+ throw new IllegalArgumentException();
+
+ this.diskCostModel = diskCostModel;
+
+ }
+
+ /**
+ *
* @param rangeCount
* The range count for the index scan.
* @param branchingFactor
@@ -58,8 +70,7 @@
*
* @return The estimated time for the range scan (milliseconds).
*/
- public double rangeScan(final DiskCostModel diskCostModel,
- final int rangeCount, final int branchingFactor,
+ public double rangeScan(final int rangeCount, final int branchingFactor,
final int averageBytesPerLeaf, final int xferBufferSize) {
if (rangeCount == 0)
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java 2010-10-01 15:14:05 UTC (rev 3712)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java 2010-10-01 15:36:09 UTC (rev 3713)
@@ -42,12 +42,24 @@
import org.apache.log4j.Logger;
import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpContextBase;
+import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.PipelineOp;
-import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.ap.Predicate;
+import com.bigdata.bop.cost.BTreeCostModel;
+import com.bigdata.bop.cost.DiskCostModel;
+import com.bigdata.btree.AbstractBTree;
import com.bigdata.btree.BTree;
+import com.bigdata.btree.IBTreeStatistics;
import com.bigdata.btree.IndexSegment;
+import com.bigdata.btree.UnisolatedReadWriteIndex;
import com.bigdata.btree.view.FusedView;
import com.bigdata.journal.IIndexManager;
+import com.bigdata.journal.Journal;
+import com.bigdata.journal.NoSuchIndexException;
+import com.bigdata.journal.TimestampUtility;
+import com.bigdata.relation.IRelation;
+import com.bigdata.relation.accesspath.IAccessPath;
import com.bigdata.resources.IndexManager;
import com.bigdata.service.IBigdataFederation;
import com.bigdata.service.IDataService;
@@ -771,4 +783,78 @@
// NOP
}
+ /*
+ * Cost models.
+ */
+
+ /**
+ * The cost model associated with the disk on which the indices are stored.
+ * For a {@link Journal}, this is just the cost model of the backing disk.
+ * For the federation, this should be an average cost model.
+ *
+ * @todo This is not parameterized. A simple cost model is always assumed.
+ * The correct cost model is necessary in order to get the tradeoff
+ * point right for SCAN+FILTER versus SUBQUERY on SSD or RAID arrays
+ * with lots of spindles versus normal disk.
+ *
+ * @todo In a shared disk deployment, we might introduce one cost model for
+ * local SSD used to cache journals, one for local non-SSD disks used
+ * to cache index segments, and one for remote storage used to
+ * materialize historical journals and index segments for query.
+ *
+ * @todo In a federation, this should be reported out as metadata for the
+ * federation. Perhaps as a Jini attribute.
+ */
+ private static final DiskCostModel diskCostModel = DiskCostModel.DEFAULT;
+
+ /**
+ * Return an estimate of the cost of a scan on the predicate.
+ *
+ * @param pred
+ * The predicate.
+ *
+ * @return The estimated cost of a scan on that predicate.
+ *
+ * @todo This tunnels through to the {@link AbstractBTree} class and is thus
+ * specific to standalone and also may run into trouble once we
+ * support unisolated access paths for reads or mutation since it may
+ * encounter an {@link UnisolatedReadWriteIndex} instead of an
+ * {@link AbstractBTree}.
+ */
+ public <E> double estimateCost(final BOpContextBase context,
+ final Predicate<E> pred) {
+
+ final IRelation<E> r = context.getRelation(pred);
+
+ final IAccessPath<E> ap = context.getAccessPath(r, pred);
+
+ final long rangeCount = ap.rangeCount(false/* exact */);
+
+ /*
+ * Drill through to the actual index object on the local index manager
+ * (standalone only). This avoids the UnisolatedReadWriteIndex class
+ * used to protect the index from concurrent modifications.
+ */
+ final String name = pred.getOnlyRelationName() + "."
+ + pred.getKeyOrder();
+
+ final long timestamp = (Long) pred
+ .getRequiredProperty(BOp.Annotations.TIMESTAMP);
+
+ final AbstractBTree index = (AbstractBTree) getIndexManager().getIndex(
+ name, timestamp);
+
+ if (index == null)
+ throw new NoSuchIndexException(name + ", timestamp="
+ + TimestampUtility.toString(timestamp));
+
+ final IBTreeStatistics stats = index.getStatistics();
+
+ final double cost = new BTreeCostModel(diskCostModel).rangeScan(
+ rangeCount, stats.getBranchingFactor(), stats.getHeight(),
+ stats.getUtilization().getLeafUtilization());
+
+ return cost;
+ }
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java 2010-10-01 15:14:05 UTC (rev 3712)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java 2010-10-01 15:36:09 UTC (rev 3713)
@@ -28,6 +28,7 @@
package com.bigdata.bop.engine;
import java.io.Serializable;
+import java.util.Arrays;
import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
@@ -47,17 +48,24 @@
import com.bigdata.bop.BOpEvaluationContext;
import com.bigdata.bop.BOpUtility;
import com.bigdata.bop.Constant;
+import com.bigdata.bop.HashBindingSet;
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.IConstant;
import com.bigdata.bop.IConstraint;
import com.bigdata.bop.IPredicate;
import com.bigdata.bop.IVariable;
+import com.bigdata.bop.IVariableOrConstant;
import com.bigdata.bop.NV;
import com.bigdata.bop.PipelineOp;
+import com.bigdata.bop.Var;
import com.bigdata.bop.ap.Predicate;
import com.bigdata.bop.bset.StartOp;
import com.bigdata.bop.join.PipelineJoin;
import com.bigdata.bop.rdf.join.DataSetJoin;
import com.bigdata.bop.solutions.SliceOp;
import com.bigdata.rdf.internal.IV;
+import com.bigdata.rdf.internal.TermId;
+import com.bigdata.rdf.internal.VTE;
import com.bigdata.rdf.lexicon.LexiconRelation;
import com.bigdata.rdf.model.BigdataURI;
import com.bigdata.rdf.sail.BigdataEvaluationStrategyImpl;
@@ -68,6 +76,7 @@
import com.bigdata.rdf.spo.NamedGraphSolutionExpander;
import com.bigdata.rdf.store.AbstractTripleStore;
import com.bigdata.rdf.store.IRawTripleStore;
+import com.bigdata.relation.IRelation;
import com.bigdata.relation.accesspath.ElementFilter;
import com.bigdata.relation.accesspath.IElementFilter;
import com.bigdata.relation.rule.IProgram;
@@ -76,6 +85,7 @@
import com.bigdata.relation.rule.eval.DefaultEvaluationPlan2;
import com.bigdata.relation.rule.eval.IRangeCountFactory;
import com.bigdata.relation.rule.eval.RuleState;
+import com.bigdata.striterator.IKeyOrder;
/**
* Utility class converts {@link IRule}s to {@link BOp}s.
@@ -168,16 +178,23 @@
* The estimated cost of a SCAN + FILTER approach to a default graph or
* named graph query.
*/
- String COST_SCAN = Rule2BOpUtility.class.getName() + ".costScan";
+ String COST_SCAN = Rule2BOpUtility.class.getName() + ".cost.scan";
/**
* The estimated cost of a SUBQUERY approach to a default graph or named
* graph query.
*/
String COST_SUBQUERY = Rule2BOpUtility.class.getName()
- + ".costSubquery";
+ + ".cost.subquery";
/**
+ * The #of samples used when estimating the cost of a SUBQUERY approach
+ * to a default graph or named graph query.
+ */
+ String COST_SUBQUERY_SAMPLE_COUNT = Rule2BOpUtility.class.getName()
+ + ".cost.subquerySampleCount";
+
+ /**
* The #of known graphs in the {@link Dataset} for a default graph or
* named graph query.
*/
@@ -254,7 +271,14 @@
// evaluation plan order.
final int[] order = plan.getOrder();
- // variables to be retained for each join.
+ // the #of variables in each tail of the rule.
+ final int[] nvars = new int[rule.getTailCount()];
+
+ // the index assigned to each tail of the rule.
+ final IKeyOrder[] keyOrder = computeKeyOrderForEachTail(rule, context,
+ order, nvars);
+
+ // the variables to be retained for each join.
final IVariable[][] selectVars = RuleState
.computeRequiredVarsForEachTail(rule, order);
@@ -304,6 +328,9 @@
// assign a bop id to the predicate
Predicate<?> pred = (Predicate<?>) rule.getTail(order[i]).setBOpId(
bopId++);
+
+ // decorate the predicate with the assigned index.
+ pred = pred.setKeyOrder(keyOrder[order[i]]);
/*
* Collect all the constraints for this predicate based on which
@@ -361,6 +388,9 @@
final boolean quads = pred.getProperty(Annotations.QUADS,
Annotations.DEFAULT_QUADS);
+ // pull of the Sesame dataset before we strip the annotations.
+ final Dataset dataset = (Dataset) pred.getProperty(Annotations.DATASET);
+
// strip off annotations that we do not want to propagate.
pred = pred.clearAnnotations(ANNS_TO_CLEAR_FROM_PREDICATE);
@@ -380,12 +410,12 @@
switch (scope) {
case NAMED_CONTEXTS:
- left = namedGraphJoin(queryEngine, left, anns, pred,
- cvar);
+ left = namedGraphJoin(queryEngine, context, left, anns,
+ pred, dataset, cvar);
break;
case DEFAULT_CONTEXTS:
- left = defaultGraphJoin(queryEngine, left, anns, pred,
- cvar);
+ left = defaultGraphJoin(queryEngine, context, left,
+ anns, pred, dataset, cvar);
break;
default:
throw new AssertionError();
@@ -468,11 +498,10 @@
* @return
*/
private static PipelineOp namedGraphJoin(final QueryEngine queryEngine,
- final PipelineOp left, final List<NV> anns, Predicate pred,
+ final BOpContextBase context, final PipelineOp left,
+ final List<NV> anns, Predicate pred, final Dataset dataset,
final org.openrdf.query.algebra.Var cvar) {
- final Dataset dataset = (Dataset) pred.getProperty(Annotations.DATASET);
-
final boolean scaleOut = queryEngine.isScaleOut();
if (scaleOut) {
anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
@@ -482,8 +511,7 @@
BOpEvaluationContext.ANY));
}
- final DataSetSummary summary = new DataSetSummary(dataset
- .getNamedGraphs());
+ final DataSetSummary summary = new DataSetSummary(dataset.getNamedGraphs());
anns.add(new NV(Annotations.NKNOWN, summary.nknown));
@@ -544,18 +572,41 @@
} else {
/*
- * Estimate cost of SCAN with C unbound)
+ * Estimate cost of SCAN with C unbound.
*/
- final double scanCost = getScanCost(pred);
+ final double scanCost = queryEngine.estimateCost(context, pred);
anns.add(new NV(Annotations.COST_SCAN, scanCost));
/*
- * Estimate cost of SUBQUERY with C bound.
+ * Estimate cost of SUBQUERY with C bound (sampling).
+ *
+ * @todo This should randomly sample in case there is bias in the
+ * order in which the URIs are presented here. However, the only
+ * thing which would be likely to create a strong bias is if someone
+ * sorted them on the IVs or if the URIs were in the same ordering
+ * in which their IVs were assigned AND the data were somehow
+ * correlated with that order. I rate the latter as pretty unlikely
+ * and the former is not true, so this sampling approach should be
+ * pretty good.
+ *
+ * @todo parameter for the #of samples to take.
*/
- final double subqueryCost = getSubqueryCost(pred);
+ double subqueryCost = 0d;
+ final int limit = 100;
+ int nsamples = 0;
+ for (URI uri : summary.graphs) {
+ if (nsamples == limit)
+ break;
+ final IV graph = ((BigdataURI) uri).getIV();
+ subqueryCost += queryEngine.estimateCost(context, pred.asBound(
+ (IVariable) pred.get(3), new Constant(graph)));
+ nsamples++;
+ }
+ subqueryCost /= nsamples;
anns.add(new NV(Annotations.COST_SUBQUERY, subqueryCost));
+ anns.add(new NV(Annotations.COST_SUBQUERY_SAMPLE_COUNT, nsamples));
if (scanCost < subqueryCost * summary.nknown) {
@@ -628,34 +679,6 @@
}
/**
- *
- * @param pred
- * @return
- *
- * FIXME Cost models have been implemented, but are not yet hooked in.
- */
- static double getScanCost(Predicate pred) {
- /*
- * @todo Scan is more expensive on the Journal so this is set to ONE (1)
- * and subquery is set to ZERO (0). This will get replaced by the actual
- * computed costs shortly.
- */
- return 1d;
- }
-
- /**
- *
- * @param pred
- * @return
- *
- * FIXME Cost models have been implemented, but are not yet hooked
- * in.
- */
- static double getSubqueryCost(Predicate pred) {
- return 0d;
- }
-
- /**
* Generate a default graph join (quads mode).
*
* @param queryEngine
@@ -669,7 +692,8 @@
* remote access paths with other remote access paths.
*/
private static PipelineOp defaultGraphJoin(final QueryEngine queryEngine,
- final PipelineOp left, final List<NV> anns, final Predicate pred,
+ final BOpContextBase context, final PipelineOp left,
+ final List<NV> anns, final Predicate pred, final Dataset dataset,
final org.openrdf.query.algebra.Var cvar) {
// @todo decision of local vs remote ap.
@@ -878,4 +902,106 @@
} // DataSetSummary
+ /**
+ * Return an array indicating the {@link IKeyOrder} that will be used when
+ * reading on each of the tail predicates. The array is formed using a
+ * private {@link IBindingSet} and propagating fake bindings to each
+ * predicate in turn using the given evaluation order.
+ *
+ * @param order
+ * The evaluation order.
+ * @param nvars
+ * The #of unbound variables for each tail predicate is assigned
+ * by side-effect.
+ *
+ * @return An array of the {@link IKeyOrder}s for each tail predicate. The
+ * array is correlated with the predicates index in the tail of the
+ * rule NOT its evaluation order.
+ */
+ @SuppressWarnings("unchecked")
+ static private IKeyOrder[] computeKeyOrderForEachTail(final IRule rule,
+ final BOpContextBase context, final int[] order, final int[] nvars) {
+
+ if (order == null)
+ throw new IllegalArgumentException();
+
+ if (order.length != rule.getTailCount())
+ throw new IllegalArgumentException();
+
+ final int tailCount = rule.getTailCount();
+
+ final IKeyOrder[] a = new IKeyOrder[tailCount];
+
+ final IBindingSet bindingSet = new HashBindingSet();
+
+ for (int orderIndex = 0; orderIndex < tailCount; orderIndex++) {
+
+ final int tailIndex = order[orderIndex];
+
+ final IPredicate pred = rule.getTail(tailIndex);
+
+ final IRelation rel = context.getRelation(pred);
+
+ final IPredicate asBound = pred.asBound(bindingSet);
+
+ final IKeyOrder keyOrder = context.getAccessPath(
+ rel, asBound).getKeyOrder();
+
+ if (log.isDebugEnabled())
+ log.debug("keyOrder=" + keyOrder + ", orderIndex=" + orderIndex
+ + ", tailIndex=" + orderIndex + ", pred=" + pred
+ + ", bindingSet=" + bindingSet + ", rule=" + rule);
+
+ // save results.
+ a[tailIndex] = keyOrder;
+ nvars[tailIndex] = keyOrder == null ? asBound.getVariableCount()
+ : asBound.getVariableCount((IKeyOrder) keyOrder);
+
+ final int arity = pred.arity();
+
+ for (int j = 0; j < arity; j++) {
+
+ final IVariableOrConstant<?> t = pred.get(j);
+
+ if (t.isVar()) {
+
+ final Var<?> var = (Var<?>) t;
+
+ if (log.isDebugEnabled()) {
+
+ log.debug("Propagating binding: pred=" + pred
+ + ", var=" + var + ", bindingSet="
+ + bindingSet);
+
+ }
+
+ bindingSet.set(var, fakeTermId);
+
+ }
+
+ }
+
+ }
+
+ if (log.isDebugEnabled()) {
+
+ log.debug("keyOrder[]=" + Arrays.toString(a) + ", nvars="
+ + Arrays.toString(nvars) + ", rule=" + rule);
+
+ }
+
+ return a;
+
+ }
+
+ /**
+ * A fake value that is propagated when we compute the {@link IKeyOrder} for
+ * a series of joins based on an assigned join evaluation order.
+ *
+ * @todo This has to be of the appropriate data type or we run into class
+ * cast exceptions.
+ */
+ final private static transient IConstant<IV> fakeTermId = new Constant<IV>(
+ new TermId(VTE.URI, -1L));
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java 2010-10-01 15:14:05 UTC (rev 3712)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java 2010-10-01 15:36:09 UTC (rev 3713)
@@ -215,11 +215,27 @@
this.resourceService = resourceService;
if(fed instanceof JiniFederation<?>) {
- // the proxy for this query engine when used as a query controller.
- this.clientProxy = (IQueryClient) ((JiniFederation<?>)fed).getProxy(this, false/*enableDGC*/);
+
+ /*
+ * The proxy for this query engine when used as a query controller.
+ *
+ *
+ * Should the data services expose their query engine in this
+ * manner?
+ *
+ * @todo We need to unexport the proxy as well when the service is
+ * shutdown. This should follow the same pattern as DataService ->
+ * DataServer. E.g., a QueryEngineServer class.
+ */
+
+ this.clientProxy = (IQueryClient) ((JiniFederation<?>) fed)
+ .getProxy(this, false/* enableDGC */);
+
} else {
- // E.g., an EmbeddedFederation in the test suite.
+
+ // E.g., an EmbeddedFederation in the test suite.
this.clientProxy = this;
+
}
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/AbstractBTree.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/AbstractBTree.java 2010-10-01 15:14:05 UTC (rev 3712)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/AbstractBTree.java 2010-10-01 15:36:09 UTC (rev 3713)
@@ -138,7 +138,7 @@
* @see KeyBuilder
*/
abstract public class AbstractBTree implements IIndex, IAutoboxBTree,
- ILinearList {
+ ILinearList, IBTreeStatistics {
/**
* The index is already closed.
@@ -787,8 +787,7 @@
// the % utilization in [0:1] for the whole tree (nodes + leaves).
counterSet.addCounter("% utilization", new Instrument<Double>(){
protected void sample() {
- final int[] tmp = getUtilization();
- setValue(tmp[2] / 100d);
+ setValue(getUtilization().getTotalUtilization() / 100d);
}
});
@@ -1617,10 +1616,11 @@
return true;
}
-
- /**
- * The branching factor for the btree.
+
+ /*
+ * IBTreeStatistics
*/
+
final public int getBranchingFactor() {
return branchingFactor;
@@ -1636,33 +1636,14 @@
*/
final int minChildren;
- /**
- * The height of the btree. The height is the #of levels minus one. A btree
- * with only a root leaf has <code>height := 0</code>. A btree with a
- * root node and one level of leaves under it has <code>height := 1</code>.
- * Note that all leaves of a btree are at the same height (this is what is
- * means for the btree to be "balanced"). Also note that the height only
- * changes when we split or join the root node (a btree maintains balance by
- * growing and shrinking in levels from the top rather than the leaves).
- */
abstract public int getHeight();
- /**
- * The #of non-leaf nodes in the {@link AbstractBTree}. This is zero (0)
- * for a new btree.
- */
abstract public int getNodeCount();
- /**
- * The #of leaf nodes in the {@link AbstractBTree}. This is one (1) for a
- * new btree.
- */
abstract public int getLeafCount();
/**
- * The #of entries (aka values) in the {@link AbstractBTree}. This is zero
- * (0) for a new B+Tree. Note that this value is tracked explicitly so it
- * requires no IOs.
+ * {@inheritDoc}
*
* @todo this could be re-defined as the exact entry count if we tracked the
* #of deleted index entries and subtracted that from the total #of
@@ -1677,6 +1658,21 @@
abstract public int getEntryCount();
/**
+ * Return a statistics snapshot of the B+Tree.
+ */
+ public IBTreeStatistics getStatistics() {
+
+ return new BTreeStatistics(this);
+
+ }
+
+ public IBTreeUtilizationReport getUtilization() {
+
+ return new BTreeUtilizationReport(this);
+
+ }
+
+ /**
* The object responsible for (de-)serializing the nodes and leaves of the
* {@link IIndex}.
*/
@@ -3197,47 +3193,6 @@
// abstract public ILeafCursor newLeafCursor(ILeafCursor leafCursor);
/**
- * Computes and returns the utilization of the tree. The utilization figures
- * do not factor in the space requirements of nodes and leaves.
- *
- * @return An array whose elements are:
- * <ul>
- * <li>0 - the leaf utilization percentage [0:100]. The leaf
- * utilization is computed as the #of values stored in the tree
- * divided by the #of values that could be stored in the #of
- * allocated leaves.</li>
- * <li>1 - the node utilization percentage [0:100]. The node
- * utilization is computed as the #of non-root nodes divided by the
- * #of non-root nodes that could be addressed by the tree.</li>
- * <li>2 - the total utilization percentage [0:100]. This is the
- * average of the leaf utilization and the node utilization.</li>
- * </ul>
- */
- public int[] getUtilization() {
-
- final int nnodes = getNodeCount();
-
- final int nleaves = getLeafCount();
-
- final int nentries = getEntryCount();
-
- final int numNonRootNodes = nnodes + nleaves - 1;
-
- final int branchingFactor = getBranchingFactor();
-
- final int nodeUtilization = nnodes == 0 ? 100 : (100 * numNonRootNodes)
- / (nnodes * branchingFactor);
-
- final int leafUtilization = (100 * nentries)
- / (nleaves * branchingFactor);
-
- final int utilization = (nodeUtilization + leafUtilization) / 2;
-
- return new int[] { nodeUtilization, leafUtilization, utilization };
-
- }
-
- /**
* Recursive dump of the tree.
*
* @param out
@@ -3256,7 +3211,7 @@
// True iff we will write out the node structure.
final boolean info = level.toInt() <= Level.INFO.toInt();
- final int[] utils = getUtilization();
+ final IBTreeUtilizationReport utils = getUtilization();
if (info) {
@@ -3273,8 +3228,9 @@
out.println("height=" + height + ", branchingFactor="
+ branchingFactor + ", #nodes=" + nnodes + ", #leaves="
+ nleaves + ", #entries=" + nentries + ", nodeUtil="
- + utils[0] + "%, leafUtil=" + utils[1] + "%, utilization="
- + utils[2] + "%");
+ + utils.getNodeUtilization() + "%, leafUtil="
+ + utils.getLeafUtilization() + "%, utilization="
+ + utils.getTotalUtilization() + "%");
}
if (root != null) {
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/BTreeStatistics.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/BTreeStatistics.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/BTreeStatistics.java 2010-10-01 15:36:09 UTC (rev 3713)
@@ -0,0 +1,90 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Oct 1, 2010
+ */
+
+package com.bigdata.btree;
+
+import java.io.Serializable;
+
+/**
+ * A snapshot of the B+Tree statistics.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class BTreeStatistics implements IBTreeStatistics, Serializable {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ private final int m;
+
+ private final int entryCount;
+
+ private final int height;
+
+ private final int leafCount;
+
+ private final int nodeCount;
+
+ private final IBTreeUtilizationReport utilReport;
+
+ public BTreeStatistics(final AbstractBTree btree) {
+ this.m = btree.getBranchingFactor();
+ this.entryCount = btree.getEntryCount();
+ this.height = btree.getHeight();
+ this.leafCount = btree.getLeafCount();
+ this.nodeCount = btree.getNodeCount();
+ this.utilReport = btree.getUtilization();
+ }
+
+ public int getBranchingFactor() {
+ return m;
+ }
+
+ public int getEntryCount() {
+ return entryCount;
+ }
+
+ public int getHeight() {
+ return height;
+ }
+
+ public int getLeafCount() {
+ return leafCount;
+ }
+
+ public int getNodeCount() {
+ return nodeCount;
+ }
+
+ public IBTreeUtilizationReport getUtilization() {
+ return utilReport;
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/BTreeStatistics.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/BTreeUtilizationReport.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/BTreeUtilizationReport.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/BTreeUtilizationReport.java 2010-10-01 15:36:09 UTC (rev 3713)
@@ -0,0 +1,85 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Oct 1, 2010
+ */
+
+package com.bigdata.btree;
+
+import java.io.Serializable;
+
+/**
+ * A btree utilization report.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class BTreeUtilizationReport implements IBTreeUtilizationReport,
+ Serializable {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ private final int leafUtilization;
+
+ private final int nodeUtilization;
+
+ private final int totalUtilization;
+
+ public BTreeUtilizationReport(final IBTreeStatistics stats) {
+
+ final int nnodes = stats.getNodeCount();
+
+ final int nleaves = stats.getLeafCount();
+
+ final int nentries = stats.getEntryCount();
+
+ final int numNonRootNodes = nnodes + nleaves - 1;
+
+ final int branchingFactor = stats.getBranchingFactor();
+
+ nodeUtilization = nnodes == 0 ? 100 : (100 * numNonRootNodes)
+ / (nnodes * branchingFactor);
+
+ leafUtilization = (100 * nentries) / (nleaves * branchingFactor);
+
+ totalUtilization = (nodeUtilization + leafUtilization) / 2;
+
+ }
+
+ public int getLeafUtilization() {
+ return leafUtilization;
+ }
+
+ public int getNodeUtilization() {
+ return nodeUtilization;
+ }
+
+ public int getTotalUtilization() {
+ return totalUtilization;
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/BTreeUtilizationReport.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/IBTreeStatistics.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/IBTreeStatistics.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/IBTreeStatistics.java 2010-10-01 15:36:09 UTC (rev 3713)
@@ -0,0 +1,83 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Oct 1, 2010
+ */
+
+package com.bigdata.btree;
+
+/**
+ * Interface used to report out some statistics about a B+Tree. These statistics
+ * may be used in combination with a disk cost model to predict the cost
+ * (latency) associated with a variety of operations on the B+Tree. All values
+ * reported by this interface are tracked explicitly by the
+ * {@link AbstractBTree} and do not require DISK IO.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public interface IBTreeStatistics {
+
+ /**
+ * The branching factor for the btree.
+ */
+ int getBranchingFactor();
+
+ /**
+ * The height of the btree. The height is the #of levels minus one. A btree
+ * with only a root leaf has <code>height := 0</code>. A btree with a
+ * root node and one level of leaves under it has <code>height := 1</code>.
+ * Note that all leaves of a btree are at the same height (this is what is
+ * means for the btree to be "balanced"). Also note that the height only
+ * changes when we split or join the root node (a btree maintains balance by
+ * growing and shrinking in levels from the top rather than the leaves).
+ */
+ int getHeight();
+
+ /**
+ * The #of non-leaf nodes in the {@link AbstractBTree}. This is zero (0)
+ * for a new btree.
+ */
+ int getNodeCount();
+
+ /**
+ * The #of leaf nodes in the {@link AbstractBTree}. This is one (1) for a
+ * new btree.
+ */
+ int getLeafCount();
+
+ /**
+ * The #of entries (aka tuples) in the {@link AbstractBTree}. This is zero
+ * (0) for a new B+Tree. When the B+Tree supports delete markers, this value
+ * also includes tuples which have been marked as deleted.
+ */
+ int getEntryCount();
+
+ /**
+ * Computes and returns the utilization of the tree. The utilization figures
+ * do not factor in the space requirements of nodes and leaves.
+ */
+ IBTreeUtilizationReport getUtilization();
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/IBTreeStatistics.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/IBTreeUtilizationReport.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/IBTreeUtilizationReport.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/IBTreeUtilizationReport.java 2010-10-01 15:36:09 UTC (rev 3713)
@@ -0,0 +1,58 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Oct 1, 2010
+ */
+
+package com.bigdata.btree;
+
+/**
+ * B+Tree utilization report.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public interface IBTreeUtilizationReport {
+
+ /**
+ * The leaf utilization percentage [0:100]. The leaf utilization is computed
+ * as the #of values stored in the tree divided by the #of values that could
+ * be stored in the #of allocated leaves.
+ */
+ int getLeafUtilization();
+
+ /**
+ * The node utilization percentage [0:100]. The node utilization is computed
+ * as the #of non-root nodes divided by the #of non-root nodes that could be
+ * addressed by the tree.
+ */
+ int getNodeUtilization();
+
+ /**
+ * The total utilization percentage [0:100]. This is the average of the leaf
+ * utilization and the node utilization.
+ */
+ int getTotalUtilization();
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/btree/IBTreeUtilizationReport.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <ble...@us...> - 2010-10-01 15:14:11
|
Revision: 3712
http://bigdata.svn.sourceforge.net/bigdata/?rev=3712&view=rev
Author: blevine218
Date: 2010-10-01 15:14:05 +0000 (Fri, 01 Oct 2010)
Log Message:
-----------
append "-clover" to the deploy dir (the directory into which the deployment tarball is extracted) when the bigdata-clover profile is enabled for code coverage.
Modified Paths:
--------------
branches/maven_scaleout/bigdata-integ/pom.xml
Modified: branches/maven_scaleout/bigdata-integ/pom.xml
===================================================================
--- branches/maven_scaleout/bigdata-integ/pom.xml 2010-10-01 15:01:16 UTC (rev 3711)
+++ branches/maven_scaleout/bigdata-integ/pom.xml 2010-10-01 15:14:05 UTC (rev 3712)
@@ -248,6 +248,7 @@
</activation>
<properties>
<integ.testOutputDirectory>${project.build.directory}/clover/test-classes</integ.testOutputDirectory>
+ <integ.deploy.dir>${integ.deploy.root.dir}/${integ.bigdata.dependency}-${project.version}-clover</integ.deploy.dir>
</properties>
<build>
<plugins>
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <sgo...@us...> - 2010-10-01 15:01:26
|
Revision: 3711
http://bigdata.svn.sourceforge.net/bigdata/?rev=3711&view=rev
Author: sgossard
Date: 2010-10-01 15:01:16 +0000 (Fri, 01 Oct 2010)
Log Message:
-----------
[maven_scaleout] : Broke all direct dependency cycles with package 'com.bigdata.mdi'. Wound up moving IMetadataIndex, LocalPartitionMetadata and JournalMetadata related classes to 'com.bigdata.btree'. This is likely a temporary move until the cycles between the btree and journal packages can be sorted out. These classes were Externalizable, so this change is not backwards compatible with existing installs. Also moved/renamed 'com.bigdata.service.Params' to 'com.bigdata.event.IEventParams'.
Modified Paths:
--------------
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/bfs/BigdataFileSystem.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/AbstractBTree.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/AbstractBTreeTupleCursor.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/BTree.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/IndexMetadata.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/IndexSegmentBuilder.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/IndexSegmentStore.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/KeyAfterPartitionException.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/KeyBeforePartitionException.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/ViewStatistics.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/proc/RangeCountProcedure.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/view/FusedView.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/journal/AbstractJournal.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/journal/Name2Addr.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/journal/RegisterIndexTask.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/mdi/AbstractResourceMetadata.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/mdi/IPartitionMetadata.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/mdi/IndexPartitionCause.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/mdi/PartitionLocator.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/mdi/SegmentMetadata.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/metadata/EmbeddedIndexStore.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/metadata/EmbeddedShardLocator.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/relation/accesspath/AbstractAccessPath.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/relation/rule/eval/IJoinNexus.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/resources/AsynchronousOverflowTask.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/resources/BTreeMetadata.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/resources/BuildResult.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/resources/CompactingMergeTask.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/resources/IncrementalBuildTask.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/resources/IndexManager.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/resources/JoinIndexPartitionTask.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/resources/JournalIndex.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/resources/MoveTask.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/resources/OverflowManager.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/resources/PurgeResult.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/resources/ResourceManager.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/resources/ScatterSplitTask.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/resources/SplitIndexPartitionTask.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/resources/SplitResult.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/resources/SplitUtility.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/resources/StoreManager.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/resources/ViewMetadata.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/AbstractScaleOutClient.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/AbstractScaleOutFederation.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/CacheOnceMetadataIndex.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/CachingMetadataIndex.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/EventResource.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/IBigdataFederation.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/IDataService.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/IMetadataService.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/IndexCache.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/MetadataIndexCache.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/MetadataService.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/NoCacheMetadataIndexView.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/ShardLocator.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/ShardManagement.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/jini/util/DumpFederation.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/ndx/AbstractScaleOutClientIndexView.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/ndx/AbstractScaleOutClientIndexView2.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/ndx/AbstractSplitter.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/ndx/ClientIndexView.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/ndx/ClientIndexViewRefactor.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/ndx/IScaleOutClientIndex.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/ndx/ISplitter.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/btree/TestIndexPartitionFencePosts.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/rdf/store/AbstractServerTestCase.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/resources/AbstractResourceManagerTestCase.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/resources/TestBuildTask.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/resources/TestBuildTask2.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/resources/TestMergeTask.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/resources/TestOverflow.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/resources/TestResourceManagerBootstrap.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/resources/TestSegSplitter.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/service/AbstractEmbeddedFederationTestCase.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/service/TestEmbeddedClient.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/service/TestEventReceiver.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/service/TestMetadataIndex.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/service/TestMove.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/service/TestOverflow.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/service/TestScatterSplit.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/service/TestSplitJoin.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/service/ndx/TestSplitter.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/service/ndx/pipeline/AbstractKeyRangeMasterTestCase.java
branches/maven_scaleout/bigdata-core/src/test/java/com/bigdata/service/ndx/pipeline/TestMasterTaskWithSplits.java
branches/maven_scaleout/bigdata-integ/src/test/java/com/bigdata/service/jini/AbstractServerTestCase.java
Added Paths:
-----------
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/IMetadataIndex.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/JournalMetadata.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/LocalPartitionMetadata.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/MetadataIndex.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/MetadataIndexView.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/event/
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/event/IEventParams.java
Removed Paths:
-------------
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/mdi/IMetadataIndex.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/mdi/JournalMetadata.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/mdi/LocalPartitionMetadata.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/mdi/MetadataIndex.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/mdi/MetadataIndexView.java
branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/service/Params.java
Modified: branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/bfs/BigdataFileSystem.java
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/bfs/BigdataFileSystem.java 2010-10-01 13:20:12 UTC (rev 3710)
+++ branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/bfs/BigdataFileSystem.java 2010-10-01 15:01:16 UTC (rev 3711)
@@ -34,7 +34,7 @@
import com.bigdata.journal.IResourceLock;
import com.bigdata.journal.ITx;
import com.bigdata.journal.Journal;
-import com.bigdata.mdi.MetadataIndex;
+import com.bigdata.btree.MetadataIndex;
import com.bigdata.rawstore.Bytes;
import com.bigdata.rawstore.IBlock;
import com.bigdata.rawstore.WormAddressManager;
Modified: branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/AbstractBTree.java
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/AbstractBTree.java 2010-10-01 13:20:12 UTC (rev 3710)
+++ branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/AbstractBTree.java 2010-10-01 15:01:16 UTC (rev 3711)
@@ -81,7 +81,6 @@
import com.bigdata.journal.IIndexManager;
import com.bigdata.journal.Journal;
import com.bigdata.mdi.IResourceMetadata;
-import com.bigdata.mdi.LocalPartitionMetadata;
import com.bigdata.rawstore.IRawStore;
import com.bigdata.resources.IndexManager;
import com.bigdata.resources.OverflowManager;
Modified: branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/AbstractBTreeTupleCursor.java
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/AbstractBTreeTupleCursor.java 2010-10-01 13:20:12 UTC (rev 3710)
+++ branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/AbstractBTreeTupleCursor.java 2010-10-01 15:01:16 UTC (rev 3711)
@@ -37,7 +37,6 @@
import com.bigdata.btree.isolation.IsolatedFusedView;
import com.bigdata.btree.view.FusedView;
import com.bigdata.io.DataOutputBuffer;
-import com.bigdata.mdi.LocalPartitionMetadata;
/**
* Class supporting random access to tuples and sequential tuple-based cursor
Modified: branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/BTree.java
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/BTree.java 2010-10-01 13:20:12 UTC (rev 3710)
+++ branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/BTree.java 2010-10-01 15:01:16 UTC (rev 3711)
@@ -41,8 +41,6 @@
import com.bigdata.journal.Name2Addr;
import com.bigdata.journal.Name2Addr.Entry;
import com.bigdata.mdi.IResourceMetadata;
-import com.bigdata.mdi.JournalMetadata;
-import com.bigdata.mdi.LocalPartitionMetadata;
import com.bigdata.rawstore.IRawStore;
/**
Copied: branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/IMetadataIndex.java (from rev 3704, branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/mdi/IMetadataIndex.java)
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/IMetadataIndex.java (rev 0)
+++ branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/IMetadataIndex.java 2010-10-01 15:01:16 UTC (rev 3711)
@@ -0,0 +1,95 @@
+/*
+
+Copyright (C) SYSTAP, LLC 2006-2008. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+*/
+/*
+ * Created on Feb 11, 2008
+ */
+
+package com.bigdata.btree;
+
+import com.bigdata.btree.MetadataIndex.MetadataIndexMetadata;
+import com.bigdata.mdi.PartitionLocator;
+import com.bigdata.service.IDataService;
+
+/**
+ * Interface for a metadata index. The metadata index stores the
+ * {@link com.bigdata.mdi.PartitionLocator}s that specify which {@link IDataService} has data
+ * for each index partition in a scale-out index.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public interface IMetadataIndex extends IRangeQuery { //extends IIndex {
+
+ /**
+ * The metadata for the metadata index itself. Note that the
+ * {@link MetadataIndexMetadata#getManagedIndexMetadata()} returns the
+ * template {@link IndexMetadata} for the scale-out index partitions.
+ *
+ * @see #getScaleOutIndexMetadata()
+ */
+ public MetadataIndexMetadata getIndexMetadata();
+
+ /**
+ * The metadata template for the scale-out index managed by this metadata
+ * index.
+ */
+ public IndexMetadata getScaleOutIndexMetadata();
+
+ /**
+ * The partition with that separator key or <code>null</code> (exact match
+ * on the separator key).
+ *
+ * @param key
+ * The separator key (the first key that would go into that
+ * partition).
+ *
+ * @return The partition with that separator key or <code>null</code>.
+ */
+ public PartitionLocator get(byte[] key);
+
+ /**
+ * Find and return the partition spanning the given key.
+ *
+ * @param key
+ * A key (optional). When <code>null</code> the locator for the
+ * last index partition will be returned.
+ *
+ * @return The partition spanning the given key or <code>null</code> if
+ * there are no partitions defined.
+ */
+ public PartitionLocator find(byte[] key);
+
+ /**
+ * Notification that a locator is stale. Caching implementations of this
+ * interface will use this notice to update their state from the
+ * authoritative metadata index. Non-caching and authoritative
+ * implementations just ignore this message.
+ *
+ * @param locator
+ * The locator.
+ */
+ public void staleLocator(PartitionLocator locator);
+
+}
Property changes on: branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/IMetadataIndex.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: svn:eol-style
+ native
Modified: branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/IndexMetadata.java
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/IndexMetadata.java 2010-10-01 13:20:12 UTC (rev 3710)
+++ branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/IndexMetadata.java 2010-10-01 15:01:16 UTC (rev 3711)
@@ -57,8 +57,6 @@
import com.bigdata.io.SerializerUtil;
import com.bigdata.io.compression.IRecordCompressorFactory;
import com.bigdata.journal.IIndexManager;
-import com.bigdata.mdi.LocalPartitionMetadata;
-import com.bigdata.mdi.MetadataIndex;
import com.bigdata.rawstore.IRawStore;
import com.bigdata.relation.accesspath.IAsynchronousIterator;
import com.bigdata.resources.OverflowManager;
Modified: branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/IndexSegmentBuilder.java
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/IndexSegmentBuilder.java 2010-10-01 13:20:12 UTC (rev 3710)
+++ branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/IndexSegmentBuilder.java 2010-10-01 15:01:16 UTC (rev 3711)
@@ -52,12 +52,10 @@
import com.bigdata.btree.raba.MutableKeyBuffer;
import com.bigdata.btree.raba.MutableValueBuffer;
import com.bigdata.btree.view.FusedView;
-import com.bigdata.cache.IGlobalLRU.ILRUCache;
import com.bigdata.journal.Journal;
import com.bigdata.journal.Name2Addr;
import com.bigdata.journal.TemporaryRawStore;
import com.bigdata.mdi.IResourceMetadata;
-import com.bigdata.mdi.LocalPartitionMetadata;
import com.bigdata.mdi.SegmentMetadata;
import com.bigdata.rawstore.Bytes;
import com.bigdata.rawstore.IAddressManager;
Modified: branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/IndexSegmentStore.java
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/IndexSegmentStore.java 2010-10-01 13:20:12 UTC (rev 3710)
+++ branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/IndexSegmentStore.java 2010-10-01 15:01:16 UTC (rev 3711)
@@ -51,7 +51,6 @@
import com.bigdata.journal.AbstractJournal;
import com.bigdata.journal.RootBlockException;
import com.bigdata.mdi.IResourceMetadata;
-import com.bigdata.mdi.LocalPartitionMetadata;
import com.bigdata.mdi.SegmentMetadata;
import com.bigdata.rawstore.AbstractRawStore;
import com.bigdata.resources.StoreManager;
Copied: branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/JournalMetadata.java (from rev 3704, branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/mdi/JournalMetadata.java)
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/JournalMetadata.java (rev 0)
+++ branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/JournalMetadata.java 2010-10-01 15:01:16 UTC (rev 3711)
@@ -0,0 +1,152 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+package com.bigdata.btree;
+
+import java.io.File;
+import java.util.UUID;
+
+import com.bigdata.journal.AbstractJournal;
+import com.bigdata.journal.IJournal;
+import com.bigdata.journal.Journal;
+import com.bigdata.mdi.AbstractResourceMetadata;
+
+/**
+ * Metadata required to locate a {@link Journal} resource.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class JournalMetadata extends AbstractResourceMetadata {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 3783897093328558238L;
+
+ public final boolean isIndexSegment() {
+
+ return false;
+
+ }
+
+ public final boolean isJournal() {
+
+ return true;
+
+ }
+
+ /**
+ * Return the file whose contents are the persistent state for the
+ * journal.
+ *
+ * @param journal
+ * The journal.
+ *
+ * @return The file.
+ */
+ private static String getFileString(final IJournal journal) {
+
+ final File file = journal.getFile();
+
+ if (file == null)
+ return "";
+
+ return file.getName();//toString();
+
+ }
+
+ /**
+ * De-serialization constructor.
+ */
+ public JournalMetadata() {
+
+ }
+
+ /**
+ * The description of a journal. The {@link JournalMetadata} state will not
+ * change as writes are made on the journal since it does not reflect
+ * anything exception the {@link UUID}, the filename, and the create time.
+ *
+ * @param journal
+ * The journal.
+ */
+ public JournalMetadata(final AbstractJournal journal) {
+
+ this(getFileString(journal), //journal.getBufferStrategy().getExtent(),
+ journal.getRootBlockView().getUUID(), //
+ journal.getRootBlockView().getCreateTime(), // createTime.
+ 0L // commitTime
+ );
+
+ }
+
+ /**
+ * Constructor variant used to indicate a read from a specific commitTime on
+ * a journal.
+ *
+ * @param journal
+ * The journal.
+ * @param commitTime
+ * The commitTime.
+ */
+ public JournalMetadata(final AbstractJournal journal, final long commitTime) {
+
+ this(getFileString(journal), //journal.getBufferStrategy().getExtent(),
+ journal.getRootBlockView().getUUID(),//
+ journal.getRootBlockView().getCreateTime(),//
+ commitTime
+ );
+
+ }
+
+ /**
+ * Data only constructor used by some unit tests.
+ *
+ * @param file
+ * @param uuid
+ * @param createTime
+ */
+ public JournalMetadata(final File file, /* long nbytes, */final UUID uuid,
+ final long createTime, final long commitTime) {
+
+ this(file.getName()/* ,nbytes */, uuid, createTime, commitTime);
+
+ }
+
+ /**
+ * Package private constructor used by the other constructors and when
+ * deserializing records.
+ *
+ * @param file
+ * @param uuid
+ * @param createTime
+ */
+ JournalMetadata(final String file, /* long nbytes, */final UUID uuid,
+ final long createTime, final long commitTime) {
+
+ super(file, /*nbytes, */ uuid, createTime, commitTime);
+
+ }
+
+}
Property changes on: branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/JournalMetadata.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: svn:eol-style
+ native
Modified: branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/KeyAfterPartitionException.java
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/KeyAfterPartitionException.java 2010-10-01 13:20:12 UTC (rev 3710)
+++ branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/KeyAfterPartitionException.java 2010-10-01 15:01:16 UTC (rev 3711)
@@ -31,7 +31,6 @@
import java.io.File;
import com.bigdata.io.BytesUtil;
-import com.bigdata.mdi.LocalPartitionMetadata;
/**
* An exception thrown when a key lies after the half-open range of an index
Modified: branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/KeyBeforePartitionException.java
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/KeyBeforePartitionException.java 2010-10-01 13:20:12 UTC (rev 3710)
+++ branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/KeyBeforePartitionException.java 2010-10-01 15:01:16 UTC (rev 3711)
@@ -31,7 +31,6 @@
import java.io.File;
import com.bigdata.io.BytesUtil;
-import com.bigdata.mdi.LocalPartitionMetadata;
/**
* Exception thrown when a key is before the start of the half-open range of an
Copied: branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/LocalPartitionMetadata.java (from rev 3704, branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/mdi/LocalPartitionMetadata.java)
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/LocalPartitionMetadata.java (rev 0)
+++ branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/LocalPartitionMetadata.java 2010-10-01 15:01:16 UTC (rev 3711)
@@ -0,0 +1,706 @@
+/*
+
+Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+*/
+package com.bigdata.btree;
+
+import java.io.Externalizable;
+import java.io.IOException;
+import java.io.ObjectInput;
+import java.io.ObjectOutput;
+import java.util.Arrays;
+import java.util.UUID;
+
+import com.bigdata.io.BytesUtil;
+import com.bigdata.mdi.*;
+import org.CognitiveWeb.extser.LongPacker;
+import org.CognitiveWeb.extser.ShortPacker;
+
+import com.bigdata.journal.Journal;
+import com.bigdata.service.DataService;
+
+/**
+ * An immutable object providing metadata about a local index partition,
+ * including the partition identifier, the left and right separator keys
+ * defining the half-open key range of the index partition, and optionally
+ * defining the {@link com.bigdata.mdi.IResourceMetadata}[] required to materialize a view of
+ * that index partition.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class LocalPartitionMetadata implements IPartitionMetadata,
+ Externalizable {
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = -1511361004851335936L;
+
+ /**
+ * The maximum length of the history string (4kb).
+ * <p>
+ * Note: The history is written each time the {@link IndexMetadata} is
+ * written and is read each time it is read so this can be the main driver
+ * of the size of the {@link IndexMetadata} record.
+ *
+ * @deprecated
+ */
+ protected final static transient int MAX_HISTORY_LENGTH = 0;//4 * Bytes.kilobyte32;
+
+ /**
+ * The unique partition identifier.
+ */
+ private int partitionId;
+
+ /**
+ *
+ * @see #getSourcePartitionId()
+ *
+ * @deprecated MoveTask manages without this field (it was required by the
+ * previous MOVE implementation).
+ */
+ private int sourcePartitionId;
+
+ /**
+ *
+ */
+ private byte[] leftSeparatorKey;
+ private byte[] rightSeparatorKey;
+
+ /**
+ * Description of the resources required to materialize a view of the index
+ * partition (optional - not stored when the partition metadata is stored on
+ * an {@link IndexSegmentStore}).
+ * <p>
+ * The entries in the array reflect the creation time of the resources. The
+ * earliest resource is listed first. The most recently created resource is
+ * listed last.
+ * <p>
+ * When present, the #of sources in the index partition view includes: the
+ * mutable {@link BTree}, any {@link BTree}s on historical journal(s)
+ * still incorporated into the view, and any {@link IndexSegment}s
+ * incorporated into the view.
+ */
+ private IResourceMetadata[] resources;
+
+ /**
+ * The reason why an index partition was created together with some metadata
+ * about when it was created.
+ */
+ private IndexPartitionCause cause;
+
+// /**
+// * A history of operations giving rise to the current partition metadata.
+// * E.g., register(timestamp), copyOnOverflow(timestamp), split(timestamp),
+// * join(partitionId,partitionId,timestamp), etc. This is truncated when
+// * serialized to keep it from growing without bound.
+// *
+// * @deprecated See {@link #getHistory()}
+// */
+// private String history;
+//
+// /**
+// * If the history string exceeds {@link #MAX_HISTORY_LENGTH} characters then
+// * truncates it to the last {@link #MAX_HISTORY_LENGTH}-3 characters,
+// * prepends "...", and returns the result. Otherwise returns the entire
+// * history string.
+// *
+// * @deprecated See {@link #history}
+// */
+// protected String getTruncatedHistory() {
+//
+// if (MAX_HISTORY_LENGTH == 0)
+// return "";
+//
+// String history = this.history;
+//
+// if(history.length() > MAX_HISTORY_LENGTH) {
+//
+// /*
+// * Truncate the history.
+// */
+//
+// final int len = history.length();
+//
+// final int fromIndex = len - (MAX_HISTORY_LENGTH - 3);
+//
+// assert fromIndex > 0 : "len=" + len + ", fromIndex=" + fromIndex
+// + ", maxHistoryLength=" + MAX_HISTORY_LENGTH;
+//
+// history = "..." + history.substring(fromIndex, len);
+//
+// }
+//
+// return history;
+//
+// }
+
+ /**
+ * De-serialization constructor.
+ */
+ public LocalPartitionMetadata() {
+
+ }
+
+ /**
+ *
+ * @param partitionId
+ * The unique partition identifier assigned by the
+ * {@link MetadataIndex}.
+ * @param sourcePartitionId
+ * <code>-1</code> unless this index partition is the target
+ * for a move, in which case this is the partition identifier of
+ * the source index partition.
+ * @param leftSeparatorKey
+ * The first key that can enter this index partition. The left
+ * separator key for the first index partition is always
+ * <code>new byte[]{}</code>. The left separator key MAY NOT
+ * be <code>null</code>.
+ * @param rightSeparatorKey
+ * The first key that is excluded from this index partition or
+ * <code>null</code> iff there is no upper bound.
+ * @param resources
+ * A description of each {@link Journal} or {@link IndexSegment}
+ * resource(s) required to compose a view of the index partition
+ * (optional).
+ * <p>
+ * The entries in the array reflect the creation time of the
+ * resources. The earliest resource is listed first. The most
+ * recently created resource is listed last.
+ * <p>
+ * Note: This is required if the {@link LocalPartitionMetadata}
+ * record will be saved on the {@link IndexMetadata} of a
+ * {@link BTree}. It is NOT recommended when it will be saved on
+ * the {@link IndexMetadata} of an {@link IndexSegment}. When
+ * the {@link IndexMetadata} is sent to a remote
+ * {@link DataService} this field MUST be <code>null</code> and
+ * the remote {@link DataService} will fill it in on arrival.
+ * @param cause
+ * The underlying cause for the creation of the index partition.
+ */
+// * @param history
+// * A human interpretable history of the index partition. The
+// * history is a series of whitespace delimited records each of
+// * more or less the form <code>foo(x,y,z)</code>. The history
+// * gets truncated when the {@link LocalPartitionMetadata} is
+// * serialized in order to prevent it from growing without bound.
+ public LocalPartitionMetadata(//
+ final int partitionId,//
+ final int sourcePartitionId,//
+ final byte[] leftSeparatorKey,//
+ final byte[] rightSeparatorKey,//
+ final IResourceMetadata[] resources,//
+ final IndexPartitionCause cause
+// final String history
+ ) {
+
+ /*
+ * Set fields first so that toString() can be used in thrown exceptions.
+ */
+
+ this.partitionId = partitionId;
+
+ this.sourcePartitionId = sourcePartitionId;
+
+ this.leftSeparatorKey = leftSeparatorKey;
+
+ this.rightSeparatorKey = rightSeparatorKey;
+
+ this.resources = resources;
+
+ this.cause = cause;
+
+// this.history = history;
+
+ /*
+ * Test arguments.
+ */
+
+ if (leftSeparatorKey == null)
+ throw new IllegalArgumentException("leftSeparatorKey");
+
+ // Note: rightSeparatorKey MAY be null.
+
+ if (rightSeparatorKey != null) {
+
+ final int cmp = BytesUtil.compareBytes(leftSeparatorKey,
+ rightSeparatorKey);
+
+ if (cmp >= 0) {
+
+ throw new IllegalArgumentException("Separator keys are "
+ + (cmp == 0 ? "equal" : "out of order") + " : " + this);
+
+ }
+
+ }
+
+ if (resources != null) {
+
+ if (resources.length == 0) {
+
+ throw new IllegalArgumentException("Empty resources array.");
+
+ }
+
+ for (IResourceMetadata t : resources) {
+
+ if (t == null)
+ throw new IllegalArgumentException(
+ "null value in resources[]");
+
+ }
+
+ /*
+ * This is the "live" journal.
+ *
+ * Note: The "live" journal is still available for writes. Index
+ * segments created off of this journal will therefore have a
+ * createTime that is greater than the firstCommitTime of this
+ * journal while being LTE to the lastCommitTime of the journal and
+ * strictly LT the commitTime of any other resource for this index
+ * partition.
+ */
+
+ if (!resources[0].isJournal()) {
+
+ throw new RuntimeException(
+ "Expecting a journal as the first resource: " + this);
+
+ }
+
+ /*
+ * Scan from 1 to n-1 - these are historical resources
+ * (non-writable). resources[0] is always the live journal. The
+ * other resources may be either historical journals or index
+ * segments.
+ *
+ * The order of the array is the order of the view. Reads on a
+ * FusedView will process the resources in the order in which they
+ * are specified by this array.
+ *
+ * The live journal gets listed first since it can continue to
+ * receive writes and therefore logically comes before any other
+ * resource in the ordering since any writes on the live index on
+ * the journal will be more recent than the data on the index
+ * segment.
+ *
+ * Normally, each successive entry in the resources[] will have an
+ * earlier createTime (smaller number) than the one that follows it.
+ * However, there is one exception. The createTime of the live
+ * journal MAY be less than the createTime of index segments created
+ * from that journal - this will be true if those indices are
+ * created from a historical view found on that journal and put into
+ * play while the journal is still the live journal. To work around
+ * this we start at the 2nd entry in the array.
+ */
+
+ /*
+ * Note: The practice of sending and index segment generated on one
+ * data service to another data service introduces another way in
+ * which the resource timestamp order can be broken. During the next
+ * synchronous overflow event you can see things like this:
+ *
+ * resourceMetadata=[
+ * JournalMetadata{filename=journal28417.jnl,uuid=add43d12-29b5-44e5-b26a-ae1b0694f67d,createTime=1236974533730},
+ * JournalMetadata{filename=journal28409.jnl,uuid=b954caf8-431b-42ae-9453-4c009398bec2,createTime=1236974293720},
+ * SegmentMetadata{filename=U8000_spo_OSP_part00050_28412.seg,uuid=cd954860-76fa-41ff-b788-e73a21b2c306,createTime=1236974525108},
+ * SegmentMetadata{filename=U8000_spo_OSP_part00050_28411.seg,uuid=35840589-6fb5-4691-b271-cf660186cd4b,createTime=1236974523976} ]
+ *
+ * This is in fact well-formed. However, because the index segments
+ * were generated on a different host, the create times get out of
+ * wack. For that reason, I have disabled checking here.
+ */
+ final boolean checkCreateTimes = false;
+
+ if (checkCreateTimes && resources.length > 2) {
+
+ long lastTimestamp = resources[1/*2ndEntry*/].getCreateTime();
+
+ for (int i = 2/* 3rd entry */; i < resources.length; i++) {
+
+ // createTime of the resource.
+ final long thisTimestamp = resources[i].getCreateTime();
+
+ if (lastTimestamp <= thisTimestamp) {
+
+ throw new RuntimeException(
+ "Resources out of timestamp order @ index="
+ + i + " : "+ this);
+
+ }
+
+ lastTimestamp = resources[i].getCreateTime();
+
+ }
+
+ }
+
+ }
+
+ }
+
+ final public int getPartitionId() {
+
+ return partitionId;
+
+ }
+
+ /**
+ * <code>-1</code> unless this index partition is the target for a move,
+ * in which case this is the partition identifier of the source index
+ * partition and the move operation has not been completed. This property is
+ * used to prevent the target data service from de-defining the index
+ * partition using a split, join or move operation while the MOVE operation
+ * is proceeding. The property is cleared to <code>-1</code> (which is an
+ * invalid index partition identifier) once the move has been completed
+ * successfully.
+ *
+ * @deprecated MoveTask manages without this field (it was required by the
+ * previous MOVE implementation).
+ */
+ final public int getSourcePartitionId() {
+
+ return sourcePartitionId;
+
+ }
+
+ final public byte[] getLeftSeparatorKey() {
+
+ return leftSeparatorKey;
+
+ }
+
+ final public byte[] getRightSeparatorKey() {
+
+ return rightSeparatorKey;
+
+ }
+
+ /**
+ * Description of the resources required to materialize a view of the index
+ * partition (optional, but required for a {@link BTree}).
+ * <p>
+ * The entries in the array reflect the creation time of the resources. The
+ * earliest resource is listed first. The most recently created resource is
+ * listed last. The order of the resources corresponds to the order in which
+ * a fused view of the index partition will be read. Reads begin with the
+ * most "recent" data for the index partition and stop as soon as there is a
+ * "hit" on one of the resources (including a hit on a deleted index entry).
+ * <p>
+ * When present, the #of sources in the index partition view includes: the
+ * mutable {@link BTree}, any {@link BTree}s on historical journal(s)
+ * still incorporated into the view, and any {@link IndexSegment}s
+ * incorporated into the view.
+ * <p>
+ * Note: the {@link IResourceMetadata}[] is only available when the
+ * {@link LocalPartitionMetadata} is attached to the {@link IndexMetadata}
+ * of a {@link BTree} and is NOT defined when the
+ * {@link LocalPartitionMetadata} is attached to an {@link IndexSegment}.
+ * The reason is that the index partition view is always described by the
+ * {@link BTree} and that view evolves as journals overflow. On the other
+ * hand, {@link IndexSegment}s are used as resources in index partition
+ * views but exist in a one to many relationship to those views.
+ */
+ final public IResourceMetadata[] getResources() {
+
+ return resources;
+
+ }
+
+ /**
+ * The reason why an index partition was created together with some metadata
+ * about when it was created.
+ */
+ final public IndexPartitionCause getIndexPartitionCause() {
+
+ return cause;
+
+ }
+
+// /**
+// * A history of the changes to the index partition.
+// *
+// * @deprecated I've essentially disabled the history (it is always empty
+// * when it is persisted). I found it nearly impossible to read.
+// * There are much saner ways to track what is going on in the
+// * federation. An analysis of the {@link Event} log is much more
+// * useful. If nothing else, you could examine the index
+// * partition in the metadata index by scanning the commit points
+// * and reading its state in each commit and reporting all state
+// * changes.
+// */
+// final public String getHistory() {
+//
+// return history;
+//
+// }
+
+ final public int hashCode() {
+
+ // per the interface contract.
+ return partitionId;
+
+ }
+
+ // Note: used by assertEquals in the test cases.
+ public boolean equals(final Object o) {
+
+ if (this == o)
+ return true;
+
+ final LocalPartitionMetadata o2 = (LocalPartitionMetadata) o;
+
+ if (partitionId != o2.partitionId)
+ return false;
+
+ if (!BytesUtil.bytesEqual(leftSeparatorKey, o2.leftSeparatorKey)) {
+
+ return false;
+
+ }
+
+ if (rightSeparatorKey == null) {
+
+ if (o2.rightSeparatorKey != null)
+ return false;
+
+ } else {
+
+ if (!BytesUtil.bytesEqual(rightSeparatorKey, o2.rightSeparatorKey)) {
+
+ return false;
+
+ }
+
+ }
+
+ if (resources.length != o2.resources.length)
+ return false;
+
+ for (int i = 0; i < resources.length; i++) {
+
+ if (!resources[i].equals(o2.resources[i]))
+ return false;
+
+ }
+
+ return true;
+
+ }
+
+ public String toString() {
+
+ return
+ "{ partitionId="+partitionId+
+ (sourcePartitionId!=-1?", sourcePartitionId="+sourcePartitionId:"")+
+ ", leftSeparator="+BytesUtil.toString(leftSeparatorKey)+
+ ", rightSeparator="+BytesUtil.toString(rightSeparatorKey)+
+ ", resourceMetadata="+Arrays.toString(resources)+
+ ", cause="+cause+
+// ", history="+history+
+ "}"
+ ;
+
+ }
+
+ /*
+ * Externalizable
+ */
+
+ private static final transient short VERSION0 = 0x0;
+
+ /**
+ * This version adds support for {@link IResourceMetadata#getCommitTime()},
+ * but that field is only serialized for a journal.
+ */
+ private static final transient short VERSION1 = 0x1;
+
+ /**
+ * This version serializes the {@link #partitionId} as 32-bits clean and
+ * gets rid of the <code>history</code> field.
+ */
+ private static final transient short VERSION2 = 0x2;
+
+ /**
+ * The current version.
+ */
+ private static final transient short VERSION = VERSION2;
+
+ public void readExternal(final ObjectInput in) throws IOException,
+ ClassNotFoundException {
+
+ final short version = ShortPacker.unpackShort(in);
+
+ switch (version) {
+ case VERSION0:
+ case VERSION1:
+ case VERSION2:
+ break;
+ default:
+ throw new IOException("Unknown version: " + version);
+ }
+
+ if (version < VERSION2) {
+ partitionId = (int) LongPacker.unpackLong(in);
+ } else {
+ partitionId = in.readInt();
+ }
+
+ sourcePartitionId = in.readInt(); // MAY be -1.
+
+ final int nresources = ShortPacker.unpackShort(in);
+
+ final int leftLen = (int) LongPacker.unpackLong(in);
+
+ final int rightLen = (int) LongPacker.unpackLong(in);
+
+ leftSeparatorKey = new byte[leftLen];
+
+ in.readFully(leftSeparatorKey);
+
+ if (rightLen != 0) {
+
+ rightSeparatorKey = new byte[rightLen];
+
+ in.readFully(rightSeparatorKey);
+
+ } else {
+
+ rightSeparatorKey = null;
+
+ }
+
+ cause = (IndexPartitionCause)in.readObject();
+
+ if (version < VERSION2) {
+ /* history = */in.readUTF();
+ }
+
+ resources = nresources>0 ? new IResourceMetadata[nresources] : null;
+
+ for (int j = 0; j < nresources; j++) {
+
+ final boolean isIndexSegment = in.readBoolean();
+
+ final String filename = in.readUTF();
+
+// long nbytes = LongPacker.unpackLong(in);
+
+ final UUID uuid = new UUID(in.readLong()/*MSB*/,in.readLong()/*LSB*/);
+
+ final long createTime = in.readLong();
+
+ long commitTime = 0L;
+ if (version >= VERSION1 && !isIndexSegment) {
+
+ commitTime = in.readLong();
+
+ }
+
+ resources[j] = (isIndexSegment //
+ ? new SegmentMetadata(filename, /*nbytes,*/ uuid, createTime) //
+ : new JournalMetadata(filename, /*nbytes,*/ uuid, createTime, commitTime) //
+ );
+
+ }
+
+ }
+
+ public void writeExternal(final ObjectOutput out) throws IOException {
+
+ ShortPacker.packShort(out, VERSION);
+
+ if (VERSION < VERSION2) {
+ LongPacker.packLong(out, partitionId);
+ } else {
+ out.writeInt(partitionId);
+ }
+
+ out.writeInt(sourcePartitionId); // MAY be -1.
+
+ final int nresources = (resources == null ? 0 : resources.length);
+
+ assert nresources < Short.MAX_VALUE;
+
+ ShortPacker.packShort(out, (short) nresources);
+
+ LongPacker.packLong(out, leftSeparatorKey.length);
+
+ LongPacker.packLong(out, rightSeparatorKey == null ? 0
+ : rightSeparatorKey.length);
+
+ out.write(leftSeparatorKey);
+
+ if (rightSeparatorKey != null) {
+
+ out.write(rightSeparatorKey);
+
+ }
+
+ out.writeObject(cause);
+
+ if (VERSION < VERSION2) {
+ out.writeUTF("");// getTruncatedHistory()
+ }
+
+ /*
+ * Note: we serialize using the IResourceMetadata interface so that we
+ * can handle different subclasses and then special case the
+ * deserialization based on the boolean flag. This is significantly more
+ * compact than using an Externalizable for each ResourceMetadata object
+ * since we do not have to write the class names for those objects.
+ */
+
+ for (int j = 0; j < nresources; j++) {
+
+ final IResourceMetadata rmd = resources[j];
+
+ final boolean isSegment = rmd.isIndexSegment();
+
+ out.writeBoolean(isSegment);
+
+ out.writeUTF(rmd.getFile());
+
+// LongPacker.packLong(out,rmd.size());
+
+ final UUID resourceUUID = rmd.getUUID();
+
+ out.writeLong(resourceUUID.getMostSignificantBits());
+
+ out.writeLong(resourceUUID.getLeastSignificantBits());
+
+ out.writeLong(rmd.getCreateTime());
+
+ if (!isSegment) {
+
+ out.writeLong(rmd.getCommitTime());
+
+ }
+
+ }
+
+ }
+
+}
Property changes on: branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/LocalPartitionMetadata.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: svn:eol-style
+ native
Copied: branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/MetadataIndex.java (from rev 3704, branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/mdi/MetadataIndex.java)
===================================================================
--- branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/MetadataIndex.java (rev 0)
+++ branches/maven_scaleout/bigdata-core/src/main/java/com/bigdata/btree/MetadataIndex.java 2010-10-01 15:01:16 UTC (rev 3711)
@@ -0,0 +1,513 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+package com.bigdata.btree;
+
+import java.io.Externalizable;
+import java.io.IOException;
+import java.io.ObjectInput;
+import java.io.ObjectOutput;
+import java.util.UUID;
+
+import com.bigdata.mdi.PartitionLocator;
+import org.CognitiveWeb.extser.LongPacker;
+
+import com.bigdata.btree.keys.IKeyBuilderFactory;
+import com.bigdata.btree.view.FusedView;
+import com.bigdata.journal.ICommitter;
+import com.bigdata.journal.IResourceManager;
+import com.bigdata.rawstore.IRawStore;
+import com.bigdata.service.MetadataService;
+
+/**
+ * A metadata index for the partitions of a distributed index. There is one
+ * metadata index for each distributed index. The keys of the metadata index are
+ * the first key that would be directed into the corresponding index segment,
+ * e.g., a <em>separator key</em> (this is just the standard btree semantics).
+ * The values are serialized {@link com.bigdata.mdi.PartitionLocator} objects.
+ * <p>
+ * Note: At this time the recommended scale-out approach for the metadata index
+ * is to place the metadata indices on a {@link MetadataService} (the same
+ * {@link MetadataService} may be used for an arbitrary #of scale-out indices)
+ * and to <em>replicate</em> the state for the {@link MetadataService} onto
+ * failover {@link MetadataService}s. Since the {@link MetadataIndex} may grow
+ * without bound, you simply need to have enough disk on hand for it (the size
+ * requirements are quite modest). Further, the {@link MetadataService} MUST NOT
+ * be used to hold the data for the scale-out indices themselves since the
+ * {@link MetadataIndex} can not undergo {@link IResourceManager#overflow()}.
+ * <p>
+ * One advantage of this approach is that the {@link MetadataIndex} is
+ * guaranteed to hold all historical states of the partition definitions for
+ * each index - effectively it is an immortal store for the partition metadata.
+ * On the other hand it is not possible to compact the metadata index without
+ * taking the database offline.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ *
+ * @todo The {@link MetadataIndex} does NOT support either overflow (it may NOT
+ * be a {@link FusedView}) NOR key-range splits. There are several issues
+ * involved:
+ * <p>
+ * (a) How to track the next partition identifier to be assigned to an
+ * index partition for the managed index. Currently this value is written
+ * in the {@link MetadataIndexCheckpoint} record and is propagated to the
+ * new backing store on overflow. However, if the metadata index is split
+ * into partitions then additional care MUST be taken to use only the
+ * value of that field on the 'meta-meta' index.
+ * <p>
+ * (b) how to locate the partitions of the metadata index itself.
+ *
+ * @todo one way to locate the metadata-index partitions is to hash partition
+ * the metadata index and range queries can be flooded to all partitions.
+ * the #of metadata service nodes can be changed by a suitable broadcast
+ * event in which clients have to change to the new hash basis. this
+ * feature can be generalized to provide hash partitioned indices as well
+ * as key-range partitioned indices.
+ *
+ * @todo A metadata index can be recovered by a distributed process running over
+ * the data services. Each data service reports all index partitions. The
+ * reports are collected and the index is rebuilt from the reports. Much
+ * like a map/reduce job.
+ */
+public class MetadataIndex extends BTree implements IMetadataIndex {
+
+ /**
+ * Used to implement find(byte[] key).
+ */
+ private transient final MetadataIndexView view;
+
+ public MetadataIndexMetadata getIndexMetadata() {
+
+ return (MetadataIndexMetadata) super.getIndexMetadata();
+
+ }
+
+ public IndexMetadata getScaleOutIndexMetadata() {
+
+ return getIndexMetadata().getManagedIndexMetadata();
+
+ }
+
+ /**
+ * Returns the value to be assigned to the next partition created on this
+ * {@link MetadataIndex} and then increments the counter. The counter will
+ * be made restart-safe iff the index is dirty, the index is registered as
+ * an {@link ICommitter}, and the store on which the index is stored is
+ * committed.
+ * <p>
+ * Note: The metadata index uses a 32-bit partition identifier rather than
+ * the {@link #getCounter()}. The reason is that the {@link Counter} uses
+ * the partition identifier in the high word and a partition local counter
+ * in the low word. Therefore we have to centralize the assignment of the
+ * partition identifier, even when the metadata index is itself split into
+ * partitions. Requests for partition identifiers need to be directed to the
+ * root partition (L0) for the {@link MetadataIndex}.
+ */
+ public int incrementAndGetNextPartitionId() {
+
+ final int tmp = nextPartitionId++;
+
+ /*
+ * Notify listener that the index is dirty.
+ */
+
+ fireDirtyEvent();
+
+ return tmp;
+
+ }
+
+ private int nextPartitionId;
+
+ /**
+ * Create a new {@link MetadataIndex}.
+ *
+ * @param store
+ * The backing store.
+ * @param indexUUID
+ * The unique identifier for the metadata index.
+ * @param managedIndexMetadata
+ * The metadata template for the managed scale-out index.
+ */
+ public static MetadataIndex create(IRawStore store, UUID indexUUID,
+ IndexMetadata managedIndexMetadata) {
+
+ final MetadataIndexMetadata metadata = new MetadataIndexMetadata(
+ managedIndexMetadata.getName(), indexUUID, managedIndexMetadata);
+
+ /*
+ * @todo the metadata index should use probably delete markers so that
+ * we can do compacting merges on it, but that is really moot until we
+ * support overflow of the metadataService and partitioning of the
+ * metadata index.
+ */
+
+ metadata.setDeleteMarkers(true);
+
+ /*
+ * Override the implementation class.
+ */
+
+ metadata.setBTreeClassName(MetadataIndex.class.getName());
+
+ /*
+ * Override the checkpoint record implementation class.
+ */
+ metadata.setCheckpointClassName(MetadataIndexCheckpoint.class.getName());
+
+ /*
+ * Tuple serializer.
+ */
+ metadata.setTupleSerializer(PartitionLocatorTupleSerializer.newInstance());
+
+ return (MetadataIndex) BTree.create(store, metadata);
+
+ }
+
+ /**
+ * Required ctor.
+ *
+ * @param store
+ * @param checkpoint
+ * @param metadata
+ */
+ public MetadataIndex(IRawStore store, Checkpoint checkpoint, IndexMetadata metadata, boolean readOnly) {
+
+ super(store, checkpoint, metadata, readOnly);
+
+ /*
+ * copy the initial value from the checkpoint record.
+ */
+
+ nextPartitionId = ((MetadataIndexCheckpoint)checkpoint).getNextPartitionId();
+
+ view = new MetadataIndexView(this);
+
+ }
+
+ /**
+ * Extended to require a checkpoint if {@link #incrementAndGetNextPartitionId()} has been
+ * invoked.
+ */
+ public boolean needsCheckpoint() {
+
+ if(nextPartitionId != ((MetadataIndexCheckpoint)getCheckpoint()).getNextPartitionId()) {
+
+ return true;
+
+ }
+
+ return super.needsCheckpoint();
+ ...
[truncated message content] |
|
From: <ble...@us...> - 2010-10-01 13:20:20
|
Revision: 3710
http://bigdata.svn.sourceforge.net/bigdata/?rev=3710&view=rev
Author: blevine218
Date: 2010-10-01 13:20:12 +0000 (Fri, 01 Oct 2010)
Log Message:
-----------
Make the test output directory variable dependent on whether we are running code coverage (Clover) or not.
Modified Paths:
--------------
branches/maven_scaleout/bigdata-integ/pom.xml
Modified: branches/maven_scaleout/bigdata-integ/pom.xml
===================================================================
--- branches/maven_scaleout/bigdata-integ/pom.xml 2010-10-01 11:54:44 UTC (rev 3709)
+++ branches/maven_scaleout/bigdata-integ/pom.xml 2010-10-01 13:20:12 UTC (rev 3710)
@@ -28,7 +28,8 @@
<integ.deploy.dir>${integ.deploy.root.dir}/${integ.bigdata.dependency}-${project.version}</integ.deploy.dir>
<integ.test.dir>${integ.deploy.dir}/testing</integ.test.dir>
- <integ.testScript>${project.build.testOutputDirectory}/services.xml</integ.testScript>
+ <integ.testOutputDirectory>${project.build.testOutputDirectory}</integ.testOutputDirectory>
+ <integ.testScript>${integ.testOutputDirectory}/services.xml</integ.testScript>
<integ.basedir>${integ.test.dir}</integ.basedir>
<integ.app.home>${integ.deploy.dir}</integ.app.home>
<integ.deploy.conf.dir>${integ.test.dir}/conf</integ.deploy.conf.dir>
@@ -245,6 +246,9 @@
<activation>
<activeByDefault>false</activeByDefault>
</activation>
+ <properties>
+ <integ.testOutputDirectory>${project.build.directory}/clover/test-classes</integ.testOutputDirectory>
+ </properties>
<build>
<plugins>
<plugin>
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <res...@us...> - 2010-10-01 11:54:50
|
Revision: 3709
http://bigdata.svn.sourceforge.net/bigdata/?rev=3709&view=rev
Author: resendes
Date: 2010-10-01 11:54:44 +0000 (Fri, 01 Oct 2010)
Log Message:
-----------
Private branch of maven_scaleout
Added Paths:
-----------
branches/bbb_cleanup/
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mar...@us...> - 2010-10-01 07:39:14
|
Revision: 3708
http://bigdata.svn.sourceforge.net/bigdata/?rev=3708&view=rev
Author: martyncutcher
Date: 2010-10-01 07:39:08 +0000 (Fri, 01 Oct 2010)
Log Message:
-----------
Fix to ensure correct deferred initialization
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Sorterator.java
Modified: branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Sorterator.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Sorterator.java 2010-09-30 23:38:12 UTC (rev 3707)
+++ branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Sorterator.java 2010-10-01 07:39:08 UTC (rev 3708)
@@ -1,77 +1,85 @@
-/*
-Striterator - transformation and mapping patterns over java Iterators
-
-Copyright (C) SYSTAP, LLC 2010. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
-
-package cutthecrap.utils.striterators;
-
-import java.util.*;
-
-/**
- * Initialized with a Sorter object, wraps a standard iterator and resolves each returned object
- */
-
-public class Sorterator implements Iterator {
-
- private final Iterator m_iter;
- protected final Object m_context;
-
- public Sorterator(Iterator iter, Object context, Sorter sorter) {
-
- this.m_context = context;
-
- // materialize the objects to be sorted.
- LinkedList tmp = new LinkedList();
-
- while(iter.hasNext()) {
- tmp.add(iter.next());
- }
-
- Object[] a = tmp.toArray();
-
- Arrays.sort(a, sorter/*.getComparator()*/);
-
- m_iter = Arrays.asList(a).iterator();
-
-// TreeSet set = new TreeSet(sorter);
-//
-// while (iter.hasNext()) {
-// set.add(iter.next());
-// }
-//
-// m_iter = set.iterator();
-
- }
-
- public boolean hasNext() {
- return m_iter.hasNext();
- }
-
- public Object next() {
- return m_iter.next();
- }
-
- public void remove() {
- m_iter.remove();
- }
+/*
+Striterator - transformation and mapping patterns over java Iterators
+
+Copyright (C) SYSTAP, LLC 2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
+
+package cutthecrap.utils.striterators;
+
+import java.util.*;
+
+/**
+ * Initialized with a Sorter object, wraps a standard iterator and resolves each
+ * returned object
+ */
+
+public class Sorterator implements Iterator {
+
+ private final Iterator m_src;
+ private Iterator m_iter;
+ private final Sorter m_sorter;
+ protected final Object m_context;
+ private boolean m_doneSort = false;
+
+ public Sorterator(Iterator iter, Object context, Sorter sorter) {
+
+ m_context = context;
+ m_src = iter;
+ m_sorter = sorter;
+
+ }
+
+ private void sort() {
+ if (!m_doneSort) {
+ // materialize the objects to be sorted.
+ LinkedList tmp = new LinkedList();
+
+ while (m_src.hasNext()) {
+ tmp.add(m_src.next());
+ }
+
+ Object[] a = tmp.toArray();
+
+ Arrays.sort(a, m_sorter/* .getComparator() */);
+
+ m_iter = Arrays.asList(a).iterator();
+ m_doneSort = true;
+ }
+ }
+
+ public boolean hasNext() {
+ sort();
+
+ return m_iter.hasNext();
+ }
+
+ public Object next() {
+ if (hasNext())
+ return m_iter.next();
+ else
+ throw new NoSuchElementException();
+ }
+
+ public void remove() {
+ m_iter.remove();
+ }
}
\ No newline at end of file
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-09-30 23:38:23
|
Revision: 3707
http://bigdata.svn.sourceforge.net/bigdata/?rev=3707&view=rev
Author: thompsonbry
Date: 2010-09-30 23:38:12 +0000 (Thu, 30 Sep 2010)
Log Message:
-----------
Rolled back a Sorterator change which broke a bunch of tests.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Sorterator.java
Modified: branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Sorterator.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Sorterator.java 2010-09-30 20:39:15 UTC (rev 3706)
+++ branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Sorterator.java 2010-09-30 23:38:12 UTC (rev 3707)
@@ -1,83 +1,77 @@
-/*
-Striterator - transformation and mapping patterns over java Iterators
-
-Copyright (C) SYSTAP, LLC 2010. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
- */
-
-package cutthecrap.utils.striterators;
-
-import java.util.*;
-
-/**
- * Initialized with a Sorter object, wraps a standard iterator and resolves each
- * returned object
- */
-
-public class Sorterator implements Iterator {
-
- private final Iterator m_src;
- private Iterator m_iter;
- private final Sorter m_sorter;
- protected final Object m_context;
- private boolean m_doneSort = false;
-
- public Sorterator(Iterator iter, Object context, Sorter sorter) {
-
- m_context = context;
- m_src = iter;
- m_sorter = sorter;
-
- }
-
- private void sort() {
- if (!m_doneSort) {
- // materialize the objects to be sorted.
- LinkedList tmp = new LinkedList();
-
- while (m_src.hasNext()) {
- tmp.add(m_src.next());
- }
-
- Object[] a = tmp.toArray();
-
- Arrays.sort(a, m_sorter/* .getComparator() */);
-
- m_iter = Arrays.asList(a).iterator();
- m_doneSort = true;
- }
- }
-
- public boolean hasNext() {
- return m_iter.hasNext();
- }
-
- public Object next() {
- if (hasNext())
- return m_iter.next();
- else
- throw new NoSuchElementException();
- }
-
- public void remove() {
- m_iter.remove();
- }
+/*
+Striterator - transformation and mapping patterns over java Iterators
+
+Copyright (C) SYSTAP, LLC 2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+
+package cutthecrap.utils.striterators;
+
+import java.util.*;
+
+/**
+ * Initialized with a Sorter object, wraps a standard iterator and resolves each returned object
+ */
+
+public class Sorterator implements Iterator {
+
+ private final Iterator m_iter;
+ protected final Object m_context;
+
+ public Sorterator(Iterator iter, Object context, Sorter sorter) {
+
+ this.m_context = context;
+
+ // materialize the objects to be sorted.
+ LinkedList tmp = new LinkedList();
+
+ while(iter.hasNext()) {
+ tmp.add(iter.next());
+ }
+
+ Object[] a = tmp.toArray();
+
+ Arrays.sort(a, sorter/*.getComparator()*/);
+
+ m_iter = Arrays.asList(a).iterator();
+
+// TreeSet set = new TreeSet(sorter);
+//
+// while (iter.hasNext()) {
+// set.add(iter.next());
+// }
+//
+// m_iter = set.iterator();
+
+ }
+
+ public boolean hasNext() {
+ return m_iter.hasNext();
+ }
+
+ public Object next() {
+ return m_iter.next();
+ }
+
+ public void remove() {
+ m_iter.remove();
+ }
}
\ No newline at end of file
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-09-30 20:39:21
|
Revision: 3706
http://bigdata.svn.sourceforge.net/bigdata/?rev=3706&view=rev
Author: thompsonbry
Date: 2010-09-30 20:39:15 +0000 (Thu, 30 Sep 2010)
Log Message:
-----------
Disabled the TestFederatedQueryEngine test suite for the moment. Something that I did when I setup the QueryEngineFactory is causing it to hang during the full com.bigdata.bop.TestAll() run.
There is one error which is related to not being the "controller" and is probably rooted in the RMI proxy object not be == to the actual object.
The hang is probably related to the tear down. if the QueryEngine is not shutdown the data service or journal on which it is running can not terminate.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestAll.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestAll.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestAll.java 2010-09-30 20:23:37 UTC (rev 3705)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/fed/TestAll.java 2010-09-30 20:39:15 UTC (rev 3706)
@@ -88,7 +88,8 @@
* bigdata-jini module since they must be executed against a full
* federation.
*/
- suite.addTestSuite(TestFederatedQueryEngine.class);
+// FIXME This is hanging for some reason. Fix it and reenable.
+// suite.addTestSuite(TestFederatedQueryEngine.class);
return suite;
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-09-30 20:23:45
|
Revision: 3705
http://bigdata.svn.sourceforge.net/bigdata/?rev=3705&view=rev
Author: thompsonbry
Date: 2010-09-30 20:23:37 +0000 (Thu, 30 Sep 2010)
Log Message:
-----------
This commit incorporates the named graph decision tree and the various cost models into Rule2BOpUtility. I will do the default graph query patterns tomorrow. The new named graph and default graph logic is not yet enabled when running CI. The cost models have been implemented based on the worksheets. However they are not currently connected to the query planner.
The only remaining features for the quads query branch are:
- PipelineJoin annotation to indicate an empty access path.
- UNION (this is basically a subquery operator).
- Advanced pattern for the ___C indices for default graph queries.
- ISimpleSplitHandler for the SPOC and SOPC shards.
CI is pretty close. The lack of a native UNION operator is responsible for most of the SPARQL query errors that remain. There are also a number of unit tests which have not yet been written for the bigdata operators, which accounts for most of the remaining errors.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/architecture/query-cost-model.xls
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPredicate.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/constraint/INBinarySearch.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/bop/rdf/join/DataSetJoin.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/InGraphBinarySearchFilter.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/InGraphHashSetFilter.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPOAccessPath.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPOPredicate.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl.java
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/BTreeCostModel.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/DiskCostModel.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/IndexSegmentCostModel.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/package.html
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/architecture/query-cost-model.xls
===================================================================
(Binary files differ)
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPredicate.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPredicate.java 2010-09-30 19:49:43 UTC (rev 3704)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/IPredicate.java 2010-09-30 20:23:37 UTC (rev 3705)
@@ -30,6 +30,7 @@
import java.io.Serializable;
+import com.bigdata.bop.ap.Predicate;
import com.bigdata.bop.ap.filter.BOpFilterBase;
import com.bigdata.bop.ap.filter.BOpTupleFilter;
import com.bigdata.bop.ap.filter.DistinctFilter;
@@ -469,9 +470,29 @@
public IConstant<?> get(E e, int index);
/**
- * A copy of this {@link IPredicate} in which zero or more variables have
- * been bound to constants using the given {@link IBindingSet}.
+ * Return a new instance in which all occurrences of the given variable have
+ * been replaced by the specified constant.
+ *
+ * @param var
+ * The variable.
+ * @param val
+ * The constant.
+ *
+ * @return A new instance of the predicate in which all occurrences of the
+ * variable have been replaced by the constant.
+ *
+ * @throws IllegalArgumentException
+ * if either argument is <code>null</code>.
*/
+ public Predicate<E> asBound(final IVariable<?> var, final IConstant<?> val);
+
+ /**
+ * Return a new instance in which all occurrences of the variable appearing
+ * in the binding set have been replaced by their bound values.
+ *
+ * @param bindingSet
+ * The binding set.
+ */
public IPredicate<E> asBound(IBindingSet bindingSet);
/**
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java 2010-09-30 19:49:43 UTC (rev 3704)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/ap/Predicate.java 2010-09-30 20:23:37 UTC (rev 3705)
@@ -34,6 +34,7 @@
import cern.colt.Arrays;
import com.bigdata.bop.AbstractAccessPathOp;
+import com.bigdata.bop.ArrayBindingSet;
import com.bigdata.bop.BOp;
import com.bigdata.bop.Constant;
import com.bigdata.bop.IBindingSet;
@@ -43,6 +44,8 @@
import com.bigdata.bop.IVariable;
import com.bigdata.bop.IVariableOrConstant;
import com.bigdata.bop.NV;
+import com.bigdata.rdf.internal.IV;
+import com.bigdata.rdf.spo.SPOPredicate;
import com.bigdata.relation.accesspath.ElementFilter;
import com.bigdata.relation.accesspath.IElementFilter;
import com.bigdata.relation.rule.ISolutionExpander;
@@ -305,6 +308,13 @@
return getProperty(Annotations.REMOTE_ACCESS_PATH,
Annotations.DEFAULT_REMOTE_ACCESS_PATH);
}
+
+ public Predicate<E> asBound(final IVariable<?> var, final IConstant<?> val) {
+
+ return asBound(new ArrayBindingSet(new IVariable[] { var },
+ new IConstant[] { val }));
+
+ }
public Predicate<E> asBound(final IBindingSet bindingSet) {
@@ -323,13 +333,13 @@
final IVariableOrConstant<?> t = (IVariableOrConstant<?>) get(i);
- if (t == null) {
- /*
- * Note: t != null handles the case where the [c] position of an
- * SPO is allowed to be null.
- */
- continue;
- }
+// if (t == null) {
+// /*
+// * Note: t != null handles the case where the [c] position of an
+// * SPO is allowed to be null.
+// */
+// continue;
+// }
if (t.isConstant())
continue;
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/constraint/INBinarySearch.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/constraint/INBinarySearch.java 2010-09-30 19:49:43 UTC (rev 3704)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/constraint/INBinarySearch.java 2010-09-30 20:23:37 UTC (rev 3705)
@@ -63,10 +63,6 @@
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
* @version $Id$
- *
- * @todo unit tests.
- *
- * @todo variant based on a {@link ConcurrentHashMap}.
*/
public class INBinarySearch<T> extends INConstraint<T> {
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/BTreeCostModel.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/BTreeCostModel.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/BTreeCostModel.java 2010-09-30 20:23:37 UTC (rev 3705)
@@ -0,0 +1,113 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Sep 30, 2010
+ */
+package com.bigdata.bop.cost;
+
+import com.bigdata.btree.AbstractBTree;
+import com.bigdata.btree.BTree;
+import com.bigdata.journal.IIndexManager;
+import com.bigdata.journal.Journal;
+
+/**
+ * A cost model for a range scan on a {@link BTree} backed by a {@link Journal}.
+ * The on disk representation of the {@link BTree} does not reflect the index
+ * order so a range scan on the {@link BTree} is basically turned into one
+ * random seek per node or leaf visited.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ *
+ * @todo Add a parameter for the write retention queue? The capacity of the
+ * queue could be turned into an estimate of the #of nodes and leaves
+ * buffered. Alternatively, we have an estimate of the #of distinct nodes
+ * and leaves on the queue in
+ * {@link AbstractBTree#ndistinctOnWriteRetentionQueue}. With that, we
+ * could decide how likely it is that the first N leaves of the
+ * {@link BTree} are in the cache. However, this is all fuzzy since a
+ * focus on one branch of the {@link BTree} could cause nothing but the
+ * root to be in the cache when probing a different branch.
+ */
+public class BTreeCostModel {
+
+ /**
+ * Return the estimated cost of a range scan of the index.
+ *
+ * @param diskCostModel
+ * The cost model for the disk.
+ * @param rangeCount
+ * The range count for the scan.
+ * @param btree
+ * The index.
+ *
+ * @return The estimated cost (milliseconds).
+ *
+ * @todo how to get the right view onto the BTree without locking? or raise
+ * the cost model into the {@link IIndexManager}?
+ */
+ public double rangeScan(final DiskCostModel diskCostModel,
+ final int rangeCount, final BTree btree) {
+
+ if (rangeCount == 0)
+ return 0d;
+
+ // double height = (Math.log(branchingFactor) / Math.log(entryCount)) -
+ // 1;
+
+ final int m = btree.getBranchingFactor();
+
+ final int entryCount = btree.getEntryCount();
+
+ final int height = btree.getHeight();
+
+ // average seek time to a leaf.
+ final double averageSeekTime = Math.max(0, (height - 1))
+ * diskCostModel.seekTime;
+
+ // the percentage of the leaves which are full.
+ // final double leafFillRate = .70d;
+ final double leafFillRate = ((double) btree.getUtilization()[1]) / 100;
+
+ /*
+ * The expected #of leaves to visit for that range scan.
+ *
+ * Note: There is an edge condition when the root leaf is empty
+ * (fillRate is zero).
+ */
+ final double expectedLeafCount = Math.ceil((rangeCount / m)
+ * Math.min(1, (1 / leafFillRate)));
+
+ /*
+ * Expected total time for the key range scan. Overestimates since
+ * ignores cache reuse and OS caching of visited nodes. Ignores transfer
+ * costs.
+ */
+ final double estimatedCost = averageSeekTime * expectedLeafCount;
+
+ return estimatedCost;
+
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/BTreeCostModel.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/DiskCostModel.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/DiskCostModel.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/DiskCostModel.java 2010-09-30 20:23:37 UTC (rev 3705)
@@ -0,0 +1,62 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Sep 30, 2010
+ */
+package com.bigdata.bop.cost;
+
+/**
+ * A cost model of the disk.
+ *
+ * @todo Develop disk models for SAS,SATA,SSD and various RAID configurations,
+ * including the #of spindles in the RAID array.
+ * @todo Develop disk models for SAN, NAS, NFS, parallel file systems, etc.
+ * @todo Conditionally copy the desired disk model parameters into the fields
+ * above to see the performance estimates for a given configuration.
+ * @todo The scattered and sustained write rates can be estimated from the
+ * transfer rate. However, SCSI does much better than SATA when it can
+ * reorder the writes for improved locality.
+ */
+public class DiskCostModel {
+
+ public static final DiskCostModel DEFAULT = new DiskCostModel(10d, 41943040);
+
+ /**
+ * The average disk seek time (milliseconds).
+ */
+ final public double seekTime;
+
+ final public double transferRate;
+
+ /**
+ *
+ * @param seekTime
+ * @param transferRate
+ */
+ public DiskCostModel(double seekTime, double transferRate) {
+ this.seekTime = seekTime;
+ this.transferRate = transferRate;
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/DiskCostModel.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/IndexSegmentCostModel.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/IndexSegmentCostModel.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/IndexSegmentCostModel.java 2010-09-30 20:23:37 UTC (rev 3705)
@@ -0,0 +1,99 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Sep 30, 2010
+ */
+package com.bigdata.bop.cost;
+
+import com.bigdata.btree.IndexSegment;
+
+/**
+ * A cost model for a range scan on an {@link IndexSegment}.
+ * <p>
+ * Note: This uses a summary description of the {@link IndexSegment} for the
+ * cost model. This makes sense because we generally have 100s of index segments
+ * in scale-out and we do not want to probe them all for their exact costs.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ *
+ * @todo
+ */
+public class IndexSegmentCostModel {
+
+ /**
+ *
+ * @param diskCostModel
+ * The disk cost model.
+ * @param rangeCount
+ * The range count for the index scan.
+ * @param branchingFactor
+ * The branching factor for the index segments for this scale-out
+ * index.
+ * @param averageBytesPerLeaf
+ * The average #of bytes per leaf for this scale-out index.
+ * @param xferBufferSize
+ * The size of the disk transfer buffer.
+ *
+ * @return The estimated time for the range scan (milliseconds).
+ */
+ public double rangeScan(final DiskCostModel diskCostModel,
+ final int rangeCount, final int branchingFactor,
+ final int averageBytesPerLeaf, final int xferBufferSize) {
+
+ if (rangeCount == 0)
+ return 0d;
+
+ if (xferBufferSize == 0)
+ throw new IllegalArgumentException();
+
+ // One seek per leaf.
+ final double averageSeekTime = diskCostModel.seekTime;
+
+ // Expected #of leaves to visit.
+ final int expectedLeafCount = (int) Math.ceil(((double) rangeCount)
+ / branchingFactor);
+
+ // Expected #of bytes to transfer.
+ final int leafBytesToXFer = expectedLeafCount * averageBytesPerLeaf;
+
+ // Expected #of disk transfers.
+ final int xfers = (int) Math.ceil(((double) leafBytesToXFer)
+ / xferBufferSize);
+
+ // Expected transfer time (ms).
+ final double xferTime = leafBytesToXFer
+ / (diskCostModel.transferRate / 1000);
+
+ // Expected disk seek time (ms).
+ final double seekTime = averageSeekTime * xfers;
+
+ // Expected total time (ms).
+ final double totalTime = seekTime + xferTime;
+
+ return totalTime;
+
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/IndexSegmentCostModel.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/package.html
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/package.html (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/package.html 2010-09-30 20:23:37 UTC (rev 3705)
@@ -0,0 +1,14 @@
+<html>
+<head>
+<title>Cost models</title>
+</head>
+<body>
+
+<p>
+
+ This package provides cost models for various things.
+
+</p>
+
+</body>
+</html>
\ No newline at end of file
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/cost/package.html
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java 2010-09-30 19:49:43 UTC (rev 3704)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/Rule2BOpUtility.java 2010-09-30 20:23:37 UTC (rev 3705)
@@ -27,6 +27,7 @@
package com.bigdata.bop.engine;
+import java.io.Serializable;
import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
@@ -37,27 +38,44 @@
import java.util.Set;
import org.apache.log4j.Logger;
+import org.openrdf.model.URI;
+import org.openrdf.query.Dataset;
+import org.openrdf.query.algebra.StatementPattern.Scope;
import com.bigdata.bop.BOp;
import com.bigdata.bop.BOpContextBase;
import com.bigdata.bop.BOpEvaluationContext;
import com.bigdata.bop.BOpUtility;
-import com.bigdata.bop.PipelineOp;
+import com.bigdata.bop.Constant;
import com.bigdata.bop.IConstraint;
import com.bigdata.bop.IPredicate;
import com.bigdata.bop.IVariable;
import com.bigdata.bop.NV;
+import com.bigdata.bop.PipelineOp;
import com.bigdata.bop.ap.Predicate;
import com.bigdata.bop.bset.StartOp;
import com.bigdata.bop.join.PipelineJoin;
+import com.bigdata.bop.rdf.join.DataSetJoin;
import com.bigdata.bop.solutions.SliceOp;
+import com.bigdata.rdf.internal.IV;
+import com.bigdata.rdf.lexicon.LexiconRelation;
+import com.bigdata.rdf.model.BigdataURI;
+import com.bigdata.rdf.sail.BigdataEvaluationStrategyImpl;
import com.bigdata.rdf.sail.BigdataSail;
+import com.bigdata.rdf.spo.DefaultGraphSolutionExpander;
+import com.bigdata.rdf.spo.ISPO;
+import com.bigdata.rdf.spo.InGraphHashSetFilter;
+import com.bigdata.rdf.spo.NamedGraphSolutionExpander;
+import com.bigdata.rdf.store.AbstractTripleStore;
+import com.bigdata.rdf.store.IRawTripleStore;
+import com.bigdata.relation.accesspath.ElementFilter;
+import com.bigdata.relation.accesspath.IElementFilter;
import com.bigdata.relation.rule.IProgram;
import com.bigdata.relation.rule.IRule;
import com.bigdata.relation.rule.IStep;
-import com.bigdata.relation.rule.Program;
import com.bigdata.relation.rule.eval.DefaultEvaluationPlan2;
import com.bigdata.relation.rule.eval.IRangeCountFactory;
+import com.bigdata.relation.rule.eval.RuleState;
/**
* Utility class converts {@link IRule}s to {@link BOp}s.
@@ -73,8 +91,113 @@
public class Rule2BOpUtility {
protected static final Logger log = Logger.getLogger(Rule2BOpUtility.class);
+
+ /**
+ * Flag to conditionally enable the new named and default graph support.
+ * <p>
+ * Note: When enabled, the {@link NamedGraphSolutionExpander} and
+ * {@link DefaultGraphSolutionExpander} must be stripped from the
+ * {@link IPredicate.Annotations#EXPANDER}. In the long term, we will simply
+ * no longer generate them in {@link BigdataEvaluationStrategyImpl}.
+ * <p>
+ * Note: If you want to test just the named graph stuff, then the default
+ * graph processing could be handed off to the
+ * {@link DefaultGraphSolutionExpander}.
+ */
+ private static boolean enableDecisionTree = false;
/**
+ * Annotations used by the {@link BigdataEvaluationStrategyImpl} to
+ * communicate with the {@link Rule2BOpUtility}.
+ * <p>
+ * <h3>Quads Mode</h3>
+ * Several annotations are used to mark named and default graph patterns on
+ * the {@link IPredicate}s. Rather than attaching a named or default graph
+ * expander, we annotate the predicate with the metadata for the access path
+ * and then convert that annotation to the appropriate bop pattern in
+ * {@link Rule2BOpUtility}.
+ */
+ public interface Annotations {
+
+ /**
+ * Boolean flag indicates that the database is operating in quads mode.
+ */
+ String QUADS = Rule2BOpUtility.class.getName() + ".quads";
+
+ boolean DEFAULT_QUADS = false;
+
+ /**
+ * The {@link Dataset} associated with the access path (quads mode
+ * only). The {@link Dataset} is only provided by openrdf when FROM or
+ * FROM NAMED was used in the query. Otherwise the {@link Dataset} will
+ * be <code>null</code> and is not attached as an annotation.
+ * <p>
+ * Note: This annotation MUST be stripped from the query plan to prevent
+ * an attempt to serialized it for RMI in scale-out (the {@link Dataset}
+ * is not {@link Serializable}, can be quite large, and is captured by
+ * other constructions in the generated query plan).
+ */
+ String DATASET = Rule2BOpUtility.class.getName() + ".dataset";
+
+ /**
+ * The {@link Scope} of the access path (quads mode only). In quads mode
+ * the {@link Scope} is always provided by openrdf.
+ *
+ * @see Scope#NAMED_CONTEXTS
+ * @see Scope#DEFAULT_CONTEXTS
+ */
+ String SCOPE = Rule2BOpUtility.class.getName() + ".scope";
+
+ /**
+ * The graph variable specified in the query (quads mode only). This is
+ * <p>
+ * Note: This is not used for SIDs mode because we use the standard
+ * triple store access paths.
+ *
+ * @see org.openrdf.query.algebra.Var
+ *
+ * @todo can we just use pred.get(3)?
+ */
+ String CVAR = Rule2BOpUtility.class.getName() + ".cvar";
+
+ /*
+ * Cost estimates.
+ */
+
+ /**
+ * The estimated cost of a SCAN + FILTER approach to a default graph or
+ * named graph query.
+ */
+ String COST_SCAN = Rule2BOpUtility.class.getName() + ".costScan";
+
+ /**
+ * The estimated cost of a SUBQUERY approach to a default graph or named
+ * graph query.
+ */
+ String COST_SUBQUERY = Rule2BOpUtility.class.getName()
+ + ".costSubquery";
+
+ /**
+ * The #of known graphs in the {@link Dataset} for a default graph or
+ * named graph query.
+ */
+ String NKNOWN = Rule2BOpUtility.class.getName() + ".nknown";
+
+ }
+
+ /**
+ * A list of annotations to be cleared from {@link Predicate} when they are
+ * copied into a query plan.
+ */
+ private static final String[] ANNS_TO_CLEAR_FROM_PREDICATE = new String[] {
+ Annotations.QUADS,//
+ Annotations.DATASET,//
+ Annotations.SCOPE,//
+ Annotations.CVAR,//
+ IPredicate.Annotations.OPTIONAL //
+ };
+
+ /**
* Convert an {@link IStep} into an operator tree. This should handle
* {@link IRule}s and {@link IProgram}s as they are currently implemented
* and used by the {@link BigdataSail}.
@@ -84,13 +207,13 @@
*
* @return
*/
- public static PipelineOp convert(final IStep step,
- final int startId, final QueryEngine queryEngine) {
-
+ public static PipelineOp convert(final IStep step, final int startId,
+ final AbstractTripleStore db, final QueryEngine queryEngine) {
+
if (step instanceof IRule)
- return convert((IRule) step, startId, queryEngine);
+ return convert((IRule) step, startId, db, queryEngine);
- throw new UnsupportedOperationException();
+ return convert((IProgram) step, startId, db, queryEngine);
}
@@ -101,11 +224,11 @@
*
* @return
*/
- public static PipelineOp convert(final IRule rule,
- final int startId, final QueryEngine queryEngine) {
+ public static PipelineOp convert(final IRule rule, final int startId,
+ final AbstractTripleStore db, final QueryEngine queryEngine) {
int bopId = startId;
-
+
final PipelineOp startOp = new StartOp(new BOp[] {},
NV.asMap(new NV[] {//
new NV(Predicate.Annotations.BOP_ID, bopId++),//
@@ -128,8 +251,13 @@
}, rule);
+ // evaluation plan order.
final int[] order = plan.getOrder();
+ // variables to be retained for each join.
+ final IVariable[][] selectVars = RuleState
+ .computeRequiredVarsForEachTail(rule, order);
+
/*
* Map the constraints from the variables they use. This way, we can
* properly attach constraints to only the first tail in which the
@@ -174,8 +302,9 @@
final int joinId = bopId++;
// assign a bop id to the predicate
- final IPredicate<?> pred = rule.getTail(order[i]).setBOpId(bopId++);
-
+ Predicate<?> pred = (Predicate<?>) rule.getTail(order[i]).setBOpId(
+ bopId++);
+
/*
* Collect all the constraints for this predicate based on which
* variables make their first appearance in this tail
@@ -189,8 +318,8 @@
* that make their first appearance in this tail.
*/
for (BOp arg : pred.args()) {
- if (arg instanceof IVariable) {
- final IVariable<?> v = (IVariable) arg;
+ if (arg instanceof IVariable<?>) {
+ final IVariable<?> v = (IVariable<?>) arg;
/*
* We do a remove because we don't ever need to run these
* constraints again during subsequent joins once they
@@ -204,22 +333,94 @@
constraints.addAll(constraintsByVar.remove(v));
}
}
+
+ // annotations for this join.
+ final List<NV> anns = new LinkedList<NV>();
- final PipelineOp joinOp = new PipelineJoin(//
- left, pred,//
- NV.asMap(new NV[] {//
- new NV(BOp.Annotations.BOP_ID, joinId),//
- new NV(PipelineJoin.Annotations.CONSTRAINTS,
- constraints.size() > 0 ?
- constraints.toArray(new IConstraint[constraints.size()]) : null),//
- new NV(PipelineJoin.Annotations.OPTIONAL, pred.isOptional()),//
- // Note: shard-partitioned joins!
- new NV( Predicate.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.SHARDED),//
- }));
+ anns.add(new NV(BOp.Annotations.BOP_ID, joinId));
+
+ anns.add(new NV(PipelineJoin.Annotations.SELECT,
+ selectVars[order[i]]));
- left = joinOp;
+ if (pred.isOptional())
+ anns.add(new NV(PipelineJoin.Annotations.OPTIONAL, pred
+ .isOptional()));
+ if (!constraints.isEmpty())
+ anns.add(new NV(PipelineJoin.Annotations.CONSTRAINTS,
+ constraints
+ .toArray(new IConstraint[constraints.size()])));
+
+ final Scope scope = (Scope) pred.getProperty(Annotations.SCOPE);
+
+ // @todo can we just use pred.get(3)?
+ final org.openrdf.query.algebra.Var cvar = (org.openrdf.query.algebra.Var) pred
+ .getProperty(Annotations.CVAR);
+
+ // true iff this is a quads access path.
+ final boolean quads = pred.getProperty(Annotations.QUADS,
+ Annotations.DEFAULT_QUADS);
+
+ // strip off annotations that we do not want to propagate.
+ pred = pred.clearAnnotations(ANNS_TO_CLEAR_FROM_PREDICATE);
+
+ if (quads) {
+
+ /*
+ * Quads mode.
+ */
+
+ if (enableDecisionTree) {
+ /*
+ * Strip off the named graph or default graph expander (in
+ * the long term it will simply not be generated.)
+ */
+ pred = pred
+ .clearAnnotations(new String[] { IPredicate.Annotations.EXPANDER });
+
+ switch (scope) {
+ case NAMED_CONTEXTS:
+ left = namedGraphJoin(queryEngine, left, anns, pred,
+ cvar);
+ break;
+ case DEFAULT_CONTEXTS:
+ left = defaultGraphJoin(queryEngine, left, anns, pred,
+ cvar);
+ break;
+ default:
+ throw new AssertionError();
+ }
+
+ } else {
+
+ /*
+ * This is basically the old way of handling quads query
+ * using expanders which were attached by
+ * BigdataEvaluationStrategyImpl.
+ */
+
+ final boolean scaleOut = queryEngine.isScaleOut();
+ if (scaleOut)
+ throw new UnsupportedOperationException();
+
+ anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
+ BOpEvaluationContext.ANY));
+
+ left = new PipelineJoin(new BOp[] { left, pred }, anns
+ .toArray(new NV[anns.size()]));
+
+ }
+
+ } else {
+
+ /*
+ * Triples or provenance mode.
+ */
+
+ left = triplesModeJoin(queryEngine, left, anns, pred);
+
+ }
+
}
// just for now while i'm debugging
@@ -228,42 +429,309 @@
return left;
}
-
- private static String toString(BOp bop) {
+
+ /**
+ * Generate a {@link PipelineJoin} for a triples mode access path.
+ *
+ * @param queryEngine
+ * @param left
+ * @param anns
+ * @param pred
+ *
+ * @return The join operator.
+ */
+ private static PipelineOp triplesModeJoin(final QueryEngine queryEngine,
+ final PipelineOp left, final List<NV> anns, final Predicate pred) {
+
+ final boolean scaleOut = queryEngine.isScaleOut();
+ if (scaleOut) {
+ anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
+ BOpEvaluationContext.SHARDED));
+ } else {
+ anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
+ BOpEvaluationContext.ANY));
+ }
+
+ return new PipelineJoin(new BOp[] { left, pred }, anns
+ .toArray(new NV[anns.size()]));
+
+ }
+
+ /**
+ * Generate a named graph join (quads mode).
+ *
+ * @param queryEngine
+ * @param left
+ * @param anns
+ * @param pred
+ * @param cvar
+ * @return
+ */
+ private static PipelineOp namedGraphJoin(final QueryEngine queryEngine,
+ final PipelineOp left, final List<NV> anns, Predicate pred,
+ final org.openrdf.query.algebra.Var cvar) {
+
+ final Dataset dataset = (Dataset) pred.getProperty(Annotations.DATASET);
+
+ final boolean scaleOut = queryEngine.isScaleOut();
+ if (scaleOut) {
+ anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
+ BOpEvaluationContext.SHARDED));
+ } else {
+ anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
+ BOpEvaluationContext.ANY));
+ }
+
+ final DataSetSummary summary = new DataSetSummary(dataset
+ .getNamedGraphs());
+
+ anns.add(new NV(Annotations.NKNOWN, summary.nknown));
+
+ // true iff C is bound to a constant.
+ final boolean isCBound = cvar.getValue() != null;
+ if (isCBound) {
+
+ /*
+ * C is already bound. The unmodified access path is used.
+ */
+
+ return new PipelineJoin(new BOp[] { left, pred }, anns
+ .toArray(new NV[anns.size()]));
+
+ } else if (summary.nknown == 0) {
+
+ /*
+ * The data set is empty (no graphs). Return a join backed by an
+ * empty access path.
+ *
+ * Note: Since the join could be optional or part of an optional
+ * join group, we can not just drop it. Instead we need to return a
+ * join against an empty access path. Since the join could also
+ * "select" for some subset of variables, it seems that we really
+ * need to modify PipelineJoin to recognize an annotation indicating
+ * an empty access path. It can then substitute the empty access
+ * path when processing the source binding sets. There should be
+ * unit tests for this.
+ *
+ * FIXME Return PipelineJoin with an EMPTY ACCESS PATH.
+ */
+
+ throw new UnsupportedOperationException();
+
+ } else if (summary.nknown == 1) {
+
+ /*
+ * The dataset contains exactly one graph. Bind C.
+ */
+
+ pred = pred.asBound((IVariable) pred.get(3), new Constant(
+ summary.firstContext));
+
+ return new PipelineJoin(new BOp[] { left, pred }, anns
+ .toArray(new NV[anns.size()]));
+
+ } else if (dataset == null) {
+
+ /*
+ * The dataset is all graphs. C is left unbound and the unmodified
+ * access path is used.
+ */
+
+ return new PipelineJoin(new BOp[] { left, pred }, anns
+ .toArray(new NV[anns.size()]));
+
+ } else {
+
+ /*
+ * Estimate cost of SCAN with C unbound)
+ */
+ final double scanCost = getScanCost(pred);
+
+ anns.add(new NV(Annotations.COST_SCAN, scanCost));
+
+ /*
+ * Estimate cost of SUBQUERY with C bound.
+ */
+ final double subqueryCost = getSubqueryCost(pred);
+
+ anns.add(new NV(Annotations.COST_SUBQUERY, subqueryCost));
+
+ if (scanCost < subqueryCost * summary.nknown) {
+
+ /*
+ * Scan and filter. C is left unbound. We do a range scan on the
+ * index and filter using an IN constraint.
+ */
+
+ // IN filter for the named graphs.
+ final IElementFilter<ISPO> test = new InGraphHashSetFilter<ISPO>(
+ summary.nknown, summary.graphs);
+
+ // layer filter onto the predicate.
+ pred = pred
+ .addIndexLocalFilter(ElementFilter.newInstance(test));
+
+ return new PipelineJoin(new BOp[] { left, pred }, anns
+ .toArray(new NV[anns.size()]));
+
+ } else {
+
+ /*
+ * Parallel Subquery.
+ */
+
+ /*
+ * Setup the data set join.
+ *
+ * @todo When the #of named graphs is large we need to do
+ * something special to avoid sending huge graph sets around
+ * with the query. For example, we should create named data sets
+ * and join against them rather than having an in-memory
+ * DataSetJoin.
+ *
+ * @todo The historical approach performed parallel subquery
+ * using an expander pattern rather than a data set join. The
+ * data set join should have very much the same effect, but it
+ * may need to emit multiple chunks to have good parallelism.
+ */
+
+ // The variable to be bound.
+ final IVariable var = (IVariable) pred.get(3);
+
+ // The data set join.
+ final DataSetJoin dataSetJoin = new DataSetJoin(
+ new BOp[] { var }, NV.asMap(new NV[] {
+ new NV(DataSetJoin.Annotations.VAR, var),
+ new NV(DataSetJoin.Annotations.GRAPHS, summary
+ .getGraphs()) }));
+
+ if (scaleOut) {
+ anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
+ BOpEvaluationContext.SHARDED));
+ anns.add(new NV(Predicate.Annotations.REMOTE_ACCESS_PATH,
+ false));
+ } else {
+ anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
+ BOpEvaluationContext.ANY));
+ anns.add(new NV(Predicate.Annotations.REMOTE_ACCESS_PATH,
+ false));
+ }
+
+ return new PipelineJoin(new BOp[] { left, pred }, anns
+ .toArray(new NV[anns.size()]));
+
+ }
+
+ }
+
+ }
+
+ /**
+ *
+ * @param pred
+ * @return
+ *
+ * FIXME Cost models have been implemented, but are not yet hooked in.
+ */
+ static double getScanCost(Predicate pred) {
+ /*
+ * @todo Scan is more expensive on the Journal so this is set to ONE (1)
+ * and subquery is set to ZERO (0). This will get replaced by the actual
+ * computed costs shortly.
+ */
+ return 1d;
+ }
+
+ /**
+ *
+ * @param pred
+ * @return
+ *
+ * FIXME Cost models have been implemented, but are not yet hooked
+ * in.
+ */
+ static double getSubqueryCost(Predicate pred) {
+ return 0d;
+ }
+
+ /**
+ * Generate a default graph join (quads mode).
+ *
+ * @param queryEngine
+ * @param left
+ * @param anns
+ * @param pred
+ * @return
+ *
+ * @todo The default graph remote access path query estimates do not take
+ * RMI costs into account. This is Ok since we are only comparing
+ * remote access paths with other remote access paths.
+ */
+ private static PipelineOp defaultGraphJoin(final QueryEngine queryEngine,
+ final PipelineOp left, final List<NV> anns, final Predicate pred,
+ final org.openrdf.query.algebra.Var cvar) {
+
+ // @todo decision of local vs remote ap.
+ final boolean scaleOut = queryEngine.isScaleOut();
+ if (scaleOut) {
+ anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
+ BOpEvaluationContext.SHARDED));
+ } else {
+ anns.add(new NV(Predicate.Annotations.EVALUATION_CONTEXT,
+ BOpEvaluationContext.ANY));
+ }
+
+ /*
+ * FIXME implement the default graph decision tree.
+ */
+ throw new UnsupportedOperationException();
+
+ }
+
+ /**
+ * Pretty print (aspects of) a bop.
+ *
+ * @param bop
+ * The bop.
+ *
+ * @return The formatted representation.
+ */
+ private static String toString(final BOp bop) {
+
StringBuilder sb = new StringBuilder();
toString(bop, sb, 0);
// chop off the last \n
- sb.setLength(sb.length()-1);
-
+ sb.setLength(sb.length() - 1);
+
return sb.toString();
-
+
}
-
- private static void toString(final BOp bop, final StringBuilder sb,
+
+ private static void toString(final BOp bop, final StringBuilder sb,
final int indent) {
-
+
for (int i = 0; i < indent; i++) {
sb.append(' ');
}
sb.append(bop).append('\n');
if (bop != null) {
- List<BOp> args = bop.args();
+ final List<BOp> args = bop.args();
for (BOp arg : args) {
- toString(arg, sb, indent+4);
+ toString(arg, sb, indent + 4);
}
- IConstraint[] constraints = (IConstraint[]) bop
+ final IConstraint[] constraints = (IConstraint[]) bop
.getProperty(PipelineJoin.Annotations.CONSTRAINTS);
if (constraints != null) {
for (IConstraint c : constraints) {
- toString(c, sb, indent+4);
+ toString(c, sb, indent + 4);
}
}
}
-
+
}
/**
@@ -275,10 +743,139 @@
*
* FIXME What is the pattern for UNION?
*/
- public static PipelineOp convert(final Program program) {
+ public static PipelineOp convert(final IProgram rule, final int startId,
+ final AbstractTripleStore db, final QueryEngine queryEngine) {
throw new UnsupportedOperationException();
}
+ /**
+ * Helper class summarizes the named graphs for a quads mode query.
+ *
+ * @todo This could be used for either named or default graphs. All it does
+ * not report the #of URIs known to the database.
+ *
+ * @todo This summary could be computed once for a given query for its named
+ * graphs and once for its default graph. We do not need to do this
+ * for each predicate in the query.
+ */
+ private static class DataSetSummary {
+
+ /**
+ * The set of graphs. The {@link URI}s MUST have been resolved against
+ * the appropriate {@link LexiconRelation} such that their term
+ * identifiers (when the exist) are known. If any term identifier is
+ * {@link IRawTripleStore#NULL}, then the corresponding graph does not
+ * exist and no access path will be queried for that graph. However, a
+ * non- {@link IRawTripleStore#NULL} term identifier may also identify a
+ * graph which does not exist, in which case an access path will be
+ * created for that {@link URI}s but will not visit any data.
+ */
+ public final Iterable<? extends URI> graphs;
+
+ /**
+ * The #of graphs in {@link #graphs} whose term identifier is known.
+ * While this is not proof that there is data in the quad store for a
+ * graph having the corresponding {@link URI}, it does allow the
+ * possibility that a graph could exist for that {@link URI}.
+ */
+ public final int nknown;
+// * <p>
+// * If {@link #nknown} is ZERO (0), then the access path is empty.
+// * <p>
+// * If {@link #nknown} is ONE (1), then the caller's {@link IAccessPath}
+// * should be used and filtered to remove the context information. If
+// * {@link #graphs} is <code>null</code>, which implies that ALL graphs
+// * in the quad store will be used as the default graph, then
+// * {@link #nknown} will be {@link Integer#MAX_VALUE}.
+
+ /**
+ * The term identifier for the first graph and
+ * {@link IRawTripleStore#NULL} if no graphs were specified having a
+ * term identifier.
+ */
+ public final IV firstContext;
+
+ /**
+ *
+ * @param graphs
+ * The set of named graphs in the SPARQL DATASET (optional).
+ * A runtime exception will be thrown during evaluation of
+ * the if the {@link URI}s are not {@link BigdataURI}s. If
+ * <code>graphs := null</code>, then the set of named graphs
+ * is understood to be ALL graphs in the quad store.
+ */
+ public DataSetSummary(final Iterable<? extends URI> graphs) {
+
+ this.graphs = graphs;
+
+ IV firstContext = null;
+
+ if (graphs == null) {
+
+ nknown = Integer.MAX_VALUE;
+
+ } else {
+
+ final Iterator<? extends URI> itr = graphs.iterator();
+
+ int nknown = 0;
+
+ while (itr.hasNext()) {
+
+ final BigdataURI uri = (BigdataURI) itr.next();
+
+ if (uri.getIV() != null) {
+
+ if (++nknown == 1) {
+
+ firstContext = uri.getIV();
+
+ }
+
+ }
+
+ } // while
+
+ this.nknown = nknown;
+
+ }
+
+ this.firstContext = firstContext;
+
+ }
+
+ /**
+ * Return a dense array of the {@link IV}s for the graphs known to the
+ * database.
+ */
+ public IV[] getGraphs() {
+
+ final IV[] a = new IV[nknown];
+
+ final Iterator<? extends URI> itr = graphs.iterator();
+
+ int nknown = 0;
+
+ while (itr.hasNext()) {
+
+ final BigdataURI uri = (BigdataURI) itr.next();
+
+ final IV id = uri.getIV();
+
+ if (id != null) {
+
+ a[nknown++] = id;
+
+ }
+
+ } // while
+
+ return a;
+
+ }
+
+ } // DataSetSummary
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/bop/rdf/join/DataSetJoin.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/bop/rdf/join/DataSetJoin.java 2010-09-30 19:49:43 UTC (rev 3704)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/bop/rdf/join/DataSetJoin.java 2010-09-30 20:23:37 UTC (rev 3705)
@@ -82,7 +82,7 @@
String VAR = DataSetJoin.class.getName() + ".var";
/**
- * The {@link IV}s to be bound. This is logically a set and SHOULD NOT
+ * The {@link IV}[]s to be bound. This is logically a set and SHOULD NOT
* include duplicates. The elements in this array SHOULD be ordered for
* improved efficiency.
*/
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/InGraphBinarySearchFilter.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/InGraphBinarySearchFilter.java 2010-09-30 19:49:43 UTC (rev 3704)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/InGraphBinarySearchFilter.java 2010-09-30 20:23:37 UTC (rev 3705)
@@ -6,12 +6,11 @@
import java.io.ObjectOutput;
import java.util.Arrays;
import java.util.HashSet;
+
import org.openrdf.model.URI;
-import com.bigdata.bop.constraint.INBinarySearch;
import com.bigdata.rdf.internal.IV;
import com.bigdata.rdf.model.BigdataURI;
-import com.bigdata.relation.rule.eval.ISolution;
/**
* "IN" filter for the context position based on a sorted long[] of the
@@ -24,8 +23,6 @@
* @version $Id$
*
* @see InGraphHashSetFilter
- *
- * @todo reconcile with {@link INBinarySearch}
*/
public final class InGraphBinarySearchFilter<E extends ISPO> extends SPOFilter<E>
implements Externalizable {
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/InGraphHashSetFilter.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/InGraphHashSetFilter.java 2010-09-30 19:49:43 UTC (rev 3704)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/InGraphHashSetFilter.java 2010-09-30 20:23:37 UTC (rev 3705)
@@ -1,31 +1,23 @@
package com.bigdata.rdf.spo;
import java.util.HashSet;
-import it.unimi.dsi.fastutil.longs.LongLinkedOpenHashSet;
import org.openrdf.model.URI;
-import com.bigdata.bop.constraint.INHashMap;
import com.bigdata.rdf.internal.IV;
import com.bigdata.rdf.model.BigdataURI;
-import com.bigdata.rdf.store.IRawTripleStore;
-import com.bigdata.relation.rule.eval.ISolution;
/**
* "IN" filter for the context position based on a native long hash set
- * containing the acceptable graph identifiers. While evaluation of the
- * access path will be ordered, the filter does not maintain evolving state
- * so a hash set will likely beat a binary search.
+ * containing the acceptable graph identifiers. While evaluation of the access
+ * path will be ordered, the filter does not maintain evolving state so a hash
+ * set will likely beat a binary search.
*
- * @author <a href="mailto:tho...@us...">Bryan
- * Thompson</a>
- * @version $Id$
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id: InGraphHashSetFilter.java 3694 2010-09-30 14:54:59Z mrpersonick
+ * $
*
* @see InGraphBinarySearchFilter
- *
- * @todo reconcile with {@link INHashMap}.
- *
- * @todo tighten serialization?
*/
public final class InGraphHashSetFilter<E extends ISPO> extends SPOFilter<E> {
@@ -34,7 +26,7 @@
*/
private static final long serialVersionUID = -6059009162692785772L;
- final HashSet<IV> contextSet;
+ private final HashSet<IV> contextSet;
/**
*
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPOAccessPath.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPOAccessPath.java 2010-09-30 19:49:43 UTC (rev 3704)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPOAccessPath.java 2010-09-30 20:23:37 UTC (rev 3705)
@@ -181,6 +181,9 @@
* The context term identifier.
*
* @return The constrained {@link IAccessPath}.
+ *
+ * @deprecated with {@link DefaultGraphSolutionExpander} and
+ * {@link NamedGraphSolutionExpander}.
*/
public SPOAccessPath bindContext(final IV c) {
@@ -205,8 +208,10 @@
* is to be set
*
* @return The constrained {@link IAccessPath}.
+ *
+ * @deprecated with {@link #bindContext(IV)}
*/
- public SPOAccessPath bindPosition(final int position, final IV v) {
+ private SPOAccessPath bindPosition(final int position, final IV v) {
if (v == null) {
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPOPredicate.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPOPredicate.java 2010-09-30 19:49:43 UTC (rev 3704)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/spo/SPOPredicate.java 2010-09-30 20:23:37 UTC (rev 3705)
@@ -25,11 +25,9 @@
import java.util.Map;
-import com.bigdata.bop.ArrayBindingSet;
import com.bigdata.bop.BOp;
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.IConstant;
-import com.bigdata.bop.IVariable;
import com.bigdata.bop.IVariableOrConstant;
import com.bigdata.bop.NV;
import com.bigdata.bop.ap.Predicate;
@@ -51,20 +49,6 @@
*/
private static final long serialVersionUID = 1L;
-// /**
-// * The arity is 3 unless the context position was given (as either a
-// * variable or bound to a constant) in which case it is 4.
-// *
-// * @todo rather than having a conditional arity, modify the SPOPredicate
-// * constructor to pass on either args[3] or args[3] depending on
-// * whether we are using triples or quads.
-// */
-// public final int arity() {
-//
-// return get(3/*c*/) == null ? 3 : 4;
-//
-// }
-
/**
* Variable argument version of the shallow copy constructor.
*/
@@ -236,191 +220,6 @@
}
-// /**
-// * Fully specified ctor.
-// *
-// * @param relationName
-// * @param partitionId
-// * @param s
-// * @param p
-// * @param o
-// * @param c
-// * MAY be <code>null</code>.
-// * @param optional
-// * @param constraint
-// * MAY be <code>null</code>.
-// * @param expander
-// * MAY be <code>null</code>.
-// * @param timestamp
-// * The timestamp or transaction identifier against which the
-// * predicate will read or write.
-// */
-// public SPOPredicate(//
-// final String[] relationName, //
-// final int partitionId, //
-// final IVariableOrConstant<IV> s,//
-// final IVariableOrConstant<IV> p,//
-// final IVariableOrConstant<IV> o,//
-// final IVariableOrConstant<IV> c,//
-// final boolean optional, //
-// final IElementFilter<ISPO> constraint,//
-// final ISolutionExpander<ISPO> expander//
-//// final long timestamp
-// ) {
-//
-// super((c == null ? new IVariableOrConstant[] { s, p, o }
-// : new IVariableOrConstant[] { s, p, o, c }), //
-// new NV(Annotations.RELATION_NAME, relationName), //
-// new NV(Annotations.PARTITION_ID, partitionId), //
-// new NV(Annotations.OPTIONAL, optional), //
-// new NV(Annotations.CONSTRAINT, constraint), //
-// new NV(Annotations.EXPANDER, expander));
-//
-//// if (relationName == null)
-//// throw new IllegalArgumentException();
-////
-//// for (int i = 0; i < relationName.length; i++) {
-////
-//// if (relationName[i] == null)
-//// throw new IllegalArgumentException();
-////
-//// }
-////
-//// if (relationName.length == 0)
-//// throw new IllegalArgumentException();
-////
-//// if (partitionId < -1)
-//// throw new IllegalArgumentException();
-////
-//// if (s == null)
-//// throw new IllegalArgumentException();
-////
-//// if (p == null)
-//// throw new IllegalArgumentException();
-////
-//// if (o == null)
-//// throw new IllegalArgumentException();
-////
-//// this.relationName = relationName;
-////
-//// this.partitionId = partitionId;
-////
-//// this.s = s;
-//// this.p = p;
-//// this.o = o;
-//// this.c = c; // MAY be null.
-////
-//// this.optional = optional;
-////
-//// this.constraint = constraint; /// MAY be null.
-////
-//// this.expander = expander;// MAY be null.
-//
-// }
-
-// /**
-// * Copy constructor overrides the relation name(s).
-// *
-// * @param relationName
-// * The new relation name(s).
-// */
-// protected SPOPredicate(final SPOPredicate src, final String[] relationName) {
-//
-// if (relationName == null)
-// throw new IllegalArgumentException();
-//
-// for(int i=0; i<relationName.length; i++) {
-//
-// if (relationName[i] == null)
-// throw new IllegalArgumentException();
-//
-// }
-//
-// if (relationName.length == 0)
-// throw new IllegalArgumentException();
-//
-// this.partitionId = src.partitionId;
-//
-// this.s = src.s;
-// this.p = src.p;
-// this.o = src.o;
-// this.c = src.c;
-//
-// this.relationName = relationName; // override.
-//
-// this.optional = src.optional;
-//
-// this.constraint = src.constraint;
-//
-// this.expander = src.expander;
-//
-// }
-
-// /**
-// * Copy constructor sets the index partition identifier.
-// *
-// * @param partitionId
-// * The index partition identifier.
-// *
-// * @throws IllegalArgumentException
-// * if the index partition identified is a negative integer.
-// * @throws IllegalStateException
-// * if the index partition identifier was already specified.
-// */
-// protected SPOPredicate(final SPOPredicate src, final int partitionId) {
-//
-// //@todo uncomment the other half of this test to make it less paranoid.
-// if (src.partitionId != -1 ) {//&& this.partitionId != partitionId) {
-//
-// throw new IllegalStateException();
-//
-// }
-//
-// if (partitionId < 0) {
-//
-// throw new IllegalArgumentException();
-//
-// }
-//
-// this.relationName = src.relationName;
-//
-// this.partitionId = partitionId;
-//
-// this.s = src.s;
-// this.p = src.p;
-// this.o = src.o;
-// this.c = src.c;
-// ...
[truncated message content] |
|
From: <tho...@us...> - 2010-09-30 19:49:49
|
Revision: 3704
http://bigdata.svn.sourceforge.net/bigdata/?rev=3704&view=rev
Author: thompsonbry
Date: 2010-09-30 19:49:43 +0000 (Thu, 30 Sep 2010)
Log Message:
-----------
Found a problem where I broke the build.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java 2010-09-30 19:03:42 UTC (rev 3703)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java 2010-09-30 19:49:43 UTC (rev 3704)
@@ -214,8 +214,13 @@
this.resourceService = resourceService;
- // the proxy for this query engine when used as a query controller.
- this.clientProxy = (IQueryClient) ((JiniFederation<?>)fed).getProxy(this, false/*enableDGC*/);
+ if(fed instanceof JiniFederation<?>) {
+ // the proxy for this query engine when used as a query controller.
+ this.clientProxy = (IQueryClient) ((JiniFederation<?>)fed).getProxy(this, false/*enableDGC*/);
+ } else {
+ // E.g., an EmbeddedFederation in the test suite.
+ this.clientProxy = this;
+ }
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mar...@us...> - 2010-09-30 19:03:48
|
Revision: 3703
http://bigdata.svn.sourceforge.net/bigdata/?rev=3703&view=rev
Author: martyncutcher
Date: 2010-09-30 19:03:42 +0000 (Thu, 30 Sep 2010)
Log Message:
-----------
Remove prefetches in constructors and ensure NoSuchElementExceltions are thrown
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Appenderator.java
branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/ArrayIterator.java
branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Contractorator.java
branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Mergerator.java
branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Resolverator.java
branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Sorterator.java
Modified: branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Appenderator.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Appenderator.java 2010-09-30 18:18:03 UTC (rev 3702)
+++ branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Appenderator.java 2010-09-30 19:03:42 UTC (rev 3703)
@@ -66,7 +66,7 @@
return m_current.next();
}
- throw new NoSuchElementException("FilterIterator");
+ throw new NoSuchElementException("Appenderator");
}
//-------------------------------------------------------------
Modified: branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/ArrayIterator.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/ArrayIterator.java 2010-09-30 18:18:03 UTC (rev 3702)
+++ branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/ArrayIterator.java 2010-09-30 19:03:42 UTC (rev 3703)
@@ -26,6 +26,7 @@
package cutthecrap.utils.striterators;
import java.util.Iterator;
+import java.util.NoSuchElementException;
/**
* Supports standard iteration over an object Array, allowing this to
@@ -47,7 +48,10 @@
/** @return current index from array **/
public Object next() {
- return m_src[m_index++];
+ if (m_index < m_src.length)
+ return m_src[m_index++];
+ else
+ throw new NoSuchElementException("ArrayIterator");
}
/** void .. does nothing **/
Modified: branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Contractorator.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Contractorator.java 2010-09-30 18:18:03 UTC (rev 3702)
+++ branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Contractorator.java 2010-09-30 19:03:42 UTC (rev 3703)
@@ -47,21 +47,29 @@
protected final Object m_ctx;
private final Contractor m_contractor;
private Object m_next;
+ private boolean m_init = false;
public Contractorator(Iterator src, final Object ctx, Contractor contractor) {
m_src = src;
m_ctx = ctx;
m_contractor = contractor;
-
- m_next = m_contractor.contract(m_src);
}
+
+ private void init() {
+ if (!m_init) {
+ m_next = m_contractor.contract(m_src);
+ m_init = true;
+ }
+ }
public boolean hasNext() {
+ init();
+
return m_next != null;
}
public Object next() {
- if (m_next == null) {
+ if (!hasNext()) {
throw new NoSuchElementException();
}
Modified: branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Mergerator.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Mergerator.java 2010-09-30 18:18:03 UTC (rev 3702)
+++ branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Mergerator.java 2010-09-30 19:03:42 UTC (rev 3703)
@@ -40,26 +40,35 @@
private Object m_valA = null;
private Object m_valB = null;
+ private boolean m_init = false;
public Mergerator(Iterator setA, Iterator setB, Object context, Comparator comparator) {
m_context = context;
m_setA = setA;
m_setB = setB;
- m_comparator = comparator;
-
- if (m_setA.hasNext()) {
- m_valA = m_setA.next();
- }
-
- if (m_setB.hasNext()) {
- m_valB = m_setB.next();
- }
+ m_comparator = comparator;
}
+
+ private void init() {
+ if (!m_init) {
+ if (m_setA.hasNext()) {
+ m_valA = m_setA.next();
+ }
+
+ if (m_setB.hasNext()) {
+ m_valB = m_setB.next();
+ }
+
+ m_init = true;
+ }
+ }
//-------------------------------------------------------------
public boolean hasNext() {
+ init();
+
return m_valA != null || m_valB != null;
}
Modified: branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Resolverator.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Resolverator.java 2010-09-30 18:18:03 UTC (rev 3702)
+++ branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Resolverator.java 2010-09-30 19:03:42 UTC (rev 3703)
@@ -50,7 +50,10 @@
}
public Object next() {
- return m_resolver.resolve(m_iter.next());
+ if (hasNext())
+ return m_resolver.resolve(m_iter.next());
+ else
+ throw new NoSuchElementException("Resolverator");
}
public void remove() {
Modified: branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Sorterator.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Sorterator.java 2010-09-30 18:18:03 UTC (rev 3702)
+++ branches/QUADS_QUERY_BRANCH/ctc-striterators/src/java/cutthecrap/utils/striterators/Sorterator.java 2010-09-30 19:03:42 UTC (rev 3703)
@@ -21,57 +21,63 @@
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
+ */
package cutthecrap.utils.striterators;
import java.util.*;
/**
- * Initialized with a Sorter object, wraps a standard iterator and resolves each returned object
+ * Initialized with a Sorter object, wraps a standard iterator and resolves each
+ * returned object
*/
public class Sorterator implements Iterator {
- private final Iterator m_iter;
+ private final Iterator m_src;
+ private Iterator m_iter;
+ private final Sorter m_sorter;
protected final Object m_context;
-
- public Sorterator(Iterator iter, Object context, Sorter sorter) {
+ private boolean m_doneSort = false;
- this.m_context = context;
+ public Sorterator(Iterator iter, Object context, Sorter sorter) {
- // materialize the objects to be sorted.
- LinkedList tmp = new LinkedList();
-
- while(iter.hasNext()) {
- tmp.add(iter.next());
- }
-
- Object[] a = tmp.toArray();
-
- Arrays.sort(a, sorter/*.getComparator()*/);
-
- m_iter = Arrays.asList(a).iterator();
-
-// TreeSet set = new TreeSet(sorter);
-//
-// while (iter.hasNext()) {
-// set.add(iter.next());
-// }
-//
-// m_iter = set.iterator();
+ m_context = context;
+ m_src = iter;
+ m_sorter = sorter;
- }
+ }
- public boolean hasNext() {
- return m_iter.hasNext();
- }
-
- public Object next() {
- return m_iter.next();
- }
-
- public void remove() {
- m_iter.remove();
- }
+ private void sort() {
+ if (!m_doneSort) {
+ // materialize the objects to be sorted.
+ LinkedList tmp = new LinkedList();
+
+ while (m_src.hasNext()) {
+ tmp.add(m_src.next());
+ }
+
+ Object[] a = tmp.toArray();
+
+ Arrays.sort(a, m_sorter/* .getComparator() */);
+
+ m_iter = Arrays.asList(a).iterator();
+ m_doneSort = true;
+ }
+ }
+
+ public boolean hasNext() {
+ return m_iter.hasNext();
+ }
+
+ public Object next() {
+ if (hasNext())
+ return m_iter.next();
+ else
+ throw new NoSuchElementException();
+ }
+
+ public void remove() {
+ m_iter.remove();
+ }
}
\ No newline at end of file
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <ble...@us...> - 2010-09-30 18:18:10
|
Revision: 3702
http://bigdata.svn.sourceforge.net/bigdata/?rev=3702&view=rev
Author: blevine218
Date: 2010-09-30 18:18:03 +0000 (Thu, 30 Sep 2010)
Log Message:
-----------
added clover profile
Modified Paths:
--------------
branches/maven_scaleout/bigdata-integ/pom.xml
Modified: branches/maven_scaleout/bigdata-integ/pom.xml
===================================================================
--- branches/maven_scaleout/bigdata-integ/pom.xml 2010-09-30 17:00:47 UTC (rev 3701)
+++ branches/maven_scaleout/bigdata-integ/pom.xml 2010-09-30 18:18:03 UTC (rev 3702)
@@ -238,6 +238,23 @@
</plugin>
</plugins>
</reporting>
+
+ <profiles>
+ <profile>
+ <id>bigdata-clover</id>
+ <activation>
+ <activeByDefault>false</activeByDefault>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>com.atlassian.maven.plugins</groupId>
+ <artifactId>maven-clover2-plugin</artifactId>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
</project>
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-09-30 17:00:57
|
Revision: 3701
http://bigdata.svn.sourceforge.net/bigdata/?rev=3701&view=rev
Author: thompsonbry
Date: 2010-09-30 17:00:47 +0000 (Thu, 30 Sep 2010)
Log Message:
-----------
Added some SVN excludes for the generated artifacts.
Property Changed:
----------------
branches/QUADS_QUERY_BRANCH/
Property changes on: branches/QUADS_QUERY_BRANCH
___________________________________________________________________
Modified: svn:ignore
- ant-build
src
bin
bigdata*.jar
ant-release
standalone
test*
countersfinal.xml
events.jnl
.settings
*.jnl
TestInsertRate.out
SYSTAP-BBT-result.txt
U10load+query
*.hprof
com.bigdata.cache.TestHardReferenceQueueWithBatchingUpdates.exp.csv
commit-log.txt
eventLog
dist
bigdata-test
com.bigdata.rdf.stress.LoadClosureAndQueryTest.*.csv
+ ant-build
src
bin
bigdata*.jar
ant-release
standalone
test*
countersfinal.xml
events.jnl
.settings
*.jnl
TestInsertRate.out
SYSTAP-BBT-result.txt
U10load+query
*.hprof
com.bigdata.cache.TestHardReferenceQueueWithBatchingUpdates.exp.csv
commit-log.txt
eventLog
dist
bigdata-test
com.bigdata.rdf.stress.LoadClosureAndQueryTest.*.csv
DIST.bigdata-*.tgz
REL.bigdata-*.tgz
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2010-09-30 16:58:52
|
Revision: 3700
http://bigdata.svn.sourceforge.net/bigdata/?rev=3700&view=rev
Author: thompsonbry
Date: 2010-09-30 16:58:46 +0000 (Thu, 30 Sep 2010)
Log Message:
-----------
Modified the QueryEngine to have a concept of its RMI "proxy" and modified the FederatedQueryEngine to use a real RMI proxy for itself as the queryController field to be sent around with the query.
Fixed build.xml. It was not staging out the ctc-striterators.
Made R.KeyOrder Serializable.
However, we still can not unit test the federated query engine because the test classes (R, KeyOrder) are not being packaged with the deployment....
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/ap/R.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/ap/TestPredicateAccessPath.java
branches/QUADS_QUERY_BRANCH/bigdata-jini/src/test/com/bigdata/bop/fed/jini/TestJiniFederatedQueryEngine.java
branches/QUADS_QUERY_BRANCH/build.xml
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java 2010-09-30 16:14:43 UTC (rev 3699)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java 2010-09-30 16:58:46 UTC (rev 3700)
@@ -271,6 +271,16 @@
}
/**
+ * The RMI proxy for this {@link QueryEngine} when used as a query controller.
+ * The default implementation returns <i>this</i>.
+ */
+ public IQueryClient getProxy() {
+
+ return this;
+
+ }
+
+ /**
* Return <code>true</code> iff running against an
* {@link IBigdataFederation}.
*/
@@ -655,7 +665,7 @@
throw new IllegalArgumentException();
final RunningQuery runningQuery = newRunningQuery(this, queryId,
- true/* controller */, this/* clientProxy */, query);
+ true/* controller */, getProxy()/*queryController*/, query);
final long timeout = query.getProperty(BOp.Annotations.TIMEOUT,
BOp.Annotations.DEFAULT_TIMEOUT);
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java 2010-09-30 16:14:43 UTC (rev 3699)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/fed/FederatedQueryEngine.java 2010-09-30 16:58:46 UTC (rev 3700)
@@ -41,8 +41,8 @@
import org.apache.log4j.Logger;
+import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.PipelineOp;
-import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.engine.IChunkMessage;
import com.bigdata.bop.engine.IQueryClient;
import com.bigdata.bop.engine.IQueryDecl;
@@ -55,6 +55,7 @@
import com.bigdata.service.IDataService;
import com.bigdata.service.ManagedResourceService;
import com.bigdata.service.ResourceService;
+import com.bigdata.service.jini.JiniFederation;
import com.bigdata.util.InnerCause;
/**
@@ -91,6 +92,11 @@
private final ManagedResourceService resourceService;
/**
+ * The proxy for this query engine when used as a query controller.
+ */
+ private final IQueryClient clientProxy;
+
+ /**
* A queue of {@link IChunkMessage}s which needs to have their data
* materialized so an operator can consume those data on this node.
*/
@@ -126,7 +132,19 @@
}
+ /**
+ * Overridden to return an RMI proxy for this {@link FederatedQueryEngine}.
+ * <p>
+ * {@inheritDoc}
+ */
@Override
+ public IQueryClient getProxy() {
+
+ return clientProxy;
+
+ }
+
+ @Override
final public boolean isScaleOut() {
return true;
@@ -196,6 +214,9 @@
this.resourceService = resourceService;
+ // the proxy for this query engine when used as a query controller.
+ this.clientProxy = (IQueryClient) ((JiniFederation<?>)fed).getProxy(this, false/*enableDGC*/);
+
}
/**
@@ -475,7 +496,7 @@
final PipelineOp query) {
return new FederatedRunningQuery(this, queryId, controller,
- clientProxy, query);
+ this.clientProxy, query);
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/ap/R.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/ap/R.java 2010-09-30 16:14:43 UTC (rev 3699)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/ap/R.java 2010-09-30 16:58:46 UTC (rev 3700)
@@ -27,6 +27,7 @@
package com.bigdata.bop.ap;
+import java.io.Serializable;
import java.util.Comparator;
import java.util.HashSet;
import java.util.Iterator;
@@ -64,9 +65,11 @@
/**
* Metadata about the index orders for this relation.
*/
- private static class KeyOrder extends AbstractKeyOrder<E> {
+ public static class KeyOrder extends AbstractKeyOrder<E> implements Serializable {
- public Comparator<E> getComparator() {
+ private static final long serialVersionUID = 1L;
+
+ public Comparator<E> getComparator() {
return new EComparator();
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/ap/TestPredicateAccessPath.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/ap/TestPredicateAccessPath.java 2010-09-30 16:14:43 UTC (rev 3699)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/ap/TestPredicateAccessPath.java 2010-09-30 16:58:46 UTC (rev 3700)
@@ -48,6 +48,7 @@
import com.bigdata.bop.engine.BOpStats;
import com.bigdata.bop.engine.MockRunningQuery;
import com.bigdata.btree.ITuple;
+import com.bigdata.io.SerializerUtil;
import com.bigdata.journal.BufferMode;
import com.bigdata.journal.ITx;
import com.bigdata.journal.Journal;
@@ -164,6 +165,12 @@
}
+ public void test_keyOrderSerializable() {
+
+ SerializerUtil.serialize(R.primaryKeyOrder);
+
+ }
+
/**
* Using a predicate with nothing bound, verify that we get the
* right range count on the relation and that we read the correct elements
Modified: branches/QUADS_QUERY_BRANCH/bigdata-jini/src/test/com/bigdata/bop/fed/jini/TestJiniFederatedQueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-jini/src/test/com/bigdata/bop/fed/jini/TestJiniFederatedQueryEngine.java 2010-09-30 16:14:43 UTC (rev 3699)
+++ branches/QUADS_QUERY_BRANCH/bigdata-jini/src/test/com/bigdata/bop/fed/jini/TestJiniFederatedQueryEngine.java 2010-09-30 16:58:46 UTC (rev 3700)
@@ -217,8 +217,8 @@
super.setUp();
-// dataService0 = fed.getDataService(dataServices[0]);
-// dataService1 = fed.getDataService(dataServices[1]);
+ dataService0 = fed.getDataService(dataServices[0]);
+ dataService1 = fed.getDataService(dataServices[1]);
// {
//
// // @todo need to wait for the dataService to be running.
Modified: branches/QUADS_QUERY_BRANCH/build.xml
===================================================================
--- branches/QUADS_QUERY_BRANCH/build.xml 2010-09-30 16:14:43 UTC (rev 3699)
+++ branches/QUADS_QUERY_BRANCH/build.xml 2010-09-30 16:58:46 UTC (rev 3700)
@@ -1124,6 +1124,9 @@
<copy toDir="${build.dir}/bigdata-sails/src">
<fileset dir="${bigdata.dir}/bigdata-sails/src" />
</copy>
+ <copy toDir="${build.dir}/ctc-striterators/src">
+ <fileset dir="${bigdata.dir}/ctc-striterators/src" />
+ </copy>
<copy toDir="${build.dir}/lgpl-utils/src">
<fileset dir="${bigdata.dir}/lgpl-utils/src" />
</copy>
@@ -1194,6 +1197,7 @@
<include name="bigdata-jini/src/**" />
<include name="bigdata-rdf/src/**" />
<include name="bigdata-sails/src/**" />
+ <include name="ctc-striterators/src/**" />
<include name="lgpl-utils/src/**" />
<include name="bigdata/lib/**" />
<include name="bigdata-jini/lib/**" />
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
4 messages has been excluded from this view by a project administrator.
