Menu
▾
▴
bigdata-commit — Commit traffic.
This list is closed, nobody may subscribe to it.
| 2010 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(139) |
Aug
(94) |
Sep
(232) |
Oct
(143) |
Nov
(138) |
Dec
(55) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2011 |
Jan
(127) |
Feb
(90) |
Mar
(101) |
Apr
(74) |
May
(148) |
Jun
(241) |
Jul
(169) |
Aug
(121) |
Sep
(157) |
Oct
(199) |
Nov
(281) |
Dec
(75) |
| 2012 |
Jan
(107) |
Feb
(122) |
Mar
(184) |
Apr
(73) |
May
(14) |
Jun
(49) |
Jul
(26) |
Aug
(103) |
Sep
(133) |
Oct
(61) |
Nov
(51) |
Dec
(55) |
| 2013 |
Jan
(59) |
Feb
(72) |
Mar
(99) |
Apr
(62) |
May
(92) |
Jun
(19) |
Jul
(31) |
Aug
(138) |
Sep
(47) |
Oct
(83) |
Nov
(95) |
Dec
(111) |
| 2014 |
Jan
(125) |
Feb
(60) |
Mar
(119) |
Apr
(136) |
May
(270) |
Jun
(83) |
Jul
(88) |
Aug
(30) |
Sep
(47) |
Oct
(27) |
Nov
(23) |
Dec
|
| 2015 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(3) |
Oct
|
Nov
|
Dec
|
| 2016 |
Jan
|
Feb
|
Mar
(4) |
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: <tho...@us...> - 2011-01-14 18:39:50
|
Revision: 4100
http://bigdata.svn.sourceforge.net/bigdata/?rev=4100&view=rev
Author: thompsonbry
Date: 2011-01-14 18:39:44 +0000 (Fri, 14 Jan 2011)
Log Message:
-----------
javadoc edit
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/striterator/AbstractChunkedResolverator.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/striterator/AbstractChunkedResolverator.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/striterator/AbstractChunkedResolverator.java 2011-01-14 17:59:00 UTC (rev 4099)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/striterator/AbstractChunkedResolverator.java 2011-01-14 18:39:44 UTC (rev 4100)
@@ -88,7 +88,7 @@
* @param state
* Application specified state (optional).
* @param src
- * The source iterator (will be closed exhausted or when this
+ * The source iterator (will be closed or exhausted when this
* iterator is closed).
* @param buffer
* The {@link BlockingBuffer} on which the resolved chunks will
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-14 17:59:06
|
Revision: 4099
http://bigdata.svn.sourceforge.net/bigdata/?rev=4099&view=rev
Author: thompsonbry
Date: 2011-01-14 17:59:00 +0000 (Fri, 14 Jan 2011)
Log Message:
-----------
Changed the DEFAULT_CHUNK_CAPACITY from 1000 to 100. This is used by things like the BigdataBindingSetResolverator, but it is no longer used by the query plans themselves (the value is not used to annotate bops).
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/AbstractResource.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/AbstractResource.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/AbstractResource.java 2011-01-14 14:51:28 UTC (rev 4098)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/relation/AbstractResource.java 2011-01-14 17:59:00 UTC (rev 4099)
@@ -260,7 +260,7 @@
*
* @deprecated by {@link BOp} annotations.
*/
- String DEFAULT_CHUNK_CAPACITY = "1000"; // was 100
+ String DEFAULT_CHUNK_CAPACITY = "100"; // was 100
/**
* The timeout in milliseconds that the {@link BlockingBuffer} will wait
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-14 14:51:34
|
Revision: 4098
http://bigdata.svn.sourceforge.net/bigdata/?rev=4098&view=rev
Author: thompsonbry
Date: 2011-01-14 14:51:28 +0000 (Fri, 14 Jan 2011)
Log Message:
-----------
Modified SubqueryOp to ignore an InterruptedException or BufferClosedException in the subquery. These exceptions are normal if the subquery employs a slice and should not cause the parent query to terminate.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/AbstractRunningQuery.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine_Slice.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java 2011-01-14 14:47:56 UTC (rev 4097)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java 2011-01-14 14:51:28 UTC (rev 4098)
@@ -42,8 +42,10 @@
import com.bigdata.bop.engine.IRunningQuery;
import com.bigdata.bop.engine.LocalChunkMessage;
import com.bigdata.bop.engine.QueryEngine;
+import com.bigdata.relation.accesspath.BufferClosedException;
import com.bigdata.relation.accesspath.IAsynchronousIterator;
import com.bigdata.relation.accesspath.ThickAsynchronousIterator;
+import com.bigdata.util.InnerCause;
import com.bigdata.util.concurrent.LatchedExecutor;
/**
@@ -361,6 +363,7 @@
public IRunningQuery call() throws Exception {
IAsynchronousIterator<IBindingSet[]> subquerySolutionItr = null;
+ IRunningQuery runningQuery = null;
try {
final QueryEngine queryEngine = parentContext.getRunningQuery()
@@ -376,7 +379,7 @@
final UUID queryId = UUID.randomUUID();
// execute the subquery, passing in the source binding set.
- final IRunningQuery runningQuery = queryEngine
+ runningQuery = queryEngine
.eval(
queryId,
(PipelineOp) subQueryOp,
@@ -415,13 +418,25 @@
} catch (Throwable t) {
/*
- * If a subquery fails, then propagate the error to the
- * parent and rethrow the first cause error out of the
- * subquery.
+ * Note: SliceOp will cause other operators to be
+ * interrupted during normal evaluation but we do not want
+ * to terminate the parent query when this occurs.
*/
- throw new RuntimeException(ControllerTask.this.context
- .getRunningQuery().halt(t));
+ if (!InnerCause.isInnerCause(t, InterruptedException.class)
+ && !InnerCause.isInnerCause(t, BufferClosedException.class)) {
+ /*
+ * If a subquery fails, then propagate the error to the
+ * parent and rethrow the first cause error out of the
+ * subquery.
+ */
+ throw new RuntimeException(ControllerTask.this.context
+ .getRunningQuery().halt(t));
+
+ }
+
+ return runningQuery;
+
} finally {
if (subquerySolutionItr != null)
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/AbstractRunningQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/AbstractRunningQuery.java 2011-01-14 14:47:56 UTC (rev 4097)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/AbstractRunningQuery.java 2011-01-14 14:51:28 UTC (rev 4098)
@@ -50,6 +50,7 @@
import com.bigdata.bop.solutions.SliceOp;
import com.bigdata.journal.IIndexManager;
import com.bigdata.journal.ITx;
+import com.bigdata.relation.accesspath.BufferClosedException;
import com.bigdata.relation.accesspath.IAsynchronousIterator;
import com.bigdata.relation.accesspath.IBlockingBuffer;
import com.bigdata.service.IBigdataFederation;
@@ -798,7 +799,13 @@
try {
- if (!InnerCause.isInnerCause(t, InterruptedException.class))
+ /*
+ * Note: SliceOp will cause other operators to be interrupted
+ * during normal evaluation so it is not useful to log an
+ * InterruptedException @ ERROR.
+ */
+ if (!InnerCause.isInnerCause(t, InterruptedException.class)
+ && !InnerCause.isInnerCause(t, BufferClosedException.class))
log.error(toString(), t);
try {
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java 2011-01-14 14:47:56 UTC (rev 4097)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java 2011-01-14 14:51:28 UTC (rev 4098)
@@ -1543,12 +1543,6 @@
* which do not succeed on the optional join are forwarded to the
* {@link SliceOp} which is the target specified by the
* {@link PipelineOp.Annotations#ALT_SINK_REF}.
- *
- * @todo Write unit test for optional join groups. Here the goal is to
- * verify that intermediate results may skip more than one join. This
- * was a problem for the old query evaluation approach since binding
- * sets had to cascade through the query one join at a time. However,
- * the new query engine design should handle this case.
*/
public void test_query_join2_optionals() throws Exception {
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine_Slice.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine_Slice.java 2011-01-14 14:47:56 UTC (rev 4097)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine_Slice.java 2011-01-14 14:51:28 UTC (rev 4098)
@@ -123,6 +123,14 @@
}
+// public void testStressThreadSafe() throws Exception {
+//
+// for(int i=0; i<1000; i++) {
+// test_slice_threadSafe();
+// }
+//
+// }
+
public void test_slice_threadSafe() throws Exception {
final long timeout = 10000; // ms
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-14 14:48:02
|
Revision: 4097
http://bigdata.svn.sourceforge.net/bigdata/?rev=4097&view=rev
Author: thompsonbry
Date: 2011-01-14 14:47:56 +0000 (Fri, 14 Jan 2011)
Log Message:
-----------
Fixed unit test for SubqueryOp (the OPTIONAL annotation needs to be set explicitly for these unit tests now that the default for that annotation is false).
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestSubqueryOp.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestSubqueryOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestSubqueryOp.java 2011-01-14 14:26:21 UTC (rev 4096)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestSubqueryOp.java 2011-01-14 14:47:56 UTC (rev 4097)
@@ -345,12 +345,12 @@
final PipelineOp joinGroup1Op = new SubqueryOp(new BOp[]{join1Op},
new NV(Predicate.Annotations.BOP_ID, joinGroup1),//
// new NV(PipelineOp.Annotations.CONDITIONAL_GROUP, joinGroup1),//
- new NV(SubqueryOp.Annotations.SUBQUERY, subQuery)//
+ new NV(SubqueryOp.Annotations.SUBQUERY, subQuery),//
// , new NV(BOp.Annotations.CONTROLLER,true)//
// new NV(BOp.Annotations.EVALUATION_CONTEXT,
// BOpEvaluationContext.CONTROLLER)//
-// // join is optional.
-// new NV(PipelineJoin.Annotations.OPTIONAL, true),//
+ // join is optional.
+ new NV(SubqueryOp.Annotations.OPTIONAL, true)//
// // optional target is the same as the default target.
// new NV(PipelineOp.Annotations.ALT_SINK_REF, sliceId)
);
@@ -610,12 +610,12 @@
final PipelineOp joinGroup1Op = new SubqueryOp(new BOp[]{join1Op},
new NV(Predicate.Annotations.BOP_ID, joinGroup1),//
// new NV(PipelineOp.Annotations.CONDITIONAL_GROUP, joinGroup1),//
- new NV(SubqueryOp.Annotations.SUBQUERY, subQuery)//
+ new NV(SubqueryOp.Annotations.SUBQUERY, subQuery),//
// new NV(BOp.Annotations.CONTROLLER,true)//
// new NV(BOp.Annotations.EVALUATION_CONTEXT,
// BOpEvaluationContext.CONTROLLER)//
-// // join is optional.
-// new NV(PipelineJoin.Annotations.OPTIONAL, true),//
+ // join is optional.
+ new NV(SubqueryOp.Annotations.OPTIONAL, true)//
// // optional target is the same as the default target.
// new NV(PipelineOp.Annotations.ALT_SINK_REF, sliceId)
);
@@ -871,12 +871,12 @@
final PipelineOp joinGroup1Op = new SubqueryOp(new BOp[]{condOp},
new NV(Predicate.Annotations.BOP_ID, joinGroup1),//
// new NV(PipelineOp.Annotations.CONDITIONAL_GROUP, joinGroup1),//
- new NV(SubqueryOp.Annotations.SUBQUERY, subQuery)//
+ new NV(SubqueryOp.Annotations.SUBQUERY, subQuery),//
// new NV(BOp.Annotations.CONTROLLER,true)//
// new NV(BOp.Annotations.EVALUATION_CONTEXT,
// BOpEvaluationContext.CONTROLLER)//
-// // join is optional.
-// new NV(PipelineJoin.Annotations.OPTIONAL, true),//
+ // join is optional.
+ new NV(SubqueryOp.Annotations.OPTIONAL, true)//
// // optional target is the same as the default target.
// new NV(PipelineOp.Annotations.ALT_SINK_REF, sliceId)
);
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-14 14:26:35
|
Revision: 4096
http://bigdata.svn.sourceforge.net/bigdata/?rev=4096&view=rev
Author: thompsonbry
Date: 2011-01-14 14:26:21 +0000 (Fri, 14 Jan 2011)
Log Message:
-----------
javadoc update and rename of the test suite for the SubqueryOp.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestAll.java
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestSubqueryOp.java
Removed Paths:
-------------
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestOptionalJoinGroup.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java 2011-01-13 22:13:47 UTC (rev 4095)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java 2011-01-14 14:26:21 UTC (rev 4096)
@@ -49,9 +49,10 @@
/**
* For each binding set presented, this operator executes a subquery. Any
* solutions produced by the subquery are copied to the default sink. If no
- * solutions are produced, then the original binding set is copied to the
- * default sink (optional join semantics). Each subquery is run as a separate
- * query but will be cancelled if the parent query is cancelled.
+ * solutions are produced and {@link Annotations#OPTIONAL} is <code>true</code>,
+ * then the original binding set is copied to the default sink (optional join
+ * semantics). Each subquery is run as a separate query but will be cancelled if
+ * the parent query is cancelled.
*
* FIXME Parallel evaluation of subqueries is not implemented. What is the
* appropriate parallelism for this operator? More parallelism should reduce
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestAll.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestAll.java 2011-01-13 22:13:47 UTC (rev 4095)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestAll.java 2011-01-14 14:26:21 UTC (rev 4096)
@@ -73,7 +73,7 @@
// test STEPS
// suite.addTestSuite(TestUnion.class);
- suite.addTestSuite(TestOptionalJoinGroup.class);
+ suite.addTestSuite(TestSubqueryOp.class);
// @todo test STAR (transitive closure).
// suite.addTestSuite(TestStar.class);
Deleted: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestOptionalJoinGroup.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestOptionalJoinGroup.java 2011-01-13 22:13:47 UTC (rev 4095)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestOptionalJoinGroup.java 2011-01-14 14:26:21 UTC (rev 4096)
@@ -1,1132 +0,0 @@
-/**
-
-Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
-/*
- * Created on Aug 23, 2010
- */
-
-package com.bigdata.bop.controller;
-
-import java.util.Iterator;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
-import java.util.Properties;
-import java.util.UUID;
-import java.util.concurrent.atomic.AtomicInteger;
-
-import junit.framework.TestCase2;
-
-import com.bigdata.bop.BOp;
-import com.bigdata.bop.BOpEvaluationContext;
-import com.bigdata.bop.Constant;
-import com.bigdata.bop.IBindingSet;
-import com.bigdata.bop.IConstant;
-import com.bigdata.bop.IConstraint;
-import com.bigdata.bop.IVariable;
-import com.bigdata.bop.IVariableOrConstant;
-import com.bigdata.bop.NV;
-import com.bigdata.bop.PipelineOp;
-import com.bigdata.bop.Var;
-import com.bigdata.bop.IPredicate.Annotations;
-import com.bigdata.bop.ap.E;
-import com.bigdata.bop.ap.Predicate;
-import com.bigdata.bop.ap.R;
-import com.bigdata.bop.bindingSet.ArrayBindingSet;
-import com.bigdata.bop.bindingSet.HashBindingSet;
-import com.bigdata.bop.bset.ConditionalRoutingOp;
-import com.bigdata.bop.bset.StartOp;
-import com.bigdata.bop.constraint.NEConstant;
-import com.bigdata.bop.engine.BOpStats;
-import com.bigdata.bop.engine.IChunkMessage;
-import com.bigdata.bop.engine.IRunningQuery;
-import com.bigdata.bop.engine.LocalChunkMessage;
-import com.bigdata.bop.engine.QueryEngine;
-import com.bigdata.bop.join.PipelineJoin;
-import com.bigdata.bop.solutions.SliceOp;
-import com.bigdata.journal.BufferMode;
-import com.bigdata.journal.ITx;
-import com.bigdata.journal.Journal;
-import com.bigdata.relation.accesspath.IAsynchronousIterator;
-import com.bigdata.relation.accesspath.ThickAsynchronousIterator;
-import com.bigdata.striterator.ChunkedArrayIterator;
-import com.bigdata.striterator.Dechunkerator;
-import com.bigdata.striterator.ICloseableIterator;
-
-/**
- * Test suite for handling of optional join groups during query evaluation
- * against a local database instance.
- *
- * <pre>
- * -Dlog4j.configuration=bigdata/src/resources/logging/log4j.properties
- * </pre>
- *
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
- */
-public class TestOptionalJoinGroup extends TestCase2 {
-
- /**
- *
- */
- public TestOptionalJoinGroup() {
- }
-
- /**
- * @param name
- */
- public TestOptionalJoinGroup(String name) {
- super(name);
- }
-
- @Override
- public Properties getProperties() {
-
- final Properties p = new Properties(super.getProperties());
-
- p.setProperty(Journal.Options.BUFFER_MODE, BufferMode.Transient
- .toString());
-
- return p;
-
- }
-
- static private final String namespace = "ns";
- private Journal jnl;
- private QueryEngine queryEngine;
-
- public void setUp() throws Exception {
-
- jnl = new Journal(getProperties());
-
- loadData(jnl);
-
- queryEngine = new QueryEngine(jnl);
-
- queryEngine.init();
-
- }
-
- /**
- * Create and populate relation in the {@link #namespace}.
- */
- private void loadData(final Journal store) {
-
- // create the relation.
- final R rel = new R(store, namespace, ITx.UNISOLATED, new Properties());
- rel.create();
-
- // data to insert (in key order for convenience).
- final E[] a = {//
- new E("Paul", "Mary"),// [0]
- new E("Paul", "Brad"),// [1]
-
- new E("John", "Mary"),// [2]
- new E("John", "Brad"),// [3]
-
- new E("Mary", "Brad"),// [4]
-
- new E("Brad", "Fred"),// [5]
- new E("Brad", "Leon"),// [6]
- };
-
- // insert data (the records are not pre-sorted).
- rel.insert(new ChunkedArrayIterator<E>(a.length, a, null/* keyOrder */));
-
- // Do commit since not scale-out.
- store.commit();
-
- }
-
- public void tearDown() throws Exception {
-
- if (queryEngine != null) {
- queryEngine.shutdownNow();
- queryEngine = null;
- }
-
- if (jnl != null) {
- jnl.destroy();
- jnl = null;
- }
-
- }
-
- /**
- * Return an {@link IAsynchronousIterator} that will read a single,
- * empty {@link IBindingSet}.
- *
- * @param bindingSet
- * the binding set.
- */
- protected ThickAsynchronousIterator<IBindingSet[]> newBindingSetIterator(
- final IBindingSet bindingSet) {
-
- return new ThickAsynchronousIterator<IBindingSet[]>(
- new IBindingSet[][] { new IBindingSet[] { bindingSet } });
-
- }
-
- /**
- * Return an {@link IAsynchronousIterator} that will read a single, chunk
- * containing all of the specified {@link IBindingSet}s.
- *
- * @param bindingSets
- * the binding sets.
- */
- protected ThickAsynchronousIterator<IBindingSet[]> newBindingSetIterator(
- final IBindingSet[] bindingSets) {
-
- return new ThickAsynchronousIterator<IBindingSet[]>(
- new IBindingSet[][] { bindingSets });
-
- }
-
- /**
- * Return an {@link IAsynchronousIterator} that will read a single, chunk
- * containing all of the specified {@link IBindingSet}s.
- *
- * @param bindingSetChunks
- * the chunks of binding sets.
- */
- protected ThickAsynchronousIterator<IBindingSet[]> newBindingSetIterator(
- final IBindingSet[][] bindingSetChunks) {
-
- return new ThickAsynchronousIterator<IBindingSet[]>(bindingSetChunks);
-
- }
-
- /**
- * Unit test for optional join group. Three joins are used and target a
- * {@link SliceOp}. The 2nd and 3rd joins are embedded in an
- * {@link SubqueryOp}.
- * <P>
- * The optional join group takes the form:
- *
- * <pre>
- * (a b)
- * optional {
- * (b c)
- * (c d)
- * }
- * </pre>
- *
- * The (a b) tail will match everything in the knowledge base. The join
- * group takes us two hops out from ?b. There should be four solutions that
- * succeed the optional join group:
- *
- * <pre>
- * (paul mary brad fred)
- * (paul mary brad leon)
- * (john mary brad fred)
- * (john mary brad leon)
- * </pre>
- *
- * and five more that don't succeed the optional join group:
- *
- * <pre>
- * (paul brad) *
- * (john brad) *
- * (mary brad) *
- * (brad fred)
- * (brad leon)
- * </pre>
- *
- * In this cases marked with a <code>*</code>, ?c will become temporarily
- * bound to fred and leon (since brad knows fred and leon), but the (c d)
- * tail will fail since fred and leon don't know anyone else. At this point,
- * the ?c binding must be removed from the solution.
- */
- public void test_query_join2_optionals() throws Exception {
-
- // main query
- final int startId = 1; //
- final int joinId1 = 2; // : base join group.
- final int predId1 = 3; // (a b)
- final int joinGroup1 = 9;
- final int sliceId = 8; //
-
- // subquery
- final int joinId2 = 4; // : joinGroup1
- final int predId2 = 5; // (b c)
- final int joinId3 = 6; // : joinGroup1
- final int predId3 = 7; // (c d)
-
- final IVariable<?> a = Var.var("a");
- final IVariable<?> b = Var.var("b");
- final IVariable<?> c = Var.var("c");
- final IVariable<?> d = Var.var("d");
-
- final PipelineOp startOp = new StartOp(new BOp[] {},
- NV.asMap(new NV[] {//
- new NV(Predicate.Annotations.BOP_ID, startId),//
- new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.CONTROLLER),//
- }));
-
- final Predicate<?> pred1Op = new Predicate<E>(
- new IVariableOrConstant[] { a, b }, NV
- .asMap(new NV[] {//
- new NV(Predicate.Annotations.RELATION_NAME,
- new String[] { namespace }),//
- new NV(Predicate.Annotations.BOP_ID, predId1),//
- new NV(Annotations.TIMESTAMP, ITx.READ_COMMITTED),//
- }));
-
- final Predicate<?> pred2Op = new Predicate<E>(
- new IVariableOrConstant[] { b, c }, NV
- .asMap(new NV[] {//
- new NV(Predicate.Annotations.RELATION_NAME,
- new String[] { namespace }),//
- new NV(Predicate.Annotations.BOP_ID, predId2),//
- new NV(Annotations.TIMESTAMP, ITx.READ_COMMITTED),//
- }));
-
- final Predicate<?> pred3Op = new Predicate<E>(
- new IVariableOrConstant[] { c, d }, NV
- .asMap(new NV[] {//
- new NV(Predicate.Annotations.RELATION_NAME,
- new String[] { namespace }),//
- new NV(Predicate.Annotations.BOP_ID, predId3),//
- new NV(Annotations.TIMESTAMP, ITx.READ_COMMITTED),//
- }));
-
- final PipelineOp join1Op = new PipelineJoin<E>(//
- new BOp[]{startOp},//
- new NV(Predicate.Annotations.BOP_ID, joinId1),//
- new NV(PipelineJoin.Annotations.PREDICATE,pred1Op));
-
- final PipelineOp subQuery;
- {
- final PipelineOp join2Op = new PipelineJoin<E>(//
- new BOp[] { /*join1Op*/ },//
- new NV(Predicate.Annotations.BOP_ID, joinId2),//
-// new NV(PipelineOp.Annotations.CONDITIONAL_GROUP, joinGroup1),//
- new NV(PipelineJoin.Annotations.PREDICATE, pred2Op)//
-// // join is optional.
-// new NV(PipelineJoin.Annotations.OPTIONAL, true),//
-// // optional target is the same as the default target.
-// new NV(PipelineOp.Annotations.ALT_SINK_REF, sliceId)
- );
-
- final PipelineOp join3Op = new PipelineJoin<E>(//
- new BOp[] { join2Op },//
- new NV(Predicate.Annotations.BOP_ID, joinId3),//
-// new NV(PipelineOp.Annotations.CONDITIONAL_GROUP, joinGroup1),//
- new NV(PipelineJoin.Annotations.PREDICATE, pred3Op)//
-// // join is optional.
-// new NV(PipelineJoin.Annotations.OPTIONAL, true),//
-// // optional target is the same as the default target.
-// new NV(PipelineOp.Annotations.ALT_SINK_REF, sliceId)
- );
- subQuery = join3Op;
- }
-
- final PipelineOp joinGroup1Op = new SubqueryOp(new BOp[]{join1Op},
- new NV(Predicate.Annotations.BOP_ID, joinGroup1),//
-// new NV(PipelineOp.Annotations.CONDITIONAL_GROUP, joinGroup1),//
- new NV(SubqueryOp.Annotations.SUBQUERY, subQuery)//
-// , new NV(BOp.Annotations.CONTROLLER,true)//
-// new NV(BOp.Annotations.EVALUATION_CONTEXT,
-// BOpEvaluationContext.CONTROLLER)//
-// // join is optional.
-// new NV(PipelineJoin.Annotations.OPTIONAL, true),//
-// // optional target is the same as the default target.
-// new NV(PipelineOp.Annotations.ALT_SINK_REF, sliceId)
- );
-
- final PipelineOp sliceOp = new SliceOp(//
- new BOp[]{joinGroup1Op},
- NV.asMap(new NV[] {//
- new NV(BOp.Annotations.BOP_ID, sliceId),//
- new NV(BOp.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.CONTROLLER),//
- }));
-
- final PipelineOp query = sliceOp;
-
- // start the query.
- final UUID queryId = UUID.randomUUID();
- final IChunkMessage<IBindingSet> initialChunkMessage;
- {
-
- final IBindingSet initialBindings = new HashBindingSet();
-
-// initialBindings.set(Var.var("x"), new Constant<String>("Mary"));
-
- initialChunkMessage = new LocalChunkMessage<IBindingSet>(queryEngine,
- queryId, startId,//
- -1, // partitionId
- newBindingSetIterator(initialBindings));
- }
- final IRunningQuery runningQuery = queryEngine.eval(queryId, query,
- initialChunkMessage);
-
- // verify solutions.
- {
-
- // the expected solutions.
- final IBindingSet[] expected = new IBindingSet[] {//
- // four solutions where the optional join succeeds.
- new ArrayBindingSet(//
- new IVariable[] { a, b, c, d },//
- new IConstant[] { new Constant<String>("Paul"),
- new Constant<String>("Mary"),
- new Constant<String>("Brad"),
- new Constant<String>("Fred") }//
- ),
- new ArrayBindingSet(//
- new IVariable[] { a, b, c, d },//
- new IConstant[] { new Constant<String>("Paul"),
- new Constant<String>("Mary"),
- new Constant<String>("Brad"),
- new Constant<String>("Leon") }//
- ),
- new ArrayBindingSet(//
- new IVariable[] { a, b, c, d },//
- new IConstant[] { new Constant<String>("John"),
- new Constant<String>("Mary"),
- new Constant<String>("Brad"),
- new Constant<String>("Fred") }//
- ),
- new ArrayBindingSet(//
- new IVariable[] { a, b, c, d },//
- new IConstant[] { new Constant<String>("John"),
- new Constant<String>("Mary"),
- new Constant<String>("Brad"),
- new Constant<String>("Leon") }//
- ),
- // plus anything we read from the first access path which did not
- // pass the optional join
- new ArrayBindingSet(//
- new IVariable[] { a, b },//
- new IConstant[] { new Constant<String>("Paul"),
- new Constant<String>("Brad") }//
- ),
- new ArrayBindingSet(//
- new IVariable[] { a, b },//
- new IConstant[] { new Constant<String>("John"),
- new Constant<String>("Brad") }//
- ),
- new ArrayBindingSet(//
- new IVariable[] { a, b },//
- new IConstant[] { new Constant<String>("Mary"),
- new Constant<String>("Brad") }//
- ),
- new ArrayBindingSet(//
- new IVariable[] { a, b },//
- new IConstant[] { new Constant<String>("Brad"),
- new Constant<String>("Fred") }//
- ),
- new ArrayBindingSet(//
- new IVariable[] { a, b },//
- new IConstant[] { new Constant<String>("Brad"),
- new Constant<String>("Leon") }//
- )
- };
-
- /*
- * junit.framework.AssertionFailedError: Iterator will deliver too
- * many objects: reminder(3)=[{ a=John, b=Brad }, { a=Mary, b=Brad
- * }, { a=Paul, b=Brad }].
- */
- assertSameSolutionsAnyOrder(expected,
- new Dechunkerator<IBindingSet>(runningQuery.iterator()));
-
- }
-
- // Wait until the query is done.
- runningQuery.get();
- final Map<Integer, BOpStats> statsMap = runningQuery.getStats();
- {
- // validate the stats map.
- assertNotNull(statsMap);
- assertEquals(4, statsMap.size());
- if (log.isInfoEnabled())
- log.info(statsMap.toString());
- }
-
- }
-
- /**
- * Unit test for optional join group with a filter. Three joins are used and
- * target a {@link SliceOp}. The 2nd and 3rd joins are embedded in an
- * optional join group. The optional join group contains a filter.
- * <p>
- * The optional join group takes the form:
- *
- * <pre>
- * (a b)
- * optional {
- * (b c)
- * (c d)
- * filter(d != Leon)
- * }
- * </pre>
- *
- * The (a b) tail will match everything in the knowledge base. The join
- * group takes us two hops out from ?b. There should be two solutions that
- * succeed the optional join group:
- *
- * <pre>
- * (paul mary brad fred)
- * (john mary brad fred)
- * </pre>
- *
- * and five more that don't succeed the optional join group:
- *
- * <pre>
- * (paul brad) *
- * (john brad) *
- * (mary brad) *
- * (brad fred)
- * (brad leon)
- * </pre>
- *
- * In the cases marked with a <code>*</code>, ?c will become temporarily
- * bound to fred and leon (since brad knows fred and leon), but the (c d)
- * tail will fail since fred and leon don't know anyone else. At this point,
- * the ?c binding must be removed from the solution.
- * <p>
- * The filter (d != Leon) will prune the two solutions:
- *
- * <pre>
- * (paul mary brad leon)
- * (john mary brad leon)
- * </pre>
- *
- * since ?d is bound to Leon in those cases.
- */
- public void test_query_optionals_filter() throws Exception {
-
- // main query
- final int startId = 1;
- final int joinId1 = 2; //
- final int predId1 = 3; // (a,b)
- final int joinGroup1 = 9;
- final int sliceId = 8;
-
- // subquery
- final int joinId2 = 4; // : group1
- final int predId2 = 5; // (b,c)
- final int joinId3 = 6; // : group1
- final int predId3 = 7; // (c,d)
-
-
- final IVariable<?> a = Var.var("a");
- final IVariable<?> b = Var.var("b");
- final IVariable<?> c = Var.var("c");
- final IVariable<?> d = Var.var("d");
-
- final PipelineOp startOp = new StartOp(new BOp[] {},
- NV.asMap(new NV[] {//
- new NV(Predicate.Annotations.BOP_ID, startId),//
- new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.CONTROLLER),//
- }));
-
- final Predicate<?> pred1Op = new Predicate<E>(
- new IVariableOrConstant[] { a, b }, NV
- .asMap(new NV[] {//
- new NV(Predicate.Annotations.RELATION_NAME,
- new String[] { namespace }),//
- new NV(Predicate.Annotations.BOP_ID, predId1),//
- new NV(Annotations.TIMESTAMP, ITx.READ_COMMITTED),//
- }));
-
- final Predicate<?> pred2Op = new Predicate<E>(
- new IVariableOrConstant[] { b, c }, NV
- .asMap(new NV[] {//
- new NV(Predicate.Annotations.RELATION_NAME,
- new String[] { namespace }),//
- new NV(Predicate.Annotations.BOP_ID, predId2),//
- new NV(Annotations.TIMESTAMP, ITx.READ_COMMITTED),//
- }));
-
- final Predicate<?> pred3Op = new Predicate<E>(
- new IVariableOrConstant[] { c, d }, NV
- .asMap(new NV[] {//
- new NV(Predicate.Annotations.RELATION_NAME,
- new String[] { namespace }),//
- new NV(Predicate.Annotations.BOP_ID, predId3),//
- new NV(Annotations.TIMESTAMP, ITx.READ_COMMITTED),//
- }));
-
- final PipelineOp join1Op = new PipelineJoin<E>(//
- new BOp[]{startOp},//
- new NV(Predicate.Annotations.BOP_ID, joinId1),//
- new NV(PipelineJoin.Annotations.PREDICATE,pred1Op));
-
- final PipelineOp subQuery;
- {
- final PipelineOp join2Op = new PipelineJoin<E>(//
- new BOp[] { /*join1Op*/ },//
- new NV(Predicate.Annotations.BOP_ID, joinId2),//
-// new NV(PipelineOp.Annotations.CONDITIONAL_GROUP, joinGroup1),//
- new NV(PipelineJoin.Annotations.PREDICATE, pred2Op)//
-// // join is optional.
-// new NV(PipelineJoin.Annotations.OPTIONAL, true),//
-// // optional target is the same as the default target.
-// new NV(PipelineOp.Annotations.ALT_SINK_REF, sliceId)
- );
-
- final PipelineOp join3Op = new PipelineJoin<E>(//
- new BOp[] { join2Op },//
- new NV(Predicate.Annotations.BOP_ID, joinId3),//
-// new NV(PipelineOp.Annotations.CONDITIONAL_GROUP, joinGroup1),//
- new NV(PipelineJoin.Annotations.PREDICATE, pred3Op),//
- // constraint d != Leon
- new NV(PipelineJoin.Annotations.CONSTRAINTS,
- new IConstraint[] { new NEConstant(d, new Constant<String>("Leon")) })
-// // join is optional.
-// new NV(PipelineJoin.Annotations.OPTIONAL, true),//
-// // optional target is the same as the default target.
-// new NV(PipelineOp.Annotations.ALT_SINK_REF, sliceId)
- );
-
- subQuery = join3Op;
- }
-
- final PipelineOp joinGroup1Op = new SubqueryOp(new BOp[]{join1Op},
- new NV(Predicate.Annotations.BOP_ID, joinGroup1),//
-// new NV(PipelineOp.Annotations.CONDITIONAL_GROUP, joinGroup1),//
- new NV(SubqueryOp.Annotations.SUBQUERY, subQuery)//
-// new NV(BOp.Annotations.CONTROLLER,true)//
-// new NV(BOp.Annotations.EVALUATION_CONTEXT,
-// BOpEvaluationContext.CONTROLLER)//
-// // join is optional.
-// new NV(PipelineJoin.Annotations.OPTIONAL, true),//
-// // optional target is the same as the default target.
-// new NV(PipelineOp.Annotations.ALT_SINK_REF, sliceId)
- );
-
- final PipelineOp sliceOp = new SliceOp(//
- new BOp[]{joinGroup1Op},
- NV.asMap(new NV[] {//
- new NV(BOp.Annotations.BOP_ID, sliceId),//
- new NV(BOp.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.CONTROLLER),//
- }));
-
- final PipelineOp query = sliceOp;
-
- // start the query.
- final UUID queryId = UUID.randomUUID();
- final IChunkMessage<IBindingSet> initialChunkMessage;
- {
-
- final IBindingSet initialBindings = new HashBindingSet();
-
-// initialBindings.set(Var.var("x"), new Constant<String>("Mary"));
-
- initialChunkMessage = new LocalChunkMessage<IBindingSet>(queryEngine,
- queryId, startId,//
- -1, // partitionId
- newBindingSetIterator(initialBindings));
- }
- final IRunningQuery runningQuery = queryEngine.eval(queryId, query,
- initialChunkMessage);
-
- // verify solutions.
- {
-
- // the expected solutions.
- final IBindingSet[] expected = new IBindingSet[] {//
- // two solutions where the optional join succeeds.
- new ArrayBindingSet(//
- new IVariable[] { a, b, c, d },//
- new IConstant[] { new Constant<String>("Paul"),
- new Constant<String>("Mary"),
- new Constant<String>("Brad"),
- new Constant<String>("Fred") }//
- ),
- new ArrayBindingSet(//
- new IVariable[] { a, b, c, d },//
- new IConstant[] { new Constant<String>("John"),
- new Constant<String>("Mary"),
- new Constant<String>("Brad"),
- new Constant<String>("Fred") }//
- ),
- // plus anything we read from the first access path which did not
- // pass the optional join
- new ArrayBindingSet(//
- new IVariable[] { a, b },//
- new IConstant[] { new Constant<String>("Paul"),
- new Constant<String>("Brad") }//
- ),
- new ArrayBindingSet(//
- new IVariable[] { a, b },//
- new IConstant[] { new Constant<String>("John"),
- new Constant<String>("Brad") }//
- ),
- new ArrayBindingSet(//
- new IVariable[] { a, b },//
- new IConstant[] { new Constant<String>("Mary"),
- new Constant<String>("Brad") }//
- ),
- new ArrayBindingSet(//
- new IVariable[] { a, b },//
- new IConstant[] { new Constant<String>("Brad"),
- new Constant<String>("Fred") }//
- ),
- new ArrayBindingSet(//
- new IVariable[] { a, b },//
- new IConstant[] { new Constant<String>("Brad"),
- new Constant<String>("Leon") }//
- )
- };
-
- assertSameSolutionsAnyOrder(expected,
- new Dechunkerator<IBindingSet>(runningQuery.iterator()));
-
- }
-
- // Wait until the query is done.
- runningQuery.get();
- final Map<Integer, BOpStats> statsMap = runningQuery.getStats();
- {
- // validate the stats map.
- assertNotNull(statsMap);
- assertEquals(4, statsMap.size());
- if (log.isInfoEnabled())
- log.info(statsMap.toString());
- }
-
- }
-
- /**
- * Unit test for optional join group with a filter on a variable outside the
- * optional join group. Three joins are used and target a {@link SliceOp}.
- * The 2nd and 3rd joins are in embedded an {@link SubqueryOp}. The
- * optional join group contains a filter that uses a variable outside the
- * optional join group.
- * <P>
- * The query takes the form:
- *
- * <pre>
- * (a b)
- * optional {
- * (b c)
- * (c d)
- * filter(a != Paul)
- * }
- * </pre>
- *
- * The (a b) tail will match everything in the knowledge base. The join
- * group takes us two hops out from ?b. There should be two solutions that
- * succeed the optional join group:
- *
- * <pre>
- * (john mary brad fred)
- * (john mary brad leon)
- * </pre>
- *
- * and six more that don't succeed the optional join group:
- *
- * <pre>
- * (paul mary) *
- * (paul brad) *
- * (john brad)
- * (mary brad)
- * (brad fred)
- * (brad leon)
- * </pre>
- *
- * In the cases marked with a <code>*</code>, ?a is bound to Paul even
- * though there is a filter that specifically prohibits a = Paul. This is
- * because the filter is inside the optional join group, which means that
- * solutions can still include a = Paul, but the optional join group should
- * not run in that case.
- */
- public void test_query_optionals_filter2() throws Exception {
-
- // main query
- final int startId = 1;
- final int joinId1 = 2;
- final int predId1 = 3; // (a,b)
- final int condId = 4; // (a != Paul)
- final int joinGroup1 = 10;
- final int sliceId = 9;
-
- // subquery (iff condition is satisfied)
- final int joinId2 = 5; // : group1
- final int predId2 = 6; // (b,c)
- final int joinId3 = 7; // : group1
- final int predId3 = 8; // (c,d)
-
- final IVariable<?> a = Var.var("a");
- final IVariable<?> b = Var.var("b");
- final IVariable<?> c = Var.var("c");
- final IVariable<?> d = Var.var("d");
-
-// final Integer joinGroup1 = Integer.valueOf(1);
-
- /*
- * Not quite sure how to write this one. I think it probably goes
- * something like this:
- *
- * 1. startOp
- * 2. join1Op(a b)
- * 3. conditionalRoutingOp( if a = Paul then goto sliceOp )
- * 4. join2Op(b c)
- * 5. join3Op(c d)
- * 6. sliceOp
- */
-
- final PipelineOp startOp = new StartOp(new BOp[] {},
- NV.asMap(new NV[] {//
- new NV(Predicate.Annotations.BOP_ID, startId),//
- new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.CONTROLLER),//
- }));
-
- final Predicate<?> pred1Op = new Predicate<E>(
- new IVariableOrConstant[] { a, b }, NV
- .asMap(new NV[] {//
- new NV(Predicate.Annotations.RELATION_NAME,
- new String[] { namespace }),//
- new NV(Predicate.Annotations.BOP_ID, predId1),//
- new NV(Annotations.TIMESTAMP, ITx.READ_COMMITTED),//
- }));
-
- final Predicate<?> pred2Op = new Predicate<E>(
- new IVariableOrConstant[] { b, c }, NV
- .asMap(new NV[] {//
- new NV(Predicate.Annotations.RELATION_NAME,
- new String[] { namespace }),//
- new NV(Predicate.Annotations.BOP_ID, predId2),//
- new NV(Annotations.TIMESTAMP, ITx.READ_COMMITTED),//
- }));
-
- final Predicate<?> pred3Op = new Predicate<E>(
- new IVariableOrConstant[] { c, d }, NV
- .asMap(new NV[] {//
- new NV(Predicate.Annotations.RELATION_NAME,
- new String[] { namespace }),//
- new NV(Predicate.Annotations.BOP_ID, predId3),//
- new NV(Annotations.TIMESTAMP, ITx.READ_COMMITTED),//
- }));
-
- final PipelineOp join1Op = new PipelineJoin<E>(//
- new BOp[]{startOp},//
- new NV(Predicate.Annotations.BOP_ID, joinId1),//
- new NV(PipelineJoin.Annotations.PREDICATE,pred1Op));
-
- final IConstraint condition = new NEConstant(a, new Constant<String>("Paul"));
-
- final ConditionalRoutingOp condOp = new ConditionalRoutingOp(new BOp[]{join1Op},
- NV.asMap(new NV[]{//
- new NV(BOp.Annotations.BOP_ID,condId),
- new NV(PipelineOp.Annotations.SINK_REF, joinGroup1), // a != Paul
- new NV(PipelineOp.Annotations.ALT_SINK_REF, sliceId), // a == Paul
- new NV(ConditionalRoutingOp.Annotations.CONDITION, condition),
- }));
-
- final PipelineOp subQuery;
- {
- final PipelineOp join2Op = new PipelineJoin<E>(//
- new BOp[] { /*condOp*/ },//
- new NV(Predicate.Annotations.BOP_ID, joinId2),//
-// new NV(PipelineOp.Annotations.CONDITIONAL_GROUP, joinGroup1),//
- new NV(PipelineJoin.Annotations.PREDICATE, pred2Op)//
-// // join is optional.
-// new NV(PipelineJoin.Annotations.OPTIONAL, true),//
-// // optional target is the same as the default target.
-// new NV(PipelineOp.Annotations.ALT_SINK_REF, sliceId)
- );
-
- final PipelineOp join3Op = new PipelineJoin<E>(//
- new BOp[] { join2Op },//
- new NV(Predicate.Annotations.BOP_ID, joinId3),//
-// new NV(PipelineOp.Annotations.CONDITIONAL_GROUP, joinGroup1),//
- new NV(PipelineJoin.Annotations.PREDICATE, pred3Op)//
-// // join is optional.
-// new NV(PipelineJoin.Annotations.OPTIONAL, true),//
-// // optional target is the same as the default target.
-// new NV(PipelineOp.Annotations.ALT_SINK_REF, sliceId)
- );
- subQuery = join3Op;
- }
-
- final PipelineOp joinGroup1Op = new SubqueryOp(new BOp[]{condOp},
- new NV(Predicate.Annotations.BOP_ID, joinGroup1),//
-// new NV(PipelineOp.Annotations.CONDITIONAL_GROUP, joinGroup1),//
- new NV(SubqueryOp.Annotations.SUBQUERY, subQuery)//
-// new NV(BOp.Annotations.CONTROLLER,true)//
-// new NV(BOp.Annotations.EVALUATION_CONTEXT,
-// BOpEvaluationContext.CONTROLLER)//
-// // join is optional.
-// new NV(PipelineJoin.Annotations.OPTIONAL, true),//
-// // optional target is the same as the default target.
-// new NV(PipelineOp.Annotations.ALT_SINK_REF, sliceId)
- );
-
- final PipelineOp sliceOp = new SliceOp(//
- new BOp[]{joinGroup1Op},
- NV.asMap(new NV[] {//
- new NV(BOp.Annotations.BOP_ID, sliceId),//
- new NV(BOp.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.CONTROLLER),//
- }));
-
- final PipelineOp query = sliceOp;
-
- // start the query.
- final UUID queryId = UUID.randomUUID();
- final IChunkMessage<IBindingSet> initialChunkMessage;
- {
-
- final IBindingSet initialBindings = new HashBindingSet();
-
-// initialBindings.set(Var.var("x"), new Constant<String>("Mary"));
-
- initialChunkMessage = new LocalChunkMessage<IBindingSet>(queryEngine,
- queryId, startId,//
- -1, // partitionId
- newBindingSetIterator(initialBindings));
- }
- final IRunningQuery runningQuery = queryEngine.eval(queryId, query,
- initialChunkMessage);
-
- // verify solutions.
- {
-
- // the expected solutions.
- final IBindingSet[] expected = new IBindingSet[] {//
- // two solutions where the optional join succeeds.
- new ArrayBindingSet(//
- new IVariable[] { a, b, c, d },//
- new IConstant[] { new Constant<String>("John"),
- new Constant<String>("Mary"),
- new Constant<String>("Brad"),
- new Constant<String>("Leon") }//
- ),
- new ArrayBindingSet(//
- new IVariable[] { a, b, c, d },//
- new IConstant[] { new Constant<String>("John"),
- new Constant<String>("Mary"),
- new Constant<String>("Brad"),
- new Constant<String>("Fred") }//
- ),
- // plus anything we read from the first access path which did not
- // pass the optional join
- new ArrayBindingSet(//
- new IVariable[] { a, b },//
- new IConstant[] { new Constant<String>("Paul"),
- new Constant<String>("Mary") }//
- ),
- new ArrayBindingSet(//
- new IVariable[] { a, b },//
- new IConstant[] { new Constant<String>("Paul"),
- new Constant<String>("Brad") }//
- ),
- new ArrayBindingSet(//
- new IVariable[] { a, b },//
- new IConstant[] { new Constant<String>("John"),
- new Constant<String>("Brad") }//
- ),
- new ArrayBindingSet(//
- new IVariable[] { a, b },//
- new IConstant[] { new Constant<String>("Mary"),
- new Constant<String>("Brad") }//
- ),
- new ArrayBindingSet(//
- new IVariable[] { a, b },//
- new IConstant[] { new Constant<String>("Brad"),
- new Constant<String>("Fred") }//
- ),
- new ArrayBindingSet(//
- new IVariable[] { a, b },//
- new IConstant[] { new Constant<String>("Brad"),
- new Constant<String>("Leon") }//
- )
- };
-
- assertSameSolutionsAnyOrder(expected,
- new Dechunkerator<IBindingSet>(runningQuery.iterator()));
-
- }
-
- // Wait until the query is done.
- runningQuery.get();
- final Map<Integer, BOpStats> statsMap = runningQuery.getStats();
- {
- // validate the stats map.
- assertNotNull(statsMap);
- assertEquals(5, statsMap.size());
- if (log.isInfoEnabled())
- log.info(statsMap.toString());
- }
-
- }
-
- /**
- * Verify the expected solutions.
- *
- * @param expected
- * @param itr
- */
- static public void assertSameSolutions(final IBindingSet[] expected,
- final IAsynchronousIterator<IBindingSet[]> itr) {
- try {
- int n = 0;
- while (itr.hasNext()) {
- final IBindingSet[] e = itr.next();
- if (log.isInfoEnabled())
- log.info(n + " : chunkSize=" + e.length);
- for (int i = 0; i < e.length; i++) {
- if (log.isInfoEnabled())
- log.info(n + " : " + e[i]);
- if (n >= expected.length) {
- fail("Willing to deliver too many solutions: n=" + n
- + " : " + e[i]);
- }
- if (!expected[n].equals(e[i])) {
- fail("n=" + n + ", expected=" + expected[n]
- + ", actual=" + e[i]);
- }
- n++;
- }
- }
- assertEquals("Wrong number of solutions", expected.length, n);
- } finally {
- itr.close();
- }
- }
-
- /**
- * Verifies that the iterator visits the specified objects in some arbitrary
- * ordering and that the iterator is exhausted once all expected objects
- * have been visited. The implementation uses a selection without
- * replacement "pattern".
- * <p>
- * Note: If the objects being visited do not correctly implement hashCode()
- * and equals() then this can fail even if the desired objects would be
- * visited. When this happens, fix the implementation classes.
- */
- static public <T> void assertSameSolutionsAnyOrder(final T[] expected,
- final Iterator<T> actual) {
-
- assertSameSolutionsAnyOrder("", expected, actual);
-
- }
-
- /**
- * Verifies that the iterator visits the specified objects in some arbitrary
- * ordering and that the iterator is exhausted once all expected objects
- * have been visited. The implementation uses a selection without
- * replacement "pattern".
- * <p>
- * Note: If the objects being visited do not correctly implement hashCode()
- * and equals() then this can fail even if the desired objects would be
- * visited. When this happens, fix the implementation classes.
- */
- static public <T> void assertSameSolutionsAnyOrder(final String msg,
- final T[] expected, final Iterator<T> actual) {
-
- try {
-
- /*
- * Populate a map that we will use to realize the match and
- * selection without replacement logic. The map uses counters to
- * handle duplicate keys. This makes it possible to write tests in
- * which two or more binding sets which are "equal" appear.
- */
-
- final int nrange = expected.length;
-
- final java.util.Map<T, AtomicInteger> range = new java.util.LinkedHashMap<T, AtomicInteger>();
-
- for (int j = 0; j < nrange; j++) {
-
- AtomicInteger count = range.get(expected[j]);
-
- if (count == null) {
-
- count = new AtomicInteger();
-
- }
-
- range.put(expected[j], count);
-
- count.incrementAndGet();
-
- }
-
- // Do selection without replacement for the objects visited by
- // iterator.
-
- for (int j = 0; j < nrange; j++) {
-
- if (!actual.hasNext()) {
-
- fail(msg
- + ": Iterator exhausted while expecting more object(s)"
- + ": index=" + j);
-
- }
-
- final T actualObject = actual.next();
-
- if (log.isInfoEnabled())
- log.info("visting: " + actualObject);
-
- AtomicInteger counter = range.get(actualObject);
-
- if (counter == null || counter.get() == 0) {
-
- fail("Object not expected" + ": index=" + j + ", object="
- + actualObject);
-
- }
-
- counter.decrementAndGet();
-
- }
-
- if (actual.hasNext()) {
-
- final List<T> remainder = new LinkedList<T>();
-
- while(actual.hasNext()) {
- remainder.add(actual.next());
- }
-
- fail("Iterator will deliver too many objects: reminder("
- + remainder.size() + ")=" + remainder);
-
- }
-
- } finally {
-
- if (actual instanceof ICloseableIterator<?>) {
-
- ((ICloseableIterator<T>) actual).close();
-
- }
-
- }
-
- }
-
-}
Copied: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestSubqueryOp.java (from rev 4086, branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestOptionalJoinGroup.java)
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestSubqueryOp.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestSubqueryOp.java 2011-01-14 14:26:21 UTC (rev 4096)
@@ -0,0 +1,1132 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Aug 23, 2010
+ */
+
+package com.bigdata.bop.controller;
+
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import junit.framework.TestCase2;
+
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpEvaluationContext;
+import com.bigdata.bop.Constant;
+import com.bigdata.bop.IBindingSet;
+import com.bigdata.bop.IConstant;
+import com.bigdata.bop.IConstraint;
+import com.bigdata.bop.IVariable;
+import com.bigdata.bop.IVariableOrConstant;
+import com.bigdata.bop.NV;
+import com.bigdata.bop.PipelineOp;
+import com.bigdata.bop.Var;
+import com.bigdata.bop.IPredicate.Annotations;
+import com.bigdata.bop.ap.E;
+import com.bigdata.bop.ap.Predicate;
+import com.bigdata.bop.ap.R;
+import com.bigdata.bop.bindingSet.ArrayBindingSet;
+import com.bigdata.bop.bindingSet.HashBindingSet;
+import com.bigdata.bop.bset.ConditionalRoutingOp;
+import com.bigdata.bop.bset.StartOp;
+import com.bigdata.bop.constraint.NEConstant;
+import com.bigdata.bop.engine.BOpStats;
+import com.bigdata.bop.engine.IChunkMessage;
+import com.bigdata.bop.engine.IRunningQuery;
+import com.bigdata.bop.engine.LocalChunkMessage;
+import com.bigdata.bop.engine.QueryEngine;
+import com.bigdata.bop.join.PipelineJoin;
+import com.bigdata.bop.solutions.SliceOp;
+import com.bigdata.journal.BufferMode;
+import com.bigdata.journal.ITx;
+import com.bigdata.journal.Journal;
+import com.bigdata.relation.accesspath.IAsynchronousIterator;
+import com.bigdata.relation.accesspath.ThickAsynchronousIterator;
+import com.bigdata.striterator.ChunkedArrayIterator;
+import com.bigdata.striterator.Dechunkerator;
+import com.bigdata.striterator.ICloseableIterator;
+
+/**
+ * Test suite for handling of optional join groups during query evaluation
+ * against a local database instance.
+ *
+ * <pre>
+ * -Dlog4j.configuration=bigdata/src/resources/logging/log4j.properties
+ * </pre>
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class TestSubqueryOp extends TestCase2 {
+
+ /**
+ *
+ */
+ public TestSubqueryOp() {
+ }
+
+ /**
+ * @param name
+ */
+ public TestSubqueryOp(String name) {
+ super(name);
+ }
+
+ @Override
+ public Properties getProperties() {
+
+ final Properties p = new Properties(super.getProperties());
+
+ p.setProperty(Journal.Options.BUFFER_MODE, BufferMode.Transient
+ .toString());
+
+ return p;
+
+ }
+
+ static private final String namespace = "ns";
+ private Journal jnl;
+ private QueryEngine queryEngine;
+
+ public void setUp() throws Exception {
+
+ jnl = new Journal(getProperties());
+
+ loadData(jnl);
+
+ queryEngine = new QueryEngine(jnl);
+
+ queryEngine.init();
+
+ }
+
+ /**
+ * Create and populate relation in the {@link #namespace}.
+ */
+ private void loadData(final Journal store) {
+
+ // create the relation.
+ final R rel = new R(store, namespace, ITx.UNISOLATED, new Properties());
+ rel.create();
+
+ // data to insert (in key order for convenience).
+ final E[] a = {//
+ new E("Paul", "Mary"),// [0]
+ new E("Paul", "Brad"),// [1]
+
+ new E("John", "Mary"),// [2]
+ new E("John", "Brad"),// [3]
+
+ new E("Mary", "Brad"),// [4]
+
+ new E("Brad", "Fred"),// [5]
+ new E("Brad", "Leon"),// [6]
+ };
+
+ // insert data (the records are not pre-sorted).
+ rel.insert(new ChunkedArrayIterator<E>(a.length, a, null/* keyOrder */));
+
+ // Do commit since not scale-out.
+ store.commit();
+
+ }
+
+ public void tearDown() throws Exception {
+
+ if (queryEngine != null) {
+ queryEngine.shutdownNow();
+ queryEngine = null;
+ }
+
+ if (jnl != null) {
+ jnl.destroy();
+ jnl = null;
+ }
+
+ }
+
+ /**
+ * Return an {@link IAsynchronousIterator} that will read a single,
+ * empty {@link IBindingSet}.
+ *
+ * @param bindingSet
+ * the binding set.
+ */
+ protected ThickAsynchronousIterator<IBindingSet[]> newBindingSetIterator(
+ final IBindingSet bindingSet) {
+
+ return new ThickAsynchronousIterator<IBindingSet[]>(
+ new IBindingSet[][] { new IBindingSet[] { bindingSet } });
+
+ }
+
+ /**
+ * Return an {@link IAsynchronousIterator} that will read a single, chunk
+ * containing all of the specified {@link IBindingSet}s.
+ *
+ * @param bindingSets
+ * the binding sets.
+ */
+ protected ThickAsynchronousIterator<IBindingSet[]> newBindingSetIterator(
+ final IBindingSet[] bindingSets) {
+
+ return new ThickAsynchronousIterator<IBindingSet[]>(
+ new IBindingSet[][] { bindingSets });
+
+ }
+
+ /**
+ * Return an {@link IAsynchronousIterator} that will read a single, chunk
+ * containing all of the specified {@link IBindingSet}s.
+ *
+ * @param bindingSetChunks
+ * the chunks of binding sets.
+ */
+ protected ThickAsynchronousIterator<IBindingSet[]> newBindingSetIterator(
+ final IBindingSet[][] bindingSetChunks) {
+
+ return new ThickAsynchronousIterator<IBindingSet[]>(bindingSetChunks);
+
+ }
+
+ /**
+ * Unit test for optional join group. Three joins are used and target a
+ * {@link SliceOp}. The 2nd and 3rd joins are embedded in an
+ * {@link SubqueryOp}.
+ * <P>
+ * The optional join group takes the form:
+ *
+ * <pre>
+ * (a b)
+ * optional {
+ * (b c)
+ * (c d)
+ * }
+ * </pre>
+ *
+ * The (a b) tail will match everything in the knowledge base. The join
+ * group takes us two hops out from ?b. There should be four solutions that
+ * succeed the optional join group:
+ *
+ * <pre>
+ * (paul mary brad fred)
+ * (paul mary brad leon)
+ * (john mary brad fred)
+ * (john mary brad leon)
+ * </pre>
+ *
+ * and five more that don't succeed the optional join group:
+ *
+ * <pre>
+ * (paul brad) *
+ * (john brad) *
+ * (mary brad) *
+ * (brad fred)
+ * (brad leon)
+ * </pre>
+ *
+ * In this cases marked with a <code>*</code>, ?c will become temporarily
+ * bound to fred and leon (since brad knows fred and leon), but the (c d)
+ * tail will fail since fred and leon don't know anyone else. At this point,
+ * the ?c binding must be removed from the solution.
+ */
+ public void test_query_join2_optionals() throws Exception {
+
+ // main query
+ final int startId = 1; //
+ final int joinId1 = 2; // : base join group.
+ final int predId1 = 3; // (a b)
+ final int joinGroup1 = 9;
+ final int sliceId = 8; //
+
+ // subquery
+ final int joinId2 = 4; // : joinGroup1
+ final int predId2 = 5; // (b c)
+ final int joinId3 = 6; // : joinGroup1
+ final int predId3 = 7; // (c d)
+
+ final IVariable<?> a = Var.var("a");
+ final IVariable<?> b = Var.var("b");
+ final IVariable<?> c = Var.var("c");
+ final IVariable<?> d = Var.var("d");
+
+ final PipelineOp startOp = new StartOp(new BOp[] {},
+ NV.asMap(new NV[] {//
+ new NV(Predicate.Annotations.BOP_ID, startId),//
+ new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
+ BOpEvaluationContext.CONTROLLER),//
+ }));
+
+ final Predicate<?> pred1Op = new Predicate<E>(
+ new IVariableOrConstant[] { a, b }, NV
+ .asMap(new NV[] {//
+ new NV(Predicate.Annotations.RELATION_NAME,
+ new String[] { namespace }),//
+ new NV(Predicate.Annotations.BOP_ID, predId1),//
+ new NV(Annotations.TIMESTAMP, ITx.READ_COMMITTED),//
+ }));
+
+ final Predicate<?> pred2Op = new Predicate<E>(
+ new IVariableOrConstant[] { b, c }, NV
+ .asMap(new NV[] {//
+ new NV(Predicate.Annotations.RELATION_NAME,
+ new String[] { namespace }),//
+ new NV(Predicate.Annotations.BOP_ID, predId2),//
+ new NV(Annotations.TIMESTAMP, ITx.READ_COMMITTED),//
+ }));
+
+ final Predicate<?> pred3Op = new Predicate<E>(
+ new IVariableOrConstant[] { c, d }, NV
+ .asMap(new NV[] {//
+ new NV(Predicate.Annotations.RELATION_NAME,
+ new String[] { namespace }),//
+ new NV(Predicate.Annotations.BOP_ID, predId3),//
+ new NV(Annotations.TIMESTAMP, ITx.READ_COMMITTED),//
+ }));
+
+ final PipelineOp join1Op = new PipelineJoin<E>(//
+ new BOp[]{startOp},//
+ new NV(Predicate.Annotations.BOP_ID, joinId1),//
+ new NV(PipelineJoin.Annotations.PREDICATE,pred1Op));
+
+ final PipelineOp subQuery;
+ {
+ final PipelineOp join2Op = new PipelineJoin<E>(//
+ new BOp[] { /*join1Op*/ },//
+ new NV(Predicate.Annotations.BOP_ID, joinId2),//
+// new NV(PipelineOp.Annotations.CONDITIONAL_GROUP, joinGroup1),//
+ new NV(PipelineJoin.Annotations.PREDICATE, pred2Op)//
+// // join is optional.
+// new NV(PipelineJoin.Annotations.OPTIONAL, true),//
+// // optional target is the same as the default target.
+// new NV(PipelineOp.Annotations.ALT_SINK_REF, sliceId)
+ );
+
+ final PipelineOp join3Op = new PipelineJoin<E>(//
+ new BOp[] { join2Op },//
+ new NV(Predicate.Annotations.BOP_ID, joinId3),//
+// new NV(PipelineOp.Annotations.CONDITIONAL_GROUP, joinGroup1),//
+ new NV(PipelineJoin.Annotations.PREDICATE, pred3Op)//
+// // join is optional.
+// new NV(PipelineJoin.Annotations.OPTIONAL, true),//
+// // optional target is the same as the default target.
+// new NV(PipelineOp.Annotations.ALT_SINK_REF, sliceId)
+ );
+ subQuery = join3Op;
+ }
+
+ final PipelineOp joinGroup1Op = new SubqueryOp(new BOp[]{join1Op},
+ new NV(Predicate.Annotations.BOP_ID, joinGroup1),//
+// new NV(PipelineOp.Annotations.CONDITIONAL_GROUP, joinGroup1),//
+ new NV(SubqueryOp.Annotations.SUBQUERY, subQuery)//
+// , new NV(BOp.Annotations.CONTROLLER,true)//
+// new NV(BOp.Annotations.EVALUATION_CONTEXT,
+// BOpEvaluationContext.CONTROLLER)//
+// // join is optional.
+// new NV(PipelineJoin.Annotations.OPTIONAL, true),//
+// // optional target is the same as the default target.
+// new NV(PipelineOp.Annotations.ALT_SINK_REF, sliceId)
+ );
+
+ final PipelineOp sliceOp = new SliceOp(//
+ new BOp[]{joinGroup1Op},
+ NV.asMap(new NV[] {//
+ new NV(BOp.Annotations.BOP_ID, sliceId),//
+ new NV(BOp.Annotations.EVALUATION_CONTEXT,
+ BOpEvaluationContext.CONTROLLER),//
+ }));
+
+ final PipelineOp query = sliceOp;
+
+ // start the query.
+ final UUID queryId = UUID.randomUUID();
+ final IChunkMessage<IBindingSet> initialChunkMessage;
+ {
+
+ final IBindingSet initialBindings = new HashBindingSet();
+
+// initialBindings.set(Var.var("x"), new Constant<String>("Mary"));
+
+ initialChunkMessage = new LocalChunkMessage<IBindingSet>(queryEngine,
+ queryId, startId,//
+ -1, // partitionId
+ newBindingSetIterator(initialBindings));
+ }
+ final IRunningQuery runningQuery = queryEngine.eval(queryId, query,
+ initialChunkMessage);
+
...
[truncated message content] |
|
From: <mrp...@us...> - 2011-01-13 22:13:53
|
Revision: 4095
http://bigdata.svn.sourceforge.net/bigdata/?rev=4095&view=rev
Author: mrpersonick
Date: 2011-01-13 22:13:47 +0000 (Thu, 13 Jan 2011)
Log Message:
-----------
added some code that will translate regex to our free text index. currently disabled
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java 2011-01-13 22:06:28 UTC (rev 4094)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java 2011-01-13 22:13:47 UTC (rev 4095)
@@ -40,6 +40,7 @@
import org.openrdf.query.algebra.ProjectionElemList;
import org.openrdf.query.algebra.QueryModelNode;
import org.openrdf.query.algebra.QueryRoot;
+import org.openrdf.query.algebra.Regex;
import org.openrdf.query.algebra.SameTerm;
import org.openrdf.query.algebra.StatementPattern;
import org.openrdf.query.algebra.StatementPattern.Scope;
@@ -52,6 +53,7 @@
import org.openrdf.query.algebra.evaluation.impl.EvaluationStrategyImpl;
import org.openrdf.query.algebra.evaluation.iterator.FilterIterator;
import org.openrdf.query.algebra.helpers.QueryModelVisitorBase;
+import org.openrdf.query.parser.serql.AnonymousVarGenerator;
import com.bigdata.bop.BOp;
import com.bigdata.bop.BOpUtility;
@@ -617,15 +619,27 @@
* UnsupportedOperatorException here must just flow through
* to Sesame evaluation of the entire query.
*/
- final ValueExpr ve = (ValueExpr) op;
- final IConstraint bop = toConstraint(ve);
- sop.setBOp(bop);
+// if (op instanceof Regex) {
+// final Regex regex = (Regex) op;
+// final IPredicate bop = toPredicate(regex);
+// sop.setBOp(bop);
+// } else {
+ final ValueExpr ve = (ValueExpr) op;
+ final IConstraint bop = toConstraint(ve);
+ sop.setBOp(bop);
+// }
} else if (op instanceof Filter) {
final Filter filter = (Filter) op;
final ValueExpr ve = filter.getCondition();
try {
- final IConstraint bop = toConstraint(ve);
- sop.setBOp(bop);
+// if (ve instanceof Regex) {
+// final Regex regex = (Regex) ve;
+// final IPredicate bop = toPredicate(regex);
+// sop.setBOp(bop);
+// } else {
+ final IConstraint bop = toConstraint(ve);
+ sop.setBOp(bop);
+// }
} catch (UnsupportedOperatorException ex) {
/*
* If we encounter a sesame filter (ValueExpr) that we
@@ -1439,6 +1453,19 @@
//
// }
+ private IPredicate toPredicate(final Regex regex)
+ throws QueryEvaluationException {
+
+ final Var s = (Var) regex.getLeftArg();
+ final ValueConstant vc = (ValueConstant) regex.getRightArg();
+ final Var p = new Var();
+ p.setValue(BD.SEARCH);
+ final Var o = new Var();
+ o.setValue(vc.getValue());
+ return toPredicate(new StatementPattern(s, p, o));
+
+ }
+
/**
* Generate a bigdata {@link IPredicate} (tail) for the supplied
* StatementPattern.
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-13 22:06:34
|
Revision: 4094
http://bigdata.svn.sourceforge.net/bigdata/?rev=4094&view=rev
Author: thompsonbry
Date: 2011-01-13 22:06:28 +0000 (Thu, 13 Jan 2011)
Log Message:
-----------
removed the statusZookeeper ant target since the output is not written to the console.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/build.xml
Modified: branches/QUADS_QUERY_BRANCH/build.xml
===================================================================
--- branches/QUADS_QUERY_BRANCH/build.xml 2011-01-13 22:04:23 UTC (rev 4093)
+++ branches/QUADS_QUERY_BRANCH/build.xml 2011-01-13 22:06:28 UTC (rev 4094)
@@ -1722,15 +1722,6 @@
</exec>
</target>
- <target name="statusZookeeper">
- <echo message="test.zookeeper.installDir=${test.zookeeper.installDir}"/>
- <echo>bin/zkServer.sh status
-</echo>
- <exec executable="bin/zkServer.sh" dir="${test.zookeeper.installDir}" logerror="true">
- <arg value="status"/>
- </exec>
- </target>
-
<!-- runs all junit tests -->
<target name="run-junit">
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-13 22:04:29
|
Revision: 4093
http://bigdata.svn.sourceforge.net/bigdata/?rev=4093&view=rev
Author: thompsonbry
Date: 2011-01-13 22:04:23 +0000 (Thu, 13 Jan 2011)
Log Message:
-----------
modified the zookeeper start/stop and targets.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/build.xml
Modified: branches/QUADS_QUERY_BRANCH/build.xml
===================================================================
--- branches/QUADS_QUERY_BRANCH/build.xml 2011-01-13 21:25:49 UTC (rev 4092)
+++ branches/QUADS_QUERY_BRANCH/build.xml 2011-01-13 22:04:23 UTC (rev 4093)
@@ -1708,18 +1708,29 @@
<echo message="test.zookeeper.installDir=${test.zookeeper.installDir}"/>
<echo>bin/zkServer.sh start
</echo>
- <exec command="bin/zkServer.sh start" dir="${test.zookeeper.installDir}" logerror="true">
-</exec>
+ <exec executable="bin/zkServer.sh" dir="${test.zookeeper.installDir}" logerror="true">
+ <arg value="start"/>
+ </exec>
</target>
<target name="stopZookeeper">
<echo message="test.zookeeper.installDir=${test.zookeeper.installDir}"/>
- <echo>bin/zkServer.sh start
+ <echo>bin/zkServer.sh stop
</echo>
- <exec command="bin/zkServer.sh stop" dir="${test.zookeeper.installDir}" logerror="true">
-</exec>
+ <exec executable="bin/zkServer.sh" dir="${test.zookeeper.installDir}" logerror="true">
+ <arg value="stop"/>
+ </exec>
</target>
+ <target name="statusZookeeper">
+ <echo message="test.zookeeper.installDir=${test.zookeeper.installDir}"/>
+ <echo>bin/zkServer.sh status
+</echo>
+ <exec executable="bin/zkServer.sh" dir="${test.zookeeper.installDir}" logerror="true">
+ <arg value="status"/>
+ </exec>
+ </target>
+
<!-- runs all junit tests -->
<target name="run-junit">
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mrp...@us...> - 2011-01-13 21:25:55
|
Revision: 4092
http://bigdata.svn.sourceforge.net/bigdata/?rev=4092&view=rev
Author: mrpersonick
Date: 2011-01-13 21:25:49 +0000 (Thu, 13 Jan 2011)
Log Message:
-----------
changing the default time zone and precision for decoding date/times
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/DateTimeExtension.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataValueFactoryImpl.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/DateTimeExtension.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/DateTimeExtension.java 2011-01-13 20:58:04 UTC (rev 4091)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/DateTimeExtension.java 2011-01-13 21:25:49 UTC (rev 4092)
@@ -109,8 +109,9 @@
*/
final long l = iv.getDelegate().longValue();
- final GregorianCalendar c =
- new GregorianCalendar(TimeZone.getTimeZone("GMT"));
+ final GregorianCalendar c = new GregorianCalendar(
+// TimeZone.getTimeZone("GMT")
+ );
c.setTimeInMillis(l);
try {
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataValueFactoryImpl.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataValueFactoryImpl.java 2011-01-13 20:58:04 UTC (rev 4091)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataValueFactoryImpl.java 2011-01-13 21:25:49 UTC (rev 4092)
@@ -271,7 +271,13 @@
// null/* languageCode */, createURI(arg0.getXMLSchemaType()
// .toString()));
- return new BigdataLiteralImpl(this, arg0.toString(),
+ String s = arg0.toString();
+ final int i = s.lastIndexOf('.');
+ if (i >= 0) {
+ s = s.substring(0, i);
+ }
+
+ return new BigdataLiteralImpl(this, s,
null/* languageCode */, createURI(XMLDatatypeUtil.qnameToURI(
arg0.getXMLSchemaType()).stringValue()));
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-13 20:58:10
|
Revision: 4091
http://bigdata.svn.sourceforge.net/bigdata/?rev=4091&view=rev
Author: thompsonbry
Date: 2011-01-13 20:58:04 +0000 (Thu, 13 Jan 2011)
Log Message:
-----------
javadoc
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java 2011-01-13 20:54:09 UTC (rev 4090)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java 2011-01-13 20:58:04 UTC (rev 4091)
@@ -173,7 +173,8 @@
BOpEvaluationContext.CONTROLLER)) {
/*
* Wrap with an operator which will be evaluated on the query
- * controller.
+ * controller so the results will be streamed back to the query
+ * controller in scale-out.
*/
left = new SliceOp(new BOp[] { left }, NV.asMap(//
new NV(BOp.Annotations.BOP_ID, idFactory
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mrp...@us...> - 2011-01-13 20:54:15
|
Revision: 4090
http://bigdata.svn.sourceforge.net/bigdata/?rev=4090&view=rev
Author: mrpersonick
Date: 2011-01-13 20:54:09 +0000 (Thu, 13 Jan 2011)
Log Message:
-----------
took out a redundant log statement
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java 2011-01-13 20:53:37 UTC (rev 4089)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java 2011-01-13 20:54:09 UTC (rev 4090)
@@ -699,10 +699,6 @@
try {
- if (log.isDebugEnabled()) {
- log.debug("running native query: " + BOpUtility.toString(query));
- }
-
final IRunningQuery runningQuery = queryEngine.eval(query);
final IAsynchronousIterator<IBindingSet[]> it1 =
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mrp...@us...> - 2011-01-13 20:53:43
|
Revision: 4089
http://bigdata.svn.sourceforge.net/bigdata/?rev=4089&view=rev
Author: mrpersonick
Date: 2011-01-13 20:53:37 +0000 (Thu, 13 Jan 2011)
Log Message:
-----------
added a slice op
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java 2011-01-13 20:47:44 UTC (rev 4088)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java 2011-01-13 20:53:37 UTC (rev 4089)
@@ -50,6 +50,7 @@
import com.bigdata.bop.controller.SubqueryOp;
import com.bigdata.bop.controller.Union;
import com.bigdata.bop.engine.QueryEngine;
+import com.bigdata.bop.solutions.SliceOp;
import com.bigdata.rdf.sail.Rule2BOpUtility;
import com.bigdata.rdf.sail.sop.SOpTree.SOpGroup;
import com.bigdata.rdf.sail.sop.SOpTree.SOpGroups;
@@ -165,14 +166,26 @@
final IRule rule = rule(join, conditionals);
- final PipelineOp joinGroup = Rule2BOpUtility.convert(
+ PipelineOp left = Rule2BOpUtility.convert(
rule, conditionals, idFactory, db, queryEngine, queryHints);
+ if (!left.getEvaluationContext().equals(
+ BOpEvaluationContext.CONTROLLER)) {
+ /*
+ * Wrap with an operator which will be evaluated on the query
+ * controller.
+ */
+ left = new SliceOp(new BOp[] { left }, NV.asMap(//
+ new NV(BOp.Annotations.BOP_ID, idFactory
+ .incrementAndGet()), //
+ new NV(BOp.Annotations.EVALUATION_CONTEXT,
+ BOpEvaluationContext.CONTROLLER)));
+ }
+
/*
* Start with left=<this join group> and add a SubqueryOp for each
* sub group.
*/
- PipelineOp left = joinGroup;
final SOpGroups children = join.getChildren();
if (children != null) {
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-13 20:47:50
|
Revision: 4088
http://bigdata.svn.sourceforge.net/bigdata/?rev=4088&view=rev
Author: thompsonbry
Date: 2011-01-13 20:47:44 +0000 (Thu, 13 Jan 2011)
Log Message:
-----------
javadoc
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/Rule2BOpUtility.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/Rule2BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/Rule2BOpUtility.java 2011-01-13 20:25:42 UTC (rev 4087)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/Rule2BOpUtility.java 2011-01-13 20:47:44 UTC (rev 4088)
@@ -258,7 +258,8 @@
BOpEvaluationContext.CONTROLLER)) {
/*
* Wrap with an operator which will be evaluated on the query
- * controller.
+ * controller so the results will be streamed back to the query
+ * controller in scale-out.
*/
tmp = new SliceOp(new BOp[] { tmp }, NV.asMap(//
new NV(BOp.Annotations.BOP_ID, idFactory
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-13 20:25:48
|
Revision: 4087
http://bigdata.svn.sourceforge.net/bigdata/?rev=4087&view=rev
Author: thompsonbry
Date: 2011-01-13 20:25:42 +0000 (Thu, 13 Jan 2011)
Log Message:
-----------
zookeeper CI ant targets added
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/build.properties
branches/QUADS_QUERY_BRANCH/build.xml
Modified: branches/QUADS_QUERY_BRANCH/build.properties
===================================================================
--- branches/QUADS_QUERY_BRANCH/build.properties 2011-01-13 17:56:43 UTC (rev 4086)
+++ branches/QUADS_QUERY_BRANCH/build.properties 2011-01-13 20:25:42 UTC (rev 4087)
@@ -376,3 +376,7 @@
# on a volume with a lot of room. The directory may be destroyed (by the test harness)
# after the performance tests have run their course.
perf.run.dir=/usr/bigdata/runs
+
+# CI properties.
+test.zookeeper.installDir=/Users/bryan/zookeeper-3.2.1
+
Modified: branches/QUADS_QUERY_BRANCH/build.xml
===================================================================
--- branches/QUADS_QUERY_BRANCH/build.xml 2011-01-13 17:56:43 UTC (rev 4086)
+++ branches/QUADS_QUERY_BRANCH/build.xml 2011-01-13 20:25:42 UTC (rev 4087)
@@ -115,7 +115,7 @@
</target>
<!-- Builds the bigdata JAR and bundles it together with all of its dependencies in the ${build.dir}/lib directory. -->
- <target name="bundleJar" depends="bundle, jar" description="Builds the bigdata JAR and bundles it together with all of its dependencies in the ${build.dir}/lib directory.">
+ <target name="bundleJar" depends="clean, bundle, jar" description="Builds the bigdata JAR and bundles it together with all of its dependencies in the ${build.dir}/lib directory.">
<copy file="${build.dir}/${version}.jar" todir="${build.dir}/lib" />
<!--<property name="myclasspath" refid="runtime.classpath" />
<echo message="${myclasspath}"/>-->
@@ -1599,6 +1599,18 @@
<property name="test.codebase.dir" value="${dist.lib.dl}" />
<property name="test.codebase" value="http://${this.hostname}:${test.codebase.port}/jsk-dl.jar" />
+ <!-- These zookeeper configuration properties used to inform the test -->
+ <!-- suite about the zookeeper instance which will be used by the -->
+ <!-- tests. These properties MUST be consistent with the actual -->
+ <!-- zookeeper configuration. Zookeeper is assumed (by the tests) to -->
+ <!-- be running on the localhost. -->
+ <property name="test.zookeeper.tickTime" value="2000" />
+ <property name="test.zookeeper.clientPort" value="2888" />
+ <property name="test.zookeeper.leaderPort" value="3888" />
+
+ <!-- The zookeeper install directory. -->
+ <property name="test.zookeeper.installDir" value="${zookeeper.installDir}" />
+
<property name="java.security.policy" value="${dist.var.config.policy}/policy.all" />
<property name="log4j.configuration" value="${bigdata.test.log4j.rel}" />
<property name="java.net.preferIPv4Stack" value="true" />
@@ -1608,6 +1620,7 @@
<copy file="${build.properties.from.file}"
todir="${build.properties.test.to.path}" />
+ <antcall target="startZookeeper" />
<antcall target="startHttpd" />
<antcall target="startLookup" />
@@ -1616,6 +1629,7 @@
<antcall target="stopLookup" />
<antcall target="stopHttpd" />
+ <antcall target="stopZookeeper" />
<!-- This message is noticed by the hudson build and is used to trigger
after various after actions. -->
@@ -1690,6 +1704,22 @@
</java>
</target>
+ <target name="startZookeeper">
+ <echo message="test.zookeeper.installDir=${test.zookeeper.installDir}"/>
+ <echo>bin/zkServer.sh start
+</echo>
+ <exec command="bin/zkServer.sh start" dir="${test.zookeeper.installDir}" logerror="true">
+</exec>
+ </target>
+
+ <target name="stopZookeeper">
+ <echo message="test.zookeeper.installDir=${test.zookeeper.installDir}"/>
+ <echo>bin/zkServer.sh start
+</echo>
+ <exec command="bin/zkServer.sh stop" dir="${test.zookeeper.installDir}" logerror="true">
+</exec>
+ </target>
+
<!-- runs all junit tests -->
<target name="run-junit">
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mrp...@us...> - 2011-01-13 17:56:49
|
Revision: 4086
http://bigdata.svn.sourceforge.net/bigdata/?rev=4086&view=rev
Author: mrpersonick
Date: 2011-01-13 17:56:43 +0000 (Thu, 13 Jan 2011)
Log Message:
-----------
trying to get bsbm to qualify
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-perf/bsbm/src/test/benchmark/bigdata/TestBSBM.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSail.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSailRepository.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-perf/bsbm/src/test/benchmark/bigdata/TestBSBM.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-perf/bsbm/src/test/benchmark/bigdata/TestBSBM.java 2011-01-13 17:55:15 UTC (rev 4085)
+++ branches/QUADS_QUERY_BRANCH/bigdata-perf/bsbm/src/test/benchmark/bigdata/TestBSBM.java 2011-01-13 17:56:43 UTC (rev 4086)
@@ -1,63 +1,143 @@
package benchmark.bigdata;
-import com.bigdata.rdf.sail.bench.NanoSparqlClient;
+import java.io.FileInputStream;
+import java.util.Properties;
+import org.apache.log4j.Logger;
+import org.openrdf.model.Statement;
+import org.openrdf.model.impl.URIImpl;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.algebra.QueryRoot;
+import org.openrdf.query.algebra.TupleExpr;
+import org.openrdf.query.algebra.UnaryTupleOperator;
+import org.openrdf.repository.RepositoryResult;
+import org.openrdf.repository.sail.SailTupleQuery;
+
+import com.bigdata.rdf.sail.BigdataSail;
+import com.bigdata.rdf.sail.BigdataSailRepository;
+import com.bigdata.rdf.sail.BigdataSailRepositoryConnection;
+import com.bigdata.rdf.sail.BigdataSailTupleQuery;
+import com.bigdata.rdf.sail.sop.SOpTree;
+import com.bigdata.rdf.sail.sop.SOpTreeBuilder;
+import com.bigdata.rdf.store.AbstractTripleStore;
+
public class TestBSBM {
- private static final String serviceURL = "http://localhost:8080";
+ private static final Logger log = Logger.getLogger(TestBSBM.class);
+
+ private static final String PROPS =
+ "/Users/mikepersonick/Documents/workspace/bigdata-query-branch/bigdata-perf/bsbm/RWStore.properties";
+
+ private static final String JNL =
+ "/Users/mikepersonick/Documents/systap/bsbm/bsbm-qual/bigdata-bsbm.RW.jnl";
- private static final String queryStr =
-/*
- "PREFIX bsbm-inst: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/> " +
- "PREFIX bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/> " +
- "PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#> " +
- "PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> " +
- "SELECT DISTINCT ?product " +
- "WHERE { " +
- " { " +
- " ?product rdfs:label ?label . " +
- " ?product rdf:type <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/ProductType75> . " +
- " ?product bsbm:productFeature <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/ProductFeature379> . " +
- " ?product bsbm:productFeature <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/ProductFeature2248> . " +
- " ?product bsbm:productPropertyTextual1 ?propertyTextual . " +
- " ?product bsbm:productPropertyNumeric1 ?p1 . " +
- " FILTER ( ?p1 > 160 ) " +
- " } UNION { " +
- " ?product rdfs:label ?label . " +
- " ?product rdf:type <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/ProductType75> . " +
- " ?product bsbm:productFeature <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/ProductFeature379> . " +
- " ?product bsbm:productFeature <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/ProductFeature2251> . " +
- " ?product bsbm:productPropertyTextual1 ?propertyTextual . " +
- " ?product bsbm:productPropertyNumeric2 ?p2 . " +
- " FILTER ( ?p2 > 461 ) " +
- " } " +
-// " FILTER ( ?p1 > 160 || ?p2> 461 ) " +
- "} "
-// "ORDER BY ?label " +
-// "OFFSET 5 " +
-// "LIMIT 10"
- ;
-*/
- "construct { <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/dataFromProducer24/Product1131> ?p ?o . } " +
- "where { " +
- " <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/dataFromProducer24/Product1131> ?p ?o . " +
- "}";
+ private static final String NS = "qual";
+ private static final String QUERY =
+
+ "PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#> " +
+ " PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> " +
+ " PREFIX bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/> " +
+ " SELECT DISTINCT ?product ?productLabel " +
+ " WHERE { " +
+ " ?product rdfs:label ?productLabel . " +
+ " FILTER (<http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/dataFromProducer36/Product1625> != ?product) " +
+ " <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/dataFromProducer36/Product1625> bsbm:productFeature ?prodFeature . " +
+ " ?product bsbm:productFeature ?prodFeature . " +
+ " <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/dataFromProducer36/Product1625> bsbm:productPropertyNumeric1 ?origProperty1 . " +
+ " ?product bsbm:productPropertyNumeric1 ?simProperty1 . " +
+ " FILTER (?simProperty1 < (?origProperty1 + 120) && ?simProperty1 > (?origProperty1 - 120)) " +
+ " <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/dataFromProducer36/Product1625> bsbm:productPropertyNumeric2 ?origProperty2 . " +
+ " ?product bsbm:productPropertyNumeric2 ?simProperty2 . " +
+ " FILTER (?simProperty2 < (?origProperty2 + 170) && ?simProperty2 > (?origProperty2 - 170)) " +
+ " } " +
+ " ORDER BY ?productLabel " +
+ " LIMIT 5";
+
+
/**
* @param args
*/
- public static void main(String[] args) {
+ public static void main(String[] args) throws Exception {
try {
- args = new String[] {
- "-query",
- queryStr,
- serviceURL
- };
+// args = new String[] {
+// "-query",
+// queryStr,
+// serviceURL
+// };
+//
+// NanoSparqlClient.main(args);
- NanoSparqlClient.main(args);
-
+
+ final Properties props = new Properties();
+ final FileInputStream fis = new FileInputStream(PROPS);
+ props.load(fis);
+ fis.close();
+ props.setProperty(AbstractTripleStore.Options.FILE, JNL);
+ props.setProperty(BigdataSail.Options.NAMESPACE, NS);
+
+ System.err.println(props);
+
+ final BigdataSail sail = new BigdataSail(props);
+ sail.initialize();
+ final BigdataSailRepository repo = new BigdataSailRepository(sail);
+ final BigdataSailRepositoryConnection cxn =
+ repo.getReadOnlyConnection();
+
+ try {
+
+// final RepositoryResult<Statement> stmts = cxn.getStatements(
+// null, RDF.TYPE,
+// new URIImpl("http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/ProductType138"), false);
+// if (stmts.hasNext()) {
+// while (stmts.hasNext()) {
+// System.err.println(stmts.next());
+// }
+// } else {
+// System.err.println("no stmts");
+// }
+
+ final SailTupleQuery tupleQuery = (SailTupleQuery)
+ cxn.prepareTupleQuery(QueryLanguage.SPARQL, QUERY);
+ tupleQuery.setIncludeInferred(false /* includeInferred */);
+
+ if (log.isInfoEnabled()) {
+
+ final BigdataSailTupleQuery bdTupleQuery =
+ (BigdataSailTupleQuery) tupleQuery;
+ final QueryRoot root = (QueryRoot) bdTupleQuery.getTupleExpr();
+ log.info(root);
+ TupleExpr te = root;
+ while (te instanceof UnaryTupleOperator) {
+ te = ((UnaryTupleOperator) te).getArg();
+ }
+
+ final SOpTreeBuilder stb = new SOpTreeBuilder();
+ final SOpTree tree = stb.collectSOps(te);
+
+ log.info(tree);
+ log.info(QUERY);
+
+ final TupleQueryResult result = tupleQuery.evaluate();
+ if (result.hasNext()) {
+ while (result.hasNext()) {
+ log.info(result.next());
+ }
+ } else {
+ log.info("no results for query");
+ }
+
+ }
+
+ } finally {
+ cxn.close();
+ repo.shutDown();
+ }
+
} catch (Exception ex) {
ex.printStackTrace();
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java 2011-01-13 17:55:15 UTC (rev 4085)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java 2011-01-13 17:56:43 UTC (rev 4086)
@@ -272,7 +272,8 @@
*/
public BigdataEvaluationStrategyImpl3(
final BigdataTripleSource tripleSource, final Dataset dataset,
- final boolean nativeJoins) {
+ final boolean nativeJoins,
+ final boolean allowSesameQueryEvaluation) {
super(tripleSource, dataset);
@@ -280,6 +281,7 @@
this.dataset = dataset;
this.database = tripleSource.getDatabase();
this.nativeJoins = nativeJoins;
+ this.allowSesameQueryEvaluation = allowSesameQueryEvaluation;
}
@@ -289,6 +291,12 @@
private boolean nativeJoins;
/**
+ * If true, allow queries that cannot be executed natively to be executed
+ * by Sesame.
+ */
+ private final boolean allowSesameQueryEvaluation;
+
+ /**
* A set of properties that act as query hints during evaluation.
*/
private Properties queryHints;
@@ -364,18 +372,30 @@
} catch (UnsupportedOperatorException ex) {
- // Use Sesame 2 evaluation
-
- log.warn("could not evaluate natively, using Sesame evaluation");
-
- if (log.isInfoEnabled()) {
- log.info(ex.getOperator());
- }
-
- nativeJoins = false;
-
- return super.evaluate(union, bs);
-
+ if (allowSesameQueryEvaluation) {
+
+ // Use Sesame 2 evaluation
+
+ log.warn("could not evaluate natively, using Sesame evaluation");
+
+ if (log.isInfoEnabled()) {
+ log.info(ex.getOperator());
+ }
+
+ // turn off native joins for the remainder, we can't do
+ // partial execution
+ nativeJoins = false;
+
+ // defer to Sesame
+ return super.evaluate(union, bs);
+
+ } else {
+
+ // allow the query to fail
+ throw ex;
+
+ }
+
}
}
@@ -409,18 +429,30 @@
} catch (UnsupportedOperatorException ex) {
- // Use Sesame 2 evaluation
+ if (allowSesameQueryEvaluation) {
+
+ // Use Sesame 2 evaluation
+
+ log.warn("could not evaluate natively, using Sesame evaluation");
+
+ if (log.isInfoEnabled()) {
+ log.info(ex.getOperator());
+ }
+
+ // turn off native joins for the remainder, we can't do
+ // partial execution
+ nativeJoins = false;
+
+ // defer to Sesame
+ return super.evaluate(join, bs);
+
+ } else {
+
+ // allow the query to fail
+ throw ex;
+
+ }
- log.warn("could not evaluate natively, using Sesame evaluation");
-
- if (log.isInfoEnabled()) {
- log.info(ex.getOperator());
- }
-
- nativeJoins = false;
-
- return super.evaluate(join, bs);
-
}
}
@@ -454,18 +486,30 @@
} catch (UnsupportedOperatorException ex) {
- // Use Sesame 2 evaluation
+ if (allowSesameQueryEvaluation) {
+
+ // Use Sesame 2 evaluation
+
+ log.warn("could not evaluate natively, using Sesame evaluation");
+
+ if (log.isInfoEnabled()) {
+ log.info(ex.getOperator());
+ }
+
+ // turn off native joins for the remainder, we can't do
+ // partial execution
+ nativeJoins = false;
+
+ // defer to Sesame
+ return super.evaluate(leftJoin, bs);
+
+ } else {
+
+ // allow the query to fail
+ throw ex;
+
+ }
- log.warn("could not evaluate natively, using Sesame evaluation");
-
- if (log.isInfoEnabled()) {
- log.info(ex.getOperator());
- }
-
- nativeJoins = false;
-
- return super.evaluate(leftJoin, bs);
-
}
}
@@ -476,6 +520,9 @@
try {
return _evaluateNatively(tupleExpr, bs);
} catch (UnrecognizedValueException ex) {
+ if (log.isInfoEnabled()) {
+ log.info("unrecognized value in query: " + ex.getValue());
+ }
return new EmptyIteration<BindingSet, QueryEvaluationException>();
} catch (QueryEvaluationException ex) {
throw ex;
@@ -652,6 +699,10 @@
try {
+ if (log.isDebugEnabled()) {
+ log.debug("running native query: " + BOpUtility.toString(query));
+ }
+
final IRunningQuery runningQuery = queryEngine.eval(query);
final IAsynchronousIterator<IBindingSet[]> it1 =
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSail.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSail.java 2011-01-13 17:55:15 UTC (rev 4085)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSail.java 2011-01-13 17:56:43 UTC (rev 4086)
@@ -384,6 +384,20 @@
public static final String DEFAULT_NEW_EVAL_STRATEGY = "false";
+ /**
+ * Option as to whether or not to allow Sesame evaluation of queries
+ * that cannot be run natively (if false, these queries will throw
+ * exceptions. If true, these queries will run heinously slow but they
+ * will run. (default
+ * {@value #DEFAULT_NEW_EVAL_STRATEGY}).
+ */
+ public static final String ALLOW_SESAME_QUERY_EVALUATION = BigdataSail.class.getPackage()
+ .getName()+ ".allowSesameQueryEvaluation";
+
+ public static final String DEFAULT_ALLOW_SESAME_QUERY_EVALUATION = "false";
+
+
+
}
/**
@@ -531,6 +545,14 @@
final private boolean starJoins;
/**
+ * When true, allow queries that cannot be executed natively to be
+ * executed by Sesame.
+ *
+ * @See {@link Options#ALLOW_SESAME_QUERY_EVALUATION}
+ */
+ final private boolean allowSesameQueryEvaluation;
+
+ /**
* <code>true</code> iff the {@link BigdataSail} has been
* {@link #initialize()}d and not {@link #shutDown()}.
*/
@@ -955,6 +977,19 @@
}
+ // Sesame query evaluation
+ {
+
+ allowSesameQueryEvaluation = Boolean.parseBoolean(properties.getProperty(
+ BigdataSail.Options.ALLOW_SESAME_QUERY_EVALUATION,
+ BigdataSail.Options.DEFAULT_ALLOW_SESAME_QUERY_EVALUATION));
+
+ if (log.isInfoEnabled())
+ log.info(BigdataSail.Options.ALLOW_SESAME_QUERY_EVALUATION + "="
+ + allowSesameQueryEvaluation);
+
+ }
+
namespaces =
Collections.synchronizedMap(new LinkedHashMap<String, String>());
@@ -3265,7 +3300,8 @@
if (newEvalStrategy) {
strategy = new BigdataEvaluationStrategyImpl3(
- tripleSource, dataset, nativeJoins
+ tripleSource, dataset, nativeJoins,
+ allowSesameQueryEvaluation
);
} else {
strategy = new BigdataEvaluationStrategyImpl(
@@ -3360,7 +3396,8 @@
if (newEvalStrategy) {
strategy = new BigdataEvaluationStrategyImpl3(
- tripleSource, dataset, nativeJoins
+ tripleSource, dataset, nativeJoins,
+ allowSesameQueryEvaluation
);
} else {
strategy = new BigdataEvaluationStrategyImpl(
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSailRepository.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSailRepository.java 2011-01-13 17:55:15 UTC (rev 4085)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSailRepository.java 2011-01-13 17:56:43 UTC (rev 4086)
@@ -57,7 +57,7 @@
*
* @return a read-only connection to the database
*/
- public SailRepositoryConnection getReadOnlyConnection()
+ public BigdataSailRepositoryConnection getReadOnlyConnection()
throws RepositoryException {
return new BigdataSailRepositoryConnection(this,
@@ -71,7 +71,7 @@
*
* @return a read-only connection to the database
*/
- public SailRepositoryConnection getReadOnlyConnection(long timestamp)
+ public BigdataSailRepositoryConnection getReadOnlyConnection(long timestamp)
throws RepositoryException {
return new BigdataSailRepositoryConnection(this,
@@ -79,7 +79,7 @@
}
- public SailRepositoryConnection getReadWriteConnection()
+ public BigdataSailRepositoryConnection getReadWriteConnection()
throws RepositoryException {
try {
@@ -95,7 +95,7 @@
}
- public SailRepositoryConnection getUnisolatedConnection()
+ public BigdataSailRepositoryConnection getUnisolatedConnection()
throws RepositoryException {
try {
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp.java 2011-01-13 17:55:15 UTC (rev 4085)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp.java 2011-01-13 17:56:43 UTC (rev 4086)
@@ -86,7 +86,8 @@
private String toString(final Var v) {
if (v.hasValue()) {
final String s = v.getValue().stringValue();
- return s.substring(s.indexOf('#'));
+ final int i = s.indexOf('#');
+ return i >= 0 ? s.substring(s.indexOf('#')) : s;
} else {
return "?"+v.getName();
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java 2011-01-13 17:55:15 UTC (rev 4085)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java 2011-01-13 17:56:43 UTC (rev 4086)
@@ -214,7 +214,7 @@
int i = 0;
for (SOpGroup child : children) {
// convert the child IStep
- args[i] = convert(child, idFactory, db, queryEngine, queryHints);
+ args[i++] = convert(child, idFactory, db, queryEngine, queryHints);
}
final LinkedList<NV> anns = new LinkedList<NV>();
@@ -225,6 +225,7 @@
final Union thisOp = new Union(args, NV
.asMap(anns.toArray(new NV[anns.size()])));
+
return thisOp;
}
@@ -283,7 +284,7 @@
final SOp sop = child.getSingletonSOp();
final BOp bop = sop.getBOp();
final IPredicate pred = (IPredicate) bop.setProperty(
- IPredicate.Annotations.OPTIONAL, "true");
+ IPredicate.Annotations.OPTIONAL, Boolean.TRUE);
preds.add(pred);
}
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mrp...@us...> - 2011-01-13 17:55:21
|
Revision: 4085
http://bigdata.svn.sourceforge.net/bigdata/?rev=4085&view=rev
Author: mrpersonick
Date: 2011-01-13 17:55:15 +0000 (Thu, 13 Jan 2011)
Log Message:
-----------
added the option to inline datetimes
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-perf/bsbm/RWStore.properties
branches/QUADS_QUERY_BRANCH/bigdata-perf/bsbm/WORMStore.properties
Modified: branches/QUADS_QUERY_BRANCH/bigdata-perf/bsbm/RWStore.properties
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-perf/bsbm/RWStore.properties 2011-01-13 15:57:45 UTC (rev 4084)
+++ branches/QUADS_QUERY_BRANCH/bigdata-perf/bsbm/RWStore.properties 2011-01-13 17:55:15 UTC (rev 4085)
@@ -48,11 +48,14 @@
#com.bigdata.rdf.store.AbstractTripleStore.termCache.capacity=50000
com.bigdata.rdf.store.AbstractTripleStore.axiomsClass=com.bigdata.rdf.axioms.NoAxioms
+# xsd:DateTime needs to be inlined for BSBM
+com.bigdata.rdf.store.AbstractTripleStore.inlineDateTimes=true
+
##
## Sail options.
##
-com.bigdata.rdf.sail.BigdataSail.truthMaintenance=false
+com.bigdata.rdf.sail.truthMaintenance=false
# 10000 is default.
-com.bigdata.rdf.sail.BigdataSail.bufferCapacity=100000
+com.bigdata.rdf.sail.bufferCapacity=100000
Modified: branches/QUADS_QUERY_BRANCH/bigdata-perf/bsbm/WORMStore.properties
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-perf/bsbm/WORMStore.properties 2011-01-13 15:57:45 UTC (rev 4084)
+++ branches/QUADS_QUERY_BRANCH/bigdata-perf/bsbm/WORMStore.properties 2011-01-13 17:55:15 UTC (rev 4085)
@@ -27,11 +27,14 @@
#com.bigdata.rdf.store.AbstractTripleStore.termCache.capacity=50000
com.bigdata.rdf.store.AbstractTripleStore.axiomsClass=com.bigdata.rdf.axioms.NoAxioms
+# xsd:DateTime needs to be inlined for BSBM
+com.bigdata.rdf.store.AbstractTripleStore.inlineDateTimes=true
+
##
## Sail options.
##
-com.bigdata.rdf.sail.BigdataSail.truthMaintenance=false
+com.bigdata.rdf.sail.truthMaintenance=false
# 10000 is default.
-com.bigdata.rdf.sail.BigdataSail.bufferCapacity=100000
+com.bigdata.rdf.sail.bufferCapacity=100000
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-13 15:57:51
|
Revision: 4084
http://bigdata.svn.sourceforge.net/bigdata/?rev=4084&view=rev
Author: thompsonbry
Date: 2011-01-13 15:57:45 +0000 (Thu, 13 Jan 2011)
Log Message:
-----------
made logger private, removed static INFO field
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/counters/PIDUtil.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/counters/PIDUtil.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/counters/PIDUtil.java 2011-01-13 15:54:48 UTC (rev 4083)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/counters/PIDUtil.java 2011-01-13 15:57:45 UTC (rev 4084)
@@ -48,10 +48,10 @@
*/
public class PIDUtil {
- static protected final Logger log = Logger
+ static private final Logger log = Logger
.getLogger(PIDUtil.class);
- static protected final boolean INFO = log.isInfoEnabled();
+// static protected final boolean INFO = log.isInfoEnabled();
/**
* Return the PID of the Java VM under Linux using bash.
@@ -156,7 +156,7 @@
final int pid = Integer.parseInt(matcher.group(1));
- if(INFO)
+ if(log.isInfoEnabled())
log.info("pid=" + pid);
return pid;
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-13 15:54:55
|
Revision: 4083
http://bigdata.svn.sourceforge.net/bigdata/?rev=4083&view=rev
Author: thompsonbry
Date: 2011-01-13 15:54:48 +0000 (Thu, 13 Jan 2011)
Log Message:
-----------
Modified the public DataLoader methods which accept a Reader or InputStream such that they do not close the Reader or InputStream.
Modified Paths:
--------------
branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java
branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rio/TestRDFXMLInterchangeWithStatementIdentifiers.java
branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestDatabaseAtOnceClosure.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphOnLubm.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rio/TestRDFXMLInterchangeWithStatementIdentifiers.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestDatabaseAtOnceClosure.java
Modified: branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java 2011-01-13 15:29:14 UTC (rev 4082)
+++ branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java 2011-01-13 15:54:48 UTC (rev 4083)
@@ -637,7 +637,7 @@
public LoadStats loadData(final Reader reader, final String baseURL,
final RDFFormat rdfFormat) throws IOException {
- try {
+// try {
final LoadStats totals = new LoadStats();
@@ -645,11 +645,11 @@
return totals;
- } finally {
-
- reader.close();
-
- }
+// } finally {
+//
+// reader.close();
+//
+// }
}
@@ -665,7 +665,7 @@
public LoadStats loadData(final InputStream is, final String baseURL,
final RDFFormat rdfFormat) throws IOException {
- try {
+// try {
final LoadStats totals = new LoadStats();
@@ -673,11 +673,11 @@
return totals;
- } finally {
-
- is.close();
-
- }
+// } finally {
+//
+// is.close();
+//
+// }
}
Modified: branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rio/TestRDFXMLInterchangeWithStatementIdentifiers.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rio/TestRDFXMLInterchangeWithStatementIdentifiers.java 2011-01-13 15:29:14 UTC (rev 4082)
+++ branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rio/TestRDFXMLInterchangeWithStatementIdentifiers.java 2011-01-13 15:54:48 UTC (rev 4083)
@@ -48,6 +48,7 @@
package com.bigdata.rdf.rio;
import java.io.IOException;
+import java.io.Reader;
import java.io.StringReader;
import java.io.StringWriter;
import java.io.Writer;
@@ -514,8 +515,13 @@
log.info("Reading RDF/XML into temp store.");
- tempStore.getDataLoader().loadData(new StringReader(rdfXml),
+ final Reader r = new StringReader(rdfXml);
+ try {
+ tempStore.getDataLoader().loadData(r,
""/* baseURL */, RDFFormat.RDFXML);
+ } finally {
+ r.close();
+ }
/*
* Verify the structure of the graph.
@@ -688,8 +694,13 @@
log.info("Reading RDF/XML into temp store.");
- tempStore.getDataLoader().loadData(new StringReader(rdfXml),
+ final Reader r = new StringReader(rdfXml);
+ try {
+ tempStore.getDataLoader().loadData(r,
""/* baseURL */, RDFFormat.RDFXML);
+ } finally {
+ r.close();
+ }
/*
* Re-define the vocabulary so that it does not use the term
Modified: branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestDatabaseAtOnceClosure.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestDatabaseAtOnceClosure.java 2011-01-13 15:29:14 UTC (rev 4082)
+++ branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestDatabaseAtOnceClosure.java 2011-01-13 15:54:48 UTC (rev 4083)
@@ -752,8 +752,12 @@
}
+ try {
closureStore.getDataLoader().loadData(is, baseURI,
RDFFormat.RDFXML);
+ } finally {
+ is.close();
+ }
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java 2011-01-13 15:29:14 UTC (rev 4082)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java 2011-01-13 15:54:48 UTC (rev 4083)
@@ -637,7 +637,7 @@
public LoadStats loadData(final Reader reader, final String baseURL,
final RDFFormat rdfFormat) throws IOException {
- try {
+// try {
final LoadStats totals = new LoadStats();
@@ -645,11 +645,11 @@
return totals;
- } finally {
-
- reader.close();
-
- }
+// } finally {
+//
+// reader.close();
+//
+// }
}
@@ -665,7 +665,7 @@
public LoadStats loadData(final InputStream is, final String baseURL,
final RDFFormat rdfFormat) throws IOException {
- try {
+// try {
final LoadStats totals = new LoadStats();
@@ -673,11 +673,11 @@
return totals;
- } finally {
-
- is.close();
-
- }
+// } finally {
+//
+// is.close();
+//
+// }
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphOnLubm.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphOnLubm.java 2011-01-13 15:29:14 UTC (rev 4082)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphOnLubm.java 2011-01-13 15:54:48 UTC (rev 4083)
@@ -380,9 +380,18 @@
}
- final LoadStats tmp = closureStore.getDataLoader()
+ final LoadStats tmp;
+ try {
+
+ tmp = closureStore.getDataLoader()
.loadData(is, baseURI, RDFFormat.RDFXML);
-
+
+ } finally {
+
+ is.close();
+
+ }
+
totals.add(tmp);
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rio/TestRDFXMLInterchangeWithStatementIdentifiers.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rio/TestRDFXMLInterchangeWithStatementIdentifiers.java 2011-01-13 15:29:14 UTC (rev 4082)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rio/TestRDFXMLInterchangeWithStatementIdentifiers.java 2011-01-13 15:54:48 UTC (rev 4083)
@@ -48,6 +48,7 @@
package com.bigdata.rdf.rio;
import java.io.IOException;
+import java.io.Reader;
import java.io.StringReader;
import java.io.StringWriter;
import java.io.Writer;
@@ -514,8 +515,13 @@
log.info("Reading RDF/XML into temp store.");
- tempStore.getDataLoader().loadData(new StringReader(rdfXml),
+ final Reader r = new StringReader(rdfXml);
+ try {
+ tempStore.getDataLoader().loadData(r,
""/* baseURL */, RDFFormat.RDFXML);
+ } finally {
+ r.close();
+ }
/*
* Verify the structure of the graph.
@@ -688,8 +694,13 @@
log.info("Reading RDF/XML into temp store.");
- tempStore.getDataLoader().loadData(new StringReader(rdfXml),
+ final Reader r = new StringReader(rdfXml);
+ try {
+ tempStore.getDataLoader().loadData(r,
""/* baseURL */, RDFFormat.RDFXML);
+ } finally {
+ r.close();
+ }
/*
* Re-define the vocabulary so that it does not use the term
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestDatabaseAtOnceClosure.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestDatabaseAtOnceClosure.java 2011-01-13 15:29:14 UTC (rev 4082)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/rules/TestDatabaseAtOnceClosure.java 2011-01-13 15:54:48 UTC (rev 4083)
@@ -771,8 +771,16 @@
}
+ try {
+
closureStore.getDataLoader().loadData(is, baseURI,
RDFFormat.RDFXML);
+
+ } finally {
+
+ is.close();
+
+ }
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-13 15:29:20
|
Revision: 4082
http://bigdata.svn.sourceforge.net/bigdata/?rev=4082&view=rev
Author: thompsonbry
Date: 2011-01-13 15:29:14 +0000 (Thu, 13 Jan 2011)
Log Message:
-----------
Made loadData3(...) public.
Modified Paths:
--------------
branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java
Modified: branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java 2011-01-13 15:29:00 UTC (rev 4081)
+++ branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java 2011-01-13 15:29:14 UTC (rev 4082)
@@ -941,14 +941,26 @@
* Loads data from the <i>source</i>. The caller is responsible for closing
* the <i>source</i> if there is an error.
*
+ * @param totals
+ * Used to report out the total {@link LoadStats}.
* @param source
* A {@link Reader} or {@link InputStream}.
* @param baseURL
+ * The baseURI (optional, when not specified the name of the each
+ * file load is converted to a URL and used as the baseURI for
+ * that file).
* @param rdfFormat
+ * The format of the file (optional, when not specified the
+ * format is deduced for each file in turn using the
+ * {@link RDFFormat} static methods).
+ * @param defaultGraph
+ * The value that will be used for the graph/context co-ordinate
+ * when loading data represented in a triple format into a quad
+ * store.
* @param endOfBatch
- * @return
+ * Signal indicates the end of a batch.
*/
- protected void loadData3(final LoadStats totals, final Object source,
+ public void loadData3(final LoadStats totals, final Object source,
final String baseURL, final RDFFormat rdfFormat,
final String defaultGraph, final boolean endOfBatch) throws IOException {
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-13 15:29:06
|
Revision: 4081
http://bigdata.svn.sourceforge.net/bigdata/?rev=4081&view=rev
Author: thompsonbry
Date: 2011-01-13 15:29:00 +0000 (Thu, 13 Jan 2011)
Log Message:
-----------
Made loadData3(...) public.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java 2011-01-13 14:27:28 UTC (rev 4080)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java 2011-01-13 15:29:00 UTC (rev 4081)
@@ -948,19 +948,31 @@
}
}
-
+
/**
* Loads data from the <i>source</i>. The caller is responsible for closing
* the <i>source</i> if there is an error.
*
+ * @param totals
+ * Used to report out the total {@link LoadStats}.
* @param source
* A {@link Reader} or {@link InputStream}.
* @param baseURL
+ * The baseURI (optional, when not specified the name of the each
+ * file load is converted to a URL and used as the baseURI for
+ * that file).
* @param rdfFormat
+ * The format of the file (optional, when not specified the
+ * format is deduced for each file in turn using the
+ * {@link RDFFormat} static methods).
+ * @param defaultGraph
+ * The value that will be used for the graph/context co-ordinate
+ * when loading data represented in a triple format into a quad
+ * store.
* @param endOfBatch
- * @return
+ * Signal indicates the end of a batch.
*/
- protected void loadData3(final LoadStats totals, final Object source,
+ public void loadData3(final LoadStats totals, final Object source,
final String baseURL, final RDFFormat rdfFormat,
final String defaultGraph, final boolean endOfBatch) throws IOException {
@@ -1203,7 +1215,23 @@
* support multiple data files within a single archive.
*
* @param args
- * [-closure][-verbose][-namespace <i>namespace</i>] propertyFile (fileOrDir)+
+ * <code>[-quiet][-closure][-verbose][-namespace <i>namespace</i>] propertyFile (fileOrDir)*</code>
+ * where
+ * <dl>
+ * <dt>-quiet</dt>
+ * <dd>Suppress all stdout messages.</dd>
+ * <dt>-verbose</dt>
+ * <dd>Show additional messages detailing the load performance.</dd>
+ * <dt>-closure</dt>
+ * <dd>Compute the RDF(S)+ closure.</dd>
+ * <dt>-namespace</dt>
+ * <dd>The namespace of the KB instance.</dd>
+ * <dt>propertyFile</dt>
+ * <dd>The configuration file for the database instance.</dd>
+ * <dt>fileOrDir</dt>
+ * <dd>Zero or more files or directories containing the data to
+ * be loaded.</dd>
+ * </dl>
*
* @throws IOException
*/
@@ -1213,6 +1241,7 @@
String namespace = "kb";
boolean doClosure = false;
boolean verbose = false;
+ boolean quiet = false;
RDFFormat rdfFormat = null;
String baseURI = null;
@@ -1243,7 +1272,13 @@
} else if (arg.equals("-verbose")) {
verbose = true;
+ quiet = false;
+ } else if (arg.equals("-quiet")) {
+
+ quiet = true;
+ verbose = false;
+
} else {
System.err.println("Unknown argument: " + arg);
@@ -1282,7 +1317,8 @@
final Properties properties = new Properties();
{
- System.out.println("Reading properties: "+propertyFile);
+ if(!quiet)
+ System.out.println("Reading properties: "+propertyFile);
final InputStream is = new FileInputStream(propertyFile);
try {
properties.load(is);
@@ -1295,7 +1331,7 @@
// // Override/set from the environment.
// final String file = System
// .getProperty(com.bigdata.journal.Options.FILE);
-// System.out.println("Using: " + com.bigdata.journal.Options.FILE
+// if(!quiet) System.out.println("Using: " + com.bigdata.journal.Options.FILE
// + "=" + file);
// properties.setProperty(com.bigdata.journal.Options.FILE, file);
// }
@@ -1323,7 +1359,8 @@
if (System.getProperty(s) != null) {
// Override/set from the environment.
final String v = System.getProperty(s);
- System.out.println("Using: " + s + "=" + v);
+ if(!quiet)
+ System.out.println("Using: " + s + "=" + v);
properties.setProperty(s, v);
}
}
@@ -1342,7 +1379,8 @@
files.add(fileOrDir);
- System.out.println("Will load from: " + fileOrDir);
+ if(!quiet)
+ System.out.println("Will load from: " + fileOrDir);
}
@@ -1357,7 +1395,8 @@
// final long firstOffset = jnl.getRootBlockView().getNextOffset();
final long userData0 = jnl.getBufferStrategy().size();
- System.out.println("Journal file: "+jnl.getFile());
+ if(!quiet)
+ System.out.println("Journal file: "+jnl.getFile());
AbstractTripleStore kb = (AbstractTripleStore) jnl
.getResourceLocator().locate(namespace, ITx.UNISOLATED);
@@ -1388,7 +1427,8 @@
dataLoader.endSource();
- System.out.println("Load: " + totals);
+ if(!quiet)
+ System.out.println("Load: " + totals);
if (dataLoader.closureEnum == ClosureEnum.None && doClosure) {
@@ -1402,11 +1442,16 @@
new StringBuilder()).toString());
}
- System.out.println("Computing closure.");
+ if (!quiet)
+ System.out.println("Computing closure.");
+ log.info("Computing closure.");
final ClosureStats stats = dataLoader.doClosure();
- System.out.println("Closure: "+stats.toString());
+ if(!quiet)
+ System.out.println("Closure: "+stats.toString());
+ if (log.isInfoEnabled())
+ log.info("Closure: " + stats.toString());
}
@@ -1440,12 +1485,16 @@
// #of bytes written (user data only)
final long bytesWritten = (userData1 - userData0);
- System.out.println("Wrote: " + bytesWritten + " bytes.");
+ if (!quiet)
+ System.out.println("Wrote: " + bytesWritten + " bytes.");
- final long elapsedTotal = System.currentTimeMillis() - begin;
-
- System.out.println("Total elapsed=" + elapsedTotal + "ms");
-
+ final long elapsedTotal = System.currentTimeMillis() - begin;
+
+ if (!quiet)
+ System.out.println("Total elapsed=" + elapsedTotal + "ms");
+ if (log.isInfoEnabled())
+ log.info("Total elapsed=" + elapsedTotal + "ms");
+
} finally {
if (jnl != null) {
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-13 14:27:34
|
Revision: 4080
http://bigdata.svn.sourceforge.net/bigdata/?rev=4080&view=rev
Author: thompsonbry
Date: 2011-01-13 14:27:28 +0000 (Thu, 13 Jan 2011)
Log Message:
-----------
Applied changes to the JOURNAL_HA_BRANCH to implement a "-quiet" option which will suppress everything on stdout.
Modified Paths:
--------------
branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java
Modified: branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java
===================================================================
--- branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java 2011-01-12 20:49:29 UTC (rev 4079)
+++ branches/JOURNAL_HA_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/DataLoader.java 2011-01-13 14:27:28 UTC (rev 4080)
@@ -1191,8 +1191,24 @@
* support multiple data files within a single archive.
*
* @param args
- * [-closure][-verbose][-namespace <i>namespace</i>] propertyFile (fileOrDir)+
- *
+ * <code>[-quiet][-closure][-verbose][-namespace <i>namespace</i>] propertyFile (fileOrDir)*</code>
+ * where
+ * <dl>
+ * <dt>-quiet</dt>
+ * <dd>Suppress all stdout messages.</dd>
+ * <dt>-verbose</dt>
+ * <dd>Show additional messages detailing the load performance.</dd>
+ * <dt>-closure</dt>
+ * <dd>Compute the RDF(S)+ closure.</dd>
+ * <dt>-namespace</dt>
+ * <dd>The namespace of the KB instance.</dd>
+ * <dt>propertyFile</dt>
+ * <dd>The configuration file for the database instance.</dd>
+ * <dt>fileOrDir</dt>
+ * <dd>Zero or more files or directories containing the data to
+ * be loaded.</dd>
+ * </dl>
+ *
* @throws IOException
*/
public static void main(final String[] args) throws IOException {
@@ -1201,6 +1217,7 @@
String namespace = "kb";
boolean doClosure = false;
boolean verbose = false;
+ boolean quiet = false;
RDFFormat rdfFormat = null;
String baseURI = null;
@@ -1231,7 +1248,13 @@
} else if (arg.equals("-verbose")) {
verbose = true;
+ quiet = false;
+ } else if (arg.equals("-quiet")) {
+
+ quiet = true;
+ verbose = false;
+
} else {
System.err.println("Unknown argument: " + arg);
@@ -1270,7 +1293,8 @@
final Properties properties = new Properties();
{
- System.out.println("Reading properties: "+propertyFile);
+ if(!quiet)
+ System.out.println("Reading properties: "+propertyFile);
final InputStream is = new FileInputStream(propertyFile);
try {
properties.load(is);
@@ -1283,7 +1307,7 @@
// // Override/set from the environment.
// final String file = System
// .getProperty(com.bigdata.journal.Options.FILE);
-// System.out.println("Using: " + com.bigdata.journal.Options.FILE
+// if(!quiet) System.out.println("Using: " + com.bigdata.journal.Options.FILE
// + "=" + file);
// properties.setProperty(com.bigdata.journal.Options.FILE, file);
// }
@@ -1311,7 +1335,8 @@
if (System.getProperty(s) != null) {
// Override/set from the environment.
final String v = System.getProperty(s);
- System.out.println("Using: " + s + "=" + v);
+ if(!quiet)
+ System.out.println("Using: " + s + "=" + v);
properties.setProperty(s, v);
}
}
@@ -1330,7 +1355,8 @@
files.add(fileOrDir);
- System.out.println("Will load from: " + fileOrDir);
+ if(!quiet)
+ System.out.println("Will load from: " + fileOrDir);
}
@@ -1345,7 +1371,8 @@
// final long firstOffset = jnl.getRootBlockView().getNextOffset();
final long userData0 = jnl.getBufferStrategy().size();
- System.out.println("Journal file: "+jnl.getFile());
+ if(!quiet)
+ System.out.println("Journal file: "+jnl.getFile());
AbstractTripleStore kb = (AbstractTripleStore) jnl
.getResourceLocator().locate(namespace, ITx.UNISOLATED);
@@ -1376,7 +1403,8 @@
dataLoader.endSource();
- System.out.println("Load: " + totals);
+ if(!quiet)
+ System.out.println("Load: " + totals);
if (dataLoader.closureEnum == ClosureEnum.None && doClosure) {
@@ -1390,11 +1418,16 @@
new StringBuilder()).toString());
}
- System.out.println("Computing closure.");
+ if(!quiet)
+ System.out.println("Computing closure.");
+ log.info("Computing closure");
final ClosureStats stats = dataLoader.doClosure();
- System.out.println("Closure: "+stats.toString());
+ if(!quiet)
+ System.out.println("Closure: "+stats.toString());
+ if(log.isInfoEnabled())
+ log.info("Closure: "+stats.toString());
}
@@ -1428,12 +1461,16 @@
// #of bytes written (user data only)
final long bytesWritten = (userData1 - userData0);
- System.out.println("Wrote: " + bytesWritten + " bytes.");
+ if(!quiet)
+ System.out.println("Wrote: " + bytesWritten + " bytes.");
final long elapsedTotal = System.currentTimeMillis() - begin;
+
+ if (!quiet)
+ System.out.println("Total elapsed=" + elapsedTotal + "ms");
+ if (log.isInfoEnabled())
+ log.info("Total elapsed=" + elapsedTotal + "ms");
- System.out.println("Total elapsed=" + elapsedTotal + "ms");
-
} finally {
if (jnl != null) {
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mrp...@us...> - 2011-01-12 20:49:35
|
Revision: 4079
http://bigdata.svn.sourceforge.net/bigdata/?rev=4079&view=rev
Author: mrpersonick
Date: 2011-01-12 20:49:29 +0000 (Wed, 12 Jan 2011)
Log Message:
-----------
inline date times back to false
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java 2011-01-12 19:37:57 UTC (rev 4078)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java 2011-01-12 20:49:29 UTC (rev 4079)
@@ -912,7 +912,7 @@
String INLINE_DATE_TIMES = AbstractTripleStore.class.getName()
+ ".inlineDateTimes";
- String DEFAULT_INLINE_DATE_TIMES = "true";
+ String DEFAULT_INLINE_DATE_TIMES = "false";
/**
* The name of the {@link IExtensionFactory} class. The implementation
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mrp...@us...> - 2011-01-12 19:38:04
|
Revision: 4078
http://bigdata.svn.sourceforge.net/bigdata/?rev=4078&view=rev
Author: mrpersonick
Date: 2011-01-12 19:37:57 +0000 (Wed, 12 Jan 2011)
Log Message:
-----------
allowing non-native Sesame filters to run
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOpTreeBuilder.java
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestSesameFilters.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java 2011-01-12 18:25:21 UTC (rev 4077)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java 2011-01-12 19:37:57 UTC (rev 4078)
@@ -548,7 +548,7 @@
* bigdata constraint, we can run it as a FilterIterator after the
* query has run natively.
*/
- final Collection<ValueExpr> sesameFilters = new LinkedList<ValueExpr>();
+ final Collection<Filter> sesameFilters = new LinkedList<Filter>();
/*
* We need to prune Sesame filters that we cannot translate into native
@@ -563,7 +563,19 @@
for (SOp sop : sopTree) {
final QueryModelNode op = sop.getOperator();
if (op instanceof ValueExpr) {
+ /*
+ * If we have a raw ValueExpr and not a Filter we know it must
+ * be the condition of a LeftJoin, in which case we cannot
+ * use the Sesame FilterIterator to safely evaluate it. A
+ * UnsupportedOperatorException here must just flow through
+ * to Sesame evaluation of the entire query.
+ */
final ValueExpr ve = (ValueExpr) op;
+ final IConstraint bop = toConstraint(ve);
+ sop.setBOp(bop);
+ } else if (op instanceof Filter) {
+ final Filter filter = (Filter) op;
+ final ValueExpr ve = filter.getCondition();
try {
final IConstraint bop = toConstraint(ve);
sop.setBOp(bop);
@@ -580,7 +592,7 @@
*/
if (sop.getGroup() == SOpTreeBuilder.ROOT_GROUP_ID) {
sopsToPrune.add(sop);
- sesameFilters.add(ve);
+ sesameFilters.add(filter);
} else {
throw ex;
}
@@ -635,7 +647,7 @@
protected CloseableIteration<BindingSet, QueryEvaluationException>
_evaluateNatively(final PipelineOp query, final BindingSet bs,
final QueryEngine queryEngine,
- final Collection<ValueExpr> sesameConstraints)
+ final Collection<Filter> sesameFilters)
throws QueryEvaluationException {
try {
@@ -658,10 +670,12 @@
runningQuery.get();
// use the basic filter iterator for remaining filters
- if (sesameConstraints != null) {
- for (ValueExpr ve : sesameConstraints) {
- final Filter filter = new Filter(null, ve);
- result = new FilterIterator(filter, result, this);
+ if (sesameFilters != null) {
+ for (Filter f : sesameFilters) {
+ if (log.isDebugEnabled()) {
+ log.debug("attaching sesame filter: " + f);
+ }
+ result = new FilterIterator(f, result, this);
}
}
@@ -1093,7 +1107,7 @@
}
protected IVariable[] gatherRequiredVariables(final TupleExpr root,
- final Collection<ValueExpr> sesameFilters) {
+ final Collection<Filter> sesameFilters) {
/*
* Collect a set of variables required beyond just the join (i.e.
@@ -1116,8 +1130,8 @@
}
if (sesameFilters.size() > 0) {
- for (ValueExpr ve : sesameFilters) {
- required.addAll(collectVariables((UnaryTupleOperator) ve));
+ for (Filter f : sesameFilters) {
+ required.addAll(collectVariables(f.getCondition()));
}
}
@@ -1140,48 +1154,47 @@
* they can be added to the list of required variables in the query for
* correct binding set pruning.
*
- * @param uto
+ * @param op
* the <code>UnaryTupleOperator</code>
* @return
* the variables it uses
*/
- protected Set<String> collectVariables(UnaryTupleOperator uto) {
+ protected Set<String> collectVariables(final QueryModelNode op) {
final Set<String> vars = new HashSet<String>();
- if (uto instanceof Projection) {
- List<ProjectionElem> elems =
- ((Projection) uto).getProjectionElemList().getElements();
+ if (op instanceof Projection) {
+ final List<ProjectionElem> elems =
+ ((Projection) op).getProjectionElemList().getElements();
for (ProjectionElem elem : elems) {
vars.add(elem.getSourceName());
}
- } else if (uto instanceof MultiProjection) {
- List<ProjectionElemList> elemLists =
- ((MultiProjection) uto).getProjections();
+ } else if (op instanceof MultiProjection) {
+ final List<ProjectionElemList> elemLists =
+ ((MultiProjection) op).getProjections();
for (ProjectionElemList list : elemLists) {
List<ProjectionElem> elems = list.getElements();
for (ProjectionElem elem : elems) {
vars.add(elem.getSourceName());
}
}
- } else if (uto instanceof Filter) {
- Filter f = (Filter) uto;
- ValueExpr ve = f.getCondition();
+ } else if (op instanceof ValueExpr) {
+ final ValueExpr ve = (ValueExpr) op;
ve.visit(new QueryModelVisitorBase<RuntimeException>() {
@Override
public void meet(Var v) {
vars.add(v.getName());
}
});
- } else if (uto instanceof Group) {
- Group g = (Group) uto;
+ } else if (op instanceof Group) {
+ final Group g = (Group) op;
g.visit(new QueryModelVisitorBase<RuntimeException>() {
@Override
public void meet(Var v) {
vars.add(v.getName());
}
});
- } else if (uto instanceof Order) {
- Order o = (Order) uto;
+ } else if (op instanceof Order) {
+ final Order o = (Order) op;
o.visit(new QueryModelVisitorBase<RuntimeException>() {
@Override
public void meet(Var v) {
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOpTreeBuilder.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOpTreeBuilder.java 2011-01-12 18:25:21 UTC (rev 4077)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOpTreeBuilder.java 2011-01-12 19:37:57 UTC (rev 4078)
@@ -222,7 +222,7 @@
final ValueExpr ve = filter.getCondition();
// make a constraint, attach it to the rule
if (ve != null) {
- sops.add(new SOp(sopId.incrementAndGet(), ve, g, pg, rslj));
+ sops.add(new SOp(sopId.incrementAndGet(), filter, g, pg, rslj));
}
final TupleExpr arg = filter.getArg();
Added: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestSesameFilters.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestSesameFilters.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestSesameFilters.java 2011-01-12 19:37:57 UTC (rev 4078)
@@ -0,0 +1,195 @@
+/**
+Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on Sep 16, 2009
+ */
+
+package com.bigdata.rdf.sail;
+
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.log4j.Logger;
+import org.openrdf.model.Literal;
+import org.openrdf.model.URI;
+import org.openrdf.model.ValueFactory;
+import org.openrdf.model.vocabulary.RDF;
+import org.openrdf.model.vocabulary.RDFS;
+import org.openrdf.query.BindingSet;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.query.algebra.Projection;
+import org.openrdf.query.algebra.QueryRoot;
+import org.openrdf.query.algebra.TupleExpr;
+import org.openrdf.query.impl.BindingImpl;
+import org.openrdf.repository.RepositoryConnection;
+import org.openrdf.repository.sail.SailTupleQuery;
+
+import com.bigdata.bop.BOpUtility;
+import com.bigdata.bop.PipelineOp;
+import com.bigdata.bop.engine.QueryEngine;
+import com.bigdata.rdf.axioms.NoAxioms;
+import com.bigdata.rdf.sail.sop.SOp;
+import com.bigdata.rdf.sail.sop.SOp2BOpUtility;
+import com.bigdata.rdf.sail.sop.SOpTree;
+import com.bigdata.rdf.sail.sop.SOpTree.SOpGroup;
+import com.bigdata.rdf.sail.sop.SOpTree.SOpGroups;
+import com.bigdata.rdf.sail.sop.SOpTreeBuilder;
+import com.bigdata.rdf.store.AbstractTripleStore;
+import com.bigdata.rdf.store.BD;
+import com.bigdata.rdf.vocab.NoVocabulary;
+
+public class TestSesameFilters extends ProxyBigdataSailTestCase {
+
+ protected static final Logger log = Logger.getLogger(TestSesameFilters.class);
+
+ protected static final boolean INFO = log.isInfoEnabled();
+
+ @Override
+ public Properties getProperties() {
+
+ Properties props = super.getProperties();
+
+ props.setProperty(BigdataSail.Options.AXIOMS_CLASS, NoAxioms.class.getName());
+ props.setProperty(BigdataSail.Options.VOCABULARY_CLASS, NoVocabulary.class.getName());
+ props.setProperty(BigdataSail.Options.TRUTH_MAINTENANCE, "false");
+ props.setProperty(BigdataSail.Options.JUSTIFY, "false");
+ props.setProperty(BigdataSail.Options.TEXT_INDEX, "false");
+
+ return props;
+
+ }
+
+ /**
+ *
+ */
+ public TestSesameFilters() {
+ }
+
+ /**
+ * @param arg0
+ */
+ public TestSesameFilters(String arg0) {
+ super(arg0);
+ }
+
+ public void testRegex() throws Exception {
+
+// final Sail sail = new MemoryStore();
+// sail.initialize();
+// final Repository repo = new SailRepository(sail);
+
+ final BigdataSail sail = getSail();
+ sail.initialize();
+ final BigdataSailRepository repo = new BigdataSailRepository(sail);
+
+ final RepositoryConnection cxn = repo.getConnection();
+ cxn.setAutoCommit(false);
+
+ try {
+
+ final ValueFactory vf = sail.getValueFactory();
+
+ /*
+ * Create some terms.
+ */
+ final URI mike = vf.createURI(BD.NAMESPACE + "mike");
+ final URI bryan = vf.createURI(BD.NAMESPACE + "bryan");
+ final URI person = vf.createURI(BD.NAMESPACE + "Person");
+ final Literal l1 = vf.createLiteral("mike personick");
+ final Literal l2 = vf.createLiteral("bryan thompson");
+
+ /*
+ * Create some statements.
+ */
+ cxn.add(mike, RDF.TYPE, person);
+ cxn.add(mike, RDFS.LABEL, l1);
+ cxn.add(bryan, RDF.TYPE, person);
+ cxn.add(bryan, RDFS.LABEL, l2);
+
+ /*
+ * Note: The either flush() or commit() is required to flush the
+ * statement buffers to the database before executing any operations
+ * that go around the sail.
+ */
+ cxn.commit();
+
+ {
+
+ String query =
+ "prefix bd: <"+BD.NAMESPACE+"> " +
+ "prefix rdf: <"+RDF.NAMESPACE+"> " +
+ "prefix rdfs: <"+RDFS.NAMESPACE+"> " +
+ "select * " +
+ "where { " +
+ " ?s rdf:type bd:Person . " +
+ " ?s rdfs:label ?label . " +
+ " FILTER regex(?label, \"mike\") . " +
+ "}";
+
+ final SailTupleQuery tupleQuery = (SailTupleQuery)
+ cxn.prepareTupleQuery(QueryLanguage.SPARQL, query);
+ tupleQuery.setIncludeInferred(false /* includeInferred */);
+
+ if (log.isInfoEnabled()) {
+
+ final BigdataSailTupleQuery bdTupleQuery =
+ (BigdataSailTupleQuery) tupleQuery;
+ final QueryRoot root = (QueryRoot) bdTupleQuery.getTupleExpr();
+ final Projection p = (Projection) root.getArg();
+ final TupleExpr tupleExpr = p.getArg();
+ final SOpTreeBuilder stb = new SOpTreeBuilder();
+ final SOpTree tree = stb.collectSOps(tupleExpr);
+
+ log.info(tree);
+ log.info(query);
+
+ final TupleQueryResult result = tupleQuery.evaluate();
+ while (result.hasNext()) {
+ log.info(result.next());
+ }
+
+ }
+
+ final Collection<BindingSet> answer = new LinkedList<BindingSet>();
+ answer.add(createBindingSet(
+ new BindingImpl("s", mike),
+ new BindingImpl("label", l1)
+ ));
+
+ final TupleQueryResult result = tupleQuery.evaluate();
+ compare(result, answer);
+
+ }
+
+ } finally {
+ cxn.close();
+ sail.shutDown();
+ }
+
+ }
+
+}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mrp...@us...> - 2011-01-12 18:25:28
|
Revision: 4077
http://bigdata.svn.sourceforge.net/bigdata/?rev=4077&view=rev
Author: mrpersonick
Date: 2011-01-12 18:25:21 +0000 (Wed, 12 Jan 2011)
Log Message:
-----------
changed to work with termIds when the compare op is NE or EQ
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/CompareBOp.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/CompareBOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/CompareBOp.java 2011-01-12 18:20:33 UTC (rev 4076)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/CompareBOp.java 2011-01-12 18:25:21 UTC (rev 4077)
@@ -92,6 +92,13 @@
if (left == null || right == null)
return true; // not yet bound.
+ final CompareOp op = (CompareOp) getProperty(Annotations.OP);
+
+ if (left.isTermId() && right.isTermId() &&
+ (op == CompareOp.EQ || op == CompareOp.NE)) {
+ return _accept(left.compareTo(right));
+ }
+
if (!IVUtility.canNumericalCompare(left))
throw new NotNumericalException("cannot numerical compare: " + left);
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mrp...@us...> - 2011-01-12 18:20:39
|
Revision: 4076
http://bigdata.svn.sourceforge.net/bigdata/?rev=4076&view=rev
Author: mrpersonick
Date: 2011-01-12 18:20:33 +0000 (Wed, 12 Jan 2011)
Log Message:
-----------
working on nested optional groups
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedOptionals.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedOptionals.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedOptionals.java 2011-01-12 18:17:51 UTC (rev 4075)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedOptionals.java 2011-01-12 18:20:33 UTC (rev 4076)
@@ -404,50 +404,50 @@
// sail.initialize();
// final Repository repo = new SailRepository(sail);
- final BigdataSail sail = getSail();
- sail.initialize();
- final BigdataSailRepository repo = new BigdataSailRepository(sail);
-
- final RepositoryConnection cxn = repo.getConnection();
- cxn.setAutoCommit(false);
-
- try {
-
- final ValueFactory vf = sail.getValueFactory();
+ final BigdataSail sail = getSail();
+ sail.initialize();
+ final BigdataSailRepository repo = new BigdataSailRepository(sail);
- /*
- * Create some terms.
- */
- final URI john = vf.createURI(BD.NAMESPACE + "john");
- final URI mary = vf.createURI(BD.NAMESPACE + "mary");
- final URI leon = vf.createURI(BD.NAMESPACE + "leon");
- final URI paul = vf.createURI(BD.NAMESPACE + "paul");
- final URI brad = vf.createURI(BD.NAMESPACE + "brad");
- final URI fred = vf.createURI(BD.NAMESPACE + "fred");
- final URI knows = vf.createURI(BD.NAMESPACE + "knows");
+ final RepositoryConnection cxn = repo.getConnection();
+ cxn.setAutoCommit(false);
- /*
- * Create some statements.
- */
- cxn.add(paul, knows, mary);
- cxn.add(paul, knows, brad);
-
- cxn.add(john, knows, mary);
- cxn.add(john, knows, brad);
+ try {
- cxn.add(mary, knows, brad);
- cxn.add(brad, knows, fred);
- cxn.add(brad, knows, leon);
+ final ValueFactory vf = sail.getValueFactory();
- /*
- * Note: The either flush() or commit() is required to flush the
- * statement buffers to the database before executing any operations
- * that go around the sail.
- */
- cxn.commit();
-
- {
-
+ /*
+ * Create some terms.
+ */
+ final URI john = vf.createURI(BD.NAMESPACE + "john");
+ final URI mary = vf.createURI(BD.NAMESPACE + "mary");
+ final URI leon = vf.createURI(BD.NAMESPACE + "leon");
+ final URI paul = vf.createURI(BD.NAMESPACE + "paul");
+ final URI brad = vf.createURI(BD.NAMESPACE + "brad");
+ final URI fred = vf.createURI(BD.NAMESPACE + "fred");
+ final URI knows = vf.createURI(BD.NAMESPACE + "knows");
+
+ /*
+ * Create some statements.
+ */
+ cxn.add(paul, knows, mary);
+ cxn.add(paul, knows, brad);
+
+ cxn.add(john, knows, mary);
+ cxn.add(john, knows, brad);
+
+ cxn.add(mary, knows, brad);
+ cxn.add(brad, knows, fred);
+ cxn.add(brad, knows, leon);
+
+ /*
+ * Note: The either flush() or commit() is required to flush the
+ * statement buffers to the database before executing any operations
+ * that go around the sail.
+ */
+ cxn.commit();
+
+ {
+
String query =
"prefix bd: <"+BD.NAMESPACE+"> " +
"select * " +
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
4 messages has been excluded from this view by a project administrator.
