Menu
▾
▴
bigdata-commit — Commit traffic.
This list is closed, nobody may subscribe to it.
| 2010 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(139) |
Aug
(94) |
Sep
(232) |
Oct
(143) |
Nov
(138) |
Dec
(55) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2011 |
Jan
(127) |
Feb
(90) |
Mar
(101) |
Apr
(74) |
May
(148) |
Jun
(241) |
Jul
(169) |
Aug
(121) |
Sep
(157) |
Oct
(199) |
Nov
(281) |
Dec
(75) |
| 2012 |
Jan
(107) |
Feb
(122) |
Mar
(184) |
Apr
(73) |
May
(14) |
Jun
(49) |
Jul
(26) |
Aug
(103) |
Sep
(133) |
Oct
(61) |
Nov
(51) |
Dec
(55) |
| 2013 |
Jan
(59) |
Feb
(72) |
Mar
(99) |
Apr
(62) |
May
(92) |
Jun
(19) |
Jul
(31) |
Aug
(138) |
Sep
(47) |
Oct
(83) |
Nov
(95) |
Dec
(111) |
| 2014 |
Jan
(125) |
Feb
(60) |
Mar
(119) |
Apr
(136) |
May
(270) |
Jun
(83) |
Jul
(88) |
Aug
(30) |
Sep
(47) |
Oct
(27) |
Nov
(23) |
Dec
|
| 2015 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(3) |
Oct
|
Nov
|
Dec
|
| 2016 |
Jan
|
Feb
|
Mar
(4) |
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: <tho...@us...> - 2011-02-02 19:00:07
|
Revision: 4175
http://bigdata.svn.sourceforge.net/bigdata/?rev=4175&view=rev
Author: thompsonbry
Date: 2011-02-02 18:59:56 +0000 (Wed, 02 Feb 2011)
Log Message:
-----------
added a step to the ant junit task to archive the generated HTML pages as a tarball. This is in support of hudson builds on non-master nodes.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/build.xml
Modified: branches/QUADS_QUERY_BRANCH/build.xml
===================================================================
--- branches/QUADS_QUERY_BRANCH/build.xml 2011-02-02 15:26:55 UTC (rev 4174)
+++ branches/QUADS_QUERY_BRANCH/build.xml 2011-02-02 18:59:56 UTC (rev 4175)
@@ -1747,6 +1747,9 @@
<report format="frames" todir="${test.results.dir}/report" />
</junitreport>
+ <!-- Archive the generated HTML report. -->
+ <tar destfile="${test.results.dir}/report.tgz" basedir="${test.results.dir}/report" compression="gzip"/>
+
</target>
<target name="clean-sparql-test-suite" description="delete the files unpacked by the Sesame SPARQL test suite.">
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-02-02 15:27:01
|
Revision: 4174
http://bigdata.svn.sourceforge.net/bigdata/?rev=4174&view=rev
Author: thompsonbry
Date: 2011-02-02 15:26:55 +0000 (Wed, 02 Feb 2011)
Log Message:
-----------
Creating a branch to work on https://sourceforge.net/apps/trac/bigdata/ticket/109
Added Paths:
-----------
branches/LARGE_LITERALS_REFACTOR/
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-02-02 15:01:57
|
Revision: 4173
http://bigdata.svn.sourceforge.net/bigdata/?rev=4173&view=rev
Author: thompsonbry
Date: 2011-02-02 15:01:51 +0000 (Wed, 02 Feb 2011)
Log Message:
-----------
Bad branch.
Removed Paths:
-------------
branches/LEXICON_REFACTOR2_BRANCH/
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-02-02 14:30:00
|
Revision: 4172
http://bigdata.svn.sourceforge.net/bigdata/?rev=4172&view=rev
Author: thompsonbry
Date: 2011-02-02 14:29:54 +0000 (Wed, 02 Feb 2011)
Log Message:
-----------
Modified the startZookeeper target to spawn the process since control was not returning to ant under Centos.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/build.xml
Modified: branches/QUADS_QUERY_BRANCH/build.xml
===================================================================
--- branches/QUADS_QUERY_BRANCH/build.xml 2011-02-01 20:23:34 UTC (rev 4171)
+++ branches/QUADS_QUERY_BRANCH/build.xml 2011-02-02 14:29:54 UTC (rev 4172)
@@ -1504,11 +1504,14 @@
</java>
</target>
+ <!-- Note: It appears to be necessary to spawn the script which starts -->
+ <!-- zookeeper under some OS platforms (for example, Centos 5.4) even -->
+ <!-- though it will start without being spawned under others (OSX). -->
<target name="startZookeeper">
<echo message="test.zookeeper.installDir=${test.zookeeper.installDir}"/>
<echo>bin/zkServer.sh start
</echo>
- <exec executable="bin/zkServer.sh" dir="${test.zookeeper.installDir}" logerror="true">
+ <exec executable="bin/zkServer.sh" dir="${test.zookeeper.installDir}" spawn="true">
<arg value="start"/>
</exec>
</target>
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-02-01 20:23:40
|
Revision: 4171
http://bigdata.svn.sourceforge.net/bigdata/?rev=4171&view=rev
Author: thompsonbry
Date: 2011-02-01 20:23:34 +0000 (Tue, 01 Feb 2011)
Log Message:
-----------
This is a branch to address [1], which covers handling of large literals and eliminating the ID2TERM index in favor of direct addressing of URIs and of those literals not amenable to inlining within the statement indices. This is being done within a branch because the change set will involve a change in the physical schema of the lexicon, dropping the ID2TERM index, and changing how the data in the TERM2ID index are represented. By creating the change set in a branch we will be able to evaluate the impact of this change on the storage file size, the IO profile when loading data, and the IO profile and SPARQL performance during query answering.
[1] https://sourceforge.net/apps/trac/bigdata/ticket/109
Added Paths:
-----------
branches/LEXICON_REFACTOR2_BRANCH/
branches/LEXICON_REFACTOR2_BRANCH/LEXICON_REFACTOR2_BRANCH/
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-29 18:10:27
|
Revision: 4170
http://bigdata.svn.sourceforge.net/bigdata/?rev=4170&view=rev
Author: thompsonbry
Date: 2011-01-29 18:10:21 +0000 (Sat, 29 Jan 2011)
Log Message:
-----------
Modified the WORMStrategy constructor to pass in the current value for the initialExtent rather than the file extent per https://sourceforge.net/apps/trac/bigdata/ticket/236.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/WORMStrategy.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/WORMStrategy.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/WORMStrategy.java 2011-01-27 20:01:36 UTC (rev 4169)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/journal/WORMStrategy.java 2011-01-29 18:10:21 UTC (rev 4170)
@@ -821,7 +821,12 @@
final FileMetadata fileMetadata,
final Quorum<?, ?> quorum) {
- super(fileMetadata.extent, maximumExtent, fileMetadata.offsetBits,
+ super(
+ // @see https://sourceforge.net/apps/trac/bigdata/ticket/236
+ Long.valueOf(fileMetadata.getProperty(Options.INITIAL_EXTENT,
+ Options.DEFAULT_INITIAL_EXTENT)),
+// fileMetadata.extent,
+ maximumExtent, fileMetadata.offsetBits,
fileMetadata.nextOffset, fileMetadata.getBufferMode(),
fileMetadata.readOnly);
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-27 20:01:43
|
Revision: 4169
http://bigdata.svn.sourceforge.net/bigdata/?rev=4169&view=rev
Author: thompsonbry
Date: 2011-01-27 20:01:36 +0000 (Thu, 27 Jan 2011)
Log Message:
-----------
Removed unused files from the test package
Removed Paths:
-------------
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/internal/ITermIdCodes.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/internal/LegacyTermIdUtility.java
Deleted: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/internal/ITermIdCodes.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/internal/ITermIdCodes.java 2011-01-27 20:01:10 UTC (rev 4168)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/internal/ITermIdCodes.java 2011-01-27 20:01:36 UTC (rev 4169)
@@ -1,89 +0,0 @@
-/**
-
-Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
-/*
- * Created on Jan 6, 2008
- */
-
-package com.bigdata.rdf.internal;
-
-import org.openrdf.model.BNode;
-import org.openrdf.model.Literal;
-import org.openrdf.model.URI;
-import com.bigdata.rdf.lexicon.LexiconKeyOrder;
-
-/**
- * An interface that defines bit masks that are applied to the low bits of the
- * assigned term identifiers and which indicate directly (without consulting the
- * term index) whether a term is a URI, Literal, BNode, or Statement. This is
- * used to quickly test the type of a term identifier without requiring the term
- * value to be materialized from the {@link LexiconKeyOrder#ID2TERM} index.
- *
- * @deprecated completely superseded by the new {@link IV} lexicon refactor
- * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
- * @version $Id$
- */
-public interface ITermIdCodes {
-
- /**
- * The bit mask that is bit-wise ANDed with a term identifier in order to
- * reveal the term code (the low two bits of the mask are set).
- *
- * @see #TERMID_CODE_URI
- * @see #TERMID_CODE_BNODE
- * @see #TERMID_CODE_LITERAL
- * @see #TERMID_CODE_STATEMENT
- */
- static final public long TERMID_CODE_MASK = 0x03L;
-
- /**
- * The #of bits in the {@link #TERMID_CODE_MASK} (
- * {@value #TERMID_CODE_MASK_BITS}).
- */
- static final public long TERMID_CODE_MASK_BITS = 2L;
-
- /**
- * The bit value used to indicate that a term identifier stands for a
- * {@link URI}.
- */
- static final public long TERMID_CODE_URI = 0x00L;
-
- /**
- * The bit value used to indicate that a term identifier stands for a
- * {@link BNode}.
- */
- static final public long TERMID_CODE_BNODE = 0x01L;
-
- /**
- * The bit value used to indicate that a term identifier stands for a
- * {@link Literal}.
- */
- static final public long TERMID_CODE_LITERAL = 0x02L;
-
- /**
- * The bit value used to indicate that a term identifier stands for a
- * statement (when support for statement identifiers is enabled).
- */
- static final public long TERMID_CODE_STATEMENT = 0x03L;
-
-}
Deleted: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/internal/LegacyTermIdUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/internal/LegacyTermIdUtility.java 2011-01-27 20:01:10 UTC (rev 4168)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/internal/LegacyTermIdUtility.java 2011-01-27 20:01:36 UTC (rev 4169)
@@ -1,245 +0,0 @@
-/**
-
-Copyright (C) SYSTAP, LLC 2006-2010. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
-/*
- * Created July 10, 2010
- */
-
-package com.bigdata.rdf.internal;
-
-import org.openrdf.model.BNode;
-import org.openrdf.model.Literal;
-import org.openrdf.model.URI;
-import com.bigdata.btree.keys.IKeyBuilder;
-import com.bigdata.btree.keys.KeyBuilder;
-import com.bigdata.rawstore.Bytes;
-import com.bigdata.rdf.store.AbstractTripleStore.Options;
-
-class LegacyTermIdUtility implements ITermIdCodes {
-
- /**
- * Return true iff the term identifier is marked as a RDF {@link Literal}.
- * <p>
- * Note: This simply examines the low bits of the term identifier, which
- * marks whether or not the term identifier is a {@link Literal}.
- * <p>
- * Note: Some entailments require the ability to filter based on whether or
- * not a term is a literal. For example, literals may not be entailed into
- * the subject position. This method makes it possible to determine whether
- * or not a term is a literal without materializing the term, thereby
- * allowing the entailments to be computed purely within the term identifier
- * space.
- *
- * @param termId
- * The term identifier.
- *
- * @return <code>true</code> iff the term identifier is marked as an RDF
- * {@link Literal}.
- */
- static final public boolean isLiteral(final long termId) {
-
- return (termId & TERMID_CODE_MASK) == TERMID_CODE_LITERAL;
-
- }
-
- /**
- * Return true iff the term identifier is marked as a RDF {@link BNode}.
- * <p>
- * Note: This simply examines the low bits of the term identifier, which
- * marks whether or not the term identifier is a {@link BNode}.
- *
- * @param termId
- * The term identifier.
- *
- * @return <code>true</code> iff the term identifier is marked as an RDF
- * {@link BNode}.
- */
- static final public boolean isBNode(final long termId) {
-
- return (termId & TERMID_CODE_MASK) == TERMID_CODE_BNODE;
-
- }
-
- /**
- * Return true iff the term identifier is marked as a RDF {@link URI}.
- * <p>
- * Note: This simply examines the low bits of the term identifier, which
- * marks whether or not the term identifier is a {@link URI}.
- *
- * @param termId
- * The term identifier.
- *
- * @return <code>true</code> iff the term identifier is marked as an RDF
- * {@link URI}.
- */
- static final public boolean isURI(final long termId) {
-
- /*
- * Note: The additional != NULL test is necessary for a URI term
- * identifier so that it will not report 'true' for 0L since the two low
- * order bits used to mark a URI are both zero.
- */
-
- return (termId & TERMID_CODE_MASK) == TERMID_CODE_URI && termId != TermId.NULL;
-
- }
-
- /**
- * Return true iff the term identifier identifies a statement (this feature
- * is enabled with {@link Options#STATEMENT_IDENTIFIERS}).
- * <p>
- * Note: This simply examines the low bits of the term identifier, which
- * marks whether or not the term identifier is actually a statement
- * identifier.
- *
- * @param termId
- * The term identifier.
- *
- * @return <code>true</code> iff the term identifier identifies a
- * statement.
- */
- static final public boolean isStatement(final long termId) {
-
- return (termId & TERMID_CODE_MASK) == TERMID_CODE_STATEMENT;
-
- }
-
- /**
- * Return the {@link VTE} identified by the LOW TWO (2)
- * bits in the caller's value.
- *
- * @param b
- * The bit flags.
- * @return The corresponding {@link VTE}.
- */
- static final public VTE VTE(final long l) {
- if (isURI(l))
- return VTE.URI;
- if (isBNode(l))
- return VTE.BNODE;
- if (isLiteral(l))
- return VTE.LITERAL;
- if (isStatement(l))
- return VTE.STATEMENT;
-
- throw new AssertionError(String.valueOf(l));
- }
-
- /**
- * Construct a new style {@link TermId} from an old style long tid. The
- * term type is encoded into the long in the old style long tid and will
- * be converted to a {@link VTE} by this method.
- *
- * @param legacyTid
- * The old style long term identifier. See {@link ITermIdCodes}.
- */
- static final public TermId termId(final long legacyTid) {
-
- if (legacyTid == TermId.NULL)
- return null;
-
- return new TermId(VTE(legacyTid), legacyTid);
-
- }
-
- /**
- * Encode an IV using the old style data scheme, which is to just write the
- * long term identifier without the flags.
- *
- * @param keyBuilder
- * The object used to encode the {@link IV}.
- * @param iv
- * The {@link IV} to encode.
- */
- static final public IKeyBuilder encode(IKeyBuilder keyBuilder, IV iv) {
-
- if (iv.isInline()) {
- throw new IllegalArgumentException();
- }
-
- keyBuilder.append(iv.getTermId());
-
- return keyBuilder;
-
- }
-
- /**
- * Decode an IV using the old style data scheme, which is to just read the
- * long term identifier without any flags.
- *
- * @param key
- * The byte[].
- * @return The {@link IV} or <code>null</code> if the
- * <code>tid == {@link TermId#NULL}</code>.
- */
- static final public TermId decode(byte[] key) {
-
- final long tid = KeyBuilder.decodeLong(key, 0);
-
- return termId(tid);
-
- }
-
- /**
- * Decode a set of {@link IV}s from a byte[].
- *
- * @param key
- * The byte[].
- * @return The set of {@link IV}s.
- *
- * @todo handle all of the inline value types.
- *
- * @todo Construct the InternalValue objects using factory if we will have
- * to scope how the RDF Value is represented to the lexicon relation
- * with which it is associated?
- */
- public static IV[] decode(final byte[] key, final IV[] ivs) {
-
- final int arity = ivs.length;
-
- int offset = 0;
-
- for (int i = 0; i < arity; i++) {
-
- // decode the term identifier.
- final long termId = KeyBuilder.decodeLong(key, offset);
-
- offset += Bytes.SIZEOF_LONG;
-
- if (termId == TermId.NULL) {
-
- ivs[i] = null;
-
- } else {
-
- ivs[i] = termId(termId);
-
- }
-
- }
-
- return ivs;
-
- }
-
-}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-27 20:01:16
|
Revision: 4168
http://bigdata.svn.sourceforge.net/bigdata/?rev=4168&view=rev
Author: thompsonbry
Date: 2011-01-27 20:01:10 +0000 (Thu, 27 Jan 2011)
Log Message:
-----------
javadoc edit for date time inlining
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java 2011-01-27 18:21:31 UTC (rev 4167)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java 2011-01-27 20:01:10 UTC (rev 4168)
@@ -38,10 +38,8 @@
import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.List;
-import java.util.Locale;
import java.util.Map;
import java.util.Properties;
-import java.util.TimeZone;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
@@ -99,7 +97,6 @@
import com.bigdata.rdf.lexicon.ITermIndexCodes;
import com.bigdata.rdf.lexicon.ITextIndexer;
import com.bigdata.rdf.lexicon.LexiconRelation;
-import com.bigdata.rdf.model.BigdataBNode;
import com.bigdata.rdf.model.BigdataResource;
import com.bigdata.rdf.model.BigdataStatement;
import com.bigdata.rdf.model.BigdataURI;
@@ -906,27 +903,33 @@
/**
* Set up database to inline date/times directly into the statement
* indices rather than using the lexicon to map them to term identifiers
- * and back. Date times will be converted to UTC, then stored as
- * milliseconds since the epoch. Thus if you inline date/times you will
- * lose the canonical representation of the date/time, and you will not
- * be able to recover the original time zone of the date/time.
+ * and back (default {@value #DEFAULT_INLINE_DATE_TIMES}). Date times
+ * will be converted to UTC, then stored as milliseconds since the
+ * epoch. Thus if you inline date/times you will lose the canonical
+ * representation of the date/time, and you will not be able to recover
+ * the original time zone of the date/time.
+ *
+ * @see #INLINE_DATE_TIMES_TIMEZONE
*/
String INLINE_DATE_TIMES = AbstractTripleStore.class.getName()
+ ".inlineDateTimes";
String DEFAULT_INLINE_DATE_TIMES = "false";
-
+
/**
* The default timezone to be used to a) encode inline xsd:datetime
* literals that do not have a time zone specified and b) decode
- * xsd:timezone literals from the statement indices where they are
- * stored as UTC milliseconds since the epoch.
+ * xsd:datetime literals from the statement indices where they are
+ * stored as UTC milliseconds since the epoch (default
+ * {@value #DEFAULT_INLINE_DATE_TIMES_TIMEZONE}).
+ *
+ * @see #INLINE_DATE_TIMES
*/
String INLINE_DATE_TIMES_TIMEZONE = AbstractTripleStore.class.getName()
+ ".inlineDateTimesTimezone";
/**
- * Defaults to the current time zone.
+ * @see #INLINE_DATE_TIMES_TIMEZONE
*/
String DEFAULT_INLINE_DATE_TIMES_TIMEZONE =
// TimeZone.getDefault().getID();
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mrp...@us...> - 2011-01-27 18:21:37
|
Revision: 4167
http://bigdata.svn.sourceforge.net/bigdata/?rev=4167&view=rev
Author: mrpersonick
Date: 2011-01-27 18:21:31 +0000 (Thu, 27 Jan 2011)
Log Message:
-----------
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java 2011-01-27 17:50:26 UTC (rev 4166)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java 2011-01-27 18:21:31 UTC (rev 4167)
@@ -929,8 +929,8 @@
* Defaults to the current time zone.
*/
String DEFAULT_INLINE_DATE_TIMES_TIMEZONE =
- TimeZone.getDefault().getID();
-// "GMT";
+// TimeZone.getDefault().getID();
+ "GMT";
/**
* The name of the {@link IExtensionFactory} class. The implementation
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mrp...@us...> - 2011-01-27 17:50:33
|
Revision: 4166
http://bigdata.svn.sourceforge.net/bigdata/?rev=4166&view=rev
Author: mrpersonick
Date: 2011-01-27 17:50:26 +0000 (Thu, 27 Jan 2011)
Log Message:
-----------
adding a timezone property for xsd:datetime inlining
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/DateTimeExtension.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/DefaultExtensionFactory.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/IExtensionFactory.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/ILexiconConfiguration.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/LexiconConfiguration.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/lexicon/LexiconRelation.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataValueFactoryImpl.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/internal/SampleExtensionFactory.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/internal/TestEncodeDecodeKeys.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/lexicon/TestInlining.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/DateTimeExtension.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/DateTimeExtension.java 2011-01-27 15:54:10 UTC (rev 4165)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/DateTimeExtension.java 2011-01-27 17:50:26 UTC (rev 4166)
@@ -28,9 +28,11 @@
import java.util.TimeZone;
import javax.xml.datatype.DatatypeConfigurationException;
+import javax.xml.datatype.DatatypeConstants;
import javax.xml.datatype.DatatypeFactory;
import javax.xml.datatype.XMLGregorianCalendar;
+import org.apache.log4j.Logger;
import org.openrdf.model.Literal;
import org.openrdf.model.URI;
import org.openrdf.model.Value;
@@ -50,9 +52,13 @@
private final BigdataURI dateTime;
- public DateTimeExtension(final IDatatypeURIResolver resolver) {
+ private final TimeZone defaultTZ;
+
+ public DateTimeExtension(final IDatatypeURIResolver resolver,
+ final TimeZone defaultTZ) {
this.dateTime = resolver.resolve(XSD.DATETIME);
+ this.defaultTZ = defaultTZ;
}
@@ -84,12 +90,21 @@
final String s = value.stringValue();
final XMLGregorianCalendar c = XMLDatatypeUtil.parseCalendar(s);
+
+ if (c.getTimezone() == DatatypeConstants.FIELD_UNDEFINED) {
+ final int offsetInMillis =
+// defaultTZ.getRawOffset();
+ defaultTZ.getOffset(c.toGregorianCalendar().getTimeInMillis());
+ final int offsetInMinutes =
+ offsetInMillis / 1000 / 60;
+ c.setTimezone(offsetInMinutes);
+ }
/*
* Returns the current time as UTC milliseconds from the epoch
*/
final long l = c.toGregorianCalendar().getTimeInMillis();
-
+
final AbstractLiteralIV delegate = new XSDLongIV(l);
return new ExtensionIV(delegate, (TermId) getDatatype().getIV());
@@ -111,6 +126,7 @@
final GregorianCalendar c = new GregorianCalendar(
// TimeZone.getTimeZone("GMT")
+ defaultTZ
);
c.setTimeInMillis(l);
@@ -119,7 +135,13 @@
final XMLGregorianCalendar xmlGC =
DatatypeFactory.newInstance().newXMLGregorianCalendar(c);
- return (V) vf.createLiteral(xmlGC);
+ String s = xmlGC.toString();
+ final int i = s.lastIndexOf('.');
+ if (i >= 0) {
+ s = s.substring(0, i);
+ }
+
+ return (V) vf.createLiteral(s, dateTime);
} catch (DatatypeConfigurationException ex) {
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/DefaultExtensionFactory.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/DefaultExtensionFactory.java 2011-01-27 15:54:10 UTC (rev 4165)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/DefaultExtensionFactory.java 2011-01-27 17:50:26 UTC (rev 4166)
@@ -3,6 +3,8 @@
import java.util.Collection;
import java.util.LinkedList;
+import com.bigdata.rdf.lexicon.LexiconRelation;
+
/**
* Empty {@link IExtensionFactory}.
*/
@@ -18,11 +20,12 @@
}
- public void init(final IDatatypeURIResolver resolver,
- final boolean inlineDateTimes) {
+ public void init(final LexiconRelation lex) {
- if (inlineDateTimes)
- extensions.add(new DateTimeExtension(resolver));
+ if (lex.isInlineDateTimes())
+ extensions.add(new DateTimeExtension(
+ lex, lex.getInlineDateTimesTimeZone()));
+
extensionsArray = extensions.toArray(new IExtension[extensions.size()]);
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/IExtensionFactory.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/IExtensionFactory.java 2011-01-27 15:54:10 UTC (rev 4165)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/IExtensionFactory.java 2011-01-27 17:50:26 UTC (rev 4166)
@@ -24,6 +24,8 @@
package com.bigdata.rdf.internal;
+import com.bigdata.rdf.lexicon.LexiconRelation;
+
/**
* IExtensionFactories are responsible for enumerating what extensions are
* supported for a particular database configuration. Bigdata comes packaged
@@ -37,16 +39,11 @@
/**
* This will be called very early in the IExtensionFactory lifecycle so that
* the {@link TermId}s for the {@link IExtension}'s datatype URIs will be on
- * hand when needed.
- *
- * @param resolver
- * the datatype URI resolver
- * @param inlineDateTimes
- * if true, inine the xsd:dateTime datatype using the
- * {@link DateTimeExtension} class.
+ * hand when needed. Also gets other relevant configuration information
+ * from the lexicon such as whether or not to inline xsd:datetimes and what
+ * timezone to use to do so.
*/
- void init(final IDatatypeURIResolver resolver,
- final boolean inlineDateTimes);
+ void init(final LexiconRelation lex);
/**
* Return the supported extensions.
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/ILexiconConfiguration.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/ILexiconConfiguration.java 2011-01-27 15:54:10 UTC (rev 4165)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/ILexiconConfiguration.java 2011-01-27 17:50:26 UTC (rev 4166)
@@ -26,8 +26,10 @@
import org.openrdf.model.Value;
+import com.bigdata.rdf.lexicon.LexiconRelation;
import com.bigdata.rdf.model.BigdataValue;
import com.bigdata.rdf.model.BigdataValueFactory;
+import com.bigdata.rdf.store.AbstractTripleStore;
/**
* Configuration determines which RDF Values are inlined into the statement
@@ -66,11 +68,8 @@
/**
* Initialize the extensions, which need to resolve their datatype URIs
* into term ids.
- *
- * @param resolver
- * the datatype URI resolver
*/
- void initExtensions(final IDatatypeURIResolver resolver);
+ void initExtensions(final LexiconRelation lex);
/**
* <code>true</code> iff the <code>vte</code> and <code>dte</code>
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/LexiconConfiguration.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/LexiconConfiguration.java 2011-01-27 15:54:10 UTC (rev 4165)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/internal/LexiconConfiguration.java 2011-01-27 17:50:26 UTC (rev 4166)
@@ -30,12 +30,15 @@
import java.util.HashMap;
import java.util.Map;
import java.util.UUID;
+
import org.apache.log4j.Logger;
import org.openrdf.model.BNode;
import org.openrdf.model.Literal;
import org.openrdf.model.URI;
import org.openrdf.model.Value;
import org.openrdf.model.datatypes.XMLDatatypeUtil;
+
+import com.bigdata.rdf.lexicon.LexiconRelation;
import com.bigdata.rdf.model.BigdataURI;
import com.bigdata.rdf.model.BigdataValue;
import com.bigdata.rdf.model.BigdataValueFactory;
@@ -75,9 +78,9 @@
}
- public void initExtensions(final IDatatypeURIResolver resolver) {
+ public void initExtensions(final LexiconRelation lex) {
- xFactory.init(resolver, inlineDateTimes);
+ xFactory.init(lex);
for (IExtension extension : xFactory.getExtensions()) {
BigdataURI datatype = extension.getDatatype();
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/lexicon/LexiconRelation.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/lexicon/LexiconRelation.java 2011-01-27 15:54:10 UTC (rev 4165)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/lexicon/LexiconRelation.java 2011-01-27 17:50:26 UTC (rev 4166)
@@ -41,6 +41,7 @@
import java.util.Map;
import java.util.Properties;
import java.util.Set;
+import java.util.TimeZone;
import java.util.concurrent.Callable;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Future;
@@ -69,10 +70,10 @@
import com.bigdata.btree.keys.KVO;
import com.bigdata.btree.keys.KeyBuilder;
import com.bigdata.btree.keys.StrengthEnum;
-import com.bigdata.btree.proc.IResultHandler;
import com.bigdata.btree.proc.AbstractKeyArrayIndexProcedure.ResultBuffer;
import com.bigdata.btree.proc.AbstractKeyArrayIndexProcedure.ResultBufferHandler;
import com.bigdata.btree.proc.BatchLookup.BatchLookupConstructor;
+import com.bigdata.btree.proc.IResultHandler;
import com.bigdata.btree.raba.IRaba;
import com.bigdata.cache.ConcurrentWeakValueCacheWithBatchedUpdates;
import com.bigdata.journal.IIndexManager;
@@ -400,6 +401,10 @@
AbstractTripleStore.Options.INLINE_DATE_TIMES,
AbstractTripleStore.Options.DEFAULT_INLINE_DATE_TIMES));
+ inlineDateTimesTimeZone = TimeZone.getTimeZone(getProperty(
+ AbstractTripleStore.Options.INLINE_DATE_TIMES_TIMEZONE,
+ AbstractTripleStore.Options.DEFAULT_INLINE_DATE_TIMES_TIMEZONE));
+
try {
final Class<IExtensionFactory> xfc =
@@ -594,6 +599,14 @@
*/
final private boolean inlineDateTimes;
+ /**
+ * The default time zone to be used for decoding inline xsd:datetime
+ * literals from the statement indices. Will use the current timezeon
+ * unless otherwise specified using
+ * {@link AbstractTripleStore.Options#DEFAULT_INLINE_DATE_TIMES_TIMEZONE}.
+ */
+ final private TimeZone inlineDateTimesTimeZone;
+
/**
* Return <code>true</code> if datatype literals are being inlined into
* the statement indices.
@@ -604,8 +617,27 @@
}
+ /**
+ * Return <code>true</code> if xsd:datetime literals are being inlined into
+ * the statement indices.
+ */
+ final public boolean isInlineDateTimes() {
+
+ return inlineDateTimes;
+
+ }
/**
+ * Return the default time zone to be used for inlining.
+ */
+ final public TimeZone getInlineDateTimesTimeZone() {
+
+ return inlineDateTimesTimeZone;
+
+ }
+
+
+ /**
* The #of low bits from the term identifier that are reversed and
* rotated into the high bits when it is assigned.
*
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataValueFactoryImpl.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataValueFactoryImpl.java 2011-01-27 15:54:10 UTC (rev 4165)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataValueFactoryImpl.java 2011-01-27 17:50:26 UTC (rev 4166)
@@ -267,20 +267,10 @@
public BigdataLiteralImpl createLiteral(final XMLGregorianCalendar arg0) {
-// return new BigdataLiteralImpl(this, arg0.toString(),
-// null/* languageCode */, createURI(arg0.getXMLSchemaType()
-// .toString()));
+ return new BigdataLiteralImpl(this, arg0.toString(),
+ null/* languageCode */, createURI(arg0.getXMLSchemaType()
+ .toString()));
- String s = arg0.toString();
- final int i = s.lastIndexOf('.');
- if (i >= 0) {
- s = s.substring(0, i);
- }
-
- return new BigdataLiteralImpl(this, s,
- null/* languageCode */, createURI(XMLDatatypeUtil.qnameToURI(
- arg0.getXMLSchemaType()).stringValue()));
-
}
public BigdataLiteralImpl createLiteral(final String label, final String language) {
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java 2011-01-27 15:54:10 UTC (rev 4165)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java 2011-01-27 17:50:26 UTC (rev 4166)
@@ -38,8 +38,10 @@
import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.Properties;
+import java.util.TimeZone;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
@@ -915,6 +917,22 @@
String DEFAULT_INLINE_DATE_TIMES = "false";
/**
+ * The default timezone to be used to a) encode inline xsd:datetime
+ * literals that do not have a time zone specified and b) decode
+ * xsd:timezone literals from the statement indices where they are
+ * stored as UTC milliseconds since the epoch.
+ */
+ String INLINE_DATE_TIMES_TIMEZONE = AbstractTripleStore.class.getName()
+ + ".inlineDateTimesTimezone";
+
+ /**
+ * Defaults to the current time zone.
+ */
+ String DEFAULT_INLINE_DATE_TIMES_TIMEZONE =
+ TimeZone.getDefault().getID();
+// "GMT";
+
+ /**
* The name of the {@link IExtensionFactory} class. The implementation
* MUST declare a constructor that accepts an
* {@link IDatatypeURIResolver} as its only argument. The
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/internal/SampleExtensionFactory.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/internal/SampleExtensionFactory.java 2011-01-27 15:54:10 UTC (rev 4165)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/internal/SampleExtensionFactory.java 2011-01-27 17:50:26 UTC (rev 4166)
@@ -3,6 +3,8 @@
import java.util.Collection;
import java.util.LinkedList;
+import com.bigdata.rdf.lexicon.LexiconRelation;
+
/**
* Simple {@link IExtensionFactory} implementation that creates two
* {@link IExtension}s - the {@link EpochExtension} and the
@@ -20,12 +22,13 @@
}
- public void init(final IDatatypeURIResolver resolver,boolean inlineDateTimes) {
+ public void init(final LexiconRelation lex) {
- if (inlineDateTimes)
- extensions.add(new DateTimeExtension(resolver));
- extensions.add(new EpochExtension(resolver));
- extensions.add(new ColorsEnumExtension(resolver));
+ if (lex.isInlineDateTimes())
+ extensions.add(new DateTimeExtension(
+ lex, lex.getInlineDateTimesTimeZone()));
+ extensions.add(new EpochExtension(lex));
+ extensions.add(new ColorsEnumExtension(lex));
extensionsArray = extensions.toArray(new IExtension[2]);
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/internal/TestEncodeDecodeKeys.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/internal/TestEncodeDecodeKeys.java 2011-01-27 15:54:10 UTC (rev 4165)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/internal/TestEncodeDecodeKeys.java 2011-01-27 17:50:26 UTC (rev 4166)
@@ -30,6 +30,7 @@
import java.math.BigDecimal;
import java.math.BigInteger;
import java.util.Random;
+import java.util.TimeZone;
import java.util.UUID;
import javax.xml.datatype.DatatypeFactory;
@@ -775,12 +776,14 @@
final DateTimeExtension<BigdataValue> ext =
new DateTimeExtension<BigdataValue>(new IDatatypeURIResolver() {
- public BigdataURI resolve(URI uri) {
- BigdataURI buri = vf.createURI(uri.stringValue());
- buri.setIV(new TermId(VTE.URI, 1024));
- return buri;
- }
- });
+ public BigdataURI resolve(URI uri) {
+ BigdataURI buri = vf.createURI(uri.stringValue());
+ buri.setIV(new TermId(VTE.URI, 1024));
+ return buri;
+ }
+ },
+ TimeZone.getDefault()
+ );
final BigdataLiteral[] dt = {
vf.createLiteral(
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/lexicon/TestInlining.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/lexicon/TestInlining.java 2011-01-27 15:54:10 UTC (rev 4165)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/lexicon/TestInlining.java 2011-01-27 17:50:26 UTC (rev 4166)
@@ -509,5 +509,72 @@
return tmp;
}
+
+ public void test_inlinedatetimes() {
+
+ final Properties properties = getProperties();
+ // test w/o predefined vocab.
+ properties.setProperty(Options.VOCABULARY_CLASS, NoVocabulary.class
+ .getName());
+
+ // test w/o axioms - they imply a predefined vocab.
+ properties.setProperty(Options.AXIOMS_CLASS, NoAxioms.class.getName());
+
+ // test w/o the full text index.
+ properties.setProperty(Options.TEXT_INDEX, "false");
+
+ // test w/o the full text index.
+ properties.setProperty(Options.INLINE_DATE_TIMES, "true");
+
+ AbstractTripleStore store = getStore(properties);
+
+ try {
+
+ final Collection<BigdataValue> terms = new HashSet<BigdataValue>();
+
+ // lookup/add some values.
+ final BigdataValueFactory f = store.getValueFactory();
+
+ terms.add(f.createLiteral("2008-03-22T00:00:00", f
+ .createURI(XSD.DATETIME.toString())));
+
+ terms.add(f.createLiteral("2007-12-25T00:00:00", f
+ .createURI(XSD.DATETIME.toString())));
+
+ final Map<IV, BigdataValue> ids = doAddTermsTest(store, terms);
+
+ if (store.isStable()) {
+
+ store.commit();
+
+ store = reopenStore(store);
+
+ // verify same reverse mappings.
+
+ final Map<IV, BigdataValue> ids2 = store.getLexiconRelation()
+ .getTerms(ids.keySet());
+
+ assertEquals(ids.size(),ids2.size());
+
+ for (IV id : ids.keySet()) {
+
+ System.err.println(ids.get(id));
+ System.err.println(ids2.get(id));
+
+ assertEquals("Id mapped to a different term? : termId="
+ + id, ids.get(id), ids2.get(id));
+
+ }
+
+ }
+
+ } finally {
+
+ store.__tearDownUnitTest();
+
+ }
+
+ }
+
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mr...@us...> - 2011-01-27 15:54:17
|
Revision: 4165
http://bigdata.svn.sourceforge.net/bigdata/?rev=4165&view=rev
Author: mroycsi
Date: 2011-01-27 15:54:10 +0000 (Thu, 27 Jan 2011)
Log Message:
-----------
Commit the current state of the csi release
Modified Paths:
--------------
branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/AbstractResource.java
branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/accesspath/AbstractAccessPath.java
branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/accesspath/BlockingBuffer.java
branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/rule/eval/pipeline/JoinTask.java
branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/striterator/IChunkedIterator.java
branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java
Added Paths:
-----------
branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/accesspath/IAbstractAccessPath.java
Modified: branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/AbstractResource.java
===================================================================
--- branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/AbstractResource.java 2011-01-25 16:01:15 UTC (rev 4164)
+++ branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/AbstractResource.java 2011-01-27 15:54:10 UTC (rev 4165)
@@ -222,7 +222,7 @@
/**
* Default for {@link #CHUNK_OF_CHUNKS_CAPACITY}
*/
- String DEFAULT_CHUNK_OF_CHUNKS_CAPACITY = "1000";
+ String DEFAULT_CHUNK_OF_CHUNKS_CAPACITY = "10";
/**
* <p>
Modified: branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/accesspath/AbstractAccessPath.java
===================================================================
--- branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/accesspath/AbstractAccessPath.java 2011-01-25 16:01:15 UTC (rev 4164)
+++ branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/accesspath/AbstractAccessPath.java 2011-01-27 15:54:10 UTC (rev 4165)
@@ -33,6 +33,7 @@
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Future;
import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
import org.apache.log4j.Logger;
@@ -82,7 +83,7 @@
* the elements we do not want from the key-range scan - the filtering
* should of course be done at the {@link IDataService}.
*/
-abstract public class AbstractAccessPath<R> implements IAccessPath<R> {
+abstract public class AbstractAccessPath<R> implements IAccessPath<R>,IAbstractAccessPath {
static final protected Logger log = Logger.getLogger(IAccessPath.class);
@@ -1027,7 +1028,7 @@
* once the elements were materialized on the client.
*/
final BlockingBuffer<R[]> buffer = new BlockingBuffer<R[]>(
- chunkOfChunksCapacity);
+ chunkOfChunksCapacity,chunkCapacity,10,TimeUnit.MILLISECONDS);
final ExecutorService executorService = indexManager
.getExecutorService();
Modified: branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/accesspath/BlockingBuffer.java
===================================================================
--- branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/accesspath/BlockingBuffer.java 2011-01-25 16:01:15 UTC (rev 4164)
+++ branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/accesspath/BlockingBuffer.java 2011-01-27 15:54:10 UTC (rev 4165)
@@ -167,12 +167,12 @@
* The default capacity for the internal {@link Queue} on which elements (or
* chunks of elements) are buffered.
*/
- public static transient final int DEFAULT_PRODUCER_QUEUE_CAPACITY = 5000;
+ public static transient final int DEFAULT_PRODUCER_QUEUE_CAPACITY = 10;
/**
* The default minimum chunk size for the chunk combiner.
*/
- public static transient final int DEFAULT_MINIMUM_CHUNK_SIZE = 10000;
+ public static transient final int DEFAULT_MINIMUM_CHUNK_SIZE = 100;
/**
* The default timeout in milliseconds during which chunks of elements may
Added: branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/accesspath/IAbstractAccessPath.java
===================================================================
--- branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/accesspath/IAbstractAccessPath.java (rev 0)
+++ branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/accesspath/IAbstractAccessPath.java 2011-01-27 15:54:10 UTC (rev 4165)
@@ -0,0 +1,6 @@
+package com.bigdata.relation.accesspath;
+
+public interface IAbstractAccessPath {
+ public byte[] getFromKey();
+ public byte[] getToKey();
+}
Property changes on: branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/accesspath/IAbstractAccessPath.java
___________________________________________________________________
Added: svn:mime-type
+ text/plain
Modified: branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/rule/eval/pipeline/JoinTask.java
===================================================================
--- branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/rule/eval/pipeline/JoinTask.java 2011-01-25 16:01:15 UTC (rev 4164)
+++ branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/relation/rule/eval/pipeline/JoinTask.java 2011-01-27 15:54:10 UTC (rev 4165)
@@ -38,6 +38,7 @@
import com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer;
import com.bigdata.relation.accesspath.BlockingBuffer;
import com.bigdata.relation.accesspath.BufferClosedException;
+import com.bigdata.relation.accesspath.IAbstractAccessPath;
import com.bigdata.relation.accesspath.IAccessPath;
import com.bigdata.relation.accesspath.IAsynchronousIterator;
import com.bigdata.relation.accesspath.IBlockingBuffer;
@@ -1409,7 +1410,7 @@
*/
protected byte[] getFromKey() {
- return ((AbstractAccessPath<?>) accessPath).getFromKey();
+ return ((IAbstractAccessPath) accessPath).getFromKey();
}
Modified: branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/striterator/IChunkedIterator.java
===================================================================
--- branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/striterator/IChunkedIterator.java 2011-01-25 16:01:15 UTC (rev 4164)
+++ branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata/src/java/com/bigdata/striterator/IChunkedIterator.java 2011-01-27 15:54:10 UTC (rev 4165)
@@ -59,7 +59,7 @@
/**
* The default chunk size.
*/
- int DEFAULT_CHUNK_SIZE = 10000;
+ int DEFAULT_CHUNK_SIZE = 100;
/**
* The next element available from the iterator.
Modified: branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java
===================================================================
--- branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java 2011-01-25 16:01:15 UTC (rev 4164)
+++ branches/BIGDATA_RELEASE_0_83_2_CSI/bigdata-rdf/src/java/com/bigdata/rdf/store/AbstractTripleStore.java 2011-01-27 15:54:10 UTC (rev 4165)
@@ -573,7 +573,7 @@
String TERM_CACHE_CAPACITY = AbstractTripleStore.class.getName()
+ ".termCache.capacity";
- String DEFAULT_TERM_CACHE_CAPACITY = "50000";
+ String DEFAULT_TERM_CACHE_CAPACITY = "5000";
/**
* The name of the class that will establish the pre-defined
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mrp...@us...> - 2011-01-25 16:01:22
|
Revision: 4164
http://bigdata.svn.sourceforge.net/bigdata/?rev=4164&view=rev
Author: mrpersonick
Date: 2011-01-25 16:01:15 +0000 (Tue, 25 Jan 2011)
Log Message:
-----------
bringing over new free text search features from HA branch
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/FreeTextSearchExpander.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestSearchQuery.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java 2011-01-22 21:13:16 UTC (rev 4163)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataEvaluationStrategyImpl3.java 2011-01-25 16:01:15 UTC (rev 4164)
@@ -7,8 +7,11 @@
import java.util.Collection;
import java.util.HashSet;
import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
import java.util.LinkedList;
import java.util.List;
+import java.util.Map;
import java.util.Properties;
import java.util.Set;
import java.util.concurrent.TimeUnit;
@@ -615,6 +618,68 @@
final Collection<SOpGroup> groupsToPrune = new LinkedList<SOpGroup>();
/*
+ * We need to prune Sesame filters that we cannot translate into native
+ * constraints (ones that require lexicon joins). We also need to
+ * prune search metadata tails.
+ */
+ final Collection<SOp> sopsToPrune = new LinkedList<SOp>();
+
+ /*
+ * deal with free text search tails first. need to match up search
+ * metadata tails with the searches themselves. ie:
+ *
+ * select *
+ * where {
+ * ?s bd:search "foo" .
+ * ?s bd:relevance ?score .
+ * }
+ */
+ // the statement patterns for metadata about the searches
+ final Map<Var, Set<StatementPattern>> searchMetadata =
+ new LinkedHashMap<Var, Set<StatementPattern>>();
+ // do a first pass to gather up the actual searches and take them out
+ // of the master list of statement patterns
+ for (SOp sop : sopTree) {
+ final QueryModelNode op = sop.getOperator();
+ if (op instanceof StatementPattern) {
+ final StatementPattern sp = (StatementPattern) op;
+ final Value s = sp.getSubjectVar().getValue();
+ final Value p = sp.getPredicateVar().getValue();
+ final Value o = sp.getObjectVar().getValue();
+ if (s == null && p != null && o != null &&
+ BD.SEARCH.equals(p)) {
+ searchMetadata.put(sp.getSubjectVar(),
+ new LinkedHashSet<StatementPattern>());
+ }
+ }
+ }
+ // do a second pass to get the search metadata
+ for (SOp sop : sopTree) {
+ final QueryModelNode op = sop.getOperator();
+ if (op instanceof StatementPattern) {
+ final StatementPattern sp = (StatementPattern) op;
+ final Value s = sp.getSubjectVar().getValue();
+ final Value p = sp.getPredicateVar().getValue();
+ if (s == null && p != null &&
+ (BD.RELEVANCE.equals(p) || BD.MAX_HITS.equals(p) ||
+ BD.MIN_RELEVANCE.equals(p))) {
+ final Var sVar = sp.getSubjectVar();
+ Set<StatementPattern> metadata = searchMetadata.get(sVar);
+ if (metadata != null) {
+ metadata.add(sp);
+ }
+ sopsToPrune.add(sop);
+ }
+ }
+ }
+
+ /*
+ * Prunes the sop tree of search metadata.
+ */
+ sopTree = stb.pruneSOps(sopTree, sopsToPrune);
+ sopsToPrune.clear();
+
+ /*
* Iterate through the sop tree and translate statement patterns into
* predicates.
*/
@@ -622,8 +687,16 @@
final QueryModelNode op = sop.getOperator();
if (op instanceof StatementPattern) {
final StatementPattern sp = (StatementPattern) op;
+ final Value p = sp.getPredicateVar().getValue();
try {
- final IPredicate bop = toPredicate((StatementPattern) op);
+ final IPredicate bop;
+ if (p != null && BD.SEARCH.equals(p)) {
+ final Set<StatementPattern> metadata =
+ searchMetadata.get(sp.getSubjectVar());
+ bop = toSearchPredicate(sp, metadata);
+ } else {
+ bop = toPredicate((StatementPattern) op);
+ }
sop.setBOp(bop);
} catch (UnrecognizedValueException ex) {
/*
@@ -657,12 +730,6 @@
final Collection<Filter> sesameFilters = new LinkedList<Filter>();
/*
- * We need to prune Sesame filters that we cannot translate into native
- * constraints (ones that require lexicon joins).
- */
- final Collection<SOp> sopsToPrune = new LinkedList<SOp>();
-
- /*
* Iterate through the sop tree and translate Sesame ValueExpr operators
* into bigdata IConstraint boperators.
*/
@@ -1842,6 +1909,118 @@
}
+ private IPredicate toSearchPredicate(final StatementPattern sp,
+ final Set<StatementPattern> metadata)
+ throws QueryEvaluationException {
+
+ final Value predValue = sp.getPredicateVar().getValue();
+ if (log.isDebugEnabled()) {
+ log.debug(predValue);
+ }
+ if (predValue == null || !BD.SEARCH.equals(predValue)) {
+ throw new IllegalArgumentException("not a valid magic search: " + sp);
+ }
+ final Value objValue = sp.getObjectVar().getValue();
+ if (log.isDebugEnabled()) {
+ log.debug(objValue);
+ }
+ if (objValue == null || !(objValue instanceof Literal)) {
+ throw new IllegalArgumentException("not a valid magic search: " + sp);
+ }
+
+ final Var subjVar = sp.getSubjectVar();
+
+ final IVariableOrConstant<IV> search =
+ com.bigdata.bop.Var.var(subjVar.getName());
+
+ IVariableOrConstant<IV> relevance = new Constant(DummyIV.INSTANCE);
+ Literal maxHits = null;
+ Literal minRelevance = null;
+
+ for (StatementPattern meta : metadata) {
+ if (!meta.getSubjectVar().equals(subjVar)) {
+ throw new IllegalArgumentException("illegal metadata: " + meta);
+ }
+ final Value pVal = meta.getPredicateVar().getValue();
+ final Var oVar = meta.getObjectVar();
+ final Value oVal = oVar.getValue();
+ if (pVal == null) {
+ throw new IllegalArgumentException("illegal metadata: " + meta);
+ }
+ if (BD.RELEVANCE.equals(pVal)) {
+ if (oVar.hasValue()) {
+ throw new IllegalArgumentException("illegal metadata: " + meta);
+ }
+ relevance = com.bigdata.bop.Var.var(oVar.getName());
+ } else if (BD.MAX_HITS.equals(pVal)) {
+ if (oVal == null || !(oVal instanceof Literal)) {
+ throw new IllegalArgumentException("illegal metadata: " + meta);
+ }
+ maxHits = (Literal) oVal;
+ } else if (BD.MIN_RELEVANCE.equals(pVal)) {
+ if (oVal == null || !(oVal instanceof Literal)) {
+ throw new IllegalArgumentException("illegal metadata: " + meta);
+ }
+ minRelevance = (Literal) oVal;
+ }
+ }
+
+ final IAccessPathExpander expander =
+ new FreeTextSearchExpander(database, (Literal) objValue,
+ maxHits, minRelevance);
+
+ // Decide on the correct arity for the predicate.
+ final BOp[] vars = new BOp[] {
+ search, // s = searchVar
+ relevance, // p = relevanceVar
+ new Constant(DummyIV.INSTANCE), // o = reserved
+ new Constant(DummyIV.INSTANCE), // c = reserved
+ };
+
+ // The annotations for the predicate.
+ final List<NV> anns = new LinkedList<NV>();
+
+ anns.add(new NV(IPredicate.Annotations.RELATION_NAME,
+ new String[] { database.getSPORelation().getNamespace() }));//
+
+ // free text search expander or named graphs expander
+ if (expander != null)
+ anns.add(new NV(IPredicate.Annotations.ACCESS_PATH_EXPANDER, expander));
+
+ // timestamp
+ anns.add(new NV(Annotations.TIMESTAMP, database
+ .getSPORelation().getTimestamp()));
+
+ /*
+ * Explicitly set the access path / iterator flags.
+ *
+ * Note: High level query generally permits iterator level parallelism.
+ * We set the PARALLEL flag here so it can be used if a global index
+ * view is chosen for the access path.
+ *
+ * Note: High level query for SPARQL always uses read-only access paths.
+ * If you are working with a SPARQL extension with UPDATE or INSERT INTO
+ * semantics then you will need to remote the READONLY flag for the
+ * mutable access paths.
+ */
+ anns.add(new NV(IPredicate.Annotations.FLAGS, IRangeQuery.DEFAULT
+ | IRangeQuery.PARALLEL | IRangeQuery.READONLY));
+
+ return new SPOPredicate(vars, anns.toArray(new NV[anns.size()]));
+// return new SPOPredicate(
+// new String[] { database.getSPORelation().getNamespace() },
+// -1, // partitionId
+// search, // s = searchVar
+// relevance, // p = relevanceVar
+// new Constant(DummyIV.INSTANCE), // o = reserved
+// new Constant(DummyIV.INSTANCE), // c = reserved
+// false, // optional
+// null, // filter on elements visited by the access path.
+// expander // free text search expander or named graphs expander
+// );
+
+ }
+
/**
* Takes a ValueExpression from a sesame Filter or LeftJoin and turns it
* into a bigdata {@link IConstraint}.
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/FreeTextSearchExpander.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/FreeTextSearchExpander.java 2011-01-22 21:13:16 UTC (rev 4163)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/FreeTextSearchExpander.java 2011-01-25 16:01:15 UTC (rev 4164)
@@ -16,6 +16,7 @@
import com.bigdata.rdf.internal.TermId;
import com.bigdata.rdf.internal.VTE;
import com.bigdata.rdf.internal.XSDDoubleIV;
+import com.bigdata.rdf.lexicon.ITextIndexer;
import com.bigdata.rdf.model.BigdataValue;
import com.bigdata.rdf.spo.ISPO;
import com.bigdata.rdf.spo.SPO;
@@ -138,14 +139,28 @@
if (hiterator == null) {
assert database!=null;
assert query != null;
- if (database.getLexiconRelation().getSearchEngine() == null)
+
+ final ITextIndexer textNdx =
+ database.getLexiconRelation().getSearchEngine();
+
+ if (textNdx == null)
throw new UnsupportedOperationException(
"No free text index?");
+
// final long begin = System.nanoTime();
- hiterator = database.getLexiconRelation()
- .getSearchEngine().search(query.getLabel(),
+
+ String s = query.getLabel();
+ final boolean prefixMatch;
+ if (s.indexOf('*') >= 0) {
+ prefixMatch = true;
+ s = s.replaceAll("\\*", "");
+ } else {
+ prefixMatch = false;
+ }
+
+ hiterator = textNdx.search(s,
query.getLanguage(),
- false/* prefixMatch */,
+ prefixMatch,
minRelevance == null ? 0d : minRelevance.doubleValue()/* minCosine */,
maxHits == null ? 10000 : maxHits.intValue()+1/* maxRank */,
1000L/* timeout */, TimeUnit.MILLISECONDS);
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestSearchQuery.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestSearchQuery.java 2011-01-22 21:13:16 UTC (rev 4163)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestSearchQuery.java 2011-01-25 16:01:15 UTC (rev 4164)
@@ -934,6 +934,138 @@
}
+ { // exact match
+
+ final String searchQuery = "brown cow";
+ final double minRelevance = 0.0d;
+
+ final String query =
+ "select ?s ?o ?score " +
+ "where " +
+ "{ " +
+ " ?s <"+RDFS.LABEL+"> ?o . " +
+ " ?o <"+BD.SEARCH+"> \""+searchQuery+"\" . " +
+ " ?o <"+BD.RELEVANCE+"> ?score . " +
+// " ?o <"+BD.MIN_RELEVANCE+"> \""+minRelevance+"\" . " +
+// " ?o <"+BD.MAX_HITS+"> \"5\" . " +
+ " filter regex(?o, \""+searchQuery+"\") " +
+ "} " +
+ "order by desc(?score)";
+
+ log.info("\n"+query);
+
+ final TupleQuery tupleQuery =
+ cxn.prepareTupleQuery(QueryLanguage.SPARQL, query);
+ tupleQuery.setIncludeInferred(true /* includeInferred */);
+ TupleQueryResult result = tupleQuery.evaluate();
+
+ int i = 0;
+ while (result.hasNext()) {
+ log.info(i++ + ": " + result.next().toString());
+ }
+ assertTrue("wrong # of results: " + i, i == 2);
+
+ result = tupleQuery.evaluate();
+
+ Collection<BindingSet> answer = new LinkedList<BindingSet>();
+
+ final ITextIndexer search =
+ sail.getDatabase().getLexiconRelation().getSearchEngine();
+ final Hiterator<IHit> hits =
+ search.search(searchQuery,
+ null, // languageCode
+ false, // prefixMatch
+ minRelevance, // minCosine
+ 10000, // maxRank (=maxResults + 1)
+ 1000L, // timeout
+ TimeUnit.MILLISECONDS // unit
+ );
+
+ while (hits.hasNext()) {
+ final IHit hit = hits.next();
+ final IV id = new TermId(VTE.LITERAL, hit.getDocId());
+ final Literal score = vf.createLiteral(hit.getCosine());
+ final URI s = uris.get(id);
+ final Literal o = literals.get(id);
+ if (!o.getLabel().contains(searchQuery))
+ continue;
+ final BindingSet bs = createBindingSet(
+ new BindingImpl("s", s),
+ new BindingImpl("o", o),
+ new BindingImpl("score", score));
+ log.info(bs);
+ answer.add(bs);
+ }
+
+ compare(result, answer);
+
+ }
+
+ { // prefix match
+
+ final String searchQuery = "bro*";
+ final double minRelevance = 0.0d;
+
+ final String query =
+ "select ?s ?o ?score " +
+ "where " +
+ "{ " +
+ " ?s <"+RDFS.LABEL+"> ?o . " +
+ " ?o <"+BD.SEARCH+"> \""+searchQuery+"\" . " +
+ " ?o <"+BD.RELEVANCE+"> ?score . " +
+// " ?o <"+BD.MIN_RELEVANCE+"> \""+minRelevance+"\" . " +
+// " ?o <"+BD.MAX_HITS+"> \"5\" . " +
+// " filter regex(?o, \""+searchQuery+"\") " +
+ "} " +
+ "order by desc(?score)";
+
+ log.info("\n"+query);
+
+ final TupleQuery tupleQuery =
+ cxn.prepareTupleQuery(QueryLanguage.SPARQL, query);
+ tupleQuery.setIncludeInferred(true /* includeInferred */);
+ TupleQueryResult result = tupleQuery.evaluate();
+
+ int i = 0;
+ while (result.hasNext()) {
+ log.info(i++ + ": " + result.next().toString());
+ }
+ assertTrue("wrong # of results: " + i, i == 3);
+
+ result = tupleQuery.evaluate();
+
+ Collection<BindingSet> answer = new LinkedList<BindingSet>();
+
+ final ITextIndexer search =
+ sail.getDatabase().getLexiconRelation().getSearchEngine();
+ final Hiterator<IHit> hits =
+ search.search(searchQuery,
+ null, // languageCode
+ true, // prefixMatch
+ minRelevance, // minCosine
+ 10000, // maxRank (=maxResults + 1)
+ 1000L, // timeout
+ TimeUnit.MILLISECONDS // unit
+ );
+
+ while (hits.hasNext()) {
+ final IHit hit = hits.next();
+ final IV id = new TermId(VTE.LITERAL, hit.getDocId());
+ final Literal score = vf.createLiteral(hit.getCosine());
+ final URI s = uris.get(id);
+ final Literal o = literals.get(id);
+ final BindingSet bs = createBindingSet(
+ new BindingImpl("s", s),
+ new BindingImpl("o", o),
+ new BindingImpl("score", score));
+ log.info(bs);
+ answer.add(bs);
+ }
+
+ compare(result, answer);
+
+ }
+
} finally {
cxn.close();
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-22 21:13:23
|
Revision: 4163
http://bigdata.svn.sourceforge.net/bigdata/?rev=4163&view=rev
Author: thompsonbry
Date: 2011-01-22 21:13:16 +0000 (Sat, 22 Jan 2011)
Log Message:
-----------
Removed the lubm-* ant build targets. See bigdata-perf/lubm, bigdata-perf/bsbm3, etc. for performance tests. Removed some utility targets which are also now accessible via the bigdata-perf package.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/build.xml
Modified: branches/QUADS_QUERY_BRANCH/build.xml
===================================================================
--- branches/QUADS_QUERY_BRANCH/build.xml 2011-01-22 12:04:30 UTC (rev 4162)
+++ branches/QUADS_QUERY_BRANCH/build.xml 2011-01-22 21:13:16 UTC (rev 4163)
@@ -627,161 +627,6 @@
</java>
</target>
-
-<!-- -->
-<!-- LUBM TARGETS (OPTIONAL) -->
-<!-- -->
-
-<target name="lubm-clean" description="Clean the lubm-integration from the build directory.">
-<delete dir="${build.dir}/lubm" />
-</target>
-
-<target name="lubm-prepare" description="Clean the lubm-integration from the build directory.">
-<mkdir dir="${build.dir}/lubm" />
-<mkdir dir="${build.dir}/lubm/classes" />
-<mkdir dir="${build.dir}/lubm/lib" />
-</target>
-
-<path id="lubm.build.classpath" description="The lubm build-time classpath (this expects to find the bigdata JAR already installed).">
-<fileset dir="${install.lib.dir}">
- <include name="**/*.jar" />
-</fileset>
-</path>
-
-<!-- And now for something totally weird. If you compile against the bigdata.jar
- rather than build.dir/classes then you will see some errors reported in
- LubmGeneratorMaster.java which otherwise are not reported... -->
-<target name="lubm-compile" depends="lubm-prepare" description="Compile the optional lubm integration.">
-<javac destdir="${build.dir}/lubm/classes" classpathref="runtime.classpath" debug="${javac.debug}" debuglevel="${javac.debuglevel}" verbose="${javac.verbose}" encoding="${javac.encoding}">
- <!-- note: must also specify -bootclasspath and -extdirs when cross-compiling -->
- <!-- target="${javac.target}" source="${javac.source}" -->
- <src path="${bigdata.dir}/bigdata-lubm/src/java" />
-</javac>
-<!-- copy resources. -->
-<copy toDir="${build.dir}/lubm/classes">
- <fileset dir="${bigdata.dir}/bigdata-lubm/src/java">
- <exclude name="**/*.java" />
- </fileset>
-</copy>
-</target>
-
-<target name="lubm-jar" depends="lubm-compile" description="Generates the JAR containing the optional LUBM integration.">
-<jar destfile="${build.dir}/lubm/lib/bigdata-lubm.jar">
- <fileset dir="${build.dir}/lubm/classes" />
-</jar>
-</target>
-
-<!-- This explicitly enumerates the lubm scripts so we don't run
- fixcrlf or set the execute bit on arbitrary files in the
- install directory. -->
-<fileset dir="${install.bin.dir}" id="lubm-scripts" description="The lubm scripts.">
-<include name="lubmMaster.sh" />
-<include name="lubmQuery.sh" />
-<include name="lubmGen.sh" />
-</fileset>
-
-<!-- While this installs the LUBM integration into the same place as the
- bigdata federation, you do not need to have the LUBM classes or the
- integration classes installed when you start the bigdata federation.
- Those classes will be automatically found when you run the lubmMaster
- script, which exposes them using a ClassServer. You can use this as
- a model for how to install and run your own software against a bigdata
- federation that is already up and running. -->
-<target name="lubm-install" depends="lubm-jar" description="Install the optional lubm integration which may be used for benchmarking the RDF database.">
-<mkdir dir="${install.lubm.dir}" />
-<mkdir dir="${install.lubm.lib.dir}" />
-<mkdir dir="${install.lubm.config.dir}" />
-<!-- install JAR. -->
-<copy toDir="${install.lubm.lib.dir}" file="${build.dir}/lubm/lib/bigdata-lubm.jar" />
-<!-- install ontology, configuration files, and query files. -->
-<copy toDir="${install.lubm.config.dir}">
- <fileset dir="bigdata-lubm/resources/config" />
-</copy>
-<!-- install scripts. -->
-<copy toDir="${install.bin.dir}">
- <fileset dir="bigdata-lubm/resources/scripts" />
-</copy>
-<!-- replace will only find those @XXX@ parameters which have not yet been
- transcribed out by the bigdata ant install. -->
-<replace dir="${install.bin.dir}" summary="true">
- <replacefilter token="@NAS@" value="${NAS}" />
- <replacefilter token="@BIN_DIR@" value="${install.bin.dir}" />
- <replacefilter token="@BIGDATA_CONFIG@" value="${bigdata.config}" />
- <replacefilter token="@LUBM_CLASS_SERVER_PORT@" value="${LUBM_CLASS_SERVER_PORT}" />
- <replacefilter token="@LUBM_CLASS_SERVER_HOSTNAME@" value="${LUBM_CLASS_SERVER_HOSTNAME}" />
- <replacefilter token="@LUBM_RMI_CODEBASE_URL@" value="${LUBM_RMI_CODEBASE_URL}" />
- <replacefilter token="@install.lubm.lib.dir@" value="${install.lubm.lib.dir}" />
- <replacefilter token="@install.lubm.config.dir@" value="${install.lubm.config.dir}" />
-</replace>
-<replace dir="${install.lubm.config.dir}" summary="true">
- <replacefilter token="@NAS@" value="${NAS}" />
- <replacefilter token="@BIN_DIR@" value="${install.bin.dir}" />
- <replacefilter token="@BIGDATA_CONFIG@" value="${bigdata.config}" />
- <replacefilter token="@LUBM_CLASS_SERVER_PORT@" value="${LUBM_CLASS_SERVER_PORT}" />
- <replacefilter token="@LUBM_CLASS_SERVER_HOSTNAME@" value="${LUBM_CLASS_SERVER_HOSTNAME}" />
- <replacefilter token="@LUBM_RMI_CODEBASE_URL@" value="${LUBM_RMI_CODEBASE_URL}" />
- <replacefilter token="@install.lubm.lib.dir@" value="${install.lubm.lib.dir}" />
- <replacefilter token="@install.lubm.config.dir@" value="${install.lubm.config.dir}" />
-</replace>
-<!-- fix newlines (otherwise substitutions cause things to break). -->
-<fixcrlf srcDir="${install.bin.dir}">
- <!-- file set not supported. <fileset refid="scripts" /> -->
-</fixcrlf>
-<!-- set execute bit for scripts in this directory (must be the last step). -->
-<chmod perm="u+x,g+rx,o-rwx">
- <fileset refid="lubm-scripts" />
-</chmod>
-<!-- Make sure that it is all accessible to the install group (ant 1.6+ plus extension module required).
- <chown file="${NAS}" type="both" owner="${install.user}.${install.group}" verbose="true"/>
- -->
-<!-- Works for earlier versions of ant LT 1.6 which do not bundle "chown". -->
-<apply executable="chown" description="set owner on NAS files" osfamily="unix">
- <arg value="-R" />
- <arg value="${install.user}.${install.group}" />
- <dirset dir="${install.bin.dir}" />
-</apply>
-<apply executable="chown" description="set owner on NAS files" osfamily="unix">
- <arg value="-R" />
- <arg value="${install.user}.${install.group}" />
- <dirset dir="${install.lubm.dir}" />
-</apply>
-</target>
-
-<!-- lubm runtime classpath w/o install. -->
-<path id="lubm.runtime.classpath">
-<pathelement location="${build.dir}/lubm/classes" />
-<pathelement location="${build.dir}/classes" />
-<path refid="build.classpath" />
-</path>
-
-<target name="lubm-load" depends="jar, lubm-compile" description="Load data into a configured lubm test harness, typically standalone.">
-
-<java classname="edu.lehigh.swat.bench.ubt.Test" failonerror="true" fork="true" logerror="true">
- <classpath refid="lubm.runtime.classpath" />
- <jvmarg value="-server" />
- <jvmarg value="-Xmx1024m" />
- <jvmarg value="-Dcom.bigdata.jmx.log4j.disable=true" />
- <jvmarg value="-Dlog4j.configuration=file:bigdata/src/resources/logging/log4j.properties" />
- <arg value="load" />
- <arg value="bigdata-lubm/src/java/edu/lehigh/swat/bench/ubt/bigdata/config.kb.bigdata" />
-</java>
-
-</target>
-
-<target name="lubm-test" depends="jar, lubm-compile" description="Run queries against a configured lubm test harness, typically standalone.">
-
-<java classname="edu.lehigh.swat.bench.ubt.Test" failonerror="true" fork="true" logerror="true">
- <classpath refid="lubm.runtime.classpath" />
- <jvmarg value="-server" />
- <jvmarg value="-Xmx1024m" />
- <jvmarg value="-Dcom.bigdata.jmx.log4j.disable=true" />
- <jvmarg value="-Dlog4j.configuration=file:bigdata/src/resources/logging/log4j.properties" />
- <arg value="query" />
- <arg value="bigdata-lubm/src/java/edu/lehigh/swat/bench/ubt/bigdata/config.kb.bigdata" />
- <arg value="bigdata-lubm/src/java/edu/lehigh/swat/bench/ubt/bigdata/config.query.sparql" />
-</java>
-
-</target>
<!-- -->
<!-- MISC. UTILITY TARGETS -->
@@ -809,54 +654,6 @@
</target>
-<target name="DataLoader" depends="compile" description="Loads RDF data into a local KB. You MUST edit this ant target before running it.">
-<java classname="com.bigdata.rdf.store.DataLoader" fork="true" failonerror="true">
- <!-- usage: [-namespace namespace] propertyFile (fileOrDir)+ -->
- <!-- Where: -->
- <!-- [-namespace namespace] is the KB namespace (default is 'kb'). -->
- <!-- propertyFile is a properties file identifying the Journal and
- giving various Journal and/or kb configuration properties if one
- or the other needs to be created. -->
- <!-- (fileOrDir)+ is a list of one or more RDF files or directories to
- be loaded. zip and gz extensions are recognized, but only one file
- is loaded per archive. -->
- <arg line="custom.properties fileOrDir" />
- <jvmarg value="-server" />
- <!-- Specify the maximum Java heap size. -->
- <jvmarg value="-Xmx10g" />
- <!-- optionally enable yourkit profiler.
- <jvmarg value="-DLD_LIBRARY_PATH=/nas/install/yjp-8.0.19/bin/linux-x86-64"/>
- <jvmarg value="-agentpath:/nas/install/yjp-8.0.20/bin/linux-x86-64/libyjpagent.so"/>
- <jvmarg value="-agentlib:yjpagent=disableexceptiontelemetry,disablestacktelemetry"/>
- -->
- <jvmarg value="-XX:+UseParallelOldGC" />
- <!-- Optional enable GC trace.
- <jvmarg line="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:jvm_gc.log"/>
- -->
- <classpath>
- <path refid="runtime.classpath" />
- </classpath>
-</java>
-</target>
-
-<target name="set-properties" depends="compile" description="Set or change properties for a kb instance. You MUST edit this target to specify the name of the journal. The new values are read from stdin.">
-<java classname="com.bigdata.rdf.sail.BigdataSailHelper" fork="true" failonerror="true" inputstring="com.bigdata.relation.rule.eval.DefaultRuleTaskFactory.nestedSubquery=false">
- <!-- Various things you might want to change:
-
- Maximum #of threads for joins.
- inputstring="com.bigdata.relation.rule.eval.ProgramTask.maxParallelSubqueries=5"
-
- Nested subquery vs pipeline joins.
- inputstring="com.bigdata.relation.rule.eval.DefaultRuleTaskFactory.nestedSubquery=true"
-
--->
- <arg line="d:/LTS.U50.jnl LTS kb" />
- <classpath>
- <path refid="runtime.classpath" />
- </classpath>
-</java>
-</target>
-
<!-- -->
<!-- STAGING -->
<!-- -->
@@ -1960,7 +1757,7 @@
<!-- PERFORMANCE TESTS -->
<!-- -->
-<target name="run-performance-tests" depends="testCompile, lubm-jar" description="Runs a variety of performance tests.">
+<target name="run-performance-tests" depends="testCompile" description="Runs a variety of performance tests.">
<!-- Note: This depends on the stage target. -->
<path id="run.class.path.id">
<pathelement location="${junit.jar}" />
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-22 12:04:36
|
Revision: 4162
http://bigdata.svn.sourceforge.net/bigdata/?rev=4162&view=rev
Author: thompsonbry
Date: 2011-01-22 12:04:30 +0000 (Sat, 22 Jan 2011)
Log Message:
-----------
Bug fix for BigdataValueFactoryImpl#createLiteral("foo",(URI)null).
See https://sourceforge.net/apps/trac/bigdata/ticket/226
Modified Paths:
--------------
trunk/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataValueFactoryImpl.java
Modified: trunk/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataValueFactoryImpl.java
===================================================================
--- trunk/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataValueFactoryImpl.java 2011-01-22 12:03:03 UTC (rev 4161)
+++ trunk/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataValueFactoryImpl.java 2011-01-22 12:04:30 UTC (rev 4162)
@@ -285,7 +285,12 @@
public BigdataLiteralImpl createLiteral(final String label, URI datatype) {
- if (!(datatype instanceof BigdataURIImpl)) {
+ /*
+ * Note: The datatype parameter may be null per the Sesame API.
+ *
+ * See https://sourceforge.net/apps/trac/bigdata/ticket/226
+ */
+ if (datatype != null && !(datatype instanceof BigdataURIImpl)) {
datatype = createURI(datatype.stringValue());
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-22 12:03:11
|
Revision: 4161
http://bigdata.svn.sourceforge.net/bigdata/?rev=4161&view=rev
Author: thompsonbry
Date: 2011-01-22 12:03:03 +0000 (Sat, 22 Jan 2011)
Log Message:
-----------
Bug fix for BigdataValueFactoryImpl#createLiteral("foo",(URI)null).
See https://sourceforge.net/apps/trac/bigdata/ticket/226
I am also going to apply the fix to createLiteral() to the trunk.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataValueFactoryImpl.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/model/TestAll.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/model/TestFactory.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataValueFactoryImpl.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataValueFactoryImpl.java 2011-01-21 21:52:18 UTC (rev 4160)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/java/com/bigdata/rdf/model/BigdataValueFactoryImpl.java 2011-01-22 12:03:03 UTC (rev 4161)
@@ -291,7 +291,12 @@
public BigdataLiteralImpl createLiteral(final String label, URI datatype) {
- if (!(datatype instanceof BigdataURIImpl)) {
+ /*
+ * Note: The datatype parameter may be null per the Sesame API.
+ *
+ * See https://sourceforge.net/apps/trac/bigdata/ticket/226
+ */
+ if (datatype != null && !(datatype instanceof BigdataURIImpl)) {
datatype = createURI(datatype.stringValue());
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/model/TestAll.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/model/TestAll.java 2011-01-21 21:52:18 UTC (rev 4160)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/model/TestAll.java 2011-01-22 12:03:03 UTC (rev 4161)
@@ -60,8 +60,9 @@
public static Test suite()
{
- TestSuite suite = new TestSuite("RDF data model");
+ final TestSuite suite = new TestSuite("RDF data model");
+ // value factory test suite.
suite.addTestSuite(TestFactory.class);
return suite;
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/model/TestFactory.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/model/TestFactory.java 2011-01-21 21:52:18 UTC (rev 4160)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/rdf/model/TestFactory.java 2011-01-22 12:03:03 UTC (rev 4161)
@@ -28,15 +28,19 @@
package com.bigdata.rdf.model;
import javax.xml.datatype.DatatypeConfigurationException;
-import javax.xml.datatype.DatatypeConstants;
import javax.xml.datatype.DatatypeFactory;
import javax.xml.datatype.XMLGregorianCalendar;
-import org.omg.CORBA.portable.ValueFactory;
-
import junit.framework.TestCase2;
+import org.openrdf.model.URI;
+import org.openrdf.model.ValueFactory;
+
+import com.bigdata.rdf.internal.XSD;
+
/**
+ * Unit tests for {@link BigdataValueFactoryImpl}.
+ *
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
* @version $Id$
*/
@@ -55,10 +59,36 @@
super(name);
}
+ final BigdataValueFactory vf = BigdataValueFactoryImpl.getInstance(getName());
+
+ public void test_create_literal_xsdInt() {
+
+ final BigdataLiteral l1 = vf.createLiteral("12", XSD.INT);
+
+ assertEquals(XSD.INT, l1.getDatatype());
+
+ assertEquals(12, l1.intValue());
+
+ }
+
+ /**
+ * Unit test for {@link ValueFactory#createLiteral(String, URI)} when the
+ * datatype URI is <code>null</code>.
+ *
+ * @see https://sourceforge.net/apps/trac/bigdata/ticket/226
+ */
+ public void test_create_literal_datatypeIsNull() {
+
+ final BigdataLiteral l1 = vf.createLiteral("12", (URI) null);
+
+ assertEquals(null, l1.getDatatype());
+
+ assertEquals(12, l1.intValue());
+
+ }
+
public void test_gregorian() throws DatatypeConfigurationException {
- final BigdataValueFactory vf = BigdataValueFactoryImpl.getInstance(getName());
-
final XMLGregorianCalendar cal = DatatypeFactory.newInstance().newXMLGregorianCalendarDate(
2010,// year
1, // month,
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-21 21:52:25
|
Revision: 4160
http://bigdata.svn.sourceforge.net/bigdata/?rev=4160&view=rev
Author: thompsonbry
Date: 2011-01-21 21:52:18 +0000 (Fri, 21 Jan 2011)
Log Message:
-----------
Added getSharedVars() to BOpUtility.
More work on the runtime query optimizer. See https://sourceforge.net/apps/trac/bigdata/ticket/64.
Javadoc on AbstractSubqueryOp and SubqueryOp.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/AbstractSubqueryOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/JoinGraph.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/PartitionedJoinGroup.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestJGraph.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestPartitionedJoinGroup.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/AbstractJoinGraphTestCase.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphOnBSBMData.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphOnBarData.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphOnLubm.java
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/NoSolutionsException.java
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestAll.java
Removed Paths:
-------------
branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphWithRDF.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java 2011-01-21 16:46:41 UTC (rev 4159)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java 2011-01-21 21:52:18 UTC (rev 4160)
@@ -34,6 +34,7 @@
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
+import java.util.Set;
import org.apache.log4j.Logger;
@@ -1053,6 +1054,83 @@
return b;
}
-
+
+ /**
+ * Return the variable references shared by tw operators. All variables
+ * spanned by either {@link BOp} are considered.
+ *
+ * @param p
+ * An operator.
+ * @param c
+ * Another operator.
+ *
+ * @param p
+ * A predicate.
+ *
+ * @param c
+ * A constraint.
+ *
+ * @return The variables in common -or- <code>null</code> iff there are no
+ * variables in common.
+ *
+ * @throws IllegalArgumentException
+ * if the two either reference is <code>null</code>.
+ * @throws IllegalArgumentException
+ * if the reference are the same.
+ *
+ * @todo unit tests.
+ */
+ public static Set<IVariable<?>> getSharedVars(final BOp p, final BOp c) {
+
+ if (p == null)
+ throw new IllegalArgumentException();
+
+ if (c == null)
+ throw new IllegalArgumentException();
+
+ if (p == c)
+ throw new IllegalArgumentException();
+
+ // The set of variables which are shared.
+ final Set<IVariable<?>> sharedVars = new LinkedHashSet<IVariable<?>>();
+
+ // Collect the variables appearing anywhere in [p].
+ final Set<IVariable<?>> p1vars = new LinkedHashSet<IVariable<?>>();
+ {
+
+ final Iterator<IVariable<?>> itr = BOpUtility
+ .getSpannedVariables(p);
+
+ while (itr.hasNext()) {
+
+ p1vars.add(itr.next());
+
+ }
+
+ }
+
+ // Consider the variables appearing anywhere in [c].
+ {
+
+ final Iterator<IVariable<?>> itr = BOpUtility
+ .getSpannedVariables(c);
+
+ while (itr.hasNext()) {
+
+ final IVariable<?> avar = itr.next();
+
+ if (p1vars.contains(avar)) {
+
+ sharedVars.add(avar);
+
+ }
+
+ }
+
+ }
+
+ return sharedVars;
+
+ }
+
}
-
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/AbstractSubqueryOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/AbstractSubqueryOp.java 2011-01-21 16:46:41 UTC (rev 4159)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/AbstractSubqueryOp.java 2011-01-21 21:52:18 UTC (rev 4160)
@@ -84,6 +84,11 @@
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
* @version $Id$
+ *
+ * @todo There is relatively little difference between this class and SubqueryOp
+ * and we should consider converging them into a single concrete subquery
+ * operator with specializations for UNION and STEPS. The main difference
+ * is that the SubqueryOp can not run multiple subqueries.
*/
abstract public class AbstractSubqueryOp extends PipelineOp {
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/JoinGraph.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/JoinGraph.java 2011-01-21 16:46:41 UTC (rev 4159)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/JoinGraph.java 2011-01-21 21:52:18 UTC (rev 4160)
@@ -67,7 +67,6 @@
import com.bigdata.bop.join.PipelineJoin;
import com.bigdata.bop.join.PipelineJoin.PipelineJoinStats;
import com.bigdata.bop.rdf.join.DataSetJoin;
-import com.bigdata.bop.solutions.SliceOp;
import com.bigdata.relation.IRelation;
import com.bigdata.relation.accesspath.BufferClosedException;
import com.bigdata.relation.accesspath.IAccessPath;
@@ -75,6 +74,7 @@
import com.bigdata.relation.accesspath.ThickAsynchronousIterator;
import com.bigdata.striterator.Dechunkerator;
import com.bigdata.striterator.IChunkedIterator;
+import com.bigdata.util.concurrent.Haltable;
/**
* A join graph with annotations for estimated cardinality and other details in
@@ -1450,13 +1450,13 @@
*
* @see #getVertices()
*/
- public IPredicate[] getPredicates() {
+ public IPredicate<?>[] getPredicates() {
// The vertices in the selected evaluation order.
final Vertex[] vertices = getVertices();
// The predicates in the same order as the vertices.
- final IPredicate[] preds = new IPredicate[vertices.length];
+ final IPredicate<?>[] preds = new IPredicate[vertices.length];
for (int i = 0; i < vertices.length; i++) {
@@ -1469,6 +1469,16 @@
}
/**
+ * Return the {@link BOp} identifiers of the predicates associated with
+ * each vertex in path order.
+ */
+ public int[] getVertexIds() {
+
+ return getVertexIds(edges);
+
+ }
+
+ /**
* Return the {@link BOp} identifiers of the predicates associated with
* each vertex in path order.
*/
@@ -1590,10 +1600,6 @@
* join path in each round then this would help to establish a
* better estimate in successive rounds.]
*
- * FIXME CONSTRAINT ORDERING : It is illegal to add a vertex to the
- * path if any variable appearing in its CONSTRAINTS would not be
- * bound.
- *
* FIXME CONSTRAINT ORDERING : Rather than constraints imposing an
* ordering on joins, constraints need to be attached dynamically to
* the first join for which their variables are known to be bound.
@@ -1848,7 +1854,7 @@
* @todo unit test for a constraint using a variable which is never
* bound.
*/
- public JGraph(final IPredicate[] v, final IConstraint[] constraints) {
+ public JGraph(final IPredicate<?>[] v, final IConstraint[] constraints) {
if (v == null)
throw new IllegalArgumentException();
@@ -1871,27 +1877,33 @@
* Identify the edges by looking for shared variables among the
* predicates.
*
- * Note: Variables may appear in the arguments of the predicate,
- * e.g., spo(?s,rdf:type,?o).
+ * Note: Variables appear in predicates or in constraints. Edges are
+ * created to represent possible joins between predicates based on
+ * those shared variables. There are two cases:
*
- * Note: Variables may ALSO appear in the CONSTRAINTS (imposed on
- * the binding sets) or FILTERS (imposed either on the local or
- * remote access path). For example, that a variable bound by
- * another predicate must take on a value having some mathematical
- * relationship to a variable bound by the predicate, e.g., BSBM Q5.
- * When a variable appears in a constraint but does not appear as an
- * argument to the predicate, then there is an additional
- * requirement that the variable MUST become bound before the
- * predicate may be evaluated (again, BSBM Q5 has this form).
+ * (1) When the target predicate shares a variable with the source
+ * predicate, then we always create an edge between those predicates
+ * to represent a possible join.
*
- * Note: If a vertex does not share ANY variables (neither in the
- * arguments of the predicate nor in its constraints or filters)
- * then it can be paired with any of the other vertices. However, in
- * such cases we always run such vertices last as they can not
- * restrict the cardinality of the rest of the join graph. Such
- * vertices are therefore inserted into a separate set and appended
- * to the join path once all edges having shared variables have been
- * exhausted.
+ * (2) When the source predicate shares a variable with a constraint
+ * which also shares a variable with the target predicate, then we
+ * will also create an edge to represent a possible join.
+ *
+ * The second case handles the case where variables are transitively
+ * shared through a constraint, but not directly shared between the
+ * predicates. BSBM Q5 is an example of this case.
+ *
+ * Note: If applying these two rules fails to create any edges for
+ * some vertex, then it does not share ANY variables and can be
+ * paired with ANY of the other vertices. However, we always run
+ * such vertices last as they can not restrict the cumulative
+ * cardinality of the solutions. Such vertices are therefore
+ * inserted into a separate set and appended to the join path once
+ * all edges having shared variables have been exhausted.
+ *
+ * FIXME VERTICES WHICH SHARE VARS THROUGH A CONSTRAINT.
+ *
+ * FIXME VERTICES WITH NO SHARED VARS.
*/
{
@@ -1917,20 +1929,14 @@
// consider a possible target vertex.
final IPredicate<?> p2 = v[j];
- final Set<IVariable<?>> shared = getSharedVars(p1, p2);
+ final Set<IVariable<?>> shared = BOpUtility
+ .getSharedVars(p1, p2);
if (shared != null && !shared.isEmpty()) {
/*
- * The source and target vertices share var(s).
- *
- * Note: A predicate having a variable which appears
- * in a CONSTRAINT MUST NOT be added to the join
- * path until that variable would be bound.
- * Therefore, when selecting the vertices to be used
- * to extend a join path, we must consider whether
- * or not the join path would bind the variable(s)
- * appearing in the CONSTRAINT.
+ * The source and target vertices share one or more
+ * variable(s).
*/
if (log.isDebugEnabled())
@@ -1945,8 +1951,68 @@
nmatched++;
- }
+ } else if (constraints != null) {
+ /*
+ * The source and target vertices do not directly
+ * share any variable(s). However, there may be a
+ * constraint which shares a variable with both the
+ * source and target vertex. If such a constraint is
+ * found, then we add an edge now as that join is
+ * potentially constrained (less than the full
+ * Cartesian cross product).
+ *
+ * Note: While this identifies possible joins via a
+ * constraint, such joins are only legal when all
+ * variables used by the constraint are known to be
+ * bound.
+ *
+ * FIXME We have to reject edges unless there are
+ * variable(s) which are directly shared between the
+ * source and target vertex until all the variables
+ * spanned by the constraint which licenses the join
+ * have become bound. [Consider marking these edges
+ * directly so we know that we need to test and see
+ * whether or not a constraint exists which shares
+ * at least one variable with both vertices and that
+ * all variables in that constraint are bound.]
+ *
+ * FIXME Since we can attach more than one
+ * constraint to a vertex, we may have to ask
+ * whether any set of the available constraints
+ * shares at least one variable with the source and
+ * target vertices. [That is, do they have to share
+ * variables via the same constraint?!?]
+ */
+
+ for(IConstraint c : constraints) {
+
+ if(BOpUtility.getSharedVars(p1, c).isEmpty())
+ continue;
+
+ if(BOpUtility.getSharedVars(p2, c).isEmpty())
+ continue;
+
+ if (log.isDebugEnabled())
+ log
+ .debug("vertices shared variable(s) via constraint: v1="
+ + p1
+ + ", v2="
+ + p2
+ + ", c=" + c);
+
+ tmp.add(new Edge(V[i], V[j], shared));
+
+ sharedEdgeVertices.add(V[i]);
+
+ sharedEdgeVertices.add(V[j]);
+
+ nmatched++;
+
+ }
+
+ }
+
}
if (nmatched == 0 && !sharedEdgeVertices.contains(V[i])) {
@@ -1972,14 +2038,24 @@
if(!unsharedEdgeVertices.isEmpty()) {
/*
- * FIXME This needs to be supported. We should explore and
- * generate the join paths based on only those vertices
- * which do share variables (and hence for which we have
- * defined edges). Once the vertices which share variables
- * have been exhausted, we should simply append edges for
- * the vertices which do not share variables in an arbitrary
- * order (they will be run last since they can not constrain
- * the evaluation).
+ * FIXME NO SHARED VARS : RUN LAST. This needs to be
+ * supported. When vertices that do not share variables
+ * either directly or via a constraint then they should run
+ * last as they can not constrain the query. In this case,
+ * they are not considered by the runtime optimizer when
+ * building up the join path until all vertices which share
+ * variables have been exhausted. At that point, the
+ * remaining vertices are just appended to whatever join
+ * path was selected as having the lowest cumulative
+ * estimated cardinality.
+ *
+ * However, if there exists for a vertex which otherwise
+ * does not share variables a constraint which should be
+ * evaluated against that vertex, then that constraint
+ * provides the basis for a edge (aka join). In this case,
+ * an edge must be created for the vertex based on the
+ * shared variable in the constraint and its position in the
+ * join path will be decided by the runtime optimizer.
*/
throw new UnsupportedOperationException(
@@ -2917,8 +2993,8 @@
final BOpIdFactory idFactory = new BOpIdFactory();
// Generate the query from the join path.
- final PipelineOp queryOp = JoinGraph.getQuery(idFactory, p
- .getPredicates(), getConstraints());
+ final PipelineOp queryOp = PartitionedJoinGroup.getQuery(idFactory,
+ p.getPredicates(), getConstraints());
// Run the query, blocking until it is done.
JoinGraph.runSubquery(context, queryOp);
@@ -2996,318 +3072,95 @@
*/
/**
- * Generate a query plan from an ordered collection of predicates.
+ * Execute the selected join path.
+ * <p>
+ * Note: When executing the query, it is actually being executed as a
+ * subquery. Therefore we have to take appropriate care to ensure that the
+ * results are copied out of the subquery and into the parent query. See
+ * {@link AbstractSubqueryOp} for how this is done.
*
- * @param p
- * The join path.
+ * @throws Exception
*
- * @return The query plan.
+ * @todo When we execute the query, we should clear the references to the
+ * samples (unless they are exact, in which case they can be used as
+ * is) in order to release memory associated with those samples if the
+ * query is long running. Samples must be held until we have
+ * identified the final join path since each vertex will be used by
+ * each maximum length join path and we use the samples from the
+ * vertices to re-sample the surviving join paths in each round.
*
- * FIXME Verify that constraints are attached correctly to the
- * returned query.
+ * @todo If there is a slice on the outer query, then the query result may
+ * well be materialized by now.
+ *
+ * @todo If there are source binding sets then they need to be applied above
+ * (when we are sampling) and below (when we evaluate the selected
+ * join path).
+ *
+ * FIXME runQuery() is not working correctly. The query is being
+ * halted by a {@link BufferClosedException} which appears before it
+ * has materialized the necessary results.
*/
- static public PipelineOp getQuery(final BOpIdFactory idFactory,
- final IPredicate[] preds, final IConstraint[] constraints) {
+ static private void runSubquery(
+ final BOpContext<IBindingSet> parentContext,
+ final PipelineOp queryOp) throws Exception {
- if (constraints != null && constraints.length != 0) {
- // FIXME Constraints must be attached to joins.
- throw new UnsupportedOperationException(
- "Constraints must be attached to joins!");
- }
-
- final PipelineJoin[] joins = new PipelineJoin[preds.length];
+ final QueryEngine queryEngine = parentContext.getRunningQuery()
+ .getQueryEngine();
-// final PipelineOp startOp = new StartOp(new BOp[] {},
-// NV.asMap(new NV[] {//
-// new NV(Predicate.Annotations.BOP_ID, idFactory
-// .nextId()),//
-// new NV(SliceOp.Annotations.EVALUATION_CONTEXT,
-// BOpEvaluationContext.CONTROLLER),//
-// }));
-//
-// PipelineOp lastOp = startOp;
- PipelineOp lastOp = null;
+ /*
+ * Run the query.
+ *
+ * @todo pass in the source binding sets here and also when sampling the
+ * vertices.
+ */
-// final Set<IVariable> vars = new LinkedHashSet<IVariable>();
-// for(IPredicate p : preds) {
-// for(BOp arg : p.args()) {
-// if(arg instanceof IVariable) {
-// vars.add((IVariable)arg);
-// }
-// }
-// }
-
- for (int i = 0; i < preds.length; i++) {
+ IAsynchronousIterator<IBindingSet[]> subquerySolutionItr = null;
- // The next vertex in the selected join order.
- final IPredicate p = preds[i];
+ final IRunningQuery runningQuery = queryEngine.eval(queryOp);
- final List<NV> anns = new LinkedList<NV>();
+ try {
- anns.add(new NV(PipelineJoin.Annotations.PREDICATE, p));
+ // Iterator visiting the subquery solutions.
+ subquerySolutionItr = runningQuery.iterator();
- anns.add(new NV(PipelineJoin.Annotations.BOP_ID, idFactory
- .nextId()));
+ // Copy solutions from the subquery to the query.
+ final long nout = BOpUtility.copy(subquerySolutionItr,
+ parentContext.getSink(), null/* sink2 */,
+ null/* constraints */, null/* stats */);
-// anns.add(new NV(PipelineJoin.Annotations.EVALUATION_CONTEXT, BOpEvaluationContext.ANY));
-//
-// anns.add(new NV(PipelineJoin.Annotations.SELECT, vars.toArray(new IVariable[vars.size()])));
+ System.out.println("nout=" + nout);
- final PipelineJoin joinOp = new PipelineJoin(
- lastOp == null ? new BOp[0] : new BOp[] { lastOp },
- anns.toArray(new NV[anns.size()]));
+ // verify no problems.
+ runningQuery.get();
- joins[i] = joinOp;
+ System.out.println("Future Ok");
- lastOp = joinOp;
+ } catch (Throwable t) {
- }
+ if (Haltable.isTerminationByInterrupt(t)) {
-// final PipelineOp queryOp = lastOp;
+ // normal termination.
+ return;
- /*
- * FIXME Why does wrapping with this slice appear to be
- * necessary? (It is causing runtime errors when not wrapped).
- * Is this a bopId collision which is not being detected?
- */
- final PipelineOp queryOp = new SliceOp(new BOp[] { lastOp }, NV
- .asMap(new NV[] {
- new NV(JoinGraph.Annotations.BOP_ID, idFactory.nextId()), //
- new NV(JoinGraph.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.CONTROLLER) }) //
- );
-
- return queryOp;
-
- }
-
- /**
- * Execute the selected join path.
- * <p>
- * Note: When executing the query, it is actually being executed as a
- * subquery. Therefore we have to take appropriate care to ensure that the
- * results are copied out of the subquery and into the parent query. See
- * {@link AbstractSubqueryOp} for how this is done.
- *
- * @todo When we execute the query, we should clear the references to the
- * samples (unless they are exact, in which case they can be used as
- * is) in order to release memory associated with those samples if the
- * query is long running. Samples must be held until we have
- * identified the final join path since each vertex will be used by
- * each maximum length join path and we use the samples from the
- * vertices to re-sample the surviving join paths in each round.
- *
- * @todo If there is a slice on the outer query, then the query result may
- * well be materialized by now.
- *
- * @todo If there are source binding sets then they need to be applied above
- * (when we are sampling) and below (when we evaluate the selected
- * join path).
- *
- * FIXME runQuery() is not working correctly. The query is being
- * halted by a {@link BufferClosedException} which appears before it
- * has materialized the necessary results.
- */
- static public void runSubquery(final BOpContext<IBindingSet> parentContext,
- final PipelineOp queryOp) {
-
- IAsynchronousIterator<IBindingSet[]> subquerySolutionItr = null;
-
- try {
-
- if (log.isInfoEnabled())
- log.info("Running: " + BOpUtility.toString(queryOp));
-
- final PipelineOp startOp = (PipelineOp) BOpUtility
- .getPipelineStart(queryOp);
-
- if (log.isInfoEnabled())
- log.info("StartOp: " + BOpUtility.toString(startOp));
-
- // Run the query.
- final UUID queryId = UUID.randomUUID();
-
- final QueryEngine queryEngine = parentContext.getRunningQuery()
- .getQueryEngine();
-
- final IRunningQuery runningQuery = queryEngine
- .eval(
- queryId,
- queryOp,
- new LocalChunkMessage<IBindingSet>(
- queryEngine,
- queryId,
- startOp.getId()/* startId */,
- -1 /* partitionId */,
- /*
- * @todo pass in the source binding sets
- * here and also when sampling the
- * vertices.
- */
- new ThickAsynchronousIterator<IBindingSet[]>(
- new IBindingSet[][] { new IBindingSet[] { new HashBindingSet() } })));
-
- // Iterator visiting the subquery solutions.
- subquerySolutionItr = runningQuery.iterator();
-
- // Copy solutions from the subquery to the query.
- final long nout = BOpUtility
- .copy(subquerySolutionItr, parentContext.getSink(),
- null/* sink2 */, null/* constraints */, null/* stats */);
-
- System.out.println("nout=" + nout);
-
- // verify no problems.
- runningQuery.get();
-
- System.out.println("Future Ok");
-
- } catch (Throwable t) {
-
- log.error(t,t);
-
- /*
- * If a subquery fails, then propagate the error to the parent
- * and rethrow the first cause error out of the subquery.
- */
- throw new RuntimeException(parentContext.getRunningQuery()
- .halt(t));
-
- } finally {
-
- if (subquerySolutionItr != null)
- subquerySolutionItr.close();
-
- }
-
- }
-
- /**
- * Return the variables in common for two {@link IPredicate}s. All variables
- * spanned by either {@link IPredicate} are considered.
- * <p>
- * Note: Variables may appear in the predicates operands, in the
- * {@link Annotations#CONSTRAINTS} associated with the
- * predicate, and in the {@link IPredicate.Annotations#ACCESS_PATH_FILTER}
- * or {@link IPredicate.Annotations#INDEX_LOCAL_FILTER}.
- * <p>
- * Note: A variable must become bound before it may be evaluated in
- * {@link Annotations#CONSTRAINTS}, an
- * {@link IPredicate.Annotations#ACCESS_PATH_FILTER} or an
- * {@link IPredicate.Annotations#INDEX_LOCAL_FILTER}. This means that the
- * {@link IPredicate}s which can bind the variable must be ordered before
- * those which merely test the variable.
- *
- *
- * @param p1
- * A predicate.
- *
- * @param p2
- * A different predicate.
- *
- * @return The variables in common -or- <code>null</code> iff there are no
- * variables in common.
- *
- * @throws IllegalArgumentException
- * if the two predicates are the same reference.
- *
- * @todo It should be an error if a variable appear in a test is not bound
- * by any possible join path. However, note that it may not be
- * possible to determine this by local examination of a join graph
- * since we do not know which variables may be presented as already
- * bound when the join graph is evaluated (but we can only run the
- * join graph currently against static source binding sets and for
- * that case this is knowable).
- *
- * @todo When a variable is only optionally bound and it is discovered at
- * runtime that the variable is not bound when it is considered by a
- * CONSTRAINT, FILTER, etc., then the SPARQL semantics are that
- * evaluation should produce a 'type' error which would cause the
- * solution should fail (at least within its current join group). See
- * https://sourceforge.net/apps/trac/bigdata/ticket/179.
- *
- * @todo Unit tests, including those which verify that variables appearing
- * in the constraints are reported as shared with those appearing in
- * the predicates operands.
- */
- static Set<IVariable<?>> getSharedVars(final IPredicate p1, final IPredicate p2) {
-
- // The set of variables which are shared by those predicates.
- final Set<IVariable<?>> sharedVars = new LinkedHashSet<IVariable<?>>();
-
- /*
- * Collect the variables appearing anyway in [p1], including the
- * predicate's operands and its constraints, filters, etc.
- */
- final Set<IVariable<?>> p1vars = new LinkedHashSet<IVariable<?>>();
- {
-
- final Iterator<IVariable<?>> itr = BOpUtility
- .getSpannedVariables(p1);
-
- while (itr.hasNext()) {
-
- p1vars.add(itr.next());
-
}
- }
+ // log.error(t,t);
- /*
- * Consider the variables appearing anyway in [p2], including the
- * predicate's operands and its constraints, filters, etc.
- */
- {
+ /*
+ * Propagate the error to the parent and rethrow the first cause
+ * error out of the subquery.
+ */
+ throw new RuntimeException(parentContext.getRunningQuery().halt(t));
- final Iterator<IVariable<?>> itr = BOpUtility
- .getSpannedVariables(p2);
+ } finally {
- while (itr.hasNext()) {
+ runningQuery.cancel(true/* mayInterruptIfRunning */);
- final IVariable<?> avar = itr.next();
-
- if(p1vars.contains(avar)) {
+ if (subquerySolutionItr != null)
+ subquerySolutionItr.close();
- sharedVars.add(avar);
-
- }
-
- }
-
}
-
- return sharedVars;
}
- /**
- * Exception thrown when the join graph does not have any solutions in the
- * data (running the query does not produce any results).
- */
- public static class NoSolutionsException extends RuntimeException
- {
-
- /**
- *
- */
- private static final long serialVersionUID = 1L;
-
- public NoSolutionsException() {
- super();
- }
-
- public NoSolutionsException(String message, Throwable cause) {
- super(message, cause);
- }
-
- public NoSolutionsException(String message) {
- super(message);
- }
-
- public NoSolutionsException(Throwable cause) {
- super(cause);
- }
-
- }
-
}
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/NoSolutionsException.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/NoSolutionsException.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/NoSolutionsException.java 2011-01-21 21:52:18 UTC (rev 4160)
@@ -0,0 +1,31 @@
+package com.bigdata.bop.controller;
+
+/**
+ * Exception thrown when the join graph does not have any solutions in the
+ * data (running the query does not produce any results).
+ */
+public class NoSolutionsException extends RuntimeException
+{
+
+ /**
+ *
+ */
+ private static final long serialVersionUID = 1L;
+
+ public NoSolutionsException() {
+ super();
+ }
+
+ public NoSolutionsException(String message, Throwable cause) {
+ super(message, cause);
+ }
+
+ public NoSolutionsException(String message) {
+ super(message);
+ }
+
+ public NoSolutionsException(Throwable cause) {
+ super(cause);
+ }
+
+}
\ No newline at end of file
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/NoSolutionsException.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/PartitionedJoinGroup.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/PartitionedJoinGroup.java 2011-01-21 16:46:41 UTC (rev 4159)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/PartitionedJoinGroup.java 2011-01-21 21:52:18 UTC (rev 4160)
@@ -10,11 +10,17 @@
import org.apache.log4j.Logger;
+import com.bigdata.bop.BOp;
+import com.bigdata.bop.BOpEvaluationContext;
+import com.bigdata.bop.BOpIdFactory;
import com.bigdata.bop.BOpUtility;
import com.bigdata.bop.IConstraint;
import com.bigdata.bop.IPredicate;
import com.bigdata.bop.IVariable;
-import com.bigdata.bop.controller.JoinGraph.NoSolutionsException;
+import com.bigdata.bop.NV;
+import com.bigdata.bop.PipelineOp;
+import com.bigdata.bop.join.PipelineJoin;
+import com.bigdata.bop.solutions.SliceOp;
/**
* Class accepts a join group and partitions it into a join graph and a tail
@@ -54,10 +60,6 @@
* predicate, which is meaningless in an of itself because the P is
* magical.]
*
- * @todo write a method which returns the set of constraints which should be run
- * for the last predicate in a given join path (a join path is just an
- * ordered array of predicates).
- *
* FIXME Add a method to generate a runnable query plan from a collection
* of predicates and constraints. This is a bit different for the join
* graph and the optionals in the tail plan. The join graph itself should
@@ -220,7 +222,54 @@
path[i] = p;
}
+
+ return getJoinGraphConstraints(path, joinGraphConstraints
+ .toArray(new IConstraint[joinGraphConstraints.size()]))[pathIds.length - 1];
+
+ }
+ /**
+ * Given a join path, return the set of constraints to be associated with
+ * each join in that join path. Only those constraints whose variables are
+ * known to be bound will be attached.
+ *
+ * @param path
+ * The join path.
+ * @param joinGraphConstraints
+ * The constraints to be applied to the join path (optional).
+ *
+ * @return The constraints to be paired with each element of the join path.
+ *
+ * @throws IllegalArgumentException
+ * if the join path is <code>null</code>.
+ * @throws IllegalArgumentException
+ * if the join path is empty.
+ * @throws IllegalArgumentException
+ * if any element of the join path is <code>null</code>.
+ * @throws IllegalArgumentException
+ * if any element of the join graph constraints is
+ * <code>null</code>.
+ *
+ * @todo It should be an error if a variable appear in a constraint is not
+ * bound by any possible join path. However, it may not be possible to
+ * determine this by local examination of a join graph since we do not
+ * know which variables may be presented as already bound when the
+ * join graph is evaluated (but we can only run the join graph
+ * currently against static source binding sets and for that case this
+ * is knowable).
+ */
+ static public IConstraint[][] getJoinGraphConstraints(
+ final IPredicate<?>[] path, final IConstraint[] joinGraphConstraints) {
+
+ if (path == null)
+ throw new IllegalArgumentException();
+
+ if (path.length == 0)
+ throw new IllegalArgumentException();
+
+ // the set of constraints for each predicate in the join path.
+ final IConstraint[][] ret = new IConstraint[path.length][];
+
/*
* For each predicate in the path in the given order, figure out which
* constraint(s) would attach to that predicate based on which variables
@@ -234,17 +283,20 @@
// the set of constraints which have been consumed.
final Set<IConstraint> used = new LinkedHashSet<IConstraint>();
- // the set of constraints for the last predicate in the join path.
- final List<IConstraint> ret = new LinkedList<IConstraint>();
-
- for(int i = 0; i<path.length; i++) {
+ for (int i = 0; i < path.length; i++) {
- // true iff this is the last join in the path.
- final boolean lastJoin = i == path.length - 1;
+// // true iff this is the last join in the path.
+// final boolean lastJoin = i == path.length - 1;
// a predicate in the path.
final IPredicate<?> p = path[i];
+ if (p == null)
+ throw new IllegalArgumentException();
+
+ // the constraints for the current predicate in the join path.
+ final List<IConstraint> constraints = new LinkedList<IConstraint>();
+
{
/*
* Visit the variables used by the predicate (and bound by it
@@ -263,80 +315,80 @@
}
}
-
- // consider each constraint.
- for(IConstraint c : joinGraphConstraints) {
- if (used.contains(c)) {
+ if (joinGraphConstraints != null) {
+
+ // consider each constraint.
+ for (IConstraint c : joinGraphConstraints) {
+
+ if (c == null)
+ throw new IllegalArgumentException();
+
+ if (used.contains(c)) {
+ /*
+ * Skip constraints which were already assigned to
+ * predicates before this one in the join path.
+ */
+ continue;
+ }
+
/*
- * Skip constraints which were already assigned to
- * predicates before this one in the join path.
+ * true iff all variables used by this constraint are bound
+ * at this point in the join path.
*/
- continue;
- }
+ boolean allVarsBound = true;
- /*
- * true iff all variables used by this constraint are bound at
- * this point in the join path.
- */
- boolean allVarsBound = true;
+ // visit the variables used by this constraint.
+ final Iterator<IVariable<?>> vitr = BOpUtility
+ .getSpannedVariables(c);
- // visit the variables used by this constraint.
- final Iterator<IVariable<?>> vitr = BOpUtility
- .getSpannedVariables(c);
+ while (vitr.hasNext()) {
- while (vitr.hasNext()) {
+ final IVariable<?> var = vitr.next();
- final IVariable<?> var = vitr.next();
-
- if(!boundVars.contains(var)) {
-
- allVarsBound = false;
+ if (!boundVars.contains(var)) {
- break;
+ allVarsBound = false;
- }
+ break;
- }
+ }
- if (allVarsBound) {
+ }
- /*
- * All variables have become bound for this constraint, so
- * add it to the set of "used" constraints.
- */
-
- used.add(c);
+ if (allVarsBound) {
- if (log.isDebugEnabled()) {
- log.debug("Constraint attached at index " + i + " of "
- + path.length + ", bopId=" + p.getId()
- + ", constraint=" + c);
- }
-
- if (lastJoin) {
-
/*
- * If we are on the last join in the join path, then
- * this constraint is one of the ones that we will
- * return.
+ * All variables have become bound for this constraint,
+ * so add it to the set of "used" constraints.
*/
-
- ret.add(c);
- }
+ used.add(c);
- } // if(allVarsBound)
-
- } // next constraint
+ if (log.isDebugEnabled()) {
+ log.debug("Constraint attached at index " + i
+ + " of " + path.length + ", bopId="
+ + p.getId() + ", constraint=" + c);
+ }
+
+ constraints.add(c);
+
+ } // if(allVarsBound)
+
+ } // next constraint
+
+ } // joinGraphConstraints != null;
+
+ // store the constraint[] for that predicate.
+ ret[i] = constraints.toArray(new IConstraint[constraints.size()]);
} // next predicate in the join path.
/*
- * Return the set of constraints to be applied as of the last predicate
- * in the join path.
+ * Return the set of constraints associated with each predicate in the
+ * join path.
*/
- return ret.toArray(new IConstraint[ret.size()]);
+ return ret;
}
@@ -589,4 +641,83 @@
}
+ /**
+ * Generate a query plan from an ordered collection of predicates.
+ *
+ * @param p
+ * The join path.
+ *
+ * @return The query plan.
+ *
+ * FIXME Select only those variables required by downstream
+ * processing or explicitly specified by the caller (in the case
+ * when this is a subquery, the caller has to declare which
+ * variables are selected and will be returned out of the subquery).
+ *
+ * FIXME For scale-out, we need to either mark the join's evaluation
+ * context based on whether or not the access path is local or
+ * remote (and whether the index is key-range distributed or hash
+ * partitioned).
+ */
+ static public PipelineOp getQuery(final BOpIdFactory idFactory,
+ final IPredicate<?>[] preds, final IConstraint[] constraints) {
+
+ // figure out which constraints are attached to which predicates.
+ final IConstraint[][] assignedConstraints = PartitionedJoinGroup
+ .getJoinGraphConstraints(preds, constraints);
+
+ final PipelineJoin<?>[] joins = new PipelineJoin[preds.length];
+
+ PipelineOp lastOp = null;
+
+ for (int i = 0; i < preds.length; i++) {
+
+ // The next vertex in the selected join order.
+ final IPredicate<?> p = preds[i];
+
+ final List<NV> anns = new LinkedList<NV>();
+
+ anns.add(new NV(PipelineJoin.Annotations.PREDICATE, p));
+
+ anns.add(new NV(PipelineJoin.Annotations.BOP_ID, idFactory
+ .nextId()));
+
+// anns.add(new NV(PipelineJoin.Annotations.EVALUATION_CONTEXT, BOpEvaluationContext.ANY));
+//
+// anns.add(new NV(PipelineJoin.Annotations.SELECT, vars.toArray(new IVariable[vars.size()])));
+
+ if (assignedConstraints[i] != null
+ && assignedConstraints[i].length > 0)
+ anns
+ .add(new NV(PipelineJoin.Annotations.CONSTRAINTS,
+ assignedConstraints[i]));
+
+ final PipelineJoin<?> joinOp = new PipelineJoin(
+ lastOp == null ? new BOp[0] : new BOp[] { lastOp }, anns
+ .toArray(new NV[anns.size()]));
+
+ joins[i] = joinOp;
+
+ lastOp = joinOp;
+
+ }
+
+// final PipelineOp queryOp = lastOp;
+
+ /*
+ * FIXME Why does wrapping with this slice appear to be
+ * necessary? (It is causing runtime errors when not wrapped).
+ * Is this a bopId collision which is not being detected?
+ */
+ final PipelineOp queryOp = new SliceOp(new BOp[] { lastOp }, NV
+ .asMap(new NV[] {
+ new NV(JoinGraph.Annotations.BOP_ID, idFactory.nextId()), //
+ new NV(JoinGraph.Annotations.EVALUATION_CONTEXT,
+ BOpEvaluationContext.CONTROLLER) }) //
+ );
+
+ return queryOp;
+
+ }
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java 2011-01-21 16:46:41 UTC (rev 4159)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java 2011-01-21 21:52:18 UTC (rev 4160)
@@ -57,6 +57,8 @@
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
* @version $Id$
+ *
+ * @see AbstractSubqueryOp
*/
public class SubqueryOp extends PipelineOp {
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestJGraph.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestJGraph.java 2011-01-21 16:46:41 UTC (rev 4159)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestJGraph.java 2011-01-21 21:52:18 UTC (rev 4160)
@@ -87,7 +87,7 @@
// fail("write test");
// }
//
-// // @todo also getEdgeCount()
+// // and also getEdgeCount()
// public void test_getEdges() {
// fail("write test");
// }
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestPartitionedJoinGroup.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestPartitionedJoinGroup.java 2011-01-21 16:46:41 UTC (rev 4159)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestPartitionedJoinGroup.java 2011-01-21 21:52:18 UTC (rev 4160)
@@ -456,9 +456,6 @@
/**
* @todo test with headPlan.
- *
- * @todo test logic to attach constraints to non-optional joins based on a
- * given join path (not yet written).
*/
public void test_something() {
fail("write tests");
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/AbstractJoinGraphTestCase.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/AbstractJoinGraphTestCase.java 2011-01-21 16:46:41 UTC (rev 4159)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/AbstractJoinGraphTestCase.java 2011-01-21 21:52:18 UTC (rev 4160)
@@ -43,7 +43,7 @@
import com.bigdata.bop.IConstraint;
import com.bigdata.bop.IPredicate;
import com.bigdata.bop.PipelineOp;
-import com.bigdata.bop.controller.JoinGraph;
+import com.bigdata.bop.controller.PartitionedJoinGroup;
import com.bigdata.bop.controller.JoinGraph.JGraph;
import com.bigdata.bop.controller.JoinGraph.Path;
import com.bigdata.bop.engine.BOpStats;
@@ -217,7 +217,7 @@
* JVM run using the known solutions produced by the runtime versus
* static query optimizers.
*/
- protected void doTest(final IPredicate[] preds,
+ protected void doTest(final IPredicate<?>[] preds,
final IConstraint[] constraints) throws Exception {
if (warmUp)
@@ -228,7 +228,7 @@
* Run the runtime query optimizer once (its cost is not counted
* thereafter).
*/
- final IPredicate[] runtimePredOrder = runRuntimeQueryOptimizer(
+ final IPredicate<?>[] runtimePredOrder = runRuntimeQueryOptimizer(
getQueryEngine(), limit, nedges, preds, constraints);
long totalRuntimeTime = 0;
@@ -296,9 +296,10 @@
*
* @throws Exception
*/
- static protected IPredicate[] runRuntimeQueryOptimizer(
+ static protected IPredicate<?>[] runRuntimeQueryOptimizer(
final QueryEngine queryEngine, final int limit, final int nedges,
- final IPredicate[] preds, IConstraint[] constraints) throws Exception {
+ final IPredicate<?>[] preds, IConstraint[] constraints)
+ throws Exception {
final Logger tmp = Logger.getLogger(QueryLog.class);
final Level oldLevel = tmp.getEffectiveLevel();
@@ -329,8 +330,8 @@
* @return The predicates in order as recommended by the static query
* optimizer.
*/
- static protected IPredicate[] runStaticQueryOptimizer(
- final QueryEngine queryEngine, final IPredicate[] preds) {
+ static protected IPredicate<?>[] runStaticQueryOptimizer(
+ final QueryEngine queryEngine, final IPredicate<?>[] preds) {
final BOpContextBase context = new BOpContextBase(queryEngine);
@@ -351,7 +352,7 @@
final int[] ids = new int[order.length];
- final IPredicate[] out = new IPredicate[order.length];
+ final IPredicate<?>[] out = new IPredicate[order.length];
for (int i = 0; i < order.length; i++) {
@@ -374,15 +375,9 @@
* @return The elapsed query time (ms).
*/
private static long runQuery(final String msg,
- final QueryEngine queryEngine, final IPredicate[] predOrder,
+ final QueryEngine queryEngine, final IPredicate<?>[] predOrder,
final IConstraint[] constraints) throws Exception {
- if (constraints != null && constraints.length != 0) {
- // FIXME Constraints must be attached to joins.
- throw new UnsupportedOperationException(
- "Constraints must be attached to joins!");
- }
-
if (log.isInfoEnabled())
log.info("Running " + msg);
@@ -400,38 +395,46 @@
}
- final PipelineOp queryOp = JoinGraph.getQuery(idFactory, predOrder,
- constraints);
+ final PipelineOp queryOp = PartitionedJoinGroup.getQuery(idFactory,
+ predOrder, constraints);
// submit query to runtime optimizer.
final IRunningQuery q = queryEngine.eval(queryOp);
- // drain the query results.
- long nout = 0;
- long nchunks = 0;
- final IAsynchronousIterator<IBindingSet[]> itr = q.iterator();
try {
- while (itr.hasNext()) {
- final IBindingSet[] chunk = itr.next();
- nout += chunk.length;
- nchunks++;
+
+ // drain the query results.
+ long nout = 0;
+ long nchunks = 0;
+ final IAsynchronousIterator<IBindingSet[]> itr = q.iterator();
+ try {
+ while (itr.hasNext()) {
+ final IBindingSet[] chunk = itr.next();
+ nout += chunk.length;
+ nchunks++;
+ }
+ } finally {
+ itr.close();
}
- } finally {
- itr.close();
- }
- // check the Future for the query.
- q.get();
+ // check the Future for the query.
+ q.get();
- // show the results.
- final BOpStats stats = q.getStats().get(queryOp.getId());
+ // show the results.
+ final BOpStats stats = q.getStats().get(queryOp.getId());
- System.err.println(msg + " : ids=" + Arrays.toString(ids)
- + ", elapsed=" + q.getElapsed() + ", nout=" + nout
- + ", nchunks=" + nchunks + ", stats=" + stats);
-
- return q.getElapsed();
+ System.err.println(msg + " : ids=" + Arrays.toString(ids)
+ + ", elapsed=" + q.getElapsed() + ", nout=" + nout
+ + ", nchunks=" + nchunks + ", stats=" + stats);
+ return q.getElapsed();
+
+ } finally {
+
+ q.cancel(true/* mayInterruptIfRunning */);
+
+ }
+
}
/**
Added: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestAll.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestAll.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestAll.java 2011-01-21 21:52:18 UTC (rev 4160)
@@ -0,0 +1,72 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2007. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+package com.bigdata.bop.rdf.joinGraph;
+
+
+import junit.framework.Test;
+import junit.framework.TestCase;
+import junit.framework.TestSuite;
+
+/**
+ * Aggregates test suites into increasing dependency order.
+ *
+ * @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @version $Id$
+ */
+public class TestAll extends TestCase {
+
+ /**
+ *
+ */
+ public TestAll() {
+
+ }
+
+ /**
+ * @param arg0
+ */
+ public TestAll(String arg0) {
+
+ super(arg0);
+
+ }
+
+ /**
+ * Returns a test that will run each of the implementation specific test
+ * suites in turn.
+ */
+ public static Test suite()
+ {
+
+ final TestSuite suite = new TestSuite("Runtime query optimizer");
+
+ suite.addTestSuite(TestJoinGraphOnLubm.class);
+ suite.addTestSuite(TestJoinGraphOnBarData.class);
+ suite.addTestSuite(TestJoinGraphOnBSBMData.class);
+
+ return suite;
+
+ }
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestAll.java
___________________________________________________________________
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphOnBSBMData.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphOnBSBMData.java 2011-01-21 16:46:41 UTC (rev 4159)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphOnBSBMData.java 2011-01-21 21:52:18 UTC (rev 4160)
@@ -176,7 +176,7 @@
return new Journal(properties);
}
-
+
/**
* BSBM Q5
*
@@ -202,6 +202,18 @@
* LIMIT 5
* </pre>
*
+ * Note: There are two predicates which bind variables (origProperty1 and
+ * origProperty2) that are not used by the other predicates and therefore do
+ * not share any variables which would form "edges" that define joins. In
+ * general, a join without shared variables means the cross product of the
+ * sources will be materialized and such joins should be run last.
+ * <p>
+ * However, in this case there are SPARQL FILTERs which (a) use those
+ * variables (origProperty1 and origProperty2); and (b) can constrain the
+ * query. This means that running the predicates without shared variables
+ * and applying the constraints before the tail of the plan can in fact lead
+ * to a more efficient join path.
+ *
* @throws Exception
*/
public void test_bsbm_q5() throws Exception {
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphOnBarData.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphOnBarData.java 2011-01-21 16:46:41 UTC (rev 4159)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphOnBarData.java 2011-01-21 21:52:18 UTC (rev 4160)
@@ -211,7 +211,7 @@
*
* @throws Exception
*/
- public void test_query() throws Exception {
+ public void test_barData_query() throws Exception {
final String namespace = getNamespace();
Modified: branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphOnLubm.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphOnLubm.java 2011-01-21 16:46:41 UTC (rev 4159)
+++ branches/QUADS_QUERY_BRANCH/bigdata-rdf/src/test/com/bigdata/bop/rdf/joinGraph/TestJoinGraphOnLubm.java 2011-01-21 21:52:18 UTC (rev 4160)
@@ -310,7 +310,7 @@
*
* @throws Exception
*/
- public void test_query2() throws Exception {
+ public void test_LUBM_Q2() throws Exception {
final String namespace = getNamespace();
@@ -449,7 +449,7 @@
* </pre>
* @throws Exception
*/
- public void test_query8() throws Exception {
+ public void test_LUBM_Q8() throws Exception {
final String namespace = getNamespace();
@@ -580,7 +580,7 @@
*
* @throws Exception
*/
- public void test_query9() throws Exception {
+ public void test_LUBM_Q9() throws Exception {
final String name...
[truncated message content] |
|
From: <tho...@us...> - 2011-01-21 16:46:48
|
Revision: 4159
http://bigdata.svn.sourceforge.net/bigdata/?rev=4159&view=rev
Author: thompsonbry
Date: 2011-01-21 16:46:41 +0000 (Fri, 21 Jan 2011)
Log Message:
-----------
Modified UNION/STEPS and AbstractSubqueryOp to use an annotation for the subqueries rather than the arguments of the UNION/STEPS. This makes it possible to use these operators at positions other than the start of the pipeline.
Bug fix to PipelineJoin where it was applying constraints for optional joins to the original solution even when the constraints was a null reference.
Fix to the semantics of an optional join test in TestQueryEngine to reflect the fact that the constraint is applied on all paths through a join.
Turning on the new eval strategy by default in the SAIL for CI builds.
13 sail failures in this commit. some appear to be new while some have disappeared.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/AbstractSubqueryOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/Steps.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/Union.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestUnion.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSail.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/Rule2BOpUtility.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/AbstractSubqueryOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/AbstractSubqueryOp.java 2011-01-21 16:41:12 UTC (rev 4158)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/AbstractSubqueryOp.java 2011-01-21 16:46:41 UTC (rev 4159)
@@ -38,7 +38,6 @@
import com.bigdata.bop.BOp;
import com.bigdata.bop.BOpContext;
-import com.bigdata.bop.BOpEvaluationContext;
import com.bigdata.bop.BOpUtility;
import com.bigdata.bop.IBindingSet;
import com.bigdata.bop.PipelineOp;
@@ -74,7 +73,7 @@
*
* <pre>
* SLICE[1](
- * UNION[2]([a{sinkRef=1},b{sinkRef=1},c{sinkRef=1}],{})
+ * UNION[2]([...],{subqueries=[a{sinkRef=1},b{sinkRef=1},c{sinkRef=1}]})
* )
* </pre>
*
@@ -96,6 +95,12 @@
public interface Annotations extends PipelineOp.Annotations {
/**
+ * The ordered {@link BOp}[] of subqueries to be evaluated for each
+ * binding set presented (required).
+ */
+ String SUBQUERIES = SubqueryOp.class.getName() + ".subqueries";
+
+ /**
* The maximum parallelism with which the subqueries will be evaluated
* (default is unlimited).
*/
@@ -132,13 +137,19 @@
super(args, annotations);
- if (!getEvaluationContext().equals(BOpEvaluationContext.CONTROLLER))
- throw new IllegalArgumentException(Annotations.EVALUATION_CONTEXT
- + "=" + getEvaluationContext());
+// if (!getEvaluationContext().equals(BOpEvaluationContext.CONTROLLER))
+// throw new IllegalArgumentException(Annotations.EVALUATION_CONTEXT
+// + "=" + getEvaluationContext());
- if (!getProperty(Annotations.CONTROLLER, Annotations.DEFAULT_CONTROLLER))
- throw new IllegalArgumentException(Annotations.CONTROLLER);
-
+// if (!getProperty(Annotations.CONTROLLER, Annotations.DEFAULT_CONTROLLER))
+// throw new IllegalArgumentException(Annotations.CONTROLLER);
+
+ // verify required annotation.
+ final BOp[] subqueries = (BOp[]) getRequiredProperty(Annotations.SUBQUERIES);
+
+ if (subqueries.length == 0)
+ throw new IllegalArgumentException(Annotations.SUBQUERIES);
+
// // The id of this operator (if any).
// final Integer thisId = (Integer)getProperty(Annotations.BOP_ID);
//
@@ -170,6 +181,7 @@
private static class ControllerTask implements Callable<Void> {
private final AbstractSubqueryOp controllerOp;
+ private final BOp[] subqueries;
private final BOpContext<IBindingSet> context;
private final int nparallel;
private final Executor executor;
@@ -187,6 +199,9 @@
this.context = context;
+ this.subqueries = (BOp[]) controllerOp
+ .getRequiredProperty(Annotations.SUBQUERIES);
+
this.nparallel = controllerOp.getProperty(Annotations.MAX_PARALLEL,
Annotations.DEFAULT_MAX_PARALLEL);
@@ -244,8 +259,8 @@
try {
- final CountDownLatch latch = new CountDownLatch(controllerOp
- .arity());
+ final CountDownLatch latch = new CountDownLatch(
+ subqueries.length);
/*
* Create FutureTasks for each subquery. The futures are not
@@ -253,7 +268,7 @@
* deferring the evaluation until call() we gain the ability to
* cancel all subqueries if any subquery fails.
*/
- for (BOp op : controllerOp.args()) {
+ for (BOp op : subqueries) {
/*
* Task runs subquery and cancels all subqueries in [tasks]
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/Steps.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/Steps.java 2011-01-21 16:41:12 UTC (rev 4158)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/Steps.java 2011-01-21 16:46:41 UTC (rev 4159)
@@ -37,7 +37,7 @@
* STEPS(ops)
*
* <pre>
- * STEPS([a,b,c],{})
+ * STEPS([],{subqueries=[a,b,c]})
* </pre>
*
* Will run the subqueries <i>a</i>, <i>b</i>, and <i>c</i> in sequence. Each
@@ -62,12 +62,11 @@
public Steps(Steps op) {
super(op);
}
-
+
/**
* Shallow copy constructor.
*
* @param args
- * Two or more operators whose union is desired.
* @param annotations
*/
public Steps(final BOp[] args,
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/Union.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/Union.java 2011-01-21 16:41:12 UTC (rev 4158)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/Union.java 2011-01-21 16:46:41 UTC (rev 4159)
@@ -34,10 +34,10 @@
import com.bigdata.bop.PipelineOp;
/**
- * UNION(ops)[maxParallel(default all)]
+ * UNION()[maxParallel(default all); subqueries=ops]
*
* <pre>
- * UNION([a,b,c],{})
+ * UNION([],{subqueries=[a,b,c]})
* </pre>
*
* Will run the subqueries <i>a</i>, <i>b</i>, and <i>c</i> in parallel for each
@@ -68,15 +68,14 @@
* Shallow copy constructor.
*
* @param args
- * Two or more operators whose union is desired.
* @param annotations
*/
public Union(final BOp[] args, final Map<String, Object> annotations) {
super(args, annotations);
- if (args.length < 2)
- throw new IllegalArgumentException();
+// if (args.length < 2)
+// throw new IllegalArgumentException();
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestUnion.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestUnion.java 2011-01-21 16:41:12 UTC (rev 4158)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestUnion.java 2011-01-21 16:46:41 UTC (rev 4159)
@@ -45,6 +45,7 @@
import com.bigdata.bop.bindingSet.ArrayBindingSet;
import com.bigdata.bop.bindingSet.EmptyBindingSet;
import com.bigdata.bop.bindingSet.HashBindingSet;
+import com.bigdata.bop.bindingSet.ListBindingSet;
import com.bigdata.bop.bset.StartOp;
import com.bigdata.bop.engine.IRunningQuery;
import com.bigdata.bop.engine.QueryEngine;
@@ -119,7 +120,7 @@
// data to insert (in key order for convenience).
final E[] a = {//
- new E("John", "Mary"),// [0]
+ new E("John", "Mary"),// [0]
new E("Leon", "Paul"),// [1]
new E("Mary", "Paul"),// [2]
new E("Paul", "Leon"),// [3]
@@ -176,12 +177,14 @@
BOpEvaluationContext.CONTROLLER),//
}));
- final BOp unionOp = new Union(new BOp[] { startOp1, startOp2 }, NV
+ final BOp unionOp = new Union(new BOp[0], NV
.asMap(new NV[] {//
new NV(Union.Annotations.BOP_ID, unionId),//
- new NV(Union.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.CONTROLLER),//
- new NV(Union.Annotations.CONTROLLER, true),//
+ new NV(Union.Annotations.SUBQUERIES, new BOp[] {
+ startOp1, startOp2 }),//
+// new NV(Union.Annotations.EVALUATION_CONTEXT,
+// BOpEvaluationContext.CONTROLLER),//
+// new NV(Union.Annotations.CONTROLLER, true),//
}));
final BOp query = unionOp;
@@ -203,6 +206,60 @@
}
+ public void test_union_consumesSource() throws Exception {
+
+ final int startId1 = 1;
+ final int startId2 = 2;
+ final int unionId = 3;
+
+ final BOp startOp1 = new StartOp(new BOp[] {}, NV.asMap(new NV[] {//
+ new NV(StartOp.Annotations.BOP_ID, startId1),//
+ new NV(StartOp.Annotations.EVALUATION_CONTEXT,
+ BOpEvaluationContext.CONTROLLER),//
+ }));
+
+ final BOp startOp2 = new StartOp(new BOp[] {}, NV.asMap(new NV[] {//
+ new NV(StartOp.Annotations.BOP_ID, startId2),//
+ new NV(StartOp.Annotations.EVALUATION_CONTEXT,
+ BOpEvaluationContext.CONTROLLER),//
+ }));
+
+ final BOp unionOp = new Union(new BOp[]{}, NV
+ .asMap(new NV[] {//
+ new NV(Union.Annotations.BOP_ID, unionId),//
+ new NV(Union.Annotations.SUBQUERIES, new BOp[] {
+ startOp1, startOp2 }) //
+// new NV(Union.Annotations.EVALUATION_CONTEXT,
+// BOpEvaluationContext.CONTROLLER),//
+// new NV(Union.Annotations.CONTROLLER, true),//
+ }));
+
+ final BOp query = unionOp;
+
+ /*
+ * Create an initial non-empty binding set.
+ */
+ final IBindingSet bset = new ListBindingSet();
+ bset.set(Var.var("x"), new Constant<String>("John"));
+ bset.set(Var.var("y"), new Constant<String>("Mary"));
+
+ // the expected solutions.
+ final IBindingSet[] expected = new IBindingSet[] {//
+ bset, // one copy from the left side of the union.
+ bset, // one copy from the right side of the union.
+ };
+
+ final IRunningQuery runningQuery = queryEngine.eval(query, bset);
+
+ // verify solutions.
+ TestQueryEngine.assertSameSolutionsAnyOrder(expected,
+ new Dechunkerator<IBindingSet>(runningQuery.iterator()));
+
+ // Wait until the query is done.
+ runningQuery.get();
+
+ }
+
/**
* Verifies that the UNION of two operators is computed. The operators do
* not route around the UNION, so their solutions are copied to the UNION
@@ -249,12 +306,14 @@
new NV(StartOp.Annotations.BINDING_SETS,bindingSets2)
}));
- final BOp unionOp = new Union(new BOp[] { startOp1, startOp2 }, NV
+ final BOp unionOp = new Union(new BOp[] {}, NV
.asMap(new NV[] {//
new NV(Union.Annotations.BOP_ID, unionId),//
- new NV(Union.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.CONTROLLER),//
- new NV(Union.Annotations.CONTROLLER, true),//
+ new NV(Union.Annotations.SUBQUERIES, new BOp[] {
+ startOp1, startOp2 }) //
+// new NV(Union.Annotations.EVALUATION_CONTEXT,
+// BOpEvaluationContext.CONTROLLER),//
+// new NV(Union.Annotations.CONTROLLER, true),//
}));
final BOp sliceOp = new SliceOp(new BOp[]{unionOp},NV.asMap(
@@ -336,12 +395,14 @@
new NV(StartOp.Annotations.BINDING_SETS,bindingSets2)
}));
- final BOp unionOp = new Union(new BOp[] { startOp1, startOp2 }, NV
+ final BOp unionOp = new Union(new BOp[] {}, NV
.asMap(new NV[] {//
new NV(Union.Annotations.BOP_ID, unionId),//
- new NV(Union.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.CONTROLLER),//
- new NV(Union.Annotations.CONTROLLER, true),//
+ new NV(Union.Annotations.SUBQUERIES, new BOp[] {
+ startOp1, startOp2 }) //
+// new NV(Union.Annotations.EVALUATION_CONTEXT,
+// BOpEvaluationContext.CONTROLLER),//
+// new NV(Union.Annotations.CONTROLLER, true),//
}));
final BOp sliceOp = new SliceOp(new BOp[]{unionOp},NV.asMap(
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java 2011-01-21 16:41:12 UTC (rev 4158)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/engine/TestQueryEngine.java 2011-01-21 16:46:41 UTC (rev 4159)
@@ -1643,17 +1643,24 @@
new Constant<String>("Leon"),
new Constant<String>("Paul") }//
),
- // plus anything we read from the first access path which did not join.
- new ArrayBindingSet(//
- new IVariable[] { Var.var("x"), Var.var("y") },//
- new IConstant[] { new Constant<String>("John"),
- new Constant<String>("Mary") }//
- ),
- new ArrayBindingSet(//
- new IVariable[] { Var.var("x"), Var.var("y") },//
- new IConstant[] { new Constant<String>("Mary"),
- new Constant<String>("Paul") }//
- )
+ /*
+ * No. The CONSTRAINT on the 2nd join [x == y] filters all
+ * solutions. For solutions where the optional join fails, [y] is
+ * not bound. Since [y] is part of the constraint on that join we DO
+ * NOT observe those solutions which only join on the first access
+ * path.
+ */
+// // plus anything we read from the first access path which did not join.
+// new ArrayBindingSet(//
+// new IVariable[] { Var.var("x"), Var.var("y") },//
+// new IConstant[] { new Constant<String>("John"),
+// new Constant<String>("Mary") }//
+// ),
+// new ArrayBindingSet(//
+// new IVariable[] { Var.var("x"), Var.var("y") },//
+// new IConstant[] { new Constant<String>("Mary"),
+// new Constant<String>("Paul") }//
+// )
};
assertSameSolutionsAnyOrder(expected,
@@ -1714,7 +1721,7 @@
// verify query solution stats details.
// assertEquals(1L, stats.chunksIn.get());
assertEquals(4L, stats.unitsIn.get());
- assertEquals(4L, stats.unitsOut.get());
+ assertEquals(2L, stats.unitsOut.get());
// assertEquals(1L, stats.chunksOut.get());
}
@@ -1727,8 +1734,8 @@
// verify query solution stats details.
// assertEquals(2L, stats.chunksIn.get());
- assertEquals(4L, stats.unitsIn.get());
- assertEquals(4L, stats.unitsOut.get());
+ assertEquals(2L, stats.unitsIn.get());
+ assertEquals(2L, stats.unitsOut.get());
// assertEquals(1L, stats.chunksOut.get());
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSail.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSail.java 2011-01-21 16:41:12 UTC (rev 4158)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/BigdataSail.java 2011-01-21 16:46:41 UTC (rev 4159)
@@ -382,7 +382,7 @@
public static final String NEW_EVAL_STRATEGY = BigdataSail.class.getPackage()
.getName()+ ".newEvalStrategy";
- public static final String DEFAULT_NEW_EVAL_STRATEGY = "false";
+ public static final String DEFAULT_NEW_EVAL_STRATEGY = "true";
/**
* Option as to whether or not to allow Sesame evaluation of queries
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/Rule2BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/Rule2BOpUtility.java 2011-01-21 16:41:12 UTC (rev 4158)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/Rule2BOpUtility.java 2011-01-21 16:46:41 UTC (rev 4159)
@@ -65,6 +65,7 @@
import com.bigdata.bop.bindingSet.HashBindingSet;
import com.bigdata.bop.bset.ConditionalRoutingOp;
import com.bigdata.bop.bset.StartOp;
+import com.bigdata.bop.controller.AbstractSubqueryOp;
import com.bigdata.bop.controller.Steps;
import com.bigdata.bop.controller.Union;
import com.bigdata.bop.cost.ScanCostReport;
@@ -1304,13 +1305,11 @@
// The bopId for the UNION or STEP.
final int thisId = idFactory.incrementAndGet();
- final int arity = program.stepCount();
-
- final IStep[] steps = program.toArray();
+ final IStep[] steps = program.toArray();
- final BOp[] args = new BOp[arity];
+ final BOp[] subqueries = new BOp[steps.length];
- for (int i = 0; i < arity; i++) {
+ for (int i = 0; i < steps.length; i++) {
// convert the child IStep
final BOpBase tmp = convert(steps[i], idFactory, db, queryEngine,
@@ -1324,28 +1323,31 @@
*/
// tmp = tmp.setProperty(PipelineOp.Annotations.SINK_REF, thisId);
- args[i] = tmp;
+ subqueries[i] = tmp;
}
final LinkedList<NV> anns = new LinkedList<NV>();
- anns.add(new NV(Union.Annotations.BOP_ID, thisId));
+ anns.add(new NV(BOp.Annotations.BOP_ID, thisId));
+
+ // the subqueries.
+ anns.add(new NV(AbstractSubqueryOp.Annotations.SUBQUERIES, subqueries));
+
+// anns.add(new NV(Union.Annotations.EVALUATION_CONTEXT,
+// BOpEvaluationContext.CONTROLLER));
+//
+// anns.add(new NV(Union.Annotations.CONTROLLER, true));
- anns.add(new NV(Union.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.CONTROLLER));
-
- anns.add(new NV(Union.Annotations.CONTROLLER, true));
-
if (!isParallel)
anns.add(new NV(Union.Annotations.MAX_PARALLEL, 1));
final PipelineOp thisOp;
if (isParallel) {
- thisOp = new Union(args, NV
+ thisOp = new Union(new BOp[]{}, NV
.asMap(anns.toArray(new NV[anns.size()])));
} else {
- thisOp = new Steps(args, NV
+ thisOp = new Steps(new BOp[]{}, NV
.asMap(anns.toArray(new NV[anns.size()])));
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java 2011-01-21 16:41:12 UTC (rev 4158)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java 2011-01-21 16:46:41 UTC (rev 4159)
@@ -273,12 +273,13 @@
}
final LinkedList<NV> anns = new LinkedList<NV>();
- anns.add(new NV(Union.Annotations.BOP_ID, thisId));
- anns.add(new NV(Union.Annotations.EVALUATION_CONTEXT,
- BOpEvaluationContext.CONTROLLER));
- anns.add(new NV(Union.Annotations.CONTROLLER, true));
+ anns.add(new NV(BOp.Annotations.BOP_ID, thisId));
+ anns.add(new NV(Union.Annotations.SUBQUERIES,args));
+// anns.add(new NV(Union.Annotations.EVALUATION_CONTEXT,
+// BOpEvaluationContext.CONTROLLER));
+// anns.add(new NV(Union.Annotations.CONTROLLER, true));
- final Union thisOp = new Union(args, NV
+ final Union thisOp = new Union(new BOp[]{}, NV
.asMap(anns.toArray(new NV[anns.size()])));
return thisOp;
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mrp...@us...> - 2011-01-21 16:41:24
|
Revision: 4158
http://bigdata.svn.sourceforge.net/bigdata/?rev=4158&view=rev
Author: mrpersonick
Date: 2011-01-21 16:41:12 +0000 (Fri, 21 Jan 2011)
Log Message:
-----------
down to 11 failures
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataSparqlTest.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataSparqlTest.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataSparqlTest.java 2011-01-21 16:35:02 UTC (rev 4157)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/tck/BigdataSparqlTest.java 2011-01-21 16:41:12 UTC (rev 4158)
@@ -181,7 +181,7 @@
static final Collection<String> testURIs = Arrays.asList(new String[] {
/*
- * working through the new query engine failures: 0 errors, 19 failures
+ * working through the new query engine failures: 0 errors, 11 failures
*/
/*
@@ -213,16 +213,6 @@
*/
/*
- * These failures have to do with nested UNIONs - we don't seem to
- * be handling them correctly at all.
- */
-// "http://www.w3.org/2001/sw/DataAccess/tests/data-r2/algebra/manifest#join-combo-1",
-// "http://www.w3.org/2001/sw/DataAccess/tests/data-r2/algebra/manifest#join-combo-2",
-// "http://www.w3.org/2001/sw/DataAccess/tests/data-r2/optional/manifest#dawg-optional-complex-1",
-// "http://www.w3.org/2001/sw/DataAccess/tests/data-r2/optional/manifest#dawg-optional-complex-2",
-// "http://www.w3.org/2001/sw/DataAccess/tests/data-r2/optional/manifest#dawg-optional-complex-4",
-
- /*
* This one is truly bizarre - involving a non-optional subquuery
* plus an optional subquery. Don't even know where to start on this
* guy.
@@ -236,18 +226,29 @@
*/
// "http://www.w3.org/2001/sw/DataAccess/tests/data-r2/algebra/manifest#filter-nested-2",
- /*
- * These demonstrate the problem of where to put non-optional
- * filters that need to be evaluated after optional tails and
- * optional join groups.
- *
- * NOTE: These are fixed. Changed constraints into conditional
- * routing ops instead of annotations on predicates or joins.
- */
-// "http://www.w3.org/2001/sw/DataAccess/tests/data-r2/bound/manifest#dawg-bound-query-001",
-// "http://www.w3.org/2001/sw/DataAccess/tests/data-r2/optional-filter/manifest#dawg-optional-filter-002",
-// "http://www.w3.org/2001/sw/DataAccess/tests/data-r2/optional-filter/manifest#dawg-optional-filter-003",
+// /*
+// * These demonstrate the problem of where to put non-optional
+// * filters that need to be evaluated after optional tails and
+// * optional join groups.
+// *
+// * NOTE: These are fixed.
+// */
+//// "http://www.w3.org/2001/sw/DataAccess/tests/data-r2/bound/manifest#dawg-bound-query-001",
+//// "http://www.w3.org/2001/sw/DataAccess/tests/data-r2/optional-filter/manifest#dawg-optional-filter-002",
+//// "http://www.w3.org/2001/sw/DataAccess/tests/data-r2/optional-filter/manifest#dawg-optional-filter-003",
+// /*
+// * These failures have to do with nested UNIONs - we don't seem to
+// * be handling them correctly at all.
+// *
+// * NOTE: These are fixed.
+// */
+//// "http://www.w3.org/2001/sw/DataAccess/tests/data-r2/algebra/manifest#join-combo-1",
+//// "http://www.w3.org/2001/sw/DataAccess/tests/data-r2/algebra/manifest#join-combo-2",
+//// "http://www.w3.org/2001/sw/DataAccess/tests/data-r2/optional/manifest#dawg-optional-complex-1",
+//// "http://www.w3.org/2001/sw/DataAccess/tests/data-r2/optional/manifest#dawg-optional-complex-2",
+//// "http://www.w3.org/2001/sw/DataAccess/tests/data-r2/optional/manifest#dawg-optional-complex-4",
+
});
/**
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mrp...@us...> - 2011-01-21 16:35:08
|
Revision: 4157
http://bigdata.svn.sourceforge.net/bigdata/?rev=4157&view=rev
Author: mrpersonick
Date: 2011-01-21 16:35:02 +0000 (Fri, 21 Jan 2011)
Log Message:
-----------
working through tck failures
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/Rule2BOpUtility.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java 2011-01-21 14:28:17 UTC (rev 4156)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java 2011-01-21 16:35:02 UTC (rev 4157)
@@ -156,17 +156,17 @@
}
- public String toString() {
-
- final StringBuilder sb = new StringBuilder(super.toString());
- sb.append("\n{\n");
- final PipelineOp subquery = (PipelineOp)
- getRequiredProperty(Annotations.SUBQUERY);
- sb.append(BOpUtility.toString(subquery));
- sb.append("\n}");
- return sb.toString();
-
- }
+// public String toString() {
+//
+// final StringBuilder sb = new StringBuilder(super.toString());
+// sb.append("\n{\n");
+// final PipelineOp subquery = (PipelineOp)
+// getRequiredProperty(Annotations.SUBQUERY);
+// sb.append(BOpUtility.toString(subquery));
+// sb.append("\n}");
+// return sb.toString();
+//
+// }
/**
* Evaluates the arguments of the operator as subqueries. The arguments are
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java 2011-01-21 14:28:17 UTC (rev 4156)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/join/PipelineJoin.java 2011-01-21 16:35:02 UTC (rev 4157)
@@ -1684,9 +1684,11 @@
final IBindingSet bs = bindingSets[bindex];
- if(!BOpUtility.isConsistent(constraints, bs)) {
- // Failed by the constraint on the join.
- continue;
+ if (constraints != null) {
+ if(!BOpUtility.isConsistent(constraints, bs)) {
+ // Failed by the constraint on the join.
+ continue;
+ }
}
if (log.isTraceEnabled())
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/Rule2BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/Rule2BOpUtility.java 2011-01-21 14:28:17 UTC (rev 4156)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/Rule2BOpUtility.java 2011-01-21 16:35:02 UTC (rev 4157)
@@ -620,12 +620,12 @@
// }
// just add all the constraints to the very last tail for now
-// if (i == (order.length-1) && rule.getConstraintCount() > 0) {
-// final Iterator<IConstraint> it = rule.getConstraints();
-// while (it.hasNext()) {
-// constraints.add(it.next());
-// }
-// }
+ if (i == (order.length-1) && rule.getConstraintCount() > 0) {
+ final Iterator<IConstraint> it = rule.getConstraints();
+ while (it.hasNext()) {
+ constraints.add(it.next());
+ }
+ }
// annotations for this join.
final List<NV> anns = new LinkedList<NV>();
@@ -730,23 +730,23 @@
}
- if (rule.getConstraintCount() > 0) {
- final Iterator<IConstraint> it = rule.getConstraints();
- while (it.hasNext()) {
- final IConstraint c = it.next();
- final int condId = idFactory.incrementAndGet();
- final PipelineOp condOp = applyQueryHints(
- new ConditionalRoutingOp(new BOp[]{left},
- NV.asMap(new NV[]{//
- new NV(BOp.Annotations.BOP_ID,condId),
- new NV(ConditionalRoutingOp.Annotations.CONDITION, c),
- })), queryHints);
- left = condOp;
- if (log.isDebugEnabled()) {
- log.debug("adding conditional routing op: " + condOp);
- }
- }
- }
+// if (rule.getConstraintCount() > 0) {
+// final Iterator<IConstraint> it = rule.getConstraints();
+// while (it.hasNext()) {
+// final IConstraint c = it.next();
+// final int condId = idFactory.incrementAndGet();
+// final PipelineOp condOp = applyQueryHints(
+// new ConditionalRoutingOp(new BOp[]{left},
+// NV.asMap(new NV[]{//
+// new NV(BOp.Annotations.BOP_ID,condId),
+// new NV(ConditionalRoutingOp.Annotations.CONDITION, c),
+// })), queryHints);
+// left = condOp;
+// if (log.isDebugEnabled()) {
+// log.debug("adding conditional routing op: " + condOp);
+// }
+// }
+// }
if (log.isInfoEnabled()) {
// just for now while i'm debugging
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java 2011-01-21 14:28:17 UTC (rev 4156)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/java/com/bigdata/rdf/sail/sop/SOp2BOpUtility.java 2011-01-21 16:35:02 UTC (rev 4157)
@@ -27,6 +27,7 @@
package com.bigdata.rdf.sail.sop;
+import java.util.Arrays;
import java.util.Collection;
import java.util.HashSet;
import java.util.Iterator;
@@ -47,6 +48,7 @@
import com.bigdata.bop.NV;
import com.bigdata.bop.PipelineOp;
import com.bigdata.bop.ap.Predicate;
+import com.bigdata.bop.bset.ConditionalRoutingOp;
import com.bigdata.bop.controller.SubqueryOp;
import com.bigdata.bop.controller.Union;
import com.bigdata.bop.engine.QueryEngine;
@@ -161,13 +163,22 @@
* join groups, and thus should be translated into ConditionalRoutingOps
* for maximum efficiency.
*/
- final Collection<IConstraint> conditionals =
+ final Collection<IConstraint> preConditionals =
new LinkedList<IConstraint>();
- final IRule rule = rule(join, conditionals);
+ /*
+ * These are constraints that use variables bound by subqueries, and
+ * thus cannot be attached to the predicates in this group. They are
+ * handled by ConditionalRoutingOps at the end of the group, after
+ * the subqueries have run.
+ */
+ final Collection<IConstraint> postConditionals =
+ new LinkedList<IConstraint>();
+ final IRule rule = rule(join, preConditionals, postConditionals);
+
PipelineOp left = Rule2BOpUtility.convert(
- rule, conditionals, idFactory, db, queryEngine, queryHints);
+ rule, preConditionals, idFactory, db, queryEngine, queryHints);
/*
* Start with left=<this join group> and add a SubqueryOp for each
@@ -190,9 +201,26 @@
new NV(SubqueryOp.Annotations.SUBQUERY, subquery),//
new NV(SubqueryOp.Annotations.OPTIONAL,optional)//
);
+ if (log.isInfoEnabled()) {
+ log.info("adding a subquery: " + subqueryId + "\n" + left);
+ }
}
}
+ for (IConstraint c : postConditionals) {
+ final int condId = idFactory.incrementAndGet();
+ final PipelineOp condOp =
+ new ConditionalRoutingOp(new BOp[]{left},
+ NV.asMap(new NV[]{//
+ new NV(BOp.Annotations.BOP_ID,condId),
+ new NV(ConditionalRoutingOp.Annotations.CONDITION, c),
+ }));
+ left = condOp;
+ if (log.isDebugEnabled()) {
+ log.debug("adding post-conditional routing op: " + condOp);
+ }
+ }
+
if (!left.getEvaluationContext()
.equals(BOpEvaluationContext.CONTROLLER)
&& !(left instanceof SubqueryOp)) {
@@ -258,7 +286,8 @@
}
protected static IRule rule(final SOpGroup group,
- final Collection<IConstraint> conditionals) {
+ final Collection<IConstraint> preConditionals,
+ final Collection<IConstraint> postConditionals) {
final Collection<IPredicate> preds = new LinkedList<IPredicate>();
final Collection<IConstraint> constraints = new LinkedList<IConstraint>();
@@ -266,36 +295,20 @@
/*
* Gather up all the variables used by non-optional parent join groups
*/
- final Set<IVariable<?>> variables = new HashSet<IVariable<?>>();
+ final Set<IVariable<?>> nonOptParentVars = new HashSet<IVariable<?>>();
SOpGroup parent = group;
while ((parent = parent.getParent()) != null) {
if (isNonOptionalJoinGroup(parent))
- collectPredicateVariables(variables, parent);
+ collectPredicateVariables(nonOptParentVars, parent);
}
+ /*
+ * Gather up all the predicates in this group.
+ */
for (SOp sop : group) {
final BOp bop = sop.getBOp();
if (bop instanceof IPredicate) {
preds.add((IPredicate) bop);
- } else if (bop instanceof IConstraint) {
- final IConstraint c = (IConstraint) bop;
- /*
- * This constraint is a conditional if all of its variables
- * appear in non-optional parent join groups
- */
- final Iterator<IVariable<?>> vars =
- BOpUtility.getSpannedVariables(c);
- boolean conditional = true;
- while (vars.hasNext()) {
- final IVariable<?> v = vars.next();
- conditional &= variables.contains(v);
- }
- if (conditional)
- conditionals.add(c);
- else
- constraints.add(c);
- } else {
- throw new IllegalArgumentException("illegal operator: " + sop);
}
}
@@ -317,8 +330,95 @@
}
}
+ /*
+ * Gather up all the variables used by predicates in this group
+ */
+ final Set<IVariable<?>> groupVars = new HashSet<IVariable<?>>();
+ for (IPredicate bop : preds) {
+ for (BOp arg : bop.args()) {
+ if (arg instanceof IVariable<?>) {
+ final IVariable<?> v = (IVariable<?>) arg;
+ groupVars.add(v);
+ }
+ }
+ }
+
+ /*
+ * Gather up the constraints, segregating into three categories:
+ * -constraints: all variables used by predicates in this group
+ * -pre-conditionals: all variables already bound by parent group(s)
+ * -post-conditionals: some or all variables bound in subqueries
+ */
+ for (SOp sop : group) {
+ final BOp bop = sop.getBOp();
+ if (bop instanceof IConstraint) {
+ final IConstraint c = (IConstraint) bop;
+
+ { // find the pre-conditionals
+
+ final Iterator<IVariable<?>> constraintVars =
+ BOpUtility.getSpannedVariables(c);
+
+ /*
+ * This constraint is a pre-conditional if all of its variables
+ * appear in non-optional parent join groups
+ */
+ boolean preConditional = true;
+ while (constraintVars.hasNext()) {
+ final IVariable<?> v = constraintVars.next();
+ preConditional &= nonOptParentVars.contains(v);
+ }
+ if (preConditional) {
+ preConditionals.add(c);
+ continue;
+ }
+
+ }
+
+ { // find the post-conditionals
+
+ final Iterator<IVariable<?>> constraintVars =
+ BOpUtility.getSpannedVariables(c);
+
+ /*
+ * This constraint is a post-conditional if not all of its
+ * variables appear in this join group or non-optional parent
+ * groups (bound by subqueries)
+ */
+ boolean postConditional = false;
+ while (constraintVars.hasNext()) {
+ final IVariable<?> v = constraintVars.next();
+ if (!nonOptParentVars.contains(v) &&
+ !groupVars.contains(v)) {
+ postConditional = true;
+ break;
+ }
+ }
+ if (postConditional) {
+ postConditionals.add(c);
+ continue;
+ }
+
+ }
+
+ /*
+ * Neither pre nor post conditional, but a constraint on the
+ * predicates in this group. done this roundabout way for the
+ * benefit of the RTO
+ */
+ constraints.add(c);
+ }
+ }
+
final IVariable<?>[] required = group.getTree().getRequiredVars();
+ if (log.isInfoEnabled()) {
+ log.info("preds: " + Arrays.toString(preds.toArray()));
+ log.info("constraints: " + Arrays.toString(constraints.toArray()));
+ log.info("preConds: " + Arrays.toString(preConditionals.toArray()));
+ log.info("postConds: " + Arrays.toString(postConditionals.toArray()));
+ }
+
final IRule rule = new Rule(
"dummy rule",
null, // head
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-21 14:28:24
|
Revision: 4156
http://bigdata.svn.sourceforge.net/bigdata/?rev=4156&view=rev
Author: thompsonbry
Date: 2011-01-21 14:28:17 +0000 (Fri, 21 Jan 2011)
Log Message:
-----------
Modified Union (and its base class AbstractSubqueryOp) to consume the binding sets from the source and added a unit test for this. However, UNION still can not be used in any position other than the 1st operator in a pipeline because its operands are the subqueries. Right now, in order to use UNION in a non-primary position you have to use a SubqueryOp to wrap the Union. And Union itself issues subqueries, so this is perhaps more overhead than we would like. In order to change this we would have to move the subqueries to be executed by the union from operands to annotations, which might be just fine.
Added some convenience methods to QueryEngine for evaluation against non-empty binding sets.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/AbstractSubqueryOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/Union.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/AbstractSubqueryOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/AbstractSubqueryOp.java 2011-01-21 13:17:01 UTC (rev 4155)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/AbstractSubqueryOp.java 2011-01-21 14:28:17 UTC (rev 4156)
@@ -32,6 +32,7 @@
import java.util.Map;
import java.util.concurrent.Callable;
import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executor;
import java.util.concurrent.FutureTask;
@@ -45,24 +46,26 @@
import com.bigdata.bop.engine.IRunningQuery;
import com.bigdata.bop.engine.QueryEngine;
import com.bigdata.relation.accesspath.IAsynchronousIterator;
+import com.bigdata.service.IBigdataFederation;
import com.bigdata.util.concurrent.LatchedExecutor;
/**
* Executes each of the operands as a subquery. The operands are evaluated in
* the order given and with the annotated parallelism. Each subquery is run as a
* separate query but is linked to the parent query in the operator is being
- * evaluated. The subqueries do not receive bindings from the parent and may be
+ * evaluated. The subqueries receive bindings from the pipeline and may be
* executed independently. By default, the subqueries are run with unlimited
- * parallelism.
+ * parallelism. Since the #of subqueries is generally small (2), this means that
+ * the subqueries run in parallel.
* <p>
* Note: This operator must execute on the query controller.
* <p>
* The {@link PipelineOp.Annotations#SINK_REF} of each child operand should be
* overridden to specify the parent of the this operator. If you fail to do
* this, then the intermediate results of the subqueries will be routed to this
- * operator, which DOES NOT pass them on. This may cause unnecessary network
- * traffic. It may also cause the query to block if the buffer capacity is
- * limited.
+ * operator. This may cause unnecessary network traffic when running against the
+ * {@link IBigdataFederation}. It may also cause the query to block if the
+ * buffer capacity is limited.
* <p>
* If you want to route intermediate results from other computations into
* subqueries, then consider a {@link Tee} pattern instead.
@@ -76,9 +79,9 @@
* </pre>
*
* Will run the subqueries <i>a</i>, <i>b</i>, and <i>c</i> in parallel. Each
- * subquery will be initialized with a single empty {@link IBindingSet}. The
- * output of those subqueries MUST be explicitly routed to the SLICE operator
- * using {@link PipelineOp.Annotations#SINK_REF} on each of the subqueries.
+ * subquery will be run once for each source {@link IBindingSet}. The output of
+ * those subqueries is explicitly routed to the SLICE operator using
+ * {@link PipelineOp.Annotations#SINK_REF} for efficiency in scale-out.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
* @version $Id$
@@ -168,13 +171,12 @@
private final AbstractSubqueryOp controllerOp;
private final BOpContext<IBindingSet> context;
- private final List<FutureTask<IRunningQuery>> tasks = new LinkedList<FutureTask<IRunningQuery>>();
- private final CountDownLatch latch;
private final int nparallel;
private final Executor executor;
-
- public ControllerTask(final AbstractSubqueryOp controllerOp, final BOpContext<IBindingSet> context) {
+ public ControllerTask(final AbstractSubqueryOp controllerOp,
+ final BOpContext<IBindingSet> context) {
+
if (controllerOp == null)
throw new IllegalArgumentException();
@@ -191,47 +193,90 @@
this.executor = new LatchedExecutor(context.getIndexManager()
.getExecutorService(), nparallel);
- this.latch = new CountDownLatch(controllerOp.arity());
+ }
- /*
- * Create FutureTasks for each subquery. The futures are not
- * submitted to the Executor yet. That happens in call(). By
- * deferring the evaluation until call() we gain the ability to
- * cancel all subqueries if any subquery fails.
- */
- for (BOp op : controllerOp.args()) {
+ /**
+ * Evaluate the subqueries with limited parallelism.
+ */
+ public Void call() throws Exception {
- /*
- * Task runs subquery and cancels all subqueries in [tasks] if
- * it fails.
- */
- tasks.add(new FutureTask<IRunningQuery>(new SubqueryTask(op,
- context)) {
- /*
- * Hook future to count down the latch when the task is
- * done.
- */
- public void run() {
- try {
- super.run();
- } finally {
- latch.countDown();
- }
+ final IAsynchronousIterator<IBindingSet[]> source = context
+ .getSource();
+
+ try {
+
+ while (source.hasNext()) {
+
+ final IBindingSet[] chunk = source.next();
+
+ for (IBindingSet bset : chunk) {
+
+ consumeBindingSet(bset);
+
}
- });
+
+ }
+ // Now that we know the subqueries ran Ok, flush the sink.
+ context.getSink().flush();
+
+ // Done.
+ return null;
+
+ } finally {
+
+ // Close the source.
+ source.close();
+
+ context.getSink().close();
+
+ if (context.getSink2() != null)
+ context.getSink2().close();
+
}
-
+
}
- /**
- * Evaluate the subqueries with limited parallelism.
- */
- public Void call() throws Exception {
+ private void consumeBindingSet(final IBindingSet bset)
+ throws InterruptedException, ExecutionException {
+ final List<FutureTask<IRunningQuery>> tasks = new LinkedList<FutureTask<IRunningQuery>>();
+
try {
+ final CountDownLatch latch = new CountDownLatch(controllerOp
+ .arity());
+
/*
+ * Create FutureTasks for each subquery. The futures are not
+ * submitted to the Executor yet. That happens in call(). By
+ * deferring the evaluation until call() we gain the ability to
+ * cancel all subqueries if any subquery fails.
+ */
+ for (BOp op : controllerOp.args()) {
+
+ /*
+ * Task runs subquery and cancels all subqueries in [tasks]
+ * if it fails.
+ */
+ tasks.add(new FutureTask<IRunningQuery>(new SubqueryTask(
+ op, context, bset)) {
+ /*
+ * Hook future to count down the latch when the task is
+ * done.
+ */
+ public void run() {
+ try {
+ super.run();
+ } finally {
+ latch.countDown();
+ }
+ }
+ });
+
+ }
+
+ /*
* Run subqueries with limited parallelism.
*/
for (FutureTask<IRunningQuery> ft : tasks) {
@@ -239,12 +284,6 @@
}
/*
- * Close the source. Controllers do not accept inputs from the
- * pipeline.
- */
- context.getSource().close();
-
- /*
* Wait for all subqueries to complete.
*/
latch.await();
@@ -255,25 +294,14 @@
for (FutureTask<IRunningQuery> ft : tasks)
ft.get();
- // Now that we know the subqueries ran Ok, flush the sink.
- context.getSink().flush();
-
- // Done.
- return null;
-
} finally {
// Cancel any tasks which are still running.
for (FutureTask<IRunningQuery> ft : tasks)
ft.cancel(true/* mayInterruptIfRunning */);
-
- context.getSink().close();
-
- if (context.getSink2() != null)
- context.getSink2().close();
}
-
+
}
/**
@@ -294,12 +322,20 @@
*/
private final BOp subQueryOp;
+ /**
+ * The input for this invocation of the subquery.
+ */
+ private final IBindingSet bset;
+
public SubqueryTask(final BOp subQuery,
- final BOpContext<IBindingSet> parentContext) {
+ final BOpContext<IBindingSet> parentContext,
+ final IBindingSet bset) {
this.subQueryOp = subQuery;
this.parentContext = parentContext;
+
+ this.bset = bset;
}
@@ -312,7 +348,7 @@
final QueryEngine queryEngine = parentContext.getRunningQuery()
.getQueryEngine();
- runningSubquery = queryEngine.eval(subQueryOp);
+ runningSubquery = queryEngine.eval(subQueryOp, bset);
// Iterator visiting the subquery solutions.
subquerySolutionItr = runningSubquery.iterator();
@@ -343,8 +379,10 @@
* Such exceptions are NOT propagated here and WILL NOT
* cause the parent query to terminate.
*/
- throw new RuntimeException(ControllerTask.this.context
- .getRunningQuery().halt(runningSubquery.getCause()));
+ throw new RuntimeException(ControllerTask.this.context
+ .getRunningQuery().halt(
+ runningSubquery == null ? t
+ : runningSubquery.getCause()));
}
return runningSubquery;
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java 2011-01-21 13:17:01 UTC (rev 4155)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java 2011-01-21 14:28:17 UTC (rev 4156)
@@ -28,7 +28,6 @@
package com.bigdata.bop.controller;
import java.util.Map;
-import java.util.UUID;
import java.util.concurrent.Callable;
import java.util.concurrent.Executor;
import java.util.concurrent.FutureTask;
@@ -40,10 +39,8 @@
import com.bigdata.bop.NV;
import com.bigdata.bop.PipelineOp;
import com.bigdata.bop.engine.IRunningQuery;
-import com.bigdata.bop.engine.LocalChunkMessage;
import com.bigdata.bop.engine.QueryEngine;
import com.bigdata.relation.accesspath.IAsynchronousIterator;
-import com.bigdata.relation.accesspath.ThickAsynchronousIterator;
/**
* For each binding set presented, this operator executes a subquery. Any
@@ -391,25 +388,28 @@
final QueryEngine queryEngine = parentContext.getRunningQuery()
.getQueryEngine();
- final BOp startOp = BOpUtility.getPipelineStart(subQueryOp);
-
- final int startId = startOp.getId();
+// final BOp startOp = BOpUtility.getPipelineStart(subQueryOp);
+//
+// final int startId = startOp.getId();
+//
+// final UUID queryId = UUID.randomUUID();
+//
+// // execute the subquery, passing in the source binding set.
+// runningSubquery = queryEngine
+// .eval(
+// queryId,
+// (PipelineOp) subQueryOp,
+// new LocalChunkMessage<IBindingSet>(
+// queryEngine,
+// queryId,
+// startId,
+// -1 /* partitionId */,
+// new ThickAsynchronousIterator<IBindingSet[]>(
+// new IBindingSet[][] { new IBindingSet[] { bset } })));
- final UUID queryId = UUID.randomUUID();
+ runningSubquery = queryEngine.eval((PipelineOp) subQueryOp,
+ bset);
- // execute the subquery, passing in the source binding set.
- runningSubquery = queryEngine
- .eval(
- queryId,
- (PipelineOp) subQueryOp,
- new LocalChunkMessage<IBindingSet>(
- queryEngine,
- queryId,
- startId,
- -1 /* partitionId */,
- new ThickAsynchronousIterator<IBindingSet[]>(
- new IBindingSet[][] { new IBindingSet[] { bset } })));
-
long ncopied = 0L;
try {
@@ -491,9 +491,11 @@
* Such exceptions are NOT propagated here and WILL NOT
* cause the parent query to terminate.
*/
- throw new RuntimeException(ControllerTask.this.context
- .getRunningQuery().halt(runningSubquery.getCause()));
- }
+ throw new RuntimeException(ControllerTask.this.context
+ .getRunningQuery().halt(
+ runningSubquery == null ? t
+ : runningSubquery.getCause()));
+ }
return runningSubquery;
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/Union.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/Union.java 2011-01-21 13:17:01 UTC (rev 4155)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/Union.java 2011-01-21 14:28:17 UTC (rev 4156)
@@ -35,15 +35,15 @@
/**
* UNION(ops)[maxParallel(default all)]
+ *
* <pre>
* UNION([a,b,c],{})
* </pre>
*
- * Will run the subqueries <i>a</i>, <i>b</i>, and <i>c</i> in parallel. Each
- * subquery will be initialized with a single empty {@link IBindingSet}. The
- * output of those subqueries will be routed to the UNION operator (their
- * parent) unless the subqueries explicitly override this behavior using
- * {@link PipelineOp.Annotations#SINK_REF}.
+ * Will run the subqueries <i>a</i>, <i>b</i>, and <i>c</i> in parallel for each
+ * source {@link IBindingSet}. The output of those subqueries will be routed to
+ * the UNION operator (their parent) unless the subqueries explicitly override
+ * this behavior using {@link PipelineOp.Annotations#SINK_REF}.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
* @version $Id$
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java 2011-01-21 13:17:01 UTC (rev 4155)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/engine/QueryEngine.java 2011-01-21 14:28:17 UTC (rev 4156)
@@ -791,16 +791,57 @@
*/
public AbstractRunningQuery eval(final BOp op) throws Exception {
+ return eval(op, new ListBindingSet());
+
+ }
+
+ /**
+ * Evaluate a query. This node will serve as the controller for the query.
+ *
+ * @param query
+ * The query to evaluate.
+ * @param bset
+ * The initial binding set to present.
+ *
+ * @return The {@link IRunningQuery}.
+ *
+ * @throws IllegalStateException
+ * if the {@link QueryEngine} has been {@link #shutdown()}.
+ * @throws Exception
+ */
+ public AbstractRunningQuery eval(final BOp op, final IBindingSet bset)
+ throws Exception {
+
+ return eval(op, newBindingSetIterator(bset));
+
+ }
+
+ /**
+ * Evaluate a query. This node will serve as the controller for the query.
+ *
+ * @param query
+ * The query to evaluate.
+ * @param bsets
+ * The binding sets to be consumed by the query.
+ *
+ * @return The {@link IRunningQuery}.
+ *
+ * @throws IllegalStateException
+ * if the {@link QueryEngine} has been {@link #shutdown()}.
+ * @throws Exception
+ */
+ public AbstractRunningQuery eval(final BOp op,
+ final IAsynchronousIterator<IBindingSet[]> bsets) throws Exception {
+
final BOp startOp = BOpUtility.getPipelineStart(op);
final int startId = startOp.getId();
-
+
final UUID queryId = UUID.randomUUID();
return eval(queryId, (PipelineOp) op,
new LocalChunkMessage<IBindingSet>(this/* queryEngine */,
- queryId, startId, -1 /* partitionId */,
- newBindingSetIterator(new ListBindingSet())));
+ queryId, startId, -1 /* partitionId */, bsets));
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-21 13:17:07
|
Revision: 4155
http://bigdata.svn.sourceforge.net/bigdata/?rev=4155&view=rev
Author: thompsonbry
Date: 2011-01-21 13:17:01 +0000 (Fri, 21 Jan 2011)
Log Message:
-----------
Bracketed the sail and connection setup with try/finally clauses and used the tear down method for the sail to ensure that the backing file is deleted by the unit test.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNamedGraphs.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedOptionals.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedUnions.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNamedGraphs.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNamedGraphs.java 2011-01-21 09:56:28 UTC (rev 4154)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNamedGraphs.java 2011-01-21 13:17:01 UTC (rev 4155)
@@ -151,13 +151,14 @@
log.info("testing: 8.2.1 Specifying the Default Graph");
final BigdataSail sail = getSail();
- sail.initialize();
- final BigdataSailRepository repo = new BigdataSailRepository(sail);
- final BigdataSailRepositoryConnection cxn =
- (BigdataSailRepositoryConnection) repo.getConnection();
- cxn.setAutoCommit(false);
try {
+ sail.initialize();
+ final BigdataSailRepository repo = new BigdataSailRepository(sail);
+ final BigdataSailRepositoryConnection cxn =
+ (BigdataSailRepositoryConnection) repo.getConnection();
+ try {
+ cxn.setAutoCommit(false);
final BNode a = new BNodeImpl("_:a");
final URI graph = new URIImpl("http://example.org/foaf/aliceFoaf");
@@ -197,8 +198,10 @@
compare(result, answer);
+ } finally {
+ cxn.close();
+ }
} finally {
- cxn.close();
sail.__tearDownUnitTest();
}
@@ -260,13 +263,14 @@
log.info("testing: 8.2.3 Combining FROM and FROM NAMED");
final BigdataSail sail = getSail();
+ try {
sail.initialize();
final BigdataSailRepository repo = new BigdataSailRepository(sail);
final BigdataSailRepositoryConnection cxn =
(BigdataSailRepositoryConnection) repo.getConnection();
- cxn.setAutoCommit(false);
try {
+ cxn.setAutoCommit(false);
final BNode a = new BNodeImpl("_:a");
final BNode b = new BNodeImpl("_:b");
@@ -350,6 +354,8 @@
} finally {
cxn.close();
+ }
+ } finally {
sail.__tearDownUnitTest();
}
@@ -418,14 +424,16 @@
log.info("testing: 8.3.1 Accessing Graph Names");
final BigdataSail sail = getSail();
+ try {
sail.initialize();
final BigdataSailRepository repo = new BigdataSailRepository(sail);
final BigdataSailRepositoryConnection cxn =
(BigdataSailRepositoryConnection) repo.getConnection();
- cxn.setAutoCommit(false);
try {
-
+
+ cxn.setAutoCommit(false);
+
final BNode a = new BNodeImpl("_:a");
final BNode b = new BNodeImpl("_:b");
final BNode z = new BNodeImpl("_:b");
@@ -536,9 +544,11 @@
new BindingImpl("bobNick", new LiteralImpl("Robert"))));
compare(result, answer);
-
+
} finally {
cxn.close();
+ }
+ } finally {
sail.__tearDownUnitTest();
}
@@ -605,14 +615,15 @@
log.info("testing: 8.3.2 Restricting by Graph IRI");
final BigdataSail sail = getSail();
+ try {
sail.initialize();
final BigdataSailRepository repo = new BigdataSailRepository(sail);
final BigdataSailRepositoryConnection cxn =
(BigdataSailRepositoryConnection) repo.getConnection();
+ try {
+
cxn.setAutoCommit(false);
- try {
-
final BNode a = new BNodeImpl("_:a");
final BNode b = new BNodeImpl("_:b");
final BNode z = new BNodeImpl("_:b");
@@ -719,9 +730,10 @@
new BindingImpl("nick", new LiteralImpl("Robert"))));
compare(result, answer);
-
} finally {
cxn.close();
+ }
+ } finally {
sail.__tearDownUnitTest();
}
@@ -810,14 +822,15 @@
log.info("testing: 8.3.3 Restricting Possible Graph IRIs");
final BigdataSail sail = getSail();
+ try {
sail.initialize();
final BigdataSailRepository repo = new BigdataSailRepository(sail);
final BigdataSailRepositoryConnection cxn =
(BigdataSailRepositoryConnection) repo.getConnection();
+ try {
+
cxn.setAutoCommit(false);
- try {
-
final BNode a = new BNodeImpl("_:a");
final BNode b = new BNodeImpl("_:b");
final BNode z = new BNodeImpl("_:b");
@@ -940,6 +953,9 @@
} finally {
cxn.close();
+
+ }
+ } finally {
sail.__tearDownUnitTest();
}
@@ -959,14 +975,15 @@
SailException, QueryEvaluationException, MalformedQueryException {
final BigdataSail sail = getSail();
+ try {
sail.initialize();
final BigdataSailRepository repo = new BigdataSailRepository(sail);
final BigdataSailRepositoryConnection cxn =
(BigdataSailRepositoryConnection) repo.getConnection();
+ try {
+
cxn.setAutoCommit(false);
- try {
-
if(!sail.getDatabase().isQuads()) {
log.warn("test requires quads.");
@@ -1192,6 +1209,10 @@
} finally {
cxn.close();
+
+ }
+ } finally {
+
sail.__tearDownUnitTest();
}
@@ -1212,14 +1233,15 @@
SailException, QueryEvaluationException, MalformedQueryException {
final BigdataSail sail = getSail();
+ try {
sail.initialize();
final BigdataSailRepository repo = new BigdataSailRepository(sail);
final BigdataSailRepositoryConnection cxn = (BigdataSailRepositoryConnection) repo
.getConnection();
+ try {
+
cxn.setAutoCommit(false);
- try {
-
if (!sail.getDatabase().isQuads()) {
log.warn("test requires quads.");
@@ -1559,8 +1581,10 @@
}
} finally {
+ cxn.close();
+ }
+ } finally {
- cxn.close();
sail.__tearDownUnitTest();
}
@@ -1570,14 +1594,15 @@
public void testSearchQuery() throws Exception {
final BigdataSail sail = getSail();
+ try {
sail.initialize();
final BigdataSailRepository repo = new BigdataSailRepository(sail);
final BigdataSailRepositoryConnection cxn = (BigdataSailRepositoryConnection) repo
.getConnection();
+ try {
+
cxn.setAutoCommit(false);
- try {
-
if (!sail.getDatabase().isQuads()) {
log.warn("test requires quads.");
@@ -1669,8 +1694,12 @@
}
} finally {
+
+ cxn.close();
- cxn.close();
+ }
+ } finally {
+
sail.__tearDownUnitTest();
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedOptionals.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedOptionals.java 2011-01-21 09:56:28 UTC (rev 4154)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedOptionals.java 2011-01-21 13:17:01 UTC (rev 4155)
@@ -130,14 +130,16 @@
// final Repository repo = new SailRepository(sail);
final BigdataSail sail = getSail();
+ try {
sail.initialize();
final BigdataSailRepository repo = new BigdataSailRepository(sail);
final RepositoryConnection cxn = repo.getConnection();
- cxn.setAutoCommit(false);
try {
-
+
+ cxn.setAutoCommit(false);
+
final ValueFactory vf = sail.getValueFactory();
/*
@@ -261,7 +263,8 @@
} finally {
cxn.close();
- sail.shutDown();
+ }} finally {
+ sail.__tearDownUnitTest();
}
}
@@ -273,13 +276,14 @@
// final Repository repo = new SailRepository(sail);
final BigdataSail sail = getSail();
+ try {
sail.initialize();
final BigdataSailRepository repo = new BigdataSailRepository(sail);
final RepositoryConnection cxn = repo.getConnection();
- cxn.setAutoCommit(false);
try {
+ cxn.setAutoCommit(false);
final ValueFactory vf = sail.getValueFactory();
@@ -391,9 +395,11 @@
}
- } finally {
- cxn.close();
- sail.shutDown();
+ } finally {
+ cxn.close();
+ }
+ } finally {
+ sail.__tearDownUnitTest();
}
}
@@ -405,13 +411,14 @@
// final Repository repo = new SailRepository(sail);
final BigdataSail sail = getSail();
+ try {
sail.initialize();
final BigdataSailRepository repo = new BigdataSailRepository(sail);
final RepositoryConnection cxn = repo.getConnection();
- cxn.setAutoCommit(false);
try {
+ cxn.setAutoCommit(false);
final ValueFactory vf = sail.getValueFactory();
@@ -528,22 +535,25 @@
}
} finally {
- cxn.close();
- sail.shutDown();
+ cxn.close();
}
+ } finally {
+ sail.__tearDownUnitTest();
+ }
}
private void __testNestedOptionals1() throws Exception {
final BigdataSail sail = getSail();
+ try {
sail.initialize();
final BigdataSailRepository repo = new BigdataSailRepository(sail);
final BigdataSailRepositoryConnection cxn =
(BigdataSailRepositoryConnection) repo.getConnection();
- cxn.setAutoCommit(false);
try {
+ cxn.setAutoCommit(false);
final ValueFactory vf = sail.getValueFactory();
@@ -634,6 +644,8 @@
} finally {
cxn.close();
+ }
+ } finally {
sail.__tearDownUnitTest();
}
@@ -642,13 +654,14 @@
private void _testNestedOptionals2() throws Exception {
final BigdataSail sail = getSail();
+ try {
sail.initialize();
final BigdataSailRepository repo = new BigdataSailRepository(sail);
final BigdataSailRepositoryConnection cxn =
(BigdataSailRepositoryConnection) repo.getConnection();
- cxn.setAutoCommit(false);
try {
+ cxn.setAutoCommit(false);
final ValueFactory vf = sail.getValueFactory();
@@ -811,9 +824,11 @@
// final TupleQueryResult result = tupleQuery.evaluate();
// compare(result, answer);
-
+
} finally {
cxn.close();
+ }
+ } finally {
sail.__tearDownUnitTest();
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedUnions.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedUnions.java 2011-01-21 09:56:28 UTC (rev 4154)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedUnions.java 2011-01-21 13:17:01 UTC (rev 4155)
@@ -130,13 +130,14 @@
// final Repository repo = new SailRepository(sail);
final BigdataSail sail = getSail();
+ try {
sail.initialize();
final BigdataSailRepository repo = new BigdataSailRepository(sail);
final RepositoryConnection cxn = repo.getConnection();
- cxn.setAutoCommit(false);
try {
+ cxn.setAutoCommit(false);
final ValueFactory vf = sail.getValueFactory();
@@ -221,8 +222,10 @@
} finally {
cxn.close();
- sail.shutDown();
}
+ } finally {
+ sail.__tearDownUnitTest();//shutDown();
+ }
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mar...@us...> - 2011-01-21 09:56:35
|
Revision: 4154
http://bigdata.svn.sourceforge.net/bigdata/?rev=4154&view=rev
Author: martyncutcher
Date: 2011-01-21 09:56:28 +0000 (Fri, 21 Jan 2011)
Log Message:
-----------
Fix build by adding IWriteCacheManager and add block offset address cache
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/rwstore/sector/MemoryManager.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/rwstore/sector/SectorAllocator.java
Added Paths:
-----------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/rwstore/IWriteCacheManager.java
Added: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/rwstore/IWriteCacheManager.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/rwstore/IWriteCacheManager.java (rev 0)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/rwstore/IWriteCacheManager.java 2011-01-21 09:56:28 UTC (rev 4154)
@@ -0,0 +1,7 @@
+package com.bigdata.rwstore;
+
+public interface IWriteCacheManager {
+
+ boolean removeWriteToAddr(long address);
+
+}
Property changes on: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/rwstore/IWriteCacheManager.java
___________________________________________________________________
Added: svn:mime-type
+ text/plain
Added: svn:keywords
+ Id Date Revision Author HeadURL
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2011-01-21 00:18:14 UTC (rev 4153)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/rwstore/RWStore.java 2011-01-21 09:56:28 UTC (rev 4154)
@@ -1273,14 +1273,7 @@
}
}
-// final Allocator na = getBlock((int) addr);
-// if (! (na instanceof BlobAllocator)) {
-// throw new IllegalStateException("Invalid Allocator index");
-// }
-// final BlobAllocator ba = (BlobAllocator) na;
-// final int hdraddr = ba.getBlobHdrAddress(getOffset((int) addr));
-// getData(hdraddr, hdrbuf); // read in header - could itself be a blob!
- getData(addr, hdrbuf); // fine but MUST NOT allow header to be a BLOB!
+ getData(addr, hdrbuf); // will work even if header is also a blob
final DataInputStream hdrstr = new DataInputStream(new ByteArrayInputStream(hdrbuf));
final int rhdrs = hdrstr.readInt();
if (rhdrs != nblocks) {
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/rwstore/sector/MemoryManager.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/rwstore/sector/MemoryManager.java 2011-01-21 00:18:14 UTC (rev 4153)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/rwstore/sector/MemoryManager.java 2011-01-21 09:56:28 UTC (rev 4154)
@@ -82,7 +82,7 @@
if ((m_allocation + m_sectorSize) > m_maxResource) {
throw new MemoryManagerResourceError();
}
- SectorAllocator sector = new SectorAllocator(this);//, null);
+ SectorAllocator sector = new SectorAllocator(this, null);
sector.setSectorAddress(m_allocation, m_sectorSize);
sector.setIndex(m_sectors.size());
m_sectors.add(sector);
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/rwstore/sector/SectorAllocator.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/rwstore/sector/SectorAllocator.java 2011-01-21 00:18:14 UTC (rev 4153)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/rwstore/sector/SectorAllocator.java 2011-01-21 09:56:28 UTC (rev 4154)
@@ -33,7 +33,7 @@
import com.bigdata.io.DirectBufferPool;
import com.bigdata.rwstore.FixedOutputStream;
-//import com.bigdata.rwstore.IWriteCacheManager;
+import com.bigdata.rwstore.IWriteCacheManager;
/**
* The SectorAllocator is designed as an alternative the the standard RWStore
@@ -126,11 +126,11 @@
final ISectorManager m_store;
boolean m_onFreeList = false;
private long m_diskAddr;
-// private final IWriteCacheManager m_writes;
+ private final IWriteCacheManager m_writes;
- public SectorAllocator(ISectorManager store) {//, IWriteCacheManager writes) {
+ public SectorAllocator(ISectorManager store, IWriteCacheManager writes) {
m_store = store;
-// m_writes = writes;
+ m_writes = writes;
}
/**
@@ -222,8 +222,14 @@
m_tags[i] = tag;
m_free[tag] += 32;
m_total[tag]++;
+
+ if (i < (m_tags.length-1)) {
+ // cache next block offset
+ m_addresses[i+1] = m_addresses[i] + (32 * ALLOC_SIZES[tag]);
+ }
+
if (log.isTraceEnabled())
- log.trace("addnewTag block for: " + ALLOC_SIZES[tag]);
+ log.trace("addNewTag block for: " + ALLOC_SIZES[tag]);
if ((i+1) == m_tags.length) {
int trim = m_maxSectorSize - (allocated + block);
@@ -232,7 +238,7 @@
return true;
} else {
if (log.isDebugEnabled())
- log.debug("addnewTag FALSE due to Sector SIZE");
+ log.debug("addNewTag FALSE due to Sector SIZE");
return false;
}
@@ -242,7 +248,7 @@
}
if (log.isDebugEnabled())
- log.debug("addnewTag FALSE due to Sector BITS");
+ log.debug("addNewTag FALSE due to Sector BITS");
return false;
}
@@ -276,10 +282,10 @@
m_store.addToFreeList(this);
}
-// if (m_writes != null && m_writes.removeWriteToAddr(getPhysicalAddress(bit))) {
-// if (log.isTraceEnabled())
-// log.trace("Removed potential DUPLICATE");
-// }
+ if (m_writes != null && m_writes.removeWriteToAddr(getPhysicalAddress(bit))) {
+ if (log.isTraceEnabled())
+ log.trace("Removed potential DUPLICATE");
+ }
}
return false;
@@ -306,11 +312,29 @@
return 0;
}
/**
+ * Uses the m_addresses block offset cache to efficiently determine the
+ * corresponding resource offset.
*
* @param bit
* @return the offset in the sector
*/
int bit2Offset(int bit) {
+ final int entry = bit / 32;
+ final int entryBit = bit % 32;
+
+ assert entry < m_addresses.length;
+
+ int offset = m_addresses[entry];
+ offset += entryBit * ALLOC_SIZES[m_tags[entry]];
+
+ return offset;
+ }
+ /**
+ * A previous version of bit2Offset that calculated the offset dynamically
+ * @param bit
+ * @return the offset in the sector
+ */
+ int calcBit2Offset(int bit) {
int offset = 0;
for (int t = 0; t < NUM_ENTRIES; t++) {
int tag = m_tags[t];
@@ -387,8 +411,17 @@
int taglen = str.read(m_tags);
assert taglen == m_tags.length;
+ m_addresses[0] = 0;
for (int i = 0; i < NUM_ENTRIES; i++) {
m_bits[i] = str.readInt();
+
+ // maintain cached block offset
+ if (i < (NUM_ENTRIES-1)) {
+ final int tag = m_tags[i];
+ if (tag != -1) {
+ m_addresses[i+1] = m_addresses[i] + (32 * ALLOC_SIZES[tag]);
+ }
+ }
}
} catch (IOException ioe) {
throw new RuntimeException(ioe);
@@ -503,10 +536,14 @@
m_sectorAddress = sectorAddress;
m_maxSectorSize = maxsize;
+ m_addresses[0] = 0;
for (int i = 0; i < ALLOC_SIZES.length; i++) {
m_tags[i] = (byte) i;
m_free[i] = 32;
m_total[i] = 1;
+
+ // cache block offset
+ m_addresses[i+1] = m_addresses[i] + (32 * ALLOC_SIZES[i]);
}
for (int i = ALLOC_SIZES.length; i < NUM_ENTRIES; i++) {
m_tags[i] = (byte) -1;
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-21 00:18:20
|
Revision: 4153
http://bigdata.svn.sourceforge.net/bigdata/?rev=4153&view=rev
Author: thompsonbry
Date: 2011-01-21 00:18:14 +0000 (Fri, 21 Jan 2011)
Log Message:
-----------
replaced System.err message with log @DEBUG.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/PartitionedJoinGroup.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/PartitionedJoinGroup.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/PartitionedJoinGroup.java 2011-01-21 00:17:18 UTC (rev 4152)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/PartitionedJoinGroup.java 2011-01-21 00:18:14 UTC (rev 4153)
@@ -173,8 +173,6 @@
* @throws IllegalArgumentException
* if any predicate specified in the join path is optional.
*
- * FIXME implement and unit tests.
- *
* @todo Implement (or refactor) the logic to decide which variables need to
* be propagated and which can be dropped. This decision logic will
* need to be available to the runtime query optimizer.
@@ -310,12 +308,11 @@
used.add(c);
-// if (log.isDebugEnabled()) {
-// log.debug
-// }
- System.err.println("Constraint attached at index " + i + " of "
- + path.length + ", bopId=" + p.getId()
- + ", constraint=" + c);
+ if (log.isDebugEnabled()) {
+ log.debug("Constraint attached at index " + i + " of "
+ + path.length + ", bopId=" + p.getId()
+ + ", constraint=" + c);
+ }
if (lastJoin) {
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-21 00:17:25
|
Revision: 4152
http://bigdata.svn.sourceforge.net/bigdata/?rev=4152&view=rev
Author: thompsonbry
Date: 2011-01-21 00:17:18 +0000 (Fri, 21 Jan 2011)
Log Message:
-----------
Added method to return an int[] of the ids for predicates to BOpUtility.
Added method to return the set of constraints attached to the last join in a join path to PartitionedJoinGroup and (a few) unit tests for that method.
Modified BOpBase#toString() to write out everything if an operand is an IValueExpression for better printing of bops.
Modified the QUADS mode SAIL proxy test suites to invoke the TestNestedOptionals and TestNestedUnions test suites.
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/PartitionedJoinGroup.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestPartitionedJoinGroup.java
branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/join/TestPipelineJoin.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailEmbeddedFederationWithQuads.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithQuads.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithQuadsWithoutInlining.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedOptionals.java
branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedUnions.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java 2011-01-20 23:25:24 UTC (rev 4151)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpBase.java 2011-01-21 00:17:18 UTC (rev 4152)
@@ -566,10 +566,14 @@
for (BOp t : args) {
if (nwritten > 0)
sb.append(',');
- sb.append(t.getClass().getSimpleName());
- final Integer tid = (Integer) t.getProperty(Annotations.BOP_ID);
- if (tid != null) {
- sb.append("[" + tid + "]");
+ if(t instanceof IValueExpression<?>) {
+ sb.append(t.toString());
+ } else {
+ sb.append(t.getClass().getSimpleName());
+ final Integer tid = (Integer) t.getProperty(Annotations.BOP_ID);
+ if (tid != null) {
+ sb.append("[" + tid + "]");
+ }
}
nwritten++;
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java 2011-01-20 23:25:24 UTC (rev 4151)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/BOpUtility.java 2011-01-21 00:17:18 UTC (rev 4152)
@@ -1022,6 +1022,37 @@
return out;
}
-
+
+ /**
+ * Return an ordered array of the bopIds associated with an ordered array of
+ * predicates (aka a join path).
+ *
+ * @param path
+ * A join path.
+ *
+ * @return The ordered array of predicates for that join path.
+ *
+ * @throws IllegalArgumentException
+ * if the argument is <code>null</code>.
+ * @throws IllegalArgumentException
+ * if any element of the argument is <code>null</code>.
+ * @throws IllegalStateException
+ * if any {@link IPredicate} does not have a defined bopId as
+ * reported by {@link BOp#getId()}.
+ */
+ public static int[] getPredIds(final IPredicate<?>[] path) {
+
+ final int[] b = new int[path.length];
+
+ for (int i = 0; i < path.length; i++) {
+
+ b[i] = path[i].getId();
+
+ }
+
+ return b;
+
+ }
+
}
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/PartitionedJoinGroup.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/PartitionedJoinGroup.java 2011-01-20 23:25:24 UTC (rev 4151)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/PartitionedJoinGroup.java 2011-01-21 00:17:18 UTC (rev 4152)
@@ -8,6 +8,8 @@
import java.util.Map;
import java.util.Set;
+import org.apache.log4j.Logger;
+
import com.bigdata.bop.BOpUtility;
import com.bigdata.bop.IConstraint;
import com.bigdata.bop.IPredicate;
@@ -65,6 +67,9 @@
*/
public class PartitionedJoinGroup {
+ private static final transient Logger log = Logger
+ .getLogger(PartitionedJoinGroup.class);
+
/**
* The set of variables bound by the non-optional predicates.
*/
@@ -139,6 +144,206 @@
}
/**
+ * Return the set of constraints which should be attached to the last join
+ * in the given the join path. All joins in the join path must be
+ * non-optional joins (that is, part of either the head plan or the join
+ * graph).
+ * <p>
+ * The rule followed by this method is that each constraint will be attached
+ * to the first non-optional join at which all of its variables are known to
+ * be bound. It is assumed that constraints are attached to each join in the
+ * join path by a consistent logic, e.g., as dictated by this method.
+ *
+ * @param joinPath
+ * An ordered array of predicate identifiers representing a
+ * specific sequence of non-optional joins.
+ *
+ * @return The constraints which should be attached to the last join in the
+ * join path.
+ *
+ * @throws IllegalArgumentException
+ * if the join path is <code>null</code>.
+ * @throws IllegalArgumentException
+ * if the join path is empty.
+ * @throws IllegalArgumentException
+ * if any element of the join path is <code>null</code>.
+ * @throws IllegalArgumentException
+ * if any predicate specified in the join path is not known to
+ * this class.
+ * @throws IllegalArgumentException
+ * if any predicate specified in the join path is optional.
+ *
+ * FIXME implement and unit tests.
+ *
+ * @todo Implement (or refactor) the logic to decide which variables need to
+ * be propagated and which can be dropped. This decision logic will
+ * need to be available to the runtime query optimizer.
+ *
+ * @todo This does not pay attention to the head plan. If there can be
+ * constraints on the head plan then either this should be modified
+ * such that it can decide where they attach or we need to have a
+ * method which does the same thing for the head plan.
+ */
+ public IConstraint[] getJoinGraphConstraints(final int[] pathIds) {
+
+ /*
+ * Verify arguments and resolve bopIds to predicates.
+ */
+ if (pathIds == null)
+ throw new IllegalArgumentException();
+
+ final IPredicate<?>[] path = new IPredicate[pathIds.length];
+
+ for (int i = 0; i < pathIds.length; i++) {
+
+ final int id = pathIds[i];
+
+ IPredicate<?> p = null;
+
+ for (IPredicate<?> tmp : joinGraph) {
+
+ if (tmp.getId() == id) {
+
+ p = tmp;
+
+ break;
+
+ }
+
+ }
+
+ if (p == null)
+ throw new IllegalArgumentException("Not found: id=" + id);
+
+ if (p.isOptional())
+ throw new AssertionError(
+ "Not expecting an optional predicate: " + p);
+
+ path[i] = p;
+
+ }
+
+ /*
+ * For each predicate in the path in the given order, figure out which
+ * constraint(s) would attach to that predicate based on which variables
+ * first become bound with that predicate. For the last predicate in the
+ * given join path, we return that set of constraints.
+ */
+
+ // the set of variables which are bound.
+ final Set<IVariable<?>> boundVars = new LinkedHashSet<IVariable<?>>();
+
+ // the set of constraints which have been consumed.
+ final Set<IConstraint> used = new LinkedHashSet<IConstraint>();
+
+ // the set of constraints for the last predicate in the join path.
+ final List<IConstraint> ret = new LinkedList<IConstraint>();
+
+ for(int i = 0; i<path.length; i++) {
+
+ // true iff this is the last join in the path.
+ final boolean lastJoin = i == path.length - 1;
+
+ // a predicate in the path.
+ final IPredicate<?> p = path[i];
+
+ {
+ /*
+ * Visit the variables used by the predicate (and bound by it
+ * since it is not an optional predicate) and add them into the
+ * total set of variables which are bound at this point in the
+ * join path.
+ */
+ final Iterator<IVariable<?>> vitr = BOpUtility
+ .getArgumentVariables(p);
+
+ while (vitr.hasNext()) {
+
+ final IVariable<?> var = vitr.next();
+
+ boundVars.add(var);
+
+ }
+ }
+
+ // consider each constraint.
+ for(IConstraint c : joinGraphConstraints) {
+
+ if (used.contains(c)) {
+ /*
+ * Skip constraints which were already assigned to
+ * predicates before this one in the join path.
+ */
+ continue;
+ }
+
+ /*
+ * true iff all variables used by this constraint are bound at
+ * this point in the join path.
+ */
+ boolean allVarsBound = true;
+
+ // visit the variables used by this constraint.
+ final Iterator<IVariable<?>> vitr = BOpUtility
+ .getSpannedVariables(c);
+
+ while (vitr.hasNext()) {
+
+ final IVariable<?> var = vitr.next();
+
+ if(!boundVars.contains(var)) {
+
+ allVarsBound = false;
+
+ break;
+
+ }
+
+ }
+
+ if (allVarsBound) {
+
+ /*
+ * All variables have become bound for this constraint, so
+ * add it to the set of "used" constraints.
+ */
+
+ used.add(c);
+
+// if (log.isDebugEnabled()) {
+// log.debug
+// }
+ System.err.println("Constraint attached at index " + i + " of "
+ + path.length + ", bopId=" + p.getId()
+ + ", constraint=" + c);
+
+ if (lastJoin) {
+
+ /*
+ * If we are on the last join in the join path, then
+ * this constraint is one of the ones that we will
+ * return.
+ */
+
+ ret.add(c);
+
+ }
+
+ } // if(allVarsBound)
+
+ } // next constraint
+
+ } // next predicate in the join path.
+
+ /*
+ * Return the set of constraints to be applied as of the last predicate
+ * in the join path.
+ */
+ return ret.toArray(new IConstraint[ret.size()]);
+
+ }
+
+ /**
* The {@link IPredicate}s representing optional joins. Any
* {@link IConstraint}s having variable(s) NOT bound by the required joins
* will already have been attached to the last {@link IPredicate} in the
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestPartitionedJoinGroup.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestPartitionedJoinGroup.java 2011-01-20 23:25:24 UTC (rev 4151)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/controller/TestPartitionedJoinGroup.java 2011-01-21 00:17:18 UTC (rev 4152)
@@ -231,6 +231,40 @@
assertSameIteratorAnyOrder("joinGraphConstraints", constraints,
Arrays.asList(fixture.getJoinGraphConstraints()).iterator());
+ /*
+ * Verify the placement of each constraint for a variety of join
+ * paths.
+ */
+ {
+// final int[] pathIds = BOpUtility.getPredIds(new IPredicate[] {
+// p0, p1, p2, p3, p4, p5 });
+// final IConstraint[] actual = fixture
+// .getJoinGraphConstraints(pathIds);
+// System.out.println(Arrays.toString(actual));
+
+ // c1 is applied when x is bound. x is bound by p0.
+ assertEquals(new IConstraint[] { c1 }, fixture
+ .getJoinGraphConstraints(new int[] { p1.getId(),
+ p0.getId() }));
+
+ /*
+ * c1 is applied when x is bound. x is bound by p0. p0 is the
+ * last predicate in this join path, so c1 is attached to p0.
+ */
+ assertEquals(new IConstraint[] { c1 }, fixture
+ .getJoinGraphConstraints(new int[] { p0.getId()}));
+
+ /*
+ * c2 is applied when y is bound. y is bound by p1. p1 is the
+ * last predicate in this join path, p1 is the last predicate in
+ * this join path so c2 is attached to p1.
+ */
+ assertEquals(new IConstraint[] { c2 }, fixture
+ .getJoinGraphConstraints(new int[] { p0.getId(),
+ p1.getId() }));
+
+ }
+
// there is no tail plan.
assertEquals("tailPlan", new IPredicate[] {}, fixture.getTailPlan());
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/join/TestPipelineJoin.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/join/TestPipelineJoin.java 2011-01-20 23:25:24 UTC (rev 4151)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/test/com/bigdata/bop/join/TestPipelineJoin.java 2011-01-21 00:17:18 UTC (rev 4152)
@@ -68,10 +68,23 @@
/**
* Unit tests for the {@link PipelineJoin} operator.
* <p>
- * Note: The operators to map binding sets over shards are tested independently.
+ * Note: The logic to map binding sets over shards is tested independently.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
* @version $Id$
+ *
+ * FIXME This test suite does not fully test the FILTER semantics for
+ * SPARQL joins. In particular, (a) optional join solutions are counted
+ * as successful <em>before</em> the FILTER is applied; and (b) if no
+ * optional join solutions exist (before filtering) for a given source
+ * binding set, then the FILTER is applied to the source binding set to
+ * decide whether or not the source solutions should be passed along.
+ * That is, the behavior is the same in both cases as if the FILTER as
+ * applied after the optional join. However, in fact, we apply it as
+ * part of the join operator in order to keep things simpler for the
+ * runtime query optimizer, which needs to be able to perform cutoff
+ * joins and which also needs to be able to reorder the predicates,
+ * creating the appropriate join operators as it does so.
*/
public class TestPipelineJoin extends TestCase2 {
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailEmbeddedFederationWithQuads.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailEmbeddedFederationWithQuads.java 2011-01-20 23:25:24 UTC (rev 4151)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailEmbeddedFederationWithQuads.java 2011-01-21 00:17:18 UTC (rev 4152)
@@ -97,6 +97,10 @@
// test suite for optionals handling (left joins).
suite.addTestSuite(TestOptionals.class);
+ suite.addTestSuite(TestNestedOptionals.class);
+
+ suite.addTestSuite(TestNestedUnions.class);
+
// test of the search magic predicate
suite.addTestSuite(TestSearchQuery.class);
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithQuads.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithQuads.java 2011-01-20 23:25:24 UTC (rev 4151)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithQuads.java 2011-01-21 00:17:18 UTC (rev 4152)
@@ -79,6 +79,10 @@
// test suite for optionals handling (left joins).
suite.addTestSuite(TestOptionals.class);
+ suite.addTestSuite(TestNestedOptionals.class);
+
+ suite.addTestSuite(TestNestedUnions.class);
+
// test of the search magic predicate
suite.addTestSuite(TestSearchQuery.class);
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithQuadsWithoutInlining.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithQuadsWithoutInlining.java 2011-01-20 23:25:24 UTC (rev 4151)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestBigdataSailWithQuadsWithoutInlining.java 2011-01-21 00:17:18 UTC (rev 4152)
@@ -81,6 +81,10 @@
// test suite for optionals handling (left joins).
suite.addTestSuite(TestOptionals.class);
+ suite.addTestSuite(TestNestedOptionals.class);
+
+ suite.addTestSuite(TestNestedUnions.class);
+
// test of the search magic predicate
suite.addTestSuite(TestSearchQuery.class);
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedOptionals.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedOptionals.java 2011-01-20 23:25:24 UTC (rev 4151)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedOptionals.java 2011-01-21 00:17:18 UTC (rev 4152)
@@ -69,7 +69,7 @@
* @author <a href="mailto:mrp...@us...">Mike Personick</a>
* @version $Id$
*/
-public class TestNestedOptionals extends ProxyBigdataSailTestCase {
+public class TestNestedOptionals extends QuadsTestCase {
/*
* TODO
Modified: branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedUnions.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedUnions.java 2011-01-20 23:25:24 UTC (rev 4151)
+++ branches/QUADS_QUERY_BRANCH/bigdata-sails/src/test/com/bigdata/rdf/sail/TestNestedUnions.java 2011-01-21 00:17:18 UTC (rev 4152)
@@ -69,7 +69,7 @@
* @author <a href="mailto:mrp...@us...">Mike Personick</a>
* @version $Id$
*/
-public class TestNestedUnions extends ProxyBigdataSailTestCase {
+public class TestNestedUnions extends QuadsTestCase {
/*
* TODO
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tho...@us...> - 2011-01-20 23:25:30
|
Revision: 4151
http://bigdata.svn.sourceforge.net/bigdata/?rev=4151&view=rev
Author: thompsonbry
Date: 2011-01-20 23:25:24 +0000 (Thu, 20 Jan 2011)
Log Message:
-----------
javadoc notes on CONSTRAINTS applied to the solutions visited by the SubqueryOp
Modified Paths:
--------------
branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java
Modified: branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java
===================================================================
--- branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java 2011-01-20 22:07:47 UTC (rev 4150)
+++ branches/QUADS_QUERY_BRANCH/bigdata/src/java/com/bigdata/bop/controller/SubqueryOp.java 2011-01-20 23:25:24 UTC (rev 4151)
@@ -440,6 +440,10 @@
/*
* Since there were no solutions for the subquery, copy
* the original binding set to the default sink.
+ *
+ * @todo If we add a CONSTRAINTS annotation to the
+ * SubqueryOp then we need to make sure that it is
+ * applied to all solutions copied out of the subquery.
*/
parentContext.getSink().add(new IBindingSet[]{bset});
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
4 messages has been excluded from this view by a project administrator.
