Menu
▾
▴
bigdata-developers — List for bigdata developers
This list is closed, nobody may subscribe to it.
| 2010 |
Jan
|
Feb
(19) |
Mar
(8) |
Apr
(25) |
May
(16) |
Jun
(77) |
Jul
(131) |
Aug
(76) |
Sep
(30) |
Oct
(7) |
Nov
(3) |
Dec
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2011 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(2) |
Jun
(2) |
Jul
(16) |
Aug
(3) |
Sep
(1) |
Oct
|
Nov
(7) |
Dec
(7) |
| 2012 |
Jan
(10) |
Feb
(1) |
Mar
(8) |
Apr
(6) |
May
(1) |
Jun
(3) |
Jul
(1) |
Aug
|
Sep
(1) |
Oct
|
Nov
(8) |
Dec
(2) |
| 2013 |
Jan
(5) |
Feb
(12) |
Mar
(2) |
Apr
(1) |
May
(1) |
Jun
(1) |
Jul
(22) |
Aug
(50) |
Sep
(31) |
Oct
(64) |
Nov
(83) |
Dec
(28) |
| 2014 |
Jan
(31) |
Feb
(18) |
Mar
(27) |
Apr
(39) |
May
(45) |
Jun
(15) |
Jul
(6) |
Aug
(27) |
Sep
(6) |
Oct
(67) |
Nov
(70) |
Dec
(1) |
| 2015 |
Jan
(3) |
Feb
(18) |
Mar
(22) |
Apr
(121) |
May
(42) |
Jun
(17) |
Jul
(8) |
Aug
(11) |
Sep
(26) |
Oct
(15) |
Nov
(66) |
Dec
(38) |
| 2016 |
Jan
(14) |
Feb
(59) |
Mar
(28) |
Apr
(44) |
May
(21) |
Jun
(12) |
Jul
(9) |
Aug
(11) |
Sep
(4) |
Oct
(2) |
Nov
(1) |
Dec
|
| 2017 |
Jan
(20) |
Feb
(7) |
Mar
(4) |
Apr
(18) |
May
(7) |
Jun
(3) |
Jul
(13) |
Aug
(2) |
Sep
(4) |
Oct
(9) |
Nov
(2) |
Dec
(5) |
| 2018 |
Jan
|
Feb
|
Mar
|
Apr
(2) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2019 |
Jan
|
Feb
|
Mar
(1) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Bryan T. <br...@sy...> - 2013-10-10 09:02:21
|
Jeremy, com.bigdata.bop.rdf.aggregate.TestMAX.test_max_with_errors (from com.bigdata.rdf.TestAll) And also TestMIN.test_min_with_errors are failing in CI. Bryan On Oct 9, 2013, at 8:30 PM, "Jeremy J Carroll" <jj...@sy...<mailto:jj...@sy...>> wrote: I have verified that MIN and MAX both inherit from INeedsMaterialization and added tests that use some of the literals from TestIVComparator that need materialization. Unless Jenkins says otherwise I believe the change is good Jeremy J Carroll Principal Architect Syapse, Inc. On Oct 9, 2013, at 5:04 PM, Jeremy J Carroll <jj...@sy...<mailto:jj...@sy...>> wrote: This is really helpful, I was just about to go home, but think I will spend a few more minutes on this! Jeremy J Carroll Principal Architect Syapse, Inc. On Oct 9, 2013, at 5:00 PM, Bryan Thompson <br...@sy...<mailto:br...@sy...>> wrote: From the ticket you mentioned -- the INeedsMaterialization interface is used to declare which bops require materialization. This gets lifted from subexpressions to decide if we need to do materialization before entering the expression. SNIP<<< There are some known issues with comparisons of inline and non-inline IVs which MikeP will be addressing shortly. Basically, we need an INeedsMaterialization option for operators which always require the materialization of non-inline IVs. ORDER_BY and aggregation both have this characteristic since they must run over all solutions at once and can not re-try solutions which have failed with a NotMaterializedException? <https://sourceforge.net/apps/trac/bigdata/wiki/NotMaterializedException>. As an initial step in that direction, I have defined an IVComparator and a test suite for that class and also added a compareLiteral(IV,Literal) to IVUtility. While this covers some cases, it does not cover all as demonstrated by TestIVComparator. SNIP<<< I think that the question for MIN and MAX is just whether they are imposing the total ordering for SPARQL over RDF Values. Again, I would have to look at the code. That might be an old FIXME. Or it might be a valid problem. However, I think that there are DAWG test cases that cover this. I would check there first and make sure that there is a problem (or an absence of test coverage). Bryan On 10/9/13 7:26 PM, "Jeremy J Carroll" <jj...@sy...<mailto:jj...@sy...>> wrote: Please glance at this code (from MIN.java or MAX.java) ... + private static IVComparator comparator = new IVComparator(); Š - /** - * FIXME This needs to use the ordering define by ORDER_BY. The - * CompareBOp imposes the ordering defined for the "<" operator - * which is less robust and will throw a type exception if you - * attempt to compare unlike Values. - * - * @see https://sourceforge.net/apps/trac/bigdata/ticket/300#comment:5 - */ - if (CompareBOp.compare(iv, min, CompareOp.LT)) { + if (comparator.compare(iv, min)<0) { min = iv; } It seems to work, but I read something about requiring materialization which I did not understand and chose to ignore - was that a mistake? Jeremy J Carroll Principal Architect Syapse, Inc. -------------------------------------------------------------------------- ---- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register > http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktr k _______________________________________________ Bigdata-developers mailing list Big...@li...<mailto:Big...@li...> https://lists.sourceforge.net/lists/listinfo/bigdata-developers |
|
From: Jeremy J C. <jj...@sy...> - 2013-10-10 00:35:14
|
I have verified that MIN and MAX both inherit from INeedsMaterialization and added tests that use some of the literals from TestIVComparator that need materialization. Unless Jenkins says otherwise I believe the change is good Jeremy J Carroll Principal Architect Syapse, Inc. On Oct 9, 2013, at 5:04 PM, Jeremy J Carroll <jj...@sy...> wrote: > This is really helpful, I was just about to go home, but think I will spend a few more minutes on this! > > > Jeremy J Carroll > Principal Architect > Syapse, Inc. > > > > On Oct 9, 2013, at 5:00 PM, Bryan Thompson <br...@sy...> wrote: > >> From the ticket you mentioned -- the INeedsMaterialization interface is >> used to declare which bops require materialization. This gets lifted from >> subexpressions to decide if we need to do materialization before entering >> the expression. >> >>>>> SNIP<<< >> There are some known issues with comparisons of inline and non-inline >> IVs which MikeP will be addressing shortly. Basically, we need an >> INeedsMaterialization option for operators which always require the >> materialization of non-inline IVs. ORDER_BY and aggregation both have >> this characteristic since they must run over all solutions at once and >> can not re-try solutions which have failed with a >> NotMaterializedException? >> <https://sourceforge.net/apps/trac/bigdata/wiki/NotMaterializedException>. >> As an initial step in that direction, I have defined an IVComparator >> and a test suite for that class and also added a >> compareLiteral(IV,Literal) to IVUtility. While this covers some cases, >> it does not cover all as demonstrated by TestIVComparator. >> >>>>> SNIP<<< >> >> I think that the question for MIN and MAX is just whether they are >> imposing the total ordering for SPARQL over RDF Values. Again, I would >> have to look at the code. That might be an old FIXME. Or it might be a >> valid problem. However, I think that there are DAWG test cases that cover >> this. I would check there first and make sure that there is a problem (or >> an absence of test coverage). >> >> >> Bryan >> >> >> On 10/9/13 7:26 PM, "Jeremy J Carroll" <jj...@sy...> wrote: >> >>> >>> Please glance at this code (from MIN.java or MAX.java) >>> >>> ... >>> >>> + private static IVComparator comparator = new IVComparator(); >>> >>> Š >>> >>> >>> - /** >>> - * FIXME This needs to use the ordering define by >>> ORDER_BY. The >>> - * CompareBOp imposes the ordering defined for the "<" >>> operator >>> - * which is less robust and will throw a type exception >>> if you >>> - * attempt to compare unlike Values. >>> - * >>> - * @see >>> https://sourceforge.net/apps/trac/bigdata/ticket/300#comment:5 >>> - */ >>> - if (CompareBOp.compare(iv, min, CompareOp.LT)) { >>> + if (comparator.compare(iv, min)<0) { >>> >>> min = iv; >>> >>> } >>> >>> >>> >>> It seems to work, but I read something about requiring materialization >>> which I did not understand and chose to ignore - was that a mistake? >>> >>> Jeremy J Carroll >>> Principal Architect >>> Syapse, Inc. >>> >>> >>> >>> >>> -------------------------------------------------------------------------- >>> ---- >>> October Webinars: Code for Performance >>> Free Intel webinars can help you accelerate application performance. >>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most >>> from >>> the latest Intel processors and coprocessors. See abstracts and register > >>> http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktr >>> k >>> _______________________________________________ >>> Bigdata-developers mailing list >>> Big...@li... >>> https://lists.sourceforge.net/lists/listinfo/bigdata-developers >> > |
|
From: Bryan T. <br...@sy...> - 2013-10-10 00:15:10
|
The code that establishes these patterns for FILTERs is in the class hierarchy of AST2BOpUtility (in one of the subclasses). The aggregation stuff is mostly handled in AST2BOPUtility and in some helper classes to analyze aggregations. Thanks, Bryan On 10/9/13 8:04 PM, "Jeremy J Carroll" <jj...@sy...> wrote: >This is really helpful, I was just about to go home, but think I will >spend a few more minutes on this! > > >Jeremy J Carroll >Principal Architect >Syapse, Inc. > > > >On Oct 9, 2013, at 5:00 PM, Bryan Thompson <br...@sy...> wrote: > >> From the ticket you mentioned -- the INeedsMaterialization interface is >> used to declare which bops require materialization. This gets lifted >>from >> subexpressions to decide if we need to do materialization before >>entering >> the expression. >> >>>>> SNIP<<< >> There are some known issues with comparisons of inline and non-inline >> IVs which MikeP will be addressing shortly. Basically, we need an >> INeedsMaterialization option for operators which always require the >> materialization of non-inline IVs. ORDER_BY and aggregation both have >> this characteristic since they must run over all solutions at once and >> can not re-try solutions which have failed with a >> NotMaterializedException? >> >><https://sourceforge.net/apps/trac/bigdata/wiki/NotMaterializedException> >>. >> As an initial step in that direction, I have defined an IVComparator >> and a test suite for that class and also added a >> compareLiteral(IV,Literal) to IVUtility. While this covers some cases, >> it does not cover all as demonstrated by TestIVComparator. >> >>>>> SNIP<<< >> >> I think that the question for MIN and MAX is just whether they are >> imposing the total ordering for SPARQL over RDF Values. Again, I would >> have to look at the code. That might be an old FIXME. Or it might be a >> valid problem. However, I think that there are DAWG test cases that >>cover >> this. I would check there first and make sure that there is a problem >>(or >> an absence of test coverage). >> >> >> Bryan >> >> >> On 10/9/13 7:26 PM, "Jeremy J Carroll" <jj...@sy...> wrote: >> >>> >>> Please glance at this code (from MIN.java or MAX.java) >>> >>> ... >>> >>> + private static IVComparator comparator = new >>>IVComparator(); >>> >>> Š >>> >>> >>> - /** >>> - * FIXME This needs to use the ordering define by >>> ORDER_BY. The >>> - * CompareBOp imposes the ordering defined for the "<" >>> operator >>> - * which is less robust and will throw a type >>>exception >>> if you >>> - * attempt to compare unlike Values. >>> - * >>> - * @see >>> https://sourceforge.net/apps/trac/bigdata/ticket/300#comment:5 >>> - */ >>> - if (CompareBOp.compare(iv, min, CompareOp.LT)) { >>> + if (comparator.compare(iv, min)<0) { >>> >>> min = iv; >>> >>> } >>> >>> >>> >>> It seems to work, but I read something about requiring materialization >>> which I did not understand and chose to ignore - was that a mistake? >>> >>> Jeremy J Carroll >>> Principal Architect >>> Syapse, Inc. >>> >>> >>> >>> >>> >>>------------------------------------------------------------------------ >>>-- >>> ---- >>> October Webinars: Code for Performance >>> Free Intel webinars can help you accelerate application performance. >>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the >>>most >>> from >>> the latest Intel processors and coprocessors. See abstracts and >>>register > >>> >>>http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clk >>>tr >>> k >>> _______________________________________________ >>> Bigdata-developers mailing list >>> Big...@li... >>> https://lists.sourceforge.net/lists/listinfo/bigdata-developers >> > |
|
From: Jeremy J C. <jj...@sy...> - 2013-10-10 00:04:31
|
This is really helpful, I was just about to go home, but think I will spend a few more minutes on this! Jeremy J Carroll Principal Architect Syapse, Inc. On Oct 9, 2013, at 5:00 PM, Bryan Thompson <br...@sy...> wrote: > From the ticket you mentioned -- the INeedsMaterialization interface is > used to declare which bops require materialization. This gets lifted from > subexpressions to decide if we need to do materialization before entering > the expression. > >>>> SNIP<<< > There are some known issues with comparisons of inline and non-inline > IVs which MikeP will be addressing shortly. Basically, we need an > INeedsMaterialization option for operators which always require the > materialization of non-inline IVs. ORDER_BY and aggregation both have > this characteristic since they must run over all solutions at once and > can not re-try solutions which have failed with a > NotMaterializedException? > <https://sourceforge.net/apps/trac/bigdata/wiki/NotMaterializedException>. > As an initial step in that direction, I have defined an IVComparator > and a test suite for that class and also added a > compareLiteral(IV,Literal) to IVUtility. While this covers some cases, > it does not cover all as demonstrated by TestIVComparator. > >>>> SNIP<<< > > I think that the question for MIN and MAX is just whether they are > imposing the total ordering for SPARQL over RDF Values. Again, I would > have to look at the code. That might be an old FIXME. Or it might be a > valid problem. However, I think that there are DAWG test cases that cover > this. I would check there first and make sure that there is a problem (or > an absence of test coverage). > > > Bryan > > > On 10/9/13 7:26 PM, "Jeremy J Carroll" <jj...@sy...> wrote: > >> >> Please glance at this code (from MIN.java or MAX.java) >> >> ... >> >> + private static IVComparator comparator = new IVComparator(); >> >> Š >> >> >> - /** >> - * FIXME This needs to use the ordering define by >> ORDER_BY. The >> - * CompareBOp imposes the ordering defined for the "<" >> operator >> - * which is less robust and will throw a type exception >> if you >> - * attempt to compare unlike Values. >> - * >> - * @see >> https://sourceforge.net/apps/trac/bigdata/ticket/300#comment:5 >> - */ >> - if (CompareBOp.compare(iv, min, CompareOp.LT)) { >> + if (comparator.compare(iv, min)<0) { >> >> min = iv; >> >> } >> >> >> >> It seems to work, but I read something about requiring materialization >> which I did not understand and chose to ignore - was that a mistake? >> >> Jeremy J Carroll >> Principal Architect >> Syapse, Inc. >> >> >> >> >> -------------------------------------------------------------------------- >> ---- >> October Webinars: Code for Performance >> Free Intel webinars can help you accelerate application performance. >> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most >> from >> the latest Intel processors and coprocessors. See abstracts and register > >> http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktr >> k >> _______________________________________________ >> Bigdata-developers mailing list >> Big...@li... >> https://lists.sourceforge.net/lists/listinfo/bigdata-developers > |
|
From: Bryan T. <br...@sy...> - 2013-10-10 00:00:57
|
>From the ticket you mentioned -- the INeedsMaterialization interface is used to declare which bops require materialization. This gets lifted from subexpressions to decide if we need to do materialization before entering the expression. >>>SNIP<<< There are some known issues with comparisons of inline and non-inline IVs which MikeP will be addressing shortly. Basically, we need an INeedsMaterialization option for operators which always require the materialization of non-inline IVs. ORDER_BY and aggregation both have this characteristic since they must run over all solutions at once and can not re-try solutions which have failed with a NotMaterializedException? <https://sourceforge.net/apps/trac/bigdata/wiki/NotMaterializedException>. As an initial step in that direction, I have defined an IVComparator and a test suite for that class and also added a compareLiteral(IV,Literal) to IVUtility. While this covers some cases, it does not cover all as demonstrated by TestIVComparator. >>>SNIP<<< I think that the question for MIN and MAX is just whether they are imposing the total ordering for SPARQL over RDF Values. Again, I would have to look at the code. That might be an old FIXME. Or it might be a valid problem. However, I think that there are DAWG test cases that cover this. I would check there first and make sure that there is a problem (or an absence of test coverage). Bryan On 10/9/13 7:26 PM, "Jeremy J Carroll" <jj...@sy...> wrote: > >Please glance at this code (from MIN.java or MAX.java) > >... > >+ private static IVComparator comparator = new IVComparator(); > >Š > > >- /** >- * FIXME This needs to use the ordering define by >ORDER_BY. The >- * CompareBOp imposes the ordering defined for the "<" >operator >- * which is less robust and will throw a type exception >if you >- * attempt to compare unlike Values. >- * >- * @see >https://sourceforge.net/apps/trac/bigdata/ticket/300#comment:5 >- */ >- if (CompareBOp.compare(iv, min, CompareOp.LT)) { >+ if (comparator.compare(iv, min)<0) { > > min = iv; > > } > > > >It seems to work, but I read something about requiring materialization >which I did not understand and chose to ignore - was that a mistake? > >Jeremy J Carroll >Principal Architect >Syapse, Inc. > > > > >-------------------------------------------------------------------------- >---- >October Webinars: Code for Performance >Free Intel webinars can help you accelerate application performance. >Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most >from >the latest Intel processors and coprocessors. See abstracts and register > >http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktr >k >_______________________________________________ >Bigdata-developers mailing list >Big...@li... >https://lists.sourceforge.net/lists/listinfo/bigdata-developers |
|
From: Bryan T. <br...@sy...> - 2013-10-09 23:57:18
|
This is a complex subject. I think that there have been postings on it before. The general approach is based on the following: - bigdata inlines a lot of different kinds of values into the statement indices rather than storing them in the dictionaries. Examples are xsd:int, xsd:integer, xsd:decimal, xsd:float, xsd:double, etc. Anything that is "inline" can be directly converted from an IV back into an RDF Value. Things that are not inline (most URIs, lots of literals) have to be converted through "materialization". Materialization is basically a JOIN against the appropriate index (either ID2TERM or BLOBS). - RDF and SPARQL are typed only at runtime for most queries. This means that we do not (in general) know the data type of the variable bindings until things are evaluated. - We handle this late discovery of typing through a pattern in which we first process those solutions that can be handled without materialization and then route the solutions that were not processed into the materialization step and then re-try the operation. This is called "conditional materialization." It gets applied for FILTERs. So, if you try to do a function and everything is xsd:int (or anything else that can be converted back into an RDF Value), then we do not perform the "materialization" join. MIN, MAX, AVG, and the other aggregates might be different. I would have to go back and look at the code. I think that we require mandatory materialization in front of the aggregation operations since we can not do the conditional materialization pattern. (Aggregation is evaluated in a single operator so the try-fail-materialize-retry pattern can not be made to work). There is an interface that is used to declare the materialization requirements of various things (bops). MIN probably requires materialization. Does that help? Bryan On 10/9/13 7:26 PM, "Jeremy J Carroll" <jj...@sy...> wrote: > >Please glance at this code (from MIN.java or MAX.java) > >... > >+ private static IVComparator comparator = new IVComparator(); > >Š > > >- /** >- * FIXME This needs to use the ordering define by >ORDER_BY. The >- * CompareBOp imposes the ordering defined for the "<" >operator >- * which is less robust and will throw a type exception >if you >- * attempt to compare unlike Values. >- * >- * @see >https://sourceforge.net/apps/trac/bigdata/ticket/300#comment:5 >- */ >- if (CompareBOp.compare(iv, min, CompareOp.LT)) { >+ if (comparator.compare(iv, min)<0) { > > min = iv; > > } > > > >It seems to work, but I read something about requiring materialization >which I did not understand and chose to ignore - was that a mistake? > >Jeremy J Carroll >Principal Architect >Syapse, Inc. > > > > >-------------------------------------------------------------------------- >---- >October Webinars: Code for Performance >Free Intel webinars can help you accelerate application performance. >Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most >from >the latest Intel processors and coprocessors. See abstracts and register > >http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktr >k >_______________________________________________ >Bigdata-developers mailing list >Big...@li... >https://lists.sourceforge.net/lists/listinfo/bigdata-developers |
|
From: Jeremy J C. <jj...@sy...> - 2013-10-09 23:26:48
|
Please glance at this code (from MIN.java or MAX.java) ... + private static IVComparator comparator = new IVComparator(); … - /** - * FIXME This needs to use the ordering define by ORDER_BY. The - * CompareBOp imposes the ordering defined for the "<" operator - * which is less robust and will throw a type exception if you - * attempt to compare unlike Values. - * - * @see https://sourceforge.net/apps/trac/bigdata/ticket/300#comment:5 - */ - if (CompareBOp.compare(iv, min, CompareOp.LT)) { + if (comparator.compare(iv, min)<0) { min = iv; } It seems to work, but I read something about requiring materialization which I did not understand and chose to ignore - was that a mistake? Jeremy J Carroll Principal Architect Syapse, Inc. |
|
From: Jeremy C. <jj...@gm...> - 2013-10-09 18:52:13
|
I drilled down on defect 739 this morning which had to do with zero length property paths with variables at either end. The specification is not very practical: it says that such a zlpp matches every node in the graph. http://www.w3.org/TR/2013/REC-sparql11-query-20130321/#defn_evalPP_ZeroOrOnePath however, I noted that sometimes bigdata was giving the right answer, and sometimes the wrong answer: and the times when it was getting the right answer was when the cardinality estimate for the zlpp was higher (because the property inside it had more matches). Thus the fix was to correct the implementation of com.bigdata.rdf.sparql.ast.ArbitraryLengthPathNode.getEstimatedCardinality(StaticOptimizer) to return the correct Long.MAX_VALUE in this case, even though we don't implement it correctly (and we actually return zero values rather than infinite); and then the rest of the optimizer does the right thing. The result is that in the test cases presented (by my colleague) in which we have a zlpp that may match or may not, we now get correct results but in the artificial case select * { ?s <eg:noSuchProperty>? ?o } LIMIT 100 we continue to incorrectly return no results. Note this sparql endpoint gets this query right: http://resource.geolba.ac.at/PoolParty/sparql/tectonicunit so it is not impossible …. (not that I would advocate this) Jeremy |
|
From: Jeremy J C. <jj...@sy...> - 2013-10-09 00:24:56
|
I believe I have now fixed the bug, which was easier than when I had misunderstood the correct behavior. The issue was that the replacing of Y by -unbound-var-Y in a FILTER did not recurse into complex FILTER expressions. This is in r7440, and I have closed the trac item Jeremy J Carroll Principal Architect Syapse, Inc. On Oct 8, 2013, at 3:39 PM, Jeremy Carroll <jj...@gm...> wrote: > I am reviewing the other 747 tests. > It is clear that the 747 tests are exploring some defects in that adding or deleting && True to a filter should not make any difference. > |
|
From: Jeremy C. <jj...@gm...> - 2013-10-08 22:39:34
|
I am more than happy to have a call this week, I am fairly unconstrained: tomorrow or Friday morning may be best, but I am open to other suggestions.
I don't believe I have made any substantial unreviewed changes to the actual code - and certainly have not changed outcomes of pre-existing tests
The commit I am most suspicious of is a one-word change: r7429, in that this was motivated by a (new) test case. The commit itself looks pretty watertight - the code change clearly conforms with the comment …
The test case that motivated r7429 is this one:
SELECT *
WHERE {
{ BIND ( "y" as $Y )
}
UNION {
FILTER (false)
}
BIND ( "x" as $X )
FILTER( BOUND($Y)
&& True
)
}
where my reading is that the $Y is in scope for the filter, and the test should produce a result. (X=x and Y=y).
I am reviewing the other 747 tests.
It is clear that the 747 tests are exploring some defects in that adding or deleting && True to a filter should not make any difference.
Jeremy
On Oct 8, 2013, at 2:56 PM, Bryan Thompson <br...@sy...> wrote:
> Jeremy,
>
> I have not worked through your recent changes yet. Maybe we can schedule a call later this week to do that?
>
> Had you changed the expected outcome for this test? The tests in com.bigdata.rdf.sparql.ast.eval.TestTck are based on DAWG tests. They should be correct in their data, expected result, and actual result. E.g., nested-filter-2 should be correct in its historical committed state.
>
> Oh. I think I understand. You are saying that you have revised your interpretation of the correct execution for the query you inlined below based on the DAWG filter-nested-2 test, right?
>
> Bryan
|
|
From: Bryan T. <br...@sy...> - 2013-10-08 21:57:08
|
Jeremy,
I have not worked through your recent changes yet. Maybe we can schedule a call later this week to do that?
Had you changed the expected outcome for this test? The tests in com.bigdata.rdf.sparql.ast.eval.TestTck are based on DAWG tests. They should be correct in their data, expected result, and actual result. E.g., nested-filter-2 should be correct in its historical committed state.
Oh. I think I understand. You are saying that you have revised your interpretation of the correct execution for the query you inlined below based on the DAWG filter-nested-2 test, right?
Bryan
From: Jeremy Carroll <jj...@gm...<mailto:jj...@gm...>>
Date: Tuesday, October 8, 2013 5:24 PM
To: "Big...@li...<mailto:Big...@li...>" <Big...@li...<mailto:Big...@li...>>
Subject: [Bigdata-developers] note: 747 FILTER(BOUND()) bug - recent incorrect code committed (by me)
I have done some work on a defect concerning the following SPARQL
SELECT *
WHERE {
BIND ( 1 as $Y ) .
{
BIND ( 2 as $X )
FILTER( BOUND($Y)
# && True
)
}
}
where, the commented out line incorrectly influenced the result set, on the basis that the correct result set should have one entry.
I am now of the belief that the correct result set should have zero entries, and I will review the work I have done on that basis.
(See DAWG test case filter-nested-2)
Jeremy
|
|
From: Jeremy C. <jj...@gm...> - 2013-10-08 21:24:43
|
I have done some work on a defect concerning the following SPARQL
SELECT *
WHERE {
BIND ( 1 as $Y ) .
{
BIND ( 2 as $X )
FILTER( BOUND($Y)
# && True
)
}
}
where, the commented out line incorrectly influenced the result set, on the basis that the correct result set should have one entry.
I am now of the belief that the correct result set should have zero entries, and I will review the work I have done on that basis.
(See DAWG test case filter-nested-2)
Jeremy
|
|
From: Bryan T. <br...@sy...> - 2013-10-08 12:32:32
|
We do sync through the build artifacts to the web site as well, but the links are not fixed up. lastSuccessful: http://www.bigdata.com/hudson-release-1.3.0/lastSuccessful/archive/BIGDATA_RELEASE_1_3_0/ant-build/classes/test/test-results/report.tgz A specific CI build: http://www.bigdata.com/hudson-release-1.3.0/builds/352/archive/BIGDATA_RELEASE_1_3_0/ant-build/classes/test/test-results/report.tgz The HA artifacts for a specific CI build: http://www.bigdata.com/hudson-release-1.3.0/builds/352/archive/BIGDATA_RELEASE_1_3_0/ant-build/classes/test/test-results/report.tgz I have updated the wiki to note these links and the (current) linking pattern at [1]. Bryan [1] https://sourceforge.net/apps/mediawiki/bigdata/index.php?title=Contributors#Continuous_Integration_.28CI.29 From: Jeremy Carroll <jj...@sy...<mailto:jj...@sy...>> Date: Monday, October 7, 2013 3:03 PM To: "Big...@li...<mailto:Big...@li...>" <Big...@li...<mailto:Big...@li...>> Subject: [Bigdata-developers] build URLs … Any chance of getting the correct links out of Jenkins … I quite wanted to click through to See <http://192.168.1.10:8080/job/bigdata-release-1.3.0/352/> Jeremy J Carroll Principal Architect Syapse, Inc. |
|
From: Bryan T. <br...@sy...> - 2013-10-08 00:25:17
|
Can provide tomorrow. From: Jeremy Carroll <jj...@sy...<mailto:jj...@sy...>> Date: Monday, October 7, 2013 3:03 PM To: "Big...@li...<mailto:Big...@li...>" <Big...@li...<mailto:Big...@li...>> Subject: [Bigdata-developers] build URLs … Any chance of getting the correct links out of Jenkins … I quite wanted to click through to See <http://192.168.1.10:8080/job/bigdata-release-1.3.0/352/> Jeremy J Carroll Principal Architect Syapse, Inc. |
|
From: Jeremy J C. <jj...@sy...> - 2013-10-07 22:22:37
|
I am reviewing StaticAnalysis and ASTBottomUpOptimizer to try and working 725, 739 and 747 I have added a new ticket https://sourceforge.net/apps/trac/bigdata/ticket/750 for the case of a non-aggregate projection in a select expression with an error. The ticket seems to come from a different belief about correct behavior than the comments in the code, so it feels that we should try and get on the same page about what correct behavior is before addressing the ticket. Unlike most of my tickets this is not motivated by an actual code failure for me or my team, but as a precursor to working on other tickets. I am looking at this code in StaticAnalysis, starting around like 1450 /* * 4. The projection of a select expression which is not an * aggregate. The projected variable will be definitely * bound if all components of the select expression are * definitely bound. * * TODO Does coalesce() change the semantics for this * analysis? If any of the values for coalesce() is * definitely bound, then the coalesce() will produce a * value. Can coalesce() be used to propagate an unbound * value? If so, then we must either not assume that any * value expression involving coalesce() is definitely bound * or we must do a more detailed analysis of the value * expression. */ final Set<IVariable<?>> usedVars = getSpannedVariables( (BOp) bind.getValueExpression(), new LinkedHashSet<IVariable<?>>()); usedVars.removeAll(definitelyBound); if (!usedVars.isEmpty()) { /* * There is at least one variable which is used by the * select expression which is not definitely bound. */ continue; } /* * All variables used by the select expression are * definitely bound so the projected variable for that * select expression will be definitely bound. */ tmp.add(bind.getVar()); This seems incorrect - more accurately would be to say that any variable involving a non-trivial expression is maybe bound; a small amount of expression analysis is probably worthwhile (e.g. definitely bound if expression is a constant; and some treatment if the expr is a variable) Jeremy J Carroll Principal Architect Syapse, Inc. |
|
From: Jeremy J C. <jj...@sy...> - 2013-10-07 21:07:35
|
On Oct 4, 2013, at 3:17 PM, Bryan Thompson <br...@sy...> wrote: > Let me suggest that we create a system propery for nspin that will let you override it and gain experience with that without pushing a change to everyone. I am very happy with this as a next step - I may split the variable into two: one for the read end (the iterator) and one for the write end. If I do that I would only expose the former as a system property. From my point of view, it would get Syapse off a forked version of the code. I see the fork as a necessary evil, use of which should be minimized. Jeremy J Carroll Principal Architect Syapse, Inc. |
|
From: Jeremy J C. <jj...@sy...> - 2013-10-07 21:04:14
|
I have just checked in further tests concerning 739 and 725, and along with tests for 747 I think this show a pattern of incorrect behavior concerning handleFiltersWithVariablesNotInScope and com.bigdata.rdf.sparql.ast.StaticAnalysis.getMaybeProducedBindings(IBindingProducerNode, Set<IVariable<?>>, boolean) My plan is to try recoding in this area (rather than simply patching existing code); and see if I can make progress on the test cases. Jeremy J Carroll Principal Architect Syapse, Inc. |
|
From: Jeremy J C. <jj...@sy...> - 2013-10-07 19:04:02
|
Any chance of getting the correct links out of Jenkins … I quite wanted to click through to See <http://192.168.1.10:8080/job/bigdata-release-1.3.0/352/> Jeremy J Carroll Principal Architect Syapse, Inc. |
|
From: Jeremy J C. <jj...@sy...> - 2013-10-07 17:30:43
|
I believe this revealed a defect in the CBD code. Adding System.err.println(astContainer); at line 461 of com.bigdata.rdf.sparql.ast.eval.CBD gives originalAST QueryType: DESCRIBE SELECT [describeMode=SymmetricOneStep]( ConstantNode(TermId(27B)[t27]) AS VarNode(-cbd-bnode-1)[anonymous] ) ( ConstantNode(TermId(26B)[t26]) AS VarNode(-cbd-bnode-1)[anonymous] ) ( ConstantNode(TermId(25B)[t25]) AS VarNode(-cbd-bnode-1)[anonymous] ) which looks suspicious because the same variable is being used multiple times for different things (this is when running the failing tests) I made a small fix so that the system print statement gives: originalAST QueryType: DESCRIBE SELECT [describeMode=ForwardOneStep]( ConstantNode(TermId(27B)[t27]) AS VarNode(-cbd-bnode-1)[anonymous] ) ( ConstantNode(TermId(24B)[t24]) AS VarNode(-cbd-bnode-2)[anonymous] ) ( ConstantNode(TermId(25B)[t25]) AS VarNode(-cbd-bnode-3)[anonymous] ) which not only looks better, but the failing tests pass. I have committed this as: r7436 and I am hoping that there are no further follow on defects. I hope it is clear that there is some guess work going on here. I am fairly confident that my change is accurately reflected in this message; I do not however claim to understand the big picture as to what is going on … My understanding as to how this relates to the fix for 748 is that in 748 the defect allowed a variable to be reset even though it had a value - which means that the reuse of the var in the CBD did what was hoped for (whatever that was) despite it not conforming with the variable semantics: variables do not get reused in this way. Jeremy J Carroll Principal Architect Syapse, Inc. On Oct 5, 2013, at 3:39 AM, Bryan Thompson <br...@sy...> wrote: > Jeremy, either this commit or the previous one broke two tests in TestDescribe related to the CBD support. Let me know if you have time to look at this. Bryan > > > Begin forwarded message: > >> From: "jer...@us..." <jer...@us...> >> Date: October 3, 2013 at 11:45:34 AM EDT >> To: "big...@li..." <big...@li...> >> Subject: [Bigdata-commit] SF.net SVN: bigdata:[7433] branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/ java/com/bigdata/rdf >> >> Revision: 7433 >> http://bigdata.svn.sourceforge.net/bigdata/?rev=7433&view=rev >> Author: jeremy_carroll >> Date: 2013-10-03 15:45:34 +0000 (Thu, 03 Oct 2013) >> Log Message: >> ----------- >> Deleted duplicate code, and added comments. >> >> Modified Paths: >> -------------- >> branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/ProjectedConstraint.java >> branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java >> >> Removed Paths: >> ------------- >> branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/BindingConstraint.java >> >> Deleted: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/BindingConstraint.java >> =================================================================== >> --- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/BindingConstraint.java 2013-10-03 15:18:56 UTC (rev 7432) >> +++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/BindingConstraint.java 2013-10-03 15:45:34 UTC (rev 7433) >> @@ -1,61 +0,0 @@ >> -/** >> - >> -Copyright (C) SYSTAP, LLC 2006-2011. All rights reserved. >> - >> -Contact: >> - SYSTAP, LLC >> - 4501 Tower Road >> - Greensboro, NC 27410 >> - lic...@bi... >> - >> -This program is free software; you can redistribute it and/or modify >> -it under the terms of the GNU General Public License as published by >> -the Free Software Foundation; version 2 of the License. >> - >> -This program is distributed in the hope that it will be useful, >> -but WITHOUT ANY WARRANTY; without even the implied warranty of >> -MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -GNU General Public License for more details. >> - >> -You should have received a copy of the GNU General Public License >> -along with this program; if not, write to the Free Software >> -Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA >> - */ >> -package com.bigdata.rdf.internal.constraints; >> - >> -import java.util.Map; >> - >> -import com.bigdata.bop.BOp; >> -import com.bigdata.bop.IBindingSet; >> -import com.bigdata.rdf.error.SparqlTypeErrorException; >> - >> -@SuppressWarnings({ "unchecked", "rawtypes" }) >> -public class BindingConstraint extends com.bigdata.bop.constraint.Constraint { >> - >> - private static final long serialVersionUID = 1L; >> - >> - public BindingConstraint(final BOp[] args, final Map<String, Object> annotations) { >> - super(args, annotations); >> - } >> - >> - public BindingConstraint(final BindingConstraint op) { >> - super(op); >> - } >> - >> - public BindingConstraint(ConditionalBind bind) { >> - super(new BOp[]{bind}, null); >> - } >> - >> - @Override >> - public boolean accept(final IBindingSet bindingSet) { >> - try{ >> - final Object result = ((ConditionalBind) get(0)).get(bindingSet); >> - >> - return result != null; >> - } catch (SparqlTypeErrorException stee) { >> - return true; >> - } >> - >> - } >> - >> -} >> >> Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/ProjectedConstraint.java >> =================================================================== >> --- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/ProjectedConstraint.java 2013-10-03 15:18:56 UTC (rev 7432) >> +++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/ProjectedConstraint.java 2013-10-03 15:45:34 UTC (rev 7433) >> @@ -1,3 +1,28 @@ >> +/* >> + >> +Copyright (C) SYSTAP, LLC 2006-2013. All rights reserved. >> + >> +Contact: >> + SYSTAP, LLC >> + 4501 Tower Road >> + Greensboro, NC 27410 >> + lic...@bi... >> + >> +This program is free software; you can redistribute it and/or modify >> +it under the terms of the GNU General Public License as published by >> +the Free Software Foundation; version 2 of the License. >> + >> +This program is distributed in the hope that it will be useful, >> +but WITHOUT ANY WARRANTY; without even the implied warranty of >> +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> +GNU General Public License for more details. >> + >> +You should have received a copy of the GNU General Public License >> +along with this program; if not, write to the Free Software >> +Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA >> + >> +*/ >> + >> package com.bigdata.rdf.internal.constraints; >> >> import java.util.Map; >> @@ -6,8 +31,34 @@ >> import com.bigdata.bop.IBindingSet; >> import com.bigdata.rdf.error.SparqlTypeErrorException; >> >> + >> +/** >> + * This class implements 'AS' in a projection as found in both >> + * subselects and bind, e.g. >> + * <pre> >> + * { SUBSELECT (Foo(?x) AS ?y) ... >> + * { ... >> + * } >> + * } >> + * </pre> >> + * or >> + * <pre> >> + * BIND (Bar(?x) AS ?y) >> + * </pre> >> + * The bind fails if the expression evaluates to some value that is different from >> + * a pre-existing binding for the target variable. >> + * If the expr (Foo or Bar above) has an error then the specification is that no binding is done, >> + * but the unbound value is a success. >> + * >> + * The specification for this operation is found as >> + * <a href="http://www.w3.org/TR/2013/REC-sparql11-query-20130321/#defn_extend">Extend</a>. >> + * >> + */ >> +@SuppressWarnings({ "unchecked", "rawtypes" }) >> public class ProjectedConstraint extends com.bigdata.bop.constraint.Constraint { >> >> + private static final long serialVersionUID = 1L; >> + >> public ProjectedConstraint(final BOp[] args, final Map<String, Object> annotations) { >> super(args, annotations); >> } >> >> Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java >> =================================================================== >> --- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java 2013-10-03 15:18:56 UTC (rev 7432) >> +++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java 2013-10-03 15:45:34 UTC (rev 7433) >> @@ -83,7 +83,6 @@ >> import com.bigdata.btree.IRangeQuery; >> import com.bigdata.rdf.error.SparqlTypeErrorException; >> import com.bigdata.rdf.internal.IV; >> -import com.bigdata.rdf.internal.constraints.BindingConstraint; >> import com.bigdata.rdf.internal.constraints.CoalesceBOp; >> import com.bigdata.rdf.internal.constraints.ConditionalBind; >> import com.bigdata.rdf.internal.constraints.ContextNotAvailableException; >> @@ -3145,8 +3144,7 @@ >> assignmentNode.getVar(), >> assignmentNode.getValueExpression(), projection); >> >> - IConstraint c = projection ? new ProjectedConstraint(b) >> - : new BindingConstraint(b); >> + IConstraint c = new ProjectedConstraint(b); >> >> /* >> * We might have already materialized everything we need for this >> >> This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. >> >> >> ------------------------------------------------------------------------------ >> October Webinars: Code for Performance >> Free Intel webinars can help you accelerate application performance. >> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from >> the latest Intel processors and coprocessors. See abstracts and register > >> http://pubads.g.doubleclick.net/gampad/clk?id=60134791&iu=/4140/ostg.clktrk >> _______________________________________________ >> Bigdata-commit mailing list >> Big...@li... >> https://lists.sourceforge.net/lists/listinfo/bigdata-commit |
|
From: Jeremy J C. <jj...@sy...> - 2013-10-05 16:05:05
|
I can prioritize that Monday: we have meetings from 10.30 through to about 12.30 so unless it is easy it won't be until the afternoon Thanks for letting me know Jeremy J Carroll Principal Architect Syapse, Inc. On Oct 5, 2013, at 3:39 AM, Bryan Thompson <br...@sy...> wrote: > Jeremy, either this commit or the previous one broke two tests in TestDescribe related to the CBD support. Let me know if you have time to look at this. Bryan > > > Begin forwarded message: > >> From: "jer...@us..." <jer...@us...> >> Date: October 3, 2013 at 11:45:34 AM EDT >> To: "big...@li..." <big...@li...> >> Subject: [Bigdata-commit] SF.net SVN: bigdata:[7433] branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/ java/com/bigdata/rdf >> >> Revision: 7433 >> http://bigdata.svn.sourceforge.net/bigdata/?rev=7433&view=rev >> Author: jeremy_carroll >> Date: 2013-10-03 15:45:34 +0000 (Thu, 03 Oct 2013) >> Log Message: >> ----------- >> Deleted duplicate code, and added comments. >> >> Modified Paths: >> -------------- >> branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/ProjectedConstraint.java >> branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java >> >> Removed Paths: >> ------------- >> branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/BindingConstraint.java >> >> Deleted: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/BindingConstraint.java >> =================================================================== >> --- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/BindingConstraint.java 2013-10-03 15:18:56 UTC (rev 7432) >> +++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/BindingConstraint.java 2013-10-03 15:45:34 UTC (rev 7433) >> @@ -1,61 +0,0 @@ >> -/** >> - >> -Copyright (C) SYSTAP, LLC 2006-2011. All rights reserved. >> - >> -Contact: >> - SYSTAP, LLC >> - 4501 Tower Road >> - Greensboro, NC 27410 >> - lic...@bi... >> - >> -This program is free software; you can redistribute it and/or modify >> -it under the terms of the GNU General Public License as published by >> -the Free Software Foundation; version 2 of the License. >> - >> -This program is distributed in the hope that it will be useful, >> -but WITHOUT ANY WARRANTY; without even the implied warranty of >> -MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -GNU General Public License for more details. >> - >> -You should have received a copy of the GNU General Public License >> -along with this program; if not, write to the Free Software >> -Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA >> - */ >> -package com.bigdata.rdf.internal.constraints; >> - >> -import java.util.Map; >> - >> -import com.bigdata.bop.BOp; >> -import com.bigdata.bop.IBindingSet; >> -import com.bigdata.rdf.error.SparqlTypeErrorException; >> - >> -@SuppressWarnings({ "unchecked", "rawtypes" }) >> -public class BindingConstraint extends com.bigdata.bop.constraint.Constraint { >> - >> - private static final long serialVersionUID = 1L; >> - >> - public BindingConstraint(final BOp[] args, final Map<String, Object> annotations) { >> - super(args, annotations); >> - } >> - >> - public BindingConstraint(final BindingConstraint op) { >> - super(op); >> - } >> - >> - public BindingConstraint(ConditionalBind bind) { >> - super(new BOp[]{bind}, null); >> - } >> - >> - @Override >> - public boolean accept(final IBindingSet bindingSet) { >> - try{ >> - final Object result = ((ConditionalBind) get(0)).get(bindingSet); >> - >> - return result != null; >> - } catch (SparqlTypeErrorException stee) { >> - return true; >> - } >> - >> - } >> - >> -} >> >> Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/ProjectedConstraint.java >> =================================================================== >> --- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/ProjectedConstraint.java 2013-10-03 15:18:56 UTC (rev 7432) >> +++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/ProjectedConstraint.java 2013-10-03 15:45:34 UTC (rev 7433) >> @@ -1,3 +1,28 @@ >> +/* >> + >> +Copyright (C) SYSTAP, LLC 2006-2013. All rights reserved. >> + >> +Contact: >> + SYSTAP, LLC >> + 4501 Tower Road >> + Greensboro, NC 27410 >> + lic...@bi... >> + >> +This program is free software; you can redistribute it and/or modify >> +it under the terms of the GNU General Public License as published by >> +the Free Software Foundation; version 2 of the License. >> + >> +This program is distributed in the hope that it will be useful, >> +but WITHOUT ANY WARRANTY; without even the implied warranty of >> +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> +GNU General Public License for more details. >> + >> +You should have received a copy of the GNU General Public License >> +along with this program; if not, write to the Free Software >> +Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA >> + >> +*/ >> + >> package com.bigdata.rdf.internal.constraints; >> >> import java.util.Map; >> @@ -6,8 +31,34 @@ >> import com.bigdata.bop.IBindingSet; >> import com.bigdata.rdf.error.SparqlTypeErrorException; >> >> + >> +/** >> + * This class implements 'AS' in a projection as found in both >> + * subselects and bind, e.g. >> + * <pre> >> + * { SUBSELECT (Foo(?x) AS ?y) ... >> + * { ... >> + * } >> + * } >> + * </pre> >> + * or >> + * <pre> >> + * BIND (Bar(?x) AS ?y) >> + * </pre> >> + * The bind fails if the expression evaluates to some value that is different from >> + * a pre-existing binding for the target variable. >> + * If the expr (Foo or Bar above) has an error then the specification is that no binding is done, >> + * but the unbound value is a success. >> + * >> + * The specification for this operation is found as >> + * <a href="http://www.w3.org/TR/2013/REC-sparql11-query-20130321/#defn_extend">Extend</a>. >> + * >> + */ >> +@SuppressWarnings({ "unchecked", "rawtypes" }) >> public class ProjectedConstraint extends com.bigdata.bop.constraint.Constraint { >> >> + private static final long serialVersionUID = 1L; >> + >> public ProjectedConstraint(final BOp[] args, final Map<String, Object> annotations) { >> super(args, annotations); >> } >> >> Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java >> =================================================================== >> --- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java 2013-10-03 15:18:56 UTC (rev 7432) >> +++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java 2013-10-03 15:45:34 UTC (rev 7433) >> @@ -83,7 +83,6 @@ >> import com.bigdata.btree.IRangeQuery; >> import com.bigdata.rdf.error.SparqlTypeErrorException; >> import com.bigdata.rdf.internal.IV; >> -import com.bigdata.rdf.internal.constraints.BindingConstraint; >> import com.bigdata.rdf.internal.constraints.CoalesceBOp; >> import com.bigdata.rdf.internal.constraints.ConditionalBind; >> import com.bigdata.rdf.internal.constraints.ContextNotAvailableException; >> @@ -3145,8 +3144,7 @@ >> assignmentNode.getVar(), >> assignmentNode.getValueExpression(), projection); >> >> - IConstraint c = projection ? new ProjectedConstraint(b) >> - : new BindingConstraint(b); >> + IConstraint c = new ProjectedConstraint(b); >> >> /* >> * We might have already materialized everything we need for this >> >> This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. >> >> >> ------------------------------------------------------------------------------ >> October Webinars: Code for Performance >> Free Intel webinars can help you accelerate application performance. >> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from >> the latest Intel processors and coprocessors. See abstracts and register > >> http://pubads.g.doubleclick.net/gampad/clk?id=60134791&iu=/4140/ostg.clktrk >> _______________________________________________ >> Bigdata-commit mailing list >> Big...@li... >> https://lists.sourceforge.net/lists/listinfo/bigdata-commit |
|
From: Bryan T. <br...@sy...> - 2013-10-05 10:40:36
|
Jeremy, either this commit or the previous one broke two tests in TestDescribe related to the CBD support. Let me know if you have time to look at this. Bryan Begin forwarded message: From: "jer...@us...<mailto:jer...@us...>" <jer...@us...<mailto:jer...@us...>> Date: October 3, 2013 at 11:45:34 AM EDT To: "big...@li...<mailto:big...@li...>" <big...@li...<mailto:big...@li...>> Subject: [Bigdata-commit] SF.net<http://SF.net> SVN: bigdata:[7433] branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/ java/com/bigdata/rdf Revision: 7433 http://bigdata.svn.sourceforge.net/bigdata/?rev=7433&view=rev Author: jeremy_carroll Date: 2013-10-03 15:45:34 +0000 (Thu, 03 Oct 2013) Log Message: ----------- Deleted duplicate code, and added comments. Modified Paths: -------------- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/ProjectedConstraint.java branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java Removed Paths: ------------- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/BindingConstraint.java Deleted: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/BindingConstraint.java =================================================================== --- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/BindingConstraint.java 2013-10-03 15:18:56 UTC (rev 7432) +++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/BindingConstraint.java 2013-10-03 15:45:34 UTC (rev 7433) @@ -1,61 +0,0 @@ -/** - -Copyright (C) SYSTAP, LLC 2006-2011. All rights reserved. - -Contact: - SYSTAP, LLC - 4501 Tower Road - Greensboro, NC 27410 - lic...@bi...<mailto:lic...@bi...> - -This program is free software; you can redistribute it and/or modify -it under the terms of the GNU General Public License as published by -the Free Software Foundation; version 2 of the License. - -This program is distributed in the hope that it will be useful, -but WITHOUT ANY WARRANTY; without even the implied warranty of -MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -GNU General Public License for more details. - -You should have received a copy of the GNU General Public License -along with this program; if not, write to the Free Software -Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA - */ -package com.bigdata.rdf.internal.constraints; - -import java.util.Map; - -import com.bigdata.bop.BOp; -import com.bigdata.bop.IBindingSet; -import com.bigdata.rdf.error.SparqlTypeErrorException; - -@SuppressWarnings({ "unchecked", "rawtypes" }) -public class BindingConstraint extends com.bigdata.bop.constraint.Constraint { - - private static final long serialVersionUID = 1L; - - public BindingConstraint(final BOp[] args, final Map<String, Object> annotations) { - super(args, annotations); - } - - public BindingConstraint(final BindingConstraint op) { - super(op); - } - - public BindingConstraint(ConditionalBind bind) { - super(new BOp[]{bind}, null); - } - - @Override - public boolean accept(final IBindingSet bindingSet) { - try{ - final Object result = ((ConditionalBind) get(0)).get(bindingSet); - - return result != null; - } catch (SparqlTypeErrorException stee) { - return true; - } - - } - -} Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/ProjectedConstraint.java =================================================================== --- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/ProjectedConstraint.java 2013-10-03 15:18:56 UTC (rev 7432) +++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/internal/constraints/ProjectedConstraint.java 2013-10-03 15:45:34 UTC (rev 7433) @@ -1,3 +1,28 @@ +/* + +Copyright (C) SYSTAP, LLC 2006-2013. All rights reserved. + +Contact: + SYSTAP, LLC + 4501 Tower Road + Greensboro, NC 27410 + lic...@bi...<mailto:lic...@bi...> + +This program is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation; version 2 of the License. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program; if not, write to the Free Software +Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +*/ + package com.bigdata.rdf.internal.constraints; import java.util.Map; @@ -6,8 +31,34 @@ import com.bigdata.bop.IBindingSet; import com.bigdata.rdf.error.SparqlTypeErrorException; + +/** + * This class implements 'AS' in a projection as found in both + * subselects and bind, e.g. + * <pre> + * { SUBSELECT (Foo(?x) AS ?y) ... + * { ... + * } + * } + * </pre> + * or + * <pre> + * BIND (Bar(?x) AS ?y) + * </pre> + * The bind fails if the expression evaluates to some value that is different from + * a pre-existing binding for the target variable. + * If the expr (Foo or Bar above) has an error then the specification is that no binding is done, + * but the unbound value is a success. + * + * The specification for this operation is found as + * <a href="http://www.w3.org/TR/2013/REC-sparql11-query-20130321/#defn_extend">Extend</a>. + * + */ +@SuppressWarnings({ "unchecked", "rawtypes" }) public class ProjectedConstraint extends com.bigdata.bop.constraint.Constraint { + private static final long serialVersionUID = 1L; + public ProjectedConstraint(final BOp[] args, final Map<String, Object> annotations) { super(args, annotations); } Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java =================================================================== --- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java 2013-10-03 15:18:56 UTC (rev 7432) +++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/eval/AST2BOpUtility.java 2013-10-03 15:45:34 UTC (rev 7433) @@ -83,7 +83,6 @@ import com.bigdata.btree.IRangeQuery; import com.bigdata.rdf.error.SparqlTypeErrorException; import com.bigdata.rdf.internal.IV; -import com.bigdata.rdf.internal.constraints.BindingConstraint; import com.bigdata.rdf.internal.constraints.CoalesceBOp; import com.bigdata.rdf.internal.constraints.ConditionalBind; import com.bigdata.rdf.internal.constraints.ContextNotAvailableException; @@ -3145,8 +3144,7 @@ assignmentNode.getVar(), assignmentNode.getValueExpression(), projection); - IConstraint c = projection ? new ProjectedConstraint(b) - : new BindingConstraint(b); + IConstraint c = new ProjectedConstraint(b); /* * We might have already materialized everything we need for this This was sent by the SourceForge.net<http://SourceForge.net> collaborative development platform, the world's largest Open Source development site. ------------------------------------------------------------------------------ October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register > http://pubads.g.doubleclick.net/gampad/clk?id=60134791&iu=/4140/ostg.clktrk _______________________________________________ Bigdata-commit mailing list Big...@li...<mailto:Big...@li...> https://lists.sourceforge.net/lists/listinfo/bigdata-commit |
|
From: Jeremy J C. <jj...@sy...> - 2013-10-05 00:01:57
|
I don't think this is through put related. The test data is fairly small, and the query has some patterns that cause a large number of blocking buffers to be used. The test shows that a large amount of time is not used at all, i.e. no cpu cycles, just the wall clock ticking, with the NSPIN value of 100, and much better CPU utilization, and much better better wall clock times with the value at 100000. It doesn't tell us that much about why it is going slow, just that the context switch is expensive. I have repeated these results with multiple concurrent queries - as far as I can tell the higher NSPIN value (at least on the iterator) is simply more performant (on my box) Jeremy J Carroll Principal Architect Syapse, Inc. On Oct 4, 2013, at 3:17 PM, Bryan Thompson <br...@sy...> wrote: > Is this a thorough put oriented test? Can you add a variant that uses an array based and linked list based queue or deque? With a reasonable workload we could characterize the time lost in terms of other implementations and make a case for changing the implementation? > > The easiest path to that goal is to establish shared interfaces. This might be possible in a limited test suite but will be harder at a more general layer. E.g., at the level of running bsbm or s different benchmark. Which is of course whay matters. > > Let me suggest that we create a system propery for nspin that will let you override it and gain experience with that without pushing a change to everyone. This would also make it easier to run different workloads ib different machines and observe the impact. > > Another key parameter for through put is the target chunk capacity. Maybe we could create an ergonomic system property to shift the bias between single query throughput and concurrent query throughput. Once defined we could shift that property automatically based on the jvm gc workload metrics and concurrent query metrics. > > B > > > > > > -------- Original message -------- > From: bigdata® <no...@so...> > Date: 10/04/2013 5:53 PM (GMT-05:00) > To: > Subject: Re: [bigdata®] #740: performance impact of NSPIN > > > #740: performance impact of NSPIN > ----------------------------+----------------------------------------------- > Reporter: jeremy_carroll | Owner: jeremy_carroll > Type: defect | Status: accepted > Priority: major | Milestone: > Component: Bigdata SAIL | Version: BIGDATA_RELEASE_1_2_2 > Keywords: | > ----------------------------+----------------------------------------------- > > Comment(by jeremy_carroll): > > I have added a test case in r7434 > com.bigdata.rdf.sparql.ast.eval.TestBlockingBufferPerformance740 > > On my system with the default value of 100 for NPSIN, the test fails with > the message: > > junit.framework.AssertionFailedError: Performance was poor: 27% [< 80%] > > > Setting the value of NSPIN to 100000, I get the message > BlockingBuffer: performance was good: 184% > (on stdout) > > I suggest a next step is to reproduce this on other h/w > > -- > Ticket URL: <http://sourceforge.net/apps/trac/bigdata/ticket/740#comment:9> > bigdata® <http://www.bigdata.com/blog> > bigdata® is a scale-out storage and computing fabric supporting optional transactions, very high concurrency, and very high aggregate IO rates. |
|
From: Jeremy J C. <jj...@sy...> - 2013-10-04 22:47:42
|
740 NSPIN performance, BlockingBuffer
added test case. Please run on your hardware, both with the current setting of NSPIN and with NSPIN = 100000
(I have only run on a Mac with an SSD and a quad core processor … I feel that this could be quite h/w specific)
The test produces a number a percentage utilization - if you run the test please let me know what value you get.
747 ASTBottomUpOptimizer handleFiltersWithVariablesNotInScope
added comment to trac item suggesting a rewrite to this method
739 BIND and optional property path issue
not started
725 FILTER EXISTS no progress
736 MIN issue - ignored
(I have closed some items too!)
I think I probably need to continue working on some of these correctness issues (725 and 747 all feel like they may be related somewhat …; I think I have seen 739 also with ALPP)
I do have other syapse work to do, so I will make a judgment call early next week as to how much time I can spend on bigdata issues
Jeremy J Carroll
Principal Architect
Syapse, Inc.
|
|
From: Bryan T. <br...@sy...> - 2013-10-04 22:18:36
|
Is this a thorough put oriented test? Can you add a variant that uses an array based and linked list based queue or deque? With a reasonable workload we could characterize the time lost in terms of other implementations and make a case for changing the implementation?
The easiest path to that goal is to establish shared interfaces. This might be possible in a limited test suite but will be harder at a more general layer. E.g., at the level of running bsbm or s different benchmark. Which is of course whay matters.
Let me suggest that we create a system propery for nspin that will let you override it and gain experience with that without pushing a change to everyone. This would also make it easier to run different workloads ib different machines and observe the impact.
Another key parameter for through put is the target chunk capacity. Maybe we could create an ergonomic system property to shift the bias between single query throughput and concurrent query throughput. Once defined we could shift that property automatically based on the jvm gc workload metrics and concurrent query metrics.
B
-------- Original message --------
From: bigdata® <no...@so...>
Date: 10/04/2013 5:53 PM (GMT-05:00)
To:
Subject: Re: [bigdata®] #740: performance impact of NSPIN
#740: performance impact of NSPIN
----------------------------+-----------------------------------------------
Reporter: jeremy_carroll | Owner: jeremy_carroll
Type: defect | Status: accepted
Priority: major | Milestone:
Component: Bigdata SAIL | Version: BIGDATA_RELEASE_1_2_2
Keywords: |
----------------------------+-----------------------------------------------
Comment(by jeremy_carroll):
I have added a test case in r7434
com.bigdata.rdf.sparql.ast.eval.TestBlockingBufferPerformance740
On my system with the default value of 100 for NPSIN, the test fails with
the message:
junit.framework.AssertionFailedError: Performance was poor: 27% [< 80%]
Setting the value of NSPIN to 100000, I get the message
BlockingBuffer: performance was good: 184%
(on stdout)
I suggest a next step is to reproduce this on other h/w
--
Ticket URL: <http://sourceforge.net/apps/trac/bigdata/ticket/740#comment:9>
bigdata® <http://www.bigdata.com/blog>
bigdata® is a scale-out storage and computing fabric supporting optional transactions, very high concurrency, and very high aggregate IO rates.
|
|
From: Jeremy J C. <jj...@sy...> - 2013-10-03 15:47:32
|
Done - I decided that ProjectedConstraint was a better name than BindingConstraint and kept that, and added documentation: boilerplate © that was missing, and a link to the spec and two syntax examples (I am procrastinating before getting on to the next bug … I suspect it is that buffer one, where Syapse is running with a different NSPIN value from everyone else …) Jeremy J Carroll Principal Architect Syapse, Inc. On Oct 3, 2013, at 8:30 AM, Bryan Thompson <br...@sy...> wrote: > Jeremy, > > Just looked at the code. Seems like a no brainer. Go ahead. > > Bryan > > From: Jeremy Carroll <jj...@sy...> > Date: Thursday, October 3, 2013 11:20 AM > To: "Big...@li..." <Big...@li...> > Subject: Re: [Bigdata-developers] trac748 fix ... > > I have committed the test cases and the fix to the ProjectedConstraint, I will make the IMO better fix of deleting ProjectConstraint as below if you would like me to. > r7432 > > Jeremy J Carroll > Principal Architect > Syapse, Inc. > > > > On Oct 2, 2013, at 6:33 PM, Jeremy J Carroll <jj...@sy...> wrote: > >> >> 748 concerned subselects. >> https://sourceforge.net/apps/trac/bigdata/ticket/748 >> >> The fix is minor fixes to ProjectedConstraint which crucially ignored the potentially null return from ConditionalBind .get(). It should be exactly the same as BindingConstraint. >> >> I have tests and code ready to commit - however, in my view a better fix is to delete the class ProjectedConstraint and change its one usage at: >> lin 3148 of AST2BopUtility >> >> i.e. from >> IConstraint c = projection ? new ProjectedConstraint(b) >> : new BindingConstraint(b); >> >> to >> >> IConstraint c = new BindingConstraint(b); >> >> >> Please advise. >> >> Jeremy J Carroll >> Principal Architect >> Syapse, Inc. >> >> >> > |
27 messages has been excluded from this view by a project administrator.
