Re: [Bigdata-developers] Subquery wildcard projections not rewritten

[Bryan T. <br...@sy...>]

Lee,

I can replicate the problem with your query (as given above) against the
sparql end point.

Can you state the SPARQL that you are trying to model with this unit test?
It appears to be not query the same as your SPARQL query above.  I would
like to make sure that it is being translated correctly into the AST.  I
can then look at the expected AST and work backwards and see if I believe
that the test shows the problem.

Thanks,
Bryan

----
Bryan Thompson
Chief Scientist & Founder
SYSTAP, LLC
4501 Tower Road
Greensboro, NC 27410
br...@sy...
http://blazegraph.com
http://blog.bigdata.com <http://bigdata.com>
http://mapgraph.io

Blazegraph™ <http://www.blazegraph.com/> is our ultra high-performance
graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints
APIs.  MapGraph™ <http://www.systap.com/mapgraph> is our disruptive new
technology to use GPUs to accelerate data-parallel graph analytics.

CONFIDENTIALITY NOTICE:  This email and its contents and attachments are
for the sole use of the intended recipient(s) and are confidential or
proprietary to SYSTAP. Any unauthorized review, use, disclosure,
dissemination or copying of this email or its contents or attachments is
prohibited. If you have received this communication in error, please notify
the sender by reply email and permanently delete all copies of the email
and its contents and attachments.

On Tue, Apr 21, 2015 at 11:07 AM, Lee Kitching <le...@sw...> wrote:

> Hi Bryan,
>
> We allow users to enter their own SPARQL queries and wrap them to do
> things like pagination so unfortunately we cannot just re-write our queries
> to do the expansion manually.
> I applied the fix detailed in the ticket and it fixes the for the query I
> provided, however it fails to rewrite the following query:
>
> SELECT (COUNT(*) as ?c) {
>   SELECT * {
>     SELECT * WHERE { ?s ?p ?o }
>   } LIMIT 21 OFFSET 0
> }
>
> I attempted to debug the issue, and it seems to re-write the * projection
> in the inner-most subquery but not the subquery with the limit and offset.
> I created a test based on the
> existing tests:
>
> public void test_wildcardProjectionOptimizer03() {
>
>         /*
>        * Note: DO NOT share structures in this test!!!!
>        */
>         final IBindingSet[] bsets = new IBindingSet[] {};
>
>         // The source AST.
>         final QueryRoot given = new QueryRoot(QueryType.SELECT);
>         {
>             final SubqueryRoot selectQuery = new
> SubqueryRoot(QueryType.SELECT);
>             {
>                 final JoinGroupNode whereClause1 = new JoinGroupNode();
>                 final StatementPatternNode spoPattern = new
> StatementPatternNode(new VarNode("s"), new VarNode("p"), new VarNode("o"),
> null, Scope.DEFAULT_CONTEXTS);
>                 whereClause1.addChild(spoPattern);
>
>                 final ProjectionNode p = new ProjectionNode();
>                 p.addProjectionVar(new VarNode("*"));
>                 selectQuery.setProjection(p);
>                 selectQuery.setWhereClause(whereClause1);
>             }
>
>             final SubqueryRoot sliceQuery = new
> SubqueryRoot(QueryType.SELECT);
>             {
>                 final ProjectionNode p = new ProjectionNode();
>                 p.addProjectionVar(new VarNode("*"));
>                 sliceQuery.setProjection(p);
>
>                 final JoinGroupNode whereClause = new JoinGroupNode();
>                 whereClause.addChild(selectQuery);
>
>                 sliceQuery.setSlice(new SliceNode(0, 21));
>             }
>
>             final FunctionNode countNode = new FunctionNode(
>                     FunctionRegistry.COUNT,
>                     Collections.EMPTY_MAP,
>                     new VarNode("*"));
>
>             final ProjectionNode countProjection = new ProjectionNode();
>             countProjection.addProjectionExpression(new AssignmentNode(new
> VarNode("c"), countNode));
>
>             JoinGroupNode countWhere = new JoinGroupNode();
>             countWhere.addChild(sliceQuery);
>
>             given.setProjection(countProjection);
>             given.setWhereClause(countWhere);
>         }
>
>         final QueryRoot expected = new QueryRoot(QueryType.SELECT);
>         {
>             final SubqueryRoot selectQuery = new
> SubqueryRoot(QueryType.SELECT);
>             {
>                 final JoinGroupNode whereClause1 = new JoinGroupNode();
>                 final StatementPatternNode spoPattern = new
> StatementPatternNode(new VarNode("s"), new VarNode("p"), new VarNode("o"),
> null, Scope.DEFAULT_CONTEXTS);
>                 whereClause1.addChild(spoPattern);
>
>                 final ProjectionNode p = new ProjectionNode();
>                 p.addProjectionVar(new VarNode("s"));
>                 p.addProjectionVar(new VarNode("p"));
>                 p.addProjectionVar(new VarNode("o"));
>                 selectQuery.setProjection(p);
>                 selectQuery.setWhereClause(whereClause1);
>             }
>
>             final SubqueryRoot sliceQuery = new
> SubqueryRoot(QueryType.SELECT);
>             {
>                 final ProjectionNode p = new ProjectionNode();
>                 p.addProjectionVar(new VarNode("s"));
>                 p.addProjectionVar(new VarNode("p"));
>                 p.addProjectionVar(new VarNode("o"));
>
>                 sliceQuery.setProjection(p);
>
>                 final JoinGroupNode whereClause = new JoinGroupNode();
>                 whereClause.addChild(selectQuery);
>
>                 sliceQuery.setSlice(new SliceNode(0, 21));
>             }
>
>             final FunctionNode countNode = new FunctionNode(
>                     FunctionRegistry.COUNT,
>                     Collections.EMPTY_MAP,
>                     new VarNode("*"));
>
>             final ProjectionNode countProjection = new ProjectionNode();
>             countProjection.addProjectionExpression(new AssignmentNode(new
> VarNode("c"), countNode));
>
>             JoinGroupNode countWhere = new JoinGroupNode();
>             countWhere.addChild(sliceQuery);
>
>             expected.setProjection(countProjection);
>             expected.setWhereClause(countWhere);
>         }
>
>         final IASTOptimizer rewriter = new
> ASTWildcardProjectionOptimizer();
>
>         final IQueryNode actual = rewriter.optimize(null/* AST2BOpContext
> */,
>                 given/* queryNode */, bsets);
>
>         assertSameAST(expected, actual);
>
>     }
>
> however I am having some problems running the tests locally so I don't
> know if it accurately models the situation.
>
> Thanks
>
>
>
> On Mon, Apr 20, 2015 at 9:05 PM, Bryan Thompson <br...@sy...> wrote:
>
>> Lee,
>>
>> I've updated the ticket with the code changes and the test changes.
>> Please try this out and let me know if you have any problems.
>>
>> Thanks,
>> Bryan
>>
>> ----
>> Bryan Thompson
>> Chief Scientist & Founder
>> SYSTAP, LLC
>> 4501 Tower Road
>> Greensboro, NC 27410
>> br...@sy...
>> http://blazegraph.com
>> http://blog.bigdata.com <http://bigdata.com>
>> http://mapgraph.io
>>
>> Blazegraph™ <http://www.blazegraph.com/> is our ultra high-performance
>> graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints
>> APIs.  MapGraph™ <http://www.systap.com/mapgraph> is our disruptive new
>> technology to use GPUs to accelerate data-parallel graph analytics.
>>
>> CONFIDENTIALITY NOTICE:  This email and its contents and attachments are
>> for the sole use of the intended recipient(s) and are confidential or
>> proprietary to SYSTAP. Any unauthorized review, use, disclosure,
>> dissemination or copying of this email or its contents or attachments is
>> prohibited. If you have received this communication in error, please notify
>> the sender by reply email and permanently delete all copies of the email
>> and its contents and attachments.
>>
>> On Mon, Apr 20, 2015 at 1:20 PM, Lee Kitching <le...@sw...> wrote:
>>
>>> Hi,
>>>
>>> We are currently evaluating using Blazegraph as our rdf database and
>>> have run in the issue described at http://trac.bigdata.com/ticket/757.
>>> The below query causes the AssertionError to be thrown:
>>>
>>> SELECT (COUNT(*) as ?c) {
>>>   SELECT ?uri ?graph where {
>>>           {
>>>             SELECT * WHERE {
>>>               GRAPH ?graph {
>>>                 ?uri a <http://object> .
>>>                 ?uri <http://purl.org/dc/terms/title> ?title .
>>>               }
>>>               MINUS {
>>>                 ?uri a <http://other>
>>>               }
>>>             }
>>>             ORDER BY ?title
>>>           }
>>>         }
>>> }
>>>
>>> Some debugging shows that the error is caused by the
>>> ASTWildcardProjectionOptimizer failing to recurse into the subqueries to
>>> rewrite the * projection. However this recursion is implemented in the
>>> BOpUtility.postOrderIterator(BOp) method - this method uses the argIterator
>>> to
>>> find child operators and therefore only visits children for nodes with
>>> an arity > 0.
>>>
>>> The root query node for the above query has an empty 'args' collection
>>> and all the associated components of the top-level query are stored in the
>>> annotations map. It looks like the iterator should search through the
>>> annotations rather than the args for query nodes.
>>>
>>> As there are a lot of implementations of the BOp interface, it seems
>>> that changing the postOrderIterator2(BOp) method is unlikely to be the
>>> correct fix. It seems that either the AST query nodes should override the
>>> arity() function to return the count of the annotations map, or the
>>> ASTWildcardProjectionOptimizer should use its own iterator for the nodes of
>>> the query. The latter option would be the least impactful change but I am
>>> not familiar with the codebase to understand the correct fix.
>>>
>>> Any help in resolving the issue would be appreciated.
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
>>> Develop your own process in accordance with the BPMN 2 standard
>>> Learn Process modeling best practices with Bonita BPM through live
>>> exercises
>>> http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual-
>>> event?utm_
>>> source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
>>> _______________________________________________
>>> Bigdata-developers mailing list
>>> Big...@li...
>>> https://lists.sourceforge.net/lists/listinfo/bigdata-developers
>>>
>>>
>>
>