Bigdata graph issues

Help
feugen24
2014-03-05
2014-04-07
  • feugen24
    feugen24
    2014-03-05

    Hi,
    I've posted some tickets on some graph/filter/union problems (831,832,833,835) few days ago.
    Those look like some major problems with the db(especially 832) and if they are valid i'm wondering how other developers work with graphs because the problems should show up pretty quick, unless most users don't use quads.

    I've also tested the issues on other triple store just to validate them, and on older version of bigdata 1.2.3 (also has the problems).

    So maybe you can take a look just to check if they are valid.

    Thanks

     
  • Bryan Thompson
    Bryan Thompson
    2014-04-07

    Hello,

    Can you please verify that you can now attach items to the ticket?

    I can view the ticket but i can't edit it so i'm attaching to this post prop file
    that has initial config + properties for namespace in last part.

    If so, please attach the data from above to the ticket.

    We are digging out from under and hope to address some of these issues this month.

    Thanks,
    Bryan

     
  • Peter Amstutz
    Peter Amstutz
    2014-04-07

    Hi Bryan,

    I hope you can address the issues with UNION and FILTER soon. I am hoping to use Bigdata for the Arvados (http://arvados.org) metadata database and have already invested substantial time working out how to build a permissions model. However, I can't build my permission rules without UNION, and I don't think I can ask users to go without FILTER, so this is a showstopper for me.

     
    • Bryan Thompson
      Bryan Thompson
      2014-04-07

      Peter,

      We've been behind on the open source support issues due to a surge in customer feature development and services. I certainly will get to this as soon as possible. If you would like to help, maybe you can drill down into the query and see if you can figure out why it is failing. That would make it easier for us to identify a fix.

      My guess is that the filter is being silently dropped during a rewrite of the UNION. This would be one of the AST optimizers. If you look at AST2BOpUtility, you can see that it runs the AST optimizers here:

          // Run the AST query rewrites / query optimizers.
      
          final QueryRoot optimizedQuery = (QueryRoot) ctx.optimizers.optimize(ctx,
      
                  originalQuery, bindingSets);
      

      Probably one of these rewrite stages is responsible for dropping the FILTER. It would help if you could (a) identify whether the FILTER is being dropped by a query rewrite stage; and (b) which one is dropping the FILTER.

      Thanks,
      Bryan

      From: Peter Amstutz tetron@users.sf.net<mailto:tetron@users.sf.net>
      Reply-To: "[bigdata:discussion]" 676946@discussion.bigdata.p.re.sf.net<mailto:676946@discussion.bigdata.p.re.sf.net>
      Date: Monday, April 7, 2014 9:42 AM
      To: "[bigdata:discussion]" 676946@discussion.bigdata.p.re.sf.net<mailto:676946@discussion.bigdata.p.re.sf.net>
      Subject: [bigdata:discussion] Bigdata graph issues

      Hi Bryan,

      I hope you can address the issues with UNION and FILTER soon. I am hoping to use Bigdata for the Arvados (http://arvados.org) metadata database and have already invested substantial time working out how to build a permissions model. However, I can't build my permission rules without UNION, and I don't think I can ask users to go without FILTER, so this is a showstopper for me.


      Bigdata graph issueshttps://sourceforge.net/p/bigdata/discussion/676946/thread/ce881ada/?limit=25#f86a


      Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/bigdata/discussion/676946/

      To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

       
      Attachments
      • Bryan Thompson
        Bryan Thompson
        2014-04-07

        Peter,

        Also, using the NSS [x] Explain should show you what is happening in the query translation phase.

        It does not show the step by step rewrite of the query, but it does show the before and after rewrite of the query. You can enable logging on the ASTOptimizerList to see the before/after of each step in the query rewrite:

        public class ASTOptimizerList extends LinkedList<IASTOptimizer> implements

            IASTOptimizer {
        
        private static final Logger log = Logger.getLogger(ASTOptimizerList.class);
        

        ….

        @Override
        
        public IQueryNode optimize(final AST2BOpContext context,
        
                IQueryNode queryNode, final IBindingSet[] bindingSets) {
        
            if (log.isDebugEnabled())
        
                log.debug("Original AST:\n" + queryNode);
        
            // Avoid side-effects on the original AST!
        
            queryNode = (IQueryNode) BOpUtility.deepCopy((BOp) queryNode);
        
            for (IASTOptimizer opt : this) {
        
                if (log.isInfoEnabled())
        
                    log.info("Applying: " + opt);
        
                queryNode = opt.optimize(context, queryNode, bindingSets);
        
                if (queryNode == null)
        
                    throw new AssertionError("Optimized discarded query: " + opt);
        
                if (log.isDebugEnabled())
        
                    log.debug("Rewritten AST:\n" + queryNode);
        
            }
        
            return queryNode;
        
        }
        

        Thanks,
        Bryan

        From: Bryan Thompson thompsonbry@users.sf.net<mailto:thompsonbry@users.sf.net>
        Reply-To: "[bigdata:discussion]" 676946@discussion.bigdata.p.re.sf.net<mailto:676946@discussion.bigdata.p.re.sf.net>
        Date: Monday, April 7, 2014 9:55 AM
        To: "[bigdata:discussion]" 676946@discussion.bigdata.p.re.sf.net<mailto:676946@discussion.bigdata.p.re.sf.net>
        Subject: [bigdata:discussion] Re: Bigdata graph issues

        Peter,

        We've been behind on the open source support issues due to a surge in customer feature development and services. I certainly will get to this as soon as possible. If you would like to help, maybe you can drill down into the query and see if you can figure out why it is failing. That would make it easier for us to identify a fix.

        My guess is that the filter is being silently dropped during a rewrite of the UNION. This would be one of the AST optimizers. If you look at AST2BOpUtility, you can see that it runs the AST optimizers here:

        // Run the AST query rewrites / query optimizers.
        
        final QueryRoot optimizedQuery = (QueryRoot) ctx.optimizers.optimize(ctx,
        
                originalQuery, bindingSets);
        

        Probably one of these rewrite stages is responsible for dropping the FILTER. It would help if you could (a) identify whether the FILTER is being dropped by a query rewrite stage; and (b) which one is dropping the FILTER.

        Thanks,
        Bryan

        From: Peter Amstutz tetron@users.sf.nettetron@users.sf.net<mailto:tetron@users.sf.net<mailto:tetron@users.sf.net>
        Reply-To: "[bigdata:discussion]" 676946@discussion.bigdata.p.re.sf.net676946@discussion.bigdata.p.re.sf.net<mailto:676946@discussion.bigdata.p.re.sf.net<mailto:676946@discussion.bigdata.p.re.sf.net>
        Date: Monday, April 7, 2014 9:42 AM
        To: "[bigdata:discussion]" 676946@discussion.bigdata.p.re.sf.net676946@discussion.bigdata.p.re.sf.net<mailto:676946@discussion.bigdata.p.re.sf.net<mailto:676946@discussion.bigdata.p.re.sf.net>
        Subject: [bigdata:discussion] Bigdata graph issues

        Hi Bryan,

        I hope you can address the issues with UNION and FILTER soon. I am hoping to use Bigdata for the Arvados (http://arvados.org) metadata database and have already invested substantial time working out how to build a permissions model. However, I can't build my permission rules without UNION, and I don't think I can ask users to go without FILTER, so this is a showstopper for me.


        Bigdata graph issueshttps://sourceforge.net/p/bigdata/discussion/676946/thread/ce881ada/?limit=25#f86a


        Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/bigdata/discussion/676946/

        To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/


        Bigdata graph issueshttp://sourceforge.net/p/bigdata/discussion/676946/thread/ce881ada/?limit=25#f86a/b540


        Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/bigdata/discussion/676946/

        To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

         
  • Mike Personick
    Mike Personick
    2014-04-07

    Peter,

    Why don't you try this:

    prefix xsd:  <http://www.w3.org/2001/XMLSchema#>SELECT * 
    where { 
      ?user <http://arvados.org/schema/api_token> <token:ckeddwmn3586gsdadbzxosz331crwpfxax58r0k8iq299nmwf> 
      {
         ?user <http://arvados.org/schema/user_is_admin> false .
         ?s ?p ?o . 
         FILTER strStarts(str(?s), "http://arvados.org/schema/modified") .
         ?user <http://arvados.org/schema/permission/can_read> ?s .
      } union {
         ?user <http://arvados.org/schema/user_is_admin> true .
         ?s ?p ?o .
         FILTER strStarts(str(?s), "http://arvados.org/schema/modified") .
      }   
    } 
    LIMIT 100
    

    -Mike