Re: [Virtuoso-users] Strange use of indices in Virtuoso 7.0.0

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Hi Hugh,

the output of "virtuoso-t -?" is the following:

Virtuoso Open Source Edition (Column Store) (multi threaded)
Version 7.0.0-rc2.3203-pthreads as of Aug  6 2013
Compiled for Linux (x86_64-unknown-linux-gnu)

As for the "git status", I don't have the clone handy but we downloaded
the build from here:

https://github.com/openlink/virtuoso-opensource/archive/48f0ef879b913c5d3b306c1f83390079c5416fe6.tar.gz

Hope this helps.

Nicola

Il 15/10/2013 04:47, Hugh Williams ha scritto:
> Hi Nicola,
> 
> What does the following report as the version of your build:
> 
> ./virtuoso-t -?
> 
> and for your build tree what does the "git status" command return when
> run against your build tree, so we can confirm which build is being used ...
> 
> Best Regards
> Hugh Williams
> Professional Services
> OpenLink Software, Inc.      //              http://www.openlinksw.com/
> Weblog   -- http://www.openlinksw.com/blogs/
> LinkedIn -- http://www.linkedin.com/company/openlink-software/
> Twitter  -- http://twitter.com/OpenLink
> Google+  -- http://plus.google.com/100570109519069333827/
> Facebook -- http://www.facebook.com/OpenLinkSoftware
> Universal Data Access, Integration, and Management Technology Providers
> 
> On 14 Oct 2013, at 13:03, Nicola Vitucci <nic...@gm...
> <mailto:nic...@gm...>> wrote:
> 
>> Hi Hugh,
>>
>> the rc2 version we used is 48f0ef879b913c5d3b306c1f83390079c5416fe6 ,
>> downloaded from <https://github.com/openlink/virtuoso-opensource>.
>>
>> I also noticed that, if I load the same data on 7.0.0 from scratch, the
>> query runs "fast". Is there maybe a problem in how indices are
>> interpreted in the two versions, so that a database build using rc2
>> cannot be used correctly on 7.0.0?
>>
>> Thanks,
>>
>> Nicola
>>
>> Il 13/10/2013 03:28, Hugh Williams ha scritto:
>>> Hi Nicola,
>>>
>>> If the profile function is not working for this query, which i have also
>>> been able to recreate locally and reported to be fixed, then the explain
>>> output is the next best form of  profiling.
>>>
>>> I tried running explain against the latest develop/7 got branch and it
>>> gave the following:
>>>
>>> SQL> explain('SPARQL PREFIX int: <namespace#> CONSTRUCT {?s int:new_prop
>>> ?n} WHERE {?s <prop> ?c . ?c rdfs:label ?n}');
>>> REPORT
>>> VARCHAR
>>> _______________________________________________________________________________
>>>
>>> {
>>>
>>> Precode:
>>>      0: {
>>>
>>> Precode:
>>>      0: vector := Call vector ( 1 ,  0 ,  3 ,  #new_prop ,  1 ,  1 )
>>>      5: vector := Call vector (vector)
>>>      10: vector := Call vector ()
>>>      15: BReturn 0
>>> { fork
>>> RDF_QUAD_POGS        40 rows(s_1_7_t2.S, s_1_7_t2.O)
>>> inlined  P =  #label
>>> RDF_QUAD    0.0018 rows(s_1_7_t1.S)
>>> inlined  P =  #prop  O = k_s_1_7_t2.S
>>>
>>> After code:
>>>      0: __ro2lo := Call __ro2lo (s_1_7_t2.O)
>>>      5: vector := Call vector (s_1_7_t1.S, __ro2lo)
>>>      10: if ($47 "user_aggr_notfirst" =  1 ) then 25 else 14 unkn 14
>>>      14: $47 "user_aggr_notfirst" :=  := artm  1
>>>      18: user_aggr_ret := Call DB.DBA.SPARQL_CONSTRUCT_INIT ($48
>>> "user_aggr_env")
>>>      25: user_aggr_ret := Call DB.DBA.SPARQL_CONSTRUCT_ACC ($48
>>> "user_aggr_env", vector, vector, vector,  1 )
>>>      32: BReturn 0
>>> }
>>> skip node  1  <none> set_ctr
>>>
>>> After code:
>>>      0: callret-0 := Call DB.DBA.SPARQL_CONSTRUCT_FIN ($48
>>> "user_aggr_env")
>>>      7: BReturn 0
>>> Subquery Select(callret-0)
>>> }
>>>
>>>      8:  <none>  := Call DB.DBA.RDF_DICT_OF_TRIPLES_TO_THREE_COLS ($100
>>> "scalar")
>>>      15: BReturn 0
>>> END Node
>>> }
>>>
>>> 37 Rows. -- 1 msec.
>>> SQL>
>>>
>>> Note the use of the "RDF_QUAD_POGS" index, which was used in your 7.0.0
>>> build but not in the "rc2" build explain output you perform.
>>>
>>> Which brings me to another question, what is the "rc2" build you refer
>>> to, as building from the latest git develop/7 branch my version is:
>>>
>>> ./virtuoso-t -?
>>> Virtuoso Open Source Edition (Column Store) (multi threaded)
>>> Version 7.0.1-dev.3203-pthreads as of Oct 13 2013
>>> Compiled for Darwin (x86_64-apple-darwin12.3.0)
>>> Copyright (C) 1998-2013 OpenLink Software
>>>
>>> What is your version ?
>>>
>>> Best Regards
>>> Hugh Williams
>>> Professional Services
>>> OpenLink Software, Inc.      //              http://www.openlinksw.com/
>>> Weblog   -- http://www.openlinksw.com/blogs/
>>> LinkedIn -- http://www.linkedin.com/company/openlink-software/
>>> Twitter  -- http://twitter.com/OpenLink
>>> Google+  -- http://plus.google.com/100570109519069333827/
>>> Facebook -- http://www.facebook.com/OpenLinkSoftware
>>> Universal Data Access, Integration, and Management Technology Providers
>>>
>>> On 11 Oct 2013, at 13:04, Nicola Vitucci <nic...@gm...
>>> <mailto:nic...@gm...>
>>> <mailto:nic...@gm...>> wrote:
>>>
>>>> Hi Hugh,
>>>>
>>>> thanks for the reply. Unfortunately I could't PROFILE any of the queries
>>>> because both ended up with an "Error 22012: [Virtuoso Driver][Virtuoso
>>>> Server]SR083: Division by zero". Any alternatives to do some profiling?
>>>>
>>>> Thanks,
>>>>
>>>> Nicola
>>>>
>>>> Il 11/10/2013 13:01, Hugh Williams ha scritto:
>>>>> Hi Nicola,
>>>>>
>>>>> Can you please provide the output of running the profile() function
>>>>>  for
>>>>> these two queries ie replace explain with profile, which provides more
>>>>> information on the query execution:
>>>>>
>>>>> http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#readingqueryprofile
>>>>>
>>>>> There has not really been changes indexes but more in the query
>>>>> optimiser were the plans may have changed which the explain and profile
>>>>> outputs should help identify for your query ...
>>>>>
>>>>> Best Regards
>>>>> Hugh Williams
>>>>> Professional Services
>>>>> OpenLink Software, Inc.      //              http://www.openlinksw.com/
>>>>> Weblog   -- http://www.openlinksw.com/blogs/
>>>>> LinkedIn -- http://www.linkedin.com/company/openlink-software/
>>>>> Twitter  -- http://twitter.com/OpenLink
>>>>> Google+  -- http://plus.google.com/100570109519069333827/
>>>>> Facebook -- http://www.facebook.com/OpenLinkSoftware
>>>>> Universal Data Access, Integration, and Management Technology Providers
>>>>>
>>>>> On 11 Oct 2013, at 09:52, Nicola Vitucci <nic...@gm...
>>>>> <mailto:nic...@gm...>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I got a problem with Virtuoso 7.0.0 and I think it's related to
>>>>>> the use
>>>>>> of indices. My question is: have there been any big changes
>>>>>> between the
>>>>>> rc2 and the stable version, so that indices are not completely
>>>>>> compatible across the two versions?
>>>>>>
>>>>>> Here is one of the queries I ran on both versions (just the names have
>>>>>> been changed):
>>>>>>
>>>>>> PREFIX int: <namespace#> CONSTRUCT {?s int:new_prop ?n} WHERE {?s
>>>>>> <prop>
>>>>>> ?c . ?c rdfs:label ?n}
>>>>>>
>>>>>> and below you can find the result of EXPLAINing the query on the rc2
>>>>>> version first and then on 7.0.0. The query is reasonably fast on rc2
>>>>>> (41659 msec with ResultSetMaxRows = 10000), while it takes an awfully
>>>>>> long time (1011741 msec, same ResultSetMaxRows) on 7.0.0. Other
>>>>>> queries
>>>>>> have a similar problem.
>>>>>>
>>>>>> Any ideas?
>>>>>>
>>>>>> Thank you,
>>>>>>
>>>>>> Nicola
>>>>>>
>>>>>> ------------------------
>>>>>> Virtuoso rc2
>>>>>>
>>>>>> {
>>>>>>
>>>>>> Precode:
>>>>>>    0: {
>>>>>> s# 63 cluster outer seq start, set no <V $44 set_ctr in>
>>>>>> save ctx:()
>>>>>>
>>>>>> Precode:
>>>>>>    0: <V $27 vector x> := Call vector ( 1 ,  0 ,  3 ,  #new_prop ,  1
>>>>>> ,  1 )
>>>>>>    5: <V $29 vector x> := Call vector (<V $27 vector x>)
>>>>>>    10: <V $31 vector x> := Call vector ()
>>>>>>    15: BReturn 0
>>>>>> s# 89 Fork 46
>>>>>> {
>>>>>> s# 70 from DB.DBA.RDF_QUAD by RDF_QUAD    2.5e+05 rows
>>>>>> Key RDF_QUAD  ASC  (<V $35 s_1_7_t1.O an>, <V $34 s_1_7_t1.S rn>)
>>>>>> inlined  P =  #prop
>>>>>> row specs:  O LIKE <c T�>
>>>>>>
>>>>>> s# 77 from DB.DBA.RDF_QUAD by RDF_QUAD          1 rows
>>>>>> Key RDF_QUAD  ASC  (<V $38 s_1_7_t2.O an>)
>>>>>> inlined  P =  #label  ,  S = <v $81 k_s_1_7_t1.O S83 rn>
>>>>>> vector param casts: <V $35 s_1_7_t1.O an>-> <v $81 k_s_1_7_t1.O
>>>>>> S83 rn>
>>>>>>
>>>>>>
>>>>>> After code:
>>>>>>    0: <V $40 __ro2lo x> := Call __ro2lo (<V $38 s_1_7_t2.O an>)
>>>>>>    5: <V $42 vector x> := Call vector (<r $34 s_1_7_t1.S via  S77
>>>>>> S83>, <V $40 __ro2lo x>)
>>>>>>    10: if ($47 "user_aggr_notfirst" =  1 ) then 25 else 14 unkn 14
>>>>>>    14: $47 "user_aggr_notfirst" :=  := artm  1
>>>>>>    18: <V $49 user_aggr_ret x> := Call DB.DBA.SPARQL_CONSTRUCT_INIT
>>>>>> ($48 "user_aggr_env")
>>>>>>    25: <V $49 user_aggr_ret x> := Call DB.DBA.SPARQL_CONSTRUCT_ACC
>>>>>> ($48 "user_aggr_env", <r $29 vector via  S77 S83 S70>, <V $42
>>>>>> vector x>,
>>>>>> <r $31 vector via  S77 S83 S70>,  1 )
>>>>>>    32: BReturn 0
>>>>>> }
>>>>>> s# 92 skip node  1  <none> <V $44 set_ctr in>
>>>>>>
>>>>>> After code:
>>>>>>    0: <V $50 callret-0 x> := Call DB.DBA.SPARQL_CONSTRUCT_FIN ($48
>>>>>> "user_aggr_env")
>>>>>>    7: BReturn 0
>>>>>> s# 96 Subquery Select(<V $50 callret-0 x>)
>>>>>> set no: <r $44 set_ctr via  S92>
>>>>>> }
>>>>>>
>>>>>>    8:  <none>  := Call DB.DBA.RDF_DICT_OF_TRIPLES_TO_THREE_COLS ($100
>>>>>> "scalar")
>>>>>>    15: BReturn 0
>>>>>> END Node
>>>>>> }
>>>>>>
>>>>>> ------------------------
>>>>>> Virtuoso 7.0.0
>>>>>>
>>>>>> {
>>>>>>
>>>>>> Precode:
>>>>>>    0: {
>>>>>>
>>>>>> Precode:
>>>>>>    0: vector := Call vector ( 1 ,  0 ,  3 ,  #new_prop ,  1 ,  1 )
>>>>>>    5: vector := Call vector (vector)
>>>>>>    10: vector := Call vector ()
>>>>>>    15: BReturn 0
>>>>>> { fork
>>>>>> RDF_QUAD   4.2e+08 rows(s_1_7_t2.S, s_1_7_t2.O)
>>>>>> inlined  P =  #label
>>>>>>
>>>>>> Precode:
>>>>>>    0: __ro2lo := Call __ro2lo (s_1_7_t2.O)
>>>>>>    5: BReturn 0
>>>>>> RDF_QUAD_POGS        33 rows(s_1_7_t1.S)
>>>>>> P =  #prop  ,  O = cast
>>>>>>
>>>>>> After code:
>>>>>>    0: vector := Call vector (s_1_7_t1.S, __ro2lo)
>>>>>>    5: if ($47 "user_aggr_notfirst" =  1 ) then 20 else 9 unkn 9
>>>>>>    9: $47 "user_aggr_notfirst" :=  := artm  1
>>>>>>    13: user_aggr_ret := Call DB.DBA.SPARQL_CONSTRUCT_INIT ($48
>>>>>> "user_aggr_env")
>>>>>>    20: user_aggr_ret := Call DB.DBA.SPARQL_CONSTRUCT_ACC ($48
>>>>>> "user_aggr_env", vector, vector, vector,  1 )
>>>>>>    27: BReturn 0
>>>>>> }
>>>>>> skip node  1  <none> set_ctr
>>>>>>
>>>>>> After code:
>>>>>>    0: callret-0 := Call DB.DBA.SPARQL_CONSTRUCT_FIN ($48
>>>>>> "user_aggr_env")
>>>>>>    7: BReturn 0
>>>>>> Subquery Select(callret-0)
>>>>>> }
>>>>>>
>>>>>>    8:  <none>  := Call DB.DBA.RDF_DICT_OF_TRIPLES_TO_THREE_COLS ($100
>>>>>> "scalar")
>>>>>>    15: BReturn 0
>>>>>> END Node
>>>>>> }
>>>>>>
>>>>>> ------------------------------------------------------------------------------
>>>>>> October Webinars: Code for Performance
>>>>>> Free Intel webinars can help you accelerate application performance.
>>>>>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the
>>>>>> most from
>>>>>> the latest Intel processors and coprocessors. See abstracts and
>>>>>> register >
>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk
>>>>>> _______________________________________________
>>>>>> Virtuoso-users mailing list
>>>>>> Vir...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> October Webinars: Code for Performance
>>>> Free Intel webinars can help you accelerate application performance.
>>>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the
>>>> most from
>>>> the latest Intel processors and coprocessors. See abstracts and
>>>> register >
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> Virtuoso-users mailing list
>>>> Vir...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>>>
>>
>> ------------------------------------------------------------------------------
>> October Webinars: Code for Performance
>> Free Intel webinars can help you accelerate application performance.
>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the
>> most from
>> the latest Intel processors and coprocessors. See abstracts and register >
>> http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Virtuoso-users mailing list
>> Vir...@li...
>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
> 