Re: [Virtuoso-users] performance tuning

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Hi Erick,

Can you confirm that the bitmap indexes are in place on your Virtuoso  
RDF Quad store table "DB.DBA.RDF_QUAD", you can check this from the  
Virtuoso Conductor by checking the indexes this table ?

The indexes we recommend be set on the table are:

create bitmap index rdf_quad_opgs on rdf_quad (o, p, g, s) partition  
(o varchar (-1, 0hexffff));
create bitmap index rdf_quad_pogs on rdf_quad (p, o, g, s) partition  
(o varchar (-1, 0hexffff));
create bitmap index rdf_quad_gpos on rdf_quad (g, p, o, s) partition  
(o varchar (-1, 0hexffff));

See:

http://docs.openlinksw.com/virtuoso/rdfperformancetuning.html

If you are increasing the memory in your machine then you should be  
able to increase the amount of memory available fro use by Virtuoso  
in the ini file to be 2/3 to 3/5 of the available memory as detailed  
in the document above which should improve performance of queries.

Regards
Hugh

/opt/Ingres/IngresII/ingres
On 10 Jun 2008, at 11:06, Erick Antezana wrote:

> Hi Hugh,
>
> Indeed, now it works. Here you are the first report:
>
> Query result:
> REPORT
> VARCHAR
>  {
>
>  Precode:
>   0: $25 "OBO#GO_0005737" := Call DB.DBA.RDF_MAKE_IID_OF_QNAME_SAFE  
> (<constant (http://www.semantic-systems-biology.org/ontology/rdf/ 
> OBO#GO_0005737)>)
>   7: $26 "ema#label" := Call DB.DBA.RDF_MAKE_IID_OF_QNAME_SAFE  
> (<constant (http://www.w3.org/2000/01/rdf-schema#label)>)
>   14: $27 "org/ontology/rdf/GOA" := Call  
> DB.DBA.RDF_MAKE_IID_OF_QNAME (<constant (http://www.semantic- 
> systems-biology.org/ontology/rdf/GOA)>)
>   21: $28 "org/ontology/rdf/OBO" := Call  
> DB.DBA.RDF_MAKE_IID_OF_QNAME (<constant (http://www.semantic- 
> systems-biology.org/ontology/rdf/OBO)>)
>   28: $29 "OBO#located_in" := Call  
> DB.DBA.RDF_MAKE_IID_OF_QNAME_SAFE (<constant (http://www.semantic- 
> systems-biology.org/ontology/rdf/OBO#located_in)>)
>   35: $30 "OBO#has_evidence_code" := Call  
> DB.DBA.RDF_MAKE_IID_OF_QNAME_SAFE (<constant (http://www.semantic- 
> systems-biology.org/ontology/rdf/OBO#has_evidence_code)>)
>   42: $31 "-ns#object" := Call DB.DBA.RDF_MAKE_IID_OF_QNAME_SAFE  
> (<constant (http://www.w3.org/1999/02/22-rdf-syntax-ns#object)>)
>   49: $32 "-ns#predicate" := Call DB.DBA.RDF_MAKE_IID_OF_QNAME_SAFE  
> (<constant (http://www.w3.org/1999/02/22-rdf-syntax-ns#predicate)>)
>   56: $33 "OBO#has_source" := Call  
> DB.DBA.RDF_MAKE_IID_OF_QNAME_SAFE (<constant (http://www.semantic- 
> systems-biology.org/ontology/rdf/OBO#has_source)>)
>   63: $34 "CBI#NCBI_471821" := Call  
> DB.DBA.RDF_MAKE_IID_OF_QNAME_SAFE (<constant (http://www.semantic- 
> systems-biology.org/ontology/rdf/NCBI#NCBI_471821)>)
>   70: $35 "-ns#type" := Call DB.DBA.RDF_MAKE_IID_OF_QNAME_SAFE  
> (<constant (http://www.w3.org/1999/02/22-rdf-syntax-ns#type)>)
>   77: $36 "GOA#prot" := Call DB.DBA.RDF_MAKE_IID_OF_QNAME_SAFE  
> (<constant (http://www.semantic-systems-biology.org/ontology/rdf/ 
> GOA#prot)>)
>   84: $37 "GOA#supported_by" := Call  
> DB.DBA.RDF_MAKE_IID_OF_QNAME_SAFE (<constant (http://www.semantic- 
> systems-biology.org/ontology/rdf/GOA#supported_by)>)
>   91: $38 "-ns#subject" := Call DB.DBA.RDF_MAKE_IID_OF_QNAME_SAFE  
> (<constant (http://www.w3.org/1999/02/22-rdf-syntax-ns#subject)>)
>   98: BReturn 0
>  in iterates $41 "in_iter"
>   over ($27 "org/ontology/rdf/GOA", $28 "org/ontology/rdf/OBO")
>  from DB.DBA.RDF_QUAD by RDF_QUAD_OGPS
>  Key RDF_QUAD_OGPS ASC ($44 "s-10-1-t1.S")
>  <col=415 O = $34 "CBI#NCBI_471821"> , <col=412 G = $41  
> "in_iter"> , <col=414 P = $33 "OBO#has_source">
>  row specs: <col=415 O LIKE <constant (Tó)>>
>
>  Current of: <$46 "<DB.DBA.RDF_QUAD s-10-1-t1>" spec 5>
>  in iterates $49 "in_iter"
>   over ($27 "org/ontology/rdf/GOA", $28 "org/ontology/rdf/OBO")
>  from DB.DBA.RDF_QUAD by RDF_QUAD_OGPS
>  Key RDF_QUAD_OGPS ASC ($52 "s-10-1-t0.S")
>  <col=415 O = $36 "GOA#prot"> , <col=412 G = $49 "in_iter"> ,  
> <col=414 P = $35 "-ns#type"> , <col=413 S = $44 "s-10-1-t1.S">
>  row specs: <col=415 O LIKE <constant (Tó)>>
>
>  Current of: <$54 "<DB.DBA.RDF_QUAD s-10-1-t0>" spec 5>
>  in iterates $57 "in_iter"
>   over ($27 "org/ontology/rdf/GOA", $28 "org/ontology/rdf/OBO")
>  from DB.DBA.RDF_QUAD by RDF_QUAD_OGPS
>  Key RDF_QUAD_OGPS ASC ($60 "s-10-1-t6.S")
>  <col=415 O = $25 "OBO#GO_0005737"> , <col=412 G = $57 "in_iter"> ,  
> <col=414 P = $31 "-ns#object">
>  row specs: <col=415 O LIKE <constant (Tó)>>
>
>  Current of: <$62 "<DB.DBA.RDF_QUAD s-10-1-t6>" spec 5>
>  in iterates $65 "in_iter"
>   over ($27 "org/ontology/rdf/GOA", $28 "org/ontology/rdf/OBO")
>  from DB.DBA.RDF_QUAD by RDF_QUAD_OGPS
>  Key RDF_QUAD_OGPS ASC ($68 "s-10-1-t4.S")
>  <col=415 O = $29 "OBO#located_in"> , <col=412 G = $65 "in_iter"> ,  
> <col=414 P = $32 "-ns#predicate"> , <col=413 S = $60 "s-10-1-t6.S">
>  row specs: <col=415 O LIKE <constant (Tó)>>
>
>  Current of: <$70 "<DB.DBA.RDF_QUAD s-10-1-t4>" spec 5>
>  in iterates $73 "in_iter"
>   over ($27 "org/ontology/rdf/GOA", $28 "org/ontology/rdf/OBO")
>  from DB.DBA.RDF_QUAD by RDF_QUAD
>  Key RDF_QUAD ASC ($76 "s-10-1-t7.O")
>   inlined <col=412 G = $73 "in_iter"> , <col=413 S = $25  
> "OBO#GO_0005737"> , <col=414 P = $26 "ema#label">
>
>  Current of: <$78 "<DB.DBA.RDF_QUAD s-10-1-t7>" spec 5>
>
>  Precode:
>   0: $79 "GO_term" := Call __rdf_sqlval_of_obj ($76 "s-10-1-t7.O")
>   5: BReturn 0
>  in iterates $82 "in_iter"
>   over ($27 "org/ontology/rdf/GOA", $28 "org/ontology/rdf/OBO")
>  from DB.DBA.RDF_QUAD by RDF_QUAD
>  Key RDF_QUAD ASC ($85 "s-10-1-t5.O")
>   inlined <col=412 G = $82 "in_iter"> , <col=413 S = $29  
> "OBO#located_in"> , <col=414 P = $26 "ema#label">
>
>  Current of: <$87 "<DB.DBA.RDF_QUAD s-10-1-t5>" spec 5>
>
>  Precode:
>   0: $88 "predicate" := Call __rdf_sqlval_of_obj ($85 "s-10-1-t5.O")
>   5: BReturn 0
>  in iterates $91 "in_iter"
>   over ($27 "org/ontology/rdf/GOA", $28 "org/ontology/rdf/OBO")
>  from DB.DBA.RDF_QUAD by RDF_QUAD
>  Key RDF_QUAD ASC ($94 "s-10-1-t9.S")
>   inlined <col=412 G = $91 "in_iter">
>  row specs: <col=414 P = $30 "OBO#has_evidence_code">
>
>  Current of: <$96 "<DB.DBA.RDF_QUAD s-10-1-t9>" spec 5>
>
>  Precode:
>   0: $97 "callret" := Call __all_eq ($44 "s-10-1-t1.S", $52 "s-10-1- 
> t0.S")
>   5: BReturn 0
>  in iterates $100 "in_iter"
>   over ($27 "org/ontology/rdf/GOA", $28 "org/ontology/rdf/OBO")
>  from DB.DBA.RDF_QUAD by RDF_QUAD
>  Key RDF_QUAD ASC ($104 "s-10-1-t2.S", $103 "s-10-1-t2.O")
>   inlined <col=412 G = $100 "in_iter"> , <col=413 S = $52 "s-10-1- 
> t0.S"> , <col=414 P = $26 "ema#label">
>  row specs: <col=413 S = $44 "s-10-1-t1.S">
>
>  Current of: <$106 "<DB.DBA.RDF_QUAD s-10-1-t2>" spec 5>
>
>  Precode:
>   0: $107 "protein" := Call __rdf_sqlval_of_obj ($103 "s-10-1-t2.O")
>   5: $108 "callret" := Call __all_eq ($60 "s-10-1-t6.S", $68  
> "s-10-1-t4.S")
>   10: BReturn 0
>  in iterates $111 "in_iter"
>   over ($27 "org/ontology/rdf/GOA", $28 "org/ontology/rdf/OBO")
>  from DB.DBA.RDF_QUAD by RDF_QUAD_OGPS
> No. of rows in result: 100
>
>
> and the one corresponding to the second query:
>
> Query result:
> REPORT
> VARCHAR
>  {
>  Fork
>  {
>
>  Precode:
>   0: $25 "org/ontology/rdf/OBO" := Call  
> DB.DBA.RDF_MAKE_IID_OF_QNAME_SAFE (<constant (http://www.semantic- 
> systems-biology.org/ontology/rdf/OBO)>)
>   7: $26 "-ns#type" := Call DB.DBA.RDF_MAKE_IID_OF_QNAME_SAFE  
> (<constant (http://www.w3.org/1999/02/22-rdf-syntax-ns#type)>)
>   14: BReturn 0
>  from DB.DBA.RDF_QUAD by RDF_QUAD
>  Key RDF_QUAD ASC ($28 "s-4-1-t0.O")
>   inlined <col=412 G = $25 "org/ontology/rdf/OBO">
>  row specs: <col=414 P = $26 "-ns#type">
>
>  Current of: <$30 "<DB.DBA.RDF_QUAD s-4-1-t0>" spec 5>
>
>  Precode:
>   0: $31 "ontology" := Call __rdf_sqlval_of_obj ($28 "s-4-1-t0.O")
>   5: BReturn 0
>  Sort (HASH) ($31 "ontology")
>  }
>  Key from temp ($31 "ontology")
>
>
>  Distinct (HASH) ($31 "ontology")
>  Select ($31 "ontology", <$30 "<DB.DBA.RDF_QUAD s-4-1-t0>" spec 5>)
>  }
> No. of rows in result: 26
>
> Erick
>
> Hugh Williams wrote:
>>
>> Hi Erick,
>>
>> Sorry, I forgot to have you prefix the queries with the keyword  
>> 'sparql' to tell Virtuoso to execute via the SPARQL rather than  
>> SQL engine, so the query below would be:
>>
>> explain('sparql prefix rdf:<http://www.w3.org/1999/02/22-rdf- 
>> syntax-ns#> select distinct ?ontology from <http://www.semantic- 
>> systems-biology.org/ontology/rdf/OBO> where{ ?term rdf:type ? 
>> ontology. } order by ?ontology ');
>>
>> This will enable the queries to work ...
>>
>> Best Regards
>> Hugh Williams
>> OpenLink Software
>>
>>
>> On 9 Jun 2008, at 18:05, Erick Antezana wrote:
>>
>>> Hugh,
>>>
>>> I tried with the isql command line ensuring that the query is all  
>>> on the same line:
>>>
>>> SQL> explain('prefix rdf:<http://www.w3.org/1999/02/22-rdf-syntax- 
>>> ns#> select distinct ?ontology from <http://www.semantic-systems- 
>>> biology.org/ontology/rdf/OBO> where{ ?term rdf:type ?ontology. }  
>>> order by ?ontology ');
>>>
>>> *** Error 37000: [Virtuoso Driver][Virtuoso Server]SQ074: Line 1:  
>>> syntax error at 'rdf' before ':'
>>> at line 23 of Top-Level:
>>> explain('prefix rdf:<http://www.w3.org/1999/02/22-rdf-syntax-ns#>  
>>> select distinct ?ontology from <http://www.semantic-systems- 
>>> biology.org/ontology/rdf/OBO> where{ ?term rdf:type ?ontology. }  
>>> order by ?ontology ')
>>> SQL>
>>>
>>> I also tried with the interactive UI in one line and multiple  
>>> lines. The result is the same :-(
>>>
>>> The statement execution did not return a result set.
>>>
>>> SQLState: 37000
>>>
>>> Message: SQ074: Line 1: syntax error at 'rdf' before ':'
>>>
>>>
>>> Erick
>>>
>>> Hugh Williams wrote:
>>>> Hi Erick,
>>>>
>>>> If you are executing from the isql commandl line then you most  
>>>> probably need to ensure the query  is all on the same line as  
>>>> multiple lines will not be passed as one query to the server.  
>>>> Multiple lines would best in the Conductor Interactive SQL UI,  
>>>> which you could try also.
>>>>
>>>> Best Regards
>>>> Hugh Williams
>>>> Professional Services
>>>> OpenLink Software
>>>>
>>>> On 9 Jun 2008, at 17:40, Erick Antezana wrote:
>>>>
>>>>> Hi Hugh,
>>>>>
>>>>> I tried to run both queries (first surrounded by the 'explain'  
>>>>> command) and I've got the following error message:
>>>>>
>>>>>
>>>>> The statement execution did not return a result set.
>>>>>
>>>>> SQLState: 37000
>>>>>
>>>>> Message: SQ074: Line 1: syntax error at 'rdf' before ':'
>>>>>
>>>>>
>>>>>
>>>>> I also tried without the 'explain' command and I've got:
>>>>>
>>>>> SQLState: 22023
>>>>>
>>>>> Message: SR007: Function trim needs a string or NULL as  
>>>>> argument 1, not an arg of type ARRAY_OF_POINTER (193)
>>>>>
>>>>>
>>>>> Moreover, I tried with the advanced tab and selecting the  
>>>>> explain option and I've got:
>>>>>
>>>>> SQLState: 22023
>>>>>
>>>>> Message: SR007: Function trim needs a string or NULL as  
>>>>> argument 1, not an arg of type ARRAY_OF_POINTER (193)
>>>>>
>>>>>
>>>>> From the command line (isql):
>>>>>
>>>>> SQL> explain('prefix rdf:<http://www.w3.org/1999/02/22-rdf- 
>>>>> syntax-ns#>
>>>>> select distinct ?ontology
>>>>> from <http://www.semantic-systems-biology.org/ontology/rdf/OBO>
>>>>> where{
>>>>> ?term rdf:type ?ontology.
>>>>> }
>>>>> order by ?ontology ')Type the rest of statement, end with a  
>>>>> semicolon (;)> Type the rest of statement, end with a semicolon  
>>>>> (;)> Type the rest of statement, end with a semicolon (;)> Type  
>>>>> the rest of statement, end with a semicolon (;)> Type the rest  
>>>>> of statement, end with a semicolon (;)> Type the rest of  
>>>>> statement, end with a semicolon (;)> ;
>>>>>
>>>>> *** Error 37000: [Virtuoso Driver][Virtuoso Server]SQ074: Line  
>>>>> 1: syntax error at 'rdf' before ':'
>>>>> in lines 1-7 of Top-Level:
>>>>> #line 1 "(console)"
>>>>> explain('prefix rdf:<http://www.w3.org/1999/02/22-rdf-syntax-ns#>
>>>>> select distinct ?ontology
>>>>> from <http://www.semantic-systems-biology.org/ontology/rdf/OBO>
>>>>> where{
>>>>> ?term rdf:type ?ontology.
>>>>> }
>>>>> order by ?ontology ');
>>>>>
>>>>>
>>>>> Am I missing something?
>>>>>
>>>>> thanks,
>>>>> Erick
>>>>>
>>>>> Hugh Williams wrote:
>>>>>> Hi Eric,
>>>>>>
>>>>>> We have been able to see the hangs you report with the sample  
>>>>>> queries below.
>>>>>>
>>>>>> Can you please provide the output of running the Virtuoso  
>>>>>> explain command against these 2 queries and provide this  
>>>>>> output. The explain command provides more details on the query  
>>>>>> execution plan with Virtuoso as detailed at:
>>>>>>
>>>>>> http://docs.openlinksw.com/virtuoso/fn_explain.html#(NULL  
>>>>>> <http://docs.openlinksw.com/virtuoso/fn_explain.html#%28NULL>)
>>>>>>
>>>>>> Note you will have to use the Virtuoso  isql command or  
>>>>>> Conductor Interactive SQL UI to execute the commands which  
>>>>>> will be as follows:
>>>>>>
>>>>>> example 1:
>>>>>>
>>>>>> explain('prefix rdf:<http://www.w3.org/1999/02/22-rdf-syntax-ns#>
>>>>>> prefix rdfs:<http://www.w3.org/2000/01/rdf-schema#>
>>>>>> prefix obo:<http://www.semantic-systems-biology.org/ontology/ 
>>>>>> rdf/OBO#>
>>>>>> prefix goa:<http://www.semantic-systems-biology.org/ontology/ 
>>>>>> rdf/GOA#>
>>>>>> prefix uni:<http://www.semantic-systems-biology.org/ontology/ 
>>>>>> rdf/UNIPROT#>
>>>>>> prefix ncbi:<http://www.semantic-systems-biology.org/ontology/ 
>>>>>> rdf/NCBI#>
>>>>>> select ?protein ?predicate ?GO_term ?GOA_association
>>>>>> from <http://www.semantic-systems-biology.org/ontology/rdf/OBO>
>>>>>> from <http://www.semantic-systems-biology.org/ontology/rdf/GOA>
>>>>>> where{
>>>>>> ?prot rdf:type goa:prot. ?prot obo:has_source ncbi:NCBI_471821.
>>>>>> ?prot rdfs:label ?protein.
>>>>>> ?Statement rdf:subject ?prot.
>>>>>> ?Statement rdf:predicate obo:located_in.
>>>>>> obo:located_in rdfs:label ?predicate.
>>>>>> ?Statement rdf:object obo:GO_0005737.
>>>>>> obo:GO_0005737 rdfs:label ?GO_term.
>>>>>> ?Statement goa:supported_by ?GOA_association.
>>>>>> ?GOA_association obo:has_evidence_code ?eco_id.
>>>>>> }')
>>>>>>
>>>>>> example 2:
>>>>>>
>>>>>> explain('prefix rdf:<http://www.w3.org/1999/02/22-rdf-syntax-ns#>
>>>>>> select distinct ?ontology
>>>>>> from <http://www.semantic-systems-biology.org/ontology/rdf/OBO>
>>>>>> where{
>>>>>> ?term rdf:type ?ontology.
>>>>>> }
>>>>>> order by ?ontology ')
>>>>>>
>>>>>> Best Regards
>>>>>> Hugh Williams
>>>>>> Professional Services
>>>>>> OpenLink Software
>>>>>>
>>>>>> On 8 Jun 2008, at 09:29, Erick Antezana wrote:
>>>>>>
>>>>>>> Hi Hugh,
>>>>>>>
>>>>>>> sorry, I forgot to add a couple of examples. here are some  
>>>>>>> examples that can be tested from:
>>>>>>>
>>>>>>> http://www.cellcycleontology.org/query/38-interactive-query/ 
>>>>>>> 82-sparql-kb
>>>>>>>
>>>>>>> example 1:
>>>>>>>
>>>>>>> prefix rdf:<http://www.w3.org/1999/02/22-rdf-syntax-ns#>
>>>>>>> prefix rdfs:<http://www.w3.org/2000/01/rdf-schema#>
>>>>>>> prefix obo:<http://www.semantic-systems-biology.org/ontology/ 
>>>>>>> rdf/OBO#>
>>>>>>> prefix goa:<http://www.semantic-systems-biology.org/ontology/ 
>>>>>>> rdf/GOA#>
>>>>>>> prefix uni:<http://www.semantic-systems-biology.org/ontology/ 
>>>>>>> rdf/UNIPROT#>
>>>>>>> prefix ncbi:<http://www.semantic-systems-biology.org/ontology/ 
>>>>>>> rdf/NCBI#>
>>>>>>> select ?protein ?predicate ?GO_term ?GOA_association
>>>>>>> from <http://www.semantic-systems-biology.org/ontology/rdf/OBO>
>>>>>>> from <http://www.semantic-systems-biology.org/ontology/rdf/GOA>
>>>>>>> where{
>>>>>>> ?prot rdf:type goa:prot. ?prot obo:has_source ncbi:NCBI_471821.
>>>>>>> ?prot rdfs:label ?protein.
>>>>>>> ?Statement rdf:subject ?prot.
>>>>>>> ?Statement rdf:predicate obo:located_in.
>>>>>>> obo:located_in rdfs:label ?predicate.
>>>>>>> ?Statement rdf:object obo:GO_0005737.
>>>>>>> obo:GO_0005737 rdfs:label ?GO_term.
>>>>>>> ?Statement goa:supported_by ?GOA_association.
>>>>>>> ?GOA_association obo:has_evidence_code ?eco_id.
>>>>>>> }
>>>>>>>
>>>>>>> example 2:
>>>>>>>
>>>>>>> prefix rdf:<http://www.w3.org/1999/02/22-rdf-syntax-ns#>
>>>>>>> select distinct ?ontology
>>>>>>> from <http://www.semantic-systems-biology.org/ontology/rdf/OBO>
>>>>>>> where{
>>>>>>> ?term rdf:type ?ontology.
>>>>>>> }
>>>>>>> order by ?ontology
>>>>>>>
>>>>>>> Since I am constantly testing it, changing the config  
>>>>>>> parameters and adding more data, the endpoint may be down  
>>>>>>> during the night (CET) duo to the entire rebuild of the  
>>>>>>> system (it is build automatically from scratch taking the  
>>>>>>> latest data sources...).
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Erick
>>>>>>>
>>>>>>>
>>>>>>> Hugh Williams wrote:
>>>>>>>> Hi Erick,
>>>>>>>>
>>>>>>>> You have not provided some sample queries for which these  
>>>>>>>> performance issues are being encountered as requested in  
>>>>>>>> point 3 ?
>>>>>>>>
>>>>>>>> Best Regards
>>>>>>>> Hugh Williams
>>>>>>>> OpenLink Software
>>>>>>>>
>>>>>>>> On 6 Jun 2008, at 14:09, Erick Antezana wrote:
>>>>>>>>
>>>>>>>>> Hi Hugh,
>>>>>>>>>
>>>>>>>>> here you are the 'top' output:
>>>>>>>>>
>>>>>>>>> top - 14:57:25 up 13 days, 22:52,  7 users,  load average:  
>>>>>>>>> 2.57, 1.74, 1.31
>>>>>>>>> Tasks: 133 total,   1 running, 132 sleeping,   0 stopped,    
>>>>>>>>> 0 zombie
>>>>>>>>> Cpu(s): 68.0% us,  1.7% sy,  0.0% ni, 29.8% id,  0.5% wa,   
>>>>>>>>> 0.0% hi,  0.0% si
>>>>>>>>> Mem:  15901500k total, 15855120k used,    46380k free,     
>>>>>>>>> 40364k buffers
>>>>>>>>> Swap:  4096564k total,  1016240k used,  3080324k free,   
>>>>>>>>> 2951672k cached
>>>>>>>>>
>>>>>>>>>  PID USER      PR  NI %CPU    TIME+  %MEM  VIRT  RES  SHR S  
>>>>>>>>> COMMAND                     16990 root      16   0  279  
>>>>>>>>> 378:08.04 75.7 12.0g  11g 8256 S virtuoso- 
>>>>>>>>> t                      1 root      16   0    0   0:00.66   
>>>>>>>>> 0.0  4756  420  388 S init                            2  
>>>>>>>>> root      RT   0    0   0:00.21  0.0     0    0    0 S  
>>>>>>>>> migration/0                     3 root      34  19    0    
>>>>>>>>> 0:18.19  0.0     0    0    0 S ksoftirqd/ 
>>>>>>>>> 0                     4 root      RT   0    0   0:00.20   
>>>>>>>>> 0.0     0    0    0 S migration/1                     5  
>>>>>>>>> root      34  19    0   0:00.88  0.0     0    0    0 S  
>>>>>>>>> ksoftirqd/1                     6 root      RT   0    0    
>>>>>>>>> 0:00.16  0.0     0    0    0 S migration/ 
>>>>>>>>> 2                     7 root      34  19    0   0:01.02   
>>>>>>>>> 0.0     0    0    0 S ksoftirqd/2                     8  
>>>>>>>>> root      RT   0    0   0:00.10  0.0     0    0    0 S  
>>>>>>>>> migration/3                     9 root      34  19    0    
>>>>>>>>> 0:15.42  0.0     0    0    0 S ksoftirqd/ 
>>>>>>>>> 3                    10 root       5 -10    0   0:00.00   
>>>>>>>>> 0.0     0    0    0 S events/0                       11  
>>>>>>>>> root       5 -10    0   0:00.01  0.0     0    0    0 S  
>>>>>>>>> events/1                       12 root       5 -10    0    
>>>>>>>>> 0:00.04  0.0     0    0    0 S events/ 
>>>>>>>>> 2                       13 root       5 -10    0   0:00.00   
>>>>>>>>> 0.0     0    0    0 S events/3                       14  
>>>>>>>>> root       5 -10    0   0:00.00  0.0     0    0    0 S  
>>>>>>>>> khelper                        15 root      15 -10    0    
>>>>>>>>> 0:00.00  0.0     0    0    0 S  
>>>>>>>>> kacpid                         64 root       5 -10    0    
>>>>>>>>> 0:00.00  0.0     0    0    0 S kblockd/ 
>>>>>>>>> 0                      65 root       5 -10    0   0:00.00   
>>>>>>>>> 0.0     0    0    0 S kblockd/1                      66  
>>>>>>>>> root       5 -10    0   0:00.00  0.0     0    0    0 S  
>>>>>>>>> kblockd/2                      67 root       5 -10    0    
>>>>>>>>> 0:00.00  0.0     0    0    0 S kblockd/ 
>>>>>>>>> 3                      68 root      15   0    0   0:00.00   
>>>>>>>>> 0.0     0    0    0 S khubd                          97  
>>>>>>>>> root      15   0    0   1:09.28  0.0     0    0    0 S  
>>>>>>>>> kswapd1                        98 root      15   0    0    
>>>>>>>>> 1:27.16  0.0     0    0    0 S  
>>>>>>>>> kswapd0                        99 root       5 -10    0    
>>>>>>>>> 0:00.00  0.0     0    0    0 S aio/0
>>>>>>>>> the machine has 2 processors, more info:
>>>>>>>>>
>>>>>>>>> $ uname -a
>>>>>>>>> Linux crunch 2.6.9-67.0.15.ELsmp #1 SMP Tue Apr 22 13:58:43  
>>>>>>>>> EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
>>>>>>>>>
>>>>>>>>> regarding the FDsPerFile param, I deleted the entire db, so  
>>>>>>>>> I began from scratch setting that param to  8 (initally I  
>>>>>>>>> tried out 32, and i got a similar error as reported then I  
>>>>>>>>> tried out 16, and the same error, then  8 and it worked).  
>>>>>>>>> Other params:
>>>>>>>>>
>>>>>>>>> [Parameters]
>>>>>>>>> ServerPort                 = 1112
>>>>>>>>> DisableUnixSocket          = 1
>>>>>>>>> ServerThreads              = 50
>>>>>>>>> CheckpointInterval         = 6000
>>>>>>>>> O_DIRECT                   = 1
>>>>>>>>> NumberOfBuffers            = 1250000
>>>>>>>>> MaxDirtyBuffers            = 1000000
>>>>>>>>> CaseMode                   = 2
>>>>>>>>> MaxStaticCursorRows        = 50000
>>>>>>>>> CheckpointAuditTrail       = 0
>>>>>>>>> AllowOSCalls               = 0
>>>>>>>>> SchedulerInterval          = 10
>>>>>>>>> DirsAllowed                = ., /virtuoso/data/virtuoso- 
>>>>>>>>> opensource-full/share/virtuoso/vad, /virtuoso/data/rdf, / 
>>>>>>>>> virtuoso/data/rdf_tc, /virtuoso/db/left, /virtuoso/db/right
>>>>>>>>> TransactionAfterImageLimit = 5000000000
>>>>>>>>> ThreadCleanupInterval      = 0
>>>>>>>>> ThreadThreshold            = 10
>>>>>>>>> ResourcesCleanupInterval   = 0
>>>>>>>>> FreeTextBatchSize          = 10000000
>>>>>>>>> SingleCPU                  = 0
>>>>>>>>> VADInstallDir              = /virtuoso/data/virtuoso- 
>>>>>>>>> opensource-full/share/virtuoso/vad/
>>>>>>>>> PrefixResultNames          = 0
>>>>>>>>> ;TraceOn                    = errors, compile, exec,  
>>>>>>>>> transact, sql_send, user_log, failed_log
>>>>>>>>> TraceOn                    = errors
>>>>>>>>> ;MaxMemPoolSize          = 0
>>>>>>>>> FDsPerFile                 = 8
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [Client]
>>>>>>>>> SQL_PREFETCH_ROWS  = 1000
>>>>>>>>> SQL_PREFETCH_BYTES = 60000
>>>>>>>>> SQL_QUERY_TIMEOUT  = 0
>>>>>>>>> SQL_TXN_TIMEOUT    = 0
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [VDB]
>>>>>>>>> ArrayOptimization           = 0
>>>>>>>>> NumArrayParameters          = 10
>>>>>>>>> VDBDisconnectTimeout        = 1000
>>>>>>>>> KeepConnectionOnFixedThread = 0
>>>>>>>>>
>>>>>>>>> [Replication]
>>>>>>>>> ServerName   = crunch
>>>>>>>>> ServerEnable = 1
>>>>>>>>> QueueMax     = 50000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ..
>>>>>>>>> ErrorLogLevel      = 7
>>>>>>>>> FileExtend         = 100000
>>>>>>>>> MaxCheckpointRemap = 3000000
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>> thanks,
>>>>>>>>> erick
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hugh Williams wrote:
>>>>>>>>>> Hi Erick,
>>>>>>>>>>
>>>>>>>>>> Can you please provide the following:
>>>>>>>>>>
>>>>>>>>>> 1.  The output of running the top command on the machine  
>>>>>>>>>> at the time you are running these problem queries such  
>>>>>>>>>> that we can see the CPU, I/O etc consumption
>>>>>>>>>> 2. How many hard disks and CPUs are in the machine currently
>>>>>>>>>> 3. Some of these problem queries for analysis
>>>>>>>>>>
>>>>>>>>>> Your FDsPerFile has too many stripes most probably,  
>>>>>>>>>> resulting in the process running out of descriptors. We  
>>>>>>>>>> would suggest you use only one stripe per physical disk.
>>>>>>>>>>
>>>>>>>>>> Best Regards
>>>>>>>>>> Hugh Williams
>>>>>>>>>> Professional Services
>>>>>>>>>> OpenLink Software
>>>>>>>>>> http://www.openlinksw.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 5 Jun 2008, at 11:33, Erick Antezana wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> could anybody point out some doc/examples for doing some  
>>>>>>>>>>> performance tuning?
>>>>>>>>>>>
>>>>>>>>>>> I have been playing (in a linux box with 16GB RAM, 1  
>>>>>>>>>>> CPU, ...) with some
>>>>>>>>>>> params (numberofbuffers, maxcheckpointremap, O_DIRECT,  
>>>>>>>>>>> etc) as indicated in:
>>>>>>>>>>>
>>>>>>>>>>> http://virtuoso.openlinksw.com/wiki/main/Main/VOSScale
>>>>>>>>>>>
>>>>>>>>>>> http://docs.openlinksw.com/virtuoso/ 
>>>>>>>>>>> rdfperformancetuning.html
>>>>>>>>>>>
>>>>>>>>>>> however, our system is still pretty slow or even no  
>>>>>>>>>>> responsive :-( while
>>>>>>>>>>> launching some SPARQL queries...
>>>>>>>>>>>
>>>>>>>>>>> on the other hand, we're planing on buying a new server.  
>>>>>>>>>>> Could anybody
>>>>>>>>>>> recommend some specs (RAM, CPU, disks, ...) so that it could
>>>>>>>>>>> "steadily/reasonably" cope with huge amounts of data  
>>>>>>>>>>> (about 160GB in RDF
>>>>>>>>>>> files). Any advice for the hard disks spec (type:  
>>>>>>>>>>> SCSI, ...; IO rate;
>>>>>>>>>>> ...)? config (striping, ...)?
>>>>>>>>>>>
>>>>>>>>>>> thanks in advance!
>>>>>>>>>>> Erick
>>>>>>>>>>> p.s. while setting the param FDsPerFile to 16 for  
>>>>>>>>>>> isntance, I've got an
>>>>>>>>>>> error while re-starting virtuoso:
>>>>>>>>>>>
>>>>>>>>>>> 12:15:54 Cannot open stripe on /virtuoso/db/right/db- 
>>>>>>>>>>> seg32-2.db (24)
>>>>>>>>>>> 12:15:54 Server exiting
>>>>>>>>>>>
>>>>>>>>>>> should that param be set the first time the system is  
>>>>>>>>>>> started (due to
>>>>>>>>>>> the striping?)?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ------------------------------------------------------------ 
>>>>>>>>>>> ------------- Check out the new SourceForge.net Marketplace.
>>>>>>>>>>> It's the best place to buy or sell services for
>>>>>>>>>>> just about anything Open Source.
>>>>>>>>>>> http://sourceforge.net/services/buy/index.php
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Virtuoso-users mailing list
>>>>>>>>>>> Vir...@li... <mailto:Virtuoso- 
>>>>>>>>>>> us...@li...>
>>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>>>>>>>>>
>>>>>>>>> -- 
>>>>>>>>> ============================================================== 
>>>>>>>>> ====
>>>>>>>>> Erick Antezana                    http:// 
>>>>>>>>> www.cellcycleontology.org
>>>>>>>>> PhD student
>>>>>>>>> Tel:+32 (0)9 331 38 24                        fax:+32 (0)9  
>>>>>>>>> 3313809
>>>>>>>>> VIB Department of Plant Systems Biology, Ghent University
>>>>>>>>> Technologiepark 927, 9052 Gent, BELGIUM
>>>>>>>>> er...@ps...  
>>>>>>>>> <mailto:er...@ps...>                  http:// 
>>>>>>>>> www.psb.ugent.be/~erant
>>>>>>>>> ============================================================== 
>>>>>>>>> ====
>>>>>>>
>>>>>>
>>>>>
>>>>> -- 
>>>>> ==================================================================
>>>>> Erick Antezana                    http://www.cellcycleontology.org
>>>>> PhD student
>>>>> Tel:+32 (0)9 331 38 24                        fax:+32 (0)9 3313809
>>>>> VIB Department of Plant Systems Biology, Ghent University
>>>>> Technologiepark 927, 9052 Gent, BELGIUM
>>>>> er...@ps...                  http://www.psb.ugent.be/~erant
>>>>> ==================================================================
>>>
>>> -- 
>>> ==================================================================
>>> Erick Antezana                    http://www.cellcycleontology.org
>>> PhD student
>>> Tel:+32 (0)9 331 38 24                        fax:+32 (0)9 3313809
>>> VIB Department of Plant Systems Biology, Ghent University
>>> Technologiepark 927, 9052 Gent, BELGIUM
>>> er...@ps...                  http://www.psb.ugent.be/~erant
>>> ==================================================================
>>
>
> -- 
> ==================================================================
> Erick Antezana                    http://www.cellcycleontology.org
> PhD student
> Tel:+32 (0)9 331 38 24                        fax:+32 (0)9 3313809
> VIB Department of Plant Systems Biology, Ghent University
> Technologiepark 927, 9052 Gent, BELGIUM
> er...@ps...                  http://www.psb.ugent.be/~erant
> ==================================================================