Re: [Exist-open] Performance decrease dramaticaly with the number of docs in a collection

[Jean-Marc V. <jm...@fr...>]

Laurent Trillaud wrote:

> Hi
> I have a serious problem when I populate the database. I use a 
> snapshot of Feb 05.
> The test is very simple, I have a file of 500 bytes with 12 tags in 3 
> nested level ( ie. /root/level2/level3).
> I made a very simple xquery :
> let $doc in document( '/db/path/to/collection/myfile.xml')
> return $doc
>
> With 30 files in the collection, I have about 125 requests/second
> With 800 files, I have 50 req/s
> With 10.000, I have 4 req/s  !!!
>
> I have increase the memory of the embedded Jetty with -Xmx512m, I have 
> increase the max poll of eXist with 20.
> Try to reindex the database. But it is not enough.
> Why it's so slow? 10.000 is not so much!

Just like a relational database, eXist has indices .
The defaults settings in eXist are to index just words, and to not index alphanumeric strings.

If you have such content, you must configure either the fulltext index, or a "range index", read this :
2. Index Configuration <http://exist.sourceforge.net/indexing.html#N10172>

The simplest for a quick try is to modify conf.xml like this :
<fulltext default="all" attributes="false" alphanum="true" >


-- 
Jean-Marc Vanel
Conseil et Services / développement & intégration logiciels
Logiciel libre, Web, Java, XML ...
A la pointe de la technique, au service des projets
http://jmvanel.free.fr/ ===) CV, software resources 

Mes journaux:
- sujets généraux en Français: http://jmvanel.free.fr/Block-note.html
- sujets informatiques en Français: http://jmvanel.free.fr/notes-informatiques.html
- computer science diary : http://jmvanel.free.fr/computer-notes.html

Worldwide Botanical Knowledge Base 
http://wwbota.free.fr/ 
test XML query engine: http://jmvanel.free.fr/protea.html