Re: [Exist-development] Benchmarking eXist

[Wolfgang M. <wol...@ex...>]

Hi,

> I see that eXist has some performance tests in the test
> suite, but I don't know whether they are still maintained or used as
> guides for further development.

I wrote those tests and I still use them whenever I make larger
changes to the query engine ("build.sh benchmark"). The tests take too
many resources to be run as part of the standard test suite. Memory
and cache settings need to be increased a bit (1gb main memory, 256mb
cache), otherwise loading the 600MB dblp bibliography file will just
take too long. Also, I don't always run all tests. However, it is a
useful tool to make sure my changes do not result in a loss of
performance.

>From my experience I would say that you need a large number of
slightly different queries in every tested area to really measure
performance. Benchmarks which are based on 10 or 20 queries are pretty
useless (at least to me as a developer). XQuery often provides more
than one way to address a problem. For example, it may make a
difference for the query engine if you use a "where" clause or an
XPath predicate, even though the result of both expressions will be
the same. Minimal changes in the query formulation may cause the query
to take 20 times as long as before. The underlying data structure also
plays a major role, so you have to test with a variety of data sets.

As a developer, I certainly want all queries to be as fast as
possible, but this requires a large number of slightly varying tests.
I thus think the current benchmark test suite is still much to small.
It only covers the basic indexes and queries, and only 3 data sets. I
would certainly welcome some additions.

I also think the performance tests should be better integrated. It
would be nice if we had a web page (similar to the one for the XQTS)
where I could just select the test sets I want to run, execute them
and get back the results.

Wolfgang