From: Ralf J. <ral...@gm...> - 2011-10-28 19:26:03
|
Hi list, in an attempt to make the development process more stable, I wanted to switch from using exist trunk in my application (which is used in production, so I need some reliability) to exist stable. This worked generally well (see my other mail for getting minimal mode to work [1]), however expensive search queries are now running much slower - up to factor 10 (30 seconds instead of less than 3, in one case). The queries use no special module or similar, just basic XPath and regular expressions to find items with nodes matching some criteria. Is that to be expected? Was there some really great step forwards since stable was branched, with the regard to basic query evaluation? Or did I probably do something wrong in the configuration or somewhere else? I can't really do the switch if it comes with such a loss, unfortunately. Any hint would be appreciated, Ralf [1] http://exist.2174344.n4.nabble.com/eXist-stable-minimal-mode-not-working- td3942142.html |
From: Palmer, E. <ep...@ri...> - 2011-10-28 19:47:53
|
Ralf I'm not on the development team. We have been profiling queries that we use here at the University of Richmond and comparing 1.4.2stable to 1.5trunk We have found several 1.5 queries that run significantly faster in trunk. For us 2 to 4 times. But I have seen reports in the 6 to 15 range or something like that. We also have seen queries slower in trunk (by 20 times). So it may be that you ahva a query that better uses the indexe and query improvements. Most are slightly faster in trunk to twice as fast. Have you tried to reindex the database? Can you show us the query and the conf file for your indexes? Eric U of Richmond On 10/28/11 3:25 PM, "Ralf Jung" <ral...@gm...> wrote: > Hi list, > > in an attempt to make the development process more stable, I wanted to switch > from using exist trunk in my application (which is used in production, so I > need some reliability) to exist stable. > This worked generally well (see my other mail for getting minimal mode to work > [1]), however expensive search queries are now running much slower - up to > factor 10 (30 seconds instead of less than 3, in one case). The queries use no > special module or similar, just basic XPath and regular expressions to find > items with nodes matching some criteria. Is that to be expected? Was there > some really great step forwards since stable was branched, with the regard to > basic query evaluation? Or did I probably do something wrong in the > configuration or somewhere else? > I can't really do the switch if it comes with such a loss, unfortunately. > > Any hint would be appreciated, > Ralf > > > [1] http://exist.2174344.n4.nabble.com/eXist-stable-minimal-mode-not-working- > td3942142.html > > ------------------------------------------------------------------------------ > The demand for IT networking professionals continues to grow, and the > demand for specialized networking skills is growing even more rapidly. > Take a complimentary Learning@Cisco Self-Assessment and learn > about Cisco certifications, training, and career opportunities. > http://p.sf.net/sfu/cisco-dev2dev > _______________________________________________ > Exist-open mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-open |
From: Dannes W. <da...@ex...> - 2011-10-28 20:00:56
|
Ralf, Compared to 1.4.x, trunk has some significant changes on the structural index, making things significant faster; these changes were not ported back to 1.4.x so that might actually explain the differences. Wolf talked about porting these changes, but I think it is not not appropriate for a stable release. And it costs time while we should focus on the 1.5 release these days. cheers Dannes On 28 Oct 2011, at 21:25 , Ralf Jung wrote: > in an attempt to make the development process more stable, I wanted to switch > from using exist trunk in my application (which is used in production, so I > need some reliability) to exist stable. > This worked generally well (see my other mail for getting minimal mode to work > [1]), however expensive search queries are now running much slower - up to > factor 10 (30 seconds instead of less than 3, in one case). The queries use no > special module or similar, just basic XPath and regular expressions to find > items with nodes matching some criteria. Is that to be expected? Was there > some really great step forwards since stable was branched, with the regard to > basic query evaluation? Or did I probably do something wrong in the > configuration or somewhere else? > I can't really do the switch if it comes with such a loss, unfortunately. > > Any hint would be appreciated, |
From: Ralf J. <ral...@gm...> - 2011-10-29 19:25:35
|
Hi, > Compared to 1.4.x, trunk has some significant changes on the structural > index, making things significant faster; these changes were not ported > back to 1.4.x so that might actually explain the differences. Yes, he query is mostly structural, so that could indeed be the culprit. > So it may be that you ahva a query that better uses the indexe and query > improvements. > > > Most are slightly faster in trunk to twice as fast. > > Have you tried to reindex the database? I actually re-created the whole database as the data format between 1.4 and 1.5 is incompatible. > Can you show us the query and the conf file for your indexes? Sure - this is the one that's 10 times slower: red-db:get-file('praeps')[((exists(darreich/(.//@samcod | .//@hstcod | @pbscod) [matches(data(.),"^201286$","i")])))] red-db:get-file returns a sequence of root nodes, i.e. collection('/db/...')/* There is no index defined on the attributes mentioned there... from my experiments, indexes did not seem to help for such case-insensitive regular expression matches. The queries are auto-generated from a more user-friendly query language that is used in the frontend, and most queries do actually contain characters, so the query generator does not check whether case- insensitivity is actually necessary. Another query that got a lot slower is red-db:get-file('praeps') [((exists(darreich/atc[matches(data(.),"^A05A.*$","i")])))] Kind regards, Ralf |
From: Wolfgang M. <wol...@ex...> - 2011-10-29 21:56:54
|
> > Compared to 1.4.x, trunk has some significant changes on the structural > > index, making things significant faster; these changes were not ported > > back to 1.4.x so that might actually explain the differences. > Yes, he query is mostly structural, so that could indeed be the culprit. In 1.5 the structural index has been rewritten completely. It performs a lot better if the context sequence of a path step is rather small. Also ancestor steps, which are used heavily by the xquery optimizer, got a lot faster. Certainly my performance tests cannot cover every use case, so it's possible that the redesign also had negative effects on some queries. But I rather suspect it's some change in the query optimizer which causes the slowdown. Eric has sent us some examples of queries which got slower. We'll work through those to eliminate any regressions performance wise. But feel free to send me some samples as well (even better if I can add them to my benchmark tests). Wolfgang |
From: Ralf J. <ral...@gm...> - 2011-10-30 11:04:04
|
Hi, > In 1.5 the structural index has been rewritten completely. It performs a > lot better if the context sequence of a path step is rather small. Also > ancestor steps, which are used heavily by the xquery optimizer, got a lot > faster. > > Certainly my performance tests cannot cover every use case, so it's > possible that the redesign also had negative effects on some queries. But > I rather suspect it's some change in the query optimizer which causes the > slowdown. > > Eric has sent us some examples of queries which got slower. We'll work > through those to eliminate any regressions performance wise. But feel free > to send me some samples as well (even better if I can add them to my > benchmark tests). The mail was about queries getting slower when I switched from 1.5 back to 1.4, so I doubt you'd want to add them :D Kind regards, Ralf |