Re: [Exist-open] Data index corruption when xpathing on whole dbcollection

[Wolfgang M. <wol...@ex...>]

Hi Nathan, 
> The only thing i have noticed when using xquery profiler is that xpath 
> expression didn't use index when running.
> We have found a way to force xpath expression to use indexes.
> 
> Do you think this issue could be the cause of data corruption?
If the query engine has to scan a large number of documents instead of using an index, memory consumption can increase considerably, potentially resulting in out of memory errors and throwing the db into an inconsistent state. For sure out of memory errors should be avoided and it is important to identify any bottleneck expression.
> On a second hand, in order to speed up index research, we are looking
> for other lines of analysis and we wonder if there wouldn't be a better 
> index than range index to deal with xpath value like 
> [value="INV_20130201_XXXXXXX"]?

It depends on the context. Is the value you are trying to look up quite frequent? Is the filter applied to a larger subset of the db or just to a few ancestor elements? If the context sequence is small and you need to do a lot of processing on single items, it can have benefits to force the root item into memory first (using util:expand). However, this really depends on the context of the expression.

I believe I still have some backups of your test data set somewhere, so if you tell me the concrete expression you are trying to optimize, I could have a closer look.
> Many thanks in advance for your help and I wish you a good easter weekend.

I actually reserved the following days to work on a rewrite of the range index which will further improve optimizations on simple key/value lookup expressions ;-)

Best,

Wolfgang