Re: [Exist-open] storing/indexing speed

[Wouter H. <ws...@gm...>]

Thanks Wolfgang

Splitting collections should be no problem.

About querying: so I can just query the top collection using a wildcard?
I read somewhere that querying using specific fields with // is faster, but
this was also a 1.4 comment.

I see the 1.4.1 changelog is up again. I will have a look.


2011/2/11 Wolfgang Meier <wol...@ex...>

> You probably have too many documents in a single collection. In 1.4, the
> time to update a single document grows proportionally with the size of the
> collection. This has something to do with the way indexes are organized.
> There are several possible solutions:
>
> 1) split your main collection into several sub-collections. This has no
> effect on queries since exist will always automatically include all
> sub-collections in a query to the main collection.
>
> 2) remove the entire collection before updating it.
>
> 3) use trunk, which addresses those problems and should be many times
> faster when updating large collections. I would actually like to port those
> changes back to 1.4 once I'm sure they have been tested well enough.
>
> Wolfgang
>
> Am 11.02.2011 00:42 schrieb "Wouter Hager" <ws...@gm...>:
>
> > Hi All,
> >
> > For some 4000 files, roughly 200MB, the xdb:store ant task I'm using has
> > been running over 24 hours now. I am using eXist 1.4.1dev-rev13702 out of
> > the box with 2GB of memory. and Lucene configured to use standard
> analyzer
> > only, with 4 qnames out of a dozen specified. I have set nodes-buffer to
> > 1000. The files in the collection need to be overwritten.
> >
> > You probably heard it all before, but is there a way to speed things up a
> > bit? The problem is that read access is slow while the store process is
> > running. Am I doing something wrong perhaps?
> >
> > One final note: there are repeating elements in the docs...
> >
> > I noticed the Lucene buffer is set to 32MB. I read somewhere in the list
> > that increasing it may speed up indexing a bit. This was in 1.4. Any idea
> if
> > this setting has effect in 1.4.1 and what is a good value?
> >
> > Thanks for any input!
> >
> >
> > --
> >
> > W.S. Hager
> > Lagua Web Solutions
> > http://lagua.nl
>



-- 

W.S. Hager
Lagua Web Solutions
http://lagua.nl