Re: [Exist-open] Performance on very large collections

[Adam R. <ada...@de...>]

I can make available my code (but it is linked to our document structure
- although not difficult to change) and there is code on the eXist
patches page for a more generalised version based on my code written by
Patrick Turcotte, but this makes use of JDOM which I wanted to replace
with SAX, I started by refactoring the eXist triggers mechanism to
better support this but got bogged down and ran out of time.

Basically it has two components, a Trigger and an XQuery Extension
module.
The Trigger passes a XML document to Lucene for indexing whenever a new
document is stored in the database or updated. If a document is removed
from the database the Trigger also removes its Lucene index entry.
The XQuery Extension module provides a mechanism to search the Lucene
index from XQuery, it provides an xquery function that simply executes a
lucene query string against the Lucene index and returns pointers to
documents in the database to the XQuery engine which then returns the
documents themselves.

If you are happy with JDOM then Patricks code may do exactly what you
want, otherwise give me a shout and I will send you my code with a bit
of blurb about how to adapt it.

Patricks code -
http://sourceforge.net/tracker/index.php?func=detail&aid=1654205&group_id=17691&atid=317691



Thanks Adam.

On Mon, 2007-08-13 at 12:02 -0400, Fabrizio Gotti wrote:
> Many thanks, Adam,
> 
> I have used Lucene in the past for the very same purpose and found it  
> impressive, and, I think, a natural extension to the eXist engine.
> 
> I would be very interested in trying the "lucenized" version of eXist  
> you mention in your message. Could you send it to me, or make it  
> available for everyone to try?
> 
> Thank you,
> 
> F.Gotti
> 
> 
> On 13 Aug 2007, at 11:55 , Adam Retter wrote:
> 
> > Or if you want really fast text matching but also the power and
> > flexibility of eXist you could use Lucene in combination with  
> > eXist. We
> > created an XQuery extension module for eXist here that uses Lucene  
> > to do
> > very fast text searches, I have made this code available in the  
> > past and
> > other people have also used it...
> >
> > Thanks Adam
> 
-- 
Adam Retter

Principal Developer
Devon Portal Project
Room 310
County Hall
Topsham Road
Exeter
EX2 4QD

t: 01392 38 3683
f: 01392 38 2966
e: ada...@de...
w: www.devonline.gov.uk