Menu
▾
▴
Re: [Aperture-devel] Lucene indexing
|
From: Florent J. <flo...@un...> - 2007-04-16 09:40:10
|
Hi,
Leo Sauermann wrote:
> Hi Florent, Aperture,
>
> I added the patch to Aperture-addons.
>
[snip]
Thanks for the pointers for the indexing configuration, I'll look at it.
>> For the LuceneSail for querying, I couldn't find the place where the
>> "matched" property to be queried using lucene are configured. Could
>> you explain me where to change it ?
> I dont understand the question, what do you mean by matched property?
>
I was speaking of luceneSail from Aduna Christiaan Fluit mentioned. I
don't know if it works exactly the same way (and I can't use the one
from Gnowsis since the API changed...)
regards,
Florent
> best
> Leo
>
>>
>> Regards,
>> Florent
>>
>> Leo Sauermann wrote:
>>> Hi Florent, Aperture
>>>
>>> It was Florent Jochaud who said at the right time 04.04.2007 13:07
>>> the following words:
>>>> Hi,
>>>>
>>>>
>>>>
>>>> I had a first look at your code and the produced index.
>>>>
>>>> Seems to work quite well (as far as I know Lucene, and at least,
>>>> the index
>>>> seems to be well populated...)
>>>>
>>>> I'm not sure how to improve it, I see several way:
>>>>
>>>> - Should all field be indexed ? or only some specifics
>>>> ones ? I
>>>> think fulltext, name, keywords, and date should be indexed (not
>>>> even sure
>>>> for the date)
>>>>
>>> for now, all fields should be dumped into lucene. we could make some
>>> config file, disabling properties. this should be done in rdf, for
>>> coolness
>>>
>>>> - Literals should be indexed without their types definition
>>>>
>>>>
>>> try this:
>>> if (node instanceof Literal)
>>> {
>>> string s = ((Literal)node).getValue();
>>>
>>> .... use s...
>>> }
>>>
>>>> - Using URI as field's name make quite longs name, can it
>>>> be a
>>>> problem for Lucene ?
>>>>
>>> The fields contain funny character that lucene seems to hate
>>> (especially #) which cannot be escaped out of queries so easily. but
>>> besides that, I see no problem.
>>>> - If date are indexed, seems that it should be first
>>>> converted (I
>>>> didn't look already the date format in aperture, I'm not quite sure
>>>> if it is
>>>> necessary)
>>>>
>>>>
>>> for a quick start, try RDFTool.toDate (or however this methid is
>>> called, RDFUtil? RDFTool? its a utility class in rdf2go)
>>>
>>>
>>>> - I think other specific lucene's improvement are specific
>>>> for each
>>>> applications, I'm not sure they are needed in this demo Handler...
>>>>
>>>> Did I miss something ?
>>>>
>>>>
>>> nope, you got it right!
>>>
>>> a small search interface with a textfield and a button would rock...
>>>
>>> best
>>> Leo
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Florent
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Von: Leo Sauermann [mailto:leo...@df...] Gesendet:
>>>> Freitag, 30. März 2007 12:56
>>>> An: Florent Jochaud; Aperture Developers
>>>> Betreff: Re: AW: [Aperture-devel] Lucene indexing
>>>>
>>>>
>>>>
>>>> Chris Fluit and we are tiddling out the details for a combination
>>>> of Lucene
>>>> and RDF indexing in one Lucenesail, we have much code and
>>>> experience laying
>>>> around, and its a nifty problem, so I would not want you to get
>>>> dragged into
>>>> it too much.
>>>>
>>>> What we would really need is a "beautifully simple" Lucene
>>>> ApertureCrawlerHandler, that is < 500 lines and shows how blissfull
>>>> Aperture
>>>> is for lucene.
>>>>
>>>> I started a first version, it compiles, it does soemthing,
>>>> though I didn't check what...
>>>> http://aperture.cvs.sourceforge.net/aperture/aperture-addons/src/java/org/se
>>>>
>>>> manticdesktop/aperture/lucenehandler/LuceneHandler.java?view=markup
>>>>
>>>>
>>>> Florent: could you pick up this ball?
>>>>
>>>> best
>>>> Leo
>>>>
>>>>
>>>> It was Florent Jochaud who said at the right time 30.03.2007 10:30 the
>>>> following words:
>>>> Hi,
>>>>
>>>>
>>>>
>>>> I'm not sure to be able to do something in this time, but I will
>>>> try :-) I
>>>> still need to go in details in lucene usage and how to set up the
>>>> index.
>>>>
>>>> Could you explain me more in details what you mean by simple store ?
>>>> Especially where in Aperture you think this storage should be set ?
>>>> I will
>>>> have a look, but I still don't know aperture in details.
>>>>
>>>>
>>>>
>>>> Florent
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Von: Leo Sauermann [mailto:leo...@df...] Gesendet:
>>>> Dienstag, 27. März 2007 17:40
>>>> An: Florent Jochaud
>>>> Betreff: Re: [Aperture-devel] Lucene indexing
>>>>
>>>>
>>>>
>>>> If you do a simple lucene store within the next week, its in the next
>>>> official release of aperture, which will happen sooner or later :-)
>>>>
>>>> this would fit perfect.
>>>>
>>>>
>>>> best
>>>> Leo
>>>>
>>>>
>>>>
>>>> It was Florent Jochaud who said at the right time 27.03.2007 08:54 the
>>>> following words:
>>>> Hi,
>>>>
>>>> Late and short answer : thanks for all theses information, this
>>>> sail seems
>>>> really interesting. I'll have a look at the way to use it for my
>>>> application
>>>> soon.
>>>>
>>>> Regards,
>>>>
>>>> Florent
>>>>
>>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: ape...@li... [mailto:aperture-
>>>> <mailto:dev...@li...>
>>>> dev...@li...] Im Auftrag von Christiaan Fluit
>>>> Gesendet: Freitag, 16. März 2007 11:50
>>>> An: <mailto:ape...@li...>
>>>> ape...@li...
>>>> Betreff: Re: [Aperture-devel] Lucene indexing
>>>>
>>>> Leo Sauermann wrote:
>>>>
>>>> I saw there is plan for a Lucene Sail architecture for storing the
>>>>
>>>> full text
>>>>
>>>> extract of documents.
>>>>
>>>>
>>>> there are two different implementations, one by us for gnowsis, one
>>>>
>>>> by
>>>>
>>>> chris fluit
>>>>
>>>> the one by chris, he knows where it is.
>>>>
>>>> The most uptodate code is located here:
>>>>
>>>> https://src.aduna-software.org/svn/info.aduna/projects/aduna-
>>>> infosource/src/main/java/info/aduna/infosource/local/
>>>>
>>>> This works with Sesame 2 beta 2 and Lucene 2.0.
>>>>
>>>> Note that this demonstrates a very particular way of combining Lucene
>>>> with Sesame. It depends on a virtual "matches" RDF property that will
>>>> be
>>>> evaluated on a Lucene index by the LuceneSail. This yields document
>>>> URIs
>>>> and relevancies that are substituted into the query before being
>>>> passed
>>>> to a wrapped Sail that evaluates it on a "real" RDF store. It does
>>>> *not*
>>>> perform a full-text search on all RDF literals. Also, the LuceneSail
>>>> only handles querying, it does not handle creation and updating of the
>>>> Lucene index.
>>>>
>>>> I plan to do an upgrade to Lucene 2.1 soon, which could make our
>>>> LuceneIndex class simpler.
>>>>
>>>>
>>>> We plan to make a better version integrating Lucene and a SAIL.
>>>> probably some people from nepomuk.semanticdesktop.org will work
>>>>
>>>> together
>>>>
>>>> with chris, the project hasn't started yet, but everyone needs it.
>>>>
>>>> I think I have gathered enough knowledge of both Lucene and Sesame to
>>>> realize the generic full-text indexing and querying Sail that Leo
>>>> mentions, I only need to find some time...
>>>>
>>>>
>>>> Also, is this Sail implementation supposed to be integrated directly
>>>>
>>>> in
>>>>
>>>> aperture one day ? Or the use of RDF2go means that store-specific
>>>>
>>>> components
>>>>
>>>> are "forbidden" ?
>>>>
>>>>
>>>> We could do demos or examples showing how to do it,
>>>> but aperture itself should stay clean of these "forbidden planets".
>>>>
>>>> I think it would be great to have this full-text indexing and
>>>> searching
>>>> Sail as a contribution in Sesame's SVN (projects/sesame-contrib). It
>>>> seems that this is of interest to the Sesame community at large, not
>>>> only for Aperture users.
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Chris
>>>> --
>>>>
>>>> -----------------------------------------------------------------------
>>>>
>>>> --
>>>> Take Surveys. Earn Cash. Influence the Future of IT
>>>> Join SourceForge.net's Techsay panel and you'll get the chance to
>>>> share
>>>> your
>>>> opinions on IT & business topics through brief surveys-and earn cash
>>>> http://www.techsay.com/default.php?page=join.php
>>>> <http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVD>
>>>>
>>>> &p=sourceforge&CID=DEVD
>>>> EV
>>>> _______________________________________________
>>>> Aperture-devel mailing list
>>>> Ape...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/aperture-devel
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------------------------
>>>>
>>>> Take Surveys. Earn Cash. Influence the Future of IT
>>>> Join SourceForge.net's Techsay panel and you'll get the chance to
>>>> share your
>>>> opinions on IT & business topics through brief surveys-and earn cash
>>>> http://www.techsay.com/default.php?page=join.php
>>>> <http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV>
>>>>
>>>> &p=sourceforge&CID=DEVDEV
>>>> _______________________________________________
>>>> Aperture-devel mailing list
>>>> Ape...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/aperture-devel
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>>
>>>> -------------------------------------------------------------------------
>>>>
>>>> Take Surveys. Earn Cash. Influence the Future of IT
>>>> Join SourceForge.net's Techsay panel and you'll get the chance to
>>>> share your
>>>> opinions on IT & business topics through brief surveys-and earn cash
>>>> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>>
>>>> _______________________________________________
>>>> Aperture-devel mailing list
>>>> Ape...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/aperture-devel
>>>>
>>
>
>
> --
> ____________________________________________________
> DI Leo Sauermann http://www.dfki.de/~sauermann
>
> Deutsches Forschungszentrum fuer
> Kuenstliche Intelligenz DFKI GmbH
> Trippstadter Strasse 122
> P.O. Box 2080 Fon: +49 631 20575-116
> D-67663 Kaiserslautern Fax: +49 631 20575-102
> Germany Mail: leo...@df...
>
> Geschaeftsfuehrung:
> Prof.Dr.Dr.h.c.mult. Wolfgang Wahlster (Vorsitzender)
> Dr. Walter Olthoff
> Vorsitzender des Aufsichtsrats:
> Prof. Dr. h.c. Hans A. Aukes
> Amtsgericht Kaiserslautern, HRB 2313
> ____________________________________________________
>
|