[Exist-open] Setting up a new Lucene tokenizer/analyzer

[Efraim F. <efr...@gm...>]

Hi,

I'm having some issues setting up a custom tokenizer/analyzer in eXist, 
and I wanted to know if I'm doing anything obviously wrong.

Here is what I did:
Download and compile hebmorph and its java Lucene analyzer from git 
(https://github.com/synhershko/HebMorph).
start with an install from a clean copy of eXist 1.5 r14395
copy lucene.hebrew.jar to $EXIST_HOME/extensions/indexes/lucene/lib/

In conf.xml, change the indexer line to:
<indexer caseSensitive="yes" index-depth="5" 
preserve-whitespace-mixed-content="no"
         stemming="no" suppress-whitespace="none"
         tokenizer="org.apache.lucene.analysis.hebrew.HebrewTokenizer"
         track-term-freq="yes">

In the database's /db/system/config/db/collection.xconf, I set:
<collection xmlns="http://exist-db.org/collection-config/1.0">
<index xmlns:tei="http://www.tei-c.org/ns/1.0" 
xmlns:j="http://jewishliturgy.org/ns/jlptei/1.0">
<fulltext default="none" attributes="no"/>
<lucene>
<analyzer class="org.apache.lucene.analysis.hebrew.MorphAnalyzer"/>
<inline qname="tei:c"/>
<text qname="tei:title"/>
<text qname="j:repository"/>
<text qname="tei:seg"/>
</lucene>
</index>
</collection>
Add some texts to the database, under the /db/text collection
 From the eXist Java client, run the XQuery:
declare namespace tei="http://www.tei-c.org/ns/1.0";
collection('/db/text')//tei:title[ft:query(.,'בראשית')]

The result is a null pointer exception, where the stack trace looks like 
it never tries to use the custom tokenizer or analyzer:
org.xmldb.api.base.XMLDBException
     at 
org.exist.xmldb.LocalXPathQueryService.execute(LocalXPathQueryService.java:398)
     at 
org.exist.xmldb.LocalXPathQueryService.execute(LocalXPathQueryService.java:145)
     at org.exist.client.QueryDialog$QueryThread.run(QueryDialog.java:531)
Caused by: java.lang.NullPointerException
     at org.apache.lucene.analysis.Tokenizer.close(Tokenizer.java:71)
     at org.apache.lucene.analysis.TokenFilter.close(TokenFilter.java:46)
     at 
org.apache.lucene.queryParser.QueryParser.getFieldQuery(QueryParser.java:619)
     at 
org.apache.lucene.queryParser.QueryParser.Term(QueryParser.java:1449)
     at 
org.apache.lucene.queryParser.QueryParser.Clause(QueryParser.java:1337)
     at 
org.apache.lucene.queryParser.QueryParser.Query(QueryParser.java:1265)
     at 
org.apache.lucene.queryParser.QueryParser.TopLevelQuery(QueryParser.java:1254)
     at 
org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:200)
     at 
org.exist.indexing.lucene.LuceneIndexWorker.query(LuceneIndexWorker.java:352)
     at org.exist.xquery.modules.lucene.Query.preSelect(Query.java:198)
     at org.exist.xquery.pragmas.Optimize.eval(Optimize.java:118)
     at 
org.exist.xquery.ExtensionExpression.eval(ExtensionExpression.java:70)
     at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:70)
     at org.exist.xquery.PathExpr.eval(PathExpr.java:243)
     at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:70)
     at org.exist.xquery.XQuery.execute(XQuery.java:246)
     at org.exist.xquery.XQuery.execute(XQuery.java:201)
     at 
org.exist.xmldb.LocalXPathQueryService.execute(LocalXPathQueryService.java:391)
     ... 2 more
Caused by: java.lang.NullPointerException
     at org.apache.lucene.analysis.Tokenizer.close(Tokenizer.java:71)
     at org.apache.lucene.analysis.TokenFilter.close(TokenFilter.java:46)
     at 
org.apache.lucene.queryParser.QueryParser.getFieldQuery(QueryParser.java:619)
     at 
org.apache.lucene.queryParser.QueryParser.Term(QueryParser.java:1449)
     at 
org.apache.lucene.queryParser.QueryParser.Clause(QueryParser.java:1337)
     at 
org.apache.lucene.queryParser.QueryParser.Query(QueryParser.java:1265)
     at 
org.apache.lucene.queryParser.QueryParser.TopLevelQuery(QueryParser.java:1254)
     at 
org.apache.lucene.queryParser.QueryParser.parse(QueryParser.java:200)
     at 
org.exist.indexing.lucene.LuceneIndexWorker.query(LuceneIndexWorker.java:352)
     at org.exist.xquery.modules.lucene.Query.preSelect(Query.java:198)
     at org.exist.xquery.pragmas.Optimize.eval(Optimize.java:118)
     at 
org.exist.xquery.ExtensionExpression.eval(ExtensionExpression.java:70)
     at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:70)
     at org.exist.xquery.PathExpr.eval(PathExpr.java:243)
     at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:70)
     at org.exist.xquery.XQuery.execute(XQuery.java:246)
     at org.exist.xquery.XQuery.execute(XQuery.java:201)
     at 
org.exist.xmldb.LocalXPathQueryService.execute(LocalXPathQueryService.java:391)
     at 
org.exist.xmldb.LocalXPathQueryService.execute(LocalXPathQueryService.java:145)
     at org.exist.client.QueryDialog$QueryThread.run(QueryDialog.java:531)


Am I doing anything obviously wrong from the eXist side?

Thanks,

-- 
---
Efraim Feinstein
Lead Developer
Open Siddur Project
http://opensiddur.net
http://wiki.jewishliturgy.org