Re: [CLucene-dev] help ! - using LanguageBasedAnalyzer/CJKTokenizer returns wrong result

[Ben v. K. <bva...@gm...>]

I'd suggest using Luke to compare what javas behaviour is
On Feb 28, 2012 10:51 PM, "Veit Jahns" <nun...@go...> wrote:

> Hi Vivien!
>
> 2012/2/27 Vivien Meng <V....@qs...>
>
>>  Hi Veit:****
>>
>> ** **
>>
>> ** **
>>
>> Thanks for your reply.****
>>
>> ** **
>>
>> Do you know if there is anyone using CJKTokenizer?
>>
>
> No, I don't know anyone. Sorry.
>
>
>> ****
>>
>> ** **
>>
>> Here is the code, no parsing error that I can see:****
>>
>> ** **
>>
>>     Directory* store = NULL;****
>>
>>     store = _CLNEW RAMDirectory();****
>>
>>     const TCHAR* contentField = _T("contents");****
>>
>>     const TCHAR* test = _T("这里我们所论及的是有儿童参加的最初级足球协会red green");****
>>
>>     const TCHAR* cjk = _T("cjk");****
>>
>>     Document doc;****
>>
>>     ****
>>
>>     IndexWriter* writer = NULL;****
>>
>>     IndexReader* reader = NULL;****
>>
>>       ****
>>
>>     LanguageBasedAnalyzer* an = _CLNEW LanguageBasedAnalyzer(cjk, false);
>> ****
>>
>>     ****
>>
>>     Query* q2 = NULL;****
>>
>>     Hits* h2 = NULL;****
>>
>>     const TCHAR* qry2 = _T("提");****
>>
>> ** **
>>
>>     writer = _CLNEW IndexWriter(store, an, true);****
>>
>>     ****
>>
>>     doc.add(*_CLNEW Field(contentField, test, Field::STORE_YES |
>> Field::INDEX_TOKENIZED));****
>>
>>       ****
>>
>>     writer->addDocument(&doc);****
>>
>>     writer->optimize();****
>>
>>       ****
>>
>>     // Close and clean up****
>>
>>     writer->close();****
>>
>>     _CLLDELETE(writer);****
>>
>>     ****
>>
>>     // verify the result****
>>
>>     reader =  IndexReader::open(store);****
>>
>>     IndexSearcher search(reader);****
>>
>>     ****
>>
>>     q2 = QueryParser::parse(qry2 , contentField, an);****
>>
>>     if ( q2 != NULL )****
>>
>>     {****
>>
>>         h2 = search.search( q2 );****
>>
>>         ****
>>
>>         size_t a = h2->length();****
>>
>>         ****
>>
>>         printf("second query is %d", (int)a);****
>>
>>     }****
>>
>>     ****
>>
>>     reader->close();
>>
>
> Looks good. What will be interesting to know is, what q2->toString()
> returns. Then you see what query is actually used for getting the hits. But
> I will try your code myself also.
>
> Best regards,
>
> Veit
>
>
> ------------------------------------------------------------------------------
> Keep Your Developer Skills Current with LearnDevNow!
> The most comprehensive online learning library for Microsoft developers
> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> Metro Style Apps, more. Free future releases when you subscribe now!
> http://p.sf.net/sfu/learndevnow-d2d
> _______________________________________________
> CLucene-developers mailing list
> CLu...@li...
> https://lists.sourceforge.net/lists/listinfo/clucene-developers
>
>