clucene-developers — CLucene development-related discussion.

[CLucene-dev] actual deletion of documents

[theorist <eg...@ma...>]

hello,

i can still read documents in my index which have been deleted with IndexModifier::deleteDocuments even after flushing, optimizing, closing the index and then reopening it with IndexReader. what's wrong?

thanks.

Re: [CLucene-dev] live query

[Ben v. K. <bva...@gm...>]

Another approach would be to create your indexes to a temporary location and
search that location for your live query before merging into the main index.

Ben

On Sun, Aug 7, 2011 at 9:05 AM, Itamar Syn-Hershko <it...@co...>wrote:

> There isn't such thing built into clucene nor Java Lucene. You are going to
> have to keep a list of document IDs that once matched a query, and to
> perform searches in the background every now and then with that document ID
> in it (use your IDs, not Lucene's internal docids).
>
>
> On Sat, Aug 6, 2011 at 9:12 AM, Clemens <cz...@au...> wrote:
>
>> Hi,
>>
>> not sure if this question better goes to the java lucene mailing list but
>> you probably can help me too.
>>
>> I'm using clucene to implement a desktop search engine for the Haiku OS. I
>> like to notify the user when a new document matches an existing query or a
>> document not match anymore. Is there something like this build in?
>> Otherwise I just need to check if an modified document matches a query.
>>
>> How can I check if a document satisfy a query? Or do I have to modify the
>> query to only search a special document? What is the best to do it?
>>
>> thank you,
>>        Clemens
>>
>>
>> ------------------------------------------------------------------------------
>> BlackBerry&reg; DevCon Americas, Oct. 18-20, San Francisco, CA
>> The must-attend event for mobile developers. Connect with experts.
>> Get tools for creating Super Apps. See the latest technologies.
>> Sessions, hands-on labs, demos & much more. Register early & save!
>> http://p.sf.net/sfu/rim-blackberry-1
>> _______________________________________________
>> CLucene-developers mailing list
>> CLu...@li...
>> https://lists.sourceforge.net/lists/listinfo/clucene-developers
>>
>>
>
>
> ------------------------------------------------------------------------------
> BlackBerry&reg; DevCon Americas, Oct. 18-20, San Francisco, CA
> The must-attend event for mobile developers. Connect with experts.
> Get tools for creating Super Apps. See the latest technologies.
> Sessions, hands-on labs, demos & much more. Register early & save!
> http://p.sf.net/sfu/rim-blackberry-1
> _______________________________________________
> CLucene-developers mailing list
> CLu...@li...
> https://lists.sourceforge.net/lists/listinfo/clucene-developers
>
>

Re: [CLucene-dev] live query

[Itamar Syn-H. <it...@co...>]

There isn't such thing built into clucene nor Java Lucene. You are going to
have to keep a list of document IDs that once matched a query, and to
perform searches in the background every now and then with that document ID
in it (use your IDs, not Lucene's internal docids).

On Sat, Aug 6, 2011 at 9:12 AM, Clemens <cz...@au...> wrote:

> Hi,
>
> not sure if this question better goes to the java lucene mailing list but
> you probably can help me too.
>
> I'm using clucene to implement a desktop search engine for the Haiku OS. I
> like to notify the user when a new document matches an existing query or a
> document not match anymore. Is there something like this build in?
> Otherwise I just need to check if an modified document matches a query.
>
> How can I check if a document satisfy a query? Or do I have to modify the
> query to only search a special document? What is the best to do it?
>
> thank you,
>        Clemens
>
>
> ------------------------------------------------------------------------------
> BlackBerry&reg; DevCon Americas, Oct. 18-20, San Francisco, CA
> The must-attend event for mobile developers. Connect with experts.
> Get tools for creating Super Apps. See the latest technologies.
> Sessions, hands-on labs, demos & much more. Register early & save!
> http://p.sf.net/sfu/rim-blackberry-1
> _______________________________________________
> CLucene-developers mailing list
> CLu...@li...
> https://lists.sourceforge.net/lists/listinfo/clucene-developers
>
>

[CLucene-dev] TermDocs methods

[theorist <eg...@ma...>]

sorry, i just made a simple syntax error. now it works fine.

[CLucene-dev] TermDocs methods

[theorist <eg...@ma...>]

hello,

what header file shall i include to get rid of this:

request for member ‘next’ in ‘documents’, which is of non-class type ‘lucene::index::TermDocs*’
request for member ‘doc’ in ‘documents’, which is of non-class type ‘lucene::index::TermDocs*’ ?

Terms.h has been already included.

thanks

[CLucene-dev] live query

[Clemens <cz...@au...>]

Hi,

not sure if this question better goes to the java lucene mailing list but
you probably can help me too.

I'm using clucene to implement a desktop search engine for the Haiku OS. I
like to notify the user when a new document matches an existing query or a
document not match anymore. Is there something like this build in?
Otherwise I just need to check if an modified document matches a query.

How can I check if a document satisfy a query? Or do I have to modify the
query to only search a special document? What is the best to do it?

thank you,
	Clemens

Re: [CLucene-dev] CLucene hackathon August 1-8. Goal: completing transition to new code-base

[Itamar Syn-H. <it...@co...>]

Just as a reminder - the CLucene hackathon is taking place now until next
week.

During this week we will be working on perfecting the LPP core, and making
it ready to be assimilated into being the new CLucene core. Following some
discussions, here are the main tasks we will be working on:

   - The biggest task - relaxing LPP's massive usage of boost::shared_ptr.
   shared_ptr uses atomic ref-counting, and as such is quite slow. The idea is
   to use raw pointers (with ownership transfers where necessary), scoped_ptrs
   or auto_ptrs instead whenever possible, without introducing memory leaks. We
   are still spec'ing this change on Skype and Google Wave, to make sure it is
   done correctly and the resulting API is still clean and easy to use. Feel
   free to ask to join the discussion.
   - Relaxing LPP's dependencies wherever possible, and making it build
   faster.
   - Massive profiling and benchmarking, to identify bottlenecks and fix
   them.
   - Surveying for memory leaks.
   - Testing on more exotic platforms.
   - Looking into creating wrappers for high-level languages and frameworks
   (Java, .NET, PHP, Perl,...).

We welcome any help.

Currently we have 7 active participants. From CLucene / LPP core: Ben, Veit,
Alan and I. We have 2 more friends from http://www.deviantart.com, and
another one from the Tomahawk player dev team.

Come say hello at #clucene-hackathon on Freenode. If we aren't answering,
thats because we are working :)

Itamar.

On Sun, Jul 17, 2011 at 11:16 PM, Itamar Syn-Hershko <it...@co...>wrote:

> Hi all,
>
> Following (quite) a recent discussion in the mailing list, we are now ready
> to begin and hopefully complete the transition to the new code base. To do
> that, we are hosting a virtual hackathon starting August 1st. It will be
> hosted on IRC (Freenode): #clucene-hackathon .
>
> For reference, here's the discussion we had:
> http://comments.gmane.org/gmane.comp.jakarta.lucene.clucene.devel/3936
>
> The goal in this hackathon is to minimize LPP (
> https://github.com/luceneplusplus/LucenePlusPlus) and make sure it runs
> smoothly on all platforms we can test on. We are likely to make intrusive
> changes to the library, and to remove some Boost usage in places where it
> isn't really needed. This is definitely something we can achieve in a week
> worth of work.
>
> You all are welcome to participate - please let us know in advance if
> you'll be willing to take significant part in this so we can delegate tasks
> appropriately.
>
> I'll be available on Skype and e-mail as well all throughout that week
> (before and after too).
>
> See you there,
>
> Itamar.
>

Re: [CLucene-dev] IndexSearcher initialization locks Index?

[Itamar Syn-H. <it...@co...>]

I'm pretty sure IndexSearcher does not *lock* the index, but rather waits on
the lock to be released. That lock is probably being set by the IW you have
running periodically. IS has setTimeout you can use.

As I said in a previous thread, we are moving forward to a more stable and
modern code base, so unless there's a test case and this is an issue that is
easy to resolve I'm afraid it will be in our backburner.

Sorry about the (temporary) inconvenience.

On Mon, Aug 1, 2011 at 11:09 PM, Andrew Victor <avi...@gm...> wrote:

> hi,
>
> > I'm not sure I'm following what you're doing with those locks
>
> Since I'm going to "swap" the IndexSearcher pointer for a new one I
> need to make sure no other thread is currently using that pointer.
> That's why the locks are needed.
>
>
> > You can safely access one index searcher from multiple threads, but
> > IndexWriter only from one writing thread. The thread accessing IW can
> work
> > in parallel to searching threads.
>
> That part is working fine.
>
> The issue is:
>  IndexSearcher* newSearcher = new IndexSearcher("index directory")
> is blocking other IndexSearcher's.
>
>
> Regards,
>  Andrew Victor
>
>
> ------------------------------------------------------------------------------
> BlackBerry&reg; DevCon Americas, Oct. 18-20, San Francisco, CA
> The must-attend event for mobile developers. Connect with experts.
> Get tools for creating Super Apps. See the latest technologies.
> Sessions, hands-on labs, demos & much more. Register early & save!
> http://p.sf.net/sfu/rim-blackberry-1
> _______________________________________________
> CLucene-developers mailing list
> CLu...@li...
> https://lists.sourceforge.net/lists/listinfo/clucene-developers
>
>

Re: [CLucene-dev] IndexSearcher initialization locks Index?

[Andrew V. <avi...@gm...>]

hi,

> I'm not sure I'm following what you're doing with those locks

Since I'm going to "swap" the IndexSearcher pointer for a new one I
need to make sure no other thread is currently using that pointer.
That's why the locks are needed.


> You can safely access one index searcher from multiple threads, but
> IndexWriter only from one writing thread. The thread accessing IW can work
> in parallel to searching threads.

That part is working fine.

The issue is:
  IndexSearcher* newSearcher = new IndexSearcher("index directory")
is blocking other IndexSearcher's.


Regards,
  Andrew Victor

Re: [CLucene-dev] IndexSearcher initialization locks Index?

[Itamar Syn-H. <it...@co...>]

I'm not sure I'm following what you're doing with those locks

You can safely access one index searcher from multiple threads, but
IndexWriter only from one writing thread. The thread accessing IW can work
in parallel to searching threads.

On Mon, Aug 1, 2011 at 10:29 PM, Andrew Victor <avi...@gm...> wrote:

> hi,
>
> We have quite a big index, but each document is pretty small.  (So +-
> 30 million documents in 3.2 Gb),
>
> In our software, all updates are via a one IndexWriter, and we have
> multiple threads using a single IndexSearcher.
> What I'm noticing that when a 2nd IndexSearcher is opened, the
> existing IndexSearcher occasionally blocks for a couple of seconds (+-
> 20-40).
>
> We pull updated and new entries (10000 - 40000) into the index every 5
> minutes.
> For each:
>   indexWriter->deleteDocuments( new Term( "a stored / untokenized key
> field" ) )
>   indexWriter->addDocument()
>
> Once the update is complete:
>   indexWriter->flush()
>
>   /* re-open the IndexSearcher to make the changes visible */
>   newSearcher = new IndexSearcher("index directory")   /* *** */
>
>   acquire_write_lock...
>       oldSearcher = indexSearcher;
>       indexSearcher = newSearcher;
>   release_write_lock..
>
>   /* cleanup */
>   oldSearcher->close();
>   delete oldSearcher;
>
> The searching threads that are "blocked" out of the index are just
> performing:
>   acquire_read_lock....
>        indexSearcher->search(BooleanQuery)
>        /* process hits .... */
>    release_read_lock
>
>
> Is there a way to stop a "new IndexSearcher()" locking the index?
> Or any suggestions to limit the amount of time the index is locked, or
> better ways of implementing this?
>
>
> Regards,
>  Andrew Victor
>
>
> ------------------------------------------------------------------------------
> BlackBerry&reg; DevCon Americas, Oct. 18-20, San Francisco, CA
> The must-attend event for mobile developers. Connect with experts.
> Get tools for creating Super Apps. See the latest technologies.
> Sessions, hands-on labs, demos & much more. Register early & save!
> http://p.sf.net/sfu/rim-blackberry-1
> _______________________________________________
> CLucene-developers mailing list
> CLu...@li...
> https://lists.sourceforge.net/lists/listinfo/clucene-developers
>
>

[CLucene-dev] IndexSearcher initialization locks Index?

[Andrew V. <avi...@gm...>]

hi,

We have quite a big index, but each document is pretty small.  (So +-
30 million documents in 3.2 Gb),

In our software, all updates are via a one IndexWriter, and we have
multiple threads using a single IndexSearcher.
What I'm noticing that when a 2nd IndexSearcher is opened, the
existing IndexSearcher occasionally blocks for a couple of seconds (+-
20-40).

We pull updated and new entries (10000 - 40000) into the index every 5 minutes.
For each:
   indexWriter->deleteDocuments( new Term( "a stored / untokenized key
field" ) )
   indexWriter->addDocument()

Once the update is complete:
   indexWriter->flush()

   /* re-open the IndexSearcher to make the changes visible */
   newSearcher = new IndexSearcher("index directory")   /* *** */

   acquire_write_lock...
       oldSearcher = indexSearcher;
       indexSearcher = newSearcher;
   release_write_lock..

   /* cleanup */
   oldSearcher->close();
   delete oldSearcher;

The searching threads that are "blocked" out of the index are just performing:
   acquire_read_lock....
        indexSearcher->search(BooleanQuery)
        /* process hits .... */
    release_read_lock


Is there a way to stop a "new IndexSearcher()" locking the index?
Or any suggestions to limit the amount of time the index is locked, or
better ways of implementing this?


Regards,
 Andrew Victor

Re: [CLucene-dev] Crash with IndexReader reopen

[Itamar Syn-H. <it...@co...>]

Can you create a failing test?

On a side note, next week we will be working on the new CLucene code-base,
so hopefully we will have a newer and better version supported soon.

On Fri, Jul 29, 2011 at 12:51 PM, Andrew Victor <avi...@gm...>wrote:

> hi,
>
> I'm consistently having this crash when trying to re-open an existing
> IndexReader.
>
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7fffe6611700 (LWP 27814)]
> 0x00007ffff673fd77 in std::_Rb_tree<wchar_t const*, std::pair<wchar_t
> const* const, lucene::index::SegmentReader::Norm*>,
> std::_Select1st<std::pair<wchar_t const* const,
> lucene::index::SegmentReader::Norm*> >, lucene::util::Compare::WChar,
> std::allocator<std::pair<wchar_t const* const,
> lucene::index::SegmentReader::Norm*> > >::_M_begin
> (this=0x7fffe01c0660, directory=<value optimized out>,
>    infos=<value optimized out>, closeDirectory=<value optimized out>,
> oldReaders=<value optimized out>, oldStarts=<value optimized out>,
> oldNormsCache=0x62f870) at /usr/include/c++/4.4/bits/stl_tree.h:488
> 488               (this->_M_impl._M_header._M_parent);
> (gdb) bt
> #0  0x00007ffff673fd77 in std::_Rb_tree<wchar_t const*,
> std::pair<wchar_t const* const, lucene::index::SegmentReader::Norm*>,
> std::_Select1st<std::pair<wchar_t const* const,
> lucene::index::SegmentReader::Norm*> >, lucene::util::Compare::WChar,
> std::allocator<std::pair<wchar_t const* const,
> lucene::index::SegmentReader::Norm*> > >::_M_begin
> (this=0x7fffe01c0660, directory=<value optimized out>,
>    infos=<value optimized out>, closeDirectory=<value optimized out>,
> oldReaders=<value optimized out>, oldStarts=<value optimized out>,
> oldNormsCache=0x62f870) at /usr/include/c++/4.4/bits/stl_tree.h:488
> #1  std::_Rb_tree<wchar_t const*, std::pair<wchar_t const* const,
> lucene::index::SegmentReader::Norm*>,
> std::_Select1st<std::pair<wchar_t const* const,
> lucene::index::SegmentReader::Norm*> >, lucene::util::Compare::WChar,
> std::allocator<std::pair<wchar_t const* const,
> lucene::index::SegmentReader::Norm*> > >::find (this=0x7fffe01c0660,
> directory=<value optimized out>, infos=<value optimized out>,
>    closeDirectory=<value optimized out>, oldReaders=<value optimized
> out>, oldStarts=<value optimized out>, oldNormsCache=0x62f870) at
> /usr/include/c++/4.4/bits/stl_tree.h:1434
> #2  std::map<wchar_t const*, lucene::index::SegmentReader::Norm*,
> lucene::util::Compare::WChar, std::allocator<std::pair<wchar_t const*
> const, lucene::index::SegmentReader::Norm*> > >::find (
>    this=0x7fffe01c0660, directory=<value optimized out>, infos=<value
> optimized out>, closeDirectory=<value optimized out>,
> oldReaders=<value optimized out>, oldStarts=<value optimized out>,
>    oldNormsCache=0x62f870) at /usr/include/c++/4.4/bits/stl_map.h:674
> #3  lucene::util::__CLMap<wchar_t const*,
> lucene::index::SegmentReader::Norm*, std::map<wchar_t const*,
> lucene::index::SegmentReader::Norm*, lucene::util::Compare::WChar,
> std::allocator<std::pair<wchar_t const* const,
> lucene::index::SegmentReader::Norm*> > >,
> lucene::util::Deletor::Dummy, lucene::index::SegmentReader::Norm>::get
> (this=0x7fffe01c0660, directory=<value optimized out>, infos=<value
> optimized out>,
>    closeDirectory=<value optimized out>, oldReaders=<value optimized
> out>, oldStarts=<value optimized out>, oldNormsCache=0x62f870) at
> /home/andrew/tmp/clucene-HEAD-772481c/src/core/CLucene/util/VoidMap.h:84
> #4  MultiSegmentReader (this=0x7fffe01c0660, directory=<value
> optimized out>, infos=<value optimized out>, closeDirectory=<value
> optimized out>, oldReaders=<value optimized out>,
>    oldStarts=<value optimized out>, oldNormsCache=0x62f870) at
>
> /home/andrew/tmp/clucene-HEAD-772481c/src/core/CLucene/index/MultiSegmentReader.cpp:163
> #5  0x00007ffff6740502 in lucene::index::MultiSegmentReader::doReopen
> (this=<value optimized out>, infos=0x7fffe01c1940)
>    at
> /home/andrew/tmp/clucene-HEAD-772481c/src/core/CLucene/index/MultiSegmentReader.cpp:207
> #6  0x00007ffff67235e4 in
> lucene::index::DirectoryIndexReader::FindSegmentsFile_Reopen::doBody
> (this=0x7fffe66107f0, segmentFileName=0x7fffe01c0888 "segments_h")
>    at
> /home/andrew/tmp/clucene-HEAD-772481c/src/core/CLucene/index/DirectoryIndexReader.cpp:185
> #7  0x00007ffff672343f in
>
> lucene::index::SegmentInfos::FindSegmentsFile<lucene::index::DirectoryIndexReader*>::tryDoBody
> (this=0x62fdc0, segmentFileName=0x0, ret_err=...)
>    at
> /home/andrew/tmp/clucene-HEAD-772481c/src/core/CLucene/index/_SegmentInfos.h:485
> #8  0x00007ffff66ebc64 in
> lucene::index::SegmentInfos::_FindSegmentsFile::doRun
> (this=0x7fffe66107f0) at
>
> /home/andrew/tmp/clucene-HEAD-772481c/src/core/CLucene/index/SegmentInfos.cpp:1046
> #9  0x00007ffff6722b78 in
>
> lucene::index::SegmentInfos::FindSegmentsFile<lucene::index::DirectoryIndexReader*>::run
> (this=0x62f810)
>    at
> /home/andrew/tmp/clucene-HEAD-772481c/src/core/CLucene/index/_SegmentInfos.h:508
> #10 lucene::index::DirectoryIndexReader::reopen (this=0x62f810) at
>
> /home/andrew/tmp/clucene-HEAD-772481c/src/core/CLucene/index/DirectoryIndexReader.cpp:214
>
>
>
> The code is basically performing:
>
>    IndexReader* reader = searcher->getReader();
>    IndexReader* newReader = reader->reopen();
>    if (newReader != reader) {
>            reader->close();
>            delete reader;
>
>            searcher->close();
>            delete searcher;
>
>            searcher = new IndexSearcher(newReader);
>    }
>
> Any ideas?
>
>
> Regards,
>  Andrew Victor
>
>
> ------------------------------------------------------------------------------
> Got Input?   Slashdot Needs You.
> Take our quick survey online.  Come on, we don't ask for help often.
> Plus, you'll get a chance to win $100 to spend on ThinkGeek.
> http://p.sf.net/sfu/slashdot-survey
> _______________________________________________
> CLucene-developers mailing list
> CLu...@li...
> https://lists.sourceforge.net/lists/listinfo/clucene-developers
>
>

[CLucene-dev] Crash with IndexReader reopen

[Andrew V. <avi...@gm...>]

hi,

I'm consistently having this crash when trying to re-open an existing
IndexReader.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe6611700 (LWP 27814)]
0x00007ffff673fd77 in std::_Rb_tree<wchar_t const*, std::pair<wchar_t
const* const, lucene::index::SegmentReader::Norm*>,
std::_Select1st<std::pair<wchar_t const* const,
lucene::index::SegmentReader::Norm*> >, lucene::util::Compare::WChar,
std::allocator<std::pair<wchar_t const* const,
lucene::index::SegmentReader::Norm*> > >::_M_begin
(this=0x7fffe01c0660, directory=<value optimized out>,
    infos=<value optimized out>, closeDirectory=<value optimized out>,
oldReaders=<value optimized out>, oldStarts=<value optimized out>,
oldNormsCache=0x62f870) at /usr/include/c++/4.4/bits/stl_tree.h:488
488		  (this->_M_impl._M_header._M_parent);
(gdb) bt
#0  0x00007ffff673fd77 in std::_Rb_tree<wchar_t const*,
std::pair<wchar_t const* const, lucene::index::SegmentReader::Norm*>,
std::_Select1st<std::pair<wchar_t const* const,
lucene::index::SegmentReader::Norm*> >, lucene::util::Compare::WChar,
std::allocator<std::pair<wchar_t const* const,
lucene::index::SegmentReader::Norm*> > >::_M_begin
(this=0x7fffe01c0660, directory=<value optimized out>,
    infos=<value optimized out>, closeDirectory=<value optimized out>,
oldReaders=<value optimized out>, oldStarts=<value optimized out>,
oldNormsCache=0x62f870) at /usr/include/c++/4.4/bits/stl_tree.h:488
#1  std::_Rb_tree<wchar_t const*, std::pair<wchar_t const* const,
lucene::index::SegmentReader::Norm*>,
std::_Select1st<std::pair<wchar_t const* const,
lucene::index::SegmentReader::Norm*> >, lucene::util::Compare::WChar,
std::allocator<std::pair<wchar_t const* const,
lucene::index::SegmentReader::Norm*> > >::find (this=0x7fffe01c0660,
directory=<value optimized out>, infos=<value optimized out>,
    closeDirectory=<value optimized out>, oldReaders=<value optimized
out>, oldStarts=<value optimized out>, oldNormsCache=0x62f870) at
/usr/include/c++/4.4/bits/stl_tree.h:1434
#2  std::map<wchar_t const*, lucene::index::SegmentReader::Norm*,
lucene::util::Compare::WChar, std::allocator<std::pair<wchar_t const*
const, lucene::index::SegmentReader::Norm*> > >::find (
    this=0x7fffe01c0660, directory=<value optimized out>, infos=<value
optimized out>, closeDirectory=<value optimized out>,
oldReaders=<value optimized out>, oldStarts=<value optimized out>,
    oldNormsCache=0x62f870) at /usr/include/c++/4.4/bits/stl_map.h:674
#3  lucene::util::__CLMap<wchar_t const*,
lucene::index::SegmentReader::Norm*, std::map<wchar_t const*,
lucene::index::SegmentReader::Norm*, lucene::util::Compare::WChar,
std::allocator<std::pair<wchar_t const* const,
lucene::index::SegmentReader::Norm*> > >,
lucene::util::Deletor::Dummy, lucene::index::SegmentReader::Norm>::get
(this=0x7fffe01c0660, directory=<value optimized out>, infos=<value
optimized out>,
    closeDirectory=<value optimized out>, oldReaders=<value optimized
out>, oldStarts=<value optimized out>, oldNormsCache=0x62f870) at
/home/andrew/tmp/clucene-HEAD-772481c/src/core/CLucene/util/VoidMap.h:84
#4  MultiSegmentReader (this=0x7fffe01c0660, directory=<value
optimized out>, infos=<value optimized out>, closeDirectory=<value
optimized out>, oldReaders=<value optimized out>,
    oldStarts=<value optimized out>, oldNormsCache=0x62f870) at
/home/andrew/tmp/clucene-HEAD-772481c/src/core/CLucene/index/MultiSegmentReader.cpp:163
#5  0x00007ffff6740502 in lucene::index::MultiSegmentReader::doReopen
(this=<value optimized out>, infos=0x7fffe01c1940)
    at /home/andrew/tmp/clucene-HEAD-772481c/src/core/CLucene/index/MultiSegmentReader.cpp:207
#6  0x00007ffff67235e4 in
lucene::index::DirectoryIndexReader::FindSegmentsFile_Reopen::doBody
(this=0x7fffe66107f0, segmentFileName=0x7fffe01c0888 "segments_h")
    at /home/andrew/tmp/clucene-HEAD-772481c/src/core/CLucene/index/DirectoryIndexReader.cpp:185
#7  0x00007ffff672343f in
lucene::index::SegmentInfos::FindSegmentsFile<lucene::index::DirectoryIndexReader*>::tryDoBody
(this=0x62fdc0, segmentFileName=0x0, ret_err=...)
    at /home/andrew/tmp/clucene-HEAD-772481c/src/core/CLucene/index/_SegmentInfos.h:485
#8  0x00007ffff66ebc64 in
lucene::index::SegmentInfos::_FindSegmentsFile::doRun
(this=0x7fffe66107f0) at
/home/andrew/tmp/clucene-HEAD-772481c/src/core/CLucene/index/SegmentInfos.cpp:1046
#9  0x00007ffff6722b78 in
lucene::index::SegmentInfos::FindSegmentsFile<lucene::index::DirectoryIndexReader*>::run
(this=0x62f810)
    at /home/andrew/tmp/clucene-HEAD-772481c/src/core/CLucene/index/_SegmentInfos.h:508
#10 lucene::index::DirectoryIndexReader::reopen (this=0x62f810) at
/home/andrew/tmp/clucene-HEAD-772481c/src/core/CLucene/index/DirectoryIndexReader.cpp:214



The code is basically performing:

    IndexReader* reader = searcher->getReader();
    IndexReader* newReader = reader->reopen();
    if (newReader != reader) {
            reader->close();
            delete reader;

            searcher->close();
            delete searcher;

            searcher = new IndexSearcher(newReader);
    }

Any ideas?


Regards,
  Andrew Victor

[CLucene-dev] CLucene hackathon August 1-8. Goal: completing transition to new code-base

[Itamar Syn-H. <it...@co...>]

Hi all,

Following (quite) a recent discussion in the mailing list, we are now ready
to begin and hopefully complete the transition to the new code base. To do
that, we are hosting a virtual hackathon starting August 1st. It will be
hosted on IRC (Freenode): #clucene-hackathon .

For reference, here's the discussion we had:
http://comments.gmane.org/gmane.comp.jakarta.lucene.clucene.devel/3936

The goal in this hackathon is to minimize LPP (
https://github.com/luceneplusplus/LucenePlusPlus) and make sure it runs
smoothly on all platforms we can test on. We are likely to make intrusive
changes to the library, and to remove some Boost usage in places where it
isn't really needed. This is definitely something we can achieve in a week
worth of work.

You all are welcome to participate - please let us know in advance if you'll
be willing to take significant part in this so we can delegate tasks
appropriately.

I'll be available on Skype and e-mail as well all throughout that week
(before and after too).

See you there,

Itamar.

Re: [CLucene-dev] StandardAnalyzer is buggy

[Timo S. <ts...@ik...>]

On Sat, 2011-07-02 at 08:59 +0300, Timo Sirainen wrote:
> Using the 2_3_2 git version and StandardAnalyzer: It looks like if I
> index document1 and then document2, at least some of document1's tokens
> go to document2 as well. If I index document3, it indexes document2's
> tokens also but not document1's. The attached patch modifies a test
> case. It works with SimpleAnalyzer, but breaks with StandardAnalyzer.
> 
> The StandardAnalyzer.cpp code itself looks rather simple. Any ideas
> where the bug could be?

Uh, sorry. That test patch is just broken. I still have this problem in
my own code (also with SimpleAnalyzer), but I can't seem to be able to
reproduce it in the test code. Continuing my debugging..

[CLucene-dev] StandardAnalyzer is buggy

[Timo S. <ts...@ik...>]

Using the 2_3_2 git version and StandardAnalyzer: It looks like if I
index document1 and then document2, at least some of document1's tokens
go to document2 as well. If I index document3, it indexes document2's
tokens also but not document1's. The attached patch modifies a test
case. It works with SimpleAnalyzer, but breaks with StandardAnalyzer.

The StandardAnalyzer.cpp code itself looks rather simple. Any ideas
where the bug could be?

Re: [CLucene-dev] how to trace memory leak in clucene

[曾小伟 <zx...@16...>]

thanks for all your reply,
I will try it.

Best regards.
Zeng Xiaowei




At 2011-06-21 04:43:06，"Itamar Syn-Hershko" <it...@co...> wrote:

>That macro will track all CLucene objects inheriting from the 
>LuceneRefBase or however it is called. It should still be usable, except 
>we are moving from that approach. The demo project or the test suite 
>should have comments on how to use this.
>
>
>I recommend using the tools Veit pointed to.
>
>
>Itamar.
>
>
>On 20/06/2011 23:33, Veit Jahns wrote:
>
>> Hi,
>>
>> I don't know, if this macro is still used in the current version. I
>> think Ben or Itamar should know more about this. But you can also use
>> other tools to trace the memory leak. Depending on your plattform and
>> compiler, you can use, for instance, the Visual Studio facilities [1]
>> or valgrind [2].
>>
>> Kind regards,
>>
>> Veit
>>
>> [1] http://msdn.microsoft.com/en-us/library/x98tx3cf.aspx
>> [2] http://valgrind.org/
>>
>> 2011/6/20 曾小伟<zx...@16...>:
>>> hi, all
>>> I find memleak in clucene, and I want to trace it,
>>> There is a macro  ： LUCENE_ENABLE_MEMLEAKTRACKING
>>> but I don't know how to use it，
>>>
>>> these are what I had tried:
>>> 1)when configure ,   I enable debug,  but It seems not worked.
>>> 2)I use CFLAG="-DLUCENE_ENABLE_MEMLEAKTRACKING", but also got nothing,
>>> 3)I define LUCENE_ENABLE_MEMLEAKTRACKING in CLcofing.h，when runing, the
>>> program crashed.
>>>
>>> anyone who can help me to use it ,
>>>
>>> Zeng Xiaowei
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> EditLive Enterprise is the world's most technically advanced content
>>> authoring tool. Experience the power of Track Changes, Inline Image
>>> Editing and ensure content is compliant with Accessibility Checking.
>>> http://p.sf.net/sfu/ephox-dev2dev
>>> _______________________________________________
>>> CLucene-developers mailing list
>>> CLu...@li...
>>> https://lists.sourceforge.net/lists/listinfo/clucene-developers
>>>
>>>
>> ------------------------------------------------------------------------------
>> EditLive Enterprise is the world's most technically advanced content
>> authoring tool. Experience the power of Track Changes, Inline Image
>> Editing and ensure content is compliant with Accessibility Checking.
>> http://p.sf.net/sfu/ephox-dev2dev
>> _______________________________________________
>> CLucene-developers mailing list
>> CLu...@li...
>> https://lists.sourceforge.net/lists/listinfo/clucene-developers
>
>------------------------------------------------------------------------------
>EditLive Enterprise is the world's most technically advanced content
>authoring tool. Experience the power of Track Changes, Inline Image
>Editing and ensure content is compliant with Accessibility Checking.
>http://p.sf.net/sfu/ephox-dev2dev
>_______________________________________________
>CLucene-developers mailing list
>CLu...@li...
>https://lists.sourceforge.net/lists/listinfo/clucene-developers

Re: [CLucene-dev] how to trace memory leak in clucene

[Itamar Syn-H. <it...@co...>]

That macro will track all CLucene objects inheriting from the 
LuceneRefBase or however it is called. It should still be usable, except 
we are moving from that approach. The demo project or the test suite 
should have comments on how to use this.


I recommend using the tools Veit pointed to.


Itamar.


On 20/06/2011 23:33, Veit Jahns wrote:

> Hi,
>
> I don't know, if this macro is still used in the current version. I
> think Ben or Itamar should know more about this. But you can also use
> other tools to trace the memory leak. Depending on your plattform and
> compiler, you can use, for instance, the Visual Studio facilities [1]
> or valgrind [2].
>
> Kind regards,
>
> Veit
>
> [1] http://msdn.microsoft.com/en-us/library/x98tx3cf.aspx
> [2] http://valgrind.org/
>
> 2011/6/20 曾小伟<zx...@16...>:
>> hi, all
>> I find memleak in clucene, and I want to trace it,
>> There is a macro  ： LUCENE_ENABLE_MEMLEAKTRACKING
>> but I don't know how to use it，
>>
>> these are what I had tried:
>> 1)when configure ,   I enable debug,  but It seems not worked.
>> 2)I use CFLAG="-DLUCENE_ENABLE_MEMLEAKTRACKING", but also got nothing,
>> 3)I define LUCENE_ENABLE_MEMLEAKTRACKING in CLcofing.h，when runing, the
>> program crashed.
>>
>> anyone who can help me to use it ,
>>
>> Zeng Xiaowei
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> EditLive Enterprise is the world's most technically advanced content
>> authoring tool. Experience the power of Track Changes, Inline Image
>> Editing and ensure content is compliant with Accessibility Checking.
>> http://p.sf.net/sfu/ephox-dev2dev
>> _______________________________________________
>> CLucene-developers mailing list
>> CLu...@li...
>> https://lists.sourceforge.net/lists/listinfo/clucene-developers
>>
>>
> ------------------------------------------------------------------------------
> EditLive Enterprise is the world's most technically advanced content
> authoring tool. Experience the power of Track Changes, Inline Image
> Editing and ensure content is compliant with Accessibility Checking.
> http://p.sf.net/sfu/ephox-dev2dev
> _______________________________________________
> CLucene-developers mailing list
> CLu...@li...
> https://lists.sourceforge.net/lists/listinfo/clucene-developers

Re: [CLucene-dev] how to trace memory leak in clucene

[Veit J. <nun...@go...>]

Hi,

I don't know, if this macro is still used in the current version. I
think Ben or Itamar should know more about this. But you can also use
other tools to trace the memory leak. Depending on your plattform and
compiler, you can use, for instance, the Visual Studio facilities [1]
or valgrind [2].

Kind regards,

Veit

[1] http://msdn.microsoft.com/en-us/library/x98tx3cf.aspx
[2] http://valgrind.org/

2011/6/20 曾小伟 <zx...@16...>:
> hi, all
> I find memleak in clucene, and I want to trace it,
> There is a macro  ： LUCENE_ENABLE_MEMLEAKTRACKING
> but I don't know how to use it，
>
> these are what I had tried:
> 1)when configure ,   I enable debug,  but It seems not worked.
> 2)I use CFLAG="-DLUCENE_ENABLE_MEMLEAKTRACKING", but also got nothing,
> 3)I define LUCENE_ENABLE_MEMLEAKTRACKING in CLcofing.h，when runing, the
> program crashed.
>
> anyone who can help me to use it ,
>
> Zeng Xiaowei
>
>
>
>
>
> ------------------------------------------------------------------------------
> EditLive Enterprise is the world's most technically advanced content
> authoring tool. Experience the power of Track Changes, Inline Image
> Editing and ensure content is compliant with Accessibility Checking.
> http://p.sf.net/sfu/ephox-dev2dev
> _______________________________________________
> CLucene-developers mailing list
> CLu...@li...
> https://lists.sourceforge.net/lists/listinfo/clucene-developers
>
>

[CLucene-dev] how to trace memory leak in clucene

[曾小伟 <zx...@16...>]

hi, all
I find memleak in clucene, and I want to trace it,
There is a macro  ： LUCENE_ENABLE_MEMLEAKTRACKING
but I don't know how to use it，

these are what I had tried:
1)when configure ,   I enable debug,  but It seems not worked.
2)I use CFLAG="-DLUCENE_ENABLE_MEMLEAKTRACKING", but also got nothing,
3)I define LUCENE_ENABLE_MEMLEAKTRACKING in CLcofing.h，when runing, the program crashed.

anyone who can help me to use it ,

Zeng Xiaowei

[CLucene-dev] memory problem of highlighter

[yunwen y. <yun...@gm...>]

Hi, there

I have used java-lucene for a couple of years and are now moving to clucene.

I have a question related to the highlighter of clucene in terms of memory
release of the fragments returned by highlighter. I read the source code of
the highlighter and realized that the result returned by getBestFragments
need to be released. However, the system always crashes, reporting
crtIsValidHeapPointer. 

I compiled clucene with Visual Studio 2010 (Windows 7).

Here is the sample code I tried. (I have tried to free, delete, delete[]
just want to make sure I did not use the wrong one; all crashed)

lucene::util::StringReader r(content);
TokenStream* ts = _pAnalyzer->tokenStream(T("field"), &r);
_TCHAR** hleds = hilighter.getBestFragments(ts, copiedLine, 1);
if (hleds) _CLDELETE_CARRAY_ALL(hleds);

Is it because I cannot free memory allocated by the dll of lucene-contribs?
If that is the case, how can I solve this memory leak problem?  

Millions of thanks.

--yunwen

Re: [CLucene-dev] Search Engine database size

[Ben v. K. <bva...@gm...>]

Hi,

Wide characters are stored using utf-8, therefore characters taking up
less than 7 bytes will take up the exact same about of space. i would
stick to the wide character format if you don't have a compelling
reason to use ascii.

reducing size: using Luke(http://code.google.com/p/luke/) will help
you figure out what you've actually stored in the index. reducing the
number of fields you 'STORE' helps a lot. There is a compressed type
field in clucene - but it was a bit hard to get going until the latest
versions (now it's just another flag - Field::STORE_COMPRESS).

ben

On Wed, Jun 8, 2011 at 2:07 AM, Teryl Taylor <ter...@gm...> wrote:
> Hi everyone,
>
> I just had a quick question about search engine size. The search engine
> takes everything as wide characters.  Since everything I'm putting in the
> database is ASCII, I thought I'd compile the search engine  with ASCII Mode
> on.  This took the TCHAR and defined it as a char rather than a wchar_t.
> When I recompiled everything, and ran it, the search engine database was the
> exact same size as the original wide char.    Anyone know why that is?  I
> would have thought using chars instead of wide chars would have reduced the
> size.  Am I missing a configuration?
>
> Also, does anyone have any tips on reducing the size of a search engine?
> Lucene doesn't support a compression mechanism right?   It's not that the
> database is bloated or anything, it's just any size reduction I can get is
> beneficial, so I'm just investigating ways to get it as small as possible.
>
>
> Thanks,
>
> Teryl
>
>
>
>
>
> ------------------------------------------------------------------------------
> EditLive Enterprise is the world's most technically advanced content
> authoring tool. Experience the power of Track Changes, Inline Image
> Editing and ensure content is compliant with Accessibility Checking.
> http://p.sf.net/sfu/ephox-dev2dev
> _______________________________________________
> CLucene-developers mailing list
> CLu...@li...
> https://lists.sourceforge.net/lists/listinfo/clucene-developers
>
>



-- 
-------------------------------------
Ben van Klinken

Mob: 0401 921847
Em: be...@vi...

[CLucene-dev] Search Engine database size

[Teryl T. <ter...@gm...>]

Hi everyone,

I just had a quick question about search engine size. The search engine
takes everything as wide characters.  Since everything I'm putting in the
database is ASCII, I thought I'd compile the search engine  with ASCII Mode
on.  This took the TCHAR and defined it as a char rather than a wchar_t.
When I recompiled everything, and ran it, the search engine database was the
exact same size as the original wide char.    Anyone know why that is?  I
would have thought using chars instead of wide chars would have reduced the
size.  Am I missing a configuration?

Also, does anyone have any tips on reducing the size of a search engine?
Lucene doesn't support a compression mechanism right?   It's not that the
database is bloated or anything, it's just any size reduction I can get is
beneficial, so I'm just investigating ways to get it as small as possible.


Thanks,

Teryl

Re: [CLucene-dev] Indexing a document

[Emerson E. <eme...@gm...>]

Terrific. Thanks a lot.

[]'s
Emerson de Lira Espínola
** <eme...@gm...>
<https://profiles.google.com/emersonespinola/buzz?hl=pt-BR>
<http://www.quora.com/emersonespinola>
<http://www.facebook.com/emersonespinola>
<http://www.linkedin.com/in/emersonespinola>
<http://spaces.live.com/profile.aspx?mem=eme...@ho...>
<http://emersonespinola.blogspot.com> <http://twitter.com/emersonespinola>
<http://www.myebook.com/emersonespinola/>

 <http://www.myebook.com/emersonespinola/>


2011/5/27 Veit Jahns <nun...@go...>

> Hi Emerson!
>
> 2011/5/27 Emerson Espínola <eme...@gm...>
> >
> > Hi Veit.
> >
> > Thank yo very much for your answer. Great explanation. You don't wonder
> how much you're helping me.
>
> You are welcome!
>
> > 1. I'll try english documents.
> > 2. Ok.
> > 3. Does BrazilianAnalyzer work similar to StandardAnalyzer? If so, that's
> what I want. I'm from Brazil. :) There are no much differences between
> portuguese from Portugal and portuguese from Brazil. But if there is already
> an analyzer for portuguese from Brazil that's perfect for me.
>
> According to the source code, it is based on the StandardAnalyzer. It
> uses the StandardTokenizer and the StandardFilter, and adds a stopword
> filter and a brazilian stemming filter.
>
> Kind regards,
>
> Veit
>
>
> ------------------------------------------------------------------------------
> vRanger cuts backup time in half-while increasing security.
> With the market-leading solution for virtual backup and recovery,
> you get blazing-fast, flexible, and affordable data protection.
> Download your free trial now.
> http://p.sf.net/sfu/quest-d2dcopy1
> _______________________________________________
> CLucene-developers mailing list
> CLu...@li...
> https://lists.sourceforge.net/lists/listinfo/clucene-developers
>

Re: [CLucene-dev] Indexing a document

[Veit J. <nun...@go...>]

Hi Emerson!

2011/5/27 Emerson Espínola <eme...@gm...>
>
> Hi Veit.
>
> Thank yo very much for your answer. Great explanation. You don't wonder how much you're helping me.

You are welcome!

> 1. I'll try english documents.
> 2. Ok.
> 3. Does BrazilianAnalyzer work similar to StandardAnalyzer? If so, that's what I want. I'm from Brazil. :) There are no much differences between portuguese from Portugal and portuguese from Brazil. But if there is already an analyzer for portuguese from Brazil that's perfect for me.

According to the source code, it is based on the StandardAnalyzer. It
uses the StandardTokenizer and the StandardFilter, and adds a stopword
filter and a brazilian stemming filter.

Kind regards,

Veit