Re: [CLucene-dev] RAM usage while indexing

[Ben v. K. <bva...@gm...>]

sounds pretty high. how big are the files? could you be leaking memory in
the 'fileDocument' function?

as a test, try not actually adding the document

ben

On Thu, Jan 6, 2011 at 7:43 AM, muhammad ismael <m.i...@gm...>wrote:

> Hello,
> I am trying to index large files as follows
>
> for(int j = 0; (j < pagesIds.count())&& !m_stop ; j++)
>
>                 {
>
>                     pagesText = m_DbManager->getBookPage(m_booksIds.at(i), pagesIds.at(j)).toUtf8();
>
>                     if(!pagesText.isEmpty())
>
>                     {
>
>                         Document* doc = fileDocument(QString::number(pagesIds.at(j)), QString::number(m_booksIds.at(i)), pagesText);
>
>                         writer->addDocument(doc);
>
>                         _CLDELETE(doc);
>
>                     }
>
>                 }
>
>
> but when the number of files exceeds 5000 files the application usage of my
> computer ram is 2 GB
> I tried to debug and i found that this happens in
>
> IndexWriter::addDocument(Document*)
>
> i tried to set
> IndexWriter->setMergeFactor(5);
> and also
> IndexWriter->setRAMBufferSizeMB(10);
> I know the default ram usage should be 16 MB but i just tried.
>
> I am working on master branch and i merged with it memory_leaks branch.
> is am i missing something?
>
> Mohammad Ismael
>
>
>
> ------------------------------------------------------------------------------
> Learn how Oracle Real Application Clusters (RAC) One Node allows customers
> to consolidate database storage, standardize their database environment,
> and,
> should the need arise, upgrade to a full multi-node Oracle RAC database
> without downtime or disruption
> http://p.sf.net/sfu/oracle-sfdevnl
> _______________________________________________
> CLucene-developers mailing list
> CLu...@li...
> https://lists.sourceforge.net/lists/listinfo/clucene-developers
>
>