Menu
▾
▴
Re: [CLucene-dev] 0.9.10 making 75,000 index files, 10x disk space of 0.8.11
|
From: Ben v. K. <bva...@gm...> - 2005-12-12 11:24:43
|
I'm sure that the index sizes should be the same. It seems as though the old segments arent being deleted. Have a look at the deletable file, it normally contains the segments which should be deleted but for some reason couldnt be - i suspect that this will have a list of all the segments except the latest one. Also look at the segments file, it contains the currently used segments. Since you optimized the index, there should only be one segment name in this. Can you verify these two points for me? thanks ben On 12/12/05, Bryan White <br...@ar...> wrote: > Ben van Klinken wrote: > > From the look of it, the old segments aren't being deleted. There > > seems to be lots of old segment files that aren't deleted. > > > > Each cfs is a compound file containing all the .f1, .fdt, .fdx, etc > > files. Is there any reason why they would not have be deleted? Is > > there something about the directory security? Or was something > > accessing the index directory while it was being written? > > This is a new index creation. The directory is created by the > IndexWriter constructor. No other process is accessing the index. The > directory is created with a temp name and after completion > (Writer->Close()) the directory is renamed. Furthermore this is a test > box and it has nothing running that would be accessing the index. > > I have verified that the indexes are readable. Also a valgrind run of > the index creation process did not show much of interest: > =3D=3D2151=3D=3D Memcheck, a memory error detector for x86-linux. > =3D=3D2151=3D=3D Copyright (C) 2002-2005, and GNU GPL'd, by Julian Seward= et al. > =3D=3D2151=3D=3D Using valgrind-2.4.0, a program supervision framework fo= r > x86-linux. > =3D=3D2151=3D=3D Copyright (C) 2000-2005, and GNU GPL'd, by Julian Seward= et al. > =3D=3D2151=3D=3D For more details, rerun with: -v > =3D=3D2151=3D=3D > =3D=3D2151=3D=3D Conditional jump or move depends on uninitialised value(= s) > =3D=3D2151=3D=3D at 0x1BBEBBE0: strstr (in /lib/libc-2.3.5.so) > =3D=3D2151=3D=3D by 0x1BCAF6E7: __pthread_initialize_minimal (in > /lib/libpthread-2.3.5.so) > =3D=3D2151=3D=3D by 0x1BCAF297: (within /lib/libpthread-2.3.5.so) > =3D=3D2151=3D=3D by 0x1BCAEE7F: (within /lib/libpthread-2.3.5.so) > =3D=3D2151=3D=3D by 0x1B8F1DCA: call_init (in /lib/ld-2.3.5.so) > =3D=3D2151=3D=3D by 0x1B8F1EEC: _dl_init (in /lib/ld-2.3.5.so) > =3D=3D2151=3D=3D by 0x1B8E47CE: (within /lib/ld-2.3.5.so) > =3D=3D2151=3D=3D > =3D=3D2151=3D=3D Conditional jump or move depends on uninitialised value(= s) > =3D=3D2151=3D=3D at 0x1BBEBBE4: strstr (in /lib/libc-2.3.5.so) > =3D=3D2151=3D=3D by 0x1BCAF6E7: __pthread_initialize_minimal (in > /lib/libpthread-2.3.5.so) > =3D=3D2151=3D=3D by 0x1BCAF297: (within /lib/libpthread-2.3.5.so) > =3D=3D2151=3D=3D by 0x1BCAEE7F: (within /lib/libpthread-2.3.5.so) > =3D=3D2151=3D=3D by 0x1B8F1DCA: call_init (in /lib/ld-2.3.5.so) > =3D=3D2151=3D=3D by 0x1B8F1EEC: _dl_init (in /lib/ld-2.3.5.so) > =3D=3D2151=3D=3D by 0x1B8E47CE: (within /lib/ld-2.3.5.so) > sh: /bin/chmod: Argument list too long > searchengine.ProcessDump: newsstory count:35591 maxid:37292 > =3D=3D2151=3D=3D > =3D=3D2151=3D=3D ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 49 = from 1) > =3D=3D2151=3D=3D malloc/free: in use at exit: 603200 bytes in 81 blocks. > =3D=3D2151=3D=3D malloc/free: 43026643 allocs, 43026562 frees, 2968442847= bytes > allocated. > =3D=3D2151=3D=3D For counts of detected errors, rerun with: -v > =3D=3D2151=3D=3D searching for pointers to 81 not-freed blocks. > =3D=3D2151=3D=3D checked 1082428 bytes. > =3D=3D2151=3D=3D > =3D=3D2151=3D=3D LEAK SUMMARY: > =3D=3D2151=3D=3D definitely lost: 0 bytes in 0 blocks. > =3D=3D2151=3D=3D possibly lost: 0 bytes in 0 blocks. > =3D=3D2151=3D=3D still reachable: 603200 bytes in 81 blocks. > =3D=3D2151=3D=3D suppressed: 0 bytes in 0 blocks. > =3D=3D2151=3D=3D Reachable blocks (those to which a pointer was found) ar= e not > shown. > =3D=3D2151=3D=3D To see them, rerun with: --show-reachable=3Dyes > > The chmod error occurs when the process attempts to chmod 666 on the > contents of the directory and values because there are two many files. > > I don't know that the two strstr errors are. It seems to have something > to do with pthreads but this is not a multithreaded program. I doubt is > is related to the problem. > > The valgrind run represents the results of a much smaller run. In this > case the resulting index directory was 163MB in 3957 files. The old > working version produces a 19MB index directory. > -- > Bryan White > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log fi= les > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=3D7637&alloc_id=3D16865&op=3Dclick > _______________________________________________ > CLucene-developers mailing list > CLu...@li... > https://lists.sourceforge.net/lists/listinfo/clucene-developers > |
