#1367 javaw memory use grows constantly

[John Fraser]

Hi Tran,

I am using the portable version of DocFetcher (current release) and am creating indexes on network folders for my co workers to use. The folders I am indexing are also network folders. The javaw.exe process memory use grows continuously while indexing is occurring, it continues to grow until all available physical memory is used and eventually slows the PC down to the point I have to kill the process as nothing else responds.

See attached video that shows the javaw.exe growing whilst network indexing is occuring.

I copied DocFetcher-4096_64-bit-Java.exe from the misc folder and use that version, running Java64 bit v1.8.0_144

Cheers

John

[John Fraser]

Hi Tran,

Just an update/extra info ...

copied the .bat file from misc folder and modified command to be:
java -d64 -enableassertions -Xms2048m -Xmx4096m -Xss64m -cp %libclasspath% -Djava.library.path=lib net.sourceforge.docfetcher.Main %1 %2 %3 %4 %5 %6 %7 %8 %9

Was hoping this might restrict its memor use to the Xmx value, but to no avail, outcome was the same, physical memory used 7.92GB out of 8.0GB after about 20 minutes indexing the one network folder.

John

[Nam-Quang Tran]

Hi John,

I can only imagine that you're trying to index an extremely large file (e.g. a 100+ MB PDF file). If that's the case, you should see the name of the file at the end of the file list on the indexing dialog.

I haven't checked the PortableApps version of DocFetcher, but portable DocFetcher from the project website ships with launchers with up to 8 GB of RAM, which is what you might need for very large files.

Alternatively, try using the exclusion filter on the indexing dialog to skip any problematic files during indexing.

Best regards
q:-) <= Quang

[John Fraser]

After several hours, I found that it appeared to be some corrupt Excel files, a different version of the same file was only 11KB in size, the second was 114MB in size and that seemed to cause the memory leak, once it started, it was a lost cause.

I managed to build a regex to exclude the file but would be concerned there could be other similarly corrupted files that could lead to the same problem.

Cheers

John

[Nam-Quang Tran]

Well, 114 MB is enormous for an Excel file, no wonder DocFetcher is choking on it. From here on out, you'll just have to try indexing and excluding files until everything works. You might also want to search for and exclude large files (e.g. everything larger than 100 MB) before indexing.