Not sure if windows is different but in Linux -Xmx512m needs to go in after Java so you should have java -Xmx512m -jar datacrow.jar
thanks, that seems to have helped a lot.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Sorry to re-animate a 2yr old thread but the problem persists. ENV
Dell Precision 670 - dual Xeon 64bit processors, 2GB RAM, Linux ubuntu 10.4 (lucid) PROBLEM
I increased the virtual memory to 1024mb and tried to import 598 books/files. The process ran with 1.4g data/stack and stopped processing after 501 files. the datacrow log (see below) reported a lack of Heap space and subsequently the garbage collector exceeded its limits.
I realize 64bit systems have a larger object size and dual processor may contribute to heap exhaustion but I would have thought 1024mb vm should be ample, so my question is - why doesn't the import process use a staging table on the backend instead of memory bound objects?
Any ideas on a workaround? I'm hoping to import 100,000+ items at some point.
Excerpt from datacrow.log INFO (DataCrow.java:840) - Data Crow 3.8.16
INFO (DataCrow.java:841) - Java version: 1.6.0_18
INFO (DataCrow.java:842) - Java vendor: Sun Microsystems Inc.
INFO (DataCrow.java:843) - Operating System: Linux
…
INFO (TextReaderFactory.java:43) - Extracting text from : /media/lcldata1/md0/text/books/Plants/Gardening/Vegeta
bles/Vegetables.pdf
INFO (TextReaderFactory.java:48) - Extraction of text done (3749 ms)
INFO (ISBNExtractor.java:74) - Searching ISBN in the first 9223372036854775807 bytes …
INFO (ISBNExtractor.java:119) - Done (1 ms).
WARN (FileISBNExtractor.java:72) - isbn not found for /media/lcldata1/md0/text/books/Plants/Gardening/Vegetables
/Vegetables.pdf
ERROR (EbookImport.java:161) - java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.awt.image.DataBufferInt.<init>(DataBufferInt.java:75)
at java.awt.image.Raster.createPackedRaster(Raster.java:470)
at java.awt.image.DirectColorModel.createCompatibleWritableRaster(DirectColorModel.java:1032)
at java.awt.image.BufferedImage.<init>(BufferedImage.java:338)
at sun.java2d.loops.GraphicsPrimitive.convertFrom(GraphicsPrimitive.java:555)
at sun.java2d.loops.GraphicsPrimitive.convertFrom(GraphicsPrimitive.java:539)
at sun.java2d.loops.MaskBlit$General.MaskBlit(MaskBlit.java:189)
at sun.java2d.loops.Blit$GeneralMaskBlit.Blit(Blit.java:204)
at sun.java2d.pipe.DrawImage.blitSurfaceData(DrawImage.java:953)
at sun.java2d.pipe.DrawImage.renderImageCopy(DrawImage.java:575)
at sun.java2d.pipe.DrawImage.copyImage(DrawImage.java:71)
at sun.java2d.pipe.DrawImage.copyImage(DrawImage.java:1008)
at sun.java2d.pipe.ValidatePipe.copyImage(ValidatePipe.java:186)
at sun.java2d.SunGraphics2D.drawImage(SunGraphics2D.java:3053)
at sun.java2d.SunGraphics2D.drawImage(SunGraphics2D.java:3038)
at sun.java2d.pipe.DrawImage.makeBufferedImage(DrawImage.java:334)
at sun.java2d.pipe.DrawImage.renderImageXform(DrawImage.java:394)
at sun.java2d.pipe.DrawImage.transformImage(DrawImage.java:268)
at sun.java2d.pipe.DrawImage.transformImage(DrawImage.java:214)
at sun.java2d.pipe.DrawImage.transformImage(DrawImage.java:1079)
at sun.java2d.pipe.ValidatePipe.transformImage(ValidatePipe.java:230)
at sun.java2d.SunGraphics2D.drawImage(SunGraphics2D.java:3179)
at com.sun.pdfview.PDFRenderer.drawImage(PDFRenderer.java:286)
at com.sun.pdfview.PDFImageCmd.execute(PDFPage.java:643)
at com.sun.pdfview.PDFRenderer.iterate(PDFRenderer.java:570)
at com.sun.pdfview.BaseWatchable.run(BaseWatchable.java:101)
at com.sun.pdfview.BaseWatchable.execute(BaseWatchable.java:263)
at com.sun.pdfview.BaseWatchable.go(BaseWatchable.java:197)
at com.sun.pdfview.PDFPage.getImage(PDFPage.java:228)
at net.datacrow.fileimporters.EbookImport.parse(EbookImport.java:149)
at net.datacrow.fileimporters.FileImporter.parse(FileImporter.java:223)
at net.datacrow.fileimporters.FileImporter$1.run(FileImporter.java:182)
INFO (FileImporter.java:395) - Searching online for vegetables
INFO (ImageCleaner.java:106) - There are 0 images loaded into memory. Removing 0.
INFO (FileImporter.java:395) - Searching online for vegetables
…
INFO (TextReaderFactory.java:43) - Extracting text from : /media/lcldata1/md0/text/books/Plants/Greenhouse Gardening/Greenhouse - PVC Plans.pdf
INFO (TextReaderFactory.java:48) - Extraction of text done (136 ms)
INFO (ISBNExtractor.java:74) - Searching ISBN in the first 9223372036854775807 bytes …
INFO (ISBNExtractor.java:119) - Done (1386 ms).
WARN (FileISBNExtractor.java:72) - isbn not found for /media/lcldata1/md0/text/books/Plants/Greenhouse Gardening/Greenhouse - PVC Plans.pdf
ERROR (EbookImport.java:161) - java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.awt.geom.Path2D.getPathIterator(Path2D.java:2348)
at sun.java2d.pisces.PiscesRenderingEngine.getAATileGenerator(PiscesRenderingEngine.java:321)
at sun.java2d.pipe.AAShapePipe.renderPath(AAShapePipe.java:93)
at sun.java2d.pipe.AAShapePipe.fill(AAShapePipe.java:65)
at sun.java2d.SunGraphics2D.fill(SunGraphics2D.java:2422)
at com.sun.pdfview.PDFPaint.fill(PDFPaint.java:69)
at com.sun.pdfview.PDFRenderer.fill(PDFRenderer.java:261)
at com.sun.pdfview.PDFShapeCmd.execute(PDFShapeCmd.java:77)
at com.sun.pdfview.PDFRenderer.iterate(PDFRenderer.java:570)
at com.sun.pdfview.BaseWatchable.run(BaseWatchable.java:101)
at com.sun.pdfview.BaseWatchable.execute(BaseWatchable.java:263)
at com.sun.pdfview.BaseWatchable.go(BaseWatchable.java:197)
at com.sun.pdfview.PDFPage.getImage(PDFPage.java:228)
at net.datacrow.fileimporters.EbookImport.parse(EbookImport.java:149)
at net.datacrow.fileimporters.FileImporter.parse(FileImporter.java:223)
at net.datacrow.fileimporters.FileImporter$1.run(FileImporter.java:182)
at java.lang.Thread.run(Thread.java:636)
INFO (FileImporter.java:395) - Searching online for green 1 pdf
INFO (FileImporter.java:395) - Searching online for green 1 pdf
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I am also getting the Java heap space error while trying to import my image library. My first question is whether Datacrow can handle 13,000+ images?
By using "java -Xmx1024m -jar datacrow.jar" I can import up to 250 images, but the program hangs at some point. I'm using XP Pro and have to use the task manager to force Datacrow to quit.
Can virtual memory be increased beyond 1024? I tried 2048, but Datacrow would not even launch.
Supposedly there is a command file for windows system named datacrow.cmd. Can't locate that.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
event log says
Starting to parse 15054 files
Starting import
consol says
Exception in thread "Thread-6" java.lang.OutOfMemoryError: java heap space
I havent stoped it yet but I assume its hung.
top and sytem mon shows only 49% of mem used
This is centos 5 java 1.6 and DataCrow 3.4.1
starting over and only selecting a few folders - should help :-)
Please check whether you are starting Data Crow using the following command:
java -jar datacrow.jar -Xmx512m
If you just start it using the following command java -jar datacrow.jar it will not have enough memory allocated.
Not sure if windows is different but in Linux -Xmx512m needs to go in after Java so you should have java -Xmx512m -jar datacrow.jar
thanks, that seems to have helped a lot.
Sorry to re-animate a 2yr old thread but the problem persists.
ENV
Dell Precision 670 - dual Xeon 64bit processors, 2GB RAM, Linux ubuntu 10.4 (lucid)
PROBLEM
I increased the virtual memory to 1024mb and tried to import 598 books/files. The process ran with 1.4g data/stack and stopped processing after 501 files. the datacrow log (see below) reported a lack of Heap space and subsequently the garbage collector exceeded its limits.
I realize 64bit systems have a larger object size and dual processor may contribute to heap exhaustion but I would have thought 1024mb vm should be ample, so my question is - why doesn't the import process use a staging table on the backend instead of memory bound objects?
Any ideas on a workaround? I'm hoping to import 100,000+ items at some point.
Excerpt from datacrow.log
INFO (DataCrow.java:840) - Data Crow 3.8.16
INFO (DataCrow.java:841) - Java version: 1.6.0_18
INFO (DataCrow.java:842) - Java vendor: Sun Microsystems Inc.
INFO (DataCrow.java:843) - Operating System: Linux
…
INFO (TextReaderFactory.java:43) - Extracting text from : /media/lcldata1/md0/text/books/Plants/Gardening/Vegeta
bles/Vegetables.pdf
INFO (TextReaderFactory.java:48) - Extraction of text done (3749 ms)
INFO (ISBNExtractor.java:74) - Searching ISBN in the first 9223372036854775807 bytes …
INFO (ISBNExtractor.java:119) - Done (1 ms).
WARN (FileISBNExtractor.java:72) - isbn not found for /media/lcldata1/md0/text/books/Plants/Gardening/Vegetables
/Vegetables.pdf
ERROR (EbookImport.java:161) - java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.awt.image.DataBufferInt.<init>(DataBufferInt.java:75)
at java.awt.image.Raster.createPackedRaster(Raster.java:470)
at java.awt.image.DirectColorModel.createCompatibleWritableRaster(DirectColorModel.java:1032)
at java.awt.image.BufferedImage.<init>(BufferedImage.java:338)
at sun.java2d.loops.GraphicsPrimitive.convertFrom(GraphicsPrimitive.java:555)
at sun.java2d.loops.GraphicsPrimitive.convertFrom(GraphicsPrimitive.java:539)
at sun.java2d.loops.MaskBlit$General.MaskBlit(MaskBlit.java:189)
at sun.java2d.loops.Blit$GeneralMaskBlit.Blit(Blit.java:204)
at sun.java2d.pipe.DrawImage.blitSurfaceData(DrawImage.java:953)
at sun.java2d.pipe.DrawImage.renderImageCopy(DrawImage.java:575)
at sun.java2d.pipe.DrawImage.copyImage(DrawImage.java:71)
at sun.java2d.pipe.DrawImage.copyImage(DrawImage.java:1008)
at sun.java2d.pipe.ValidatePipe.copyImage(ValidatePipe.java:186)
at sun.java2d.SunGraphics2D.drawImage(SunGraphics2D.java:3053)
at sun.java2d.SunGraphics2D.drawImage(SunGraphics2D.java:3038)
at sun.java2d.pipe.DrawImage.makeBufferedImage(DrawImage.java:334)
at sun.java2d.pipe.DrawImage.renderImageXform(DrawImage.java:394)
at sun.java2d.pipe.DrawImage.transformImage(DrawImage.java:268)
at sun.java2d.pipe.DrawImage.transformImage(DrawImage.java:214)
at sun.java2d.pipe.DrawImage.transformImage(DrawImage.java:1079)
at sun.java2d.pipe.ValidatePipe.transformImage(ValidatePipe.java:230)
at sun.java2d.SunGraphics2D.drawImage(SunGraphics2D.java:3179)
at com.sun.pdfview.PDFRenderer.drawImage(PDFRenderer.java:286)
at com.sun.pdfview.PDFImageCmd.execute(PDFPage.java:643)
at com.sun.pdfview.PDFRenderer.iterate(PDFRenderer.java:570)
at com.sun.pdfview.BaseWatchable.run(BaseWatchable.java:101)
at com.sun.pdfview.BaseWatchable.execute(BaseWatchable.java:263)
at com.sun.pdfview.BaseWatchable.go(BaseWatchable.java:197)
at com.sun.pdfview.PDFPage.getImage(PDFPage.java:228)
at net.datacrow.fileimporters.EbookImport.parse(EbookImport.java:149)
at net.datacrow.fileimporters.FileImporter.parse(FileImporter.java:223)
at net.datacrow.fileimporters.FileImporter$1.run(FileImporter.java:182)
INFO (FileImporter.java:395) - Searching online for vegetables
INFO (ImageCleaner.java:106) - There are 0 images loaded into memory. Removing 0.
INFO (FileImporter.java:395) - Searching online for vegetables
…
INFO (TextReaderFactory.java:43) - Extracting text from : /media/lcldata1/md0/text/books/Plants/Greenhouse Gardening/Greenhouse - PVC Plans.pdf
INFO (TextReaderFactory.java:48) - Extraction of text done (136 ms)
INFO (ISBNExtractor.java:74) - Searching ISBN in the first 9223372036854775807 bytes …
INFO (ISBNExtractor.java:119) - Done (1386 ms).
WARN (FileISBNExtractor.java:72) - isbn not found for /media/lcldata1/md0/text/books/Plants/Greenhouse Gardening/Greenhouse - PVC Plans.pdf
ERROR (EbookImport.java:161) - java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.awt.geom.Path2D.getPathIterator(Path2D.java:2348)
at sun.java2d.pisces.PiscesRenderingEngine.getAATileGenerator(PiscesRenderingEngine.java:321)
at sun.java2d.pipe.AAShapePipe.renderPath(AAShapePipe.java:93)
at sun.java2d.pipe.AAShapePipe.fill(AAShapePipe.java:65)
at sun.java2d.SunGraphics2D.fill(SunGraphics2D.java:2422)
at com.sun.pdfview.PDFPaint.fill(PDFPaint.java:69)
at com.sun.pdfview.PDFRenderer.fill(PDFRenderer.java:261)
at com.sun.pdfview.PDFShapeCmd.execute(PDFShapeCmd.java:77)
at com.sun.pdfview.PDFRenderer.iterate(PDFRenderer.java:570)
at com.sun.pdfview.BaseWatchable.run(BaseWatchable.java:101)
at com.sun.pdfview.BaseWatchable.execute(BaseWatchable.java:263)
at com.sun.pdfview.BaseWatchable.go(BaseWatchable.java:197)
at com.sun.pdfview.PDFPage.getImage(PDFPage.java:228)
at net.datacrow.fileimporters.EbookImport.parse(EbookImport.java:149)
at net.datacrow.fileimporters.FileImporter.parse(FileImporter.java:223)
at net.datacrow.fileimporters.FileImporter$1.run(FileImporter.java:182)
at java.lang.Thread.run(Thread.java:636)
INFO (FileImporter.java:395) - Searching online for green 1 pdf
INFO (FileImporter.java:395) - Searching online for green 1 pdf
Data Crow 3.9 will deliver the solution!
There are a lot of improvements related to the memory usage and such.
Cheers,
Robert
Thanks Robert,
That makes me happy as DC is the only feature rich catalog s/w out there. i look forward to 3.9
Regards
Mike
I am also getting the Java heap space error while trying to import my image library. My first question is whether Datacrow can handle 13,000+ images?
By using "java -Xmx1024m -jar datacrow.jar" I can import up to 250 images, but the program hangs at some point. I'm using XP Pro and have to use the task manager to force Datacrow to quit.
Can virtual memory be increased beyond 1024? I tried 2048, but Datacrow would not even launch.
Supposedly there is a command file for windows system named datacrow.cmd. Can't locate that.