out of memory error?

2008-10-16
2013-01-20
  • event log says
    Starting to parse 15054 files
    Starting import

    consol says
    Exception in thread "Thread-6" java.lang.OutOfMemoryError: java heap space

    I havent stoped it yet but I assume its hung.
    top and sytem mon shows only 49% of mem used
    This is centos 5 java 1.6 and DataCrow 3.4.1

    starting over and only selecting a few folders - should help :-)

     
    • Please check whether you are starting Data Crow using the following command:
      java -jar datacrow.jar -Xmx512m

      If you just start it using the following command java -jar datacrow.jar it will not have enough memory allocated.

       
    • Not sure if windows is different but in Linux -Xmx512m needs to go in after Java so you should have java -Xmx512m -jar datacrow.jar
      thanks, that seems to have helped a lot.

       
  • MICHAEL
    MICHAEL
    2010-09-04

    Sorry to re-animate a 2yr old thread but the problem persists.
    ENV
    Dell Precision 670 - dual Xeon 64bit processors, 2GB RAM, Linux ubuntu 10.4 (lucid)
    PROBLEM
    I increased the virtual memory to 1024mb and tried to import 598 books/files. The process ran with 1.4g data/stack and stopped processing after 501 files. the datacrow log (see below) reported a lack of Heap space and subsequently the garbage collector exceeded its limits.

    I realize 64bit systems have a larger object size and dual processor may contribute to heap exhaustion but I would have thought 1024mb vm should be ample, so my question is - why doesn't the import process use a staging table on the backend instead of memory bound objects?

    Any ideas on a workaround? I'm hoping to import 100,000+ items at some point.

    Excerpt from datacrow.log
    INFO  (DataCrow.java:840) - Data Crow 3.8.16
    INFO  (DataCrow.java:841) - Java version: 1.6.0_18
    INFO  (DataCrow.java:842) - Java vendor: Sun Microsystems Inc.
    INFO  (DataCrow.java:843) - Operating System: Linux


    INFO  (TextReaderFactory.java:43) - Extracting text from : /media/lcldata1/md0/text/books/Plants/Gardening/Vegeta
    bles/Vegetables.pdf
    INFO  (TextReaderFactory.java:48) - Extraction of text done (3749 ms)
    INFO  (ISBNExtractor.java:74) - Searching ISBN in the first 9223372036854775807 bytes …
    INFO  (ISBNExtractor.java:119) - Done (1 ms).
    WARN  (FileISBNExtractor.java:72) - isbn not found for /media/lcldata1/md0/text/books/Plants/Gardening/Vegetables
    /Vegetables.pdf
    ERROR  (EbookImport.java:161) - java.lang.OutOfMemoryError: Java heap space
    java.lang.OutOfMemoryError: Java heap space
            at java.awt.image.DataBufferInt.<init>(DataBufferInt.java:75)
            at java.awt.image.Raster.createPackedRaster(Raster.java:470)
            at java.awt.image.DirectColorModel.createCompatibleWritableRaster(DirectColorModel.java:1032)
            at java.awt.image.BufferedImage.<init>(BufferedImage.java:338)
            at sun.java2d.loops.GraphicsPrimitive.convertFrom(GraphicsPrimitive.java:555)
            at sun.java2d.loops.GraphicsPrimitive.convertFrom(GraphicsPrimitive.java:539)
            at sun.java2d.loops.MaskBlit$General.MaskBlit(MaskBlit.java:189)
            at sun.java2d.loops.Blit$GeneralMaskBlit.Blit(Blit.java:204)
            at sun.java2d.pipe.DrawImage.blitSurfaceData(DrawImage.java:953)
            at sun.java2d.pipe.DrawImage.renderImageCopy(DrawImage.java:575)
            at sun.java2d.pipe.DrawImage.copyImage(DrawImage.java:71)
            at sun.java2d.pipe.DrawImage.copyImage(DrawImage.java:1008)
            at sun.java2d.pipe.ValidatePipe.copyImage(ValidatePipe.java:186)
            at sun.java2d.SunGraphics2D.drawImage(SunGraphics2D.java:3053)
            at sun.java2d.SunGraphics2D.drawImage(SunGraphics2D.java:3038)
            at sun.java2d.pipe.DrawImage.makeBufferedImage(DrawImage.java:334)
            at sun.java2d.pipe.DrawImage.renderImageXform(DrawImage.java:394)
            at sun.java2d.pipe.DrawImage.transformImage(DrawImage.java:268)
            at sun.java2d.pipe.DrawImage.transformImage(DrawImage.java:214)
            at sun.java2d.pipe.DrawImage.transformImage(DrawImage.java:1079)
            at sun.java2d.pipe.ValidatePipe.transformImage(ValidatePipe.java:230)
            at sun.java2d.SunGraphics2D.drawImage(SunGraphics2D.java:3179)
            at com.sun.pdfview.PDFRenderer.drawImage(PDFRenderer.java:286)
            at com.sun.pdfview.PDFImageCmd.execute(PDFPage.java:643)
            at com.sun.pdfview.PDFRenderer.iterate(PDFRenderer.java:570)
            at com.sun.pdfview.BaseWatchable.run(BaseWatchable.java:101)
            at com.sun.pdfview.BaseWatchable.execute(BaseWatchable.java:263)
            at com.sun.pdfview.BaseWatchable.go(BaseWatchable.java:197)
            at com.sun.pdfview.PDFPage.getImage(PDFPage.java:228)
            at net.datacrow.fileimporters.EbookImport.parse(EbookImport.java:149)
            at net.datacrow.fileimporters.FileImporter.parse(FileImporter.java:223)
            at net.datacrow.fileimporters.FileImporter$1.run(FileImporter.java:182)
    INFO  (FileImporter.java:395) - Searching online for vegetables
    INFO  (ImageCleaner.java:106) - There are 0 images loaded into memory. Removing 0.
    INFO  (FileImporter.java:395) - Searching online for vegetables

    INFO  (TextReaderFactory.java:43) - Extracting text from : /media/lcldata1/md0/text/books/Plants/Greenhouse Gardening/Greenhouse -  PVC Plans.pdf
    INFO  (TextReaderFactory.java:48) - Extraction of text done (136 ms)
    INFO  (ISBNExtractor.java:74) - Searching ISBN in the first 9223372036854775807 bytes …
    INFO  (ISBNExtractor.java:119) - Done (1386 ms).
    WARN  (FileISBNExtractor.java:72) - isbn not found for /media/lcldata1/md0/text/books/Plants/Greenhouse Gardening/Greenhouse -  PVC Plans.pdf
    ERROR  (EbookImport.java:161) - java.lang.OutOfMemoryError: GC overhead limit exceeded
    java.lang.OutOfMemoryError: GC overhead limit exceeded
            at java.awt.geom.Path2D.getPathIterator(Path2D.java:2348)
            at sun.java2d.pisces.PiscesRenderingEngine.getAATileGenerator(PiscesRenderingEngine.java:321)
            at sun.java2d.pipe.AAShapePipe.renderPath(AAShapePipe.java:93)
            at sun.java2d.pipe.AAShapePipe.fill(AAShapePipe.java:65)
            at sun.java2d.SunGraphics2D.fill(SunGraphics2D.java:2422)
            at com.sun.pdfview.PDFPaint.fill(PDFPaint.java:69)
            at com.sun.pdfview.PDFRenderer.fill(PDFRenderer.java:261)
            at com.sun.pdfview.PDFShapeCmd.execute(PDFShapeCmd.java:77)
            at com.sun.pdfview.PDFRenderer.iterate(PDFRenderer.java:570)
            at com.sun.pdfview.BaseWatchable.run(BaseWatchable.java:101)
            at com.sun.pdfview.BaseWatchable.execute(BaseWatchable.java:263)
            at com.sun.pdfview.BaseWatchable.go(BaseWatchable.java:197)
            at com.sun.pdfview.PDFPage.getImage(PDFPage.java:228)
            at net.datacrow.fileimporters.EbookImport.parse(EbookImport.java:149)
            at net.datacrow.fileimporters.FileImporter.parse(FileImporter.java:223)
            at net.datacrow.fileimporters.FileImporter$1.run(FileImporter.java:182)
            at java.lang.Thread.run(Thread.java:636)
    INFO  (FileImporter.java:395) - Searching online for green 1 pdf
    INFO  (FileImporter.java:395) - Searching online for green 1 pdf

     
  • Data Crow 3.9 will deliver the solution!
    There are a lot of improvements related to the memory usage and such.

    Cheers,
    Robert

     
  • MICHAEL
    MICHAEL
    2010-09-05

    Thanks Robert,
    That makes me happy as DC is the only feature rich catalog s/w out there. i look forward to 3.9

    Regards
    Mike

     
  • Mannikin
    Mannikin
    2011-07-01

    I am also getting the Java heap space error while trying to import my image library. My first question is whether Datacrow can handle 13,000+ images?
    By using "java -Xmx1024m -jar datacrow.jar" I can import up to 250 images, but the program hangs at some point. I'm using XP Pro and have to use the task manager to force Datacrow to quit.
    Can virtual memory be increased beyond 1024? I tried 2048, but Datacrow would not even launch.
    Supposedly there is a command file for windows system named datacrow.cmd. Can't locate that.