Re: [djatoka-devel] compress.sh | heap space

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Hi Nick,

If you're on a Linux64 box you can only push the heap size out to 4GB or so, which may do the job for most of the images, but definitely not the 2GB images.  I think I'd take advantage of the native memory and write a script to call the kakadu app directly:

The general command-line djatoka uses to compress images is:
kdu_compress -quiet -i <tmpTiff> -o <outoutJP2> -slope 51651,51337,51186,50804,50548,50232 Cprecincts="{256,256},{256,256},{128,128}" Clayers=6 Corder=RPCL ORGtparts=R Cblk="{32,32}" ORGgen_plt=yes Creversible=yes

The only command line arguments you'll care about are the input (i.e. -i), output (i.e. -o), and the number of resolution levels (i.e. Clayers).  Using http://sourceforge.net/apps/mediawiki/djatoka/index.php?title=Djatoka_Level_Logic as a baseline, Clayers is the number of DWT levels (i.e. the number of time you can reasonably halve the image).  So if you start from the bottom figuring size 0 is 96px x 96px, and size1 is 192, size2 is 382, size3 is 768, and so on you just need to determine the appropriate Clayer value for each image.  If you have a standard capture resolution, you can calculate this value once manually and just write a shell script to process the images.  Note: The TIFF library used by kakadu does not support LZW and is pretty base standard.  If you have any high bit images or such, you may need to use ImageMagick or the like to convert the image.  

Is there a file you can provide as a reference, I might me able to throw together a quick util to facilitate the steps above.

Cheers,
Ryan

On Sep 3, 2010, at 6:50 AM, Nick Ruest wrote:

> The images are 600 dpi tifs and range in size from 350M - 2GB. World War 1 trench maps. I have 16GB of memory available on the machine I am running it one. Do you think it would help if I jacked up the memory even more or just do them in batches or around 200?
> 
> I'm really liking the compress script over mogrify. The output files are about 10x smaller than the originals and the speed at which the script processes the files is light years quicker.
> 
> thanks!
> 
> -nruest
> 
> On Sep 3, 2010, at 9:46 AM, Ryan Chute wrote:
> 
>> Hi Nick,
>> 
>> What resolution are the files?  For high resolution images ( eg > 6k x 6k) you're best off using a native library to convert to tiff then use the tiff as djatoka input. Java and large tiffs will use up lots of memory. 
>> 
>> Let us know how it goes. 
>> 
>> Cheers,
>> Ryan
>> 
>> On Sep 3, 2010, at 6:38 AM, Nick Ruest <ru...@mc...> wrote:
>> 
>>> Hello all,
>>> 
>>> I am trying to convert a folder of large tif files to jp2. It gets about halfway through each time, and quits with the following error:
>>> 
>>> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>>>      at java.awt.image.DataBufferByte.<init>(DataBufferByte.java:92)
>>>      at java.awt.image.ComponentSampleModel.createDataBuffer(ComponentSampleModel.java:415)
>>>      at non_com.media.jai.RasterFactory.createWritableRaster(Unknown Source)
>>>      at non_com.media.jai.codec.SimpleRenderedImage.getData(Unknown Source)
>>>      at non_com.media.jai.codec.SimpleRenderedImage.getData(Unknown Source)
>>>      at net.sf.ij.jaiio.ImagePlusCreator.forceTileUpdate(ImagePlusCreator.java:51)
>>>      at net.sf.ij.jaiio.JAIReader.read(JAIReader.java:292)
>>>      at net.sf.ij.jaiio.JAIReader.read(JAIReader.java:174)
>>>      at net.sf.ij.jaiio.JAIReader.read(JAIReader.java:129)
>>>      at gov.lanl.adore.djatoka.io.reader.ImageJReader.open(ImageJReader.java:84)
>>>      at gov.lanl.adore.djatoka.io.reader.DjatokaReader.open(DjatokaReader.java:55)
>>>      at gov.lanl.adore.djatoka.util.IOUtils.createTempTiff(IOUtils.java:60)
>>>      at gov.lanl.adore.djatoka.kdu.KduCompressExe.compressImage(KduCompressExe.java:293)
>>>      at gov.lanl.adore.djatoka.DjatokaCompress.compress(DjatokaCompress.java:173)
>>>      at gov.lanl.adore.djatoka.DjatokaCompress.main(DjatokaCompress.java:131)
>>> 
>>> I have tried boosting up the memory in compress.sh with the following addition:
>>> < java -Xms32m -Xmx512m -classpath ${CLASSPATH} ${JAVA_OPTS} gov.lanl.adore.djatoka.DjatokaCompress $* 
>>> ---
>>>> java -classpath ${CLASSPATH} ${JAVA_OPTS} gov.lanl.adore.djatoka.DjatokaCompress $* 
>>> 
>>> I tried 128, 256, and 512. It errored out in the same exact spot each time. I know it isn't the file, because I can run the compress script on the file itself, and it successfully converts.
>>> 
>>> Any suggestions?
>>> 
>>> thanks!
>>> 
>>> ------------------------------------------------------------
>>> Nick Ruest
>>> Digital Strategies Librarian
>>> Vice President - McMaster University Academic Librarians Association
>>> 
>>> McMaster University
>>> Mills Memorial Library
>>> 1280 Main Street West
>>> Hamilton, ON L8S 4L6
>>> Phone: 905.525.9140 ext. 21276
>>> Email: ru...@mc...
>>> http://library.mcmaster.ca/contact/ruest-nicholas
>>> http://ruebot.net/
>>> ------------------------------------------------------------
>>> 
>>> "Revolution is not something fixed in ideology, nor is it something fashioned to a particular decade.  It is a personal process embedded in the human spirit." - Abbie Hoffman
>>> 
>>> 
>>> ------------------------------------------------------------------------------
>>> This SF.net Dev2Dev email is sponsored by:
>>> 
>>> Show off your parallel programming skills.
>>> Enter the Intel(R) Threading Challenge 2010.
>>> http://p.sf.net/sfu/intel-thread-sfd
>>> _______________________________________________
>>> djatoka-devel mailing list
>>> dja...@li...
>>> https://lists.sourceforge.net/lists/listinfo/djatoka-devel
> 
> ------------------------------------------------------------
> Nick Ruest
> Digital Strategies Librarian
> Vice President - McMaster University Academic Librarians Association
> 
> McMaster University
> Mills Memorial Library
> 1280 Main Street West
> Hamilton, ON L8S 4L6
> Phone: 905.525.9140 ext. 21276
> Email: ru...@mc...
> http://library.mcmaster.ca/contact/ruest-nicholas
> http://ruebot.net/
> ------------------------------------------------------------
> 
> "Revolution is not something fixed in ideology, nor is it something fashioned to a particular decade.  It is a personal process embedded in the human spirit." - Abbie Hoffman
> 