Thread: [djatoka-devel] Extending exec() in KduExtractExe/KduCompressExe
Status: Beta
Brought to you by:
rchute
From: Spencer T. <spe...@it...> - 2011-02-24 18:32:26
|
We have a need to use a different method than Runtime.exec() to run the underlying kdu commands. Why? Because our JVM heap is over 16Gb and we can't afford to fork it. Instead we use RMI to send a fork request to a small external JVM that actually runs the command and returns the resulting output stream. The modifications to accomplish this were small, refactoring the exec() calls into a separate, protected method so we can override it. I can submit patches against the distributed adore-djatoka-1.1 package if you are interested. -- Spencer Thomas Technical Architect, JSTOR spe...@it... <mailto:spe...@it...> +1-734-887-7004 ITHAKA is a not-for-profit organization dedicated to helping the academic community take full advantage of rapidly advancing information and networking technologies. We serve scholars, researchers and students by providing the content, tools, and services needed to preserve the scholarly record and to advance research and teaching in sustainable ways. We are committed to working in collaboration with other organizations to maximize benefits to our stakeholders. |
From: Younos A. <You...@bi...> - 2011-03-02 07:06:19
|
Yes please Spencer, I'd really appreciate if you provide the patch or source code.. I'm getting an out of memory exception in java.lang.ProcessBuilder.start upon calling kdu_expand. Best regards, Younos ABOULNAGA Software Development Engineer Bibliotheca Alexandrina Cell: +2 012 4490444 From: Spencer Thomas [mailto:spe...@it...] Sent: Thursday, February 24, 2011 7:57 PM To: dja...@li... Subject: [djatoka-devel] Extending exec() in KduExtractExe/KduCompressExe We have a need to use a different method than Runtime.exec() to run the underlying kdu commands. Why? Because our JVM heap is over 16Gb and we can't afford to fork it. Instead we use RMI to send a fork request to a small external JVM that actually runs the command and returns the resulting output stream. The modifications to accomplish this were small, refactoring the exec() calls into a separate, protected method so we can override it. I can submit patches against the distributed adore-djatoka-1.1 package if you are interested. -- Spencer Thomas Technical Architect, JSTOR spe...@it...<mailto:spe...@it...> +1-734-887-7004 ITHAKA is a not-for-profit organization dedicated to helping the academic community take full advantage of rapidly advancing information and networking technologies. We serve scholars, researchers and students by providing the content, tools, and services needed to preserve the scholarly record and to advance research and teaching in sustainable ways. We are committed to working in collaboration with other organizations to maximize benefits to our stakeholders. |
From: Fabio N. K. <fab...@gm...> - 2011-05-09 19:08:01
|
Hi Spencer, I would also really appreciate if you could provide the patch or source code for replacing calls to Runtime.exec(). We are currently having hanging issues with our Tomcat instance and my bet is that it's because of the high frequency of calls to Runtime.exec(). I think we've correctly taken care of consuming the streams, so the cause may be forking too much. By the way, we're using -Xms1024M for our JVM (Tomcat). Should we first try to decrease that number, or it won't work anyway if we call Runtime.exec() a lot? Best regards, Fabio On Thu, Feb 24, 2011 at 14:57, Spencer Thomas <spe...@it...>wrote: > We have a need to use a different method than Runtime.exec() to run the > underlying kdu commands. Why? Because our JVM heap is over 16Gb and we can't > afford to fork it. Instead we use RMI to send a fork request to a small > external JVM that actually runs the command and returns the resulting output > stream. > > The modifications to accomplish this were small, refactoring the exec() > calls into a separate, protected method so we can override it. > > I can submit patches against the distributed adore-djatoka-1.1 package if > you are interested. > -- > > Spencer Thomas > Technical Architect, JSTOR > spe...@it... > +1-734-887-7004 > > ITHAKA is a not-for-profit organization dedicated to helping the academic > community take full advantage of rapidly advancing information and > networking technologies. We serve scholars, researchers and students by > providing the content, tools, and services needed to preserve the scholarly > record and to advance research and teaching in sustainable ways. We are > committed to working in collaboration with other organizations to maximize > benefits to our stakeholders. > > > ------------------------------------------------------------------------------ > Free Software Download: Index, Search & Analyze Logs and other IT data in > Real-Time with Splunk. Collect, index and harness all the fast moving IT > data > generated by your applications, servers and devices whether physical, > virtual > or in the cloud. Deliver compliance at lower cost and gain new business > insights. http://p.sf.net/sfu/splunk-dev2dev > _______________________________________________ > djatoka-devel mailing list > dja...@li... > https://lists.sourceforge.net/lists/listinfo/djatoka-devel > > |
From: Spencer T. <Spe...@it...> - 2011-05-10 15:49:47
Attachments:
djatoka.patch
|
The patches are simple (attached), but the support required to use the patch is not. We wrote a separate process to which the exec() calls were transmitted by RMI, with the stdin/stdout/stderr piped back to the main process. The patch allows me to subclass in order to call the RMI: KduExtractExe kee = new KduExtractExe() { // Requires a refactored version of KduExtractExe with // Runtime.getRuntime().exec(...) replaced with a call to getProcess protected Process getProcess(String command, String[] envParms, File dir) throws IOException { return ProcessUtil.exec(command, envParms, dir); } }; Another change, which might help you, is to wrap the invocation of the extractor with a semaphore to permit only a certain number to run at one time. This could prevent you from being overwhelmed. private static final Semaphore conversionRateLimit = new Semaphore(maxJp2Rate, true); … if (maxJp2Rate > 0) { try { if (!conversionRateLimit.tryAcquire(0, TimeUnit.SECONDS)) { log.debug("Waiting for semaphore"); conversionRateLimit.acquire(); log.debug("Acquired semaphore"); } } catch (InterruptedException e) { // Shouldn't happen? log.error("JP2 conversion interrupted waiting for semaphore"); } } try { return new ResponseBinaryData("image/jpeg", jp2.extractRegion(region)); } finally { if (maxJp2Rate > 0) conversionRateLimit.release(); } From: "Fabio N. Kepler" <fab...@gm...<mailto:fab...@gm...>> Date: Mon, 9 May 2011 15:07:34 -0400 To: Spencer Thomas <spe...@it...<mailto:spe...@it...>> Cc: "dja...@li...<mailto:dja...@li...>" <dja...@li...<mailto:dja...@li...>> Subject: Re: [djatoka-devel] Extending exec() in KduExtractExe/KduCompressExe Hi Spencer, I would also really appreciate if you could provide the patch or source code for replacing calls to Runtime.exec(). We are currently having hanging issues with our Tomcat instance and my bet is that it's because of the high frequency of calls to Runtime.exec(). I think we've correctly taken care of consuming the streams, so the cause may be forking too much. By the way, we're using -Xms1024M for our JVM (Tomcat). Should we first try to decrease that number, or it won't work anyway if we call Runtime.exec() a lot? Best regards, Fabio On Thu, Feb 24, 2011 at 14:57, Spencer Thomas <spe...@it...<mailto:spe...@it...>> wrote: We have a need to use a different method than Runtime.exec() to run the underlying kdu commands. Why? Because our JVM heap is over 16Gb and we can't afford to fork it. Instead we use RMI to send a fork request to a small external JVM that actually runs the command and returns the resulting output stream. The modifications to accomplish this were small, refactoring the exec() calls into a separate, protected method so we can override it. I can submit patches against the distributed adore-djatoka-1.1 package if you are interested. -- Spencer Thomas Technical Architect, JSTOR spe...@it...<mailto:spe...@it...> +1-734-887-7004 ITHAKA is a not-for-profit organization dedicated to helping the academic community take full advantage of rapidly advancing information and networking technologies. We serve scholars, researchers and students by providing the content, tools, and services needed to preserve the scholarly record and to advance research and teaching in sustainable ways. We are committed to working in collaboration with other organizations to maximize benefits to our stakeholders. ------------------------------------------------------------------------------ Free Software Download: Index, Search & Analyze Logs and other IT data in Real-Time with Splunk. Collect, index and harness all the fast moving IT data generated by your applications, servers and devices whether physical, virtual or in the cloud. Deliver compliance at lower cost and gain new business insights. http://p.sf.net/sfu/splunk-dev2dev _______________________________________________ djatoka-devel mailing list dja...@li...<mailto:dja...@li...> https://lists.sourceforge.net/lists/listinfo/djatoka-devel |