Jython

At 09:31 30.06.2003 -0400, Birkinbine, Brian wrote:
>Hello,
>
>I'm having problems releasing memory using jython with java objects.
>
>I have an infinite loop where I read in XML files from a directory, process
>the XML files using XSLT to create new XML files in another directory.  I
>then wait x number of seconds (in this case 1 hour) before checking for more
>files to be processed.
>
>I've seen cases where my program will use around 500meg of memory if it
>processes a larger XML file (around 100meg), but during future file
>processing it holds onto the 500meg for much smaller XML file processing
>(less than 1meg).
>
>I've tried using the del keywork for my java objects, and I've also tried
>calling java.lang.System.gc() without success at releasing the memory.
>
>Here are the java classes I'm using.
>java.util.Properties (to read from properties file for configuration)
>javax.xml.transform.stream.StreamSource for the XML input, output, and XSLT
>source.
>javax.xml.transform.TransformerFactory.newInstance() and
>javax.xml.transform.Transformer.newTransformer() for XSLT.
>
>Thanks,
>Brian Birkinbine

I don't think it has something to do with Jython. I would not be surprised 
if after processing 100mb the JVM is holding onto 500mb of OS mem.

To check, as you did, use del, java.lang.System.gc() and then print

java.lang.Runtime.getRuntime().freeMemory()
java.lang.Runtime.getRuntime().totalMemory()

regards.

RE: [Jython-users] jython and java garbage collection From: Birkinbine, Brian <brian.birkinbine@ci...> - 2003-07-01 19:17 I'll have to read more about how the JVM uses memory. Are there any good tools/methods for determining the amount of memory for different java objects? I used jythonc to compile a jar containing my jython script. Here are the results of Samuele's suggestion for gathering memory statistics. DEBUG:Before XSLT infinite loop start: Memory free: 1,244,032 DEBUG:Before XSLT infinite loop start: Total memory: 4,389,376 ------------------------------------- processing 100,447,720 byte XML file DEBUG:After XSLT before del: Memory free: 143,054,200 DEBUG:After XSLT before del: Total memory: 388,102,656 DEBUG:Sleeping 60 seconds DEBUG:After XSLT after del before gc Memory free: 143,032,168 DEBUG:After XSLT after del before gc Total memory: 388,102,656 DEBUG:Sleeping 60 seconds after calling gc DEBUG:After XSLT after del after gc Memory free: 201,611,176 DEBUG:After XSLT after del after gc Total memory: 368,769,536 167,158,360 still in use ------------------------------------- processing 154 byte XML file (1 hour later) DEBUG:After XSLT before del: Memory free: 199,753,576 DEBUG:After XSLT before del: Total memory: 368,769,536 DEBUG:Sleeping 60 seconds DEBUG:After XSLT after del before gc Memory free: 199,753,576 DEBUG:After XSLT after del before gc Total memory: 368,769,536 DEBUG:Sleeping 60 seconds after calling gc DEBUG:After XSLT after del after gc Memory free: 201,441,776 DEBUG:After XSLT after del after gc Total memory: 368,769,536 167,327,760 still in use ------------------------------------- processing 154 byte XML file (20 hours later) DEBUG:After XSLT before del: Memory free: 199,289,240 DEBUG:After XSLT before del: Total memory: 368,769,536 DEBUG:Sleeping 60 seconds DEBUG:After XSLT after del before gc Memory free: 199,246,960 DEBUG:After XSLT after del before gc Total memory: 368,769,536 DEBUG:Sleeping 60 seconds after calling gc DEBUG:After XSLT after del after gc Memory free: 201,411,704 DEBUG:After XSLT after del after gc Total memory: 368,769,536 167,357,832 still in use -----Original Message----- From: Samuele Pedroni [mailto:pedronis@...] Sent: Monday, June 30, 2003 10:42 AM To: Birkinbine, Brian; jython-users@... Subject: Re: [Jython-users] jython and java garbage collection At 09:31 30.06.2003 -0400, Birkinbine, Brian wrote: >Hello, > >I'm having problems releasing memory using jython with java objects. > >I have an infinite loop where I read in XML files from a directory, process >the XML files using XSLT to create new XML files in another directory. I >then wait x number of seconds (in this case 1 hour) before checking for more >files to be processed. > >I've seen cases where my program will use around 500meg of memory if it >processes a larger XML file (around 100meg), but during future file >processing it holds onto the 500meg for much smaller XML file processing >(less than 1meg). > >I've tried using the del keywork for my java objects, and I've also tried >calling java.lang.System.gc() without success at releasing the memory. > >Here are the java classes I'm using. >java.util.Properties (to read from properties file for configuration) >javax.xml.transform.stream.StreamSource for the XML input, output, and XSLT >source. >javax.xml.transform.TransformerFactory.newInstance() and >javax.xml.transform.Transformer.newTransformer() for XSLT. > >Thanks, >Brian Birkinbine I don't think it has something to do with Jython. I would not be surprised if after processing 100mb the JVM is holding onto 500mb of OS mem. To check, as you did, use del, java.lang.System.gc() and then print java.lang.Runtime.getRuntime().freeMemory() java.lang.Runtime.getRuntime().totalMemory() regards.

RE: [Jython-users] jython and java garbage collection From: Birkinbine, Brian <brian.birkinbine@ci...> - 2003-07-02 20:37 I seem to have a lot better results if I change my jython code to just return javax.xml.transform.stream.StreamSource(), then use the setInputStream() method to load the input files instead of passing an argument to the StreamSource constructor directly. I'm not sure if jython/jythonc handles arguments passed to a constructor differently than using another method to load the input file. If not, I probably just wasn't patient enough during my firsts tests (although 20 hours later, I would have thought the memory would have been freed up). Perhaps there are differences in calling this way (original test results) xslinput = javax.xml.transform.stream.StreamSource(xslinputfile) instead of (new test results) xslinput = javax.xml.transform.stream.StreamSource() xslinput.setInputStream(xslinputfile) Thanks, Brian - - new test results - - DEBUG:Before XSLT infinite loop start: Memory free: 1,244,000 DEBUG:Before XSLT infinite loop start: Total memory: 4,389,376 3,145,376 in use ------------------------------------- processing 100,447,720 byte XML file DEBUG:After XSLT before del: Memory free: 36,371,320 DEBUG:After XSLT before del: Total memory: 387,774,976 DEBUG:Sleeping 60 seconds DEBUG:After XSLT after del before gc Memory free: 36,327,912 DEBUG:After XSLT after del before gc Total memory: 387,774,976 DEBUG:Sleeping 60 seconds after calling gc DEBUG:After XSLT after del after gc Memory free: 385,602,344 DEBUG:After XSLT after del after gc Total memory: 387,774,976 2,172,632 still in use ------------------------------------- processing 154 byte XML file (1 hour later) DEBUG:After XSLT before del: Memory free: 383,744,656 DEBUG:After XSLT before del: Total memory: 387,774,976 DEBUG:Sleeping 60 seconds DEBUG:After XSLT after del before gc Memory free: 383,744,656 DEBUG:After XSLT after del before gc Total memory: 387,774,976 DEBUG:Sleeping 60 seconds after calling gc DEBUG:After XSLT after del after gc Memory free: 385,435,416 DEBUG:After XSLT after del after gc Total memory: 387,774,976 2,339,560 still in use ------------------------------------- processing 100,447,720 byte XML file (2 hours later) DEBUG:After XSLT before del: Memory free: 114635448 DEBUG:After XSLT before del: Total memory: 405076480 DEBUG:Sleeping 60 seconds DEBUG:After XSLT after del before gc Memory free: 114635448 DEBUG:After XSLT after del before gc Total memory: 405076480 DEBUG:Sleeping 60 seconds after calling gc DEBUG:After XSLT after del after gc Memory free: 237882928 DEBUG:After XSLT after del after gc Total memory: 405076480 167,193,552 still in use ------------------------------------- processing 154 byte XML file (3 hours later) DEBUG:After XSLT before del: Memory free: 236,091,240 DEBUG:After XSLT before del: Total memory: 405,076,480 DEBUG:Sleeping 60 seconds DEBUG:After XSLT after del before gc Memory free: 236,091,240 DEBUG:After XSLT after del before gc Total memory: 405,076,480 DEBUG:Sleeping 60 seconds after calling gc DEBUG:After XSLT after del after gc Memory free: 402,385,376 DEBUG:After XSLT after del after gc Total memory: 405,076,480 2,691,104 still in use -----Original Message----- From: Birkinbine, Brian Sent: Tuesday, July 01, 2003 1:56 PM To: Samuele Pedroni; jython-users@... Subject: RE: [Jython-users] jython and java garbage collection I'll have to read more about how the JVM uses memory. Are there any good tools/methods for determining the amount of memory for different java objects? I used jythonc to compile a jar containing my jython script. Here are the results of Samuele's suggestion for gathering memory statistics. DEBUG:Before XSLT infinite loop start: Memory free: 1,244,032 DEBUG:Before XSLT infinite loop start: Total memory: 4,389,376 ------------------------------------- processing 100,447,720 byte XML file DEBUG:After XSLT before del: Memory free: 143,054,200 DEBUG:After XSLT before del: Total memory: 388,102,656 DEBUG:Sleeping 60 seconds DEBUG:After XSLT after del before gc Memory free: 143,032,168 DEBUG:After XSLT after del before gc Total memory: 388,102,656 DEBUG:Sleeping 60 seconds after calling gc DEBUG:After XSLT after del after gc Memory free: 201,611,176 DEBUG:After XSLT after del after gc Total memory: 368,769,536 167,158,360 still in use ------------------------------------- processing 154 byte XML file (1 hour later) DEBUG:After XSLT before del: Memory free: 199,753,576 DEBUG:After XSLT before del: Total memory: 368,769,536 DEBUG:Sleeping 60 seconds DEBUG:After XSLT after del before gc Memory free: 199,753,576 DEBUG:After XSLT after del before gc Total memory: 368,769,536 DEBUG:Sleeping 60 seconds after calling gc DEBUG:After XSLT after del after gc Memory free: 201,441,776 DEBUG:After XSLT after del after gc Total memory: 368,769,536 167,327,760 still in use ------------------------------------- processing 154 byte XML file (20 hours later) DEBUG:After XSLT before del: Memory free: 199,289,240 DEBUG:After XSLT before del: Total memory: 368,769,536 DEBUG:Sleeping 60 seconds DEBUG:After XSLT after del before gc Memory free: 199,246,960 DEBUG:After XSLT after del before gc Total memory: 368,769,536 DEBUG:Sleeping 60 seconds after calling gc DEBUG:After XSLT after del after gc Memory free: 201,411,704 DEBUG:After XSLT after del after gc Total memory: 368,769,536 167,357,832 still in use -----Original Message----- From: Samuele Pedroni [mailto:pedronis@...] Sent: Monday, June 30, 2003 10:42 AM To: Birkinbine, Brian; jython-users@... Subject: Re: [Jython-users] jython and java garbage collection At 09:31 30.06.2003 -0400, Birkinbine, Brian wrote: >Hello, > >I'm having problems releasing memory using jython with java objects. > >I have an infinite loop where I read in XML files from a directory, process >the XML files using XSLT to create new XML files in another directory. I >then wait x number of seconds (in this case 1 hour) before checking for more >files to be processed. > >I've seen cases where my program will use around 500meg of memory if it >processes a larger XML file (around 100meg), but during future file >processing it holds onto the 500meg for much smaller XML file processing >(less than 1meg). > >I've tried using the del keywork for my java objects, and I've also tried >calling java.lang.System.gc() without success at releasing the memory. > >Here are the java classes I'm using. >java.util.Properties (to read from properties file for configuration) >javax.xml.transform.stream.StreamSource for the XML input, output, and XSLT >source. >javax.xml.transform.TransformerFactory.newInstance() and >javax.xml.transform.Transformer.newTransformer() for XSLT. > >Thanks, >Brian Birkinbine I don't think it has something to do with Jython. I would not be surprised if after processing 100mb the JVM is holding onto 500mb of OS mem. To check, as you did, use del, java.lang.System.gc() and then print java.lang.Runtime.getRuntime().freeMemory() java.lang.Runtime.getRuntime().totalMemory() regards. ------------------------------------------------------- This SF.Net email sponsored by: Free pre-built ASP.NET sites including Data Reports, E-commerce, Portals, and Forums are available now. Download today and enter to win an XBOX or Visual Studio .NET. http://aspnet.click-url.com/go/psa00100006ave/direct;at.asp_061203_01/01 _______________________________________________ Jython-users mailing list Jython-users@... https://lists.sourceforge.net/lists/listinfo/jython-users

1 message has been excluded from this view by a project administrator.