I'm running the StyleSheet transformation provided in saxon:
java com.icl.saxon.StyleSheet -o output.txt input.xml stylesheet.xsl
It runs nicely for a 2Mb xml file. But I need to parse a 50Mb file, in which
case I get a java.lang.outOfMemoryError even when I upped the JVM memory
from 32Mb to 100Mb. The stylesheet does not have any deep recursion.
Is it possible that this is because the parsed output is accumulating in
memory instead of being written directly to output.txt file? If not, any
other suggestions as to why I'd be having this trouble?
This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.
I'm using Saxon 6.5 to process many small xml files. The output file =
should be around 30M. I'm getting a java.lang.outofmemory error message =
even after increasing the memory parameters. I tried using saxon:preview =
without luck (I'm new at this and may not have had all my ducks in a =
row.) I'm able to process the files in three batches; but, I would =
prefer to do it in one shot. Any suggestions?
From: Michael Kay <mike@sa...> - 2004-12-30 23:29:04
> I'm using Saxon 6.5 to process many small xml files. The
> output file should be around 30M. I'm getting a
> java.lang.outofmemory error message even after increasing the
> memory parameters. I tried using saxon:preview without luck
> (I'm new at this and may not have had all my ducks in a row.)
> I'm able to process the files in three batches; but, I would
> prefer to do it in one shot. Any suggestions?
What is the total size of the many small files, and how much memory did you
Are you copying the files? If so, is the copy necessary?
If the input files are processed one at a time, you may find the extension
function saxon:discard-document() useful. This discards a document from
memory once it has been processed. However, to use this you will have to
upgrade to Saxon 8.x.
From: Michael Kay <michael.kay@nt...> - 2002-02-01 15:04:40
> From: saxon-help-admin@...
> [mailto:saxon-help-admin@...]On Behalf Of Gottesman,
> I'm running the StyleSheet transformation provided in saxon:
> java com.icl.saxon.StyleSheet -o output.txt input.xml stylesheet.xsl
> It runs nicely for a 2Mb xml file. But I need to parse a 50Mb
> file, in which
> case I get a java.lang.outOfMemoryError even when I upped the
> JVM memory
> from 32Mb to 100Mb.
The memory needed for building the source tree is around 2 bytes per
character of content plus 19 bytes per node (with the tinytree
implementation). If your 50 Mb XML file consisted of
<b>Some data and more</b>
repeated two million times, the tree would occupy around 148Mbytes of
XSLT is an in-memory technology. I've heard reports of people transforming a
100Mb document successfully, but it needs a lot of memory to achieve it.
If your transformation is basically serial in nature, take a look at
saxon:preview to reduce the memory requirements.