Saxon XSLT and XQuery Processor

On 17/12/2007, Julio de la Vega <juvepo@...> wrote:
> I am working in a case where I have to transform an XML input in a more
> complex XML output. I have developed an XSLT that creates my XML output
> according to my requirements. My problems have started when I have begun to
> run huge xml input files because of problems of memory.

> Could you please give me an overview about how could I solve this case?

Increase the amount of memory available to the JVM using the xmx switch, eg:

java -Xmx1024M ........


cheers
-- 
Andrew Welch
http://andrewjwelch.com
Kernow: http://kernowforsaxon.sf.net/

How big is "huge"? For some people it's 100Mb, for others it's 100Gb.
 
There are two facilities now in Saxon for handling files that are too large
to fit in memory:
 
(1) "Streaming of large documents" in XSLT - see
http://www.saxonica.com/documentation/sourcedocs/serial.html
 
(2) "Document projection" in XQuery - see
http://www.saxonica.com/documentation/sourcedocs/projection.html
 
In both cases the feature only works for certain kinds of processing, so it
depends on the exact nature of the XML.
 
But if "huge" means say 200Mb, then it might be that all you need to do is
to set the right options to allocate memory to Java.
 
I can advise you at the coding level if you're having difficulty using
particular Saxon features, but if you need help at the design level, or if
you need help with the implementation, then please contact me off-list to
see if we can organize some consultancy arrangement.
 
Regards,
 
Michael Kay
http://www.saxonica.com/


  _____  

From: saxon-help-bounces@...
[mailto:saxon-help-bounces@...] On Behalf Of Julio de la
Vega
Sent: 17 December 2007 10:15
To: saxon-help@...
Subject: [saxon] Question about XSLT >> Huge Files



Hi *,

 

First of all, thank you for you time and help. 

 

I am working in a case where I have to transform an XML input in a more
complex XML output. I have developed an XSLT that creates my XML output
according to my requirements. My problems have started when I have begun to
run huge xml input files because of problems of memory. 

I have found information that makes me think than you can help me to solve
my issue. 

 

Could you please give me an overview about how could I solve this case?

 

Please do not hesitate to contact me if you need more information

 

Thanks again

 

Best Regards

 

Julio de la Vega

Thank you for your answers.



Increasing the amount of memory available to the JVM using the xmx was the
first step we did but it is no enough for the size of our files. At the
moment my input xml files are more or less 3 Gb but they will be bigger in
the future.



In my first tests with files about 500 Mb, the amount of memory in use
increased to big values so it was impossible to do the XSLT transformation
because of java heap errors.


2007/12/17, Andrew Welch <andrew.j.welch@...>:
>
> On 17/12/2007, Julio de la Vega <juvepo@...> wrote:
> > I am working in a case where I have to transform an XML input in a more
> > complex XML output. I have developed an XSLT that creates my XML output
> > according to my requirements. My problems have started when I have begun
> to
> > run huge xml input files because of problems of memory.
>
> > Could you please give me an overview about how could I solve this case?
>
> Increase the amount of memory available to the JVM using the xmx switch,
> eg:
>
> java -Xmx1024M ........
>
>
> cheers
> --
> Andrew Welch
> http://andrewjwelch.com
> Kernow: http://kernowforsaxon.sf.net/
>
> -------------------------------------------------------------------------
> SF.Net email is sponsored by:
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services
> for just about anything Open Source.
>
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________
> saxon-help mailing list
> saxon-help@...
> https://lists.sourceforge.net/lists/listinfo/saxon-help
>

OK, for 3Gb you need to look at the other options such as streaming
transformations or document projection. Note, both these are Saxon-SA
options. (Or you could do your own preprocessing with a SAX filter, etc.)
 
Michael Kay
http://www.saxonica.com/


  _____  

From: saxon-help-bounces@...
[mailto:saxon-help-bounces@...] On Behalf Of Julio de la
Vega
Sent: 17 December 2007 11:02
To: Mailing list for SAXON XSLT queries
Subject: Re: [saxon] Question about XSLT >> Huge Files



Thank you for your answers.

 

Increasing the amount of memory available to the JVM using the xmx was the
first step we did but it is no enough for the size of our files. At the
moment my input xml files are more or less 3 Gb but they will be bigger in
the future. 

 

In my first tests with files about 500 Mb, the amount of memory in use
increased to big values so it was impossible to do the XSLT transformation
because of java heap errors. 



2007/12/17, Andrew Welch <andrew.j.welch@...>: 

On 17/12/2007, Julio de la Vega <juvepo@...> wrote:
> I am working in a case where I have to transform an XML input in a more 
> complex XML output. I have developed an XSLT that creates my XML output
> according to my requirements. My problems have started when I have begun
to
> run huge xml input files because of problems of memory. 

> Could you please give me an overview about how could I solve this case?

Increase the amount of memory available to the JVM using the xmx switch, eg:

java -Xmx1024M ........


cheers
--
Andrew Welch
http://andrewjwelch.com
Kernow: http://kernowforsaxon.sf.net/

------------------------------------------------------------------------- 
SF.Net email is sponsored by:
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services
for just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
saxon-help mailing list
saxon-help@...  <mailto:saxon-help@...> 
https://lists.sourceforge.net/lists/listinfo/saxon-help

On 17/12/2007, Julio de la Vega <juvepo@...> wrote:
> Increasing the amount of memory available to the JVM using the xmx was the
> first step we did but it is no enough for the size of our files. At the
> moment my input xml files are more or less 3 Gb but they will be bigger in
> the future.
>
> In my first tests with files about 500 Mb, the amount of memory in use
> increased to big values so it was impossible to do the XSLT transformation
> because of java heap errors.

yep, 3gb is huge.

In which case you should investigate:

(1) "Streaming of large documents" in XSLT - see
http://www.saxonica.com/documentation/sourcedocs/serial.html

The alternative is to roll your own - write a wrapper around the
transformation that chunks the input into manageable sizes (usually
based the repeating structure of the input), transforms each one, and
then re-constructs the multiple output files into a single file - if
needed.

Out of interest, what industry are you in?  I wonder what sort of
applications use or create 3gb xml files?

Maybe the solution isn't to cope with an unmanageable 3gb file, but to
alter whatever creates it to generate many smaller files...


-- 
Andrew Welch
http://andrewjwelch.com
Kernow: http://kernowforsaxon.sf.net/

I am working for a bank in this issue. They generate huge flat files that I
have to transform to xml. I stimate that my ouput files will be between 2
and 3 Gb. I will review your information and try to find the best solution.
Please, more ideas are wellcome

Thanks again


2007/12/17, Andrew Welch <andrew.j.welch@...>:
>
> On 17/12/2007, Julio de la Vega <juvepo@...> wrote:
> > Increasing the amount of memory available to the JVM using the xmx was
> the
> > first step we did but it is no enough for the size of our files. At the
> > moment my input xml files are more or less 3 Gb but they will be bigger
> in
> > the future.
> >
> > In my first tests with files about 500 Mb, the amount of memory in use
> > increased to big values so it was impossible to do the XSLT
> transformation
> > because of java heap errors.
>
> yep, 3gb is huge.
>
> In which case you should investigate:
>
> (1) "Streaming of large documents" in XSLT - see
> http://www.saxonica.com/documentation/sourcedocs/serial.html
>
> The alternative is to roll your own - write a wrapper around the
> transformation that chunks the input into manageable sizes (usually
> based the repeating structure of the input), transforms each one, and
> then re-constructs the multiple output files into a single file - if
> needed.
>
> Out of interest, what industry are you in?  I wonder what sort of
> applications use or create 3gb xml files?
>
> Maybe the solution isn't to cope with an unmanageable 3gb file, but to
> alter whatever creates it to generate many smaller files...
>
>
> --
> Andrew Welch
> http://andrewjwelch.com
> Kernow: http://kernowforsaxon.sf.net/
>
> -------------------------------------------------------------------------
> SF.Net email is sponsored by:
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services
> for just about anything Open Source.
>
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________
> saxon-help mailing list
> saxon-help@...
> https://lists.sourceforge.net/lists/listinfo/saxon-help
>

On Dec 17, 2007 3:25 AM, Julio de la Vega <juvepo@...> wrote:
> I am working for a bank in this issue. They generate huge flat files that I
> have to transform to xml. I stimate that my ouput files will be between 2
> and 3 Gb. I will review your information and try to find the best solution.

Then your application should not just transform the huge text input
file into a single xml file. You are in full control to try to produce
many xml files from the huge text file.

One solution could be to provide a single index.xml file, describing
the structure, ordering and relationships of these files:

<index>
  <file name="file0001"/>
  <file name="file0002"/>
  <!-- ................ -->
  <file name="file9999"/>
</index>



-- 
Cheers,
Dimitre Novatchev
---------------------------------------
Truly great madness cannot be achieved without significant intelligence.
---------------------------------------
To invent, you need a good imagination and a pile of junk
-------------------------------------
Never fight an inanimate object
-------------------------------------
You've achieved success in your field when you don't know whether what
you're doing is work or play

--- Julio de la Vega <juvepo@...> wrote:

> I am working for a bank in this issue. They generate
> huge flat files that I
> have to transform to xml. I stimate that my ouput
> files will be between 2
> and 3 Gb. I will review your information and try to
> find the best solution.
> Please, more ideas are wellcome

You should consider streaming xml processing. SAX is
an option, but for many simple transformations Stax is
more convenient as it does not require you to reverse
the control flow in unnatural ways. SAX is good if you
need pipelining, you can use unix-style small filters
to cobble together bigger transformations.

For some types of transformations I have used StaxMate
helper library with some success -- it can do limited
amount of buffering (-> out-of-order output) which is
nice if sub-trees require structural changes and
sub-trees themselves have limited size.

-+ Tatu +-



      ____________________________________________________________________________________
Be a better friend, newshound, and 
know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

--- Dimitre Novatchev <dnovatchev@...> wrote:

> On Dec 17, 2007 3:25 AM, Julio de la Vega
> <juvepo@...> wrote:
> > I am working for a bank in this issue. They
> generate huge flat files that I
> > have to transform to xml. I stimate that my ouput
> files will be between 2
> > and 3 Gb. I will review your information and try
> to find the best solution.
> 
> Then your application should not just transform the
> huge text input
> file into a single xml file. You are in full control
> to try to produce
> many xml files from the huge text file.

Besides, if input is text, not xml, why even bother
with xslt in the first place?
If you have to parse/splice textual content and
produce xml output, it's likely to be rather simple to
use any old xml writers for the output side. The
result could be done as multiple smaller files, or as
one huge file, depending on who has to process them
next.

-+ Tatu +-



      ____________________________________________________________________________________
Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs

I use Perl a lot to create xml out of flat files when each row can create a
part of the tree with no dependencies on what came before or after.
Generally this data comes from a relational database...

On Dec 17, 2007 11:28 PM, Tatu Saloranta <cowtowncoder@...> wrote:

> --- Dimitre Novatchev <dnovatchev@...> wrote:
>
> > On Dec 17, 2007 3:25 AM, Julio de la Vega
> > <juvepo@...> wrote:
> > > I am working for a bank in this issue. They
> > generate huge flat files that I
> > > have to transform to xml. I stimate that my ouput
> > files will be between 2
> > > and 3 Gb. I will review your information and try
> > to find the best solution.
> >
> > Then your application should not just transform the
> > huge text input
> > file into a single xml file. You are in full control
> > to try to produce
> > many xml files from the huge text file.
>
> Besides, if input is text, not xml, why even bother
> with xslt in the first place?
> If you have to parse/splice textual content and
> produce xml output, it's likely to be rather simple to
> use any old xml writers for the output side. The
> result could be done as multiple smaller files, or as
> one huge file, depending on who has to process them
> next.
>
> -+ Tatu +-
>
>
>
>
>  ____________________________________________________________________________________
> Never miss a thing.  Make Yahoo your home page.
> http://www.yahoo.com/r/hs
>
> -------------------------------------------------------------------------
> SF.Net email is sponsored by:
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services
> for just about anything Open Source.
>
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________
> saxon-help mailing list
> saxon-help@...
> https://lists.sourceforge.net/lists/listinfo/saxon-help
>

I think there is a streaming problem(?) related to how Saxon handles writing
to databases.

I am processing two huge files, one 2GB and another 30GB. Initially, I had
written code Saxon/XSLT that processes the source XML files and outputs the
results to csv files. This worked fine for both files - meaning that the
code was well written using all the necessary streaming related code etc to
enable processing of huge files. In both cases, I didn't even have to use
the -Xmx java switch to allocate more memory to the JVM.

I am now processing the same files but with the Saxon/XSLT code modified to
output to a MySQL database. I am always getting a "Exception in thread
"main" java.lang.OutOfMemoryError: Java heap space" error. This error occurs
when its building a tree; specifically, the heap error occurs after this
statement has been displayed: 

"Building tree for file:<file path> using class
net.sf.saxon.tree.TreeBuilder"

(I have set it to use linked instead of tiny trees though none of the two
options seem to make a difference)

The only way I could make it work was to use the -Xmx switch to allocate
alot more memory to the JVM. For the 30GB file, I set it at a very large
value (30000M); it managed to build the tree write out records but it was
extremely slow.

Questions:

1. Does Saxon use different techniques to build the trees depending on what
type of output it has to generate?

2. Is streaming disabled when output is a database as opposed to csv files?
Why didn't I need to use the -Xmx switch when outputting to csv files?



Michael Kay wrote:
> 
> How big is "huge"? For some people it's 100Mb, for others it's 100Gb.
>  
> There are two facilities now in Saxon for handling files that are too
> large
> to fit in memory:
>  
> (1) "Streaming of large documents" in XSLT - see
> http://www.saxonica.com/documentation/sourcedocs/serial.html
>  
> (2) "Document projection" in XQuery - see
> http://www.saxonica.com/documentation/sourcedocs/projection.html
>  
> In both cases the feature only works for certain kinds of processing, so
> it
> depends on the exact nature of the XML.
>  
> But if "huge" means say 200Mb, then it might be that all you need to do is
> to set the right options to allocate memory to Java.
>  
> I can advise you at the coding level if you're having difficulty using
> particular Saxon features, but if you need help at the design level, or if
> you need help with the implementation, then please contact me off-list to
> see if we can organize some consultancy arrangement.
>  
> Regards,
>  
> Michael Kay
> http://www.saxonica.com/
> 
> 
>   _____  
> 
> From: saxon-help-bounces@...
> [mailto:saxon-help-bounces@...] On Behalf Of Julio de la
> Vega
> Sent: 17 December 2007 10:15
> To: saxon-help@...
> Subject: [saxon] Question about XSLT >> Huge Files
> 
> 
> 
> Hi *,
> 
>  
> 
> First of all, thank you for you time and help. 
> 
>  
> 
> I am working in a case where I have to transform an XML input in a more
> complex XML output. I have developed an XSLT that creates my XML output
> according to my requirements. My problems have started when I have begun
> to
> run huge xml input files because of problems of memory. 
> 
> I have found information that makes me think than you can help me to solve
> my issue. 
> 
>  
> 
> Could you please give me an overview about how could I solve this case?
> 
>  
> 
> Please do not hesitate to contact me if you need more information
> 
>  
> 
> Thanks again
> 
>  
> 
> Best Regards
> 
>  
> 
> Julio de la Vega
> 
> 
> -------------------------------------------------------------------------
> SF.Net email is sponsored by:
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services
> for just about anything Open Source.
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________
> saxon-help mailing list
> saxon-help@...
> https://lists.sourceforge.net/lists/listinfo/saxon-help
> 
> 

-- 
View this message in context: http://www.nabble.com/Question-about-XSLT-%3E%3E-Huge-Files-tp14373477p21559911.html
Sent from the saxon-help mailing list archive at Nabble.com.