Thanks for these measurements.
First, it's important to note that the primary benefit of streaming is a reduction in memory usage, not an improvement in elapsed time. If you have enough memory to build the tree representation of the source document then it will usually be faster to use a non-streaming transformation; Saxon can take advantage of the navigation paths that exist in the tree. It's more complicated and therefore slower to do a streaming transformation. There's actually a lot of scope in Saxon for making it faster, but the emphasis so far has been on making it work at all.
The differences between piping the input to Saxon on stdin, and having it read from a file, only affect performance and memory use in the Java I/O subsystem, they are exactly the same at the Saxon level.
When you're streaming, there's going to be constant memory usage independent of file size, for holding the Java VM and class library, the Saxon software, the data tables these use, and the compiled stylesheet. I don't think it really matters whether this is 2Mb or 15Mb, the important thing is that it doesn't grow with file size.
When you are taking measurements at the Java command line, there's a lot of overhead (several seconds) in loading the Java VM and warming up the hotspot compiler (not to mention Saxon initialization such as locating and validating the license file). Running Java apps from the command line therefore always takes longer than non-Java apps. If running an individual transformation from the command line is what you are doing in your production workload, then it's fair to make measurements that include this overhead. If not, it's important to be aware that you measurements for your production workload will be different. Rather than an absolute figure for an identity transform of 100Mb, it would be interesting to see how this figure changes with file size. Other users have reported streaming rates of around 1Gb/min, which is roughly in line with my own experience.
I'm not familiar with GCX but I read here
that it doesn't support the whole of XML or the whole of XQuery. In other words, it's a research prototype. It's very easy in an acedemic project, by leaving out the difficult parts of the problem, to achieve significantly better performance than industrial products with a wider set of engineering goals. Such projects are useful as "pioneers" to identify targets one can aim at, but they cannot be compared directly with a project like Saxon.
On 14 May 2013, at 15:00, Sebastian Maneth wrote:
I am trying to stream the identity in XSLT, using Saxon.
I've done it in two different ways:
1) cat the xml file and pipe to Saxon
2) take the xml as input
3) and 4) as above, but without the streaming directive;
so, 3) and 4) run from memory.
What confuses me:
3) and 4) are *faster* than the streaming execution.
How can that be??
I find it also confusing that for streaming,
memory consumption increases by 50%
when I run from the cat-pipe, as when the file name is given.
Can this be explained? (-- it is faster though) -- see the numbers below.
Moreover, if I use another streaming tool, such as GCX (written in C++)
and run the identity (written in XQuery) then this is *much* faster; it takes
less than half of the time of Saxon.
Is this only due to the fact that Saxon is written in java and uses a java
I'm running the Saxon EE (9.5) edition with an evaluation copy.
I use this style sheet:
I ran in four different ways:
(1) cat 100M.xml | java -jar saxon9ee.jar .... -s:/dev/stdin
This takes 23.5s and takes 18Mb memory on my machine.
(2) java -jar .. -s:100M.xml
This takes 24.4s and 12.6Mb memory.
(3) I remove the "streamable=yes" line above and run non-streamed,
using the cat and stdin.
This takes 16.3s and uses 310M of memory.
(4) Then I run without cat but with the file name given to saxon, and this
takes 13.8s and 368M of memory.
As a comparison: GCX will use 2.6M of memory and stream the
identity (in XQuery) in 10s.
Thanks for your help,
AlienVault Unified Security Management (USM) platform delivers complete
security visibility with the essential security capabilities. Easily and
efficiently configure, manage, and operate all of your security controls
from a single console and one unified framework. Download a free trial.
saxon-help mailing list archived at http://saxon.markmail.org/