The problem seems to be with the class LargeStringBuffer when handling very large strings (10m characters). I haven't got to the bottom of it yet - it's years since I looked at this code - but I'm on the trail.

Michael Kay
Saxonica

On 17/03/2012 00:11, Michael Kay wrote:

Thanks.

This fails for me with default memory settings, but succeeds if I give it -Xmx2048m, reporting

Execution time: 55.811s (55811ms)
Memory used: 1333035256

It's definitely using streaming on the input file, but appears to be using memory heavily for something else. I suspect that for some reason the space used for the temporary trees isn't being released for garbage collection.

The problem seems to be related to the existence of very large text nodes, of which the longest seems to be almost 10m characters.

The investigation continues...

Michael Kay
Saxonica

On 16/03/2012 21:11, Todd Gochenour wrote:
Sample file can be accessed on Google Docs at https://docs.google.com/open?id=0B6HMTxI5Bg7QaTF2YWYtZjFUQS1mM1l1Si1oUlpVdw
 
Sample XSLT is:

<?xml version="1.0" encoding="UTF-8"?>


<xsl:stylesheet version="2.1"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:saxon="http://saxon.sf.net/"
xmlns:xsi="http://dummy.instance.net/"
xmlns:xs="http://dummy.schema.net/"
>

<xsl:template match="/">
<xsl:for-each select="saxon:stream(doc('test7.xml')/*/*/table_data)">
<xsl:variable name="table-name" select="@name"/>
<xsl:variable name="collection-uri" select="concat('db/',parent::*/@name,'/',$table-name)"/>
<xsl:for-each select="row">
<xsl:variable name="record">
<xsl:element name="{$table-name}">
<xsl:for-each select="field[@name]">
<xsl:element name="{if(number(substring(@name,1,1))=number(substring(@name,1,1))) then concat('_',@name) else @name}">
<xsl:value-of select="text()"/>
</xsl:element>
</xsl:for-each>
</xsl:element>
</xsl:variable>
<xsl:variable name="id" select="
if($table-name='cal_cyc') then concat('id-',$record/*/id,'-mnth-',$record/*/mnth,'-wk-',$record/*/wk)
else if($record/*/id) then concat('id-',$record/*/id)
else concat('ndx-',position())"/>
<xsl:result-document href="{$collection-uri}/{$id}.xml">
<xsl:copy-of select="$record"/>
</xsl:result-document>
</xsl:for-each>
</xsl:for-each>
</xsl:template>
</xsl:stylesheet>



------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sfd2d-msazure


_______________________________________________
saxon-help mailing list archived at http://saxon.markmail.org/
saxon-help@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/saxon-help