Sorry, I should have specific. Yes, I am using saxon:discard-document to mark documents for GC.

Here is my typical work-flow.

I have a file-list of relative URIs (either csv or in an xml file), each file has the format
<pedagogicalTrace>
<student/>
<session/>
<session/>
...
<pedagogicalTrace>

and these URIs are fed one at a time into:

<xsl:apply-templates select="saxon:discard-document(document(concat($path,@session.URI)))/pedagogicalTrace">
                    <xsl:with-param name="student.information" select="."/>
         </xsl:apply-templates>

[@session.URI is the local file-name in my xml table of files, so concat($path,@session.URI) is the document's absolute file name.]

Then, my main script will often start off with something like:

<xsl:template match="pedagogicalTrace">
        <xsl:param name="student.information"/>
        <xsl:variable name="session.file" select="saxon:discard-document(document(base-uri(.)))"/>       
        <xsl:text>
</xsl:text>
        <student username="{$session.file/pedagogicalTrace[1]/student[1]/@userName}">
            <xsl:for-each-group select="$session.file//session" group-adjacent="date:date(@startedOn)">

....
</xsl:for-each-group>
</student>

I just noticed that this means I'm actually marking the files as available for GC in two different places, once when the <xsl:apply-templates> is called and again when the $session.file variable is utilized.

-David









On Sat, Feb 8, 2014 at 3:29 PM, Michael Kay <mike@saxonica.com> wrote:

Are you using saxon:discard-document() to enable garbage collection of documents?

Without it, documents are locked in memory for the duration of a transformation, with or without accumulators. It's very hard to do anything else because of the stability rules (reading the same document twice is required to produce the same nodes).

Michael Kay
Saxonica


On 8 Feb 2014, at 00:16, David Rudel <fwqhgads@gmail.com> wrote:

> So I got my accumulator application working, and I loved it.
>
> Then I went to apply it to a large collection of files and ran into memory problems.
>
> My accumulator is non-streaming (though I may be able to create a streamable version).
>
> I assume one of two things is happening:
> The existence of the accumulator prevents the documents from being discarded.
> or
> The documents are being successfully discarded, but the memory from the accumulator itself is not being released.
>
> Is this expected behavior? Is there a way around it?
>
> -David
>
> --
>
> "A false conclusion, once arrived at and widely accepted is not dislodged easily, and the less it is understood, the more tenaciously it is held." - Cantor's Law of Preservation of Ignorance.
> ------------------------------------------------------------------------------
> Managing the Performance of Cloud-Based Applications
> Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
> Read the Whitepaper.
> http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk_______________________________________________
> saxon-help mailing list archived at http://saxon.markmail.org/
> saxon-help@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/saxon-help


------------------------------------------------------------------------------
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
_______________________________________________
saxon-help mailing list archived at http://saxon.markmail.org/
saxon-help@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/saxon-help



--

"A false conclusion, once arrived at and widely accepted is not dislodged easily, and the less it is understood, the more tenaciously it is held." - Cantor's Law of Preservation of Ignorance.