You might find it helpful to do this kind of investigation by running from the command line, for example:

java net.sf.saxon.Query -qs:"saxon:stream(doc('temp/test.xml')/*/*)/string()" -explain  -qversion:3.0

The -explain output in this case tells you:

OPT ======================================
OPT : Cannot use streaming copy: expression is not provably in document order
OPT ======================================

though it's a little bit confusing because some of the expressions that can be streamed give you the same message, followed by

OPT ======================================
OPT : Using streaming copy
OPT ======================================

because the optimizer succeeds on its second attempt. But it's a lot better than trying to work out what's streaming and what isn't by measuring the time and memory usage.


On 24 Feb 2014, at 10:56, Rademacher, Gunther <Gunther.Rademacher@softwareag.com> wrote:

Thanks for the advice. I now have streaming queries working as I want them.
 
However I found that the extra FLWOR does make a difference. These queries are streaming:
 
for $x in saxon:stream(doc('uri')/*/*) return string($x)
 
for $x in saxon:stream(doc('uri')/*/*) return $x
 
for $x in saxon:stream(doc('uri')/*/*) return $x/*
 
but these are not:
 
saxon:stream(doc('uri')/*/*)/string()
 
saxon:stream(doc('uri')/*/*)
 
saxon:stream(doc('uri')/*/*)/*

What's happening here is that in all 6 cases, the first attempt to convert to a streaming expression happens before the analysis that the expression doesn't need sorting, and therefore fails. In the first three cases there is a second attempt (because it tries again after inlining the variable), and the second attempt succeeds. But in the last three cases there is no second attempt. I will fix this.

 
Where those three run out of memory in roughly 10 seconds, the following takes 160 seconds
before running out of memory, too:
 
for $x in saxon:stream(doc('uri')/*/*)/* return $x

can't see any reason for that difference!

Michael Kay
Saxonica

 
My test code is shown below. Tested on Saxon-EE 9.5.1.4J.
 
Best regards
Gunther
 
 
       @Test(dataProvider="streamingQueries")
       public void testStreamingSaxon(String query, long limit) throws Exception {
              XQueryExecutable xqueryExecutable = xqueryCompiler.compile(query);
              XQueryEvaluator xqueryEvaluator = xqueryExecutable.load();
              xqueryEvaluator.setURIResolver(new StandardURIResolver() {
                     private static final long serialVersionUID = 7097605951872171305L;
 
                     @Override
                     public Source resolve(String href, String base) throws XPathException {
                           return new StreamSource(new InputStream() {
                                  private static final String intro =
                                         "<song>\n";
                                  private static final String chorus =
                                         "  <chorus>\n" +
                                         "    <line>Let me take you down</line>\n" +
                                         "    <line>Cause I'm going to Strawberry Fields</line>\n" +
                                         "    <line>Nothing is real</line>\n" +
                                         "    <line>And nothing to get hung about</line>\n" +
                                         "    <line>Strawberry Fields forever</line>\n" +
                                         "  </chorus>\n";
                                  private int i = - intro.length();
                                 
                                  @Override
                                  public int read() throws IOException {
                                         if (i++ < 0) {
                                                return intro.charAt(i + intro.length() - 1);
                                         }
                                         else {
                                                i %= chorus.length();
                                                return chorus.charAt(i);
                                         }
                                  }
                           });
                     }
              });
 
              long start = System.currentTimeMillis();
              long count = 0;
              long nextStop = 1;
              for (Iterator<XdmItem> xdmItemIterator = xqueryEvaluator.iterator(); xdmItemIterator.hasNext(); ) {
                     xdmItemIterator.next();
                     if (++count == nextStop) {
                           long elapsed = System.currentTimeMillis() - start;
                           System.out.println("t(" + count + "): " + elapsed + " msec");
                           if (count >= limit)
                                  break;
                           nextStop <<= 1;
                     }
              }
       }
 
 
From: Michael Kay [mailto:mike@saxonica.com] 
Sent: Donnerstag, 20. Februar 2014 19:33
To: Mailing list for the SAXON XSLT and XQuery processor
Subject: Re: [saxon] Question on streaming via s9api
 
 
saxon:stream() is a pseudo-function; its effect depends on the form of the expression in its argument, not just on the value of the expression, and the simplest rule is that the expression must be a call on doc() followed by something that looks like an XSLT pattern. So having /string() on the end will spoil it. You don't need a FLWOR: the following should work equally well:
 
saxon:stream(doc('" + input + "')/*/*)/string()
 
 

Software AG – Sitz/Registered office: Uhlandstraße 12, 64297 Darmstadt, Germany – Registergericht/Commercial register: Darmstadt HRB 1562 - Vorstand/Management Board: Karl-Heinz Streibich (Vorsitzender/Chairman), Dr. Wolfram Jost, Arnd Zinnhardt; - Aufsichtsratsvorsitzender/Chairman of the Supervisory Board: Dr. Andreas Bereczky - http://www.softwareag.com
------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool.
Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer
Customize your own dashboards, set traffic alerts and generate reports.
Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk_______________________________________________
saxon-help mailing list archived at http://saxon.markmail.org/
saxon-help@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/saxon-help