It's not a tail call because the result of the recursive call is processed by the caller after return: when you do
<xsl:sequence select="xyz"/>
<xsl:sequence select="f(xxx)"/>
it's the same as
<xsl:sequence select="xyz, f(xxx)"/>
where the "," is a sequence concatenation operator.
One way around this is to pass xyz as an extra parameter to the recursive function call, and have the concatenation done by the callee rather than the caller.
In fact the mechanism used for tail-call optimization on templates handles this situation, because it relies on the code being evaluated in push mode, where results are written to a Receiver as soon as they are generated, which means the function/template doesn't actually do a real concatenation. But functions don't currently work that way. So: no, it's not a bug, just a limitation of the current design.
Michael Kay

From: Cary Millsap []
Sent: 15 October 2008 17:50
Subject: [saxon] Too many nested template or function calls

Attached is a self-contained, reproducible test case (it includes Saxon that runs two slightly different XSL transforms. Each causes a different error:

1) Too many nested template or function calls. The stylesheet may be looping.
2) Exception in thread "main" java.lang.StackOverflowError

It seems to us that the function call in the XSLT should qualify as optimizable tail recursion, but it's seems not to be behaving that way...

Is this a Saxon bug?

Thank you in advance,

Cary Millsap