Update for this issue... the upgrade to saxon9 did not correct the issue.  It still occurrs after about 2 - 3 days of up time for the application we start to see this exception:
 
java.lang.ArrayIndexOutOfBoundsException: 5
 at net.sf.saxon.om.NamespaceIterator.getInScopeNamespaceCodes(NamespaceIterator.java:288)
 at net.sf.saxon.tree.ElementImpl.getInScopeNamespaceCodes(ElementImpl.java:670)
 at net.sf.saxon.style.LiteralResultElement.validate(LiteralResultElement.java:200)
 at net.sf.saxon.style.StyleElement.validateSubtree(StyleElement.java:1234)
 at net.sf.saxon.style.StyleElement.validateChildren(StyleElement.java:1263)
 at net.sf.saxon.style.StyleElement.validateSubtree(StyleElement.java:1238)
 at net.sf.saxon.style.StyleElement.validateChildren(StyleElement.java:1263)
 at net.sf.saxon.style.StyleElement.validateSubtree(StyleElement.java:1238)
 at net.sf.saxon.style.XSLStylesheet.preprocess(XSLStylesheet.java:705)
 at net.sf.saxon.PreparedStylesheet.setStylesheetDocument(PreparedStylesheet.java:331)
 at net.sf.saxon.PreparedStylesheet.prepare(PreparedStylesheet.java:163)
 at net.sf.saxon.TransformerFactoryImpl.newTemplates(TransformerFactoryImpl.java:139)
 at weblogic.xml.jaxp.RegistryTransformerFactory.newTemplates(RegistryTransformerFactory.java:173)
 
Once I restart the application it all works fine.  There seems to be plenty of free memory available and other transformers still seem to work okay.  It seems rather random as it's not always the same style sheets which have issues.
 
I'm really at a loss as to what the problem could be.  I cannot purposefully create the issue by stressing the application... and this latest time the application only had 38 transformations over the span of 3 days before the problem started to happen....
 
The application is running in a WebLogic 9.2 App Server using Java 1.5.0_10
 
Thanks!
Paul
 
 


From: Ackley, Paul [mailto:Paul.Ackley@qwest.com]
Sent: Tuesday, March 10, 2009 10:14 AM
To: Mailing list for the SAXON XSLT and XQuery processor
Subject: Re: [saxon] ArrayIndexOutOfBoundsException in Saxon 8.6

Slavek
 
I'll let you know if upgrading fixes the issue.  Odd thing is that no matter how many times I hit our application and cause it to regenerate the transformer, I cannot purposefuly cause this error to happen.  It seems to only happen over the span of a few days...  I've hit the application hard for hours, with many more requests then the app will see in 3 days and still the error doesn't happen.... strange.
 
I'm trying saxon9 in the next couple of days.
 
Paul


From: Michael Kay [mailto:mike@saxonica.com]
Sent: Thursday, February 26, 2009 10:45 AM
To: 'Mailing list for the SAXON XSLT and XQuery processor'
Subject: Re: [saxon] ArrayIndexOutOfBoundsException in Saxon 8.6

Yes, this is definitely what I would recommend: keep the Templates object around for as long as you can, but create a new Transformer for each transformation.
 
I'm puzzled by your previous message referring to a WebLogic TransformerFactory creating Saxon transformers. I don't really know WebLogic in any detail, people do occasionally report problems getting it to choose correctly between Saxon and Xalan, but if you're running Saxon then I can't see where the problem would be.
 
Saxon incidentally never caches compiled stylesheets itself. It leaves that to the application (or the next layer of software in the component hierarchy!).
 
I think that doing fewer compilations will reduce the stress on the NamePool and this may make the problem less frequent, but it doesn't really fix the bug. Moving to a more recent Saxon release might well remove the problem entirely. But it's still worth reducing unnecessary recompilation of stylesheets if you can.
 
Regards,
 
Michael Kay
Saxonica


From: Ackley, Paul [mailto:Paul.Ackley@qwest.com]
Sent: 26 February 2009 15:30
To: Ackley, Paul; Mailing list for the SAXON XSLTand XQuery processor
Subject: Re: [saxon] ArrayIndexOutOfBoundsException in Saxon 8.6

I am considering abandoning my caching of the Transformer instances and just use the Templates feature of JAXP/Saxon.  I ran some tests once the initial Templates object has been created, subsequent creations of the Transformer instance for a given style sheet are very quick (less than 10ms).  I did keep the "Templates" instance around to reuse it create the Transformer object.  From what I've read the Templates object can be reused and is thread safe..... just the Transformer objects are not thread safe...
 
Does this sound like a reasonable approach?
 
Thanks!
Paul


From: Ackley, Paul
Sent: Thursday, February 26, 2009 12:05 AM
To: Ackley, Paul; Mailing list for the SAXON XSLT and XQuery processor
Subject: RE: [saxon] ArrayIndexOutOfBoundsException in Saxon 8.6

Found out a few more details.  We were not using the TransformerFactory class from Saxon, we were using one from WebLogic.  Now this transformer factory was creating Saxon transformers... but the factory class was WebLogic's not Saxon's.  Do you think this is okay?  Could this be causing issues?  This is all configured under the XML Registry service inside WebLogic, so it's easy to switch to Saxon's TransformerFactory class...
 
Paul


From: Ackley, Paul
Sent: Wednesday, February 25, 2009 7:51 PM
To: Mailing list for the SAXON XSLT and XQuery processor
Cc: Ackley, Paul
Subject: RE: [saxon] ArrayIndexOutOfBoundsException in Saxon 8.6

Well yes what you said makes sense.  As to why the stylesheet is still compiling after 2 or 3 days it has historical reasons. Back when this was written (5 or 6 years ago) we handled the caching of transformer objects ourselves.  We cache them for a set amount of time.  If they are not accessed after a certain amount of time they are disposed of.  Then when they are needed we load them and cache them again.
 
Just today I was reading the Saxon documentation for 8.6 and I was starting to get the idea that Saxon is also capable of caching the transformer object (templates?) for better performance.  So I was begining to think that it may have something to do with this problem.
 
Our framework is capable to using an XSLT API - is this caching part of the standard now?  Is there a way to Saxon (newer version is okay) which doesn't use this feature?  May not matter as I may look into just letting Saxon handle the caching...
 
BTW - I did hit the application hard with a load test to try to duplicate the problem in a shorter amount of time, but have not been able to cause this issue intentionally.  It's pretty elusive.  I will look at the statistics part of things too and see if there is some growth (in usage) over time....
 
Thanks for the quick response.
Paul Ackley
 
 

From: Michael Kay [mike@saxonica.com]
Sent: Wednesday, February 25, 2009 4:48 PM
To: 'Mailing list for the SAXON XSLT and XQuery processor'
Subject: Re: [saxon] ArrayIndexOutOfBoundsException in Saxon 8.6

I don't recognize this one - but 8.6 is way out of date, so that's not surprising. Is there anything that stops you moving to a more recent release? It's one that would be pretty tough to debug if it happened on the current release, but on a release that old, it's nigh impossible.
 
It's failing during compilation of a stylesheet, which is unusual. In fact, I'm slightly surprised to see a stylesheet being compiled two days into a continuously running service - I would expect everything to be cached.
 
A possibility is that the NamePool is filling and that it's not failing cleanly when it fills. I seem to remember older releases weren't checking too carefully for this condition. Also, older releases of Saxon tended to consume entries in the NamePool during stylesheet compilation by generating randomized variable names. There's no direct evidence of a NamePool problem in the stack trace, but it could have distant effects - and when problems occur in a shared long-running workload it's usually the NamePool that's to blame. You could try dumping the NamePool when it fails: you can get to it from the TransformerFactory via the Configuration, and it has methods diagnosticDump() and statistics() which might yield some insights. Calling statistics() every hour or so can also be useful to see if it's growing steadily.
 
Michael Kay


From: Ackley, Paul [mailto:Paul.Ackley@qwest.com]
Sent: 25 February 2009 20:59
To: saxon-help@lists.sourceforge.net
Subject: [saxon] ArrayIndexOutOfBoundsException in Saxon 8.6

We are experiencing an odd problem with Saxon inside of a WebLogic version 9.2 Container.  When the application starts up it is able to load the XSL file fine and the transformer object is able to transfor the XML just fine.  Eventually, usually after 2 or 3 days of continuous uptime with not too heavy usage Saxon starts to throw this exception when loading a style sheet which has previously worked.  Once it starts having this issue all stylesheets fail to load with the same message. 
 
Here is the exception:
com.qwest.ecomm.xslt.XTPoolableObjectFactory INFO  2009-02-24 09:17:36,961 - Creating new XT object, XSL=./xsl/nw/ViewEBill.xsl
com.ecomm.xslt.XSLTObject INFO  2009-02-24 09:17:36,961 - TransformerFactory hard class=class weblogic.xml.jaxp.RegistrySAXTransformerFactory
com.ecomm.xslt.ECSSURIResolver INFO  2009-02-24 09:17:36,965 - href=lib_Common.xsl
com.ecomm.xslt.ECSSURIResolver INFO  2009-02-24 09:17:36,965 - base=file:///prod/ecom2/local/apps/apps92/5.0.0.22/xsl/nw/ViewEBill.xsl
com.ecomm.xslt.ECSSURIResolver INFO  2009-02-24 09:17:36,966 - Location of included file took 1ms to find.
com.ecomm.xslt.ECSSURIResolver INFO  2009-02-24 09:17:36,969 - href=lib_CreateEncryptedUrl.xsl
com.ecomm.xslt.ECSSURIResolver INFO  2009-02-24 09:17:36,969 - base=file:/prod/ecom2/local/apps/apps92/5.0.0.22/xsl/nw/lib_Common.xsl
com.ecomm.xslt.ECSSURIResolver INFO  2009-02-24 09:17:36,971 - Location of included file took 2ms to find.
com.ecomm.xslt.ECSSURIResolver INFO  2009-02-24 09:17:36,972 - href=lib_FormatColumnTitles.xsl
com.ecomm.xslt.ECSSURIResolver INFO  2009-02-24 09:17:36,972 - base=file:/prod/ecom2/local/apps/apps92/5.0.0.22/xsl/nw/lib_Common.xsl
com.ecomm.xslt.ECSSURIResolver INFO  2009-02-24 09:17:36,974 - Location of included file took 2ms to find.
com.ecomm.xslt.XSLTObject ERROR 2009-02-24 09:17:36,983 - Caught generic exception while creating a transformer object.
java.lang.ArrayIndexOutOfBoundsException: 5
 at net.sf.saxon.om.NamespaceIterator.getInScopeNamespaceCodes(NamespaceIterator.java:170)
 at net.sf.saxon.tree.ElementWithAttributes.getInScopeNamespaceCodes(ElementWithAttributes.java:260)
 at net.sf.saxon.style.LiteralResultElement.validate(LiteralResultElement.java:198)
 at net.sf.saxon.style.StyleElement.validateSubtree(StyleElement.java:1167)
 at net.sf.saxon.style.StyleElement.validateChildren(StyleElement.java:1201)
 at net.sf.saxon.style.StyleElement.validateSubtree(StyleElement.java:1176)
 at net.sf.saxon.style.StyleElement.validateChildren(StyleElement.java:1201)
 at net.sf.saxon.style.StyleElement.validateSubtree(StyleElement.java:1176)
 at net.sf.saxon.style.XSLStylesheet.preprocess(XSLStylesheet.java:643)
 at net.sf.saxon.PreparedStylesheet.setStylesheetDocument(PreparedStylesheet.java:279)
 at net.sf.saxon.PreparedStylesheet.prepare(PreparedStylesheet.java:117)
 at net.sf.saxon.TransformerFactoryImpl.newTemplates(TransformerFactoryImpl.java:130)
 at net.sf.saxon.TransformerFactoryImpl.newTransformer(TransformerFactoryImpl.java:85)
 at weblogic.xml.jaxp.RegistryTransformerFactory.newTransformer(RegistryTransformerFactory.java:209)
 at com.ecomm.xslt.XSLTObject.<init>(Unknown Source)
 at com.ecomm.xslt.XTPoolableObjectFactory.makeObject(Unknown Source)
 at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:747)
 at com.ecomm.services.ECSS.doTransformation(Unknown Source)
 at com.ecomm.services.ECSS.fetchScreenContent(Unknown Source)
 at com.ecomm.services.ECSSProxy.fetchScreenContent(Unknown Source)
 at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at weblogic.webservice.component.javaclass.JavaClassInvocationHandler.invoke(JavaClassInvocationHandler.java:134)
 at weblogic.webservice.core.handler.InvokeHandler.handleRequest(InvokeHandler.java:105)
 at weblogic.webservice.core.HandlerChainImpl.handleRequest(HandlerChainImpl.java:144)
 at weblogic.webservice.core.DefaultOperation.process(DefaultOperation.java:551)
 at weblogic.webservice.server.Dispatcher.process(Dispatcher.java:204)
 at weblogic.webservice.server.Dispatcher.doDispatch(Dispatcher.java:175)
 at weblogic.webservice.server.Dispatcher.dispatch(Dispatcher.java:97)
 at weblogic.webservice.server.WebServiceManager.dispatch(WebServiceManager.java:101)
 at weblogic.webservice.server.servlet.WebServiceServlet.serverSideInvoke(WebServiceServlet.java:321)
Any ideas as to what the problem could be?
                                                    
Paul Ackley
Staff Software Engineer
Qwest Communications