Found out a few more details.  We were not using the TransformerFactory class from Saxon, we were using one from WebLogic.  Now this transformer factory was creating Saxon transformers... but the factory class was WebLogic's not Saxon's.  Do you think this is okay?  Could this be causing issues?  This is all configured under the XML Registry service inside WebLogic, so it's easy to switch to Saxon's TransformerFactory class...
 
Paul


From: Ackley, Paul
Sent: Wednesday, February 25, 2009 7:51 PM
To: Mailing list for the SAXON XSLT and XQuery processor
Cc: Ackley, Paul
Subject: RE: [saxon] ArrayIndexOutOfBoundsException in Saxon 8.6

Well yes what you said makes sense.  As to why the stylesheet is still compiling after 2 or 3 days it has historical reasons. Back when this was written (5 or 6 years ago) we handled the caching of transformer objects ourselves.  We cache them for a set amount of time.  If they are not accessed after a certain amount of time they are disposed of.  Then when they are needed we load them and cache them again.
 
Just today I was reading the Saxon documentation for 8.6 and I was starting to get the idea that Saxon is also capable of caching the transformer object (templates?) for better performance.  So I was begining to think that it may have something to do with this problem.
 
Our framework is capable to using an XSLT API - is this caching part of the standard now?  Is there a way to Saxon (newer version is okay) which doesn't use this feature?  May not matter as I may look into just letting Saxon handle the caching...
 
BTW - I did hit the application hard with a load test to try to duplicate the problem in a shorter amount of time, but have not been able to cause this issue intentionally.  It's pretty elusive.  I will look at the statistics part of things too and see if there is some growth (in usage) over time....
 
Thanks for the quick response.
Paul Ackley
 
 

From: Michael Kay [mike@saxonica.com]
Sent: Wednesday, February 25, 2009 4:48 PM
To: 'Mailing list for the SAXON XSLT and XQuery processor'
Subject: Re: [saxon] ArrayIndexOutOfBoundsException in Saxon 8.6

I don't recognize this one - but 8.6 is way out of date, so that's not surprising. Is there anything that stops you moving to a more recent release? It's one that would be pretty tough to debug if it happened on the current release, but on a release that old, it's nigh impossible.
 
It's failing during compilation of a stylesheet, which is unusual. In fact, I'm slightly surprised to see a stylesheet being compiled two days into a continuously running service - I would expect everything to be cached.
 
A possibility is that the NamePool is filling and that it's not failing cleanly when it fills. I seem to remember older releases weren't checking too carefully for this condition. Also, older releases of Saxon tended to consume entries in the NamePool during stylesheet compilation by generating randomized variable names. There's no direct evidence of a NamePool problem in the stack trace, but it could have distant effects - and when problems occur in a shared long-running workload it's usually the NamePool that's to blame. You could try dumping the NamePool when it fails: you can get to it from the TransformerFactory via the Configuration, and it has methods diagnosticDump() and statistics() which might yield some insights. Calling statistics() every hour or so can also be useful to see if it's growing steadily.
 
Michael Kay
http://www.saxonica.com/


From: Ackley, Paul [mailto:Paul.Ackley@qwest.com]
Sent: 25 February 2009 20:59
To: saxon-help@lists.sourceforge.net
Subject: [saxon] ArrayIndexOutOfBoundsException in Saxon 8.6

We are experiencing an odd problem with Saxon inside of a WebLogic version 9.2 Container.  When the application starts up it is able to load the XSL file fine and the transformer object is able to transfor the XML just fine.  Eventually, usually after 2 or 3 days of continuous uptime with not too heavy usage Saxon starts to throw this exception when loading a style sheet which has previously worked.  Once it starts having this issue all stylesheets fail to load with the same message. 
 
Here is the exception:
com.qwest.ecomm.xslt.XTPoolableObjectFactory INFO  2009-02-24 09:17:36,961 - Creating new XT object, XSL=./xsl/nw/ViewEBill.xsl
com.ecomm.xslt.XSLTObject INFO  2009-02-24 09:17:36,961 - TransformerFactory hard class=class weblogic.xml.jaxp.RegistrySAXTransformerFactory
com.ecomm.xslt.ECSSURIResolver INFO  2009-02-24 09:17:36,965 - href=lib_Common.xsl
com.ecomm.xslt.ECSSURIResolver INFO  2009-02-24 09:17:36,965 - base=file:///prod/ecom2/local/apps/apps92/5.0.0.22/xsl/nw/ViewEBill.xsl
com.ecomm.xslt.ECSSURIResolver INFO  2009-02-24 09:17:36,966 - Location of included file took 1ms to find.
com.ecomm.xslt.ECSSURIResolver INFO  2009-02-24 09:17:36,969 - href=lib_CreateEncryptedUrl.xsl
com.ecomm.xslt.ECSSURIResolver INFO  2009-02-24 09:17:36,969 - base=file:/prod/ecom2/local/apps/apps92/5.0.0.22/xsl/nw/lib_Common.xsl
com.ecomm.xslt.ECSSURIResolver INFO  2009-02-24 09:17:36,971 - Location of included file took 2ms to find.
com.ecomm.xslt.ECSSURIResolver INFO  2009-02-24 09:17:36,972 - href=lib_FormatColumnTitles.xsl
com.ecomm.xslt.ECSSURIResolver INFO  2009-02-24 09:17:36,972 - base=file:/prod/ecom2/local/apps/apps92/5.0.0.22/xsl/nw/lib_Common.xsl
com.ecomm.xslt.ECSSURIResolver INFO  2009-02-24 09:17:36,974 - Location of included file took 2ms to find.
com.ecomm.xslt.XSLTObject ERROR 2009-02-24 09:17:36,983 - Caught generic exception while creating a transformer object.
java.lang.ArrayIndexOutOfBoundsException: 5
 at net.sf.saxon.om.NamespaceIterator.getInScopeNamespaceCodes(NamespaceIterator.java:170)
 at net.sf.saxon.tree.ElementWithAttributes.getInScopeNamespaceCodes(ElementWithAttributes.java:260)
 at net.sf.saxon.style.LiteralResultElement.validate(LiteralResultElement.java:198)
 at net.sf.saxon.style.StyleElement.validateSubtree(StyleElement.java:1167)
 at net.sf.saxon.style.StyleElement.validateChildren(StyleElement.java:1201)
 at net.sf.saxon.style.StyleElement.validateSubtree(StyleElement.java:1176)
 at net.sf.saxon.style.StyleElement.validateChildren(StyleElement.java:1201)
 at net.sf.saxon.style.StyleElement.validateSubtree(StyleElement.java:1176)
 at net.sf.saxon.style.XSLStylesheet.preprocess(XSLStylesheet.java:643)
 at net.sf.saxon.PreparedStylesheet.setStylesheetDocument(PreparedStylesheet.java:279)
 at net.sf.saxon.PreparedStylesheet.prepare(PreparedStylesheet.java:117)
 at net.sf.saxon.TransformerFactoryImpl.newTemplates(TransformerFactoryImpl.java:130)
 at net.sf.saxon.TransformerFactoryImpl.newTransformer(TransformerFactoryImpl.java:85)
 at weblogic.xml.jaxp.RegistryTransformerFactory.newTransformer(RegistryTransformerFactory.java:209)
 at com.ecomm.xslt.XSLTObject.<init>(Unknown Source)
 at com.ecomm.xslt.XTPoolableObjectFactory.makeObject(Unknown Source)
 at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:747)
 at com.ecomm.services.ECSS.doTransformation(Unknown Source)
 at com.ecomm.services.ECSS.fetchScreenContent(Unknown Source)
 at com.ecomm.services.ECSSProxy.fetchScreenContent(Unknown Source)
 at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at weblogic.webservice.component.javaclass.JavaClassInvocationHandler.invoke(JavaClassInvocationHandler.java:134)
 at weblogic.webservice.core.handler.InvokeHandler.handleRequest(InvokeHandler.java:105)
 at weblogic.webservice.core.HandlerChainImpl.handleRequest(HandlerChainImpl.java:144)
 at weblogic.webservice.core.DefaultOperation.process(DefaultOperation.java:551)
 at weblogic.webservice.server.Dispatcher.process(Dispatcher.java:204)
 at weblogic.webservice.server.Dispatcher.doDispatch(Dispatcher.java:175)
 at weblogic.webservice.server.Dispatcher.dispatch(Dispatcher.java:97)
 at weblogic.webservice.server.WebServiceManager.dispatch(WebServiceManager.java:101)
 at weblogic.webservice.server.servlet.WebServiceServlet.serverSideInvoke(WebServiceServlet.java:321)
Any ideas as to what the problem could be?
                                                    
Paul Ackley
Staff Software Engineer
Qwest Communications