Phew!
 
The diagnostics show that at the time the crash occurs, the contents of the IntHashSet are indeed very strange. It appears to contain two "real" values (65537 and 131074), 15 unused slots (shown as "*"), and 15 slots containing the value -1. The size=2 is consistent with the number of "real" values, while the iterator is going to return all the -1 values as well, so the code in NamespaceIterator will allocate an array of size 2 and then try to put 17 values into it, which accounts for the AIOOB exception.
 
So, how did these -1 values get there, and what is the significance of the fact that they occupy alternate slots?
 
My suspicion would fall on the instruction used to initialize the array right at the start:
 
java.util.Arrays.fill(_values, ndv);
 
where _values is the array of 32 slots, and ndv is Integer.MIN_VALUE. It looks to me as if the implementation of this call might be filling the array 8 bytes (64 bits) at a time, and might be writing the value FFFFFFFF80000000 to these 64 bits rather than 8000000080000000.
 
Hopefully that will give the people who support the JVM something more concrete to look for; I don't think I can go any further than that at the Saxon level.
 
As a trial workaround, we could try using a value other than MIN_VALUE to represent unoccupied slots. For namespace codes, any negative number would do, and the obvious choice is -1. It's tricky though when it takes months for the bug to reveal itself!
 

Regards,

Michael Kay
http://www.saxonica.com/
http://twitter.com/michaelhkay



From: Ackley, Paul [mailto:Paul.Ackley@qwest.com]
Sent: 28 May 2009 17:34
To: Ackley, Paul; Mailing list for the SAXON XSLTand XQuery processor
Subject: Re: [saxon] ArrayIndexOutOfBoundsException in Saxon 8.6

Michael,
 
After a couple months of testing the new debugging build of Saxon 9.1.0.6 this error has finally occurred.  The debugging information you put in the code printed the following to our logs:
 
186479642 [[ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'] INFO com.qwest.ecomm.xslt.XSLTObject  -
 TransformerFactory implementing class=class weblogic.xml.jaxp.RegistrySAXTransformerFactory
*** Internal error in NamespaceIterator ***
Contents of IntHashSet
 
*, *, 65537, -1, 131074, -1, *, -1, *, -1,
*, -1, *, -1, *, -1, *, -1, *, -1,
*, -1, *, -1, *, -1, *, -1, *, -1,
*, -1,
size: 2
ndv: -2147483648
nlo: 8
nhi: 268435456
nmax: 32
shift: 26
mask: 31
Result of iterator:
 
65537, -1, 131074, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1,
=====================
186479643 [[ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR com.qwest.ecomm.xslt.XSLTObject
- Caught generic exception while creating a transformer object.
java.lang.ArrayIndexOutOfBoundsException: 2
        at net.sf.saxon.om.NamespaceIterator.getInScopeNamespaceCodes(NamespaceIterator.java:289)
        at net.sf.saxon.tree.ElementImpl.getInScopeNamespaceCodes(ElementImpl.java:674)
        at net.sf.saxon.style.LiteralResultElement.validate(LiteralResultElement.java:200)
        at net.sf.saxon.style.StyleElement.validateSubtree(StyleElement.java:1234)
        at net.sf.saxon.style.StyleElement.validateChildren(StyleElement.java:1263)
        at net.sf.saxon.style.StyleElement.validateSubtree(StyleElement.java:1238)
        at net.sf.saxon.style.XSLStylesheet.preprocess(XSLStylesheet.java:705)
        at net.sf.saxon.PreparedStylesheet.setStylesheetDocument(PreparedStylesheet.java:331)
        at net.sf.saxon.PreparedStylesheet.prepare(PreparedStylesheet.java:163)
        at net.sf.saxon.TransformerFactoryImpl.newTemplates(TransformerFactoryImpl.java:139)
        at weblogic.xml.jaxp.RegistryTransformerFactory.newTemplates(RegistryTransformerFactory.java:173)
        at com.qwest.ecomm.xslt.XSLTObject.getTemplates(XSLTObject.java:217)
        at com.qwest.ecomm.xslt.XSLTObject.<init>(XSLTObject.java:170)
        at com.qwest.ecomm.services.ECommScreenService.doTransformation(ECommScreenService.java:186)
        at com.qwest.ecomm.services.ECommScreenService.fetchScreenContent(ECommScreenService.java:254)
        at com.qwest.ecomm.services.ECommScreenServiceProxy.fetchScreenContent(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at weblogic.webservice.component.javaclass.JavaClassInvocationHandler.invoke(JavaClassInvocationHandler.java:134)
        at weblogic.webservice.core.handler.InvokeHandler.handleRequest(InvokeHandler.java:105)
        at weblogic.webservice.core.HandlerChainImpl.handleRequest(HandlerChainImpl.java:144)
        at weblogic.webservice.core.DefaultOperation.process(DefaultOperation.java:551)
        at weblogic.webservice.server.Dispatcher.process(Dispatcher.java:204)
        at weblogic.webservice.server.Dispatcher.doDispatch(Dispatcher.java:175)
        at weblogic.webservice.server.Dispatcher.dispatch(Dispatcher.java:97)
        at weblogic.webservice.server.WebServiceManager.dispatch(WebServiceManager.java:101)

Does this give you any indication of what is going wrong?
 
Thanks!
Paul Ackley
 
 


From: Ackley, Paul
Sent: Thursday, March 19, 2009 4:16 PM
To: Ackley, Paul; Mailing list for the SAXON XSLT and XQuery processor
Subject: RE: [saxon] ArrayIndexOutOfBoundsException in Saxon 8.6

Here is an interesting article on how the JRockit JVM (maybe Sun's as well?) does optimize the code on the fly:
 
http://download.oracle.com/docs/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/underst_jit.html#wp1076474
 
The JVM first runs JIT compilation, so that the application can get running.  JRockit then monitor's threads and samples what could benefit from being optimized.  I guess my hypothesis is that the initial JIT compilation of the loop in NamespaceIterator.getInScopeNamespaceCodes() runs okay, but then after time the JVM decides to optimize this loop even more and possibly that's when we experience this issue?
 
Just a thought.
 
Thanks!
Paul
 
 


From: Ackley, Paul
Sent: Thursday, March 19, 2009 4:02 PM
To: 'Mailing list for the SAXON XSLT and XQuery processor'
Subject: RE: [saxon] ArrayIndexOutOfBoundsException in Saxon 8.6

Michael,
 
The 9.1.0.6 build of Saxon has been installed and the application has been running for 3 days without running into this "ArrayIndexOutOfBoundsException" exception.  I'll let you know if it happens and what the debugging output looks like.  Typically, by now the problem has occurred.  I'm wondering if the JVM does some optimization of the loop which iterates through the hash and if putting this try/catch in this code possibly caused the JVM to not do the optimization - but really this is just a very weak hypothesis on my part.  Just because it hasn't happened yet doesn't mean it wont.  Time will tell.
 
I'll keep you updated.
 
Thanks for all your help thus far.
 
Paul Ackley