Stale Connections with NamedPipe

[Thomas Yates]

Hello!

I have a jTDS (JCIFS?) issue that is killing me.

Environment:

DataSource in WebSphere 5.1 (jdk 1.4.1), and I am using NamedPipes.

Let me say right off the bat that the whole jTDS/JCIFS thing works fine except for this intermittent issue,and for that I am very grateful - THANKS!!

Here is the intermittent issue:

I set up a DataSource in WebSphere, set min pool size to 1 and max pool size to 5.

In a given thread of execution (i.e. a given user executing one method), I get a connection, close it, get another connection, close it etc 30+ times.
No Problem - connections are being returned to the pool and reused as expected.

However, in the next thread of execution, for the same user, when I try to get/use even 1 connection from the pool I get the stack included below.

WEIRDER STILL, if I then click the back button on the browser and kick off the same thread of
execution that just failed, everything works fine.

I suspect the root cause is JCIFS related (see root cause below). Any ideas? I updated my JCIFS to the latest (1.1.7), but that didnt fix the issue.

[2/4/05 11:02:39:713 CST] 5a97a2 WSRdbManagedC W DSRA0080E: An exception was received by the Data Store Adapter. See original exception message: java.sql.SQLException: Invalid state, the Connection object is closed.
at net.sourceforge.jtds.jdbc.ConnectionJDBC2.checkOpen(ConnectionJDBC2.java:1134)
at net.sourceforge.jtds.jdbc.ConnectionJDBC2.getCatalog(ConnectionJDBC2.java:1560)
at net.sourceforge.jtds.jdbcx.proxy.ConnectionProxy.getCatalog(ConnectionProxy.java:175)
at com.ibm.ws.rsadapter.spi.WSRdbManagedConnectionImpl.getCatalog(WSRdbManagedConnectionImpl.java:620)
at com.ibm.ws.rsadapter.spi.WSRdbManagedConnectionImpl.introspectSelf(WSRdbManagedConnectionImpl.java:740)
at com.ibm.ws.rsadapter.FFDCLogger.introspect(FFDCLogger.java:169)
at com.ibm.ws.rsadapter.jdbc.WSJdbcConnection.introspectSelf(WSJdbcConnection.java:1190)
at com.ibm.ws.rsadapter.jdbc.WSJdbcObject.introspectSelf(WSJdbcObject.java:355)
at com.ibm.ws.ffdc.IntrospectionLevelMember.getNextMembers(IntrospectionLevelMember.java:440)
at com.ibm.ws.ffdc.IntrospectionLevel.getNextLevel(IntrospectionLevel.java:181)
at com.ibm.ws.ffdc.ObjectIntrospectorImpl.dumpContents(ObjectIntrospectorImpl.java:67)
at com.ibm.ws.ffdc.ObjectIntrospectorImpl.dumpContents(ObjectIntrospectorImpl.java:51)
at com.ibm.ws.ffdc.IncidentStreamImpl.introspectAndWrite(IncidentStreamImpl.java:396)
at com.ibm.ws.ffdc.IncidentStreamImpl.introspectAndWriteLine(IncidentStreamImpl.java:632)
at com.ibm.ws.ffdc.DiagnosticEngine.dumpObjectAndStack(DiagnosticEngine.java:301)
at com.ibm.ws.ffdc.DiagnosticEngine.processIncident(DiagnosticEngine.java:147)
at com.ibm.ws.ffdc.FFDCFilter.filterEngine(FFDCFilter.java(Compiled Code))
at com.ibm.ws.ffdc.FFDCFilter.processException(FFDCFilter.java:192)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.execute(WSJdbcPreparedStatement.java:394)
at com.springs.costestimating.dao.ProjectDAO.copyProject(ProjectDAO.java:843)
at com.springs.costestimating.handler.ProjectHandler.copyProjects(ProjectHandler.java:493)
at com.springs.costestimating.handler.ProjectHandler.processConfirmAction(ProjectHandler.java:259)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled Code))
at java.lang.reflect.Method.invoke(Method.java(Compiled Code))
at com.sun.faces.el.MethodBindingImpl.invoke(MethodBindingImpl.java:126)
at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:72)
at javax.faces.component.UICommand.broadcast(UICommand.java:312)
at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:267)
at javax.faces.component.UIViewRoot.processDecodes(UIViewRoot.java:307)
at com.sun.faces.lifecycle.ApplyRequestValuesPhase.execute(ApplyRequestValuesPhase.java:79)
at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:200)
at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:90)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:197)
at com.ibm.ws.webcontainer.servlet.StrictServletInstance.doService(StrictServletInstance.java:110)
at com.ibm.ws.webcontainer.servlet.StrictLifecycleServlet._service(StrictLifecycleServlet.java:174)
at com.ibm.ws.webcontainer.servlet.IdleServletState.service(StrictLifecycleServlet.java:313)
at com.ibm.ws.webcontainer.servlet.StrictLifecycleServlet.service(StrictLifecycleServlet.java:116)
at com.ibm.ws.webcontainer.servlet.ServletInstance.service(ServletInstance.java:283)
at com.ibm.ws.webcontainer.servlet.ValidServletReferenceState.dispatch(ValidServletReferenceState.java:42)
at com.ibm.ws.webcontainer.servlet.ServletInstanceReference.dispatch(ServletInstanceReference.java:40)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:76)
at com.springs.utilities.authorization.AuthorizationFilter.doFilter(AuthorizationFilter.java:62)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:132)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:71)
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.handleWebAppDispatch(WebAppRequestDispatcher.java:974)
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.dispatch(WebAppRequestDispatcher.java:564)
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java:200)
at com.ibm.ws.webcontainer.srt.WebAppInvoker.doForward(WebAppInvoker.java:119)
at com.ibm.ws.webcontainer.srt.WebAppInvoker.handleInvocationHook(WebAppInvoker.java:276)
at com.ibm.ws.webcontainer.cache.invocation.CachedInvocation.handleInvocation(CachedInvocation.java:71)
at com.ibm.ws.webcontainer.cache.invocation.CacheableInvocationContext.invoke(CacheableInvocationContext.java:116)
at com.ibm.ws.webcontainer.srp.ServletRequestProcessor.dispatchByURI(ServletRequestProcessor.java:186)
at com.ibm.ws.webcontainer.oselistener.OSEListenerDispatcher.service(OSEListener.java:334)
at com.ibm.ws.webcontainer.http.HttpConnection.handleRequest(HttpConnection.java:56)
at com.ibm.ws.http.HttpConnection.readAndHandleRequest(HttpConnection.java:618)
at com.ibm.ws.http.HttpConnection.run(HttpConnection.java:443)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:672)

[2/4/05 11:02:40:023 CST] 5a97a2 StaleConnecti A CONM7007I: Mapping the following SQLException, with ErrorCode 0 and SQLState 08S01, to a StaleConnectionException: java.sql.SQLException: I/O Error: The handle is invalid.
at net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:859)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:362)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.execute(JtdsPreparedStatement.java:431)
at net.sourceforge.jtds.jdbcx.proxy.PreparedStatementProxy.execute(PreparedStatementProxy.java:451)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.execute(WSJdbcPreparedStatement.java:386)
at com.springs.costestimating.dao.ProjectDAO.copyProject(ProjectDAO.java:843)
at com.springs.costestimating.handler.ProjectHandler.copyProjects(ProjectHandler.java:493)
at com.springs.costestimating.handler.ProjectHandler.processConfirmAction(ProjectHandler.java:259)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled Code))
at java.lang.reflect.Method.invoke(Method.java(Compiled Code))
at com.sun.faces.el.MethodBindingImpl.invoke(MethodBindingImpl.java:126)
at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:72)
at javax.faces.component.UICommand.broadcast(UICommand.java:312)
at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:267)
at javax.faces.component.UIViewRoot.processDecodes(UIViewRoot.java:307)
at com.sun.faces.lifecycle.ApplyRequestValuesPhase.execute(ApplyRequestValuesPhase.java:79)
at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:200)
at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:90)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:197)
at com.ibm.ws.webcontainer.servlet.StrictServletInstance.doService(StrictServletInstance.java:110)
at com.ibm.ws.webcontainer.servlet.StrictLifecycleServlet._service(StrictLifecycleServlet.java:174)
at com.ibm.ws.webcontainer.servlet.IdleServletState.service(StrictLifecycleServlet.java:313)
at com.ibm.ws.webcontainer.servlet.StrictLifecycleServlet.service(StrictLifecycleServlet.java:116)
at com.ibm.ws.webcontainer.servlet.ServletInstance.service(ServletInstance.java:283)
at com.ibm.ws.webcontainer.servlet.ValidServletReferenceState.dispatch(ValidServletReferenceState.java:42)
at com.ibm.ws.webcontainer.servlet.ServletInstanceReference.dispatch(ServletInstanceReference.java:40)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:76)
at com.springs.utilities.authorization.AuthorizationFilter.doFilter(AuthorizationFilter.java:62)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:132)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:71)
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.handleWebAppDispatch(WebAppRequestDispatcher.java:974)
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.dispatch(WebAppRequestDispatcher.java:564)
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java:200)
at com.ibm.ws.webcontainer.srt.WebAppInvoker.doForward(WebAppInvoker.java:119)
at com.ibm.ws.webcontainer.srt.WebAppInvoker.handleInvocationHook(WebAppInvoker.java:276)
at com.ibm.ws.webcontainer.cache.invocation.CachedInvocation.handleInvocation(CachedInvocation.java:71)
at com.ibm.ws.webcontainer.cache.invocation.CacheableInvocationContext.invoke(CacheableInvocationContext.java:116)
at com.ibm.ws.webcontainer.srp.ServletRequestProcessor.dispatchByURI(ServletRequestProcessor.java:186)
at com.ibm.ws.webcontainer.oselistener.OSEListenerDispatcher.service(OSEListener.java:334)
at com.ibm.ws.webcontainer.http.HttpConnection.handleRequest(HttpConnection.java:56)
at com.ibm.ws.http.HttpConnection.readAndHandleRequest(HttpConnection.java:618)
at com.ibm.ws.http.HttpConnection.run(HttpConnection.java:443)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:672)
Caused by: jcifs.smb.SmbException: The handle is invalid.
at jcifs.smb.SmbTransport.send(SmbTransport.java:704)
at jcifs.smb.SmbSession.send(SmbSession.java:234)
at jcifs.smb.SmbTree.send(SmbTree.java:103)
at jcifs.smb.SmbFile.send(SmbFile.java:724)
at jcifs.smb.SmbFileOutputStream.write(SmbFileOutputStream.java:220)
at java.io.DataOutputStream.write(DataOutputStream.java:102)
at net.sourceforge.jtds.jdbc.SharedSocket.sendNetPacket(SharedSocket.java:561)
at net.sourceforge.jtds.jdbc.RequestStream.putPacket(RequestStream.java:558)
at net.sourceforge.jtds.jdbc.RequestStream.flush(RequestStream.java:504)
at net.sourceforge.jtds.jdbc.TdsCore.executeSQL70(TdsCore.java:3243)
at net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:847)
... 44 more

[2/4/05 11:02:40:073 CST] 5a97a2 ConnectionEve A J2CA0056I: The Connection Manager received a fatal connection error from the Resource Adaptor for resource jdbc/jtds-wfmddt2-springstransactions. The exception which was received is com.ibm.websphere.ce.cm.StaleConnectionException: I/O Error: The handle is invalid.
ERROR - [11:02:40,124] com.springs.utilities.exception.DAOExceptionHelper: CAUSE:: SQLException SQL State = 08S01 STACK:: com.ibm.websphere.ce.cm.StaleConnectionException: I/O Error: The handle is invalid.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:79)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java(Compiled Code))
at java.lang.reflect.Constructor.newInstance(Constructor.java(Compiled Code))
at com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapExceptionHelper(GenericDataStoreHelper.java:473)
at com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapException(GenericDataStoreHelper.java:516)
at com.ibm.ws.rsadapter.jdbc.WSJdbcUtil.mapException(WSJdbcUtil.java:824)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.execute(WSJdbcPreparedStatement.java:396)
at com.springs.costestimating.dao.ProjectDAO.copyProject(ProjectDAO.java:843)
at com.springs.costestimating.handler.ProjectHandler.copyProjects(ProjectHandler.java:493)
at com.springs.costestimating.handler.ProjectHandler.processConfirmAction(ProjectHandler.java:259)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled Code))
at java.lang.reflect.Method.invoke(Method.java(Compiled Code))
at com.sun.faces.el.MethodBindingImpl.invoke(MethodBindingImpl.java:126)
at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:72)
at javax.faces.component.UICommand.broadcast(UICommand.java:312)
at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:267)
at javax.faces.component.UIViewRoot.processDecodes(UIViewRoot.java:307)
at com.sun.faces.lifecycle.ApplyRequestValuesPhase.execute(ApplyRequestValuesPhase.java:79)
at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:200)
at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:90)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:197)
at com.ibm.ws.webcontainer.servlet.StrictServletInstance.doService(StrictServletInstance.java:110)
at com.ibm.ws.webcontainer.servlet.StrictLifecycleServlet._service(StrictLifecycleServlet.java:174)
at com.ibm.ws.webcontainer.servlet.IdleServletState.service(StrictLifecycleServlet.java:313)
at com.ibm.ws.webcontainer.servlet.StrictLifecycleServlet.service(StrictLifecycleServlet.java:116)
at com.ibm.ws.webcontainer.servlet.ServletInstance.service(ServletInstance.java:283)
at com.ibm.ws.webcontainer.servlet.ValidServletReferenceState.dispatch(ValidServletReferenceState.java:42)
at com.ibm.ws.webcontainer.servlet.ServletInstanceReference.dispatch(ServletInstanceReference.java:40)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:76)
at com.springs.utilities.authorization.AuthorizationFilter.doFilter(AuthorizationFilter.java:62)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:132)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:71)
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.handleWebAppDispatch(WebAppRequestDispatcher.java:974)
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.dispatch(WebAppRequestDispatcher.java:564)
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java:200)
at com.ibm.ws.webcontainer.srt.WebAppInvoker.doForward(WebAppInvoker.java:119)
at com.ibm.ws.webcontainer.srt.WebAppInvoker.handleInvocationHook(WebAppInvoker.java:276)
at com.ibm.ws.webcontainer.cache.invocation.CachedInvocation.handleInvocation(CachedInvocation.java:71)
at com.ibm.ws.webcontainer.cache.invocation.CacheableInvocationContext.invoke(CacheableInvocationContext.java:116)
at com.ibm.ws.webcontainer.srp.ServletRequestProcessor.dispatchByURI(ServletRequestProcessor.java:186)
at com.ibm.ws.webcontainer.oselistener.OSEListenerDispatcher.service(OSEListener.java:334)
at com.ibm.ws.webcontainer.http.HttpConnection.handleRequest(HttpConnection.java:56)
at com.ibm.ws.http.HttpConnection.readAndHandleRequest(HttpConnection.java:618)
at com.ibm.ws.http.HttpConnection.run(HttpConnection.java:443)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:672)
.

[Alin Sinpalean]

Thomas,

The stack traces are pretty hard to understand. From what I can tell, the second stack trace should actually be the first (compare the line numbers: WSJdbcPreparedStatement.java:386 in the second vs. WSJdbcPreparedStatement.java:396 in the first).

Considering this, the first exception that occurs is "jcifs.smb.SmbException: The handle is invalid". Now I'm not really sure what this means. It sure seems like the named pipe connection has failed, but I guess asking the JCIFS cuys about this (exclusively about the second stack trace, I don't hink they are interested in the other ones) could be a good idea.

Anyway, the thread stuff seems interesting. I would say the first thread does something wrong causing the named pipe connection to fail, then the second thread tries to use it and gets this exception. When reloading the page you no longer get the exception because the pool implementation realizes the connection has failed (see the StaleConnectionException), drops it and creates a new one.

And as a side note, named pipes over JCIFS are by far the slowest network support for jTDS. The fastest is local named pipes (but these only work when jTDS is running on the same machine as SQL Server) and close in second are TCP/IP connections. JCIFS uses named pipes over TCP/IP, so it has a pretty big overhead.

You might take a look at the SQL Server logs, maybe you can find an error there. Or in your own logs, see if the first thread logs any error message (but I guess you already did that).

Alin.

[Thomas Yates]

alin-

yea, I know named pipes over jCIFS to a remote db is s-l-o-w....but its a customer requirement :(

thanks for taking the time to dig thru my stack....you validated to some degree what i suspected (jCIFS issue due to flaky network).

thanks again!
tom