Re: [Opaltoolkit-users] Problem reloading opal

[Anthony B. <ant...@ir...>]

Hi,
Now that I use the stop opal webapp, wait a minute and then restart it I 
no longer have this HSQLDB problem (although I still get this when stopping:
Exception in thread "HSQLDB Timer @4e8890da" java.lang.NullPointerException
    at org.hsqldb.lib.HsqlTimer.nextTask(Unknown Source)
    at org.hsqldb.lib.HsqlTimer$TaskRunner.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:595)
)

However, I still have a problem: services are usually not accessible 
after restarting. I get this kind of error message from clients:

Exception in thread "main" AxisFault
 faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
 faultSubcode:
 faultString: java.lang.reflect.InvocationTargetException
 faultActor:
 faultNode:
 faultDetail:
    {http://xml.apache.org/axis/}hostname:genoweb3

java.lang.reflect.InvocationTargetException
    at 
org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:221)
    at 
org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:128)
    at 
org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)
    at 
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:604)
    at 
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1750)
    at 
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2906)
    at 
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:624)
    at 
com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:116)
    at 
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:486)
    at 
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:810)
    at 
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:740)
    at 
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:110)
    at 
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1208)
    at 
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:525)
    at javax.xml.parsers.SAXParser.parse(SAXParser.java:392)
    at 
org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)
    at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)
    at org.apache.axis.Message.getSOAPEnvelope(Message.java:424)
    at 
org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62)
    at org.apache.axis.client.AxisClient.invoke(AxisClient.java:206)
    at org.apache.axis.client.Call.invokeEngine(Call.java:2765)
    at org.apache.axis.client.Call.invoke(Call.java:2748)
    at org.apache.axis.client.Call.invoke(Call.java:2424)
    at org.apache.axis.client.Call.invoke(Call.java:2347)
    at org.apache.axis.client.Call.invoke(Call.java:1804)
    at 
edu.sdsc.nbcr.opal.AppServicePortTypeSoapBindingStub.launchJob(AppServicePortTypeSoapBindingStub.java:551)
    at 
CustomOpalClient.CustomOpalClient.launchJob(CustomOpalClient.java:397)
    at Test.main(Test.java:49)

It seems like a server problem, maybe with axis? Is this a known 
problem? Is there a solution (apart from restarting tomcat server)?
Thanks!

Luca Clementi a écrit :
> Anthony Bretaudeau wrote:
>> Hi,
>> I've got a problem when reloading opal2 app from tomcat manager. After
>> doing so, registered webservices stop working and the only solution is
>> to restart tomcat (which is quite uncomfortable as we have other apps in
>> production on the same tomcat server).
>> Looking at the log, I see some db related errors when reloading: see 
>> below.
>> Can you help me?
>> Thanks!
>
> Anthony,
> we already saw this problem.
> It is caused by a a race condition of HSQL during the reload of the 
> webapp, basically HSQL is restarted before the old HSQL is completely 
> destroyed. It doesn't always happen, sometime the reload works well.
>
> If you use external database this problem won't appear.
>
> My suggestion is to stop the webapp, wait some seconds and then start 
> the webapp again, this should avoid this race condition.
>
> Sincerely,
> Luca
>
>
>>
>> Jun 18, 2009 3:18:48 PM org.apache.catalina.core.StandardContext reload
>> INFO: Reloading this Context has started
>> Exception in thread "HSQLDB Timer @5d802746" 
>> java.lang.NullPointerException
>>     at org.hsqldb.lib.HsqlTimer.nextTask(Unknown Source)
>>     at org.hsqldb.lib.HsqlTimer$TaskRunner.run(Unknown Source)
>>     at java.lang.Thread.run(Thread.java:595)
>> [http-8383-Processor19] ERROR hbm2ddl.SchemaUpdate  - could not get
>> database metadata
>> java.sql.SQLException: The database is already in use by another
>> process: org.hsqldb.NIOLockFile@f6fbb688[file
>> =/index/www-tmp/opal/opaldb.lck, exists=true, locked=false, valid=false,
>> fl =null]: java.lang.Exception: checkHeartbeat(): lock file
>> [/index/www-tmp/opal/opaldb.lck] is presumably locked by another 
>> process.
>>     at org.hsqldb.jdbc.jdbcUtil.sqlException(Unknown Source)
>>     at org.hsqldb.jdbc.jdbcConnection.<init>(Unknown Source)
>>     at org.hsqldb.jdbcDriver.getConnection(Unknown Source)
>>     at org.hsqldb.jdbcDriver.connect(Unknown Source)
>>     at java.sql.DriverManager.getConnection(DriverManager.java:525)
>>     at java.sql.DriverManager.getConnection(DriverManager.java:140)
>>     at
>> org.hibernate.connection.DriverManagerConnectionProvider.getConnection(DriverManagerConnectionProvider.java:133) 
>>
>>     at
>> org.hibernate.tool.hbm2ddl.SuppliedConnectionProviderConnectionHelper.prepare(SuppliedConnectionProviderConnectionHelper.java:51) 
>>
>>     at
>> org.hibernate.tool.hbm2ddl.SchemaUpdate.execute(SchemaUpdate.java:168)
>>     at
>> org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:346) 
>>
>>     at
>> org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1327) 
>>
>>     at
>> edu.sdsc.nbcr.opal.state.HibernateUtil.<clinit>(HibernateUtil.java:48)
>>     at
>> edu.sdsc.nbcr.opal.dashboard.persistence.DBManager.<init>(DBManager.java:79) 
>>
>>     at
>> edu.sdsc.nbcr.opal.dashboard.servlet.OpalInfoServlet.init(OpalInfoServlet.java:103) 
>>
>>     at
>> org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1139) 
>>
>>     at
>> org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:966)
>>     at
>> org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:3956) 
>>
>>     at
>> org.apache.catalina.core.StandardContext.start(StandardContext.java:4230) 
>>
>>     at
>> org.apache.catalina.core.StandardContext.reload(StandardContext.java:3025) 
>>
>>     at
>> org.apache.catalina.manager.ManagerServlet.reload(ManagerServlet.java:904) 
>>
>>     at
>> org.apache.catalina.manager.HTMLManagerServlet.reload(HTMLManagerServlet.java:495) 
>>
>>     at
>> org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:99) 
>>
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>     at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) 
>>
>>     at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) 
>>
>>     at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) 
>>
>>     at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) 
>>
>>     at
>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525) 
>>
>>     at
>> org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:276) 
>>
>>     at
>> org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81) 
>>
>>     at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) 
>>
>>     at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) 
>>
>>     at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) 
>>
>>     at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) 
>>
>>     at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875) 
>>
>>     at
>> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665) 
>>
>>     at
>> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528) 
>>
>>     at
>> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81) 
>>
>>     at
>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) 
>>
>>     at java.lang.Thread.run(Thread.java:595)
>> [http-8383-Processor19] ERROR hbm2ddl.SchemaUpdate  - could not complete
>> schema update
>> java.sql.SQLException: The database is already in use by another
>> process: org.hsqldb.NIOLockFile@f6fbb688[file
>> =/index/www-tmp/opal/opaldb.lck, exists=true, locked=false, valid=false,
>> fl =null]: java.lang.Exception: checkHeartbeat(): lock file
>> [/index/www-tmp/opal/opaldb.lck] is presumably locked by another 
>> process.
>>     at org.hsqldb.jdbc.jdbcUtil.sqlException(Unknown Source)
>>     at org.hsqldb.jdbc.jdbcConnection.<init>(Unknown Source)
>>     at org.hsqldb.jdbcDriver.getConnection(Unknown Source)
>>     at org.hsqldb.jdbcDriver.connect(Unknown Source)
>>     at java.sql.DriverManager.getConnection(DriverManager.java:525)
>>     at java.sql.DriverManager.getConnection(DriverManager.java:140)
>>     at
>> org.hibernate.connection.DriverManagerConnectionProvider.getConnection(DriverManagerConnectionProvider.java:133) 
>>
>>     at
>> org.hibernate.tool.hbm2ddl.SuppliedConnectionProviderConnectionHelper.prepare(SuppliedConnectionProviderConnectionHelper.java:51) 
>>
>>     at
>> org.hibernate.tool.hbm2ddl.SchemaUpdate.execute(SchemaUpdate.java:168)
>>     at
>> org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:346) 
>>
>>     at
>> org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1327) 
>>
>>     at
>> edu.sdsc.nbcr.opal.state.HibernateUtil.<clinit>(HibernateUtil.java:48)
>>     at
>> edu.sdsc.nbcr.opal.dashboard.persistence.DBManager.<init>(DBManager.java:79) 
>>
>>     at
>> edu.sdsc.nbcr.opal.dashboard.servlet.OpalInfoServlet.init(OpalInfoServlet.java:103) 
>>
>>     at
>> org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1139) 
>>
>>     at
>> org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:966)
>>     at
>> org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:3956) 
>>
>>     at
>> org.apache.catalina.core.StandardContext.start(StandardContext.java:4230) 
>>
>>     at
>> org.apache.catalina.core.StandardContext.reload(StandardContext.java:3025) 
>>
>>     at
>> org.apache.catalina.manager.ManagerServlet.reload(ManagerServlet.java:904) 
>>
>>     at
>> org.apache.catalina.manager.HTMLManagerServlet.reload(HTMLManagerServlet.java:495) 
>>
>>     at
>> org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:99) 
>>
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>     at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) 
>>
>>     at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) 
>>
>>     at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) 
>>
>>     at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) 
>>
>>     at
>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525) 
>>
>>     at
>> org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:276) 
>>
>>     at
>> org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81) 
>>
>>     at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) 
>>
>>     at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) 
>>
>>     at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) 
>>
>>     at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) 
>>
>>     at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875) 
>>
>>     at
>> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665) 
>>
>>     at
>> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528) 
>>
>>     at
>> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81) 
>>
>>     at
>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) 
>>
>>     at java.lang.Thread.run(Thread.java:595)
>> [http-8383-Processor19] ERROR util.JDBCExceptionReporter  - The database
>> is already in use by another process:
>> org.hsqldb.NIOLockFile@f6fbb688[file =/index/www-tmp/opal/opaldb.lck,
>> exists=true, locked=false, valid=false, fl =null]: java.lang.Exception:
>> checkHeartbeat(): lock file [/index/www-tmp/opal/opaldb.lck] is
>> presumably locked by another process.
>> org.hibernate.exception.JDBCConnectionException: Cannot open connection
>>     at
>> org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:97) 
>>
>>     at
>> org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66) 
>>
>>     at
>> org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52) 
>>
>>     at
>> org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449) 
>>
>>     at
>> org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167) 
>>
>>     at
>> org.hibernate.jdbc.BorrowedConnectionProxy.invoke(BorrowedConnectionProxy.java:74) 
>>
>>     at $Proxy3.getMetaData(Unknown Source)
>>     at
>> edu.sdsc.nbcr.opal.dashboard.persistence.DBManager.<init>(DBManager.java:85) 
>>
>>     at
>> edu.sdsc.nbcr.opal.dashboard.servlet.OpalInfoServlet.init(OpalInfoServlet.java:103) 
>>
>>     at
>> org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1139) 
>>
>>     at
>> org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:966)
>>     at
>> org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:3956) 
>>
>>     at
>> org.apache.catalina.core.StandardContext.start(StandardContext.java:4230) 
>>
>>     at
>> org.apache.catalina.core.StandardContext.reload(StandardContext.java:3025) 
>>
>>     at
>> org.apache.catalina.manager.ManagerServlet.reload(ManagerServlet.java:904) 
>>
>>     at
>> org.apache.catalina.manager.HTMLManagerServlet.reload(HTMLManagerServlet.java:495) 
>>
>>     at
>> org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:99) 
>>
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>     at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) 
>>
>>     at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) 
>>
>>     at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) 
>>
>>     at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) 
>>
>>     at
>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525) 
>>
>>     at
>> org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:276) 
>>
>>     at
>> org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81) 
>>
>>     at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) 
>>
>>     at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) 
>>
>>     at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) 
>>
>>     at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) 
>>
>>     at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875) 
>>
>>     at
>> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665) 
>>
>>     at
>> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528) 
>>
>>     at
>> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81) 
>>
>>     at
>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) 
>>
>>     at java.lang.Thread.run(Thread.java:595)
>> Caused by: java.sql.SQLException: The database is already in use by
>> another process: org.hsqldb.NIOLockFile@f6fbb688[file
>> =/index/www-tmp/opal/opaldb.lck, exists=true, locked=false, valid=false,
>> fl =null]: java.lang.Exception: checkHeartbeat(): lock file
>> [/index/www-tmp/opal/opaldb.lck] is presumably locked by another 
>> process.
>>     at org.hsqldb.jdbc.jdbcUtil.sqlException(Unknown Source)
>>     at org.hsqldb.jdbc.jdbcConnection.<init>(Unknown Source)
>>     at org.hsqldb.jdbcDriver.getConnection(Unknown Source)
>>     at org.hsqldb.jdbcDriver.connect(Unknown Source)
>>     at java.sql.DriverManager.getConnection(DriverManager.java:525)
>>     at java.sql.DriverManager.getConnection(DriverManager.java:140)
>>     at
>> org.hibernate.connection.DriverManagerConnectionProvider.getConnection(DriverManagerConnectionProvider.java:133) 
>>
>>     at
>> org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446) 
>>
>>     ... 32 more
>>
>> ------------------------------------------------------------------------------ 
>>
>> Crystal Reports - New Free Runtime and 30 Day Trial
>> Check out the new simplified licensing option that enables unlimited
>> royalty-free distribution of the report engine for externally facing
>> server and web deployment.
>> http://p.sf.net/sfu/businessobjects
>> _______________________________________________
>> Opaltoolkit-users mailing list
>> Opa...@li...
>> https://lists.sourceforge.net/lists/listinfo/opaltoolkit-users
>