#78 Tomcat hangs on shutdown on Linux with GSearch

General (6)
Bill Branan

When running Fedora on Linux with messaging and GSearch enabled and ActiveMQ running as an embedded service (the default configuration) the Tomcat process will hang on shutdown. The same error occurs on Windows, but the tomcat process does not hang.

The error below is seen in the tomcat log right after the shutdown script is executed:

INFO: Stopping service Catalina
log4j:ERROR LogMananger.repositorySelector was null likely due to error in class reloading, using NOPLoggerRepository.
ERROR 2008-07-23 17:52:12,022 ([/fedoragsearch]) Servlet UpdateListener threw unload() exception
javax.servlet.ServletException: Servlet.destroy() for servlet UpdateListener threw exception
at org.apache.catalina.core.StandardWrapper.unload(StandardWrapper.java:1373)
at org.apache.catalina.core.StandardWrapper.stop(StandardWrapper.java:1688)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4350)
at org.apache.catalina.core.ContainerBase.removeChild(ContainerBase.java:893)
at org.apache.catalina.startup.HostConfig.undeployApps(HostConfig.java:1180)
at org.apache.catalina.startup.HostConfig.stop(HostConfig.java:1151)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:313)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:120)
at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1055)
at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1067)
at org.apache.catalina.core.StandardEngine.stop(StandardEngine.java:448)
at org.apache.catalina.core.StandardService.stop(StandardService.java:510)
at org.apache.catalina.core.StandardServer.stop(StandardServer.java:734)
at org.apache.catalina.startup.Catalina.stop(Catalina.java:602)
at org.apache.catalina.startup.Catalina.start(Catalina.java:577)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:295)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:433)


  • Bill Branan

    Bill Branan - 2008-07-28

    Logged In: YES
    Originator: YES

    There appear to be two related issues here, one is the log4j Error which shows up even with messaging off and with no gsearch installed. I'm not sure yet where this error originates.

    The other issue is the exception being thrown by the gsearch UpdateListener. This exception seems to be due to the fact that the ActiveMQ service running as part of the Fedora app is shutdown prior to the GSearch app, so attempting to close the connection to ActiveMQ does not work. I verified this by running ActiveMQ externally and shutdown ran through properly (no UpdateListener exception.) So I'm guessing that running an external ActiveMQ will allow the tomcat process close properly on Linux.

    When running with the embedded ActiveMQ, the exception due to not being able to close the connection is caught properly, but when trying to log the problem in the messaging client an uncaught exception (java.util.MissingResourceException: Can't find bundle for base name fedora.server.resources.Server, locale en_US) is thrown, which is what is likely causing the hanging process.

    I don't see the fact that the GSearch JMS connections cannot be closed because the JMS provider is already shutdown as a real issue since the connections have been closed (when ActiveMQ was shutdown), and any durable subscriptions (which GSearch uses) will be reestablished when started again. It's likely that the log4j error showing up in the log is due to the same missing resource exception as that causing the problem in the messaging client exception handler.

    So the question is, what might cause the Server.properties to become unavailable during shutdown?

  • Bill Branan

    Bill Branan - 2008-07-28

    Logged In: YES
    Originator: YES

    Comment from Eddie:

    Are you saying that the messaging client needs Server.properties? If that's the case, we can either a) identify & remove the dependency or b) just drop Server.properties into fedora-3.0-messaging-client.jar.

    The log4j error might be a red herring. I was trying to run this down before and it looked to be more of a issue with Tomcat's classloader manipulation that log4j used to be silent about but as of some recent version now warns about.

  • Bill Branan

    Bill Branan - 2008-08-15

    Logged In: YES
    Originator: YES

    Chris identified the dependency:

    I see that MessagingException (and any subclass of
    ServerException) has a runtime dependency on this bundle if .getMessage()
    is called. Fulfilling that dependency should just be a matter of making
    server/fedora/server/resources/Server.properties available in the classpath of
    the caller. One easy way to do that is for us to make sure it gets built as
    part of the fedora-client.jar file.

  • Daniel Davis

    Daniel Davis - 2008-09-15
    • status: open --> closed

Log in to post a comment.