wrapper-user

[Wrapper-user] Starting and stopping multiple configured services

[Paul C. <cas...@au...>]

Hi,

This is my first post to this forum.

First up Leif, I would like to thank and congratulate you and everyone else
who has worked on the Java Service Wrapper for making such a fantastic
configurable and scaleable product available to us all.

I manage a complex Java application for a client.  The application has
(now) 5 components as follows:
Tomcat 4.1.24
RMI Registry
AutoUpdater
MainServer (-Xmx1100)
AdditionalServer (-Xmx1100)

All of these now start and stop beautifully individually via the Windows
Services applet.  Previously when we took over support of the application,
the "srvany" Microsoft tool (horrid) was being used to start and stop the
components of the application individually and as a whole using "net
stop..." and "net start..." commands inside batch scripts.

I can start them all at once successfully from a batch file using "net
start..." eg:
net start service1
net start service2
...

However, using this approach to stop all the services together (eg for
nightly database backup) now times out though because the applications are
being shut down cleanly, rather than just being killed off (ala "srvany").

I have a dependency for all of the services (except Tomcat) on the
RMIRegistry, which works fine through the Services applet - when it starts,
they all start; when it stops, they all stop.  No problem.

However, the "net stop" command ignores this dependency, and doesn't stop
or start the other services when it is stopped or started.

What is the preferred method of stopping several Java Service Wrapper
services at the same time?  The first 3 services mentioned aren't a problem
because they are relatively small and simple (< 20MB).  However, the last 2
use up to 1.1GB of RAM each (yeah, I know - quite large) and have large
database connection pools and things they need to clean up before they can
exit cleanly.

Is there any alternative to "net stop"?
Can the wrapper be configured to have more than 1 shut down method?

Any ideas on this would be much appreciated.

Thanks,

Paul Casanova
AMS Ballarat Application Centre
IBM Global Services Australia
Mount Helen

Telephone:
61-3-53273245
Fax:
61-3-53273297

Re: [Wrapper-user] Starting and stopping multiple configured services

[Leif M. <le...@ta...>]

Paul Casanova wrote:

>First up Leif, I would like to thank and congratulate you and everyone else
>who has worked on the Java Service Wrapper for making such a fantastic
>configurable and scaleable product available to us all.
>
Thanks, glad you find it useful.

>I manage a complex Java application for a client.  The application has
>(now) 5 components as follows:
>Tomcat 4.1.24
>RMI Registry
>AutoUpdater
>MainServer (-Xmx1100)
>AdditionalServer (-Xmx1100)
>
Wow.  My limit so far has been 512MB

>All of these now start and stop beautifully individually via the Windows
>Services applet.  Previously when we took over support of the application,
>the "srvany" Microsoft tool (horrid) was being used to start and stop the
>components of the application individually and as a whole using "net
>stop..." and "net start..." commands inside batch scripts.
>
>I can start them all at once successfully from a batch file using "net
>start..." eg:
>net start service1
>net start service2
>...
>
>However, using this approach to stop all the services together (eg for
>nightly database backup) now times out though because the applications are
>being shut down cleanly, rather than just being killed off (ala "srvany").
>
>I have a dependency for all of the services (except Tomcat) on the
>RMIRegistry, which works fine through the Services applet - when it starts,
>they all start; when it stops, they all stop.  No problem.
>
>However, the "net stop" command ignores this dependency, and doesn't stop
>or start the other services when it is stopped or started.
>
>What is the preferred method of stopping several Java Service Wrapper
>services at the same time?  The first 3 services mentioned aren't a problem
>because they are relatively small and simple (< 20MB).  However, the last 2
>use up to 1.1GB of RAM each (yeah, I know - quite large) and have large
>database connection pools and things they need to clean up before they can
>exit cleanly.
>
I want to confirm that your problem is not that the Wrapper is timing 
out and
killing the JVM prematurely.  I don't think it is from what you said.

The only thing I can think of would be to either write a C (or Java) 
application
that would be a little smarter about waiting for each process to stop.

An easy way to do this would be to have your batch file pause for a 
minute or
so after each call to "net stop <apl>".  This way you could guarantee that
enough time was allowed for the application to actually stop.   I assume 
that
net stop is timing out, but that the Wrapper is continuing to stop the 
service
anyway and it will eventually exit on its own.

Looking into this, there does not seem to be an easy way to wait in a batch
file.  But I did locate this page which describes how to use the PING
command to ping a host for n seconds and then complete.  This will
effectively pause your batch file for n seconds.

http://www.robvanderwoude.com/index.html

Their example is pretty complex because it is designed to work on any
windows version.  But you basically just place a call like the following 
into
your batch file where you want to wait for 60 seconds:

PING 1.1.1.1 -n 60 -w 1000 >NUL

Just change the "60" to the number of seconds to wait.

>Is there any alternative to "net stop"?
>Can the wrapper be configured to have more than 1 shut down method?
>
What kind of additional shutdown method are you thinking would be useful?

If you found the product and support useful, please think about supporting
the project:  http://wrapper.tanukisoftware.org/doc/english/donate.html

Cheers,
Leif

Re: [Wrapper-user] Starting and stopping multiple configured services

[Paul C. <cas...@au...>]

Thanks for responding so fast!

>I want to confirm that your problem is not that the Wrapper is timing
>out and
>killing the JVM prematurely.  I don't think it is from what you said.

I initially set it all up with DEBUG turned on, until I got it running with
all errors gone.  I did have to increase the timeout periods, but yes I
believe that when the services are stopped individually from the Services
applet, they exit without error.

You're right - even though "net stop" bombs out with an error ("service
could not be stopped..." or something), the services do appear to have
stopped in the Services applet.  I'm just a little nervous about putting
that into production.

The extra method I was thinking of is for the nightly backup situation,
where it's pretty much guaranteed that users will be off the system
(midnight), perhaps another class could be written which ends the services
more abruptly or something.  I'm pretty new on the services side of things,
so this may either be a valid idea, or complete nonsense that I'm
suggesting!!

I'll speak to management to see what can be done about supporting the Java
Service Wrapper project.

Thanks,

Paul Casanova

Re: [Wrapper-user] Starting and stopping multiple configured services

[Leif M. <le...@ta...>]

Paul Casanova wrote:

>Thanks for responding so fast!
>
>>I want to confirm that your problem is not that the Wrapper is timing
>>out and
>>killing the JVM prematurely.  I don't think it is from what you said.
>>    
>>
>I initially set it all up with DEBUG turned on, until I got it running with
>all errors gone.  I did have to increase the timeout periods, but yes I
>believe that when the services are stopped individually from the Services
>applet, they exit without error.
>
>You're right - even though "net stop" bombs out with an error ("service
>could not be stopped..." or something), the services do appear to have
>stopped in the Services applet.  I'm just a little nervous about putting
>that into production.
>
I'll do some more testing to make sure that "net stop" is not timing out
because of a problem with the Wrapper.  The Wrapper should be notifying
the Service Manager that it needs more time to shutdown cleanly every few
moments.  Same happens as the server is starting up.

>The extra method I was thinking of is for the nightly backup situation,
>where it's pretty much guaranteed that users will be off the system
>(midnight), perhaps another class could be written which ends the services
>more abruptly or something.  I'm pretty new on the services side of things,
>so this may either be a valid idea, or complete nonsense that I'm
>suggesting!!
>
That is possible by setting the shutdown timeouts to very short values 
within
the wrapper.conf file.  But there is only one of those.  The problem is that
there are not multiple commands coming from the ServiceManager based
on the time.

It would be possible to add something to the wrapper.conf where you could
define windows of time where the Wrapper would immediately kill the JVM
rather than requesting the JVM to shutdown cleanly.  If possible I would 
like
to avoid doing this as I don't think this a very common problem however.

You could implement this behavior easily in your own applications by
adding a check to the beginning of your application's shutdown hook.  If
the current time is within a certain window, execute the following code.
If you call halt outside a shutdown hook, and the Wrapper has not
requested that the JVM shutdown, then it will interpret the JVM exiting
as it having crashed and will restart the JVM.  The call to signalStopped
is to tell the Wrapper that the JVM is exiting intentionally so it is not
restarted.  System.exit will shutdown cleanly and the Wrapper will
handle it correctly.

WrapperManager.signalStopped( 0 )
Runtime.getRuntime().halt(0)

Hope this helps.

Cheers,
Leif

Re: [Wrapper-user] Starting and stopping multiple configured services

[Leif M. <le...@ta...>]

Leif Mortenson wrote:

> I'll do some more testing to make sure that "net stop" is not timing out
> because of a problem with the Wrapper.  The Wrapper should be notifying
> the Service Manager that it needs more time to shutdown cleanly every few
> moments.  Same happens as the server is starting up. 

I verified that the Wrapper is indeed correctly informing the
ServiceManager that the Wrapper needs more time to stop cleanly.  It
appears that the "net stop" command is just ignoring that and timing out
on its own.  Probably so the user on the command line doesn't think its
hung or something...  All well.

Good news though.  I think I came up with a solution to your problem.  See
if you think it is acceptable.
Rather than using "net stop" to stop the service, use the Wrapper to remove
the service.  The Wrapper will wait until the service has completely stopped
before it is removed, so this should give you the desired behavior.  Then
when you want to start the services back up, you will have to reinstall 
them.

To stop your services, you would use this:
---
C:\srv1\bin\Wrapper -r ..\conf\wrapper.conf
C:\srv2\bin\Wrapper -r ..\conf\wrapper.conf
C:\srv3\bin\Wrapper -r ..\conf\wrapper.conf
C:\srv4\bin\Wrapper -r ..\conf\wrapper.conf
C:\srv5\bin\Wrapper -r ..\conf\wrapper.conf
---

Then to start them back up, you would do this:
---
C:\srv5\bin\Wrapper -i ..\conf\wrapper.conf
net start srv5
C:\srv4\bin\Wrapper -i ..\conf\wrapper.conf
net start srv4
C:\srv3\bin\Wrapper -i ..\conf\wrapper.conf
net start srv3
C:\srv2\bin\Wrapper -i ..\conf\wrapper.conf
net start srv2
C:\srv1\bin\Wrapper -i ..\conf\wrapper.conf
net start srv1
---

Version 3.0.3 and below will display a message once per second saying
that the Wrapper is waiting for the service to stop.  I just reduced that
frequency to once every five seconds.  That will be in 3.0.4 when it is
released.

Let me know whether or not this will work for you.  Another possibility 
is to
create a simple tool that has the sole job of replacing the "net start"
command with something that does not time out.  But the above works as
is.

Cheers,
Leif

Re: [Wrapper-user] Starting and stopping multiple configured services

[Paul C. <cas...@au...>]

Sorry for not replying sooner - it's 9am here in Australia now.

Leif, this sounds like the way to go!  I'll do some testing on it this
morning and give you some feedback.

The only concern I have is that sometimes Windows NT (our dev server
anyway) plays up when services are added and removed several times.  It
comes up with a message saying that "the services database is locked".
It's really weird, but this started happening after we switched the server
from Token Ring to Ethernet (finally!).  It reoccurred when I was testing
JavaService, but to date I haven't seen the problem occur with Java Service
Wrapper, so here's hoping it was because of buggy tools.

Thanks,

Paul Casanova

Re: [Wrapper-user] Starting and stopping multiple configured services

[Paul C. <cas...@au...>]

Hi Leif,

Having some trouble here gettings Windows NT to stop the services and/or
remove the services without problem.

I was thinking about writing a new class to control shutdown of the
application components at backup time - although I'm not really sure how to
do this in light of the fact that I'm using method 2 of the integration
methods (org.tanukisoftware.wrapper.WrapperStartStopApp).

I'm really only having trouble with the complicated classes - ie those with
database connections to close etc.  Tomcat and the registry server shut
down ok via "net stop".

I was thinking of adding a new method for each of my classes (eg "public
void rapidShutdown()") which would just call System.exit().  This method in
each of these classes could then be called by a new class whose only
purpose is to shut them all down at backup time, and then stop the service
("net stop").  Would the Java Service Wrapper see this as a JVM crash and
try to restart the application(s)?

Thanks,

Paul Casanova

Re: [Wrapper-user] Starting and stopping multiple configured services

[Leif M. <le...@ta...>]

Paul,

>Having some trouble here gettings Windows NT to stop the services and/or
>remove the services without problem.
>
What is the problem that you are having?  net stop will time out but the
removing the service should always be waiting and I need to know if it is
not.

>I was thinking about writing a new class to control shutdown of the
>application components at backup time - although I'm not really sure how to
>do this in light of the fact that I'm using method 2 of the integration
>methods (org.tanukisoftware.wrapper.WrapperStartStopApp).
>
>I'm really only having trouble with the complicated classes - ie those with
>database connections to close etc.  Tomcat and the registry server shut
>down ok via "net stop".
>
>I was thinking of adding a new method for each of my classes (eg "public
>void rapidShutdown()") which would just call System.exit().  This method in
>each of these classes could then be called by a new class whose only
>purpose is to shut them all down at backup time, and then stop the service
>("net stop").  Would the Java Service Wrapper see this as a JVM crash and
>try to restart the application(s)?
>  
>
I described how to shut your application down instantly in a previous
email, but I'll go into more detail this time.
---
You could implement this behavior easily in your own applications by
adding a check to the beginning of your application's shutdown hook.  If
the current time is within a certain window, execute the following code.
If you call halt outside a shutdown hook, and the Wrapper has not
requested that the JVM shutdown, then it will interpret the JVM exiting
as it having crashed and will restart the JVM.  The call to signalStopped
is to tell the Wrapper that the JVM is exiting intentionally so it is not
restarted.  System.exit will shutdown cleanly and the Wrapper will
handle it correctly.

WrapperManager.signalStopped( 0 )
Runtime.getRuntime().halt(0)
---

I think you should let your system stop the Wrapper controlled Java apps
as usual.   On the Java side, when the Wrapper starts to shut them down,
their shutdown hooks will be started.  I don't know if you have created any
of your own shutdown hooks, but they are pretty easy to do.
Just add the following code and write the isTimeForQuickShutdown()
method to suit your needs.

---SOURCE BEGIN
Thread shutdownHook = new Thread()
{
public void run()
{
System.out.println( "My shutdown hook starting." );

if ( isTimeForQuickShutdown() )
{
System.out.println( "Forcing an immediate shutdown." );
WrapperManager.signalStopped( 0 )
Runtime.getRuntime().halt(0)
}

System.out.println( "My shutdown hook completed." );
}
}
Runtime runtime = Runtime.getRuntime();
runtime.addShutdownHook( shutdownHook );
---SOURCE END

One idea for the isTimeForQuickShutdown() would be to look for a marker
file in a certain directory.  If it is there, halt abruptly, otherwise 
shutdown
cleanly.  Your batch file could then simply create this file when it 
wants to
shutdown all of your applications for their nightly backups.  This way you
could run the script at any time of the day.

Cheers,
Leif

Re: [Wrapper-user] Starting and stopping multiple configured services

[Leif M. <le...@ta...>]

Paul,
    Ok, I spent some time today and added two new commands to the
Wrapper on Windows to start and stop an NT Service.  These commands
are actually quite useful as they use the wrapper.conf file to stop the
service rather than requiring the name of the server as do the net start
and net stop commands.
    In addition, these commands work much more reliably than their
net start/stop counterparts.  The Wrapper now gives nice descriptive
explanations of exactly what is happening as it tries to stop or stop a
service.
    For example if you start a service that takes a while to start in one
command window and stop it from another, you will get clear messages
in both command windows and they will both block until the service has
stopped.
    Likewise, if you stop a service that takes a long time to stop in one
window and then attempt to stop it again from another window, the
second attempt will tell the user that the service was already stopping
and they will both wait until the service has completely stopped.
    I think I have all the various error cases worked out as well, but 
please
try out any that you can think of.  If any of the messages are confusing,
let me know and I'll try to make them a little clearer.

    As you are the main reason for these new features, I was hoping you
could help test them out and verify that they are working with your
applications as expected.  You can download version 3.0.4b from the
following URL.  This is not an official release and has not yet gone through
my full wave of tests, but it should be fine.
http://wrapper.tanukisoftware.com/tmp/wrapper_win32_3.0.4b.zip

    If you type "Wrapper.exe -?" you will see the new commands.

    Installing and removing the service is the same as before.  To start a
service that has been installed. use the following command:
Wrapper.exe -t ../conf/wrapper.conf

    To stop it, use the following:
Wrapper.exe -p ../conf/wrapper.conf

    Hope this helps,
Cheers,
Leif

Re: [Wrapper-user] Starting and stopping multiple configured services

[<da...@ix...>]

In article <3ED...@ta...>,
Leif Mortenson  <wra...@li...> wrote:
>    To stop it, use the following:
>Wrapper.exe -p ../conf/wrapper.conf

It would be nice is the wrapper.bat file handled start|stop|console in the
same way the unix scripts did.

Would make less special casing of documenation.  :->

mrc
-- 
     Mike Castle      da...@ix...      www.netcom.com/~dalgoda/
    We are all of us living in the shadow of Manhattan.  -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc

Re: [Wrapper-user] Starting and stopping multiple configured services

[<da...@ix...>]

In article <3ED...@ta...>,
Leif Mortenson  <wra...@li...> wrote:
>Looking into this, there does not seem to be an easy way to wait in a batch
>file.  But I did locate this page which describes how to use the PING
>command to ping a host for n seconds and then complete.  This will
>effectively pause your batch file for n seconds.

We used to use this method.

These days I recommend a java app with nothing more than Thread.sleep(X);
in it.  I mean, you're already requiring java anyway.  Might as well make
use of it.  :->

mrc

-- 
     Mike Castle      da...@ix...      www.netcom.com/~dalgoda/
    We are all of us living in the shadow of Manhattan.  -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc