Re: [Apcupsd-users] Graceful shutdown on comm or network loss

[Fabian W. <fa...@we...>]

Hello Nicholas

On 20.04.2012 15:52, Nicholas.Pirollo@L-3com.com wrote:
> Hello Fabian,
>       Thank you for your reply.  This was a request of our customer, but
> we brought up the same issues to them as you did to me.  Perhaps we can
> convince them of dropping the feature if say, when on battery and you
> lose connection the server would shut down.  I know currently if I put
> the servers under UPS power and then cut the connection between server 1
> (master) and server 2 (slave) that the slave says "NETWORK ERROR" and
> never shuts down (assumably the timer for TIMEOUT is cleared?) - and
> server 1 turns off as expected.

I do not know how you have configured BATTERYLEVEL and MINUTES on 
the server connected directly to the UPS (master apcupsd) and to 
the other servers (slave apcupsd).

My setup is like this:

SMT1500I - has around 39 minutes battery run time
superman - server with USB cable to SMT1500I and master apcupsd
goliath  - server with slave apcupsd

superman has the following options in apcupsd.conf:
POLLTIME 10
BATTERYLEVEL 25
MINUTES 15

apcupsd runs with option --kill-on-powerfail, as is it a FreeBSD, 
so on system shutdown it should tell the USP to turn power off. 
The UPS will turn off the outlets after a delay (have set it to 5 
minutes on the UPS). This delay depends on the shutdown time 
needed for your systems.

goliath has the following options in apcupsd.conf:
POLLTIME 10
BATTERYLEVEL 30
MINUTES 17

I have not (yet) tested this with a "real" power failure, but as 
far as I think, it should work like this:

1. goliath starts first to shut down, without telling the UPS 
anything to turn off (as it is a slave apcupsd)

2. superman starts about 2 mintues later to shutdown and then 
will tell the UPS to turn off the outlets (with the 5 minutes 
delay configured in the UPS)

3. 5 minutes later UPS turns outlets off

4. utility power returns, UPS will turn on the power on the 
outlets only when there is at least battery power for 8 minutes 
(configured on the UPS)

5. Power returns on the UPS outlets and superman and goliath 
should power up (because they are set to "last state" in the BIOS)


With this scenario it should work perfectly if the power outage 
is long enough. But it will probably fail and the systems will 
not boot up, when the power returns during the time when superman 
already has shut down, but the UPS is still running in the 5 
minutes delay to turn off the outlets.

But I still have the problem, when superman is down or the 
network connection from goliath to superman is lost during a 
power outage. But when utility power is normal, a interruption of 
the communications between superman and goliath will not hurt 
anything. But when you use this to initiate a shut down, this 
will probably hurt more (as the system goes down).

There is no easy 100% solution for all possible situations. The 
best thing is, to have a proper system monitoring (I use 
check_apcupsd [1] in Nagios) and staff which is able to react 
with the right action to any failure at all the time.

   [1] 
http://exchange.nagios.org/directory/Plugins/Hardware/UPS/APC/check_apcupsd/details

What happens if eg. the UPS just dies? With only a single power 
supply in the system, it will just turn off and also not do a 
proper shutdown. I have some ideas to have a better availability, 
but this adds only more complexity on what to do depending on the 
event. It probably also depends on the quality of the utility 
power. Somehow I am lucky, as I have seen only one real power 
outage (around 1.5 hour) in the past 8 years here, else there 
where only some really short outages (seconds or fraction of a 
second).

For example, use only systems with dual power supply, connect one 
directly to the utility power and the other one to the UPS. This 
does protect from failing UPS, as long as the utility power is 
there. Depending on your hardware, it is possible to also monitor 
the power supplies and see if one of them fails or lost power. 
But here you need to be sure, which power supply has lost power 
(or failed), the one from the UPS or the one from utility power. 
An other solution could be with using different UPS for each of 
the dual power supplies, but here it will probably get more 
complicated when using apcupsd.

I guess you have now some input for doing your own brainstorming 
and finding the perfect solution for your (or your customer) needs.


bye
Fabian