[Devmon-devel] [ devmon-Bugs-2897345 ] devmon goes purple

[SourceForge.net <no...@so...>]

Bugs item #2897345, was opened at 2009-11-13 19:05
Message generated for change (Comment added) made by buchanmilne
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=816977&aid=2897345&group_id=160720

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Devmon
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: devmon goes purple

Initial Comment:
Regarding devmon version 0.3.1-beta1
- Devmon stops working at irregular intervals. 
- The dm column in xymon "goes purple" for lack of new information. 
- devmon processes remain running.
- running "sudo killall devmon" (on ubuntu 8.10) is enough to return to a "green" state
- no errors or new entries in devmon.log file after communication stops
- upon sending a SIGTERM to the devmon master process, it stops all other processes, so it looks like it is responding to signals as it should

Other users have commented that running a cron job to kill the processes is their workaround. Or to use xymon to run a SCRIPT upon receiving a "purple" alert to restart the services.

(comments compiled from thorsten.erdmann and gthomas on hobbit/xymon mailing list)

----------------------------------------------------------------------

>Comment By: Buchan Milne (buchanmilne)
Date: 2011-01-25 14:25

Message:
I have now been able to test current svn in an environment that experiences
this problem. The changes in svn didn't cure this.

In this case, it seems like devmon hung while trying to send the actual
status messages to hobbit.

I'm working on adding timeouts here as well.

----------------------------------------------------------------------

Comment By: Buchan Milne (buchanmilne)
Date: 2010-03-11 01:26

Message:
I have now committed some changes I have been working on, that should
hopefully:

1)Log more information in case of any socket communication errors
2)Provide information on fork behaviour in the dm test for the polling
host
3)Provide for terminating forks that seem to be stalled (by sending data
to idle forks to ensure they are alive), as a workaround that should
prevent having to have scripts to restart devmon
4)Add timeouts for all socket communication (using "alarm" and "eval"),
hopefully fixing the original problem

If you can reproduce this issue with any better frequency, I would
appreciate it if you could run the version currently in subversion (rev 180
or later). Preferably, run it in debug mode with high (5) verbosity, e.g.:

./devmon --debug -vvvvv -d /var/lib/devmon/hosts.db -c /etc/devmon.cfg

If the problem persists, watch to see if the dm test for the polling
device goes yellow or red, or polled devices go purple, and provide the log
contents for at least the last two poll cycles before any of these changes
(or, the relevant problems).


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=816977&aid=2897345&group_id=160720