Re: [Ganglia-developers] Bad summary data in gmetad.

[Jason A. S. <sm...@bn...>]

I figured out what the problem was.  I looked at the gmetad source code
here: gmetad/process_xml.c, and remembered that I had one gmetad that
was older than 2.5.  For older versions of the xml data, gmetad uses a
different method for calculating the host_up flag:

abs(cluster_localtime - reported) < 60

while for xml data newer that ganglia 2.5 it uses:  tn < tmax * 4

Because of how we have ganglia setup, the xml data goes through a couple
levels of gmetads and some host's data can easily be older than 60
seconds which causes sporadic intervals where many hosts were being
marked as down.

This does point out a few problems with gmetad and the webfrontend:

1. gmetad and the webfrontend currently have different methods for
determining then a host should be marked as down.  gmetad will use 60
seconds since the last reported metric for old xml and 4 times tmax for
new xml, while the webfrontend uses the same method for new xml data but
uses: abs(cluster_localtime - reported) < 4*60 for old xml data, see the
ganglia.php:host_alive function.  This inconsistency and the fact that
we had an old gmetad, made the webfrontend view confusing.  The summary
data indicated that some nodes were down because the xml data was a
little older than 60 seconds for some nodes, while the webfrontend
showed no nodes down in the cluster view because it was older than 60
seconds, but wasn't older than 4 minutes.  These should be made
consistant.

2. 60 seconds seems a little low to me, especially if you are using the
scalability option and a few gmetads or a longer than default update
interval.

A separate problem we are seeing is when one of gmetad's data source
threads gets stuck (maybe when it can't contact any hosts in its
cluster), then that cluster's data stops being updated.  When this
happens the webfrontend is very misleading.  The grid summary xml data
and rrd graphs decrease their totals because of that "down" cluster, but
the rest of the grid view on the webfrontend doesn't indicate which
cluster is down unless you look very closely at the time values on the
cluster graphs.  What happens is that the right edge of the rrd graphs
for all other working clusters appears normal (right edge is the current
time), but for the down cluster the right edge never advances in time. 
It stays at the last time which that cluster was updated.  Since this is
displayed in a table format you would expect the horizontal range to all
be the same for each column, right?  Also, the cpu and host totals to
the left of the graphs still show no problem.  Is this a problem with
gmetad summary data, the webfrontend or both?  Maybe the total grid
summary is showing decreased numbers but the summary for the affected
cluster still has old data?  If the right edge on the graph of the
affected cluster was the current time and the summary data for that
cluster correctly indicated it was old, maybe by marking all hosts as
down then we could more easily identify where the problem is.

Sorry for the long email....
~Jason


On Wed, 2004-07-14 at 15:05, Jason A. Smith wrote:
> I have two gmetad servers, setup using the scalability option so they
> each keep their own copy of the rrds.  The second gmetad, which gets
> data from the first one, appears to be keeping corrupted summary
> statistics occasionally.  Every few minutes it thinks some of the nodes
> are down.  The result is bad summary data being written to the rrds,
> which causes graphs like the example I have attached.
> 
> I have tried looking at the raw xml data, from both gmetad servers, to
> see if I could find the cause, but I didn't see anything unusual. 
> During the small intervals that it thinks some of the nodes are down, if
> I query the second gmetad for its filter=summary output, I get results
> like this:
> 
> $ echo "/?filter=summary" | nc localhost 8652 | grep '<HOSTS'
> <HOSTS UP="63" DOWN="29" SOURCE="gmetad"/>
> 
> Also, when this happens the top-level grid display on the webfrontend
> shows 29 hosts down for the whole grid and for the cluster reported from
> that second gmetad.  If I look at that cluster view though, none of the
> hosts are marked as down and the raw xml looks good.  It is only the
> filter=summary xml that looks bad and the rrd graphs.  From this, I can
> only assume that gmetad must have corrupt summary statistics internally
> which it reports on the query port and writes into the rrds.  Is this
> some thread timing or data locking issue?  Does anyone else see this?
> 
> Ganglia version: 2.5.6
> 
> ~Jason
-- 
/------------------------------------------------------------------\
|  Jason A. Smith                          Email:  sm...@bn... |
|  Atlas Computing Facility, Bldg. 510M    Phone:  (631)344-4226   |
|  Brookhaven National Lab, P.O. Box 5000  Fax:    (631)344-7616   |
|  Upton, NY 11973-5000                                            |
\------------------------------------------------------------------/