Ganglia

Joel,

I have replied to your questions below.  Just a little background: My setup 
monitor almost 600 hosts.  I also wanted to measure many other metrics beyond 
the default 30 or so that are built into Ganglia, so I hacked the Ganglia source 
to compile in a few others and added some cron jobs to report other metrics too. 
I estimate that I am monitoring about 85 metrics per host.

>
> - Put gmetad rrd files on a ramdisk.
>

Definitely.  When I originally set things up, I had the rrds on an ext3 
filesystem.  The load on the system was always around 4.  I tweaked some of the 
VM settings in /proc and got the load down to around 3, but that was the best I 
could do. Turns out that the real killer was the journaling.  I put the VM 
settings back to their default values and remounted the filesystem as ext2. 
The load dropped to about 0.5!  Huge improvement.

But since we had plans to put even more stuff on the server, I decided to play 
it safe and write to a tmpfs filesystem.  For my setup, the metrics use about 
600 MB of RAM.  I rsync it to disk every 10 minutes, and I modified the 
start/stop init scripts to restore/save data.  I think when I moved to a 
ram-bsaed filesystem, the load from Ganglia dropped to something like 0.2 or 
0.3.  (I don't remember the exact numbers, but I do know that we have since 
added Nagios on the same system to monitor about 670 hosts and 850 services. 
And the average load on that system is still only around 0.5 or less.)

>
> - Use TCP polling queries instead of UDP or Multicast push. (disable 
> UDP/multicast pushing) I'd prefer to let gmetad poll instead of having 1000 
> UDP messages flying around on odd intervals.  A good practice?
>

Since gmond can't poll, I assume that you are talking about having gmetad query 
every gmond to gather the data?  If so, I doubt it is a good idea.

First of all, every node would appear to be its own cluster, so you wouldn't be 
able to see graphs for all nodes on the same page or see nice summary graphs for 
the entire cluster.

Second, (assuming the default poll rate) gmetad would be trying to setup and 
tear down 1000 TCP connections every 15 secs. Sure, you could change that poll 
rate (more on that below), but I think UDP is much better way to handle it.  I 
would suggest however that you consider using UPD unicast instead of multicast. 
It would cut down on the number of packets on your network.  Even though a 
single metric packet is only about 60-70 bytes, I don't see the need to send 
around any extra data that I don't have to.  The multicast method does provide 
you with redundancy (since every node knows the state of every other node), but 
if that is a concern, just unicast the packets to two hosts instead of one.

Richard Grevis made a comment in a previous reply about possibly having the 
nodes unicast their metrics back to the server running gmetad so that the gmetad 
process only had to connect to the loopback interface to retrieve data.  He said 
that he hadn't tried it so he wasn't sure if it would work well.  Fortunately, I 
can tell you for certain that it does since that is how I run my setup.  We have 
a couple of clusters with about 250 nodes each as well as several smaller ones. 
On our server, I run a separate gmond process for each cluster.  The nodes 
report their metrics to these gmonds.  The gmetad process then contacts them to 
get its data.  It seems to work quite well.

(I have one other comment on using UDP.  It is a rather long story, so I 
included it at the end.)


>
> - Alter timers for lighter network load? examples? ideas? Was going to just go 
> to 30 or 60s timers in gmetad.conf cluster definition to start.
>

That could definitely help.  I still use gmetad's default 15 secs because I 
haven't had any problems with it so far.  But I have done some experimenting to 
see how easily I could change that if it ever becomes an issue in the future.

Plus there is the fact that even if gmetad gets information every 15 secs, that 
doesn't mean the gmond's have to report it every 15 secs :-)  You should 
certainly take a look at what metrics are being collected and decide which ones 
are truly useful to you.  Keep those and disable the rest.  And for those that 
you do keep, look at the default reporting intervals.  You may find that you 
don't need to report the metrics as often, or perhaps the thresholds can be set 
higher so that only larger changes are supported.

>
> - Consider "federating"? Create groups of 100 gmond hosts managed by single 
> gmetas, all linking up to a core gmetad.
>

That is certainly a possibility.  Ganglia makes it easy to treat 1000 nodes as 
one cluster or as 10 100-node clusters.  Although, you don't really need to have 
each 100 node group have its own separate gmetad associated with it.  You can 
still just use a single one.


I should mention one final thing about large setups using UDP.  It is probably 
pretty rare, and you may never run across this problem.  But it is certainly 
something to keep in mind. When I first set things up, I had several metrics 
being reported by hourly cron jobs.  I noticed that every hour, there were a 
small number of nodes that failed to report back a couple of those metrics. 
The nodes that failed to do so changed every hour.  After some extensive 
testing, it seems that it was the result of lost UDP packets.  Part of the 
problem was that every node kicked of the cron job at the same time, and then 
transmitted the metrics at roughly the same time.  The other part appeared to be 
related to the interaction of the gmetad and gmond processes.  I had these "cron 
metrics" sent to two gmond servers which I'll call gmond-A and gmond-B.  When 
gmetad got its periodic info from gmond-A, there were a few nodes that showed up 
as not reporting metrics.  But when I checked the info on gmond-B, their recent 
values were indeed there.  So I pointed gmetad at gmond-B to gather info 
thinking it was because of a problem w/ gmond-A.  After that, gmetad still 
showed a few nodes as not reporting, but now gmond-A had their recent values. 
After some other tests, I came to the conclusion that when the gmetad process 
connected to the gmond process, it "engaged" the gmond long enough to cause the 
socket buffer to fill up and lose a few UDP packets.  So I added a random 10 sec 
sleep to all my cronjob metrics, and that seemed to spread out the load enough 
to make things OK.

>
> I've seen similar scaling questions asked, but not a lot of answers.
>

How's that for answers ;-)

-- Rick

--------------------------
Rick Mohr
Systems Developer
Ohio Supercomputer Center

I have commented on this multicast question before and I honestly do not
understand the fear of using multicast.  Like Rick said below, the
packets are small (60 bytes) and each node sends only about 20/minute on
average.  Do the math, even with a large cluster you are talking about a
tiny fraction of the network's capacity.  For us, redundancy is
important because we have many clusters and do not want to single out
one node in each cluster to make it a special monitoring node that needs
to be up 24/7 in order to collect ganglia monitoring data.  With
multicast's redundancy, we won't have to worry about the possibility
that the one special node out of our 400+ node cluster has crashed
taking ganglia down with it.  If the node gmetad is currently getting
data from has a problem, it will transparently switch to another.

Our largest cluster has 427 linux nodes with a basic gmond using all of
the default metrics.  I ran a tcpdump for about 10 minutes to capture
all of the multicast data in that cluster and here is what I found (*):

5,256,772 bytes collected in 628.45 seconds from 87,582 packets.

In other words, the average rate was:

139.362 packets/second
8364.668 bytes/second
0.066 Mbits/second

* Directly from ethereal's Summary window.

This is noise on modern 1 Gbit network, or even on a 100Mbit network.

~Jason


On Fri, 2006-01-27 at 11:26 -0500, Rick Mohr wrote:
> Joel,
> 
> I have replied to your questions below.  Just a little background: My setup 
> monitor almost 600 hosts.  I also wanted to measure many other metrics beyond 
> the default 30 or so that are built into Ganglia, so I hacked the Ganglia source 
> to compile in a few others and added some cron jobs to report other metrics too. 
> I estimate that I am monitoring about 85 metrics per host.
> 
> >
> > - Put gmetad rrd files on a ramdisk.
> >
> 
> Definitely.  When I originally set things up, I had the rrds on an ext3 
> filesystem.  The load on the system was always around 4.  I tweaked some of the 
> VM settings in /proc and got the load down to around 3, but that was the best I 
> could do. Turns out that the real killer was the journaling.  I put the VM 
> settings back to their default values and remounted the filesystem as ext2. 
> The load dropped to about 0.5!  Huge improvement.
> 
> But since we had plans to put even more stuff on the server, I decided to play 
> it safe and write to a tmpfs filesystem.  For my setup, the metrics use about 
> 600 MB of RAM.  I rsync it to disk every 10 minutes, and I modified the 
> start/stop init scripts to restore/save data.  I think when I moved to a 
> ram-bsaed filesystem, the load from Ganglia dropped to something like 0.2 or 
> 0.3.  (I don't remember the exact numbers, but I do know that we have since 
> added Nagios on the same system to monitor about 670 hosts and 850 services. 
> And the average load on that system is still only around 0.5 or less.)
> 
> >
> > - Use TCP polling queries instead of UDP or Multicast push. (disable 
> > UDP/multicast pushing) I'd prefer to let gmetad poll instead of having 1000 
> > UDP messages flying around on odd intervals.  A good practice?
> >
> 
> Since gmond can't poll, I assume that you are talking about having gmetad query 
> every gmond to gather the data?  If so, I doubt it is a good idea.
> 
> First of all, every node would appear to be its own cluster, so you wouldn't be 
> able to see graphs for all nodes on the same page or see nice summary graphs for 
> the entire cluster.
> 
> Second, (assuming the default poll rate) gmetad would be trying to setup and 
> tear down 1000 TCP connections every 15 secs. Sure, you could change that poll 
> rate (more on that below), but I think UDP is much better way to handle it.  I 
> would suggest however that you consider using UPD unicast instead of multicast. 
> It would cut down on the number of packets on your network.  Even though a 
> single metric packet is only about 60-70 bytes, I don't see the need to send 
> around any extra data that I don't have to.  The multicast method does provide 
> you with redundancy (since every node knows the state of every other node), but 
> if that is a concern, just unicast the packets to two hosts instead of one.
> 
> Richard Grevis made a comment in a previous reply about possibly having the 
> nodes unicast their metrics back to the server running gmetad so that the gmetad 
> process only had to connect to the loopback interface to retrieve data.  He said 
> that he hadn't tried it so he wasn't sure if it would work well.  Fortunately, I 
> can tell you for certain that it does since that is how I run my setup.  We have 
> a couple of clusters with about 250 nodes each as well as several smaller ones. 
> On our server, I run a separate gmond process for each cluster.  The nodes 
> report their metrics to these gmonds.  The gmetad process then contacts them to 
> get its data.  It seems to work quite well.
> 
> (I have one other comment on using UDP.  It is a rather long story, so I 
> included it at the end.)
> 
> 
> >
> > - Alter timers for lighter network load? examples? ideas? Was going to just go 
> > to 30 or 60s timers in gmetad.conf cluster definition to start.
> >
> 
> That could definitely help.  I still use gmetad's default 15 secs because I 
> haven't had any problems with it so far.  But I have done some experimenting to 
> see how easily I could change that if it ever becomes an issue in the future.
> 
> Plus there is the fact that even if gmetad gets information every 15 secs, that 
> doesn't mean the gmond's have to report it every 15 secs :-)  You should 
> certainly take a look at what metrics are being collected and decide which ones 
> are truly useful to you.  Keep those and disable the rest.  And for those that 
> you do keep, look at the default reporting intervals.  You may find that you 
> don't need to report the metrics as often, or perhaps the thresholds can be set 
> higher so that only larger changes are supported.
> 
> >
> > - Consider "federating"? Create groups of 100 gmond hosts managed by single 
> > gmetas, all linking up to a core gmetad.
> >
> 
> That is certainly a possibility.  Ganglia makes it easy to treat 1000 nodes as 
> one cluster or as 10 100-node clusters.  Although, you don't really need to have 
> each 100 node group have its own separate gmetad associated with it.  You can 
> still just use a single one.
> 
> 
> I should mention one final thing about large setups using UDP.  It is probably 
> pretty rare, and you may never run across this problem.  But it is certainly 
> something to keep in mind. When I first set things up, I had several metrics 
> being reported by hourly cron jobs.  I noticed that every hour, there were a 
> small number of nodes that failed to report back a couple of those metrics. 
> The nodes that failed to do so changed every hour.  After some extensive 
> testing, it seems that it was the result of lost UDP packets.  Part of the 
> problem was that every node kicked of the cron job at the same time, and then 
> transmitted the metrics at roughly the same time.  The other part appeared to be 
> related to the interaction of the gmetad and gmond processes.  I had these "cron 
> metrics" sent to two gmond servers which I'll call gmond-A and gmond-B.  When 
> gmetad got its periodic info from gmond-A, there were a few nodes that showed up 
> as not reporting metrics.  But when I checked the info on gmond-B, their recent 
> values were indeed there.  So I pointed gmetad at gmond-B to gather info 
> thinking it was because of a problem w/ gmond-A.  After that, gmetad still 
> showed a few nodes as not reporting, but now gmond-A had their recent values. 
> After some other tests, I came to the conclusion that when the gmetad process 
> connected to the gmond process, it "engaged" the gmond long enough to cause the 
> socket buffer to fill up and lose a few UDP packets.  So I added a random 10 sec 
> sleep to all my cronjob metrics, and that seemed to spread out the load enough 
> to make things OK.
> 
> >
> > I've seen similar scaling questions asked, but not a lot of answers.
> >
> 
> How's that for answers ;-)
> 
> -- Rick
> 
> --------------------------
> Rick Mohr
> Systems Developer
> Ohio Supercomputer Center
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
> _______________________________________________
> Ganglia-general mailing list
> Ganglia-general@...
> https://lists.sourceforge.net/lists/listinfo/ganglia-general
> 
-- 
/------------------------------------------------------------------\
|  Jason A. Smith                          Email:  smithj4@... |
|  Atlas Computing Facility, Bldg. 510M    Phone:  (631)344-4226   |
|  Brookhaven National Lab, P.O. Box 5000  Fax:    (631)344-7616   |
|  Upton, NY 11973-5000                                            |
\------------------------------------------------------------------/

On Fri, Jan 27, 2006 at 01:41:28AM -0800, Joel Krauska wrote:
> I've seen similar scaling questions asked, but not a lot of answers.
>=20
> I hope this query falls on some ears with experience in the brains=20
> behind them.
>=20
> I'm looking to deploy ganglia on a largish cluster.
> (1000+ nodes)
>=20
> Here were some of my thoughts on how I could help scale the system.
>=20
> Any opinions or suggestions are greatly appreciated.
>=20
> - Put gmetad rrd files on a ramdisk.
> This should decrease the frequency of disk writes
> during normal runs.
> If I rsync to a local disk every hour or so, I can get away
> with limited disk writes, and still have a reasonable backup of data.

I use a ramdisk for my rrds, but rather than using rsync I store them in
a tar.gz file.  The time difference between the two is night and day.
When you use rsync you waste a tremendous amount of time opening and
reading the backup files.  By just writing a tarball you only have to
open the files on the ramdisk and then by compressing the data you
reduce the number of writes.  The data is so small that using bzip is a
significant slowdown on the 2.4GHz Xeon so I use gzip instead because
it doesn't need to read a large chunk of data before it can start
writing.  I've integrated this into the FreeBSD port start and stop
scripts which automaticly take a snapshot at shutdown, restore from it
at startup, and can be called from cron to take period snapshots.  Even
if you lose a snapshot file, the system will recover from one of the
temporary files.

The script I use is part of the FreeBSD port:

http://www.freebsd.org/cgi/cvsweb.cgi/ports/sysutils/ganglia-monitor-core/f=
iles/gmetasnap.sh?rev=3D1.2&content-type=3Dtext/x-cvsweb-markup

> - Use TCP polling queries instead of UDP or Multicast push.
> (disable UDP/multicast pushing)
> I'd prefer to let gmetad poll instead of having 1000 UDP messages flying=
=20
> around on odd intervals.  A good practice?
>=20
>=20
> - Alter timers for lighter network load?
> examples? ideas?
> Was going to just go to 30 or 60s timers in gmetad.conf cluster=20
> definition to start.
>=20
>=20
> - Consider "federating"?
> Create groups of 100 gmond hosts managed by single gmetas, all linking
> up to a core gmetad.

As to the rest, IMO ganglia's traffic below the noise floor and I
don't bother to look at it.

-- Brooks

--=20
Any statement of the form "X is the one, true Y" is FALSE.
PGP fingerprint 655D 519C 26A7 82E7 2529  9BF0 5D8E 8BE9 F238 1AD4

It's not so much that I "fear" multicast, it's just that I see no need for it. 
Admittedly, my setup is not necessarily like others, and therefore my 
preferences don't necessarily apply to other setups.  And I wouldn't claim the 
my setup is the "best" setup by any means.  But if you are interested, I will 
try to explain my personal resoning for not using multicast. (I have replied to 
parts of your previous email below.)

> Like Rick said below, the packets are small (60 bytes) and each node sends 
> only about 20/minute on average.  Do the math, even with a large cluster you 
> are talking about a tiny fraction of the network's capacity.

This is not the whole picture (more on this below).  But even if it is a small 
amount, it's extra traffic that I do not need.  At best, it is unnoticable and 
serves no purpose. (So why keep it?)  At worst, the UDP traffic intereferes with 
my NFS traffic and results in extra RPC retransmits.

> For us, redundancy is important because we have many clusters and do not want 
> to single out one node in each cluster to make it a special monitoring node 
> that needs to be up 24/7 in order to collect ganglia monitoring data.

I agree which is why our monitoring node is not part of any cluster.  It is a 
separate machine that provides other monitoring services, so it is expected to 
be up 24x7 anyway.  Besides, I can easily get redundancy by sending unicast to 
two or three hosts.  I don't need 400 copies of my cluster's data.  And even if 
I did have that many copies, when I modify gmetad.conf, I certainly don't plan 
to enter 400 host names as alternative sources for gmond info.

>  With multicast's redundancy, we won't have to worry about the possibility 
> that the one special node out of our 400+ node cluster has crashed taking 
> ganglia down with it.  If the node gmetad is currently getting data from has a 
> problem, it will transparently switch to another.

I don't worry about redundancy either, but this also fails to address the 
availability of the gmetad process.  Ganglia isn't very useful if the gmetad 
server goes down and it is not recording the metrics to disk.  Since I pretty 
much need to make sure that gmetad is running 24x7 anyway, why not let it 
collect gmond info?  The server I run gmetad on is better equipped for 
availability than any of the compute nodes in the cluster.

> Our largest cluster has 427 linux nodes with a basic gmond using all of
> the default metrics.  I ran a tcpdump for about 10 minutes to capture
> all of the multicast data in that cluster and here is what I found (*):
>
> 5,256,772 bytes collected in 628.45 seconds from 87,582 packets.
>
> In other words, the average rate was:
>
> 139.362 packets/second
> 8364.668 bytes/second
> 0.066 Mbits/second

Thanks for supplying this data.  It is a good baseline for people to use when 
planning their Ganglia deployments.  However, I add a lot more metrics than the 
default.  I estimate that I get about 500 packets/second which translates to 
about 30,000 bytes/sec.  Plus, this "background noise" scales linearly with the 
number of nodes in the cluster.

I should also mention that the 60 byte metric packet actually consumes more than 
60 bytes in the linux socket receive buffer.  On my system, I ran gmond and then 
sent it the STOP signal.  I then used gmetric to submit a single metric by hand. 
By looking at the rx_queue column in /proc/net/udp, I saw that the one metric 
actually took up 304 bytes.  The buffer fills up pretty quick and UDP packets 
start dropping when the number of metrics gets big.

Then you also have to consider the bandwidth used by gmetad when it contacts 
gmond.  I tested having gmetad contact one of my cluster nodes to get the 
cluster info.  My test showed that it took about 0.7 secs for it to retreive 
about 2.5 MB of data.  That's 28.5 Mbps of the node's bandwidth used every 15 
seconds.  (In a previous post I also mentioned how I saw increased UDP drop 
rates whenever the gmetad contacted a gmond.)

So those are my reasons for choosing unicast.  Plenty of possibility for 
problems, very little benefit.  Other people's mileage may vary.

-- Rick

--------------------------
Rick Mohr
Systems Developer
Ohio Supercomputer Center

On Fri, 2006-01-27 at 15:05 -0500, Rick Mohr wrote: 
> It's not so much that I "fear" multicast, it's just that I see no need for it. 
> Admittedly, my setup is not necessarily like others, and therefore my 
> preferences don't necessarily apply to other setups.  And I wouldn't claim the 
> my setup is the "best" setup by any means.  But if you are interested, I will 
> try to explain my personal resoning for not using multicast. (I have replied to 
> parts of your previous email below.)

I didn't mean to make sound like I think that everyone should be using
multicast.  I am sure there are some places where it isn't necessary or
where unicast is better.  It is just it seems like every few months,
someone posts to the list asking how to set up ganglia without multicast
because they think it is too heavy for them or their network people
don't like it for some reason.  Its strength is in its redundancy and we
haven't noticed any negative impact in our environment.

> I agree which is why our monitoring node is not part of any cluster.  It is a 
> separate machine that provides other monitoring services, so it is expected to 
> be up 24x7 anyway.  Besides, I can easily get redundancy by sending unicast to 
> two or three hosts.  I don't need 400 copies of my cluster's data.  And even if 
> I did have that many copies, when I modify gmetad.conf, I certainly don't plan 
> to enter 400 host names as alternative sources for gmond info.

We have 10 clusters here and we don't want to pick a dedicated node from
each one and convert it into a ganglia node, for us, this would be a
waste of a compute node.  If we didn't use 10 dedicated nodes then we
would have to run 10 separate gmond or gmetad processes on a single node
which is more complications than we want.  Also, I forgot to mention
that we are using a modified gmetad that gets the list of nodes in each
cluster from our database so we don't have to create a huge config file,
we only specify the appropriate database query.  It also refreshes this
list periodically and sticks with a "good" source node until it has
problems, then tries another one.

> I don't worry about redundancy either, but this also fails to address the 
> availability of the gmetad process.  Ganglia isn't very useful if the gmetad 
> server goes down and it is not recording the metrics to disk.  Since I pretty 
> much need to make sure that gmetad is running 24x7 anyway, why not let it 
> collect gmond info?  The server I run gmetad on is better equipped for 
> availability than any of the compute nodes in the cluster.

Of course the gmetad server is critical, but it is better to have a
single critical server rather than the 10 that we would need.  Because
we want to keep these 10 clusters separate, we would need to configure
10 separate gmond or gmetad on our ganglia server if we wanted to stay
with just a single server.

> Thanks for supplying this data.  It is a good baseline for people to use when 
> planning their Ganglia deployments.  However, I add a lot more metrics than the 
> default.  I estimate that I get about 500 packets/second which translates to 
> about 30,000 bytes/sec.  Plus, this "background noise" scales linearly with the 
> number of nodes in the cluster.

This is true, and the default metrics are different for different OSes I
believe.

> I should also mention that the 60 byte metric packet actually consumes more than 
> 60 bytes in the linux socket receive buffer.  On my system, I ran gmond and then 
> sent it the STOP signal.  I then used gmetric to submit a single metric by hand. 
> By looking at the rx_queue column in /proc/net/udp, I saw that the one metric 
> actually took up 304 bytes.  The buffer fills up pretty quick and UDP packets 
> start dropping when the number of metrics gets big.

I was only considering the ganglia multicast bandwidth usage on the
wire, which is something most people seem to be concerned with, not the
extra overhead in the kernel.

> Then you also have to consider the bandwidth used by gmetad when it contacts 
> gmond.  I tested having gmetad contact one of my cluster nodes to get the 
> cluster info.  My test showed that it took about 0.7 secs for it to retreive 
> about 2.5 MB of data.  That's 28.5 Mbps of the node's bandwidth used every 15 
> seconds.  (In a previous post I also mentioned how I saw increased UDP drop 
> rates whenever the gmetad contacted a gmond.)

This is one reason why we have increased our polling interval to 60
seconds instead of the default 15.  I'm glad that you brought this up
because it reminds me of something that I remember seeing in an early
ganglia beta, optional support for compressed xml tcp streams.  Does
anyone know what happened to this feature?  Any chance of resurrecting
the code?

~Jason


-- 
/------------------------------------------------------------------\
|  Jason A. Smith                          Email:  smithj4@... |
|  Atlas Computing Facility, Bldg. 510M    Phone:  (631)344-4226   |
|  Brookhaven National Lab, P.O. Box 5000  Fax:    (631)344-7616   |
|  Upton, NY 11973-5000                                            |
\------------------------------------------------------------------/