Ganglia

Hi Brad,

On Thursday 08 November 2007 02:25:21 Brad Nicholes wrote:
> [...]  Briefly I have been trying to add some additional data to the
> gmond XDR packets so that we can communicate attributes such as an
> alternate TITLE for the metric or which GROUP(s) the metric belongs to (or
> anything else that we dream up in the future).  What I have done to solve
> this issue is the following [...]

=46rom a quick skim, this all looks pretty good. =20

Although it's early days, are you planning on supporting "metric packing" (=
for=20
want of a better name) with several metrics will end up in a single UDP=20
packet?

So, if I've understood correctly, the potential attributes are now:

  1. name:  unique(*) short string for metric (potential use: create a
	suitable filename for RRD storage)

  2. host: the FQDN of the host from which the metric proports to come.

  3. type: what kind of data is collected.

  4. value: a (current/recent) measured value of this metric

  5. title:  short name (potential use: a title of an RRD graph)

  6. description: a longer string, giving more information (potential use:
	pop-up box in web-interface)

  7. group: some indication of how this metric can be grouped with others.

  8. units:  a suffix to each number (potential use: label in y-axis, suffix
	after static values)

  9. slope: an indication of whether the data is a counter-, gauge- or
	static- metric.

  10. tmax: indicate how regularly the metric should be updated.

  11. dmax: how quickly a metric should be dropped when no further=20
measurements are received.

(*) The (name,host) ordered pairs must be unique for any one metric.

Data packet contains 1.--4.
Metadata packet contains 1.--3. and 5.--11.

Do we also want to have a timestamp when the metric was measured, or do we=
=20
assume it was sufficiently close to "now"?

If we're going to record metrics in base-units (bytes rather than megabytes=
)=20
and let RRDTool do the scaling, then we should also include a flag about=20
whether the SI scaling prefix should be in powers of 10 (kilo, Mega, ...) o=
r=20
powers of 2 (kibi, Mebi, ...).  IIRC, RRDTool supports both.

Perhaps we could add a bit-wise set of attributes one bit could be wither t=
he=20
metric should be scaled by powers-of-10 or powers-of-2.  The current slope=
=20
value could be folded in with this attribute, or left separate.

Did we come to any conclusion about the semantics of the group field? =20
Something like a space separated list of groups.  Would membership of a gro=
up=20
imply plotting together on the same graph, or somehow close together on a=20
webpage?

Sorry, too many questions!

Cheers,

Paul.

>>> On 11/8/2007 at 12:37 PM, in message
<200711082037.39314.p.millar@...>, Paul Millar
<p.millar@...> wrote:
> Hi Brad,
>=20
> On Thursday 08 November 2007 02:25:21 Brad Nicholes wrote:
>> [...]  Briefly I have been trying to add some additional data to the
>> gmond XDR packets so that we can communicate attributes such as an
>> alternate TITLE for the metric or which GROUP(s) the metric belongs to =
(or
>> anything else that we dream up in the future).  What I have done to =
solve
>> this issue is the following [...]
>=20
> From a quick skim, this all looks pretty good. =20
>=20
> Although it's early days, are you planning on supporting "metric =
packing"=20
> (for=20
> want of a better name) with several metrics will end up in a single =
UDP=20
> packet?
>=20

Not at this point.  I think that we can try to tackle metric packing in a =
later patch.

> So, if I've understood correctly, the potential attributes are now:
>=20
>   1. name:  unique(*) short string for metric (potential use: create a
> 	suitable filename for RRD storage)
>=20
>   2. host: the FQDN of the host from which the metric proports to come.
>=20
>   3. type: what kind of data is collected.
>=20
>   4. value: a (current/recent) measured value of this metric
>=20
>   5. title:  short name (potential use: a title of an RRD graph)
>=20
>   6. description: a longer string, giving more information (potential =
use:
> 	pop-up box in web-interface)
>=20
>   7. group: some indication of how this metric can be grouped with =
others.
>=20
>   8. units:  a suffix to each number (potential use: label in y-axis, =
suffix
> 	after static values)
>=20
>   9. slope: an indication of whether the data is a counter-, gauge- or
> 	static- metric.
>=20
>   10. tmax: indicate how regularly the metric should be updated.
>=20
>   11. dmax: how quickly a metric should be dropped when no further=20
> measurements are received.
>=20
> (*) The (name,host) ordered pairs must be unique for any one metric.
>=20
> Data packet contains 1.--4.
> Metadata packet contains 1.--3. and 5.--11.
>=20

Correct.

> Do we also want to have a timestamp when the metric was measured, or do =
we=20
> assume it was sufficiently close to "now"?
>=20

I haven't thought about this.  Not sure if it is necessary, but could be =
added later.

> If we're going to record metrics in base-units (bytes rather than =
megabytes)=20
>=20
> and let RRDTool do the scaling, then we should also include a flag =
about=20
> whether the SI scaling prefix should be in powers of 10 (kilo, Mega, =
...) or=20
>=20
> powers of 2 (kibi, Mebi, ...).  IIRC, RRDTool supports both.
>=20
> Perhaps we could add a bit-wise set of attributes one bit could be =
wither=20
> the=20
> metric should be scaled by powers-of-10 or powers-of-2.  The current =
slope=20
> value could be folded in with this attribute, or left separate.
>=20
> Did we come to any conclusion about the semantics of the group field? =
=20
> Something like a space separated list of groups.  Would membership of =
a=20
> group=20
> imply plotting together on the same graph, or somehow close together on =
a=20
> webpage?
>=20

I don't know what the answer to this question is as far as the web-frontend=
 goes.  But for gmond and the xml output the answer is pretty basic.  The =
built-in metrics do not have any group designation assigned to them.  This =
is mainly because there isn't a convenient way to add a group designation =
to the hardcoded array.  This will be resolved when the builtin metrics =
are ported to modules.  For modular metrics I have added a macro that =
allows the C code to easily designate one or more groups for a metric.  =
For the python interface a comma delimited list should be provided in the =
metric definition with the keyword GROUPS.  In the XML data, groups are =
represented by one or more <GROUP> tags within the <METRIC>.  The only =
thing that I don't like about this is that I have implemented a flexible =
way for a metric module to include any additional metadata for a metric.  =
However, the XML output of both gmond and gmetad aren't as flexible.  I'm =
not sure how to resolve that issue at this point.

Brad=20