[Ganglia-developers] rrdtool XDR Format

[Matt M. <ma...@cs...>]

Tobi et al-

first let me start by giving tobi and the rrdtool developers kudoes for 
building such a great piece of software.  i challenge anyone to find a 
better tool for storing and visualizing time series data.

i have a simple proposal to make rrdtool even better with what i think 
will be very little retooling of the rrdtool code.

to give some brief background, i manager the ganglia 
(http://ganglia.info/) distributed system monitoring program that 
currently runs on dozen of platforms.

there are two big limitations to rrdtool that really hurt: (a) rrds are 
platform-specific (non-portable) and (b) every rrd update requires an 
immediate sync to disk.

(a) it would be wonderful if i could move rrds around the network to 
different platforms and not deal with endianness problems.  i know that 
rrdtool allows you to "dump" and contents from one platform and 
"restore" it however xml is such a bloated communication format.  even 
more importantly, in the time is takes to export the data, move the data 
over the network, restore the data... the original data may have changed 
  (and i've burned unecessary CPU in the process parsing XML).

(b) it would also be wonderful if rrdtool supported in-memory databases. 
  the disk io cost of writing to a rrd is pretty significant.  it is not 
uncommon for ganglia to be monitoring thousands of hosts that update 
their data every 15 seconds or so.  that is a lot of disk io!

one workaround to this is to save multiple data points to memory before 
calling rrdtool "update" (although that reason to use rrdtool in the 
first place is to not have to implement storage of time series data 
yourself).

one solution that solves both of these problems is for me to use jrobin 
(http://jrobin.org/) which is an implementation of rrdtool in java.  the 
format of the jrobin rrds is portable and it supports in-memory rrds. 
however, i want the speed i get with C/C++ and don't want to implement 
portion of ganglia in C and others in Java.

here is my solution to problem (a) and (b) above.

leverage the power of the XDR: External Data Representation Standard 
(http://www.faqs.org/rfcs/rfc1832.html).  this standard has been around 
since the 1980s and serves as the communication format for RPC (Remote 
Procedure Calls).

this solution doesn't just solve these two problems, but it also opens 
up a number of new possibilities for rrdtool (not the least of which is 
rrdtool rpc services).

here are the steps to make this dream a reality.

1.  copy rrd_format.h to rrd_format.x (.x is the standard extension for 
   rpc/xdr protocol description files).

2.  make some slight modifications to rrd_format.x (i attached my final 
version of rrd_format.x to this email).  the only real changes that i 
needed to make are as follows...

2a.  change structure definitions from

typedef struct foo_t {
   ...
} foo_t;

to

struct foo_t {
   ...
};

2b. changed unival union definition from

typedef union unival {
     unsigned long u_cnt;
     rrd_value_t   u_val;
} unival;

to

enum unival_type {
    UNIVAL_COUNT,
    UNIVAL_VALUE
};

union unival switch (enum unival_type type)
{
   case UNIVAL_COUNT:
     unsigned long u_cnt;
   case UNIVAL_VALUE:
     rrd_value_t   u_val;
};

the 2b change is necessary because xdr protocol descriptions require 
that unions have a defined discriminant.  when you think about it, this 
makes sense... if i'm sending a union over the network or to a file or 
whereever xdr needs to know what the format of the data in union is 
(given that it has multiple choices).

2c. i added the line

typedef double       rrd_value_t;

and removed the include for "rrd.h" which had the definition for a 
rrd_value_t.

2d. i needed to change the contents of stat_head_t from

struct stat_head_t {
       /* Data Base Identification Section ***/
     char             cookie[4];          /* RRD */
     char             version[5];         /* version of the format */
     ...
};

to

struct stat_head_t {
     string           rrd_cookie<4>;          /* RRD */
     string           rrd_version<5>;         /* version of the format */
     ...
};


3.  run a protocol compiler (rpcgen) on this new protocol description 
file (rrd_format.x).

e.g.

% rpcgen rrd_format.x

rpcgen creates two files from rrd_format.x: rrd_format.h and 
rrd_format_xdr.c for you.

the generated rrd_format.h file is not much different than the original 
but it also has a number of XDR stub functions which are implemented in 
rrd_format_xdr.c.

if you take a look at the man page for xdr (man xdr) or visit
http://linux.ctyme.com/man/man3885.htm you will see the basic functions 
for XDR.  these functions are implemented on every operating system I 
know of (including Cygwin on Windows and 32bit/64bit platforms) and 
require only that link to a single library (-lxdr).

4. modify the rrdtool code to use these XDR stub functions

the rest of the code in rrdtool would need to call these stub functions 
instead of operating on rrd_t structures directly.  XDR will deal with 
serializing the structures you define in order to send them over the 
wire or to a file or to memory.

for example,

rrd_create() as it is written now builds a rrd_t structure (mallocing 
substructures) and then calls

int rrd_create_fn(char *file_name, rrd_t *rrd);

the rrd_create_fn walks the rrd structure and writes the data to the 
specified file using calls to fwrite() which saves the data in a 
non-portable format.

here is the new implementation of rrd_create_fn() using XDR...

int rrd_create_fn(char *filename, rrd_t *rrd)
{
   int rval,
   XDR xdrs;
   FILE *rrd_file = fopen(file_name,"wb"));

   if(!rrd_file)
     {
       rrd_set_error("creating '%s': %s",file_name,strerror(errno));
       free(rrd->stat_head);
       free(rrd->ds_def);
       free(rrd->rra_def);
       return(-1);
     }

   /* Create XDR stream */
   xdrstdio_create(&xdrs, rrd_file,  XDR_ENCODE);

   /* Returns TRUE on SUCCESS, FALSE on FAILURE */
   rval = xdr_rrd_t(xdrs, rrd);
   if(rval == FALSE)
     {
       rrd_set_error("a file error occurred while creating '%s'",file_name);
       xdr_destroy(xdrs);
       return(-1);
     }

   /* We're finished with the stream */
   xdr_destroy(xdrs);
   rrd_free(rrd);
   return(0);
}

the next function to change would be rrd_open() since it is used to open
databases for update, graph, etc.

here is the new rrd_open() in using XDR..

int
rrd_open(char *file_name, FILE **in_file, rrd_t *rrd, int rdwr)
{
     int rval;
     char *mode = NULL;
     XDR xdrs;

     rrd_init(rrd);
     if (rdwr == RRD_READONLY) {
#ifndef WIN32
         mode = "r";
#else
         mode = "rb";
#endif
     } else {
#ifndef WIN32
         mode = "r+";
#else
         mode = "rb+";
#endif
     }

     if (((*in_file) = fopen(file_name,mode)) == NULL ){
         rrd_set_error("opening '%s': %s",file_name, strerror(errno));
         return (-1);
     }

     /* Create the XDR stream */
     xdrstdio_create(&xdrs, in_file, XDR_DECODE);

     /* De-serialize the stream into a rrd_t structure */
     rval = xdr_rrd_t(xdrs, rrd);
     if(rval == FALSE)
       {
         rrd_set_error("'%s' is not an RRD file",file_name);
         free(rrd->stat_head);
         fclose(*in_file);
         return(-1);
       }

     return(0);
}

i kid you not... it's that easy.

if you are running the XDR stream in the XDR_ENCODE direction (as in 
rrd_create_fn() above) then you must have malloc'd all the deep 
structures of rrd_t.

*however*, if you are running the XDR stream in the XDR_DECODE direction 
(as in the rrd_open() above), you don't have to worry about mallocing 
memory at all since the XDR library handles it for you.

if you want to free the deep structures malloc'd in a rrd_t structure by 
XDR... all you need to do is run.

xdr_free( (xdr_proc_t)xdr_rrd_t, rrdptr );

the xdr_free function takes two arguments.  the function used to decode 
the data (xdr_rrd_t in this case) and a pointer to the data you want to 
free.

here is the man page entry for xdr_free()...

void
xdr_free(proc, objp)
        xdrproc_t proc;
        char *objp;

Generic freeing routine. The first argument is the XDR  routine  for 
the object  being  freed.  The  second  argument  is  a pointer to the 
object itself. Note: the pointer passed to this routine is not freed, 
but what it points to is freed (recursively).

so here is the new rrd_free() using XDR

void rrd_free(rrd_t *rrd)
{
   xdr_free(xdr_rrd_t, rrd);
}

easy.

i think you can see how this approach would really make the rrdtool code 
a lot cleaner and easier to understand.  it would also make databases 
completely portable... you can just copy them around in the raw to any 
platform that you want and they will just work.

100% backward compatibility

using this approach will make updating the rrdtool database format much 
easier in the future AND rrdtool will always be 100% backward compatible 
from this point forward.  to achieve this you simply need to alter the 
rrd_format.x that i attached to this email a bit more (i didn't want to 
alter it much from original in order to make it simpler to understand).

all we need to do is specify a rrd_versions enum

enum rrd_versions {
    RRD_VERSION_1,
    RRD_VERSION_2
};

and then define a rrd_t not as a structure but as a discriminant union 
as such

struct rrd_version_1_t {
   ...
};

struct rrd_version_2_t {
   ...
};

union rrd_t switch ( enum rrd_versions )
{
   case RRD_VERSION_1:
      rrd_version_1 version_1;
   case RRD_VERSION_2:
      rrd_version_2 version_2;
}


this means that all rrdtool data formats from here on out would be part 
of a single union.  that would give you 100% backward compatibility.

if a new rrdtool received data from an older rrdtool, it would know 
because the rrd_versions != RRD_VERSION_2.  each revision of the format 
would require that you create a conversion function that converts 
RRD_VERSION_1 data to RRD_VERSION_2, and RRD_VERSION_2 to RRD_VERSION_3, 
and so on.

this might sound like a lot of work but because all the stub functions 
are created by rpcgen you don't need to think about it.  you just define 
a new and better format and plop it into the protocol description file 
and let rpcgen worry about it.

by the way, talking about formats... a nice feature addition to rrdtool 
would be to allow user-defined meta-data in rrdtool databases in order 
to give some context to the data (which host is came from, 
application-specific data, etc).  it would be easy to add that to the 
rrdtool format without increasing the size of the databases (unless you 
had user-defined info to store).

this would allow me to collect data from distributed hosts into a 
central repository and have the database define itself (in the 
user-defined variable) as it moves through the system.

i do not know the development contribution process for rrdtool.  i'm 
willing to contribute the time to make this proposal a reality.

-matt

-- 
PGP fingerprint 'A7C2 3C2F 8445 AD3C 135E F40B 242A 5984 ACBC 91D3'

    They that can give up essential liberty to obtain a little
       temporary safety deserve neither liberty nor safety.
   --Benjamin Franklin, Historical Review of Pennsylvania, 1759