[Qmail-scanner-general]Mail statistics on qmail/qmailscanner log files

[Philip C. <pb...@hp...>]

I have been looking for a tool to give me good mail receipt/rejection
stats for my qmail/qmail-scanner installation.  I reject, bounce, or tag
mail based on RBLs, policies, uvscan, perscanner, and spamassassin.  The
only qmail scanner stats system I have found is Qmail-Scanner Statics,
http://sourceforge.net/projects/qss/, and it only addresses virus
stats based on the quarantine log.  There is also a qmail monitoring
system for MRTG from Inter7, http://www.inter7.com/mrtg.html  It tracks
a lot of good qmail stats but only the allows/denies data it collects is
of interest to me for this project.

Finding these inadequate, I decided to write my own stats tool.  I
wrote a parser for mailstats.csv which has fairly complete data. 
Parsing the qmail-smtpd log file for accepts and rejects by block list
should also be easy.  However I am now at an impasse on issues of where
to store the summary data and how much to store.  So I thought I would
ask the list.  

The data storage possibilities are endless but the common options are
round robin arrays (RRAs), MRTG data files, mysql tables, and good old
fashioned text files.  I have looked at the sendmail oriented mailstats,
http://staff.cie.uce.ac.uk/~id001869/mailstats/, and like what it
presents.  It uses a mix of text files and MRTG data files to store the
data. I shudder at the thought of returning to MRTG now that I have a
Cricket installation to do my monitoring.  I like mysql because I'm very
comfortable in SQL.  I like RRAs because there are lots of tools to
manipulate them, and they automatically age out and summarize data, but
I have very little experience with them outside of Cricket.

Also, how much data should we keep.  If you want to grow it
indefinitely, mysql is a good tool.  If you want to automatically age it
out, RRAs are a natural choice.  

I am curious to know what others would want to see in such a tool in
terms of data storage mechanism, the fineness of detail to store (by
hour, day, week, month, etc.) and the length of time that data needs to
be around.  Opinions on the report itself are welcome as well.  I like
what I see in mailstats, but I am open to suggestions.

By the way I'm working in perl.  At least that much has been decided.
;-)

TIA,
Philip