#2 Large Audit Logs

[Don Grodecki]

We have the requirement to log and archive potentially massive amounts of information. For example it is often necessary to log all queries and all responses. Both of these are available in XML form in OpenHRE, and so this should be a "simple" matter of using log4j with some sort of XML Appender to write out all of this XML to a log file, along with axiliary information, such as userid, date, and time. log4j could be configured to start up a new log file every day, and leave the old files around for archiving.

One could imagine a LoggingTransformer that could be inserted into a pipeline, which would echo all of its input to its output, but at the same time write to a log file. The problem with this is that the log file now becomes a critical resource, and processes will block until they get their chance at the log file.

We want some elasticity in the process, which begs the use of queues. OpenHRE recently gained a MuleTransformer (not yet tested) which should be able to not only echo its input to its output, but also write its input to a Mule queue. From there, asynchronously, Mule should be able to empty the queue and process its data, by, for example, using log4j to write it to a log file.

Mule would be a separate /sbin/service from tomcat. Eventually both might be under control of the same JMX console, perhaps the one integrated into JBoss.

Note that the queing system used must be able to guarantee delivery, so if Mule's internal queuing system is not this robust, to preserve messages through a machine failure, for instance, then a more robust JMS system must be used, again perhaps the one already integrated into JBoss.