Re: [Exist-development] Cache logging [was: log entries]

[Andrzej J. T. <an...@ch...>]

Wolfgang:

> Mmmh, I was hoping that JMX could become the central interface for
> getting administrative information from the db - and to some extend
> even change configuration parameters at runtime. As I understand it,
> JMX provides a large number of options to use it without compromising
> security. Even if I don't enable JMX, I can still use jconsole on the
> same machine to get information from my running eXist.

JMX is great for ad-hoc, realtime access. Using it for runtime config changes is great.

In many cases it's just a lot easier and more efficient to go look in a log file, rather than fire up a JMX interface
tool.  With long distance, flakey and slow net connections (common in the healthcare space, down at the primary care
clinic levels), JMX just doesn't work well....downloading a log file and then perusing it locally at your leisure (ie.
eliminating the network instabilities) works better.  Not everyone is on high speed connections yet. ;-)

JMX doesn't obviate the need for longer term and historical logging though. I think we need both in the long run.

>> After the fact, how do you easily get a picture of how the caching was doing over a historical period of time?
> 
> Normally I'm only interested in the maximum cache use over a longer
> period of time. As long as everything runs fine, I usually don't check
> the caches. The main information the caching statistics provide is if
> a certain cache can become a bottleneck under high load. The
> historical development of the cache doesn't really tell me that much -
> but I'm happy to learn.

OK...makes sense to me to some degree....but often you can't see the high load condition when it happens in
production...so having this info  (optionally) stream to a log would seem to make some sense. In fact, if you are in a
high load situation, getting response out of a JMX interface can be an issue, made worse if the network infrastructure
is flakey (as it can be in our client situations).

> Ok, but then we could also just assign a special logger to the cache
> logs. I *really* prefer to have all logs going into the same log file.
> Advantage: I can see all log messages in context. Having all logs in
> the same file makes it easy to relate events to each other. I can
> easily see what query causes an increase in memory consumption or
> cache space ;-)

I can see the value of that, and the changes that I put in place the other day actually take us very close to what
you're asking for, since I've assigned a special logger category to the cache logs at this stage ("org.exist.statistics"
instead of the class name). Though in that scenario, I think opening up two logs and looking at timestamps to
co-ordinate the cause/effect is pretty simple to do, at least in the case of the cache performance logging (though it is
harder for more general purpose debugging as Dannes noted).

How about if I change the default appender for the cache log entries back to the exist.core (ie. exist.log file)
appender in the log4j.xml file?  That would basically bring us back to the same logging output/destination file to what
we had before my change, but.....

I could now easily use our application's config file customization infrastructure (ant targets that dynamically modify
things like log4j.xml, conf.xml and other key config files), to route the cache logs to a different file
(statistics.log) for just our application implementation.

Say the word, Wolfie, and I shall "make it so" (said in my best Pickard voice. ;-) ).

-- 
Andrzej Taramina
Chaeron Corporation: Enterprise System Solutions
http://www.chaeron.com