#62 logging overhead - time for a different logging system?
There was recently some discussion on the jsbsim mailing list regarding the performance overhead imposed by logging, and it was shown how significantly jsbsim may be slowed down due to logging.
This is in fact a problem that has also been striking FlightGear every now and then during the last couple of years, which is also documented by numerous mailing list postings and forum discussions, as well as related feature requests [1].
And recently this is something that has become awfully obvious due to the issues related to TriangleIntersect
and NaN/CullVisitor log messages flooding the console [2] [3], basically rendering FlightGear unusable thereby-usually requiring a restart.
Often the only "working" workaround is to either throttle logging by reducing the granularity or completely disabling it.
However, this may not always be a viable option.
Basically, these are architectural shortcomings: log messages don't necessarily have to be processed sequentially in the scope of the main loop.
Instead, they could easily be processed in a different thread, where they could be written to a memory buffer or even a socket to facilitate remote logging.
In fact, even in a single-threaded context, just writing them to a memory buffer would address this issue efficiently.
Many mainstream software packages support the notion of "remote logging", often there are even corresponding standards so that existing infrastructure/code could be leveraged (think syslog).
In the long run, the most promising approach would probably be to use a client/server architecture, where a logging application may optionally become a log server itself, so that clients could connect to it in order to read log messages. This could be rather easily achieved without too much effort required, in fact just using "telnet" for this purpose would probably be sufficient and have the advantage of clients being widely available and standardized, as well as the necessary server code being readily available, too (plib netchat, which is also used by fgfs for its props interface).
Given that a client/server approach to logging would inevitably imply that client and server are running as different processes, this would automatically have the advantage of the server not being slowed down by output the client has to echo to the console, in fact a client/server approach would basically boil down to writing to a memory buffer.
In addition, there'd also be the advantage of remote logging itself, which would mean that users would be able to leverage help from community members by enabling remote logging during a debugging session.
[1] http://sourceforge.net/tracker/index.php?func=detail&aid=1910344&group_id=583&atid=350583
[2] http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg19994.html
[3] http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg20263.html
This may also be helpful: http://en.wikipedia.org/wiki/Syslog
well, from a debugging point of view it's worth to note that there may be scenarios where the executable that's being debugged terminates/segfaults at some point, in such a situation it would be sort of unfortunate if the logserver -including all relevant log info- was also terminated, only because it is run as part of the logging process.
So from that point of view, it might be worthwhile to use a conventional syslog approach, where a standalone/separate log server is running that could be used by arbitrary applications that simply send log messages via UDP to the server.
The server could still be inspected via standard software such as a telnet client or conventional syslog frontends, that exist for both-linux and windows (e.g. see http://www.sysrose.com or http://www.softpanorama.org/Logs/Syslog/syslog_for_windows.shtml )
On the other hand, it's important to keep in mind that this scenario, where the logging process terminates, isn't in any way different from the current situation.
The workaround we are using now is to either pipe the log output to some file or have the process itself write to some log file.
So, this would also be an option in a logserver scenario - be it by having the logserver write locally to a log file to buffer all log messages, or alternatively by optionally having clients buffer all log messages, so that they can still be inspected even if the server terminates.
It's probably also worth to point out that there are several existing multi-platform logging frameworks/libraries that already provide syslog support: http://freshmeat.net/projects/log4cpp/
The following open source (LGPL), cross-platform logging library with remote logging support may be interesting:
http://freshmeat.net/projects/rlog/
http://www.arg0.net/rlog
http://code.google.com/p/rlog/
rlog is for message logs, not high-rate data output, as far as I can tell.
If syslog support and performance are the primary goals, one should probably check out "pantheios":
"Pantheios is an Open Source C/C++ Logging API library, offering an optimal combination of 100% type-safety, efficiency, genericity and extensibility."
"Pantheios was forked from a previous logging architecture, proprietary to Synesis Software, in 2005, and has been used in commercial projects in Australia, Europe and the US since that time. In one project it serves a middleware suite that handles 10s of millions of financial transactions per week, wherein it has an unmeasurably low impact on performance when logging (of a given severity level) is switched off. (It's also decidedly quick with logging switched on, of course ...). In another, it's rock-solid reliability has served 10^11 logging events in a transactional financial system in the heart of New York without a flutter."
http://freshmeat.net/projects/pantheios/
http://www.pantheios.org/
http://www.pantheios.org/performance.html#sweet-spot
http://www.pantheios.org/essentials.html