qb4j (pronounced 'cube for j') is the first open source multi-dimensional logging framework for Java coupled with an OLAP analytical platform.
qb4j doesn't create log files. It creates log cubes (or cube logs).
Logs are there to be analyzed
The end product of the qb4j's logging framework is a multidimensional hierarchical matrix called cube log vs traditional log files. Information is best analyzed in the multidimensional databases, not in text files. Superior multidimensional databases are commercially available, the vendors are well known: Oracle/Hyperion Essbase, Microsoft Analytical Services etc. There is also a Java open source thingy called mondrian that is probably the way to go for qb4j.
It is a waste of human energy to look at the log files as text!
Why is it sometimes difficult to analyze your data - there could be only two reasons for this.
- you don't have all the information in the cube
- you are using wrong tools to access the cube
What is cube? Cube is the OLAP term for a multidimensional matrix. OLAP stands for Online Analytical Processing. One way to think of it is a big sparse multidimensional matrix, where each dimension is a hierarchy. Sounds abstract, but that what they are using to analyze data in business world. You know why? Because it is THE BEST way to analyze data. And that what we need our log files for, right? We want to analyze them.
Not the flat file should be the end product of the logging mechanism, but the OLAP cube that you can query. The most common analytical query is called drill-through, that is you slice and dice your cube by dimensions.
Those dimensions and hierarchies are familiar entities in Java. You just need to think about the log record as a data on the intersection of multiple hierarchies, which are
Log record divided into dimensions
___Process_________________main class: net.sf.Sample
___________________________started: 2009 Apr 4:20:2:1032
(who wrote the line of code in the class)
__Developer ________________Vasya Pupkin
____________________________title: Software Engineer
____________________________manager: Peter Z. Hopkin
(method being executed)
____________________________args: name="x", val="y"
_____Line_______________________33: this.map=new HashMap();
(os user that initiated the action)
Values Dimension [Local Variables]
[Object and static class properties]
You should start seeing by now how each level in each dimension has its own information and how comprehensive the log record becomes?
Now we have it in source control, right? Yet another dimension
The most often missed dimension!
Using comments from the source files. This will allow a nice google-finance like view of your thread's runtime with your comments popping out as news articles.
__________________________text: /*slicing and dicing the analytical cube*/"
__________________________text:"//todo need to think of a better way to analyze my log files"
Adding application specific dimensions
And let's put it inside a servlet, so you have another dimension of users - your web users
Web Users Dimension
And web requests
Web Requests Dimension
Remember - the right column is just one big log record... And all of this happens at the same time, in our case it was somebody who has logged in to the web server from 192.168.1.10, sent a request and that caused invocation of a method that changes some values and is written by some developer in some thread at the certain time.
Our log records are strings and java objects on the intersection of all of those hierarchical dimensions.
THE ULTIMATE GOAL
Converting all the log files of this world to log cubes (or cube logs?)
- Support qb4j so we can have our first release sooner. Currently accepting architecture ideas, source code, feedback and $ donations.
- Creation of the PERFECT CUBE for your organization.