Qb4j
From qb4j
qb4j (pronounced 'cube for j') is the first open source multi-dimensional logging framework for Java coupled with an OLAP analytical platform.
qb4j doesn't create log files. It creates log cubes (or cube logs).
Logs are there to be analyzed
The end product of the qb4j's logging framework is a multidimensional hierarchical matrix called cube log vs traditional log files. Information is best analyzed in the multidimensional databases, not in text files. Superior multidimensional databases are commercially available, the vendors are well known: Oracle/Hyperion Essbase, Microsoft Analytical Services etc. There is also a Java open source thingy called mondrian that is probably the way to go for qb4j.
It is a waste of human energy to look at the log files as text!
Why is it sometimes difficult to analyze your data - there could be only two reasons for this.
- you don't have all the information in the cube
- you are using wrong tools to access the cube
What is cube? Cube is the OLAP term for a multidimensional matrix. OLAP stands for Online Analytical Processing. One way to think of it is a big sparse multidimensional matrix, where each dimension is a hierarchy. Sounds abstract, but that what they are using to analyze data in business world. You know why? Because it is THE BEST way to analyze data. And that what we need our log files for, right? We want to analyze them.
Not the flat file should be the end product of the logging mechanism, but the OLAP cube that you can query. The most common analytical query is called drill-through, that is you slice and dice your cube by dimensions.
Those dimensions and hierarchies are familiar entities in Java. You just need to think about the log record as a data on the intersection of multiple hierarchies, which are
Log record divided into dimensions
Time Dimension
Year___________________________2009
_Month________________________ Apr
__Hour_________________________4
___Minute______________________21
____Second_____________________3
_____Nanosecond________________325
Threads Dimension
Domain___________________qb4j.sourceforge.net
_Computer________________name:vasya-pc1
_________________________ip:192.168.0.1
___Process_________________main class: net.sf.Sample
___________________________started: 2009 Apr 4:20:2:1032
___________________________arguments: String[]:{}
_____Thread_________________ main
_____________________________isAlive: yes
_____________________________isDaemon: no
Exceptions Dimension
Library
_Lib-Version
__Package
___Class
____Exception
Developers Dimension
(who wrote the line of code in the class)
Department ________________Engineering/Essbase
__Developer ________________Vasya Pupkin
____________________________title: Software Engineer
____________________________manager: Peter Z. Hopkin
Methods Dimension
(method being executed)
Library________________myproject
_Lib-Version________________1.2
___Package________________com.myproject.utils
____ Class_________________MyHashMap
_______Method_______________MyHashMap(String, String)
____________________________args: name="x", val="y"
_____Line_______________________33: this.map=new HashMap();
Users Dimension
(os user that initiated the action)
User_____________________________vasya-pc1\vasya
Values Dimension [Local Variables]
(our values)
Library
__Lib-Version
___Package
_____Class
________Instance Object
___________Method
______________Variable
Values Dimension
[Object and static class properties]
Library__________________________myproject
_Lib-Version_____________________1.2
__Package________________________com.myproject.utils
___Class_________________________MyHashMap
____Instance_Object______________MyHashMap(1)
_____Property____________________map
_________________________________old_val:_null
_________________________________new_val:_HashMap()
You should start seeing by now how each level in each dimension has its own information and how comprehensive the log record becomes?
Now we have it in source control, right? Yet another dimension
SVC Dimension
SVN_rep_____https://qb4j.svn.sourceforge.net/svnroot/qb4j
_dir_________________________src/com/myproject
__file_______________________MyHashMap.java
___revision__________________4
____line_____________________33
The most often missed dimension!
Configuration Dimension
All Properties
_System.getProperties()_______{"classpath=blabla...",_""}
_log4j.properties_____________{"logger.bla.level=DEBUG",...}
...
Explanation Dimension
Using comments from the source files. This will allow a nice google-finance like view of your thread's runtime with your comments popping out as news articles.
__/*Comments*/____________author:Vasya Pupkin
__________________________text: /*slicing and dicing the analytical cube*/"
__//TODO__________________author:Vasya Pupkin
__________________________text:"//todo need to think of a better way to analyze my log files"
Adding application specific dimensions
And let's put it inside a servlet, so you have another dimension of users - your web users
Web Users Dimension
Realm________________________tomcat
_User________________________manager
And web requests
Web Requests Dimension
Session______________HttpSessionImpl_
_____________________started:_10Jan2009_10:10am
_Request________________HttpRequestImpl
________________________urlparams=reply.jsp?message_id=12
________________________ip=192.168.1.10
Remember - the right column is just one big log record...
And all of this happens at the same time, in our case it was somebody who has logged in to the web server from 192.168.1.10, sent a request and that caused invocation of a method that changes some values and is written by some developer in some thread at the certain time.
Our log records are strings and java objects on the intersection of all of those hierarchical dimensions.
THE ULTIMATE GOAL
For qb4j
Converting all the log files of this world to log cubes (or cube logs?)
For you
- Support qb4j so we can have our first release sooner. Currently accepting architecture ideas, source code, feedback and $ donations.
- Creation of the PERFECT CUBE for your organization.
