mod_trace_output Files

/*************************/
/** Version : 1.1-beta  **/
/** Update : 04/05/2002 **/
/*************************/

You will find the following topics in this file:
-Description
-Module workflow
-Installation
-Usage

---------------------------------
Description
---------------------------------

The first idea was to be able to do statistics not only on page
viewed but also on the content viewed by users. One of the problems 
is that content can be generated by other modules and we don't have
programmatic access to them. There comes mod_trace_output.

Mod_trace_output will catch the content of the web pages sent by Apache
after they were generated.It works with most common modules, including 
mod_jk (wich is using the uri translate phase to set the r->handler 
to "jakarta-servlet" instead of the usual mechanism based on mime-types, 
this was quite a headach!).

It can store pages visited in files on a specified directory or in a specified 
MySQL database. This content can also be gzipped in order to use less disk 
space and/or less bandwith in case of remote MySQL database. Furthermore, 
mod_trace_output sends the data gzipped to browsers that support it.

The module is designed to run on LINUX with Apache 1.2.23 and 1.2.24. But my 
objective is to port it soon to Apache 2.0 (and maybe later to windows).

It is written in c, using the libmysqlclient (from MySQL), the libz (for 
gzip) libraries.

---------------------------------
Module workflow
---------------------------------
 
First, you have to know that mod_trace_output is creating a subrequest to 
have the job done and get the data back from this subrequest.

Mod_trace_output has a handler for the fixup phase, where it sets the 
r->handler to "trace-output", but only if the following conditions are met : 
the current request is not a subrequest of mod_trace_output handler 
(otherwise I have an endless loop!), the request is not for headers only, 
r->filename is not a directory (mod_dir has to find the rigth directory index 
file before we do something), if r->content_type is NULL : check 
for "HANLDER" in ToMimeTypeIn (see directives below) otherwise check for 
r->content_type in ToMimeTypeIn.

So if one of these condition is not met, the fixup handler will 
decline and won't set r->handler = "trace-output", and mod_trace_output 
won't handle the request.

If all the conditions are met, we have r->hanlder="trace-output", and, at 
the content handling phase, mod_trace_output handler is called first. 
It then creates a subrequest and adds a filter_callback function to it, 
so that it can get a copy of the data sent. Then it uses a little trick 
(many thanks to Ian McRae and to his mod_mmap_dynamic) and sets the 
client side connection to point to /dev/null, so that the subrequest doesn't 
actually send the data.

Then subrequest runs, the module gets the copy of the data, it gzips it, and 
sends it to the browser. The response created by the subrequest can be of 
arbitrary mime type. If the response's mime type is in the ToMimeTypesOut list 
(see directive ToMimeTypesOut below), it stores it either in files or in MySQL 
db, gzipped or not.

---------------------------------
Installation
---------------------------------

In previous version, mod_trace_output had no handler for the fixup phase, so the 
module needed to be executed first, but this is not a requirement anymore.

I'll give the installation steps if you compile mod_trace_output statically 
with Apache. But you can also use DSO and apxs to compile it out of Apache.

Here we go. Create a directory mod_trace_output in the modules directory of 
the source distribution of apache, copy all the files provided in the 
mod_trace_output release (mod_trace_output.c, mod_trace_output.h, filter.c, 
gzipcomp.c, mysql.c and MakeFile.tmpl), then add the following line in the configuration 
file.

AddModule modules/mod_trace_output/mod_trace_output.o

It uses libmysqlclient (.a or .so) so you need to set extra options in the 
Configuration file.<BR>Add -L/path/to/libmysqlclient/folder -lz -lmysqlclient 
(if you have libmysqlclient.so) or -L/path/to/libmysqlclient/folder/libmysqlclient.a 
(if you have libmysqlclient.a) to EXTRA_LDFLAGS.<BR>Add -I/path/to/mysql/include to 
EXTRA_INCLUDES.

Then compile apache as usual. See http://httpd.apache.org/ for more information 
on how to do it.

Mod_trace_output is now statically linked into your apache binary, you can 
configure it with several directives, see below.

---------------------------------
Usage
---------------------------------
The following directives cannot be set in a <Directory> or a <Location> tag,
so their value is for the entire server.

-ToMySQLInfo: take three arguments, the MySQL database name, user and password.
-ToMySQLHost: host of the MySQL database.
-ToMySQLPort: TCP-IP port to connect to the MySQL database. You need this only if the host is not localhost. 
-ToMySQLSocket: path to socket(ex:"/var/lib/mysql/mysql.sock") to connect to the MySQL database. You need 
		this only if the mysql.sock file is not at its default place.
-ToMySQLNonPersistent: flag to use non-persistent MySQL connections (values: ON/OFF). OFF is the best value 
		for performance.
-ToLog: flag to allow / disallow the log (values: ON/OFF).
-ToLogFile: file where to log mod_trace_output actions.

The following directives must be set in a <Directory> or a <Location> tag,
so their scope is the specified directory or location.

-ToMimeTypesIn: mime types of files requested that mod_trace_output will handle(separated by comas). 
You can also specify "HANDLER", if you want to handle the requests that have no content_type and 
just has a handler (like in the case of mod_jk). The default is  "", NO mime types are handled.
-ToMimeTypesOut: mime types of files that mod_trace_output will store (separated by comas) (default = "text/html").
-ToDataGroup: string used to group data in the database, for example by directory (default = directory path).
-ToStoreInMySQL: flag to allow / disallow storing of the pages in MySQL (values: ON/OFF).
-ToGzipInMySQL: flag to allow / disallow to gzip compress data in MySQL (values: ON/OFF).
-ToStoreInFiles: flag to allow / disallow storing of the pages in a specified directory (values: ON/OFF).
-ToGzipInFiles: flag to allow / disallow to gzip compress data in stored files (values: ON/OFF).
-ToDir: directory where to store pages (default /tmp).

These directives allow to specify different configuration for each location, you can set 
content from a location to a MySQL database, while storing the contents in gzipped html 
files for another location.

Example: 
ToMySQLInfo dbname user password
ToMySQLHost localhost
ToMySQLPort 3306
ToMySQLSocket /var/lib/mysql/mysql.sock
ToMySQLNonPersistent "OFF"
ToLogFile "/etc/httpd/logs/mod_trace_output_log"
ToLog "ON"

<Location /directory_sql>
	ToMimeTypesIn "text/html,HANDLER"
	ToMimeTypesOut "text/html"
	ToDataGroup "test NON gzipped"
	ToStoreInMySQL "ON"
	ToGzipInMySQL "OFF"
</Location>

<Location /directory_sql_gz>
	ToMimeTypesIn "text/php"
	ToDataGroup "test gzipped"
	ToStoreInMySQL "ON"
	ToGzipInMySQL "ON"
</Location>

<Location /directory_files>
	ToMimeTypesIn "text/plain"
	ToMimeTypesOut "text/plain"
	ToDir "/var/wasa_store/"
	ToStoreInFiles "ON"
	ToGzipInFiles "OFF"
</Location>

<Location /directory_files_gz>
	ToMimeTypesIn "text/html"
	ToDir "/var/wasa_store"
	ToStoreInFiles "ON"
	ToGzipInFiles "ON"
</Location>