Re: [Wonder-disc] WOLogging

[David T. <dav...@cl...>]

Bon Jour :-)

> Well, with - say - 10 instances, I think your asynchronouse ('delayed
> write') solution may generate non-neglegible overhead, coming from:
> competition for an exclusive table lock, memory (database cache, socket
> buffers), extra IPC (to log writer, to database), CPU cache hit ratio
> degradation, ...
>
So is your conclusion that no one should use log4j? I think you missed 
the point: avoiding synchronous logging (aka writing every request 
statistik directly with jdbc to the database) means that the logging 
stuff is very very fast.

If you write the SQL down to a file and once an hour or once a day or 
once every 10 minutes or once ever 10 seconds you execute it on a 
database it will not be a big deal. You can even do this on a slow, old 
ugly server which is just used for the statistics. You see my point: 
its adjustable. There is no real difference if apache is logging or if 
log4j is logging if both log to a file in the filesystem in the first 
place.

And in the end we could write the sql to the database just before its 
needed, so completely the same situation if we ran a ruby script over a 
large large logfile, dont you think?

> But this is not a good way of measuring the overhead. Under this test
> scenario you have db cache hit ratio 100%, same fo CPU cache hit 
> ration,
> and no competition for the exclusive table lock. You could at least run
> this test from as many processes, as there are in production, but 
> that's
> not really enough, as in production the logging module must compete for
> resources with other parts of the system.
>
No, the database is out of scope here because the db is only filled 
ever 10 seconds or 10 minutes or 10 hours or once a day. Its just about 
logging to a file in the first place.

> It is extremely hard to create realistic performace test scenarios. 
> This is
> one of the reasons why we came up with WOLogging, which is designed to 
> have
> overhead as low as possible, and meant to run in production.
And i am pretty sure its not much less or much more than using log4j to 
write to a file. Socket communication is maybe (well, IMHO its 
definitely) slower than writing to a file and has more overhead because 
WOLogging write to a socket and then (through apache) to a file. But 
again, we are talking about small, aceptable overheads.

> Obviously, with a http load generator run from another machine(s).
> Applicable to both solutions.
>
> Another issue is that you did not mention: your solution does not 
> measure
> the total time it takes to process requests. We perceive it as a 
> serious
> drawback. If appache is the bottleneck, your solution will not catch 
> this.
> (Yes, we had such case, this is why we upgraded to apache 2.0).
>
It helps you only if you provide the time WO needed to generate the 
response in your customized 'descriptForResponse', which is not too 
difficult.

You should mention the possibility that the firewall, which comes after 
the webserver can also be a bottleneck and then there is the router and 
then there is maybe a lot of other stuff...

database - appserver - webserver - firewall - router - and so on

So this is also not a point. I prefer to check the units independently 
and instead of

'database - appserver - webserver'

i like to test

'database' (executing SQL logged from the WOApp)

'database - appserver' using our logging stuff and connecting to the 
app directly

'database - appserver - webserver' connecting to the app via webserver

But of course everyone checks performance differently, its a very 
complex situation.

>> I think its great to have different way of logging, each one
>> can use what suits better its problem.
>
> I couldn't agree more, but the users should be given full information.
>
No on prevents you from writing a book about performance measuring, 
IPC, mach kernel in general, log4j overhead using File logger instead 
of /dev/null, socket IO compared to filesystem IO :-)

In serious: i said 'synchronous is sometimes slow' which in our case is 
correct. Asynchronous is faster, in our case also correct. WOLogging is 
not better nor is our stuff better, its just a different approach and 
if you want to test it i would be interested which one is faster, i bet 
avoiding the TCP overhead is faster and if you can prove that your 
solution is faster i pay the beer next time we are both on a wocoa :-)

Have a nice day,


David