log4perl-devel

[log4perl-devel] appender/dispatcher that writes to per-category files

[Mike S. <m...@pe...>]

From: Jonathan Swartz <sw...@po...>
To: log...@li...
Subject: appender/dispatcher that writes to per-category files
Date: Thu, 1 Nov 2007 17:00:36 -0700

I'd like to write simultaneously to a central log file, and to a
separate log file for each category. e.g. A log to category Foo.Bar
would go to app.log and Foo.Bar.log. I want to do this automatically
rather than having to configure each category separately.

Before I set out to create this, I was curious if anyone else has
done this and has advice, or if there is an existing appender/
dispatcher that does this (couldn't find anything on cpan).

Seems like one of the potential problems will be reaching a maximum
number of open file handles, given the large number of possible
categories. So I might have to maintain a "cache" of open
filehandles, closing old ones as needed when reaching a limit.

Feedback appreciated.

Re: [log4perl-devel] appender/dispatcher that writes to per-category files

[Mike S. <m...@pe...>]

On Sat, 3 Nov 2007, Mike Schilli wrote:

> I'd like to write simultaneously to a central log file, and to
> a separate log file for each category. e.g. A log to category Foo.Bar
> would go to app.log and Foo.Bar.log. I want to do this automatically
> rather than having to configure each category separately.

Interesting problem ... do you know ahead of time which categories
you're gonna run into? A script could call all kinds of Log4perl-enabled
modules at runtime, adding new categories on-the-fly.

-- Mike

Mike Schilli
m...@pe...

> Before I set out to create this, I was curious if anyone else has done
> this and has advice, or if there is an existing appender/ dispatcher
> that does this (couldn't find anything on cpan).
>
> Seems like one of the potential problems will be reaching a maximum
> number of open file handles, given the large number of possible
> categories. So I might have to maintain a "cache" of open
> filehandles, closing old ones as needed when reaching a limit.
>
> Feedback appreciated.
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> log4perl-devel mailing list
> log...@li...
> https://lists.sourceforge.net/lists/listinfo/log4perl-devel
>

Re: [log4perl-devel] appender/dispatcher that writes to per-category files

[Jonathan S. <sw...@po...>]

> On Sat, 3 Nov 2007, Mike Schilli wrote:
>
>> I'd like to write simultaneously to a central log file, and to
>> a separate log file for each category. e.g. A log to category Foo.Bar
>> would go to app.log and Foo.Bar.log. I want to do this automatically
>> rather than having to configure each category separately.
>
> Interesting problem ... do you know ahead of time which categories
> you're gonna run into? A script could call all kinds of Log4perl- 
> enabled
> modules at runtime, adding new categories on-the-fly.
>

Right, I expected that new categories would keep cropping up. Why  
would it be a problem for an appender to open new filehandles on the  
fly? Other than worrying about maximum # of filehandles?

Jon

Re: [log4perl-devel] appender/dispatcher that writes to per-category files

[Mike S. <m...@pe...>]

On Mon, 5 Nov 2007, Jonathan Swartz wrote:

> Right, I expected that new categories would keep cropping up. Why
> would it be a problem for an appender to open new filehandles on the
> fly? Other than worrying about maximum # of filehandles?

Categories are added at init() time, not at log() time. If you add a new
category to Log4perl, you have to run init().

-- Mike

Mike Schilli
m...@pe...

Re: [log4perl-devel] appender/dispatcher that writes to per-category files

[Jonathan S. <sw...@po...>]

> On Mon, 5 Nov 2007, Jonathan Swartz wrote:
>
>> Right, I expected that new categories would keep cropping up. Why
>> would it be a problem for an appender to open new filehandles on the
>> fly? Other than worrying about maximum # of filehandles?
>
> Categories are added at init() time, not at log() time. If you add  
> a new
> category to Log4perl, you have to run init().

Huh? I can add a category at runtime like

     my $log = Log::Log4perl->get_logger('foo.bar.baz');

without having declared foo.bar.baz at init() time.

I think you mean that categories are associated with appenders at init 
() time, right? But I'm not looking to create a ton of different  
appenders. I'm looking to create a single appender that handles all  
categories, and automatically writes to the right filename based on  
the category passed to log().

Jon

Re: [log4perl-devel] appender/dispatcher that writes to per-category files

[Mike S. <m...@pe...>]

On Tue, 6 Nov 2007, Jonathan Swartz wrote:

> >> Right, I expected that new categories would keep cropping up. Why
> >> would it be a problem for an appender to open new filehandles on
> >> the fly? Other than worrying about maximum # of filehandles?
> >
> > Categories are added at init() time, not at log() time. If you add
> > a new
> > category to Log4perl, you have to run init().
>
> Huh? I can add a category at runtime like
>      my $log = Log::Log4perl->get_logger('foo.bar.baz');
> without having declared foo.bar.baz at init() time.

Ah, I see, we're talking about different things. I was talking about
adding categories and associate appenders with them. You're adding
them to the hierarchy and rely on category inheritance.

> I think you mean that categories are associated with appenders at init
> () time, right? But I'm not looking to create a ton of different
> appenders. I'm looking to create a single appender that handles all
> categories, and automatically writes to the right filename based on
> the category passed to log().

So you want a custom appender that dynamically changes the file it's
writing to. Since the appender's log() function gets 'log4p_category' in
its %param hash, this should be straight-forward to implement.

Sounds like an interesting idea, I'm looking forward to it!

-- Mike

Mike Schilli
m...@pe...

Re: [log4perl-devel] appender/dispatcher that writes to per-category files

[Jonathan S. <sw...@po...>]

> So you want a custom appender that dynamically changes the file it's
> writing to. Since the appender's log() function gets  
> 'log4p_category' in
> its %param hash, this should be straight-forward to implement.
>
> Sounds like an interesting idea, I'm looking forward to it!
>

Heh, no doubt. :)

My main worry is number of open file handles. Especially with  
mod_perl, since a lot of those handles will be opened individually in  
the child processes. At the same time, I don't want to have to open  
and close each handle every time. Any idea what kind of limits on  
number of open handles I can expect to hit on a reasonable Unix system?

Re: [log4perl-devel] appender/dispatcher that writes to per-category files

[Mike S. <m...@pe...>]

On Fri, 9 Nov 2007, Jonathan Swartz wrote:

> My main worry is number of open file handles. Especially with
> mod_perl, since a lot of those handles will be opened individually in
> the child processes. At the same time, I don't want to have to open
> and close each handle every time. Any idea what kind of limits on
> number of open handles I can expect to hit on a reasonable Unix
> system?

Depends. The easiest way to find out is this one:

    #!/usr/bin/perl -w
    use strict;
    my @handles;
    my $count;

    while(1) {
	open my $fh, "</etc/passwd" or die "died at $count";
	push @handles, $fh;
	$count++;
    }

prints

    died at 1020 at ./t line 7.

so that's it :). I would suggest that you find a compromise between
good performance and not running out of file handles by having the
appender maintain a configurable pool of file handles that it keeps open.

-- Mike

Mike Schilli
m...@pe...

Re: [log4perl-devel] appender/dispatcher that writes to per-category files

[Kevin M. G. <cp...@go...>]

Mike Schilli wrote:
> On Fri, 9 Nov 2007, Jonathan Swartz wrote:
> 
>> My main worry is number of open file handles. Especially with
>> mod_perl, since a lot of those handles will be opened individually in
>> the child processes. Any idea what kind of limits on
>> number of open handles I can expect to hit on a reasonable Unix
>> system?
> 
> Depends. The easiest way to find out is this one:
...
> 
> prints
> 
>     died at 1020 at ./t line 7.
> 
> so that's it :). 


I think he was looking for the total number of open file handles per
system, not per process. Try this:

	cat /proc/sys/fs/file-max