Menu
▾
▴
log4perl-devel
|
From: Martin J. E. <mar...@ea...> - 2006-03-03 14:16:52
|
Hi,
I am using log4perl extensively in a number of daemon processes and running out
of file descriptors. To investigate I ran:
perl -e 'use Log::Log4perl qw(get_logger
:levels);Log::Log4perl->init_and_watch("/etc/log4.conf", 10);sleep 600;'
and then ran lsof to see what open files there were in this process. I was
surprised to see every log file named in my log4.conf file under
"appender.XX.filename" had been created and was open. This may be my
misunderstanding of the way log4perl works but it is undesirable for me so I
wondered if either a) these log files should not be open or b) if my conf file
was wrong. Given:
log4perl.logger.Server = WARN
log4perl.logger.Server.Markets.Close = DEBUG, A1
log4perl.appender.A1=Log::Dispatch::File
log4perl.appender.A1.filename=/tmp/markets_close.log
log4perl.appender.A1.mode=append
log4perl.appender.A1.layout=Log::Log4perl::Layout::PatternLayout
log4perl.appender.A1.layout.ConversionPattern=%d %p> %F{1}:%L %M - %m%n
log4perl.appender.A1.Threshold=INFO
log4perl.logger.Server.CGI.Post = DEBUG, A3
log4perl.appender.A3=Log::Dispatch::File
log4perl.appender.A3.filename=/tmp/cgi_post.log
log4perl.appender.A3.mode=append
log4perl.appender.A3.layout=Log::Log4perl::Layout::PatternLayout
log4perl.appender.A3.layout.ConversionPattern=%d %p> %F{1}:%L %M - %m%n
log4perl.logger.Server.Queued = DEBUG, A5
log4perl.appender.A5=Log::Dispatch::File
log4perl.appender.A5.filename=/tmp/queued.log
log4perl.appender.A5.mode=append
log4perl.appender.A5.layout=Log::Log4perl::Layout::PatternLayout
log4perl.appender.A5.layout.ConversionPattern=%d %p> %F{1}:%L %M - %m%n
why are A1.filename, A3.filename and A5.filename all created and open when I
run the above Perl. I would have expected A5.filename to be opened when I did
get_logger("Server::Queued") but no files other than log4.conf to be open when
all I've done is init_and_watch.
Any ideas.
Martin
--
Martin J. Evans
Easysoft Ltd, UK
http://www.easysoft.com
|
|
From: Mike S. <m...@pe...> - 2006-03-03 19:39:12
|
On Fri, 3 Mar 2006, Martin J. Evans wrote:
> I was surprised to see every log file named in my log4.conf file
> under "appender.XX.filename" had been created and was open.
That's exactly how Log::Log4perl's file appender works. It opens all
log files at init(), not when you obtain a logger.
I wonder how it is possible to run out of file descriptors with only
three open files per process, though. Are you running hundreds of
daemons simultaneously?
If opening the log files at startup imposes a problem, you could write
your own log file appender that opens files at log() instead. If you
look at the implementation of Log::Log4perl::Appender::File you'll see
that it's actually quite simple.
-- Mike
Mike Schilli
m...@pe...
> This may
> be my misunderstanding of the way log4perl works but it is
> undesirable for me so I wondered if either a) these log files should
> not be open or b) if my conf file was wrong. Given:
>
> log4perl.logger.Server = WARN
> log4perl.logger.Server.Markets.Close = DEBUG, A1
> log4perl.appender.A1=Log::Dispatch::File
> log4perl.appender.A1.filename=/tmp/markets_close.log
> log4perl.appender.A1.mode=append
> log4perl.appender.A1.layout=Log::Log4perl::Layout::PatternLayout
> log4perl.appender.A1.layout.ConversionPattern=%d %p> %F{1}:%L %M - %m%n
> log4perl.appender.A1.Threshold=INFO
>
> log4perl.logger.Server.CGI.Post = DEBUG, A3
> log4perl.appender.A3=Log::Dispatch::File
> log4perl.appender.A3.filename=/tmp/cgi_post.log
> log4perl.appender.A3.mode=append
> log4perl.appender.A3.layout=Log::Log4perl::Layout::PatternLayout
> log4perl.appender.A3.layout.ConversionPattern=%d %p> %F{1}:%L %M - %m%n
>
> log4perl.logger.Server.Queued = DEBUG, A5
> log4perl.appender.A5=Log::Dispatch::File
> log4perl.appender.A5.filename=/tmp/queued.log
> log4perl.appender.A5.mode=append
> log4perl.appender.A5.layout=Log::Log4perl::Layout::PatternLayout
> log4perl.appender.A5.layout.ConversionPattern=%d %p> %F{1}:%L %M - %m%n
>
> why are A1.filename, A3.filename and A5.filename all created and open when I
> run the above Perl. I would have expected A5.filename to be opened when I did
> get_logger("Server::Queued") but no files other than log4.conf to be open when
> all I've done is init_and_watch.
>
> Any ideas.
>
> Martin
> --
> Martin J. Evans
> Easysoft Ltd, UK
> http://www.easysoft.com
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
> _______________________________________________
> log4perl-devel mailing list
> log...@li...
> https://lists.sourceforge.net/lists/listinfo/log4perl-devel
>
|
|
From: Martin J. E. <mar...@ea...> - 2006-03-06 09:16:36
|
Mike,
Thanks for the reply
see below:
Mike Schilli wrote:
> On Fri, 3 Mar 2006, Martin J. Evans wrote:
>
>
>>I was surprised to see every log file named in my log4.conf file
>>under "appender.XX.filename" had been created and was open.
>
>
> That's exactly how Log::Log4perl's file appender works. It opens all
> log files at init(), not when you obtain a logger.
why?
Perhaps this highlights my misunderstanding then.
I don't understand why it does this.
I thought calling get_logger would actually open the file
and init_and_watch would simply parse the conf file, check it
was OK and watch the conf file.
If I'd called get_logger(x::y::z) then I could understand
the log file associated with x::y::z and x::y being opened
but If I call get_logger(x::y) then why would the log
file associated with x::y::z be opened.
> I wonder how it is possible to run out of file descriptors with only
> three open files per process, though. Are you running hundreds of
> daemons simultaneously?
Perhaps I should have been clearer. I am running out of file descriptors
(but within a single process, not system wide) and it was that
which prompted me to look at what files I've got open. By the
time you add a file descriptor for the perl binary and every module
you've "use"ed, then every shared object you need it amounts to quite a
bit (easily more than 64).
I did not expect log4perl to have created and opened the files I have
not called get_logger for (or at least those below it). I wasn't
saying the 3 that log4perl open are the one and only cause of my
problem but they do include 2 I did not expect to be created
and opened.
> If opening the log files at startup imposes a problem, you could write
> your own log file appender that opens files at log() instead. If you
> look at the implementation of Log::Log4perl::Appender::File you'll see
> that it's actually quite simple.
I don't want files created and opened which I'm not going to log to.
In my mind if I've not called get_logger("x::y") then I'm
not going to log to the file that is defined by x::y (unless I've
called get_logger(x::y::z). I will look
at Appender but it seems to get ONLY the files opened which I'm
actually going to use in the current process I need a separate
conf file for each process which only defines the appenders
I intend using this time around. If that is the way it is
- fair enough - I can live with that.
As I said, perhaps I've misunderstood appenders and the conf file
but I'm "currently" at a loss to understand why a file is created
and opened when there is no likelihood of anything being written
to it. Perhaps you could put me right on any misunderstanding
I have.
Martin
>>This may
>>be my misunderstanding of the way log4perl works but it is
>>undesirable for me so I wondered if either a) these log files should
>>not be open or b) if my conf file was wrong. Given:
>>
>>log4perl.logger.Server = WARN
>>log4perl.logger.Server.Markets.Close = DEBUG, A1
>>log4perl.appender.A1=Log::Dispatch::File
>>log4perl.appender.A1.filename=/tmp/markets_close.log
>>log4perl.appender.A1.mode=append
>>log4perl.appender.A1.layout=Log::Log4perl::Layout::PatternLayout
>>log4perl.appender.A1.layout.ConversionPattern=%d %p> %F{1}:%L %M - %m%n
>>log4perl.appender.A1.Threshold=INFO
>>
>>log4perl.logger.Server.CGI.Post = DEBUG, A3
>>log4perl.appender.A3=Log::Dispatch::File
>>log4perl.appender.A3.filename=/tmp/cgi_post.log
>>log4perl.appender.A3.mode=append
>>log4perl.appender.A3.layout=Log::Log4perl::Layout::PatternLayout
>>log4perl.appender.A3.layout.ConversionPattern=%d %p> %F{1}:%L %M - %m%n
>>
>>log4perl.logger.Server.Queued = DEBUG, A5
>>log4perl.appender.A5=Log::Dispatch::File
>>log4perl.appender.A5.filename=/tmp/queued.log
>>log4perl.appender.A5.mode=append
>>log4perl.appender.A5.layout=Log::Log4perl::Layout::PatternLayout
>>log4perl.appender.A5.layout.ConversionPattern=%d %p> %F{1}:%L %M - %m%n
>>
>>why are A1.filename, A3.filename and A5.filename all created and open when I
>>run the above Perl. I would have expected A5.filename to be opened when I did
>>get_logger("Server::Queued") but no files other than log4.conf to be open when
>>all I've done is init_and_watch.
>>
>>Any ideas.
>>
>>Martin
|
|
From: Mike S. <m...@pe...> - 2006-03-07 03:53:16
|
On Mon, 6 Mar 2006, Martin J. Evans wrote:
> I thought calling get_logger would actually open the file
> and init_and_watch would simply parse the conf file, check it
> was OK and watch the conf file.
>
> If I'd called get_logger(x::y::z) then I could understand
> the log file associated with x::y::z and x::y being opened
> but If I call get_logger(x::y) then why would the log
> file associated with x::y::z be opened.
Loggers and appenders are orthogonal concepts in Log4perl. If you get
a logger, nothing happens with the appender. Only when the logger decides
to fire, the appender comes into play (according to the configuration).
The standard Log::Log4perl::Appender::File appender in Log4perl opens
its files at init() because it's usually not a problem having a handful
of files open, even if they're not written to. Alternatively, it could
open the files at log() time, which would impose a small penalty on
the appenders with every log().
> > I wonder how it is possible to run out of file descriptors with only
> > three open files per process, though. Are you running hundreds of
> > daemons simultaneously?
>
> Perhaps I should have been clearer. I am running out of file descriptors
> (but within a single process, not system wide) and it was that
> which prompted me to look at what files I've got open. By the
> time you add a file descriptor for the perl binary and every module
> you've "use"ed, then every shared object you need it amounts to quite a
> bit (easily more than 64).
64 file descriptors per process sounds really low. On my linux box at home it
is
$ ulimit -n
1024
> I don't want files created and opened which I'm not going to log to.
> In my mind if I've not called get_logger("x::y") then I'm
> not going to log to the file that is defined by x::y (unless I've
> called get_logger(x::y::z). I will look
> at Appender but it seems to get ONLY the files opened which I'm
> actually going to use in the current process I need a separate
> conf file for each process which only defines the appenders
> I intend using this time around. If that is the way it is
> - fair enough - I can live with that.
If you really need to save those 3 file descriptors, I'd recommend writing
your own file appender that opens the file at log() time, on demand. It's really
a 20-liner.
-- Mike
Mike Schilli
m...@pe...
> As I said, perhaps I've misunderstood appenders and the conf file
> but I'm "currently" at a loss to understand why a file is created
> and opened when there is no likelihood of anything being written
> to it. Perhaps you could put me right on any misunderstanding
> I have.
>
> Martin
>
> >>This may
> >>be my misunderstanding of the way log4perl works but it is
> >>undesirable for me so I wondered if either a) these log files should
> >>not be open or b) if my conf file was wrong. Given:
> >>
> >>log4perl.logger.Server = WARN
> >>log4perl.logger.Server.Markets.Close = DEBUG, A1
> >>log4perl.appender.A1=Log::Dispatch::File
> >>log4perl.appender.A1.filename=/tmp/markets_close.log
> >>log4perl.appender.A1.mode=append
> >>log4perl.appender.A1.layout=Log::Log4perl::Layout::PatternLayout
> >>log4perl.appender.A1.layout.ConversionPattern=%d %p> %F{1}:%L %M - %m%n
> >>log4perl.appender.A1.Threshold=INFO
> >>
> >>log4perl.logger.Server.CGI.Post = DEBUG, A3
> >>log4perl.appender.A3=Log::Dispatch::File
> >>log4perl.appender.A3.filename=/tmp/cgi_post.log
> >>log4perl.appender.A3.mode=append
> >>log4perl.appender.A3.layout=Log::Log4perl::Layout::PatternLayout
> >>log4perl.appender.A3.layout.ConversionPattern=%d %p> %F{1}:%L %M - %m%n
> >>
> >>log4perl.logger.Server.Queued = DEBUG, A5
> >>log4perl.appender.A5=Log::Dispatch::File
> >>log4perl.appender.A5.filename=/tmp/queued.log
> >>log4perl.appender.A5.mode=append
> >>log4perl.appender.A5.layout=Log::Log4perl::Layout::PatternLayout
> >>log4perl.appender.A5.layout.ConversionPattern=%d %p> %F{1}:%L %M - %m%n
> >>
> >>why are A1.filename, A3.filename and A5.filename all created and open when I
> >>run the above Perl. I would have expected A5.filename to be opened when I did
> >>get_logger("Server::Queued") but no files other than log4.conf to be open when
> >>all I've done is init_and_watch.
> >>
> >>Any ideas.
> >>
> >>Martin
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
> _______________________________________________
> log4perl-devel mailing list
> log...@li...
> https://lists.sourceforge.net/lists/listinfo/log4perl-devel
>
|
|
From: Kevin G. <cp...@go...> - 2006-03-07 16:51:22
|
> The standard Log::Log4perl::Appender::File appender in Log4perl opens > its files at init() because it's usually not a problem having a handful > of files open, even if they're not written to. Alternatively, it could > open the files at log() time, which would impose a small penalty on > the appenders with every log(). There's also the matter of reliability. It would be pretty annoying to have your program run for an hour, die of a fatal error, and then have the error message vanish into the ether because the program didn't have permissions to write to the log file. By opening the files at the beginning, log4perl ensures that it will actually be able to write to each file if it every needs to. -- Happy Trails . . . Kevin M. Goess |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X