Thread: Re: [log4perl-devel] A question on disk usage... (fwd)

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Interesting claim by a module called Log::Cabin on CPAN :).

-- Mike

Mike Schilli
m...@pe...

---------- Forwarded message ----------
From: Mike Schilli <m...@pe...>
To: Robert Hicks <rob...@ma...>
Cc: m...@pe...
Subject: Re: A question on disk usage...
Date: Thu, 19 Apr 2007 15:19:30 -0700 (PDT)

On Thu, 19 Apr 2007, Robert Hicks wrote:

> Log::XXX provides a selection of the features of Log::Log4perl but
> with a focus on reduced disk IO. Just calling 'use Log::Log4perl'
> results in hundreds of stat calls to the file system. If you have a
> shared file system with many nodes running perl scripts at once, this
> could result in a significant decrease in performance.

Interesting, never heard of this claim before. Let's look at the facts:

Perl does a lot of stat() calls because of the way it searches for modules
in the %INC path:

   $ strace perl -MFoo::Bar::Baz -e1 2>&1 | grep stat64 | wc -l
        54

So for a *non-existent* module, you get 54 stat() calls!

Let's try the same thing with Log::Log4perl:

   $ strace perl -MLog::Log4perl -e1 2>&1 | grep stat64 | wc -l
        151

Definitely more, but I've never seen that to be a problem. However,
if you run short scripts on NFS in quick succession, I can see that to
be a problem. Which we could resolve easily by offering something like

    use Log::Log4perl qw(:fast);

implementing lazy loading. As a matter of fact, I'm gonna put that on
the list for one of the next releases.

> After implementing this module we were able to cut up to 70,000 stat/
> open calls per second on our NFS.

That's an odd claim, because it says nothing about how many stat calls
there were before and after. If it got improved by only 10%, then the
new module is useless. And, as I've already mentioned, there can be
a negative impact if you have many calls to short scripts -- in which
case you should be running a persistent perl interpreter like mod_perl
or pperl and get rid of module loading I/O entirely.

> Of course, this module doesn't
> currently support all the features of Log::Log4perl, but many of the
> most comment ones are implemented.

Would be interesting to know how much functionality is actually implemented
and if it's any different than the 20 other log modules on CPAN :).
Do you happen to know more about where it's available?

Would you be ok if I forwarded this email to the L4p mailing list?

Thanks for the note!

-- Mike

Mike Schilli
m...@pe...