log4perl-devel

[log4perl-devel] Setting parameters globally in Log::Log4perl::Appender::DBI

[Alex M. <ale...@cl...>]

Hello,

I wonder if there is a way, using Log::Log4perl::Appender::DBI, to set a
value for a logger, which will be inserted at all subsequent log
statements.

Maybe an exemple will help me to be understandable :

################################################################################

my $logger = Log::Log4perl::get_logger();

my $customer = get_cust_id();

do_something_dangerous() || $logger->error("Big problem !", $customer);
# ...
do_something_uncredibly_dangerous() || $logger->error("Aaargh !", $customer);

################################################################################

As you can see, I pass the same value as a second argument to several 
statements, breaking the rule of once-and-only-once.

I'm looking for a way to affect values onces to my $logger, and then
this values will be inserted in the database for each call to logging
statements.

Any idea ?

-- 
Alex Marandon
CLARISYS Informatique
http://clarisys.fr

Re: [log4perl-devel] Setting parameters globally in Log::Log4perl::Appender::DBI

[Kevin G. <ke...@go...>]

> I wonder if there is a way, using Log::Log4perl::Appender::DBI, to set a
> value for a logger, which will be inserted at all subsequent log
> statements.

See 'Mapped Diagnostic Context' in the perldoc 
http://log4perl.sourceforge.net/releases/Log-Log4perl/docs/html/Log/Log4perl.html#mapped_diagnostic_context_(mdc) 



> my $logger = Log::Log4perl::get_logger();
> 
> my $customer = get_cust_id();

#stash the value in a global location for the logger to find
Log::Log4perl::MDC->put('customer', $customer);

> do_something_dangerous() || $logger->error("Big problem !");
> # ...
> do_something_uncredibly_dangerous() || $logger->error("Aaargh !");

And set up your appender to look like something this:

  log4j.appender.DBAppndr.sql         = \
        insert into logtable           \
        (custid, message) \
        values (?,?)
     log4j.appender.DBAppndr.params.1 = %X{customer}

Does that answer your question?



-- 
Happy Trails . . .

Kevin M. Goess
(and Anne and Frank)
904 Carmel Ave.
Albany, CA  94706
(510) 525-5217

Log::Log4perl::Appender::DBI questions (Was: Re: [log4perl-devel] Setting parameters globally in Log::Log4perl::Appender::DBI)

[Alex M. <ale...@cl...>]

On Mon, Oct 20, 2003 at 01:28:04PM -0700, Kevin Goess wrote:
> >I wonder if there is a way, using Log::Log4perl::Appender::DBI, to set a
> >value for a logger, which will be inserted at all subsequent log
> >statements.
> 
> See 'Mapped Diagnostic Context' in the perldoc 
> http://log4perl.sourceforge.net/releases/Log-Log4perl/docs/html/Log/Log4perl.html#mapped_diagnostic_context_(mdc) 
> 
> 
> >my $logger = Log::Log4perl::get_logger();
> >
> >my $customer = get_cust_id();
> 
> #stash the value in a global location for the logger to find
> Log::Log4perl::MDC->put('customer', $customer);
> 
> >do_something_dangerous() || $logger->error("Big problem !");
> ># ...
> >do_something_uncredibly_dangerous() || $logger->error("Aaargh !");
> 
> And set up your appender to look like something this:
> 
>  log4j.appender.DBAppndr.sql         = \
>        insert into logtable           \
>        (custid, message) \
>        values (?,?)
>     log4j.appender.DBAppndr.params.1 = %X{customer}
> 
> Does that answer your question?

That's great !

Well, I have a couple of more questions :)

1) I'd like to have optional parameters to logging statements. The
documentation says that if it lacks a parameter, it will just be
replaced by the undef value. But I got a different behaviour.

2) I would like to do some preprocessing before the value are inserted
in the database. I can manage to do that on %x's values by overriding
the log() method (and I think I could use this trick to solve the
problem #1). But I don't know how to do it on the values in
%X. I'd like to do that because when a value from %X is not initialized,
the string '[undef]' is inserted, but when the corresponding database
column is of type INTEGER, obviously the SQL statements fails. I'd like
to have a way to put a zero or a NULL value instead of the string
'[undef]'.

Here are pieces of code in order to explain my problems :


################################################################################
package MyAppender;
use base 'Log::Log4perl::Appender::DBI';

sub log {
    # Here I'm subclassing to modify value from the %x's parameters
    my $self = shift;
    my %p = @_;
    # Categories are shown as Perl packages
    $p{log4p_category} =~ s/\./::/g;
    Log::Log4perl::Appender::DBI::log($self, %p);
}

1;

################################################################################

package My::Cute::Package;

my $log_conf = <<END;

log4perl.appender.dbLog=MyAppender
log4perl.appender.dbLog.datasource=dbi:Pg:dbname=test
log4perl.appender.dbLog.username=
log4perl.appender.dbLog.password=
log4perl.appender.dbLog.sql=INSERT INTO log (category, level, customer, message, details) VALUES (?, ?, ?, ?, ?)

log4perl.appender.dbLog.params.1 = %c
log4perl.appender.dbLog.params.2 = %p
log4perl.appender.dbLog.params.3 = %X{customer}

log4perl.appender.dbLog.usePreparedStmt = 1
log4perl.appender.dbLog.layout = Log::Log4perl::Layout::NoopLayout
log4perl.appender.dbLog.warp_message = 0

log4perl.logger=DEBUG, dbLog

END

use Log::Log4perl;
Log::Log4perl->init(\$log_conf);
my $logger = Log::Log4perl::get_logger();

# This one fails with the following message because the database wants an
# integer (or a NULL value).
#
# DBD::Pg::st execute failed: ERROR:  pg_atoi: error in "[undef]": can't parse
# "[undef]" at /usr/local/lib/perl5/site_perl/5.8.0/Log/Log4perl/Appender/DBI.pm

$logger->error(
    "Something wrong.", 
    "A problem has happened because of this and that."
);


# %X{customer} is initialized
Log::Log4perl::MDC->put('customer', 1234);

# This one works

$logger->error(
    "Something wrong.", 
    "A problem has happened because of this and that."
);

# This one fails with the message :
#
#  execute called with 4 bind variables, 5 needed at
#  /usr/local/lib/perl5/site_perl/5.8.0/Log/Log4perl/Appender/DBI.pm line 96.

$logger->error(
    "Some other thing wrong."
);

################################################################################

-- 
Alex Marandon
CLARISYS Informatique
http://clarisys.fr

[log4perl-devel] MDC and [undef] (was Re: Log::Log4perl::Appender::DBI questions)

[Kevin G. <ke...@go...>]

> 2) ...when a value from %X is not initialized,
> the string '[undef]' is inserted, but when the corresponding database
> column is of type INTEGER, obviously the SQL statements fails. I'd like
> to have a way to put a zero or a NULL value instead of the string
> '[undef]'.

Hey, Mike, that's this bit from MDC.pm:

     if(exists $MDC_HASH{$key}) {
         return $MDC_HASH{$key};
     } else {
         return "[undef]";
     }

Do you remember the rationale for returning a string instead of just 
undef?  Maybe we should at least make that configurable?  What do you think?


--
Happy Trails. . .

Kevin M. Goess
(and Anne and Frank)
904 Carmel Ave.
Albany, CA  94706
(510)525-5217

[log4perl-devel] Re: MDC and [undef] (was Re: Log::Log4perl::Appender::DBI questions)

[Mike S. <msc...@ao...>]

Kevin Goess wrote on 10/21/2003, 10:39 PM:

 > Do you remember the rationale for returning a string instead of just
 > undef?  Maybe we should at least make that configurable?  What do you
 > think?

Think the rationale was that if the configuration file specifies a 
layout with an MDC field and it's not defined in the code, 
'uninitialized variable' warnings are going to be all over the place. If 
it causes problems somewhere else, it's more consistent to just return 
undef. Fixed and checked in ...

-- 
-- Mike
Mike Schilli
m...@pe...

[log4perl-devel] Re: MDC and [undef] (was Re: Log::Log4perl::Appender::DBI questions)

[Alex M. <ale...@cl...>]

On Wed, Oct 22, 2003 at 11:18:50AM -0700, Mike Schilli wrote:
> Kevin Goess wrote on 10/21/2003, 10:39 PM:
> 
>  > Do you remember the rationale for returning a string instead of just
>  > undef?  Maybe we should at least make that configurable?  What do you
>  > think?
> 
> Think the rationale was that if the configuration file specifies a 
> layout with an MDC field and it's not defined in the code, 
> 'uninitialized variable' warnings are going to be all over the place. If 
> it causes problems somewhere else, it's more consistent to just return 
> undef. Fixed and checked in ...

But now it fails with the message : 

DBD::Pg::st execute failed: ERROR: pg_atoi: zero-length string at
/usr/local/lib/perl5/site_perl/5.8.0/Log/Log4perl/Appender/DBI.pm line 96.

When I look at the query using DBI->trace(2), I see that it actually
tries to insert a zero-length string instead of the expected NULL value.

I've tried to figure out where the undef value is changed to a
zero-length string in the code, but didn't success for now.

-- 
Alex Marandon
CLARISYS Informatique
http://clarisys.fr

Re: [log4perl-devel] Re: MDC and [undef] (was Re: Log::Log4perl::Appender::DBI questions)

[Kevin G. <ke...@go...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

|>it causes problems somewhere else, it's more consistent to just return
|>undef. Fixed and checked in ...
|
| But now it fails with the message :
|
| DBD::Pg::st execute failed: ERROR: pg_atoi: zero-length string at
| /usr/local/lib/perl5/site_perl/5.8.0/Log/Log4perl/Appender/DBI.pm line 96.

Alex, sorry for the delay.  I'm not reproducing your problem.  The
following code runs fine, the two MDC columns end up with '' and 0
respectively.  Does it work for you?  Can you send sample code that
produces your error?


- ------------------------------------------------

use Log::Log4perl;
use warnings;
use strict;
use DBI;

my $pguser = 'postgres';
my $pwd = 'xxx';

my $dbh = DBI->connect('DBI:Pg:dbname=testdb',$pguser, $pwd,
{ RaiseError => 1, PrintError => 1 });

$dbh->do('DROP TABLE log4perltest');

my $stmt = <<EOL;
~    CREATE TABLE log4perltest (
~      message   char(128),
~      mdc_char char(16),
~      mdc_int int,
~      last_field char(1)
~  )
EOL

$dbh->do($stmt);

#creating a log statement where bind values 1,3,5 and 6 are
#calculated from conversion specifiers and 2,4,7,8 are
#calculated at runtime and fed to the $logger->whatever(...)
#statement

my $config = <<EOT;
#log4j.category = WARN, DBAppndr, console
log4j.category = WARN, DBAppndr
log4j.appender.DBAppndr             = Log::Log4perl::Appender::DBI
log4j.appender.DBAppndr.datasource = DBI:Pg:dbname=testdb
log4j.appender.DBAppndr.username  = $pguser
log4j.appender.DBAppndr.password = $pwd
log4j.appender.DBAppndr.sql = \\
~   insert into log4perltest \\
~   (message, mdc_char, mdc_int,last_field) \\
~   values (?,?,?,?)
log4j.appender.DBAppndr.params.2 = %X{fubar}
log4j.appender.DBAppndr.params.3 = %X{fubar}
log4j.appender.DBAppndr.params.4 = x

log4j.appender.DBAppndr.warp_message=0

#noop layout to pass it through
log4j.appender.DBAppndr.layout    = Log::Log4perl::Layout::NoopLayout

EOT

Log::Log4perl::init(\$config);

#Log::Log4perl::MDC->put('fubar', 'xxx');


my $logger = Log::Log4perl->get_logger();

$logger->warn('warning message'); #xxx






- --
- --
Happy Trails. . .

Kevin M. Goess
(and Anne and Frank)
904 Carmel Ave.
Albany, CA  94706
(510)525-5217
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Netscape - http://enigmail.mozdev.org

iD8DBQE/n2dy4g4/Tl71vUkRAtD3AJ9gYfx0NOJ5ZiJnzcMiyRWPi/zhLACdHglr
mm3LaTcInZ8iNMxFhSt3qV0=
=+EgU
-----END PGP SIGNATURE-----

Re: [log4perl-devel] Re: MDC and [undef] (was Re: Log::Log4perl::Appender::DBI questions)

[Alex M. <ale...@cl...>]

On Tue, Oct 28, 2003 at 11:08:35PM -0800, Kevin Goess wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> |>it causes problems somewhere else, it's more consistent to just return
> |>undef. Fixed and checked in ...
> |
> | But now it fails with the message :
> |
> | DBD::Pg::st execute failed: ERROR: pg_atoi: zero-length string at
> | /usr/local/lib/perl5/site_perl/5.8.0/Log/Log4perl/Appender/DBI.pm line 96.
> 
> Alex, sorry for the delay.  I'm not reproducing your problem.  The
> following code runs fine, the two MDC columns end up with '' and 0
> respectively.  Does it work for you?  Can you send sample code that
> produces your error?

Hello,

Well, your code produce the same error like mine. Maybe this is a matter
of PostgreSQL version, what's yours ? Mine is 7.3.3.

One can read this in 7.3 release notes :

 "An empty string ('') is no longer allowed as the input into an integer
  field. Formerly, it was silently interpreted as 0."

http://www.postgresql.org/docs/7.3/interactive/release.html#AEN23568

-- 
Alex Marandon
CLARISYS Informatique
http://clarisys.fr

Re: [log4perl-devel] Re: MDC and [undef]

[Kevin G. <ke...@go...>]

> Well, your code produce the same error like mine. Maybe this is a matter
> of PostgreSQL version, what's yours ? Mine is 7.3.3.
> 
> One can read this in 7.3 release notes :
> 
>  "An empty string ('') is no longer allowed as the input into an integer
>   field. Formerly, it was silently interpreted as 0."

Well, darn, mine is a venerable 7.1.3, nice find.  If postgres is 
requiring at least a 0 but only for numeric fields, not for all fields, 
I don't see a way for the MDC stuff to have a 'conditional default' 
value, it seems to me the solution is that you have to set the MDC value 
to 0 as soon as you start up.  What do you think?


-- 
Happy Trails . . .

Kevin M. Goess
(and Anne and Frank)
904 Carmel Ave.
Albany, CA  94706
(510) 525-5217

Re: [log4perl-devel] Re: MDC and [undef]

[Alex M. <ale...@cl...>]

On Wed, Oct 29, 2003 at 08:26:03AM -0800, Kevin Goess wrote:
> >Well, your code produce the same error like mine. Maybe this is a matter
> >of PostgreSQL version, what's yours ? Mine is 7.3.3.
> >
> >One can read this in 7.3 release notes :
> >
> > "An empty string ('') is no longer allowed as the input into an integer
> >  field. Formerly, it was silently interpreted as 0."
> 
> Well, darn, mine is a venerable 7.1.3, nice find.  If postgres is 
> requiring at least a 0 but only for numeric fields, not for all fields, 
> I don't see a way for the MDC stuff to have a 'conditional default' 
> value, it seems to me the solution is that you have to set the MDC value 
> to 0 as soon as you start up.  What do you think?

I think this not coherent to insert an empty string when we have an
undefined value. An empty string is a defined value. That's an undef
value which should be inserted (which is converted to NULL by DBI).

I guess that will be a problem when printing to a log file because
of warnings about undefined value.

That's why it would be nice to provide a way for the users to do some
kind of preprocessing on MDC values in order to handle problems which
are specific to some kind of appenders.

I see that MDC.pm has an undef value (which was recently '[undef]')
but I don't see where it's changed to an empty string. By subclassing the
place where undef values are changed to empty string, I could certainly
get rid of my problem.

-- 
Alex Marandon
CLARISYS Informatique
http://clarisys.fr

Re: [log4perl-devel] Re: MDC and [undef]

[Kevin G. <ke...@go...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

| I see that MDC.pm has an undef value (which was recently '[undef]')
| but I don't see where it's changed to an empty string. By subclassing the
| place where undef values are changed to empty string, I could certainly
| get rid of my problem.

It's changed in the sprintf() in PatternLayout::render(), since this:

	sprintf("%s", undef)

produces the empty string, not undef.

| That's why it would be nice to provide a way for the users to do some
| kind of preprocessing on MDC values in order to handle problems which
| are specific to some kind of appenders.

Tell me if I'm just being dense (which is more than likely), but is it
not true that you *do* have complete control the MDC values, you can
init the troublesome one via MDC->put('somenumeric',0) right next to
where you call Log4perl::init(), or even in a BEGIN{} block somewhere.
Right?

What am I missing?  Do you have another suggestion?  I'm open to ideas....



- --
Happy Trails. . .

Kevin M. Goess
(and Anne and Frank)
904 Carmel Ave.
Albany, CA  94706
(510)525-5217
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Netscape - http://enigmail.mozdev.org

iD8DBQE/oLiu4g4/Tl71vUkRAqnNAJ4/ixvTH9XfA/W2hdbfvR44Vg6IxgCgmdC8
Bc4MC3qYUR2ViCUcXLHav1w=
=+Xkz
-----END PGP SIGNATURE-----

Re: [log4perl-devel] Re: MDC and [undef]

[Alex M. <ale...@cl...>]

On Wed, Oct 29, 2003 at 11:07:26PM -0800, Kevin Goess wrote:
> | I see that MDC.pm has an undef value (which was recently '[undef]')
> | but I don't see where it's changed to an empty string. By subclassing the
> | place where undef values are changed to empty string, I could certainly
> | get rid of my problem.
> 
> It's changed in the sprintf() in PatternLayout::render(), since this:
> 
> 	sprintf("%s", undef)
> 
> produces the empty string, not undef.

Thanks I'll have a look at it. I'll inform you of my research :)

> | That's why it would be nice to provide a way for the users to do some
> | kind of preprocessing on MDC values in order to handle problems which
> | are specific to some kind of appenders.
> 
> Tell me if I'm just being dense (which is more than likely), but is it
> not true that you *do* have complete control the MDC values, you can
> init the troublesome one via MDC->put('somenumeric',0) right next to
> where you call Log4perl::init(), or even in a BEGIN{} block somewhere.
> Right?

Let me explain a bit the context. My application sends email. But it
also does other things (like providing the user a web interface). I have
a table where I insert informations about sent mail. When there is a
problem while sending a mail, I log it in a file, but also in an error
table, and I put in the error table the id of the failed mail. So, the
integer column in the error table references primary keys of the
sent mail table. As I said, error could happen when doing something else
than sending a mail, so the integer column of my error table can hold
NULL values.

If I put a default value (0 for example) it could work if :
 - I remove the referecing integrity constraint
 or
 - I create a fake sent-mail which have an id of 0 and put if clauses
   everywhere in my code in order to not display information about this
   fake sent-mail

Both of these solutions could work as quick workaround, but it's
not clean.

[log4perl-devel] Appender::DBI lacking parameters bug (was Re: Log::Log4perl::Appender::DBI questions)

[Kevin G. <ke...@go...>]

> 1) I'd like to have optional parameters to logging statements. The
> documentation says that if it lacks a parameter, it will just be
> replaced by the undef value. But I got a different behaviour.
> 
> log4perl.appender.dbLog.sql=INSERT INTO log (category, level, customer, message, details) VALUES (?, ?, ?, ?, ?)
> log4perl.appender.dbLog.params.1 = %c
> log4perl.appender.dbLog.params.2 = %p
> log4perl.appender.dbLog.params.3 = %X{customer}
> 
> #  execute called with 4 bind variables, 5 needed at
> #  /usr/local/lib/perl5/site_perl/5.8.0/Log/Log4perl/Appender/DBI.pm line 96.
> 
> $logger->error(
>     "Some other thing wrong."
> );

That's a plain old bug you found.  The way it's coded now, missing 
parameters will be replaced by undef ONLY if they come before the 
highest declared parameter number in the config, i.e. if you changed the 
config file to look like this you should get what you want.

  log4perl.appender.dbLog.sql=INSERT INTO log (category, level, message, 
details, customer) VALUES (?, ?, ?, ?, ?)
  log4perl.appender.dbLog.params.1 = %c
  log4perl.appender.dbLog.params.2 = %p
  log4perl.appender.dbLog.params.5 = %X{customer}

A bad assumption on my part in the code.  Fixing it is going to take 
some thought, so try that workaround rather than waiting for me to patch.


On another note, does the test t/034DBI.t pass with your version of the 
postgres dbd?  I noticed it fails with mine, it looks like there was a 
bug in the dbd that was only fixed in March 
(http://gborg.postgresql.org/pipermail/dbdpg-general/2003-March/000038.html).


--
Happy Trails. . .

Kevin M. Goess
(and Anne and Frank)
904 Carmel Ave.
Albany, CA  94706
(510)525-5217