log4perl-devel

[log4perl-devel] Structured logging in log4perl

[Dmitry B. <ic...@in...>]

Hi there!

I was searching the mailing-list history but didn't find any topics related to the subject I'd like to discuss. Maybe I didn't look close enough, sorry if so.
Anyway, the subject I want to discuss is how to use log4perl for structured logging.
There are a couple of modules on CPAN related to this task, like for example Log::Log4perl::Layout::GELF, Log::Log4perl::Appender::Fluent or Log::Message::JSON. But different approaches implemented there gave me the impression that there is no a general convenient way of doing structured logging in log4perl.
Yes, you can pass an arbitrary list of parameters to the logger and it really depends on the appender if it can work with that or not,  Log::Log4perl::Appender::DBI is a good example of how to do that. But what if I want to log my data to database, log file and lets say ElasticSearch at the same time. The problem here is that all three appenders expect data in different formats, File appender expects to get a string, DBI - a list of parameters and ElasticSearch - JSON. This is the place where you get stuck and has to choose something one.
The Log::Message::JSON module, as I understand it, is trying to address the problem using objects which get serialized to JSON in appenders expecting stings. But calling a special function (logmsg) every time you log something is quite a hassle and this is only JSON, what if I want to use XML? Here we go, there is no way in log4perl you can format you data before they are passed to appender. Layouts are meant to render messages, not arbitrary data. I also was trying to play with the warp_message appender option but it's result is always an array ref which doesn't work with File appender for example and it's probably not the best way to solve the problem.
To don't introduce new entities I'd say that Layouts is a proper place to make such transformations but it is just not how things work now.

Can someone suggest a good solution for that or maybe an idea?
What are you thoughts in general on the problem, should it be solved in log4perl?
Thanks.
--
Dmitry
e-mail: ic...@in...

Re: [log4perl-devel] Structured logging in log4perl

[Lee <le...@gm...>]

Sorry to reply to such a long and thoughtful e-mail with a URI, but do 
you think being given access to the raw data would give you sufficient 
means to format it?

http://log4perl.sourceforge.net/releases/Log-Log4perl/docs/html/Log/Log4perl/FAQ.html#e8a46

Could you then forward the formatted data to an appender of your 
choice/via appenders listed in the config?

Lee

On 14/03/2013 17:25, Dmitry Bigunyak wrote:
> Hi there!
>
> I was searching the mailing-list history but didn't find any topics related to the subject I'd like to discuss. Maybe I didn't look close enough, sorry if so.
> Anyway, the subject I want to discuss is how to use log4perl for structured logging.
> There are a couple of modules on CPAN related to this task, like for example Log::Log4perl::Layout::GELF, Log::Log4perl::Appender::Fluent or Log::Message::JSON. But different approaches implemented there gave me the impression that there is no a general convenient way of doing structured logging in log4perl.
> Yes, you can pass an arbitrary list of parameters to the logger and it really depends on the appender if it can work with that or not,  Log::Log4perl::Appender::DBI is a good example of how to do that. But what if I want to log my data to database, log file and lets say ElasticSearch at the same time. The problem here is that all three appenders expect data in different formats, File appender expects to get a string, DBI - a list of parameters and ElasticSearch - JSON. This is the place where you get stuck and has to choose something one.
> The Log::Message::JSON module, as I understand it, is trying to address the problem using objects which get serialized to JSON in appenders expecting stings. But calling a special function (logmsg) every time you log something is quite a hassle and this is only JSON, what if I want to use XML? Here we go, there is no way in log4perl you can format you data before they are passed to appender. Layouts are meant to render messages, not arbitrary data. I also was trying to play with the warp_message appender option but it's result is always an array ref which doesn't work with File appender for example and it's probably not the best way to solve the problem.
> To don't introduce new entities I'd say that Layouts is a proper place to make such transformations but it is just not how things work now.
>
> Can someone suggest a good solution for that or maybe an idea?
> What are you thoughts in general on the problem, should it be solved in log4perl?
> Thanks.
> --
> Dmitry
> e-mail: ic...@in...
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_mar
> _______________________________________________
> log4perl-devel mailing list
> log...@li...
> https://lists.sourceforge.net/lists/listinfo/log4perl-devel

Re: [log4perl-devel] Structured logging in log4perl

[Dmitry B. <ic...@in...>]

Yes, that's actually how I see it, get the raw data, format them to a certain type and pass to an appender.
As a matter of fact, this is exactly what Layouts allow me to do. I was wrong saying that they are meant to render strings only. If fact you get an array ref with all raw data inside the render() method. Maybe this is a bit which is missing in the FAQ, "How can I write my own layout?". :)
So, I can implement my own Layout::JSON or/and Layout::XML and use them with different appenders which working with strings. Nice! The only thing I'm missing here is the ability to use built in layouts like PatternLayout when I want to dump JSON to a log file as a formatted string. Or another example, compressing the data before passing them to the appender. That brings me to the idea of having a chain of layout (in current terminology) when you get an actual result passed through a number of layouts and formatted in each of them accordingly. But maybe this is a controversial feature request and should be discussed in a separate thread.

/Dmitry

Пятница, 15 марта 2013, 8:01 +01:00 от Lee < le...@gm... >:
> Sorry to reply to such a long and thoughtful e-mail with a URI, but do 
> you think being given access to the raw data would give you sufficient 
> means to format it?
> 
>  http://log4perl.sourceforge.net/releases/Log-Log4perl/docs/html/Log/Log4perl/FAQ.html#e8a46
> 
> Could you then forward the formatted data to an appender of your 
> choice/via appenders listed in the config?
> 
> Lee
> 

Re: [log4perl-devel] Structured logging in log4perl

[Alexander H. <ale...@gm...>]

On Fri, Mar 15, 2013 at 11:29 AM, Dmitry Bigunyak <ic...@in...> wrote:

> Yes, that's actually how I see it, get the raw data, format them to a
> certain type and pass to an appender.
> As a matter of fact, this is exactly what Layouts allow me to do. I was
> wrong saying that they are meant to render strings only. If fact you get an
> array ref with all raw data inside the render() method. Maybe this is a bit
> which is missing in the FAQ, "How can I write my own layout?". :)
> So, I can implement my own Layout::JSON or/and Layout::XML and use them
> with different appenders which working with strings. Nice! The only thing
> I'm missing here is the ability to use built in layouts like PatternLayout
> when I want to dump JSON to a log file as a formatted string. Or another
> example, compressing the data before passing them to the appender. That
> brings me to the idea of having a chain of layout (in current terminology)
> when you get an actual result passed through a number of layouts and
> formatted in each of them accordingly. But maybe this is a controversial
> feature request and should be discussed in a separate thread.
>
> /Dmitry
>
> Пятница, 15 марта 2013, 8:01 +01:00 от Lee < le...@gm... >:
> > Sorry to reply to such a long and thoughtful e-mail with a URI, but do
> > you think being given access to the raw data would give you sufficient
> > means to format it?
> >
> >
> http://log4perl.sourceforge.net/releases/Log-Log4perl/docs/html/Log/Log4perl/FAQ.html#e8a46
> >
> > Could you then forward the formatted data to an appender of your
> > choice/via appenders listed in the config?
> >
> > Lee
> >
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_mar
> _______________________________________________
> log4perl-devel mailing list
> log...@li...
> https://lists.sourceforge.net/lists/listinfo/log4perl-devel
>

You might want to check out Message::Passing [1],
Log::Dispatch::Message::Passing [2] to get your Log4perl messages into it
and my Message::Passing::Output::Log::Any::Adapter [3] for the other
direction.
I wrote it to use Message::Passing to parse syslog messages and mail them
using Log4perl with Log::Dispatch::Email.

Keep us updated because I'm planning to push the log messages of all my
Log4perl based Catalyst and MooseX::App::Cmd apps to a central log store,
probably ElasticSearch and am interested in your use case and solution.

[1] https://metacpan.org/module/Message::Passing
[2] https://metacpan.org/module/Log::Dispatch::Message::Passing
[3] https://metacpan.org/module/Message::Passing::Output::Log::Any::Adapter

Re: [log4perl-devel] Structured logging in log4perl

[Mike S. <m...@pe...>]

On Fri, 15 Mar 2013, Dmitry Bigunyak wrote:

> That brings me to the idea of having a chain of layout (in current 
> terminology) when you get an actual result passed through a number of 
> layouts and formatted in each of them accordingly.

Have you looked at composite appenders yet?

     http://search.cpan.org/~mschilli/Log-Log4perl-1.40/lib/Log/Log4perl/Appender.pm#Composite_Appenders

You can use a composite appender to preprocess the message and have it
forwarded to the target appender right after.

For the type of processing you have in mind, I've originally put
warp_message in place, but I agree that the fact that it always produces
an array ref isn't helpful. Maybe we need another preprocess hook for
appenders in Log4perl.

-- 
-- Mike

Mike Schilli
m...@pe...