openinteract-dev

[Openinteract-dev] what to log and when

[Chris W. <ch...@cw...>]

I've been rewriting the error module to work with the filesystem 
instead of the database. That part is (mostly) done, but my initial 
idea of hooking this to log4perl with a custom appender has hit a 
little snag.

  - If we set the threshold to 'FATAL' we don't store anything -- 
nothing is logged with a 'FATAL' level.
  - If we set the threshold to 'ERROR' we store a ton of messages, too 
many to be useful.

This is my fault for not setting guidelines in the beginning as to what 
the different logging levels should mean. I don't have a good feel for 
how other people do this, so before I go through and try and make 
everything at least roughly consistent I'd like to hear what you think:

What are appropriate messages to log at these levels?

   DEBUG
   INFO
   WARN
   ERROR
   FATAL

FWIW, related JIRA issues are at:

   http://jira.openinteract.org/browse/OIN-46
   http://jira.openinteract.org/browse/OIN-35

Thanks,

Chris

PS - Would it be useful to post here a periodic status message as to 
how OI2 is doing? In theory this should be on the wiki and I'll put it 
there if people find it useful, but I didn't get much feedback...

--
Chris Winters
Creating enterprise-capable snack systems since 1988

Re: [Openinteract-dev] what to log and when

[Perrin H. <pe...@el...>]

On Wed, 2005-02-23 at 23:24 -0500, Chris Winters wrote:
>   - If we set the threshold to 'ERROR' we store a ton of messages, too 
> many to be useful.

Hmmm, that doesn't sound good.  Maybe the Apache loglevel docs would
help you:
http://httpd.apache.org/docs/mod/core.html#loglevel

> PS - Would it be useful to post here a periodic status message as to 
> how OI2 is doing?

Yes, I think so.

- Perrin

Re: [Openinteract-dev] what to log and when

[Chris W. <ch...@cw...>]

On Feb 23, 2005, at 11:35 PM, Perrin Harkins wrote:
> Hmmm, that doesn't sound good.  Maybe the Apache loglevel docs would
> help you:
> http://httpd.apache.org/docs/mod/core.html#loglevel

That's very useful, thanks. They have more granularity but it's still 
helpful.

>> PS - Would it be useful to post here a periodic status message as to
>> how OI2 is doing?
>
> Yes, I think so.

I'll look into doing this and not relying on people to just read my 
blog and follow JIRA :-)

Chris

--
Chris Winters
Creating enterprise-capable snack systems since 1988

Re: [Openinteract-dev] what to log and when

[Perrin H. <pe...@el...>]

On Wed, 2005-02-23 at 23:48 -0500, Chris Winters wrote:
> That's very useful, thanks. They have more granularity but it's still 
> helpful.

Most admins will tell you they don't want to hear about it unless they
need to do something about it, and most developers will tell you they
take out their debugging stuff when they find the bug, so my advice is
to have very minimal logging in general.  Excessive attention to logging
is usually a sign of trouble in my opinion.

> I'll look into doing this and not relying on people to just read my 
> blog and follow JIRA :-)

I do follow your journal, but I wouldn't mind a brief OI2 update in my
mailbox now and then.

- Perrin

Re: [Openinteract-dev] what to log and when

[Chris W. <ch...@cw...>]

Perrin Harkins said:
> Most admins will tell you they don't want to hear about it unless they
> need to do something about it, and most developers will tell you they
> take out their debugging stuff when they find the bug, so my advice is
> to have very minimal logging in general.  Excessive attention to logging
> is usually a sign of trouble in my opinion.

I agree with you on administrators. And the error browser + files is
mainly intended for them.

But regarding developers I think there are differences between good
logging messages and the quick-iteration debugging messages ("got here",
data structure dumps) that usually take the place of the debugger.
log4perl makes it awfully easy to assign different logging levels to
different logging categories; combined with a consistent logging strategy
for a framework people can peer into the innards of what's going on in a
particular area of the system. I compare it to turning on DBI's trace
functionality. IMO this is extremely useful -- moreso because it's
externally configurable.

The trick is to get the logging at least roughly consistent, otherwise
you'll get too little/too much whenever you change a logging level.

Chris

-- 
Chris Winters (ch...@cw...)
Building enterprise-capable snack solutions since 1988.

Re: [Openinteract-dev] what to log and when

[Salve J N. <sal...@me...>]

Chris Winters wrote:
> I've been rewriting the error module to work with the filesystem instead 
> of the database. That part is (mostly) done, but my initial idea of 
> hooking this to log4perl with a custom appender has hit a little snag.
> 
>  - If we set the threshold to 'FATAL' we don't store anything -- nothing 
> is logged with a 'FATAL' level.

This doesn't necessary have to be a bad thing... Are there any 
situations that should be fatal to OI2? e.g. missing critical 
dependencies, inability to contact critical databases, badly broken 
(truncated?) primary config files, inability to access local files at 
all (broken filesystem.)

At this level, I hope never to get any output to the log files. :)

>  - If we set the threshold to 'ERROR' we store a ton of messages, too 
> many to be useful.

Do _all_ these messages deserve an "ERROR" status, or are some of them 
only informational? The ones that actually _are_ errors, should of 
course be fixed. :)


> This is my fault for not setting guidelines in the beginning as to what 
> the different logging levels should mean. I don't have a good feel for 
> how other people do this, so before I go through and try and make 
> everything at least roughly consistent I'd like to hear what you think:
> 
> What are appropriate messages to log at these levels?

Ok, here's my interpretation (With "most important" first :)

 >   FATAL

Fatalities (death). Situations where the system/application cannot 
continue safely and must abort. Manual intervention is necessary to get 
things running again. (Hopefully, this is something you'll never need to 
see. :)

 >   ERROR

Actual errors that need to be looked into. The application should be 
able to continue running despite having these errors, but should 
probably tell the user why the expected data can't be delivered.

>   WARN

For situations that at some point later may lead to errors. E.g. missing 
directories, use of default values in system-dependent configuration 
(e.g. when using a default IP address of 127.0.0.1, when it should have 
been defined explicitly in some config file.)

 >   INFO

High/Coarse-level status messages. I try to use this to tell when 
important subsystems are used, and what data they recieve/pass along.

 >   DEBUG

Detailed status messages, debugging, and all the rest. Personally I'd 
like several debug levels, but existing logging frameworks like 
syslog(1) and log4* seem to disagree with me. :)


> PS - Would it be useful to post here a periodic status message as to how 
> OI2 is doing?

Yes, please do! :)


- Salve

-- 
Salve J. Nilsen <salvejn at met dot no> / Systems Developer
Norwegian Meteorological Institute           http://met.no/
Information Technology Department / Section for Development

Re: [Openinteract-dev] what to log and when

[Chris W. <ch...@cw...>]

Salve J Nilsen said:
>>  - If we set the threshold to 'FATAL' we don't store anything -- nothing
>> is logged with a 'FATAL' level.
>
> This doesn't necessary have to be a bad thing... Are there any
> situations that should be fatal to OI2? e.g. missing critical
> dependencies, inability to contact critical databases, badly broken
> (truncated?) primary config files, inability to access local files at
> all (broken filesystem.)

These were all I could think of, although at least the dependencies should
get caught at server startup vs runtime.

> At this level, I hope never to get any output to the log files. :)
>
>>  - If we set the threshold to 'ERROR' we store a ton of messages, too
>> many to be useful.
>
> Do _all_ these messages deserve an "ERROR" status, or are some of them
> only informational? The ones that actually _are_ errors, should of
> course be fixed. :)

That's the problem -- while all of them are errors, some aren't ones you'd
actually care about. For instance, when you lookup an action with an
incorrect name OI2 will log an error. Yes, it is an error, but is it one
we need to log? Probably not, since the caller is in a better position to
evaluate whether it's actually an error.

There are lots of these, unfortunately. It'll just take some time to go
through them but it's worth the time.

>  >   FATAL
>
> Fatalities (death). Situations where the system/application cannot
> continue safely and must abort. Manual intervention is necessary to get
> things running again. (Hopefully, this is something you'll never need to
> see. :)

Right -- database crash, cannot load vital components (e.g., packages), etc.

>  >   ERROR
>
> Actual errors that need to be looked into. The application should be
> able to continue running despite having these errors, but should
> probably tell the user why the expected data can't be delivered.

So would this include not being able to store data in the database because
of bad data? Or failing to execute a query because of a typo in the
fieldname?

>>   WARN
>
> For situations that at some point later may lead to errors. E.g. missing
> directories, use of default values in system-dependent configuration
> (e.g. when using a default IP address of 127.0.0.1, when it should have
> been defined explicitly in some config file.)

I'd probably add security violations to this.

>  >   INFO
>
> High/Coarse-level status messages. I try to use this to tell when
> important subsystems are used, and what data they recieve/pass along.

Good guidelines. Not a lot of data is in these but more like a guide as to
what path the application is taking.

>  >   DEBUG
>
> Detailed status messages, debugging, and all the rest. Personally I'd
> like several debug levels, but existing logging frameworks like
> syslog(1) and log4* seem to disagree with me. :)

Yeah, everything else goes here :-)

Thanks!

Chris

-- 
Chris Winters (ch...@cw...)
Building enterprise-capable snack solutions since 1988.