From: Lowell H. <lha...@vi...> - 2001-07-16 18:18:40
|
Rotation in general is bad though unless it is vitally important to keep the logs ... a busy crawler can generate 10k/minute in log data .. that's 14mb at the end of the day plus the rotation file... plus the arch folder growing and shrinking .. it all adds up. A solution might be log levels defined in the conf with a string like dnet pproxy: LogLevel: urls info stats server errors ... or general sets: LogLevel: [Minimum|Verbose|Debug] so one can define the amount of log data generated. Not everyone wants the whole url list that they are crawling or even any debug info, but some general info would be nice for everyone, especially statistics info once that is put in there. Lowell Vaclav Barta wro > > Lowell Hamilton wrote: > > > > Yeah... there are ways around the logfile growing ... I actually > > just linked grublog.log and putlog.log to /dev/null since I > > would never use it and capture stderr to a file A distributed > > client should not need maintaince by an outside application or > > consume a lot of resources. > Well, it shouldn't, but I would consider the handling/disposal > of logfiles (with logrotate or otherwise) an integral part of the > system, rather than an outside application... If you have apache, > what are you doing with apache logs? Maybe the client should just > use the system log (and hope it's configured correctly) - but > perhaps there's a reason so few applications do that and everybody > keeps their own files... > > > The average user probably wouldn't know how to use logrotate or > > how to write a shell script to rotate the logs ... or even where > > the logs are located when their hard drive fills up. Plus, only > > redhat-ish linux distributions even come with logrotate... and > That's why I'm saying that it should be disabled by default - if > the client doesn't work, people who are inclined to debug > the problem may enable logging, and the vast majority will just scrap > the application, whether they have logs or not... > > Bye > Vasek |