syslog-summary-commits — Commits Mailing List

[Syslog-summary-commits] SF.net SVN: syslog-summary: [4] trunk

[<dpa...@us...>]

Revision: 4
          http://syslog-summary.svn.sourceforge.net/syslog-summary/?rev=4&view=rev
Author:   dpaleino
Date:     2008-04-26 09:15:14 -0700 (Sat, 26 Apr 2008)

Log Message:
-----------
Added AUTHORS, NEWS, README.
Removed useless comments from the docbook source of the manpage.

Modified Paths:
--------------
    trunk/syslog-summary.1.xml

Added Paths:
-----------
    trunk/AUTHORS
    trunk/NEWS
    trunk/README

Added: trunk/AUTHORS
===================================================================
--- trunk/AUTHORS	                        (rev 0)
+++ trunk/AUTHORS	2008-04-26 16:15:14 UTC (rev 4)
@@ -0,0 +1 @@
+David Paleino <d.p...@gm...>

Added: trunk/NEWS
===================================================================

Added: trunk/README
===================================================================
--- trunk/README	                        (rev 0)
+++ trunk/README	2008-04-26 16:15:14 UTC (rev 4)
@@ -0,0 +1 @@
+Still to be written, sorry :(

Modified: trunk/syslog-summary.1.xml
===================================================================
--- trunk/syslog-summary.1.xml	2008-04-26 15:48:51 UTC (rev 3)
+++ trunk/syslog-summary.1.xml	2008-04-26 16:15:14 UTC (rev 4)
@@ -3,61 +3,12 @@
         href="http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl"?>
 <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
 "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
-
-<!--
-
-`xsltproc -''-nonet \
-          -''-param man.charmap.use.subset "0" \
-          -''-param make.year.ranges "1" \
-          -''-param make.single.year.ranges "1" \
-          /usr/share/xml/docbook/stylesheet/nwalsh/manpages/docbook.xsl \
-          manpage.xml'
-
-A manual page <package>.<section> will be generated. You may view the
-manual page with: nroff -man <package>.<section> | less'. A typical entry
-in a Makefile or Makefile.am is:
-
-DB2MAN = /usr/share/sgml/docbook/stylesheet/xsl/nwalsh/manpages/docbook.xsl
-XP     = xsltproc -''-nonet -''-param man.charmap.use.subset "0"
-
-manpage.1: manpage.xml
-        $(XP) $(DB2MAN) $<
-    
-The xsltproc binary is found in the xsltproc package. The XSL files are in
-docbook-xsl. A description of the parameters you can use can be found in the
-docbook-xsl-doc-* packages. Please remember that if you create the nroff
-version in one of the debian/rules file targets (such as build), you will need
-to include xsltproc and docbook-xsl in your Build-Depends control field.
-Alternatively use the xmlto command/package. That will also automatically
-pull in xsltproc and docbook-xsl.
-
-Notes for using docbook2x: docbook2x-man does not automatically create the
-AUTHOR(S) and COPYRIGHT sections. In this case, please add them manually as
-<refsect1> ... </refsect1>.
-
-To disable the automatic creation of the AUTHOR(S) and COPYRIGHT sections
-read /usr/share/doc/docbook-xsl/doc/manpages/authors.html. This file can be
-found in the docbook-xsl-doc-html package.
-
-Validation can be done using: `xmllint -''-noout -''-valid manpage.xml`
-
-General documentation about man-pages and man-page-formatting:
-man(1), man(7), http://www.tldp.org/HOWTO/Man-Page/
-
--->
-
-  <!-- Fill in your name for FIRSTNAME and SURNAME. -->
   <!ENTITY dhfirstname "David">
   <!ENTITY dhsurname   "Paleino">
   <!-- dhusername could also be set to "&firstname; &surname;". -->  
   <!ENTITY dhusername  "David Paleino">
   <!ENTITY dhemail     "d.p...@gm...">
-  <!-- SECTION should be 1-8, maybe w/ subsection other parameters are
-       allowed: see man(7), man(1) and
-       http://www.tldp.org/HOWTO/Man-Page/q2.html. -->
   <!ENTITY dhsection   "1">
-  <!-- TITLE should be something like "User commands" or similar (see
-       http://www.tldp.org/HOWTO/Man-Page/q2.html). -->
   <!ENTITY dhtitle     "Syslog-Summary User Manual">
   <!ENTITY dhucpackage "SYSLOG-SUMMARY">
   <!ENTITY dhpackage   "syslog-summary">


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[Syslog-summary-commits] SF.net SVN: syslog-summary: [3] trunk

[<dpa...@us...>]

Revision: 3
          http://syslog-summary.svn.sourceforge.net/syslog-summary/?rev=3&view=rev
Author:   dpaleino
Date:     2008-04-26 08:48:51 -0700 (Sat, 26 Apr 2008)

Log Message:
-----------
* branches/, tags/ and trunk/ created
* Files moved under the new hierarchy.

Added Paths:
-----------
    branches/
    tags/
    trunk/
    trunk/ChangeLog
    trunk/Makefile
    trunk/ignore.rules
    trunk/syslog-summary
    trunk/syslog-summary.1
    trunk/syslog-summary.1.xml

Removed Paths:
-------------
    ChangeLog
    Makefile
    ignore.rules
    syslog-summary
    syslog-summary.1
    syslog-summary.1.xml

Deleted: ChangeLog
===================================================================
--- ChangeLog	2008-04-26 15:46:22 UTC (rev 2)
+++ ChangeLog	2008-04-26 15:48:51 UTC (rev 3)
@@ -1,13 +0,0 @@
-syslog-summary (1.14)
-
-  * Added ability to read gzip compressed files, using python-magic or, if
-    unavailable, os.path.splitext as fallback.
-  * Added this ChangeLog file to track changes made to the sourcecode.
-  * Moved regexp matching (ignore rules) earlier in the code (i.e. right
-    after reading the line).
-  * Using hashlib (Python 2.5) instead of the deprecated md5 module
-  * printable_md5() removed: using hexdigest() method of the hashlib object.
-  * Using SHA-256 instead of MD5 now.
-  * Imported into SourceForge's SVN repository.
-
- -- David Paleino <d.p...@gm...>  Sat, 26 Apr 2008 17:45:51 +0200

Deleted: Makefile
===================================================================
--- Makefile	2008-04-26 15:46:22 UTC (rev 2)
+++ Makefile	2008-04-26 15:48:51 UTC (rev 3)
@@ -1,16 +0,0 @@
-XP = xsltproc --nonet \
-				--param man.charmap.use.subset "0" \
-				--param make.year.ranges "1" \
-				--param make.single.year.ranges "1"
-
-install:
-	install -m 755 syslog-summary $(DESTDIR)/usr/bin/syslog-summary
-	install -m 644 ignore.rules $(DESTDIR)/etc/syslog-summary/ignore.rules
-
-uninstall:
-	[ ! -f $(DESTDIR)/usr/bin/syslog-summary ] || rm -v $(DESTDIR)/usr/bin/syslog-summary
-	[ ! -d $(DESTDIR)/etc/syslog-summary ] || rm -vrf $(DESTDIR)/etc/syslog-summary/
-
-syslog-summary.1: syslog-summary.1.xml
-	$(XP) $<
-

Deleted: ignore.rules
===================================================================
--- ignore.rules	2008-04-26 15:46:22 UTC (rev 2)
+++ ignore.rules	2008-04-26 15:48:51 UTC (rev 3)
@@ -1,5 +0,0 @@
-#
-# Use this file to define regular expressions to ignore.
-#
-#last message repeated [0-9]* times
-#kernel: Checking 'hlt' instruction... Ok. 

Deleted: syslog-summary
===================================================================
--- syslog-summary	2008-04-26 15:46:22 UTC (rev 2)
+++ syslog-summary	2008-04-26 15:48:51 UTC (rev 3)
@@ -1,310 +0,0 @@
-#!/usr/bin/env python2.5
-# -*- coding: utf-8 -*-
-
-# Copyright © 2008, David Paleino <d.p...@gm...>
-#           © 2001-2008, Tommi Virtanen <tv...@de...>
-#           © 1998-2000, Lars Wirzenius <li...@ik...>
-# 
-#      This program is free software; you can redistribute it and/or modify
-#      it under the terms of the GNU General Public License as published by
-#      the Free Software Foundation; either version 3 of the License, or
-#      (at your option) any later version.
-#      
-#      This program is distributed in the hope that it will be useful,
-#      but WITHOUT ANY WARRANTY; without even the implied warranty of
-#      MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#      GNU General Public License for more details.
-#      
-#      You should have received a copy of the GNU General Public License
-#      along with this program; if not, write to the Free Software
-#      Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
-#      MA 02110-1301, USA.
- 
-"""Summarize the contents of a syslog log file.
-
-The syslog(3) service writes system log messages in a certain format:
-
-	Jan 17 19:21:50 zeus kernel: klogd 1.3-3, log source = /proc/kmsg started.
-
-This program summarizes the contents of such a file, by displaying each
-unique (except for the time) line once, and also the number of times such
-a line occurs in the input. The lines are displayed in the order they occur
-in the input.
-
-Lars Wirzenius <li...@ik...>"""
-
-IGNORE_FILENAME = "/etc/syslog-summary/ignore"
-STATE_FILENAME = None
-REPEAT = False
-QUIET = False
-DEBUG = False
-
-version = "1.14"
-
-import sys
-import re
-import getopt
-import string
-import hashlib
-from gzip import open as gzopen
-
-datepats = [
-	re.compile(r"^(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [ 0-9][0-9] [ 0-9][0-9]:[0-9][0-9]:[0-9][0-9] "),
-	re.compile(r"^(Mon|Tue|Wed|Thu|Fri|Sat|Sun) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [ 0-9][0-9][0-9][0-9]:[0-9][0-9] "),
-	re.compile(r"^(Mon|Tue|Wed|Thu|Fri|Sat|Sun) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [ 0-9][0-9][0-9][0-9]:[0-9][0-9]:[0-9][0-9] "),
-]
-pidpat = re.compile(r"^([^ ]* [^ ]*)\[[0-9][0-9]*\]: ")
-repeatpat = re.compile(r"^[^ ]* last message repeated (\d+) times$")
-
-ignore_pats = []
-
-def io_error(err, filename, die=True):
-	"""Prints a nice error message, i.e. Tracebacks are ugly to end users"""
-	import os, errno, traceback
-	num = err.errno
-	# DEBUG && die ensures that if it's a non-fatal exception, we don't
-	# show all the traceback mess...
-	if DEBUG:
-		if die:
-			traceback.print_exc(file=sys.stderr)
-		else:
-			print "[E] %s [%s(%s) - %s]" % (os.strerror(num), errno.errorcode[num], num, filename)
-
-	if die:
-		sys.exit(1)
-
-def read_patterns(filename):
-	"""Reads patterns to ignore from file specified by -i | --ignore="""
-	pats = []
-	try:
-		f = open(filename, "r")
-	except IOError, e:
-		io_error(e, filename, False)
-		return []
-	for line in f:
-		rule = line.strip()
-		if rule[0:1] == "#":
-			continue
-		else:
-			pats.append(re.compile(rule))
-	f.close()
-	return pats
-
-def read_states(filename):
-	"""Reads the previous state saved into the argument of -s | --state="""
-	states = {}
-	if not filename:
-		return states
-	try:
-		f = open(filename, "r")
-	except IOError, e:
-		io_error(e, filename, False)
-		return states
-	for line in f:
-		fields = string.split(line)
-		states[fields[0]] = (string.atoi(fields[1]), fields[2])
-	f.close()
-	return states
-
-def save_states(filename, states):
-	if not filename:
-		return
-	try:
-		f = open(filename, "w")
-	except IOError, e:
-		io_error(e, filename, True)
-	for filename in states.keys():
-		value = states[filename]
-		f.write("%s %d %s\n" % (filename, value[0], value[1]))
-	f.close()
-
-def should_be_ignored(line):
-	for pat in ignore_pats:
-		if pat.search(line):
-			return 1
-	return 0
-
-def split_date(line):
-	for pat in datepats:
-		m = pat.match(line)
-		if m:
-			return line[:m.end()], line[m.end():]
-	print "line has bad date", "<" + string.rstrip(line) + ">"
-	return None, line
-
-def is_gzipped(filename):
-	"""Returns True if the filename is a gzipped compressed file"""	
-	try:
-		import magic
-		ms = magic.open(magic.MAGIC_NONE)
-		ms.load()
-		if re.search("^gzip compressed data.*", ms.file(filename)):
-			return True
-		else:
-			return False
-	except:
-		from os.path import splitext
-		
-		if not QUIET:
-			print "Using fallback detection... please install python-magic for better gzip detection."
-		
-		if splitext(filename)[1] == ".gz":
-			return True
-		else:
-			return False
-
-def summarize(filename, states):
-	counts = {}
-	order = []
-	ignored_count = 0
-	if not QUIET:
-		print "Summarizing %s" % filename
-	
-	# If the file is a gzipped log, open it
-	# using the proper function from the gzip
-	# module.
-	try:
-		if is_gzipped(filename):
-			file = gzopen(filename, "rb")
-		else:
-			file = open(filename, "r")
-	except IOError, e:
-		io_error(e, filename, True)
-		
-	linecount = 0
-
-	shaobj = hashlib.sha256()
-	if filename in states:
-		oldlines, oldsha = states[filename]
-		for i in xrange(oldlines):
-			line = file.readline()
-			shaobj.update(line)
-		print "OLD-new: %s" % shaobj.hexdigest()
-		print "OLD-file: %s" % oldsha
-		if shaobj.hexdigest() != oldsha:
-			#file.seek(0, 0)
-			file.seek(0)
-			shaobj = hashlib.sha256()
-		else:
-			linecount = oldlines
-	if not QUIET:
-		print "%8d Lines skipped (already processed)" % linecount
-
-	line = file.readline()
-	previous = None
-	print "BEFORE-while: %s" % shaobj.hexdigest()
-	foo=0
-	while line:
-		foo+=1
-		shaobj.update(line)
-		linecount += 1
-		
-		if should_be_ignored(line):
-			ignored_count += 1
-			if DEBUG:
-				print "Ignoring: %s" % line
-			line = file.readline()
-		
-		date, rest = split_date(line)
-		if date:
-			found = pidpat.search(rest)
-			if found:
-				rest = found.group(1) + ": " + rest[found.end():]
-
-		count = 1
-		repeated = None
-		if REPEAT:
-			repeated = repeatpat.search(rest)
-		if repeated and previous:
-			count = int(repeated.group(1))
-			rest = previous
-
-		if counts.has_key(rest):
-			counts[rest] = counts[rest] + count
-		else:
-			assert count == 1
-			counts[rest] = count
-			order.append(rest)
-
-		if not repeated:
-			previous = rest
-		line = file.readline()
-	file.close()
-
-	print "TOT-lines: %d" % linecount
-	print "TOT-ignor: %d" % ignored_count
-	print "AFTER-while: %s" % shaobj.hexdigest()
-	print foo
-	states[filename] = (linecount + ignored_count, shaobj.hexdigest())
-	print states
-	
-	if QUIET and order:
-		print "Summarizing %s" % filename
-	if not QUIET or order:
-		print "%8d Patterns to ignore" % len(ignore_pats)
-		print "%8d Ignored lines" % ignored_count
-	for rest in order:
-		print "%8d %s" % (counts[rest], rest),
-	if not QUIET or order:
-		print
-
-def show_usage(opt = ""):
-	print """Syslog-Summary %s
-Usage: syslog-summary [options] <logfile> [<logfile> ...]
-
-Options available:
-
--i <arg>
---ignore=<arg>     read regular expressions from <arg>, and ignore lines in the
-                   <logfile> that match them
--s <arg>
---state=<arg>      read state information from filename (see the man page)
--r
---repeat           merge "last message repeated x times" with the event repeated
--q
---quiet            don't output anything, unless there were unmatched lines
--d
---debug            shows additional messages in case of error
--h
---help             show this help message and exit
-
-This program must be run with proper privileges to read log files!""" % version
-	
-	if opt:
-		print "Option \"%s\" not recognized." % opt
-	
-	sys.exit(0)
-
-def main():
-	global ignore_pats, IGNORE_FILENAME, STATE_FILENAME, REPEAT, QUIET, DEBUG
-
-	opts, args = getopt.getopt(sys.argv[1:], "i:qs:rhd", [
-		"ignore=", "quiet", "state=", "repeat", "help", "debug" ])
-
-	if len(sys.argv) == 1:
-		show_usage()
-	
-	for opt, optarg in opts:
-		if opt == "-i" or opt == "--ignore":
-			IGNORE_FILENAME = optarg
-		elif opt == "-s" or opt == "--state":
-			STATE_FILENAME = optarg
-		elif opt == "-r" or opt == "--repeat":
-			REPEAT = True
-		elif opt == "-q" or opt == "--quiet":
-			QUIET = True
-		elif opt == "-h" or opt == "--help":
-			show_usage()
-		elif opt == "-d" or opt == "--debug":
-			DEBUG = True
-		else:
-			show_usage(opt)
-
-	ignore_pats = read_patterns(IGNORE_FILENAME)
-	states = read_states(STATE_FILENAME)
-	for filename in args:
-		summarize(filename, states)
-	save_states(STATE_FILENAME, states)
-
-if __name__ == "__main__":
-	main()

Deleted: syslog-summary.1
===================================================================
--- syslog-summary.1	2008-04-26 15:46:22 UTC (rev 2)
+++ syslog-summary.1	2008-04-26 15:48:51 UTC (rev 3)
@@ -1,137 +0,0 @@
-.\"     Title: SYSLOG-SUMMARY
-.\"    Author: Lars Wirzenius <li...@ik...>
-.\" Generator: DocBook XSL Stylesheets v1.73.2 <http://docbook.sf.net/>
-.\"      Date: 02/07/2008
-.\"    Manual: Syslog-Summary User Manual
-.\"    Source: syslog-summary
-.\"
-.TH "SYSLOG\-SUMMARY" "1" "02/07/2008" "syslog-summary" "Syslog-Summary User Manual"
-.\" disable hyphenation
-.nh
-.\" disable justification (adjust text to left margin only)
-.ad l
-.SH "NAME"
-syslog-summary - summarize the contents of a syslog log file
-.SH "SYNOPSIS"
-.HP 15
-\fBsyslog\-summary\fR [\fB\-s\fR\ |\ \fB\-\-state=\fR\ \fIfilename\fR] [\fB\-i\fR\ |\ \fB\-\-ignore=\fR\ \fIfilename\fR] [\fB\-d\fR\ |\ \fB\-\-debug\fR] [\fB\-r\fR\ |\ \fB\-\-repeat\fR] \fIlogfile\fR...
-.HP 15
-\fBsyslog\-summary\fR [\fB\-h\fR\ |\ \fB\-\-help\fR]
-.SH "DESCRIPTION"
-.PP
-This manual page documents briefly the
-\fBsyslog\-summary\fR
-command\.
-.PP
-\fBsyslog\-summary\fR
-summarizes the contents of log files via the
-\fBsyslog\fR(3)
-service, by displaying each unique (except for the time) line once, and also the number of times such a line occurs in the input\. The lines are displayed in the order they occur in the input\.
-.SH "OPTIONS"
-.PP
-\fB\-i \fR\fB\fIfilename\fR\fR, \fB\-\-ignore=\fR\fB\fIfilename\fR\fR
-.RS 4
-Read regular expressions from
-\fIfilename\fR
-and ignore lines in the logfiles that match them\.
-.RE
-.PP
-\fB\-s \fR\fB\fIfilename\fR\fR, \fB\-\-state=\fR\fB\fIfilename\fR\fR
-.RS 4
-Read state information from
-\fIfilename\fR\. The state contains information about the already reported parts of a log file, and prevents
-\fBsyslog\-summary\fR
-from reporting the same things many times\. This is useful when
-\fBsyslog\-summary\fR
-is run from
-\fBcrontab\fR
-every hour\. The file is created, if it doesn\'t exist already\.
-.RE
-.PP
-\fB\-r\fR, \fB\-\-repeat\fR
-.RS 4
-Merge "last message repeated * times" lines with the repeated event\.
-.RE
-.PP
-\fB\-d\fR, \fB\-\-debug\fR
-.RS 4
-Enable verbose messages when errors occur (i\.e\. "debug mode")\.
-.RE
-.PP
-\fB\-h\fR, \fB\-\-help\fR
-.RS 4
-Show summary of options\.
-.RE
-.SH "FILES"
-.PP
-\fI/etc/syslog\-summary/ignore\.rules\fR
-.RS 4
-Default ignore\.rules file with default regular expressions\.
-.RE
-.SH "DIAGNOSTICS"
-.PP
-When reporting a bug, please run
-\fBsyslog\-summary\fR
-with the
-\fB\-d\fR
-(or
-\fB\-\-debug\fR) flag enabled\.
-.PP
-\fBsyslog\-summary\fR
-provides some return codes, that can be used in scripts:
-.\" line length increase to cope w/ tbl weirdness
-.ll +(\n(LLu * 62u / 100u)
-.TS
-ll.
-\fICode\fR	\fIDiagnostic\fR
-T{
-\fB0\fR
-T}	T{
-Program exited successfully\.
-T}
-T{
-\fB1\fR
-T}	T{
-Something went wrong, please run the program with the debug messages enabled\.
-T}
-.TE
-.\" line length decrease back to previous value
-.ll -(\n(LLu * 62u / 100u)
-.sp
-.SH "AUTHORS"
-.PP
-\fBLars Wirzenius\fR <\&liw@iki\.fi\&>
-.sp -1n
-.IP "" 4
-Wrote this manpage for the Debian system\.
-.sp -1n
-.IP "" 4
-Wrote the first version of syslog\-summary\.
-.sp -1n
-.IP "" 4
-Maintained the Debian package from 1998 to 2000\.
-.PP
-\fBTommi Virtanen\fR <\&tv@debian\.org\&>
-.sp -1n
-.IP "" 4
-Maintained the package from 2001 to early 2008\.
-.PP
-\fBDavid Paleino\fR <\&d\.paleino@gmail\.com\&>
-.sp -1n
-.IP "" 4
-Maintains the package since early 2008\.
-.SH "COPYRIGHT"
-Copyright \(co 2008 David Paleino
-.br
-Copyright \(co 2001-2007 Tommi Virtanen
-.br
-Copyright \(co 1998-2000 Lars Wirzenius
-.br
-.PP
-This manual page was written for the Debian system (but may be used by others)\.
-.PP
-Permission is granted to copy, distribute and/or modify this document under the terms of the GNU General Public License, Version 3 or (at your option) any later version published by the Free Software Foundation\.
-.PP
-On Debian systems, the complete text of the GNU General Public License can be found in
-\fI/usr/share/common\-licenses/GPL\fR\.
-.sp

Deleted: syslog-summary.1.xml
===================================================================
--- syslog-summary.1.xml	2008-04-26 15:46:22 UTC (rev 2)
+++ syslog-summary.1.xml	2008-04-26 15:48:51 UTC (rev 3)
@@ -1,276 +0,0 @@
-<?xml version='1.0' encoding='UTF-8'?>
-<?xml-stylesheet type="text/xsl"
-        href="http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl"?>
-<!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
-"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
-
-<!--
-
-`xsltproc -''-nonet \
-          -''-param man.charmap.use.subset "0" \
-          -''-param make.year.ranges "1" \
-          -''-param make.single.year.ranges "1" \
-          /usr/share/xml/docbook/stylesheet/nwalsh/manpages/docbook.xsl \
-          manpage.xml'
-
-A manual page <package>.<section> will be generated. You may view the
-manual page with: nroff -man <package>.<section> | less'. A typical entry
-in a Makefile or Makefile.am is:
-
-DB2MAN = /usr/share/sgml/docbook/stylesheet/xsl/nwalsh/manpages/docbook.xsl
-XP     = xsltproc -''-nonet -''-param man.charmap.use.subset "0"
-
-manpage.1: manpage.xml
-        $(XP) $(DB2MAN) $<
-    
-The xsltproc binary is found in the xsltproc package. The XSL files are in
-docbook-xsl. A description of the parameters you can use can be found in the
-docbook-xsl-doc-* packages. Please remember that if you create the nroff
-version in one of the debian/rules file targets (such as build), you will need
-to include xsltproc and docbook-xsl in your Build-Depends control field.
-Alternatively use the xmlto command/package. That will also automatically
-pull in xsltproc and docbook-xsl.
-
-Notes for using docbook2x: docbook2x-man does not automatically create the
-AUTHOR(S) and COPYRIGHT sections. In this case, please add them manually as
-<refsect1> ... </refsect1>.
-
-To disable the automatic creation of the AUTHOR(S) and COPYRIGHT sections
-read /usr/share/doc/docbook-xsl/doc/manpages/authors.html. This file can be
-found in the docbook-xsl-doc-html package.
-
-Validation can be done using: `xmllint -''-noout -''-valid manpage.xml`
-
-General documentation about man-pages and man-page-formatting:
-man(1), man(7), http://www.tldp.org/HOWTO/Man-Page/
-
--->
-
-  <!-- Fill in your name for FIRSTNAME and SURNAME. -->
-  <!ENTITY dhfirstname "David">
-  <!ENTITY dhsurname   "Paleino">
-  <!-- dhusername could also be set to "&firstname; &surname;". -->  
-  <!ENTITY dhusername  "David Paleino">
-  <!ENTITY dhemail     "d.p...@gm...">
-  <!-- SECTION should be 1-8, maybe w/ subsection other parameters are
-       allowed: see man(7), man(1) and
-       http://www.tldp.org/HOWTO/Man-Page/q2.html. -->
-  <!ENTITY dhsection   "1">
-  <!-- TITLE should be something like "User commands" or similar (see
-       http://www.tldp.org/HOWTO/Man-Page/q2.html). -->
-  <!ENTITY dhtitle     "Syslog-Summary User Manual">
-  <!ENTITY dhucpackage "SYSLOG-SUMMARY">
-  <!ENTITY dhpackage   "syslog-summary">
-]>
-
-<refentry>
-  <refentryinfo>
-    <title>&dhtitle;</title>
-    <productname>&dhpackage;</productname>
-    <authorgroup>
-      <author>
-       <firstname>Lars</firstname>
-        <surname>Wirzenius</surname>
-        <contrib>Wrote this manpage for the Debian system.</contrib>
-        <contrib>Wrote the first version of &dhpackage;.</contrib>
-        <contrib>Maintained the Debian package from 1998 to 2000.</contrib>
-        <address>
-          <email>li...@ik...</email>
-        </address>
-      </author>
-      <author>
-        <firstname>Tommi</firstname>
-        <surname>Virtanen</surname>
-        <contrib>Maintained the package from 2001 to early 2008.</contrib>
-        <address>
-          <email>tv...@de...</email>
-        </address>
-      </author>
-      <author>
-        <firstname>David</firstname>
-        <surname>Paleino</surname>
-        <contrib>Maintains the package since early 2008.</contrib>
-        <address>
-          <email>d.p...@gm...</email>
-        </address>
-      </author>
-    </authorgroup>
-    <copyright>
-      <year>2008</year>
-      <holder>David Paleino</holder>
-    </copyright>
-    <copyright>
-      <year>2001</year>
-      <year>2002</year>
-      <year>2003</year>
-      <year>2004</year>
-      <year>2005</year>
-      <year>2006</year>
-      <year>2007</year>
-      <holder>Tommi Virtanen</holder>
-    </copyright>
-    <copyright>
-      <year>1998</year>
-      <year>1999</year>
-      <year>2000</year>
-      <holder>Lars Wirzenius</holder>
-    </copyright>
-    <legalnotice>
-      <para>This manual page was written for the Debian system
-        (but may be used by others).</para>
-      <para>Permission is granted to copy, distribute and/or modify this
-        document under the terms of the GNU General Public License,
-        Version 3 or (at your option) any later version published by
-        the Free Software Foundation.</para>
-      <para>On Debian systems, the complete text of the GNU General Public
-        License can be found in
-	<filename>/usr/share/common-licenses/GPL</filename>.</para>
-    </legalnotice>
-  </refentryinfo>
-  <refmeta>
-    <refentrytitle>&dhucpackage;</refentrytitle>
-    <manvolnum>&dhsection;</manvolnum>
-  </refmeta>
-  <refnamediv>
-    <refname>&dhpackage;</refname>
-    <refpurpose>summarize the contents of a syslog log file</refpurpose>
-  </refnamediv>
-  <refsynopsisdiv>
-    <cmdsynopsis>
-      <command>&dhpackage;</command>
-      <!-- These are several examples, how syntaxes could look -->
-      <arg choice="opt">
-        <group choice="plain">
-          <arg choice="plain"><option>-s</option></arg>
-          <arg choice="plain"><option>--state=</option></arg>
-        </group>
-        <replaceable class="option">filename</replaceable>
-      </arg>
-      <arg choice="opt">
-        <group choice="plain">
-          <arg choice="plain"><option>-i</option></arg>
-          <arg choice="plain"><option>--ignore=</option></arg>
-        </group>
-        <replaceable class="option">filename</replaceable>
-      </arg>
-      <arg choice="opt">
-        <group choice="plain">
-          <arg choice="plain"><option>-d</option></arg>
-          <arg choice="plain"><option>--debug</option></arg>
-        </group>
-      </arg>
-      <arg choice="opt">
-        <group choice="plain">
-          <arg choice="plain"><option>-r</option></arg>
-          <arg choice="plain"><option>--repeat</option></arg>
-        </group>
-      </arg>
-      <arg choice="plain" rep="repeat"><replaceable>logfile</replaceable></arg>
-    </cmdsynopsis>
-    <cmdsynopsis>
-      <command>&dhpackage;</command>
-      <!-- Normally the help and version options make the programs stop
-           right after outputting the requested information. -->
-      <arg choice="opt">
-        <group choice="plain">
-          <arg choice="plain"><option>-h</option></arg>
-          <arg choice="plain"><option>--help</option></arg>
-        </group>
-      </arg>
-    </cmdsynopsis>
-  </refsynopsisdiv>
-  <refsect1 id="description">
-    <title>DESCRIPTION</title>
-    <para>This manual page documents briefly the
-      <command>&dhpackage;</command> command.</para>
-    <para><command>&dhpackage;</command> summarizes the contents of log files
-    via the
-    <citerefentry>
-      <refentrytitle>syslog</refentrytitle>
-      <manvolnum>3</manvolnum>
-    </citerefentry>
-    service, by displaying each unique (except for the time) line once, and also
-    the number of times such a line occurs in the input. The lines are displayed
-    in the order they occur in the input.</para>
-  </refsect1>
-  <refsect1 id="options">
-    <title>OPTIONS</title>
-    <variablelist>
-      <!-- Use the variablelist.term.separator and the
-           variablelist.term.break.after parameters to
-           control the term elements. -->
-      <varlistentry>
-        <term><option>-i <replaceable>filename</replaceable></option></term>
-        <term><option>--ignore=<replaceable>filename</replaceable></option></term>
-        <listitem>
-          <para>Read regular expressions from <replaceable>filename</replaceable>
-          and ignore lines in the logfiles that match them.</para>
-        </listitem>
-      </varlistentry>
-      <varlistentry>
-        <term><option>-s <replaceable>filename</replaceable></option></term>
-        <term><option>--state=<replaceable>filename</replaceable></option></term>
-        <listitem>
-          <para>Read state information from <replaceable>filename</replaceable>.
-          The state contains information about the already reported parts of a log
-          file, and prevents <command>syslog-summary</command> from reporting the
-          same things many times. This is useful when <command>syslog-summary</command>
-          is run from <command>crontab</command> every hour. The file is created,
-          if it doesn't exist already.</para>
-        </listitem>
-      </varlistentry>
-      <varlistentry>
-        <term><option>-r</option></term>
-        <term><option>--repeat</option></term>
-        <listitem>
-          <para>Merge "last message repeated * times" lines with the repeated
-          event.</para>
-        </listitem>
-      </varlistentry>
-      <varlistentry>
-        <term><option>-d</option></term>
-        <term><option>--debug</option></term>
-        <listitem>
-          <para>Enable verbose messages when errors occur (i.e. "debug mode").</para>
-        </listitem>
-      </varlistentry>
-      <varlistentry>
-        <term><option>-h</option></term>
-        <term><option>--help</option></term>
-        <listitem>
-          <para>Show summary of options.</para>
-        </listitem>
-      </varlistentry>
-    </variablelist>
-  </refsect1>
-  <refsect1 id="files">
-    <title>FILES</title>
-    <variablelist>
-      <varlistentry>
-        <term><filename>/etc/syslog-summary/ignore.rules</filename></term>
-        <listitem>
-          <para>Default ignore.rules file with default regular expressions.</para>
-        </listitem>
-      </varlistentry>
-    </variablelist>
-  </refsect1>
-  <refsect1 id="diagnostics">
-    <title>DIAGNOSTICS</title>
-    <para>When reporting a bug, please run <command>syslog-summary</command>
-    with the <option>-d</option> (or <option>--debug</option>) flag enabled.</para>
-    <para><command>&dhpackage;</command> provides some return codes, that can
-      be used in scripts:</para>
-    <segmentedlist>
-      <segtitle>Code</segtitle>
-      <segtitle>Diagnostic</segtitle>
-      <seglistitem>
-        <seg><errorcode>0</errorcode></seg>
-        <seg>Program exited successfully.</seg>
-      </seglistitem>
-      <seglistitem>
-        <seg><errorcode>1</errorcode></seg>
-        <seg>Something went wrong, please run the program with the debug messages enabled.</seg>
-      </seglistitem>
-    </segmentedlist>
-  </refsect1>
-</refentry>

Copied: trunk/ChangeLog (from rev 2, ChangeLog)
===================================================================
--- trunk/ChangeLog	                        (rev 0)
+++ trunk/ChangeLog	2008-04-26 15:48:51 UTC (rev 3)
@@ -0,0 +1,13 @@
+syslog-summary (1.14)
+
+  * Added ability to read gzip compressed files, using python-magic or, if
+    unavailable, os.path.splitext as fallback.
+  * Added this ChangeLog file to track changes made to the sourcecode.
+  * Moved regexp matching (ignore rules) earlier in the code (i.e. right
+    after reading the line).
+  * Using hashlib (Python 2.5) instead of the deprecated md5 module
+  * printable_md5() removed: using hexdigest() method of the hashlib object.
+  * Using SHA-256 instead of MD5 now.
+  * Imported into SourceForge's SVN repository.
+
+ -- David Paleino <d.p...@gm...>  Sat, 26 Apr 2008 17:45:51 +0200

Copied: trunk/Makefile (from rev 1, Makefile)
===================================================================
--- trunk/Makefile	                        (rev 0)
+++ trunk/Makefile	2008-04-26 15:48:51 UTC (rev 3)
@@ -0,0 +1,16 @@
+XP = xsltproc --nonet \
+				--param man.charmap.use.subset "0" \
+				--param make.year.ranges "1" \
+				--param make.single.year.ranges "1"
+
+install:
+	install -m 755 syslog-summary $(DESTDIR)/usr/bin/syslog-summary
+	install -m 644 ignore.rules $(DESTDIR)/etc/syslog-summary/ignore.rules
+
+uninstall:
+	[ ! -f $(DESTDIR)/usr/bin/syslog-summary ] || rm -v $(DESTDIR)/usr/bin/syslog-summary
+	[ ! -d $(DESTDIR)/etc/syslog-summary ] || rm -vrf $(DESTDIR)/etc/syslog-summary/
+
+syslog-summary.1: syslog-summary.1.xml
+	$(XP) $<
+

Copied: trunk/ignore.rules (from rev 1, ignore.rules)
===================================================================
--- trunk/ignore.rules	                        (rev 0)
+++ trunk/ignore.rules	2008-04-26 15:48:51 UTC (rev 3)
@@ -0,0 +1,5 @@
+#
+# Use this file to define regular expressions to ignore.
+#
+#last message repeated [0-9]* times
+#kernel: Checking 'hlt' instruction... Ok. 

Copied: trunk/syslog-summary (from rev 1, syslog-summary)
===================================================================
--- trunk/syslog-summary	                        (rev 0)
+++ trunk/syslog-summary	2008-04-26 15:48:51 UTC (rev 3)
@@ -0,0 +1,310 @@
+#!/usr/bin/env python2.5
+# -*- coding: utf-8 -*-
+
+# Copyright © 2008, David Paleino <d.p...@gm...>
+#           © 2001-2008, Tommi Virtanen <tv...@de...>
+#           © 1998-2000, Lars Wirzenius <li...@ik...>
+# 
+#      This program is free software; you can redistribute it and/or modify
+#      it under the terms of the GNU General Public License as published by
+#      the Free Software Foundation; either version 3 of the License, or
+#      (at your option) any later version.
+#      
+#      This program is distributed in the hope that it will be useful,
+#      but WITHOUT ANY WARRANTY; without even the implied warranty of
+#      MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#      GNU General Public License for more details.
+#      
+#      You should have received a copy of the GNU General Public License
+#      along with this program; if not, write to the Free Software
+#      Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+#      MA 02110-1301, USA.
+ 
+"""Summarize the contents of a syslog log file.
+
+The syslog(3) service writes system log messages in a certain format:
+
+	Jan 17 19:21:50 zeus kernel: klogd 1.3-3, log source = /proc/kmsg started.
+
+This program summarizes the contents of such a file, by displaying each
+unique (except for the time) line once, and also the number of times such
+a line occurs in the input. The lines are displayed in the order they occur
+in the input.
+
+Lars Wirzenius <li...@ik...>"""
+
+IGNORE_FILENAME = "/etc/syslog-summary/ignore"
+STATE_FILENAME = None
+REPEAT = False
+QUIET = False
+DEBUG = False
+
+version = "1.14"
+
+import sys
+import re
+import getopt
+import string
+import hashlib
+from gzip import open as gzopen
+
+datepats = [
+	re.compile(r"^(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [ 0-9][0-9] [ 0-9][0-9]:[0-9][0-9]:[0-9][0-9] "),
+	re.compile(r"^(Mon|Tue|Wed|Thu|Fri|Sat|Sun) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [ 0-9][0-9][0-9][0-9]:[0-9][0-9] "),
+	re.compile(r"^(Mon|Tue|Wed|Thu|Fri|Sat|Sun) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [ 0-9][0-9][0-9][0-9]:[0-9][0-9]:[0-9][0-9] "),
+]
+pidpat = re.compile(r"^([^ ]* [^ ]*)\[[0-9][0-9]*\]: ")
+repeatpat = re.compile(r"^[^ ]* last message repeated (\d+) times$")
+
+ignore_pats = []
+
+def io_error(err, filename, die=True):
+	"""Prints a nice error message, i.e. Tracebacks are ugly to end users"""
+	import os, errno, traceback
+	num = err.errno
+	# DEBUG && die ensures that if it's a non-fatal exception, we don't
+	# show all the traceback mess...
+	if DEBUG:
+		if die:
+			traceback.print_exc(file=sys.stderr)
+		else:
+			print "[E] %s [%s(%s) - %s]" % (os.strerror(num), errno.errorcode[num], num, filename)
+
+	if die:
+		sys.exit(1)
+
+def read_patterns(filename):
+	"""Reads patterns to ignore from file specified by -i | --ignore="""
+	pats = []
+	try:
+		f = open(filename, "r")
+	except IOError, e:
+		io_error(e, filename, False)
+		return []
+	for line in f:
+		rule = line.strip()
+		if rule[0:1] == "#":
+			continue
+		else:
+			pats.append(re.compile(rule))
+	f.close()
+	return pats
+
+def read_states(filename):
+	"""Reads the previous state saved into the argument of -s | --state="""
+	states = {}
+	if not filename:
+		return states
+	try:
+		f = open(filename, "r")
+	except IOError, e:
+		io_error(e, filename, False)
+		return states
+	for line in f:
+		fields = string.split(line)
+		states[fields[0]] = (string.atoi(fields[1]), fields[2])
+	f.close()
+	return states
+
+def save_states(filename, states):
+	if not filename:
+		return
+	try:
+		f = open(filename, "w")
+	except IOError, e:
+		io_error(e, filename, True)
+	for filename in states.keys():
+		value = states[filename]
+		f.write("%s %d %s\n" % (filename, value[0], value[1]))
+	f.close()
+
+def should_be_ignored(line):
+	for pat in ignore_pats:
+		if pat.search(line):
+			return 1
+	return 0
+
+def split_date(line):
+	for pat in datepats:
+		m = pat.match(line)
+		if m:
+			return line[:m.end()], line[m.end():]
+	print "line has bad date", "<" + string.rstrip(line) + ">"
+	return None, line
+
+def is_gzipped(filename):
+	"""Returns True if the filename is a gzipped compressed file"""	
+	try:
+		import magic
+		ms = magic.open(magic.MAGIC_NONE)
+		ms.load()
+		if re.search("^gzip compressed data.*", ms.file(filename)):
+			return True
+		else:
+			return False
+	except:
+		from os.path import splitext
+		
+		if not QUIET:
+			print "Using fallback detection... please install python-magic for better gzip detection."
+		
+		if splitext(filename)[1] == ".gz":
+			return True
+		else:
+			return False
+
+def summarize(filename, states):
+	counts = {}
+	order = []
+	ignored_count = 0
+	if not QUIET:
+		print "Summarizing %s" % filename
+	
+	# If the file is a gzipped log, open it
+	# using the proper function from the gzip
+	# module.
+	try:
+		if is_gzipped(filename):
+			file = gzopen(filename, "rb")
+		else:
+			file = open(filename, "r")
+	except IOError, e:
+		io_error(e, filename, True)
+		
+	linecount = 0
+
+	shaobj = hashlib.sha256()
+	if filename in states:
+		oldlines, oldsha = states[filename]
+		for i in xrange(oldlines):
+			line = file.readline()
+			shaobj.update(line)
+		print "OLD-new: %s" % shaobj.hexdigest()
+		print "OLD-file: %s" % oldsha
+		if shaobj.hexdigest() != oldsha:
+			#file.seek(0, 0)
+			file.seek(0)
+			shaobj = hashlib.sha256()
+		else:
+			linecount = oldlines
+	if not QUIET:
+		print "%8d Lines skipped (already processed)" % linecount
+
+	line = file.readline()
+	previous = None
+	print "BEFORE-while: %s" % shaobj.hexdigest()
+	foo=0
+	while line:
+		foo+=1
+		shaobj.update(line)
+		linecount += 1
+		
+		if should_be_ignored(line):
+			ignored_count += 1
+			if DEBUG:
+				print "Ignoring: %s" % line
+			line = file.readline()
+		
+		date, rest = split_date(line)
+		if date:
+			found = pidpat.search(rest)
+			if found:
+				rest = found.group(1) + ": " + rest[found.end():]
+
+		count = 1
+		repeated = None
+		if REPEAT:
+			repeated = repeatpat.search(rest)
+		if repeated and previous:
+			count = int(repeated.group(1))
+			rest = previous
+
+		if counts.has_key(rest):
+			counts[rest] = counts[rest] + count
+		else:
+			assert count == 1
+			counts[rest] = count
+			order.append(rest)
+
+		if not repeated:
+			previous = rest
+		line = file.readline()
+	file.close()
+
+	print "TOT-lines: %d" % linecount
+	print "TOT-ignor: %d" % ignored_count
+	print "AFTER-while: %s" % shaobj.hexdigest()
+	print foo
+	states[filename] = (linecount + ignored_count, shaobj.hexdigest())
+	print states
+	
+	if QUIET and order:
+		print "Summarizing %s" % filename
+	if not QUIET or order:
+		print "%8d Patterns to ignore" % len(ignore_pats)
+		print "%8d Ignored lines" % ignored_count
+	for rest in order:
+		print "%8d %s" % (counts[rest], rest),
+	if not QUIET or order:
+		print
+
+def show_usage(opt = ""):
+	print """Syslog-Summary %s
+Usage: syslog-summary [options] <logfile> [<logfile> ...]
+
+Options available:
+
+-i <arg>
+--ignore=<arg>     read regular expressions from <arg>, and ignore lines in the
+                   <logfile> that match them
+-s <arg>
+--state=<arg>      read state information from filename (see the man page)
+-r
+--repeat           merge "last message repeated x times" with the event repeated
+-q
+--quiet            don't output anything, unless there were unmatched lines
+-d
+--debug            shows additional messages in case of error
+-h
+--help             show this help message and exit
+
+This program must be run with proper privileges to read log files!""" % version
+	
+	if opt:
+		print "Option \"%s\" not recognized." % opt
+	
+	sys.exit(0)
+
+def main():
+	global ignore_pats, IGNORE_FILENAME, STATE_FILENAME, REPEAT, QUIET, DEBUG
+
+	opts, args = getopt.getopt(sys.argv[1:], "i:qs:rhd", [
+		"ignore=", "quiet", "state=", "repeat", "help", "debug" ])
+
+	if len(sys.argv) == 1:
+		show_usage()
+	
+	for opt, optarg in opts:
+		if opt == "-i" or opt == "--ignore":
+			IGNORE_FILENAME = optarg
+		elif opt == "-s" or opt == "--state":
+			STATE_FILENAME = optarg
+		elif opt == "-r" or opt == "--repeat":
+			REPEAT = True
+		elif opt == "-q" or opt == "--quiet":
+			QUIET = True
+		elif opt == "-h" or opt == "--help":
+			show_usage()
+		elif opt == "-d" or opt == "--debug":
+			DEBUG = True
+		else:
+			show_usage(opt)
+
+	ignore_pats = read_patterns(IGNORE_FILENAME)
+	states = read_states(STATE_FILENAME)
+	for filename in args:
+		summarize(filename, states)
+	save_states(STATE_FILENAME, states)
+
+if __name__ == "__main__":
+	main()

Copied: trunk/syslog-summary.1 (from rev 1, syslog-summary.1)
===================================================================
--- trunk/syslog-summary.1	                        (rev 0)
+++ trunk/syslog-summary.1	2008-04-26 15:48:51 UTC (rev 3)
@@ -0,0 +1,137 @@
+.\"     Title: SYSLOG-SUMMARY
+.\"    Author: Lars Wirzenius <li...@ik...>
+.\" Generator: DocBook XSL Stylesheets v1.73.2 <http://docbook.sf.net/>
+.\"      Date: 02/07/2008
+.\"    Manual: Syslog-Summary User Manual
+.\"    Source: syslog-summary
+.\"
+.TH "SYSLOG\-SUMMARY" "1" "02/07/2008" "syslog-summary" "Syslog-Summary User Manual"
+.\" disable hyphenation
+.nh
+.\" disable justification (adjust text to left margin only)
+.ad l
+.SH "NAME"
+syslog-summary - summarize the contents of a syslog log file
+.SH "SYNOPSIS"
+.HP 15
+\fBsyslog\-summary\fR [\fB\-s\fR\ |\ \fB\-\-state=\fR\ \fIfilename\fR] [\fB\-i\fR\ |\ \fB\-\-ignore=\fR\ \fIfilename\fR] [\fB\-d\fR\ |\ \fB\-\-debug\fR] [\fB\-r\fR\ |\ \fB\-\-repeat\fR] \fIlogfile\fR...
+.HP 15
+\fBsyslog\-summary\fR [\fB\-h\fR\ |\ \fB\-\-help\fR]
+.SH "DESCRIPTION"
+.PP
+This manual page documents briefly the
+\fBsyslog\-summary\fR
+command\.
+.PP
+\fBsyslog\-summary\fR
+summarizes the contents of log files via the
+\fBsyslog\fR(3)
+service, by displaying each unique (except for the time) line once, and also the number of times such a line occurs in the input\. The lines are displayed in the order they occur in the input\.
+.SH "OPTIONS"
+.PP
+\fB\-i \fR\fB\fIfilename\fR\fR, \fB\-\-ignore=\fR\fB\fIfilename\fR\fR
+.RS 4
+Read regular expressions from
+\fIfilename\fR
+and ignore lines in the logfiles that match them\.
+.RE
+.PP
+\fB\-s \fR\fB\fIfilename\fR\fR, \fB\-\-state=\fR\fB\fIfilename\fR\fR
+.RS 4
+Read state information from
+\fIfilename\fR\. The state contains information about the already reported parts of a log file, and prevents
+\fBsyslog\-summary\fR
+from reporting the same things many times\. This is useful when
+\fBsyslog\-summary\fR
+is run from
+\fBcrontab\fR
+every hour\. The file is created, if it doesn\'t exist already\.
+.RE
+.PP
+\fB\-r\fR, \fB\-\-repeat\fR
+.RS 4
+Merge "last message repeated * times" lines with the repeated event\.
+.RE
+.PP
+\fB\-d\fR, \fB\-\-debug\fR
+.RS 4
+Enable verbose messages when errors occur (i\.e\. "debug mode")\.
+.RE
+.PP
+\fB\-h\fR, \fB\-\-help\fR
+.RS 4
+Show summary of options\.
+.RE
+.SH "FILES"
+.PP
+\fI/etc/syslog\-summary/ignore\.rules\fR
+.RS 4
+Default ignore\.rules file with default regular expressions\.
+.RE
+.SH "DIAGNOSTICS"
+.PP
+When reporting a bug, please run
+\fBsyslog\-summary\fR
+with the
+\fB\-d\fR
+(or
+\fB\-\-debug\fR) flag enabled\.
+.PP
+\fBsyslog\-summary\fR
+provides some return codes, that can be used in scripts:
+.\" line length increase to cope w/ tbl weirdness
+.ll +(\n(LLu * 62u / 100u)
+.TS
+ll.
+\fICode\fR	\fIDiagnostic\fR
+T{
+\fB0\fR
+T}	T{
+Program exited successfully\.
+T}
+T{
+\fB1\fR
+T}	T{
+Something went wrong, please run the program with the debug messages enabled\.
+T}
+.TE
+.\" line length decrease back to previous value
+.ll -(\n(LLu * 62u / 100u)
+.sp
+.SH "AUTHORS"
+.PP
+\fBLars Wirzenius\fR <\&liw@iki\.fi\&>
+.sp -1n
+.IP "" 4
+Wrote this manpage for the Debian system\.
+.sp -1n
+.IP "" 4
+Wrote the first version of syslog\-summary\.
+.sp -1n
+.IP "" 4
+Maintained the Debian package from 1998 to 2000\.
+.PP
+\fBTommi Virtanen\fR <\&tv@debian\.org\&>
+.sp -1n
+.IP "" 4
+Maintained the package from 2001 to early 2008\.
+.PP
+\fBDavid Paleino\fR <\&d\.paleino@gmail\.com\&>
+.sp -1n
+.IP "" 4
+Maintains the package since early 2008\.
+.SH "COPYRIGHT"
+Copyright \(co 2008 David Paleino
+.br
+Copyright \(co 2001-2007 Tommi Virtanen
+.br
+Copyright \(co 1998-2000 Lars Wirzenius
+.br
+.PP
+This manual page was written for the Debian system (but may be used by others)\.
+.PP
+Permission is granted to copy, distribute and/or modify this document under the terms of the GNU General Public License, Version 3 or (at your option) any later version published by the Free Software Foundation\.
+.PP
+On Debian systems, the complete text of the GNU General Public License can be found in
+\fI/usr/share/common\-licenses/GPL\fR\.
+.sp

Copied: trunk/syslog-summary.1.xml (from rev 1, syslog-summary.1.xml)
===================================================================
--- trunk/syslog-summary.1.xml	                        (rev 0)
+++ trunk/syslog-summary.1.xml	2008-04-26 15:48:51 UTC (rev 3)
@@ -0,0 +1,276 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<?xml-stylesheet type="text/xsl"
+        href="http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl"?>
+<!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
+"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
+
+<!--
+
+`xsltproc -''-nonet \
+          -''-param man.charmap.use.subset "0" \
+          -''-param make.year.ranges "1" \
+          -''-param make.single.year.ranges "1" \
+          /usr/share/xml/docbook/stylesheet/nwalsh/manpages/docbook.xsl \
+          manpage.xml'
+
+A manual page <package>.<section> will be generated. You may view the
+manual page with: nroff -man <package>.<section> | less'. A typical entry
+in a Makefile or Makefile.am is:
+
+DB2MAN = /usr/share/sgml/docbook/stylesheet/xsl/nwalsh/manpages/docbook.xsl
+XP     = xsltproc -''-nonet -''-param man.charmap.use.subset "0"
+
+manpage.1: manpage.xml
+        $(XP) $(DB2MAN) $<
+    
+The xsltproc binary is found in the xsltproc package. The XSL files are in
+docbook-xsl. A description of the parameters you can use can be found in the
+docbook-xsl-doc-* packages. Please remember that if you create the nroff
+version in one of the debian/rules file targets (such as build), you will need
+to include xsltproc and docbook-xsl in your Build-Depends control field.
+Alternatively use the xmlto command/package. That will also automatically
+pull in xsltproc and docbook-xsl.
+
+Notes for using docbook2x: docbook2x-man does not automatically create the
+AUTHOR(S) and COPYRIGHT sections. In this case, please add them manually as
+<refsect1> ... </refsect1>.
+
+To disable the automatic creation of the AUTHOR(S) and COPYRIGHT sections
+read /usr/share/doc/docbook-xsl/doc/manpages/authors.html. This file can be
+found in the docbook-xsl-doc-html package.
+
+Validation can be done using: `xmllint -''-noout -''-valid manpage.xml`
+
+General documentation about man-pages and man-page-formatting:
+man(1), man(7), http://www.tldp.org/HOWTO/Man-Page/
+
+-->
+
+  <!-- Fill in your name for FIRSTNAME and SURNAME. -->
+  <!ENTITY dhfirstname "David">
+  <!ENTITY dhsurname   "Paleino">
+  <!-- dhusername could also be set to "&firstname; &surname;". -->  
+  <!ENTITY dhusername  "David Paleino">
+  <!ENTITY dhemail     "d.p...@gm...">
+  <!-- SECTION should be 1-8, maybe w/ subsection other parameters are
+       allowed: see man(7), man(1) and
+       http://www.tldp.org/HOWTO/Man-Page/q2.html. -->
+  <!ENTITY dhsection   "1">
+  <!-- TITLE should be something like "User commands" or similar (see
+       http://www.tldp.org/HOWTO/Man-Page/q2.html). -->
+  <!ENTITY dhtitle     "Syslog-Summary User Manual">
+  <!ENTITY dhucpackage "SYSLOG-SUMMARY">
+  <!ENTITY dhpackage   "syslog-summary">
+]>
+
+<refentry>
+  <refentryinfo>
+    <title>&dhtitle;</title>
+    <productname>&dhpackage;</productname>
+    <authorgroup>
+      <author>
+       <firstname>Lars</firstname>
+        <surname>Wirzenius</surname>
+        <contrib>Wrote this manpage for the Debian system.</contrib>
+        <contrib>Wrote the first version of &dhpackage;.</contrib>
+        <contrib>Maintained the Debian package from 1998 to 2000.</contrib>
+        <address>
+          <email>li...@ik...</email>
+        </address>
+      </author>
+      <author>
+        <firstname>Tommi</firstname>
+        <surname>Virtanen</surname>
+        <contrib>Maintained the package from 2001 to early 2008.</contrib>
+        <address>
+          <email>tv...@de...</email>
+        </address>
+      </author>
+      <author>
+        <firstname>David</firstname>
+        <surname>Paleino</surname>
+        <contrib>Maintains the package since early 2008.</contrib>
+        <address>
+          <email>d.p...@gm...</email>
+        </address>
+      </author>
+    </authorgroup>
+    <copyright>
+      <year>2008</year>
+      <holder>David Paleino</holder>
+    </copyright>
+    <copyright>
+      <year>2001</year>
+      <year>2002</year>
+      <year>2003</year>
+      <year>2004</year>
+      <year>2005</year>
+      <year>2006</year>
+      <year>2007</year>
+      <holder>Tommi Virtanen</holder>
+    </copyright>
+    <copyright>
+      <year>1998</year>
+      <year>1999</year>
+      <year>2000</year>
+      <holder>Lars Wirzenius</holder>
+    </copyright>
+    <legalnotice>
+      <para>This manual page was written for the Debian system
+        (but may be used by others).</para>
+      <para>Permission is granted to copy, distribute and/or modify this
+        document under the terms of the GNU General Public License,
+        Version 3 or (at your option) any later version published by
+        the Free Software Foundation.</para>
+      <para>On Debian systems, the complete text of the GNU General Public
+        License can be found in
+	<filename>/usr/share/common-licenses/GPL</filename>.</para>
+    </legalnotice>
+  </refentryinfo>
+  <refmeta>
+    <refentrytitle>&dhucpackage;</refentrytitle>
+    <manvolnum>&dhsection;</manvolnum>
+  </refmeta>
+  <refnamediv>
+    <refname>&dhpackage;</refname>
+    <refpurpose>summarize the contents of a syslog log file</refpurpose>
+  </refnamediv>
+  <refsynopsisdiv>
+    <cmdsynopsis>
+      <command>&dhpackage;</command>
+      <!-- These are several examples, how syntaxes could look -->
+      <arg choice="opt">
+        <group choice="plain">
+          <arg choice="plain"><option>-s</option></arg>
+          <arg choice="plain"><option>--state=</option></arg>
+        </group>
+        <replaceable class="option">filename</replaceable>
+      </arg>
+      <arg choice="opt">
+        <group choice="plain">
+          <arg choice="plain"><option>-i</option></arg>
+          <arg choice="plain"><option>--ignore=</option></arg>
+        </group>
+        <replaceable class="option">filename</replaceable>
+      </arg>
+      <arg choice="opt">
+        <group choice="plain">
+          <arg choice="plain"><option>-d</option></arg>
+          <arg choice="plain"><option>--debug</option></arg>
+        </group>
+      </arg>
+      <arg choice="opt">
+        <group choice="plain">
+          <arg choice="plain"><option>-r</option></arg>
+          <arg choice="plain"><option>--repeat</option></arg>
+        </group>
+      </arg>
+      <arg choice="plain" rep="repeat"><replaceable>logfile</replaceable></arg>
+    </cmdsynopsis>
+    <cmdsynopsis>
+      <command>&dhpackage;</command>
+      <!-- Normally the help and version options make the programs stop
+           right after outputting the requested information. -->
+      <arg choice="opt">
+        <group choice="plain">
+          <arg choice="plain"><option>-h</option></arg>
+          <arg choice="plain"><option>--help</option></arg>
+        </group>
+      </arg>
+    </cmdsynopsis>
+  </refsynopsisdiv>
+  <refsect1 id="description">
+    <title>DESCRIPTION</title>
+    <para>This manual page documents briefly the
+      <command>&dhpackage;</command> command.</para>
+    <para><command>&dhpackage;</command> summarizes the contents of log files
+    via the
+    <citerefentry>
+      <refentrytitle>syslog</refentrytitle>
+      <manvolnum>3</manvolnum>
+    </citerefentry>
+    service, by displaying each unique (except for the time) line once, and also
+    the number of times such a line occurs in the input. The lines are displayed
+    in the order they occur in the input.</para>
+  </refsect1>
+  <refsect1 id="options">
+    <title>OPTIONS</title>
+    <variablelist>
+      <!-- Use the variablelist.term.separator and the
+           variablelist.term.break.after parameters to
+           control the term elements. -->
+      <varlistentry>
+        <term><option>-i <replaceable>filename</replaceable></option></term>
+        <term><option>--ignore=<replaceable>filename</replaceable></option></term>
+        <listitem>
+          <para>Read regular expressions from <replaceable>filename</replaceable>
+          and ignore lines in the logfiles that match them.</para>
+        </listitem>
+      </varlistentry>
+      <varlistentry>
+        <term><option>-s <replaceable>filename</replaceable></option></term>
+        <term><option>--state=<replaceable>filename</replaceable></option></term>
+        <listitem>
+          <para>Read state information from <replaceable>filename</replaceable>.
+          The state contains information about the already reported parts of a log
+          file, and prevents <command>syslog-summary</command> from reporting the
+          same things many times. This is useful when <command>syslog-summary</command>
+          is run from <command>crontab</command> every hour. The file is created,
+          if it doesn't exist already.</para>
+        </listitem>
+      </varlistentry>
+      <varlistentry>
+        <term><option>-r</option></term>
+        <term><option>--repeat</option></term>
+        <listitem>
+          <para>Merge "last message repeated * times" lines with the repeated
+          event.</para>
+        </listitem>
+      </varlistentry>
+      <varlistentry>
+        <term><option>-d</option></term>
+        <term><option>--debug</option></term>
+        <listitem>
+          <para>Enable verbose messages when errors occur (i.e. "debug mode").</para>
+        </listitem>
+      </varlistentry>
+      <varlistentry>
+        <term><option>-h</option></term>
+        <term><option>--help</option></term>
+        <listitem>
+          <para>Show summary of options.</para>
+        </listitem>
+      </varlistentry>
+    </variablelist>
+  </refsect1>
+  <refsect1 id="files">
+    <title>FILES</title>
+    <variablelist>
+      <varlistentry>
+        <term><filename>/etc/syslog-summary/ignore.rules</filename></term>
+        <listitem>
+          <para>Default ignore.rules file with default regular expressions.</para>
+        </listitem>
+      </varlistentry>
+    </variablelist>
+  </refsect1>
+  <refsect1 id="diagnostics">
+    <title>DIAGNOSTICS</title>
+    <para>When reporting a bug, please run <command>syslog-summary</command>
+    with the <option>-d</option> (or <option>--debug</option>) flag enabled.</para>
+    <para><command>&dhpackage;</command> provides some return codes, that can
+      be used in scripts:</para>
+    <segmentedlist>
+      <segtitle>Code</segtitle>
+      <segtitle>Diagnostic</segtitle>
+      <seglistitem>
+        <seg><errorcode>0</errorcode></seg>
+        <seg>Program exited successfully.</seg>
+      </seglistitem>
+      <seglistitem>
+        <seg><errorcode>1</errorcode></seg>
+        <seg>Something went wrong, please run the program with the debug messages enabled.</seg>
+      </seglistitem>
+    </segmentedlist>
+  </refsect1>
+</refentry>


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[Syslog-summary-commits] SF.net SVN: syslog-summary: [2] ChangeLog

[<dpa...@us...>]

Revision: 2
          http://syslog-summary.svn.sourceforge.net/syslog-summary/?rev=2&view=rev
Author:   dpaleino
Date:     2008-04-26 08:46:22 -0700 (Sat, 26 Apr 2008)

Log Message:
-----------
Changelog updated: information about SVN repository available.

Modified Paths:
--------------
    ChangeLog

Modified: ChangeLog
===================================================================
--- ChangeLog	2008-04-26 15:43:34 UTC (rev 1)
+++ ChangeLog	2008-04-26 15:46:22 UTC (rev 2)
@@ -8,5 +8,6 @@
   * Using hashlib (Python 2.5) instead of the deprecated md5 module
   * printable_md5() removed: using hexdigest() method of the hashlib object.
   * Using SHA-256 instead of MD5 now.
+  * Imported into SourceForge's SVN repository.
 
- -- David Paleino <d.p...@gm...>  Sun, 24 Feb 2008 17:44:06 +0100
+ -- David Paleino <d.p...@gm...>  Sat, 26 Apr 2008 17:45:51 +0200


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[Syslog-summary-commits] SF.net SVN: syslog-summary: [1]

[<dpa...@us...>]

Revision: 1
          http://syslog-summary.svn.sourceforge.net/syslog-summary/?rev=1&view=rev
Author:   dpaleino
Date:     2008-04-26 08:43:34 -0700 (Sat, 26 Apr 2008)

Log Message:
-----------
First sourcecode commit

Added Paths:
-----------
    ChangeLog
    Makefile
    ignore.rules
    syslog-summary
    syslog-summary.1
    syslog-summary.1.xml

Added: ChangeLog
===================================================================
--- ChangeLog	                        (rev 0)
+++ ChangeLog	2008-04-26 15:43:34 UTC (rev 1)
@@ -0,0 +1,12 @@
+syslog-summary (1.14)
+
+  * Added ability to read gzip compressed files, using python-magic or, if
+    unavailable, os.path.splitext as fallback.
+  * Added this ChangeLog file to track changes made to the sourcecode.
+  * Moved regexp matching (ignore rules) earlier in the code (i.e. right
+    after reading the line).
+  * Using hashlib (Python 2.5) instead of the deprecated md5 module
+  * printable_md5() removed: using hexdigest() method of the hashlib object.
+  * Using SHA-256 instead of MD5 now.
+
+ -- David Paleino <d.p...@gm...>  Sun, 24 Feb 2008 17:44:06 +0100


Property changes on: ChangeLog
___________________________________________________________________
Name: svn:mime-type
   + text/plain

Added: Makefile
===================================================================
--- Makefile	                        (rev 0)
+++ Makefile	2008-04-26 15:43:34 UTC (rev 1)
@@ -0,0 +1,16 @@
+XP = xsltproc --nonet \
+				--param man.charmap.use.subset "0" \
+				--param make.year.ranges "1" \
+				--param make.single.year.ranges "1"
+
+install:
+	install -m 755 syslog-summary $(DESTDIR)/usr/bin/syslog-summary
+	install -m 644 ignore.rules $(DESTDIR)/etc/syslog-summary/ignore.rules
+
+uninstall:
+	[ ! -f $(DESTDIR)/usr/bin/syslog-summary ] || rm -v $(DESTDIR)/usr/bin/syslog-summary
+	[ ! -d $(DESTDIR)/etc/syslog-summary ] || rm -vrf $(DESTDIR)/etc/syslog-summary/
+
+syslog-summary.1: syslog-summary.1.xml
+	$(XP) $<
+


Property changes on: Makefile
___________________________________________________________________
Name: svn:mime-type
   + text/plain

Added: ignore.rules
===================================================================
--- ignore.rules	                        (rev 0)
+++ ignore.rules	2008-04-26 15:43:34 UTC (rev 1)
@@ -0,0 +1,5 @@
+#
+# Use this file to define regular expressions to ignore.
+#
+#last message repeated [0-9]* times
+#kernel: Checking 'hlt' instruction... Ok. 


Property changes on: ignore.rules
___________________________________________________________________
Name: svn:mime-type
   + text/plain

Added: syslog-summary
===================================================================
--- syslog-summary	                        (rev 0)
+++ syslog-summary	2008-04-26 15:43:34 UTC (rev 1)
@@ -0,0 +1,310 @@
+#!/usr/bin/env python2.5
+# -*- coding: utf-8 -*-
+
+# Copyright © 2008, David Paleino <d.p...@gm...>
+#           © 2001-2008, Tommi Virtanen <tv...@de...>
+#           © 1998-2000, Lars Wirzenius <li...@ik...>
+# 
+#      This program is free software; you can redistribute it and/or modify
+#      it under the terms of the GNU General Public License as published by
+#      the Free Software Foundation; either version 3 of the License, or
+#      (at your option) any later version.
+#      
+#      This program is distributed in the hope that it will be useful,
+#      but WITHOUT ANY WARRANTY; without even the implied warranty of
+#      MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#      GNU General Public License for more details.
+#      
+#      You should have received a copy of the GNU General Public License
+#      along with this program; if not, write to the Free Software
+#      Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+#      MA 02110-1301, USA.
+ 
+"""Summarize the contents of a syslog log file.
+
+The syslog(3) service writes system log messages in a certain format:
+
+	Jan 17 19:21:50 zeus kernel: klogd 1.3-3, log source = /proc/kmsg started.
+
+This program summarizes the contents of such a file, by displaying each
+unique (except for the time) line once, and also the number of times such
+a line occurs in the input. The lines are displayed in the order they occur
+in the input.
+
+Lars Wirzenius <li...@ik...>"""
+
+IGNORE_FILENAME = "/etc/syslog-summary/ignore"
+STATE_FILENAME = None
+REPEAT = False
+QUIET = False
+DEBUG = False
+
+version = "1.14"
+
+import sys
+import re
+import getopt
+import string
+import hashlib
+from gzip import open as gzopen
+
+datepats = [
+	re.compile(r"^(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [ 0-9][0-9] [ 0-9][0-9]:[0-9][0-9]:[0-9][0-9] "),
+	re.compile(r"^(Mon|Tue|Wed|Thu|Fri|Sat|Sun) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [ 0-9][0-9][0-9][0-9]:[0-9][0-9] "),
+	re.compile(r"^(Mon|Tue|Wed|Thu|Fri|Sat|Sun) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [ 0-9][0-9][0-9][0-9]:[0-9][0-9]:[0-9][0-9] "),
+]
+pidpat = re.compile(r"^([^ ]* [^ ]*)\[[0-9][0-9]*\]: ")
+repeatpat = re.compile(r"^[^ ]* last message repeated (\d+) times$")
+
+ignore_pats = []
+
+def io_error(err, filename, die=True):
+	"""Prints a nice error message, i.e. Tracebacks are ugly to end users"""
+	import os, errno, traceback
+	num = err.errno
+	# DEBUG && die ensures that if it's a non-fatal exception, we don't
+	# show all the traceback mess...
+	if DEBUG:
+		if die:
+			traceback.print_exc(file=sys.stderr)
+		else:
+			print "[E] %s [%s(%s) - %s]" % (os.strerror(num), errno.errorcode[num], num, filename)
+
+	if die:
+		sys.exit(1)
+
+def read_patterns(filename):
+	"""Reads patterns to ignore from file specified by -i | --ignore="""
+	pats = []
+	try:
+		f = open(filename, "r")
+	except IOError, e:
+		io_error(e, filename, False)
+		return []
+	for line in f:
+		rule = line.strip()
+		if rule[0:1] == "#":
+			continue
+		else:
+			pats.append(re.compile(rule))
+	f.close()
+	return pats
+
+def read_states(filename):
+	"""Reads the previous state saved into the argument of -s | --state="""
+	states = {}
+	if not filename:
+		return states
+	try:
+		f = open(filename, "r")
+	except IOError, e:
+		io_error(e, filename, False)
+		return states
+	for line in f:
+		fields = string.split(line)
+		states[fields[0]] = (string.atoi(fields[1]), fields[2])
+	f.close()
+	return states
+
+def save_states(filename, states):
+	if not filename:
+		return
+	try:
+		f = open(filename, "w")
+	except IOError, e:
+		io_error(e, filename, True)
+	for filename in states.keys():
+		value = states[filename]
+		f.write("%s %d %s\n" % (filename, value[0], value[1]))
+	f.close()
+
+def should_be_ignored(line):
+	for pat in ignore_pats:
+		if pat.search(line):
+			return 1
+	return 0
+
+def split_date(line):
+	for pat in datepats:
+		m = pat.match(line)
+		if m:
+			return line[:m.end()], line[m.end():]
+	print "line has bad date", "<" + string.rstrip(line) + ">"
+	return None, line
+
+def is_gzipped(filename):
+	"""Returns True if the filename is a gzipped compressed file"""	
+	try:
+		import magic
+		ms = magic.open(magic.MAGIC_NONE)
+		ms.load()
+		if re.search("^gzip compressed data.*", ms.file(filename)):
+			return True
+		else:
+			return False
+	except:
+		from os.path import splitext
+		
+		if not QUIET:
+			print "Using fallback detection... please install python-magic for better gzip detection."
+		
+		if splitext(filename)[1] == ".gz":
+			return True
+		else:
+			return False
+
+def summarize(filename, states):
+	counts = {}
+	order = []
+	ignored_count = 0
+	if not QUIET:
+		print "Summarizing %s" % filename
+	
+	# If the file is a gzipped log, open it
+	# using the proper function from the gzip
+	# module.
+	try:
+		if is_gzipped(filename):
+			file = gzopen(filename, "rb")
+		else:
+			file = open(filename, "r")
+	except IOError, e:
+		io_error(e, filename, True)
+		
+	linecount = 0
+
+	shaobj = hashlib.sha256()
+	if filename in states:
+		oldlines, oldsha = states[filename]
+		for i in xrange(oldlines):
+			line = file.readline()
+			shaobj.update(line)
+		print "OLD-new: %s" % shaobj.hexdigest()
+		print "OLD-file: %s" % oldsha
+		if shaobj.hexdigest() != oldsha:
+			#file.seek(0, 0)
+			file.seek(0)
+			shaobj = hashlib.sha256()
+		else:
+			linecount = oldlines
+	if not QUIET:
+		print "%8d Lines skipped (already processed)" % linecount
+
+	line = file.readline()
+	previous = None
+	print "BEFORE-while: %s" % shaobj.hexdigest()
+	foo=0
+	while line:
+		foo+=1
+		shaobj.update(line)
+		linecount += 1
+		
+		if should_be_ignored(line):
+			ignored_count += 1
+			if DEBUG:
+				print "Ignoring: %s" % line
+			line = file.readline()
+		
+		date, rest = split_date(line)
+		if date:
+			found = pidpat.search(rest)
+			if found:
+				rest = found.group(1) + ": " + rest[found.end():]
+
+		count = 1
+		repeated = None
+		if REPEAT:
+			repeated = repeatpat.search(rest)
+		if repeated and previous:
+			count = int(repeated.group(1))
+			rest = previous
+
+		if counts.has_key(rest):
+			counts[rest] = counts[rest] + count
+		else:
+			assert count == 1
+			counts[rest] = count
+			order.append(rest)
+
+		if not repeated:
+			previous = rest
+		line = file.readline()
+	file.close()
+
+	print "TOT-lines: %d" % linecount
+	print "TOT-ignor: %d" % ignored_count
+	print "AFTER-while: %s" % shaobj.hexdigest()
+	print foo
+	states[filename] = (linecount + ignored_count, shaobj.hexdigest())
+	print states
+	
+	if QUIET and order:
+		print "Summarizing %s" % filename
+	if not QUIET or order:
+		print "%8d Patterns to ignore" % len(ignore_pats)
+		print "%8d Ignored lines" % ignored_count
+	for rest in order:
+		print "%8d %s" % (counts[rest], rest),
+	if not QUIET or order:
+		print
+
+def show_usage(opt = ""):
+	print """Syslog-Summary %s
+Usage: syslog-summary [options] <logfile> [<logfile> ...]
+
+Options available:
+
+-i <arg>
+--ignore=<arg>     read regular expressions from <arg>, and ignore lines in the
+                   <logfile> that match them
+-s <arg>
+--state=<arg>      read state information from filename (see the man page)
+-r
+--repeat           merge "last message repeated x times" with the event repeated
+-q
+--quiet            don't output anything, unless there were unmatched lines
+-d
+--debug            shows additional messages in case of error
+-h
+--help             show this help message and exit
+
+This program must be run with proper privileges to read log files!""" % version
+	
+	if opt:
+		print "Option \"%s\" not recognized." % opt
+	
+	sys.exit(0)
+
+def main():
+	global ignore_pats, IGNORE_FILENAME, STATE_FILENAME, REPEAT, QUIET, DEBUG
+
+	opts, args = getopt.getopt(sys.argv[1:], "i:qs:rhd", [
+		"ignore=", "quiet", "state=", "repeat", "help", "debug" ])
+
+	if len(sys.argv) == 1:
+		show_usage()
+	
+	for opt, optarg in opts:
+		if opt == "-i" or opt == "--ignore":
+			IGNORE_FILENAME = optarg
+		elif opt == "-s" or opt == "--state":
+			STATE_FILENAME = optarg
+		elif opt == "-r" or opt == "--repeat":
+			REPEAT = True
+		elif opt == "-q" or opt == "--quiet":
+			QUIET = True
+		elif opt == "-h" or opt == "--help":
+			show_usage()
+		elif opt == "-d" or opt == "--debug":
+			DEBUG = True
+		else:
+			show_usage(opt)
+
+	ignore_pats = read_patterns(IGNORE_FILENAME)
+	states = read_states(STATE_FILENAME)
+	for filename in args:
+		summarize(filename, states)
+	save_states(STATE_FILENAME, states)
+
+if __name__ == "__main__":
+	main()


Property changes on: syslog-summary
___________________________________________________________________
Name: svn:executable
   + *

Added: syslog-summary.1
===================================================================
--- syslog-summary.1	                        (rev 0)
+++ syslog-summary.1	2008-04-26 15:43:34 UTC (rev 1)
@@ -0,0 +1,137 @@
+.\"     Title: SYSLOG-SUMMARY
+.\"    Author: Lars Wirzenius <li...@ik...>
+.\" Generator: DocBook XSL Stylesheets v1.73.2 <http://docbook.sf.net/>
+.\"      Date: 02/07/2008
+.\"    Manual: Syslog-Summary User Manual
+.\"    Source: syslog-summary
+.\"
+.TH "SYSLOG\-SUMMARY" "1" "02/07/2008" "syslog-summary" "Syslog-Summary User Manual"
+.\" disable hyphenation
+.nh
+.\" disable justification (adjust text to left margin only)
+.ad l
+.SH "NAME"
+syslog-summary - summarize the contents of a syslog log file
+.SH "SYNOPSIS"
+.HP 15
+\fBsyslog\-summary\fR [\fB\-s\fR\ |\ \fB\-\-state=\fR\ \fIfilename\fR] [\fB\-i\fR\ |\ \fB\-\-ignore=\fR\ \fIfilename\fR] [\fB\-d\fR\ |\ \fB\-\-debug\fR] [\fB\-r\fR\ |\ \fB\-\-repeat\fR] \fIlogfile\fR...
+.HP 15
+\fBsyslog\-summary\fR [\fB\-h\fR\ |\ \fB\-\-help\fR]
+.SH "DESCRIPTION"
+.PP
+This manual page documents briefly the
+\fBsyslog\-summary\fR
+command\.
+.PP
+\fBsyslog\-summary\fR
+summarizes the contents of log files via the
+\fBsyslog\fR(3)
+service, by displaying each unique (except for the time) line once, and also the number of times such a line occurs in the input\. The lines are displayed in the order they occur in the input\.
+.SH "OPTIONS"
+.PP
+\fB\-i \fR\fB\fIfilename\fR\fR, \fB\-\-ignore=\fR\fB\fIfilename\fR\fR
+.RS 4
+Read regular expressions from
+\fIfilename\fR
+and ignore lines in the logfiles that match them\.
+.RE
+.PP
+\fB\-s \fR\fB\fIfilename\fR\fR, \fB\-\-state=\fR\fB\fIfilename\fR\fR
+.RS 4
+Read state information from
+\fIfilename\fR\. The state contains information about the already reported parts of a log file, and prevents
+\fBsyslog\-summary\fR
+from reporting the same things many times\. This is useful when
+\fBsyslog\-summary\fR
+is run from
+\fBcrontab\fR
+every hour\. The file is created, if it doesn\'t exist already\.
+.RE
+.PP
+\fB\-r\fR, \fB\-\-repeat\fR
+.RS 4
+Merge "last message repeated * times" lines with the repeated event\.
+.RE
+.PP
+\fB\-d\fR, \fB\-\-debug\fR
+.RS 4
+Enable verbose messages when errors occur (i\.e\. "debug mode")\.
+.RE
+.PP
+\fB\-h\fR, \fB\-\-help\fR
+.RS 4
+Show summary of options\.
+.RE
+.SH "FILES"
+.PP
+\fI/etc/syslog\-summary/ignore\.rules\fR
+.RS 4
+Default ignore\.rules file with default regular expressions\.
+.RE
+.SH "DIAGNOSTICS"
+.PP
+When reporting a bug, please run
+\fBsyslog\-summary\fR
+with the
+\fB\-d\fR
+(or
+\fB\-\-debug\fR) flag enabled\.
+.PP
+\fBsyslog\-summary\fR
+provides some return codes, that can be used in scripts:
+.\" line length increase to cope w/ tbl weirdness
+.ll +(\n(LLu * 62u / 100u)
+.TS
+ll.
+\fICode\fR	\fIDiagnostic\fR
+T{
+\fB0\fR
+T}	T{
+Program exited successfully\.
+T}
+T{
+\fB1\fR
+T}	T{
+Something went wrong, please run the program with the debug messages enabled\.
+T}
+.TE
+.\" line length decrease back to previous value
+.ll -(\n(LLu * 62u / 100u)
+.sp
+.SH "AUTHORS"
+.PP
+\fBLars Wirzenius\fR <\&liw@iki\.fi\&>
+.sp -1n
+.IP "" 4
+Wrote this manpage for the Debian system\.
+.sp -1n
+.IP "" 4
+Wrote the first version of syslog\-summary\.
+.sp -1n
+.IP "" 4
+Maintained the Debian package from 1998 to 2000\.
+.PP
+\fBTommi Virtanen\fR <\&tv@debian\.org\&>
+.sp -1n
+.IP "" 4
+Maintained the package from 2001 to early 2008\.
+.PP
+\fBDavid Paleino\fR <\&d\.paleino@gmail\.com\&>
+.sp -1n
+.IP "" 4
+Maintains the package since early 2008\.
+.SH "COPYRIGHT"
+Copyright \(co 2008 David Paleino
+.br
+Copyright \(co 2001-2007 Tommi Virtanen
+.br
+Copyright \(co 1998-2000 Lars Wirzenius
+.br
+.PP
+This manual page was written for the Debian system (but may be used by others)\.
+.PP
+Permission is granted to copy, distribute and/or modify this document under the terms of the GNU General Public License, Version 3 or (at your option) any later version published by the Free Software Foundation\.
+.PP
+On Debian systems, the complete text of the GNU General Public License can be found in
+\fI/usr/share/common\-licenses/GPL\fR\.
+.sp


Property changes on: syslog-summary.1
___________________________________________________________________
Name: svn:mime-type
   + text/plain

Added: syslog-summary.1.xml
===================================================================
--- syslog-summary.1.xml	                        (rev 0)
+++ syslog-summary.1.xml	2008-04-26 15:43:34 UTC (rev 1)
@@ -0,0 +1,276 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<?xml-stylesheet type="text/xsl"
+        href="http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl"?>
+<!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
+"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
+
+<!--
+
+`xsltproc -''-nonet \
+          -''-param man.charmap.use.subset "0" \
+          -''-param make.year.ranges "1" \
+          -''-param make.single.year.ranges "1" \
+          /usr/share/xml/docbook/stylesheet/nwalsh/manpages/docbook.xsl \
+          manpage.xml'
+
+A manual page <package>.<section> will be generated. You may view the
+manual page with: nroff -man <package>.<section> | less'. A typical entry
+in a Makefile or Makefile.am is:
+
+DB2MAN = /usr/share/sgml/docbook/stylesheet/xsl/nwalsh/manpages/docbook.xsl
+XP     = xsltproc -''-nonet -''-param man.charmap.use.subset "0"
+
+manpage.1: manpage.xml
+        $(XP) $(DB2MAN) $<
+    
+The xsltproc binary is found in the xsltproc package. The XSL files are in
+docbook-xsl. A description of the parameters you can use can be found in the
+docbook-xsl-doc-* packages. Please remember that if you create the nroff
+version in one of the debian/rules file targets (such as build), you will need
+to include xsltproc and docbook-xsl in your Build-Depends control field.
+Alternatively use the xmlto command/package. That will also automatically
+pull in xsltproc and docbook-xsl.
+
+Notes for using docbook2x: docbook2x-man does not automatically create the
+AUTHOR(S) and COPYRIGHT sections. In this case, please add them manually as
+<refsect1> ... </refsect1>.
+
+To disable the automatic creation of the AUTHOR(S) and COPYRIGHT sections
+read /usr/share/doc/docbook-xsl/doc/manpages/authors.html. This file can be
+found in the docbook-xsl-doc-html package.
+
+Validation can be done using: `xmllint -''-noout -''-valid manpage.xml`
+
+General documentation about man-pages and man-page-formatting:
+man(1), man(7), http://www.tldp.org/HOWTO/Man-Page/
+
+-->
+
+  <!-- Fill in your name for FIRSTNAME and SURNAME. -->
+  <!ENTITY dhfirstname "David">
+  <!ENTITY dhsurname   "Paleino">
+  <!-- dhusername could also be set to "&firstname; &surname;". -->  
+  <!ENTITY dhusername  "David Paleino">
+  <!ENTITY dhemail     "d.p...@gm...">
+  <!-- SECTION should be 1-8, maybe w/ subsection other parameters are
+       allowed: see man(7), man(1) and
+       http://www.tldp.org/HOWTO/Man-Page/q2.html. -->
+  <!ENTITY dhsection   "1">
+  <!-- TITLE should be something like "User commands" or similar (see
+       http://www.tldp.org/HOWTO/Man-Page/q2.html). -->
+  <!ENTITY dhtitle     "Syslog-Summary User Manual">
+  <!ENTITY dhucpackage "SYSLOG-SUMMARY">
+  <!ENTITY dhpackage   "syslog-summary">
+]>
+
+<refentry>
+  <refentryinfo>
+    <title>&dhtitle;</title>
+    <productname>&dhpackage;</productname>
+    <authorgroup>
+      <author>
+       <firstname>Lars</firstname>
+        <surname>Wirzenius</surname>
+        <contrib>Wrote this manpage for the Debian system.</contrib>
+        <contrib>Wrote the first version of &dhpackage;.</contrib>
+        <contrib>Maintained the Debian package from 1998 to 2000.</contrib>
+        <address>
+          <email>li...@ik...</email>
+        </address>
+      </author>
+      <author>
+        <firstname>Tommi</firstname>
+        <surname>Virtanen</surname>
+        <contrib>Maintained the package from 2001 to early 2008.</contrib>
+        <address>
+          <email>tv...@de...</email>
+        </address>
+      </author>
+      <author>
+        <firstname>David</firstname>
+        <surname>Paleino</surname>
+        <contrib>Maintains the package since early 2008.</contrib>
+        <address>
+          <email>d.p...@gm...</email>
+        </address>
+      </author>
+    </authorgroup>
+    <copyright>
+      <year>2008</year>
+      <holder>David Paleino</holder>
+    </copyright>
+    <copyright>
+      <year>2001</year>
+      <year>2002</year>
+      <year>2003</year>
+      <year>2004</year>
+      <year>2005</year>
+      <year>2006</year>
+      <year>2007</year>
+      <holder>Tommi Virtanen</holder>
+    </copyright>
+    <copyright>
+      <year>1998</year>
+      <year>1999</year>
+      <year>2000</year>
+      <holder>Lars Wirzenius</holder>
+    </copyright>
+    <legalnotice>
+      <para>This manual page was written for the Debian system
+        (but may be used by others).</para>
+      <para>Permission is granted to copy, distribute and/or modify this
+        document under the terms of the GNU General Public License,
+        Version 3 or (at your option) any later version published by
+        the Free Software Foundation.</para>
+      <para>On Debian systems, the complete text of the GNU General Public
+        License can be found in
+	<filename>/usr/share/common-licenses/GPL</filename>.</para>
+    </legalnotice>
+  </refentryinfo>
+  <refmeta>
+    <refentrytitle>&dhucpackage;</refentrytitle>
+    <manvolnum>&dhsection;</manvolnum>
+  </refmeta>
+  <refnamediv>
+    <refname>&dhpackage;</refname>
+    <refpurpose>summarize the contents of a syslog log file</refpurpose>
+  </refnamediv>
+  <refsynopsisdiv>
+    <cmdsynopsis>
+      <command>&dhpackage;</command>
+      <!-- These are several examples, how syntaxes could look -->
+      <arg choice="opt">
+        <group choice="plain">
+          <arg choice="plain"><option>-s</option></arg>
+          <arg choice="plain"><option>--state=</option></arg>
+        </group>
+        <replaceable class="option">filename</replaceable>
+      </arg>
+      <arg choice="opt">
+        <group choice="plain">
+          <arg choice="plain"><option>-i</option></arg>
+          <arg choice="plain"><option>--ignore=</option></arg>
+        </group>
+        <replaceable class="option">filename</replaceable>
+      </arg>
+      <arg choice="opt">
+        <group choice="plain">
+          <arg choice="plain"><option>-d</option></arg>
+          <arg choice="plain"><option>--debug</option></arg>
+        </group>
+      </arg>
+      <arg choice="opt">
+        <group choice="plain">
+          <arg choice="plain"><option>-r</option></arg>
+          <arg choice="plain"><option>--repeat</option></arg>
+        </group>
+      </arg>
+      <arg choice="plain" rep="repeat"><replaceable>logfile</replaceable></arg>
+    </cmdsynopsis>
+    <cmdsynopsis>
+      <command>&dhpackage;</command>
+      <!-- Normally the help and version options make the programs stop
+           right after outputting the requested information. -->
+      <arg choice="opt">
+        <group choice="plain">
+          <arg choice="plain"><option>-h</option></arg>
+          <arg choice="plain"><option>--help</option></arg>
+        </group>
+      </arg>
+    </cmdsynopsis>
+  </refsynopsisdiv>
+  <refsect1 id="description">
+    <title>DESCRIPTION</title>
+    <para>This manual page documents briefly the
+      <command>&dhpackage;</command> command.</para>
+    <para><command>&dhpackage;</command> summarizes the contents of log files
+    via the
+    <citerefentry>
+      <refentrytitle>syslog</refentrytitle>
+      <manvolnum>3</manvolnum>
+    </citerefentry>
+    service, by displaying each unique (except for the time) line once, and also
+    the number of times such a line occurs in the input. The lines are displayed
+    in the order they occur in the input.</para>
+  </refsect1>
+  <refsect1 id="options">
+    <title>OPTIONS</title>
+    <variablelist>
+      <!-- Use the variablelist.term.separator and the
+           variablelist.term.break.after parameters to
+           control the term elements. -->
+      <varlistentry>
+        <term><option>-i <replaceable>filename</replaceable></option></term>
+        <term><option>--ignore=<replaceable>filename</replaceable></option></term>
+        <listitem>
+          <para>Read regular expressions from <replaceable>filename</replaceable>
+          and ignore lines in the logfiles that match them.</para>
+        </listitem>
+      </varlistentry>
+      <varlistentry>
+        <term><option>-s <replaceable>filename</replaceable></option></term>
+        <term><option>--state=<replaceable>filename</replaceable></option></term>
+        <listitem>
+          <para>Read state information from <replaceable>filename</replaceable>.
+          The state contains information about the already reported parts of a log
+          file, and prevents <command>syslog-summary</command> from reporting the
+          same things many times. This is useful when <command>syslog-summary</command>
+          is run from <command>crontab</command> every hour. The file is created,
+          if it doesn't exist already.</para>
+        </listitem>
+      </varlistentry>
+      <varlistentry>
+        <term><option>-r</option></term>
+        <term><option>--repeat</option></term>
+        <listitem>
+          <para>Merge "last message repeated * times" lines with the repeated
+          event.</para>
+        </listitem>
+      </varlistentry>
+      <varlistentry>
+        <term><option>-d</option></term>
+        <term><option>--debug</option></term>
+        <listitem>
+          <para>Enable verbose messages when errors occur (i.e. "debug mode").</para>
+        </listitem>
+      </varlistentry>
+      <varlistentry>
+        <term><option>-h</option></term>
+        <term><option>--help</option></term>
+        <listitem>
+          <para>Show summary of options.</para>
+        </listitem>
+      </varlistentry>
+    </variablelist>
+  </refsect1>
+  <refsect1 id="files">
+    <title>FILES</title>
+    <variablelist>
+      <varlistentry>
+        <term><filename>/etc/syslog-summary/ignore.rules</filename></term>
+        <listitem>
+          <para>Default ignore.rules file with default regular expressions.</para>
+        </listitem>
+      </varlistentry>
+    </variablelist>
+  </refsect1>
+  <refsect1 id="diagnostics">
+    <title>DIAGNOSTICS</title>
+    <para>When reporting a bug, please run <command>syslog-summary</command>
+    with the <option>-d</option> (or <option>--debug</option>) flag enabled.</para>
+    <para><command>&dhpackage;</command> provides some return codes, that can
+      be used in scripts:</para>
+    <segmentedlist>
+      <segtitle>Code</segtitle>
+      <segtitle>Diagnostic</segtitle>
+      <seglistitem>
+        <seg><errorcode>0</errorcode></seg>
+        <seg>Program exited successfully.</seg>
+      </seglistitem>
+      <seglistitem>
+        <seg><errorcode>1</errorcode></seg>
+        <seg>Something went wrong, please run the program with the debug messages enabled.</seg>
+      </seglistitem>
+    </segmentedlist>
+  </refsect1>
+</refentry>


Property changes on: syslog-summary.1.xml
___________________________________________________________________
Name: svn:mime-type
   + text/plain


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.