It would be very helpful to have even a very simple example
of an efficient (again simple) log parser.
Looking at cl-httpd's common-log-parser isn't helpful
at all, because
it's *quite* complex..
Log parsing is just string parsing and file manipulation.
The below can be used in place of awk for the very simple case of extracting columns from the
interesting lines of a file (those matching a simple string):
(defun split (delim line)
"Return a list of subsequences separated by a one character delimiter, delim itself is not returned"
(loop :for mark = 0 :then (1+ point)
:for point = (position delim line :start mark)
:collect (subseq line mark point)
:while point))
(defun splitcols (columns file &key (delim #\Space) grep)
"Iterate over the lines of a file, returning the specified columns.
If :grep is supplied only lines containing that string will be returned."
(with-open-file (s file)
(loop :for line = (read-line s nil)
:while line
:when (or (eq grep nil) (search grep line))
:collect (loop :with chunks = (split delim line)
:for c :in columns
:collect (elt chunks c)))))
This is not particularly efficient. On my machine, comparing awk (including startup) to
SBCL 1.0.18 (not including startup) and sinking the output shows awk to be more than 3x
faster.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Logged In: YES
user_id=78907
Originator: NO
Log parsing is just string parsing and file manipulation.
The below can be used in place of awk for the very simple case of extracting columns from the
interesting lines of a file (those matching a simple string):
(defun split (delim line)
"Return a list of subsequences separated by a one character delimiter, delim itself is not returned"
(loop :for mark = 0 :then (1+ point)
:for point = (position delim line :start mark)
:collect (subseq line mark point)
:while point))
(defun splitcols (columns file &key (delim #\Space) grep)
"Iterate over the lines of a file, returning the specified columns.
If :grep is supplied only lines containing that string will be returned."
(with-open-file (s file)
(loop :for line = (read-line s nil)
:while line
:when (or (eq grep nil) (search grep line))
:collect (loop :with chunks = (split delim line)
:for c :in columns
:collect (elt chunks c)))))
This is not particularly efficient. On my machine, comparing awk (including startup) to
SBCL 1.0.18 (not including startup) and sinking the output shows awk to be more than 3x
faster.