I want to create a command-line tool that I myself am in need of, and think that others may find of use as well: a very simple to use program to quickly extract the logfile entries for a certain period, that are spread out over multiple logs, and show me the entries in sorted order, so that you do not have to do so by hand. The things that make it slightly non-trivial:
Log files can get big, we are talking tens of gigabytes. Too big to load into memory or memmap.
Logrotate can chop up logs into multiple files, and compress some of them.
Log entry formats can vary.
It may have to run on resource-constrained systems, and 32-bit architectures.
Approach: We start out by building some code to get a logical view on the logs, to handle the details of decompression, logfile formats, and finding out how files were split up. Then, we add a thin layer of commands on top to view, filter and search the logs. Finally, we wrap it up nicely as a Debian package and maybe an RPM.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The goal is now to produce two separate tools: a CLI version to use from scripts, and an terminal-based UI vaguely inspired by less(1), both interfacing to the same back-end code. They will be called lograw(1) and logras(1) respectively. Because the second one requires more design and work, I may release the first one by itself initially.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
To expedite a first release, I'm going to put out a "simple" UI version first, meaning restricted to a single log view screen, a help screen, and not much more. I think this will cover 90% of likely usage scenarios anyway.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I want to create a command-line tool that I myself am in need of, and think that others may find of use as well: a very simple to use program to quickly extract the logfile entries for a certain period, that are spread out over multiple logs, and show me the entries in sorted order, so that you do not have to do so by hand. The things that make it slightly non-trivial:
Approach: We start out by building some code to get a logical view on the logs, to handle the details of decompression, logfile formats, and finding out how files were split up. Then, we add a thin layer of commands on top to view, filter and search the logs. Finally, we wrap it up nicely as a Debian package and maybe an RPM.
The goal is now to produce two separate tools: a CLI version to use from scripts, and an terminal-based UI vaguely inspired by less(1), both interfacing to the same back-end code. They will be called lograw(1) and logras(1) respectively. Because the second one requires more design and work, I may release the first one by itself initially.
To expedite a first release, I'm going to put out a "simple" UI version first, meaning restricted to a single log view screen, a help screen, and not much more. I think this will cover 90% of likely usage scenarios anyway.