Re: [Fetchmail-users] logrotate options?

[Peter P. <ro...@ri...>]

On Wed, Jan 19, 2022 at 10:18:47PM -0500, Joe Acquisto-j4 wrote:
> > On Wed, Jan 19, 2022 at 04:35:35PM -0500, Joe Acquisto-j4 wrote:
> >> This is fetchmail release 6.4.21+SSL-SSLv2-SSLv3+NLS
> >> 
> >> Looking for examples I can follow to implement log rotation with
> >> fetchmail.   Links, examples, tutorials kindly accepted.
> > 
> > If you have the logrotate tool installed on your system, the fetchmail
> > source contains a sample fetchmail.logrotate file:
> > 
> >   
> > https://gitlab.com/fetchmail/fetchmail/-/blob/next/contrib/fetchmail.logrota 
> > te
> > 
> > Of course, you can customize it however you wish - specify different
> > rotation criteria, point to a different fetchmail logfile - but, most
> > importantly, you will most probably need to modify the command to be
> > executed after the logfile has been rotated - the contents of
> > the logrotate configuration file's "postrotate" section.
> > In the sample file, the command in the postrotate section is a POSIX
> > shell "if" statement that examines the server that it is running on and
> > figures out the appropriate command to run to make the fetchmail
> > instance that is running as a system service reopen its logfiles.
> > 
> > If you run fetchmail as root, something like that will probably work for
> > you. If you run fetchmail from a normal user account, there might be
> > a slight problem here, since the logrotate tool will need to restart
> > fetchmail. So if your fetchmail instance is run under some kind of
> > service manager (supervise, runsv, a systemd user service, etc), then
> > the postrotate section is where you place the appropriate command to
> > restart that particular service.
> > 
> > Of course, if you run fetchmail under some kind of service manager, then
> > there may be no need for a separate logfile at all, since most of
> > the service managers can already handle a program that sends messages to
> > its standard output stream, as fetchmail does by default in daemon mode.
> > So that service manager's functionality may already provide some kind of
> > log storage and rotation, and you don't need the logrotate tool.
> > 
> > Hope that helps! In short, it all depends on how you run fetchmail.
> > 
> > G'luck,
> > Peter
> > 
> > -- 
> > Peter Pentchev  ro...@ri... ro...@de... pp...@st... 
> > PGP key:        http://people.FreeBSD.org/~roam/roam.key.asc 
> > Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13
> 
> Some well strained infant food and spoon feeding may be required here.  I've never
> had to deal with logrotate before this, which may be obvious.
> 
> It is not clear to me how logrotate runs.  That would be a first spoonful.

On most installations logrotate is run automatically through cron - both
my Debian and CentOS Linux systems have an /etc/cron.daily/logrotate
file that makes sure that it runs at least once a day. When it does,
it first reads the /etc/logrotate.conf file, then it checks whether
there are any files in the /etc/logrotate.d/ directory, and it processes
all the logs and commands in all of these files. So, if you have
an /etc/logrotate.d/ directory, then placing a file similar to
fetchmail.logrotate there (with the rotate conditions and
pre/post-rotation commands updated for your specific installation) could
handle it... if you run fetchmail in a way that can be restarted
automatically. So see below.

> The linked config seems decipherable but I pause at the post rotate part as it is not apparent 
> where fetchmail was stopped prior to this.  I imagine it must be paused  to "quiesce" 
> the log.  Or is that necessary?

Not exactly. This is how logrotate works:
- logrotate reads a configuration file that says "check this logfile
  and, if it fits these criteria (it has grown too large, or too many
  days have passed since it was last rotated, or something else), then
  proceed, else skip it"
- logrotate renames the logfile while the program that writes to it
  probably still has it open
- at this point, if the program (fetchmail) wants to log more into the
  file, it will automatically write into the renamed file (the kernel
  should make sure of that)
- logrotate reads the "postrotate" section and runs the command that is
  specified there to tell the program "please stop writing to this file,
  I have renamed it, start writing to a new one"
- the program (fetchmail) starts writing to a new, currently empty,
  logfile
- logrotate compresses the renamed file

So the point of the postrotate section is to have a command that will
tell the program to stop writing to the old file and start writing to
the new one. Some programs can be told that by sending them a signal,
some don't really need that level of control or, for some reason, it may
be hard to implement (e.g. they may have already changed their user and
group ID, switched to some chroot environment, etc, so it maynot even be
possible for them to create/open a new logfile). For the latter - and
fetchmail is one of them - the postrotate section contains instructions
for the OS to restart the program altogether. This is what
fetchmail.logrotate does - its postrotate section says "find out how we
can restart the fetchmail system service and do it".

> That config post rotate stuff looks to be for init.d, while my stuff uses systemd.

This does not really matter. When systemd first appeared, there were no
programs at all that had systemd service files written for them, and
there were many programs that had SysV init scripts
(/etc/init.d/program) files already written. So what the OS packages did
was make sure that all the /etc/init.d/program files would check whether
systemd is currently running and, if so, automatically run systemctl
with the command that was passed to the SysV init script - so on most
OSs and Linux distributions that run systemd, /etc/init.d/program
stop/start/restart will still work, it will automatically pass
the command to systemctl to control the systemd service.

> I never got fetchmail to start (or maybe restart) properly, so am just
> starting with fetchmail -v (I like noisy logs).  But that is another
> story.

Right... this is actually the most important part of this conversation.
When I wrote at the end of my first message "it all depends on how you
run fetchmail", the point was that for the logrotate file to work, there
must be a way to restart fetchmail automatically, and you must put it in
the postrotate section of the logrotate file.

So if what you are saying is that you run fetchmail by hand, e.g. in
a xterm window or a screen/tmux pane, and just let it run there, then
that might be a problem, since logrotate will need to have a way to stop
it and start it again. If your system runs systemd, then it might be
possible to create a relatively simple systemd user service file that
you place in the ~/.config/systemd/user/ directory (creating that
directory if necessary). Something like this works for me (note that
I do not have a logfile directive in my fetchmail config file, so
it will send its output to the standard output stream by default):

  [roam@straylight ~]$ cat ~/.config/systemd/user/fetchmail.service 
  [Unit]
  Description=Run fetchmail in daemon mode.
  Documentation=man:fetchmail(1)
  After=network-online.target
  Wants=network-online.target
  
  [Service]
  Type=simple
  ExecStartPre=-fetchmail -ve200
  ExecStart=fetchmail -ve25 -Nd120
  ExecStop=fetchmail --quit
  KillMode=process

This is somewhat modeled on the /usr/lib/systemd/user/fetchmail.service
file that the Debian package of fetchmail already ships, but I had to
modify it a bit for my own use case, which is that sometimes I do not
have a stable Internet connection for a long time and then I'd like to
fetch a lot of mail in a single burst. If this does not apply to you,
remove the ExecStartPre line.  After creating this file, I ran:

  systemctl --user daemon-reload
  systemctl --user disable fetchmail
  journalctl --user --follow -u fetchmail.service

And then, in another shell, since journalctl took over this one:

  systemctl --user start fetchmail

...and the journalctl window started showing me fetchmail log messages.

This particular setup leads to fetchmail logging to its standard output
stream, not to a logfile (as noted above, I do not have a logfile line
in my fetchmail config file), and systemd capturing those log lines and
storing them in its own journal (that's why journalctl can show them).
If you want a separate logfile (there are many reasons people do not
necessarily like systemd's journal, some of them are valid for some use
cases), you can still specify a logfile in the fetchmail config file and
it will log to that file. In that case, if you want to rotate that file
through logrotate, there are two ways you can go about it:

- make use of the fact that logrotate runs every day as a system
  service, create an /etc/logrotate.d/fetchmail.service file, point it
  to the logfile that your fetchmail instance creates, tell it to run
  from your user account (look at its "su" config directive), and
  use `systemctl --user restart fetchmail.service` as the postrotate
  command

- find a way to run logrotate periodically from your own user account,
  either from a cron job or from a systemd timer, and either point it
  specifically at a config file for fetchmail, or at a directory that
  contains a config file for fetchmail. Again, the postrotate command
  should be "systemctl --user restart fetchmail.service".

Hope that helps!

G'luck,
Peter

-- 
Peter Pentchev  ro...@ri... ro...@de... pp...@st...
PGP key:        http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13