fetchmail-users

[Fetchmail-users] WG: Running multiple fetchmail processes at the same time

[Christopher M. <C.M...@da...>]

Hello,

we have the following problem:

We are downloading emails from our mailserver through fetchmail. Right now we are downloading from 50 Mailboxes (via POP3) and have it setup that every mailbox has its own config file. For example:

fetchmail -F -f /dokuweb/log/.fetchmailrc_0042
fetchmail -F -f /dokuweb/log/.fetchmailrc_einvoice_00D4

.fetchmailrc_0042
poll XXX.XXX.XXX.XXX protocol pop3 auth password
user XXX password XXX is XXX
poll XXX.XXX.XXX.XXXX protocol pop3 auth password
user YYY password XXX is YYY

.fetchmailrc_einvoice_00D4:
poll XXX.XXX.XXX.XXX protocol pop3 auth password
user XXX password XXX is XXX


Since we are processing the emails after they downloaded we have multiple poll processes running that initiate the above commands. If one of those poll processes is initiating the fetchmail command and another one is already running we get the following message:

fetchmail: WARNING: Running as root is discouraged.
fetchmail: another foreground fetchmail is running at 28646.

Is there any way we can run multiple fetchmail processes at the same time?

Thank you very much.

________________________________
Amtsgericht Siegen HRB 4697 - Gesch?ftsf?hrer Sebastian Weber

Das Web-basierte Archivierungs- und Informationsmanagementsystem http://www.datasec.de

Oder per e-Mail / or send a mailto:co...@da...

Diese elektronisch uebermittelte Post ist nur fuer den aufgefuehrten Empfaenger bestimmt. Im Falle einer irrtuemlichen Uebermittlung bitten wir um Ruecksendung an den Absender. Wir weisen darauf hin, dass diese Sendung einschliesslich aller Anlagen insbesondere Dateianlagen unser Eigentum ist und ohne unsere schriftliche Zustimmung weder abgeaendert, kopiert noch dritten Personen zugaenglich gemacht werden darf.

This e-mail is private and confidential and should only be read by those to whom it is addressed. If you are not the intended recipient, please return this e-mail to the sender. We notify that any dissemination, distribution, copy, reproduction, modification or publication of this communication is strictly prohibited. This message is not intended to be relied upon by any person without subsequent written confirmation of its contents.

Re: [Fetchmail-users] WG: Running multiple fetchmail processes at the same time

[grarpamp <gra...@gm...>]

IIRC, try specifying --pidfile $(mktemp blah.XXXXXXXXX)

Of course pid files are foolish broken antique races, running
without them should be made possible, and daemon
monitors are better... but until then.

Re: [Fetchmail-users] WG: Running multiple fetchmail processes at the same time

[Matthias A. <mat...@gm...>]

Am 19.05.2017 um 08:38 schrieb grarpamp:
> IIRC, try specifying --pidfile $(mktemp blah.XXXXXXXXX)
>
> Of course pid files are foolish broken antique races, running
> without them should be made possible, and daemon
> monitors are better... but until then.

This does not contribute to answering the originator's questions.

Re: [Fetchmail-users] WG: Running multiple fetchmail processes at the same time

[Carlos E. R. <rob...@te...>]

On 2017-05-19 08:24, Christopher Mudersbach wrote:
> Hello,
...

> Is there any way we can run multiple fetchmail processes at the same time?

As a different user (fetch_01, fetch_02...)

See the documentation:

       Only one daemon process is permitted per user; in daemon mode,
       fetchmail sets up a per-user lockfile to  guar-
       antee  this.   (You can however cheat and set the FETCHMAILHOME
       environment variable to overcome this setting,
       but in that case, it is your responsibility to make sure you
       aren't polling the same server with two processes
       at the same time.)

I don't exactly know why not, as long as you don't poll the same
account. It should be able to poll two different accounts from the same
server, I think.

-- 
Cheers / Saludos,

		Carlos E. R.

  (from 42.2 x86_64 "Malachite" (Minas Tirith))

Re: [Fetchmail-users] WG: Running multiple fetchmail processes at the same time

[Matthias A. <mat...@gm...>]

Am 19.05.2017 um 13:25 schrieb Carlos E. R.:
> See the documentation:
>        Only one daemon process is permitted per user; in daemon mode,
>        fetchmail sets up a per-user lockfile to  guar-
>        antee  this.   (You can however cheat and set the FETCHMAILHOME
>        environment variable to overcome this setting,
>        but in that case, it is your responsibility to make sure you
>        aren't polling the same server with two processes
>        at the same time.)
>
> I don't exactly know why not, as long as you don't poll the same
> account. It should be able to poll two different accounts from the same
> server, I think.

Seconded.

The documentation could be more precise and should read "polling the
same account ... at the same time" instead of "polling the same
server...", and FETCHMAILHOME is the way to go.

The GNU "parallel" utility might then also be an easy way to launch a
limited number of concurrent processes.

Answering "Why not" is simple, with the documentation refinement shown
above:
It's useless and doesn't work right. Fetchmail isn't designed for itself
running twice on the same mailbox and would not properly synchronize its
data between its processes, so both processes might end up corrupting
each other's state, or fetching mail in duplicate, or bump into server
locks and be refused, or thereabouts.

But that would not be Christopher's approach anyways, I presume.

Re: [Fetchmail-users] WG: Running multiple fetchmail processes at the same time

[Matthias A. <mat...@gm...>]

Am 19.05.2017 um 08:24 schrieb Christopher Mudersbach:
> Since we are processing the emails after they downloaded we have multiple poll processes running that initiate the above commands. If one of those poll processes is initiating the fetchmail command and another one is already running we get the following message:
>
> fetchmail: WARNING: Running as root is discouraged.
> fetchmail: another foreground fetchmail is running at 28646.
>
> Is there any way we can run multiple fetchmail processes at the same time?

See Carlos E.R.'s message and my followup for an approach to do that.

Regarding the messages you see, you are changing two items at the same
time and drawing the wrong conclusion from the warning. The warning
above is because you've run fetchmail from a privileged account (root),
not because you're running it twice. In the latter case you'd get
different diagnostics.


Be sure that whatever runs your 50 fetchmail processes with the
individual configuration data:

1. also gives each process their own .fetchids file and .pid file
location, easiest done through the FETCHMAILHOME environment variable
(also see the env(1) manual page) which flips the whole set of
directories in one go;

2. you run fetchmail from an unprivileged user account, not root or toor
or similar.

Re: [Fetchmail-users] WG: Running multiple fetchmail processes at the same time

[Carlos E. R. <rob...@te...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



On Friday, 2017-05-19 at 20:55 +0200, Matthias Andree wrote:

> Be sure that whatever runs your 50 fetchmail processes with the
> individual configuration data:
>
> 1. also gives each process their own .fetchids file and .pid file
> location, easiest done through the FETCHMAILHOME environment variable
> (also see the env(1) manual page) which flips the whole set of
> directories in one go;
>
> 2. you run fetchmail from an unprivileged user account, not root or toor
> or similar.

I went ahead and distributed my fetchmail run in two processes, but it 
can be easily expanded, manually. I created two directories:

cer@Telcontar:~> ls -a .fetchmail_0*
.fetchmail_01:
.  ..  .fetchmailrc  .fetchmailrc~

.fetchmail_02:
.  ..  .fetchmailrc
cer@Telcontar:~>


Each directory contains one .fetchmailrc file with a part of the 
original .fetchmailrc, different on each file so the is no interference. 
In my case, one file polls gmail, which is particularly slow, and the 
other polls the rest.

Then I created a script:

+++......................
#!/bin/bash

function bucle_N()
{
   cd $HOME/.fetchmail_$1
   echo "Launching fetchmail $1"
   FETCHMAILHOME=$HOME/.fetchmail_$1 fetchmail -v --fetchmailrc .fetchmailrc
   echo "Fetchmail $1 finished."
}


echo "Launching two fetchmail processes"
date --iso-8601=seconds

bucle_N 01 &
bucle_N 02 &

echo "Waiting for the two fetchmail processes to finish"

wait

echo "All fetchmails processes have finished."
date --iso-8601=seconds
mailq | grep Requests
......................++-


Notice that it may not be worth it to download a lot of emails fast, if 
one is going to filter for spam using spamassassin, which is heavy and 
slow.

- -- 
Cheers,
        Carlos E. R.
        (from 42.2 x86_64 "Malachite" at Telcontar)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlkz7CUACgkQtTMYHG2NR9UDpQCfZybLgZ+gEXOxbBe1AuVig+vl
+scAnAxg+q2X1iJdUHTpuqaaO1XMJy2t
=K0M8
-----END PGP SIGNATURE-----

Re: [Fetchmail-users] WG: Running multiple fetchmail processes at the same time

[grarpamp <gra...@gm...>]

> (You can however cheat and set the FETCHMAILHOME
>        environment variable to overcome this setting,
>        but in that case, it is your responsibility to make sure you
>        aren't polling the same server with two processes
>        at the same time.)

Polling the same server with N processes at the same time is a
highly valid use case. There are various ways to do that today.
Having enhanced fetchmail config file semantics would help
streamline it.

Today if you're dealing with multiple accounts and servers in
various parallel polling modes / timings, including say a daemon
fetch and random check oneshots of the same account, you're almost
guaranteed to need end up writing configs on the fly and forking off
polling instances. Program instances will always be needed, but
config writing or multiple configs shouldn't be needed for them,
one config file and an address target should do.

The list archives are unfortunately not in a usable format but
there were some initial posts on this config stuff for 7.x a while back.

Re: [Fetchmail-users] WG: Running multiple fetchmail processes at the same time

[Carlos E. R. <rob...@te...>]

On 2017-05-20 12:46, grarpamp wrote:
>> (You can however cheat and set the FETCHMAILHOME
>>        environment variable to overcome this setting,
>>        but in that case, it is your responsibility to make sure you
>>        aren't polling the same server with two processes
>>        at the same time.)
> 
> Polling the same server with N processes at the same time is a
> highly valid use case. 

I assume the issue is not polling the same account. Polling the same
server should be no problem, except that fetchmail doesn't do it on its own.

> There are various ways to do that today.
> Having enhanced fetchmail config file semantics would help
> streamline it.

That would be nice :-)

-- 
Cheers / Saludos,

		Carlos E. R.

  (from 42.2 x86_64 "Malachite" (Minas Tirith))