Re: [fetchmail-users] more clever resource limitations

[Matthias A. <mat...@gm...>]

Greg Lindahl <li...@pb...> writes:

> I was pleased to see on LWN that you guys are working on fetchmail,
> I had no idea it had active maintainers.

Good to see that the word is spreading. :-)

> Due to the advent of spamassassin, I am no longer able to accept lots
>of messages at once via fetchmail -- it causes my laptop to thrash.  I

My personal experience is that SpamAssassin's biggest hog is its Bayes
database, disabling it speeds up SA quite a bit. External filters such
as bogofilter, spamprobe or qsf are much faster than SpamAssassin, and
as you are already using procmail, integration is quite easy.

It takes a bit of manual work training the filter, though. Also, if you
use "spamassassin" in your procmail script, consider running a "spamd"
daemon and then use "spamc" in your script instead, this also gives you
some resource control given proper spamd settings.

You can also use a procmail "local lock file" to make sure only one
spamassassin process is running at a time, and you can also precede the
spamassassin line with "nice". (You'd change your ":0fw" rule to
":0fw:.spamassassin.lock" or similar, depending on how _exactly_ you use
spamassassin.)

>use sendmail -> procmail -> spamassassin, so using sendmail's load
>average limit didn't really help the situation, since sendmail has no
>idea it's kicked off a bunch of spamassassin processes that in a few
>seconds will drive the load way up. (In this analysis I'm assuming that
>it's only cpu time and not disk I/O which is causing the problem.)

The actual problem is that sendmail is somewhat limited in terms of
resource control. With Postfix for instance you'd set the
local_concurrency_limit to some low value and that's it, but as shown
above there are ways around your problem.

>  --fetchlimitsize n (kilobytes, default to 100 kilobytes? 1 meg?)
>  --expungesize n (kilobytes, same default as fetchlimitsize)
>
> In addition, we could use a rate limiter sending messages to sendmail
> to dodge the spamassassin problem. My laptop takes 2 cpu seconds to
> spamassassin a small email message (20 seconds elapsed) and so perhaps
> a time-based measure with an optional addition for the message size
> would do it. If you wanted to be clever you could compute this
> dynamically by watching what happens when you throw a bunch of email
> messages at sendmail, but to start:
>
>  --mtacpu n (seconds, default to 0)
>  --mtacpupermeg n (seconds per megabyte, default to 0)
>
> so fetchmail would wait "mtacpu + size / 1meg * mtacpupermeg" seconds
> before sending another message to the MTA. I'd set this to 2. Hm,
> given that this value is small a floating-point value would be a
> good idea instead of an integer.

As you talked about varying network connection quality, a time limit is
probably one easy way to go and it makes good sense for --expunge* like
options, too -- fetchmail might then checkpoint the mailbox every two
minutes or so (or whichever the user set).

-- 
Matthias Andree