Re: [fetchmail-devel] Re: deleting oversized messages

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

>> No need, I found a similar report (IIRC in bugs.debian.org) and
>> investigated. The upcoming fetchmail release and IIRC the latest -pre
>> release from BerliOS.de also will delete oversized messages if NOT run
>> in daemon mode - be sure to use the --flush option.
>
> The 'flush' option is very dangerous as it can even delete mails which
> have been read by another email client. In particular, if one has the
> habit of checking a remote mailbox through an interactive email
> client, new mails can get marked as read. Then, when fetchmail polls
> that mailbox, it will think that those mails have already been
> downloaded and just delete those mails due to the 'flush' option.

Nice to see you back, and to see you've found the new site.

> Here, I am assuming a setup like:
>
> poll server
>     protocol imap	# or "pop3 no uidl"
>     fetchall
>     limit 1000000
>     flush		# this has been put to delete oversized only
>
> Without the 'flush' option, mails which are read through another email
> client also get downloaded without problems. With the 'flush' option,
> such apparently read mails get deleted.

Yup. There's also a difference between "flush" and "nokeep". I believe
if it's not deliberate, then it's at least useful enough to warrant
further documentation.

> I had created a patch which adds another option to fetchmail. The old
> patch (against 6.2.4) along with some discussion can be found at:
>
> <http://lists.ccil.org/pipermail/fetchmail-friends/2003-October/008017.html>.
>
> Eric was against adding new options, hence that patch was not accepted
> then. If the idea of the above patch is acceptable, I can recreate it
> for the latest fetchmail. In any case, it would be better if the
> 'flush' option is not overloaded to delete oversized mails.

Well, I have some plans for fetchmail, and I believe your option (as
well as others) might fit in the medium term - not on short notice
though.

WRT release management, I hope to release 6.3.0 in early September, only
trouble is I have zero feedback on the *RC stuff yet. Does it work for
people? Does it fail? Do the recent IPv6 patches break anything?

Anyways, with 6.3.0 out, I will open a 6.3.X side branch for bugfixes
only, and continue development in the trunk.

The most important change will be to switch fetchmail to client-side
tracking which mail it has seen. POP3 LAST and IMAP \Seen flags will be
removed and we'll no longer care about them. If fetchmail understands
itself as mail transfer agent, this translates to "transport each message
exactly once, and do it reliably". OK, we may sometimes transfer a
message twice, but we must not skip or lose any message unless the user
explicitly tells us to discard it.

Perhaps this change already involves some database for the UIDs (perhaps
SQLite3), and I plan to reuse your "one fetchids per server" patches at
least in part, that you wrote two years ago.

This would enable us to add a "delete after N days" sort of thing later
that users have been longing for for so long. A sponsor is needed for
this work however. (ESR's fetchmail version already deletes mail and has
limited client-side tracking support ("POP3 UIDL"), so there is no
reason why it would not delete mail after some days or as the mailbox
grows too large.)

Taking a step back from the blackboard, we may need to go less from
offering fixed options with a small functionality, but offer fewer but
more flexible options. Read: interfaces where users can hook up
scripts/programs that decide which messages get deleted, perhaps in
which order messages are fetched, and such. This is a long-term plan
though.

Some details at <http://fetchmail.berlios.de/design-notes.html>.

-- 
Matthias Andree