fetchmail-devel — Development related discussion

Re: [fetchmail-devel] Re: deleting oversized messages

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> I had created a patch which adds another option to fetchmail. The old
> patch (against 6.2.4) along with some discussion can be found at:
>
> <http://lists.ccil.org/pipermail/fetchmail-friends/2003-October/008017.html>.

I had a very quick glance at this patch - I have two minor nits to pick:

1. overloading the sign as a flag is plain ugly. A size is a size and as
   such should be an unsigned type. If we need to track state, we'll add
   fields (perhaps a bitfield if we're concerned about memory
   consumption).

2. There are several places where this check, in prose an "if limitflush
   caused deletion of this message" flag, is coded. The check "if
   limitflush is effective and the message is too large" had better be
   put in a macro or function so that the whole logic about this
   function is in one central place.

   I've recently wasted some time finding out the gory "#if
   (defined(__linux__) && !defined(INET6_ENABLE)) || defined(FreeBSD)"
   or similar, and variants thereof, that pertained to the
   monitor/interface options. Not all places used the same check, so
   there were inconsistencies, and about half a dozen files would have
   to change should someone, for instance, provide --interface support
   for NetBSD or Solaris. I'm still not too happy with the way it is
   now, at least it's only in fetchmail.h and interface.c.

-- 
Matthias Andree

Re: [fetchmail-devel] Re: deleting oversized messages

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> Rather well put. However, this is not the only thing I am after.
>
> Please check the revision r4173 which attempts to fix Debian Bug
> #212240. This fix actually encourages the use of the 'flush'
> option.
>
> It is contradictory to say that
>
> $ fetchmail --flush
>
> is dangerous, but 
>
> $ fetchmail --daemon 0 --limit 100000 --flush
>
> is ok. In both cases, mails read through another email client are
> going to get deleted. Thus, both the above usages of 'flush' are
> eqaually dangerous.
>
> What I want us to do:
>
> - reverse the commit in r4173: this will break the link between
>   'flush' and 'limit'.
>
> - add a FAQ item which tells that 'flush' option cannot be overloaded
>   to delete oversized mails as 'flush' is dangerous (with or without
>   the overloading) when one is using another email client to check
>   one's mailbox and that 'flush' should be immediately removed from
>   the rc file.
>
> - treat Debian Bug #212240 as a new feature request, not a bug.
>
> - decide whether this request should should be implemented before
>   6.3.0.

> In case, it is decided to implement now:
>
> - add a new option which will delete only oversized mails. The patch I
>   had mentioned earlier did exactly that.

> - update the FAQ item mentioned above.

Thinking about the options (yet another would be to ignore --flush for
all configurations except POP3 UIDL), all are more or less ugly, even a
flushoversized option or however you called it in your patch (haven't
had time yet to look at it). Some people may want to delete oversized
messages depending on mailbox size, some depending on age...

-- 
Matthias Andree

[fetchmail-devel] Re: deleting oversized messages

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Wed, Aug 31, 2005 at 01:04:04AM +0200:
> Current wording, in roff man(7) format:
> 
> .TP
> .B \-F | \-\-flush
> POP3/IMAP only.  This is a dangerous option and can cause mail loss when
> used improperly. It deletes old (seen) messages from the mailserver
> before retrieving new messages.  Warning: This can cause mail loss if
> you check your mail with other clients than fetchmail, and cause
> fetchmail to delete a message it had never fetched before. Similarly, if
> your local MTA hangs and fetchmail is aborted, the next time you run
> fetchmail, it will delete mail that was never delivered to you.  You
> should probably not use this option in your configuration file.
> What you probably want is the default setting: if you don't specify
> '-k', then fetchmail will automatically delete messages after successful
> delivery. This option does not work with ETRN and ODMR.

Rather well put. However, this is not the only thing I am after.

Please check the revision r4173 which attempts to fix Debian Bug
#212240. This fix actually encourages the use of the 'flush'
option.

It is contradictory to say that

$ fetchmail --flush

is dangerous, but 

$ fetchmail --daemon 0 --limit 100000 --flush

is ok. In both cases, mails read through another email client are
going to get deleted. Thus, both the above usages of 'flush' are
eqaually dangerous.

What I want us to do:

- reverse the commit in r4173: this will break the link between
  'flush' and 'limit'.

- add a FAQ item which tells that 'flush' option cannot be overloaded
  to delete oversized mails as 'flush' is dangerous (with or without
  the overloading) when one is using another email client to check
  one's mailbox and that 'flush' should be immediately removed from
  the rc file.

- treat Debian Bug #212240 as a new feature request, not a bug.

- decide whether this request should should be implemented before
  6.3.0.

In case, it is decided to implement now:

- add a new option which will delete only oversized mails. The patch I
  had mentioned earlier did exactly that.

- update the FAQ item mentioned above.

-- 
Sunil Shetye.

Re: [fetchmail-devel] Re: deleting oversized messages

[Rob F. <rf...@fu...>]

Matthias Andree wrote:
> So for the time being, until we fix IMAP to use client-side tracking
> and get rid of all server-side tracking junk in POP3 and IMAP,

While I agree that the major goal post-6.3.0 should be to fix UIDL for 
POP3 and get rid of the server-side tracking assumptions there, IMAP is a 
different story.  The whole IMAP protocol is oriented toward server-side 
state, so it seems appropriate to use it, while allowing the option of 
using client-side state instead.

Though I might just need to read this conversation more carefully.


BTW, welcome back Sunil.  :-)

-- 
==============================|   "A microscope locked in on one point
 Rob Funk <rf...@fu...> |Never sees what kind of room that it's in"
 http://www.funknet.net/rfunk |    -- Chris Mars, "Stuck in Rewind"

Re: [fetchmail-devel] Re: deleting oversized messages

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> Quoting from Matthias Andree's mail on Tue, Aug 30, 2005 at 10:09:19AM +0200:
>> > Here, I am assuming a setup like:
>> >
>> > poll server
>> >     protocol imap	# or "pop3 no uidl"
>> >     fetchall
>> >     limit 1000000
>> >     flush		# this has been put to delete oversized only
>> >
>> > Without the 'flush' option, mails which are read through another email
>> > client also get downloaded without problems. With the 'flush' option,
>> > such apparently read mails get deleted.
>> 
>> Yup. There's also a difference between "flush" and "nokeep". I believe
>> if it's not deliberate, then it's at least useful enough to warrant
>> further documentation.
>
> I think there is some misunderstanding here. The change made in r4173
> is reintroducing a bug which had been removed and encouraging users to
> use the 'flush' option without documenting that even mails under
> 'limit' can get lost permanently.
>
> The 'flush' option is dangerous because it can delete mails which have
> never been downloaded previously by fetchmail.

> The 'flush' option is supposed to delete previously downloaded mail.
> fetchmail thinks a mail has been previously downloaded if its \Seen
> flag is set. Thus, the 'flush' option will delete all \Seen mails.

I don't think there is a misunderstanding. I am well aware that --flush
is dangerous, and I think it can bear some more explicit wording that it
is. Perhaps I see the problem smaller than it currently is because I
don't subscribe to ESR's pushing people towards IMAP, and I am only
using fetchmail for POP3. I do need fetchmail-side tracking of seen mail
for fetchmail, but I don't see this happen in 6.3.0.

> However, when a user uses another email client to check the users'
> remote mailbox, the \Seen flag gets set here too. In this case, the
> mails which have been viewed remotely but not downloaded yet too will
> get deleted. In short, the 'flush' option will in this case delete
> mails which have not been downloaded even once.

Right.

> In r4173, the 'flush' option is being connected to the 'limit' option.
> But, the point is if a user add 'flush' to the rc file (or, on the
> command line) just to delete oversized mails, mails seen through other
> email clients can get deleted without any warning.
>
> [The above comments are applicable only when uidl is off. Obviously,
> when uidl is used, fetchmail does not use the server side \Seen flag
> at all.]

Unfortunately, these preclude one another. The UIDL option only works
for POP3 which does not know \Seen flags, IMAP on the other hand does
not support UIDL at all.

We seem to agree that we both want fetchmail to track, on its inbound
side, which messages it has fetched and which it has not.

So for the time being, until we fix IMAP to use client-side tracking and
get rid of all server-side tracking junk in POP3 and IMAP, we should
just document this problem in the --flush section, and put a pointer
into the BUGS section.

Current wording, in roff man(7) format:

.TP
.B \-F | \-\-flush
POP3/IMAP only.  This is a dangerous option and can cause mail loss when
used improperly. It deletes old (seen) messages from the mailserver
before retrieving new messages.  Warning: This can cause mail loss if
you check your mail with other clients than fetchmail, and cause
fetchmail to delete a message it had never fetched before. Similarly, if
your local MTA hangs and fetchmail is aborted, the next time you run
fetchmail, it will delete mail that was never delivered to you.  You
should probably not use this option in your configuration file.
What you probably want is the default setting: if you don't specify
'-k', then fetchmail will automatically delete messages after successful
delivery. This option does not work with ETRN and ODMR.

-- 
Matthias Andree

[fetchmail-devel] Re: deleting oversized messages

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Tue, Aug 30, 2005 at 10:09:19AM +0200:
> > Here, I am assuming a setup like:
> >
> > poll server
> >     protocol imap	# or "pop3 no uidl"
> >     fetchall
> >     limit 1000000
> >     flush		# this has been put to delete oversized only
> >
> > Without the 'flush' option, mails which are read through another email
> > client also get downloaded without problems. With the 'flush' option,
> > such apparently read mails get deleted.
> 
> Yup. There's also a difference between "flush" and "nokeep". I believe
> if it's not deliberate, then it's at least useful enough to warrant
> further documentation.

I think there is some misunderstanding here. The change made in r4173
is reintroducing a bug which had been removed and encouraging users to
use the 'flush' option without documenting that even mails under
'limit' can get lost permanently.

The 'flush' option is dangerous because it can delete mails which have
never been downloaded previously by fetchmail.

The 'flush' option is supposed to delete previously downloaded mail.
fetchmail thinks a mail has been previously downloaded if its \Seen
flag is set. Thus, the 'flush' option will delete all \Seen mails.

However, when a user uses another email client to check the users'
remote mailbox, the \Seen flag gets set here too. In this case, the
mails which have been viewed remotely but not downloaded yet too will
get deleted. In short, the 'flush' option will in this case delete
mails which have not been downloaded even once.

In r4173, the 'flush' option is being connected to the 'limit' option.
But, the point is if a user add 'flush' to the rc file (or, on the
command line) just to delete oversized mails, mails seen through other
email clients can get deleted without any warning.

[The above comments are applicable only when uidl is off. Obviously,
when uidl is used, fetchmail does not use the server side \Seen flag
at all.]

-- 
Sunil Shetye.

Re: [fetchmail-devel] Re: deleting oversized messages

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

>> No need, I found a similar report (IIRC in bugs.debian.org) and
>> investigated. The upcoming fetchmail release and IIRC the latest -pre
>> release from BerliOS.de also will delete oversized messages if NOT run
>> in daemon mode - be sure to use the --flush option.
>
> The 'flush' option is very dangerous as it can even delete mails which
> have been read by another email client. In particular, if one has the
> habit of checking a remote mailbox through an interactive email
> client, new mails can get marked as read. Then, when fetchmail polls
> that mailbox, it will think that those mails have already been
> downloaded and just delete those mails due to the 'flush' option.

Nice to see you back, and to see you've found the new site.

> Here, I am assuming a setup like:
>
> poll server
>     protocol imap	# or "pop3 no uidl"
>     fetchall
>     limit 1000000
>     flush		# this has been put to delete oversized only
>
> Without the 'flush' option, mails which are read through another email
> client also get downloaded without problems. With the 'flush' option,
> such apparently read mails get deleted.

Yup. There's also a difference between "flush" and "nokeep". I believe
if it's not deliberate, then it's at least useful enough to warrant
further documentation.

> I had created a patch which adds another option to fetchmail. The old
> patch (against 6.2.4) along with some discussion can be found at:
>
> <http://lists.ccil.org/pipermail/fetchmail-friends/2003-October/008017.html>.
>
> Eric was against adding new options, hence that patch was not accepted
> then. If the idea of the above patch is acceptable, I can recreate it
> for the latest fetchmail. In any case, it would be better if the
> 'flush' option is not overloaded to delete oversized mails.

Well, I have some plans for fetchmail, and I believe your option (as
well as others) might fit in the medium term - not on short notice
though.

WRT release management, I hope to release 6.3.0 in early September, only
trouble is I have zero feedback on the *RC stuff yet. Does it work for
people? Does it fail? Do the recent IPv6 patches break anything?

Anyways, with 6.3.0 out, I will open a 6.3.X side branch for bugfixes
only, and continue development in the trunk.

The most important change will be to switch fetchmail to client-side
tracking which mail it has seen. POP3 LAST and IMAP \Seen flags will be
removed and we'll no longer care about them. If fetchmail understands
itself as mail transfer agent, this translates to "transport each message
exactly once, and do it reliably". OK, we may sometimes transfer a
message twice, but we must not skip or lose any message unless the user
explicitly tells us to discard it.

Perhaps this change already involves some database for the UIDs (perhaps
SQLite3), and I plan to reuse your "one fetchids per server" patches at
least in part, that you wrote two years ago.

This would enable us to add a "delete after N days" sort of thing later
that users have been longing for for so long. A sponsor is needed for
this work however. (ESR's fetchmail version already deletes mail and has
limited client-side tracking support ("POP3 UIDL"), so there is no
reason why it would not delete mail after some days or as the mailbox
grows too large.)

Taking a step back from the blackboard, we may need to go less from
offering fixed options with a small functionality, but offer fewer but
more flexible options. Read: interfaces where users can hook up
scripts/programs that decide which messages get deleted, perhaps in
which order messages are fetched, and such. This is a long-term plan
though.

Some details at <http://fetchmail.berlios.de/design-notes.html>.

-- 
Matthias Andree

[fetchmail-devel] Re: deleting oversized messages

[Sunil S. <sh...@bo...>]

Hi,

Quoting from Matthias Andree's mail on Sun, Aug 21, 2005 at 11:12:45PM +0200:
> Rob MacGregor <rob...@gm...> writes:
> 
> > On 21/08/05, David Laight <dav...@l8...> wrote:
> >> 
> >> But, IIRC, the code got broken at some point.
> >> ISTR patching my local copy.
> >
> > Then report the bug to the fetchmail-devel list, detailing what you
> > believe is broken.
> 
> No need, I found a similar report (IIRC in bugs.debian.org) and
> investigated. The upcoming fetchmail release and IIRC the latest -pre
> release from BerliOS.de also will delete oversized messages if NOT run
> in daemon mode - be sure to use the --flush option.

The 'flush' option is very dangerous as it can even delete mails which
have been read by another email client. In particular, if one has the
habit of checking a remote mailbox through an interactive email
client, new mails can get marked as read. Then, when fetchmail polls
that mailbox, it will think that those mails have already been
downloaded and just delete those mails due to the 'flush' option.

Here, I am assuming a setup like:

poll server
    protocol imap	# or "pop3 no uidl"
    fetchall
    limit 1000000
    flush		# this has been put to delete oversized only

Without the 'flush' option, mails which are read through another email
client also get downloaded without problems. With the 'flush' option,
such apparently read mails get deleted.

I had created a patch which adds another option to fetchmail. The old
patch (against 6.2.4) along with some discussion can be found at:

<http://lists.ccil.org/pipermail/fetchmail-friends/2003-October/008017.html>.

Eric was against adding new options, hence that patch was not accepted
then. If the idea of the above patch is acceptable, I can recreate it
for the latest fetchmail. In any case, it would be better if the
'flush' option is not overloaded to delete oversized mails.

-- 
Sunil Shetye.

Re: [fetchmail-devel] Fetchmail From space or not?

[Matthias A. <mat...@gm...>]

Nico Golde <ni...@ng...> writes:

> --- sink.c      2005-08-15 18:54:07.000000000 +0200
> +++ la.c        2005-08-15 18:54:58.000000000 +0200
> @@ -721,7 +721,7 @@
>
>      /* see the ap computation under the SMTP branch */
>      fprintf(sinkfp,
> -           "MAIL FROM:%s", (msg->return_path[0]) ? msg->return_path : user);
> +           "MAIL FROM:<%s>", (msg->return_path[0]) ? msg->return_path : user);

Thanks for your patch.  However, the interactive SMTP sink checks if it
needs to add angle brackets, so let's use the same in BSMTP for
consistency. (Actually, the output drivers should be merged so there's
only one SMTP/LMTP/BSMTP sink that talks to either file or socket. But
that's a major change and for later, i. e. post 6.3.0.)

-- 
Matthias Andree

Re: [fetchmail-devel] Fetchmail From space or not?

[Matthias A. <mat...@gm...>]

Nico Golde <ni...@ng...> writes:

> Hi,
>
> I received this bug message:
> When i do 'fetchmail --bsmtp test.mail'
> test.mail looks like that:
>
> MAIL FROM:us...@us...
> RCPT TO: us...@us...
> DATA..
>
> Before 6.2.5.2 there was a space after the from.
> This is caused by:
> -           "MAIL FROM: %s", (msg->return_path[0]) ? msg->return_path : user);
> +           "MAIL FROM:%s", (msg->return_path[0]) ? msg->return_path : user);
>
> Why was this space removed?

Protocol conformance.  The BSMTP writer is b0rked in 6.2.5.2, it lacks
the angle brackets, too.  What a mess.

Anyways, the next 6.2.6-pre snapshot and the SVN repo versions have this
fixed, too.

Here's the patch from SVN, I don't know if it works for 6.2.5.2:

Index: sink.c
===================================================================
--- sink.c	(Revision 4234)
+++ sink.c	(Revision 4237)
@@ -713,6 +713,7 @@
 /* open a BSMTP stream */
 {
     struct	idlist *idp;
+    int		need_anglebrs;
 
     if (strcmp(ctl->bsmtp, "-") == 0)
 	sinkfp = stdout;
@@ -720,8 +721,12 @@
 	sinkfp = fopen(ctl->bsmtp, "a");
 
     /* see the ap computation under the SMTP branch */
-    fprintf(sinkfp, 
-	    "MAIL FROM:%s", (msg->return_path[0]) ? msg->return_path : user);
+    need_anglebrs = (msg->return_path[0] != '<');
+    fprintf(sinkfp,
+	    "MAIL FROM:%s%s%s",
+	    need_anglebrs ? "<" : "",
+	    (msg->return_path[0]) ? msg->return_path : user,
+	    need_anglebrs ? ">" : "");
 
     if (ctl->pass8bits || (ctl->mimemsg & MSG_IS_8BIT))
 	fputs(" BODY=8BITMIME", sinkfp);
@@ -746,7 +751,7 @@
     for (idp = msg->recipients; idp; idp = idp->next)
 	if (idp->val.status.mark == XMIT_ACCEPT)
 	{
-	    fprintf(sinkfp, "RCPT TO: %s\r\n",
+	    fprintf(sinkfp, "RCPT TO:<%s>\r\n",
 		rcpt_address (ctl, idp->id, 1));
 	    (*good_addresses)++;
 	}


-- 
Matthias Andree

Re: [fetchmail-devel] Fetchmail From space or not?

[Nico G. <ni...@ng...>]

Hi,
* Rob Funk <rf...@fu...> [2005-08-15 18:48]:
> Nico Golde wrote:
> > I received this bug message:
> > When i do 'fetchmail --bsmtp test.mail'
> > test.mail looks like that:
> >
> > MAIL FROM:us...@us...
> > RCPT TO: us...@us...
> > DATA..
> 
> There are supposed to be <angle brackets> around those email addresses.  
> I'm a bit concerned that they aren't there.  It should look like:
> 
> MAIL FROM:<us...@us...>
> RCPT TO:<us...@us...>

--- sink.c      2005-08-15 18:54:07.000000000 +0200
+++ la.c        2005-08-15 18:54:58.000000000 +0200
@@ -721,7 +721,7 @@

     /* see the ap computation under the SMTP branch */
     fprintf(sinkfp,
-           "MAIL FROM:%s", (msg->return_path[0]) ? msg->return_path : user);
+           "MAIL FROM:<%s>", (msg->return_path[0]) ? msg->return_path : user);

     if (ctl->pass8bits || (ctl->mimemsg & MSG_IS_8BIT))
        fputs(" BODY=8BITMIME", sinkfp);
@@ -746,7 +746,7 @@
     for (idp = msg->recipients; idp; idp = idp->next)
        if (idp->val.status.mark == XMIT_ACCEPT)
        {
-           fprintf(sinkfp, "RCPT TO: %s\r\n",
+           fprintf(sinkfp, "RCPT TO:<%s>\r\n",
                rcpt_address (ctl, idp->id, 1));
            (*good_addresses)++;
        }

Regards Nico

-- 
Nico Golde - JAB: ni...@ja... | GPG: 0x73647CFF
http://www.ngolde.de | http://www.muttng.org | http://grml.org 
VIM has two modes - the one in which it beeps 
and the one in which it doesn't -- encrypted mail preferred

Re: [fetchmail-devel] Fetchmail From space or not?

[Rob F. <rf...@fu...>]

Nico Golde wrote:
> I received this bug message:
> When i do 'fetchmail --bsmtp test.mail'
> test.mail looks like that:
>
> MAIL FROM:us...@us...
> RCPT TO: us...@us...
> DATA..

There are supposed to be <angle brackets> around those email addresses.  
I'm a bit concerned that they aren't there.  It should look like:

MAIL FROM:<us...@us...>
RCPT TO:<us...@us...>

-- 
==============================|   "A microscope locked in on one point
 Rob Funk <rf...@fu...> |Never sees what kind of room that it's in"
 http://www.funknet.net/rfunk |    -- Chris Mars, "Stuck in Rewind"

Re: [fetchmail-devel] Fetchmail From space or not?

[Nico G. <ni...@ng...>]

Hi,
* Nico Golde <ni...@ng...> [2005-08-15 14:58]:
> I received this bug message:
> When i do 'fetchmail --bsmtp test.mail'
> test.mail looks like that:
> 
> MAIL FROM:us...@us...
> RCPT TO: us...@us...
> DATA..
> 
> Before 6.2.5.2 there was a space after the from.
> This is caused by:
> -           "MAIL FROM: %s", (msg->return_path[0]) ? msg->return_path : user);
> +           "MAIL FROM:%s", (msg->return_path[0]) ? msg->return_path : user);
> 
> Why was this space removed?

Ah ok, it was because of cyrus.
Regards Nico
-- 
Nico Golde - JAB: ni...@ja... | GPG: 0x73647CFF
http://www.ngolde.de | http://www.muttng.org | http://grml.org 
VIM has two modes - the one in which it beeps 
and the one in which it doesn't -- encrypted mail preferred

[fetchmail-devel] Fetchmail From space or not?

[Nico G. <ni...@ng...>]

Hi,

I received this bug message:
When i do 'fetchmail --bsmtp test.mail'
test.mail looks like that:

MAIL FROM:us...@us...
RCPT TO: us...@us...
DATA..

Before 6.2.5.2 there was a space after the from.
This is caused by:
-           "MAIL FROM: %s", (msg->return_path[0]) ? msg->return_path : user);
+           "MAIL FROM:%s", (msg->return_path[0]) ? msg->return_path : user);

Why was this space removed?
Regards Nico

-- 
Nico Golde - JAB: ni...@ja... | GPG: 0x73647CFF
http://www.ngolde.de | http://www.muttng.org | http://grml.org 
VIM has two modes - the one in which it beeps 
and the one in which it doesn't -- encrypted mail preferred

Re: [fetchmail-devel] [art666@poczta.onet.pl: Bug#322683: fetchmail can't get some "special" messages from pop3 o2.pl]

[Matthias A. <mat...@gm...>]

Brian Candler <B.C...@po...> writes:

> You're quite right. If a server sends a dot on a line of its own without
> dot-stuffing, then (a) the server is broken, and (b) there's nothing *any*
> POP3 client can do about it.

Well, it should be possible to work around this in many cases on systems
with 4.4BSD or IEEE Std 1003.1-2001 compatible stacks, and I have some
ideas how to do it -- we can peek if there's data behind DOT CR LF and
if it looks like some POP3 protocol (in response to further commands we
might have sent) response or not. This assumes that the .\r\n doesn't
fall on a packet boundary.

I'm not doing this now, 6.3.0 is long overdue, with some bug feedback
pending, and after 6.3.0 I'd request payment to add such features that
make up for somebody else's mistakes. It's not fetchmail's fault after all.

-- 
Matthias Andree

Re: [fetchmail-devel] [art666@poczta.onet.pl: Bug#322683: fetchmail can't get some "special" messages from pop3 o2.pl]

[Brian C. <B.C...@po...>]

On Fri, Aug 12, 2005 at 12:45:27PM -0400, Rob Funk wrote:
> Brian Candler wrote:
> > IMO this is definitely a fetchmail bug.
> >
> > fetchmail should read data until \r\n.\r\n which marks the end of the
> > message. Even though RFC1939 insists that the size from a 'LIST'
> > command needs to be calculated exactly, it is bizarre for a client to
> > perform a LIST command, then issue a RETR, then blindly read that
> > number of bytes, when it could just scan the stream for \r\n.\r\n in
> > the first place.
> > 
> > Why does fetchmail do this silly thing?
> 
> Um, what the bug reporter showed was that does look for \r\n.\r\n and 
> finds it, but the server is sending it too early because it's in the 
> message and the server is not escaping it.
> 
> In other words, fetchmail is doing exactly what you say it should be 
> doing, but the server seems to expect fetchmail to do what you say is 
> silly.  (And I agree with you that it would be silly.)

Oops, sorry :-)

You're quite right. If a server sends a dot on a line of its own without
dot-stuffing, then (a) the server is broken, and (b) there's nothing *any*
POP3 client can do about it.

Cheers,

Brian.

Re: [fetchmail-devel] [art666@poczta.onet.pl: Bug#322683: fetchmail can't get some "special" messages from pop3 o2.pl]

[Rob F. <rf...@fu...>]

Brian Candler wrote:
> IMO this is definitely a fetchmail bug.
>
> fetchmail should read data until \r\n.\r\n which marks the end of the
> message. Even though RFC1939 insists that the size from a 'LIST'
> command needs to be calculated exactly, it is bizarre for a client to
> perform a LIST command, then issue a RETR, then blindly read that
> number of bytes, when it could just scan the stream for \r\n.\r\n in
> the first place.
> 
> Why does fetchmail do this silly thing?

Um, what the bug reporter showed was that does look for \r\n.\r\n and 
finds it, but the server is sending it too early because it's in the 
message and the server is not escaping it.

In other words, fetchmail is doing exactly what you say it should be 
doing, but the server seems to expect fetchmail to do what you say is 
silly.  (And I agree with you that it would be silly.)

-- 
==============================|   "A microscope locked in on one point
 Rob Funk <rf...@fu...> |Never sees what kind of room that it's in"
 http://www.funknet.net/rfunk |    -- Chris Mars, "Stuck in Rewind"

Re: [fetchmail-devel] [art666@poczta.onet.pl: Bug#322683: fetchmail can't get some "special" messages from pop3 o2.pl]

[Brian C. <B.C...@po...>]

On Fri, Aug 12, 2005 at 10:14:21AM +0200, Nico Golde wrote:
> 1 list dla artii at poczta.o2.pl. (5193 octets).
> fetchmail: POP3> LIST 1
> fetchmail: POP3< +OK 1 5193
> fetchmail: POP3> RETR 1
> fetchmail: POP3< +OK 
> fetchmail: SMTP> MAIL
> FROM:<bounce-debian-security=artii=o2...@li...> BODY=8BITMIME
> SIZE=5193
> fetchmail: SMTP< 250 Ok
> fetchmail: SMTP> RCPT TO:<ar...@lo...>
> fetchmail: SMTP< 250 Ok
> fetchmail: SMTP> DATA
> fetchmail: SMTP< 354 End data with <CR><LF>.<CR><LF>
> #********************fetchmail: message ar...@po...:1 was not the
> expected length (4411 actual != 5193 expected)

IMO this is definitely a fetchmail bug.

fetchmail should read data until \r\n.\r\n which marks the end of the
message. Even though RFC1939 insists that the size from a 'LIST' command
needs to be calculated exactly, it is bizarre for a client to perform a LIST
command, then issue a RETR, then blindly read that number of bytes, when it
could just scan the stream for \r\n.\r\n in the first place.

Why does fetchmail do this silly thing?

Regards,

Brian.

Re: [fetchmail-devel] [art666@poczta.onet.pl: Bug#322683: fetchmail can't get some "special" messages from pop3 o2.pl]

[Rob F. <rf...@fu...>]

Nico Golde wrote:
> * Rob Funk <rf...@fu...> [2005-08-12 15:01]:
> > Looks like a server problem

> so you think closing the bug is ok?

Probably.  Depends on whether you subscribe to the philosophy that 
fetchmail should work around server bugs it encounters.

One problem here is that we have contradictory workarounds -- fetchmail 
could handle this by using the server's reported message size rather than 
using the dot (and that may be what the authors of the o2.pl server 
expect the client to do), but fetchmail doesn't use the reported message 
size because some servers report the wrong message size (usually due to 
EOL issues).

-- 
==============================|   "A microscope locked in on one point
 Rob Funk <rf...@fu...> |Never sees what kind of room that it's in"
 http://www.funknet.net/rfunk |    -- Chris Mars, "Stuck in Rewind"

Re: [fetchmail-devel] [art666@poczta.onet.pl: Bug#322683: fetchmail can't get some "special" messages from pop3 o2.pl]

[Nico G. <ni...@ng...>]

Hallo Rob,

* Rob Funk <rf...@fu...> [2005-08-12 15:01]:
> > #********************fetchmail: message ar...@po...:1 was not
> > the expected length (4411 actual != 5193 expected)
> > fetchmail: SMTP>. (EOM)
> > fetchmail: SMTP< 250 Ok: queued as 75E0960E3
> >  skasowany
> > fetchmail: POP3> DELE 1
> > fetchmail: POP3< No insecure packages installed would generate no
> >  output and thus no email.
> 
> > i use telnet to connect and interesting part of mesage :
> > [start]
> > One could then run a cronjob (or whatever) for dpkg-secure and it
> > would report any of the packages that are both installed and have a
> > security-tagged bug assosiated with it.
> > The result, of course, would end up in whomever crond emails it's
> > output to=
> > .
> > No insecure packages installed would generate no output and thus no
> > email.
> >
> > Maybe there could be two states?  "Insecure, unpatched" and "insecure,
> > patc=
> > hed"?
> > [stop]
> >
> > q: this is fetchmail problem or server o2.pl problem?
> 
> Looks like a server problem -- the lone dot on a line is supposed to 
> indicate the end of the message, and if it appears in the message the 
> server is supposed to make it two dots.  It's in the message just because 
> of an unfortunate line break in its quoted-printable encoding.
> 
> Unfortunately there's not even any indication of what server software 
> they're running.  The greeting flag is just:
> +OK POP3 Ready poczta.o2.pl

so you think closing the bug is ok?
regards nico
-- 
Nico Golde - JAB: ni...@ja... | GPG: 0x73647CFF
http://www.ngolde.de | http://www.muttng.org | http://grml.org 
VIM has two modes - the one in which it beeps 
and the one in which it doesn't -- encrypted mail preferred

Re: [fetchmail-devel] [art666@poczta.onet.pl: Bug#322683: fetchmail can't get some "special" messages from pop3 o2.pl]

[Rob F. <rf...@fu...>]

> #********************fetchmail: message ar...@po...:1 was not
> the expected length (4411 actual != 5193 expected)
> fetchmail: SMTP>. (EOM)
> fetchmail: SMTP< 250 Ok: queued as 75E0960E3
>  skasowany
> fetchmail: POP3> DELE 1
> fetchmail: POP3< No insecure packages installed would generate no
>  output and thus no email.

> i use telnet to connect and interesting part of mesage :
> [start]
> One could then run a cronjob (or whatever) for dpkg-secure and it
> would report any of the packages that are both installed and have a
> security-tagged bug assosiated with it.
> The result, of course, would end up in whomever crond emails it's
> output to=
> .
> No insecure packages installed would generate no output and thus no
> email.
>
> Maybe there could be two states?  "Insecure, unpatched" and "insecure,
> patc=
> hed"?
> [stop]
>
> q: this is fetchmail problem or server o2.pl problem?

Looks like a server problem -- the lone dot on a line is supposed to 
indicate the end of the message, and if it appears in the message the 
server is supposed to make it two dots.  It's in the message just because 
of an unfortunate line break in its quoted-printable encoding.

Unfortunately there's not even any indication of what server software 
they're running.  The greeting flag is just:
+OK POP3 Ready poczta.o2.pl

-- 
==============================|   "A microscope locked in on one point
 Rob Funk <rf...@fu...> |Never sees what kind of room that it's in"
 http://www.funknet.net/rfunk |    -- Chris Mars, "Stuck in Rewind"

[fetchmail-devel] [art666@poczta.onet.pl: Bug#322683: fetchmail can't get some "special" messages from pop3 o2.pl]

[Nico G. <ni...@ng...>]

Hi,
any idea?
Regrds Nico
----- Forwarded message from "Artur Polaczynski (art)" <ar...@po...> -----

Subject: Bug#322683: fetchmail can't get some "special" messages from pop3
    o2.pl
Resent-Date: Fri, 12 Aug 2005 08:03:05 UTC
From: "Artur Polaczynski (art)" <ar...@po...>
To: Debian Bug Tracking System <su...@bu...>
X-Mailer: reportbug 3.15

Package: fetchmail
Version: 6.2.5-18
Severity: important

fetchmail can't get some "special" messages from pop3 o2.pl
i have 3 "special" mesages that can't get from this server 
other messages: no problem 

log from fetchmail -v 
[snip]
fetchmail: POP3> STAT
fetchmail: POP3< +OK 1 5193
1 list dla artii at poczta.o2.pl. (5193 octets).
fetchmail: POP3> LIST 1
fetchmail: POP3< +OK 1 5193
fetchmail: POP3> RETR 1
fetchmail: POP3< +OK 
fetchmail: SMTP> MAIL
FROM:<bounce-debian-security=artii=o2...@li...> BODY=8BITMIME
SIZE=5193
fetchmail: SMTP< 250 Ok
fetchmail: SMTP> RCPT TO:<ar...@lo...>
fetchmail: SMTP< 250 Ok
fetchmail: SMTP> DATA
fetchmail: SMTP< 354 End data with <CR><LF>.<CR><LF>
#********************fetchmail: message ar...@po...:1 was not the
expected length (4411 actual != 5193 expected)
fetchmail: SMTP>. (EOM)
fetchmail: SMTP< 250 Ok: queued as 75E0960E3
 skasowany
fetchmail: POP3> DELE 1
fetchmail: POP3< No insecure packages installed would generate no
 output and thus no email.
fetchmail: POP3> QUIT
fetchmail: POP3< 
fetchmail: b??d protoko?u mi?dzy serwerem i klientem podczas pobierania
listów z poczta.o2.pl
fetchmail: 6.2.5 querying poczta.o2.pl (protocol POP3) at pi? 12 sie
2005 09:28:14 CEST: poll completed
fetchmail: Query status=4 (PROTOCOL)

i use telnet to connect and interesting part of mesage :
[start]
One could then run a cronjob (or whatever) for dpkg-secure and it
would report any of the packages that are both installed and have a
security-tagged bug assosiated with it.
The result, of course, would end up in whomever crond emails it's output
to=
.
No insecure packages installed would generate no output and thus no
email.

Maybe there could be two states?  "Insecure, unpatched" and "insecure,
patc=
hed"?
[stop]

q: this is fetchmail problem or server o2.pl problem?

-- System Information:
Debian Release: testing/unstable
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'stable')
Architecture: i386 (i686)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.11-686-art
Locale: LANG=C, LC_CTYPE=C (charmap=ISO-8859-2) (ignored: LC_ALL set to pl_PL)

Versions of packages fetchmail depends on:
ii  adduser                       3.66       Add and remove users and groups
ii  base-files                    3.1.6-1    Debian base system miscellaneous f
ii  debianutils                   2.14.2     Miscellaneous utilities specific t
ii  libc6                         2.3.5-3    GNU C Library: Shared libraries an
ii  libssl0.9.7                   0.9.7g-2   SSL shared libraries

Versions of packages fetchmail recommends:
ii  ca-certificates               20050804   Common CA Certificates PEM files

-- debconf information:
* fetchmail/confwarn:
* fetchmail/systemwide: false
  fetchmail/initdefaultswarn:
  fetchmail/runasroot: false
  fetchmail/fetchidswarn:



----- End forwarded message -----

-- 
Nico Golde - JAB: ni...@ja... | GPG: 0x73647CFF
http://www.ngolde.de | http://www.muttng.org | http://grml.org 
VIM has two modes - the one in which it beeps 
and the one in which it doesn't -- encrypted mail preferred

[fetchmail-devel] New Russian PO file for `fetchmail'

[Translation P. R. <tra...@ir...>]

Hello, gentle maintainer. This is a message from the Translation
Project robot.

A revised PO file, for programs using the textual domain `fetchmail',
has been submitted by the team of translators taking care of the
Russian language.  This particular file, along with all other PO files
pertaining to the same textual domain, is available as:
>    http://www.iro.umontreal.ca/translation/maint/fetchmail/ru.po

The file should soon be made available in mirror sites as:

>    ftp://ftp.unex.es/pub/gnu-i18n/po/maint/fetchmail/ru.po
>    http://translation.sf.net/maint/fetchmail/ru.po
>    ftp://tiger.informatik.hu-berlin.de/pub/po/maint/fetchmail/ru.po

This file has already been sent to you separately on 2005-08-04,
as a MIME invoice unpacking the file `fetchmail-6.2.6-pre8.ru.po'.

The following HTML page should also be updated by tomorrow.

>    http://www.iro.umontreal.ca/translation/HTML/domain-fetchmail.html

Please consider including all PO files, as they stand, in the `po/'
subdirectory of your next release of programs using that textual domain,
whether it is official or pretest.  Whenever you have a distribution ready
which holds a newer PO Template, please send the URL of this distribution
to the address below.  The distribution could be a pretest or a snapshot,
it does not even have to compile.  This is to be used by translators,
when they need to get some translation context from your sources.

Within the Translation Project, each PO Template file should have different
version numbers, but since it is not OK to have two different distributions
using same version numbers, this is not a problem in practice.

Contact me if any question arises.  Thanks for your collaboration,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                mailto:tra...@ir...

[fetchmail-devel] fetchmail 6.2.6-pre9 released - please test

[Matthias A. <mat...@gm...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I have released a new fetchmail snapshot, 6.2.6-pre9, available from:

<http://developer.berlios.de/project/showfiles.php?group_id=1824&release_id=6730>

mirror: <http://home.pages.de/~mandree/fetchmail/>

Please test, particularly if you're running a multidrop setup.

Also, we still need translators, we have some old translations that need
to be updated and aren't enabled by default, but new translations will
also be welcome. Contact me if you're interested and fetchmail doesn't
currently ship with a translation into your favorite language.


Notes:

This release has fairly large internal cleanups - I decided to get rid
of the obsolete inet6_apps/inner_connect/ipv6-connect code rather than
carrying it into a new release - the "--netsec" and "-T" options are now
gone. The INET6_ENABLE code was also simplified, everything is now a
"service" internally, port is converted into service, although this
should be transparent.

There will be some more IPv6 awareness fixes, too many places still use
the hardcoded IPv4-only gethostbyname function, so we'll have one or two
more releases down the road before calling out for other release
candidates.

Summarized changes:

    * The multidrop code was severely broken in several places:

          o a fairly fundamental RFC822 parser bug was fixed that
            stripped the last character of bare addresses
            i. e. addresses without <> angle brackets (from
            Delivered-To: headers, for instance), responsible for many
            unrecognized domains in multidrop mode.

          o The multidrop IP address matching code was broken and
            checked only the first address of the configured servername
            against the aliases of the domain found in the envelope.

          o The multidrop MX matching code was broken and didn't resolve
            the MX hosts into IP addresses.

    * The netsec and -T options are gone, corresponding code was
      removed. See Notes above.
  
    * The gettext code (intl/ directory) was removed from the fetchmail
      package and, if NLS is desired, needs to be installed separately
      before fetchmail is built.

    * The default distribution format has been switched to bzip2.

    * The Python script is now optimized, byte-compiled and installed
      into the Python site-packages directory.

    * Some internal code shuffling and cleanups.


- -- 
Matthias Andree
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)

iD8DBQFC7t0hvmGDOQUufZURAkDnAKCoJ9cIoHddmqODswE9IpSEH4WWBQCaA7kF
ld16PJwbItP+ixO+aqjrNYw=
=Zdqo
-----END PGP SIGNATURE-----

Re: [fetchmail-devel] [PATCH] printf ( #ifdef ... #endif ... )

[Miloslav T. <mi...@re...>]

Hello,
Matthias Andree wrote:
> Miloslav Trmac <mi...@re...> writes:
>>printf () may be a macro (and it actually is a macro e.g. with recent
>>glibc and -D_FORTIFY_SOURCE=2) and preprocessor directives within
>>macros are not permitted.
> 
> What is this _FORTIFY_SOURCE about?  I have it in my include files, but
> ZERO documentation, and all I found out is that it remaps to ominuous
> builtins. So there must be recent gcc as well as it appears.
http://www.redhat.com/magazine/009jul05/features/execshield/#checks
contains a short introduction; basically buffer sizes, when known
to the compiler, are passed to libc and libc aborts the program if
it would overwrite the buffer.

> If this is supposed to be a security helper/feature/whatever and it's
> not documented, it's useless.
The necessary features have not been released in gcc yet (they
are scheduled to be in gcc 4.1); currently this requires using
the Fedora/RHEL toolchain, so evangelizing this for cross-distribution
usage would be a bit premature.
	Mirek