fetchmail-users

[fetchmail-users] How to handle mail with broken headers?

[Johann 'M. O. <myr...@ai...>]

Hi all,

I have a few emails with broken headers in my yahoo mailbox and
fetchmail refuses to retrieve/forward them.  I have repeated examples of

  fetchmail: reading message
  myr...@po...:3695 of
  21103 (3139 octets) (log message incomplete)
  fetchmail: incorrect header line found while scanning headers
  fetchmail:  not flushed

In my log file.  And yes, you're right.  I do have 21k emails, and I'm
not deleting them from my mailbox.

Interestingly enough, that particular mail is now being skipped by
fetchmail, though it does have a broken header:

  X-YahooFilteredBulk: 206.40.205.124
  smedley.info>
  X-Originating-IP: [206.40.205.124]

Checking the UIDL in a manual POP3 session:

  UIDL 3639
  +OK 3639 AMZEv9EAAUj4R3RxaQBpvXThslA

And test to see if it's been retrieved.

  % grep AMZEv9EAAUj4R3RxaQBpvXThslA .fetchids
  myr...@po... AMZEv9EAAUj4R3RxaQBpvXThslA

Yet I can not find it anywhere when I grep for the message id:

  find . -exec grep 47797E8A.3040107 '{}' +

So, apparently there are two issues I'm running into.  One is that I
have broken headers and somehow would like Fetchmail to handle that.
Going manually through the log file and picking them out is not really
what I want to do.  Once, maybe, but there are several such mails (I
suspect a broken port of Thunderbird to the OS/2 operating system) and
it can happen again.

Two, for some reason fetchmail has marked a broken message as downloaded
even though it can't be found locally.  I'm using maildrop as an mda and
I somehow doupt that is the fault - maildrop has been very good at
complaining when I managed to misconfigure it.  Of course I did not put
that particular mail through it just to see what it does -- yet.

Going back to the former issue.  Is there a way to get fetchmail do
somehow fix/download broken mail?  Perhaps take broken header lines and
prefix them with X-Fetchmail-Broken-Header: ?

Here is the last such message from my log file.  Apparently I haven't
gotten a broken mail for some time now:

  fetchmail: reading message
  myr...@po...:13512 of
   21115 (306940 octets) (log message incomplete)
  fetchmail: incorrect header line found while scanning headers
  fetchmail:  not flushed

And here are both messages from manual POP3 sesssions:

  http://www.myrkraverk.com/~johann/3649.invalid.header
  http://www.myrkraverk.com/~johann/13512.invalid.header


Johann

Re: [fetchmail-users] How to handle mail with broken headers?

[Rob M. <rob...@gm...>]

On Mon, Jan 30, 2012 at 15:53, Johann 'Myrkraverk' Oskarsson
<myr...@ai...> wrote:
> Hi all,
>
> I have a few emails with broken headers in my yahoo mailbox and
> fetchmail refuses to retrieve/forward them.  I have repeated examples of

What version number of Fetchmail are you running and on what operating
system? You may want to read the FAQ
(http://fetchmail.berlios.de/fetchmail-FAQ.html#G3) and the list
archive too.

I know this has been discussed a lot on the list and either it is
still planned or support to handle this has been added. Unfortunately
I can't find a list of changes included with the source, and Berlios
only displays the changes for the most recent release.

-- 
                 Please keep list traffic on the list.

Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

Re: [fetchmail-users] How to handle mail with broken headers?

[Matthias A. <mat...@gm...>]

Am 30.01.2012 19:16, schrieb Rob MacGregor:
> On Mon, Jan 30, 2012 at 15:53, Johann 'Myrkraverk' Oskarsson
> <myr...@ai...> wrote:
>> Hi all,
>>
>> I have a few emails with broken headers in my yahoo mailbox and
>> fetchmail refuses to retrieve/forward them.  I have repeated examples of
> 
> What version number of Fetchmail are you running and on what operating
> system? You may want to read the FAQ
> (http://fetchmail.berlios.de/fetchmail-FAQ.html#G3) and the list
> archive too.

The version Johann is running is outdated, because a relevant switch
named "bad-header" has been added in release 6.3.15 in Early 2010, and
has been more prominent since 6.3.18.

> I know this has been discussed a lot on the list and either it is
> still planned or support to handle this has been added. Unfortunately
> I can't find a list of changes included with the source, and Berlios
> only displays the changes for the most recent release.

The list of changes is called "NEWS", along the lines of other GNU
software (although fetchmail isn't a GNU project, just uses some GNU
libraries and the GNU license).

Best regards
Matthias

Re: [fetchmail-users] How to handle mail with broken headers?

[Rob M. <rob...@gm...>]

On Mon, Jan 30, 2012 at 19:22, Matthias Andree <mat...@gm...> wrote:
>
> The list of changes is called "NEWS", along the lines of other GNU
> software (although fetchmail isn't a GNU project, just uses some GNU
> libraries and the GNU license).

Ah, for some reason I was looking for something named Changes (or at
least starting with Changes).

Ta

-- 
                 Please keep list traffic on the list.

Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

Re: [fetchmail-users] How to handle mail with broken headers?

[Johann 'M. O. <myr...@ai...>]

Matthias Andree <mat...@gm...> writes:

> The version Johann is running is outdated, because a relevant switch
> named "bad-header" has been added in release 6.3.15 in Early 2010, and
> has been more prominent since 6.3.18.

Thank you.  I'm using 6.3.21 now and life is better.  It could be made
more explicit in the manual how to use the option in the run control
file.  I had to read the description on the command line flag to learn I
have to use "accept" in there too.

For the record, this is how my yahoo entry reads at the moment:

  poll pop.mail.yahoo.com proto pop3 service 995 uidl bad-header accept
    user "myrkraverk" 
    ssl keep 
    mda "maildrop -f %F /home/johann/.maildroprc.yahoo"

and the password is stored in the .netrc file.


Johann

Re: [fetchmail-users] How to handle mail with broken headers?

[Matthias A. <mat...@gm...>]

Am 30.01.2012 21:15, schrieb Johann 'Myrkraverk' Oskarsson:
> Matthias Andree <mat...@gm...> writes:
> 
>> The version Johann is running is outdated, because a relevant switch
>> named "bad-header" has been added in release 6.3.15 in Early 2010, and
>> has been more prominent since 6.3.18.
> 
> Thank you.  I'm using 6.3.21 now and life is better.  It could be made
> more explicit in the manual how to use the option in the run control
> file.  I had to read the description on the command line flag to learn I
> have to use "accept" in there too.

Johann,

improving documentation is a constant struggle, so please help me a bit:
 Where did you look first, so that I can amend to that section, or
redirect users to the command line options?

Basically the options are the same, with two major differences:

- rcfile options don't use the "--" hyphens

- rcfile options need to be grouped into per-server and per-user
options, whereas command line options can be given in arbitrary order
(and then override all options from all servers in the rcfile).

> For the record, this is how my yahoo entry reads at the moment:
> 
>   poll pop.mail.yahoo.com proto pop3 service 995 uidl bad-header accept
>     user "myrkraverk" 
>     ssl keep 
>     mda "maildrop -f %F /home/johann/.maildroprc.yahoo"
> 
> and the password is stored in the .netrc file.

I routinely suggest to add "sslcertck" (without quotes), too, it would
go somewhere after the user option somewhere, for instance, "ssl
sslcertck keep".  A future fetchmail release will make that default, but
considerations about compatibility with older versions made me not do
that in 6.X.Y versions.

To use that, one may need to install root certificates (if such a
package is available and how it's named depends on the respective
distribution).

HTH

Re: [fetchmail-users] How to handle mail with broken headers?

[Johann 'M. O. <myr...@ai...>]

Matthias Andree <mat...@gm...> writes:

> improving documentation is a constant struggle, so please help me a
> bit:

>  Where did you look first, so that I can amend to that section, or
> redirect users to the command line options?

In this particular case, once I had learned of the option, I looked to
the tables for the run control file.

     bad-header                    How to treat messages with  a  bad
                                   header. Can be reject (default) or
                                   accept.

Notice that just above that we have

     uidl             -U           Force  POP3  to  use   client-side
                                   UIDLs (recommended)
     no uidl                       Turn off POP3 use  of  client-side
                                   UIDLs (default)

So it stands to reason one is supposed to use "bad-header" to accept bad
headers and just leave it out to reject it.

I recommend having the format given for the command line option
reflected in the table too.

     --bad-header {reject|accept}

Somewhat akin to

     bad-header {reject|accept}    How to treat messages with  a  bad
                                   header. Can be reject (default) or
                                   accept.

or have two entries like other options.

     bad-header reject             Reject messages with bad headers (the
                                   default).
     bad-header accept             Accept messages with bad headers.

I can not propose any one definite answer, only to have it clear that
the keywords "accept" and "reject" are also required in the run control
file without having to scroll all the way back to the command line
options to see it; in a manner that is consistent with the rest of the
man page.

> I routinely suggest to add "sslcertck" (without quotes), too, it would
> go somewhere after the user option somewhere, for instance, "ssl
> sslcertck keep".  A future fetchmail release will make that default,
> but considerations about compatibility with older versions made me not
> do that in 6.X.Y versions.

Done.  I've simplified my run control file and added it to my defaults:

defaults proto pop3 service 995 uidl fastuidl 6
	ssl sslcertck keep

> To use that, one may need to install root certificates (if such a
> package is available and how it's named depends on the respective
> distribution).

Seems fine at first test.  I know where to find my root certificates if
I have to tell fetchmail about them.


Johann