fetchmail-users

[fetchmail-users] Incorrect header line and lost mails

[Frederic M. <fre...@wo...>]

Hello,

I found 15 lines like this in my log this morning and 3 more earlier 
last week:

fetchmail[1226]: incorrect header line found while scanning headers


I read the source of fetchmail to learn more about it and I found out 
that any header containing at least one line without a colon and not 
starting with a space is considered illegal as per rfc822 and send to 
oblivion without any other consideration (at least none I could find).

Alas, it turns out that one of the three e-mails I could recover was 
definitely not a spam... I agree it was not fully rfc822 compliant but 
it was a good e-mail according to *my* standards and it could have been 
delivered since the invalid line was a very long TO header from hotmail 
(not rfc822 compliant either) wrapped by my ISP  !

I agree that the sin was in the original e-mail, but is it really the 
purpose of fetchmail to enforce rfc822 so drastically ? With such a 
name, I would expect it to fetch the e-mails and leave any 
"counter-measure" to a program designed for such a task (or at least to 
have an option to drop or accept invalid e-mails).

Now, I'm reluctant to patch and recompile the program myself. Is it 
possible to configure fetchmail to save a copy of every mail in a local 
folder before it is processed ? Could that be achieved with the --plugin 
option and how ?

I'm using fetchmail 6.2.5 as provided by the stable branch of debian.

Frederic







WOW Company or its affiliates do not accept legal responsibility 
for the contents of this message. The views or opinions 
presented are solely those of the author and do not necessarily 
represent those of WOW Company or any of its affiliates.

No contract will be established with WOW Company other than 
in writing (letter) and with the explicit authorization of the director 
of WOW Company.

Re: [fetchmail-users] Incorrect header line and lost mails

[Rob M. <rob...@gm...>]

On 2/13/06, Frederic Marchal <fre...@wo...> wrote:
> Hello,
>
> I found 15 lines like this in my log this morning and 3 more earlier
> last week:
>
> fetchmail[1226]: incorrect header line found while scanning headers

You'll find threads about this in the list archive.  I don't remember
the outcome, but I've a feeling that 6.3.x handles these (illegal)
messages differently.

It would be worth giving the latest version a try and see if that helps.

--
                 Please keep list traffic on the list.
Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

Re: [fetchmail-users] Incorrect header line and lost mails

[Matthias A. <mat...@gm...>]

Rob MacGregor <rob...@gm...> writes:

> On 2/13/06, Frederic Marchal <fre...@wo...> wrote:
>> Hello,
>>
>> I found 15 lines like this in my log this morning and 3 more earlier
>> last week:
>>
>> fetchmail[1226]: incorrect header line found while scanning headers
>
> You'll find threads about this in the list archive.  I don't remember
> the outcome, but I've a feeling that 6.3.x handles these (illegal)
> messages differently.

I don't recall having a fix for this particular issue, but 6.3.2
generally has some dozens of bugfixes worth having anyways. I'll put
this on my 6.3.3 TODO, but don't think it will appear within the next
three weeks (unless someone else sends a patch). Don't hold your breath,
I'm busy ATM.

-- 
Matthias Andree

Re: [fetchmail-users] Incorrect header line and lost mails

[Frederic M. <fre...@wo...>]

Rob MacGregor wrote:
> On 2/13/06, Frederic Marchal <fre...@wo...> wrote:
>   
>> Hello,
>>
>> I found 15 lines like this in my log this morning and 3 more earlier
>> last week:
>>
>> fetchmail[1226]: incorrect header line found while scanning headers
>>     
>
> You'll find threads about this in the list archive.  I don't remember
> the outcome, but I've a feeling that 6.3.x handles these (illegal)
> messages differently.
>
> It would be worth giving the latest version a try and see if that helps.
>   
I read the source code of 6.3.2 and the handling of those illegal mails 
is still exactly the same. Those e-mails are discarded and the 
"incorrect header line" message is written to the log.

Frederic


WOW Company or its affiliates do not accept legal responsibility 
for the contents of this message. The views or opinions 
presented are solely those of the author and do not necessarily 
represent those of WOW Company or any of its affiliates.

No contract will be established with WOW Company other than 
in writing (letter) and with the explicit authorization of the director 
of WOW Company.

Re: [fetchmail-users] Incorrect header line and lost mails

[Frederic M. <fre...@wo...>]

Matthias Andree wrote:
> Rob MacGregor <rob...@gm...> writes:
>
>   
>> On 2/13/06, Frederic Marchal <fre...@wo...> wrote:
>>     
>>> Hello,
>>>
>>> I found 15 lines like this in my log this morning and 3 more earlier
>>> last week:
>>>
>>> fetchmail[1226]: incorrect header line found while scanning headers
>>>       
>> You'll find threads about this in the list archive.  I don't remember
>> the outcome, but I've a feeling that 6.3.x handles these (illegal)
>> messages differently.
>>     
>
> I don't recall having a fix for this particular issue, but 6.3.2
> generally has some dozens of bugfixes worth having anyways. I'll put
> this on my 6.3.3 TODO, but don't think it will appear within the next
> three weeks (unless someone else sends a patch). Don't hold your breath,
> I'm busy ATM.
>   

Well, I'm a C programmer and I may be able to help but I'm mainly a 
windows programmer (yes, I know, shame on me :-) ) and I'm not yet all 
too familiar with the Linux programming and especially with the patch 
mechanism and none at all with the diff tool. Nevertheless, I have 
identified the core of the problem (its the call to readheaders in 
drive.c and the response to PS_REFUSED). I still have to check what 
happens if an invalid FROM or TO header is passed after readheaders but 
I think I can fix the program. I'll also need some direction about the 
proper way of doing it for a good integration in the current source.

If you think it is reasonable, we can discuss this further on the 
fetchmail-devel list.

Frederic

Re: [fetchmail-users] Incorrect header line and lost mails

[Matthias A. <mat...@gm...>]

Frederic Marchal <fre...@wo...> writes:

> Well, I'm a C programmer and I may be able to help but I'm mainly a
> windows programmer (yes, I know, shame on me :-) ) and I'm not yet all
> too familiar with the Linux programming and especially with the patch
> mechanism and none at all with the diff tool.

The workflow is:

1. create a copy of all files you need to edit, for instance, 
   copy foo.c to foo.c.orig

   alternatively, copy the whole directory tree recursively

2. edit until you're happy with the code

3. test the edits (compile, install, run)

4. generate a unified (preferred) or context patch.  Not all diff
   utilities support unified output, but all are to support context
   format, as mandated by the relevant standard, IEEE Std 1003.1-2001.

   Plain ed-style patches (unfortunately still the default in the diff
   utility) are impractical so they are not usually accepted.

The diff tool is quite simple actually, if you created file copies:

diff -u OLDFILE NEWFILE >PATCHFILE (unified diff, preferred)

diff -c OLDFILE NEWFILE >PATCHFILE (context diff, alternative)

If you copied the whole directory before the edits:

diff -u -r BACKUP_DIR EDITED_DIR >PATCHFILE    (or -c instead of -u)

If you added new files to EDITED_DIR, you'd use -N but I don't think
we'll need this here.

diff, used either way, will produce a file (PATCHFILE) that you'd attach
to a message and mail to me (or Rob, or preferably the -devel@ list) or
upload to the BerliOS patch tracker.

If I have the choice, I prefer the -devel@ list. That's public, archived
and easiest for me since I don't need to fetch anything, but get it
pushed into my inbox.


The patch utility does the reverse and will usually just be used like
this, it derives the file names from the PATCHFILE.

patch <PATCHFILE

perhaps with -d (to change directory) or -p (to strip leading path
components from the paths shown in the patch file; used if patch cannot
find the files).

> Nevertheless, I have identified the core of the problem (its the call
> to readheaders in drive.c and the response to PS_REFUSED). I still
> have to check what happens if an invalid FROM or TO header is passed
> after readheaders but I think I can fix the program.

That would be great.

Basically they should not matter that much. fetchmail has some
deprecated code to guess from To:/Cc: or similar, but that is an
unreliable concept and doesn't deserve attention, From: shouldn't matter
as long as Return-Path: is given (and even then, From: shouldn't
matter). Missing or corrupted Return-Path: headers should be treated as
though the message had contained "Return-Path: <>".

> I'll also need some direction about the proper way of doing it for a
> good integration in the current source.

The bigger concern is that fetchmail shouldn't be emitting headers it
knows are broken, so that the next hop (the MTA or MDA) does not get
confused. I wonder if fetchmail should prefix the broken lines with
X-Fetchmail-Escaped-Broken-Header: or something similar.

-- 
Matthias Andree