fetchmail-users — User questions and discussion

[fetchmail-users] Re: broken POP3 servers - future fetchmail policy (IMPORTANT) (was: [fetchmail]comcast.net and attachments - workaround)

[Jakob H. <jh...@pl...>]

Matthias Andree wrote:

> [Cc:ing BerliOS users

Right, we better continue on the new list.

>> Dropping it completely would be a pity. I have one upstream server
>> (Mercury on a Netware server...) that deletes messages after RETR
>> automatically. That may be a nice feature for the admin, but it's
>> annoying if you like to read mail from more than one place.
> It is not only "annoying", but up front a critical data loss defect of
> the server, needless to say it is a violation of the protocol, too.

Well, RFC 1939 says "Such message deletions are outside the scope of the
POP3 protocol and are not considered a protocol violation." (thanks for
pointing me to this chapter, btw)

Site policies always override RFC-specified behaviour, there's not much
you can do about that.

> If talking to the admins doesn't resolve their bad behavior, find a
> different server.

It's my university's server...

> Loss scenario: What happens if the client crashes between the server's
> shipping CRLF.CRLF and storing the message? Whoops, the server deleted
> the message, and the client retry.

Not necessarily. Consider a normal POP3 server, only that he does an
implicit DELE after each RETR. Of course, a client could still fail to
store the message (or to forward it somewhere, like fetchmail) without the
server noticing. But only if the client tries to behave nicely and sends
QUIT even after something went wrong. In this case, nice behaviour does
not pay off.

>> What changes do you plan for the seen tracking? To do it on the client
>> side is already possible with the UIDL option.
> UIDL is not an option, but a necessity.

For simply retrieving mail, delivering locally, retry in case of failure?
I don't see why.

[fetchmail-users] Re: [fetchmail]comcast.net and attachments - workaround

[Matthias A. <mat...@gm...>]

On Tue, 03 Jan 2006, Ed Wilts wrote:

> On Tue, Jan 03, 2006 at 04:43:28PM +0100, Matthias Andree wrote:
> > In the meanwhile, please complain to comcast.net for violating the POP3
> > standard, which states expressis verbis that the full message is to be
> > fetched if the number passed to TOP exceeds the actual number of lines
> > in the message.
> 
> That would be a waste of time.  After all, it works fine with all the
> normal Windows clients.

The bug doesn't depend on the operating system.

After all, comcast need not offer TOP at all if they don't plan to do it
right. TOP is optional.

Now, if Novell or Red Hat could phone comcast (rather than individual
users) and demand the fix, that might make a difference...

> In my opinion, we (the Linux community) need to spend more time working
> around broken implementations and less time standing on our podiums
> claiming we're right.

I don't plan to spend much time working around broken implementations,
there are more important things to do in fetchmail.

Standards are there for a reason, and not only Microsoft will notice
that soon enough.

In this particular example, you're lucky as the test is trivial and
"more RETR, less TOP" coincides with my goals for fetchmail.

Please test the attached patch (save it to a file, type "patch
<workaround-Maillennium.patch"), then recompile ("make") and reinstall
("make install") fetchmail and see if it solves your problem. It will
however mark messages as downloaded on the server, so I'm not sure if
or how I'll merge it into 6.3.2 or if it needs to wait until 6.4.0.

It is untested (I have no comcast account) and should apply against 6.3.1.
As usual, no warranties, use at your own risk.

If someone else wishes to test, go ahead and let me know what you find.
The warning message only appears if fetchmail had used TOP otherwise.

> I know it's disgusting, but unfortunately we're
> propogating the attitudes that Windows works better than Linux, because
> in this case, it does.  My Windows clients work fine (which is how I
> narrowed it down to fetchmail), my Linux one doesn't (until I
> implemented my workaround), yet the server is unlikely to change.

They, too, must learn that standards conformance is a way to keep
customers happy and money flowing in. If they truncate your messages
because their server is buggy, why give them your best^W money?

> I do see that fetchmail will be fixed to work better in cases like this
> so I'm not putting you in the same camp as some other Linux developers
> I've run into.

My motivation to add _complex_ workaround code as little as that of many
fellow developers. You may get the workaround if it's simple and the
problem hurts really bad.

Software should work efficiently on conformant servers, and our
installed base is large enough we can take some risks of not pleasing
everyone.  Fetchmail is default install in many systems.

I won't be adding workarounds that are disadvantages to users of intact
servers.

Kind regards,

-- 
Matthias Andree

Re: [fetchmail-users] broken POP3 servers - future fetchmail policy (IMPORTANT)

[Rob M. <rob...@gm...>]

On 03/01/06, Matthias Andree <mat...@gm...> wrote:
>
> That's what I was trying to say: download completely, hand on
> completely, obtain status of destination SMTP/LMTP/BSMTP/MDA, and only
> assume "completed" if every step succeeded. If any intermediate step
> fails, we'll retry the message next time. Although it's bad luck we've
> missed the goal of one attempt, and we'll have 1.7 downloads or so.

That's what I thought.  Thanks for confirming that :)

--
                 Please keep list traffic on the list.
Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

Re: [fetchmail-users] broken POP3 servers - future fetchmail policy (IMPORTANT)

[Matthias A. <mat...@gm...>]

Rob MacGregor <rob...@gm...> writes:

> On 03/01/06, Matthias Andree <mat...@gm...> wrote:
>>
>> fetchmail's task is to fetch all messages exactly once, nothing more,
>> nothing less. If we must fetch a message twice because the transfer was
>> interrupted, well, bad luck.
>
> You sure you're not ESR in disguise :-)

100% sure I'm not ESR. 

> I'm happy with that statement, if by fetch you mean "download and hand
> it on to the specified MTA".

That's what I was trying to say: download completely, hand on
completely, obtain status of destination SMTP/LMTP/BSMTP/MDA, and only
assume "completed" if every step succeeded. If any intermediate step
fails, we'll retry the message next time. Although it's bad luck we've
missed the goal of one attempt, and we'll have 1.7 downloads or so.

> Otherwise the statement sounds like you don't care of there's a
> problem with the local MTA, fetchmail will have done it's bit by
> downloading the email, chucking it at /dev/null and deleting it from
> the remote server...

Fetchmail will continue to make as many attempts as necessary to
retrieve the message for reliable protocols such as POP3+UIDL.

This requires co-operation of the server though, like not doing stupid
things such as deleteing the message without knowing if the client was
able to store the message. Looking at code and FAQ, many TOP
implementations leave things to be desired.

Some minor polishing has yet to happen though, for instance, after a
crash, fetchmail may retrieve messages it had successfully retrieved and
delivered before the crash, causing duplicates, but better get a message
again that you have already read than miss out on one you hadn't
read. We could reduce the number of duplicates by checkpointing the
.fetchids more often.

-- 
Matthias Andree

Re: [fetchmail-users] broken POP3 servers - future fetchmail policy (IMPORTANT) (was: [fetchmail]comcast.net and attachments - workaround)

[Rob M. <rob...@gm...>]

On 03/01/06, Matthias Andree <mat...@gm...> wrote:
>
> fetchmail's task is to fetch all messages exactly once, nothing more,
> nothing less. If we must fetch a message twice because the transfer was
> interrupted, well, bad luck.

You sure you're not ESR in disguise :-)

I'm happy with that statement, if by fetch you mean "download and hand
it on to the specified MTA".  Otherwise the statement sounds like you
don't care of there's a problem with the local MTA, fetchmail will
have done it's bit by downloading the email, chucking it at /dev/null
and deleting it from the remote server...

--
                 Please keep list traffic on the list.
Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

[fetchmail-users] broken POP3 servers - future fetchmail policy (IMPORTANT) (was: [fetchmail]comcast.net and attachments - workaround)

[Matthias A. <mat...@gm...>]

[Cc:ing BerliOS users 

Jakob Hirsch schrieb am 2006-01-03:

> Matthias Andree wrote:
> 
> > fetchmail will gradually move to client-side seen tracking since that's
> > the only reliable way to do it, and we'll probably get rid of TOP
> > because it's just too painful with the many b0rked upstreams.
> 
> Dropping it completely would be a pity. I have one upstream server
> (Mercury on a Netware server...) that deletes messages after RETR
> automatically. That may be a nice feature for the admin, but it's annoying
> if you like to read mail from more than one place.

It is not only "annoying", but up front a critical data loss defect of
the server, needless to say it is a violation of the protocol, too.

And it has legal implications at least in Germany for most practical
purposes, unless you signed a clause to the extent that "$ISP is allowed
to delete messages even after unsuccessful retrieval attempts, in
violation of pertinent standards" or it's a company machine with no
permission for you to use the mailbox for private purposes, too.

If talking to the admins doesn't resolve their bad behavior, find a
different server.

Loss scenario: What happens if the client crashes between the server's
shipping CRLF.CRLF and storing the message? Whoops, the server deleted
the message, and the client retry.

There's a reason why messages MUST NOT be deleted unless the client
sends DELE 1234 for message 1234 and then commits the changes with QUIT
("UPDATE" state), and that reason is called reliability.

Working around such massively broken server behavior will just carry on
the users' suffering perpetually because server admins will think that
clients have alternatives.

> What changes do you plan for the seen tracking? To do it on the client
> side is already possible with the UIDL option.

UIDL is not an option, but a necessity.

The only reason why I didn't make it default in 6.3.X is that I wanted
6.3.X to be as compatible with 6.2.X as needed, so that most people
could just drop 6.3.X in to have their 6.2.X bugs fixed, without further
thinking. We'll need to enable UID/UIDVALIDITY for IMAP, too.

fetchmail's task is to fetch all messages exactly once, nothing more,
nothing less. If we must fetch a message twice because the transfer was
interrupted, well, bad luck.

-- 
Matthias Andree

Re: [fetchmail-users] Fetchmail is running slow on Mac OS X

[Stéphane L. <ste...@lu...>]

Am 03.01.2006 um 00:50 schrieb Rob MacGregor:

> [ Please don't CC me - I get the mail on the list ]
>
> On 02/01/06, Stéphane Lux <ste...@lu...> wrote:
>>
>> I have tried fetchmail 6.3.1. I still have the one minute delays for
>> setting up a connection:
>
> Next thing to do is run fetchmail under truss to get a dump of what's
> going on.  Watching that it should be fairly obvious where the problem
> lies.  The output of that, with the area the pause occurs marked,
> would help in tracking down the problem.


Here is the output of kdump:

...
2710 fetchmail RET   sigaction 0
12710 fetchmail CALL  sigaction(0xd,0xbfffd3d8,0xbfffd450)
12710 fetchmail RET   sigaction 0
12710 fetchmail CALL  sigprocmask(0x1,0,0x32c28)
12710 fetchmail RET   sigprocmask 0
12710 fetchmail CALL  setitimer(0,0xbfffd460,0)
12710 fetchmail RET   setitimer 0

--- here is the one minute delay ---

12710 fetchmail CALL  socket(0x2,0x1,0)
12710 fetchmail RET   socket 4
12710 fetchmail CALL  connect(0x4,0x3018b0,0x10)
12710 fetchmail RET   connect 0
12710 fetchmail CALL  setitimer(0,0xbfffd460,0)
12710 fetchmail RET   setitimer 0
12710 fetchmail CALL  setitimer(0,0xbfffd460,0)
12710 fetchmail RET   setitimer 0
12710 fetchmail CALL  setitimer(0,0xbfffb3a0,0)
12710 fetchmail RET   setitimer 0
12710 fetchmail CALL  recvfrom(0x4,0xbfffb450,0x2000,0x2,0,0)
12710 fetchmail GIO   fd 4 wrote 147 bytes
...

Does this help in tracking down the problem?

[fetchmail-users] Re: [fetchmail]Re: SSL authentication problems with Gmail

[Matthias A. <mat...@gm...>]

[Cc:ing fet...@li..., JFTR]

Sebastian Tennant <se...@sm...> writes:

> I unpacked the 6.3.1 tarball in my workspace directory, downloaded
> the openssl source from the Debian apt repositories (testing) and
> copied it across to my workspace too.  Then I ran:
>
>   $ cd fetchmail-6.3.1
>   $ ./configure --with-ssl=/home/sebyte/workspace/openssl-0.9.8a
>   [...]
>   configure: error: cannot link with SSL - check config.log
>
> I read the INSTALL file for building with ssl and the directory I
> specified on the comand line definitely includes the include/
> directory...

Actually, the system SSL should have been fine after installation of the
-dev package. I know not a single system with OpenSSL in the base system
where --with-ssl=/usr hasn't worked for me.

> I hope I've attached the relevant section from the config.log.

Yup.

> configure:12595: gcc -o conftest -g -O2   conftest.c  -L/home/sebyte/workspace/openssl-0.9.8a/lib -lcrypt  -lresolv -lssl -lcrypto  >&5
> /usr/bin/ld: cannot find -lssl

Try running "ldconfig -n /home/sebyte/workspace/openssl-0.9.8a/lib" and
re-run ./configure.

-- 
Matthias Andree

Re: [fetchmail-users] Fetchmail is running slow on Mac OS X

[Rob M. <rob...@gm...>]

[ Please don't CC me - I get the mail on the list ]

On 02/01/06, Stéphane Lux <ste...@lu...> wrote:
>
> I have tried fetchmail 6.3.1. I still have the one minute delays for
> setting up a connection:

Next thing to do is run fetchmail under truss to get a dump of what's
going on.  Watching that it should be fairly obvious where the problem
lies.  The output of that, with the area the pause occurs marked,
would help in tracking down the problem.

--
                 Please keep list traffic on the list.
Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

Re: [fetchmail-users] Fetchmail is running slow on Mac OS X

[Stéphane L. <ste...@lu...>]

Am 02.01.2006 um 00:32 schrieb Rob MacGregor:

> On 01/01/06, Stéphane Lux <ste...@lu...> wrote:
>>
>> When I start fetchmail, I get this message:
>>
>> fetchmail: starting fetchmail 6.2.5 daemon
>
> As per my message on the other list - please try 6.3.1.

;)
I still have the same delays wit 6.3.1.

Re: [fetchmail-users] Fetchmail is running slow on Mac OS X

[Stéphane L. <ste...@lu...>]

Am 02.01.2006 um 16:38 schrieb Rob Funk:

> Stéphane Lux wrote:
>> fetchmail: starting fetchmail 6.2.5 daemon
>> fetchmail: 6.2.5 querying ... (protocol IMAP) at Sat, 31 Dec 2005
>> 03:03:13 +0100 (CET): poll started
>>
>> Then nothing happens for about one minute  (timout?), then fetchmail
>> ist running normaly:
>
> Normally delays of a minute or two indicate a DNS problem somewhere.

What kind of DNS problem? Server or client? I can telnet to the port  
143 without a delay.


>> On my linux server it takes only a few seconds for fetchmail to check
>> the mail accounts? What can I do? Is it a DNS or IDENT problem? I'm
>> using a NAT router.
>
> Ah, ident could be a possibility too, depending on the server  
> software, but
> DNS is more likely.
>
> As the other Rob recommended, first try fetchmail 6.3.1 from
> http://fetchmail.berlios.de/.  If that doesn't help let us know.

I have tried fetchmail 6.3.1. I still have the one minute delays for  
setting up a connection:

mac-mini:~/Desktop root# ./fetchmail -Nv -f /var/root/.fetchmailrc
fetchmail: WARNING: Running as root is discouraged.
fetchmail: starting fetchmail 6.3.1 daemon
fetchmail: 6.3.1 querying *** (protocol IMAP) at Mon, 02 Jan 2006  
22:56:38 +0100 (CET): poll started
fetchmail: IMAP< * OK [CAPABILITY IMAP4REV1 LOGIN-REFERRALS STARTTLS  
AUTH=LOGIN] *** IMAP4rev1 2003.339 at Mon, 2 Jan 2006 22:57:37 +0100  
(CET)
...

Re: [fetchmail-users] Fetchmail is running slow on Mac OS X

[Rob F. <rf...@fu...>]

Stéphane Lux wrote:
> fetchmail: starting fetchmail 6.2.5 daemon
> fetchmail: 6.2.5 querying ... (protocol IMAP) at Sat, 31 Dec 2005
> 03:03:13 +0100 (CET): poll started
>
> Then nothing happens for about one minute  (timout?), then fetchmail
> ist running normaly:

Normally delays of a minute or two indicate a DNS problem somewhere.  

> On my linux server it takes only a few seconds for fetchmail to check
> the mail accounts? What can I do? Is it a DNS or IDENT problem? I'm
> using a NAT router.

Ah, ident could be a possibility too, depending on the server software, but 
DNS is more likely.

As the other Rob recommended, first try fetchmail 6.3.1 from 
http://fetchmail.berlios.de/.  If that doesn't help let us know.

-- 
==============================|   "A microscope locked in on one point
 Rob Funk <rf...@fu...> |Never sees what kind of room that it's in"
 http://www.funknet.net/rfunk |    -- Chris Mars, "Stuck in Rewind"

Re: [fetchmail-users] Fetchmail is running slow on Mac OS X

[Rob M. <rob...@gm...>]

On 01/01/06, Stéphane Lux <ste...@lu...> wrote:
>
> When I start fetchmail, I get this message:
>
> fetchmail: starting fetchmail 6.2.5 daemon

As per my message on the other list - please try 6.3.1.

--
                 Please keep list traffic on the list.
Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

[fetchmail-users] Fetchmail is running slow on Mac OS X

[Stéphane L. <ste...@lu...>]

Hi,

I hope, that this is the right place for my question regarding  
fetchmail:

I have switched my mailserver from linux to Mac OS X 10.4. I  
apreciated, that fetchmail was already included in  OS 10.4. But on  
OS X it takes 1-2 minutes for fetchmail to set up a connection:

When I start fetchmail, I get this message:

fetchmail: starting fetchmail 6.2.5 daemon
fetchmail: 6.2.5 querying ... (protocol IMAP) at Sat, 31 Dec 2005  
03:03:13 +0100 (CET): poll started

Then nothing happens for about one minute  (timout?), then fetchmail  
ist running normaly:

fetchmail: IMAP< * OK [CAPABILITY IMAP4REV1 LOGIN-REFERRALS STARTTLS  
AUTH=LOGIN] ...  03:04:11 +0100 (CET)
fetchmail: IMAP> A0001 CAPABILITY
fetchmail: IMAP< * CAPABILITY IMAP4REV1 IDLE NAMESPACE MAILBOX- 
REFERRALS BINARY UNSELECT SCAN SORT THREAD=REFERENCES  
THREAD=ORDEREDSUBJECT MULTIAPPEND LOGIN-REFERRALS STARTTLS AUTH=LOGIN

When I try to connect manualy, I get an answer at once:

mac-mini:/var/root root# telnet ... 143
Trying ... ...
Connected to ....
Escape character is '^]'.
* OK [CAPABILITY IMAP4REV1 LOGIN-REFERRALS STARTTLS AUTH=LOGIN] ...  
IMAP4rev1 2003.339 at Sun, 1 Jan 2006 23:47:33 +0100 (CET)

On my linux server it takes only a few seconds for fetchmail to check  
the mail accounts? What can I do? Is it a DNS or IDENT problem? I'm  
using a NAT router.


Kind regards,

Stephane

[fetchmail-users] Re: message was not the expected length errors

[Sunil S. <sh...@bo...>]

Quoting from Frederik's mail on Fri, Dec 23, 2005:
> Since tonight, I'm suffering from "message [...] was not the expected
> length" errors. I see it on one mailbox on my ISP (the other mailbox is
> unaffected though), and it makes fetchmail completely fail to retrieve
> mails from the mailbox concerned.
> 
> I tried my ISP's webmail, and could access my mailbox without any
> problems. I deleted the whole contents of the mailbox, in the hope that it
> was one particual mail causing the problem. Unfortunately, on the first
> mail which arrived, the problem was there again.
> 
> I have this problem with both fetchmail from sarge (6.2.5-12sarge3), and
> with self compiled fetchmail 6.3.1.
> 
> Here's part of the output of an /etc/init.d/fetchmail debug-run:
> 
> fetchmail: 6.2.5 querying pop.pandora.be (protocol POP3) at Fri 23 Dec
> 2005 07:24:18 PM CET: poll started

...

> fetchmail: forwarding to localhost
> fetchmail: SMTP> MAIL FROM:<coo...@ma...> SIZE=4693
> fetchmail: SMTP< 250 OK
> fetchmail: SMTP> RCPT TO:<frederik@localhost>
> fetchmail: SMTP< 250 Accepted
> fetchmail: SMTP> DATA
> fetchmail: SMTP< 354 Enter message, ending with "." on a line by itself
> #**********************************.****************fetchmail: message
> use...@po...:1 was not the expected length (4803 actual != 4693
> expected) fetchmail: SMTP>. (EOM)
> fetchmail: terminated with signal 2

Please try adding the "forcecr" option to your fetchmailrc.

In case it doesn't work, please send the output of "fetchmail -V" and
the details of your smtp server.

Is the "message ... was not the expected length" error coming on every
mail or on specific mails only?

-- 
Sunil Shetye.

Re: [fetchmail-users] message was not the expected length errors

[Matthias A. <mat...@gm...>]

Rob MacGregor <rob...@gm...> writes:

>> #**********************************.****************fetchmail: message
>> use...@po...:1 was not the expected length (4803 actual != 4693
>> expected) fetchmail: SMTP>. (EOM)
>> fetchmail: terminated with signal 2
>
> And it turns out to be a bit longer.  It would be useful to see what
> the message from your SMTP server is - I suspect the problem may be on
> that end.

Would the SMTP server send a SIGINT to its client? I doubt that.
How would it find out the PID of its client?

-- 
Matthias Andree

Re: [fetchmail-users] message was not the expected length errors

[Matthias A. <mat...@gm...>]

Frederik <fr...@gm...> writes:

> fetchmail: SMTP< 354 Enter message, ending with "." on a line by itself
> #**********************************.****************fetchmail: message
> use...@po...:1 was not the expected length (4803 actual != 4693
> expected) fetchmail: SMTP>. (EOM)
> fetchmail: terminated with signal 2

These "not the expected length" messages are hints that the upstream
server is broken and is misreporting the actual message size.

Many POP3 server software packages are broken like this, for instance
some Exchange versions, qmail-pop3d and the one your server uses,
fetchmail just reports them and continues.  This "not the expected
length" message does not cause fetchmail to fail.

fetchmail aborts because someone or something is sending signal 2
(SIGINT, ^C), and it's not fetchmail, which has no code to raise
SIGINT.

Stop that something from sending fetchmail signals like SIGINT. Check
your scripts that run fetchmail, the shells that you run this script
from, and perhaps check if fetchmail is somehow part of a process group
with rampant programs. plugin, plugout are also possible candidates.

Details about your setup, such as fetchmail configuration and how you
start fetchmail, may help in tracking down the problem.

-- 
Matthias Andree

Re: [fetchmail-users] message was not the expected length errors

[Jason W. <jas...@in...>]

On Fri, Dec 23, 2005 at 08:45:18PM +0000, Rob MacGregor wrote:
 
> > #**********************************.****************fetchmail: message
> > use...@po...:1 was not the expected length (4803 actual != 4693
> > expected) fetchmail: SMTP>. (EOM)
> > fetchmail: terminated with signal 2
> 
> And it turns out to be a bit longer.  It would be useful to see what
> the message from your SMTP server is - I suspect the problem may be on
> that end.

If, as I also suspect, that is where the problem is, try setting the mda
option in your fetchmailrc file to, e.g., procmail.
This will bypass your local MTA. You might also be able to configure your MTA
somehow to ignore the declared message size.

Re: [fetchmail-users] message was not the expected length errors

[Rob M. <rob...@gm...>]

On 23/12/05, Frederik <fr...@gm...> wrote:
> Since tonight, I'm suffering from "message [...] was not the expected
> length" errors. I see it on one mailbox on my ISP (the other mailbox is
> unaffected though), and it makes fetchmail completely fail to retrieve
> mails from the mailbox concerned.

I've often seen this before and it's never caused a problem in the
past.  However my days of using that horribly buggy POP server are
long over :-)


> fetchmail: POP3< +OK 1 4693
> 1 message for username at pop.pandora.be (4693 octets).
> fetchmail: POP3< +OK 1 4693
> fetchmail: POP3< +OK 4693 octets
> reading message use...@po...:1 of 1 (4693 octets)

Right, so the POP server has stated that the message is 4693 bytes long, total.

> fetchmail: SMTP> MAIL FROM:<coo...@ma...> SIZE=4693

And fetchmail passes this on.

> #**********************************.****************fetchmail: message
> use...@po...:1 was not the expected length (4803 actual != 4693
> expected) fetchmail: SMTP>. (EOM)
> fetchmail: terminated with signal 2

And it turns out to be a bit longer.  It would be useful to see what
the message from your SMTP server is - I suspect the problem may be on
that end.

--
                 Please keep list traffic on the list.
Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

[fetchmail-users] message was not the expected length errors

[Frederik <fr...@gm...>]

Since tonight, I'm suffering from "message [...] was not the expected
length" errors. I see it on one mailbox on my ISP (the other mailbox is
unaffected though), and it makes fetchmail completely fail to retrieve
mails from the mailbox concerned.

I tried my ISP's webmail, and could access my mailbox without any
problems. I deleted the whole contents of the mailbox, in the hope that it
was one particual mail causing the problem. Unfortunately, on the first
mail which arrived, the problem was there again.

I have this problem with both fetchmail from sarge (6.2.5-12sarge3), and
with self compiled fetchmail 6.3.1.

Here's part of the output of an /etc/init.d/fetchmail debug-run:

fetchmail: 6.2.5 querying pop.pandora.be (protocol POP3) at Fri 23 Dec
2005 07:24:18 PM CET: poll started
fetchmail: POP3< +OK pop s`maHegdelS
fetchmail: POP3> CAPA
fetchmail: POP3< -ERR not implemented
fetchmail: not implemented
fetchmail: Repoll immediately on use...@po...
fetchmail: POP3< +OK pop s`maHegdelS
fetchmail: POP3> USER username
fetchmail: POP3< +OK 1332
fetchmail: POP3> PASS *
fetchmail: POP3< +OK .. welcome
fetchmail: selecting or re-polling default folder
fetchmail: POP3> STAT
fetchmail: POP3< +OK 1 4693
fetchmail: POP3> LAST
fetchmail: POP3< +OK 0
1 message for username at pop.pandora.be (4693 octets).
fetchmail: POP3> LIST 1
fetchmail: POP3< +OK 1 4693
fetchmail: POP3> TOP 1 99999999
fetchmail: POP3< +OK 4693 octets
reading message use...@po...:1 of 1 (4693 octets)
About to rewrite Return-Path: <coo...@ma...>
Rewritten version is Return-Path: <coo...@ma...>

About to rewrite To: co...@ma...
Rewritten version is To: co...@ma...

About to rewrite From: "dvg...@xs..." <bug...@qa...>
Rewritten version is From: "dvg...@xs..." <bug...@qa...>

About to rewrite Sender: ap...@qa...
Rewritten version is Sender: ap...@qa...

About to rewrite Reply-To: co...@ma...
Rewritten version is Reply-To: co...@ma...

fetchmail: SMTP< 220 localhost.localdomain ESMTP Exim 4.50 Fri, 23 Dec
2005 19:24:22 +0100
fetchmail: SMTP> EHLO localhost
fetchmail: SMTP< 250-localhost.localdomain Hello localhost [127.0.0.1]
fetchmail: SMTP< 250-SIZE 52428800
fetchmail: SMTP< 250-PIPELINING
fetchmail: SMTP< 250 HELP
fetchmail: forwarding to localhost
fetchmail: SMTP> MAIL FROM:<coo...@ma...> SIZE=4693
fetchmail: SMTP< 250 OK
fetchmail: SMTP> RCPT TO:<frederik@localhost>
fetchmail: SMTP< 250 Accepted
fetchmail: SMTP> DATA
fetchmail: SMTP< 354 Enter message, ending with "." on a line by itself
#**********************************.****************fetchmail: message
use...@po...:1 was not the expected length (4803 actual != 4693
expected) fetchmail: SMTP>. (EOM)
fetchmail: terminated with signal 2


The mail in question is this mail:
http://article.gmane.org/gmane.linux.mandrake.cooker.devel/177796


Any help to further debug this problem, would be appreciated, as now I
cannot access my e-mail anymore.

--
Frederik

[fetchmail-users] fetchmail security announcement 2005-03 (CVE-2005-4348)

[Matthias A. <mat...@gm...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

fetchmail-SA-2005-03: security announcement

Topics:		#1 crash retrieving headerless message in multidrop mode
		#2 fetchmail 6.2.5.X end of life

Author:		Matthias Andree
Version:	1.00
Announced:	2005-12-19
Type:		null pointer dereference
Impact:		fetchmail crashes
Danger:		low
Credits:	Daniel Drake, Gentoo (bug report)
		Sunil Shetye (bug fix)
CVE Name:	CVE-2005-4348
URL:		http://fetchmail.berlios.de/fetchmail-SA-2005-03.txt
		http://article.gmane.org/gmane.mail.fetchmail.user/7573
		http://bugs.debian.org/343836
Project URL:	http://fetchmail.berlios.de/

Affects:	fetchmail version 6.2.5.4
		fetchmail version 6.3.0

Not affected:	fetchmail 6.3.1
		fetchmail 6.2.5.5
		other versions not mentioned here or in the previous
		sections have not been checked

Corrected:	2005-12-19 - released fetchmail 6.3.1
		2005-12-18 - released fetchmail 6.3.1-rc1
		2005-12-19 - released fetchmail 6.2.5.5


0. Release history
==================

2005-12-19	1.00 - initial version


1. Background
=============

fetchmail is a software package to retrieve mail from remote POP2, POP3,
IMAP, ETRN or ODMR servers and forward it to local SMTP, LMTP servers or
message delivery agents.

fetchmail ships with a graphical, Python/Tkinter based configuration
utility named "fetchmailconf" to help the user create configuration (run
control) files for fetchmail.


2. Problem description and Impact
=================================

Fetchmail contains a bug that causes an application crash when fetchmail
is configured for multidrop mode and the upstream mail server sends a
message without headers.  As fetchmail does not record this message as
"previously fetched", it will crash with the same message if it is
re-executed, so it cannot make progress. A malicious or broken-into
upstream server could thus cause a denial of service in fetchmail
clients.

Note that such messages are not RFC-822 conformant, so if the server has
not been tampered with, the server software is faulty.


3. Workaround
=============

Where possible, singledrop mode may be an alternative.

For sites, where multidrop mode is required, no workaround is known.


4. Solution
===========

Download and install fetchmail 6.3.1 or a newer stable release from
fetchmail's project site at
<http://developer.berlios.de/project/showfiles.php?group_id=1824>.

The fix has also been backported to the 6.2.5.5 legacy release which is
available from the same site.

Note however that 6.3.X has very few incompatible changes since 6.2.5.X
so 6.3.X should be viable for most sites.  It is therefore recommended
that every user and distributor upgrade to 6.3.1 or newer.


5. End of life announcement
===========================

The fetchmail 6.2.5.X branch will be discontinued early in 2006.

The new 6.3.X stable branch has been available since 2005-11-30
and will not change except for bugfixes, documentation and translations.


A. Copyright, License and Warranty
==================================

(C) Copyright 2005 by Matthias Andree, <mat...@gm...>.
Some rights reserved.

This work is licensed under the Creative Commons
Attribution-NonCommercial-NoDerivs German License. To view a copy of
this license, visit http://creativecommons.org/licenses/by-nc-nd/2.0/de/
or send a letter to Creative Commons; 559 Nathan Abbott Way;
Stanford, California 94305; USA.

THIS WORK IS PROVIDED FREE OF CHARGE AND WITHOUT ANY WARRANTIES.
Use the information herein at your own risk.

END OF fetchmail-SA-2005-03.txt
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)

iD8DBQFDqejWvmGDOQUufZURArTuAJ9/6xIB+DU4e0CAKNCGC3xE8pRzwwCeKLLT
CtudIJhoxM7h/HBsPgpnetg=
=pI3F
-----END PGP SIGNATURE-----

[fetchmail-users] fetchmail 6.3.1 stable release with security relevant bugfixes

[Matthias A. <mat...@gm...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Greetings,

I am announcing the release of fetchmail 6.3.1.

This release fixes a denial of service bug/fetchmail crash in multidrop
mode and several other annoyances, see below for a list of bugs fixed.

This is a recommended upgrade for all users of any previous fetchmail
versions. Distributors are sought to check opportunity to offer 6.3.1 as
upgrade for 6.2.X or previous releases.

The software is available from:
<https://developer.berlios.de/project/showfiles.php?group_id=1824&release_id=8405>

The SMTP/LMTP bug recently discussed on the fetchmail-devel mailing list
remains unfixed, I preferred the quick security fix over delaying the
release to have all fixes in. We still have the chance to a 6.3.2
release later :-)

These are the relevant changes in 6.3.1 since 6.3.0:

# DEPRECATED FEATURES AND MAJOR INCOMPATIBLE CHANGE ADVANCE WARNINGS
* The MX and host alias DNS lookups that fetchmail performs in multidrop mode
  are obsolete, deprecated and may be removed from a future fetchmail version.
  They have never supported IPv6 (including IPv6-mapped IPv4) anyhow. (MA)
* The monitor and interface options may be removed from a future fetchmail
  version as they are not sufficiently portable. (MA)
* POP2 is obsolete.
  Support for POP2 may be removed from a future fetchmail version. (MA)
* RPOP is obsolete, support may be removed from a future fetchmail release. (MA)
* --sslcertck may become a default setting in a future fetchmail version. (MA)
* The multidrop To/Cc guessing code along with the fragile duplicate suppressor
  is deprecated and may be removed from a future release. (MA)

# SECURITY FIX IN THIS RELEASE
* CVE-2005-4348 Fix segmentation fault (null pointer dereference) in
  multidrop mode with headerless email.  See fetchmail-SA-2005-03.txt.
  Reported by Daniel Drake, patch by Sunil Shetye. (MA)

# OTHER BUG FIXES, DOCUMENTATION AND TRANSLATION UPDATES
* Fix broken default port in POP2. Patch by Stanislav Brabec, SUSE [CZ]. (MA)
* Fix manual page, some lines starting with ' were escaped by \&.
  Reported by Simon Barner. (MA)
* Ship with gettext-0.14.3 again, as 6.2.9-rc10 did. Found by Sunil Shetye. (MA)
* Actually set default SSL certificate path if --sslcertpath is unset.
  Reported by Heino Tiedemann and Rob MacGregor. (MA)
* Remove bogus Netscape IMAP4rev1 Service >= 3.6 warning about BODY[TEXT]
  that we are not using. Patch by Sunil Shetye. (MA)
* Plug potential memory and socket leak when polling multiple folders or when
  the upstream sends bogus message sizes.  Patch by Sunil Shetye. (MA)
* Update Catalan translation, by Ernest Adrogué Calveras. (MA)
* Fix segfault (null pointer dereference) on some operating systems with
  fetchmail's obsolete DNS MX/host alias lookups in multidrop mode.
  Patch by Dr.-Ing. Andreas Haakh. (MA)
* Close SMTP sockets early, to reduce resource usage, trigger earlier delivery
  with some MTAs and avoid SIGPIPE (SIG 13) when the SMTP listener gets bored
  and drops the connection after timeout.  Patch by Sunil Shetye. (MA)
* Don't treat hitting a fetch limit as error.  Patch by Sunil Shetye. (MA)
* Fix negative "messages left on server" on idle/repoll with fetchlimit.
  Patch by Sunil Shetye. (MA)
* Properly track logout stage.  Patch by Sunil Shetye. (MA)
* Preserve error conditions across postconnect script.  Sunil Shetye. (MA)
* Do not trash destination domain if multiple messages are forwarded into the
  same SMTP/LMTP connection. Reported by Joachim Feise, Berlios Bug #5849. (MA)
* Manual page: Add "-md5" to "openssl x509" example in --sslfingerprint
  documentation, since OpenSSL 0.9.8 changed the default to SHA1.
  Suggested by Jason White. (MA)
* Cope with servers that return UID information in response to non-UID
  RFC822.{SIZE|HEADER} requests. Reported by Jason White.
  Patch suggestion by by Sunil Shetye, simplified by MA.

Regards,

- -- 
Matthias Andree
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)

iD8DBQFDppmVvmGDOQUufZURAsmjAJ4hRWPQf/xFiW6Uf0hscZqjLL1JywCfZqcx
HWL8U9SWHyOOQY1tqM4xDys=
=iql6
-----END PGP SIGNATURE-----

[fetchmail-users] fetchmail 6.2.5.5 legacy release fixes Denial of Service bug

[Matthias A. <mat...@gm...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Greetings,

I am announcing the release of fetchmail 6.2.5.5.

This release fixes a denial of service bug/fetchmail crash in multidrop
mode, plugs a socket leak when SSL negotiation fails and adds the three
security announcements from 2005 that the project issued so far.

The software is available from:
<https://developer.berlios.de/project/showfiles.php?group_id=1824&release_id=8403>

fetchmail-6.2.5.X is a security fix branch that forked off
fetchmail-6.2.5. It does not change for anything but security and the
most severe bug fixes.

Note this 6.2.5.X branch is going to be discontinued in Early 2006, all
users are advised to upgrade to the new 6.3.1 fetchmail release instead.
There have been very few incompatible changes, most sites should be
unaffected. 6.3.1 however fixes dozens of bugs (literally) that 6.2.5.5
still has. fetchmail 6.3.1 is available from
<https://developer.berlios.de/project/showfiles.php?group_id=1824&release_id=8405>

These are the relevant changes in 6.2.5.5 since (and excluding) 6.2.5.4:

* SECURITY FIX CVE-2005-4348: fix null pointer dereference in
  multidrop mode when the message is empty. Reported by Daniel Drake
  <http://article.gmane.org/gmane.mail.fetchmail.user/7573> and others
  (Debian Bug #343836). Fix by Sunil Shetye.
* Fix Debian bug #301964, fetchmail leaks sockets when SSL negotiation
  fails. Fix suggested by Goswin Brederlow.
* Add fetchmail-SA-2005-{01,02,03}.txt

Regards,

- -- 
Matthias Andree
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)

iD8DBQFDppiZvmGDOQUufZURAiVVAKCrH0gGmn/GCjFa8jag7FeUoPSyOQCgsezV
BQuopaSln4QWcgLAYBm4OPM=
=IvLw
-----END PGP SIGNATURE-----

Re: [fetchmail-users] Correction to example in fetchmail(1) manual page

[Matthias A. <mat...@gm...>]

Jason White <jas...@in...> writes:

> On Mon, Dec 19, 2005 at 01:33:33AM +0100, Matthias Andree wrote:
>> 
>> Thank you.  Does it document relevant changes to the x509 command, or
>> what does the x509 manual page say WRT the default hash?
>
>        -md2|-md5|-sha1|-mdc2
>            the digest to use. This affects any signing or display option that
>            uses a message digest, such as the -fingerprint, -signkey and -CA
>            options. If not specified then SHA1 is used. If the key being used
>            to sign with is a DSA key then this option has no effect: SHA1 is
>            always used with DSA keys.

Makes sense. Thank you!

-- 
Matthias Andree

Re: [fetchmail-users] [PATCH] Re: IMAP problem (fetchmail 6.3.1-pre1)

[Jason W. <jas...@in...>]

On Sun, Dec 18, 2005 at 11:31:49AM +0100, Matthias Andree wrote:
 
> Well, I'll just use an equivalent modification of Sunil's patch for
> 6.3.1, because, after looking at the respective code, the entire IMAP
> response parser must be rewritten - for instance, to support the ALERT
> response code or * BYE. And this rewrite is stuff for 6.4.0.
I'll volunteer to test it when the new parser is implemented.
> 
> We'll just use the minimal change that fixes the problem in 6.3.1. I'll
> release 6.3.1-rc1 later, and I'd ask you to test again so that my
> changes haven't broken the fix.
>
The fix still works in 6.3.1-rc1.