fetchmail-devel — Development related discussion

[fetchmail-devel] New template for 'fetchmail' made available

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.  (If you have
any questions, send them to <coo...@tr...>.)

A new POT file for textual domain 'fetchmail' has been made available
to the language teams for translation.  It is archived as:

    http://translationproject.org/POT-files/fetchmail-6.3.10-pre1.pot

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

Below is the URL which has been provided to the translators of your
package.  Please inform the translation coordinator, at the address
at the bottom, if this information is not current:

    http://www.dt.e-technik.uni-dortmund.de/~ma/fetchmail/fetchmail-6.3.10-beta1.tar.bz2

Translated PO files will later be automatically e-mailed to you.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

Re: [fetchmail-devel] [fetchmail-users] fetchmail 6.3.10-beta1 test version

[Rob M. <rob...@gm...>]

On Mon, May 25, 2009 at 23:30, Matthias Andree <mat...@gm...> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Greetings,
>
> I have uploaded a fetchmail 6.3.10 beta version - "beta" means that some
> of the bug fixes have been nontrivial, and testing is required. See the
> log below for bugs supposed to be fixed.

Compiles cleanly, with SSL support, on FreeBSD 7.0 (I'll get around to
upgrading it to something current in a few weeks, hopefully).

-- 
                 Please keep list traffic on the list.

Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

[fetchmail-devel] fetchmail 6.3.10-beta1 test version

[Matthias A. <mat...@gm...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Greetings,

I have uploaded a fetchmail 6.3.10 beta version - "beta" means that some
of the bug fixes have been nontrivial, and testing is required. See the
log below for bugs supposed to be fixed.

Find the tarball at the usual download location:
<http://home.pages.de/~mandree/fetchmail/>.

Happy fetching
Matthias

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Changes in fetchmail 6.3.10-beta1 since 6.3.9:

# ADVANCE WARNING OF FEATURES TO BE REMOVED OR CHANGED IN FUTURE VERSIONS
* The maintainer may migrate fetchmail to C++ with STL or C#, and impose further
  requirements (dependencies), such as Boost or other class libraries.
* The softbounce option default will change to "false" in the next release.

fetchmail 6.3.10 (not yet released, beta1 on 2009-05-25):

# INCOMPATIBLE BUGFIXES AND CHANGES
* Fetchmail no longer drops permanently undelivered messages by default, to
  match historic documentation.  It does this by adding a new "softbounce"
  option, see below.
  Fixes Debian Bug#471283, demotes Debian Bug#494418 to wishlist.
* There is a new "softbounce" global option that prevents the deletion of
  messages that have not been forwarded. It defaults to "true" for fetchmail
  6.3.X in order to match historic documentation. This may change its default
  in the next major release.

# BUGFIXES
* Fix misuse of canonical autoconf target as _TARGET when it should have been
  _HOST. Report and patch courtesy of Diego E. "Flameeyes" Pettenò.
  Details: http://blog.flameeyes.eu/2009/01/01/the-canonical-target
* Do not lose PS_MAXFETCH (13) exit status when hitting maxpoll. Reported by
  Michelle Konzack, Debian Bug#508667.
* Do not overlap source and destination fields in snprintf() in interface.c.
  Courtesy of Nico Golde, Debian.
* When a pre- or post-connect command fails, now report the exit status or
  termination signal properly through sys/wait.h macros.
* When acquiring a body, understand NIL ("no such data item"), as returned by
  some MS Exchange versions. Fixes BerliOS Bug #11980 by KB Sriram.
* Make progress tickers (-v/--showdots) consistent, and update documentation 
  accordingly ("." for each 1024 octets read, "#" for a header written, and "*" 
  for each body line written.)
  The conditions under which these had been printed were inconsistent,
  illogical, and documentation hadn't matched real behaviour for long.
* For NTLM authentication, use dynamically allocated buffers.
  Fixes Debian Bug#449179, reported by Stepan Golosunov.
* Non-delivery notice ("bounce mail") now mentions the original reason again,
  before the address list. This fixes a regression introduced in 6.3.0.
* Several compiler warnings were fixed.
* The minimum recommended SMTP (RFC-5321) timeouts are enforced to leave
  sufficient time for the listener to respond. Some synchronous listeners,
  particularly when used with spam filtering and other policy enforcement
  services, take extended amounts of time to process messages after the sender,
  recipient, or data block and EOM line. This can cause fetchmail to not wait
  long enough for the "250 Ok" and make fetchmail believe the message wasn't
  properly delivered when in fact it was; fetchmail would then retry the
  download next time and never make progress.
  Fixes Berlios Bug #10972, reported by Viktor Binzberger.
* The ESMTP/LMTP client will now apply an application-specific timeout while
  waiting for the EHLO/LHLO response, rather than wait for the server or TCP
  connection timeout.
* Treat 530 errors as temporary, so as not to delete messages on configuration
  errors. Partially taken from Petr Cerny's patch in Novell Bugzilla #246829.
  The 501 part of said patch was not added, as the maintainer is not convinced
  501 is a temporary condition, and softbounce takes care of this anyways.

# CHANGES
* Make the comparison of the SSL fingerprints case insensitive, to
  ease its use. Suggested by Daniel Richard G.

# CHANGES TO CONTRIB
* Fix bashism in contrib/fetchsetup. Fixes Debian Bug#530081.

# DOCUMENTATION
* Some parts of the the manual page were revised for clarity, accuracy, and
  updated recommendations (particularly SSL/TLS) and formatting conventions from
  man-pages(7).
* The README and README.SSL documents were updated.
* A document, README.SSL-SERVER, was added to describe server-side requirements
  for proper SSL and/or TLS service offerings. These are not specific to
  fetchmail.

# TRANSLATION UPDATES AND ADDITIONS (ordered by language name):
* [en_GB] English/British
* [de]    German
* [it]    Italian (Vincenzo Campanella)
* [es]    Spanish/Castilian (Francisco Molinero)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEARECAAYFAkobHBYACgkQvmGDOQUufZVlLQCcC3BGECLvxLPFRnFoK3kPhDUL
aMwAoKnIMkE/WmOevCCPC/G/aR0ROl/z
=o/Ah
-----END PGP SIGNATURE-----

[fetchmail-devel] New Spanish PO file for 'fetchmail' (version 6.3.9-rc3)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Spanish team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/es.po

(This file, 'fetchmail-6.3.9-rc3.es.po', has just now been sent to you in
a separate email.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] New Spanish PO file for 'fetchmail' (version 6.3.9-rc3)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Spanish team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/es.po

(This file, 'fetchmail-6.3.9-rc3.es.po', has just now been sent to you in
a separate email.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] recent format string fixes (was: r5291 - branches/BRANCH_6-3)

[Matthias A. <mat...@gm...>]

Am 05.05.2009, 00:00 Uhr, schrieb <sv...@mk...>:

> Author: m-a
> Date: 2009-05-04 17:00:18 -0500 (Mon, 04 May 2009)
> New Revision: 5291
>
> Modified:
>    branches/BRANCH_6-3/pop3.c
>    branches/BRANCH_6-3/report.c
>    branches/BRANCH_6-3/smtp.c
> Log:
> Fix format string bugs.

Note that these are uncritical, meaning they cannot be exploited to mount  
attacks on fetchmail. Analysis:

pop3.c uses generated data of the TOP 12345 1 form => no % here => safe.

smtp.c uses report() to print string literals without placeholders,  
possibly translated through gettext. The English strings do not contain %  
fields. => safe. Even if a translation introduced %-strings, gettext()  
would reject such fuzzy translations and use the English text instead.

report.c was sort-of-unsafe, but the bug was visible only for around 8  
minutes, and never part of a release. Nevermind.

> Modified: branches/BRANCH_6-3/pop3.c
> ===================================================================
> --- branches/BRANCH_6-3/pop3.c	2009-05-04 21:52:32 UTC (rev 5290)
> +++ branches/BRANCH_6-3/pop3.c	2009-05-04 22:00:18 UTC (rev 5291)
> @@ -771,7 +771,7 @@
>      int got_it;
>      char buf [POPBUFSIZE+1];
>      snprintf(buf, sizeof(buf), "TOP %d 1", num);
> -    if ((ok = gen_transact(sock, buf )) != 0)
> +    if ((ok = gen_transact(sock, "%s", buf)) != 0)
>         return ok;
>      got_it = 0;
>      while ((ok = gen_recv(sock, buf, sizeof(buf))) == 0)
>
> Modified: branches/BRANCH_6-3/report.c
> ===================================================================
> --- branches/BRANCH_6-3/report.c	2009-05-04 21:52:32 UTC (rev 5290)
> +++ branches/BRANCH_6-3/report.c	2009-05-04 22:00:18 UTC (rev 5291)
> @@ -274,7 +274,7 @@
>      if (partial_message_size_used != 0)
>      {
>  	partial_message_size_used = 0;
> -	report(errfp, partial_message);
> +	report(errfp, "%s", partial_message);
>  	partial_suppress_tag = 1;
>      }
>  }
>
> Modified: branches/BRANCH_6-3/smtp.c
> ===================================================================
> --- branches/BRANCH_6-3/smtp.c	2009-05-04 21:52:32 UTC (rev 5290)
> +++ branches/BRANCH_6-3/smtp.c	2009-05-04 22:00:18 UTC (rev 5291)
> @@ -55,7 +55,7 @@
>  {
>      SockPrintf(sock, "*\r\n");
>      SockRead(sock, smtp_response, sizeof(smtp_response) - 1);
> -    if (outlevel >= O_MONITOR) report(stdout, msg);
> +    if (outlevel >= O_MONITOR) report(stdout, "%s", msg);
>  }
> static void SMTP_auth(int sock, char smtp_mode, char *username, char  
> *password, char *buf)
>
> _______________________________________________
> fetchmail-svn mailing list
> fet...@li...
> https://lists.berlios.de/mailman/listinfo/fetchmail-svn



-- 
Matthias Andree

Re: [fetchmail-devel] Drop privileges

[Sunil S. <sh...@bo...>]

Hi Thomas,

Quoting from Thomas Jarosch's mail on Thu, Apr 30, 2009:
> We don't run fetchmail in daemon mode, we start it for a single run
> as it doesn't know if the online connection is available or if the
> mail system is currently reconfigured on our site. Basically fetchmail
> is controlled by another daemon.

You may still run fetchmail in daemon mode with a high poll interval:

$ fetchmail --daemon 86400               # poll just once a day ...

Whenever you are online, just wake it up:

$ fetchmail                              # ... or more

> The new features are completly optional and allow you to harden
> the security in basic setups if running in non-daemon mode.

Unfortunately, the sandbox is not going to work in two common setups
even in non-daemon mode:

1. POP3+UIDL: the idfile is read from /home/user/.fetchids and will be
written back to /sandbox/home/user/.fetchids. In effect, the idfile is
not getting updated at all!

2. MDA: this delivery program (like maildrop, procmail, or even a
user-defined script) is not going to work in the sandbox.

> If the normal fetchmail usage is the daemon mode,
> then I guess it's not very useful. Hmm.

Well, you too should try using the daemon mode...

-- 
Sunil Shetye.

Re: [fetchmail-devel] Drop privileges

[Thomas J. <tho...@in...>]

Hello Sunil,

> Here are some thoughts on the options in your patch.

Thanks for your review!

> The run_chroot option:
> ...
> Your patch will work correctly only if fetchmail does not open any
> file or run any script after the chroot.
>
> Also, ensuring that the preconnect script runs properly in the chroot
> will be next to impossible.

Yes, that is true.

> The run_user and run_group options:
>
> fetchmail is essentially a user-level daemon program. Unlike a system
> daemon, it does not require special privileges to read the
> configuration files and it does not bind to a reserved port. So, it is
> far simpler to start fetchmail with a command like:
>
> # su user -c fetchmail
>
> rather than
>
> # fetchmail --run_user user
>
> Is it really useful to have passwords in a root-owned file? Does it
> add to security? In case you really want that, you may run fetchmail
> in this fashion as root:
>
> # su user -c fetchmail -f - < /etc/fetchmailrc
>
> The added advantage is that the lockfile is the correct place.
>
> Another problem will be that if /etc/fetchmailrc is modified after
> fetchmail is invoked with the --run_user option, it will detect the
> file modification and exit because the file cannot be read again due
> to dropping of privileges. Try this:
>
> # fetchmail --run_user user --daemon 1800
> [ wait for one poll to complete ]
> # touch /etc/fetchmailrc
> # fetchmail
>
> This will not happen with the invocation with redirection!

Interesting approach with the redirection, didn't think of it before.

We don't run fetchmail in daemon mode, we start it for a single run
as it doesn't know if the online connection is available or if the
mail system is currently reconfigured on our site. Basically fetchmail
is controlled by another daemon.

The new features are completly optional and allow you to harden
the security in basic setups if running in non-daemon mode.

If the normal fetchmail usage is the daemon mode,
then I guess it's not very useful. Hmm.

Cheers,
Thomas

Re: [fetchmail-devel] Drop privileges

[Sunil S. <sh...@bo...>]

Hi Thomas Jarosch,

Quoting from Thomas Jarosch's mail on Fri, Mar 27, 2009:
> attached is a patch which allows fetchmail to drop root privileges
> and run inside a chroot. The idea is to read the config file as root (as it 
> contains passwords) and setup the logging. The root privileges get dropped 
> after initialization.
> 
> All new features are optional. This will leave a stale lockfile after
> the fetchmail run (not enough privileges to write to /var/run or even
> not accessible inside the chroot), fetchmail perfectly
> handles that on the next invocation.
> 
> Upstream integration or feedback is welcome :-)
> The patch is against fetchmail 6.3.9.

Here are some thoughts on the options in your patch.


The run_chroot option:

The patch does not handle files which are being opened after chroot.
Any file opened after the chroot will be in the wrong location. For
example, in this invocation with your patch:

# fetchmail --run_chroot /home/user --logfile /home/user/fetchmail.logs \
    --idfile /home/user/.fetchids --bsmtp /home/user/bsmtp.file \
    --preconnect '/home/user/preconnect.py script-arguments'

the logfile is being opened before the chroot, the bsmtp file is being
opened after the chroot (i.e. /home/user/home/user/bsmtp.file), and
the idfile is being opened before the chroot for reading *and* after
the chroot for writing! The preconnect script is running in the
chroot.

Your patch will work correctly only if fetchmail does not open any
file or run any script after the chroot.

Also, ensuring that the preconnect script runs properly in the chroot
will be next to impossible.


The run_user and run_group options:

fetchmail is essentially a user-level daemon program. Unlike a system
daemon, it does not require special privileges to read the
configuration files and it does not bind to a reserved port. So, it is
far simpler to start fetchmail with a command like:

# su user -c fetchmail

rather than

# fetchmail --run_user user

Is it really useful to have passwords in a root-owned file? Does it
add to security? In case you really want that, you may run fetchmail
in this fashion as root:

# su user -c fetchmail -f - < /etc/fetchmailrc

The added advantage is that the lockfile is the correct place.

Another problem will be that if /etc/fetchmailrc is modified after
fetchmail is invoked with the --run_user option, it will detect the
file modification and exit because the file cannot be read again due
to dropping of privileges. Try this:

# fetchmail --run_user user --daemon 1800
[ wait for one poll to complete ]
# touch /etc/fetchmailrc
# fetchmail

This will not happen with the invocation with redirection!

-- 
Sunil Shetye.

Re: [fetchmail-devel] [patch] Drop privileges

[Thomas J. <tho...@in...>]

Hello Matthias,

On Wednesday, 22. April 2009 18:22:57 Matthias Andree wrote:
> We'd probably need more detailed instructions of what doesn't work inside
> chroots (you mentioned /var/run -- and this may require revising the stale
> lock detection and/or logging), and what needs to be inside the chroot
> because it's re-read at run-time, for instance, the resolver is
> re-initialized frequently in daemon mode, so we need some libs and config
> files inside the chroot, and depending on how much caching the libc does,
> we may need f. i. /etc/services... unfortunately, this is highly OS and
> version dependent.

If someone uses the daemon mode inside the chroot, I guess /etc/fetchmailrc
needs to be owned by the unprivileged user, too.
Though I don't use the daemon mode myself.

My current chroot sandbox looks like this:

[root@intranator fetchmail]# ls -alR
.:
total 5
drwxr-xr-x    2 root     root         1024 Apr 24 16:03 etc
drwxr-xr-x    2 root     root         1024 Apr 24 16:00 lib
drwxr-x---    2 fetchmai fetchmai     1024 Apr 24 11:59 tmp

./etc:
total 18
-rw-r--r--    1 root     root           26 Apr 24 16:03 host.conf
-rw-r--r--    1 root     root           48 Apr 24 16:03 hosts
-rw-r--r--    1 root     root          837 Apr 24 16:03 localtime
-rw-r--r--    1 root     root           34 Apr 24 16:03 resolv.conf
-rw-r--r--    1 root     root        11375 Apr 24 16:03 services

./lib:
total 1264
-rwxr-xr-x    1 root     root       218749 Nov  2  2004 libnss_compat-2.1.3.so
lrwxrwxrwx    1 root     root           22 Apr 24 16:00 libnss_compat.so.2 -> libnss_compat-2.1.3.so
-rwxr-xr-x    1 root     root        68957 Nov  2  2004 libnss_dns-2.1.3.so
lrwxrwxrwx    1 root     root           19 Apr 24 16:00 libnss_dns.so.2 -> libnss_dns-2.1.3.so
-rwxr-xr-x    1 root     root       245542 Nov  2  2004 libnss_files-2.1.3.so
lrwxrwxrwx    1 root     root           21 Apr 24 16:00 libnss_files.so.2 -> libnss_files-2.1.3.so
-rwxr-xr-x    1 root     root        69129 Nov  2  2004 libnss_hesiod-2.1.3.so
lrwxrwxrwx    1 root     root           22 Apr 24 16:00 libnss_hesiod.so.2 -> libnss_hesiod-2.1.3.so
-rwxr-xr-x    1 root     root       255929 Nov  2  2004 libnss_nis-2.1.3.so
lrwxrwxrwx    1 root     root           19 Apr 24 16:00 libnss_nis.so.2 -> libnss_nis-2.1.3.so
-rwxr-xr-x    1 root     root       254328 Nov  2  2004 libnss_nisplus-2.1.3.so
lrwxrwxrwx    1 root     root           23 Apr 24 16:00 libnss_nisplus.so.2 -> libnss_nisplus-2.1.3.so
-rwxr-xr-x    1 root     root       169644 Apr 24 16:00 libresolv-2.1.3.so
lrwxrwxrwx    1 root     root           18 Apr 24 16:00 libresolv.so.2 -> libresolv-2.1.3.so

> I still need to think how to integrate all this, since 6.3.X is supposed to
> be a bug-fix branch... it seems I need to revive the bit-rotten 6.4.X
> branch for new features, mark 6.3 "regression fixes only" and move on...

Put it in 6.4.x if you like, it works fine for me in 6.3.

> I'm also pondering whether fetchmail needs a split-process model some day,
> which might then solve issues such as the /var/run (PID file) removal
> problem you mentioned and perhaps allow for a pool of concurrent fetch
> children - useful with multiple accounts. That's certainly for later...

So much ideas - so little time :-)

Cheers,
Thomas

Re: [fetchmail-devel] [Bug #15551] Fetmail envelope option - Reg

[Rob M. <rob...@gm...>]

Note - this really belongs on the fetchmail-users mailing list, it's
not a bug in fetchmail.

<---SNIP--->
>
> But if I give the option in the .fetchmailrc as follows
>
> envelope [1] Deivered-To
>
> it throws an error as follows,
> fetchmail:/root/.fetchmailrc:9: syntax error at Delivered-To
>
> Can any one Help me to rectify this error ?

The universal standard in Unix documentation (and others) is to
enclose optional arguments in square brackets.  You do not include
those brackets when providing the argument.

envelope 1 Delivered-To

Would be the correct entry

-- 
                 Please keep list traffic on the list.

Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

[fetchmail-devel] [Bug #15551] Fetmail envelope option - Reg

[<ad...@be...>]

Bug #15551, was updated on 2009-Apr-23 14:30
Here is a current snapshot of the bug.

Project: Community Fetchmail
Category: None
Status: Open
Resolution: None
Bug Group: None
Priority: 5
Submitted by: arunvasan
Assigned to : none
Summary: Fetmail envelope option - Reg

Details: Hi and Hello to all,

I have one clsrification in the Fetchmail. 
My Fetchmail version is 6.3.6
My .fetchmailrc file is as follows

set logfile '/var/log/fetchmail.log'
set invisible
set bouncemail
set showdots

poll mail.mydomain.com
protocol pop3
no dns
envelope Delivered-To
qvirtual 'mailindx-mydomain:com-'
localdomains mydomain.com
user test
password test
is * here
smtphost intranet.mydomain.com
no rewrite
flush
fetchall

All our mail users mail will be received at mail.mydomain.com with in a comman user id having their original email id in Delivered-To header. (Mutidrop)

This is Delivered-To header will be as a second header to my email.

In the fetchmail man pages I came to know that 

-E <line> | --envelope <line>
(Keyword: envelope; Multidrop only)
In the configuration file, an enhanced syntax is used:
envelope [<count>] <line>

This option changes the header fetchmail assumes will carry a copy of the mail's envelope address. Normally this is 'X-Envelope-To', but as this
header is not standard, practice varies. See the discussion of multidrop address handling below. As a special case, 'envelope "Received"' enables
parsing of sendmail-style Received lines. This is the default, and it should not be necessary unless you have globally disabled Received parsing
with 'no envelope' in the .fetchmailrc file.

The optional count argument (only available in the configuration file) determines how many header lines of this kind are skipped. A count of 1
means: skip the first, take the second. A count of 2 means: skip the first and second, take the third, and so on.

But if I give the option in the .fetchmailrc as follows

envelope [1] Deivered-To

it throws an error as follows,
fetchmail:/root/.fetchmailrc:9: syntax error at Delivered-To

Can any one Help me to rectify this error ?
Thanks in advance.

Regards,
Arun


For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=15551&group_id=1824

Re: [fetchmail-devel] [patch] Drop privileges

[Matthias A. <mat...@gm...>]

Thomas Jarosch schrieb:
> Hello together,
> 
> attached is a patch which allows fetchmail to drop root privileges
> and run inside a chroot. The idea is to read the config file as root (as it 
> contains passwords) and setup the logging. The root privileges get dropped 
> after initialization.
> 
> All new features are optional. This will leave a stale lockfile after
> the fetchmail run (not enough privileges to write to /var/run or even
> not accessible inside the chroot), fetchmail perfectly
> handles that on the next invocation.
> 
> Upstream integration or feedback is welcome :-)
> The patch is against fetchmail 6.3.9.
> 
> Have a nice weekend,
> Thomas

Hi Thomas,

sorry for not giving immediate feedback - I've been busy with other things,
among them real life and also leafnode releases...

At first glance, the patch looks sound, thanks a lot.
There's some point in integrating such functionality inside fetchmail, i.
e. keeping sensitive data outside fetchmail's accessible area.

We'd probably need more detailed instructions of what doesn't work inside
chroots (you mentioned /var/run -- and this may require revising the stale
lock detection and/or logging), and what needs to be inside the chroot
because it's re-read at run-time, for instance, the resolver is
re-initialized frequently in daemon mode, so we need some libs and config
files inside the chroot, and depending on how much caching the libc does,
we may need f. i. /etc/services... unfortunately, this is highly OS and
version dependent.

I still need to think how to integrate all this, since 6.3.X is supposed to
be a bug-fix branch... it seems I need to revive the bit-rotten 6.4.X
branch for new features, mark 6.3 "regression fixes only" and move on...

I'm also pondering whether fetchmail needs a split-process model some day,
which might then solve issues such as the /var/run (PID file) removal
problem you mentioned and perhaps allow for a pool of concurrent fetch
children - useful with multiple accounts. That's certainly for later...

Best
Matthias

Re: [fetchmail-devel] [patch] Drop privileges

[Thomas J. <tho...@in...>]

On Friday, 27. March 2009 17:14:02 Thomas Jarosch wrote:
> Upstream integration or feedback is welcome :-)
> The patch is against fetchmail 6.3.9.

Any comment on the patch?

Regards,
Thomas

Re: [fetchmail-devel] Password prompt on stderr

[Matthias A. <mat...@gm...>]

Am 09.02.2009, 12:58 Uhr, schrieb Prasad J Pandit <pj....@ya...>:

>    Hello folks,
>
> I was using fetchmail-6.3.9 as
>
> $ fetchmail -cu <name> -D na...@my... --ssl mail-server 2> /dev/null
>
> and noticed that the prompt which says: "Enter password for %s@%s" is  
> displayed on stderr instead of steout.

Hi Prasad,

thanks for your effort to contribute.

While undocumented, chances are that this was intentional for some reason.  
I've checked the relevant code, and fetchmail opens /dev/tty from file  
descriptor #2 with fdopen(), which is also stderr, to save/change/restore  
settings such as not echoing the password back to the user.

I wonder if it's more common that people discard stdout (progress) instead  
of stderr (problems) - and I suspect that was the reason to use stderr...

At least, if we go this way, we need to fix the fdopen as well to use  
stdout.

I wonder if "-s" (--silent) is an option that works for you if you want  
less output, rather than discarding 2>/dev/null - it is currently unclear  
to me which ACTUAL problem you try to solve by redirecting stderr to the  
bit bucket. Would you care to enlighten me a bit?

> The attached patch fixes this bug and a warning message encountered  
> while compiling the source.
>
>
> The other warning messages were:
>
> smbutil.c:207: warning: the address of ‘lmRespData’ will always evaluate  
> as ‘true’
> smbutil.c:208: warning: the address of ‘ntRespData’ will always evaluate  
> as ‘true’
>
> that's because both are statically allocated array variables.

Yes. :-( I currently understand too little of NTLM authentication to  
actually touch this code myself. I'd be more comfortable with using an  
external NTLM library. I still need to look at the GSoC 2008 patches to  
add OpenChange support (my fault - haven't had the time, and I feel sorry  
for not yet merging Yangyan's code) - perhaps this is a way to do away  
with our old NTLM code.

-- 
Matthias Andree

[fetchmail-devel] [patch] Drop privileges

[Thomas J. <tho...@in...>]

Hello together,

attached is a patch which allows fetchmail to drop root privileges
and run inside a chroot. The idea is to read the config file as root (as it 
contains passwords) and setup the logging. The root privileges get dropped 
after initialization.

All new features are optional. This will leave a stale lockfile after
the fetchmail run (not enough privileges to write to /var/run or even
not accessible inside the chroot), fetchmail perfectly
handles that on the next invocation.

Upstream integration or feedback is welcome :-)
The patch is against fetchmail 6.3.9.

Have a nice weekend,
Thomas

[fetchmail-devel] Password prompt on stderr

[Prasad J P. <pj....@ya...>]

   Hello folks,

I was using fetchmail-6.3.9 as

$ fetchmail -cu <name> -D na...@my... --ssl mail-server 2> /dev/null

and noticed that the prompt which says: "Enter password for %s@%s" is displayed on stderr instead of steout. 

The attached patch fixes this bug and a warning message encountered while compiling the source.


The other warning messages were:

smbutil.c:207: warning: the address of ‘lmRespData’ will always evaluate as ‘true’
smbutil.c:208: warning: the address of ‘ntRespData’ will always evaluate as ‘true’

that's because both are statically allocated array variables.

Hope you find it(patch) useful. I would like to hear your comments about the same.

Thank you.
---
Regards
    -Prasad
PS: Please don't send me html/attachment/Fwd mails


      Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/

[fetchmail-devel] test

[Prasad J P. <pj....@ya...>]

---
Regards
    -Prasad
PS: Please don't send me html/attachment/Fwd mails


      Download prohibited? No problem. CHAT from any browser, without download. Go to http://in.webmessenger.yahoo.com/

Re: [fetchmail-devel] POP3 UIDL behavior

[Seth M. <se...@ro...>]

And here's the debug output using my patched version of fetchmail:


fetchmail:/var/lib/fetchmail/test# cat uidl
se...@ma... 0001179e45a1729f
se...@ma... 0001179f45a1729f
se...@ma... 000117a045a1729f
se...@ma... 000117a145a1729f
se...@ma... 0001179e45a1729f
se...@ma... 0001179f45a1729f
se...@ma... 000117a045a1729f
se...@ma... 000117a145a1729f

fetchmail:/var/lib/fetchmail/test# su fetchmail -c 'fetchmail --quit -f 
/var/lib/fetchmail/test/fetchmailrc --pidfile 
/var/lib/fetchmail/test/fetchmail.pid -i /var/lib/fetchmail/test/uidl -vvN'
fetchmail: Old UID list from mail.mattinen.org: 0001179e45a1729f 
0001179f45a1729f 000117a045a1729f 000117a145a1729f 0001179e45a1729f 
0001179f45a1729f 000117a045a1729f 000117a145a1729f <empty>
fetchmail: Scratch list of UIDs: <empty>
fetchmail: starting fetchmail 6.3.9 daemon
fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 
Jan 2009 10:08:20 PM PST: poll started
fetchmail: Trying to connect to 67.118.43.90/995...connected.
fetchmail: Issuer Organization: Courier Mail Server
fetchmail: Issuer CommonName: mail.mattinen.org
fetchmail: Server CommonName: mail.mattinen.org
fetchmail: mail.mattinen.org key fingerprint: 
0D:B9:98:10:A0:C7:B6:4C:70:92:6C:EE:C6:BB:99:1C
fetchmail: Server certificate verification error: self signed certificate
fetchmail: Server certificate verification error: certificate has expired
fetchmail: POP3< +OK Dovecot ready.
fetchmail: POP3> CAPA
fetchmail: POP3< +OK
fetchmail: POP3< CAPA
fetchmail: POP3< TOP
fetchmail: POP3< UIDL
fetchmail: POP3< RESP-CODES
fetchmail: POP3< PIPELINING
fetchmail: POP3< USER
fetchmail: POP3< SASL PLAIN
fetchmail: POP3< .
fetchmail: POP3> USER sethm
fetchmail: POP3< +OK
fetchmail: POP3> PASS *
fetchmail: POP3< +OK Logged in.
fetchmail: selecting or re-polling default folder
fetchmail: POP3> STAT
fetchmail: POP3< +OK 4 1792
fetchmail: POP3> UIDL
fetchmail: POP3< +OK
fetchmail: POP3< 1 0001179e45a1729f
fetchmail: POP3< 2 0001179f45a1729f
fetchmail: POP3< 3 000117a045a1729f
fetchmail: POP3< 4 000117a145a1729f
fetchmail: POP3< .
fetchmail: 4 messages (4 seen) for sethm at mail.mattinen.org (1792 octets).
fetchmail: skipping message se...@ma...:1 not flushed
fetchmail: skipping message se...@ma...:2 not flushed
fetchmail: skipping message se...@ma...:3 not flushed
fetchmail: skipping message se...@ma...:4 not flushed
fetchmail: POP3> QUIT
fetchmail: POP3< +OK Logging out.
fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 
Jan 2009 10:08:20 PM PST: poll completed
fetchmail: New UID list from mail.mattinen.org: 0001179e45a1729f = 1 
0001179f45a1729f = 1 000117a045a1729f = 1 000117a145a1729f = 1 <empty>
fetchmail: swapping UID lists
fetchmail: Query status=1 (NOMAIL)
fetchmail: Writing fetchids file.
fetchmail: sleeping at Wed 28 Jan 2009 10:08:20 PM PST for 15 seconds
fetchmail: terminated with signal 2
fetchmail: Writing fetchids file.

fetchmail:/var/lib/fetchmail/test# cat uidl
se...@ma... 0001179e45a1729f
se...@ma... 0001179f45a1729f
se...@ma... 000117a045a1729f
se...@ma... 000117a145a1729f


Nothing in the configuration was changed and the same uidl file 
(including the duplicates, which were weeded out) was used. Note the 
oldlist is now populated at start time whereas it wasn't with the 
unpatched version.

-- 
Seth Mattinen		se...@ro...
Roller Network LLC

Re: [fetchmail-devel] POP3 UIDL behavior

[Seth M. <se...@ro...>]

As promised here's my test run with debugging enabled that prompted me 
to dig into the uid.c file. This is with stock fetchmail-6.3.9 
configured with "./configure --with-ssl":


fetchmail:/var/lib/fetchmail/test# cat uidl
cat: uidl: No such file or directory
fetchmail:/var/lib/fetchmail/test# su fetchmail -c 'fetchmail --quit -f 
/var/lib/fetchmail/test/fetchmailrc --pidfile 
/var/lib/fetchmail/test/fetchmail.pid -i /var/lib/fetchmail/test/uidl -vvN'
fetchmail: Old UID list from mail.mattinen.org: <empty>
fetchmail: Scratch list of UIDs: <empty>
fetchmail: starting fetchmail 6.3.9 daemon
fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 
Jan 2009 09:55:50 PM PST: poll started
fetchmail: Trying to connect to 67.118.43.90/995...connected.
fetchmail: Issuer Organization: Courier Mail Server
fetchmail: Issuer CommonName: mail.mattinen.org
fetchmail: Server CommonName: mail.mattinen.org
fetchmail: mail.mattinen.org key fingerprint: 
0D:B9:98:10:A0:C7:B6:4C:70:92:6C:EE:C6:BB:99:1C
fetchmail: Server certificate verification error: self signed certificate
fetchmail: Server certificate verification error: certificate has expired
fetchmail: POP3< +OK Dovecot ready.
fetchmail: POP3> CAPA
fetchmail: POP3< +OK
fetchmail: POP3< CAPA
fetchmail: POP3< TOP
fetchmail: POP3< UIDL
fetchmail: POP3< RESP-CODES
fetchmail: POP3< PIPELINING
fetchmail: POP3< USER
fetchmail: POP3< SASL PLAIN
fetchmail: POP3< .
fetchmail: POP3> USER sethm
fetchmail: POP3< +OK
fetchmail: POP3> PASS *
fetchmail: POP3< +OK Logged in.
fetchmail: selecting or re-polling default folder
fetchmail: POP3> STAT
fetchmail: POP3< +OK 4 1792
fetchmail: POP3> UIDL
fetchmail: POP3< +OK
fetchmail: POP3< 1 0001179e45a1729f
fetchmail: 1 is unseen
fetchmail: POP3< 2 0001179f45a1729f
fetchmail: 2 is unseen
fetchmail: POP3< 3 000117a045a1729f
fetchmail: 3 is unseen
fetchmail: POP3< 4 000117a145a1729f
fetchmail: 4 is unseen
fetchmail: POP3< .
fetchmail: 4 messages for sethm at mail.mattinen.org (1792 octets).
...downloading happens here...
fetchmail: POP3> QUIT
fetchmail: POP3< +OK Logging out.
fetchmail: SMTP> QUIT
fetchmail: SMTP< 221 2.0.0 Bye
fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 
Jan 2009 09:55:55 PM PST: poll completed
fetchmail: New UID list from mail.mattinen.org: 0001179e45a1729f = 1 
0001179f45a1729f = 1 000117a045a1729f = 1 000117a145a1729f = 1 <empty>
fetchmail: swapping UID lists
fetchmail: Writing fetchids file.
fetchmail: sleeping at Wed 28 Jan 2009 09:55:55 PM PST for 15 seconds
fetchmail: awakened at Wed 28 Jan 2009 09:56:10 PM PST
fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 
Jan 2009 09:56:10 PM PST: poll started
fetchmail: Trying to connect to 67.118.43.90/995...connected.
fetchmail: Issuer Organization: Courier Mail Server
fetchmail: Issuer CommonName: mail.mattinen.org
fetchmail: Server CommonName: mail.mattinen.org
fetchmail: mail.mattinen.org key fingerprint: 
0D:B9:98:10:A0:C7:B6:4C:70:92:6C:EE:C6:BB:99:1C
fetchmail: Server certificate verification error: self signed certificate
fetchmail: Server certificate verification error: certificate has expired
fetchmail: POP3< +OK Dovecot ready.
fetchmail: POP3> CAPA
fetchmail: POP3< +OK
fetchmail: POP3< CAPA
fetchmail: POP3< TOP
fetchmail: POP3< UIDL
fetchmail: POP3< RESP-CODES
fetchmail: POP3< PIPELINING
fetchmail: POP3< USER
fetchmail: POP3< SASL PLAIN
fetchmail: POP3< .
fetchmail: POP3> USER sethm
fetchmail: POP3< +OK
fetchmail: POP3> PASS *
fetchmail: POP3< +OK Logged in.
fetchmail: selecting or re-polling default folder
fetchmail: POP3> STAT
fetchmail: POP3< +OK 4 1792
fetchmail: POP3> UIDL
fetchmail: POP3< +OK
fetchmail: POP3< 1 0001179e45a1729f
fetchmail: POP3< 2 0001179f45a1729f
fetchmail: POP3< 3 000117a045a1729f
fetchmail: POP3< 4 000117a145a1729f
fetchmail: POP3< .
fetchmail: 4 messages (4 seen) for sethm at mail.mattinen.org (1792 octets).
fetchmail: skipping message se...@ma...:1 not flushed
fetchmail: skipping message se...@ma...:2 not flushed
fetchmail: skipping message se...@ma...:3 not flushed
fetchmail: skipping message se...@ma...:4 not flushed
fetchmail: POP3> QUIT
fetchmail: POP3< +OK Logging out.
fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 
Jan 2009 09:56:10 PM PST: poll completed
fetchmail: New UID list from mail.mattinen.org: 0001179e45a1729f = 1 
0001179f45a1729f = 1 000117a045a1729f = 1 000117a145a1729f = 1 <empty>
fetchmail: swapping UID lists
fetchmail: Query status=1 (NOMAIL)
fetchmail: Writing fetchids file.
fetchmail: sleeping at Wed 28 Jan 2009 09:56:10 PM PST for 15 seconds
fetchmail: terminated with signal 2
fetchmail: Writing fetchids file.

fetchmail:/var/lib/fetchmail/test# cat uidl
se...@ma... 0001179e45a1729f
se...@ma... 0001179f45a1729f
se...@ma... 000117a045a1729f
se...@ma... 000117a145a1729f


Everything was normal. The uidl file looks fine too. I'm expecting it to 
read the uidl file and skip those four messages, correct? So I restart 
the daemon:


fetchmail:/var/lib/fetchmail/test# su fetchmail -c 'fetchmail --quit -f 
/var/lib/fetchmail/test/fetchmailrc --pidfile 
/var/lib/fetchmail/test/fetchmail.pid -i /var/lib/fetchmail/test/uidl -vvN'
fetchmail: Old UID list from mail.mattinen.org: <empty>
fetchmail: Scratch list of UIDs: <empty>
fetchmail: starting fetchmail 6.3.9 daemon
fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 
Jan 2009 09:56:16 PM PST: poll started
fetchmail: Trying to connect to 67.118.43.90/995...connected.
fetchmail: Issuer Organization: Courier Mail Server
fetchmail: Issuer CommonName: mail.mattinen.org
fetchmail: Server CommonName: mail.mattinen.org
fetchmail: mail.mattinen.org key fingerprint: 
0D:B9:98:10:A0:C7:B6:4C:70:92:6C:EE:C6:BB:99:1C
fetchmail: Server certificate verification error: self signed certificate
fetchmail: Server certificate verification error: certificate has expired
fetchmail: POP3< +OK Dovecot ready.
fetchmail: POP3> CAPA
fetchmail: POP3< +OK
fetchmail: POP3< CAPA
fetchmail: POP3< TOP
fetchmail: POP3< UIDL
fetchmail: POP3< RESP-CODES
fetchmail: POP3< PIPELINING
fetchmail: POP3< USER
fetchmail: POP3< SASL PLAIN
fetchmail: POP3< .
fetchmail: POP3> USER sethm
fetchmail: POP3< +OK
fetchmail: POP3> PASS *
fetchmail: POP3< +OK Logged in.
fetchmail: selecting or re-polling default folder
fetchmail: POP3> STAT
fetchmail: POP3< +OK 4 1792
fetchmail: POP3> UIDL
fetchmail: POP3< +OK
fetchmail: POP3< 1 0001179e45a1729f
fetchmail: 1 is unseen
fetchmail: POP3< 2 0001179f45a1729f
fetchmail: 2 is unseen
fetchmail: POP3< 3 000117a045a1729f
fetchmail: 3 is unseen
fetchmail: POP3< 4 000117a145a1729f
fetchmail: 4 is unseen
fetchmail: POP3< .
fetchmail: 4 messages for sethm at mail.mattinen.org (1792 octets).
...downloading happens here again even though it shouldn't...
fetchmail: POP3> QUIT
fetchmail: POP3< +OK Logging out.
fetchmail: SMTP> QUIT
fetchmail: SMTP< 221 2.0.0 Bye
fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 
Jan 2009 09:56:21 PM PST: poll completed
fetchmail: New UID list from mail.mattinen.org: 0001179e45a1729f = 1 
0001179f45a1729f = 1 000117a045a1729f = 1 000117a145a1729f = 1 <empty>
fetchmail: swapping UID lists
fetchmail: Writing fetchids file.
fetchmail: sleeping at Wed 28 Jan 2009 09:56:21 PM PST for 15 seconds
fetchmail: terminated with signal 2
fetchmail: Writing fetchids file.

fetchmail:/var/lib/fetchmail/test# cat uidl
se...@ma... 0001179e45a1729f
se...@ma... 0001179f45a1729f
se...@ma... 000117a045a1729f
se...@ma... 000117a145a1729f
se...@ma... 0001179e45a1729f
se...@ma... 0001179f45a1729f
se...@ma... 000117a045a1729f
se...@ma... 000117a145a1729f


It re-downloaded everything in the POP3 box and duplicated the uidl file 
after the daemon was restarted. According to the debug output it didn't 
bother loading the existing uidl file. Continuing this pattern causes 
the uidl file to grow out of control.

-- 
Seth Mattinen		se...@ro...
Roller Network LLC

[fetchmail-devel] New Vietnamese PO file for 'fetchmail' (version 6.3.9-rc3)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Vietnamese team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/vi.po

(This file, 'fetchmail-6.3.9-rc3.vi.po', has just now been sent to you in
a separate email.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

Re: [fetchmail-devel] POP3 UIDL behavior

[Seth M. <se...@ro...>]

Matthias Andree wrote:

> We have the following structure (roughly):
> 
>     while (fgets(buf, POPBUFSIZE, tmpfp) != (char *)NULL) {
>     // ...
>         /* find proper list and save it */
>         for (ctl = hostlist; ctl; ctl = ctl->next) {
>             if (strcasecmp(host, ctl->server.queryname) == 0
>                     && strcasecmp(user, ctl->remotename) == 0) {
>                 save_str(&ctl->oldsaved, id, UID_SEEN);
>                 break;
>             }
>         }
>     }
 >
> So if the "break;" breaks out of anything but the for ctl=... list,
> something's seriously wrong. The "break;" applies to the for... loop for
> as long as I remember C (that's nearly two decades now), not to the
> outer while... loop. Anything else is a hardware problem,
> compiler/optimizer bug or something is corrupting the heap or stack or
> unrelated variables. Can you check that? The latter may be revealed by
> running under valgrind (OTOH, OpenSSL reads uninitialized memory to
> harvest entropy, so expect a gazillion of complaints unless you have an
> ignore file for OpenSSL).

I completely agree; something weird is going on.


> What compiler, version, compilers flags, and operating system are you
> using? What platform/hardware does this happen on?

Vanilla Debian "stable" (kernel 2.6.18) on amd64.

fetchmail:/usr/local/src/fetchmail-6.3.9# gcc -v
Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure -v 
--enable-languages=c,c++,fortran,objc,obj-c++,treelang --prefix=/usr 
--enable-shared --with-system-zlib --libexecdir=/usr/lib 
--without-included-gettext --enable-threads=posix --enable-nls 
--program-suffix=-4.1 --enable-__cxa_atexit --enable-clocale=gnu 
--enable-libstdcxx-debug --enable-mpfr --enable-checking=release 
x86_64-linux-gnu
Thread model: posix
gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)


fetchmail:/usr/local/src/fetchmail-6.3.9-roller# make
make  all-recursive
make[1]: Entering directory `/usr/local/src/fetchmail-6.3.9-roller'
Making all in m4
make[2]: Entering directory `/usr/local/src/fetchmail-6.3.9-roller/m4'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/usr/local/src/fetchmail-6.3.9-roller/m4'
Making all in po
make[2]: Entering directory `/usr/local/src/fetchmail-6.3.9-roller/po'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/usr/local/src/fetchmail-6.3.9-roller/po'
make[2]: Entering directory `/usr/local/src/fetchmail-6.3.9-roller'
gcc -DHAVE_CONFIG_H -DLOCALEDIR=\"/usr/local/share/locale\" -I. 
-I./libesmtp   -g -O2 -I/usr/kerberos/include -MT uid.o -MD -MP -MF 
.deps/uid.Tpo -c -o uid.o uid.c
mv -f .deps/uid.Tpo .deps/uid.Po
gcc  -g -O2 -I/usr/kerberos/include  -L/usr/lib -o fetchmail socket.o 
getpass.o fetchmail.o env.o idle.o options.o daemon.o driver.o 
transact.o sink.o smtp.o uid.o mxget.o md5ify.o cram.o gssapi.o opie.o 
interface.o netrc.o unmime.o conf.o checkalias.o lock.o rcfile_l.o 
rcfile_y.o norm_charmap.o  pop3.o imap.o etrn.o odmr.o     libfm.a 
strlcpy.o strlcat.o  -lcrypt  -lresolv -lssl -lcrypto
make[2]: Leaving directory `/usr/local/src/fetchmail-6.3.9-roller'
make[1]: Leaving directory `/usr/local/src/fetchmail-6.3.9-roller'


> 
>> This is the specific fetchmailrc file that it's currently running with:
>>
>> set daemon 60
>> set no syslog
>> set logfile /var/lib/fetchmail/test/log
>> set idfile /var/lib/fetchmail/test/uidl
>> set no bouncemail
>> set spambounce
> 
> Unrelated, but please don't to the latter. Spambouncing hurts innocent
> bystanders.

I've made several changes to sink.c so it will only send bounce notices 
to the defined postmaster address. ;)


> I'm not convinced it's actually fixing bugs in fetchmail, rather than
> working around them in strange ways. I do not dispute that the patch is
> effective, but I'm loathe to apply the patch before I understand why
> exactly it's needed. My configuration looks similar to yours as far as
> UIDs are concerned, and I don't see problems here.
> 
> I'm inclined to believe in compiler bugs for now, but there may be a
> fetchmail bug in a different location that corrupts innocent storage and
> just happens to hit uid.c.
> 

When I have time (probably next week) I'll collect the debug output from 
before and after the patch to illustrate. The event that started me on 
this quest was fetchmail re-downloading the entire contents of a "keep" 
POP3 box every time the daemon started even though I had uidl enabled.

-- 
Seth Mattinen		se...@ro...
Roller Network LLC

Re: [fetchmail-devel] POP3 UIDL behavior

[Matthias A. <mat...@gm...>]

Seth Mattinen schrieb am 2009-01-23:

> Something other than a linear list written to disk on every iteration 
> would obviously be better. Personally I wouldn't worry about backwards 
> compatibility. If it were me, I probably wouldn't bother with loading 
> anything into memory and just do queries against the database. It'll end 
> up in the disk cache anyway.

Yeah, makes sense, although it will make Cygwin users suffer.
Not that I care much.

> > As far as I can see (it's around line 218 in my current uid.c version as
> > of post-6.3.9 SVN, in initialize_saved_lists()), it just breaks out of
> > "find the correct list" (there's a for loop three lines above) AFTER it
> > has saved the UID.
> > 
> > So if commenting out that break; at line 219 fixes anything for you,
> > your fix is for the symptoms, but not the root cause. We should then
> > find the latter.
> 
> I was using the released 6.3.9 version, not the bleeding edge from SVN 
> since I'm not going for a development environment. The break seemed to 
> be causing it to exit the outermost loop though and neither oldlist nor 
> scratchlist were being populated at start time when a uidl file was present.

We have the following structure (roughly):

    while (fgets(buf, POPBUFSIZE, tmpfp) != (char *)NULL) {
    // ...
        /* find proper list and save it */
        for (ctl = hostlist; ctl; ctl = ctl->next) {
            if (strcasecmp(host, ctl->server.queryname) == 0
                    && strcasecmp(user, ctl->remotename) == 0) {
                save_str(&ctl->oldsaved, id, UID_SEEN);
                break;
            }
        }
    }

So if the "break;" breaks out of anything but the for ctl=... list,
something's seriously wrong. The "break;" applies to the for... loop for
as long as I remember C (that's nearly two decades now), not to the
outer while... loop. Anything else is a hardware problem,
compiler/optimizer bug or something is corrupting the heap or stack or
unrelated variables. Can you check that? The latter may be revealed by
running under valgrind (OTOH, OpenSSL reads uninitialized memory to
harvest entropy, so expect a gazillion of complaints unless you have an
ignore file for OpenSSL).

What compiler, version, compilers flags, and operating system are you
using? What platform/hardware does this happen on?

I can provide fetchmail 6.3.9 executables for FreeBSD 7 i386, Linux i386
and Solaris SPARC if we want to see if it's a compiler issue on your
machine.

> >> 2) Everything is duplicated in the scratchlist even for UID entries we 
> >> should know about so I added a boolean flag to skips duplicates. 
> >> Otherwise when the UID file is written it'll include oldlist plus 
> >> duplicates in scratchlist. Is it supposed to duplicate everything into 
> >> the scratchlist for some reason? This isn't a functional problem, but it 
> >> does waste memory and disk space if you're thinking about large deployments.
> > 
> > This appears to be a side effect of your commenting out the "break;",
> > since the for loop you made complete now terminates with uid == NULL.
> 
> It is since according to the debug output oldlist was never populated by 
> initialize_saved_lists(), but without the break they both ran.

fetchmail switches the current list to oldlist somewhere, and picks
either new or old uid lists in other places; check uid_swap_lists() and
expunge_uids(), and "fastuidl" also bears a meaning here WRT which list
fetchmail actually works on.

> This is the specific fetchmailrc file that it's currently running with:
> 
> set daemon 60
> set no syslog
> set logfile /var/lib/fetchmail/test/log
> set idfile /var/lib/fetchmail/test/uidl
> set no bouncemail
> set spambounce

Unrelated, but please don't to the latter. Spambouncing hurts innocent
bystanders.

Otherwise, configuration and command line are sound.

> And one more thing:
> 3) If two servers of the same name were present but one was "skip" and 
> the other "poll", every "skip" would cause the UID file to be written 
> with duplicate data because the server name is in ctl more than once so 
> each loop through ctl during the uidl file write duplicated data. Added 
> cases to check the skip/uidl flags and ignore servers without those set.
> 
> Here's my patch that I ended up with that fixed the issues I observed. 
> I've been running it for several days without any side effects. The only 
> changes are to uid.c.
> 
> http://www.rollernet.us/opensource/patches/fetchmail-rollernet-uidfixes2.patch
> 
> I haven't done extensive testing against it beyond the few cases where I 
> had issues.

I'm not convinced it's actually fixing bugs in fetchmail, rather than
working around them in strange ways. I do not dispute that the patch is
effective, but I'm loathe to apply the patch before I understand why
exactly it's needed. My configuration looks similar to yours as far as
UIDs are concerned, and I don't see problems here.

I'm inclined to believe in compiler bugs for now, but there may be a
fetchmail bug in a different location that corrupts innocent storage and
just happens to hit uid.c.

HTH

-- 
Matthias Andree

Re: [fetchmail-devel] POP3 UIDL behavior

[Seth M. <se...@ro...>]

Matthias Andree wrote:
> Seth Mattinen schrieb am 2009-01-19:
> 
>> I was working on developing fetchmail for use in a large multiuser 
>> environment and came across some issues with the POP3 UID lists. I'm 
>> very new to fetchmail's code, so I don't know of these things are 
>> intentional or expected behavior, but they seemed "broken" to me.
> 
> Hi Seth,
> 
> you're scratching a sore spot ;-) Except for minimal streamlining and
> minor fixes, the uid.c code is what we inherited from Eric S. Raymond,
> who was very loathe to touch anything of it since it broke all too
> easily.
> 
> A while back, Sunil and I had exchanged patches to save UIDs in one file
> per account, but I didn't merge that code into fetchmail, to make
> fetchmail 6.3.X as drop-in compatible with 6.2.5 so that nobody would
> have excuses for not upgrading from 6.2.X to 6.3.X.
> 
> Let me beforehand state that the data structures for storing the UID
> lists are very inefficient, and costly at run-time. Several hundred kept
> messages bog down old computers, and several thousands bog down modern
> computers. We have cascaded linear lists, and we store the file very
> inefficiently on-disk (which is less of an issue today).
> 
> The whole UID issue is for reconsideration when I really start working
> on 6.4.X. At least, we need efficient memory and on-disk structures and
> we'd preferably save to one uid file per account, so we can do away with
> all the "load unrelated host info to scratchlist" stuff. It may be
> useful to store everything in some lightweight/embedded database instead
> (such as TokyoCabinet, Berkeley DB, SQLite). No decisions made yet, and
> maybe we need to benchmark our options here during development.

Something other than a linear list written to disk on every iteration 
would obviously be better. Personally I wouldn't worry about backwards 
compatibility. If it were me, I probably wouldn't bother with loading 
anything into memory and just do queries against the database. It'll end 
up in the disk cache anyway.


>> 1) The old UID list is never loaded when the daemon starts or restarts 
>> and the whole POP3 box downloaded every time for "keep" sources. My fix 
>> was to comment out the break after save_str() in uid.c because as far as 
>> I can tell, having that break in there causes it to never load the id 
>> file and it'll self-break when the for loop hits a null anyway.
> 
> I do not observe such behaviour - which version of fetchmail are you
> looking at?
> 
> As far as I can see (it's around line 218 in my current uid.c version as
> of post-6.3.9 SVN, in initialize_saved_lists()), it just breaks out of
> "find the correct list" (there's a for loop three lines above) AFTER it
> has saved the UID.
> 
> So if commenting out that break; at line 219 fixes anything for you,
> your fix is for the symptoms, but not the root cause. We should then
> find the latter.

I was using the released 6.3.9 version, not the bleeding edge from SVN 
since I'm not going for a development environment. The break seemed to 
be causing it to exit the outermost loop though and neither oldlist nor 
scratchlist were being populated at start time when a uidl file was present.



>> 2) Everything is duplicated in the scratchlist even for UID entries we 
>> should know about so I added a boolean flag to skips duplicates. 
>> Otherwise when the UID file is written it'll include oldlist plus 
>> duplicates in scratchlist. Is it supposed to duplicate everything into 
>> the scratchlist for some reason? This isn't a functional problem, but it 
>> does waste memory and disk space if you're thinking about large deployments.
> 
> This appears to be a side effect of your commenting out the "break;",
> since the for loop you made complete now terminates with uid == NULL.

It is since according to the debug output oldlist was never populated by 
initialize_saved_lists(), but without the break they both ran.


> Make sure the spelling on the command line matches the one in the rcfile
> exactly (including case) for a test and see if that helps.
> 
> Could you show me your configuration file (remove/mask passwords!) and
> your command line? Please do not edit host names, they are crucial here.
> If you're concerned about disclosing that in public, send the material
> to me GnuPG encrypted off-list (my key is 0x052e7d95), but please again
> without passwords.
> 


This is the specific fetchmailrc file that it's currently running with:

set daemon 60
set no syslog
set logfile /var/lib/fetchmail/test/log
set idfile /var/lib/fetchmail/test/uidl
set no bouncemail
set spambounce
set postmaster "fet...@bo..."

defaults
	pass8bits
	smtphost mail2.rollernet.us,mail.rollernet.us
	antispam 554

skip mail.mattinen.org with proto imap and tracepolls
	user "sethm"
	pass "xxxxxxxxxxxxx"
	smtpname "se...@ro..."
	ssl

poll mail.mattinen.org proto pop3 uidl tracepolls
	user "sethm"
	pass "xxxxxxxxxxxxx"
	smtpname "se...@ro..."
	ssl
	keep
	fastuidl 4

su fetchmail -c \
'fetchmail --quit \
-f /var/lib/fetchmail/test/fetchmailrc \
--pidfile /var/lib/fetchmail/test/fetchmail.pid'


And one more thing:
3) If two servers of the same name were present but one was "skip" and 
the other "poll", every "skip" would cause the UID file to be written 
with duplicate data because the server name is in ctl more than once so 
each loop through ctl during the uidl file write duplicated data. Added 
cases to check the skip/uidl flags and ignore servers without those set.

Here's my patch that I ended up with that fixed the issues I observed. 
I've been running it for several days without any side effects. The only 
changes are to uid.c.

http://www.rollernet.us/opensource/patches/fetchmail-rollernet-uidfixes2.patch

I haven't done extensive testing against it beyond the few cases where I 
had issues.

-- 
Seth Mattinen		se...@ro...
Roller Network LLC

Re: [fetchmail-devel] [Bug #15103] Fetchmail Fails with ASSP (http://assp.sourceforge.net/)

[Frederic M. <fre...@wo...>]

Matthias Andree a écrit :
> Frederic Marchal schrieb:
>
> [in response to BerliOS Bug #15103]
>
>   
>> I solved this issue by running ASSP in test mode (check option "Bayesian 
>> Test Mode") and prefixing the subject with something like [SPAM] (in 
>> option "Prepend Spam Subject"). It means ASSP won't stop mails seen as 
>> bayesian spam but will tag the subject. The mails can then be filtered 
>> out automatically by the final recipient.
>>     
>
> This, and my being bogofilter's co-maintainer (bogofilter also does
> Bayesian filtering), prompted me to have a glimpse at ASSP, and I must say
> that the promises it makes are way too bold. ASSP's self-advertising as the
> best tool aside, the assertion that Bayes filter were to intelligently
> decide is just crap. Bayes filters are making statistical decisions based
> on past training. "Maintenance free" doesn't work for them.
>   
ASSP's implementation of Paul Graham's algorithm is the most inefficient 
I know but it does more than bayesian statistics and that compensate for 
its incapacity at properly classify e-mails. In my case, I have a bunch 
of spamtraps addresses and ASSP learns from the e-mails it nets. Since 
spammers are sending tons of e-mails to every addresses they know, we 
can filter them out by simple comparison between the mails received by 
the spamtraps and the mails received by regular users. So, it does its 
own training even though a Bayes filter is not adequate considering how 
it performs in ASSP.

Beside, ASSP sits on the SMTP port. Therefore, it sees the mails sent by 
local users too and can whitelist the responses, preventing them from 
being filtered out and learning hams in the process. It can also learn 
from SMTP violations the spammers usually do and blacklist the offending 
senders very early and for every user of the server.

I don't believe ASSP is a great piece of software but it is the only one 
I know of that can do that and it does require very little or no action 
from the users it protect (especially no command line to run to learn 
more spam/ham).

>> By the way, I don't advise to let ASSP delete or reject blindly bayesian 
>> spams because, as you noticed, it doesn't take the whole mail into 
>> account and, in general, it does a poor job at filtering spams. It tends 
>> to have a high rate of false positives and you will loose good e-mails 
>> if you don't keep an eye on what it does.
>>     
>
> But then it seems to me that if you're using fetchmail or getmail, that
> integrating bogofilter, spamprobe or qsf in the mail system is more
> efficient than ASSP if it's prone to high resource use.
>
> At least bogofilter and SpamProbe are much faster than SpamAssassin's bayes
> mode. :-)
>   
Thanks for the advertisement :-) but my users are all windows 
(mal-)formatted and it is impossible to get them to run a linux command 
to classify their e-mails...

Frederic