fetchmail-devel — Development related discussion

Re: [fetchmail-devel] lost UIDs

[Brian C. <B.C...@po...>]

On Sun, Mar 06, 2005 at 03:15:36PM +0100, Manfred Weihs wrote:
> > Perhaps keeping all spam mail on the server isn't the best idea then?
> 
> But since I fetch the mails from two PCs (at home and at work), I have 
> to leave them on the server.

This was a problem for me, downloading to laptop on the go and desktop where
I kept everything archived. I finally solved it by downloading to whichever
I happened to be on, keeping all my messages in Maildir format (so each
message is in a separate file), and then using 'unison' to synchronise the
two home directories from time to time. Works beautifully, and also means my
laptop is fully backed up to my desktop PC, and vice versa.

Regards,

Brian.

Re: [fetchmail-devel] lost UIDs

[Matthias A. <mat...@gm...>]

Manfred Weihs schrieb am 2005-03-06:

> You might have a look at the attached patch. It removes the (completely 
> unnecessary) recursion in free_str_list. So it reduces stack usage and 

OK that far. However...

> avoids possible problems with very long lists. Furthermore I slightly 
> changed the interface and replaced the pointer to a pointer to the first 
> list element by just a pointer to the first list element. I think, this 

...now is not the time for interface changes, as the next fetchmail
release has been long overdue and intrusive changes should wait.

So I've left the interface unchanged and just removed the recursion. I
have ignored your change to the __UNUSED__ code which isn't compiled in.

> > Perhaps keeping all spam mail on the server isn't the best idea then?
> 
> But since I fetch the mails from two PCs (at home and at work), I have 
> to leave them on the server. Maybe I should delete mails from the server 
> more frequently. But I have to do this manually, because fetchmail does 
> not support to keep the mails only for X days or to keep only the latest 
> X mails (the later would be very easy to implement, but according to the 
> fetchmail FAQ it will not be implemented due to political reasons).

Yup, that was ESR's opinion. I don't share this view, but the UID code
needs to be cleaned up *massively* before any feature changes should be
allowed in. The UID code is a mess, and it's not used for IMAP.

> And an automatic cron job is no alternative, because, in this case I
> cannot be sure, that fetchmail got all the mails before the cron job
> deletes them. The deletion should be done _after_ fetchmail delivered
> the mails locally, ideally within the same pop3 session.

Right. If you can live with a slower software that has less features in
several areas, Charles Cazabon's getmail-4 has this feature.

-- 
Matthias Andree

Re: [fetchmail-devel] lost UIDs

[Manfred W. <we...@ic...>]

Matthias Andree wrote:
> Of course it's not, we'll be erasing everything.
> 
> Right code is (watch where the & is!)
> 
>     if (ctl->newsaved)
>     {
>         /* old state of mailbox may now be irrelevant */
>         struct idlist *temp = ctl->oldsaved;
>         if (outlevel >= O_DEBUG)
>             report(stdout, GT_("swapping UID lists\n"));
>         ctl->oldsaved = ctl->newsaved;
>         ctl->newsaved = (struct idlist *) NULL;
>         free_str_list(&temp);
>     }
> 
> Committed as SVN revision 4021.

OK, this is now definitely the correct version. Sorry for the small bug 
in my first version, and thanks for committing the two fixes to svn. I 
think this should solve my problems.

You might have a look at the attached patch. It removes the (completely 
unnecessary) recursion in free_str_list. So it reduces stack usage and 
avoids possible problems with very long lists. Furthermore I slightly 
changed the interface and replaced the pointer to a pointer to the first 
list element by just a pointer to the first list element. I think, this 
is cleaner; the only difference is that it is now in some cases 
necessary to set the pointer to NULL afterwards, but this is already 
done at many locations anyway.

 > Perhaps keeping all spam mail on the server isn't the best idea then?

But since I fetch the mails from two PCs (at home and at work), I have 
to leave them on the server. Maybe I should delete mails from the server 
more frequently. But I have to do this manually, because fetchmail does 
not support to keep the mails only for X days or to keep only the latest 
X mails (the later would be very easy to implement, but according to the 
fetchmail FAQ it will not be implemented due to political reasons). And 
an automatic cron job is no alternative, because, in this case I cannot 
be sure, that fetchmail got all the mails before the cron job deletes 
them. The deletion should be done _after_ fetchmail delivered the mails 
locally, ideally within the same pop3 session.

Manfred Weihs

Re: [fetchmail-devel] lost UIDs

[Matthias A. <mat...@gm...>]

Matthias Andree <mat...@gm...> writes:

> Actually, the right code is:
>
>     if (ctl->newsaved)
>     {
>         /* old state of mailbox may now be irrelevant */
>         struct idlist **temp = &ctl->oldsaved;
>         if (outlevel >= O_DEBUG)
>             report(stdout, GT_("swapping UID lists\n"));
>         ctl->oldsaved = ctl->newsaved;
>         ctl->newsaved = (struct idlist *) NULL;
>         free_str_list(temp);
>     }

Of course it's not, we'll be erasing everything.

Right code is (watch where the & is!)

    if (ctl->newsaved)
    {
        /* old state of mailbox may now be irrelevant */
        struct idlist *temp = ctl->oldsaved;
        if (outlevel >= O_DEBUG)
            report(stdout, GT_("swapping UID lists\n"));
        ctl->oldsaved = ctl->newsaved;
        ctl->newsaved = (struct idlist *) NULL;
        free_str_list(&temp);
    }

Committed as SVN revision 4021.

-- 
Matthias Andree

Re: [fetchmail-devel] lost UIDs

[Matthias A. <mat...@gm...>]

Manfred Weihs <we...@ic...> writes:

> I have a problem with fetchmail, which occurs every once in a while (and
> also happened with older versions of fetchmail). I use fetchmail in
> daemon mode (started from my KDE startup script when I log in) to poll
> two mail servers.

I'll just copy part of the message I've sent to fetchmail-friends, for
completeness's sake:
-------------------------------------------------------------------------
Manfred Weihs <we...@ic...> writes:

> Sometimes when I log in the .fetchids file is very short, it contains
> only very few uids. And therefore fetchmail loads down very many old
> mails. And since I get very much spam, this is very much, often several
> thousand mails.

Perhaps keeping all spam mail on the server isn't the best idea then?

> My first thought was, that probably fetchmail is killed
> in a bad moment during writing the .fetchids by system shutdown. This is
> one possibility, but I had a look into the code and found a bug, which
> also could cause this problem.
>
> In the function uid_swap_lists (in uid.c) there is this code:
>     if (ctl->newsaved)
>     {
> 	/* old state of mailbox may now be irrelevant */
> 	if (outlevel >= O_DEBUG)
> 	    report(stdout, GT_("swapping UID lists\n"));
> 	free_str_list(&ctl->oldsaved);
> 	ctl->oldsaved = ctl->newsaved;
> 	ctl->newsaved = (struct idlist *) NULL;
>     }
>
> So first the old oldsaved list is freed, and then it is overwritten with
> the newsaved list. If the TERM signal arrives during the free_str_list,
> an incomplete oldsaved list will be saved to .fetchids. And since
> free_str_list is implemented by a recursion, it really shortens the list
> beginning at the end (BTW I do not know, why this is implemented by a
> recursion rather then by a loop. If there are many messages on the
> server this could lead to a really deep recursion with the possibility
> of a stack overflow).

This is one of the many flaws in existing UID code. The problem is that
the current maintainer team including backup maintainers has evidently
zero time. A handover from ESR was started half a year ago, but
effectively starved in action; Rob Funk has essentially dropped from the
net as ESR did before, and Graham Wilson has officially quit but is
still operating the SVN server.

If you'd be willing to help, that would be much appreciated.

> IMHO the correct solution would be:
>     if (ctl->newsaved)
>     {
> 	struct idlist *temp;
> 	/* old state of mailbox may now be irrelevant */
> 	if (outlevel >= O_DEBUG)
> 	    report(stdout, GT_("swapping UID lists\n"));
> 	temp = ctrl->oldsaved;
> 	ctl->oldsaved = ctl->newsaved;
> 	ctl->newsaved = (struct idlist *) NULL;
> 	free_str_list(&ctl->oldsaved);
>     }

Actually, the right code is:

    if (ctl->newsaved)
    {
        /* old state of mailbox may now be irrelevant */
        struct idlist **temp = &ctl->oldsaved;
        if (outlevel >= O_DEBUG)
            report(stdout, GT_("swapping UID lists\n"));
        ctl->oldsaved = ctl->newsaved;
        ctl->newsaved = (struct idlist *) NULL;
        free_str_list(temp);
    }

> An additional good idea would be not to overwrite the old idfile, but 
> write to new idfile (e.g. ~/.fetchids-new), and after completion rename 
> the file (e.g. "rename("~/.fetchids-new", "~/.fetchids");" or similar). 
> Since there is a comment in the code "FIXME: do not overwrite the old 
> idfile" I assume that you already planned such change and I wonder, why 
> you did not implement it. I think, it is a very easy change, it is 
> probably just about 3 or 4 additional lines; if you want, I can provide 
> you a patch.

Well, we have code from Sunil Shetye, another "backup maintainer" who
disappeared without notice, to use one .fetchids file per server, and
that might (I'd need to check again) already implement that, so the
immediate fix may have been shelved for that reason. The patch was
queued for 6.2.7 as 6.2.6 was supposed to be bugfix-only.

I have made the fix of writing to a temp file first, as this has been
long-standing and is not too intrusive. (SVN revision 4019)

-- 
Matthias Andree

[fetchmail-devel] lost UIDs

[Manfred W. <we...@ic...>]

I have a problem with fetchmail, which occurs every once in a while (and
also happened with older versions of fetchmail). I use fetchmail in
daemon mode (started from my KDE startup script when I log in) to poll
two mail servers.
Sometimes when I log in the .fetchids file is very short, it contains
only very few uids. And therefore fetchmail loads down very many old
mails. And since I get very much spam, this is very much, often several
thousand mails. My first thought was, that probably fetchmail is killed
in a bad moment during writing the .fetchids by system shutdown. This is
one possibility, but I had a look into the code and found a bug, which
also could cause this problem.

In the function uid_swap_lists (in uid.c) there is this code:
     if (ctl->newsaved)
     {
	/* old state of mailbox may now be irrelevant */
	if (outlevel >= O_DEBUG)
	    report(stdout, GT_("swapping UID lists\n"));
	free_str_list(&ctl->oldsaved);
	ctl->oldsaved = ctl->newsaved;
	ctl->newsaved = (struct idlist *) NULL;
     }

So first the old oldsaved list is freed, and then it is overwritten with
the newsaved list. If the TERM signal arrives during the free_str_list,
an incomplete oldsaved list will be saved to .fetchids. And since
free_str_list is implemented by a recursion, it really shortens the list
beginning at the end (BTW I do not know, why this is implemented by a
recursion rather then by a loop. If there are many messages on the
server this could lead to a really deep recursion with the possibility
of a stack overflow).
IMHO the correct solution would be:
     if (ctl->newsaved)
     {
	struct idlist *temp;
	/* old state of mailbox may now be irrelevant */
	if (outlevel >= O_DEBUG)
	    report(stdout, GT_("swapping UID lists\n"));
	temp = ctrl->oldsaved;
	ctl->oldsaved = ctl->newsaved;
	ctl->newsaved = (struct idlist *) NULL;
	free_str_list(&ctl->oldsaved);
     }


An additional good idea would be not to overwrite the old idfile, but
write to new idfile (e.g. ~/.fetchids-new), and after completion rename
the file (e.g. "rename("~/.fetchids-new", "~/.fetchids");" or similar).
Since there is a comment in the code "FIXME: do not overwrite the old
idfile" I assume that you already planned such change and I wonder, why
you did not implement it. I think, it is a very easy change, it is
probably just about 3 or 4 additional lines; if you want, I can provide
you a patch.


Here is the information you require (according to the FAQ) in a bug report:
1. Linux 2.6.10 (but the problem occurred also with older versions)
2. fetchmail_6.2.5-12_i386.deb
3. I do not see a greeting when logging into the pop3 server (not
relevant for the problem anyway).
4. Exim 4.34
5. -d 600
5.
This is fetchmail release 6.2.5+NTLM+SDPS+SSL+NLS
Fallback MDA: (none)
Linux pc235 2.6.10 #1 Mon Dec 27 10:43:40 CET 2004 i686 GNU/Linux
Taking options from command line and /home/weihs/.fetchmailrc
Idfile is /home/weihs/.fetchids
Fetchmail will show progress dots even in logfiles.
Fetchmail will forward misaddressed multidrop messages to weihs.
Options for retrieving from XX...@XX....XX:
   True name of server is mail.ict.tuwien.ac.at.
   Protocol is POP3 (forcing UIDL use).
   All available authentication methods will be tried.
   SSL encrypted sessions enabled.
   Server nonresponse timeout is 300 seconds (default).
   Default mailbox selected.
   Only new messages will be retrieved (--all off).
   Fetched messages will be kept on the server (--keep on).
   Old messages will not be flushed before message retrieval (--flush off).
   Rewrite of server-local addresses is enabled (--norewrite off).
   Carriage-return stripping is disabled (stripcr off).
   Carriage-return forcing is disabled (forcecr off).
   Interpretation of Content-Transfer-Encoding is enabled (pass8bits off).
   MIME decoding is disabled (mimedecode off).
   Idle after poll is disabled (idle off).
   Nonempty Status lines will be kept (dropstatus off)
   Delivered-To lines will be kept (dropdelivered off)
   Fetch message size limit is 100 (--fetchsizelimit 100).
   Do binary search of UIDs during 9 out of 10 polls (--fastuidl 10).
   Messages will be SMTP-forwarded to: localhost (default)
   Single-drop mode: 1 local name(s) recognized.
   4098 UIDs saved.


Manfred Weihs

Re: [fetchmail-devel] translations for message catalogs, info and inquiry

[Matthias A. <mat...@gm...>]

Rob Funk <rf...@fu...> wrote on 2004-09-02:

> Sounds like we should start compiling a list of everyone to email when we do 
> the changeover announcement.

I have requested that the translation registry be changed, as can be
seen from the Cc: to this list that should appear in a few minutes.

>>   Do we want to have this mailing list listed as the "mailto" and
>>   the place where the .po files be sent?
>
> Probably should, unless anyone can think of a better place.

No better suggestions have been posted, so I've asked for this list's
address.

>>   We'll need to allow the bot's 
>>   address to the list, apparently it's either gnutra@ or translation@
>>   iro.umontreal.ca.
>
> Worst case we add it when Mailman holds it back.

Yup. Do you want to add me as moderator for fetchmail-devel?

> Thanks for handling the translation issues!

OK. I'll release a snapshot under the version of 6.2.5.991 to an
inofficial FTP site and have it translated so we can see to updating the
.po files.

We should probably get a 6.2.6 or 6.3.0 release out the door quick
(i. e. in March 2005) to provide the public with the fixes we have
merged, and so that fetchmail sees a new release within 18 months after
the previous one. If you do not have the time to work on fetchmail,
we'll have to find more developers and perhaps reassign maintainership
again to help, now that Graham has had to quit.

-- 
Matthias Andree

[fetchmail-devel] fetchmail domain migration

[Matthias A. <mat...@gm...>]

Greetings,

fetchmail is in the process of being handed over to new maintainers, and
so we'd like to have the fetchmail domain entry changed as shown below.

I'll be translating fetchmail for the "de" (German) team but given I
have SVN access, I can commit directly, so I mark <ext>de. Not sure if I
need a translator entry.

--- registry.sgml	2005-02-25 15:48:32.000000000 +0100
+++ registry.sgml.new	2005-03-01 20:58:53.000000000 +0100
@@ -234,11 +234,11 @@
    <note>This is a compendium like textdomain
 
   <domain>fetchmail
-   <mailto>es...@th...
-   <keep>6.0.0<keep>6.1.3<keep>6.2.0<keep>6.2.4
+   <mailto>fet...@li...
+   <keep>6.2.0<keep>6.2.4<keep>6.2.5
    <autosend>
    <url>http://www.catb.org/~esr/fetchmail/fetchmail-6.2.5.tar.gz
-   <ext>cs<ext>pt_BR
+   <ext>de
    <remark>was: http://www.tuxedo.org/~esr/fetchmail/fetchmail-6.2.2.tar.gz
    <remark>XXX: gl,es is also initial
    <remark>Keep more versions than usual because of strange version number scheme

-- 
Matthias Andree

[fetchmail-devel] Re: [fetchmail-svn] r4017 - trunk/po

[Matthias A. <ma...@dt...>]

sv...@de... writes:

> Author: m-a
> Date: 2004-12-27 14:00:41 -0600 (Mon, 27 Dec 2004)
> New Revision: 4017
>
> Removed:
>    trunk/po/.pot
> Modified:
>    trunk/po/ca.po
>    trunk/po/da.po

...
ARGH! Sorry. There were not meant to be committed, but I'll leave them
in for now.

-- 
Matthias Andree

[fetchmail-devel] Re: [fetchmail]Observations of a new user

[R H. B. <arg...@ya...>]

Matthias,

This is a follow-up post to my earlier thread on the Fetchmail-Friends list.

You replied to my original email:
> To: fet...@li...
> Subject: Re: [fetchmail]Observations of a new user
> From: Matthias Andree <ma...@dt...>
> Date: Mon, 08 Nov 2004 12:09:11 +0100
>
> R Hannes Beinert <arg...@ya...> writes:
>
>>  2) The documentation for FM is excellent, and sets a high bar for other
>>     OSS projects!  One small detail caused me to trip, initially, although
>>     (he says apologetically) this may have had more to do with the reader
>>     than the writer.  I found it might've helped me initially if there had
>>     been an explicit discussion of the "singledrop" versus "multidrop" mode
>>     of operation early in the man page.  Then, it might additionally have
>>     helped if each option had been explicitly labeled as having
>>     applicability for single- vs multidrop. [...]
>
> Well, the single- vs. multidrop issue is difficult; fetchmail implicitly
> derives this information from the "is ... here" stuff, and that is
> pretty unclear from the documentation.
>
> If you can suggest better wordings, perhaps even a patch, that would be
> most welcome - we have a fetchmail-devel list for such things, see
> http://developer.berlios.de/projects/fetchmail/

I have attached a patch on trunk/fetchmail.man (rev 3876).  Hopefully I
have managed to put it into a form which is useful to you.  I also hope
I have not made any technical errors, however I am by no means a
Fetchmail guru.

There are two basic changes I'm suggesting.  First, an explanation of
Fetchmail's modes of operation under the man-page DESCRIPTION heading.
Second, I've added some annotations to various options when the option
only applies to multidrop.  I've also added an additional column to the
keyword/option summary with this information, as well (I hope I was
able to find all the relevant instances).

>>  3) On the subject of logging - I find myself missing MTA-style logging, ie,
>>     timestamp, mailserver, MAIL FROM, RCPT TO, message size, and receiving
>>     MTA queueid.  This would make tracking email through the various logs
>>     much easier.  The standard FM log is too terse to include this
>>     information, yet while the "single-v" logging contains the information
>>     in question, it is probably more intended for debugging than logging of
>>     the sort I'm looking for.  Also, in the non-v logging, I miss having
>>     information on when a server with no messages was polled.  I do realize,
>>     though, that the non-v logging was intended to be very economical.  I've
>>     been using -v logging, and I'm happy.  Disks are cheap.  ;-)
>
> If you can offer a scheme what information exactly should be printed,
> that would help. MTA-style logging might be useful for fetchmail indeed.

I'm sure there are many details I've neglected which might be helpful,
but here is a sketch of what I have in mind.  All messages should be
prefixed by a time/date stamp which can be easily sorted.

At daemon start:

04/12/08@04:03:00[pid]: Starting fetchmail 6.2.3 daemon (/etc/fetchmailrc)
04/12/08@04:03:01[pid]: Postmaster=pos...@ho...m; Poll-Interval=1234 secs

When a poll starts.  Should only specify information for one of single
or multidrop modes.  Message line specifies the number of messages and
the size in bytes/octets:

04/12/08@04:03:10[pid]: Poll started; US...@ma... [1.2.3.4] (POP3); Interval=1
04/12/08@04:03:11[pid]: Multidrop; Fetchdomains DOM1, DOM2, DOM3, ...
04/12/08@04:03:12[pid]: Multidrop; Users <USER1>, <USER2>, <USER3>, ...
04/12/08@04:03:13[pid]: Singledrop; Recipient=<me...@ho...m>
04/12/08@04:03:14[pid]: Messages new 254 (1780000) / old 40 (9589) / total 294 (1789589)

If a poll specification includes an INTERVAL keyword which calls
for the server to be skipped on the current cycle:

04/12/08@04:03:10[pid]: Poll skipped; US...@ma... [1.2.3.4] (POP3); Interval=2

If a server poll is attempted, yet unsuccessful:

04/12/08@04:03:20[pid]: Poll started; US...@ma... [1.2.3.4] (POP3)
04/12/08@04:03:21[pid]: Server unreachable (specific error message)
04/12/08@04:03:22[pid]: Poll completed; US...@ma... [1.2.3.4] (POP3)

For each retrieved message.  "Resolved" refers to the envelope recipient which
has been derived by Fetchmail.  The parenthetical number is the sequence number
of the message in the mail account (or any other quasi-unique identifier).  The
Delivered/Rejected line comes from the reply code to the DATA & <CRLF>.<CRLF>
sequence (RFC2821 sec 4.2.5).  A positive completion status often contains the
queue-ID for the SMTP server, and could be included in the parentheses.  It is
my hope that this information is sufficient to track a particular message from
the retrieval server, through Fetchmail, and into the SMTP server:

04/12/08@04:03:30[pid]: (001) Message-Id=<612...@dr...>
04/12/08@04:03:31[pid]: (001) From=<ad...@dr...>; Size=38830
04/12/08@04:03:32[pid]: (001) To=<ad...@fo...>; Resolved=<admin@localhost>
04/12/08@04:03:33[pid]: (001) Delivered; SMTP.RELAYHOST.DOM:PORT (250 Ok: queued as 1B8D615F8F)
04/12/08@04:03:34[pid]: (001) Rejected; SMTP.RELAYHOST.DOM:PORT (540 Service unavailable)

If Fetchmail needs to break a server session due to a resource limit:

04/12/08@04:03:40[pid]: Poll fetchlimit reached (100); US...@ma... [1.2.3.4] (POP3)
04/12/08@04:03:41[pid]: Poll restarted; US...@ma... [1.2.3.4] (POP3)
04/12/08@04:03:42[pid]: Messages new 254 (1780000) / old 40 (9589) / total 294 (1789589)

Finally, when the daemon terminates a polling cycle:

04/12/08@04:03:50[pid]: Poll completed; US...@ma... [1.2.3.4] (POP3)
04/12/08@04:03:51[pid]: Sleeping until 04/12/08@05:03:40

>>  4) Closely releted to item #3, I also found it might be useful to be able
>>     to control logging from the fetchmailrc file, so that system start-up
>>     scripts wouldn't need to be touched.  So, eg, one could conceivably
>>     use global options such as "set verbose 2", or "set logging-level 1"
>>     in one's FMrc configuration to change the logging verbosity.
>
> Hm. That should be doable for the release after the next. The next one
> is for bugfixes, since we've had it pending for over a year now.
>
> Again, detailed suggestions on fetchmail-devel (see above) will be
> appreciated. Patches even more so. :)

I wish I could oblige with some patches to the code, but I don't have the
time to familiarize myself with the codebase at the moment.  In the meantime
I hope this email might be a little helpful.

Many grateful regards,

Hannes.

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [fetchmail-devel] some .po files not included in tarball

[Matthias A. <ma...@dt...>]

Graham Wilson <gr...@mk...> writes:

> Matthias, I noticed that some .po files aren't included in the generated
> tarball. Is that intentional, or a mistake?

I dropped them because they're way out of date; I considered them unfit
to ship if they had more than 50 quirks each. If the translation teams
can send in updated files in time after our request, we'll ship them,
otherwise, they won't. I don't like programs that are only halfway
translated.

> The missing files are:
>
>   po/cs.po
>   po/gl.po
>   po/pt_BR.po
>   po/sk.po

And additionally, in languages I don't know.

-- 
Matthias Andree

[fetchmail-devel] some .po files not included in tarball

[Graham W. <gr...@mk...>]

Matthias, I noticed that some .po files aren't included in the generated
tarball. Is that intentional, or a mistake?

The missing files are:

  po/cs.po
  po/gl.po
  po/pt_BR.po
  po/sk.po

-- 
gram

Re: [fetchmail-devel] running make dist changes po/*.po files

[Matthias A. <ma...@dt...>]

Graham Wilson <gr...@mk...> writes:

> Running make dist change po/*.po files. Should the changes be committed
> to the tree? If not, why are the files changed in the first place?

As far as I see it, the common cause for *.po files being a moving
target for version control is:

1. we change some code in a source file, possibly inserting or removing
   a line, at any rate changing line numbers of
   strings-to-be-translated.

2. fetchmail.pot is updated automatically by make and has new line
   numbers for the same old strings

3. make dist or make install run "make update-po", which then propagates
   the new line numbers.

This must be one of the reasons gettext still deserves a 0.X version number.

I haven't yet checked if we can disable line numbers and what that would
mean for gettext or helper tools such as Emacs' po-mode.

I have only committed .po changes if I'd changed the po files' content
at the same time, usually that means de.po and sometimes fr.po.

We'll do a pre-release tarball for the translation teams a week or two
before the actual release of 6.2.6 or 6.3.0, and we'll also ask them to
report pluaral form errors or shortcomings.
  gettext has proper support for proper plural form translations, but ESR
didn't use it anywhere. Perhaps his gettext version didn't support that
at its time.

-- 
Matthias Andree

[fetchmail-devel] running make dist changes po/*.po files

[Graham W. <gr...@mk...>]

Running make dist change po/*.po files. Should the changes be committed
to the tree? If not, why are the files changed in the first place?

-- 
gram

[fetchmail-devel] Re: r3995 - trunk

[Graham W. <gr...@mk...>]

On Thu, Nov 18, 2004 at 12:35:16PM +0100, Matthias Andree wrote:
> Brian Candler <B.C...@po...> writes:
> > What do other projects do with regard to snprintf or lack of it? How many
> > Unix systems lack snprintf these days?
> 
> I don't know, but I wouldn't want to break compatibility in a patchlevel
> or minot update. If we drop all the cruft,  we need to call it 7.0 - and
> we'd instead need to get out 6.2.6 or 6.3.0 soon.  We can change
> requirements an drop compatibility cruft for a later major version.

I think this is probably a good path to go down. Forget about sprintf
usage for now, and consider our alternatives after we have a few more
minor 6.2.x releases done.

At that point we could consider dropping support for systems without
snprintf, or figuring what other code to integrate.

-- 
gram

Re: [fetchmail-devel] Re: r3995 - trunk

[Matthias A. <ma...@dt...>]

Graham Wilson <gr...@mk...> writes:

> Matthias Andree:

>> Ripping code out of other software is harder to track WRT updates.
>
> I'm not really concerned as much about updates to an snprintf
> implementation;

I am.

> I wouldn't think there would need to be much updating.

More than 4/5 of the snprintf replacements I've looked at were broken in
one way or another.

> I'm more concerned with adding a whole other project into the fetchmail
> tarball.

We aren't doing that.

>> What is the claim mutt had a "good snprintf" implementation based on?
>
> Probably some unfounded claim that I read somewhere else I suppose. If
> you don't think the implementation is good, I'm fine with going with
> something else.

I think the mutt implementation is too incomplete to be useful for
fetchmail.

> What about the implementation at [1]? The downside is that it doesn't
> support a number of conversion characters (f, e, E, g, G, lc, ls, n),
> none of which we seem to use though. There are a few other small things
> missing which I don't think we use (or will) either.

(from different mail)
> [1] http://www.ijs.si/software/snprintf/

It doesn't support %n$ either, which is needed for internationalized
projects, so it, too, is out of the game. I haven't yet had the time to
look _closely_ at snprintfv.


Brian Candler <B.C...@po...> writes:

> What do other projects do with regard to snprintf or lack of it? How many
> Unix systems lack snprintf these days?

I don't know, but I wouldn't want to break compatibility in a patchlevel
or minot update. If we drop all the cruft,  we need to call it 7.0 - and
we'd instead need to get out 6.2.6 or 6.3.0 soon.  We can change
requirements an drop compatibility cruft for a later major version.

> According to the FreeBSD manpage,
>
>      the snprintf() and vsnprintf()
>      functions conform to ISO/IEC 9899:1999 (`ISO C99'').
>
> Would it be reasonable simply to expect a decent O/S to have these functions
> now? I don't know if SunOS 4.1 has snprintf or not,

It doesn't, but SunOS older than 5.7 (IIRC) isn't supported by the
vendor any more.

> If there is a free-standing external snprintf library, then we could just
> point people at it and tell them to install it first if it's needed.

The replacements I've looked at all assumed to be packaged as integral
part of another application.

-- 
Matthias Andree

Re: [fetchmail-devel] Re: r3995 - trunk

[Brian C. <B.C...@po...>]

On Thu, Nov 18, 2004 at 12:41:12AM -0600, Graham Wilson wrote:
> On Fri, Nov 12, 2004 at 11:54:51PM +0100, Matthias Andree wrote:
> > Graham Wilson <gr...@mk...> writes:
> > > Why do we want it as a separate project?
> > 
> > Ripping code out of other software is harder to track WRT updates.
> 
> I'm not really concerned as much about updates to an snprintf
> implementation; I wouldn't think there would need to be much updating.
> I'm more concerned with adding a whole other project into the fetchmail
> tarball.

What do other projects do with regard to snprintf or lack of it? How many
Unix systems lack snprintf these days?

According to the FreeBSD manpage,

     the snprintf() and vsnprintf()
     functions conform to ISO/IEC 9899:1999 (`ISO C99'').

Would it be reasonable simply to expect a decent O/S to have these functions
now? I don't know if SunOS 4.1 has snprintf or not, but if it doesn't and
you're still running it, you'll have worse problems than not being able to
compile fetchmail.

If there is a free-standing external snprintf library, then we could just
point people at it and tell them to install it first if it's needed.

Regards,

Brian.

[fetchmail-devel] Re: r3995 - trunk

[Graham W. <gr...@mk...>]

On Thu, Nov 18, 2004 at 12:41:12AM -0600, Graham Wilson wrote:
> What about the implementation at [1]? The downside is that it doesn't
> support a number of conversion characters (f, e, E, g, G, lc, ls, n),
> none of which we seem to use though. There are a few other small things
> missing which I don't think we use (or will) either.
> 
> On the other hand, the source doesn't seem to be the derived from
> Powell's implementation and the return semantics seem correct. The code
> is also distributed in tarball form, so it seems updates would be easy
> to track (though there hasn't been a release since late 2000 it seems).

[1] http://www.ijs.si/software/snprintf/

-- 
gram

[fetchmail-devel] Re: r3995 - trunk

[Graham W. <gr...@mk...>]

On Fri, Nov 12, 2004 at 11:54:51PM +0100, Matthias Andree wrote:
> Graham Wilson <gr...@mk...> writes:
> > Why do we want it as a separate project?
> 
> Ripping code out of other software is harder to track WRT updates.

I'm not really concerned as much about updates to an snprintf
implementation; I wouldn't think there would need to be much updating.
I'm more concerned with adding a whole other project into the fetchmail
tarball.

> > I would have gone with a single file implementation of snprintf instead
> > of using the whole Trio. I don't see why we need to add 54 files when we
> > could just add one. For example, mutt is known to have a good snprintf
> > implmentation.
> 
> What is the claim mutt had a "good snprintf" implementation based on?

Probably some unfounded claim that I read somewhere else I suppose. If
you don't think the implementation is good, I'm fine with going with
something else.

> It's one of the hordes of the Patrick Powell snprintf.c derivates, and
> as such, one that's both incomplete and pretty broken - it uses
> va_arg(args, short int) which is outright nonsense given the promotion
> rules, and it doesn't get return values right as documented.

Well, I thought that was the case. I'm fine not using though if you
don't think the code is good.

What about the implementation at [1]? The downside is that it doesn't
support a number of conversion characters (f, e, E, g, G, lc, ls, n),
none of which we seem to use though. There are a few other small things
missing which I don't think we use (or will) either.

On the other hand, the source doesn't seem to be the derived from
Powell's implementation and the return semantics seem correct. The code
is also distributed in tarball form, so it seems updates would be easy
to track (though there hasn't been a release since late 2000 it seems).

-- 
gram

[fetchmail-devel] Re: quick overview of branching and tagging in SVN

[Graham W. <gr...@mk...>]

On Fri, Nov 12, 2004 at 09:44:59PM +0100, Matthias Andree wrote:
> Graham Wilson <gr...@mk...> writes:
> >   $ cd /path/to/my/wc
> >   $ svn switch https://decoy.wox.org/svn/fetchmail/branches/debian
> 
> I presume that is ...fetchmail/trunk

Yes.

> > Tagging is done the same way as branching, except you copy to the tags
> > directory, not the branhes directory, and you never checkin anything on
> > a tag.
> 
> Are these "check in only on branches" enforced or is that just a
> convention? For this reminds me of what the svnbook recommended on the
> handling of vendor branches - and I see nothing but different names.

No, it is only by convention.

> > To see what would be merged before doing the merge operation:
> >
> >   $ svn diff https://decoy.wox.org/svn/fetchmail/trunk \
> >              https://decoy.wox.org/svn/fetchmail/branches/debian
> 
> Helluvalottotype :)

Since I access all of my projects on my server, I set the SVN
environment variable and do the following:

  $ echo $SVN
  https://decoy.wox.org/svn
  $ svn diff $SVN/fetchmail/trunk $SVN/fetchmail/branches/debian

-- 
gram

Re: [fetchmail-devel] Re: r3995 - trunk

[Matthias A. <ma...@dt...>]

Graham Wilson <gr...@mk...> writes:

> On Wed, Nov 10, 2004 at 09:14:37PM +0100, Matthias Andree wrote:
>> Graham Wilson <gr...@mk...> writes:
>> > What is this and why do we need it?
>> 
>> This is the best GPL-compatible snprintf/sscanf replacement (including
>> varargs variants) that I've been able to find as a separate project.
>
> Why do we want it as a separate project?

Ripping code out of other software is harder to track WRT updates.

>> I'm not fixed on Trio, if you know something better, we can flush Trio
>> and use something else,
>
> I would have gone with a single file implementation of snprintf instead
> of using the whole Trio. I don't see why we need to add 54 files when we
> could just add one. For example, mutt is known to have a good snprintf
> implmentation.
>
> What would you think about using that?

What is the claim mutt had a "good snprintf" implementation based on?

It's one of the hordes of the Patrick Powell snprintf.c derivates, and
as such, one that's both incomplete and pretty broken - it uses
va_arg(args, short int) which is outright nonsense given the promotion
rules, and it doesn't get return values right as documented.

Note that the tarball only includes a small subset of the Trio files,
namely these (try "make distdir" to figure):

total 328K
 20K CHANGES       160K trio.c      24K trionan.c   44K triostr.c
4.0K README        8.0K trio.h     4.0K trionan.h   12K triostr.h
 36K regression.c  8.0K triodef.h  8.0K triop.h


I'll have a look at Gary V. Vaughan's snprintfv but the tarball also is
463k in size so I'm not sure if it'll be much lighter than Trio.

http://savannah.nongnu.org/projects/libsnprintfv
ftp://alpha.gnu.org/gnu/snprintfv

-- 
Matthias Andree

[fetchmail-devel] Re: quick overview of branching and tagging in SVN

[Graham W. <gr...@mk...>]

On Fri, Nov 12, 2004 at 09:46:21PM +0100, Matthias Andree wrote:
> Brian Candler <B.C...@po...> writes:
> 
> > I mean you tag a particular version of a file ready for a release, and then
> > at the last minute change your mind, squeeze in a change, and move the
> > release tag to the modified file.
> >
> > FreeBSD do it occasionally just before they make a release :-)
> 
> Haven't tried it, but I'd probably try killing (with svn remove) the tag
> directory, and then tag new (with svn tags).

That is what I usually do. I don't know if there is a better way to do
that or not though.

-- 
gram

Re: [fetchmail-devel] Re: r4000 - trunk

[Matthias A. <ma...@dt...>]

Graham Wilson <gr...@mk...> writes:

>>      if (getuid() == ROOT_UID) {
>>  	lockfile = (char *)xmalloc(
>>  		sizeof(PID_DIR) + sizeof(FETCHMAIL_PIDFILE) + 1);
>> -	sprintf(lockfile, "%s/%s", PID_DIR, FETCHMAIL_PIDFILE);
>> +	strcpy(lockfile, PID_DIR);
>> +	strcat(lockfile, "/");
>> +	strcat(lockfile, FETCHMAIL_PIDFILE);
>>      } else {
>>  	lockfile = (char *)xmalloc(strlen(fmhome)+sizeof(FETCHMAIL_PIDFILE)+2);
>>  	strcpy(lockfile, fmhome);
>
> Why did you switch to strcat (which doesn't check bounds) instead of
> snprintf? I would think we should just change that sprintf call to
> snprintf.

Consistency in these particular if...else branches.

The memory has been allocated anew so we know everything fits.

>> Modified: trunk/unmime.c
>> ===================================================================
>> --- trunk/unmime.c	2004-11-10 20:14:18 UTC (rev 3999)
>> +++ trunk/unmime.c	2004-11-10 20:38:06 UTC (rev 4000)
>> @@ -669,9 +669,9 @@
>>    char fnam[100];
>>  
>>    pid = getpid();
>> -  sprintf(fnam, "/tmp/i_unmime.%x", pid);
>> +  sprintf(fnam, "/tmp/i_unmime.%lx", (long)pid);
>>    fd_orig = fopen(fnam, "w");
>> -  sprintf(fnam, "/tmp/o_unmime.%x", pid);
>> +  sprintf(fnam, "/tmp/o_unmime.%lx", (long)pid);
>>    fd_conv = fopen(fnam, "w");
>>  #endif
>
> These should be changed to snprintf as well I assume?

The buffer holds 100 bytes, we'll stuff at most 31 on 64bit machines or
47 on 128bit machines - and it's only used when the macro STANDALONE is
defined, but not for the module compiled into fetchmail.

We can change the stuff just for completeness though.

-- 
Matthias Andree

Re: [fetchmail-devel] Re: quick overview of branching and tagging in SVN

[Matthias A. <ma...@dt...>]

Brian Candler <B.C...@po...> writes:

> I mean you tag a particular version of a file ready for a release, and then
> at the last minute change your mind, squeeze in a change, and move the
> release tag to the modified file.
>
> FreeBSD do it occasionally just before they make a release :-)

Haven't tried it, but I'd probably try killing (with svn remove) the tag
directory, and then tag new (with svn tags).

-- 
Matthias Andree

Re: [fetchmail-devel] quick overview of branching and tagging in SVN

[Matthias A. <ma...@dt...>]

Graham Wilson <gr...@mk...> writes:

> You can then switch your working copy of fetchmail to the Debian branch
> by doing the following:
>
>   $ cd /path/to/my/wc
>   $ svn switch https://decoy.wox.org/svn/fetchmail/branches/debian
>
> Switching back to the trunk is just as easy:
>
>   $ cd /path/to/my/wc
>   $ svn switch https://decoy.wox.org/svn/fetchmail/branches/debian

I presume that is ...fetchmail/trunk

> Tagging is done the same way as branching, except you copy to the tags
> directory, not the branhes directory, and you never checkin anything on
> a tag.

Are these "check in only on branches" enforced or is that just a
convention? For this reminds me of what the svnbook recommended on the
handling of vendor branches - and I see nothing but different names.

> To see what would be merged before doing the merge operation:
>
>   $ svn diff https://decoy.wox.org/svn/fetchmail/trunk \
>              https://decoy.wox.org/svn/fetchmail/branches/debian

Helluvalottotype :)

> I'd be happy to answer any questions anyone has (assuming I can, of
> course :).

Thank you.

-- 
Matthias Andree