fetchmail-users — User questions and discussion

Re: [Fetchmail-users] Fetchmail without passwords in plaintext file?

[Matthias A. <mat...@gm...>]

Am 11.12.18 um 13:31 schrieb Bjoern Voigt:
> Matthias Andree wrote:
>> There is no encryption option. You could use a .netrc file to save
>> passwords instead, but that would not be encrypted either.
>>
>> If you are running fetchmail from a shell (not from cron), you can omit
>> the passwords from .fetchmailrc and .netrc files, and fetchmail will ask
>> you for all of them. This should also work with daemon mode, so you
>> enter them once and fetchmail polls every N seconds (whatever --daemon
>> delay you give it).
> I think storing passwords unencrypted in files is an absolute no-go
> today. Business users and administrators may come into conflict with
> their company security policies. Especially users with unencrypted disks
> on laptops are are at high risk.
>
> The only work-around which I can recommend is a variation of Matthias'
> tip: Use a password manager and copy-paste passwords into fetchmail at
> start-up.
>
> But Matthias, why you refused to accept the following poll request?
> https://gitlab.com/fetchmail/fetchmail/merge_requests/1
>
> PWMD is probably not perfect (I haven't tried it), but it would be the
> first official solution for the unencrypted password storing problem, if
> Fetchmail would contain the libpwmd patches.
>
[The messages appears to not have made it to the list yet, but since
Björn has written a Cc: to the list, I'll provide a full quote.]


Hi Björn,

Regarding plaintext password storage, you don't have to, but fetchmail
had originally been written for end-user consumption and not high-grade
datacenter use.

If I need to manually (through Git downloads) integrate a merge request
(such as !1 or !11 or others), possibly rebasing it to linearize
history, I cannot properly have Gitlab reflect that, you'll need to read
the comments to see that I did in fact merge them - but not into the
legacy_64 branch, (6.4 BETA) but instead into the master branch, where I
take the 7.0 ALPHA snapshots from. So, that point rejected, but no
offense taken.

Feel free to check if there are relevant Gitlab bug/feature request
tickets and otherwise write one to permit me to mark things as merged. I
believe Gitorous, or some other Git hosting site, permitted that.

HTH
Matthias

Re: [Fetchmail-users] Fetchmail without passwords in plaintext file?

[Gene H. <ghe...@sh...>]

On Tuesday 11 December 2018 07:31:57 Bjoern Voigt wrote:

> Matthias Andree wrote:
> > There is no encryption option. You could use a .netrc file to save
> > passwords instead, but that would not be encrypted either.
> >
> > If you are running fetchmail from a shell (not from cron), you can
> > omit the passwords from .fetchmailrc and .netrc files, and fetchmail
> > will ask you for all of them. This should also work with daemon
> > mode, so you enter them once and fetchmail polls every N seconds
> > (whatever --daemon delay you give it).
>
> I think storing passwords unencrypted in files is an absolute no-go
> today. Business users and administrators may come into conflict with
> their company security policies. Especially users with unencrypted
> disks on laptops are are at high risk.
>
> The only work-around which I can recommend is a variation of Matthias'
> tip: Use a password manager and copy-paste passwords into fetchmail at
> start-up.
>
> But Matthias, why you refused to accept the following poll request?
> https://gitlab.com/fetchmail/fetchmail/merge_requests/1
>
> PWMD is probably not perfect (I haven't tried it), but it would be the
> first official solution for the unencrypted password storing problem,
> if Fetchmail would contain the libpwmd patches.
>
> Greetings,
> Björn
>
While I realize there is a need for encrypted passwd's in the more densly 
populated and/or industrial areas where business critical information 
might be sought by the shadier types, that need has not reached out here 
in the puckerbrush. Yet... I am not aware of any pop3 server in this 
moderately rural area, ever asking us to use encrypted passwds. Not 
saying there aren't such isp's but I've not encountered any yet.  What 
I'm saying is that forcing it on us, would force us to find another 
fetchmail like agent. In fact, they've not even forced us to longer 
passwds yet. I'd druther they did, to at least 20 alpha/numeric chars 
but too many winders users think 12345 or 54321Boom is good enough.
And they are the majority by far. I'm an island of all linux stuff here.
>
> _______________________________________________
> Fetchmail-users mailing list
> Fet...@li...
> https://lists.sourceforge.net/lists/listinfo/fetchmail-users



-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: [Fetchmail-users] Fetchmail without passwords in plaintext file?

[Bjoern V. <bj...@ar...>]

Matthias Andree wrote:
> There is no encryption option. You could use a .netrc file to save
> passwords instead, but that would not be encrypted either.
>
> If you are running fetchmail from a shell (not from cron), you can omit
> the passwords from .fetchmailrc and .netrc files, and fetchmail will ask
> you for all of them. This should also work with daemon mode, so you
> enter them once and fetchmail polls every N seconds (whatever --daemon
> delay you give it).

I think storing passwords unencrypted in files is an absolute no-go
today. Business users and administrators may come into conflict with
their company security policies. Especially users with unencrypted disks
on laptops are are at high risk.

The only work-around which I can recommend is a variation of Matthias'
tip: Use a password manager and copy-paste passwords into fetchmail at
start-up.

But Matthias, why you refused to accept the following poll request?
https://gitlab.com/fetchmail/fetchmail/merge_requests/1

PWMD is probably not perfect (I haven't tried it), but it would be the
first official solution for the unencrypted password storing problem, if
Fetchmail would contain the libpwmd patches.

Greetings,
Björn

Re: [Fetchmail-users] Duplicate emails running fetchmail

[Lucio C. <lu...@la...>]

On Wed, 28 Nov 2018, Susan Ruiz wrote:

> Hello, I have a problem, I am currently using fetchmail to retrieve my
> emails from two accounts, one gmail.com, another corporate one with
> POP3.

> and I recover but this duplicate the messages, does anyone know if
> there is any option for this not to happen?

I do not understand whether the duplication occurs only for the gmail 
inbox or for both, and therefore it is due to the known hydiosinchratic 
imap implementation of gmail, or simply to the fact the message is not 
deleted when retrieved the first time, so it is fetched again.

I invoke fetchmail via crontab every 5 min (not manually nor daemon)

The details of what I did are described in
http://sax.iasf-milano.inaf.it/~lucio/WWW/WhereManWins/gs.html

My MUA is alpine not mutt, but you should be able to mimic my behaviour if 
you like it.

- in general I am content to wait max 5 min until fetchmail drops the
   message in my local inbox

- I am able to access (or see) gsuite inbox as an incoming folder in
   my MUA. I never read mail from there, I just use to see the count of
   pending messages waiting to be fetched

- I am able to access all the other gsuite folders (without inbox) as
   a separate folder collection. I use only two folders in there

   - Spam. In case of false positives (good messages classified as
     spam) I use google's webmail i/f to flag them "not spam". This
     moves them temporarily back to gsuite inbox where fetchmail
     retrieves them.

     In case of false negatives (spam received as ham) I can use
     my MUA to move it from local inbox to google Spam

   - Bin. With my configuration, messages retrieved by fetchmail
     and "deleted" end there. Daily I clean it from my MUA with
     a single keystroke

- so I never keep anything but spam on google for more than 1 day

- the peculiarity of google imap is that their "folders" are not
   REAL folders but "labels" and some of them do not reply to
   delete/expunge commands issued from my MUA

But I found a reasonable arrangement,

-- 
Lucio Chiappetti - INAF/IASF - via Bassini 15 - I-20133 Milano (Italy)
For more info : http://www.iasf-milano.inaf.it/~lucio/personal.html
------------------------------------------------------------------------
"All that is google does not glitter
  Nor all who use alpine/procmail are lost"

Re: [Fetchmail-users] fetchmail can't delete a fetched email

[Gene H. <ghe...@sh...>]

On Thursday 29 November 2018 13:04:03 Matthias Andree wrote:

> Am 29.11.18 um 16:08 schrieb Gene Heskett:
> > Greetins all;
> >
> > My isp seems to handle a request into its mailserver, which I
> > believe is dovecot, in pop3 mode, by disabling fetchmails
> > housekeeping cleanup of successfully fetched emails, so in a days
> > time I have seen 450+ emails pile up, the I have to log into that
> > same server with a browser and the browser can then delete these
> > msgs 100 at a time.
> >
> > Is there a keyword stronger than "no-keep" that can override its
> > inability to expunge read messages by using the same expunge method
> > a browser uses?
>
> Well, we need to establish what's causing it. POP3 does not require
> some kind of expiration or expunging, it's just that if you delete
> messages and then close the session properly, the messages are
> supposed to be gone. I know some servers ignore such requests, for
> instance, Google.
>
> There are options in fetchmail that can limit how many messages it
> fetches (and hence, without keep, deletes) in one run, but the best
> thing is to look at the logs... the usual recommendations are at
> <http://www.fetchmail.info/fetchmail-FAQ.html#G3>.
>
These "missfires" do not show in the fetchmail logs. The server just 
silently ignores them, but a web browser can clean house just fine.
I'm just wondering if fetchmail has some sort of a method to spoof itself 
as a browser for the delete command.

Thanks Matthias Andree.

-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: [Fetchmail-users] fetchmail can't delete a fetched email

[Matthias A. <mat...@gm...>]

Am 29.11.18 um 16:08 schrieb Gene Heskett:
> Greetins all;
>
> My isp seems to handle a request into its mailserver, which I believe is 
> dovecot, in pop3 mode, by disabling fetchmails housekeeping cleanup of 
> successfully fetched emails, so in a days time I have seen 450+ emails 
> pile up, the I have to log into that same server with a browser and the 
> browser can then delete these msgs 100 at a time.
>
> Is there a keyword stronger than "no-keep" that can override its 
> inability to expunge read messages by using the same expunge method a 
> browser uses?


Well, we need to establish what's causing it. POP3 does not require some
kind of expiration or expunging, it's just that if you delete messages
and then close the session properly, the messages are supposed to be
gone. I know some servers ignore such requests, for instance, Google.

There are options in fetchmail that can limit how many messages it
fetches (and hence, without keep, deletes) in one run, but the best
thing is to look at the logs... the usual recommendations are at
<http://www.fetchmail.info/fetchmail-FAQ.html#G3>.

[Fetchmail-users] fetchmail can't delete a fetched email

[Gene H. <ghe...@sh...>]

Greetins all;

My isp seems to handle a request into its mailserver, which I believe is 
dovecot, in pop3 mode, by disabling fetchmails housekeeping cleanup of 
successfully fetched emails, so in a days time I have seen 450+ emails 
pile up, the I have to log into that same server with a browser and the 
browser can then delete these msgs 100 at a time.

Is there a keyword stronger than "no-keep" that can override its 
inability to expunge read messages by using the same expunge method a 
browser uses?

Thanks all.

-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: [Fetchmail-users] Duplicate emails running fetchmail

[Carlos E. R. <rob...@te...>]

On 28/11/2018 20.56, Susan Ruiz wrote:
> Hello, I have a problem, I am currently using fetchmail to retrieve my
> emails from two accounts, one gmail.com, another corporate one with
> POP3.
> 
> This is my fetchmailrc file:
> poll "pop.gmail.com" with proto POP3 port 995 user "xx...@gm..."
> password "xxxxxxx"
> sslfingerprint "xxxxxxxxxxx"
> no rewrite keep ssl fetchall mda "/usr/bin/maildrop"


I use this recipe with gmail and it works perfectly and automatically:

poll imap.gmail.com with interval 0 proto imap timeout 50, and tracepolls
        user R_...@gm..., with password "PASSWD", is L_USER here, expunge 20, and ssl, and fetchall

And for other, non gmail, accounts:

poll SERVER with proto imap, timeout 70, and tracepolls
        user R_USER, with password PASSWD, is L_USER here, and fetchall


I don't touch pop3 if I can avoid it.

In my case, mail is sent for further processing by the local postfix server.


-- 
Cheers / Saludos,

		Carlos E. R.
		(from 42.3 x86_64 "Malachite" at Telcontar)

[Fetchmail-users] Duplicate emails running fetchmail

[Susan R. <sus...@gm...>]

Hello, I have a problem, I am currently using fetchmail to retrieve my
emails from two accounts, one gmail.com, another corporate one with
POP3.

This is my fetchmailrc file:
poll "pop.gmail.com" with proto POP3 port 995 user "xx...@gm..."
password "xxxxxxx"
sslfingerprint "xxxxxxxxxxx"
no rewrite keep ssl fetchall mda "/usr/bin/maildrop"

poll "mail.xxxxxxxx.co"
proto POP3
port 995
user "sus...@xx..."
password "xxxxxxx"
sslfingerprint "xxxxxxxxxxxxxxx"
no rewrite
keep
ssl
fetchall
mda "/usr/bin/maildrop"

In the first execution I use it in the following way, and I invoke it
inside mutt with the macro "G"
macro index G "!fetchmail\n"
macro pager G "!fetchmail\n"

My problem comes when I enter mutt again and invoke G to activate the
daemon (for the second execution I make these changes in my file:)

set daemon 60
poll "pop.gmail.com" with proto POP3 port 995 user "xx...@gm..."
password "xxxxxxx"
sslfingerprint "xxxxxxxxxxx"
no rewrite keep ssl nofetchall mda "/usr/bin/maildrop"

set daemon 60
poll "mail.xxxxxxxx.co"
proto POP3
port 995
user "sus...@xx..."
password "xxxxxxx"
sslfingerprint "xxxxxxxxxxxxxxx"
no rewrite
keep
ssl
nofetchall
mda "/usr/bin/maildrop"

and I recover but this duplicate the messages, does anyone know if
there is any option for this not to happen? I change previously to
notfechall but even that way it brings me duplicate emails.

Thank you in advance for your help.

Regards,


-- 
Susan Ruiz

Re: [Fetchmail-users] bug in pop3.c of 6.4.0.beta4?

[Dr P. K. <p.k...@im...>]

> I am wondering how find_uid_by_num() could return 0 in this situation,

I don't know.

I know that fetchmail-6.3.26 doesn't have the same problem
with any emails that cause the issue in fetchmail-6.4.0.beta4; since
reverting to that downloaded such emails fine. 

fetchmail-6.4.0.beta4 seemed to have a problem only on the last remaining
email on the server, although the sample size was small (less than ten).

For one email account, switching the download from POP3 to IMAP solved
the problem (in beta4), so presumably it's a POP-only problem (or at least
a POP-only symptom).

I saw no problems after adding in my change, although it is possible
that there are silent problems that exist. The troublesome emails, 
rather than being left on the server because of the fetchmail crash, 
got downloaded ok, and deleted off the server, so everything seemed fine.

I do have an email that I can bounce to myself that triggers the problem,
but it's not one for public consumption. I could use it to test new 
versions though.



On Sun, Nov 25, 2018 at 01:04:11AM +0100, Matthias Andree wrote:
> Am 15.11.18 um 15:46 schrieb Dr P. Kinsler:
> > In pop3.c lines 1373-1374 (routine pop3_delete()) from fetchmail-6.4.0.beta4
> > we have the lines:
> >
> >     rec = find_uid_by_num(dofastuidl ? &ctl->oldsaved : &ctl->newsaved, number);
> >     rec->status = UID_DELETED;
> >
> > If find_uid_by_num() returns a 0 (NULL), the "rec->status = UID_DELETED"
> > line then can trigger a segmentation fault.
> 
> Paul,
> 
> thanks for reporting and debugging this.
> 
> I am wondering how find_uid_by_num() could return 0 in this situation,
> so I wonder if there is a flaw in the program logic in driver.c (in the
> next two outer frame), but I think that your proposal to safeguard the
> assignment is right to avoid the immediate crash. If the message isn't
> known, we shouldn't be trying to save state for it.
> 
> I have committed similar code to what you've proposed, and uploaded to
> the Git repositories as 8c57ec38. The "similar" is my adding another
> pair of parentheses, which suppresses warnings in some
> compiler/version/flag combinations. The fix should be in beta5 or rc1 or
> whatever I'll call the next tarball.
> 
> I assume you have already tested such a change an know it to fix your
> crashes?
> 
> Thanks again.
> 
> Best regards,
> Matthias
> 
> 
> 
> 
> _______________________________________________
> Fetchmail-users mailing list
> Fet...@li...
> https://lists.sourceforge.net/lists/listinfo/fetchmail-users

-- 
---------------------------------+---------------------------------
Dr. Paul Kinsler                        Dr....@ph...

Re: [Fetchmail-users] Fetchmail without passwords in plaintext file?

[Matthias A. <mat...@gm...>]

Am 18.11.18 um 19:08 schrieb Susan Ruiz:
> Hi, to retrieve my emails using POP3, both from gmail and from my
> company email account, I am working with fetchmail version 6.3.26, I
> would like to know if there is any way to avoid leaving my password in
> the .fetchmailrc file.
>
> How to encrypt it? Thanks in advance

Susan,

There is no encryption option. You could use a .netrc file to save
passwords instead, but that would not be encrypted either.

If you are running fetchmail from a shell (not from cron), you can omit
the passwords from .fetchmailrc and .netrc files, and fetchmail will ask
you for all of them. This should also work with daemon mode, so you
enter them once and fetchmail polls every N seconds (whatever --daemon
delay you give it).

Hope that helps.

Matthias

Re: [Fetchmail-users] bug in pop3.c of 6.4.0.beta4?

[Matthias A. <mat...@gm...>]

Am 15.11.18 um 15:46 schrieb Dr P. Kinsler:
> In pop3.c lines 1373-1374 (routine pop3_delete()) from fetchmail-6.4.0.beta4
> we have the lines:
>
>     rec = find_uid_by_num(dofastuidl ? &ctl->oldsaved : &ctl->newsaved, number);
>     rec->status = UID_DELETED;
>
> If find_uid_by_num() returns a 0 (NULL), the "rec->status = UID_DELETED"
> line then can trigger a segmentation fault.

Paul,

thanks for reporting and debugging this.

I am wondering how find_uid_by_num() could return 0 in this situation,
so I wonder if there is a flaw in the program logic in driver.c (in the
next two outer frame), but I think that your proposal to safeguard the
assignment is right to avoid the immediate crash. If the message isn't
known, we shouldn't be trying to save state for it.

I have committed similar code to what you've proposed, and uploaded to
the Git repositories as 8c57ec38. The "similar" is my adding another
pair of parentheses, which suppresses warnings in some
compiler/version/flag combinations. The fix should be in beta5 or rc1 or
whatever I'll call the next tarball.

I assume you have already tested such a change an know it to fix your
crashes?

Thanks again.

Best regards,
Matthias

Re: [Fetchmail-users] Fetchmail without passwords in plaintext file?

[grarpamp <gra...@gm...>]

On 11/18/18, Susan Ruiz <sus...@gm...> wrote:
> Hi, to retrieve my emails using POP3, both from gmail and from my
> company email account, I am working with fetchmail version 6.3.26, I
> would like to know if there is any way to avoid leaving my password in
> the .fetchmailrc file.
>
> How to encrypt it? Thanks in advance

Fetchmail does not yet support any sort of file
encryption passphrase input during invocation.

If it did, that could cover all sorts of key material,
server, and account disclosure, etc that is present
in the rc, statefiles, and even possibly the
maildir / mbox itself for later export.

There's probably some password options
in the manpage for use on command line.
Though some programs do not hide that
from the ps or environment.

The .fetchmail rc should be chmod 0600.

And you should be using full disk encryption.
And or any relavant filesystem or file based
encryption if FDE is insufficient defense, or
unavailable. Some VM schemes may offer
encrypted containers.

[Fetchmail-users] Fetchmail without passwords in plaintext file?

[Susan R. <sus...@gm...>]

Hi, to retrieve my emails using POP3, both from gmail and from my
company email account, I am working with fetchmail version 6.3.26, I
would like to know if there is any way to avoid leaving my password in
the .fetchmailrc file.

How to encrypt it? Thanks in advance

Regards.

-- 
Susan Ruiz

[Fetchmail-users] bug in pop3.c of 6.4.0.beta4?

[Dr P. K. <p.k...@im...>]

In pop3.c lines 1373-1374 (routine pop3_delete()) from fetchmail-6.4.0.beta4
we have the lines:

    rec = find_uid_by_num(dofastuidl ? &ctl->oldsaved : &ctl->newsaved, number);
    rec->status = UID_DELETED;

If find_uid_by_num() returns a 0 (NULL), the "rec->status = UID_DELETED"
line then can trigger a segmentation fault.

Other calls to find_uid_by_num() in pop3.c (i.e. at lines 1207,
1352, 1358) use an if statement that avoids subsequently assigning 
something to rec->status when rec is NULL.

Presumably this is a bug, and lines 1373-4 should instead be 
something like those other invocations, e.g.

if (rec = find_uid_by_num(dofastuidl ? &ctl->oldsaved : &ctl->newsaved, number)){
  rec->status = UID_DELETED;
}

However, I don't fully understand how this is intended to
work, so my suggestion may well be lacking (it *does* stop 
the segmentation faults, and everything then seemed fine, 
but for all I know may [sometimes] get something else wrong).


NB: I installed fetchmail-6.4.0.beta4 on my Slackware64-14.2 system,
got fetchmail crashes on some emails, after connecting to two different
POP servers. Gdb told me where the crash was, and then adding a write 
statement and recompiling told me that line 1373 indeed set rec=0 for 
those emails. Here is the gdb backtrace:

Program received signal SIGSEGV, Segmentation fault.
pop3_delete (sock=<optimized out>, ctl=<optimized out>, number=1) at pop3.c:1374
1374        rec->status = UID_DELETED;
(gdb) bt
#0  pop3_delete (sock=<optimized out>, ctl=<optimized out>, number=1) at pop3.c:1374
#1  0x000000000040f8ba in fetch_messages (msgsizes=0x640b60 <msgsizes>, transient_errors=<synthetic pointer>, 
    deletions=<synthetic pointer>, dispatches=<synthetic pointer>, fetches=<synthetic pointer>, maxfetch=0, 
    count=<optimized out>, ctl=0x654670, mailserver_socket=3) at driver.c:812
#2  do_session (ctl=0x654670, proto=proto@entry=0x436820 <pop3>, maxfetch=0) at driver.c:1435
#3  0x0000000000410fa2 in do_protocol (ctl=<optimized out>, proto=proto@entry=0x436820 <pop3>) at driver.c:1660
#4  0x00000000004216ea in doPOP3 (ctl=ctl@entry=0x654670) at pop3.c:1449
#5  0x000000000040b380 in query_host (ctl=ctl@entry=0x654670) at fetchmail.c:1546
#6  0x0000000000406cab in main (argc=<optimized out>, argv=0x7fffffffe2f8) at fetchmail.c:793
(gdb) 


-- 
---------------------------------+---------------------------------
Dr. Paul Kinsler                        Dr....@ph...

Re: [Fetchmail-users] some question and thoughts about imap and uids

[Matthias A. <mat...@gm...>]

NOTE: Followup-To set to a public mailing list (which requires
subscription for postings) - be sure to double-check where your mailer
is going to send your reply!


Am 09.10.18 um 18:44 schrieb Kamil Jońca:
> I have read imap.c code and (it is possible I have overlooked something)

Hello Kamil,

Thanks for looking into this. I think this should continue on
fetchmail-devel@, I am redirecting there and please do subscribe so you
can post.

The model that fetchmail followed so far is that it relied on the \Seen
flags to track which messages had been downloaded, and without --keep,
it would mark messages as deleted.

> 1. (imap_search)
>    sometimes fetchmail issues "SEARCH UNSEEN" and sometimes "SEARCH
>    UNSEEN UNDELETED" (it depends on "keep" flag )
>    What is use case to issue "SEARCH UNSEEN" (without undeleted?)

ad 1. What the original intention was, I don't know. Part of it was
compatibility with IMAP2 (see the revision log for dca4a906), and part
may be the assumption that without --keep, we mark messages for deletion
but possibly without an immediate EXPUNGE, so disconnects or crashes
could theoretically leave deleted messages somehow. I am Cc:'ing Sunil
Shetye who contributed this code in early 2010 if he remembers. I hope
his mail address works, haven't heard in a long time.

> commit dca4a906d60a197b09159bc8d8a16625b1790215
> Committer: Matthias Andree <mat...@gm...>
> Date:   Thu Feb 4 09:51:01 2010 +0000
>
>     IMAP SEARCH fixes & FETCH fallback by Sunil Shetye
>     
>     * The IMAP client now uses "SEARCH UNSEEN" rather than "SEARCH
> UNSEEN NOT
>       DELETED" again on IMAP2, to fix a regression in fetchmail 6.2.5
> reported by
>       Will Stringer in June 2004. (Sunil Shetye)
>     * The IMAP client now uses "SEARCH UNSEEN UNDELETED" on IMAP4 and
> IMAP4r1
>       servers (Sunil Shetye).
>     * Workaround: The IMAP client now falls back to "FETCH n:m FLAGS"
> if the server
>       does not support "SEARCH". (Sunil Shetye)
>     * The IMAP client now requests message numbers in batches of 1,000
> to avoid
>       problems if there are more than 1860 unseen messages. (Sunil Shetye)
>         Note that this wasn't security relevant because fetchmail
> would only read up
>       to the maximum buffer size and leave the remainder of the string
> unread, going
>       out of synch afterwards.
>     
>     svn path=/branches/BRANCH_6-3/; revision=5468

> 2. After fetchin we set "\Seen" flag. Will be this neccessary with uids
> tracking?

It will no longer be necessary. We might make it an option, or we might
forego it entirely.

> 3. my first idea is to keep "not seen" ranges attached to (host,folder)
> ie something like 5:6,1000:* this can easilly be mapped to
> SEARCH UNSEEN UNDELETED UID <ranges here>

Makes sense. If you find the current .fetchids format non-workable, we
can address that, too. We used to have a different layout for .fetchids
that was more efficient, and split up, however the patches were for
6.2.5, surely will no longer apply, and I haven't forward-ported them to
6.4.x beta, let alone 7.0.x alpha.  And if you find awkward things that
need fixing, but should not distract from your current work, feel free
to file issues on Gitlab even without patches.

Thanks again for looking into this!

Cheers,
Matthias

[Fetchmail-users] some question and thoughts about imap and uids

[<kj...@o2...>]

I have read imap.c code and (it is possible I have overlooked something)

1. (imap_search)
   sometimes fetchmail issues "SEARCH UNSEEN" and sometimes "SEARCH
   UNSEEN UNDELETED" (it depends on "keep" flag )
   What is use case to issue "SEARCH UNSEEN" (without undeleted?)

2. After fetchin we set "\Seen" flag. Will be this neccessary with uids
tracking?

3. my first idea is to keep "not seen" ranges attached to (host,folder)
ie something like 5:6,1000:* this can easilly be mapped to
SEARCH UNSEEN UNDELETED UID <ranges here>

Any comments?
KJ


-- 
http://stopstopnop.pl/stop_stopnop.pl_o_nas.html
"Turn on, tune up, rock out."
		-- Billy Gibbons

Re: [Fetchmail-users] how to log date/time in daemon mode?

[Carlos E. R. <rob...@te...>]

On 03/10/2018 02.17, Matthias Andree wrote:
> Am 03.10.18 um 00:42 schrieb Carlos E. R.:
>> On 02/10/2018 21.23, Kamil Jońca wrote:
>>> "Carlos E. R." <> writes:
>>>
>>>> On 02/10/2018 20.03, Kamil Jońca wrote:
>>>>> "Carlos E. R." <> writes:
>>>>>
>>>>>> On 01/10/2018 20.09, Kamil Jońca wrote:
>>>>>>> Matthias Andree <mat...@gm...> writes:
>>>>>>>
>>>>>>> [..]
>>>>>>>> The consideration is whether this could go into 6.4, or would have to
>>>>>>>> wait for 7.0. The log file format is not really specified, so changing
>>>>>>>> it isn't breaking any documented interface, just breaking past practice,
>>>>>>>> so it might cause some astonishment. Opinions?
>>>>>>> Regardless of these questions I would as NOT to change logging via
>>>>>>> syslog.
>>>>>> Timestamping via syslog is done by the syslog daemon, not by the
>>>>>> application that sends the messages.
>>>> Don't you understand? Fetchmail can not change logging via syslog.
>>> Please correct me if I am wrong.
>>> 1. So far fetchmail produces "log messages"
>>> 2. these messages are identical, in case when logging to file/stderr or
>>> syslog.
>>> 3. syslog adds timestamp to messages received from fetchmail
>>> 4. now we want to extend tetchmail log message with timestamps.
>>> 5. so syslog will get messages with timestamps, and adds its own
>>> timestamp.
>>> 6. so, after adding timestamp to fetchmail messages and logging to
>>> syslog we will have TWO timestamps in logs. I think this is ugly, and I
>>> kindly ask Mathhias, not to change messages when someone pick "set
>>> syslog" option.
>> Well, I have never seen a programmer make that error in a released
>> version. I'm sure they know. The message is formed, sent to syslog, or
>> some extra entries are added, like a timestapmp, then printed (or
>> written to file).
>>
>>
> To calm your concerns,
> 
> 1. I thank Kamil for the hint to make sure we don't bluntly apply
> timestamping all over the map, to avoid double logging in syslog;
> 
> 2. I thank Carlos for the trust. Carlos, rest assured that oversights
> and mistakes do happen, and if the code that formats the message is
> shared between syslog and fetchmail code, that concern of Kamil's is
> indeed rather substantiated. And the formatting itself _is_ shared code.
> O:-) See report.c for the function report(). Hans's patch, however, only
> changed the file output, not the syslog output, and generated an
> additional system call, and would not change the log buffer, so syslog
> output would remain unadorned with the new logfile time stamp.

Welcome :-)

Without looking at the code, I was doing educated guesses. For instance,
from my own small programs I know that if I goof at that, I will notice
the error fast enough on testing it - and yes, errors do happen, but we
all do tests for that reason ;-)

-- 
Cheers / Saludos,

		Carlos E. R.

  (from openSUSE 15.0 (Legolas))

Re: [Fetchmail-users] how to log date/time in daemon mode?

[Matthias A. <mat...@gm...>]

Am 03.10.18 um 00:42 schrieb Carlos E. R.:
> On 02/10/2018 21.23, Kamil Jońca wrote:
>> "Carlos E. R." <> writes:
>>
>>> On 02/10/2018 20.03, Kamil Jońca wrote:
>>>> "Carlos E. R." <> writes:
>>>>
>>>>> On 01/10/2018 20.09, Kamil Jońca wrote:
>>>>>> Matthias Andree <mat...@gm...> writes:
>>>>>>
>>>>>> [..]
>>>>>>> The consideration is whether this could go into 6.4, or would have to
>>>>>>> wait for 7.0. The log file format is not really specified, so changing
>>>>>>> it isn't breaking any documented interface, just breaking past practice,
>>>>>>> so it might cause some astonishment. Opinions?
>>>>>> Regardless of these questions I would as NOT to change logging via
>>>>>> syslog.
>>>>> Timestamping via syslog is done by the syslog daemon, not by the
>>>>> application that sends the messages.
>>> Don't you understand? Fetchmail can not change logging via syslog.
>> Please correct me if I am wrong.
>> 1. So far fetchmail produces "log messages"
>> 2. these messages are identical, in case when logging to file/stderr or
>> syslog.
>> 3. syslog adds timestamp to messages received from fetchmail
>> 4. now we want to extend tetchmail log message with timestamps.
>> 5. so syslog will get messages with timestamps, and adds its own
>> timestamp.
>> 6. so, after adding timestamp to fetchmail messages and logging to
>> syslog we will have TWO timestamps in logs. I think this is ugly, and I
>> kindly ask Mathhias, not to change messages when someone pick "set
>> syslog" option.
> Well, I have never seen a programmer make that error in a released
> version. I'm sure they know. The message is formed, sent to syslog, or
> some extra entries are added, like a timestapmp, then printed (or
> written to file).
>
>
To calm your concerns,

1. I thank Kamil for the hint to make sure we don't bluntly apply
timestamping all over the map, to avoid double logging in syslog;

2. I thank Carlos for the trust. Carlos, rest assured that oversights
and mistakes do happen, and if the code that formats the message is
shared between syslog and fetchmail code, that concern of Kamil's is
indeed rather substantiated. And the formatting itself _is_ shared code.
O:-) See report.c for the function report(). Hans's patch, however, only
changed the file output, not the syslog output, and generated an
additional system call, and would not change the log buffer, so syslog
output would remain unadorned with the new logfile time stamp.

Re: [Fetchmail-users] how to log date/time in daemon mode?

[Carlos E. R. <rob...@te...>]

On 02/10/2018 21.23, Kamil Jońca wrote:
> "Carlos E. R." <> writes:
> 
>> On 02/10/2018 20.03, Kamil Jońca wrote:
>>> "Carlos E. R." <> writes:
>>>
>>>> On 01/10/2018 20.09, Kamil Jońca wrote:
>>>>> Matthias Andree <mat...@gm...> writes:
>>>>>
>>>>> [..]
>>>>>>
>>>>>> The consideration is whether this could go into 6.4, or would have to
>>>>>> wait for 7.0. The log file format is not really specified, so changing
>>>>>> it isn't breaking any documented interface, just breaking past practice,
>>>>>> so it might cause some astonishment. Opinions?
>>>>>
>>>>> Regardless of these questions I would as NOT to change logging via
>>>>> syslog.
>>>>
>>>> Timestamping via syslog is done by the syslog daemon, not by the
>>>> application that sends the messages.
>>
>> Don't you understand? Fetchmail can not change logging via syslog.
> 
> Please correct me if I am wrong.
> 1. So far fetchmail produces "log messages"
> 2. these messages are identical, in case when logging to file/stderr or
> syslog.
> 3. syslog adds timestamp to messages received from fetchmail
> 4. now we want to extend tetchmail log message with timestamps.
> 5. so syslog will get messages with timestamps, and adds its own
> timestamp.
> 6. so, after adding timestamp to fetchmail messages and logging to
> syslog we will have TWO timestamps in logs. I think this is ugly, and I
> kindly ask Mathhias, not to change messages when someone pick "set
> syslog" option.

Well, I have never seen a programmer make that error in a released
version. I'm sure they know. The message is formed, sent to syslog, or
some extra entries are added, like a timestapmp, then printed (or
written to file).

-- 
Cheers / Saludos,

		Carlos E. R.

  (from openSUSE 15.0 (Legolas))

Re: [Fetchmail-users] how to log date/time in daemon mode?

[<kj...@o2...>]

"Carlos E. R." <rob...@te...> writes:

> On 02/10/2018 20.03, Kamil Jońca wrote:
>> "Carlos E. R." <rob...@te...> writes:
>> 
>>> On 01/10/2018 20.09, Kamil Jońca wrote:
>>>> Matthias Andree <mat...@gm...> writes:
>>>>
>>>> [..]
>>>>>
>>>>> The consideration is whether this could go into 6.4, or would have to
>>>>> wait for 7.0. The log file format is not really specified, so changing
>>>>> it isn't breaking any documented interface, just breaking past practice,
>>>>> so it might cause some astonishment. Opinions?
>>>>
>>>> Regardless of these questions I would as NOT to change logging via
>>>> syslog.
>>>
>>> Timestamping via syslog is done by the syslog daemon, not by the
>>> application that sends the messages.
>
> Don't you understand? Fetchmail can not change logging via syslog.

Please correct me if I am wrong.
1. So far fetchmail produces "log messages"
2. these messages are identical, in case when logging to file/stderr or
syslog.
3. syslog adds timestamp to messages received from fetchmail
4. now we want to extend tetchmail log message with timestamps.
5. so syslog will get messages with timestamps, and adds its own
timestamp.
6. so, after adding timestamp to fetchmail messages and logging to
syslog we will have TWO timestamps in logs. I think this is ugly, and I
kindly ask Mathhias, not to change messages when someone pick "set
syslog" option.

KJ

-- 
http://stopstopnop.pl/stop_stopnop.pl_o_nas.html
If you don't know what game you're playing, don't ask what the score is.

Re: [Fetchmail-users] how to log date/time in daemon mode?

[Carlos E. R. <rob...@te...>]

On 02/10/2018 20.03, Kamil Jońca wrote:
> "Carlos E. R." <rob...@te...> writes:
> 
>> On 01/10/2018 20.09, Kamil Jońca wrote:
>>> Matthias Andree <mat...@gm...> writes:
>>>
>>> [..]
>>>>
>>>> The consideration is whether this could go into 6.4, or would have to
>>>> wait for 7.0. The log file format is not really specified, so changing
>>>> it isn't breaking any documented interface, just breaking past practice,
>>>> so it might cause some astonishment. Opinions?
>>>
>>> Regardless of these questions I would as NOT to change logging via
>>> syslog.
>>
>> Timestamping via syslog is done by the syslog daemon, not by the
>> application that sends the messages.

Don't you understand? Fetchmail can not change logging via syslog.


-- 
Cheers / Saludos,

		Carlos E. R.

  (from openSUSE 15.0 (Legolas))

Re: [Fetchmail-users] how to log date/time in daemon mode?

[<kj...@o2...>]

"Carlos E. R." <rob...@te...> writes:

> On 01/10/2018 20.09, Kamil Jońca wrote:
>> Matthias Andree <mat...@gm...> writes:
>> 
>> [..]
>>>
>>> The consideration is whether this could go into 6.4, or would have to
>>> wait for 7.0. The log file format is not really specified, so changing
>>> it isn't breaking any documented interface, just breaking past practice,
>>> so it might cause some astonishment. Opinions?
>> 
>> Regardless of these questions I would as NOT to change logging via
>> syslog.
>
> Timestamping via syslog is done by the syslog daemon, not by the
> application that sends the messages.

And?
KJ
-- 
http://stopstopnop.pl/stop_stopnop.pl_o_nas.html
You climb to reach the summit, but once there, discover that all roads
lead down.
		-- Stanislaw Lem, "The Cyberiad"

Re: [Fetchmail-users] how to log date/time in daemon mode?

[Carlos E. R. <rob...@te...>]

On 01/10/2018 20.09, Kamil Jońca wrote:
> Matthias Andree <mat...@gm...> writes:
> 
> [..]
>>
>> The consideration is whether this could go into 6.4, or would have to
>> wait for 7.0. The log file format is not really specified, so changing
>> it isn't breaking any documented interface, just breaking past practice,
>> so it might cause some astonishment. Opinions?
> 
> Regardless of these questions I would as NOT to change logging via
> syslog.

Timestamping via syslog is done by the syslog daemon, not by the
application that sends the messages.

-- 
Cheers / Saludos,

		Carlos E. R.

  (from openSUSE 15.0 (Legolas))

Re: [Fetchmail-users] how to log date/time in daemon mode?

[grarpamp <gra...@gm...>]

> Would it be unreasonable to include both ISO8601 and time_t? eg.
>
>    2018-10-01T22:50:44+00:00 (1538434244)
>
> The time_t would simply be easier for an external script to parse.

Some apps have user configurable global choice to make happy / useful...
a) iso8601 (default)
b) $(date)
c) epoch
d) strftime format string, ie above: "%Y-%m-%dT%H:%M:%S%z (%s)"

Rightly leaving the box, localtime, and TZ env,
to handle the timezones and such.

As to when / if anything, 7.0 would seem best oppurtunity,
as there will be some change and other new things and
oppurtunities there anyways.