fetchmail-users — User questions and discussion

Re: [Fetchmail-users] Fetchmail seems to silently fail when I use the --pidfile option

[Matthias A. <mat...@gm...>]

Am 12.10.20 um 23:04 schrieb Alex Hall:
> Hello,
> I've used fetchmail for a couple years, as a way of pulling emails
> into the Request Tracker ticket system I set up for my work. I'm
> trying to better automate it, by setting it up with Monit. To do that,
> I want to use the --pidfile option when starting or quitting
> fetchmail, which I've not tried before.
>
> First, the details. I'm on Debian 8, with fetchmail 6.3.26. I
> installed this from Debian's package manager; I didn't compile any
> source. I know Debian 8 is outdated, but we're stuck on this server
> for a while for various reasons. The server reports that this is the
> most recent version of fetchmail it can find.
>
> In the past, I'd start fetchmail manually (this is part of what I want
> to automate with Monit). I use this command:
> sudo -u rt-user fetchmail -f /path/rt4/fetchmailrc
>
> To let Monit have a known pid file to monitor, I tried this command:
> sudo -u rt-user fetchmail -f /path/rt4/fetchmailrc --pidfile
> /var/run/rt_fetchmail.pid
>
> When I run that command in the terminal, nothing at all happens. I get
> no output, /var/run/rt_fetchmail.pid is not created, and fetchmail
> doesn't seem to run. I should say here that the fetchmailrc file being
> used commands fetchmail to start as a daemon. If I leave off the
> --pidfile bit, the daemon starts as expected. I can then
> sudo -u rt-user fetchmail --quit
> to kill it. Running that quit command after using the --pidfile
> variant produces no output.
>
Assuming it's a stopgap before the server is going to be replaced:

You can try and run fetchmail with the additional options     --nosyslog
--logfile /dev/stderr
and see if that gives you output, or possibly add -d0 to override any
"set daemon" directives in your fetchmailrc file.

> It seems like the --pidfile option causes a silent failure, but I
> can't figure out what's going on. The log only shows normal mail
> managing stuff, with no errors that I've found, and the terminal has
> no output whatsoever.

This is strange, check if the user you "sudo" into has a proper shell
and permissions and the session doesn't crash before starting fetchmail.
Possibly you may also want to install strace and try

sudo -u rt-user strace -f fetchmail -f /path/rt4/fetchmailrc --pidfile
/var/run/rt_fetchmail.pid

and see where it all ends up. (strace -o /path/to/dumpfile <- that saves
the output).

[Fetchmail-users] test

[Alex H. <ah...@au...>]

Test. Please ignore.

[Fetchmail-users] Fetchmail seems to silently fail when I use the --pidfile option

[Alex H. <ah...@au...>]

Hello,
I've used fetchmail for a couple years, as a way of pulling emails into 
the Request Tracker ticket system I set up for my work. I'm trying to 
better automate it, by setting it up with Monit. To do that, I want to 
use the --pidfile option when starting or quitting fetchmail, which I've 
not tried before.

First, the details. I'm on Debian 8, with fetchmail 6.3.26. I installed 
this from Debian's package manager; I didn't compile any source. I know 
Debian 8 is outdated, but we're stuck on this server for a while for 
various reasons. The server reports that this is the most recent version 
of fetchmail it can find.

In the past, I'd start fetchmail manually (this is part of what I want 
to automate with Monit). I use this command:
sudo -u rt-user fetchmail -f /path/rt4/fetchmailrc

To let Monit have a known pid file to monitor, I tried this command:
sudo -u rt-user fetchmail -f /path/rt4/fetchmailrc --pidfile 
/var/run/rt_fetchmail.pid

When I run that command in the terminal, nothing at all happens. I get 
no output, /var/run/rt_fetchmail.pid is not created, and fetchmail 
doesn't seem to run. I should say here that the fetchmailrc file being 
used commands fetchmail to start as a daemon. If I leave off the 
--pidfile bit, the daemon starts as expected. I can then
sudo -u rt-user fetchmail --quit
to kill it. Running that quit command after using the --pidfile variant 
produces no output.

It seems like the --pidfile option causes a silent failure, but I can't 
figure out what's going on. The log only shows normal mail managing 
stuff, with no errors that I've found, and the terminal has no output 
whatsoever.

I've found nothing online about this problem, so am not sure where else 
to turn. If anyone on this list has ideas, I'd love to hear them. If 
I've left out any important information, let me know. Thanks for any help.


--

Alex Hall

ah...@au...

Re: [Fetchmail-users] making fetchmail delete a successfully downloaded email from dovecot

[Gene H. <ghe...@sh...>]

On Sunday 11 October 2020 15:29:02 Lucio Chiappetti wrote:

> On Sun, 11 Oct 2020, Gene Heskett wrote:
> >>>>> Shentel.net does not allow fetchmail to delete a downloaded
> >>>>> email because their dovecot server handles imap and pop3 from
> >>>>> the same directory and somebody might want to come in from imap.
>
> I cannot talk for your provider, but can tell my experience with
> gsuite (gmail) since my institution moved to it.

Where do I send flowers of sympathy? gmail was broken the 6 months I put 
up with it

It took a wee bit of tinkering & studying of the manpage, but its now 
using imap and successfully expunging what it fetches. So until they 
fill the spam folder up too, I am a happy camper.

> An account of my tests and setup is in
> http://sax.iasf-milano.inaf.it/~lucio/WWW/WhereManWins/gs.html

I'll take a look, thanks for shareing.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: [Fetchmail-users] making fetchmail delete a successfully downloaded email from dovecot

[Lucio C. <lu...@la...>]

On Sun, 11 Oct 2020, Gene Heskett wrote:

>>>>> Shentel.net does not allow fetchmail to delete a downloaded email
>>>>> because their dovecot server handles imap and pop3 from the same
>>>>> directory and somebody might want to come in from imap.

I cannot talk for your provider, but can tell my experience with gsuite 
(gmail) since my institution moved to it.

Before I used to receive e-mail directly on my institute workstation and 
had it filtered via procmail, and read it with alpine (only occasionally I 
activated a local imapd and used a local webalpine web mailer from 
remote).

I wanted to preserve the same, so I run fetchmail every 5 min via crontab 
and get all "good" email from gsuite via pop3.

I choose pop3 because the imap by google is faulty in my opinion, i.e. a 
"delete and expunge" *in its inbox* does not delete messages (it makes 
them disappear from inbox, but they remain in an "All mail" PSEUDOfolder.

pop3 by google is not perfect too but allows a viable workaround. In this 
case stuff got via fetchmail ends in the "Bin" pseudofolder.

>>>> What I would do, is log it to the imap server with any mail client
>>>> (say mutt, Alpine, Thunderbird), select all the messages and
>>>> "click" delete.

This is possible and in fact I have two imap connections to gsuite in my 
alpine .pinerc.

- one goes to gsuite inbox. I use it very seldom (essentially it allows
   me to see mail "transiting" in inbox in the 5 min before it is fetched
   by fetchmail ... used for some "confirmation e-mail")

- the other one is a folder collection and sees all other pseudofolders.
   I check it once per day for the following purposes:

   - to check false positives in the Spam folder (to tag "non spam" I
     have to go the Gsuite webmail)

   - to select all "deleted" messages of the last day in the Bin folder
     and delete-and-expunge them

An account of my tests and setup is in
http://sax.iasf-milano.inaf.it/~lucio/WWW/WhereManWins/gs.html


-- 
Lucio Chiappetti - INAF/IASF - via Corti 12 - I-20133 Milano (Italy)
For more info : http://www.iasf-milano.inaf.it/~lucio/personal.html

Re: [Fetchmail-users] making fetchmail delete a successfully downloaded email from dovecot

[Carlos E. R. <rob...@te...>]

On 11/10/2020 20.33, Gene Heskett wrote:
> On Sunday 11 October 2020 14:24:00 Gene Heskett wrote:
>> On Sunday 11 October 2020 13:22:28 Carlos E. R. wrote:
>>
>> Ohhkayy, I just made it work, and it fetched everything using imap
>> that had accumulated in the last 2 hours. So I used kmails delete dups
>> on those.
>> from one pass:
>> fetchmail: IMAP> A0003 SELECT "INBOX"
>> fetchmail: IMAP< * FLAGS (\Answered \Flagged \Deleted \Seen \Draft
>> $POP3Deleted)
>> fetchmail: IMAP< * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted
>> \Seen \Draft $POP3Deleted \*)] Flags permitted.
>> fetchmail: IMAP< * 1 EXISTS
>> fetchmail: IMAP< * 1 RECENT
>> fetchmail: IMAP< * OK [UNSEEN 1] First unseen.
>> fetchmail: IMAP< * OK [UIDVALIDITY 1470639859] UIDs valid
>> fetchmail: IMAP< * OK [UIDNEXT 242714] Predicted next UID
>> fetchmail: IMAP< * OK [HIGHESTMODSEQ 427314] Highest
>> fetchmail: IMAP< A0003 OK [READ-WRITE] Select completed (0.013 + 0.000
>> + 0.012 secs).
>> fetchmail: IMAP> A0004 EXPUNGE

Expunge is delete, I understand.

>> fetchmail: IMAP< A0004 OK Expunge completed (0.001 + 0.000 secs).
>>
>> was it able to actually delete the one msg it pulled?
>> Or am I stuck using a browser to clean it out about weekly?
>>
>> Thanks.
> 
> logging in with a browser answered that question with a resounding yes.
> So fetchmail is actually doing what it was told to do for the first time
> in a couple years.  Thank you very much.

Welcome :-)

-- 
Cheers / Saludos,

		Carlos E. R.
		(from 15.1 x86_64 at Telcontar)

Re: [Fetchmail-users] making fetchmail delete a successfully downloaded email from dovecot

[Gene H. <ghe...@sh...>]

On Sunday 11 October 2020 14:24:00 Gene Heskett wrote:

> On Sunday 11 October 2020 13:22:28 Carlos E. R. wrote:
> > On 11/10/2020 18.50, Gene Heskett wrote:
> > > On Sunday 11 October 2020 08:10:37 Carlos E. R. wrote:
> > >> On 11/10/2020 14.03, Gene Heskett wrote:
> > >>> Greetings all;
> > >>>
> > >>> I just spent slightly over an hour deleting old email from my
> > >>> ISP mail server 100 at a time, about 8 months worth and over
> > >>> 30,000 msgs.
> > >>>
> > >>> Shentel.net does not allow fetchmail to delete a downloaded
> > >>> email because their dovecot server handles imap and pop3 from
> > >>> the same directory and somebody might want to come in from imap.
> > >>
> > >> I don't understand this.
> > >>
> > >> What I would do, is log it to the imap server with any mail
> > >> client (say mutt, Alpine, Thunderbird), select all the messages
> > >> and "click" delete.
> > >>
> > >> Surely you can also log to that imap-server using fetchmail, too.
> > >
> > > I don't think it works when coming in thru its pop.isp-name port
> > > 995, Does fetchmail 6.4.8 even speak imap?
> >
> > Certainly fetchmail talks imap. I am on fetchmail-6.3.26 myself, I
> > have been using imap for more than a decade or two. It was some mail
> > servers which did not provide imap, but fetchmail had it as long as
> > I remember.
>
> Ohhkayy, I just made it work, and it fetched everything using imap
> that had accumulated in the last 2 hours. So I used kmails delete dups
> on those.
> from one pass:
> fetchmail: IMAP> A0003 SELECT "INBOX"
> fetchmail: IMAP< * FLAGS (\Answered \Flagged \Deleted \Seen \Draft
> $POP3Deleted)
> fetchmail: IMAP< * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted
> \Seen \Draft $POP3Deleted \*)] Flags permitted.
> fetchmail: IMAP< * 1 EXISTS
> fetchmail: IMAP< * 1 RECENT
> fetchmail: IMAP< * OK [UNSEEN 1] First unseen.
> fetchmail: IMAP< * OK [UIDVALIDITY 1470639859] UIDs valid
> fetchmail: IMAP< * OK [UIDNEXT 242714] Predicted next UID
> fetchmail: IMAP< * OK [HIGHESTMODSEQ 427314] Highest
> fetchmail: IMAP< A0003 OK [READ-WRITE] Select completed (0.013 + 0.000
> + 0.012 secs).
> fetchmail: IMAP> A0004 EXPUNGE
> fetchmail: IMAP< A0004 OK Expunge completed (0.001 + 0.000 secs).
>
> was it able to actually delete the one msg it pulled?
> Or am I stuck using a browser to clean it out about weekly?
>
> Thanks.

logging in with a browser answered that question with a resounding yes.
So fetchmail is actually doing what it was told to do for the first time 
in a couple years.  Thank you very much.

> Cheers, Gene Heskett


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: [Fetchmail-users] making fetchmail delete a successfully downloaded email from dovecot

[Gene H. <ghe...@sh...>]

On Sunday 11 October 2020 13:22:28 Carlos E. R. wrote:

> On 11/10/2020 18.50, Gene Heskett wrote:
> > On Sunday 11 October 2020 08:10:37 Carlos E. R. wrote:
> >> On 11/10/2020 14.03, Gene Heskett wrote:
> >>> Greetings all;
> >>>
> >>> I just spent slightly over an hour deleting old email from my ISP
> >>> mail server 100 at a time, about 8 months worth and over 30,000
> >>> msgs.
> >>>
> >>> Shentel.net does not allow fetchmail to delete a downloaded email
> >>> because their dovecot server handles imap and pop3 from the same
> >>> directory and somebody might want to come in from imap.
> >>
> >> I don't understand this.
> >>
> >> What I would do, is log it to the imap server with any mail client
> >> (say mutt, Alpine, Thunderbird), select all the messages and
> >> "click" delete.
> >>
> >> Surely you can also log to that imap-server using fetchmail, too.
> >
> > I don't think it works when coming in thru its pop.isp-name port
> > 995, Does fetchmail 6.4.8 even speak imap?
>
> Certainly fetchmail talks imap. I am on fetchmail-6.3.26 myself, I
> have been using imap for more than a decade or two. It was some mail
> servers which did not provide imap, but fetchmail had it as long as I
> remember.
Ohhkayy, I just made it work, and it fetched everything using imap that 
had accumulated in the last 2 hours. So I used kmails delete dups on 
those.
from one pass:
fetchmail: IMAP> A0003 SELECT "INBOX"
fetchmail: IMAP< * FLAGS (\Answered \Flagged \Deleted \Seen \Draft 
$POP3Deleted)
fetchmail: IMAP< * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen 
\Draft $POP3Deleted \*)] Flags permitted.
fetchmail: IMAP< * 1 EXISTS
fetchmail: IMAP< * 1 RECENT
fetchmail: IMAP< * OK [UNSEEN 1] First unseen.
fetchmail: IMAP< * OK [UIDVALIDITY 1470639859] UIDs valid
fetchmail: IMAP< * OK [UIDNEXT 242714] Predicted next UID
fetchmail: IMAP< * OK [HIGHESTMODSEQ 427314] Highest
fetchmail: IMAP< A0003 OK [READ-WRITE] Select completed (0.013 + 0.000 + 
0.012 secs).
fetchmail: IMAP> A0004 EXPUNGE
fetchmail: IMAP< A0004 OK Expunge completed (0.001 + 0.000 secs).                                 

was it able to actually delete the one msg it pulled?
Or am I stuck using a browser to clean it out about weekly?

Thanks.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: [Fetchmail-users] making fetchmail delete a successfully downloaded email from dovecot

[Carlos E. R. <rob...@te...>]

On 11/10/2020 18.50, Gene Heskett wrote:
> On Sunday 11 October 2020 08:10:37 Carlos E. R. wrote:
>> On 11/10/2020 14.03, Gene Heskett wrote:
>>> Greetings all;
>>>
>>> I just spent slightly over an hour deleting old email from my ISP
>>> mail server 100 at a time, about 8 months worth and over 30,000
>>> msgs.
>>>
>>> Shentel.net does not allow fetchmail to delete a downloaded email
>>> because their dovecot server handles imap and pop3 from the same
>>> directory and somebody might want to come in from imap.
>>
>> I don't understand this.
>>
>> What I would do, is log it to the imap server with any mail client
>> (say mutt, Alpine, Thunderbird), select all the messages and "click"
>> delete.
>>
>> Surely you can also log to that imap-server using fetchmail, too.
> 
> I don't think it works when coming in thru its pop.isp-name port 995,
> Does fetchmail 6.4.8 even speak imap?

Certainly fetchmail talks imap. I am on fetchmail-6.3.26 myself, I have 
been using imap for more than a decade or two. It was some mail servers 
which did not provide imap, but fetchmail had it as long as I remember.



-- 
Cheers / Saludos,

		Carlos E. R.
		(from 15.1 x86_64 at Telcontar)

Re: [Fetchmail-users] making fetchmail delete a successfully downloaded email from dovecot

[Gene H. <ghe...@sh...>]

On Sunday 11 October 2020 08:10:37 Carlos E. R. wrote:

> On 11/10/2020 14.03, Gene Heskett wrote:
> > Greetings all;
> >
> > I just spent slightly over an hour deleting old email from my ISP
> > mail server 100 at a time, about 8 months worth and over 30,000
> > msgs.
> >
> > Shentel.net does not allow fetchmail to delete a downloaded email
> > because their dovecot server handles imap and pop3 from the same
> > directory and somebody might want to come in from imap.
>
> I don't understand this.
>
> What I would do, is log it to the imap server with any mail client
> (say mutt, Alpine, Thunderbird), select all the messages and "click"
> delete.
>
> Surely you can also log to that imap-server using fetchmail, too.

I don't think it works when coming in thru its pop.isp-name port 995,
Does fetchmail 6.4.8 even speak imap? 


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: [Fetchmail-users] making fetchmail delete a successfully downloaded email from dovecot

[Carlos E. R. <rob...@te...>]

On 11/10/2020 14.03, Gene Heskett wrote:
> Greetings all;
> 
> I just spent slightly over an hour deleting old email from my ISP mail
> server 100 at a time, about 8 months worth and over 30,000 msgs.
> 
> Shentel.net does not allow fetchmail to delete a downloaded email because
> their dovecot server handles imap and pop3 from the same directory and
> somebody might want to come in from imap.

I don't understand this.

What I would do, is log it to the imap server with any mail client (say 
mutt, Alpine, Thunderbird), select all the messages and "click" delete.

Surely you can also log to that imap-server using fetchmail, too.



-- 
Cheers / Saludos,

		Carlos E. R.
		(from 15.1 x86_64 at Telcontar)

[Fetchmail-users] making fetchmail delete a successfully downloaded email from dovecot

[Gene H. <ghe...@sh...>]

Greetings all;

I just spent slightly over an hour deleting old email from my ISP mail 
server 100 at a time, about 8 months worth and over 30,000 msgs.

Shentel.net does not allow fetchmail to delete a downloaded email because 
their dovecot server handles imap and pop3 from the same directory and 
somebody might want to come in from imap.

I've got my own archive, so I log in useing a web browser and clean my 
account out occasionally.

Is there any way to make fetchmail use an html command to do the dele 
under these conditions??
 
Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: [Fetchmail-users] Setting Up OAUTH2 Support for Fetchmail and Postfix

[Matthias A. <mat...@gm...>]

Am 10.10.20 um 08:04 schrieb Ranjan Maitra:
>
> Thanks! I was looking around git and I guess the python code to get the token file is perhaps here: https://sourceforge.net/p/fetchmail/git/ci/next/tree/contrib/fetchmail-oauth2.py#l140
> But then, I was wondering where is the code in fetchmail that makes it work (i.e. makes use of the oauth2 written in the rc file):

The code is somewhat spread out, as Peter explains in his e-mail in more
detail (thanks, Peter!)

>     poll imap.gmail.com protocol imap
>       auth oauthbearer username "US...@gm..."
>       passwordfile "/home/path/to/access_token_file"
>       is LOCALUSER here sslmode wrapped sslcertck
>
> This is for gmail, mine is for office 365 so I am wondering if there are areas in the main source code that I have to modify. (I have the python code needed to get the token.) It seems to me that fetchmail has to recognize and make use of the oauthbearer hence my question.

Indeed I have accepted OAuth as contributed code but never used it
myself, it is not a first-class citizen.

It is somewhat coarse, derived from Google code, and may require heavy
tinkering to make it work for other providers/sites, because the sites
all use their own URI (URLs) to provide tokens.

On some past days, I looked at the specs, since there are various ways
to obtain tokens, and I've read (but not confirmed or disproven) that
some sites do not implement all, and some scheme hinges on a user
registering a copy of the open source fetchmail code on some developer
page of the ISP's website because the authentication scheme requires me
to bake app[lication]-specific credentials into the code, the concept is
already dead before the first line of code has been written, because
open source won't keep those secret and the whole concept hinges on
"closed source" and possibly "use my app store".

The easiest would be to try if the respective site permits you to
generate app-specific passwords and use those with a traditional login
scheme.

Re: [Fetchmail-users] Setting Up OAUTH2 Support for Fetchmail and Postfix

[Peter P. <ro...@ri...>]

On Sat, Oct 10, 2020 at 01:04:24AM -0500, Ranjan Maitra wrote:
> On Sat, 10 Oct 2020 01:28:02 +0200 Matthias Andree <mat...@gm...> wrote:
> 
> > Am 09.10.20 um 23:16 schrieb Ranjan Maitra:
> > > On Wed, 7 Oct 2020 20:33:13 +0200 Matthias Andree <mat...@gm...> wrote:
> > >
> > >> Am 07.10.20 um 19:00 schrieb Ranjan Maitra:
> > >>> On Wed, 7 Oct 2020 18:39:41 +0200 Matthias Andree <mat...@gm...> wrote:
> > >>>
> > >>>> Am 07.10.20 um 06:51 schrieb Ranjan Maitra:
> > >>>>> According to the site, that may be dated, but fetchmail has oauth2 support from fetchmail7. Will it be available soon?
> > >>>> Ranjan,
> > >>>>
> > >>>> that depends on how leniency of your definition of "soon".
> > >>>>
> > >>>> First we will have a 6.5 release branch, and that has not yet happened,
> > >>>> so I expect that 7.x will be a matter of 2021, but if you are running it
> > >>>> from the up-to-date "next" branch in Git (available on Gitlab and
> > >>>> Sourceforge), feel free to file bug reports.
> > >>>>
> > >>>> Regards,
> > >>>> Matthias
> > >>> Thanks,
> > >>>
> > >>> I have used fetchmail 7 without much of a problem (or something that I have noticed, anyway). Is it possible to backport the patch for oauth2 in the  6.x.x releases? I prefer using the rpms released by the distributions.
> > >> I am not doing that, and the purpose of the existing 6.x.y branches is
> > >> compatiblity as we update y and if possibly also when updating x. There
> > >> may be minor features in 6.5.y.
> > >>
> > >> 7.x.y is for major changes.
> > >>
> > > So, I was looking at https://sourceforge.net/projects/fetchmail/files/branch_7-alpha/ and here, there has not been an update here since about 2 years ago. Is the git version more up-to-date?
> > >
> > Yes it is, and I don't think I will be doing many releases from
> > development branches before it's release candidate time.
> > We have Git and we have autoreconf.
> >
> 
> Thanks! I was looking around git and I guess the python code to get the token file is perhaps here: https://sourceforge.net/p/fetchmail/git/ci/next/tree/contrib/fetchmail-oauth2.py#l140
> But then, I was wondering where is the code in fetchmail that makes it work (i.e. makes use of the oauth2 written in the rc file):
> 
>     poll imap.gmail.com protocol imap
>       auth oauthbearer username "US...@gm..."
>       passwordfile "/home/path/to/access_token_file"
>       is LOCALUSER here sslmode wrapped sslcertck
> 
> This is for gmail, mine is for office 365 so I am wondering if there are areas in the main source code that I have to modify. (I have the python code needed to get the token.) It seems to me that fetchmail has to recognize and make use of the oauthbearer hence my question.

To answer your direct question, the code is in parts of imap.c and
pop3.c, as well as the aptly named oauth2.c and oauth2.h files, also
with configuration-related pieces sprinkled across several other files.

However, if you are thinking of backporting fetchmail OAUTH supprt to
a 6.x branch... please bear in mind that I would be very, very wary of
(in fact, I would not do it at all) trying to copy five, ten, or even
two hundred lines from a heavily-reworked codebase. There may be several
types of issues:
- the new code uses helper functions that are not present in the old
  code: you have to either backport them, too, or rewrite the code to
  not use them. This is a rabbit hole that once you start falling in,
  it takes you days and weeks to reach any kind of bottom... and then
  you find out that you have practically "backported" two-thirds of
  the rest of the new code
- the new code uses functions with new or changed arguments: you have to
  either backport them (and adapt all the other pieces in the old code
  that call them), or rewrite the code to not use them; see above about
  the rabbit hole
- the new code uses data structures that exist in the old code, *but*
  have some of the fields removed or the meaning of some of the fields
  changed: this is the most "evil" one, since the code will compile, but
  then, when you try to run it, some other function will expect this
  field to have this value or that field to have a value consistent with
  the other one, and things will break, either badly, or in subtle ways
  that will be very, very hard to detect

So... yeah. I would either try to use the 7.x version directly, or wait.

G'luck,
Peter

-- 
Peter Pentchev  ro...@ri... ro...@de... pp...@st...
PGP key:        http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13

Re: [Fetchmail-users] Setting Up OAUTH2 Support for Fetchmail and Postfix

[Ranjan M. <ma...@em...>]

On Sat, 10 Oct 2020 01:28:02 +0200 Matthias Andree <mat...@gm...> wrote:

> Am 09.10.20 um 23:16 schrieb Ranjan Maitra:
> > On Wed, 7 Oct 2020 20:33:13 +0200 Matthias Andree <mat...@gm...> wrote:
> >
> >> Am 07.10.20 um 19:00 schrieb Ranjan Maitra:
> >>> On Wed, 7 Oct 2020 18:39:41 +0200 Matthias Andree <mat...@gm...> wrote:
> >>>
> >>>> Am 07.10.20 um 06:51 schrieb Ranjan Maitra:
> >>>>> According to the site, that may be dated, but fetchmail has oauth2 support from fetchmail7. Will it be available soon?
> >>>> Ranjan,
> >>>>
> >>>> that depends on how leniency of your definition of "soon".
> >>>>
> >>>> First we will have a 6.5 release branch, and that has not yet happened,
> >>>> so I expect that 7.x will be a matter of 2021, but if you are running it
> >>>> from the up-to-date "next" branch in Git (available on Gitlab and
> >>>> Sourceforge), feel free to file bug reports.
> >>>>
> >>>> Regards,
> >>>> Matthias
> >>> Thanks,
> >>>
> >>> I have used fetchmail 7 without much of a problem (or something that I have noticed, anyway). Is it possible to backport the patch for oauth2 in the  6.x.x releases? I prefer using the rpms released by the distributions.
> >> I am not doing that, and the purpose of the existing 6.x.y branches is
> >> compatiblity as we update y and if possibly also when updating x. There
> >> may be minor features in 6.5.y.
> >>
> >> 7.x.y is for major changes.
> >>
> > So, I was looking at https://sourceforge.net/projects/fetchmail/files/branch_7-alpha/ and here, there has not been an update here since about 2 years ago. Is the git version more up-to-date?
> >
> Yes it is, and I don't think I will be doing many releases from
> development branches before it's release candidate time.
> We have Git and we have autoreconf.
>

Thanks! I was looking around git and I guess the python code to get the token file is perhaps here: https://sourceforge.net/p/fetchmail/git/ci/next/tree/contrib/fetchmail-oauth2.py#l140
But then, I was wondering where is the code in fetchmail that makes it work (i.e. makes use of the oauth2 written in the rc file):

    poll imap.gmail.com protocol imap
      auth oauthbearer username "US...@gm..."
      passwordfile "/home/path/to/access_token_file"
      is LOCALUSER here sslmode wrapped sslcertck

This is for gmail, mine is for office 365 so I am wondering if there are areas in the main source code that I have to modify. (I have the python code needed to get the token.) It seems to me that fetchmail has to recognize and make use of the oauthbearer hence my question.

Many thanks,
Ranjan

Re: [Fetchmail-users] Setting Up OAUTH2 Support for Fetchmail and Postfix

[Matthias A. <mat...@gm...>]

Am 09.10.20 um 23:16 schrieb Ranjan Maitra:
> On Wed, 7 Oct 2020 20:33:13 +0200 Matthias Andree <mat...@gm...> wrote:
>
>> Am 07.10.20 um 19:00 schrieb Ranjan Maitra:
>>> On Wed, 7 Oct 2020 18:39:41 +0200 Matthias Andree <mat...@gm...> wrote:
>>>
>>>> Am 07.10.20 um 06:51 schrieb Ranjan Maitra:
>>>>> According to the site, that may be dated, but fetchmail has oauth2 support from fetchmail7. Will it be available soon?
>>>> Ranjan,
>>>>
>>>> that depends on how leniency of your definition of "soon".
>>>>
>>>> First we will have a 6.5 release branch, and that has not yet happened,
>>>> so I expect that 7.x will be a matter of 2021, but if you are running it
>>>> from the up-to-date "next" branch in Git (available on Gitlab and
>>>> Sourceforge), feel free to file bug reports.
>>>>
>>>> Regards,
>>>> Matthias
>>> Thanks,
>>>
>>> I have used fetchmail 7 without much of a problem (or something that I have noticed, anyway). Is it possible to backport the patch for oauth2 in the  6.x.x releases? I prefer using the rpms released by the distributions.
>> I am not doing that, and the purpose of the existing 6.x.y branches is
>> compatiblity as we update y and if possibly also when updating x. There
>> may be minor features in 6.5.y.
>>
>> 7.x.y is for major changes.
>>
> So, I was looking at https://sourceforge.net/projects/fetchmail/files/branch_7-alpha/ and here, there has not been an update here since about 2 years ago. Is the git version more up-to-date?
>
Yes it is, and I don't think I will be doing many releases from
development branches before it's release candidate time.
We have Git and we have autoreconf.

Re: [Fetchmail-users] Setting Up OAUTH2 Support for Fetchmail and Postfix

[Peter P. <ro...@ri...>]

On Fri, Oct 09, 2020 at 04:16:54PM -0500, Ranjan Maitra wrote:
> On Wed, 7 Oct 2020 20:33:13 +0200 Matthias Andree <mat...@gm...> wrote:
> 
> > Am 07.10.20 um 19:00 schrieb Ranjan Maitra:
> > > On Wed, 7 Oct 2020 18:39:41 +0200 Matthias Andree <mat...@gm...> wrote:
> > >
> > >> Am 07.10.20 um 06:51 schrieb Ranjan Maitra:
> > >>> According to the site, that may be dated, but fetchmail has oauth2 support from fetchmail7. Will it be available soon?
> > >> Ranjan,
> > >>
> > >> that depends on how leniency of your definition of "soon".
> > >>
> > >> First we will have a 6.5 release branch, and that has not yet happened,
> > >> so I expect that 7.x will be a matter of 2021, but if you are running it
> > >> from the up-to-date "next" branch in Git (available on Gitlab and
> > >> Sourceforge), feel free to file bug reports.
> > >>
> > >> Regards,
> > >> Matthias
> > > Thanks,
> > >
> > > I have used fetchmail 7 without much of a problem (or something that I have noticed, anyway). Is it possible to backport the patch for oauth2 in the  6.x.x releases? I prefer using the rpms released by the distributions.
> >
> > I am not doing that, and the purpose of the existing 6.x.y branches is
> > compatiblity as we update y and if possibly also when updating x. There
> > may be minor features in 6.5.y.
> >
> > 7.x.y is for major changes.
> >
> 
> So, I was looking at https://sourceforge.net/projects/fetchmail/files/branch_7-alpha/ and here, there has not been an update here since about 2 years ago. Is the git version more up-to-date?

The "next" branch in the Git repository has commits from this year:

https://sourceforge.net/p/fetchmail/git/ci/next/tree/

G'luck,
Peter

-- 
Peter Pentchev  ro...@ri... ro...@de... pp...@st...
PGP key:        http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13

Re: [Fetchmail-users] Setting Up OAUTH2 Support for Fetchmail and Postfix

[Ranjan M. <ma...@em...>]

On Wed, 7 Oct 2020 20:33:13 +0200 Matthias Andree <mat...@gm...> wrote:

> Am 07.10.20 um 19:00 schrieb Ranjan Maitra:
> > On Wed, 7 Oct 2020 18:39:41 +0200 Matthias Andree <mat...@gm...> wrote:
> >
> >> Am 07.10.20 um 06:51 schrieb Ranjan Maitra:
> >>> According to the site, that may be dated, but fetchmail has oauth2 support from fetchmail7. Will it be available soon?
> >> Ranjan,
> >>
> >> that depends on how leniency of your definition of "soon".
> >>
> >> First we will have a 6.5 release branch, and that has not yet happened,
> >> so I expect that 7.x will be a matter of 2021, but if you are running it
> >> from the up-to-date "next" branch in Git (available on Gitlab and
> >> Sourceforge), feel free to file bug reports.
> >>
> >> Regards,
> >> Matthias
> > Thanks,
> >
> > I have used fetchmail 7 without much of a problem (or something that I have noticed, anyway). Is it possible to backport the patch for oauth2 in the  6.x.x releases? I prefer using the rpms released by the distributions.
>
> I am not doing that, and the purpose of the existing 6.x.y branches is
> compatiblity as we update y and if possibly also when updating x. There
> may be minor features in 6.5.y.
>
> 7.x.y is for major changes.
>

So, I was looking at https://sourceforge.net/projects/fetchmail/files/branch_7-alpha/ and here, there has not been an update here since about 2 years ago. Is the git version more up-to-date?

Ranjan

Re: [Fetchmail-users] Setting Up OAUTH2 Support for Fetchmail and Postfix

[Matthias A. <mat...@gm...>]

Am 07.10.20 um 19:00 schrieb Ranjan Maitra:
> On Wed, 7 Oct 2020 18:39:41 +0200 Matthias Andree <mat...@gm...> wrote:
>
>> Am 07.10.20 um 06:51 schrieb Ranjan Maitra:
>>> According to the site, that may be dated, but fetchmail has oauth2 support from fetchmail7. Will it be available soon?
>> Ranjan,
>>
>> that depends on how leniency of your definition of "soon".
>>
>> First we will have a 6.5 release branch, and that has not yet happened,
>> so I expect that 7.x will be a matter of 2021, but if you are running it
>> from the up-to-date "next" branch in Git (available on Gitlab and
>> Sourceforge), feel free to file bug reports.
>>
>> Regards,
>> Matthias
> Thanks,
>
> I have used fetchmail 7 without much of a problem (or something that I have noticed, anyway). Is it possible to backport the patch for oauth2 in the  6.x.x releases? I prefer using the rpms released by the distributions.

I am not doing that, and the purpose of the existing 6.x.y branches is
compatiblity as we update y and if possibly also when updating x. There
may be minor features in 6.5.y.

7.x.y is for major changes.

Re: [Fetchmail-users] Setting Up OAUTH2 Support for Fetchmail and Postfix

[Ranjan M. <ma...@em...>]

On Wed, 7 Oct 2020 18:39:41 +0200 Matthias Andree <mat...@gm...> wrote:

> Am 07.10.20 um 06:51 schrieb Ranjan Maitra:
> > According to the site, that may be dated, but fetchmail has oauth2 support from fetchmail7. Will it be available soon?
>
> Ranjan,
>
> that depends on how leniency of your definition of "soon".
>
> First we will have a 6.5 release branch, and that has not yet happened,
> so I expect that 7.x will be a matter of 2021, but if you are running it
> from the up-to-date "next" branch in Git (available on Gitlab and
> Sourceforge), feel free to file bug reports.
>
> Regards,
> Matthias

Thanks,

I have used fetchmail 7 without much of a problem (or something that I have noticed, anyway). Is it possible to backport the patch for oauth2 in the  6.x.x releases? I prefer using the rpms released by the distributions.

Ranjan

Re: [Fetchmail-users] Setting Up OAUTH2 Support for Fetchmail and Postfix

[Matthias A. <mat...@gm...>]

Am 07.10.20 um 06:51 schrieb Ranjan Maitra:
> According to the site, that may be dated, but fetchmail has oauth2 support from fetchmail7. Will it be available soon?

Ranjan,

that depends on how leniency of your definition of "soon".

First we will have a 6.5 release branch, and that has not yet happened,
so I expect that 7.x will be a matter of 2021, but if you are running it
from the up-to-date "next" branch in Git (available on Gitlab and
Sourceforge), feel free to file bug reports.

Regards,
Matthias

Re: [Fetchmail-users] Setting Up OAUTH2 Support for Fetchmail and Postfix

[Ranjan M. <ma...@em...>]

According to the site, that may be dated, but fetchmail has oauth2 support from fetchmail7. Will it be available soon?

Many thanks,
Ranjan

On Sat, 3 Oct 2020 13:52:52 -0400 "Gerard E. Seibert" <ger...@ou...> wrote:

> I just found this site,
> http://mmogilvi.users.sourceforge.net/software/oauthbearer.html
>
> I was wondering if anyone had tried it.
>
> --
> Gerard

Re: [Fetchmail-users] Polling interval: fetchmailrc 'set daemon 180' not being honoured

[Matthias A. <mat...@gm...>]

Am 21.09.20 um 03:56 schrieb Chris James:
> Hi,
>
> I’ve been having issues (potential bug) with the polling interval using fetchmail-6.4.8
>
> I’m running FreeBSD 11.3 inside an iocage jail. The package was pulled from the ‘quarterly’ repository, and is the latest currently available from that source. I’ve checked the bug tracker and see nothing related, nor any fix in releases 6-4-9 >> 6-4-12.
>
> My mail stack is comprised of:
>
> - fetchmail
> - postfix
> - rspamd
> - dovecot
>
> The full stack has been operating for months with no issues.
>
> Fetchmail is running in daemon mode, with the config file located at:
>
> /usr/local/etc/fetchmailrc
>
> … and (redacted) contains:
>
> set daemon 180

[...]

Chris,

FreeBSD's fetchmail port maintainer and I have worked to fix this bug in
FreeBSD's port version 6.4.12_1, for now it's only on the trunk, and I
have not yet requested to merge back to quarterly, but might if it
proves to work out for everyone, or else in January 2021 the fix will
propagate to quarterly.

https://svnweb.freebsd.org/ports?view=revision&revision=551537

Cheers,
Matthias

[Fetchmail-users] Setting Up OAUTH2 Support for Fetchmail and Postfix

[Gerard E. S. <ger...@ou...>]

I just found this site,
http://mmogilvi.users.sourceforge.net/software/oauthbearer.html

I was wondering if anyone had tried it.

-- 
Gerard

Re: [Fetchmail-users] somehow broke fetchmail trying to fix outgoing email :(

[Matthias A. <mat...@gm...>]

Am 02.10.20 um 08:05 schrieb Abe Stone via Fetchmail-users:
> Hi,
>
> When I got up this morning I found that my outgoing mail
> wasn't working. Of course that had nothing to do with
> fetchmail, and I was still able to retrieve my mail from
> gmail using POP3 as usual. I spent most of the day getting
> the outgoing mail to work again. But now I find that
> fetchmail can't deliver the incoming mail locally anymore. I
> get "smtp listener protocol error."
>
> The weird thing is, even when I change main.cf back to what
> it was before (which would break my outgoing mail again, but
> never mind that), the problem persists.
>
> I'm running fetchmail 6.4.4 on Mac OS 10.13.6 (High Sierra).
>
> The verbose fetchmail log looks like this (as you can see,
> everything works fine until it tries to deliver locally):
>
> fetchmail: 6.4.4 querying pop.gmail.com (protocol POP3) at Thu, 01 Oct 2020 21:50:34 -0700 (PDT): poll started
> [...]
> fetchmail: reading message abe...@gm...@pop.gmail.com:1 of 59 (35221 octets)Trying to connect to 127.0.0.1/25...connected.
> fetchmail: smtp listener protocol error
> fetchmail: Trying to connect to 127.0.0.1/25...connected.
> fetchmail: smtp listener protocol error
> fetchmail: SMTP connect to localhost failed
> fetchmail: POP3> QUIT
> fetchmail: POP3< --Mjk5MjIxMDM3OTAzNjQ2MDg
> fetchmail: SMTP transaction error while fetching from abe...@gm...@pop.gmail.com and delivering to SMTP host localhost
> fetchmail: 6.4.4 querying pop.gmail.com (protocol POP3) at Thu, 01 Oct 2020 22:00:35 -0700 (PDT): poll completed
> fetchmail: Query status=10 (SMTP)
> fetchmail: normal termination, status 10
>
> The verbose log from smtpd looks like this:
>
> 2020-10-01 21:50:35.514198-0700 0x72ee     Info        0x0                  1414   smtp: unknown_stream: wanted attribute: (any attribute name or list terminator)

Abe,

there is one long-standing conceptual design, umm, idiosyncrasy in
Postfix. That is that Postfix uses "smtp" and "smtpd" names which are
quite similar, rather than smtp-sender and smtp-listener or something.

All logs and configuration items you're showing from Postfix are smtp
(the client = sender) logs and configuration, no smtpd (receiver) logs,
but the latter is what fetchmail wants to talk to. Mind the trailing d
of the Postfix daemon name. You are looking in the wrong place.
Apparently the listener connects and then dies. Check the master and
smtpd logs, and if smtpd misses any services or you see them crash, or
if some configuration does not permissively allow clients from 127.0.0.1.

You may then also stand a better chance of getting replies by asking on
the postfix-users mailing list, where you best provide the output of
postconf -n (main.cf differences from defaults) and possibly postconf -M
(master.cf)

HTH,
Matthias