fetchmail-users — User questions and discussion

Re: [Fetchmail-users] Fetchmail logrotate script works once then fails next run

[Chris <cpo...@em...>]

On Sun, 2017-02-12 at 16:25 +0100, Matthias Andree wrote:
> Am 12.02.2017 um 15:56 schrieb Chris:
> > 
> >         postrotate
> >         /usr/local/bin/fetchmail --quit
> >         /usr/local/bin/fetchmail -v --nokeep --nosyslog --logfile
> > /home/chris/fetchmaillog --uidl -m procmail
> >         endscript
> > [...]
> > running postrotate script
> > fetchmail: background fetchmail at 3786 killed.
> > fetchmail: can't accept options while a background fetchmail is
> > running.
> 
> > 
> > I don't understand why it worked the first time but not the 2nd as
> > nothing was changed in between.
> 
> Hi Chris,
> 
> Looks like a race between the three instances of fetchmail involved:
> 
> 1. you have 3786
> 2. postrotate starts fetchmail -q, which just signals 3786 to exit,
> but
> does not wait for its exit (because it's not its child that wouldn't
> be
> 100% safe anyhow)
> 3. postrotate immediately starts a new fetchmail, which stumbles over
> the still existing instance that's still cleaning up.
> 
> It'd probably be good to wait until the prior instance has really
> quit
> before starting the next one, like so (adjust the pid file's path):
> 
> /usr/local/bin/fetchmail --quit
> 
> while [ -e /path/to/fetchmail.pid ] ; do sleep 1 ; done
> 
> /usr/local/bin/fetchmail -v --nokeep --nosyslog --logfile
> /home/chris/fetchmaillog --uidl -m procmail
> 
> 
> and possibly adding a counter so it does not wait too long or
> otherwise
> use SIGKILL to get rid of the old instance.
> 
> Neither will be 100% bullet-proof unless you defer to a service
> supervisor such as runit, upstart, systemd, or thereabouts, and just
> request a fetchmail restart from such a supervisor in your postrotate
> script.
> 
> HTH
> Matthias
> 
Thanks Matthias, I'm sure it will. I'll make the changes and give it
some test runs until I get it right. Appreciate the assistance.

Chris

-- 
Chris
KeyID 0xE372A7DA98E6705C
31.11972; -97.90167 (Elev. 1092 ft)
11:01:44 up 9 days, 2:59, 2 users, load average: 0.22, 0.16, 0.10
Ubuntu 16.04.1 LTS, kernel 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18
14:10:15 UTC 2017

Re: [Fetchmail-users] Fetchmail logrotate script works once then fails next run

[Matthias A. <mat...@gm...>]

Am 12.02.2017 um 15:56 schrieb Chris:
>         postrotate
>         /usr/local/bin/fetchmail --quit
>         /usr/local/bin/fetchmail -v --nokeep --nosyslog --logfile
> /home/chris/fetchmaillog --uidl -m procmail
>         endscript
> [...]
> running postrotate script
> fetchmail: background fetchmail at 3786 killed.
> fetchmail: can't accept options while a background fetchmail is
> running.

> I don't understand why it worked the first time but not the 2nd as
> nothing was changed in between.

Hi Chris,

Looks like a race between the three instances of fetchmail involved:

1. you have 3786
2. postrotate starts fetchmail -q, which just signals 3786 to exit, but
does not wait for its exit (because it's not its child that wouldn't be
100% safe anyhow)
3. postrotate immediately starts a new fetchmail, which stumbles over
the still existing instance that's still cleaning up.

It'd probably be good to wait until the prior instance has really quit
before starting the next one, like so (adjust the pid file's path):

/usr/local/bin/fetchmail --quit

while [ -e /path/to/fetchmail.pid ] ; do sleep 1 ; done

/usr/local/bin/fetchmail -v --nokeep --nosyslog --logfile /home/chris/fetchmaillog --uidl -m procmail


and possibly adding a counter so it does not wait too long or otherwise
use SIGKILL to get rid of the old instance.

Neither will be 100% bullet-proof unless you defer to a service
supervisor such as runit, upstart, systemd, or thereabouts, and just
request a fetchmail restart from such a supervisor in your postrotate
script.

HTH
Matthias

[Fetchmail-users] Fetchmail logrotate script works once then fails next run

[Chris <cpo...@em...>]

This is a logrotate configuration file that I modified from what
Matthias had put out about 10yrs ago. I modified it to fit my needs:

/home/chris/fetchmaillog {
	weekly
	rotate 8
	compress
	dateext
	missingok
	notifempty
	create 0664 chris chris
        sharedscripts
        postrotate
        /usr/local/bin/fetchmail --quit
        /usr/local/bin/fetchmail -v --nokeep --nosyslog --logfile
/home/chris/fetchmaillog --uidl -m procmail
        endscript
}

This is the command to execute the rotation Sunday mornings at 8am. For
some reason without the -f it will tell me the log doesn't need
rotation. 

/usr/sbin/logrotate -f -v --state=/home/chris/logrotate/status
/home/chris/logrotate/fetchmail.logrotate

It ran fine on 5 Feb:

reading config file /home/chris/logrotate/fetchmail.logrotate

Handling 1 logs

rotating pattern: /home/chris/fetchmaillog  forced from command line (8
rotations)
empty log files are not rotated, old logs are removed
considering log /home/chris/fetchmaillog
  log needs rotating
rotating log /home/chris/fetchmaillog, log->rotateCount is 8
dateext suffix '-20170205'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /home/chris/fetchmaillog to /home/chris/fetchmaillog-20170205
creating new /home/chris/fetchmaillog mode = 0664 uid = 1000 gid = 1000
running postrotate script
fetchmail: background fetchmail at 1700 killed.
compressing log with: /bin/gzip

Although to be honest I can't recall if it was restarted or not. That's
why I have it running at 8am when I can observe. When logrotate was run
this morning however:

reading config file /home/chris/logrotate/fetchmail.logrotate

Handling 1 logs

rotating pattern: /home/chris/fetchmaillog  forced from command line (8
rotations)
empty log files are not rotated, old logs are removed
considering log /home/chris/fetchmaillog
  log needs rotating
rotating log /home/chris/fetchmaillog, log->rotateCount is 8
dateext suffix '-20170212'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /home/chris/fetchmaillog to /home/chris/fetchmaillog-20170212
creating new /home/chris/fetchmaillog mode = 0664 uid = 1000 gid = 1000
running postrotate script
fetchmail: background fetchmail at 3786 killed.
fetchmail: can't accept options while a background fetchmail is
running.
argc = 9, arg list:
arg 1 = "-v"
arg 2 = "--nokeep"
arg 3 = "--nosyslog"
arg 4 = "--logfile"
arg 5 = "/home/chris/fetchmaillog"
arg 6 = "--uidl"
arg 7 = "-m"
arg 8 = "procmail"
error: error running shared postrotate script for
'/home/chris/fetchmaillog '

I don't understand why it worked the first time but not the 2nd as
nothing was changed in between.

Apologies Matthias when I said yesterday it was working.

Chris

-- 
Chris
KeyID 0xE372A7DA98E6705C
31.11972; -97.90167 (Elev. 1092 ft)
08:40:34 up 9 days, 38 min, 2 users, load average: 0.61, 0.44, 0.34
Ubuntu 16.04.1 LTS, kernel 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18
14:10:15 UTC 2017

Re: [Fetchmail-users] 6.4.0.beta2

[Matthias A. <mat...@gm...>]

Am 10.02.2017 um 16:12 schrieb Chris:
> Matthias, thought I'd drop by and let you know I'm still having no
> problems at all with 6.4.0.beta2. I also now have a log rotate working
> for my fetchmail log, finally.

Hi Chris,

thanks a lot for the feedback, very reassuring.

Cheers,
Matthias

Re: [Fetchmail-users] IMAP< A0002 NO LOGIN failed

[Carlos E. R. <rob...@te...>]

On 2017-02-10 22:09, Matthias Andree wrote:
> Am 10.02.2017 um 11:32 schrieb Axel Jantsch:

>> But I do not know how I could use that name because then I get the error
>> for ax...@kt...@webmail.kth.se
>>
>> Maybe I should use some Exchange specific format like
>>
>> ug.kth.se\axel  ???
> 
> Not sure if that helps, I am unaware of how Exchange's IMAP server works
> internally,
> and how the IMAP and MAPI (or whatever Exchange uses these days to
> natively talk to Outlook) settings differ.

Sometimes I use a trick: I try to configure Thunderbird to retrieve
email (they seem to have some sort of autoconfigure and database), and
see if it figures out things, then I replicate the resulting working
config in fetchmail ;-)

If Thunderbird also fails, you have two tools to report to the IT staff
when you ask. You can even lie and say you use Thunderbird on Windows,
so they can not say "we don't support your software choice" :-P

-- 
Cheers / Saludos,

		Carlos E. R.

  (from 42.2 x86_64 "Malachite" (Minas Tirith))

Re: [Fetchmail-users] IMAP< A0002 NO LOGIN failed

[Matthias A. <mat...@gm...>]

Am 10.02.2017 um 11:32 schrieb Axel Jantsch:
> Hi Matthias,
>
> Many thanks for analyzing this problem so carefully.
>
> So somehow I use the wrong username or password. I suspect the username
> because the error message says

I do not think that you were doing anything wrong on your end -
fetchmail's reporting appears to be misleading you.

Fetchmail, specifically, does _not_ report your e-mail address (which it
does not normally even know, or need to know), but it reports the
username and the server's name, and concatenates them with an @. It's a
bit unfortunate because it looks like an e-mail address when it usually
isn't, but there's so much secondary literature on fetchmail I can't
change that lightly without confusing everyone.

Fetchmail reports following a pattern LLLLL@SSSSS, LLLLL is the username
or login, SSSSS is the server name, in your case the user name axel, and
the server's name webmail.kth.se. For me, it would report ...
"mat...@gm...@imap.gmx.net". See the two at signs and be
totally confused. However, KTH appears to use only the unqualified user
name for login.

Look at the screenshot below item #5 at
<https://www.kth.se/en/student/kth-it-support/work-online/email/settings-and-config/installningar-for-thunderbird-1.282051>
again, it just has the bare user name as Username (at the bottom of the
screenshot), and the imap server name above. To me, your .fetchmailrc as
originally posted looks sane.

> but my email address and account is really ax...@kt...
>
> But I do not know how I could use that name because then I get the error
> for ax...@kt...@webmail.kth.se
>
> Maybe I should use some Exchange specific format like
>
> ug.kth.se\axel  ???

Not sure if that helps, I am unaware of how Exchange's IMAP server works
internally,
and how the IMAP and MAPI (or whatever Exchange uses these days to
natively talk to Outlook) settings differ.

So unless something changed recently on the server's end, which isn't on
any of the KTH IT pages that I've skimmed,
I'd suggest to get a hold of KTH IT support next week and ask for
assistance. Perhaps some flags on the account are wrong, sometimes there
are mistakes... call the guys during office ours, explain and ask.

HTH
Matthias

Re: [Fetchmail-users] IMAP< A0002 NO LOGIN failed

[grarpamp <gra...@gm...>]

On Fri, Feb 10, 2017 at 5:32 AM, Axel Jantsch <axe...@tu...> wrote:
> So somehow I use the wrong username or password. I suspect the username
> because the error message says
>
>>> fetchmail: Authorization failure on ax...@we...
>
> but my email address and account is really ax...@kt...
>
> But I do not know how I could use that name because then I get the error
> for ax...@kt...@webmail.kth.se
>
> Maybe I should use some Exchange specific format like
> ug.kth.se\axel  ???
>
> I tried that but it still appends @webmail.kth.se with the corresponding
> error message.
>
> Is there a way to have fetchmail use "ax...@kt..." ?

>     > https://www.kth.se/en/student/kth-it-support/work-online/email/settings-and-config/installningar-for-thunderbird-1.282051
>     > https://www.kth.se/en/student/kth-it-support/work-online/email/settings-and-config/imap-installningar-for-kth-e-post-1.3887
>     >> fetchmail: IMAP< * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN

First, test this by reading the RFC spec for IMAP4 auth, which people
have posted links to or search, then login to the mail server
manually using openssl / gnutls command, to confirm your
username and password are exactly what you think they are.

openssl s_client -connect server:port [-starttls imap]

>     >> fetchmail: IMAP> A0002 LOGIN "axel" *
>     >> fetchmail: IMAP< A0002 NO LOGIN failed.

It is extremely unlikely these days that your usename is not
required by the server to be qualified with the @fqdn. Fix that and retry.


>     >> my .fetchmailrc file looks like this:
>     >>
>     >> poll webmail.kth.se protocol imap
>     >> port 993
>     >> authenticate password
>     >> user 'axel'
>     >> password 'topsecret'
>     >> #     ssl sslproto 'auto'
>     >> ssl sslproto tls1 sslcertck
>     >> mda "/usr/bin/procmail -f sendmail"

Break all these multiple config statements into one per line.
'user' is 'username' then lines up nicely with 'password'.
'poll' is nothing more than a nop config stanza label string,
when you add a proper 'via' statement. I've talked / ticketed
about this regarding fetchmail-7 full config flexibility.

Today, you can have nonideal part of it with 6, and I do have,
 poll blah2
 via <fqdn9>
 username <user3@fqdn7>
and then fetchmail "blah2".

I say all this because your unqualified 'axel' username is being appended
by silly 'poll' mashup of concept to submit LOGIN 'axel@poll'.

I'm guessing kth changed to require @fqdn to support virtuals,
so you would have to test and change your config to match.

>     >> My email address there is ax...@kt..., but the IMAP server is webmail.kth.se

[Fetchmail-users] 6.4.0.beta2

[Chris <cpo...@em...>]

Matthias, thought I'd drop by and let you know I'm still having no
problems at all with 6.4.0.beta2. I also now have a log rotate working
for my fetchmail log, finally.

Chris

-- 
Chris
KeyID 0xE372A7DA98E6705C
31.11972; -97.90167 (Elev. 1092 ft)
09:10:31 up 7 days, 1:08, 1 user, load average: 0.47, 0.26, 0.27
Ubuntu 16.04.1 LTS, kernel 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18
14:10:15 UTC 2017

Re: [Fetchmail-users] IMAP< A0002 NO LOGIN failed

[Axel J. <axe...@tu...>]

Hi Matthias,

Many thanks for analyzing this problem so carefully.

So somehow I use the wrong username or password. I suspect the username
because the error message says

>> fetchmail: Authorization failure on ax...@we...

but my email address and account is really ax...@kt...

But I do not know how I could use that name because then I get the error
for ax...@kt...@webmail.kth.se

Maybe I should use some Exchange specific format like

ug.kth.se\axel  ???

I tried that but it still appends @webmail.kth.se with the corresponding
error message.

Is there a way to have fetchmail use "ax...@kt..." ?

Best,
Axel



>>>>> On Fri, 10 Feb 2017 10:04:29 +0100, Matthias Andree <mat...@gm...> said:

    > Am 09.02.2017 um 18:09 schrieb Axel Jantsch:
    >> Hi,
    >> 
    >> I am struggling to fetch my email from an exchange IMAP server. It used
    >> to work, but since a few weeks ago it does not anymore.
    > Hi Axel,

    > All it's saying is that the server doesn't accept your login, but the
    > server does not reveal why (which is normal behaviour in such situations).

    > Either you have mistyped things after a password change (or confused
    > certain similarly-looking letters, or case, such as 1 l I 0 O), or one
    > of the topsecret characters is the ' (in which case you could use "..."
    > for quoting). Note that lower-case vs. capitalization usually _does_
    > matter in passwords and often also in user names.

    > Have you changed anything on your end? If yes, double-check those
    > changes, especially user and password. Have you changed your password
    > somewhere for the KTH access, then you may need to change the password
    > in your .fetchmailrc as well.

    > If you still cannot figure it out, and the problem persists, ask the
    > helpdesk/support/whatever the KTH offers for assistance and +++
    > important +++ do tell them you're trying IMAP so they look at the right
    > places in their logging and when reproducing it.

    > The configuration snippet you're showing is in line with
    > https://www.kth.se/en/student/kth-it-support/work-online/email/settings-and-config/installningar-for-thunderbird-1.282051
    > and
    > https://www.kth.se/en/student/kth-it-support/work-online/email/settings-and-config/imap-installningar-for-kth-e-post-1.3887
    > but of course I cannot check the username or password for you, and
    > sometimes the backend password servers might have hiccups, it's
    > Microsoft Exchange and probably a Microsoft Active Directory underneath
    > after all, and things get wedged once in a while.

    >> When running fetchmail -v I get
    >> 
    >> fetchmail: SSL/TLS: using protocol TLSv1, cipher ECDHE-RSA-AES256-SHA, 256/256 secret/processed bits
    >> fetchmail: IMAP< * OK The Microsoft Exchange IMAP4 service is ready.
    >> fetchmail: IMAP> A0001 CAPABILITY
    >> fetchmail: IMAP< * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN UIDPLUS ID CHILDREN IDLE NAMESPACE LITERAL+
    >> fetchmail: IMAP< A0001 OK CAPABILITY completed.
    >> fetchmail: IMAP> A0002 LOGIN "axel" *
    >> fetchmail: IMAP< A0002 NO LOGIN failed.
    >> fetchmail: Authorization failure on ax...@we...
    >> fetchmail: IMAP> A0003 LOGOUT
    >> fetchmail: IMAP< * BYE Microsoft Exchange Server 2013 IMAP4 server signing off.
    >> fetchmail: IMAP< A0003 OK LOGOUT completed.
    >> fetchmail: 6.3.26 querying webmail.kth.se (protocol IMAP) at Don 09 Feb 2017 18:02:53 CET: poll completed
    >> fetchmail: Query status=3 (AUTHFAIL)
    >> fetchmail: normal termination, status 3
    >> 
    >> my .fetchmailrc file looks like this:
    >> 
    >> poll webmail.kth.se protocol imap
    >> port 993
    >> authenticate password
    >> user 'axel'
    >> password 'topsecret'
    >> #     ssl sslproto 'auto'
    >> ssl sslproto tls1 sslcertck
    >> mda "/usr/bin/procmail -f sendmail"
    >> 
    >> 
    >> My email address there is ax...@kt..., but the IMAP server is webmail.kth.se
    >> 
    >> I cannot figure out what exactly the problem is.
    >> Any idea? anybody?

    > Give them a head start of a few hours during normal office hours, and if
    > the problem persists, ask the KTH IT support (don't wait too long
    > though, on a Friday ;-))

    > HTH.

    > Best regards,
    > Matthias


    > ------------------------------------------------------------------------------
    > Check out the vibrant tech community on one of the world's most
    > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
    > _______________________________________________
    > Fetchmail-users mailing list
    > Fet...@li...
    > https://lists.sourceforge.net/lists/listinfo/fetchmail-users


-- 
Axel Jantsch, Prof. of Systems on Chip
TU Wien
Gusshausstrasse 27-29/384, 1040 Vienna, Austria
Phone: +43 1 58801 38415
Email: axe...@tu...
Web: www.ict.tuwien.ac.at

Re: [Fetchmail-users] IMAP< A0002 NO LOGIN failed

[Matthias A. <mat...@gm...>]

Am 09.02.2017 um 18:09 schrieb Axel Jantsch:
> Hi,
>
> I am struggling to fetch my email from an exchange IMAP server. It used
> to work, but since a few weeks ago it does not anymore.
Hi Axel,

All it's saying is that the server doesn't accept your login, but the
server does not reveal why (which is normal behaviour in such situations).

Either you have mistyped things after a password change (or confused
certain similarly-looking letters, or case, such as 1 l I 0 O), or one
of the topsecret characters is the ' (in which case you could use "..."
for quoting). Note that lower-case vs. capitalization usually _does_
matter in passwords and often also in user names.

Have you changed anything on your end? If yes, double-check those
changes, especially user and password. Have you changed your password
somewhere for the KTH access, then you may need to change the password
in your .fetchmailrc as well.

If you still cannot figure it out, and the problem persists, ask the
helpdesk/support/whatever the KTH offers for assistance and +++
important +++ do tell them you're trying IMAP so they look at the right
places in their logging and when reproducing it.

The configuration snippet you're showing is in line with
https://www.kth.se/en/student/kth-it-support/work-online/email/settings-and-config/installningar-for-thunderbird-1.282051
and
https://www.kth.se/en/student/kth-it-support/work-online/email/settings-and-config/imap-installningar-for-kth-e-post-1.3887
but of course I cannot check the username or password for you, and
sometimes the backend password servers might have hiccups, it's
Microsoft Exchange and probably a Microsoft Active Directory underneath
after all, and things get wedged once in a while.

> When running fetchmail -v I get
>
> fetchmail: SSL/TLS: using protocol TLSv1, cipher ECDHE-RSA-AES256-SHA, 256/256 secret/processed bits
> fetchmail: IMAP< * OK The Microsoft Exchange IMAP4 service is ready.
> fetchmail: IMAP> A0001 CAPABILITY
> fetchmail: IMAP< * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN UIDPLUS ID CHILDREN IDLE NAMESPACE LITERAL+
> fetchmail: IMAP< A0001 OK CAPABILITY completed.
> fetchmail: IMAP> A0002 LOGIN "axel" *
> fetchmail: IMAP< A0002 NO LOGIN failed.
> fetchmail: Authorization failure on ax...@we...
> fetchmail: IMAP> A0003 LOGOUT
> fetchmail: IMAP< * BYE Microsoft Exchange Server 2013 IMAP4 server signing off.
> fetchmail: IMAP< A0003 OK LOGOUT completed.
> fetchmail: 6.3.26 querying webmail.kth.se (protocol IMAP) at Don 09 Feb 2017 18:02:53 CET: poll completed
> fetchmail: Query status=3 (AUTHFAIL)
> fetchmail: normal termination, status 3
>
> my .fetchmailrc file looks like this:
>
> poll webmail.kth.se protocol imap
>      port 993
>      authenticate password
>      user 'axel'
>      password 'topsecret'
> #     ssl sslproto 'auto'
>      ssl sslproto tls1 sslcertck
>      mda "/usr/bin/procmail -f sendmail"
>
>
> My email address there is ax...@kt..., but the IMAP server is webmail.kth.se
>
> I cannot figure out what exactly the problem is.
> Any idea? anybody?

Give them a head start of a few hours during normal office hours, and if
the problem persists, ask the KTH IT support (don't wait too long
though, on a Friday ;-))

HTH.

Best regards,
Matthias

[Fetchmail-users] IMAP< A0002 NO LOGIN failed

[Axel J. <axe...@tu...>]

Hi,

I am struggling to fetch my email from an exchange IMAP server. It used
to work, but since a few weeks ago it does not anymore.

When running fetchmail -v I get

fetchmail: SSL/TLS: using protocol TLSv1, cipher ECDHE-RSA-AES256-SHA, 256/256 secret/processed bits
fetchmail: IMAP< * OK The Microsoft Exchange IMAP4 service is ready.
fetchmail: IMAP> A0001 CAPABILITY
fetchmail: IMAP< * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN UIDPLUS ID CHILDREN IDLE NAMESPACE LITERAL+
fetchmail: IMAP< A0001 OK CAPABILITY completed.
fetchmail: IMAP> A0002 LOGIN "axel" *
fetchmail: IMAP< A0002 NO LOGIN failed.
fetchmail: Authorization failure on ax...@we...
fetchmail: IMAP> A0003 LOGOUT
fetchmail: IMAP< * BYE Microsoft Exchange Server 2013 IMAP4 server signing off.
fetchmail: IMAP< A0003 OK LOGOUT completed.
fetchmail: 6.3.26 querying webmail.kth.se (protocol IMAP) at Don 09 Feb 2017 18:02:53 CET: poll completed
fetchmail: Query status=3 (AUTHFAIL)
fetchmail: normal termination, status 3

my .fetchmailrc file looks like this:

poll webmail.kth.se protocol imap
     port 993
     authenticate password
     user 'axel'
     password 'topsecret'
#     ssl sslproto 'auto'
     ssl sslproto tls1 sslcertck
     mda "/usr/bin/procmail -f sendmail"


My email address there is ax...@kt..., but the IMAP server is webmail.kth.se

I cannot figure out what exactly the problem is.
Any idea? anybody?

Many thanks!
--Axel

Re: [Fetchmail-users] delay interval

[Carlos E. R. <rob...@te...>]

On 2017-01-08 13:41, Matthias Andree wrote:
> Am 06.01.2017 um 18:00 schrieb piequiex:
>>> El 2017-01-04 a las 17:24 -0000, piequiex escribi??:


>>> Probably your script is exiting too early, before it finishes processing.
>> Yes. External application, executed from a script, sometimes terminates with
>> the strange error - "Cannot allocate memory". This does not happen in example
>> above.
> Getting error handling right in procmail requires jumping through hoops
> and adding an error trapping recipe after each and every action that you
> code, that's why I'm advocating replacing procmail.

Yes, procmail is definitely not easy to trace, but I use log entries to
help. After years of familiarity and that my setup "simply" works, one
finds no need to invest many days in changing it ;-)

Eventually I may have to change, but meanwhile...

> While you're writing in a later message that you've found the cause for
> the allocation errors, I still believe that the error non-handling in
> your procmailrc setup persists. Reason why I think that: Fetchmail
> 6.3.26 in default configuration will leave mail on the server and retry
> it later (--softbounce is default), but for that to happen it needs to
> know that the --mda failed, either by it crashing, or from its nonzero
> exit status. Since you wrote about mail loss, I believe that procmail
> hasn't propagated that script failing status back to fetchmail.

I have postfix in the mix, so mail is queued inside postfix. It helps.

One trick is that I make procmail to copy every mail to a backup folder.
Thus If something bad happens later on, the posts are not lost. So far I
had no need.



-- 
Cheers / Saludos,

		Carlos E. R.
		(from 42.2 x86_64 "Malachite" at Telcontar)

Re: [Fetchmail-users] delay interval

[Matthias A. <mat...@gm...>]

Am 06.01.2017 um 18:00 schrieb piequiex:
>> El 2017-01-04 a las 17:24 -0000, piequiex escribi??:
>>
>>> The problem is I loss all messages, except first two. They all must be parsed
>>> with script, which I execute from procmailrc.
>>>
>>> I have discovered that all messages will delivered in case when delay
>>> interval present in loop. For example:
>>>
>>> for msg in file.???; do
>>> cat $msg | script;
>>> sleep 1;
>>> done
>> Probably your script is exiting too early, before it finishes processing.
> Yes. External application, executed from a script, sometimes terminates with
> the strange error - "Cannot allocate memory". This does not happen in example
> above.
Getting error handling right in procmail requires jumping through hoops
and adding an error trapping recipe after each and every action that you
code, that's why I'm advocating replacing procmail.

While you're writing in a later message that you've found the cause for
the allocation errors, I still believe that the error non-handling in
your procmailrc setup persists. Reason why I think that: Fetchmail
6.3.26 in default configuration will leave mail on the server and retry
it later (--softbounce is default), but for that to happen it needs to
know that the --mda failed, either by it crashing, or from its nonzero
exit status. Since you wrote about mail loss, I believe that procmail
hasn't propagated that script failing status back to fetchmail.

Re: [Fetchmail-users] delay interval

[piequiex <pie...@ny...>]

> > El 2017-01-04 a las 17:24 -0000, piequiex escribi??:
> > 
> > > The problem is I loss all messages, except first two. They all must be parsed
> > > with script, which I execute from procmailrc.

Finally I have find out what is going wrong.
Wrong environment value.

-- 
0x16E684E1A170D8A3

Re: [Fetchmail-users] delay interval

[piequiex <pie...@ny...>]

> El 2017-01-04 a las 17:24 -0000, piequiex escribi??:
> 
> > The problem is I loss all messages, except first two. They all must be parsed
> > with script, which I execute from procmailrc.
> > 
> > I have discovered that all messages will delivered in case when delay
> > interval present in loop. For example:
> > 
> > for msg in file.???; do
> > cat $msg | script;
> > sleep 1;
> > done
> 
> Probably your script is exiting too early, before it finishes processing.

Yes. External application, executed from a script, sometimes terminates with
the strange error - "Cannot allocate memory". This does not happen in example
above.

> Depending on how you do things, procmail can be called in parallel by
> several emails, so you either change your script to be able to cope with
> that situation, or tell the other software to only send one.
> 
> For example, if you are using postfix in the mix, you can tell it to process
> one email at a time for local delivery (slowing things down a lot).
> 
> You can also use lock files in procmail so that a section blocks if a
> parallel process is already in there.

Thanks for advices.

-- 
0x16E684E1A170D8A3

Re: [Fetchmail-users] delay interval

[Gene H. <ghe...@sh...>]

On Thursday 05 January 2017 22:57:46 grarpamp wrote:

> On Thu, Jan 5, 2017 at 9:45 PM, Gene Heskett <ghe...@sh...> 
wrote:
> >> https://tracker.debian.org/pkg/maildrop
>
> Another portal...
>
> https://packages.debian.org/search?keywords=maildrop&searchon=names&ex
>act=1&suite=all&section=all
>
> > One thing I have wanted to do for several years was to setup an imap
> > repo an R-Pi
> > so, could this maildrop do a direct deposit
>
> Yes. You probably want to use Maildir's, they're a bit more reliable
> than mbox. More common tools below. Use whatever you find that
> works for you after fetchmail's mda option hands it off.
>
> dovecot.org, mailpile.is, neomutt.org
> www.courier-mta.org/maildrop/
>
> > how would I go about
> > pulling in a 14 year old collection of msgs totalling around 20GB
> > that kmail now maintains into the imap database.
>
> Make backups, test them, store them offline.
>
> Read about if there's some way to export the database to Maildir.

I run amanda, to hd storage using vtapes.  So backups I have, although 
the storage "depth" is only the last 30 days as the drive space is 
recycled after the 30th day, one days vtape at a time.

And because the mailfile is much slower to access than the maildir, I 
converted it all to maildir years ago.  So thats not a problem, but 
thanks for the reminder.

> ----------------------------------------------------------------------
>-------- Check out the vibrant tech community on one of the world's
> most engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Fetchmail-users mailing list
> Fet...@li...
> https://lists.sourceforge.net/lists/listinfo/fetchmail-users


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: [Fetchmail-users] delay interval

[Carlos E. R. <rob...@te...>]

On 2017-01-06 04:44, Gene Heskett wrote:
> On Thursday 05 January 2017 22:12:39 Carlos E. R. wrote:


>> Why do I have the feeling that you are angry at me? Why would that be,
>> have I done something to you?
> 
> The attitude that procmail is gone and "good riddance", when its been 
> working rather well here for yonks once I had grokked how to write a 
> recipe that worked, is what pulled my trigger.

Well, but you see, I use procmail, I am happy with it, and I hope to
keep using it for years to come, till forced otherwise ;-)

Yes, I am aware that it has got no maintenance upstream, and that it is
not perfect, but I find the replacements more problematic.

https://lwn.net/Articles/416901/

> I do apologize for biting back as you did stir the pot and maildrop was 
> brought to my attention, for that I thank you.
> 
> Take care Carlos.

Ok!

> 
> Cheers, Gene Heskett
> 


-- 
Cheers / Saludos,

		Carlos E. R.
		(from 42.2 x86_64 "Malachite" at Telcontar)

Re: [Fetchmail-users] delay interval

[grarpamp <gra...@gm...>]

On Thu, Jan 5, 2017 at 9:45 PM, Gene Heskett <ghe...@sh...> wrote:
>> https://tracker.debian.org/pkg/maildrop

Another portal...

https://packages.debian.org/search?keywords=maildrop&searchon=names&exact=1&suite=all&section=all

> One thing I have wanted to do for several years was to setup an imap repo
> an R-Pi
> so, could this maildrop do a direct deposit

Yes. You probably want to use Maildir's, they're a bit more reliable
than mbox. More common tools below. Use whatever you find that
works for you after fetchmail's mda option hands it off.

dovecot.org, mailpile.is, neomutt.org
www.courier-mta.org/maildrop/

> how would I go about
> pulling in a 14 year old collection of msgs totalling around 20GB that
> kmail now maintains into the imap database.

Make backups, test them, store them offline.

Read about if there's some way to export the database to Maildir.

Re: [Fetchmail-users] delay interval

[Gene H. <ghe...@sh...>]

On Thursday 05 January 2017 22:12:39 Carlos E. R. wrote:

> On 2017-01-06 00:15, Gene Heskett wrote:
> > On Thursday 05 January 2017 16:56:11 Carlos E. R. wrote:
> >> Procmail is offtopic here, and you don't say what distribution you
> >> are using, but surely it has a help forum or mail list. I use
> >> openSUSE, you can find me there and I'll be happy to try to help.
> >> Otherwise, mail me directly, and time permitting, I'll try to help.
> >
> > Pray tell me why procmail is off-topic here?  It's been doing an
> > excellent job here as the mta for fetchmail for about 15 years.
>
> Perhaps you didn't notice that this is a fetchmail mail list?  :-?
> It is not for me to nor you decide what is offtopic here, anyway.
>
> > Before you go touting a new mta, you should make sure it is in all
> > the big name repositories, but its not in debian for wheezy. The
> > only competition is getmail, which is a competitor of fetchmail
> > itself.
> >
> > Methinks you are out of line from your bully pulpit.
>
> Why do I have the feeling that you are angry at me? Why would that be,
> have I done something to you?

The attitude that procmail is gone and "good riddance", when its been 
working rather well here for yonks once I had grokked how to write a 
recipe that worked, is what pulled my trigger.  The fact that its been 
declared dead, and last rights given 2 years ago is news to me as it was 
apparently not well publicized in the lists I sub to, currently about 
40.  It is also in the debian repo's I access, and no noises made about 
dropping it that I am aware of.  Unless the switchover can be done 
rather painlessly, I will likely use it until I miss roll call, an 
undetermined time in the future but looming as I am now 82 yo, diabetic, 
and a blue hang tag on the rear view mirror of my WV Cadillac, aka a 99 
GMC 4wd 3 door. Might come in handy as we've about 4" of new snow on 
warm roads that none of the local youngun's have ever learned how to 
drive on. And still falling slowly.

I do apologize for biting back as you did stir the pot and maildrop was 
brought to my attention, for that I thank you.

Take care Carlos.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: [Fetchmail-users] delay interval

[Carlos E. R. <rob...@te...>]

On 2017-01-06 00:15, Gene Heskett wrote:
> On Thursday 05 January 2017 16:56:11 Carlos E. R. wrote:
>
>>
>> Procmail is offtopic here, and you don't say what distribution you are
>> using, but surely it has a help forum or mail list. I use openSUSE,
>> you can find me there and I'll be happy to try to help. Otherwise,
>> mail me directly, and time permitting, I'll try to help.
>
> Pray tell me why procmail is off-topic here?  It's been doing an
> excellent job here as the mta for fetchmail for about 15 years.

Perhaps you didn't notice that this is a fetchmail mail list?  :-?
It is not for me to nor you decide what is offtopic here, anyway.


> Before you go touting a new mta, you should make sure it is in all the
> big name repositories, but its not in debian for wheezy. The only
> competition is getmail, which is a competitor of fetchmail itself.
>
> Methinks you are out of line from your bully pulpit.

Why do I have the feeling that you are angry at me? Why would that be, 
have I done something to you?

-- 
Cheers/Saludos

          Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor)

Re: [Fetchmail-users] delay interval

[Gene H. <ghe...@sh...>]

On Thursday 05 January 2017 19:02:58 Matthias Andree wrote:

> Am 06.01.2017 um 00:15 schrieb Gene Heskett:
> > On Thursday 05 January 2017 16:56:11 Carlos E. R. wrote:
> >> Procmail is offtopic here, and you don't say what distribution you
> >> are using, but surely it has a help forum or mail list. I use
> >> openSUSE, you can find me there and I'll be happy to try to help.
> >> Otherwise, mail me directly, and time permitting, I'll try to help.
> >
> > Pray tell me why procmail is off-topic here?  It's been doing an
> > excellent job here as the mta for fetchmail for about 15 years.
>
> Gene,
>
> a happy new year. The good news is that procmail is dead (good
> riddance), and isn’t a part of fetchmail. Procmail got deorbited by
> its authors, after more than a decade of being abandonware.
> It’s always been next to impossible to configure in a way to NOT goof
> up in adverse conditions (such as full disks, loaded systems,
> thereabouts).
>
> The old news is that procmail configuration isn’t subject of this
> list. Procmail used to have its own lists, and while questions of “how
> do I execute procmail from fetchmail’s --mda option” are on topic
> here, the actual procmail configuration is not.
>
> > Before you go touting a new mta, you should make sure it is in all
> > the big name repositories, but its not in debian for wheezy. The
> > only competition is getmail, which is a competitor of fetchmail
> > itself.
>
> https://tracker.debian.org/pkg/maildrop looks plausible for Debian,
> and maildrop has been more mature and usable than procmail ever was.
>
Humm, interestink.  A search in synaptic just before I posted that 
message did not find it.  After chasing thru the links on the above page 
and finally find it as a .deb in the wheezy main repo, I went back to 
synaptic after checking my sources.list entries, making no changes 
because the correct repo was already there and reloaded again, and lo, 
behold even, the search function spit it right out.  So now it and a 
couple dependencies are installed so I can at least study its man pages 
and how to tie it into fetchmail as the default mta.

One thing I have wanted to do for several years was to setup an imap repo 
on this machine that fetchmails mta could deposit the incoming mail 
into, and all kmails on this private network could reach into so that I 
could access it from any of the 6 machines behind dd-wrt here, including 
an R-Pi soon to be running a cnc lathe here.  From the name of one of 
the dependency's pulled in, I am wondering it that is possible, and if 
so, could this maildrop do a direct deposit, and how would I go about 
pulling in a 14 year old collection of msgs totalling around 20GB that 
kmail now maintains into the imap database.  But that part might be more 
apropo to be asked on the tde mailing list since I am running tde. Kde 
has wandered off into the desert and no longer suits my tastes, nor 
stability needs. The simplest way would be to just turn this mail corpus 
into an imap served by kmail to the rest of the house.  I'll have to ask 
a few questions I believe.

Thank you for bringing it to my attention.

> ----------------------------------------------------------------------
>-------- Check out the vibrant tech community on one of the world's
> most engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Fetchmail-users mailing list
> Fet...@li...
> https://lists.sourceforge.net/lists/listinfo/fetchmail-users


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: [Fetchmail-users] delay interval

[Matthias A. <mat...@gm...>]

Am 06.01.2017 um 00:15 schrieb Gene Heskett:
> On Thursday 05 January 2017 16:56:11 Carlos E. R. wrote:
>
>> Procmail is offtopic here, and you don't say what distribution you are
>> using, but surely it has a help forum or mail list. I use openSUSE,
>> you can find me there and I'll be happy to try to help. Otherwise,
>> mail me directly, and time permitting, I'll try to help.
> Pray tell me why procmail is off-topic here?  It's been doing an 
> excellent job here as the mta for fetchmail for about 15 years.

Gene,

a happy new year. The good news is that procmail is dead (good
riddance), and isn’t a part of fetchmail. Procmail got deorbited by its
authors, after more than a decade of being abandonware.
It’s always been next to impossible to configure in a way to NOT goof up
in adverse conditions (such as full disks, loaded systems, thereabouts).

The old news is that procmail configuration isn’t subject of this list.
Procmail used to have its own lists, and while questions of “how do I
execute procmail from fetchmail’s --mda option” are on topic here, the
actual procmail configuration is not.

> Before you go touting a new mta, you should make sure it is in all the 
> big name repositories, but its not in debian for wheezy. The only 
> competition is getmail, which is a competitor of fetchmail itself.

https://tracker.debian.org/pkg/maildrop looks plausible for Debian, and
maildrop has been more mature and usable than procmail ever was.

Re: [Fetchmail-users] delay interval

[Gene H. <ghe...@sh...>]

On Thursday 05 January 2017 16:56:11 Carlos E. R. wrote:

>
> Procmail is offtopic here, and you don't say what distribution you are
> using, but surely it has a help forum or mail list. I use openSUSE,
> you can find me there and I'll be happy to try to help. Otherwise,
> mail me directly, and time permitting, I'll try to help.

Pray tell me why procmail is off-topic here?  It's been doing an 
excellent job here as the mta for fetchmail for about 15 years.

Before you go touting a new mta, you should make sure it is in all the 
big name repositories, but its not in debian for wheezy. The only 
competition is getmail, which is a competitor of fetchmail itself.

Methinks you are out of line from your bully pulpit.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: [Fetchmail-users] delay interval

[Carlos E. R. <rob...@te...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



El 2017-01-04 a las 17:24 -0000, piequiex escribió:

> The problem is I loss all messages, except first two. They all must be parsed
> with script, which I execute from procmailrc.
>
> I have discovered that all messages will delivered in case when delay
> interval present in loop. For example:
>
> for msg in file.???; do
> cat $msg | script;
> sleep 1;
> done

Probably your script is exiting too early, before it finishes 
processing.

Depending on how you do things, procmail can be called in parallel by 
several emails, so you either change your script to be able to cope with 
that situation, or tell the other software to only send one.

For example, if you are using postfix in the mix, you can tell it to 
process one email at a time for local delivery (slowing things down a 
lot).

You can also use lock files in procmail so that a section blocks if a 
parallel process is already in there.


Procmail is offtopic here, and you don't say what distribution you are 
using, but surely it has a help forum or mail list. I use openSUSE, you 
can find me there and I'll be happy to try to help. Otherwise, mail me 
directly, and time permitting, I'll try to help.


- -- 
Cheers
        Carlos E. R.

        (from 13.1 x86_64 "Bottle" (Minas Tirith))
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iF4EAREIAAYFAlhuwQQACgkQja8UbcUWM1ypHgD8C6ef8G4HACmIfjZm+rPHPj6K
RgeHPDDSE0ijupeSbyUA/2QcceAc3PSFv2OLmoMXRpdZAWThKHiWVihQOAah/oRb
=mR5g
-----END PGP SIGNATURE-----

Re: [Fetchmail-users] delay interval

[Matthias A. <mat...@gm...>]

Am 04.01.2017 um 18:24 schrieb piequiex:
>> Am 03.01.2017 um 18:00 schrieb piequiex:
>>> Hello.
>>>
>>> Is it possible to set delay interval after every received message? 1-2
>>> seconds.
>>>
>> Hi,
>> no there isn't unless you want to change code or are using the --mda option.
> I have use --mda.
>
>> However, this should never be necessary. Why do you try to do that?
> The problem is I loss all messages, except first two. They all must be parsed
> with script, which I execute from procmailrc.

Then your script or the corresponding .procmailrc is faulty, or procmail
is, or your upstream server is, and it appears something's
asynchronously messing up things for you.

I don't see how you could find a delay interval that would *guarantee*
your setup to not lose mail. Imagine your computer running fetchmail
gets under load or swap pressure and the given delay that worked "in
good weather" is insufficient "in bad weather".

fetchmail waits until the command --mda has completed and obtains its
exit code (or termination signal) through pclose(), before fetching the
next message. You must make sure your script does not terminate
prematurely without losing its exit status, and that procmail properly
processes your script's exit status and conveys it back to fetchmail.
I know this is hard to get really robust with procmail, which is why
I've replaced it more than a decade ago, and warnings against using
procmail have been in fetchmail's manual for years.
(Error-handling endeavours get the typical non-trivial procmailrc on the
brink of becoming unreadable because you need to code all error-trapping
yourself.)

I am not going to help with debugging your procmailrc though, procmail
was abandoned more than 15 years and was retired more than 2 years
ago[1], and its website has disappeared.
I suggest that you debug your script and rigging and plan to migrate to
maildrop or another filtering solution that is still maintained and
supported.

Questions on how to use maildrop with fetchmail's --mda option can be
asked here, for maildrop itself, it has lists on its own.

[1] <https://marc.info/?l=openbsd-ports&m=141634350915839&w=2>

Sorry, this isn't terribly helpful on the short term, but I don't think
that changes to fetchmail's setup would make your script setup robust.
You really need to debug it and prepare yourself to exchange procmail in
the long run.