courier-cone

[cone] Automatic Refresh for local Maildir's mail counts?

[Linux-Fan <Ma_...@we...>]

Dear Cone users,

In [1] I asked if it might be possible to add a means of filtering
accross account boundaries to Cone in order to be able to replicate
my mail setup with it. I have understood it does not work that way
because moving across accounts is conceptually different from moving
inside the same account.

Still, I have not given up on trying to configure Cone in a way I could
use it as my mail client: While I did not try to amend Cone's filtering
system to work across server boundaries, I have thought of another
solution to the problem: In order to have filtering across account
boundaries, I could use an external program to process incoming e-mails
and then only read (and write) them in Cone.

The setup is intended to look something like this

  Offlineimap               Filtering tool¹        Cone
  ---------------------     ------------------     ---------------------
  Downloads mails and   --> Moves mails inside --> Is used to read from
  uploads changes to an     a local Maildir        the local Maildir and
  IMAP Server               structure              send via SMTP.

Now in order for this to work as I expect it, Cone needs to
automatically update the e-mail counts from local Maildirs to display
the changes made externally via Offlineimap and the filtering tool.

As far as I understand, there is an option called `/noop=N` [2] to
poll IMAP accounts. I tried to use that (testing with an IMAP account),
but it did not automatically tell me `... unread` after receiving a new
mail until I opened and closed the folder again?

Alternatively, I have tried to implement a refresh via a `SIGUSR1` to
Cone by collapsing and expanding the currently selected subtree. This is
a hack and while it does something similar to what I want (if the
correct entry is selected, it refreshes the mail counters), it does not
do it right: It depends on the current selection and it marks all
folders (as if they were highlighted by the cursor) for reasons unknown
to me :)

Now my question is: Is there a recommended way to achieve automatic
updating of (new-)mail counters for offline accounts / e-mail data
processed by a separate tool? If not, is there a chance of getting the
`/noop=N` thing to work with local Maildirs making it update
e-mail counts without interaction or is there a chance of implementing a
`SIGUSR1` to refresh in a more sensible way than I did?

In order to illustrate what I am thinking about, I have attached my
patch to enhance cone-0.92 with a hacky `SIGUSR1` support. If you think
it might be a good idea to rewrite the patch to apply to the newest
Cone version, I will try to do that. However, I do not think that
my hack is the way to go with respect to the issue I want to solve.

¹) I do not know which filtering tool to use, yet.

TIA and Yours Sincerely
Linux-Fan

[1] https://sourceforge.net/p/courier/mailman/message/35086282/
[2] http://www.courier-mta.org/cone/cone06newaccount.html

-- 
http://masysma.lima-city.de/

Re: [cone] Automatic Refresh for local Maildir's mail counts?

[Sam V. <mr...@co...>]

Linux-Fan writes:

> Now in order for this to work as I expect it, Cone needs to
> automatically update the e-mail counts from local Maildirs to display
> the changes made externally via Offlineimap and the filtering tool.

This should happen automatically already.

If you have a local maildir open, a new message delivered there should  
result in the display immediately updated (well, within a few seconds).

> As far as I understand, there is an option called `/noop=N` [2] to
> poll IMAP accounts. I tried to use that (testing with an IMAP account),

IMAP and local maildirs have nothing to do with each other.

If the IMAP server supports IDLE, and an IMAP folder is open, a new message  
in the IMAP folder will also result in an immediate refresh.

If you do not have a folder open, but looking at the folder hierarchy,  
the message counts are not going to get updated in realtime. There's no  
facility to do that.

Re: [cone] Automatic Refresh for local Maildir's mail counts?

[Linux-Fan <Ma_...@we...>]

[Sat, 31 Dec 2016 10:23:26 -0500] Sam Varshavchik
<mr...@co...> wrote:
> Linux-Fan writes:
> 
> > Now in order for this to work as I expect it, Cone needs to
> > automatically update the e-mail counts from local Maildirs to
> > display the changes made externally via Offlineimap and the
> > filtering tool.
> 
> This should happen automatically already.
> 
> If you have a local maildir open, a new message delivered there
> should result in the display immediately updated (well, within a few
> seconds).
> 
> > As far as I understand, there is an option called `/noop=N` [2] to
> > poll IMAP accounts. I tried to use that (testing with an IMAP
> > account),
> 
> IMAP and local maildirs have nothing to do with each other.
> 
> If the IMAP server supports IDLE, and an IMAP folder is open, a new
> message in the IMAP folder will also result in an immediate refresh.

Just for the record: I have tested it with viewing the folder contents
in Cone and it works just as described above for IMAP and local
Maildirs.

> If you do not have a folder open, but looking at the folder
> hierarchy, the message counts are not going to get updated in
> realtime. There's no facility to do that.

Ah OK, that's it: I misunderstood the meaning of ``open'' thinking it
was enough to view the folder hierarchy instead of looking at the actual
folder contents. The updates for the folder currently opened in a sense
that the contents are displayed work well (see above).

Is it possible/planned to add a means of updating the message counts
from the folder hierarchy view automatically as well (similar to what
I did upon receiving a SIGUSR1 for my proof-of-concept patch from the
previous mail)?

TIA and Yours Sincerely
Linux-Fan

-- 
http://masysma.lima-city.de/

Re: [cone] Automatic Refresh for local Maildir's mail counts?

[Sam V. <mr...@co...>]

Linux-Fan writes:

> > If you do not have a folder open, but looking at the folder
> > hierarchy, the message counts are not going to get updated in
> > realtime. There's no facility to do that.
>
> Ah OK, that's it: I misunderstood the meaning of ``open'' thinking it
> was enough to view the folder hierarchy instead of looking at the actual
> folder contents. The updates for the folder currently opened in a sense
> that the contents are displayed work well (see above).
>
> Is it possible/planned to add a means of updating the message counts
> from the folder hierarchy view automatically as well (similar to what
> I did upon receiving a SIGUSR1 for my proof-of-concept patch from the
> previous mail)?

I'm open to such a feature; however this approach won't work. Signals are  
asynchronous, and none of the high levels function that the handler invokes  
are async-safe. If cone is doing nothing but polling for the next event,  
it'll work fine. If cone is in the middle of something, there's a high  
likelyhood of a crash. And if cone is not displaying the folder hierarchy,  
this is pretty much a guaranteed crash.

The only safe way to do this: in addition to installing a signal handler, an  
internal pipe needs to be created, and put into nonblocking mode. The only  
thing the signal handler does is write a byte to the pipe, ignoring the  
results, and immediately terminating.

Then, on the folder screen only, in addition to handling all the regular  
events the read end of the pipe gets polled as well, and when something is  
read, what your patch does would happen.

Even with that, things may not work 100%. All the code needs to be audited  
and verified that it'll correctly handle an interrupted system call.

Correct signal handling is hard.

Re: [cone] Automatic Refresh for local Maildir's mail counts?

[Linux-Fan <Ma_...@we...>]

[Sat, 31 Dec 2016 12:27:38 -0500] Sam Varshavchik
<mr...@co...> wrote:
> Linux-Fan writes:
> 
> > > If you do not have a folder open, but looking at the folder
> > > hierarchy, the message counts are not going to get updated in
> > > realtime. There's no facility to do that.

[...]

> > Is it possible/planned to add a means of updating the message counts
> > from the folder hierarchy view automatically as well (similar to
> > what I did upon receiving a SIGUSR1 for my proof-of-concept patch
> > from the previous mail)?
> 
> I'm open to such a feature; however this approach won't work. Signals

[...]

> The only safe way to do this: in addition to installing a signal
> handler, an internal pipe needs to be created, and put into
> nonblocking mode. The only thing the signal handler does is write a
> byte to the pipe, ignoring the results, and immediately terminating.
> 
> Then, on the folder screen only, in addition to handling all the
> regular events the read end of the pipe gets polled as well, and when
> something is read, what your patch does would happen.

OK. I am going to check this out when I find time to dig into the
source code again :) How about the other option: Just extending the (as
far as I understand it) existing event polling to also check for new
messages every NN seconds or such? I do not like polling that much but
in this case it sounds easier and probably more reliable?

> Even with that, things may not work 100%. All the code needs to be
> audited and verified that it'll correctly handle an interrupted
> system call.
>
> Correct signal handling is hard.

Yes it is :( Especially signal handling from within multithreaded
applications is probably asking for trouble, so I understand that might
not be the best solution here...

Thank you for the quick and thorough reply, I really appreciate it.

Happy new year to all list members :) !

Yours Sincerely
Linux-Fan

-- 
http://masysma.lima-city.de/

Re: [cone] Automatic Refresh for local Maildir's mail counts

[Linux-Fan <Ma_...@we...>]

[Sun, 1 Jan 2017 23:15:02 +0100] Linux-Fan <Ma_...@we...> wrote:
> [Sat, 31 Dec 2016 12:27:38 -0500] Sam Varshavchik
> <mr...@co...> wrote:
> > Linux-Fan writes:

[...]

> > > Is it possible/planned to add a means of updating the message
> > > counts from the folder hierarchy view automatically as well
> > > (similar to what I did upon receiving a SIGUSR1 for my
> > > proof-of-concept patch from the previous mail)?
> > 
> > I'm open to such a feature; however this approach won't work.
> > Signals
> 
> [...]
> 
> > The only safe way to do this: in addition to installing a signal
> > handler, an internal pipe needs to be created, and put into
> > nonblocking mode. The only thing the signal handler does is write a
> > byte to the pipe, ignoring the results, and immediately terminating.
> > 
> > Then, on the folder screen only, in addition to handling all the
> > regular events the read end of the pipe gets polled as well, and
> > when something is read, what your patch does would happen.
> 
> OK. I am going to check this out when I find time to dig into the
> source code again :)

[...]

I have taken on the challenge and tried to create a (more) correct
patch which enables cone-0.92 to update all folder's message counts
upon receiving a SIGUSR1.

It is not using a pipe, but a C++11 std::atomic instead. If that cannot
be used, because Cone needs to compile without C++11 (I used
`./configure CPPFLAGS=-std=c++11` to get it to work), I'd replace this
by a variable which is marked `volatile sig_atomic_t`.

What do you think? Does it make sense to put it that way or have I
misplaced the call to `processHierharcyRefresh()`? Is it better to use
`SA_RESTART` or not? I am not quite sure...
Also, in order to improve the folder refreshing mechanism, I have
implemented a `RefreshIterator` which uses `updateInfo` on all folder
entries. Have I used the API correctly?

> > Even with that, things may not work 100%. All the code needs to be
> > audited and verified that it'll correctly handle an interrupted
> > system call.
> >
> > Correct signal handling is hard.  
> 
> Yes it is :( Especially signal handling from within multithreaded
> applications is probably asking for trouble, so I understand that
> might not be the best solution here...

If signal handling is still too much of an issue with regard to the
overall application stability, might it make sense to replace my check
for `pendingUpdates.exchange(0) > 0` with a simple timer
(like if now - lastTimeOfCheck > NN seconds refresh)?

By the way: If you wonder about the `masysma` prefixes and annotations:
I will happily remove these if the patch has a chance of being
incorporated but for now I have added them to easily navigate to my
modifications within the code and make sure I do not confuse my
changes with what was already there...

Yours Sincerely and TIA
Linux-Fan

-- 
http://masysma.lima-city.de/

Re: [cone] Automatic Refresh for local Maildir's mail counts

[Sam V. <mr...@co...>]

Linux-Fan writes:

> It is not using a pipe, but a C++11 std::atomic instead. If that cannot
> be used, because Cone needs to compile without C++11 (I used
> `./configure CPPFLAGS=-std=c++11` to get it to work), I'd replace this
> by a variable which is marked `volatile sig_atomic_t`.

Cone is still compiled on relatively old platforms. C++11 is not a viable  
option right now; but probably in a little while this would be an option.

I do not see a good reason not to use an internal pipe, in non-blocking  
mode, and with O_CLOEXEC set. sig_atomic_t has been around only since C99;  
but I suppose at this time requiring C99 would be acceptable.

> What do you think? Does it make sense to put it that way or have I
> misplaced the call to `processHierharcyRefresh()`? Is it better to use
> `SA_RESTART` or not? I am not quite sure...
> Also, in order to improve the folder refreshing mechanism, I have
> implemented a `RefreshIterator` which uses `updateInfo` on all folder
> entries. Have I used the API correctly?

I would have to look at this closer, to determine the right answer.

> By the way: If you wonder about the `masysma` prefixes and annotations:
> I will happily remove these if the patch has a chance of being
> incorporated but for now I have added them to easily navigate to my
> modifications within the code and make sure I do not confuse my
> changes with what was already there...

No big rush here. You can take your time and test your changes thoroughly.  
In general, this looks fine but I have not had a closer look; but I will  
consider it. A few procedural things will need to happen for that; but that  
can be left for later...

Re: [cone] Automatic Refresh for local Maildir's mail counts

[Linux-Fan <Ma_...@we...>]

[Wed, 01 Feb 2017 21:44:39 -0500] Sam Varshavchik
<mr...@co...> wrote:
> Linux-Fan writes:
> 
> > It is not using a pipe, but a C++11 std::atomic instead. If that
> > cannot be used, because Cone needs to compile without C++11 (I used
> > `./configure CPPFLAGS=-std=c++11` to get it to work), I'd replace
> > this by a variable which is marked `volatile sig_atomic_t`.
> 
> Cone is still compiled on relatively old platforms. C++11 is not a
> viable option right now; but probably in a little while this would be
> an option.

OK. I have changed the code to no longer rely on an atomic primitive. I
have implemented this by maintaining two counters where the first one
counts the number of signals received and the second one counts the
number of signals processed. This way, there is no potential for losing
a signal as there are no concurrent writes to the variables. (Well,
theoretically signals may still get lost: If exactly 2^32 signals are
received before Cone runs the function to detect pending updates, there
will not be an update, because advancing the counter 2^32 times puts it
back to it's initial position)

> I do not see a good reason not to use an internal pipe, in
> non-blocking mode, and with O_CLOEXEC set.

I have not tried to do it with pipes because I find them more complex:
Pipes need to be opened explicitely and I have not found a definite
answer about the question whether POSIX specifies writing to a pipe
from inside a signal handler to behave well-defined.

Also, I am not sure (for the pipe-idea) what happens if for any reason,
a lot of signals are received: My current solution will just keep
updating the messages all the times it is polled by the event loop.
Whenever a flood of signals stops, it will also do at most one update
after that. If I were to implement a pipe to detect signals, the pipe
would be filled (completely) whenever a flood of signals arises. On the
other end, my implementation should attempt to read as many entries as
possible (at once) from the pipe in order not to keep updating multiple
times even though the time the update was necessary is long in the
past. It might introduce a potential for nontermination if one end of
the pipe is continuously populated by signals while the other end is
continuously read (but never empty as for a steady stream of arriving
signals)...

All in all, I find pipes harder to reason about and thus went for the
easiest solution: Initially one atomic integer and now: two integers.

> sig_atomic_t has been around only since C99; but I suppose at this
> time requiring C99 would be acceptable.

Additionally, it should be available on every POSIX-compatible system.
This time, no additional options to `configure` were needed.

[...]

> No big rush here. You can take your time and test your changes
> thoroughly. In general, this looks fine but I have not had a closer
> look; but I will consider it. A few procedural things will need to
> happen for that; but that can be left for later...

Very well. I shall test this in practice :)

Thank you for the constructive reply :)

Yours Sincerely
Linux-Fan

-- 
http://masysma.lima-city.de/

Re: [cone] Automatic Refresh for local Maildir's mail counts

[Sam V. <mr...@co...>]

Linux-Fan writes:

> OK. I have changed the code to no longer rely on an atomic primitive. I
> have implemented this by maintaining two counters where the first one
> counts the number of signals received and the second one counts the
> number of signals processed. This way, there is no potential for losing
> a signal as there are no concurrent writes to the variables. (Well,
> theoretically signals may still get lost: If exactly 2^32 signals are
> received before Cone runs the function to detect pending updates, there
> will not be an update, because advancing the counter 2^32 times puts it
> back to it's initial position)

There is a reason why sig_atomic_t exists. The problem with sig_atomic_t is  
that it's unavailable on older platforms. There, platform-specific types  
were typically used for this purpose.

> I have not tried to do it with pipes because I find them more complex:
> Pipes need to be opened explicitely and I have not found a definite
> answer about the question whether POSIX specifies writing to a pipe
> from inside a signal handler to behave well-defined.

The write() system call is explicitly listed as being safe to call from a  
signal handler, see signal(7).

> Also, I am not sure (for the pipe-idea) what happens if for any reason,
> a lot of signals are received: My current solution will just keep

I believe I already pointed you to using non-blocking mode, for both writing  
and reading from the pipe. Once something is read from the pipe, loop and  
keep reading from it until it is completely drained.

This is, of course, entirely up to you. I'm just pointing out the direction  
I would've gone, in the same situation.