isync-devel — All-purpose list, actually - but RTFM first. Moderated for non-subscribers for SPAM protection.

[commit] master: fix --dry-run without --debug-driver

[ossi <os...@us...>]

commit 1e0b661d09c5a4bbb0bb2061db8071363145d5ac
Author: Oswald Buddenhagen <os...@us...>
Date:   Tue Jan 28 00:49:17 2025 +0100

    fix --dry-run without --debug-driver
    
    the stubbing is implemented by the proxy driver, so we need to hook it
    in in that case as well. the driver is already prepared for that, as
    it's also used to implement -Ta/-TA.

 src/main_sync.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/main_sync.c b/src/main_sync.c
index 2ea4a92..f4ec34b 100644
--- a/src/main_sync.c
+++ b/src/main_sync.c
@@ -450,7 +450,7 @@ do_sync_chans( main_vars_t *mvars )
 			labels[F] = labels[N] = "";
 		for (int t = 0; t < 2; t++) {
 			store_t *ctx = mvars->drv[t]->alloc_store( mvars->chan->stores[t], labels[t] );
-			if ((DFlags & DEBUG_DRV) || ((DFlags & FORCEASYNC(t)) && !(dcaps[t] & DRV_ASYNC))) {
+			if ((DFlags & (DEBUG_DRV | DRYRUN)) || ((DFlags & FORCEASYNC(t)) && !(dcaps[t] & DRV_ASYNC))) {
 				mvars->drv[t] = &proxy_driver;
 				ctx = proxy_alloc_store( ctx, labels[t], DFlags & FORCEASYNC(t) );
 			}

Re: Does isync report an error in case of data corruption in RAM?

[Oswald B. <osw...@gm...>]

On Mon, Jan 13, 2025 at 04:33:32PM -0800, Hong Xu wrote:
>However, consumer-grade hardware is prone to RAM bit flips, at a higher
>rate than what most people probably think. The most recent research I
>knew is discussed on Wikipedia [2], which shows the error rate was on
>the magnitude of 1 bit error per gigabyte of RAM per 1.8 hours.
>
did you actually read that google paper?
as far as i'm concerned, its conclusion is "bad ram is bad, duh".
a subsequent study by facebook
(https://www.dpss.inesc-id.pt/~romanop/files/secMemErr.pdf) drives the
point home even harder.
it's not a non-issue, but it's by far not as dramatic as the number
above suggests, and running a ram test on a regular basis should reveal
most problems before they cause much damage.

Re: Does isync report an error in case of data corruption in RAM?

[Hong Xu <ho...@to...>]

On 2025-01-13 Mon 04:25 GMT-08, Oswald Buddenhagen <osw...@gm...> wrote:

> On Mon, Jan 13, 2025 at 01:33:54AM -0800, Hong Xu wrote:
>>Does isync perform any checks (like some checksum) to prevent data loss
>>in an event of RAM error?
>>
> no, and the idea doesn't even make much sense. there would have to be a
> checksum to compare to, and that couldn't be calculated before the data
> was actually loaded into ram.

I did a bit of research into IMAP, and I agree that there's no way to
fully fix this unless there are data integrity support at the IMAP
level.

>
> the whole idea of software-based ram-checking would make sense only for
> data that lives in ram for a long time, where corruption due to storage
> deterioration would be much more likely than corruption during transfer,
> or the added time due to computing the checksums at storage and
> retrieval.
>
> isync stores message payloads only for a few seconds in its network
> buffers. if you are concerned about integrity at that time scale, get
> hardware that can do it for you automatically. it would be total
> insanity to half-assedly harden each application.

Even though I don't think isync can fully fix the issue here, I'd still
like to clarify the concern for isync users, which is real and shouldn't
be dismissed. I'd also like to share what I do to mitigate.

=== Concern ===

With proper backup, storage corruption is not a concern here, because
storage corruptions can hardly go undetected due to the builtin error
checks in hard drives, even on consumer-grade hardware. When a
corruption occurs, data can be read without error only at a super-low
probability. See [1] and its reference.

However, consumer-grade hardware is prone to RAM bit flips, at a higher
rate than what most people probably think. The most recent research I
knew is discussed on Wikipedia [2], which shows the error rate was on
the magnitude of 1 bit error per gigabyte of RAM per 1.8 hours.

And they truly affect data integrity, which is a frequent topic
discussed in the NAS community for one example, because data can go
corrupted in RAM in the period of being copied from one location to
another (e.g., [3]).

=== My Solution ===

I'd also like to share what I do to mitigate silent data corruption when
it's impractical to use an ECC RAM:

- Regularly back up email folders. This can become a life-saver if a RAM
  becomes faulty, in which case many file corruptions and system crashes
  becomes noticeable. I can then restore the backups and let them sync
  back to the server with isync.

- Save all important data, like attachment, to somewhere else with data
  integrity checks, such as a local git repository, or a subversion
  repository if there are large binaries. In this way, I don't worry
  about silent data corruptions when copying them around in the future,
  and occasional bit flip in emails becomes non-concerning.

[1] https://serverfault.com/a/77721/158338
[2] https://en.wikipedia.org/wiki/ECC_memory#Research
[3] https://forums.truenas.com/t/ecc-memory-for-home-servers-or-not/7099



-- 
Hong

Re: Does isync report an error in case of data corruption in RAM?

[Oswald B. <osw...@gm...>]

On Mon, Jan 13, 2025 at 01:33:54AM -0800, Hong Xu wrote:
>Does isync perform any checks (like some checksum) to prevent data loss
>in an event of RAM error?
>
no, and the idea doesn't even make much sense. there would have to be a
checksum to compare to, and that couldn't be calculated before the data
was actually loaded into ram.
the whole idea of software-based ram-checking would make sense only for
data that lives in ram for a long time, where corruption due to storage
deterioration would be much more likely than corruption during transfer,
or the added time due to computing the checksums at storage and
retrieval.
isync stores message payloads only for a few seconds in its network
buffers. if you are concerned about integrity at that time scale, get
hardware that can do it for you automatically. it would be total
insanity to half-assedly harden each application.

[commit] master: explicitly strip newline from contents of VERSION

[ossi <os...@us...>]

commit 277e3cee1b5c31e221b2bfd4592d09c41a9a2226
Author: Oswald Buddenhagen <os...@us...>
Date:   Mon Jan 13 13:09:18 2025 +0100

    explicitly strip newline from contents of VERSION
    
    while m4 1.4.19 + autoconf 2.72 from my debian unstable drop the
    trailing newline automatically, m4 1.4.18 + autoconf 2.69 shipped
    with ubuntu 20.4 don't, leading to an invalid configure script.
    
    reported and verified by Donat Wegner <dm...@we...>.

 configure.ac | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/configure.ac b/configure.ac
index 7f1fc6e..8c46b8b 100644
--- a/configure.ac
+++ b/configure.ac
@@ -3,7 +3,7 @@
 dnl SPDX-License-Identifier: GPL-2.0-or-later
 
 m4_syscmd([./version.sh])
-AC_INIT([isync], m4_sinclude([VERSION]))
+AC_INIT([isync], m4_chomp(m4_sinclude([VERSION])))
 AC_CONFIG_HEADERS([autodefs.h])
 
 AC_CANONICAL_TARGET

Does isync report an error in case of data corruption in RAM?

[Hong Xu <ho...@to...>]

Hi all,

I'm interested in understanding whether isync is capable of preventing
data loss in case of RAM errors.

Bit corruption in RAM is a low-probability event but does happen. ECC
memories [1] exist to address this concern. However, today, most
personal computers use non-ECC memories and are prone to such
corruption. [2]

In the context of email syncing, this means that some silent data
corruption may occur if there's no integrity check. For example, a
message is synced to local, but its content can become corrupted in RAM
before written to disk.

Another consequence may occur if the RAM starts to be faulty. An ECC RAM
will refuse to work at all if it's too faulty, preventing data loss. But
a non-ECC RAM can just corrupt whatever is being read from the RAM and
written to the disk at the moment.

Does isync perform any checks (like some checksum) to prevent data loss
in an event of RAM error?

Thank you!


[1] https://en.wikipedia.org/wiki/ECC_memory
[2] You may have heard that all DDR5 RAMs have "ECC", but this is untrue. See https://superuser.com/a/1797176/139328
-- 
Hong

Re: synchronize Opentext/Groupwise Notes+Appointments?

[Peter P. <pet...@fa...>]

Thanks for the reply Oswald!

Meanwhile I found out that these "Notes" and "Appointments" are only
displayed in the webmail Inbox, but are in fact stored in a separate
imap folder which I wasn't syncing so far. I am doing this now and
mbsync gracefully syncs these objects as well. Sorry for the noise and
thanks again!

Peter

* Oswald Buddenhagen via isync-devel <isy...@li...> [2024-12-20 11:03]:
> On Fri, Dec 20, 2024 at 10:08:29AM +0100, Peter P. wrote:
> > Can someone possibly confirm that these
> > messages are not transferred by mbsync from an imap server?
> > 
> isync transfers everything it sees on imap, and that doesn't even have
> to be messages (though objects not compliant with rfc822 would get
> garbled). you can use another imap client (say, thunderbird) to confirm
> that isync isn't just blind. maybe these specials are virtually mapped
> to separate mailboxes?
>
> obviously, nobody here knows anything about this. have you googled
> and/or asked in a forum related to that server?

Re: synchronize Opentext/Groupwise Notes+Appointments?

[Oswald B. <osw...@gm...>]

On Fri, Dec 20, 2024 at 10:08:29AM +0100, Peter P. wrote:
>Can someone possibly confirm that these
>messages are not transferred by mbsync from an imap server?
>
isync transfers everything it sees on imap, and that doesn't even have
to be messages (though objects not compliant with rfc822 would get
garbled). you can use another imap client (say, thunderbird) to confirm
that isync isn't just blind. maybe these specials are virtually mapped
to separate mailboxes?

obviously, nobody here knows anything about this. have you googled
and/or asked in a forum related to that server?

Re: synchronize Opentext/Groupwise Notes+Appointments?

[Peter P. <pet...@fa...>]

Dear list,

still trying to figure out how to sync the aforementioned "Notes" and
"Appointments" using mbsync. Can someone possibly confirm that these
messages are not transferred by mbsync from an imap server?

Thanks again,
Peter

* Peter P. <pet...@fa...> [2024-12-15 06:57]:
> Hi list,
> 
> I discovered that people are from time to time sending me "Notes"
> instead of emails, or "Appointments". These seem to be something
> different and are not synced by mbsync, making me miss messages and
> appointments.
> 
> The anatomy of such an "appointment" is:
> MESSAGE 	2 KB 
> TEXT.htm 	6 KB 	
> rfc2445.ics 	7 KB 	
> Mime.822 	40 KB 
> 
> and of a "Note":
> MESSAGE 	1 KB
> text.htm 	1 KB 
> 
> Is there a way to sync them, or at least convert them on the server to
> something that can be synced?
> 
> Thanks!
> Peter

Re: config working on Ubuntu not working an arch?

[Bence F. <be...@fe...>]

On Thu Dec 19, 2024 at 10:35, Tassilo Horn <ts...@gn...> wrote:
> "Bence Ferdinandy" <be...@fe...> writes:
>
> Hi Bence,
>
>> My previous machine was running Ubuntu now I moved to CachyOS (arch
>> btw).  I supposedly didn't change anything in my configs and I can
>> actually send messages with msmtp for my oauth accounts (gmail and
>> outlook), but mbsync if failing for both.
>>
>> For example with `mbsync -D`:
>>
>>     Connecting to imap.gmail.com (173.194.76.109:993)...
>>     Connection is now encrypted
>>     F: * OK Gimap ready for requests from 89.223.195.239 o37mb130304551wms
>>     F: >>> 1 CAPABILITY
>>     F: * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN
>> X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN
>> AUTH=OAUTHBEARER
>>     F: 1 OK Thats all she wrote! o37mb130304551wms
>>     Logging in...
>>     Authenticating with SASL mechanism PLAIN...
>>     F: >>> 2 AUTHENTICATE PLAIN <authdata>
>>     F: 2 NO [AUTHENTICATIONFAILED] Invalid credentials (Failure)
>>     IMAP command 'AUTHENTICATE PLAIN <authdata>' returned an error:
>> [AUTHENTICATIONFAILED] Invalid credentials (Failure)
>
> Hm, I use Arch with the extra/isync package which is currently version
> 1.5.0-2 and it works for me (on the AUR there's only an isync-git
> package).  That's my debug log for my gmail account:
>
> --8<---------------cut here---------------start------------->8---
> Connecting to imap.gmail.com ([2a00:1450:400c:c00::6c]:993)... 
> Connection is now encrypted
> F: * OK Gimap ready for requests from 2003:df:1703:d00:5e51:4fff:fef3:ece7 b9mb127754308wmb
> F: >>> 1 CAPABILITY
> F: * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN AUTH=OAUTHBEARER
> F: 1 OK Thats all she wrote! b9mb127754308wmb
> Logging in...
> Authenticating with SASL mechanism PLAIN...
> F: >>> 2 AUTHENTICATE PLAIN <authdata>
> F: * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL- SPECIAL-USE APPENDLIMIT=35651584
> F: 2 OK ts...@gm... authenticated (Success)
> F: >>> 3 COMPRESS DEFLATE
> F: 3 OK Success
> F: >>> 4 ENABLE UTF8=ACCEPT
> F: * ENABLED UTF8=ACCEPT
> F: 4 OK Success
> F: >>> 5 NAMESPACE
> F: * NAMESPACE (("" "/")) NIL NIL
> F: 5 OK Success
> --8<---------------cut here---------------end--------------->8---
>
> However, I don't use OAuth2 for authentication but an app password.  It
> seems the plugin you've built yourself on Ubuntu is available on the
> AUR:
>
> --8<---------------cut here---------------start------------->8---
> ❯ yay -s xoauth2                                                                        
> 3 aur/sasl-xoauth2-git r203.47ff232-2 (+0 0.00) 
>     SASL plugin that enables client-side use of OAuth 2.0
> 2 aur/oauth2ms-git r10.a1ef0cabfdea-1 (+0 0.00) (Orphaned) 
>     XOAUTH2 compatible O365 token fetcher
> 1 aur/cyrus-sasl-xoauth2-git r24.36aabca54fd6-1 (+5 0.13) 
>     XOAUTH2 mechanism plugin for cyrus-sasl
> --8<---------------cut here---------------end--------------->8---
>
> Maybe try installing that.

Thanks everybody for the pointers, one step closer now :)


Logging in...
Authenticating with SASL mechanism XOAUTH2...
Error performing SASL authentication step: SASL(-1): generic failure: Unable to find a callback: 32775

❯ paru -Q | grep sasl
cyrus-sasl 2.1.28-4.1
cyrus-sasl-xoauth2-git r24.36aabca54fd6-1
gsasl 2.2.1-1.1
libsasl 2.1.28-5.1
sasl-xoauth2-git r206.bb5c890-1

Tried building isync also, but that has the same output. I also restarted, just
to make sure.

Nevermind, after uninstalling `sasl-xoauth2-git` it works (thanks Marci!). So tldr:

these two packages are needed on arch:
cyrus-sasl 
cyrus-sasl-xoauth2-git 

Best,
Bence

Re: config working on Ubuntu not working an arch?

[Tassilo H. <ts...@gn...>]

"Bence Ferdinandy" <be...@fe...> writes:

Hi Bence,

> My previous machine was running Ubuntu now I moved to CachyOS (arch
> btw).  I supposedly didn't change anything in my configs and I can
> actually send messages with msmtp for my oauth accounts (gmail and
> outlook), but mbsync if failing for both.
>
> For example with `mbsync -D`:
>
>     Connecting to imap.gmail.com (173.194.76.109:993)...
>     Connection is now encrypted
>     F: * OK Gimap ready for requests from 89.223.195.239 o37mb130304551wms
>     F: >>> 1 CAPABILITY
>     F: * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN
> X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN
> AUTH=OAUTHBEARER
>     F: 1 OK Thats all she wrote! o37mb130304551wms
>     Logging in...
>     Authenticating with SASL mechanism PLAIN...
>     F: >>> 2 AUTHENTICATE PLAIN <authdata>
>     F: 2 NO [AUTHENTICATIONFAILED] Invalid credentials (Failure)
>     IMAP command 'AUTHENTICATE PLAIN <authdata>' returned an error:
> [AUTHENTICATIONFAILED] Invalid credentials (Failure)

Hm, I use Arch with the extra/isync package which is currently version
1.5.0-2 and it works for me (on the AUR there's only an isync-git
package).  That's my debug log for my gmail account:

--8<---------------cut here---------------start------------->8---
Connecting to imap.gmail.com ([2a00:1450:400c:c00::6c]:993)... 
Connection is now encrypted
F: * OK Gimap ready for requests from 2003:df:1703:d00:5e51:4fff:fef3:ece7 b9mb127754308wmb
F: >>> 1 CAPABILITY
F: * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN AUTH=OAUTHBEARER
F: 1 OK Thats all she wrote! b9mb127754308wmb
Logging in...
Authenticating with SASL mechanism PLAIN...
F: >>> 2 AUTHENTICATE PLAIN <authdata>
F: * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL- SPECIAL-USE APPENDLIMIT=35651584
F: 2 OK ts...@gm... authenticated (Success)
F: >>> 3 COMPRESS DEFLATE
F: 3 OK Success
F: >>> 4 ENABLE UTF8=ACCEPT
F: * ENABLED UTF8=ACCEPT
F: 4 OK Success
F: >>> 5 NAMESPACE
F: * NAMESPACE (("" "/")) NIL NIL
F: 5 OK Success
--8<---------------cut here---------------end--------------->8---

However, I don't use OAuth2 for authentication but an app password.  It
seems the plugin you've built yourself on Ubuntu is available on the
AUR:

--8<---------------cut here---------------start------------->8---
❯ yay -s xoauth2                                                                        
3 aur/sasl-xoauth2-git r203.47ff232-2 (+0 0.00) 
    SASL plugin that enables client-side use of OAuth 2.0
2 aur/oauth2ms-git r10.a1ef0cabfdea-1 (+0 0.00) (Orphaned) 
    XOAUTH2 compatible O365 token fetcher
1 aur/cyrus-sasl-xoauth2-git r24.36aabca54fd6-1 (+5 0.13) 
    XOAUTH2 mechanism plugin for cyrus-sasl
--8<---------------cut here---------------end--------------->8---

Maybe try installing that.

HTH,
  Tassilo

Re: isync duplicated some, but not all messages when moving computers

[Mathieu A. <ma...@ma...>]

Le Wed, Dec 18, 2024 at 10:37:43PM +0100, Bence Ferdinandy a écrit :
> 
> On Wed Dec 18, 2024 at 21:29, Oswald Buddenhagen via isync-devel <isy...@li...> wrote:
> > On Wed, Dec 18, 2024 at 02:33:51PM +0100, Bence Ferdinandy wrote:
> >>it was probably not the best idea, but I moved computers, by coping all
> >>my
> >>maildir and isync config, but not the isync state (in retrospect, I probably
> >>should have done that).
> >>
> > yeah ...
> >
> >>I ran isync (1.5.0) and some emails got duplicated, but
> >>not everything and more heavily in Archive folders. Is this the
> >>expected behaviour?
> >>
> > no. i expect that there is some confounding factor.
> > maybe misleading sort order?
> > or some messages were never pushed up because of reasons?
> 
> There were probably messages never pulled, but who knows. Anyway, I'm now
> professional in mail de-duplication.

A while back, I collected all my various imap tools written in various
languages over the years into a single tool, and it does deduplication
removal, amongst other things:
https://crates.io/crates/imap-tools
https://gitlab.com/mat813/imap-tools-rs

Though you'll need "cargo" to install it.

-- 
Mathieu Arnold

Re: config working on Ubuntu not working an arch?

[Oswald B. <osw...@gm...>]

On Wed, Dec 18, 2024 at 10:53:23PM +0100, Bence Ferdinandy wrote:
>    F: >>> 2 AUTHENTICATE PLAIN <authdata>
>
>Now for Ubuntu I know I had to mess around SASL and even build
>moriyoshi/cyrus-sasl-xoauth2.git manually to make things work,

>but this doesn't look like an issue with that.
>
but it totally does. it's clearly not even trying to use oauth. it must
be something with that plugin, but i have no clue beyond that.

google whether you can enable some sasl debugging via env vars or config
files.

beyond that, i guess i'd strace it and see if it does anything
unexpected. compare with a trace from the working box maybe.

config working on Ubuntu not working an arch?

[Bence F. <be...@fe...>]

Hi,

sorry for a second round of having issues moving ... Hopefully the last.

My previous machine was running Ubuntu now I moved to CachyOS (arch btw).
I supposedly didn't change anything in my configs and I can actually send
messages with msmtp for my oauth accounts (gmail and outlook), but mbsync if
failing for both.

For example with `mbsync -D`:

    Connecting to imap.gmail.com (173.194.76.109:993)...
    Connection is now encrypted
    F: * OK Gimap ready for requests from 89.223.195.239 o37mb130304551wms
    F: >>> 1 CAPABILITY
    F: * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN AUTH=OAUTHBEARER
    F: 1 OK Thats all she wrote! o37mb130304551wms
    Logging in...
    Authenticating with SASL mechanism PLAIN...
    F: >>> 2 AUTHENTICATE PLAIN <authdata>
    F: 2 NO [AUTHENTICATIONFAILED] Invalid credentials (Failure)
    IMAP command 'AUTHENTICATE PLAIN <authdata>' returned an error: [AUTHENTICATIONFAILED] Invalid credentials (Failure)

Since it's failing for _both_ in mbsync and both can send I'm assuming it must
be mbsync. Now for Ubuntu I know I had to mess around SASL and even build
moriyoshi/cyrus-sasl-xoauth2.git manually to make things work, but this doesn't
look like an issue with that. Currently I have the package installed from aur.

❯ paru -Qe | grep isync
isync 1.5.0-2.1

I'm a bit lost what can be going wrong here. Any ideas?

Thanks,
Bence

Re: isync duplicated some, but not all messages when moving computers

[Bence F. <be...@fe...>]

On Wed Dec 18, 2024 at 21:29, Oswald Buddenhagen via isync-devel <isy...@li...> wrote:
> On Wed, Dec 18, 2024 at 02:33:51PM +0100, Bence Ferdinandy wrote:
>>it was probably not the best idea, but I moved computers, by coping all
>>my
>>maildir and isync config, but not the isync state (in retrospect, I probably
>>should have done that).
>>
> yeah ...
>
>>I ran isync (1.5.0) and some emails got duplicated, but
>>not everything and more heavily in Archive folders. Is this the
>>expected behaviour?
>>
> no. i expect that there is some confounding factor.
> maybe misleading sort order?
> or some messages were never pushed up because of reasons?

There were probably messages never pulled, but who knows. Anyway, I'm now
professional in mail de-duplication.

>
>>I was hoping that without zero state, isync would just rescan and
>>match up everything nicely if slowly.
>>
> no, isync does not look at the content, not even message ids.
> one could implement that as a fallback if one was so inclined ...
>
>>For future reference? Is copying everything, including state the
>>preferred way of moving?
>>
> yes.
> unless your connection to the imap server is faster then a local
> transfer. then just sync up, and sync from scratch on the new box.

Well, now I know :)

Thanks,
Bence

Re: isync duplicated some, but not all messages when moving computers

[Oswald B. <osw...@gm...>]

On Wed, Dec 18, 2024 at 02:33:51PM +0100, Bence Ferdinandy wrote:
>it was probably not the best idea, but I moved computers, by coping all
>my
>maildir and isync config, but not the isync state (in retrospect, I probably
>should have done that).
>
yeah ...

>I ran isync (1.5.0) and some emails got duplicated, but
>not everything and more heavily in Archive folders. Is this the
>expected behaviour?
>
no. i expect that there is some confounding factor.
maybe misleading sort order?
or some messages were never pushed up because of reasons?

>I was hoping that without zero state, isync would just rescan and
>match up everything nicely if slowly.
>
no, isync does not look at the content, not even message ids.
one could implement that as a fallback if one was so inclined ...

>For future reference? Is copying everything, including state the
>preferred way of moving?
>
yes.
unless your connection to the imap server is faster then a local
transfer. then just sync up, and sync from scratch on the new box.

isync duplicated some, but not all messages when moving computers

[Bence F. <be...@fe...>]

Hi,

it was probably not the best idea, but I moved computers, by coping all my
maildir and isync config, but not the isync state (in retrospect, I probably
should have done that). I ran isync (1.5.0) and some emails got duplicated, but
not everything and more heavily in Archive folders. Is this the expected
behaviour? I was hoping that without zero state, isync would just rescan and
match up everything nicely if slowly.

For future reference? Is copying everything, including state the preferred way
of moving?

Thanks,
Bence

synchronize Opentext/Groupwise Notes+Appointments?

[Peter P. <pet...@fa...>]

Hi list,

I discovered that people are from time to time sending me "Notes"
instead of emails, or "Appointments". These seem to be something
different and are not synced by mbsync, making me miss messages and
appointments.

The anatomy of such an "appointment" is:
MESSAGE 	2 KB 
TEXT.htm 	6 KB 	
rfc2445.ics 	7 KB 	
Mime.822 	40 KB 

and of a "Note":
MESSAGE 	1 KB
text.htm 	1 KB 

Is there a way to sync them, or at least convert them on the server to
something that can be synced?

Thanks!
Peter

Re: Suddenly errors regarding "near side box INBOX/XXX cannot be opened."

[Oswald B. <osw...@gm...>]

On Sun, Dec 08, 2024 at 09:23:11AM +0100, Johannes Kastl wrote:
>> Error: channel XXX: near side box INBOX/20140102_BACKUP cannot be opened.
>
>I can send the log directly, if it is any help?
>
yes, but let's go for the complete set:
- your config file
- cd <local Path>; find -type d
- mbsync -D -ls
- mbsync -D -l <channel>
- mbsync -D <channel>

Re: Suddenly errors regarding "near side box INBOX/XXX cannot be opened."

[Johannes K. <ma...@oj...>]

Hi all,

On 04.12.24 06:43 Johannes Kastl wrote:
> Hi Oswald,
> 
> On 02.12.24 11:28 Oswald Buddenhagen via isync-devel wrote:
>> On Sun, Dec 01, 2024 at 08:47:48PM +0100, Johannes Kastl wrote:
>>>> Error: channel YYY: near side box INBOX/XXX cannot be opened.
>>>
>>>Funnily enough, in one case this happened after the folder was created
>>>locally and the sync succeeded. On the next run the error above was shown.
>>>
>> weird. try 'mbsync -ls' for starters.
>> if that looks good, try 'mbsync -l'.
>> if that still looks good, it's presumably not a user error.
>> in that case try syncing only one failing mailbox (YYY:INBOX/XXX).
>> add -D to the above command and the normal YYY run, und mail me the
>> logs.
> 
> Will try to do some more debugging.

Hmmm, I find no obvious errors (at least, obvious to me...):

> Opening far side box INBOX/20140102_BACKUP...
> F: [ 10] Enter open_box
> F: >>> 7 SELECT "INBOX/20140102_BACKUP"^M
> F: [ 10] Leave open_box 
> Opening near side box INBOX/20140102_BACKUP...
> F: [ 7] Callback leave load_box
> F: * 97 EXISTS
> F: * 0 RECENT 
> F: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
> F: * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Unlimited
> F: * OK [UIDNEXT 1396] Predicted next UID
> F: * OK [UIDVALIDITY 1388669892] UIDs valid
> F: 7 OK [READ-WRITE] SELECT completed
> F: [ 10] Callback enter open_box, sts=0, uidvalidity=1388669892
> Error: channel XXX: near side box INBOX/20140102_BACKUP cannot be opened.
> F: Called get_caps, ret=0x7
> N: Called get_caps, ret=0

And the same for all children of this folder.

I can send the log directly, if it is any help?

Kind Regards,
Johannes

Re: Repeatable crash in 1.5.0

[Robin P. <rlp...@di...>]

Awesome, thanks!

On Sat, Dec 7, 2024 at 12:47 AM Oswald Buddenhagen via isync-devel <
isy...@li...> wrote:

> On Fri, Dec 06, 2024 at 05:07:49PM -0800, Robin Lee Powell via isync-devel
> wrote:
> >mbsync: socket.c:957: socket_read: Assertion `min_len <= max_len' failed.
> >
> this is already fixed in master as of a few days ago.
>
>
> _______________________________________________
> isync-devel mailing list
> isy...@li...
> https://lists.sourceforge.net/lists/listinfo/isync-devel
>

Re: Repeatable crash in 1.5.0

[Oswald B. <osw...@gm...>]

On Fri, Dec 06, 2024 at 05:07:49PM -0800, Robin Lee Powell via isync-devel wrote:
>mbsync: socket.c:957: socket_read: Assertion `min_len <= max_len' failed.
>
this is already fixed in master as of a few days ago.

Repeatable crash in 1.5.0

[Robin L. P. <rlp...@di...>]

I have sync set up for my gmail to local maildir, it's been working
fine for months.

I recently updated my system and went from (probably)
isync-0:1.4.4-5.fc37.x86_64 to (definitely)
isync-0:1.5.0-2.fc41.x86_64 , and now it does this:

    Opening far side box [Gmail]/Chats...
    Opening near side box [Gmail]/Chats...
    Loading far side box...
    Loading near side box...
    near side: 503 messages, 0 recent
    far side: 938 messages, 0 recent
    Synchronizing...
    C: 0/1  B: 34/44  F: +2/2 *0/0 #0/0 -0/0  N: +2/437 *2/2 #0/0 -0/0mbsync: socket.c:957: socket_read: Assertion `min_len <= max_len' failed.
    [2]    3876479 IOT instruction (core dumped)  mbsync -V gmail

I don't actually care about the Chats folder (it's had no activity
in ~10 years), but since it does this with perfect reliability, I thought y'all
my be interested in exploring it as a possible bug; lemme know how I can help.

Re: IMAP Timeouts During Local Operations—Correct Behaviour?

[Sabahattin G. <lis...@me...>]

On 4 Dec 2024, at 09:31, Oswald Buddenhagen via isync-devel <isy...@li...> wrote:
> On Tue, Dec 03, 2024 at 04:12:09PM +0000, Sabahattin Gucukoglu via isync-devel wrote:
>> So my question is this: is it expected behaviour that timeouts occur
>> because of long local operations,
>> 
> yes, sort of.
> 
>> and is it *correct* or desirable behaviour?
>> 
> certainly not desirable.

Agreed. It’s just not intuitive IMO and is not what one would expect from a control intended to detect unreachability. Yes, it’s sort of obvious that it happens at all because by the time a response is expected that much time has passed. But of course the delay is being imposed by mbsync and not the server.

>> Isn’t the “right thing” here to have a dedicated idle timeout, which
>> would take effect while isync is doing purely local computation?
>> 
> is that timeout reported by isync or the server? what's the end of the
> log with -Dn?

Reported by mbsync. Here are the log lines starting with the opening of the problem mailbox:
Opening far side box GRC...
>>> 32 SELECT "GRC"
Opening near side box GRC...
* 0 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1692709377]
* OK [UIDNEXT 1]
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen \Recent $MailFlagBit0 $MailFlagBit1 $MailFlagBit2 $Forwarded Redirected $NotJunk NotJunk)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \Recent $MailFlagBit0 $MailFlagBit1 $MailFlagBit2 $Forwarded Redirected $NotJunk NotJunk $iCloudCleanup \*)]
32 OK [READ-WRITE] SELECT completed (took 2 ms)
Loading far side box...
far side: 0 messages, 0 recent
Warning: lost track of 1018382 pushed message(s)
Loading near side box...
near side: 1018382 messages, 1018382 recent
Synchronizing...
>>> 33 APPEND "GRC" "04-Dec-2024 18:25:24 +0000" {7001}
Socket error on imap.mail.me.com (17.42.251.69:993): timeout.

That APPEND is the very first message in the mailbox. With even more debugging I see that every other near-far transaction is pending right before the timeout occurs. That “lost track” warning seems to be about uncommitted messages from the last failed attempt, but as you see not a single APPEND was actually successful, so it’s fine. I initially wasn’t really willing to download all the source messages again, so that was fortunate. But as it happens running mbsync interactively with a 300 second timeout (see below) was still giving me lots of “Server too busy” warnings from the IMAP server anyway, so I was willing to start again fresh from source for science.

>> I am successfully working around this situation by using a separate
>> config file for the duration of the initial upload [...]
>> 
> how is it different?

Sorry. Increase timeout from 30 to 300s. It is now running in background; it’s too slow and painful to watch interactively, so I’m hoping it’ll complete over the course of a few days. It is however making the IMAP server very grumpy and I seem to be pushing some limit or other because it sometimes just refuses to store a message, or disconnects me. I can’t wait to finish setting up my self-hosted mail again after years.

> my first idea would be to limit the operations to the minimum, in your
> case that should be --push-new if i understand your case correctly.

Yes, I’ll look into this, thanks, although it now seems to be on the rails.

> secondly, if the problem is local i/o and you're pushing, then copying
> the box to a ram drive might be a solution. (make sure to move the state
> file into place when switching back to the proper box.)
> 
> did you try to identify what the time is actually spent on? i'd first
> try htop and iotop, and then for finer results valgrind/cachegrind and
> strace -tt, resp.

Interesting idea re the ramdisk! I will probably try that next if I need to, though I’m hoping the 30s timeout will be acceptable again after the initial transfer.

This is macOS so I’ll need to look at the equivalents, however a quick look at ActivityMonitor doesn’t show substantial disk I/O at all, less than 10 K/s, so we’re probably being held back by latency here.

Thanks for the help.

Cheers,
Sabahattin

Re: [commit] master: build: also consider builds off of git with `git clone --depth 1`

[Paymon <dar...@gm...>]

On Wed, Dec 04, 2024 at 05:28:33PM +0100, Oswald Buddenhagen via isync-devel wrote:
> but maybe this could be still handled better? i guess not, as the branch
> name above is meaningless, and presumably no other refs are present?  does
> gitk --all show anything sensible in this repo?

it's a remote machine with wayland only.
what would `gitk --all show` tell us? (never used it before)
people at #git tell me to use `git log --all`... and there is something:

```sh
portage@frostbite /var/tmp/portage/net-mail/isync-9999/work/isync-9999 $ git --no-pager log --all
commit 884413b48871361cf6c0cfa41d01c22ed1ec8d30 (grafted, HEAD)
Author: Paymon MARANDI <dar...@gm...>
Date:   Tue Dec 3 15:25:09 2024 -0500

    build: also consider builds off of git with `git clone --depth 1`

    one case where this could happen is a shallow clone, purely for build
    purposes. in source based distros, like gentoo, setting depth of the
    clone to 1, globally, is a common configuration. this patch avoids that
    those builds fail.

    Signed-off-by: Paymon MARANDI <dar...@gm...>
```

```sh
portage@frostbite /var/tmp/portage/net-mail/isync-9999/work/isync-9999 $ cat .git/refs/git-r3/HEAD
884413b48871361cf6c0cfa41d01c22ed1ec8d30
```

git-r3 is poratge's doing. (portage is the package manager in gentoo).

-- 

               Paymon