Stephan Gerth

FWIW: Source server is Dovecot 2.3.13 (Debian 11), target server is Dovecot 2.4.1-4 (Debian 13).

I've sent you a log of a -Ddn run with BufferLimit 1, PipelineDepth 1 on the target and FSync no to the maintainer address.

I think the disk setup is not the actual problem since the network connection could always be the main bottleneck (imagine syncing over a 50MBit internet connection). But I think I figured out why the BufferLimit has no effect (lines refer latest git 42cfa1): Line sync.c:1468 checks via get_memory_usage if BufferLimit has been already reached or overrun and handles that. However, according to the logs all calls to get_memory_usage returned no usage at all: N: Called get_memory_usage, ret=0 This explains...

I had a look at iotop and I could see that the write rate was 1,5x compared the Fsync yes setting. So it seems to make a difference but does not increase write rate enough. Setting BufferLimit 1 pushes message storage further to the end in the log file (-Dd) but it still fetches all messages so that the connection to the source server runs into the timeout (while it is working on storing all mails into the target mailbox folder (about 120k lines only enter/leave store_msg lines).

I've tried with the -Dd option: The output makes sense so far: A lot of interleaved messages of such as this snippet (within the large mailbox folder already): F: [ 120772] Callback enter fetch_msg, sts=0, flags=, date=1789297202, size=110049 N: [ 228630] Enter store_msg, flags=, date=1789297202, size=110071, to_trash=no N: [ 228630] Leave store_msg F: [ 120772] Callback leave fetch_msg F: [ 120773] Callback enter fetch_msg, sts=0, flags=, date=1789279202, size=122022 N: [ 228631] Enter store_msg,...

Regarding the working of isync this was just guessing from what I've observed, sorry! The mailbox folder in question has about 108k emails, so that's far from millions of messages. I've tried to speed-up the target server by putting the target mailbox on a ramdisk. This lets the synchronization finish before the timeout hits which solves it in my case. However, for larger mailboxes or slower servers the issue would surely occur again. How is the general mechanism working when target server takes...

No, this happens somewhere in the middle (on the 43rd folder of 48 to be specific which happens to be quite large). I watched some numbers this time while running the synchronization: fetching from the source seems to be faster than pushing to the target mbsync seems to cache the whole(?) mailbox folder (memory usage went to max. 10G) when it tries to push the remaining mails to the target while the source mails have been already cached, the source connection is not seeing any activity During the...

Connection loss for IMAP2IMAP sync on large mailboxes