Oswald Buddenhagen

Connection loss for IMAP2IMAP sync on large mailboxes

ok, thanks, something is obviously going wrong. first it queues 22 fetches (which is incompatible with a pipeline depth of 1), then it queries the buffer size for all remaining messages (which is zero, because no appends have been scheduled), then it queues all remaining fetches. then the first fetches are completing, upon which it correctly schedules the first append, and the next one only once the first one completes (this is how pipelinedepth 1 should work). one notable observation is that the...

the only legitimate way how that can happen is that the IMAP commands were actually all already sent out, and the queuing is happening on the target server. -Ddn should confirm that. the way to work around that would be limiting PipelineDepth to something reasonable, say 50.

the fs load result is surprising. what are the actual numbers? network bw, disk bw, fs type & mount options, etc. that effect of bufferlimit sounds just wrong. please mail me the log.

Setting FSYNC no did not make any difference. weird. have a look at iotop on the target server with the two settings. also check the mount options of the volume. it's kinda unexpected that the process is disk-bound to such a degree. Would it make sense to set it to some smaller value than the default value so that it cannot fetch the whole mailbox folder [...]? yes. you could actually use something ridiculously small (even 1) to avoid that multiple STOREs are queued up at all.

there is no such mechanism; i just didn't anticipate this situation. assuming you are using the default BufferLimit, the timeout cannot possibly occur while messages are being transferred. (*) so it must be during message enumeration. but that should be also on the order of 10 MB with your mailbox size, so i dunno. have you tried the -Dd log? if you can't make sense of it, you can mail it to me privately. EDIT: (*) well, actually, if the target server is using a legacy fs that does fsync inefficiently,...

there is no such mechanism, i just didn't anticipate this situation. assuming you are using the default BufferLimit, the timeout cannot possibly occur while messages are being transferred. so it must be during message enumeration. but that should be also on the order of 10 MB with your mailbox size, so i dunno. have you tried the -Dd log? if you can't make sense of it, you can mail it to me privately.

that's just not how isync works. you can watch it by adding -Dd. -Dn (or just -D) will produce even more output, but based on the info so far i expect that to be somewhat useless due to the sheer volume of repetetive content. how large is "quite large"? at normal speeds, such a duration would translate to millions of messages.