Large Maildir Synced fast for first 78% of ~600k message, now at snails pace...
mailbox synchronizer
Brought to you by:
ossi
Menu
▾
▴
#79 Large Maildir Synced fast for first 78% of ~600k message, now at snails pace one per email for rest
Large Maildir of 579637 messages (originally from Gmail) chokes at ~ 450,000 messages on syncing from local Maildir to my self-run on docker-mailserver IMAP account run on server with 8 GB RAM and 160GB SSD, with a 4096 MB swap disk . The setup cut thru ~450,000 messages efficiently and easily.
BUT, according to the mailing list, others successfully limped forward by running a loop to get messages at smaller amount.
Did my RTFM: https://sourceforge.net/p/isync/mailman/search/?q=unexpected+EOF+2023&mail_list=isync-devel
I did this with setup of default PipelineDepth and set Timeout to 0 (actually no change from my initial setup).
And for a time, was getting 1 email message per run. Been doing this for lilke 2-4 weeks, I lost count...
I would like to fix this because I have more accounts to convert of other users and would prefer to get a proper fix in.
docker-compose exec isync /sync
Reading configuration file /root/.mbsyncrc
C: 0/1 B: 0/0 F: +0/0 *0/0 #0/0 N: +0/0 *0/0 #0/0
Channel my-inbox
Opening far side store my-remote...
Resolving myemailserver.org... ok
Connecting to myemailserver.org (xx.xxx.xxx.xxx:993)...
Opening near side store my-local...
Connection is now encrypted
Logging in...
C: 0/1 B: 0/1 F: +0/0 *0/0 #0/0 N: +0/0 *0/0 #0/0
Opening far side box Archive...
Opening near side box Archive...
Loading far side box...
Loading near side box...
near side: 579637 messages, 0 recent
far side: 553961 messages, 0 recent
Warning: lost track of 753 pushed message(s)
Synchronizing...
C: 0/1 B: 0/1 F: +756/1008 *0/0 #0/0 N: +0/0 *0/0 #0/0
Socket error from myemailserver.org (myemailserver:993): Connection reset by peer
C: 1/1 B: 1/1 F: +756/1008 *0/0 #0/0 N: +0/0 *0/0 #0/0
make: *** [Makefile:46: isync] Error 1 # (NOTE: This is just mbsync called)
Command exited with non-zero status 2
real 11m 0.40s
user 0m 0.10s
sys 0m 0.05s
# another run, the next run in the loop
docker-compose exec isync /sync
Reading configuration file /root/.mbsyncrc
C: 0/1 B: 0/0 F: +0/0 *0/0 #0/0 N: +0/0 *0/0 #0/0
Channel my-inbox
Opening far side store my-remote...
Resolving myemailserver.org... ok
Connecting to myemailserver.org (xx.xxx.xxx.xxx:993)...
Opening near side store my-local...
Connection is now encrypted
Logging in...
C: 0/1 B: 0/1 F: +0/0 *0/0 #0/0 N: +0/0 *0/0 #0/0
Opening far side box Archive...
Opening near side box Archive...
Loading far side box...
Loading near side box...
near side: 579637 messages, 0 recent
far side: 554718 messages, 0 recent
Warning: lost track of 251 pushed message(s)
Synchronizing...
C: 0/1 B: 0/1 F: +359/880 *0/0 #0/0 N: +0/0 *0/0 #0/0
Socket error from myemailserver.org (xx.xxx.xxx.xxx:993): Connection reset by peer
C: 1/1 B: 1/1 F: +359/880 *0/0 #0/0 N: +0/0 *0/0 #0/0
make: *** [Makefile:46: isync] Error 1
Command exited with non-zero status 2
real 5m 58.65s
user 0m 0.06s
sys 0m 0.04s
Let me know what you need to analyze this!
Thanks
GH
Related
are you sure you aren't hitting some configured server limits? or that the server isn't thrashing?
in principle, you could be hitting limits even when just re-syncing old messages, as retrieving such huge message lists alone causes quite some traffic.
experiment some with partial syncs, e.g. --push-new.
next, inspect the traffic itself, looking at -Dn output. do you see a pattern across multiple runs?
next you could also inspect the low-level traffic using wireshark, but if something turns out odd at this level, the problem is most likely outside isync's purview.
Thank you for your response
Please see below:
Probably also hitting some configured limits there, but first will try to improve this side as loads of people are using docker mailserver, which is postfix.
This is an old archive, so I'm not sure what --push-new does exactly. There are not new emails being placed actively into this Maildir.
Regardless, I took your suggestion and ran with the "--push-new --verbose" options that in loop over night, as was still getting disconnects from my server and "lost messages".
I revisited my settings for channel:
Channel my-inbox
Far :my-remote:Archive
Near :my-local:Archive
Patterns *
Create Far
Expunge Far
SyncState /root/.mbsync/
Sync Push
CopyArrivalDate yes
On the above, I do not want to alter the static Near archive. Maybe a setting is incorrect here?
My goal was to see if by running in loop that I could complete the message sync. But, what found is that now my FAR side has MORE message than my NEAR side.
Here is a sample of a couple of runs.
Reading configuration file /root/.mbsyncrc
C: 0/1 B: 0/0 F: +0/0 0/0 #0/0 N: +0/0 0/0 #0/0
Channel my-inbox
Opening far side store my-remote...
Resolving myemailserver.org... ok
Connecting to myemailserver.org (XX.XXX.XXX.XXX:993)...
Opening near side store my-local...
Connection is now encrypted
Logging in...
C: 0/1 B: 0/1 F: +0/0 0/0 #0/0 N: +0/0 0/0 #0/0
Opening far side box Archive...
Opening near side box Archive...
Loading far side box...
Loading near side box...
near side: 579637 messages, 0 recent
far side: 614870 messages, 0 recent
Warning: lost track of 218 pushed message(s)
Synchronizing...
C: 0/1 B: 0/1 F: +670/1483 0/0 #0/0 N: +0/0 0/0 #0/0
Socket error from myemailserver.org (XX.XXX.XXX.XXX:993): Connection reset by peer
C: 1/1 B: 1/1 F: +670/1483 0/0 #0/0 N: +0/0 0/0 #0/0
Reading configuration file /root/.mbsyncrc
C: 0/1 B: 0/0 F: +0/0 0/0 #0/0 N: +0/0 0/0 #0/0
Channel my-inbox
Opening far side store my-remote...
Resolving myemailserver.org... ok
Connecting to myemailserver.org (XX.XXX.XXX.XXX:993)...
Opening near side store my-local...
Connection is now encrypted
Logging in...
C: 0/1 B: 0/1 F: +0/0 0/0 #0/0 N: +0/0 0/0 #0/0
Opening far side box Archive...
Opening near side box Archive...
Loading far side box...
Loading near side box...
near side: 579637 messages, 0 recent
far side: 614510 messages, 0 recent
Warning: lost track of 468 pushed message(s)
Synchronizing...
C: 0/1 B: 0/1 F: +359/578 0/0 #0/0 N: +0/0 0/0 #0/0
Socket error from myemailserver.org (XX.XXX.XXX.XXX:993): Connection reset by peer
C: 1/1 B: 1/1 F: +359/578 0/0 #0/0 N: +0/0 0/0 #0/0
And, when I look at the FAR side, to me looks like new messages.
I am not sure how to analyze if there are duplicates here or why this number is larger. Help?
Will try these later if need be if cannot understand why my NEAR and FAR are not synced one to one.
Thankyou!
GH
duplicates may come exactly from the "lost" messages. though it seems to be quite many. alternatively, you'd get dupes if you deleted the sync state without deleting the box.
you can find dupes by opening the box directly with mutt (on the server, or via imap) and tagging the pattern
~=, iirc.your configuration seems just fine. the use of Patterns seems pointless, but not problematic.
you should probably re-read the manual, as you seem confused about some things.
What I found from using docker mailserver is that I can get past these issues by setting a mailbox quota and then using PipelineDepth 1
Somehow that all worked. BUT, I think this is a good converstaino for anyone who might have similar issues. Feel free to close.
ok, thanks for the update.
@Githubbie, out of curiosity, since docker-mailserver's Dovecot is backed by Maildir by default, why not just do a Syncthing or Resilio Sync (or rsync/rclone/unison or even straight cp/scp if this is a one-off migration) to do a Maildir-to-Maildir transfer? I imagine it'd be faster and possibly have higher fidelity
Last edit: Huy 2024-12-09
Good idea. Did not cross my mind somehow! Next time!
:)
December 9, 2024 at 7:26 AM, "Huy" <huyz@users.sourceforge.net mailto:huyz@users.sourceforge.net?to=%22Huy%22%20%3Chuyz%40users.sourceforge.net%3E > wrote:
Related
Bugs:
#79