#13 Archiving large IMAP folders fails.

[Florian Ragwitz]

Hi,

I'd like to archive some of my IMAP folders
(cyrus21-imapd).
I call archivemail like that:
/usr/bin/archivemail --suffix= --days 30 -o
Mail.archive/
imap://florian:pass@imap.foobar.lan/florian.debian.powerpc

That workes for some folders, but expecially on bigger
ones (about 3.000 messages and above) it fails with the
following traceback:

Deleting messages
Traceback (most recent call last):
File "/usr/bin/archivemail", line 1305, in ?
main()
File "/usr/bin/archivemail", line 623, in main
archive(mailbox_path)
File "/usr/bin/archivemail", line 1040, in archive
_archive_imap(mailbox_name, final_archive_name)
File "/usr/bin/archivemail", line 1258, in _archive_imap
'+FLAGS.SILENT', '\\Deleted')
File "/usr/lib/python2.3/imaplib.py", line 670, in store
typ, dat = self._simple_command('STORE',
message_set, command, flags)
File "/usr/lib/python2.3/imaplib.py", line 1000, in
_simple_command
return self._command_complete(name,
self._command(name, *args))
File "/usr/lib/python2.3/imaplib.py", line 832, in
_command_complete
raise self.abort('command: %s => %s' % (name, val))
imaplib.abort: command: STORE => socket error: EOF
cleaning up ...

The mailbox is archived correct, but the archived
messages in the folder are not deleted.
I'm not sure whether this is an archivemail- or
imaplib-bug. I hope to have reported it at the right place.

Regards,
Florian

[Morten Lied Johansen]

Logged In: YES
user_id=574105

I have had to deal with this problem too, and my tests
suggests that the command to delete a message from the IMAP
server gets a list of messages to delete, and that there is
a limit to the lenght of that list.

The solution could probably be to split the command into
manageable chunks before sending it to the server?

[Peter Poeml]

Logged In: YES
user_id=78531

I fixed it with the following patch:

--- archivemail.py (revision 97)
+++ archivemail.py (working copy)
@@ -1279,9 +1279,14 @@
if archive:
archive.close()
archive.finalise()
- vprint("Deleting messages")
- imap_srv.store(string.join(message_list, ','),
- '+FLAGS.SILENT', '\\Deleted')
+ # do not delete more than a certain number of messages at a time, because the
+ # command length is limited. This avoids that servers terminate the connection with
+ # EOF or TCP RST.
+ vprint("Deleting %s messages" % len(message_list))
+ max_delete = 100
+ for i in range(0, len(message_list), max_delete):
+ imap_srv.store(string.join(message_list[i:i+max_delete], ','),
+ '+FLAGS.SILENT', '\\Deleted')
imap_srv.close()
imap_srv.logout()