[Courier-imap] Performance on huge maildir folders

[Fran F. <fr...@ci...>]

First off, let me say that I'm very happy with Courier-IMAP's 
performance and that the purpose of my post is not out of frustration 
for the software.  I'm merely trying to push it to its limits and see 
how far I can go. :-)  Having said that, I've been running some tests in 
my mail environment.  I have Courier-IMAP installed on a Xeon 2.8GHz 
system with 1GB memory.  OS is Redhat 9.  On this same box I have 
Apache, PHP and SquirrelMail running for webmail, as well as imapproxy 
to recycle connections from SquirrelMail to Courier-IMAP.  Within 
SquirrelMail I have enabled server-side SORT and server-side threads. 

The general user will access their Maildirs from this box via NFS to an 
ext2 filesystem.  However, for the purposes of my tests, I also set up 
some other scenarios, as outlined below.  The reason I am doing these 
tests is because I happen to have a Maildir that has 27,000 messages in 
its INBOX. :-)  I realize this is outside of the realm of what you might 
call "reasonable" and that I should not be expecting great performance 
in this case.  To be honest, I was just curious to see if I could get it 
to work at all. :-)  To my surprise, it did work, but load times were 
very long, about 45-60 seconds from the time I logged into SquirrelMail 
until the time I saw the contents of my INBOX.  So I started hunting for 
where the slowdown was, imapd, SquirrelMail, filesystem, CPU bound, 
memory, network, etc....

I used the same Maildir ~850MB in size with 27k messages in my INBOX in 
each case.  I logged into SquirrelMail with each user three times, and 
took the average number of seconds before the INBOX folder contents 
appeared.

Case 1:  Maildir lives on an ext2 filesystem within a RAID5 set of five 
disks.  This filesystem is then NFS mounted to the IMAP server.  Average 
time to display: 41 seconds.
Case 2:  Maildir lives on a reiserfs (Reiser4) filesystem on a RAID0 set 
of two disks.  This filesystem is then NFS mounted to the IMAP server.  
Average time to display: 64 seconds.
Case 3:  Maildir lives on an ext3 filesystem on a RAID0 set of two 
disks.  This filesystem is local to the IMAP server.  Average time to 
display: 37 seconds.

Not at all what I expected.  Pretty much everyone was saying that 
ReiserFS was the way to go for many small files, and yet it was by far 
the slowest.  Also, the effect of NFS vs. local was small.
I can see that the entire time I'm waiting for the page to return, imapd 
is pegging the CPU, so this might be CPU-bound.  Loads on the machine 
aren't going above 0.6 or so.  I would hope that one single imap session 
would not bring a Xeon 2.8GHz/1GB mem to its knees.  Even still, I'll be 
able to compare in a couple of weeks when the 2nd processor and another 
1GB of RAM comes in for these systems.

Now, I might just write this off and tell users "if you have folders 
with 27k messages in them, prepare for horrid performance", but it's 
just one of those things that you just want to make faster. :-)  Do 
people have ideas of where the most time might be spent in this 
transaction?  And more importantly, other areas I might look at to 
improve performance, if possible?

Any and all ideas much appreciated! 

Thanks,
Fran

-- 
Fran Fabrizio
Senior Systems Analyst
Department of Computer and Information Sciences
University of Alabama at Birmingham 
http://www.cis.uab.edu/
205.934.0653