[JAWmail-devel] Holy shit.

[Rudi B. <ru...@ja...>]

Hey guys,

Errr... Well... Hate to say this, but... There was a major performance
bug in JAWmail pretty much since its very beginning. Check the
attached patch. You'll probably find it amusing. :)

But, heck, I have to defend myself a bit, no? ;-)

So, here it goes:


Three months ago we have deployed JAWmail on a 220,000+ users system.
It is a free email hosting service with something like 28k active
users per day and 5.2k active users per hour (looking at IMAP/POP3
access graphs).

The biggest obstacle was migration from the old system to the new.
I wrote some scripts that put users into a queue and scheduled them
for a LDAP->SQL & IMAP->IMAP transaction. Systems could handle
something like 30 simultaneous migrations, but it still lasted 20
days. 400 gigs of smaller messages was quite a load.


Database is MySQL 4.0.12 on a dual P4 1.4/512Mb RAM, RedHat AS (like
everyt other box), RAID 1 with 10K RPM SCSI disks, ext3 FS, InnoDB
tables. The current average is 172 queries per second and the daily
peaks are currently (summertime, holidays) around 550 queries per
second. Index efficiency is 99.9%. :)

JAWmail currently stores: 420,000 address book items, 388,000
preferences rows, 800,000 sessions (these don't get cleaned regulary
[1]) and 300,000 persistent objects.

Database also stores all user infos, Postfix & Perdition routing
tables,... so all boxes have to connect to it: 1 IMAP/POP3 proxy, 3
MXes, 1 Mailstore, 1 Web server and who knows what else.

Load on the box? Usually around 0.15. MySQL kicks ass. Totally.


Mail storage is done with a fail-over cluster of two dual Xeon
2.4G/2Gb boxes connected to 2x15 15K RPM disks in JBOD via FC, handled
with Linux SW RAID (10), with ReiserFS.

Cyrus 2.2.0-alpha with virtual domains, Sieve is used for redirecting
messages & limited SMS forwarding. Works great. Load around 1.

Currently we have reliability problems with disks that unfortunately
end with a required reiserfsck which does its work in a few days. Go
figure. Some firmware upgrades will supposedly fix this, but disk
manufacturer is a bit lazy. Or something.


An IMAP/POP3 proxy box is used for potential multiple mail storages.
That box is currently an... errr Sun 220R... well, it works.


Ah, the webserver thing. A quad Xeon 2Ghz/4Gb RAM. Well, it's mostly
idling. ;) Load is usually around 1, anything higher is probably
suspicious. And as stated on the beginning of this message, that patch
might fix even that. Egh. Applied it a few minutes ago.

Apache 1.3.27 and currently PHP 4.3.2-RC ... Usually from CVS. ;)

The best PHP improvement is most definitily a PHP caching compiler. We
use the ionCube PHP Accelerator, v1.3.3r2. It boosts everything _a
lot_, especially JAWmail which has a quite big codebase. Lingerd also
helps.

We handle around 200,000 page views (so, PHP files alone) per day. The
peaks would be around 5 page views per second. "ab" says something
like 40 requests per seconds on the first page. We haven't done any
inside tests... It just plain works.

JAWmail from CVS, a little custom template addon for fixed-width
pages, a bunch of plugins for registration, lost passwords, FAQ and
such stupid-users(TM) crap. Also some other background libraries that
handle user managment (generating IMAP accounts, very custom Sieve
filters, logging,...). Nothing very interesting.

The whole system is VERY fast. Every page loads in a snap. Messages
get delivered in a second. Unfortunately, those stupid disks totally
crash our reliability. Oh hell. Guys are on it.


So, what the heck is this thing? "Email.si", supposedly the largest
free email provider in Slovenia (that's a country ;). Go to
http://www.email.si/, use the account jaw:aa ("Uporabniško ime" stands
for "Username", "Geslo" for "Password"), I set it to English. You
might be curious. Everything else is in Slovenian... Well, we're a
local provider. Ads... :/

And I have commited the theme used there to CVS today, named it
"Purply" ;-)


Whoah. Hopefully this stuff will help someone. Ask if you're curious.

----

[1] -
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/jawmail/jaw/lib/session/sql.php?rev=1.6&content-type=text/vnd.viewcvs-markup
- have to test on such big setup. But searching for matching dates on
a 300+k rows table is still slow... Have to figure something out... A
cleanup daemon is one idea...

--

Rudi Benkovic             ru...@ja...