|
From: Rudi B. <ru...@ja...> - 2003-07-30 21:04:51
|
Hey guys, Errr... Well... Hate to say this, but... There was a major performance bug in JAWmail pretty much since its very beginning. Check the attached patch. You'll probably find it amusing. :) But, heck, I have to defend myself a bit, no? ;-) So, here it goes: Three months ago we have deployed JAWmail on a 220,000+ users system. It is a free email hosting service with something like 28k active users per day and 5.2k active users per hour (looking at IMAP/POP3 access graphs). The biggest obstacle was migration from the old system to the new. I wrote some scripts that put users into a queue and scheduled them for a LDAP->SQL & IMAP->IMAP transaction. Systems could handle something like 30 simultaneous migrations, but it still lasted 20 days. 400 gigs of smaller messages was quite a load. Database is MySQL 4.0.12 on a dual P4 1.4/512Mb RAM, RedHat AS (like everyt other box), RAID 1 with 10K RPM SCSI disks, ext3 FS, InnoDB tables. The current average is 172 queries per second and the daily peaks are currently (summertime, holidays) around 550 queries per second. Index efficiency is 99.9%. :) JAWmail currently stores: 420,000 address book items, 388,000 preferences rows, 800,000 sessions (these don't get cleaned regulary [1]) and 300,000 persistent objects. Database also stores all user infos, Postfix & Perdition routing tables,... so all boxes have to connect to it: 1 IMAP/POP3 proxy, 3 MXes, 1 Mailstore, 1 Web server and who knows what else. Load on the box? Usually around 0.15. MySQL kicks ass. Totally. Mail storage is done with a fail-over cluster of two dual Xeon 2.4G/2Gb boxes connected to 2x15 15K RPM disks in JBOD via FC, handled with Linux SW RAID (10), with ReiserFS. Cyrus 2.2.0-alpha with virtual domains, Sieve is used for redirecting messages & limited SMS forwarding. Works great. Load around 1. Currently we have reliability problems with disks that unfortunately end with a required reiserfsck which does its work in a few days. Go figure. Some firmware upgrades will supposedly fix this, but disk manufacturer is a bit lazy. Or something. An IMAP/POP3 proxy box is used for potential multiple mail storages. That box is currently an... errr Sun 220R... well, it works. Ah, the webserver thing. A quad Xeon 2Ghz/4Gb RAM. Well, it's mostly idling. ;) Load is usually around 1, anything higher is probably suspicious. And as stated on the beginning of this message, that patch might fix even that. Egh. Applied it a few minutes ago. Apache 1.3.27 and currently PHP 4.3.2-RC ... Usually from CVS. ;) The best PHP improvement is most definitily a PHP caching compiler. We use the ionCube PHP Accelerator, v1.3.3r2. It boosts everything _a lot_, especially JAWmail which has a quite big codebase. Lingerd also helps. We handle around 200,000 page views (so, PHP files alone) per day. The peaks would be around 5 page views per second. "ab" says something like 40 requests per seconds on the first page. We haven't done any inside tests... It just plain works. JAWmail from CVS, a little custom template addon for fixed-width pages, a bunch of plugins for registration, lost passwords, FAQ and such stupid-users(TM) crap. Also some other background libraries that handle user managment (generating IMAP accounts, very custom Sieve filters, logging,...). Nothing very interesting. The whole system is VERY fast. Every page loads in a snap. Messages get delivered in a second. Unfortunately, those stupid disks totally crash our reliability. Oh hell. Guys are on it. So, what the heck is this thing? "Email.si", supposedly the largest free email provider in Slovenia (that's a country ;). Go to http://www.email.si/, use the account jaw:aa ("Uporabniško ime" stands for "Username", "Geslo" for "Password"), I set it to English. You might be curious. Everything else is in Slovenian... Well, we're a local provider. Ads... :/ And I have commited the theme used there to CVS today, named it "Purply" ;-) Whoah. Hopefully this stuff will help someone. Ask if you're curious. ---- [1] - http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/jawmail/jaw/lib/session/sql.php?rev=1.6&content-type=text/vnd.viewcvs-markup - have to test on such big setup. But searching for matching dates on a 300+k rows table is still slow... Have to figure something out... A cleanup daemon is one idea... -- Rudi Benkovic ru...@ja... |