Menu
▾
▴
mod-pubsub-developer
|
From: BC S. <bsi...@in...> - 2004-03-27 00:06:40
|
Just to be sure I understand... there is a kqueue [*bsd] reactor already in twisted, but not one for epoll [linux], right? And Joyce needs one for Linux? ----- Original Message ----- From: Alex Russell <al...@ne...> Date: Fri, 26 Mar 2004 15:11:42 -0800 To: George Schlossnagle <ge...@om...> Subject: Re: [Mod-pubsub-developer] port to twisted > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Friday 26 March 2004 2:05 pm, George Schlossnagle wrote: > > Hi, > > > > Joyce Park has been talking to me about porting mod_pubsub to use > > the twisted event framework. I've done some poking around and > > wanted to share some assessments before I start work in earnest: > > > > In your main event loop, you currently do this: > > > > while server.alive: > > asyncore.poll(scheduler.timeout()) > > scheduler.run() > > Right. This was the primary discussion point around > poll/select/kqueue/epoll. We all realize that this is horrendously > inefficient. The Plan (TM), as I understand it, is to move this to a > Twisted reactor and then use the appropriate reactor for the system > at hand (kqueue, epoll, etc...). > > Someone had mentioned libevent on Wed, and I'm considering writing a > reactor for it, which would ease the portability issues further. > > [ snip ] > > Actual implementation aside, is their any reason I'm missing as to > > why this is a bad idea? > > Nope, it's exactly the right approach. > > Regards > > - -- > Alex Russell > al...@bu... BD10 7AFC 87F6 63F9 1691 83FA 9884 3A15 AFC9 61B7 > al...@ne... F687 1964 1EF6 453E 9BD0 5148 A15D 1D43 AB92 9A46 > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.2.4 (Darwin) > > iD8DBQFAZLiuoV0dQ6uSmkYRAunUAKDI9jskaFwpZCmvIAp/xaFhN+OP0gCgsOS+ > /z3jZEZfppqRU3PpIdsCufc= > =fAKv > -----END PGP SIGNATURE----- > > > > ------------------------------------------------------- > This SF.Net email is sponsored by: IBM Linux Tutorials > Free Linux tutorial presented by Daniel Robbins, President and CEO of > GenToo technologies. Learn everything from fundamentals to system > administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=click > _______________________________________________ > Mod-pubsub-developer mailing list > Mod...@li... > https://lists.sourceforge.net/lists/listinfo/mod-pubsub-developer -- _______________________________________________ Get your free email from http://www.iname.com |
|
From: BC S. <bsi...@in...> - 2004-03-27 00:38:48
|
George, the approach you lay out seems like the right one. The scheduler in there now is only a last-minute hack and should be replaced by a real scheduling system (Twisted, for example) at the first opportunity. -- _______________________________________________ Get your free email from http://www.iname.com |
|
From: Alex R. <al...@ne...> - 2004-03-27 00:12:34
|
=2D----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday 26 March 2004 4:06 pm, BC Sittler wrote: > Just to be sure I understand... there is a kqueue [*bsd] reactor > already in twisted, but not one for epoll [linux], right? And Joyce > needs one for Linux? That's correct. PyKqueue should build on OS X, but so far gcc 3.3 is=20 giving me fits (even after hacking the makefile a couple of times).=20 PyKqueue doesn't seem to have seen active maintenance since about=20 2000-2001, but is reported to still work. As for doing the libevent reactor, the odds are very high that=20 libevent exposes sane semantics for what we need, and that writing a=20 reactor for it is the same effort (or lower) as writing an epoll=20 reactor. Whichever one Joyce chooses to go with is fine by me. Either=20 way we get coverage of all the OSes we need (collectively) need. Regards. =2D --=20 Alex Russell al...@bu... BD10 7AFC 87F6 63F9 1691 83FA 9884 3A15 AFC9 61B7 al...@ne... F687 1964 1EF6 453E 9BD0 5148 A15D 1D43 AB92 9A46 =2D----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (Darwin) iD8DBQFAZMeBoV0dQ6uSmkYRAp7jAJ9mhqjRDsgugeExPcFy4Dy1iKofawCcCP9h fuN/4xSO8LvQ2m9uzlRYOm4=3D =3DeGYb =2D----END PGP SIGNATURE----- |
|
From: George S. <ge...@om...> - 2004-03-27 02:02:15
|
On Mar 26, 2004, at 7:06 PM, BC Sittler wrote: > Just to be sure I understand... there is a kqueue [*bsd] reactor > already in twisted, but not one for epoll [linux], right? And Joyce > needs one for Linux? Just fyi, the epoll backport to 2.4 is slightly flaky (will build products that exploit that target that patch). Even without epoll, the poll reactor will give you the scalability you need. Either way, adding an epoll reactor should not be terribly difficult though. George |
|
From: Alex R. <al...@ne...> - 2004-03-29 00:13:30
|
=2D----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday 26 March 2004 6:02 pm, George Schlossnagle wrote: > On Mar 26, 2004, at 7:06 PM, BC Sittler wrote: > > Just to be sure I understand... there is a kqueue [*bsd] reactor > > already in twisted, but not one for epoll [linux], right? And > > Joyce needs one for Linux? > > Just fyi, the epoll backport to 2.4 is slightly flaky (will build > products that exploit that target that patch). Even without epoll, > the poll reactor will give you the scalability you need. I have no doubt that poll() will do the job, but as to whether or not=20 poll() will induce high latency with a huge number of long-lived=20 clients, I'm not sure: http://lse.sf.net/epoll/index.html#compare Solaris added /dev/poll for many of the same reasons that Linux has=20 added epoll and FreeBSD kqueue/kevent. It's just another step in the=20 research-driven evolution of Unix in general. Select breaks down well=20 before poll when faced with lots of connnections, but with 10K active=20 clients, it's not unreasonable to expect even poll()'s O(changed FDs)=20 performance to get problematic. > Either way, adding an epoll reactor should not be terribly=20 > difficult though. Agreed. Regards =2D --=20 Alex Russell al...@bu... BD10 7AFC 87F6 63F9 1691 83FA 9884 3A15 AFC9 61B7 al...@ne... F687 1964 1EF6 453E 9BD0 5148 A15D 1D43 AB92 9A46 =2D----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (Darwin) iD8DBQFAZ2q8oV0dQ6uSmkYRApgwAJ97pkcWkAzCz9fvpY8zdvh0BkfQCACgzHJs Micv7DZCMhbiBCPNlXgaeBM=3D =3DEpow =2D----END PGP SIGNATURE----- |
|
From: George S. <ge...@om...> - 2004-03-29 01:27:47
|
On Mar 28, 2004, at 7:15 PM, Alex Russell wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Friday 26 March 2004 6:02 pm, George Schlossnagle wrote: >> On Mar 26, 2004, at 7:06 PM, BC Sittler wrote: >>> Just to be sure I understand... there is a kqueue [*bsd] reactor >>> already in twisted, but not one for epoll [linux], right? And >>> Joyce needs one for Linux? >> >> Just fyi, the epoll backport to 2.4 is slightly flaky (will build >> products that exploit that target that patch). Even without epoll, >> the poll reactor will give you the scalability you need. > > I have no doubt that poll() will do the job, but as to whether or not > poll() will induce high latency with a huge number of long-lived > clients, I'm not sure: > > http://lse.sf.net/epoll/index.html#compare As a bit of an aside (since everything you've said is completely true), while epoll(), /dev/poll and kqueue() are much more scalable than poll, you can still get pretty decent performance out of stock poll(). For instance, our MTA (ecelerity) can support 100,000 inbound connections at 450 messages per second, which is pretty darn fast (200-300 Mb/s). You get better performance out of the platform-specific event systems (and we support those on systems that support them), but the poll() performance is not shabby. At any rate, even without it's implementation in twisted, porting to twisted seems to be the fastest way to get epoll() support, as it is likely to show up there soon and be well supported. // George Schlossnagle // Postal Engine -- http://www.postalengine.com/ // Ecelerity: fastest MTA on earth |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X