From: Daniel B. <da...@st...> - 2007-07-13 02:42:06
|
What are people using serve-event for, and what features of it do they use? In the course of attempting (with at best qualified success, which I blame on me at least as much as on sbcl) to write software which is robust in the face of the internet, I have been rubbing up against bits of serve-event and fd-streams and suchlike, and I suspect it to be over-complex for most uses it gets put to. Herewith a bunch of random observations about which experienced fd-handler programmers could probably say "sure, we could have told you that": * 'bogus' handlers (those on descriptors for which fstat() returns false) never get unmarked or removed except by restarts. A handler may go bogus if, say, the socket it refers to is closed. It will not be unbogused - as far as I can tell at a glance, anyway - even if the fd is reused for something else (say, an fd-stream created by CL:OPEN) * user-installed 'ready for output' handlers may get trodden on by frob-output-later, if the fd is one under the care of fd-streams. "So don't do that then" * bogus handlers are detected and signalled in serve-event - which in a typical program is as likely at a random unexpected time (e.g. while waiting at the repl) than when doing file io on the associated stream. The condition which is signalled has no type more specialised than ERROR, so we can't pick it off selectively either with a finely-specialised handler visible to most of the program or with a coarsely-specialised handler bound around only a very small part of the program. * oh, and what leads me to think I may not be alone in my assessment of this stuff as "cruft": this comment from serve-event.lisp which has all the hallmarks of 1999-era WHN doing CMUCL archeology - : ;; FIXME: unused. At some point this used to be set to T ;; around the call to the handler-function, but that was commented ;; out with the verbose explantion "Doesn't work -- ACK". active (I haven't checked CVS logs, I'm just guessing from the text style) In the back of my mind I also have Eric Marsden's presentation (at an LSM sometime?) demonstrating among other things the Bad Things that may happen if serve-event handlers - even nice unobtrusive ready-for-input serve-evet handlers - are called recursively. For my own entertainment/edification I am thinking about gutting fd-streams and replacing with something shorter and simpler that (a) serves the needs of ordinary ansi cl functions, and (b) provides sufficient hooks for "tell me when input is available" that one could write, say, a reasonably traditional network daemon or an event loop for a gui or similar with it. However, I'm obviously not about to entertain the idea of a long-term fork of sbcl to do so, and so I wonder if other people have similar ideas/radically different ideas/uses for serve-event that require it to be kept in its full complexity. (The only actual /need/ for serve-event in fd-streams is so that force-output can return without having written the buffers it's forcing. ANSI language lawyers are welcome to comment on the necessity for this, but my feeling is that in the absence of any portable way of determining whether a stream is buffered or not, "does not wait" in the standard is merely an unclear way of writing "does not necessarily wait" rather than a requirement that function returns before the write finishes. I've only recently resubscribed to sbcl-devel: I'm aware of (and have skimmed but not fully digested) the recent talk about replacing the serve-event internals poll() or epoll() or kevent, but if there's also been discussion of its external api then apologies, I haven't found that yet. -dan |
From: Timothy R. <tri...@ma...> - 2007-07-13 03:13:44
|
Dan, I can't really speak to your API points, although having been through the serve-event code, it does appear to be on the crufty side. In our case, we are handling our heavy connection load through our own epoll/ kqueue networking library (not dissimilar to iolib I would assume), and serve-event only gets invoked for lower socket count activity such as sending email and the cl-sql -> mysql connection. Our motivation for an epoll serve-event implementation was to simply avoid everything coming to a shuddering halt whenever the mysql or smtp socket was assigned an FD > 1024. If someone has an idea for a better serve-event design, I would be more than happy to work on refactoring our platform-specific epoll and kqueue code. On Jul 12, 2007, at 10:42 PM, Daniel Barlow wrote: > What are people using serve-event for, and what features of it do > they use? <snip> > I've only recently resubscribed to sbcl-devel: I'm aware of (and have > skimmed but not fully digested) the recent talk about replacing the > serve-event internals poll() or epoll() or kevent, but if there's also > been discussion of its external api then apologies, I haven't found > that > yet. > > > -dan > > ---------------------------------------------------------------------- > --- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > Sbcl-devel mailing list > Sbc...@li... > https://lists.sourceforge.net/lists/listinfo/sbcl-devel |
From: Timothy R. <tri...@ma...> - 2007-07-14 02:31:27
|
One issue to contend with is that serve-event is created somewhat =20 early in the sbcl build process, so not all CL facilities are =20 necessarily available. IIRC, CLOS is one thing=97there are certainly =20 others that I'm not remembering. In any case, none of the contribs =20 are available yet. Dependence on CFFI would probably be another. =20 These may not be an issue with IOLib - I haven't had a chance to look =20= over the source yet, but something to consider in any move. On Jul 13, 2007, at 3:45 PM, Far=E9 wrote: > The IOLib guy is rewriting in portable CL with CFFI wrappers to basic > C functions a lot of the things that could go in the guts of > SERVE-EVENTS. He supports and auto-selects various backends (select, > epoll, kqueue, etc.). Could serve-events be made something > semi-portable this way, instead of remaining a huge unportable > duplication of efforts between all CL implementations? > > Note: I am using IOlib together with arnesi's call/cc to write a > green-threaded network server in semi-portable CL. > > [ Fran=E7ois-Ren=E9 =D0VB Rideau | Reflection&Cybernethics | http://=20= > fare.tunes.org ] > I'll start exercising as soon as I'm into shape. > |
From: Nathan F. <fr...@gm...> - 2007-07-14 02:54:47
|
On 7/13/07, Timothy Ritchey <tri...@ma...> wrote: > One issue to contend with is that serve-event is created somewhat > early in the sbcl build process, so not all CL facilities are > necessarily available. IIRC, CLOS is one thing=97there are certainly > others that I'm not remembering. In any case, none of the contribs > are available yet. Here's a thought: what about being *very* careful in all code executed pre-cold init to only use, say, string-streams and fd-streams sans serve-event, and then write the event-driven I/O part + full streams in warm-init so we can use things like CLOS and so forth? I know there would likely be a little code duplication, but depending on how much there would be, it might be worth it. We wouldn't necessarily have to be all that smart in the pre-cold init stuff, either--it just has to be goo= d enough to get us through cold-init and compiling in warm-init. So a bunch of the fancy bits (e.g. direct I/O for {read,write}-sequence) don't have to be there. -Nathan |
From: Timothy R. <tri...@ma...> - 2007-07-14 14:27:32
|
When serve-event isn't working (IOW when I have broken it in my local build) the first thing that obviously breaks is the sb-simple-streams contrib. Whether this means there are no other issues, I couldn't say. However, it does seem to indicate that most of the build is currently safe without serve-event. -- Paragent.com (765) 749-5063 On Jul 13, 2007, at 10:54 PM, Nathan Froyd <fr...@gm...> wrote: > > Here's a thought: what about being *very* careful in all code executed > pre-cold init to only use, say, string-streams and fd-streams sans > serve-event, and then write the event-driven I/O part + full streams > in warm-init so we can use things like CLOS and so forth? I know > there > would likely be a little code duplication, but depending on how much > there would be, it might be worth it. We wouldn't necessarily have to > be all that smart in the pre-cold init stuff, either--it just has to > be good > enough to get us through cold-init and compiling in warm-init. So a > bunch of the fancy bits (e.g. direct I/O for {read,write}-sequence) > don't have to be there. > > -Nathan > > --- > ---------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > Sbcl-devel mailing list > Sbc...@li... > https://lists.sourceforge.net/lists/listinfo/sbcl-devel |
From: Nikodemus S. <nik...@ra...> - 2007-07-14 15:06:07
|
Timothy Ritchey wrote: > When serve-event isn't working (IOW when I have broken it in my local > build) the first thing that obviously breaks is the sb-simple-streams > contrib. Whether this means there are no other issues, I couldn't say. > However, it does seem to indicate that most of the build is currently > safe without serve-event. Yep. Serve event can be stubbed out just fine. Cheers, -- Nikodemus |
From: Matthew S. <ako...@gm...> - 2007-07-14 23:20:39
|
On Fri, 13 Jul 2007 22:31:24 -0400, Timothy Ritchey wrote: ... > On Jul 13, 2007, at 3:45 PM, Faré wrote: ... >> Could serve-events be made something >> semi-portable this way, instead of remaining a huge unportable >> duplication of efforts between all CL implementations? >> Is there momentum to do this? Because that's an exciting prospect (though understandably a bit far afield of sbcl-devel). Matt -- "You do not really understand something unless you can explain it to your grandmother." - Albert Einstein. |