From: Roman P. <pu...@x-...> - 2009-08-27 11:19:01
|
Hi Alex, thanks for coming back on my points. I already sent an answer to Daniel discussing the application scenario with some hundred threads. Regarding the broadcast channel: well, that might lead to http://en.wikipedia.org/wiki/Thundering_herd_problem - so maybe it's not the best idea. What do you think? Back to the screaming metal: I guess this is sort of theological discussion wether to server each client with a single thread, or to have one process serve all clients (Tcl), or have thread pools with each thread serving many clients. My believe (of course) is the latter, especially to avoid stack space waste, so I am currently working on libevent 2.x as an extension for Tcl, having SSL, buffer events, multi-threading, epoll and IOCP on windows (when it's available in libevent). My first tests however show an incredible performance boost, having moderate CPU usage serving some hundred clients. As for the Tcl-core: a relatively easy step to increase the total number of sockets (and performance) would be to replace select by epoll/poll, where available. With pthread_kill I remember that signal handling can be quite expensive (time) and overruns may occur. Personally, I'll stay with pipes (which can overrun as well) ;) Finally, and as asked before in the email to Daniel, are there any general plans on modifying / extending the io handling in the core? Is there a place for a wish list? Let me know! @David: does the current IOCP support worker threads, or is it a single worker thread? What do you think? Thanks & take care! Roman PS: what I am doing right now is to have libevent live in separate threads and have inter-thread communication over fd's (to the tcl core). Whenever a worker thread has decrypted and re-assembled a complete message, it simply places an event to the main tcl thread. That's nice in terms of that the application logic remains single-threaded, but having worker threads which can share the ssl encryption etc. > -----Original Message----- > From: Alexandre Ferrieux [mailto:ale...@gm...] > Sent: Thursday, August 27, 2009 10:56 AM > To: Roman Puls > Cc: tcl...@li...;David Gravereaux > Subject: Re: [TCLCORE] multithreaded Tcl IO / select vs. > epoll and IOCP > > On 8/26/09, Roman Puls <pu...@x-...> wrote: > > > > Hi Alex, > > > > > > > About the extra-fd solution: indeed we had thought of > one > extra > > pipe per thread for interthread communication, but it > is > judged Too > > Fscking Expensive by many... Now you're saying > "shared > between the > > threads"... > > > what do you have in mind ? One single pipe, with N readers > > > selecting on it ? Is this supposed to work ? Also, this would > > > awaken everybody whenever a thread::send is done; I'm > concerned > > about the efficiency. > > > Please elaborate. > > > > > > Well, it depends on the application or architecture: if > we're talking about io-centric worker threads (which > typically reflect the number of CPUs or cores), the overhead > for pipes will not be large. Have one fd per thread, and > you're done writing to it. Also, for Tcl, that pipe could > handle all sort of event notification. > > The problem is that a generic thing like the Tcl core cannot > make restrictive assumptions on the app :-}. I remember > people talking about thousands of threads were one pipe for > each one would be unthinkable; all the more so if > inter-thread communication is *not* used in the app... which > is notably often the case in the category of high-throughput > TCP servers ! > > > > However, reading the man pages for 'eventfd', 'signalfd', > 'timerfd_create', select/epoll seems to be the way on how to > realize that sort of event multiplexing / inter-thread > communication. But of course, these are for kernels >= 2.6.22 > only, and linux only. > > See http://lwn.net/Articles/225714/: "It brings Linux > toward a single interface for event delivery without the need > for a new, complex API." > > > > Very interesting... for our children :) (I have 2.4 machines around) > > > For the rest of the unix world, still pipes seem to be the only way. > > Yes. Pipes *or* pthread_kill when applicable. > > > And, as you were asking: select/epoll'ing the same handle > will probably not work (or lead to unwanted results), but > duping the pipe handles will do. But as written before, the > cleanest approach for me is to have one pipe per thread. > > Just checked: both duping and sharing work with select() on my 2.6.9. > So we do have a cheap broadcast channel after all, even > though it doesn't fit the purpose. > > > About the 'screaming metal': well, for the server side, I > need one accept socket (true), but N reading/writing sockets > (with let's say N around 512). Your proposal was probably not > to have 512 threads sitting on a blocked select, right? > > My proposal was to have no select at all: > > while(1) { > s2=accept(...); > pthread_create(worker,s2); > } > > void worker(int fd) { > while(1) { > n=read(fd,...) > if (n<=0) break; > ... > } > close(fd); > } > > > > > The only clean way to handle it is to have either one big > IO/loop (what I do now, with the known limits) or to have > several IO/loops, each sitting on its own (worker)thread. > > Yup. The latter. But no notifier needed. > > > Being curious if Daniel has something to add, and - what > do the windows guys think? Who's actually taking care of that > part? Do you plan to integrate IOCP? > > IIRC David Gravereaux (Cced) maintains a branch where he > rewrote all the Windows socket code using IOCP. Maybe he can step in ? > > -Alex > > |