Re: [TCLCORE] multithreaded Tcl IO / select vs. epoll and IOCP

[Alexandre F. <ale...@gm...>]

On 8/26/09, Roman Puls <pu...@x-...> wrote:
>
>  Hi Alex,
>
>
>  > About the extra-fd solution: indeed we had thought of one
>  > extra pipe per thread for interthread communication, but it
>  > is judged Too Fscking Expensive by many... Now you're saying
>  > "shared between the threads"...
>  > what do you have in mind ? One single pipe, with N readers
>  > selecting on it ? Is this supposed to work ? Also, this would
>  > awaken everybody whenever a thread::send is done; I'm
>  > concerned about the efficiency.
>  > Please elaborate.
>
>
> Well, it depends on the application or architecture: if we're talking about io-centric worker threads (which typically reflect the number of CPUs or cores), the overhead for pipes will not be large. Have one fd per thread, and you're done writing to it. Also, for Tcl, that pipe could handle all sort of event notification.

The problem is that a generic thing like the Tcl core cannot make
restrictive assumptions on the app :-}. I remember people talking
about thousands of threads were one pipe for each one would be
unthinkable; all the more so if inter-thread communication is *not*
used in the app... which is notably often the case in the category of
high-throughput TCP servers !


>  However, reading the man pages for 'eventfd', 'signalfd', 'timerfd_create', select/epoll seems to be the way on how to realize that sort of event multiplexing / inter-thread communication. But of course, these are for kernels >= 2.6.22 only, and linux only.
>  See http://lwn.net/Articles/225714/: "It brings Linux toward a single interface for event delivery without the need for a new, complex API."
>

Very interesting... for our children :) (I have 2.4 machines around)

> For the rest of the unix world, still pipes seem to be the only way.

Yes. Pipes *or* pthread_kill when applicable.

>  And, as you were asking: select/epoll'ing the same handle will probably not work (or lead to unwanted results), but duping the pipe handles will do. But as written before, the cleanest approach for me is to have one pipe per thread.

Just checked: both duping and sharing work with select() on my 2.6.9.
So we do have a cheap broadcast channel after all, even though it
doesn't fit the purpose.

>  About the 'screaming metal': well, for the server side, I need one accept socket (true), but N reading/writing sockets (with let's say N around 512). Your proposal was probably not to have 512 threads sitting on a blocked select, right?

My proposal was to have no select at all:

   while(1) {
       s2=accept(...);
       pthread_create(worker,s2);
   }

   void worker(int fd) {
       while(1) {
           n=read(fd,...)
           if (n<=0) break;
           ...
       }
       close(fd);
   }

>
>  The only clean way to handle it is to have either one big IO/loop (what I do now, with the known limits) or to have several IO/loops, each sitting on its own (worker)thread.

Yup. The latter. But no notifier needed.

>  Being curious if Daniel has something to add, and - what do the windows guys think? Who's actually taking care of that part? Do you plan to integrate IOCP?

IIRC David Gravereaux (Cced) maintains a branch where he rewrote all
the Windows socket code using IOCP. Maybe he can step in ?

-Alex