Re: [Sigsafe-devel] Re: advogato diary

[Scott L. <sl...@sl...>]

On Feb 17, 2004, at 1:18 PM, Pierre Phaneuf wrote:

> Scott Lamb wrote:
>
>> If you're not using thread cancellation, I think that trick is safe.
>> But if you're using a thread per connection (and not already using
>> poll()), adding a poll() before every read() or write() would be
>> slower - twice the system call overhead. It might be too slow to be
>> acceptable.
>
> If you don't use poll(), I suppose you're kind of screwed. But  
> personally, I think that if you use block on read() and use threads to  
> cope with that, you're already screwed, for other reasons. :-)

Well, you won't have the fastest code around, but a lot of tremendously  
popular code does it this way: Apache, most Java-based servers, etc. I  
imagine many of their authors and users would disagree with you. ;)

> System call overhead nowadays is almost nil, so that's not a very good  
> reason.

In my tests, it seems significant on some platforms. In the  
tests/bench_read_* microbenchmark, I found that:

- On Linux 2.6 with a 1GHz Athlon, bench_read_raw takes ~9 seconds and  
bench_read_select takes ~21.
- On OS X 10.3 with a 1GHz G4, bench_read_raw takes ~140 seconds and  
bench_read_select takes ~200.

So I think that supports your assertion on Linux - it's a threefold  
speed decrease but it did an awful lot of system calls in that time,  
enough so that I don't think it's significant in the context of the  
larger program.

But on OS X with a slightly-faster processor, both tests ran an order  
of magnitude slower. So I think that extra 40% is quite significant.

These results match my expectations - Linux is well-known for having  
fast system calls and context switches, while OS X is based on the Mach  
microkernel.

Is that convincing or do I need to do a different benchmark? I suppose  
I could take a real thread-based webserver and butcher it to double the  
number of IO system calls it makes. I'm fairly confident I'd find a  
substantial change in throughput.

Arguably you just shouldn't use platforms so slow, but again, people do  
anyway. (You can have my Mac when you pry it from my cold, dead  
hands...) Besides, if you start saying "my platform had better do X  
really well", before long you'll be left with no platforms, because in  
my experience no platform does _everything_ well.

> Also, I think you might be wrong about the setjmp() trick (which would  
> be the proper trick for read()), read() will return EINTR only if the  
> I/O operation did not complete

I've added (since the 0.1.0 release) a test for this to race_checker.  
And it returns the result I expected - the jump can happen after the  
system call completes. (With the setjmp() method, it doesn't really use  
EINTR at all.) I also added some more description of this to the  
documentation.

>  (look at the implementation of fread() or fwrite() in libc).

Which libc are you talking about? glibc? I don't see any code that uses  
longjmp in the libc/libio, which is where fread seems to live. I see a  
lot of IO_jump_t stuff, but that seems to just be a virtual function  
table rather than a jump buffer for longjmp()/setjmp(). In libioP.h:


  * The _IO_FILE type is used to implement the FILE type in GNU libc,
  * as well as the streambuf class in GNU iostreams for C++.
  * These are all the same, just used differently.
  * An _IO_FILE (or FILE) object is allows followed by a pointer to
  * a jump table (of pointers to functions).  The pointer is accessed
  * with the _IO_JUMPS macro.  The jump table has a eccentric format,
  * so as to be compatible with the layout of a C++ virtual function  
table.
  * (as implemented by g++).  When a pointer to a streambuf object is
  * coerced to an (_IO_FILE*), then _IO_JUMPS on the result just
  * happens to point to the virtual function table of the streambuf.
  * Thus the _IO_JUMPS function table used for C stdio/libio does
  * double duty as the virtual function table for C++ streambuf.

...

/* The 'sysread' hook is used to read data from the external file into
    an existing buffer.  It generalizes the Unix read(2) function.
    It matches the streambuf::sys_read virtual function, which is
    specific to this implementation. */
typedef _IO_ssize_t (*_IO_read_t) __PMT ((_IO_FILE *, void *,  
_IO_ssize_t));
#define _IO_SYSREAD(FP, DATA, LEN) JUMP2 (__read, FP, DATA, LEN)
#define _IO_WSYSREAD(FP, DATA, LEN) WJUMP2 (__read, FP, DATA, LEN)


I did see this one comment in fileops.c:


           /* These must be set before the sysread as we might longjmp  
out
              waiting for input. */


but that doesn't lead me to believe they put a lot of thought into it.

>  If you're using the other calls you mention, you should be using the  
> write() trick and you already don't have the race. The trick doesn't  
> work in Cygwin, but handily, Windows doesn't have signals! So you're  
> still good (just put the code in and watch it never do the unsupported  
> thing by never getting a signal). But seeing that you actually report  
> that the trick doesn't work with Cygwin would kind of assume that you  
> can get signals on Cygwin, so I might be wrong here.

You can get signals on cygwin. They do it with some weird tricks I  
don't really understand. If you're curious, it's described here:

<http://sources.redhat.com/cgi-bin/cvsweb.cgi/src/winsup/cygwin/how- 
signals-work.txt?rev=1.11&content-type=text/x-cvsweb- 
markup&cvsroot=src>

But if you look around in that same directory, it's pretty obvious that  
it's not safe to longjmp() out of a signal handler during select() as  
well as other functions. They use the C++ guard/monitor pattern, so  
they'd probably have to implement forced unwinding on longjmp to make  
it work right.

>> btw, feel free to ask anything like this on the sigsafe-devel mailing
>> list. I'd like to get a discussion going on there.
>
> I sent this to the list, but I'm not really interesting on joining  
> that list (I'm on too many already). You can keep on Cc'ing me on this  
> thread though.

Works for me.

>
> -- 
> Pierre Phaneuf
> http://advogato.org/person/pphaneuf/
> "I am denial, guilt and fear -- and I control you"

Thanks. I appreciate the feedback.

Scott Lamb