[ros-kernel] Re: IOCP and fibers - not strictly related to ROS

[Phillip S. <ps...@cf...>]

I don't know a whole lot about fibers, but I think they are implemented on 
top of threads, so you should not be using one per client.  If you are 
using IOCPs and a worker thread pool, why do you want to mess with 
fibers?  I wrote an ftp server a few years ago using IOCPs that I think is 
a good example of how to use them.  I'm going to put it on a web site in 
the next few days as I know several others on the list are interested in 
it.  Anyhow, why do you want to use fibers?  I defined a socket abstract 
base class, and used a pointer to it as the completion key, and derive 
classes from it to handle control and data connections.  The GQCS loop 
casts the key to a socket pointer, and calls a virtual OnRecv() or OnSend() 
method.  I also make use of zero copy overlapped IO requests and memory 
mapped files to reach performance levels of TransmitFile().  Specifically, 
on a dual PII-233 I was able to saturate 2 ethernet nics, one at 10,000 
KB/s, the other at 5,500 KB/s with around 50% of the cpu.

At 01:44 AM 9/25/2001 +0100, you wrote:
>Sorry for the OT post, but you guys seem to have a good overall sense of the
>internals of NT and I wondered if you might be able to help with development
>of an idea I've got.
>
>My idea, in outline, is to facilate the use of IOCP by combining them with
>fibers.  Effectively what you do is this, using a sockets server as an
>example:
>
>* Create a single IOCP and a thread pool as per usual.
>* For each client, create a fiber.  The fiber will run as the client
>connection, and you can then just treat this as normal piece of linear code.
>If you are reasonably careful then you can get away with a teeny stack for
>this (say a couple of pages).  You associate the socket handle with the IOCP
>using the fiber as the key:
>         CreateIoCompletionPort((HANDLE)client_socket, g_CompletionPort,
>(DWORD)GetCurrentFiber(), NUM_THREADS);
>
>* Process the client connection.  When you need to do some I/O you do
>something like:
>
>         BOOL bRet = ReadFile(...)
>         if (!bRet) {
>                 DWORD dwError = GetLastError();
>                 if (dwError == ERROR_IO_PENDING) {
>                         GetQueuedCompletionStatus(...)
>                         SwitchToFiber(CompletionKey);
>                 }
>         }
>
>OK, so now this *nearly* works.  The problem is that after calling
>ReadFile() I/O can complete immediately.  In this case, another thread may
>pick up the completion from GQCS and switch to the fiber BEFORE the first
>thread has finished executing GetLastError() and SwitchToFiber.  In this
>case, the fiber restart address is bound to wrong.  Now I don't have an easy
>solution to this.  Then one that I have come up with is somewhat nasty:
>
>Before calling ReadFile, "fiddle" the fiber restart address to point to just
>after the !bRet code block and store interesting registers.  In other words,
>set the correct fiber state for a restart BEFORE calling ReadFile.  Now we
>can't use the stack after calling ReadFile because the fiber switch from the
>completed IO on a different thread will have started to use the same stack
>again.  To get around this, immediately after if (!bRet) { we switch to a
>small, safe stack.  You only need one of these stacks per worker thread, so
>this is an acceptable overhead.  The stack pointers can be saved in TLS.  We
>then continue with the GetLastError(), ... code.  We also can't call
>SwitchToFiber as this will save the current fiber state, destroying the
>carefully set up restart information.  So what we do is just a fiber state
>restore, without a save.  So I have split the SwitchToFiber into two halves;
>firstly a save and secondly a restore, with intervening code.  This neatly
>sidesteps my restart problem, but is just possibly a bit too low level!
>
>I can't see a problem with this method, but it certainly isn't supported -
>this kind of meddling with fibers just isn't catered for in the API.
>However, looking at the idea and not the implementation, you get the
>scalability of IOCP without paying any extra from what you would have done
>by writing the code in a stateful sense in the first place.  Fiber switches
>don't even involve a kernel call, and end up as being 20-30 instructions at
>most.  I've already got my own fiber switching code that I wrote a while
>back (I was doing some work with VenturCom's real-time extensions to NT -
>they implemented a new subsystem to do this that basically schedules its own
>threads ahead of anyone else's in response to device or timer interrupts -
>I'm not clear on all the details here, but it works very well - and for my
>work, I needed fibers, which weren't implemented in their subsystem), so I
>am planning on using that to handle the switches.
>
>My questions are:
>
>(1) Can anyone see a problem with this?
>(2) Is there any *portable* (i.e. API based way) of doing this?
>(3) If not, is there any portable way of switching stacks/exception records
>in NT other than fibers?  Basically that's all I need - a portable way of
>swapping stacks and exception records around.  I can take care of the
>details of save/restore registers without getting the OS involved, as this
>is just normal application level code.
>
>Thanks in advance,
>
>Mark

====================================================
= To remove yourself from this mailing list, go to =
= http://www.reactos.com/home/mailing.html         =
====================================================