[ros-kernel] IOCP and fibers - not strictly related to ROS

[Mark W. <ma...@np...>]

Sorry for the OT post, but you guys seem to have a good overall sense of the
internals of NT and I wondered if you might be able to help with development
of an idea I've got.

My idea, in outline, is to facilate the use of IOCP by combining them with
fibers.  Effectively what you do is this, using a sockets server as an
example:

* Create a single IOCP and a thread pool as per usual.
* For each client, create a fiber.  The fiber will run as the client
connection, and you can then just treat this as normal piece of linear code.
If you are reasonably careful then you can get away with a teeny stack for
this (say a couple of pages).  You associate the socket handle with the IOCP
using the fiber as the key:
	CreateIoCompletionPort((HANDLE)client_socket, g_CompletionPort,
(DWORD)GetCurrentFiber(), NUM_THREADS);

* Process the client connection.  When you need to do some I/O you do
something like:

	BOOL bRet = ReadFile(...)
	if (!bRet) {
		DWORD dwError = GetLastError();
		if (dwError == ERROR_IO_PENDING) {
			GetQueuedCompletionStatus(...)
			SwitchToFiber(CompletionKey);
		}
	}

OK, so now this *nearly* works.  The problem is that after calling
ReadFile() I/O can complete immediately.  In this case, another thread may
pick up the completion from GQCS and switch to the fiber BEFORE the first
thread has finished executing GetLastError() and SwitchToFiber.  In this
case, the fiber restart address is bound to wrong.  Now I don't have an easy
solution to this.  Then one that I have come up with is somewhat nasty:

Before calling ReadFile, "fiddle" the fiber restart address to point to just
after the !bRet code block and store interesting registers.  In other words,
set the correct fiber state for a restart BEFORE calling ReadFile.  Now we
can't use the stack after calling ReadFile because the fiber switch from the
completed IO on a different thread will have started to use the same stack
again.  To get around this, immediately after if (!bRet) { we switch to a
small, safe stack.  You only need one of these stacks per worker thread, so
this is an acceptable overhead.  The stack pointers can be saved in TLS.  We
then continue with the GetLastError(), ... code.  We also can't call
SwitchToFiber as this will save the current fiber state, destroying the
carefully set up restart information.  So what we do is just a fiber state
restore, without a save.  So I have split the SwitchToFiber into two halves;
firstly a save and secondly a restore, with intervening code.  This neatly
sidesteps my restart problem, but is just possibly a bit too low level!

I can't see a problem with this method, but it certainly isn't supported -
this kind of meddling with fibers just isn't catered for in the API.
However, looking at the idea and not the implementation, you get the
scalability of IOCP without paying any extra from what you would have done
by writing the code in a stateful sense in the first place.  Fiber switches
don't even involve a kernel call, and end up as being 20-30 instructions at
most.  I've already got my own fiber switching code that I wrote a while
back (I was doing some work with VenturCom's real-time extensions to NT -
they implemented a new subsystem to do this that basically schedules its own
threads ahead of anyone else's in response to device or timer interrupts -
I'm not clear on all the details here, but it works very well - and for my
work, I needed fibers, which weren't implemented in their subsystem), so I
am planning on using that to handle the switches.

My questions are:

(1) Can anyone see a problem with this?
(2) Is there any *portable* (i.e. API based way) of doing this?
(3) If not, is there any portable way of switching stacks/exception records
in NT other than fibers?  Basically that's all I need - a portable way of
swapping stacks and exception records around.  I can take care of the
details of save/restore registers without getting the OS involved, as this
is just normal application level code.

Thanks in advance,

Mark

====================================================
= To remove yourself from this mailing list, go to =
= http://www.reactos.com/home/mailing.html         =
====================================================