Menu
▾
▴
[ros-kernel] RE: Re: IOCP and fibers - not strictly related to ROS
|
From: Mark W. <ma...@np...> - 2001-09-25 01:24:16
|
Fibers are just a stack and register context that you can switch out on
demand. They are very lightweight self-scheduled threads. They therefore
have ideal properties for sitting with IOCP. The OnRecv and OnSend
functions require you to keep track of the current state of a connection,
and make only very light use of the stack. IMO a stack is a pretty
reasonable place to keep all of this data - having to store it yourself in a
connection state structure or class is a pain. Stacks were designed to make
your coding life easier, IOCP basically mean that you can't use them
anywhere near as easily as intended. Keeping the current execution location
also saves you from storing the current state in a variable, and having to
choose the next bit of code to execute based on that. It's not entirely a
bad way of doing it, but consider an SMTP conversation (outgoing):
connect
read response
send HELO
read response
send MAIL
read response
send RCPT
...
This is basically a linear process. Breaking the code down into little
chunks doesn't make a huge amount of sense. With my method, the code path
above would be virtually unchanged, all you do is fiddle with `read
response' and `send' to work as listed below and you get IOCP without the
extra coding hassle. The other important driving factor behind this is I
have a fair bit of legacy code that I want to convert to use IOCP. If I
only have to change a few functions, then I'm much happier than if I have to
rework the whole thing! The main other point is that my method, although it
seems a bit of a fiddle at first instance, actually adds no more weight than
keeping track of all this yourself. It's just that the data for the
connection is stored on the stack, not in a class.
Just on the subject of IOCP they do seem to be a total hack (`set the low
bit of the event handle in the overlapped structure to 1 to prevent IOCP
notification' is my favourite bit). What is it that means that threads have
so much weight in NT anyway?
Nice FTP server by the way! I managed to get the source code out of the
.bz2 file you sent, but not the support libraries - I would definitely be
interested in seeing the whole thing!
Thanks,
Mark
> -----Original Message-----
> From: Phillip Susi [mailto:ps...@cf...]
> Sent: 25 September 2001 01:59
> To: ros...@re...
> Subject: [ros-kernel] Re: IOCP and fibers - not strictly related to ROS
>
>
> I don't know a whole lot about fibers, but I think they are
> implemented on
> top of threads, so you should not be using one per client. If you are
> using IOCPs and a worker thread pool, why do you want to mess with
> fibers? I wrote an ftp server a few years ago using IOCPs that I
> think is
> a good example of how to use them. I'm going to put it on a web site in
> the next few days as I know several others on the list are interested in
> it. Anyhow, why do you want to use fibers? I defined a socket abstract
> base class, and used a pointer to it as the completion key, and derive
> classes from it to handle control and data connections. The GQCS loop
> casts the key to a socket pointer, and calls a virtual OnRecv()
> or OnSend()
> method. I also make use of zero copy overlapped IO requests and memory
> mapped files to reach performance levels of TransmitFile().
> Specifically,
> on a dual PII-233 I was able to saturate 2 ethernet nics, one at 10,000
> KB/s, the other at 5,500 KB/s with around 50% of the cpu.
>
> At 01:44 AM 9/25/2001 +0100, you wrote:
> >Sorry for the OT post, but you guys seem to have a good overall
> sense of the
> >internals of NT and I wondered if you might be able to help with
> development
> >of an idea I've got.
> >
> >My idea, in outline, is to facilate the use of IOCP by combining
> them with
> >fibers. Effectively what you do is this, using a sockets server as an
> >example:
> >
> >* Create a single IOCP and a thread pool as per usual.
> >* For each client, create a fiber. The fiber will run as the client
> >connection, and you can then just treat this as normal piece of
> linear code.
> >If you are reasonably careful then you can get away with a teeny
> stack for
> >this (say a couple of pages). You associate the socket handle
> with the IOCP
> >using the fiber as the key:
> > CreateIoCompletionPort((HANDLE)client_socket, g_CompletionPort,
> >(DWORD)GetCurrentFiber(), NUM_THREADS);
> >
> >* Process the client connection. When you need to do some I/O you do
> >something like:
> >
> > BOOL bRet = ReadFile(...)
> > if (!bRet) {
> > DWORD dwError = GetLastError();
> > if (dwError == ERROR_IO_PENDING) {
> > GetQueuedCompletionStatus(...)
> > SwitchToFiber(CompletionKey);
> > }
> > }
> >
> >OK, so now this *nearly* works. The problem is that after calling
> >ReadFile() I/O can complete immediately. In this case, another
> thread may
> >pick up the completion from GQCS and switch to the fiber BEFORE the first
> >thread has finished executing GetLastError() and SwitchToFiber. In this
> >case, the fiber restart address is bound to wrong. Now I don't
> have an easy
> >solution to this. Then one that I have come up with is somewhat nasty:
> >
> >Before calling ReadFile, "fiddle" the fiber restart address to
> point to just
> >after the !bRet code block and store interesting registers. In
> other words,
> >set the correct fiber state for a restart BEFORE calling
> ReadFile. Now we
> >can't use the stack after calling ReadFile because the fiber
> switch from the
> >completed IO on a different thread will have started to use the
> same stack
> >again. To get around this, immediately after if (!bRet) { we switch to a
> >small, safe stack. You only need one of these stacks per worker
> thread, so
> >this is an acceptable overhead. The stack pointers can be saved
> in TLS. We
> >then continue with the GetLastError(), ... code. We also can't call
> >SwitchToFiber as this will save the current fiber state, destroying the
> >carefully set up restart information. So what we do is just a
> fiber state
> >restore, without a save. So I have split the SwitchToFiber into
> two halves;
> >firstly a save and secondly a restore, with intervening code.
> This neatly
> >sidesteps my restart problem, but is just possibly a bit too low level!
> >
> >I can't see a problem with this method, but it certainly isn't
> supported -
> >this kind of meddling with fibers just isn't catered for in the API.
> >However, looking at the idea and not the implementation, you get the
> >scalability of IOCP without paying any extra from what you would
> have done
> >by writing the code in a stateful sense in the first place.
> Fiber switches
> >don't even involve a kernel call, and end up as being 20-30
> instructions at
> >most. I've already got my own fiber switching code that I wrote a while
> >back (I was doing some work with VenturCom's real-time extensions to NT -
> >they implemented a new subsystem to do this that basically
> schedules its own
> >threads ahead of anyone else's in response to device or timer
> interrupts -
> >I'm not clear on all the details here, but it works very well -
> and for my
> >work, I needed fibers, which weren't implemented in their
> subsystem), so I
> >am planning on using that to handle the switches.
> >
> >My questions are:
> >
> >(1) Can anyone see a problem with this?
> >(2) Is there any *portable* (i.e. API based way) of doing this?
> >(3) If not, is there any portable way of switching
> stacks/exception records
> >in NT other than fibers? Basically that's all I need - a portable way of
> >swapping stacks and exception records around. I can take care of the
> >details of save/restore registers without getting the OS
> involved, as this
> >is just normal application level code.
> >
> >Thanks in advance,
> >
> >Mark
>
> ====================================================
> = To remove yourself from this mailing list, go to =
> = http://www.reactos.com/home/mailing.html =
> ====================================================
>
>
====================================================
= To remove yourself from this mailing list, go to =
= http://www.reactos.com/home/mailing.html =
====================================================
|
