Thanks for the reply, what I am looking at is not only commodity,
but cheap ... very cheap.Actually I don't consider
latency too big a problem, providing intelligent memory allocation is
available, and shared data structures are avoided
as much as possible, AND the system supports pipelined access (ie I can
issue another memory fetch before the first one
has been satisfied - this tends to lessen latency issues)...
As for the paper, I mentioned this because it addresses memory
allocation accross a network of systems. I actually dont
care if there is one kernel image (would be impossible without some sort
of micro-kernel on each machine) or several, as long as
programs do not need to be rewritten for the system. It is a case of
taking ideas that fit etc... (Unless somebody already has a workable
system that is).
Having read around a bit, there are various suggestions that
user-mode-linux could be used to provide something similar, running
on top of a linux kernel on each machine. What would be needed then
would be an abstraction layer providing global file, memory and thread
management to sit between the shared user-mode-linux kernel and each
machines individual kernel.
Keean Schupke, Computer Support Group,
Department of Electrical & Electronic Engineering,
Imperial College London.
Bogdan Costescu wrote:
>On Mon, 14 Apr 2003, Keean Schupke wrote:
>>It seems to me with the advent of 10Gb over copper networking that
>>reasonable memory fetch times could be had...
>One usually overlooks the most important factor when dealing with remote
>memory: communication latency. The fact that 1 Gbit networking is
>theoretically capable of transferring 10 times more information than 100
>Mbit networking per time unit does not indicate that the first bit of
>memory transfer arrives at the destination 1 microsecond rather than 10
>microseconds after the transfer was started. Using Jumbo frames, one can
>transfer more data per packet, but the latency for sending/receiving each
>packet has to be paid _per packet_. I have not seen (but didn't look hard
>enough either) latency figures for 10 Gbit equipment; but speaking
>strictly on "widely" used interconnect that can be bought in quantities
>now, 1 Gbit does not compare favorable to SCI, Myrinet, GigaNet, etc. and
>I somehow doubt that 10 GBit would make wonders in this area...
>So, after this rant, I would say that you need to define the meaning
>of "reasonable memory fetch times".
>>... using a switched ethernet, and a bunch of ordinary PCs.
>In 5 years time, maybe. 10 Gbit networking is not widely available now,
>which makes it expensive. And ordinary PCs these would be not - "ordinary"
>PCs using 64/66 PCI are barely capable today of offering high performance
>with 1 Gbit networking - otherwise why would Intel shortcut PCI to connect
>the network chips directly to the northbridge.
>>I am interested here in a single kernel not a clustering approach, and
>>was wondering if any work had already been done on this? I know some
>>non-linux work has been done
>>(http://www.cs.washington.edu/homes/levy/gms/) on networked memory
>>systems, but has anything linux-specific been done?
>I'm afraid that you contradict yourself here: one of the last pages from
>the first publication mentioned on this page (Dec. 95) says:
>"All of the kernels must trust each other in various ways".
>To me this indicates that the system you are talking about is indeed a
>clustering approach and not a single kernel.
>Are you looking to solve a specific problem or just to have a COTS NUMA ?
>(COTS=comodity of the shelf)