Re: [Discontig-devel] NUMA aware __alloc_pages function

[Jack S. <st...@sg...>]

> 
> Hi kimi,
> 
> 
> On Wed, 6 Feb 2002, Kimio Suganuma wrote:
> 
> > > 2) It appears that the algorithm attempts to keep memory free on each node. If
> > >    a node runs low on memory, alloc_pages kicks the swapd before taking memory
> > >    from other nodes.
> > >
> > >    This is ok for many workloads. However, I'm concerned about workloads where
> > >    a single process attempts to use more memory than is physically present on a node.
> > >    In this case, you dont want/need kswapd involved, nor do you want a single
> > >    process to have it memory pages as long as sufficient memory is available
> > >    on other nodes.
> > >
> > >    As part of your testing, try a testcase where you:
> > >         - use "runon" to pin a process to a node
> > >         - malloc N bytes, then bzero the mallo'ed space
> > >
> > >
> > >    Measure the wallclock & cpu time required to run the test. Vary N so that
> > >    you hit both cases where memory is fully contained on a single node and
> > >    where memory crosses multiple nodes.
> >
> > I agree with you.
> > I think the best behavior of memory allocation at local memory shortage
> > depends on the numa-ratio of the platform and policy of the system
> > administration. So, I don't think it's good idea to hard code such the
> > mechanism.
> What do you mean by numa-ration ?

Typo - numa ratio. If the platform has a high penalty for remote references,
it may chose a different strategy for using remote memory than platforms
that have only a small remote access penalty.


> I think the previous mechanism (launch the swap daemon when the entire
> machine is short on memory) was hard coded as well.

True. I think the algorithm may need to be tunable. 

At a minimum, a task (possible threaded) needs to be able to 
use more memory than contained on a single node without causing
swapping or excessive system overhead.



> I also think that having a swap daemon per node will allow processes to
> get pages from their local node more often.
> 
> Thanks,
> Samuel.
 
On IRIX, the algorithms that select cpus for tasks will consider the amount of
free memory on a node as part of the selection process. This is 
another way to improve memory locality. 




-- 
Thanks

Jack Steiner    (651-683-5302)   (vnet 233-5302)      st...@sg...