Linux Scalability Effort

Hi Bill,
I'm pretty sure there's no exported interface in the standard kernels.
I'm sure I've seen code out there to do it somewhere. I thought it might have
been in a SGI patch. SGI guys, do you know ?

Tim

On Mon, Feb 12, 2001 at 03:11:15PM -0500, Bill Hartner wrote:
> 
> > On Mon, Feb 12, 2001 at 01:10:21PM -0500, Bill Hartner wrote:
> > > Is it exported thru the proc file system ?
> >
> > AFAICS not.
> >
> > >
> > > For performance analysis, I would like to set the processor affinity
> > > for process X.  I do not have the option of modifying the source code
> > > and rebuilding the program for process X.  Therefore, I need some
> > > external means for setting processor affinity.
> > >
> > > I suppose one way of setting processor affinity for some process X
> > > is to create a device driver and send an ioctl down to it with a the
> > > PID of process X and a set of processors to set affinity to it.
> >
> > I don't think that is a good idea.  Because we have procfs for setting
> > process specific data.  A file in /proc/<pid>/ (e.g. cpu_affinity or
> > overloading cpu) looks a lot cleaner to me.
> >
> >    Christoph
> 
> I agree that /proc/<pid> is the correct place for this.  I was just
> needing a hack to do some performance analysis and wanted to avoid
> reinventing the wheel if someone has done a patch to export processor
> affinity as I need to use it.
> 
> Thanks,
>      Bill
> 
> 
> 
> 
> _______________________________________________
> Lse-tech mailing list
> Lse-tech@...
> http://lists.sourceforge.net/lists/listinfo/lse-tech

-- 
Tim Wright - timw@... or timw@... or twright@...
IBM Linux Technology Center, Beaverton, Oregon
Interested in Linux scalability ? Look at http://lse.sourceforge.net/
"Nobody ever said I was charming, they said "Rimmer, you're a git!"" RD VI

Tim Wright wrote:
> 
> Hi Bill,
> I'm pretty sure there's no exported interface in the standard kernels.
> I'm sure I've seen code out there to do it somewhere. I thought it might have
> been in a SGI patch. SGI guys, do you know ?
> 

I'm cc:ing Nick Pollitt on this mail who did the patch under discussion.
No, there is no interface in the standard kernel. Nick, can you please
elaborate?

cheers,

--------------------------------------------------------------------------
Rajagopal Ananthanarayanan ("ananth")
Member Technical Staff, SGI.
--------------------------------------------------------------------------

The interface is in sys_prctl, and uses cpus_allowed in the task struct.  The patch
I have (based partly on work by Ingo and Dimitris) was posted here several weeks
ago and should be in the email archives - but I'll attach the latest incarnation 
here, which includes a change to the scheduler by IBM that is necessary.

Nick

On Mon, Feb 12, 2001 at 02:30:50PM -0800, Rajagopal Ananthanarayanan wrote:
> Tim Wright wrote:
> > 
> > Hi Bill,
> > I'm pretty sure there's no exported interface in the standard kernels.
> > I'm sure I've seen code out there to do it somewhere. I thought it might have
> > been in a SGI patch. SGI guys, do you know ?
> > 
> 
> I'm cc:ing Nick Pollitt on this mail who did the patch under discussion.
> No, there is no interface in the standard kernel. Nick, can you please
> elaborate?
> 
> cheers,
> 
> --------------------------------------------------------------------------
> Rajagopal Ananthanarayanan ("ananth")
> Member Technical Staff, SGI.
> --------------------------------------------------------------------------

Rajagopal Ananthanarayanan wrote:
> 
> Tim Wright wrote:
> >
> > Hi Bill,
> > I'm pretty sure there's no exported interface in the standard kernels.
> > I'm sure I've seen code out there to do it somewhere. I thought it might have
> > been in a SGI patch. SGI guys, do you know ?
> >
> 
> I'm cc:ing Nick Pollitt on this mail who did the patch under discussion.
> No, there is no interface in the standard kernel. Nick, can you please
> elaborate?

I did a patch against 2.4.1.  It creates /proc/<pid>/cpus_allowed,
which is exactly what you're after.

	http://www.uow.edu.au/~andrewm/linux/#cpus_allowed

Note that if a process calls schedule() in state TASK_RUNNING,
it does not honour new settings of cpus_allowed.  A process
must pass through a sleeping state before is picks up the
new setting.  I think this is reasonable behaviour, actually.

If you grab http://www.uow.edu.au/~andrewm/linux/zc.tar.gz
you'll find a little proglet in there called `run_on_cpu'
which allows you to run another program on a selected CPU set.

Andrew, why not simply set the need_resched flag + (call schedule or send
IPI for remote cpus) ?
That should get ride of the sleep requirements.
On Nick's patch I didn't get the requirement in
#define can_schedule(.......)    to check for (cpu == p->processor)
We did this some time ago through a new system call on the calling process
and we didn't
have any problems with the need_resched + force reschedule approach.

Hubertus Franke
Enterprise Linux Group (Mgr),  Linux Technology Center (Member Scalability)
, OS-PIC (Chair)
email: frankeh@...
(w) 914-945-2003    (fax) 914-945-4425   TL: 862-2003



Andrew Morton <andrewm@... on 02/13/2001
10:27:42 PM

Sent by:  lse-tech-admin@...


To:   Rajagopal Ananthanarayanan <ananth@...>
cc:   timw@..., Bill Hartner/Austin/IBM@...,
      lse-tech@..., npollitt@...
Subject:  Re: [Lse-tech] setting processor affinity for a process.



Rajagopal Ananthanarayanan wrote:
>
> Tim Wright wrote:
> >
> > Hi Bill,
> > I'm pretty sure there's no exported interface in the standard kernels.
> > I'm sure I've seen code out there to do it somewhere. I thought it
might have
> > been in a SGI patch. SGI guys, do you know ?
> >
>
> I'm cc:ing Nick Pollitt on this mail who did the patch under discussion.
> No, there is no interface in the standard kernel. Nick, can you please
> elaborate?

I did a patch against 2.4.1.  It creates /proc/<pid>/cpus_allowed,
which is exactly what you're after.

     http://www.uow.edu.au/~andrewm/linux/#cpus_allowed

Note that if a process calls schedule() in state TASK_RUNNING,
it does not honour new settings of cpus_allowed.  A process
must pass through a sleeping state before is picks up the
new setting.  I think this is reasonable behaviour, actually.

If you grab http://www.uow.edu.au/~andrewm/linux/zc.tar.gz
you'll find a little proglet in there called `run_on_cpu'
which allows you to run another program on a selected CPU set.

_______________________________________________
Lse-tech mailing list
Lse-tech@...
http://lists.sourceforge.net/lists/listinfo/lse-tech

Hubertus Franke wrote:
> 
> Andrew, why not simply set the need_resched flag + (call schedule or send
> IPI for remote cpus) ?
> That should get ride of the sleep requirements.

The problem is that if a task calls schedule() in state TASK_RUNNING
it will simply keep on running.  On the same CPU.  It doesn't inspect
cpus_allowed.  The only place where we can migrate a task between CPUs
due to a changed cpus_allowed is in reschedule_idle().  So the sleep
is a way of making this happen.

Rusty's hotplug CPU patch added a cpus_allowed test in schedule(),
but I'm not sure this is warranted for such a rare operation.

Hi Andrew,

I'm interested in hotplug CPU support for Linux. Where can I find
Rusty's hotplug CPU patch?

Thanks,

Andrew Morton wrote:
> 
> Hubertus Franke wrote:
> >
> > Andrew, why not simply set the need_resched flag + (call schedule or send
> > IPI for remote cpus) ?
> > That should get ride of the sleep requirements.
> 
> The problem is that if a task calls schedule() in state TASK_RUNNING
> it will simply keep on running.  On the same CPU.  It doesn't inspect
> cpus_allowed.  The only place where we can migrate a task between CPUs
> due to a changed cpus_allowed is in reschedule_idle().  So the sleep
> is a way of making this happen.
> 
> Rusty's hotplug CPU patch added a cpus_allowed test in schedule(),
> but I'm not sure this is warranted for such a rare operation.
> 
> _______________________________________________
> Lse-tech mailing list
> Lse-tech@...
> http://lists.sourceforge.net/lists/listinfo/lse-tech

-- 
Jun U Nakajima
Core OS Development
SCO/Murray Hill, NJ
Email: jun@..., Phone: 908-790-2352 Fax: 908-790-2426

Jun Nakajima wrote:
> 
> Hi Andrew,
> 
> I'm interested in hotplug CPU support for Linux. Where can I find
> Rusty's hotplug CPU patch?
> 

http://www.uwsg.indiana.edu/hypermail/linux/kernel/0102.0/0751.html

Thanks, Andrew.

Someone else might be intested in this, and I looked at the code. To be
accurate, this is not "hotplug" but "hotswap", as Rusty is saying. All
the CPUs need to be first online at boot time. Then one can place a CPU
in offline state, or the idle loop with interrupt disabled. Placing a
CPU in online state simply means to get it out of the loop, instead of
resetting it (which some other Unix does).


Andrew Morton wrote:
> 
> Jun Nakajima wrote:
> >
> > Hi Andrew,
> >
> > I'm interested in hotplug CPU support for Linux. Where can I find
> > Rusty's hotplug CPU patch?
> >
> 
> http://www.uwsg.indiana.edu/hypermail/linux/kernel/0102.0/0751.html
> 
-- 
Jun U Nakajima
Core OS Development
SCO/Murray Hill, NJ
Email: jun@..., Phone: 908-790-2352 Fax: 908-790-2426