Re: [Lse-tech] 2.4.5 scheduler cpu affinity

[Mike K. <mkr...@se...>]

On Tue, Jul 03, 2001 at 10:50:06PM +0200, Andi Kleen wrote:
> On Tue, Jul 03, 2001 at 09:15:38AM -0700, Mike Kravetz wrote:
> 
> Yes. Both CPUs get about 50% load when just a single CPU should be fully 
> loaded (this load estimation is checked with top which disturbs the results
> a bit, but it seems to match to observation of a vmstat 5 which should not
> disturb anything) 
> 
> > In the context switching test of LMbench, you can run with only 2 tasks
> > passing a token between each other (via a pipe).  Therefore, at most
> > two tasks are on the runqueue.  I have run this on an 8 CPU system.
> > 
> > At first thought, you would think the two tasks would stay confined to
> > 2 CPUs of the 8 way.  However, in reality the tasks are 'round robined'
> > among all the CPUs.
> 
> Ok, it's not just me then.
> 
> > 
> > I thought I understood why this was happening.  However, when I wrote
> > it down it didn't make sense.  I'll try to look into it more.  As a
> > 'hint' my theory had to do with the 'awakened' task's CPU always being
> > non-idle and reschedule_idle() choosing the CPU with the longest idle
> > time for the awakened task.
> 
> I got to a similar theory independently.
> 

I looked into this a little more.  As stated in my previous
note, it doesn't appear that reschedule_idle() is making bad
decisions.  You can explore this yourself by trying to come
up with a scenario where this is the case.  However, I was
unable to come up with such a scenario.  If reschedule_idle()
was not to blame, what was?

In my next theory I assumed that reschedule_idle() was choosing
the correct target CPU with respect to task affinity.  However,
before the target CPU could schedule the task, another CPU would
schedule the task (on the 'not as appropriate' CPU).  Remember
that reschedule_idle() does not actually schedule tasks.  It
simply 'informs' the target CPU that it should run schedule() to
schedule the task.

To test this theory, I made a few modifications to the scheduler
code as follows:
- Added a new field 'saved_cpus_allowed' to the task structure
- In reschedule_idle(), after choosing a target CPU save off the
  cpus_allowed field in saved_cpus_allowed of the target task.
  Then overwrite the cpus_allowed field such that the only CPU
  the task will run on is the one chosen by reschedule_idle().
- When schedule() schedules a task, restore the original contents
  of the cpus_allowed field.
  
After these modifications, I once again ran the 2 task context
switching test of LMbench on an 8 way system.  Now, the two tasks
were confined to 2 of the 8 CPUs.  This is different that the
'default' behavior where the tasks are 'round robined' among all
8 CPUs.

This seems to support my 'theory' that reschedule_idle() is making
appropriate decisions and the behavior we are seeing is caused by
other CPUs scheduling the task before the 'appropriate' CPU.

Additional thoughts/Comments,
-- 
Mike Kravetz                                 mkr...@se...
IBM Linux Technology Center