Linux Scalability Effort

On Tue, Oct 01, 2002 at 04:28:56PM +1000, Mark Goodwin wrote:
> The read handler for /proc/discontig does not scale. The problem
> is get_discontig_info() walks pg_data_t->node_mem_map on each node
> calculating the stats, but this is clearly not scalable on very large
> memory systems. The best fix I can think of is to restrict the output
> to Node, Total and Free columns only - the Free statistic is scalable
> because per-node zone allocation dynamically maintains the free_pages
> count.
> Would anyone miss the Reserved, Swap, Slab and Active columns if we
> just deleted them?
> thanks
> -- Mark Goodwin
> SGI Engineering.

Could you point me to the kernel patch you're using that contains
/proc/discontig?

Thanks.
Bill

> The read handler for /proc/discontig does not scale. The problem

is this some ia64 special thing? 
Does /proc/meminfo.numa in 2.5-mm do what you want?
If there's more info that you want, and it's already tracked,
it would be trivial to add it.

M.

On Mon, 30 Sep 2002, Martin J. Bligh wrote:

> > The read handler for /proc/discontig does not scale. The problem
> 
> is this some ia64 special thing? 

I think the home for the discontig memory patch is:
http://sourceforge.net/projects/discontig

and the specifc patch for the proc info is :
http://easynews.dl.sourceforge.net/sourceforge/discontig/discontig-proc_info-2.4.18.diff

> Does /proc/meminfo.numa in 2.5-mm do what you want?

we're using 2.4.18 and .19 at present. Is /proc/meminfo.numa available
for 2.4 kernels?

> If there's more info that you want, and it's already tracked,
> it would be trivial to add it.

I don't think active, reserved (etc) page stats are tracked dynamically
(that's why these fields in /proc/discontig are not scalable).

-- Mark

On Tue, Oct 01, 2002 at 04:51:37PM +1000, Mark Goodwin wrote:
> we're using 2.4.18 and .19 at present. Is /proc/meminfo.numa available
> for 2.4 kernels?

I suggest moving development to the development kernel, though it's
very, very, very late for this.


On Mon, 30 Sep 2002, Martin J. Bligh wrote:
>> If there's more info that you want, and it's already tracked,
>> it would be trivial to add it.

On Tue, Oct 01, 2002 at 04:51:37PM +1000, Mark Goodwin wrote:
> I don't think active, reserved (etc) page stats are tracked dynamically
> (that's why these fields in /proc/discontig are not scalable).

You could try adding the counters yourself, of course. It's not too
tough. I did a deferred coalescing patch that needed to keep track of
some statistics with similar properties and it wasn't difficult at all.
Note unbounded hold time with zone->lock held will NMI oops.

Basically the bits missing from 2.5 are reserved, cached, and slab.
Neither should be difficult to reconstruct as all changes thereof are
relatively well-contained.

Is your architecture's port running on 2.5?


Bill

On Tue, 1 Oct 2002, William Lee Irwin III wrote:
> You could try adding the counters yourself, of course. It's not too
> tough. I did a deferred coalescing patch that needed to keep track of
> some statistics with similar properties and it wasn't difficult at all.
> Note unbounded hold time with zone->lock held will NMI oops.

I was concerned doing that would add unjustified overhead to
assorted page management macros; they look rather lean. So rather than
add code in performance sentitive places, I'd rather remove code
in the discontig memory patch and only export stuff that is scalable
in /proc/discontig.

> Basically the bits missing from 2.5 are reserved, cached, and slab.
> Neither should be difficult to reconstruct as all changes thereof are
> relatively well-contained.
> 
> Is your architecture's port running on 2.5?

no. 2.4.18 and 2.4.19 at present.

-- Mark

> I think the home for the discontig memory patch is:
> http://sourceforge.net/projects/discontig

Ah, you mean the ia64 patch.
 
> and the specifc patch for the proc info is :
> http://easynews.dl.sourceforge.net/sourceforge/discontig/discontig-proc_info-2.4.18.diff

OK, thanks for that - much easier ;-)

>> Does /proc/meminfo.numa in 2.5-mm do what you want?
> 
> we're using 2.4.18 and .19 at present. Is /proc/meminfo.numa available
> for 2.4 kernels?

Should backport trivially. Attatched. Can't remember if this is the
exact version that's in 2.5 or not.

Would be nice if the ia64 people decided to play on the same planet
as the rest of us one of these days (specifically one day in October
before the freeze) .... maybe if people submitted their stuff to the 
2.5 tree you wouldn't all still be living on Mars when 2.6 came out ...

M.

On Tue, 1 Oct 2002, William Lee Irwin III wrote:
>> You could try adding the counters yourself, of course. It's not too
>> tough. I did a deferred coalescing patch that needed to keep track of
>> some statistics with similar properties and it wasn't difficult at all.
>> Note unbounded hold time with zone->lock held will NMI oops.

On Tue, Oct 01, 2002 at 05:32:41PM +1000, Mark Goodwin wrote:
> I was concerned doing that would add unjustified overhead to
> assorted page management macros; they look rather lean. So rather than
> add code in performance sentitive places, I'd rather remove code
> in the discontig memory patch and only export stuff that is scalable
> in /proc/discontig.

Well, there is a tradeoff between latency of requests serviced and the
ability to monitor workloads. In your case I suspect monitoring your
workload takes precedence, and the overhead may very well be acceptable
for the general case -- try and see.


On Tue, 1 Oct 2002, William Lee Irwin III wrote:
>> Basically the bits missing from 2.5 are reserved, cached, and slab.
>> Neither should be difficult to reconstruct as all changes thereof are
>> relatively well-contained.
>> Is your architecture's port running on 2.5?

On Tue, Oct 01, 2002 at 05:32:41PM +1000, Mark Goodwin wrote:
> no. 2.4.18 and 2.4.19 at present.

Sounds ugly. I'd try to help out your maintainer however you can so you
don't end up in a similar situation with respect to 2.6 vs. 2.7 vs. 2.8.


Bill

On Tuesday 01 October 2002 09:32, Mark Goodwin wrote:
> > Is your architecture's port running on 2.5?
>
> no. 2.4.18 and 2.4.19 at present.

That is a bit unexact.
The ia64 architecture port is at 2.5.39, but Mark means that the
SGI platform doesn't run with that version.

The reason is that there is no discontigmem-ia64 patch for 2.5, yet,
I guess...

Regards,
Erich

On Tue, 1 Oct 2002, Erich Focht wrote:

> On Tuesday 01 October 2002 09:32, Mark Goodwin wrote:
> > > Is your architecture's port running on 2.5?
> > no. 2.4.18 and 2.4.19 at present.
> That is a bit unexact.
> The ia64 architecture port is at 2.5.39, but Mark means that the
> SGI platform doesn't run with that version.

yes, sorry I ment the SGI platform runs on 2.4, not that ia64 arch
doesn't run on 2.5.x

> The reason is that there is no discontigmem-ia64 patch for 2.5, yet,
> I guess...

Should there be? Or is it superceded by the (obviously related)
topology and numa support in 2.5?

-- Mark

William Lee Irwin III wrote:
> 
> ...
> Basically the bits missing from 2.5 are reserved, cached, and slab.
> Neither should be difficult to reconstruct as all changes thereof are
> relatively well-contained.
> 

um.  2.5 has:

MemTotal:      4143236 kB
MemFree:       4093680 kB
MemShared:           0 kB
Buffers:          9596 kB
Cached:          13700 kB
SwapCached:          0 kB
Active:          12336 kB
Inactive:        14248 kB
HighTotal:     3277216 kB
HighFree:      3257344 kB
LowTotal:       866020 kB
LowFree:        836336 kB
SwapTotal:     3951844 kB
SwapFree:      3951844 kB
Dirty:              60 kB
Writeback:           0 kB
Mapped:           5916 kB
Slab:             5224 kB
Committed_AS:     4160 kB
PageTables:        260 kB
ReverseMaps:      3821

So we have cache, slab and swapcache there.  It's all done with
the lockless per-CPU accounting.

But no reserved page accounting there.  It's easy to add, but
I'd rather delete the obnoxious PageReserved altogether.

On Tue, Oct 01, 2002 at 01:18:11AM -0700, Andrew Morton wrote:
> So we have cache, slab and swapcache there.  It's all done with
> the lockless per-CPU accounting.
> But no reserved page accounting there.  It's easy to add, but
> I'd rather delete the obnoxious PageReserved altogether.

I thought he had all of these in mind as per-node statistics.


Bill

On Tuesday 01 October 2002 09:56, Mark Goodwin wrote:
> > The reason is that there is no discontigmem-ia64 patch for 2.5, yet,
> > I guess...
>
> Should there be? Or is it superceded by the (obviously related)
> topology and numa support in 2.5?

Yes, there needs to be an additional discontigmem patch. SGI needs
the kernel relocation feature for booting.

Regards,
Erich

On Tue, 1 Oct 2002, William Lee Irwin III wrote:
> On Tue, Oct 01, 2002 at 01:18:11AM -0700, Andrew Morton wrote:
> > So we have cache, slab and swapcache there.  It's all done with
> > the lockless per-CPU accounting.
> > But no reserved page accounting there.  It's easy to add, but
> > I'd rather delete the obnoxious PageReserved altogether.
> 
> I thought he had all of these in mind as per-node statistics.

Well actually both system-wide and per-node, but we can always aggregate
the latter. The per-node statistics are needed on large memory numa machines
for process/memory placement tools and performance monitoring.

-- Mark

On Tue, Oct 01, 2002 at 04:51:37PM +1000, Mark Goodwin wrote:
> On Mon, 30 Sep 2002, Martin J. Bligh wrote:
> 
> > > The read handler for /proc/discontig does not scale. The problem
> > 
> > is this some ia64 special thing? 
> 
> I think the home for the discontig memory patch is:
> http://sourceforge.net/projects/discontig

That patch is so ugly that it will never make it into mainline.
Could you take a look at the work Martin and I did for 2.5? (Same
concept as that patch, but gets rid of horrible ia64-isms).

I plan to do a backport to 2.4.20-pre this week.

On Tue, Oct 01, 2002 at 05:56:59PM +1000, Mark Goodwin wrote:
> Should there be? Or is it superceded by the (obviously related)
> topology and numa support in 2.5?

2.5 has proper replacement for most of te crap in the 2.4 IA64
discontigmem patch.  But there is no support for IA64 architectures
yet.

> On Tue, Oct 01, 2002 at 05:56:59PM +1000, Mark Goodwin wrote:
>> Should there be? Or is it superceded by the (obviously related)
>> topology and numa support in 2.5?
>
> Christoph:
> 2.5 has proper replacement for most of te crap in the 2.4 IA64
> discontigmem patch.  But there is no support for IA64 architectures
> yet.
> 
> Erich:
> Yes, there needs to be an additional discontigmem patch. SGI needs
> the kernel relocation feature for booting.

You also need discontig mem within a node, don't you? In which case,
I think you need config_nonlinear if you're not going to drown in
clunks and chumps. I'm sure Daniel and I would be willing to try help 
move that forward, I had half a 2.5 port at some point - I was going
to use it for ia32 but changed my mind for complicated reasons.

But you need to start *now*!. At the very least, get the mergeable
stuff merged, otherwise your life is going to be hell as you drift
further and further from mainline ... I have no qualms about changing
things which break your out of tree patches horribly if you're not
going to make them mergeable and push them.

Erich, does your NEC machine run OK with discontigmem support for
ia64 with no patches on mainline or -mm? Or do you have a secret
stash of patches somewhere that we could see to make your machine
work?

M.

On Tuesday 01 October 2002 17:46, Martin J. Bligh wrote:
> Erich, does your NEC machine run OK with discontigmem support for
> ia64 with no patches on mainline or -mm? Or do you have a secret
> stash of patches somewhere that we could see to make your machine
> work?

Our machine rearranges the physical memory to present it more or less
linearly (and starting from 0) to the OS. Therefore we can boot with
2.5 but don't have per-node zones yet. So we need something much simpler
than SGI. In 2.4.X we use only per node zonelists for the production
kernels, but for 2.5 we'd like to have discontigmem. Kimi is currently
looking into this, I'm tied up with other stuff the next few days :-/

Regards,
Erich

>> Erich, does your NEC machine run OK with discontigmem support for
>> ia64 with no patches on mainline or -mm? Or do you have a secret
>> stash of patches somewhere that we could see to make your machine
>> work?
> 
> Our machine rearranges the physical memory to present it more or less
> linearly (and starting from 0) to the OS. Therefore we can boot with
> 2.5 but don't have per-node zones yet. So we need something much simpler
> than SGI. In 2.4.X we use only per node zonelists for the production
> kernels, but for 2.5 we'd like to have discontigmem. Kimi is currently
> looking into this, I'm tied up with other stuff the next few days :-/

OK, well please publish what you have ASAP, even if it's somewhat 
half-baked at first ... would be nice to get the stuff apart from 
discontig-inside-node merged at least - that's the hardest chunk 
of it all, I suspect.

Is there still a huge external ia64 patch for 2.5 you need, or is that
all merged now?

M.

On Tue, 01 Oct 2002 09:25:15 -0700
"Martin J. Bligh" <mbligh@...> wrote:

> >> Erich, does your NEC machine run OK with discontigmem support for
> >> ia64 with no patches on mainline or -mm? Or do you have a secret
> >> stash of patches somewhere that we could see to make your machine
> >> work?
> > 
> > Our machine rearranges the physical memory to present it more or less
> > linearly (and starting from 0) to the OS. Therefore we can boot with
> > 2.5 but don't have per-node zones yet. So we need something much simpler
> > than SGI. In 2.4.X we use only per node zonelists for the production
> > kernels, but for 2.5 we'd like to have discontigmem. Kimi is currently
> > looking into this, I'm tied up with other stuff the next few days :-/
> 
> OK, well please publish what you have ASAP, even if it's somewhat 
> half-baked at first ... would be nice to get the stuff apart from 
> discontig-inside-node merged at least - that's the hardest chunk 
> of it all, I suspect.

First, I'll just port the current discontig-core patch to 2.5.
And then I'll think about using config_nonlinear, I'm not sure
will it work on IA64 though.

> Is there still a huge external ia64 patch for 2.5 you need, or is that
> all merged now?

Not so huge. About 6k lines for 2.5.39.

Regards,
Kimi

-- 
Kimio Suganuma <k-suganuma@...>

> First, I'll just port the current discontig-core patch to 2.5.

OK ... hopefully that's not the scary bit with all the clumps
and chunks and everything? If so, I'd suggest you just start from
scratch, and pull out the pieces you need to make your box work.

> And then I'll think about using config_nonlinear, I'm not sure
> will it work on IA64 though.

The concept is generic, it can be made to work anywhere.
 
M.

On Tue, 1 Oct 2002, Martin J. Bligh wrote:

> > On Tue, Oct 01, 2002 at 05:56:59PM +1000, Mark Goodwin wrote:
> >> Should there be? Or is it superceded by the (obviously related)
> >> topology and numa support in 2.5?
> >
> > Christoph:
> > 2.5 has proper replacement for most of te crap in the 2.4 IA64
> > discontigmem patch.  But there is no support for IA64 architectures
> > yet.
> > 
> > Erich:
> > Yes, there needs to be an additional discontigmem patch. SGI needs
> > the kernel relocation feature for booting.
> 
> You also need discontig mem within a node, don't you? In which case,
> I think you need config_nonlinear if you're not going to drown in
> clunks and chumps. I'm sure Daniel and I would be willing to try help 
> move that forward, I had half a 2.5 port at some point - I was going
> to use it for ia32 but changed my mind for complicated reasons.

Yes, the SGI SN2/IPF platform has discontig memory both within
and between nodes. 

-- Mark

> You also need discontig mem within a node, don't you? In which case,
> I think you need config_nonlinear if you're not going to drown in
> clunks and chumps. I'm sure Daniel and I would be willing to try help 
> move that forward, I had half a 2.5 port at some point - I was going
> to use it for ia32 but changed my mind for complicated reasons.

Martin do you guys have any need for config_nonlinear? (I seem to
remember you wanting to use it for some reason)

Im keen to see it go in so we can clean up ppc64 iSeries. It would also
help sparc32.

Anton

>> You also need discontig mem within a node, don't you? In which case,
>> I think you need config_nonlinear if you're not going to drown in
>> clunks and chumps. I'm sure Daniel and I would be willing to try help 
>> move that forward, I had half a 2.5 port at some point - I was going
>> to use it for ia32 but changed my mind for complicated reasons.
> 
> Martin do you guys have any need for config_nonlinear? (I seem to
> remember you wanting to use it for some reason)

I don't ... I was going to use it to spread ZONE_NORMAL around nodes,
but that turns out to be unfeasibly hard because every driver in the
world stuffs phys pointers from ZONE_NORMAL into an unsigned long and
DMAs to them, amongst other things.
 
> Im keen to see it go in so we can clean up ppc64 iSeries. It would 
> also help sparc32.

I'd love to see it go in, it cleans up a lot of stuff ... I'll help
out where I can, and I'm sure Daniel will too. Wanna try it? If so,
I'll clean up my 2.5 port ...

M.

On Wednesday 02 October 2002 16:13, Anton Blanchard wrote:
> Martin do you guys have any need for config_nonlinear? (I seem to
> remember you wanting to use it for some reason)
>
> Im keen to see it go in so we can clean up ppc64 iSeries. It would also
> help sparc32.
>
> Anton

A clear "Yes, please!" also from us ;-)

Having this we could put a very much simplified discontigmem on top
of it, which wouldn't see the eventually scattered memblks (which we have
on IA64 any more) but nice and continuous pieces of memory for each node.=
=2E.
I know that the concept was to replace discontigmem, but there's no NUMA
infrastructure left if you remove it.

Regards,
Erich

> A clear "Yes, please!" also from us ;-)
> 
> Having this we could put a very much simplified discontigmem on top
> of it, which wouldn't see the eventually scattered memblks (which we 
> have on IA64 any more) but nice and continuous pieces of memory for 
> each node...
> I know that the concept was to replace discontigmem, but there's no 
> NUMA infrastructure left if you remove it.

That wasn't really the concept - it was to make discontigmem only used
for NUMA (and probably get a more sensible name). That's not going to
happen in 2.5 ... but nonlinear would provide a good way to do 
discontig inside NUMA nodes, and an *alternative* way to do basic
discontig for non-NUMA (especially for highly fragmented spaces that
would end up with a billion zones otherwise). If we can migrate all
the current non-NUMA discontig people, we could make discontig NUMA-only
later.

So discontig's not going away, maybe getting renamed ;-)

M.