bproc-users

[BProc] page allocation problems

[Dale H. <ro...@ma...>]

Hey I'm seeing some page allocation errors.  Has anyone seen anything
like this.  Of course it a 2.6.9 vanilla kernel patched for bproc, and
myrinet GM driver is running.


swapper: page allocation failure. order:2, mode:0x20
 [<c013e28e>] __alloc_pages+0x1b3/0x358
 [<c013e458>] __get_free_pages+0x25/0x3f
 [<c01416dc>] kmem_getpages+0x21/0xc9
 [<c01423bb>] cache_grow+0xab/0x14d
 [<c01425d1>] cache_alloc_refill+0x174/0x219
 [<c0142a24>] __kmalloc+0x85/0x8c
 [<c0219da1>] alloc_skb+0x47/0xe0
 [<f8febfb8>] gmip_recv_interrupt+0x216/0x4c7 [gm]
 [<c011a76a>] load_balance+0x15c/0x170
 [<f8ff1691>] __gm_ethernet_wake_callback+0x6a/0x9c [gm]
 [<f8fde584>] gm_handle_claimed_interrupt+0x580/0x62e [gm]
 [<c025d0ae>] udp_queue_rcv_skb+0x174/0x2a4
 [<c025d6c6>] udp_rcv+0x164/0x407
 [<c023b6d8>] ip_defrag+0x112/0x1bf
 [<c0239d4d>] ip_local_deliver+0xe8/0x279
 [<c023a26d>] ip_rcv+0x38f/0x510
 [<c021ffe3>] netif_receive_skb+0x1c7/0x2a1
 [<c022013b>] process_backlog+0x7e/0x10b
 [<c022023f>] net_rx_action+0x77/0xf6
 [<f8fea40e>] gm_linux_intr+0x9c/0xac [gm]
 [<c010899d>] handle_IRQ_event+0x31/0x65
 [<c0108d0b>] do_IRQ+0x9e/0x130
 [<c0106b6c>] common_interrupt+0x18/0x20
 [<c010401e>] default_idle+0x0/0x2c
 [<c0104047>] default_idle+0x29/0x2c
 [<c01040bc>] cpu_idle+0x3f/0x58
 [<c034a895>] start_kernel+0x197/0x1d5
 [<c034a336>] unknown_bootoption+0x0/0x15c
swapper: page allocation failure. order:2, mode:0x20
 [<c013e28e>] __alloc_pages+0x1b3/0x358
 [<c013e458>] __get_free_pages+0x25/0x3f
 [<c01416dc>] kmem_getpages+0x21/0xc9
 [<c01423bb>] cache_grow+0xab/0x14d
 [<c01425d1>] cache_alloc_refill+0x174/0x219
 [<c0142a24>] __kmalloc+0x85/0x8c
 [<c0219da1>] alloc_skb+0x47/0xe0
 [<f8febfb8>] gmip_recv_interrupt+0x216/0x4c7 [gm]
 [<c011a76a>] load_balance+0x15c/0x170
 [<f8ff1691>] __gm_ethernet_wake_callback+0x6a/0x9c [gm]
 [<f8fde584>] gm_handle_claimed_interrupt+0x580/0x62e [gm]
 [<c025d0ae>] udp_queue_rcv_skb+0x174/0x2a4
 [<c025d6c6>] udp_rcv+0x164/0x407
 [<c023b6d8>] ip_defrag+0x112/0x1bf
 [<c0239d4d>] ip_local_deliver+0xe8/0x279
 [<c023a26d>] ip_rcv+0x38f/0x510
 [<c021ffe3>] netif_receive_skb+0x1c7/0x2a1
 [<c022013b>] process_backlog+0x7e/0x10b
 [<c022023f>] net_rx_action+0x77/0xf6
 [<f8fea40e>] gm_linux_intr+0x9c/0xac [gm]
 [<c010899d>] handle_IRQ_event+0x31/0x65
 [<c0108d0b>] do_IRQ+0x9e/0x130
 [<c0106b6c>] common_interrupt+0x18/0x20
 [<c010401e>] default_idle+0x0/0x2c
 [<c0104047>] default_idle+0x29/0x2c
 [<c01040bc>] cpu_idle+0x3f/0x58
 [<c034a895>] start_kernel+0x197/0x1d5
 [<c034a336>] unknown_bootoption+0x0/0x15c
 [<c013e28e>] __alloc_pages+0x1b3/0x358
 [<c013e458>] __get_free_pages+0x25/0x3f
 [<c01416dc>] kmem_getpages+0x21/0xc9
 [<c0142239>] alloc_slabmgmt+0x55/0x5f
 [<c01423bb>] cache_grow+0xab/0x14d
 [<c01425d1>] cache_alloc_refill+0x174/0x219
 [<c0142a24>] __kmalloc+0x85/0x8c
 [<c0219da1>] alloc_skb+0x47/0xe0
 [<f8febfb8>] gmip_recv_interrupt+0x216/0x4c7 [gm]
 [<c0106b6c>] common_interrupt+0x18/0x20
 [<f8ff1691>] __gm_ethernet_wake_callback+0x6a/0x9c [gm]
 [<f8fde584>] gm_handle_claimed_interrupt+0x580/0x62e [gm]
 [<c0219206>] sock_alloc_send_skb+0x2f/0x33
 [<c023ea78>] ip_append_data+0x7f7/0x8a7
 [<c021997f>] release_sock+0x1b/0x71
 [<c025c3d5>] udp_sendmsg+0x2eb/0x759
 [<c023e1c4>] ip_generic_getfrag+0x0/0xbd
 [<f8fea40e>] gm_linux_intr+0x9c/0xac [gm]
 [<c010899d>] handle_IRQ_event+0x31/0x65
 [<c0108d0b>] do_IRQ+0x9e/0x130
 [<c0106b6c>] common_interrupt+0x18/0x20
 [<c021007b>] input_devices_read+0x42c/0x593
 [<c0146c9d>] page_address+0x19/0xa5
 [<c0219458>] sock_no_sendpage+0x35/0x85
 [<c01579e7>] do_readv_writev+0x1ec/0x273
 [<c025c952>] udp_sendpage+0x10f/0x136
 [<c0118e76>] task_rq_lock+0x36/0x66
 [<c026426b>] inet_sendpage+0x9b/0xca
 [<fa0b25db>] svc_sendto+0xac/0x29e [sunrpc]
 [<fa1129e9>] nfsd_cache_update+0x8c/0x14c [nfsd]
 [<fa0b2cbb>] svc_udp_sendto+0x1e/0x3d [sunrpc]
 [<fa0b3f0c>] svc_send+0xb9/0xfc [sunrpc]
 [<fa0b1b58>] svc_process+0x2b4/0x788 [sunrpc]
 [<fa1094b2>] nfsd+0x1f3/0x39e [nfsd]
 [<fa1092bf>] nfsd+0x0/0x39e [nfsd]
 [<c010428d>] kernel_thread_helper+0x5/0xb

1 warning and 1 error issued.  Results may not be reliable.

-- 
Dale Harris   
ro...@ma...
/.-)

Re: [BProc] page allocation problems

[Erik H. <eah...@gm...>]

On 5/7/05, Dale Harris <ro...@ma...> wrote:
>=20
> Hey I'm seeing some page allocation errors.  Has anyone seen anything
> like this.  Of course it a 2.6.9 vanilla kernel patched for bproc, and
> myrinet GM driver is running.

I don't think this has anything to do with BProc per se.  I've seen
stuff like this before whenever I turn on jumbo frames on any machine
and start shoving a lot of data through.  I'm not 100% sure I'm right
about what's going on but here's my guess:

Once you start using jumbo frame sizes (~9k) it gets harder to
allocate skuffs in the kernel.  These are the buffers that hold
network packets.  Pages for this kind of stuff are allocated in powers
of two.  9k will require 4 pages.  Also, since it's kernel space stuff
that will be used for DMA buffers, it will want 4 contiguous pages.=20
That can be hard to find since memory gets fragmented.  Normally it
would be possible for the swapper to page some stuff out (e.g. disk
blocks) but kmalloc was called from an interrupt in this case which
makes that impossible.  The allocator has no options left so it gives
up and the allocation fails.

Allocation failures in cases like this shouldn't be treated as a major
problem.  Network drivers need to be able to deal with this sort of
thing - and they do.  I think the message below is supposed to be a
helpful debugging aid.  It's considered a warning.

I don't know if BProc is causing this to happen in some subtle way.=20
If anything, I would expect that BProc could cause bigger traffic
bursts than would normally be experienced by most servers.

- Erik

> swapper: page allocation failure. order:2, mode:0x20
>  [<c013e28e>] __alloc_pages+0x1b3/0x358
>  [<c013e458>] __get_free_pages+0x25/0x3f
>  [<c01416dc>] kmem_getpages+0x21/0xc9
>  [<c01423bb>] cache_grow+0xab/0x14d
>  [<c01425d1>] cache_alloc_refill+0x174/0x219
>  [<c0142a24>] __kmalloc+0x85/0x8c
>  [<c0219da1>] alloc_skb+0x47/0xe0
>  [<f8febfb8>] gmip_recv_interrupt+0x216/0x4c7 [gm]
>  [<c011a76a>] load_balance+0x15c/0x170
>  [<f8ff1691>] __gm_ethernet_wake_callback+0x6a/0x9c [gm]
>  [<f8fde584>] gm_handle_claimed_interrupt+0x580/0x62e [gm]
>  [<c025d0ae>] udp_queue_rcv_skb+0x174/0x2a4
>  [<c025d6c6>] udp_rcv+0x164/0x407
>  [<c023b6d8>] ip_defrag+0x112/0x1bf
>  [<c0239d4d>] ip_local_deliver+0xe8/0x279
>  [<c023a26d>] ip_rcv+0x38f/0x510
>  [<c021ffe3>] netif_receive_skb+0x1c7/0x2a1
>  [<c022013b>] process_backlog+0x7e/0x10b
>  [<c022023f>] net_rx_action+0x77/0xf6
>  [<f8fea40e>] gm_linux_intr+0x9c/0xac [gm]
>  [<c010899d>] handle_IRQ_event+0x31/0x65
>  [<c0108d0b>] do_IRQ+0x9e/0x130
>  [<c0106b6c>] common_interrupt+0x18/0x20
>  [<c010401e>] default_idle+0x0/0x2c
>  [<c0104047>] default_idle+0x29/0x2c
>  [<c01040bc>] cpu_idle+0x3f/0x58
>  [<c034a895>] start_kernel+0x197/0x1d5
>  [<c034a336>] unknown_bootoption+0x0/0x15c

[ snip ]

Re: [BProc] page allocation problems

[Dale H. <ro...@ma...>]

On Mon, May 09, 2005 at 10:54:16AM -0700, Erik Hendriks elucidated:
> On 5/7/05, Dale Harris <ro...@ma...> wrote:
> > 
> > Hey I'm seeing some page allocation errors.  Has anyone seen anything
> > like this.  Of course it a 2.6.9 vanilla kernel patched for bproc, and
> > myrinet GM driver is running.
> 
> I don't think this has anything to do with BProc per se.  I've seen
> stuff like this before whenever I turn on jumbo frames on any machine
> and start shoving a lot of data through.  I'm not 100% sure I'm right
> about what's going on but here's my guess:
> 
> Once you start using jumbo frame sizes (~9k) it gets harder to
> allocate skuffs in the kernel.  These are the buffers that hold
> network packets.  Pages for this kind of stuff are allocated in powers
> of two.  9k will require 4 pages.  Also, since it's kernel space stuff


Okay, so this does make sense.  Myrinet is using Jumbo frames and set
itself to a default of MTU of 9000.  Myrinet suggested dropping the
frame size down to 4000.  


With bproc is can I just put a:

plugin ifconfig myri0 mtu 4000


To set the MTU when the nodes boot?  Or does it support setting the MTU?


Dale

Re: [BProc] page allocation problems

[Sean <se...@la...>]

HI
    I wrote the attached patch to ifdup.c that lets you set the mtu in 
beoboot quite some time ago.  I guess I should check it in.

Sean



Dale Harris wrote:

>On Mon, May 09, 2005 at 10:54:16AM -0700, Erik Hendriks elucidated:
>  
>
>>On 5/7/05, Dale Harris <ro...@ma...> wrote:
>>    
>>
>>>Hey I'm seeing some page allocation errors.  Has anyone seen anything
>>>like this.  Of course it a 2.6.9 vanilla kernel patched for bproc, and
>>>myrinet GM driver is running.
>>>      
>>>
>>I don't think this has anything to do with BProc per se.  I've seen
>>stuff like this before whenever I turn on jumbo frames on any machine
>>and start shoving a lot of data through.  I'm not 100% sure I'm right
>>about what's going on but here's my guess:
>>
>>Once you start using jumbo frame sizes (~9k) it gets harder to
>>allocate skuffs in the kernel.  These are the buffers that hold
>>network packets.  Pages for this kind of stuff are allocated in powers
>>of two.  9k will require 4 pages.  Also, since it's kernel space stuff
>>    
>>
>
>
>Okay, so this does make sense.  Myrinet is using Jumbo frames and set
>itself to a default of MTU of 9000.  Myrinet suggested dropping the
>frame size down to 4000.  
>
>
>With bproc is can I just put a:
>
>plugin ifconfig myri0 mtu 4000
>
>
>To set the MTU when the nodes boot?  Or does it support setting the MTU?
>
>
>Dale
>
>
>
>-------------------------------------------------------
>This SF.Net email is sponsored by: NEC IT Guy Games.
>Get your fingers limbered up and give it your best shot. 4 great events, 4
>opportunities to win big! Highest score wins.NEC IT Guy Games. Play to
>win an NEC 61 plasma display. Visit http://www.necitguy.com/?r=20
>_______________________________________________
>BProc-users mailing list
>BPr...@li...
>https://lists.sourceforge.net/lists/listinfo/bproc-users
>
>  
>

Re: [BProc] page allocation problems

[Erik H. <eah...@gm...>]

Whups, looks like I forgot to reply to all.

---------- Forwarded message ----------
From: Erik Hendriks <eah...@gm...>
Date: May 12, 2005 8:59 AM
Subject: Re: [BProc] page allocation problems
To: Sean <se...@la...>


On 5/11/05, Sean <se...@la...> wrote:
> HI
>     I wrote the attached patch to ifdup.c that lets you set the mtu in
> beoboot quite some time ago.  I guess I should check it in.

I believe it made it in as part of ifdup, just not part of the
ifconfig plugin.  IIRC, ifdup doesn't handle the case where you're not
trying to copy another interface's configuration.

Maybe ifdup isn't an appropriate place for it at all.  Maybe the more
appriopriate thing would be to expect ifdup followed by ifconfig to
tweak parameters like that.  Ideally, ifconfig would be just like the
real ifconfig but that would take work.

<crazyidea>
Write a plugin that would let beoboot use stuff like system's ifconfig
program directly.  For example a line like this:

useutil ifconfig myri0 mtu 4000

The useutil plugin would do bproc_execdump ifconfig with those options
and the go undump it on the slave node.  It should work as long as
this plugin is called after vmadlib to setup the libraries.  It would
make the size of the setup process a bit bigger but I think it'll
likely be worth it for the flexibility it provides.
</crazyidea>

- Erik

Re: [BProc] page allocation problems

[Dale H. <ro...@ma...>]

On Thu, May 12, 2005 at 10:27:43AM -0700, Erik Hendriks elucidated:
> <crazyidea>
> Write a plugin that would let beoboot use stuff like system's ifconfig
> program directly.  For example a line like this:
> 
> useutil ifconfig myri0 mtu 4000
> 


It would be useful, I think.  

FWIW, dropping the MTU down to 4000 worked for me.  I'd probably apply
some of the patches if it wasn't that tomorrow (Friday) is my last day
at work, I'm being laid off.  Such is the way of academia when the
grants dry up. 


Dale

Re: [BProc] page allocation problems

[Erik H. <eah...@gm...>]

On 5/9/05, Dale Harris <ro...@ma...> wrote:
> On Mon, May 09, 2005 at 10:54:16AM -0700, Erik Hendriks elucidated:
> > On 5/7/05, Dale Harris <ro...@ma...> wrote:
> > >
> > > Hey I'm seeing some page allocation errors.  Has anyone seen anything
> > > like this.  Of course it a 2.6.9 vanilla kernel patched for bproc, an=
d
> > > myrinet GM driver is running.
> >
> > I don't think this has anything to do with BProc per se.  I've seen
> > stuff like this before whenever I turn on jumbo frames on any machine
> > and start shoving a lot of data through.  I'm not 100% sure I'm right
> > about what's going on but here's my guess:
> >
> > Once you start using jumbo frame sizes (~9k) it gets harder to
> > allocate skuffs in the kernel.  These are the buffers that hold
> > network packets.  Pages for this kind of stuff are allocated in powers
> > of two.  9k will require 4 pages.  Also, since it's kernel space stuff
>=20
> Okay, so this does make sense.  Myrinet is using Jumbo frames and set
> itself to a default of MTU of 9000.  Myrinet suggested dropping the
> frame size down to 4000.
>=20
> With bproc is can I just put a:
>=20
> plugin ifconfig myri0 mtu 4000
>=20
> To set the MTU when the nodes boot?  Or does it support setting the MTU?

It doesn't although it certainly should...  There's code in the ifdup
beoboot module to do it.  That could probably be copied to the
ifconfig module without too much trouble.

- Erik