Re: [tipc-discussion] Page allocation failures

[Ying X. <yin...@wi...>]

Andy,

By your listed result of "tipc-config -ls", it seems your system state of TIPC cluster is very normal because the rates of packet duplication and packet out-of-order are also within acceptable range. 
So no any exceptional things are found. Maybe you can consider whether other applications causes this issue.

Ying



Andy Stec wrote:
> Hi Ying,
>
> Thanks for your response. Below is the output from tipc-config -ls command.  I included 3 snapshots of TIPC stats, they were collected at one second intervals
> during the time when page allocation failure happened.
>
>
> We're using TIPC 1.7.7 that runs on Redhat 6.0 in virtual environment (vmware).   At high CPU loads we see periodic page allocation failures and kernel panics.  
> >From the stack trace (below) it appears that TIPC is failing to allocate memory.   There is over 5GB of cached memory when the page allocation failures happen.   
> We tried periodically clearing cached memory and that seems to fix the problem.  Setting vm.max_free_kbytes to a high value (1 GB) also alleviates this problem.  
> Has anybody encountered this problem and been able to get to the root cause?
>
>
>
>
> Mon Aug 22 20:19:59 UTC 2011
>
> Link statistics:
> Link <broadcast-link>
>   Window:50 packets
>   RX packets:169 fragments:0/0 bundles:17/328
>   TX packets:1180 fragments:0/0 bundles:0/0
>   RX naks:0 defs:0 dups:0
>   TX naks:0 acks:11 dups:0
>   Congestion bearer:0 link:0  Send queue max:18 avg:4
>
> Link <1.1.2:eth0-1.1.1:eth0>
>   ACTIVE  MTU:1500  Priority:10  Tolerance:1500 ms  Window:50 packets
>   RX packets:143587 fragments:113560/12610 bundles:0/0
>   TX packets:151466 fragments:122319/13591 bundles:4/5
>   TX profile sample:3065 packets  average:2690 octets
>   0-64:0% -256:14% -1024:54% -4096:16% -16354:16% -32768:0% -66000:0%
>   RX states:4606 probes:237 naks:23 defs:0 dups:0
>   TX states:4061 probes:280 naks:0 acks:3544 dups:214
>   Congestion bearer:0 link:27  Send queue max:56 avg:2
>
> Link <1.1.2:eth0-1.1.9:eth0>
>   ACTIVE  MTU:1500  Priority:10  Tolerance:1500 ms  Window:50 packets
>   RX packets:1 fragments:0/0 bundles:0/0
>   TX packets:1 fragments:0/0 bundles:0/0
>   TX profile sample:0 packets  average:0 octets
>   0-64:0% -256:0% -1024:0% -4096:0% -16354:0% -32768:0% -66000:0%
>   RX states:780777 probes:390297 naks:0 defs:0 dups:0
>   TX states:781229 probes:390560 naks:0 acks:0 dups:0
>   Congestion bearer:0 link:0  Send queue max:0 avg:0
>
> Link <1.1.2:eth0-1.1.10:eth0>
>   ACTIVE  MTU:1500  Priority:10  Tolerance:1500 ms  Window:50 packets
>   RX packets:5196993 fragments:0/0 bundles:0/0
>   TX packets:5197040 fragments:0/0 bundles:0/0
>   TX profile sample:14897 packets  average:179 octets
>   0-64:0% -256:100% -1024:0% -4096:0% -16354:0% -32768:0% -66000:0%
>   RX states:35969 probes:18659 naks:0 defs:1 dups:0
>   TX states:36118 probes:17236 naks:1 acks:0 dups:0
>   Congestion bearer:0 link:0  Send queue max:4 avg:0
>
>
> Mon Aug 22 20:20:00 UTC 2011
>
> Link statistics:
> Link <broadcast-link>
>   Window:50 packets
>   RX packets:169 fragments:0/0 bundles:17/328
>   TX packets:1180 fragments:0/0 bundles:0/0
>   RX naks:0 defs:0 dups:0
>   TX naks:0 acks:11 dups:0
>   Congestion bearer:0 link:0  Send queue max:18 avg:4
>
> Link <1.1.2:eth0-1.1.1:eth0>
>   ACTIVE  MTU:1500  Priority:10  Tolerance:1500 ms  Window:50 packets
>   RX packets:143639 fragments:113596/12614 bundles:0/0
>   TX packets:151533 fragments:122373/13597 bundles:4/5
>   TX profile sample:3066 packets  average:2690 octets
>   0-64:0% -256:14% -1024:54% -4096:16% -16354:16% -32768:0% -66000:0%
>   RX states:4608 probes:237 naks:23 defs:0 dups:0
>   TX states:4062 probes:280 naks:0 acks:3545 dups:214
>   Congestion bearer:0 link:27  Send queue max:56 avg:2
>
> Link <1.1.2:eth0-1.1.9:eth0>
>   ACTIVE  MTU:1500  Priority:10  Tolerance:1500 ms  Window:50 packets
>   RX packets:1 fragments:0/0 bundles:0/0
>   TX packets:1 fragments:0/0 bundles:0/0
>   TX profile sample:0 packets  average:0 octets
>   0-64:0% -256:0% -1024:0% -4096:0% -16354:0% -32768:0% -66000:0%
>   RX states:780778 probes:390298 naks:0 defs:0 dups:0
>   TX states:781230 probes:390560 naks:0 acks:0 dups:0
>   Congestion bearer:0 link:0  Send queue max:0 avg:0
>
> Link <1.1.2:eth0-1.1.10:eth0>
>   ACTIVE  MTU:1500  Priority:10  Tolerance:1500 ms  Window:50 packets
>   RX packets:5196996 fragments:0/0 bundles:0/0
>   TX packets:5197043 fragments:0/0 bundles:0/0
>   TX profile sample:14897 packets  average:179 octets
>   0-64:0% -256:100% -1024:0% -4096:0% -16354:0% -32768:0% -66000:0%
>   RX states:35969 probes:18659 naks:0 defs:1 dups:0
>   TX states:36118 probes:17236 naks:1 acks:0 dups:0
>   Congestion bearer:0 link:0  Send queue max:4 avg:0
>
>
> Mon Aug 22 20:20:01 UTC 2011
>
>
> Link statistics:
> Link <broadcast-link>
>   Window:50 packets
>   RX packets:169 fragments:0/0 bundles:17/328
>   TX packets:1180 fragments:0/0 bundles:0/0
>   RX naks:0 defs:0 dups:0
>   TX naks:0 acks:11 dups:0
>   Congestion bearer:0 link:0  Send queue max:18 avg:4
>
> Link <1.1.2:eth0-1.1.1:eth0>
>   ACTIVE  MTU:1500  Priority:10  Tolerance:1500 ms  Window:50 packets
>   RX packets:143712 fragments:113659/12621 bundles:0/0
>   TX packets:151570 fragments:122400/13600 bundles:4/5
>   TX profile sample:3068 packets  average:2688 octets
>   0-64:0% -256:14% -1024:54% -4096:16% -16354:16% -32768:0% -66000:0%
>   RX states:4609 probes:237 naks:23 defs:11 dups:46
>   TX states:4069 probes:280 naks:3 acks:3551 dups:214
>   Congestion bearer:0 link:27  Send queue max:56 avg:2
>
> Link <1.1.2:eth0-1.1.9:eth0>
>   ACTIVE  MTU:1500  Priority:10  Tolerance:1500 ms  Window:50 packets
>   RX packets:1 fragments:0/0 bundles:0/0
>   TX packets:1 fragments:0/0 bundles:0/0
>   TX profile sample:0 packets  average:0 octets
>   0-64:0% -256:0% -1024:0% -4096:0% -16354:0% -32768:0% -66000:0%
>   RX states:780780 probes:390299 naks:0 defs:0 dups:0
>   TX states:781232 probes:390561 naks:0 acks:0 dups:0
>   Congestion bearer:0 link:0  Send queue max:0 avg:0
>
> Link <1.1.2:eth0-1.1.10:eth0>
>   ACTIVE  MTU:1500  Priority:10  Tolerance:1500 ms  Window:50 packets
>   RX packets:5197001 fragments:0/0 bundles:0/0
>   TX packets:5197048 fragments:0/0 bundles:0/0
>   TX profile sample:14897 packets  average:179 octets
>   0-64:0% -256:100% -1024:0% -4096:0% -16354:0% -32768:0% -66000:0%
>   RX states:35970 probes:18660 naks:0 defs:1 dups:0
>   TX states:36119 probes:17236 naks:1 acks:0 dups:0
>   Congestion bearer:0 link:0  Send queue max:4 avg:0
>
>
>
> -----Original Message-----
> From: Ying Xue [mailto:yin...@wi...] 
> Sent: Monday, August 22, 2011 12:42 AM
> To: Andy Stec
> Cc: tip...@li...
> Subject: Re: [tipc-discussion] Page allocation failures
>
> Hi Andy,
>
> When the issue happens, please post the result of executing "tipc-config 
> -ls" command. Maybe we can obtain some valuable information from it.
>
> Best Regards,
> Ying
>
>
> Andy Stec wrote:
>   
>> Hello,
>>
>> We're using TIPC 1.7.7 that runs on Redhat 6.0 in virtual environment (vmware).   At high CPU loads we see periodic page allocation failures and kernel panics.  From the stack trace (below) it appears that TIPC is failing to allocate memory.   There is over 5GB of cached memory when the page allocation failures happen.   We tried periodically clearing cached memory and that seems to fix the problem.  Setting vm.max_free_kbytes to a high value (1 GB) also alleviates this problem.  Has anybody encountered this problem and been able to get to the root cause?
>>
>> Aug 18 03:50:42 pl0_2 kernel: Call Trace:
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff8111ea66>] __alloc_pages_nodemask+0x706/0x850
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff811561a2>] kmem_getpages+0x62/0x170
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff81156dba>] fallback_alloc+0x1ba/0x270
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff8115680f>] ? cache_grow+0x2cf/0x320
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff81156b39>] ____cache_alloc_node+0x99/0x160
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff814070ca>] ? __alloc_skb+0x7a/0x180
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff811579cf>] kmem_cache_alloc_node_notrace+0x6f/0x130
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff81157c0b>] __kmalloc_node+0x7b/0x100
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff814070ca>] __alloc_skb+0x7a/0x180
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffffa0149d52>] buf_acquire+0x32/0x70 [tipc]
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffffa0150081>] tipc_msg_build+0x91/0x1c0 [tipc]
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffffa0157fe7>] tipc_port_recv_sections+0x37/0x50 [tipc]
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffffa0158146>] tipc_forward2port+0x146/0x150 [tipc]
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffffa0158178>] tipc_send2port+0x28/0x30 [tipc]
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffffa015b2f8>] send_msg+0x208/0x420 [tipc]
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff814cbacb>] ? _spin_unlock_bh+0x1b/0x20
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff81401d57>] ? release_sock+0xb7/0xd0
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff8140028e>] sock_sendmsg+0x11e/0x150
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff81091e50>] ? autoremove_wake_function+0x0/0x40
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff81427ffd>] ? verify_compat_iovec+0x7d/0x120
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff81400553>] sys_sendmsg+0x233/0x3a0
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff8116d12a>] ? do_sync_read+0xfa/0x140
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff81183e28>] ? d_free+0x58/0x60
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff8109bb59>] ? ktime_get_ts+0xa9/0xe0
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff8101a4f9>] ? read_tsc+0x9/0x20
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff8109b7e0>] ? getnstimeofday+0x60/0xf0
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff81426ad4>] compat_sys_sendmsg+0x14/0x20
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff81427858>] compat_sys_socketcall+0xa8/0x200
>> Aug 18 03:50:42 pl0_2 kernel: [<ffffffff8104cb10>] sysenter_dispatch+0x7/0x2e
>>
>>
>> Andy Stec
>> Infinite Convergence Solutions
>>
>>
>>
>>
>>
>> ______________________________________________________________________________________________________________________________
>>
>> This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify so to the sender by e-mail and delete the original message. In such cases, please notify us immediately at san...@in... . Further, you are not to copy, disclose, or distribute this e-mail or its contents to any unauthorized person(s) .Any such actions are considered unlawful. This e-mail may contain viruses. Infinite has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachments. Infinite reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infinite e-mail system. 
>>
>> ***INFINITE******** End of Disclaimer********INFINITE******** 
>> ------------------------------------------------------------------------------
>> Get a FREE DOWNLOAD! and learn more about uberSVN rich system, 
>> user administration capabilities and model configuration. Take 
>> the hassle out of deploying and managing Subversion and the 
>> tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
>> _______________________________________________
>> tipc-discussion mailing list
>> tip...@li...
>> https://lists.sourceforge.net/lists/listinfo/tipc-discussion
>>
>>   
>>     
>
>
> ______________________________________________________________________________________________________________________________
>
> This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify so to the sender by e-mail and delete the original message. In such cases, please notify us immediately at san...@in... . Further, you are not to copy, disclose, or distribute this e-mail or its contents to any unauthorized person(s) .Any such actions are considered unlawful. This e-mail may contain viruses. Infinite has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachments. Infinite reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infinite e-mail system. 
>
> ***INFINITE******** End of Disclaimer********INFINITE******** 
>
>
>