From: Toralf F. <tor...@gm...> - 2012-10-13 15:01:22
|
I'm still trying to get a reliable scenario which either crashes UML or let simple 1 linux process at the host system runs at 100%. Till now I only know that it happens if a have apache2 running at the UML and fetch some http pages (either the apache manual itself or something from phpsysinfo via firefox (not via wget !). The hostfs feature seems to be involved too. Today the current git kernel (linux-v3.6-10970-g4d7127d ) crashes within an unstable Gentoo Linux UML with : ... * Starting local [ ok ] Kernel panic - not syncing: BUG! 083d7178: [<08322f18>] dump_stack+0x22/0x24 083d7190: [<08322fcc>] panic+0x7a/0x181 083d71b8: [<08238d21>] blk_flush_plug_list+0x201/0x220 083d71f8: [<08327202>] schedule+0x52/0x70 083d720c: [<08327ab5>] __down_read+0x75/0xa0 083d7238: [<08326a11>] down_read+0x11/0x20 083d7244: [<0806205f>] handle_page_fault+0x7f/0x270 083d7288: [<080623e1>] segv+0x161/0x2b0 083d7338: [<08062583>] segv_handler+0x53/0x90 083d7360: [<08073b77>] sig_handler_common+0x57/0xe0 083d75e4: [<08073d25>] sig_handler+0x45/0x60 083d75f4: [<0807383b>] hard_handler+0x6b/0xb0 083d761c: [<b77e240c>] 0xb77e240c 083d7758: [<08326a11>] down_read+0x11/0x20 083d7764: [<0806205f>] handle_page_fault+0x7f/0x270 083d77a8: [<080623e1>] segv+0x161/0x2b0 083d7858: [<08062583>] segv_handler+0x53/0x90 083d7880: [<08073b77>] sig_handler_common+0x57/0xe0 083d7b04: [<08073d25>] sig_handler+0x45/0x60 083d7b14: [<0807383b>] hard_handler+0x6b/0xb0 083d7b3c: [<b77e240c>] 0xb77e240c /home/tfoerste/workspace/bin/start_uml.sh: line 97: 22043 Segmentation fault (core dumped) $LINUX earlyprintk ubda=$ROOT_FS ubdb=$SWAP_FS $CD eth0=$NET mem=256M $TTY umid=uml "$@" The uml exe (linux-v3.6-10970-g4d7127d) can be delivered/examined if wanted. -- MfG/Sincerely Toralf Förster pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3 |
From: Toralf F. <tor...@gm...> - 2012-10-13 19:08:57
|
The following data I gathered from a UML process running at 100% COU of 1 core : Is something of those data are helpful ? 0x0805f2aa in to_irq_stack (mask_out=0x83d7b28) at arch/um/kernel/irq.c:439 439 task = cpu_tasks[ti->cpu].task; (gdb) cont Continuing. Program received signal SIGSEGV, Segmentation fault. 0x0805f2aa in to_irq_stack (mask_out=0x83d7b28) at arch/um/kernel/irq.c:439 439 task = cpu_tasks[ti->cpu].task; (gdb) bt #0 0x0805f2aa in to_irq_stack (mask_out=0x83d7b28) at arch/um/kernel/irq.c:439 #1 0x080737fb in hard_handler (sig=<optimized out>, si=0x83d7b4c, p=0x83d7bcc) at arch/um/os-Linux/signal.c:155 #2 <signal handler called> #3 0x08286867 in __kfree_skb (skb=0x37fa1240) at net/core/skbuff.c:614 #4 0x08286710 in kfree_skb (skb=0x37fa1240) at net/core/skbuff.c:633 #5 0x08068ce8 in uml_net_rx (dev=0x37cc2000) at arch/um/drivers/net_kern.c:104 #6 0x0806975f in uml_net_interrupt (irq=5, dev_id=0x37cc2000) at arch/um/drivers/net_kern.c:125 #7 0x080c9edd in handle_irq_event_percpu (desc=0x83e21f0, action=0x37fe5880) at kernel/irq/handle.c:142 #8 0x080ca017 in handle_irq_event (desc=0x83e21f0) at kernel/irq/handle.c:191 #9 0x080cc2ef in handle_edge_irq (irq=5, desc=0x83e21f0) at kernel/irq/chip.c:519 #10 0x080c9888 in generic_handle_irq_desc (desc=<optimized out>, irq=5) at include/linux/irqdesc.h:111 #11 generic_handle_irq (irq=5) at kernel/irq/irqdesc.c:314 #12 0x0805ef35 in do_IRQ (irq=5, regs=0x3716a0f8) at arch/um/kernel/irq.c:294 #13 0x0805efab in sigio_handler (sig=29, unused_si=0x0, regs=0x3716a0f8) at arch/um/kernel/irq.c:53 #14 0x08073b77 in sig_handler_common (sig=29, si=0x0, mc=0x0) at arch/um/os-Linux/signal.c:44 #15 0x08073b00 in unblock_signals () at arch/um/os-Linux/signal.c:282 #16 0x08073c90 in set_signals (enable=1) at arch/um/os-Linux/signal.c:302 #17 0x0807ffe5 in arch_local_irq_restore (flags=1) at /home/tfoerste/devel/linux/arch/um/include/asm/irqflags.h:16 #18 vprintk_emit (facility=0, level=<optimized out>, dict=0x0, dictlen=0, fmt=0x838ba23 "\001\064WARNING: at %s:%d %pS()\n", args=0x3716a3e0 "*\273\070\b\240") at kernel/printk.c:1619 #19 0x0832310b in printk (fmt=0x838ba23 "\001\064WARNING: at %s:%d %pS()\n") at kernel/printk.c:1681 #20 0x0807e470 in warn_slowpath_common (file=0x838bb2a "kernel/softirq.c", line=160, caller=<optimized out>, taint=9, args=0x0) at kernel/panic.c:413 #21 0x0807e563 in warn_slowpath_null (file=0x838bb2a "kernel/softirq.c", line=160) at kernel/panic.c:454 #22 0x08085b9f in _local_bh_enable_ip (ip=<optimized out>) at kernel/softirq.c:160 #23 local_bh_enable () at kernel/softirq.c:187 #24 0x08291998 in rcu_read_unlock_bh () at include/linux/rcupdate.h:802 #25 dev_queue_xmit (skb=0x37fa13c0) at net/core/dev.c:2637 #26 0x0829d5f2 in neigh_resolve_output (neigh=0x37e97800, skb=0x37fa13c0) at net/core/neighbour.c:1320 #27 0x082b444d in dst_neigh_output (skb=0x37fa13c0, n=0x37e97800, dst=<optimized out>) at include/net/dst.h:414 #28 ip_finish_output2 (skb=0x37fa13c0) at net/ipv4/ip_output.c:201 #29 ip_finish_output (skb=<optimized out>) at net/ipv4/ip_output.c:234 #30 0x082b4613 in ip_output (skb=0x37fa13c0) at net/ipv4/ip_output.c:307 #31 0x082b34e3 in dst_output (skb=0x37fa13c0) at include/net/dst.h:449 #32 ip_local_out (skb=0x37fa13c0) at net/ipv4/ip_output.c:110 #33 0x082b3624 in ip_build_and_send_pkt (skb=0x37fa13c0, sk=0x3711cf00, saddr=838969536, daddr=4261456064, opt=0x0) at net/ipv4/ip_output.c:162 #34 0x082d1060 in tcp_v4_conn_request (sk=0x3711cf00, skb=0x37fa1300) at net/ipv4/tcp_ipv4.c:1652 #35 0x082c7f3c in tcp_rcv_state_process (sk=0x3711cf00, skb=0x37fa1300, th=0x3724c662, len=40) at net/ipv4/tcp_input.c:5927 #36 0x082ced1d in tcp_v4_do_rcv (sk=0x3711cf00, skb=0x37fa1300) at net/ipv4/tcp_ipv4.c:1896 #37 0x082d1dd6 in tcp_v4_rcv (skb=0x37fa1300) at net/ipv4/tcp_ipv4.c:2037 #38 0x082af936 in ip_local_deliver_finish (skb=0x37fa1300) at net/ipv4/ip_input.c:225 ---Type <return> to continue, or q <return> to quit--- #39 ip_local_deliver (skb=0x37fa1300) at net/ipv4/ip_input.c:263 #40 0x082afd38 in dst_input (skb=0x37fa1300) at include/net/dst.h:455 #41 ip_rcv_finish (skb=0x37fa1300) at net/ipv4/ip_input.c:373 #42 ip_rcv (skb=<optimized out>, dev=0x37cc2000, pt=0x83ef740, orig_dev=0x37cc2000) at net/ipv4/ip_input.c:451 #43 0x0828ecd4 in __netif_receive_skb (skb=0x37fa1300) at net/core/dev.c:3356 #44 0x0828f039 in process_backlog (napi=0x83f2040, quota=1) at net/core/dev.c:3826 #45 0x08292b2c in net_rx_action (h=0x84a314c) at net/core/dev.c:3984 #46 0x080859a5 in __do_softirq () at kernel/softirq.c:247 #47 0x08085b05 in do_softirq () at kernel/softirq.c:295 #48 do_softirq () at kernel/softirq.c:282 #49 0x08085d55 in invoke_softirq () at kernel/softirq.c:329 #50 irq_exit () at kernel/softirq.c:348 #51 0x0805ef3a in do_IRQ (irq=5, regs=0x3716a8a0) at arch/um/kernel/irq.c:295 #52 0x0805efab in sigio_handler (sig=29, unused_si=0x0, regs=0x3716a8a0) at arch/um/kernel/irq.c:53 #53 0x08073b77 in sig_handler_common (sig=29, si=0x0, mc=0x0) at arch/um/os-Linux/signal.c:44 #54 0x08073b00 in unblock_signals () at arch/um/os-Linux/signal.c:282 #55 0x08238f75 in arch_local_irq_enable () at /home/tfoerste/devel/linux/arch/um/include/asm/irqflags.h:21 #56 spin_unlock_irq (lock=<optimized out>) at include/linux/spinlock.h:335 #57 blk_queue_bio (q=0x37ea0000, bio=0x37136e00) at block/blk-core.c:1529 #58 0x08237876 in generic_make_request (bio=0x37136e00) at block/blk-core.c:1792 #59 0x08237959 in submit_bio (rw=96, bio=0x37136e00) at block/blk-core.c:1844 #60 0x0812a225 in submit_bh (rw=96, bh=0x379d42a0) at fs/buffer.c:2904 #61 0x081601ca in __ext3_get_inode_loc (iloc=0x3716ac60, in_mem=0, inode=<optimized out>) at fs/ext3/inode.c:2824 #62 0x08160764 in ext3_iget (sb=0x37e9a000, ino=59551) at fs/ext3/inode.c:2906 #63 0x0816716c in ext3_lookup (dentry=0x3797de80, dir=0x379e6f6c, flags=<optimized out>) at fs/ext3/namei.c:1033 #64 ext3_lookup (dir=0x379e6f6c, dentry=0x3797de80, flags=1) at fs/ext3/namei.c:1014 #65 0x08109f2f in lookup_real (dir=<optimized out>, dentry=0x3797de80, flags=<optimized out>) at fs/namei.c:1321 #66 0x0810a041 in __lookup_hash (name=<optimized out>, base=<optimized out>, flags=1) at fs/namei.c:1339 #67 0x0810b41b in lookup_slow (nd=0x3716ad8c, name=<optimized out>, path=0x3716ad24) at fs/namei.c:1458 #68 0x0810c74b in walk_component (follow=1, name=0x3716ad94, type=<optimized out>, path=0x3716ad24, nd=0x3716ad8c) at fs/namei.c:1547 #69 lookup_last (path=0x3716ad24, nd=0x3716ad8c) at fs/namei.c:1949 #70 path_lookupat (dfd=<optimized out>, name=<optimized out>, flags=65, nd=0x3716ad8c) at fs/namei.c:1984 #71 0x0810c7d6 in filename_lookup (dfd=-100, name=0x3723e000, flags=1, nd=0x3716ad8c) at fs/namei.c:2023 #72 0x0810cd4f in user_path_at_empty (dfd=-100, name=0x4f407858 <Address 0x4f407858 out of bounds>, flags=1, path=0x3716ae24, empty=0x0) at fs/namei.c:2166 #73 0x0810cdde in user_path_at (dfd=-100, name=0x4f407858 <Address 0x4f407858 out of bounds>, flags=1, path=0x3716ae24) at fs/namei.c:2177 #74 0x08105a87 in vfs_fstatat (dfd=-100, filename=0x4f407858 <Address 0x4f407858 out of bounds>, stat=0x3716ae58, flag=<optimized out>) at fs/stat.c:88 #75 0x08105ae8 in vfs_stat (name=0x4f407858 <Address 0x4f407858 out of bounds>, stat=0x3716ae58) at fs/stat.c:101 #76 0x08105d39 in sys_stat64 (filename=0x4f407858 <Address 0x4f407858 out of bounds>, statbuf=0x42afffb0) at fs/stat.c:376 #77 0x08062ec6 in handle_syscall (r=0x37145e90) at arch/um/kernel/skas/syscall.c:35 #78 0x0807710f in handle_trap (local_using_sysemu=<optimized out>, regs=0x37145e90, pid=2053) at arch/um/os-Linux/skas/process.c:193 ---Type <return> to continue, or q <return> to quit--- #79 userspace (regs=0x37145e90) at arch/um/os-Linux/skas/process.c:426 #80 0x0805f8ec in fork_handler () at arch/um/kernel/process.c:160 #81 0x00000000 in ?? () (gdb) quit A debugging session is active. Inferior 1 [process 997] will be detached. Quit anyway? (y or n) y Detaching from program: /usr/local/bin/linux-v3.6-10970-g4d7127d, process 997 -- MfG/Sincerely Toralf Förster pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3 |
From: richard -r. w. <ric...@gm...> - 2012-10-13 19:42:51
|
On Sat, Oct 13, 2012 at 9:08 PM, Toralf Förster <tor...@gm...> wrote: > The following data I gathered from a UML process running at 100% COU of 1 core : > Is something of those data are helpful ? If UML is looping, no. Sorry. -- Thanks, //richard |
From: Toralf F. <tor...@gm...> - 2012-10-14 08:21:48
|
On 10/13/2012 09:42 PM, richard -rw- weinberger wrote: > On Sat, Oct 13, 2012 at 9:08 PM, Toralf Förster <tor...@gm...> wrote: >> The following data I gathered from a UML process running at 100% COU of 1 core : >> Is something of those data are helpful ? > > If UML is looping, no. > Sorry. > Understood, BTW this crash happens after fetching few hundreds HTTP docs from the UML apache and shutdown the UML afterwards: ... [ ok ] * Unmounting /usr/portage ... Kernel panic - not syncing: Kernel mode fault at addr 0x0, ip 0x0 083d3358: [<0831e238>] dump_stack+0x22/0x24 083d3370: [<0831e2ec>] panic+0x7a/0x181 083d3398: [<0806143d>] segv+0x1bd/0x2b0 083d3448: [<08061583>] segv_handler+0x53/0x90 083d3470: [<08072b77>] sig_handler_common+0x57/0xe0 083d36f4: [<08072d25>] sig_handler+0x45/0x60 083d3704: [<0807283b>] hard_handler+0x6b/0xb0 083d372c: [<b773f40c>] 0xb773f40c 083d3850: [<080c8888>] generic_handle_irq+0x28/0x30 083d3860: [<0805df35>] do_IRQ+0x25/0x40 083d3870: [<08061799>] winch+0x19/0x20 083d3880: [<08072b77>] sig_handler_common+0x57/0xe0 083d3b04: [<08072d25>] sig_handler+0x45/0x60 083d3b14: [<0807283b>] hard_handler+0x6b/0xb0 083d3b3c: [<b773f40c>] 0xb773f40c /home/tfoerste/workspace/bin/start_uml.sh: line 97: 14610 Segmentation fault (core dumped) $LINUX earlyprintk ubda=$ROOT_FS ubdb=$SWAP_FS $CD eth0=$NET mem=768M $TTY umid=uml "$@" Just for completeness: $ addr2line -e /usr/local/bin/linux-v3.6-11019-g3d6ee36 0831e238 /home/tfoerste/devel/linux/arch/um/kernel/sysrq.c:47 ... -- MfG/Sincerely Toralf Förster pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3 |
From: Toralf F. <tor...@gm...> - 2012-10-14 16:25:56
|
On 10/13/2012 09:42 PM, richard -rw- weinberger wrote: > If UML is looping, no. > Sorry. > Now I got a crash. Running this command at the host: $ for i in $(seq 1 1000); do echo -en "$i\r"; for p in manual phpsysinfo server-info server-status; do wget --quiet --no-check-certificate -O /dev/null https://n22_uml/$p; done; done crashes the UML after 337 HTTP requests : ... * Starting vixie-cron ... [ ok ] * Starting local [ ok ] Kernel panic - not syncing: Kernel mode fault at addr 0xd4, ip 0x8084c68 083d3768: [<0831e238>] dump_stack+0x22/0x24 083d3780: [<0831e2ec>] panic+0x7a/0x181 083d37a8: [<0806143d>] segv+0x1bd/0x2b0 083d3858: [<08061583>] segv_handler+0x53/0x90 083d3880: [<08072b77>] sig_handler_common+0x57/0xe0 083d3b04: [<08072d25>] sig_handler+0x45/0x60 083d3b14: [<0807283b>] hard_handler+0x6b/0xb0 083d3b3c: [<b777740c>] 0xb777740c EIP: 0073:[<400011d2>] CPU: 0 Not tainted ESP: 007b:bfede77c EFLAGS: 00000246 Not tainted EAX: ffffffda EBX: 00000005 ECX: 00000000 EDX: bfede870 ESI: 00000000 EDI: 00000000 EBP: bfede9f8 DS: 007b ES: 007b 083d371c: [<080793cd>] show_regs+0xed/0x120 083d3738: [<0806192c>] panic_exit+0x2c/0x50 083d3748: [<0809ec71>] notifier_call_chain+0x41/0x60 083d3770: [<0809edb3>] atomic_notifier_call_chain+0x23/0x30 083d3780: [<0831e314>] panic+0xa2/0x181 083d37a8: [<0806143d>] segv+0x1bd/0x2b0 083d3858: [<08061583>] segv_handler+0x53/0x90 083d3880: [<08072b77>] sig_handler_common+0x57/0xe0 083d3b04: [<08072d25>] sig_handler+0x45/0x60 083d3b14: [<0807283b>] hard_handler+0x6b/0xb0 083d3b3c: [<b777740c>] 0xb777740c Terminated $ addr2line -e /usr/local/bin/linux-v3.6-11019-g3d6ee36 0x8084c68 /home/tfoerste/devel/linux/kernel/softirq.c:310 -- MfG/Sincerely Toralf Förster pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3 |
From: richard -r. w. <ric...@gm...> - 2012-10-14 22:18:40
|
On Sun, Oct 14, 2012 at 6:25 PM, Toralf Förster <tor...@gm...> wrote: > On 10/13/2012 09:42 PM, richard -rw- weinberger wrote: >> If UML is looping, no. >> Sorry. >> > Now I got a crash. Running this command at the host: > > $ for i in $(seq 1 1000); do echo -en "$i\r"; for p in manual phpsysinfo server-info server-status; do wget --quiet --no-check-certificate -O /dev/null https://n22_uml/$p; done; done > > crashes the UML after 337 HTTP requests : Is UML server or client? > ... > * Starting vixie-cron ... > [ ok ] > * Starting local > [ ok ] > Kernel panic - not syncing: Kernel mode fault at addr 0xd4, ip 0x8084c68 > 083d3768: [<0831e238>] dump_stack+0x22/0x24 > 083d3780: [<0831e2ec>] panic+0x7a/0x181 > 083d37a8: [<0806143d>] segv+0x1bd/0x2b0 > 083d3858: [<08061583>] segv_handler+0x53/0x90 > 083d3880: [<08072b77>] sig_handler_common+0x57/0xe0 > 083d3b04: [<08072d25>] sig_handler+0x45/0x60 > 083d3b14: [<0807283b>] hard_handler+0x6b/0xb0 > 083d3b3c: [<b777740c>] 0xb777740c > > > EIP: 0073:[<400011d2>] CPU: 0 Not tainted ESP: 007b:bfede77c EFLAGS: 00000246 > Not tainted > EAX: ffffffda EBX: 00000005 ECX: 00000000 EDX: bfede870 > ESI: 00000000 EDI: 00000000 EBP: bfede9f8 DS: 007b ES: 007b > 083d371c: [<080793cd>] show_regs+0xed/0x120 > 083d3738: [<0806192c>] panic_exit+0x2c/0x50 > 083d3748: [<0809ec71>] notifier_call_chain+0x41/0x60 > 083d3770: [<0809edb3>] atomic_notifier_call_chain+0x23/0x30 > 083d3780: [<0831e314>] panic+0xa2/0x181 > 083d37a8: [<0806143d>] segv+0x1bd/0x2b0 > 083d3858: [<08061583>] segv_handler+0x53/0x90 > 083d3880: [<08072b77>] sig_handler_common+0x57/0xe0 > 083d3b04: [<08072d25>] sig_handler+0x45/0x60 > 083d3b14: [<0807283b>] hard_handler+0x6b/0xb0 > 083d3b3c: [<b777740c>] 0xb777740c > > Terminated > > > $ addr2line -e /usr/local/bin/linux-v3.6-11019-g3d6ee36 0x8084c68 > /home/tfoerste/devel/linux/kernel/softirq.c:310 I'm sure next time its a completely different source position. What consoles are you using? xterm? -- Thanks, //richard |