Hi,
attached below is the output of ksymoops regarding a
recent oops.
The machine is currently running Kernel 2.4.4 /
atm-0.78.
In the past, I had had several crashes on the same box
running
different kernel/atm-versions,. The machine is a HP
Netserver
with a ForeRunner LE ATM-NIC (hardware is checked and
o.k.). The
crashes seem to occur always at night without any
significant
load on the machine or the network, probably while
answering a
DHCP request (there is a DHCP server running on it).
All the
crashes occured in ATM/LANE code, unfortunately on
different
places (often arp-related).
Maybe somebody with more insight into LANE has any
idea, what is
going wrong ...
Regards,
Peter Daum
- - - - - 8< - - - - - 8< - - - - - 8< - - - - - 8< -
- - - - 8< - - - - -
ksymoops 2.3.7 on i686 2.4.4. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.4/ (default)
-m /boot/System.map-2.4.4 (specified)
No modules in ksyms, skipping objects
Warning (read_lsmod): no symbols in lsmod, is
/proc/modules a valid lsmod file?
Warning (compare_maps): ksyms_base symbol
__VERSIONED_SYMBOL(shmem_file_setup)
not found in System.map. Ignoring ksyms_base entry
Oops: 0000
CPU: 0
EIP: 0010:[<c0212dd1>]
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010086
eax: 00000000 ebx: c186cea0 ecx: c62a4800 edx:
00074b21
esi: 00000006 edi: c12e1560 ebp: c186cea0 esp:
c4179dd8
ds: 0018 es: 0018 ss: 0018
Process zeppelin (pid: 27287, stackpage=c4179000)
Stack: c12c0d40 00000006 c12e1560 c186cea0 00000000
c0199a81 c62a4800 c186cea0
c12a0000 c186cea0 c12a0000 0f0c0020 c12e1560
c4179eb4 c4997720 4000d000
00000001 c19c7034 c7ee35a0 4000d000 00000006
c62a4800 c12c0d40 c7f5f580
Call Trace: [<c0199a81>] [<c01992e7>] [<c019825d>]
[<c0107cff>] [<c0107e5e>]
[<c0106be8>] [<c0212f94>]
[<c0211560>] [<c0210825>] [<c011cbfe>]
[<c011cf92>] [<c01d2005>]
[<c0136e87>] [<c0106b3f>]
Code: 8b 68 60 85 db 75 18 8b 54 24 18 52 55 e8 cd 17
00 00 83 c4
>>EIP; c0212dd1 <lec_push+19/140> <=====
Trace; c0199a81 <dequeue_rx+765/dbc>
Trace; c01992e7 <process_rsq+1b/50>
Trace; c019825d <ns_irq_handler+119/33c>
Trace; c0107cff <handle_IRQ_event+2f/58>
Trace; c0107e5e <do_IRQ+6e/b0>
Trace; c0106be8 <ret_from_intr+0/20>
Trace; c0212f94 <lec_vcc_attach+9c/c0>
Trace; c0211560 <atm_push_raw+0/44>
Trace; c0210825 <atm_ioctl+429/9a8>
Trace; c011cbfe <unmap_fixup+62/12c>
Trace; c011cf92 <do_munmap+246/254>
Trace; c01d2005 <sock_ioctl+21/28>
Trace; c0136e87 <sys_ioctl+16b/184>
Trace; c0106b3f <system_call+33/38>
Code; c0212dd1 <lec_push+19/140>
00000000 <_EIP>:
Code; c0212dd1 <lec_push+19/140> <=====
0: 8b 68 60 mov
0x60(%eax),%ebp <=====
Code; c0212dd4 <lec_push+1c/140>
3: 85 db test %ebx,%ebx
Code; c0212dd6 <lec_push+1e/140>
5: 75 18 jne 1f <_EIP+0x1f>
c0212df0
<lec_push+38/140>
Code; c0212dd8 <lec_push+20/140>
7: 8b 54 24 18 mov
0x18(%esp,1),%edx
Code; c0212ddc <lec_push+24/140>
b: 52 push %edx
Code; c0212ddd <lec_push+25/140>
c: 55 push %ebp
Code; c0212dde <lec_push+26/140>
d: e8 cd 17 00 00 call 17df
<_EIP+0x17df> c02145b0
<lec_vcc_close+0/308>
Code; c0212de3 <lec_push+2b/140>
12: 83 c4 00 add $0x0,%esp
Kernel panic: Aiee, killing interrupt handler!
Receiver lock-up bug exists -- enabling work-around.
2 warnings issued. Results may not be reliable.
Logged In: YES
user_id=43292
From duplicate #454141 (more ATM (LANE) - related
Kernel-Crashes):
Hi,
... it happened again -
attached below is the output
of ksymoops regarding a
recent oops.
The machine is currently
running Kernel 2.4.7 /
atm-0.78.
In the past, I had had
several crashes running
different
kernel/atm-versions,. The
machines are HP Netserver
with a ForeRunner
LE ATM-NIC (hardware is
checked and o.k.). The other
network equipment
(ATM-, Ether-switches .. is
also mostly by Fore
Systems. The crashes
seem to occur always at night
without any significant
load on the
machine or the network. All
the crashes occured in
ATM/LANE code,
unfortunately on different
places. Besides that, I
personally can't
identify any particular
pattern.
Maybe somebody with more
insight into LANE has any
idea, what is
going wrong ...
Is anybody else experiencing
similar problems? The
ATM-system is
labelled "experimental", is
it generally that unstable?
As far as I
can see, the MTBF seems to be
about 2 weeks ...
Regards,
Peter Daum
<gator@cs.tu-berlin.de>
- - - - - 8< - - - - - 8< - -
- - - 8< - - - - - 8< -
- - - - 8< - - - - -
ksymoops 2.4.0 on i686 2.4.7.
Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.7/
(default)
-m /boot/System.map-2.4.7
(default)
Warning: You did not tell me
where to find symbol
information. I will
assume that the log matches
the kernel and modules that
are running
right now and I'll use the
default options above for
symbol resolution.
If the current kernel and/or
modules do not match the
log, you can get
more accurate output by
telling me the kernel version
and where to find
map, modules, ksyms etc.
ksymoops -h explains the
options.
No modules in ksyms, skipping
objects
Warning (read_lsmod): no
symbols in lsmod, is
/proc/modules a valid lsmod
file?
Oops: 0000
CPU: 0
EIP: 0010:[<c02118d1>]
Using defaults from ksymoops
-t elf32-i386 -a i386
EFLAGS: 00010086
eax: 00000000 ebx: c1592960
ecx: ce5d3600 edx:
000332be
esi: 00000006 edi: c15dd560
ebp: c1592960 esp:
cf221dd8
ds: 0018 es: 0018 ss: 0018
Process zeppelin (pid: 214,
stackpage=cf221000)
Stack: cfece440 00000006
c15dd560 c1592960 00000000
c019e041 ce5d3600 c1592960
c15a0000 c1592960 c15a0000
0f0c0020 c15dd560
cf221eb4 cfccc800 c2dd03a0
40017000 00000001 cf21705c
cfccc800 00000006
ce5d3600 cfece440 cfecf860
Call Trace: [<c019e041>]
[<c019d8ad>] [<c019c80d>]
[<c0107eaf>] [<c010800e>]
[<c0106d98>] [<c0211a94>]
[<c020ffe0>] [<c020f2a5>]
[<c011d1fe>]
[<c011d59d>] [<c01ce885>]
[<c01372e7>] [<c0106d0f>]
Code: 8b 68 60 85 db 75 18 8b
54 24 18 52 55 e8 dd 17
00 00 83 c4
>>EIP; c02118d1
<lec_push+19/140> <=====
Trace; c019e041
<dequeue_rx+75d/db4>
Trace; c019d8ad
<process_rsq+15/4c>
Trace; c019c80d
<ns_irq_handler+119/33c>
Trace; c0107eaf
<handle_IRQ_event+2f/58>
Trace; c010800e
<do_IRQ+6e/b0>
Trace; c0106d98
<ret_from_intr+0/7>
Trace; c0211a94
<lec_vcc_attach+9c/c0>
Trace; c020ffe0
<atm_push_raw+0/44>
Trace; c020f2a5
<atm_ioctl+429/9a8>
Trace; c011d1fe
<unmap_fixup+62/12c>
Trace; c011d59d
<do_munmap+251/260>
Trace; c01ce885
<sock_ioctl+21/28>
Trace; c01372e7
<sys_ioctl+16b/184>
Trace; c0106d0f
<system_call+33/38>
Code; c02118d1
<lec_push+19/140>
00000000 <_EIP>:
Code; c02118d1
<lec_push+19/140> <=====
0: 8b 68 60 mov
0x60(%eax),%ebp <=====
Code; c02118d4
<lec_push+1c/140>
3: 85 db test %ebx,%ebx
Code; c02118d6
<lec_push+1e/140>
5: 75 18 jne 1f <_EIP+0x1f>
c02118f0 <lec_push+38/140>
Code; c02118d8
<lec_push+20/140>
7: 8b 54 24 18 mov
0x18(%esp,1),%edx
Code; c02118dc
<lec_push+24/140>
b: 52 push %edx
Code; c02118dd
<lec_push+25/140>
c: 55 push %ebp
Code; c02118de
<lec_push+26/140>
d: e8 dd 17 00 00 call 17ef
<_EIP+0x17ef> c02130c0
<lec_vcc_close+0/308>
Code; c02118e3
<lec_push+2b/140>
12: 83 c4 00 add $0x0,%esp
Kernel panic: Aiee, killing
interrupt handler!
2 warnings issued. Results
may not be reliable.
Logged In: YES
user_id=43292
From duplicate [#468750 ] still: ATM (LANE) -related
Kernel-Crashe:
attached below is the output of another kernel crash.
The machine is running Kernel
2.4.7 / atm-0.78.
In the past, I had had
several crashes running
different
kernel/atm-versions,. The
machines are HP Netserver
with a ForeRunner LE ATM-NIC
(hardware is checked and
o.k.). The other network
equipment (ATM-,
Ether-switches .. is also
mostly by Fore Systems.
The crashes seem to occur
always at night without any
significant load on the
machine or the network. All the
crashes occured in ATM/LANE
code, unfortunately on
different places. Besides
that, I personally can't
identify any particular
pattern.
Maybe somebody with more
insight into LANE has any
idea, what is
going wrong ...
Is anybody else experiencing
similar problems?
Regards,
Peter Daum
- - - - - 8< - - - - - 8< - -
- - - 8< - - - - - 8< -
- - - - 8< - - - - -
Oops: 0000
CPU: 0
EIP: 0010:[<c0213e61>]
EFLAGS: 00010086
eax: 00000000 ebx: c12c8d20
ecx: c6664400 edx:
000bbb4a
esi: 00000006 edi: c12db560
ebp: c12c8d20 esp:
c62f9dd8
ds: 0018 es: 0018 ss: 0018
Process zeppelin (pid: 84,
stackpage=c62f9000)
Stack: c1298ac0 00000006
c12db560 c12c8d20 00000000
c0198451 c6664400
c12c8d20
c12a0000 c12c8d20 c12a0000
0f0c0020 c12db560
c62f9eb4 c7edd580
c6807920
4000d000 00000001 c62fe034
c7edd580 00000006
c6664400 c1298ac0
c7f57260
Call Trace: [<c0198451>]
[<c0197cbd>] [<c0196c1d>]
[<c0107eaf>]
[<c010800e>] [<c0106d98>]
[<c021402b>]
[<c0212570>] [<c0211835>]
[<c011d1fe>]
[<c011d59d>] [<c01d1315>]
[<c01372e7>] [<c0106d0f>]
Code: 8b 68 60 85 db 75 18 8b
54 24 18 52 55 e8 dd 17
00 00 83 c4
>>EIP: c0213e61
<lec_push+19/140>
Trace: c0198451
<dequeue_rx+75d/db4>
Trace: c0197cbd
<process_rsq+15/4c>
Trace: c0196c1d
<ns_irq_handler+119/33c>
Trace: c0107eaf
<handle_IRQ_event+2f/58>
Trace: c010800e
<do_IRQ+6e/b0>
Trace: c0106d98
<ret_from_intr+0/7>
Trace: c021402b
<lec_vcc_attach+a3/c0>
Trace: c0212570
<atm_push_raw+0/44>
Code: c0213e61
<lec_push+19/140> 00000000
<_EIP>: <===
Code: c0213e61
<lec_push+19/140> 0: 8b
68 60
mov 0x60(%eax),%ebp <===
Code: c0213e64
<lec_push+1c/140> 3: 85
db
test %ebx,%ebx
Code: c0213e66
<lec_push+1e/140> 5: 75
18
jne c0213e80
<lec_push+38/140>
Code: c0213e68
<lec_push+20/140> 7: 8b
54 24 18
mov 0x18(%esp,1),%edx
Code: c0213e6c
<lec_push+24/140> b:
52
push %edx
Code: c0213e6d
<lec_push+25/140> c:
55
push %ebp
Code: c0213e6e
<lec_push+26/140> d: e8
dd 17 00 00
call c0215650
<lec_vcc_close+0/308>
Code: c0213e73
<lec_push+2b/140> 12: 83
c4 00
add $0x0,%esp
Kernel panic: Aiee, killing
interrupt handler!
Logged In: YES
user_id=43292
Duplicate bug reports #454141 and #468750 were deleted and
their related info put into the comments below...
Logged In: NO
.. here some more of the same - this time Kernel 2.4.12,
still with atm 0.78
Regards ,
Peter Daum
ksymoops 2.4.0 on i686 2.4.12. Options used
Oops: 0000
CPU: 0
EIP: 0010:[<c020ff51>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010082
eax: 00000000 ebx: c610b120 ecx: c438a200 edx:
00089ffc
esi: cbb448d4 edi: cf190030 ebp: c610b120 esp:
c6819de8
ds: 0018 es: 0018 ss: 0018
Process zeppelin (pid: 766, stackpage=c6819000)
Stack: c610b080 cbb448d4 cf190030 c610b120 00000000 c019c45b
c438a200 c610b120
c15e0000 0f0c0020 c14de560 c6819eb8 cff614e0 c15e2cc0
c019bb1d c15e0000
c14cce70 0000003e c438a200 cff626c8 cff61580 c15e2cc0
c019bb1d c15e0000
Call Trace: [<c019c45b>] [<c019bb1d>] [<c019bb1d>]
[<c019aa7d>] [<c0107f1f>]
[<c010807e>] [<c0109e88>] [<c0210114>] [<c020e6b0>]
[<c020d9a5>] [<c011da19>]
[<c01cd6d5>] [<c0134cf7>] [<c0106d1f>]
Code: 8b 68 60 85 db 75 18 8b 54 24 18 52 55 e8 dd 17 00 00
83 c4
>>EIP; c020ff51 <lec_push+19/140> <=====
Trace; c019c45b <dequeue_rx+907/d1c>
Trace; c019bb1d <process_rsq+15/4c>
Trace; c019bb1d <process_rsq+15/4c>
Trace; c019aa7d <ns_irq_handler+119/33c>
Trace; c0107f1f <handle_IRQ_event+2f/58>
Trace; c010807e <do_IRQ+6e/b0>
Trace; c0109e88 <call_do_IRQ+5/d>
Trace; c0210114 <lec_vcc_attach+9c/c0>
Trace; c020e6b0 <atm_push_raw+0/44>
Trace; c020d9a5 <atm_ioctl+405/954>
Trace; c011da19 <do_munmap+22d/23c>
Trace; c01cd6d5 <sock_ioctl+21/28>
Trace; c0134cf7 <sys_ioctl+16b/184>
Trace; c0106d1f <system_call+33/38>
Code; c020ff51 <lec_push+19/140>
00000000 <_EIP>:
Code; c020ff51 <lec_push+19/140> <=====
0: 8b 68 60 mov 0x60(%eax),%ebp
<=====
Code; c020ff54 <lec_push+1c/140>
3: 85 db test %ebx,%ebx
Code; c020ff56 <lec_push+1e/140>
5: 75 18 jne 1f <_EIP+0x1f>
c020ff70 <lec_push+38/140>
Code; c020ff58 <lec_push+20/140>
7: 8b 54 24 18 mov 0x18(%esp,1),%edx
Code; c020ff5c <lec_push+24/140>
b: 52 push %edx
Code; c020ff5d <lec_push+25/140>
c: 55 push %ebp
Code; c020ff5e <lec_push+26/140>
d: e8 dd 17 00 00 call 17ef <_EIP+0x17ef>
c0211740 <lec_vcc_close+0/308>
Code; c020ff63 <lec_push+2b/140>
12: 83 c4 00 add $0x0,%esp
<0>Kernel panic: Aiee, killing interrupt handler!
Logged In: NO
Hi,
... here's yet another crash, presumably caused by the same
problem,
still Linux 2.4.12/atm-0.78, this time on a different
machine ...
Regards,
Peter Daum
- - - - 8< - - - - - 8< - - - - - 8< - - - - - 8< - - - - -
8< - -
Oops: 0000
CPU: 0
EIP: 0010:[<c0213371>] Not tainted
EFLAGS: 00010086
eax: 00000000 ebx: c23c4c00 ecx: c5fca400 edx:
000194db
esi: 00000006 edi: c12fc560 ebp: c23c4c00 esp:
c6305de0
ds: 0018 es: 0018 ss: 0018
Process zeppelin (pid: 613, stackpage=c6305000)
Stack: c7f7f740 00000006 c12fc560 c23c4c00 00000000 c01966e0
c5fca400 c23c4c00
c12c0000 c23c4c00 c12c0000 100c0020 c12fc560 c6305eb8
4000d000 00000001
c7a66034 c7f0f580 00000001 00000006 c5fca400 c7f7f740
c7f7e760 c12c0cec
Call Trace: [<c01966e0>] [<c0195f8d>] [<c0194eed>]
[<c0107f1f>] [<c010807e>]
[<c0109e88>] [<c0213534>] [<c0211ad0>] [<c0210dc5>]
[<c011da19>] [<c01d1015>]
[<c0134cf7>] [<c0106d1f>]
Code: 8b 68 60 85 db 75 18 8b 54 24 18 52 55 e8 dd 17 00 00
83 c4
>>EIP: c0213371 <bind_vcc+132d/358c>
Trace: c01966e0 <autoirq_report+2374/3be0>
Trace: c0195f8d <autoirq_report+1c21/3be0>
Trace: c0194eed <autoirq_report+b81/3be0>
Trace: c0107f1f <__up_wakeup+223b/2264>
Trace: c010807e <enable_irq+e2/124>
Trace: c0109e88 <disable_irq_nosync+18a4/220c>
Trace: c0134cf7 <kill_fasync+2ef/308>
Code: c0213371 <bind_vcc+132d/358c> 00000000
<_EIP>: <===
Code: c0213371 <bind_vcc+132d/358c> 0: 8b 68
60
mov 0x60(%eax),%ebp <===
Code: c0213374 <bind_vcc+1330/358c> 3: 85
db
test %ebx,%ebx
Code: c0213376 <bind_vcc+1332/358c> 5: 75
18
jne c0213390 <bind_vcc+134c/358c>
Code: c0213378 <bind_vcc+1334/358c> 7: 8b 54
24 18
mov 0x18(%esp,1),%edx
Code: c021337c <bind_vcc+1338/358c> b:
52
push %edx
Code: c021337d <bind_vcc+1339/358c> c:
55
push %ebp
Code: c021337e <bind_vcc+133a/358c> d: e8 dd
17 00 00
call c0214b60 <bind_vcc+2b1c/358c>
Code: c0213383 <bind_vcc+133f/358c> 12: 83 c4
00
add $0x0,%esp
<0>Kernel panic: Aiee, killing interrupt handler!
Logged In: NO
I guess, it is still the same problem. This time kernel
version 2.4.25:
Oops: 0000
CPU: 0
EIP: 0010:[<c02a4f5b>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010082
eax: c5934800 ebx: c645a080 ecx: c645a080 edx: 00000000
esi: 00000000 edi: 0000000e ebp: c7fc0000 esp: c680fdf4
ds: 0018 es: 0018 ss: 0018
Process lt-zeppelin (pid: 107, stackpage=c680f000)
Stack: c7fc0000 c01ea242 c7fc0000 00000000 00000000 c5934800
c7fe6384 00000000
0000000e c7fc0000 c01e984a c5934800 c645a080 ffffffff
0009f25f 00030002
0000000a ffffffff 00000005 00000000 0000006c 00000246
ffffffff c56145a0
Call Trace: [<c01ea242>] [<c01e984a>] [<c01e8f3c>]
[<c01e7e0d>] [<c010a6f5>]
[<c010a8b9>] [<c010cfc8>] [<c02a51e3>] [<c02a2380>]
[<c02a044e>] [<c02a16b8>]
[<c02460f6>] [<c014c2e0>] [<c012c0f2>] [<c010911f>]
Code: 8b 6a 6c 0f 84 70 01 00 00 fc 8b b3 84 00 00 00 bf c0
ed 31
>>EIP; c02a4f5b <lec_push+2b/220> <=====
Trace; c01ea242 <dequeue_sm_buf+142/170>
Trace; c01e984a <dequeue_rx+8da/e90>
Trace; c01e8f3c <process_rsq+3c/70>
Trace; c01e7e0d <ns_irq_handler+36d/430>
Trace; c010a6f5 <handle_IRQ_event+45/70>
Trace; c010a8b9 <do_IRQ+69/b0>
Trace; c010cfc8 <call_do_IRQ+5/d>
Trace; c02a51e3 <lec_vcc_attach+93/b0>
Trace; c02a2380 <atm_push_raw+0/70>
Trace; c02a044e <try_atm_lane_ops+3e/60>
Trace; c02a16b8 <vcc_ioctl+258/350>
Trace; c02460f6 <sock_ioctl+26/30>
Trace; c014c2e0 <sys_ioctl+b0/260>
Trace; c012c0f2 <sys_munmap+42/70>
Trace; c010911f <system_call+33/38>
Code; c02a4f5b <lec_push+2b/220>
00000000 <_EIP>:
Code; c02a4f5b <lec_push+2b/220> <=====
0: 8b 6a 6c mov 0x6c(%edx),%ebp
<=====
Code; c02a4f5e <lec_push+2e/220>
3: 0f 84 70 01 00 00 je 179 <_EIP+0x179>
c02a50d4 <lec_push+1a4/220>
Code; c02a4f64 <lec_push+34/220>
9: fc cld
Code; c02a4f65 <lec_push+35/220>
a: 8b b3 84 00 00 00 mov 0x84(%ebx),%esi
Code; c02a4f6b <lec_push+3b/220>
10: bf c0 ed 31 00 mov $5x31edc0,%edi
<0>Kernel panic: Aiee, killing interrupt handler!