Re: [SSI-devel] 1.9 .text.lock.open race with .text.lock.inode
Brought to you by:
brucewalker,
rogertsang
From: Roger T. <rog...@gm...> - 2005-10-01 06:32:00
|
I found out that all Apache processes on both nodes were stuck waiting on the same inode. I sending a separate mail with the attached screen logs of bta on both nodes. Roger On 9/30/05, Roger Tsang <rog...@gm...> wrote: > > I ran into this text lock again.. The other processes aren't waiting for > IO, but can't be killed. Will get back with some stack traces. > > 5 S apache 83974 1 0 81 5 - 23689 tok_wa Sep29 ? 00:00:10 /usr/bin/httpd > -DSSL > 5 S apache 83975 1 0 80 5 - 25519 tok_wa Sep29 ? 00:00:32 /usr/bin/httpd > -DSSL > 5 S apache 83976 1 0 80 5 - 23183 tok_wa Sep29 ? 00:00:09 /usr/bin/httpd > -DSSL > 5 S apache 83977 1 0 80 5 - 24101 tok_wa Sep29 ? 00:00:09 /usr/bin/httpd > -DSSL > 5 D apache 83978 1 0 81 5 - 23212 text.l Sep29 ? 00:00:12 /usr/bin/httpd > -DSSL > 5 S apache 84006 1 0 80 5 - 23064 tok_wa Sep29 ? 00:00:07 /usr/bin/httpd > -DSSL > 5 S apache 84007 1 0 81 5 - 23514 tok_wa Sep29 ? 00:00:05 /usr/bin/httpd > -DSSL > 5 S apache 84008 1 0 80 5 - 25419 tok_wa Sep29 ? 00:00:31 /usr/bin/httpd > -DSSL > 5 S apache 84009 1 0 81 5 - 23143 tok_wa Sep29 ? 00:00:47 /usr/bin/httpd > -DSSL > 5 D apache 94810 1 0 81 5 - 23084 text.l Sep29 ? 00:01:43 /usr/bin/httpd > -DSSL > 5 S apache 116167 1 0 83 5 - 22937 tok_wa 01:39 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 116249 1 0 81 5 - 22315 tok_wa 01:40 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 121178 1 0 80 5 - 22315 tok_wa 01:50 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 121264 1 0 80 5 - 22315 tok_wa 01:50 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 121294 1 0 80 5 - 22327 tok_wa 01:50 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 121490 1 0 80 5 - 22315 tok_wa 01:51 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 121508 1 0 80 5 - 22315 tok_wa 01:51 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 121530 1 0 80 5 - 22315 tok_wa 01:51 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 122059 1 0 81 5 - 22315 tok_wa 01:53 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 122071 1 0 80 5 - 22315 tok_wa 01:53 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 122101 1 0 80 5 - 22315 tok_wa 01:53 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 122294 1 0 80 5 - 22315 tok_wa 01:54 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 122313 1 0 80 5 - 22315 tok_wa 01:54 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 122343 1 0 80 5 - 22327 tok_wa 01:54 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 122936 1 0 80 5 - 22315 tok_wa 01:55 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 123277 1 0 80 5 - 22315 tok_wa 01:55 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 123295 1 0 81 5 - 22315 tok_wa 01:55 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 123450 1 0 80 5 - 22315 tok_wa 01:55 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 123860 1 0 81 5 - 22315 tok_wa 01:57 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 123874 1 0 80 5 - 22343 tok_wa 01:57 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 123901 1 0 80 5 - 22327 tok_wa 01:57 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 124077 1 0 81 5 - 22952 tok_wa 01:58 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 124106 1 0 80 5 - 22315 tok_wa 01:58 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 124253 1 0 80 5 - 22315 tok_wa 01:58 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 124665 1 0 80 5 - 22327 tok_wa 02:00 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 126253 1 0 80 5 - 22315 tok_wa 02:03 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 126267 1 0 80 5 - 22315 tok_wa 02:03 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 126303 1 0 83 5 - 22909 tok_wa 02:03 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 127866 1 0 81 5 - 22315 tok_wa 02:06 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 127885 1 0 80 5 - 22343 tok_wa 02:06 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 127914 1 0 80 5 - 22316 tok_wa 02:06 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 128090 1 0 80 5 - 22315 tok_wa 02:07 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 128109 1 0 81 5 - 22327 tok_wa 02:07 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 128259 1 0 80 5 - 22315 tok_wa 02:07 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 129846 1 0 80 5 - 22315 tok_wa 02:10 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 129848 1 0 80 5 - 22315 tok_wa 02:10 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 129872 1 0 80 5 - 22315 tok_wa 02:10 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 130025 1 0 80 5 - 22315 tok_wa 02:10 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 130430 1 0 81 5 - 22315 tok_wa 02:12 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 130447 1 0 80 5 - 22343 tok_wa 02:12 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 130483 1 0 81 5 - 22315 tok_wa 02:12 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 130665 1 0 81 5 - 22315 tok_wa 02:13 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 130684 1 0 81 5 - 22315 tok_wa 02:13 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 130788 1 0 81 5 - 22937 tok_wa 02:13 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 66061 1 0 80 5 - 22315 tok_wa 02:15 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 66873 1 0 80 5 - 22315 tok_wa 02:15 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 67916 1 0 81 5 - 22315 tok_wa 02:19 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 67932 1 0 81 5 - 22952 tok_wa 02:19 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 68086 1 0 80 5 - 22327 tok_wa 02:19 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 68201 1 0 80 5 - 22315 tok_wa 02:20 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 69664 1 0 80 5 - 22315 tok_wa 02:22 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 69814 1 0 80 5 - 22327 tok_wa 02:22 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 70975 1 0 80 5 - 22315 tok_wa 02:25 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 71336 1 0 80 5 - 22315 tok_wa 02:25 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 71350 1 0 80 5 - 22315 tok_wa 02:25 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 71501 1 0 80 5 - 22327 tok_wa 02:25 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 72139 1 0 81 5 - 22315 tok_wa 02:28 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 72161 1 0 81 5 - 22315 tok_wa 02:28 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 72319 1 0 80 5 - 22952 tok_wa 02:28 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 72495 1 0 80 5 - 22319 tok_wa 02:29 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 72527 1 0 80 5 - 22315 tok_wa 02:29 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 72552 1 0 80 5 - 22315 tok_wa 02:29 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 72692 1 0 80 5 - 22315 tok_wa 02:30 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 73655 1 0 80 5 - 22343 tok_wa 02:30 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 73866 1 0 80 5 - 22315 tok_wa 02:31 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 73887 1 0 80 5 - 22315 tok_wa 02:31 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 74030 1 0 81 5 - 22315 tok_wa 02:31 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 74205 1 0 80 5 - 22315 tok_wa 02:32 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 74233 1 0 80 5 - 22315 tok_wa 02:32 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 74259 1 0 81 5 - 22315 tok_wa 02:32 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 74683 1 0 83 5 - 22925 tok_wa 02:34 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 74697 1 0 80 5 - 22315 tok_wa 02:34 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 74846 1 0 80 5 - 22973 tok_wa 02:34 ? 00:00:02 /usr/bin/httpd > -DSSL > 5 S apache 75132 1 0 80 5 - 22327 tok_wa 02:35 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 76407 1 0 80 5 - 22315 tok_wa 02:37 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 76429 1 0 80 5 - 22315 tok_wa 02:37 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 76573 1 0 80 5 - 22315 tok_wa 02:37 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 76986 1 0 81 5 - 22973 tok_wa 02:39 ? 00:00:01 /usr/bin/httpd > -DSSL > 5 S apache 76999 1 0 80 5 - 22973 tok_wa 02:39 ? 00:00:01 /usr/bin/httpd > -DSSL > 5 S apache 77036 1 0 80 5 - 22973 tok_wa 02:39 ? 00:00:02 /usr/bin/httpd > -DSSL > 5 S apache 77207 1 0 81 5 - 22973 tok_wa 02:40 ? 00:00:01 /usr/bin/httpd > -DSSL > 5 S apache 92715 1 0 80 5 - 22349 tok_wa 03:13 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 120716 1 0 81 5 - 22973 tok_wa 04:13 ? 00:00:01 /usr/bin/httpd > -DSSL > 5 S apache 68312 1 0 80 5 - 22348 tok_wa 07:02 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 93850 1 0 80 5 - 22348 tok_wa 07:58 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 78641 1 0 80 5 - 22343 tok_wa 09:45 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 78681 1 0 81 5 - 22315 tok_wa 09:45 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 79188 1 0 80 5 - 22315 tok_wa 09:47 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 79201 1 0 80 5 - 22315 tok_wa 09:47 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 79233 1 0 81 5 - 22315 tok_wa 09:47 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 79429 1 0 81 5 - 22343 tok_wa 09:48 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 79444 1 0 80 5 - 22327 tok_wa 09:48 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 79484 1 0 81 5 - 22319 tok_wa 09:48 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 80017 1 0 81 5 - 22315 tok_wa 09:50 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 81833 1 0 81 5 - 22315 tok_wa 09:53 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 81847 1 0 81 5 - 22327 tok_wa 09:53 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 81884 1 0 80 5 - 22315 tok_wa 09:53 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 82080 1 0 80 5 - 22315 tok_wa 09:54 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 82104 1 0 80 5 - 22315 tok_wa 09:54 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 82145 1 0 80 5 - 22319 tok_wa 09:54 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 83563 1 0 80 5 - 22315 tok_wa 09:55 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 83794 1 0 83 5 - 22915 tok_wa 09:56 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 83828 1 0 81 5 - 22315 tok_wa 09:56 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 84830 1 0 81 5 - 22343 tok_wa 10:00 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 88155 1 0 80 5 - 22973 tok_wa 10:05 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 88645 1 0 81 5 - 22327 tok_wa 10:05 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 88683 1 0 80 5 - 22315 tok_wa 10:05 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 88911 1 0 80 5 - 23039 tok_wa 10:06 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 88931 1 0 80 5 - 22316 tok_wa 10:06 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 89919 1 0 80 5 - 22316 tok_wa 10:10 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 90131 1 0 81 5 - 22314 tok_wa 10:10 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 90132 1 0 80 5 - 22327 tok_wa 10:10 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 98729 1 0 81 5 - 22315 tok_wa 10:25 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 98743 1 0 80 5 - 22315 tok_wa 10:25 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 98889 1 0 80 5 - 22315 tok_wa 10:25 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 99305 1 0 81 5 - 22315 tok_wa 10:27 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 99319 1 0 80 5 - 22315 tok_wa 10:27 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 99362 1 0 80 5 - 22316 tok_wa 10:27 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 99538 1 0 80 5 - 22327 tok_wa 10:28 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 99551 1 0 80 5 - 22343 tok_wa 10:28 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 99700 1 0 81 5 - 22315 tok_wa 10:28 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 100080 1 0 81 5 - 22343 tok_wa 10:30 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 100780 1 0 81 5 - 22315 tok_wa 10:30 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 102821 1 0 80 5 - 22316 tok_wa 10:35 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 106004 1 0 80 5 - 22327 tok_wa 10:40 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 108117 1 0 80 5 - 22314 tok_wa 10:45 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 109326 1 0 81 5 - 22314 tok_wa 10:45 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 109436 1 0 83 5 - 22917 tok_wa 10:45 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 111506 1 0 80 5 - 22315 tok_wa 10:50 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 112613 1 0 80 5 - 22343 tok_wa 10:52 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 112628 1 0 81 5 - 22315 tok_wa 10:52 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 112954 1 0 80 5 - 22315 tok_wa 10:53 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 112976 1 0 80 5 - 22315 tok_wa 10:53 ? 00:00:00 /usr/bin/httpd > -DSSL > 5 S apache 114824 1 0 81 5 - 22315 tok_wa 10:55 ? 00:00:00 /usr/bin/httpd > -DSSL > 0 R root 85473 80394 0 77 0 - 931 - 14:37 pts/0 00:00:00 grep httpd > > On 9/26/05, John Byrne <joh...@hp...> wrote: > > > > Roger Tsang wrote: > > > Hi, > > > > > > I found these two processes in a deadlock at text.lock and I can't > > kill > > > them either. Any ideas? > > > > > > Roger > > > > There is at probably least one more thread involved that has the > > semaphore and is deadlocked. One way to look for this is to use screen > > to capture all the output of the "bta" command in kdb (use LINES=3D1000= 0 > > to keep from having to page the output) and look for 0xc6bd5af4 which i= s > > > > the address of the dentry involved. You may also have to look for the > > inode address inode address, which you'd have to pull out of the dentry= . > > > > John > > > > > > > > > > > 0xccbc5020 120553 1 0 0 D 0xccbc51e0 httpd > > > EBP EIP Function (args) > > > 0xdbff1e50 0xc03befc9 schedule+0x2a9 (0xc027fb87, 0xe7f93884, > > > 0xe7f93804, 0x0, 0xe7f93884) > > > 0xdbff1e84 0xc03bfb25 rwsem_down_write_failed+0x75 (0xe7f93804, 0x0, > > > 0x0, 0x48, 0xc027514f) > > > 0xc0159168 .text.lock.open+0x13 > > > 0xdbff1ee4 0xc015782f do_truncate+0x5f (0xc6bd5af4, 0x0, 0x0, > > > 0xc6bd5af4, 0xdbff1f58) > > > 0xdbff1f08 0xc0169185 may_open+0x245 (0xdbff1f58, 0x2, 0x242, 0x0, > > > 0xdbff1f60) > > > 0xdbff1f40 0xc01692f6 open_namei+0x126 (0xe1937000, 0x242, 0x1b6, > > > 0xdbff1f58, 0xc6bd5af4) > > > 0xdbff1f9c 0xc0158afa filp_open+0x3a (0xe1937000, 0x241, 0x1b6, > > > 0x89f5e1c, 0x89f5e1c) > > > 0xdbff1fbc 0xc0158ef5 sys_open+0x55 > > > 0xc0103c55 sysenter_past_esp+0x52 > > > > > > 0xe9eb5550 123638 1 0 0 D 0xe9eb5710 httpd > > > EBP EIP Function (args) > > > 0xd3003dd0 0xc03befc9 schedule+0x2a9 (0xe7f93878, 0x1, 0xe9eb5550, > > > 0xc0118ea0, 0xe7f93878) > > > 0xd3003e08 0xc03beb74 __down+0x64 (0x0, 0xe7f93734) > > > 0xd3003e18 0xc03bece2 __down_failed+0xa (0xe7f93804, 0x0, 0x0, 0x2, > > 0x0) > > > 0xc0270095 .text.lock.inode+0x15 > > > 0xd3003e58 0xc026f2f3 cfs_setattr+0x93 (0xc6bd5af4, 0xd3003e98, > > 0x81a4, > > > 0x43383bdf, 0x6ebc849) > > > 0xd3003e88 0xc0175dd0 notify_change+0x240 (0xc6bd5af4, 0xd3003e98, > > 0x48, > > > 0xc027514f, 0xe7f93804) > > > 0xd3003ee4 0xc015783e do_truncate+0x6e (0xc6bd5af4, 0x0, 0x0, > > > 0xc6bd5af4, 0xd3003f58) > > > 0xd3003f08 0xc0169185 may_open+0x245 (0xd3003f58, 0x2, 0x242, 0x0, > > > 0xd3003f60) > > > 0xd3003f40 0xc01692f6 open_namei+0x126 (0xf6c0f000, 0x242, 0x1b6, > > > 0xd3003f58, 0xc6bd5af4) > > > 0xd3003f9c 0xc0158afa filp_open+0x3a (0xf6c0f000, 0x241, 0x1b6, > > > 0x891481c, 0x891481c) > > > 0xd3003fbc 0xc0158ef5 sys_open+0x55 > > > 0xc0103c55 sysenter_past_esp+0x52 > > > > > |