From: Chris E. <cem...@ch...> - 2001-05-15 03:08:50
|
The handling of PTRACE_{PEEK,POKE}USR in sys_ptrace had a hard-coded 17 used for deciding the highest allowed offset. I've changed the "17*sizeof(long)" to UM_MAX_USER_OFFSET, with appropriate defines in sysdep-$(SUBARCH)/ptrace.h. Someone please come up with a better name for the define! After this fix strace seems to work fine on uml/ppc, which should help with my debugging. :-) Cheers, Chris -- Chris Emerson, obsessed Cambridge juggler E-mail: cem...@ch... Web page: http://www.chiark.greenend.org.uk/~cemerson/ |
From: Jeff D. <jd...@ka...> - 2001-05-15 04:11:18
|
cem...@ch... said: > The handling of PTRACE_{PEEK,POKE}USR in sys_ptrace had a hard-coded > 17 used for deciding the highest allowed offset. Oops. > I've changed the "17*sizeof(long)" to UM_MAX_USER_OFFSET, with > appropriate defines in sysdep-$(SUBARCH)/ptrace.h. OK, I'll add that to my pool. After I think of a good name for it. > After this fix strace seems to work fine on uml/ppc, which should help > with my debugging. :-) Yeah, having strace working is definitely useful... Have you figured out the ld.so problem yet? Jeff |
From: Chris E. <cem...@ch...> - 2001-05-15 13:09:05
|
On Tuesday, 15 May 2001, Jeff Dike wrote: > cem...@ch... said: > > I've changed the "17*sizeof(long)" to UM_MAX_USER_OFFSET, with > > appropriate defines in sysdep-$(SUBARCH)/ptrace.h. > > OK, I'll add that to my pool. After I think of a good name for it. Ok, cool. > > After this fix strace seems to work fine on uml/ppc, which should help > > with my debugging. :-) > > Yeah, having strace working is definitely useful... I'd say essential. :-) Is it worth trying gdb, BTW, or is it not likely to work? > Have you figured out the ld.so problem yet? Yes, I think so. On ppc start_thread() has to re-arrange the elf data on the stack slightly to align it to 16 bytes. This wasn't being done, and this sometimes confused ld.so into thinking that it had been invoked directly rather than as the elf interpreter. Hence ignoring argv[0]. Phew! I've added a "UM_ARCH_ADJUST_STACK(sp)" macro which gets called (if defined) from start_thread(). I'll know in a few minutes whether it works or not. Cheers, Chris -- Chris Emerson, obsessed Cambridge juggler E-mail: cem...@ch... Web page: http://www.chiark.greenend.org.uk/~cemerson/ |
From: Jeff D. <jd...@ka...> - 2001-05-15 14:47:14
|
cem...@ch... said: > I'd say essential. :-) I actually implemented ptrace fairly late in the game, but life was definitely simpler once I did it. > Is it worth trying gdb, BTW, or is it not > likely to work? If ptrace works, gdb should. And if it doesn't, you need to find out anyway. > On ppc start_thread() has to re-arrange the elf data on the stack > slightly to align it to 16 bytes. This wasn't being done, and this > sometimes confused ld.so into thinking that it had been invoked > directly rather than as the elf interpreter. Hence ignoring argv[0]. > Phew! > I've added a "UM_ARCH_ADJUST_STACK(sp)" macro which gets called (if > defined) from start_thread(). OK, that sounds like the right thing. Jeff |
From: Chris E. <cem...@ch...> - 2001-05-15 23:25:29
|
On Tuesday, 15 May 2001, Jeff Dike wrote: > cem...@ch... said: > > Is it worth trying gdb, BTW, or is it not > > likely to work? > > If ptrace works, gdb should. And if it doesn't, you need to find out anyway. Cool, I'll see how I get on with it at some point then. > > On ppc start_thread() has to re-arrange the elf data on the stack > > slightly to align it to 16 bytes. This wasn't being done, and > > this sometimes confused ld.so into thinking that it had been > > invoked directly rather than as the elf interpreter. Hence > > ignoring argv[0]. Phew! > > > I've added a "UM_ARCH_ADJUST_STACK(sp)" macro which gets called > > (if defined) from start_thread(). > > OK, that sounds like the right thing. Great. The fix does work, and now I don't have to count the arguments carefully anymore. :-) I've also fixed another problem. Turns out that on PPC it's enough to set the SYSCALL_RET register with the result. When it's an error (ie result < 0) you have to negate the result _and_ set one of the flags. I've added a macro UM_SET_SYSCALL_RET(regs, result), which is used instead of UM_SYSCALL_RET(regs). The latter could probably be renamed UM_GET_SYSCALL_RET(regs), or removed entirely. After this I can actually run an interactive bash shell. I haven't managed to boot my Debian base system normally yet, since I generally get one of the infamous "Unexpectedly got signal 4 in signals" somewhere along the line. Rather than tracer_panicing, do you think that gdb could be usefully used to handle the signal? It might help to give some sort of clue about what's happened. Cheers, Chris -- Chris Emerson, obsessed Cambridge juggler E-mail: cem...@ch... Web page: http://www.chiark.greenend.org.uk/~cemerson/ |
From: Jeff D. <jd...@ka...> - 2001-05-16 00:40:02
|
cem...@ch... said: > The fix does work, and now I don't have to count the arguments > carefully anymore. :-) Excellent. > Turns out that on PPC it's enough to set the SYSCALL_RET register with > the result. ^^^ not? > When it's an error (ie result < 0) you have to negate the result _and_ > set one of the flags. I've added a macro UM_SET_SYSCALL_RET(regs, > result), which is used instead of UM_SYSCALL_RET(regs). The latter > could probably be renamed UM_GET_SYSCALL_RET(regs), or removed > entirely. Removing it is probably the best idea. Unfortunately, sys_sigreturn gets the return value, so a UM_GET_SYSCALL_RET(regs) is necessary for now. > Rather than tracer_panicing, do you think that gdb could be usefully > used to handle the signal? It might help to give some sort of clue > about what's happened. That should have been done a long time ago. In signals(), just move SIGBUS and SIGILL down to the SIGIO/SIGALRM/SIGVTALRM/SIGFPE block. Jeff |
From: Chris E. <cem...@ch...> - 2001-05-17 08:57:03
|
On Tuesday, 15 May 2001, Jeff Dike wrote: > cem...@ch... said: > > Rather than tracer_panicing, do you think that gdb could be > > usefully used to handle the signal? It might help to give some > > sort of clue about what's happened. > > That should have been done a long time ago. In signals(), just move > SIGBUS and SIGILL down to the SIGIO/SIGALRM/SIGVTALRM/SIGFPE block. That's not quite the right thing to do, since the child is continued and immediately dies from the SIGILL, and the debugger doesn't hear anything. Cheers, Chris -- Chris Emerson, obsessed Cambridge juggler E-mail: cem...@ch... Web page: http://www.chiark.greenend.org.uk/~cemerson/ |
From: Jeff D. <jd...@ka...> - 2001-05-17 14:21:23
|
cem...@ch... said: > That's not quite the right thing to do, since the child is continued > and immediately dies from the SIGILL OK, try this patch: --- cvs/linux/arch/um/kernel/trap_user.c Tue May 15 17:00:04 2001 +++ um/arch/um/kernel/trap_user.c Thu May 17 09:16:28 2001 @@ -208,11 +208,6 @@ if(debugger_pid != -1) child_signal(pid, status); continue; - case SIGBUS: - case SIGILL: - tracer_panic("Unexpectedly got signal %d in " - "signals", sig); - break; case SIGTRAP: sig = 0; if(switching_modes(task)) tracing = 0; @@ -236,7 +231,7 @@ case SIGALRM: case SIGVTALRM: case SIGFPE: - if(!tracing && debugger_pid != -1){ + if(!tracing && (debugger_pid != -1)){ child_signal(pid, status); continue; } @@ -248,11 +243,14 @@ case SIGCHLD: sig = 0; break; + case SIGBUS: + case SIGILL: default: if(debugger_pid != -1){ child_signal(pid, status); continue; } + tracing = 0; break; } set_tracing(task, tracing); |
From: Chris E. <cem...@ch...> - 2001-05-17 15:00:47
|
On Thursday, 17 May 2001, Jeff Dike wrote: > cem...@ch... said: > > That's not quite the right thing to do, since the child is continued > > and immediately dies from the SIGILL > > OK, try this patch: Thanks, that works. Unfortunately it looks like both the PC and return address are at the same illegal instruction. Speaking of debugger things... I'd quite like an option to use something other than an xterm in the cases where an xterm is used. In particular, I'd find it useful to open consoles or the debugger as "screen" windows when I'm working remotely. Since screen doesn't have the equivalent to xterm's -S option, I guess it would need a helper program to do some pty proxying (or pass fds around). This isn't an area I'm very familiar with; do you think it's reasonably feasible? Cheers, Chris -- Chris Emerson, obsessed Cambridge juggler E-mail: cem...@ch... Web page: http://www.chiark.greenend.org.uk/~cemerson/ |
From: Jeff D. <jd...@ka...> - 2001-05-17 16:01:52
|
cem...@ch... said: > Unfortunately it looks like both the PC and return address are at the > same illegal instruction. Have a look at the stack. See if it looks "stack-like" :-) > I'd quite like an option to use something other than an xterm in the > cases where an xterm is used. I've tried that. You can try 'gdb=<chan>' just like the 'ssl=' and 'con=' switches. The one I'd like most to have is 'gdb=pty', but I can't get the pty modes right. Try it and you'll see the problem. Jeff |
From: Chris E. <cem...@ch...> - 2001-05-17 22:33:30
|
On Thursday, 17 May 2001, Jeff Dike wrote: > cem...@ch... said: > > Unfortunately it looks like both the PC and return address are at the > > same illegal instruction. > > Have a look at the stack. See if it looks "stack-like" :-) It looks stack-like. :-) I might see if I can build a version of bash with debugging symbols, and get gdb to load those. Might make it slightly easier. > > I'd quite like an option to use something other than an xterm in > > the cases where an xterm is used. > > I've tried that. You can try 'gdb=<chan>' just like the 'ssl=' and > 'con=' switches. The one I'd like most to have is 'gdb=pty', but I > can't get the pty modes right. > > Try it and you'll see the problem. Ok, I'll try it sometime. Cheers, Chris -- Chris Emerson, obsessed Cambridge juggler E-mail: cem...@ch... Web page: http://www.chiark.greenend.org.uk/~cemerson/ |
From: <lis...@os...> - 2001-05-18 01:46:45
|
Chris Emerson writes: > On Thursday, 17 May 2001, Jeff Dike wrote: > > > cem...@ch... said: > > > That's not quite the right thing to do, since the child is continued > > > and immediately dies from the SIGILL > > > > OK, try this patch: > > Thanks, that works. Unfortunately it looks like both the PC and > return address are at the same illegal instruction. > > Speaking of debugger things... I'd quite like an option to use > something other than an xterm in the cases where an xterm is used. In > particular, I'd find it useful to open consoles or the debugger as > "screen" windows when I'm working remotely. Since screen doesn't have > the equivalent to xterm's -S option, I guess it would need a helper > program to do some pty proxying (or pass fds around). This isn't an > area I'm very familiar with; do you think it's reasonably feasible? It's something that will be useful fo me too, if I decude to try the AIX port. The AIX computer being in Melbourne, and me being one continent (Australia) and modem away in Perth. |
From: Chris E. <cem...@ch...> - 2001-05-18 12:10:00
|
On Thursday, 17 May 2001, Chris Emerson wrote: > On Thursday, 17 May 2001, Jeff Dike wrote: > > > cem...@ch... said: > > > Unfortunately it looks like both the PC and return address are at the > > > same illegal instruction. > > > > Have a look at the stack. See if it looks "stack-like" :-) > > It looks stack-like. :-) I might see if I can build a version of > bash with debugging symbols, and get gdb to load those. Might make it > slightly easier. Well, I've got slightly more information now. I can reproduce the SIGILL on ppc by booting with init=/bin/ash, running /bin/bash, and from there typing "ls". The directory listing completes, but the bash process gets the illegal instruction. I've compared the code at the problem address with the same part of the same bash under gdb in the host. It's within bash's xmalloc() function. The code is identical up to and including the branch to malloc(), but after that in the UML it's garbage. Some of the values in memory there look like stack addresses, and some others look like code addresses. It looks like something somehow tried to use one of the return addresses on the stack (or maybe in the link register) somewhere below malloc() as a stack pointer, hence corrupting everything. I'll try to further narrow down where it's happening. Cheers, Chris -- Chris Emerson, obsessed Cambridge juggler E-mail: cem...@ch... Web page: http://www.chiark.greenend.org.uk/~cemerson/ |
From: Chris E. <cem...@ch...> - 2001-05-18 14:55:56
|
On Friday, 18 May 2001, Chris Emerson wrote: > I've compared the code at the problem address with the same part of > the same bash under gdb in the host. It's within bash's xmalloc() > function. The code is identical up to and including the branch to > malloc(), but after that in the UML it's garbage. Some of the > values in memory there look like stack addresses, and some others > look like code addresses. Actually, it looks more like a register set than part of a stack. The first corrupted word is 0x000000ae, which happens to be the number of the last syscall before the SIGILL (rt_sigprocmask), in the right place for the syscall number within the register set. Still don't know why it's happening, but I'm working on it. Cheers, Chris -- Chris Emerson, obsessed Cambridge juggler E-mail: cem...@ch... Web page: http://www.chiark.greenend.org.uk/~cemerson/ |
From: Jeff D. <jd...@ka...> - 2001-05-18 16:07:43
|
cem...@ch... said: > The code is identical up to and including the branch to malloc(), but > after that in the UML it's garbage. Some of the values in memory > there look like stack addresses, and some others look like code > addresses. Is the boundary between goodness and garbage a page boundary, by any chance? If so, then there's an incorrectly mapped page. Jeff |
From: Chris E. <cem...@ch...> - 2001-05-18 16:38:39
|
On Friday, 18 May 2001, Jeff Dike wrote: > cem...@ch... said: > > The code is identical up to and including the branch to malloc(), > > but after that in the UML it's garbage. Some of the values in > > memory there look like stack addresses, and some others look like > > code addresses. > > Is the boundary between goodness and garbage a page boundary, by any > chance? > > If so, then there's an incorrectly mapped page. No, it's not. I've seen it in two places now, and both times it was just after a subroutine call (so a return address looks misused). Looking at the corrupted data as a struct sys_pt_regs is consistent with register state on entry to the rt_sigprocmask system call (ie r0 is __NR_rt_sigprocmask, the other registers look "reasonable", and trap is 0xc00, which means a system call trap). Everything looks reasonable after sys_rt_sigprocmask returns, so it may be somewhere near set_user_thread() or one of the signal handlers. Cheers, Chris -- Chris Emerson, obsessed Cambridge juggler E-mail: cem...@ch... Web page: http://www.chiark.greenend.org.uk/~cemerson/ |
From: Jeff D. <jd...@ka...> - 2001-05-18 17:30:51
|
cem...@ch... said: > I've seen it in two places now, and both times it was just after a > subroutine call (so a return address looks misused). I don't think I'm understanding exactly what's happening. The picture I've got so far is that you're ending up with code in the process that looks like good asm nice asm call some proc garbage garbage garbage And you're getting a SIGILL when the process starts executing the garbage. And the garbage looks like a stack that has just run a signal handler. Is this close? And if that garbage is a stack, you should see some UML code addresses in it, which ought to let you build a partial UML stack. Jeff |
From: Chris E. <cem...@ch...> - 2001-05-18 18:11:34
|
On Friday, 18 May 2001, Jeff Dike wrote: > cem...@ch... said: > > I've seen it in two places now, and both times it was just after a > > subroutine call (so a return address looks misused). > > I don't think I'm understanding exactly what's happening. The picture I've > got so far is that you're ending up with code in the process that looks like > > good asm > nice asm > call some proc > garbage > garbage > garbage > > And you're getting a SIGILL when the process starts executing the > garbage. Right so far. > And the garbage looks like a stack that has just run a signal handler. Is > this close? Close. The garbage looks (to me) like a struct sys_pt_regs you might get in a system call entry point: r0 has __NR_rt_sigprocmask (which was indeed a recent system call). Several other registers have addresses in bash' text segment or the user stack. The trap register has the value you'd expect from a system call trap. It doesn't look like a stack after all. I've set a breakpoint in sys_rt_sigprocmask and traced through until returning from sig_handler; all seems to be fine at that point. By the time we get back to execute_syscall for the next system call, the corruption has happened. The signal handler bit is a guess, but I'm not sure how else it could happen. I hope that's clearer, but I'm not sure it is. :-) Let me know how I can clarify further. Or even better, some good ideas on what's going on. :-) Cheers, Chris -- Chris Emerson, obsessed Cambridge juggler E-mail: cem...@ch... Web page: http://www.chiark.greenend.org.uk/~cemerson/ |
From: Jeff D. <jd...@ka...> - 2001-05-18 19:11:16
|
cem...@ch... said: > The garbage looks (to me) like a struct sys_pt_regs you might get in a > system call entry point: r0 has __NR_rt_sigprocmask (which was indeed > a recent system call). I was thinking it might be a struct sigcontext, which also contains a register set. But if r0 is __NR_rt_sigprocmask, then you're probably right about it being a struct sys_pt_regs. If i386 is any guide, that would be -1 in a signal, so if it's not, it's the register set from a system call. > I've set a breakpoint in sys_rt_sigprocmask and traced through until > returning from sig_handler; all seems to be fine at that point. It sounds like you can predict when it's about to happen. Can you also predict where it's going to happen? > By > the time we get back to execute_syscall for the next system call, the > corruption has happened. If you can, then you just need better granularity, which you can get by switching over to debugging the tracing thread. The general idea would be to watch every signal that the tracing thread sees, watch the area that's going to be corrupted, and see which signal is responsible. Then narrow in on what the tracing thread is doing during that signal. Though, if I had to guess, I'd say that something odd is happening in exit_kernel when the rt_sigprocmask goes back to userspace. The process registers are restored then, and maybe something is going wrong at that point. Jeff |
From: Chris E. <cem...@ch...> - 2001-05-18 21:20:18
|
On Friday, 18 May 2001, Jeff Dike wrote: > I was thinking it might be a struct sigcontext, which also contains > a register set. But if r0 is __NR_rt_sigprocmask, then you're > probably right about it being a struct sys_pt_regs. If i386 is any > guide, that would be -1 in a signal, so if it's not, it's the > register set from a system call. *nods* > It sounds like you can predict when it's about to happen. Can you > also predict where it's going to happen? Yep. It's currently quite consistent. > > By the time we get back to execute_syscall for the next system > > call, the corruption has happened. > > If you can, then you just need better granularity, which you can get > by switching over to debugging the tracing thread. Ah, ok. That makes sense. > The general idea would be to watch every signal that the tracing > thread sees, watch the area that's going to be corrupted, and see > which signal is responsible. Then narrow in on what the tracing > thread is doing during that signal. *nods* I'll have a play (in a day or two, when I'm back near the machine) and see what I can find out. I'll report back when I've done so. Cheers, Chris -- Chris Emerson, obsessed Cambridge juggler E-mail: cem...@ch... Web page: http://www.chiark.greenend.org.uk/~cemerson/ |
From: Jeff D. <jd...@ka...> - 2001-05-18 19:29:21
|
cem...@ch... said: > The garbage looks (to me) like a struct sys_pt_regs you might get in a > system call entry point: r0 has __NR_rt_sigprocmask (which was indeed > a recent system call). One thing I forgot to mention - look at current_task.thread.process_regs and see if it's the same as the garbage. If so, that's pretty much a smoking gun. Somehow, those registers are being 'restored' to the middle of the process text. Jeff |
From: Chris E. <cem...@ch...> - 2001-05-19 13:58:33
|
On Friday, 18 May 2001, Jeff Dike wrote: > One thing I forgot to mention - look at > current_task.thread.process_regs and see if it's the same as the > garbage. If so, that's pretty much a smoking gun. Somehow, those > registers are being 'restored' to the middle of the process text. They're identical. What's the easiest way of checking bits of process memory from the tracing thread? I guess um_virt_to_phys or similar. Cheers, Chris -- Chris Emerson, obsessed Cambridge juggler E-mail: cem...@ch... Web page: http://www.chiark.greenend.org.uk/~cemerson/ |
From: Jeff D. <jd...@ka...> - 2001-05-19 14:40:50
|
cem...@ch... said: > They're identical. Excellent. I would watch what's happening in exit_kernel right after that system call veeerrrry carefully. > What's the easiest way of checking bits of process memory from the > tracing thread? If you only know the virtual address: call pgd_offset_proc(current_task.mm, 0xdeadbeef) call pmd_offset_proc($, 0xdeadbeef) call pte_offset_proc($, 0xdeadbeef) p/x $.pte_low That will give you the physical page corresponding the process virtual 0xdeadbeef. You'll have to stick the low bits of the page in to it to finish the virt -> phys conversion of the address. Jeff |
From: Chris E. <cem...@ch...> - 2001-05-21 15:25:09
|
On Saturday, 19 May 2001, Jeff Dike wrote: > cem...@ch... said: > > What's the easiest way of checking bits of process memory from the > > tracing thread? > > If you only know the virtual address: > > call pgd_offset_proc(current_task.mm, 0xdeadbeef) > call pmd_offset_proc($, 0xdeadbeef) > call pte_offset_proc($, 0xdeadbeef) > p/x $.pte_low Hmm, that doesn't work for me: # (gdb) call pgd_offset_proc(current_task->next_task->mm, 0x10048000) # # Program received signal SIGSEGV, Segmentation fault. # 0x200e521c in strlen () at hostfs_kern.c:672 # 672 } # The program being debugged was signaled while in a function called # from GDB. # GDB remains in the frame where the signal was received. # To change this behavior use "set unwindonsignal on" # Evaluation of the expression containing the function (pgd_offset_proc) # will be abandoned. I have found the problem though. signal_handler() allocates a struct sigcontext on the stack and uses fill_in_sigcontext() on it. On ppc, the sigcontext only has a *pointer* to a register set (ie a sys_pt_regs), not the actual register values themselves. fill_in_sigcontext() was then dereferencing that (uninitialised) pointer. This is one of those "I can't believe it worked at all" bugs. :-) I've fixed it by adding YA macro, UM_ALLOCATE_SC(name), which sets up a sys_pt_regs and the pointer in the sigcontext. Maybe SC_ALLOCATE would be more consistent with other bits in sysdep/sigcontext.h. For future reference, I found out the culprit by watching for the address being corrupted being passed in to do_wp_page() in mm/memory.c. Since the address was in the code segment, it was initially mapped read-only until someone wrote to it. Next problem is that the mount options are being corrupted, but that's easy in comparision. :-) I guess I'm due to make another patch soon. Cheers, Chris -- Chris Emerson, obsessed Cambridge juggler E-mail: cem...@ch... Web page: http://www.chiark.greenend.org.uk/~cemerson/ |
From: Jeff D. <jd...@ka...> - 2001-05-21 16:54:02
|
cem...@ch... said: > I've fixed it by adding YA macro, UM_ALLOCATE_SC(name), which sets up > a sys_pt_regs and the pointer in the sigcontext. Maybe SC_ALLOCATE > would be more consistent with other bits in sysdep/sigcontext.h. OK, the sigcontext stuff is partly gone because it wasn't really being used. I'm not calling fill_in_regs and fill_in_sigcontext any more, but the code is still there. The code that you replaced with GET_SC is gone. I declared the sigcontext as a parameter to the signal handlers so the compiler should get references to it right on all platforms. > For future reference, I found out the culprit by watching for the > address being corrupted being passed in to do_wp_page() in mm/ > memory.c. Yup. Been there, done that. That's how I managed to track down a few of the trickiest corruption bugs. Write-protect the memory and see where it faults. > I guess I'm due to make another patch soon. Yeah, sync up with my CVS. I've been whittling down your patch and I'd like to see how much I have left. Nice work, BTW. Jeff |