Re: [BProc] vmadump: mips port; apparenlt bug; etc

[Erik A. H. <er...@he...>]

On Sun, Mar 03, 2002 at 05:59:48PM -0500, Grant Taylor wrote:
> I've made vmadump go on the sb1 embedded mips cpu, at least mostly.
> Currently it only works in non-SMP mode...
> 
> The main thing for the port, and something which I'm not sure how
> you'll want to deal with, is the fact that on MIPS, kernel entry
> points save only some registers (as there are plenty of registers, and
> only a few are really used for syscalls, this saves a little time).
> At key times later (context switch, etc) there is code to save the
> rest of them.  Unfortunately it isn't at all easy to get at the
> unsaved registers, so I ended up saving them all always.  The slightly
> better choice is to have kernel entry magically recognize the vmadump
> syscall number and save all just for it.  
>
> Anyway, this somewhat precludes the current vmadump-as-a-module
> arrangement.  There might be other platforms with this property that
> vmadump will need to deal with someday, so it's worth pondering a bit.

The alpha has the same problem.  See the big (and ugly) syscall entry
code there for an example of what I did.  Basically the first bit of
the syscall handler is a hunk of asm code that saves what doesn't
normally get saved.  The alpha code is basically copied from context
switch and fork.  It seems (from a glance at MIPS fork code) that this
should be easily accomplished with the "save_static_function" macro.
It looks like FP will still be an issue on that platform.

> Another key thing, and an apparent general bug, is the (somewhat new)
> flush_icache_range() call in load_map, which explodes every time.
> This appears to be passing in a userspace address, while flush_icache
> functions seem to expect a real address.  I replaced this with a
> get_user_pages(), flush_icache_page() incantation:

[snip]

> Curiously, the bad flush_icache thing didn't fail in uniprocessor
> mode.  I don't know why this would be so; it seems to me that flushing
> effectively random addresses would be poor either way.  Evidently it's
> a nonfatal failure on my platform in uniprocessor mode.
> 
> Regardless, now thawing under SMP never panics the kernel.  Sometimes,
> thawing even works completely.  However, it usually still fails, and
> in a funky way.  The thawed process merely segfaults immediately or
> shortly after resurrection.  Occasionally it (vmadtest) will even
> print some of it's .+'s as it does things and then segv.  What's
> really interesting, is that often *my shell* will exit for no reason
> shortly after a segv'd thawed process dies.

Hrm.  Well, it works fine (and is required) on PPC.  I'm quite certain
the icache flush functions take user addresses - see kernel/module.c
and kernel/ptrace.c.  I'm basically doing the same thing there that
module.c does to make the I and D caches consistent.  If I was
flushing the wrong addresses, the PPC port should be broken too.

Since it's blowing up only on SMP and (I presume) you can successfully
load modules, I believe something else is wrong.

An immediate segfault in user space is the symptom I saw of failing to
sync the I and D caches on PPC.

PPC faulted in user space if I didn't set the permissions on the
segments properly before doing a flush_icache_range.  rwx should be
loose enough permission-wise for anything though.

* snip *

> If I run the thaw under GDB, then the GDB tends to bizzarely exit,
> so I can't really poke around easily to verify registers and memory
> contents ;(

Can you get a core dump out of it?  If it's really a user mode trap
and exit, that should be no problem.  GDB might be happier looking at
the core file.  I doubt the core file will show you anything other
than "all is well".  It's probably a waste of time.  GDB seems to get
confused fairly easily when messing with weird stuff.

> All in all, I'm inclined to think that the icache stuff is still
> wrong; it seems to be time-dependant as to whether or not the child
> will work, which I could see if there were dcache/icache interactions
> giving my core old code to run in the newly undumped process.
> Interestingly, using a flush_icache_all() at the end of the page
> loading behaves exactly the same.
> 
> Can anyone offer any suggestions?  I'm a bit puzzled...

Try takeing a closer look at how you're restoring the values that
aren't saved in the default syscall entry.  The text switch back to
user space needs to restore these values properly which means the
return from syscall needs to do some magic there.  Failing to restore
some of that will likely lead to a user space crash.  I saw that on
alpha.

The other thing I would double check was that you're not restoring any
important control registers from the dump file.  You usually have to
be careful about that to avoid opening up security holes or putting
the CPU in some bogus state once it tries to return to user mode.

> * Signal state can apparently be shared with another process, so
>   vmadump may have a bug here if something clones keeping signals and
>   then calls the vmadump thaw syscall before exec.  There's a static
>   function which breaks the sharing of task->sig in fs/exec.c.

VMADump should just overwrite the signal handler state for both
processes in that case.  VMADump behavior for processes that are
sharing resources (files, sig handlers, memory) is mostly undefined at
this point.

- Erik
-- 
Erik Arjan Hendriks          Printed On 100 Percent Recycled Electrons
er...@he...                   Contents may settle during shipment