Re: [BProc] vmadump performance

[Erik A. H. <er...@he...>]

On Tue, Jun 04, 2002 at 04:58:30PM -0400, Grant Taylor wrote:
> > On Thu, Mar 07, 2002 at 07:20:28PM -0500, Grant Taylor wrote:
> >> Anyway, it's a little frentic here now, but after things settle down
> >> I'll put together a clean patch for mips.  
> 
> Well, I haven't gotten to the clean patch point yet (I still have
> nasty hacks in syscall entry to make it work) but now we're at the
> performance tuning stage...
> 
> I'm trying to make vmadump run faster.  Currently it looks like the
> following things are true:
> 
>  - Freezing is faster than thawing by around a factor of 2.

That's interesting...

>  - Having the dumping kernel connected directly to the undumping
>    kernel* pipelines the whole process and makes thawing the only wall
>    time spent.  So that's a 33% speedup.

This is what BProc does, btw.

>  - Having four CPUs migrate processes to four other CPUs all in
>    parallel pipelines that; so that's a 4X speedup.
> 
>  - Removing seemingly uneeded icache flushes for non-executable pages
>    in load_map makes no difference, time-wise, on my platform.  This
>    is a little surprising given 250MB of cache flushing that went
>    away, but so be it.

That's good.  I don't find it surprising though since most of the
addresses involved in the flush aren't going to be in the icache
anyway.  Flushing stuff out to main memory early shouldn't cause any
problem either since you're going to end up forcing that to happen
anyway.

>  - The network is not the bottleneck.  My cluster will TCP within
>    itself at nearly a gigabit, and the dumps go at maybe 10MB/s.

Hrm.  That's pretty slow.  I'm seeing 550m+bits/sec on our alpha
cluster here (IP over myrinet).

> This leaves me wondering why thawing is so much more expensive than
> freezing.  It's curious that writing the arriving dump to a file in
> tmpfs is way faster than thawing.  My assumption is that the extra
> time is sunk in the allocation of pages through mmap; this is the main
> difference between the various paths.  Is each read of a page's data
> triggering a page fault, or are the pages allocated at mmap time?  Is
> there something I can do to speed this up?

Every new page is going to cause a page fault.  mmap really just sets
up a memory region so that the fault handler knows what to do.  I
don't have any experience with Linux on MIPS myself.  It could be that
page faults are particularly slow or something.  That certainly
wouldn't surprise me.

> I don't suppose there would be a way to map the pages directly from a
> dump file and have a sort of execute-in-place semantic between the
> file and new process?  I'm statistically unlikely to touch pages right
> away in the process, so some of this work seems unnecessary.

No, not with the current dump file format.  The basic problem is
VMADump goes out of its way to send less than the whole process image.
Therefore the dump file is a bunch of pages that aren't necessarily
contiguous.  There's a tiny big of goop (page address) in front of
every page in the file too.

> * This was a PITA to make work right.  Does bproc do this?

BProc process migration runs freeze and thaw directly connected with a
TCP socket.

>   We found that merely closing the TCP socket on the transmitting
>   end would cause a premature connection reset on the receiving end
>   and thus hose the thaw. No variation on lingering or cloexec or
>   even sleeping worked reliably.  In the end I make the sender wait
>   in a loop for the TCP_INFO ioctl to return a TCP state of
>   TCP_CLOSE.  No way should this be necessary; something is horribly
>   amiss in my kernel, I think.

Whoa.  That's messed up.  I've run into number of TCP bugs including
one that results in a spurious connection reset but nothing that
matches that.  VMADump certainly should not cause any problems since
the only I/O related calls it makes are read and write.

The reset bug I saw looks like this: If two machines are connected (A
and B) and A calls shutdown(fd, 1) (i.e. no more sends from A, send
TCP FIN), then B will sometimes get connection reset by peer on
writes.

- Erik
-- 
Erik Arjan Hendriks          Printed On 100 Percent Recycled Electrons
er...@he...                   Contents may settle during shipment