[Lifelines-dev] load performance

[amores p. <lif...@ho...>]

>From: "Matt Emmerton" <ma...@gs...>
>Subject: Re: [Lifelines-dev] Created cvs branch "dbf_2005_branch"
>Date: Sat, 15 Oct 2005 23:39:07 -0400
>

>The MSDN documentation points out two MS-specific features of fopen() that
>we should be using:
>- the "R" flag, which hints to the filesystem cache that the file will be
>accessed in "random" mode.
>- the "T" flag, which hints to the fs cache that the file is temporary and
>should not be commited to disk until fclose().
>
>I've added #defines for these two options (LLFILERANDOM and LLFILETEMP)
>which are empty strings on UNIX/Cygwin.
>I've gone through the btree code and added LLFILETEMP to the two places
>where we open temporary files.
>(I also made a separate commit to clean up lots of places where we were
>specifying hard-coded strings such as "rb" or "w" instead of the
>corresponding LL macros.)
>
>It would be interesting if the "temp" change improves the performance on
>Windows; if so, we can consider using it in more places (where appropriate)
>and also consider where the "random" flag would be appropriate.
>

That sounded promising, but my tests
(all on the same box, all done by importing
some of the stobie ged file -- each to a new
database), showed sadly no improvement.


MSVC6, btree import, release mode, network drive

    679 @ 20:24
  2,460 @ 20:33
  5,250 @ 20:47
16,386 @ 21:54
40,644 @ 00:53 (40,054/4h29m ~150ppm)

MSVC, btree import, release mode, local drive
(didn't time verify)
actual load
~15,000 @ +10min (1,500ppm)
~30,000 @ +20min (1,500ppm)

cygwin, btree import, local drive
verify 12min
actual load
~15,000 @ +10min (1,500ppm)
~30,000 @ +20min (1,500ppm)

cygwin, btree import, local drive (w/Matthew's fread fix)
verify from 13min
actual load
~15,000 @ +10min (1,500ppm)
~35,000 @ +23min (1,500ppm)

MSVC, btree import, release mode, local drive (w/Matthew's fread fix)
verify took 17min (100,000 at +3min)
actual load
~15,000 @ +10min (1,500ppm)
~39,000 @ +26min (1,500ppm)


Cordially,

Perry