Re: [htdig-dev] Compression related performance

[David B. <dav...@bo...>]

>   I think you are right.. doing a dump & reinsert would be a nice way to 
> optimize for size.
>
>   However in my tests both of them had about the same fill factor.. so the 
> compression db files will still be smaller.

Well that means that it didn't optimize for size then ;)
Perhaps the theory about inserting in key order yielding 100% FF is 
incorrect.

>   The more odd result is that Jim has about a 5x speed improvement when he 
> disables compression.  I also see a speedup.

Obviously you'd expect that in the case that the task is CPU-bound.

> Joe gets a 14% slowdown.

If Joe's disks are very slow compared to yours, that might
explain the difference (because the compressed output file
is smaller and hence can be written to disk quicker). Another
possibility is differences in filesystem : if in one case the data
is not actually being flushed to disk during the measured elapsed test time,
and in the other case it is, that could make a big difference, and again the
difference would depend on the size of the file.

Just some random ideas...

I'd try running tools like top, vmstat and iostat during the
test execution and see if they reveal anything interesting.