Re: [Moosefs-users] Bad write performance of mfschunkserver

[Chris P. <ch...@ec...>]

Hi Michal

In some tests I was running, with 50 disks spread over 18 machines.

I was using bonnie++ as its rewrite test matches some of our access
pattens.  The moosefs standard 1.6.20 was reporting a rewrite speed of
about 1.4MB/s

With a patch to remove fsyncs totally, the speed went up to about
20MB/s, which is a substantial speed increase.

However, forcing a power failure while disks were writing caused chunk
loss (because lots of data was in OS cache), and as you say, the blocks
data and checksum may not match, which was only picked up later when
using the filesystem again.  This was why I was thinking about forcing a
full checksum of all blocks.  Noting blocks which could possibly be
dirty is a more elegant way of dealing with this, though.

Currently I am testing with the patch from Davies Liu on the mailing
list yesterday, which runs the fsync on block close, or delayed.  I am
seeing about 1/3 the fsyncs of before, leading to percieved higher
performance - with no numbers to back that statement up yet  :)

Chris







On Tue, 2012-03-06 at 11:22 +0100, Michał Borychowski wrote:
> Hi!
> 
> I'm curious whether you made some tests with your solution? What was the performance gain? Was it like 20-30% or rather 2-3%?
> 
> 
> Actually here we are quite skeptical about this. In one of your emails you suggest to do fsync more rarely (eg. every 30 seconds). But when CS is closed cleanly, OS will do all fsyncs before closing the files but when CS is not closed cleanly so how should we know which files to test? CS upon startup doesn't do 'stat' on every file (it took too long time). So it won't know which files to check.
> 
> We could create some extra file (eg. named '.dirty') where we could save id of the file upon opening it (and do fsync on the '.dirty' file). Upon file closing we delete its id from '.dirty' file. When CS closes cleanly, '.dirty' file should be empty. If not, upon starting of CS, it reads the '.dirty' file and it scans all the chunks which are saved in this file.
> 
> 
> You also gave some suggestions to use this options:
> 1. FLUSH_ON_WRITE - option easy to implement, but not that secure 
> 2. FLUSH_DELAY - as above
> 3. CHECKSUM_INITIAL - this would mean to read all chunks on all the disks upon startup which is just impossible (in some environments would take more than 24hrs). 
> 
> 
> And still we are afraid that there may be scenario that malfunction of CS without fsyncs could cause that chunk will "return" to a proper form (in sense of CRC) from before the save. It will mean that there would be several "proper" copies of the same chunk, but with different content - we cannot allow this to happen. 
> 
> Possible it would be necessary to inform master server that CS has some 'unfsynced' chunks. So this gets still more complicated. That's why we are curious whether the performance gain is substantial enough to start doing this fine tuning.
> 
> 
> Kind regards
> Michał Borychowski 
> MooseFS Support Manager
> 
> 
> 

-- 
Chris Picton

Executive Manager - Systems
ECN Telecommunications (Pty) Ltd
t:   010 590 0031 m: 079 721 8521
f:   087 941 0813
e:  ch...@ec...

"Lowering the cost of doing business"