Re: [MooseFS-Users] MooseFS 4.0 and new features

[Marin B. <li...@ol...>]

> On 20.05.2018 15:38, Marin Bernard wrote:
> > I may have some answers.
> > 
> > > There where some posts saying that MooseFS doesn't support fsync
> > > at
> > > all, is this fixed? Can we force the use of fsync to be sure that
> > > an
> > > ACK is sent if and only if all data is really wrote on disks?
> > 
> > MooseFS chunkservers have the HDD_FSYNC_BEFORE_CLOSE config setting
> > (default off) which does just that. The equivalent setting on
> > LizardFS
> > is PERFORM_FSYNC (default on). The performance penalty with
> > PERFORM_FSYNC=1 is very high, though.
> > 
> > 
>  
> Funny thing, just this morning I did some testing with this flag on
> and off.
> 
> Basically, I was running with fsync on ever since, because it just
> felt as the right thing to do. And it's the only change I ever made
> to mfschunkserver.cfg, all other settings being at their default
> values. But, just yesterday, suddenly I wondered, if it would make
> any difference for my case, where chunkservers are rather far away
> from the mount point (about 10 - 15 ms away).
> 
> So, I did some basic tests, writing a large file, and untarring linux
> kernel source (lots of small files) with fsync setting on/off. 
> 
> Much as expected, there was no difference at all for large files
> (network interface saturated, anyway). In case of very small files
> (linux kernel source), performance per chunkserver improved from
> about 350 writes/sec to about 420 writes/sec. IOW, depending on your
> workload, it seems that turning HDD_FSYNC_BEFORE_CLOSE on could
> result in 0-15% performance degradation. Of course, depending on your
> infrastructure (network, disks...), the results may vary.
> 
> I eventually decided, it's not a big price to pay, and I'm dealing
> mostly with bigger files anyway, so I turned the setting back on and
> don't intend to do any more testing.

I saw a much important difference with LizardFS on a ZFS backend, even
with large files. Roughly 50-60% faster without fsync. I never ran the
test on MooseFS, so it's hard to tell the difference. You gave me the
idea to try it though!

It is possible that LizardFS performs slower than MooseFS when fsync is
enabled. Afterall, their implementations have had enough time to drift.
 However, such a perf gap between both solutions seem unlikely to me.

What is certain, however, is that the performance cost of fsync with
ZFS is far higher than with another FS. In standard async mode, ZFS
delays and batches disk writes to minimize IO latency. fsync predates
this optimization without disabling it, and as a result performance
drops quickly. I could try to set sync=always on the dataset, leave
fsync off at chunkserver level, and see what happens. This would offer
the same warranty than fsync=on while letting the filesystem handle
synchronous writes by itself (and trigger its own optimization
strategies, maybe not using batched transactions at all with
synchronous writes).

Thanks for sharing your figures with us!

Marin.