Re: [MooseFS-Users] MooseFS consistency model ?

[Krzysztof K. <krz...@mo...>]

Hi,

There is no simple answer to this question since in MooseFS, to gain the best performance, we use different kind of caching mechanisms on different levels:
- read-ahead cache (on client and chunkservers),
- write-back cache (on client and chunkservers),
- Metadata cache on clients.

MooseFS does enforce consistency model for synchronisation of writes to all copies of a single chunk - data written to all copies of the same chunk performed by different clients is always consistent. 

MooseFS however does not enforce any consistency model on a file level by default due to number of caches in the system, so let me discuss two typical model cases.

MapReduce Model

In typical MapReduce computational model, where reads are scheduled after completed writes to new files  (mapper results are read by reduce jobs) there is a guarantee that read performed by any client on the system will get consistent view of last write operation performed on any client  

write operation (creating new file - i.e. on the mapper job):

fd = open(“file.data",O_WRONLY|O_CREAT|O_EXCL);
write(fd,...);
fsync(fd);
close(fd);

read operation (after successful close in write operation - i.e. reduce job on mapper results):

fd = open(“file.data",O_RDONLY);
read(fd,...);
close(fd);

General Model

In general the only schema of operations in MooseFS that guarantees synchronisation between read and write workers should be coded as following (written as C pseudo-code): 

Synchronised Write:

fdl = open(“file.lock",O_RDWR|O_CREAT);
flock(fdl,LOCK_EX);
fd = open(“file.data",O_WRONLY);
write(fd,.....);
fsync(fd);
close(fd);
flock(fdl,LOCK_UN); /* formally should be there, but in practice close should remove the lock */
close(fdl);

Synchronised Read:

fdl = open(“file.lock",O_RDWR|O_CREAT);
flock(fdl,LOCK_SH);
fd = open(“file.data",O_RDONLY);
read(fd,.....);
close(fd);
flock(fdl,LOCK_UN); /* formally */
close(fdl);

To achieve “full” synchronisation you must use MooseFS locks mechanism (available in MooseFS 3.0+) on external objects - in our example “file.lock”. You can think of those external objects as a r/w locks in the operating system for thread synchronisation in typical consumer/producer scenario. 

In above example for clarity we used flock() mechanism (which is more common in BSD system and requires FUSE 2.9+), but in your application you can use any POSIX compliant locks mechanism that can be used with FUSE 2.6+

Best Regards,
Krzysztof Kielak
Director of Operations and Customer Support
Mobile: +48 601 476 440

> On 02 Sep 2015, at 10:08, Valerio Schiavoni <val...@gm...> wrote:
> 
> Hello Aleksander,
> for now I need simply a short answer, that is to understand if MooseFS guarantees eventual, weak, causal, strong consistency (or other flavours).
>  
> The how's and gory details are also relevant and useful but I can quietly wait for the longer answer.
> 
> Thanks a lot,
> --
> Valerio
> 
> 
> On Wed, Sep 2, 2015 at 7:53 AM, Aleksander Wieliczko <ale...@mo... <mailto:ale...@mo...>> wrote:
> Hi.
> We made so many changes till 2011 so you need to give as some time to generate answer on your question.
> We will return to this topic as fast as it is possible.
> 
> Best regards
> Aleksander Wieliczko
> Technical Support Engineer
> MooseFS.com <http://moosefs.com/>
> 
> On 31.08.2015 19:02, Valerio Schiavoni wrote:
>> Hello,
>> what is the consistency model of MooseFS? 
>> I've found a 2011 thread [1] on this ML that seems to be remained unanswered, and since we need to know the same details... 
>> 
>> Thanks.
>> 
>> 1 - http://sourceforge.net/p/moosefs/mailman/message/26888539/ <http://sourceforge.net/p/moosefs/mailman/message/26888539/>
>> 
>> --
>> Valerio
>> _________________________________________
>> moosefs-users mailing list
>> moo...@li... <mailto:moo...@li...>
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users <https://lists.sourceforge.net/lists/listinfo/moosefs-users>
> 
> 
> ------------------------------------------------------------------------------
> Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
> Get real-time metrics from all of your servers, apps and tools
> in one place.
> SourceForge users - Click here to start your Free Trial of Datadog now!
> http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 <http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140>
> _________________________________________
> moosefs-users mailing list
> moo...@li... <mailto:moo...@li...>
> https://lists.sourceforge.net/lists/listinfo/moosefs-users <https://lists.sourceforge.net/lists/listinfo/moosefs-users>
> 
> 
> ------------------------------------------------------------------------------
> Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
> Get real-time metrics from all of your servers, apps and tools
> in one place.
> SourceForge users - Click here to start your Free Trial of Datadog now!
> http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140_________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users