Re: [MooseFS-Users] Append operator not working on FreeBSD 10.3+

[Marin B. <li...@ol...>]

Hi,

Thank you for taking the time to deal with this.

I have tried to mount MFS with explit -o mfscachemode=YES on a FreeBSD 10.3-RELEASE server with MooseFS 3.0.84. It works fine. It also seems to work with mfscachemode=AUTO. Do you recommend using it or should we stick to the YES mode?

I also tried to mount MFS with the same options on a FreeBSD 11.0-RELEASE-p1 box, with MooseFS 3.0.81. This time, specifying the cache mode did not fix the issue: append mode is still overwriting the destination file. I do not know if this happens because I'm using an older version of the client (3.0.81 vs 3.0.84), or if this is related to changes in FreeBSD 11.0 regarding the FUSE implementation.

Since we're dealing with cache, I noticed another strange thing, both on FreeBSD 10.3 and 11.0. If my understanding is correct, data caching is disabled by default by mfsmount when it's running on FreeBSD. Yet, it seems caching is still operating behind the scene. Here is what I noticed a few days ago:
- Let's say node A and node B have access to the same MFS file system, and run mfsmount with default options.
- The file system contains a file, e.g. "file.txt".
- If I run "cat file.txt" on both nodes, I see the same contents.
- Now, node A deletes the file.
- Then, node A creates a new file with the same name but different contents.
- On node A: "cat file.txt" displays the contents of the new file
- On node B: "cat file.txt" displays the contents of the old file

How can this happen if data caching is disabled? Is it related to FreeBSD own caching process? Do you know about a way to fix this?

Many thanks, again, for your time.

Marin.

On Thu, 20 Oct 2016 15:39:48 +0200
Aleksander Wieliczko <ale...@mo...> wrote:

> Hi,
> Thank you for all this information.
> 
> I would like to inform that during our investigation we have noticed,
> that problem is connected with fuse/kernel.
> Problem appears when we disable cache. In such a situation, when we open
> file with O_APPEND flag, write process starts with offset 0, and size of
> the file is the size of last written data.
> 
> MooseFS client by default starts with mfscachemode=DIRECT on FreeBSD
> operating system, because only then FreeBSD kernel can use larger block
> size than 4k for read and write operations.
> We are trying to find the best solution for this issue but this process
> can take some time.
> 
> Right now we strongly advise to mount MooseFS client with
> mfscachemode=YES option.
> 
> mfsmount -o mfscachemode=YES -H mfsmaster.host /mfs/path
> 
> This step will cause performance decrease but you will be able to append
> file properly.
> 
> 
> We will inform you, when we find some reasonable solution for this case.
> 
> Best regards
> Aleksander Wieliczko
> Technical Support Engineer
> MooseFS.com <moosefs.com>


-- 
Marin Bernard <li...@ol...>