Re: [fuse-devel] Question about big_writes

[Bradley W. S. <set...@or...>]

Performance.  I want to write the data out in parallel -- a special 
parallel file system implemented in fuse.  I could build my own 
buffering, but then a 128MB write could require as much as 256MB of RAM.

   I can perhaps get by with 128K writes, it just doesn't work in my 
prototype implementation.  I may still be be able to get decent 
performance with less parallelism.  I will see if I can engineer around 
it.  Thanks for the response.

Cheers,
Brad

On 11/06/2009 04:11 AM, Miklos Szeredi wrote:
> On Thu, 05 Nov 2009, Bradley W. Settlemyer wrote:
>> Hello
>>
>>     I am developing a fuse file system that will need to leverage
>> reasonably large write buffers.  For example my clients tend to write
>> hundreds of MB in a single read or write call, and I need my fuse file
>> system to operate on write buffers of at least 128MB.
>>
>>     I've mounted with -odirect_io,big_writes,max_write=128000000 flags,
>> but I still seem to only be receiving 128KB buffers for each write.
>>
>>    For example, my fuse debug output:
>>
>> write[1] 131072 bytes to 127401968 flags: 0x8002
>>      write[1] 131072 bytes to 127401968
>>      unique: 1123, success, outsize: 24
>> unique: 1124, opcode: WRITE (16), nodeid: 22, insize: 131152
>> write[1] 131072 bytes to 127533040 flags: 0x8002
>>      write[1] 131072 bytes to 127533040
>>      unique: 1124, success, outsize: 24
>> unique: 1125, opcode: WRITE (16), nodeid: 22, insize: 131152
>> write[1] 131072 bytes to 127664112 flags: 0x8002
>>      write[1] 131072 bytes to 127664112
>>      unique: 1125, success, outsize: 24
>>
>> So it appears that fuse knows the client's write buffer is 128MB, but
>> fuse seems to be chunking up the write no matter what I do.  Any guidance?
>
> Currently fuse doesn't support>128k writes "out of the box".  There
> have been patches floating around that raise the limit, but they have
> some side effects, so I'm not integrating these yet.
>
> Why do you need these very large buffers?
>
> Thanks,
> Miklos
>