Re: [Moosefs-users] Fwd: Append and seek while writing functionality

[Michał B. <mic...@ge...>]

 

From: Stas Oskin [mailto:sta...@gm...] 
Sent: Monday, June 14, 2010 1:27 PM
To: moo...@li...
Subject: [Moosefs-users] Fwd: Append and seek while writing functionality

 

 

Also, can any process / machine read file being written by other process /
machine?
(I presume it can from the ability to write to same file from multiple
machines).

[MB] Yes, it can.

So this covers appending, but can a write operation to file pause, jump to
file start for example, and update some data?

[MB] Yes, seeking (jumping) is fully supported by MooseFS (and writing in
that place).

The only thing you could not do is when two clients would like to
simultaneously append (write at the end) to the same file. FUSE library make
only operations similar to "pwrite" - saving of X data starting at Y
position. The starting position is dependent on the operating system.
Probably operation "append" would be consisted from several operations:

open,

getsize,

N*pwrite

close

So if two clients' computers make these operations in parallel then at the
beginning "getsize" would return on both clients the same value and both
would start writing data on the same position and they would overwrite data
by each other and after finishing the file would have a doubled length.

The only possible solution would be to allow only one computer for writing
from "open" till "close". Thanks to it only on computer at the given moment
could have the file opened for writing. For the moment we are not going to
implement this limitation because appending to the same file by several
computers is quite a strange thing. (Do you have a real life example where
it would be necessary and helpful?) The other option is just to use by
yourself lockfiles.

Or it would also be possible to introduce parallel appending to the same
file but in the form of "booking" the space - so every client which wants to
append knows how many bytes it wants to append. The scenario of writing X
bytes to Y file would like similarly to this:

lock(Y+".lock")

FD=open(Y)

S=FD.getsize()

FD.setsize(S+X)

unlock(Y+".lock")

FD.write(data,S,X)

FD.close()

 

Regards

Michał 

 

2010/6/14 Stas Oskin <sta...@gm...>

 

Hi.

Thanks for the explanation.

So this covers appending, but can a write operation to file pause, jump to
file start for example, and update some data?

Regards.

2010/6/14 MichaĹ, Borychowski <mic...@ge...>

 

MooseFS fully supports appending to a file and writing to any position of a
file. It also supports creating sparse files. 

 

Two independent processes / machines may write in parallel to the same file
at different positions. If two positions are located in different chunks
(pos1 / 64Mi != pos2 / 64Mi) the writing process would be run at normal
speed; if two positions are in the same chunk you can expect a substantial
loss of writing speed (due to chunk lock for writing).

 

 

Kind regards

MichaĹ, Borychowski 

MooseFS Support Manager

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Gemius S.A.

ul. WoĹ,oska 7, 02-672 Warszawa

Budynek MARS, klatka D

Tel.: +4822 874-41-00

Fax : +4822 874-41-01

 

 

 

From: Stas Oskin [mailto:sta...@gm...] 
Sent: Thursday, June 10, 2010 2:13 PM
To: moo...@li...
Cc: MooseFS
Subject: [Moosefs-users] Append and seek while writing functionality

 

Hi.

Does MooseFS supports append functionality?

Also, does it support the ability to seek file being it's being written and
write data in other place (like regular file system)?

Thanks!