Re: [Moosefs-users] Reliably detecting MFS is online

[Michal B. <mic...@ge...>]

Hi!

When the master server goes down while mfsmount is already running, mfsmount
doesn't disconnect the mounted resource and files awaiting to be saved would
stay quite long in the queue while trying to reconnect to the master server.
After a specified number of tries they eventually return EIO - "input/output
error". On the other hand it is not possible to start mfsmount when the
master doesn't work.

There are several ways to assure that the master server is online, we
present them below. 

1. You can check if you can connect to the port of the master server.

2. In order to assure that a MooseFS resource is mounted it is enough to
check the inode number - MooseFS root will always have inode equal to 1. For
example if we have MooseFS installation in "/mnt/mfs" then "stat /mnt/mfs"
command (in Linux) will show:

$ stat /mnt/mfs

File: `/mnt/mfs'

Size: xxxxxx Blocks: xxx IO Block: 4096 directory

Device: 13h/19d Inode: 1 Links: xx

(...)

 

3. Additionaly "mfsmount" simulates in the root folder a hidden file
".stats". So when MooseFS is mounted we would get statistics data of
mfsmount, eg.:

 

$ cat /mnt/mfs/.stats

fuse_ops.statfs: 241

fuse_ops.access: 0

fuse_ops.lookup-cached: 707553

fuse_ops.lookup: 603335

fuse_ops.getattr-cached: 24927

fuse_ops.getattr: 687750

fuse_ops.setattr: 24018

fuse_ops.mknod: 0

fuse_ops.unlink: 23083

fuse_ops.mkdir: 4

fuse_ops.rmdir: 1

fuse_ops.symlink: 3

fuse_ops.readlink: 454

fuse_ops.rename: 269

(...)

 

4. If you want to be sure that master server properly responds you need to
try to read the goal of any object, specifically of the root folder:

$ mfsgetgoal /mnt/mfs

/mnt/mfs: 2

If you get a proper goal of the root folder you can be sure that the master
server responds properly (hasn't hung up).

 

Regards

Michal

 

 

 

 

 

 

> Hi.

> 

> I wonder how other MFS users detect if their cluster is online?

> Because if the MFS goes down, from application point of view, the mount

> directory still up, and it will continue writing to local disk

> 

> We solve this by placing a special control file in root of the cluster,
but

> I wonder if there is a better solution, perhaps on MFS level?

> 

> Regards.