Re: [Moosefs-users] Some questions about MooseFS

[Robert S. <rsa...@ne...>]

Comments in the email.

Incomprehensible email compliments of my iPad

On Jul 24, 2012, at 7:41 PM, Michael Kagerbauer <in...@ka...> wrote:

> Hello,
> 
> I'm new to MooseFS and would like to ask you some questions:
> 
> 1.
> Is it safe to use a MooseFS-Cluster with chunkservers which are located 
> in different datacenters? What happens if one chunkserver goes down or 
> cannot reach the masterserver? What happens if the communication will 
> only be interrupted for some seconds or minutes, will MooseFS still 
> serve the files as long as there are enough chunks available and will it 
> detect when a chunkserver comes back online?
> 

No, you really do not want to do this. MooseFS does not respond well to lag and other network issues that will be introduced by this. Not only this but all chunk servers should be reachable by all clients.

> 2.
> We have a project with over 35 Million files with an approx. filesize of 
> 450KiB per file, 1000-1500 requests/sec. Is MooseFS suitable for such a 
> configuration or would you suggest a different filesystem?
> 

There are several bottlenecks in MooseFS that can be worked around. I have not found anything better. But in the end you are at best limited by the IOPS of the total number of storage devices you have. A single 7200 RPM drive can handle about 150 IOPS per second. Opening and reading a file is at best 2 IOPS. The only way to get 2000+ IOPS per second is to either use a very large number of disks, SSD or significant caching. 

> 3.
> 95% of all accesses are read-only. Is there some sort of "Caching 
> Frontend" available which can use MooseFS as a backend system (e.g. 
> Proxy-software, memcached or others) to lower the requests? Has anyone 
> documented some information about this and could provide me a link?
> 

I am currently testing a Squid CARP cluster using SSD as a caching front end for MooseFS. The initial tests look promising. Our data is only read through a custom HTTP server which has been written to load balance between multiple local instances of mfsmount.

> 4.
> We built a small cluster to test the behavior of MooseFS:
> Hardware: Intel QuadCore Xeon 2.53GH with 8 GB RAM and 2x 500 GB SATA 
> HDD with RAID0
> 2 Gigabit NIC's (one dedicated on private network for MooseFS), Gigabit 
> Switch, no Jumboframes
> 1x Masterserver, 1x Metalogger, 5x Chunkserver, 1 Client
> Goal-Setting: 3
> 
> Writing speed:
> dd if=/dev/zero of=/mnt/mfs/tempfile bs=1M count=1024 conv=fdatasync,notrunc
> 1073741824 bytes (1.1 GB) copied, 10.4868 s, 102 MB/s
> 
> Reading speed (without buffer/cache):
> dd if=/mnt/mfs/tempfile3 of=/dev/null bs=1M count=1024
> 1073741824 bytes (1.1 GB) copied, 13.0057 s, 82.6 MB/s
> 
> I expected that the read speed would be faster then writing, but the 
> opposite is the case. What can be the reason for this? This is the speed 
> on a single chunkserver locally:
> 
> Writing: 163 MB/s, Reading (without cache): 264 MB/sec
> 

Did you wait for all the files to reach the specified goal before you did the read test? A distributed file system introduces some overhead that can not be avoided, only minimized.

> 5.
> Can we use MooseFS for streaming video files (filesize approx. 30-60 MiB)?
> 

It is probably better for this than for lots of small files.

> 
> Michael
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users