Re: [Moosefs-users] How big can MooseFS really go?

[Thomas S H. <tha...@gm...>]

Thanks Michal! I really appreciate your efforts for us, I hope I can give
back more to moosefs in the future.

we are planning on giving it 128 MB of ram, minimum, and then we are going
to toss a fusionio drive into the master and set it up as swap space <just
in case>

Right now we are serving files via apache, and the web front-end is entirely
in django using wsgi as the python interface. We are looking into moving it
to nginx, as most of our frontends use nginx, but an apache module would
encourage us to stick with apache!

But one of the big epiphanies came yesterday when we realized that we can
use fusionio drives as swap space, because they perform at almost the speed
of ram. We are probably not looking at this for a while, but I am serious
about giving you guys a fusionio drive in the future if it will help moosefs
development.

We did have one question, what are the chances that the master servers
exports could be dynamically loaded?

This and the client caching we are really counting on! We have been
crunching the numbers and it looks like we will be resting somewhere around
3.2 Petabytes when we are done with the build out. We will keep you posted
and we deeply appreciate your efforts, not only with MooseFS, but also on
our behalf!

-Tom

2011/1/21 Michal Borychowski <mic...@ge...>

>  Hi Thomas!
>
>
>
> Sorry for late reply but the last two days were very busy and full of
> meetings.
>
>
>
> We consulted your installation among our devs and admins and to be honest
> we cannot give you any more tips to what Jose and Reinis have already said.
> For a start you should have about 64GB RAM available in the master server
> (probably 48GB would be also fine). Performance is not that affected by the
> space in the cluster as by the number of files (objects) in the system and
> number of current operations (recorded in changelogs).
>
>
>
> As we already talked your files don’t change often (if ever) so if you
> prepare client machines with much RAM quite a lot of files would get cached
> in RAM on them (see
> http://www.moosefs.org/news-reader/items/moose-file-system-v-1617-released.html).
>
>
>
>
> And what server would you use to serve the files? Lighttpd / Apache or any
> other? We had some plans to implement a direct module in Apache for MooseFS
> – it should also speed up the communication.
>
>
>
>
>
> Regards
>
> Michal
>
>
>
>
>
>
>
>
>
> *From:* Thomas S Hatch [mailto:tha...@gm...]
> *Sent:* Tuesday, January 18, 2011 6:56 PM
> *To:* Michal Borychowski
> *Cc:* moosefs-users
> *Subject:* Re: [Moosefs-users] How big can MooseFS really go?
>
>
>
> FusionIO in all the chunkservers, thats a little too rich for my blood :)
>
>
>
> one of the problems we are seeing is that we need to have the failover work
> faster, and the bottleneck for mfsmetarestore looks like it is IO, and the
> mfsmetarestore in the failover process is what takes up the most time. Thats
> why we want the fusionio drives.
>
>
>
> As for number of files, we have about 13 million right now, but we have
> only imported a small percentage of the total number of files we are
> contracted to get (we are making a music "cloud", and we are getting all the
> music in the world, a lot is still coming). By the end of the year we should
> have about 100-150 million files.
>
>
>
> Right now the idea is to have two types of storage, the moose for large
> scale storage where write speed is not a major issue, and then a "cluster"
> of ssd pci cards to handle high speed storage needs, like databases, and the
> master and metalogger to speed up restores and to make sure the changelogs
> can be written fast enough when the activity is so high.
>
>
>
> Paint a better picture?
>
>
>
> -Tom
>
> 2011/1/18 Michal Borychowski <mic...@ge...>
>
> WOW!!!
>
>
>
> And what about FusionIO in all the chunkservers? ;) But as you are talking
> about FusionIO in mfsmaster so probably you are also going to use them in
> metaloggers? I just think if it is necessary… Metadata is cached in RAM in
> mfsmaster, but there are changelogs… If the system will be busy (and for
> sure it will be) there would be lots of operations logged to the files and
> transmitted to metaloggers…
>
>
>
> Please tell us how many files do you plan to store?
>
>
>
>
>
> Regards
>
> Michal
>
>
>
>
>
> *From:* Thomas S Hatch [mailto:tha...@gm...]
> *Sent:* Tuesday, January 18, 2011 6:06 PM
> *To:* moosefs-users
> *Subject:* [Moosefs-users] How big can MooseFS really go?
>
>
>
> I am architecting a potential 3P MooseFS install.... 3 Peta-bytes... and
> maybe even 4P (assuming we will be moving to 4T disks when they come out).
>
>
>
> My question is this, can MooseFS handle that kind of load? Are there any
> additional considerations that I will need to take as I approach such a high
> volume.
>
>
>
> As I am seeing I will have over 100 chunkservers attached to one master. I
> am going to change out the mfsmaster metadata store with fusionio (
> http://www.fusionio.com/) drives to maintain the disk speed that
> metadata operations will need.
>
>
>
> This deployment will also require well over 100 million chunks.
>
>
>
> So my question again is, what, if any, special considerations should I take
> as I roll this out?
>
>
>
> -Thomas S Hatch
>
>
>