Re: [Moosefs-users] Small file sizes revisited - 12x space used

[Ken <ken...@gm...>]

On Fri, Sep 30, 2011 at 3:44 PM, Patrick Feliciano <fus...@gm...> wrote:
> On 09/28/2011 07:26 PM, Kristofer Pettijohn wrote:
>> GFS2 in Google was redesigned for smaller files.  Multi-master design is needed, but that is a huge overhaul and a lot of work to complete.
>>
>> Ask and beg for it; you might see it some day.
>>
> Those are interesting points, that MooseFS has an architecture like
> GoogleFS and now Google has the GFS2 aka Colossus.  Colossus is designed
> for smaller files and has a distributed master design.  Maybe that is
> what MooseFS 2 will work to emulate as well.
>> On Sep 28, 2011, at 8:55 PM, Ken wrote:
>>
>>> Distribute filesystem always design for huge space. Waste often exist. eg:
>>> Haystack in facebook, GFS in google never recycling space of delete
>>> files, they mark flag for deleted status.
>>>
> It isn't true that all distributed file systems are designed for huge
> files.  Lustre for instance uses the block size of the underlying file
> system.  I disagree that the concept of distributed file systems is
> synonymous with large files.  That doesn't strike me as a valid reason
> to dismiss the idea of variable block sizes at compile time.
It is true what you said.
We have plan to use moosefs for a photo storage which growth is 1
terabyte per week.
Sincerely hope moosefs support small files. You know photos are small files.


>>> Much small size files put into moose filesystem cause master server
>>> memory bottleneck.
>>> IMHO, space saving will never be main target in these systems.
>>>
> My servers can support 148GB of RAM which is enough for hundreds of
> millions of files.  That would give our site years of growth, I'm not as
> worried about that as I am about the fact that we only have 10TB of
> space unused on the web farm that I want to use with MooseFS. With 64KB
> blocks we will run out of that space well before we reach a hundred
> million files.  With 3 copies of the data we'd be out already with just
> the 50 million files  we currently have.
Let's count a few.
In master server failover, mfsmetarestore should read meta log for
building filesystem.
Generally read speed can reach 100MB per second, 148G RAM meta data
should recover in 148*1024/100=1515 seconds
That mean a failure restore more than 25 minute. It's not easy to
resolve these problem.

In moosefs source core, filesystem.c and chunk.c have been too
difficult to understand now. These feature may make it worse.


>>> If we must handle much small files, just like photo files, should
>>> bundle them into a big file(s). And use URL locate content, like
>>> '/prefix/bundle_filename/offset/length/check_sum.jpg'.
> That is an interesting idea and I'm not against it if you can tell me
> what tools will do that and allow me to present it as a standard POSIX
> filesystem path.   Seems to me though that a smaller block size for this
> awesome filesystem is still the better fix.
We have plan to open source these tools.

>
>
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2dcopy2
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>