Re: [Moosefs-users] Solution of small file store

[Ken <ken...@gm...>]

This way is too Linux. ha. How about growing?

The business maybe same as Flickr, Facebook photo and DropBox, etc...

If small files store resolved, moosefs will be used in more product
applications scenarios.

Thanks

-Ken


On Tue, Apr 24, 2012 at 4:10 PM,  <ro...@mm...> wrote:
> If you're on linux,
>
> i'd try a filesystem-in-a-file which would be used to store the photo and
> this file itselfs stored in a MooseFS volume.
>
> For example something along these lines:
>
> dd if=/dev/zero of=FILENAME bs=1024 count=1 seek=$((2*1024*1024-1))
> mkfs -t ext3 FILENAME
> mount -t ext3 -o loop FILENAME MOUNTPOINT
>
> where MOUNTPOINT is a MooseFS volume.
>
> -Stathis
>
> Wang Jian wrote:
>> TFS has strict and random URL scheme, so it's difficult to do URL based
>> tuning.
>>
>> δΊ  2012/4/24 15:11, Davies Liu ε  ι  :
>>> On Tue, Apr 24, 2012 at 2:29 PM, Ken <ken...@gm...
>>> <mailto:ken...@gm...>> wrote:
>>>
>>>     On Tue, Apr 24, 2012 at 2:22 PM, Davies Liu <dav...@gm...
>>> <mailto:dav...@gm...>> wrote:
>>>     > Maybe leveldb + MooseFS is better.
>>>     Is this your mean?
>>>      leveldb store url
>>>      moosefs store photo
>>>
>>> No, store url and photo in leveldb, which use MooseFS as disks.
>>> the performance may be not perfect, but enough.
>>> If not, TFS from taobao.com <http://taobao.com> may be the better
>>> choice.
>>>
>>>     >
>>>     > On Tue, Apr 24, 2012 at 1:59 PM, Ken <ken...@gm...
>>> <mailto:ken...@gm...>> wrote:
>>>     >>
>>>     >> We need to store tons of small files(photo), as noticed in faq,
>>> file
>>>     >> count is limited in moosefs, I think bundle small files to a huge
>>> file
>>>     >> maybe work.
>>>     >>
>>>     >> Write photos procedure like:
>>>     >>   allocate a huge file
>>>     >>   write head and photo content
>>>     >>   return (huge file, position, size)
>>>     >>   write another head and another photo
>>>     >>   return (huge file, position, size)
>>>     >>   ...
>>>     >>
>>>     >> Before read a photo, we should have enough information: huge
>>> file,
>>>     >> position and length, the reading procedure is expected normally.
>>>     >>
>>>     >> To read a photo, we should provide an URL, like
>>>     >> 'http://xxx.com/prefix/huge file/offset/size.jpg'
>>>     >>
>>>     >> And to be useful in WEB, build a fastcgi program for read/write
>>> access
>>>     >> to the huge file.
>>>     >>
>>>     >> ps:
>>>     >> *  The matchup information photo and url should store outside of
>>> mooosefs.
>>>     >> *  Huge file size limited in 2G.
>>>     >> *  Huge file maybe cause race condition.
>>>     >>
>>>     >> Is there anyone interested in this?
>>>     >> or better solution?
>>>     >>
>>>     >> -Ken
>>>     >>
>>>     >>
>>>     >> ------------------------------------------------------------------------------
>>>     >> Live Security Virtual Conference
>>>     >> Exclusive live event will cover all the ways today's security and
>>>     >> threat landscape has changed and how IT managers can respond.
>>> Discussions
>>>     >> will include endpoint security, mobile security and the latest in
>>> malware
>>>     >> threats.
>>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>     >> _______________________________________________
>>>     >> moosefs-users mailing list
>>>     >> moo...@li...
>>> <mailto:moo...@li...>
>>>     >> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>>     >
>>>     >
>>>     >
>>>     >
>>>     > --
>>>     >  - Davies
>>>
>>>
>>>
>>>
>>> --
>>>  - Davies
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond.
>>> Discussions
>>> will include endpoint security, mobile security and the latest in
>>> malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>
>>>
>>> _______________________________________________
>>> moosefs-users mailing list
>>> moo...@li...
>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats.
>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users