Re: [Moosefs-users] Solution of small file store

[Wang J. <jia...@re...>]

TFS has strict and random URL scheme, so it's difficult to do URL based tuning.

于 2012/4/24 15:11, Davies Liu 写道:
> On Tue, Apr 24, 2012 at 2:29 PM, Ken <ken...@gm... <mailto:ken...@gm...>> wrote:
>
>     On Tue, Apr 24, 2012 at 2:22 PM, Davies Liu <dav...@gm... <mailto:dav...@gm...>> wrote:
>     > Maybe leveldb + MooseFS is better.
>     Is this your mean?
>      leveldb store url
>      moosefs store photo
>
> No, store url and photo in leveldb, which use MooseFS as disks.
> the performance may be not perfect, but enough.
> If not, TFS from taobao.com <http://taobao.com> may be the better choice.
>
>     >
>     > On Tue, Apr 24, 2012 at 1:59 PM, Ken <ken...@gm... <mailto:ken...@gm...>> wrote:
>     >>
>     >> We need to store tons of small files(photo), as noticed in faq, file
>     >> count is limited in moosefs, I think bundle small files to a huge file
>     >> maybe work.
>     >>
>     >> Write photos procedure like:
>     >>   allocate a huge file
>     >>   write head and photo content
>     >>   return (huge file, position, size)
>     >>   write another head and another photo
>     >>   return (huge file, position, size)
>     >>   ...
>     >>
>     >> Before read a photo, we should have enough information: huge file,
>     >> position and length, the reading procedure is expected normally.
>     >>
>     >> To read a photo, we should provide an URL, like
>     >> 'http://xxx.com/prefix/huge file/offset/size.jpg'
>     >>
>     >> And to be useful in WEB, build a fastcgi program for read/write access
>     >> to the huge file.
>     >>
>     >> ps:
>     >> *  The matchup information photo and url should store outside of mooosefs.
>     >> *  Huge file size limited in 2G.
>     >> *  Huge file maybe cause race condition.
>     >>
>     >> Is there anyone interested in this?
>     >> or better solution?
>     >>
>     >> -Ken
>     >>
>     >>
>     >> ------------------------------------------------------------------------------
>     >> Live Security Virtual Conference
>     >> Exclusive live event will cover all the ways today's security and
>     >> threat landscape has changed and how IT managers can respond. Discussions
>     >> will include endpoint security, mobile security and the latest in malware
>     >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>     >> _______________________________________________
>     >> moosefs-users mailing list
>     >> moo...@li... <mailto:moo...@li...>
>     >> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>     >
>     >
>     >
>     >
>     > --
>     >  - Davies
>
>
>
>
> -- 
>  - Davies
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>
>
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users