Re: [Moosefs-users] Solution of small file store

[Ken <ken...@gm...>]

>> store url and photo in leveldb, which use MooseFS as disks.
I understand this solution now.

Leveldb maybe write 'disk' 2 times in add/insert once,  impact of
Moosefs goal, the write performance is worse.

The most important, only one writer can access one leveldb file.
Single server is too limited.

-Ken


On Tue, Apr 24, 2012 at 3:11 PM, Davies Liu <dav...@gm...> wrote:
> On Tue, Apr 24, 2012 at 2:29 PM, Ken <ken...@gm...> wrote:
>>
>> On Tue, Apr 24, 2012 at 2:22 PM, Davies Liu <dav...@gm...> wrote:
>> > Maybe leveldb + MooseFS is better.
>> Is this your mean?
>>  leveldb store url
>>  moosefs store photo
>
>
> No, store url and photo in leveldb, which use MooseFS as disks.
> the performance may be not perfect, but enough.
> If not, TFS from taobao.com may be the better choice.
>
>> >
>> > On Tue, Apr 24, 2012 at 1:59 PM, Ken <ken...@gm...> wrote:
>> >>
>> >> We need to store tons of small files(photo), as noticed in faq, file
>> >> count is limited in moosefs, I think bundle small files to a huge file
>> >> maybe work.
>> >>
>> >> Write photos procedure like:
>> >>   allocate a huge file
>> >>   write head and photo content
>> >>   return (huge file, position, size)
>> >>   write another head and another photo
>> >>   return (huge file, position, size)
>> >>   ...
>> >>
>> >> Before read a photo, we should have enough information: huge file,
>> >> position and length, the reading procedure is expected normally.
>> >>
>> >> To read a photo, we should provide an URL, like
>> >> 'http://xxx.com/prefix/huge file/offset/size.jpg'
>> >>
>> >> And to be useful in WEB, build a fastcgi program for read/write access
>> >> to the huge file.
>> >>
>> >> ps:
>> >> *  The matchup information photo and url should store outside of
>> >> mooosefs.
>> >> *  Huge file size limited in 2G.
>> >> *  Huge file maybe cause race condition.
>> >>
>> >> Is there anyone interested in this?
>> >> or better solution?
>> >>
>> >> -Ken
>> >>
>> >>
>> >>
>> >> ------------------------------------------------------------------------------
>> >> Live Security Virtual Conference
>> >> Exclusive live event will cover all the ways today's security and
>> >> threat landscape has changed and how IT managers can respond.
>> >> Discussions
>> >> will include endpoint security, mobile security and the latest in
>> >> malware
>> >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> >> _______________________________________________
>> >> moosefs-users mailing list
>> >> moo...@li...
>> >> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>> >
>> >
>> >
>> >
>> > --
>> >  - Davies
>
>
>
>
> --
>  - Davies