Menu
▾
▴
Re: [Moosefs-users] Solution of small file store
|
From: <ro...@mm...> - 2012-04-24 09:06:26
|
Ken wrote: > This way is too Linux. ha. How about growing? It should work on any decent *nix system. Growing? Option-1 How about using resize2fs. Option-2 [sci-fi-ON] You could create a PV group out of filesystems-in-a-file. Every time you need more space, you just add a new filesystem-in-a-file to the PV group [sci-fi-OFF] Option-3 Use sparse files. The file containing the file system could be a sparse file. This way, you can can specify, for example, a 2TB sparse file that will really only consume the actual data as a storage space. I don't know how MooseFS handles sparse files though... Option-4 Forget about filesystems (including MooseFS) and use a database for storing the photos. Use sharding or partioning to keep the database file down to a reasonable(whatever this may be) size and improve performance. Plenty of options for on-line replication of database data (so no single point of failure) -Stathis > > The business maybe same as Flickr, Facebook photo and DropBox, etc... > > If small files store resolved, moosefs will be used in more product > applications scenarios. > > Thanks > > -Ken > > > On Tue, Apr 24, 2012 at 4:10 PM, <ro...@mm...> wrote: >> If you're on linux, >> >> i'd try a filesystem-in-a-file which would be used to store the photo >> and >> this file itselfs stored in a MooseFS volume. >> >> For example something along these lines: >> >> dd if=/dev/zero of=FILENAME bs=1024 count=1 seek=$((2*1024*1024-1)) >> mkfs -t ext3 FILENAME >> mount -t ext3 -o loop FILENAME MOUNTPOINT >> >> where MOUNTPOINT is a MooseFS volume. >> >> -Stathis >> >> Wang Jian wrote: >>> TFS has strict and random URL scheme, so it's difficult to do URL based >>> tuning. >>> >>> δΊ 2012/4/24 15:11, Davies Liu ε ι : >>>> On Tue, Apr 24, 2012 at 2:29 PM, Ken <ken...@gm... >>>> <mailto:ken...@gm...>> wrote: >>>> >>>> On Tue, Apr 24, 2012 at 2:22 PM, Davies Liu <dav...@gm... >>>> <mailto:dav...@gm...>> wrote: >>>> > Maybe leveldb + MooseFS is better. >>>> Is this your mean? >>>> leveldb store url >>>> moosefs store photo >>>> >>>> No, store url and photo in leveldb, which use MooseFS as disks. >>>> the performance may be not perfect, but enough. >>>> If not, TFS from taobao.com <http://taobao.com> may be the better >>>> choice. >>>> >>>> > >>>> > On Tue, Apr 24, 2012 at 1:59 PM, Ken <ken...@gm... >>>> <mailto:ken...@gm...>> wrote: >>>> >> >>>> >> We need to store tons of small files(photo), as noticed in faq, >>>> file >>>> >> count is limited in moosefs, I think bundle small files to a >>>> huge >>>> file >>>> >> maybe work. >>>> >> >>>> >> Write photos procedure like: >>>> >> allocate a huge file >>>> >> write head and photo content >>>> >> return (huge file, position, size) >>>> >> write another head and another photo >>>> >> return (huge file, position, size) >>>> >> ... >>>> >> >>>> >> Before read a photo, we should have enough information: huge >>>> file, >>>> >> position and length, the reading procedure is expected >>>> normally. >>>> >> >>>> >> To read a photo, we should provide an URL, like >>>> >> 'http://xxx.com/prefix/huge file/offset/size.jpg' >>>> >> >>>> >> And to be useful in WEB, build a fastcgi program for read/write >>>> access >>>> >> to the huge file. >>>> >> >>>> >> ps: >>>> >> * The matchup information photo and url should store outside >>>> of >>>> mooosefs. >>>> >> * Huge file size limited in 2G. >>>> >> * Huge file maybe cause race condition. >>>> >> >>>> >> Is there anyone interested in this? >>>> >> or better solution? >>>> >> >>>> >> -Ken >>>> >> >>>> >> >>>> >> >>>> ------------------------------------------------------------------------------ >>>> >> Live Security Virtual Conference >>>> >> Exclusive live event will cover all the ways today's security >>>> and >>>> >> threat landscape has changed and how IT managers can respond. >>>> Discussions >>>> >> will include endpoint security, mobile security and the latest >>>> in >>>> malware >>>> >> threats. >>>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >>>> >> _______________________________________________ >>>> >> moosefs-users mailing list >>>> >> moo...@li... >>>> <mailto:moo...@li...> >>>> >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >>>> > >>>> > >>>> > >>>> > >>>> > -- >>>> > - Davies >>>> >>>> >>>> >>>> >>>> -- >>>> - Davies >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Live Security Virtual Conference >>>> Exclusive live event will cover all the ways today's security and >>>> threat landscape has changed and how IT managers can respond. >>>> Discussions >>>> will include endpoint security, mobile security and the latest in >>>> malware >>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >>>> >>>> >>>> _______________________________________________ >>>> moosefs-users mailing list >>>> moo...@li... >>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users >>> >>> ------------------------------------------------------------------------------ >>> Live Security Virtual Conference >>> Exclusive live event will cover all the ways today's security and >>> threat landscape has changed and how IT managers can respond. >>> Discussions >>> will include endpoint security, mobile security and the latest in >>> malware >>> threats. >>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________ >>> moosefs-users mailing list >>> moo...@li... >>> https://lists.sourceforge.net/lists/listinfo/moosefs-users >>> >> >> >> >> ------------------------------------------------------------------------------ >> Live Security Virtual Conference >> Exclusive live event will cover all the ways today's security and >> threat landscape has changed and how IT managers can respond. >> Discussions >> will include endpoint security, mobile security and the latest in >> malware >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >> _______________________________________________ >> moosefs-users mailing list >> moo...@li... >> https://lists.sourceforge.net/lists/listinfo/moosefs-users > |
