Re: [Moosefs-users] ChunkServer in different Level

[Davies L. <dav...@gm...>]

On Thu, Jan 12, 2012 at 5:28 PM, Ken <ken...@gm...> wrote:
> hi, moosefs
>
> We plan to use moosefs as storage for huge amount photos uploaded by users.

It's not good ideal to use moosefs as storage for huge amount of small files,
because the mfsmaster will be the bottle neck when you have more than 100M
files. At that time, the whole size of files may be 1T (10k per file),
can be stored
by one local disk.

Huge amount small files need other solutions, just like TFS [1] from
taobao.com,
or beansdb [2] from douban.com.

[1] http://code.taobao.org/p/tfs/src/
[2] http://code.google.com/p/beansdb/

> Because of read operations of new files are very more than old files,
> maybe write new files to SSD is a choice.
> For strict safe reason, we must backup content to an other data center.
> And more features in maintain purpose are required.
>
> I don't think moosefs can work fine in these situation. We try to
> implement these features several weeks ago. Till now, it's almost
> done.
>
> Is there anyone interested in this?
>
> more detail:
> # Add access_mode(none, read, write capability) to struct
> matocserventry(matocserv.c). This value can be changed from
> outside(maybe from the python cgi)
> # mfschunkserver.cfg add 'LEVEL' config, if not, LEVEL=0 as normal.
> ChunkServer report it to Master if need.
> # Add uint32_t levelgoal into struct fsnode(filesystem.c).
> # Add uint32_t levelgoal into sturct chunk(chunk.c).
>  As seen, uint32_t levelgoal = uint8_t levelgoal[4], implied LEVEL
> should be 1,2,3 or 4.
>  [2,1,0,0] mean store 2 copies in level=1 ChunkServer, store 1 copy in
> level=2 ChunkServer.
> # In chunk_do_jobs(chunk.c), send replicated command to ChunkServer.
> This policy should be very complicated in future.
> # Also, we add read/write levelgoal support in mfstools.
>
> We plan to put these trivial change into github or somewhere else.
>
> It's a very incipient prototype. We appreciate any advice from the
> develop team and other users.
>
> Regards
> -Ken
>
> ------------------------------------------------------------------------------
> RSA(R) Conference 2012
> Mar 27 - Feb 2
> Save $400 by Jan. 27
> Register now!
> http://p.sf.net/sfu/rsa-sfdev2dev2
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users



-- 
 - Davies