Re: [Moosefs-users] ChunkServer in different Level

[Ken <ken...@gm...>]

We also use ucarp, but changed a little from Thomas version. The auto
switching never failed.

A friend of mine use DRBD and LVS. It also work fine, but I think it
is much smaller than the Douban's.

-Ken



On Fri, Jan 13, 2012 at 3:42 PM, Davies Liu <dav...@gm...> wrote:
> On Fri, Jan 13, 2012 at 3:25 PM, Ken <ken...@gm...> wrote:
>> I noticed the go-mfsclient in this mail-list before, we also write a
>> moose client in C++. ;-)
>> It work find when mfsmaster failover.
>
> It's the most interesting part, how do you failover mfsmaster ?
> I have tried the method with ucarp,  provided by Thomas S Hatch,
> It's seemed not stable enough, failed sometimes.
>
> Before a stable solution come up, we decide to do manually fail-over by ops,
> and do not deploy it in heavy online system.
>
>> We plan to build it as a preload dynamic library, and auto hook the file API.
>> I think it is high availability enough.
>>
>> Thanks
>>
>> Regards
>> -Ken
>>
>>
>>
>> On Fri, Jan 13, 2012 at 2:52 PM, Davies Liu <dav...@gm...> wrote:
>>> On Fri, Jan 13, 2012 at 2:40 PM, Ken <ken...@gm...> wrote:
>>>>>> It's not good ideal to use moosefs as storage for huge amount of small files
>>>> I agree. We combine some small files into one big file, and read the
>>>> small files with offset/length infomation.
>>>
>>> Is not safe to write to same file concurrently.
>>> We use this method to backup the original file user uploaded, with
>>> tar, when offline.
>>> Some times, some file will be broken.
>>>
>>> MFS is not good enough for online system, not high available, and some IO
>>> operations will be block when error in mfsmaster or mfschunkserver.
>>>
>>> So we serve some video files (>10M) in MFS this way:
>>>     Nginx -> nginx + FUSE -> MFS
>>> or
>>>     Nginx -> go-mfsclient [1] -> MFS
>>>
>>> If there something wrong with MFS, it will not block the first Nginx
>>> and the whole
>>> site will not be affected.
>>>
>>> Davies
>>>
>>> [1] github.com/davies/go-mfsclient
>>>
>>>> Thanks.
>>>>
>>>> Regards
>>>> -Ken
>>>>
>>>>
>>>>
>>>> On Fri, Jan 13, 2012 at 2:32 PM, Davies Liu <dav...@gm...> wrote:
>>>>> On Thu, Jan 12, 2012 at 5:28 PM, Ken <ken...@gm...> wrote:
>>>>>> hi, moosefs
>>>>>>
>>>>>> We plan to use moosefs as storage for huge amount photos uploaded by users.
>>>>>
>>>>> It's not good ideal to use moosefs as storage for huge amount of small files,
>>>>> because the mfsmaster will be the bottle neck when you have more than 100M
>>>>> files. At that time, the whole size of files may be 1T (10k per file),
>>>>> can be stored
>>>>> by one local disk.
>>>>>
>>>>> Huge amount small files need other solutions, just like TFS [1] from
>>>>> taobao.com,
>>>>> or beansdb [2] from douban.com.
>>>>>
>>>>> [1] http://code.taobao.org/p/tfs/src/
>>>>> [2] http://code.google.com/p/beansdb/
>>>>>
>>>>>> Because of read operations of new files are very more than old files,
>>>>>> maybe write new files to SSD is a choice.
>>>>>> For strict safe reason, we must backup content to an other data center.
>>>>>> And more features in maintain purpose are required.
>>>>>>
>>>>>> I don't think moosefs can work fine in these situation. We try to
>>>>>> implement these features several weeks ago. Till now, it's almost
>>>>>> done.
>>>>>>
>>>>>> Is there anyone interested in this?
>>>>>>
>>>>>> more detail:
>>>>>> # Add access_mode(none, read, write capability) to struct
>>>>>> matocserventry(matocserv.c). This value can be changed from
>>>>>> outside(maybe from the python cgi)
>>>>>> # mfschunkserver.cfg add 'LEVEL' config, if not, LEVEL=0 as normal.
>>>>>> ChunkServer report it to Master if need.
>>>>>> # Add uint32_t levelgoal into struct fsnode(filesystem.c).
>>>>>> # Add uint32_t levelgoal into sturct chunk(chunk.c).
>>>>>>  As seen, uint32_t levelgoal = uint8_t levelgoal[4], implied LEVEL
>>>>>> should be 1,2,3 or 4.
>>>>>>  [2,1,0,0] mean store 2 copies in level=1 ChunkServer, store 1 copy in
>>>>>> level=2 ChunkServer.
>>>>>> # In chunk_do_jobs(chunk.c), send replicated command to ChunkServer.
>>>>>> This policy should be very complicated in future.
>>>>>> # Also, we add read/write levelgoal support in mfstools.
>>>>>>
>>>>>> We plan to put these trivial change into github or somewhere else.
>>>>>>
>>>>>> It's a very incipient prototype. We appreciate any advice from the
>>>>>> develop team and other users.
>>>>>>
>>>>>> Regards
>>>>>> -Ken
>>>>>>
>>>>>> ------------------------------------------------------------------------------
>>>>>> RSA(R) Conference 2012
>>>>>> Mar 27 - Feb 2
>>>>>> Save $400 by Jan. 27
>>>>>> Register now!
>>>>>> http://p.sf.net/sfu/rsa-sfdev2dev2
>>>>>> _______________________________________________
>>>>>> moosefs-users mailing list
>>>>>> moo...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>  - Davies
>>>
>>>
>>>
>>> --
>>>  - Davies
>
>
>
> --
>  - Davies