Re: [Moosefs-users] Replication Priority?

[rxknhe <rx...@gm...>]

Hi there,
  If it can be given an option as 'on the fly tunable', that would be great.
Here is the reason for asking.

(1) We like conservative re-balancing approach of MooseFS, because in case
of hardware failure or new chunkserver addition, MooseFS won't go crazy and
start pounding disks and consuming I/O resource, thus causing  i/o
starvation for applications using MooseFS share. This works nicely most of
the time.

(2) However if we replace a dead chunkserver or mark area with '*' (i.e disk
area to be replaced) then it takes several days to re-balance. With goals=2,
this is a critical moment(when one chunkserver dies down), because if we
loose one more chunk server (or disk area), we are on the verge of loosing
data.
 In this case, if MooseFS can be told to speedup re-balancing and use more
resources if needed. Hence we can live with goals=2 (i.e more usable space
for data, instead of setting goals=3) comfortably.

Depending upon nature of application served via MooseFS, admin can decide if
willing to take risk with longer re-balance cycle and better i/o for
applications or choose re-balance speedup and sacrifice i/o for application
for a while.

Although to begin with, your approach of speeding up based on "high priority
(chunks with one copy with goal>1; orange color in CGI
monitor)" and such, is a very welcome enhancement.

regards
rxknhe



2011/6/8 Michal Borychowski <mic...@ge...>

> Hi!
>
> Thanks for pointing this out. Yes, it would be really important to make
> some
> priorities in the rebalancing process which we put on our to-do list.
>
> We have this plan for prioritizing:
> - high priority (chunks with one copy with goal>1; orange color in CGI
> monitor)
> - middle priority (chunks undergoal and overgoal; yellow and blue color)
> - low priority (normal and deleted chunks; green and grey color)
> - "special" priority (chunks with 0 copies; red color)
>
> What is your opinion about it?
>
>
> Kind regards
> Michał Borychowski
> MooseFS Support Manager
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> Gemius S.A.
> ul. Wołoska 7, 02-672 Warszawa
> Budynek MARS, klatka D
> Tel.: +4822 874-41-00
> Fax : +4822 874-41-01
>
>
>
>
> -----Original Message-----
> From: WK [mailto:wk...@bn...]
> Sent: Thursday, May 26, 2011 2:41 AM
> To: moo...@li...
> Subject: Re: [Moosefs-users] Replication Priority?
>
> On 5/23/2011 3:32 PM, W Kern wrote:
> > So now the CGI is showing 10,000+ chunks with a single copy (red), 2
> > million+ chunks are now orange (2 copies) and the system is happily
> > increasing the 'green' 3 valid copy column.
> >
> > The problem is that it seems to be concentrating on the orange (2 copy)
> > files and ignoring the 10,000+ red ones that are most at risk. In the
> > last hour we've seen a few 'red' chunks
> > disappear but the vast majority of activity is occuring in the orange (2
> > copy) column.
> >
> > Shouldn't the replication worry about the single copy files first?
> >
> > I also realize we could simply set the goal back to 2 let it finish that
> > up and THEN switch it to 3 but I'm curious as to what the community says.
> >
>
> Just a followup, after a day or so we still had "red" single copy
> chunks. Obviously the under goal routine doesn't look at how badly under
> goal a given chunk is.
>
> So we dropped the Goal back down to 2 and MFS immediately focused on the
> single copy chunks.
>
> The only problem observed was that shortly after dropping the goal back
> down to 2, the mount complained of connection issues and people were
> kicked out of their IMAP sessions.
> That condition returned to normal less than a minute later and no files
> were lost.
>
> Once the under goal of 2 was completed an hour or so later, we reset the
> Goal to 3 and in a few days we should be fully green. In the meantime,
> we have at least two copies and are not vulnerable to an additional
> failure.
>
> I would still suggest that the "under goal" routine might want to look
> first at those chunks that are more severely out of goal, then go back
> and fix the others. Assuming that doesn't impact overall performance.
>
> -bill
>
>
> ----------------------------------------------------------------------------
> --
> vRanger cuts backup time in half-while increasing security.
> With the market-leading solution for virtual backup and recovery,
> you get blazing-fast, flexible, and affordable data protection.
> Download your free trial now.
> http://p.sf.net/sfu/quest-d2dcopy1
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>
>
> ------------------------------------------------------------------------------
> EditLive Enterprise is the world's most technically advanced content
> authoring tool. Experience the power of Track Changes, Inline Image
> Editing and ensure content is compliant with Accessibility Checking.
> http://p.sf.net/sfu/ephox-dev2dev
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>