moosefs-users

[Moosefs-users] CGI curiosity

[Steve T. <sm...@cb...>]

mfs 1.6.20

I have marked a disk for removal with * in the mfshdd.cfg file. There are 
approximately 1.6 million chunks on this disk, and so far about 1.3 
million chunks have been replicated elsewhere. All files have goal = 2.

When viewing the CGI with Firefox on a Windows box, it shows 1.3 million 
in blue on the valid copies = 3 column of the goal = 2 line and nothing in 
the valid copies = 1 line. This number is increasing.

When viewing the CGI with Firefox on a Linux box, it shows 300,000 in 
orange on the valid copies = 1 column of the goal = 2 line, and nothing in 
the valid copies = 3 line. This number is decreasing.

The CGI is running on the mfsmaster in both cases. Why the difference?

BTW, this has taken 5 days so far. A little slow, methinks.

Steve
-- 
----------------------------------------------------------------------------
Steve Thompson, Cornell School of Chemical and Biomolecular Engineering
smt AT cbe DOT cornell DOT edu
   "186,282 miles per second: it's not just a good idea, it's the law"
----------------------------------------------------------------------------

Re: [Moosefs-users] CGI curiosity

[Steve W. <st...@pu...>]

On 04/09/2012 02:51 PM, Steve Thompson wrote:
> mfs 1.6.20
>
> I have marked a disk for removal with * in the mfshdd.cfg file. There are
> approximately 1.6 million chunks on this disk, and so far about 1.3
> million chunks have been replicated elsewhere. All files have goal = 2.
>
> When viewing the CGI with Firefox on a Windows box, it shows 1.3 million
> in blue on the valid copies = 3 column of the goal = 2 line and nothing in
> the valid copies = 1 line. This number is increasing.
>
> When viewing the CGI with Firefox on a Linux box, it shows 300,000 in
> orange on the valid copies = 1 column of the goal = 2 line, and nothing in
> the valid copies = 3 line. This number is decreasing.
>
> The CGI is running on the mfsmaster in both cases. Why the difference?
>
> BTW, this has taken 5 days so far. A little slow, methinks.
>
> Steve

Regarding the speed issue, have you modified the default 
CHUNKS_WRITE_REP_LIMIT and CHUNKS_READ_REP_LIMIT in mfsmaster.cfg?  
Following suggestions on the list, I have mine permanently set to:
     CHUNKS_WRITE_REP_LIMIT = 5
     CHUNKS_READ_REP_LIMIT = 15
instead of the default:
     CHUNKS_WRITE_REP_LIMIT = 1
     CHUNKS_READ_REP_LIMIT = 5

Steve

Re: [Moosefs-users] CGI curiosity

[Steve T. <sm...@cb...>]

On Mon, 9 Apr 2012, Steve Wilson wrote:

> Regarding the speed issue, have you modified the default
> CHUNKS_WRITE_REP_LIMIT and CHUNKS_READ_REP_LIMIT in mfsmaster.cfg?

No, for now I have chosen not to change the defaults, because we are 
having issues with performance for regular application I/O, and in any 
event I don't really want to have to restart the master while we are in 
production for an unknown effect. It's just replicating at a much slower 
rate than I expected.

Steve
-- 
----------------------------------------------------------------------------
Steve Thompson, Cornell School of Chemical and Biomolecular Engineering
smt AT cbe DOT cornell DOT edu
   "186,282 miles per second: it's not just a good idea, it's the law"
----------------------------------------------------------------------------

Re: [Moosefs-users] CGI curiosity

[Markus K. <mar...@tu...>]

On Monday 09 April 2012, Steve Wilson wrote:
> On 04/09/2012 02:51 PM, Steve Thompson wrote:
> > mfs 1.6.20
> >
> > I have marked a disk for removal with * in the mfshdd.cfg file. There are
> > approximately 1.6 million chunks on this disk, and so far about 1.3
> > million chunks have been replicated elsewhere. All files have goal = 2.
> >
> > When viewing the CGI with Firefox on a Windows box, it shows 1.3 million
> > in blue on the valid copies = 3 column of the goal = 2 line and nothing
> > in the valid copies = 1 line. This number is increasing.
> >
> > When viewing the CGI with Firefox on a Linux box, it shows 300,000 in
> > orange on the valid copies = 1 column of the goal = 2 line, and nothing
> > in the valid copies = 3 line. This number is decreasing.
> >
> > The CGI is running on the mfsmaster in both cases. Why the difference?
> >
> > BTW, this has taken 5 days so far. A little slow, methinks.
> >
> > Steve
>
> Regarding the speed issue, have you modified the default
> CHUNKS_WRITE_REP_LIMIT and CHUNKS_READ_REP_LIMIT in mfsmaster.cfg?
> Following suggestions on the list, I have mine permanently set to:
>      CHUNKS_WRITE_REP_LIMIT = 5
>      CHUNKS_READ_REP_LIMIT = 15
> instead of the default:
>      CHUNKS_WRITE_REP_LIMIT = 1
>      CHUNKS_READ_REP_LIMIT = 5


I tried this settings today and it worked very good. Thanks for sharing your 
configuration.


For me today a similar question followed up:
How to find out if all chunks are migrated from a chunk server?

At the moment I can see:

chunk server marked for removal: 
chunks=4545
All chunks state matrix 'regular':
goal=2: 1=1, 2=8028
All chunks state matrix 'all':
goal=2: 1=1, 2=3483, 3=4545

So I know that I can remove the chunk server now because chunks state 
matrix 'all' says I have a overgoal of 4545 which is exactly the same number 
of chunks on my chunk server. 

At the moment I only run some test with goal=2. In production we will have 
different goals for different types of data. My plan is to use all desktop 
and laboratory hosts in our environment as a chunk server. Ones a year we 
need to reboot our laboratory hosts into windows for about two weeks. Which 
means I have to mark all partitions on this chunk servers for removal at the 
same time. Which I guess will make it hard to say which one is ready to 
reboot. At least not before all chunks of all chunk server are migrated.

It would be very nice to see the status at the disk status table. At the 
moment I only can see status 'ok' or 'marked for removal'.
Nice would be see something like: 'marked for removal, migration in progress' 
and 'marked for removal, migration finished'


Markus
-- 
Markus Köberl
Graz University of Technology
Signal Processing and Speech Communication Laboratory
E-mail: mar...@tu...

Re: [Moosefs-users] CGI curiosity

[Michał B. <mic...@co...>]

For the moment you need to see the 'regular' view mode and check if none of the chunks is in undergoal. If not you are ready to switch the machines off. 

And from 1.6.26 on it would be possible to reload config files on the fly (including rep speeds) without stopping the master process.


Kind regards
Michał Borychowski 
MooseFS Support Manager


-----Original Message-----
From: Markus Köberl [mailto:mar...@tu...] 
Sent: Tuesday, April 10, 2012 11:49 AM
To: moo...@li...
Subject: Re: [Moosefs-users] CGI curiosity

On Monday 09 April 2012, Steve Wilson wrote:
> On 04/09/2012 02:51 PM, Steve Thompson wrote:
> > mfs 1.6.20
> >
> > I have marked a disk for removal with * in the mfshdd.cfg file. 
> > There are approximately 1.6 million chunks on this disk, and so far 
> > about 1.3 million chunks have been replicated elsewhere. All files have goal = 2.
> >
> > When viewing the CGI with Firefox on a Windows box, it shows 1.3 
> > million in blue on the valid copies = 3 column of the goal = 2 line 
> > and nothing in the valid copies = 1 line. This number is increasing.
> >
> > When viewing the CGI with Firefox on a Linux box, it shows 300,000 
> > in orange on the valid copies = 1 column of the goal = 2 line, and 
> > nothing in the valid copies = 3 line. This number is decreasing.
> >
> > The CGI is running on the mfsmaster in both cases. Why the difference?
> >
> > BTW, this has taken 5 days so far. A little slow, methinks.
> >
> > Steve
>
> Regarding the speed issue, have you modified the default 
> CHUNKS_WRITE_REP_LIMIT and CHUNKS_READ_REP_LIMIT in mfsmaster.cfg?
> Following suggestions on the list, I have mine permanently set to:
>      CHUNKS_WRITE_REP_LIMIT = 5
>      CHUNKS_READ_REP_LIMIT = 15
> instead of the default:
>      CHUNKS_WRITE_REP_LIMIT = 1
>      CHUNKS_READ_REP_LIMIT = 5


I tried this settings today and it worked very good. Thanks for sharing your configuration.


For me today a similar question followed up:
How to find out if all chunks are migrated from a chunk server?

At the moment I can see:

chunk server marked for removal: 
chunks=4545
All chunks state matrix 'regular':
goal=2: 1=1, 2=8028
All chunks state matrix 'all':
goal=2: 1=1, 2=3483, 3=4545

So I know that I can remove the chunk server now because chunks state 
matrix 'all' says I have a overgoal of 4545 which is exactly the same number 
of chunks on my chunk server. 

At the moment I only run some test with goal=2. In production we will have 
different goals for different types of data. My plan is to use all desktop 
and laboratory hosts in our environment as a chunk server. Ones a year we 
need to reboot our laboratory hosts into windows for about two weeks. Which 
means I have to mark all partitions on this chunk servers for removal at the 
same time. Which I guess will make it hard to say which one is ready to 
reboot. At least not before all chunks of all chunk server are migrated.

It would be very nice to see the status at the disk status table. At the 
moment I only can see status 'ok' or 'marked for removal'.
Nice would be see something like: 'marked for removal, migration in progress' 
and 'marked for removal, migration finished'


Markus
-- 
Markus Köberl
Graz University of Technology
Signal Processing and Speech Communication Laboratory
E-mail: mar...@tu...

------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second 
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] CGI curiosity

[Michał B. <mic...@co...>]

Hi Steve!

Please observe that there are two "view modes" in the Info tab:
- 'all' - All chunks state matrix (counts 'regular' hdd space and 'marked
for removal' hdd space)
- 'regular' - Regular chunks state matrix (counts only 'regular' hdd space)

So in case you are in the 'all' mode you'd have lots of chunks in overgoal
(the disk marked for removal still keeps the chunks - they are copied not
deleted from this drive) and when in 'regular' you'd see lots of chunks in
undergoal - still waiting to be copied to the target goal.


Kind regards
Michał Borychowski 
MooseFS Support Manager



-----Original Message-----
From: Steve Thompson [mailto:sm...@cb...] 
Sent: Monday, April 09, 2012 8:52 PM
To: moo...@li...
Subject: [Moosefs-users] CGI curiosity

mfs 1.6.20

I have marked a disk for removal with * in the mfshdd.cfg file. There are
approximately 1.6 million chunks on this disk, and so far about 1.3 million
chunks have been replicated elsewhere. All files have goal = 2.

When viewing the CGI with Firefox on a Windows box, it shows 1.3 million in
blue on the valid copies = 3 column of the goal = 2 line and nothing in the
valid copies = 1 line. This number is increasing.

When viewing the CGI with Firefox on a Linux box, it shows 300,000 in orange
on the valid copies = 1 column of the goal = 2 line, and nothing in the
valid copies = 3 line. This number is decreasing.

The CGI is running on the mfsmaster in both cases. Why the difference?

BTW, this has taken 5 days so far. A little slow, methinks.

Steve
--
----------------------------------------------------------------------------
Steve Thompson, Cornell School of Chemical and Biomolecular Engineering smt
AT cbe DOT cornell DOT edu
   "186,282 miles per second: it's not just a good idea, it's the law"
----------------------------------------------------------------------------

----------------------------------------------------------------------------
--
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] CGI curiosity

[Steve T. <sm...@cb...>]

On Tue, 17 Apr 2012, Micha? Borychowski wrote:

> Please observe that there are two "view modes" in the Info tab: - 'all' 
> - All chunks state matrix (counts 'regular' hdd space and 'marked for 
> removal' hdd space) - 'regular' - Regular chunks state matrix (counts 
> only 'regular' hdd space)

Yes, you are of course correct. I screwed up; how embarrassing!

Steve