Re: [MooseFS-Users] Chunk servers not conforming to CS_HEAVY_LOAD parameters

[Aleksander W. <ale...@mo...>]

Hi,
Basically to slow down internal re-balancing, you have to modify last
two parameters in CHUNKS_(READ/WRITE)_REP_LIMIT in mfsmaster.cfg file.
On chunkserver: HDD_REBALANCE_UTILIZATION and WOREKRS_MAX.

By the way. We will try to add some extra parameter to avoid such a
situation in the feature.

Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com <moosefs.com>

On 17.05.2017 20:02, David Myer wrote:
> Hi MooseFS users,
>
> I'm trying to reduce the impact of chunk server rebalancing after new
> partitions were added. I have set some very conservative master and
> chunkserver cfg parameters but I can only seem to get the linux server
> loads down to around 2.
>
> http://i.imgur.com/cnomEaU.png
>
> Here are some params I have set to try to slow the master down:
>
> ACCEPTABLE_PERCENTAGE_DIFFERENCE = 10.0
> NICE_LEVEL = 19
> REPLICATIONS_DELAY_INIT = 600
> CHUNKS_LOOP_MAX_CPS = 1
> CHUNKS_LOOP_MIN_TIME = 3600
> CHUNKS_SOFT_DEL_LIMIT = 1
> CHUNKS_HARD_DEL_LIMIT = 2
> CHUNKS_WRITE_REP_LIMIT = 2,1,1,1
> CHUNKS_READ_REP_LIMIT = 10,1,1,1
> ATIME_MODE = 4
> CS_HEAVY_LOAD_THRESHOLD = 5
> CS_HEAVY_LOAD_RATIO_THRESHOLD = 1.0
>
> Here are some params I have set to try to slow the chunkservers down:
>
> HDD_REBALANCE_UTILIZATION = 1
> NICE_LEVEL = 19
> WORKERS_MAX = 1
> WORKERS_MAX_IDLE = 1
>
> The parameter changes have definitely helped, reducing the linux
> server loads from 30 down to ~2. It is hard to say which parameters
> have the most effect or if some parameters are even working at all
> (such as CS_HEAVY_LOAD_*, which doesn't seem to be putting the chunk
> servers into grace mode).
>
> From watching IO via iotop I think the main problem is how much IO
> mfschunkserver still uses even when greatly restricted by the
> aforementioned parameters. It would be great to throttle its
> read/write speeds somehow.
>
> Any general ideas on mitigating the effect of big chunkserver work
> while in production and under heavy use by mfsmount? I have also moved
> all other services off the chunkserver machines. Could mfsmount be
> turned off on the chunkservers and only used on the master? 
>
> Any help would be greatly appreciated.
>
> Thank you,
> Dave
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users