[MooseFS-Users] Chunk servers not conforming to CS_HEAVY_LOAD parameters

[David M. <dav...@pr...>]

Hi MooseFS users,

I'm trying to reduce the impact of chunk server rebalancing after new partitions were added. I have set some very conservative master and chunkserver cfg parameters but I can only seem to get the linux server loads down to around 2.

http://i.imgur.com/cnomEaU.png

Here are some params I have set to try to slow the master down:

ACCEPTABLE_PERCENTAGE_DIFFERENCE = 10.0
NICE_LEVEL = 19
REPLICATIONS_DELAY_INIT = 600
CHUNKS_LOOP_MAX_CPS = 1
CHUNKS_LOOP_MIN_TIME = 3600
CHUNKS_SOFT_DEL_LIMIT = 1
CHUNKS_HARD_DEL_LIMIT = 2
CHUNKS_WRITE_REP_LIMIT = 2,1,1,1
CHUNKS_READ_REP_LIMIT = 10,1,1,1
ATIME_MODE = 4
CS_HEAVY_LOAD_THRESHOLD = 5
CS_HEAVY_LOAD_RATIO_THRESHOLD = 1.0

Here are some params I have set to try to slow the chunkservers down:

HDD_REBALANCE_UTILIZATION = 1
NICE_LEVEL = 19
WORKERS_MAX = 1
WORKERS_MAX_IDLE = 1

The parameter changes have definitely helped, reducing the linux server loads from 30 down to ~2. It is hard to say which parameters have the most effect or if some parameters are even working at all (such as CS_HEAVY_LOAD_*, which doesn't seem to be putting the chunk servers into grace mode).

From watching IO via iotop I think the main problem is how much IO mfschunkserver still uses even when greatly restricted by the aforementioned parameters. It would be great to throttle its read/write speeds somehow.

Any general ideas on mitigating the effect of big chunkserver work while in production and under heavy use by mfsmount? I have also moved all other services off the chunkserver machines. Could mfsmount be turned off on the chunkservers and only used on the master?

Any help would be greatly appreciated.

Thank you,
Dave