Re: [Moosefs-users] MFS Access very slow during chunk deletions

[Ólafur Ó. <osv...@ne...>]

Hi,
I agree it seems to have no effect on anything else than mfsexports.cfg reloading, we are running Xen VM images on the MFS partition and I'm not sure how they will handle the mfsmaster restarting, will have to setup a test enviroment and test it.

When reading the man page for mfsmaster.cfg I see the comments for CHUNKS_LOOP_TIME and CHUNKS_DEL_LIMIT and my understanding is that with the default values the maximum number of chunks to delete in one loop (300 sek) is 100, it does not say if that is pr. chunkserver or for the whole system, but each server here was around and over 5000 chunk deletions pr. minute and with 10 servers thats over 50k chunk deletions pr. minute for the whole system.

       CHUNKS_LOOP_TIME
              Chunks loop frequency in seconds (default is 300)

       CHUNKS_DEL_LIMIT
              Maximum number of chunks to delete in one loop (default is 100)

       CHUNKS_WRITE_REP_LIMIT
              Maximum  number of chunks to replicate to one chunkserver in one
              loop (default is 1)

       CHUNKS_READ_REP_LIMIT
              Maximum number of chunks to replicate from  one  chunkserver  in
              one loop (default is 5)

Deleteing 100 and only replicating 1, thats quite a difference.

Stil puzzled by this.

/Oli

On 28.6.2011, at 10:54, rxknhe wrote:

I am not sure if SIGHUP is going to work. It works for few things like re-reading mfsexports.cfg file, but we also noticed that it won't work when changing parameters in mfsmaster.cfg and one has to restart master server process.
  Although we found that restarting master server process is normally safe, but I agree that better way is to handle via SIGHUP as it is a little scary thought to restart master in a production system.



2011/6/28 Ólafur Ósvaldsson <osv...@ne...<mailto:osv...@ne...>>
Hi
When it was at its peak I tried changing that value down to 10 and ultimately to 1 and sending the mfsmaster a SIGHUP, but nothing changed, for us restarting the master is not an option since all the systems were still getting service although very slow.

/Oli

On 27.6.2011, at 21:52, rxknhe wrote:

On master may be try tweaking option in mfsmaster.cfg file and restart master server process.
# CHUNKS_DEL_LIMIT =100
un-comment and try lower values,



2011/6/27 Ólafur Ósvaldsson <osv...@ne...<mailto:osv...@ne...>>
Hi,
Our system had just over 4.5 million chunks untill one week ago when a good deal of the data was deleted (on purpose).

Today the trash time expired and it seems that our master is deleting all the unused chunks which is quite normal except each server of 10 is doing up to 5.5k chunk deletions pr. minute and all systems accessing the MFS partitions are very slow and that started at the same time as the chunk deletions.

Is there any way to tune MFS so that chunk deletions don't have such an impact on the system, can we somehow control how many chunks are deleted at once in max rate pr. minute or maybe have it somehow linked to chunkserver/master load?

The network links are not saturated and none of the servers seem to be loaded at all?

The disk io on the master even went down when this started, as did the CPU usage of the server, but the memory usage has gone up about 30%

Could it be that the master process is not handling all this at once?

/Oli

--
Ólafur Osvaldsson
System Administrator
Nethonnun ehf.
e-mail: osv...@ne...<mailto:osv...@ne...>
phone: +354 517 3400<tel:%2B354%20517%203400>


------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
moosefs-users mailing list
moo...@li...<mailto:moo...@li...>
https://lists.sourceforge.net/lists/listinfo/moosefs-users


--
Ólafur Osvaldsson
System Administrator
Nethonnun ehf.
e-mail: osv...@ne...<mailto:osv...@ne...>
phone: +354 517 3400<tel:%2B354%20517%203400>



--
Ólafur Osvaldsson
System Administrator
Nethonnun ehf.
e-mail: osv...@ne...<mailto:osv...@ne...>
phone: +354 517 3400