Re: [Moosefs-users] mfsmaster performance and hardware

[Robert S. <rsa...@ne...>]

Hi Elliot,

There is nothing in the code to change the priority.

Taking virtually all other load from the chunk and master servers seems 
to have improved this significantly. I still see timeouts from mfsmount, 
but not enough to  be problematic.

To try and optimize the performance I am experimenting with accessing 
the data using different APIs and block sizes but this has been 
inconclusive. I have tried the effect of posix_fadvise(), sendfile() and 
different sized buffers for read(). I still want to try mmap(). 
Sendfile() did seem to be slightly slower than read().

Robert

On 8/24/11 11:05 AM, Elliot Finley wrote:
> On Tue, Aug 9, 2011 at 6:46 PM, Robert Sandilands<rsa...@ne...>  wrote:
>> Increasing the swap space fixed the fork() issue. It seems that you have to
>> ensure that memory available is always double the memory needed by
>> mfsmaster. None of the swap space was used over the last 24 hours.
>>
>> This did solve the extreme comb-like behavior of mfsmaster. It still does
>> not resolve its sensitivity to load on the server. I am still seeing
>> timeouts on the chunkservers and mounts on the hour due to the high CPU and
>> I/O load when the meta data is dumped to disk. It did however decrease
>> significantly.
> Here is another thought on this...
>
> The process is niced to -19 (very high priority) so that it has good
> performance.  It forks once per hour to write out the metadata.  I
> haven't checked the code for this, but is the forked process lowering
> it's priority so it doesn't compete with the original process?
>
> If it's not, it should be an easy code change to lower the priority in
> the child process (metadata writer) so that it doesn't compete with
> the original process at the same priority.
>
> If you check into this, I'm sure the list would appreciate an update. :)
>
> Elliot