Re: [Moosefs-users] Write starvation

[Robert S. <rsa...@ne...>]

Based on some tests I think the limit in this case is the number of 
opens per minute. I think I need to understand what happens with an open 
before I can make guesses on what can be done to get the number higher.

But then it still does not quite explain the write starvation except if 
the number of pending reads are just so much higher than the number of 
pending writes that it seems to starve the writes. Maybe this will 
resolve itself as I add more chunk servers.

Some questions:

1. Is there a limit to the number of handles that client applications 
can open per mount, per chunk server, per disk?
2. What happens when an application does fopen() on a mount? Can 
somebody give a quick overview or do I have to read some code?

Robert

On 6/30/11 11:32 AM, Ricardo J. Barberis wrote:
> El Miércoles 29 Junio 2011, Robert escribió:
>> Yes, we use Centos, but installing and using the ktune package generally
>> resolves most of the performance issues and differences I have seen with
>> Ubuntu/Debian.
> Nice to know about ktune and thank you for bringing it up, I'll take a look a
> it.
>
>> I don't understand the comment on hitting metadata a lot? What is a lot?
> A lot = reading / (re)writing / ls -l'ing / stat'ing too often.
>
> If the client can't cache the metadata but uses it often, that means it has to
> query the master every time.
>
> Network latencies might also play a role in the performance degradation.
>
>> Why would it make a difference? All the metadata is in RAM anyway? The
>> biggest limit to speed seems to be the number of IOPS that you can get out
>> of your disks you have available to you. Looking up the metadata from RAM
>> should be several orders of magnitude faster than that.
> Yep, and you have plenty of RAM, so that shouldn't be an issue in your case.
>
>> The activity reported through the CGI interface on the master is around
>> 2,400 opens per minute average. Reads and writes are also around 2400 per
>> minute alternating with each other. mknod has some peaks around 2,800 per
>> minute but is generally much lower. Lookup's are around 8,000 per minute
>> and getattr is around 700 per minute. Chunk replication and deletion is
>> around 50 per minute. The other numbers are generally very low.
> Mmm, maybe 2 chunkservers are just too litle to handle that activity but I
> would also check the network latencies.
>
> I'm also not really confident about having master and cunkserver on the same
> server but I don't have any hard evidence to support my feelings ;)
>
>> Is there a guide/hints specific to MooseFS on what IO/Net/Process
>> parameters would be good to investigate for mfsmaster?
> I'd like to know that too!
>
> Cheers,