Re: [Moosefs-users] mfsmaster performance and hardware

[Robert S. <rsa...@ne...>]

Hi Michal,

The paper is an interesting read. I think that the growth of technology 
is however making it impractical. Both Intel and AMD are working on 20+ 
core CPU's and there are some non-x86 64-core systems available today. 
These systems focus on slower cores but many of them. For systems to be 
able to scale in the future they need to be able to effectively use the 
hardware that will be available then. Unfortunately (according to the 
paper) the paradigm that is winning on both the hardware and software 
fronts are threads. Does threads have problems? Yes, but it may be 
slightly less problematic than pointers ;-)

With a 2 GHz Xeon I am seeing scaling problems when you approach 94 
million files. I had another crash this weekend and had to increase 
timeouts yet again. At this stage the master is unresponsive for at 
least 5 minutes every hour. The graphs in the CGI look like a comb with 
0 activity on the hour every hour for about 5 minutes. That is except 
for CPU usage on the master which spikes to 100% for the same period. We 
did see an increase in performance and stability when we moved some 
tasks from the master server to other machines but at this stage we 
can't move more tasks off the master without buying more hardware. 
During the time of 0 activity we see read and write timeouts and the 
filesystem is completely unresponsive to users.

I am convinced that part of the scalability issue is related to the fact 
that everything is single threaded and that any single task that can 
take a long time has the potential to cause problems affecting 
scalability and stability.

We still have another approximately 16 TB to move to MooseFS so I do 
expect us to easily pass the 100 million file mark. As we are 
deduplicating the files as we move them it is hard to predict how much 
space/files it will be when we are done. We are also adding more than 4 
million files and 2 TB per month (before deduplication).

Robert

On 8/8/11 2:52 AM, Michal Borychowski wrote:
> Hi Robert
>
> I wrote shortly about multithreading in mfsmaster here:
> http://sourceforge.net/mailarchive/message.php?msg_id=26680860
>
> So no, it is not on our roadmap.
>
> And yes, performance of MooseFS is dependent on the performance of
> mfsmaster. CPU load depends on amount of operations in the filesystem. In
> our environment the master server consumes about 30% of CPU (ca. 1500
> operations per second).  HDD doesn't have to be huge, but still it should be
> quick for dumps of metadata and continuous saving of changelogs.
>
> Rough estimate how much RAM you need is here:
> http://www.moosefs.org/moosefs-faq.html#sort
>
> And to be honest metalogger machines should be as good as the master itself
> because in case of emergency metalogger should be switched to the role of
> the master.
>
>
> Kind regards
> -Michal
>
>
> -----Original Message-----
> From: Robert Sandilands [mailto:rsa...@ne...]
> Sent: Thursday, August 04, 2011 2:42 AM
> To: moo...@li...
> Subject: [Moosefs-users] mfsmaster performance and hardware
>
> We have been spending a lot of time trying to get MooseFS stable and
> optimized.
>
> Something I have noticed is that mfsmaster seems to be a bottleneck in
> our setup. What I also noticed is that mfsmaster is single threaded.
>   From reading the source code it seems to use a very interesting polling
> loop to handle all communications and actions.
>
> So a question: Is there anything on the roadmap to make mfsmaster
> multithreaded?
>
> It also seems that the performance of MooseFS is very dependent on the
> performance of mfsmaster. If the machine running mfsmaster is slow or is
> busy then it can slow everything down significantly or even cause
> instability in the file system.
>
> This also implies that if you want to buy a dedicated machine for
> mfsmaster that you have to buy the fastest possible CPU and as much RAM
> as you need. Local disk space and multiple CPUs and cores are not
> important. Is this correct? What would the recommendation be for an
> optimal machine to run mfsmaster?
>
> Robert
>
>
> ----------------------------------------------------------------------------
> --
> BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA
> The must-attend event for mobile developers. Connect with experts.
> Get tools for creating Super Apps. See the latest technologies.
> Sessions, hands-on labs, demos&  much more. Register early&  save!
> http://p.sf.net/sfu/rim-blackberry-1
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>