Re: [Moosefs-users] architecture question

[Robert S. <rsa...@ne...>]

The only time that many spindles will be remotely useful on 1 controller 
is if you are doing a large amount of small random read/write accesses. 
For anything else you are more likely to saturate the bus connecting the 
machine to the controller/expander.

Performance, as I see it, is divided into at least 2 types of performance:

1. Transfer speed
2. IOPS (I/O Operations per second)

For completely serial jobs you can get high transfer speeds with very 
few spindles. The more clients or the more random the load generated by 
your clients the more spindles becomes useful. But there is a tradeoff. 
If you are reading large numbers of large files then transfer speed may 
be your limiting factor. If you read large numbers of small files then 
IOPS may be.

We have 68 spindles distributed between 2 chunkservers and we are maxed 
out on IOPS on the one chunkserver. On the other chunkserver we are 
around 60%. But our load pattern is random reads of millions of small 
files ( < 600 kB ). The reason for the imbalance is historical and will 
go away in the next few weeks.

When you start dealing with large numbers of disks on a single 
controller/device you will be surprised what bottlenecks exist. The RAID 
controller rapidly becomes your bottleneck when it comes to transfer 
speed. In my experience it seems to be better to have either multiple 
controllers, multiple servers or multiple channels if you are talking to 
so many spindles. In the case of multiple controllers and channels you 
may still end up saturating the PCI bus or your network interfaces.

At some stage a storage company tried to convince us to buy an enclosure 
with 12 x SSD drives. I asked them to do some performance tests and the 
numbers were quite interesting. The SSD's could quickly saturate the 
bus. In transfer speed 12 x SSD was not any faster than 12 x SAS drives. 
The reason: the SAS bus got saturated in both cases. In IOPS the SSD 
drives were better until it got to a point where the bus were saturated 
and then adding more SSD drives did not provide any advantage.

Robert

On 7/21/11 2:46 PM, Elliot Finley wrote:
> Does putting more spindles on a chunkserver increase performance?  If
> so, to what point?
>
> I'm specifically thinking of a SAS2 controller/SAS2 expander with 127
> drives attached.  Does the chunkserver daemon have the capability to
> efficiently use that many spindles?
>
> TIA
> Elliot
>
> ------------------------------------------------------------------------------
> 5 Ways to Improve&  Secure Unified Communications
> Unified Communications promises greater efficiencies for business. UC can
> improve internal communications as well as offer faster, more efficient ways
> to interact with customers and streamline customer service. Learn more!
> http://www.accelacomm.com/jaw/sfnl/114/51426253/
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users