Re: [MooseFS-Users] Performance suggestions for millions of small files

[Marco M. <mar...@gm...>]

Steve,

I don't have a solution for this problem.
Just out of curiosity:

-- what is your large file create speed on this cluster
compared to your other clusters?
(i.e how long does it take to create a single 10GB file
on this one compared to others ?)

-- You said you have a mix of 3.0.97 and 3.0.101,
are the versions of MooseFS uniform on this "slow cluster" ?

-- Marco

On 10/18/18 8:35 PM, Wilson, Steven M wrote:
> We have a mix of 3.0.97 and 3.0.101.
> 
> Steve
> 
> ________________________________________
> From: Marco Milano <mar...@gm...>
> Sent: Thursday, October 18, 2018 5:43 PM
> To: moo...@li...
> Subject: Re: [MooseFS-Users] Performance suggestions for millions of small files
> 
> Steve,
> 
> What is the version of the MooseFS ?
> 
> -- Marco
> 
> On 10/18/18 4:47 PM, Wilson, Steven M wrote:
>> Hi,
>>
>>
>> We have ten different MooseFS installations in our research group and
>> one, in particular, is struggling with poor I/O performance.  This
>> installation currently has 315 million files occupying 170TB of disk
>> space (goal = 2).  If anyone else has a similar installation, I would
>> like to hear what you have done to maintain performance at a reasonable
>> level.
>>
>>
>> Here are some metrics to give a basic idea of the performance
>> characteristics.  I'll include in parentheses the range of measurements
>> from other MFS installations with far fewer files for comparison.
>>
>>       * tar xf linux-4.9-rc3.tar: 1185 secs (220 - 296 secs)
>>
>>       * smallfile test, create MB/s: 0.8 (2.3 - 4.8)  <== Ouch!
>>
>>       * smallfile test, read MB/s: 10.7 (12.8 - 15.4)
>>
>>       * smallfile test, append MB/s: 6.1 (3.0 - 7.7)
>>
>>
>> It looks file creation is where I'm losing most of my performance
>> compared to the other installations.  My master server has a Xeon
>> E5-1630v3 3.7GHz CPU with 256GB of DDR4 2133MHz memory.
>>
>>
>> I tried several mfsmount options but the only one that showed any
>> significant improvement was the mfsfsyncmintime option
>> ("mfsfsyncmintime=5").  As to be expected, the improvement gained was
>> during the write/append operation.  Here are the results using the same
>> tests as above:
>>
>>       * tar xf linux-4.9-rc3.tar: 683 secs
>>       * smallfile test, create MB/s: 1.2
>>       * smallfile test, read MB/s: 11.7
>>       * smallfile test, append MB/s: 11.4  <== Dramatic improvement over
>> 6.1 MB/s
>>
>>
>> The smallfile benchmark test I used is ​from
>> https://github.com/distributed-system-analysis/smallfile.
>>
>>
>> Thanks for any suggestions you might have!
>>
>>
>> Regards,
>>
>> Steve
>>
>>
>>
>> _________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>
> 
> 
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>