moosefs-users

[Moosefs-users] MooseFS performance benchmarking.

[<leo...@ar...>]

Greetings!
Resently I've started performance benchmarking for MooseFS,
_____________ __ _ _
Chunkservers:
CPU: Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz
RAM: 12G
HDD: storage used for mfs chunks is raid0 of two WD 7200rpm caviar 
black disks.
OS: OpenSuSE 11.3
Chunk servers number: 5

-------------------------------
Master:

CPU: Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz
OS: OpenSuSE 11.3
RAM: 12G

--------------------------------
Client:
One of chunkservers.

---------------------------------------------------
LAN:
1Gbit/s LAN.

____________ _ _ __
Problem;
I've experimented alot with bonnie, fio, iozone and others in testing 
other storages... We have people working with the source code here, so 
we need good random I/O for small files with moderate blocksize from 8k 
to 128k... Comparative testing of other storage solutions involving ZFS, 
different hardware raids etc showed that simple taring and untaring 
Linux kernel sources shows how good can the storage be for that kind of 
work... and it always correlates with more advanced fio and iozone 
tests.

So, simple untaring Linux kernel sources takes about 7 seconds on 
chunkservers storage bypassing moosefs... but when I untar it to mounted 
moosefs it takes more than 230 seconds.
Goal is set to 1. CPU load is OK on all the servers, RAM is sufficient. 
Network is not overloaded and I can untar this file in about7 secs to 
our nfs-mounted NAS....
I even turned on files caching on the client.
And this is all is very strange... maybe fuse is the bottleneck?...

Re: [Moosefs-users] MooseFS performance benchmarking.

[Flow J. <fl...@gm...>]

We are experiencing the same issue when dealing with a lot of small files.

One observation is, when scp a large file through Gigabyte link onto our 
NFS server (Served by a VERY old Sun Raid5 Disk Array), there seems to 
have a large write cache to allow the speed reported at the first 
several seconds greater than the writing limit of hard drives. But when 
copying the same file onto MFS (Served by SAS 300G 10k disk on Dell PE 
1950), the speed never reaches the hard drive limit. And our NFS server 
do perform better than MFS when copying large amount of source code files.

I'm not sure if this issue is only related to our SW/HW configuration. 
But seems the write caches works better with NFS than MFS. Does any one 
know how does write cache of MFS work?

Thanks
Flow


On 10/17/2011 10:46 PM, leo...@ar... wrote:
> Greetings!
> Resently I've started performance benchmarking for MooseFS,
> _____________ __ _ _
> Chunkservers:
> CPU: Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz
> RAM: 12G
> HDD: storage used for mfs chunks is raid0 of two WD 7200rpm caviar
> black disks.
> OS: OpenSuSE 11.3
> Chunk servers number: 5
>
> -------------------------------
> Master:
>
> CPU: Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz
> OS: OpenSuSE 11.3
> RAM: 12G
>
> --------------------------------
> Client:
> One of chunkservers.
>
> ---------------------------------------------------
> LAN:
> 1Gbit/s LAN.
>
> ____________ _ _ __
> Problem;
> I've experimented alot with bonnie, fio, iozone and others in testing
> other storages... We have people working with the source code here, so
> we need good random I/O for small files with moderate blocksize from 8k
> to 128k... Comparative testing of other storage solutions involving ZFS,
> different hardware raids etc showed that simple taring and untaring
> Linux kernel sources shows how good can the storage be for that kind of
> work... and it always correlates with more advanced fio and iozone
> tests.
>
> So, simple untaring Linux kernel sources takes about 7 seconds on
> chunkservers storage bypassing moosefs... but when I untar it to mounted
> moosefs it takes more than 230 seconds.
> Goal is set to 1. CPU load is OK on all the servers, RAM is sufficient.
> Network is not overloaded and I can untar this file in about7 secs to
> our nfs-mounted NAS....
> I even turned on files caching on the client.
> And this is all is very strange... maybe fuse is the bottleneck?...
>
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2d-oct
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] MooseFS performance benchmarking.

[<leo...@ar...>]

Something is very wrong with MooseFS regarding writing 'lots of small 
files'.
You can even see it 'visually': if you tar xzvf linux*.tgz the names of 
the files appear much slooowly than this is in case of NFS or local 
storage...
This is to say that we use quite a powerfull HW. And we have NO load on 
MFS at the moment. The only thing done at the moment is the untaring the 
linux kernel sources.

I'd sincerelly appresiate any comments from the developers of the 
MooseFS, the project I had big hopes for )
Do they observe the same in their environment?..


P.S.
sometimes it takes less than 100 seconds to untar, but still far beyond 
NFS, where I get it in 4-7 seconds or so.




On Fri, 21 Oct 2011 00:00:06 +0800, Flow Jiang wrote:
> We are experiencing the same issue when dealing with a lot of small 
> files.
>
> One observation is, when scp a large file through Gigabyte link onto
> our NFS server (Served by a VERY old Sun Raid5 Disk Array), there
> seems to have a large write cache to allow the speed reported at the
> first several seconds greater than the writing limit of hard drives.
> But when copying the same file onto MFS (Served by SAS 300G 10k disk
> on Dell PE 1950), the speed never reaches the hard drive limit. And
> our NFS server do perform better than MFS when copying large amount 
> of
> source code files.
>
> I'm not sure if this issue is only related to our SW/HW
> configuration. But seems the write caches works better with NFS than
> MFS. Does any one know how does write cache of MFS work?
>
> Thanks
> Flow
>
>
> On 10/17/2011 10:46 PM, leo...@ar... wrote:
>> Greetings!
>> Resently I've started performance benchmarking for MooseFS,
>> _____________ __ _ _
>> Chunkservers:
>> CPU: Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz
>> RAM: 12G
>> HDD: storage used for mfs chunks is raid0 of two WD 7200rpm caviar
>> black disks.
>> OS: OpenSuSE 11.3
>> Chunk servers number: 5
>>
>> -------------------------------
>> Master:
>>
>> CPU: Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz
>> OS: OpenSuSE 11.3
>> RAM: 12G
>>
>> --------------------------------
>> Client:
>> One of chunkservers.
>>
>> ---------------------------------------------------
>> LAN:
>> 1Gbit/s LAN.
>>
>> ____________ _ _ __
>> Problem;
>> I've experimented alot with bonnie, fio, iozone and others in 
>> testing
>> other storages... We have people working with the source code here, 
>> so
>> we need good random I/O for small files with moderate blocksize from 
>> 8k
>> to 128k... Comparative testing of other storage solutions involving 
>> ZFS,
>> different hardware raids etc showed that simple taring and untaring
>> Linux kernel sources shows how good can the storage be for that kind 
>> of
>> work... and it always correlates with more advanced fio and iozone
>> tests.
>>
>> So, simple untaring Linux kernel sources takes about 7 seconds on
>> chunkservers storage bypassing moosefs... but when I untar it to 
>> mounted
>> moosefs it takes more than 230 seconds.
>> Goal is set to 1. CPU load is OK on all the servers, RAM is 
>> sufficient.
>> Network is not overloaded and I can untar this file in about7 secs 
>> to
>> our nfs-mounted NAS....
>> I even turned on files caching on the client.
>> And this is all is very strange... maybe fuse is the bottleneck?...
>>
>>
>> 
>> ------------------------------------------------------------------------------
>> All the data continuously generated in your IT infrastructure 
>> contains a
>> definitive record of customers, application performance, security
>> threats, fraudulent activity and more. Splunk takes this data and 
>> makes
>> sense of it. Business sense. IT sense. Common sense.
>> http://p.sf.net/sfu/splunk-d2d-oct
>> _______________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] MooseFS performance benchmarking.

[Kristofer P. <kri...@cy...>]

MooseFS was designed to match Google File System (v1).  The GFS 
whitepaper even says its not ideal for many small files. It is most 
ideal for large files that are appended to.

You are going to need to hope for a multi-master set up and a lot of 
work on the internals to MooseFS to get better performance with small files.


On 10/24/2011 05:11 AM, leo...@ar... wrote:
> Something is very wrong with MooseFS regarding writing 'lots of small
> files'.
> You can even see it 'visually': if you tar xzvf linux*.tgz the names of
> the files appear much slooowly than this is in case of NFS or local
> storage...
> This is to say that we use quite a powerfull HW. And we have NO load on
> MFS at the moment. The only thing done at the moment is the untaring the
> linux kernel sources.
>
> I'd sincerelly appresiate any comments from the developers of the
> MooseFS, the project I had big hopes for )
> Do they observe the same in their environment?..
>
>
> P.S.
> sometimes it takes less than 100 seconds to untar, but still far beyond
> NFS, where I get it in 4-7 seconds or so.
>
>
>
>
> On Fri, 21 Oct 2011 00:00:06 +0800, Flow Jiang wrote:
>> We are experiencing the same issue when dealing with a lot of small
>> files.
>>
>> One observation is, when scp a large file through Gigabyte link onto
>> our NFS server (Served by a VERY old Sun Raid5 Disk Array), there
>> seems to have a large write cache to allow the speed reported at the
>> first several seconds greater than the writing limit of hard drives.
>> But when copying the same file onto MFS (Served by SAS 300G 10k disk
>> on Dell PE 1950), the speed never reaches the hard drive limit. And
>> our NFS server do perform better than MFS when copying large amount
>> of
>> source code files.
>>
>> I'm not sure if this issue is only related to our SW/HW
>> configuration. But seems the write caches works better with NFS than
>> MFS. Does any one know how does write cache of MFS work?
>>
>> Thanks
>> Flow
>>
>>
>> On 10/17/2011 10:46 PM, leo...@ar... wrote:
>>> Greetings!
>>> Resently I've started performance benchmarking for MooseFS,
>>> _____________ __ _ _
>>> Chunkservers:
>>> CPU: Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz
>>> RAM: 12G
>>> HDD: storage used for mfs chunks is raid0 of two WD 7200rpm caviar
>>> black disks.
>>> OS: OpenSuSE 11.3
>>> Chunk servers number: 5
>>>
>>> -------------------------------
>>> Master:
>>>
>>> CPU: Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz
>>> OS: OpenSuSE 11.3
>>> RAM: 12G
>>>
>>> --------------------------------
>>> Client:
>>> One of chunkservers.
>>>
>>> ---------------------------------------------------
>>> LAN:
>>> 1Gbit/s LAN.
>>>
>>> ____________ _ _ __
>>> Problem;
>>> I've experimented alot with bonnie, fio, iozone and others in
>>> testing
>>> other storages... We have people working with the source code here,
>>> so
>>> we need good random I/O for small files with moderate blocksize from
>>> 8k
>>> to 128k... Comparative testing of other storage solutions involving
>>> ZFS,
>>> different hardware raids etc showed that simple taring and untaring
>>> Linux kernel sources shows how good can the storage be for that kind
>>> of
>>> work... and it always correlates with more advanced fio and iozone
>>> tests.
>>>
>>> So, simple untaring Linux kernel sources takes about 7 seconds on
>>> chunkservers storage bypassing moosefs... but when I untar it to
>>> mounted
>>> moosefs it takes more than 230 seconds.
>>> Goal is set to 1. CPU load is OK on all the servers, RAM is
>>> sufficient.
>>> Network is not overloaded and I can untar this file in about7 secs
>>> to
>>> our nfs-mounted NAS....
>>> I even turned on files caching on the client.
>>> And this is all is very strange... maybe fuse is the bottleneck?...
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> All the data continuously generated in your IT infrastructure
>>> contains a
>>> definitive record of customers, application performance, security
>>> threats, fraudulent activity and more. Splunk takes this data and
>>> makes
>>> sense of it. Business sense. IT sense. Common sense.
>>> http://p.sf.net/sfu/splunk-d2d-oct
>>> _______________________________________________
>>> moosefs-users mailing list
>>> moo...@li...
>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
> ------------------------------------------------------------------------------
> The demand for IT networking professionals continues to grow, and the
> demand for specialized networking skills is growing even more rapidly.
> Take a complimentary Learning@Cisco Self-Assessment and learn
> about Cisco certifications, training, and career opportunities.
> http://p.sf.net/sfu/cisco-dev2dev
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users