Re: [MooseFS-Users] slow lookup

[Aleksander W. <ale...@mo...>]

Hi.
Thank you for all this information.
Hardware is really nice.

Would you be so kind and execute this two tests in your environment?

1. Mount Moosefs Client on MooseFS master and execute time ls -l in
folder with over 100k files
2. Execute ping command from "slow" MooseFS client to MooseFS master

Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com <moosefs.com>

On 18.05.2017 10:27, Eugene Diatlov wrote:
> CPU Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
> RAM 32 GB with ECC
> 1 Gbit connection.
> load average: 0.00, 0.01, 0.05
> Linux mfs 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19)
> x86_64 GNU/Linux
>
> It is a totally new empty server. If I copy 1 gig file to mfs it tooks
> around 22 seconds, so it is couldn't be a network issue. 
> The problem only with lookup operation.
>
> mfs metadata files are placed on ssd system drive. read/write test:
>
> dd if=/dev/zero of=/test bs=1024 count=5024000
> 5024000+0 records in
> 5024000+0 records out
> 5144576000 bytes (5.1 GB) copied, 19.4822 s, 264 MB/s
>
> dd if=/test of=/dev/null bs=1024 count=5024000
> 5024000+0 records in
> 5024000+0 records out
> 5144576000 bytes (5.1 GB) copied, 1.56791 s, 3.3 GB/s
>
> mfschukserver raid read/write local test (not thru mfs):
> dd if=/dev/zero of=/mnt/data/test bs=1024 count=5024000
> 5024000+0 records in
> 5024000+0 records out
> 5144576000 bytes (5.1 GB) copied, 6.66198 s, 772 MB/s
>
> dd if=/mnt/data/test of=/dev/null bs=1024 count=5024000
> 5024000+0 records in
> 5024000+0 records out
> 5144576000 bytes (5.1 GB) copied, 1.52796 s, 3.4 GB/s 
>
> -- 
> Best regards,
> Eugene
>
>
>
>> 18 мая 2017 г., в 10:51, Aleksander Wieliczko
>> <ale...@mo...
>> <mailto:ale...@mo...>> написал(а):
>>
>> Hi,
>> metadata operations like ls, mostly depends on MooseFS master CPU
>> speed and network latency.
>>
>> Would you be so kind and tell us something more about this parameters?
>>
>>
>> Best regards
>> Aleksander Wieliczko
>> Technical Support Engineer
>> MooseFS.com <x-msg://2/moosefs.com>
>>
>> On 18.05.2017 09:05, Ben Harker wrote:
>>> I've experienced weirdness when using various raid setups. It's
>>> stated in the documentation that a non raid setup is preferred - at
>>> least from what I've remembered. MFS is a fuse filesystem that's
>>> bound to have a performance penalty somewhere when it comes to huge
>>> numbers of small files due to it's distributed nature.
>>>
>>>  I've mitigated this somewhat by introducing an SSD based tier of
>>> storage that holds files smaller than a certain size. This one's
>>> definitely helped, and you can also set it as the C (create) tier
>>> for when files get created if you get creative with your storage
>>> classes - the initial writes to SSD are much quicker and can make
>>> your setup feel much faster when writing files. 
>>>
>>> Hope that's helpful!
>>>
>>> On May 18, 2017 07:06, Eugene Diatlov <it...@da...> wrote:
>>>
>>>     Hi,
>>>
>>>     I have a hardware server with latest moosefs. HDD's are in a
>>>     hardware RAID5. Debian 8.
>>>     I have a lot of small files in folders (100k+). When I enter the
>>>     folder I have a big delay while the system reading directory's
>>>     files list.
>>>     Using moosefs metrics I found that the speed of lookup
>>>     operations is equal to 1,8k operations per minute at maximum.
>>>     Why this could happen? I have another server with moosefs and
>>>     the speed of lookup could be 300k per minute and higher.  
>>>
>>>
>>>     -- 
>>>     Best regards,
>>>     Eugene
>>>
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org <http://Slashdot.org>! http://sdm.link/slashdot
>>>
>>>
>>> _________________________________________
>>> moosefs-users mailing list
>>> moo...@li...
>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>
>