Re: [MooseFS-Users] slow lookup

[David M. <dav...@pr...>]

38ms is very high latency - are the servers in the same datacenter? You should expect <1ms network latency if they are.

Cheers,
Dave

Sent with [ProtonMail](https://protonmail.com) Secure Email.

-------- Original Message --------
Subject: Re: [MooseFS-Users] slow lookup
Local Time: May 18, 2017 5:13 AM
UTC Time: May 18, 2017 9:13 AM
From: it...@da...
To: Aleksander Wieliczko <ale...@mo...>
moo...@li...

Ping stat:

124 packets transmitted, 124 received, 0% packet loss, time 123143ms
rtt min/avg/max/mdev = 38.024/38.077/38.209/0.252 ms

I found an issue with FUSE on master server, I cannot mount mfs localy:

fuse: device not found, try 'modprobe fuse' first
error in fuse_mount

modprobe fuse
modprobe: ERROR: could not insert 'fuse': Unknown symbol in module, or unknown parameter (see dmesg)

dmesg | grep fuse

[4248962.398993] fuse: Unknown symbol setattr_prepare (err 0)
[4410205.380837] fuse: Unknown symbol setattr_prepare (err 0)
[4410216.902707] fuse: Unknown symbol setattr_prepare (err 0)

I will try to find out what is the issue with FUSE. Can it cause my lookup issue?

--
Best regards,
Eugene

18 мая 2017 г., в 12:01, Aleksander Wieliczko <ale...@mo...> написал(а):

Hi.
Thank you for all this information.
Hardware is really nice.

Would you be so kind and execute this two tests in your environment?

1. Mount Moosefs Client on MooseFS master and execute time ls -l in folder with over 100k files
2. Execute ping command from "slow" MooseFS client to MooseFS master

Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com

On 18.05.2017 10:27, Eugene Diatlov wrote:
CPU Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
RAM 32 GB with ECC
1 Gbit connection.

load average: 0.00, 0.01, 0.05

Linux mfs 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux

It is a totally new empty server. If I copy 1 gig file to mfs it tooks around 22 seconds, so it is couldn't be a network issue.
The problem only with lookup operation.

mfs metadata files are placed on ssd system drive. read/write test:

dd if=/dev/zero of=/test bs=1024 count=5024000
5024000+0 records in
5024000+0 records out
5144576000 bytes (5.1 GB) copied, 19.4822 s, 264 MB/s

dd if=/test of=/dev/null bs=1024 count=5024000
5024000+0 records in
5024000+0 records out
5144576000 bytes (5.1 GB) copied, 1.56791 s, 3.3 GB/s

mfschukserver raid read/write local test (not thru mfs):
dd if=/dev/zero of=/mnt/data/test bs=1024 count=5024000
5024000+0 records in
5024000+0 records out
5144576000 bytes (5.1 GB) copied, 6.66198 s, 772 MB/s

dd if=/mnt/data/test of=/dev/null bs=1024 count=5024000
5024000+0 records in
5024000+0 records out
5144576000 bytes (5.1 GB) copied, 1.52796 s, 3.4 GB/s

--
Best regards,
Eugene

18 мая 2017 г., в 10:51, Aleksander Wieliczko <ale...@mo...> написал(а):

Hi,
metadata operations like ls, mostly depends on MooseFS master CPU speed and network latency.

Would you be so kind and tell us something more about this parameters?

Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com

On 18.05.2017 09:05, Ben Harker wrote:

I've experienced weirdness when using various raid setups. It's stated in the documentation that a non raid setup is preferred - at least from what I've remembered. MFS is a fuse filesystem that's bound to have a performance penalty somewhere when it comes to huge numbers of small files due to it's distributed nature.

I've mitigated this somewhat by introducing an SSD based tier of storage that holds files smaller than a certain size. This one's definitely helped, and you can also set it as the C (create) tier for when files get created if you get creative with your storage classes - the initial writes to SSD are much quicker and can make your setup feel much faster when writing files.

Hope that's helpful!

On May 18, 2017 07:06, Eugene Diatlov [<it...@da...>](mailto:it...@da...) wrote:

Hi,

I have a hardware server with latest moosefs. HDD's are in a hardware RAID5. Debian 8.
I have a lot of small files in folders (100k+). When I enter the folder I have a big delay while the system reading directory's files list.
Using moosefs metrics I found that the speed of lookup operations is equal to 1,8k operations per minute at maximum.

Why this could happen? I have another server with moosefs and the speed of lookup could be 300k per minute and higher.

--
Best regards,
Eugene

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites,
[Slashdot.org](http://slashdot.org/)
!
http://sdm.link/slashdot

_________________________________________
moosefs-users mailing list
moo...@li...

https://lists.sourceforge.net/lists/listinfo/moosefs-users