Menu
▾
▴
Re: [MooseFS-Users] slow lookup
|
From: Alex C. <ac...@in...> - 2017-05-18 18:21:59
|
It doesn't matter how stable your 1GB bandwidth is if you have that kind of latency. There is no way you can expect to get the same lookup performance if your Master/Client/Chunkserver setup has that kind of geographical/topological separation. Alex On 18/05/17 18:26, Eugene Diatlov wrote: > Different DC, but connection is stable 1GB. > > -- > Best regards, > Eugene > > > > >> 18 мая 2017 г., в 19:30, David Myer <dav...@pr... >> <mailto:dav...@pr...>> написал(а): >> >> 38ms is very high latency - are the servers in the same datacenter? >> You should expect <1ms network latency if they are. >> >> Cheers, >> Dave >> >> >> Sent with ProtonMail <https://protonmail.com/> Secure Email. >> >>> -------- Original Message -------- >>> Subject: Re: [MooseFS-Users] slow lookup >>> Local Time: May 18, 2017 5:13 AM >>> UTC Time: May 18, 2017 9:13 AM >>> From: it...@da... <mailto:it...@da...> >>> To: Aleksander Wieliczko <ale...@mo... >>> <mailto:ale...@mo...>> >>> moo...@li... >>> <mailto:moo...@li...> >>> >>> Ping stat: >>> 124 packets transmitted, 124 received, 0% packet loss, time 123143ms >>> rtt min/avg/max/mdev = 38.024/38.077/38.209/0.252 ms >>> >>> I found an issue with FUSE on master server, I cannot mount mfs localy: >>> fuse: device not found, try 'modprobe fuse' first >>> error in fuse_mount >>> >>> modprobe fuse >>> modprobe: ERROR: could not insert 'fuse': Unknown symbol in module, >>> or unknown parameter (see dmesg) >>> >>> dmesg | grep fuse >>> [4248962.398993] fuse: Unknown symbol setattr_prepare (err 0) >>> [4410205.380837] fuse: Unknown symbol setattr_prepare (err 0) >>> [4410216.902707] fuse: Unknown symbol setattr_prepare (err 0) >>> >>> I will try to find out what is the issue with FUSE. Can it cause my >>> lookup issue? >>> >>> >>> -- >>> Best regards, >>> Eugene >>> >>> >>> >>>> 18 мая 2017 г., в 12:01, Aleksander Wieliczko >>>> <ale...@mo... >>>> <mailto:ale...@mo...>> написал(а): >>>> >>>> Hi. >>>> Thank you for all this information. >>>> Hardware is really nice. >>>> >>>> Would you be so kind and execute this two tests in your environment? >>>> >>>> 1. Mount Moosefs Client on MooseFS master and execute time ls -l in >>>> folder with over 100k files >>>> 2. Execute ping command from "slow" MooseFS client to MooseFS master >>>> >>>> Best regards >>>> Aleksander Wieliczko >>>> Technical Support Engineer >>>> MooseFS.com >>>> >>>> On 18.05.2017 10:27, Eugene Diatlov wrote: >>>>> CPU Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz >>>>> RAM 32 GB with ECC >>>>> 1 Gbit connection. >>>>> load average: 0.00, 0.01, 0.05 >>>>> Linux mfs 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 >>>>> (2016-10-19) x86_64 GNU/Linux >>>>> >>>>> It is a totally new empty server. If I copy 1 gig file to mfs it >>>>> tooks around 22 seconds, so it is couldn't be a network issue. >>>>> The problem only with lookup operation. >>>>> >>>>> mfs metadata files are placed on ssd system drive. read/write test: >>>>> >>>>> dd if=/dev/zero of=/test bs=1024 count=5024000 >>>>> 5024000+0 records in >>>>> 5024000+0 records out >>>>> 5144576000 bytes (5.1 GB) copied, 19.4822 s, 264 MB/s >>>>> >>>>> dd if=/test of=/dev/null bs=1024 count=5024000 >>>>> 5024000+0 records in >>>>> 5024000+0 records out >>>>> 5144576000 bytes (5.1 GB) copied, 1.56791 s, 3.3 GB/s >>>>> >>>>> mfschukserver raid read/write local test (not thru mfs): >>>>> dd if=/dev/zero of=/mnt/data/test bs=1024 count=5024000 >>>>> 5024000+0 records in >>>>> 5024000+0 records out >>>>> 5144576000 bytes (5.1 GB) copied, 6.66198 s, 772 MB/s >>>>> >>>>> dd if=/mnt/data/test of=/dev/null bs=1024 count=5024000 >>>>> 5024000+0 records in >>>>> 5024000+0 records out >>>>> 5144576000 bytes (5.1 GB) copied, 1.52796 s, 3.4 GB/s >>>>> >>>>> -- >>>>> Best regards, >>>>> Eugene >>>>> >>>>> >>>>> >>>>>> 18 мая 2017 г., в 10:51, Aleksander Wieliczko >>>>>> <ale...@mo... >>>>>> <mailto:ale...@mo...>> написал(а): >>>>>> >>>>>> Hi, >>>>>> metadata operations like ls, mostly depends on MooseFS master CPU >>>>>> speed and network latency. >>>>>> >>>>>> Would you be so kind and tell us something more about this >>>>>> parameters? >>>>>> >>>>>> >>>>>> Best regards >>>>>> Aleksander Wieliczko >>>>>> Technical Support Engineer >>>>>> MooseFS.com >>>>>> >>>>>> On 18.05.2017 09:05, Ben Harker wrote: >>>>>>> I've experienced weirdness when using various raid setups. It's >>>>>>> stated in the documentation that a non raid setup is preferred - >>>>>>> at least from what I've remembered. MFS is a fuse filesystem >>>>>>> that's bound to have a performance penalty somewhere when it >>>>>>> comes to huge numbers of small files due to it's distributed nature. >>>>>>> >>>>>>> I've mitigated this somewhat by introducing an SSD based tier >>>>>>> of storage that holds files smaller than a certain size. This >>>>>>> one's definitely helped, and you can also set it as the C >>>>>>> (create) tier for when files get created if you get creative >>>>>>> with your storage classes - the initial writes to SSD are much >>>>>>> quicker and can make your setup feel much faster when writing >>>>>>> files. >>>>>>> >>>>>>> Hope that's helpful! >>>>>>> >>>>>>> On May 18, 2017 07:06, Eugene Diatlov <it...@da...> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I have a hardware server with latest moosefs. HDD's are in a >>>>>>> hardware RAID5. Debian 8. >>>>>>> I have a lot of small files in folders (100k+). When I enter >>>>>>> the folder I have a big delay while the system reading >>>>>>> directory's files list. >>>>>>> Using moosefs metrics I found that the speed of lookup >>>>>>> operations is equal to 1,8k operations per minute at maximum. >>>>>>> Why this could happen? I have another server with moosefs >>>>>>> and the speed of lookup could be 300k per minute and higher. >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Best regards, >>>>>>> Eugene >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> ------------------------------------------------------------------------------ >>>>>>> Check out the vibrant tech community on one of the world's most >>>>>>> engaging tech sites,Slashdot.org <http://slashdot.org/>!http://sdm.link/slashdot >>>>>>> >>>>>>> >>>>>>> _________________________________________ >>>>>>> moosefs-users mailing list >>>>>>> moo...@li... >>>>>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users >>>>>>> >> > > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > _________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice. "Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856). |
