Menu
▾
▴
Re: [MooseFS-Users] question about mfs reading speed slower than writing speed???
|
From: Aleksander W. <ale...@mo...> - 2015-04-14 09:40:53
|
Hi Peter Thank you for your replay. First of all. Reading speed will be slower than writing because this data need to be read physically from all hard drives. Writing data can be done asynchronously - I/O is directed to cache and completion is immediately confirmed to the host. This results in low latency and high throughput for write-intensive applications. This mechanism is called write-back cache. Network latency is very important not only for MooseFS but for all distributed file systems. We need to remember that each file is generating many ACK to all components in cluster. So if you want to copy file from MFS to your local disc from folder with goal 3, client need to ask master where is the file. Then master responds to the client and tells where are chunks and on which chunkserver - 27.0 ms Than client starts to communicate with all chunkservers to download file. Now you can see that the time, which is taken to talk with all components is quite high. By default MFS performs read-ahead using 1MB block. Read of each block will be delayed by about 54ms (master+chunkserver latency), so in one second it will perform only about 20 reads 1MB each, so with such latency maximum read speed is about 20MB/s. To increase throughput you may increase read-ahead block size (and also other parameters). For example you may try such parameters for mfsmount: -o mfsreadaheadsize=256 -o mfsreadaheadleng=4194304 -o mfsreadaheadtrigger=8388608 This will set read-ahead block size to 4MB and in your environment it should increase linear reading speed significantly. Best regards Aleksander Wieliczko Technical Support Engineer MooseFS.com <moosefs.com> On 04/13/2015 02:57 PM, Peter wrote: > Hi,Aleks, > > According to your reply, I list out all the detail information about > my moosefs file system. you can see below, > > After several test, I find that , the read and write speed diff > because of the network latency. > > When the client and mfs master in the same IDC, the reading and > writing speed ,almost the same. > but when the client and mfs master in different city, it show a result > that the reading speed much slow than the write speed. > > can you explain why the network latency make such difference in > reading and writing ? > Also ,according to the test, seems that the reading speed never faster > than the write speed. it's strange, mostly, reading should be faster > than writing? > > > network topology: > ---------------------------------------------------------- > > master: > 10.153.136.230 > > master -metalogger: > 10.153.136.227 > > chunkserver: > 10.153.142.237 > 10.153.142.239 > 10.153.143.98 > > client: > same city: > 10.135.32.170 > > another city: > 10.149.131.111 > > mfs master , mfs metalogger, mfs chunk server are in same IDC. > > client 10.135.32.170 in same city ,different IDC. > > client 10.149.131.111 in other city. > > > Network Latency Test: > -------------------------------------------------------------------------------- > I do this test from the moosefs master server, ping all the other > component. > > mfs master ----> chunkserver > ping 10.153.142.237 > PING 10.153.142.237 (10.153.142.237) 56(84) bytes of data. > 64 bytes from 10.153.142.237 <http://10.153.142.237>: icmp_seq=1 > ttl=61 time=0.102 ms > 64 bytes from 10.153.142.237 <http://10.153.142.237>: icmp_seq=2 > ttl=61 time=0.084 ms > > > mfs master -----> another city client > ping 10.149.131.111 > PING 10.149.131.111 (10.149.131.111) 56(84) bytes of data. > 64 bytes from 10.149.131.111 <http://10.149.131.111>: icmp_seq=1 > ttl=55 time=27.0 ms > 64 bytes from 10.149.131.111 <http://10.149.131.111>: icmp_seq=2 > ttl=55 time=27.0 ms > > > mfs master -----------> same city client > ping 10.135.32.170 > PING 10.135.32.170 (10.135.32.170) 56(84) bytes of data. > 64 bytes from 10.135.32.170 <http://10.135.32.170>: icmp_seq=2 ttl=57 > time=2.28 ms > 64 bytes from 10.135.32.170 <http://10.135.32.170>: icmp_seq=3 ttl=57 > time=2.27 ms > > > mfs version information: > ----------------------------------------------------------------------------------- > the mfs version（current version): > ./mfsmaster -v > version: 2.0.61-1 > > also, I do the test in > ./mfsmaster -v > version: 1.6.27 > > ./mfsmount -V > MFS version 2.0.61-1 > FUSE library version: 2.8.3 > fusermount version: 2.8.3 > > > > *client read and write test:* > ------------------------------------------------------------------------------------------- > > > *Test1: same city client: 10.135.32.170* > read from mfs: > [root@TENCENT64 /mnt/mfs]# time dd if=/mnt/mfs/1-1.iso of=/dev/null > > 1024000+0 records in > 1024000+0 records out > 524288000 bytes (524 MB) copied, 12.8321 s, 40.9 MB/s > > real 0m12.841s > user 0m0.092s > sys 0m0.364s > [root@TENCENT64 /mnt/mfs]# time dd if=/mnt/mfs/1-2.iso of=/dev/null > 1024000+0 records in > 1024000+0 records out > 524288000 bytes (524 MB) copied, 10.148 s, 51.7 MB/s > > real 0m10.154s > user 0m0.120s > sys 0m0.328s > > write to mfs: > [root@TENCENT64 /mnt/mfs]# time dd if=/dev/zero of=/mnt/mfs/2-2.iso > bs=1M count=500 > 500+0 records in > 500+0 records out > 524288000 bytes (524 MB) copied, 5.04111 s, 104 MB/s > > real 0m5.047s > user 0m0.000s > sys 0m0.204s > [root@TENCENT64 /mnt/mfs]# time dd if=/dev/zero of=/mnt/mfs/2-3.iso > bs=1M count=500 > 500+0 records in > 500+0 records out > 524288000 bytes (524 MB) copied, 5.46042 s, 96.0 MB/s > > real 0m5.467s > user 0m0.000s > sys 0m0.208s > > > *another client in same city:* > read from mfs: > [root@TENCENT64 /mnt/mfs]# time cp /mnt/mfs/555-new.iso /data/ > > real 0m9.869s > user 0m0.000s > sys 0m0.669s > > [root@TENCENT64 /mnt/mfs]# time cp /mnt/mfs/555.iso /data/ > > real 0m9.980s > user 0m0.004s > sys 0m0.687s > > write to mfs: > [root@TENCENT64 /data]# time dd if=/dev/zero of=/mnt/mfs/1-1.iso bs=1M > count=500 > 500+0 records in > 500+0 records out > 524288000 bytes (524 MB) copied, 5.30263 s, 98.9 MB/s > > real 0m5.332s > user 0m0.000s > sys 0m0.349s > [root@TENCENT64 /data]# time dd if=/dev/zero of=/mnt/mfs/1-2.iso bs=1M > count=500 > 500+0 records in > 500+0 records out > 524288000 bytes (524 MB) copied, 5.12036 s, 102 MB/s > > real 0m5.128s > user 0m0.004s > sys 0m0.345s > > > > *client in other city test:* > > write to mfs: > [root@Tencent-SNG /mnt/mfs]# time dd if=/dev/zero of=/mnt/mfs/3-1.iso > bs=1M count=500 > 500+0 records in > 500+0 records out > 524288000 bytes (524 MB) copied, 13.2458 s, 39.6 MB/s > > real 0m13.332s > user 0m0.002s > sys 0m0.467s > [root@Tencent-SNG /mnt/mfs]# time dd if=/dev/zero of=/mnt/mfs/3-2.iso > bs=1M count=500 > 500+0 records in > 500+0 records out > 524288000 bytes (524 MB) copied, 12.1528 s, 43.1 MB/s > > real 0m12.211s > user 0m0.002s > sys 0m0.414s > > > > read from mfs: > > [root@Tencent-SNG /mnt/mfs]# time dd if=/mnt/mfs/1-1.iso of=/dev/null > 1024000+0 records in > 1024000+0 records out > 524288000 bytes (524 MB) copied, 44.355 s, 11.8 MB/s > > real 0m44.413s > user 0m0.202s > sys 0m0.653s > [root@Tencent-SNG /mnt/mfs]# time dd if=/mnt/mfs/1-2.iso of=/dev/null > 1024000+0 records in > 1024000+0 records out > 524288000 bytes (524 MB) copied, 45.0923 s, 11.6 MB/s > > real 0m45.151s > user 0m0.197s > sys 0m0.743s > > > > *last , test the client and mfs master in the same server (means in > same IDC):* > write to mfs: > > root@Tencent-SNG /mnt/mfs]# time dd if=/dev/zero of=/mnt/mfs/4-1.iso > bs=1M count=500 > 500+0 records in > 500+0 records out > 524288000 bytes (524 MB) copied, 4.71052 s, 111 MB/s > > real 0m4.712s > user 0m0.000s > sys 0m0.272s > [root@Tencent-SNG /mnt/mfs]# time dd if=/dev/zero of=/mnt/mfs/4-2.iso > bs=1M count=500 > 500+0 records in > 500+0 records out > 524288000 bytes (524 MB) copied, 4.63003 s, 113 MB/s > > real 0m4.632s > user 0m0.000s > sys 0m0.284s > > > read from mfs : > > [root@Tencent-SNG /mnt/mfs]# time dd if=/mnt/mfs/1-1.iso of=/dev/null > 1024000+0 records in > 1024000+0 records out > 524288000 bytes (524 MB) copied, 4.68863 s, 112 MB/s > > real 0m4.690s > user 0m0.140s > sys 0m0.412s > [root@Tencent-SNG /mnt/mfs]# time dd if=/mnt/mfs/1-2.iso of=/dev/null > 1024000+0 records in > 1024000+0 records out > 524288000 bytes (524 MB) copied, 4.68907 s, 112 MB/s > > real 0m4.690s > user 0m0.120s > sys 0m0.412s > > > > > > > > > > On Mon, Apr 13, 2015 at 2:57 AM, Aleksander Wieliczko > <ale...@mo... > <mailto:ale...@mo...>> wrote: > > Hi Peter > > Which version of MooseFS are you using? > > Can you tell something more about your topology - configuration? > Is your MooseFS components are in the same LAN ? > Can you check latency between all MooseFS components? > > > Best regards > Aleksander Wieliczko > Technical Support Engineer > MooseFS.com <http://moosefs.com> > > > On 12.04.2015 09:52, Peter wrote: >> Hi moosefs team, >> >> Recently, I setup a moosefs filesystem. >> with the following topology: >> >> Master server: 1 >> Mesta logger server: 1 >> chunkserver: 3 >> >> and each file will keep 3 copy. >> >> base in this topology, I have run some test: >> >> *and I find that write data to the mfs file system is much faster >> than the reading speed, this test result is confusing me. can you >> guys explain a little bit this for me?* >> >> >> all the test file are the same size >> /mnt/mfs --> moosefs file system. >> /data --> local disk >> --------------------------------------------------------------------------------------------- >> >> [root@TENCENT64 /mnt/mfs]# time cp 111.iso /mnt/mfs/ >> cp: cannot stat `111.iso': No such file or directory >> >> real 0m0.002s >> user 0m0.000s >> sys 0m0.000s >> [root@TENCENT64 /mnt/mfs]# time cp /data/111.iso /mnt/mfs/ >> >> real 0m5.065s >> user 0m0.000s >> sys 0m0.316s >> [root@TENCENT64 /mnt/mfs]# time cp 555.iso /data/ >> >> real 0m11.156s >> user 0m0.000s >> sys 0m0.780s >> [root@TENCENT64 /mnt/mfs]# ^C >> [root@TENCENT64 /mnt/mfs]# time cp 666.iso /data >> >> real 0m11.165s >> user 0m0.004s >> sys 0m0.864s >> [root@TENCENT64 /mnt/mfs]# time cp /data/555.iso /mnt/mfs/555-new.iso >> >> real 0m5.045s >> user 0m0.000s >> sys 0m0.300s >> [root@TENCENT64 /mnt/mfs]# ll -h >> total 3.0G >> -rw-r--r-- 1 root root 500M Apr 12 14:36 111.iso >> -rw-r--r-- 1 root root 500M Apr 12 15:33 555-new.iso >> -rw-r--r-- 1 root root 500M Apr 11 01:43 555.iso >> -rw-r--r-- 1 root root 500M Apr 11 01:46 666.iso >> -rw-r--r-- 1 root root 500M Apr 11 01:06 aaa.iso >> -rw-r--r-- 1 root root 500M Apr 11 01:39 bbb.iso >> >> >> also, i use another client server to run the test , (the client >> server in other city) >> >> we can see, the read speed is much more slow than the writing speed. >> >> [root@Tencent-SNG /mnt/mfs]# time cp 111.iso /data/ >> >> real 0m44.736s >> user 0m0.005s >> sys 0m1.299s >> [root@Tencent-SNG /mnt/mfs]# time cp /data/222.iso ./ >> >> [root@Tencent-SNG /data]# time cp 333.iso /mnt/mfs/333-new.iso >> >> real 0m12.432s >> user 0m0.008s >> sys 0m0.614s >> >> Also, in case there is some kernel or linux version problem, >> I setup the moosefs master in centos 6.3 and centos 7.0. >> still have the same problem. >> >> I also search the internet , and find that, the other also have >> the same problem. >> >> Other people's performance test data: >> Block size 1M Filesize20G >> Client1 write：68.4MB/s read：25.3MB/s >> Client2 write：67.5MB/s read：24.7MB/s >> >> >> >> ------------------------------------------------------------------------------ >> BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT >> Develop your own process in accordance with the BPMN 2 standard >> Learn Process modeling best practices with Bonita BPM through live exercises >> http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ >> source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF >> >> >> _________________________________________ >> moosefs-users mailing list >> moo...@li... <mailto:moo...@li...> >> https://lists.sourceforge.net/lists/listinfo/moosefs-users > > |
