Re: [Moosefs-users] Bad write performance of mfschunkserver

[Michał B. <mic...@ge...>]

Hi Davies!

Here is our analysis of this situation. Different files are written simultaneously on the same CS - that's why pwrites are written do different files. Block size of 64kB is not that small. Writes in OS are sent through write cache so all saves being multiplication of 4096B should work equally fast.

Our tests:

dd on Linux 64k : 640k
$ dd if=/dev/zero of=/tmp/test bs=64k count=10000
10000+0 records in
10000+0 records out
655360000 bytes (655 MB) copied, 22.1836 s, 29.5 MB/s 

$ dd if=/dev/zero of=/tmp/test bs=640k count=1000 
1000+0 records in
1000+0 records out
655360000 bytes (655 MB) copied, 23.1311 s, 28.3 MB/s

dd on Mac OS X 64k : 640k
$ dd if=/dev/zero of=/tmp/test bs=64k count=10000
10000+0 records in
10000+0 records out
655360000 bytes transferred in 14.874652 secs (44058846 bytes/sec) 

$ dd if=/dev/zero of=/tmp/test bs=640k count=1000
1000+0 records in
1000+0 records out
655360000 bytes transferred in 14.578427 secs (44954096 bytes/sec)

So the times are similar. Saves going to different files also should not be a problem as a kernel scheduler takes care of this.

If you have some specific idea how to improve the saves please share it with us.


Kind regards
Michał 

-----Original Message-----
From: Davies Liu [mailto:dav...@gm...] 
Sent: Wednesday, February 22, 2012 8:24 AM
To: moo...@li...
Subject: [Moosefs-users] Bad write performance of mfschunkserver

Hi，devs:

Today, We found that some mfschunkserver were not responsive, caused many timeout in mfsmount, then all the write operation were blocked.

After some digging, we found that there were some small but continuous write bandwidth, strace show that many small pwrite() between several files:

[pid  7087] 12:28:28 pwrite(19, "baggins3 60.210.18.235 sE7NtNQU7"..., 25995, 55684725 <unfinished ...> [pid  7078] 12:28:28 pwrite(17, "2012/02/22 12:28:28:root: WARNIN"..., 69, 21768909 <unfinished ...> [pid  7080] 12:28:28 pwrite(20, "gardner4 183.7.50.169 mr5vi+Z4H3"..., 47663, 34550257 <unfinished ...> [pid  7079] 12:28:28 pwrite(19, "\" \"Mozilla/5.0 (Windows NT 6.1) "..., 40377, 55710720 <unfinished ...> [pid  7086] 12:28:28 pwrite(23, "MATP; InfoPath.2; .NET4.0C; 360S"..., 65536, 6427648 <unfinished ...> [pid  7082] 12:28:28 pwrite(23, "; GTB7.2; SLCC2; .NET CLR 2.0.50"..., 65536, 6493184 <unfinished ...> [pid  7083] 12:28:28 pwrite(20, "\255BYU\355\237\347\226s\261\307N{A\355\203S\306\244\255\322[\322\rJ\32[z3\31\311\327"...,
4096, 1024 <unfinished ...>
[pid  7078] 12:28:28 pwrite(23, "ovie/subject/4724373/reviews?sta"...,
65536, 6558720 <unfinished ...>
[pid  7080] 12:28:28 pwrite(19,
"[\"[\4\5\266v\324\366\245n\t\315\202\227\\\343=\336-\r
k)\316\354\335\353\373\340\331;"..., 4096, 1024 <unfinished ...> [pid  7079] 12:28:28 pwrite(23, "ta-Python/2.0.15\"
0.016\n211.147."..., 65536, 6624256 <unfinished ...> [pid  7081] 12:28:28 pwrite(23, "4034093?apikey=0eb695f25995d7eb2"...,
65536, 6689792 <unfinished ...>
[pid  7084] 12:28:28 pwrite(23, " y8G23n95BKY:43534427:wind8vssc4"...,
65536, 6755328) = 65536 <0.000108>
[pid  7078] 12:28:28 pwrite(23, "TkVvKuXfug:3248233:5Yo9vFoOIuo \""..., 65536, 6820864 <unfinished ...> [pid  7086] 12:28:28 pwrite(23, ":s|1563396:s|1040897:s|1395290:s"...,
65536, 6886400 <unfinished ...>
[pid  7085] 12:28:28 pwrite(23, "dows%3B%20U%3B%20Windows%20NT%20"...,
65536, 6951936 <unfinished ...>
[pid  7087] 12:28:28 pwrite(23, "/533.17.9 (KHTML, like Gecko) Ve"..., 65536, 7017472 <unfinished ...> [pid  7079] 12:28:28 pwrite(23, " r1m+tFW1T5M::
\"22/Feb/2012:00:0"..., 65536, 7083008 <unfinished ...> [pid  7086] 12:28:28 pwrite(19, "baggins5 61.174.60.117 i6MSCBvE1"..., 25159, 55751097 <unfinished ...> [pid  7084] 12:28:28 pwrite(20, "gardner1 182.118.7.64 TjxzPKdqNU"..., 10208, 34597920 <unfinished ...> [pid  7080] 12:28:28 pwrite(23, "d7eb2c23c1d70cc187c1&alt=json HT"..., 65536, 7148544 <unfinished ...> [pid  7083] 12:28:28 pwrite(23, "5_Google&type=n&channel=-3&user_"...,
65536, 7214080 <unfinished ...>
[pid  7085] 12:28:28 pwrite(19, "12-02-22 12:28:27 1861 \"GET /ser"..., 23179, 55776256 <unfinished ...> [pid  7082] 12:28:28 pwrite(23, "\"http://douban.fm/swf/53035/fmpl"..., 65536, 7279616 <unfinished ...> [pid  7078] 12:28:28 pwrite(20, "opic/27639291/add_comment HTTP/1"..., 18576, 34608128 <unfinished ...> [pid  7087] 12:28:28 pwrite(19, "[\"[\4\5\266v\324\366\245n\t\315\202\227\\\343=\336-\r
k)\316\354\335\353\373\340\331;"..., 4096, 1024 <unfinished ...> [pid  7079] 12:28:28 pwrite(23, "ww.douban.com%2Fgroup%2Ftopic%2F"...,
65536, 7345152 <unfinished ...>
[pid  7081] 12:28:28 pwrite(20,
"\255BYU\355\237\347\226s\261\307N{A\355\203S\306\244\255\322[\322\rJ\32[z3\31\311\327"...,
4096, 1024 <unfinished ...>
[pid  7086] 12:28:28 pwrite(23, "patible; MSIE 7.0; Windows NT 6."..., 65536, 7410688 <unfinished ...> [pid  7084] 12:28:28 pwrite(23, "fari/535.7 360EE\"
0.006\n211.147."..., 65536, 7476224 <unfinished ...> [pid  7080] 12:28:28 pwrite(23, "1:OUIVR8CIG5c \"22/Feb/2012:00:03"..., 65536, 7541760 <unfinished ...> [pid  7085] 12:28:28 pwrite(23, "fm \"GET /j/mine/playlist?type=s&"..., 65536, 7607296 <unfinished ...> [pid  7083] 12:28:28 pwrite(23, "pe=n&channel=18&user_id=39266798"...,
65536, 7672832 <unfinished ...>
[pid  7082] 12:28:28 pwrite(23, " 0.023\n125.34.190.128 ::
\"22/Feb"..., 65536, 7738368 <unfinished ...> [pid  7078] 12:28:28 pwrite(23, "00 5859 \"http://www.douban.com/p"..., 65536, 7803904 <unfinished ...> [pid  7079] 12:28:28 pwrite(23, "03:08 +0800\" www.douban.com \"GET"..., 65536, 7869440 <unfinished ...> [pid  7086] 12:28:28 pwrite(23, "type=all HTTP/1.1\" 200 1492 \"-\"
"..., 65536, 7934976 <unfinished ...>
[pid  7084] 12:28:28 pwrite(23, "Hiapk&user_id=57982902&expire=13"...,
65536, 8000512 <unfinished ...>
[pid  7080] 12:28:28 pwrite(23, "0.011\n116.253.89.216 rxASuWZf1wg"..., 65536, 8066048 <unfinished ...> [pid  7085] 12:28:28 pwrite(23, "9 +0800\" www.douban.com \"GET /ph"..., 65536, 8131584) = 65536 <0.000062> [pid  7083] 12:28:28 pwrite(23, " +0800\" www.douban.com \"GET /eve"..., 65536, 8197120 <unfinished ...> [pid  7082] 12:28:28 pwrite(23, " +0800\" www.douban.com \"POST /se"..., 65536, 8262656) = 65536 <0.000103> [pid  7087] 12:28:28 pwrite(23, "0 12971 \"http://www.douban.com/g"..., 65536, 8328192 <unfinished ...> [pid  7081] 12:28:28 pwrite(23, ".0 (compatible; MSIE 7.0; Window"..., 65536, 8393728) = 65536 <0.000065>

In order to get better performance, the chunk server should merge the continuous sequential write operations into larger ones.

--
 - Davies

------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning Cloud computing makes use of virtualization - but cloud computing also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users