[Aoetools-discuss] vblade performance issues
Brought to you by:
ecashin,
elcapitansam
From: kelsey h. <kh...@ts...> - 2007-07-30 21:32:14
|
Hello. I've been using AoE for some time now, and I notice a considerable slowdown on sustained writes to the backend device. Bursted writes are quite fast, but anything sustained crawls to a snail's pace. I'm exporting my target volumes over an 802.3ad LACP bonded ethernet interface and on burst reads from cache can saturate this interface. I'm using jumbo frames and I have the raw socket read/write default/max buffer memory set to 256K. There are no dropped packets reported by /dev/etherd/err. I have done a bit of research and I've observed that this slowdown is related to a block read that happens on the same underlying device when the device is also trying to write. There is nothing else using the volume, so therefore nothing should be trying to read. Here is what I am currently doing: My disk box exports a 36 GB LV using AoE to a client box on the network. On the client box, I partition, then create and mount a filesystem on the underlying device. I run: dd if=/dev/zero of=/mnt/whatever/somelarge.file bs=4096 count=262144 to create a sustained write condition. The underlying device is capable of ~300 MByte/sec sustained write. I'm seeing approximately 15 Mbyte/sec sustained. Here are iostat outputs from the two machines during this process. From the machine mounting the exported disk: Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn etherd/e6.2 4168.32 0.00 33346.53 0 33680 Notice how there are no reads pending from the device. From the machine exporting the disk: Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dm-9 9697.00 36136.00 41440.00 36136 41440 Notice now how there are a significant amount of reads. There is nothing else using this logical volume. I can't understand why anything would read from the volume as it's writing. Is there a reason for this behaviour? It's a real detriment to performance. For reference, the same thing done on the local machine results in these numbers: Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dm-8 68833.66 0.00 550669.31 0 556176 Again, notice the lack of reads to the device (as one would expect). Also, notice the over tenfold increase in performance. Here are some bonnie++ results from both machines (same parameters). Machine importing AoE device: (trimmed to fit in this space) --Sequential Output-- --Sequential Input- --Random- --Block-- -Rewrite- --Block-- --Seeks-- Size K/sec %CP K/sec %CP K/sec %CP /sec %CP 12G 19994 7 33011 10 71552 4 218.8 0 Notice how the rewrite speed is much higher than the write speed. This is very curious. I'd also expect the read speed to be a bit better, but that's not my primary concern right now. Machine exporting AoE device: (trimmed to fit in this space) --Sequential Output-- --Sequential Input- --Random- --Block-- -Rewrite- --Block-- --Seeks-- Size K/sec %CP K/sec %CP K/sec %CP /sec %CP 12G 243176 64 63382 22 150545 35 379.8 1 These numbers are about right. Does anyone else see anything strange or unexpected with these numbers? Can anyone shed some light onto the subject of why vblade might do a block read when it does a block write? Thanks in advance. -Kelsey Hudson |