[Aoetools-discuss] vblade performance issues

[kelsey h. <kh...@ts...>]

Hello.

I've been using AoE for some time now, and I notice a considerable 
slowdown on sustained writes to the backend device. Bursted writes are 
quite fast, but anything sustained crawls to a snail's pace.

I'm exporting my target volumes over an 802.3ad LACP bonded ethernet 
interface and on burst reads from cache can saturate this interface. I'm 
using jumbo frames and I have the raw socket read/write default/max 
buffer memory set to 256K. There are no dropped packets reported by 
/dev/etherd/err.

I have done a bit of research and I've observed that this slowdown is 
related to a block read that happens on the same underlying device when 
the device is also trying to write. There is nothing else using the 
volume, so therefore nothing should be trying to read.

Here is what I am currently doing:

My disk box exports a 36 GB LV using AoE to a client box on the network. 
On the client box, I partition, then create and mount a filesystem on 
the underlying device. I run:

dd if=/dev/zero of=/mnt/whatever/somelarge.file bs=4096 count=262144

to create a sustained write condition.

The underlying device is capable of ~300 MByte/sec sustained write. I'm 
seeing approximately 15 Mbyte/sec sustained.

Here are iostat outputs from the two machines during this process.

 From the machine mounting the exported disk:
Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
etherd/e6.2    4168.32         0.00     33346.53          0      33680

Notice how there are no reads pending from the device.


 From the machine exporting the disk:
Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
dm-9           9697.00     36136.00     41440.00      36136      41440

Notice now how there are a significant amount of reads.


There is nothing else using this logical volume. I can't understand why 
anything would read from the volume as it's writing. Is there a reason 
for this behaviour? It's a real detriment to performance.

For reference, the same thing done on the local machine results in these 
numbers:

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
dm-8          68833.66         0.00    550669.31          0     556176

Again, notice the lack of reads to the device (as one would expect). 
Also, notice the over tenfold increase in performance.

Here are some bonnie++ results from both machines (same parameters).

Machine importing AoE device: (trimmed to fit in this space)
        --Sequential Output-- --Sequential Input- --Random-
        --Block--   -Rewrite-      --Block--      --Seeks--
  Size  K/sec %CP   K/sec %CP      K/sec %CP      /sec  %CP
   12G  19994   7   33011  10      71552   4      218.8   0

Notice how the rewrite speed is much higher than the write speed. This 
is very curious.  I'd also expect the read speed to be a bit better, but 
that's not my primary concern right now.


Machine exporting AoE device: (trimmed to fit in this space)

           --Sequential Output--  --Sequential Input- --Random-
           --Block--   -Rewrite-     --Block--        --Seeks--
     Size  K/sec  %CP  K/sec %CP     K/sec   %CP      /sec   %CP
      12G  243176  64  63382  22     150545  35       379.8   1

These numbers are about right.


Does anyone else see anything strange or unexpected with these numbers? 
Can anyone shed some light onto the subject of why vblade might do a 
block read when it does a block write?

Thanks in advance.

-Kelsey Hudson