[Aoetools-discuss] domU is causing misaligned disk writes

[Tracy R. <tr...@ul...>]

Anyone know why my xen xvda devices would be doing (apparently)
unaligned writes to my SAN causing horrible performance and massive
seeking and lots of reading for page cache backfill? BUT writing to
the device in the dom0 is very fast and causes no extra reads?

I am running the 2.6.18-164.11.1.el5xen xen/kernel which came with
CentOS 5.4

After spending a lot of time banging my head on this I seem to have
finally tracked it down to a difference between domU and dom0.  I
never would have thought it would be this but it is extremely
reproduceable. We're talking a difference of 4-5x in write speed.
Reads are equally fast everywhere.

I am using AoE v72 kernel module (initiator) on a Dell R610's to talk
to vblade-19 (target) on Dell R710's all running CentOS 5.4. I have
striped two 7200 RPM SATA disks and exported the md with AoE (although
I have done these tests with individual disks also). Read performance
is excellent:

# dd of=/dev/null if=/dev/xvdg1 bs=4096 count=3000000
3000000+0 records in
3000000+0 records out
12288000000 bytes (12 GB) copied, 106.749 seconds, 115 MB/s

I dropped the cache with:

echo 1 > /proc/sys/vm/drop_caches

on both target and initiator before starting the test. This is great
for just a single gig-e link. This suggests that the network is fine.

However, write performance is odious. Typically around 20MB/s. It
should be more like 70MB/s per disk or better (7200rpm SATA) and max
out my gig-e with write performance similar to the above read
performance. I mentioned above that these are unaligned writes because
when running iostat on the target machine I can see lots of reads
happening which are surely causing seeks and killing
performance. Typical is something like 8MB/s of reads while doing
16MB/s of writes.

HOWEVER, if I do the writes from the dom0 the performance is
excellent:

# dd if=/dev/zero of=/dev/etherd/e6.2 bs=4096 count=3000000
3000000+0 records in
3000000+0 records out
12288000000 bytes (12 GB) copied, 104.679 seconds, 117 MB/s

And I see no reads happening on the disks being written to in
iostat. Purely streaming writes at high speeds.

I have had AoE working very well with Xen previously although not with
this particular hardware/xen/aoe version. Also it occurs to me that in
the past when I have done this I network booted the domU's and they
got root over AoE using a complicated initrd that I cooked up. In the
last year or so I decided that it was too complicated and went to
booting my dom0's from compact flash with the AoE driver in the dom0
instead of the domU. I now handing the domU xvd's from the AoE driver
in dom0. I strongly suspect that this is why things worked great
before but stink now. Unfortunately I don't have a working network
boot initrd setup like I used to and although I still have all of the
code etc. it would take a while to set up. I don't want to run that
setup in production anymore anyway if I can help it.

I have tried manually aligning the disk by setting the beginning of
data on the partition from 63 to 64 (although this is usually done for
RAID alignment) and I have tried changing the disk geometry to account
for the extra partition table which causes a half-block page-cache
misalignment as described by the ever insightful Kelsey Hudson in his
writeup on the issue here:

http://copilotco.com/Virtualization/wiki/aoe-caching-alignment.pdf/at_download/file

All to no avail. What am I missing here? Why is domU apparently
fudging my writes?

-- 
Tracy Reed
http://tracyreed.org