Menu
▾
▴
Re: [Iscsitarget-devel] Re : ietd & VMware ESX best practices?
|
From: Ming Z. <bla...@gm...> - 2007-07-08 14:59:29
|
On Sun, 2007-07-08 at 08:37 +0000, philippe florent wrote: > hi, > sorry to interfeer > I was wondering if you achieved to make ietd reachable on ESX 3.0 what u meant about "reachable"? > I know there are issues with some commands that vmware don't interpret > correctly like > 'refresh' or something > I can make it work on my laptop runin VMWare Workstation so far... > > sheers > > ----- Message d'origine ---- > De : Ross S. W. Walker <rw...@me...> > À : bla...@gm...; Jim Kusznir <jku...@gm...> > Cc : isc...@li... > Envoyé le : Samedi, 7 Juillet 2007, 4h50mn 52s > Objet : Re: [Iscsitarget-devel] ietd & VMware ESX best practices? > > > -----Original Message----- > > From: Ming Zhang [mailto:bla...@gm...] > > Sent: Friday, July 06, 2007 10:00 PM > > To: Jim Kusznir > > Cc: Ross S. W. Walker; isc...@li... > > Subject: Re: [Iscsitarget-devel] ietd & VMware ESX best practices? > > > > On Fri, 2007-07-06 at 15:30 -0700, Jim Kusznir wrote: > > > Hi: > > > > > > Here's the vmstat: > > > > > > vs1 ~ # vmstat 1 > > > procs -----------memory---------- ---swap-- -----io---- > > -system-- ----cpu---- > > > r b swpd free buff cache si so bi bo > > in cs us sy id wa > > > 0 0 0 1914768 70648 42700 0 0 4 4 > > 1 2 0 0 100 0 > > > 0 0 0 1914768 70648 42700 0 0 10979 387 > > 7568 22738 0 4 96 0 > > > 0 0 0 1914768 70648 42700 0 0 8114 255 > > 5761 17297 0 1 99 0 > > > 4 0 0 1914768 70648 42700 0 0 6838 226 > > 5102 15032 0 2 98 0 > > > 0 0 0 1914644 70648 42700 0 0 7236 511 > > 5576 16039 0 4 96 0 > > > 0 0 0 1914644 70648 42700 0 0 7254 221 > > 5306 15296 0 1 99 0 > > > 0 0 0 1914644 70648 42700 0 0 6204 1169 > > 5208 14831 0 3 97 0 > > > 0 0 0 1914644 70648 42700 0 0 7140 724 > > 5829 17238 0 4 96 0 > > > 0 0 0 1914768 70648 42700 0 0 8816 252 > > 5758 18129 0 5 95 0 > > > 0 0 0 1914768 70648 42700 0 0 10388 324 > > 6405 20782 0 3 97 0 > > > 0 0 0 1914768 70648 42700 0 0 9224 207 > > 5805 19057 0 2 98 0 > > > 0 0 0 1914768 70648 42700 0 0 8112 52 > > 5261 16623 0 2 98 0 > > > > little ram usage, bi and bo is not much as well. strange thing is > the > > interrupt number and context switch is not low at all. > > Interrupts appear ok for 1500 mtu and 8000 block avg, idle system can > generate 1000 interrupts a sec, bump mtu to 9000 and interrupts will > probably drop to 2000 avg. but latency for small io will increase. > > > cat /proc/interrupt couple times and find out which is the main > > interrupt consumer. > > > > common consumer is NIC and storage. but that means quite a > > lot io going > > on. which is not shown in bi and bo. > > > > also cpu is so idle. even iowait is near 0. > > block-io is minimal on cpu and iowait needs a big load with block-io > to throw up some numbers. > > > run "iostat -x -k 1" and show us the result. > > > > also we need HW and SW info. maybe a buggy kernel? ;) > > nah, just light load, but let us know how you think your VMs are > running. > > Most virtualization software caches io to memory and lets it go in > streams to even it out. With MS Virtual Server and 16 VMs here is a > vmstat sample with block-io: > > procs -----------memory---------- ---swap-- -----io---- --system-- > ----cpu---- > r b swpd free buff cache si so bi bo in cs us > sy id wa > 1 0 0 2670784 42952 251216 0 0 1 52 1070 > 235 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 0 179 1182 > 579 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 0 29 1048 > 181 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 0 2 1045 > 128 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 0 63 1075 > 270 0 1 99 0 > 0 0 0 2670784 42952 251216 0 0 0 17 1055 > 165 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 1 51 1063 > 240 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 0 3 1041 > 114 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 0 19 1049 > 195 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 0 0 > 1030 79 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 0 25 1037 > 137 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 0 28 1094 > 315 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 0 300 1216 > 615 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 0 10 1057 > 188 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 0 17 1036 > 143 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 0 44 1068 > 184 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 0 51 1076 > 269 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 0 20 1055 > 179 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 0 11 1043 > 169 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 0 20 1061 > 192 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 0 37 1048 > 182 0 0 100 0 > 0 0 0 2670784 42952 251216 0 0 0 22 1057 > 172 0 0 100 0 > > Of course this is after hours and the VMs aren't doing much, but even > if I hit > them hard the io pattern increases just a little bit more. It's only > when I > shut them down, start them up or save their state that io is hit hard. > > > > > > > This machine is dedicated exclusively to iscsi serving for our ESX > > > farm and 20 VMs. > > > > > > I think the critical issue we were seeing was related to the > mptscsi > > > bug that someone else cited here. I've been upgrading all our VMs > > > with the improved drivers today, and that should stop the > read-only > > > filesystem problem. > > > > > > I'm still very interested in optimizing our iet > > implementation to get > > > the most out of the box we have, and I'd greatly appreciate any > > > suggestions you all have! > > > > > > Thanks! > > > --Jim > > > > > > On 7/6/07, Ming Zhang <bla...@gm...> wrote: > > > > On Fri, 2007-07-06 at 13:42 -0400, Ross S. W. Walker wrote: > > > > > > -----Original Message----- > > > > > > From: isc...@li... > > > > > > [mailto:isc...@li...] On > > > > > > Behalf Of Jim Kusznir > > > > > > Sent: Friday, July 06, 2007 1:31 PM > > > > > > To: isc...@li... > > > > > > Subject: [Iscsitarget-devel] ietd & VMware ESX best > practices? > > > > > > > > > > > > Hi all: > > > > > > > > > > > > We're trying to use ietd (on a gentoo box built > > specifically for the > > > > > > process -- i.e., very lean, using a CVS version > > including the ESX > > > > > > serial num patch) to serve up 1.1TB of space for our > > ESX cluster > > > > > > (currently 2 ESX servers; ~20 VMs). We keep running > > into issues with > > > > > > some of our linux VM filesystems getting an error and > > switching to > > > > > > read-only. > > > > > > > > > > Does anything appear in the ESX logs or iSCSI target > > logs at the time? > > > > > > > > > > > We are using LVM and block-level LUNs. We currently > > are using 2 LUNs > > > > > > with the load roughly equal. We've seen problems > > with machines on > > > > > > both luns. So far, we have not been able to > > determine the cause. > > > > > > > > > > Well we would need more information otherwise it could > > be as simple as > > > > > improper handling of the VMs on the operator side. > > > > > > > > > > > I have noticed while watching load on the iscsi > > server (dual P4-3Ghz > > > > > > w/ hyperthreading, 2GB RAM, hardware RAID5 -- a Dell > > PowerEdge 2650) > > > > > > that the load average will spike frequently (right > > now, I'm seeing a > > > > > > load of 6.72), but the cpu will be 97.7% idle, 0% > > iowait. Originally > > > > > > I was thinking there was no way we're overloading the > > server, but now > > > > > > I'm beginning to wonder. > > > > > > > > looks like your IET box is very idle? do "vmstat 1" for > > couple seconds > > > > and post here. we could like to see why the box is loaded. > > > > > > > > > > > > > > > > > > It's IO wait load. What controller is this Perc 5e? > > > > > > > > > > You could try going with the deadline or noop io > > scheduler on the RAID > > > > > array and see if it lightens. > > > > > > > > > > > Here's our config: > > > > > > > > > > > > > > > > > > Target iqn.2001-04.edu.wsu.eecs:storage.esx.sys1 > > > > > > Lun 0 Path=/dev/iscsiLuns/lun0,Type=blockio > > > > > > Lun 1 Path=/dev/iscsiLuns/lun1,Type=blockio > > > > > > > > try to use 2 targets. one lu per target might have better > > performance. > > > > > > > > > > > > > > > > (comments removed) > > > > > > > > > > Because of a bug in block-io and how it auto-generates > > it's ScsiId you > > > > > will need to manually set the ScsiId for each Lun. > > > > > > > > > > That is probably where the disk problems are occurring. > > > > > > > > > > It's also prudent to manually set the ScsiSN when using > > it with ESX. > > > > > > > > > > > I'm still pretty new to iSCSI, and am not at all > > confidant we've used > > > > > > the ideal configuration. I'd greatly appreciate any > > suggestions > > > > > > (including "you're crazy" if approprate :) ). One > > thought we had was > > > > > > possibly a form of lock contention with so many VMs > > running on one > > > > > > LUN. We're currently thinking about creating a > > seperate LUN for each > > > > > > VM, but I'm not convinced that's right either. > > > > > > > > > > I would have probably created a separate target for > > each Lun, so you > > > > > could easily move a Lun from one ESX to another, but > > with VMfs and > > > > > VMotion the current setup should work too. > > > > > > > > > > > > > ______________________________________________________________________ > > > > > This e-mail, and any attachments thereto, is intended > > only for use by > > > > > the addressee(s) named herein and may contain legally > privileged > > > > > and/or confidential information. If you are not the > > intended recipient > > > > > of this e-mail, you are hereby notified that any > dissemination, > > > > > distribution or copying of this e-mail, and any > > attachments thereto, > > > > > is strictly prohibited. If you have received this > > e-mail in error, > > > > > please immediately notify the sender and permanently delete > the > > > > > original and any copy or printout thereof. > > > > > > > > > > > > > > > > > -------------------------------------------------------------- > > ----------- > > > > > This SF.net email is sponsored by DB2 Express > > > > > Download DB2 Express C - the FREE version of DB2 > > express and take > > > > > control of your XML. No limits. Just data. Click to get it > now. > > > > > http://sourceforge.net/powerbar/db2/ > > > > > _______________________________________________ > > > > > Iscsitarget-devel mailing list > > > > > Isc...@li... > > > > > https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel > > > > -- > > > > Ming Zhang > > > > > > > > > > > > @#$%^ purging memory... (*!% > > > > http://blackmagic02881.wordpress.com/ > > > > http://www.linkedin.com/in/blackmagic02881 > > > > -------------------------------------------- > > > > > > > > > > -- > > Ming Zhang > > > > > > @#$%^ purging memory... (*!% > > http://blackmagic02881.wordpress.com/ > > http://www.linkedin.com/in/blackmagic02881 > > -------------------------------------------- > > > > ______________________________________________________________________ > This e-mail, and any attachments thereto, is intended only for use by > the addressee(s) named herein and may contain legally privileged > and/or confidential information. If you are not the intended recipient > of this e-mail, you are hereby notified that any dissemination, > distribution or copying of this e-mail, and any attachments thereto, > is strictly prohibited. If you have received this e-mail in error, > please immediately notify the sender and permanently delete the > original and any copy or printout thereof. > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > Iscsitarget-devel mailing list > Isc...@li... > https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel > > > > > ______________________________________________________________________ > Ne gardez plus qu'une seule adresse mail ! Copiez vos mails vers > Yahoo! Mail > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ Iscsitarget-devel mailing list Isc...@li... https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel -- Ming Zhang @#$%^ purging memory... (*!% http://blackmagic02881.wordpress.com/ http://www.linkedin.com/in/blackmagic02881 -------------------------------------------- |