From: Ross S. W. W. <RW...@me...> - 2008-11-17 22:10:36
|
No, should be in the same place. Errors on the initiator and not the target usually mean network communications problems. I suspect it's a problem with trunking jumbo frames between the two PowerConnects, or you don't have the management NIC (with 1500 mtu) on a different broadcast domain (either isolated switch or vlan) then the iscsi bond and ARP filtering isn't enabled, causing eth0 with mtu 1500 trying to service the 9000 frames. -Ross ________________________________ From: David Levinger [mailto:da...@ma...] Sent: Monday, November 17, 2008 4:57 PM To: Ross S. W. Walker; isc...@li... Subject: RE: [Iscsitarget-devel] Jumbo Frame iSCSI failure centos 5.2 kernel2.6.18-92.1.18.el5xen I didn't see any, but I have a feeling it is because I'm not looking in the right place. I was seeing errors in /var/log/messages on the clients, on the server is there a difference location that I should be looking for? d ________________________________ From: Ross S. W. Walker [mailto:RW...@me...] Sent: Monday, November 17, 2008 12:59 PM To: David Levinger; isc...@li... Subject: RE: [Iscsitarget-devel] Jumbo Frame iSCSI failure centos 5.2 kernel2.6.18-92.1.18.el5xen What errors did you get on the target side? -Ross ________________________________ From: David Levinger [mailto:da...@ma...] Sent: Monday, November 17, 2008 3:12 PM To: Ross S. W. Walker; isc...@li... Subject: RE: [Iscsitarget-devel] Jumbo Frame iSCSI failure centos 5.2 kernel2.6.18-92.1.18.el5xen I'm certainly not stuck on using Jumbo Frames, if the consensus is that until we end up using 10Gbe it isn't worth it then I'm fine turning it off and leaving it turned off. I am curious as to what is causing those errors to show up. I checked on the switch and flow control is disabled. d ________________________________ From: Ross S. W. Walker [mailto:RW...@me...] Sent: Monday, November 17, 2008 12:08 PM To: David Levinger; isc...@li... Subject: RE: [Iscsitarget-devel] Jumbo Frame iSCSI failure centos 5.2 kernel2.6.18-92.1.18.el5xen When doing Jumbo Frames with Dell switches you must ensure that flow control is disabled. I always recommend not using Jumbo frames for speeds < 10Gbe, it isn't worth the load on the switches and it's actually slower on normal workloads then standard 1500 MTU frames. If you insist on Jumbo frames though, make sure flow control is disabled on the switches. -Ross ________________________________ From: David Levinger [mailto:da...@ma...] Sent: Monday, November 17, 2008 2:18 PM To: isc...@li... Subject: [Iscsitarget-devel] Jumbo Frame iSCSI failure centos 5.2 kernel2.6.18-92.1.18.el5xen Hey guys, I'm working on configuring an iSCSI setup on Centos 5.2 x86_64with the 2.6.18-92.1.18.el5xen kernel on the clients and 2.6.18-92.1.18.el5 #1 SMP on the server and whenever I enable jumbo frames and iSCSI I receive the errors below from the clients, iSCSI works great without Jumbo Frames enabled and it is enabled on the server/client/switch: Nov 17 11:59:47 rho kernel: scsi1 : iSCSI Initiator over TCP/IP Nov 17 11:59:47 rho kernel: Vendor: IET Model: VIRTUAL-DISK Rev: 0 Nov 17 11:59:47 rho kernel: Type: Direct-Access ANSI SCSI revision: 04 Nov 17 11:59:47 rho kernel: SCSI device sdb: 3996634572 512-byte hdwr sectors (2046277 MB) Nov 17 11:59:47 rho kernel: sdb: Write Protect is off Nov 17 11:59:47 rho kernel: SCSI device sdb: drive cache: write through Nov 17 11:59:47 rho kernel: SCSI device sdb: 3996634572 512-byte hdwr sectors (2046277 MB) Nov 17 11:59:47 rho kernel: sdb: Write Protect is off Nov 17 11:59:47 rho kernel: SCSI device sdb: drive cache: write through Nov 17 11:59:48 rho iscsid: received iferror -38 Nov 17 11:59:48 rho last message repeated 2 times Nov 17 11:59:48 rho iscsid: connection1:0 is operational now Nov 17 11:59:51 rho udevd-event[7505]: wait_for_sysfs: waiting for '/sys/devices/platform/host1/session1/target1:0:0/1:0:0:1/ioerr_cnt' failed Nov 17 11:59:57 rho kernel: sdb:<3>ping timeout of 5 secs expired, last rx 4295061736, last ping 4295062986, now 4295064236 Nov 17 11:59:57 rho kernel: connection1:0: iscsi: detected conn error (1011) Nov 17 11:59:58 rho iscsid: Kernel reported iSCSI connection 1:0 error (1011) state (3) Nov 17 12:00:01 rho iscsid: received iferror -38 Nov 17 12:00:01 rho last message repeated 2 times Nov 17 12:00:01 rho iscsid: connection1:0 is operational after recovery (1 attempts) Nov 17 12:00:10 rho kernel: ping timeout of 5 secs expired, last rx 4295064992, last ping 4295066242, now 4295067492 Nov 17 12:00:10 rho kernel: connection1:0: iscsi: detected conn error (1011) Nov 17 12:00:11 rho iscsid: Kernel reported iSCSI connection 1:0 error (1011) state (3) Nov 17 12:00:14 rho iscsid: received iferror -38 Nov 17 12:00:14 rho last message repeated 2 times Nov 17 12:00:14 rho iscsid: connection1:0 is operational after recovery (1 attempts) I'd been led to believe that enabling Jumbo Frames was important when setting up an iSCSI server that was going to be under heavy load. Below is the layout of how I have things configured: ----iSCSI Server---- Dell 2950 Dual Quad Core CPU 4GB of memory 2 onboard Broadcom Corporation NetXtreme II BCM5708, currently enabled and being used for management but NOT for iSCSI communication 1 PCI-E 4 port intel network card Intel Corporation 82571EB in a 4 port bond being used for iSCSI communication Dell MD1000 connected to Dell 2950 for storage, 15 drive 15k SAS array in a hardward raid 5 configuration. ----iSCSI ietd.conf---- Target iqn.2008-11.com.test:storage.tester1 IncomingUser OutgoingUser Lun 1 Path=/dev/sdb1,Type=blockio Alias tester1 ----iSCSI Client---- Dell 2950 Dual Quad Core CPU 4GB of memory 2 onboard Broadcom Corporation NetXtreme II BCM5708, currently bonded together enabled and being used for iSCSI communication 2 PCI-E Marvell Technology Group Ltd. 88E8053, currently bonded together and enabled for vlan Xen networking ----Network Infrastructure---- 2 Dell 6248 switches stacked together to form one logical switch connected over 10gbe to the core stack of 2 Dell 6248 switches. All ports associated with iSCSI communication have Maximum Frame Size (1518 - 9216) set to 9216. ----Things I've tried---- I have tried using the Marvell nics instead of the Broadcom for the iSCSI communication but that makes no difference. We have introduced another server, non-dell, non-broadcom nics and it exhibits the same behavior. It doesn't matter what I set the MTU size to be, if it's anything other then standard it causes the aforementioned errors (I tried setting it to 2000, 4000, and 6000). If I don't have iSCSI turned on but I do have the MTU set to 9000 everything "appears" to be normal, I can get to the machines via their Jumbo enabled IP address, write to it via dd_rescue etc and it doesn't throw any errors, but iSCSI will not function. Soon as I turn off the MTU=9000 in the /etc/sysconfig/network-scripts/ifcfg-bond1 config file iSCSI works great no issues. Any ideas as to what I'm doing wrong? Is Jumbo Frames something I should make sure I get working? Are there better performance settings inside ietd.conf that I should be using that I'm not? Thanks in advance, David ________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof. ________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof. ______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof. |