From: Ming Z. <bla...@gm...> - 2006-12-26 13:24:43
|
Hi All Result about reserve/release with MPIO. Seems that our way to support R/R with MPIO has problem. So current code has to be fixed. Ming -------- Forwarded Message -------- > From: Bla...@em... > To: bla...@gm..., ip...@ie... > Subject: RE: [Ips] MPIO with Persistent Reserve Out/In > Date: Wed, 20 Dec 2006 11:36:54 -0500 > > > I can not find this from original rfc or the new guide draft about how > > MPIO work with Persistent Reserve Out/In or old Reserve/Release. > > > > In iSCSI, a session is a I_T nexus. With MPIO, then between one > > initiator and one target will have multiple I_T nexus. If initiator > > setup a reservation via 1st session, can it send out commands like > write > > via 2nd session? Intuitively, I think it should be able to. > > No - if you want that behavior, use multiple TCP connections in a single > iSCSI session. > > > But from SPC, reservation will base on I_T nexus so 2 sessions are 2 > > different I_T nexus. > > That is how iSCSI behaves - no change to SPC. > > Thanks, > --David > ---------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > bla...@em... Mobile: +1 (978) 394-7754 > ---------------------------------------------------- |
From: Juhani R. <jr...@ik...> - 2006-12-26 14:18:04
|
On Tue, 26 Dec 2006, Ming Zhang wrote: > Hi All > > Result about reserve/release with MPIO. Seems that our way to support > R/R with MPIO has problem. So current code has to be fixed. So my original idea using sid was basicly correct? What about dropped connections? If new session is new I_T nexus then we don't need current iscsi_initiator struct at all and we can survive with older sid idea. This means that my cleanup patch can be forgotten in any case. > Ming Juhani > -------- Forwarded Message -------- > > From: Bla...@em... > > To: bla...@gm..., ip...@ie... > > Subject: RE: [Ips] MPIO with Persistent Reserve Out/In > > Date: Wed, 20 Dec 2006 11:36:54 -0500 > > > > > I can not find this from original rfc or the new guide draft about how > > > MPIO work with Persistent Reserve Out/In or old Reserve/Release. > > > > > > In iSCSI, a session is a I_T nexus. With MPIO, then between one > > > initiator and one target will have multiple I_T nexus. If initiator > > > setup a reservation via 1st session, can it send out commands like > > write > > > via 2nd session? Intuitively, I think it should be able to. > > > > No - if you want that behavior, use multiple TCP connections in a single > > iSCSI session. > > > > > But from SPC, reservation will base on I_T nexus so 2 sessions are 2 > > > different I_T nexus. > > > > That is how iSCSI behaves - no change to SPC. > > > > Thanks, > > --David > > ---------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 176 South St., Hopkinton, MA 01748 > > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > > bla...@em... Mobile: +1 (978) 394-7754 > > ---------------------------------------------------- -- Juhani Rautiainen jr...@ik... |
From: Ming Z. <bla...@gm...> - 2006-12-26 16:24:40
|
On Tue, 2006-12-26 at 16:17 +0200, Juhani Rautiainen wrote: > On Tue, 26 Dec 2006, Ming Zhang wrote: > > > Hi All > > > > Result about reserve/release with MPIO. Seems that our way to support > > R/R with MPIO has problem. So current code has to be fixed. > > So my original idea using sid was basicly correct? What about dropped > connections? If new session is new I_T nexus then we don't need current > iscsi_initiator struct at all and we can survive with older sid idea. This > means that my cleanup patch can be forgotten in any case. > i saw u cleanup patch and then realized i forgot to forward the result to list. i know it is a little bit counter-intuitive, but different I_T nexus will be treated differently in reserve. > > Ming > > Juhani > > -------- Forwarded Message -------- > > > From: Bla...@em... > > > To: bla...@gm..., ip...@ie... > > > Subject: RE: [Ips] MPIO with Persistent Reserve Out/In > > > Date: Wed, 20 Dec 2006 11:36:54 -0500 > > > > > > > I can not find this from original rfc or the new guide draft about how > > > > MPIO work with Persistent Reserve Out/In or old Reserve/Release. > > > > > > > > In iSCSI, a session is a I_T nexus. With MPIO, then between one > > > > initiator and one target will have multiple I_T nexus. If initiator > > > > setup a reservation via 1st session, can it send out commands like > > > write > > > > via 2nd session? Intuitively, I think it should be able to. > > > > > > No - if you want that behavior, use multiple TCP connections in a single > > > iSCSI session. > > > > > > > But from SPC, reservation will base on I_T nexus so 2 sessions are 2 > > > > different I_T nexus. > > > > > > That is how iSCSI behaves - no change to SPC. > > > > > > Thanks, > > > --David > > > ---------------------------------------------------- > > > David L. Black, Senior Technologist > > > EMC Corporation, 176 South St., Hopkinton, MA 01748 > > > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > > > bla...@em... Mobile: +1 (978) 394-7754 > > > ---------------------------------------------------- > > |
From: Ross S. W. W. <rw...@me...> - 2006-12-26 16:20:46
|
So, does this mean that using MPIO with say Microsoft Cluster services = that the cluster services are to respect the reserve/release over = multiple paths from the same initiator? I suspect that they may not = actually do, as Microsoft uses SCSI reserve/release as the basis of = their shared storage, and expect that having the reservation held = per-session instead of per-initiator may break it. This can be analyzed = though with wireshark to see if each session issues a = release->reserve->release for round-robin IO from a single initiator and = if we have a two node cluster if this constant switching of reservations = causes a split brain along the way. I think we should research several vendors concepts of this before = deciding the defacto standard. I think we may find several major vendors = interpreting this differently and picking the one with the widest = deployment is certainly better then the one that is more technically = correct. -Ross -----Original Message----- From: isc...@li... on behalf of Ming = Zhang Sent: Tue 12/26/2006 8:24 AM To: iet-dev Subject: [Iscsitarget-devel] [Fwd: RE: [Ips] MPIO with Persistent = ReserveOut/In] =20 Hi All Result about reserve/release with MPIO. Seems that our way to support R/R with MPIO has problem. So current code has to be fixed. Ming -------- Forwarded Message -------- > From: Bla...@em... > To: bla...@gm..., ip...@ie... > Subject: RE: [Ips] MPIO with Persistent Reserve Out/In > Date: Wed, 20 Dec 2006 11:36:54 -0500 >=20 > > I can not find this from original rfc or the new guide draft about = how > > MPIO work with Persistent Reserve Out/In or old Reserve/Release. > >=20 > > In iSCSI, a session is a I_T nexus. With MPIO, then between one > > initiator and one target will have multiple I_T nexus. If initiator > > setup a reservation via 1st session, can it send out commands like > write > > via 2nd session? Intuitively, I think it should be able to. >=20 > No - if you want that behavior, use multiple TCP connections in a = single > iSCSI session. >=20 > > But from SPC, reservation will base on I_T nexus so 2 sessions are 2 > > different I_T nexus. >=20 > That is how iSCSI behaves - no change to SPC. >=20 > Thanks, > --David > ---------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > bla...@em... Mobile: +1 (978) 394-7754 > ---------------------------------------------------- -------------------------------------------------------------------------= Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share = your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D= DEVDEV _______________________________________________ Iscsitarget-devel mailing list Isc...@li... https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel ______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof. |
From: Ming Z. <bla...@gm...> - 2006-12-26 16:27:08
|
On Tue, 2006-12-26 at 11:20 -0500, Ross S. W. Walker wrote: > > So, does this mean that using MPIO with say Microsoft Cluster services > that the cluster services are to respect the reserve/release over > multiple paths from the same initiator? I suspect that they may not > actually do, as Microsoft uses SCSI reserve/release as the basis of > their shared storage, and expect that having the reservation held > per-session instead of per-initiator may break it. This can be > analyzed though with wireshark to see if each session issues a > release->reserve->release for round-robin IO from a single initiator > and if we have a two node cluster if this constant switching of > reservations causes a split brain along the way. > > I think we should research several vendors concepts of this before > deciding the defacto standard. I think we may find several major > vendors interpreting this differently and picking the one with the > widest deployment is certainly better then the one that is more > technically correct. we have to conform with the RFC. i know it sounds weired if one initiator connect to one target with MPIO (multiple session), one session do reservation and another can not send a write command. :P either use multiple connections per session, or use persistent reserve out/in with reservation key. > > -Ross > > > > -----Original Message----- > From: isc...@li... on behalf of > Ming Zhang > Sent: Tue 12/26/2006 8:24 AM > To: iet-dev > Subject: [Iscsitarget-devel] [Fwd: RE: [Ips] MPIO with Persistent > ReserveOut/In] > > Hi All > > Result about reserve/release with MPIO. Seems that our way to support > R/R with MPIO has problem. So current code has to be fixed. > > Ming > > -------- Forwarded Message -------- > > From: Bla...@em... > > To: bla...@gm..., ip...@ie... > > Subject: RE: [Ips] MPIO with Persistent Reserve Out/In > > Date: Wed, 20 Dec 2006 11:36:54 -0500 > > > > > I can not find this from original rfc or the new guide draft about > how > > > MPIO work with Persistent Reserve Out/In or old Reserve/Release. > > > > > > In iSCSI, a session is a I_T nexus. With MPIO, then between one > > > initiator and one target will have multiple I_T nexus. If > initiator > > > setup a reservation via 1st session, can it send out commands like > > write > > > via 2nd session? Intuitively, I think it should be able to. > > > > No - if you want that behavior, use multiple TCP connections in a > single > > iSCSI session. > > > > > But from SPC, reservation will base on I_T nexus so 2 sessions are > 2 > > > different I_T nexus. > > > > That is how iSCSI behaves - no change to SPC. > > > > Thanks, > > --David > > ---------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 176 South St., Hopkinton, MA 01748 > > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > > bla...@em... Mobile: +1 (978) 394-7754 > > ---------------------------------------------------- > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to > share your > opinions on IT & business topics through brief surveys - and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Iscsitarget-devel mailing list > Isc...@li... > https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel > > > > > ______________________________________________________________________ > This e-mail, and any attachments thereto, is intended only for use by > the addressee(s) named herein and may contain legally privileged > and/or confidential information. If you are not the intended recipient > of this e-mail, you are hereby notified that any dissemination, > distribution or copying of this e-mail, and any attachments thereto, > is strictly prohibited. If you have received this e-mail in error, > please immediately notify the sender and permanently delete the > original and any copy or printout thereof. |
From: Juhani R. <jr...@ik...> - 2006-12-26 16:55:00
|
On Tue, 26 Dec 2006, Ming Zhang wrote: > On Tue, 2006-12-26 at 11:20 -0500, Ross S. W. Walker wrote: > > > > So, does this mean that using MPIO with say Microsoft Cluster services > > that the cluster services are to respect the reserve/release over > > multiple paths from the same initiator? I suspect that they may not > > actually do, as Microsoft uses SCSI reserve/release as the basis of > > their shared storage, and expect that having the reservation held > > per-session instead of per-initiator may break it. This can be > > analyzed though with wireshark to see if each session issues a > > release->reserve->release for round-robin IO from a single initiator > > and if we have a two node cluster if this constant switching of > > reservations causes a split brain along the way. > > > > I think we should research several vendors concepts of this before > > deciding the defacto standard. I think we may find several major > > vendors interpreting this differently and picking the one with the > > widest deployment is certainly better then the one that is more > > technically correct. > > we have to conform with the RFC. i know it sounds weired if one > initiator connect to one target with MPIO (multiple session), one > session do reservation and another can not send a write command. :P > > either use multiple connections per session, or use persistent reserve > out/in with reservation key. I agree that we have to conform to RFC (or could we make it option to break RFC with warnings of course). BTW how do you get multiple connections per session? I don't think that you can use persistent reservations with MS cluster. In Longhorn they are probably needed. Juhani -- Juhani Rautiainen jr...@ik... |
From: Ming Z. <bla...@gm...> - 2006-12-26 19:07:53
|
On Tue, 2006-12-26 at 18:54 +0200, Juhani Rautiainen wrote: > On Tue, 26 Dec 2006, Ming Zhang wrote: > > > On Tue, 2006-12-26 at 11:20 -0500, Ross S. W. Walker wrote: > > > > > > So, does this mean that using MPIO with say Microsoft Cluster services > > > that the cluster services are to respect the reserve/release over > > > multiple paths from the same initiator? I suspect that they may not > > > actually do, as Microsoft uses SCSI reserve/release as the basis of > > > their shared storage, and expect that having the reservation held > > > per-session instead of per-initiator may break it. This can be > > > analyzed though with wireshark to see if each session issues a > > > release->reserve->release for round-robin IO from a single initiator > > > and if we have a two node cluster if this constant switching of > > > reservations causes a split brain along the way. > > > > > > I think we should research several vendors concepts of this before > > > deciding the defacto standard. I think we may find several major > > > vendors interpreting this differently and picking the one with the > > > widest deployment is certainly better then the one that is more > > > technically correct. > > > > we have to conform with the RFC. i know it sounds weired if one > > initiator connect to one target with MPIO (multiple session), one > > session do reservation and another can not send a write command. :P > > > > either use multiple connections per session, or use persistent reserve > > out/in with reservation key. > > I agree that we have to conform to RFC (or could we make it option to > break RFC with warnings of course). BTW how do you get multiple > connections per session? I don't think that you can use persistent > reservations with MS cluster. In Longhorn they are probably needed. MC/S need extra work. I am not familiar with MSCS, I ran out of my windows license #. > > Juhani |
From: Ross S. W. W. <rw...@me...> - 2006-12-26 17:21:08
|
-----Original Message----- From: Juhani Rautiainen [mailto:jr...@ik...] Sent: Tue 12/26/2006 11:54 AM To: Ming Zhang Cc: Ross S. W. Walker; iet-dev Subject: Re: [Iscsitarget-devel] [Fwd: RE: [Ips] MPIO with Persistent = ReserveOut/In] =20 > On Tue, 26 Dec 2006, Ming Zhang wrote: >=20 > > On Tue, 2006-12-26 at 11:20 -0500, Ross S. W. Walker wrote: > > >=20 > > > So, does this mean that using MPIO with say Microsoft Cluster = services > > > that the cluster services are to respect the reserve/release over > > > multiple paths from the same initiator? I suspect that they may = not > > > actually do, as Microsoft uses SCSI reserve/release as the basis = of > > > their shared storage, and expect that having the reservation held > > > per-session instead of per-initiator may break it. This can be > > > analyzed though with wireshark to see if each session issues a > > > release->reserve->release for round-robin IO from a single = initiator > > > and if we have a two node cluster if this constant switching of > > > reservations causes a split brain along the way. > > >=20 > > > I think we should research several vendors concepts of this before > > > deciding the defacto standard. I think we may find several major > > > vendors interpreting this differently and picking the one with the > > > widest deployment is certainly better then the one that is more > > > technically correct. > >=20 > > we have to conform with the RFC. i know it sounds weired if one > > initiator connect to one target with MPIO (multiple session), one > > session do reservation and another can not send a write command. :P > >=20 > > either use multiple connections per session, or use persistent = reserve > > out/in with reservation key. >=20 > I agree that we have to conform to RFC (or could we make it option to=20 > break RFC with warnings of course). BTW how do you get multiple=20 > connections per session? I don't think that you can use persistent=20 > reservations with MS cluster. In Longhorn they are probably needed. =20 Well Ming read and re-read the original RFC and draft RFC and couldn't = find it explicitly mentioned how reservations should be handled = per-initiator, per-session, per-connection, so I wouldn't say that the = way EMC does it is any more RFC compliant as IET currently does it. I guess at this point it would defer to the SCSI/SCSI-2/SCSI-3 RFCs in = how reserve/release should be handled, but how do you translate the = iSCSI session/connection idea to SCSI? I know you have the concept of = initiator and target with SCSI, but I don't think sessions and = connections exist? Maybe that is what EMC is doing, if "sessions" do exist in SCSI they = would take the SCSI reserve/release RFC that reservations should be = per-session and use that in their iSCSI implementation as nothing is = mentioned in the iSCSI RFC, but since any given host on a shared SCSI = bus will have only 1 "session" per-lun at a time, I don't think it = translates very into iSCSI, as any single host in iSCSI can have = multiple sessions to a single lun at any given time. Currently IET does not support multiple connections per-session. If it = did then I suppose it wouldn't really be an issue, we would just change = the procedure to use multiple connections instead of multiple sessions = for MPIO and implement the reserve/release per-session, but right now I = think throughput would be seriously hampered if the initiators had to do = a reserve/release for each MPIO PDU transfer. It would make the R2T = overhead seem like nothing and we will still have to contend with R2T = overhead on top of that. -Ross ______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof. |
From: Juhani R. <jr...@ik...> - 2006-12-26 18:21:27
|
On Tue, 26 Dec 2006, Ross S. W. Walker wrote: > Well Ming read and re-read the original RFC and draft RFC and couldn't > find it explicitly mentioned how reservations should be handled > per-initiator, per-session, per-connection, so I wouldn't say that the > way EMC does it is any more RFC compliant as IET currently does it. RFC-3783 pretty much says that session is I_T nexus (atleast in 3.2). Then again I'm not sure what is the difference in reality between one session with multiple connections vs. multiple sessions. As far as I know there hasn't been any data corruption with current solution. Then again question can be raised if iet's session is same as RFC's session? > I guess at this point it would defer to the SCSI/SCSI-2/SCSI-3 RFCs in > how reserve/release should be handled, but how do you translate the > iSCSI session/connection idea to SCSI? I know you have the concept of > initiator and target with SCSI, but I don't think sessions and > connections exist? They seem to depend on implemantion so basicly RFC makes rules in this behaviour. > Currently IET does not support multiple connections per-session. If it > did then I suppose it wouldn't really be an issue, we would just change > the procedure to use multiple connections instead of multiple sessions > for MPIO and implement the reserve/release per-session, but right now I > think throughput would be seriously hampered if the initiators had to do > a reserve/release for each MPIO PDU transfer. It would make the R2T > overhead seem like nothing and we will still have to contend with R2T > overhead on top of that. Does anyone have idea how multiple connections per session should be done? RFC doesn't see to be much help in this question. Is there something that initiator should do or is this something target should do automatically? If it is latter then what is the real difference between multible connections per session and multiple sessions? I just can't see the difference when it comes to initiators. I agree with your throughput conserns. This was originally why I made those changes to original implementation. > -Ross Juhani -- Juhani Rautiainen jr...@ik... |
From: Ming Z. <bla...@gm...> - 2006-12-26 19:10:48
|
On Tue, 2006-12-26 at 20:21 +0200, Juhani Rautiainen wrote: > On Tue, 26 Dec 2006, Ross S. W. Walker wrote: > > > Well Ming read and re-read the original RFC and draft RFC and couldn't > > find it explicitly mentioned how reservations should be handled > > per-initiator, per-session, per-connection, so I wouldn't say that the > > way EMC does it is any more RFC compliant as IET currently does it. > > > RFC-3783 pretty much says that session is I_T nexus (atleast in 3.2). Then > again I'm not sure what is the difference in reality between one session > with multiple connections vs. multiple sessions. As far as I know there > hasn't been any data corruption with current solution. Then again question > can be raised if iet's session is same as RFC's session? of course, same session concept. > > > I guess at this point it would defer to the SCSI/SCSI-2/SCSI-3 RFCs in > > how reserve/release should be handled, but how do you translate the > > iSCSI session/connection idea to SCSI? I know you have the concept of > > initiator and target with SCSI, but I don't think sessions and > > connections exist? > > They seem to depend on implemantion so basicly RFC makes rules in this > behaviour. have to check scsi spec. i found out this but wish all of you who has interest to read and find out more. see spc321b, p198 "The PERSISTENT RESERVE OUT command (see table 111) is used to request service actions that reserve a logical unit for the exclusive or shared use of a particular I_T nexus." see sam4, page 55, NOTE 4 - The implications of this view of a SCSI initiator device are more far reaching than are immediately apparent (e.g., after a SCSI initiator device makes a persistent exclusive access reservation via one SCSI initiator port, access is denied to the other SCSI initiator port(s) on that same SCSI initiator device). also check 2.1 in iscsi rfc. > > > Currently IET does not support multiple connections per-session. If it > > did then I suppose it wouldn't really be an issue, we would just change > > the procedure to use multiple connections instead of multiple sessions > > for MPIO and implement the reserve/release per-session, but right now I > > think throughput would be seriously hampered if the initiators had to do > > a reserve/release for each MPIO PDU transfer. It would make the R2T > > overhead seem like nothing and we will still have to contend with R2T > > overhead on top of that. > > Does anyone have idea how multiple connections per session should be done? > RFC doesn't see to be much help in this question. Is there something that > initiator should do or is this something target should do automatically? > If it is latter then what is the real difference between multible > connections per session and multiple sessions? I just can't see the > difference when it comes to initiators. I agree with your throughput > conserns. This was originally why I made those changes to > original implementation. both ini and target need extra code to support MC/S. open-iscsi does not support that. MS ini support that. unh-iscsi support it. i think difference between MC/S and multiple session is in error recovery. > > > -Ross > > Juhani |
From: Arne R. <ag...@po...> - 2006-12-26 19:07:22
|
Juhani Rautiainen <jr...@ik...> writes: > On Tue, 26 Dec 2006, Ross S. W. Walker wrote: > >> Well Ming read and re-read the original RFC and draft RFC and couldn't >> find it explicitly mentioned how reservations should be handled >> per-initiator, per-session, per-connection, so I wouldn't say that the >> way EMC does it is any more RFC compliant as IET currently does it. > > > RFC-3783 pretty much says that session is I_T nexus (atleast in 3.2). Yep. > Then > again I'm not sure what is the difference in reality between one session > with multiple connections vs. multiple sessions. As far as I know there > hasn't been any data corruption with current solution. Then again question > can be raised if iet's session is same as RFC's session? It is. >> I guess at this point it would defer to the SCSI/SCSI-2/SCSI-3 RFCs in >> how reserve/release should be handled, but how do you translate the >> iSCSI session/connection idea to SCSI? I know you have the concept of >> initiator and target with SCSI, but I don't think sessions and >> connections exist? Connections are transparent to the SCSI devices and a iSCSI-only thing. No need to translate it. > They seem to depend on implemantion so basicly RFC makes rules in this > behaviour. > >> Currently IET does not support multiple connections per-session. If it >> did then I suppose it wouldn't really be an issue, we would just change >> the procedure to use multiple connections instead of multiple sessions >> for MPIO and implement the reserve/release per-session, but right now I >> think throughput would be seriously hampered if the initiators had to do >> a reserve/release for each MPIO PDU transfer. It would make the R2T >> overhead seem like nothing and we will still have to contend with R2T >> overhead on top of that. > > Does anyone have idea how multiple connections per session should be done? > RFC doesn't see to be much help in this question. Is there something that > initiator should do or is this something target should do > automatically? You might want to re-read the RFC, it does explain all that is required to support it. Basically, it's nothing more than using several TCP connections per session. However, it does have quite some impact on cmnd processing ("connection allegiance", IIRC) and error handling. > If it is latter then what is the real difference between multible > connections per session and multiple sessions? I just can't see the > difference when it comes to initiators. I agree with your throughput > conserns. This was originally why I made those changes to > original implementation. You pretty much summed it up yourself: MC/S == one I_T nexus Multiple sessions == several I_T nexuses. With all the consequences outlined in this thread. HTH, Arne |
From: Ming Z. <bla...@gm...> - 2006-12-26 19:13:03
|
On Tue, 2006-12-26 at 20:07 +0100, Arne Redlich wrote: > Juhani Rautiainen <jr...@ik...> writes: > > > On Tue, 26 Dec 2006, Ross S. W. Walker wrote: > > > >> Well Ming read and re-read the original RFC and draft RFC and couldn't > >> find it explicitly mentioned how reservations should be handled > >> per-initiator, per-session, per-connection, so I wouldn't say that the > >> way EMC does it is any more RFC compliant as IET currently does it. > > > > > > RFC-3783 pretty much says that session is I_T nexus (atleast in 3.2). > > Yep. > > > Then > > again I'm not sure what is the difference in reality between one session > > with multiple connections vs. multiple sessions. As far as I know there > > hasn't been any data corruption with current solution. Then again question > > can be raised if iet's session is same as RFC's session? > > It is. > > >> I guess at this point it would defer to the SCSI/SCSI-2/SCSI-3 RFCs in > >> how reserve/release should be handled, but how do you translate the > >> iSCSI session/connection idea to SCSI? I know you have the concept of > >> initiator and target with SCSI, but I don't think sessions and > >> connections exist? > > Connections are transparent to the SCSI devices and a iSCSI-only > thing. No need to translate it. > > > They seem to depend on implemantion so basicly RFC makes rules in this > > behaviour. > > > >> Currently IET does not support multiple connections per-session. If it > >> did then I suppose it wouldn't really be an issue, we would just change > >> the procedure to use multiple connections instead of multiple sessions > >> for MPIO and implement the reserve/release per-session, but right now I > >> think throughput would be seriously hampered if the initiators had to do > >> a reserve/release for each MPIO PDU transfer. It would make the R2T > >> overhead seem like nothing and we will still have to contend with R2T > >> overhead on top of that. > > > > Does anyone have idea how multiple connections per session should be done? > > RFC doesn't see to be much help in this question. Is there something that > > initiator should do or is this something target should do > > automatically? > > You might want to re-read the RFC, it does explain all that is required > to support it. > > Basically, it's nothing more than using several TCP connections per > session. However, it does have quite some impact on cmnd processing > ("connection allegiance", IIRC) and error handling. > > > If it is latter then what is the real difference between multible > > connections per session and multiple sessions? I just can't see the > > difference when it comes to initiators. I agree with your throughput > > conserns. This was originally why I made those changes to > > original implementation. > > You pretty much summed it up yourself: > MC/S == one I_T nexus > Multiple sessions == several I_T nexuses. With all the consequences outlined in > this thread. and then scsi spec seems to bind reservation to nexus. though i can not figure out why SAME ini can not do reservation in one path and do io in another. that is why i think it is weired. > > HTH, > Arne > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys - and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Iscsitarget-devel mailing list > Isc...@li... > https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel |
From: Arne R. <ag...@po...> - 2006-12-26 19:25:40
|
Ming Zhang <bla...@gm...> writes: > On Tue, 2006-12-26 at 20:07 +0100, Arne Redlich wrote: >> Juhani Rautiainen <jr...@ik...> writes: >> <snip> >> > If it is latter then what is the real difference between multible >> > connections per session and multiple sessions? I just can't see the >> > difference when it comes to initiators. I agree with your throughput >> > conserns. This was originally why I made those changes to >> > original implementation. >> >> You pretty much summed it up yourself: >> MC/S == one I_T nexus >> Multiple sessions == several I_T nexuses. With all the consequences outlined in >> this thread. > > and then scsi spec seems to bind reservation to nexus. Yes, I think so. > though i can not figure out why SAME ini can not do reservation in one > path and do io in another. that is why i think it is weired. Indeed. It's probably a historical thing, i.e. Res/Rel is simply "older" than multipathing? Anyway, it's fixed with persistent reservations based on reservation keys, so we should look into supporting that, I guess? Arne |
From: Ming Z. <bla...@gm...> - 2006-12-26 19:44:23
|
On Tue, 2006-12-26 at 20:25 +0100, Arne Redlich wrote: > Ming Zhang <bla...@gm...> writes: > > > On Tue, 2006-12-26 at 20:07 +0100, Arne Redlich wrote: > >> Juhani Rautiainen <jr...@ik...> writes: > >> > <snip> > > >> > If it is latter then what is the real difference between multible > >> > connections per session and multiple sessions? I just can't see the > >> > difference when it comes to initiators. I agree with your throughput > >> > conserns. This was originally why I made those changes to > >> > original implementation. > >> > >> You pretty much summed it up yourself: > >> MC/S == one I_T nexus > >> Multiple sessions == several I_T nexuses. With all the consequences outlined in > >> this thread. > > > > and then scsi spec seems to bind reservation to nexus. > > Yes, I think so. > > > though i can not figure out why SAME ini can not do reservation in one > > path and do io in another. that is why i think it is weired. > > Indeed. It's probably a historical thing, i.e. Res/Rel is simply > "older" than multipathing? other than this, i can not figure out why. > Anyway, it's fixed with persistent reservations based on reservation > keys, so we should look into supporting that, I guess? not sure but i remember persistent reservation also backward support reservation? if so, then how to handle old application which support only reservation? > > Arne |
From: Juhani R. <jr...@ik...> - 2006-12-26 20:01:47
|
On Tue, 26 Dec 2006, Ming Zhang wrote: > On Tue, 2006-12-26 at 20:25 +0100, Arne Redlich wrote: > > Indeed. It's probably a historical thing, i.e. Res/Rel is simply > > "older" than multipathing? > > other than this, i can not figure out why. > > > Anyway, it's fixed with persistent reservations based on reservation > > keys, so we should look into supporting that, I guess? > > not sure but i remember persistent reservation also backward support > reservation? if so, then how to handle old application which support > only reservation? It's possible to support old reserve/release commands with persistent reservations. There are rules for that. So is consensus that we should fall back old sid based checking for reserve/release? What do you think: Should we rollback all the way to original version or change the current one little bit? > > > > Arne Juhani -- Juhani Rautiainen jr...@ik... |
From: Steffen P. <swp...@am...> - 2006-12-26 20:49:12
|
Hi, If we were to go back to the old SID checking, could anything be done with dropped and reestablished sessions? The original version exposed this issue, one initiator does a reserve, the network cable is unplugged and plugged back in, the reservation could only be cleared by doing a lun reset, because the original SID is gone. Steffen -----Original Message----- From: isc...@li... [mailto:isc...@li...] On Behalf Of Juhani Rautiainen Sent: Tuesday, December 26, 2006 3:02 PM To: Ming Zhang Cc: iet-dev; Arne Redlich Subject: {SPAM?} Re: [Iscsitarget-devel] [Fwd: RE: [Ips] MPIO with Persistent ReserveOut/In] It's possible to support old reserve/release commands with persistent reservations. There are rules for that. So is consensus that we should fall back old sid based checking for reserve/release? What do you think: Should we rollback all the way to original version or change the current one little bit? Juhani --=20 Juhani Rautiainen jr...@ik... ------------------------------------------------------------------------ - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D= DEVDE V _______________________________________________ Iscsitarget-devel mailing list Isc...@li... https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel |
From: Ming Z. <bla...@gm...> - 2006-12-26 20:52:17
|
On Tue, 2006-12-26 at 15:49 -0500, Steffen Plotner wrote: > Hi, > > If we were to go back to the old SID checking, could anything be done > with dropped and reestablished sessions? The original version exposed > this issue, one initiator does a reserve, the network cable is unplugged > and plugged back in, the reservation could only be cleared by doing a > lun reset, because the original SID is gone. guess we have to do reset since a new session is a new nexus. so in this case, MC/S will protect the session from dropped. > > Steffen > > -----Original Message----- > From: isc...@li... > [mailto:isc...@li...] On Behalf Of > Juhani Rautiainen > Sent: Tuesday, December 26, 2006 3:02 PM > To: Ming Zhang > Cc: iet-dev; Arne Redlich > Subject: {SPAM?} Re: [Iscsitarget-devel] [Fwd: RE: [Ips] MPIO with > Persistent ReserveOut/In] > > > It's possible to support old reserve/release commands with persistent > reservations. There are rules for that. So is consensus that we should > fall back old sid based checking for reserve/release? What do you think: > > Should we rollback all the way to original version or change the current > one little bit? > > > Juhani |
From: Juhani R. <jr...@ik...> - 2006-12-26 21:15:40
|
On Tue, 26 Dec 2006, Ming Zhang wrote: > guess we have to do reset since a new session is a new nexus. so in this > case, MC/S will protect the session from dropped. Check my other mail. I'd like to see some proof about that I_T nexus requirement. Juhani -- Juhani Rautiainen jr...@ik... |
From: Ming Z. <bla...@gm...> - 2006-12-26 20:04:13
|
On Tue, 2006-12-26 at 22:01 +0200, Juhani Rautiainen wrote: > On Tue, 26 Dec 2006, Ming Zhang wrote: > > > On Tue, 2006-12-26 at 20:25 +0100, Arne Redlich wrote: > > > > Indeed. It's probably a historical thing, i.e. Res/Rel is simply > > > "older" than multipathing? > > > > other than this, i can not figure out why. > > > > > Anyway, it's fixed with persistent reservations based on reservation > > > keys, so we should look into supporting that, I guess? > > > > not sure but i remember persistent reservation also backward support > > reservation? if so, then how to handle old application which support > > only reservation? > > It's possible to support old reserve/release commands with persistent > reservations. There are rules for that. So is consensus that we should > fall back old sid based checking for reserve/release? What do you think: > Should we rollback all the way to original version or change the current > one little bit? 1st we need to read and figure out which way is correct for reservation. do u think u will have time to go ahead and add persistent reservation? Ming > > > > > > > Arne > > Juhani |
From: Juhani R. <jr...@ik...> - 2006-12-26 21:14:26
|
On Tue, 26 Dec 2006, Ming Zhang wrote: > > 1st we need to read and figure out which way is correct for reservation. > > do u think u will have time to go ahead and add persistent reservation? Can't promise anything. I cheched it few weeks back and it's not as complicated as it used to be in SPC-3 compared to SPC-2 (no more partial reservation et al). Anyway I_T nexus problems are present there. BTW I checked reserve/release commands from SPC-2 and it doesn't speak anything about I_T nexus when it comes to those commands. It speaks about exclusive use by INITIATOR and nothing about I_T nexus. So if you take that as guideline current implemantation is correct. In SPC-3 I_T nexus is mentioned put reserve/relaase are obsoleted. You can still support then but they are are handled like in SPC-2 (with exceptions found in 5.6.3). This makes me wonder where EMC got this I_T nexus requirement? Also I didn't find anything about this from RFC's so I'm bit puzzled. > Ming Juhani -- Juhani Rautiainen jr...@ik... |
From: Ming Z. <bla...@gm...> - 2006-12-26 21:58:46
|
On Tue, 2006-12-26 at 23:14 +0200, Juhani Rautiainen wrote: > On Tue, 26 Dec 2006, Ming Zhang wrote: > > > > > 1st we need to read and figure out which way is correct for reservation. > > > > do u think u will have time to go ahead and add persistent reservation? > > Can't promise anything. I cheched it few weeks back and it's not as > complicated as it used to be in SPC-3 compared to SPC-2 (no more partial > reservation et al). Anyway I_T nexus problems are present there. > > BTW I checked reserve/release commands from SPC-2 and it doesn't speak > anything about I_T nexus when it comes to those commands. It speaks about > exclusive use by INITIATOR and nothing about I_T nexus. So if you take > that as guideline current implemantation is correct. In SPC-3 I_T > nexus is mentioned put reserve/relaase are obsoleted. You can still > support then but they are are handled like in SPC-2 (with exceptions found > in 5.6.3). This makes me wonder where EMC got this I_T nexus requirement? > Also I didn't find anything about this from RFC's so I'm bit puzzled. yes, spc2 does not mention how to bind reservation with nexus. but sam3 mentions this (p50). This standard does not require a SCSI target device to have the ability to detect the presence of a SCSI initiator device with multiple SCSI initiator ports. Therefore, a SCSI target device handles a SCSI initiator device with multiple SCSI initiator ports exactly as it would handle multiple separate SCSI initiator devices (e.g., a SCSI target device handles the configurations shown in figure 21 and figure 22 in exactly the same way it handles the configuration shown in figure 20). NOTE 4 - The implications of this view of a SCSI initiator device are more far reaching than are immediately apparent (e.g., after a SCSI initiator device makes a persistent exclusive access reservation via one SCSI initiator port, access is denied to the other SCSI initiator port(s) on that same SCSI initiator device). > > > Ming > > Juhani |
From: Juhani R. <jr...@ik...> - 2006-12-27 06:13:16
|
On Tue, 26 Dec 2006, Ming Zhang wrote: > yes, spc2 does not mention how to bind reservation with nexus. but sam3 > mentions this (p50). I thought that SAM-2 goes with SPC-2. And SAM-2 really doesn't specify either. Not much SAN's then probably. > This standard does not require a SCSI target device to have the ability > to detect the presence of a SCSI initiator > device with multiple SCSI initiator ports. Therefore, a SCSI target > device handles a SCSI initiator device with > multiple SCSI initiator ports exactly as it would handle multiple > separate SCSI initiator devices (e.g., a SCSI target > device handles the configurations shown in figure 21 and figure 22 in > exactly the same way it handles the configuration > shown in figure 20). > NOTE 4 - The implications of this view of a SCSI initiator device are > more far reaching than are immediately > apparent (e.g., after a SCSI initiator device makes a persistent > exclusive access reservation via one SCSI initiator > port, access is denied to the other SCSI initiator port(s) on that same > SCSI initiator device). It's persistent reservation again. And as you can see MPIO RR problem is still there. Juhani -- Juhani Rautiainen jr...@ik... |
From: Arne R. <ag...@po...> - 2006-12-27 10:07:20
|
Juhani Rautiainen <jr...@ik...> writes: > On Tue, 26 Dec 2006, Ming Zhang wrote: > >> yes, spc2 does not mention how to bind reservation with nexus. but sam3 >> mentions this (p50). > > I thought that SAM-2 goes with SPC-2. And SAM-2 really doesn't specify > either. Not much SAN's then probably. Sorry, but SAM-2 has a statement very similar to the one quoted by Ming - see 4.12.7. Btw., I think we're actually bound to SAM-2 because the iSCSI RFC is based on it. Which leads to the interesting question if/how we can support later SPC or SBC revisions referring to newer SAM revisions. > >> This standard does not require a SCSI target device to have the ability >> to detect the presence of a SCSI initiator >> device with multiple SCSI initiator ports. Therefore, a SCSI target >> device handles a SCSI initiator device with >> multiple SCSI initiator ports exactly as it would handle multiple >> separate SCSI initiator devices (e.g., a SCSI target >> device handles the configurations shown in figure 21 and figure 22 in >> exactly the same way it handles the configuration >> shown in figure 20). >> NOTE 4 - The implications of this view of a SCSI initiator device are >> more far reaching than are immediately >> apparent (e.g., after a SCSI initiator device makes a persistent >> exclusive access reservation via one SCSI initiator >> port, access is denied to the other SCSI initiator port(s) on that same >> SCSI initiator device). > > It's persistent reservation again. And as you can see MPIO RR problem is > still there. No, it's gone IMO, because multiple initiators can participate in a persistent reservation. See SPC-2, 5.5 or SPC-3, 5.6. Arne |
From: Juhani R. <jr...@ik...> - 2006-12-27 11:33:58
|
On Wed, 27 Dec 2006, Arne Redlich wrote: > Juhani Rautiainen <jr...@ik...> writes: > > > Sorry, but SAM-2 has a statement very similar to the one quoted by Ming - see > 4.12.7. You are right. I just checked 4.7 last night. > Btw., I think we're actually bound to SAM-2 because the iSCSI RFC is > based on it. Which leads to the interesting question if/how we can > support later SPC or SBC revisions referring to newer SAM revisions. And if I do persistent reservations they are done SPC-3 way because it's much simpler and cleaner (that's compared to SPC-2). Then again I'm not sure that anyone has used those scope's if they can be so easily dropped. > >> NOTE 4 - The implications of this view of a SCSI initiator device are > >> more far reaching than are immediately > >> apparent (e.g., after a SCSI initiator device makes a persistent > >> exclusive access reservation via one SCSI initiator > >> port, access is denied to the other SCSI initiator port(s) on that same > >> SCSI initiator device). > > > > It's persistent reservation again. And as you can see MPIO RR problem is > > still there. > > No, it's gone IMO, because multiple initiators can participate in a > persistent reservation. See SPC-2, 5.5 or SPC-3, 5.6. I've been reading them many times and it still feels foggy to me. You can multiple I_T nexuses registered but in many places it is said that only one I_T nexus can have reservation. This is confirmed in table 11 in SPC-2 and table 32 in SPC-3. If it's gone other I_T nexuses have to clear it. That is if you use I_T nexus type of reservations. If you use All Registrants type of reservations all registered I_T nexuses (including different cluster nodes) can do whatever they want (except do RESERVE). Funny that SPC-2 has table for allowed SBC commands for different reservations in Annex B but SPC-3 doesn't. So in certain cases MPIO RR is possible but then again there is little protection against possible corruption. I'm starting to think that implementing this is not worth the trouble when somebody writes standard as unclearly as it's done here. > Arne Juhani -- Juhani Rautiainen jr...@ik... |
From: Arne R. <ag...@po...> - 2006-12-27 21:22:17
|
Juhani Rautiainen <jr...@ik...> writes: > On Wed, 27 Dec 2006, Arne Redlich wrote: > <snip> >> No, it's gone IMO, because multiple initiators can participate in a >> persistent reservation. See SPC-2, 5.5 or SPC-3, 5.6. > > I've been reading them many times and it still feels foggy to me. You can > multiple I_T nexuses registered but in many places it is said that only > one I_T nexus can have reservation. This is confirmed in table 11 in SPC-2 > and table 32 in SPC-3. If it's gone other I_T nexuses have to clear it. > That is if you use I_T nexus type of reservations. If you use All > Registrants type of reservations all registered I_T nexuses (including > different cluster nodes) can do whatever they want (except do RESERVE). > Funny that SPC-2 has table for allowed SBC commands for different > reservations in Annex B but SPC-3 doesn't. > > So in certain cases MPIO RR is possible but then again there is little > protection against possible corruption. I'm starting to think that > implementing this is not worth the trouble when somebody writes standard > as unclearly as it's done here. The part of the spec describing reservations is giving me a hard time, too, so I can feel your pain. ;) Arne |