iSCSI Enterprise Target

On Tue, 2007-01-30 at 17:46 -0500, Ross S. W. Walker wrote:
> > -----Original Message-----
> > From: iscsitarget-devel-bounces@... 
> > [mailto:iscsitarget-devel-bounces@...] On 
> > Behalf Of Steve Wray
> > Sent: Tuesday, January 30, 2007 5:25 PM
> > To: iscsitarget-devel@...; 
> > open-iscsi@...
> > Subject: [Iscsitarget-devel] tracking down cause of 
> > filesystem corruption
> > 
> > Hi there,
> > 
> > I am posting this to two mailing lists, I understand that 
> > there will be
> > people on both to whom this may be a bother but I think its 
> > important to
> > get both the initiator and the target perspective on this.
> 
> You should have added xen-devel list too and made it a trifecta.


i would say get rid of xen first. run same version of open-iscsi and iet
and see if any corruption here.

-- 
http://blackmagic02881.wordpress.com/

> -----Original Message-----
> From: Ming Zhang [mailto:blackmagic02881@...] 
> Sent: Tuesday, January 30, 2007 5:59 PM
> To: Ross S. W. Walker
> Cc: Steve Wray; iscsitarget-devel@...; 
> open-iscsi@...
> Subject: Re: [Iscsitarget-devel] tracking down cause of 
> filesystemcorruption
> 
> On Tue, 2007-01-30 at 17:46 -0500, Ross S. W. Walker wrote:
> > > -----Original Message-----
> > > From: iscsitarget-devel-bounces@... 
> > > [mailto:iscsitarget-devel-bounces@...] On 
> > > Behalf Of Steve Wray
> > > Sent: Tuesday, January 30, 2007 5:25 PM
> > > To: iscsitarget-devel@...; 
> > > open-iscsi@...
> > > Subject: [Iscsitarget-devel] tracking down cause of 
> > > filesystem corruption
> > > 
> > > Hi there,
> > > 
> > > I am posting this to two mailing lists, I understand that 
> > > there will be
> > > people on both to whom this may be a bother but I think its 
> > > important to
> > > get both the initiator and the target perspective on this.
> > 
> > You should have added xen-devel list too and made it a trifecta.
> 
> 
> i would say get rid of xen first. run same version of 
> open-iscsi and iet
> and see if any corruption here.

Iet is actually running on a completely different host and is reporting
no errors. It's garbage in, garbage out I bet. The data is getting
corrupted before it reaches iet.

I would be interested in seeing if the domU could run open-iscsi and
mount the volume and work on it without errors, and narrow it down to
where the problem is actually happening a bit, but once again, not
really for this list, better off on the xen list.

-Ross

______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.

Ross S. W. Walker wrote:
[snip]
>> I've been testing iscsi for use in a XEN virtualisation 
>> environment and
>> have been getting pretty bad filesystem corruption after only about 30
>> minutes of use.
> 
> Which version of IET?

root@...# ietd --version
iscsid version 0.4.14


[sip]
>> and the domU sees its root filesystem disappear and hangs.
>>
>> I see no errors in logs on the machine providing the target.
> 
> Sounds a lot like open-iscsi vs. xen.
> 
> I know there have been problems running open-iscsi and iet target on the
> same box which might mean there may be problems with open-iscsi and Xen
> on the same box. You could try going to an iSCSI HBA and remove the
> open-iscsi part. Might be better solution then having both run on same
> host.

Hmmm in this case the xen dom0 with the iscsi target also has open-iscsi.

I've been starting xen instances on the iscsi target dom0 and
live-migrating them to other hosts which have access to the same iscsi
targets. I've also started the domU from dom0's with no iscsi targets.


>> The network interfaces on the initiator and target show no errors, no
>> dropped and no overruns (according to ifconfig).
> 
> No, sounds a lot like page-cache snafu between Xen and open-iscsi, but
> I'm just shooting arrows in the dark.

Could it be a filesystem issue? I've been wondering about the wisdom in
using journaling filesystems over iscsi... I don't know why but
something seems dodgy. I certainly wouldn't want to use XFS or reiser
over iscsi. They could do some really nasty things if the target
disappeared on them (similar to spontaneously losing power).


>> My ietd.conf for the iscsi target used by the domU is attached
>> (passwords changed) as is the iscsid node config (again, 
>> passwords changed).
>>
>>
>> You will notice that this domU has been running spamassassin; 
>> I figured
>> that would give the iscsi layer a good workout since it was running
>> spamassassin for 3 seperate mail servers each with fairly 
>> high throughput.
> 
> iSCSI HBA is starting to sound more like the ticket here...

Sadly thats really not in the budget at this stage... its either
software or nothing.

Ming Zhang wrote:
> On Tue, 2007-01-30 at 17:46 -0500, Ross S. W. Walker wrote:
>>> -----Original Message-----
>>> From: iscsitarget-devel-bounces@... 
>>> [mailto:iscsitarget-devel-bounces@...] On 
>>> Behalf Of Steve Wray
>>> Sent: Tuesday, January 30, 2007 5:25 PM
>>> To: iscsitarget-devel@...; 
>>> open-iscsi@...
>>> Subject: [Iscsitarget-devel] tracking down cause of 
>>> filesystem corruption
>>>
>>> Hi there,
>>>
>>> I am posting this to two mailing lists, I understand that 
>>> there will be
>>> people on both to whom this may be a bother but I think its 
>>> important to
>>> get both the initiator and the target perspective on this.
>> You should have added xen-devel list too and made it a trifecta.
> 
> 
> i would say get rid of xen first. run same version of open-iscsi and iet
> and see if any corruption here.

Thats just not practical; the only reason I am considering using iscsi
at all is for use with xen. I actually don't have any servers suitable
for testing this which arn't running xen...

On Wed, 2007-01-31 at 12:56 +1300, Steve Wray wrote:
> Ming Zhang wrote:
> > On Tue, 2007-01-30 at 17:46 -0500, Ross S. W. Walker wrote:
> >>> -----Original Message-----
> >>> From: iscsitarget-devel-bounces@... 
> >>> [mailto:iscsitarget-devel-bounces@...] On 
> >>> Behalf Of Steve Wray
> >>> Sent: Tuesday, January 30, 2007 5:25 PM
> >>> To: iscsitarget-devel@...; 
> >>> open-iscsi@...
> >>> Subject: [Iscsitarget-devel] tracking down cause of 
> >>> filesystem corruption
> >>>
> >>> Hi there,
> >>>
> >>> I am posting this to two mailing lists, I understand that 
> >>> there will be
> >>> people on both to whom this may be a bother but I think its 
> >>> important to
> >>> get both the initiator and the target perspective on this.
> >> You should have added xen-devel list too and made it a trifecta.
> > 
> > 
> > i would say get rid of xen first. run same version of open-iscsi and iet
> > and see if any corruption here.
> 
> Thats just not practical; the only reason I am considering using iscsi
> at all is for use with xen. I actually don't have any servers suitable
> for testing this which arn't running xen...

it is part of the test. remove non-related component first.

> 
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Iscsitarget-devel mailing list
> Iscsitarget-devel@...
> https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel
-- 
http://blackmagic02881.wordpress.com/

Ming Zhang wrote:
> On Wed, 2007-01-31 at 12:56 +1300, Steve Wray wrote:
>> Ming Zhang wrote:
>>> On Tue, 2007-01-30 at 17:46 -0500, Ross S. W. Walker wrote:
[snip]
>>>
>>> i would say get rid of xen first. run same version of open-iscsi and iet
>>> and see if any corruption here.
>> Thats just not practical; the only reason I am considering using iscsi
>> at all is for use with xen. I actually don't have any servers suitable
>> for testing this which arn't running xen...
> 
> it is part of the test. remove non-related component first.

Yes, I can appreciate that, however under the circumstances I cannot
duplicate the conditions without using virtualisation and xen is all I
have available.

I've tried to reproduce the problem using simple iscsi mounts and
transferring data around but without having a busy machine using iscsi
for its root filesystem and swap space this isn't going to exercise the
problem.

So its not as easy as 'get rid of xen'; I'd have to set up a network
booting machine on physical hardware with iscsi root filesystem and
swap. Thats not as easy for me as you might think.

On Wed, 2007-01-31 at 13:37 +1300, Steve Wray wrote:
> Ming Zhang wrote:
> > On Wed, 2007-01-31 at 12:56 +1300, Steve Wray wrote:
> >> Ming Zhang wrote:
> >>> On Tue, 2007-01-30 at 17:46 -0500, Ross S. W. Walker wrote:
> [snip]
> >>>
> >>> i would say get rid of xen first. run same version of open-iscsi and iet
> >>> and see if any corruption here.
> >> Thats just not practical; the only reason I am considering using iscsi
> >> at all is for use with xen. I actually don't have any servers suitable
> >> for testing this which arn't running xen...
> > 
> > it is part of the test. remove non-related component first.
> 
> Yes, I can appreciate that, however under the circumstances I cannot
> duplicate the conditions without using virtualisation and xen is all I
> have available.
> 
> I've tried to reproduce the problem using simple iscsi mounts and
> transferring data around but without having a busy machine using iscsi
> for its root filesystem and swap space this isn't going to exercise the
> problem.
> 
> So its not as easy as 'get rid of xen'; I'd have to set up a network
> booting machine on physical hardware with iscsi root filesystem and
> swap. Thats not as easy for me as you might think.

ic. maybe u can choose a open-iscsi stable release which is known to
work well with IET and try again with xen.


> 
> 
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Iscsitarget-devel mailing list
> Iscsitarget-devel@...
> https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel
-- 
http://blackmagic02881.wordpress.com/

> -----Original Message-----
> From: iscsitarget-devel-bounces@... 
> [mailto:iscsitarget-devel-bounces@...] On 
> Behalf Of Steve Wray
> Sent: Tuesday, January 30, 2007 6:55 PM
> To: iscsitarget-devel@...
> Cc: open-iscsi@...
> Subject: Re: [Iscsitarget-devel] tracking down cause of 
> filesystem corruption
> 
> Ross S. W. Walker wrote:
> [snip]
> >> I've been testing iscsi for use in a XEN virtualisation 
> >> environment and
> >> have been getting pretty bad filesystem corruption after 
> only about 30
> >> minutes of use.
> > 
> > Which version of IET?
> 
> root@...# ietd --version
> iscsid version 0.4.14
> 
> 
> [sip]
> >> and the domU sees its root filesystem disappear and hangs.
> >>
> >> I see no errors in logs on the machine providing the target.
> > 
> > Sounds a lot like open-iscsi vs. xen.
> > 
> > I know there have been problems running open-iscsi and iet 
> target on the
> > same box which might mean there may be problems with 
> open-iscsi and Xen
> > on the same box. You could try going to an iSCSI HBA and remove the
> > open-iscsi part. Might be better solution then having both 
> run on same
> > host.
> 
> Hmmm in this case the xen dom0 with the iscsi target also has 
> open-iscsi.
> 
> I've been starting xen instances on the iscsi target dom0 and
> live-migrating them to other hosts which have access to the same iscsi
> targets. I've also started the domU from dom0's with no iscsi targets.

So let me understand this correctly. On the host running IET you had run
open-iscsi and mounted the locally exported volumes as iscsi volumes on
the same host. You then ran Xen guests off those mounted iscsi volumes
with no corruption.

Now you mount those volumes on another Xen box, but when you run the Xen
guests off them corruption occurs.

What is different between host1 and host2?


> >> The network interfaces on the initiator and target show no 
> errors, no
> >> dropped and no overruns (according to ifconfig).
> > 
> > No, sounds a lot like page-cache snafu between Xen and 
> open-iscsi, but
> > I'm just shooting arrows in the dark.
> 
> Could it be a filesystem issue? I've been wondering about the 
> wisdom in
> using journaling filesystems over iscsi... I don't know why but
> something seems dodgy. I certainly wouldn't want to use XFS or reiser
> over iscsi. They could do some really nasty things if the target
> disappeared on them (similar to spontaneously losing power).

There is no problem with running a journaled filesystem over iSCSI,
journaled filesystems are the norm. Ext3 and NTFS all run fine over
iSCSI and even stand up well to network interruptions. iSCSI will queue
up pending requests during the disconnect, and as long as the target
comes back before the timeout, all requests are then issued.

> >> My ietd.conf for the iscsi target used by the domU is attached
> >> (passwords changed) as is the iscsid node config (again, 
> >> passwords changed).
> >>
> >>
> >> You will notice that this domU has been running spamassassin; 
> >> I figured
> >> that would give the iscsi layer a good workout since it was running
> >> spamassassin for 3 seperate mail servers each with fairly 
> >> high throughput.
> > 
> > iSCSI HBA is starting to sound more like the ticket here...
> 
> Sadly thats really not in the budget at this stage... its either
> software or nothing.

Ok, well lets look at how the two hosts differ in configuration. Maybe
that will tell us something.

Do these volumes corrupt when mounted on the Xen host and accessed that
way or only when interacting with the Xen guest?

-Ross


______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.

iSCSI really is just the handling of data blocks to and from the hard
disk. Think of it as an extension of the hard disk that goes across the
wire.
 
If you have multiple servers connected to a volume it is exactly like
having a shared SCSI bus between multiple hosts. There is nothing in the
SCSI controllers to prevent you from corrupting your shared volumes.
This kind of control needs to come from the operating system. The OS
needs to know the volume is shared and either a special file-system or
special handling needs to take place. I don't know of a shared
file-system in Windows, but Microsoft Cluster Services allow one server
at a time access to a shared volume using SCSI Reserve/Release.
 
iSCSI must follow the SCSI protocol and the only function available to
aid in shared access is SCSI RESERVE/RELEASE, which is a form of mutex.
 
-Ross
 


________________________________

	From: Bob Zhang [mailto:bob_zhang2004@...] 
	Sent: Tuesday, January 30, 2007 9:12 PM
	To: Scott Kaelin; Ross S. W. Walker; blackmagic02881@...
	Cc: blackmagic02881@...;
iscsitarget-devel@...
	Subject: Re: [Iscsitarget-devel] about mutex between multi-users
to write a same target disk ? for example , multi-users write a same
directory of /dev/hdb1 at the same time
	
	
	thanks very much ! But I can't open this page :
http://en.wikipedia.org/wiki/ISCSI <http://en.wikipedia.org/wiki/ISCSI> 
	 
	Ross and Ming ,do you have any review ? 
	 
	By the way , now I am reading the code of IET , can I find a
good development document ? 
	 
	Thanks ahead , 
	 
	Best Regards ,
	Bob
	 

		----- Original Message ----- 
		From: Scott Kaelin <mailto:skaelin@...>  
		To: Bob Zhang <mailto:bob_zhang2004@...>  
		Cc: blackmagic02881@... ;
iscsitarget-devel@... 
		Sent: Monday, January 29, 2007 9:30 PM
		Subject: Re: [Iscsitarget-devel] about mutex between
multi-users to write a same target disk ? for example , multi-users
write a same directory of /dev/hdb1 at the same time

		Bob,
		It is my understanding that in order to mount the same
target device on multiple initiator machines the file system should be
cluster aware, i.e GFS or it's microsoft counterpart. When multiple
initiators mount the same device, except in read-only mounts, the device
will experience corruption. See http://en.wikipedia.org/wiki/ISCSI
specifically the 'Storage Device' section for why this happens.
		 
		Cheers,
		
		Scott Kaelin
		Sitrof Technologies


______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.

From: "Ross S. W. Walker" <rwalker@...>
Subject: Re: [Iscsitarget-devel] New Block-io Module
Date: Tue, 30 Jan 2007 12:48:13 -0500

> > > > We would need additional counters for that array and additional
> > > > logic,
> > > 
> > > The bio structure has something useful for this purpose. See
> > > blk_rq_map_user() post 2.6.19.
> > 
> > Not really needed, tio already has data in page vectors, this 
> > just takes
> > an unstructured buffer and creates page vector based bio 
> > using zero-copy
> > methodology.
> 
> Let me add that REQ_BLOCK_PC is for page-cache data too. Block-io does
> absolutely no page-caching, doesn't touch it and does no userland memory
> handling so there are no context switches between kernel space and user
> space.
>
> I worked this code on the principle that all caching should be done on
> either the controller side or the initiator side or both, but not in the
> middle. The middle layer should process requests with as close to
> zero-latency as is physically possible.

Sorry, I cannot see what you meant.

I just said the way to handle multiple bios in blk_rq_map_user might
gives some hints.

From: "Ross S. W. Walker" <rwalker@...>
Subject: Re: [Iscsitarget-devel] New Block-io Module
Date: Tue, 30 Jan 2007 13:03:18 -0500

> > -----Original Message-----
> > From: Ming Zhang [mailto:blackmagic02881@...] 
> > Sent: Tuesday, January 30, 2007 11:51 AM
> > To: FUJITA Tomonori
> > Cc: Ross S. W. Walker; iscsitarget-devel@...
> > Subject: Re: [Iscsitarget-devel] New Block-io Module
> > 
> > On Wed, 2007-01-31 at 01:33 +0900, FUJITA Tomonori wrote:
> > > > > > + kfree(tio_work);
> > > > > > +
> > > > > > + return err;
> > > > > > +}
> > > > > 
> > > > > If we set up all bios first and then submit them, the 
> > logic would
> > > be
> > > > > more simple?
> > > > 
> > > > It would be more complex to setup all bios first and would add
> > > latency.
> > > > To do so we would need an array of bio pointers up to the request
> > > size.
> > > 
> > > I'm not sure about this.
> > > 
> > > In most cases, you have only one bio with a decent HBA.
> > > 
> > 
> > fyi, one exception here is drbd. it will lead to multiple bio.
> 
> Actually any initiator request > max_sectors or 128K or some unknown
> variable request sized MD array will generate multiple bios.
> 
> MD arrays will vary their max request size depending on the offset of
> the data in the array, different parts of an array will have different
> request sizes due to the different types of underlying hardware. That is
> why we cannot use max_sectors or max_phys_segments to pre-determine the
> size of the bio and we need to rely on the return code from bio_add_page
> to figure out when that limit has been reached.

I think that dm code properly set the limits with the different types
of underlying hardware (see combine_restrictions_low,
dm_table_set_restrictions, etc). I'm not sure about md code (I think
that the old md code wrongly did, but I've followed it nowadays).

> -----Original Message-----
> From: FUJITA Tomonori [mailto:fujita.tomonori@...] 
> Sent: Tuesday, January 30, 2007 6:32 PM
> To: Ross S. W. Walker
> Cc: fujita.tomonori@...; 
> iscsitarget-devel@...
> Subject: Re: [Iscsitarget-devel] New Block-io Module
> 
> From: "Ross S. W. Walker" <rwalker@...>
> Subject: Re: [Iscsitarget-devel] New Block-io Module
> Date: Tue, 30 Jan 2007 12:48:13 -0500
> 
> > > > > We would need additional counters for that array and 
> additional
> > > > > logic,
> > > > 
> > > > The bio structure has something useful for this purpose. See
> > > > blk_rq_map_user() post 2.6.19.
> > > 
> > > Not really needed, tio already has data in page vectors, this 
> > > just takes
> > > an unstructured buffer and creates page vector based bio 
> > > using zero-copy
> > > methodology.
> > 
> > Let me add that REQ_BLOCK_PC is for page-cache data too. 
> Block-io does
> > absolutely no page-caching, doesn't touch it and does no 
> userland memory
> > handling so there are no context switches between kernel 
> space and user
> > space.
> >
> > I worked this code on the principle that all caching should 
> be done on
> > either the controller side or the initiator side or both, 
> but not in the
> > middle. The middle layer should process requests with as close to
> > zero-latency as is physically possible.
> 
> Sorry, I cannot see what you meant.
> 
> I just said the way to handle multiple bios in blk_rq_map_user might
> gives some hints.

Blk_req_map_user calls bio_map_user, bio_map_user calls
bio_map_user_iov, bio_map_user_iov calls __bio_map_user_iov.

Take a look at lines 643 - 691 in bio.c, function __bio_map_user_iov.
You will see a similarity in that code to mine. Now of course I don't
have to take an unstructured page cache backed user space buffer and
break it down to vectors, I am already given vectors and I don't deal
with page cache, so my code is lower level blk_req_map_user.

http://www.gelato.unsw.edu.au/lxr/source/fs/bio.c#L643

There really isn't too many ways to do this, so all code will look
similar. Like I said off-list, if BIO_MAX_PAGES increases my code will
take that into account, and if a user uses an HBA that can handle really
large requests, my code will take that into account too, so you will see
dramatic increases in performance based on which kernel version and HBA
combo you choose, without modification or tuning of the code.


-Ross

______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.

From: "Ross S. W. Walker" <rwalker@...>
Subject: Re: [Iscsitarget-devel] New Block-io Module
Date: Tue, 30 Jan 2007 19:36:00 -0500

> > -----Original Message-----
> > From: FUJITA Tomonori [mailto:fujita.tomonori@...] 
> > Sent: Tuesday, January 30, 2007 6:32 PM
> > To: Ross S. W. Walker
> > Cc: fujita.tomonori@...; 
> > iscsitarget-devel@...
> > Subject: Re: [Iscsitarget-devel] New Block-io Module
> > 
> > From: "Ross S. W. Walker" <rwalker@...>
> > Subject: Re: [Iscsitarget-devel] New Block-io Module
> > Date: Tue, 30 Jan 2007 12:48:13 -0500
> > 
> > > > > > We would need additional counters for that array and 
> > additional
> > > > > > logic,
> > > > > 
> > > > > The bio structure has something useful for this purpose. See
> > > > > blk_rq_map_user() post 2.6.19.
> > > > 
> > > > Not really needed, tio already has data in page vectors, this 
> > > > just takes
> > > > an unstructured buffer and creates page vector based bio 
> > > > using zero-copy
> > > > methodology.
> > > 
> > > Let me add that REQ_BLOCK_PC is for page-cache data too. 
> > Block-io does
> > > absolutely no page-caching, doesn't touch it and does no 
> > userland memory
> > > handling so there are no context switches between kernel 
> > space and user
> > > space.
> > >
> > > I worked this code on the principle that all caching should 
> > be done on
> > > either the controller side or the initiator side or both, 
> > but not in the
> > > middle. The middle layer should process requests with as close to
> > > zero-latency as is physically possible.
> > 
> > Sorry, I cannot see what you meant.
> > 
> > I just said the way to handle multiple bios in blk_rq_map_user might
> > gives some hints.
> 
> Blk_req_map_user calls bio_map_user, bio_map_user calls
> bio_map_user_iov, bio_map_user_iov calls __bio_map_user_iov.
> 
> Take a look at lines 643 - 691 in bio.c, function __bio_map_user_iov.
> You will see a similarity in that code to mine. Now of course I don't
> have to take an unstructured page cache backed user space buffer and
> break it down to vectors, I am already given vectors and I don't deal
> with page cache, so my code is lower level blk_req_map_user.
> 
> http://www.gelato.unsw.edu.au/lxr/source/fs/bio.c#L643

You don't need to explain me how it works. I'm pretty familiar with it
because I ported Mike's patch to handle multipe bios in
blk_rq_map_user to upstream. And the scsi target infrastructure (that
Mike and I put into mainline) does the similar stuff (handling
multiple bios per SCSI command).

I just said that like blk_rq_map_user and blk_execute_rq do, first
allocating all bios and then submitting them looks a better (simple)
way for me.

> -----Original Message-----
> From: FUJITA Tomonori [mailto:fujita.tomonori@...] 
> Sent: Tuesday, January 30, 2007 6:32 PM
> To: Ross S. W. Walker
> Cc: fujita.tomonori@...; 
> iscsitarget-devel@...
> Subject: Re: [Iscsitarget-devel] New Block-io Module
> 
> From: "Ross S. W. Walker" <rwalker@...>
> Subject: Re: [Iscsitarget-devel] New Block-io Module
> Date: Tue, 30 Jan 2007 12:16:32 -0500
> 
> > > > > If we set up all bios first and then submit them, the 
> > > logic would be
> > > > > more simple?
> > > > 
> > > > It would be more complex to setup all bios first and would 
> > > add latency.
> > > > To do so we would need an array of bio pointers up to the 
> > > request size.
> > > 
> > > I'm not sure about this.
> > > 
> > > In most cases, you have only one bio with a decent HBA.
> > 
> > Not true, BIO_MAX_PAGES = 32 which = 128K,
> 
> AFAIK, BIO_MAX_PAGES is 256.

Ok, my bad I just looked at the headers again and you are correct it is
256. I don't know why I thought it was 32, something else I was looking
up at the time probably. So bio request side primarily limited by device
capability.

> > you can of course have
> > request sizes > then 128K from initiator, most high end HBA's limit
> > their max_sector size to 128K, some are adjustable, but 
> Linux limits the
> 
> Decent HBAs has no such limit.

Then my code against those HBAs should provide substantial performance
gains for large request sizes as it will be able to get the whole
request up to 1MB request size into the bio.

> > bio size anyways. The current implementation will work both 
> ways though.
> > If Linux increases BIO_MAX_PAGES to 64 and HBA's 
> max_sectors are upped
> > to match, our code will just start sending 256K sized bios, 
> we can still
> > get 1M, 2M, 4M... request sizes from initiator though so we 
> will still
> > need multiple bios.
> > 
> > > > We would need additional counters for that array and additional
> > > > logic,
> > > 
> > > The bio structure has something useful for this purpose. See
> > > blk_rq_map_user() post 2.6.19.
> > 
> > Not really needed, tio already has data in page vectors, 
> this just takes
> > an unstructured buffer and creates page vector based bio 
> using zero-copy
> > methodology.
> > 
> > The current implementation has gone through several 
> variations on this
> > and the current version has been tested as the quickest 
> implementation
> > so far.
> 
> I don't buy it until I see the performance comparison.

I can provide a spreadsheet of this between file-io, old block-io and
new block-io. The old code was posted on the list not too long ago. The
old code performed very poorly with drbd, it submitted and waited on
each bio, and crashed with MD, it's use of max_sectors to determine bio
request size breaks on variably sectored devices.

> > I don't think the logic can get any simpler then, build 
> bio, submit to
> > queue, build bio, submit to queue, when request is 
> complete, kick the
> > queue.
> 
> Not sure.

At this low level... Find me an algorithm or function that can do it
better (and not just abstract the same thing) and I'll use it. Of course
I want it to run the best it can...

> > > > This leads into another issue I am experiencing. The 
> > > majority of Linux
> > > > Enterprise users are running a stable distribution 
> based on earlier
> > > > kernels, 2.6.9 the most prevailent. These are going to 
> be the target
> > > > users of IET as most home users will not have a need for 
> > > iSCSI, yet we
> > > > do not maintain our code for these users.
> > > 
> > > In the first place, enterprise users don't install out-of-mainline
> > > kernel modules, I think. iSCSI target feature should be 
> provided by
> > > default (like Solaris).
> > 
> > Only if they are not provided upstream and the feature is 
> needed would
> > enterprise users (myself) build from scratch.
> > 
> > Only the bleeding edge distros carry IET in their repositories right
> > now.
> > 
> > > There is no chance to have IET in mainline (for 
> reasonable reasons) so
> > > that RHEL, CentOS, feadora, Debian and etc will not include
> > > IET. That's the main reason why I've been working on another iSCSI
> > > implementation, tgt.
> > 
> > I don't think I followed the argument in the first 
> sentence, but if I
> > did then the second would make sense.
> 
> IET will not be merged into upstream. Thus the majority of
> distributions will not include IET.

Does IET have an upstream? I thought Ardis was discontinued?

I know Suse is starting to include it in their distro.

It is a candidate for RHEL5, so eventually it will be carried by a lot.
Just not today.

-Ross



______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.

From: "Ross S. W. Walker" <rwalker@...>
Subject: Re: [Iscsitarget-devel] New Block-io Module
Date: Tue, 30 Jan 2007 19:54:00 -0500

> > > I don't think I followed the argument in the first 
> > sentence, but if I
> > > did then the second would make sense.
> > 
> > IET will not be merged into upstream. Thus the majority of
> > distributions will not include IET.
> 
> Does IET have an upstream? I thought Ardis was discontinued?

As I have said multiple times, IET will never be merged into upstream
Linux kernel.


> I know Suse is starting to include it in their distro. 

As far as I know, only SUSE includes IET.


> It is a candidate for RHEL5, so eventually it will be carried by a
> lot.  Just not today.

As far as I know, RHEL will never include IET.

> -----Original Message-----
> From: FUJITA Tomonori [mailto:fujita.tomonori@...] 
> Sent: Tuesday, January 30, 2007 8:11 PM
> To: Ross S. W. Walker
> Cc: fujita.tomonori@...; 
> iscsitarget-devel@...
> Subject: Re: [Iscsitarget-devel] New Block-io Module
> 
> From: "Ross S. W. Walker" <rwalker@...>
> Subject: Re: [Iscsitarget-devel] New Block-io Module
> Date: Tue, 30 Jan 2007 19:54:00 -0500
> 
> > > > I don't think I followed the argument in the first 
> > > sentence, but if I
> > > > did then the second would make sense.
> > > 
> > > IET will not be merged into upstream. Thus the majority of
> > > distributions will not include IET.
> > 
> > Does IET have an upstream? I thought Ardis was discontinued?
> 
> As I have said multiple times, IET will never be merged into upstream
> Linux kernel.

Oh, it never crossed my mind that it would be included into the Linux
kernel. Not everything even needs to be included into the kernel. That
doesn't mean anything anyways, a lot of software that uses kernel
drivers stand very well on their own. Take drbd for example, I can
download the kernel module in CentOS extras nobody needs drbd integrated
into the kernel.

> > I know Suse is starting to include it in their distro. 
> 
> As far as I know, only SUSE includes IET.

If SUSE has it others will too. Users want it, demand is increasing,
distros will carry it.

> > It is a candidate for RHEL5, so eventually it will be carried by a
> > lot.  Just not today.
> 
> As far as I know, RHEL will never include IET.

Would you care to set a wager... Maybe not in main, but extras... I can
definitely see it.


______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.

> -----Original Message-----
> From: FUJITA Tomonori [mailto:fujita.tomonori@...] 
> Sent: Tuesday, January 30, 2007 6:32 PM
> To: Ross S. W. Walker
> Cc: blackmagic02881@...; fujita.tomonori@...; 
> iscsitarget-devel@...
> Subject: Re: [Iscsitarget-devel] New Block-io Module
> 
> From: "Ross S. W. Walker" <rwalker@...>
> Subject: Re: [Iscsitarget-devel] New Block-io Module
> Date: Tue, 30 Jan 2007 13:03:18 -0500
> 
> > > -----Original Message-----
> > > From: Ming Zhang [mailto:blackmagic02881@...] 
> > > Sent: Tuesday, January 30, 2007 11:51 AM
> > > To: FUJITA Tomonori
> > > Cc: Ross S. W. Walker; iscsitarget-devel@...
> > > Subject: Re: [Iscsitarget-devel] New Block-io Module
> > > 
> > > On Wed, 2007-01-31 at 01:33 +0900, FUJITA Tomonori wrote:
> > > > > > > + kfree(tio_work);
> > > > > > > +
> > > > > > > + return err;
> > > > > > > +}
> > > > > > 
> > > > > > If we set up all bios first and then submit them, the 
> > > logic would
> > > > be
> > > > > > more simple?
> > > > > 
> > > > > It would be more complex to setup all bios first and would add
> > > > latency.
> > > > > To do so we would need an array of bio pointers up to 
> the request
> > > > size.
> > > > 
> > > > I'm not sure about this.
> > > > 
> > > > In most cases, you have only one bio with a decent HBA.
> > > > 
> > > 
> > > fyi, one exception here is drbd. it will lead to multiple bio.
> > 
> > Actually any initiator request > max_sectors or 128K or some unknown
> > variable request sized MD array will generate multiple bios.
> > 
> > MD arrays will vary their max request size depending on the 
> offset of
> > the data in the array, different parts of an array will 
> have different
> > request sizes due to the different types of underlying 
> hardware. That is
> > why we cannot use max_sectors or max_phys_segments to 
> pre-determine the
> > size of the bio and we need to rely on the return code from 
> bio_add_page
> > to figure out when that limit has been reached.
> 
> I think that dm code properly set the limits with the different types
> of underlying hardware (see combine_restrictions_low,
> dm_table_set_restrictions, etc). I'm not sure about md code (I think
> that the old md code wrongly did, but I've followed it nowadays).


Here is an email exchange with Jens Axboe on the problems I experienced
with the old block-io code:

On Wed, Jan 24 2007, Ross S. W. Walker wrote:
> Jens,
> 
> I have invited you into this conversation as the authoritative expert
on
> the current implementation of Linux's generic block layer. We are
hoping
> some light could be shed on this problem.
> 
> I am having a problem with the code that calculates max request size
in
> blockio_make_request, it doesn't do so accurately for all devices. I
> found this when testing out against an lvm volume on an md raid1
running
> on top of a pair of SATA drives. The drives report themselves as
> supported max_sectors/max_hw_sectors = 255, the system max. This
> translated into 31 pages and the physical and hardware segment
> constraints allowed this. Of course it must have been lying as the
> kernel panic'd, so there must be a more accurate way to calculate
this.
> 
> 	if (bdev_q) {
> 		if (bdev_q->max_hw_sectors > 0) {
> 			max_pages = (bdev_q->max_hw_sectors <<
> volume->blk_shift) >> PAGE_SHIFT;
> 		}
> 		if (max_pages > bdev_q->max_phys_segments) {
> 			max_pages = bdev_q->max_phys_segments;
> 		}
> 		if (max_pages > bdev_q->max_hw_segments) {
> 			max_pages = bdev_q->max_hw_segments;
> 		}
> 		if (max_pages > tio->pg_cnt) {
> 			max_pages = tio->pg_cnt;
> 		}
> 	} else {
> 		max_pages = 1;
> 	}

You can't calculate it like this, it's not a static limit. Sometimes the
size will depend on the device offset, that is the case for eg raid with
stripes and such. That is why we have the bio_add_page() and
->merge_bvec_fn() apis.

-- 
Jens Axboe
--------------------------------------

I took this as saying the only reliable way to determine when the max
request size of a bio has been reached is to examine the return code
from bio_add_page because different devices within a stripe set can have
different max_sectors depending on their driver implementation and the
underlying hardware.

-Ross

______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.

> -----Original Message-----
> From: FUJITA Tomonori [mailto:fujita.tomonori@...] 
> Sent: Tuesday, January 30, 2007 7:53 PM
> To: Ross S. W. Walker
> Cc: fujita.tomonori@...; 
> iscsitarget-devel@...
> Subject: Re: [Iscsitarget-devel] New Block-io Module
> 
> From: "Ross S. W. Walker" <rwalker@...>
> Subject: Re: [Iscsitarget-devel] New Block-io Module
> Date: Tue, 30 Jan 2007 19:36:00 -0500
> 
> > > -----Original Message-----
> > > From: FUJITA Tomonori [mailto:fujita.tomonori@...] 
> > > Sent: Tuesday, January 30, 2007 6:32 PM
> > > To: Ross S. W. Walker
> > > Cc: fujita.tomonori@...; 
> > > iscsitarget-devel@...
> > > Subject: Re: [Iscsitarget-devel] New Block-io Module
> > > 
> > > From: "Ross S. W. Walker" <rwalker@...>
> > > Subject: Re: [Iscsitarget-devel] New Block-io Module
> > > Date: Tue, 30 Jan 2007 12:48:13 -0500
> > > 
> > > > > > > We would need additional counters for that array and 
> > > additional
> > > > > > > logic,
> > > > > > 
> > > > > > The bio structure has something useful for this purpose. See
> > > > > > blk_rq_map_user() post 2.6.19.
> > > > > 
> > > > > Not really needed, tio already has data in page vectors, this 
> > > > > just takes
> > > > > an unstructured buffer and creates page vector based bio 
> > > > > using zero-copy
> > > > > methodology.
> > > > 
> > > > Let me add that REQ_BLOCK_PC is for page-cache data too. 
> > > Block-io does
> > > > absolutely no page-caching, doesn't touch it and does no 
> > > userland memory
> > > > handling so there are no context switches between kernel 
> > > space and user
> > > > space.
> > > >
> > > > I worked this code on the principle that all caching should 
> > > be done on
> > > > either the controller side or the initiator side or both, 
> > > but not in the
> > > > middle. The middle layer should process requests with 
> as close to
> > > > zero-latency as is physically possible.
> > > 
> > > Sorry, I cannot see what you meant.
> > > 
> > > I just said the way to handle multiple bios in 
> blk_rq_map_user might
> > > gives some hints.
> > 
> > Blk_req_map_user calls bio_map_user, bio_map_user calls
> > bio_map_user_iov, bio_map_user_iov calls __bio_map_user_iov.
> > 
> > Take a look at lines 643 - 691 in bio.c, function 
> __bio_map_user_iov.
> > You will see a similarity in that code to mine. Now of 
> course I don't
> > have to take an unstructured page cache backed user space buffer and
> > break it down to vectors, I am already given vectors and I 
> don't deal
> > with page cache, so my code is lower level blk_req_map_user.
> > 
> > http://www.gelato.unsw.edu.au/lxr/source/fs/bio.c#L643
> 
> You don't need to explain me how it works. I'm pretty familiar with it
> because I ported Mike's patch to handle multipe bios in
> blk_rq_map_user to upstream. And the scsi target infrastructure (that
> Mike and I put into mainline) does the similar stuff (handling
> multiple bios per SCSI command).
> 
> I just said that like blk_rq_map_user and blk_execute_rq do, first
> allocating all bios and then submitting them looks a better (simple)
> way for me.

Well, when I looked at it I figured if the request size was so big to
fill 1 bio and go on to another, then what do we do? Continue to hang on
to the filled bio for the other(s) to fill, or how about just dropping
it on the queue so the device can start processing it if it wants to.
The queue is there, the bio is ready, why wait, submit_bio doesn't
block, and allows me to go on to the next and maybe the queue will
unplug in the meantime and start processing, which will give it a head
start.

I suppose I could kick the queue on each submission... But my idea was
to just keep dropping them on the queue and use some elevator merge
action to possibly squeeze a little more performance and possibly
improve read starvation a bit.

-Ross

______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.

On Tue, 2007-01-30 at 23:28 +0100, Arne Redlich wrote:
> Ming Zhang <blackmagic02881@...> writes:
> 
> > On Tue, 2007-01-30 at 22:26 +0100, Arne Redlich wrote:
> >> Hi Ming,
> >> 
> >> Ming Zhang <blackmagic02881@...> writes:
> >> 
> >> > Hi All
> >> >
> >> > Wonder if we need a unstable branch for IET. So we can keep the
> >> > stability of trunk while absorb more unstable patches into iet and allow
> >> > more people to test. Otherwise patches are buried easily.
> >> 
> >> Could you elaborate a bit more on how that is supposed to improve our workflow?
> >> Or do you think of using separate svn repositories / git trees?
> >
> > i meant create a branch in same svn repos. let all not that bad patches
> > get merged first. people know it is test branch so only play with it
> > instead of running production system. once the feature is validated and
> > bugs are sort out, we merge that to trunk. so trunk is always stable.
> 
> I see. However, I think trunk is already the right place for it:
> - We don't have to merge code back and forth between trunk and a
> unstable branch (which would certainly be a major PITA for Tomo, I
> guess).

yes, this need some work, but not that much. if he agree to merge one
from branch to trunk, there will always be some people, or most time the
original patch author will do that.

> - Those wo are willing to test-drive our development code just have to
> look at trunk.
> - Those who want stable releases should use the released tarballs, and,
> judging from the messages on this list, already do so). I don't think
> anyone looking for stable code will look in the project's development
> repository for it. ;-)

our trunk is stable enough, sometime even better than release code.


> 
> Arne
-- 
http://blackmagic02881.wordpress.com/

From: Arne Redlich <agr@...>
Subject: Re: [Iscsitarget-devel] [RFC] unstable branch
Date: Tue, 30 Jan 2007 23:28:04 +0100

> Ming Zhang <blackmagic02881@...> writes:
> 
> > On Tue, 2007-01-30 at 22:26 +0100, Arne Redlich wrote:
> >> Hi Ming,
> >> 
> >> Ming Zhang <blackmagic02881@...> writes:
> >> 
> >> > Hi All
> >> >
> >> > Wonder if we need a unstable branch for IET. So we can keep the
> >> > stability of trunk while absorb more unstable patches into iet and allow
> >> > more people to test. Otherwise patches are buried easily.
> >> 
> >> Could you elaborate a bit more on how that is supposed to improve our workflow?
> >> Or do you think of using separate svn repositories / git trees?
> >
> > i meant create a branch in same svn repos. let all not that bad patches
> > get merged first. people know it is test branch so only play with it
> > instead of running production system. once the feature is validated and
> > bugs are sort out, we merge that to trunk. so trunk is always stable.
> 
> I see. However, I think trunk is already the right place for it:
> - We don't have to merge code back and forth between trunk and a
> unstable branch (which would certainly be a major PITA for Tomo, I
> guess).
> - Those wo are willing to test-drive our development code just have to
> look at trunk.
> - Those who want stable releases should use the released tarballs, and,
> judging from the messages on this list, already do so). I don't think
> anyone looking for stable code will look in the project's development
> repository for it. ;-)

Agreed. I don't think we need another branch (with more effort to
maintain it).

Just putting an experimental patches on the sf project site (or the
Wiki site) should work. The latter is better because everyone can do
that. However, as I said before, I have no idea how to setup MediaWiki
with Captcha. The Wiki site will not be available until someone tell
me how to do that.

On Wed, 2007-01-31 at 08:41 +0900, FUJITA Tomonori wrote:
> From: Arne Redlich <agr@...>
> Subject: Re: [Iscsitarget-devel] [RFC] unstable branch
> Date: Tue, 30 Jan 2007 23:28:04 +0100
> 
> > Ming Zhang <blackmagic02881@...> writes:
> > 
> > > On Tue, 2007-01-30 at 22:26 +0100, Arne Redlich wrote:
> > >> Hi Ming,
> > >> 
> > >> Ming Zhang <blackmagic02881@...> writes:
> > >> 
> > >> > Hi All
> > >> >
> > >> > Wonder if we need a unstable branch for IET. So we can keep the
> > >> > stability of trunk while absorb more unstable patches into iet and allow
> > >> > more people to test. Otherwise patches are buried easily.
> > >> 
> > >> Could you elaborate a bit more on how that is supposed to improve our workflow?
> > >> Or do you think of using separate svn repositories / git trees?
> > >
> > > i meant create a branch in same svn repos. let all not that bad patches
> > > get merged first. people know it is test branch so only play with it
> > > instead of running production system. once the feature is validated and
> > > bugs are sort out, we merge that to trunk. so trunk is always stable.
> > 
> > I see. However, I think trunk is already the right place for it:
> > - We don't have to merge code back and forth between trunk and a
> > unstable branch (which would certainly be a major PITA for Tomo, I
> > guess).
> > - Those wo are willing to test-drive our development code just have to
> > look at trunk.
> > - Those who want stable releases should use the released tarballs, and,
> > judging from the messages on this list, already do so). I don't think
> > anyone looking for stable code will look in the project's development
> > repository for it. ;-)
> 
> Agreed. I don't think we need another branch (with more effort to
> maintain it).
> 
> Just putting an experimental patches on the sf project site (or the
> Wiki site) should work. The latter is better because everyone can do
> that. However, as I said before, I have no idea how to setup MediaWiki
> with Captcha. The Wiki site will not be available until someone tell
> me how to do that.

ok, as a temporary fix, can u enable the patch tracker on sf site so
people can use it?


-- 
http://blackmagic02881.wordpress.com/