iSCSI Enterprise Target

>You,Carl, said: 
>"I have some experiences on storage, storage area networks, databases and
>other commercial systems used for handling real workload. But if you >don't
need such experience I won't bother here anymore."
>Fine, we need all experience there is out there! But why don´t offer it to
>us in a little bit humble way, not try to force your solutions on us?

Sorry, it never was my intention to force any solution on you. I wanted to
share my experiences in real workload enterprise computing with you for
helping further development of IET, unfortunately, I'm no real programmer,
so I cannot supplying any patches to you.

>It seems to me, that you need iet so badly, and the way you want it, that
>you try to force the developers to go your way.

Nope, I never needed it, just interested in technology and usability. And
after all discussion I'm knowing some things more than before. So my time
here wasn't useless, at least for me.

>You also seem to have all the answers, and on everything. I have 
>difficulties to trust people who have all the answers. 

Never claimed fully wisdom, I'm sorry if I leaved this impression.

>We want a nice and constructive discussion climate on the list, and if 
>you can live with that, please be wellcome to join us. But if you need to 
>bulley us around, do it some other place!

If I bothered anyone here - sorry, wasn't my intention. 

-- 
NEU +++ DSL Komplett von GMX +++ http://www.gmx.net/de/go/dsl
GMX DSL-Netzanschluss + Tarif zum supergünstigen Komplett-Preis!

From: FUJITA Tomonori <tomof@...>
Subject: Re: [Iscsitarget-devel] Write caching - subjective summary
Date: Wed, 27 Oct 2004 10:39:34 +0900

> > XFS is more mature than ext2/ext3. AFAIK XFS was designed on journaling and 
> > not like ext3 on a already existing filesystem. Probably the designers have 
> > taken the caching issues into design.
> 
> I quato Stephen Lord's mail in the XFS mailing list. He is a SGI XFS developer.
> 
> http://oss.sgi.com/archives/linux-xfs/2002-11/msg00057.html

I found his more informative quotation.

http://oss.sgi.com/archives/linux-xfs/2001-07/msg00246.html

--
> I've asked about write caching before, and now I've got another
> question.
> 
> If I've got write caching turned on, and the power goes out, do
> I face the possibility of silent metadata corruption?
> 
> With ext2, the fsck would attempt to fix metadata problems on
> bootup; on the other hand, with xfs, if write caching resulted
> in a corrupted log, would the simple log replay that xfs does on
> mount perhaps fail to catch possible metadata problems?

Yes, and the same will be true for all write behind journalled filesystems,
if the log does not make it to disk when the driver says it has then in
theory a subsequent metadata write could make it out to the media. A crash
at this point would mean that part of a transaction had made it to disk,
but the rest which only existed in the disks internal cache would not.

I would expect reiserfs, ext3 and jfs would all have the same issue.

So I would not in general recommend write caching on a device, unless
you know enough about it to be satisfied that its cache makes it out
to disk on power down. Using the rotational power of the spindle to
generate power to move the head to a special track and flush the cache
is not unheard of for instance.
---

If you are serious about your data, don't use 'write back with
reorder' policy with a disk drive that does not have something like a
battery to flush disk cache in the presence of a system crash.

Hi,

> "So I would not in general recommend write caching on a device, unless
> you know enough about it to be satisfied that its cache makes it out
> to disk on power down. Using the rotational power of the spindle to
> generate power to move the head to a special track and flush the cache
> is not unheard of for instance."

a) You can force unbuffered writes if you need it and there are some systems
out there doing that. I'll get some SAN logs the next few hours.
b) A harddisk reorders the buffered data before writing it. 
c) Operating write back caching without battery backup was always a risk. d)
Everyone who claims operation enterprise computing without backup systems is
a fool.

I don't see a problem here. If some filesystems rely on ordered write back
behaviour they are crap. You don't get ordered write back on today's
harddisks, that's fact (ask Hitachi, Seagate, Maxtor and so on). And no
professional sysadmin could operate his storages write through, the
performance pentalty is just too big. 

To be really safe, be sure to turn off all write buffers anywhere in the io
process. 

Unless IET is strictly write through it will never play in the league of
enterprise computing. You have to offer features of typical SCSI drives and
this means, offering unbuffered and buffered writes on request. 

bye
Carl

-- 
Geschenkt: 3 Monate GMX ProMail + 3 Ausgaben der TV Movie mit DVD
++++ Jetzt anmelden und testen http://www.gmx.net/de/go/mail ++++

>Unless IET is strictly write through it will never play in the league of
>enterprise computing. You have to offer features of typical SCSI drives and
>this means, offering unbuffered and buffered writes on request. 
>  
>
Question is, how does this "unbuffered/buffered write request" look like 
(which SCSI command?) and if we should use by default buffered (Carl 
approach?) or unbuffered (Fujita approach?)


-- 

Libor Vanek


+-------------------------------------+
| Email:    libor@...            |
| ICQ:      124529939                 |
| WWW:      http://www.discobolos.net |
| Tel/fax:  +420 541 22 5091, 6293    |
| Mobil:    +420 777 703 642          |
+-------------------------------------+

Hi,

> Question is, how does this "unbuffered/buffered write request" look like 
> (which SCSI command?) and if we should use by default buffered (Carl 
> approach?) or unbuffered (Fujita approach?)

For a untagged queueing SCSI device it's really simple: Let the initiator
choose if he wants to write buffered (simple SCSI WRITE), write unbuffered
(SCSI WRITE with FUA bit set) or flushing the cache (SYNCHRONIZE CACHE).

Typical access pattern found in my SAN traces are: Writing buffered and
synchronize a particular range (SYNCHRONIZE CACHE supports syncing a range
only). This is common between the traced OS/filesystems.

As long as IET doesn't reorder the SCSI commands received there's AFAIK no
problem.

If you're using initiators not supporting the neccessary SCSI command set or
using journaling filesystems which aren't using these features you have to
set write through, else you can safely use write back. 

BTW: Newer 2.6 kernels are supporting write barriers for reiser & ext3.

bye
Carl

-- 
NEU +++ DSL Komplett von GMX +++ http://www.gmx.net/de/go/dsl
GMX DSL-Netzanschluss + Tarif zum supergünstigen Komplett-Preis!

From: "Carl Rueder" <carl.rueder@...>
Subject: Re: [Iscsitarget-devel] Write caching - subjective summary
Date: Wed, 27 Oct 2004 09:41:26 +0200 (MEST)

> > "So I would not in general recommend write caching on a device, unless
> > you know enough about it to be satisfied that its cache makes it out
> > to disk on power down. Using the rotational power of the spindle to
> > generate power to move the head to a special track and flush the cache
> > is not unheard of for instance."
> 
> a) You can force unbuffered writes if you need it and there are some systems
> out there doing that. I'll get some SAN logs the next few hours.
> b) A harddisk reorders the buffered data before writing it. 
> c) Operating write back caching without battery backup was always a risk. d)
> Everyone who claims operation enterprise computing without backup systems is
> a fool.

With write back caching without battery backup, you may lose some of
your data.

With write back caching with reordering without battery backup, you
may lose the whole your file system.

Please read Stephen Lord's mail more carefully. He said that
write back may corrupt file system metadata.

Note that he said that write-back may corrupt file system metadata,
however, write-back preserves the ordering is safe for modern
journaling file systems, I think.


> I don't see a problem here. If some filesystems rely on ordered write back
> behaviour they are crap. You don't get ordered write back on today's
> harddisks, that's fact (ask Hitachi, Seagate, Maxtor and so on). And no
> professional sysadmin could operate his storages write through, the
> performance pentalty is just too big. 

Why is it so difficult for you to understand the fact that some
journaling file systems in Linux don't support write back caching with
reordering?

Please don't insult file system developers by saying that they are
crap. Your theory says that ext3 was crap (I showed you Stephen
C. Tweedie's mail) and XFS was crap (I showed you Stephen Lord's
mail). Note that you thought that XFS is designed to handle write-back
with reorder.

You don't know anything about file systems and Linux I/O design. Why
are you so rude?


And you said,

XFS is more mature than ext2/ext3. AFAIK XFS was designed on journaling and 
not like ext3 on a already existing filesystem. Probably the designers have 
taken the caching issues into design.

Again, how could you said such thing even though you don't know
anything about XFS and ext3 implementations in Linux.


I think that you have some experience and read white papers about
these file systems, however, how can you be sure that they are
true? You can find truth only in the source code.

Don't insult or criticize people by only your assumption. As I showed,
you know no truth about file system implementations.


> Unless IET is strictly write through it will never play in the league of
> enterprise computing. You have to offer features of typical SCSI drives and
> this means, offering unbuffered and buffered writes on request. 

You don't need to say "You have to do something.". You have to do what
you want for yourself.

If IET is not useful for you, please go away and find another solution.

Hi,

> I don't know IET so much in details but I'm afraid that correct 
> implementation of this requires to wait for implementation of SCSI 
> "middleware" (which is needed also for virtual tape etc.).

That's part of the OS, the fileio must only use the corresponding functions.
And the SCSI interface must offer the appropriate commands.

> I don't think there is reason in IET to reorder SCSI commands.

Right. It acts as a untagged queued SCSI device.

> >BTW: Newer 2.6 kernels are supporting write barriers for reiser & ext3.
> What's that?

In short, ordered write back
E.g.: W1, W2, W3, W4, WB, W5, W6, ...

Without write barriers any layer could choose to reorder the commands and so
there's no guarantee that W1-W4 are always written before W5 & W6.
With write barrier there's a guarantee that W1-W4 are always written before
W5 & W6. A kind of block device transaction & isolation. 

That's the problem with journaling filesystems or database transaction logs:
The logfile must be updated before the data is written. Even if a harddisk
isn't reordering the commands (no queueing or untagged queueing), the data
stored in the harddrive's cache isn't written back ordered. So, if you're
doing only buffered writes, you cannot be sure in which order the data is
written. To solve this, there are two approaches:

a) Writing logdata strictly unbuffered. This is the most reliable approach
but it degrades performance too much as every block is written synchronously
to disk.

b) Writing logdata buffered but set a write barrier. All outstanding changes
to the logdata is written asynchronously and flushed from disk cache to disk
platter afterwards before writing the next data. The performance degradation
isn't very much compared to fully buffered, non-barrier but it's much more
safe.

c) Write always buffered disregarding any write barriers. Only recommended
for fail-safe systems. Best performance. 

This is a feature a journaling filesystem must use and it must be ensured
that the appropriate commands are supported within the block io layer. 

As far as my research gained my knowledge, method b) is common on commercial
filesystems and databases. On some systems you have the choice of doing a),
b) or c). a) for the paranoid, c) for the careless and b) for the rest :-)

bye
Carl

-- 
NEU +++ DSL Komplett von GMX +++ http://www.gmx.net/de/go/dsl
GMX DSL-Netzanschluss + Tarif zum supergünstigen Komplett-Preis!

>As far as my research gained my knowledge, method b) is common on commercial
>filesystems and databases. On some systems you have the choice of doing a),
>b) or c). a) for the paranoid, c) for the careless and b) for the rest :-)
>
>  
>
I think that we're discussing too much "filesystem implementation 
details". It's good to have insight BUT it should be the "decision 
maker". What I think is that IET should work JUST LIKE any other block 
device (SCSI/SATA/RAID) with same advantages and disadvantages (and 
possible (meta)data corruption).

-- 

Libor Vanek


+-------------------------------------+
| Email:    libor@...            |
| ICQ:      124529939                 |
| WWW:      http://www.discobolos.net |
| Tel/fax:  +420 541 22 5091, 6293    |
| Mobil:    +420 777 703 642          |
+-------------------------------------+

> Please read Stephen Lord's mail more carefully. He said that
> write back may corrupt file system metadata.

Did I ever ignored this? 

> Note that he said that write-back may corrupt file system metadata,
> however, write-back preserves the ordering is safe for modern
> journaling file systems, I think.

That's the reason why some people implemented write barriers for io &
filesystems (see the current kernel 2.6). To ensure transaction isolation on
IO.

> Why is it so difficult for you to understand the fact that some
> journaling file systems in Linux don't support write back caching with
> reordering?

Why is it so difficult for you to see that there are some other systems
beyond Linux doing the things right? You want to provide a target, not an
initiator!

What are the facts (from my point of view):
1) Some filesystem are having problems with reordering
2) Some filesystems aren't having any problems with reordering
3) Harddisks are doing reordering
4) Write through cuts real database workload performance down to 20% 

> Please don't insult file system developers by saying that they are
> crap. Your theory says that ext3 was crap (I showed you Stephen
> C. Tweedie's mail) and XFS was crap (I showed you Stephen Lord's
> mail). Note that you thought that XFS is designed to handle write-back
> with reorder.

If a filesystem has a problem with write caching it's imho buggy. That's my
opinion. Why was write barrier support developed? Why not just forcing write
through?

> You don't know anything about file systems and Linux I/O design. Why
> are you so rude?

Why am I rude? Why are you so focused on only ext3 and Linux? 

> Again, how could you said such thing even though you don't know
> anything about XFS and ext3 implementations in Linux.

Sure, reordering was an issue, but now there are solutions. It won't improve
IET any further if we dispute on past issues. 

> I think that you have some experience and read white papers about
> these file systems, however, how can you be sure that they are
> true? You can find truth only in the source code.

Why do I have to take care only for these filesystems? Why should I ignore
other implementations? 

> Don't insult or criticize people by only your assumption. As I showed,
> you know no truth about file system implementations.

I assume I know as much about linux filesystems as you know about commercial
systems. I don't claim to know everything, do you?

> You don't need to say "You have to do something.". You have to do what
> you want for yourself.

Unfortunately I dont have too much programming experience on kernel layer
but I have some experiences on storage, storage area networks, databases and
other commercial systems used for handling real workload. But if you don't
need such experience I won't bother here anymore.

> If IET is not useful for you, please go away and find another solution.

Isn't your objective the development of an "iSCSI target with professional
features, that works well in enterprise environment under real workload, and
is scalable and versatile enough to meet the challenge of future storage
needs and developements" anymore?

I only wanted a constructive discussion about improvements to write caching
but your only arguments were the handicaps of some Linux filesystems. If you
don't want to get feedback from users who are having experience in real
workload systems, ok, no problem. But don't expect to play in the enterprise
league. Your highest scoring will be the toys club. Just my few cents.

Bye
Carl

-- 
Geschenkt: 3 Monate GMX ProMail + 3 Ausgaben der TV Movie mit DVD
++++ Jetzt anmelden und testen http://www.gmx.net/de/go/mail ++++

Hey, just calm down. Why not provide "write caching" as option (by 
default disabled) to those who knows that this can corrupt data?

-- 
Libor Vanek


+-------------------------------------+
| Email:    libor@...            |
| ICQ:      124529939                 |
| WWW:      http://www.discobolos.net |
| Tel/fax:  +420 541 22 5091, 6293    |
| Mobil:    +420 777 703 642          |
+-------------------------------------+

> Hey, just calm down. Why not provide "write caching" as option (by 
> default disabled) to those who knows that this can corrupt data?

Strictly write through isn't a solution, strictly write back isn't a
solution, too. 

bye
Carl

-- 
NEU +++ DSL Komplett von GMX +++ http://www.gmx.net/de/go/dsl
GMX DSL-Netzanschluss + Tarif zum supergünstigen Komplett-Preis!

yes, pls calm down.

i agree with Libor's suggestion.

1) make write back as an option here, as a patch, not be merged.
2) try to implement support for WRITE FUA, SYNCHRONIZE_CACHE (this is
useful for both write through and write back)
3) pursue a true write back solution with order preserve. or extra write
redo log. implement a new io handler.
4)...


ming



On Wed, 2004-10-27 at 09:08, Carl Rueder wrote:
> > Hey, just calm down. Why not provide "write caching" as option (by 
> > default disabled) to those who knows that this can corrupt data?
> 
> Strictly write through isn't a solution, strictly write back isn't a
> solution, too. 
> 
> bye
> Carl

Why 3)?

> yes, pls calm down.
> 
> i agree with Libor's suggestion.
> 
> 1) make write back as an option here, as a patch, not be merged.
> 2) try to implement support for WRITE FUA, SYNCHRONIZE_CACHE (this is
> useful for both write through and write back)
> 3) pursue a true write back solution with order preserve. or extra write
> redo log. implement a new io handler.
> 4)...
> 
> 
> ming
> 
> 
> 
> On Wed, 2004-10-27 at 09:08, Carl Rueder wrote:
> > > Hey, just calm down. Why not provide "write caching" as option (by 
> > > default disabled) to those who knows that this can corrupt data?
> > 
> > Strictly write through isn't a solution, strictly write back isn't a
> > solution, too. 
> > 
> > bye
> > Carl
> 

-- 
NEU +++ DSL Komplett von GMX +++ http://www.gmx.net/de/go/dsl
GMX DSL-Netzanschluss + Tarif zum supergünstigen Komplett-Preis!

From: "Carl Rueder" <carl.rueder@...>
Subject: Re: [Iscsitarget-devel] Write caching - subjective summary
Date: Wed, 27 Oct 2004 14:35:16 +0200 (MEST)

> > Please read Stephen Lord's mail more carefully. He said that
> > write back may corrupt file system metadata.
> 
> Did I ever ignored this? 

You said,

> c) Operating write back caching without battery backup was always a
> risk.

Modern journaling file systems are safe with write-back without reorder.

FYI, you want to protect not only metadata but also file-data, use a
journaling file systems which logs everything.


> > Note that he said that write-back may corrupt file system metadata,
> > however, write-back preserves the ordering is safe for modern
> > journaling file systems, I think.
> 
> That's the reason why some people implemented write barriers for io &
> filesystems (see the current kernel 2.6). To ensure transaction isolation on
> IO.

Have you read my mails?

I said that ext3 uses BIO barrier to work with write-back with
reordering and it was done this summer. And I also said that reiserfs
can handle write-back with reorder. And I know how Linux I/O
architecture has evolved.

You don't need to teach me such stuff.


> > Please don't insult file system developers by saying that they are
> > crap. Your theory says that ext3 was crap (I showed you Stephen
> > C. Tweedie's mail) and XFS was crap (I showed you Stephen Lord's
> > mail). Note that you thought that XFS is designed to handle write-back
> > with reorder.
> 
> If a filesystem has a problem with write caching it's imho buggy. That's my

So you think that XFS in Linux is buggy (at least, it was buggy in the
past due to the developer's mail). And you thought that it is designed
for enterprise use.


> opinion. Why was write barrier support developed? Why not just forcing write
> through?

If a disk drive, that uses write-back with reorder, cannot flush disk
cache in the presence of a system crash, it is evil for file systems.

However, as I said, some file system developers started to work on
this issue. That's why write barrier support was developed. But it
takes a long time. You quoted Linus's mail (2001) and some of file
systems in Linux have supported this since 2004.

So you call the file systems, which cannot handle write-back issue,
crap, it's wrong.


> > You don't know anything about file systems and Linux I/O design. Why
> > are you so rude?
> 
> Why am I rude? Why are you so focused on only ext3 and Linux? 

How do you think about people who say that something is crap without
knowledge about it?

How do you think about people who critisize developers without
knowledge about their work?

If they are not rude, who is rude?

And when I said that some file systems in Linux, like XFS and JFS,
can't handle "write back with reorder",

You replied, "I just won't believe that current file systems are such
a misdesign." even though you don't know anything about file system
implementations in Linux. So I looked for XFS developer's mail saying
that write-back cache may destroy your file system. Why did I do
such things to correct you? You should have done this for yourself.

Isn't it rude to deny someone's opinion without any knowledge and
waste his time to correct your misunderstanding?

Note that it is possible that XFS developers had worked this issue. So
please correct me by asking the developers if I'm wrong.


> > Don't insult or criticize people by only your assumption. As I
> > showed, you know no truth about file system implementations.
>
> I assume I know as much about linux filesystems as you know about
> commercial systems. I don't claim to know everything, do you?

I don't know anything about commercial systems, so did I criticize
them by calling them crap like you?


> > You don't need to say "You have to do something.". You have to do what
> > you want for yourself.
> 
> Unfortunately I dont have too much programming experience on kernel layer
> but I have some experiences on storage, storage area networks, databases and
> other commercial systems used for handling real workload. But if you don't
> need such experience I won't bother here anymore.

Please don't bother me any more.


> > If IET is not useful for you, please go away and find another solution.
> 
> Isn't your objective the development of an "iSCSI target with professional
> features, that works well in enterprise environment under real workload, and
> is scalable and versatile enough to meet the challenge of future storage
> needs and developements" anymore?
> 
> I only wanted a constructive discussion about improvements to write caching

I don't think so.

For example, I explained one of reasons why it is difficult to achieve
write-back policy in IET, which disk drives provide. I don't think
that your reply was constructive.

I love to listen to users' opinions and collaborate with
people. However, I don't think that I can work with you
constructively, though I'm not sure about how other people feel.

All I can say is that please go away and find another solution.

From: Ming Zhang <mingz@...>
Subject: Re: [Iscsitarget-devel] Write caching - subjective summary
Date: Wed, 27 Oct 2004 10:11:41 -0400

> i agree with Libor's suggestion.
> 
> 1) make write back as an option here, as a patch, not be merged.
> 2) try to implement support for WRITE FUA, SYNCHRONIZE_CACHE (this is
> useful for both write through and write back)
> 3) pursue a true write back solution with order preserve. or extra write
> redo log. implement a new io handler.

Tomorrow (maybe day after tomorrow), I'll post reasons why it is to
implement write-back, which disk drives provide, though I've already
said some here. And I'll also suggest a possible way to implement it.

If it is implemented properly and you guys want, merging it is OK for
me. But I don't think that it is easy to implement it. And as I said,
I'm not interested in implementing this so much.

I prefer other jobs like the new I/O design. And the new I/O design
make it easier for you to work independently. For example, you can
implement the virtual tape library without modifications to the core
code. So I think that I should do it first.

3 is useful for trying to recovery some data after crash, same reason as
why people need journalling fs.

On Wed, 2004-10-27 at 10:41, Carl Rueder wrote:
> Why 3)?
> 
> > yes, pls calm down.
> > 
> > i agree with Libor's suggestion.
> > 
> > 1) make write back as an option here, as a patch, not be merged.
> > 2) try to implement support for WRITE FUA, SYNCHRONIZE_CACHE (this is
> > useful for both write through and write back)
> > 3) pursue a true write back solution with order preserve. or extra write
> > redo log. implement a new io handler.
> > 4)...
> > 
> > 
> > ming
> > 
> > 
> > 
> > On Wed, 2004-10-27 at 09:08, Carl Rueder wrote:
> > > > Hey, just calm down. Why not provide "write caching" as option (by 
> > > > default disabled) to those who knows that this can corrupt data?
> > > 
> > > Strictly write through isn't a solution, strictly write back isn't a
> > > solution, too. 
> > > 
> > > bye
> > > Carl
> > 
-- 
 --------------------------------------------------
| Ming Zhang, PhD. Student
| Dept. of Electrical & Computer Engineering
| College of Engineering
| University of Rhode Island
| Kingston RI. 02881
| e-mail: mingz at ele.uri.edu
| Tel. (401) 874-2293 
| Fax. (401) 782-6422
| http://www.ele.uri.edu/~mingz/
| http://crab.ele.uri.edu/gallery/albums.php
 --------------------------------------------------

On Wed, 2004-10-27 at 10:48, FUJITA Tomonori wrote:
> From: Ming Zhang <mingz@...>
> Subject: Re: [Iscsitarget-devel] Write caching - subjective summary
> Date: Wed, 27 Oct 2004 10:11:41 -0400
> 
> > i agree with Libor's suggestion.
> > 
> > 1) make write back as an option here, as a patch, not be merged.
> > 2) try to implement support for WRITE FUA, SYNCHRONIZE_CACHE (this is
> > useful for both write through and write back)
> > 3) pursue a true write back solution with order preserve. or extra write
> > redo log. implement a new io handler.
> 
> Tomorrow (maybe day after tomorrow), I'll post reasons why it is to
> implement write-back, which disk drives provide, though I've already
> said some here. And I'll also suggest a possible way to implement it.
ok. looking forward to seeing.

> 
> If it is implemented properly and you guys want, merging it is OK for
> me. But I don't think that it is easy to implement it. And as I said,
> I'm not interested in implementing this so much.
> 
> I prefer other jobs like the new I/O design. And the new I/O design
> make it easier for you to work independently. For example, you can
> implement the virtual tape library without modifications to the core
> code. So I think that I should do it first.
> 
i vote on this. if we can get this done first. then libor can work on VT
without modifying others. i may work on a bypass interface to allow use
system real scsi device directly.

> 
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Sybase ASE Linux Express Edition - download now for FREE
> LinuxWorld Reader's Choice Award Winner for best database on Linux.
> http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
> _______________________________________________
> Iscsitarget-devel mailing list
> Iscsitarget-devel@...
> https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel
-- 
 --------------------------------------------------
| Ming Zhang, PhD. Student
| Dept. of Electrical & Computer Engineering
| College of Engineering
| University of Rhode Island
| Kingston RI. 02881
| e-mail: mingz at ele.uri.edu
| Tel. (401) 874-2293 
| Fax. (401) 782-6422
| http://www.ele.uri.edu/~mingz/
| http://crab.ele.uri.edu/gallery/albums.php
 --------------------------------------------------