Re: [Iometer-devel] Sequential IO with SignleworkerandMultipleworkers

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

I am little confused about the O_DIRECT flag? How does that matter with
running multiple workers each with either single outstanding IO or
multiple outstanding Ios?=20

-----Original Message-----
From: iom...@li...
[mailto:iom...@li...] On Behalf Of
Harder, David W.
Sent: Wednesday, June 28, 2006 10:07 AM
To: mi...@el...
Cc: iom...@li...
Subject: Re: [Iometer-devel] Sequential IO with
SignleworkerandMultipleworkers

No, unfortunately I haven't and I don't currently have the resources
available to do so.

Actually, if I remember right the kernel implementation of aio is
present in 2.4 (at least the 2.4 filesystems I looked at) it just
neglects to check for the O_DIRECT flag and act accordingly.  Our
workaround was to add a command-line option that allowed us to switch to
posix read/write commands when we needed O_DIRECT but didn't care about
multiple outstanding I/Os (if and only if O_DIRECT was also enabled for
which we also had a command-line option).

-Dave-

-----Original Message-----
From: Ming Zhang [mailto:mi...@el...]
Sent: Wednesday, June 28, 2006 10:55 AM
To: Harder, David W.
Cc: iometer-devel-list
Subject: RE: [Iometer-devel] Sequential IO with Signleworker
andMultipleworkers

cc to iometer list. seems that original email was bounced.

yes, aio is broken in 2.4 and is emulated by user space thread in glibc.
i did not check the correctness in 2.6 yet.

have u tried with block device instead of file system?

On Wed, 2006-06-28 at 10:33 -0500, Harder, David W. wrote:
> Has anyone done any testing lately to determine if queueing multiple=20
> I/Os through AIO+O_DIRECT actually results in multiple I/Os to the=20
> disk in Linux?
>=20
> I seem to recall doing some testing a year or so ago that showed that=20
> if we modified IOmeter to use O_DIRECT (back when it wasn't in the
> tree) that it didn't matter how many transactions IOmeter kept=20
> outstanding, because there was a bottleneck in Linux 2.6 somewhere.
> (We have protocol analyzers that I think we used to verify our
> conclusion.) Of course, that was against a filesystem, but I think we=20
> reproduced it on multiple filesystems.  I think we assumed it had to=20
> do with something Direct I/O was doing in order to maintain sanity=20
> when dealing with DMA to/from user-space pages.
>=20
> The other problem I recall encountering is that the Linux 2.4.x AIO=20
> implementation is broken and it doesn't matter if O_DIRECT is set.  If

> you use AIO in Linux 2.4, you get buffered I/O regardless of O_DIRECT.
> However, I just noticed that we're dealing with 2.6.16, so that=20
> couldn't be the problem.
>=20
> -Dave Harder-
>=20
> P.S.  Judging from the numbers below, I would guess that either=20
> IOmeter is not using O_DIRECT in this case or that the disk has=20
> write-caching enabled.  Otherwise, the rotational latency between=20
> writes would prevent you from getting that kind of throughput with
'oustanding I/Os =3D 1'.
>=20
> -----Original Message-----
> From: iom...@li...
> [mailto:iom...@li...] On Behalf Of Ming

> Zhang
> Sent: Tuesday, June 27, 2006 8:17 PM
> To: P M, Priya
> Cc: iom...@li...
> Subject: Re: [Iometer-devel] Sequential IO with Signleworker=20
> andMultipleworkers
>=20
> On Wed, 2006-06-28 at 06:38 +0530, P M, Priya wrote:
> > could you tell me which source code you are using?
> > I am using the source code downloaded from "The latest code can be=20
> > found from  svn://svn.berlios.de/svnroot/repos/iometer/  "
>=20
> it has trunk and branch. so i am still unclear about which code u are=20
> using. and specifically, i want to check the file opened in O_DIRECT=20
> or not.
>=20
> or can u tell me that when u run the test, what is the system memory=20
> usage. run free and post here.
>=20
>=20
> >=20
> > Our actual doubt is
> > If we run an iometer sequential workload with No of outstanding =
Ios=3D

> > 7,
>=20
> run 1 worker with outstanding io 7 is not the same as run 7 worker and

> each with outstanding io =3D 1;
>=20
>=20
> > what does that mean:
> > 1)       Each thread runs its own sequential workload independently?
>=20
> yes.
>=20
> > 2)       Sequential block I/Os are parceled out to the threads in
> > sequential order?
>=20
> each worker will generate its own io stream.
>=20
>=20
> >=20
> > So we thought Running an iometer sequential workload with queue=20
> > depth
> > 7 means, in our case, a single thread is used to queue 7 requests.=20
> > Becasue in IOMeter, a worker is nothing but a single thread and we=20
> > ran
>=20
> > all our tests with single worker.
>=20
> 1 worker run 1 io stream.
>=20
> but u said in previous emails that u ran with 7 workers as well.
>=20
>=20
> > If we want to simulate the Option 1 ( each thread with its own=20
> > sequential workload independently), we need to create 7 workers,=20
> > each one which submits one request.
>=20
> yes. and iometer will sum up all the number get from 7 workers. so if
> 7 worker send out (even independently but happen to be same) 7 io=20
> streams,
> 7 io streams will be merged to 1 io stream in os and eventually arrive

> disk as 1 stream. then here u get a fake result (because same result=20
> summed by 7 times).
>=20
>=20
>=20
>=20
>=20
>=20
> >=20
> > Is our understanding correct?
> >=20
> > -----Original Message-----
> > From: Ming Zhang [mailto:mi...@el...]
> > Sent: Tuesday, June 27, 2006 7:00 PM
> > To: P M, Priya
> > Cc: iom...@li...
> > Subject: RE: [Iometer-devel] Sequential IO with Signle worker=20
> > andMultipleworkers
> >=20
> > On Wed, 2006-06-28 at 06:22 +0530, P M, Priya wrote:
> > > u tested with physical disk or fs on it?
> > >=20
> > > Answer >> I tested with Physical disk.
> > >=20
> > > > =20
> > > >         512 Bytes       4KB     16KB    32KB   =20
> > > > Queue depth =3D7  6254 IOPs       19077 IOPs      4769 IOPs
> > > 2464
> > > > IOPs      =20
> > >               ~~~guess this is a typo?
> > >=20
> > > Answer >> No. It is not typo. This is the behavior we have seen.
> >=20
> > no i mean it is "queue depth=3D1"?
> >=20
> > >=20
> > > > Queue depth =3D7  5849 IOPs       19180 IOPs      4909 IOPs
> > > 2426
> > > > IOPs      =20
> > > > # Workers=3D7 =20
> > > > Depth =3D 1       33488 IOPs      83673 IOPs      30872 IOPs
> > > 15669
> > > > IOPs    =20
> > >=20
> > >=20
> > > u disk is quite fast at 80MB/s.
> > >=20
> > >=20
> > > > =20
> > > > I have some doubts about the results what IOMeter produces.
> > >=20
> > > i have one as well. u 512B IOPS is MUCH lower than 4KB. can u test

> > > 1KB
> >=20
> > > as well?
> > >=20
> > >=20
> > >=20
> > > > =20
> > > > 1. How can the result for Queue depth =3D 7 and 1 be almost =
same?=20
> > >=20
> > > why? if iometer can send out request fast enough.
> > >=20
> > > check scsi device queue depth, that is the one you meant i feel.=20
> > > Answer >> Yes. I meant the No of outstanding Ios as Queue depth.
> >=20
> > could you tell me which source code you are using?
> >=20
> > >=20
> > >=20
> > > > 2. If I increase the number of workers to 7 and queue depth as 1
> > > for
> > > > each worker, I see a huge change in the performance results?
> > >=20
> > > i do not think this value is meaningful. stick to worker =3D 1 =
here.

> > > i
>=20
> > > would guess all 7 workers send out same request at same time so=20
> > > all of
> >=20
> > > them are merged into 1 request at lower level.
> > >=20
> > > Answer >> I would like to know how IOMeter handles each request,=20
> > > if I create 7 workers.  Do you have any idea???
> >=20
> > if you create 7 workers in one dynamo, and when dynamo start to
work.=20
> > 7 workers work independently, but since u are using sequential=20
> > write, so each worker will generate same request sequence. then=20
> > these will be
>=20
> > merged at linux os level. it is like
> >=20
> > dd if=3D/dev/zero of=3D/dev/sda bs=3Dx &
> > dd if=3D/dev/zero of=3D/dev/sda bs=3Dx &
> > dd if=3D/dev/zero of=3D/dev/sda bs=3Dx &
> > dd if=3D/dev/zero of=3D/dev/sda bs=3Dx &
> > ...
> >=20
> > if u start these at almost same time, os will merge most of them and

> > each dd can get a good number. but u can not sum them up.
> >=20
> >=20
> >=20
> > >=20
> > > Thanks,
> > > Priya
> > >=20
> > > -----Original Message-----
> > > From: Ming Zhang [mailto:mi...@el...]
> > > Sent: Tuesday, June 27, 2006 6:26 PM
> > > To: P M, Priya
> > > Cc: iom...@li...
> > > Subject: Re: [Iometer-devel] Sequential IO with Signle worker and=20
> > > Multipleworkers
> > >=20
> > > On Tue, 2006-06-27 at 03:47 +0530, P M, Priya wrote:=20
> > > > Hi,
> > > >   =20
> > > > I am using the latest IOMeter ( got the code and compiled it)=20
> > > > with
>=20
> > > > Linux 2.6.16.
> > > > =20
> > > > Test Configuration: =20
> > > > Sequential Write
> > > > SCSI Ultra 3 Disk with RPM 10K
> > >=20
> > > u tested with physical disk or fs on it?
> > >=20
> > > > =20
> > > >         512 Bytes       4KB     16KB    32KB   =20
> > > > Queue depth =3D7  6254 IOPs       19077 IOPs      4769 IOPs
> > > 2464
> > > > IOPs      =20
> > >               ~~~guess this is a typo?
> > >=20
> > >=20
> > > > Queue depth =3D7  5849 IOPs       19180 IOPs      4909 IOPs
> > > 2426
> > > > IOPs      =20
> > > > # Workers=3D7 =20
> > > > Depth =3D 1       33488 IOPs      83673 IOPs      30872 IOPs
> > > 15669
> > > > IOPs    =20
> > >=20
> > >=20
> > > u disk is quite fast at 80MB/s.
> > >=20
> > >=20
> > > > =20
> > > > I have some doubts about the results what IOMeter produces.
> > >=20
> > > i have one as well. u 512B IOPS is MUCH lower than 4KB. can u test

> > > 1KB
> >=20
> > > as well?
> > >=20
> > >=20
> > > > =20
> > > > 1. How can the result for Queue depth =3D 7 and 1 be almost =
same?=20
> > >=20
> > > why? if iometer can send out request fast enough.
> > >=20
> > > check scsi device queue depth, that is the one you meant i feel.
> > >=20
> > >=20
> > >=20
> > > > 2. If I increase the number of workers to 7 and queue depth as 1
> > > for
> > > > each worker, I see a huge change in the performance results?
> > >=20
> > > i do not think this value is meaningful. stick to worker =3D 1 =
here.

> > > i
>=20
> > > would guess all 7 workers send out same request at same time so=20
> > > all of
> >=20
> > > them are merged into 1 request at lower level.
> > >=20
> > >=20
> > > > =20
> > > > Can anyone please explain why this behavior I see in Linux??? =20
> > > > Is there any problem in IOMeter?=20
> > > > =20
> > > > Thanks,
> > > > Priya
> > > > =20
> > > > =20
> > > > =20
> > > > Using Tomcat but need to do more? Need to support web services,
> > > security?=20
> > > > Get stuff done quickly with pre-integrated technology to make=20
> > > > your
> > > job
> > > > easier Download IBM WebSphere Application Server v.1.0.1 based=20
> > > > on Apache Geronimo
> > > >
> > > =
http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D=

> > > 12
> > > 16
> > > > 42 _______________________________________________ Iometer-devel

> > > > mailing list Iom...@li...
> > > > https://lists.sourceforge.net/lists/listinfo/iometer-devel
> > >=20
> >=20
>=20
>=20
> Using Tomcat but need to do more? Need to support web services,=20
> security?
> Get stuff done quickly with pre-integrated technology to make your job

> easier Download IBM WebSphere Application Server v.1.0.1 based on=20
> Apache Geronimo
> =
http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D=
1216
> 42 _______________________________________________
> Iometer-devel mailing list
> Iom...@li...
> https://lists.sourceforge.net/lists/listinfo/iometer-devel

Using Tomcat but need to do more? Need to support web services,
security?
Get stuff done quickly with pre-integrated technology to make your job
easier Download IBM WebSphere Application Server v.1.0.1 based on Apache
Geronimo
http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D=
121642
_______________________________________________
Iometer-devel mailing list
Iom...@li...
https://lists.sourceforge.net/lists/listinfo/iometer-devel