Linux Scalability Effort

Hanna Linder <hannal@...> wrote:
>
> In mainline, once block size is over 32k our throuput actually drops off.
> It levels off around 128k but at a greater cpu utilization.
> Dont really understand why that is. 

Probably thrashing the CPU's L1 cache.

> In mm tree, maintains throughput for all block size but the cpu utilzation
> keeps going up to do the same throughput.  Readprofile shows the extra time 
> is being spent in copy_to_user (in mm tree). Backing out readahead patch 
> reduces cpu by 10% for all block sizes but still shows the spike. So that
> isnt the main problem.

If you have a loop like:

	char *buf;

	for (lots) {
		read(fd, buf, size);
	}


the optimum value of `size' is small: as little as 8k.  Once `size' gets
close to half the size of the L1 cache you end up pushing the memory at
`buf' out of CPU cache all the time.

On Wed, Oct 01, 2003 at 04:29:16PM -0700, Andrew Morton wrote:
> If you have a loop like:
> 
> 	char *buf;
> 
> 	for (lots) {
> 		read(fd, buf, size);
> 	}
> 
> the optimum value of `size' is small: as little as 8k.  Once `size' gets
> close to half the size of the L1 cache you end up pushing the memory at
> `buf' out of CPU cache all the time.

I've seen this too, not that Andrew needs me to back him up, but in many 
cases even 4k is big enough.  Linux has a very thin system call layer so
it is OK, good even, to use reasonable buffer sizes.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Larry McVoy wrote:
> On Wed, Oct 01, 2003 at 04:29:16PM -0700, Andrew Morton wrote:
> 
>>If you have a loop like:
>>
>>	char *buf;
>>
>>	for (lots) {
>>		read(fd, buf, size);
>>	}
>>
>>the optimum value of `size' is small: as little as 8k.  Once `size' gets
>>close to half the size of the L1 cache you end up pushing the memory at
>>`buf' out of CPU cache all the time.
> 
> 
> I've seen this too, not that Andrew needs me to back him up, but in many 
> cases even 4k is big enough.  Linux has a very thin system call layer so
> it is OK, good even, to use reasonable buffer sizes.


Slight tangent, FWIW...   Back when I was working on my "race-free 
userland" project, I noticed that the fastest cp(1) implementation was 
GNU's:  read/write from a single, statically allocated, page-aligned 4K 
buffer.  I experimented with various buffer sizes, mmap-based copies, 
and even with sendfile(2) where both arguments were files. 
read(2)/write(2) of a single 4K buffer was always the fastest.

	Jeff

On Thursday 02 October 2003 03:23, Jeff Garzik wrote:
> Larry McVoy wrote:
> > On Wed, Oct 01, 2003 at 04:29:16PM -0700, Andrew Morton wrote:
> >>If you have a loop like:
> >>
> >>	char *buf;
> >>
> >>	for (lots) {
> >>		read(fd, buf, size);
> >>	}
> >>
> >>the optimum value of `size' is small: as little as 8k.  Once `size' gets
> >>close to half the size of the L1 cache you end up pushing the memory at
> >>`buf' out of CPU cache all the time.
> >
> > I've seen this too, not that Andrew needs me to back him up, but in many
> > cases even 4k is big enough.  Linux has a very thin system call layer so
> > it is OK, good even, to use reasonable buffer sizes.
>
> Slight tangent, FWIW...   Back when I was working on my "race-free
> userland" project, I noticed that the fastest cp(1) implementation was
> GNU's:  read/write from a single, statically allocated, page-aligned 4K
> buffer.  I experimented with various buffer sizes, mmap-based copies,
> and even with sendfile(2) where both arguments were files.
> read(2)/write(2) of a single 4K buffer was always the fastest.

That sounds reasonable, but today's RAM throughput is on the order
of 1GB/s, not 100Mb/s. 'Out of L1' theory can't explain 100Mb/s ceiling
it seems.
--
vda

insecure wrote:
> That sounds reasonable, but today's RAM throughput is on the order
> of 1GB/s, not 100Mb/s. 'Out of L1' theory can't explain 100Mb/s ceiling
> it seems.


cp(1) data, at least, will never ever be in L1.  Copying data you need 
to look at the ends of the pipeline -- hard drive throughput, PCI bus 
bandwidth, FSB bandwidth, speed at which ext2/3 allocates blocks, and 
similar things are likely bottlenecks.

You'll never hit RAM bandwidth limits, unless your copies are extremely 
tiny, and entirely in L2 or pagecache.

	Jeff

Andrew Morton wrote:

>Hanna Linder <hannal@...> wrote:
>  
>
>>In mainline, once block size is over 32k our throuput actually drops off.
>>It levels off around 128k but at a greater cpu utilization.
>>Dont really understand why that is. 
>>    
>>
>
>Probably thrashing the CPU's L1 cache.
>  
>
>>In mm tree, maintains throughput for all block size but the cpu utilzation
>>keeps going up to do the same throughput.  Readprofile shows the extra time 
>>is being spent in copy_to_user (in mm tree). Backing out readahead patch 
>>reduces cpu by 10% for all block sizes but still shows the spike. So that
>>isnt the main problem.
>>    
>>
>
>If you have a loop like:
>
>	char *buf;
>
>	for (lots) {
>		read(fd, buf, size);
>	}
>
>
>the optimum value of `size' is small: as little as 8k.  Once `size' gets
>close to half the size of the L1 cache you end up pushing the memory at
>`buf' out of CPU cache all the time.
>
>  
>
Sure, but why do I only see this is the mm tree, and not the mainline 
tree. Also, there are 160 threads doing this loop, so even at 32k block 
size the  working set of 'buf's is over 5MB with only a 2MB L2 cache.

Steve

>
>-------------------------------------------------------
>This sf.net email is sponsored by:ThinkGeek
>Welcome to geek heaven.
>http://thinkgeek.com/sf
>_______________________________________________
>Lse-tech mailing list
>Lse-tech@...
>https://lists.sourceforge.net/lists/listinfo/lse-tech
>  
>

Steven Pratt <slpratt@...> wrote:
>
>  Sure, but why do I only see this is the mm tree, and not the mainline 
>  tree.

Please send a full description of how to reproduce it and I'll take a look.

On Thursday 02 October 2003 22:10, Jeff Garzik wrote:
> insecure wrote:
> > That sounds reasonable, but today's RAM throughput is on the order
> > of 1GB/s, not 100Mb/s. 'Out of L1' theory can't explain 100Mb/s ceiling
> > it seems.
>
> cp(1) data, at least, will never ever be in L1.  Copying data you need
> to look at the ends of the pipeline -- hard drive throughput, PCI bus
> bandwidth, FSB bandwidth, speed at which ext2/3 allocates blocks, and
> similar things are likely bottlenecks.

Hmm.

On Wednesday 01 October 2003 22:19, Hanna Linder wrote:
> We got about 100 mb/sec using the bonie benchmark for block io writes,
> for writes we hit a ceiling around 100-120 mb/sec. Stopped scaling
> after about 3 spindles.
>
> Tried to focus on the block io part of it. Have not tried
> direct or raw io yet.  With block IO we got about 133 mb/sec
> doing a simple dd to dev/null from multiple spindles. This
> was on the 2.6 test 3.
> ....
> Odirect on large block sizes has low cpu utilization ( 3-5% ).
> However with buffered IO we can easily get to 100% cpu utilization.
> If you look at the profile most of that is in the copy_to_user function.

So:
* we hit a ceiling of ~133 Mb/s, no matter how many disks
* CPU utilization is 100%, spent mostly in copy_to_user
* RAM bandwidth is >1Gb/s

These can't be true at once.
At least one of these three statements must be false (imho).
-- 
vda

Ahh. I see the confusion. These are excerpts from two different speakers.
Notice the line of ===== separating speakers in the minutes. 
This part is from Steve Pratt's mm vs mainline comparisons:


> On Thursday 02 October 2003 22:10, Jeff Garzik wrote:
>> insecure wrote:
>> > That sounds reasonable, but today's RAM throughput is on the order
>> > of 1GB/s, not 100Mb/s. 'Out of L1' theory can't explain 100Mb/s ceiling
>> > it seems.
>> 
>> cp(1) data, at least, will never ever be in L1.  Copying data you need
>> to look at the ends of the pipeline -- hard drive throughput, PCI bus
>> bandwidth, FSB bandwidth, speed at which ext2/3 allocates blocks, and
>> similar things are likely bottlenecks.
> 
> Hmm.
> 

This part is from Mark Gross's Real Time Application issues discussion:


> On Wednesday 01 October 2003 22:19, Hanna Linder wrote:
>> We got about 100 mb/sec using the bonie benchmark for block io writes,
>> for writes we hit a ceiling around 100-120 mb/sec. Stopped scaling
>> after about 3 spindles.
>> 
>> Tried to focus on the block io part of it. Have not tried
>> direct or raw io yet.  With block IO we got about 133 mb/sec
>> doing a simple dd to dev/null from multiple spindles. This
>> was on the 2.6 test 3.
>> ....
>> Odirect on large block sizes has low cpu utilization ( 3-5% ).
>> However with buffered IO we can easily get to 100% cpu utilization.
>> If you look at the profile most of that is in the copy_to_user function.
> 
> So:
> * we hit a ceiling of ~133 Mb/s, no matter how many disks
> * CPU utilization is 100%, spent mostly in copy_to_user
> * RAM bandwidth is >1Gb/s


Hanna

Andrew Morton wrote:

>Steven Pratt <slpratt@...> wrote:
>  
>
>> Sure, but why do I only see this is the mm tree, and not the mainline 
>> tree.
>>    
>>
>
>Please send a full description of how to reproduce it and I'll take a look.
>
>  
>
Get the latest rawread from 
http://www-124.ibm.com/developerworks/opensource/linuxperf/rawread/rawread.html

mkfs devices and mount on /mnt/mntN  where N is increasing index.  
Create file 'foo'  in each filesystem of size 1GB (for this example).  
Unmount and remount the partitions/devices to flush the cache.  
Filesystems are also umounted and re-mounted between each test run.

The following rawread commands will run the tests for block sizes 
ranging from 1k-512k.  The "-d 1" parameters assumes that you mounted 
starting at /mnt/mnt1  and the "-m2 -p16" say to run 8 threads on each 
of 2 devices /mnt/mnt1 and /mnt/mnt2.

rawread -m 2 -p 16 -d 6 -n 20480 -f -c -t 0 -s 1024

rawread -m 2 -p 16 -d 6 -n 10240 -f -c -t 0 -s 2048

rawread -m 2 -p 16 -d 6 -n 5120 -f -c -t 0 -s 4096

rawread -m 2 -p 16 -d 6 -n 2560 -f -c -t 0 -s 8192

rawread -m 2 -p 16 -d 1 -n 1280 -f -c -t 0 -s 16384

rawread -m 2 -p 16 -d 1 -n 640 -f -c -t 0 -s 32768

rawread -m 2 -p 16 -d 1 -n 320 -f -c -t 0 -s 65536

rawread -m 2 -p 16 -d 1 -n 160 -f -c -t 0 -s 131072

rawread -m 2 -p 16 -d 1 -n 80 -f -c -t 0 -s 262144

rawread -m 2 -p 16 -d 1 -n 40 -f -c -t 0 -s 524288


2 devices is the smallest number I have been able to run which shows 
this problem.  With only 1 device I did not see it.  My original tests 
were done with 20 devices.  One thing of interest is that with only 2 
devices the point at which CPU starts to increase again is at 128k 
instead of at 32k which I saw with 20 devices.  This would support your 
theory that this is casued by cache misses with more/larger buffers.  
I'm still not sure this accounts for all of the extra CPU usage, but I 
am less worried about it.

But as long as I have your attention, there is one other thing about 
these runs which bothers me, which is that the mm tree is doing horribly 
on 1k and 2k block sizes.  I looks like readahead is not functioning 
properly for these requst sizes.

Here is a comparison for 2 devices between test6 and test6mm1.  You can 
see that the mm1 tree does great at larger block sizes, but poorly at 
small ones.


Results:seqread-_vs_.seqread-

                                          tolerance = 0.00 + 3.00% of A
                test6         test6-mm1
 Blocksize      KBs/sec      KBs/sec    %diff         diff    tolerance
---------- ------------ ------------ -------- ------------ ------------
      1024        44083        22641   -48.64    -21442.00      1322.49  *
      2048        45276        26371   -41.76    -18905.00      1358.28  *
      4096        44024        45260     2.81      1236.00      1320.72
      8192        44519        50073    12.48      5554.00      1335.57  *
     16384        46869        51528     9.94      4659.00      1406.07  *
     32768        47900        52231     9.04      4331.00      1437.00  *
     65536        42803        52183    21.91      9380.00      1284.09  *
    131072        36525        49724    36.14     13199.00      1095.75  *
    262144        34628        46192    33.39     11564.00      1038.84  *
    524288        28997        48005    65.55     19008.00       869.91  *


Results:seqread-_vs_.seqread-
                                          tolerance = 0.50 + 3.00% of A
               test6         test6-mm1
 Blocksize         %CPU         %CPU    %diff         diff    tolerance
---------- ------------ ------------ -------- ------------ ------------
      1024        27.87        11.72   -57.95       -16.15         1.34  *
      2048        13.77         8.84   -35.80        -4.93         0.91  *
      4096            9         9.99    11.00         0.99         0.77  *
      8192         8.07         8.31     2.97         0.24         0.74
     16384          5.7         6.63    16.32         0.93         0.67  *
     32768         4.93         5.59    13.39         0.66         0.65  *
     65536         3.76          4.7    25.00         0.94         0.61  *
    131072         3.25         4.53    39.38         1.28         0.60  *
    262144         3.23         6.15    90.40         2.92         0.60  *
    524288         2.97         8.19   175.76         5.22         0.59  *

Steve

Steven Pratt <slpratt@...> wrote:
>
> Get the latest rawread from 
> http://www-124.ibm.com/developerworks/opensource/linuxperf/rawread/rawread.html

Sigh, I was afraid of that.  My previous outings with rawread have not been
happy.  Maybe your detailed descriptions will help.

> But as long as I have your attention, there is one other thing about 
> these runs which bothers me, which is that the mm tree is doing horribly 
> on 1k and 2k block sizes.  I looks like readahead is not functioning 
> properly for these requst sizes.

There are several problems with readahead under specific circumstances.  I
need to have a session with it.

insecure <insecure@...> wrote:
>
> So:
>  * we hit a ceiling of ~133 Mb/s, no matter how many disks
>  * CPU utilization is 100%, spent mostly in copy_to_user
>  * RAM bandwidth is >1Gb/s
> 
>  These can't be true at once.

True.  But bear in mind that the data crosses the memory busses up to three
times: disk to pagecache, pagecache to CPU, CPU to user memory.

So top speed may be as little as 300 MB/sec.