From: Rik v. R. <ri...@co...> - 2002-09-18 16:45:31
|
On Wed, 18 Sep 2002, Bill Hartner wrote: > I will baseline on 2.4.19 and run both the 3GB and 4GB VoloanoMark test. > > I will also test with rmap14a. I released rmap14b last night, with an SMP bugfix you'll want to have: http://surriel.com/patches/2.4/2.4.19-rmap14b > I am currently running (a) rawio on scsi devices and (b) direct io on scsi > devices for both read and readv on 2.5.35. For this test, I am using an > 8-way 700 Mhz with (4) IBM 4Mx controllers and 32 disks. Hmmm, with near certainty rmap in 2.4 still has a bunch of SMP inefficiencies that'll slow you down on an 8-way. If these are bothering you I'll do a backport of the 2.5 rmap speedups... regards, Rik -- Spamtrap of the month: sep...@su... http://www.surriel.com/ http://distro.conectiva.com/ |
From: Bill H. <bha...@au...> - 2002-09-18 16:53:23
|
Rik van Riel wrote: > > On Tue, 17 Sep 2002, Andrew Morton wrote: > > Bill Hartner wrote: > > > > > > I ran VolanoMark 2.1.2 under memory pressure to test rmap. > > > --------------- > > > 2.5.26 vs 2.5.26 + rmap patch > > > ----------------------------- > > > It appears as though the page stealing decisions made when using the > > > 2.5.26 rmap patch may not be as good as the baseline for this workload. > > > There was more swap activity and idle time. > > > > Do you have similar results for 2.4 and 2.4-rmap? > > If Bill is going to test this, I'd appreciate it if he could use > rmap14a (or newer, if I've released it by the time he gets around > to testing). > Rik, I will baseline on 2.4.19 and run both the 3GB and 4GB VoloanoMark test. I will also test with rmap14a. I am currently running (a) rawio on scsi devices and (b) direct io on scsi devices for both read and readv on 2.5.35. For this test, I am using an 8-way 700 Mhz with (4) IBM 4Mx controllers and 32 disks. I should be able to get to the 2.4.19 VolanoMark tests by the end of the week. Both rawio and VolanoMark test use the same machine. Bill |
From: Bill H. <ha...@au...> - 2002-09-27 17:02:49
|
Rik van Riel wrote: > > > > > 2.5.26 vs 2.5.26 + rmap patch > > > ----------------------------- > > > It appears as though the page stealing decisions made when using the > > > 2.5.26 rmap patch may not be as good as the baseline for this workload. > > > There was more swap activity and idle time. > > > > Do you have similar results for 2.4 and 2.4-rmap? > > If Bill is going to test this, I'd appreciate it if he could use > rmap14a (or newer, if I've released it by the time he gets around > to testing). > More VolanoMark results using an 8-way 700 Mhz under memory pressure for 2.4.19 and rmap14b. Details of SUT same as : http://marc.theaimsgroup.com/?l=linux-mm&m=103229747000714&w=2 NOTE : the swap device is on ServeRAID which is probably bouncing for the HIGHMEM pages in most if not all of the tests so results will likely improve when bouncing is eliminated. 2419 = 2.4.19 + o(1) scheduler 2419rmap = 2.4.19 + rmap14b + o(1) scheduler %sys/%user = ratio of %system CPU utilization to %user CPU utilization. ======================================== The results for the 3 GB mem test were : ======================================== kernel msg/s %CPU %sys/%user Total swpin Total swpout Total swapio ----------- ----- ---- ---------- ------------ ------------ ------------ 2.4.19 ***** system hard hangs - requires reset. ***** 2.4.19rmap 37767 76.9 1.46 2,274,380 KB 3,800,336 KB 6,074,716 KB =============================== old data below=============================== 2.5.26 51824 96.3 1.42 1,987,024 KB 2,148,100 KB 4,135,124 KB 2.5.26rmap 46053 90.8 1.55 3,139,324 KB 3,887,368 KB 7,026,692 KB 2.5.35 44693 86.1 1.45 1,982,236 KB 5,393,152 KB 7,375,388 KB 2.5.35mm1 39679 99.6 1.50 *2,720,600 KB *6,154,512 KB *8,875,112 KB * used pgin/pgout instead of swapin/swapout since /proc/stat changed. 2.4.19 + o(1) hangs the system - requires reset. 2.4.19 + rmap13 + o(1) performance is degraded. The baseline 2.4.19 hangs after a couple of attempts so no direct comparision. There are also peaks of idle time during high swap activity. ======================================== The results for the 4 GB mem test were : ======================================== kernel msg/s %CPU %sys/%user Total swpin Total swpout Total swapio ----------- ----- ---- ---------- ------------ ------------ ------------ 2.4.19 55386 99.8 1.40 0 0 0 2.4.19rmap 52330 99.5 1.43 0 2,363,388 KB 2,363,388 KB =============================== old data below=============================== 2.5.26 55446 99.4 1.40 0 0 0 2.5.35 52845 99.9 1.38 0 0 0 2.5.35mm1 52755 99.9 1.42 0 0 0 2.4.19 + o(1) using 4GB memory performs as well as 2.5.26. 2.4.19 + rmap14b + o(1) performance is down 5.5 % (52330/55386). There was swap io even though we had 500MB free mem. Bill |
From: Andrew M. <ak...@di...> - 2002-09-27 18:32:34
|
Bill Hartner wrote: > > ... > 2.5.35 44693 86.1 1.45 1,982,236 KB 5,393,152 KB 7,375,388 KB > 2.5.35mm1 39679 99.6 1.50 *2,720,600 KB *6,154,512 KB *8,875,112 KB > 2.5.35 was fairly wretched from the swapout point of view. Would be interesting to retest on 2.5.38-mm/2.5.39 sometime. (This always happens, sorry. But stuff is changing fast) |
From: Bill H. <ha...@au...> - 2002-10-02 18:54:39
|
Andrew Morton wrote: > > Bill Hartner wrote: > > > > ... > > 2.5.35 44693 86.1 1.45 1,982,236 KB 5,393,152 KB 7,375,388 KB > > 2.5.35mm1 39679 99.6 1.50 *2,720,600 KB *6,154,512 KB *8,875,112 KB > > > > 2.5.35 was fairly wretched from the swapout point of view. > Would be interesting to retest on 2.5.38-mm/2.5.39 sometime. > Here are VolanoMark results for 2.5.38 and 2.5.38-mm3 for both 3GB (memory pressure) and 4GB. I will repeat for 2.5.40 mm1 or what ever is the latest and greatest on Friday. SUT same as : http://marc.theaimsgroup.com/?l=linux-mm&m=103229747000714&w=2 NOTE : the swap device is on ServeRAID which is probably bouncing for the HIGHMEM pages in most if not all of the tests so results will likely improve when bouncing is eliminated. Need to work this problem next. 2419 = 2.4.19 + o(1) scheduler 2419rmap = 2.4.19 + rmap14b + o(1) scheduler %sys/%user = ratio of %system CPU utilization to %user CPU utilization. ======================================== The results for the 3 GB mem test were : ======================================== kernel msg/s %CPU %sys/%user Total swpin Total swpout Total swapio ----------- ----- ---- ---------- ------------ ------------ ------------ 2.5.38 46081 90.1 1.44 1,992,608 KB 2,881,056 KB 4,873,664 KB 2.5.38mm3 44950 99.8 1.52 did not collect io - /proc/stat changed =============================== old data below=============================== 2.4.19 ***** system hard hangs - requires reset. ***** 2.4.19rmap 37767 76.9 1.46 2,274,380 KB 3,800,336 KB 6,074,716 KB 2.5.26 51824 96.3 1.42 1,987,024 KB 2,148,100 KB 4,135,124 KB 2.5.26rmap 46053 90.8 1.55 3,139,324 KB 3,887,368 KB 7,026,692 KB 2.5.35 44693 86.1 1.45 1,982,236 KB 5,393,152 KB 7,375,388 KB 2.5.35mm1 39679 99.6 1.50 *2,720,600 KB *6,154,512 KB *8,875,112 KB * used pgin/pgout instead of swapin/swapout since /proc/stat changed. 2.5.38 does not perform as well as 2.5.26 (before rmap). 46081/51284 = 89.9 % or 10.1 % degradation. 2.5.38mm3 does not perform as well as 2.5.38. 44950/46081 = 97.5 % or 2.5 % degradation. CPU utilization is also higher - 99.8 vs 90.1. ======================================== The results for the 4 GB mem test were : ======================================== kernel msg/s %CPU %sys/%user Total swpin Total swpout Total swapio ----------- ----- ---- ---------- ------------ ------------ ------------ 2.5.38 53084 99.9 1.41 0 0 0 2.5.38mm3 49933 99.9 1.47 0 0 0 =============================== old data below=============================== 2.4.19 55386 99.8 1.40 0 0 0 2.4.19rmap 52330 99.5 1.43 0 2,363,388 KB 2,363,388 KB 2.5.26 55446 99.4 1.40 0 0 0 2.5.35 52845 99.9 1.38 0 0 0 2.5.35mm1 52755 99.9 1.42 0 0 0 2.5.38 does not perform as well as 2.5.26. 53084/55426 = 95.8 % or 4.2 % degradation. 2.5.38mm3 does not perform as well as 2.5.38. 49933/53084 = 94.1 % or 5.9 % degradation. Higher ratio of system CPU. Bill |
From: Andrew M. <ak...@di...> - 2002-10-02 19:36:46
|
Bill Hartner wrote: > > Andrew Morton wrote: > > > > Bill Hartner wrote: > > > > > > ... > > > 2.5.35 44693 86.1 1.45 1,982,236 KB 5,393,152 KB 7,375,388 KB > > > 2.5.35mm1 39679 99.6 1.50 *2,720,600 KB *6,154,512 KB *8,875,112 KB > > > > > > > 2.5.35 was fairly wretched from the swapout point of view. > > Would be interesting to retest on 2.5.38-mm/2.5.39 sometime. > > > > Here are VolanoMark results for 2.5.38 and 2.5.38-mm3 for both > 3GB (memory pressure) and 4GB. I will repeat for 2.5.40 mm1 or > what ever is the latest and greatest on Friday. Thanks again. > SUT same as : > > http://marc.theaimsgroup.com/?l=linux-mm&m=103229747000714&w=2 > > NOTE : the swap device is on ServeRAID which is probably bouncing for > the HIGHMEM pages in most if not all of the tests so results will > likely improve when bouncing is eliminated. Need to work this problem next. > > 2419 = 2.4.19 + o(1) scheduler > 2419rmap = 2.4.19 + rmap14b + o(1) scheduler > > %sys/%user = ratio of %system CPU utilization to %user CPU utilization. > > ======================================== > The results for the 3 GB mem test were : > ======================================== > > kernel msg/s %CPU %sys/%user Total swpin Total swpout Total swapio > ----------- ----- ---- ---------- ------------ ------------ ------------ > > 2.5.38 46081 90.1 1.44 1,992,608 KB 2,881,056 KB 4,873,664 KB > 2.5.38mm3 44950 99.8 1.52 did not collect io - /proc/stat changed That's probably due to the more aggressive promote-reads-before-writes tuning. The same is observable with the `qsbench' benchmark. > =============================== old data below=============================== > 2.4.19 ***** system hard hangs - requires reset. ***** > 2.4.19rmap 37767 76.9 1.46 2,274,380 KB 3,800,336 KB 6,074,716 KB > 2.5.26 51824 96.3 1.42 1,987,024 KB 2,148,100 KB 4,135,124 KB > 2.5.26rmap 46053 90.8 1.55 3,139,324 KB 3,887,368 KB 7,026,692 KB > 2.5.35 44693 86.1 1.45 1,982,236 KB 5,393,152 KB 7,375,388 KB > 2.5.35mm1 39679 99.6 1.50 *2,720,600 KB *6,154,512 KB *8,875,112 KB > > * used pgin/pgout instead of swapin/swapout since /proc/stat changed. > > 2.5.38 does not perform as well as 2.5.26 (before rmap). > 46081/51284 = 89.9 % or 10.1 % degradation. > > 2.5.38mm3 does not perform as well as 2.5.38. > 44950/46081 = 97.5 % or 2.5 % degradation. > CPU utilization is also higher - 99.8 vs 90.1. Yes, we're generally more eager to start swapout. > ======================================== > The results for the 4 GB mem test were : > ======================================== > > kernel msg/s %CPU %sys/%user Total swpin Total swpout Total swapio > ----------- ----- ---- ---------- ------------ ------------ ------------ > > 2.5.38 53084 99.9 1.41 0 0 0 > 2.5.38mm3 49933 99.9 1.47 0 0 0 > > =============================== old data below=============================== > 2.4.19 55386 99.8 1.40 0 0 0 > 2.4.19rmap 52330 99.5 1.43 0 2,363,388 KB 2,363,388 KB > 2.5.26 55446 99.4 1.40 0 0 0 > 2.5.35 52845 99.9 1.38 0 0 0 > 2.5.35mm1 52755 99.9 1.42 0 0 0 > > 2.5.38 does not perform as well as 2.5.26. > 53084/55426 = 95.8 % or 4.2 % degradation. > > 2.5.38mm3 does not perform as well as 2.5.38. > 49933/53084 = 94.1 % or 5.9 % degradation. Higher ratio of system CPU. Davem says that the loopback network device is currently doing an extra copy, which will go away soon. (That was news to me). I wonder if volanomark does tcp to localhost? `ifconfig lo' will tell us. |
From: Andrew M. <ak...@di...> - 2002-10-02 21:03:44
|
Andrew Morton wrote: > > ... > I wonder if volanomark does tcp to localhost? `ifconfig lo' will > tell us. OK, I googled up some kernel profiles. Volanomark does tcp to localhost. We'll need kernel profiles to take this further. |
From: Dave H. <hav...@us...> - 2002-10-02 20:59:55
|
Bill Hartner wrote: > Andrew Morton wrote: > >>Bill Hartner wrote: >> >>>... >>>2.5.35 44693 86.1 1.45 1,982,236 KB 5,393,152 KB 7,375,388 KB >>>2.5.35mm1 39679 99.6 1.50 *2,720,600 KB *6,154,512 KB *8,875,112 KB >>> >> >>2.5.35 was fairly wretched from the swapout point of view. >>Would be interesting to retest on 2.5.38-mm/2.5.39 sometime. >> > > Here are VolanoMark results for 2.5.38 and 2.5.38-mm3 for both > 3GB (memory pressure) and 4GB. I will repeat for 2.5.40 mm1 or > what ever is the latest and greatest on Friday. Could you possibly include profiling data as well? oprofile would be preferred, but readprofile would be fine if you can get it. We can guess what is causing the degredation, but profiles will offer some hard proof. -- Dave Hansen hav...@us... |
From: Bill H. <ha...@au...> - 2002-10-03 14:01:39
|
Dave Hansen wrote: > > Bill Hartner wrote: > > Andrew Morton wrote: > > > >>Bill Hartner wrote: > >> > >>>... > >>>2.5.35 44693 86.1 1.45 1,982,236 KB 5,393,152 KB 7,375,388 KB > >>>2.5.35mm1 39679 99.6 1.50 *2,720,600 KB *6,154,512 KB *8,875,112 KB > >>> > >> > >>2.5.35 was fairly wretched from the swapout point of view. > >>Would be interesting to retest on 2.5.38-mm/2.5.39 sometime. > >> > > > > Here are VolanoMark results for 2.5.38 and 2.5.38-mm3 for both > > 3GB (memory pressure) and 4GB. I will repeat for 2.5.40 mm1 or > > what ever is the latest and greatest on Friday. > > Could you possibly include profiling data as well? oprofile would be > preferred, but readprofile would be fine if you can get it. We can > guess what is causing the degredation, but profiles will offer some > hard proof. I will get 2.5.40 mm1 results and then get a profile before and after the point that we start swapping. I think that the ips driver may be bouncing here - so I would like to resolve that 1st - could change results - possibly quite a bit. Bill > > -- > Dave Hansen > hav...@us... -- IBM Linux Technology Center Performance Team bha...@au... |
From: Andrew M. <ak...@di...> - 2002-10-03 16:44:02
|
Bill Hartner wrote: > > Dave Hansen wrote: > > > > Bill Hartner wrote: > > > Andrew Morton wrote: > > > > > >>Bill Hartner wrote: > > >> > > >>>... > > >>>2.5.35 44693 86.1 1.45 1,982,236 KB 5,393,152 KB 7,375,388 KB > > >>>2.5.35mm1 39679 99.6 1.50 *2,720,600 KB *6,154,512 KB *8,875,112 KB > > >>> > > >> > > >>2.5.35 was fairly wretched from the swapout point of view. > > >>Would be interesting to retest on 2.5.38-mm/2.5.39 sometime. > > >> > > > > > > Here are VolanoMark results for 2.5.38 and 2.5.38-mm3 for both > > > 3GB (memory pressure) and 4GB. I will repeat for 2.5.40 mm1 or > > > what ever is the latest and greatest on Friday. > > > > Could you possibly include profiling data as well? oprofile would be > > preferred, but readprofile would be fine if you can get it. We can > > guess what is causing the degredation, but profiles will offer some > > hard proof. > > I will get 2.5.40 mm1 results and then get a profile before and after the > point that we start swapping. > > I think that the ips driver may be bouncing here - so I would like to > resolve that 1st - could change results - possibly quite a bit. > Profiles will tell. Bill, I'd recommend that you simply *always* generate a kernel profile. Just make it a part of the routine. They tell us so much. It's a matter of replacing test with readprofile -r test readprofile -v -m /boot/System.map | sort -n +2 |
From: Bill H. <bha...@au...> - 2002-09-18 18:45:02
|
Rik van Riel wrote: > > On Wed, 18 Sep 2002, Bill Hartner wrote: > > > I will baseline on 2.4.19 and run both the 3GB and 4GB VoloanoMark test. > > > > I will also test with rmap14a. > > I released rmap14b last night, with an SMP bugfix you'll want to have: > > http://surriel.com/patches/2.4/2.4.19-rmap14b > > > I am currently running (a) rawio on scsi devices and (b) direct io on scsi > > devices for both read and readv on 2.5.35. For this test, I am using an > > 8-way 700 Mhz with (4) IBM 4Mx controllers and 32 disks. > > Hmmm, with near certainty rmap in 2.4 still has a bunch of SMP > inefficiencies that'll slow you down on an 8-way. If these are > bothering you I'll do a backport of the 2.5 rmap speedups... I have not ran on a UP with memory pressure - could try that. VolanoMark has looooooong run queues - so I will look for a o(1) scheduler patch to lay down and then rmap14b - do you see any problem with rmap14b on top of o(1) ? Bill |