From: Shailabh N. <na...@us...> - 2001-09-07 22:28:30
|
Apologies from the scheduler team for missing out on the call. We were trying to get some numbers running on an OSDL machine for which our time slot runs out today ! The OSDLab environment has been very helpful to us for getting scalability results for the pooling extensions to the multiqueue scheduler. Watch out for our paper on the same at ALS2001 (we'll send the camera-ready version out on LSE by the end of next week). As far as future projects go, some of us at IBM are looking at ways to improve the scalability of the block I/O subsystem. In particular, we're looking at how we can eliminate the use of multiple buffer heads while reading a file page into the page cache. Another item on our agenda is splitting up the lru_list_lock. Is anyone in the group looking at these issues or interested in joining in ? For the lru_list_lock, it seems that one obvious way of splitting it up would be to make atleast as many locks as there are buffer lists (NR_LIST=4). The same concept is already being used for the "free" buffer head lists (there's one lock per buffer size). But for better scalability in the long term, it might need to be split even further (=SMP_NUM_CPUS perhaps). The changes will be mostly confined to fs/buffer.c and look like they should be relatively simple to write. I guess the only downside to attempting this is the proposed overhaul of the entire block I/O subsystem by Ben LaHaise (which might make this redundant) or if this lock isn't hot in the first place. Any comments/numbers for this lock ? I'd like to put these issues on the agenda for the next call. Regards, Shailabh Nagar Enterprise Linux Group, IBM TJ Watson Research Center (914) 945 2851, T/L 862 2851 Hanna Linder/Beaverton/IBM@IB...@li... on 09/07/2001 05:36:06 PM Sent by: lse...@li... To: lse...@li... cc: Subject: [Lse-tech] Minutes from 9/7 Con Call Thank you everyone for calling in it was a good opportunity to communicate. There were about 20 people. Mainly from OSDL, SGI, Intel, and IBM. I tried to get everyone's name so I am sorry if I missed you: Tim Witham, Nathan ?, Gerrit Huizenga, Paul Jackson, Mathew Barnes, John Hawkes, Richard Griffith, Steve Carbonari, Terry Prickett, Bill Hartner, Peter Wong, Jonathan Lahr, Hans Tannenberger, Hanna Linder, Ruth Forester, Dave Olien, Pat Gaughen, Jack Vogel, Paul McKenney A brief summary: I. Tim Witham from the Open Source Development Lab presented their new Scalable Test Platform. It looks like it is going to be a great tool. You can check your patches into their CVS tree and pick the tests and machines to run them on then get email with the results (I think that is how it works). Check out the web site at: http://www.osdlab.org/stp use login guest password guest to look around and see what they have available. They also have a sourceforge project at: http://stp.sourceforge.net II. Jonathan Lahr talked about his io_request_lock patch and some performance results should come out soon. III. Dave Olien described his I/O APIC routing patch for IA 32 systems. He assigns priorities to processes then routes interrupts to the lower priority processes. He will backport the patch from 2.4.9 to 2.4.4. and 2.4.6 for groups in IBM and Intel to apply to some of their test machines. IV. A bit of benchmarking and test results talk with some coordination of which patches to test on which systems. V. Discussion of where to go after finishing current projects (ie- which locks are next in line of high contention). Next call Friday September 21st. Please send agenda items to the list or hl...@us... by the 20th. _______________________________________________ Lse-tech mailing list Lse...@li... https://lists.sourceforge.net/lists/listinfo/lse-tech |
From: Shailabh N. <na...@us...> - 2001-09-10 13:34:09
|
Thats right. If PAGE_SIZE==blocksize, there is no problem. It is when blocksize < PAGE_SIZE that inefficiencies could arise (though I can't tell how much they show up in application performance). Shailabh Nagar Enterprise Linux Group, IBM TJ Watson Research Center (914) 945 2851, T/L 862 2851 Andi Kleen <ak...@su...>@lists.sourceforge.net on 09/07/2001 07:29:13 PM Sent by: lse...@li... To: Shailabh Nagar/Watson/IBM@IBMUS cc: Hanna Linder/Beaverton/IBM@IBMUS, lse...@li..., Kenneth Rozendal/Austin/IBM@IBMUS Subject: Re: [Lse-tech] Minutes from 9/7 Con Call On Fri, Sep 07, 2001 at 06:28:17PM -0400, Shailabh Nagar wrote: > As far as future projects go, some of us at IBM are looking at ways to > improve the scalability of the block I/O subsystem. In particular, we're > looking at how we can eliminate the use of multiple buffer heads while > reading a file page into the page cache. Another item on our agenda is This somehow doesn't sound right: Reading a single page on ext2 when PAGE_SIZE==blocksize (=on i386 with 4K blocks) should already only involve a single buffer head. -Andi _______________________________________________ Lse-tech mailing list Lse...@li... https://lists.sourceforge.net/lists/listinfo/lse-tech |
From: Andi K. <ak...@su...> - 2001-09-10 16:43:28
|
On Mon, Sep 10, 2001 at 09:33:57AM -0400, Shailabh Nagar wrote: > > Thats right. If PAGE_SIZE==blocksize, there is no problem. > It is when blocksize < PAGE_SIZE that inefficiencies could arise (though I > can't tell how much they show up in application performance). AFAIK nobody uses blocksizes < page size at least on i386 anymore. ext2 defaults to 4K for any reasonably sized filesystems. Many filesystems do not even support smaller block sizes (e.g. XFS, JFS, reiserfs don't) On IA64/alpha/.. with bigger page sizes it is probably different though; but I did not see any problems with them yet. A bigger problem is probably the buffer head overhead for raw IO. -Andi |
From: Mala A. <ma...@us...> - 2001-09-11 13:03:21
|
> Dave Olien described his I/O APIC routing patch for IA 32 > systems. He assigns priorities to processes then routes interrupts > to the lower priority processes. He will backport the patch from > 2.4.9 to 2.4.4. and 2.4.6 for groups in IBM and Intel to apply to some > of their test machines Dave Olien has agreed to send us his patch ported to 2.4.4 kernel to test it on our netperf3 workload. I will be posting the results to lse-tech. Right now my netperf3 (updated version) workload stresses tcp streaming using 4 adapters on both the server (4-way) and the client (8-way). Regards, Mala Mala Anand E-mail:ma...@us... Linux Technology Center - Performance Phone:838-8088; Tie-line:678-8088 Hanna Linder/Beaverton/IBM@IB...@li... on 09/07/2001 04:36:06 PM Sent by: lse...@li... To: lse...@li... cc: Subject: [Lse-tech] Minutes from 9/7 Con Call Thank you everyone for calling in it was a good opportunity to communicate. There were about 20 people. Mainly from OSDL, SGI, Intel, and IBM. I tried to get everyone's name so I am sorry if I missed you: Tim Witham, Nathan ?, Gerrit Huizenga, Paul Jackson, Mathew Barnes, John Hawkes, Richard Griffith, Steve Carbonari, Terry Prickett, Bill Hartner, Peter Wong, Jonathan Lahr, Hans Tannenberger, Hanna Linder, Ruth Forester, Dave Olien, Pat Gaughen, Jack Vogel, Paul McKenney A brief summary: I. Tim Witham from the Open Source Development Lab presented their new Scalable Test Platform. It looks like it is going to be a great tool. You can check your patches into their CVS tree and pick the tests and machines to run them on then get email with the results (I think that is how it works). Check out the web site at: http://www.osdlab.org/stp use login guest password guest to look around and see what they have available. They also have a sourceforge project at: http://stp.sourceforge.net II. Jonathan Lahr talked about his io_request_lock patch and some performance results should come out soon. III. Dave Olien described his I/O APIC routing patch for IA 32 systems. He assigns priorities to processes then routes interrupts to the lower priority processes. He will backport the patch from 2.4.9 to 2.4.4. and 2.4.6 for groups in IBM and Intel to apply to some of their test machines. IV. A bit of benchmarking and test results talk with some coordination of which patches to test on which systems. V. Discussion of where to go after finishing current projects (ie- which locks are next in line of high contention). Next call Friday September 21st. Please send agenda items to the list or hl...@us... by the 20th. _______________________________________________ Lse-tech mailing list Lse...@li... https://lists.sourceforge.net/lists/listinfo/lse-tech |
From: Alan C. <al...@lx...> - 2001-09-11 22:48:01
|
> that Jonathon's small patch goes a long way towards resolving the most > significant performance bottlenecks without a full subsystem rework, > thereby dramatically reducing the risk of regression in a 2.4.X > kernel. I am very concerned about the stability of such a change. What would be useful data would be to know how much of the time that lock is held is by the drivers not by the request queue code. Alan |
From: Jonathan L. <la...@us...> - 2001-09-12 19:56:27
|
Alan Cox [al...@lx...] wrote: > > that Jonathon's small patch goes a long way towards resolving the most > > significant performance bottlenecks without a full subsystem rework, > > thereby dramatically reducing the risk of regression in a 2.4.X > > kernel. > > I am very concerned about the stability of such a change. What would be > useful data would be to know how much of the time that lock is held is by > the drivers not by the request queue code. Lock contention measurements using lockmeter indicate that block i/o subsystem function __make_request is the primary source of io_request_lock contention. (see http://marc.theaimsgroup.com/?l=lse-tech&m=100017134609802&w=2) My io_request_lock patch changes __make_request to use a per device queue lock (q->queue_lock) instead of io_request_lock without reducing the scope of other io_request_lock usage. Device drivers are not changed nor are they required to change by that patch. Please let me know if can address any other questions. Jonathan -- Jonathan Lahr IBM Linux Technology Center Beaverton, Oregon la...@us... 503-578-3385 |
From: Benjamin L. <bc...@re...> - 2001-09-13 00:58:46
|
On Wed, Sep 12, 2001 at 12:49:34PM -0700, Jonathan Lahr wrote: > Lock contention measurements using lockmeter indicate that block i/o > subsystem function __make_request is the primary source of io_request_lock > contention. > (see http://marc.theaimsgroup.com/?l=lse-tech&m=100017134609802&w=2) > My io_request_lock patch changes __make_request to use a per device queue > lock (q->queue_lock) instead of io_request_lock without reducing the > scope of other io_request_lock usage. Device drivers are not changed nor > are they required to change by that patch. My immediate suspicion is not that the lock itself is the problem, but that the insertion method used in the block request layer, combined with the large queue length is resulting in O(n^2) or worse behaviour. Of course, fixing that is even worse for stability. Keep in mind that the last time the request code was changed, it resulted in a subtle bug that wasn't corrected until ~2.4.8. -ben |
From: Jonathan L. <la...@us...> - 2001-09-13 19:18:51
|
Benjamin LaHaise [bc...@re...] wrote: > On Wed, Sep 12, 2001 at 12:49:34PM -0700, Jonathan Lahr wrote: > > Lock contention measurements using lockmeter indicate that block i/o > > subsystem function __make_request is the primary source of io_request_lock > > contention. > > (see http://marc.theaimsgroup.com/?l=lse-tech&m=100017134609802&w=2) > > My io_request_lock patch changes __make_request to use a per device queue > > lock (q->queue_lock) instead of io_request_lock without reducing the > > scope of other io_request_lock usage. Device drivers are not changed nor > > are they required to change by that patch. > > My immediate suspicion is not that the lock itself is the problem, but > that the insertion method used in the block request layer, combined with > the large queue length is resulting in O(n^2) or worse behaviour. Of > course, fixing that is even worse for stability. Keep in mind that the > last time the request code was changed, it resulted in a subtle bug > that wasn't corrected until ~2.4.8. I haven't studied the insertion code, but I believe a lot of the i/o performance gain from this patch is due to the concurrency it allows in __make_request. On multiprocessor machines, i/o request submissions can now be processed in parallel. It has been made extraordinarily clear that changes to 2.4 io_request_lock locking are considered risky. I am still exploring this problem only because io_request_lock contention is showing up as a major i/o performance limitation. I consider this patch conservative with respect to changing existing locking. It reduces the scope of io_request_lock locking only in __make_request. I believe the warnings of the risks involved in changing io_request_lock are fully justified, but it still seems reasonable to consider both the risks and the gains when evaluating a patch. Jonathan -- Jonathan Lahr IBM Linux Technology Center Beaverton, Oregon la...@us... 503-578-3385 |
From: Jens A. <ax...@su...> - 2001-09-15 22:27:30
|
On Thu, Sep 13 2001, Jonathan Lahr wrote: > I haven't studied the insertion code, but I believe a lot of the i/o > performance gain from this patch is due to the concurrency it allows > in __make_request. On multiprocessor machines, i/o request submissions > can now be processed in parallel. So you're just guessing now? You are attacking this problem from the wrong angle. > It has been made extraordinarily clear that changes to 2.4 io_request_lock > locking are considered risky. I am still exploring this problem only > because io_request_lock contention is showing up as a major i/o performance Changes in the order of what you presented are of course frowned upon, since _they break stuff_. > limitation. I consider this patch conservative with respect to changing > existing locking. It reduces the scope of io_request_lock locking only > in __make_request. Please start listening! You patch is broken, I've told you so many times it's not even funny anymore. > I believe the warnings of the risks involved in changing io_request_lock > are fully justified, but it still seems reasonable to consider both the > risks and the gains when evaluating a patch. I'm afraid that reducing io_request_lock scope in 2.4 will always be too great a risk. -- Jens Axboe |
From: Jonathan L. <la...@us...> - 2001-09-17 18:36:57
|
> > I haven't studied the insertion code, but I believe a lot of the i/o > > performance gain from this patch is due to the concurrency it allows > > in __make_request. On multiprocessor machines, i/o request submissions > > can now be processed in parallel. > > So you're just guessing now? You are attacking this problem from the > wrong angle. Allowing multiple processors to concurrently submit i/o requests to multiple device queues rather than unnecessarily serializing all i/o request submissions is generally a good idea for performance. > > It has been made extraordinarily clear that changes to 2.4 io_request_lock > > locking are considered risky. I am still exploring this problem only > > because io_request_lock contention is showing up as a major i/o performance > > Changes in the order of what you presented are of course frowned upon, > since _they break stuff_. ... > Please start listening! You patch is broken, I've told you so many times > it's not even funny anymore. The patch has been run on IDE and SCSI systems with heavy i/o loads without data corruption or deadlocks. Describe in detail, not generalizations, how it's broken. > I'm afraid that reducing io_request_lock scope in 2.4 will always be too > great a risk. If the performance gain is high enough, some risk should be accepted. See http://marc.theaimsgroup.com/?l=lse-tech&m=100017134609802&w=2. Jonathan -- Jonathan Lahr IBM Linux Technology Center Beaverton, Oregon la...@us... 503-578-3385 |
From: Andrea A. <an...@su...> - 2001-09-17 18:45:53
|
On Mon, Sep 17, 2001 at 11:30:52AM -0700, Jonathan Lahr wrote: > data corruption or deadlocks. Describe in detail, not generalizations, how > it's broken. I recall Jens did the same patch as you (just move the lock from global to per-queue) last year, you can find it on l-k, Manfred pointed out it could have lock inversions with the different ide subchannels for example. Other than those subtle deadlocks (that make it obviously not suitable for production) I can't see much other problems. I assume those subtle problems is what make Jens not happy about it. Andrea |
From: Jonathan L. <la...@us...> - 2001-09-17 19:53:13
|
> I recall Jens did the same patch as you (just move the lock from global > to per-queue) last year, I have not seen Jens' previous patch, but based on your description, this patch is dissimilar. Except in __make_request, whenever the queue_lock is required to protect the queue, queue_lock is taken *in addition to* io_request_lock, rather than instead of io_request_lock. > you can find it on l-k, Manfred pointed out it > could have lock inversions with the different ide subchannels for > example. Other than those subtle deadlocks (that make it obviously not > suitable for production) I can't see much other problems. I assume those > subtle problems is what make Jens not happy about it. Thank you for examining the patch. I originally posted the patch to explore the feasibility and possible performance gains for server environments which would probably exclude IDE. I will look further into IDE correctness. Jonathan -- Jonathan Lahr IBM Linux Technology Center Beaverton, Oregon la...@us... 503-578-3385 |
From: Andrea A. <an...@su...> - 2001-09-17 22:06:10
|
On Mon, Sep 17, 2001 at 12:45:46PM -0700, Jonathan Lahr wrote: > > > I recall Jens did the same patch as you (just move the lock from global > > to per-queue) last year, > > I have not seen Jens' previous patch, but based on your description, > this patch is dissimilar. Except in __make_request, whenever the well, possibly yes, I didn't looked your patch very well yet (I will try soon to have a closer look), I mosty read the emails and I guessed what the patch is doing based on when I read previous Jens's patch last year. > environments which would probably exclude IDE. I will look further Strictly speaking today I wouldn't exclude IDE from servers but I see what you mean. Andrea |
From: Alan C. <al...@lx...> - 2001-09-17 18:47:21
|
> Allowing multiple processors to concurrently submit i/o requests to > multiple device queues rather than unnecessarily serializing all i/o > request submissions is generally a good idea for performance. Except when the layers rely on the same lock for multiple purposes. The io_request lock isnt just a queue lock, its internal locking in some drivers and since we sometimes use it for layered drivers (ataraid, md, lvm..) its asking for problems. > If the performance gain is high enough, some risk should be accepted. > See http://marc.theaimsgroup.com/?l=lse-tech&m=100017134609802&w=2. The best way to handle such things is to shove them in early 2.5, then back port if they prove ok |
From: Jonathan L. <la...@us...> - 2001-09-17 19:42:50
|
Alan Cox [al...@lx...] wrote: > > Allowing multiple processors to concurrently submit i/o requests to > > multiple device queues rather than unnecessarily serializing all i/o > > request submissions is generally a good idea for performance. > > Except when the layers rely on the same lock for multiple purposes. The > io_request lock isnt just a queue lock, its internal locking in some drivers > and since we sometimes use it for layered drivers (ataraid, md, lvm..) its > asking for problems. I agree it is clear that io_request_lock is used for multiple purposes by block i/o, scsi-mid, and drivers. One purpose for io_request_lock is to protect the request queue (queue_head). The patch changes protection of the queue by having __make_request, the producer of the queue, and scsi_request_fn, the consumer of the queue, use q->queue_lock instead of io_request_lock. The scope of io_request_lock is left intact elsewhere except for scsi_request_fn, in which case is it increased. This leaves other existing io_request_lock serialization in place. So, unless a driver accesses the request queue without taking q->queue_lock, queue integrity is protected. > > If the performance gain is high enough, some risk should be accepted. > > See http://marc.theaimsgroup.com/?l=lse-tech&m=100017134609802&w=2. > > The best way to handle such things is to shove them in early 2.5, then > back port if they prove ok In this particular case, that not possible since Jens is removing io_request_lock altogether in 2.5 which I think is a good idea after studying it's use in 2.4. -- Jonathan Lahr IBM Linux Technology Center Beaverton, Oregon la...@us... 503-578-3385 |
From: Alan C. <al...@lx...> - 2001-09-17 21:24:27
|
> elsewhere except for scsi_request_fn, in which case is it increased. > This leaves other existing io_request_lock serialization in place. > So, unless a driver accesses the request queue without taking q->queue_lock, > queue integrity is protected. Well drivers do. io_request_lock in the IDE layer synchronizes the ide internal request queue against the queue_lock. If you take queue_lock insode of io_request_lock you seem to have IDE deadlocks |
From: Jonathan L. <la...@us...> - 2001-09-17 21:50:39
|
Alan Cox [al...@lx...] wrote: > > elsewhere except for scsi_request_fn, in which case is it increased. > > This leaves other existing io_request_lock serialization in place. > > So, unless a driver accesses the request queue without taking q->queue_lock, > > queue integrity is protected. > > Well drivers do. io_request_lock in the IDE layer synchronizes the ide > internal request queue against the queue_lock. If you take queue_lock > insode of io_request_lock you seem to have IDE deadlocks __make_request is the only function in which io_request_lock serialization is removed and replaced with q->queue_lock. I would be interested in knowing exactly what code now conflicts with __make_request in accessing the request queue. I have seen no deadlocks in testing on machines running both IDE and SCSI simultaneously. -- Jonathan Lahr IBM Linux Technology Center Beaverton, Oregon la...@us... 503-578-3385 |
From: Jens A. <ax...@su...> - 2001-09-18 06:13:36
|
On Mon, Sep 17 2001, Jonathan Lahr wrote: > > > I haven't studied the insertion code, but I believe a lot of the i/o > > > performance gain from this patch is due to the concurrency it allows > > > in __make_request. On multiprocessor machines, i/o request submissions > > > can now be processed in parallel. > > > > So you're just guessing now? You are attacking this problem from the > > wrong angle. > > Allowing multiple processors to concurrently submit i/o requests to > multiple device queues rather than unnecessarily serializing all i/o > request submissions is generally a good idea for performance. Well duh > > > It has been made extraordinarily clear that changes to 2.4 io_request_lock > > > locking are considered risky. I am still exploring this problem only > > > because io_request_lock contention is showing up as a major i/o performance > > > > Changes in the order of what you presented are of course frowned upon, > > since _they break stuff_. > ... > > Please start listening! You patch is broken, I've told you so many times > > it's not even funny anymore. > > The patch has been run on IDE and SCSI systems with heavy i/o loads without > data corruption or deadlocks. Describe in detail, not generalizations, how > it's broken. No, listen. You tell me why it's safe to remove io_request_lock in IDE for instance. Read the code carefully and justify why it's safe. If you can do that, then I can begin to trust that you know what you are doing here. > > I'm afraid that reducing io_request_lock scope in 2.4 will always be too > > great a risk. > > If the performance gain is high enough, some risk should be accepted. Wrong -- Jens Axboe |
From: Jonathan L. <la...@us...> - 2001-09-18 18:42:25
|
> > The patch has been run on IDE and SCSI systems with heavy i/o loads without > > data corruption or deadlocks. Describe in detail, not generalizations, how > > it's broken. > > No, listen. You tell me why it's safe to remove io_request_lock in IDE > for instance. Read the code carefully and justify why it's safe. If you > can do that, then I can begin to trust that you know what you are doing > here. Please bear in mind that this preliminary patch was posted as a snapshot of an exploration into the costs and gains of one approach to reducing io_request_lock contention. It has shown significant gains, e.g. io_request_lock utilization/contention was 61.8%/63.6% before the patch and is 10.7%/11.1% after the patch, and performance improved by 20%. (http://marc.theaimsgroup.com/?l=lse-tech&m=100017134609802&w=2) Since the patch produced positive preliminary results, I am developing it further. However, I acknowledge that even though the current patch has not exhibited locking problems, problems could nevertheless exist which would necessitate further changes. At this point, my intent is only to continue to assess the gains and costs (code impacts) of this approach. I appreciate the feedback that has been provided. Jonathan -- Jonathan Lahr IBM Linux Technology Center Beaverton, Oregon la...@us... 503-578-3385 |
From: Alan C. <al...@lx...> - 2001-09-18 19:15:03
|
> it further. However, I acknowledge that even though the current patch has > not exhibited locking problems, problems could nevertheless exist which > would necessitate further changes. At this point, my intent is only to > continue to assess the gains and costs (code impacts) of this approach. I dont think anyone is going to dispute the data - its made it very obvious there is a real problem there |
From: Jens A. <ax...@su...> - 2001-09-15 22:27:25
|
On Wed, Sep 12 2001, Benjamin LaHaise wrote: > On Wed, Sep 12, 2001 at 12:49:34PM -0700, Jonathan Lahr wrote: > > Lock contention measurements using lockmeter indicate that block i/o > > subsystem function __make_request is the primary source of io_request_lock > > contention. > > (see http://marc.theaimsgroup.com/?l=lse-tech&m=100017134609802&w=2) > > My io_request_lock patch changes __make_request to use a per device queue > > lock (q->queue_lock) instead of io_request_lock without reducing the > > scope of other io_request_lock usage. Device drivers are not changed nor > > are they required to change by that patch. > > My immediate suspicion is not that the lock itself is the problem, but > that the insertion method used in the block request layer, combined with > the large queue length is resulting in O(n^2) or worse behaviour. Of > course, fixing that is even worse for stability. Keep in mind that the > last time the request code was changed, it resulted in a subtle bug > that wasn't corrected until ~2.4.8. Ben is spot on. Jonathan, instead of wasting your time with the per-queue lock stuff you are doing now, I would suggest you spend some time checking how much of the contention is due to insertion scans and how much is due to merge scans. The latter could possibly be fixed in 2.4 (it's fixed completely in the bio patches). -- Jens Axboe |
From: Jonathan L. <la...@us...> - 2001-09-17 18:42:04
|
> I would suggest you spend some > time checking how much of the contention is due to insertion scans and > how much is due to merge scans. I have already eliminated the contention problem by increasing concurrency through queue-specific locking. Lockmeter measurements with the patch applied indicate no need to examine the insertion code. -- Jonathan Lahr IBM Linux Technology Center Beaverton, Oregon la...@us... 503-578-3385 |
From: Jens A. <ax...@su...> - 2001-09-18 06:16:26
|
On Mon, Sep 17 2001, Jonathan Lahr wrote: > > > I would suggest you spend some > > time checking how much of the contention is due to insertion scans and > > how much is due to merge scans. > > I have already eliminated the contention problem by increasing concurrency > through queue-specific locking. Lockmeter measurements with the patch > applied indicate no need to examine the insertion code. "if an algorithm is broken, just fine grain the locking and everything is perfect" That may be how you work elsewhere, but in Linux we like to do things the other way around. Ok, let me help you out here. If it just so happens that merging is what is eating lots of CPU cycles and thus the primary contention problem, then we can probably fix that _without_ changing the locking scope. -- Jens Axboe |
From: Jonathan L. <la...@us...> - 2001-09-18 17:37:52
|
Jens Axboe [ax...@su...] wrote: > On Mon, Sep 17 2001, Jonathan Lahr wrote: > > > > > I would suggest you spend some > > > time checking how much of the contention is due to insertion scans and > > > how much is due to merge scans. > > > > I have already eliminated the contention problem by increasing concurrency > > through queue-specific locking. Lockmeter measurements with the patch > > applied indicate no need to examine the insertion code. > > "if an algorithm is broken, just fine grain the locking and everything > is perfect" > > That may be how you work elsewhere, but in Linux we like to do things > the other way around. Ok, let me help you out here. If it just so > happens that merging is what is eating lots of CPU cycles and thus the > primary contention problem, then we can probably fix that _without_ > changing the locking scope. I agree that contention could be reduced also by improving algorithmic efficiency. I'll look into that as well as an alternative to locking scope changes. The narrower scope of merge code improvement seems less risky than pervasive locking changes, but I'm not sure as much performance improvement can be made there. Thanks for your comments. Jonathan -- Jonathan Lahr IBM Linux Technology Center Beaverton, Oregon la...@us... 503-578-3385 |
From: Jens A. <ax...@su...> - 2001-09-15 22:27:25
|
On Wed, Sep 12 2001, Jonathan Lahr wrote: > Alan Cox [al...@lx...] wrote: > > > that Jonathon's small patch goes a long way towards resolving the most > > > significant performance bottlenecks without a full subsystem rework, > > > thereby dramatically reducing the risk of regression in a 2.4.X > > > kernel. > > > > I am very concerned about the stability of such a change. What would be > > useful data would be to know how much of the time that lock is held is by > > the drivers not by the request queue code. > > Lock contention measurements using lockmeter indicate that block i/o > subsystem function __make_request is the primary source of io_request_lock > contention. elevator_linus_merge would be the primary source, really. > My io_request_lock patch changes __make_request to use a per device queue > lock (q->queue_lock) instead of io_request_lock without reducing the > scope of other io_request_lock usage. Device drivers are not changed nor > are they required to change by that patch. Since when? Sorry, but now you're just talking crap. Last version of this patch I saw did indeed require substantial changes, and only worked on a very limited cut of the kernel drivers. -- Jens Axboe |