Share

Direct Rendering Infrastructure

Email Archive: dri-devel (read-only)

1999:
Jan
   
Feb
   
Mar
   
Apr
   
May
   
Jun
   
Jul
   
Aug
   
Sep
   
Oct
   
Nov
   
Dec
(115)
2000:
Jan
(143)
Feb
(177)
Mar
(390)
Apr
(285)
May
(316)
Jun
(241)
Jul
(303)
Aug
(504)
Sep
(322)
Oct
(368)
Nov
(398)
Dec
(474)
2001:
Jan
(734)
Feb
(712)
Mar
(736)
Apr
(358)
May
(403)
Jun
(317)
Jul
(286)
Aug
(299)
Sep
(304)
Oct
(398)
Nov
(169)
Dec
(313)
2002:
Jan
(406)
Feb
(506)
Mar
(520)
Apr
(629)
May
(714)
Jun
(711)
Jul
(761)
Aug
(665)
Sep
(542)
Oct
(713)
Nov
(641)
Dec
(639)
2003:
Jan
(468)
Feb
(748)
Mar
(781)
Apr
(812)
May
(789)
Jun
(731)
Jul
(567)
Aug
(579)
Sep
(624)
Oct
(647)
Nov
(387)
Dec
(422)
2004:
Jan
(592)
Feb
(630)
Mar
(514)
Apr
(457)
May
(647)
Jun
(388)
Jul
(276)
Aug
(528)
Sep
(840)
Oct
(831)
Nov
(350)
Dec
(458)
2005:
Jan
(584)
Feb
(654)
Mar
(706)
Apr
(229)
May
(411)
Jun
(594)
Jul
(341)
Aug
(435)
Sep
(251)
Oct
(297)
Nov
(196)
Dec
(286)
2006:
Jan
(295)
Feb
(378)
Mar
(300)
Apr
(204)
May
(241)
Jun
(316)
Jul
(256)
Aug
(346)
Sep
(338)
Oct
(352)
Nov
(288)
Dec
(272)
2007:
Jan
(194)
Feb
(242)
Mar
(329)
Apr
(357)
May
(254)
Jun
(309)
Jul
(291)
Aug
(370)
Sep
(279)
Oct
(336)
Nov
(357)
Dec
(465)
2008:
Jan
(396)
Feb
(370)
Mar
(407)
Apr
(350)
May
(337)
Jun
(339)
Jul
(352)
Aug
(314)
Sep
(338)
Oct
(299)
Nov
(279)
Dec
(365)
2009:
Jan
(596)
Feb
(601)
Mar
(588)
Apr
(542)
May
(731)
Jun
(701)
Jul
(673)
Aug
(1050)
Sep
(740)
Oct
(750)
Nov
(521)
Dec
   
From: Thomas Hellström <thomas@tu...> - 2008-05-14 15:20
Ben Skeggs wrote:
>>>> 1) I feel there hasn't been enough open driver coverage to prove it. So far
>>>> we have done an Intel IGD, we have a lot of code that isn't required for
>>>> these devices, so the question of how much code exists purely to support
>>>> poulsbo closed source userspace there is and why we need to live with it.
>>>> Both radeon and nouveau developers have expressed frustration about the
>>>> fencing internals being really hard to work with which doesn't bode well for
>>>> maintainability in the future.
>>>>
>>>>
>>> OK. So basically what I'm asking is that when we have full-feathered open
>>> source drivers available that
>>> utilize TTM, either as part of DRM core, or, if needed, as part of
>>> driver-specific code, do you see anything
>>> else that prevents that from being pushed? That would be very valuable to know
>>> for anyone starting porting work. ?
>>>
>> I was hoping that by now, one of the radeon or nouveau drivers would have
>> adopted TTM, or at least demoed something working using it, this hasn't
>> happened which worries me, perhaps glisse or darktama could fill in on
>> what limited them from doing it. The fencing internals are very very scary
>> and seem to be a major stumbling block.
>>
> The fencing internals do seem overly complicated indeed, but that's
> something that I'm personally OK with taking the time to figure out how
> to get right. Is there any good documentation around that describes it
> in detail?
>
Yes, there is a wiki page.
http://dri.freedesktop.org/wiki/TTMFencing
> I actually started working on nouveau/ttm again a month or so back, with
> the intention of actually having the work land this time. Overall, I
> don't have much problem with TTM and would be willing to work with it.
> Supporting G8x/G9x chips was the reason the work's stalled again, I
> wasn't sure at the time what requirements we'd have from a memory
> manager.
>
> The issue on G8x is that the 3D engine will refuse to render to linear
> surfaces, and in order to setup tiling we need to make use of a
> channel's page tables. The driver doesn't get any control when VRAM is
> allocated so that it can setup the page tables appropriately etc. I
> just had a thought that the driver-specific validation ioctl could
> probably handle that at the last minute, so perhaps that's also not an
> issue. I'll look more into G8x/ttm after I finish my current G8x work.
>
> Another minor issue (probably doesn't effect merging?): Nouveau makes
> extensive use fence classes, we assign 1 fence class to each GPU channel
> (read: context + command submission mechanism). We have 128 of these on
> G80 cards, the current _DRM_FENCE_CLASSES is 8 which is insufficient
> even for NV1x hardware.
>
Ouch. Yes it should be OK to bump that as long as kmalloc doesn't complain.
> So overall, I'm basically fine with TTM now that I've actually made a
> proper attempt at using it.. GEM does seem interesting, I'll also
> follow its development while I continue with other non-mm G80 work.
>
> Cheers,
> Ben.
>
Nice to know Ben. Anyway whatever happens, the fencing code will remain
for some drivers either device specific or common, so if you find ways
to simplify or things that doesn't look right, please let me know.

/Thomas


    From: Keith Packard <keithp@ke...> - 2008-05-14 15:52

    Attachments: Message as HTML     
    On Wed, 2008-05-14 at 12:09 +0200, Thomas Hellström wrote:

    > 1) The inability to map device memory. The design arguments and proposed
    > solution for VRAM are not really valid. Think of this, probably not too
    > uncommon, scenario of a single pixel fallback composite to a scanout
    > buffer in vram. Or a texture or video frame upload:

    Nothing prevents you from mapping device memory; it's just that on a UMA
    device, there's no difference, and some significant advantages to using
    the direct mapping. I wrote the API I needed for my device; I think it's
    simple enough that other devices can add the APIs they need.

    But, what we've learned in the last few months is that mapping *any*
    pages into user space is a last-resort mechanism. Mapping pages WC or UC
    requires inter-processor interrupts, and using normal WB pages means
    invoking clflush on regions written from user space.

    The glxgears "benchmark" demonstrates this with some clarity -- using
    pwrite to send batch buffers is nearly three times faster (888 fps using
    pwrite vs 300 fps using mmap) than mapping pages to user space and then
    clflush'ing them in the kernel.

    > A) Page in all GEM pages, because they've been paged out.
    > B) Copy the complete scanout buffer to GEM because it's dirty. Untile.
    > C) Write the pixel.
    > D) Copy the complete buffer back while tiling.

    First off, I don't care about fallbacks; any driver using fallbacks is
    broken.

    Second, if you had to care about fallbacks on non-UMA hardware, you'd
    compute the pages necessary for the fallback and only map/copy those
    anyway.

    > 2) Reserving pages when allocating VRAM buffers is also a very bad
    > solution particularly on systems with a lot of VRAM and little system
    > RAM. (Multiple card machines?). GEM basically needs to reserve
    > swap-space when buffers are created, and put a limit on the pinned
    > physical pages. We basically should not be able to fail memory
    > allocation during execbuf, because we cannot recover from that.

    As far as I know, any device using VRAM will not save it across
    suspend/resume. From my perspective, this means you don't get a choice
    about allocating backing store for that data

    Because GEM has backing store, we can limit pinned memory to only those
    pages needed for the current operation, waiting to pin pages until the
    device is ready to execute the operation. As I said in my earlier email,
    that part of the kernel driver is not written yet. I was hoping to get
    that finished before launching into this discussion as it is always
    better to argue with running code.

    > This means that the dependency on SHMEMFS propably needs to be dropped
    > and replaced with some sort of DRMFS that allows overloading of mmap and
    > a correct swap handling, address the caching issue and also avoids the
    > driver do_mmap().

    Because GEM doesn't expose the use of shmfs to the user, there's no
    requirement that all objects use this abstraction. You could even have
    multiple object creation functions if that made sense in your driver.

    --
    keith.packard@in...

    From: Jerome Glisse <glisse@fr...> - 2008-05-14 17:09
    On Wed, 14 May 2008 16:36:54 +0200
    Thomas Hellström <thomas@tu...> wrote:

    > Jerome Glisse wrote:
    > I don't agree with you here. EXA is much faster for small composite
    > operations and even small fill blits if fallbacks are used. Even to
    > write-combined memory, but that of course depends on the hardware. This
    > is going to be even more pronounced with acceleration architectures like
    > Glucose and similar, that don't have an optimized path for small
    > hardware composite operations.
    >
    > My personal feeling is that pwrites are a workaround for a workaround
    > for a very bad decision:
    >
    > To avoid user-space allocators on device-mapped memory. This lead to a
    > hack to avoid cahing-policy changes which lead to cache trashing
    > problems which put us in the current situation. How far are we going to
    > follow this path before people wake up? What's wrong with the
    > performance of good old i915tex which even beats "classic" i915 in many
    > cases.
    >
    > Having to go through potentially (and even probably) paged-out memory to
    > access buffers to make that are present in VRAM sounds like a very odd
    > approach (to say the least) to me. Even if it's a single page and
    > implementing per-page dirty checks for domain flushing isn't very
    > appealing either.

    I don't have number or benchmark to check how fast pread/pwrite path might
    be in this use so i am just expressing my feeling which happen to just be
    to avoid vma tlb flush as most as we can. I got the feeling that kernel
    goes through numerous trick to avoid tlb flushing for a good reason and
    also i am pretty sure that with number of core keeping growing anythings
    that need cpu broad synchronization is to be avoided.

    Hopefully once i got decent amount of time to do benchmark with gem i will
    check out my theory. I think simple benchmark can be done on intel hw just
    return false in EXA prepare access to force use of download from screen,
    and in download from screen use pread then comparing benchmark of this
    hacked intel ddx with a normal one should already give some numbers.

    > Why should we have to when we can do it right?

    Well my point was that mapping vram is not right, i am not saying that
    i know the truth. It's just a feeling based on my experiment with ttm
    and on the bar restriction stuff and others consideration of same kind.

    > No. Gem can't coop with it. Let's say you have a 512M system with two 1G
    > video cards, 4G swap space, and you want to fill both card's videoram
    > with render-and-forget textures for whatever purpose.
    >
    > What happens? After you've generated the first say 300M, The system
    > mysteriously starts to page, and when, after a a couple of minutes of
    > crawling texture upload speeds, you're done, The system is using and
    > have written almost 2G of swap. Now, you want to update the textures and
    > expect fast texsubimage...
    >
    > So having a backing object that you have to access to get things into
    > VRAM is not the way to go.
    > The correct way to do this is to reserve, but not use swap space. Then
    > you can start using it on suspend, provided that the swapping system is
    > still up (which is has to be with the current GEM approach anyway). If
    > pwrite is used in this case, it must not dirty any backing object pages.
    >

    For normal desktop i don't expect VRAM amount > RAM amount, people with
    1Go VRAM are usually hard gamer with 4G of ram :). Also most object in
    3d world are stored in memory, if program are not stupid and trust gl
    to keep their texture then you just have the usual ram copy and possibly
    a vram copy, so i don't see any waste in the normal use case. Of course
    we can always come up with crazy weird setup, but i am more interested
    in dealing well with average Joe than dealing mostly well with every
    use case.

    That said i do see GPGPU as a possible users of temporary big vram buffer
    ie buffer you can trash away. For that kind of stuff it does make sense
    to not have backing ram/swap area. But i would rather add somethings in
    gem like intercepting allocation of such buffer and not creating backing
    buffer, or adding driver specific ioctl for that case.

    Anyway i think we need benchmark to know what in the end is really the
    best option. I don't have code to support my general feeling, so i might
    be wrong. Sadly we don't have 2^32 monkeys doing code days and night for
    drm to test all solutions :)

    Cheers,
    Jerome Glisse <glisse@fr...>

    From: Eric Anholt <eric@an...> - 2008-05-14 17:53

    Attachments: Message as HTML     
    On Wed, 2008-05-14 at 16:36 +0200, Thomas Hellström wrote:
    > >> 2) Reserving pages when allocating VRAM buffers is also a very bad
    > >> solution particularly on systems with a lot of VRAM and little system
    > >> RAM. (Multiple card machines?). GEM basically needs to reserve
    > >> swap-space when buffers are created, and put a limit on the pinned
    > >> physical pages. We basically should not be able to fail memory
    > >> allocation during execbuf, because we cannot recover from that.
    > >>
    > >
    > > Well this solve the suspend problem we were discussing at xds ie what
    > > to do on buffer. If we know that we have room to put buffer then we
    > > don't to worry about which buffer we are ready to loose. Given that
    > > opengl don't give any clue on that this sounds like a good approach.
    > >
    > > For embedded device where every piece of ram still matter i guess
    > > you also have to deal with suspend case so you have a way to either
    > > save vram content or to preserve it. I don't see any problem with
    > > gem to cop with this case too.
    > >
    > No. Gem can't coop with it. Let's say you have a 512M system with two 1G
    > video cards, 4G swap space, and you want to fill both card's videoram
    > with render-and-forget textures for whatever purpose.

    Who's selling that system? Who's building that system at home?

    --
    Eric Anholt anholt@Fr...
    eric@an... eric.anholt@in...

    From: Eric Anholt <eric@an...> - 2008-05-14 18:31

    Attachments: Message as HTML     
    On Wed, 2008-05-14 at 02:33 +0200, Thomas Hellström wrote:
    > > The real question is whether TTM suits the driver writers for use in Linux
    > > desktop and embedded environments, and I think so far I'm not seeing
    > > enough positive feedback from the desktop side.
    > >
    > I actually haven't seen much feedback at all. At least not on the
    > mailing lists.
    > Anyway we need to look at the alternatives which currently is GEM.
    >
    > GEM, while still in development basically brings us back to the
    > functionality of TTM 0.1, with added paging support but without
    > fine-grained locking and caching policy support.
    >
    > I might have misunderstood things but quickly browsing the code raises
    > some obvious questions:
    >
    > 1) Some AGP chipsets don't support page addresses > 32bits. GEM objects
    > use GFP_HIGHUSER, and it's hardcoded into the linux swap code.

    The obvious solution here is what many DMA APIs do for IOMMUs that can't
    address all of memory -- keep a pool of pages within the addressable
    range and bounce data through them. I think the Linux kernel even has
    interfaces to support us in this. Since it's not going to be a very
    common case, we may not care about the performance. If we do find that
    we care about the performance, we should first attempt to get what we
    need into the linux kernel so we don't have to duplicate code, and only
    if that fails do the duplication.

    I'm pretty sure the AGP chipsets versus >32-bits pages danger has been
    overstated, though. Besides the fact that you needed to load one of
    these older supposed machines with a full 4GB of memory (well,
    theoretically 3.5GB but how often can you even boot a system with a 2,
    1, .5gb combo?), you also need a chipset that does >32-bit addressing.

    At least all AMD and Intel chipsets don't appear to have this problem in
    the survey I did last night, as they've either got >32-bit chipset and
    >32-bit gart, or 32-bit chipset and 32-bit gart. Basically all I'm
    worried about is ATI PCI[E]GART at this point.

    http://dri.freedesktop.org/wiki/GARTAddressingLimits

    <snip bits that have been covered in other mails>

    > 5) What's protecting i915 GEM object privates and lists in a
    > multi-threaded environment?

    Nothing at the moment. That's my current project. dev->struct_mutex is
    the plan -- I don't want to see finer-grained locking until we show that
    contention on that locking is an issue. Fine-grained locking takes
    significant care, and there's a lot more important performance
    improvements to work on before then.

    > 6) Isn't do_mmap() strictly forbidden in new drivers? I remember seeing
    > some severe ranting about it on the lkml?

    We've talked it over with Arjan, and until we can use real fds as our
    handles to objects, he thought it sounded OK. But apparently Al Viro's
    working on making it so that allocating a thousand fds would be feasible
    for us. At that point mmap/pread/pwrite/close ioctls could be replaced
    with the syscalls they were named for, and the kernel guys love us.

    > TTM is designed to cope with most hardware quirks I've come across with
    > different chipsets so far, including Intel UMA, Unichrome, Poulsbo, and
    > some other ones. GEM basically leaves it up to the driver writer to
    > reinvent the wheel..

    The problem with TTM is that it's designed to expose one general API for
    all hardware, when that's not what our drivers want. The GPU-GPU cache
    handling for intel, for example, mapped the hardware so poorly that
    every batch just flushed everything. Bolting on the clflush-based
    cpu-gpu caching management for our platform recovered a lot of
    performance, but we're still having to reuse buffers in userland at a
    memory cost because allocating buffers is overly expensive for the
    general supporting-everybody (but oops, it's not swappable!) object
    allocator.

    We're trying to come at it from the other direction: Implement one
    driver well. When someone else implements another driver and finds that
    there's code that should be common, make it into a support library and
    share it.

    I actually would have liked the whole interface to userland to be
    driver-specific with a support library for the parts we think other
    people would want, but DRI2 wants to use buffer objects for its shared
    memory transport and I didn't want to rock its boat too hard, so the
    ioctls that should be supportable for everyone got moved to generic.

    If the implementation of those ioctls in generic code doesn't work for
    some drivers (say, early shmfs object creation turns out to be a bad
    idea for VRAM drivers), I'll happily push it out to the driver.

    --
    Eric Anholt anholt@Fr...
    eric@an... eric.anholt@in...

    From: Thomas Hellström <thomas@tu...> - 2008-05-14 19:05
    Eric Anholt wrote:
    > On Wed, 2008-05-14 at 02:33 +0200, Thomas Hellström wrote:
    >
    >>> The real question is whether TTM suits the driver writers for use in Linux
    >>> desktop and embedded environments, and I think so far I'm not seeing
    >>> enough positive feedback from the desktop side.
    >>>
    >>>
    >> I actually haven't seen much feedback at all. At least not on the
    >> mailing lists.
    >> Anyway we need to look at the alternatives which currently is GEM.
    >>
    >> GEM, while still in development basically brings us back to the
    >> functionality of TTM 0.1, with added paging support but without
    >> fine-grained locking and caching policy support.
    >>
    >> I might have misunderstood things but quickly browsing the code raises
    >> some obvious questions:
    >>
    >> 1) Some AGP chipsets don't support page addresses > 32bits. GEM objects
    >> use GFP_HIGHUSER, and it's hardcoded into the linux swap code.
    >>
    >
    > The obvious solution here is what many DMA APIs do for IOMMUs that can't
    > address all of memory -- keep a pool of pages within the addressable
    > range and bounce data through them. I think the Linux kernel even has
    > interfaces to support us in this. Since it's not going to be a very
    > common case, we may not care about the performance. If we do find that
    > we care about the performance, we should first attempt to get what we
    > need into the linux kernel so we don't have to duplicate code, and only
    > if that fails do the duplication.
    >
    > I'm pretty sure the AGP chipsets versus >32-bits pages danger has been
    > overstated, though. Besides the fact that you needed to load one of
    > these older supposed machines with a full 4GB of memory (well,
    > theoretically 3.5GB but how often can you even boot a system with a 2,
    > 1, .5gb combo?), you also need a chipset that does >32-bit addressing.
    >
    > At least all AMD and Intel chipsets don't appear to have this problem in
    > the survey I did last night, as they've either got >32-bit chipset and
    >
    >> 32-bit gart, or 32-bit chipset and 32-bit gart. Basically all I'm
    >>
    > worried about is ATI PCI[E]GART at this point.
    >
    > http://dri.freedesktop.org/wiki/GARTAddressingLimits
    >
    > <snip bits that have been covered in other mails>
    >
    >
    There will probably turn up a couple of more devices or incomplete
    drivers, but in the long run this is a fixable problem.
    >> 5) What's protecting i915 GEM object privates and lists in a
    >> multi-threaded environment?
    >>
    >
    > Nothing at the moment. That's my current project. dev->struct_mutex is
    > the plan -- I don't want to see finer-grained locking until we show that
    > contention on that locking is an issue. Fine-grained locking takes
    > significant care, and there's a lot more important performance
    > improvements to work on before then.
    >
    >
    >> 6) Isn't do_mmap() strictly forbidden in new drivers? I remember seeing
    >> some severe ranting about it on the lkml?
    >>
    >
    > We've talked it over with Arjan, and until we can use real fds as our
    > handles to objects, he thought it sounded OK. But apparently Al Viro's
    > working on making it so that allocating a thousand fds would be feasible
    > for us. At that point mmap/pread/pwrite/close ioctls could be replaced
    > with the syscalls they were named for, and the kernel guys love us.
    >
    >
    >> TTM is designed to cope with most hardware quirks I've come across with
    >> different chipsets so far, including Intel UMA, Unichrome, Poulsbo, and
    >> some other ones. GEM basically leaves it up to the driver writer to
    >> reinvent the wheel..
    >>
    >
    > The problem with TTM is that it's designed to expose one general API for
    > all hardware, when that's not what our drivers want. The GPU-GPU cache
    > handling for intel, for example, mapped the hardware so poorly that
    > every batch just flushed everything. Bolting on the clflush-based
    > cpu-gpu caching management for our platform recovered a lot of
    > performance, but we're still having to reuse buffers in userland at a
    > memory cost because allocating buffers is overly expensive for the
    > general supporting-everybody (but oops, it's not swappable!) object
    > allocator.
    >
    >
    Swapping drmBOs is a couple of days implementation and some core kernel
    exports. It's just that someone needs find the time and the right person
    to talk to in the right way to get certain swapping functions exported.
    > We're trying to come at it from the other direction: Implement one
    > driver well. When someone else implements another driver and finds that
    > there's code that should be common, make it into a support library and
    > share it.
    >
    > I actually would have liked the whole interface to userland to be
    > driver-specific with a support library for the parts we think other
    > people would want, but DRI2 wants to use buffer objects for its shared
    > memory transport and I didn't want to rock its boat too hard, so the
    > ioctls that should be supportable for everyone got moved to generic.
    >
    > If the implementation of those ioctls in generic code doesn't work for
    > some drivers (say, early shmfs object creation turns out to be a bad
    > idea for VRAM drivers), I'll happily push it out to the driver.
    >
    >
    That would basically make the rest of my issues non-issues, as it would
    allow us to without too much effort reuse much of the TTM functionality
    over the GEM interface, and everybody's hopefully happy.

    /Thomas

    From: Keith Packard <keithp@ke...> - 2008-05-14 19:10

    Attachments: Message as HTML     
    On Wed, 2008-05-14 at 16:36 +0200, Thomas Hellström wrote:

    > My personal feeling is that pwrites are a workaround for a workaround
    > for a very bad decision

    Feel free to map VRAM then if you can; I didn't need to on Intel as
    there isn't any difference.

    --
    keith.packard@in...

    From: Keith Packard <keithp@ke...> - 2008-05-14 19:13

    Attachments: Message as HTML     
    On Wed, 2008-05-14 at 19:08 +0200, Jerome Glisse wrote:

    > I don't have number or benchmark to check how fast pread/pwrite path might
    > be in this use so i am just expressing my feeling which happen to just be
    > to avoid vma tlb flush as most as we can.

    For batch buffers, pwrite is 3X faster than map/write/unmap, at least as
    measured by that most estimable benchmark 'glxgears'. Take that with as
    much skepticism as it deserves.

    --
    keith.packard@in...

    From: Alex Deucher <alexdeucher@gm...> - 2008-05-14 19:18
    On Wed, May 14, 2008 at 2:30 PM, Eric Anholt <eric@an...> wrote:
    > On Wed, 2008-05-14 at 02:33 +0200, Thomas Hellström wrote:
    > > > The real question is whether TTM suits the driver writers for use in Linux
    > > > desktop and embedded environments, and I think so far I'm not seeing
    > > > enough positive feedback from the desktop side.
    > > >
    > > I actually haven't seen much feedback at all. At least not on the
    > > mailing lists.
    > > Anyway we need to look at the alternatives which currently is GEM.
    > >
    > > GEM, while still in development basically brings us back to the
    > > functionality of TTM 0.1, with added paging support but without
    > > fine-grained locking and caching policy support.
    > >
    > > I might have misunderstood things but quickly browsing the code raises
    > > some obvious questions:
    > >
    > > 1) Some AGP chipsets don't support page addresses > 32bits. GEM objects
    > > use GFP_HIGHUSER, and it's hardcoded into the linux swap code.
    >
    > The obvious solution here is what many DMA APIs do for IOMMUs that can't
    > address all of memory -- keep a pool of pages within the addressable
    > range and bounce data through them. I think the Linux kernel even has
    > interfaces to support us in this. Since it's not going to be a very
    > common case, we may not care about the performance. If we do find that
    > we care about the performance, we should first attempt to get what we
    > need into the linux kernel so we don't have to duplicate code, and only
    > if that fails do the duplication.
    >
    > I'm pretty sure the AGP chipsets versus >32-bits pages danger has been
    > overstated, though. Besides the fact that you needed to load one of
    > these older supposed machines with a full 4GB of memory (well,
    > theoretically 3.5GB but how often can you even boot a system with a 2,
    > 1, .5gb combo?), you also need a chipset that does >32-bit addressing.
    >
    > At least all AMD and Intel chipsets don't appear to have this problem in
    > the survey I did last night, as they've either got >32-bit chipset and
    > >32-bit gart, or 32-bit chipset and 32-bit gart. Basically all I'm
    > worried about is ATI PCI[E]GART at this point.

    AMD PCIE and IGP GART support 40 bits (Dave just committed support
    this morning) so we should be fine on r3xx and newer PCIE cards.

    Alex

    From: Thomas Hellström <thomas@tu...> - 2008-05-14 19:42
    Keith Packard wrote:
    > On Wed, 2008-05-14 at 16:36 +0200, Thomas Hellström wrote:
    >
    >
    >> My personal feeling is that pwrites are a workaround for a workaround
    >> for a very bad decision
    >>
    >
    > Feel free to map VRAM then if you can; I didn't need to on Intel as
    > there isn't any difference.
    >
    >

    With mapping device memory on UMA devices I'm referring to mapping
    through the GTT aperture. Either as stolen memory, Pre-bound GTT pools
    or simply buffer object memory temporarily bound to the GTT.

    As you've previously mentioned, this requires caching policy changes and
    it needs to be used with some care.

    /Thomas

    From: Thomas Hellström <thomas@tu...> - 2008-05-14 19:51
    Eric Anholt wrote:
    >
    > If the implementation of those ioctls in generic code doesn't work for
    > some drivers (say, early shmfs object creation turns out to be a bad
    > idea for VRAM drivers), I'll happily push it out to the driver.
    >
    >
    Or perhaps use generic ioctls, but provide hooks in the driver to
    overload the core GEM functions with other implementations.

    /Thomas

    From: Keith Packard <keithp@ke...> - 2008-05-14 22:51

    Attachments: Message as HTML     
    On Wed, 2008-05-14 at 21:41 +0200, Thomas Hellström wrote:

    > As you've previously mentioned, this requires caching policy changes and
    > it needs to be used with some care.

    I did't need that in my drivers as GEM handles the WB -> GPU object
    transfer already.

    Object mapping is really the least important part of the system; it
    should only be necessary when your GPU is deficient, or your API so
    broken as to require this inefficient mechanism. I suspect we'll be
    tracking 965 performance as we work to eliminate mapping, we should see
    a steady increase until we're no longer mapping anything that the GPU
    uses into the application's address space.

    --
    keith.packard@in...

    From: Eric Anholt <eric@an...> - 2008-05-14 23:01

    Attachments: Message as HTML     
    On Wed, 2008-05-14 at 21:51 +0200, Thomas Hellström wrote:
    > Eric Anholt wrote:
    > >
    > > If the implementation of those ioctls in generic code doesn't work for
    > > some drivers (say, early shmfs object creation turns out to be a bad
    > > idea for VRAM drivers), I'll happily push it out to the driver.
    > >
    > >
    > Or perhaps use generic ioctls, but provide hooks in the driver to
    > overload the core GEM functions with other implementations.

    Yeah, that's what I was thinking.

    --
    Eric Anholt anholt@Fr...
    eric@an... eric.anholt@in...

    From: Allen Akin <akin@po...> - 2008-05-14 23:34
    On Wed, May 14, 2008 at 03:48:47PM -0700, Keith Packard wrote:
    | Object mapping is really the least important part of the system; it
    | should only be necessary when your GPU is deficient, or your API so
    | broken as to require this inefficient mechanism.

    In the OpenGL case, object mapping wasn't originally a part of the API.
    It was added because people building hardware and apps for Intel-based
    PCs determined that it was worthwhile, and demanded it.

    This wasn't on my watch, so I can't give you the history in detail, but
    my recollection is that the primary uses were texture loading for games
    and video apps, and incremental changes to vertex arrays for games and
    rendering apps.

    So maybe the hardware has changed sufficiently that the old reasoning
    and performance measurements are no longer valid. It would still be
    good to know for sure that eliminating low-level support for the
    mechanism won't be drastically bad for the classes of apps that use it.

    Allen

    From: Keith Packard <keithp@ke...> - 2008-05-15 02:24

    Attachments: Message as HTML     
    On Wed, 2008-05-14 at 16:34 -0700, Allen Akin wrote:
    > On Wed, May 14, 2008 at 03:48:47PM -0700, Keith Packard wrote:
    > | Object mapping is really the least important part of the system; it
    > | should only be necessary when your GPU is deficient, or your API so
    > | broken as to require this inefficient mechanism.
    >
    > In the OpenGL case, object mapping wasn't originally a part of the API.
    > It was added because people building hardware and apps for Intel-based
    > PCs determined that it was worthwhile, and demanded it.

    In a UMA environment, it seems so obvious to map objects into the
    application and just bypass the whole kernel API issue. That, however,
    ignores caching effects, which appear to dominate performance effects
    these days.

    > This wasn't on my watch, so I can't give you the history in detail, but
    > my recollection is that the primary uses were texture loading for games
    > and video apps, and incremental changes to vertex arrays for games and
    > rendering apps.

    Most of which can be efficiently performed with a pwrite-like system
    where the application explicitly tells the system which portions of the
    object to modify. Again, it seems insane when everything is a uniform
    mass of pages, except for the subtle differences in cache behaviour.

    > So maybe the hardware has changed sufficiently that the old reasoning
    > and performance measurements are no longer valid. It would still be
    > good to know for sure that eliminating low-level support for the
    > mechanism won't be drastically bad for the classes of apps that use it.

    I'm not sure we can (or want to) eliminate it entirely, all that I
    discovered was that it should be avoided as it has negative performance
    consequences. Not dire, but certainly not positive either.

    I don't know how old these measurements were, but certainly the gap
    between CPU and memory speed has been rapidly increasing for years,
    along with cache sizes, both of which have a fairly dramatic effect on
    how best to access actual memory.

    --
    keith.packard@in...

    From: Allen Akin <akin@po...> - 2008-05-15 03:37
    On Wed, May 14, 2008 at 05:22:06PM -0700, Keith Packard wrote:
    | On Wed, 2008-05-14 at 16:34 -0700, Allen Akin wrote:
    | > In the OpenGL case, object mapping wasn't originally a part of the API.
    | > It was added because people building hardware and apps for Intel-based
    | > PCs determined that it was worthwhile, and demanded it.
    |
    | In a UMA environment, it seems so obvious to map objects into the
    | application and just bypass the whole kernel API issue. That, however,
    | ignores caching effects, which appear to dominate performance effects
    | these days.

    I think the confusion arises because the mechanism is used for several
    purposes, some of which are likely to be dominated by cache effects on
    some implementations, and others that aren't. I'm thinking about the
    differences between piecemeal updating of the elements of a vertex
    array, versus grabbing an image from a video capture card or a direct
    read() from a file into a texture buffer. The API is intended to allow
    apps and drivers to make intelligent choices between cases like those.
    Check out BufferData() and MapBuffer() in section 2.9 of the OpenGL 2.1
    spec for a discussion which specifically mentions cache effects.

    | > This wasn't on my watch, so I can't give you the history in detail, but
    | > my recollection is that the primary uses were texture loading for games
    | > and video apps, and incremental changes to vertex arrays for games and
    | > rendering apps.
    |
    | Most of which can be efficiently performed with a pwrite-like system
    | where the application explicitly tells the system which portions of the
    | object to modify. ...

    Interfaces of that style are present in OpenGL, and predate the mapping
    interfaces. I know they were regarded as too slow for some apps, so the
    mapping interfaces were added. The early extensions were driven by
    vendors who didn't support UMA, so that couldn't have been the only
    model they were concerned about. Beyond that I'm not sure.

    | > So maybe the hardware has changed sufficiently that the old reasoning
    | > and performance measurements are no longer valid. It would still be
    | > good to know for sure that eliminating low-level support for the
    | > mechanism won't be drastically bad for the classes of apps that use it.
    |
    | I'm not sure we can (or want to) eliminate it entirely, all that I
    | discovered was that it should be avoided as it has negative performance
    | consequences. Not dire, but certainly not positive either.
    |
    | I don't know how old these measurements were, but certainly the gap
    | between CPU and memory speed has been rapidly increasing for years,
    | along with cache sizes, both of which have a fairly dramatic effect on
    | how best to access actual memory.

    The first reference I can find to an object-mapping API in OpenGL is
    from 2001. I'm sure the vendors had implementations internally before
    then, but that's when things were mature enough to start standardizing.
    Since the functionality is present in OpenGL 2.0 (vintage 2006?),
    apparently someone thought it was still useful enough to carry over from
    OpenGL 1.X.

    Again, sorry I don't know the entire history on this one.

    Allen

    From: Thomas Hellström <thomas@tu...> - 2008-05-15 05:31
    Keith Packard wrote:
    > On Wed, 2008-05-14 at 21:41 +0200, Thomas Hellström wrote:
    >
    >
    >> As you've previously mentioned, this requires caching policy changes and
    >> it needs to be used with some care.
    >>
    >
    > I did't need that in my drivers as GEM handles the WB -> GPU object
    > transfer already.
    >
    > Object mapping is really the least important part of the system; it
    > should only be necessary when your GPU is deficient, or your API so
    > broken as to require this inefficient mechanism. I suspect we'll be
    > tracking 965 performance as we work to eliminate mapping, we should see
    > a steady increase until we're no longer mapping anything that the GPU
    > uses into the application's address space.
    >
    >
    Static wc-d maps into fairly static objects like scanout buffers or
    buffer pools are not inefficient. They provide the by far highest
    throughput for writing (even beats cache-coherent). But they may take
    some time to set up or tear down, which means you should avoid that as
    much as possible.

    For things like scanout buffers or video buffers you should really use
    such mappings, otherwise you lose big.

    The current implementation of GEM that doesn't allow overloading of the
    core GEM functions blocked the possibility to set up such mappings. This
    is about to change, and I'm happy with that.

    /Thomas

    From: Keith Packard <keithp@ke...> - 2008-05-15 07:13

    Attachments: Message as HTML     
    On Thu, 2008-05-15 at 07:30 +0200, Thomas Hellström wrote:

    > Static wc-d maps into fairly static objects like scanout buffers or
    > buffer pools are not inefficient. They provide the by far highest
    > throughput for writing (even beats cache-coherent). But they may take
    > some time to set up or tear down, which means you should avoid that as
    > much as possible.

    Yeah, the 2D driver uses the GTT mapping for drawing to the front
    buffer. Otherwise, tiling is way too much work as the code isn't neatly
    wrapped up like the Mesa swrast. For that, I just added detiling code.

    I'm not sure how I'd manage objects that can move or be evicted though;
    perhaps some page table tricks could be used to avoid locking objects to
    the GTT in a way visible to the user application.

    > The current implementation of GEM that doesn't allow overloading of the
    > core GEM functions blocked the possibility to set up such mappings. This
    > is about to change, and I'm happy with that.

    The requirement is that GEM provide the interfaces used by drivers; if a
    driver needs some new functionality, we'd naturally work out how to
    incorporate that. Right now, our 3D drivers don't care about GTT maps,
    but I suspect our 2D ones will as I don't really want to try to deal
    with tiling from all of the X server drawing code.

    --
    keith.packard@in...

    From: Thomas Hellström <thomas@tu...> - 2008-05-15 12:04
    Keith Packard wrote:
    > On Thu, 2008-05-15 at 07:30 +0200, Thomas Hellström wrote:
    >
    >
    >> Static wc-d maps into fairly static objects like scanout buffers or
    >> buffer pools are not inefficient. They provide the by far highest
    >> throughput for writing (even beats cache-coherent). But they may take
    >> some time to set up or tear down, which means you should avoid that as
    >> much as possible.
    >>
    >
    > Yeah, the 2D driver uses the GTT mapping for drawing to the front
    > buffer. Otherwise, tiling is way too much work as the code isn't neatly
    > wrapped up like the Mesa swrast. For that, I just added detiling code.
    >
    > I'm not sure how I'd manage objects that can move or be evicted though;
    > perhaps some page table tricks could be used to avoid locking objects to
    > the GTT in a way visible to the user application.
    >

    Yes, TTM does this by killing user-space mappings when an object is
    evicted. When they are again accessed by the app, they are simply
    faulted back mapping the new location wherever that is, changing caching
    policy if necessary.

    That approach is in principle reusable for a GEM implementation but I
    think not with the shmem backing objects, since we cannot overload the
    SHMEM fault() method, but that's only a limitiation of the current GEM
    implementation, not of the API design.

    One way to implement this in the GEM context I'd use the (small) part of
    drmBOs that deals with this, and use that as base GEM objects instead of
    SHMEMFS objects. Each GEM object would also always hold a page list.
    When the object has real pages attached to it (Like UMA objects) it
    would be populated with pages. When the object points to a place in
    VRAM, it holds swap_entry references (To reserve swap space used at
    suspend).

    If a GEM object, at creation, fails to allocate either pages or swap
    entries, object creation should fail to avoid failures later at
    unpredictable points.

    With this approach the GEM driver needs to decide when to push a GEM
    object's pages to the swap cache and when to reclaim them. It doesn't
    happen automatically as with the SMEMFS objects, but OTOH that puts the
    GEM driver in control on how much memory it will pin. That will avoid
    bad SHMEMFS decisions.
    >
    >
    >> The current implementation of GEM that doesn't allow overloading of the
    >> core GEM functions blocked the possibility to set up such mappings. This
    >> is about to change, and I'm happy with that.
    >>
    >
    > The requirement is that GEM provide the interfaces used by drivers; if a
    > driver needs some new functionality, we'd naturally work out how to
    > incorporate that. Right now, our 3D drivers don't care about GTT maps,
    > but I suspect our 2D ones will as I don't really want to try to deal
    > with tiling from all of the X server drawing code.
    >
    >
    Sounds good to me. Modesetting people will also be happy.

    /Thomas.

    From: Ian Romanick <idr@us...> - 2008-05-15 17:06
    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    Keith Packard wrote:
    | On Wed, 2008-05-14 at 21:41 +0200, Thomas Hellström wrote:
    |
    |> As you've previously mentioned, this requires caching policy changes and
    |> it needs to be used with some care.
    |
    | I did't need that in my drivers as GEM handles the WB -> GPU object
    | transfer already.
    |
    | Object mapping is really the least important part of the system; it
    | should only be necessary when your GPU is deficient, or your API so
    | broken as to require this inefficient mechanism. I suspect we'll be
    | tracking 965 performance as we work to eliminate mapping, we should see
    | a steady increase until we're no longer mapping anything that the GPU
    | uses into the application's address space.

    Er...what about glMapBuffer? Are we now going to force drivers to
    implement that via copies?

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.7 (GNU/Linux)

    iD8DBQFILGzGX1gOwKyEAw8RAhYRAJsF4k9TjewdZseLAvXPlibJdKChrwCgih0p
    L/D5WQlfEpN+DyDgYvOUA20=
    =wYa/
    -----END PGP SIGNATURE-----

    From: Keith Packard <keithp@ke...> - 2008-05-15 18:10

    Attachments: Message as HTML     
    On Thu, 2008-05-15 at 10:03 -0700, Ian Romanick wrote:

    > Er...what about glMapBuffer? Are we now going to force drivers to
    > implement that via copies?

    No, we'll support it, and make it as fast as possible. The goal is to
    not use it inside the driver, not to break GL apps.

    --
    keith.packard@in...

    From: Philipp Klaus Krause <pkk@sp...> - 2008-05-15 18:50
    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    Eric Anholt schrieb:

    >> No. Gem can't coop with it. Let's say you have a 512M system with two 1G
    >> video cards, 4G swap space, and you want to fill both card's videoram
    >> with render-and-forget textures for whatever purpose.
    >
    > Who's selling that system? Who's building that system at home?

    Video game consoles?

    According to Wikipedia PS3 has 256 MB of RAM vs 256 MB of VRAM.

    Philipp

    P.S.: Even my ColecoVision, has 1 KB of RAM vs 16 KB of VRAM.
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.6 (GNU/Linux)
    Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

    iD8DBQFILHQVbtUV+xsoLpoRAsaLAJ0fXyrk1n4TE0m/egvm10uACnIxLwCgqjl3
    BE0DdbGE1R61oBsbf/zi8cU=
    =nq1l
    -----END PGP SIGNATURE-----

    From: Stephane Marchesin <marchesin@ic...> - 2008-05-18 14:37
    >
    > Jerome, Dave, Keith
    >
    > It's hard to argue against people trying things out and finding it's not
    > really what they want, so I'm not going to do that.
    >
    > The biggest argument (apart from the fencing) seems to be that people
    > thinks TTM stops them from doing what they want with the hardware,
    > although it seems like the Nouveau needs and Intel UMA needs are quite
    > opposite. In an open-source community where people work on things
    > because they want to, not being able to do what you want to is a bad thing,

    What you fail to notice here is that I think most people intend to
    have only one memory manager in the kernel. So making the wrong
    decisions here will pretty much enforce those decisions on all
    drivers. And therefore, we will not be "able to do what you want to"

    >
    > OTOH a stall and disagreement about what's the best thing to use is
    > even worse. It confuses the users and it's particularly bad for people
    > trying to write drivers on a commercial basis.

    I don't see how the needs are opposed. A memory manager is just
    handling pieces of memory, and you should get some kind of flexibility
    from it, especially if it's going to be the de-facto memory manager
    for all DRI/X.Org.

    Stephane

    From: Thomas Hellström <thomas@tu...> - 2008-05-18 16:09
    Stephane Marchesin wrote:
    >> Jerome, Dave, Keith
    >>
    >> It's hard to argue against people trying things out and finding it's not
    >> really what they want, so I'm not going to do that.
    >>
    >> The biggest argument (apart from the fencing) seems to be that people
    >> thinks TTM stops them from doing what they want with the hardware,
    >> although it seems like the Nouveau needs and Intel UMA needs are quite
    >> opposite. In an open-source community where people work on things
    >> because they want to, not being able to do what you want to is a bad thing,
    >>
    >
    > What you fail to notice here is that I think most people intend to
    > have only one memory manager in the kernel.

    How on earth can you draw that conclusion from the above statement?

    > So making the wrong
    > decisions here will pretty much enforce those decisions on all
    > drivers. And therefore, we will not be "able to do what you want to"
    >

    What GEM protagonists have been arguing and propagating for is not a
    single memory manager, but a single small common simple memory
    management interface to that would allow any driver writer to do pretty
    much what they want with their driver. As you might have noticed we're
    not really arguing against that.

    /Thomas

    From: Stephane Marchesin <marchesin@ic...> - 2008-05-18 16:52
    On 5/18/08, Thomas Hellström <thomas@tu...> wrote:
    > >
    > > What you fail to notice here is that I think most people intend to
    > > have only one memory manager in the kernel.
    >
    >
    > How on earth can you draw that conclusion from the above statement?
    >

    Well, Dave has been saying this to me all along... otherwise I'd
    probably have my own memory manager too. I also think most people
    agree that a single memory manager would make things simpler for
    everyone (especially since there is a need for some glue with things
    like EXA).

    >
    > > So making the wrong
    > > decisions here will pretty much enforce those decisions on all
    > > drivers. And therefore, we will not be "able to do what you want to"
    > >
    >
    >
    > What GEM protagonists have been arguing and propagating for is not a
    > single memory manager, but a single small common simple memory
    > management interface to that would allow any driver writer to do pretty
    > much what they want with their driver. As you might have noticed we're
    > not really arguing against that.

    Yeah, again, I'm not taking sides, I'm just concerned that we'll have
    to revisit the memory manager issue in 2 years when all cards
    implement full memory protection and paging (and that'll be the case,
    since the windows driver model pretty much requires that). But maybe
    it's the right thing to do to move forward now.

    Stephane

    Next Messages