Re: [E1000-devel] How the ixgbe and Linux network stack manage memory

[Alexander D. <ale...@gm...>]

On Sun, Oct 16, 2016 at 4:52 AM, Michał Purzyński
<mic...@gm...> wrote:
> Could you fill me in on how the bounce buffer approach to memory management
> in IXGBE and I40E works?
>
> Why do you allocate the same amount of memory in ixgbe_setup_rx_resources()
> twice?
> First time with a call to rx_ring->rx_buffer_info = vzalloc_node() - where
> size represents what I set with ethtool -G (N buffers, each represented
> with a struct ixgbe_rx_buffer).
> Second in the rx_ring->desc = dma_alloc_coherent() of the same size as
> above.

There are two different buffers being allocated here.  The first is
the rx_buffer_info which store data used by the driver to track memory
allocations, DMA addresses, and page offsets.  The allocation into
desc is for descriptor rings used to signal between the device and the
driver where the buffers are at, and information about the packets
written back.

> Second question. So we have a single ring (per queue) with a chains of
> buffers "attached" to it - a chunk of memory composed from multiple pages.
> Each page will have a single packet.
> Does the driver replace "used" pages every time it uses one? I'm confused,
> because that's kind of logical it does, there are functions doing that, yet
> there are comments in the driver that "we are reusing buffers".
>
> My understanding is that once a page (2048 bytes here) contains a packet
> data, it is only "re-attached" to SKB with minimal copying (header,
> basically). That would require allocating a new page every time we used
> one  - attached it to SKB.

A page is 4096 bytes in size.  One trick we are using is to only use
2048 bytes of it at a time, then we move over to the other half by
XORing the page offset value by 2048.  The pages themselves have a
reference count and as long as the reference count is 1 we can
increase the reference count by one and always use the other half of
the page.  The stack itself should release the page reference reducing
the reference count back to 1 hopefully before we will need to use
that portion of the buffer again.  If the reference count is greater
than 1 we will free hand the page off to the stack and instead
allocate a new page to replace it via the <driver>_alloc_mapped_page
call.

Hope that helps to clarify things a bit.

- Alex