Re: [tipc-discussion] [PATCH 5/5] tipc: add ip/udp media type

[Ying X. <yin...@wi...>]

On 01/19/2015 04:47 PM, Erik Hugne wrote:
> On Mon, Jan 19, 2015 at 10:43:56AM +0800, Ying Xue wrote:
>> On 01/16/2015 07:49 PM, Erik Hugne wrote:
>>>>>>> +/**
>>>>>>> + * tipc_send_msg - enqueue a send request
>>>>>>> + *
>>>>>>> + * The send request need to be deferred since we cannot call kernel
>>>>>>> + * socket API functions while holding the node spinlock.
>>>>>>> + */
>>>>>
>>>>> Yes. Unfortunately this is necessary for now, but we should aim at
>>>>> releasing the node spinlock before sending the buffers, on all
>>>>> bearers. This will probably have some positive impact on performance.
>>>>>
>>> What's holding me from sending another version now is the skb reallocation
>>> from kernel_sendmsg().
>>> On lowmem systems this is causing the kworker thread to spin heavily on memory
>>> allocation because it has run out of memory in the kmalloc-2048 pool.
>>> It should be possible to reuse the TIPC skb and pass that to the IP stack, but
>>> i don't know exactly how to do this.
>>
>>
>> If this is another reason why we have to send TIPC buffers in a separate
>> thread, maybe we should seriously consider the idea I proposed before.
>> If we are really able to make the TIPC sending path non-atomic, this is
>> not only of benefit to UDP bearer, but also it gives TIPC an extreme
>> flexibility, for example, we can use all kinds of tunnels implemented in
>> networking stack as TIPC bearers, such as, ipip, vti, vxlan,
>> ipv6_tunnels etc.
>>
> 
> Making the send path non-atomic would help in the sense that we'd get rid of the
> bearer-level send workqueue and related stuff, but there's still the issue with
> each packet going through kernel_sendmsg is reallocated/copied to a new skb.
> 

Until now, I eventually understood your met issue.

> I'm wondering if this would be feasible:
> 1. export the udp_send_skb() function
> 2. skb_push(sizeof udphdr) on the TIPC SKB.
> 3. Do a routing lookup for the destination ip and pass that, together with the
>    TIPC SKB to udp_send_skb()
>

It sounds like this is a hack method. Instead, we can understand how
these tunnels deal with our encountered issue as long as we look through
the tunnel interfaces implemented in networking with UDP socket. For
example, if we take a closer look at VXLAN code, we will find a better
approach to solve our issue. Actually VXLAN uses udp_tunnel to overcome
the difficulty:

1. Create UDP socket:

vxlan_create_sock()
  udp_sock_create()

2. Deliver SKB:

vxlan_xmit_one()
  vxlan_xmit_skb()
    udp_tunnel_xmit_skb()

In the entire sending path, no SKB copy happens. Moreover, when we use
the interfaces provided by udp_tunnel, both IPV4 and IPv6 are supported :)

It seems that this is a perfect solution for us :)

Maybe my analysis is not completely right as I just look through the
code of VXLAN. If I am wrong, please correct me :)

Regards,
Ying

> 
> //E
> 
>