I have a question about how mode 5 functions...
Is the traffic dispatched from a bonded interface configured for mode 5
balanced by conversation? Or, is the balancing done on a
per-packet/segment/frame basis? I am curious if mode 5 could possibly
generate packet out of sequence errors due to packets from the same
conversation being sent out of more than one interface.
Aaron J. Greusel
Walt Disney Feature Animation
"There are only 10 types of people in the world: Those who understand
binary, and those who don't."
From: Jay Vosburgh <fubar@us...> - 2006-08-31 18:48:28
Greusel, Aaron <Aaron.Greusel@...> wrote:
>Is the traffic dispatched from a bonded interface configured for mode 5
>balanced by conversation? Or, is the balancing done on a
>per-packet/segment/frame basis? I am curious if mode 5 could possibly
>generate packet out of sequence errors due to packets from the same
>conversation being sent out of more than one interface.
For IP traffic (other than broadcasts, IGMP and ARP), the
transmit traffic for balance-tlb (and alb) is balanced according to a
XOR hash of the destination IP address, wherein each hashed value is
assigned a slave at first use by selecting the least loaded slave at
It should not normally stripe traffic from a single conversation
across multiple interfaces, although the balance is recomputed from
scratch every 10 seconds (the hash table is cleared, and repopulated as
During the rebalance, it is theoretically possible that a packet
for a given conversation sent on interface A, then followed immediately
by a packet for the same conversation on interface B (because a
rebalance has occured between them), could be delivered out of order,
but only if some interim or destination system doesn't process them in
true real time.
I haven't tested to try and induce this, but I think it would be
unlikely to occur if all ports of the bond are connected to a common
(non-crappy) switch. In that case, all packets traverse the same path
(apart from the chosen bonding interface) to their destination. If the
different interfaces of the bond have differing paths to the
destination, then it would be more likely to get a packet out of order,
depending upon whether or not the paths are the same length, and what
the relative traffic loads of the two paths is.
The same type of potential out of order delivery issues exist in
the other balance modes, as well as 802.3ad, when an interface either
enters or leaves service. That's obviously a much less frequent event
than the periodic rebalance of balance-tlb, though.
The classic manifestation of packets being delivered out of
order manifests at receive time, when the receiving system has multiple
interfaces that are being passed traffic simultaneously (because modern
network interfaces generally receive packets in bunches, so striping of
traffic and the vagaries of interrupt delivery or the like can induce
out of order reception). I've never seen the switches I have (Cisco
Catalyst midrange) exhibit this problem; it's always been a problem on
end hosts with multiple links into the bond, typically when running a
round robin type of mode.
The balance-xor mode when used in conjuction with the layer3+4
xmit_hash_policy does balance traffic by conversation, but it's
according to a hash formula; the relative loading of the slaves is not
taken into account as it is for balance-tlb/alb. The traffic won't jump
from slave to slave (unless slaves enter or leave service), but if
you're mathematically unlucky, traffic can end up unbalanced.
-Jay Vosburgh, IBM Linux Technology Center, fubar@...