Improve performance of single TCP connection?

Help
jbreher
2006-10-05
2013-06-06
  • jbreher
    jbreher
    2006-10-05

    I was just hoping for a sanity check. I am hoping to increase the throughput possible on a single TCP connection past the wire speed of a single Ethernet link. I boldly marched forward with the assumption that this is possible with the bonding mechanism. However, after combing the documentation, I believe I may have been mistaken. It appears that no matter how many NICs I throw at the problem, I will still be limited in performance due to the specific hashing mechanisms used to identify which NICs a given packet will be sent to/from.

    So for a point blank question, is there any bonding mode which can improve the performance of a single TCP connection across multiple NICs?

     
    • Jay Vosburgh
      Jay Vosburgh
      2006-10-05

      Yes, it's possible in balance-rr mode, but it usually won't show linear scaling as you add interfaces.  This is caused by out of order delivery if the receiver is a bonding / etherchannel, which will in turn trigger TCP/IP's congestion control algorithms (those can be mitigated to some degree, see below).

      I did some tests on this some time ago; if memory serves, a four interface bond could obtain roughly 2.25 interface's worth of throughput to one TCP/IP connection.  I don't recall what the scaling was for a two interface bond, I'd expect something around 1.5.  There is some discussion of this in the bonding-devel archives.

      On the other hand, if you run, e.g., several lower speed links connected in an balance-rr bond, connected to an etherchannel switch, where the outbound interface from the switch is a single higher speed link, you can get good scaling transmitting out to the single higher speed link, provided that your switch is peppy enough to not reorder the packets between their arrival at the etherchannel and delivery to the outbound port.  Realisitically, I'd expect some impact here, but not as much as the "receiving at the bond" case.

      The relevant bit of the bonding documentation follows:

      balance-rr: This mode is the only mode that will permit a single
              TCP/IP connection to stripe traffic across multiple
              interfaces. It is therefore the only mode that will allow a
              single TCP/IP stream to utilize more than one interface's
              worth of throughput.  This comes at a cost, however: the
              striping often results in peer systems receiving packets out
              of order, causing TCP/IP's congestion control system to kick
              in, often by retransmitting segments.

              It is possible to adjust TCP/IP's congestion limits by
              altering the net.ipv4.tcp_reordering sysctl parameter.  The
              usual default value is 3, and the maximum useful value is 127.
              For a four interface balance-rr bond, expect that a single
              TCP/IP stream will utilize no more than approximately 2.3
              interface's worth of throughput, even after adjusting
              tcp_reordering.

              Note that this out of order delivery occurs when both the
              sending and receiving systems are utilizing a multiple
              interface bond.  Consider a configuration in which a
              balance-rr bond feeds into a single higher capacity network
              channel (e.g., multiple 100Mb/sec ethernets feeding a single
              gigabit ethernet via an etherchannel capable switch).  In this
              configuration, traffic sent from the multiple 100Mb devices to
              a destination connected to the gigabit device will not see
              packets out of order.  However, traffic sent from the gigabit
              device to the multiple 100Mb devices may or may not see
              traffic out of order, depending upon the balance policy of the
              switch.  Many switches do not support any modes that stripe
              traffic (instead choosing a port based upon IP or MAC level
              addresses); for those devices, traffic flowing from the
              gigabit device to the many 100Mb devices will only utilize one
              interface.

              If you are utilizing protocols other than TCP/IP, UDP for
              example, and your application can tolerate out of order
              delivery, then this mode can allow for single stream datagram
              performance that scales near linearly as interfaces are added
              to the bond.