[Linux-l7sw-devel] [ANNOUNCE] Haproxy 1.3.5 + tcp splicing

[Willy T. <w...@1w...>]

Hi Alex,

as promised, I've adapted Haproxy to support your TCP splicing code.
It was really easy, I'm relying on the libtcpsplice. I think we will
have to slightly change the initialization method, because I would
like to be able to init the socket without doing the modprobe, so
I would need to access tcp_splice_init(), which currently is static.
As you see, that's no big deal.

I've noticed that the process must run as root all the time because
a check on CAP_NET_ADMIN is performed at the top of the function
do_tcp_splice_set_ctl(). I think that we should relax the test so
that once the process has grabbed its IP_RAW socket and done the
initialization, it should be able to splice sockets without being
root anymore.

The performance gain is very interesting. I have installed haproxy
on my notebook, tux on my PC and my injector on the PC too. So the
HTTP requests travel from PC to NB to PC. Without TCP splicing, I
reach about 370 Mbps in both directions with 65% CPU used on the
notebook. With TCP splicing, the bit rate is exactly the same, but
the load goes down by a half. Since in both cases, the poor dirty
NIC is saturated first, it is normal that I get same data rates.
But seeing the CPU usage being cut in half is impressive. And the
CPU usage for haproxy goes down from 65% to about 5% in such a case.

Everything's not pink or blue though. I migrated the tests on my
notebook because the kernel repeatedly freezes on my main PC after
a random number of sessions (a few thousands), with 2.6.19.1,
2.6.16.37 and 2.4.33+patches. Since it's a dual athlon, I suspect
an SMP race condition somewhere. Everytime, all processes stopped
scheduling (vmstat stopped, the keyboard and mouse did not respond),
but Alt-SysRQ could still dump crap on the frame buffer, sync,
unmount and reboot. It has never happened even once on the notebook.

I have also noticed that when sessions are terminated, they seem to
be reused without passing through the process if a SYN comes in
during the TIME_WAIT state. I could even maintain a constant load
of about 45 sessions/s each transfering 1 MB while the proxy had
been stopped for one minute.

If I transfer small objects (say, empty or 1 kB pages), the ports
quickly wrap around (of course) and it seems like sessions cannot
establish anymore. I'll have to check this further, because it
looked exactly like a firewall I have benchmarked, and which
randomized SEQ numbers. I had to turn timestamps on to achieve
"decent" speeds of 50000 hits/s. Right now they are disabled, so
it might be a valid reason, but it's too late to retry right now.

I have written a small introduction doc explaining how to build
L7SW and haproxy and how to get them to work. There is also a
sample configuration. That way, you will not have to waste your
time maintaining your switchd (except for fun and experimentation).
BTW, I don't know why, but it often crashes with the following
message :

  switchd: scheduler.c:572: thread_fetch: Assertion `0' failed.
  Aborted

Oh, last but not least, before I forget. Running tcpdump on eth0
with a client connecting to the proxy from 127.0.0.1 made me see
the packets from/to 127.0.0.1 on eth0. I found it a bit strange,
but I think it's just the interface pointer associated to the
skb which causes this, and no packet really goes out of the NIC,
but it might be worth finding why it does this.


Useful resources :

   Haproxy 1.3.5 :
   http://haproxy.1wt.eu/download/1.3/bin/
   http://haproxy.1wt.eu/download/1.3/src/
   http://haproxy.1wt.eu/download/1.3/src/haproxy-1.3.5.tar.gz

   Install doc :
   http://haproxy.1wt.eu/download/1.3/doc/
   http://haproxy.1wt.eu/download/1.3/doc/tcp-splicing.txt

   Sample configuration :
   http://haproxy.1wt.eu/download/1.3/test/
   http://haproxy.1wt.eu/download/1.3/test/tcp-splicing-sample.cfg

I will send an announce on freshmeat to try to get more testers.

Have fun
Willy