Re: [Linux-l7sw-devel] [ANNOUNCE] Haproxy 1.3.5 + tcp splicing

[Alexandre C. <ac...@fr...>]

Hi Willy,

> as promised, I've adapted Haproxy to support your TCP splicing code.
> It was really easy, I'm relying on the libtcpsplice. I think we will
> have to slightly change the initialization method, because I would
> like to be able to init the socket without doing the modprobe, so
> I would need to access tcp_splice_init(), which currently is static.
> As you see, that's no big deal.

agreed will extend lib.

> I've noticed that the process must run as root all the time because
> a check on CAP_NET_ADMIN is performed at the top of the function
> do_tcp_splice_set_ctl(). I think that we should relax the test so
> that once the process has grabbed its IP_RAW socket and done the
> initialization, it should be able to splice sockets without being
> root anymore.

hmm... agreed too.

> The performance gain is very interesting. I have installed haproxy
> on my notebook, tux on my PC and my injector on the PC too. So the
> HTTP requests travel from PC to NB to PC. Without TCP splicing, I
> reach about 370 Mbps in both directions with 65% CPU used on the
> notebook. With TCP splicing, the bit rate is exactly the same, but
> the load goes down by a half. Since in both cases, the poor dirty
> NIC is saturated first, it is normal that I get same data rates.
> But seeing the CPU usage being cut in half is impressive. And the
> CPU usage for haproxy goes down from 65% to about 5% in such a case.

:) excellent :)

> Everything's not pink or blue though.

yes there still some pending points...

> I migrated the tests on my
> notebook because the kernel repeatedly freezes on my main PC after
> a random number of sessions (a few thousands), with 2.6.19.1,
> 2.6.16.37 and 2.4.33+patches. Since it's a dual athlon, I suspect
> an SMP race condition somewhere. Everytime, all processes stopped
> scheduling (vmstat stopped, the keyboard and mouse did not respond),
> but Alt-SysRQ could still dump crap on the frame buffer, sync,
> unmount and reboot. It has never happened even once on the notebook.

outch... a lock issue IMHO, have you got calltrace, I will try on a SMP
box during the week.

> I have also noticed that when sessions are terminated, they seem to
> be reused without passing through the process if a SYN comes in
> during the TIME_WAIT state. I could even maintain a constant load
> of about 45 sessions/s each transfering 1 MB while the proxy had
> been stopped for one minute.

hmm... a mistake in the FSM, will investigate.

> If I transfer small objects (say, empty or 1 kB pages), the ports
> quickly wrap around (of course) and it seems like sessions cannot
> establish anymore. I'll have to check this further, because it
> looked exactly like a firewall I have benchmarked, and which
> randomized SEQ numbers. I had to turn timestamps on to achieve
> "decent" speeds of 50000 hits/s. Right now they are disabled, so
> it might be a valid reason, but it's too late to retry right now.

IMHO, the same issue as previous point.

> I have written a small introduction doc explaining how to build
> L7SW and haproxy and how to get them to work. There is also a
> sample configuration. That way, you will not have to waste your
> time maintaining your switchd (except for fun and experimentation).
> BTW, I don't know why, but it often crashes with the following
> message :
> 
>   switchd: scheduler.c:572: thread_fetch: Assertion `0' failed.
>   Aborted

? outch... anyway switchd is just my debugging testbed, your haproxy is
BEST !

> Oh, last but not least, before I forget. Running tcpdump on eth0
> with a client connecting to the proxy from 127.0.0.1 made me see
> the packets from/to 127.0.0.1 on eth0. I found it a bit strange,
> but I think it's just the interface pointer associated to the
> skb which causes this, and no packet really goes out of the NIC,
> but it might be worth finding why it does this.

will investigate.

> Useful resources :
> 
>    Haproxy 1.3.5 :
>    http://haproxy.1wt.eu/download/1.3/bin/
>    http://haproxy.1wt.eu/download/1.3/src/
>    http://haproxy.1wt.eu/download/1.3/src/haproxy-1.3.5.tar.gz
> 
>    Install doc :
>    http://haproxy.1wt.eu/download/1.3/doc/
>    http://haproxy.1wt.eu/download/1.3/doc/tcp-splicing.txt
> 
>    Sample configuration :
>    http://haproxy.1wt.eu/download/1.3/test/
>    http://haproxy.1wt.eu/download/1.3/test/tcp-splicing-sample.cfg
> 
> I will send an announce on freshmeat to try to get more testers.

Nice, I will merge all your work during this week.

Thanks lot for your feedback !


best regs,
Alex