                     Using Linux TCP Splicing with HAProxy
                           Willy Tarreau <w@1wt.eu>
                                - 2007/01/06 -


Alexandre Cassen has started a project called Linux Layer7 Switching (L7SW),
whose goal is to provide kernel services to help userland proxies achieving
very high performance. Right now, the project consists in a loadable kernel
module providing TCP Splicing under Linux.

TCP Splicing is a method by which a userland proxy can tell the kernel that
it considers it has no added value on the data part of a connection, and that
the kernel can perform the transfers it itself, thus relieving the proxy from
a potentially heavy job. There are two advantages to this method :

  - it reduces the number of process wakeups
  - it reduces the number of data copies between user-space and kernel buffers

This method is particularly suited to protocols in which data is sent till
the end of the session. This is the case for FTP data for instance, and it
is also the case for the BODY part of HTTP/1.0.

The great news is that haproxy has been designed from the beginning with a
clear distinction between the headers and the DATA phase, so it was a child's
game to add hooks to Alex's library in it

Be careful! Both versions are to be considered BETA software ! Run them on
your systems if you want, but do not complain if it crashes twice a day !
Anyway, it seems stable on our test machines.

In order to use TCP Splicing on haproxy, you need :

  - Linux Layer7 Switching code version 0.1.1 : [ http://linux-l7sw.sf.net/ ]
  - Haproxy version 1.3.5 : [ http://haproxy.1wt.eu/download/1.3/src/ ]

Then, you must untar both packages in any location, let's assume you'll
be using /tmp. First extract l7sw and :

  $ cd /tmp
  $ tar zxf layer7switch-0.1.1.tar.gz
  $ cd layer7switch-0.1.1

L7SW currently only supports Linux kernel 2.6.19+. If you prefer to use it
on a more stable kernel, such as 2.6.16.X, you can apply this patch to the
L7SW directory :

  [ http://haproxy.1wt.eu/download/patches/tcp_splice-0.1.1-linux-2.6.16.diff ]

  $ patch -p1 -d kernel < tcp_splice-0.1.1-linux-2.6.16.diff

Alternatively, if you prefer to run it on 2.4.33+, you can apply this patch
to the L7SW directory :

  [ http://haproxy.1wt.eu/download/patches/tcp_splice-0.1.1-linux-2.4.33.diff ]

  $ patch -p1 -d kernel < tcp_splice-0.1.1-linux-2.4.33.diff

Then build the kernel module as described in the L7SW README. Basically, you
just have to do this once your tree has been patched :

  $ cd kernel
  $ make

You can either install the resulting module (tcp_splice) or load it now. During
early testing periods, it might be preferable to avoid installing anything and
just load it manually :

  $ sudo insmod tcp_splice.*o
  $ cd ..

Now that the module is loaded, you need to build the libtcpsplice library on
which haproxy currently relies :

  $ cd userland/libtcpsplice
  $ make
  $ cd ..

For the adventurous, there's also a proof of concept in the userlan/switchd
directory, it may be useful if you encounter problems with haproxy for
instance. But it is not needed at all here.

OK, L7SW is ready. Now you have to extract haproxy and tell it to build using
libtcpsplice :

  $ cd /tmp
  $ tar zxf haproxy-1.3.5.tar.gz
  $ cd haproxy-1.3.5
  $ make USE_TCPSPLICE=1 TCPSPLICEDIR=/tmp/layer7switch-0.1.1/userland/libtcpsplice

There are other options to make, which are hugely recommended, such as
CPU=, REGEX=, and above all, TARGET= so that you use the best syscalls and
functions for your system. Generally you will use TARGET=linux26, but 2.4 users
with an epoll-patched kernel will use TARGET=linux24e. This is very important
because failing to specify those options will disable important optimizations
which might hide the tcpsplice benefits ! Please consult the haproxy's README.

Now that you have haproxy built with support for tcpsplice, and that the module
is loaded, you have to write a config. There is an example in the 'examples'
directory. Basically, you just have to add the "option tcpsplice" keyword BOTH
in the frontend AND in the backend sections that you want to accelerate.

If the option is specified only in the frontend or in the backend, then no
acceleration will be used. It is designed this way to allow some front-back
combinations to use it without forcing others to use it. Of course, if you use
a single "listen" section, you just have to specify it once.

As of now (l7sw-0.1.1 and haproxy-1.3.5), you need the CAP_NETADMIN capability
to START and to RUN. For human beings, it means that you have to start haproxy
as root and keep it running as root, so it must not drop its priviledges. This
is somewhat annoying, but we'll try to find a solution later.

Also, l7sw-0.1.1 does not yet support TCP window scaling nor SACK. So you have
to disable both features on the proxy :

  $ sudo sysctl -w net.ipv4.tcp_window_scaling=0
  $ sudo sysctl -w net.ipv4.tcp_sack=0
  $ sudo sysctl -w net.ipv4.tcp_dsack=0
  $ sudo sysctl -w net.ipv4.tcp_tw_recycle=1

You can now check that everything works as expected. Run "vmstat 1" or "top"
in one terminal, and haproxy in another one :

  $ sudo ./haproxy -f examples/tcp-splicing-sample.cfg

Transfering large file through it should not affect it much. You should observe
something like 10% CPU instead of 95% when transferring 1 MB files at full
speed. You can play with the tcpsplice option in the configuration to see the
effects.


Troubleshooting
---------------

This software is still beta, and you will probably encounter some caveats.
I personnally ran into a few issues that we'll try to address with Alex. First
of all, I had occasionnal lockups on my SMP machine which I never had on an UP
one. So if you get problems on an SMP machine, please reboot it in UP and do
not lose your time on this.

I also noticed that sometimes, some sessions remained established even after
the end of the program. You might also see some situtations where even after
the proxy's exit, the traffic still passes through the system. It may happen
when you have a limited source port range and that you reuse a TIME_WAIT
session matching exactly the same source and destinations. This will need
to be addressed too.

You can play with tcp_splice variables and timeouts here in /proc/sys/net/ :

  $ ls /proc/sys/net/tcp_splice/
  debug_level        timeout_established  timeout_listen   timeout_synsent
  timeout_close      timeout_finwait      timeout_synack   timeout_timewait
  timeout_closewait  timeout_lastack      timeout_synrecv
  
  $ sysctl net/tcp_splice
  net.tcp_splice.debug_level = 0
  net.tcp_splice.timeout_synack = 120
  net.tcp_splice.timeout_listen = 120
  net.tcp_splice.timeout_lastack = 30
  net.tcp_splice.timeout_closewait = 60
  net.tcp_splice.timeout_close = 10
  net.tcp_splice.timeout_timewait = 120
  net.tcp_splice.timeout_finwait = 120
  net.tcp_splice.timeout_synrecv = 60
  net.tcp_splice.timeout_synsent = 120
  net.tcp_splice.timeout_established = 900

You can also consult the full session list here :

$ head /proc/net/tcp_splice_conn 
FromIP   FPrt ToIP     TPrt LocalIP  LPrt DestIP   DPrt State       Expires
0A000301 4EBB 0A000302 1F40 0A000302 817B 0A000301 0050 CLOSE             7
0A000301 4E9B 0A000302 1F40 0A000302 8165 0A000301 0050 CLOSE             7

Since a session exists at least in CLOSE state for 10 seconds, you just have
to consult this entry less than 10 seconds after a test to see a session.

Please report your successes, failures, suggestions or fixes to the L7SW
mailing list here (do not use the list to report other haproxy bugs) :

  https://lists.sourceforge.net/lists/listinfo/linux-l7sw-devel


Motivations
-----------

I've always wanted haproxy to be the fastest and most reliable software load
balancer available. L7SW is an opportunity to make get a huge performance boost
on high traffic sites (eg: photo sharing, streaming, ...). In turn, I find it a
shame that Alex wastes his time redevelopping a proxy as a proof of concept for
his kernel code. While it is a fun game to enter into, it really becomes harder
when you need to get close to customers' needs. So by porting haproxy early to
L7SW, I get both the opportunity to get an idea of what it will soon be capable
of, and help Alex spend more time on the complex kernel part.

Have fun !
Willy