blob: 1341aea53951811937bd8dccd8b4134f3935c196 [file] [log] [blame]
willy tarreau0174f312005-12-18 01:02:42 +01001 -------------------
Willy Tarreau94b45912006-05-31 06:40:15 +02002 HAProxy
willy tarreau0174f312005-12-18 01:02:42 +01003 Reference Manual
4 -------------------
Willy Tarreau2272dc12006-09-03 10:19:38 +02005 version 1.3.2
willy tarreauc5f73ed2005-12-18 01:26:38 +01006 willy tarreau
Willy Tarreau2272dc12006-09-03 10:19:38 +02007 2006/09/03
willy tarreaueedaa9f2005-12-17 14:08:03 +01008
9============
10| Abstract |
11============
12
Willy Tarreau94b45912006-05-31 06:40:15 +020013HAProxy is a TCP/HTTP reverse proxy which is particularly suited for high
willy tarreaueedaa9f2005-12-17 14:08:03 +010014availability environments. Indeed, it can :
15 - route HTTP requests depending on statically assigned cookies ;
16 - spread the load among several servers while assuring server persistence
17 through the use of HTTP cookies ;
18 - switch to backup servers in the event a main one fails ;
19 - accept connections to special ports dedicated to service monitoring ;
20 - stop accepting connections without breaking existing ones ;
21 - add/modify/delete HTTP headers both ways ;
22 - block requests matching a particular pattern ;
willy tarreau532bb552006-05-13 18:40:37 +020023 - hold clients to the right application server depending on application
24 cookies
willy tarreau481132e2006-05-21 21:43:10 +020025 - report detailed status as HTML pages to authenticated users from an URI
26 intercepted from the application.
willy tarreaueedaa9f2005-12-17 14:08:03 +010027
28It needs very little resource. Its event-driven architecture allows it to easily
29handle thousands of simultaneous connections on hundreds of instances without
30risking the system's stability.
31
32====================
33| Start parameters |
34====================
35
36There are only a few command line options :
37
38 -f <configuration file>
39 -n <high limit for the total number of simultaneous connections>
willy tarreau532bb552006-05-13 18:40:37 +020040 = 'maxconn' in 'global' section
41 -N <high limit for the per-listener number of simultaneous connections>
42 = 'maxconn' in 'listen' or 'default' sections
willy tarreaueedaa9f2005-12-17 14:08:03 +010043 -d starts in foregreound with debugging mode enabled
44 -D starts in daemon mode
willy tarreau982249e2005-12-18 00:57:06 +010045 -q disable messages on output
46 -V displays messages on output even when -q or 'quiet' are specified.
47 -c only checks config file and exits with code 0 if no error was found, or
48 exits with code 1 if a syntax error was found.
willy tarreaufe2c5c12005-12-17 14:14:34 +010049 -p <pidfile> asks the process to write down each of its children's
50 pids to this file in daemon mode.
willy tarreau34f45302006-04-15 21:37:14 +020051 -sf specifies a list of pids to send a FINISH signal to after startup.
52 -st specifies a list of pids to send a TERMINATE signal to after startup.
willy tarreaueedaa9f2005-12-17 14:08:03 +010053 -s shows statistics (only if compiled in)
54 -l shows even more statistics (implies '-s')
Willy Tarreaude99e992007-04-16 00:53:59 +020055 -dk disables use of kqueue()
56 -ds disables use of speculative epoll()
willy tarreau64a3cc32005-12-18 01:13:11 +010057 -de disables use of epoll()
58 -dp disables use of poll()
willy tarreau34f45302006-04-15 21:37:14 +020059 -db disables background mode (stays in foreground, useful for debugging)
60 -m <megs> enforces a memory usage limit to a maximum of <megs> megabytes.
willy tarreaueedaa9f2005-12-17 14:08:03 +010061
willy tarreau532bb552006-05-13 18:40:37 +020062The maximal number of connections per proxy instance is used as the default
63parameter for each instance for which the 'maxconn' paramter is not set in the
64'listen' section.
willy tarreaueedaa9f2005-12-17 14:08:03 +010065
66The maximal number of total connections limits the number of connections used by
67the whole process if the 'maxconn' parameter is not set in the 'global' section.
68
69The debugging mode has the same effect as the 'debug' option in the 'global'
70section. When the proxy runs in this mode, it dumps every connections,
71disconnections, timestamps, and HTTP headers to stdout. This should NEVER
72be used in an init script since it will prevent the system from starting up.
73
willy tarreau34f45302006-04-15 21:37:14 +020074For debugging, the '-db' option is very useful as it temporarily disables
75daemon mode and multi-process mode. The service can then be stopped by simply
76pressing Ctrl-C, without having to edit the config nor run full debug.
77
willy tarreaueedaa9f2005-12-17 14:08:03 +010078Statistics are only available if compiled in with the 'STATTIME' option. It's
willy tarreau481132e2006-05-21 21:43:10 +020079only used during code optimization phases, and will soon disappear.
willy tarreaueedaa9f2005-12-17 14:08:03 +010080
willy tarreau532bb552006-05-13 18:40:37 +020081The '-st' and '-sf' options are used for hot reconfiguration (see below).
willy tarreau34f45302006-04-15 21:37:14 +020082
willy tarreaueedaa9f2005-12-17 14:08:03 +010083======================
84| Configuration file |
85======================
86
87Structure
88=========
89
90The configuration file parser ignores empty lines, spaces, tabs. Anything
91between a sharp ('#') not following a backslash ('\'), and the end of a line
92constitutes a comment and is ignored too.
93
94The configuration file is segmented in sections. A section begins whenever
95one of these 3 keywords are encountered :
96
97 - 'global'
98 - 'listen'
99 - 'defaults'
100
101Every parameter refer to the section beginning at the last one of these 3
102keywords.
103
104
1051) Global parameters
106====================
107
108Global parameters affect the whole process behaviour. They are all set in the
109'global' section. There may be several 'global' sections if needed, but their
110parameters will only be merged. Allowed parameters in 'global' section include
111the following ones :
112
113 - log <address> <facility> [max_level]
114 - maxconn <number>
115 - uid <user id>
116 - gid <group id>
Willy Tarreau95c20ac2007-03-25 15:39:23 +0200117 - user <user name>
118 - group <group name>
willy tarreaueedaa9f2005-12-17 14:08:03 +0100119 - chroot <directory>
120 - nbproc <number>
121 - daemon
122 - debug
Willy Tarreaude99e992007-04-16 00:53:59 +0200123 - nokqueue
124 - nosepoll
willy tarreau64a3cc32005-12-18 01:13:11 +0100125 - noepoll
126 - nopoll
willy tarreaueedaa9f2005-12-17 14:08:03 +0100127 - quiet
willy tarreaufe2c5c12005-12-17 14:14:34 +0100128 - pidfile <file>
willy tarreauc5f73ed2005-12-18 01:26:38 +0100129 - ulimit-n <number>
willy tarreau598da412005-12-18 01:07:29 +0100130 - stats
willy tarreaueedaa9f2005-12-17 14:08:03 +0100131
willy tarreauc5f73ed2005-12-18 01:26:38 +0100132
willy tarreaueedaa9f2005-12-17 14:08:03 +01001331.1) Event logging
134------------------
135Most events are logged : start, stop, servers going up and down, connections and
136errors. Each event generates a syslog message which can be sent to up to 2
137servers. The syntax is :
138
139 log <ip_address> <facility> [max_level]
140
141Connections are logged at level "info". Services initialization and servers
142going up are logged at level "notice", termination signals are logged at
143"warning", and definitive service termination, as well as loss of servers are
144logged at level "alert". The optional parameter <max_level> specifies above
145what level messages should be sent. Level can take one of these 8 values :
146
147 emerg, alert, crit, err, warning, notice, info, debug
148
149For backwards compatibility with versions 1.1.16 and earlier, the default level
150value is "debug" if not specified.
151
152Permitted facilities are :
153 kern, user, mail, daemon, auth, syslog, lpr, news,
154 uucp, cron, auth2, ftp, ntp, audit, alert, cron2,
155 local0, local1, local2, local3, local4, local5, local6, local7
156
157According to RFC3164, messages are truncated to 1024 bytes before being emitted.
158
159Example :
160---------
161 global
willy tarreauc5f73ed2005-12-18 01:26:38 +0100162 log 192.168.2.200 local3
163 log 127.0.0.1 local4 notice
164
willy tarreaueedaa9f2005-12-17 14:08:03 +0100165
1661.2) limiting the number of connections
167---------------------------------------
168It is possible and recommended to limit the global number of per-process
willy tarreauc5f73ed2005-12-18 01:26:38 +0100169connections using the 'maxconn' global keyword. Since one connection includes
170both a client and a server, it means that the max number of TCP sessions will
171be about the double of this number. It's important to understand this when
172trying to find best values for 'ulimit -n' before starting the proxy. To
173anticipate the number of sockets needed, all these parameters must be counted :
willy tarreaueedaa9f2005-12-17 14:08:03 +0100174
175 - 1 socket per incoming connection
176 - 1 socket per outgoing connection
177 - 1 socket per address/port/proxy tuple.
178 - 1 socket per server being health-checked
179 - 1 socket for all logs
180
181In simple configurations where each proxy only listens one one address/port,
willy tarreauc5f73ed2005-12-18 01:26:38 +0100182set the limit of file descriptors (ulimit -n) to
183(2 * maxconn + nbproxies + nbservers + 1). Starting with versions 1.1.32/1.2.6,
184it is now possible to set the limit in the configuration using the 'ulimit-n'
185global keyword, provided the proxy is started as root. This puts an end to the
186recurrent problem of ensuring that the system limits are adapted to the proxy
187values. Note that these limits are per-process.
188
189Example :
190---------
191 global
192 maxconn 32000
193 ulimit-n 65536
194
willy tarreaueedaa9f2005-12-17 14:08:03 +0100195
1961.3) Drop of priviledges
197------------------------
198In order to reduce the risk and consequences of attacks, in the event where a
199yet non-identified vulnerability would be successfully exploited, it's possible
200to lower the process priviledges and even isolate it in a riskless directory.
201
202In the 'global' section, the 'uid' parameter sets a numerical user identifier
203which the process will switch to after binding its listening sockets. The value
204'0', which normally represents the super-user, here indicates that the UID must
205not change during startup. It's the default behaviour. The 'gid' parameter does
Willy Tarreau95c20ac2007-03-25 15:39:23 +0200206the same for the group identifier. If setting an uid is not possible because of
207deployment constraints, it is possible to set a user name with the 'user'
208keyword followed by a valid user name. The same is true for the gid. It is
209possible to specify a group name after the 'group' keyword.
210
211It is particularly advised against use of generic accounts such as 'nobody'
212because it has the same consequences as using 'root' if other services use
213them.
willy tarreaueedaa9f2005-12-17 14:08:03 +0100214
215The 'chroot' parameter makes the process isolate itself in an empty directory
216just before switching its UID. This type of isolation (chroot) can sometimes
217be worked around on certain OS (Linux, Solaris), provided that the attacker
218has gained 'root' priviledges and has the ability to use or create a directory.
219For this reason, it's capital to use a dedicated directory and not to share one
220between several services of different nature. To make isolation more resistant,
221it's recommended to use an empty directory without any right, and to change the
222UID of the process so that it cannot do anything there.
223
224Note: in the event where such a vulnerability would be exploited, it's most
225likely that first attempts would kill the process due to 'Segmentation Fault',
226'Bus Error' or 'Illegal Instruction' signals. Eventhough it's true that
227isolating the server reduces the risks of intrusion, it's sometimes useful to
228find why a process dies, via the analysis of a 'core' file, although very rare
229(the last bug of this sort was fixed in 1.1.9). For security reasons, most
230systems disable the generation of core file when a process changes its UID. So
231the two workarounds are either to start the process from a restricted user
232account, which will not be able to chroot itself, or start it as root and not
233change the UID. In both cases the core will be either in the start or the chroot
234directories. Do not forget to allow core dumps prior to start the process :
235
236# ulimit -c unlimited
237
238Example :
239---------
240
Willy Tarreau95c20ac2007-03-25 15:39:23 +0200241 # with uid/gid
willy tarreaueedaa9f2005-12-17 14:08:03 +0100242 global
willy tarreauc5f73ed2005-12-18 01:26:38 +0100243 uid 30000
244 gid 30000
245 chroot /var/chroot/haproxy
246
Willy Tarreau95c20ac2007-03-25 15:39:23 +0200247 # with user/group
248 global
249 user haproxy
250 group public
251 chroot /var/chroot/haproxy
252
willy tarreaueedaa9f2005-12-17 14:08:03 +0100253
2541.4) Startup modes
255------------------
willy tarreau34f45302006-04-15 21:37:14 +0200256The service can start in several different modes :
willy tarreaueedaa9f2005-12-17 14:08:03 +0100257 - foreground / background
258 - quiet / normal / debug
259
260The default mode is normal, foreground, which means that the program doesn't
261return once started. NEVER EVER use this mode in a system startup script, or
262the system won't boot. It needs to be started in background, so that it
263returns immediately after forking. That's accomplished by the 'daemon' option
264in the 'global' section, which is the equivalent of the '-D' command line
265argument.
266
willy tarreau34f45302006-04-15 21:37:14 +0200267The '-db' command line argument overrides the 'daemon' and 'nbproc' global
268options to make the process run in normal, foreground mode.
269
willy tarreaueedaa9f2005-12-17 14:08:03 +0100270Moreover, certain alert messages are still sent to the standard output even
271in 'daemon' mode. To make them disappear, simply add the 'quiet' option in the
272'global' section. This option has no command-line equivalent.
273
274Last, the 'debug' mode, enabled with the 'debug' option in the 'global' section,
275and which is equivalent of the '-d' option, allows deep TCP/HTTP analysis, with
276timestamped display of each connection, disconnection, and HTTP headers for both
277ways. This mode is incompatible with 'daemon' and 'quiet' modes for obvious
278reasons.
279
willy tarreauc5f73ed2005-12-18 01:26:38 +0100280
willy tarreaueedaa9f2005-12-17 14:08:03 +01002811.5) Increasing the overall processing power
282--------------------------------------------
283On multi-processor systems, it may seem to be a shame to use only one processor,
willy tarreau982249e2005-12-18 00:57:06 +0100284eventhough the load needed to saturate a recent processor is far above common
willy tarreaueedaa9f2005-12-17 14:08:03 +0100285usage. Anyway, for very specific needs, the proxy can start several processes
286between which the operating system will spread the incoming connections. The
287number of processes is controlled by the 'nbproc' parameter in the 'global'
willy tarreau4302f492005-12-18 01:00:37 +0100288section. It defaults to 1, and obviously works only in 'daemon' mode. One
289typical usage of this parameter has been to workaround the default per-process
290file-descriptor limit that Solaris imposes to user processes.
willy tarreaueedaa9f2005-12-17 14:08:03 +0100291
292Example :
293---------
294
295 global
willy tarreauc5f73ed2005-12-18 01:26:38 +0100296 daemon
297 quiet
298 nbproc 2
willy tarreaueedaa9f2005-12-17 14:08:03 +0100299
300
willy tarreaufe2c5c12005-12-17 14:14:34 +01003011.6) Helping process management
302-------------------------------
303Haproxy now supports the notion of pidfile. If the '-p' command line argument,
304or the 'pidfile' global option is followed with a file name, this file will be
305removed, then filled with all children's pids, one per line (only in daemon
306mode). This file is NOT within the chroot, which allows to work with a readonly
307 chroot. It will be owned by the user starting the process, and will have
308permissions 0644.
309
310Example :
311---------
312
313 global
314 daemon
315 quiet
willy tarreauc5f73ed2005-12-18 01:26:38 +0100316 nbproc 2
willy tarreaufe2c5c12005-12-17 14:14:34 +0100317 pidfile /var/run/haproxy-private.pid
318
319 # to stop only those processes among others :
320 # kill $(</var/run/haproxy-private.pid)
321
willy tarreau34f45302006-04-15 21:37:14 +0200322 # to reload a new configuration with minimal service impact and without
323 # breaking existing sessions :
324 # haproxy -f haproxy.cfg -p $(</var/run/haproxy-private.pid) -st $(</var/run/haproxy-private.pid)
willy tarreaufe2c5c12005-12-17 14:14:34 +0100325
willy tarreau64a3cc32005-12-18 01:13:11 +01003261.7) Polling mechanisms
327-----------------------
328Starting from version 1.2.5, haproxy supports the poll() and epoll() polling
329mechanisms. On systems where select() is limited by FD_SETSIZE (like Solaris),
330poll() can be an interesting alternative. Performance tests show that Solaris'
331poll() performance does not decay as fast as the numbers of sockets increase,
332making it a safe solution for high loads. However, Solaris already uses poll()
333to emulate select(), so as long as the number of sockets has no reason to go
334higher than FD_SETSIZE, poll() should not provide any better performance. On
335Linux systems with the epoll() patch (or any 2.6 version), haproxy will use
336epoll() which is extremely fast and non dependant on the number of sockets.
337Tests have shown constant performance from 1 to 20000 simultaneous sessions.
Willy Tarreaude99e992007-04-16 00:53:59 +0200338Version 1.3.9 introduced kqueue() for FreeBSD/OpenBSD, and speculative epoll()
339which consists in trying to perform I/O before queuing the events via syscalls.
willy tarreau64a3cc32005-12-18 01:13:11 +0100340
Willy Tarreaude99e992007-04-16 00:53:59 +0200341Haproxy will use kqueue() or speculative epoll() when available, then epoll(),
342and will fall back to poll(), then to select(). However, if for any reason you
343need to disable epoll() or poll() (eg. because of a bug or just to compare
344performance), new global options have been created for this matter : 'nosepoll',
345'nokqueue', 'noepoll' and 'nopoll'.
willy tarreau64a3cc32005-12-18 01:13:11 +0100346
347Example :
348---------
349
350 global
351 # use only select()
352 noepoll
353 nopoll
354
355Note :
356------
357For the sake of configuration file portability, these options are accepted but
358ignored if the poll() or epoll() mechanisms have not been enabled at compile
359time.
360
Willy Tarreaude99e992007-04-16 00:53:59 +0200361To make debugging easier, the '-de' runtime argument disables epoll support,
362the '-dp' argument disables poll support, '-dk' disables kqueue and '-ds'
363disables speculative epoll(). They are respectively equivalent to 'noepoll',
364'nopoll', 'nokqueue' and 'nosepoll'.
willy tarreau64a3cc32005-12-18 01:13:11 +0100365
366
willy tarreaueedaa9f2005-12-17 14:08:03 +01003672) Declaration of a listening service
368=====================================
369
370Service sections start with the 'listen' keyword :
371
372 listen <instance_name> [ <IP_address>:<port_range>[,...] ]
373
374- <instance_name> is the name of the instance. This name will be reported in
375 logs, so it is good to have it reflect the proxied service. No unicity test
376 is done on this name, and it's not mandatory for it to be unique, but highly
377 recommended.
378
379- <IP_address> is the IP address the proxy binds to. Empty address, '*' and
380 '0.0.0.0' all mean that the proxy listens to all valid addresses on the
381 system.
382
383- <port_range> is either a unique port, or a port range for which the proxy will
384 accept connections for the IP address specified above. This range can be :
385 - a numerical port (ex: '80')
386 - a dash-delimited ports range explicitly stating the lower and upper bounds
387 (ex: '2000-2100') which are included in the range.
388
389 Particular care must be taken against port ranges, because every <addr:port>
390 couple consumes one socket (=a file descriptor), so it's easy to eat lots of
391 descriptors with a simple range. The <addr:port> couple must be used only once
392 among all instances running on a same system. Please note that attaching to
393 ports lower than 1024 need particular priviledges to start the program, which
394 are independant of the 'uid' parameter.
395
396- the <IP_address>:<port_range> couple may be repeated indefinitely to require
397 the proxy to listen to other addresses and/or ports. To achieve this, simply
398 separate them with a coma.
399
400Examples :
401---------
402 listen http_proxy :80
403 listen x11_proxy 127.0.0.1:6000-6009
404 listen smtp_proxy 127.0.0.1:25,127.0.0.1:587
405 listen ldap_proxy :389,:663
406
407In the event that all addresses do not fit line width, it's preferable to
408detach secondary addresses on other lines with the 'bind' keyword. If this
409keyword is used, it's not even necessary to specify the first address on the
410'listen' line, which sometimes makes multiple configuration handling easier :
411
412 bind [ <IP_address>:<port_range>[,...] ]
413
414Examples :
415----------
416 listen http_proxy
417 bind :80,:443
willy tarreauc5f73ed2005-12-18 01:26:38 +0100418 bind 10.0.0.1:10080,10.0.0.1:10443
419
willy tarreaueedaa9f2005-12-17 14:08:03 +0100420
4212.1) Inhibiting a service
422-------------------------
423A service may be disabled for maintenance reasons, without needing to comment
424out the whole section, simply by specifying the 'disabled' keyword in the
425section to be disabled :
426
427 listen smtp_proxy 0.0.0.0:25
willy tarreauc5f73ed2005-12-18 01:26:38 +0100428 disabled
willy tarreaueedaa9f2005-12-17 14:08:03 +0100429
430Note: the 'enabled' keyword allows to enable a service which has been disabled
431 previously by a default configuration.
432
willy tarreauc5f73ed2005-12-18 01:26:38 +0100433
willy tarreaueedaa9f2005-12-17 14:08:03 +01004342.2) Modes of operation
435-----------------------
436A service can work in 3 different distinct modes :
437 - TCP
438 - HTTP
willy tarreau532bb552006-05-13 18:40:37 +0200439 - health
willy tarreaueedaa9f2005-12-17 14:08:03 +0100440
441TCP mode
442--------
443In this mode, the service relays TCP connections as soon as they're established,
444towards one or several servers. No processing is done on the stream. It's only
445an association of source(addr:port) -> destination(addr:port). To use this mode,
446you must specify 'mode tcp' in the 'listen' section. This is the default mode.
447
448Example :
449---------
450 listen smtp_proxy 0.0.0.0:25
willy tarreauc5f73ed2005-12-18 01:26:38 +0100451 mode tcp
willy tarreaueedaa9f2005-12-17 14:08:03 +0100452
453HTTP mode
454---------
455In this mode, the service relays TCP connections towards one or several servers,
456when it has enough informations to decide, which normally means that all HTTP
457headers have been read. Some of them may be scanned for a cookie or a pattern
458matching a regex. To use this mode, specify 'mode http' in the 'listen' section.
459
460Example :
461---------
462 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +0100463 mode http
willy tarreaueedaa9f2005-12-17 14:08:03 +0100464
465Health-checking mode
466--------------------
467This mode provides a way for external components to check the proxy's health.
468It is meant to be used with intelligent load-balancers which can use send/expect
469scripts to check for all of their servers' availability. This one simply accepts
willy tarreau197e8ec2005-12-17 14:10:59 +0100470the connection, returns the word 'OK' and closes it. If the 'option httpchk' is
471set, then the reply will be 'HTTP/1.0 200 OK' with no data, so that it can be
472tested from a tool which supports HTTP health-checks. To enable it, simply
willy tarreaueedaa9f2005-12-17 14:08:03 +0100473specify 'health' as the working mode :
474
475Example :
476---------
willy tarreau197e8ec2005-12-17 14:10:59 +0100477 # simple response : 'OK'
willy tarreaueedaa9f2005-12-17 14:08:03 +0100478 listen health_check 0.0.0.0:60000
willy tarreauc5f73ed2005-12-18 01:26:38 +0100479 mode health
willy tarreaueedaa9f2005-12-17 14:08:03 +0100480
willy tarreau197e8ec2005-12-17 14:10:59 +0100481 # HTTP response : 'HTTP/1.0 200 OK'
482 listen http_health_check 0.0.0.0:60001
willy tarreauc5f73ed2005-12-18 01:26:38 +0100483 mode health
484 option httpchk
485
willy tarreau532bb552006-05-13 18:40:37 +02004862.2.1 Monitoring
487----------------
willy tarreauc5f73ed2005-12-18 01:26:38 +0100488Versions 1.1.32 and 1.2.6 provide a new solution to check the proxy's
489availability without perturbating the service. The 'monitor-net' keyword was
490created to specify a network of equipments which CANNOT use the service for
491anything but health-checks. This is particularly suited to TCP proxies, because
492it prevents the proxy from relaying the monitor's connection to the remote
493server.
494
495When used with TCP, the connection is accepted then closed and nothing is
496logged. This is enough for a front-end load-balancer to detect the service as
497available.
willy tarreau197e8ec2005-12-17 14:10:59 +0100498
willy tarreauc5f73ed2005-12-18 01:26:38 +0100499When used with HTTP, the connection is accepted, nothing is logged, the
500following response is sent, then the session is closed : "HTTP/1.0 200 OK".
501This is normally enough for any front-end HTTP load-balancer to detect the
502service as available too, both with TCP and HTTP checks.
503
504Proxies using the "monitor-net" keyword can remove the "option dontlognull", as
505it will make them log empty connections from hosts outside the monitoring
506network.
507
508Example :
509---------
510
511 listen tse-proxy
512 bind :3389,:1494,:5900 # TSE, ICA and VNC at once.
513 mode tcp
514 balance roundrobin
515 server tse-farm 192.168.1.10
516 monitor-net 192.168.1.252/31 # L4 load-balancers on .252 and .253
517
willy tarreaueedaa9f2005-12-17 14:08:03 +0100518
Willy Tarreau1c47f852006-07-09 08:22:27 +0200519When the system executing the checks is located behind a proxy, the monitor-net
520keyword cannot be used because haproxy will always see the proxy's address. To
521overcome this limitation, version 1.2.15 brought the 'monitor-uri' keyword. It
522defines an URI which will not be forwarded nor logged, but for which haproxy
523will immediately send an "HTTP/1.0 200 OK" response. This makes it possible to
524check the validity of the reverse-proxy->haproxy chain with one request. It can
525be used in HTTPS checks in front of an stunnel -> haproxy combination for
526instance. Obviously, this keyword is only valid in HTTP mode, otherwise there
527is no notion of URI. Note that the method and HTTP versions are simply ignored.
528
529Example :
530---------
531
532 listen stunnel_backend :8080
533 mode http
534 balance roundrobin
535 server web1 192.168.1.10:80 check
536 server web2 192.168.1.11:80 check
537 monitor-uri /haproxy_test
538
539
willy tarreaueedaa9f2005-12-17 14:08:03 +01005402.3) Limiting the number of simultaneous connections
541----------------------------------------------------
542The 'maxconn' parameter allows a proxy to refuse connections above a certain
543amount of simultaneous ones. When the limit is reached, it simply stops
544listening, but the system may still be accepting them because of the back log
willy tarreau982249e2005-12-18 00:57:06 +0100545queue. These connections will be processed later when other ones have freed
willy tarreaueedaa9f2005-12-17 14:08:03 +0100546some slots. This provides a serialization effect which helps very fragile
willy tarreau34f45302006-04-15 21:37:14 +0200547servers resist to high loads. See further for system limitations.
willy tarreaueedaa9f2005-12-17 14:08:03 +0100548
549Example :
550---------
551 listen tiny_server 0.0.0.0:80
552 maxconn 10
553
554
5552.4) Soft stop
556--------------
557It is possible to stop services without breaking existing connections by the
willy tarreau22739ef2006-01-20 20:43:32 +0100558sending of the SIGUSR1 signal to the process. All services are then put into
willy tarreaueedaa9f2005-12-17 14:08:03 +0100559soft-stop state, which means that they will refuse to accept new connections,
560except for those which have a non-zero value in the 'grace' parameter, in which
561case they will still accept connections for the specified amount of time, in
willy tarreau22739ef2006-01-20 20:43:32 +0100562milliseconds. This makes it possible to tell a load-balancer that the service
563is failing, while still doing the job during the time it needs to detect it.
willy tarreaueedaa9f2005-12-17 14:08:03 +0100564
565Note: active connections are never killed. In the worst case, the user will have
566to wait for all of them to close or to time-out, or simply kill the process
willy tarreau22739ef2006-01-20 20:43:32 +0100567normally (SIGTERM). The default 'grace' value is '0'.
willy tarreaueedaa9f2005-12-17 14:08:03 +0100568
569Example :
570---------
571 # enter soft stop after 'killall -USR1 haproxy'
572 # the service will still run 10 seconds after the signal
573 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +0100574 mode http
575 grace 10000
willy tarreaueedaa9f2005-12-17 14:08:03 +0100576
577 # this port is dedicated to a load-balancer, and must fail immediately
578 listen health_check 0.0.0.0:60000
willy tarreauc5f73ed2005-12-18 01:26:38 +0100579 mode health
580 grace 0
willy tarreaueedaa9f2005-12-17 14:08:03 +0100581
582
willy tarreau39df2dc2006-01-29 21:56:05 +0100583As of version 1.2.8, a new soft-reconfiguration mechanism has been introduced.
willy tarreau22739ef2006-01-20 20:43:32 +0100584It is now possible to "pause" all the proxies by sending a SIGTTOU signal to
585the processes. This will disable the listening socket without breaking existing
586connections. After that, sending a SIGTTIN signal to those processes enables
587the listening sockets again. This is very useful to try to load a new
588configuration or even a new version of haproxy without breaking existing
589connections. If the load succeeds, then simply send a SIGUSR1 which will make
590the previous proxies exit immediately once their sessions are closed ; and if
591the load fails, then simply send a SIGTTIN to restore the service immediately.
592Please note that the 'grace' parameter is ignored for SIGTTOU, as well as for
593SIGUSR1 when the process was in the pause mode. Please also note that it would
594be useful to save the pidfile before starting a new instance.
595
willy tarreau34f45302006-04-15 21:37:14 +0200596This mechanism fully exploited since 1.2.11 with the '-st' and '-sf' options
willy tarreau532bb552006-05-13 18:40:37 +0200597(see below).
598
5992.4.1) Hot reconfiguration
600--------------------------
601The '-st' and '-sf' command line options are used to inform previously running
602processes that a configuration is being reloaded. They will receive the SIGTTOU
603signal to ask them to temporarily stop listening to the ports so that the new
604process can grab them. If anything wrong happens, the new process will send
605them a SIGTTIN to tell them to re-listen to the ports and continue their normal
606work. Otherwise, it will either ask them to finish (-sf) their work then softly
607exit, or immediately terminate (-st), breaking existing sessions. A typical use
608of this allows a configuration reload without service interruption :
609
610 # haproxy -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
611
willy tarreau22739ef2006-01-20 20:43:32 +0100612
willy tarreaueedaa9f2005-12-17 14:08:03 +01006132.5) Connections expiration time
614--------------------------------
615It is possible (and recommended) to configure several time-outs on TCP
616connections. Three independant timers are adjustable with values specified
617in milliseconds. A session will be terminated if either one of these timers
618expire.
619
620 - the time we accept to wait for data from the client, or for the client to
621 accept data : 'clitimeout' :
622
willy tarreauc5f73ed2005-12-18 01:26:38 +0100623 # client time-out set to 2mn30.
624 clitimeout 150000
willy tarreaueedaa9f2005-12-17 14:08:03 +0100625
626 - the time we accept to wait for data from the server, or for the server to
627 accept data : 'srvtimeout' :
628
willy tarreauc5f73ed2005-12-18 01:26:38 +0100629 # server time-out set to 30s.
630 srvtimeout 30000
willy tarreaueedaa9f2005-12-17 14:08:03 +0100631
632 - the time we accept to wait for a connection to establish on a server :
633 'contimeout' :
634
635 # we give up if the connection does not complete within 4 seconds
willy tarreauc5f73ed2005-12-18 01:26:38 +0100636 contimeout 4000
willy tarreaueedaa9f2005-12-17 14:08:03 +0100637
638Notes :
639-------
640 - 'contimeout' and 'srvtimeout' have no sense on 'health' mode servers ;
641 - under high loads, or with a saturated or defective network, it's possible
642 that some packets get lost. Since the first TCP retransmit only happens
643 after 3 seconds, a time-out equal to, or lower than 3 seconds cannot
644 compensate for a packet loss. A 4 seconds time-out seems a reasonable
645 minimum which will considerably reduce connection failures.
646
willy tarreauc5f73ed2005-12-18 01:26:38 +0100647
willy tarreaueedaa9f2005-12-17 14:08:03 +01006482.6) Attempts to reconnect
649--------------------------
650After a connection failure to a server, it is possible to retry, potentially
651on another server. This is useful if health-checks are too rare and you don't
652want the clients to see the failures. The number of attempts to reconnect is
653set by the 'retries' paramter.
654
655Example :
656---------
willy tarreauc5f73ed2005-12-18 01:26:38 +0100657 # we can retry 3 times max after a failure
658 retries 3
willy tarreaueedaa9f2005-12-17 14:08:03 +0100659
willy tarreau34f45302006-04-15 21:37:14 +0200660Please note that the reconnection attempt may lead to getting the connection
661sent to a new server if the original one died between connection attempts.
662
willy tarreaueedaa9f2005-12-17 14:08:03 +0100663
6642.7) Address of the dispatch server (deprecated)
665------------------------------------------------
666The server which will be sent all new connections is defined by the 'dispatch'
667parameter, in the form <address>:<port>. It generally is dedicated to unknown
668connections and will assign them a cookie, in case of HTTP persistence mode,
669or simply is a single server in case of generic TCP proxy. This old mode is only
670provided for backwards compatibility, but doesn't allow to check remote servers
671state, and has a rather limited usage. All new setups should switch to 'balance'
672mode. The principle of the dispatcher is to be able to perform the load
673balancing itself, but work only on new clients so that the server doesn't need
674to be a big machine.
675
676Example :
677---------
willy tarreauc5f73ed2005-12-18 01:26:38 +0100678 # all new connections go there
679 dispatch 192.168.1.2:80
willy tarreaueedaa9f2005-12-17 14:08:03 +0100680
681Note :
682------
683This parameter has no sense for 'health' servers, and is incompatible with
684'balance' mode.
685
686
6872.8) Outgoing source address
688----------------------------
689It is often necessary to bind to a particular address when connecting to some
690remote hosts. This is done via the 'source' parameter which is a per-proxy
691parameter. A newer version may allow to fix different sources to reach different
692servers. The syntax is 'source <address>[:<port>]', where <address> is a valid
693local address (or '0.0.0.0' or '*' or empty to let the system choose), and
694<port> is an optional parameter allowing the user to force the source port for
695very specific needs. If the port is not specified or is '0', the system will
696choose a free port. Note that as of version 1.1.18, the servers health checks
697are also performed from the same source.
698
699Examples :
700----------
701 listen http_proxy *:80
willy tarreauc5f73ed2005-12-18 01:26:38 +0100702 # all connections take 192.168.1.200 as source address
703 source 192.168.1.200:0
willy tarreaueedaa9f2005-12-17 14:08:03 +0100704
705 listen rlogin_proxy *:513
willy tarreauc5f73ed2005-12-18 01:26:38 +0100706 # use address 192.168.1.200 and the reserved port 900 (needs to be root)
707 source 192.168.1.200:900
willy tarreaueedaa9f2005-12-17 14:08:03 +0100708
709
7102.9) Setting the cookie name
711----------------------------
712In HTTP mode, it is possible to look for a particular cookie which will contain
713a server identifier which should handle the connection. The cookie name is set
714via the 'cookie' parameter.
715
716Example :
717---------
718 listen http_proxy :80
willy tarreauc5f73ed2005-12-18 01:26:38 +0100719 mode http
720 cookie SERVERID
willy tarreaueedaa9f2005-12-17 14:08:03 +0100721
722It is possible to change the cookie behaviour to get a smarter persistence,
723depending on applications. It is notably possible to delete or modify a cookie
724emitted by a server, insert a cookie identifying the server in an HTTP response
725and even add a header to tell upstream caches not to cache this response.
726
727Examples :
728----------
729
730To remove the cookie for direct accesses (ie when the server matches the one
731which was specified in the client cookie) :
732
willy tarreauc5f73ed2005-12-18 01:26:38 +0100733 cookie SERVERID indirect
willy tarreaueedaa9f2005-12-17 14:08:03 +0100734
735To replace the cookie value with the one assigned to the server if any (no
736cookie will be created if the server does not provide one, nor if the
737configuration does not provide one). This lets the application put the cookie
738exactly on certain pages (eg: successful authentication) :
739
willy tarreauc5f73ed2005-12-18 01:26:38 +0100740 cookie SERVERID rewrite
willy tarreaueedaa9f2005-12-17 14:08:03 +0100741
742To create a new cookie and assign the server identifier to it (in this case, all
743servers should be associated with a valid cookie, since no cookie will simply
744delete the cookie from the client's browser) :
745
willy tarreauc5f73ed2005-12-18 01:26:38 +0100746 cookie SERVERID insert
willy tarreaueedaa9f2005-12-17 14:08:03 +0100747
willy tarreau0174f312005-12-18 01:02:42 +0100748To reuse an existing application cookie and prefix it with the server's
749identifier, and remove it in the request, use the 'prefix' option. This allows
750to insert a haproxy in front of an application without risking to break clients
751which does not support more than one cookie :
752
willy tarreauc5f73ed2005-12-18 01:26:38 +0100753 cookie JSESSIONID prefix
willy tarreau0174f312005-12-18 01:02:42 +0100754
willy tarreaueedaa9f2005-12-17 14:08:03 +0100755To insert a cookie and ensure that no upstream cache will store it, add the
756'nocache' option :
757
willy tarreauc5f73ed2005-12-18 01:26:38 +0100758 cookie SERVERID insert nocache
willy tarreaueedaa9f2005-12-17 14:08:03 +0100759
760To insert a cookie only after a POST request, add 'postonly' after 'insert'.
761This has the advantage that there's no risk of caching, and that all pages
762seen before the POST one can still be cached :
763
willy tarreauc5f73ed2005-12-18 01:26:38 +0100764 cookie SERVERID insert postonly
willy tarreaueedaa9f2005-12-17 14:08:03 +0100765
766Notes :
767-----------
768- it is possible to combine 'insert' with 'indirect' or 'rewrite' to adapt to
769 applications which already generate the cookie with an invalid content.
770
771- in the case where 'insert' and 'indirect' are both specified, the cookie is
willy tarreau0174f312005-12-18 01:02:42 +0100772 never transmitted to the server, since it wouldn't understand it. This is the
773 most application-transparent mode.
willy tarreaueedaa9f2005-12-17 14:08:03 +0100774
775- it is particularly recommended to use 'nocache' in 'insert' mode if any
776 upstream HTTP/1.0 cache is susceptible to cache the result, because this may
777 lead to many clients going to the same server, or even worse, some clients
778 having their server changed while retrieving a page from the cache.
779
willy tarreau0174f312005-12-18 01:02:42 +0100780- the 'prefix' mode normally does not need 'indirect', 'nocache', nor
781 'postonly', because just as in the 'rewrite' mode, it relies on the
782 application to know when a cookie can be emitted. However, since it has to
783 fix the cookie name in every subsequent requests, you must ensure that the
784 proxy will be used without any "HTTP keep-alive". Use option "httpclose" if
785 unsure.
786
willy tarreaueedaa9f2005-12-17 14:08:03 +0100787- when the application is well known and controlled, the best method is to
788 only add the persistence cookie on a POST form because it's up to the
willy tarreau0174f312005-12-18 01:02:42 +0100789 application to select which page it wants the upstream servers to cache. In
790 this case, you would use 'insert postonly indirect'.
willy tarreaueedaa9f2005-12-17 14:08:03 +0100791
willy tarreauc5f73ed2005-12-18 01:26:38 +0100792
willy tarreaueedaa9f2005-12-17 14:08:03 +01007932.10) Associating a cookie value with a server
794----------------------------------------------
795In HTTP mode, it's possible to associate a cookie value to each server. This
796was initially used in combination with 'dispatch' mode to handle direct accesses
797but it is now the standard way of doing the load balancing. The syntax is :
798
799 server <identifier> <address>:<port> cookie <value>
800
801- <identifier> is any name which can be used to identify the server in the logs.
802- <address>:<port> specifies where the server is bound.
803- <value> is the value to put in or to read from the cookie.
804
805Example : the 'SERVERID' cookie can be either 'server01' or 'server02'
806---------
807 listen http_proxy :80
willy tarreauc5f73ed2005-12-18 01:26:38 +0100808 mode http
809 cookie SERVERID
810 dispatch 192.168.1.100:80
811 server web1 192.168.1.1:80 cookie server01
812 server web2 192.168.1.2:80 cookie server02
willy tarreaueedaa9f2005-12-17 14:08:03 +0100813
814Warning : the syntax has changed since version 1.0 !
815---------
816
willy tarreauc5f73ed2005-12-18 01:26:38 +0100817
willy tarreau598da412005-12-18 01:07:29 +01008182.11) Application Cookies
819-------------------------
820Since 1.2.4 it is possible to catch the cookie that comes from an
821application server in order to apply "application session stickyness".
822The server's response is searched for 'appsession' cookie, the first
823'len' bytes are used for matching and it is stored for a period of
824'timeout'.
825The syntax is:
826
willy tarreau532bb552006-05-13 18:40:37 +0200827 appsession <session_cookie> len <match_length> timeout <holdtime>
willy tarreau598da412005-12-18 01:07:29 +0100828
willy tarreau532bb552006-05-13 18:40:37 +0200829- <session_cookie> is the cookie, the server uses for it's session-handling
830- <match_length> how many bytes/characters should be used for matching equal
willy tarreau598da412005-12-18 01:07:29 +0100831 sessions
willy tarreau532bb552006-05-13 18:40:37 +0200832- <holdtime> after this inactivaty time, in ms, the cookie will be deleted
willy tarreau598da412005-12-18 01:07:29 +0100833 from the sessionstore
834
835The appsession is only per 'listen' section possible.
836
837Example :
838---------
willy tarreau532bb552006-05-13 18:40:37 +0200839 listen http_lb1 192.168.3.4:80
840 mode http
841 capture request header Cookie len 200
842 # Havind a ServerID cookie on the client allows him to reach
843 # the right server even after expiration of the appsession.
844 cookie ServerID insert nocache indirect
845 # Will memorize 52 bytes of the cookie 'JSESSIONID' and keep them
846 # for 3 hours. It will match it in the cookie and the URL field.
847 appsession JSESSIONID len 52 timeout 10800000
848 server first1 10.3.9.2:10805 check inter 3000 cookie first
849 server secon1 10.3.9.3:10805 check inter 3000 cookie secon
850 server first1 10.3.9.4:10805 check inter 3000 cookie first
851 server secon2 10.3.9.5:10805 check inter 3000 cookie secon
852 option httpchk GET /test.jsp
willy tarreau598da412005-12-18 01:07:29 +0100853
willy tarreauc5f73ed2005-12-18 01:26:38 +0100854
willy tarreaueedaa9f2005-12-17 14:08:03 +01008553) Autonomous load balancer
856===========================
857
858The proxy can perform the load-balancing itself, both in TCP and in HTTP modes.
859This is the most interesting mode which obsoletes the old 'dispatch' mode
860described above. It has advantages such as server health monitoring, multiple
861port binding and port mapping. To use this mode, the 'balance' keyword is used,
willy tarreau34f45302006-04-15 21:37:14 +0200862followed by the selected algorithm. Up to version 1.2.11, only 'roundrobin' was
863available, which is also the default value if unspecified. Starting with
Willy Tarreau2fcb5002007-05-08 13:35:26 +0200864version 1.2.12, a new 'source' keyword appeared. A new 'uri' keyword was added
865in version 1.3.10. In this mode, there will be no dispatch address, but the
866proxy needs at least one server.
willy tarreaueedaa9f2005-12-17 14:08:03 +0100867
868Example : same as the last one, with internal load balancer
869---------
870
871 listen http_proxy :80
willy tarreauc5f73ed2005-12-18 01:26:38 +0100872 mode http
873 cookie SERVERID
874 balance roundrobin
875 server web1 192.168.1.1:80 cookie server01
876 server web2 192.168.1.2:80 cookie server02
willy tarreaueedaa9f2005-12-17 14:08:03 +0100877
878
879Since version 1.1.22, it is possible to automatically determine on which port
880the server will get the connection, depending on the port the client connected
881to. Indeed, there now are 4 possible combinations for the server's <port> field:
882
883 - unspecified or '0' :
884 the connection will be sent to the same port as the one on which the proxy
885 received the client connection itself.
886
887 - numerical value (the only one supported in versions earlier than 1.1.22) :
888 the connection will always be sent to the specified port.
889
890 - '+' followed by a numerical value :
891 the connection will be sent to the same port as the one on which the proxy
892 received the connection, plus this value.
893
894 - '-' followed by a numerical value :
895 the connection will be sent to the same port as the one on which the proxy
896 received the connection, minus this value.
897
898Examples :
899----------
900
901# same as previous example
902
903 listen http_proxy :80
willy tarreauc5f73ed2005-12-18 01:26:38 +0100904 mode http
905 cookie SERVERID
906 balance roundrobin
907 server web1 192.168.1.1 cookie server01
908 server web2 192.168.1.2 cookie server02
willy tarreaueedaa9f2005-12-17 14:08:03 +0100909
910# simultaneous relaying of ports 80, 81 and 8080-8089
911
912 listen http_proxy :80,:81,:8080-8089
willy tarreauc5f73ed2005-12-18 01:26:38 +0100913 mode http
914 cookie SERVERID
915 balance roundrobin
916 server web1 192.168.1.1 cookie server01
917 server web2 192.168.1.2 cookie server02
willy tarreaueedaa9f2005-12-17 14:08:03 +0100918
919# relaying of TCP ports 25, 389 and 663 to ports 1025, 1389 and 1663
920
921 listen http_proxy :25,:389,:663
willy tarreauc5f73ed2005-12-18 01:26:38 +0100922 mode tcp
923 balance roundrobin
924 server srv1 192.168.1.1:+1000
925 server srv2 192.168.1.2:+1000
willy tarreaueedaa9f2005-12-17 14:08:03 +0100926
willy tarreau34f45302006-04-15 21:37:14 +0200927As previously stated, version 1.2.12 brought the 'source' keyword. When this
928keyword is used, the client's IP address is hashed and evenly distributed among
929the available servers so that a same source IP will always go to the same
930server as long as there are no change in the number of available servers. This
931can be used for instance to bind HTTP and HTTPS to the same server. It can also
932be used to improve stickyness when one part of the client population does not
933accept cookies. In this case, only those ones will be perturbated should a
934server fail.
935
936NOTE: It is important to consider the fact that many clients surf the net
937 through proxy farms which assign different IP addresses for each
938 request. Others use dialup connections with a different IP at each
939 connection. Thus, the 'source' parameter should be used with extreme
940 care.
941
942Examples :
943----------
944
945# make a same IP go to the same server whatever the service
946
947 listen http_proxy
948 bind :80,:443
949 mode http
950 balance source
951 server web1 192.168.1.1
952 server web2 192.168.1.2
953
954# try to improve client-server binding by using both source IP and cookie :
955
956 listen http_proxy :80
957 mode http
958 cookie SERVERID
959 balance source
960 server web1 192.168.1.1 cookie server01
961 server web2 192.168.1.2 cookie server02
962
Willy Tarreau2fcb5002007-05-08 13:35:26 +0200963As indicated above, the 'uri' keyword was introduced in version 1.3.10. It is
964useful when load-balancing between reverse proxy-caches, because it will hash
965the URI and use the hash result to select a server, thus optimizing the hit
966rate on the caches, because the same URI will always reach the same cache. This
967keyword is only allowed in HTTP mode.
968
969Example :
970---------
971
972# Always send a given URI to the same server
973
974 listen http_proxy
975 bind :3128
976 mode http
977 balance uri
978 server squid1 192.168.1.1
979 server squid2 192.168.1.2
980
willy tarreaueedaa9f2005-12-17 14:08:03 +0100981
willy tarreau197e8ec2005-12-17 14:10:59 +01009823.1) Server monitoring
983----------------------
willy tarreaueedaa9f2005-12-17 14:08:03 +0100984It is possible to check the servers status by trying to establish TCP
985connections or even sending HTTP requests to them. A server which fails to
986reply to health checks as expected will not be used by the load balancing
987algorithms. To enable monitoring, add the 'check' keyword on a server line.
988It is possible to specify the interval between tests (in milliseconds) with
989the 'inter' parameter, the number of failures supported before declaring that
990the server has fallen down with the 'fall' parameter, and the number of valid
991checks needed for the server to fully get up with the 'rise' parameter. Since
992version 1.1.22, it is also possible to send checks to a different port
993(mandatory when none is specified) with the 'port' parameter. The default
994values are the following ones :
995
996 - inter : 2000
997 - rise : 2
998 - fall : 3
999 - port : default server port
Willy Tarreau2ea3abb2007-03-25 16:45:16 +02001000 - addr : specific address for the test (default = address server)
1001
willy tarreaueedaa9f2005-12-17 14:08:03 +01001002The default mode consists in establishing TCP connections only. But in certain
1003types of application failures, it is often that the server continues to accept
1004connections because the system does it itself while the application is running
1005an endless loop, or is completely stuck. So in version 1.1.16 were introduced
1006HTTP health checks which only performed simple lightweight requests and analysed
1007the response. Now, as of version 1.1.23, it is possible to change the HTTP
1008method, the URI, and the HTTP version string (which even allows to send headers
1009with a dirty trick). To enable HTTP health-checks, use 'option httpchk'.
1010
1011By default, requests use the 'OPTIONS' method because it's very light and easy
1012to filter from logs, and does it on '/'. Only HTTP responses 2xx and 3xx are
1013considered valid ones, and only if they come before the time to send a new
1014request is reached ('inter' parameter). If some servers block this type of
1015request, 3 other forms help to forge a request :
1016
1017 - option httpchk -> OPTIONS / HTTP/1.0
1018 - option httpchk URI -> OPTIONS <URI> HTTP/1.0
1019 - option httpchk METH URI -> <METH> <URI> HTTP/1.0
1020 - option httpchk METH URI VER -> <METH> <URI> <VER>
1021
Willy Tarreauf3c69202006-07-09 16:42:34 +02001022Some people are using HAProxy to relay various TCP-based protocols such as
1023HTTPS, SMTP or LDAP, with the most common one being HTTPS. One problem commonly
1024encountered in data centers is the need to forward the traffic to far remote
1025servers while providing server fail-over. Often, TCP-only checks are not enough
1026because intermediate firewalls, load balancers or proxies might acknowledge the
1027connection before it reaches the real server. The only solution to this problem
1028is to send application-level health checks. Since the demand for HTTPS checks
1029is high, it has been implemented in 1.2.15 based on SSLv3 Client Hello packets.
1030To enable it, use 'option ssl-hello-chk'. It will send SSL CLIENT HELLO packets
1031to the servers, announcing support for most common cipher suites. If the server
1032responds what looks like a SERVER HELLO or an ALERT (refuses the ciphers) then
1033the response is considered as valid. Note that Apache does not generate a log
1034when it receives only an HELLO message, which makes this type of message
1035perfectly suit this need.
1036
Willy Tarreau23677902007-05-08 23:50:35 +02001037Version 1.3.10 introduced the SMTP health check. By default, it sends
1038"HELO localhost" to the servers, and waits for the 250 message. Note that it
1039can also send a specific request :
1040
1041 - option smtpchk -> sends "HELO localhost"
1042 - option smtpchk EHLO mail.mydomain.com -> sends this ESMTP greeting
1043
willy tarreaueedaa9f2005-12-17 14:08:03 +01001044See examples below.
1045
1046Since version 1.1.17, it is possible to specify backup servers. These servers
1047are only sollicited when no other server is available. This may only be useful
1048to serve a maintenance page, or define one active and one backup server (seldom
1049used in TCP mode). To make a server a backup one, simply add the 'backup' option
1050on its line. These servers also support cookies, so if a cookie is specified for
1051a backup server, clients assigned to this server will stick to it even when the
1052other ones come back. Conversely, if no cookie is assigned to such a server,
1053the clients will get their cookies removed (empty cookie = removal), and will
1054be balanced against other servers once they come back. Please note that there
Willy TARREAU3481c462006-03-01 22:37:57 +01001055is no load-balancing among backup servers by default. If there are several
1056backup servers, the second one will only be used when the first one dies, and
1057so on. To force load-balancing between backup servers, specify the 'allbackups'
1058option.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001059
Willy Tarreau2ea3abb2007-03-25 16:45:16 +02001060Since version 1.1.22, it is possible to send health checks to a different port
1061than the service. It is mainly needed in setups where the server does not have
1062any predefined port, for instance when the port is deduced from the listening
1063port. For this, use the 'port' parameter followed by the port number which must
1064respond to health checks. It is also possible to send health checks to a
1065different address than the service. It makes it easier to use a dedicated check
1066daemon on the servers, for instance, check return contents and stop several
1067farms at once in the event of an error anywhere.
1068
willy tarreaueedaa9f2005-12-17 14:08:03 +01001069Since version 1.1.17, it is also possible to visually check the status of all
1070servers at once. For this, you just have to send a SIGHUP signal to the proxy.
1071The servers status will be dumped into the logs at the 'notice' level, as well
1072as on <stderr> if not closed. For this reason, it's always a good idea to have
1073one local log server at the 'notice' level.
1074
willy tarreau982249e2005-12-18 00:57:06 +01001075Since version 1.1.28 and 1.2.1, if an instance loses all its servers, an
willy tarreau0174f312005-12-18 01:02:42 +01001076emergency message will be sent in the logs to inform the administator that an
willy tarreau982249e2005-12-18 00:57:06 +01001077immediate action must be taken.
1078
willy tarreau0174f312005-12-18 01:02:42 +01001079Since version 1.1.30 and 1.2.3, several servers can share the same cookie
1080value. This is particularly useful in backup mode, to select alternate paths
1081for a given server for example, to provide soft-stop, or to direct the clients
1082to a temporary page during an application restart. The principle is that when
1083a server is dead, the proxy will first look for another server which shares the
1084same cookie value for every client which presents the cookie. If there is no
1085standard server for this cookie, it will then look for a backup server which
1086shares the same name. Please consult the architecture guide for more information.
willy tarreau982249e2005-12-18 00:57:06 +01001087
willy tarreaueedaa9f2005-12-17 14:08:03 +01001088Examples :
1089----------
1090# same setup as in paragraph 3) with TCP monitoring
1091 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001092 mode http
1093 cookie SERVERID
1094 balance roundrobin
1095 server web1 192.168.1.1:80 cookie server01 check
1096 server web2 192.168.1.2:80 cookie server02 check inter 500 rise 1 fall 2
willy tarreaueedaa9f2005-12-17 14:08:03 +01001097
1098# same with HTTP monitoring via 'OPTIONS / HTTP/1.0'
1099 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001100 mode http
1101 cookie SERVERID
1102 balance roundrobin
1103 option httpchk
1104 server web1 192.168.1.1:80 cookie server01 check
1105 server web2 192.168.1.2:80 cookie server02 check inter 500 rise 1 fall 2
willy tarreaueedaa9f2005-12-17 14:08:03 +01001106
1107# same with HTTP monitoring via 'OPTIONS /index.html HTTP/1.0'
1108 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001109 mode http
1110 cookie SERVERID
1111 balance roundrobin
1112 option httpchk /index.html
1113 server web1 192.168.1.1:80 cookie server01 check
1114 server web2 192.168.1.2:80 cookie server02 check inter 500 rise 1 fall 2
willy tarreaueedaa9f2005-12-17 14:08:03 +01001115
1116# same with HTTP monitoring via 'HEAD /index.jsp? HTTP/1.1\r\nHost: www'
1117 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001118 mode http
1119 cookie SERVERID
1120 balance roundrobin
1121 option httpchk HEAD /index.jsp? HTTP/1.1\r\nHost:\ www
1122 server web1 192.168.1.1:80 cookie server01 check
1123 server web2 192.168.1.2:80 cookie server02 check inter 500 rise 1 fall 2
willy tarreaueedaa9f2005-12-17 14:08:03 +01001124
willy tarreau0174f312005-12-18 01:02:42 +01001125# Load-balancing with 'prefixed cookie' persistence, and soft-stop using an
1126# alternate port 81 on the server for health-checks.
1127 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001128 mode http
1129 cookie JSESSIONID prefix
1130 balance roundrobin
1131 option httpchk HEAD /index.jsp? HTTP/1.1\r\nHost:\ www
1132 server web1-norm 192.168.1.1:80 cookie s1 check port 81
1133 server web2-norm 192.168.1.2:80 cookie s2 check port 81
1134 server web1-stop 192.168.1.1:80 cookie s1 check port 80 backup
1135 server web2-stop 192.168.1.2:80 cookie s2 check port 80 backup
willy tarreau0174f312005-12-18 01:02:42 +01001136
willy tarreaueedaa9f2005-12-17 14:08:03 +01001137# automatic insertion of a cookie in the server's response, and automatic
1138# deletion of the cookie in the client request, while asking upstream caches
1139# not to cache replies.
1140 listen web_appl 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001141 mode http
1142 cookie SERVERID insert nocache indirect
1143 balance roundrobin
1144 server web1 192.168.1.1:80 cookie server01 check
1145 server web2 192.168.1.2:80 cookie server02 check
willy tarreaueedaa9f2005-12-17 14:08:03 +01001146
1147# same with off-site application backup and local error pages server
1148 listen web_appl 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001149 mode http
1150 cookie SERVERID insert nocache indirect
1151 balance roundrobin
1152 server web1 192.168.1.1:80 cookie server01 check
1153 server web2 192.168.1.2:80 cookie server02 check
1154 server web-backup 192.168.2.1:80 cookie server03 check backup
1155 server web-excuse 192.168.3.1:80 check backup
willy tarreaueedaa9f2005-12-17 14:08:03 +01001156
willy tarreauc5f73ed2005-12-18 01:26:38 +01001157# SMTP+TLS relaying with health-checks and backup servers
willy tarreaueedaa9f2005-12-17 14:08:03 +01001158
1159 listen http_proxy :25,:587
willy tarreauc5f73ed2005-12-18 01:26:38 +01001160 mode tcp
1161 balance roundrobin
1162 server srv1 192.168.1.1 check port 25 inter 30000 rise 1 fall 2
1163 server srv2 192.168.1.2 backup
willy tarreaueedaa9f2005-12-17 14:08:03 +01001164
Willy Tarreauf3c69202006-07-09 16:42:34 +02001165# HTTPS relaying with health-checks and backup servers
1166
1167 listen http_proxy :443
1168 mode tcp
1169 option ssl-hello-chk
1170 balance roundrobin
1171 server srv1 192.168.1.1 check inter 30000 rise 1 fall 2
1172 server srv2 192.168.1.2 backup
1173
Willy TARREAU3481c462006-03-01 22:37:57 +01001174# Load-balancing using a backup pool (requires haproxy 1.2.9)
1175 listen http_proxy 0.0.0.0:80
1176 mode http
1177 balance roundrobin
1178 option httpchk
1179 server inst1 192.168.1.1:80 cookie s1 check
1180 server inst2 192.168.1.2:80 cookie s2 check
1181 server inst3 192.168.1.3:80 cookie s3 check
1182 server back1 192.168.1.10:80 check backup
1183 server back2 192.168.1.11:80 check backup
1184 option allbackups # all backups will be used
1185
willy tarreaueedaa9f2005-12-17 14:08:03 +01001186
11873.2) Redistribute connections in case of failure
1188------------------------------------------------
1189In HTTP mode, if a server designated by a cookie does not respond, the clients
1190may definitely stick to it because they cannot flush the cookie, so they will
1191not be able to access the service anymore. Specifying 'redispatch' will allow
1192the proxy to break their persistence and redistribute them to working servers.
1193
1194Example :
1195---------
1196 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001197 mode http
1198 cookie SERVERID
1199 dispatch 192.168.1.100:80
1200 server web1 192.168.1.1:80 cookie server01
1201 server web2 192.168.1.2:80 cookie server02
1202 redispatch # send back to dispatch in case of connection failure
willy tarreaueedaa9f2005-12-17 14:08:03 +01001203
1204Up to, and including version 1.1.16, this parameter only applied to connection
1205failures. Since version 1.1.17, it also applies to servers which have been
1206detected as failed by the health check mechanism. Indeed, a server may be broken
1207but still accepting connections, which would not solve every case. But it is
1208possible to conserve the old behaviour, that is, make a client insist on trying
1209to connect to a server even if it is said to be down, by setting the 'persist'
1210option :
1211
1212 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001213 mode http
1214 option persist
1215 cookie SERVERID
1216 dispatch 192.168.1.100:80
1217 server web1 192.168.1.1:80 cookie server01
1218 server web2 192.168.1.2:80 cookie server02
1219 redispatch # send back to dispatch in case of connection failure
willy tarreaueedaa9f2005-12-17 14:08:03 +01001220
1221
willy tarreau34f45302006-04-15 21:37:14 +020012223.3) Assigning different weights to servers
1223-------------------------------------------
1224Sometimes you will need to bring new servers to increase your server farm's
1225capacity, but the new server will be either smaller (emergency use of anything
1226that fits) or bigger (when investing in new hardware). For this reason, it
1227might be wise to be able to send more clients to biggest servers. Till version
12281.2.11, it was necessary to replicate the same server multiple times in the
1229configuration. Starting with 1.2.12, the 'weight' option is available. HAProxy
1230then computes the most homogenous possible map of servers based on their
willy tarreau532bb552006-05-13 18:40:37 +02001231weights so that the load gets distributed as smoothly as possible among them.
1232The weight, between 1 and 256, should reflect one server's capacity relative to
1233others. Weight 1 represents the lowest frequency and 256 the highest. This way,
1234if a server fails, the remaining capacities are still respected.
willy tarreau34f45302006-04-15 21:37:14 +02001235
1236Example :
1237---------
1238# fair distribution among two opterons and one old pentium3
1239
1240 listen web_appl 0.0.0.0:80
1241 mode http
1242 cookie SERVERID insert nocache indirect
1243 balance roundrobin
1244 server pentium3-800 192.168.1.1:80 cookie server01 weight 8 check
1245 server opteron-2.0G 192.168.1.2:80 cookie server02 weight 20 check
1246 server opteron-2.4G 192.168.1.3:80 cookie server03 weight 24 check
1247 server web-backup1 192.168.2.1:80 cookie server04 check backup
1248 server web-excuse 192.168.3.1:80 check backup
1249
1250Notes :
1251-------
1252 - if unspecified, the default weight is 1
1253
1254 - the weight does not impact health checks, so it is cleaner to use weights
1255 than replicating the same server several times
1256
1257 - weights also work on backup servers if the 'allbackups' option is used
1258
1259 - the weights also apply to the source address load balancing
1260 ('balance source').
1261
1262 - whatever the weights, the first server will always be assigned first. This
1263 is helpful for troubleshooting.
1264
1265 - for the purists, the map calculation algorithm gives precedence to first
1266 server, so the map is the most uniform when servers are declared in
1267 ascending order relative to their weights.
1268
willy tarreau532bb552006-05-13 18:40:37 +02001269The load distribution will follow exactly this sequence :
1270
1271 Request| 1 1 1 1
1272 number | 1 2 3 4 5 6 7 8 9 0 1 2 3
1273 --------+---------------------------
1274 p3-800 | X . . . . . . X . . . . .
1275 opt-20 | . X . X . X . . . X . X .
1276 opt-24 | . . X . X . X . X . X . X
1277
1278
12793.4) Limiting the number of concurrent sessions on each server
1280--------------------------------------------------------------
1281Some pre-forked servers such as Apache suffer from too many concurrent
1282sessions, because it's very expensive to run hundreds or thousands of
1283processes on one system. One solution is to increase the number of servers
1284and load-balance between them, but it is a problem when the only goal is
1285to resist to short surges.
1286
1287To solve this problem, a new feature was implemented in HAProxy 1.2.13.
1288It's a per-server 'maxconn', associated with a per-server and a per-proxy
1289queue. This transforms haproxy into a request buffer between the thousands of
1290clients and the few servers. On many circumstances, lowering the maxconn value
1291will increase the server's performance and decrease the overall response times
1292because the servers will be less congested.
1293
1294When a request tries to reach any server, the first non-saturated server is
1295used, respective to the load balancing algorithm. If all servers are saturated,
1296then the request gets queued into the instance's global queue. It will be
1297dequeued once a server will have freed a session and all previously queued
1298requests have been processed.
1299
1300If a request references a particular server (eg: source hashing, or persistence
1301cookie), and if this server is full, then the request will be queued into the
1302server's dedicated queue. This queue has higher priority than the global queue,
1303so it's easier for already registered users to enter the site than for new
1304users.
1305
1306For this, the logs have been enhanced to show the number of sessions per
1307server, the request's position in the queue and the time spent in the queue.
1308This helps doing capacity planning. See the 'logs' section below for more info.
1309
1310Example :
1311---------
1312 # be nice with P3 which only has 256 MB of RAM.
1313 listen web_appl 0.0.0.0:80
1314 maxconn 10000
1315 mode http
1316 cookie SERVERID insert nocache indirect
1317 balance roundrobin
1318 server pentium3-800 192.168.1.1:80 cookie s1 weight 8 maxconn 100 check
1319 server opteron-2.0G 192.168.1.2:80 cookie s2 weight 20 maxconn 300 check
1320 server opteron-2.4G 192.168.1.3:80 cookie s3 weight 24 maxconn 300 check
1321 server web-backup1 192.168.2.1:80 cookie s4 check maxconn 200 backup
1322 server web-excuse 192.168.3.1:80 check backup
1323
willy tarreauf76e6ca2006-05-21 21:09:55 +02001324
1325This was so much efficient at reducing the server's response time that some
1326users wanted to use low values to improve their server's performance. However,
1327they were not able anymore to handle very large loads because it was not
1328possible anymore to saturate the servers. For this reason, version 1.2.14 has
1329brought dynamic limitation with the addition of the parameter 'minconn'. When
1330this parameter is set along with maxconn, it will enable dynamic limitation
1331based on the instance's load. The maximum number of concurrent sessions on a
1332server will be proportionnal to the number of sessions on the instance relative
1333to its maxconn. A minimum of <minconn> will be allowed whatever the load. This
1334will ensure that servers will perform at their best level under normal loads,
1335while still handling surges when needed. The dynamic limit is computed like
1336this :
1337
1338 srv.dyn_limit = max(srv.minconn, srv.maxconn * inst.sess / inst.maxconn)
1339
1340Example :
1341---------
1342 # be nice with P3 which only has 256 MB of RAM.
1343 listen web_appl 0.0.0.0:80
1344 maxconn 10000
1345 mode http
1346 cookie SERVERID insert nocache indirect
1347 balance roundrobin
1348 server pentium3-800 192.168.1.1:80 cookie s1 weight 8 minconn 10 maxconn 100 check
1349 server opteron-2.0G 192.168.1.2:80 cookie s2 weight 20 minconn 30 maxconn 300 check
1350 server opteron-2.4G 192.168.1.3:80 cookie s3 weight 24 minconn 30 maxconn 300 check
1351 server web-backup1 192.168.2.1:80 cookie s4 check maxconn 200 backup
1352 server web-excuse 192.168.3.1:80 check backup
1353
1354In the example above, the server 'pentium3-800' will receive at most 100
1355simultaneous sessions when the proxy instance will reach 10000 sessions, and
1356will receive only 10 simultaneous sessions when the proxy will be under 1000
1357sessions.
1358
willy tarreau532bb552006-05-13 18:40:37 +02001359Notes :
1360-------
1361 - The requests will not stay indefinitely in the queue, they follow the
1362 'contimeout' parameter, and if a request cannot be dequeued within this
1363 timeout because the server is saturated or because the queue is filled,
1364 the session will expire with a 503 error.
1365
willy tarreauf76e6ca2006-05-21 21:09:55 +02001366 - if only <minconn> is specified, it has the same effect as <maxconn>
1367
willy tarreau532bb552006-05-13 18:40:37 +02001368 - setting too low values for maxconn might improve performance but might also
1369 allow slow users to block access to the server for other users.
1370
willy tarreau34f45302006-04-15 21:37:14 +02001371
willy tarreaue0bdd622006-05-21 20:51:54 +020013723.5) Dropping aborted requests
1373------------------------------
1374In presence of very high loads, the servers will take some time to respond. The
1375per-proxy's connection queue will inflate, and the response time will increase
1376respective to the size of the queue times the average per-session response
1377time. When clients will wait for more than a few seconds, they will often hit
1378the 'STOP' button on their browser, leaving a useless request in the queue, and
1379slowing down other users.
1380
1381As there is no way to distinguish between a full STOP and a simple
1382shutdown(SHUT_WR) on the client side, HTTP agents should be conservative and
1383consider that the client might only have closed its output channel while
1384waiting for the response. However, this introduces risks of congestion when
1385lots of users do the same, and is completely useless nowadays because probably
1386no client at all will close the session while waiting for the response. Some
1387HTTP agents support this (Squid, Apache, HAProxy), and others do not (TUX, most
1388hardware-based load balancers). So the probability for a closed input channel
1389to represent a user hitting the 'STOP' button is close to 100%, and it is very
1390tempting to be able to abort the session early without polluting the servers.
1391
1392For this reason, a new option "abortonclose" was introduced in version 1.2.14.
1393By default (without the option) the behaviour is HTTP-compliant. But when the
1394option is specified, a session with an incoming channel closed will be aborted
1395if it's still possible, which means that it's either waiting for a connect() to
1396establish or it is queued waiting for a connection slot. This considerably
1397reduces the queue size and the load on saturated servers when users are tempted
1398to click on STOP, which in turn reduces the response time for other users.
1399
1400Example :
1401---------
1402 listen web_appl 0.0.0.0:80
1403 maxconn 10000
1404 mode http
1405 cookie SERVERID insert nocache indirect
1406 balance roundrobin
1407 server web1 192.168.1.1:80 cookie s1 weight 10 maxconn 100 check
1408 server web2 192.168.1.2:80 cookie s2 weight 10 maxconn 100 check
1409 server web3 192.168.1.3:80 cookie s3 weight 10 maxconn 100 check
1410 server bck1 192.168.2.1:80 cookie s4 check maxconn 200 backup
1411 option abortonclose
1412
1413
willy tarreaueedaa9f2005-12-17 14:08:03 +010014144) Additionnal features
1415=======================
1416
willy tarreau481132e2006-05-21 21:43:10 +02001417Other features are available. They are transparent mode, event logging, header
1418rewriting/filtering, and the status as an HTML page.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001419
willy tarreauc5f73ed2005-12-18 01:26:38 +01001420
willy tarreau0174f312005-12-18 01:02:42 +010014214.1) Network features
willy tarreaueedaa9f2005-12-17 14:08:03 +01001422---------------------
willy tarreau0174f312005-12-18 01:02:42 +010014234.1.1) Transparent mode
1424-----------------------
willy tarreaueedaa9f2005-12-17 14:08:03 +01001425In HTTP mode, the 'transparent' keyword allows to intercept sessions which are
1426routed through the system hosting the proxy. This mode was implemented as a
1427replacement for the 'dispatch' mode, since connections without cookie will be
1428sent to the original address while known cookies will be sent to the servers.
1429This mode implies that the system can redirect sessions to a local port.
1430
1431Example :
1432---------
1433 listen http_proxy 0.0.0.0:65000
willy tarreauc5f73ed2005-12-18 01:26:38 +01001434 mode http
1435 transparent
1436 cookie SERVERID
1437 server server01 192.168.1.1:80
1438 server server02 192.168.1.2:80
willy tarreaueedaa9f2005-12-17 14:08:03 +01001439
1440 # iptables -t nat -A PREROUTING -i eth0 -p tcp -d 192.168.1.100 \
1441 --dport 80 -j REDIRECT --to-ports 65000
1442
1443Note :
1444------
1445If the port is left unspecified on the server, the port the client connected to
1446will be used. This allows to relay a full port range without using transparent
1447mode nor thousands of file descriptors, provided that the system can redirect
1448sessions to local ports.
1449
1450Example :
1451---------
1452 # redirect all ports to local port 65000, then forward to the server on the
1453 # original port.
1454 listen http_proxy 0.0.0.0:65000
willy tarreauc5f73ed2005-12-18 01:26:38 +01001455 mode tcp
1456 server server01 192.168.1.1 check port 60000
1457 server server02 192.168.1.2 check port 60000
willy tarreaueedaa9f2005-12-17 14:08:03 +01001458
1459 # iptables -t nat -A PREROUTING -i eth0 -p tcp -d 192.168.1.100 \
1460 -j REDIRECT --to-ports 65000
1461
willy tarreau0174f312005-12-18 01:02:42 +010014624.1.2) Per-server source address binding
1463----------------------------------------
1464As of versions 1.1.30 and 1.2.3, it is possible to specify a particular source
1465to reach each server. This is useful when reaching backup servers from a
1466different LAN, or to use an alternate path to reach the same server. It is also
1467usable to provide source load-balancing for outgoing connections. Obviously,
1468the same source address is used to send health-checks.
1469
1470Example :
1471---------
1472 # use a particular source to reach both servers
1473 listen http_proxy 0.0.0.0:65000
willy tarreauc5f73ed2005-12-18 01:26:38 +01001474 mode http
1475 balance roundrobin
1476 server server01 192.168.1.1:80 source 192.168.2.13
1477 server server02 192.168.1.2:80 source 192.168.2.13
willy tarreau0174f312005-12-18 01:02:42 +01001478
1479Example :
1480---------
1481 # use a particular source to reach each servers
1482 listen http_proxy 0.0.0.0:65000
willy tarreauc5f73ed2005-12-18 01:26:38 +01001483 mode http
1484 balance roundrobin
1485 server server01 192.168.1.1:80 source 192.168.1.1
1486 server server02 192.168.2.1:80 source 192.168.2.1
willy tarreau0174f312005-12-18 01:02:42 +01001487
1488Example :
1489---------
1490 # provide source load-balancing to reach the same proxy through 2 WAN links
1491 listen http_proxy 0.0.0.0:65000
willy tarreauc5f73ed2005-12-18 01:26:38 +01001492 mode http
1493 balance roundrobin
1494 server remote-proxy-way1 192.168.1.1:3128 source 192.168.2.1
1495 server remote-proxy-way2 192.168.1.1:3128 source 192.168.3.1
willy tarreau0174f312005-12-18 01:02:42 +01001496
1497Example :
1498---------
1499 # force a TCP connection to bind to a specific port
1500 listen http_proxy 0.0.0.0:2000
willy tarreauc5f73ed2005-12-18 01:26:38 +01001501 mode tcp
1502 balance roundrobin
1503 server srv1 192.168.1.1:80 source 192.168.2.1:20
1504 server srv2 192.168.1.2:80 source 192.168.2.1:20
willy tarreau0174f312005-12-18 01:02:42 +01001505
willy tarreaub952e1d2005-12-18 01:31:20 +010015064.1.3) TCP keep-alive
1507---------------------
1508With version 1.2.7, it becomes possible to enable TCP keep-alives on both the
1509client and server sides. This makes it possible to prevent long sessions from
1510expiring on external layer 4 components such as firewalls and load-balancers.
1511It also allows the system to terminate dead sessions when no timeout has been
1512set (not recommanded). The proxy cannot set the keep-alive probes intervals nor
1513maximal count, consult your operating system manual for this. There are 3
1514options to enable TCP keep-alive :
1515
1516 option tcpka # enables keep-alive both on client and server side
1517 option clitcpka # enables keep-alive only on client side
1518 option srvtcpka # enables keep-alive only on server side
1519
willy tarreaueedaa9f2005-12-17 14:08:03 +01001520
15214.2) Event logging
1522------------------
willy tarreauc5f73ed2005-12-18 01:26:38 +01001523
1524HAProxy's strength certainly lies in its precise logs. It probably provides the
1525finest level of information available for such a product, which is very
1526important for troubleshooting complex environments. Standard log information
1527include client ports, TCP/HTTP state timers, precise session state at
1528termination and precise termination cause, information about decisions to
1529direct trafic to a server, and of course the ability to capture arbitrary
1530headers.
1531
1532In order to improve administrators reactivity, it offers a great transparency
1533about encountered problems, both internal and external, and it is possible to
1534send logs to different sources at the same time with different level filters :
1535
1536 - global process-level logs (system errors, start/stop, etc..)
1537 - per-listener system and internal errors (lack of resource, bugs, ...)
1538 - per-listener external troubles (servers up/down, max connections)
1539 - per-listener activity (client connections), either at the establishment or
1540 at the termination.
1541
1542The ability to distribute different levels of logs to different log servers
1543allow several production teams to interact and to fix their problems as soon
1544as possible. For example, the system team might monitor system-wide errors,
1545while the application team might be monitoring the up/down for their servers in
1546real time, and the security team might analyze the activity logs with one hour
1547delay.
1548
willy tarreauc1cae632005-12-17 14:12:23 +010015494.2.1) Log levels
1550-----------------
willy tarreau197e8ec2005-12-17 14:10:59 +01001551TCP and HTTP connections can be logged with informations such as date, time,
1552source IP address, destination address, connection duration, response times,
1553HTTP request, the HTTP return code, number of bytes transmitted, the conditions
1554in which the session ended, and even exchanged cookies values, to track a
1555particular user's problems for example. All messages are sent to up to two
1556syslog servers. Consult section 1.1 for more info about log facilities. The
1557syntax follows :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001558
willy tarreau197e8ec2005-12-17 14:10:59 +01001559 log <address_1> <facility_1> [max_level_1]
1560 log <address_2> <facility_2> [max_level_2]
1561or
willy tarreaueedaa9f2005-12-17 14:08:03 +01001562 log global
1563
willy tarreau197e8ec2005-12-17 14:10:59 +01001564Note :
1565------
1566The particular syntax 'log global' means that the same log configuration as the
1567'global' section will be used.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001568
willy tarreau197e8ec2005-12-17 14:10:59 +01001569Example :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001570---------
1571 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001572 mode http
1573 log 192.168.2.200 local3
1574 log 192.168.2.201 local4
willy tarreaueedaa9f2005-12-17 14:08:03 +01001575
willy tarreauc1cae632005-12-17 14:12:23 +010015764.2.2) Log format
1577-----------------
1578By default, connections are logged at the TCP level, as soon as the session
1579establishes between the client and the proxy. By enabling the 'tcplog' option,
1580the proxy will wait until the session ends to generate an enhanced log
1581containing more information such as session duration and its state during the
willy tarreau532bb552006-05-13 18:40:37 +02001582disconnection. The number of remaining session after disconnection is also
1583indicated (for the server, the listener, and the process).
willy tarreauc1cae632005-12-17 14:12:23 +01001584
willy tarreauc5f73ed2005-12-18 01:26:38 +01001585Example of TCP logging :
1586------------------------
willy tarreau982249e2005-12-18 00:57:06 +01001587 listen relais-tcp 0.0.0.0:8000
willy tarreauc5f73ed2005-12-18 01:26:38 +01001588 mode tcp
1589 option tcplog
1590 log 192.168.2.200 local3
willy tarreau982249e2005-12-18 00:57:06 +01001591
willy tarreau532bb552006-05-13 18:40:37 +02001592>>> haproxy[18989]: 127.0.0.1:34550 [15/Oct/2003:15:24:28] relais-tcp Srv1 0/0/5007 0 -- 1/1/1 0/0
willy tarreauc5f73ed2005-12-18 01:26:38 +01001593
willy tarreau532bb552006-05-13 18:40:37 +02001594 Field Format Example
willy tarreauc5f73ed2005-12-18 01:26:38 +01001595
willy tarreau532bb552006-05-13 18:40:37 +02001596 1 process_name '[' pid ']:' haproxy[18989]:
1597 2 client_ip ':' client_port 127.0.0.1:34550
1598 3 '[' date ']' [15/Oct/2003:15:24:28]
1599 4 listener_name relais-tcp
1600 5 server_name Srv1
1601 6 queue_time '/' connect_time '/' total_time 0/0/5007
1602 7 bytes_read 0
1603 8 termination_state --
1604 9 srv_conn '/' listener_conn '/' process_conn 1/1/1
1605 10 position in srv_queue / listener_queue 0/0
1606
willy tarreau982249e2005-12-18 00:57:06 +01001607
willy tarreauc1cae632005-12-17 14:12:23 +01001608Another option, 'httplog', provides more detailed information about HTTP
1609contents, such as the request and some cookies. In the event where an external
1610component would establish frequent connections to check the service, logs may be
1611full of useless lines. So it is possible not to log any session which didn't
1612transfer any data, by the setting of the 'dontlognull' option. This only has
1613effect on sessions which are established then closed.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001614
willy tarreauc5f73ed2005-12-18 01:26:38 +01001615Example of HTTP logging :
1616-------------------------
willy tarreaueedaa9f2005-12-17 14:08:03 +01001617 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001618 mode http
1619 option httplog
1620 option dontlognull
1621 log 192.168.2.200 local3
1622
willy tarreau532bb552006-05-13 18:40:37 +02001623>>> haproxy[674]: 127.0.0.1:33319 [15/Oct/2003:08:31:57] relais-http Srv1 9/0/7/147/723 200 243 - - ---- 2/3/3 0/0 "HEAD / HTTP/1.0"
willy tarreaueedaa9f2005-12-17 14:08:03 +01001624
willy tarreauc5f73ed2005-12-18 01:26:38 +01001625More complete example
willy tarreau532bb552006-05-13 18:40:37 +02001626 haproxy[18989]: 10.0.0.1:34552 [15/Oct/2003:15:26:31] relais-http Srv1 3183/-1/-1/-1/11215 503 0 - - SC-- 137/202/205 0/0 {w.ods.org|Mozilla} {} "HEAD / HTTP/1.0"
willy tarreauc5f73ed2005-12-18 01:26:38 +01001627
willy tarreau532bb552006-05-13 18:40:37 +02001628 Field Format Example
willy tarreauc5f73ed2005-12-18 01:26:38 +01001629
willy tarreau532bb552006-05-13 18:40:37 +02001630 1 process_name '[' pid ']:' haproxy[18989]:
1631 2 client_ip ':' client_port 10.0.0.1:34552
1632 3 '[' date ']' [15/Oct/2003:15:26:31]
1633 4 listener_name relais-http
1634 5 server_name Srv1
1635 6 Tq '/' Tw '/' Tc '/' Tr '/' Tt 3183/-1/-1/-1/11215
1636 7 HTTP_return_code 503
1637 8 bytes_read 0
1638 9 captured_request_cookie -
1639 10 captured_response_cookie -
1640 11 termination_state SC--
1641 12 srv_conn '/' listener_conn '/' process_conn 137/202/205
1642 13 position in srv_queue / listener_queue 0/0
1643 14 '{' captured_request_headers '}' {w.ods.org|Mozilla}
1644 15 '{' captured_response_headers '}' {}
1645 16 '"' HTTP_request '"' "HEAD / HTTP/1.0"
willy tarreauc5f73ed2005-12-18 01:26:38 +01001646
1647Note for log parsers: the URI is ALWAYS the end of the line starting with the
1648 first double quote '"'.
willy tarreau982249e2005-12-18 00:57:06 +01001649
1650The problem when logging at end of connection is that you have no clue about
1651what is happening during very long sessions. To workaround this problem, a
1652new option 'logasap' has been introduced in 1.1.28/1.2.1. When specified, the
1653proxy will log as soon as possible, just before data transfer begins. This means
1654that in case of TCP, it will still log the connection status to the server, and
1655in case of HTTP, it will log just after processing the server headers. In this
1656case, the number of bytes reported is the number of header bytes sent to the
1657client.
1658
1659In order to avoid confusion with normal logs, the total time field and the
1660number of bytes are prefixed with a '+' sign which mean that real numbers are
1661certainly bigger.
1662
1663Example :
1664---------
1665
1666 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001667 mode http
1668 option httplog
1669 option dontlognull
1670 option logasap
1671 log 192.168.2.200 local3
willy tarreau982249e2005-12-18 00:57:06 +01001672
willy tarreau532bb552006-05-13 18:40:37 +02001673>>> haproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17] relais-http Srv1 9/10/7/14/+30 200 +243 - - ---- 1/1/3 1/0 "GET /image.iso HTTP/1.0"
willy tarreau982249e2005-12-18 00:57:06 +01001674
willy tarreauc1cae632005-12-17 14:12:23 +010016754.2.3) Timing events
1676--------------------
1677Timers provide a great help in trouble shooting network problems. All values
1678are reported in milliseconds (ms). In HTTP mode, four control points are
willy tarreau532bb552006-05-13 18:40:37 +02001679reported under the form 'Tq/Tw/Tc/Tr/Tt' :
willy tarreauc1cae632005-12-17 14:12:23 +01001680
1681 - Tq: total time to get the client request.
1682 It's the time elapsed between the moment the client connection was accepted
1683 and the moment the proxy received the last HTTP header. The value '-1'
1684 indicates that the end of headers (empty line) has never been seen.
1685
willy tarreau532bb552006-05-13 18:40:37 +02001686 - Tw: total time spent in the queues waiting for a connection slot. It
1687 accounts for listener's queue as well as the server's queue, and depends
1688 on the queue size, and the time needed for the server to complete previous
1689 sessions. The value '-1' means that the request was killed before reaching
1690 the queue.
1691
willy tarreauc1cae632005-12-17 14:12:23 +01001692 - Tc: total time to establish the TCP connection to the server.
1693 It's the time elapsed between the moment the proxy sent the connection
1694 request, and the moment it was acknowledged, or between the TCP SYN packet
1695 and the matching SYN/ACK in return. The value '-1' means that the
1696 connection never established.
1697
1698 - Tr: server response time. It's the time elapsed between the moment the
1699 TCP connection was established to the server and the moment it send its
1700 complete response header. It purely shows its request processing time,
1701 without the network overhead due to the data transmission. The value '-1'
1702 means that the last the response header (empty line) was never seen.
1703
1704 - Tt: total session duration time, between the moment the proxy accepted it
willy tarreau982249e2005-12-18 00:57:06 +01001705 and the moment both ends were closed. The exception is when the 'logasap'
willy tarreau532bb552006-05-13 18:40:37 +02001706 option is specified. In this case, it only equals (Tq+Tw+Tc+Tr), and is
willy tarreau982249e2005-12-18 00:57:06 +01001707 prefixed with a '+' sign. From this field, we can deduce Td, the data
1708 transmission time, by substracting other timers when valid :
willy tarreauc1cae632005-12-17 14:12:23 +01001709
willy tarreau532bb552006-05-13 18:40:37 +02001710 Td = Tt - (Tq + Tw + Tc + Tr)
willy tarreauc1cae632005-12-17 14:12:23 +01001711
1712 Timers with '-1' values have to be excluded from this equation.
1713
willy tarreau532bb552006-05-13 18:40:37 +02001714In TCP mode ('option tcplog'), only Tw, Tc and Tt are reported.
willy tarreauc1cae632005-12-17 14:12:23 +01001715
1716These timers provide precious indications on trouble causes. Since the TCP
1717protocol defines retransmit delays of 3, 6, 12... seconds, we know for sure
1718that timers close to multiples of 3s are nearly always related to packets lost
1719due to network problems (wires or negociation). Moreover, if <Tt> is close to
1720a timeout value specified in the configuration, it often means that a session
1721has been aborted on time-out.
1722
1723Most common cases :
1724
1725 - If Tq is close to 3000, a packet has probably been lost between the client
1726 and the proxy.
1727 - If Tc is close to 3000, a packet has probably been lost between the server
1728 and the proxy during the server connection phase. This one should always be
1729 very low (less than a few tens).
1730 - If Tr is nearly always lower than 3000 except some rare values which seem to
1731 be the average majored by 3000, there are probably some packets lost between
1732 the proxy and the server.
1733 - If Tt is often slightly higher than a time-out, it's often because the
1734 client and the server use HTTP keep-alive and the session is maintained
1735 after the response ends. Se further for how to disable HTTP keep-alive.
1736
1737Other cases ('xx' means any value to be ignored) :
willy tarreau532bb552006-05-13 18:40:37 +02001738 -1/xx/xx/xx/Tt: the client was not able to send its complete request in time,
1739 or that it aborted it too early.
1740 Tq/-1/xx/xx/Tt: it was not possible to process the request, maybe because
1741 servers were out of order.
1742 Tq/Tw/-1/xx/Tt: the connection could not establish on the server. Either it
1743 refused it or it timed out after Tt-(Tq+Tw) ms.
1744 Tq/Tw/Tc/-1/Tt: the server has accepted the connection but did not return a
1745 complete response in time, or it closed its connexion
1746 unexpectedly, after Tt-(Tq+Tw+Tc) ms.
willy tarreauc1cae632005-12-17 14:12:23 +01001747
17484.2.4) Session state at disconnection
1749-------------------------------------
willy tarreauc5f73ed2005-12-18 01:26:38 +01001750TCP and HTTP logs provide a session completion indicator in the
1751<termination_state> field, just before the number of active
1752connections. It is 2-characters long in TCP, and 4-characters long in
1753HTTP, each of which has a special meaning :
1754
willy tarreau197e8ec2005-12-17 14:10:59 +01001755 - On the first character, a code reporting the first event which caused the
1756 session to terminate :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001757
willy tarreauc5f73ed2005-12-18 01:26:38 +01001758 C : the TCP session was unexpectedly aborted by the client.
1759
1760 S : the TCP session was unexpectedly aborted by the server, or the
1761 server explicitly refused it.
1762
1763 P : the session was prematurely aborted by the proxy, because of a
1764 connection limit enforcement, because a DENY filter was matched,
1765 or because of a security check which detected and blocked a
1766 dangerous error in server response which might have caused
1767 information leak (eg: cacheable cookie).
1768
1769 R : a resource on the proxy has been exhausted (memory, sockets, source
1770 ports, ...). Usually, this appears during the connection phase, and
1771 system logs should contain a copy of the precise error.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001772
willy tarreauc5f73ed2005-12-18 01:26:38 +01001773 I : an internal error was identified by the proxy during a self-check.
1774 This should NEVER happen, and you are encouraged to report any log
1775 containing this, because this is a bug.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001776
willy tarreauc5f73ed2005-12-18 01:26:38 +01001777 c : the client-side time-out expired first.
1778
1779 s : the server-side time-out expired first.
1780
1781 - : normal session completion.
1782
1783 - on the second character, the TCP/HTTP session state when it was closed :
1784
1785 R : waiting for complete REQUEST from the client (HTTP only). Nothing
1786 was sent to any server.
1787
willy tarreau532bb552006-05-13 18:40:37 +02001788 Q : waiting in the QUEUE for a connection slot. This can only happen on
1789 servers which have a 'maxconn' parameter set. No connection attempt
1790 was made to any server.
1791
willy tarreauc5f73ed2005-12-18 01:26:38 +01001792 C : waiting for CONNECTION to establish on the server. The server might
1793 at most have noticed a connection attempt.
1794
1795 H : waiting for, receiving and processing server HEADERS (HTTP only).
1796
1797 D : the session was in the DATA phase.
1798
1799 L : the proxy was still transmitting LAST data to the client while the
1800 server had already finished.
1801
Willy Tarreau2272dc12006-09-03 10:19:38 +02001802 T : the request was tarpitted. It has been held open on with the client
Willy Tarreau08fa2e32006-09-03 10:47:37 +02001803 during the whole contimeout duration or untill the client closed.
Willy Tarreau2272dc12006-09-03 10:19:38 +02001804
willy tarreauc5f73ed2005-12-18 01:26:38 +01001805 - : normal session completion after end of data transfer.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001806
willy tarreau197e8ec2005-12-17 14:10:59 +01001807 - the third character tells whether the persistence cookie was provided by
willy tarreauc1cae632005-12-17 14:12:23 +01001808 the client (only in HTTP mode) :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001809
willy tarreauc5f73ed2005-12-18 01:26:38 +01001810 N : the client provided NO cookie. This is usually the case on new
1811 connections.
1812
1813 I : the client provided an INVALID cookie matching no known
1814 server. This might be caused by a recent configuration change,
1815 mixed cookies between HTTP/HTTPS sites, or an attack.
1816
1817 D : the client provided a cookie designating a server which was DOWN,
1818 so either the 'persist' option was used and the client was sent to
1819 this server, or it was not set and the client was redispatched to
1820 another server.
1821
1822 V : the client provided a valid cookie, and was sent to the associated
1823 server.
1824
1825 - : does not apply (no cookie set in configuration).
willy tarreaueedaa9f2005-12-17 14:08:03 +01001826
willy tarreau197e8ec2005-12-17 14:10:59 +01001827 - the last character reports what operations were performed on the persistence
willy tarreauc1cae632005-12-17 14:12:23 +01001828 cookie returned by the server (only in HTTP mode) :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001829
willy tarreauc5f73ed2005-12-18 01:26:38 +01001830 N : NO cookie was provided by the server, and none was inserted either.
1831
1832 I : no cookie was provided by the server, and the proxy INSERTED one.
1833
willy tarreau197e8ec2005-12-17 14:10:59 +01001834 P : a cookie was PROVIDED by the server and transmitted as-is.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001835
willy tarreauc5f73ed2005-12-18 01:26:38 +01001836 R : the cookie provided by the server was REWRITTEN by the proxy.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001837
willy tarreauc5f73ed2005-12-18 01:26:38 +01001838 D : the cookie provided by the server was DELETED by the proxy.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001839
willy tarreauc5f73ed2005-12-18 01:26:38 +01001840 - : does not apply (no cookie set in configuration).
willy tarreaueedaa9f2005-12-17 14:08:03 +01001841
willy tarreauc5f73ed2005-12-18 01:26:38 +01001842The combination of the two first flags give a lot of information about what was
1843happening when the session terminated. It can be helpful to detect server
1844saturation, network troubles, local system resource starvation, attacks, etc...
willy tarreaueedaa9f2005-12-17 14:08:03 +01001845
willy tarreauc5f73ed2005-12-18 01:26:38 +01001846The most common termination flags combinations are indicated here.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001847
willy tarreauc5f73ed2005-12-18 01:26:38 +01001848 Flags Reason
1849 CR The client aborted before sending a full request. Most probably the
1850 request was done by hand using a telnet client, and aborted early.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001851
willy tarreauc5f73ed2005-12-18 01:26:38 +01001852 cR The client timed out before sending a full request. This is sometimes
1853 caused by too large TCP MSS values on the client side for PPPoE
1854 networks which cannot transport full-sized packets, or by clients
1855 sending requests by hand and not typing fast enough.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001856
willy tarreauc5f73ed2005-12-18 01:26:38 +01001857 SC The server explicitly refused the connection (the proxy received a
1858 TCP RST or an ICMP in return). Under some circumstances, it can
1859 also be the network stack telling the proxy that the server is
1860 unreachable (eg: no route, or no ARP response on local network).
willy tarreau982249e2005-12-18 00:57:06 +01001861
willy tarreauc5f73ed2005-12-18 01:26:38 +01001862 sC The connection to the server did not complete during contimeout.
willy tarreau982249e2005-12-18 00:57:06 +01001863
willy tarreauc5f73ed2005-12-18 01:26:38 +01001864 PC The proxy refused to establish a connection to the server because the
1865 maxconn limit has been reached. The listener's maxconn parameter may
1866 be increased in the proxy configuration, as well as the global
1867 maxconn parameter.
willy tarreauc1cae632005-12-17 14:12:23 +01001868
willy tarreauc5f73ed2005-12-18 01:26:38 +01001869 RC A local resource has been exhausted (memory, sockets, source ports)
1870 preventing the connection to the server from establishing. The error
1871 logs will tell precisely what was missing. Anyway, this can only be
1872 solved by system tuning.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001873
willy tarreauc5f73ed2005-12-18 01:26:38 +01001874 cH The client timed out during a POST request. This is sometimes caused
1875 by too large TCP MSS values for PPPoE networks which cannot transport
1876 full-sized packets.
willy tarreauc1cae632005-12-17 14:12:23 +01001877
willy tarreau078c79a2006-05-13 12:23:58 +02001878 CH The client aborted while waiting for the server to start responding.
1879 It might be the server taking too long to respond or the client
1880 clicking the 'Stop' button too fast.
1881
1882 CQ The client aborted while its session was queued, waiting for a server
1883 with enough empty slots to accept it. It might be that either all the
1884 servers were saturated or the assigned server taking too long to
1885 respond.
1886
Willy Tarreau08fa2e32006-09-03 10:47:37 +02001887 CT The client aborted while its session was tarpitted.
1888
willy tarreau078c79a2006-05-13 12:23:58 +02001889 sQ The session spent too much time in queue and has been expired.
1890
willy tarreauc5f73ed2005-12-18 01:26:38 +01001891 SH The server aborted before sending its full headers, or it crashed.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001892
willy tarreauc5f73ed2005-12-18 01:26:38 +01001893 sH The server failed to reply during the srvtimeout delay, which
1894 indicates too long transactions, probably caused by back-end
1895 saturation. The only solutions are to fix the problem on the
1896 application or to increase the 'srvtimeout' parameter to support
1897 longer delays (at the risk of the client giving up anyway).
1898
1899 PR The proxy blocked the client's request, either because of an invalid
1900 HTTP syntax, in which case it returned an HTTP 400 error to the
1901 client, or because a deny filter matched, in which case it returned
1902 an HTTP 403 error.
1903
1904 PH The proxy blocked the server's response, because it was invalid,
1905 incomplete, dangerous (cache control), or matched a security filter.
1906 In any case, an HTTP 502 error is sent to the client.
1907
Willy Tarreau2272dc12006-09-03 10:19:38 +02001908 PT The proxy blocked the client's request and has tarpitted its
1909 connection before returning it a 500 server error. Nothing was sent
1910 to the server.
1911
willy tarreauc5f73ed2005-12-18 01:26:38 +01001912 cD The client did not read any data for as long as the clitimeout delay.
1913 This is often caused by network failures on the client side.
1914
1915 CD The client unexpectedly aborted during data transfer. This is either
1916 caused by a browser crash, or by a keep-alive session between the
1917 server and the client terminated first by the client.
1918
1919 sD The server did nothing during the srvtimeout delay. This is often
1920 caused by too short timeouts on L4 equipements before the server
1921 (firewalls, load-balancers, ...).
1922
19234.2.5) Non-printable characters
willy tarreau4302f492005-12-18 01:00:37 +01001924-------------------------------
1925As of version 1.1.29, non-printable characters are not sent as-is into log
1926files, but are converted to their two-digits hexadecimal representation,
1927prefixed by the character '#'. The only characters that can now be logged
1928without being escaped are between 32 and 126 (inclusive). Obviously, the
1929escape character '#' is also encoded to avoid any ambiguity. It is the same for
1930the character '"', as well as '{', '|' and '}' when logging headers.
1931
willy tarreauc5f73ed2005-12-18 01:26:38 +010019324.2.6) Capturing HTTP headers and cookies
1933-----------------------------------------
1934Version 1.1.23 brought cookie capture, and 1.1.29 the header capture. All this
1935is performed using the 'capture' keyword.
1936
1937Cookie capture makes it easy to track a complete user session. The syntax is :
1938
1939 capture cookie <cookie_prefix> len <capture_length>
1940
1941This will enable cookie capture from both requests and responses. This way,
1942it's easy to detect when a user switches to a new session for example, because
1943the server will reassign it a new cookie.
1944
1945The FIRST cookie whose name starts with <cookie_prefix> will be captured, and
1946logged as 'NAME=value', without exceeding <capture_length> characters (64 max).
1947When the cookie name is fixed and known, it's preferable to suffix '=' to it to
1948ensure that no other cookie will be logged.
1949
1950Examples :
1951----------
1952 # capture the first cookie whose name starts with "ASPSESSION"
1953 capture cookie ASPSESSION len 32
1954
1955 # capture the first cookie whose name is exactly "vgnvisitor"
1956 capture cookie vgnvisitor= len 32
1957
1958In the logs, the field preceeding the completion indicator contains the cookie
1959value as sent by the server, preceeded by the cookie value as sent by the
1960client. Each of these field is replaced with '-' when no cookie was seen or
1961when the option is disabled.
1962
1963Header captures have a different goal. They are useful to track unique request
1964identifiers set by a previous proxy, virtual host names, user-agents, POST
1965content-length, referrers, etc. In the response, one can search for information
1966about the response length, how the server asked the cache to behave, or an
1967object location during a redirection. As for cookie captures, it is both
1968possible to include request headers and response headers at the same time. The
1969syntax is :
willy tarreau4302f492005-12-18 01:00:37 +01001970
1971 capture request header <name> len <max length>
1972 capture response header <name> len <max length>
1973
1974Note: Header names are not case-sensitive.
1975
1976Examples:
1977---------
1978 # keep the name of the virtual server
1979 capture request header Host len 20
1980 # keep the amount of data uploaded during a POST
1981 capture request header Content-Length len 10
1982
1983 # note the expected cache behaviour on the response
1984 capture response header Cache-Control len 8
1985 # note the URL location during a redirection
1986 capture response header Location len 20
1987
1988Non-existant headers are logged as empty strings, and if one header appears more
1989than once, only its last occurence will be kept. Request headers are grouped
1990within braces '{' and '}' in the same order as they were declared, and delimited
1991with a vertical bar '|' without any space. Response headers follow the same
1992representation, but are displayed after a space following the request headers
1993block. These blocks are displayed just before the HTTP request in the logs.
willy tarreauc5f73ed2005-12-18 01:26:38 +01001994
willy tarreau4302f492005-12-18 01:00:37 +01001995Example :
1996
willy tarreauc5f73ed2005-12-18 01:26:38 +01001997 Config:
1998
1999 capture request header Host len 20
2000 capture request header Content-Length len 10
2001 capture request header Referer len 20
2002 capture response header Server len 20
2003 capture response header Content-Length len 10
2004 capture response header Cache-Control len 8
2005 capture response header Via len 20
2006 capture response header Location len 20
2007
2008 Log :
2009
willy tarreau532bb552006-05-13 18:40:37 +02002010 Aug 9 20:26:09 localhost haproxy[2022]: 127.0.0.1:34014 [09/Aug/2004:20:26:09] relais-http netcache 0/0/0/162/+162 200 +350 - - ---- 0/0/0 0/0 {fr.adserver.yahoo.co||http://fr.f416.mail.} {|864|private||} "GET http://fr.adserver.yahoo.com/"
2011 Aug 9 20:30:46 localhost haproxy[2022]: 127.0.0.1:34020 [09/Aug/2004:20:30:46] relais-http netcache 0/0/0/182/+182 200 +279 - - ---- 0/0/0 0/0 {w.ods.org||} {Formilux/0.1.8|3495|||} "GET http://w.ods.org/sytadin.html HTTP/1.1"
2012 Aug 9 20:30:46 localhost haproxy[2022]: 127.0.0.1:34028 [09/Aug/2004:20:30:46] relais-http netcache 0/0/2/126/+128 200 +223 - - ---- 0/0/0 0/0 {www.infotrafic.com||http://w.ods.org/syt} {Apache/2.0.40 (Red H|9068|||} "GET http://www.infotrafic.com/images/live/cartesidf/grandes/idf_ne.png HTTP/1.1"
willy tarreauc5f73ed2005-12-18 01:26:38 +01002013
2014
20154.2.7) Examples of logs
2016-----------------------
willy tarreau532bb552006-05-13 18:40:37 +02002017- haproxy[674]: 127.0.0.1:33319 [15/Oct/2003:08:31:57] relais-http Srv1 6559/0/7/147/6723 200 243 - - ---- 1/3/5 0/0 "HEAD / HTTP/1.0"
willy tarreauc5f73ed2005-12-18 01:26:38 +01002018 => long request (6.5s) entered by hand through 'telnet'. The server replied
2019 in 147 ms, and the session ended normally ('----')
2020
willy tarreau532bb552006-05-13 18:40:37 +02002021- haproxy[674]: 127.0.0.1:33319 [15/Oct/2003:08:31:57] relais-http Srv1 6559/1230/7/147/6870 200 243 - - ---- 99/239/324 0/9 "HEAD / HTTP/1.0"
2022 => Idem, but the request was queued in the global queue behind 9 other
2023 requests, and waited there for 1230 ms.
2024
2025- haproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17] relais-http Srv1 9/0/7/14/+30 200 +243 - - ---- 1/3/3 0/0 "GET /image.iso HTTP/1.0"
willy tarreauc5f73ed2005-12-18 01:26:38 +01002026 => request for a long data transfer. The 'logasap' option was specified, so
2027 the log was produced just before transfering data. The server replied in
2028 14 ms, 243 bytes of headers were sent to the client, and total time from
2029 accept to first data byte is 30 ms.
2030
willy tarreau532bb552006-05-13 18:40:37 +02002031- haproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17] relais-http Srv1 9/0/7/14/30 502 243 - - PH-- 0/2/3 0/0 "GET /cgi-bin/bug.cgi? HTTP/1.0"
willy tarreauc5f73ed2005-12-18 01:26:38 +01002032 => the proxy blocked a server response either because of an 'rspdeny' or
2033 'rspideny' filter, or because it blocked sensible information which risked
2034 being cached. In this case, the response is replaced with a '502 bad
2035 gateway'.
2036
willy tarreau532bb552006-05-13 18:40:37 +02002037- haproxy[18113]: 127.0.0.1:34548 [15/Oct/2003:15:18:55] relais-http <NOSRV> -1/-1/-1/-1/8490 -1 0 - - CR-- 0/2/2 0/0 ""
willy tarreauc5f73ed2005-12-18 01:26:38 +01002038 => the client never completed its request and aborted itself ('C---') after
2039 8.5s, while the proxy was waiting for the request headers ('-R--').
2040 Nothing was sent to the server.
2041
willy tarreau532bb552006-05-13 18:40:37 +02002042- haproxy[18113]: 127.0.0.1:34549 [15/Oct/2003:15:19:06] relais-http <NOSRV> -1/-1/-1/-1/50001 408 0 - - cR-- 2/2 0/0 ""
willy tarreauc5f73ed2005-12-18 01:26:38 +01002043 => The client never completed its request, which was aborted by the time-out
2044 ('c---') after 50s, while the proxy was waiting for the request headers ('-R--').
2045 Nothing was sent to the server, but the proxy could send a 408 return code
2046 to the client.
willy tarreau4302f492005-12-18 01:00:37 +01002047
willy tarreau532bb552006-05-13 18:40:37 +02002048- haproxy[18989]: 127.0.0.1:34550 [15/Oct/2003:15:24:28] relais-tcp Srv1 0/0/5007 0 cD 0/0/0 0/0
willy tarreauc5f73ed2005-12-18 01:26:38 +01002049 => This is a 'tcplog' entry. Client-side time-out ('c----') occured after 5s.
willy tarreau4302f492005-12-18 01:00:37 +01002050
willy tarreau532bb552006-05-13 18:40:37 +02002051- haproxy[18989]: 10.0.0.1:34552 [15/Oct/2003:15:26:31] relais-http Srv1 3183/-1/-1/-1/11215 503 0 - - SC-- 115/202/205 0/0 "HEAD / HTTP/1.0"
willy tarreauc5f73ed2005-12-18 01:26:38 +01002052 => The request took 3s to complete (probably a network problem), and the
2053 connection to the server failed ('SC--') after 4 attemps of 2 seconds
2054 (config says 'retries 3'), then a 503 error code was sent to the client.
willy tarreau532bb552006-05-13 18:40:37 +02002055 There were 115 connections on this server, 202 connections on this proxy,
2056 and 205 on the global process. It is possible that the server refused the
2057 connection because of too many already established.
willy tarreau4302f492005-12-18 01:00:37 +01002058
willy tarreau4302f492005-12-18 01:00:37 +01002059
willy tarreau197e8ec2005-12-17 14:10:59 +010020604.3) HTTP header manipulation
2061-----------------------------
2062In HTTP mode, it is possible to rewrite, add or delete some of the request and
2063response headers based on regular expressions. It is also possible to block a
2064request or a response if a particular header matches a regular expression,
2065which is enough to stops most elementary protocol attacks, and to protect
2066against information leak from the internal network. But there is a limitation
2067to this : since haproxy's HTTP engine knows nothing about keep-alive, only
2068headers passed during the first request of a TCP session will be seen. All
2069subsequent headers will be considered data only and not analyzed. Furthermore,
2070haproxy doesn't touch data contents, it stops at the end of headers.
willy tarreaueedaa9f2005-12-17 14:08:03 +01002071
willy tarreau197e8ec2005-12-17 14:10:59 +01002072The syntax is :
2073 reqadd <string> to add a header to the request
2074 reqrep <search> <replace> to modify the request
2075 reqirep <search> <replace> same, but ignoring the case
2076 reqdel <search> to delete a header in the request
2077 reqidel <search> same, but ignoring the case
2078 reqallow <search> definitely allow a request if a header matches <search>
2079 reqiallow <search> same, but ignoring the case
2080 reqdeny <search> denies a request if a header matches <search>
2081 reqideny <search> same, but ignoring the case
2082 reqpass <search> ignore a header matching <search>
2083 reqipass <search> same, but ignoring the case
Willy Tarreau2272dc12006-09-03 10:19:38 +02002084 reqtarpit <search> tarpit a request matching <search>
2085 reqitarpit <search> same, but ignoring the case
willy tarreaueedaa9f2005-12-17 14:08:03 +01002086
willy tarreau197e8ec2005-12-17 14:10:59 +01002087 rspadd <string> to add a header to the response
2088 rsprep <search> <replace> to modify the response
2089 rspirep <search> <replace> same, but ignoring the case
2090 rspdel <search> to delete the response
2091 rspidel <search> same, but ignoring the case
willy tarreau982249e2005-12-18 00:57:06 +01002092 rspdeny <search> replaces a response with a HTTP 502 if a header matches <search>
2093 rspideny <search> same, but ignoring the case
willy tarreaueedaa9f2005-12-17 14:08:03 +01002094
2095
willy tarreau197e8ec2005-12-17 14:10:59 +01002096<search> is a POSIX regular expression (regex) which supports grouping through
2097parenthesis (without the backslash). Spaces and other delimiters must be
2098prefixed with a backslash ('\') to avoid confusion with a field delimiter.
2099Other characters may be prefixed with a backslash to change their meaning :
willy tarreaueedaa9f2005-12-17 14:08:03 +01002100
willy tarreau197e8ec2005-12-17 14:10:59 +01002101 \t for a tab
2102 \r for a carriage return (CR)
2103 \n for a new line (LF)
2104 \ to mark a space and differentiate it from a delimiter
2105 \# to mark a sharp and differentiate it from a comment
2106 \\ to use a backslash in a regex
2107 \\\\ to use a backslash in the text (*2 for regex, *2 for haproxy)
2108 \xXX to write the ASCII hex code XX as in the C language
willy tarreaueedaa9f2005-12-17 14:08:03 +01002109
2110
Willy Tarreau2272dc12006-09-03 10:19:38 +02002111<replace> contains the string to be used to replace the largest portion of text
willy tarreau197e8ec2005-12-17 14:10:59 +01002112matching the regex. It can make use of the special characters above, and can
2113reference a substring delimited by parenthesis in the regex, by the group
Willy Tarreau2272dc12006-09-03 10:19:38 +02002114numerical order from 0 to 9 (0 being the entire line). In this case, you would
2115write a backslash ('\') immediately followed by one digit indicating the group
2116position.
willy tarreaueedaa9f2005-12-17 14:08:03 +01002117
willy tarreau197e8ec2005-12-17 14:10:59 +01002118<string> represents the string which will systematically be added after the last
2119header line. It can also use special characters above.
willy tarreaueedaa9f2005-12-17 14:08:03 +01002120
willy tarreau197e8ec2005-12-17 14:10:59 +01002121Notes :
2122-------
2123 - the first line is considered as a header, which makes it possible to rewrite
2124 or filter HTTP requests URIs or response codes.
2125 - 'reqrep' is the equivalent of 'cliexp' in version 1.0, and 'rsprep' is the
2126 equivalent of 'srvexp' in 1.0. Those names are still supported but
2127 deprecated.
2128 - for performances reasons, the number of characters added to a request or to
2129 a response is limited to 4096 since version 1.1.5 (it was 256 before). This
2130 value is easy to modify in the code if needed (#define). If it is too short
2131 on occasional uses, it is possible to gain some space by removing some
2132 useless headers before adding new ones.
willy tarreau982249e2005-12-18 00:57:06 +01002133 - a denied request will generate an "HTTP 403 forbidden" response, while a
2134 denied response will generate an "HTTP 502 Bad gateway" response.
Willy Tarreau2272dc12006-09-03 10:19:38 +02002135 - a tarpitted request will be held open on the client side for a duration
Willy Tarreau08fa2e32006-09-03 10:47:37 +02002136 defined in the contimeout parameter, or untill the client aborts. Nothing
2137 will be sent to any server. When the timeout is reached, the proxy will
2138 reply with a 500 server error response so that the attacker does not
2139 suspect it has been tarpitted. The logs may report the 500, but the
2140 termination flags will indicate 'PT' in this case.
Willy Tarreau2272dc12006-09-03 10:19:38 +02002141
willy tarreaueedaa9f2005-12-17 14:08:03 +01002142
willy tarreau197e8ec2005-12-17 14:10:59 +01002143Examples :
2144----------
willy tarreauc5f73ed2005-12-18 01:26:38 +01002145 ###### a few examples ######
willy tarreau197e8ec2005-12-17 14:10:59 +01002146
willy tarreauc5f73ed2005-12-18 01:26:38 +01002147 # rewrite 'online.fr' instead of 'free.fr' for GET and POST requests
2148 reqrep ^(GET\ .*)(.free.fr)(.*) \1.online.fr\3
2149 reqrep ^(POST\ .*)(.free.fr)(.*) \1.online.fr\3
willy tarreau197e8ec2005-12-17 14:10:59 +01002150
willy tarreauc5f73ed2005-12-18 01:26:38 +01002151 # force proxy connections to close
2152 reqirep ^Proxy-Connection:.* Proxy-Connection:\ close
2153 # rewrite locations
2154 rspirep ^(Location:\ )([^:]*://[^/]*)(.*) \1\3
willy tarreaueedaa9f2005-12-17 14:08:03 +01002155
willy tarreauc5f73ed2005-12-18 01:26:38 +01002156 ###### A full configuration being used on production ######
willy tarreaueedaa9f2005-12-17 14:08:03 +01002157
willy tarreau197e8ec2005-12-17 14:10:59 +01002158 # Every header should end with a colon followed by one space.
willy tarreauc5f73ed2005-12-18 01:26:38 +01002159 reqideny ^[^:\ ]*[\ ]*$
willy tarreaueedaa9f2005-12-17 14:08:03 +01002160
willy tarreau197e8ec2005-12-17 14:10:59 +01002161 # block Apache chunk exploit
willy tarreauc5f73ed2005-12-18 01:26:38 +01002162 reqideny ^Transfer-Encoding:[\ ]*chunked
2163 reqideny ^Host:\ apache-
willy tarreaueedaa9f2005-12-17 14:08:03 +01002164
willy tarreau197e8ec2005-12-17 14:10:59 +01002165 # block annoying worms that fill the logs...
willy tarreauc5f73ed2005-12-18 01:26:38 +01002166 reqideny ^[^:\ ]*\ .*(\.|%2e)(\.|%2e)(%2f|%5c|/|\\\\)
2167 reqideny ^[^:\ ]*\ ([^\ ]*\ [^\ ]*\ |.*%00)
2168 reqideny ^[^:\ ]*\ .*<script
2169 reqideny ^[^:\ ]*\ .*/(root\.exe\?|cmd\.exe\?|default\.ida\?)
willy tarreau197e8ec2005-12-17 14:10:59 +01002170
Willy Tarreau2272dc12006-09-03 10:19:38 +02002171 # tarpit attacks on the login page.
2172 reqtarpit ^[^:\ ]*\ .*\.php?login=[^0-9]
2173
willy tarreau197e8ec2005-12-17 14:10:59 +01002174 # allow other syntactically valid requests, and block any other method
willy tarreauc5f73ed2005-12-18 01:26:38 +01002175 reqipass ^(GET|POST|HEAD|OPTIONS)\ /.*\ HTTP/1\.[01]$
2176 reqipass ^OPTIONS\ \\*\ HTTP/1\.[01]$
2177 reqideny ^[^:\ ]*\
willy tarreau197e8ec2005-12-17 14:10:59 +01002178
2179 # force connection:close, thus disabling HTTP keep-alive
willy tarreauc5f73ed2005-12-18 01:26:38 +01002180 option httpclose
willy tarreau197e8ec2005-12-17 14:10:59 +01002181
willy tarreauc5f73ed2005-12-18 01:26:38 +01002182 # change the server name
2183 rspidel ^Server:\
2184 rspadd Server:\ Formilux/0.1.8
willy tarreau197e8ec2005-12-17 14:10:59 +01002185
2186
willy tarreau982249e2005-12-18 00:57:06 +01002187Also, the 'forwardfor' option creates an HTTP 'X-Forwarded-For' header which
willy tarreauc1cae632005-12-17 14:12:23 +01002188contains the client's IP address. This is useful to let the final web server
Willy Tarreau7ac51f62007-03-25 16:00:04 +02002189know what the client address was (eg for statistics on domains). Starting with
2190version 1.3.8, it is possible to specify the "except" keyword followed by a
2191source IP address or network for which no header will be added. This is very
2192useful when another reverse-proxy which already adds the header runs on the
2193same machine or in a known DMZ, the most common case being the local use of
2194stunnel on the same system.
willy tarreauc1cae632005-12-17 14:12:23 +01002195
willy tarreau982249e2005-12-18 00:57:06 +01002196Last, the 'httpclose' option removes any 'Connection' header both ways, and
2197adds a 'Connection: close' header in each direction. This makes it easier to
Willy TARREAU767ba712006-03-01 22:40:50 +01002198disable HTTP keep-alive than the previous 4-rules block.
willy tarreau982249e2005-12-18 00:57:06 +01002199
willy tarreauc1cae632005-12-17 14:12:23 +01002200Example :
2201---------
2202 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01002203 mode http
2204 log global
2205 option httplog
2206 option dontlognull
Willy Tarreau7ac51f62007-03-25 16:00:04 +02002207 option forwardfor except 127.0.0.1/8
willy tarreauc5f73ed2005-12-18 01:26:38 +01002208 option httpclose
2209
Willy TARREAU767ba712006-03-01 22:40:50 +01002210Note that some HTTP servers do not necessarily close the connections when they
2211receive the 'Connection: close', and if the client does not close either, then
2212the connection will be maintained up to the time-out. This translates into high
2213number of simultaneous sessions and high global session times in the logs. To
2214workaround this, a new option 'forceclose' appeared in version 1.2.9 to enforce
2215the closing of the outgoing server channel as soon as the server begins to
2216reply and only if the request buffer is empty. Note that this should NOT be
2217used if CONNECT requests are expected between the client and the server. The
2218'forceclose' option implies the 'httpclose' option.
2219
2220Example :
2221---------
2222 listen http_proxy 0.0.0.0:80
2223 mode http
2224 log global
2225 option httplog
2226 option dontlognull
2227 option forwardfor
2228 option forceclose
2229
willy tarreau197e8ec2005-12-17 14:10:59 +01002230
22314.4) Load balancing with persistence
2232------------------------------------
willy tarreau197e8ec2005-12-17 14:10:59 +01002233Combining cookie insertion with internal load balancing allows to transparently
2234bring persistence to applications. The principle is quite simple :
2235 - assign a cookie value to each server
2236 - enable the load balancing between servers
2237 - insert a cookie into responses resulting from the balancing algorithm
2238 (indirect accesses), end ensure that no upstream proxy will cache it.
2239 - remove the cookie in the request headers so that the application never sees
2240 it.
2241
2242Example :
willy tarreaueedaa9f2005-12-17 14:08:03 +01002243---------
2244 listen application 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01002245 mode http
2246 cookie SERVERID insert nocache indirect
2247 balance roundrobin
2248 server srv1 192.168.1.1:80 cookie server01 check
2249 server srv2 192.168.1.2:80 cookie server02 check
willy tarreaueedaa9f2005-12-17 14:08:03 +01002250
willy tarreau0174f312005-12-18 01:02:42 +01002251The other solution brought by versions 1.1.30 and 1.2.3 is to reuse a cookie
2252from the server, and prefix the server's name to it. In this case, don't forget
2253to force "httpclose" mode so that you can be assured that every subsequent
2254request will have its cookie fixed.
2255
2256 listen application 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01002257 mode http
2258 cookie JSESSIONID prefix
2259 balance roundrobin
2260 server srv1 192.168.1.1:80 cookie srv1 check
2261 server srv2 192.168.1.2:80 cookie srv2 check
2262 option httpclose
willy tarreau0174f312005-12-18 01:02:42 +01002263
2264
willy tarreau982249e2005-12-18 00:57:06 +010022654.5) Protection against information leak from the servers
2266---------------------------------------------------------
2267In versions 1.1.28/1.2.1, a new option 'checkcache' was created. It carefully
2268checks 'Cache-control', 'Pragma' and 'Set-cookie' headers in server response
2269to check if there's a risk of caching a cookie on a client-side proxy. When this
2270option is enabled, the only responses which can be delivered to the client are :
2271 - all those without 'Set-Cookie' header ;
2272 - all those with a return code other than 200, 203, 206, 300, 301, 410,
2273 provided that the server has not set a 'Cache-control: public' header ;
2274 - all those that come from a POST request, provided that the server has not
2275 set a 'Cache-Control: public' header ;
2276 - those with a 'Pragma: no-cache' header
2277 - those with a 'Cache-control: private' header
2278 - those with a 'Cache-control: no-store' header
2279 - those with a 'Cache-control: max-age=0' header
2280 - those with a 'Cache-control: s-maxage=0' header
2281 - those with a 'Cache-control: no-cache' header
2282 - those with a 'Cache-control: no-cache="set-cookie"' header
2283 - those with a 'Cache-control: no-cache="set-cookie,' header
2284 (allowing other fields after set-cookie)
willy tarreaueedaa9f2005-12-17 14:08:03 +01002285
willy tarreau982249e2005-12-18 00:57:06 +01002286If a response doesn't respect these requirements, then it will be blocked just
2287as if it was from an 'rspdeny' filter, with an "HTTP 502 bad gateway". The
2288session state shows "PH--" meaning that the proxy blocked the response during
2289headers processing. Additionnaly, an alert will be sent in the logs so that
2290admins are told that there's something to be done.
2291
willy tarreauc5f73ed2005-12-18 01:26:38 +01002292
willy tarreau982249e2005-12-18 00:57:06 +010022934.6) Customizing errors
2294-----------------------
willy tarreau197e8ec2005-12-17 14:10:59 +01002295Some situations can make haproxy return an HTTP error code to the client :
2296 - invalid or too long request => HTTP 400
2297 - request not completely sent in time => HTTP 408
2298 - forbidden request (matches a deny filter) => HTTP 403
2299 - internal error in haproxy => HTTP 500
2300 - the server returned an invalid or incomplete response => HTTP 502
2301 - no server was available to handle the request => HTTP 503
2302 - the server failed to reply in time => HTTP 504
willy tarreaueedaa9f2005-12-17 14:08:03 +01002303
willy tarreau197e8ec2005-12-17 14:10:59 +01002304A succint error message taken from the RFC accompanies these return codes.
2305But depending on the clients knowledge, it may be better to return custom, user
2306friendly, error pages. This is made possible through the use of the 'errorloc'
2307command :
willy tarreaueedaa9f2005-12-17 14:08:03 +01002308
willy tarreau197e8ec2005-12-17 14:10:59 +01002309 errorloc <HTTP_code> <location>
willy tarreaueedaa9f2005-12-17 14:08:03 +01002310
willy tarreau197e8ec2005-12-17 14:10:59 +01002311Instead of generating an HTTP error <HTTP_code> among those above, the proxy
2312will return a temporary redirection code (HTTP 302) towards the address
2313specified in <location>. This address may be either relative to the site or
2314absolute. Since this request will be handled by the client's browser, it's
2315mandatory that the returned address be reachable from the outside.
willy tarreaueedaa9f2005-12-17 14:08:03 +01002316
willy tarreau197e8ec2005-12-17 14:10:59 +01002317Example :
willy tarreaueedaa9f2005-12-17 14:08:03 +01002318---------
2319 listen application 0.0.0.0:80
2320 errorloc 400 /badrequest.html
2321 errorloc 403 /forbidden.html
2322 errorloc 408 /toolong.html
willy tarreauc5f73ed2005-12-18 01:26:38 +01002323 errorloc 500 http://haproxy.domain.net/bugreport.html
willy tarreaueedaa9f2005-12-17 14:08:03 +01002324 errorloc 502 http://192.168.114.58/error50x.html
2325 errorloc 503 http://192.168.114.58/error50x.html
2326 errorloc 504 http://192.168.114.58/error50x.html
2327
willy tarreauc1f47532005-12-18 01:08:26 +01002328Note: RFC2616 says that a client must reuse the same method to fetch the
2329Location returned by a 302, which causes problems with the POST method.
2330The return code 303 was designed explicitly to force the client to fetch the
2331Location URL with the GET method, but there are some browsers pre-dating
2332HTTP/1.1 which don't support it. Anyway, most browsers still behave with 302 as
willy tarreauc5f73ed2005-12-18 01:26:38 +01002333if it was a 303. In order to allow the user to chose, versions 1.1.31 and 1.2.5
2334bring two new keywords to replace 'errorloc' : 'errorloc302' and 'errorloc303'.
willy tarreauc1f47532005-12-18 01:08:26 +01002335
2336They are preffered over errorloc (which still does 302). Consider using
2337errorloc303 everytime you know that your clients support HTTP 303 responses..
2338
2339
willy tarreau982249e2005-12-18 00:57:06 +010023404.7) Modifying default values
willy tarreau197e8ec2005-12-17 14:10:59 +01002341-----------------------------
willy tarreau197e8ec2005-12-17 14:10:59 +01002342Version 1.1.22 introduced the notion of default values, which eliminates the
2343pain of often repeating common parameters between many instances, such as
2344logs, timeouts, modes, etc...
willy tarreaueedaa9f2005-12-17 14:08:03 +01002345
willy tarreau197e8ec2005-12-17 14:10:59 +01002346Default values are set in a 'defaults' section. Each of these section clears
2347all previously set default parameters, so there may be as many default
2348parameters as needed. Only the last one before a 'listen' section will be
2349used for this section. The 'defaults' section uses the same syntax as the
2350'listen' section, for the supported parameters. The 'defaults' keyword ignores
2351everything on its command line, so that fake instance names can be specified
2352there for better clarity.
willy tarreaueedaa9f2005-12-17 14:08:03 +01002353
willy tarreau982249e2005-12-18 00:57:06 +01002354In version 1.1.28/1.2.1, only those parameters can be preset in the 'default'
willy tarreau197e8ec2005-12-17 14:10:59 +01002355section :
2356 - log (the first and second one)
willy tarreaueedaa9f2005-12-17 14:08:03 +01002357 - mode { tcp, http, health }
2358 - balance { roundrobin }
willy tarreau197e8ec2005-12-17 14:10:59 +01002359 - disabled (to disable every further instances)
2360 - enabled (to enable every further instances, this is the default)
willy tarreaueedaa9f2005-12-17 14:08:03 +01002361 - contimeout, clitimeout, srvtimeout, grace, retries, maxconn
willy tarreau982249e2005-12-18 00:57:06 +01002362 - option { redispatch, transparent, keepalive, forwardfor, logasap, httpclose,
2363 checkcache, httplog, tcplog, dontlognull, persist, httpchk }
willy tarreaueedaa9f2005-12-17 14:08:03 +01002364 - redispatch, redisp, transparent, source { addr:port }
2365 - cookie, capture
2366 - errorloc
2367
willy tarreau197e8ec2005-12-17 14:10:59 +01002368As of 1.1.24, it is not possible to put certain parameters in a 'defaults'
2369section, mainly regular expressions and server configurations :
willy tarreaueedaa9f2005-12-17 14:08:03 +01002370 - dispatch, server,
willy tarreau197e8ec2005-12-17 14:10:59 +01002371 - req*, rsp*
willy tarreaueedaa9f2005-12-17 14:08:03 +01002372
willy tarreau197e8ec2005-12-17 14:10:59 +01002373Last, there's no way yet to change a boolean option from its assigned default
2374value. So if an 'option' statement is set in a 'defaults' section, the only
2375way to flush it is to redefine a new 'defaults' section without this 'option'.
willy tarreaueedaa9f2005-12-17 14:08:03 +01002376
willy tarreau197e8ec2005-12-17 14:10:59 +01002377Examples :
willy tarreaueedaa9f2005-12-17 14:08:03 +01002378----------
2379 defaults applications TCP
willy tarreauc5f73ed2005-12-18 01:26:38 +01002380 log global
2381 mode tcp
2382 balance roundrobin
2383 clitimeout 180000
2384 srvtimeout 180000
2385 contimeout 4000
2386 retries 3
2387 redispatch
willy tarreaueedaa9f2005-12-17 14:08:03 +01002388
2389 listen app_tcp1 10.0.0.1:6000-6063
willy tarreauc5f73ed2005-12-18 01:26:38 +01002390 server srv1 192.168.1.1 check port 6000 inter 10000
2391 server srv2 192.168.1.2 backup
willy tarreaueedaa9f2005-12-17 14:08:03 +01002392
2393 listen app_tcp2 10.0.0.2:6000-6063
willy tarreauc5f73ed2005-12-18 01:26:38 +01002394 server srv1 192.168.2.1 check port 6000 inter 10000
2395 server srv2 192.168.2.2 backup
willy tarreaueedaa9f2005-12-17 14:08:03 +01002396
2397 defaults applications HTTP
willy tarreauc5f73ed2005-12-18 01:26:38 +01002398 log global
2399 mode http
2400 option httplog
2401 option forwardfor
2402 option dontlognull
2403 balance roundrobin
2404 clitimeout 20000
2405 srvtimeout 20000
2406 contimeout 4000
2407 retries 3
willy tarreaueedaa9f2005-12-17 14:08:03 +01002408
2409 listen app_http1 10.0.0.1:80-81
willy tarreauc5f73ed2005-12-18 01:26:38 +01002410 cookie SERVERID postonly insert indirect
2411 capture cookie userid= len 10
2412 server srv1 192.168.1.1:+8000 cookie srv1 check port 8080 inter 1000
2413 server srv1 192.168.1.2:+8000 cookie srv2 check port 8080 inter 1000
willy tarreaueedaa9f2005-12-17 14:08:03 +01002414
2415 defaults
willy tarreauc5f73ed2005-12-18 01:26:38 +01002416 # this empty section voids all default parameters
willy tarreaueedaa9f2005-12-17 14:08:03 +01002417
willy tarreau481132e2006-05-21 21:43:10 +02002418
24194.8) Status report in HTML page
2420-------------------------------
2421Starting with 1.2.14, it is possible for HAProxy to intercept requests for a
2422particular URI and return a full report of the proxy's activity and servers
2423statistics. This is available through the 'stats' keyword, associated to any
2424such options :
2425
2426 - stats enable
2427 - stats uri <uri prefix>
2428 - stats realm <authentication realm>
2429 - stats auth <user:password>
2430 - stats scope <proxy_id> | '.'
2431
2432By default, the status report is disabled. Specifying any combination above
2433enables it for the proxy instance referencing it. The easiest solution is to
2434use "stats enable" which will enable the report with default parameters :
2435
2436 - default URI : "/haproxy?stats" (CONFIG_STATS_DEFAULT_URI)
2437 - default auth : unspecified (no authentication)
2438 - default realm : "HAProxy Statistics" (CONFIG_STATS_DEFAULT_REALM)
2439 - default scope : unspecified (access to all instances)
2440
2441The "stats uri <uri_prefix>" option allows one to intercept another URI prefix.
2442Note that any URI that BEGINS with this string will match. For instance, one
2443proxy instance might be dedicated to status page only and would reply to any
2444URI.
2445
2446Example :
2447---------
2448 # catches any URI and returns the status page.
2449 listen stats :8080
2450 mode http
2451 stats uri /
2452
2453The "stats auth <user:password>" option enables Basic authentication and adds a
2454valid user:password combination to the list of authorized accounts. The user
2455and password are passed in the configuration file as clear text, and since this
2456is HTTP Basic authentication, you should be aware that it transits as clear
2457text on the network, so you must not use any sensible account. The list is
2458unlimited in order to provide easy accesses to developpers or customers.
2459
2460The "stats realm <realm>" option defines the "realm" name which is displayed
2461in the popup box when the browser asks for a password. It's important to ensure
2462that this one is not used by the application, otherwise the browser will try to
2463use a cached one from the application. Note that any space in the realm name
2464should be escaped with a backslash ('\').
2465
2466The "stats scope <proxy_id>" option limits the scope of the status report. By
2467default, all proxy instances are listed. But under some circumstances, it would
2468be better to limit the listing to some proxies or only to the current one. This
2469is what this option does. The special proxy name "." (a single dot) references
2470the current proxy. The proxy name can be repeated multiple times, even for
2471proxies defined later in the configuration or some which do not exist. The name
2472is the one which appears after the 'listen' keyword.
2473
2474Example :
2475---------
2476 # simple application with authenticated embedded status report
2477 listen app1 192.168.1.100:80
2478 mode http
willy tarreaud4ba08d2006-05-21 21:54:14 +02002479 option httpclose
willy tarreau481132e2006-05-21 21:43:10 +02002480 balance roundrobin
2481 cookie SERVERID postonly insert indirect
2482 server srv1 192.168.1.1:8080 cookie srv1 check inter 1000
2483 server srv1 192.168.1.2:8080 cookie srv2 check inter 1000
2484 stats uri /my_stats
willy tarreaud4ba08d2006-05-21 21:54:14 +02002485 stats realm Statistics\ for\ MyApp1-2
2486 stats auth guest:guest
2487 stats auth admin:AdMiN123
2488 stats scope .
2489 stats scope app2
willy tarreau481132e2006-05-21 21:43:10 +02002490
2491 # simple application with anonymous embedded status report
2492 listen app2 192.168.2.100:80
2493 mode http
willy tarreaud4ba08d2006-05-21 21:54:14 +02002494 option httpclose
willy tarreau481132e2006-05-21 21:43:10 +02002495 balance roundrobin
2496 cookie SERVERID postonly insert indirect
2497 server srv1 192.168.2.1:8080 cookie srv1 check inter 1000
2498 server srv1 192.168.2.2:8080 cookie srv2 check inter 1000
2499 stats uri /my_stats
willy tarreaud4ba08d2006-05-21 21:54:14 +02002500 stats realm Statistics\ for\ MyApp2
2501 stats scope .
willy tarreau481132e2006-05-21 21:43:10 +02002502
2503 listen admin_page :8080
2504 mode http
2505 stats uri /my_stats
willy tarreaud4ba08d2006-05-21 21:54:14 +02002506 stats realm Global\ statistics
2507 stats auth admin:AdMiN123
willy tarreau481132e2006-05-21 21:43:10 +02002508
2509Notes :
2510-------
2511 - The 'stats' options can also be specified in the 'defaults' section, in
2512 which case it will provide the exact same configuration to all further
2513 instances (hence the usefulness of the scope "."). However, if an instance
2514 redefines any 'stats' parameter, defaults will not be used for this
2515 instance.
2516
2517 - HTTP Basic authentication is very basic and unsecure from snooping. No
2518 sensible password should be used, and be aware that there is no way to
2519 remove it from the browser so it will be sent to the whole application
2520 upon further accesses.
2521
willy tarreaud4ba08d2006-05-21 21:54:14 +02002522 - It is very important that the 'option httpclose' is specified, otherwise
2523 the proxy will not be able to detect the URI within keep-alive sessions
2524 maintained between the browser and the servers, so the stats URI will be
2525 forwarded unmodified to the server as if the option was not set.
2526
willy tarreau481132e2006-05-21 21:43:10 +02002527
willy tarreau197e8ec2005-12-17 14:10:59 +01002528=========================
2529| System-specific setup |
2530=========================
willy tarreaueedaa9f2005-12-17 14:08:03 +01002531
willy tarreau197e8ec2005-12-17 14:10:59 +01002532Linux 2.4
2533=========
willy tarreaueedaa9f2005-12-17 14:08:03 +01002534
2535-- cut here --
2536#!/bin/sh
2537# set this to about 256/4M (16384 for 256M machine)
2538MAXFILES=16384
2539echo $MAXFILES > /proc/sys/fs/file-max
2540ulimit -n $MAXFILES
2541
2542if [ -e /proc/sys/net/ipv4/ip_conntrack_max ]; then
willy tarreauc5f73ed2005-12-18 01:26:38 +01002543 echo 65536 > /proc/sys/net/ipv4/ip_conntrack_max
willy tarreaueedaa9f2005-12-17 14:08:03 +01002544fi
2545
2546if [ -e /proc/sys/net/ipv4/netfilter/ip_ct_tcp_timeout_fin_wait ]; then
willy tarreauc5f73ed2005-12-18 01:26:38 +01002547 # 30 seconds for fin, 15 for time wait
2548 echo 3000 > /proc/sys/net/ipv4/netfilter/ip_ct_tcp_timeout_fin_wait
2549 echo 1500 > /proc/sys/net/ipv4/netfilter/ip_ct_tcp_timeout_time_wait
2550 echo 0 > /proc/sys/net/ipv4/netfilter/ip_ct_tcp_log_invalid_scale
2551 echo 0 > /proc/sys/net/ipv4/netfilter/ip_ct_tcp_log_out_of_window
willy tarreaueedaa9f2005-12-17 14:08:03 +01002552fi
2553
2554echo 1024 60999 > /proc/sys/net/ipv4/ip_local_port_range
2555echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
2556echo 4096 > /proc/sys/net/ipv4/tcp_max_syn_backlog
2557echo 262144 > /proc/sys/net/ipv4/tcp_max_tw_buckets
2558echo 262144 > /proc/sys/net/ipv4/tcp_max_orphans
2559echo 300 > /proc/sys/net/ipv4/tcp_keepalive_time
2560echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
2561echo 0 > /proc/sys/net/ipv4/tcp_timestamps
2562echo 0 > /proc/sys/net/ipv4/tcp_ecn
willy tarreauc5f73ed2005-12-18 01:26:38 +01002563echo 1 > /proc/sys/net/ipv4/tcp_sack
willy tarreaueedaa9f2005-12-17 14:08:03 +01002564echo 0 > /proc/sys/net/ipv4/tcp_dsack
2565
2566# auto-tuned on 2.4
2567#echo 262143 > /proc/sys/net/core/rmem_max
2568#echo 262143 > /proc/sys/net/core/rmem_default
2569
2570echo 16384 65536 524288 > /proc/sys/net/ipv4/tcp_rmem
2571echo 16384 349520 699040 > /proc/sys/net/ipv4/tcp_wmem
2572
2573-- cut here --
2574
willy tarreau197e8ec2005-12-17 14:10:59 +01002575
2576FreeBSD
2577=======
2578
2579A FreeBSD port of HA-Proxy is now available and maintained, thanks to
2580Clement Laforet <sheepkiller@cultdeadsheep.org>.
2581
2582For more information :
2583http://www.freebsd.org/cgi/url.cgi?ports/net/haproxy/pkg-descr
2584http://www.freebsd.org/cgi/cvsweb.cgi/ports/net/haproxy/
2585http://www.freshports.org/net/haproxy
2586
2587
2588-- end --