blob: 59cba6869dade7883af39b2820dd40df5dc0bb9d [file] [log] [blame]
willy tarreau0174f312005-12-18 01:02:42 +01001 -------------------
2 H A - P r o x y
3 Reference Manual
4 -------------------
willy tarreau34f45302006-04-15 21:37:14 +02005 version 1.2.12
willy tarreauc5f73ed2005-12-18 01:26:38 +01006 willy tarreau
willy tarreau34f45302006-04-15 21:37:14 +02007 2006/04/15
willy tarreaueedaa9f2005-12-17 14:08:03 +01008
9============
10| Abstract |
11============
12
13HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high
14availability environments. Indeed, it can :
15 - route HTTP requests depending on statically assigned cookies ;
16 - spread the load among several servers while assuring server persistence
17 through the use of HTTP cookies ;
18 - switch to backup servers in the event a main one fails ;
19 - accept connections to special ports dedicated to service monitoring ;
20 - stop accepting connections without breaking existing ones ;
21 - add/modify/delete HTTP headers both ways ;
22 - block requests matching a particular pattern ;
23
24It needs very little resource. Its event-driven architecture allows it to easily
25handle thousands of simultaneous connections on hundreds of instances without
26risking the system's stability.
27
28====================
29| Start parameters |
30====================
31
32There are only a few command line options :
33
34 -f <configuration file>
35 -n <high limit for the total number of simultaneous connections>
36 -N <high limit for the per-proxy number of simultaneous connections>
37 -d starts in foregreound with debugging mode enabled
38 -D starts in daemon mode
willy tarreau982249e2005-12-18 00:57:06 +010039 -q disable messages on output
40 -V displays messages on output even when -q or 'quiet' are specified.
41 -c only checks config file and exits with code 0 if no error was found, or
42 exits with code 1 if a syntax error was found.
willy tarreaufe2c5c12005-12-17 14:14:34 +010043 -p <pidfile> asks the process to write down each of its children's
44 pids to this file in daemon mode.
willy tarreau34f45302006-04-15 21:37:14 +020045 -sf specifies a list of pids to send a FINISH signal to after startup.
46 -st specifies a list of pids to send a TERMINATE signal to after startup.
willy tarreaueedaa9f2005-12-17 14:08:03 +010047 -s shows statistics (only if compiled in)
48 -l shows even more statistics (implies '-s')
willy tarreau64a3cc32005-12-18 01:13:11 +010049 -de disables use of epoll()
50 -dp disables use of poll()
willy tarreau34f45302006-04-15 21:37:14 +020051 -db disables background mode (stays in foreground, useful for debugging)
52 -m <megs> enforces a memory usage limit to a maximum of <megs> megabytes.
willy tarreaueedaa9f2005-12-17 14:08:03 +010053
54The maximal number of connections per proxy is used as the default parameter for
55each instance for which the 'maxconn' paramter is not set in the 'listen' section.
56
57The maximal number of total connections limits the number of connections used by
58the whole process if the 'maxconn' parameter is not set in the 'global' section.
59
60The debugging mode has the same effect as the 'debug' option in the 'global'
61section. When the proxy runs in this mode, it dumps every connections,
62disconnections, timestamps, and HTTP headers to stdout. This should NEVER
63be used in an init script since it will prevent the system from starting up.
64
willy tarreau34f45302006-04-15 21:37:14 +020065For debugging, the '-db' option is very useful as it temporarily disables
66daemon mode and multi-process mode. The service can then be stopped by simply
67pressing Ctrl-C, without having to edit the config nor run full debug.
68
willy tarreaueedaa9f2005-12-17 14:08:03 +010069Statistics are only available if compiled in with the 'STATTIME' option. It's
70only used during code optimization phases.
71
willy tarreau34f45302006-04-15 21:37:14 +020072The '-st' and '-sf' options are used to inform previously running processes
73that a configuration is being reloaded. They will receive the SIGTTOU signal to
74ask them to temporarily stop listening to the ports so that the new process
75can grab them. If anything wrong happens, the new process will send them a
76SIGTTIN to tell them to re-listen to the ports and continue their normal
77work. Otherwise, it will either ask them to finish (-sf) their work then softly
78exit, or immediately terminate (-st), breaking existing sessions. A typical use
79of this allows a configuration reload without service interruption :
80
81 # haproxy -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
82
willy tarreaueedaa9f2005-12-17 14:08:03 +010083======================
84| Configuration file |
85======================
86
87Structure
88=========
89
90The configuration file parser ignores empty lines, spaces, tabs. Anything
91between a sharp ('#') not following a backslash ('\'), and the end of a line
92constitutes a comment and is ignored too.
93
94The configuration file is segmented in sections. A section begins whenever
95one of these 3 keywords are encountered :
96
97 - 'global'
98 - 'listen'
99 - 'defaults'
100
101Every parameter refer to the section beginning at the last one of these 3
102keywords.
103
104
1051) Global parameters
106====================
107
108Global parameters affect the whole process behaviour. They are all set in the
109'global' section. There may be several 'global' sections if needed, but their
110parameters will only be merged. Allowed parameters in 'global' section include
111the following ones :
112
113 - log <address> <facility> [max_level]
114 - maxconn <number>
115 - uid <user id>
116 - gid <group id>
117 - chroot <directory>
118 - nbproc <number>
119 - daemon
120 - debug
willy tarreau64a3cc32005-12-18 01:13:11 +0100121 - noepoll
122 - nopoll
willy tarreaueedaa9f2005-12-17 14:08:03 +0100123 - quiet
willy tarreaufe2c5c12005-12-17 14:14:34 +0100124 - pidfile <file>
willy tarreauc5f73ed2005-12-18 01:26:38 +0100125 - ulimit-n <number>
willy tarreau598da412005-12-18 01:07:29 +0100126 - stats
willy tarreaueedaa9f2005-12-17 14:08:03 +0100127
willy tarreauc5f73ed2005-12-18 01:26:38 +0100128
willy tarreaueedaa9f2005-12-17 14:08:03 +01001291.1) Event logging
130------------------
131Most events are logged : start, stop, servers going up and down, connections and
132errors. Each event generates a syslog message which can be sent to up to 2
133servers. The syntax is :
134
135 log <ip_address> <facility> [max_level]
136
137Connections are logged at level "info". Services initialization and servers
138going up are logged at level "notice", termination signals are logged at
139"warning", and definitive service termination, as well as loss of servers are
140logged at level "alert". The optional parameter <max_level> specifies above
141what level messages should be sent. Level can take one of these 8 values :
142
143 emerg, alert, crit, err, warning, notice, info, debug
144
145For backwards compatibility with versions 1.1.16 and earlier, the default level
146value is "debug" if not specified.
147
148Permitted facilities are :
149 kern, user, mail, daemon, auth, syslog, lpr, news,
150 uucp, cron, auth2, ftp, ntp, audit, alert, cron2,
151 local0, local1, local2, local3, local4, local5, local6, local7
152
153According to RFC3164, messages are truncated to 1024 bytes before being emitted.
154
155Example :
156---------
157 global
willy tarreauc5f73ed2005-12-18 01:26:38 +0100158 log 192.168.2.200 local3
159 log 127.0.0.1 local4 notice
160
willy tarreaueedaa9f2005-12-17 14:08:03 +0100161
1621.2) limiting the number of connections
163---------------------------------------
164It is possible and recommended to limit the global number of per-process
willy tarreauc5f73ed2005-12-18 01:26:38 +0100165connections using the 'maxconn' global keyword. Since one connection includes
166both a client and a server, it means that the max number of TCP sessions will
167be about the double of this number. It's important to understand this when
168trying to find best values for 'ulimit -n' before starting the proxy. To
169anticipate the number of sockets needed, all these parameters must be counted :
willy tarreaueedaa9f2005-12-17 14:08:03 +0100170
171 - 1 socket per incoming connection
172 - 1 socket per outgoing connection
173 - 1 socket per address/port/proxy tuple.
174 - 1 socket per server being health-checked
175 - 1 socket for all logs
176
177In simple configurations where each proxy only listens one one address/port,
willy tarreauc5f73ed2005-12-18 01:26:38 +0100178set the limit of file descriptors (ulimit -n) to
179(2 * maxconn + nbproxies + nbservers + 1). Starting with versions 1.1.32/1.2.6,
180it is now possible to set the limit in the configuration using the 'ulimit-n'
181global keyword, provided the proxy is started as root. This puts an end to the
182recurrent problem of ensuring that the system limits are adapted to the proxy
183values. Note that these limits are per-process.
184
185Example :
186---------
187 global
188 maxconn 32000
189 ulimit-n 65536
190
willy tarreaueedaa9f2005-12-17 14:08:03 +0100191
1921.3) Drop of priviledges
193------------------------
194In order to reduce the risk and consequences of attacks, in the event where a
195yet non-identified vulnerability would be successfully exploited, it's possible
196to lower the process priviledges and even isolate it in a riskless directory.
197
198In the 'global' section, the 'uid' parameter sets a numerical user identifier
199which the process will switch to after binding its listening sockets. The value
200'0', which normally represents the super-user, here indicates that the UID must
201not change during startup. It's the default behaviour. The 'gid' parameter does
202the same for the group identifier. It's particularly advised against use of
203generic accounts such as 'nobody' because it has the same consequences as using
204'root' if other services use them.
205
206The 'chroot' parameter makes the process isolate itself in an empty directory
207just before switching its UID. This type of isolation (chroot) can sometimes
208be worked around on certain OS (Linux, Solaris), provided that the attacker
209has gained 'root' priviledges and has the ability to use or create a directory.
210For this reason, it's capital to use a dedicated directory and not to share one
211between several services of different nature. To make isolation more resistant,
212it's recommended to use an empty directory without any right, and to change the
213UID of the process so that it cannot do anything there.
214
215Note: in the event where such a vulnerability would be exploited, it's most
216likely that first attempts would kill the process due to 'Segmentation Fault',
217'Bus Error' or 'Illegal Instruction' signals. Eventhough it's true that
218isolating the server reduces the risks of intrusion, it's sometimes useful to
219find why a process dies, via the analysis of a 'core' file, although very rare
220(the last bug of this sort was fixed in 1.1.9). For security reasons, most
221systems disable the generation of core file when a process changes its UID. So
222the two workarounds are either to start the process from a restricted user
223account, which will not be able to chroot itself, or start it as root and not
224change the UID. In both cases the core will be either in the start or the chroot
225directories. Do not forget to allow core dumps prior to start the process :
226
227# ulimit -c unlimited
228
229Example :
230---------
231
232 global
willy tarreauc5f73ed2005-12-18 01:26:38 +0100233 uid 30000
234 gid 30000
235 chroot /var/chroot/haproxy
236
willy tarreaueedaa9f2005-12-17 14:08:03 +0100237
2381.4) Startup modes
239------------------
willy tarreau34f45302006-04-15 21:37:14 +0200240The service can start in several different modes :
willy tarreaueedaa9f2005-12-17 14:08:03 +0100241 - foreground / background
242 - quiet / normal / debug
243
244The default mode is normal, foreground, which means that the program doesn't
245return once started. NEVER EVER use this mode in a system startup script, or
246the system won't boot. It needs to be started in background, so that it
247returns immediately after forking. That's accomplished by the 'daemon' option
248in the 'global' section, which is the equivalent of the '-D' command line
249argument.
250
willy tarreau34f45302006-04-15 21:37:14 +0200251The '-db' command line argument overrides the 'daemon' and 'nbproc' global
252options to make the process run in normal, foreground mode.
253
willy tarreaueedaa9f2005-12-17 14:08:03 +0100254Moreover, certain alert messages are still sent to the standard output even
255in 'daemon' mode. To make them disappear, simply add the 'quiet' option in the
256'global' section. This option has no command-line equivalent.
257
258Last, the 'debug' mode, enabled with the 'debug' option in the 'global' section,
259and which is equivalent of the '-d' option, allows deep TCP/HTTP analysis, with
260timestamped display of each connection, disconnection, and HTTP headers for both
261ways. This mode is incompatible with 'daemon' and 'quiet' modes for obvious
262reasons.
263
willy tarreauc5f73ed2005-12-18 01:26:38 +0100264
willy tarreaueedaa9f2005-12-17 14:08:03 +01002651.5) Increasing the overall processing power
266--------------------------------------------
267On multi-processor systems, it may seem to be a shame to use only one processor,
willy tarreau982249e2005-12-18 00:57:06 +0100268eventhough the load needed to saturate a recent processor is far above common
willy tarreaueedaa9f2005-12-17 14:08:03 +0100269usage. Anyway, for very specific needs, the proxy can start several processes
270between which the operating system will spread the incoming connections. The
271number of processes is controlled by the 'nbproc' parameter in the 'global'
willy tarreau4302f492005-12-18 01:00:37 +0100272section. It defaults to 1, and obviously works only in 'daemon' mode. One
273typical usage of this parameter has been to workaround the default per-process
274file-descriptor limit that Solaris imposes to user processes.
willy tarreaueedaa9f2005-12-17 14:08:03 +0100275
276Example :
277---------
278
279 global
willy tarreauc5f73ed2005-12-18 01:26:38 +0100280 daemon
281 quiet
282 nbproc 2
willy tarreaueedaa9f2005-12-17 14:08:03 +0100283
284
willy tarreaufe2c5c12005-12-17 14:14:34 +01002851.6) Helping process management
286-------------------------------
287Haproxy now supports the notion of pidfile. If the '-p' command line argument,
288or the 'pidfile' global option is followed with a file name, this file will be
289removed, then filled with all children's pids, one per line (only in daemon
290mode). This file is NOT within the chroot, which allows to work with a readonly
291 chroot. It will be owned by the user starting the process, and will have
292permissions 0644.
293
294Example :
295---------
296
297 global
298 daemon
299 quiet
willy tarreauc5f73ed2005-12-18 01:26:38 +0100300 nbproc 2
willy tarreaufe2c5c12005-12-17 14:14:34 +0100301 pidfile /var/run/haproxy-private.pid
302
303 # to stop only those processes among others :
304 # kill $(</var/run/haproxy-private.pid)
305
willy tarreau34f45302006-04-15 21:37:14 +0200306 # to reload a new configuration with minimal service impact and without
307 # breaking existing sessions :
308 # haproxy -f haproxy.cfg -p $(</var/run/haproxy-private.pid) -st $(</var/run/haproxy-private.pid)
willy tarreaufe2c5c12005-12-17 14:14:34 +0100309
willy tarreau64a3cc32005-12-18 01:13:11 +01003101.7) Polling mechanisms
311-----------------------
312Starting from version 1.2.5, haproxy supports the poll() and epoll() polling
313mechanisms. On systems where select() is limited by FD_SETSIZE (like Solaris),
314poll() can be an interesting alternative. Performance tests show that Solaris'
315poll() performance does not decay as fast as the numbers of sockets increase,
316making it a safe solution for high loads. However, Solaris already uses poll()
317to emulate select(), so as long as the number of sockets has no reason to go
318higher than FD_SETSIZE, poll() should not provide any better performance. On
319Linux systems with the epoll() patch (or any 2.6 version), haproxy will use
320epoll() which is extremely fast and non dependant on the number of sockets.
321Tests have shown constant performance from 1 to 20000 simultaneous sessions.
322
323Haproxy will use epoll() when available, and will fall back to poll(), then to
324select(). However, if for any reason you need to disable epoll() or poll() (eg.
325because of a bug or just to compare performance), two new global options have
326been created for this matter : 'noepoll' and 'nopoll'.
327
328Example :
329---------
330
331 global
332 # use only select()
333 noepoll
334 nopoll
335
336Note :
337------
338For the sake of configuration file portability, these options are accepted but
339ignored if the poll() or epoll() mechanisms have not been enabled at compile
340time.
341
342To make debugging easier, the '-de' runtime argument disables epoll support and
343the '-dp' argument disables poll support. They are respectively equivalent to
344'noepoll' and 'nopoll'.
345
346
willy tarreaueedaa9f2005-12-17 14:08:03 +01003472) Declaration of a listening service
348=====================================
349
350Service sections start with the 'listen' keyword :
351
352 listen <instance_name> [ <IP_address>:<port_range>[,...] ]
353
354- <instance_name> is the name of the instance. This name will be reported in
355 logs, so it is good to have it reflect the proxied service. No unicity test
356 is done on this name, and it's not mandatory for it to be unique, but highly
357 recommended.
358
359- <IP_address> is the IP address the proxy binds to. Empty address, '*' and
360 '0.0.0.0' all mean that the proxy listens to all valid addresses on the
361 system.
362
363- <port_range> is either a unique port, or a port range for which the proxy will
364 accept connections for the IP address specified above. This range can be :
365 - a numerical port (ex: '80')
366 - a dash-delimited ports range explicitly stating the lower and upper bounds
367 (ex: '2000-2100') which are included in the range.
368
369 Particular care must be taken against port ranges, because every <addr:port>
370 couple consumes one socket (=a file descriptor), so it's easy to eat lots of
371 descriptors with a simple range. The <addr:port> couple must be used only once
372 among all instances running on a same system. Please note that attaching to
373 ports lower than 1024 need particular priviledges to start the program, which
374 are independant of the 'uid' parameter.
375
376- the <IP_address>:<port_range> couple may be repeated indefinitely to require
377 the proxy to listen to other addresses and/or ports. To achieve this, simply
378 separate them with a coma.
379
380Examples :
381---------
382 listen http_proxy :80
383 listen x11_proxy 127.0.0.1:6000-6009
384 listen smtp_proxy 127.0.0.1:25,127.0.0.1:587
385 listen ldap_proxy :389,:663
386
387In the event that all addresses do not fit line width, it's preferable to
388detach secondary addresses on other lines with the 'bind' keyword. If this
389keyword is used, it's not even necessary to specify the first address on the
390'listen' line, which sometimes makes multiple configuration handling easier :
391
392 bind [ <IP_address>:<port_range>[,...] ]
393
394Examples :
395----------
396 listen http_proxy
397 bind :80,:443
willy tarreauc5f73ed2005-12-18 01:26:38 +0100398 bind 10.0.0.1:10080,10.0.0.1:10443
399
willy tarreaueedaa9f2005-12-17 14:08:03 +0100400
4012.1) Inhibiting a service
402-------------------------
403A service may be disabled for maintenance reasons, without needing to comment
404out the whole section, simply by specifying the 'disabled' keyword in the
405section to be disabled :
406
407 listen smtp_proxy 0.0.0.0:25
willy tarreauc5f73ed2005-12-18 01:26:38 +0100408 disabled
willy tarreaueedaa9f2005-12-17 14:08:03 +0100409
410Note: the 'enabled' keyword allows to enable a service which has been disabled
411 previously by a default configuration.
412
willy tarreauc5f73ed2005-12-18 01:26:38 +0100413
willy tarreaueedaa9f2005-12-17 14:08:03 +01004142.2) Modes of operation
415-----------------------
416A service can work in 3 different distinct modes :
417 - TCP
418 - HTTP
419 - monitoring
420
421TCP mode
422--------
423In this mode, the service relays TCP connections as soon as they're established,
424towards one or several servers. No processing is done on the stream. It's only
425an association of source(addr:port) -> destination(addr:port). To use this mode,
426you must specify 'mode tcp' in the 'listen' section. This is the default mode.
427
428Example :
429---------
430 listen smtp_proxy 0.0.0.0:25
willy tarreauc5f73ed2005-12-18 01:26:38 +0100431 mode tcp
willy tarreaueedaa9f2005-12-17 14:08:03 +0100432
433HTTP mode
434---------
435In this mode, the service relays TCP connections towards one or several servers,
436when it has enough informations to decide, which normally means that all HTTP
437headers have been read. Some of them may be scanned for a cookie or a pattern
438matching a regex. To use this mode, specify 'mode http' in the 'listen' section.
439
440Example :
441---------
442 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +0100443 mode http
willy tarreaueedaa9f2005-12-17 14:08:03 +0100444
445Health-checking mode
446--------------------
447This mode provides a way for external components to check the proxy's health.
448It is meant to be used with intelligent load-balancers which can use send/expect
449scripts to check for all of their servers' availability. This one simply accepts
willy tarreau197e8ec2005-12-17 14:10:59 +0100450the connection, returns the word 'OK' and closes it. If the 'option httpchk' is
451set, then the reply will be 'HTTP/1.0 200 OK' with no data, so that it can be
452tested from a tool which supports HTTP health-checks. To enable it, simply
willy tarreaueedaa9f2005-12-17 14:08:03 +0100453specify 'health' as the working mode :
454
455Example :
456---------
willy tarreau197e8ec2005-12-17 14:10:59 +0100457 # simple response : 'OK'
willy tarreaueedaa9f2005-12-17 14:08:03 +0100458 listen health_check 0.0.0.0:60000
willy tarreauc5f73ed2005-12-18 01:26:38 +0100459 mode health
willy tarreaueedaa9f2005-12-17 14:08:03 +0100460
willy tarreau197e8ec2005-12-17 14:10:59 +0100461 # HTTP response : 'HTTP/1.0 200 OK'
462 listen http_health_check 0.0.0.0:60001
willy tarreauc5f73ed2005-12-18 01:26:38 +0100463 mode health
464 option httpchk
465
466Monitoring
467----------
468Versions 1.1.32 and 1.2.6 provide a new solution to check the proxy's
469availability without perturbating the service. The 'monitor-net' keyword was
470created to specify a network of equipments which CANNOT use the service for
471anything but health-checks. This is particularly suited to TCP proxies, because
472it prevents the proxy from relaying the monitor's connection to the remote
473server.
474
475When used with TCP, the connection is accepted then closed and nothing is
476logged. This is enough for a front-end load-balancer to detect the service as
477available.
willy tarreau197e8ec2005-12-17 14:10:59 +0100478
willy tarreauc5f73ed2005-12-18 01:26:38 +0100479When used with HTTP, the connection is accepted, nothing is logged, the
480following response is sent, then the session is closed : "HTTP/1.0 200 OK".
481This is normally enough for any front-end HTTP load-balancer to detect the
482service as available too, both with TCP and HTTP checks.
483
484Proxies using the "monitor-net" keyword can remove the "option dontlognull", as
485it will make them log empty connections from hosts outside the monitoring
486network.
487
488Example :
489---------
490
491 listen tse-proxy
492 bind :3389,:1494,:5900 # TSE, ICA and VNC at once.
493 mode tcp
494 balance roundrobin
495 server tse-farm 192.168.1.10
496 monitor-net 192.168.1.252/31 # L4 load-balancers on .252 and .253
497
willy tarreaueedaa9f2005-12-17 14:08:03 +0100498
4992.3) Limiting the number of simultaneous connections
500----------------------------------------------------
501The 'maxconn' parameter allows a proxy to refuse connections above a certain
502amount of simultaneous ones. When the limit is reached, it simply stops
503listening, but the system may still be accepting them because of the back log
willy tarreau982249e2005-12-18 00:57:06 +0100504queue. These connections will be processed later when other ones have freed
willy tarreaueedaa9f2005-12-17 14:08:03 +0100505some slots. This provides a serialization effect which helps very fragile
willy tarreau34f45302006-04-15 21:37:14 +0200506servers resist to high loads. See further for system limitations.
willy tarreaueedaa9f2005-12-17 14:08:03 +0100507
508Example :
509---------
510 listen tiny_server 0.0.0.0:80
511 maxconn 10
512
513
5142.4) Soft stop
515--------------
516It is possible to stop services without breaking existing connections by the
willy tarreau22739ef2006-01-20 20:43:32 +0100517sending of the SIGUSR1 signal to the process. All services are then put into
willy tarreaueedaa9f2005-12-17 14:08:03 +0100518soft-stop state, which means that they will refuse to accept new connections,
519except for those which have a non-zero value in the 'grace' parameter, in which
520case they will still accept connections for the specified amount of time, in
willy tarreau22739ef2006-01-20 20:43:32 +0100521milliseconds. This makes it possible to tell a load-balancer that the service
522is failing, while still doing the job during the time it needs to detect it.
willy tarreaueedaa9f2005-12-17 14:08:03 +0100523
524Note: active connections are never killed. In the worst case, the user will have
525to wait for all of them to close or to time-out, or simply kill the process
willy tarreau22739ef2006-01-20 20:43:32 +0100526normally (SIGTERM). The default 'grace' value is '0'.
willy tarreaueedaa9f2005-12-17 14:08:03 +0100527
528Example :
529---------
530 # enter soft stop after 'killall -USR1 haproxy'
531 # the service will still run 10 seconds after the signal
532 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +0100533 mode http
534 grace 10000
willy tarreaueedaa9f2005-12-17 14:08:03 +0100535
536 # this port is dedicated to a load-balancer, and must fail immediately
537 listen health_check 0.0.0.0:60000
willy tarreauc5f73ed2005-12-18 01:26:38 +0100538 mode health
539 grace 0
willy tarreaueedaa9f2005-12-17 14:08:03 +0100540
541
willy tarreau39df2dc2006-01-29 21:56:05 +0100542As of version 1.2.8, a new soft-reconfiguration mechanism has been introduced.
willy tarreau22739ef2006-01-20 20:43:32 +0100543It is now possible to "pause" all the proxies by sending a SIGTTOU signal to
544the processes. This will disable the listening socket without breaking existing
545connections. After that, sending a SIGTTIN signal to those processes enables
546the listening sockets again. This is very useful to try to load a new
547configuration or even a new version of haproxy without breaking existing
548connections. If the load succeeds, then simply send a SIGUSR1 which will make
549the previous proxies exit immediately once their sessions are closed ; and if
550the load fails, then simply send a SIGTTIN to restore the service immediately.
551Please note that the 'grace' parameter is ignored for SIGTTOU, as well as for
552SIGUSR1 when the process was in the pause mode. Please also note that it would
553be useful to save the pidfile before starting a new instance.
554
willy tarreau34f45302006-04-15 21:37:14 +0200555This mechanism fully exploited since 1.2.11 with the '-st' and '-sf' options
556(see above).
willy tarreau22739ef2006-01-20 20:43:32 +0100557
willy tarreaueedaa9f2005-12-17 14:08:03 +01005582.5) Connections expiration time
559--------------------------------
560It is possible (and recommended) to configure several time-outs on TCP
561connections. Three independant timers are adjustable with values specified
562in milliseconds. A session will be terminated if either one of these timers
563expire.
564
565 - the time we accept to wait for data from the client, or for the client to
566 accept data : 'clitimeout' :
567
willy tarreauc5f73ed2005-12-18 01:26:38 +0100568 # client time-out set to 2mn30.
569 clitimeout 150000
willy tarreaueedaa9f2005-12-17 14:08:03 +0100570
571 - the time we accept to wait for data from the server, or for the server to
572 accept data : 'srvtimeout' :
573
willy tarreauc5f73ed2005-12-18 01:26:38 +0100574 # server time-out set to 30s.
575 srvtimeout 30000
willy tarreaueedaa9f2005-12-17 14:08:03 +0100576
577 - the time we accept to wait for a connection to establish on a server :
578 'contimeout' :
579
580 # we give up if the connection does not complete within 4 seconds
willy tarreauc5f73ed2005-12-18 01:26:38 +0100581 contimeout 4000
willy tarreaueedaa9f2005-12-17 14:08:03 +0100582
583Notes :
584-------
585 - 'contimeout' and 'srvtimeout' have no sense on 'health' mode servers ;
586 - under high loads, or with a saturated or defective network, it's possible
587 that some packets get lost. Since the first TCP retransmit only happens
588 after 3 seconds, a time-out equal to, or lower than 3 seconds cannot
589 compensate for a packet loss. A 4 seconds time-out seems a reasonable
590 minimum which will considerably reduce connection failures.
591
willy tarreauc5f73ed2005-12-18 01:26:38 +0100592
willy tarreaueedaa9f2005-12-17 14:08:03 +01005932.6) Attempts to reconnect
594--------------------------
595After a connection failure to a server, it is possible to retry, potentially
596on another server. This is useful if health-checks are too rare and you don't
597want the clients to see the failures. The number of attempts to reconnect is
598set by the 'retries' paramter.
599
600Example :
601---------
willy tarreauc5f73ed2005-12-18 01:26:38 +0100602 # we can retry 3 times max after a failure
603 retries 3
willy tarreaueedaa9f2005-12-17 14:08:03 +0100604
willy tarreau34f45302006-04-15 21:37:14 +0200605Please note that the reconnection attempt may lead to getting the connection
606sent to a new server if the original one died between connection attempts.
607
willy tarreaueedaa9f2005-12-17 14:08:03 +0100608
6092.7) Address of the dispatch server (deprecated)
610------------------------------------------------
611The server which will be sent all new connections is defined by the 'dispatch'
612parameter, in the form <address>:<port>. It generally is dedicated to unknown
613connections and will assign them a cookie, in case of HTTP persistence mode,
614or simply is a single server in case of generic TCP proxy. This old mode is only
615provided for backwards compatibility, but doesn't allow to check remote servers
616state, and has a rather limited usage. All new setups should switch to 'balance'
617mode. The principle of the dispatcher is to be able to perform the load
618balancing itself, but work only on new clients so that the server doesn't need
619to be a big machine.
620
621Example :
622---------
willy tarreauc5f73ed2005-12-18 01:26:38 +0100623 # all new connections go there
624 dispatch 192.168.1.2:80
willy tarreaueedaa9f2005-12-17 14:08:03 +0100625
626Note :
627------
628This parameter has no sense for 'health' servers, and is incompatible with
629'balance' mode.
630
631
6322.8) Outgoing source address
633----------------------------
634It is often necessary to bind to a particular address when connecting to some
635remote hosts. This is done via the 'source' parameter which is a per-proxy
636parameter. A newer version may allow to fix different sources to reach different
637servers. The syntax is 'source <address>[:<port>]', where <address> is a valid
638local address (or '0.0.0.0' or '*' or empty to let the system choose), and
639<port> is an optional parameter allowing the user to force the source port for
640very specific needs. If the port is not specified or is '0', the system will
641choose a free port. Note that as of version 1.1.18, the servers health checks
642are also performed from the same source.
643
644Examples :
645----------
646 listen http_proxy *:80
willy tarreauc5f73ed2005-12-18 01:26:38 +0100647 # all connections take 192.168.1.200 as source address
648 source 192.168.1.200:0
willy tarreaueedaa9f2005-12-17 14:08:03 +0100649
650 listen rlogin_proxy *:513
willy tarreauc5f73ed2005-12-18 01:26:38 +0100651 # use address 192.168.1.200 and the reserved port 900 (needs to be root)
652 source 192.168.1.200:900
willy tarreaueedaa9f2005-12-17 14:08:03 +0100653
654
6552.9) Setting the cookie name
656----------------------------
657In HTTP mode, it is possible to look for a particular cookie which will contain
658a server identifier which should handle the connection. The cookie name is set
659via the 'cookie' parameter.
660
661Example :
662---------
663 listen http_proxy :80
willy tarreauc5f73ed2005-12-18 01:26:38 +0100664 mode http
665 cookie SERVERID
willy tarreaueedaa9f2005-12-17 14:08:03 +0100666
667It is possible to change the cookie behaviour to get a smarter persistence,
668depending on applications. It is notably possible to delete or modify a cookie
669emitted by a server, insert a cookie identifying the server in an HTTP response
670and even add a header to tell upstream caches not to cache this response.
671
672Examples :
673----------
674
675To remove the cookie for direct accesses (ie when the server matches the one
676which was specified in the client cookie) :
677
willy tarreauc5f73ed2005-12-18 01:26:38 +0100678 cookie SERVERID indirect
willy tarreaueedaa9f2005-12-17 14:08:03 +0100679
680To replace the cookie value with the one assigned to the server if any (no
681cookie will be created if the server does not provide one, nor if the
682configuration does not provide one). This lets the application put the cookie
683exactly on certain pages (eg: successful authentication) :
684
willy tarreauc5f73ed2005-12-18 01:26:38 +0100685 cookie SERVERID rewrite
willy tarreaueedaa9f2005-12-17 14:08:03 +0100686
687To create a new cookie and assign the server identifier to it (in this case, all
688servers should be associated with a valid cookie, since no cookie will simply
689delete the cookie from the client's browser) :
690
willy tarreauc5f73ed2005-12-18 01:26:38 +0100691 cookie SERVERID insert
willy tarreaueedaa9f2005-12-17 14:08:03 +0100692
willy tarreau0174f312005-12-18 01:02:42 +0100693To reuse an existing application cookie and prefix it with the server's
694identifier, and remove it in the request, use the 'prefix' option. This allows
695to insert a haproxy in front of an application without risking to break clients
696which does not support more than one cookie :
697
willy tarreauc5f73ed2005-12-18 01:26:38 +0100698 cookie JSESSIONID prefix
willy tarreau0174f312005-12-18 01:02:42 +0100699
willy tarreaueedaa9f2005-12-17 14:08:03 +0100700To insert a cookie and ensure that no upstream cache will store it, add the
701'nocache' option :
702
willy tarreauc5f73ed2005-12-18 01:26:38 +0100703 cookie SERVERID insert nocache
willy tarreaueedaa9f2005-12-17 14:08:03 +0100704
705To insert a cookie only after a POST request, add 'postonly' after 'insert'.
706This has the advantage that there's no risk of caching, and that all pages
707seen before the POST one can still be cached :
708
willy tarreauc5f73ed2005-12-18 01:26:38 +0100709 cookie SERVERID insert postonly
willy tarreaueedaa9f2005-12-17 14:08:03 +0100710
711Notes :
712-----------
713- it is possible to combine 'insert' with 'indirect' or 'rewrite' to adapt to
714 applications which already generate the cookie with an invalid content.
715
716- in the case where 'insert' and 'indirect' are both specified, the cookie is
willy tarreau0174f312005-12-18 01:02:42 +0100717 never transmitted to the server, since it wouldn't understand it. This is the
718 most application-transparent mode.
willy tarreaueedaa9f2005-12-17 14:08:03 +0100719
720- it is particularly recommended to use 'nocache' in 'insert' mode if any
721 upstream HTTP/1.0 cache is susceptible to cache the result, because this may
722 lead to many clients going to the same server, or even worse, some clients
723 having their server changed while retrieving a page from the cache.
724
willy tarreau0174f312005-12-18 01:02:42 +0100725- the 'prefix' mode normally does not need 'indirect', 'nocache', nor
726 'postonly', because just as in the 'rewrite' mode, it relies on the
727 application to know when a cookie can be emitted. However, since it has to
728 fix the cookie name in every subsequent requests, you must ensure that the
729 proxy will be used without any "HTTP keep-alive". Use option "httpclose" if
730 unsure.
731
willy tarreaueedaa9f2005-12-17 14:08:03 +0100732- when the application is well known and controlled, the best method is to
733 only add the persistence cookie on a POST form because it's up to the
willy tarreau0174f312005-12-18 01:02:42 +0100734 application to select which page it wants the upstream servers to cache. In
735 this case, you would use 'insert postonly indirect'.
willy tarreaueedaa9f2005-12-17 14:08:03 +0100736
willy tarreauc5f73ed2005-12-18 01:26:38 +0100737
willy tarreaueedaa9f2005-12-17 14:08:03 +01007382.10) Associating a cookie value with a server
739----------------------------------------------
740In HTTP mode, it's possible to associate a cookie value to each server. This
741was initially used in combination with 'dispatch' mode to handle direct accesses
742but it is now the standard way of doing the load balancing. The syntax is :
743
744 server <identifier> <address>:<port> cookie <value>
745
746- <identifier> is any name which can be used to identify the server in the logs.
747- <address>:<port> specifies where the server is bound.
748- <value> is the value to put in or to read from the cookie.
749
750Example : the 'SERVERID' cookie can be either 'server01' or 'server02'
751---------
752 listen http_proxy :80
willy tarreauc5f73ed2005-12-18 01:26:38 +0100753 mode http
754 cookie SERVERID
755 dispatch 192.168.1.100:80
756 server web1 192.168.1.1:80 cookie server01
757 server web2 192.168.1.2:80 cookie server02
willy tarreaueedaa9f2005-12-17 14:08:03 +0100758
759Warning : the syntax has changed since version 1.0 !
760---------
761
willy tarreauc5f73ed2005-12-18 01:26:38 +0100762
willy tarreau598da412005-12-18 01:07:29 +01007632.11) Application Cookies
764-------------------------
765Since 1.2.4 it is possible to catch the cookie that comes from an
766application server in order to apply "application session stickyness".
767The server's response is searched for 'appsession' cookie, the first
768'len' bytes are used for matching and it is stored for a period of
769'timeout'.
770The syntax is:
771
772 appsession <SESSIONCOOKIE> len <MATCHLENGTH> timeout <HOLDTIME>
773
774- <SESSIONCOOKIE> is the cookie, the server uses for it's session-handling
775- <MATCHLENGTH> how many bytes/characters should be used for matching equal
776 sessions
777- <HOLDTIME> after this inactivaty time, in ms, the cookie will be deleted
778 from the sessionstore
779
780The appsession is only per 'listen' section possible.
781
782Example :
783---------
784 listen http_proxy :80
willy tarreauc5f73ed2005-12-18 01:26:38 +0100785 mode http
willy tarreau598da412005-12-18 01:07:29 +0100786 appsession JSESSIONID len 52 timeout 300000
787 .
788 .
789
willy tarreauc5f73ed2005-12-18 01:26:38 +0100790
willy tarreaueedaa9f2005-12-17 14:08:03 +01007913) Autonomous load balancer
792===========================
793
794The proxy can perform the load-balancing itself, both in TCP and in HTTP modes.
795This is the most interesting mode which obsoletes the old 'dispatch' mode
796described above. It has advantages such as server health monitoring, multiple
797port binding and port mapping. To use this mode, the 'balance' keyword is used,
willy tarreau34f45302006-04-15 21:37:14 +0200798followed by the selected algorithm. Up to version 1.2.11, only 'roundrobin' was
799available, which is also the default value if unspecified. Starting with
800version 1.2.12, a new 'source' keyword appeared. In this mode, there will be no
801dispatch address, but the proxy needs at least one server.
willy tarreaueedaa9f2005-12-17 14:08:03 +0100802
803Example : same as the last one, with internal load balancer
804---------
805
806 listen http_proxy :80
willy tarreauc5f73ed2005-12-18 01:26:38 +0100807 mode http
808 cookie SERVERID
809 balance roundrobin
810 server web1 192.168.1.1:80 cookie server01
811 server web2 192.168.1.2:80 cookie server02
willy tarreaueedaa9f2005-12-17 14:08:03 +0100812
813
814Since version 1.1.22, it is possible to automatically determine on which port
815the server will get the connection, depending on the port the client connected
816to. Indeed, there now are 4 possible combinations for the server's <port> field:
817
818 - unspecified or '0' :
819 the connection will be sent to the same port as the one on which the proxy
820 received the client connection itself.
821
822 - numerical value (the only one supported in versions earlier than 1.1.22) :
823 the connection will always be sent to the specified port.
824
825 - '+' followed by a numerical value :
826 the connection will be sent to the same port as the one on which the proxy
827 received the connection, plus this value.
828
829 - '-' followed by a numerical value :
830 the connection will be sent to the same port as the one on which the proxy
831 received the connection, minus this value.
832
833Examples :
834----------
835
836# same as previous example
837
838 listen http_proxy :80
willy tarreauc5f73ed2005-12-18 01:26:38 +0100839 mode http
840 cookie SERVERID
841 balance roundrobin
842 server web1 192.168.1.1 cookie server01
843 server web2 192.168.1.2 cookie server02
willy tarreaueedaa9f2005-12-17 14:08:03 +0100844
845# simultaneous relaying of ports 80, 81 and 8080-8089
846
847 listen http_proxy :80,:81,:8080-8089
willy tarreauc5f73ed2005-12-18 01:26:38 +0100848 mode http
849 cookie SERVERID
850 balance roundrobin
851 server web1 192.168.1.1 cookie server01
852 server web2 192.168.1.2 cookie server02
willy tarreaueedaa9f2005-12-17 14:08:03 +0100853
854# relaying of TCP ports 25, 389 and 663 to ports 1025, 1389 and 1663
855
856 listen http_proxy :25,:389,:663
willy tarreauc5f73ed2005-12-18 01:26:38 +0100857 mode tcp
858 balance roundrobin
859 server srv1 192.168.1.1:+1000
860 server srv2 192.168.1.2:+1000
willy tarreaueedaa9f2005-12-17 14:08:03 +0100861
willy tarreau34f45302006-04-15 21:37:14 +0200862As previously stated, version 1.2.12 brought the 'source' keyword. When this
863keyword is used, the client's IP address is hashed and evenly distributed among
864the available servers so that a same source IP will always go to the same
865server as long as there are no change in the number of available servers. This
866can be used for instance to bind HTTP and HTTPS to the same server. It can also
867be used to improve stickyness when one part of the client population does not
868accept cookies. In this case, only those ones will be perturbated should a
869server fail.
870
871NOTE: It is important to consider the fact that many clients surf the net
872 through proxy farms which assign different IP addresses for each
873 request. Others use dialup connections with a different IP at each
874 connection. Thus, the 'source' parameter should be used with extreme
875 care.
876
877Examples :
878----------
879
880# make a same IP go to the same server whatever the service
881
882 listen http_proxy
883 bind :80,:443
884 mode http
885 balance source
886 server web1 192.168.1.1
887 server web2 192.168.1.2
888
889# try to improve client-server binding by using both source IP and cookie :
890
891 listen http_proxy :80
892 mode http
893 cookie SERVERID
894 balance source
895 server web1 192.168.1.1 cookie server01
896 server web2 192.168.1.2 cookie server02
897
willy tarreaueedaa9f2005-12-17 14:08:03 +0100898
willy tarreau197e8ec2005-12-17 14:10:59 +01008993.1) Server monitoring
900----------------------
willy tarreaueedaa9f2005-12-17 14:08:03 +0100901It is possible to check the servers status by trying to establish TCP
902connections or even sending HTTP requests to them. A server which fails to
903reply to health checks as expected will not be used by the load balancing
904algorithms. To enable monitoring, add the 'check' keyword on a server line.
905It is possible to specify the interval between tests (in milliseconds) with
906the 'inter' parameter, the number of failures supported before declaring that
907the server has fallen down with the 'fall' parameter, and the number of valid
908checks needed for the server to fully get up with the 'rise' parameter. Since
909version 1.1.22, it is also possible to send checks to a different port
910(mandatory when none is specified) with the 'port' parameter. The default
911values are the following ones :
912
913 - inter : 2000
914 - rise : 2
915 - fall : 3
916 - port : default server port
917
918The default mode consists in establishing TCP connections only. But in certain
919types of application failures, it is often that the server continues to accept
920connections because the system does it itself while the application is running
921an endless loop, or is completely stuck. So in version 1.1.16 were introduced
922HTTP health checks which only performed simple lightweight requests and analysed
923the response. Now, as of version 1.1.23, it is possible to change the HTTP
924method, the URI, and the HTTP version string (which even allows to send headers
925with a dirty trick). To enable HTTP health-checks, use 'option httpchk'.
926
927By default, requests use the 'OPTIONS' method because it's very light and easy
928to filter from logs, and does it on '/'. Only HTTP responses 2xx and 3xx are
929considered valid ones, and only if they come before the time to send a new
930request is reached ('inter' parameter). If some servers block this type of
931request, 3 other forms help to forge a request :
932
933 - option httpchk -> OPTIONS / HTTP/1.0
934 - option httpchk URI -> OPTIONS <URI> HTTP/1.0
935 - option httpchk METH URI -> <METH> <URI> HTTP/1.0
936 - option httpchk METH URI VER -> <METH> <URI> <VER>
937
938See examples below.
939
940Since version 1.1.17, it is possible to specify backup servers. These servers
941are only sollicited when no other server is available. This may only be useful
942to serve a maintenance page, or define one active and one backup server (seldom
943used in TCP mode). To make a server a backup one, simply add the 'backup' option
944on its line. These servers also support cookies, so if a cookie is specified for
945a backup server, clients assigned to this server will stick to it even when the
946other ones come back. Conversely, if no cookie is assigned to such a server,
947the clients will get their cookies removed (empty cookie = removal), and will
948be balanced against other servers once they come back. Please note that there
Willy TARREAU3481c462006-03-01 22:37:57 +0100949is no load-balancing among backup servers by default. If there are several
950backup servers, the second one will only be used when the first one dies, and
951so on. To force load-balancing between backup servers, specify the 'allbackups'
952option.
willy tarreaueedaa9f2005-12-17 14:08:03 +0100953
954Since version 1.1.17, it is also possible to visually check the status of all
955servers at once. For this, you just have to send a SIGHUP signal to the proxy.
956The servers status will be dumped into the logs at the 'notice' level, as well
957as on <stderr> if not closed. For this reason, it's always a good idea to have
958one local log server at the 'notice' level.
959
willy tarreau982249e2005-12-18 00:57:06 +0100960Since version 1.1.28 and 1.2.1, if an instance loses all its servers, an
willy tarreau0174f312005-12-18 01:02:42 +0100961emergency message will be sent in the logs to inform the administator that an
willy tarreau982249e2005-12-18 00:57:06 +0100962immediate action must be taken.
963
willy tarreau0174f312005-12-18 01:02:42 +0100964Since version 1.1.30 and 1.2.3, several servers can share the same cookie
965value. This is particularly useful in backup mode, to select alternate paths
966for a given server for example, to provide soft-stop, or to direct the clients
967to a temporary page during an application restart. The principle is that when
968a server is dead, the proxy will first look for another server which shares the
969same cookie value for every client which presents the cookie. If there is no
970standard server for this cookie, it will then look for a backup server which
971shares the same name. Please consult the architecture guide for more information.
willy tarreau982249e2005-12-18 00:57:06 +0100972
willy tarreaueedaa9f2005-12-17 14:08:03 +0100973Examples :
974----------
975# same setup as in paragraph 3) with TCP monitoring
976 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +0100977 mode http
978 cookie SERVERID
979 balance roundrobin
980 server web1 192.168.1.1:80 cookie server01 check
981 server web2 192.168.1.2:80 cookie server02 check inter 500 rise 1 fall 2
willy tarreaueedaa9f2005-12-17 14:08:03 +0100982
983# same with HTTP monitoring via 'OPTIONS / HTTP/1.0'
984 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +0100985 mode http
986 cookie SERVERID
987 balance roundrobin
988 option httpchk
989 server web1 192.168.1.1:80 cookie server01 check
990 server web2 192.168.1.2:80 cookie server02 check inter 500 rise 1 fall 2
willy tarreaueedaa9f2005-12-17 14:08:03 +0100991
992# same with HTTP monitoring via 'OPTIONS /index.html HTTP/1.0'
993 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +0100994 mode http
995 cookie SERVERID
996 balance roundrobin
997 option httpchk /index.html
998 server web1 192.168.1.1:80 cookie server01 check
999 server web2 192.168.1.2:80 cookie server02 check inter 500 rise 1 fall 2
willy tarreaueedaa9f2005-12-17 14:08:03 +01001000
1001# same with HTTP monitoring via 'HEAD /index.jsp? HTTP/1.1\r\nHost: www'
1002 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001003 mode http
1004 cookie SERVERID
1005 balance roundrobin
1006 option httpchk HEAD /index.jsp? HTTP/1.1\r\nHost:\ www
1007 server web1 192.168.1.1:80 cookie server01 check
1008 server web2 192.168.1.2:80 cookie server02 check inter 500 rise 1 fall 2
willy tarreaueedaa9f2005-12-17 14:08:03 +01001009
willy tarreau0174f312005-12-18 01:02:42 +01001010# Load-balancing with 'prefixed cookie' persistence, and soft-stop using an
1011# alternate port 81 on the server for health-checks.
1012 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001013 mode http
1014 cookie JSESSIONID prefix
1015 balance roundrobin
1016 option httpchk HEAD /index.jsp? HTTP/1.1\r\nHost:\ www
1017 server web1-norm 192.168.1.1:80 cookie s1 check port 81
1018 server web2-norm 192.168.1.2:80 cookie s2 check port 81
1019 server web1-stop 192.168.1.1:80 cookie s1 check port 80 backup
1020 server web2-stop 192.168.1.2:80 cookie s2 check port 80 backup
willy tarreau0174f312005-12-18 01:02:42 +01001021
willy tarreaueedaa9f2005-12-17 14:08:03 +01001022# automatic insertion of a cookie in the server's response, and automatic
1023# deletion of the cookie in the client request, while asking upstream caches
1024# not to cache replies.
1025 listen web_appl 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001026 mode http
1027 cookie SERVERID insert nocache indirect
1028 balance roundrobin
1029 server web1 192.168.1.1:80 cookie server01 check
1030 server web2 192.168.1.2:80 cookie server02 check
willy tarreaueedaa9f2005-12-17 14:08:03 +01001031
1032# same with off-site application backup and local error pages server
1033 listen web_appl 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001034 mode http
1035 cookie SERVERID insert nocache indirect
1036 balance roundrobin
1037 server web1 192.168.1.1:80 cookie server01 check
1038 server web2 192.168.1.2:80 cookie server02 check
1039 server web-backup 192.168.2.1:80 cookie server03 check backup
1040 server web-excuse 192.168.3.1:80 check backup
willy tarreaueedaa9f2005-12-17 14:08:03 +01001041
willy tarreauc5f73ed2005-12-18 01:26:38 +01001042# SMTP+TLS relaying with health-checks and backup servers
willy tarreaueedaa9f2005-12-17 14:08:03 +01001043
1044 listen http_proxy :25,:587
willy tarreauc5f73ed2005-12-18 01:26:38 +01001045 mode tcp
1046 balance roundrobin
1047 server srv1 192.168.1.1 check port 25 inter 30000 rise 1 fall 2
1048 server srv2 192.168.1.2 backup
willy tarreaueedaa9f2005-12-17 14:08:03 +01001049
Willy TARREAU3481c462006-03-01 22:37:57 +01001050# Load-balancing using a backup pool (requires haproxy 1.2.9)
1051 listen http_proxy 0.0.0.0:80
1052 mode http
1053 balance roundrobin
1054 option httpchk
1055 server inst1 192.168.1.1:80 cookie s1 check
1056 server inst2 192.168.1.2:80 cookie s2 check
1057 server inst3 192.168.1.3:80 cookie s3 check
1058 server back1 192.168.1.10:80 check backup
1059 server back2 192.168.1.11:80 check backup
1060 option allbackups # all backups will be used
1061
willy tarreaueedaa9f2005-12-17 14:08:03 +01001062
10633.2) Redistribute connections in case of failure
1064------------------------------------------------
1065In HTTP mode, if a server designated by a cookie does not respond, the clients
1066may definitely stick to it because they cannot flush the cookie, so they will
1067not be able to access the service anymore. Specifying 'redispatch' will allow
1068the proxy to break their persistence and redistribute them to working servers.
1069
1070Example :
1071---------
1072 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001073 mode http
1074 cookie SERVERID
1075 dispatch 192.168.1.100:80
1076 server web1 192.168.1.1:80 cookie server01
1077 server web2 192.168.1.2:80 cookie server02
1078 redispatch # send back to dispatch in case of connection failure
willy tarreaueedaa9f2005-12-17 14:08:03 +01001079
1080Up to, and including version 1.1.16, this parameter only applied to connection
1081failures. Since version 1.1.17, it also applies to servers which have been
1082detected as failed by the health check mechanism. Indeed, a server may be broken
1083but still accepting connections, which would not solve every case. But it is
1084possible to conserve the old behaviour, that is, make a client insist on trying
1085to connect to a server even if it is said to be down, by setting the 'persist'
1086option :
1087
1088 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001089 mode http
1090 option persist
1091 cookie SERVERID
1092 dispatch 192.168.1.100:80
1093 server web1 192.168.1.1:80 cookie server01
1094 server web2 192.168.1.2:80 cookie server02
1095 redispatch # send back to dispatch in case of connection failure
willy tarreaueedaa9f2005-12-17 14:08:03 +01001096
1097
willy tarreau34f45302006-04-15 21:37:14 +020010983.3) Assigning different weights to servers
1099-------------------------------------------
1100Sometimes you will need to bring new servers to increase your server farm's
1101capacity, but the new server will be either smaller (emergency use of anything
1102that fits) or bigger (when investing in new hardware). For this reason, it
1103might be wise to be able to send more clients to biggest servers. Till version
11041.2.11, it was necessary to replicate the same server multiple times in the
1105configuration. Starting with 1.2.12, the 'weight' option is available. HAProxy
1106then computes the most homogenous possible map of servers based on their
1107weights so that the load gets distributed as smoothly as possible among
1108them. The weight, between 1 and 256, should reflect one server's capacity
1109relative to others. This way, if a server fails, the remaining capacities are
1110still respected.
1111
1112Example :
1113---------
1114# fair distribution among two opterons and one old pentium3
1115
1116 listen web_appl 0.0.0.0:80
1117 mode http
1118 cookie SERVERID insert nocache indirect
1119 balance roundrobin
1120 server pentium3-800 192.168.1.1:80 cookie server01 weight 8 check
1121 server opteron-2.0G 192.168.1.2:80 cookie server02 weight 20 check
1122 server opteron-2.4G 192.168.1.3:80 cookie server03 weight 24 check
1123 server web-backup1 192.168.2.1:80 cookie server04 check backup
1124 server web-excuse 192.168.3.1:80 check backup
1125
1126Notes :
1127-------
1128 - if unspecified, the default weight is 1
1129
1130 - the weight does not impact health checks, so it is cleaner to use weights
1131 than replicating the same server several times
1132
1133 - weights also work on backup servers if the 'allbackups' option is used
1134
1135 - the weights also apply to the source address load balancing
1136 ('balance source').
1137
1138 - whatever the weights, the first server will always be assigned first. This
1139 is helpful for troubleshooting.
1140
1141 - for the purists, the map calculation algorithm gives precedence to first
1142 server, so the map is the most uniform when servers are declared in
1143 ascending order relative to their weights.
1144
1145
willy tarreaueedaa9f2005-12-17 14:08:03 +010011464) Additionnal features
1147=======================
1148
1149Other features are available. They are transparent mode, event logging and
1150header rewriting/filtering.
1151
willy tarreauc5f73ed2005-12-18 01:26:38 +01001152
willy tarreau0174f312005-12-18 01:02:42 +010011534.1) Network features
willy tarreaueedaa9f2005-12-17 14:08:03 +01001154---------------------
willy tarreau0174f312005-12-18 01:02:42 +010011554.1.1) Transparent mode
1156-----------------------
willy tarreaueedaa9f2005-12-17 14:08:03 +01001157In HTTP mode, the 'transparent' keyword allows to intercept sessions which are
1158routed through the system hosting the proxy. This mode was implemented as a
1159replacement for the 'dispatch' mode, since connections without cookie will be
1160sent to the original address while known cookies will be sent to the servers.
1161This mode implies that the system can redirect sessions to a local port.
1162
1163Example :
1164---------
1165 listen http_proxy 0.0.0.0:65000
willy tarreauc5f73ed2005-12-18 01:26:38 +01001166 mode http
1167 transparent
1168 cookie SERVERID
1169 server server01 192.168.1.1:80
1170 server server02 192.168.1.2:80
willy tarreaueedaa9f2005-12-17 14:08:03 +01001171
1172 # iptables -t nat -A PREROUTING -i eth0 -p tcp -d 192.168.1.100 \
1173 --dport 80 -j REDIRECT --to-ports 65000
1174
1175Note :
1176------
1177If the port is left unspecified on the server, the port the client connected to
1178will be used. This allows to relay a full port range without using transparent
1179mode nor thousands of file descriptors, provided that the system can redirect
1180sessions to local ports.
1181
1182Example :
1183---------
1184 # redirect all ports to local port 65000, then forward to the server on the
1185 # original port.
1186 listen http_proxy 0.0.0.0:65000
willy tarreauc5f73ed2005-12-18 01:26:38 +01001187 mode tcp
1188 server server01 192.168.1.1 check port 60000
1189 server server02 192.168.1.2 check port 60000
willy tarreaueedaa9f2005-12-17 14:08:03 +01001190
1191 # iptables -t nat -A PREROUTING -i eth0 -p tcp -d 192.168.1.100 \
1192 -j REDIRECT --to-ports 65000
1193
willy tarreau0174f312005-12-18 01:02:42 +010011944.1.2) Per-server source address binding
1195----------------------------------------
1196As of versions 1.1.30 and 1.2.3, it is possible to specify a particular source
1197to reach each server. This is useful when reaching backup servers from a
1198different LAN, or to use an alternate path to reach the same server. It is also
1199usable to provide source load-balancing for outgoing connections. Obviously,
1200the same source address is used to send health-checks.
1201
1202Example :
1203---------
1204 # use a particular source to reach both servers
1205 listen http_proxy 0.0.0.0:65000
willy tarreauc5f73ed2005-12-18 01:26:38 +01001206 mode http
1207 balance roundrobin
1208 server server01 192.168.1.1:80 source 192.168.2.13
1209 server server02 192.168.1.2:80 source 192.168.2.13
willy tarreau0174f312005-12-18 01:02:42 +01001210
1211Example :
1212---------
1213 # use a particular source to reach each servers
1214 listen http_proxy 0.0.0.0:65000
willy tarreauc5f73ed2005-12-18 01:26:38 +01001215 mode http
1216 balance roundrobin
1217 server server01 192.168.1.1:80 source 192.168.1.1
1218 server server02 192.168.2.1:80 source 192.168.2.1
willy tarreau0174f312005-12-18 01:02:42 +01001219
1220Example :
1221---------
1222 # provide source load-balancing to reach the same proxy through 2 WAN links
1223 listen http_proxy 0.0.0.0:65000
willy tarreauc5f73ed2005-12-18 01:26:38 +01001224 mode http
1225 balance roundrobin
1226 server remote-proxy-way1 192.168.1.1:3128 source 192.168.2.1
1227 server remote-proxy-way2 192.168.1.1:3128 source 192.168.3.1
willy tarreau0174f312005-12-18 01:02:42 +01001228
1229Example :
1230---------
1231 # force a TCP connection to bind to a specific port
1232 listen http_proxy 0.0.0.0:2000
willy tarreauc5f73ed2005-12-18 01:26:38 +01001233 mode tcp
1234 balance roundrobin
1235 server srv1 192.168.1.1:80 source 192.168.2.1:20
1236 server srv2 192.168.1.2:80 source 192.168.2.1:20
willy tarreau0174f312005-12-18 01:02:42 +01001237
willy tarreaub952e1d2005-12-18 01:31:20 +010012384.1.3) TCP keep-alive
1239---------------------
1240With version 1.2.7, it becomes possible to enable TCP keep-alives on both the
1241client and server sides. This makes it possible to prevent long sessions from
1242expiring on external layer 4 components such as firewalls and load-balancers.
1243It also allows the system to terminate dead sessions when no timeout has been
1244set (not recommanded). The proxy cannot set the keep-alive probes intervals nor
1245maximal count, consult your operating system manual for this. There are 3
1246options to enable TCP keep-alive :
1247
1248 option tcpka # enables keep-alive both on client and server side
1249 option clitcpka # enables keep-alive only on client side
1250 option srvtcpka # enables keep-alive only on server side
1251
willy tarreaueedaa9f2005-12-17 14:08:03 +01001252
12534.2) Event logging
1254------------------
willy tarreauc5f73ed2005-12-18 01:26:38 +01001255
1256HAProxy's strength certainly lies in its precise logs. It probably provides the
1257finest level of information available for such a product, which is very
1258important for troubleshooting complex environments. Standard log information
1259include client ports, TCP/HTTP state timers, precise session state at
1260termination and precise termination cause, information about decisions to
1261direct trafic to a server, and of course the ability to capture arbitrary
1262headers.
1263
1264In order to improve administrators reactivity, it offers a great transparency
1265about encountered problems, both internal and external, and it is possible to
1266send logs to different sources at the same time with different level filters :
1267
1268 - global process-level logs (system errors, start/stop, etc..)
1269 - per-listener system and internal errors (lack of resource, bugs, ...)
1270 - per-listener external troubles (servers up/down, max connections)
1271 - per-listener activity (client connections), either at the establishment or
1272 at the termination.
1273
1274The ability to distribute different levels of logs to different log servers
1275allow several production teams to interact and to fix their problems as soon
1276as possible. For example, the system team might monitor system-wide errors,
1277while the application team might be monitoring the up/down for their servers in
1278real time, and the security team might analyze the activity logs with one hour
1279delay.
1280
willy tarreauc1cae632005-12-17 14:12:23 +010012814.2.1) Log levels
1282-----------------
willy tarreau197e8ec2005-12-17 14:10:59 +01001283TCP and HTTP connections can be logged with informations such as date, time,
1284source IP address, destination address, connection duration, response times,
1285HTTP request, the HTTP return code, number of bytes transmitted, the conditions
1286in which the session ended, and even exchanged cookies values, to track a
1287particular user's problems for example. All messages are sent to up to two
1288syslog servers. Consult section 1.1 for more info about log facilities. The
1289syntax follows :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001290
willy tarreau197e8ec2005-12-17 14:10:59 +01001291 log <address_1> <facility_1> [max_level_1]
1292 log <address_2> <facility_2> [max_level_2]
1293or
willy tarreaueedaa9f2005-12-17 14:08:03 +01001294 log global
1295
willy tarreau197e8ec2005-12-17 14:10:59 +01001296Note :
1297------
1298The particular syntax 'log global' means that the same log configuration as the
1299'global' section will be used.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001300
willy tarreau197e8ec2005-12-17 14:10:59 +01001301Example :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001302---------
1303 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001304 mode http
1305 log 192.168.2.200 local3
1306 log 192.168.2.201 local4
willy tarreaueedaa9f2005-12-17 14:08:03 +01001307
willy tarreauc1cae632005-12-17 14:12:23 +010013084.2.2) Log format
1309-----------------
1310By default, connections are logged at the TCP level, as soon as the session
1311establishes between the client and the proxy. By enabling the 'tcplog' option,
1312the proxy will wait until the session ends to generate an enhanced log
1313containing more information such as session duration and its state during the
1314disconnection.
1315
willy tarreauc5f73ed2005-12-18 01:26:38 +01001316Example of TCP logging :
1317------------------------
willy tarreau982249e2005-12-18 00:57:06 +01001318 listen relais-tcp 0.0.0.0:8000
willy tarreauc5f73ed2005-12-18 01:26:38 +01001319 mode tcp
1320 option tcplog
1321 log 192.168.2.200 local3
willy tarreau982249e2005-12-18 00:57:06 +01001322
willy tarreauc5f73ed2005-12-18 01:26:38 +01001323>>> haproxy[18989]: 127.0.0.1:34550 [15/Oct/2003:15:24:28] relais-tcp Srv1 0/5007 0 -- 1/1
1324
1325 Field Format Example
1326
1327 1 process_name '[' pid ']:' haproxy[18989]:
1328 2 client_ip ':' client_port 127.0.0.1:34550
1329 3 '[' date ']' [15/Oct/2003:15:24:28]
1330 4 listener_name relais-tcp
1331 5 server_name Srv1
1332 6 connect_time '/' total_time 0/5007
1333 7 bytes_read 0
1334 8 termination_state --
1335 9 listener_conns '/' process_conns 1/1
1336
willy tarreau982249e2005-12-18 00:57:06 +01001337
willy tarreauc1cae632005-12-17 14:12:23 +01001338Another option, 'httplog', provides more detailed information about HTTP
1339contents, such as the request and some cookies. In the event where an external
1340component would establish frequent connections to check the service, logs may be
1341full of useless lines. So it is possible not to log any session which didn't
1342transfer any data, by the setting of the 'dontlognull' option. This only has
1343effect on sessions which are established then closed.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001344
willy tarreauc5f73ed2005-12-18 01:26:38 +01001345Example of HTTP logging :
1346-------------------------
willy tarreaueedaa9f2005-12-17 14:08:03 +01001347 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001348 mode http
1349 option httplog
1350 option dontlognull
1351 log 192.168.2.200 local3
1352
1353>>> haproxy[674]: 127.0.0.1:33319 [15/Oct/2003:08:31:57] relais-http Srv1 9/7/147/723 200 243 - - ---- 3/3 "HEAD / HTTP/1.0"
willy tarreaueedaa9f2005-12-17 14:08:03 +01001354
willy tarreauc5f73ed2005-12-18 01:26:38 +01001355More complete example
1356 haproxy[18989]: 10.0.0.1:34552 [15/Oct/2003:15:26:31] relais-http Srv1 3183/-1/-1/11215 503 0 - - SC-- 202/205 {w.ods.org|Mozilla} {} "HEAD / HTTP/1.0"
1357
1358 Field Format Example
1359
1360 1 process_name '[' pid ']:' haproxy[18989]:
1361 2 client_ip ':' client_port 10.0.0.1:34552
1362 3 '[' date ']' [15/Oct/2003:15:26:31]
1363 4 listener_name relais-http
1364 5 server_name Srv1
1365 6 Tq '/' Tc '/' Tr '/' Tt 3183/-1/-1/11215
1366 7 HTTP_return_code 503
1367 8 bytes_read 0
1368 9 captured_request_cookie -
1369 10 captured_response_cookie -
1370 11 termination_state SC--
1371 12 listener_conns '/' process_conns 202/205
1372 13 '{' captured_request_headers '}' {w.ods.org|Mozilla}
1373 14 '{' captured_response_headers '}' {}
1374 15 '"' HTTP_request '"' "HEAD / HTTP/1.0"
1375
1376Note for log parsers: the URI is ALWAYS the end of the line starting with the
1377 first double quote '"'.
willy tarreau982249e2005-12-18 00:57:06 +01001378
1379The problem when logging at end of connection is that you have no clue about
1380what is happening during very long sessions. To workaround this problem, a
1381new option 'logasap' has been introduced in 1.1.28/1.2.1. When specified, the
1382proxy will log as soon as possible, just before data transfer begins. This means
1383that in case of TCP, it will still log the connection status to the server, and
1384in case of HTTP, it will log just after processing the server headers. In this
1385case, the number of bytes reported is the number of header bytes sent to the
1386client.
1387
1388In order to avoid confusion with normal logs, the total time field and the
1389number of bytes are prefixed with a '+' sign which mean that real numbers are
1390certainly bigger.
1391
1392Example :
1393---------
1394
1395 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001396 mode http
1397 option httplog
1398 option dontlognull
1399 option logasap
1400 log 192.168.2.200 local3
willy tarreau982249e2005-12-18 00:57:06 +01001401
willy tarreauc5f73ed2005-12-18 01:26:38 +01001402>>> haproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17] relais-http Srv1 9/7/14/+30 200 +243 - - ---- 3/3 "GET /image.iso HTTP/1.0"
willy tarreau982249e2005-12-18 00:57:06 +01001403
willy tarreauc1cae632005-12-17 14:12:23 +010014044.2.3) Timing events
1405--------------------
1406Timers provide a great help in trouble shooting network problems. All values
1407are reported in milliseconds (ms). In HTTP mode, four control points are
1408reported under the form 'Tq/Tc/Tr/Tt' :
1409
1410 - Tq: total time to get the client request.
1411 It's the time elapsed between the moment the client connection was accepted
1412 and the moment the proxy received the last HTTP header. The value '-1'
1413 indicates that the end of headers (empty line) has never been seen.
1414
1415 - Tc: total time to establish the TCP connection to the server.
1416 It's the time elapsed between the moment the proxy sent the connection
1417 request, and the moment it was acknowledged, or between the TCP SYN packet
1418 and the matching SYN/ACK in return. The value '-1' means that the
1419 connection never established.
1420
1421 - Tr: server response time. It's the time elapsed between the moment the
1422 TCP connection was established to the server and the moment it send its
1423 complete response header. It purely shows its request processing time,
1424 without the network overhead due to the data transmission. The value '-1'
1425 means that the last the response header (empty line) was never seen.
1426
1427 - Tt: total session duration time, between the moment the proxy accepted it
willy tarreau982249e2005-12-18 00:57:06 +01001428 and the moment both ends were closed. The exception is when the 'logasap'
1429 option is specified. In this case, it only equals (Tq+Tc+Tr), and is
1430 prefixed with a '+' sign. From this field, we can deduce Td, the data
1431 transmission time, by substracting other timers when valid :
willy tarreauc1cae632005-12-17 14:12:23 +01001432
1433 Td = Tt - (Tq + Tc + Tr)
1434
1435 Timers with '-1' values have to be excluded from this equation.
1436
1437In TCP mode ('option tcplog'), only Tc and Tt are reported.
1438
1439These timers provide precious indications on trouble causes. Since the TCP
1440protocol defines retransmit delays of 3, 6, 12... seconds, we know for sure
1441that timers close to multiples of 3s are nearly always related to packets lost
1442due to network problems (wires or negociation). Moreover, if <Tt> is close to
1443a timeout value specified in the configuration, it often means that a session
1444has been aborted on time-out.
1445
1446Most common cases :
1447
1448 - If Tq is close to 3000, a packet has probably been lost between the client
1449 and the proxy.
1450 - If Tc is close to 3000, a packet has probably been lost between the server
1451 and the proxy during the server connection phase. This one should always be
1452 very low (less than a few tens).
1453 - If Tr is nearly always lower than 3000 except some rare values which seem to
1454 be the average majored by 3000, there are probably some packets lost between
1455 the proxy and the server.
1456 - If Tt is often slightly higher than a time-out, it's often because the
1457 client and the server use HTTP keep-alive and the session is maintained
1458 after the response ends. Se further for how to disable HTTP keep-alive.
1459
1460Other cases ('xx' means any value to be ignored) :
1461 -1/xx/xx/Tt : the client was not able to send its complete request in time,
willy tarreauc5f73ed2005-12-18 01:26:38 +01001462 or that it aborted it too early.
willy tarreauc1cae632005-12-17 14:12:23 +01001463 Tq/-1/xx/Tt : the connection could not establish on the server. Either it
willy tarreauc5f73ed2005-12-18 01:26:38 +01001464 refused it or it timed out after Tt-Tq ms.
willy tarreauc1cae632005-12-17 14:12:23 +01001465 Tq/Tc/-1/Tt : the server has accepted the connection but did not return a
willy tarreauc5f73ed2005-12-18 01:26:38 +01001466 complete response in time, or it closed its connexion
1467 unexpectedly, after Tt-(Tq+Tc) ms.
willy tarreauc1cae632005-12-17 14:12:23 +01001468
14694.2.4) Session state at disconnection
1470-------------------------------------
willy tarreauc5f73ed2005-12-18 01:26:38 +01001471TCP and HTTP logs provide a session completion indicator in the
1472<termination_state> field, just before the number of active
1473connections. It is 2-characters long in TCP, and 4-characters long in
1474HTTP, each of which has a special meaning :
1475
willy tarreau197e8ec2005-12-17 14:10:59 +01001476 - On the first character, a code reporting the first event which caused the
1477 session to terminate :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001478
willy tarreauc5f73ed2005-12-18 01:26:38 +01001479 C : the TCP session was unexpectedly aborted by the client.
1480
1481 S : the TCP session was unexpectedly aborted by the server, or the
1482 server explicitly refused it.
1483
1484 P : the session was prematurely aborted by the proxy, because of a
1485 connection limit enforcement, because a DENY filter was matched,
1486 or because of a security check which detected and blocked a
1487 dangerous error in server response which might have caused
1488 information leak (eg: cacheable cookie).
1489
1490 R : a resource on the proxy has been exhausted (memory, sockets, source
1491 ports, ...). Usually, this appears during the connection phase, and
1492 system logs should contain a copy of the precise error.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001493
willy tarreauc5f73ed2005-12-18 01:26:38 +01001494 I : an internal error was identified by the proxy during a self-check.
1495 This should NEVER happen, and you are encouraged to report any log
1496 containing this, because this is a bug.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001497
willy tarreauc5f73ed2005-12-18 01:26:38 +01001498 c : the client-side time-out expired first.
1499
1500 s : the server-side time-out expired first.
1501
1502 - : normal session completion.
1503
1504 - on the second character, the TCP/HTTP session state when it was closed :
1505
1506 R : waiting for complete REQUEST from the client (HTTP only). Nothing
1507 was sent to any server.
1508
1509 C : waiting for CONNECTION to establish on the server. The server might
1510 at most have noticed a connection attempt.
1511
1512 H : waiting for, receiving and processing server HEADERS (HTTP only).
1513
1514 D : the session was in the DATA phase.
1515
1516 L : the proxy was still transmitting LAST data to the client while the
1517 server had already finished.
1518
1519 - : normal session completion after end of data transfer.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001520
willy tarreau197e8ec2005-12-17 14:10:59 +01001521 - the third character tells whether the persistence cookie was provided by
willy tarreauc1cae632005-12-17 14:12:23 +01001522 the client (only in HTTP mode) :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001523
willy tarreauc5f73ed2005-12-18 01:26:38 +01001524 N : the client provided NO cookie. This is usually the case on new
1525 connections.
1526
1527 I : the client provided an INVALID cookie matching no known
1528 server. This might be caused by a recent configuration change,
1529 mixed cookies between HTTP/HTTPS sites, or an attack.
1530
1531 D : the client provided a cookie designating a server which was DOWN,
1532 so either the 'persist' option was used and the client was sent to
1533 this server, or it was not set and the client was redispatched to
1534 another server.
1535
1536 V : the client provided a valid cookie, and was sent to the associated
1537 server.
1538
1539 - : does not apply (no cookie set in configuration).
willy tarreaueedaa9f2005-12-17 14:08:03 +01001540
willy tarreau197e8ec2005-12-17 14:10:59 +01001541 - the last character reports what operations were performed on the persistence
willy tarreauc1cae632005-12-17 14:12:23 +01001542 cookie returned by the server (only in HTTP mode) :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001543
willy tarreauc5f73ed2005-12-18 01:26:38 +01001544 N : NO cookie was provided by the server, and none was inserted either.
1545
1546 I : no cookie was provided by the server, and the proxy INSERTED one.
1547
willy tarreau197e8ec2005-12-17 14:10:59 +01001548 P : a cookie was PROVIDED by the server and transmitted as-is.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001549
willy tarreauc5f73ed2005-12-18 01:26:38 +01001550 R : the cookie provided by the server was REWRITTEN by the proxy.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001551
willy tarreauc5f73ed2005-12-18 01:26:38 +01001552 D : the cookie provided by the server was DELETED by the proxy.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001553
willy tarreauc5f73ed2005-12-18 01:26:38 +01001554 - : does not apply (no cookie set in configuration).
willy tarreaueedaa9f2005-12-17 14:08:03 +01001555
willy tarreauc5f73ed2005-12-18 01:26:38 +01001556The combination of the two first flags give a lot of information about what was
1557happening when the session terminated. It can be helpful to detect server
1558saturation, network troubles, local system resource starvation, attacks, etc...
willy tarreaueedaa9f2005-12-17 14:08:03 +01001559
willy tarreauc5f73ed2005-12-18 01:26:38 +01001560The most common termination flags combinations are indicated here.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001561
willy tarreauc5f73ed2005-12-18 01:26:38 +01001562 Flags Reason
1563 CR The client aborted before sending a full request. Most probably the
1564 request was done by hand using a telnet client, and aborted early.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001565
willy tarreauc5f73ed2005-12-18 01:26:38 +01001566 cR The client timed out before sending a full request. This is sometimes
1567 caused by too large TCP MSS values on the client side for PPPoE
1568 networks which cannot transport full-sized packets, or by clients
1569 sending requests by hand and not typing fast enough.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001570
willy tarreauc5f73ed2005-12-18 01:26:38 +01001571 SC The server explicitly refused the connection (the proxy received a
1572 TCP RST or an ICMP in return). Under some circumstances, it can
1573 also be the network stack telling the proxy that the server is
1574 unreachable (eg: no route, or no ARP response on local network).
willy tarreau982249e2005-12-18 00:57:06 +01001575
willy tarreauc5f73ed2005-12-18 01:26:38 +01001576 sC The connection to the server did not complete during contimeout.
willy tarreau982249e2005-12-18 00:57:06 +01001577
willy tarreauc5f73ed2005-12-18 01:26:38 +01001578 PC The proxy refused to establish a connection to the server because the
1579 maxconn limit has been reached. The listener's maxconn parameter may
1580 be increased in the proxy configuration, as well as the global
1581 maxconn parameter.
willy tarreauc1cae632005-12-17 14:12:23 +01001582
willy tarreauc5f73ed2005-12-18 01:26:38 +01001583 RC A local resource has been exhausted (memory, sockets, source ports)
1584 preventing the connection to the server from establishing. The error
1585 logs will tell precisely what was missing. Anyway, this can only be
1586 solved by system tuning.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001587
willy tarreauc5f73ed2005-12-18 01:26:38 +01001588 cH The client timed out during a POST request. This is sometimes caused
1589 by too large TCP MSS values for PPPoE networks which cannot transport
1590 full-sized packets.
willy tarreauc1cae632005-12-17 14:12:23 +01001591
willy tarreauc5f73ed2005-12-18 01:26:38 +01001592 SH The server aborted before sending its full headers, or it crashed.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001593
willy tarreauc5f73ed2005-12-18 01:26:38 +01001594 sH The server failed to reply during the srvtimeout delay, which
1595 indicates too long transactions, probably caused by back-end
1596 saturation. The only solutions are to fix the problem on the
1597 application or to increase the 'srvtimeout' parameter to support
1598 longer delays (at the risk of the client giving up anyway).
1599
1600 PR The proxy blocked the client's request, either because of an invalid
1601 HTTP syntax, in which case it returned an HTTP 400 error to the
1602 client, or because a deny filter matched, in which case it returned
1603 an HTTP 403 error.
1604
1605 PH The proxy blocked the server's response, because it was invalid,
1606 incomplete, dangerous (cache control), or matched a security filter.
1607 In any case, an HTTP 502 error is sent to the client.
1608
1609 cD The client did not read any data for as long as the clitimeout delay.
1610 This is often caused by network failures on the client side.
1611
1612 CD The client unexpectedly aborted during data transfer. This is either
1613 caused by a browser crash, or by a keep-alive session between the
1614 server and the client terminated first by the client.
1615
1616 sD The server did nothing during the srvtimeout delay. This is often
1617 caused by too short timeouts on L4 equipements before the server
1618 (firewalls, load-balancers, ...).
1619
16204.2.5) Non-printable characters
willy tarreau4302f492005-12-18 01:00:37 +01001621-------------------------------
1622As of version 1.1.29, non-printable characters are not sent as-is into log
1623files, but are converted to their two-digits hexadecimal representation,
1624prefixed by the character '#'. The only characters that can now be logged
1625without being escaped are between 32 and 126 (inclusive). Obviously, the
1626escape character '#' is also encoded to avoid any ambiguity. It is the same for
1627the character '"', as well as '{', '|' and '}' when logging headers.
1628
willy tarreauc5f73ed2005-12-18 01:26:38 +010016294.2.6) Capturing HTTP headers and cookies
1630-----------------------------------------
1631Version 1.1.23 brought cookie capture, and 1.1.29 the header capture. All this
1632is performed using the 'capture' keyword.
1633
1634Cookie capture makes it easy to track a complete user session. The syntax is :
1635
1636 capture cookie <cookie_prefix> len <capture_length>
1637
1638This will enable cookie capture from both requests and responses. This way,
1639it's easy to detect when a user switches to a new session for example, because
1640the server will reassign it a new cookie.
1641
1642The FIRST cookie whose name starts with <cookie_prefix> will be captured, and
1643logged as 'NAME=value', without exceeding <capture_length> characters (64 max).
1644When the cookie name is fixed and known, it's preferable to suffix '=' to it to
1645ensure that no other cookie will be logged.
1646
1647Examples :
1648----------
1649 # capture the first cookie whose name starts with "ASPSESSION"
1650 capture cookie ASPSESSION len 32
1651
1652 # capture the first cookie whose name is exactly "vgnvisitor"
1653 capture cookie vgnvisitor= len 32
1654
1655In the logs, the field preceeding the completion indicator contains the cookie
1656value as sent by the server, preceeded by the cookie value as sent by the
1657client. Each of these field is replaced with '-' when no cookie was seen or
1658when the option is disabled.
1659
1660Header captures have a different goal. They are useful to track unique request
1661identifiers set by a previous proxy, virtual host names, user-agents, POST
1662content-length, referrers, etc. In the response, one can search for information
1663about the response length, how the server asked the cache to behave, or an
1664object location during a redirection. As for cookie captures, it is both
1665possible to include request headers and response headers at the same time. The
1666syntax is :
willy tarreau4302f492005-12-18 01:00:37 +01001667
1668 capture request header <name> len <max length>
1669 capture response header <name> len <max length>
1670
1671Note: Header names are not case-sensitive.
1672
1673Examples:
1674---------
1675 # keep the name of the virtual server
1676 capture request header Host len 20
1677 # keep the amount of data uploaded during a POST
1678 capture request header Content-Length len 10
1679
1680 # note the expected cache behaviour on the response
1681 capture response header Cache-Control len 8
1682 # note the URL location during a redirection
1683 capture response header Location len 20
1684
1685Non-existant headers are logged as empty strings, and if one header appears more
1686than once, only its last occurence will be kept. Request headers are grouped
1687within braces '{' and '}' in the same order as they were declared, and delimited
1688with a vertical bar '|' without any space. Response headers follow the same
1689representation, but are displayed after a space following the request headers
1690block. These blocks are displayed just before the HTTP request in the logs.
willy tarreauc5f73ed2005-12-18 01:26:38 +01001691
willy tarreau4302f492005-12-18 01:00:37 +01001692Example :
1693
willy tarreauc5f73ed2005-12-18 01:26:38 +01001694 Config:
1695
1696 capture request header Host len 20
1697 capture request header Content-Length len 10
1698 capture request header Referer len 20
1699 capture response header Server len 20
1700 capture response header Content-Length len 10
1701 capture response header Cache-Control len 8
1702 capture response header Via len 20
1703 capture response header Location len 20
1704
1705 Log :
1706
1707 Aug 9 20:26:09 localhost haproxy[2022]: 127.0.0.1:34014 [09/Aug/2004:20:26:09] relais-http netcache 0/0/162/+162 200 +350 - - ---- {fr.adserver.yahoo.co||http://fr.f416.mail.} {|864|private||} "GET http://fr.adserver.yahoo.com/"
1708 Aug 9 20:30:46 localhost haproxy[2022]: 127.0.0.1:34020 [09/Aug/2004:20:30:46] relais-http netcache 0/0/182/+182 200 +279 - - ---- {w.ods.org||} {Formilux/0.1.8|3495|||} "GET http://w.ods.org/sytadin.html HTTP/1.1"
1709 Aug 9 20:30:46 localhost haproxy[2022]: 127.0.0.1:34028 [09/Aug/2004:20:30:46] relais-http netcache 0/2/126/+128 200 +223 - - ---- {www.infotrafic.com||http://w.ods.org/syt} {Apache/2.0.40 (Red H|9068|||} "GET http://www.infotrafic.com/images/live/cartesidf/grandes/idf_ne.png HTTP/1.1"
1710
1711
17124.2.7) Examples of logs
1713-----------------------
1714- haproxy[674]: 127.0.0.1:33319 [15/Oct/2003:08:31:57] relais-http Srv1 6559/7/147/6723 200 243 - - ---- 3/5 "HEAD / HTTP/1.0"
1715 => long request (6.5s) entered by hand through 'telnet'. The server replied
1716 in 147 ms, and the session ended normally ('----')
1717
1718- haproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17] relais-http Srv1 9/7/14/+30 200 +243 - - ---- 3/3 "GET /image.iso HTTP/1.0"
1719 => request for a long data transfer. The 'logasap' option was specified, so
1720 the log was produced just before transfering data. The server replied in
1721 14 ms, 243 bytes of headers were sent to the client, and total time from
1722 accept to first data byte is 30 ms.
1723
1724- haproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17] relais-http Srv1 9/7/14/30 502 243 - - PH-- 2/3 "GET /cgi-bin/bug.cgi? HTTP/1.0"
1725 => the proxy blocked a server response either because of an 'rspdeny' or
1726 'rspideny' filter, or because it blocked sensible information which risked
1727 being cached. In this case, the response is replaced with a '502 bad
1728 gateway'.
1729
1730- haproxy[18113]: 127.0.0.1:34548 [15/Oct/2003:15:18:55] relais-http <NOSRV> -1/-1/-1/8490 -1 0 - - CR-- 2/2 ""
1731 => the client never completed its request and aborted itself ('C---') after
1732 8.5s, while the proxy was waiting for the request headers ('-R--').
1733 Nothing was sent to the server.
1734
1735- haproxy[18113]: 127.0.0.1:34549 [15/Oct/2003:15:19:06] relais-http <NOSRV> -1/-1/-1/50001 408 0 - - cR-- 2/2 ""
1736 => The client never completed its request, which was aborted by the time-out
1737 ('c---') after 50s, while the proxy was waiting for the request headers ('-R--').
1738 Nothing was sent to the server, but the proxy could send a 408 return code
1739 to the client.
willy tarreau4302f492005-12-18 01:00:37 +01001740
willy tarreauc5f73ed2005-12-18 01:26:38 +01001741- haproxy[18989]: 127.0.0.1:34550 [15/Oct/2003:15:24:28] relais-tcp Srv1 0/5007 0 cD
1742 => This is a 'tcplog' entry. Client-side time-out ('c----') occured after 5s.
willy tarreau4302f492005-12-18 01:00:37 +01001743
willy tarreauc5f73ed2005-12-18 01:26:38 +01001744- haproxy[18989]: 10.0.0.1:34552 [15/Oct/2003:15:26:31] relais-http Srv1 3183/-1/-1/11215 503 0 - - SC-- 202/205 "HEAD / HTTP/1.0"
1745 => The request took 3s to complete (probably a network problem), and the
1746 connection to the server failed ('SC--') after 4 attemps of 2 seconds
1747 (config says 'retries 3'), then a 503 error code was sent to the client.
1748 There were 202 connections on this proxy, and 205 on the global process.
1749 It is possible that the server refused the connection because of too many
1750 already established.
willy tarreau4302f492005-12-18 01:00:37 +01001751
willy tarreau4302f492005-12-18 01:00:37 +01001752
willy tarreau197e8ec2005-12-17 14:10:59 +010017534.3) HTTP header manipulation
1754-----------------------------
1755In HTTP mode, it is possible to rewrite, add or delete some of the request and
1756response headers based on regular expressions. It is also possible to block a
1757request or a response if a particular header matches a regular expression,
1758which is enough to stops most elementary protocol attacks, and to protect
1759against information leak from the internal network. But there is a limitation
1760to this : since haproxy's HTTP engine knows nothing about keep-alive, only
1761headers passed during the first request of a TCP session will be seen. All
1762subsequent headers will be considered data only and not analyzed. Furthermore,
1763haproxy doesn't touch data contents, it stops at the end of headers.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001764
willy tarreau197e8ec2005-12-17 14:10:59 +01001765The syntax is :
1766 reqadd <string> to add a header to the request
1767 reqrep <search> <replace> to modify the request
1768 reqirep <search> <replace> same, but ignoring the case
1769 reqdel <search> to delete a header in the request
1770 reqidel <search> same, but ignoring the case
1771 reqallow <search> definitely allow a request if a header matches <search>
1772 reqiallow <search> same, but ignoring the case
1773 reqdeny <search> denies a request if a header matches <search>
1774 reqideny <search> same, but ignoring the case
1775 reqpass <search> ignore a header matching <search>
1776 reqipass <search> same, but ignoring the case
willy tarreaueedaa9f2005-12-17 14:08:03 +01001777
willy tarreau197e8ec2005-12-17 14:10:59 +01001778 rspadd <string> to add a header to the response
1779 rsprep <search> <replace> to modify the response
1780 rspirep <search> <replace> same, but ignoring the case
1781 rspdel <search> to delete the response
1782 rspidel <search> same, but ignoring the case
willy tarreau982249e2005-12-18 00:57:06 +01001783 rspdeny <search> replaces a response with a HTTP 502 if a header matches <search>
1784 rspideny <search> same, but ignoring the case
willy tarreaueedaa9f2005-12-17 14:08:03 +01001785
1786
willy tarreau197e8ec2005-12-17 14:10:59 +01001787<search> is a POSIX regular expression (regex) which supports grouping through
1788parenthesis (without the backslash). Spaces and other delimiters must be
1789prefixed with a backslash ('\') to avoid confusion with a field delimiter.
1790Other characters may be prefixed with a backslash to change their meaning :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001791
willy tarreau197e8ec2005-12-17 14:10:59 +01001792 \t for a tab
1793 \r for a carriage return (CR)
1794 \n for a new line (LF)
1795 \ to mark a space and differentiate it from a delimiter
1796 \# to mark a sharp and differentiate it from a comment
1797 \\ to use a backslash in a regex
1798 \\\\ to use a backslash in the text (*2 for regex, *2 for haproxy)
1799 \xXX to write the ASCII hex code XX as in the C language
willy tarreaueedaa9f2005-12-17 14:08:03 +01001800
1801
willy tarreau197e8ec2005-12-17 14:10:59 +01001802<replace> containst the string to be used to replace the largest portion of text
1803matching the regex. It can make use of the special characters above, and can
1804reference a substring delimited by parenthesis in the regex, by the group
1805numerical order from 1 to 9. In this case, you would write a backslah ('\')
1806immediately followed by one digit indicating the group position.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001807
willy tarreau197e8ec2005-12-17 14:10:59 +01001808<string> represents the string which will systematically be added after the last
1809header line. It can also use special characters above.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001810
willy tarreau197e8ec2005-12-17 14:10:59 +01001811Notes :
1812-------
1813 - the first line is considered as a header, which makes it possible to rewrite
1814 or filter HTTP requests URIs or response codes.
1815 - 'reqrep' is the equivalent of 'cliexp' in version 1.0, and 'rsprep' is the
1816 equivalent of 'srvexp' in 1.0. Those names are still supported but
1817 deprecated.
1818 - for performances reasons, the number of characters added to a request or to
1819 a response is limited to 4096 since version 1.1.5 (it was 256 before). This
1820 value is easy to modify in the code if needed (#define). If it is too short
1821 on occasional uses, it is possible to gain some space by removing some
1822 useless headers before adding new ones.
willy tarreau982249e2005-12-18 00:57:06 +01001823 - a denied request will generate an "HTTP 403 forbidden" response, while a
1824 denied response will generate an "HTTP 502 Bad gateway" response.
1825
willy tarreaueedaa9f2005-12-17 14:08:03 +01001826
willy tarreau197e8ec2005-12-17 14:10:59 +01001827Examples :
1828----------
willy tarreauc5f73ed2005-12-18 01:26:38 +01001829 ###### a few examples ######
willy tarreau197e8ec2005-12-17 14:10:59 +01001830
willy tarreauc5f73ed2005-12-18 01:26:38 +01001831 # rewrite 'online.fr' instead of 'free.fr' for GET and POST requests
1832 reqrep ^(GET\ .*)(.free.fr)(.*) \1.online.fr\3
1833 reqrep ^(POST\ .*)(.free.fr)(.*) \1.online.fr\3
willy tarreau197e8ec2005-12-17 14:10:59 +01001834
willy tarreauc5f73ed2005-12-18 01:26:38 +01001835 # force proxy connections to close
1836 reqirep ^Proxy-Connection:.* Proxy-Connection:\ close
1837 # rewrite locations
1838 rspirep ^(Location:\ )([^:]*://[^/]*)(.*) \1\3
willy tarreaueedaa9f2005-12-17 14:08:03 +01001839
willy tarreauc5f73ed2005-12-18 01:26:38 +01001840 ###### A full configuration being used on production ######
willy tarreaueedaa9f2005-12-17 14:08:03 +01001841
willy tarreau197e8ec2005-12-17 14:10:59 +01001842 # Every header should end with a colon followed by one space.
willy tarreauc5f73ed2005-12-18 01:26:38 +01001843 reqideny ^[^:\ ]*[\ ]*$
willy tarreaueedaa9f2005-12-17 14:08:03 +01001844
willy tarreau197e8ec2005-12-17 14:10:59 +01001845 # block Apache chunk exploit
willy tarreauc5f73ed2005-12-18 01:26:38 +01001846 reqideny ^Transfer-Encoding:[\ ]*chunked
1847 reqideny ^Host:\ apache-
willy tarreaueedaa9f2005-12-17 14:08:03 +01001848
willy tarreau197e8ec2005-12-17 14:10:59 +01001849 # block annoying worms that fill the logs...
willy tarreauc5f73ed2005-12-18 01:26:38 +01001850 reqideny ^[^:\ ]*\ .*(\.|%2e)(\.|%2e)(%2f|%5c|/|\\\\)
1851 reqideny ^[^:\ ]*\ ([^\ ]*\ [^\ ]*\ |.*%00)
1852 reqideny ^[^:\ ]*\ .*<script
1853 reqideny ^[^:\ ]*\ .*/(root\.exe\?|cmd\.exe\?|default\.ida\?)
willy tarreau197e8ec2005-12-17 14:10:59 +01001854
1855 # allow other syntactically valid requests, and block any other method
willy tarreauc5f73ed2005-12-18 01:26:38 +01001856 reqipass ^(GET|POST|HEAD|OPTIONS)\ /.*\ HTTP/1\.[01]$
1857 reqipass ^OPTIONS\ \\*\ HTTP/1\.[01]$
1858 reqideny ^[^:\ ]*\
willy tarreau197e8ec2005-12-17 14:10:59 +01001859
1860 # force connection:close, thus disabling HTTP keep-alive
willy tarreauc5f73ed2005-12-18 01:26:38 +01001861 option httpclose
willy tarreau197e8ec2005-12-17 14:10:59 +01001862
willy tarreauc5f73ed2005-12-18 01:26:38 +01001863 # change the server name
1864 rspidel ^Server:\
1865 rspadd Server:\ Formilux/0.1.8
willy tarreau197e8ec2005-12-17 14:10:59 +01001866
1867
willy tarreau982249e2005-12-18 00:57:06 +01001868Also, the 'forwardfor' option creates an HTTP 'X-Forwarded-For' header which
willy tarreauc1cae632005-12-17 14:12:23 +01001869contains the client's IP address. This is useful to let the final web server
1870know what the client address was (eg for statistics on domains).
1871
willy tarreau982249e2005-12-18 00:57:06 +01001872Last, the 'httpclose' option removes any 'Connection' header both ways, and
1873adds a 'Connection: close' header in each direction. This makes it easier to
Willy TARREAU767ba712006-03-01 22:40:50 +01001874disable HTTP keep-alive than the previous 4-rules block.
willy tarreau982249e2005-12-18 00:57:06 +01001875
willy tarreauc1cae632005-12-17 14:12:23 +01001876Example :
1877---------
1878 listen http_proxy 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001879 mode http
1880 log global
1881 option httplog
1882 option dontlognull
1883 option forwardfor
1884 option httpclose
1885
Willy TARREAU767ba712006-03-01 22:40:50 +01001886Note that some HTTP servers do not necessarily close the connections when they
1887receive the 'Connection: close', and if the client does not close either, then
1888the connection will be maintained up to the time-out. This translates into high
1889number of simultaneous sessions and high global session times in the logs. To
1890workaround this, a new option 'forceclose' appeared in version 1.2.9 to enforce
1891the closing of the outgoing server channel as soon as the server begins to
1892reply and only if the request buffer is empty. Note that this should NOT be
1893used if CONNECT requests are expected between the client and the server. The
1894'forceclose' option implies the 'httpclose' option.
1895
1896Example :
1897---------
1898 listen http_proxy 0.0.0.0:80
1899 mode http
1900 log global
1901 option httplog
1902 option dontlognull
1903 option forwardfor
1904 option forceclose
1905
willy tarreau197e8ec2005-12-17 14:10:59 +01001906
19074.4) Load balancing with persistence
1908------------------------------------
willy tarreau197e8ec2005-12-17 14:10:59 +01001909Combining cookie insertion with internal load balancing allows to transparently
1910bring persistence to applications. The principle is quite simple :
1911 - assign a cookie value to each server
1912 - enable the load balancing between servers
1913 - insert a cookie into responses resulting from the balancing algorithm
1914 (indirect accesses), end ensure that no upstream proxy will cache it.
1915 - remove the cookie in the request headers so that the application never sees
1916 it.
1917
1918Example :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001919---------
1920 listen application 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001921 mode http
1922 cookie SERVERID insert nocache indirect
1923 balance roundrobin
1924 server srv1 192.168.1.1:80 cookie server01 check
1925 server srv2 192.168.1.2:80 cookie server02 check
willy tarreaueedaa9f2005-12-17 14:08:03 +01001926
willy tarreau0174f312005-12-18 01:02:42 +01001927The other solution brought by versions 1.1.30 and 1.2.3 is to reuse a cookie
1928from the server, and prefix the server's name to it. In this case, don't forget
1929to force "httpclose" mode so that you can be assured that every subsequent
1930request will have its cookie fixed.
1931
1932 listen application 0.0.0.0:80
willy tarreauc5f73ed2005-12-18 01:26:38 +01001933 mode http
1934 cookie JSESSIONID prefix
1935 balance roundrobin
1936 server srv1 192.168.1.1:80 cookie srv1 check
1937 server srv2 192.168.1.2:80 cookie srv2 check
1938 option httpclose
willy tarreau0174f312005-12-18 01:02:42 +01001939
1940
willy tarreau982249e2005-12-18 00:57:06 +010019414.5) Protection against information leak from the servers
1942---------------------------------------------------------
1943In versions 1.1.28/1.2.1, a new option 'checkcache' was created. It carefully
1944checks 'Cache-control', 'Pragma' and 'Set-cookie' headers in server response
1945to check if there's a risk of caching a cookie on a client-side proxy. When this
1946option is enabled, the only responses which can be delivered to the client are :
1947 - all those without 'Set-Cookie' header ;
1948 - all those with a return code other than 200, 203, 206, 300, 301, 410,
1949 provided that the server has not set a 'Cache-control: public' header ;
1950 - all those that come from a POST request, provided that the server has not
1951 set a 'Cache-Control: public' header ;
1952 - those with a 'Pragma: no-cache' header
1953 - those with a 'Cache-control: private' header
1954 - those with a 'Cache-control: no-store' header
1955 - those with a 'Cache-control: max-age=0' header
1956 - those with a 'Cache-control: s-maxage=0' header
1957 - those with a 'Cache-control: no-cache' header
1958 - those with a 'Cache-control: no-cache="set-cookie"' header
1959 - those with a 'Cache-control: no-cache="set-cookie,' header
1960 (allowing other fields after set-cookie)
willy tarreaueedaa9f2005-12-17 14:08:03 +01001961
willy tarreau982249e2005-12-18 00:57:06 +01001962If a response doesn't respect these requirements, then it will be blocked just
1963as if it was from an 'rspdeny' filter, with an "HTTP 502 bad gateway". The
1964session state shows "PH--" meaning that the proxy blocked the response during
1965headers processing. Additionnaly, an alert will be sent in the logs so that
1966admins are told that there's something to be done.
1967
willy tarreauc5f73ed2005-12-18 01:26:38 +01001968
willy tarreau982249e2005-12-18 00:57:06 +010019694.6) Customizing errors
1970-----------------------
willy tarreau197e8ec2005-12-17 14:10:59 +01001971Some situations can make haproxy return an HTTP error code to the client :
1972 - invalid or too long request => HTTP 400
1973 - request not completely sent in time => HTTP 408
1974 - forbidden request (matches a deny filter) => HTTP 403
1975 - internal error in haproxy => HTTP 500
1976 - the server returned an invalid or incomplete response => HTTP 502
1977 - no server was available to handle the request => HTTP 503
1978 - the server failed to reply in time => HTTP 504
willy tarreaueedaa9f2005-12-17 14:08:03 +01001979
willy tarreau197e8ec2005-12-17 14:10:59 +01001980A succint error message taken from the RFC accompanies these return codes.
1981But depending on the clients knowledge, it may be better to return custom, user
1982friendly, error pages. This is made possible through the use of the 'errorloc'
1983command :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001984
willy tarreau197e8ec2005-12-17 14:10:59 +01001985 errorloc <HTTP_code> <location>
willy tarreaueedaa9f2005-12-17 14:08:03 +01001986
willy tarreau197e8ec2005-12-17 14:10:59 +01001987Instead of generating an HTTP error <HTTP_code> among those above, the proxy
1988will return a temporary redirection code (HTTP 302) towards the address
1989specified in <location>. This address may be either relative to the site or
1990absolute. Since this request will be handled by the client's browser, it's
1991mandatory that the returned address be reachable from the outside.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001992
willy tarreau197e8ec2005-12-17 14:10:59 +01001993Example :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001994---------
1995 listen application 0.0.0.0:80
1996 errorloc 400 /badrequest.html
1997 errorloc 403 /forbidden.html
1998 errorloc 408 /toolong.html
willy tarreauc5f73ed2005-12-18 01:26:38 +01001999 errorloc 500 http://haproxy.domain.net/bugreport.html
willy tarreaueedaa9f2005-12-17 14:08:03 +01002000 errorloc 502 http://192.168.114.58/error50x.html
2001 errorloc 503 http://192.168.114.58/error50x.html
2002 errorloc 504 http://192.168.114.58/error50x.html
2003
willy tarreauc1f47532005-12-18 01:08:26 +01002004Note: RFC2616 says that a client must reuse the same method to fetch the
2005Location returned by a 302, which causes problems with the POST method.
2006The return code 303 was designed explicitly to force the client to fetch the
2007Location URL with the GET method, but there are some browsers pre-dating
2008HTTP/1.1 which don't support it. Anyway, most browsers still behave with 302 as
willy tarreauc5f73ed2005-12-18 01:26:38 +01002009if it was a 303. In order to allow the user to chose, versions 1.1.31 and 1.2.5
2010bring two new keywords to replace 'errorloc' : 'errorloc302' and 'errorloc303'.
willy tarreauc1f47532005-12-18 01:08:26 +01002011
2012They are preffered over errorloc (which still does 302). Consider using
2013errorloc303 everytime you know that your clients support HTTP 303 responses..
2014
2015
willy tarreau982249e2005-12-18 00:57:06 +010020164.7) Modifying default values
willy tarreau197e8ec2005-12-17 14:10:59 +01002017-----------------------------
willy tarreau197e8ec2005-12-17 14:10:59 +01002018Version 1.1.22 introduced the notion of default values, which eliminates the
2019pain of often repeating common parameters between many instances, such as
2020logs, timeouts, modes, etc...
willy tarreaueedaa9f2005-12-17 14:08:03 +01002021
willy tarreau197e8ec2005-12-17 14:10:59 +01002022Default values are set in a 'defaults' section. Each of these section clears
2023all previously set default parameters, so there may be as many default
2024parameters as needed. Only the last one before a 'listen' section will be
2025used for this section. The 'defaults' section uses the same syntax as the
2026'listen' section, for the supported parameters. The 'defaults' keyword ignores
2027everything on its command line, so that fake instance names can be specified
2028there for better clarity.
willy tarreaueedaa9f2005-12-17 14:08:03 +01002029
willy tarreau982249e2005-12-18 00:57:06 +01002030In version 1.1.28/1.2.1, only those parameters can be preset in the 'default'
willy tarreau197e8ec2005-12-17 14:10:59 +01002031section :
2032 - log (the first and second one)
willy tarreaueedaa9f2005-12-17 14:08:03 +01002033 - mode { tcp, http, health }
2034 - balance { roundrobin }
willy tarreau197e8ec2005-12-17 14:10:59 +01002035 - disabled (to disable every further instances)
2036 - enabled (to enable every further instances, this is the default)
willy tarreaueedaa9f2005-12-17 14:08:03 +01002037 - contimeout, clitimeout, srvtimeout, grace, retries, maxconn
willy tarreau982249e2005-12-18 00:57:06 +01002038 - option { redispatch, transparent, keepalive, forwardfor, logasap, httpclose,
2039 checkcache, httplog, tcplog, dontlognull, persist, httpchk }
willy tarreaueedaa9f2005-12-17 14:08:03 +01002040 - redispatch, redisp, transparent, source { addr:port }
2041 - cookie, capture
2042 - errorloc
2043
willy tarreau197e8ec2005-12-17 14:10:59 +01002044As of 1.1.24, it is not possible to put certain parameters in a 'defaults'
2045section, mainly regular expressions and server configurations :
willy tarreaueedaa9f2005-12-17 14:08:03 +01002046 - dispatch, server,
willy tarreau197e8ec2005-12-17 14:10:59 +01002047 - req*, rsp*
willy tarreaueedaa9f2005-12-17 14:08:03 +01002048
willy tarreau197e8ec2005-12-17 14:10:59 +01002049Last, there's no way yet to change a boolean option from its assigned default
2050value. So if an 'option' statement is set in a 'defaults' section, the only
2051way to flush it is to redefine a new 'defaults' section without this 'option'.
willy tarreaueedaa9f2005-12-17 14:08:03 +01002052
willy tarreau197e8ec2005-12-17 14:10:59 +01002053Examples :
willy tarreaueedaa9f2005-12-17 14:08:03 +01002054----------
2055 defaults applications TCP
willy tarreauc5f73ed2005-12-18 01:26:38 +01002056 log global
2057 mode tcp
2058 balance roundrobin
2059 clitimeout 180000
2060 srvtimeout 180000
2061 contimeout 4000
2062 retries 3
2063 redispatch
willy tarreaueedaa9f2005-12-17 14:08:03 +01002064
2065 listen app_tcp1 10.0.0.1:6000-6063
willy tarreauc5f73ed2005-12-18 01:26:38 +01002066 server srv1 192.168.1.1 check port 6000 inter 10000
2067 server srv2 192.168.1.2 backup
willy tarreaueedaa9f2005-12-17 14:08:03 +01002068
2069 listen app_tcp2 10.0.0.2:6000-6063
willy tarreauc5f73ed2005-12-18 01:26:38 +01002070 server srv1 192.168.2.1 check port 6000 inter 10000
2071 server srv2 192.168.2.2 backup
willy tarreaueedaa9f2005-12-17 14:08:03 +01002072
2073 defaults applications HTTP
willy tarreauc5f73ed2005-12-18 01:26:38 +01002074 log global
2075 mode http
2076 option httplog
2077 option forwardfor
2078 option dontlognull
2079 balance roundrobin
2080 clitimeout 20000
2081 srvtimeout 20000
2082 contimeout 4000
2083 retries 3
willy tarreaueedaa9f2005-12-17 14:08:03 +01002084
2085 listen app_http1 10.0.0.1:80-81
willy tarreauc5f73ed2005-12-18 01:26:38 +01002086 cookie SERVERID postonly insert indirect
2087 capture cookie userid= len 10
2088 server srv1 192.168.1.1:+8000 cookie srv1 check port 8080 inter 1000
2089 server srv1 192.168.1.2:+8000 cookie srv2 check port 8080 inter 1000
willy tarreaueedaa9f2005-12-17 14:08:03 +01002090
2091 defaults
willy tarreauc5f73ed2005-12-18 01:26:38 +01002092 # this empty section voids all default parameters
willy tarreaueedaa9f2005-12-17 14:08:03 +01002093
willy tarreau197e8ec2005-12-17 14:10:59 +01002094=========================
2095| System-specific setup |
2096=========================
willy tarreaueedaa9f2005-12-17 14:08:03 +01002097
willy tarreau197e8ec2005-12-17 14:10:59 +01002098Linux 2.4
2099=========
willy tarreaueedaa9f2005-12-17 14:08:03 +01002100
2101-- cut here --
2102#!/bin/sh
2103# set this to about 256/4M (16384 for 256M machine)
2104MAXFILES=16384
2105echo $MAXFILES > /proc/sys/fs/file-max
2106ulimit -n $MAXFILES
2107
2108if [ -e /proc/sys/net/ipv4/ip_conntrack_max ]; then
willy tarreauc5f73ed2005-12-18 01:26:38 +01002109 echo 65536 > /proc/sys/net/ipv4/ip_conntrack_max
willy tarreaueedaa9f2005-12-17 14:08:03 +01002110fi
2111
2112if [ -e /proc/sys/net/ipv4/netfilter/ip_ct_tcp_timeout_fin_wait ]; then
willy tarreauc5f73ed2005-12-18 01:26:38 +01002113 # 30 seconds for fin, 15 for time wait
2114 echo 3000 > /proc/sys/net/ipv4/netfilter/ip_ct_tcp_timeout_fin_wait
2115 echo 1500 > /proc/sys/net/ipv4/netfilter/ip_ct_tcp_timeout_time_wait
2116 echo 0 > /proc/sys/net/ipv4/netfilter/ip_ct_tcp_log_invalid_scale
2117 echo 0 > /proc/sys/net/ipv4/netfilter/ip_ct_tcp_log_out_of_window
willy tarreaueedaa9f2005-12-17 14:08:03 +01002118fi
2119
2120echo 1024 60999 > /proc/sys/net/ipv4/ip_local_port_range
2121echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
2122echo 4096 > /proc/sys/net/ipv4/tcp_max_syn_backlog
2123echo 262144 > /proc/sys/net/ipv4/tcp_max_tw_buckets
2124echo 262144 > /proc/sys/net/ipv4/tcp_max_orphans
2125echo 300 > /proc/sys/net/ipv4/tcp_keepalive_time
2126echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
2127echo 0 > /proc/sys/net/ipv4/tcp_timestamps
2128echo 0 > /proc/sys/net/ipv4/tcp_ecn
willy tarreauc5f73ed2005-12-18 01:26:38 +01002129echo 1 > /proc/sys/net/ipv4/tcp_sack
willy tarreaueedaa9f2005-12-17 14:08:03 +01002130echo 0 > /proc/sys/net/ipv4/tcp_dsack
2131
2132# auto-tuned on 2.4
2133#echo 262143 > /proc/sys/net/core/rmem_max
2134#echo 262143 > /proc/sys/net/core/rmem_default
2135
2136echo 16384 65536 524288 > /proc/sys/net/ipv4/tcp_rmem
2137echo 16384 349520 699040 > /proc/sys/net/ipv4/tcp_wmem
2138
2139-- cut here --
2140
willy tarreau197e8ec2005-12-17 14:10:59 +01002141
2142FreeBSD
2143=======
2144
2145A FreeBSD port of HA-Proxy is now available and maintained, thanks to
2146Clement Laforet <sheepkiller@cultdeadsheep.org>.
2147
2148For more information :
2149http://www.freebsd.org/cgi/url.cgi?ports/net/haproxy/pkg-descr
2150http://www.freebsd.org/cgi/cvsweb.cgi/ports/net/haproxy/
2151http://www.freshports.org/net/haproxy
2152
2153
2154-- end --