blob: 115be18213fa8a14fa8ac196833e3f2a11d75687 [file] [log] [blame]
willy tarreaueedaa9f2005-12-17 14:08:03 +01001
2 H A - P r o x y
3 ---------------
willy tarreau982249e2005-12-18 00:57:06 +01004 version 1.2.1
willy tarreaueedaa9f2005-12-17 14:08:03 +01005 willy tarreau
willy tarreau982249e2005-12-18 00:57:06 +01006 2004/06/06
willy tarreaueedaa9f2005-12-17 14:08:03 +01007
8============
9| Abstract |
10============
11
12HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high
13availability environments. Indeed, it can :
14 - route HTTP requests depending on statically assigned cookies ;
15 - spread the load among several servers while assuring server persistence
16 through the use of HTTP cookies ;
17 - switch to backup servers in the event a main one fails ;
18 - accept connections to special ports dedicated to service monitoring ;
19 - stop accepting connections without breaking existing ones ;
20 - add/modify/delete HTTP headers both ways ;
21 - block requests matching a particular pattern ;
22
23It needs very little resource. Its event-driven architecture allows it to easily
24handle thousands of simultaneous connections on hundreds of instances without
25risking the system's stability.
26
27====================
28| Start parameters |
29====================
30
31There are only a few command line options :
32
33 -f <configuration file>
34 -n <high limit for the total number of simultaneous connections>
35 -N <high limit for the per-proxy number of simultaneous connections>
36 -d starts in foregreound with debugging mode enabled
37 -D starts in daemon mode
willy tarreau982249e2005-12-18 00:57:06 +010038 -q disable messages on output
39 -V displays messages on output even when -q or 'quiet' are specified.
40 -c only checks config file and exits with code 0 if no error was found, or
41 exits with code 1 if a syntax error was found.
willy tarreaufe2c5c12005-12-17 14:14:34 +010042 -p <pidfile> asks the process to write down each of its children's
43 pids to this file in daemon mode.
willy tarreaueedaa9f2005-12-17 14:08:03 +010044 -s shows statistics (only if compiled in)
45 -l shows even more statistics (implies '-s')
46
47
48The maximal number of connections per proxy is used as the default parameter for
49each instance for which the 'maxconn' paramter is not set in the 'listen' section.
50
51The maximal number of total connections limits the number of connections used by
52the whole process if the 'maxconn' parameter is not set in the 'global' section.
53
54The debugging mode has the same effect as the 'debug' option in the 'global'
55section. When the proxy runs in this mode, it dumps every connections,
56disconnections, timestamps, and HTTP headers to stdout. This should NEVER
57be used in an init script since it will prevent the system from starting up.
58
59Statistics are only available if compiled in with the 'STATTIME' option. It's
60only used during code optimization phases.
61
62======================
63| Configuration file |
64======================
65
66Structure
67=========
68
69The configuration file parser ignores empty lines, spaces, tabs. Anything
70between a sharp ('#') not following a backslash ('\'), and the end of a line
71constitutes a comment and is ignored too.
72
73The configuration file is segmented in sections. A section begins whenever
74one of these 3 keywords are encountered :
75
76 - 'global'
77 - 'listen'
78 - 'defaults'
79
80Every parameter refer to the section beginning at the last one of these 3
81keywords.
82
83
841) Global parameters
85====================
86
87Global parameters affect the whole process behaviour. They are all set in the
88'global' section. There may be several 'global' sections if needed, but their
89parameters will only be merged. Allowed parameters in 'global' section include
90the following ones :
91
92 - log <address> <facility> [max_level]
93 - maxconn <number>
94 - uid <user id>
95 - gid <group id>
96 - chroot <directory>
97 - nbproc <number>
98 - daemon
99 - debug
100 - quiet
willy tarreaufe2c5c12005-12-17 14:14:34 +0100101 - pidfile <file>
willy tarreaueedaa9f2005-12-17 14:08:03 +0100102
1031.1) Event logging
104------------------
105Most events are logged : start, stop, servers going up and down, connections and
106errors. Each event generates a syslog message which can be sent to up to 2
107servers. The syntax is :
108
109 log <ip_address> <facility> [max_level]
110
111Connections are logged at level "info". Services initialization and servers
112going up are logged at level "notice", termination signals are logged at
113"warning", and definitive service termination, as well as loss of servers are
114logged at level "alert". The optional parameter <max_level> specifies above
115what level messages should be sent. Level can take one of these 8 values :
116
117 emerg, alert, crit, err, warning, notice, info, debug
118
119For backwards compatibility with versions 1.1.16 and earlier, the default level
120value is "debug" if not specified.
121
122Permitted facilities are :
123 kern, user, mail, daemon, auth, syslog, lpr, news,
124 uucp, cron, auth2, ftp, ntp, audit, alert, cron2,
125 local0, local1, local2, local3, local4, local5, local6, local7
126
127According to RFC3164, messages are truncated to 1024 bytes before being emitted.
128
129Example :
130---------
131 global
132 log 192.168.2.200 local3
133 log 127.0.0.1 local4 notice
134
1351.2) limiting the number of connections
136---------------------------------------
137It is possible and recommended to limit the global number of per-process
138connections. Since one connection includes both a client and a server, it
139means that the max number of TCP sessions will be about the double of this
140number. It's important to understand this when trying to find best values
141for 'ulimit -n' before starting the proxy. To anticipate the number of
142sockets needed, all these parameters must be counted :
143
144 - 1 socket per incoming connection
145 - 1 socket per outgoing connection
146 - 1 socket per address/port/proxy tuple.
147 - 1 socket per server being health-checked
148 - 1 socket for all logs
149
150In simple configurations where each proxy only listens one one address/port,
151set the limit of file descriptors (ulimit -n) to
152(2 * maxconn + nbproxies + nbservers + 1). In a future release, haproxy may
153be able to set this value itself.
154
1551.3) Drop of priviledges
156------------------------
157In order to reduce the risk and consequences of attacks, in the event where a
158yet non-identified vulnerability would be successfully exploited, it's possible
159to lower the process priviledges and even isolate it in a riskless directory.
160
161In the 'global' section, the 'uid' parameter sets a numerical user identifier
162which the process will switch to after binding its listening sockets. The value
163'0', which normally represents the super-user, here indicates that the UID must
164not change during startup. It's the default behaviour. The 'gid' parameter does
165the same for the group identifier. It's particularly advised against use of
166generic accounts such as 'nobody' because it has the same consequences as using
167'root' if other services use them.
168
169The 'chroot' parameter makes the process isolate itself in an empty directory
170just before switching its UID. This type of isolation (chroot) can sometimes
171be worked around on certain OS (Linux, Solaris), provided that the attacker
172has gained 'root' priviledges and has the ability to use or create a directory.
173For this reason, it's capital to use a dedicated directory and not to share one
174between several services of different nature. To make isolation more resistant,
175it's recommended to use an empty directory without any right, and to change the
176UID of the process so that it cannot do anything there.
177
178Note: in the event where such a vulnerability would be exploited, it's most
179likely that first attempts would kill the process due to 'Segmentation Fault',
180'Bus Error' or 'Illegal Instruction' signals. Eventhough it's true that
181isolating the server reduces the risks of intrusion, it's sometimes useful to
182find why a process dies, via the analysis of a 'core' file, although very rare
183(the last bug of this sort was fixed in 1.1.9). For security reasons, most
184systems disable the generation of core file when a process changes its UID. So
185the two workarounds are either to start the process from a restricted user
186account, which will not be able to chroot itself, or start it as root and not
187change the UID. In both cases the core will be either in the start or the chroot
188directories. Do not forget to allow core dumps prior to start the process :
189
190# ulimit -c unlimited
191
192Example :
193---------
194
195 global
196 uid 30000
197 gid 30000
198 chroot /var/chroot/haproxy
199
2001.4) Startup modes
201------------------
202The service can start in several different :
203 - foreground / background
204 - quiet / normal / debug
205
206The default mode is normal, foreground, which means that the program doesn't
207return once started. NEVER EVER use this mode in a system startup script, or
208the system won't boot. It needs to be started in background, so that it
209returns immediately after forking. That's accomplished by the 'daemon' option
210in the 'global' section, which is the equivalent of the '-D' command line
211argument.
212
213Moreover, certain alert messages are still sent to the standard output even
214in 'daemon' mode. To make them disappear, simply add the 'quiet' option in the
215'global' section. This option has no command-line equivalent.
216
217Last, the 'debug' mode, enabled with the 'debug' option in the 'global' section,
218and which is equivalent of the '-d' option, allows deep TCP/HTTP analysis, with
219timestamped display of each connection, disconnection, and HTTP headers for both
220ways. This mode is incompatible with 'daemon' and 'quiet' modes for obvious
221reasons.
222
2231.5) Increasing the overall processing power
224--------------------------------------------
225On multi-processor systems, it may seem to be a shame to use only one processor,
willy tarreau982249e2005-12-18 00:57:06 +0100226eventhough the load needed to saturate a recent processor is far above common
willy tarreaueedaa9f2005-12-17 14:08:03 +0100227usage. Anyway, for very specific needs, the proxy can start several processes
228between which the operating system will spread the incoming connections. The
229number of processes is controlled by the 'nbproc' parameter in the 'global'
230section. It defaults to 1, and obviously works only in 'daemon' mode.
231
232Example :
233---------
234
235 global
236 daemon
237 quiet
238 nbproc 2
239
240
willy tarreaufe2c5c12005-12-17 14:14:34 +01002411.6) Helping process management
242-------------------------------
243Haproxy now supports the notion of pidfile. If the '-p' command line argument,
244or the 'pidfile' global option is followed with a file name, this file will be
245removed, then filled with all children's pids, one per line (only in daemon
246mode). This file is NOT within the chroot, which allows to work with a readonly
247 chroot. It will be owned by the user starting the process, and will have
248permissions 0644.
249
250Example :
251---------
252
253 global
254 daemon
255 quiet
256 nbproc 2
257 pidfile /var/run/haproxy-private.pid
258
259 # to stop only those processes among others :
260 # kill $(</var/run/haproxy-private.pid)
261
262
willy tarreaueedaa9f2005-12-17 14:08:03 +01002632) Declaration of a listening service
264=====================================
265
266Service sections start with the 'listen' keyword :
267
268 listen <instance_name> [ <IP_address>:<port_range>[,...] ]
269
270- <instance_name> is the name of the instance. This name will be reported in
271 logs, so it is good to have it reflect the proxied service. No unicity test
272 is done on this name, and it's not mandatory for it to be unique, but highly
273 recommended.
274
275- <IP_address> is the IP address the proxy binds to. Empty address, '*' and
276 '0.0.0.0' all mean that the proxy listens to all valid addresses on the
277 system.
278
279- <port_range> is either a unique port, or a port range for which the proxy will
280 accept connections for the IP address specified above. This range can be :
281 - a numerical port (ex: '80')
282 - a dash-delimited ports range explicitly stating the lower and upper bounds
283 (ex: '2000-2100') which are included in the range.
284
285 Particular care must be taken against port ranges, because every <addr:port>
286 couple consumes one socket (=a file descriptor), so it's easy to eat lots of
287 descriptors with a simple range. The <addr:port> couple must be used only once
288 among all instances running on a same system. Please note that attaching to
289 ports lower than 1024 need particular priviledges to start the program, which
290 are independant of the 'uid' parameter.
291
292- the <IP_address>:<port_range> couple may be repeated indefinitely to require
293 the proxy to listen to other addresses and/or ports. To achieve this, simply
294 separate them with a coma.
295
296Examples :
297---------
298 listen http_proxy :80
299 listen x11_proxy 127.0.0.1:6000-6009
300 listen smtp_proxy 127.0.0.1:25,127.0.0.1:587
301 listen ldap_proxy :389,:663
302
303In the event that all addresses do not fit line width, it's preferable to
304detach secondary addresses on other lines with the 'bind' keyword. If this
305keyword is used, it's not even necessary to specify the first address on the
306'listen' line, which sometimes makes multiple configuration handling easier :
307
308 bind [ <IP_address>:<port_range>[,...] ]
309
310Examples :
311----------
312 listen http_proxy
313 bind :80,:443
314 bind 10.0.0.1:10080,10.0.0.1:10443
315
3162.1) Inhibiting a service
317-------------------------
318A service may be disabled for maintenance reasons, without needing to comment
319out the whole section, simply by specifying the 'disabled' keyword in the
320section to be disabled :
321
322 listen smtp_proxy 0.0.0.0:25
323 disabled
324
325Note: the 'enabled' keyword allows to enable a service which has been disabled
326 previously by a default configuration.
327
3282.2) Modes of operation
329-----------------------
330A service can work in 3 different distinct modes :
331 - TCP
332 - HTTP
333 - monitoring
334
335TCP mode
336--------
337In this mode, the service relays TCP connections as soon as they're established,
338towards one or several servers. No processing is done on the stream. It's only
339an association of source(addr:port) -> destination(addr:port). To use this mode,
340you must specify 'mode tcp' in the 'listen' section. This is the default mode.
341
342Example :
343---------
344 listen smtp_proxy 0.0.0.0:25
345 mode tcp
346
347HTTP mode
348---------
349In this mode, the service relays TCP connections towards one or several servers,
350when it has enough informations to decide, which normally means that all HTTP
351headers have been read. Some of them may be scanned for a cookie or a pattern
352matching a regex. To use this mode, specify 'mode http' in the 'listen' section.
353
354Example :
355---------
356 listen http_proxy 0.0.0.0:80
357 mode http
358
359Health-checking mode
360--------------------
361This mode provides a way for external components to check the proxy's health.
362It is meant to be used with intelligent load-balancers which can use send/expect
363scripts to check for all of their servers' availability. This one simply accepts
willy tarreau197e8ec2005-12-17 14:10:59 +0100364the connection, returns the word 'OK' and closes it. If the 'option httpchk' is
365set, then the reply will be 'HTTP/1.0 200 OK' with no data, so that it can be
366tested from a tool which supports HTTP health-checks. To enable it, simply
willy tarreaueedaa9f2005-12-17 14:08:03 +0100367specify 'health' as the working mode :
368
369Example :
370---------
willy tarreau197e8ec2005-12-17 14:10:59 +0100371 # simple response : 'OK'
willy tarreaueedaa9f2005-12-17 14:08:03 +0100372 listen health_check 0.0.0.0:60000
373 mode health
374
willy tarreau197e8ec2005-12-17 14:10:59 +0100375 # HTTP response : 'HTTP/1.0 200 OK'
376 listen http_health_check 0.0.0.0:60001
377 mode health
378 option httpchk
379
willy tarreaueedaa9f2005-12-17 14:08:03 +0100380
3812.3) Limiting the number of simultaneous connections
382----------------------------------------------------
383The 'maxconn' parameter allows a proxy to refuse connections above a certain
384amount of simultaneous ones. When the limit is reached, it simply stops
385listening, but the system may still be accepting them because of the back log
willy tarreau982249e2005-12-18 00:57:06 +0100386queue. These connections will be processed later when other ones have freed
willy tarreaueedaa9f2005-12-17 14:08:03 +0100387some slots. This provides a serialization effect which helps very fragile
388servers resist to high loads. Se further for system limitations.
389
390Example :
391---------
392 listen tiny_server 0.0.0.0:80
393 maxconn 10
394
395
3962.4) Soft stop
397--------------
398It is possible to stop services without breaking existing connections by the
399sending of the SIG_USR1 signal to the process. All services are then put into
400soft-stop state, which means that they will refuse to accept new connections,
401except for those which have a non-zero value in the 'grace' parameter, in which
402case they will still accept connections for the specified amount of time, in
403milliseconds. This allows to tell a load-balancer that the service is failing,
404while still doing the job during the time it needs to detect it.
405
406Note: active connections are never killed. In the worst case, the user will have
407to wait for all of them to close or to time-out, or simply kill the process
408normally (SIG_TERM). The default 'grace' value is '0'.
409
410Example :
411---------
412 # enter soft stop after 'killall -USR1 haproxy'
413 # the service will still run 10 seconds after the signal
414 listen http_proxy 0.0.0.0:80
415 mode http
416 grace 10000
417
418 # this port is dedicated to a load-balancer, and must fail immediately
419 listen health_check 0.0.0.0:60000
420 mode health
421 grace 0
422
423
4242.5) Connections expiration time
425--------------------------------
426It is possible (and recommended) to configure several time-outs on TCP
427connections. Three independant timers are adjustable with values specified
428in milliseconds. A session will be terminated if either one of these timers
429expire.
430
431 - the time we accept to wait for data from the client, or for the client to
432 accept data : 'clitimeout' :
433
434 # client time-out set to 2mn30.
435 clitimeout 150000
436
437 - the time we accept to wait for data from the server, or for the server to
438 accept data : 'srvtimeout' :
439
440 # server time-out set to 30s.
441 srvtimeout 30000
442
443 - the time we accept to wait for a connection to establish on a server :
444 'contimeout' :
445
446 # we give up if the connection does not complete within 4 seconds
447 contimeout 4000
448
449Notes :
450-------
451 - 'contimeout' and 'srvtimeout' have no sense on 'health' mode servers ;
452 - under high loads, or with a saturated or defective network, it's possible
453 that some packets get lost. Since the first TCP retransmit only happens
454 after 3 seconds, a time-out equal to, or lower than 3 seconds cannot
455 compensate for a packet loss. A 4 seconds time-out seems a reasonable
456 minimum which will considerably reduce connection failures.
457
4582.6) Attempts to reconnect
459--------------------------
460After a connection failure to a server, it is possible to retry, potentially
461on another server. This is useful if health-checks are too rare and you don't
462want the clients to see the failures. The number of attempts to reconnect is
463set by the 'retries' paramter.
464
465Example :
466---------
467 # we can retry 3 times max after a failure
468 retries 3
469
470
4712.7) Address of the dispatch server (deprecated)
472------------------------------------------------
473The server which will be sent all new connections is defined by the 'dispatch'
474parameter, in the form <address>:<port>. It generally is dedicated to unknown
475connections and will assign them a cookie, in case of HTTP persistence mode,
476or simply is a single server in case of generic TCP proxy. This old mode is only
477provided for backwards compatibility, but doesn't allow to check remote servers
478state, and has a rather limited usage. All new setups should switch to 'balance'
479mode. The principle of the dispatcher is to be able to perform the load
480balancing itself, but work only on new clients so that the server doesn't need
481to be a big machine.
482
483Example :
484---------
485 # all new connections go there
486 dispatch 192.168.1.2:80
487
488Note :
489------
490This parameter has no sense for 'health' servers, and is incompatible with
491'balance' mode.
492
493
4942.8) Outgoing source address
495----------------------------
496It is often necessary to bind to a particular address when connecting to some
497remote hosts. This is done via the 'source' parameter which is a per-proxy
498parameter. A newer version may allow to fix different sources to reach different
499servers. The syntax is 'source <address>[:<port>]', where <address> is a valid
500local address (or '0.0.0.0' or '*' or empty to let the system choose), and
501<port> is an optional parameter allowing the user to force the source port for
502very specific needs. If the port is not specified or is '0', the system will
503choose a free port. Note that as of version 1.1.18, the servers health checks
504are also performed from the same source.
505
506Examples :
507----------
508 listen http_proxy *:80
509 # all connections take 192.168.1.200 as source address
510 source 192.168.1.200:0
511
512 listen rlogin_proxy *:513
513 # use address 192.168.1.200 and the reserved port 900 (needs to be root)
514 source 192.168.1.200:900
515
516
5172.9) Setting the cookie name
518----------------------------
519In HTTP mode, it is possible to look for a particular cookie which will contain
520a server identifier which should handle the connection. The cookie name is set
521via the 'cookie' parameter.
522
523Example :
524---------
525 listen http_proxy :80
526 mode http
527 cookie SERVERID
528
529It is possible to change the cookie behaviour to get a smarter persistence,
530depending on applications. It is notably possible to delete or modify a cookie
531emitted by a server, insert a cookie identifying the server in an HTTP response
532and even add a header to tell upstream caches not to cache this response.
533
534Examples :
535----------
536
537To remove the cookie for direct accesses (ie when the server matches the one
538which was specified in the client cookie) :
539
540 cookie SERVERID indirect
541
542To replace the cookie value with the one assigned to the server if any (no
543cookie will be created if the server does not provide one, nor if the
544configuration does not provide one). This lets the application put the cookie
545exactly on certain pages (eg: successful authentication) :
546
547 cookie SERVERID rewrite
548
549To create a new cookie and assign the server identifier to it (in this case, all
550servers should be associated with a valid cookie, since no cookie will simply
551delete the cookie from the client's browser) :
552
553 cookie SERVERID insert
554
555To insert a cookie and ensure that no upstream cache will store it, add the
556'nocache' option :
557
558 cookie SERVERID insert nocache
559
560To insert a cookie only after a POST request, add 'postonly' after 'insert'.
561This has the advantage that there's no risk of caching, and that all pages
562seen before the POST one can still be cached :
563
564 cookie SERVERID insert postonly
565
566Notes :
567-----------
568- it is possible to combine 'insert' with 'indirect' or 'rewrite' to adapt to
569 applications which already generate the cookie with an invalid content.
570
571- in the case where 'insert' and 'indirect' are both specified, the cookie is
572 never transmitted to the server, since it wouldn't understand it. This is
573 the most application-transparent mode.
574
575- it is particularly recommended to use 'nocache' in 'insert' mode if any
576 upstream HTTP/1.0 cache is susceptible to cache the result, because this may
577 lead to many clients going to the same server, or even worse, some clients
578 having their server changed while retrieving a page from the cache.
579
580- when the application is well known and controlled, the best method is to
581 only add the persistence cookie on a POST form because it's up to the
582 application to select which page it wants the upstream servers to cache.
583 In this case, you would use 'insert postonly indirect'.
584
5852.10) Associating a cookie value with a server
586----------------------------------------------
587In HTTP mode, it's possible to associate a cookie value to each server. This
588was initially used in combination with 'dispatch' mode to handle direct accesses
589but it is now the standard way of doing the load balancing. The syntax is :
590
591 server <identifier> <address>:<port> cookie <value>
592
593- <identifier> is any name which can be used to identify the server in the logs.
594- <address>:<port> specifies where the server is bound.
595- <value> is the value to put in or to read from the cookie.
596
597Example : the 'SERVERID' cookie can be either 'server01' or 'server02'
598---------
599 listen http_proxy :80
600 mode http
601 cookie SERVERID
602 dispatch 192.168.1.100:80
603 server web1 192.168.1.1:80 cookie server01
604 server web2 192.168.1.2:80 cookie server02
605
606Warning : the syntax has changed since version 1.0 !
607---------
608
6093) Autonomous load balancer
610===========================
611
612The proxy can perform the load-balancing itself, both in TCP and in HTTP modes.
613This is the most interesting mode which obsoletes the old 'dispatch' mode
614described above. It has advantages such as server health monitoring, multiple
615port binding and port mapping. To use this mode, the 'balance' keyword is used,
616followed by the selected algorithm. As of version 1.1.23, only 'roundrobin' is
617available, which is also the default value if unspecified. In this mode, there
618will be no dispatch address, but the proxy needs at least one server.
619
620Example : same as the last one, with internal load balancer
621---------
622
623 listen http_proxy :80
624 mode http
625 cookie SERVERID
626 balance roundrobin
627 server web1 192.168.1.1:80 cookie server01
628 server web2 192.168.1.2:80 cookie server02
629
630
631Since version 1.1.22, it is possible to automatically determine on which port
632the server will get the connection, depending on the port the client connected
633to. Indeed, there now are 4 possible combinations for the server's <port> field:
634
635 - unspecified or '0' :
636 the connection will be sent to the same port as the one on which the proxy
637 received the client connection itself.
638
639 - numerical value (the only one supported in versions earlier than 1.1.22) :
640 the connection will always be sent to the specified port.
641
642 - '+' followed by a numerical value :
643 the connection will be sent to the same port as the one on which the proxy
644 received the connection, plus this value.
645
646 - '-' followed by a numerical value :
647 the connection will be sent to the same port as the one on which the proxy
648 received the connection, minus this value.
649
650Examples :
651----------
652
653# same as previous example
654
655 listen http_proxy :80
656 mode http
657 cookie SERVERID
658 balance roundrobin
659 server web1 192.168.1.1 cookie server01
660 server web2 192.168.1.2 cookie server02
661
662# simultaneous relaying of ports 80, 81 and 8080-8089
663
664 listen http_proxy :80,:81,:8080-8089
665 mode http
666 cookie SERVERID
667 balance roundrobin
668 server web1 192.168.1.1 cookie server01
669 server web2 192.168.1.2 cookie server02
670
671# relaying of TCP ports 25, 389 and 663 to ports 1025, 1389 and 1663
672
673 listen http_proxy :25,:389,:663
674 mode tcp
675 balance roundrobin
676 server srv1 192.168.1.1:+1000
677 server srv2 192.168.1.2:+1000
678
679
willy tarreau197e8ec2005-12-17 14:10:59 +01006803.1) Server monitoring
681----------------------
willy tarreaueedaa9f2005-12-17 14:08:03 +0100682
683It is possible to check the servers status by trying to establish TCP
684connections or even sending HTTP requests to them. A server which fails to
685reply to health checks as expected will not be used by the load balancing
686algorithms. To enable monitoring, add the 'check' keyword on a server line.
687It is possible to specify the interval between tests (in milliseconds) with
688the 'inter' parameter, the number of failures supported before declaring that
689the server has fallen down with the 'fall' parameter, and the number of valid
690checks needed for the server to fully get up with the 'rise' parameter. Since
691version 1.1.22, it is also possible to send checks to a different port
692(mandatory when none is specified) with the 'port' parameter. The default
693values are the following ones :
694
695 - inter : 2000
696 - rise : 2
697 - fall : 3
698 - port : default server port
699
700The default mode consists in establishing TCP connections only. But in certain
701types of application failures, it is often that the server continues to accept
702connections because the system does it itself while the application is running
703an endless loop, or is completely stuck. So in version 1.1.16 were introduced
704HTTP health checks which only performed simple lightweight requests and analysed
705the response. Now, as of version 1.1.23, it is possible to change the HTTP
706method, the URI, and the HTTP version string (which even allows to send headers
707with a dirty trick). To enable HTTP health-checks, use 'option httpchk'.
708
709By default, requests use the 'OPTIONS' method because it's very light and easy
710to filter from logs, and does it on '/'. Only HTTP responses 2xx and 3xx are
711considered valid ones, and only if they come before the time to send a new
712request is reached ('inter' parameter). If some servers block this type of
713request, 3 other forms help to forge a request :
714
715 - option httpchk -> OPTIONS / HTTP/1.0
716 - option httpchk URI -> OPTIONS <URI> HTTP/1.0
717 - option httpchk METH URI -> <METH> <URI> HTTP/1.0
718 - option httpchk METH URI VER -> <METH> <URI> <VER>
719
720See examples below.
721
722Since version 1.1.17, it is possible to specify backup servers. These servers
723are only sollicited when no other server is available. This may only be useful
724to serve a maintenance page, or define one active and one backup server (seldom
725used in TCP mode). To make a server a backup one, simply add the 'backup' option
726on its line. These servers also support cookies, so if a cookie is specified for
727a backup server, clients assigned to this server will stick to it even when the
728other ones come back. Conversely, if no cookie is assigned to such a server,
729the clients will get their cookies removed (empty cookie = removal), and will
730be balanced against other servers once they come back. Please note that there
731is no load-balancing among backup servers. If there are several backup servers,
732the second one will only be used when the first one dies, and so on.
733
734Since version 1.1.17, it is also possible to visually check the status of all
735servers at once. For this, you just have to send a SIGHUP signal to the proxy.
736The servers status will be dumped into the logs at the 'notice' level, as well
737as on <stderr> if not closed. For this reason, it's always a good idea to have
738one local log server at the 'notice' level.
739
willy tarreau982249e2005-12-18 00:57:06 +0100740Since version 1.1.28 and 1.2.1, if an instance loses all its servers, an
741emergency mesasge will be sent in the logs to inform the administator that an
742immediate action must be taken.
743
744
willy tarreaueedaa9f2005-12-17 14:08:03 +0100745Examples :
746----------
747# same setup as in paragraph 3) with TCP monitoring
748 listen http_proxy 0.0.0.0:80
749 mode http
750 cookie SERVERID
751 balance roundrobin
752 server web1 192.168.1.1:80 cookie server01 check
753 server web2 192.168.1.2:80 cookie server02 check inter 500 rise 1 fall 2
754
755# same with HTTP monitoring via 'OPTIONS / HTTP/1.0'
756 listen http_proxy 0.0.0.0:80
757 mode http
758 cookie SERVERID
759 balance roundrobin
760 option httpchk
761 server web1 192.168.1.1:80 cookie server01 check
762 server web2 192.168.1.2:80 cookie server02 check inter 500 rise 1 fall 2
763
764# same with HTTP monitoring via 'OPTIONS /index.html HTTP/1.0'
765 listen http_proxy 0.0.0.0:80
766 mode http
767 cookie SERVERID
768 balance roundrobin
769 option httpchk /index.html
770 server web1 192.168.1.1:80 cookie server01 check
771 server web2 192.168.1.2:80 cookie server02 check inter 500 rise 1 fall 2
772
773# same with HTTP monitoring via 'HEAD /index.jsp? HTTP/1.1\r\nHost: www'
774 listen http_proxy 0.0.0.0:80
775 mode http
776 cookie SERVERID
777 balance roundrobin
778 option httpchk HEAD /index.jsp? HTTP/1.1\r\nHost:\ www
779 server web1 192.168.1.1:80 cookie server01 check
780 server web2 192.168.1.2:80 cookie server02 check inter 500 rise 1 fall 2
781
782# automatic insertion of a cookie in the server's response, and automatic
783# deletion of the cookie in the client request, while asking upstream caches
784# not to cache replies.
785 listen web_appl 0.0.0.0:80
786 mode http
787 cookie SERVERID insert nocache indirect
788 balance roundrobin
789 server web1 192.168.1.1:80 cookie server01 check
790 server web2 192.168.1.2:80 cookie server02 check
791
792# same with off-site application backup and local error pages server
793 listen web_appl 0.0.0.0:80
794 mode http
795 cookie SERVERID insert nocache indirect
796 balance roundrobin
797 server web1 192.168.1.1:80 cookie server01 check
798 server web2 192.168.1.2:80 cookie server02 check
799 server web-backup 192.168.2.1:80 cookie server03 check backup
800 server web-excuse 192.168.3.1:80 check backup
801
802# SMTP+TLS relaying with heakth-checks and backup servers
803
804 listen http_proxy :25,:587
805 mode tcp
806 balance roundrobin
807 server srv1 192.168.1.1 check port 25 inter 30000 rise 1 fall 2
808 server srv2 192.168.1.2 backup
809
810
8113.2) Redistribute connections in case of failure
812------------------------------------------------
813In HTTP mode, if a server designated by a cookie does not respond, the clients
814may definitely stick to it because they cannot flush the cookie, so they will
815not be able to access the service anymore. Specifying 'redispatch' will allow
816the proxy to break their persistence and redistribute them to working servers.
817
818Example :
819---------
820 listen http_proxy 0.0.0.0:80
821 mode http
822 cookie SERVERID
823 dispatch 192.168.1.100:80
824 server web1 192.168.1.1:80 cookie server01
825 server web2 192.168.1.2:80 cookie server02
826 redispatch # send back to dispatch in case of connection failure
827
828Up to, and including version 1.1.16, this parameter only applied to connection
829failures. Since version 1.1.17, it also applies to servers which have been
830detected as failed by the health check mechanism. Indeed, a server may be broken
831but still accepting connections, which would not solve every case. But it is
832possible to conserve the old behaviour, that is, make a client insist on trying
833to connect to a server even if it is said to be down, by setting the 'persist'
834option :
835
836 listen http_proxy 0.0.0.0:80
837 mode http
838 option persist
839 cookie SERVERID
840 dispatch 192.168.1.100:80
841 server web1 192.168.1.1:80 cookie server01
842 server web2 192.168.1.2:80 cookie server02
843 redispatch # send back to dispatch in case of connection failure
844
845
8464) Additionnal features
847=======================
848
849Other features are available. They are transparent mode, event logging and
850header rewriting/filtering.
851
8524.1) Transparent mode
853---------------------
854In HTTP mode, the 'transparent' keyword allows to intercept sessions which are
855routed through the system hosting the proxy. This mode was implemented as a
856replacement for the 'dispatch' mode, since connections without cookie will be
857sent to the original address while known cookies will be sent to the servers.
858This mode implies that the system can redirect sessions to a local port.
859
860Example :
861---------
862 listen http_proxy 0.0.0.0:65000
863 mode http
864 transparent
865 cookie SERVERID
866 server server01 192.168.1.1:80
867 server server02 192.168.1.2:80
868
869 # iptables -t nat -A PREROUTING -i eth0 -p tcp -d 192.168.1.100 \
870 --dport 80 -j REDIRECT --to-ports 65000
871
872Note :
873------
874If the port is left unspecified on the server, the port the client connected to
875will be used. This allows to relay a full port range without using transparent
876mode nor thousands of file descriptors, provided that the system can redirect
877sessions to local ports.
878
879Example :
880---------
881 # redirect all ports to local port 65000, then forward to the server on the
882 # original port.
883 listen http_proxy 0.0.0.0:65000
884 mode tcp
885 server server01 192.168.1.1 check port 60000
886 server server02 192.168.1.2 check port 60000
887
888 # iptables -t nat -A PREROUTING -i eth0 -p tcp -d 192.168.1.100 \
889 -j REDIRECT --to-ports 65000
890
891
8924.2) Event logging
893------------------
willy tarreauc1cae632005-12-17 14:12:23 +01008944.2.1) Log levels
895-----------------
willy tarreau197e8ec2005-12-17 14:10:59 +0100896TCP and HTTP connections can be logged with informations such as date, time,
897source IP address, destination address, connection duration, response times,
898HTTP request, the HTTP return code, number of bytes transmitted, the conditions
899in which the session ended, and even exchanged cookies values, to track a
900particular user's problems for example. All messages are sent to up to two
901syslog servers. Consult section 1.1 for more info about log facilities. The
902syntax follows :
willy tarreaueedaa9f2005-12-17 14:08:03 +0100903
willy tarreau197e8ec2005-12-17 14:10:59 +0100904 log <address_1> <facility_1> [max_level_1]
905 log <address_2> <facility_2> [max_level_2]
906or
willy tarreaueedaa9f2005-12-17 14:08:03 +0100907 log global
908
willy tarreau197e8ec2005-12-17 14:10:59 +0100909Note :
910------
911The particular syntax 'log global' means that the same log configuration as the
912'global' section will be used.
willy tarreaueedaa9f2005-12-17 14:08:03 +0100913
willy tarreau197e8ec2005-12-17 14:10:59 +0100914Example :
willy tarreaueedaa9f2005-12-17 14:08:03 +0100915---------
916 listen http_proxy 0.0.0.0:80
917 mode http
918 log 192.168.2.200 local3
919 log 192.168.2.201 local4
920
willy tarreauc1cae632005-12-17 14:12:23 +01009214.2.2) Log format
922-----------------
923By default, connections are logged at the TCP level, as soon as the session
924establishes between the client and the proxy. By enabling the 'tcplog' option,
925the proxy will wait until the session ends to generate an enhanced log
926containing more information such as session duration and its state during the
927disconnection.
928
willy tarreau982249e2005-12-18 00:57:06 +0100929Example :
930---------
931 listen relais-tcp 0.0.0.0:8000
932 mode tcp
933 option tcplog
934 log 192.168.2.200 local3
935
936>>> haproxy[18989]: 127.0.0.1:34550 [15/Oct/2003:15:24:28] relais-tcp Srv1 0/5007 0 --
937
willy tarreauc1cae632005-12-17 14:12:23 +0100938Another option, 'httplog', provides more detailed information about HTTP
939contents, such as the request and some cookies. In the event where an external
940component would establish frequent connections to check the service, logs may be
941full of useless lines. So it is possible not to log any session which didn't
942transfer any data, by the setting of the 'dontlognull' option. This only has
943effect on sessions which are established then closed.
willy tarreaueedaa9f2005-12-17 14:08:03 +0100944
willy tarreau197e8ec2005-12-17 14:10:59 +0100945Example :
willy tarreaueedaa9f2005-12-17 14:08:03 +0100946---------
947 listen http_proxy 0.0.0.0:80
948 mode http
949 option httplog
950 option dontlognull
951 log 192.168.2.200 local3
952
willy tarreau982249e2005-12-18 00:57:06 +0100953>>> haproxy[674]: 127.0.0.1:33319 [15/Oct/2003:08:31:57] relais-http Srv1 9/7/147/723 200 243 - - ---- "HEAD / HTTP/1.0"
954
955The problem when logging at end of connection is that you have no clue about
956what is happening during very long sessions. To workaround this problem, a
957new option 'logasap' has been introduced in 1.1.28/1.2.1. When specified, the
958proxy will log as soon as possible, just before data transfer begins. This means
959that in case of TCP, it will still log the connection status to the server, and
960in case of HTTP, it will log just after processing the server headers. In this
961case, the number of bytes reported is the number of header bytes sent to the
962client.
963
964In order to avoid confusion with normal logs, the total time field and the
965number of bytes are prefixed with a '+' sign which mean that real numbers are
966certainly bigger.
967
968Example :
969---------
970
971 listen http_proxy 0.0.0.0:80
972 mode http
973 option httplog
974 option dontlognull
975 option logasap
976 log 192.168.2.200 local3
977
978>>> haproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17] relais-http Srv1 9/7/14/+30 200 +243 - - ---- "GET /image.iso HTTP/1.0"
979
980
willy tarreauc1cae632005-12-17 14:12:23 +01009814.2.3) Timing events
982--------------------
983Timers provide a great help in trouble shooting network problems. All values
984are reported in milliseconds (ms). In HTTP mode, four control points are
985reported under the form 'Tq/Tc/Tr/Tt' :
986
987 - Tq: total time to get the client request.
988 It's the time elapsed between the moment the client connection was accepted
989 and the moment the proxy received the last HTTP header. The value '-1'
990 indicates that the end of headers (empty line) has never been seen.
991
992 - Tc: total time to establish the TCP connection to the server.
993 It's the time elapsed between the moment the proxy sent the connection
994 request, and the moment it was acknowledged, or between the TCP SYN packet
995 and the matching SYN/ACK in return. The value '-1' means that the
996 connection never established.
997
998 - Tr: server response time. It's the time elapsed between the moment the
999 TCP connection was established to the server and the moment it send its
1000 complete response header. It purely shows its request processing time,
1001 without the network overhead due to the data transmission. The value '-1'
1002 means that the last the response header (empty line) was never seen.
1003
1004 - Tt: total session duration time, between the moment the proxy accepted it
willy tarreau982249e2005-12-18 00:57:06 +01001005 and the moment both ends were closed. The exception is when the 'logasap'
1006 option is specified. In this case, it only equals (Tq+Tc+Tr), and is
1007 prefixed with a '+' sign. From this field, we can deduce Td, the data
1008 transmission time, by substracting other timers when valid :
willy tarreauc1cae632005-12-17 14:12:23 +01001009
1010 Td = Tt - (Tq + Tc + Tr)
1011
1012 Timers with '-1' values have to be excluded from this equation.
1013
1014In TCP mode ('option tcplog'), only Tc and Tt are reported.
1015
1016These timers provide precious indications on trouble causes. Since the TCP
1017protocol defines retransmit delays of 3, 6, 12... seconds, we know for sure
1018that timers close to multiples of 3s are nearly always related to packets lost
1019due to network problems (wires or negociation). Moreover, if <Tt> is close to
1020a timeout value specified in the configuration, it often means that a session
1021has been aborted on time-out.
1022
1023Most common cases :
1024
1025 - If Tq is close to 3000, a packet has probably been lost between the client
1026 and the proxy.
1027 - If Tc is close to 3000, a packet has probably been lost between the server
1028 and the proxy during the server connection phase. This one should always be
1029 very low (less than a few tens).
1030 - If Tr is nearly always lower than 3000 except some rare values which seem to
1031 be the average majored by 3000, there are probably some packets lost between
1032 the proxy and the server.
1033 - If Tt is often slightly higher than a time-out, it's often because the
1034 client and the server use HTTP keep-alive and the session is maintained
1035 after the response ends. Se further for how to disable HTTP keep-alive.
1036
1037Other cases ('xx' means any value to be ignored) :
1038 -1/xx/xx/Tt : the client was not able to send its complete request in time,
1039 or that it aborted it too early.
1040 Tq/-1/xx/Tt : the connection could not establish on the server. Either it
1041 refused it or it timed out after Tt-Tq ms.
1042 Tq/Tc/-1/Tt : the server has accepted the connection but did not return a
1043 complete response in time, or it closed its connexion
1044 unexpectedly, after Tt-(Tq+Tc) ms.
1045
10464.2.4) Session state at disconnection
1047-------------------------------------
1048TCP and HTTP logs provide a session completion indicator. It's a 4-characters
1049(2 in TCP) field preceeding the HTTP request, and indicating :
willy tarreau197e8ec2005-12-17 14:10:59 +01001050 - On the first character, a code reporting the first event which caused the
1051 session to terminate :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001052
willy tarreau197e8ec2005-12-17 14:10:59 +01001053 C : the TCP session was aborted by the client.
1054 S : the TCP session was aborted by the server, or the server refused it.
1055 P : the session was abordted prematurely by the proxy, either because of
willy tarreau982249e2005-12-18 00:57:06 +01001056 an internal error, because a DENY filter was matched, or because of
1057 a security check which detected a dangerous error in server
1058 response.
willy tarreau197e8ec2005-12-17 14:10:59 +01001059 c : the client time-out expired first.
1060 s : the server time-out expired first.
1061 - : normal session completion.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001062
willy tarreau197e8ec2005-12-17 14:10:59 +01001063 - on the second character, the HTTP session state when it was closed :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001064
willy tarreau197e8ec2005-12-17 14:10:59 +01001065 R : waiting for complete REQUEST from the client
1066 C : waiting for CONNECTION to establish on the server
willy tarreau982249e2005-12-18 00:57:06 +01001067 H : processing server HEADERS
willy tarreau197e8ec2005-12-17 14:10:59 +01001068 D : the session was in the DATA phase
1069 L : the proxy was still transmitting LAST data to the client while the
1070 server had already finished.
1071 - : normal session completion after end of data transfer.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001072
willy tarreau197e8ec2005-12-17 14:10:59 +01001073 - the third character tells whether the persistence cookie was provided by
willy tarreauc1cae632005-12-17 14:12:23 +01001074 the client (only in HTTP mode) :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001075
willy tarreau197e8ec2005-12-17 14:10:59 +01001076 N : the client provided NO cookie.
1077 I : the client provided an INVALID cookie matching no known server.
1078 D : the client provided a cookie designating a server which was DOWN,
1079 so either the 'persist' option was used and the client was sent to
1080 this server, or it was not set and the client was redispatched to
1081 another server.
1082 V : the client provided a valid cookie, and was sent to the associated
1083 server.
1084 - : does not apply (no cookie set in configuration).
willy tarreaueedaa9f2005-12-17 14:08:03 +01001085
willy tarreau197e8ec2005-12-17 14:10:59 +01001086 - the last character reports what operations were performed on the persistence
willy tarreauc1cae632005-12-17 14:12:23 +01001087 cookie returned by the server (only in HTTP mode) :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001088
willy tarreau197e8ec2005-12-17 14:10:59 +01001089 N : NO cookie was provided by the server.
1090 P : a cookie was PROVIDED by the server and transmitted as-is.
1091 I : no cookie was provided by the server, and one was INSERTED by the
willy tarreaueedaa9f2005-12-17 14:08:03 +01001092 proxy.
willy tarreau197e8ec2005-12-17 14:10:59 +01001093 D : the cookie provided by the server was DELETED by the proxy.
1094 R : the cookie provided by the server was REWRITTEN by the proxy.
1095 - : does not apply (no cookie set in configuration).
willy tarreaueedaa9f2005-12-17 14:08:03 +01001096
willy tarreau197e8ec2005-12-17 14:10:59 +01001097The 'capture' keyword allows to capture and log informations exchanged between
1098clients and servers. As of version 1.1.23, only cookies can be captured, which
1099makes it easy to track a complete user session. The syntax is :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001100
willy tarreau197e8ec2005-12-17 14:10:59 +01001101 capture cookie <cookie_prefix> len <capture_length>
willy tarreaueedaa9f2005-12-17 14:08:03 +01001102
willy tarreau197e8ec2005-12-17 14:10:59 +01001103The FIRST cookie whose name starts with <cookie_prefix> will be captured, and
1104logged as 'NAME=value', without exceeding <capture_length> characters (64 max).
1105When the cookie name is fixed and known, it's preferable to suffix '=' to it to
1106ensure that no other cookie will be logged.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001107
willy tarreau197e8ec2005-12-17 14:10:59 +01001108Examples :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001109----------
willy tarreau197e8ec2005-12-17 14:10:59 +01001110 # capture the first cookie whose name starts with "ASPSESSION"
willy tarreaueedaa9f2005-12-17 14:08:03 +01001111 capture cookie ASPSESSION len 32
1112
willy tarreau197e8ec2005-12-17 14:10:59 +01001113 # capture the first cookie whose name is exactly "vgnvisitor"
willy tarreaueedaa9f2005-12-17 14:08:03 +01001114 capture cookie vgnvisitor= len 32
1115
willy tarreau197e8ec2005-12-17 14:10:59 +01001116In the logs, the field preceeding the completion indicator contains the cookie
1117value as sent by the server, preceeded by the cookie value as sent by the
1118client. Each of these field is replaced with '-' when no cookie was seen.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001119
willy tarreauc1cae632005-12-17 14:12:23 +010011204.2.5) Examples of logs
1121-----------------------
1122- haproxy[674]: 127.0.0.1:33319 [15/Oct/2003:08:31:57] relais-http Srv1 6559/7/147/6723 200 243 - - ---- "HEAD / HTTP/1.0"
1123 => long request (6.5s) entered by hand through 'telnet'. The server replied
1124 in 147 ms, and the session ended normally ('----')
willy tarreaueedaa9f2005-12-17 14:08:03 +01001125
willy tarreau982249e2005-12-18 00:57:06 +01001126- haproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17] relais-http Srv1 9/7/14/+30 200 +243 - - ---- "GET /image.iso HTTP/1.0"
1127 => request for a long data transfer. The 'logasap' option was specified, so
1128 the log was produced just before transfering data. The server replied in
1129 14 ms, 243 bytes of headers were sent to the client, and total time from
1130 accept to first data byte is 30 ms.
1131
1132- haproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17] relais-http Srv1 9/7/14/30 502 243 - - PH-- "GET /cgi-bin/bug.cgi? HTTP/1.0"
1133 => the proxy blocked a server response either because of an 'rspdeny' or
1134 'rspideny' filter, or because it blocked sensible information which risked
1135 being cached. In this case, the response is replaced with a '502 bad
1136 gateway'.
1137
willy tarreauc1cae632005-12-17 14:12:23 +01001138- haproxy[18113]: 127.0.0.1:34548 [15/Oct/2003:15:18:55] relais-http <NOSRV> -1/-1/-1/8490 -1 0 - - CR-- ""
1139 => the client never completed its request and aborted itself ('C---') after
1140 8.5s, while the proxy was waiting for the request headers ('-R--').
1141 Nothing was sent to the server.
1142
1143- haproxy[18113]: 127.0.0.1:34549 [15/Oct/2003:15:19:06] relais-http <NOSRV> -1/-1/-1/50001 408 0 - - cR-- ""
1144 => The client never completed its request, which was aborted by the time-out
1145 ('c---') after 50s, while the proxy was waiting for the request headers ('-R--').
1146 Nothing was sent to the server, but the proxy could send a 408 return code
1147 to the client.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001148
willy tarreauc1cae632005-12-17 14:12:23 +01001149- haproxy[18989]: 127.0.0.1:34550 [15/Oct/2003:15:24:28] relais-tcp Srv1 0/5007 0 cD
1150 => This is a 'tcplog' entry. Client-side time-out ('c----') occured after 5s.
1151
1152- haproxy[18989]: 10.0.0.1:34552 [15/Oct/2003:15:26:31] relais-http Srv1 3183/-1/-1/11215 503 0 - - SC-- "HEAD / HTTP/1.0"
1153 => The request took 3s to complete (probably a network problem), and the
1154 connection to the server failed ('SC--') after 4 attemps of 2 seconds
1155 (config says 'retries 3'), then a 503 error code was sent to the client.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001156
willy tarreau197e8ec2005-12-17 14:10:59 +010011574.3) HTTP header manipulation
1158-----------------------------
1159In HTTP mode, it is possible to rewrite, add or delete some of the request and
1160response headers based on regular expressions. It is also possible to block a
1161request or a response if a particular header matches a regular expression,
1162which is enough to stops most elementary protocol attacks, and to protect
1163against information leak from the internal network. But there is a limitation
1164to this : since haproxy's HTTP engine knows nothing about keep-alive, only
1165headers passed during the first request of a TCP session will be seen. All
1166subsequent headers will be considered data only and not analyzed. Furthermore,
1167haproxy doesn't touch data contents, it stops at the end of headers.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001168
willy tarreau197e8ec2005-12-17 14:10:59 +01001169The syntax is :
1170 reqadd <string> to add a header to the request
1171 reqrep <search> <replace> to modify the request
1172 reqirep <search> <replace> same, but ignoring the case
1173 reqdel <search> to delete a header in the request
1174 reqidel <search> same, but ignoring the case
1175 reqallow <search> definitely allow a request if a header matches <search>
1176 reqiallow <search> same, but ignoring the case
1177 reqdeny <search> denies a request if a header matches <search>
1178 reqideny <search> same, but ignoring the case
1179 reqpass <search> ignore a header matching <search>
1180 reqipass <search> same, but ignoring the case
willy tarreaueedaa9f2005-12-17 14:08:03 +01001181
willy tarreau197e8ec2005-12-17 14:10:59 +01001182 rspadd <string> to add a header to the response
1183 rsprep <search> <replace> to modify the response
1184 rspirep <search> <replace> same, but ignoring the case
1185 rspdel <search> to delete the response
1186 rspidel <search> same, but ignoring the case
willy tarreau982249e2005-12-18 00:57:06 +01001187 rspdeny <search> replaces a response with a HTTP 502 if a header matches <search>
1188 rspideny <search> same, but ignoring the case
willy tarreaueedaa9f2005-12-17 14:08:03 +01001189
1190
willy tarreau197e8ec2005-12-17 14:10:59 +01001191<search> is a POSIX regular expression (regex) which supports grouping through
1192parenthesis (without the backslash). Spaces and other delimiters must be
1193prefixed with a backslash ('\') to avoid confusion with a field delimiter.
1194Other characters may be prefixed with a backslash to change their meaning :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001195
willy tarreau197e8ec2005-12-17 14:10:59 +01001196 \t for a tab
1197 \r for a carriage return (CR)
1198 \n for a new line (LF)
1199 \ to mark a space and differentiate it from a delimiter
1200 \# to mark a sharp and differentiate it from a comment
1201 \\ to use a backslash in a regex
1202 \\\\ to use a backslash in the text (*2 for regex, *2 for haproxy)
1203 \xXX to write the ASCII hex code XX as in the C language
willy tarreaueedaa9f2005-12-17 14:08:03 +01001204
1205
willy tarreau197e8ec2005-12-17 14:10:59 +01001206<replace> containst the string to be used to replace the largest portion of text
1207matching the regex. It can make use of the special characters above, and can
1208reference a substring delimited by parenthesis in the regex, by the group
1209numerical order from 1 to 9. In this case, you would write a backslah ('\')
1210immediately followed by one digit indicating the group position.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001211
willy tarreau197e8ec2005-12-17 14:10:59 +01001212<string> represents the string which will systematically be added after the last
1213header line. It can also use special characters above.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001214
willy tarreau197e8ec2005-12-17 14:10:59 +01001215Notes :
1216-------
1217 - the first line is considered as a header, which makes it possible to rewrite
1218 or filter HTTP requests URIs or response codes.
1219 - 'reqrep' is the equivalent of 'cliexp' in version 1.0, and 'rsprep' is the
1220 equivalent of 'srvexp' in 1.0. Those names are still supported but
1221 deprecated.
1222 - for performances reasons, the number of characters added to a request or to
1223 a response is limited to 4096 since version 1.1.5 (it was 256 before). This
1224 value is easy to modify in the code if needed (#define). If it is too short
1225 on occasional uses, it is possible to gain some space by removing some
1226 useless headers before adding new ones.
willy tarreau982249e2005-12-18 00:57:06 +01001227 - a denied request will generate an "HTTP 403 forbidden" response, while a
1228 denied response will generate an "HTTP 502 Bad gateway" response.
1229
willy tarreaueedaa9f2005-12-17 14:08:03 +01001230
willy tarreau197e8ec2005-12-17 14:10:59 +01001231Examples :
1232----------
1233 ###### a few examples ######
1234
1235 # rewrite 'online.fr' instead of 'free.fr' for GET and POST requests
1236 reqrep ^(GET\ .*)(.free.fr)(.*) \1.online.fr\3
1237 reqrep ^(POST\ .*)(.free.fr)(.*) \1.online.fr\3
1238
1239 # force proxy connections to close
willy tarreaueedaa9f2005-12-17 14:08:03 +01001240 reqirep ^Proxy-Connection:.* Proxy-Connection:\ close
willy tarreau197e8ec2005-12-17 14:10:59 +01001241 # rewrite locations
willy tarreaueedaa9f2005-12-17 14:08:03 +01001242 rspirep ^(Location:\ )([^:]*://[^/]*)(.*) \1\3
willy tarreaueedaa9f2005-12-17 14:08:03 +01001243
willy tarreau197e8ec2005-12-17 14:10:59 +01001244 ###### A full configuration being used on production ######
willy tarreaueedaa9f2005-12-17 14:08:03 +01001245
willy tarreau197e8ec2005-12-17 14:10:59 +01001246 # Every header should end with a colon followed by one space.
1247 reqideny ^[^:\ ]*[\ ]*$
willy tarreaueedaa9f2005-12-17 14:08:03 +01001248
willy tarreau197e8ec2005-12-17 14:10:59 +01001249 # block Apache chunk exploit
1250 reqideny ^Transfer-Encoding:[\ ]*chunked
1251 reqideny ^Host:\ apache-
willy tarreaueedaa9f2005-12-17 14:08:03 +01001252
willy tarreau197e8ec2005-12-17 14:10:59 +01001253 # block annoying worms that fill the logs...
1254 reqideny ^[^:\ ]*\ .*(\.|%2e)(\.|%2e)(%2f|%5c|/|\\\\)
1255 reqideny ^[^:\ ]*\ ([^\ ]*\ [^\ ]*\ |.*%00)
1256 reqideny ^[^:\ ]*\ .*<script
1257 reqideny ^[^:\ ]*\ .*/(root\.exe\?|cmd\.exe\?|default\.ida\?)
1258
1259 # allow other syntactically valid requests, and block any other method
1260 reqipass ^(GET|POST|HEAD|OPTIONS)\ /.*\ HTTP/1\.[01]$
1261 reqipass ^OPTIONS\ \\*\ HTTP/1\.[01]$
1262 reqideny ^[^:\ ]*\
1263
1264 # force connection:close, thus disabling HTTP keep-alive
willy tarreau982249e2005-12-18 00:57:06 +01001265 option httpclose
willy tarreau197e8ec2005-12-17 14:10:59 +01001266
1267 # change the server name
1268 rspidel ^Server:\
1269 rspadd Server:\ Formilux/0.1.8
1270
1271
willy tarreau982249e2005-12-18 00:57:06 +01001272Also, the 'forwardfor' option creates an HTTP 'X-Forwarded-For' header which
willy tarreauc1cae632005-12-17 14:12:23 +01001273contains the client's IP address. This is useful to let the final web server
1274know what the client address was (eg for statistics on domains).
1275
willy tarreau982249e2005-12-18 00:57:06 +01001276Last, the 'httpclose' option removes any 'Connection' header both ways, and
1277adds a 'Connection: close' header in each direction. This makes it easier to
1278disable HTTP keep-alive than the previous 4-rules block..
1279
willy tarreauc1cae632005-12-17 14:12:23 +01001280Example :
1281---------
1282 listen http_proxy 0.0.0.0:80
1283 mode http
1284 log global
1285 option httplog
1286 option dontlognull
1287 option forwardfor
willy tarreau982249e2005-12-18 00:57:06 +01001288 option httpclose
willy tarreau197e8ec2005-12-17 14:10:59 +01001289
12904.4) Load balancing with persistence
1291------------------------------------
willy tarreau197e8ec2005-12-17 14:10:59 +01001292Combining cookie insertion with internal load balancing allows to transparently
1293bring persistence to applications. The principle is quite simple :
1294 - assign a cookie value to each server
1295 - enable the load balancing between servers
1296 - insert a cookie into responses resulting from the balancing algorithm
1297 (indirect accesses), end ensure that no upstream proxy will cache it.
1298 - remove the cookie in the request headers so that the application never sees
1299 it.
1300
1301Example :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001302---------
1303 listen application 0.0.0.0:80
1304 mode http
1305 cookie SERVERID insert nocache indirect
1306 balance roundrobin
1307 server 192.168.1.1:80 cookie server01 check
1308 server 192.168.1.2:80 cookie server02 check
1309
willy tarreau982249e2005-12-18 00:57:06 +010013104.5) Protection against information leak from the servers
1311---------------------------------------------------------
1312In versions 1.1.28/1.2.1, a new option 'checkcache' was created. It carefully
1313checks 'Cache-control', 'Pragma' and 'Set-cookie' headers in server response
1314to check if there's a risk of caching a cookie on a client-side proxy. When this
1315option is enabled, the only responses which can be delivered to the client are :
1316 - all those without 'Set-Cookie' header ;
1317 - all those with a return code other than 200, 203, 206, 300, 301, 410,
1318 provided that the server has not set a 'Cache-control: public' header ;
1319 - all those that come from a POST request, provided that the server has not
1320 set a 'Cache-Control: public' header ;
1321 - those with a 'Pragma: no-cache' header
1322 - those with a 'Cache-control: private' header
1323 - those with a 'Cache-control: no-store' header
1324 - those with a 'Cache-control: max-age=0' header
1325 - those with a 'Cache-control: s-maxage=0' header
1326 - those with a 'Cache-control: no-cache' header
1327 - those with a 'Cache-control: no-cache="set-cookie"' header
1328 - those with a 'Cache-control: no-cache="set-cookie,' header
1329 (allowing other fields after set-cookie)
willy tarreaueedaa9f2005-12-17 14:08:03 +01001330
willy tarreau982249e2005-12-18 00:57:06 +01001331If a response doesn't respect these requirements, then it will be blocked just
1332as if it was from an 'rspdeny' filter, with an "HTTP 502 bad gateway". The
1333session state shows "PH--" meaning that the proxy blocked the response during
1334headers processing. Additionnaly, an alert will be sent in the logs so that
1335admins are told that there's something to be done.
1336
13374.6) Customizing errors
1338-----------------------
willy tarreau197e8ec2005-12-17 14:10:59 +01001339Some situations can make haproxy return an HTTP error code to the client :
1340 - invalid or too long request => HTTP 400
1341 - request not completely sent in time => HTTP 408
1342 - forbidden request (matches a deny filter) => HTTP 403
1343 - internal error in haproxy => HTTP 500
1344 - the server returned an invalid or incomplete response => HTTP 502
1345 - no server was available to handle the request => HTTP 503
1346 - the server failed to reply in time => HTTP 504
willy tarreaueedaa9f2005-12-17 14:08:03 +01001347
willy tarreau197e8ec2005-12-17 14:10:59 +01001348A succint error message taken from the RFC accompanies these return codes.
1349But depending on the clients knowledge, it may be better to return custom, user
1350friendly, error pages. This is made possible through the use of the 'errorloc'
1351command :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001352
willy tarreau197e8ec2005-12-17 14:10:59 +01001353 errorloc <HTTP_code> <location>
willy tarreaueedaa9f2005-12-17 14:08:03 +01001354
willy tarreau197e8ec2005-12-17 14:10:59 +01001355Instead of generating an HTTP error <HTTP_code> among those above, the proxy
1356will return a temporary redirection code (HTTP 302) towards the address
1357specified in <location>. This address may be either relative to the site or
1358absolute. Since this request will be handled by the client's browser, it's
1359mandatory that the returned address be reachable from the outside.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001360
willy tarreau197e8ec2005-12-17 14:10:59 +01001361Example :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001362---------
1363 listen application 0.0.0.0:80
1364 errorloc 400 /badrequest.html
1365 errorloc 403 /forbidden.html
1366 errorloc 408 /toolong.html
1367 errorloc 500 http://haproxy.domain.net/bugreport.html
1368 errorloc 502 http://192.168.114.58/error50x.html
1369 errorloc 503 http://192.168.114.58/error50x.html
1370 errorloc 504 http://192.168.114.58/error50x.html
1371
willy tarreau982249e2005-12-18 00:57:06 +010013724.7) Modifying default values
willy tarreau197e8ec2005-12-17 14:10:59 +01001373-----------------------------
willy tarreau197e8ec2005-12-17 14:10:59 +01001374Version 1.1.22 introduced the notion of default values, which eliminates the
1375pain of often repeating common parameters between many instances, such as
1376logs, timeouts, modes, etc...
willy tarreaueedaa9f2005-12-17 14:08:03 +01001377
willy tarreau197e8ec2005-12-17 14:10:59 +01001378Default values are set in a 'defaults' section. Each of these section clears
1379all previously set default parameters, so there may be as many default
1380parameters as needed. Only the last one before a 'listen' section will be
1381used for this section. The 'defaults' section uses the same syntax as the
1382'listen' section, for the supported parameters. The 'defaults' keyword ignores
1383everything on its command line, so that fake instance names can be specified
1384there for better clarity.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001385
willy tarreau982249e2005-12-18 00:57:06 +01001386In version 1.1.28/1.2.1, only those parameters can be preset in the 'default'
willy tarreau197e8ec2005-12-17 14:10:59 +01001387section :
1388 - log (the first and second one)
willy tarreaueedaa9f2005-12-17 14:08:03 +01001389 - mode { tcp, http, health }
1390 - balance { roundrobin }
willy tarreau197e8ec2005-12-17 14:10:59 +01001391 - disabled (to disable every further instances)
1392 - enabled (to enable every further instances, this is the default)
willy tarreaueedaa9f2005-12-17 14:08:03 +01001393 - contimeout, clitimeout, srvtimeout, grace, retries, maxconn
willy tarreau982249e2005-12-18 00:57:06 +01001394 - option { redispatch, transparent, keepalive, forwardfor, logasap, httpclose,
1395 checkcache, httplog, tcplog, dontlognull, persist, httpchk }
willy tarreaueedaa9f2005-12-17 14:08:03 +01001396 - redispatch, redisp, transparent, source { addr:port }
1397 - cookie, capture
1398 - errorloc
1399
willy tarreau197e8ec2005-12-17 14:10:59 +01001400As of 1.1.24, it is not possible to put certain parameters in a 'defaults'
1401section, mainly regular expressions and server configurations :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001402 - dispatch, server,
willy tarreau197e8ec2005-12-17 14:10:59 +01001403 - req*, rsp*
willy tarreaueedaa9f2005-12-17 14:08:03 +01001404
willy tarreau197e8ec2005-12-17 14:10:59 +01001405Last, there's no way yet to change a boolean option from its assigned default
1406value. So if an 'option' statement is set in a 'defaults' section, the only
1407way to flush it is to redefine a new 'defaults' section without this 'option'.
willy tarreaueedaa9f2005-12-17 14:08:03 +01001408
willy tarreau197e8ec2005-12-17 14:10:59 +01001409Examples :
willy tarreaueedaa9f2005-12-17 14:08:03 +01001410----------
1411 defaults applications TCP
1412 log global
1413 mode tcp
1414 balance roundrobin
1415 clitimeout 180000
1416 srvtimeout 180000
1417 contimeout 4000
1418 retries 3
1419 redispatch
1420
1421 listen app_tcp1 10.0.0.1:6000-6063
1422 server srv1 192.168.1.1 check port 6000 inter 10000
1423 server srv2 192.168.1.2 backup
1424
1425 listen app_tcp2 10.0.0.2:6000-6063
1426 server srv1 192.168.2.1 check port 6000 inter 10000
1427 server srv2 192.168.2.2 backup
1428
1429 defaults applications HTTP
1430 log global
1431 mode http
1432 option httplog
1433 option forwardfor
1434 option dontlognull
1435 balance roundrobin
1436 clitimeout 20000
1437 srvtimeout 20000
1438 contimeout 4000
1439 retries 3
1440
1441 listen app_http1 10.0.0.1:80-81
1442 cookie SERVERID postonly insert indirect
1443 capture cookie userid= len 10
1444 server srv1 192.168.1.1:+8000 cookie srv1 check port 8080 inter 1000
1445 server srv1 192.168.1.2:+8000 cookie srv2 check port 8080 inter 1000
1446
1447 defaults
willy tarreau197e8ec2005-12-17 14:10:59 +01001448 # this empty section voids all default parameters
willy tarreaueedaa9f2005-12-17 14:08:03 +01001449
willy tarreau197e8ec2005-12-17 14:10:59 +01001450=========================
1451| System-specific setup |
1452=========================
willy tarreaueedaa9f2005-12-17 14:08:03 +01001453
willy tarreau197e8ec2005-12-17 14:10:59 +01001454Linux 2.4
1455=========
willy tarreaueedaa9f2005-12-17 14:08:03 +01001456
1457-- cut here --
1458#!/bin/sh
1459# set this to about 256/4M (16384 for 256M machine)
1460MAXFILES=16384
1461echo $MAXFILES > /proc/sys/fs/file-max
1462ulimit -n $MAXFILES
1463
1464if [ -e /proc/sys/net/ipv4/ip_conntrack_max ]; then
1465 echo 65536 > /proc/sys/net/ipv4/ip_conntrack_max
1466fi
1467
1468if [ -e /proc/sys/net/ipv4/netfilter/ip_ct_tcp_timeout_fin_wait ]; then
1469 # 30 seconds for fin, 15 for time wait
1470 echo 3000 > /proc/sys/net/ipv4/netfilter/ip_ct_tcp_timeout_fin_wait
1471 echo 1500 > /proc/sys/net/ipv4/netfilter/ip_ct_tcp_timeout_time_wait
1472 echo 0 > /proc/sys/net/ipv4/netfilter/ip_ct_tcp_log_invalid_scale
1473 echo 0 > /proc/sys/net/ipv4/netfilter/ip_ct_tcp_log_out_of_window
1474fi
1475
1476echo 1024 60999 > /proc/sys/net/ipv4/ip_local_port_range
1477echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
1478echo 4096 > /proc/sys/net/ipv4/tcp_max_syn_backlog
1479echo 262144 > /proc/sys/net/ipv4/tcp_max_tw_buckets
1480echo 262144 > /proc/sys/net/ipv4/tcp_max_orphans
1481echo 300 > /proc/sys/net/ipv4/tcp_keepalive_time
1482echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
1483echo 0 > /proc/sys/net/ipv4/tcp_timestamps
1484echo 0 > /proc/sys/net/ipv4/tcp_ecn
1485echo 0 > /proc/sys/net/ipv4/tcp_sack
1486echo 0 > /proc/sys/net/ipv4/tcp_dsack
1487
1488# auto-tuned on 2.4
1489#echo 262143 > /proc/sys/net/core/rmem_max
1490#echo 262143 > /proc/sys/net/core/rmem_default
1491
1492echo 16384 65536 524288 > /proc/sys/net/ipv4/tcp_rmem
1493echo 16384 349520 699040 > /proc/sys/net/ipv4/tcp_wmem
1494
1495-- cut here --
1496
willy tarreau197e8ec2005-12-17 14:10:59 +01001497
1498FreeBSD
1499=======
1500
1501A FreeBSD port of HA-Proxy is now available and maintained, thanks to
1502Clement Laforet <sheepkiller@cultdeadsheep.org>.
1503
1504For more information :
1505http://www.freebsd.org/cgi/url.cgi?ports/net/haproxy/pkg-descr
1506http://www.freebsd.org/cgi/cvsweb.cgi/ports/net/haproxy/
1507http://www.freshports.org/net/haproxy
1508
1509
1510-- end --