blob: 8b04f99548d71cf23c32533c86cfa37b57219b29 [file] [log] [blame]
willy tarreau0174f312005-12-18 01:02:42 +01001 -------------------
Willy Tarreau94b45912006-05-31 06:40:15 +02002 HAProxy
willy tarreau0174f312005-12-18 01:02:42 +01003 Architecture Guide
4 -------------------
willy tarreau065f1c02006-01-29 22:10:07 +01005 version 1.1.34
willy tarreau0174f312005-12-18 01:02:42 +01006 willy tarreau
willy tarreau065f1c02006-01-29 22:10:07 +01007 2006/01/29
willy tarreau0174f312005-12-18 01:02:42 +01008
9
10This document provides real world examples with working configurations.
11Please note that except stated otherwise, global configuration parameters
12such as logging, chrooting, limits and time-outs are not described here.
13
14===================================================
151. Simple HTTP load-balancing with cookie insertion
16===================================================
17
18A web application often saturates the front-end server with high CPU loads,
19due to the scripting language involved. It also relies on a back-end database
20which is not much loaded. User contexts are stored on the server itself, and
21not in the database, so that simply adding another server with simple IP/TCP
22load-balancing would not work.
23
24 +-------+
25 |clients| clients and/or reverse-proxy
26 +---+---+
27 |
28 -+-----+--------+----
29 | _|_db
30 +--+--+ (___)
31 | web | (___)
32 +-----+ (___)
33 192.168.1.1 192.168.1.2
34
35
36Replacing the web server with a bigger SMP system would cost much more than
37adding low-cost pizza boxes. The solution is to buy N cheap boxes and install
38the application on them. Install haproxy on the old one which will spread the
39load across the new boxes.
40
41 192.168.1.1 192.168.1.11-192.168.1.14 192.168.1.2
42 -------+-----------+-----+-----+-----+--------+----
43 | | | | | _|_db
44 +--+--+ +-+-+ +-+-+ +-+-+ +-+-+ (___)
45 | LB1 | | A | | B | | C | | D | (___)
46 +-----+ +---+ +---+ +---+ +---+ (___)
47 haproxy 4 cheap web servers
48
49
50Config on haproxy (LB1) :
51-------------------------
52
willy tarreauc5f73ed2005-12-18 01:26:38 +010053 listen webfarm 192.168.1.1:80
willy tarreau0174f312005-12-18 01:02:42 +010054 mode http
55 balance roundrobin
56 cookie SERVERID insert indirect
57 option httpchk HEAD /index.html HTTP/1.0
58 server webA 192.168.1.11:80 cookie A check
59 server webB 192.168.1.12:80 cookie B check
60 server webC 192.168.1.13:80 cookie C check
61 server webD 192.168.1.14:80 cookie D check
62
63
64Description :
65-------------
66 - LB1 will receive clients requests.
67 - if a request does not contain a cookie, it will be forwarded to a valid
68 server
69 - in return, a cookie "SERVERID" will be inserted in the response holding the
70 server name (eg: "A").
71 - when the client comes again with the cookie "SERVERID=A", LB1 will know that
72 it must be forwarded to server A. The cookie will be removed so that the
73 server does not see it.
74 - if server "webA" dies, the requests will be sent to another valid server
75 and a cookie will be reassigned.
76
77
78Flows :
79-------
80
81(client) (haproxy) (server A)
82 >-- GET /URI1 HTTP/1.0 ------------> |
83 ( no cookie, haproxy forwards in load-balancing mode. )
84 | >-- GET /URI1 HTTP/1.0 ---------->
85 | <-- HTTP/1.0 200 OK -------------<
86 ( the proxy now adds the server cookie in return )
87 <-- HTTP/1.0 200 OK ---------------< |
88 Set-Cookie: SERVERID=A |
89 >-- GET /URI2 HTTP/1.0 ------------> |
90 Cookie: SERVERID=A |
91 ( the proxy sees the cookie. it forwards to server A and deletes it )
92 | >-- GET /URI2 HTTP/1.0 ---------->
93 | <-- HTTP/1.0 200 OK -------------<
94 ( the proxy does not add the cookie in return because the client knows it )
95 <-- HTTP/1.0 200 OK ---------------< |
96 >-- GET /URI3 HTTP/1.0 ------------> |
97 Cookie: SERVERID=A |
98 ( ... )
99
100
101Limits :
102--------
103 - if clients use keep-alive (HTTP/1.1), only the first response will have
104 a cookie inserted, and only the first request of each session will be
105 analyzed. This does not cause trouble in insertion mode because the cookie
106 is put immediately in the first response, and the session is maintained to
107 the same server for all subsequent requests in the same session. However,
108 the cookie will not be removed from the requests forwarded to the servers,
109 so the server must not be sensitive to unknown cookies. If this causes
110 trouble, you can disable keep-alive by adding the following option :
111
112 option httpclose
113
114 - if for some reason the clients cannot learn more than one cookie (eg: the
115 clients are indeed some home-made applications or gateways), and the
116 application already produces a cookie, you can use the "prefix" mode (see
117 below).
118
119 - LB1 becomes a very sensible server. If LB1 dies, nothing works anymore.
120 => you can back it up using keepalived.
121
122 - if the application needs to log the original client's IP, use the
123 "forwardfor" option which will add an "X-Forwarded-For" header with the
124 original client's IP address. You must also use "httpclose" to ensure
125 that you will rewrite every requests and not only the first one of each
126 session :
127
128 option httpclose
129 option forwardfor
130
131 The web server will have to be configured to use this header instead.
132 For example, on apache, you can use LogFormat for this :
133
134 LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b " combined
135 CustomLog /var/log/httpd/access_log combined
136
137
138==================================================================
1392. HTTP load-balancing with cookie prefixing and high availability
140==================================================================
141
142Now you don't want to add more cookies, but rather use existing ones. The
143application already generates a "JSESSIONID" cookie which is enough to track
144sessions, so we'll prefix this cookie with the server name when we see it.
145Since the load-balancer becomes critical, it will be backed up with a second
willy tarreauc5f73ed2005-12-18 01:26:38 +0100146one in VRRP mode using keepalived under Linux.
willy tarreau0174f312005-12-18 01:02:42 +0100147
148Download the latest version of keepalived from this site and install it
149on each load-balancer LB1 and LB2 :
150
151 http://www.keepalived.org/
152
153You then have a shared IP between the two load-balancers (we will still use the
154original IP). It is active only on one of them at any moment. To allow the
willy tarreauc5f73ed2005-12-18 01:26:38 +0100155proxy to bind to the shared IP on Linux 2.4, you must enable it in /proc :
willy tarreau0174f312005-12-18 01:02:42 +0100156
157# echo 1 >/proc/sys/net/ipv4/ip_nonlocal_bind
158
159
160 shared IP=192.168.1.1
161 192.168.1.3 192.168.1.4 192.168.1.11-192.168.1.14 192.168.1.2
162 -------+------------+-----------+-----+-----+-----+--------+----
163 | | | | | | _|_db
164 +--+--+ +--+--+ +-+-+ +-+-+ +-+-+ +-+-+ (___)
165 | LB1 | | LB2 | | A | | B | | C | | D | (___)
166 +-----+ +-----+ +---+ +---+ +---+ +---+ (___)
167 haproxy haproxy 4 cheap web servers
168 keepalived keepalived
169
170
171Config on both proxies (LB1 and LB2) :
172--------------------------------------
173
willy tarreauc5f73ed2005-12-18 01:26:38 +0100174 listen webfarm 192.168.1.1:80
willy tarreau0174f312005-12-18 01:02:42 +0100175 mode http
176 balance roundrobin
177 cookie JSESSIONID prefix
178 option httpclose
179 option forwardfor
180 option httpchk HEAD /index.html HTTP/1.0
181 server webA 192.168.1.11:80 cookie A check
182 server webB 192.168.1.12:80 cookie B check
183 server webC 192.168.1.13:80 cookie C check
184 server webD 192.168.1.14:80 cookie D check
185
186
187Notes: the proxy will modify EVERY cookie sent by the client and the server,
188so it is important that it can access to ALL cookies in ALL requests for
189each session. This implies that there is no keep-alive (HTTP/1.1), thus the
190"httpclose" option. Only if you know for sure that the client(s) will never
willy tarreauc5f73ed2005-12-18 01:26:38 +0100191use keep-alive (eg: Apache 1.3 in reverse-proxy mode), you can remove this
192option.
willy tarreau0174f312005-12-18 01:02:42 +0100193
194
195Description :
196-------------
197 - LB1 is VRRP master (keepalived), LB2 is backup.
198 - LB1 will receive clients requests on IP 192.168.1.1.
199 - both load-balancers send their checks from their native IP.
200 - if a request does not contain a cookie, it will be forwarded to a valid
201 server
202 - in return, if a JESSIONID cookie is seen, the server name will be prefixed
203 into it, followed by a delimitor ('~')
204 - when the client comes again with the cookie "JSESSIONID=A~xxx", LB1 will
205 know that it must be forwarded to server A. The server name will then be
206 extracted from cookie before it is sent to the server.
207 - if server "webA" dies, the requests will be sent to another valid server
208 and a cookie will be reassigned.
209
210
211Flows :
212-------
213
214(client) (haproxy) (server A)
215 >-- GET /URI1 HTTP/1.0 ------------> |
216 ( no cookie, haproxy forwards in load-balancing mode. )
217 | >-- GET /URI1 HTTP/1.0 ---------->
218 | X-Forwarded-For: 10.1.2.3
219 | <-- HTTP/1.0 200 OK -------------<
220 ( no cookie, nothing changed )
221 <-- HTTP/1.0 200 OK ---------------< |
222 >-- GET /URI2 HTTP/1.0 ------------> |
223 ( no cookie, haproxy forwards in lb mode, possibly to another server. )
224 | >-- GET /URI2 HTTP/1.0 ---------->
225 | X-Forwarded-For: 10.1.2.3
226 | <-- HTTP/1.0 200 OK -------------<
227 | Set-Cookie: JSESSIONID=123
228 ( the cookie is identified, it will be prefixed with the server name )
229 <-- HTTP/1.0 200 OK ---------------< |
230 Set-Cookie: JSESSIONID=A~123 |
231 >-- GET /URI3 HTTP/1.0 ------------> |
232 Cookie: JSESSIONID=A~123 |
233 ( the proxy sees the cookie, removes the server name and forwards
234 to server A which sees the same cookie as it previously sent )
235 | >-- GET /URI3 HTTP/1.0 ---------->
236 | Cookie: JSESSIONID=123
237 | X-Forwarded-For: 10.1.2.3
238 | <-- HTTP/1.0 200 OK -------------<
239 ( no cookie, nothing changed )
240 <-- HTTP/1.0 200 OK ---------------< |
241 ( ... )
242
243
244
245========================================================
2462.1 Variations involving external layer 4 load-balancers
247========================================================
248
249Instead of using a VRRP-based active/backup solution for the proxies,
250they can also be load-balanced by a layer4 load-balancer (eg: Alteon)
251which will also check that the services run fine on both proxies :
252
253 | VIP=192.168.1.1
254 +----+----+
255 | Alteon |
256 +----+----+
257 |
258 192.168.1.3 | 192.168.1.4 192.168.1.11-192.168.1.14 192.168.1.2
259 -------+-----+------+-----------+-----+-----+-----+--------+----
260 | | | | | | _|_db
261 +--+--+ +--+--+ +-+-+ +-+-+ +-+-+ +-+-+ (___)
262 | LB1 | | LB2 | | A | | B | | C | | D | (___)
263 +-----+ +-----+ +---+ +---+ +---+ +---+ (___)
264 haproxy haproxy 4 cheap web servers
265
266
267Config on both proxies (LB1 and LB2) :
268--------------------------------------
269
willy tarreauc5f73ed2005-12-18 01:26:38 +0100270 listen webfarm 0.0.0.0:80
willy tarreau0174f312005-12-18 01:02:42 +0100271 mode http
272 balance roundrobin
273 cookie JSESSIONID prefix
274 option httpclose
275 option forwardfor
276 option httplog
277 option dontlognull
278 option httpchk HEAD /index.html HTTP/1.0
279 server webA 192.168.1.11:80 cookie A check
280 server webB 192.168.1.12:80 cookie B check
281 server webC 192.168.1.13:80 cookie C check
282 server webD 192.168.1.14:80 cookie D check
283
284The "dontlognull" option is used to prevent the proxy from logging the health
285checks from the Alteon. If a session exchanges no data, then it will not be
286logged.
287
willy tarreauc5f73ed2005-12-18 01:26:38 +0100288Config on the Alteon :
289----------------------
290
291 /c/slb/real 11
292 ena
293 name "LB1"
294 rip 192.168.1.3
295 /c/slb/real 12
296 ena
297 name "LB2"
298 rip 192.168.1.4
299 /c/slb/group 10
300 name "LB1-2"
301 metric roundrobin
302 health tcp
303 add 11
304 add 12
305 /c/slb/virt 10
306 ena
307 vip 192.168.1.1
308 /c/slb/virt 10/service http
309 group 10
310
311
312Note: the health-check on the Alteon is set to "tcp" to prevent the proxy from
313forwarding the connections. It can also be set to "http", but for this the
314proxy must specify a "monitor-net" with the Alteons' addresses, so that the
315Alteon can really check that the proxies can talk HTTP but without forwarding
316the connections to the end servers. Check next section for an example on how to
317use monitor-net.
318
319
320============================================================
3212.2 Generic TCP relaying and external layer 4 load-balancers
322============================================================
323
324Sometimes it's useful to be able to relay generic TCP protocols (SMTP, TSE,
325VNC, etc...), for example to interconnect private networks. The problem comes
326when you use external load-balancers which need to send periodic health-checks
327to the proxies, because these health-checks get forwarded to the end servers.
328The solution is to specify a network which will be dedicated to monitoring
329systems and must not lead to a forwarding connection nor to any log, using the
330"monitor-net" keyword. Note: this feature expects a version of haproxy greater
331than or equal to 1.1.32 or 1.2.6.
332
333
334 | VIP=172.16.1.1 |
335 +----+----+ +----+----+
336 | Alteon1 | | Alteon2 |
337 +----+----+ +----+----+
338 192.168.1.252 | GW=192.168.1.254 | 192.168.1.253
339 | |
340 ------+---+------------+--+-----------------> TSE farm : 192.168.1.10
341 192.168.1.1 | | 192.168.1.2
342 +--+--+ +--+--+
343 | LB1 | | LB2 |
344 +-----+ +-----+
345 haproxy haproxy
346
347
348Config on both proxies (LB1 and LB2) :
349--------------------------------------
350
351 listen tse-proxy
352 bind :3389,:1494,:5900 # TSE, ICA and VNC at once.
353 mode tcp
354 balance roundrobin
355 server tse-farm 192.168.1.10
356 monitor-net 192.168.1.252/31
357
358The "monitor-net" option instructs the proxies that any connection coming from
359192.168.1.252 or 192.168.1.253 will not be logged nor forwarded and will be
360closed immediately. The Alteon load-balancers will then see the proxies alive
361without perturbating the service.
362
willy tarreau0174f312005-12-18 01:02:42 +0100363Config on the Alteon :
364----------------------
365
willy tarreauc5f73ed2005-12-18 01:26:38 +0100366 /c/l3/if 1
367 ena
368 addr 192.168.1.252
369 mask 255.255.255.0
370 /c/slb/real 11
371 ena
372 name "LB1"
373 rip 192.168.1.1
374 /c/slb/real 12
375 ena
376 name "LB2"
377 rip 192.168.1.2
378 /c/slb/group 10
379 name "LB1-2"
380 metric roundrobin
381 health tcp
382 add 11
383 add 12
384 /c/slb/virt 10
385 ena
386 vip 172.16.1.1
387 /c/slb/virt 10/service 1494
388 group 10
389 /c/slb/virt 10/service 3389
390 group 10
391 /c/slb/virt 10/service 5900
392 group 10
willy tarreau0174f312005-12-18 01:02:42 +0100393
394
395=========================================================
3963. Simple HTTP/HTTPS load-balancing with cookie insertion
397=========================================================
398
399This is the same context as in example 1 above, but the web
400server uses HTTPS.
401
402 +-------+
403 |clients| clients
404 +---+---+
405 |
406 -+-----+--------+----
407 | _|_db
408 +--+--+ (___)
409 | SSL | (___)
410 | web | (___)
411 +-----+
412 192.168.1.1 192.168.1.2
413
414
415Since haproxy does not handle SSL, this part will have to be extracted from the
416servers (freeing even more ressources) and installed on the load-balancer
417itself. Install haproxy and apache+mod_ssl on the old box which will spread the
418load between the new boxes. Apache will work in SSL reverse-proxy-cache. If the
419application is correctly developped, it might even lower its load. However,
420since there now is a cache between the clients and haproxy, some security
421measures must be taken to ensure that inserted cookies will not be cached.
422
423
424 192.168.1.1 192.168.1.11-192.168.1.14 192.168.1.2
425 -------+-----------+-----+-----+-----+--------+----
426 | | | | | _|_db
427 +--+--+ +-+-+ +-+-+ +-+-+ +-+-+ (___)
428 | LB1 | | A | | B | | C | | D | (___)
429 +-----+ +---+ +---+ +---+ +---+ (___)
430 apache 4 cheap web servers
431 mod_ssl
432 haproxy
433
434
435Config on haproxy (LB1) :
436-------------------------
437
438 listen 127.0.0.1:8000
439 mode http
440 balance roundrobin
441 cookie SERVERID insert indirect nocache
442 option httpchk HEAD /index.html HTTP/1.0
443 server webA 192.168.1.11:80 cookie A check
444 server webB 192.168.1.12:80 cookie B check
445 server webC 192.168.1.13:80 cookie C check
446 server webD 192.168.1.14:80 cookie D check
447
448
449Description :
450-------------
451 - apache on LB1 will receive clients requests on port 443
452 - it forwards it to haproxy bound to 127.0.0.1:8000
453 - if a request does not contain a cookie, it will be forwarded to a valid
454 server
455 - in return, a cookie "SERVERID" will be inserted in the response holding the
456 server name (eg: "A"), and a "Cache-control: private" header will be added
457 so that the apache does not cache any page containing such cookie.
458 - when the client comes again with the cookie "SERVERID=A", LB1 will know that
459 it must be forwarded to server A. The cookie will be removed so that the
460 server does not see it.
461 - if server "webA" dies, the requests will be sent to another valid server
462 and a cookie will be reassigned.
463
464Notes :
465-------
466 - if the cookie works in "prefix" mode, there is no need to add the "nocache"
467 option because it is an application cookie which will be modified, and the
468 application flags will be preserved.
469 - if apache 1.3 is used as a front-end before haproxy, it always disables
470 HTTP keep-alive on the back-end, so there is no need for the "httpclose"
471 option on haproxy.
472 - configure apache to set the X-Forwarded-For header itself, and do not do
473 it on haproxy if you need the application to know about the client's IP.
474
475
476Flows :
477-------
478
479(apache) (haproxy) (server A)
480 >-- GET /URI1 HTTP/1.0 ------------> |
481 ( no cookie, haproxy forwards in load-balancing mode. )
482 | >-- GET /URI1 HTTP/1.0 ---------->
483 | <-- HTTP/1.0 200 OK -------------<
484 ( the proxy now adds the server cookie in return )
485 <-- HTTP/1.0 200 OK ---------------< |
486 Set-Cookie: SERVERID=A |
487 Cache-Control: private |
488 >-- GET /URI2 HTTP/1.0 ------------> |
489 Cookie: SERVERID=A |
490 ( the proxy sees the cookie. it forwards to server A and deletes it )
491 | >-- GET /URI2 HTTP/1.0 ---------->
492 | <-- HTTP/1.0 200 OK -------------<
493 ( the proxy does not add the cookie in return because the client knows it )
494 <-- HTTP/1.0 200 OK ---------------< |
495 >-- GET /URI3 HTTP/1.0 ------------> |
496 Cookie: SERVERID=A |
497 ( ... )
498
499
500
501========================================
5024. Soft-stop for application maintenance
503========================================
504
willy tarreau22739ef2006-01-20 20:43:32 +0100505When an application is spread across several servers, the time to update all
willy tarreau0174f312005-12-18 01:02:42 +0100506instances increases, so the application seems jerky for a longer period.
507
508HAproxy offers several solutions for this. Although it cannot be reconfigured
willy tarreauc5f73ed2005-12-18 01:26:38 +0100509without being stopped, nor does it offer any external command, there are other
willy tarreau0174f312005-12-18 01:02:42 +0100510working solutions.
511
512
513=========================================
5144.1 Soft-stop using a file on the servers
515=========================================
516
517This trick is quite common and very simple: put a file on the server which will
518be checked by the proxy. When you want to stop the server, first remove this
519file. The proxy will see the server as failed, and will not send it any new
520session, only the old ones if the "persist" option is used. Wait a bit then
521stop the server when it does not receive anymore connections.
522
523
524 listen 192.168.1.1:80
525 mode http
526 balance roundrobin
527 cookie SERVERID insert indirect
528 option httpchk HEAD /running HTTP/1.0
529 server webA 192.168.1.11:80 cookie A check inter 2000 rise 2 fall 2
530 server webB 192.168.1.12:80 cookie B check inter 2000 rise 2 fall 2
531 server webC 192.168.1.13:80 cookie C check inter 2000 rise 2 fall 2
532 server webD 192.168.1.14:80 cookie D check inter 2000 rise 2 fall 2
533 option persist
534 redispatch
535 contimeout 5000
536
537
538Description :
539-------------
540 - every 2 seconds, haproxy will try to access the file "/running" on the
541 servers, and declare the server as down after 2 attempts (4 seconds).
542 - only the servers which respond with a 200 or 3XX response will be used.
543 - if a request does not contain a cookie, it will be forwarded to a valid
544 server
545 - if a request contains a cookie for a failed server, haproxy will insist
546 on trying to reach the server anyway, to let the user finish what he was
547 doing. ("persist" option)
548 - if the server is totally stopped, the connection will fail and the proxy
549 will rebalance the client to another server ("redispatch")
550
551Usage on the web servers :
552--------------------------
553- to start the server :
554 # /etc/init.d/httpd start
555 # touch /home/httpd/www/running
556
557- to soft-stop the server
558 # rm -f /home/httpd/www/running
559
560- to completely stop the server :
561 # /etc/init.d/httpd stop
562
563Limits
564------
565If the server is totally powered down, the proxy will still try to reach it
566for those clients who still have a cookie referencing it, and the connection
567attempt will expire after 5 seconds ("contimeout"), and only after that, the
568client will be redispatched to another server. So this mode is only useful
569for software updates where the server will suddenly refuse the connection
570because the process is stopped. The problem is the same if the server suddenly
571crashes. All of its users will be fairly perturbated.
572
573
574==================================
5754.2 Soft-stop using backup servers
576==================================
577
578A better solution which covers every situation is to use backup servers.
579Version 1.1.30 fixed a bug which prevented a backup server from sharing
580the same cookie as a standard server.
581
582
583 listen 192.168.1.1:80
584 mode http
585 balance roundrobin
586 redispatch
587 cookie SERVERID insert indirect
588 option httpchk HEAD / HTTP/1.0
589 server webA 192.168.1.11:80 cookie A check port 81 inter 2000
590 server webB 192.168.1.12:80 cookie B check port 81 inter 2000
591 server webC 192.168.1.13:80 cookie C check port 81 inter 2000
592 server webD 192.168.1.14:80 cookie D check port 81 inter 2000
593
594 server bkpA 192.168.1.11:80 cookie A check port 80 inter 2000 backup
595 server bkpB 192.168.1.12:80 cookie B check port 80 inter 2000 backup
596 server bkpC 192.168.1.13:80 cookie C check port 80 inter 2000 backup
597 server bkpD 192.168.1.14:80 cookie D check port 80 inter 2000 backup
598
599Description
600-----------
601Four servers webA..D are checked on their port 81 every 2 seconds. The same
602servers named bkpA..D are checked on the port 80, and share the exact same
603cookies. Those servers will only be used when no other server is available
604for the same cookie.
605
606When the web servers are started, only the backup servers are seen as
607available. On the web servers, you need to redirect port 81 to local
608port 80, either with a local proxy (eg: a simple haproxy tcp instance),
609or with iptables (linux) or pf (openbsd). This is because we want the
610real web server to reply on this port, and not a fake one. Eg, with
611iptables :
612
613 # /etc/init.d/httpd start
614 # iptables -t nat -A PREROUTING -p tcp --dport 81 -j REDIRECT --to-port 80
615
616A few seconds later, the standard server is seen up and haproxy starts to send
617it new requests on its real port 80 (only new users with no cookie, of course).
618
619If a server completely crashes (even if it does not respond at the IP level),
620both the standard and backup servers will fail, so clients associated to this
621server will be redispatched to other live servers and will lose their sessions.
622
623Now if you want to enter a server into maintenance, simply stop it from
624responding on port 81 so that its standard instance will be seen as failed,
625but the backup will still work. Users will not notice anything since the
626service is still operational :
627
628 # iptables -t nat -D PREROUTING -p tcp --dport 81 -j REDIRECT --to-port 80
629
630The health checks on port 81 for this server will quickly fail, and the
631standard server will be seen as failed. No new session will be sent to this
632server, and existing clients with a valid cookie will still reach it because
633the backup server will still be up.
634
635Now wait as long as you want for the old users to stop using the service, and
636once you see that the server does not receive any traffic, simply stop it :
637
638 # /etc/init.d/httpd stop
639
640The associated backup server will in turn fail, and if any client still tries
641to access this particular server, he will be redispatched to any other valid
642server because of the "redispatch" option.
643
644This method has an advantage : you never touch the proxy when doing server
645maintenance. The people managing the servers can make them disappear smoothly.
646
647
6484.2.1 Variations for operating systems without any firewall software
649--------------------------------------------------------------------
650
651The downside is that you need a redirection solution on the server just for
652the health-checks. If the server OS does not support any firewall software,
653this redirection can also be handled by a simple haproxy in tcp mode :
654
655 global
656 daemon
657 quiet
658 pidfile /var/run/haproxy-checks.pid
659 listen 0.0.0.0:81
660 mode tcp
661 dispatch 127.0.0.1:80
662 contimeout 1000
663 clitimeout 10000
664 srvtimeout 10000
665
666To start the web service :
667
668 # /etc/init.d/httpd start
669 # haproxy -f /etc/haproxy/haproxy-checks.cfg
670
671To soft-stop the service :
672
673 # kill $(</var/run/haproxy-checks.pid)
674
willy tarreauc5f73ed2005-12-18 01:26:38 +0100675The port 81 will stop responding and the load-balancer will notice the failure.
willy tarreau0174f312005-12-18 01:02:42 +0100676
677
6784.2.2 Centralizing the server management
679----------------------------------------
680
willy tarreauc5f73ed2005-12-18 01:26:38 +0100681If one finds it preferable to manage the servers from the load-balancer itself,
willy tarreau0174f312005-12-18 01:02:42 +0100682the port redirector can be installed on the load-balancer itself. See the
683example with iptables below.
684
685Make the servers appear as operational :
686 # iptables -t nat -A OUTPUT -d 192.168.1.11 -p tcp --dport 81 -j DNAT --to-dest :80
687 # iptables -t nat -A OUTPUT -d 192.168.1.12 -p tcp --dport 81 -j DNAT --to-dest :80
688 # iptables -t nat -A OUTPUT -d 192.168.1.13 -p tcp --dport 81 -j DNAT --to-dest :80
689 # iptables -t nat -A OUTPUT -d 192.168.1.14 -p tcp --dport 81 -j DNAT --to-dest :80
690
691Soft stop one server :
692 # iptables -t nat -D OUTPUT -d 192.168.1.12 -p tcp --dport 81 -j DNAT --to-dest :80
693
694Another solution is to use the "COMAFILE" patch provided by Alexander Lazic,
695which is available for download here :
696
697 http://w.ods.org/tools/haproxy/contrib/
698
699
7004.2.3 Notes :
701-------------
702 - Never, ever, start a fake service on port 81 for the health-checks, because
703 a real web service failure will not be detected as long as the fake service
704 runs. You must really forward the check port to the real application.
705
706 - health-checks will be sent twice as often, once for each standard server,
willy tarreau22739ef2006-01-20 20:43:32 +0100707 and once for each backup server. All this will be multiplicated by the
willy tarreauc5f73ed2005-12-18 01:26:38 +0100708 number of processes if you use multi-process mode. You will have to ensure
709 that all the checks sent to the server do not overload it.
willy tarreau0174f312005-12-18 01:02:42 +0100710
willy tarreau22739ef2006-01-20 20:43:32 +0100711=======================
7124.3 Hot reconfiguration
713=======================
714
715There are two types of haproxy users :
716 - those who can never do anything in production out of maintenance periods ;
717 - those who can do anything at any time provided that the consequences are
718 limited.
719
720The first ones have no problem stopping the server to change configuration
721because they got some maintenance periods during which they can break anything.
722So they will even prefer doing a clean stop/start sequence to ensure everything
723will work fine upon next reload. Since those have represented the majority of
724haproxy uses, there has been little effort trying to improve this.
725
726However, the second category is a bit different. They like to be able to fix an
727error in a configuration file without anyone noticing. This can sometimes also
728be the case for the first category because humans are not failsafe.
729
730For this reason, a new hot reconfiguration mechanism has been introduced in
731version 1.1.34. Its usage is very simple and works even in chrooted
732environments with lowered privileges. The principle is very simple : upon
733reception of a SIGTTOU signal, the proxy will stop listening to all the ports.
734This will release the ports so that a new instance can be started. Existing
735connections will not be broken at all. If the new instance fails to start,
736then sending a SIGTTIN signal back to the original processes will restore
737the listening ports. This is possible without any special privileges because
738the sockets will not have been closed, so the bind() is still valid. Otherwise,
739if the new process starts successfully, then sending a SIGUSR1 signal to the
740old one ensures that it will exit as soon as its last session ends.
741
742A hot reconfiguration script would look like this :
743
744 # save previous state
745 mv /etc/haproxy/config /etc/haproxy/config.old
746 mv /var/run/haproxy.pid /var/run/haproxy.pid.old
747
748 mv /etc/haproxy/config.new /etc/haproxy/config
749 kill -TTOU $(cat /var/run/haproxy.pid.old)
750 if haproxy -p /var/run/haproxy.pid -f /etc/haproxy/config; then
751 echo "New instance successfully loaded, stopping previous one."
752 kill -USR1 $(cat /var/run/haproxy.pid.old)
753 rm -f /var/run/haproxy.pid.old
754 exit 1
755 else
756 echo "New instance failed to start, resuming previous one."
757 kill -TTIN $(cat /var/run/haproxy.pid.old)
758 rm -f /var/run/haproxy.pid
759 mv /var/run/haproxy.pid.old /var/run/haproxy.pid
760 mv /etc/haproxy/config /etc/haproxy/config.new
761 mv /etc/haproxy/config.old /etc/haproxy/config
762 exit 0
763 fi
764
765After this, you can still force old connections to end by sending
766a SIGTERM to the old process if it still exists :
767
768 kill $(cat /var/run/haproxy.pid.old)
769 rm -f /var/run/haproxy.pid.old
770
771Be careful with this as in multi-process mode, some pids might already
772have been reallocated to completely different processes.
773
willy tarreau0174f312005-12-18 01:02:42 +0100774
775==================================================
7765. Multi-site load-balancing with local preference
777==================================================
778
7795.1 Description of the problem
780==============================
781
782Consider a world-wide company with sites on several continents. There are two
783production sites SITE1 and SITE2 which host identical applications. There are
784many offices around the world. For speed and communication cost reasons, each
785office uses the nearest site by default, but can switch to the backup site in
786the event of a site or application failure. There also are users on the
787production sites, which use their local sites by default, but can switch to the
788other site in case of a local application failure.
789
790The main constraints are :
791
792 - application persistence : although the application is the same on both
793 sites, there is no session synchronisation between the sites. A failure
794 of one server or one site can cause a user to switch to another server
795 or site, but when the server or site comes back, the user must not switch
796 again.
797
798 - communication costs : inter-site communication should be reduced to the
799 minimum. Specifically, in case of a local application failure, every
800 office should be able to switch to the other site without continuing to
801 use the default site.
802
8035.2 Solution
804============
805 - Each production site will have two haproxy load-balancers in front of its
806 application servers to balance the load across them and provide local HA.
807 We will call them "S1L1" and "S1L2" on site 1, and "S2L1" and "S2L2" on
808 site 2. These proxies will extend the application's JSESSIONID cookie to
809 put the server name as a prefix.
810
811 - Each production site will have one front-end haproxy director to provide
812 the service to local users and to remote offices. It will load-balance
813 across the two local load-balancers, and will use the other site's
814 load-balancers as backup servers. It will insert the local site identifier
815 in a SITE cookie for the local load-balancers, and the remote site
816 identifier for the remote load-balancers. These front-end directors will
817 be called "SD1" and "SD2" for "Site Director".
818
819 - Each office will have one haproxy near the border gateway which will direct
820 local users to their preference site by default, or to the backup site in
821 the event of a previous failure. It will also analyze the SITE cookie, and
822 direct the users to the site referenced in the cookie. Thus, the preferred
823 site will be declared as a normal server, and the backup site will be
824 declared as a backup server only, which will only be used when the primary
825 site is unreachable, or when the primary site's director has forwarded
826 traffic to the second site. These proxies will be called "OP1".."OPXX"
827 for "Office Proxy #XX".
828
829
8305.3 Network diagram
831===================
832
833Note : offices 1 and 2 are on the same continent as site 1, while
834 office 3 is on the same continent as site 3. Each production
835 site can reach the second one either through the WAN or through
836 a dedicated link.
837
838
839 Office1 Office2 Office3
840 users users users
841192.168 # # # 192.168 # # # # # #
842.1.0/24 | | | .2.0/24 | | | 192.168.3.0/24 | | |
843 --+----+-+-+- --+----+-+-+- ---+----+-+-+-
844 | | .1 | | .1 | | .1
845 | +-+-+ | +-+-+ | +-+-+
846 | |OP1| | |OP2| | |OP3| ...
847 ,-:-. +---+ ,-:-. +---+ ,-:-. +---+
848 ( X ) ( X ) ( X )
849 `-:-' `-:-' ,---. `-:-'
850 --+---------------+------+----~~~( X )~~~~-------+---------+-
851 | `---' |
852 | |
853 +---+ ,-:-. +---+ ,-:-.
854 |SD1| ( X ) |SD2| ( X )
855 ( SITE 1 ) +-+-+ `-:-' ( SITE 2 ) +-+-+ `-:-'
856 |.1 | |.1 |
857 10.1.1.0/24 | | ,---. 10.2.1.0/24 | |
858 -+-+-+-+-+-+-+-----+-+--( X )------+-+-+-+-+-+-+-----+-+--
859 | | | | | | | `---' | | | | | | |
860 ...# # # # # |.11 |.12 ...# # # # # |.11 |.12
861 Site 1 +-+--+ +-+--+ Site 2 +-+--+ +-+--+
862 Local |S1L1| |S1L2| Local |S2L1| |S2L2|
863 users +-+--+ +--+-+ users +-+--+ +--+-+
864 | | | |
865 10.1.2.0/24 -+-+-+--+--++-- 10.2.2.0/24 -+-+-+--+--++--
866 |.1 |.4 |.1 |.4
867 +-+-+ +-+-+ +-+-+ +-+-+
868 |W11| ~~~ |W14| |W21| ~~~ |W24|
869 +---+ +---+ +---+ +---+
870 4 application servers 4 application servers
871 on site 1 on site 2
872
873
874
8755.4 Description
876===============
877
8785.4.1 Local users
879-----------------
880 - Office 1 users connect to OP1 = 192.168.1.1
881 - Office 2 users connect to OP2 = 192.168.2.1
882 - Office 3 users connect to OP3 = 192.168.3.1
883 - Site 1 users connect to SD1 = 10.1.1.1
884 - Site 2 users connect to SD2 = 10.2.1.1
885
8865.4.2 Office proxies
887--------------------
888 - Office 1 connects to site 1 by default and uses site 2 as a backup.
889 - Office 2 connects to site 1 by default and uses site 2 as a backup.
890 - Office 3 connects to site 2 by default and uses site 1 as a backup.
891
892The offices check the local site's SD proxy every 30 seconds, and the
893remote one every 60 seconds.
894
895
896Configuration for Office Proxy OP1
897----------------------------------
898
899 listen 192.168.1.1:80
900 mode http
901 balance roundrobin
902 redispatch
903 cookie SITE
904 option httpchk HEAD / HTTP/1.0
905 server SD1 10.1.1.1:80 cookie SITE1 check inter 30000
906 server SD2 10.2.1.1:80 cookie SITE2 check inter 60000 backup
907
908
909Configuration for Office Proxy OP2
910----------------------------------
911
912 listen 192.168.2.1:80
913 mode http
914 balance roundrobin
915 redispatch
916 cookie SITE
917 option httpchk HEAD / HTTP/1.0
918 server SD1 10.1.1.1:80 cookie SITE1 check inter 30000
919 server SD2 10.2.1.1:80 cookie SITE2 check inter 60000 backup
920
921
922Configuration for Office Proxy OP3
923----------------------------------
924
925 listen 192.168.3.1:80
926 mode http
927 balance roundrobin
928 redispatch
929 cookie SITE
930 option httpchk HEAD / HTTP/1.0
931 server SD2 10.2.1.1:80 cookie SITE2 check inter 30000
932 server SD1 10.1.1.1:80 cookie SITE1 check inter 60000 backup
933
934
9355.4.3 Site directors ( SD1 and SD2 )
936------------------------------------
937The site directors forward traffic to the local load-balancers, and set a
938cookie to identify the site. If no local load-balancer is available, or if
939the local application servers are all down, it will redirect traffic to the
940remote site, and report this in the SITE cookie. In order not to uselessly
941load each site's WAN link, each SD will check the other site at a lower
942rate. The site directors will also insert their client's address so that
943the application server knows which local user or remote site accesses it.
944
945The SITE cookie which is set by these directors will also be understood
946by the office proxies. This is important because if SD1 decides to forward
947traffic to site 2, it will write "SITE2" in the "SITE" cookie, and on next
948request, the office proxy will automatically and directly talk to SITE2 if
949it can reach it. If it cannot, it will still send the traffic to SITE1
950where SD1 will in turn try to reach SITE2.
951
952The load-balancers checks are performed on port 81. As we'll see further,
953the load-balancers provide a health monitoring port 81 which reroutes to
954port 80 but which allows them to tell the SD that they are going down soon
955and that the SD must not use them anymore.
956
957
958Configuration for SD1
959---------------------
960
961 listen 10.1.1.1:80
962 mode http
963 balance roundrobin
964 redispatch
965 cookie SITE insert indirect
966 option httpchk HEAD / HTTP/1.0
967 option forwardfor
968 server S1L1 10.1.1.11:80 cookie SITE1 check port 81 inter 4000
969 server S1L2 10.1.1.12:80 cookie SITE1 check port 81 inter 4000
970 server S2L1 10.2.1.11:80 cookie SITE2 check port 81 inter 8000 backup
971 server S2L2 10.2.1.12:80 cookie SITE2 check port 81 inter 8000 backup
972
973Configuration for SD2
974---------------------
975
976 listen 10.2.1.1:80
977 mode http
978 balance roundrobin
979 redispatch
980 cookie SITE insert indirect
981 option httpchk HEAD / HTTP/1.0
982 option forwardfor
983 server S2L1 10.2.1.11:80 cookie SITE2 check port 81 inter 4000
984 server S2L2 10.2.1.12:80 cookie SITE2 check port 81 inter 4000
985 server S1L1 10.1.1.11:80 cookie SITE1 check port 81 inter 8000 backup
986 server S1L2 10.1.1.12:80 cookie SITE1 check port 81 inter 8000 backup
987
988
9895.4.4 Local load-balancers S1L1, S1L2, S2L1, S2L2
990-------------------------------------------------
991Please first note that because SD1 and SD2 use the same cookie for both
992servers on a same site, the second load-balancer of each site will only
993receive load-balanced requests, but as soon as the SITE cookie will be
994set, only the first LB will receive the requests because it will be the
995first one to match the cookie.
996
997The load-balancers will spread the load across 4 local web servers, and
998use the JSESSIONID provided by the application to provide server persistence
999using the new 'prefix' method. Soft-stop will also be implemented as described
1000in section 4 above. Moreover, these proxies will provide their own maintenance
1001soft-stop. Port 80 will be used for application traffic, while port 81 will
1002only be used for health-checks and locally rerouted to port 80. A grace time
1003will be specified to service on port 80, but not on port 81. This way, a soft
1004kill (kill -USR1) on the proxy will only kill the health-check forwarder so
1005that the site director knows it must not use this load-balancer anymore. But
1006the service will still work for 20 seconds and as long as there are established
1007sessions.
1008
1009These proxies will also be the only ones to disable HTTP keep-alive in the
1010chain, because it is enough to do it at one place, and it's necessary to do
1011it with 'prefix' cookies.
1012
1013Configuration for S1L1/S1L2
1014---------------------------
1015
1016 listen 10.1.1.11:80 # 10.1.1.12:80 for S1L2
1017 grace 20000 # don't kill us until 20 seconds have elapsed
1018 mode http
1019 balance roundrobin
1020 cookie JSESSIONID prefix
1021 option httpclose
1022 option forwardfor
1023 option httpchk HEAD / HTTP/1.0
1024 server W11 10.1.2.1:80 cookie W11 check port 81 inter 2000
1025 server W12 10.1.2.2:80 cookie W12 check port 81 inter 2000
1026 server W13 10.1.2.3:80 cookie W13 check port 81 inter 2000
1027 server W14 10.1.2.4:80 cookie W14 check port 81 inter 2000
1028
1029 server B11 10.1.2.1:80 cookie W11 check port 80 inter 4000 backup
1030 server B12 10.1.2.2:80 cookie W12 check port 80 inter 4000 backup
1031 server B13 10.1.2.3:80 cookie W13 check port 80 inter 4000 backup
1032 server B14 10.1.2.4:80 cookie W14 check port 80 inter 4000 backup
1033
1034 listen 10.1.1.11:81 # 10.1.1.12:81 for S1L2
1035 mode tcp
1036 dispatch 10.1.1.11:80 # 10.1.1.12:80 for S1L2
1037
1038
1039Configuration for S2L1/S2L2
1040---------------------------
1041
1042 listen 10.2.1.11:80 # 10.2.1.12:80 for S2L2
1043 grace 20000 # don't kill us until 20 seconds have elapsed
1044 mode http
1045 balance roundrobin
1046 cookie JSESSIONID prefix
1047 option httpclose
1048 option forwardfor
1049 option httpchk HEAD / HTTP/1.0
1050 server W21 10.2.2.1:80 cookie W21 check port 81 inter 2000
1051 server W22 10.2.2.2:80 cookie W22 check port 81 inter 2000
1052 server W23 10.2.2.3:80 cookie W23 check port 81 inter 2000
1053 server W24 10.2.2.4:80 cookie W24 check port 81 inter 2000
1054
1055 server B21 10.2.2.1:80 cookie W21 check port 80 inter 4000 backup
1056 server B22 10.2.2.2:80 cookie W22 check port 80 inter 4000 backup
1057 server B23 10.2.2.3:80 cookie W23 check port 80 inter 4000 backup
1058 server B24 10.2.2.4:80 cookie W24 check port 80 inter 4000 backup
1059
1060 listen 10.2.1.11:81 # 10.2.1.12:81 for S2L2
1061 mode tcp
1062 dispatch 10.2.1.11:80 # 10.2.1.12:80 for S2L2
1063
1064
10655.5 Comments
1066------------
1067Since each site director sets a cookie identifying the site, remote office
1068users will have their office proxies direct them to the right site and stick
1069to this site as long as the user still uses the application and the site is
1070available. Users on production sites will be directed to the right site by the
1071site directors depending on the SITE cookie.
1072
1073If the WAN link dies on a production site, the remote office users will not
1074see their site anymore, so they will redirect the traffic to the second site.
1075If there are dedicated inter-site links as on the diagram above, the second
1076SD will see the cookie and still be able to reach the original site. For
1077example :
1078
1079Office 1 user sends the following to OP1 :
1080 GET / HTTP/1.0
1081 Cookie: SITE=SITE1; JSESSIONID=W14~123;
1082
1083OP1 cannot reach site 1 because its external router is dead. So the SD1 server
1084is seen as dead, and OP1 will then forward the request to SD2 on site 2,
1085regardless of the SITE cookie.
1086
1087SD2 on site 2 receives a SITE cookie containing "SITE1". Fortunately, it
1088can reach Site 1's load balancers S1L1 and S1L2. So it forwards the request
1089so S1L1 (the first one with the same cookie).
1090
1091S1L1 (on site 1) finds "W14" in the JSESSIONID cookie, so it can forward the
1092request to the right server, and the user session will continue to work. Once
1093the Site 1's WAN link comes back, OP1 will see SD1 again, and will not route
1094through SITE 2 anymore.
1095
1096However, when a new user on Office 1 connects to the application during a
1097site 1 failure, it does not contain any cookie. Since OP1 does not see SD1
1098because of the network failure, it will direct the request to SD2 on site 2,
1099which will by default direct the traffic to the local load-balancers, S2L1 and
1100S2L2. So only initial users will load the inter-site link, not the new ones.
1101
1102
1103===================
11046. Source balancing
1105===================
1106
1107Sometimes it may reveal useful to access servers from a pool of IP addresses
1108instead of only one or two. Some equipments (NAT firewalls, load-balancers)
1109are sensible to source address, and often need many sources to distribute the
1110load evenly amongst their internal hash buckets.
1111
1112To do this, you simply have to use several times the same server with a
1113different source. Example :
1114
1115 listen 0.0.0.0:80
1116 mode tcp
1117 balance roundrobin
1118 server from1to1 10.1.1.1:80 source 10.1.2.1
1119 server from2to1 10.1.1.1:80 source 10.1.2.2
1120 server from3to1 10.1.1.1:80 source 10.1.2.3
1121 server from4to1 10.1.1.1:80 source 10.1.2.4
1122 server from5to1 10.1.1.1:80 source 10.1.2.5
1123 server from6to1 10.1.1.1:80 source 10.1.2.6
1124 server from7to1 10.1.1.1:80 source 10.1.2.7
1125 server from8to1 10.1.1.1:80 source 10.1.2.8
1126