blob: 052177d175d280b0ce21d69936a1db7b955f117c [file] [log] [blame]
Willy Tarreaud8e42b62015-08-18 21:51:36 +02001 -----------------------
2 HAProxy Starter Guide
3 -----------------------
Willy Tarreau7d1b48f2016-05-10 15:36:58 +02004 version 1.7
Willy Tarreaud8e42b62015-08-18 21:51:36 +02005
6
7This document is an introduction to HAProxy for all those who don't know it, as
8well as for those who want to re-discover it when they know older versions. Its
9primary focus is to provide users with all the elements to decide if HAProxy is
10the product they're looking for or not. Advanced users may find here some parts
11of solutions to some ideas they had just because they were not aware of a given
12new feature. Some sizing information are also provided, the product's lifecycle
13is explained, and comparisons with partially overlapping products are provided.
14
15This document doesn't provide any configuration help nor hint, but it explains
16where to find the relevant documents. The summary below is meant to help you
17search sections by name and navigate through the document.
18
19Note to documentation contributors :
20 This document is formatted with 80 columns per line, with even number of
21 spaces for indentation and without tabs. Please follow these rules strictly
22 so that it remains easily printable everywhere. If you add sections, please
23 update the summary below for easier searching.
24
25
26Summary
27-------
28
291. Available documentation
30
312. Quick introduction to load balancing and load balancers
32
333. Introduction to HAProxy
343.1. What HAProxy is and is not
353.2. How HAProxy works
363.3. Basic features
373.3.1. Proxying
383.3.2. SSL
393.3.3. Monitoring
403.3.4. High availability
413.3.5. Load balancing
423.3.6. Stickiness
433.3.7. Sampling and converting information
443.3.8. Maps
453.3.9. ACLs and conditions
463.3.10. Content switching
473.3.11. Stick-tables
483.3.12. Formated strings
493.3.13. HTTP rewriting and redirection
503.3.14. Server protection
513.3.15. Logging
523.3.16. Statistics
533.4. Advanced features
543.4.1. Management
553.4.2. System-specific capabilities
563.4.3. Scripting
573.5. Sizing
583.6. How to get HAProxy
59
604. Companion products and alternatives
614.1. Apache HTTP server
624.2. NGINX
634.3. Varnish
644.4. Alternatives
65
66
671. Available documentation
68--------------------------
69
70The complete HAProxy documentation is contained in the following documents.
71Please ensure to consult the relevant documentation to save time and to get the
72most accurate response to your needs. Also please refrain from sending questions
73to the mailing list whose responses are present in these documents.
74
75 - intro.txt (this document) : it presents the basics of load balancing,
76 HAProxy as a product, what it does, what it doesn't do, some known traps to
77 avoid, some OS-specific limitations, how to get it, how it evolves, how to
78 ensure you're running with all known fixes how to update it, complements and
79 alternatives.
80
Willy Tarreau373933d2015-10-13 16:32:20 +020081 - management.txt : it explains how to start haproxy, how to manage it at
82 runtime, how to manage it on multiple nodes, how to proceed with seamless
83 upgrades.
84
Willy Tarreaud8e42b62015-08-18 21:51:36 +020085 - configuration.txt : the reference manual details all configuration keywords
86 and their options. It is used when a configuration change is needed.
87
88 - architecture.txt : the architecture manual explains how to best architect a
89 load-balanced infrastructure and how to interact with third party products.
90
91 - coding-style.txt : this is for developers who want to propose some code to
92 the project. It explains the style to adopt for the code. It's not very
93 strict and not all the code base completely respects it but contributions
94 which diverge too much from it will be rejected.
95
96 - proxy-protocol.txt : this is the de-facto specification of the PROXY
97 protocol which is implemented by HAProxy and a number of third party
98 products.
99
100 - README : how to build haproxy from sources
101
102
1032. Quick introduction to load balancing and load balancers
104----------------------------------------------------------
105
106Load balancing consists in aggregating multiple components in order to achieve
107a total processing capacity above each component's individual capacity, without
108any intervention from the end user and in a scalable way. This results in more
Willy Tarreaueff04f42015-08-27 14:44:43 +0200109operations being performed simultaneously by the time it takes a component to
Willy Tarreaud8e42b62015-08-18 21:51:36 +0200110perform only one. A single operation however will still be performed on a single
111component at a time and will not get faster than without load balancing. It
112always requires at least as many operations as available components and an
113efficient load balancing mechanism to make use of all components and to fully
114benefit from the load balancing. A good example of this is the number of lanes
115on a highway which allows as many cars to pass during the same time frame
116without increasing their individual speed.
117
118Examples of load balancing :
119
120 - Process scheduling in multi-processor systems
121 - Link load balancing (eg: EtherChannel, Bonding)
122 - IP address load balancing (eg: ECMP, DNS roundrobin)
123 - Server load balancing (via load balancers)
124
125The mechanism or component which performs the load balancing operation is
126called a load balancer. In web environments these components are called a
127"network load balancer", and more commonly a "load balancer" given that this
128activity is by far the best known case of load balancing.
129
130A load balancer may act :
131
132 - at the link level : this is called link load balancing, and it consists in
133 chosing what network link to send a packet to;
134
135 - at the network level : this is called network load balancing, and it
136 consists in chosing what route a series of packets will follow;
137
138 - at the server level : this is called server load balancing and it consists
139 in deciding what server will process a connection or request.
140
141Two distinct technologies exist and address different needs, though with some
Willy Tarreaueff04f42015-08-27 14:44:43 +0200142overlapping. In each case it is important to keep in mind that load balancing
143consists in diverting the traffic from its natural flow and that doing so always
144requires a minimum of care to maintain the required level of consistency between
145all routing decisions.
Willy Tarreaud8e42b62015-08-18 21:51:36 +0200146
147The first one acts at the packet level and processes packets more or less
148individually. There is a 1-to-1 relation between input and output packets, so
149it is possible to follow the traffic on both sides of the load balancer using a
150regular network sniffer. This technology can be very cheap and extremely fast.
151It is usually implemented in hardware (ASICs) allowing to reach line rate, such
152as switches doing ECMP. Usually stateless, it can also be stateful (consider
153the session a packet belongs to and called layer4-LB or L4), may support DSR
154(direct server return, without passing through the LB again) if the packets
155were not modified, but provides almost no content awareness. This technology is
156very well suited to network-level load balancing, though it is sometimes used
157for very basic server load balancing at high speed.
158
159The second one acts on session contents. It requires that the input streams is
160reassembled and processed as a whole. The contents may be modified, and the
161output stream is segmented into new packets. For this reason it is generally
162performed by proxies and they're often called layer 7 load balancers or L7.
163This implies that there are two distinct connections on each side, and that
164there is no relation between input and output packets sizes nor counts. Clients
165and servers are not required to use the same protocol (for example IPv4 vs
166IPv6, clear vs SSL). The operations are always stateful, and the return traffic
167must pass through the load balancer. The extra processing comes with a cost so
168it's not always possible to achieve line rate, especially with small packets.
169On the other hand, it offers wide possibilities and is generally achieved by
170pure software, even if embedded into hardware appliances. This technology is
171very well suited for server load balancing.
172
173Packet-based load balancers are generally deployed in cut-through mode, so they
174are installed on the normal path of the traffic and divert it according to the
175configuration. The return traffic doesn't necessarily pass through the load
176balancer. Some modifications may be applied to the network destination address
177in order to direct the traffic to the proper destination. In this case, it is
178mandatory that the return traffic passes through the load balancer. If the
179routes doesn't make this possible, the load balancer may also replace the
180packets' source address with its own in order to force the return traffic to
181pass through it.
182
183Proxy-based load balancers are deployed as a server with their own IP address
184and ports, without architecture changes. Sometimes this requires to perform some
185adaptations to the applications so that clients are properly directed to the
186load balancer's IP address and not directly to the server's. Some load balancers
187may have to adjust some servers' responses to make this possible (eg: the HTTP
188Location header field used in HTTP redirects). Some proxy-based load balancers
189may intercept traffic for an address they don't own, and spoof the client's
190address when connecting to the server. This allows them to be deployed as if
191they were a regular router or firewall, in a cut-through mode very similar to
192the packet based load balancers. This is particularly appreciated for products
193which combine both packet mode and proxy mode. In this case DSR is obviously
194still not possible and the return traffic still has to be routed back to the
195load balancer.
196
197A very scalable layered approach would consist in having a front router which
198receives traffic from multiple load balanced links, and uses ECMP to distribute
199this traffic to a first layer of multiple stateful packet-based load balancers
200(L4). These L4 load balancers in turn pass the traffic to an even larger number
201of proxy-based load balancers (L7), which have to parse the contents to decide
202what server will ultimately receive the traffic.
203
204The number of components and possible paths for the traffic increases the risk
205of failure; in very large environments, it is even normal to permanently have
206a few faulty components being fixed or replaced. Load balancing done without
207awareness of the whole stack's health significantly degrades availability. For
208this reason, any sane load balancer will verify that the components it intends
209to deliver the traffic to are still alive and reachable, and it will stop
210delivering traffic to faulty ones. This can be achieved using various methods.
211
212The most common one consists in periodically sending probes to ensure the
213component is still operational. These probes are called "health checks". They
214must be representative of the type of failure to address. For example a ping-
215based check will not detect that a web server has crashed and doesn't listen to
216a port anymore, while a connection to the port will verify this, and a more
217advanced request may even validate that the server still works and that the
218database it relies on is still accessible. Health checks often involve a few
219retries to cover for occasional measuring errors. The period between checks
220must be small enough to ensure the faulty component is not used for too long
221after an error occurs.
222
223Other methods consist in sampling the production traffic sent to a destination
224to observe if it is processed correctly or not, and to evince the components
225which return inappropriate responses. However this requires to sacrify a part
226of the production traffic and this is not always acceptable. A combination of
227these two mechanisms provides the best of both worlds, with both of them being
228used to detect a fault, and only health checks to detect the end of the fault.
229A last method involves centralized reporting : a central monitoring agent
230periodically updates all load balancers about all components' state. This gives
231a global view of the infrastructure to all components, though sometimes with
232less accuracy or responsiveness. It's best suited for environments with many
233load balancers and many servers.
234
235Layer 7 load balancers also face another challenge known as stickiness or
236persistence. The principle is that they generally have to direct multiple
237subsequent requests or connections from a same origin (such as an end user) to
238the same target. The best known example is the shopping cart on an online
239store. If each click leads to a new connection, the user must always be sent
240to the server which holds his shopping cart. Content-awareness makes it easier
241to spot some elements in the request to identify the server to deliver it to,
242but that's not always enough. For example if the source address is used as a
243key to pick a server, it can be decided that a hash-based algorithm will be
244used and that a given IP address will always be sent to the same server based
245on a divide of the address by the number of available servers. But if one
246server fails, the result changes and all users are suddenly sent to a different
247server and lose their shopping cart. The solution against this issue consists
248in memorizing the chosen target so that each time the same visitor is seen,
249he's directed to the same server regardless of the number of available servers.
250The information may be stored in the load balancer's memory, in which case it
251may have to be replicated to other load balancers if it's not alone, or it may
252be stored in the client's memory using various methods provided that the client
253is able to present this information back with every request (cookie insertion,
254redirection to a sub-domain, etc). This mechanism provides the extra benefit of
255not having to rely on unstable or unevenly distributed information (such as the
256source IP address). This is in fact the strongest reason to adopt a layer 7
257load balancer instead of a layer 4 one.
258
259In order to extract information such as a cookie, a host header field, a URL
260or whatever, a load balancer may need to decrypt SSL/TLS traffic and even
261possibly to reencrypt it when passing it to the server. This expensive task
262explains why in some high-traffic infrastructures, sometimes there may be a
263lot of load balancers.
264
265Since a layer 7 load balancer may perform a number of complex operations on the
266traffic (decrypt, parse, modify, match cookies, decide what server to send to,
267etc), it can definitely cause some trouble and will very commonly be accused of
268being responsible for a lot of trouble that it only revealed. Often it will be
269discovered that servers are unstable and periodically go up and down, or for
270web servers, that they deliver pages with some hard-coded links forcing the
271clients to connect directly to one specific server without passing via the load
272balancer, or that they take ages to respond under high load causing timeouts.
273That's why logging is an extremely important aspect of layer 7 load balancing.
274Once a trouble is reported, it is important to figure if the load balancer took
275a wrong decision and if so why so that it doesn't happen anymore.
276
277
2783. Introduction to HAProxy
279--------------------------
280
281HAProxy is written "HAProxy" to designate the product, "haproxy" to designate
282the executable program, software package or a process, though both are commonly
283used for both purposes, and is pronounced H-A-Proxy. Very early it used to stand
284for "high availability proxy" and the name was written in two separate words,
285though by now it means nothing else than "HAProxy".
286
287
2883.1. What HAProxy is and is not
289-------------------------------
290
291HAProxy is :
292
293 - a TCP proxy : it can accept a TCP connection from a listening socket,
294 connect to a server and attach these sockets together allowing traffic to
295 flow in both directions;
296
297 - an HTTP reverse-proxy (called a "gateway" in HTTP terminology) : it presents
298 itself as a server, receives HTTP requests over connections accepted on a
299 listening TCP socket, and passes the requests from these connections to
300 servers using different connections.
301
302 - an SSL terminator / initiator / offloader : SSL/TLS may be used on the
303 connection coming from the client, on the connection going to the server,
304 or even on both connections.
305
306 - a TCP normalizer : since connections are locally terminated by the operating
307 system, there is no relation between both sides, so abnormal traffic such as
308 invalid packets, flag combinations, window advertisements, sequence numbers,
309 incomplete connections (SYN floods), or so will not be passed to the other
310 side. This protects fragile TCP stacks from protocol attacks, and also
311 allows to optimize the connection parameters with the client without having
312 to modify the servers' TCP stack settings.
313
314 - an HTTP normalizer : when configured to process HTTP traffic, only valid
315 complete requests are passed. This protects against a lot of protocol-based
316 attacks. Additionally, protocol deviations for which there is a tolerance
317 in the specification are fixed so that they don't cause problem on the
318 servers (eg: multiple-line headers).
319
320 - an HTTP fixing tool : it can modify / fix / add / remove / rewrite the URL
321 or any request or response header. This helps fixing interoperability issues
322 in complex environments.
323
324 - a content-based switch : it can consider any element from the request to
325 decide what server to pass the request or connection to. Thus it is possible
326 to handle multiple protocols over a same port (eg: http, https, ssh).
327
328 - a server load balancer : it can load balance TCP connections and HTTP
329 requests. In TCP mode, load balancing decisions are taken for the whole
330 connection. In HTTP mode, decisions are taken per request.
331
332 - a traffic regulator : it can apply some rate limiting at various points,
333 protect the servers against overloading, adjust traffic priorities based on
334 the contents, and even pass such information to lower layers and outer
335 network components by marking packets.
336
337 - a protection against DDoS and service abuse : it can maintain a wide number
338 of statistics per IP address, URL, cookie, etc and detect when an abuse is
339 happening, then take action (slow down the offenders, block them, send them
340 to outdated contents, etc).
341
342 - an observation point for network troubleshooting : due to the precision of
343 the information reported in logs, it is often used to narrow down some
344 network-related issues.
345
346 - an HTTP compression offloader : it can compress responses which were not
347 compressed by the server, thus reducing the page load time for clients with
348 poor connectivity or using high-latency, mobile networks.
349
350HAProxy is not :
351
352 - an explicit HTTP proxy, ie, the proxy that browsers use to reach the
353 internet. There are excellent open-source software dedicated for this task,
354 such as Squid. However HAProxy can be installed in front of such a proxy to
355 provide load balancing and high availability.
356
357 - a caching proxy : it will return as-is the contents its received from the
358 server and will not interfere with any caching policy. There are excellent
359 open-source software for this task such as Varnish. HAProxy can be installed
360 in front of such a cache to provide SSL offloading, and scalability through
361 smart load balancing.
362
363 - a data scrubber : it will not modify the body of requests nor responses.
364
365 - a web server : during startup, it isolates itself inside a chroot jail and
366 drops its privileges, so that it will not perform any single file-system
367 access once started. As such it cannot be turned into a web server. There
368 are excellent open-source software for this such as Apache or Nginx, and
369 HAProxy can be installed in front of them to provide load balancing and
370 high availability.
371
372 - a packet-based load balancer : it will not see IP packets nor UDP datagrams,
373 will not perform NAT or even less DSR. These are tasks for lower layers.
374 Some kernel-based components such as IPVS (Linux Virtual Server) already do
375 this pretty well and complement perfectly with HAProxy.
376
377
3783.2. How HAProxy works
379----------------------
380
381HAProxy is a single-threaded, event-driven, non-blocking engine combining a very
382fast I/O layer with a priority-based scheduler. As it is designed with a data
383forwarding goal in mind, its architecture is optimized to move data as fast as
384possible with the least possible operations. As such it implements a layered
385model offering bypass mechanisms at each level ensuring data don't reach higher
386levels when not needed. Most of the processing is performed in the kernel, and
387HAProxy does its best to help the kernel do the work as fast as possible by
388giving some hints or by avoiding certain operation when it guesses they could
389be grouped later. As a result, typical figures show 15% of the processing time
390spent in HAProxy versus 85% in the kernel in TCP or HTTP close mode, and about
39130% for HAProxy versus 70% for the kernel in HTTP keep-alive mode.
392
393A single process can run many proxy instances; configurations as large as
394300000 distinct proxies in a single process were reported to run fine. Thus
395there is usually no need to start more than one process for all instances.
396
397It is possible to make HAProxy run over multiple processes, but it comes with
398a few limitations. In general it doesn't make sense in HTTP close or TCP modes
399because the kernel-side doesn't scale very well with some operations such as
400connect(). It scales pretty well for HTTP keep-alive mode but the performance
401that can be achieved out of a single process generaly outperforms common needs
402by an order of magnitude. It does however make sense when used as an SSL
403offloader, and this feature is well supported in multi-process mode.
404
405HAProxy only requires the haproxy executable and a configuration file to run.
406For logging it is highly recommended to have a properly configured syslog daemon
407and log rotations in place. The configuration files are parsed before starting,
408then HAProxy tries to bind all listening sockets, and refuses to start if
409anything fails. Past this point it cannot fail anymore. This means that there
410are no runtime failures and that if it accepts to start, it will work until it
411is stopped.
412
413Once HAProxy is started, it does exactly 3 things :
414
415 - process incoming connections;
416
417 - periodically check the servers' status (known as health checks);
418
419 - exchange information with other haproxy nodes.
420
421Processing incoming connections is by far the most complex task as it depends
422on a lot of configuration possibilities, but it can be summarized as the 9 steps
423below :
424
425 - accept incoming connections from listening sockets that belong to a
426 configuration entity known as a "frontend", which references one or multiple
427 listening addresses;
428
429 - apply the frontend-specific processing rules to these connections that may
430 result in blocking them, modifying some headers, or intercepting them to
431 execute some internal applets such as the statistics page or the CLI;
432
433 - pass these incoming connections to another configuration entity representing
434 a server farm known as a "backend", which contains the list of servers and
435 the load balancing strategy for this server farm;
436
437 - apply the backend-specific processing rules to these connections;
438
439 - decide which server to forward the connection to according to the load
440 balancing strategy;
441
442 - apply the backend-specific processing rules to the response data;
443
444 - apply the frontend-specific processing rules to the response data;
445
446 - emit a log to report what happened in fine details;
447
448 - in HTTP, loop back to the second step to wait for a new request, otherwise
449 close the connection.
450
451Frontends and backends are sometimes considered as half-proxies, since they only
452look at one side of an end-to-end connection; the frontend only cares about the
453clients while the backend only cares about the servers. HAProxy also supports
454full proxies which are exactly the union of a frontend and a backend. When HTTP
455processing is desired, the configuration will generally be split into frontends
456and backends as they open a lot of possibilities since any frontend may pass a
457connection to any backend. With TCP-only proxies, using frontends and backends
458rarely provides a benefit and the configuration can be more readable with full
459proxies.
460
461
4623.3. Basic features
463-------------------
464
465This section will enumerate a number of features that HAProxy implements, some
466of which are generally expected from any modern load balancer, and some of
467which are a direct benefit of HAProxy's architecture. More advanced features
468will be detailed in the next section.
469
470
4713.3.1. Basic features : Proxying
472--------------------------------
473
474Proxying is the action of transferring data between a client and a server over
475two independant connections. The following basic features are supported by
476HAProxy regarding proxying and connection management :
477
478 - Provide the server with a clean connection to protect them against any
479 client-side defect or attack;
480
481 - Listen to multiple IP address and/or ports, even port ranges;
482
483 - Transparent accept : intercept traffic targetting any arbitrary IP address
484 that doesn't even belong to the local system;
485
486 - Server port doesn't need to be related to listening port, and may even be
487 translated by a fixed offset (useful with ranges);
488
489 - Transparent connect : spoof the client's (or any) IP address if needed
490 when connecting to the server;
491
492 - Provide a reliable return IP address to the servers in multi-site LBs;
493
494 - Offload the server thanks to buffers and possibly short-lived connections
495 to reduce their concurrent connection count and their memory footprint;
496
497 - Optimize TCP stacks (eg: SACK), congestion control, and reduce RTT impacts;
498
499 - Support different protocol families on both sides (eg: IPv4/IPv6/Unix);
500
501 - Timeout enforcement : HAProxy supports multiple levels of timeouts depending
502 on the stage the connection is, so that a dead client or server, or an
503 attacker cannot be granted resources for too long;
504
505 - Protocol validation: HTTP, SSL, or payload are inspected and invalid
506 protocol elements are rejected, unless instructed to accept them anyway;
507
508 - Policy enforcement : ensure that only what is allowed may be forwarded;
509
510 - Both incoming and outgoing connections may be limited to certain network
511 namespaces (Linux only), making it easy to build a cross-container,
512 multi-tenant load balancer;
513
514 - PROXY protocol presents the client's IP address to the server even for
515 non-HTTP traffic. This is an HAProxy extension that was adopted by a number
516 of third-party products by now, at least these ones at the time of writing :
517 - client : haproxy, stud, stunnel, exaproxy, ELB, squid
518 - server : haproxy, stud, postfix, exim, nginx, squid, node.js, varnish
519
520
5213.3.2. Basic features : SSL
522---------------------------
523
524HAProxy's SSL stack is recognized as one of the most featureful according to
525Google's engineers (http://istlsfastyet.com/). The most commonly used features
526making it quite complete are :
527
528 - SNI-based multi-hosting with no limit on sites count and focus on
529 performance. At least one deployment is known for running 50000 domains
530 with their respective certificates;
531
532 - support for wildcard certificates reduces the need for many certificates ;
533
534 - certificate-based client authentication with configurable policies on
535 failure to present a valid certificate. This allows to present a different
536 server farm to regenerate the client certificate for example;
537
538 - authentication of the backend server ensures the backend server is the real
539 one and not a man in the middle;
540
541 - authentication with the backend server lets the backend server it's really
542 the expected haproxy node that is connecting to it;
543
544 - TLS NPN and ALPN extensions make it possible to reliably offload SPDY/HTTP2
545 connections and pass them in clear text to backend servers;
546
547 - OCSP stapling further reduces first page load time by delivering inline an
548 OCSP response when the client requests a Certificate Status Request;
549
550 - Dynamic record sizing provides both high performance and low latency, and
551 significantly reduces page load time by letting the browser start to fetch
552 new objects while packets are still in flight;
553
554 - permanent access to all relevant SSL/TLS layer information for logging,
555 access control, reporting etc... These elements can be embedded into HTTP
556 header or even as a PROXY protocol extension so that the offloaded server
557 gets all the information it would have had if it performed the SSL
558 termination itself.
559
560 - Detect, log and block certain known attacks even on vulnerable SSL libs,
561 such as the Heartbleed attack affecting certain versions of OpenSSL.
562
Pavlos Parissisba56d9c2015-08-24 13:14:32 +0200563 - support for stateless session resumption (RFC 5077 TLS Ticket extension).
564 TLS tickets can be updated from CLI which provides them means to implement
565 Perfect Forward Secrecy by frequently rotating the tickets.
566
Willy Tarreaud8e42b62015-08-18 21:51:36 +0200567
5683.3.3. Basic features : Monitoring
569----------------------------------
570
571HAProxy focuses a lot on availability. As such it cares about servers state,
572and about reporting its own state to other network components :
573
574 - Servers state is continuously monitored using per-server parameters. This
575 ensures the path to the server is operational for regular traffic;
576
577 - Health checks support two hysteresis for up and down transitions in order
578 to protect against state flapping;
579
580 - Checks can be sent to a different address/port/protocol : this makes it
581 easy to check a single service that is considered representative of multiple
582 ones, for example the HTTPS port for an HTTP+HTTPS server.
583
584 - Servers can track other servers and go down simultaneously : this ensures
585 that servers hosting multiple services can fail atomically and that noone
586 will be sent to a partially failed server;
587
588 - Agents may be deployed on the server to monitor load and health : a server
589 may be interested in reporting its load, operational status, administrative
590 status independantly from what health checks can see. By running a simple
591 agent on the server, it's possible to consider the server's view of its own
592 health in addition to the health checks validating the whole path;
593
594 - Various check methods are available : TCP connect, HTTP request, SMTP hello,
595 SSL hello, LDAP, SQL, Redis, send/expect scripts, all with/without SSL;
596
597 - State change is notified in the logs and stats page with the failure reason
Willy Tarreaueff04f42015-08-27 14:44:43 +0200598 (eg: the HTTP response received at the moment the failure was detected). An
599 e-mail can also be sent to a configurable address upon such a change ;
Willy Tarreaud8e42b62015-08-18 21:51:36 +0200600
601 - Server state is also reported on the stats interface and can be used to take
602 routing decisions so that traffic may be sent to different farms depending
603 on their sizes and/or health (eg: loss of an inter-DC link);
604
605 - HAProxy can use health check requests to pass information to the servers,
606 such as their names, weight, the number of other servers in the farm etc...
607 so that servers can adjust their response and decisions based on this
608 knowledge (eg: postpone backups to keep more CPU available);
609
610 - Servers can use health checks to report more detailed state than just on/off
611 (eg: I would like to stop, please stop sending new visitors);
612
613 - HAProxy itself can report its state to external components such as routers
614 or other load balancers, allowing to build very complete multi-path and
615 multi-layer infrastructures.
616
617
6183.3.4. Basic features : High availability
619-----------------------------------------
620
621Just like any serious load balancer, HAProxy cares a lot about availability to
622ensure the best global service continuity :
623
624 - Only valid servers are used ; the other ones are automatically evinced from
625 load balancing farms ; under certain conditions it is still possible to
626 force to use them though;
627
628 - Support for a graceful shutdown so that it is possible to take servers out
629 of a farm without affecting any connection;
630
631 - Backup servers are automatically used when active servers are down and
632 replace them so that sessions are not lost when possible. This also allows
633 to build multiple paths to reach the same server (eg: multiple interfaces);
634
635 - Ability to return a global failed status for a farm when too many servers
636 are down. This, combined with the monitoring capabilities makes it possible
637 for an upstream component to choose a different LB node for a given service;
638
639 - Stateless design makes it easy to build clusters : by design, HAProxy does
640 its best to ensure the highest service continuity without having to store
641 information that could be lost in the event of a failure. This ensures that
642 a takeover is the most seamless possible;
643
644 - Integrates well with standard VRRP daemon keepalived : HAProxy easily tells
645 keepalived about its state and copes very will with floating virtual IP
646 addresses. Note: only use IP redundancy protocols (VRRP/CARP) over cluster-
647 based solutions (Heartbeat, ...) as they're the ones offering the fastest,
648 most seamless, and most reliable switchover.
649
650
6513.3.5. Basic features : Load balancing
652--------------------------------------
653
654HAProxy offers a fairly complete set of load balancing features, most of which
655are unfortunately not available in a number of other load balancing products :
656
657 - no less than 9 load balancing algorithms are supported, some of which apply
658 to input data to offer an infinite list of possibilities. The most common
659 ones are round-robin (for short connections, pick each server in turn),
660 leastconn (for long connections, pick the least recently used of the servers
661 with the lowest connection count), source (for SSL farms or terminal server
662 farms, the server directly depends on the client's source address), uri (for
663 HTTP caches, the server directly depends on the HTTP URI), hdr (the server
664 directly depends on the contents of a specific HTTP header field), first
665 (for short-lived virtual machines, all connections are packed on the
666 smallest possible subset of servers so that unused ones can be powered
667 down);
668
669 - all algorithms above support per-server weights so that it is possible to
670 accommodate from different server generations in a farm, or direct a small
671 fraction of the traffic to specific servers (debug mode, running the next
672 version of the software, etc);
673
674 - dynamic weights are supported for round-robin, leastconn and consistent
675 hashing ; this allows server weights to be modified on the fly from the CLI
676 or even by an agent running on the server;
677
678 - slow-start is supported whenever a dynamic weight is supported; this allows
679 a server to progressively take the traffic. This is an important feature
680 for fragile application servers which require to compile classes at runtime
681 as well as cold caches which need to fill up before being run at full
682 throttle;
683
684 - hashing can apply to various elements such as client's source address, URL
685 components, query string element, header field values, POST parameter, RDP
686 cookie;
687
688 - consistent hashing protects server farms against massive redistribution when
689 adding or removing servers in a farm. That's very important in large cache
690 farms and it allows slow-start to be used to refill cold caches;
691
692 - a number of internal metrics such as the number of connections per server,
693 per backend, the amount of available connection slots in a backend etc makes
694 it possible to build very advanced load balancing strategies.
695
696
6973.3.6. Basic features : Stickiness
698----------------------------------
699
700Application load balancing would be useless without stickiness. HAProxy provides
701a fairly comprehensive set of possibilities to maintain a visitor on the same
702server even across various events such as server addition/removal, down/up
703cycles, and some methods are designed to be resistant to the distance between
704multiple load balancing nodes in that they don't require any replication :
705
706 - stickiness information can be individually matched and learned from
707 different places if desired. For example a JSESSIONID cookie may be matched
708 both in a cookie and in the URL. Up to 8 parallel sources can be learned at
709 the same time and each of them may point to a different stick-table;
710
711 - stickiness information can come from anything that can be seen within a
712 request or response, including source address, TCP payload offset and
713 length, HTTTP query string elements, header field values, cookies, and so
714 on...
715
716 - stick-tables are replicated between all nodes in a multi-master fashion ;
717
718 - commonly used elements such as SSL-ID or RDP cookies (for TSE farms) are
719 directly accessible to ease manipulation;
720
721 - all sticking rules may be dynamically conditionned by ACLs;
722
723 - it is possible to decide not to stick to certain servers, such as backup
724 servers, so that when the nominal server comes back, it automatically takes
725 the load back. This is often used in multi-path environments;
726
727 - in HTTP it is often prefered not to learn anything and instead manipulate
728 a cookie dedicated to stickiness. For this, it's possible to detect,
729 rewrite, insert or prefix such a cookie to let the client remember what
730 server was assigned;
731
732 - the server may decide to change or clean the stickiness cookie on logout,
733 so that leaving visitors are automatically unbound from the server;
734
735 - using ACL-based rules it is also possible to selectively ignore or enforce
736 stickiness regardless of the server's state; combined with advanced health
737 checks, that helps admins verify that the server they're installing is up
738 and running before presenting it to the whole world;
739
740 - an innovative mechanism to set a maximum idle time and duration on cookies
741 ensures that stickiness can be smoothly stopped on devices which are never
742 closed (smartphones, TVs, home appliances) without having to store them on
743 persistent storage;
744
745 - multiple server entries may share the same stickiness keys so that
746 stickiness is not lost in multi-path environments when one path goes down;
747
748 - soft-stop ensures that only users with stickiness information will continue
749 to reach the server they've been assigned to but no new users will go there.
750
751
7523.3.7. Basic features : Sampling and converting information
753-----------------------------------------------------------
754
755HAProxy supports information sampling using a wide set of "sample fetch
756functions". The principle is to extract pieces of information known as samples,
757for immediate use. This is used for stickiness, to build conditions, to produce
758information in logs or to enrich HTTP headers.
759
760Samples can be fetched from various sources :
761
762 - constants : integers, strings, IP addresses, binary blocks;
763
764 - the process : date, environment variables, server/frontend/backend/process
765 state, byte/connection counts/rates, queue length, random generator, ...
766
767 - variables : per-session, per-request, per-response variables;
768
769 - the client connection : source and destination addresses and ports, and all
770 related statistics counters;
771
772 - the SSL client session : protocol, version, algorithm, cipher, key size,
773 session ID, all client and server certificate fields, certificate serial,
774 SNI, ALPN, NPN, client support for certain extensions;
775
776 - request and response buffers contents : arbitrary payload at offset/length,
777 data length, RDP cookie, decoding of SSL hello type, decoding of TLS SNI;
778
779 - HTTP (request and response) : method, URI, path, query string arguments,
780 status code, headers values, positionnal header value, cookies, captures,
781 authentication, body elements;
782
783A sample may then pass through a number of operators known as "converters" to
784experience some transformation. A converter consumes a sample and produces a
785new one, possibly of a completely different type. For example, a converter may
786be used to return only the integer length of the input string, or could turn a
787string to upper case. Any arbitrary number of converters may be applied in
788series to a sample before final use. Among all available sample converters, the
789following ones are the most commonly used :
790
791 - arithmetic and logic operators : they make it possible to perform advanced
792 computation on input data, such as computing ratios, percentages or simply
793 converting from one unit to another one;
794
795 - IP address masks are useful when some addresses need to be grouped by larger
796 networks;
797
798 - data representation : url-decode, base64, hex, JSON strings, hashing;
799
800 - string conversion : extract substrings at fixed positions, fixed length,
801 extract specific fields around certain delimiters, extract certain words,
802 change case, apply regex-based substitution ;
803
804 - date conversion : convert to http date format, convert local to UTC and
805 conversely, add or remove offset;
806
807 - lookup an entry in a stick table to find statistics or assigned server;
808
809 - map-based key-to-value conversion from a file (mostly used for geolocation).
810
811
8123.3.8. Basic features : Maps
813----------------------------
814
815Maps are a powerful type of converter consisting in loading a two-columns file
816into memory at boot time, then looking up each input sample from the first
817column and either returning the corresponding pattern on the second column if
818the entry was found, or returning a default value. The output information also
819being a sample, it can in turn experience other transformations including other
820map lookups. Maps are most commonly used to translate the client's IP address
821to an AS number or country code since they support a longest match for network
822addresses but they can be used for various other purposes.
823
824Part of their strength comes from being updatable on the fly either from the CLI
825or from certain actions using other samples, making them capable of storing and
826retrieving information between subsequent accesses. Another strength comes from
827the binary tree based indexation which makes them extremely fast event when they
828contain hundreds of thousands of entries, making geolocation very cheap and easy
829to set up.
830
831
8323.3.9. Basic features : ACLs and conditions
833-------------------------------------------
834
835Most operations in HAProxy can be made conditional. Conditions are built by
836combining multiple ACLs using logic operators (AND, OR, NOT). Each ACL is a
837series of tests based on the following elements :
838
839 - a sample fetch method to retrieve the element to test ;
840
841 - an optional series of converters to transform the element ;
842
843 - a list of patterns to match against ;
844
845 - a matching method to indicate how to compare the patterns with the sample
846
847For example, the sample may be taken from the HTTP "Host" header, it could then
848be converted to lower case, then matched against a number of regex patterns
849using the regex matching method.
850
851Technically, ACLs are built on the same core as the maps, they share the exact
852same internal structure, pattern matching methods and performance. The only real
853difference is that instead of returning a sample, they only return "found" or
854or "not found". In terms of usage, ACL patterns may be declared inline in the
855configuration file and do not require their own file. ACLs may be named for ease
856of use or to make configurations understandable. A named ACL may be declared
857multiple times and it will evaluate all definitions in turn until one matches.
858
859About 13 different pattern matching methods are provided, among which IP address
860mask, integer ranges, substrings, regex. They work like functions, and just like
861with any programming language, only what is needed is evaluated, so when a
862condition involving an OR is already true, next ones are not evaluated, and
863similarly when a condition involving an AND is already false, the rest of the
864condition is not evaluated.
865
866There is no practical limit to the number of declared ACLs, and a handful of
867commonly used ones are provided. However experience has shown that setups using
868a lot of named ACLs are quite hard to troubleshoot and that sometimes using
869anynmous ACLs inline is easier as it requires less references out of the scope
870being analysed.
871
872
8733.3.10. Basic features : Content switching
874------------------------------------------
875
876HAProxy implements a mechanism known as content-based switching. The principle
877is that a connection or request arrives on a frontend, then the information
878carried with this request or connection are processed, and at this point it is
879possible to write ACLs-based conditions making use of these information to
880decide what backend will process the request. Thus the traffic is directed to
881one backend or another based on the request's contents. The most common example
882consists in using the Host header and/or elements from the path (sub-directories
883or file-name extensions) to decide whether an HTTP request targets a static
884object or the application, and to route static objects traffic to a backend made
885of fast and light servers, and all the remaining traffic to a more complex
886application server, thus constituting a fine-grained virtual hosting solution.
887This is quite convenient to make multiple technologies coexist as a more global
888solution.
889
890Another use case of content-switching consists in using different load balancing
891algorithms depending on various criteria. A cache may use a URI hash while an
892application would use round robin.
893
894Last but not least, it allows multiple customers to use a small share of a
895common resource by enforcing per-backend (thus per-customer connection limits).
896
897Content switching rules scale very well, though their performance may depend on
898the number and complexity of the ACLs in use. But it is also possible to write
899dynamic content switching rules where a sample value directly turns into a
900backend name and without making use of ACLs at all. Such configurations have
901been reported to work fine at least with 300000 backends in production.
902
903
9043.3.11. Basic features : Stick-tables
905-------------------------------------
906
907Stick-tables are commonly used to store stickiness information, that is, to keep
908a reference to the server a certain visitor was directed to. The key is then the
909identifier associated with the visitor (its source address, the SSL ID of the
910connection, an HTTP or RDP cookie, the customer number extracted from the URL or
911from the payload, ...) and the stored value is then the server's identifier.
912
913Stick tables may use 3 different types of samples for their keys : integers,
914strings and addresses. Only one stick-table may be referenced in a proxy, and it
915is designated everywhere with the proxy name. Up to 8 key may be tracked in
916parallel. The server identifier is committed during request or response
917processing once both the key and the server are known.
918
919Stick-table contents may be replicated in active-active mode with other HAProxy
920nodes known as "peers" as well as with the new process during a reload operation
921so that all load balancing nodes share the same information and take the same
922routing decision if a client's requests are spread over multiple nodes.
923
924Since stick-tables are indexed on what allows to recognize a client, they are
925often also used to store extra information such as per-client statistics. The
926extra statistics take some extra space and need to be explicitly declared. The
927type of statistics that may be stored includes the input and output bandwidth,
928the number of concurrent connections, the connection rate and count over a
929period, the amount and frequency of errors, some specific tags and counters,
930etc... In order to support keeping such information without being forced to
931stick to a given server, a special "tracking" feature is implemented and allows
932to track up to 3 simultaneous keys from different tables at the same time
933regardless of stickiness rules. Each stored statistics may be searched, dumped
934and cleared from the CLI and adds to the live troubleshooting capabilities.
935
936While this mechanism can be used to surclass a returning visitor or to adjust
937the delivered quality of service depending on good or bad behaviour, it is
938mostly used to fight against service abuse and more generally DDoS as it allows
939to build complex models to detect certain bad behaviours at a high processing
940speed.
941
942
9433.3.12. Basic features : Formated strings
944-----------------------------------------
945
946There are many places where HAProxy needs to manipulate character strings, such
947as logs, redirects, header additions, and so on. In order to provide the
948greatest flexibility, the notion of formated strings was introduced, initially
949for logging purposes, which explains why it's still called "log-format". These
950strings contain escape characters allowing to introduce various dynamic data
951including variables and sample fetch expressions into strings, and even to
952adjust the encoding while the result is being turned into a string (for example,
953adding quotes). This provides a powerful way to build header contents or to
954customize log lines. Additionally, in order to remain simple to build most
955common strings, about 50 special tags are provided as shortcuts for information
956commonly used in logs.
957
958
9593.3.13. Basic features : HTTP rewriting and redirection
960-------------------------------------------------------
961
962Installing a load balancer in front of an application that was never designed
963for this can be a challenging task without the proper tools. One of the most
964commonly requested operation in this case is to adjust requests and response
965headers to make the load balancer appear as the origin server and to fix hard
966coded information. This comes with changing the path in requests (which is
967strongly advised against), modifying Host header field, modifying the Location
968response header field for redirects, modifying the path and domain attribute
969for cookies, and so on. It also happens that a number of servers are somewhat
970verbose and tend to leak too much information in the response, making them more
971vulnerable to targetted attacks. While it's theorically not the role of a load
972balancer to clean this up, in practice it's located at the best place in the
973infrastructure to guarantee that everything is cleaned up.
974
975Similarly, sometimes the load balancer will have to intercept some requests and
976respond with a redirect to a new target URL. While some people tend to confuse
977redirects and rewriting, these are two completely different concepts, since the
978rewriting makes the client and the server see different things (and disagree on
979the location of the page being visited) while redirects ask the client to visit
980the new URL so that it sees the same location as the server.
981
982In order to do this, HAProxy supports various possibilities for rewriting and
983redirect, among which :
984
985 - regex-based URL and header rewriting in requests and responses. Regex are
986 the most commonly used tool to modify header values since they're easy to
987 manipulate and well understood;
988
989 - headers may also be appended, deleted or replaced based on formated strings
990 so that it is possible to pass information there (eg: client side TLS
991 algorithm and cipher);
992
993 - HTTP redirects can use any 3xx code to a relative, absolute, or completely
994 dynamic (formated string) URI;
995
996 - HTTP redirects also support some extra options such as setting or clearing
997 a specific cookie, dropping the query string, appending a slash if missing,
998 and so on;
999
1000 - all operations support ACL-based conditions;
1001
1002
10033.3.14. Basic features : Server protection
1004------------------------------------------
1005
1006HAProxy does a lot to maximize service availability, and for this it deploys
1007large efforts to protect servers against overloading and attacks. The first
1008and most important point is that only complete and valid requests are forwarded
1009to the servers. The initial reason is that HAProxy needs to find the protocol
1010elements it needs to stay synchronized with the byte stream, and the second
1011reason is that until the request is complete, there is no way to know if some
1012elements will change its semantics. The direct benefit from this is that servers
1013are not exposed to invalid or incomplete requests. This is a very effective
1014protection against slowloris attacks, which have almost no impact on HAProxy.
1015
1016Another important point is that HAProxy contains buffers to store requests and
1017responses, and that by only sending a request to a server when it's complete and
1018by reading the whole response very quickly from the local network, the server
1019side connection is used for a very short time and this preserves server
1020resources as much as possible.
1021
1022A direct extension to this is that HAProxy can artificially limit the number of
1023concurrent connections or outstanding requests to a server, which guarantees
1024that the server will never be overloaded even if it continuously runs at 100% of
1025its capacity during traffic spikes. All excess requests will simply be queued to
1026be processed when one slot is released. In the end, this huge resource savings
1027most often ensures so much better server response times that it ends up actually
1028being faster than by overloading the server. Queued requests may be redispatched
1029to other servers, or even aborted in queue when the client aborts, which also
1030protects the servers against the "reload effect", where each click on "reload"
1031by a visitor on a slow-loading page usually induces a new request and maintains
1032the server in an overloaded state.
1033
1034The slow-start mechanism also protects restarting servers against high traffic
1035levels while they're still finalizing their startup or compiling some classes.
1036
1037Regarding the protocol-level protection, it is possible to relax the HTTP parser
1038to accept non stardard-compliant but harmless requests or responses and even to
1039fix them. This allows bogus applications to be accessible while a fix is being
1040developped. In parallel, offending messages are completely captured with a
1041detailed report that help developers spot the issue in the application. The most
1042dangerous protocol violations are properly detected and dealt with and fixed.
1043For example malformed requests or responses with two Content-length headers are
1044either fixed if the values are exactly the same, or rejected if they differ,
1045since it becomes a security problem. Protocol inspection is not limited to HTTP,
1046it is also available for other protocols like TLS or RDP.
1047
1048When a protocol violation or attack is detected, there are various options to
1049respond to the user, such as returning the common "HTTP 400 bad request",
1050closing the connection with a TCP reset, faking an error after a long delay
1051("tarpit") to confuse the attacker. All of these contribute to protecting the
1052servers by discouraging the offending client from pursuing an attack that
1053becomes very expensive to maintain.
1054
1055HAProxy also proposes some more advanced options to protect against accidental
1056data leaks and session crossing. Not only it can log suspicious server responses
1057but it will also log and optionally block a response which might affect a given
1058visitors' confidentiality. One such example is a cacheable cookie appearing in a
1059cacheable response and which may result in an intermediary cache to deliver it
1060to another visitor, causing an accidental session sharing.
1061
1062
10633.3.15. Basic features : Logging
1064--------------------------------
1065
1066Logging is an extremely important feature for a load balancer, first because a
1067load balancer is often accused of the trouble it reveals, and second because it
1068is placed at a critical point in an infrastructure where all normal and abnormal
1069activity needs to be analysed and correlated with other components.
1070
1071HAProxy provides very detailed logs, with millisecond accuracy and the exact
1072connection accept time that can be searched in firewalls logs (eg: for NAT
1073correlation). By default, TCP and HTTP logs are quite detailed an contain
1074everything needed for troubleshooting, such as source IP address and port,
1075frontend, backend, server, timers (request receipt duration, queue duration,
1076connection setup time, response headers time, data transfer time), global
1077process state, connection counts, queue status, retries count, detailed
1078stickiness actions and disconnect reasons, header captures with a safe output
1079encoding. It is then possible to extend or replace this format to include any
1080sampled data, variables, captures, resulting in very detailed information. For
1081example it is possible to log the number cumulated requests for this client or
1082the number of different URLs for the client.
1083
1084The log level may be adjusted per request using standard ACLs, so it is possible
1085to automatically silent some logs considered as pollution and instead raise
1086warnings when some abnormal behaviour happen for a small part of the traffic
1087(eg: too many URLs or HTTP errors for a source address). Administrative logs are
1088also emitted with their own levels to inform about the loss or recovery of a
1089server for example.
1090
1091Each frontend and backend may use multiple independant log outputs, which eases
1092multi-tenancy. Logs are preferably sent over UDP, maybe JSON-encoded, and are
1093truncated after a configurable line length in order to guarantee delivery.
1094
1095
10963.3.16. Basic features : Statistics
1097-----------------------------------
1098
1099HAProxy provides a web-based statistics reporting interface with authentication,
1100security levels and scopes. It is thus possible to provide each hosted customer
1101with his own page showing only his own instances. This page can be located in a
1102hidden URL part of the regular web site so that no new port needs to be opened.
1103This page may also report the availability of other HAProxy nodes so that it is
1104easy to spot if everything works as expected at a glance. The view is synthetic
1105with a lot of details accessible (such as error causes, last access and last
1106change duration, etc), which are also accessible as a CSV table that other tools
1107may import to draw graphs. The page may self-refresh to be used as a monitoring
1108page on a large display. In administration mode, the page also allows to change
1109server state to ease maintenance operations.
1110
1111
11123.4. Advanced features
1113----------------------
1114
11153.4.1. Advanced features : Management
1116-------------------------------------
1117
1118HAProxy is designed to remain extremely stable and safe to manage in a regular
1119production environment. It is provided as a single executable file which doesn't
1120require any installation process. Multiple versions can easily coexist, meaning
1121that it's possible (and recommended) to upgrade instances progressively by
1122order of criticity instead of migrating all of them at once. Configuration files
1123are easily versionned. Configuration checking is done off-line so it doesn't
1124require to restart a service that will possibly fail. During configuration
1125checks, a number of advanced mistakes may be detected (eg: for example, a rule
1126hiding another one, or stickiness that will not work) and detailed warnings and
1127configuration hints are proposed to fix them. Backwards configuration file
1128compatibility goes very far away in time, with version 1.5 still fully
1129supporting configurations for versions 1.1 written 13 years before, and 1.6
1130only dropping support for almost unused, obsolete keywords that can be done
1131differently. The configuration and software upgrade mechanism is smooth and non
1132disruptive in that it allows old and new processes to coexist on the system,
1133each handling its own connections. System status, build options and library
1134compatibility are reported on startup.
1135
1136Some advanced features allow an application administrator to smoothly stop a
1137server, detect when there's no activity on it anymore, then take it off-line,
1138stop it, upgrade it and ensure it doesn't take any traffic while being upgraded,
1139then test it again through the normal path without opening it to the public, and
1140all of this without touching HAProxy at all. This ensures that even complicated
1141production operations may be done during opening hours with all technical
1142resources available.
1143
1144The process tries to save resources as much as possible, uses memory pools to
1145save on allocation time and limit memory fragmentation, releases payload buffers
1146as soon as their contents are sent, and supports enforcing strong memory limits
1147above which connections have to wait for a buffer to become available instead of
1148allocating more memory. This system helps guarantee memory usage in certain
1149strict environments.
1150
1151A command line interface (CLI) is available as a UNIX or TCP socket, to perform
1152a number of operations and to retrieve troubleshooting information. Everything
1153done on this socket doesn't require a configuration change, so it is mostly used
1154for temporary changes. Using this interface it is possible to change a server's
1155address, weight and status, to consult statistics and clear counters, dump and
1156clear stickiness tables, possibly selectively by key criteria, dump and kill
1157client-side and server-side connections, dump captured errors with a detailed
1158analysis of the exact cause and location of the error, dump, add and remove
1159entries from ACLs and maps, update TLS shared secrets, apply connection limits
1160and rate limits on the fly to arbitrary frontends (useful in shared hosting
1161environments), and disable a specific frontend to release a listening port
1162(useful when daytime operations are forbidden and a fix is needed nonetheless).
1163
1164For environments where SNMP is mandatory, at least two agents exist, one is
1165provided with the HAProxy sources and relies on the Net-SNMP perl module.
1166Another one is provided with the commercial packages and doesn't require Perl.
1167Both are roughly equivalent in terms of coverage.
1168
1169It is often recommended to install 4 utilities on the machine where HAProxy is
1170deployed :
1171
1172 - socat (in order to connect to the CLI, though certain forks of netcat can
1173 also do it to some extents);
1174
1175 - halog from the latest HAProxy version : this is the log analysis tool, it
1176 parses native TCP and HTTP logs extremely fast (1 to 2 GB per second) and
1177 extracts useful information and statistics such as requests per URL, per
1178 source address, URLs sorted by response time or error rate, termination
1179 codes etc... It was designed to be deployed on the production servers to
1180 help troubleshoot live issues so it has to be there ready to be used;
1181
1182 - tcpdump : this is highly recommended to take the network traces needed to
1183 troubleshoot an issue that was made visible in the logs. There is a moment
1184 where application and haproxy's analysis will diverge and the network traces
1185 are the only way to say who's right and who's wrong. It's also fairly common
1186 to detect bugs in network stacks and hypervisors thanks to tcpdump;
1187
1188 - strace : it is tcpdump's companion. It will report what HAProxy really sees
1189 and will help sort out the issues the operating system is responsible for
1190 from the ones HAProxy is responsible for. Strace is often requested when a
1191 bug in HAProxy is suspected;
1192
1193
11943.4.2. Advanced features : System-specific capabilities
1195-------------------------------------------------------
1196
1197Depending on the operating system HAProxy is deployed on, certain extra features
1198may be available or needed. While it is supported on a number of platforms,
1199HAProxy is primarily developped on Linux, which explains why some features are
1200only available on this platform.
1201
1202The transparent bind and connect features, the support for binding connections
1203to a specific network interface, as well as the ability to bind multiple
1204processes to the same IP address and ports are only available on Linux and BSD
1205systems, though only Linux performs a kernel-side load balancing of the incoming
1206requests between the available processes.
1207
1208On Linux, there are also a number of extra features and optimizations including
1209support for network namespaces (also known as "containers") allowing HAProxy to
1210be a gateway between all containers, the ability to set the MSS, Netfilter marks
1211and IP TOS field on the client side connection, support for TCP FastOpen on the
1212listening side, TCP user timeouts to let the kernel quickly kill connections
1213when it detects the client has disappeared before the configured timeouts, TCP
1214splicing to let the kernel forward data between the two sides of a connections
1215thus avoiding multiple memory copies, the ability to enable the "defer-accept"
1216bind option to only get notified of an incoming connection once data become
1217available in the kernel buffers, and the ability to send the request with the
1218ACK confirming a connect (sometimes called "biggy-back") which is enabled with
1219the "tcp-smart-connect" option. On Linux, HAProxy also takes great care of
1220manipulating the TCP delayed ACKs to save as many packets as possible on the
1221network.
1222
1223Some systems have an unreliable clock which jumps back and forth in the past
1224and in the future. This used to happen with some NUMA systems where multiple
1225processors didn't see the exact same time of day, and recently it became more
1226common in virtualized environments where the virtual clock has no relation with
1227the real clock, resulting in huge time jumps (sometimes up to 30 seconds have
1228been observed). This causes a lot of trouble with respect to timeout enforcement
1229in general. Due to this flaw of these systems, HAProxy maintains its own
1230monotonic clock which is based on the system's clock but where drift is measured
1231and compensated for. This ensures that even with a very bad system clock, timers
1232remain reasonably accurate and timeouts continue to work. Note that this problem
1233affects all the software running on such systems and is not specific to HAProxy.
1234The common effects are spurious timeouts or application freezes. Thus if this
1235behaviour is detected on a system, it must be fixed, regardless of the fact that
1236HAProxy protects itself against it.
1237
1238
12393.4.3. Advanced features : Scripting
1240------------------------------------
1241
1242HAProxy can be built with support for the Lua embedded language, which opens a
1243wide area of new possibilities related to complex manipulation of requests or
1244responses, routing decisions, statistics processing and so on. Using Lua it is
1245even possible to establish parallel connections to other servers to exchange
1246information. This way it becomes possible (though complex) to develop an
1247authentication system for example. Please refer to the documentation in the file
1248"doc/lua-api/index.rst" for more information on how to use Lua.
1249
1250
12513.5. Sizing
1252-----------
1253
1254Typical CPU usage figures show 15% of the processing time spent in HAProxy
1255versus 85% in the kernel in TCP or HTTP close mode, and about 30% for HAProxy
1256versus 70% for the kernel in HTTP keep-alive mode. This means that the operating
1257system and its tuning have a strong impact on the global performance.
1258
1259Usages vary a lot between users, some focus on bandwidth, other ones on request
1260rate, others on connection concurrency, others on SSL performance. this section
1261aims at providing a few elements to help in this task.
1262
1263It is important to keep in mind that every operation comes with a cost, so each
1264individual operation adds its overhead on top of the other ones, which may be
1265negligible in certain circumstances, and which may dominate in other cases.
1266
1267When processing the requests from a connection, we can say that :
1268
1269 - forwarding data costs less than parsing request or response headers;
1270
1271 - parsing request or response headers cost less than establishing then closing
1272 a connection to a server;
1273
1274 - establishing an closing a connection costs less than a TLS resume operation;
1275
1276 - a TLS resume operation costs less than a full TLS handshake with a key
1277 computation;
1278
1279 - an idle connection costs less CPU than a connection whose buffers hold data;
1280
1281 - a TLS context costs even more memory than a connection with data;
1282
1283So in practice, it is cheaper to process payload bytes than header bytes, thus
1284it is easier to achieve high network bandwidth with large objects (few requests
1285per volume unit) than with small objects (many requests per volume unit). This
1286explains why maximum bandwidth is always measured with large objects, while
1287request rate or connection rates are measured with small objects.
1288
1289Some operations scale well on multiple process spread over multiple processors,
1290and others don't scale as well. Network bandwidth doesn't scale very far because
1291the CPU is rarely the bottleneck for large objects, it's mostly the network
1292bandwidth and data busses to reach the network interfaces. The connection rate
1293doesn't scale well over multiple processors due to a few locks in the system
1294when dealing with the local ports table. The request rate over persistent
1295connections scales very well as it doesn't involve much memory nor network
1296bandwidth and doesn't require to access locked structures. TLS key computation
1297scales very well as it's totally CPU-bound. TLS resume scales moderately well,
1298but reaches its limits around 4 processes where the overhead of accessing the
1299shared table offsets the small gains expected from more power.
1300
1301The performance numbers one can expect from a very well tuned system are in the
1302following range. It is important to take them as orders of magnitude and to
1303expect significant variations in any direction based on the processor, IRQ
1304setting, memory type, network interface type, operating system tuning and so on.
1305
1306The following numbers were found on a Core i7 running at 3.7 GHz equiped with
1307a dual-port 10 Gbps NICs running Linux kernel 3.10, HAProxy 1.6 and OpenSSL
13081.0.2. HAProxy was running as a single process on a single dedicated CPU core,
1309and two extra cores were dedicated to network interrupts :
1310
1311 - 20 Gbps of maximum network bandwidth in clear text for objects 256 kB or
1312 higher, 10 Gbps for 41kB or higher;
1313
1314 - 4.6 Gbps of TLS traffic using AES256-GCM cipher with large objects;
1315
1316 - 83000 TCP connections per second from client to server;
1317
1318 - 82000 HTTP connections per second from client to server;
1319
1320 - 97000 HTTP requests per second in server-close mode (keep-alive with the
1321 client, close with the server);
1322
1323 - 243000 HTTP requests per second in end-to-end keep-alive mode;
1324
1325 - 300000 filtered TCP connections per second (anti-DDoS)
1326
1327 - 160000 HTTPS requests per second in keep-alive mode over persistent TLS
1328 connections;
1329
1330 - 13100 HTTPS requests per second using TLS resumed connections;
1331
1332 - 1300 HTTPS connections per second using TLS connections renegociated with
1333 RSA2048;
1334
1335 - 20000 concurrent saturated connections per GB of RAM, including the memory
1336 required for system buffers; it is possible to do better with careful tuning
1337 but this setting it easy to achieve.
1338
1339 - about 8000 concurrent TLS connections (client-side only) per GB of RAM,
1340 including the memory required for system buffers;
1341
1342 - about 5000 concurrent end-to-end TLS connections (both sides) per GB of
1343 RAM including the memory required for system buffers;
1344
1345Thus a good rule of thumb to keep in mind is that the request rate is divided
1346by 10 between TLS keep-alive and TLS resume, and between TLS resume and TLS
1347renegociation, while it's only divided by 3 between HTTP keep-alive and HTTP
1348close. Another good rule of thumb is to remember that a high frequency core
1349with AES instructions can do around 5 Gbps of AES-GCM per core.
1350
1351Having more core rarely helps (except for TLS) and is even counter-productive
1352due to the lower frequency. In general a small number of high frequency cores
1353is better.
1354
1355Another good rule of thumb is to consider that on the same server, HAProxy will
1356be able to saturate :
1357
1358 - about 5-10 static file servers or caching proxies;
1359
1360 - about 100 anti-virus proxies;
1361
Willy Tarreau16af23c2015-08-27 16:30:53 +02001362 - and about 100-1000 application servers depending on the technology in use.
Willy Tarreaud8e42b62015-08-18 21:51:36 +02001363
1364
13653.6. How to get HAProxy
1366-----------------------
1367
1368HAProxy is an opensource project covered by the GPLv2 license, meaning that
1369everyone is allowed to redistribute it provided that access to the sources is
1370also provided upon request, especially if any modifications were made.
1371
1372HAProxy evolves as a main development branch called "master" or "mainline", from
1373which new branches are derived once the code is considered stable. A lot of web
1374sites run some development branches in production on a voluntarily basis, either
1375to participate to the project or because they need a bleeding edge feature, and
1376their feedback is highly valuable to fix bugs and judge the overall quality and
1377stability of the version being developped.
1378
1379The new branches that are created when the code is stable enough constitute a
1380stable version and are generally maintained for several years, so that there is
1381no emergency to migrate to a newer branch even when you're not on the latest.
1382Once a stable branch is issued, it may only receive bug fixes, and very rarely
1383minor feature updates when that makes users' life easier. All fixes that go into
1384a stable branch necessarily come from the master branch. This guarantees that no
1385fix will be lost after an upgrade. For this reason, if you fix a bug, please
1386make the patch against the master branch, not the stable branch. You may even
1387discover it was already fixed. This process also ensures that regressions in a
1388stable branch are extremely rare, so there is never any excuse for not upgrading
1389to the latest version in your current branch.
1390
1391Branches are numberred with two digits delimited with a dot, such as "1.6". A
1392complete version includes one or two sub-version numbers indicating the level of
1393fix. For example, version 1.5.14 is the 14th fix release in branch 1.5 after
1394version 1.5.0 was issued. It contains 126 fixes for individual bugs, 24 updates
1395on the documentation, and 75 other backported patches, most of which were needed
1396to fix the aforementionned 126 bugs. An existing feature may never be modified
1397nor removed in a stable branch, in order to guarantee that upgrades within the
1398same branch will always be harmless.
1399
1400HAProxy is available from multiple sources, at different release rhythms :
1401
1402 - The official community web site : http://www.haproxy.org/ : this site
1403 provides the sources of the latest development release, all stable releases,
1404 as well as nightly snapshots for each branch. The release cycle is not fast,
1405 several months between stable releases, or between development snapshots.
1406 Very old versions are still supported there. Everything is provided as
1407 sources only, so whatever comes from there needs to be rebuilt and/or
1408 repackaged;
1409
1410 - A number of operating systems such as Linux distributions and BSD ports.
1411 These systems generally provide long-term maintained versions which do not
1412 always contain all the fixes from the official ones, but which at least
1413 contain the critical fixes. It often is a good option for most users who do
1414 not seek advanced configurations and just want to keep updates easy;
1415
1416 - Commercial versions from http://www.haproxy.com/ : these are supported
1417 professional packages built for various operating systems or provided as
1418 appliances, based on the latest stable versions and including a number of
1419 features backported from the next release for which there is a strong
1420 demand. It is the best option for users seeking the latest features with
1421 the reliability of a stable branch, the fastest response time to fix bugs,
1422 or simply support contracts on top of an opensource product;
1423
1424
1425In order to ensure that the version you're using is the latest one in your
1426branch, you need to proceed this way :
1427
1428 - verify which HAProxy executable you're running : some systems ship it by
1429 default and administrators install their versions somewhere else on the
1430 system, so it is important to verify in the startup scripts which one is
1431 used;
1432
1433 - determine which source your HAProxy version comes from. For this, it's
1434 generally sufficient to type "haproxy -v". A development version will
1435 appear like this, with the "dev" word after the branch number :
1436
1437 HA-Proxy version 1.6-dev3-385ecc-68 2015/08/18
1438
1439 A stable version will appear like this, as well as unmodified stable
1440 versions provided by operating system vendors :
1441
1442 HA-Proxy version 1.5.14 2015/07/02
1443
1444 And a nightly snapshot of a stable version will appear like this with an
1445 hexadecimal sequence after the version, and with the date of the snapshot
1446 instead of the date of the release :
1447
1448 HA-Proxy version 1.5.14-e4766ba 2015/07/29
1449
1450 Any other format may indicate a system-specific package with its own
1451 patch set. For example HAProxy Enterprise versions will appear with the
1452 following format (<branch>-<latest commit>-<revision>) :
1453
1454 HA-Proxy version 1.5.0-994126-357 2015/07/02
1455
1456 - for system-specific packages, you have to check with your vendor's package
1457 repository or update system to ensure that your system is still supported,
1458 and that fixes are still provided for your branch. For community versions
1459 coming from haproxy.org, just visit the site, verify the status of your
1460 branch and compare the latest version with yours to see if you're on the
1461 latest one. If not you can upgrade. If your branch is not maintained
1462 anymore, you're definitely very late and will have to consider an upgrade
1463 to a more recent branch (carefully read the README when doing so).
1464
1465HAProxy will have to be updated according to the source it came from. Usually it
1466follows the system vendor's way of upgrading a package. If it was taken from
1467sources, please read the README file in the sources directory after extracting
1468the sources and follow the instructions for your operating system.
1469
1470
14714. Companion products and alternatives
1472--------------------------------------
1473
1474HAProxy integrates fairly well with certain products listed below, which is why
1475they are mentionned here even if not directly related to HAProxy.
1476
1477
14784.1. Apache HTTP server
1479-----------------------
1480
1481Apache is the de-facto standard HTTP server. It's a very complete and modular
1482project supporting both file serving and dynamic contents. It can serve as a
1483frontend for some application servers. In can even proxy requests and cache
1484responses. In all of these use cases, a front load balancer is commonly needed.
1485Apache can work in various modes, certain being heavier than other ones. Certain
1486modules still require the heavier pre-forked model and will prevent Apache from
1487scaling well with a high number of connections. In this case HAProxy can provide
1488a tremendous help by enforcing the per-server connection limits to a safe value
1489and will significantly speed up the server and preserve its resources that will
1490be better used by the application.
1491
1492Apache can extract the client's address from the X-Forwarded-For header by using
1493the "mod_rpaf" extension. HAProxy will automatically feed this header when
1494"option forwardfor" is specified in its configuration. HAProxy may also offer a
1495nice protection to Apache when exposed to the internet, where it will better
1496resist to a wide number of types of DoS.
1497
1498
14994.2. NGINX
1500----------
1501
1502NGINX is the second de-facto standard HTTP server. Just like Apache, it covers a
1503wide range of features. NGINX is built on a similar model as HAProxy so it has
1504no problem dealing with tens of thousands of concurrent connections. When used
1505as a gateway to some applications (eg: using the included PHP FPM), it can often
1506be beneficial to set up some frontend connection limiting to reduce the load
1507on the PHP application. HAProxy will clearly be useful there both as a regular
1508load balancer and as the traffic regulator to speed up PHP by decongestionning
1509it. Also since both products use very little CPU thanks to their event-driven
1510architecture, it's often easy to install both of them on the same system. NGINX
1511implements HAProxy's PROXY protocol, thus it is easy for HAProxy to pass the
1512client's connection information to NGINX so that the application gets all the
1513relevant information. Some benchmarks have also shown that for large static
1514file serving, implementing consistent hash on HAProxy in front of NGINX can be
1515beneficial by optimizing the OS' cache hit ratio, which is basically multiplied
1516by the number of server nodes.
1517
1518
15194.3. Varnish
1520------------
1521
1522Varnish is a smart caching reverse-proxy, probably best described as a web
1523application accelerator. Varnish doesn't implement SSL/TLS and wants to dedicate
1524all of its CPU cycles to what it does best. Varnish also implements HAProxy's
1525PROXY protocol so that HAProxy can very easily be deployed in front of Varnish
1526as an SSL offloader as well as a load balancer and pass it all relevant client
1527information. Also, Varnish naturally supports decompression from the cache when
1528a server has provided a compressed object, but doesn't compress however. HAProxy
1529can then be used to compress outgoing data when backend servers do not implement
1530compression, though it's rarely a good idea to compress on the load balancer
1531unless the traffic is low.
1532
1533When building large caching farms across multiple nodes, HAProxy can make use of
1534consistent URL hashing to intelligently distribute the load to the caching nodes
1535and avoid cache duplication, resulting in a total cache size which is the sum of
1536all caching nodes.
1537
1538
15394.4. Alternatives
1540-----------------
1541
1542Linux Virtual Server (LVS or IPVS) is the layer 4 load balancer included within
1543the Linux kernel. It works at the packet level and handles TCP and UDP. In most
1544cases it's more a complement than an alternative since it doesn't have layer 7
1545knowledge at all.
1546
1547Pound is another well-known load balancer. It's much simpler and has much less
1548features than HAProxy but for many very basic setups both can be used. Its
1549author has always focused on code auditability first and wants to maintain the
1550set of features low. Its thread-based architecture scales less well with high
1551connection counts, but it's a good product.
1552
1553Pen is a quite light load balancer. It supports SSL, maintains persistence using
1554a fixed-size table of its clients' IP addresses. It supports a packet-oriented
1555mode allowing it to support direct server return and UDP to some extents. It is
1556meant for small loads (the persistence table only has 2048 entries).
1557
1558NGINX can do some load balancing to some extents, though it's clearly not its
1559primary function. Production traffic is used to detect server failures, the
1560load balancing algorithms are more limited, and the stickiness is very limited.
1561But it can make sense in some simple deployment scenarios where it is already
1562present. The good thing is that since it integrates very well with HAProxy,
1563there's nothing wrong with adding HAProxy later when its limits have been faced.
1564
1565Varnish also does some load balancing of its backend servers and does support
1566real health checks. It doesn't implement stickiness however, so just like with
1567NGINX, as long as stickiness is not needed that can be enough to start with.
1568And similarly, since HAProxy and Varnish integrate so well together, it's easy
1569to add it later into the mix to complement the feature set.
1570