blob: 683b997901609f477538770adab0f3ac03a90677 [file] [log] [blame]
Willy Tarreau2212e6a2015-10-13 14:40:55 +02001 ------------------------
2 HAProxy Management Guide
3 ------------------------
Willy Tarreau0e658fb2016-11-25 16:55:50 +01004 version 1.8
Willy Tarreau2212e6a2015-10-13 14:40:55 +02005
6
7This document describes how to start, stop, manage, and troubleshoot HAProxy,
8as well as some known limitations and traps to avoid. It does not describe how
9to configure it (for this please read configuration.txt).
10
11Note to documentation contributors :
12 This document is formatted with 80 columns per line, with even number of
13 spaces for indentation and without tabs. Please follow these rules strictly
14 so that it remains easily printable everywhere. If you add sections, please
15 update the summary below for easier searching.
16
17
18Summary
19-------
20
211. Prerequisites
222. Quick reminder about HAProxy's architecture
233. Starting HAProxy
244. Stopping and restarting HAProxy
255. File-descriptor limitations
266. Memory management
277. CPU usage
288. Logging
299. Statistics and monitoring
Willy Tarreau44aed902015-10-13 14:45:29 +0200309.1. CSV format
Willy Tarreau5d8b9792016-03-11 11:09:34 +0100319.2. Typed output format
329.3. Unix Socket commands
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003310. Tricks for easier configuration management
3411. Well-known traps to avoid
3512. Debugging and performance issues
3613. Security considerations
37
38
391. Prerequisites
40----------------
41
42In this document it is assumed that the reader has sufficient administration
43skills on a UNIX-like operating system, uses the shell on a daily basis and is
44familiar with troubleshooting utilities such as strace and tcpdump.
45
46
472. Quick reminder about HAProxy's architecture
48----------------------------------------------
49
50HAProxy is a single-threaded, event-driven, non-blocking daemon. This means is
51uses event multiplexing to schedule all of its activities instead of relying on
52the system to schedule between multiple activities. Most of the time it runs as
53a single process, so the output of "ps aux" on a system will report only one
54"haproxy" process, unless a soft reload is in progress and an older process is
55finishing its job in parallel to the new one. It is thus always easy to trace
56its activity using the strace utility.
57
58HAProxy is designed to isolate itself into a chroot jail during startup, where
59it cannot perform any file-system access at all. This is also true for the
60libraries it depends on (eg: libc, libssl, etc). The immediate effect is that
61a running process will not be able to reload a configuration file to apply
62changes, instead a new process will be started using the updated configuration
63file. Some other less obvious effects are that some timezone files or resolver
64files the libc might attempt to access at run time will not be found, though
65this should generally not happen as they're not needed after startup. A nice
66consequence of this principle is that the HAProxy process is totally stateless,
67and no cleanup is needed after it's killed, so any killing method that works
68will do the right thing.
69
70HAProxy doesn't write log files, but it relies on the standard syslog protocol
71to send logs to a remote server (which is often located on the same system).
72
73HAProxy uses its internal clock to enforce timeouts, that is derived from the
74system's time but where unexpected drift is corrected. This is done by limiting
75the time spent waiting in poll() for an event, and measuring the time it really
76took. In practice it never waits more than one second. This explains why, when
77running strace over a completely idle process, periodic calls to poll() (or any
78of its variants) surrounded by two gettimeofday() calls are noticed. They are
79normal, completely harmless and so cheap that the load they imply is totally
80undetectable at the system scale, so there's nothing abnormal there. Example :
81
82 16:35:40.002320 gettimeofday({1442759740, 2605}, NULL) = 0
83 16:35:40.002942 epoll_wait(0, {}, 200, 1000) = 0
84 16:35:41.007542 gettimeofday({1442759741, 7641}, NULL) = 0
85 16:35:41.007998 gettimeofday({1442759741, 8114}, NULL) = 0
86 16:35:41.008391 epoll_wait(0, {}, 200, 1000) = 0
87 16:35:42.011313 gettimeofday({1442759742, 11411}, NULL) = 0
88
89HAProxy is a TCP proxy, not a router. It deals with established connections that
90have been validated by the kernel, and not with packets of any form nor with
91sockets in other states (eg: no SYN_RECV nor TIME_WAIT), though their existence
92may prevent it from binding a port. It relies on the system to accept incoming
93connections and to initiate outgoing connections. An immediate effect of this is
94that there is no relation between packets observed on the two sides of a
95forwarded connection, which can be of different size, numbers and even family.
96Since a connection may only be accepted from a socket in LISTEN state, all the
97sockets it is listening to are necessarily visible using the "netstat" utility
98to show listening sockets. Example :
99
100 # netstat -ltnp
101 Active Internet connections (only servers)
102 Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
103 tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1629/sshd
104 tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2847/haproxy
105 tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 2847/haproxy
106
107
1083. Starting HAProxy
109-------------------
110
111HAProxy is started by invoking the "haproxy" program with a number of arguments
112passed on the command line. The actual syntax is :
113
114 $ haproxy [<options>]*
115
116where [<options>]* is any number of options. An option always starts with '-'
117followed by one of more letters, and possibly followed by one or multiple extra
118arguments. Without any option, HAProxy displays the help page with a reminder
119about supported options. Available options may vary slightly based on the
120operating system. A fair number of these options overlap with an equivalent one
121if the "global" section. In this case, the command line always has precedence
122over the configuration file, so that the command line can be used to quickly
123enforce some settings without touching the configuration files. The current
124list of options is :
125
126 -- <cfgfile>* : all the arguments following "--" are paths to configuration
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200127 file/directory to be loaded and processed in the declaration order. It is
128 mostly useful when relying on the shell to load many files that are
129 numerically ordered. See also "-f". The difference between "--" and "-f" is
130 that one "-f" must be placed before each file name, while a single "--" is
131 needed before all file names. Both options can be used together, the
132 command line ordering still applies. When more than one file is specified,
133 each file must start on a section boundary, so the first keyword of each
134 file must be one of "global", "defaults", "peers", "listen", "frontend",
135 "backend", and so on. A file cannot contain just a server list for example.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200136
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200137 -f <cfgfile|cfgdir> : adds <cfgfile> to the list of configuration files to be
138 loaded. If <cfgdir> is a directory, all the files (and only files) it
Dan Lloyd8e48b872016-07-01 21:01:18 -0400139 contains are added in lexical order (using LC_COLLATE=C) to the list of
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200140 configuration files to be loaded ; only files with ".cfg" extension are
141 added, only non hidden files (not prefixed with ".") are added.
142 Configuration files are loaded and processed in their declaration order.
143 This option may be specified multiple times to load multiple files. See
144 also "--". The difference between "--" and "-f" is that one "-f" must be
145 placed before each file name, while a single "--" is needed before all file
146 names. Both options can be used together, the command line ordering still
147 applies. When more than one file is specified, each file must start on a
148 section boundary, so the first keyword of each file must be one of
149 "global", "defaults", "peers", "listen", "frontend", "backend", and so on.
150 A file cannot contain just a server list for example.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200151
152 -C <dir> : changes to directory <dir> before loading configuration
153 files. This is useful when using relative paths. Warning when using
154 wildcards after "--" which are in fact replaced by the shell before
155 starting haproxy.
156
157 -D : start as a daemon. The process detaches from the current terminal after
158 forking, and errors are not reported anymore in the terminal. It is
159 equivalent to the "daemon" keyword in the "global" section of the
160 configuration. It is recommended to always force it in any init script so
161 that a faulty configuration doesn't prevent the system from booting.
162
163 -Ds : work in systemd mode. Only used by the systemd wrapper.
164
165 -L <name> : change the local peer name to <name>, which defaults to the local
166 hostname. This is used only with peers replication.
167
168 -N <limit> : sets the default per-proxy maxconn to <limit> instead of the
169 builtin default value (usually 2000). Only useful for debugging.
170
171 -V : enable verbose mode (disables quiet mode). Reverts the effect of "-q" or
172 "quiet".
173
174 -c : only performs a check of the configuration files and exits before trying
175 to bind. The exit status is zero if everything is OK, or non-zero if an
176 error is encountered.
177
178 -d : enable debug mode. This disables daemon mode, forces the process to stay
179 in foreground and to show incoming and outgoing events. It is equivalent to
180 the "global" section's "debug" keyword. It must never be used in an init
181 script.
182
183 -dG : disable use of getaddrinfo() to resolve host names into addresses. It
184 can be used when suspecting that getaddrinfo() doesn't work as expected.
185 This option was made available because many bogus implementations of
186 getaddrinfo() exist on various systems and cause anomalies that are
187 difficult to troubleshoot.
188
Dan Lloyd8e48b872016-07-01 21:01:18 -0400189 -dM[<byte>] : forces memory poisoning, which means that each and every
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200190 memory region allocated with malloc() or pool_alloc2() will be filled with
191 <byte> before being passed to the caller. When <byte> is not specified, it
192 defaults to 0x50 ('P'). While this slightly slows down operations, it is
193 useful to reliably trigger issues resulting from missing initializations in
194 the code that cause random crashes. Note that -dM0 has the effect of
195 turning any malloc() into a calloc(). In any case if a bug appears or
196 disappears when using this option it means there is a bug in haproxy, so
197 please report it.
198
199 -dS : disable use of the splice() system call. It is equivalent to the
200 "global" section's "nosplice" keyword. This may be used when splice() is
201 suspected to behave improperly or to cause performance issues, or when
202 using strace to see the forwarded data (which do not appear when using
203 splice()).
204
205 -dV : disable SSL verify on the server side. It is equivalent to having
206 "ssl-server-verify none" in the "global" section. This is useful when
207 trying to reproduce production issues out of the production
208 environment. Never use this in an init script as it degrades SSL security
209 to the servers.
210
211 -db : disable background mode and multi-process mode. The process remains in
212 foreground. It is mainly used during development or during small tests, as
213 Ctrl-C is enough to stop the process. Never use it in an init script.
214
215 -de : disable the use of the "epoll" poller. It is equivalent to the "global"
216 section's keyword "noepoll". It is mostly useful when suspecting a bug
217 related to this poller. On systems supporting epoll, the fallback will
218 generally be the "poll" poller.
219
220 -dk : disable the use of the "kqueue" poller. It is equivalent to the
221 "global" section's keyword "nokqueue". It is mostly useful when suspecting
222 a bug related to this poller. On systems supporting kqueue, the fallback
223 will generally be the "poll" poller.
224
225 -dp : disable the use of the "poll" poller. It is equivalent to the "global"
226 section's keyword "nopoll". It is mostly useful when suspecting a bug
227 related to this poller. On systems supporting poll, the fallback will
228 generally be the "select" poller, which cannot be disabled and is limited
229 to 1024 file descriptors.
230
Willy Tarreau3eed10e2016-11-07 21:03:16 +0100231 -dr : ignore server address resolution failures. It is very common when
232 validating a configuration out of production not to have access to the same
233 resolvers and to fail on server address resolution, making it difficult to
234 test a configuration. This option simply appends the "none" method to the
235 list of address resolution methods for all servers, ensuring that even if
236 the libc fails to resolve an address, the startup sequence is not
237 interrupted.
238
Willy Tarreau70060452015-12-14 12:46:07 +0100239 -m <limit> : limit the total allocatable memory to <limit> megabytes across
240 all processes. This may cause some connection refusals or some slowdowns
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200241 depending on the amount of memory needed for normal operations. This is
Willy Tarreau70060452015-12-14 12:46:07 +0100242 mostly used to force the processes to work in a constrained resource usage
243 scenario. It is important to note that the memory is not shared between
244 processes, so in a multi-process scenario, this value is first divided by
245 global.nbproc before forking.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200246
247 -n <limit> : limits the per-process connection limit to <limit>. This is
248 equivalent to the global section's keyword "maxconn". It has precedence
249 over this keyword. This may be used to quickly force lower limits to avoid
250 a service outage on systems where resource limits are too low.
251
252 -p <file> : write all processes' pids into <file> during startup. This is
253 equivalent to the "global" section's keyword "pidfile". The file is opened
254 before entering the chroot jail, and after doing the chdir() implied by
255 "-C". Each pid appears on its own line.
256
257 -q : set "quiet" mode. This disables some messages during the configuration
258 parsing and during startup. It can be used in combination with "-c" to
259 just check if a configuration file is valid or not.
260
261 -sf <pid>* : send the "finish" signal (SIGUSR1) to older processes after boot
262 completion to ask them to finish what they are doing and to leave. <pid>
263 is a list of pids to signal (one per argument). The list ends on any
264 option starting with a "-". It is not a problem if the list of pids is
265 empty, so that it can be built on the fly based on the result of a command
266 like "pidof" or "pgrep".
267
268 -st <pid>* : send the "terminate" signal (SIGTERM) to older processes after
269 boot completion to terminate them immediately without finishing what they
270 were doing. <pid> is a list of pids to signal (one per argument). The list
271 is ends on any option starting with a "-". It is not a problem if the list
272 of pids is empty, so that it can be built on the fly based on the result of
273 a command like "pidof" or "pgrep".
274
275 -v : report the version and build date.
276
277 -vv : display the version, build options, libraries versions and usable
278 pollers. This output is systematically requested when filing a bug report.
279
Dan Lloyd8e48b872016-07-01 21:01:18 -0400280A safe way to start HAProxy from an init file consists in forcing the daemon
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200281mode, storing existing pids to a pid file and using this pid file to notify
282older processes to finish before leaving :
283
284 haproxy -f /etc/haproxy.cfg \
285 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
286
287When the configuration is split into a few specific files (eg: tcp vs http),
288it is recommended to use the "-f" option :
289
290 haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
291 -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
292 -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
293 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
294
295When an unknown number of files is expected, such as customer-specific files,
296it is recommended to assign them a name starting with a fixed-size sequence
297number and to use "--" to load them, possibly after loading some defaults :
298
299 haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
300 -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
301 -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
302 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) \
303 -f /etc/haproxy/default-customers.cfg -- /etc/haproxy/customers/*
304
305Sometimes a failure to start may happen for whatever reason. Then it is
306important to verify if the version of HAProxy you are invoking is the expected
307version and if it supports the features you are expecting (eg: SSL, PCRE,
308compression, Lua, etc). This can be verified using "haproxy -vv". Some
309important information such as certain build options, the target system and
310the versions of the libraries being used are reported there. It is also what
311you will systematically be asked for when posting a bug report :
312
313 $ haproxy -vv
314 HA-Proxy version 1.6-dev7-a088d3-4 2015/10/08
315 Copyright 2000-2015 Willy Tarreau <willy@haproxy.org>
316
317 Build options :
318 TARGET = linux2628
319 CPU = generic
320 CC = gcc
321 CFLAGS = -pg -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement \
322 -DBUFSIZE=8030 -DMAXREWRITE=1030 -DSO_MARK=36 -DTCP_REPAIR=19
323 OPTIONS = USE_ZLIB=1 USE_DLMALLOC=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
324
325 Default settings :
326 maxconn = 2000, bufsize = 8030, maxrewrite = 1030, maxpollevents = 200
327
328 Encrypted password support via crypt(3): yes
329 Built with zlib version : 1.2.6
330 Compression algorithms supported : identity("identity"), deflate("deflate"), \
331 raw-deflate("deflate"), gzip("gzip")
332 Built with OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
333 Running on OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
334 OpenSSL library supports TLS extensions : yes
335 OpenSSL library supports SNI : yes
336 OpenSSL library supports prefer-server-ciphers : yes
337 Built with PCRE version : 8.12 2011-01-15
338 PCRE library supports JIT : no (USE_PCRE_JIT not set)
339 Built with Lua version : Lua 5.3.1
340 Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND
341
342 Available polling systems :
343 epoll : pref=300, test result OK
344 poll : pref=200, test result OK
345 select : pref=150, test result OK
346 Total: 3 (3 usable), will use epoll.
347
348The relevant information that many non-developer users can verify here are :
349 - the version : 1.6-dev7-a088d3-4 above means the code is currently at commit
350 ID "a088d3" which is the 4th one after after official version "1.6-dev7".
351 Version 1.6-dev7 would show as "1.6-dev7-8c1ad7". What matters here is in
352 fact "1.6-dev7". This is the 7th development version of what will become
353 version 1.6 in the future. A development version not suitable for use in
354 production (unless you know exactly what you are doing). A stable version
355 will show as a 3-numbers version, such as "1.5.14-16f863", indicating the
356 14th level of fix on top of version 1.5. This is a production-ready version.
357
358 - the release date : 2015/10/08. It is represented in the universal
359 year/month/day format. Here this means August 8th, 2015. Given that stable
360 releases are issued every few months (1-2 months at the beginning, sometimes
361 6 months once the product becomes very stable), if you're seeing an old date
362 here, it means you're probably affected by a number of bugs or security
363 issues that have since been fixed and that it might be worth checking on the
364 official site.
365
366 - build options : they are relevant to people who build their packages
367 themselves, they can explain why things are not behaving as expected. For
368 example the development version above was built for Linux 2.6.28 or later,
Dan Lloyd8e48b872016-07-01 21:01:18 -0400369 targeting a generic CPU (no CPU-specific optimizations), and lacks any
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200370 code optimization (-O0) so it will perform poorly in terms of performance.
371
372 - libraries versions : zlib version is reported as found in the library
373 itself. In general zlib is considered a very stable product and upgrades
374 are almost never needed. OpenSSL reports two versions, the version used at
375 build time and the one being used, as found on the system. These ones may
376 differ by the last letter but never by the numbers. The build date is also
377 reported because most OpenSSL bugs are security issues and need to be taken
378 seriously, so this library absolutely needs to be kept up to date. Seeing a
379 4-months old version here is highly suspicious and indeed an update was
380 missed. PCRE provides very fast regular expressions and is highly
381 recommended. Certain of its extensions such as JIT are not present in all
382 versions and still young so some people prefer not to build with them,
Dan Lloyd8e48b872016-07-01 21:01:18 -0400383 which is why the build status is reported as well. Regarding the Lua
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200384 scripting language, HAProxy expects version 5.3 which is very young since
385 it was released a little time before HAProxy 1.6. It is important to check
386 on the Lua web site if some fixes are proposed for this branch.
387
388 - Available polling systems will affect the process's scalability when
389 dealing with more than about one thousand of concurrent connections. These
390 ones are only available when the correct system was indicated in the TARGET
391 variable during the build. The "epoll" mechanism is highly recommended on
392 Linux, and the kqueue mechanism is highly recommended on BSD. Lacking them
393 will result in poll() or even select() being used, causing a high CPU usage
394 when dealing with a lot of connections.
395
396
3974. Stopping and restarting HAProxy
398----------------------------------
399
400HAProxy supports a graceful and a hard stop. The hard stop is simple, when the
401SIGTERM signal is sent to the haproxy process, it immediately quits and all
402established connections are closed. The graceful stop is triggered when the
403SIGUSR1 signal is sent to the haproxy process. It consists in only unbinding
404from listening ports, but continue to process existing connections until they
405close. Once the last connection is closed, the process leaves.
406
407The hard stop method is used for the "stop" or "restart" actions of the service
408management script. The graceful stop is used for the "reload" action which
409tries to seamlessly reload a new configuration in a new process.
410
411Both of these signals may be sent by the new haproxy process itself during a
412reload or restart, so that they are sent at the latest possible moment and only
413if absolutely required. This is what is performed by the "-st" (hard) and "-sf"
414(graceful) options respectively.
415
416To understand better how these signals are used, it is important to understand
417the whole restart mechanism.
418
419First, an existing haproxy process is running. The administrator uses a system
420specific command such as "/etc/init.d/haproxy reload" to indicate he wants to
421take the new configuration file into effect. What happens then is the following.
422First, the service script (/etc/init.d/haproxy or equivalent) will verify that
423the configuration file parses correctly using "haproxy -c". After that it will
424try to start haproxy with this configuration file, using "-st" or "-sf".
425
426Then HAProxy tries to bind to all listening ports. If some fatal errors happen
427(eg: address not present on the system, permission denied), the process quits
428with an error. If a socket binding fails because a port is already in use, then
429the process will first send a SIGTTOU signal to all the pids specified in the
430"-st" or "-sf" pid list. This is what is called the "pause" signal. It instructs
431all existing haproxy processes to temporarily stop listening to their ports so
432that the new process can try to bind again. During this time, the old process
433continues to process existing connections. If the binding still fails (because
434for example a port is shared with another daemon), then the new process sends a
435SIGTTIN signal to the old processes to instruct them to resume operations just
436as if nothing happened. The old processes will then restart listening to the
437ports and continue to accept connections. Not that this mechanism is system
Dan Lloyd8e48b872016-07-01 21:01:18 -0400438dependent and some operating systems may not support it in multi-process mode.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200439
440If the new process manages to bind correctly to all ports, then it sends either
441the SIGTERM (hard stop in case of "-st") or the SIGUSR1 (graceful stop in case
442of "-sf") to all processes to notify them that it is now in charge of operations
443and that the old processes will have to leave, either immediately or once they
444have finished their job.
445
446It is important to note that during this timeframe, there are two small windows
447of a few milliseconds each where it is possible that a few connection failures
448will be noticed during high loads. Typically observed failure rates are around
4491 failure during a reload operation every 10000 new connections per second,
450which means that a heavily loaded site running at 30000 new connections per
451second may see about 3 failed connection upon every reload. The two situations
452where this happens are :
453
454 - if the new process fails to bind due to the presence of the old process,
455 it will first have to go through the SIGTTOU+SIGTTIN sequence, which
456 typically lasts about one millisecond for a few tens of frontends, and
457 during which some ports will not be bound to the old process and not yet
458 bound to the new one. HAProxy works around this on systems that support the
459 SO_REUSEPORT socket options, as it allows the new process to bind without
460 first asking the old one to unbind. Most BSD systems have been supporting
461 this almost forever. Linux has been supporting this in version 2.0 and
462 dropped it around 2.2, but some patches were floating around by then. It
463 was reintroduced in kernel 3.9, so if you are observing a connection
Dan Lloyd8e48b872016-07-01 21:01:18 -0400464 failure rate above the one mentioned above, please ensure that your kernel
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200465 is 3.9 or newer, or that relevant patches were backported to your kernel
466 (less likely).
467
468 - when the old processes close the listening ports, the kernel may not always
469 redistribute any pending connection that was remaining in the socket's
470 backlog. Under high loads, a SYN packet may happen just before the socket
471 is closed, and will lead to an RST packet being sent to the client. In some
472 critical environments where even one drop is not acceptable, these ones are
473 sometimes dealt with using firewall rules to block SYN packets during the
474 reload, forcing the client to retransmit. This is totally system-dependent,
475 as some systems might be able to visit other listening queues and avoid
476 this RST. A second case concerns the ACK from the client on a local socket
477 that was in SYN_RECV state just before the close. This ACK will lead to an
478 RST packet while the haproxy process is still not aware of it. This one is
Dan Lloyd8e48b872016-07-01 21:01:18 -0400479 harder to get rid of, though the firewall filtering rules mentioned above
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200480 will work well if applied one second or so before restarting the process.
481
482For the vast majority of users, such drops will never ever happen since they
483don't have enough load to trigger the race conditions. And for most high traffic
484users, the failure rate is still fairly within the noise margin provided that at
485least SO_REUSEPORT is properly supported on their systems.
486
487
4885. File-descriptor limitations
489------------------------------
490
491In order to ensure that all incoming connections will successfully be served,
492HAProxy computes at load time the total number of file descriptors that will be
493needed during the process's life. A regular Unix process is generally granted
4941024 file descriptors by default, and a privileged process can raise this limit
495itself. This is one reason for starting HAProxy as root and letting it adjust
496the limit. The default limit of 1024 file descriptors roughly allow about 500
497concurrent connections to be processed. The computation is based on the global
498maxconn parameter which limits the total number of connections per process, the
499number of listeners, the number of servers which have a health check enabled,
500the agent checks, the peers, the loggers and possibly a few other technical
501requirements. A simple rough estimate of this number consists in simply
502doubling the maxconn value and adding a few tens to get the approximate number
503of file descriptors needed.
504
505Originally HAProxy did not know how to compute this value, and it was necessary
506to pass the value using the "ulimit-n" setting in the global section. This
507explains why even today a lot of configurations are seen with this setting
508present. Unfortunately it was often miscalculated resulting in connection
509failures when approaching maxconn instead of throttling incoming connection
510while waiting for the needed resources. For this reason it is important to
Dan Lloyd8e48b872016-07-01 21:01:18 -0400511remove any vestigial "ulimit-n" setting that can remain from very old versions.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200512
513Raising the number of file descriptors to accept even moderate loads is
514mandatory but comes with some OS-specific adjustments. First, the select()
515polling system is limited to 1024 file descriptors. In fact on Linux it used
516to be capable of handling more but since certain OS ship with excessively
517restrictive SELinux policies forbidding the use of select() with more than
5181024 file descriptors, HAProxy now refuses to start in this case in order to
519avoid any issue at run time. On all supported operating systems, poll() is
520available and will not suffer from this limitation. It is automatically picked
Dan Lloyd8e48b872016-07-01 21:01:18 -0400521so there is nothing to do to get a working configuration. But poll's becomes
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200522very slow when the number of file descriptors increases. While HAProxy does its
523best to limit this performance impact (eg: via the use of the internal file
524descriptor cache and batched processing), a good rule of thumb is that using
525poll() with more than a thousand concurrent connections will use a lot of CPU.
526
527For Linux systems base on kernels 2.6 and above, the epoll() system call will
528be used. It's a much more scalable mechanism relying on callbacks in the kernel
529that guarantee a constant wake up time regardless of the number of registered
530monitored file descriptors. It is automatically used where detected, provided
531that HAProxy had been built for one of the Linux flavors. Its presence and
532support can be verified using "haproxy -vv".
533
534For BSD systems which support it, kqueue() is available as an alternative. It
535is much faster than poll() and even slightly faster than epoll() thanks to its
536batched handling of changes. At least FreeBSD and OpenBSD support it. Just like
537with Linux's epoll(), its support and availability are reported in the output
538of "haproxy -vv".
539
540Having a good poller is one thing, but it is mandatory that the process can
541reach the limits. When HAProxy starts, it immediately sets the new process's
542file descriptor limits and verifies if it succeeds. In case of failure, it
543reports it before forking so that the administrator can see the problem. As
544long as the process is started by as root, there should be no reason for this
545setting to fail. However, it can fail if the process is started by an
546unprivileged user. If there is a compelling reason for *not* starting haproxy
547as root (eg: started by end users, or by a per-application account), then the
548file descriptor limit can be raised by the system administrator for this
549specific user. The effectiveness of the setting can be verified by issuing
550"ulimit -n" from the user's command line. It should reflect the new limit.
551
552Warning: when an unprivileged user's limits are changed in this user's account,
553it is fairly common that these values are only considered when the user logs in
554and not at all in some scripts run at system boot time nor in crontabs. This is
555totally dependent on the operating system, keep in mind to check "ulimit -n"
556before starting haproxy when running this way. The general advice is never to
557start haproxy as an unprivileged user for production purposes. Another good
558reason is that it prevents haproxy from enabling some security protections.
559
560Once it is certain that the system will allow the haproxy process to use the
561requested number of file descriptors, two new system-specific limits may be
562encountered. The first one is the system-wide file descriptor limit, which is
563the total number of file descriptors opened on the system, covering all
564processes. When this limit is reached, accept() or socket() will typically
565return ENFILE. The second one is the per-process hard limit on the number of
566file descriptors, it prevents setrlimit() from being set higher. Both are very
567dependent on the operating system. On Linux, the system limit is set at boot
568based on the amount of memory. It can be changed with the "fs.file-max" sysctl.
569And the per-process hard limit is set to 1048576 by default, but it can be
570changed using the "fs.nr_open" sysctl.
571
572File descriptor limitations may be observed on a running process when they are
573set too low. The strace utility will report that accept() and socket() return
574"-1 EMFILE" when the process's limits have been reached. In this case, simply
575raising the "ulimit-n" value (or removing it) will solve the problem. If these
576system calls return "-1 ENFILE" then it means that the kernel's limits have
577been reached and that something must be done on a system-wide parameter. These
578trouble must absolutely be addressed, as they result in high CPU usage (when
579accept() fails) and failed connections that are generally visible to the user.
580One solution also consists in lowering the global maxconn value to enforce
581serialization, and possibly to disable HTTP keep-alive to force connections
582to be released and reused faster.
583
584
5856. Memory management
586--------------------
587
588HAProxy uses a simple and fast pool-based memory management. Since it relies on
589a small number of different object types, it's much more efficient to pick new
590objects from a pool which already contains objects of the appropriate size than
591to call malloc() for each different size. The pools are organized as a stack or
592LIFO, so that newly allocated objects are taken from recently released objects
593still hot in the CPU caches. Pools of similar sizes are merged together, in
594order to limit memory fragmentation.
595
596By default, since the focus is set on performance, each released object is put
597back into the pool it came from, and allocated objects are never freed since
598they are expected to be reused very soon.
599
600On the CLI, it is possible to check how memory is being used in pools thanks to
601the "show pools" command :
602
603 > show pools
604 Dumping pools usage. Use SIGQUIT to flush them.
605 - Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 3 users [SHARED]
606 - Pool hlua_com (48 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
607 - Pool vars (64 bytes) : 0 allocated (0 bytes), 0 used, 2 users [SHARED]
608 - Pool task (112 bytes) : 5 allocated (560 bytes), 5 used, 1 users [SHARED]
609 - Pool session (128 bytes) : 1 allocated (128 bytes), 1 used, 2 users [SHARED]
610 - Pool http_txn (272 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
611 - Pool connection (352 bytes) : 2 allocated (704 bytes), 2 used, 1 users [SHARED]
612 - Pool hdr_idx (416 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
613 - Pool stream (864 bytes) : 1 allocated (864 bytes), 1 used, 1 users [SHARED]
614 - Pool requri (1024 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
615 - Pool buffer (8064 bytes) : 3 allocated (24192 bytes), 2 used, 1 users [SHARED]
616 Total: 11 pools, 26608 bytes allocated, 18544 used.
617
618The pool name is only indicative, it's the name of the first object type using
619this pool. The size in parenthesis is the object size for objects in this pool.
620Object sizes are always rounded up to the closest multiple of 16 bytes. The
621number of objects currently allocated and the equivalent number of bytes is
622reported so that it is easy to know which pool is responsible for the highest
623memory usage. The number of objects currently in use is reported as well in the
624"used" field. The difference between "allocated" and "used" corresponds to the
625objects that have been freed and are available for immediate use.
626
627It is possible to limit the amount of memory allocated per process using the
628"-m" command line option, followed by a number of megabytes. It covers all of
629the process's addressable space, so that includes memory used by some libraries
630as well as the stack, but it is a reliable limit when building a resource
631constrained system. It works the same way as "ulimit -v" on systems which have
632it, or "ulimit -d" for the other ones.
633
634If a memory allocation fails due to the memory limit being reached or because
635the system doesn't have any enough memory, then haproxy will first start to
636free all available objects from all pools before attempting to allocate memory
637again. This mechanism of releasing unused memory can be triggered by sending
638the signal SIGQUIT to the haproxy process. When doing so, the pools state prior
639to the flush will also be reported to stderr when the process runs in
640foreground.
641
642During a reload operation, the process switched to the graceful stop state also
643automatically performs some flushes after releasing any connection so that all
644possible memory is released to save it for the new process.
645
646
6477. CPU usage
648------------
649
650HAProxy normally spends most of its time in the system and a smaller part in
651userland. A finely tuned 3.5 GHz CPU can sustain a rate about 80000 end-to-end
652connection setups and closes per second at 100% CPU on a single core. When one
653core is saturated, typical figures are :
654 - 95% system, 5% user for long TCP connections or large HTTP objects
655 - 85% system and 15% user for short TCP connections or small HTTP objects in
656 close mode
657 - 70% system and 30% user for small HTTP objects in keep-alive mode
658
659The amount of rules processing and regular expressions will increase the user
660land part. The presence of firewall rules, connection tracking, complex routing
661tables in the system will instead increase the system part.
662
663On most systems, the CPU time observed during network transfers can be cut in 4
664parts :
665 - the interrupt part, which concerns all the processing performed upon I/O
666 receipt, before the target process is even known. Typically Rx packets are
667 accounted for in interrupt. On some systems such as Linux where interrupt
668 processing may be deferred to a dedicated thread, it can appear as softirq,
669 and the thread is called ksoftirqd/0 (for CPU 0). The CPU taking care of
670 this load is generally defined by the hardware settings, though in the case
671 of softirq it is often possible to remap the processing to another CPU.
672 This interrupt part will often be perceived as parasitic since it's not
673 associated with any process, but it actually is some processing being done
674 to prepare the work for the process.
675
676 - the system part, which concerns all the processing done using kernel code
677 called from userland. System calls are accounted as system for example. All
678 synchronously delivered Tx packets will be accounted for as system time. If
679 some packets have to be deferred due to queues filling up, they may then be
680 processed in interrupt context later (eg: upon receipt of an ACK opening a
681 TCP window).
682
683 - the user part, which exclusively runs application code in userland. HAProxy
684 runs exclusively in this part, though it makes heavy use of system calls.
685 Rules processing, regular expressions, compression, encryption all add to
686 the user portion of CPU consumption.
687
688 - the idle part, which is what the CPU does when there is nothing to do. For
689 example HAProxy waits for an incoming connection, or waits for some data to
690 leave, meaning the system is waiting for an ACK from the client to push
691 these data.
692
693In practice regarding HAProxy's activity, it is in general reasonably accurate
694(but totally inexact) to consider that interrupt/softirq are caused by Rx
695processing in kernel drivers, that user-land is caused by layer 7 processing
696in HAProxy, and that system time is caused by network processing on the Tx
697path.
698
699Since HAProxy runs around an event loop, it waits for new events using poll()
700(or any alternative) and processes all these events as fast as possible before
701going back to poll() waiting for new events. It measures the time spent waiting
702in poll() compared to the time spent doing processing events. The ratio of
703polling time vs total time is called the "idle" time, it's the amount of time
704spent waiting for something to happen. This ratio is reported in the stats page
705on the "idle" line, or "Idle_pct" on the CLI. When it's close to 100%, it means
706the load is extremely low. When it's close to 0%, it means that there is
707constantly some activity. While it cannot be very accurate on an overloaded
708system due to other processes possibly preempting the CPU from the haproxy
709process, it still provides a good estimate about how HAProxy considers it is
710working : if the load is low and the idle ratio is low as well, it may indicate
711that HAProxy has a lot of work to do, possibly due to very expensive rules that
712have to be processed. Conversely, if HAProxy indicates the idle is close to
713100% while things are slow, it means that it cannot do anything to speed things
714up because it is already waiting for incoming data to process. In the example
715below, haproxy is completely idle :
716
717 $ echo "show info" | socat - /var/run/haproxy.sock | grep ^Idle
718 Idle_pct: 100
719
720When the idle ratio starts to become very low, it is important to tune the
721system and place processes and interrupts correctly to save the most possible
722CPU resources for all tasks. If a firewall is present, it may be worth trying
723to disable it or to tune it to ensure it is not responsible for a large part
724of the performance limitation. It's worth noting that unloading a stateful
725firewall generally reduces both the amount of interrupt/softirq and of system
726usage since such firewalls act both on the Rx and the Tx paths. On Linux,
727unloading the nf_conntrack and ip_conntrack modules will show whether there is
728anything to gain. If so, then the module runs with default settings and you'll
729have to figure how to tune it for better performance. In general this consists
730in considerably increasing the hash table size. On FreeBSD, "pfctl -d" will
731disable the "pf" firewall and its stateful engine at the same time.
732
733If it is observed that a lot of time is spent in interrupt/softirq, it is
734important to ensure that they don't run on the same CPU. Most systems tend to
735pin the tasks on the CPU where they receive the network traffic because for
736certain workloads it improves things. But with heavily network-bound workloads
737it is the opposite as the haproxy process will have to fight against its kernel
738counterpart. Pinning haproxy to one CPU core and the interrupts to another one,
739all sharing the same L3 cache tends to sensibly increase network performance
740because in practice the amount of work for haproxy and the network stack are
741quite close, so they can almost fill an entire CPU each. On Linux this is done
742using taskset (for haproxy) or using cpu-map (from the haproxy config), and the
743interrupts are assigned under /proc/irq. Many network interfaces support
744multiple queues and multiple interrupts. In general it helps to spread them
745across a small number of CPU cores provided they all share the same L3 cache.
746Please always stop irq_balance which always does the worst possible thing on
747such workloads.
748
749For CPU-bound workloads consisting in a lot of SSL traffic or a lot of
750compression, it may be worth using multiple processes dedicated to certain
751tasks, though there is no universal rule here and experimentation will have to
752be performed.
753
754In order to increase the CPU capacity, it is possible to make HAProxy run as
755several processes, using the "nbproc" directive in the global section. There
756are some limitations though :
757 - health checks are run per process, so the target servers will get as many
758 checks as there are running processes ;
759 - maxconn values and queues are per-process so the correct value must be set
760 to avoid overloading the servers ;
761 - outgoing connections should avoid using port ranges to avoid conflicts
762 - stick-tables are per process and are not shared between processes ;
763 - each peers section may only run on a single process at a time ;
764 - the CLI operations will only act on a single process at a time.
765
766With this in mind, it appears that the easiest setup often consists in having
767one first layer running on multiple processes and in charge for the heavy
768processing, passing the traffic to a second layer running in a single process.
769This mechanism is suited to SSL and compression which are the two CPU-heavy
770features. Instances can easily be chained over UNIX sockets (which are cheaper
fengpeiyuancc123c62016-01-15 16:40:53 +0800771than TCP sockets and which do not waste ports), and the proxy protocol which is
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200772useful to pass client information to the next stage. When doing so, it is
773generally a good idea to bind all the single-process tasks to process number 1
774and extra tasks to next processes, as this will make it easier to generate
775similar configurations for different machines.
776
777On Linux versions 3.9 and above, running HAProxy in multi-process mode is much
778more efficient when each process uses a distinct listening socket on the same
779IP:port ; this will make the kernel evenly distribute the load across all
780processes instead of waking them all up. Please check the "process" option of
781the "bind" keyword lines in the configuration manual for more information.
782
783
7848. Logging
785----------
786
787For logging, HAProxy always relies on a syslog server since it does not perform
788any file-system access. The standard way of using it is to send logs over UDP
789to the log server (by default on port 514). Very commonly this is configured to
790127.0.0.1 where the local syslog daemon is running, but it's also used over the
791network to log to a central server. The central server provides additional
792benefits especially in active-active scenarios where it is desirable to keep
793the logs merged in arrival order. HAProxy may also make use of a UNIX socket to
794send its logs to the local syslog daemon, but it is not recommended at all,
795because if the syslog server is restarted while haproxy runs, the socket will
796be replaced and new logs will be lost. Since HAProxy will be isolated inside a
797chroot jail, it will not have the ability to reconnect to the new socket. It
798has also been observed in field that the log buffers in use on UNIX sockets are
799very small and lead to lost messages even at very light loads. But this can be
800fine for testing however.
801
802It is recommended to add the following directive to the "global" section to
803make HAProxy log to the local daemon using facility "local0" :
804
805 log 127.0.0.1:514 local0
806
807and then to add the following one to each "defaults" section or to each frontend
808and backend section :
809
810 log global
811
812This way, all logs will be centralized through the global definition of where
813the log server is.
814
815Some syslog daemons do not listen to UDP traffic by default, so depending on
816the daemon being used, the syntax to enable this will vary :
817
818 - on sysklogd, you need to pass argument "-r" on the daemon's command line
819 so that it listens to a UDP socket for "remote" logs ; note that there is
820 no way to limit it to address 127.0.0.1 so it will also receive logs from
821 remote systems ;
822
823 - on rsyslogd, the following lines must be added to the configuration file :
824
825 $ModLoad imudp
826 $UDPServerAddress *
827 $UDPServerRun 514
828
829 - on syslog-ng, a new source can be created the following way, it then needs
830 to be added as a valid source in one of the "log" directives :
831
832 source s_udp {
833 udp(ip(127.0.0.1) port(514));
834 };
835
836Please consult your syslog daemon's manual for more information. If no logs are
837seen in the system's log files, please consider the following tests :
838
839 - restart haproxy. Each frontend and backend logs one line indicating it's
840 starting. If these logs are received, it means logs are working.
841
842 - run "strace -tt -s100 -etrace=sendmsg -p <haproxy's pid>" and perform some
843 activity that you expect to be logged. You should see the log messages
844 being sent using sendmsg() there. If they don't appear, restart using
845 strace on top of haproxy. If you still see no logs, it definitely means
846 that something is wrong in your configuration.
847
848 - run tcpdump to watch for port 514, for example on the loopback interface if
849 the traffic is being sent locally : "tcpdump -As0 -ni lo port 514". If the
850 packets are seen there, it's the proof they're sent then the syslogd daemon
851 needs to be troubleshooted.
852
853While traffic logs are sent from the frontends (where the incoming connections
854are accepted), backends also need to be able to send logs in order to report a
855server state change consecutive to a health check. Please consult HAProxy's
856configuration manual for more information regarding all possible log settings.
857
Dan Lloyd8e48b872016-07-01 21:01:18 -0400858It is convenient to chose a facility that is not used by other daemons. HAProxy
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200859examples often suggest "local0" for traffic logs and "local1" for admin logs
860because they're never seen in field. A single facility would be enough as well.
861Having separate logs is convenient for log analysis, but it's also important to
862remember that logs may sometimes convey confidential information, and as such
Dan Lloyd8e48b872016-07-01 21:01:18 -0400863they must not be mixed with other logs that may accidentally be handed out to
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200864unauthorized people.
865
866For in-field troubleshooting without impacting the server's capacity too much,
867it is recommended to make use of the "halog" utility provided with HAProxy.
868This is sort of a grep-like utility designed to process HAProxy log files at
869a very fast data rate. Typical figures range between 1 and 2 GB of logs per
870second. It is capable of extracting only certain logs (eg: search for some
871classes of HTTP status codes, connection termination status, search by response
872time ranges, look for errors only), count lines, limit the output to a number
873of lines, and perform some more advanced statistics such as sorting servers
874by response time or error counts, sorting URLs by time or count, sorting client
875addresses by access count, and so on. It is pretty convenient to quickly spot
876anomalies such as a bot looping on the site, and block them.
877
878
8799. Statistics and monitoring
880----------------------------
881
Willy Tarreau44aed902015-10-13 14:45:29 +0200882It is possible to query HAProxy about its status. The most commonly used
883mechanism is the HTTP statistics page. This page also exposes an alternative
884CSV output format for monitoring tools. The same format is provided on the
885Unix socket.
886
887
8889.1. CSV format
889---------------
890
891The statistics may be consulted either from the unix socket or from the HTTP
892page. Both means provide a CSV format whose fields follow. The first line
893begins with a sharp ('#') and has one word per comma-delimited field which
894represents the title of the column. All other lines starting at the second one
895use a classical CSV format using a comma as the delimiter, and the double quote
896('"') as an optional text delimiter, but only if the enclosed text is ambiguous
897(if it contains a quote or a comma). The double-quote character ('"') in the
898text is doubled ('""'), which is the format that most tools recognize. Please
899do not insert any column before these ones in order not to break tools which
900use hard-coded column positions.
901
902In brackets after each field name are the types which may have a value for
903that field. The types are L (Listeners), F (Frontends), B (Backends), and
904S (Servers).
905
906 0. pxname [LFBS]: proxy name
907 1. svname [LFBS]: service name (FRONTEND for frontend, BACKEND for backend,
908 any name for server/listener)
909 2. qcur [..BS]: current queued requests. For the backend this reports the
910 number queued without a server assigned.
911 3. qmax [..BS]: max value of qcur
912 4. scur [LFBS]: current sessions
913 5. smax [LFBS]: max sessions
914 6. slim [LFBS]: configured session limit
Willy Tarreauc73810f2016-01-11 13:52:04 +0100915 7. stot [LFBS]: cumulative number of sessions
Willy Tarreau44aed902015-10-13 14:45:29 +0200916 8. bin [LFBS]: bytes in
917 9. bout [LFBS]: bytes out
918 10. dreq [LFB.]: requests denied because of security concerns.
919 - For tcp this is because of a matched tcp-request content rule.
920 - For http this is because of a matched http-request or tarpit rule.
921 11. dresp [LFBS]: responses denied because of security concerns.
922 - For http this is because of a matched http-request rule, or
923 "option checkcache".
924 12. ereq [LF..]: request errors. Some of the possible causes are:
925 - early termination from the client, before the request has been sent.
926 - read error from the client
927 - client timeout
928 - client closed connection
929 - various bad requests from the client.
930 - request was tarpitted.
931 13. econ [..BS]: number of requests that encountered an error trying to
932 connect to a backend server. The backend stat is the sum of the stat
933 for all servers of that backend, plus any connection errors not
934 associated with a particular server (such as the backend having no
935 active servers).
936 14. eresp [..BS]: response errors. srv_abrt will be counted here also.
937 Some other errors are:
938 - write error on the client socket (won't be counted for the server stat)
939 - failure applying filters to the response.
940 15. wretr [..BS]: number of times a connection to a server was retried.
941 16. wredis [..BS]: number of times a request was redispatched to another
942 server. The server value counts the number of times that server was
943 switched away from.
Willy Tarreaub96dd282016-11-09 14:45:51 +0100944 17. status [LFBS]: status (UP/DOWN/NOLB/MAINT/MAINT(via)/MAINT(resolution)...)
Willy Tarreau44aed902015-10-13 14:45:29 +0200945 18. weight [..BS]: total weight (backend), server weight (server)
946 19. act [..BS]: number of active servers (backend), server is active (server)
947 20. bck [..BS]: number of backup servers (backend), server is backup (server)
948 21. chkfail [...S]: number of failed checks. (Only counts checks failed when
949 the server is up.)
950 22. chkdown [..BS]: number of UP->DOWN transitions. The backend counter counts
951 transitions to the whole backend being down, rather than the sum of the
952 counters for each server.
953 23. lastchg [..BS]: number of seconds since the last UP<->DOWN transition
954 24. downtime [..BS]: total downtime (in seconds). The value for the backend
955 is the downtime for the whole backend, not the sum of the server downtime.
956 25. qlimit [...S]: configured maxqueue for the server, or nothing in the
957 value is 0 (default, meaning no limit)
958 26. pid [LFBS]: process id (0 for first instance, 1 for second, ...)
959 27. iid [LFBS]: unique proxy id
960 28. sid [L..S]: server id (unique inside a proxy)
961 29. throttle [...S]: current throttle percentage for the server, when
962 slowstart is active, or no value if not in slowstart.
963 30. lbtot [..BS]: total number of times a server was selected, either for new
964 sessions, or when re-dispatching. The server counter is the number
965 of times that server was selected.
966 31. tracked [...S]: id of proxy/server if tracking is enabled.
967 32. type [LFBS]: (0=frontend, 1=backend, 2=server, 3=socket/listener)
968 33. rate [.FBS]: number of sessions per second over last elapsed second
969 34. rate_lim [.F..]: configured limit on new sessions per second
970 35. rate_max [.FBS]: max number of new sessions per second
971 36. check_status [...S]: status of last health check, one of:
972 UNK -> unknown
973 INI -> initializing
974 SOCKERR -> socket error
975 L4OK -> check passed on layer 4, no upper layers testing enabled
976 L4TOUT -> layer 1-4 timeout
977 L4CON -> layer 1-4 connection problem, for example
978 "Connection refused" (tcp rst) or "No route to host" (icmp)
979 L6OK -> check passed on layer 6
980 L6TOUT -> layer 6 (SSL) timeout
981 L6RSP -> layer 6 invalid response - protocol error
982 L7OK -> check passed on layer 7
983 L7OKC -> check conditionally passed on layer 7, for example 404 with
984 disable-on-404
985 L7TOUT -> layer 7 (HTTP/SMTP) timeout
986 L7RSP -> layer 7 invalid response - protocol error
987 L7STS -> layer 7 response error, for example HTTP 5xx
988 37. check_code [...S]: layer5-7 code, if available
989 38. check_duration [...S]: time in ms took to finish last health check
990 39. hrsp_1xx [.FBS]: http responses with 1xx code
991 40. hrsp_2xx [.FBS]: http responses with 2xx code
992 41. hrsp_3xx [.FBS]: http responses with 3xx code
993 42. hrsp_4xx [.FBS]: http responses with 4xx code
994 43. hrsp_5xx [.FBS]: http responses with 5xx code
995 44. hrsp_other [.FBS]: http responses with other codes (protocol error)
996 45. hanafail [...S]: failed health checks details
997 46. req_rate [.F..]: HTTP requests per second over last elapsed second
998 47. req_rate_max [.F..]: max number of HTTP requests per second observed
Willy Tarreaufb981bd2016-12-12 14:31:46 +0100999 48. req_tot [.FB.]: total number of HTTP requests received
Willy Tarreau44aed902015-10-13 14:45:29 +02001000 49. cli_abrt [..BS]: number of data transfers aborted by the client
1001 50. srv_abrt [..BS]: number of data transfers aborted by the server
1002 (inc. in eresp)
1003 51. comp_in [.FB.]: number of HTTP response bytes fed to the compressor
1004 52. comp_out [.FB.]: number of HTTP response bytes emitted by the compressor
1005 53. comp_byp [.FB.]: number of bytes that bypassed the HTTP compressor
1006 (CPU/BW limit)
1007 54. comp_rsp [.FB.]: number of HTTP responses that were compressed
1008 55. lastsess [..BS]: number of seconds since last session assigned to
1009 server/backend
1010 56. last_chk [...S]: last health check contents or textual error
1011 57. last_agt [...S]: last agent check contents or textual error
1012 58. qtime [..BS]: the average queue time in ms over the 1024 last requests
1013 59. ctime [..BS]: the average connect time in ms over the 1024 last requests
1014 60. rtime [..BS]: the average response time in ms over the 1024 last requests
1015 (0 for TCP)
1016 61. ttime [..BS]: the average total session time in ms over the 1024 last
1017 requests
Willy Tarreau7f618842016-01-08 11:40:03 +01001018 62. agent_status [...S]: status of last agent check, one of:
1019 UNK -> unknown
1020 INI -> initializing
1021 SOCKERR -> socket error
1022 L4OK -> check passed on layer 4, no upper layers testing enabled
1023 L4TOUT -> layer 1-4 timeout
1024 L4CON -> layer 1-4 connection problem, for example
1025 "Connection refused" (tcp rst) or "No route to host" (icmp)
1026 L7OK -> agent reported "up"
1027 L7STS -> agent reported "fail", "stop", or "down"
1028 63. agent_code [...S]: numeric code reported by agent if any (unused for now)
1029 64. agent_duration [...S]: time in ms taken to finish last check
Willy Tarreaudd7354b2016-01-08 13:47:26 +01001030 65. check_desc [...S]: short human-readable description of check_status
1031 66. agent_desc [...S]: short human-readable description of agent_status
Willy Tarreau3141f592016-01-08 14:25:28 +01001032 67. check_rise [...S]: server's "rise" parameter used by checks
1033 68. check_fall [...S]: server's "fall" parameter used by checks
1034 69. check_health [...S]: server's health check value between 0 and rise+fall-1
1035 70. agent_rise [...S]: agent's "rise" parameter, normally 1
1036 71. agent_fall [...S]: agent's "fall" parameter, normally 1
1037 72. agent_health [...S]: agent's health parameter, between 0 and rise+fall-1
Willy Tarreaua6f5a732016-01-08 16:59:56 +01001038 73. addr [L..S]: address:port or "unix". IPv6 has brackets around the address.
Willy Tarreaue4847c62016-01-08 15:43:54 +01001039 74: cookie [..BS]: server's cookie value or backend's cookie name
Willy Tarreauf8211df2016-01-11 14:09:38 +01001040 75: mode [LFBS]: proxy mode (tcp, http, health, unknown)
Willy Tarreauf1516d92016-01-11 14:48:36 +01001041 76: algo [..B.]: load balancing algorithm
Willy Tarreauc73810f2016-01-11 13:52:04 +01001042 77: conn_rate [.F..]: number of connections over the last elapsed second
1043 78: conn_rate_max [.F..]: highest known conn_rate
1044 79: conn_tot [.F..]: cumulative number of connections
Willy Tarreau5b9bdff2016-01-11 14:40:47 +01001045 80: intercepted [.FB.]: cum. number of intercepted requests (monitor, stats)
Willy Tarreau8a90b8e2016-10-21 18:15:32 +02001046 81: dcon [LF..]: requests denied by "tcp-request connection" rules
Willy Tarreaua5bc36b2016-10-21 18:16:27 +02001047 82: dses [LF..]: requests denied by "tcp-request session" rules
Willy Tarreau44aed902015-10-13 14:45:29 +02001048
1049
Willy Tarreau5d8b9792016-03-11 11:09:34 +010010509.2) Typed output format
1051------------------------
1052
1053Both "show info" and "show stat" support a mode where each output value comes
1054with its type and sufficient information to know how the value is supposed to
1055be aggregated between processes and how it evolves.
1056
1057In all cases, the output consists in having a single value per line with all
1058the information split into fields delimited by colons (':').
1059
1060The first column designates the object or metric being dumped. Its format is
1061specific to the command producing this output and will not be described in this
1062section. Usually it will consist in a series of identifiers and field names.
1063
1064The second column contains 3 characters respectively indicating the origin, the
1065nature and the scope of the value being reported. The first character (the
1066origin) indicates where the value was extracted from. Possible characters are :
1067
1068 M The value is a metric. It is valid at one instant any may change depending
1069 on its nature .
1070
1071 S The value is a status. It represents a discrete value which by definition
1072 cannot be aggregated. It may be the status of a server ("UP" or "DOWN"),
1073 the PID of the process, etc.
1074
1075 K The value is a sorting key. It represents an identifier which may be used
1076 to group some values together because it is unique among its class. All
1077 internal identifiers are keys. Some names can be listed as keys if they
1078 are unique (eg: a frontend name is unique). In general keys come from the
Dan Lloyd8e48b872016-07-01 21:01:18 -04001079 configuration, even though some of them may automatically be assigned. For
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001080 most purposes keys may be considered as equivalent to configuration.
1081
1082 C The value comes from the configuration. Certain configuration values make
1083 sense on the output, for example a concurrent connection limit or a cookie
1084 name. By definition these values are the same in all processes started
1085 from the same configuration file.
1086
1087 P The value comes from the product itself. There are very few such values,
1088 most common use is to report the product name, version and release date.
1089 These elements are also the same between all processes.
1090
1091The second character (the nature) indicates the nature of the information
1092carried by the field in order to let an aggregator decide on what operation to
1093use to aggregate multiple values. Possible characters are :
1094
1095 A The value represents an age since a last event. This is a bit different
1096 from the duration in that an age is automatically computed based on the
1097 current date. A typical example is how long ago did the last session
1098 happen on a server. Ages are generally aggregated by taking the minimum
1099 value and do not need to be stored.
1100
1101 a The value represents an already averaged value. The average response times
1102 and server weights are of this nature. Averages can typically be averaged
1103 between processes.
1104
1105 C The value represents a cumulative counter. Such measures perpetually
1106 increase until they wrap around. Some monitoring protocols need to tell
1107 the difference between a counter and a gauge to report a different type.
1108 In general counters may simply be summed since they represent events or
1109 volumes. Examples of metrics of this nature are connection counts or byte
1110 counts.
1111
1112 D The value represents a duration for a status. There are a few usages of
1113 this, most of them include the time taken by the last health check and
1114 the time a server has spent down. Durations are generally not summed,
1115 most of the time the maximum will be retained to compute an SLA.
1116
1117 G The value represents a gauge. It's a measure at one instant. The memory
1118 usage or the current number of active connections are of this nature.
1119 Metrics of this type are typically summed during aggregation.
1120
1121 L The value represents a limit (generally a configured one). By nature,
1122 limits are harder to aggregate since they are specific to the point where
1123 they were retrieved. In certain situations they may be summed or be kept
1124 separate.
1125
1126 M The value represents a maximum. In general it will apply to a gauge and
1127 keep the highest known value. An example of such a metric could be the
1128 maximum amount of concurrent connections that was encountered in the
1129 product's life time. To correctly aggregate maxima, you are supposed to
1130 output a range going from the maximum of all maxima and the sum of all
1131 of them. There is indeed no way to know if they were encountered
1132 simultaneously or not.
1133
1134 m The value represents a minimum. In general it will apply to a gauge and
1135 keep the lowest known value. An example of such a metric could be the
1136 minimum amount of free memory pools that was encountered in the product's
1137 life time. To correctly aggregate minima, you are supposed to output a
1138 range going from the minimum of all minima and the sum of all of them.
1139 There is indeed no way to know if they were encountered simultaneously
1140 or not.
1141
1142 N The value represents a name, so it is a string. It is used to report
1143 proxy names, server names and cookie names. Names have configuration or
1144 keys as their origin and are supposed to be the same among all processes.
1145
1146 O The value represents a free text output. Outputs from various commands,
1147 returns from health checks, node descriptions are of such nature.
1148
1149 R The value represents an event rate. It's a measure at one instant. It is
1150 quite similar to a gauge except that the recipient knows that this measure
1151 moves slowly and may decide not to keep all values. An example of such a
1152 metric is the measured amount of connections per second. Metrics of this
1153 type are typically summed during aggregation.
1154
1155 T The value represents a date or time. A field emitting the current date
1156 would be of this type. The method to aggregate such information is left
1157 as an implementation choice. For now no field uses this type.
1158
1159The third character (the scope) indicates what extent the value reflects. Some
1160elements may be per process while others may be per configuration or per system.
1161The distinction is important to know whether or not a single value should be
1162kept during aggregation or if values have to be aggregated. The following
1163characters are currently supported :
1164
1165 C The value is valid for a whole cluster of nodes, which is the set of nodes
1166 communicating over the peers protocol. An example could be the amount of
1167 entries present in a stick table that is replicated with other peers. At
1168 the moment no metric use this scope.
1169
1170 P The value is valid only for the process reporting it. Most metrics use
1171 this scope.
1172
1173 S The value is valid for the whole service, which is the set of processes
1174 started together from the same configuration file. All metrics originating
1175 from the configuration use this scope. Some other metrics may use it as
1176 well for some shared resources (eg: shared SSL cache statistics).
1177
1178 s The value is valid for the whole system, such as the system's hostname,
1179 current date or resource usage. At the moment this scope is not used by
1180 any metric.
1181
1182Consumers of these information will generally have enough of these 3 characters
1183to determine how to accurately report aggregated information across multiple
1184processes.
1185
1186After this column, the third column indicates the type of the field, among "s32"
1187(signed 32-bit integer), "s64" (signed 64-bit integer), "u32" (unsigned 32-bit
1188integer), "u64" (unsigned 64-bit integer), "str" (string). It is important to
1189know the type before parsing the value in order to properly read it. For example
1190a string containing only digits is still a string an not an integer (eg: an
1191error code extracted by a check).
1192
1193Then the fourth column is the value itself, encoded according to its type.
1194Strings are dumped as-is immediately after the colon without any leading space.
1195If a string contains a colon, it will appear normally. This means that the
1196output should not be exclusively split around colons or some check outputs
1197or server addresses might be truncated.
1198
1199
12009.3. Unix Socket commands
Willy Tarreau44aed902015-10-13 14:45:29 +02001201-------------------------
1202
1203The stats socket is not enabled by default. In order to enable it, it is
1204necessary to add one line in the global section of the haproxy configuration.
1205A second line is recommended to set a larger timeout, always appreciated when
1206issuing commands by hand :
1207
1208 global
1209 stats socket /var/run/haproxy.sock mode 600 level admin
1210 stats timeout 2m
1211
1212It is also possible to add multiple instances of the stats socket by repeating
1213the line, and make them listen to a TCP port instead of a UNIX socket. This is
1214never done by default because this is dangerous, but can be handy in some
1215situations :
1216
1217 global
1218 stats socket /var/run/haproxy.sock mode 600 level admin
1219 stats socket ipv4@192.168.0.1:9999 level admin
1220 stats timeout 2m
1221
1222To access the socket, an external utility such as "socat" is required. Socat is
1223a swiss-army knife to connect anything to anything. We use it to connect
1224terminals to the socket, or a couple of stdin/stdout pipes to it for scripts.
1225The two main syntaxes we'll use are the following :
1226
1227 # socat /var/run/haproxy.sock stdio
1228 # socat /var/run/haproxy.sock readline
1229
1230The first one is used with scripts. It is possible to send the output of a
1231script to haproxy, and pass haproxy's output to another script. That's useful
1232for retrieving counters or attack traces for example.
1233
1234The second one is only useful for issuing commands by hand. It has the benefit
1235that the terminal is handled by the readline library which supports line
1236editing and history, which is very convenient when issuing repeated commands
1237(eg: watch a counter).
1238
1239The socket supports two operation modes :
1240 - interactive
1241 - non-interactive
1242
1243The non-interactive mode is the default when socat connects to the socket. In
1244this mode, a single line may be sent. It is processed as a whole, responses are
1245sent back, and the connection closes after the end of the response. This is the
1246mode that scripts and monitoring tools use. It is possible to send multiple
1247commands in this mode, they need to be delimited by a semi-colon (';'). For
1248example :
1249
1250 # echo "show info;show stat;show table" | socat /var/run/haproxy stdio
1251
Dragan Dosena1c35ab2016-11-24 11:33:12 +01001252If a command needs to use a semi-colon or a backslash (eg: in a value), it
1253must be preceeded by a backslash ('\').
Chad Lavoiee3f50312016-05-26 16:42:25 -04001254
Willy Tarreau44aed902015-10-13 14:45:29 +02001255The interactive mode displays a prompt ('>') and waits for commands to be
1256entered on the line, then processes them, and displays the prompt again to wait
1257for a new command. This mode is entered via the "prompt" command which must be
1258sent on the first line in non-interactive mode. The mode is a flip switch, if
1259"prompt" is sent in interactive mode, it is disabled and the connection closes
1260after processing the last command of the same line.
1261
1262For this reason, when debugging by hand, it's quite common to start with the
1263"prompt" command :
1264
1265 # socat /var/run/haproxy readline
1266 prompt
1267 > show info
1268 ...
1269 >
1270
1271Since multiple commands may be issued at once, haproxy uses the empty line as a
1272delimiter to mark an end of output for each command, and takes care of ensuring
1273that no command can emit an empty line on output. A script can thus easily
1274parse the output even when multiple commands were pipelined on a single line.
1275
1276It is important to understand that when multiple haproxy processes are started
1277on the same sockets, any process may pick up the request and will output its
1278own stats.
1279
1280The list of commands currently supported on the stats socket is provided below.
1281If an unknown command is sent, haproxy displays the usage message which reminds
1282all supported commands. Some commands support a more complex syntax, generally
1283it will explain what part of the command is invalid when this happens.
1284
1285add acl <acl> <pattern>
1286 Add an entry into the acl <acl>. <acl> is the #<id> or the <file> returned by
1287 "show acl". This command does not verify if the entry already exists. This
1288 command cannot be used if the reference <acl> is a file also used with a map.
1289 In this case, you must use the command "add map" in place of "add acl".
1290
1291add map <map> <key> <value>
1292 Add an entry into the map <map> to associate the value <value> to the key
1293 <key>. This command does not verify if the entry already exists. It is
1294 mainly used to fill a map after a clear operation. Note that if the reference
1295 <map> is a file and is shared with a map, this map will contain also a new
1296 pattern entry.
1297
1298clear counters
1299 Clear the max values of the statistics counters in each proxy (frontend &
Dan Lloyd8e48b872016-07-01 21:01:18 -04001300 backend) and in each server. The accumulated counters are not affected. This
Willy Tarreau44aed902015-10-13 14:45:29 +02001301 can be used to get clean counters after an incident, without having to
1302 restart nor to clear traffic counters. This command is restricted and can
1303 only be issued on sockets configured for levels "operator" or "admin".
1304
1305clear counters all
1306 Clear all statistics counters in each proxy (frontend & backend) and in each
1307 server. This has the same effect as restarting. This command is restricted
1308 and can only be issued on sockets configured for level "admin".
1309
1310clear acl <acl>
1311 Remove all entries from the acl <acl>. <acl> is the #<id> or the <file>
1312 returned by "show acl". Note that if the reference <acl> is a file and is
1313 shared with a map, this map will be also cleared.
1314
1315clear map <map>
1316 Remove all entries from the map <map>. <map> is the #<id> or the <file>
1317 returned by "show map". Note that if the reference <map> is a file and is
1318 shared with a acl, this acl will be also cleared.
1319
1320clear table <table> [ data.<type> <operator> <value> ] | [ key <key> ]
1321 Remove entries from the stick-table <table>.
1322
1323 This is typically used to unblock some users complaining they have been
1324 abusively denied access to a service, but this can also be used to clear some
1325 stickiness entries matching a server that is going to be replaced (see "show
1326 table" below for details). Note that sometimes, removal of an entry will be
1327 refused because it is currently tracked by a session. Retrying a few seconds
1328 later after the session ends is usual enough.
1329
1330 In the case where no options arguments are given all entries will be removed.
1331
1332 When the "data." form is used entries matching a filter applied using the
1333 stored data (see "stick-table" in section 4.2) are removed. A stored data
1334 type must be specified in <type>, and this data type must be stored in the
1335 table otherwise an error is reported. The data is compared according to
1336 <operator> with the 64-bit integer <value>. Operators are the same as with
1337 the ACLs :
1338
1339 - eq : match entries whose data is equal to this value
1340 - ne : match entries whose data is not equal to this value
1341 - le : match entries whose data is less than or equal to this value
1342 - ge : match entries whose data is greater than or equal to this value
1343 - lt : match entries whose data is less than this value
1344 - gt : match entries whose data is greater than this value
1345
1346 When the key form is used the entry <key> is removed. The key must be of the
1347 same type as the table, which currently is limited to IPv4, IPv6, integer and
1348 string.
1349
1350 Example :
1351 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1352 >>> # table: http_proxy, type: ip, size:204800, used:2
1353 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
1354 bytes_out_rate(60000)=187
1355 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
1356 bytes_out_rate(60000)=191
1357
1358 $ echo "clear table http_proxy key 127.0.0.1" | socat stdio /tmp/sock1
1359
1360 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1361 >>> # table: http_proxy, type: ip, size:204800, used:1
1362 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
1363 bytes_out_rate(60000)=191
1364 $ echo "clear table http_proxy data.gpc0 eq 1" | socat stdio /tmp/sock1
1365 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1366 >>> # table: http_proxy, type: ip, size:204800, used:1
1367
1368del acl <acl> [<key>|#<ref>]
1369 Delete all the acl entries from the acl <acl> corresponding to the key <key>.
1370 <acl> is the #<id> or the <file> returned by "show acl". If the <ref> is used,
1371 this command delete only the listed reference. The reference can be found with
1372 listing the content of the acl. Note that if the reference <acl> is a file and
1373 is shared with a map, the entry will be also deleted in the map.
1374
1375del map <map> [<key>|#<ref>]
1376 Delete all the map entries from the map <map> corresponding to the key <key>.
1377 <map> is the #<id> or the <file> returned by "show map". If the <ref> is used,
1378 this command delete only the listed reference. The reference can be found with
1379 listing the content of the map. Note that if the reference <map> is a file and
1380 is shared with a acl, the entry will be also deleted in the map.
1381
1382disable agent <backend>/<server>
1383 Mark the auxiliary agent check as temporarily stopped.
1384
1385 In the case where an agent check is being run as a auxiliary check, due
1386 to the agent-check parameter of a server directive, new checks are only
Dan Lloyd8e48b872016-07-01 21:01:18 -04001387 initialized when the agent is in the enabled. Thus, disable agent will
Willy Tarreau44aed902015-10-13 14:45:29 +02001388 prevent any new agent checks from begin initiated until the agent
1389 re-enabled using enable agent.
1390
1391 When an agent is disabled the processing of an auxiliary agent check that
1392 was initiated while the agent was set as enabled is as follows: All
1393 results that would alter the weight, specifically "drain" or a weight
1394 returned by the agent, are ignored. The processing of agent check is
1395 otherwise unchanged.
1396
1397 The motivation for this feature is to allow the weight changing effects
1398 of the agent checks to be paused to allow the weight of a server to be
1399 configured using set weight without being overridden by the agent.
1400
1401 This command is restricted and can only be issued on sockets configured for
1402 level "admin".
1403
1404disable frontend <frontend>
1405 Mark the frontend as temporarily stopped. This corresponds to the mode which
1406 is used during a soft restart : the frontend releases the port but can be
1407 enabled again if needed. This should be used with care as some non-Linux OSes
1408 are unable to enable it back. This is intended to be used in environments
1409 where stopping a proxy is not even imaginable but a misconfigured proxy must
1410 be fixed. That way it's possible to release the port and bind it into another
1411 process to restore operations. The frontend will appear with status "STOP"
1412 on the stats page.
1413
1414 The frontend may be specified either by its name or by its numeric ID,
1415 prefixed with a sharp ('#').
1416
1417 This command is restricted and can only be issued on sockets configured for
1418 level "admin".
1419
1420disable health <backend>/<server>
1421 Mark the primary health check as temporarily stopped. This will disable
1422 sending of health checks, and the last health check result will be ignored.
1423 The server will be in unchecked state and considered UP unless an auxiliary
1424 agent check forces it down.
1425
1426 This command is restricted and can only be issued on sockets configured for
1427 level "admin".
1428
1429disable server <backend>/<server>
1430 Mark the server DOWN for maintenance. In this mode, no more checks will be
1431 performed on the server until it leaves maintenance.
1432 If the server is tracked by other servers, those servers will be set to DOWN
1433 during the maintenance.
1434
1435 In the statistics page, a server DOWN for maintenance will appear with a
1436 "MAINT" status, its tracking servers with the "MAINT(via)" one.
1437
1438 Both the backend and the server may be specified either by their name or by
1439 their numeric ID, prefixed with a sharp ('#').
1440
1441 This command is restricted and can only be issued on sockets configured for
1442 level "admin".
1443
1444enable agent <backend>/<server>
1445 Resume auxiliary agent check that was temporarily stopped.
1446
1447 See "disable agent" for details of the effect of temporarily starting
1448 and stopping an auxiliary agent.
1449
1450 This command is restricted and can only be issued on sockets configured for
1451 level "admin".
1452
1453enable frontend <frontend>
1454 Resume a frontend which was temporarily stopped. It is possible that some of
1455 the listening ports won't be able to bind anymore (eg: if another process
1456 took them since the 'disable frontend' operation). If this happens, an error
1457 is displayed. Some operating systems might not be able to resume a frontend
1458 which was disabled.
1459
1460 The frontend may be specified either by its name or by its numeric ID,
1461 prefixed with a sharp ('#').
1462
1463 This command is restricted and can only be issued on sockets configured for
1464 level "admin".
1465
1466enable health <backend>/<server>
1467 Resume a primary health check that was temporarily stopped. This will enable
1468 sending of health checks again. Please see "disable health" for details.
1469
1470 This command is restricted and can only be issued on sockets configured for
1471 level "admin".
1472
1473enable server <backend>/<server>
1474 If the server was previously marked as DOWN for maintenance, this marks the
1475 server UP and checks are re-enabled.
1476
1477 Both the backend and the server may be specified either by their name or by
1478 their numeric ID, prefixed with a sharp ('#').
1479
1480 This command is restricted and can only be issued on sockets configured for
1481 level "admin".
1482
1483get map <map> <value>
1484get acl <acl> <value>
1485 Lookup the value <value> in the map <map> or in the ACL <acl>. <map> or <acl>
1486 are the #<id> or the <file> returned by "show map" or "show acl". This command
1487 returns all the matching patterns associated with this map. This is useful for
1488 debugging maps and ACLs. The output format is composed by one line par
1489 matching type. Each line is composed by space-delimited series of words.
1490
1491 The first two words are:
1492
1493 <match method>: The match method applied. It can be "found", "bool",
1494 "int", "ip", "bin", "len", "str", "beg", "sub", "dir",
1495 "dom", "end" or "reg".
1496
1497 <match result>: The result. Can be "match" or "no-match".
1498
1499 The following words are returned only if the pattern matches an entry.
1500
1501 <index type>: "tree" or "list". The internal lookup algorithm.
1502
1503 <case>: "case-insensitive" or "case-sensitive". The
1504 interpretation of the case.
1505
1506 <entry matched>: match="<entry>". Return the matched pattern. It is
1507 useful with regular expressions.
1508
1509 The two last word are used to show the returned value and its type. With the
1510 "acl" case, the pattern doesn't exist.
1511
1512 return=nothing: No return because there are no "map".
1513 return="<value>": The value returned in the string format.
1514 return=cannot-display: The value cannot be converted as string.
1515
1516 type="<type>": The type of the returned sample.
1517
1518get weight <backend>/<server>
1519 Report the current weight and the initial weight of server <server> in
1520 backend <backend> or an error if either doesn't exist. The initial weight is
1521 the one that appears in the configuration file. Both are normally equal
1522 unless the current weight has been changed. Both the backend and the server
1523 may be specified either by their name or by their numeric ID, prefixed with a
1524 sharp ('#').
1525
1526help
1527 Print the list of known keywords and their basic usage. The same help screen
1528 is also displayed for unknown commands.
1529
1530prompt
1531 Toggle the prompt at the beginning of the line and enter or leave interactive
1532 mode. In interactive mode, the connection is not closed after a command
1533 completes. Instead, the prompt will appear again, indicating the user that
1534 the interpreter is waiting for a new command. The prompt consists in a right
1535 angle bracket followed by a space "> ". This mode is particularly convenient
1536 when one wants to periodically check information such as stats or errors.
1537 It is also a good idea to enter interactive mode before issuing a "help"
1538 command.
1539
1540quit
1541 Close the connection when in interactive mode.
1542
1543set map <map> [<key>|#<ref>] <value>
1544 Modify the value corresponding to each key <key> in a map <map>. <map> is the
1545 #<id> or <file> returned by "show map". If the <ref> is used in place of
1546 <key>, only the entry pointed by <ref> is changed. The new value is <value>.
1547
1548set maxconn frontend <frontend> <value>
1549 Dynamically change the specified frontend's maxconn setting. Any positive
1550 value is allowed including zero, but setting values larger than the global
1551 maxconn does not make much sense. If the limit is increased and connections
1552 were pending, they will immediately be accepted. If it is lowered to a value
1553 below the current number of connections, new connections acceptation will be
1554 delayed until the threshold is reached. The frontend might be specified by
1555 either its name or its numeric ID prefixed with a sharp ('#').
1556
Andrew Hayworthedb93a72015-10-27 21:46:25 +00001557set maxconn server <backend/server> <value>
1558 Dynamically change the specified server's maxconn setting. Any positive
1559 value is allowed including zero, but setting values larger than the global
1560 maxconn does not make much sense.
1561
Willy Tarreau44aed902015-10-13 14:45:29 +02001562set maxconn global <maxconn>
1563 Dynamically change the global maxconn setting within the range defined by the
1564 initial global maxconn setting. If it is increased and connections were
1565 pending, they will immediately be accepted. If it is lowered to a value below
1566 the current number of connections, new connections acceptation will be
1567 delayed until the threshold is reached. A value of zero restores the initial
1568 setting.
1569
1570set rate-limit connections global <value>
1571 Change the process-wide connection rate limit, which is set by the global
1572 'maxconnrate' setting. A value of zero disables the limitation. This limit
1573 applies to all frontends and the change has an immediate effect. The value
1574 is passed in number of connections per second.
1575
1576set rate-limit http-compression global <value>
1577 Change the maximum input compression rate, which is set by the global
1578 'maxcomprate' setting. A value of zero disables the limitation. The value is
1579 passed in number of kilobytes per second. The value is available in the "show
1580 info" on the line "CompressBpsRateLim" in bytes.
1581
1582set rate-limit sessions global <value>
1583 Change the process-wide session rate limit, which is set by the global
1584 'maxsessrate' setting. A value of zero disables the limitation. This limit
1585 applies to all frontends and the change has an immediate effect. The value
1586 is passed in number of sessions per second.
1587
1588set rate-limit ssl-sessions global <value>
1589 Change the process-wide SSL session rate limit, which is set by the global
1590 'maxsslrate' setting. A value of zero disables the limitation. This limit
1591 applies to all frontends and the change has an immediate effect. The value
1592 is passed in number of sessions per second sent to the SSL stack. It applies
1593 before the handshake in order to protect the stack against handshake abuses.
1594
Baptiste Assmann3749ebf2016-08-03 22:34:12 +02001595set server <backend>/<server> addr <ip4 or ip6 address> [port <port>]
Willy Tarreau44aed902015-10-13 14:45:29 +02001596 Replace the current IP address of a server by the one provided.
Baptiste Assmann3749ebf2016-08-03 22:34:12 +02001597 Optionnaly, the port can be changed using the 'port' parameter.
1598 Note that changing the port also support switching from/to port mapping
1599 (notation with +X or -Y), only if a port is configured for the health check.
Willy Tarreau44aed902015-10-13 14:45:29 +02001600
1601set server <backend>/<server> agent [ up | down ]
1602 Force a server's agent to a new state. This can be useful to immediately
1603 switch a server's state regardless of some slow agent checks for example.
1604 Note that the change is propagated to tracking servers if any.
1605
1606set server <backend>/<server> health [ up | stopping | down ]
1607 Force a server's health to a new state. This can be useful to immediately
1608 switch a server's state regardless of some slow health checks for example.
1609 Note that the change is propagated to tracking servers if any.
1610
Baptiste Assmann50946562016-08-31 23:26:29 +02001611set server <backend>/<server> check-port <port>
1612 Change the port used for health checking to <port>
1613
Willy Tarreau44aed902015-10-13 14:45:29 +02001614set server <backend>/<server> state [ ready | drain | maint ]
1615 Force a server's administrative state to a new state. This can be useful to
1616 disable load balancing and/or any traffic to a server. Setting the state to
1617 "ready" puts the server in normal mode, and the command is the equivalent of
1618 the "enable server" command. Setting the state to "maint" disables any traffic
1619 to the server as well as any health checks. This is the equivalent of the
1620 "disable server" command. Setting the mode to "drain" only removes the server
1621 from load balancing but still allows it to be checked and to accept new
1622 persistent connections. Changes are propagated to tracking servers if any.
1623
1624set server <backend>/<server> weight <weight>[%]
1625 Change a server's weight to the value passed in argument. This is the exact
1626 equivalent of the "set weight" command below.
1627
1628set ssl ocsp-response <response>
1629 This command is used to update an OCSP Response for a certificate (see "crt"
1630 on "bind" lines). Same controls are performed as during the initial loading of
1631 the response. The <response> must be passed as a base64 encoded string of the
1632 DER encoded response from the OCSP server.
1633
1634 Example:
1635 openssl ocsp -issuer issuer.pem -cert server.pem \
1636 -host ocsp.issuer.com:80 -respout resp.der
1637 echo "set ssl ocsp-response $(base64 -w 10000 resp.der)" | \
1638 socat stdio /var/run/haproxy.stat
1639
1640set ssl tls-key <id> <tlskey>
1641 Set the next TLS key for the <id> listener to <tlskey>. This key becomes the
1642 ultimate key, while the penultimate one is used for encryption (others just
1643 decrypt). The oldest TLS key present is overwritten. <id> is either a numeric
1644 #<id> or <file> returned by "show tls-keys". <tlskey> is a base64 encoded 48
1645 bit TLS ticket key (ex. openssl rand -base64 48).
1646
1647set table <table> key <key> [data.<data_type> <value>]*
1648 Create or update a stick-table entry in the table. If the key is not present,
1649 an entry is inserted. See stick-table in section 4.2 to find all possible
1650 values for <data_type>. The most likely use consists in dynamically entering
1651 entries for source IP addresses, with a flag in gpc0 to dynamically block an
1652 IP address or affect its quality of service. It is possible to pass multiple
1653 data_types in a single call.
1654
1655set timeout cli <delay>
1656 Change the CLI interface timeout for current connection. This can be useful
1657 during long debugging sessions where the user needs to constantly inspect
1658 some indicators without being disconnected. The delay is passed in seconds.
1659
1660set weight <backend>/<server> <weight>[%]
1661 Change a server's weight to the value passed in argument. If the value ends
1662 with the '%' sign, then the new weight will be relative to the initially
1663 configured weight. Absolute weights are permitted between 0 and 256.
1664 Relative weights must be positive with the resulting absolute weight is
1665 capped at 256. Servers which are part of a farm running a static
1666 load-balancing algorithm have stricter limitations because the weight
1667 cannot change once set. Thus for these servers, the only accepted values
1668 are 0 and 100% (or 0 and the initial weight). Changes take effect
1669 immediately, though certain LB algorithms require a certain amount of
1670 requests to consider changes. A typical usage of this command is to
1671 disable a server during an update by setting its weight to zero, then to
1672 enable it again after the update by setting it back to 100%. This command
1673 is restricted and can only be issued on sockets configured for level
1674 "admin". Both the backend and the server may be specified either by their
1675 name or by their numeric ID, prefixed with a sharp ('#').
1676
William Lallemand51132162016-12-16 16:38:58 +01001677show cli sockets
1678 List CLI sockets. The output format is composed of 3 fields separated by
1679 spaces. The first field is the socket address, it can be a unix socket, a
1680 ipv4 address:port couple or a ipv6 one. Socket of other types won't be dump.
1681 The second field describe the level of the socket: 'admin', 'user' or
1682 'operator'. The last field list the processes on which the socket is bound,
1683 separated by commas, it can be numbers or 'all'.
1684
1685 Example :
1686
1687 $ echo 'show cli sockets' | socat stdio /tmp/sock1
1688 # socket lvl processes
1689 /tmp/sock1 admin all
1690 127.0.0.1:9999 user 2,3,4
1691 127.0.0.2:9969 user 2
1692 [::1]:9999 operator 2
1693
Willy Tarreauae795722016-02-16 11:27:28 +01001694show env [<name>]
1695 Dump one or all environment variables known by the process. Without any
1696 argument, all variables are dumped. With an argument, only the specified
1697 variable is dumped if it exists. Otherwise "Variable not found" is emitted.
1698 Variables are dumped in the same format as they are stored or returned by the
1699 "env" utility, that is, "<name>=<value>". This can be handy when debugging
1700 certain configuration files making heavy use of environment variables to
1701 ensure that they contain the expected values. This command is restricted and
1702 can only be issued on sockets configured for levels "operator" or "admin".
1703
Willy Tarreau35069f82016-11-25 09:16:37 +01001704show errors [<iid>|<proxy>] [request|response]
Willy Tarreau44aed902015-10-13 14:45:29 +02001705 Dump last known request and response errors collected by frontends and
1706 backends. If <iid> is specified, the limit the dump to errors concerning
Willy Tarreau234ba2d2016-11-25 08:39:10 +01001707 either frontend or backend whose ID is <iid>. Proxy ID "-1" will cause
1708 all instances to be dumped. If a proxy name is specified instead, its ID
Willy Tarreau35069f82016-11-25 09:16:37 +01001709 will be used as the filter. If "request" or "response" is added after the
1710 proxy name or ID, only request or response errors will be dumped. This
1711 command is restricted and can only be issued on sockets configured for
1712 levels "operator" or "admin".
Willy Tarreau44aed902015-10-13 14:45:29 +02001713
1714 The errors which may be collected are the last request and response errors
1715 caused by protocol violations, often due to invalid characters in header
1716 names. The report precisely indicates what exact character violated the
1717 protocol. Other important information such as the exact date the error was
1718 detected, frontend and backend names, the server name (when known), the
1719 internal session ID and the source address which has initiated the session
1720 are reported too.
1721
1722 All characters are returned, and non-printable characters are encoded. The
1723 most common ones (\t = 9, \n = 10, \r = 13 and \e = 27) are encoded as one
1724 letter following a backslash. The backslash itself is encoded as '\\' to
1725 avoid confusion. Other non-printable characters are encoded '\xNN' where
1726 NN is the two-digits hexadecimal representation of the character's ASCII
1727 code.
1728
1729 Lines are prefixed with the position of their first character, starting at 0
1730 for the beginning of the buffer. At most one input line is printed per line,
1731 and large lines will be broken into multiple consecutive output lines so that
1732 the output never goes beyond 79 characters wide. It is easy to detect if a
1733 line was broken, because it will not end with '\n' and the next line's offset
1734 will be followed by a '+' sign, indicating it is a continuation of previous
1735 line.
1736
1737 Example :
Willy Tarreau35069f82016-11-25 09:16:37 +01001738 $ echo "show errors -1 response" | socat stdio /tmp/sock1
Willy Tarreau44aed902015-10-13 14:45:29 +02001739 >>> [04/Mar/2009:15:46:56.081] backend http-in (#2) : invalid response
1740 src 127.0.0.1, session #54, frontend fe-eth0 (#1), server s2 (#1)
1741 response length 213 bytes, error at position 23:
1742
1743 00000 HTTP/1.0 200 OK\r\n
1744 00017 header/bizarre:blah\r\n
1745 00038 Location: blah\r\n
1746 00054 Long-line: this is a very long line which should b
1747 00104+ e broken into multiple lines on the output buffer,
1748 00154+ otherwise it would be too large to print in a ter
1749 00204+ minal\r\n
1750 00211 \r\n
1751
1752 In the example above, we see that the backend "http-in" which has internal
1753 ID 2 has blocked an invalid response from its server s2 which has internal
1754 ID 1. The request was on session 54 initiated by source 127.0.0.1 and
1755 received by frontend fe-eth0 whose ID is 1. The total response length was
1756 213 bytes when the error was detected, and the error was at byte 23. This
1757 is the slash ('/') in header name "header/bizarre", which is not a valid
1758 HTTP character for a header name.
1759
1760show backend
1761 Dump the list of backends available in the running process
1762
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001763show info [typed]
1764 Dump info about haproxy status on current process. If "typed" is passed as an
1765 optional argument, field numbers, names and types are emitted as well so that
1766 external monitoring products can easily retrieve, possibly aggregate, then
1767 report information found in fields they don't know. Each field is dumped on
1768 its own line. By default, the format contains only two columns delimited by a
1769 colon (':'). The left one is the field name and the right one is the value.
1770 It is very important to note that in typed output format, the dump for a
Dan Lloyd8e48b872016-07-01 21:01:18 -04001771 single object is contiguous so that there is no need for a consumer to store
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001772 everything at once.
1773
1774 When using the typed output format, each line is made of 4 columns delimited
1775 by colons (':'). The first column is a dot-delimited series of 3 elements. The
1776 first element is the numeric position of the field in the list (starting at
1777 zero). This position shall not change over time, but holes are to be expected,
1778 depending on build options or if some fields are deleted in the future. The
1779 second element is the field name as it appears in the default "show info"
1780 output. The third element is the relative process number starting at 1.
1781
1782 The rest of the line starting after the first colon follows the "typed output
1783 format" described in the section above. In short, the second column (after the
1784 first ':') indicates the origin, nature and scope of the variable. The third
1785 column indicates the type of the field, among "s32", "s64", "u32", "u64" and
1786 "str". Then the fourth column is the value itself, which the consumer knows
1787 how to parse thanks to column 3 and how to process thanks to column 2.
1788
1789 Thus the overall line format in typed mode is :
1790
1791 <field_pos>.<field_name>.<process_num>:<tags>:<type>:<value>
1792
1793 Example :
1794
1795 > show info
1796 Name: HAProxy
1797 Version: 1.7-dev1-de52ea-146
1798 Release_date: 2016/03/11
1799 Nbproc: 1
1800 Process_num: 1
1801 Pid: 28105
1802 Uptime: 0d 0h00m04s
1803 Uptime_sec: 4
1804 Memmax_MB: 0
1805 PoolAlloc_MB: 0
1806 PoolUsed_MB: 0
1807 PoolFailed: 0
1808 (...)
1809
1810 > show info typed
1811 0.Name.1:POS:str:HAProxy
1812 1.Version.1:POS:str:1.7-dev1-de52ea-146
1813 2.Release_date.1:POS:str:2016/03/11
1814 3.Nbproc.1:CGS:u32:1
1815 4.Process_num.1:KGP:u32:1
1816 5.Pid.1:SGP:u32:28105
1817 6.Uptime.1:MDP:str:0d 0h00m08s
1818 7.Uptime_sec.1:MDP:u32:8
1819 8.Memmax_MB.1:CLP:u32:0
1820 9.PoolAlloc_MB.1:MGP:u32:0
1821 10.PoolUsed_MB.1:MGP:u32:0
1822 11.PoolFailed.1:MCP:u32:0
1823 (...)
1824
Simon Horman1084a362016-11-21 17:00:24 +01001825 In the typed format, the presence of the process ID at the end of the
1826 first column makes it very easy to visually aggregate outputs from
1827 multiple processes.
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001828 Example :
1829
1830 $ ( echo show info typed | socat /var/run/haproxy.sock1 ; \
1831 echo show info typed | socat /var/run/haproxy.sock2 ) | \
1832 sort -t . -k 1,1n -k 2,2 -k 3,3n
1833 0.Name.1:POS:str:HAProxy
1834 0.Name.2:POS:str:HAProxy
1835 1.Version.1:POS:str:1.7-dev1-868ab3-148
1836 1.Version.2:POS:str:1.7-dev1-868ab3-148
1837 2.Release_date.1:POS:str:2016/03/11
1838 2.Release_date.2:POS:str:2016/03/11
1839 3.Nbproc.1:CGS:u32:2
1840 3.Nbproc.2:CGS:u32:2
1841 4.Process_num.1:KGP:u32:1
1842 4.Process_num.2:KGP:u32:2
1843 5.Pid.1:SGP:u32:30120
1844 5.Pid.2:SGP:u32:30121
1845 6.Uptime.1:MDP:str:0d 0h01m28s
1846 6.Uptime.2:MDP:str:0d 0h01m28s
1847 (...)
Willy Tarreau44aed902015-10-13 14:45:29 +02001848
1849show map [<map>]
1850 Dump info about map converters. Without argument, the list of all available
1851 maps is returned. If a <map> is specified, its contents are dumped. <map> is
1852 the #<id> or <file>. The first column is a unique identifier. It can be used
1853 as reference for the operation "del map" and "set map". The second column is
1854 the pattern and the third column is the sample if available. The data returned
1855 are not directly a list of available maps, but are the list of all patterns
1856 composing any map. Many of these patterns can be shared with ACL.
1857
1858show acl [<acl>]
1859 Dump info about acl converters. Without argument, the list of all available
1860 acls is returned. If a <acl> is specified, its contents are dumped. <acl> if
1861 the #<id> or <file>. The dump format is the same than the map even for the
1862 sample value. The data returned are not a list of available ACL, but are the
1863 list of all patterns composing any ACL. Many of these patterns can be shared
1864 with maps.
1865
1866show pools
1867 Dump the status of internal memory pools. This is useful to track memory
1868 usage when suspecting a memory leak for example. It does exactly the same
1869 as the SIGQUIT when running in foreground except that it does not flush
1870 the pools.
1871
1872show servers state [<backend>]
1873 Dump the state of the servers found in the running configuration. A backend
1874 name or identifier may be provided to limit the output to this backend only.
1875
1876 The dump has the following format:
1877 - first line contains the format version (1 in this specification);
1878 - second line contains the column headers, prefixed by a sharp ('#');
1879 - third line and next ones contain data;
1880 - each line starting by a sharp ('#') is considered as a comment.
1881
Dan Lloyd8e48b872016-07-01 21:01:18 -04001882 Since multiple versions of the output may co-exist, below is the list of
Willy Tarreau44aed902015-10-13 14:45:29 +02001883 fields and their order per file format version :
1884 1:
1885 be_id: Backend unique id.
1886 be_name: Backend label.
1887 srv_id: Server unique id (in the backend).
1888 srv_name: Server label.
1889 srv_addr: Server IP address.
1890 srv_op_state: Server operational state (UP/DOWN/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01001891 0 = SRV_ST_STOPPED
1892 The server is down.
1893 1 = SRV_ST_STARTING
1894 The server is warming up (up but
1895 throttled).
1896 2 = SRV_ST_RUNNING
1897 The server is fully up.
1898 3 = SRV_ST_STOPPING
1899 The server is up but soft-stopping
1900 (eg: 404).
Willy Tarreau44aed902015-10-13 14:45:29 +02001901 srv_admin_state: Server administrative state (MAINT/DRAIN/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01001902 The state is actually a mask of values :
1903 0x01 = SRV_ADMF_FMAINT
1904 The server was explicitly forced into
1905 maintenance.
1906 0x02 = SRV_ADMF_IMAINT
1907 The server has inherited the maintenance
1908 status from a tracked server.
1909 0x04 = SRV_ADMF_CMAINT
1910 The server is in maintenance because of
1911 the configuration.
1912 0x08 = SRV_ADMF_FDRAIN
1913 The server was explicitly forced into
1914 drain state.
1915 0x10 = SRV_ADMF_IDRAIN
1916 The server has inherited the drain status
1917 from a tracked server.
Baptiste Assmann89aa7f32016-11-02 21:31:27 +01001918 0x20 = SRV_ADMF_RMAINT
1919 The server is in maintenance because of an
1920 IP address resolution failure.
Willy Tarreau44aed902015-10-13 14:45:29 +02001921 srv_uweight: User visible server's weight.
1922 srv_iweight: Server's initial weight.
1923 srv_time_since_last_change: Time since last operational change.
1924 srv_check_status: Last health check status.
1925 srv_check_result: Last check result (FAILED/PASSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01001926 0 = CHK_RES_UNKNOWN
1927 Initialized to this by default.
1928 1 = CHK_RES_NEUTRAL
1929 Valid check but no status information.
1930 2 = CHK_RES_FAILED
1931 Check failed.
1932 3 = CHK_RES_PASSED
1933 Check succeeded and server is fully up
1934 again.
1935 4 = CHK_RES_CONDPASS
1936 Check reports the server doesn't want new
1937 sessions.
Willy Tarreau44aed902015-10-13 14:45:29 +02001938 srv_check_health: Checks rise / fall current counter.
1939 srv_check_state: State of the check (ENABLED/PAUSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01001940 The state is actually a mask of values :
1941 0x01 = CHK_ST_INPROGRESS
1942 A check is currently running.
1943 0x02 = CHK_ST_CONFIGURED
1944 This check is configured and may be
1945 enabled.
1946 0x04 = CHK_ST_ENABLED
1947 This check is currently administratively
1948 enabled.
1949 0x08 = CHK_ST_PAUSED
1950 Checks are paused because of maintenance
1951 (health only).
Willy Tarreau44aed902015-10-13 14:45:29 +02001952 srv_agent_state: State of the agent check (ENABLED/PAUSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01001953 This state uses the same mask values as
1954 "srv_check_state", adding this specific one :
1955 0x10 = CHK_ST_AGENT
1956 Check is an agent check (otherwise it's a
1957 health check).
Willy Tarreau44aed902015-10-13 14:45:29 +02001958 bk_f_forced_id: Flag to know if the backend ID is forced by
1959 configuration.
1960 srv_f_forced_id: Flag to know if the server's ID is forced by
1961 configuration.
1962
1963show sess
1964 Dump all known sessions. Avoid doing this on slow connections as this can
1965 be huge. This command is restricted and can only be issued on sockets
1966 configured for levels "operator" or "admin".
1967
1968show sess <id>
1969 Display a lot of internal information about the specified session identifier.
1970 This identifier is the first field at the beginning of the lines in the dumps
1971 of "show sess" (it corresponds to the session pointer). Those information are
1972 useless to most users but may be used by haproxy developers to troubleshoot a
1973 complex bug. The output format is intentionally not documented so that it can
1974 freely evolve depending on demands. You may find a description of all fields
1975 returned in src/dumpstats.c
1976
1977 The special id "all" dumps the states of all sessions, which must be avoided
1978 as much as possible as it is highly CPU intensive and can take a lot of time.
1979
Willy Tarreaua1b1ed52016-11-25 08:50:58 +01001980show stat [{<iid>|<proxy>} <type> <sid>] [typed]
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001981 Dump statistics using the CSV format, or using the extended typed output
1982 format described in the section above if "typed" is passed after the other
1983 arguments. By passing <id>, <type> and <sid>, it is possible to dump only
1984 selected items :
Willy Tarreaua1b1ed52016-11-25 08:50:58 +01001985 - <iid> is a proxy ID, -1 to dump everything. Alternatively, a proxy name
1986 <proxy> may be specified. In this case, this proxy's ID will be used as
1987 the ID selector.
Willy Tarreau44aed902015-10-13 14:45:29 +02001988 - <type> selects the type of dumpable objects : 1 for frontends, 2 for
1989 backends, 4 for servers, -1 for everything. These values can be ORed,
1990 for example:
1991 1 + 2 = 3 -> frontend + backend.
1992 1 + 2 + 4 = 7 -> frontend + backend + server.
1993 - <sid> is a server ID, -1 to dump everything from the selected proxy.
1994
1995 Example :
1996 $ echo "show info;show stat" | socat stdio unix-connect:/tmp/sock1
1997 >>> Name: HAProxy
1998 Version: 1.4-dev2-49
1999 Release_date: 2009/09/23
2000 Nbproc: 1
2001 Process_num: 1
2002 (...)
2003
2004 # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq, (...)
2005 stats,FRONTEND,,,0,0,1000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,1,0, (...)
2006 stats,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,250,(...)
2007 (...)
2008 www1,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,0,250, (...)
2009
2010 $
2011
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002012 In this example, two commands have been issued at once. That way it's easy to
2013 find which process the stats apply to in multi-process mode. This is not
2014 needed in the typed output format as the process number is reported on each
2015 line. Notice the empty line after the information output which marks the end
2016 of the first block. A similar empty line appears at the end of the second
2017 block (stats) so that the reader knows the output has not been truncated.
2018
2019 When "typed" is specified, the output format is more suitable to monitoring
2020 tools because it provides numeric positions and indicates the type of each
2021 output field. Each value stands on its own line with process number, element
2022 number, nature, origin and scope. This same format is available via the HTTP
2023 stats by passing ";typed" after the URI. It is very important to note that in
Dan Lloyd8e48b872016-07-01 21:01:18 -04002024 typed output format, the dump for a single object is contiguous so that there
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002025 is no need for a consumer to store everything at once.
2026
2027 When using the typed output format, each line is made of 4 columns delimited
2028 by colons (':'). The first column is a dot-delimited series of 5 elements. The
2029 first element is a letter indicating the type of the object being described.
2030 At the moment the following object types are known : 'F' for a frontend, 'B'
2031 for a backend, 'L' for a listener, and 'S' for a server. The second element
2032 The second element is a positive integer representing the unique identifier of
2033 the proxy the object belongs to. It is equivalent to the "iid" column of the
2034 CSV output and matches the value in front of the optional "id" directive found
2035 in the frontend or backend section. The third element is a positive integer
2036 containing the unique object identifier inside the proxy, and corresponds to
2037 the "sid" column of the CSV output. ID 0 is reported when dumping a frontend
2038 or a backend. For a listener or a server, this corresponds to their respective
2039 ID inside the proxy. The fourth element is the numeric position of the field
2040 in the list (starting at zero). This position shall not change over time, but
2041 holes are to be expected, depending on build options or if some fields are
2042 deleted in the future. The fifth element is the field name as it appears in
2043 the CSV output. The sixth element is a positive integer and is the relative
2044 process number starting at 1.
2045
2046 The rest of the line starting after the first colon follows the "typed output
2047 format" described in the section above. In short, the second column (after the
2048 first ':') indicates the origin, nature and scope of the variable. The third
2049 column indicates the type of the field, among "s32", "s64", "u32", "u64" and
2050 "str". Then the fourth column is the value itself, which the consumer knows
2051 how to parse thanks to column 3 and how to process thanks to column 2.
2052
2053 Thus the overall line format in typed mode is :
2054
2055 <obj>.<px_id>.<id>.<fpos>.<fname>.<process_num>:<tags>:<type>:<value>
2056
2057 Here's an example of typed output format :
2058
2059 $ echo "show stat typed" | socat stdio unix-connect:/tmp/sock1
2060 F.2.0.0.pxname.1:MGP:str:private-frontend
2061 F.2.0.1.svname.1:MGP:str:FRONTEND
2062 F.2.0.8.bin.1:MGP:u64:0
2063 F.2.0.9.bout.1:MGP:u64:0
2064 F.2.0.40.hrsp_2xx.1:MGP:u64:0
2065 L.2.1.0.pxname.1:MGP:str:private-frontend
2066 L.2.1.1.svname.1:MGP:str:sock-1
2067 L.2.1.17.status.1:MGP:str:OPEN
2068 L.2.1.73.addr.1:MGP:str:0.0.0.0:8001
2069 S.3.13.60.rtime.1:MCP:u32:0
2070 S.3.13.61.ttime.1:MCP:u32:0
2071 S.3.13.62.agent_status.1:MGP:str:L4TOUT
2072 S.3.13.64.agent_duration.1:MGP:u64:2001
2073 S.3.13.65.check_desc.1:MCP:str:Layer4 timeout
2074 S.3.13.66.agent_desc.1:MCP:str:Layer4 timeout
2075 S.3.13.67.check_rise.1:MCP:u32:2
2076 S.3.13.68.check_fall.1:MCP:u32:3
2077 S.3.13.69.check_health.1:SGP:u32:0
2078 S.3.13.70.agent_rise.1:MaP:u32:1
2079 S.3.13.71.agent_fall.1:SGP:u32:1
2080 S.3.13.72.agent_health.1:SGP:u32:1
2081 S.3.13.73.addr.1:MCP:str:1.255.255.255:8888
2082 S.3.13.75.mode.1:MAP:str:http
2083 B.3.0.0.pxname.1:MGP:str:private-backend
2084 B.3.0.1.svname.1:MGP:str:BACKEND
2085 B.3.0.2.qcur.1:MGP:u32:0
2086 B.3.0.3.qmax.1:MGP:u32:0
2087 B.3.0.4.scur.1:MGP:u32:0
2088 B.3.0.5.smax.1:MGP:u32:0
2089 B.3.0.6.slim.1:MGP:u32:1000
2090 B.3.0.55.lastsess.1:MMP:s32:-1
2091 (...)
2092
Simon Horman1084a362016-11-21 17:00:24 +01002093 In the typed format, the presence of the process ID at the end of the
2094 first column makes it very easy to visually aggregate outputs from
2095 multiple processes, as show in the example below where each line appears
2096 for each process :
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002097
2098 $ ( echo show stat typed | socat /var/run/haproxy.sock1 - ; \
2099 echo show stat typed | socat /var/run/haproxy.sock2 - ) | \
2100 sort -t . -k 1,1 -k 2,2n -k 3,3n -k 4,4n -k 5,5 -k 6,6n
2101 B.3.0.0.pxname.1:MGP:str:private-backend
2102 B.3.0.0.pxname.2:MGP:str:private-backend
2103 B.3.0.1.svname.1:MGP:str:BACKEND
2104 B.3.0.1.svname.2:MGP:str:BACKEND
2105 B.3.0.2.qcur.1:MGP:u32:0
2106 B.3.0.2.qcur.2:MGP:u32:0
2107 B.3.0.3.qmax.1:MGP:u32:0
2108 B.3.0.3.qmax.2:MGP:u32:0
2109 B.3.0.4.scur.1:MGP:u32:0
2110 B.3.0.4.scur.2:MGP:u32:0
2111 B.3.0.5.smax.1:MGP:u32:0
2112 B.3.0.5.smax.2:MGP:u32:0
2113 B.3.0.6.slim.1:MGP:u32:1000
2114 B.3.0.6.slim.2:MGP:u32:1000
2115 (...)
Willy Tarreau44aed902015-10-13 14:45:29 +02002116
2117show stat resolvers [<resolvers section id>]
2118 Dump statistics for the given resolvers section, or all resolvers sections
2119 if no section is supplied.
2120
2121 For each name server, the following counters are reported:
2122 sent: number of DNS requests sent to this server
2123 valid: number of DNS valid responses received from this server
2124 update: number of DNS responses used to update the server's IP address
2125 cname: number of CNAME responses
2126 cname_error: CNAME errors encountered with this server
2127 any_err: number of empty response (IE: server does not support ANY type)
2128 nx: non existent domain response received from this server
2129 timeout: how many time this server did not answer in time
2130 refused: number of requests refused by this server
2131 other: any other DNS errors
2132 invalid: invalid DNS response (from a protocol point of view)
2133 too_big: too big response
2134 outdated: number of response arrived too late (after an other name server)
2135
2136show table
2137 Dump general information on all known stick-tables. Their name is returned
2138 (the name of the proxy which holds them), their type (currently zero, always
2139 IP), their size in maximum possible number of entries, and the number of
2140 entries currently in use.
2141
2142 Example :
2143 $ echo "show table" | socat stdio /tmp/sock1
2144 >>> # table: front_pub, type: ip, size:204800, used:171454
2145 >>> # table: back_rdp, type: ip, size:204800, used:0
2146
2147show table <name> [ data.<type> <operator> <value> ] | [ key <key> ]
2148 Dump contents of stick-table <name>. In this mode, a first line of generic
2149 information about the table is reported as with "show table", then all
2150 entries are dumped. Since this can be quite heavy, it is possible to specify
2151 a filter in order to specify what entries to display.
2152
2153 When the "data." form is used the filter applies to the stored data (see
2154 "stick-table" in section 4.2). A stored data type must be specified
2155 in <type>, and this data type must be stored in the table otherwise an
2156 error is reported. The data is compared according to <operator> with the
2157 64-bit integer <value>. Operators are the same as with the ACLs :
2158
2159 - eq : match entries whose data is equal to this value
2160 - ne : match entries whose data is not equal to this value
2161 - le : match entries whose data is less than or equal to this value
2162 - ge : match entries whose data is greater than or equal to this value
2163 - lt : match entries whose data is less than this value
2164 - gt : match entries whose data is greater than this value
2165
2166
2167 When the key form is used the entry <key> is shown. The key must be of the
2168 same type as the table, which currently is limited to IPv4, IPv6, integer,
2169 and string.
2170
2171 Example :
2172 $ echo "show table http_proxy" | socat stdio /tmp/sock1
2173 >>> # table: http_proxy, type: ip, size:204800, used:2
2174 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
2175 bytes_out_rate(60000)=187
2176 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2177 bytes_out_rate(60000)=191
2178
2179 $ echo "show table http_proxy data.gpc0 gt 0" | socat stdio /tmp/sock1
2180 >>> # table: http_proxy, type: ip, size:204800, used:2
2181 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2182 bytes_out_rate(60000)=191
2183
2184 $ echo "show table http_proxy data.conn_rate gt 5" | \
2185 socat stdio /tmp/sock1
2186 >>> # table: http_proxy, type: ip, size:204800, used:2
2187 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2188 bytes_out_rate(60000)=191
2189
2190 $ echo "show table http_proxy key 127.0.0.2" | \
2191 socat stdio /tmp/sock1
2192 >>> # table: http_proxy, type: ip, size:204800, used:2
2193 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2194 bytes_out_rate(60000)=191
2195
2196 When the data criterion applies to a dynamic value dependent on time such as
2197 a bytes rate, the value is dynamically computed during the evaluation of the
2198 entry in order to decide whether it has to be dumped or not. This means that
2199 such a filter could match for some time then not match anymore because as
2200 time goes, the average event rate drops.
2201
2202 It is possible to use this to extract lists of IP addresses abusing the
2203 service, in order to monitor them or even blacklist them in a firewall.
2204 Example :
2205 $ echo "show table http_proxy data.gpc0 gt 0" \
2206 | socat stdio /tmp/sock1 \
2207 | fgrep 'key=' | cut -d' ' -f2 | cut -d= -f2 > abusers-ip.txt
2208 ( or | awk '/key/{ print a[split($2,a,"=")]; }' )
2209
William Lallemandbb933462016-05-31 21:09:53 +02002210show tls-keys [id|*]
2211 Dump all loaded TLS ticket keys references. The TLS ticket key reference ID
2212 and the file from which the keys have been loaded is shown. Both of those
2213 can be used to update the TLS keys using "set ssl tls-key". If an ID is
2214 specified as parameter, it will dump the tickets, using * it will dump every
2215 keys from every references.
Willy Tarreau44aed902015-10-13 14:45:29 +02002216
2217shutdown frontend <frontend>
2218 Completely delete the specified frontend. All the ports it was bound to will
2219 be released. It will not be possible to enable the frontend anymore after
2220 this operation. This is intended to be used in environments where stopping a
2221 proxy is not even imaginable but a misconfigured proxy must be fixed. That
2222 way it's possible to release the port and bind it into another process to
2223 restore operations. The frontend will not appear at all on the stats page
2224 once it is terminated.
2225
2226 The frontend may be specified either by its name or by its numeric ID,
2227 prefixed with a sharp ('#').
2228
2229 This command is restricted and can only be issued on sockets configured for
2230 level "admin".
2231
2232shutdown session <id>
2233 Immediately terminate the session matching the specified session identifier.
2234 This identifier is the first field at the beginning of the lines in the dumps
2235 of "show sess" (it corresponds to the session pointer). This can be used to
2236 terminate a long-running session without waiting for a timeout or when an
2237 endless transfer is ongoing. Such terminated sessions are reported with a 'K'
2238 flag in the logs.
2239
2240shutdown sessions server <backend>/<server>
2241 Immediately terminate all the sessions attached to the specified server. This
2242 can be used to terminate long-running sessions after a server is put into
2243 maintenance mode, for instance. Such terminated sessions are reported with a
2244 'K' flag in the logs.
2245
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002246
224710. Tricks for easier configuration management
2248----------------------------------------------
2249
2250It is very common that two HAProxy nodes constituting a cluster share exactly
2251the same configuration modulo a few addresses. Instead of having to maintain a
2252duplicate configuration for each node, which will inevitably diverge, it is
2253possible to include environment variables in the configuration. Thus multiple
2254configuration may share the exact same file with only a few different system
2255wide environment variables. This started in version 1.5 where only addresses
2256were allowed to include environment variables, and 1.6 goes further by
2257supporting environment variables everywhere. The syntax is the same as in the
2258UNIX shell, a variable starts with a dollar sign ('$'), followed by an opening
2259curly brace ('{'), then the variable name followed by the closing brace ('}').
2260Except for addresses, environment variables are only interpreted in arguments
2261surrounded with double quotes (this was necessary not to break existing setups
2262using regular expressions involving the dollar symbol).
2263
2264Environment variables also make it convenient to write configurations which are
2265expected to work on various sites where only the address changes. It can also
2266permit to remove passwords from some configs. Example below where the the file
2267"site1.env" file is sourced by the init script upon startup :
2268
2269 $ cat site1.env
2270 LISTEN=192.168.1.1
2271 CACHE_PFX=192.168.11
2272 SERVER_PFX=192.168.22
2273 LOGGER=192.168.33.1
2274 STATSLP=admin:pa$$w0rd
2275 ABUSERS=/etc/haproxy/abuse.lst
2276 TIMEOUT=10s
2277
2278 $ cat haproxy.cfg
2279 global
2280 log "${LOGGER}:514" local0
2281
2282 defaults
2283 mode http
2284 timeout client "${TIMEOUT}"
2285 timeout server "${TIMEOUT}"
2286 timeout connect 5s
2287
2288 frontend public
2289 bind "${LISTEN}:80"
2290 http-request reject if { src -f "${ABUSERS}" }
2291 stats uri /stats
2292 stats auth "${STATSLP}"
2293 use_backend cache if { path_end .jpg .css .ico }
2294 default_backend server
2295
2296 backend cache
2297 server cache1 "${CACHE_PFX}.1:18080" check
2298 server cache2 "${CACHE_PFX}.2:18080" check
2299
2300 backend server
2301 server cache1 "${SERVER_PFX}.1:8080" check
2302 server cache2 "${SERVER_PFX}.2:8080" check
2303
2304
230511. Well-known traps to avoid
2306-----------------------------
2307
2308Once in a while, someone reports that after a system reboot, the haproxy
2309service wasn't started, and that once they start it by hand it works. Most
2310often, these people are running a clustered IP address mechanism such as
2311keepalived, to assign the service IP address to the master node only, and while
2312it used to work when they used to bind haproxy to address 0.0.0.0, it stopped
2313working after they bound it to the virtual IP address. What happens here is
2314that when the service starts, the virtual IP address is not yet owned by the
2315local node, so when HAProxy wants to bind to it, the system rejects this
2316because it is not a local IP address. The fix doesn't consist in delaying the
2317haproxy service startup (since it wouldn't stand a restart), but instead to
2318properly configure the system to allow binding to non-local addresses. This is
2319easily done on Linux by setting the net.ipv4.ip_nonlocal_bind sysctl to 1. This
2320is also needed in order to transparently intercept the IP traffic that passes
2321through HAProxy for a specific target address.
2322
2323Multi-process configurations involving source port ranges may apparently seem
2324to work but they will cause some random failures under high loads because more
2325than one process may try to use the same source port to connect to the same
2326server, which is not possible. The system will report an error and a retry will
2327happen, picking another port. A high value in the "retries" parameter may hide
2328the effect to a certain extent but this also comes with increased CPU usage and
2329processing time. Logs will also report a certain number of retries. For this
2330reason, port ranges should be avoided in multi-process configurations.
2331
Dan Lloyd8e48b872016-07-01 21:01:18 -04002332Since HAProxy uses SO_REUSEPORT and supports having multiple independent
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002333processes bound to the same IP:port, during troubleshooting it can happen that
2334an old process was not stopped before a new one was started. This provides
2335absurd test results which tend to indicate that any change to the configuration
2336is ignored. The reason is that in fact even the new process is restarted with a
2337new configuration, the old one also gets some incoming connections and
2338processes them, returning unexpected results. When in doubt, just stop the new
2339process and try again. If it still works, it very likely means that an old
2340process remains alive and has to be stopped. Linux's "netstat -lntp" is of good
2341help here.
2342
2343When adding entries to an ACL from the command line (eg: when blacklisting a
2344source address), it is important to keep in mind that these entries are not
2345synchronized to the file and that if someone reloads the configuration, these
2346updates will be lost. While this is often the desired effect (for blacklisting)
2347it may not necessarily match expectations when the change was made as a fix for
2348a problem. See the "add acl" action of the CLI interface.
2349
2350
235112. Debugging and performance issues
2352------------------------------------
2353
2354When HAProxy is started with the "-d" option, it will stay in the foreground
2355and will print one line per event, such as an incoming connection, the end of a
2356connection, and for each request or response header line seen. This debug
2357output is emitted before the contents are processed, so they don't consider the
2358local modifications. The main use is to show the request and response without
2359having to run a network sniffer. The output is less readable when multiple
2360connections are handled in parallel, though the "debug2ansi" and "debug2html"
2361scripts found in the examples/ directory definitely help here by coloring the
2362output.
2363
2364If a request or response is rejected because HAProxy finds it is malformed, the
2365best thing to do is to connect to the CLI and issue "show errors", which will
2366report the last captured faulty request and response for each frontend and
2367backend, with all the necessary information to indicate precisely the first
2368character of the input stream that was rejected. This is sometimes needed to
2369prove to customers or to developers that a bug is present in their code. In
2370this case it is often possible to relax the checks (but still keep the
2371captures) using "option accept-invalid-http-request" or its equivalent for
2372responses coming from the server "option accept-invalid-http-response". Please
2373see the configuration manual for more details.
2374
2375Example :
2376
2377 > show errors
2378 Total events captured on [13/Oct/2015:13:43:47.169] : 1
2379
2380 [13/Oct/2015:13:43:40.918] frontend HAProxyLocalStats (#2): invalid request
2381 backend <NONE> (#-1), server <NONE> (#-1), event #0
2382 src 127.0.0.1:51981, session #0, session flags 0x00000080
2383 HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000
2384 HTTP chunk len 0 bytes, HTTP body len 0 bytes
2385 buffer flags 0x00808002, out 0 bytes, total 31 bytes
2386 pending 31 bytes, wrapping at 8040, error at position 13:
2387
2388 00000 GET /invalid request HTTP/1.1\r\n
2389
2390
2391The output of "show info" on the CLI provides a number of useful information
2392regarding the maximum connection rate ever reached, maximum SSL key rate ever
2393reached, and in general all information which can help to explain temporary
2394issues regarding CPU or memory usage. Example :
2395
2396 > show info
2397 Name: HAProxy
2398 Version: 1.6-dev7-e32d18-17
2399 Release_date: 2015/10/12
2400 Nbproc: 1
2401 Process_num: 1
2402 Pid: 7949
2403 Uptime: 0d 0h02m39s
2404 Uptime_sec: 159
2405 Memmax_MB: 0
2406 Ulimit-n: 120032
2407 Maxsock: 120032
2408 Maxconn: 60000
2409 Hard_maxconn: 60000
2410 CurrConns: 0
2411 CumConns: 3
2412 CumReq: 3
2413 MaxSslConns: 0
2414 CurrSslConns: 0
2415 CumSslConns: 0
2416 Maxpipes: 0
2417 PipesUsed: 0
2418 PipesFree: 0
2419 ConnRate: 0
2420 ConnRateLimit: 0
2421 MaxConnRate: 1
2422 SessRate: 0
2423 SessRateLimit: 0
2424 MaxSessRate: 1
2425 SslRate: 0
2426 SslRateLimit: 0
2427 MaxSslRate: 0
2428 SslFrontendKeyRate: 0
2429 SslFrontendMaxKeyRate: 0
2430 SslFrontendSessionReuse_pct: 0
2431 SslBackendKeyRate: 0
2432 SslBackendMaxKeyRate: 0
2433 SslCacheLookups: 0
2434 SslCacheMisses: 0
2435 CompressBpsIn: 0
2436 CompressBpsOut: 0
2437 CompressBpsRateLim: 0
2438 ZlibMemUsage: 0
2439 MaxZlibMemUsage: 0
2440 Tasks: 5
2441 Run_queue: 1
2442 Idle_pct: 100
2443 node: wtap
2444 description:
2445
2446When an issue seems to randomly appear on a new version of HAProxy (eg: every
2447second request is aborted, occasional crash, etc), it is worth trying to enable
Dan Lloyd8e48b872016-07-01 21:01:18 -04002448memory poisoning so that each call to malloc() is immediately followed by the
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002449filling of the memory area with a configurable byte. By default this byte is
24500x50 (ASCII for 'P'), but any other byte can be used, including zero (which
2451will have the same effect as a calloc() and which may make issues disappear).
Dan Lloyd8e48b872016-07-01 21:01:18 -04002452Memory poisoning is enabled on the command line using the "-dM" option. It
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002453slightly hurts performance and is not recommended for use in production. If
Dan Lloyd8e48b872016-07-01 21:01:18 -04002454an issue happens all the time with it or never happens when poisoning uses
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002455byte zero, it clearly means you've found a bug and you definitely need to
2456report it. Otherwise if there's no clear change, the problem it is not related.
2457
2458When debugging some latency issues, it is important to use both strace and
2459tcpdump on the local machine, and another tcpdump on the remote system. The
2460reason for this is that there are delays everywhere in the processing chain and
2461it is important to know which one is causing latency to know where to act. In
2462practice, the local tcpdump will indicate when the input data come in. Strace
2463will indicate when haproxy receives these data (using recv/recvfrom). Warning,
2464openssl uses read()/write() syscalls instead of recv()/send(). Strace will also
2465show when haproxy sends the data, and tcpdump will show when the system sends
2466these data to the interface. Then the external tcpdump will show when the data
2467sent are really received (since the local one only shows when the packets are
2468queued). The benefit of sniffing on the local system is that strace and tcpdump
2469will use the same reference clock. Strace should be used with "-tts200" to get
2470complete timestamps and report large enough chunks of data to read them.
2471Tcpdump should be used with "-nvvttSs0" to report full packets, real sequence
2472numbers and complete timestamps.
2473
2474In practice, received data are almost always immediately received by haproxy
2475(unless the machine has a saturated CPU or these data are invalid and not
2476delivered). If these data are received but not sent, it generally is because
2477the output buffer is saturated (ie: recipient doesn't consume the data fast
2478enough). This can be confirmed by seeing that the polling doesn't notify of
2479the ability to write on the output file descriptor for some time (it's often
2480easier to spot in the strace output when the data finally leave and then roll
2481back to see when the write event was notified). It generally matches an ACK
2482received from the recipient, and detected by tcpdump. Once the data are sent,
2483they may spend some time in the system doing nothing. Here again, the TCP
2484congestion window may be limited and not allow these data to leave, waiting for
2485an ACK to open the window. If the traffic is idle and the data take 40 ms or
2486200 ms to leave, it's a different issue (which is not an issue), it's the fact
2487that the Nagle algorithm prevents empty packets from leaving immediately, in
2488hope that they will be merged with subsequent data. HAProxy automatically
2489disables Nagle in pure TCP mode and in tunnels. However it definitely remains
2490enabled when forwarding an HTTP body (and this contributes to the performance
2491improvement there by reducing the number of packets). Some HTTP non-compliant
2492applications may be sensitive to the latency when delivering incomplete HTTP
2493response messages. In this case you will have to enable "option http-no-delay"
2494to disable Nagle in order to work around their design, keeping in mind that any
2495other proxy in the chain may similarly be impacted. If tcpdump reports that data
2496leave immediately but the other end doesn't see them quickly, it can mean there
Dan Lloyd8e48b872016-07-01 21:01:18 -04002497is a congested WAN link, a congested LAN with flow control enabled and
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002498preventing the data from leaving, or more commonly that HAProxy is in fact
2499running in a virtual machine and that for whatever reason the hypervisor has
2500decided that the data didn't need to be sent immediately. In virtualized
2501environments, latency issues are almost always caused by the virtualization
2502layer, so in order to save time, it's worth first comparing tcpdump in the VM
2503and on the external components. Any difference has to be credited to the
2504hypervisor and its accompanying drivers.
2505
2506When some TCP SACK segments are seen in tcpdump traces (using -vv), it always
2507means that the side sending them has got the proof of a lost packet. While not
2508seeing them doesn't mean there are no losses, seeing them definitely means the
2509network is lossy. Losses are normal on a network, but at a rate where SACKs are
2510not noticeable at the naked eye. If they appear a lot in the traces, it is
2511worth investigating exactly what happens and where the packets are lost. HTTP
2512doesn't cope well with TCP losses, which introduce huge latencies.
2513
2514The "netstat -i" command will report statistics per interface. An interface
2515where the Rx-Ovr counter grows indicates that the system doesn't have enough
2516resources to receive all incoming packets and that they're lost before being
2517processed by the network driver. Rx-Drp indicates that some received packets
2518were lost in the network stack because the application doesn't process them
2519fast enough. This can happen during some attacks as well. Tx-Drp means that
2520the output queues were full and packets had to be dropped. When using TCP it
Dan Lloyd8e48b872016-07-01 21:01:18 -04002521should be very rare, but will possibly indicate a saturated outgoing link.
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002522
2523
252413. Security considerations
2525---------------------------
2526
2527HAProxy is designed to run with very limited privileges. The standard way to
2528use it is to isolate it into a chroot jail and to drop its privileges to a
2529non-root user without any permissions inside this jail so that if any future
2530vulnerability were to be discovered, its compromise would not affect the rest
2531of the system.
2532
Dan Lloyd8e48b872016-07-01 21:01:18 -04002533In order to perform a chroot, it first needs to be started as a root user. It is
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002534pointless to build hand-made chroots to start the process there, these ones are
2535painful to build, are never properly maintained and always contain way more
2536bugs than the main file-system. And in case of compromise, the intruder can use
2537the purposely built file-system. Unfortunately many administrators confuse
2538"start as root" and "run as root", resulting in the uid change to be done prior
2539to starting haproxy, and reducing the effective security restrictions.
2540
2541HAProxy will need to be started as root in order to :
2542 - adjust the file descriptor limits
2543 - bind to privileged port numbers
2544 - bind to a specific network interface
2545 - transparently listen to a foreign address
2546 - isolate itself inside the chroot jail
2547 - drop to another non-privileged UID
2548
2549HAProxy may require to be run as root in order to :
2550 - bind to an interface for outgoing connections
2551 - bind to privileged source ports for outgoing connections
Dan Lloyd8e48b872016-07-01 21:01:18 -04002552 - transparently bind to a foreign address for outgoing connections
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002553
2554Most users will never need the "run as root" case. But the "start as root"
2555covers most usages.
2556
2557A safe configuration will have :
2558
2559 - a chroot statement pointing to an empty location without any access
2560 permissions. This can be prepared this way on the UNIX command line :
2561
2562 # mkdir /var/empty && chmod 0 /var/empty || echo "Failed"
2563
2564 and referenced like this in the HAProxy configuration's global section :
2565
2566 chroot /var/empty
2567
2568 - both a uid/user and gid/group statements in the global section :
2569
2570 user haproxy
2571 group haproxy
2572
2573 - a stats socket whose mode, uid and gid are set to match the user and/or
2574 group allowed to access the CLI so that nobody may access it :
2575
2576 stats socket /var/run/haproxy.stat uid hatop gid hatop mode 600
2577