blob: 27f86dbc8ce075acdab3630ecf50b21ac81e8416 [file] [log] [blame]
Willy Tarreau2212e6a2015-10-13 14:40:55 +02001 ------------------------
2 HAProxy Management Guide
3 ------------------------
Willy Tarreau0e658fb2016-11-25 16:55:50 +01004 version 1.8
Willy Tarreau2212e6a2015-10-13 14:40:55 +02005
6
7This document describes how to start, stop, manage, and troubleshoot HAProxy,
8as well as some known limitations and traps to avoid. It does not describe how
9to configure it (for this please read configuration.txt).
10
11Note to documentation contributors :
12 This document is formatted with 80 columns per line, with even number of
13 spaces for indentation and without tabs. Please follow these rules strictly
14 so that it remains easily printable everywhere. If you add sections, please
15 update the summary below for easier searching.
16
17
18Summary
19-------
20
211. Prerequisites
222. Quick reminder about HAProxy's architecture
233. Starting HAProxy
244. Stopping and restarting HAProxy
255. File-descriptor limitations
266. Memory management
277. CPU usage
288. Logging
299. Statistics and monitoring
Willy Tarreau44aed902015-10-13 14:45:29 +0200309.1. CSV format
Willy Tarreau5d8b9792016-03-11 11:09:34 +0100319.2. Typed output format
329.3. Unix Socket commands
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003310. Tricks for easier configuration management
3411. Well-known traps to avoid
3512. Debugging and performance issues
3613. Security considerations
37
38
391. Prerequisites
40----------------
41
42In this document it is assumed that the reader has sufficient administration
43skills on a UNIX-like operating system, uses the shell on a daily basis and is
44familiar with troubleshooting utilities such as strace and tcpdump.
45
46
472. Quick reminder about HAProxy's architecture
48----------------------------------------------
49
50HAProxy is a single-threaded, event-driven, non-blocking daemon. This means is
51uses event multiplexing to schedule all of its activities instead of relying on
52the system to schedule between multiple activities. Most of the time it runs as
53a single process, so the output of "ps aux" on a system will report only one
54"haproxy" process, unless a soft reload is in progress and an older process is
55finishing its job in parallel to the new one. It is thus always easy to trace
56its activity using the strace utility.
57
58HAProxy is designed to isolate itself into a chroot jail during startup, where
59it cannot perform any file-system access at all. This is also true for the
60libraries it depends on (eg: libc, libssl, etc). The immediate effect is that
61a running process will not be able to reload a configuration file to apply
62changes, instead a new process will be started using the updated configuration
63file. Some other less obvious effects are that some timezone files or resolver
64files the libc might attempt to access at run time will not be found, though
65this should generally not happen as they're not needed after startup. A nice
66consequence of this principle is that the HAProxy process is totally stateless,
67and no cleanup is needed after it's killed, so any killing method that works
68will do the right thing.
69
70HAProxy doesn't write log files, but it relies on the standard syslog protocol
71to send logs to a remote server (which is often located on the same system).
72
73HAProxy uses its internal clock to enforce timeouts, that is derived from the
74system's time but where unexpected drift is corrected. This is done by limiting
75the time spent waiting in poll() for an event, and measuring the time it really
76took. In practice it never waits more than one second. This explains why, when
77running strace over a completely idle process, periodic calls to poll() (or any
78of its variants) surrounded by two gettimeofday() calls are noticed. They are
79normal, completely harmless and so cheap that the load they imply is totally
80undetectable at the system scale, so there's nothing abnormal there. Example :
81
82 16:35:40.002320 gettimeofday({1442759740, 2605}, NULL) = 0
83 16:35:40.002942 epoll_wait(0, {}, 200, 1000) = 0
84 16:35:41.007542 gettimeofday({1442759741, 7641}, NULL) = 0
85 16:35:41.007998 gettimeofday({1442759741, 8114}, NULL) = 0
86 16:35:41.008391 epoll_wait(0, {}, 200, 1000) = 0
87 16:35:42.011313 gettimeofday({1442759742, 11411}, NULL) = 0
88
89HAProxy is a TCP proxy, not a router. It deals with established connections that
90have been validated by the kernel, and not with packets of any form nor with
91sockets in other states (eg: no SYN_RECV nor TIME_WAIT), though their existence
92may prevent it from binding a port. It relies on the system to accept incoming
93connections and to initiate outgoing connections. An immediate effect of this is
94that there is no relation between packets observed on the two sides of a
95forwarded connection, which can be of different size, numbers and even family.
96Since a connection may only be accepted from a socket in LISTEN state, all the
97sockets it is listening to are necessarily visible using the "netstat" utility
98to show listening sockets. Example :
99
100 # netstat -ltnp
101 Active Internet connections (only servers)
102 Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
103 tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1629/sshd
104 tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2847/haproxy
105 tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 2847/haproxy
106
107
1083. Starting HAProxy
109-------------------
110
111HAProxy is started by invoking the "haproxy" program with a number of arguments
112passed on the command line. The actual syntax is :
113
114 $ haproxy [<options>]*
115
116where [<options>]* is any number of options. An option always starts with '-'
117followed by one of more letters, and possibly followed by one or multiple extra
118arguments. Without any option, HAProxy displays the help page with a reminder
119about supported options. Available options may vary slightly based on the
120operating system. A fair number of these options overlap with an equivalent one
121if the "global" section. In this case, the command line always has precedence
122over the configuration file, so that the command line can be used to quickly
123enforce some settings without touching the configuration files. The current
124list of options is :
125
126 -- <cfgfile>* : all the arguments following "--" are paths to configuration
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200127 file/directory to be loaded and processed in the declaration order. It is
128 mostly useful when relying on the shell to load many files that are
129 numerically ordered. See also "-f". The difference between "--" and "-f" is
130 that one "-f" must be placed before each file name, while a single "--" is
131 needed before all file names. Both options can be used together, the
132 command line ordering still applies. When more than one file is specified,
133 each file must start on a section boundary, so the first keyword of each
134 file must be one of "global", "defaults", "peers", "listen", "frontend",
135 "backend", and so on. A file cannot contain just a server list for example.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200136
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200137 -f <cfgfile|cfgdir> : adds <cfgfile> to the list of configuration files to be
138 loaded. If <cfgdir> is a directory, all the files (and only files) it
Dan Lloyd8e48b872016-07-01 21:01:18 -0400139 contains are added in lexical order (using LC_COLLATE=C) to the list of
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200140 configuration files to be loaded ; only files with ".cfg" extension are
141 added, only non hidden files (not prefixed with ".") are added.
142 Configuration files are loaded and processed in their declaration order.
143 This option may be specified multiple times to load multiple files. See
144 also "--". The difference between "--" and "-f" is that one "-f" must be
145 placed before each file name, while a single "--" is needed before all file
146 names. Both options can be used together, the command line ordering still
147 applies. When more than one file is specified, each file must start on a
148 section boundary, so the first keyword of each file must be one of
149 "global", "defaults", "peers", "listen", "frontend", "backend", and so on.
150 A file cannot contain just a server list for example.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200151
152 -C <dir> : changes to directory <dir> before loading configuration
153 files. This is useful when using relative paths. Warning when using
154 wildcards after "--" which are in fact replaced by the shell before
155 starting haproxy.
156
157 -D : start as a daemon. The process detaches from the current terminal after
158 forking, and errors are not reported anymore in the terminal. It is
159 equivalent to the "daemon" keyword in the "global" section of the
160 configuration. It is recommended to always force it in any init script so
161 that a faulty configuration doesn't prevent the system from booting.
162
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200163 -L <name> : change the local peer name to <name>, which defaults to the local
164 hostname. This is used only with peers replication.
165
166 -N <limit> : sets the default per-proxy maxconn to <limit> instead of the
167 builtin default value (usually 2000). Only useful for debugging.
168
169 -V : enable verbose mode (disables quiet mode). Reverts the effect of "-q" or
170 "quiet".
171
William Lallemande202b1e2017-06-01 17:38:56 +0200172 -W : master-worker mode. It is equivalent to the "master-worker" keyword in
173 the "global" section of the configuration. This mode will launch a "master"
174 which will monitor the "workers". Using this mode, you can reload HAProxy
175 directly by sending a SIGUSR2 signal to the master. The master-worker mode
176 is compatible either with the foreground or daemon mode. It is
177 recommended to use this mode with multiprocess and systemd.
178
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200179 -c : only performs a check of the configuration files and exits before trying
180 to bind. The exit status is zero if everything is OK, or non-zero if an
181 error is encountered.
182
183 -d : enable debug mode. This disables daemon mode, forces the process to stay
184 in foreground and to show incoming and outgoing events. It is equivalent to
185 the "global" section's "debug" keyword. It must never be used in an init
186 script.
187
188 -dG : disable use of getaddrinfo() to resolve host names into addresses. It
189 can be used when suspecting that getaddrinfo() doesn't work as expected.
190 This option was made available because many bogus implementations of
191 getaddrinfo() exist on various systems and cause anomalies that are
192 difficult to troubleshoot.
193
Dan Lloyd8e48b872016-07-01 21:01:18 -0400194 -dM[<byte>] : forces memory poisoning, which means that each and every
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200195 memory region allocated with malloc() or pool_alloc2() will be filled with
196 <byte> before being passed to the caller. When <byte> is not specified, it
197 defaults to 0x50 ('P'). While this slightly slows down operations, it is
198 useful to reliably trigger issues resulting from missing initializations in
199 the code that cause random crashes. Note that -dM0 has the effect of
200 turning any malloc() into a calloc(). In any case if a bug appears or
201 disappears when using this option it means there is a bug in haproxy, so
202 please report it.
203
204 -dS : disable use of the splice() system call. It is equivalent to the
205 "global" section's "nosplice" keyword. This may be used when splice() is
206 suspected to behave improperly or to cause performance issues, or when
207 using strace to see the forwarded data (which do not appear when using
208 splice()).
209
210 -dV : disable SSL verify on the server side. It is equivalent to having
211 "ssl-server-verify none" in the "global" section. This is useful when
212 trying to reproduce production issues out of the production
213 environment. Never use this in an init script as it degrades SSL security
214 to the servers.
215
216 -db : disable background mode and multi-process mode. The process remains in
217 foreground. It is mainly used during development or during small tests, as
218 Ctrl-C is enough to stop the process. Never use it in an init script.
219
220 -de : disable the use of the "epoll" poller. It is equivalent to the "global"
221 section's keyword "noepoll". It is mostly useful when suspecting a bug
222 related to this poller. On systems supporting epoll, the fallback will
223 generally be the "poll" poller.
224
225 -dk : disable the use of the "kqueue" poller. It is equivalent to the
226 "global" section's keyword "nokqueue". It is mostly useful when suspecting
227 a bug related to this poller. On systems supporting kqueue, the fallback
228 will generally be the "poll" poller.
229
230 -dp : disable the use of the "poll" poller. It is equivalent to the "global"
231 section's keyword "nopoll". It is mostly useful when suspecting a bug
232 related to this poller. On systems supporting poll, the fallback will
233 generally be the "select" poller, which cannot be disabled and is limited
234 to 1024 file descriptors.
235
Willy Tarreau3eed10e2016-11-07 21:03:16 +0100236 -dr : ignore server address resolution failures. It is very common when
237 validating a configuration out of production not to have access to the same
238 resolvers and to fail on server address resolution, making it difficult to
239 test a configuration. This option simply appends the "none" method to the
240 list of address resolution methods for all servers, ensuring that even if
241 the libc fails to resolve an address, the startup sequence is not
242 interrupted.
243
Willy Tarreau70060452015-12-14 12:46:07 +0100244 -m <limit> : limit the total allocatable memory to <limit> megabytes across
245 all processes. This may cause some connection refusals or some slowdowns
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200246 depending on the amount of memory needed for normal operations. This is
Willy Tarreau70060452015-12-14 12:46:07 +0100247 mostly used to force the processes to work in a constrained resource usage
248 scenario. It is important to note that the memory is not shared between
249 processes, so in a multi-process scenario, this value is first divided by
250 global.nbproc before forking.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200251
252 -n <limit> : limits the per-process connection limit to <limit>. This is
253 equivalent to the global section's keyword "maxconn". It has precedence
254 over this keyword. This may be used to quickly force lower limits to avoid
255 a service outage on systems where resource limits are too low.
256
257 -p <file> : write all processes' pids into <file> during startup. This is
258 equivalent to the "global" section's keyword "pidfile". The file is opened
259 before entering the chroot jail, and after doing the chdir() implied by
260 "-C". Each pid appears on its own line.
261
262 -q : set "quiet" mode. This disables some messages during the configuration
263 parsing and during startup. It can be used in combination with "-c" to
264 just check if a configuration file is valid or not.
265
266 -sf <pid>* : send the "finish" signal (SIGUSR1) to older processes after boot
267 completion to ask them to finish what they are doing and to leave. <pid>
268 is a list of pids to signal (one per argument). The list ends on any
269 option starting with a "-". It is not a problem if the list of pids is
270 empty, so that it can be built on the fly based on the result of a command
271 like "pidof" or "pgrep".
272
273 -st <pid>* : send the "terminate" signal (SIGTERM) to older processes after
274 boot completion to terminate them immediately without finishing what they
275 were doing. <pid> is a list of pids to signal (one per argument). The list
276 is ends on any option starting with a "-". It is not a problem if the list
277 of pids is empty, so that it can be built on the fly based on the result of
278 a command like "pidof" or "pgrep".
279
280 -v : report the version and build date.
281
282 -vv : display the version, build options, libraries versions and usable
283 pollers. This output is systematically requested when filing a bug report.
284
Olivier Houchardd33fc3a2017-04-05 22:50:59 +0200285 -x <unix_socket> : connect to the specified socket and try to retrieve any
286 listening sockets from the old process, and use them instead of trying to
287 bind new ones. This is useful to avoid missing any new connection when
William Lallemandf6975e92017-05-26 17:42:10 +0200288 reloading the configuration on Linux. The capability must be enable on the
289 stats socket using "expose-fd listeners" in your configuration.
Olivier Houchardd33fc3a2017-04-05 22:50:59 +0200290
Dan Lloyd8e48b872016-07-01 21:01:18 -0400291A safe way to start HAProxy from an init file consists in forcing the daemon
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200292mode, storing existing pids to a pid file and using this pid file to notify
293older processes to finish before leaving :
294
295 haproxy -f /etc/haproxy.cfg \
296 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
297
298When the configuration is split into a few specific files (eg: tcp vs http),
299it is recommended to use the "-f" option :
300
301 haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
302 -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
303 -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
304 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
305
306When an unknown number of files is expected, such as customer-specific files,
307it is recommended to assign them a name starting with a fixed-size sequence
308number and to use "--" to load them, possibly after loading some defaults :
309
310 haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
311 -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
312 -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
313 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) \
314 -f /etc/haproxy/default-customers.cfg -- /etc/haproxy/customers/*
315
316Sometimes a failure to start may happen for whatever reason. Then it is
317important to verify if the version of HAProxy you are invoking is the expected
318version and if it supports the features you are expecting (eg: SSL, PCRE,
319compression, Lua, etc). This can be verified using "haproxy -vv". Some
320important information such as certain build options, the target system and
321the versions of the libraries being used are reported there. It is also what
322you will systematically be asked for when posting a bug report :
323
324 $ haproxy -vv
325 HA-Proxy version 1.6-dev7-a088d3-4 2015/10/08
326 Copyright 2000-2015 Willy Tarreau <willy@haproxy.org>
327
328 Build options :
329 TARGET = linux2628
330 CPU = generic
331 CC = gcc
332 CFLAGS = -pg -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement \
333 -DBUFSIZE=8030 -DMAXREWRITE=1030 -DSO_MARK=36 -DTCP_REPAIR=19
334 OPTIONS = USE_ZLIB=1 USE_DLMALLOC=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
335
336 Default settings :
337 maxconn = 2000, bufsize = 8030, maxrewrite = 1030, maxpollevents = 200
338
339 Encrypted password support via crypt(3): yes
340 Built with zlib version : 1.2.6
341 Compression algorithms supported : identity("identity"), deflate("deflate"), \
342 raw-deflate("deflate"), gzip("gzip")
343 Built with OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
344 Running on OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
345 OpenSSL library supports TLS extensions : yes
346 OpenSSL library supports SNI : yes
347 OpenSSL library supports prefer-server-ciphers : yes
348 Built with PCRE version : 8.12 2011-01-15
349 PCRE library supports JIT : no (USE_PCRE_JIT not set)
350 Built with Lua version : Lua 5.3.1
351 Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND
352
353 Available polling systems :
354 epoll : pref=300, test result OK
355 poll : pref=200, test result OK
356 select : pref=150, test result OK
357 Total: 3 (3 usable), will use epoll.
358
359The relevant information that many non-developer users can verify here are :
360 - the version : 1.6-dev7-a088d3-4 above means the code is currently at commit
361 ID "a088d3" which is the 4th one after after official version "1.6-dev7".
362 Version 1.6-dev7 would show as "1.6-dev7-8c1ad7". What matters here is in
363 fact "1.6-dev7". This is the 7th development version of what will become
364 version 1.6 in the future. A development version not suitable for use in
365 production (unless you know exactly what you are doing). A stable version
366 will show as a 3-numbers version, such as "1.5.14-16f863", indicating the
367 14th level of fix on top of version 1.5. This is a production-ready version.
368
369 - the release date : 2015/10/08. It is represented in the universal
370 year/month/day format. Here this means August 8th, 2015. Given that stable
371 releases are issued every few months (1-2 months at the beginning, sometimes
372 6 months once the product becomes very stable), if you're seeing an old date
373 here, it means you're probably affected by a number of bugs or security
374 issues that have since been fixed and that it might be worth checking on the
375 official site.
376
377 - build options : they are relevant to people who build their packages
378 themselves, they can explain why things are not behaving as expected. For
379 example the development version above was built for Linux 2.6.28 or later,
Dan Lloyd8e48b872016-07-01 21:01:18 -0400380 targeting a generic CPU (no CPU-specific optimizations), and lacks any
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200381 code optimization (-O0) so it will perform poorly in terms of performance.
382
383 - libraries versions : zlib version is reported as found in the library
384 itself. In general zlib is considered a very stable product and upgrades
385 are almost never needed. OpenSSL reports two versions, the version used at
386 build time and the one being used, as found on the system. These ones may
387 differ by the last letter but never by the numbers. The build date is also
388 reported because most OpenSSL bugs are security issues and need to be taken
389 seriously, so this library absolutely needs to be kept up to date. Seeing a
390 4-months old version here is highly suspicious and indeed an update was
391 missed. PCRE provides very fast regular expressions and is highly
392 recommended. Certain of its extensions such as JIT are not present in all
393 versions and still young so some people prefer not to build with them,
Dan Lloyd8e48b872016-07-01 21:01:18 -0400394 which is why the build status is reported as well. Regarding the Lua
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200395 scripting language, HAProxy expects version 5.3 which is very young since
396 it was released a little time before HAProxy 1.6. It is important to check
397 on the Lua web site if some fixes are proposed for this branch.
398
399 - Available polling systems will affect the process's scalability when
400 dealing with more than about one thousand of concurrent connections. These
401 ones are only available when the correct system was indicated in the TARGET
402 variable during the build. The "epoll" mechanism is highly recommended on
403 Linux, and the kqueue mechanism is highly recommended on BSD. Lacking them
404 will result in poll() or even select() being used, causing a high CPU usage
405 when dealing with a lot of connections.
406
407
4084. Stopping and restarting HAProxy
409----------------------------------
410
411HAProxy supports a graceful and a hard stop. The hard stop is simple, when the
412SIGTERM signal is sent to the haproxy process, it immediately quits and all
413established connections are closed. The graceful stop is triggered when the
414SIGUSR1 signal is sent to the haproxy process. It consists in only unbinding
415from listening ports, but continue to process existing connections until they
416close. Once the last connection is closed, the process leaves.
417
418The hard stop method is used for the "stop" or "restart" actions of the service
419management script. The graceful stop is used for the "reload" action which
420tries to seamlessly reload a new configuration in a new process.
421
422Both of these signals may be sent by the new haproxy process itself during a
423reload or restart, so that they are sent at the latest possible moment and only
424if absolutely required. This is what is performed by the "-st" (hard) and "-sf"
425(graceful) options respectively.
426
William Lallemande202b1e2017-06-01 17:38:56 +0200427In master-worker mode, it is not needed to start a new haproxy process in
428order to reload the configuration. The master process reacts to the SIGUSR2
429signal by reexecuting itself with the -sf parameter followed by the PIDs of
430the workers. The master will then parse the configuration file and fork new
431workers.
432
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200433To understand better how these signals are used, it is important to understand
434the whole restart mechanism.
435
436First, an existing haproxy process is running. The administrator uses a system
437specific command such as "/etc/init.d/haproxy reload" to indicate he wants to
438take the new configuration file into effect. What happens then is the following.
439First, the service script (/etc/init.d/haproxy or equivalent) will verify that
440the configuration file parses correctly using "haproxy -c". After that it will
441try to start haproxy with this configuration file, using "-st" or "-sf".
442
443Then HAProxy tries to bind to all listening ports. If some fatal errors happen
444(eg: address not present on the system, permission denied), the process quits
445with an error. If a socket binding fails because a port is already in use, then
446the process will first send a SIGTTOU signal to all the pids specified in the
447"-st" or "-sf" pid list. This is what is called the "pause" signal. It instructs
448all existing haproxy processes to temporarily stop listening to their ports so
449that the new process can try to bind again. During this time, the old process
450continues to process existing connections. If the binding still fails (because
451for example a port is shared with another daemon), then the new process sends a
452SIGTTIN signal to the old processes to instruct them to resume operations just
453as if nothing happened. The old processes will then restart listening to the
454ports and continue to accept connections. Not that this mechanism is system
Dan Lloyd8e48b872016-07-01 21:01:18 -0400455dependent and some operating systems may not support it in multi-process mode.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200456
457If the new process manages to bind correctly to all ports, then it sends either
458the SIGTERM (hard stop in case of "-st") or the SIGUSR1 (graceful stop in case
459of "-sf") to all processes to notify them that it is now in charge of operations
460and that the old processes will have to leave, either immediately or once they
461have finished their job.
462
463It is important to note that during this timeframe, there are two small windows
464of a few milliseconds each where it is possible that a few connection failures
465will be noticed during high loads. Typically observed failure rates are around
4661 failure during a reload operation every 10000 new connections per second,
467which means that a heavily loaded site running at 30000 new connections per
468second may see about 3 failed connection upon every reload. The two situations
469where this happens are :
470
471 - if the new process fails to bind due to the presence of the old process,
472 it will first have to go through the SIGTTOU+SIGTTIN sequence, which
473 typically lasts about one millisecond for a few tens of frontends, and
474 during which some ports will not be bound to the old process and not yet
475 bound to the new one. HAProxy works around this on systems that support the
476 SO_REUSEPORT socket options, as it allows the new process to bind without
477 first asking the old one to unbind. Most BSD systems have been supporting
478 this almost forever. Linux has been supporting this in version 2.0 and
479 dropped it around 2.2, but some patches were floating around by then. It
480 was reintroduced in kernel 3.9, so if you are observing a connection
Dan Lloyd8e48b872016-07-01 21:01:18 -0400481 failure rate above the one mentioned above, please ensure that your kernel
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200482 is 3.9 or newer, or that relevant patches were backported to your kernel
483 (less likely).
484
485 - when the old processes close the listening ports, the kernel may not always
486 redistribute any pending connection that was remaining in the socket's
487 backlog. Under high loads, a SYN packet may happen just before the socket
488 is closed, and will lead to an RST packet being sent to the client. In some
489 critical environments where even one drop is not acceptable, these ones are
490 sometimes dealt with using firewall rules to block SYN packets during the
491 reload, forcing the client to retransmit. This is totally system-dependent,
492 as some systems might be able to visit other listening queues and avoid
493 this RST. A second case concerns the ACK from the client on a local socket
494 that was in SYN_RECV state just before the close. This ACK will lead to an
495 RST packet while the haproxy process is still not aware of it. This one is
Dan Lloyd8e48b872016-07-01 21:01:18 -0400496 harder to get rid of, though the firewall filtering rules mentioned above
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200497 will work well if applied one second or so before restarting the process.
498
499For the vast majority of users, such drops will never ever happen since they
500don't have enough load to trigger the race conditions. And for most high traffic
501users, the failure rate is still fairly within the noise margin provided that at
502least SO_REUSEPORT is properly supported on their systems.
503
504
5055. File-descriptor limitations
506------------------------------
507
508In order to ensure that all incoming connections will successfully be served,
509HAProxy computes at load time the total number of file descriptors that will be
510needed during the process's life. A regular Unix process is generally granted
5111024 file descriptors by default, and a privileged process can raise this limit
512itself. This is one reason for starting HAProxy as root and letting it adjust
513the limit. The default limit of 1024 file descriptors roughly allow about 500
514concurrent connections to be processed. The computation is based on the global
515maxconn parameter which limits the total number of connections per process, the
516number of listeners, the number of servers which have a health check enabled,
517the agent checks, the peers, the loggers and possibly a few other technical
518requirements. A simple rough estimate of this number consists in simply
519doubling the maxconn value and adding a few tens to get the approximate number
520of file descriptors needed.
521
522Originally HAProxy did not know how to compute this value, and it was necessary
523to pass the value using the "ulimit-n" setting in the global section. This
524explains why even today a lot of configurations are seen with this setting
525present. Unfortunately it was often miscalculated resulting in connection
526failures when approaching maxconn instead of throttling incoming connection
527while waiting for the needed resources. For this reason it is important to
Dan Lloyd8e48b872016-07-01 21:01:18 -0400528remove any vestigial "ulimit-n" setting that can remain from very old versions.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200529
530Raising the number of file descriptors to accept even moderate loads is
531mandatory but comes with some OS-specific adjustments. First, the select()
532polling system is limited to 1024 file descriptors. In fact on Linux it used
533to be capable of handling more but since certain OS ship with excessively
534restrictive SELinux policies forbidding the use of select() with more than
5351024 file descriptors, HAProxy now refuses to start in this case in order to
536avoid any issue at run time. On all supported operating systems, poll() is
537available and will not suffer from this limitation. It is automatically picked
Dan Lloyd8e48b872016-07-01 21:01:18 -0400538so there is nothing to do to get a working configuration. But poll's becomes
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200539very slow when the number of file descriptors increases. While HAProxy does its
540best to limit this performance impact (eg: via the use of the internal file
541descriptor cache and batched processing), a good rule of thumb is that using
542poll() with more than a thousand concurrent connections will use a lot of CPU.
543
544For Linux systems base on kernels 2.6 and above, the epoll() system call will
545be used. It's a much more scalable mechanism relying on callbacks in the kernel
546that guarantee a constant wake up time regardless of the number of registered
547monitored file descriptors. It is automatically used where detected, provided
548that HAProxy had been built for one of the Linux flavors. Its presence and
549support can be verified using "haproxy -vv".
550
551For BSD systems which support it, kqueue() is available as an alternative. It
552is much faster than poll() and even slightly faster than epoll() thanks to its
553batched handling of changes. At least FreeBSD and OpenBSD support it. Just like
554with Linux's epoll(), its support and availability are reported in the output
555of "haproxy -vv".
556
557Having a good poller is one thing, but it is mandatory that the process can
558reach the limits. When HAProxy starts, it immediately sets the new process's
559file descriptor limits and verifies if it succeeds. In case of failure, it
560reports it before forking so that the administrator can see the problem. As
561long as the process is started by as root, there should be no reason for this
562setting to fail. However, it can fail if the process is started by an
563unprivileged user. If there is a compelling reason for *not* starting haproxy
564as root (eg: started by end users, or by a per-application account), then the
565file descriptor limit can be raised by the system administrator for this
566specific user. The effectiveness of the setting can be verified by issuing
567"ulimit -n" from the user's command line. It should reflect the new limit.
568
569Warning: when an unprivileged user's limits are changed in this user's account,
570it is fairly common that these values are only considered when the user logs in
571and not at all in some scripts run at system boot time nor in crontabs. This is
572totally dependent on the operating system, keep in mind to check "ulimit -n"
573before starting haproxy when running this way. The general advice is never to
574start haproxy as an unprivileged user for production purposes. Another good
575reason is that it prevents haproxy from enabling some security protections.
576
577Once it is certain that the system will allow the haproxy process to use the
578requested number of file descriptors, two new system-specific limits may be
579encountered. The first one is the system-wide file descriptor limit, which is
580the total number of file descriptors opened on the system, covering all
581processes. When this limit is reached, accept() or socket() will typically
582return ENFILE. The second one is the per-process hard limit on the number of
583file descriptors, it prevents setrlimit() from being set higher. Both are very
584dependent on the operating system. On Linux, the system limit is set at boot
585based on the amount of memory. It can be changed with the "fs.file-max" sysctl.
586And the per-process hard limit is set to 1048576 by default, but it can be
587changed using the "fs.nr_open" sysctl.
588
589File descriptor limitations may be observed on a running process when they are
590set too low. The strace utility will report that accept() and socket() return
591"-1 EMFILE" when the process's limits have been reached. In this case, simply
592raising the "ulimit-n" value (or removing it) will solve the problem. If these
593system calls return "-1 ENFILE" then it means that the kernel's limits have
594been reached and that something must be done on a system-wide parameter. These
595trouble must absolutely be addressed, as they result in high CPU usage (when
596accept() fails) and failed connections that are generally visible to the user.
597One solution also consists in lowering the global maxconn value to enforce
598serialization, and possibly to disable HTTP keep-alive to force connections
599to be released and reused faster.
600
601
6026. Memory management
603--------------------
604
605HAProxy uses a simple and fast pool-based memory management. Since it relies on
606a small number of different object types, it's much more efficient to pick new
607objects from a pool which already contains objects of the appropriate size than
608to call malloc() for each different size. The pools are organized as a stack or
609LIFO, so that newly allocated objects are taken from recently released objects
610still hot in the CPU caches. Pools of similar sizes are merged together, in
611order to limit memory fragmentation.
612
613By default, since the focus is set on performance, each released object is put
614back into the pool it came from, and allocated objects are never freed since
615they are expected to be reused very soon.
616
617On the CLI, it is possible to check how memory is being used in pools thanks to
618the "show pools" command :
619
620 > show pools
621 Dumping pools usage. Use SIGQUIT to flush them.
622 - Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 3 users [SHARED]
623 - Pool hlua_com (48 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
624 - Pool vars (64 bytes) : 0 allocated (0 bytes), 0 used, 2 users [SHARED]
625 - Pool task (112 bytes) : 5 allocated (560 bytes), 5 used, 1 users [SHARED]
626 - Pool session (128 bytes) : 1 allocated (128 bytes), 1 used, 2 users [SHARED]
627 - Pool http_txn (272 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
628 - Pool connection (352 bytes) : 2 allocated (704 bytes), 2 used, 1 users [SHARED]
629 - Pool hdr_idx (416 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
630 - Pool stream (864 bytes) : 1 allocated (864 bytes), 1 used, 1 users [SHARED]
631 - Pool requri (1024 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
632 - Pool buffer (8064 bytes) : 3 allocated (24192 bytes), 2 used, 1 users [SHARED]
633 Total: 11 pools, 26608 bytes allocated, 18544 used.
634
635The pool name is only indicative, it's the name of the first object type using
636this pool. The size in parenthesis is the object size for objects in this pool.
637Object sizes are always rounded up to the closest multiple of 16 bytes. The
638number of objects currently allocated and the equivalent number of bytes is
639reported so that it is easy to know which pool is responsible for the highest
640memory usage. The number of objects currently in use is reported as well in the
641"used" field. The difference between "allocated" and "used" corresponds to the
642objects that have been freed and are available for immediate use.
643
644It is possible to limit the amount of memory allocated per process using the
645"-m" command line option, followed by a number of megabytes. It covers all of
646the process's addressable space, so that includes memory used by some libraries
647as well as the stack, but it is a reliable limit when building a resource
648constrained system. It works the same way as "ulimit -v" on systems which have
649it, or "ulimit -d" for the other ones.
650
651If a memory allocation fails due to the memory limit being reached or because
652the system doesn't have any enough memory, then haproxy will first start to
653free all available objects from all pools before attempting to allocate memory
654again. This mechanism of releasing unused memory can be triggered by sending
655the signal SIGQUIT to the haproxy process. When doing so, the pools state prior
656to the flush will also be reported to stderr when the process runs in
657foreground.
658
659During a reload operation, the process switched to the graceful stop state also
660automatically performs some flushes after releasing any connection so that all
661possible memory is released to save it for the new process.
662
663
6647. CPU usage
665------------
666
667HAProxy normally spends most of its time in the system and a smaller part in
668userland. A finely tuned 3.5 GHz CPU can sustain a rate about 80000 end-to-end
669connection setups and closes per second at 100% CPU on a single core. When one
670core is saturated, typical figures are :
671 - 95% system, 5% user for long TCP connections or large HTTP objects
672 - 85% system and 15% user for short TCP connections or small HTTP objects in
673 close mode
674 - 70% system and 30% user for small HTTP objects in keep-alive mode
675
676The amount of rules processing and regular expressions will increase the user
677land part. The presence of firewall rules, connection tracking, complex routing
678tables in the system will instead increase the system part.
679
680On most systems, the CPU time observed during network transfers can be cut in 4
681parts :
682 - the interrupt part, which concerns all the processing performed upon I/O
683 receipt, before the target process is even known. Typically Rx packets are
684 accounted for in interrupt. On some systems such as Linux where interrupt
685 processing may be deferred to a dedicated thread, it can appear as softirq,
686 and the thread is called ksoftirqd/0 (for CPU 0). The CPU taking care of
687 this load is generally defined by the hardware settings, though in the case
688 of softirq it is often possible to remap the processing to another CPU.
689 This interrupt part will often be perceived as parasitic since it's not
690 associated with any process, but it actually is some processing being done
691 to prepare the work for the process.
692
693 - the system part, which concerns all the processing done using kernel code
694 called from userland. System calls are accounted as system for example. All
695 synchronously delivered Tx packets will be accounted for as system time. If
696 some packets have to be deferred due to queues filling up, they may then be
697 processed in interrupt context later (eg: upon receipt of an ACK opening a
698 TCP window).
699
700 - the user part, which exclusively runs application code in userland. HAProxy
701 runs exclusively in this part, though it makes heavy use of system calls.
702 Rules processing, regular expressions, compression, encryption all add to
703 the user portion of CPU consumption.
704
705 - the idle part, which is what the CPU does when there is nothing to do. For
706 example HAProxy waits for an incoming connection, or waits for some data to
707 leave, meaning the system is waiting for an ACK from the client to push
708 these data.
709
710In practice regarding HAProxy's activity, it is in general reasonably accurate
711(but totally inexact) to consider that interrupt/softirq are caused by Rx
712processing in kernel drivers, that user-land is caused by layer 7 processing
713in HAProxy, and that system time is caused by network processing on the Tx
714path.
715
716Since HAProxy runs around an event loop, it waits for new events using poll()
717(or any alternative) and processes all these events as fast as possible before
718going back to poll() waiting for new events. It measures the time spent waiting
719in poll() compared to the time spent doing processing events. The ratio of
720polling time vs total time is called the "idle" time, it's the amount of time
721spent waiting for something to happen. This ratio is reported in the stats page
722on the "idle" line, or "Idle_pct" on the CLI. When it's close to 100%, it means
723the load is extremely low. When it's close to 0%, it means that there is
724constantly some activity. While it cannot be very accurate on an overloaded
725system due to other processes possibly preempting the CPU from the haproxy
726process, it still provides a good estimate about how HAProxy considers it is
727working : if the load is low and the idle ratio is low as well, it may indicate
728that HAProxy has a lot of work to do, possibly due to very expensive rules that
729have to be processed. Conversely, if HAProxy indicates the idle is close to
730100% while things are slow, it means that it cannot do anything to speed things
731up because it is already waiting for incoming data to process. In the example
732below, haproxy is completely idle :
733
734 $ echo "show info" | socat - /var/run/haproxy.sock | grep ^Idle
735 Idle_pct: 100
736
737When the idle ratio starts to become very low, it is important to tune the
738system and place processes and interrupts correctly to save the most possible
739CPU resources for all tasks. If a firewall is present, it may be worth trying
740to disable it or to tune it to ensure it is not responsible for a large part
741of the performance limitation. It's worth noting that unloading a stateful
742firewall generally reduces both the amount of interrupt/softirq and of system
743usage since such firewalls act both on the Rx and the Tx paths. On Linux,
744unloading the nf_conntrack and ip_conntrack modules will show whether there is
745anything to gain. If so, then the module runs with default settings and you'll
746have to figure how to tune it for better performance. In general this consists
747in considerably increasing the hash table size. On FreeBSD, "pfctl -d" will
748disable the "pf" firewall and its stateful engine at the same time.
749
750If it is observed that a lot of time is spent in interrupt/softirq, it is
751important to ensure that they don't run on the same CPU. Most systems tend to
752pin the tasks on the CPU where they receive the network traffic because for
753certain workloads it improves things. But with heavily network-bound workloads
754it is the opposite as the haproxy process will have to fight against its kernel
755counterpart. Pinning haproxy to one CPU core and the interrupts to another one,
756all sharing the same L3 cache tends to sensibly increase network performance
757because in practice the amount of work for haproxy and the network stack are
758quite close, so they can almost fill an entire CPU each. On Linux this is done
759using taskset (for haproxy) or using cpu-map (from the haproxy config), and the
760interrupts are assigned under /proc/irq. Many network interfaces support
761multiple queues and multiple interrupts. In general it helps to spread them
762across a small number of CPU cores provided they all share the same L3 cache.
763Please always stop irq_balance which always does the worst possible thing on
764such workloads.
765
766For CPU-bound workloads consisting in a lot of SSL traffic or a lot of
767compression, it may be worth using multiple processes dedicated to certain
768tasks, though there is no universal rule here and experimentation will have to
769be performed.
770
771In order to increase the CPU capacity, it is possible to make HAProxy run as
772several processes, using the "nbproc" directive in the global section. There
773are some limitations though :
774 - health checks are run per process, so the target servers will get as many
775 checks as there are running processes ;
776 - maxconn values and queues are per-process so the correct value must be set
777 to avoid overloading the servers ;
778 - outgoing connections should avoid using port ranges to avoid conflicts
779 - stick-tables are per process and are not shared between processes ;
780 - each peers section may only run on a single process at a time ;
781 - the CLI operations will only act on a single process at a time.
782
783With this in mind, it appears that the easiest setup often consists in having
784one first layer running on multiple processes and in charge for the heavy
785processing, passing the traffic to a second layer running in a single process.
786This mechanism is suited to SSL and compression which are the two CPU-heavy
787features. Instances can easily be chained over UNIX sockets (which are cheaper
fengpeiyuancc123c62016-01-15 16:40:53 +0800788than TCP sockets and which do not waste ports), and the proxy protocol which is
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200789useful to pass client information to the next stage. When doing so, it is
790generally a good idea to bind all the single-process tasks to process number 1
791and extra tasks to next processes, as this will make it easier to generate
792similar configurations for different machines.
793
794On Linux versions 3.9 and above, running HAProxy in multi-process mode is much
795more efficient when each process uses a distinct listening socket on the same
796IP:port ; this will make the kernel evenly distribute the load across all
797processes instead of waking them all up. Please check the "process" option of
798the "bind" keyword lines in the configuration manual for more information.
799
800
8018. Logging
802----------
803
804For logging, HAProxy always relies on a syslog server since it does not perform
805any file-system access. The standard way of using it is to send logs over UDP
806to the log server (by default on port 514). Very commonly this is configured to
807127.0.0.1 where the local syslog daemon is running, but it's also used over the
808network to log to a central server. The central server provides additional
809benefits especially in active-active scenarios where it is desirable to keep
810the logs merged in arrival order. HAProxy may also make use of a UNIX socket to
811send its logs to the local syslog daemon, but it is not recommended at all,
812because if the syslog server is restarted while haproxy runs, the socket will
813be replaced and new logs will be lost. Since HAProxy will be isolated inside a
814chroot jail, it will not have the ability to reconnect to the new socket. It
815has also been observed in field that the log buffers in use on UNIX sockets are
816very small and lead to lost messages even at very light loads. But this can be
817fine for testing however.
818
819It is recommended to add the following directive to the "global" section to
820make HAProxy log to the local daemon using facility "local0" :
821
822 log 127.0.0.1:514 local0
823
824and then to add the following one to each "defaults" section or to each frontend
825and backend section :
826
827 log global
828
829This way, all logs will be centralized through the global definition of where
830the log server is.
831
832Some syslog daemons do not listen to UDP traffic by default, so depending on
833the daemon being used, the syntax to enable this will vary :
834
835 - on sysklogd, you need to pass argument "-r" on the daemon's command line
836 so that it listens to a UDP socket for "remote" logs ; note that there is
837 no way to limit it to address 127.0.0.1 so it will also receive logs from
838 remote systems ;
839
840 - on rsyslogd, the following lines must be added to the configuration file :
841
842 $ModLoad imudp
843 $UDPServerAddress *
844 $UDPServerRun 514
845
846 - on syslog-ng, a new source can be created the following way, it then needs
847 to be added as a valid source in one of the "log" directives :
848
849 source s_udp {
850 udp(ip(127.0.0.1) port(514));
851 };
852
853Please consult your syslog daemon's manual for more information. If no logs are
854seen in the system's log files, please consider the following tests :
855
856 - restart haproxy. Each frontend and backend logs one line indicating it's
857 starting. If these logs are received, it means logs are working.
858
859 - run "strace -tt -s100 -etrace=sendmsg -p <haproxy's pid>" and perform some
860 activity that you expect to be logged. You should see the log messages
861 being sent using sendmsg() there. If they don't appear, restart using
862 strace on top of haproxy. If you still see no logs, it definitely means
863 that something is wrong in your configuration.
864
865 - run tcpdump to watch for port 514, for example on the loopback interface if
866 the traffic is being sent locally : "tcpdump -As0 -ni lo port 514". If the
867 packets are seen there, it's the proof they're sent then the syslogd daemon
868 needs to be troubleshooted.
869
870While traffic logs are sent from the frontends (where the incoming connections
871are accepted), backends also need to be able to send logs in order to report a
872server state change consecutive to a health check. Please consult HAProxy's
873configuration manual for more information regarding all possible log settings.
874
Dan Lloyd8e48b872016-07-01 21:01:18 -0400875It is convenient to chose a facility that is not used by other daemons. HAProxy
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200876examples often suggest "local0" for traffic logs and "local1" for admin logs
877because they're never seen in field. A single facility would be enough as well.
878Having separate logs is convenient for log analysis, but it's also important to
879remember that logs may sometimes convey confidential information, and as such
Dan Lloyd8e48b872016-07-01 21:01:18 -0400880they must not be mixed with other logs that may accidentally be handed out to
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200881unauthorized people.
882
883For in-field troubleshooting without impacting the server's capacity too much,
884it is recommended to make use of the "halog" utility provided with HAProxy.
885This is sort of a grep-like utility designed to process HAProxy log files at
886a very fast data rate. Typical figures range between 1 and 2 GB of logs per
887second. It is capable of extracting only certain logs (eg: search for some
888classes of HTTP status codes, connection termination status, search by response
889time ranges, look for errors only), count lines, limit the output to a number
890of lines, and perform some more advanced statistics such as sorting servers
891by response time or error counts, sorting URLs by time or count, sorting client
892addresses by access count, and so on. It is pretty convenient to quickly spot
893anomalies such as a bot looping on the site, and block them.
894
895
8969. Statistics and monitoring
897----------------------------
898
Willy Tarreau44aed902015-10-13 14:45:29 +0200899It is possible to query HAProxy about its status. The most commonly used
900mechanism is the HTTP statistics page. This page also exposes an alternative
901CSV output format for monitoring tools. The same format is provided on the
902Unix socket.
903
904
9059.1. CSV format
906---------------
907
908The statistics may be consulted either from the unix socket or from the HTTP
909page. Both means provide a CSV format whose fields follow. The first line
910begins with a sharp ('#') and has one word per comma-delimited field which
911represents the title of the column. All other lines starting at the second one
912use a classical CSV format using a comma as the delimiter, and the double quote
913('"') as an optional text delimiter, but only if the enclosed text is ambiguous
914(if it contains a quote or a comma). The double-quote character ('"') in the
915text is doubled ('""'), which is the format that most tools recognize. Please
916do not insert any column before these ones in order not to break tools which
917use hard-coded column positions.
918
919In brackets after each field name are the types which may have a value for
920that field. The types are L (Listeners), F (Frontends), B (Backends), and
921S (Servers).
922
923 0. pxname [LFBS]: proxy name
924 1. svname [LFBS]: service name (FRONTEND for frontend, BACKEND for backend,
925 any name for server/listener)
926 2. qcur [..BS]: current queued requests. For the backend this reports the
927 number queued without a server assigned.
928 3. qmax [..BS]: max value of qcur
929 4. scur [LFBS]: current sessions
930 5. smax [LFBS]: max sessions
931 6. slim [LFBS]: configured session limit
Willy Tarreauc73810f2016-01-11 13:52:04 +0100932 7. stot [LFBS]: cumulative number of sessions
Willy Tarreau44aed902015-10-13 14:45:29 +0200933 8. bin [LFBS]: bytes in
934 9. bout [LFBS]: bytes out
935 10. dreq [LFB.]: requests denied because of security concerns.
936 - For tcp this is because of a matched tcp-request content rule.
937 - For http this is because of a matched http-request or tarpit rule.
938 11. dresp [LFBS]: responses denied because of security concerns.
939 - For http this is because of a matched http-request rule, or
940 "option checkcache".
941 12. ereq [LF..]: request errors. Some of the possible causes are:
942 - early termination from the client, before the request has been sent.
943 - read error from the client
944 - client timeout
945 - client closed connection
946 - various bad requests from the client.
947 - request was tarpitted.
948 13. econ [..BS]: number of requests that encountered an error trying to
949 connect to a backend server. The backend stat is the sum of the stat
950 for all servers of that backend, plus any connection errors not
951 associated with a particular server (such as the backend having no
952 active servers).
953 14. eresp [..BS]: response errors. srv_abrt will be counted here also.
954 Some other errors are:
955 - write error on the client socket (won't be counted for the server stat)
956 - failure applying filters to the response.
957 15. wretr [..BS]: number of times a connection to a server was retried.
958 16. wredis [..BS]: number of times a request was redispatched to another
959 server. The server value counts the number of times that server was
960 switched away from.
Willy Tarreaub96dd282016-11-09 14:45:51 +0100961 17. status [LFBS]: status (UP/DOWN/NOLB/MAINT/MAINT(via)/MAINT(resolution)...)
Willy Tarreau44aed902015-10-13 14:45:29 +0200962 18. weight [..BS]: total weight (backend), server weight (server)
963 19. act [..BS]: number of active servers (backend), server is active (server)
964 20. bck [..BS]: number of backup servers (backend), server is backup (server)
965 21. chkfail [...S]: number of failed checks. (Only counts checks failed when
966 the server is up.)
967 22. chkdown [..BS]: number of UP->DOWN transitions. The backend counter counts
968 transitions to the whole backend being down, rather than the sum of the
969 counters for each server.
970 23. lastchg [..BS]: number of seconds since the last UP<->DOWN transition
971 24. downtime [..BS]: total downtime (in seconds). The value for the backend
972 is the downtime for the whole backend, not the sum of the server downtime.
973 25. qlimit [...S]: configured maxqueue for the server, or nothing in the
974 value is 0 (default, meaning no limit)
975 26. pid [LFBS]: process id (0 for first instance, 1 for second, ...)
976 27. iid [LFBS]: unique proxy id
977 28. sid [L..S]: server id (unique inside a proxy)
978 29. throttle [...S]: current throttle percentage for the server, when
979 slowstart is active, or no value if not in slowstart.
980 30. lbtot [..BS]: total number of times a server was selected, either for new
981 sessions, or when re-dispatching. The server counter is the number
982 of times that server was selected.
983 31. tracked [...S]: id of proxy/server if tracking is enabled.
984 32. type [LFBS]: (0=frontend, 1=backend, 2=server, 3=socket/listener)
985 33. rate [.FBS]: number of sessions per second over last elapsed second
986 34. rate_lim [.F..]: configured limit on new sessions per second
987 35. rate_max [.FBS]: max number of new sessions per second
988 36. check_status [...S]: status of last health check, one of:
989 UNK -> unknown
990 INI -> initializing
991 SOCKERR -> socket error
992 L4OK -> check passed on layer 4, no upper layers testing enabled
993 L4TOUT -> layer 1-4 timeout
994 L4CON -> layer 1-4 connection problem, for example
995 "Connection refused" (tcp rst) or "No route to host" (icmp)
996 L6OK -> check passed on layer 6
997 L6TOUT -> layer 6 (SSL) timeout
998 L6RSP -> layer 6 invalid response - protocol error
999 L7OK -> check passed on layer 7
1000 L7OKC -> check conditionally passed on layer 7, for example 404 with
1001 disable-on-404
1002 L7TOUT -> layer 7 (HTTP/SMTP) timeout
1003 L7RSP -> layer 7 invalid response - protocol error
1004 L7STS -> layer 7 response error, for example HTTP 5xx
Daniel Schnellerb6c8b0d2017-09-01 19:13:55 +02001005 Notice: If a check is currently running, the last known status will be
1006 reported, prefixed with "* ". e. g. "* L7OK".
Willy Tarreau44aed902015-10-13 14:45:29 +02001007 37. check_code [...S]: layer5-7 code, if available
1008 38. check_duration [...S]: time in ms took to finish last health check
1009 39. hrsp_1xx [.FBS]: http responses with 1xx code
1010 40. hrsp_2xx [.FBS]: http responses with 2xx code
1011 41. hrsp_3xx [.FBS]: http responses with 3xx code
1012 42. hrsp_4xx [.FBS]: http responses with 4xx code
1013 43. hrsp_5xx [.FBS]: http responses with 5xx code
1014 44. hrsp_other [.FBS]: http responses with other codes (protocol error)
1015 45. hanafail [...S]: failed health checks details
1016 46. req_rate [.F..]: HTTP requests per second over last elapsed second
1017 47. req_rate_max [.F..]: max number of HTTP requests per second observed
Willy Tarreaufb981bd2016-12-12 14:31:46 +01001018 48. req_tot [.FB.]: total number of HTTP requests received
Willy Tarreau44aed902015-10-13 14:45:29 +02001019 49. cli_abrt [..BS]: number of data transfers aborted by the client
1020 50. srv_abrt [..BS]: number of data transfers aborted by the server
1021 (inc. in eresp)
1022 51. comp_in [.FB.]: number of HTTP response bytes fed to the compressor
1023 52. comp_out [.FB.]: number of HTTP response bytes emitted by the compressor
1024 53. comp_byp [.FB.]: number of bytes that bypassed the HTTP compressor
1025 (CPU/BW limit)
1026 54. comp_rsp [.FB.]: number of HTTP responses that were compressed
1027 55. lastsess [..BS]: number of seconds since last session assigned to
1028 server/backend
1029 56. last_chk [...S]: last health check contents or textual error
1030 57. last_agt [...S]: last agent check contents or textual error
1031 58. qtime [..BS]: the average queue time in ms over the 1024 last requests
1032 59. ctime [..BS]: the average connect time in ms over the 1024 last requests
1033 60. rtime [..BS]: the average response time in ms over the 1024 last requests
1034 (0 for TCP)
1035 61. ttime [..BS]: the average total session time in ms over the 1024 last
1036 requests
Willy Tarreau7f618842016-01-08 11:40:03 +01001037 62. agent_status [...S]: status of last agent check, one of:
1038 UNK -> unknown
1039 INI -> initializing
1040 SOCKERR -> socket error
1041 L4OK -> check passed on layer 4, no upper layers testing enabled
1042 L4TOUT -> layer 1-4 timeout
1043 L4CON -> layer 1-4 connection problem, for example
1044 "Connection refused" (tcp rst) or "No route to host" (icmp)
1045 L7OK -> agent reported "up"
1046 L7STS -> agent reported "fail", "stop", or "down"
1047 63. agent_code [...S]: numeric code reported by agent if any (unused for now)
1048 64. agent_duration [...S]: time in ms taken to finish last check
Willy Tarreaudd7354b2016-01-08 13:47:26 +01001049 65. check_desc [...S]: short human-readable description of check_status
1050 66. agent_desc [...S]: short human-readable description of agent_status
Willy Tarreau3141f592016-01-08 14:25:28 +01001051 67. check_rise [...S]: server's "rise" parameter used by checks
1052 68. check_fall [...S]: server's "fall" parameter used by checks
1053 69. check_health [...S]: server's health check value between 0 and rise+fall-1
1054 70. agent_rise [...S]: agent's "rise" parameter, normally 1
1055 71. agent_fall [...S]: agent's "fall" parameter, normally 1
1056 72. agent_health [...S]: agent's health parameter, between 0 and rise+fall-1
Willy Tarreaua6f5a732016-01-08 16:59:56 +01001057 73. addr [L..S]: address:port or "unix". IPv6 has brackets around the address.
Willy Tarreaue4847c62016-01-08 15:43:54 +01001058 74: cookie [..BS]: server's cookie value or backend's cookie name
Willy Tarreauf8211df2016-01-11 14:09:38 +01001059 75: mode [LFBS]: proxy mode (tcp, http, health, unknown)
Willy Tarreauf1516d92016-01-11 14:48:36 +01001060 76: algo [..B.]: load balancing algorithm
Willy Tarreauc73810f2016-01-11 13:52:04 +01001061 77: conn_rate [.F..]: number of connections over the last elapsed second
1062 78: conn_rate_max [.F..]: highest known conn_rate
1063 79: conn_tot [.F..]: cumulative number of connections
Willy Tarreau5b9bdff2016-01-11 14:40:47 +01001064 80: intercepted [.FB.]: cum. number of intercepted requests (monitor, stats)
Willy Tarreau8a90b8e2016-10-21 18:15:32 +02001065 81: dcon [LF..]: requests denied by "tcp-request connection" rules
Willy Tarreaua5bc36b2016-10-21 18:16:27 +02001066 82: dses [LF..]: requests denied by "tcp-request session" rules
Willy Tarreau44aed902015-10-13 14:45:29 +02001067
1068
Willy Tarreau5d8b9792016-03-11 11:09:34 +010010699.2) Typed output format
1070------------------------
1071
1072Both "show info" and "show stat" support a mode where each output value comes
1073with its type and sufficient information to know how the value is supposed to
1074be aggregated between processes and how it evolves.
1075
1076In all cases, the output consists in having a single value per line with all
1077the information split into fields delimited by colons (':').
1078
1079The first column designates the object or metric being dumped. Its format is
1080specific to the command producing this output and will not be described in this
1081section. Usually it will consist in a series of identifiers and field names.
1082
1083The second column contains 3 characters respectively indicating the origin, the
1084nature and the scope of the value being reported. The first character (the
1085origin) indicates where the value was extracted from. Possible characters are :
1086
1087 M The value is a metric. It is valid at one instant any may change depending
1088 on its nature .
1089
1090 S The value is a status. It represents a discrete value which by definition
1091 cannot be aggregated. It may be the status of a server ("UP" or "DOWN"),
1092 the PID of the process, etc.
1093
1094 K The value is a sorting key. It represents an identifier which may be used
1095 to group some values together because it is unique among its class. All
1096 internal identifiers are keys. Some names can be listed as keys if they
1097 are unique (eg: a frontend name is unique). In general keys come from the
Dan Lloyd8e48b872016-07-01 21:01:18 -04001098 configuration, even though some of them may automatically be assigned. For
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001099 most purposes keys may be considered as equivalent to configuration.
1100
1101 C The value comes from the configuration. Certain configuration values make
1102 sense on the output, for example a concurrent connection limit or a cookie
1103 name. By definition these values are the same in all processes started
1104 from the same configuration file.
1105
1106 P The value comes from the product itself. There are very few such values,
1107 most common use is to report the product name, version and release date.
1108 These elements are also the same between all processes.
1109
1110The second character (the nature) indicates the nature of the information
1111carried by the field in order to let an aggregator decide on what operation to
1112use to aggregate multiple values. Possible characters are :
1113
1114 A The value represents an age since a last event. This is a bit different
1115 from the duration in that an age is automatically computed based on the
1116 current date. A typical example is how long ago did the last session
1117 happen on a server. Ages are generally aggregated by taking the minimum
1118 value and do not need to be stored.
1119
1120 a The value represents an already averaged value. The average response times
1121 and server weights are of this nature. Averages can typically be averaged
1122 between processes.
1123
1124 C The value represents a cumulative counter. Such measures perpetually
1125 increase until they wrap around. Some monitoring protocols need to tell
1126 the difference between a counter and a gauge to report a different type.
1127 In general counters may simply be summed since they represent events or
1128 volumes. Examples of metrics of this nature are connection counts or byte
1129 counts.
1130
1131 D The value represents a duration for a status. There are a few usages of
1132 this, most of them include the time taken by the last health check and
1133 the time a server has spent down. Durations are generally not summed,
1134 most of the time the maximum will be retained to compute an SLA.
1135
1136 G The value represents a gauge. It's a measure at one instant. The memory
1137 usage or the current number of active connections are of this nature.
1138 Metrics of this type are typically summed during aggregation.
1139
1140 L The value represents a limit (generally a configured one). By nature,
1141 limits are harder to aggregate since they are specific to the point where
1142 they were retrieved. In certain situations they may be summed or be kept
1143 separate.
1144
1145 M The value represents a maximum. In general it will apply to a gauge and
1146 keep the highest known value. An example of such a metric could be the
1147 maximum amount of concurrent connections that was encountered in the
1148 product's life time. To correctly aggregate maxima, you are supposed to
1149 output a range going from the maximum of all maxima and the sum of all
1150 of them. There is indeed no way to know if they were encountered
1151 simultaneously or not.
1152
1153 m The value represents a minimum. In general it will apply to a gauge and
1154 keep the lowest known value. An example of such a metric could be the
1155 minimum amount of free memory pools that was encountered in the product's
1156 life time. To correctly aggregate minima, you are supposed to output a
1157 range going from the minimum of all minima and the sum of all of them.
1158 There is indeed no way to know if they were encountered simultaneously
1159 or not.
1160
1161 N The value represents a name, so it is a string. It is used to report
1162 proxy names, server names and cookie names. Names have configuration or
1163 keys as their origin and are supposed to be the same among all processes.
1164
1165 O The value represents a free text output. Outputs from various commands,
1166 returns from health checks, node descriptions are of such nature.
1167
1168 R The value represents an event rate. It's a measure at one instant. It is
1169 quite similar to a gauge except that the recipient knows that this measure
1170 moves slowly and may decide not to keep all values. An example of such a
1171 metric is the measured amount of connections per second. Metrics of this
1172 type are typically summed during aggregation.
1173
1174 T The value represents a date or time. A field emitting the current date
1175 would be of this type. The method to aggregate such information is left
1176 as an implementation choice. For now no field uses this type.
1177
1178The third character (the scope) indicates what extent the value reflects. Some
1179elements may be per process while others may be per configuration or per system.
1180The distinction is important to know whether or not a single value should be
1181kept during aggregation or if values have to be aggregated. The following
1182characters are currently supported :
1183
1184 C The value is valid for a whole cluster of nodes, which is the set of nodes
1185 communicating over the peers protocol. An example could be the amount of
1186 entries present in a stick table that is replicated with other peers. At
1187 the moment no metric use this scope.
1188
1189 P The value is valid only for the process reporting it. Most metrics use
1190 this scope.
1191
1192 S The value is valid for the whole service, which is the set of processes
1193 started together from the same configuration file. All metrics originating
1194 from the configuration use this scope. Some other metrics may use it as
1195 well for some shared resources (eg: shared SSL cache statistics).
1196
1197 s The value is valid for the whole system, such as the system's hostname,
1198 current date or resource usage. At the moment this scope is not used by
1199 any metric.
1200
1201Consumers of these information will generally have enough of these 3 characters
1202to determine how to accurately report aggregated information across multiple
1203processes.
1204
1205After this column, the third column indicates the type of the field, among "s32"
1206(signed 32-bit integer), "s64" (signed 64-bit integer), "u32" (unsigned 32-bit
1207integer), "u64" (unsigned 64-bit integer), "str" (string). It is important to
1208know the type before parsing the value in order to properly read it. For example
1209a string containing only digits is still a string an not an integer (eg: an
1210error code extracted by a check).
1211
1212Then the fourth column is the value itself, encoded according to its type.
1213Strings are dumped as-is immediately after the colon without any leading space.
1214If a string contains a colon, it will appear normally. This means that the
1215output should not be exclusively split around colons or some check outputs
1216or server addresses might be truncated.
1217
1218
12199.3. Unix Socket commands
Willy Tarreau44aed902015-10-13 14:45:29 +02001220-------------------------
1221
1222The stats socket is not enabled by default. In order to enable it, it is
1223necessary to add one line in the global section of the haproxy configuration.
1224A second line is recommended to set a larger timeout, always appreciated when
1225issuing commands by hand :
1226
1227 global
1228 stats socket /var/run/haproxy.sock mode 600 level admin
1229 stats timeout 2m
1230
1231It is also possible to add multiple instances of the stats socket by repeating
1232the line, and make them listen to a TCP port instead of a UNIX socket. This is
1233never done by default because this is dangerous, but can be handy in some
1234situations :
1235
1236 global
1237 stats socket /var/run/haproxy.sock mode 600 level admin
1238 stats socket ipv4@192.168.0.1:9999 level admin
1239 stats timeout 2m
1240
1241To access the socket, an external utility such as "socat" is required. Socat is
1242a swiss-army knife to connect anything to anything. We use it to connect
1243terminals to the socket, or a couple of stdin/stdout pipes to it for scripts.
1244The two main syntaxes we'll use are the following :
1245
1246 # socat /var/run/haproxy.sock stdio
1247 # socat /var/run/haproxy.sock readline
1248
1249The first one is used with scripts. It is possible to send the output of a
1250script to haproxy, and pass haproxy's output to another script. That's useful
1251for retrieving counters or attack traces for example.
1252
1253The second one is only useful for issuing commands by hand. It has the benefit
1254that the terminal is handled by the readline library which supports line
1255editing and history, which is very convenient when issuing repeated commands
1256(eg: watch a counter).
1257
1258The socket supports two operation modes :
1259 - interactive
1260 - non-interactive
1261
1262The non-interactive mode is the default when socat connects to the socket. In
1263this mode, a single line may be sent. It is processed as a whole, responses are
1264sent back, and the connection closes after the end of the response. This is the
1265mode that scripts and monitoring tools use. It is possible to send multiple
1266commands in this mode, they need to be delimited by a semi-colon (';'). For
1267example :
1268
1269 # echo "show info;show stat;show table" | socat /var/run/haproxy stdio
1270
Dragan Dosena1c35ab2016-11-24 11:33:12 +01001271If a command needs to use a semi-colon or a backslash (eg: in a value), it
1272must be preceeded by a backslash ('\').
Chad Lavoiee3f50312016-05-26 16:42:25 -04001273
Willy Tarreau44aed902015-10-13 14:45:29 +02001274The interactive mode displays a prompt ('>') and waits for commands to be
1275entered on the line, then processes them, and displays the prompt again to wait
1276for a new command. This mode is entered via the "prompt" command which must be
1277sent on the first line in non-interactive mode. The mode is a flip switch, if
1278"prompt" is sent in interactive mode, it is disabled and the connection closes
1279after processing the last command of the same line.
1280
1281For this reason, when debugging by hand, it's quite common to start with the
1282"prompt" command :
1283
1284 # socat /var/run/haproxy readline
1285 prompt
1286 > show info
1287 ...
1288 >
1289
1290Since multiple commands may be issued at once, haproxy uses the empty line as a
1291delimiter to mark an end of output for each command, and takes care of ensuring
1292that no command can emit an empty line on output. A script can thus easily
1293parse the output even when multiple commands were pipelined on a single line.
1294
1295It is important to understand that when multiple haproxy processes are started
1296on the same sockets, any process may pick up the request and will output its
1297own stats.
1298
1299The list of commands currently supported on the stats socket is provided below.
1300If an unknown command is sent, haproxy displays the usage message which reminds
1301all supported commands. Some commands support a more complex syntax, generally
1302it will explain what part of the command is invalid when this happens.
1303
Olivier Doucetd8703e82017-08-31 11:05:10 +02001304Some commands require a higher level of privilege to work. If you do not have
1305enough privilege, you will get an error "Permission denied". Please check
1306the "level" option of the "bind" keyword lines in the configuration manual
1307for more information.
1308
Willy Tarreau44aed902015-10-13 14:45:29 +02001309add acl <acl> <pattern>
1310 Add an entry into the acl <acl>. <acl> is the #<id> or the <file> returned by
1311 "show acl". This command does not verify if the entry already exists. This
1312 command cannot be used if the reference <acl> is a file also used with a map.
1313 In this case, you must use the command "add map" in place of "add acl".
1314
1315add map <map> <key> <value>
1316 Add an entry into the map <map> to associate the value <value> to the key
1317 <key>. This command does not verify if the entry already exists. It is
1318 mainly used to fill a map after a clear operation. Note that if the reference
1319 <map> is a file and is shared with a map, this map will contain also a new
1320 pattern entry.
1321
1322clear counters
1323 Clear the max values of the statistics counters in each proxy (frontend &
Dan Lloyd8e48b872016-07-01 21:01:18 -04001324 backend) and in each server. The accumulated counters are not affected. This
Willy Tarreau44aed902015-10-13 14:45:29 +02001325 can be used to get clean counters after an incident, without having to
1326 restart nor to clear traffic counters. This command is restricted and can
1327 only be issued on sockets configured for levels "operator" or "admin".
1328
1329clear counters all
1330 Clear all statistics counters in each proxy (frontend & backend) and in each
1331 server. This has the same effect as restarting. This command is restricted
1332 and can only be issued on sockets configured for level "admin".
1333
1334clear acl <acl>
1335 Remove all entries from the acl <acl>. <acl> is the #<id> or the <file>
1336 returned by "show acl". Note that if the reference <acl> is a file and is
1337 shared with a map, this map will be also cleared.
1338
1339clear map <map>
1340 Remove all entries from the map <map>. <map> is the #<id> or the <file>
1341 returned by "show map". Note that if the reference <map> is a file and is
1342 shared with a acl, this acl will be also cleared.
1343
1344clear table <table> [ data.<type> <operator> <value> ] | [ key <key> ]
1345 Remove entries from the stick-table <table>.
1346
1347 This is typically used to unblock some users complaining they have been
1348 abusively denied access to a service, but this can also be used to clear some
1349 stickiness entries matching a server that is going to be replaced (see "show
1350 table" below for details). Note that sometimes, removal of an entry will be
1351 refused because it is currently tracked by a session. Retrying a few seconds
1352 later after the session ends is usual enough.
1353
1354 In the case where no options arguments are given all entries will be removed.
1355
1356 When the "data." form is used entries matching a filter applied using the
1357 stored data (see "stick-table" in section 4.2) are removed. A stored data
1358 type must be specified in <type>, and this data type must be stored in the
1359 table otherwise an error is reported. The data is compared according to
1360 <operator> with the 64-bit integer <value>. Operators are the same as with
1361 the ACLs :
1362
1363 - eq : match entries whose data is equal to this value
1364 - ne : match entries whose data is not equal to this value
1365 - le : match entries whose data is less than or equal to this value
1366 - ge : match entries whose data is greater than or equal to this value
1367 - lt : match entries whose data is less than this value
1368 - gt : match entries whose data is greater than this value
1369
1370 When the key form is used the entry <key> is removed. The key must be of the
1371 same type as the table, which currently is limited to IPv4, IPv6, integer and
1372 string.
1373
1374 Example :
1375 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1376 >>> # table: http_proxy, type: ip, size:204800, used:2
1377 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
1378 bytes_out_rate(60000)=187
1379 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
1380 bytes_out_rate(60000)=191
1381
1382 $ echo "clear table http_proxy key 127.0.0.1" | socat stdio /tmp/sock1
1383
1384 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1385 >>> # table: http_proxy, type: ip, size:204800, used:1
1386 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
1387 bytes_out_rate(60000)=191
1388 $ echo "clear table http_proxy data.gpc0 eq 1" | socat stdio /tmp/sock1
1389 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1390 >>> # table: http_proxy, type: ip, size:204800, used:1
1391
1392del acl <acl> [<key>|#<ref>]
1393 Delete all the acl entries from the acl <acl> corresponding to the key <key>.
1394 <acl> is the #<id> or the <file> returned by "show acl". If the <ref> is used,
1395 this command delete only the listed reference. The reference can be found with
1396 listing the content of the acl. Note that if the reference <acl> is a file and
1397 is shared with a map, the entry will be also deleted in the map.
1398
1399del map <map> [<key>|#<ref>]
1400 Delete all the map entries from the map <map> corresponding to the key <key>.
1401 <map> is the #<id> or the <file> returned by "show map". If the <ref> is used,
1402 this command delete only the listed reference. The reference can be found with
1403 listing the content of the map. Note that if the reference <map> is a file and
1404 is shared with a acl, the entry will be also deleted in the map.
1405
1406disable agent <backend>/<server>
1407 Mark the auxiliary agent check as temporarily stopped.
1408
1409 In the case where an agent check is being run as a auxiliary check, due
1410 to the agent-check parameter of a server directive, new checks are only
Dan Lloyd8e48b872016-07-01 21:01:18 -04001411 initialized when the agent is in the enabled. Thus, disable agent will
Willy Tarreau44aed902015-10-13 14:45:29 +02001412 prevent any new agent checks from begin initiated until the agent
1413 re-enabled using enable agent.
1414
1415 When an agent is disabled the processing of an auxiliary agent check that
1416 was initiated while the agent was set as enabled is as follows: All
1417 results that would alter the weight, specifically "drain" or a weight
1418 returned by the agent, are ignored. The processing of agent check is
1419 otherwise unchanged.
1420
1421 The motivation for this feature is to allow the weight changing effects
1422 of the agent checks to be paused to allow the weight of a server to be
1423 configured using set weight without being overridden by the agent.
1424
1425 This command is restricted and can only be issued on sockets configured for
1426 level "admin".
1427
Olivier Houchard614f8d72017-03-14 20:08:46 +01001428disable dynamic-cookie backend <backend>
1429 Disable the generation of dynamic cookies fot the backend <backend>
1430
Willy Tarreau44aed902015-10-13 14:45:29 +02001431disable frontend <frontend>
1432 Mark the frontend as temporarily stopped. This corresponds to the mode which
1433 is used during a soft restart : the frontend releases the port but can be
1434 enabled again if needed. This should be used with care as some non-Linux OSes
1435 are unable to enable it back. This is intended to be used in environments
1436 where stopping a proxy is not even imaginable but a misconfigured proxy must
1437 be fixed. That way it's possible to release the port and bind it into another
1438 process to restore operations. The frontend will appear with status "STOP"
1439 on the stats page.
1440
1441 The frontend may be specified either by its name or by its numeric ID,
1442 prefixed with a sharp ('#').
1443
1444 This command is restricted and can only be issued on sockets configured for
1445 level "admin".
1446
1447disable health <backend>/<server>
1448 Mark the primary health check as temporarily stopped. This will disable
1449 sending of health checks, and the last health check result will be ignored.
1450 The server will be in unchecked state and considered UP unless an auxiliary
1451 agent check forces it down.
1452
1453 This command is restricted and can only be issued on sockets configured for
1454 level "admin".
1455
1456disable server <backend>/<server>
1457 Mark the server DOWN for maintenance. In this mode, no more checks will be
1458 performed on the server until it leaves maintenance.
1459 If the server is tracked by other servers, those servers will be set to DOWN
1460 during the maintenance.
1461
1462 In the statistics page, a server DOWN for maintenance will appear with a
1463 "MAINT" status, its tracking servers with the "MAINT(via)" one.
1464
1465 Both the backend and the server may be specified either by their name or by
1466 their numeric ID, prefixed with a sharp ('#').
1467
1468 This command is restricted and can only be issued on sockets configured for
1469 level "admin".
1470
1471enable agent <backend>/<server>
1472 Resume auxiliary agent check that was temporarily stopped.
1473
1474 See "disable agent" for details of the effect of temporarily starting
1475 and stopping an auxiliary agent.
1476
1477 This command is restricted and can only be issued on sockets configured for
1478 level "admin".
1479
Olivier Houchard614f8d72017-03-14 20:08:46 +01001480enable dynamic-cookie backend <backend>
1481 Enable the generation of dynamic cookies fot the backend <backend>
1482 A secret key must also be provided
1483
Willy Tarreau44aed902015-10-13 14:45:29 +02001484enable frontend <frontend>
1485 Resume a frontend which was temporarily stopped. It is possible that some of
1486 the listening ports won't be able to bind anymore (eg: if another process
1487 took them since the 'disable frontend' operation). If this happens, an error
1488 is displayed. Some operating systems might not be able to resume a frontend
1489 which was disabled.
1490
1491 The frontend may be specified either by its name or by its numeric ID,
1492 prefixed with a sharp ('#').
1493
1494 This command is restricted and can only be issued on sockets configured for
1495 level "admin".
1496
1497enable health <backend>/<server>
1498 Resume a primary health check that was temporarily stopped. This will enable
1499 sending of health checks again. Please see "disable health" for details.
1500
1501 This command is restricted and can only be issued on sockets configured for
1502 level "admin".
1503
1504enable server <backend>/<server>
1505 If the server was previously marked as DOWN for maintenance, this marks the
1506 server UP and checks are re-enabled.
1507
1508 Both the backend and the server may be specified either by their name or by
1509 their numeric ID, prefixed with a sharp ('#').
1510
1511 This command is restricted and can only be issued on sockets configured for
1512 level "admin".
1513
1514get map <map> <value>
1515get acl <acl> <value>
1516 Lookup the value <value> in the map <map> or in the ACL <acl>. <map> or <acl>
1517 are the #<id> or the <file> returned by "show map" or "show acl". This command
1518 returns all the matching patterns associated with this map. This is useful for
1519 debugging maps and ACLs. The output format is composed by one line par
1520 matching type. Each line is composed by space-delimited series of words.
1521
1522 The first two words are:
1523
1524 <match method>: The match method applied. It can be "found", "bool",
1525 "int", "ip", "bin", "len", "str", "beg", "sub", "dir",
1526 "dom", "end" or "reg".
1527
1528 <match result>: The result. Can be "match" or "no-match".
1529
1530 The following words are returned only if the pattern matches an entry.
1531
1532 <index type>: "tree" or "list". The internal lookup algorithm.
1533
1534 <case>: "case-insensitive" or "case-sensitive". The
1535 interpretation of the case.
1536
1537 <entry matched>: match="<entry>". Return the matched pattern. It is
1538 useful with regular expressions.
1539
1540 The two last word are used to show the returned value and its type. With the
1541 "acl" case, the pattern doesn't exist.
1542
1543 return=nothing: No return because there are no "map".
1544 return="<value>": The value returned in the string format.
1545 return=cannot-display: The value cannot be converted as string.
1546
1547 type="<type>": The type of the returned sample.
1548
1549get weight <backend>/<server>
1550 Report the current weight and the initial weight of server <server> in
1551 backend <backend> or an error if either doesn't exist. The initial weight is
1552 the one that appears in the configuration file. Both are normally equal
1553 unless the current weight has been changed. Both the backend and the server
1554 may be specified either by their name or by their numeric ID, prefixed with a
1555 sharp ('#').
1556
1557help
1558 Print the list of known keywords and their basic usage. The same help screen
1559 is also displayed for unknown commands.
1560
1561prompt
1562 Toggle the prompt at the beginning of the line and enter or leave interactive
1563 mode. In interactive mode, the connection is not closed after a command
1564 completes. Instead, the prompt will appear again, indicating the user that
1565 the interpreter is waiting for a new command. The prompt consists in a right
1566 angle bracket followed by a space "> ". This mode is particularly convenient
1567 when one wants to periodically check information such as stats or errors.
1568 It is also a good idea to enter interactive mode before issuing a "help"
1569 command.
1570
1571quit
1572 Close the connection when in interactive mode.
1573
Olivier Houchard614f8d72017-03-14 20:08:46 +01001574set dynamic-cookie-key backend <backend> <value>
1575 Modify the secret key used to generate the dynamic persistent cookies.
1576 This will break the existing sessions.
1577
Willy Tarreau44aed902015-10-13 14:45:29 +02001578set map <map> [<key>|#<ref>] <value>
1579 Modify the value corresponding to each key <key> in a map <map>. <map> is the
1580 #<id> or <file> returned by "show map". If the <ref> is used in place of
1581 <key>, only the entry pointed by <ref> is changed. The new value is <value>.
1582
1583set maxconn frontend <frontend> <value>
1584 Dynamically change the specified frontend's maxconn setting. Any positive
1585 value is allowed including zero, but setting values larger than the global
1586 maxconn does not make much sense. If the limit is increased and connections
1587 were pending, they will immediately be accepted. If it is lowered to a value
1588 below the current number of connections, new connections acceptation will be
1589 delayed until the threshold is reached. The frontend might be specified by
1590 either its name or its numeric ID prefixed with a sharp ('#').
1591
Andrew Hayworthedb93a72015-10-27 21:46:25 +00001592set maxconn server <backend/server> <value>
1593 Dynamically change the specified server's maxconn setting. Any positive
1594 value is allowed including zero, but setting values larger than the global
1595 maxconn does not make much sense.
1596
Willy Tarreau44aed902015-10-13 14:45:29 +02001597set maxconn global <maxconn>
1598 Dynamically change the global maxconn setting within the range defined by the
1599 initial global maxconn setting. If it is increased and connections were
1600 pending, they will immediately be accepted. If it is lowered to a value below
1601 the current number of connections, new connections acceptation will be
1602 delayed until the threshold is reached. A value of zero restores the initial
1603 setting.
1604
1605set rate-limit connections global <value>
1606 Change the process-wide connection rate limit, which is set by the global
1607 'maxconnrate' setting. A value of zero disables the limitation. This limit
1608 applies to all frontends and the change has an immediate effect. The value
1609 is passed in number of connections per second.
1610
1611set rate-limit http-compression global <value>
1612 Change the maximum input compression rate, which is set by the global
1613 'maxcomprate' setting. A value of zero disables the limitation. The value is
1614 passed in number of kilobytes per second. The value is available in the "show
1615 info" on the line "CompressBpsRateLim" in bytes.
1616
1617set rate-limit sessions global <value>
1618 Change the process-wide session rate limit, which is set by the global
1619 'maxsessrate' setting. A value of zero disables the limitation. This limit
1620 applies to all frontends and the change has an immediate effect. The value
1621 is passed in number of sessions per second.
1622
1623set rate-limit ssl-sessions global <value>
1624 Change the process-wide SSL session rate limit, which is set by the global
1625 'maxsslrate' setting. A value of zero disables the limitation. This limit
1626 applies to all frontends and the change has an immediate effect. The value
1627 is passed in number of sessions per second sent to the SSL stack. It applies
1628 before the handshake in order to protect the stack against handshake abuses.
1629
Baptiste Assmann3749ebf2016-08-03 22:34:12 +02001630set server <backend>/<server> addr <ip4 or ip6 address> [port <port>]
Willy Tarreau44aed902015-10-13 14:45:29 +02001631 Replace the current IP address of a server by the one provided.
Baptiste Assmann3749ebf2016-08-03 22:34:12 +02001632 Optionnaly, the port can be changed using the 'port' parameter.
1633 Note that changing the port also support switching from/to port mapping
1634 (notation with +X or -Y), only if a port is configured for the health check.
Willy Tarreau44aed902015-10-13 14:45:29 +02001635
1636set server <backend>/<server> agent [ up | down ]
1637 Force a server's agent to a new state. This can be useful to immediately
1638 switch a server's state regardless of some slow agent checks for example.
1639 Note that the change is propagated to tracking servers if any.
1640
Misiek43972902017-01-09 09:53:06 +01001641set server <backend>/<server> agent-addr <addr>
1642 Change addr for servers agent checks. Allows to migrate agent-checks to
1643 another address at runtime. You can specify both IP and hostname, it will be
1644 resolved.
1645
1646set server <backend>/<server> agent-send <value>
1647 Change agent string sent to agent check target. Allows to update string while
1648 changing server address to keep those two matching.
1649
Willy Tarreau44aed902015-10-13 14:45:29 +02001650set server <backend>/<server> health [ up | stopping | down ]
1651 Force a server's health to a new state. This can be useful to immediately
1652 switch a server's state regardless of some slow health checks for example.
1653 Note that the change is propagated to tracking servers if any.
1654
Baptiste Assmann50946562016-08-31 23:26:29 +02001655set server <backend>/<server> check-port <port>
1656 Change the port used for health checking to <port>
1657
Willy Tarreau44aed902015-10-13 14:45:29 +02001658set server <backend>/<server> state [ ready | drain | maint ]
1659 Force a server's administrative state to a new state. This can be useful to
1660 disable load balancing and/or any traffic to a server. Setting the state to
1661 "ready" puts the server in normal mode, and the command is the equivalent of
1662 the "enable server" command. Setting the state to "maint" disables any traffic
1663 to the server as well as any health checks. This is the equivalent of the
1664 "disable server" command. Setting the mode to "drain" only removes the server
1665 from load balancing but still allows it to be checked and to accept new
1666 persistent connections. Changes are propagated to tracking servers if any.
1667
1668set server <backend>/<server> weight <weight>[%]
1669 Change a server's weight to the value passed in argument. This is the exact
1670 equivalent of the "set weight" command below.
1671
Frédéric Lécailleb418c122017-04-26 11:24:02 +02001672set server <backend>/<server> fqdn <FQDN>
1673 Change a server's FQDN to the value passed in argument.
1674
Willy Tarreau44aed902015-10-13 14:45:29 +02001675set ssl ocsp-response <response>
1676 This command is used to update an OCSP Response for a certificate (see "crt"
1677 on "bind" lines). Same controls are performed as during the initial loading of
1678 the response. The <response> must be passed as a base64 encoded string of the
Emmanuel Hocdet2c32d8f2017-05-22 14:58:00 +02001679 DER encoded response from the OCSP server. This command is not supported with
1680 BoringSSL.
Willy Tarreau44aed902015-10-13 14:45:29 +02001681
1682 Example:
1683 openssl ocsp -issuer issuer.pem -cert server.pem \
1684 -host ocsp.issuer.com:80 -respout resp.der
1685 echo "set ssl ocsp-response $(base64 -w 10000 resp.der)" | \
1686 socat stdio /var/run/haproxy.stat
1687
1688set ssl tls-key <id> <tlskey>
1689 Set the next TLS key for the <id> listener to <tlskey>. This key becomes the
1690 ultimate key, while the penultimate one is used for encryption (others just
1691 decrypt). The oldest TLS key present is overwritten. <id> is either a numeric
1692 #<id> or <file> returned by "show tls-keys". <tlskey> is a base64 encoded 48
1693 bit TLS ticket key (ex. openssl rand -base64 48).
1694
1695set table <table> key <key> [data.<data_type> <value>]*
1696 Create or update a stick-table entry in the table. If the key is not present,
1697 an entry is inserted. See stick-table in section 4.2 to find all possible
1698 values for <data_type>. The most likely use consists in dynamically entering
1699 entries for source IP addresses, with a flag in gpc0 to dynamically block an
1700 IP address or affect its quality of service. It is possible to pass multiple
1701 data_types in a single call.
1702
1703set timeout cli <delay>
1704 Change the CLI interface timeout for current connection. This can be useful
1705 during long debugging sessions where the user needs to constantly inspect
1706 some indicators without being disconnected. The delay is passed in seconds.
1707
1708set weight <backend>/<server> <weight>[%]
1709 Change a server's weight to the value passed in argument. If the value ends
1710 with the '%' sign, then the new weight will be relative to the initially
1711 configured weight. Absolute weights are permitted between 0 and 256.
1712 Relative weights must be positive with the resulting absolute weight is
1713 capped at 256. Servers which are part of a farm running a static
1714 load-balancing algorithm have stricter limitations because the weight
1715 cannot change once set. Thus for these servers, the only accepted values
1716 are 0 and 100% (or 0 and the initial weight). Changes take effect
1717 immediately, though certain LB algorithms require a certain amount of
1718 requests to consider changes. A typical usage of this command is to
1719 disable a server during an update by setting its weight to zero, then to
1720 enable it again after the update by setting it back to 100%. This command
1721 is restricted and can only be issued on sockets configured for level
1722 "admin". Both the backend and the server may be specified either by their
1723 name or by their numeric ID, prefixed with a sharp ('#').
1724
Willy Tarreaud6129fc2017-07-28 16:52:23 +02001725show acl [<acl>]
1726 Dump info about acl converters. Without argument, the list of all available
1727 acls is returned. If a <acl> is specified, its contents are dumped. <acl> if
1728 the #<id> or <file>. The dump format is the same than the map even for the
1729 sample value. The data returned are not a list of available ACL, but are the
1730 list of all patterns composing any ACL. Many of these patterns can be shared
1731 with maps.
1732
1733show backend
1734 Dump the list of backends available in the running process
1735
William Lallemand51132162016-12-16 16:38:58 +01001736show cli sockets
1737 List CLI sockets. The output format is composed of 3 fields separated by
1738 spaces. The first field is the socket address, it can be a unix socket, a
1739 ipv4 address:port couple or a ipv6 one. Socket of other types won't be dump.
1740 The second field describe the level of the socket: 'admin', 'user' or
1741 'operator'. The last field list the processes on which the socket is bound,
1742 separated by commas, it can be numbers or 'all'.
1743
1744 Example :
1745
1746 $ echo 'show cli sockets' | socat stdio /tmp/sock1
1747 # socket lvl processes
1748 /tmp/sock1 admin all
1749 127.0.0.1:9999 user 2,3,4
1750 127.0.0.2:9969 user 2
1751 [::1]:9999 operator 2
1752
Willy Tarreauae795722016-02-16 11:27:28 +01001753show env [<name>]
1754 Dump one or all environment variables known by the process. Without any
1755 argument, all variables are dumped. With an argument, only the specified
1756 variable is dumped if it exists. Otherwise "Variable not found" is emitted.
1757 Variables are dumped in the same format as they are stored or returned by the
1758 "env" utility, that is, "<name>=<value>". This can be handy when debugging
1759 certain configuration files making heavy use of environment variables to
1760 ensure that they contain the expected values. This command is restricted and
1761 can only be issued on sockets configured for levels "operator" or "admin".
1762
Willy Tarreau35069f82016-11-25 09:16:37 +01001763show errors [<iid>|<proxy>] [request|response]
Willy Tarreau44aed902015-10-13 14:45:29 +02001764 Dump last known request and response errors collected by frontends and
1765 backends. If <iid> is specified, the limit the dump to errors concerning
Willy Tarreau234ba2d2016-11-25 08:39:10 +01001766 either frontend or backend whose ID is <iid>. Proxy ID "-1" will cause
1767 all instances to be dumped. If a proxy name is specified instead, its ID
Willy Tarreau35069f82016-11-25 09:16:37 +01001768 will be used as the filter. If "request" or "response" is added after the
1769 proxy name or ID, only request or response errors will be dumped. This
1770 command is restricted and can only be issued on sockets configured for
1771 levels "operator" or "admin".
Willy Tarreau44aed902015-10-13 14:45:29 +02001772
1773 The errors which may be collected are the last request and response errors
1774 caused by protocol violations, often due to invalid characters in header
1775 names. The report precisely indicates what exact character violated the
1776 protocol. Other important information such as the exact date the error was
1777 detected, frontend and backend names, the server name (when known), the
1778 internal session ID and the source address which has initiated the session
1779 are reported too.
1780
1781 All characters are returned, and non-printable characters are encoded. The
1782 most common ones (\t = 9, \n = 10, \r = 13 and \e = 27) are encoded as one
1783 letter following a backslash. The backslash itself is encoded as '\\' to
1784 avoid confusion. Other non-printable characters are encoded '\xNN' where
1785 NN is the two-digits hexadecimal representation of the character's ASCII
1786 code.
1787
1788 Lines are prefixed with the position of their first character, starting at 0
1789 for the beginning of the buffer. At most one input line is printed per line,
1790 and large lines will be broken into multiple consecutive output lines so that
1791 the output never goes beyond 79 characters wide. It is easy to detect if a
1792 line was broken, because it will not end with '\n' and the next line's offset
1793 will be followed by a '+' sign, indicating it is a continuation of previous
1794 line.
1795
1796 Example :
Willy Tarreau35069f82016-11-25 09:16:37 +01001797 $ echo "show errors -1 response" | socat stdio /tmp/sock1
Willy Tarreau44aed902015-10-13 14:45:29 +02001798 >>> [04/Mar/2009:15:46:56.081] backend http-in (#2) : invalid response
1799 src 127.0.0.1, session #54, frontend fe-eth0 (#1), server s2 (#1)
1800 response length 213 bytes, error at position 23:
1801
1802 00000 HTTP/1.0 200 OK\r\n
1803 00017 header/bizarre:blah\r\n
1804 00038 Location: blah\r\n
1805 00054 Long-line: this is a very long line which should b
1806 00104+ e broken into multiple lines on the output buffer,
1807 00154+ otherwise it would be too large to print in a ter
1808 00204+ minal\r\n
1809 00211 \r\n
1810
1811 In the example above, we see that the backend "http-in" which has internal
1812 ID 2 has blocked an invalid response from its server s2 which has internal
1813 ID 1. The request was on session 54 initiated by source 127.0.0.1 and
1814 received by frontend fe-eth0 whose ID is 1. The total response length was
1815 213 bytes when the error was detected, and the error was at byte 23. This
1816 is the slash ('/') in header name "header/bizarre", which is not a valid
1817 HTTP character for a header name.
1818
Willy Tarreau7a4a0ac2017-07-25 19:32:50 +02001819show fd [<fd>]
1820 Dump the list of either all open file descriptors or just the one number <fd>
1821 if specified. This is only aimed at developers who need to observe internal
1822 states in order to debug complex issues such as abnormal CPU usages. One fd
1823 is reported per lines, and for each of them, its state in the poller using
1824 upper case letters for enabled flags and lower case for disabled flags, using
1825 "P" for "polled", "R" for "ready", "A" for "active", the events status using
1826 "H" for "hangup", "E" for "error", "O" for "output", "P" for "priority" and
1827 "I" for "input", a few other flags like "N" for "new" (just added into the fd
1828 cache), "U" for "updated" (received an update in the fd cache), "L" for
1829 "linger_risk", "C" for "cloned", then the cached entry position, the pointer
1830 to the internal owner, the pointer to the I/O callback and its name when
1831 known. When the owner is a connection, the connection flags, and the target
1832 are reported (frontend, proxy or server). When the owner is a listener, the
1833 listener's state and its frontend are reported. There is no point in using
1834 this command without a good knowledge of the internals. It's worth noting
1835 that the output format may evolve over time so this output must not be parsed
1836 by tools designed to be durable.
1837
Simon Horman05ee2132017-01-04 09:37:25 +01001838show info [typed|json]
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001839 Dump info about haproxy status on current process. If "typed" is passed as an
1840 optional argument, field numbers, names and types are emitted as well so that
1841 external monitoring products can easily retrieve, possibly aggregate, then
1842 report information found in fields they don't know. Each field is dumped on
Simon Horman05ee2132017-01-04 09:37:25 +01001843 its own line. If "json" is passed as an optional argument then
1844 information provided by "typed" output is provided in JSON format as a
1845 list of JSON objects. By default, the format contains only two columns
1846 delimited by a colon (':'). The left one is the field name and the right
1847 one is the value. It is very important to note that in typed output
1848 format, the dump for a single object is contiguous so that there is no
1849 need for a consumer to store everything at once.
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001850
1851 When using the typed output format, each line is made of 4 columns delimited
1852 by colons (':'). The first column is a dot-delimited series of 3 elements. The
1853 first element is the numeric position of the field in the list (starting at
1854 zero). This position shall not change over time, but holes are to be expected,
1855 depending on build options or if some fields are deleted in the future. The
1856 second element is the field name as it appears in the default "show info"
1857 output. The third element is the relative process number starting at 1.
1858
1859 The rest of the line starting after the first colon follows the "typed output
1860 format" described in the section above. In short, the second column (after the
1861 first ':') indicates the origin, nature and scope of the variable. The third
1862 column indicates the type of the field, among "s32", "s64", "u32", "u64" and
1863 "str". Then the fourth column is the value itself, which the consumer knows
1864 how to parse thanks to column 3 and how to process thanks to column 2.
1865
1866 Thus the overall line format in typed mode is :
1867
1868 <field_pos>.<field_name>.<process_num>:<tags>:<type>:<value>
1869
1870 Example :
1871
1872 > show info
1873 Name: HAProxy
1874 Version: 1.7-dev1-de52ea-146
1875 Release_date: 2016/03/11
1876 Nbproc: 1
1877 Process_num: 1
1878 Pid: 28105
1879 Uptime: 0d 0h00m04s
1880 Uptime_sec: 4
1881 Memmax_MB: 0
1882 PoolAlloc_MB: 0
1883 PoolUsed_MB: 0
1884 PoolFailed: 0
1885 (...)
1886
1887 > show info typed
1888 0.Name.1:POS:str:HAProxy
1889 1.Version.1:POS:str:1.7-dev1-de52ea-146
1890 2.Release_date.1:POS:str:2016/03/11
1891 3.Nbproc.1:CGS:u32:1
1892 4.Process_num.1:KGP:u32:1
1893 5.Pid.1:SGP:u32:28105
1894 6.Uptime.1:MDP:str:0d 0h00m08s
1895 7.Uptime_sec.1:MDP:u32:8
1896 8.Memmax_MB.1:CLP:u32:0
1897 9.PoolAlloc_MB.1:MGP:u32:0
1898 10.PoolUsed_MB.1:MGP:u32:0
1899 11.PoolFailed.1:MCP:u32:0
1900 (...)
1901
Simon Horman1084a362016-11-21 17:00:24 +01001902 In the typed format, the presence of the process ID at the end of the
1903 first column makes it very easy to visually aggregate outputs from
1904 multiple processes.
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001905 Example :
1906
1907 $ ( echo show info typed | socat /var/run/haproxy.sock1 ; \
1908 echo show info typed | socat /var/run/haproxy.sock2 ) | \
1909 sort -t . -k 1,1n -k 2,2 -k 3,3n
1910 0.Name.1:POS:str:HAProxy
1911 0.Name.2:POS:str:HAProxy
1912 1.Version.1:POS:str:1.7-dev1-868ab3-148
1913 1.Version.2:POS:str:1.7-dev1-868ab3-148
1914 2.Release_date.1:POS:str:2016/03/11
1915 2.Release_date.2:POS:str:2016/03/11
1916 3.Nbproc.1:CGS:u32:2
1917 3.Nbproc.2:CGS:u32:2
1918 4.Process_num.1:KGP:u32:1
1919 4.Process_num.2:KGP:u32:2
1920 5.Pid.1:SGP:u32:30120
1921 5.Pid.2:SGP:u32:30121
1922 6.Uptime.1:MDP:str:0d 0h01m28s
1923 6.Uptime.2:MDP:str:0d 0h01m28s
1924 (...)
Willy Tarreau44aed902015-10-13 14:45:29 +02001925
Simon Horman05ee2132017-01-04 09:37:25 +01001926 The format of JSON output is described in a schema which may be output
Simon Horman6f6bb382017-01-04 09:37:26 +01001927 using "show schema json".
Simon Horman05ee2132017-01-04 09:37:25 +01001928
1929 The JSON output contains no extra whitespace in order to reduce the
1930 volume of output. For human consumption passing the output through a
1931 pretty printer may be helpful. Example :
1932
1933 $ echo "show info json" | socat /var/run/haproxy.sock stdio | \
1934 python -m json.tool
1935
Simon Horman6f6bb382017-01-04 09:37:26 +01001936 The JSON output contains no extra whitespace in order to reduce the
1937 volume of output. For human consumption passing the output through a
1938 pretty printer may be helpful. Example :
1939
1940 $ echo "show info json" | socat /var/run/haproxy.sock stdio | \
1941 python -m json.tool
1942
Willy Tarreau44aed902015-10-13 14:45:29 +02001943show map [<map>]
1944 Dump info about map converters. Without argument, the list of all available
1945 maps is returned. If a <map> is specified, its contents are dumped. <map> is
1946 the #<id> or <file>. The first column is a unique identifier. It can be used
1947 as reference for the operation "del map" and "set map". The second column is
1948 the pattern and the third column is the sample if available. The data returned
1949 are not directly a list of available maps, but are the list of all patterns
1950 composing any map. Many of these patterns can be shared with ACL.
1951
Willy Tarreau44aed902015-10-13 14:45:29 +02001952show pools
1953 Dump the status of internal memory pools. This is useful to track memory
1954 usage when suspecting a memory leak for example. It does exactly the same
1955 as the SIGQUIT when running in foreground except that it does not flush
1956 the pools.
1957
1958show servers state [<backend>]
1959 Dump the state of the servers found in the running configuration. A backend
1960 name or identifier may be provided to limit the output to this backend only.
1961
1962 The dump has the following format:
1963 - first line contains the format version (1 in this specification);
1964 - second line contains the column headers, prefixed by a sharp ('#');
1965 - third line and next ones contain data;
1966 - each line starting by a sharp ('#') is considered as a comment.
1967
Dan Lloyd8e48b872016-07-01 21:01:18 -04001968 Since multiple versions of the output may co-exist, below is the list of
Willy Tarreau44aed902015-10-13 14:45:29 +02001969 fields and their order per file format version :
1970 1:
1971 be_id: Backend unique id.
1972 be_name: Backend label.
1973 srv_id: Server unique id (in the backend).
1974 srv_name: Server label.
1975 srv_addr: Server IP address.
1976 srv_op_state: Server operational state (UP/DOWN/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01001977 0 = SRV_ST_STOPPED
1978 The server is down.
1979 1 = SRV_ST_STARTING
1980 The server is warming up (up but
1981 throttled).
1982 2 = SRV_ST_RUNNING
1983 The server is fully up.
1984 3 = SRV_ST_STOPPING
1985 The server is up but soft-stopping
1986 (eg: 404).
Willy Tarreau44aed902015-10-13 14:45:29 +02001987 srv_admin_state: Server administrative state (MAINT/DRAIN/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01001988 The state is actually a mask of values :
1989 0x01 = SRV_ADMF_FMAINT
1990 The server was explicitly forced into
1991 maintenance.
1992 0x02 = SRV_ADMF_IMAINT
1993 The server has inherited the maintenance
1994 status from a tracked server.
1995 0x04 = SRV_ADMF_CMAINT
1996 The server is in maintenance because of
1997 the configuration.
1998 0x08 = SRV_ADMF_FDRAIN
1999 The server was explicitly forced into
2000 drain state.
2001 0x10 = SRV_ADMF_IDRAIN
2002 The server has inherited the drain status
2003 from a tracked server.
Baptiste Assmann89aa7f32016-11-02 21:31:27 +01002004 0x20 = SRV_ADMF_RMAINT
2005 The server is in maintenance because of an
2006 IP address resolution failure.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02002007 0x40 = SRV_ADMF_HMAINT
2008 The server FQDN was set from stats socket.
2009
Willy Tarreau44aed902015-10-13 14:45:29 +02002010 srv_uweight: User visible server's weight.
2011 srv_iweight: Server's initial weight.
2012 srv_time_since_last_change: Time since last operational change.
2013 srv_check_status: Last health check status.
2014 srv_check_result: Last check result (FAILED/PASSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002015 0 = CHK_RES_UNKNOWN
2016 Initialized to this by default.
2017 1 = CHK_RES_NEUTRAL
2018 Valid check but no status information.
2019 2 = CHK_RES_FAILED
2020 Check failed.
2021 3 = CHK_RES_PASSED
2022 Check succeeded and server is fully up
2023 again.
2024 4 = CHK_RES_CONDPASS
2025 Check reports the server doesn't want new
2026 sessions.
Willy Tarreau44aed902015-10-13 14:45:29 +02002027 srv_check_health: Checks rise / fall current counter.
2028 srv_check_state: State of the check (ENABLED/PAUSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002029 The state is actually a mask of values :
2030 0x01 = CHK_ST_INPROGRESS
2031 A check is currently running.
2032 0x02 = CHK_ST_CONFIGURED
2033 This check is configured and may be
2034 enabled.
2035 0x04 = CHK_ST_ENABLED
2036 This check is currently administratively
2037 enabled.
2038 0x08 = CHK_ST_PAUSED
2039 Checks are paused because of maintenance
2040 (health only).
Willy Tarreau44aed902015-10-13 14:45:29 +02002041 srv_agent_state: State of the agent check (ENABLED/PAUSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002042 This state uses the same mask values as
2043 "srv_check_state", adding this specific one :
2044 0x10 = CHK_ST_AGENT
2045 Check is an agent check (otherwise it's a
2046 health check).
Willy Tarreau44aed902015-10-13 14:45:29 +02002047 bk_f_forced_id: Flag to know if the backend ID is forced by
2048 configuration.
2049 srv_f_forced_id: Flag to know if the server's ID is forced by
2050 configuration.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02002051 srv_fqdn: Server FQDN.
Frédéric Lécaille31694712017-08-01 08:47:19 +02002052 srv_port: Server port.
Willy Tarreau44aed902015-10-13 14:45:29 +02002053
2054show sess
2055 Dump all known sessions. Avoid doing this on slow connections as this can
2056 be huge. This command is restricted and can only be issued on sockets
2057 configured for levels "operator" or "admin".
2058
2059show sess <id>
2060 Display a lot of internal information about the specified session identifier.
2061 This identifier is the first field at the beginning of the lines in the dumps
2062 of "show sess" (it corresponds to the session pointer). Those information are
2063 useless to most users but may be used by haproxy developers to troubleshoot a
2064 complex bug. The output format is intentionally not documented so that it can
2065 freely evolve depending on demands. You may find a description of all fields
2066 returned in src/dumpstats.c
2067
2068 The special id "all" dumps the states of all sessions, which must be avoided
2069 as much as possible as it is highly CPU intensive and can take a lot of time.
2070
Simon Horman05ee2132017-01-04 09:37:25 +01002071show stat [{<iid>|<proxy>} <type> <sid>] [typed|json]
2072 Dump statistics using the CSV format; using the extended typed output
2073 format described in the section above if "typed" is passed after the
2074 other arguments; or in JSON if "json" is passed after the other arguments
2075 . By passing <id>, <type> and <sid>, it is possible to dump only selected
2076 items :
Willy Tarreaua1b1ed52016-11-25 08:50:58 +01002077 - <iid> is a proxy ID, -1 to dump everything. Alternatively, a proxy name
2078 <proxy> may be specified. In this case, this proxy's ID will be used as
2079 the ID selector.
Willy Tarreau44aed902015-10-13 14:45:29 +02002080 - <type> selects the type of dumpable objects : 1 for frontends, 2 for
2081 backends, 4 for servers, -1 for everything. These values can be ORed,
2082 for example:
2083 1 + 2 = 3 -> frontend + backend.
2084 1 + 2 + 4 = 7 -> frontend + backend + server.
2085 - <sid> is a server ID, -1 to dump everything from the selected proxy.
2086
2087 Example :
2088 $ echo "show info;show stat" | socat stdio unix-connect:/tmp/sock1
2089 >>> Name: HAProxy
2090 Version: 1.4-dev2-49
2091 Release_date: 2009/09/23
2092 Nbproc: 1
2093 Process_num: 1
2094 (...)
2095
2096 # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq, (...)
2097 stats,FRONTEND,,,0,0,1000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,1,0, (...)
2098 stats,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,250,(...)
2099 (...)
2100 www1,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,0,250, (...)
2101
2102 $
2103
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002104 In this example, two commands have been issued at once. That way it's easy to
2105 find which process the stats apply to in multi-process mode. This is not
2106 needed in the typed output format as the process number is reported on each
2107 line. Notice the empty line after the information output which marks the end
2108 of the first block. A similar empty line appears at the end of the second
2109 block (stats) so that the reader knows the output has not been truncated.
2110
2111 When "typed" is specified, the output format is more suitable to monitoring
2112 tools because it provides numeric positions and indicates the type of each
2113 output field. Each value stands on its own line with process number, element
2114 number, nature, origin and scope. This same format is available via the HTTP
2115 stats by passing ";typed" after the URI. It is very important to note that in
Dan Lloyd8e48b872016-07-01 21:01:18 -04002116 typed output format, the dump for a single object is contiguous so that there
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002117 is no need for a consumer to store everything at once.
2118
2119 When using the typed output format, each line is made of 4 columns delimited
2120 by colons (':'). The first column is a dot-delimited series of 5 elements. The
2121 first element is a letter indicating the type of the object being described.
2122 At the moment the following object types are known : 'F' for a frontend, 'B'
2123 for a backend, 'L' for a listener, and 'S' for a server. The second element
2124 The second element is a positive integer representing the unique identifier of
2125 the proxy the object belongs to. It is equivalent to the "iid" column of the
2126 CSV output and matches the value in front of the optional "id" directive found
2127 in the frontend or backend section. The third element is a positive integer
2128 containing the unique object identifier inside the proxy, and corresponds to
2129 the "sid" column of the CSV output. ID 0 is reported when dumping a frontend
2130 or a backend. For a listener or a server, this corresponds to their respective
2131 ID inside the proxy. The fourth element is the numeric position of the field
2132 in the list (starting at zero). This position shall not change over time, but
2133 holes are to be expected, depending on build options or if some fields are
2134 deleted in the future. The fifth element is the field name as it appears in
2135 the CSV output. The sixth element is a positive integer and is the relative
2136 process number starting at 1.
2137
2138 The rest of the line starting after the first colon follows the "typed output
2139 format" described in the section above. In short, the second column (after the
2140 first ':') indicates the origin, nature and scope of the variable. The third
2141 column indicates the type of the field, among "s32", "s64", "u32", "u64" and
2142 "str". Then the fourth column is the value itself, which the consumer knows
2143 how to parse thanks to column 3 and how to process thanks to column 2.
2144
2145 Thus the overall line format in typed mode is :
2146
2147 <obj>.<px_id>.<id>.<fpos>.<fname>.<process_num>:<tags>:<type>:<value>
2148
2149 Here's an example of typed output format :
2150
2151 $ echo "show stat typed" | socat stdio unix-connect:/tmp/sock1
2152 F.2.0.0.pxname.1:MGP:str:private-frontend
2153 F.2.0.1.svname.1:MGP:str:FRONTEND
2154 F.2.0.8.bin.1:MGP:u64:0
2155 F.2.0.9.bout.1:MGP:u64:0
2156 F.2.0.40.hrsp_2xx.1:MGP:u64:0
2157 L.2.1.0.pxname.1:MGP:str:private-frontend
2158 L.2.1.1.svname.1:MGP:str:sock-1
2159 L.2.1.17.status.1:MGP:str:OPEN
2160 L.2.1.73.addr.1:MGP:str:0.0.0.0:8001
2161 S.3.13.60.rtime.1:MCP:u32:0
2162 S.3.13.61.ttime.1:MCP:u32:0
2163 S.3.13.62.agent_status.1:MGP:str:L4TOUT
2164 S.3.13.64.agent_duration.1:MGP:u64:2001
2165 S.3.13.65.check_desc.1:MCP:str:Layer4 timeout
2166 S.3.13.66.agent_desc.1:MCP:str:Layer4 timeout
2167 S.3.13.67.check_rise.1:MCP:u32:2
2168 S.3.13.68.check_fall.1:MCP:u32:3
2169 S.3.13.69.check_health.1:SGP:u32:0
2170 S.3.13.70.agent_rise.1:MaP:u32:1
2171 S.3.13.71.agent_fall.1:SGP:u32:1
2172 S.3.13.72.agent_health.1:SGP:u32:1
2173 S.3.13.73.addr.1:MCP:str:1.255.255.255:8888
2174 S.3.13.75.mode.1:MAP:str:http
2175 B.3.0.0.pxname.1:MGP:str:private-backend
2176 B.3.0.1.svname.1:MGP:str:BACKEND
2177 B.3.0.2.qcur.1:MGP:u32:0
2178 B.3.0.3.qmax.1:MGP:u32:0
2179 B.3.0.4.scur.1:MGP:u32:0
2180 B.3.0.5.smax.1:MGP:u32:0
2181 B.3.0.6.slim.1:MGP:u32:1000
2182 B.3.0.55.lastsess.1:MMP:s32:-1
2183 (...)
2184
Simon Horman1084a362016-11-21 17:00:24 +01002185 In the typed format, the presence of the process ID at the end of the
2186 first column makes it very easy to visually aggregate outputs from
2187 multiple processes, as show in the example below where each line appears
2188 for each process :
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002189
2190 $ ( echo show stat typed | socat /var/run/haproxy.sock1 - ; \
2191 echo show stat typed | socat /var/run/haproxy.sock2 - ) | \
2192 sort -t . -k 1,1 -k 2,2n -k 3,3n -k 4,4n -k 5,5 -k 6,6n
2193 B.3.0.0.pxname.1:MGP:str:private-backend
2194 B.3.0.0.pxname.2:MGP:str:private-backend
2195 B.3.0.1.svname.1:MGP:str:BACKEND
2196 B.3.0.1.svname.2:MGP:str:BACKEND
2197 B.3.0.2.qcur.1:MGP:u32:0
2198 B.3.0.2.qcur.2:MGP:u32:0
2199 B.3.0.3.qmax.1:MGP:u32:0
2200 B.3.0.3.qmax.2:MGP:u32:0
2201 B.3.0.4.scur.1:MGP:u32:0
2202 B.3.0.4.scur.2:MGP:u32:0
2203 B.3.0.5.smax.1:MGP:u32:0
2204 B.3.0.5.smax.2:MGP:u32:0
2205 B.3.0.6.slim.1:MGP:u32:1000
2206 B.3.0.6.slim.2:MGP:u32:1000
2207 (...)
Willy Tarreau44aed902015-10-13 14:45:29 +02002208
Simon Horman05ee2132017-01-04 09:37:25 +01002209 The format of JSON output is described in a schema which may be output
Simon Horman6f6bb382017-01-04 09:37:26 +01002210 using "show schema json".
2211
2212 The JSON output contains no extra whitespace in order to reduce the
2213 volume of output. For human consumption passing the output through a
2214 pretty printer may be helpful. Example :
2215
2216 $ echo "show stat json" | socat /var/run/haproxy.sock stdio | \
2217 python -m json.tool
Simon Horman05ee2132017-01-04 09:37:25 +01002218
2219 The JSON output contains no extra whitespace in order to reduce the
2220 volume of output. For human consumption passing the output through a
2221 pretty printer may be helpful. Example :
2222
2223 $ echo "show stat json" | socat /var/run/haproxy.sock stdio | \
2224 python -m json.tool
2225
Willy Tarreau44aed902015-10-13 14:45:29 +02002226show stat resolvers [<resolvers section id>]
2227 Dump statistics for the given resolvers section, or all resolvers sections
2228 if no section is supplied.
2229
2230 For each name server, the following counters are reported:
2231 sent: number of DNS requests sent to this server
2232 valid: number of DNS valid responses received from this server
2233 update: number of DNS responses used to update the server's IP address
2234 cname: number of CNAME responses
2235 cname_error: CNAME errors encountered with this server
2236 any_err: number of empty response (IE: server does not support ANY type)
2237 nx: non existent domain response received from this server
2238 timeout: how many time this server did not answer in time
2239 refused: number of requests refused by this server
2240 other: any other DNS errors
2241 invalid: invalid DNS response (from a protocol point of view)
2242 too_big: too big response
2243 outdated: number of response arrived too late (after an other name server)
2244
2245show table
2246 Dump general information on all known stick-tables. Their name is returned
2247 (the name of the proxy which holds them), their type (currently zero, always
2248 IP), their size in maximum possible number of entries, and the number of
2249 entries currently in use.
2250
2251 Example :
2252 $ echo "show table" | socat stdio /tmp/sock1
2253 >>> # table: front_pub, type: ip, size:204800, used:171454
2254 >>> # table: back_rdp, type: ip, size:204800, used:0
2255
2256show table <name> [ data.<type> <operator> <value> ] | [ key <key> ]
2257 Dump contents of stick-table <name>. In this mode, a first line of generic
2258 information about the table is reported as with "show table", then all
2259 entries are dumped. Since this can be quite heavy, it is possible to specify
2260 a filter in order to specify what entries to display.
2261
2262 When the "data." form is used the filter applies to the stored data (see
2263 "stick-table" in section 4.2). A stored data type must be specified
2264 in <type>, and this data type must be stored in the table otherwise an
2265 error is reported. The data is compared according to <operator> with the
2266 64-bit integer <value>. Operators are the same as with the ACLs :
2267
2268 - eq : match entries whose data is equal to this value
2269 - ne : match entries whose data is not equal to this value
2270 - le : match entries whose data is less than or equal to this value
2271 - ge : match entries whose data is greater than or equal to this value
2272 - lt : match entries whose data is less than this value
2273 - gt : match entries whose data is greater than this value
2274
2275
2276 When the key form is used the entry <key> is shown. The key must be of the
2277 same type as the table, which currently is limited to IPv4, IPv6, integer,
2278 and string.
2279
2280 Example :
2281 $ echo "show table http_proxy" | socat stdio /tmp/sock1
2282 >>> # table: http_proxy, type: ip, size:204800, used:2
2283 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
2284 bytes_out_rate(60000)=187
2285 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2286 bytes_out_rate(60000)=191
2287
2288 $ echo "show table http_proxy data.gpc0 gt 0" | socat stdio /tmp/sock1
2289 >>> # table: http_proxy, type: ip, size:204800, used:2
2290 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2291 bytes_out_rate(60000)=191
2292
2293 $ echo "show table http_proxy data.conn_rate gt 5" | \
2294 socat stdio /tmp/sock1
2295 >>> # table: http_proxy, type: ip, size:204800, used:2
2296 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2297 bytes_out_rate(60000)=191
2298
2299 $ echo "show table http_proxy key 127.0.0.2" | \
2300 socat stdio /tmp/sock1
2301 >>> # table: http_proxy, type: ip, size:204800, used:2
2302 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2303 bytes_out_rate(60000)=191
2304
2305 When the data criterion applies to a dynamic value dependent on time such as
2306 a bytes rate, the value is dynamically computed during the evaluation of the
2307 entry in order to decide whether it has to be dumped or not. This means that
2308 such a filter could match for some time then not match anymore because as
2309 time goes, the average event rate drops.
2310
2311 It is possible to use this to extract lists of IP addresses abusing the
2312 service, in order to monitor them or even blacklist them in a firewall.
2313 Example :
2314 $ echo "show table http_proxy data.gpc0 gt 0" \
2315 | socat stdio /tmp/sock1 \
2316 | fgrep 'key=' | cut -d' ' -f2 | cut -d= -f2 > abusers-ip.txt
2317 ( or | awk '/key/{ print a[split($2,a,"=")]; }' )
2318
William Lallemandbb933462016-05-31 21:09:53 +02002319show tls-keys [id|*]
2320 Dump all loaded TLS ticket keys references. The TLS ticket key reference ID
2321 and the file from which the keys have been loaded is shown. Both of those
2322 can be used to update the TLS keys using "set ssl tls-key". If an ID is
2323 specified as parameter, it will dump the tickets, using * it will dump every
2324 keys from every references.
Willy Tarreau44aed902015-10-13 14:45:29 +02002325
Simon Horman6f6bb382017-01-04 09:37:26 +01002326show schema json
2327 Dump the schema used for the output of "show info json" and "show stat json".
2328
2329 The contains no extra whitespace in order to reduce the volume of output.
2330 For human consumption passing the output through a pretty printer may be
2331 helpful. Example :
2332
2333 $ echo "show schema json" | socat /var/run/haproxy.sock stdio | \
2334 python -m json.tool
2335
2336 The schema follows "JSON Schema" (json-schema.org) and accordingly
2337 verifiers may be used to verify the output of "show info json" and "show
2338 stat json" against the schema.
2339
2340
Willy Tarreau44aed902015-10-13 14:45:29 +02002341shutdown frontend <frontend>
2342 Completely delete the specified frontend. All the ports it was bound to will
2343 be released. It will not be possible to enable the frontend anymore after
2344 this operation. This is intended to be used in environments where stopping a
2345 proxy is not even imaginable but a misconfigured proxy must be fixed. That
2346 way it's possible to release the port and bind it into another process to
2347 restore operations. The frontend will not appear at all on the stats page
2348 once it is terminated.
2349
2350 The frontend may be specified either by its name or by its numeric ID,
2351 prefixed with a sharp ('#').
2352
2353 This command is restricted and can only be issued on sockets configured for
2354 level "admin".
2355
2356shutdown session <id>
2357 Immediately terminate the session matching the specified session identifier.
2358 This identifier is the first field at the beginning of the lines in the dumps
2359 of "show sess" (it corresponds to the session pointer). This can be used to
2360 terminate a long-running session without waiting for a timeout or when an
2361 endless transfer is ongoing. Such terminated sessions are reported with a 'K'
2362 flag in the logs.
2363
2364shutdown sessions server <backend>/<server>
2365 Immediately terminate all the sessions attached to the specified server. This
2366 can be used to terminate long-running sessions after a server is put into
2367 maintenance mode, for instance. Such terminated sessions are reported with a
2368 'K' flag in the logs.
2369
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002370
237110. Tricks for easier configuration management
2372----------------------------------------------
2373
2374It is very common that two HAProxy nodes constituting a cluster share exactly
2375the same configuration modulo a few addresses. Instead of having to maintain a
2376duplicate configuration for each node, which will inevitably diverge, it is
2377possible to include environment variables in the configuration. Thus multiple
2378configuration may share the exact same file with only a few different system
2379wide environment variables. This started in version 1.5 where only addresses
2380were allowed to include environment variables, and 1.6 goes further by
2381supporting environment variables everywhere. The syntax is the same as in the
2382UNIX shell, a variable starts with a dollar sign ('$'), followed by an opening
2383curly brace ('{'), then the variable name followed by the closing brace ('}').
2384Except for addresses, environment variables are only interpreted in arguments
2385surrounded with double quotes (this was necessary not to break existing setups
2386using regular expressions involving the dollar symbol).
2387
2388Environment variables also make it convenient to write configurations which are
2389expected to work on various sites where only the address changes. It can also
2390permit to remove passwords from some configs. Example below where the the file
2391"site1.env" file is sourced by the init script upon startup :
2392
2393 $ cat site1.env
2394 LISTEN=192.168.1.1
2395 CACHE_PFX=192.168.11
2396 SERVER_PFX=192.168.22
2397 LOGGER=192.168.33.1
2398 STATSLP=admin:pa$$w0rd
2399 ABUSERS=/etc/haproxy/abuse.lst
2400 TIMEOUT=10s
2401
2402 $ cat haproxy.cfg
2403 global
2404 log "${LOGGER}:514" local0
2405
2406 defaults
2407 mode http
2408 timeout client "${TIMEOUT}"
2409 timeout server "${TIMEOUT}"
2410 timeout connect 5s
2411
2412 frontend public
2413 bind "${LISTEN}:80"
2414 http-request reject if { src -f "${ABUSERS}" }
2415 stats uri /stats
2416 stats auth "${STATSLP}"
2417 use_backend cache if { path_end .jpg .css .ico }
2418 default_backend server
2419
2420 backend cache
2421 server cache1 "${CACHE_PFX}.1:18080" check
2422 server cache2 "${CACHE_PFX}.2:18080" check
2423
2424 backend server
2425 server cache1 "${SERVER_PFX}.1:8080" check
2426 server cache2 "${SERVER_PFX}.2:8080" check
2427
2428
242911. Well-known traps to avoid
2430-----------------------------
2431
2432Once in a while, someone reports that after a system reboot, the haproxy
2433service wasn't started, and that once they start it by hand it works. Most
2434often, these people are running a clustered IP address mechanism such as
2435keepalived, to assign the service IP address to the master node only, and while
2436it used to work when they used to bind haproxy to address 0.0.0.0, it stopped
2437working after they bound it to the virtual IP address. What happens here is
2438that when the service starts, the virtual IP address is not yet owned by the
2439local node, so when HAProxy wants to bind to it, the system rejects this
2440because it is not a local IP address. The fix doesn't consist in delaying the
2441haproxy service startup (since it wouldn't stand a restart), but instead to
2442properly configure the system to allow binding to non-local addresses. This is
2443easily done on Linux by setting the net.ipv4.ip_nonlocal_bind sysctl to 1. This
2444is also needed in order to transparently intercept the IP traffic that passes
2445through HAProxy for a specific target address.
2446
2447Multi-process configurations involving source port ranges may apparently seem
2448to work but they will cause some random failures under high loads because more
2449than one process may try to use the same source port to connect to the same
2450server, which is not possible. The system will report an error and a retry will
2451happen, picking another port. A high value in the "retries" parameter may hide
2452the effect to a certain extent but this also comes with increased CPU usage and
2453processing time. Logs will also report a certain number of retries. For this
2454reason, port ranges should be avoided in multi-process configurations.
2455
Dan Lloyd8e48b872016-07-01 21:01:18 -04002456Since HAProxy uses SO_REUSEPORT and supports having multiple independent
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002457processes bound to the same IP:port, during troubleshooting it can happen that
2458an old process was not stopped before a new one was started. This provides
2459absurd test results which tend to indicate that any change to the configuration
2460is ignored. The reason is that in fact even the new process is restarted with a
2461new configuration, the old one also gets some incoming connections and
2462processes them, returning unexpected results. When in doubt, just stop the new
2463process and try again. If it still works, it very likely means that an old
2464process remains alive and has to be stopped. Linux's "netstat -lntp" is of good
2465help here.
2466
2467When adding entries to an ACL from the command line (eg: when blacklisting a
2468source address), it is important to keep in mind that these entries are not
2469synchronized to the file and that if someone reloads the configuration, these
2470updates will be lost. While this is often the desired effect (for blacklisting)
2471it may not necessarily match expectations when the change was made as a fix for
2472a problem. See the "add acl" action of the CLI interface.
2473
2474
247512. Debugging and performance issues
2476------------------------------------
2477
2478When HAProxy is started with the "-d" option, it will stay in the foreground
2479and will print one line per event, such as an incoming connection, the end of a
2480connection, and for each request or response header line seen. This debug
2481output is emitted before the contents are processed, so they don't consider the
2482local modifications. The main use is to show the request and response without
2483having to run a network sniffer. The output is less readable when multiple
2484connections are handled in parallel, though the "debug2ansi" and "debug2html"
2485scripts found in the examples/ directory definitely help here by coloring the
2486output.
2487
2488If a request or response is rejected because HAProxy finds it is malformed, the
2489best thing to do is to connect to the CLI and issue "show errors", which will
2490report the last captured faulty request and response for each frontend and
2491backend, with all the necessary information to indicate precisely the first
2492character of the input stream that was rejected. This is sometimes needed to
2493prove to customers or to developers that a bug is present in their code. In
2494this case it is often possible to relax the checks (but still keep the
2495captures) using "option accept-invalid-http-request" or its equivalent for
2496responses coming from the server "option accept-invalid-http-response". Please
2497see the configuration manual for more details.
2498
2499Example :
2500
2501 > show errors
2502 Total events captured on [13/Oct/2015:13:43:47.169] : 1
2503
2504 [13/Oct/2015:13:43:40.918] frontend HAProxyLocalStats (#2): invalid request
2505 backend <NONE> (#-1), server <NONE> (#-1), event #0
2506 src 127.0.0.1:51981, session #0, session flags 0x00000080
2507 HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000
2508 HTTP chunk len 0 bytes, HTTP body len 0 bytes
2509 buffer flags 0x00808002, out 0 bytes, total 31 bytes
2510 pending 31 bytes, wrapping at 8040, error at position 13:
2511
2512 00000 GET /invalid request HTTP/1.1\r\n
2513
2514
2515The output of "show info" on the CLI provides a number of useful information
2516regarding the maximum connection rate ever reached, maximum SSL key rate ever
2517reached, and in general all information which can help to explain temporary
2518issues regarding CPU or memory usage. Example :
2519
2520 > show info
2521 Name: HAProxy
2522 Version: 1.6-dev7-e32d18-17
2523 Release_date: 2015/10/12
2524 Nbproc: 1
2525 Process_num: 1
2526 Pid: 7949
2527 Uptime: 0d 0h02m39s
2528 Uptime_sec: 159
2529 Memmax_MB: 0
2530 Ulimit-n: 120032
2531 Maxsock: 120032
2532 Maxconn: 60000
2533 Hard_maxconn: 60000
2534 CurrConns: 0
2535 CumConns: 3
2536 CumReq: 3
2537 MaxSslConns: 0
2538 CurrSslConns: 0
2539 CumSslConns: 0
2540 Maxpipes: 0
2541 PipesUsed: 0
2542 PipesFree: 0
2543 ConnRate: 0
2544 ConnRateLimit: 0
2545 MaxConnRate: 1
2546 SessRate: 0
2547 SessRateLimit: 0
2548 MaxSessRate: 1
2549 SslRate: 0
2550 SslRateLimit: 0
2551 MaxSslRate: 0
2552 SslFrontendKeyRate: 0
2553 SslFrontendMaxKeyRate: 0
2554 SslFrontendSessionReuse_pct: 0
2555 SslBackendKeyRate: 0
2556 SslBackendMaxKeyRate: 0
2557 SslCacheLookups: 0
2558 SslCacheMisses: 0
2559 CompressBpsIn: 0
2560 CompressBpsOut: 0
2561 CompressBpsRateLim: 0
2562 ZlibMemUsage: 0
2563 MaxZlibMemUsage: 0
2564 Tasks: 5
2565 Run_queue: 1
2566 Idle_pct: 100
2567 node: wtap
2568 description:
2569
2570When an issue seems to randomly appear on a new version of HAProxy (eg: every
2571second request is aborted, occasional crash, etc), it is worth trying to enable
Dan Lloyd8e48b872016-07-01 21:01:18 -04002572memory poisoning so that each call to malloc() is immediately followed by the
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002573filling of the memory area with a configurable byte. By default this byte is
25740x50 (ASCII for 'P'), but any other byte can be used, including zero (which
2575will have the same effect as a calloc() and which may make issues disappear).
Dan Lloyd8e48b872016-07-01 21:01:18 -04002576Memory poisoning is enabled on the command line using the "-dM" option. It
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002577slightly hurts performance and is not recommended for use in production. If
Dan Lloyd8e48b872016-07-01 21:01:18 -04002578an issue happens all the time with it or never happens when poisoning uses
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002579byte zero, it clearly means you've found a bug and you definitely need to
2580report it. Otherwise if there's no clear change, the problem it is not related.
2581
2582When debugging some latency issues, it is important to use both strace and
2583tcpdump on the local machine, and another tcpdump on the remote system. The
2584reason for this is that there are delays everywhere in the processing chain and
2585it is important to know which one is causing latency to know where to act. In
2586practice, the local tcpdump will indicate when the input data come in. Strace
2587will indicate when haproxy receives these data (using recv/recvfrom). Warning,
2588openssl uses read()/write() syscalls instead of recv()/send(). Strace will also
2589show when haproxy sends the data, and tcpdump will show when the system sends
2590these data to the interface. Then the external tcpdump will show when the data
2591sent are really received (since the local one only shows when the packets are
2592queued). The benefit of sniffing on the local system is that strace and tcpdump
2593will use the same reference clock. Strace should be used with "-tts200" to get
2594complete timestamps and report large enough chunks of data to read them.
2595Tcpdump should be used with "-nvvttSs0" to report full packets, real sequence
2596numbers and complete timestamps.
2597
2598In practice, received data are almost always immediately received by haproxy
2599(unless the machine has a saturated CPU or these data are invalid and not
2600delivered). If these data are received but not sent, it generally is because
2601the output buffer is saturated (ie: recipient doesn't consume the data fast
2602enough). This can be confirmed by seeing that the polling doesn't notify of
2603the ability to write on the output file descriptor for some time (it's often
2604easier to spot in the strace output when the data finally leave and then roll
2605back to see when the write event was notified). It generally matches an ACK
2606received from the recipient, and detected by tcpdump. Once the data are sent,
2607they may spend some time in the system doing nothing. Here again, the TCP
2608congestion window may be limited and not allow these data to leave, waiting for
2609an ACK to open the window. If the traffic is idle and the data take 40 ms or
2610200 ms to leave, it's a different issue (which is not an issue), it's the fact
2611that the Nagle algorithm prevents empty packets from leaving immediately, in
2612hope that they will be merged with subsequent data. HAProxy automatically
2613disables Nagle in pure TCP mode and in tunnels. However it definitely remains
2614enabled when forwarding an HTTP body (and this contributes to the performance
2615improvement there by reducing the number of packets). Some HTTP non-compliant
2616applications may be sensitive to the latency when delivering incomplete HTTP
2617response messages. In this case you will have to enable "option http-no-delay"
2618to disable Nagle in order to work around their design, keeping in mind that any
2619other proxy in the chain may similarly be impacted. If tcpdump reports that data
2620leave immediately but the other end doesn't see them quickly, it can mean there
Dan Lloyd8e48b872016-07-01 21:01:18 -04002621is a congested WAN link, a congested LAN with flow control enabled and
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002622preventing the data from leaving, or more commonly that HAProxy is in fact
2623running in a virtual machine and that for whatever reason the hypervisor has
2624decided that the data didn't need to be sent immediately. In virtualized
2625environments, latency issues are almost always caused by the virtualization
2626layer, so in order to save time, it's worth first comparing tcpdump in the VM
2627and on the external components. Any difference has to be credited to the
2628hypervisor and its accompanying drivers.
2629
2630When some TCP SACK segments are seen in tcpdump traces (using -vv), it always
2631means that the side sending them has got the proof of a lost packet. While not
2632seeing them doesn't mean there are no losses, seeing them definitely means the
2633network is lossy. Losses are normal on a network, but at a rate where SACKs are
2634not noticeable at the naked eye. If they appear a lot in the traces, it is
2635worth investigating exactly what happens and where the packets are lost. HTTP
2636doesn't cope well with TCP losses, which introduce huge latencies.
2637
2638The "netstat -i" command will report statistics per interface. An interface
2639where the Rx-Ovr counter grows indicates that the system doesn't have enough
2640resources to receive all incoming packets and that they're lost before being
2641processed by the network driver. Rx-Drp indicates that some received packets
2642were lost in the network stack because the application doesn't process them
2643fast enough. This can happen during some attacks as well. Tx-Drp means that
2644the output queues were full and packets had to be dropped. When using TCP it
Dan Lloyd8e48b872016-07-01 21:01:18 -04002645should be very rare, but will possibly indicate a saturated outgoing link.
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002646
2647
264813. Security considerations
2649---------------------------
2650
2651HAProxy is designed to run with very limited privileges. The standard way to
2652use it is to isolate it into a chroot jail and to drop its privileges to a
2653non-root user without any permissions inside this jail so that if any future
2654vulnerability were to be discovered, its compromise would not affect the rest
2655of the system.
2656
Dan Lloyd8e48b872016-07-01 21:01:18 -04002657In order to perform a chroot, it first needs to be started as a root user. It is
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002658pointless to build hand-made chroots to start the process there, these ones are
2659painful to build, are never properly maintained and always contain way more
2660bugs than the main file-system. And in case of compromise, the intruder can use
2661the purposely built file-system. Unfortunately many administrators confuse
2662"start as root" and "run as root", resulting in the uid change to be done prior
2663to starting haproxy, and reducing the effective security restrictions.
2664
2665HAProxy will need to be started as root in order to :
2666 - adjust the file descriptor limits
2667 - bind to privileged port numbers
2668 - bind to a specific network interface
2669 - transparently listen to a foreign address
2670 - isolate itself inside the chroot jail
2671 - drop to another non-privileged UID
2672
2673HAProxy may require to be run as root in order to :
2674 - bind to an interface for outgoing connections
2675 - bind to privileged source ports for outgoing connections
Dan Lloyd8e48b872016-07-01 21:01:18 -04002676 - transparently bind to a foreign address for outgoing connections
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002677
2678Most users will never need the "run as root" case. But the "start as root"
2679covers most usages.
2680
2681A safe configuration will have :
2682
2683 - a chroot statement pointing to an empty location without any access
2684 permissions. This can be prepared this way on the UNIX command line :
2685
2686 # mkdir /var/empty && chmod 0 /var/empty || echo "Failed"
2687
2688 and referenced like this in the HAProxy configuration's global section :
2689
2690 chroot /var/empty
2691
2692 - both a uid/user and gid/group statements in the global section :
2693
2694 user haproxy
2695 group haproxy
2696
2697 - a stats socket whose mode, uid and gid are set to match the user and/or
2698 group allowed to access the CLI so that nobody may access it :
2699
2700 stats socket /var/run/haproxy.stat uid hatop gid hatop mode 600
2701