blob: c78dc7486225ddf0ab465309a324d91c34d055cc [file] [log] [blame]
Willy Tarreau2212e6a2015-10-13 14:40:55 +02001 ------------------------
2 HAProxy Management Guide
3 ------------------------
Willy Tarreaub3066502017-11-26 19:50:17 +01004 version 1.9
Willy Tarreau2212e6a2015-10-13 14:40:55 +02005
6
7This document describes how to start, stop, manage, and troubleshoot HAProxy,
8as well as some known limitations and traps to avoid. It does not describe how
9to configure it (for this please read configuration.txt).
10
11Note to documentation contributors :
12 This document is formatted with 80 columns per line, with even number of
13 spaces for indentation and without tabs. Please follow these rules strictly
14 so that it remains easily printable everywhere. If you add sections, please
15 update the summary below for easier searching.
16
17
18Summary
19-------
20
211. Prerequisites
222. Quick reminder about HAProxy's architecture
233. Starting HAProxy
244. Stopping and restarting HAProxy
255. File-descriptor limitations
266. Memory management
277. CPU usage
288. Logging
299. Statistics and monitoring
Willy Tarreau44aed902015-10-13 14:45:29 +0200309.1. CSV format
Willy Tarreau5d8b9792016-03-11 11:09:34 +0100319.2. Typed output format
329.3. Unix Socket commands
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003310. Tricks for easier configuration management
3411. Well-known traps to avoid
3512. Debugging and performance issues
3613. Security considerations
37
38
391. Prerequisites
40----------------
41
42In this document it is assumed that the reader has sufficient administration
43skills on a UNIX-like operating system, uses the shell on a daily basis and is
44familiar with troubleshooting utilities such as strace and tcpdump.
45
46
472. Quick reminder about HAProxy's architecture
48----------------------------------------------
49
50HAProxy is a single-threaded, event-driven, non-blocking daemon. This means is
51uses event multiplexing to schedule all of its activities instead of relying on
52the system to schedule between multiple activities. Most of the time it runs as
53a single process, so the output of "ps aux" on a system will report only one
54"haproxy" process, unless a soft reload is in progress and an older process is
55finishing its job in parallel to the new one. It is thus always easy to trace
56its activity using the strace utility.
57
58HAProxy is designed to isolate itself into a chroot jail during startup, where
59it cannot perform any file-system access at all. This is also true for the
60libraries it depends on (eg: libc, libssl, etc). The immediate effect is that
61a running process will not be able to reload a configuration file to apply
62changes, instead a new process will be started using the updated configuration
63file. Some other less obvious effects are that some timezone files or resolver
64files the libc might attempt to access at run time will not be found, though
65this should generally not happen as they're not needed after startup. A nice
66consequence of this principle is that the HAProxy process is totally stateless,
67and no cleanup is needed after it's killed, so any killing method that works
68will do the right thing.
69
70HAProxy doesn't write log files, but it relies on the standard syslog protocol
71to send logs to a remote server (which is often located on the same system).
72
73HAProxy uses its internal clock to enforce timeouts, that is derived from the
74system's time but where unexpected drift is corrected. This is done by limiting
75the time spent waiting in poll() for an event, and measuring the time it really
76took. In practice it never waits more than one second. This explains why, when
77running strace over a completely idle process, periodic calls to poll() (or any
78of its variants) surrounded by two gettimeofday() calls are noticed. They are
79normal, completely harmless and so cheap that the load they imply is totally
80undetectable at the system scale, so there's nothing abnormal there. Example :
81
82 16:35:40.002320 gettimeofday({1442759740, 2605}, NULL) = 0
83 16:35:40.002942 epoll_wait(0, {}, 200, 1000) = 0
84 16:35:41.007542 gettimeofday({1442759741, 7641}, NULL) = 0
85 16:35:41.007998 gettimeofday({1442759741, 8114}, NULL) = 0
86 16:35:41.008391 epoll_wait(0, {}, 200, 1000) = 0
87 16:35:42.011313 gettimeofday({1442759742, 11411}, NULL) = 0
88
89HAProxy is a TCP proxy, not a router. It deals with established connections that
90have been validated by the kernel, and not with packets of any form nor with
91sockets in other states (eg: no SYN_RECV nor TIME_WAIT), though their existence
92may prevent it from binding a port. It relies on the system to accept incoming
93connections and to initiate outgoing connections. An immediate effect of this is
94that there is no relation between packets observed on the two sides of a
95forwarded connection, which can be of different size, numbers and even family.
96Since a connection may only be accepted from a socket in LISTEN state, all the
97sockets it is listening to are necessarily visible using the "netstat" utility
98to show listening sockets. Example :
99
100 # netstat -ltnp
101 Active Internet connections (only servers)
102 Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
103 tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1629/sshd
104 tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2847/haproxy
105 tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 2847/haproxy
106
107
1083. Starting HAProxy
109-------------------
110
111HAProxy is started by invoking the "haproxy" program with a number of arguments
112passed on the command line. The actual syntax is :
113
114 $ haproxy [<options>]*
115
116where [<options>]* is any number of options. An option always starts with '-'
117followed by one of more letters, and possibly followed by one or multiple extra
118arguments. Without any option, HAProxy displays the help page with a reminder
119about supported options. Available options may vary slightly based on the
120operating system. A fair number of these options overlap with an equivalent one
121if the "global" section. In this case, the command line always has precedence
122over the configuration file, so that the command line can be used to quickly
123enforce some settings without touching the configuration files. The current
124list of options is :
125
126 -- <cfgfile>* : all the arguments following "--" are paths to configuration
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200127 file/directory to be loaded and processed in the declaration order. It is
128 mostly useful when relying on the shell to load many files that are
129 numerically ordered. See also "-f". The difference between "--" and "-f" is
130 that one "-f" must be placed before each file name, while a single "--" is
131 needed before all file names. Both options can be used together, the
132 command line ordering still applies. When more than one file is specified,
133 each file must start on a section boundary, so the first keyword of each
134 file must be one of "global", "defaults", "peers", "listen", "frontend",
135 "backend", and so on. A file cannot contain just a server list for example.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200136
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200137 -f <cfgfile|cfgdir> : adds <cfgfile> to the list of configuration files to be
138 loaded. If <cfgdir> is a directory, all the files (and only files) it
Dan Lloyd8e48b872016-07-01 21:01:18 -0400139 contains are added in lexical order (using LC_COLLATE=C) to the list of
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200140 configuration files to be loaded ; only files with ".cfg" extension are
141 added, only non hidden files (not prefixed with ".") are added.
142 Configuration files are loaded and processed in their declaration order.
143 This option may be specified multiple times to load multiple files. See
144 also "--". The difference between "--" and "-f" is that one "-f" must be
145 placed before each file name, while a single "--" is needed before all file
146 names. Both options can be used together, the command line ordering still
147 applies. When more than one file is specified, each file must start on a
148 section boundary, so the first keyword of each file must be one of
149 "global", "defaults", "peers", "listen", "frontend", "backend", and so on.
150 A file cannot contain just a server list for example.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200151
152 -C <dir> : changes to directory <dir> before loading configuration
153 files. This is useful when using relative paths. Warning when using
154 wildcards after "--" which are in fact replaced by the shell before
155 starting haproxy.
156
157 -D : start as a daemon. The process detaches from the current terminal after
158 forking, and errors are not reported anymore in the terminal. It is
159 equivalent to the "daemon" keyword in the "global" section of the
160 configuration. It is recommended to always force it in any init script so
161 that a faulty configuration doesn't prevent the system from booting.
162
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200163 -L <name> : change the local peer name to <name>, which defaults to the local
William Lallemanddaf4cd22018-04-17 16:46:13 +0200164 hostname. This is used only with peers replication. You can use the
165 variable $HAPROXY_LOCALPEER in the configuration file to reference the
166 peer name.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200167
168 -N <limit> : sets the default per-proxy maxconn to <limit> instead of the
169 builtin default value (usually 2000). Only useful for debugging.
170
171 -V : enable verbose mode (disables quiet mode). Reverts the effect of "-q" or
172 "quiet".
173
William Lallemande202b1e2017-06-01 17:38:56 +0200174 -W : master-worker mode. It is equivalent to the "master-worker" keyword in
175 the "global" section of the configuration. This mode will launch a "master"
176 which will monitor the "workers". Using this mode, you can reload HAProxy
177 directly by sending a SIGUSR2 signal to the master. The master-worker mode
178 is compatible either with the foreground or daemon mode. It is
179 recommended to use this mode with multiprocess and systemd.
180
Pavlos Parissisf65f2572018-02-07 21:42:16 +0100181 -Ws : master-worker mode with support of `notify` type of systemd service.
182 This option is only available when HAProxy was built with `USE_SYSTEMD`
183 build option enabled.
184
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200185 -c : only performs a check of the configuration files and exits before trying
186 to bind. The exit status is zero if everything is OK, or non-zero if an
187 error is encountered.
188
189 -d : enable debug mode. This disables daemon mode, forces the process to stay
190 in foreground and to show incoming and outgoing events. It is equivalent to
191 the "global" section's "debug" keyword. It must never be used in an init
192 script.
193
194 -dG : disable use of getaddrinfo() to resolve host names into addresses. It
195 can be used when suspecting that getaddrinfo() doesn't work as expected.
196 This option was made available because many bogus implementations of
197 getaddrinfo() exist on various systems and cause anomalies that are
198 difficult to troubleshoot.
199
Dan Lloyd8e48b872016-07-01 21:01:18 -0400200 -dM[<byte>] : forces memory poisoning, which means that each and every
Willy Tarreaubafbe012017-11-24 17:34:44 +0100201 memory region allocated with malloc() or pool_alloc() will be filled with
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200202 <byte> before being passed to the caller. When <byte> is not specified, it
203 defaults to 0x50 ('P'). While this slightly slows down operations, it is
204 useful to reliably trigger issues resulting from missing initializations in
205 the code that cause random crashes. Note that -dM0 has the effect of
206 turning any malloc() into a calloc(). In any case if a bug appears or
207 disappears when using this option it means there is a bug in haproxy, so
208 please report it.
209
210 -dS : disable use of the splice() system call. It is equivalent to the
211 "global" section's "nosplice" keyword. This may be used when splice() is
212 suspected to behave improperly or to cause performance issues, or when
213 using strace to see the forwarded data (which do not appear when using
214 splice()).
215
216 -dV : disable SSL verify on the server side. It is equivalent to having
217 "ssl-server-verify none" in the "global" section. This is useful when
218 trying to reproduce production issues out of the production
219 environment. Never use this in an init script as it degrades SSL security
220 to the servers.
221
222 -db : disable background mode and multi-process mode. The process remains in
223 foreground. It is mainly used during development or during small tests, as
224 Ctrl-C is enough to stop the process. Never use it in an init script.
225
226 -de : disable the use of the "epoll" poller. It is equivalent to the "global"
227 section's keyword "noepoll". It is mostly useful when suspecting a bug
228 related to this poller. On systems supporting epoll, the fallback will
229 generally be the "poll" poller.
230
231 -dk : disable the use of the "kqueue" poller. It is equivalent to the
232 "global" section's keyword "nokqueue". It is mostly useful when suspecting
233 a bug related to this poller. On systems supporting kqueue, the fallback
234 will generally be the "poll" poller.
235
236 -dp : disable the use of the "poll" poller. It is equivalent to the "global"
237 section's keyword "nopoll". It is mostly useful when suspecting a bug
238 related to this poller. On systems supporting poll, the fallback will
239 generally be the "select" poller, which cannot be disabled and is limited
240 to 1024 file descriptors.
241
Willy Tarreau3eed10e2016-11-07 21:03:16 +0100242 -dr : ignore server address resolution failures. It is very common when
243 validating a configuration out of production not to have access to the same
244 resolvers and to fail on server address resolution, making it difficult to
245 test a configuration. This option simply appends the "none" method to the
246 list of address resolution methods for all servers, ensuring that even if
247 the libc fails to resolve an address, the startup sequence is not
248 interrupted.
249
Willy Tarreau70060452015-12-14 12:46:07 +0100250 -m <limit> : limit the total allocatable memory to <limit> megabytes across
251 all processes. This may cause some connection refusals or some slowdowns
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200252 depending on the amount of memory needed for normal operations. This is
Willy Tarreau70060452015-12-14 12:46:07 +0100253 mostly used to force the processes to work in a constrained resource usage
254 scenario. It is important to note that the memory is not shared between
255 processes, so in a multi-process scenario, this value is first divided by
256 global.nbproc before forking.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200257
258 -n <limit> : limits the per-process connection limit to <limit>. This is
259 equivalent to the global section's keyword "maxconn". It has precedence
260 over this keyword. This may be used to quickly force lower limits to avoid
261 a service outage on systems where resource limits are too low.
262
263 -p <file> : write all processes' pids into <file> during startup. This is
264 equivalent to the "global" section's keyword "pidfile". The file is opened
265 before entering the chroot jail, and after doing the chdir() implied by
266 "-C". Each pid appears on its own line.
267
268 -q : set "quiet" mode. This disables some messages during the configuration
269 parsing and during startup. It can be used in combination with "-c" to
270 just check if a configuration file is valid or not.
271
272 -sf <pid>* : send the "finish" signal (SIGUSR1) to older processes after boot
273 completion to ask them to finish what they are doing and to leave. <pid>
274 is a list of pids to signal (one per argument). The list ends on any
275 option starting with a "-". It is not a problem if the list of pids is
276 empty, so that it can be built on the fly based on the result of a command
277 like "pidof" or "pgrep".
278
279 -st <pid>* : send the "terminate" signal (SIGTERM) to older processes after
280 boot completion to terminate them immediately without finishing what they
281 were doing. <pid> is a list of pids to signal (one per argument). The list
282 is ends on any option starting with a "-". It is not a problem if the list
283 of pids is empty, so that it can be built on the fly based on the result of
284 a command like "pidof" or "pgrep".
285
286 -v : report the version and build date.
287
288 -vv : display the version, build options, libraries versions and usable
289 pollers. This output is systematically requested when filing a bug report.
290
Olivier Houchardd33fc3a2017-04-05 22:50:59 +0200291 -x <unix_socket> : connect to the specified socket and try to retrieve any
292 listening sockets from the old process, and use them instead of trying to
293 bind new ones. This is useful to avoid missing any new connection when
William Lallemandf6975e92017-05-26 17:42:10 +0200294 reloading the configuration on Linux. The capability must be enable on the
295 stats socket using "expose-fd listeners" in your configuration.
Olivier Houchardd33fc3a2017-04-05 22:50:59 +0200296
Dan Lloyd8e48b872016-07-01 21:01:18 -0400297A safe way to start HAProxy from an init file consists in forcing the daemon
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200298mode, storing existing pids to a pid file and using this pid file to notify
299older processes to finish before leaving :
300
301 haproxy -f /etc/haproxy.cfg \
302 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
303
304When the configuration is split into a few specific files (eg: tcp vs http),
305it is recommended to use the "-f" option :
306
307 haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
308 -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
309 -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
310 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
311
312When an unknown number of files is expected, such as customer-specific files,
313it is recommended to assign them a name starting with a fixed-size sequence
314number and to use "--" to load them, possibly after loading some defaults :
315
316 haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
317 -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
318 -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
319 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) \
320 -f /etc/haproxy/default-customers.cfg -- /etc/haproxy/customers/*
321
322Sometimes a failure to start may happen for whatever reason. Then it is
323important to verify if the version of HAProxy you are invoking is the expected
324version and if it supports the features you are expecting (eg: SSL, PCRE,
325compression, Lua, etc). This can be verified using "haproxy -vv". Some
326important information such as certain build options, the target system and
327the versions of the libraries being used are reported there. It is also what
328you will systematically be asked for when posting a bug report :
329
330 $ haproxy -vv
331 HA-Proxy version 1.6-dev7-a088d3-4 2015/10/08
332 Copyright 2000-2015 Willy Tarreau <willy@haproxy.org>
333
334 Build options :
335 TARGET = linux2628
336 CPU = generic
337 CC = gcc
338 CFLAGS = -pg -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement \
339 -DBUFSIZE=8030 -DMAXREWRITE=1030 -DSO_MARK=36 -DTCP_REPAIR=19
340 OPTIONS = USE_ZLIB=1 USE_DLMALLOC=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
341
342 Default settings :
343 maxconn = 2000, bufsize = 8030, maxrewrite = 1030, maxpollevents = 200
344
345 Encrypted password support via crypt(3): yes
346 Built with zlib version : 1.2.6
347 Compression algorithms supported : identity("identity"), deflate("deflate"), \
348 raw-deflate("deflate"), gzip("gzip")
349 Built with OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
350 Running on OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
351 OpenSSL library supports TLS extensions : yes
352 OpenSSL library supports SNI : yes
353 OpenSSL library supports prefer-server-ciphers : yes
354 Built with PCRE version : 8.12 2011-01-15
355 PCRE library supports JIT : no (USE_PCRE_JIT not set)
356 Built with Lua version : Lua 5.3.1
357 Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND
358
359 Available polling systems :
360 epoll : pref=300, test result OK
361 poll : pref=200, test result OK
362 select : pref=150, test result OK
363 Total: 3 (3 usable), will use epoll.
364
365The relevant information that many non-developer users can verify here are :
366 - the version : 1.6-dev7-a088d3-4 above means the code is currently at commit
367 ID "a088d3" which is the 4th one after after official version "1.6-dev7".
368 Version 1.6-dev7 would show as "1.6-dev7-8c1ad7". What matters here is in
369 fact "1.6-dev7". This is the 7th development version of what will become
370 version 1.6 in the future. A development version not suitable for use in
371 production (unless you know exactly what you are doing). A stable version
372 will show as a 3-numbers version, such as "1.5.14-16f863", indicating the
373 14th level of fix on top of version 1.5. This is a production-ready version.
374
375 - the release date : 2015/10/08. It is represented in the universal
376 year/month/day format. Here this means August 8th, 2015. Given that stable
377 releases are issued every few months (1-2 months at the beginning, sometimes
378 6 months once the product becomes very stable), if you're seeing an old date
379 here, it means you're probably affected by a number of bugs or security
380 issues that have since been fixed and that it might be worth checking on the
381 official site.
382
383 - build options : they are relevant to people who build their packages
384 themselves, they can explain why things are not behaving as expected. For
385 example the development version above was built for Linux 2.6.28 or later,
Dan Lloyd8e48b872016-07-01 21:01:18 -0400386 targeting a generic CPU (no CPU-specific optimizations), and lacks any
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200387 code optimization (-O0) so it will perform poorly in terms of performance.
388
389 - libraries versions : zlib version is reported as found in the library
390 itself. In general zlib is considered a very stable product and upgrades
391 are almost never needed. OpenSSL reports two versions, the version used at
392 build time and the one being used, as found on the system. These ones may
393 differ by the last letter but never by the numbers. The build date is also
394 reported because most OpenSSL bugs are security issues and need to be taken
395 seriously, so this library absolutely needs to be kept up to date. Seeing a
396 4-months old version here is highly suspicious and indeed an update was
397 missed. PCRE provides very fast regular expressions and is highly
398 recommended. Certain of its extensions such as JIT are not present in all
399 versions and still young so some people prefer not to build with them,
Dan Lloyd8e48b872016-07-01 21:01:18 -0400400 which is why the build status is reported as well. Regarding the Lua
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200401 scripting language, HAProxy expects version 5.3 which is very young since
402 it was released a little time before HAProxy 1.6. It is important to check
403 on the Lua web site if some fixes are proposed for this branch.
404
405 - Available polling systems will affect the process's scalability when
406 dealing with more than about one thousand of concurrent connections. These
407 ones are only available when the correct system was indicated in the TARGET
408 variable during the build. The "epoll" mechanism is highly recommended on
409 Linux, and the kqueue mechanism is highly recommended on BSD. Lacking them
410 will result in poll() or even select() being used, causing a high CPU usage
411 when dealing with a lot of connections.
412
413
4144. Stopping and restarting HAProxy
415----------------------------------
416
417HAProxy supports a graceful and a hard stop. The hard stop is simple, when the
418SIGTERM signal is sent to the haproxy process, it immediately quits and all
419established connections are closed. The graceful stop is triggered when the
420SIGUSR1 signal is sent to the haproxy process. It consists in only unbinding
421from listening ports, but continue to process existing connections until they
422close. Once the last connection is closed, the process leaves.
423
424The hard stop method is used for the "stop" or "restart" actions of the service
425management script. The graceful stop is used for the "reload" action which
426tries to seamlessly reload a new configuration in a new process.
427
428Both of these signals may be sent by the new haproxy process itself during a
429reload or restart, so that they are sent at the latest possible moment and only
430if absolutely required. This is what is performed by the "-st" (hard) and "-sf"
431(graceful) options respectively.
432
William Lallemande202b1e2017-06-01 17:38:56 +0200433In master-worker mode, it is not needed to start a new haproxy process in
434order to reload the configuration. The master process reacts to the SIGUSR2
435signal by reexecuting itself with the -sf parameter followed by the PIDs of
436the workers. The master will then parse the configuration file and fork new
437workers.
438
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200439To understand better how these signals are used, it is important to understand
440the whole restart mechanism.
441
442First, an existing haproxy process is running. The administrator uses a system
443specific command such as "/etc/init.d/haproxy reload" to indicate he wants to
444take the new configuration file into effect. What happens then is the following.
445First, the service script (/etc/init.d/haproxy or equivalent) will verify that
446the configuration file parses correctly using "haproxy -c". After that it will
447try to start haproxy with this configuration file, using "-st" or "-sf".
448
449Then HAProxy tries to bind to all listening ports. If some fatal errors happen
450(eg: address not present on the system, permission denied), the process quits
451with an error. If a socket binding fails because a port is already in use, then
452the process will first send a SIGTTOU signal to all the pids specified in the
453"-st" or "-sf" pid list. This is what is called the "pause" signal. It instructs
454all existing haproxy processes to temporarily stop listening to their ports so
455that the new process can try to bind again. During this time, the old process
456continues to process existing connections. If the binding still fails (because
457for example a port is shared with another daemon), then the new process sends a
458SIGTTIN signal to the old processes to instruct them to resume operations just
459as if nothing happened. The old processes will then restart listening to the
460ports and continue to accept connections. Not that this mechanism is system
Dan Lloyd8e48b872016-07-01 21:01:18 -0400461dependent and some operating systems may not support it in multi-process mode.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200462
463If the new process manages to bind correctly to all ports, then it sends either
464the SIGTERM (hard stop in case of "-st") or the SIGUSR1 (graceful stop in case
465of "-sf") to all processes to notify them that it is now in charge of operations
466and that the old processes will have to leave, either immediately or once they
467have finished their job.
468
469It is important to note that during this timeframe, there are two small windows
470of a few milliseconds each where it is possible that a few connection failures
471will be noticed during high loads. Typically observed failure rates are around
4721 failure during a reload operation every 10000 new connections per second,
473which means that a heavily loaded site running at 30000 new connections per
474second may see about 3 failed connection upon every reload. The two situations
475where this happens are :
476
477 - if the new process fails to bind due to the presence of the old process,
478 it will first have to go through the SIGTTOU+SIGTTIN sequence, which
479 typically lasts about one millisecond for a few tens of frontends, and
480 during which some ports will not be bound to the old process and not yet
481 bound to the new one. HAProxy works around this on systems that support the
482 SO_REUSEPORT socket options, as it allows the new process to bind without
483 first asking the old one to unbind. Most BSD systems have been supporting
484 this almost forever. Linux has been supporting this in version 2.0 and
485 dropped it around 2.2, but some patches were floating around by then. It
486 was reintroduced in kernel 3.9, so if you are observing a connection
Dan Lloyd8e48b872016-07-01 21:01:18 -0400487 failure rate above the one mentioned above, please ensure that your kernel
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200488 is 3.9 or newer, or that relevant patches were backported to your kernel
489 (less likely).
490
491 - when the old processes close the listening ports, the kernel may not always
492 redistribute any pending connection that was remaining in the socket's
493 backlog. Under high loads, a SYN packet may happen just before the socket
494 is closed, and will lead to an RST packet being sent to the client. In some
495 critical environments where even one drop is not acceptable, these ones are
496 sometimes dealt with using firewall rules to block SYN packets during the
497 reload, forcing the client to retransmit. This is totally system-dependent,
498 as some systems might be able to visit other listening queues and avoid
499 this RST. A second case concerns the ACK from the client on a local socket
500 that was in SYN_RECV state just before the close. This ACK will lead to an
501 RST packet while the haproxy process is still not aware of it. This one is
Dan Lloyd8e48b872016-07-01 21:01:18 -0400502 harder to get rid of, though the firewall filtering rules mentioned above
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200503 will work well if applied one second or so before restarting the process.
504
505For the vast majority of users, such drops will never ever happen since they
506don't have enough load to trigger the race conditions. And for most high traffic
507users, the failure rate is still fairly within the noise margin provided that at
508least SO_REUSEPORT is properly supported on their systems.
509
510
5115. File-descriptor limitations
512------------------------------
513
514In order to ensure that all incoming connections will successfully be served,
515HAProxy computes at load time the total number of file descriptors that will be
516needed during the process's life. A regular Unix process is generally granted
5171024 file descriptors by default, and a privileged process can raise this limit
518itself. This is one reason for starting HAProxy as root and letting it adjust
519the limit. The default limit of 1024 file descriptors roughly allow about 500
520concurrent connections to be processed. The computation is based on the global
521maxconn parameter which limits the total number of connections per process, the
522number of listeners, the number of servers which have a health check enabled,
523the agent checks, the peers, the loggers and possibly a few other technical
524requirements. A simple rough estimate of this number consists in simply
525doubling the maxconn value and adding a few tens to get the approximate number
526of file descriptors needed.
527
528Originally HAProxy did not know how to compute this value, and it was necessary
529to pass the value using the "ulimit-n" setting in the global section. This
530explains why even today a lot of configurations are seen with this setting
531present. Unfortunately it was often miscalculated resulting in connection
532failures when approaching maxconn instead of throttling incoming connection
533while waiting for the needed resources. For this reason it is important to
Dan Lloyd8e48b872016-07-01 21:01:18 -0400534remove any vestigial "ulimit-n" setting that can remain from very old versions.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200535
536Raising the number of file descriptors to accept even moderate loads is
537mandatory but comes with some OS-specific adjustments. First, the select()
538polling system is limited to 1024 file descriptors. In fact on Linux it used
539to be capable of handling more but since certain OS ship with excessively
540restrictive SELinux policies forbidding the use of select() with more than
5411024 file descriptors, HAProxy now refuses to start in this case in order to
542avoid any issue at run time. On all supported operating systems, poll() is
543available and will not suffer from this limitation. It is automatically picked
Dan Lloyd8e48b872016-07-01 21:01:18 -0400544so there is nothing to do to get a working configuration. But poll's becomes
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200545very slow when the number of file descriptors increases. While HAProxy does its
546best to limit this performance impact (eg: via the use of the internal file
547descriptor cache and batched processing), a good rule of thumb is that using
548poll() with more than a thousand concurrent connections will use a lot of CPU.
549
550For Linux systems base on kernels 2.6 and above, the epoll() system call will
551be used. It's a much more scalable mechanism relying on callbacks in the kernel
552that guarantee a constant wake up time regardless of the number of registered
553monitored file descriptors. It is automatically used where detected, provided
554that HAProxy had been built for one of the Linux flavors. Its presence and
555support can be verified using "haproxy -vv".
556
557For BSD systems which support it, kqueue() is available as an alternative. It
558is much faster than poll() and even slightly faster than epoll() thanks to its
559batched handling of changes. At least FreeBSD and OpenBSD support it. Just like
560with Linux's epoll(), its support and availability are reported in the output
561of "haproxy -vv".
562
563Having a good poller is one thing, but it is mandatory that the process can
564reach the limits. When HAProxy starts, it immediately sets the new process's
565file descriptor limits and verifies if it succeeds. In case of failure, it
566reports it before forking so that the administrator can see the problem. As
567long as the process is started by as root, there should be no reason for this
568setting to fail. However, it can fail if the process is started by an
569unprivileged user. If there is a compelling reason for *not* starting haproxy
570as root (eg: started by end users, or by a per-application account), then the
571file descriptor limit can be raised by the system administrator for this
572specific user. The effectiveness of the setting can be verified by issuing
573"ulimit -n" from the user's command line. It should reflect the new limit.
574
575Warning: when an unprivileged user's limits are changed in this user's account,
576it is fairly common that these values are only considered when the user logs in
577and not at all in some scripts run at system boot time nor in crontabs. This is
578totally dependent on the operating system, keep in mind to check "ulimit -n"
579before starting haproxy when running this way. The general advice is never to
580start haproxy as an unprivileged user for production purposes. Another good
581reason is that it prevents haproxy from enabling some security protections.
582
583Once it is certain that the system will allow the haproxy process to use the
584requested number of file descriptors, two new system-specific limits may be
585encountered. The first one is the system-wide file descriptor limit, which is
586the total number of file descriptors opened on the system, covering all
587processes. When this limit is reached, accept() or socket() will typically
588return ENFILE. The second one is the per-process hard limit on the number of
589file descriptors, it prevents setrlimit() from being set higher. Both are very
590dependent on the operating system. On Linux, the system limit is set at boot
591based on the amount of memory. It can be changed with the "fs.file-max" sysctl.
592And the per-process hard limit is set to 1048576 by default, but it can be
593changed using the "fs.nr_open" sysctl.
594
595File descriptor limitations may be observed on a running process when they are
596set too low. The strace utility will report that accept() and socket() return
597"-1 EMFILE" when the process's limits have been reached. In this case, simply
598raising the "ulimit-n" value (or removing it) will solve the problem. If these
599system calls return "-1 ENFILE" then it means that the kernel's limits have
600been reached and that something must be done on a system-wide parameter. These
601trouble must absolutely be addressed, as they result in high CPU usage (when
602accept() fails) and failed connections that are generally visible to the user.
603One solution also consists in lowering the global maxconn value to enforce
604serialization, and possibly to disable HTTP keep-alive to force connections
605to be released and reused faster.
606
607
6086. Memory management
609--------------------
610
611HAProxy uses a simple and fast pool-based memory management. Since it relies on
612a small number of different object types, it's much more efficient to pick new
613objects from a pool which already contains objects of the appropriate size than
614to call malloc() for each different size. The pools are organized as a stack or
615LIFO, so that newly allocated objects are taken from recently released objects
616still hot in the CPU caches. Pools of similar sizes are merged together, in
617order to limit memory fragmentation.
618
619By default, since the focus is set on performance, each released object is put
620back into the pool it came from, and allocated objects are never freed since
621they are expected to be reused very soon.
622
623On the CLI, it is possible to check how memory is being used in pools thanks to
624the "show pools" command :
625
626 > show pools
627 Dumping pools usage. Use SIGQUIT to flush them.
628 - Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 3 users [SHARED]
629 - Pool hlua_com (48 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
630 - Pool vars (64 bytes) : 0 allocated (0 bytes), 0 used, 2 users [SHARED]
631 - Pool task (112 bytes) : 5 allocated (560 bytes), 5 used, 1 users [SHARED]
632 - Pool session (128 bytes) : 1 allocated (128 bytes), 1 used, 2 users [SHARED]
633 - Pool http_txn (272 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
634 - Pool connection (352 bytes) : 2 allocated (704 bytes), 2 used, 1 users [SHARED]
635 - Pool hdr_idx (416 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
636 - Pool stream (864 bytes) : 1 allocated (864 bytes), 1 used, 1 users [SHARED]
637 - Pool requri (1024 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
638 - Pool buffer (8064 bytes) : 3 allocated (24192 bytes), 2 used, 1 users [SHARED]
639 Total: 11 pools, 26608 bytes allocated, 18544 used.
640
641The pool name is only indicative, it's the name of the first object type using
642this pool. The size in parenthesis is the object size for objects in this pool.
643Object sizes are always rounded up to the closest multiple of 16 bytes. The
644number of objects currently allocated and the equivalent number of bytes is
645reported so that it is easy to know which pool is responsible for the highest
646memory usage. The number of objects currently in use is reported as well in the
647"used" field. The difference between "allocated" and "used" corresponds to the
648objects that have been freed and are available for immediate use.
649
650It is possible to limit the amount of memory allocated per process using the
651"-m" command line option, followed by a number of megabytes. It covers all of
652the process's addressable space, so that includes memory used by some libraries
653as well as the stack, but it is a reliable limit when building a resource
654constrained system. It works the same way as "ulimit -v" on systems which have
655it, or "ulimit -d" for the other ones.
656
657If a memory allocation fails due to the memory limit being reached or because
658the system doesn't have any enough memory, then haproxy will first start to
659free all available objects from all pools before attempting to allocate memory
660again. This mechanism of releasing unused memory can be triggered by sending
661the signal SIGQUIT to the haproxy process. When doing so, the pools state prior
662to the flush will also be reported to stderr when the process runs in
663foreground.
664
665During a reload operation, the process switched to the graceful stop state also
666automatically performs some flushes after releasing any connection so that all
667possible memory is released to save it for the new process.
668
669
6707. CPU usage
671------------
672
673HAProxy normally spends most of its time in the system and a smaller part in
674userland. A finely tuned 3.5 GHz CPU can sustain a rate about 80000 end-to-end
675connection setups and closes per second at 100% CPU on a single core. When one
676core is saturated, typical figures are :
677 - 95% system, 5% user for long TCP connections or large HTTP objects
678 - 85% system and 15% user for short TCP connections or small HTTP objects in
679 close mode
680 - 70% system and 30% user for small HTTP objects in keep-alive mode
681
682The amount of rules processing and regular expressions will increase the user
683land part. The presence of firewall rules, connection tracking, complex routing
684tables in the system will instead increase the system part.
685
686On most systems, the CPU time observed during network transfers can be cut in 4
687parts :
688 - the interrupt part, which concerns all the processing performed upon I/O
689 receipt, before the target process is even known. Typically Rx packets are
690 accounted for in interrupt. On some systems such as Linux where interrupt
691 processing may be deferred to a dedicated thread, it can appear as softirq,
692 and the thread is called ksoftirqd/0 (for CPU 0). The CPU taking care of
693 this load is generally defined by the hardware settings, though in the case
694 of softirq it is often possible to remap the processing to another CPU.
695 This interrupt part will often be perceived as parasitic since it's not
696 associated with any process, but it actually is some processing being done
697 to prepare the work for the process.
698
699 - the system part, which concerns all the processing done using kernel code
700 called from userland. System calls are accounted as system for example. All
701 synchronously delivered Tx packets will be accounted for as system time. If
702 some packets have to be deferred due to queues filling up, they may then be
703 processed in interrupt context later (eg: upon receipt of an ACK opening a
704 TCP window).
705
706 - the user part, which exclusively runs application code in userland. HAProxy
707 runs exclusively in this part, though it makes heavy use of system calls.
708 Rules processing, regular expressions, compression, encryption all add to
709 the user portion of CPU consumption.
710
711 - the idle part, which is what the CPU does when there is nothing to do. For
712 example HAProxy waits for an incoming connection, or waits for some data to
713 leave, meaning the system is waiting for an ACK from the client to push
714 these data.
715
716In practice regarding HAProxy's activity, it is in general reasonably accurate
717(but totally inexact) to consider that interrupt/softirq are caused by Rx
718processing in kernel drivers, that user-land is caused by layer 7 processing
719in HAProxy, and that system time is caused by network processing on the Tx
720path.
721
722Since HAProxy runs around an event loop, it waits for new events using poll()
723(or any alternative) and processes all these events as fast as possible before
724going back to poll() waiting for new events. It measures the time spent waiting
725in poll() compared to the time spent doing processing events. The ratio of
726polling time vs total time is called the "idle" time, it's the amount of time
727spent waiting for something to happen. This ratio is reported in the stats page
728on the "idle" line, or "Idle_pct" on the CLI. When it's close to 100%, it means
729the load is extremely low. When it's close to 0%, it means that there is
730constantly some activity. While it cannot be very accurate on an overloaded
731system due to other processes possibly preempting the CPU from the haproxy
732process, it still provides a good estimate about how HAProxy considers it is
733working : if the load is low and the idle ratio is low as well, it may indicate
734that HAProxy has a lot of work to do, possibly due to very expensive rules that
735have to be processed. Conversely, if HAProxy indicates the idle is close to
736100% while things are slow, it means that it cannot do anything to speed things
737up because it is already waiting for incoming data to process. In the example
738below, haproxy is completely idle :
739
740 $ echo "show info" | socat - /var/run/haproxy.sock | grep ^Idle
741 Idle_pct: 100
742
743When the idle ratio starts to become very low, it is important to tune the
744system and place processes and interrupts correctly to save the most possible
745CPU resources for all tasks. If a firewall is present, it may be worth trying
746to disable it or to tune it to ensure it is not responsible for a large part
747of the performance limitation. It's worth noting that unloading a stateful
748firewall generally reduces both the amount of interrupt/softirq and of system
749usage since such firewalls act both on the Rx and the Tx paths. On Linux,
750unloading the nf_conntrack and ip_conntrack modules will show whether there is
751anything to gain. If so, then the module runs with default settings and you'll
752have to figure how to tune it for better performance. In general this consists
753in considerably increasing the hash table size. On FreeBSD, "pfctl -d" will
754disable the "pf" firewall and its stateful engine at the same time.
755
756If it is observed that a lot of time is spent in interrupt/softirq, it is
757important to ensure that they don't run on the same CPU. Most systems tend to
758pin the tasks on the CPU where they receive the network traffic because for
759certain workloads it improves things. But with heavily network-bound workloads
760it is the opposite as the haproxy process will have to fight against its kernel
761counterpart. Pinning haproxy to one CPU core and the interrupts to another one,
762all sharing the same L3 cache tends to sensibly increase network performance
763because in practice the amount of work for haproxy and the network stack are
764quite close, so they can almost fill an entire CPU each. On Linux this is done
765using taskset (for haproxy) or using cpu-map (from the haproxy config), and the
766interrupts are assigned under /proc/irq. Many network interfaces support
767multiple queues and multiple interrupts. In general it helps to spread them
768across a small number of CPU cores provided they all share the same L3 cache.
769Please always stop irq_balance which always does the worst possible thing on
770such workloads.
771
772For CPU-bound workloads consisting in a lot of SSL traffic or a lot of
773compression, it may be worth using multiple processes dedicated to certain
774tasks, though there is no universal rule here and experimentation will have to
775be performed.
776
777In order to increase the CPU capacity, it is possible to make HAProxy run as
778several processes, using the "nbproc" directive in the global section. There
779are some limitations though :
780 - health checks are run per process, so the target servers will get as many
781 checks as there are running processes ;
782 - maxconn values and queues are per-process so the correct value must be set
783 to avoid overloading the servers ;
784 - outgoing connections should avoid using port ranges to avoid conflicts
785 - stick-tables are per process and are not shared between processes ;
786 - each peers section may only run on a single process at a time ;
787 - the CLI operations will only act on a single process at a time.
788
789With this in mind, it appears that the easiest setup often consists in having
790one first layer running on multiple processes and in charge for the heavy
791processing, passing the traffic to a second layer running in a single process.
792This mechanism is suited to SSL and compression which are the two CPU-heavy
793features. Instances can easily be chained over UNIX sockets (which are cheaper
fengpeiyuancc123c62016-01-15 16:40:53 +0800794than TCP sockets and which do not waste ports), and the proxy protocol which is
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200795useful to pass client information to the next stage. When doing so, it is
796generally a good idea to bind all the single-process tasks to process number 1
797and extra tasks to next processes, as this will make it easier to generate
798similar configurations for different machines.
799
800On Linux versions 3.9 and above, running HAProxy in multi-process mode is much
801more efficient when each process uses a distinct listening socket on the same
802IP:port ; this will make the kernel evenly distribute the load across all
803processes instead of waking them all up. Please check the "process" option of
804the "bind" keyword lines in the configuration manual for more information.
805
806
8078. Logging
808----------
809
810For logging, HAProxy always relies on a syslog server since it does not perform
811any file-system access. The standard way of using it is to send logs over UDP
812to the log server (by default on port 514). Very commonly this is configured to
813127.0.0.1 where the local syslog daemon is running, but it's also used over the
814network to log to a central server. The central server provides additional
815benefits especially in active-active scenarios where it is desirable to keep
816the logs merged in arrival order. HAProxy may also make use of a UNIX socket to
817send its logs to the local syslog daemon, but it is not recommended at all,
818because if the syslog server is restarted while haproxy runs, the socket will
819be replaced and new logs will be lost. Since HAProxy will be isolated inside a
820chroot jail, it will not have the ability to reconnect to the new socket. It
821has also been observed in field that the log buffers in use on UNIX sockets are
822very small and lead to lost messages even at very light loads. But this can be
823fine for testing however.
824
825It is recommended to add the following directive to the "global" section to
826make HAProxy log to the local daemon using facility "local0" :
827
828 log 127.0.0.1:514 local0
829
830and then to add the following one to each "defaults" section or to each frontend
831and backend section :
832
833 log global
834
835This way, all logs will be centralized through the global definition of where
836the log server is.
837
838Some syslog daemons do not listen to UDP traffic by default, so depending on
839the daemon being used, the syntax to enable this will vary :
840
841 - on sysklogd, you need to pass argument "-r" on the daemon's command line
842 so that it listens to a UDP socket for "remote" logs ; note that there is
843 no way to limit it to address 127.0.0.1 so it will also receive logs from
844 remote systems ;
845
846 - on rsyslogd, the following lines must be added to the configuration file :
847
848 $ModLoad imudp
849 $UDPServerAddress *
850 $UDPServerRun 514
851
852 - on syslog-ng, a new source can be created the following way, it then needs
853 to be added as a valid source in one of the "log" directives :
854
855 source s_udp {
856 udp(ip(127.0.0.1) port(514));
857 };
858
859Please consult your syslog daemon's manual for more information. If no logs are
860seen in the system's log files, please consider the following tests :
861
862 - restart haproxy. Each frontend and backend logs one line indicating it's
863 starting. If these logs are received, it means logs are working.
864
865 - run "strace -tt -s100 -etrace=sendmsg -p <haproxy's pid>" and perform some
866 activity that you expect to be logged. You should see the log messages
867 being sent using sendmsg() there. If they don't appear, restart using
868 strace on top of haproxy. If you still see no logs, it definitely means
869 that something is wrong in your configuration.
870
871 - run tcpdump to watch for port 514, for example on the loopback interface if
872 the traffic is being sent locally : "tcpdump -As0 -ni lo port 514". If the
873 packets are seen there, it's the proof they're sent then the syslogd daemon
874 needs to be troubleshooted.
875
876While traffic logs are sent from the frontends (where the incoming connections
877are accepted), backends also need to be able to send logs in order to report a
878server state change consecutive to a health check. Please consult HAProxy's
879configuration manual for more information regarding all possible log settings.
880
Dan Lloyd8e48b872016-07-01 21:01:18 -0400881It is convenient to chose a facility that is not used by other daemons. HAProxy
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200882examples often suggest "local0" for traffic logs and "local1" for admin logs
883because they're never seen in field. A single facility would be enough as well.
884Having separate logs is convenient for log analysis, but it's also important to
885remember that logs may sometimes convey confidential information, and as such
Dan Lloyd8e48b872016-07-01 21:01:18 -0400886they must not be mixed with other logs that may accidentally be handed out to
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200887unauthorized people.
888
889For in-field troubleshooting without impacting the server's capacity too much,
890it is recommended to make use of the "halog" utility provided with HAProxy.
891This is sort of a grep-like utility designed to process HAProxy log files at
892a very fast data rate. Typical figures range between 1 and 2 GB of logs per
893second. It is capable of extracting only certain logs (eg: search for some
894classes of HTTP status codes, connection termination status, search by response
895time ranges, look for errors only), count lines, limit the output to a number
896of lines, and perform some more advanced statistics such as sorting servers
897by response time or error counts, sorting URLs by time or count, sorting client
898addresses by access count, and so on. It is pretty convenient to quickly spot
899anomalies such as a bot looping on the site, and block them.
900
901
9029. Statistics and monitoring
903----------------------------
904
Willy Tarreau44aed902015-10-13 14:45:29 +0200905It is possible to query HAProxy about its status. The most commonly used
906mechanism is the HTTP statistics page. This page also exposes an alternative
907CSV output format for monitoring tools. The same format is provided on the
908Unix socket.
909
910
9119.1. CSV format
912---------------
913
914The statistics may be consulted either from the unix socket or from the HTTP
915page. Both means provide a CSV format whose fields follow. The first line
916begins with a sharp ('#') and has one word per comma-delimited field which
917represents the title of the column. All other lines starting at the second one
918use a classical CSV format using a comma as the delimiter, and the double quote
919('"') as an optional text delimiter, but only if the enclosed text is ambiguous
920(if it contains a quote or a comma). The double-quote character ('"') in the
921text is doubled ('""'), which is the format that most tools recognize. Please
922do not insert any column before these ones in order not to break tools which
923use hard-coded column positions.
924
925In brackets after each field name are the types which may have a value for
926that field. The types are L (Listeners), F (Frontends), B (Backends), and
927S (Servers).
928
929 0. pxname [LFBS]: proxy name
930 1. svname [LFBS]: service name (FRONTEND for frontend, BACKEND for backend,
931 any name for server/listener)
932 2. qcur [..BS]: current queued requests. For the backend this reports the
933 number queued without a server assigned.
934 3. qmax [..BS]: max value of qcur
935 4. scur [LFBS]: current sessions
936 5. smax [LFBS]: max sessions
937 6. slim [LFBS]: configured session limit
Willy Tarreauc73810f2016-01-11 13:52:04 +0100938 7. stot [LFBS]: cumulative number of sessions
Willy Tarreau44aed902015-10-13 14:45:29 +0200939 8. bin [LFBS]: bytes in
940 9. bout [LFBS]: bytes out
941 10. dreq [LFB.]: requests denied because of security concerns.
942 - For tcp this is because of a matched tcp-request content rule.
943 - For http this is because of a matched http-request or tarpit rule.
944 11. dresp [LFBS]: responses denied because of security concerns.
945 - For http this is because of a matched http-request rule, or
946 "option checkcache".
947 12. ereq [LF..]: request errors. Some of the possible causes are:
948 - early termination from the client, before the request has been sent.
949 - read error from the client
950 - client timeout
951 - client closed connection
952 - various bad requests from the client.
953 - request was tarpitted.
954 13. econ [..BS]: number of requests that encountered an error trying to
955 connect to a backend server. The backend stat is the sum of the stat
956 for all servers of that backend, plus any connection errors not
957 associated with a particular server (such as the backend having no
958 active servers).
959 14. eresp [..BS]: response errors. srv_abrt will be counted here also.
960 Some other errors are:
961 - write error on the client socket (won't be counted for the server stat)
962 - failure applying filters to the response.
963 15. wretr [..BS]: number of times a connection to a server was retried.
964 16. wredis [..BS]: number of times a request was redispatched to another
965 server. The server value counts the number of times that server was
966 switched away from.
Willy Tarreaub96dd282016-11-09 14:45:51 +0100967 17. status [LFBS]: status (UP/DOWN/NOLB/MAINT/MAINT(via)/MAINT(resolution)...)
Willy Tarreau44aed902015-10-13 14:45:29 +0200968 18. weight [..BS]: total weight (backend), server weight (server)
969 19. act [..BS]: number of active servers (backend), server is active (server)
970 20. bck [..BS]: number of backup servers (backend), server is backup (server)
971 21. chkfail [...S]: number of failed checks. (Only counts checks failed when
972 the server is up.)
973 22. chkdown [..BS]: number of UP->DOWN transitions. The backend counter counts
974 transitions to the whole backend being down, rather than the sum of the
975 counters for each server.
976 23. lastchg [..BS]: number of seconds since the last UP<->DOWN transition
977 24. downtime [..BS]: total downtime (in seconds). The value for the backend
978 is the downtime for the whole backend, not the sum of the server downtime.
979 25. qlimit [...S]: configured maxqueue for the server, or nothing in the
980 value is 0 (default, meaning no limit)
981 26. pid [LFBS]: process id (0 for first instance, 1 for second, ...)
982 27. iid [LFBS]: unique proxy id
983 28. sid [L..S]: server id (unique inside a proxy)
984 29. throttle [...S]: current throttle percentage for the server, when
985 slowstart is active, or no value if not in slowstart.
986 30. lbtot [..BS]: total number of times a server was selected, either for new
987 sessions, or when re-dispatching. The server counter is the number
988 of times that server was selected.
989 31. tracked [...S]: id of proxy/server if tracking is enabled.
990 32. type [LFBS]: (0=frontend, 1=backend, 2=server, 3=socket/listener)
991 33. rate [.FBS]: number of sessions per second over last elapsed second
992 34. rate_lim [.F..]: configured limit on new sessions per second
993 35. rate_max [.FBS]: max number of new sessions per second
994 36. check_status [...S]: status of last health check, one of:
995 UNK -> unknown
996 INI -> initializing
997 SOCKERR -> socket error
998 L4OK -> check passed on layer 4, no upper layers testing enabled
999 L4TOUT -> layer 1-4 timeout
1000 L4CON -> layer 1-4 connection problem, for example
1001 "Connection refused" (tcp rst) or "No route to host" (icmp)
1002 L6OK -> check passed on layer 6
1003 L6TOUT -> layer 6 (SSL) timeout
1004 L6RSP -> layer 6 invalid response - protocol error
1005 L7OK -> check passed on layer 7
1006 L7OKC -> check conditionally passed on layer 7, for example 404 with
1007 disable-on-404
1008 L7TOUT -> layer 7 (HTTP/SMTP) timeout
1009 L7RSP -> layer 7 invalid response - protocol error
1010 L7STS -> layer 7 response error, for example HTTP 5xx
Daniel Schnellerb6c8b0d2017-09-01 19:13:55 +02001011 Notice: If a check is currently running, the last known status will be
1012 reported, prefixed with "* ". e. g. "* L7OK".
Willy Tarreau44aed902015-10-13 14:45:29 +02001013 37. check_code [...S]: layer5-7 code, if available
1014 38. check_duration [...S]: time in ms took to finish last health check
1015 39. hrsp_1xx [.FBS]: http responses with 1xx code
1016 40. hrsp_2xx [.FBS]: http responses with 2xx code
1017 41. hrsp_3xx [.FBS]: http responses with 3xx code
1018 42. hrsp_4xx [.FBS]: http responses with 4xx code
1019 43. hrsp_5xx [.FBS]: http responses with 5xx code
1020 44. hrsp_other [.FBS]: http responses with other codes (protocol error)
1021 45. hanafail [...S]: failed health checks details
1022 46. req_rate [.F..]: HTTP requests per second over last elapsed second
1023 47. req_rate_max [.F..]: max number of HTTP requests per second observed
Willy Tarreaufb981bd2016-12-12 14:31:46 +01001024 48. req_tot [.FB.]: total number of HTTP requests received
Willy Tarreau44aed902015-10-13 14:45:29 +02001025 49. cli_abrt [..BS]: number of data transfers aborted by the client
1026 50. srv_abrt [..BS]: number of data transfers aborted by the server
1027 (inc. in eresp)
1028 51. comp_in [.FB.]: number of HTTP response bytes fed to the compressor
1029 52. comp_out [.FB.]: number of HTTP response bytes emitted by the compressor
1030 53. comp_byp [.FB.]: number of bytes that bypassed the HTTP compressor
1031 (CPU/BW limit)
1032 54. comp_rsp [.FB.]: number of HTTP responses that were compressed
1033 55. lastsess [..BS]: number of seconds since last session assigned to
1034 server/backend
1035 56. last_chk [...S]: last health check contents or textual error
1036 57. last_agt [...S]: last agent check contents or textual error
1037 58. qtime [..BS]: the average queue time in ms over the 1024 last requests
1038 59. ctime [..BS]: the average connect time in ms over the 1024 last requests
1039 60. rtime [..BS]: the average response time in ms over the 1024 last requests
1040 (0 for TCP)
1041 61. ttime [..BS]: the average total session time in ms over the 1024 last
1042 requests
Willy Tarreau7f618842016-01-08 11:40:03 +01001043 62. agent_status [...S]: status of last agent check, one of:
1044 UNK -> unknown
1045 INI -> initializing
1046 SOCKERR -> socket error
1047 L4OK -> check passed on layer 4, no upper layers testing enabled
1048 L4TOUT -> layer 1-4 timeout
1049 L4CON -> layer 1-4 connection problem, for example
1050 "Connection refused" (tcp rst) or "No route to host" (icmp)
1051 L7OK -> agent reported "up"
1052 L7STS -> agent reported "fail", "stop", or "down"
1053 63. agent_code [...S]: numeric code reported by agent if any (unused for now)
1054 64. agent_duration [...S]: time in ms taken to finish last check
Willy Tarreaudd7354b2016-01-08 13:47:26 +01001055 65. check_desc [...S]: short human-readable description of check_status
1056 66. agent_desc [...S]: short human-readable description of agent_status
Willy Tarreau3141f592016-01-08 14:25:28 +01001057 67. check_rise [...S]: server's "rise" parameter used by checks
1058 68. check_fall [...S]: server's "fall" parameter used by checks
1059 69. check_health [...S]: server's health check value between 0 and rise+fall-1
1060 70. agent_rise [...S]: agent's "rise" parameter, normally 1
1061 71. agent_fall [...S]: agent's "fall" parameter, normally 1
1062 72. agent_health [...S]: agent's health parameter, between 0 and rise+fall-1
Willy Tarreaua6f5a732016-01-08 16:59:56 +01001063 73. addr [L..S]: address:port or "unix". IPv6 has brackets around the address.
Willy Tarreaue4847c62016-01-08 15:43:54 +01001064 74: cookie [..BS]: server's cookie value or backend's cookie name
Willy Tarreauf8211df2016-01-11 14:09:38 +01001065 75: mode [LFBS]: proxy mode (tcp, http, health, unknown)
Willy Tarreauf1516d92016-01-11 14:48:36 +01001066 76: algo [..B.]: load balancing algorithm
Willy Tarreauc73810f2016-01-11 13:52:04 +01001067 77: conn_rate [.F..]: number of connections over the last elapsed second
1068 78: conn_rate_max [.F..]: highest known conn_rate
1069 79: conn_tot [.F..]: cumulative number of connections
Willy Tarreau5b9bdff2016-01-11 14:40:47 +01001070 80: intercepted [.FB.]: cum. number of intercepted requests (monitor, stats)
Willy Tarreau8a90b8e2016-10-21 18:15:32 +02001071 81: dcon [LF..]: requests denied by "tcp-request connection" rules
Willy Tarreaua5bc36b2016-10-21 18:16:27 +02001072 82: dses [LF..]: requests denied by "tcp-request session" rules
Willy Tarreauea96a822018-05-28 15:15:43 +02001073 83: wrew [LFBS]: cumulative number of failed header rewriting warnings
Willy Tarreau44aed902015-10-13 14:45:29 +02001074
1075
Willy Tarreau5d8b9792016-03-11 11:09:34 +010010769.2) Typed output format
1077------------------------
1078
1079Both "show info" and "show stat" support a mode where each output value comes
1080with its type and sufficient information to know how the value is supposed to
1081be aggregated between processes and how it evolves.
1082
1083In all cases, the output consists in having a single value per line with all
1084the information split into fields delimited by colons (':').
1085
1086The first column designates the object or metric being dumped. Its format is
1087specific to the command producing this output and will not be described in this
1088section. Usually it will consist in a series of identifiers and field names.
1089
1090The second column contains 3 characters respectively indicating the origin, the
1091nature and the scope of the value being reported. The first character (the
1092origin) indicates where the value was extracted from. Possible characters are :
1093
1094 M The value is a metric. It is valid at one instant any may change depending
1095 on its nature .
1096
1097 S The value is a status. It represents a discrete value which by definition
1098 cannot be aggregated. It may be the status of a server ("UP" or "DOWN"),
1099 the PID of the process, etc.
1100
1101 K The value is a sorting key. It represents an identifier which may be used
1102 to group some values together because it is unique among its class. All
1103 internal identifiers are keys. Some names can be listed as keys if they
1104 are unique (eg: a frontend name is unique). In general keys come from the
Dan Lloyd8e48b872016-07-01 21:01:18 -04001105 configuration, even though some of them may automatically be assigned. For
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001106 most purposes keys may be considered as equivalent to configuration.
1107
1108 C The value comes from the configuration. Certain configuration values make
1109 sense on the output, for example a concurrent connection limit or a cookie
1110 name. By definition these values are the same in all processes started
1111 from the same configuration file.
1112
1113 P The value comes from the product itself. There are very few such values,
1114 most common use is to report the product name, version and release date.
1115 These elements are also the same between all processes.
1116
1117The second character (the nature) indicates the nature of the information
1118carried by the field in order to let an aggregator decide on what operation to
1119use to aggregate multiple values. Possible characters are :
1120
1121 A The value represents an age since a last event. This is a bit different
1122 from the duration in that an age is automatically computed based on the
1123 current date. A typical example is how long ago did the last session
1124 happen on a server. Ages are generally aggregated by taking the minimum
1125 value and do not need to be stored.
1126
1127 a The value represents an already averaged value. The average response times
1128 and server weights are of this nature. Averages can typically be averaged
1129 between processes.
1130
1131 C The value represents a cumulative counter. Such measures perpetually
1132 increase until they wrap around. Some monitoring protocols need to tell
1133 the difference between a counter and a gauge to report a different type.
1134 In general counters may simply be summed since they represent events or
1135 volumes. Examples of metrics of this nature are connection counts or byte
1136 counts.
1137
1138 D The value represents a duration for a status. There are a few usages of
1139 this, most of them include the time taken by the last health check and
1140 the time a server has spent down. Durations are generally not summed,
1141 most of the time the maximum will be retained to compute an SLA.
1142
1143 G The value represents a gauge. It's a measure at one instant. The memory
1144 usage or the current number of active connections are of this nature.
1145 Metrics of this type are typically summed during aggregation.
1146
1147 L The value represents a limit (generally a configured one). By nature,
1148 limits are harder to aggregate since they are specific to the point where
1149 they were retrieved. In certain situations they may be summed or be kept
1150 separate.
1151
1152 M The value represents a maximum. In general it will apply to a gauge and
1153 keep the highest known value. An example of such a metric could be the
1154 maximum amount of concurrent connections that was encountered in the
1155 product's life time. To correctly aggregate maxima, you are supposed to
1156 output a range going from the maximum of all maxima and the sum of all
1157 of them. There is indeed no way to know if they were encountered
1158 simultaneously or not.
1159
1160 m The value represents a minimum. In general it will apply to a gauge and
1161 keep the lowest known value. An example of such a metric could be the
1162 minimum amount of free memory pools that was encountered in the product's
1163 life time. To correctly aggregate minima, you are supposed to output a
1164 range going from the minimum of all minima and the sum of all of them.
1165 There is indeed no way to know if they were encountered simultaneously
1166 or not.
1167
1168 N The value represents a name, so it is a string. It is used to report
1169 proxy names, server names and cookie names. Names have configuration or
1170 keys as their origin and are supposed to be the same among all processes.
1171
1172 O The value represents a free text output. Outputs from various commands,
1173 returns from health checks, node descriptions are of such nature.
1174
1175 R The value represents an event rate. It's a measure at one instant. It is
1176 quite similar to a gauge except that the recipient knows that this measure
1177 moves slowly and may decide not to keep all values. An example of such a
1178 metric is the measured amount of connections per second. Metrics of this
1179 type are typically summed during aggregation.
1180
1181 T The value represents a date or time. A field emitting the current date
1182 would be of this type. The method to aggregate such information is left
1183 as an implementation choice. For now no field uses this type.
1184
1185The third character (the scope) indicates what extent the value reflects. Some
1186elements may be per process while others may be per configuration or per system.
1187The distinction is important to know whether or not a single value should be
1188kept during aggregation or if values have to be aggregated. The following
1189characters are currently supported :
1190
1191 C The value is valid for a whole cluster of nodes, which is the set of nodes
1192 communicating over the peers protocol. An example could be the amount of
1193 entries present in a stick table that is replicated with other peers. At
1194 the moment no metric use this scope.
1195
1196 P The value is valid only for the process reporting it. Most metrics use
1197 this scope.
1198
1199 S The value is valid for the whole service, which is the set of processes
1200 started together from the same configuration file. All metrics originating
1201 from the configuration use this scope. Some other metrics may use it as
1202 well for some shared resources (eg: shared SSL cache statistics).
1203
1204 s The value is valid for the whole system, such as the system's hostname,
1205 current date or resource usage. At the moment this scope is not used by
1206 any metric.
1207
1208Consumers of these information will generally have enough of these 3 characters
1209to determine how to accurately report aggregated information across multiple
1210processes.
1211
1212After this column, the third column indicates the type of the field, among "s32"
1213(signed 32-bit integer), "s64" (signed 64-bit integer), "u32" (unsigned 32-bit
1214integer), "u64" (unsigned 64-bit integer), "str" (string). It is important to
1215know the type before parsing the value in order to properly read it. For example
1216a string containing only digits is still a string an not an integer (eg: an
1217error code extracted by a check).
1218
1219Then the fourth column is the value itself, encoded according to its type.
1220Strings are dumped as-is immediately after the colon without any leading space.
1221If a string contains a colon, it will appear normally. This means that the
1222output should not be exclusively split around colons or some check outputs
1223or server addresses might be truncated.
1224
1225
12269.3. Unix Socket commands
Willy Tarreau44aed902015-10-13 14:45:29 +02001227-------------------------
1228
1229The stats socket is not enabled by default. In order to enable it, it is
1230necessary to add one line in the global section of the haproxy configuration.
1231A second line is recommended to set a larger timeout, always appreciated when
1232issuing commands by hand :
1233
1234 global
1235 stats socket /var/run/haproxy.sock mode 600 level admin
1236 stats timeout 2m
1237
1238It is also possible to add multiple instances of the stats socket by repeating
1239the line, and make them listen to a TCP port instead of a UNIX socket. This is
1240never done by default because this is dangerous, but can be handy in some
1241situations :
1242
1243 global
1244 stats socket /var/run/haproxy.sock mode 600 level admin
1245 stats socket ipv4@192.168.0.1:9999 level admin
1246 stats timeout 2m
1247
1248To access the socket, an external utility such as "socat" is required. Socat is
1249a swiss-army knife to connect anything to anything. We use it to connect
1250terminals to the socket, or a couple of stdin/stdout pipes to it for scripts.
1251The two main syntaxes we'll use are the following :
1252
1253 # socat /var/run/haproxy.sock stdio
1254 # socat /var/run/haproxy.sock readline
1255
1256The first one is used with scripts. It is possible to send the output of a
1257script to haproxy, and pass haproxy's output to another script. That's useful
1258for retrieving counters or attack traces for example.
1259
1260The second one is only useful for issuing commands by hand. It has the benefit
1261that the terminal is handled by the readline library which supports line
1262editing and history, which is very convenient when issuing repeated commands
1263(eg: watch a counter).
1264
1265The socket supports two operation modes :
1266 - interactive
1267 - non-interactive
1268
1269The non-interactive mode is the default when socat connects to the socket. In
1270this mode, a single line may be sent. It is processed as a whole, responses are
1271sent back, and the connection closes after the end of the response. This is the
1272mode that scripts and monitoring tools use. It is possible to send multiple
1273commands in this mode, they need to be delimited by a semi-colon (';'). For
1274example :
1275
1276 # echo "show info;show stat;show table" | socat /var/run/haproxy stdio
1277
Dragan Dosena1c35ab2016-11-24 11:33:12 +01001278If a command needs to use a semi-colon or a backslash (eg: in a value), it
1279must be preceeded by a backslash ('\').
Chad Lavoiee3f50312016-05-26 16:42:25 -04001280
Willy Tarreau44aed902015-10-13 14:45:29 +02001281The interactive mode displays a prompt ('>') and waits for commands to be
1282entered on the line, then processes them, and displays the prompt again to wait
1283for a new command. This mode is entered via the "prompt" command which must be
1284sent on the first line in non-interactive mode. The mode is a flip switch, if
1285"prompt" is sent in interactive mode, it is disabled and the connection closes
1286after processing the last command of the same line.
1287
1288For this reason, when debugging by hand, it's quite common to start with the
1289"prompt" command :
1290
1291 # socat /var/run/haproxy readline
1292 prompt
1293 > show info
1294 ...
1295 >
1296
1297Since multiple commands may be issued at once, haproxy uses the empty line as a
1298delimiter to mark an end of output for each command, and takes care of ensuring
1299that no command can emit an empty line on output. A script can thus easily
1300parse the output even when multiple commands were pipelined on a single line.
1301
Aurélien Nephtaliabbf6072018-04-18 13:26:46 +02001302Some commands may take an optional payload. To add one to a command, the first
1303line needs to end with the "<<\n" pattern. The next lines will be treated as
1304the payload and can contain as many lines as needed. To validate a command with
1305a payload, it needs to end with an empty line.
1306
1307Limitations do exist: the length of the whole buffer passed to the CLI must
1308not be greater than tune.bfsize and the pattern "<<" must not be glued to the
1309last word of the line.
1310
1311When entering a paylod while in interactive mode, the prompt will change from
1312"> " to "+ ".
1313
Willy Tarreau44aed902015-10-13 14:45:29 +02001314It is important to understand that when multiple haproxy processes are started
1315on the same sockets, any process may pick up the request and will output its
1316own stats.
1317
1318The list of commands currently supported on the stats socket is provided below.
1319If an unknown command is sent, haproxy displays the usage message which reminds
1320all supported commands. Some commands support a more complex syntax, generally
1321it will explain what part of the command is invalid when this happens.
1322
Olivier Doucetd8703e82017-08-31 11:05:10 +02001323Some commands require a higher level of privilege to work. If you do not have
1324enough privilege, you will get an error "Permission denied". Please check
1325the "level" option of the "bind" keyword lines in the configuration manual
1326for more information.
1327
Willy Tarreau44aed902015-10-13 14:45:29 +02001328add acl <acl> <pattern>
1329 Add an entry into the acl <acl>. <acl> is the #<id> or the <file> returned by
1330 "show acl". This command does not verify if the entry already exists. This
1331 command cannot be used if the reference <acl> is a file also used with a map.
1332 In this case, you must use the command "add map" in place of "add acl".
1333
1334add map <map> <key> <value>
Aurélien Nephtali25650ce2018-04-18 14:04:47 +02001335add map <map> <payload>
Willy Tarreau44aed902015-10-13 14:45:29 +02001336 Add an entry into the map <map> to associate the value <value> to the key
1337 <key>. This command does not verify if the entry already exists. It is
1338 mainly used to fill a map after a clear operation. Note that if the reference
1339 <map> is a file and is shared with a map, this map will contain also a new
Aurélien Nephtali25650ce2018-04-18 14:04:47 +02001340 pattern entry. Using the payload syntax it is possible to add multiple
1341 key/value pairs by entering them on separate lines. On each new line, the
1342 first word is the key and the rest of the line is considered to be the value
1343 which can even contains spaces.
1344
1345 Example:
1346
1347 # socat /tmp/sock1 -
1348 prompt
1349
1350 > add map #-1 <<
1351 + key1 value1
1352 + key2 value2 with spaces
1353 + key3 value3 also with spaces
1354 + key4 value4
1355
1356 >
Willy Tarreau44aed902015-10-13 14:45:29 +02001357
1358clear counters
1359 Clear the max values of the statistics counters in each proxy (frontend &
Willy Tarreaud80cb4e2018-01-20 19:30:13 +01001360 backend) and in each server. The accumulated counters are not affected. The
1361 internal activity counters reported by "show activity" are also reset. This
Willy Tarreau44aed902015-10-13 14:45:29 +02001362 can be used to get clean counters after an incident, without having to
1363 restart nor to clear traffic counters. This command is restricted and can
1364 only be issued on sockets configured for levels "operator" or "admin".
1365
1366clear counters all
1367 Clear all statistics counters in each proxy (frontend & backend) and in each
1368 server. This has the same effect as restarting. This command is restricted
1369 and can only be issued on sockets configured for level "admin".
1370
1371clear acl <acl>
1372 Remove all entries from the acl <acl>. <acl> is the #<id> or the <file>
1373 returned by "show acl". Note that if the reference <acl> is a file and is
1374 shared with a map, this map will be also cleared.
1375
1376clear map <map>
1377 Remove all entries from the map <map>. <map> is the #<id> or the <file>
1378 returned by "show map". Note that if the reference <map> is a file and is
1379 shared with a acl, this acl will be also cleared.
1380
1381clear table <table> [ data.<type> <operator> <value> ] | [ key <key> ]
1382 Remove entries from the stick-table <table>.
1383
1384 This is typically used to unblock some users complaining they have been
1385 abusively denied access to a service, but this can also be used to clear some
1386 stickiness entries matching a server that is going to be replaced (see "show
1387 table" below for details). Note that sometimes, removal of an entry will be
1388 refused because it is currently tracked by a session. Retrying a few seconds
1389 later after the session ends is usual enough.
1390
1391 In the case where no options arguments are given all entries will be removed.
1392
1393 When the "data." form is used entries matching a filter applied using the
1394 stored data (see "stick-table" in section 4.2) are removed. A stored data
1395 type must be specified in <type>, and this data type must be stored in the
1396 table otherwise an error is reported. The data is compared according to
1397 <operator> with the 64-bit integer <value>. Operators are the same as with
1398 the ACLs :
1399
1400 - eq : match entries whose data is equal to this value
1401 - ne : match entries whose data is not equal to this value
1402 - le : match entries whose data is less than or equal to this value
1403 - ge : match entries whose data is greater than or equal to this value
1404 - lt : match entries whose data is less than this value
1405 - gt : match entries whose data is greater than this value
1406
1407 When the key form is used the entry <key> is removed. The key must be of the
1408 same type as the table, which currently is limited to IPv4, IPv6, integer and
1409 string.
1410
1411 Example :
1412 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1413 >>> # table: http_proxy, type: ip, size:204800, used:2
1414 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
1415 bytes_out_rate(60000)=187
1416 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
1417 bytes_out_rate(60000)=191
1418
1419 $ echo "clear table http_proxy key 127.0.0.1" | socat stdio /tmp/sock1
1420
1421 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1422 >>> # table: http_proxy, type: ip, size:204800, used:1
1423 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
1424 bytes_out_rate(60000)=191
1425 $ echo "clear table http_proxy data.gpc0 eq 1" | socat stdio /tmp/sock1
1426 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1427 >>> # table: http_proxy, type: ip, size:204800, used:1
1428
1429del acl <acl> [<key>|#<ref>]
1430 Delete all the acl entries from the acl <acl> corresponding to the key <key>.
1431 <acl> is the #<id> or the <file> returned by "show acl". If the <ref> is used,
1432 this command delete only the listed reference. The reference can be found with
1433 listing the content of the acl. Note that if the reference <acl> is a file and
1434 is shared with a map, the entry will be also deleted in the map.
1435
1436del map <map> [<key>|#<ref>]
1437 Delete all the map entries from the map <map> corresponding to the key <key>.
1438 <map> is the #<id> or the <file> returned by "show map". If the <ref> is used,
1439 this command delete only the listed reference. The reference can be found with
1440 listing the content of the map. Note that if the reference <map> is a file and
1441 is shared with a acl, the entry will be also deleted in the map.
1442
1443disable agent <backend>/<server>
1444 Mark the auxiliary agent check as temporarily stopped.
1445
1446 In the case where an agent check is being run as a auxiliary check, due
1447 to the agent-check parameter of a server directive, new checks are only
Dan Lloyd8e48b872016-07-01 21:01:18 -04001448 initialized when the agent is in the enabled. Thus, disable agent will
Willy Tarreau44aed902015-10-13 14:45:29 +02001449 prevent any new agent checks from begin initiated until the agent
1450 re-enabled using enable agent.
1451
1452 When an agent is disabled the processing of an auxiliary agent check that
1453 was initiated while the agent was set as enabled is as follows: All
1454 results that would alter the weight, specifically "drain" or a weight
1455 returned by the agent, are ignored. The processing of agent check is
1456 otherwise unchanged.
1457
1458 The motivation for this feature is to allow the weight changing effects
1459 of the agent checks to be paused to allow the weight of a server to be
1460 configured using set weight without being overridden by the agent.
1461
1462 This command is restricted and can only be issued on sockets configured for
1463 level "admin".
1464
Olivier Houchard614f8d72017-03-14 20:08:46 +01001465disable dynamic-cookie backend <backend>
1466 Disable the generation of dynamic cookies fot the backend <backend>
1467
Willy Tarreau44aed902015-10-13 14:45:29 +02001468disable frontend <frontend>
1469 Mark the frontend as temporarily stopped. This corresponds to the mode which
1470 is used during a soft restart : the frontend releases the port but can be
1471 enabled again if needed. This should be used with care as some non-Linux OSes
1472 are unable to enable it back. This is intended to be used in environments
1473 where stopping a proxy is not even imaginable but a misconfigured proxy must
1474 be fixed. That way it's possible to release the port and bind it into another
1475 process to restore operations. The frontend will appear with status "STOP"
1476 on the stats page.
1477
1478 The frontend may be specified either by its name or by its numeric ID,
1479 prefixed with a sharp ('#').
1480
1481 This command is restricted and can only be issued on sockets configured for
1482 level "admin".
1483
1484disable health <backend>/<server>
1485 Mark the primary health check as temporarily stopped. This will disable
1486 sending of health checks, and the last health check result will be ignored.
1487 The server will be in unchecked state and considered UP unless an auxiliary
1488 agent check forces it down.
1489
1490 This command is restricted and can only be issued on sockets configured for
1491 level "admin".
1492
1493disable server <backend>/<server>
1494 Mark the server DOWN for maintenance. In this mode, no more checks will be
1495 performed on the server until it leaves maintenance.
1496 If the server is tracked by other servers, those servers will be set to DOWN
1497 during the maintenance.
1498
1499 In the statistics page, a server DOWN for maintenance will appear with a
1500 "MAINT" status, its tracking servers with the "MAINT(via)" one.
1501
1502 Both the backend and the server may be specified either by their name or by
1503 their numeric ID, prefixed with a sharp ('#').
1504
1505 This command is restricted and can only be issued on sockets configured for
1506 level "admin".
1507
1508enable agent <backend>/<server>
1509 Resume auxiliary agent check that was temporarily stopped.
1510
1511 See "disable agent" for details of the effect of temporarily starting
1512 and stopping an auxiliary agent.
1513
1514 This command is restricted and can only be issued on sockets configured for
1515 level "admin".
1516
Olivier Houchard614f8d72017-03-14 20:08:46 +01001517enable dynamic-cookie backend <backend>
1518 Enable the generation of dynamic cookies fot the backend <backend>
1519 A secret key must also be provided
1520
Willy Tarreau44aed902015-10-13 14:45:29 +02001521enable frontend <frontend>
1522 Resume a frontend which was temporarily stopped. It is possible that some of
1523 the listening ports won't be able to bind anymore (eg: if another process
1524 took them since the 'disable frontend' operation). If this happens, an error
1525 is displayed. Some operating systems might not be able to resume a frontend
1526 which was disabled.
1527
1528 The frontend may be specified either by its name or by its numeric ID,
1529 prefixed with a sharp ('#').
1530
1531 This command is restricted and can only be issued on sockets configured for
1532 level "admin".
1533
1534enable health <backend>/<server>
1535 Resume a primary health check that was temporarily stopped. This will enable
1536 sending of health checks again. Please see "disable health" for details.
1537
1538 This command is restricted and can only be issued on sockets configured for
1539 level "admin".
1540
1541enable server <backend>/<server>
1542 If the server was previously marked as DOWN for maintenance, this marks the
1543 server UP and checks are re-enabled.
1544
1545 Both the backend and the server may be specified either by their name or by
1546 their numeric ID, prefixed with a sharp ('#').
1547
1548 This command is restricted and can only be issued on sockets configured for
1549 level "admin".
1550
1551get map <map> <value>
1552get acl <acl> <value>
1553 Lookup the value <value> in the map <map> or in the ACL <acl>. <map> or <acl>
1554 are the #<id> or the <file> returned by "show map" or "show acl". This command
1555 returns all the matching patterns associated with this map. This is useful for
1556 debugging maps and ACLs. The output format is composed by one line par
1557 matching type. Each line is composed by space-delimited series of words.
1558
1559 The first two words are:
1560
1561 <match method>: The match method applied. It can be "found", "bool",
1562 "int", "ip", "bin", "len", "str", "beg", "sub", "dir",
1563 "dom", "end" or "reg".
1564
1565 <match result>: The result. Can be "match" or "no-match".
1566
1567 The following words are returned only if the pattern matches an entry.
1568
1569 <index type>: "tree" or "list". The internal lookup algorithm.
1570
1571 <case>: "case-insensitive" or "case-sensitive". The
1572 interpretation of the case.
1573
1574 <entry matched>: match="<entry>". Return the matched pattern. It is
1575 useful with regular expressions.
1576
1577 The two last word are used to show the returned value and its type. With the
1578 "acl" case, the pattern doesn't exist.
1579
1580 return=nothing: No return because there are no "map".
1581 return="<value>": The value returned in the string format.
1582 return=cannot-display: The value cannot be converted as string.
1583
1584 type="<type>": The type of the returned sample.
1585
1586get weight <backend>/<server>
1587 Report the current weight and the initial weight of server <server> in
1588 backend <backend> or an error if either doesn't exist. The initial weight is
1589 the one that appears in the configuration file. Both are normally equal
1590 unless the current weight has been changed. Both the backend and the server
1591 may be specified either by their name or by their numeric ID, prefixed with a
1592 sharp ('#').
1593
1594help
1595 Print the list of known keywords and their basic usage. The same help screen
1596 is also displayed for unknown commands.
1597
1598prompt
1599 Toggle the prompt at the beginning of the line and enter or leave interactive
1600 mode. In interactive mode, the connection is not closed after a command
1601 completes. Instead, the prompt will appear again, indicating the user that
1602 the interpreter is waiting for a new command. The prompt consists in a right
1603 angle bracket followed by a space "> ". This mode is particularly convenient
1604 when one wants to periodically check information such as stats or errors.
1605 It is also a good idea to enter interactive mode before issuing a "help"
1606 command.
1607
1608quit
1609 Close the connection when in interactive mode.
1610
Olivier Houchard614f8d72017-03-14 20:08:46 +01001611set dynamic-cookie-key backend <backend> <value>
1612 Modify the secret key used to generate the dynamic persistent cookies.
1613 This will break the existing sessions.
1614
Willy Tarreau44aed902015-10-13 14:45:29 +02001615set map <map> [<key>|#<ref>] <value>
1616 Modify the value corresponding to each key <key> in a map <map>. <map> is the
1617 #<id> or <file> returned by "show map". If the <ref> is used in place of
1618 <key>, only the entry pointed by <ref> is changed. The new value is <value>.
1619
1620set maxconn frontend <frontend> <value>
1621 Dynamically change the specified frontend's maxconn setting. Any positive
1622 value is allowed including zero, but setting values larger than the global
1623 maxconn does not make much sense. If the limit is increased and connections
1624 were pending, they will immediately be accepted. If it is lowered to a value
1625 below the current number of connections, new connections acceptation will be
1626 delayed until the threshold is reached. The frontend might be specified by
1627 either its name or its numeric ID prefixed with a sharp ('#').
1628
Andrew Hayworthedb93a72015-10-27 21:46:25 +00001629set maxconn server <backend/server> <value>
1630 Dynamically change the specified server's maxconn setting. Any positive
1631 value is allowed including zero, but setting values larger than the global
1632 maxconn does not make much sense.
1633
Willy Tarreau44aed902015-10-13 14:45:29 +02001634set maxconn global <maxconn>
1635 Dynamically change the global maxconn setting within the range defined by the
1636 initial global maxconn setting. If it is increased and connections were
1637 pending, they will immediately be accepted. If it is lowered to a value below
1638 the current number of connections, new connections acceptation will be
1639 delayed until the threshold is reached. A value of zero restores the initial
1640 setting.
1641
1642set rate-limit connections global <value>
1643 Change the process-wide connection rate limit, which is set by the global
1644 'maxconnrate' setting. A value of zero disables the limitation. This limit
1645 applies to all frontends and the change has an immediate effect. The value
1646 is passed in number of connections per second.
1647
1648set rate-limit http-compression global <value>
1649 Change the maximum input compression rate, which is set by the global
1650 'maxcomprate' setting. A value of zero disables the limitation. The value is
1651 passed in number of kilobytes per second. The value is available in the "show
1652 info" on the line "CompressBpsRateLim" in bytes.
1653
1654set rate-limit sessions global <value>
1655 Change the process-wide session rate limit, which is set by the global
1656 'maxsessrate' setting. A value of zero disables the limitation. This limit
1657 applies to all frontends and the change has an immediate effect. The value
1658 is passed in number of sessions per second.
1659
1660set rate-limit ssl-sessions global <value>
1661 Change the process-wide SSL session rate limit, which is set by the global
1662 'maxsslrate' setting. A value of zero disables the limitation. This limit
1663 applies to all frontends and the change has an immediate effect. The value
1664 is passed in number of sessions per second sent to the SSL stack. It applies
1665 before the handshake in order to protect the stack against handshake abuses.
1666
Baptiste Assmann3749ebf2016-08-03 22:34:12 +02001667set server <backend>/<server> addr <ip4 or ip6 address> [port <port>]
Willy Tarreau44aed902015-10-13 14:45:29 +02001668 Replace the current IP address of a server by the one provided.
Baptiste Assmann3749ebf2016-08-03 22:34:12 +02001669 Optionnaly, the port can be changed using the 'port' parameter.
1670 Note that changing the port also support switching from/to port mapping
1671 (notation with +X or -Y), only if a port is configured for the health check.
Willy Tarreau44aed902015-10-13 14:45:29 +02001672
1673set server <backend>/<server> agent [ up | down ]
1674 Force a server's agent to a new state. This can be useful to immediately
1675 switch a server's state regardless of some slow agent checks for example.
1676 Note that the change is propagated to tracking servers if any.
1677
Misiek43972902017-01-09 09:53:06 +01001678set server <backend>/<server> agent-addr <addr>
1679 Change addr for servers agent checks. Allows to migrate agent-checks to
1680 another address at runtime. You can specify both IP and hostname, it will be
1681 resolved.
1682
1683set server <backend>/<server> agent-send <value>
1684 Change agent string sent to agent check target. Allows to update string while
1685 changing server address to keep those two matching.
1686
Willy Tarreau44aed902015-10-13 14:45:29 +02001687set server <backend>/<server> health [ up | stopping | down ]
1688 Force a server's health to a new state. This can be useful to immediately
1689 switch a server's state regardless of some slow health checks for example.
1690 Note that the change is propagated to tracking servers if any.
1691
Baptiste Assmann50946562016-08-31 23:26:29 +02001692set server <backend>/<server> check-port <port>
1693 Change the port used for health checking to <port>
1694
Willy Tarreau44aed902015-10-13 14:45:29 +02001695set server <backend>/<server> state [ ready | drain | maint ]
1696 Force a server's administrative state to a new state. This can be useful to
1697 disable load balancing and/or any traffic to a server. Setting the state to
1698 "ready" puts the server in normal mode, and the command is the equivalent of
1699 the "enable server" command. Setting the state to "maint" disables any traffic
1700 to the server as well as any health checks. This is the equivalent of the
1701 "disable server" command. Setting the mode to "drain" only removes the server
1702 from load balancing but still allows it to be checked and to accept new
1703 persistent connections. Changes are propagated to tracking servers if any.
1704
1705set server <backend>/<server> weight <weight>[%]
1706 Change a server's weight to the value passed in argument. This is the exact
1707 equivalent of the "set weight" command below.
1708
Frédéric Lécailleb418c122017-04-26 11:24:02 +02001709set server <backend>/<server> fqdn <FQDN>
Lukas Tribusc5dd5a52018-08-14 11:39:35 +02001710 Change a server's FQDN to the value passed in argument. This requires the
1711 internal run-time DNS resolver to be configured and enabled for this server.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02001712
Andjelko Iharosc4df59e2017-07-20 11:59:48 +02001713set severity-output [ none | number | string ]
1714 Change the severity output format of the stats socket connected to for the
1715 duration of the current session.
1716
Aurélien Nephtali1e0867c2018-04-18 14:04:58 +02001717set ssl ocsp-response <response | payload>
Willy Tarreau44aed902015-10-13 14:45:29 +02001718 This command is used to update an OCSP Response for a certificate (see "crt"
1719 on "bind" lines). Same controls are performed as during the initial loading of
1720 the response. The <response> must be passed as a base64 encoded string of the
Emmanuel Hocdet2c32d8f2017-05-22 14:58:00 +02001721 DER encoded response from the OCSP server. This command is not supported with
1722 BoringSSL.
Willy Tarreau44aed902015-10-13 14:45:29 +02001723
1724 Example:
1725 openssl ocsp -issuer issuer.pem -cert server.pem \
1726 -host ocsp.issuer.com:80 -respout resp.der
1727 echo "set ssl ocsp-response $(base64 -w 10000 resp.der)" | \
1728 socat stdio /var/run/haproxy.stat
1729
Aurélien Nephtali1e0867c2018-04-18 14:04:58 +02001730 using the payload syntax:
1731 echo -e "set ssl ocsp-response <<\n$(base64 resp.der)\n" | \
1732 socat stdio /var/run/haproxy.stat
1733
Willy Tarreau44aed902015-10-13 14:45:29 +02001734set ssl tls-key <id> <tlskey>
1735 Set the next TLS key for the <id> listener to <tlskey>. This key becomes the
1736 ultimate key, while the penultimate one is used for encryption (others just
1737 decrypt). The oldest TLS key present is overwritten. <id> is either a numeric
1738 #<id> or <file> returned by "show tls-keys". <tlskey> is a base64 encoded 48
1739 bit TLS ticket key (ex. openssl rand -base64 48).
1740
1741set table <table> key <key> [data.<data_type> <value>]*
1742 Create or update a stick-table entry in the table. If the key is not present,
1743 an entry is inserted. See stick-table in section 4.2 to find all possible
1744 values for <data_type>. The most likely use consists in dynamically entering
1745 entries for source IP addresses, with a flag in gpc0 to dynamically block an
1746 IP address or affect its quality of service. It is possible to pass multiple
1747 data_types in a single call.
1748
1749set timeout cli <delay>
1750 Change the CLI interface timeout for current connection. This can be useful
1751 during long debugging sessions where the user needs to constantly inspect
1752 some indicators without being disconnected. The delay is passed in seconds.
1753
1754set weight <backend>/<server> <weight>[%]
1755 Change a server's weight to the value passed in argument. If the value ends
1756 with the '%' sign, then the new weight will be relative to the initially
1757 configured weight. Absolute weights are permitted between 0 and 256.
1758 Relative weights must be positive with the resulting absolute weight is
1759 capped at 256. Servers which are part of a farm running a static
1760 load-balancing algorithm have stricter limitations because the weight
1761 cannot change once set. Thus for these servers, the only accepted values
1762 are 0 and 100% (or 0 and the initial weight). Changes take effect
1763 immediately, though certain LB algorithms require a certain amount of
1764 requests to consider changes. A typical usage of this command is to
1765 disable a server during an update by setting its weight to zero, then to
1766 enable it again after the update by setting it back to 100%. This command
1767 is restricted and can only be issued on sockets configured for level
1768 "admin". Both the backend and the server may be specified either by their
1769 name or by their numeric ID, prefixed with a sharp ('#').
1770
Willy Tarreaud6129fc2017-07-28 16:52:23 +02001771show acl [<acl>]
1772 Dump info about acl converters. Without argument, the list of all available
1773 acls is returned. If a <acl> is specified, its contents are dumped. <acl> if
1774 the #<id> or <file>. The dump format is the same than the map even for the
1775 sample value. The data returned are not a list of available ACL, but are the
1776 list of all patterns composing any ACL. Many of these patterns can be shared
1777 with maps.
1778
1779show backend
1780 Dump the list of backends available in the running process
1781
William Lallemand51132162016-12-16 16:38:58 +01001782show cli sockets
1783 List CLI sockets. The output format is composed of 3 fields separated by
1784 spaces. The first field is the socket address, it can be a unix socket, a
1785 ipv4 address:port couple or a ipv6 one. Socket of other types won't be dump.
1786 The second field describe the level of the socket: 'admin', 'user' or
1787 'operator'. The last field list the processes on which the socket is bound,
1788 separated by commas, it can be numbers or 'all'.
1789
1790 Example :
1791
1792 $ echo 'show cli sockets' | socat stdio /tmp/sock1
1793 # socket lvl processes
1794 /tmp/sock1 admin all
1795 127.0.0.1:9999 user 2,3,4
1796 127.0.0.2:9969 user 2
1797 [::1]:9999 operator 2
1798
William Lallemand86d0df02017-11-24 21:36:45 +01001799show cache
Cyril Bonté7b888f12017-11-26 22:24:31 +01001800 List the configured caches and the objects stored in each cache tree.
William Lallemand86d0df02017-11-24 21:36:45 +01001801
1802 $ echo 'show cache' | socat stdio /tmp/sock1
1803 0x7f6ac6c5b03a: foobar (shctx:0x7f6ac6c5b000, available blocks:3918)
1804 1 2 3 4
1805
1806 1. pointer to the cache structure
1807 2. cache name
1808 3. pointer to the mmap area (shctx)
1809 4. number of blocks available for reuse in the shctx
1810
1811 0x7f6ac6c5b4cc hash:286881868 size:39114 (39 blocks), refcount:9, expire:237
1812 1 2 3 4 5 6
1813
1814 1. pointer to the cache entry
1815 2. first 32 bits of the hash
1816 3. size of the object in bytes
1817 4. number of blocks used for the object
1818 5. number of transactions using the entry
1819 6. expiration time, can be negative if already expired
1820
Willy Tarreauae795722016-02-16 11:27:28 +01001821show env [<name>]
1822 Dump one or all environment variables known by the process. Without any
1823 argument, all variables are dumped. With an argument, only the specified
1824 variable is dumped if it exists. Otherwise "Variable not found" is emitted.
1825 Variables are dumped in the same format as they are stored or returned by the
1826 "env" utility, that is, "<name>=<value>". This can be handy when debugging
1827 certain configuration files making heavy use of environment variables to
1828 ensure that they contain the expected values. This command is restricted and
1829 can only be issued on sockets configured for levels "operator" or "admin".
1830
Willy Tarreau35069f82016-11-25 09:16:37 +01001831show errors [<iid>|<proxy>] [request|response]
Willy Tarreau44aed902015-10-13 14:45:29 +02001832 Dump last known request and response errors collected by frontends and
1833 backends. If <iid> is specified, the limit the dump to errors concerning
Willy Tarreau234ba2d2016-11-25 08:39:10 +01001834 either frontend or backend whose ID is <iid>. Proxy ID "-1" will cause
1835 all instances to be dumped. If a proxy name is specified instead, its ID
Willy Tarreau35069f82016-11-25 09:16:37 +01001836 will be used as the filter. If "request" or "response" is added after the
1837 proxy name or ID, only request or response errors will be dumped. This
1838 command is restricted and can only be issued on sockets configured for
1839 levels "operator" or "admin".
Willy Tarreau44aed902015-10-13 14:45:29 +02001840
1841 The errors which may be collected are the last request and response errors
1842 caused by protocol violations, often due to invalid characters in header
1843 names. The report precisely indicates what exact character violated the
1844 protocol. Other important information such as the exact date the error was
1845 detected, frontend and backend names, the server name (when known), the
1846 internal session ID and the source address which has initiated the session
1847 are reported too.
1848
1849 All characters are returned, and non-printable characters are encoded. The
1850 most common ones (\t = 9, \n = 10, \r = 13 and \e = 27) are encoded as one
1851 letter following a backslash. The backslash itself is encoded as '\\' to
1852 avoid confusion. Other non-printable characters are encoded '\xNN' where
1853 NN is the two-digits hexadecimal representation of the character's ASCII
1854 code.
1855
1856 Lines are prefixed with the position of their first character, starting at 0
1857 for the beginning of the buffer. At most one input line is printed per line,
1858 and large lines will be broken into multiple consecutive output lines so that
1859 the output never goes beyond 79 characters wide. It is easy to detect if a
1860 line was broken, because it will not end with '\n' and the next line's offset
1861 will be followed by a '+' sign, indicating it is a continuation of previous
1862 line.
1863
1864 Example :
Willy Tarreau35069f82016-11-25 09:16:37 +01001865 $ echo "show errors -1 response" | socat stdio /tmp/sock1
Willy Tarreau44aed902015-10-13 14:45:29 +02001866 >>> [04/Mar/2009:15:46:56.081] backend http-in (#2) : invalid response
1867 src 127.0.0.1, session #54, frontend fe-eth0 (#1), server s2 (#1)
1868 response length 213 bytes, error at position 23:
1869
1870 00000 HTTP/1.0 200 OK\r\n
1871 00017 header/bizarre:blah\r\n
1872 00038 Location: blah\r\n
1873 00054 Long-line: this is a very long line which should b
1874 00104+ e broken into multiple lines on the output buffer,
1875 00154+ otherwise it would be too large to print in a ter
1876 00204+ minal\r\n
1877 00211 \r\n
1878
1879 In the example above, we see that the backend "http-in" which has internal
1880 ID 2 has blocked an invalid response from its server s2 which has internal
1881 ID 1. The request was on session 54 initiated by source 127.0.0.1 and
1882 received by frontend fe-eth0 whose ID is 1. The total response length was
1883 213 bytes when the error was detected, and the error was at byte 23. This
1884 is the slash ('/') in header name "header/bizarre", which is not a valid
1885 HTTP character for a header name.
1886
Willy Tarreau7a4a0ac2017-07-25 19:32:50 +02001887show fd [<fd>]
1888 Dump the list of either all open file descriptors or just the one number <fd>
1889 if specified. This is only aimed at developers who need to observe internal
1890 states in order to debug complex issues such as abnormal CPU usages. One fd
1891 is reported per lines, and for each of them, its state in the poller using
1892 upper case letters for enabled flags and lower case for disabled flags, using
1893 "P" for "polled", "R" for "ready", "A" for "active", the events status using
1894 "H" for "hangup", "E" for "error", "O" for "output", "P" for "priority" and
1895 "I" for "input", a few other flags like "N" for "new" (just added into the fd
1896 cache), "U" for "updated" (received an update in the fd cache), "L" for
1897 "linger_risk", "C" for "cloned", then the cached entry position, the pointer
1898 to the internal owner, the pointer to the I/O callback and its name when
1899 known. When the owner is a connection, the connection flags, and the target
1900 are reported (frontend, proxy or server). When the owner is a listener, the
1901 listener's state and its frontend are reported. There is no point in using
1902 this command without a good knowledge of the internals. It's worth noting
1903 that the output format may evolve over time so this output must not be parsed
1904 by tools designed to be durable.
1905
Willy Tarreaud80cb4e2018-01-20 19:30:13 +01001906show activity
1907 Reports some counters about internal events that will help developers and
1908 more generally people who know haproxy well enough to narrow down the causes
1909 of reports of abnormal behaviours. A typical example would be a properly
1910 running process never sleeping and eating 100% of the CPU. The output fields
1911 will be made of one line per metric, and per-thread counters on the same
1912 line. These counters are 32-bit and will wrap during the process' life, which
1913 is not a problem since calls to this command will typically be performed
1914 twice. The fields are purposely not documented so that their exact meaning is
1915 verified in the code where the counters are fed. These values are also reset
1916 by the "clear counters" command.
1917
Simon Horman05ee2132017-01-04 09:37:25 +01001918show info [typed|json]
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001919 Dump info about haproxy status on current process. If "typed" is passed as an
1920 optional argument, field numbers, names and types are emitted as well so that
1921 external monitoring products can easily retrieve, possibly aggregate, then
1922 report information found in fields they don't know. Each field is dumped on
Simon Horman05ee2132017-01-04 09:37:25 +01001923 its own line. If "json" is passed as an optional argument then
1924 information provided by "typed" output is provided in JSON format as a
1925 list of JSON objects. By default, the format contains only two columns
1926 delimited by a colon (':'). The left one is the field name and the right
1927 one is the value. It is very important to note that in typed output
1928 format, the dump for a single object is contiguous so that there is no
1929 need for a consumer to store everything at once.
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001930
1931 When using the typed output format, each line is made of 4 columns delimited
1932 by colons (':'). The first column is a dot-delimited series of 3 elements. The
1933 first element is the numeric position of the field in the list (starting at
1934 zero). This position shall not change over time, but holes are to be expected,
1935 depending on build options or if some fields are deleted in the future. The
1936 second element is the field name as it appears in the default "show info"
1937 output. The third element is the relative process number starting at 1.
1938
1939 The rest of the line starting after the first colon follows the "typed output
1940 format" described in the section above. In short, the second column (after the
1941 first ':') indicates the origin, nature and scope of the variable. The third
1942 column indicates the type of the field, among "s32", "s64", "u32", "u64" and
1943 "str". Then the fourth column is the value itself, which the consumer knows
1944 how to parse thanks to column 3 and how to process thanks to column 2.
1945
1946 Thus the overall line format in typed mode is :
1947
1948 <field_pos>.<field_name>.<process_num>:<tags>:<type>:<value>
1949
1950 Example :
1951
1952 > show info
1953 Name: HAProxy
1954 Version: 1.7-dev1-de52ea-146
1955 Release_date: 2016/03/11
1956 Nbproc: 1
1957 Process_num: 1
1958 Pid: 28105
1959 Uptime: 0d 0h00m04s
1960 Uptime_sec: 4
1961 Memmax_MB: 0
1962 PoolAlloc_MB: 0
1963 PoolUsed_MB: 0
1964 PoolFailed: 0
1965 (...)
1966
1967 > show info typed
1968 0.Name.1:POS:str:HAProxy
1969 1.Version.1:POS:str:1.7-dev1-de52ea-146
1970 2.Release_date.1:POS:str:2016/03/11
1971 3.Nbproc.1:CGS:u32:1
1972 4.Process_num.1:KGP:u32:1
1973 5.Pid.1:SGP:u32:28105
1974 6.Uptime.1:MDP:str:0d 0h00m08s
1975 7.Uptime_sec.1:MDP:u32:8
1976 8.Memmax_MB.1:CLP:u32:0
1977 9.PoolAlloc_MB.1:MGP:u32:0
1978 10.PoolUsed_MB.1:MGP:u32:0
1979 11.PoolFailed.1:MCP:u32:0
1980 (...)
1981
Simon Horman1084a362016-11-21 17:00:24 +01001982 In the typed format, the presence of the process ID at the end of the
1983 first column makes it very easy to visually aggregate outputs from
1984 multiple processes.
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001985 Example :
1986
1987 $ ( echo show info typed | socat /var/run/haproxy.sock1 ; \
1988 echo show info typed | socat /var/run/haproxy.sock2 ) | \
1989 sort -t . -k 1,1n -k 2,2 -k 3,3n
1990 0.Name.1:POS:str:HAProxy
1991 0.Name.2:POS:str:HAProxy
1992 1.Version.1:POS:str:1.7-dev1-868ab3-148
1993 1.Version.2:POS:str:1.7-dev1-868ab3-148
1994 2.Release_date.1:POS:str:2016/03/11
1995 2.Release_date.2:POS:str:2016/03/11
1996 3.Nbproc.1:CGS:u32:2
1997 3.Nbproc.2:CGS:u32:2
1998 4.Process_num.1:KGP:u32:1
1999 4.Process_num.2:KGP:u32:2
2000 5.Pid.1:SGP:u32:30120
2001 5.Pid.2:SGP:u32:30121
2002 6.Uptime.1:MDP:str:0d 0h01m28s
2003 6.Uptime.2:MDP:str:0d 0h01m28s
2004 (...)
Willy Tarreau44aed902015-10-13 14:45:29 +02002005
Simon Horman05ee2132017-01-04 09:37:25 +01002006 The format of JSON output is described in a schema which may be output
Simon Horman6f6bb382017-01-04 09:37:26 +01002007 using "show schema json".
Simon Horman05ee2132017-01-04 09:37:25 +01002008
2009 The JSON output contains no extra whitespace in order to reduce the
2010 volume of output. For human consumption passing the output through a
2011 pretty printer may be helpful. Example :
2012
2013 $ echo "show info json" | socat /var/run/haproxy.sock stdio | \
2014 python -m json.tool
2015
Simon Horman6f6bb382017-01-04 09:37:26 +01002016 The JSON output contains no extra whitespace in order to reduce the
2017 volume of output. For human consumption passing the output through a
2018 pretty printer may be helpful. Example :
2019
2020 $ echo "show info json" | socat /var/run/haproxy.sock stdio | \
2021 python -m json.tool
2022
Willy Tarreau44aed902015-10-13 14:45:29 +02002023show map [<map>]
2024 Dump info about map converters. Without argument, the list of all available
2025 maps is returned. If a <map> is specified, its contents are dumped. <map> is
2026 the #<id> or <file>. The first column is a unique identifier. It can be used
2027 as reference for the operation "del map" and "set map". The second column is
2028 the pattern and the third column is the sample if available. The data returned
2029 are not directly a list of available maps, but are the list of all patterns
2030 composing any map. Many of these patterns can be shared with ACL.
2031
Willy Tarreau44aed902015-10-13 14:45:29 +02002032show pools
2033 Dump the status of internal memory pools. This is useful to track memory
2034 usage when suspecting a memory leak for example. It does exactly the same
2035 as the SIGQUIT when running in foreground except that it does not flush
2036 the pools.
2037
2038show servers state [<backend>]
2039 Dump the state of the servers found in the running configuration. A backend
2040 name or identifier may be provided to limit the output to this backend only.
2041
2042 The dump has the following format:
2043 - first line contains the format version (1 in this specification);
2044 - second line contains the column headers, prefixed by a sharp ('#');
2045 - third line and next ones contain data;
2046 - each line starting by a sharp ('#') is considered as a comment.
2047
Dan Lloyd8e48b872016-07-01 21:01:18 -04002048 Since multiple versions of the output may co-exist, below is the list of
Willy Tarreau44aed902015-10-13 14:45:29 +02002049 fields and their order per file format version :
2050 1:
2051 be_id: Backend unique id.
2052 be_name: Backend label.
2053 srv_id: Server unique id (in the backend).
2054 srv_name: Server label.
2055 srv_addr: Server IP address.
2056 srv_op_state: Server operational state (UP/DOWN/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002057 0 = SRV_ST_STOPPED
2058 The server is down.
2059 1 = SRV_ST_STARTING
2060 The server is warming up (up but
2061 throttled).
2062 2 = SRV_ST_RUNNING
2063 The server is fully up.
2064 3 = SRV_ST_STOPPING
2065 The server is up but soft-stopping
2066 (eg: 404).
Willy Tarreau44aed902015-10-13 14:45:29 +02002067 srv_admin_state: Server administrative state (MAINT/DRAIN/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002068 The state is actually a mask of values :
2069 0x01 = SRV_ADMF_FMAINT
2070 The server was explicitly forced into
2071 maintenance.
2072 0x02 = SRV_ADMF_IMAINT
2073 The server has inherited the maintenance
2074 status from a tracked server.
2075 0x04 = SRV_ADMF_CMAINT
2076 The server is in maintenance because of
2077 the configuration.
2078 0x08 = SRV_ADMF_FDRAIN
2079 The server was explicitly forced into
2080 drain state.
2081 0x10 = SRV_ADMF_IDRAIN
2082 The server has inherited the drain status
2083 from a tracked server.
Baptiste Assmann89aa7f32016-11-02 21:31:27 +01002084 0x20 = SRV_ADMF_RMAINT
2085 The server is in maintenance because of an
2086 IP address resolution failure.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02002087 0x40 = SRV_ADMF_HMAINT
2088 The server FQDN was set from stats socket.
2089
Willy Tarreau44aed902015-10-13 14:45:29 +02002090 srv_uweight: User visible server's weight.
2091 srv_iweight: Server's initial weight.
2092 srv_time_since_last_change: Time since last operational change.
2093 srv_check_status: Last health check status.
2094 srv_check_result: Last check result (FAILED/PASSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002095 0 = CHK_RES_UNKNOWN
2096 Initialized to this by default.
2097 1 = CHK_RES_NEUTRAL
2098 Valid check but no status information.
2099 2 = CHK_RES_FAILED
2100 Check failed.
2101 3 = CHK_RES_PASSED
2102 Check succeeded and server is fully up
2103 again.
2104 4 = CHK_RES_CONDPASS
2105 Check reports the server doesn't want new
2106 sessions.
Willy Tarreau44aed902015-10-13 14:45:29 +02002107 srv_check_health: Checks rise / fall current counter.
2108 srv_check_state: State of the check (ENABLED/PAUSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002109 The state is actually a mask of values :
2110 0x01 = CHK_ST_INPROGRESS
2111 A check is currently running.
2112 0x02 = CHK_ST_CONFIGURED
2113 This check is configured and may be
2114 enabled.
2115 0x04 = CHK_ST_ENABLED
2116 This check is currently administratively
2117 enabled.
2118 0x08 = CHK_ST_PAUSED
2119 Checks are paused because of maintenance
2120 (health only).
Willy Tarreau44aed902015-10-13 14:45:29 +02002121 srv_agent_state: State of the agent check (ENABLED/PAUSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002122 This state uses the same mask values as
2123 "srv_check_state", adding this specific one :
2124 0x10 = CHK_ST_AGENT
2125 Check is an agent check (otherwise it's a
2126 health check).
Willy Tarreau44aed902015-10-13 14:45:29 +02002127 bk_f_forced_id: Flag to know if the backend ID is forced by
2128 configuration.
2129 srv_f_forced_id: Flag to know if the server's ID is forced by
2130 configuration.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02002131 srv_fqdn: Server FQDN.
Frédéric Lécaille31694712017-08-01 08:47:19 +02002132 srv_port: Server port.
Willy Tarreau44aed902015-10-13 14:45:29 +02002133
2134show sess
2135 Dump all known sessions. Avoid doing this on slow connections as this can
2136 be huge. This command is restricted and can only be issued on sockets
2137 configured for levels "operator" or "admin".
2138
2139show sess <id>
2140 Display a lot of internal information about the specified session identifier.
2141 This identifier is the first field at the beginning of the lines in the dumps
2142 of "show sess" (it corresponds to the session pointer). Those information are
2143 useless to most users but may be used by haproxy developers to troubleshoot a
2144 complex bug. The output format is intentionally not documented so that it can
2145 freely evolve depending on demands. You may find a description of all fields
2146 returned in src/dumpstats.c
2147
2148 The special id "all" dumps the states of all sessions, which must be avoided
2149 as much as possible as it is highly CPU intensive and can take a lot of time.
2150
Simon Horman05ee2132017-01-04 09:37:25 +01002151show stat [{<iid>|<proxy>} <type> <sid>] [typed|json]
2152 Dump statistics using the CSV format; using the extended typed output
2153 format described in the section above if "typed" is passed after the
2154 other arguments; or in JSON if "json" is passed after the other arguments
2155 . By passing <id>, <type> and <sid>, it is possible to dump only selected
2156 items :
Willy Tarreaua1b1ed52016-11-25 08:50:58 +01002157 - <iid> is a proxy ID, -1 to dump everything. Alternatively, a proxy name
2158 <proxy> may be specified. In this case, this proxy's ID will be used as
2159 the ID selector.
Willy Tarreau44aed902015-10-13 14:45:29 +02002160 - <type> selects the type of dumpable objects : 1 for frontends, 2 for
2161 backends, 4 for servers, -1 for everything. These values can be ORed,
2162 for example:
2163 1 + 2 = 3 -> frontend + backend.
2164 1 + 2 + 4 = 7 -> frontend + backend + server.
2165 - <sid> is a server ID, -1 to dump everything from the selected proxy.
2166
2167 Example :
2168 $ echo "show info;show stat" | socat stdio unix-connect:/tmp/sock1
2169 >>> Name: HAProxy
2170 Version: 1.4-dev2-49
2171 Release_date: 2009/09/23
2172 Nbproc: 1
2173 Process_num: 1
2174 (...)
2175
2176 # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq, (...)
2177 stats,FRONTEND,,,0,0,1000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,1,0, (...)
2178 stats,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,250,(...)
2179 (...)
2180 www1,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,0,250, (...)
2181
2182 $
2183
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002184 In this example, two commands have been issued at once. That way it's easy to
2185 find which process the stats apply to in multi-process mode. This is not
2186 needed in the typed output format as the process number is reported on each
2187 line. Notice the empty line after the information output which marks the end
2188 of the first block. A similar empty line appears at the end of the second
2189 block (stats) so that the reader knows the output has not been truncated.
2190
2191 When "typed" is specified, the output format is more suitable to monitoring
2192 tools because it provides numeric positions and indicates the type of each
2193 output field. Each value stands on its own line with process number, element
2194 number, nature, origin and scope. This same format is available via the HTTP
2195 stats by passing ";typed" after the URI. It is very important to note that in
Dan Lloyd8e48b872016-07-01 21:01:18 -04002196 typed output format, the dump for a single object is contiguous so that there
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002197 is no need for a consumer to store everything at once.
2198
2199 When using the typed output format, each line is made of 4 columns delimited
2200 by colons (':'). The first column is a dot-delimited series of 5 elements. The
2201 first element is a letter indicating the type of the object being described.
2202 At the moment the following object types are known : 'F' for a frontend, 'B'
2203 for a backend, 'L' for a listener, and 'S' for a server. The second element
2204 The second element is a positive integer representing the unique identifier of
2205 the proxy the object belongs to. It is equivalent to the "iid" column of the
2206 CSV output and matches the value in front of the optional "id" directive found
2207 in the frontend or backend section. The third element is a positive integer
2208 containing the unique object identifier inside the proxy, and corresponds to
2209 the "sid" column of the CSV output. ID 0 is reported when dumping a frontend
2210 or a backend. For a listener or a server, this corresponds to their respective
2211 ID inside the proxy. The fourth element is the numeric position of the field
2212 in the list (starting at zero). This position shall not change over time, but
2213 holes are to be expected, depending on build options or if some fields are
2214 deleted in the future. The fifth element is the field name as it appears in
2215 the CSV output. The sixth element is a positive integer and is the relative
2216 process number starting at 1.
2217
2218 The rest of the line starting after the first colon follows the "typed output
2219 format" described in the section above. In short, the second column (after the
2220 first ':') indicates the origin, nature and scope of the variable. The third
2221 column indicates the type of the field, among "s32", "s64", "u32", "u64" and
2222 "str". Then the fourth column is the value itself, which the consumer knows
2223 how to parse thanks to column 3 and how to process thanks to column 2.
2224
2225 Thus the overall line format in typed mode is :
2226
2227 <obj>.<px_id>.<id>.<fpos>.<fname>.<process_num>:<tags>:<type>:<value>
2228
2229 Here's an example of typed output format :
2230
2231 $ echo "show stat typed" | socat stdio unix-connect:/tmp/sock1
2232 F.2.0.0.pxname.1:MGP:str:private-frontend
2233 F.2.0.1.svname.1:MGP:str:FRONTEND
2234 F.2.0.8.bin.1:MGP:u64:0
2235 F.2.0.9.bout.1:MGP:u64:0
2236 F.2.0.40.hrsp_2xx.1:MGP:u64:0
2237 L.2.1.0.pxname.1:MGP:str:private-frontend
2238 L.2.1.1.svname.1:MGP:str:sock-1
2239 L.2.1.17.status.1:MGP:str:OPEN
2240 L.2.1.73.addr.1:MGP:str:0.0.0.0:8001
2241 S.3.13.60.rtime.1:MCP:u32:0
2242 S.3.13.61.ttime.1:MCP:u32:0
2243 S.3.13.62.agent_status.1:MGP:str:L4TOUT
2244 S.3.13.64.agent_duration.1:MGP:u64:2001
2245 S.3.13.65.check_desc.1:MCP:str:Layer4 timeout
2246 S.3.13.66.agent_desc.1:MCP:str:Layer4 timeout
2247 S.3.13.67.check_rise.1:MCP:u32:2
2248 S.3.13.68.check_fall.1:MCP:u32:3
2249 S.3.13.69.check_health.1:SGP:u32:0
2250 S.3.13.70.agent_rise.1:MaP:u32:1
2251 S.3.13.71.agent_fall.1:SGP:u32:1
2252 S.3.13.72.agent_health.1:SGP:u32:1
2253 S.3.13.73.addr.1:MCP:str:1.255.255.255:8888
2254 S.3.13.75.mode.1:MAP:str:http
2255 B.3.0.0.pxname.1:MGP:str:private-backend
2256 B.3.0.1.svname.1:MGP:str:BACKEND
2257 B.3.0.2.qcur.1:MGP:u32:0
2258 B.3.0.3.qmax.1:MGP:u32:0
2259 B.3.0.4.scur.1:MGP:u32:0
2260 B.3.0.5.smax.1:MGP:u32:0
2261 B.3.0.6.slim.1:MGP:u32:1000
2262 B.3.0.55.lastsess.1:MMP:s32:-1
2263 (...)
2264
Simon Horman1084a362016-11-21 17:00:24 +01002265 In the typed format, the presence of the process ID at the end of the
2266 first column makes it very easy to visually aggregate outputs from
2267 multiple processes, as show in the example below where each line appears
2268 for each process :
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002269
2270 $ ( echo show stat typed | socat /var/run/haproxy.sock1 - ; \
2271 echo show stat typed | socat /var/run/haproxy.sock2 - ) | \
2272 sort -t . -k 1,1 -k 2,2n -k 3,3n -k 4,4n -k 5,5 -k 6,6n
2273 B.3.0.0.pxname.1:MGP:str:private-backend
2274 B.3.0.0.pxname.2:MGP:str:private-backend
2275 B.3.0.1.svname.1:MGP:str:BACKEND
2276 B.3.0.1.svname.2:MGP:str:BACKEND
2277 B.3.0.2.qcur.1:MGP:u32:0
2278 B.3.0.2.qcur.2:MGP:u32:0
2279 B.3.0.3.qmax.1:MGP:u32:0
2280 B.3.0.3.qmax.2:MGP:u32:0
2281 B.3.0.4.scur.1:MGP:u32:0
2282 B.3.0.4.scur.2:MGP:u32:0
2283 B.3.0.5.smax.1:MGP:u32:0
2284 B.3.0.5.smax.2:MGP:u32:0
2285 B.3.0.6.slim.1:MGP:u32:1000
2286 B.3.0.6.slim.2:MGP:u32:1000
2287 (...)
Willy Tarreau44aed902015-10-13 14:45:29 +02002288
Simon Horman05ee2132017-01-04 09:37:25 +01002289 The format of JSON output is described in a schema which may be output
Simon Horman6f6bb382017-01-04 09:37:26 +01002290 using "show schema json".
2291
2292 The JSON output contains no extra whitespace in order to reduce the
2293 volume of output. For human consumption passing the output through a
2294 pretty printer may be helpful. Example :
2295
2296 $ echo "show stat json" | socat /var/run/haproxy.sock stdio | \
2297 python -m json.tool
Simon Horman05ee2132017-01-04 09:37:25 +01002298
2299 The JSON output contains no extra whitespace in order to reduce the
2300 volume of output. For human consumption passing the output through a
2301 pretty printer may be helpful. Example :
2302
2303 $ echo "show stat json" | socat /var/run/haproxy.sock stdio | \
2304 python -m json.tool
2305
Willy Tarreau44aed902015-10-13 14:45:29 +02002306show stat resolvers [<resolvers section id>]
2307 Dump statistics for the given resolvers section, or all resolvers sections
2308 if no section is supplied.
2309
2310 For each name server, the following counters are reported:
2311 sent: number of DNS requests sent to this server
2312 valid: number of DNS valid responses received from this server
2313 update: number of DNS responses used to update the server's IP address
2314 cname: number of CNAME responses
2315 cname_error: CNAME errors encountered with this server
2316 any_err: number of empty response (IE: server does not support ANY type)
2317 nx: non existent domain response received from this server
2318 timeout: how many time this server did not answer in time
2319 refused: number of requests refused by this server
2320 other: any other DNS errors
2321 invalid: invalid DNS response (from a protocol point of view)
2322 too_big: too big response
2323 outdated: number of response arrived too late (after an other name server)
2324
2325show table
2326 Dump general information on all known stick-tables. Their name is returned
2327 (the name of the proxy which holds them), their type (currently zero, always
2328 IP), their size in maximum possible number of entries, and the number of
2329 entries currently in use.
2330
2331 Example :
2332 $ echo "show table" | socat stdio /tmp/sock1
2333 >>> # table: front_pub, type: ip, size:204800, used:171454
2334 >>> # table: back_rdp, type: ip, size:204800, used:0
2335
2336show table <name> [ data.<type> <operator> <value> ] | [ key <key> ]
2337 Dump contents of stick-table <name>. In this mode, a first line of generic
2338 information about the table is reported as with "show table", then all
2339 entries are dumped. Since this can be quite heavy, it is possible to specify
2340 a filter in order to specify what entries to display.
2341
2342 When the "data." form is used the filter applies to the stored data (see
2343 "stick-table" in section 4.2). A stored data type must be specified
2344 in <type>, and this data type must be stored in the table otherwise an
2345 error is reported. The data is compared according to <operator> with the
2346 64-bit integer <value>. Operators are the same as with the ACLs :
2347
2348 - eq : match entries whose data is equal to this value
2349 - ne : match entries whose data is not equal to this value
2350 - le : match entries whose data is less than or equal to this value
2351 - ge : match entries whose data is greater than or equal to this value
2352 - lt : match entries whose data is less than this value
2353 - gt : match entries whose data is greater than this value
2354
2355
2356 When the key form is used the entry <key> is shown. The key must be of the
2357 same type as the table, which currently is limited to IPv4, IPv6, integer,
2358 and string.
2359
2360 Example :
2361 $ echo "show table http_proxy" | socat stdio /tmp/sock1
2362 >>> # table: http_proxy, type: ip, size:204800, used:2
2363 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
2364 bytes_out_rate(60000)=187
2365 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2366 bytes_out_rate(60000)=191
2367
2368 $ echo "show table http_proxy data.gpc0 gt 0" | socat stdio /tmp/sock1
2369 >>> # table: http_proxy, type: ip, size:204800, used:2
2370 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2371 bytes_out_rate(60000)=191
2372
2373 $ echo "show table http_proxy data.conn_rate gt 5" | \
2374 socat stdio /tmp/sock1
2375 >>> # table: http_proxy, type: ip, size:204800, used:2
2376 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2377 bytes_out_rate(60000)=191
2378
2379 $ echo "show table http_proxy key 127.0.0.2" | \
2380 socat stdio /tmp/sock1
2381 >>> # table: http_proxy, type: ip, size:204800, used:2
2382 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2383 bytes_out_rate(60000)=191
2384
2385 When the data criterion applies to a dynamic value dependent on time such as
2386 a bytes rate, the value is dynamically computed during the evaluation of the
2387 entry in order to decide whether it has to be dumped or not. This means that
2388 such a filter could match for some time then not match anymore because as
2389 time goes, the average event rate drops.
2390
2391 It is possible to use this to extract lists of IP addresses abusing the
2392 service, in order to monitor them or even blacklist them in a firewall.
2393 Example :
2394 $ echo "show table http_proxy data.gpc0 gt 0" \
2395 | socat stdio /tmp/sock1 \
2396 | fgrep 'key=' | cut -d' ' -f2 | cut -d= -f2 > abusers-ip.txt
2397 ( or | awk '/key/{ print a[split($2,a,"=")]; }' )
2398
William Lallemandbb933462016-05-31 21:09:53 +02002399show tls-keys [id|*]
2400 Dump all loaded TLS ticket keys references. The TLS ticket key reference ID
2401 and the file from which the keys have been loaded is shown. Both of those
2402 can be used to update the TLS keys using "set ssl tls-key". If an ID is
2403 specified as parameter, it will dump the tickets, using * it will dump every
2404 keys from every references.
Willy Tarreau44aed902015-10-13 14:45:29 +02002405
Simon Horman6f6bb382017-01-04 09:37:26 +01002406show schema json
2407 Dump the schema used for the output of "show info json" and "show stat json".
2408
2409 The contains no extra whitespace in order to reduce the volume of output.
2410 For human consumption passing the output through a pretty printer may be
2411 helpful. Example :
2412
2413 $ echo "show schema json" | socat /var/run/haproxy.sock stdio | \
2414 python -m json.tool
2415
2416 The schema follows "JSON Schema" (json-schema.org) and accordingly
2417 verifiers may be used to verify the output of "show info json" and "show
2418 stat json" against the schema.
2419
2420
Willy Tarreau44aed902015-10-13 14:45:29 +02002421shutdown frontend <frontend>
2422 Completely delete the specified frontend. All the ports it was bound to will
2423 be released. It will not be possible to enable the frontend anymore after
2424 this operation. This is intended to be used in environments where stopping a
2425 proxy is not even imaginable but a misconfigured proxy must be fixed. That
2426 way it's possible to release the port and bind it into another process to
2427 restore operations. The frontend will not appear at all on the stats page
2428 once it is terminated.
2429
2430 The frontend may be specified either by its name or by its numeric ID,
2431 prefixed with a sharp ('#').
2432
2433 This command is restricted and can only be issued on sockets configured for
2434 level "admin".
2435
2436shutdown session <id>
2437 Immediately terminate the session matching the specified session identifier.
2438 This identifier is the first field at the beginning of the lines in the dumps
2439 of "show sess" (it corresponds to the session pointer). This can be used to
2440 terminate a long-running session without waiting for a timeout or when an
2441 endless transfer is ongoing. Such terminated sessions are reported with a 'K'
2442 flag in the logs.
2443
2444shutdown sessions server <backend>/<server>
2445 Immediately terminate all the sessions attached to the specified server. This
2446 can be used to terminate long-running sessions after a server is put into
2447 maintenance mode, for instance. Such terminated sessions are reported with a
2448 'K' flag in the logs.
2449
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002450
245110. Tricks for easier configuration management
2452----------------------------------------------
2453
2454It is very common that two HAProxy nodes constituting a cluster share exactly
2455the same configuration modulo a few addresses. Instead of having to maintain a
2456duplicate configuration for each node, which will inevitably diverge, it is
2457possible to include environment variables in the configuration. Thus multiple
2458configuration may share the exact same file with only a few different system
2459wide environment variables. This started in version 1.5 where only addresses
2460were allowed to include environment variables, and 1.6 goes further by
2461supporting environment variables everywhere. The syntax is the same as in the
2462UNIX shell, a variable starts with a dollar sign ('$'), followed by an opening
2463curly brace ('{'), then the variable name followed by the closing brace ('}').
2464Except for addresses, environment variables are only interpreted in arguments
2465surrounded with double quotes (this was necessary not to break existing setups
2466using regular expressions involving the dollar symbol).
2467
2468Environment variables also make it convenient to write configurations which are
2469expected to work on various sites where only the address changes. It can also
2470permit to remove passwords from some configs. Example below where the the file
2471"site1.env" file is sourced by the init script upon startup :
2472
2473 $ cat site1.env
2474 LISTEN=192.168.1.1
2475 CACHE_PFX=192.168.11
2476 SERVER_PFX=192.168.22
2477 LOGGER=192.168.33.1
2478 STATSLP=admin:pa$$w0rd
2479 ABUSERS=/etc/haproxy/abuse.lst
2480 TIMEOUT=10s
2481
2482 $ cat haproxy.cfg
2483 global
2484 log "${LOGGER}:514" local0
2485
2486 defaults
2487 mode http
2488 timeout client "${TIMEOUT}"
2489 timeout server "${TIMEOUT}"
2490 timeout connect 5s
2491
2492 frontend public
2493 bind "${LISTEN}:80"
2494 http-request reject if { src -f "${ABUSERS}" }
2495 stats uri /stats
2496 stats auth "${STATSLP}"
2497 use_backend cache if { path_end .jpg .css .ico }
2498 default_backend server
2499
2500 backend cache
2501 server cache1 "${CACHE_PFX}.1:18080" check
2502 server cache2 "${CACHE_PFX}.2:18080" check
2503
2504 backend server
2505 server cache1 "${SERVER_PFX}.1:8080" check
2506 server cache2 "${SERVER_PFX}.2:8080" check
2507
2508
250911. Well-known traps to avoid
2510-----------------------------
2511
2512Once in a while, someone reports that after a system reboot, the haproxy
2513service wasn't started, and that once they start it by hand it works. Most
2514often, these people are running a clustered IP address mechanism such as
2515keepalived, to assign the service IP address to the master node only, and while
2516it used to work when they used to bind haproxy to address 0.0.0.0, it stopped
2517working after they bound it to the virtual IP address. What happens here is
2518that when the service starts, the virtual IP address is not yet owned by the
2519local node, so when HAProxy wants to bind to it, the system rejects this
2520because it is not a local IP address. The fix doesn't consist in delaying the
2521haproxy service startup (since it wouldn't stand a restart), but instead to
2522properly configure the system to allow binding to non-local addresses. This is
2523easily done on Linux by setting the net.ipv4.ip_nonlocal_bind sysctl to 1. This
2524is also needed in order to transparently intercept the IP traffic that passes
2525through HAProxy for a specific target address.
2526
2527Multi-process configurations involving source port ranges may apparently seem
2528to work but they will cause some random failures under high loads because more
2529than one process may try to use the same source port to connect to the same
2530server, which is not possible. The system will report an error and a retry will
2531happen, picking another port. A high value in the "retries" parameter may hide
2532the effect to a certain extent but this also comes with increased CPU usage and
2533processing time. Logs will also report a certain number of retries. For this
2534reason, port ranges should be avoided in multi-process configurations.
2535
Dan Lloyd8e48b872016-07-01 21:01:18 -04002536Since HAProxy uses SO_REUSEPORT and supports having multiple independent
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002537processes bound to the same IP:port, during troubleshooting it can happen that
2538an old process was not stopped before a new one was started. This provides
2539absurd test results which tend to indicate that any change to the configuration
2540is ignored. The reason is that in fact even the new process is restarted with a
2541new configuration, the old one also gets some incoming connections and
2542processes them, returning unexpected results. When in doubt, just stop the new
2543process and try again. If it still works, it very likely means that an old
2544process remains alive and has to be stopped. Linux's "netstat -lntp" is of good
2545help here.
2546
2547When adding entries to an ACL from the command line (eg: when blacklisting a
2548source address), it is important to keep in mind that these entries are not
2549synchronized to the file and that if someone reloads the configuration, these
2550updates will be lost. While this is often the desired effect (for blacklisting)
2551it may not necessarily match expectations when the change was made as a fix for
2552a problem. See the "add acl" action of the CLI interface.
2553
2554
255512. Debugging and performance issues
2556------------------------------------
2557
2558When HAProxy is started with the "-d" option, it will stay in the foreground
2559and will print one line per event, such as an incoming connection, the end of a
2560connection, and for each request or response header line seen. This debug
2561output is emitted before the contents are processed, so they don't consider the
2562local modifications. The main use is to show the request and response without
2563having to run a network sniffer. The output is less readable when multiple
2564connections are handled in parallel, though the "debug2ansi" and "debug2html"
2565scripts found in the examples/ directory definitely help here by coloring the
2566output.
2567
2568If a request or response is rejected because HAProxy finds it is malformed, the
2569best thing to do is to connect to the CLI and issue "show errors", which will
2570report the last captured faulty request and response for each frontend and
2571backend, with all the necessary information to indicate precisely the first
2572character of the input stream that was rejected. This is sometimes needed to
2573prove to customers or to developers that a bug is present in their code. In
2574this case it is often possible to relax the checks (but still keep the
2575captures) using "option accept-invalid-http-request" or its equivalent for
2576responses coming from the server "option accept-invalid-http-response". Please
2577see the configuration manual for more details.
2578
2579Example :
2580
2581 > show errors
2582 Total events captured on [13/Oct/2015:13:43:47.169] : 1
2583
2584 [13/Oct/2015:13:43:40.918] frontend HAProxyLocalStats (#2): invalid request
2585 backend <NONE> (#-1), server <NONE> (#-1), event #0
2586 src 127.0.0.1:51981, session #0, session flags 0x00000080
2587 HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000
2588 HTTP chunk len 0 bytes, HTTP body len 0 bytes
2589 buffer flags 0x00808002, out 0 bytes, total 31 bytes
2590 pending 31 bytes, wrapping at 8040, error at position 13:
2591
2592 00000 GET /invalid request HTTP/1.1\r\n
2593
2594
2595The output of "show info" on the CLI provides a number of useful information
2596regarding the maximum connection rate ever reached, maximum SSL key rate ever
2597reached, and in general all information which can help to explain temporary
2598issues regarding CPU or memory usage. Example :
2599
2600 > show info
2601 Name: HAProxy
2602 Version: 1.6-dev7-e32d18-17
2603 Release_date: 2015/10/12
2604 Nbproc: 1
2605 Process_num: 1
2606 Pid: 7949
2607 Uptime: 0d 0h02m39s
2608 Uptime_sec: 159
2609 Memmax_MB: 0
2610 Ulimit-n: 120032
2611 Maxsock: 120032
2612 Maxconn: 60000
2613 Hard_maxconn: 60000
2614 CurrConns: 0
2615 CumConns: 3
2616 CumReq: 3
2617 MaxSslConns: 0
2618 CurrSslConns: 0
2619 CumSslConns: 0
2620 Maxpipes: 0
2621 PipesUsed: 0
2622 PipesFree: 0
2623 ConnRate: 0
2624 ConnRateLimit: 0
2625 MaxConnRate: 1
2626 SessRate: 0
2627 SessRateLimit: 0
2628 MaxSessRate: 1
2629 SslRate: 0
2630 SslRateLimit: 0
2631 MaxSslRate: 0
2632 SslFrontendKeyRate: 0
2633 SslFrontendMaxKeyRate: 0
2634 SslFrontendSessionReuse_pct: 0
2635 SslBackendKeyRate: 0
2636 SslBackendMaxKeyRate: 0
2637 SslCacheLookups: 0
2638 SslCacheMisses: 0
2639 CompressBpsIn: 0
2640 CompressBpsOut: 0
2641 CompressBpsRateLim: 0
2642 ZlibMemUsage: 0
2643 MaxZlibMemUsage: 0
2644 Tasks: 5
2645 Run_queue: 1
2646 Idle_pct: 100
2647 node: wtap
2648 description:
2649
2650When an issue seems to randomly appear on a new version of HAProxy (eg: every
2651second request is aborted, occasional crash, etc), it is worth trying to enable
Dan Lloyd8e48b872016-07-01 21:01:18 -04002652memory poisoning so that each call to malloc() is immediately followed by the
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002653filling of the memory area with a configurable byte. By default this byte is
26540x50 (ASCII for 'P'), but any other byte can be used, including zero (which
2655will have the same effect as a calloc() and which may make issues disappear).
Dan Lloyd8e48b872016-07-01 21:01:18 -04002656Memory poisoning is enabled on the command line using the "-dM" option. It
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002657slightly hurts performance and is not recommended for use in production. If
Dan Lloyd8e48b872016-07-01 21:01:18 -04002658an issue happens all the time with it or never happens when poisoning uses
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002659byte zero, it clearly means you've found a bug and you definitely need to
2660report it. Otherwise if there's no clear change, the problem it is not related.
2661
2662When debugging some latency issues, it is important to use both strace and
2663tcpdump on the local machine, and another tcpdump on the remote system. The
2664reason for this is that there are delays everywhere in the processing chain and
2665it is important to know which one is causing latency to know where to act. In
2666practice, the local tcpdump will indicate when the input data come in. Strace
2667will indicate when haproxy receives these data (using recv/recvfrom). Warning,
2668openssl uses read()/write() syscalls instead of recv()/send(). Strace will also
2669show when haproxy sends the data, and tcpdump will show when the system sends
2670these data to the interface. Then the external tcpdump will show when the data
2671sent are really received (since the local one only shows when the packets are
2672queued). The benefit of sniffing on the local system is that strace and tcpdump
2673will use the same reference clock. Strace should be used with "-tts200" to get
2674complete timestamps and report large enough chunks of data to read them.
2675Tcpdump should be used with "-nvvttSs0" to report full packets, real sequence
2676numbers and complete timestamps.
2677
2678In practice, received data are almost always immediately received by haproxy
2679(unless the machine has a saturated CPU or these data are invalid and not
2680delivered). If these data are received but not sent, it generally is because
2681the output buffer is saturated (ie: recipient doesn't consume the data fast
2682enough). This can be confirmed by seeing that the polling doesn't notify of
2683the ability to write on the output file descriptor for some time (it's often
2684easier to spot in the strace output when the data finally leave and then roll
2685back to see when the write event was notified). It generally matches an ACK
2686received from the recipient, and detected by tcpdump. Once the data are sent,
2687they may spend some time in the system doing nothing. Here again, the TCP
2688congestion window may be limited and not allow these data to leave, waiting for
2689an ACK to open the window. If the traffic is idle and the data take 40 ms or
2690200 ms to leave, it's a different issue (which is not an issue), it's the fact
2691that the Nagle algorithm prevents empty packets from leaving immediately, in
2692hope that they will be merged with subsequent data. HAProxy automatically
2693disables Nagle in pure TCP mode and in tunnels. However it definitely remains
2694enabled when forwarding an HTTP body (and this contributes to the performance
2695improvement there by reducing the number of packets). Some HTTP non-compliant
2696applications may be sensitive to the latency when delivering incomplete HTTP
2697response messages. In this case you will have to enable "option http-no-delay"
2698to disable Nagle in order to work around their design, keeping in mind that any
2699other proxy in the chain may similarly be impacted. If tcpdump reports that data
2700leave immediately but the other end doesn't see them quickly, it can mean there
Dan Lloyd8e48b872016-07-01 21:01:18 -04002701is a congested WAN link, a congested LAN with flow control enabled and
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002702preventing the data from leaving, or more commonly that HAProxy is in fact
2703running in a virtual machine and that for whatever reason the hypervisor has
2704decided that the data didn't need to be sent immediately. In virtualized
2705environments, latency issues are almost always caused by the virtualization
2706layer, so in order to save time, it's worth first comparing tcpdump in the VM
2707and on the external components. Any difference has to be credited to the
2708hypervisor and its accompanying drivers.
2709
2710When some TCP SACK segments are seen in tcpdump traces (using -vv), it always
2711means that the side sending them has got the proof of a lost packet. While not
2712seeing them doesn't mean there are no losses, seeing them definitely means the
2713network is lossy. Losses are normal on a network, but at a rate where SACKs are
2714not noticeable at the naked eye. If they appear a lot in the traces, it is
2715worth investigating exactly what happens and where the packets are lost. HTTP
2716doesn't cope well with TCP losses, which introduce huge latencies.
2717
2718The "netstat -i" command will report statistics per interface. An interface
2719where the Rx-Ovr counter grows indicates that the system doesn't have enough
2720resources to receive all incoming packets and that they're lost before being
2721processed by the network driver. Rx-Drp indicates that some received packets
2722were lost in the network stack because the application doesn't process them
2723fast enough. This can happen during some attacks as well. Tx-Drp means that
2724the output queues were full and packets had to be dropped. When using TCP it
Dan Lloyd8e48b872016-07-01 21:01:18 -04002725should be very rare, but will possibly indicate a saturated outgoing link.
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002726
2727
272813. Security considerations
2729---------------------------
2730
2731HAProxy is designed to run with very limited privileges. The standard way to
2732use it is to isolate it into a chroot jail and to drop its privileges to a
2733non-root user without any permissions inside this jail so that if any future
2734vulnerability were to be discovered, its compromise would not affect the rest
2735of the system.
2736
Dan Lloyd8e48b872016-07-01 21:01:18 -04002737In order to perform a chroot, it first needs to be started as a root user. It is
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002738pointless to build hand-made chroots to start the process there, these ones are
2739painful to build, are never properly maintained and always contain way more
2740bugs than the main file-system. And in case of compromise, the intruder can use
2741the purposely built file-system. Unfortunately many administrators confuse
2742"start as root" and "run as root", resulting in the uid change to be done prior
2743to starting haproxy, and reducing the effective security restrictions.
2744
2745HAProxy will need to be started as root in order to :
2746 - adjust the file descriptor limits
2747 - bind to privileged port numbers
2748 - bind to a specific network interface
2749 - transparently listen to a foreign address
2750 - isolate itself inside the chroot jail
2751 - drop to another non-privileged UID
2752
2753HAProxy may require to be run as root in order to :
2754 - bind to an interface for outgoing connections
2755 - bind to privileged source ports for outgoing connections
Dan Lloyd8e48b872016-07-01 21:01:18 -04002756 - transparently bind to a foreign address for outgoing connections
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002757
2758Most users will never need the "run as root" case. But the "start as root"
2759covers most usages.
2760
2761A safe configuration will have :
2762
2763 - a chroot statement pointing to an empty location without any access
2764 permissions. This can be prepared this way on the UNIX command line :
2765
2766 # mkdir /var/empty && chmod 0 /var/empty || echo "Failed"
2767
2768 and referenced like this in the HAProxy configuration's global section :
2769
2770 chroot /var/empty
2771
2772 - both a uid/user and gid/group statements in the global section :
2773
2774 user haproxy
2775 group haproxy
2776
2777 - a stats socket whose mode, uid and gid are set to match the user and/or
2778 group allowed to access the CLI so that nobody may access it :
2779
2780 stats socket /var/run/haproxy.stat uid hatop gid hatop mode 600
2781