blob: 56f119e9e59b6d875d20256e73a346f32f697e7a [file] [log] [blame]
Willy Tarreau2212e6a2015-10-13 14:40:55 +02001 ------------------------
2 HAProxy Management Guide
3 ------------------------
Willy Tarreau1db55792020-11-05 17:20:35 +01004 version 2.4
Willy Tarreau2212e6a2015-10-13 14:40:55 +02005
6
7This document describes how to start, stop, manage, and troubleshoot HAProxy,
8as well as some known limitations and traps to avoid. It does not describe how
9to configure it (for this please read configuration.txt).
10
11Note to documentation contributors :
12 This document is formatted with 80 columns per line, with even number of
13 spaces for indentation and without tabs. Please follow these rules strictly
14 so that it remains easily printable everywhere. If you add sections, please
15 update the summary below for easier searching.
16
17
18Summary
19-------
20
211. Prerequisites
222. Quick reminder about HAProxy's architecture
233. Starting HAProxy
244. Stopping and restarting HAProxy
255. File-descriptor limitations
266. Memory management
277. CPU usage
288. Logging
299. Statistics and monitoring
Willy Tarreau44aed902015-10-13 14:45:29 +0200309.1. CSV format
Willy Tarreau5d8b9792016-03-11 11:09:34 +0100319.2. Typed output format
329.3. Unix Socket commands
William Lallemand142db372018-12-11 18:56:45 +0100339.4. Master CLI
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003410. Tricks for easier configuration management
3511. Well-known traps to avoid
3612. Debugging and performance issues
3713. Security considerations
38
39
401. Prerequisites
41----------------
42
43In this document it is assumed that the reader has sufficient administration
44skills on a UNIX-like operating system, uses the shell on a daily basis and is
45familiar with troubleshooting utilities such as strace and tcpdump.
46
47
482. Quick reminder about HAProxy's architecture
49----------------------------------------------
50
Willy Tarreau3f364482019-02-27 15:01:46 +010051HAProxy is a multi-threaded, event-driven, non-blocking daemon. This means is
Willy Tarreau2212e6a2015-10-13 14:40:55 +020052uses event multiplexing to schedule all of its activities instead of relying on
53the system to schedule between multiple activities. Most of the time it runs as
54a single process, so the output of "ps aux" on a system will report only one
55"haproxy" process, unless a soft reload is in progress and an older process is
56finishing its job in parallel to the new one. It is thus always easy to trace
Willy Tarreau3f364482019-02-27 15:01:46 +010057its activity using the strace utility. In order to scale with the number of
58available processors, by default haproxy will start one worker thread per
59processor it is allowed to run on. Unless explicitly configured differently,
60the incoming traffic is spread over all these threads, all running the same
61event loop. A great care is taken to limit inter-thread dependencies to the
62strict minimum, so as to try to achieve near-linear scalability. This has some
63impacts such as the fact that a given connection is served by a single thread.
64Thus in order to use all available processing capacity, it is needed to have at
65least as many connections as there are threads, which is almost always granted.
Willy Tarreau2212e6a2015-10-13 14:40:55 +020066
67HAProxy is designed to isolate itself into a chroot jail during startup, where
68it cannot perform any file-system access at all. This is also true for the
69libraries it depends on (eg: libc, libssl, etc). The immediate effect is that
70a running process will not be able to reload a configuration file to apply
71changes, instead a new process will be started using the updated configuration
72file. Some other less obvious effects are that some timezone files or resolver
73files the libc might attempt to access at run time will not be found, though
74this should generally not happen as they're not needed after startup. A nice
75consequence of this principle is that the HAProxy process is totally stateless,
76and no cleanup is needed after it's killed, so any killing method that works
77will do the right thing.
78
79HAProxy doesn't write log files, but it relies on the standard syslog protocol
80to send logs to a remote server (which is often located on the same system).
81
82HAProxy uses its internal clock to enforce timeouts, that is derived from the
83system's time but where unexpected drift is corrected. This is done by limiting
84the time spent waiting in poll() for an event, and measuring the time it really
85took. In practice it never waits more than one second. This explains why, when
86running strace over a completely idle process, periodic calls to poll() (or any
87of its variants) surrounded by two gettimeofday() calls are noticed. They are
88normal, completely harmless and so cheap that the load they imply is totally
89undetectable at the system scale, so there's nothing abnormal there. Example :
90
91 16:35:40.002320 gettimeofday({1442759740, 2605}, NULL) = 0
92 16:35:40.002942 epoll_wait(0, {}, 200, 1000) = 0
93 16:35:41.007542 gettimeofday({1442759741, 7641}, NULL) = 0
94 16:35:41.007998 gettimeofday({1442759741, 8114}, NULL) = 0
95 16:35:41.008391 epoll_wait(0, {}, 200, 1000) = 0
96 16:35:42.011313 gettimeofday({1442759742, 11411}, NULL) = 0
97
98HAProxy is a TCP proxy, not a router. It deals with established connections that
99have been validated by the kernel, and not with packets of any form nor with
100sockets in other states (eg: no SYN_RECV nor TIME_WAIT), though their existence
101may prevent it from binding a port. It relies on the system to accept incoming
102connections and to initiate outgoing connections. An immediate effect of this is
103that there is no relation between packets observed on the two sides of a
104forwarded connection, which can be of different size, numbers and even family.
105Since a connection may only be accepted from a socket in LISTEN state, all the
106sockets it is listening to are necessarily visible using the "netstat" utility
107to show listening sockets. Example :
108
109 # netstat -ltnp
110 Active Internet connections (only servers)
111 Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
112 tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1629/sshd
113 tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2847/haproxy
114 tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 2847/haproxy
115
116
1173. Starting HAProxy
118-------------------
119
120HAProxy is started by invoking the "haproxy" program with a number of arguments
121passed on the command line. The actual syntax is :
122
123 $ haproxy [<options>]*
124
125where [<options>]* is any number of options. An option always starts with '-'
126followed by one of more letters, and possibly followed by one or multiple extra
127arguments. Without any option, HAProxy displays the help page with a reminder
128about supported options. Available options may vary slightly based on the
129operating system. A fair number of these options overlap with an equivalent one
130if the "global" section. In this case, the command line always has precedence
131over the configuration file, so that the command line can be used to quickly
132enforce some settings without touching the configuration files. The current
133list of options is :
134
135 -- <cfgfile>* : all the arguments following "--" are paths to configuration
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200136 file/directory to be loaded and processed in the declaration order. It is
137 mostly useful when relying on the shell to load many files that are
138 numerically ordered. See also "-f". The difference between "--" and "-f" is
139 that one "-f" must be placed before each file name, while a single "--" is
140 needed before all file names. Both options can be used together, the
141 command line ordering still applies. When more than one file is specified,
142 each file must start on a section boundary, so the first keyword of each
143 file must be one of "global", "defaults", "peers", "listen", "frontend",
144 "backend", and so on. A file cannot contain just a server list for example.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200145
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200146 -f <cfgfile|cfgdir> : adds <cfgfile> to the list of configuration files to be
147 loaded. If <cfgdir> is a directory, all the files (and only files) it
Dan Lloyd8e48b872016-07-01 21:01:18 -0400148 contains are added in lexical order (using LC_COLLATE=C) to the list of
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200149 configuration files to be loaded ; only files with ".cfg" extension are
150 added, only non hidden files (not prefixed with ".") are added.
151 Configuration files are loaded and processed in their declaration order.
152 This option may be specified multiple times to load multiple files. See
153 also "--". The difference between "--" and "-f" is that one "-f" must be
154 placed before each file name, while a single "--" is needed before all file
155 names. Both options can be used together, the command line ordering still
156 applies. When more than one file is specified, each file must start on a
157 section boundary, so the first keyword of each file must be one of
158 "global", "defaults", "peers", "listen", "frontend", "backend", and so on.
159 A file cannot contain just a server list for example.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200160
161 -C <dir> : changes to directory <dir> before loading configuration
162 files. This is useful when using relative paths. Warning when using
163 wildcards after "--" which are in fact replaced by the shell before
164 starting haproxy.
165
166 -D : start as a daemon. The process detaches from the current terminal after
167 forking, and errors are not reported anymore in the terminal. It is
168 equivalent to the "daemon" keyword in the "global" section of the
169 configuration. It is recommended to always force it in any init script so
170 that a faulty configuration doesn't prevent the system from booting.
171
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200172 -L <name> : change the local peer name to <name>, which defaults to the local
William Lallemanddaf4cd22018-04-17 16:46:13 +0200173 hostname. This is used only with peers replication. You can use the
174 variable $HAPROXY_LOCALPEER in the configuration file to reference the
175 peer name.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200176
177 -N <limit> : sets the default per-proxy maxconn to <limit> instead of the
178 builtin default value (usually 2000). Only useful for debugging.
179
180 -V : enable verbose mode (disables quiet mode). Reverts the effect of "-q" or
181 "quiet".
182
William Lallemande202b1e2017-06-01 17:38:56 +0200183 -W : master-worker mode. It is equivalent to the "master-worker" keyword in
184 the "global" section of the configuration. This mode will launch a "master"
185 which will monitor the "workers". Using this mode, you can reload HAProxy
186 directly by sending a SIGUSR2 signal to the master. The master-worker mode
187 is compatible either with the foreground or daemon mode. It is
188 recommended to use this mode with multiprocess and systemd.
189
Pavlos Parissisf65f2572018-02-07 21:42:16 +0100190 -Ws : master-worker mode with support of `notify` type of systemd service.
191 This option is only available when HAProxy was built with `USE_SYSTEMD`
192 build option enabled.
193
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200194 -c : only performs a check of the configuration files and exits before trying
195 to bind. The exit status is zero if everything is OK, or non-zero if an
Willy Tarreaubebd2122020-04-15 16:06:11 +0200196 error is encountered. Presence of warnings will be reported if any.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200197
198 -d : enable debug mode. This disables daemon mode, forces the process to stay
Willy Tarreauccf42992020-10-09 19:15:03 +0200199 in foreground and to show incoming and outgoing events. It must never be
200 used in an init script.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200201
Amaury Denoyelle7b01a8d2021-03-29 10:29:07 +0200202 -dD : enable diagnostic mode. This mode will output extra warnings about
203 suspicious configuration statements. This will never prevent startup even in
204 "zero-warning" mode nor change the exit status code.
205
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200206 -dG : disable use of getaddrinfo() to resolve host names into addresses. It
207 can be used when suspecting that getaddrinfo() doesn't work as expected.
208 This option was made available because many bogus implementations of
209 getaddrinfo() exist on various systems and cause anomalies that are
210 difficult to troubleshoot.
211
Willy Tarreau0aa9dbe2021-12-28 15:43:11 +0100212 -dL : dumps the list of dynamic shared libraries that are loaded at the end
213 of the config processing. This will generally also include deep dependencies
214 such as anything loaded from Lua code for example, as well as the executable
215 itself. The list is printed in a format that ought to be easy enough to
216 sanitize to directly produce a tarball of all dependencies. Since it doesn't
217 stop the program's startup, it is recommended to only use it in combination
218 with "-c" and "-q" where only the list of loaded objects will be displayed
219 (or nothing in case of error). In addition, keep in mind that when providing
220 such a package to help with a core file analysis, most libraries are in fact
221 symbolic links that need to be dereferenced when creating the archive:
222
223 ./haproxy -W -q -c -dL -f foo.cfg | tar -T - -hzcf archive.tgz
224
Dan Lloyd8e48b872016-07-01 21:01:18 -0400225 -dM[<byte>] : forces memory poisoning, which means that each and every
Willy Tarreaubafbe012017-11-24 17:34:44 +0100226 memory region allocated with malloc() or pool_alloc() will be filled with
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200227 <byte> before being passed to the caller. When <byte> is not specified, it
228 defaults to 0x50 ('P'). While this slightly slows down operations, it is
229 useful to reliably trigger issues resulting from missing initializations in
230 the code that cause random crashes. Note that -dM0 has the effect of
231 turning any malloc() into a calloc(). In any case if a bug appears or
232 disappears when using this option it means there is a bug in haproxy, so
233 please report it.
234
235 -dS : disable use of the splice() system call. It is equivalent to the
236 "global" section's "nosplice" keyword. This may be used when splice() is
237 suspected to behave improperly or to cause performance issues, or when
238 using strace to see the forwarded data (which do not appear when using
239 splice()).
240
241 -dV : disable SSL verify on the server side. It is equivalent to having
242 "ssl-server-verify none" in the "global" section. This is useful when
243 trying to reproduce production issues out of the production
244 environment. Never use this in an init script as it degrades SSL security
245 to the servers.
246
Willy Tarreau3eb10b82020-04-15 16:42:39 +0200247 -dW : if set, haproxy will refuse to start if any warning was emitted while
248 processing the configuration. This helps detect subtle mistakes and keep the
249 configuration clean and portable across versions. It is recommended to set
250 this option in service scripts when configurations are managed by humans,
251 but it is recommended not to use it with generated configurations, which
252 tend to emit more warnings. It may be combined with "-c" to cause warnings
253 in checked configurations to fail. This is equivalent to global option
254 "zero-warning".
255
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200256 -db : disable background mode and multi-process mode. The process remains in
257 foreground. It is mainly used during development or during small tests, as
258 Ctrl-C is enough to stop the process. Never use it in an init script.
259
260 -de : disable the use of the "epoll" poller. It is equivalent to the "global"
261 section's keyword "noepoll". It is mostly useful when suspecting a bug
262 related to this poller. On systems supporting epoll, the fallback will
263 generally be the "poll" poller.
264
265 -dk : disable the use of the "kqueue" poller. It is equivalent to the
266 "global" section's keyword "nokqueue". It is mostly useful when suspecting
267 a bug related to this poller. On systems supporting kqueue, the fallback
268 will generally be the "poll" poller.
269
270 -dp : disable the use of the "poll" poller. It is equivalent to the "global"
271 section's keyword "nopoll". It is mostly useful when suspecting a bug
272 related to this poller. On systems supporting poll, the fallback will
273 generally be the "select" poller, which cannot be disabled and is limited
274 to 1024 file descriptors.
275
Willy Tarreau3eed10e2016-11-07 21:03:16 +0100276 -dr : ignore server address resolution failures. It is very common when
277 validating a configuration out of production not to have access to the same
278 resolvers and to fail on server address resolution, making it difficult to
279 test a configuration. This option simply appends the "none" method to the
280 list of address resolution methods for all servers, ensuring that even if
281 the libc fails to resolve an address, the startup sequence is not
282 interrupted.
283
Willy Tarreau70060452015-12-14 12:46:07 +0100284 -m <limit> : limit the total allocatable memory to <limit> megabytes across
285 all processes. This may cause some connection refusals or some slowdowns
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200286 depending on the amount of memory needed for normal operations. This is
Willy Tarreau70060452015-12-14 12:46:07 +0100287 mostly used to force the processes to work in a constrained resource usage
288 scenario. It is important to note that the memory is not shared between
289 processes, so in a multi-process scenario, this value is first divided by
290 global.nbproc before forking.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200291
292 -n <limit> : limits the per-process connection limit to <limit>. This is
293 equivalent to the global section's keyword "maxconn". It has precedence
294 over this keyword. This may be used to quickly force lower limits to avoid
295 a service outage on systems where resource limits are too low.
296
297 -p <file> : write all processes' pids into <file> during startup. This is
298 equivalent to the "global" section's keyword "pidfile". The file is opened
299 before entering the chroot jail, and after doing the chdir() implied by
300 "-C". Each pid appears on its own line.
301
302 -q : set "quiet" mode. This disables some messages during the configuration
303 parsing and during startup. It can be used in combination with "-c" to
304 just check if a configuration file is valid or not.
305
William Lallemand142db372018-12-11 18:56:45 +0100306 -S <bind>[,bind_options...]: in master-worker mode, bind a master CLI, which
307 allows the access to every processes, running or leaving ones.
308 For security reasons, it is recommended to bind the master CLI to a local
309 UNIX socket. The bind options are the same as the keyword "bind" in
310 the configuration file with words separated by commas instead of spaces.
311
312 Note that this socket can't be used to retrieve the listening sockets from
313 an old process during a seamless reload.
314
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200315 -sf <pid>* : send the "finish" signal (SIGUSR1) to older processes after boot
316 completion to ask them to finish what they are doing and to leave. <pid>
317 is a list of pids to signal (one per argument). The list ends on any
318 option starting with a "-". It is not a problem if the list of pids is
319 empty, so that it can be built on the fly based on the result of a command
320 like "pidof" or "pgrep".
321
322 -st <pid>* : send the "terminate" signal (SIGTERM) to older processes after
323 boot completion to terminate them immediately without finishing what they
324 were doing. <pid> is a list of pids to signal (one per argument). The list
325 is ends on any option starting with a "-". It is not a problem if the list
326 of pids is empty, so that it can be built on the fly based on the result of
327 a command like "pidof" or "pgrep".
328
329 -v : report the version and build date.
330
331 -vv : display the version, build options, libraries versions and usable
332 pollers. This output is systematically requested when filing a bug report.
333
Olivier Houchardd33fc3a2017-04-05 22:50:59 +0200334 -x <unix_socket> : connect to the specified socket and try to retrieve any
335 listening sockets from the old process, and use them instead of trying to
336 bind new ones. This is useful to avoid missing any new connection when
William Lallemandf6975e92017-05-26 17:42:10 +0200337 reloading the configuration on Linux. The capability must be enable on the
338 stats socket using "expose-fd listeners" in your configuration.
Olivier Houchardd33fc3a2017-04-05 22:50:59 +0200339
Dan Lloyd8e48b872016-07-01 21:01:18 -0400340A safe way to start HAProxy from an init file consists in forcing the daemon
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200341mode, storing existing pids to a pid file and using this pid file to notify
342older processes to finish before leaving :
343
344 haproxy -f /etc/haproxy.cfg \
345 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
346
347When the configuration is split into a few specific files (eg: tcp vs http),
348it is recommended to use the "-f" option :
349
350 haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
351 -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
352 -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
353 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
354
355When an unknown number of files is expected, such as customer-specific files,
356it is recommended to assign them a name starting with a fixed-size sequence
357number and to use "--" to load them, possibly after loading some defaults :
358
359 haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
360 -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
361 -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
362 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) \
363 -f /etc/haproxy/default-customers.cfg -- /etc/haproxy/customers/*
364
365Sometimes a failure to start may happen for whatever reason. Then it is
366important to verify if the version of HAProxy you are invoking is the expected
367version and if it supports the features you are expecting (eg: SSL, PCRE,
368compression, Lua, etc). This can be verified using "haproxy -vv". Some
369important information such as certain build options, the target system and
370the versions of the libraries being used are reported there. It is also what
371you will systematically be asked for when posting a bug report :
372
373 $ haproxy -vv
Willy Tarreau58000fe2021-05-09 06:25:16 +0200374 HAProxy version 1.6-dev7-a088d3-4 2015/10/08
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200375 Copyright 2000-2015 Willy Tarreau <willy@haproxy.org>
376
377 Build options :
378 TARGET = linux2628
379 CPU = generic
380 CC = gcc
381 CFLAGS = -pg -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement \
382 -DBUFSIZE=8030 -DMAXREWRITE=1030 -DSO_MARK=36 -DTCP_REPAIR=19
383 OPTIONS = USE_ZLIB=1 USE_DLMALLOC=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
384
385 Default settings :
386 maxconn = 2000, bufsize = 8030, maxrewrite = 1030, maxpollevents = 200
387
388 Encrypted password support via crypt(3): yes
389 Built with zlib version : 1.2.6
390 Compression algorithms supported : identity("identity"), deflate("deflate"), \
391 raw-deflate("deflate"), gzip("gzip")
392 Built with OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
393 Running on OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
394 OpenSSL library supports TLS extensions : yes
395 OpenSSL library supports SNI : yes
396 OpenSSL library supports prefer-server-ciphers : yes
397 Built with PCRE version : 8.12 2011-01-15
398 PCRE library supports JIT : no (USE_PCRE_JIT not set)
399 Built with Lua version : Lua 5.3.1
400 Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND
401
402 Available polling systems :
403 epoll : pref=300, test result OK
404 poll : pref=200, test result OK
405 select : pref=150, test result OK
406 Total: 3 (3 usable), will use epoll.
407
408The relevant information that many non-developer users can verify here are :
409 - the version : 1.6-dev7-a088d3-4 above means the code is currently at commit
410 ID "a088d3" which is the 4th one after after official version "1.6-dev7".
411 Version 1.6-dev7 would show as "1.6-dev7-8c1ad7". What matters here is in
412 fact "1.6-dev7". This is the 7th development version of what will become
413 version 1.6 in the future. A development version not suitable for use in
414 production (unless you know exactly what you are doing). A stable version
415 will show as a 3-numbers version, such as "1.5.14-16f863", indicating the
416 14th level of fix on top of version 1.5. This is a production-ready version.
417
418 - the release date : 2015/10/08. It is represented in the universal
419 year/month/day format. Here this means August 8th, 2015. Given that stable
420 releases are issued every few months (1-2 months at the beginning, sometimes
421 6 months once the product becomes very stable), if you're seeing an old date
422 here, it means you're probably affected by a number of bugs or security
423 issues that have since been fixed and that it might be worth checking on the
424 official site.
425
426 - build options : they are relevant to people who build their packages
427 themselves, they can explain why things are not behaving as expected. For
428 example the development version above was built for Linux 2.6.28 or later,
Dan Lloyd8e48b872016-07-01 21:01:18 -0400429 targeting a generic CPU (no CPU-specific optimizations), and lacks any
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200430 code optimization (-O0) so it will perform poorly in terms of performance.
431
432 - libraries versions : zlib version is reported as found in the library
433 itself. In general zlib is considered a very stable product and upgrades
434 are almost never needed. OpenSSL reports two versions, the version used at
435 build time and the one being used, as found on the system. These ones may
436 differ by the last letter but never by the numbers. The build date is also
437 reported because most OpenSSL bugs are security issues and need to be taken
438 seriously, so this library absolutely needs to be kept up to date. Seeing a
439 4-months old version here is highly suspicious and indeed an update was
440 missed. PCRE provides very fast regular expressions and is highly
441 recommended. Certain of its extensions such as JIT are not present in all
442 versions and still young so some people prefer not to build with them,
Dan Lloyd8e48b872016-07-01 21:01:18 -0400443 which is why the build status is reported as well. Regarding the Lua
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200444 scripting language, HAProxy expects version 5.3 which is very young since
445 it was released a little time before HAProxy 1.6. It is important to check
446 on the Lua web site if some fixes are proposed for this branch.
447
448 - Available polling systems will affect the process's scalability when
449 dealing with more than about one thousand of concurrent connections. These
450 ones are only available when the correct system was indicated in the TARGET
451 variable during the build. The "epoll" mechanism is highly recommended on
452 Linux, and the kqueue mechanism is highly recommended on BSD. Lacking them
453 will result in poll() or even select() being used, causing a high CPU usage
454 when dealing with a lot of connections.
455
456
4574. Stopping and restarting HAProxy
458----------------------------------
459
460HAProxy supports a graceful and a hard stop. The hard stop is simple, when the
461SIGTERM signal is sent to the haproxy process, it immediately quits and all
462established connections are closed. The graceful stop is triggered when the
463SIGUSR1 signal is sent to the haproxy process. It consists in only unbinding
464from listening ports, but continue to process existing connections until they
465close. Once the last connection is closed, the process leaves.
466
467The hard stop method is used for the "stop" or "restart" actions of the service
468management script. The graceful stop is used for the "reload" action which
469tries to seamlessly reload a new configuration in a new process.
470
471Both of these signals may be sent by the new haproxy process itself during a
472reload or restart, so that they are sent at the latest possible moment and only
473if absolutely required. This is what is performed by the "-st" (hard) and "-sf"
474(graceful) options respectively.
475
William Lallemande202b1e2017-06-01 17:38:56 +0200476In master-worker mode, it is not needed to start a new haproxy process in
477order to reload the configuration. The master process reacts to the SIGUSR2
478signal by reexecuting itself with the -sf parameter followed by the PIDs of
479the workers. The master will then parse the configuration file and fork new
480workers.
481
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200482To understand better how these signals are used, it is important to understand
483the whole restart mechanism.
484
485First, an existing haproxy process is running. The administrator uses a system
Jackie Tapia749f74c2020-07-22 18:59:40 -0500486specific command such as "/etc/init.d/haproxy reload" to indicate they want to
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200487take the new configuration file into effect. What happens then is the following.
488First, the service script (/etc/init.d/haproxy or equivalent) will verify that
489the configuration file parses correctly using "haproxy -c". After that it will
490try to start haproxy with this configuration file, using "-st" or "-sf".
491
492Then HAProxy tries to bind to all listening ports. If some fatal errors happen
493(eg: address not present on the system, permission denied), the process quits
494with an error. If a socket binding fails because a port is already in use, then
495the process will first send a SIGTTOU signal to all the pids specified in the
496"-st" or "-sf" pid list. This is what is called the "pause" signal. It instructs
497all existing haproxy processes to temporarily stop listening to their ports so
498that the new process can try to bind again. During this time, the old process
499continues to process existing connections. If the binding still fails (because
500for example a port is shared with another daemon), then the new process sends a
501SIGTTIN signal to the old processes to instruct them to resume operations just
502as if nothing happened. The old processes will then restart listening to the
Jonathon Lacherb6ed0cb2021-08-04 00:29:05 -0500503ports and continue to accept connections. Note that this mechanism is system
Dan Lloyd8e48b872016-07-01 21:01:18 -0400504dependent and some operating systems may not support it in multi-process mode.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200505
506If the new process manages to bind correctly to all ports, then it sends either
507the SIGTERM (hard stop in case of "-st") or the SIGUSR1 (graceful stop in case
508of "-sf") to all processes to notify them that it is now in charge of operations
509and that the old processes will have to leave, either immediately or once they
510have finished their job.
511
512It is important to note that during this timeframe, there are two small windows
513of a few milliseconds each where it is possible that a few connection failures
514will be noticed during high loads. Typically observed failure rates are around
5151 failure during a reload operation every 10000 new connections per second,
516which means that a heavily loaded site running at 30000 new connections per
517second may see about 3 failed connection upon every reload. The two situations
518where this happens are :
519
520 - if the new process fails to bind due to the presence of the old process,
521 it will first have to go through the SIGTTOU+SIGTTIN sequence, which
522 typically lasts about one millisecond for a few tens of frontends, and
523 during which some ports will not be bound to the old process and not yet
524 bound to the new one. HAProxy works around this on systems that support the
525 SO_REUSEPORT socket options, as it allows the new process to bind without
526 first asking the old one to unbind. Most BSD systems have been supporting
527 this almost forever. Linux has been supporting this in version 2.0 and
528 dropped it around 2.2, but some patches were floating around by then. It
529 was reintroduced in kernel 3.9, so if you are observing a connection
Dan Lloyd8e48b872016-07-01 21:01:18 -0400530 failure rate above the one mentioned above, please ensure that your kernel
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200531 is 3.9 or newer, or that relevant patches were backported to your kernel
532 (less likely).
533
534 - when the old processes close the listening ports, the kernel may not always
535 redistribute any pending connection that was remaining in the socket's
536 backlog. Under high loads, a SYN packet may happen just before the socket
537 is closed, and will lead to an RST packet being sent to the client. In some
538 critical environments where even one drop is not acceptable, these ones are
539 sometimes dealt with using firewall rules to block SYN packets during the
540 reload, forcing the client to retransmit. This is totally system-dependent,
541 as some systems might be able to visit other listening queues and avoid
542 this RST. A second case concerns the ACK from the client on a local socket
543 that was in SYN_RECV state just before the close. This ACK will lead to an
544 RST packet while the haproxy process is still not aware of it. This one is
Dan Lloyd8e48b872016-07-01 21:01:18 -0400545 harder to get rid of, though the firewall filtering rules mentioned above
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200546 will work well if applied one second or so before restarting the process.
547
548For the vast majority of users, such drops will never ever happen since they
549don't have enough load to trigger the race conditions. And for most high traffic
550users, the failure rate is still fairly within the noise margin provided that at
551least SO_REUSEPORT is properly supported on their systems.
552
553
5545. File-descriptor limitations
555------------------------------
556
557In order to ensure that all incoming connections will successfully be served,
558HAProxy computes at load time the total number of file descriptors that will be
559needed during the process's life. A regular Unix process is generally granted
5601024 file descriptors by default, and a privileged process can raise this limit
561itself. This is one reason for starting HAProxy as root and letting it adjust
562the limit. The default limit of 1024 file descriptors roughly allow about 500
563concurrent connections to be processed. The computation is based on the global
564maxconn parameter which limits the total number of connections per process, the
565number of listeners, the number of servers which have a health check enabled,
566the agent checks, the peers, the loggers and possibly a few other technical
567requirements. A simple rough estimate of this number consists in simply
568doubling the maxconn value and adding a few tens to get the approximate number
569of file descriptors needed.
570
571Originally HAProxy did not know how to compute this value, and it was necessary
572to pass the value using the "ulimit-n" setting in the global section. This
573explains why even today a lot of configurations are seen with this setting
574present. Unfortunately it was often miscalculated resulting in connection
575failures when approaching maxconn instead of throttling incoming connection
576while waiting for the needed resources. For this reason it is important to
Dan Lloyd8e48b872016-07-01 21:01:18 -0400577remove any vestigial "ulimit-n" setting that can remain from very old versions.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200578
579Raising the number of file descriptors to accept even moderate loads is
580mandatory but comes with some OS-specific adjustments. First, the select()
581polling system is limited to 1024 file descriptors. In fact on Linux it used
582to be capable of handling more but since certain OS ship with excessively
583restrictive SELinux policies forbidding the use of select() with more than
5841024 file descriptors, HAProxy now refuses to start in this case in order to
585avoid any issue at run time. On all supported operating systems, poll() is
586available and will not suffer from this limitation. It is automatically picked
Dan Lloyd8e48b872016-07-01 21:01:18 -0400587so there is nothing to do to get a working configuration. But poll's becomes
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200588very slow when the number of file descriptors increases. While HAProxy does its
589best to limit this performance impact (eg: via the use of the internal file
590descriptor cache and batched processing), a good rule of thumb is that using
591poll() with more than a thousand concurrent connections will use a lot of CPU.
592
593For Linux systems base on kernels 2.6 and above, the epoll() system call will
594be used. It's a much more scalable mechanism relying on callbacks in the kernel
595that guarantee a constant wake up time regardless of the number of registered
596monitored file descriptors. It is automatically used where detected, provided
597that HAProxy had been built for one of the Linux flavors. Its presence and
598support can be verified using "haproxy -vv".
599
600For BSD systems which support it, kqueue() is available as an alternative. It
601is much faster than poll() and even slightly faster than epoll() thanks to its
602batched handling of changes. At least FreeBSD and OpenBSD support it. Just like
603with Linux's epoll(), its support and availability are reported in the output
604of "haproxy -vv".
605
606Having a good poller is one thing, but it is mandatory that the process can
607reach the limits. When HAProxy starts, it immediately sets the new process's
608file descriptor limits and verifies if it succeeds. In case of failure, it
609reports it before forking so that the administrator can see the problem. As
610long as the process is started by as root, there should be no reason for this
611setting to fail. However, it can fail if the process is started by an
612unprivileged user. If there is a compelling reason for *not* starting haproxy
613as root (eg: started by end users, or by a per-application account), then the
614file descriptor limit can be raised by the system administrator for this
615specific user. The effectiveness of the setting can be verified by issuing
616"ulimit -n" from the user's command line. It should reflect the new limit.
617
618Warning: when an unprivileged user's limits are changed in this user's account,
619it is fairly common that these values are only considered when the user logs in
620and not at all in some scripts run at system boot time nor in crontabs. This is
621totally dependent on the operating system, keep in mind to check "ulimit -n"
622before starting haproxy when running this way. The general advice is never to
623start haproxy as an unprivileged user for production purposes. Another good
624reason is that it prevents haproxy from enabling some security protections.
625
626Once it is certain that the system will allow the haproxy process to use the
627requested number of file descriptors, two new system-specific limits may be
628encountered. The first one is the system-wide file descriptor limit, which is
629the total number of file descriptors opened on the system, covering all
630processes. When this limit is reached, accept() or socket() will typically
631return ENFILE. The second one is the per-process hard limit on the number of
632file descriptors, it prevents setrlimit() from being set higher. Both are very
633dependent on the operating system. On Linux, the system limit is set at boot
634based on the amount of memory. It can be changed with the "fs.file-max" sysctl.
635And the per-process hard limit is set to 1048576 by default, but it can be
636changed using the "fs.nr_open" sysctl.
637
638File descriptor limitations may be observed on a running process when they are
639set too low. The strace utility will report that accept() and socket() return
640"-1 EMFILE" when the process's limits have been reached. In this case, simply
641raising the "ulimit-n" value (or removing it) will solve the problem. If these
642system calls return "-1 ENFILE" then it means that the kernel's limits have
643been reached and that something must be done on a system-wide parameter. These
644trouble must absolutely be addressed, as they result in high CPU usage (when
645accept() fails) and failed connections that are generally visible to the user.
646One solution also consists in lowering the global maxconn value to enforce
647serialization, and possibly to disable HTTP keep-alive to force connections
648to be released and reused faster.
649
650
6516. Memory management
652--------------------
653
654HAProxy uses a simple and fast pool-based memory management. Since it relies on
655a small number of different object types, it's much more efficient to pick new
656objects from a pool which already contains objects of the appropriate size than
657to call malloc() for each different size. The pools are organized as a stack or
658LIFO, so that newly allocated objects are taken from recently released objects
659still hot in the CPU caches. Pools of similar sizes are merged together, in
660order to limit memory fragmentation.
661
662By default, since the focus is set on performance, each released object is put
663back into the pool it came from, and allocated objects are never freed since
664they are expected to be reused very soon.
665
666On the CLI, it is possible to check how memory is being used in pools thanks to
667the "show pools" command :
668
669 > show pools
670 Dumping pools usage. Use SIGQUIT to flush them.
Willy Tarreau0a93b642018-10-16 07:58:39 +0200671 - Pool cache_st (16 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccc40=03 [SHARED]
672 - Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 0 failures, 2 users, @0x9ccac0=00 [SHARED]
673 - Pool comp_state (48 bytes) : 3 allocated (144 bytes), 3 used, 0 failures, 5 users, @0x9cccc0=04 [SHARED]
674 - Pool filter (64 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 3 users, @0x9ccbc0=02 [SHARED]
675 - Pool vars (80 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccb40=01 [SHARED]
676 - Pool uniqueid (128 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9cd240=15 [SHARED]
677 - Pool task (144 bytes) : 55 allocated (7920 bytes), 55 used, 0 failures, 1 users, @0x9cd040=11 [SHARED]
678 - Pool session (160 bytes) : 1 allocated (160 bytes), 1 used, 0 failures, 1 users, @0x9cd140=13 [SHARED]
679 - Pool h2s (208 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccec0=08 [SHARED]
680 - Pool h2c (288 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cce40=07 [SHARED]
681 - Pool spoe_ctx (304 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccf40=09 [SHARED]
682 - Pool connection (400 bytes) : 2 allocated (800 bytes), 2 used, 0 failures, 1 users, @0x9cd1c0=14 [SHARED]
683 - Pool hdr_idx (416 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cd340=17 [SHARED]
684 - Pool dns_resolut (480 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccdc0=06 [SHARED]
685 - Pool dns_answer_ (576 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccd40=05 [SHARED]
686 - Pool stream (960 bytes) : 1 allocated (960 bytes), 1 used, 0 failures, 1 users, @0x9cd0c0=12 [SHARED]
687 - Pool requri (1024 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cd2c0=16 [SHARED]
688 - Pool buffer (8030 bytes) : 3 allocated (24090 bytes), 2 used, 0 failures, 1 users, @0x9cd3c0=18 [SHARED]
689 - Pool trash (8062 bytes) : 1 allocated (8062 bytes), 1 used, 0 failures, 1 users, @0x9cd440=19
690 Total: 19 pools, 42296 bytes allocated, 34266 used.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200691
692The pool name is only indicative, it's the name of the first object type using
693this pool. The size in parenthesis is the object size for objects in this pool.
694Object sizes are always rounded up to the closest multiple of 16 bytes. The
695number of objects currently allocated and the equivalent number of bytes is
696reported so that it is easy to know which pool is responsible for the highest
697memory usage. The number of objects currently in use is reported as well in the
698"used" field. The difference between "allocated" and "used" corresponds to the
Willy Tarreau0a93b642018-10-16 07:58:39 +0200699objects that have been freed and are available for immediate use. The address
700at the end of the line is the pool's address, and the following number is the
701pool index when it exists, or is reported as -1 if no index was assigned.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200702
703It is possible to limit the amount of memory allocated per process using the
704"-m" command line option, followed by a number of megabytes. It covers all of
705the process's addressable space, so that includes memory used by some libraries
706as well as the stack, but it is a reliable limit when building a resource
707constrained system. It works the same way as "ulimit -v" on systems which have
708it, or "ulimit -d" for the other ones.
709
710If a memory allocation fails due to the memory limit being reached or because
711the system doesn't have any enough memory, then haproxy will first start to
712free all available objects from all pools before attempting to allocate memory
713again. This mechanism of releasing unused memory can be triggered by sending
714the signal SIGQUIT to the haproxy process. When doing so, the pools state prior
715to the flush will also be reported to stderr when the process runs in
716foreground.
717
718During a reload operation, the process switched to the graceful stop state also
719automatically performs some flushes after releasing any connection so that all
720possible memory is released to save it for the new process.
721
722
7237. CPU usage
724------------
725
726HAProxy normally spends most of its time in the system and a smaller part in
727userland. A finely tuned 3.5 GHz CPU can sustain a rate about 80000 end-to-end
728connection setups and closes per second at 100% CPU on a single core. When one
729core is saturated, typical figures are :
730 - 95% system, 5% user for long TCP connections or large HTTP objects
731 - 85% system and 15% user for short TCP connections or small HTTP objects in
732 close mode
733 - 70% system and 30% user for small HTTP objects in keep-alive mode
734
735The amount of rules processing and regular expressions will increase the user
736land part. The presence of firewall rules, connection tracking, complex routing
737tables in the system will instead increase the system part.
738
739On most systems, the CPU time observed during network transfers can be cut in 4
740parts :
741 - the interrupt part, which concerns all the processing performed upon I/O
742 receipt, before the target process is even known. Typically Rx packets are
743 accounted for in interrupt. On some systems such as Linux where interrupt
744 processing may be deferred to a dedicated thread, it can appear as softirq,
745 and the thread is called ksoftirqd/0 (for CPU 0). The CPU taking care of
746 this load is generally defined by the hardware settings, though in the case
747 of softirq it is often possible to remap the processing to another CPU.
748 This interrupt part will often be perceived as parasitic since it's not
749 associated with any process, but it actually is some processing being done
750 to prepare the work for the process.
751
752 - the system part, which concerns all the processing done using kernel code
753 called from userland. System calls are accounted as system for example. All
754 synchronously delivered Tx packets will be accounted for as system time. If
755 some packets have to be deferred due to queues filling up, they may then be
756 processed in interrupt context later (eg: upon receipt of an ACK opening a
757 TCP window).
758
759 - the user part, which exclusively runs application code in userland. HAProxy
760 runs exclusively in this part, though it makes heavy use of system calls.
761 Rules processing, regular expressions, compression, encryption all add to
762 the user portion of CPU consumption.
763
764 - the idle part, which is what the CPU does when there is nothing to do. For
765 example HAProxy waits for an incoming connection, or waits for some data to
766 leave, meaning the system is waiting for an ACK from the client to push
767 these data.
768
769In practice regarding HAProxy's activity, it is in general reasonably accurate
770(but totally inexact) to consider that interrupt/softirq are caused by Rx
771processing in kernel drivers, that user-land is caused by layer 7 processing
772in HAProxy, and that system time is caused by network processing on the Tx
773path.
774
775Since HAProxy runs around an event loop, it waits for new events using poll()
776(or any alternative) and processes all these events as fast as possible before
777going back to poll() waiting for new events. It measures the time spent waiting
778in poll() compared to the time spent doing processing events. The ratio of
779polling time vs total time is called the "idle" time, it's the amount of time
780spent waiting for something to happen. This ratio is reported in the stats page
781on the "idle" line, or "Idle_pct" on the CLI. When it's close to 100%, it means
782the load is extremely low. When it's close to 0%, it means that there is
783constantly some activity. While it cannot be very accurate on an overloaded
784system due to other processes possibly preempting the CPU from the haproxy
785process, it still provides a good estimate about how HAProxy considers it is
786working : if the load is low and the idle ratio is low as well, it may indicate
787that HAProxy has a lot of work to do, possibly due to very expensive rules that
788have to be processed. Conversely, if HAProxy indicates the idle is close to
789100% while things are slow, it means that it cannot do anything to speed things
790up because it is already waiting for incoming data to process. In the example
791below, haproxy is completely idle :
792
793 $ echo "show info" | socat - /var/run/haproxy.sock | grep ^Idle
794 Idle_pct: 100
795
796When the idle ratio starts to become very low, it is important to tune the
797system and place processes and interrupts correctly to save the most possible
798CPU resources for all tasks. If a firewall is present, it may be worth trying
799to disable it or to tune it to ensure it is not responsible for a large part
800of the performance limitation. It's worth noting that unloading a stateful
801firewall generally reduces both the amount of interrupt/softirq and of system
802usage since such firewalls act both on the Rx and the Tx paths. On Linux,
803unloading the nf_conntrack and ip_conntrack modules will show whether there is
804anything to gain. If so, then the module runs with default settings and you'll
805have to figure how to tune it for better performance. In general this consists
806in considerably increasing the hash table size. On FreeBSD, "pfctl -d" will
807disable the "pf" firewall and its stateful engine at the same time.
808
809If it is observed that a lot of time is spent in interrupt/softirq, it is
810important to ensure that they don't run on the same CPU. Most systems tend to
811pin the tasks on the CPU where they receive the network traffic because for
812certain workloads it improves things. But with heavily network-bound workloads
813it is the opposite as the haproxy process will have to fight against its kernel
814counterpart. Pinning haproxy to one CPU core and the interrupts to another one,
815all sharing the same L3 cache tends to sensibly increase network performance
816because in practice the amount of work for haproxy and the network stack are
817quite close, so they can almost fill an entire CPU each. On Linux this is done
818using taskset (for haproxy) or using cpu-map (from the haproxy config), and the
819interrupts are assigned under /proc/irq. Many network interfaces support
820multiple queues and multiple interrupts. In general it helps to spread them
821across a small number of CPU cores provided they all share the same L3 cache.
822Please always stop irq_balance which always does the worst possible thing on
823such workloads.
824
825For CPU-bound workloads consisting in a lot of SSL traffic or a lot of
826compression, it may be worth using multiple processes dedicated to certain
827tasks, though there is no universal rule here and experimentation will have to
828be performed.
829
830In order to increase the CPU capacity, it is possible to make HAProxy run as
831several processes, using the "nbproc" directive in the global section. There
832are some limitations though :
833 - health checks are run per process, so the target servers will get as many
834 checks as there are running processes ;
835 - maxconn values and queues are per-process so the correct value must be set
836 to avoid overloading the servers ;
837 - outgoing connections should avoid using port ranges to avoid conflicts
838 - stick-tables are per process and are not shared between processes ;
839 - each peers section may only run on a single process at a time ;
840 - the CLI operations will only act on a single process at a time.
841
842With this in mind, it appears that the easiest setup often consists in having
843one first layer running on multiple processes and in charge for the heavy
844processing, passing the traffic to a second layer running in a single process.
845This mechanism is suited to SSL and compression which are the two CPU-heavy
846features. Instances can easily be chained over UNIX sockets (which are cheaper
fengpeiyuancc123c62016-01-15 16:40:53 +0800847than TCP sockets and which do not waste ports), and the proxy protocol which is
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200848useful to pass client information to the next stage. When doing so, it is
849generally a good idea to bind all the single-process tasks to process number 1
850and extra tasks to next processes, as this will make it easier to generate
851similar configurations for different machines.
852
853On Linux versions 3.9 and above, running HAProxy in multi-process mode is much
854more efficient when each process uses a distinct listening socket on the same
855IP:port ; this will make the kernel evenly distribute the load across all
856processes instead of waking them all up. Please check the "process" option of
857the "bind" keyword lines in the configuration manual for more information.
858
859
8608. Logging
861----------
862
863For logging, HAProxy always relies on a syslog server since it does not perform
864any file-system access. The standard way of using it is to send logs over UDP
865to the log server (by default on port 514). Very commonly this is configured to
866127.0.0.1 where the local syslog daemon is running, but it's also used over the
867network to log to a central server. The central server provides additional
868benefits especially in active-active scenarios where it is desirable to keep
869the logs merged in arrival order. HAProxy may also make use of a UNIX socket to
870send its logs to the local syslog daemon, but it is not recommended at all,
871because if the syslog server is restarted while haproxy runs, the socket will
872be replaced and new logs will be lost. Since HAProxy will be isolated inside a
873chroot jail, it will not have the ability to reconnect to the new socket. It
874has also been observed in field that the log buffers in use on UNIX sockets are
875very small and lead to lost messages even at very light loads. But this can be
876fine for testing however.
877
878It is recommended to add the following directive to the "global" section to
879make HAProxy log to the local daemon using facility "local0" :
880
881 log 127.0.0.1:514 local0
882
883and then to add the following one to each "defaults" section or to each frontend
884and backend section :
885
886 log global
887
888This way, all logs will be centralized through the global definition of where
889the log server is.
890
891Some syslog daemons do not listen to UDP traffic by default, so depending on
892the daemon being used, the syntax to enable this will vary :
893
894 - on sysklogd, you need to pass argument "-r" on the daemon's command line
895 so that it listens to a UDP socket for "remote" logs ; note that there is
896 no way to limit it to address 127.0.0.1 so it will also receive logs from
897 remote systems ;
898
899 - on rsyslogd, the following lines must be added to the configuration file :
900
901 $ModLoad imudp
902 $UDPServerAddress *
903 $UDPServerRun 514
904
905 - on syslog-ng, a new source can be created the following way, it then needs
906 to be added as a valid source in one of the "log" directives :
907
908 source s_udp {
909 udp(ip(127.0.0.1) port(514));
910 };
911
912Please consult your syslog daemon's manual for more information. If no logs are
913seen in the system's log files, please consider the following tests :
914
915 - restart haproxy. Each frontend and backend logs one line indicating it's
916 starting. If these logs are received, it means logs are working.
917
918 - run "strace -tt -s100 -etrace=sendmsg -p <haproxy's pid>" and perform some
919 activity that you expect to be logged. You should see the log messages
920 being sent using sendmsg() there. If they don't appear, restart using
921 strace on top of haproxy. If you still see no logs, it definitely means
922 that something is wrong in your configuration.
923
924 - run tcpdump to watch for port 514, for example on the loopback interface if
925 the traffic is being sent locally : "tcpdump -As0 -ni lo port 514". If the
926 packets are seen there, it's the proof they're sent then the syslogd daemon
927 needs to be troubleshooted.
928
929While traffic logs are sent from the frontends (where the incoming connections
930are accepted), backends also need to be able to send logs in order to report a
931server state change consecutive to a health check. Please consult HAProxy's
932configuration manual for more information regarding all possible log settings.
933
Dan Lloyd8e48b872016-07-01 21:01:18 -0400934It is convenient to chose a facility that is not used by other daemons. HAProxy
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200935examples often suggest "local0" for traffic logs and "local1" for admin logs
936because they're never seen in field. A single facility would be enough as well.
937Having separate logs is convenient for log analysis, but it's also important to
938remember that logs may sometimes convey confidential information, and as such
Dan Lloyd8e48b872016-07-01 21:01:18 -0400939they must not be mixed with other logs that may accidentally be handed out to
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200940unauthorized people.
941
942For in-field troubleshooting without impacting the server's capacity too much,
943it is recommended to make use of the "halog" utility provided with HAProxy.
944This is sort of a grep-like utility designed to process HAProxy log files at
945a very fast data rate. Typical figures range between 1 and 2 GB of logs per
946second. It is capable of extracting only certain logs (eg: search for some
947classes of HTTP status codes, connection termination status, search by response
948time ranges, look for errors only), count lines, limit the output to a number
949of lines, and perform some more advanced statistics such as sorting servers
950by response time or error counts, sorting URLs by time or count, sorting client
951addresses by access count, and so on. It is pretty convenient to quickly spot
952anomalies such as a bot looping on the site, and block them.
953
954
9559. Statistics and monitoring
956----------------------------
957
Willy Tarreau44aed902015-10-13 14:45:29 +0200958It is possible to query HAProxy about its status. The most commonly used
959mechanism is the HTTP statistics page. This page also exposes an alternative
960CSV output format for monitoring tools. The same format is provided on the
961Unix socket.
962
Amaury Denoyelle072f97e2020-10-05 11:49:37 +0200963Statistics are regroup in categories labelled as domains, corresponding to the
Ilya Shipitsin2272d8a2020-12-21 01:22:40 +0500964multiple components of HAProxy. There are two domains available: proxy and dns.
Amaury Denoyellefbd0bc92020-10-05 11:49:46 +0200965If not specified, the proxy domain is selected. Note that only the proxy
966statistics are printed on the HTTP page.
Willy Tarreau44aed902015-10-13 14:45:29 +0200967
9689.1. CSV format
969---------------
970
971The statistics may be consulted either from the unix socket or from the HTTP
972page. Both means provide a CSV format whose fields follow. The first line
973begins with a sharp ('#') and has one word per comma-delimited field which
974represents the title of the column. All other lines starting at the second one
975use a classical CSV format using a comma as the delimiter, and the double quote
976('"') as an optional text delimiter, but only if the enclosed text is ambiguous
977(if it contains a quote or a comma). The double-quote character ('"') in the
978text is doubled ('""'), which is the format that most tools recognize. Please
979do not insert any column before these ones in order not to break tools which
980use hard-coded column positions.
981
Amaury Denoyelle50660a82020-10-05 11:49:39 +0200982For proxy statistics, after each field name, the types which may have a value
983for that field are specified in brackets. The types are L (Listeners), F
984(Frontends), B (Backends), and S (Servers). There is a fixed set of static
985fields that are always available in the same order. A column containing the
986character '-' delimits the end of the static fields, after which presence or
987order of the fields are not guaranteed.
Willy Tarreau44aed902015-10-13 14:45:29 +0200988
Amaury Denoyelle50660a82020-10-05 11:49:39 +0200989Here is the list of static fields using the proxy statistics domain:
Willy Tarreau44aed902015-10-13 14:45:29 +0200990 0. pxname [LFBS]: proxy name
991 1. svname [LFBS]: service name (FRONTEND for frontend, BACKEND for backend,
992 any name for server/listener)
993 2. qcur [..BS]: current queued requests. For the backend this reports the
994 number queued without a server assigned.
995 3. qmax [..BS]: max value of qcur
996 4. scur [LFBS]: current sessions
997 5. smax [LFBS]: max sessions
998 6. slim [LFBS]: configured session limit
Willy Tarreauc73810f2016-01-11 13:52:04 +0100999 7. stot [LFBS]: cumulative number of sessions
Willy Tarreau44aed902015-10-13 14:45:29 +02001000 8. bin [LFBS]: bytes in
1001 9. bout [LFBS]: bytes out
1002 10. dreq [LFB.]: requests denied because of security concerns.
1003 - For tcp this is because of a matched tcp-request content rule.
1004 - For http this is because of a matched http-request or tarpit rule.
1005 11. dresp [LFBS]: responses denied because of security concerns.
1006 - For http this is because of a matched http-request rule, or
1007 "option checkcache".
1008 12. ereq [LF..]: request errors. Some of the possible causes are:
1009 - early termination from the client, before the request has been sent.
1010 - read error from the client
1011 - client timeout
1012 - client closed connection
1013 - various bad requests from the client.
1014 - request was tarpitted.
1015 13. econ [..BS]: number of requests that encountered an error trying to
1016 connect to a backend server. The backend stat is the sum of the stat
1017 for all servers of that backend, plus any connection errors not
1018 associated with a particular server (such as the backend having no
1019 active servers).
1020 14. eresp [..BS]: response errors. srv_abrt will be counted here also.
1021 Some other errors are:
1022 - write error on the client socket (won't be counted for the server stat)
1023 - failure applying filters to the response.
1024 15. wretr [..BS]: number of times a connection to a server was retried.
1025 16. wredis [..BS]: number of times a request was redispatched to another
1026 server. The server value counts the number of times that server was
1027 switched away from.
Willy Tarreaub96dd282016-11-09 14:45:51 +01001028 17. status [LFBS]: status (UP/DOWN/NOLB/MAINT/MAINT(via)/MAINT(resolution)...)
Willy Tarreaubd715102020-10-23 22:44:30 +02001029 18. weight [..BS]: total effective weight (backend), effective weight (server)
Willy Tarreau44aed902015-10-13 14:45:29 +02001030 19. act [..BS]: number of active servers (backend), server is active (server)
1031 20. bck [..BS]: number of backup servers (backend), server is backup (server)
1032 21. chkfail [...S]: number of failed checks. (Only counts checks failed when
1033 the server is up.)
1034 22. chkdown [..BS]: number of UP->DOWN transitions. The backend counter counts
1035 transitions to the whole backend being down, rather than the sum of the
1036 counters for each server.
1037 23. lastchg [..BS]: number of seconds since the last UP<->DOWN transition
1038 24. downtime [..BS]: total downtime (in seconds). The value for the backend
1039 is the downtime for the whole backend, not the sum of the server downtime.
1040 25. qlimit [...S]: configured maxqueue for the server, or nothing in the
1041 value is 0 (default, meaning no limit)
1042 26. pid [LFBS]: process id (0 for first instance, 1 for second, ...)
1043 27. iid [LFBS]: unique proxy id
1044 28. sid [L..S]: server id (unique inside a proxy)
1045 29. throttle [...S]: current throttle percentage for the server, when
1046 slowstart is active, or no value if not in slowstart.
1047 30. lbtot [..BS]: total number of times a server was selected, either for new
1048 sessions, or when re-dispatching. The server counter is the number
1049 of times that server was selected.
1050 31. tracked [...S]: id of proxy/server if tracking is enabled.
1051 32. type [LFBS]: (0=frontend, 1=backend, 2=server, 3=socket/listener)
1052 33. rate [.FBS]: number of sessions per second over last elapsed second
1053 34. rate_lim [.F..]: configured limit on new sessions per second
1054 35. rate_max [.FBS]: max number of new sessions per second
1055 36. check_status [...S]: status of last health check, one of:
1056 UNK -> unknown
1057 INI -> initializing
1058 SOCKERR -> socket error
1059 L4OK -> check passed on layer 4, no upper layers testing enabled
1060 L4TOUT -> layer 1-4 timeout
1061 L4CON -> layer 1-4 connection problem, for example
1062 "Connection refused" (tcp rst) or "No route to host" (icmp)
1063 L6OK -> check passed on layer 6
1064 L6TOUT -> layer 6 (SSL) timeout
1065 L6RSP -> layer 6 invalid response - protocol error
1066 L7OK -> check passed on layer 7
1067 L7OKC -> check conditionally passed on layer 7, for example 404 with
1068 disable-on-404
1069 L7TOUT -> layer 7 (HTTP/SMTP) timeout
1070 L7RSP -> layer 7 invalid response - protocol error
1071 L7STS -> layer 7 response error, for example HTTP 5xx
Daniel Schnellerb6c8b0d2017-09-01 19:13:55 +02001072 Notice: If a check is currently running, the last known status will be
1073 reported, prefixed with "* ". e. g. "* L7OK".
Willy Tarreau44aed902015-10-13 14:45:29 +02001074 37. check_code [...S]: layer5-7 code, if available
1075 38. check_duration [...S]: time in ms took to finish last health check
1076 39. hrsp_1xx [.FBS]: http responses with 1xx code
1077 40. hrsp_2xx [.FBS]: http responses with 2xx code
1078 41. hrsp_3xx [.FBS]: http responses with 3xx code
1079 42. hrsp_4xx [.FBS]: http responses with 4xx code
1080 43. hrsp_5xx [.FBS]: http responses with 5xx code
1081 44. hrsp_other [.FBS]: http responses with other codes (protocol error)
1082 45. hanafail [...S]: failed health checks details
1083 46. req_rate [.F..]: HTTP requests per second over last elapsed second
1084 47. req_rate_max [.F..]: max number of HTTP requests per second observed
Willy Tarreaufb981bd2016-12-12 14:31:46 +01001085 48. req_tot [.FB.]: total number of HTTP requests received
Willy Tarreau44aed902015-10-13 14:45:29 +02001086 49. cli_abrt [..BS]: number of data transfers aborted by the client
1087 50. srv_abrt [..BS]: number of data transfers aborted by the server
1088 (inc. in eresp)
1089 51. comp_in [.FB.]: number of HTTP response bytes fed to the compressor
1090 52. comp_out [.FB.]: number of HTTP response bytes emitted by the compressor
1091 53. comp_byp [.FB.]: number of bytes that bypassed the HTTP compressor
1092 (CPU/BW limit)
1093 54. comp_rsp [.FB.]: number of HTTP responses that were compressed
1094 55. lastsess [..BS]: number of seconds since last session assigned to
1095 server/backend
1096 56. last_chk [...S]: last health check contents or textual error
1097 57. last_agt [...S]: last agent check contents or textual error
1098 58. qtime [..BS]: the average queue time in ms over the 1024 last requests
1099 59. ctime [..BS]: the average connect time in ms over the 1024 last requests
1100 60. rtime [..BS]: the average response time in ms over the 1024 last requests
1101 (0 for TCP)
1102 61. ttime [..BS]: the average total session time in ms over the 1024 last
1103 requests
Willy Tarreau7f618842016-01-08 11:40:03 +01001104 62. agent_status [...S]: status of last agent check, one of:
1105 UNK -> unknown
1106 INI -> initializing
1107 SOCKERR -> socket error
1108 L4OK -> check passed on layer 4, no upper layers testing enabled
1109 L4TOUT -> layer 1-4 timeout
1110 L4CON -> layer 1-4 connection problem, for example
1111 "Connection refused" (tcp rst) or "No route to host" (icmp)
1112 L7OK -> agent reported "up"
1113 L7STS -> agent reported "fail", "stop", or "down"
1114 63. agent_code [...S]: numeric code reported by agent if any (unused for now)
1115 64. agent_duration [...S]: time in ms taken to finish last check
Willy Tarreaudd7354b2016-01-08 13:47:26 +01001116 65. check_desc [...S]: short human-readable description of check_status
1117 66. agent_desc [...S]: short human-readable description of agent_status
Willy Tarreau3141f592016-01-08 14:25:28 +01001118 67. check_rise [...S]: server's "rise" parameter used by checks
1119 68. check_fall [...S]: server's "fall" parameter used by checks
1120 69. check_health [...S]: server's health check value between 0 and rise+fall-1
1121 70. agent_rise [...S]: agent's "rise" parameter, normally 1
1122 71. agent_fall [...S]: agent's "fall" parameter, normally 1
1123 72. agent_health [...S]: agent's health parameter, between 0 and rise+fall-1
Willy Tarreaua6f5a732016-01-08 16:59:56 +01001124 73. addr [L..S]: address:port or "unix". IPv6 has brackets around the address.
Willy Tarreaue4847c62016-01-08 15:43:54 +01001125 74: cookie [..BS]: server's cookie value or backend's cookie name
Willy Tarreauf8211df2016-01-11 14:09:38 +01001126 75: mode [LFBS]: proxy mode (tcp, http, health, unknown)
Willy Tarreauf1516d92016-01-11 14:48:36 +01001127 76: algo [..B.]: load balancing algorithm
Willy Tarreauc73810f2016-01-11 13:52:04 +01001128 77: conn_rate [.F..]: number of connections over the last elapsed second
1129 78: conn_rate_max [.F..]: highest known conn_rate
1130 79: conn_tot [.F..]: cumulative number of connections
Willy Tarreau5b9bdff2016-01-11 14:40:47 +01001131 80: intercepted [.FB.]: cum. number of intercepted requests (monitor, stats)
Willy Tarreau8a90b8e2016-10-21 18:15:32 +02001132 81: dcon [LF..]: requests denied by "tcp-request connection" rules
Willy Tarreaua5bc36b2016-10-21 18:16:27 +02001133 82: dses [LF..]: requests denied by "tcp-request session" rules
Willy Tarreauea96a822018-05-28 15:15:43 +02001134 83: wrew [LFBS]: cumulative number of failed header rewriting warnings
Jérôme Magnin708eb882019-07-17 09:24:46 +02001135 84: connect [..BS]: cumulative number of connection establishment attempts
1136 85: reuse [..BS]: cumulative number of connection reuses
Willy Tarreau72974292019-11-08 07:29:34 +01001137 86: cache_lookups [.FB.]: cumulative number of cache lookups
Jérôme Magnin34ebb5c2019-07-17 14:04:40 +02001138 87: cache_hits [.FB.]: cumulative number of cache hits
Christopher Faulet2ac25742019-11-08 15:27:27 +01001139 88: srv_icur [...S]: current number of idle connections available for reuse
1140 89: src_ilim [...S]: limit on the number of available idle connections
1141 90. qtime_max [..BS]: the maximum observed queue time in ms
1142 91. ctime_max [..BS]: the maximum observed connect time in ms
1143 92. rtime_max [..BS]: the maximum observed response time in ms (0 for TCP)
1144 93. ttime_max [..BS]: the maximum observed total session time in ms
Christopher Faulet0159ee42019-12-16 14:40:39 +01001145 94. eint [LFBS]: cumulative number of internal errors
Pierre Cheynier08eb7182020-10-08 16:37:14 +02001146 95. idle_conn_cur [...S]: current number of unsafe idle connections
1147 96. safe_conn_cur [...S]: current number of safe idle connections
1148 97. used_conn_cur [...S]: current number of connections in use
1149 98. need_conn_est [...S]: estimated needed number of connections
Willy Tarreaubd715102020-10-23 22:44:30 +02001150 99. uweight [..BS]: total user weight (backend), server user weight (server)
Willy Tarreau44aed902015-10-13 14:45:29 +02001151
Amaury Denoyelle50660a82020-10-05 11:49:39 +02001152For all other statistics domains, the presence or the order of the fields are
1153not guaranteed. In this case, the header line should always be used to parse
1154the CSV data.
Willy Tarreau44aed902015-10-13 14:45:29 +02001155
Phil Schererb931f962020-12-02 19:36:08 +000011569.2. Typed output format
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001157------------------------
1158
1159Both "show info" and "show stat" support a mode where each output value comes
1160with its type and sufficient information to know how the value is supposed to
1161be aggregated between processes and how it evolves.
1162
1163In all cases, the output consists in having a single value per line with all
1164the information split into fields delimited by colons (':').
1165
1166The first column designates the object or metric being dumped. Its format is
1167specific to the command producing this output and will not be described in this
1168section. Usually it will consist in a series of identifiers and field names.
1169
1170The second column contains 3 characters respectively indicating the origin, the
1171nature and the scope of the value being reported. The first character (the
1172origin) indicates where the value was extracted from. Possible characters are :
1173
1174 M The value is a metric. It is valid at one instant any may change depending
1175 on its nature .
1176
1177 S The value is a status. It represents a discrete value which by definition
1178 cannot be aggregated. It may be the status of a server ("UP" or "DOWN"),
1179 the PID of the process, etc.
1180
1181 K The value is a sorting key. It represents an identifier which may be used
1182 to group some values together because it is unique among its class. All
1183 internal identifiers are keys. Some names can be listed as keys if they
1184 are unique (eg: a frontend name is unique). In general keys come from the
Dan Lloyd8e48b872016-07-01 21:01:18 -04001185 configuration, even though some of them may automatically be assigned. For
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001186 most purposes keys may be considered as equivalent to configuration.
1187
1188 C The value comes from the configuration. Certain configuration values make
1189 sense on the output, for example a concurrent connection limit or a cookie
1190 name. By definition these values are the same in all processes started
1191 from the same configuration file.
1192
1193 P The value comes from the product itself. There are very few such values,
1194 most common use is to report the product name, version and release date.
1195 These elements are also the same between all processes.
1196
1197The second character (the nature) indicates the nature of the information
1198carried by the field in order to let an aggregator decide on what operation to
1199use to aggregate multiple values. Possible characters are :
1200
1201 A The value represents an age since a last event. This is a bit different
1202 from the duration in that an age is automatically computed based on the
1203 current date. A typical example is how long ago did the last session
1204 happen on a server. Ages are generally aggregated by taking the minimum
1205 value and do not need to be stored.
1206
1207 a The value represents an already averaged value. The average response times
1208 and server weights are of this nature. Averages can typically be averaged
1209 between processes.
1210
1211 C The value represents a cumulative counter. Such measures perpetually
1212 increase until they wrap around. Some monitoring protocols need to tell
1213 the difference between a counter and a gauge to report a different type.
1214 In general counters may simply be summed since they represent events or
1215 volumes. Examples of metrics of this nature are connection counts or byte
1216 counts.
1217
1218 D The value represents a duration for a status. There are a few usages of
1219 this, most of them include the time taken by the last health check and
1220 the time a server has spent down. Durations are generally not summed,
1221 most of the time the maximum will be retained to compute an SLA.
1222
1223 G The value represents a gauge. It's a measure at one instant. The memory
1224 usage or the current number of active connections are of this nature.
1225 Metrics of this type are typically summed during aggregation.
1226
1227 L The value represents a limit (generally a configured one). By nature,
1228 limits are harder to aggregate since they are specific to the point where
1229 they were retrieved. In certain situations they may be summed or be kept
1230 separate.
1231
1232 M The value represents a maximum. In general it will apply to a gauge and
1233 keep the highest known value. An example of such a metric could be the
1234 maximum amount of concurrent connections that was encountered in the
1235 product's life time. To correctly aggregate maxima, you are supposed to
1236 output a range going from the maximum of all maxima and the sum of all
1237 of them. There is indeed no way to know if they were encountered
1238 simultaneously or not.
1239
1240 m The value represents a minimum. In general it will apply to a gauge and
1241 keep the lowest known value. An example of such a metric could be the
1242 minimum amount of free memory pools that was encountered in the product's
1243 life time. To correctly aggregate minima, you are supposed to output a
1244 range going from the minimum of all minima and the sum of all of them.
1245 There is indeed no way to know if they were encountered simultaneously
1246 or not.
1247
1248 N The value represents a name, so it is a string. It is used to report
1249 proxy names, server names and cookie names. Names have configuration or
1250 keys as their origin and are supposed to be the same among all processes.
1251
1252 O The value represents a free text output. Outputs from various commands,
1253 returns from health checks, node descriptions are of such nature.
1254
1255 R The value represents an event rate. It's a measure at one instant. It is
1256 quite similar to a gauge except that the recipient knows that this measure
1257 moves slowly and may decide not to keep all values. An example of such a
1258 metric is the measured amount of connections per second. Metrics of this
1259 type are typically summed during aggregation.
1260
1261 T The value represents a date or time. A field emitting the current date
1262 would be of this type. The method to aggregate such information is left
1263 as an implementation choice. For now no field uses this type.
1264
1265The third character (the scope) indicates what extent the value reflects. Some
1266elements may be per process while others may be per configuration or per system.
1267The distinction is important to know whether or not a single value should be
1268kept during aggregation or if values have to be aggregated. The following
1269characters are currently supported :
1270
1271 C The value is valid for a whole cluster of nodes, which is the set of nodes
1272 communicating over the peers protocol. An example could be the amount of
1273 entries present in a stick table that is replicated with other peers. At
1274 the moment no metric use this scope.
1275
1276 P The value is valid only for the process reporting it. Most metrics use
1277 this scope.
1278
1279 S The value is valid for the whole service, which is the set of processes
1280 started together from the same configuration file. All metrics originating
1281 from the configuration use this scope. Some other metrics may use it as
1282 well for some shared resources (eg: shared SSL cache statistics).
1283
1284 s The value is valid for the whole system, such as the system's hostname,
1285 current date or resource usage. At the moment this scope is not used by
1286 any metric.
1287
1288Consumers of these information will generally have enough of these 3 characters
1289to determine how to accurately report aggregated information across multiple
1290processes.
1291
1292After this column, the third column indicates the type of the field, among "s32"
1293(signed 32-bit integer), "s64" (signed 64-bit integer), "u32" (unsigned 32-bit
1294integer), "u64" (unsigned 64-bit integer), "str" (string). It is important to
1295know the type before parsing the value in order to properly read it. For example
1296a string containing only digits is still a string an not an integer (eg: an
1297error code extracted by a check).
1298
1299Then the fourth column is the value itself, encoded according to its type.
1300Strings are dumped as-is immediately after the colon without any leading space.
1301If a string contains a colon, it will appear normally. This means that the
1302output should not be exclusively split around colons or some check outputs
1303or server addresses might be truncated.
1304
1305
13069.3. Unix Socket commands
Willy Tarreau44aed902015-10-13 14:45:29 +02001307-------------------------
1308
1309The stats socket is not enabled by default. In order to enable it, it is
1310necessary to add one line in the global section of the haproxy configuration.
1311A second line is recommended to set a larger timeout, always appreciated when
1312issuing commands by hand :
1313
1314 global
1315 stats socket /var/run/haproxy.sock mode 600 level admin
1316 stats timeout 2m
1317
1318It is also possible to add multiple instances of the stats socket by repeating
1319the line, and make them listen to a TCP port instead of a UNIX socket. This is
1320never done by default because this is dangerous, but can be handy in some
1321situations :
1322
1323 global
1324 stats socket /var/run/haproxy.sock mode 600 level admin
1325 stats socket ipv4@192.168.0.1:9999 level admin
1326 stats timeout 2m
1327
1328To access the socket, an external utility such as "socat" is required. Socat is
1329a swiss-army knife to connect anything to anything. We use it to connect
1330terminals to the socket, or a couple of stdin/stdout pipes to it for scripts.
1331The two main syntaxes we'll use are the following :
1332
1333 # socat /var/run/haproxy.sock stdio
1334 # socat /var/run/haproxy.sock readline
1335
1336The first one is used with scripts. It is possible to send the output of a
1337script to haproxy, and pass haproxy's output to another script. That's useful
1338for retrieving counters or attack traces for example.
1339
1340The second one is only useful for issuing commands by hand. It has the benefit
1341that the terminal is handled by the readline library which supports line
1342editing and history, which is very convenient when issuing repeated commands
1343(eg: watch a counter).
1344
1345The socket supports two operation modes :
1346 - interactive
1347 - non-interactive
1348
1349The non-interactive mode is the default when socat connects to the socket. In
1350this mode, a single line may be sent. It is processed as a whole, responses are
1351sent back, and the connection closes after the end of the response. This is the
1352mode that scripts and monitoring tools use. It is possible to send multiple
1353commands in this mode, they need to be delimited by a semi-colon (';'). For
1354example :
1355
1356 # echo "show info;show stat;show table" | socat /var/run/haproxy stdio
1357
Dragan Dosena1c35ab2016-11-24 11:33:12 +01001358If a command needs to use a semi-colon or a backslash (eg: in a value), it
Joseph Herlant71b4b152018-11-13 16:55:16 -08001359must be preceded by a backslash ('\').
Chad Lavoiee3f50312016-05-26 16:42:25 -04001360
Willy Tarreau44aed902015-10-13 14:45:29 +02001361The interactive mode displays a prompt ('>') and waits for commands to be
1362entered on the line, then processes them, and displays the prompt again to wait
1363for a new command. This mode is entered via the "prompt" command which must be
1364sent on the first line in non-interactive mode. The mode is a flip switch, if
1365"prompt" is sent in interactive mode, it is disabled and the connection closes
1366after processing the last command of the same line.
1367
1368For this reason, when debugging by hand, it's quite common to start with the
1369"prompt" command :
1370
1371 # socat /var/run/haproxy readline
1372 prompt
1373 > show info
1374 ...
1375 >
1376
1377Since multiple commands may be issued at once, haproxy uses the empty line as a
1378delimiter to mark an end of output for each command, and takes care of ensuring
1379that no command can emit an empty line on output. A script can thus easily
1380parse the output even when multiple commands were pipelined on a single line.
1381
Aurélien Nephtaliabbf6072018-04-18 13:26:46 +02001382Some commands may take an optional payload. To add one to a command, the first
1383line needs to end with the "<<\n" pattern. The next lines will be treated as
1384the payload and can contain as many lines as needed. To validate a command with
1385a payload, it needs to end with an empty line.
1386
1387Limitations do exist: the length of the whole buffer passed to the CLI must
1388not be greater than tune.bfsize and the pattern "<<" must not be glued to the
1389last word of the line.
1390
1391When entering a paylod while in interactive mode, the prompt will change from
1392"> " to "+ ".
1393
Willy Tarreau44aed902015-10-13 14:45:29 +02001394It is important to understand that when multiple haproxy processes are started
1395on the same sockets, any process may pick up the request and will output its
1396own stats.
1397
1398The list of commands currently supported on the stats socket is provided below.
1399If an unknown command is sent, haproxy displays the usage message which reminds
1400all supported commands. Some commands support a more complex syntax, generally
1401it will explain what part of the command is invalid when this happens.
1402
Olivier Doucetd8703e82017-08-31 11:05:10 +02001403Some commands require a higher level of privilege to work. If you do not have
1404enough privilege, you will get an error "Permission denied". Please check
1405the "level" option of the "bind" keyword lines in the configuration manual
1406for more information.
1407
William Lallemand6ab08b32019-11-29 16:48:43 +01001408abort ssl cert <filename>
1409 Abort and destroy a temporary SSL certificate update transaction.
1410
1411 See also "set ssl cert" and "commit ssl cert".
1412
Willy Tarreaubb51c442021-04-30 15:23:36 +02001413add acl [@<ver>] <acl> <pattern>
Willy Tarreau44aed902015-10-13 14:45:29 +02001414 Add an entry into the acl <acl>. <acl> is the #<id> or the <file> returned by
Willy Tarreaubb51c442021-04-30 15:23:36 +02001415 "show acl". This command does not verify if the entry already exists. Entries
1416 are added to the current version of the ACL, unless a specific version is
1417 specified with "@<ver>". This version number must have preliminary been
1418 allocated by "prepare acl", and it will be comprised between the versions
1419 reported in "curr_ver" and "next_ver" on the output of "show acl". Entries
1420 added with a specific version number will not match until a "commit acl"
1421 operation is performed on them. They may however be consulted using the
1422 "show acl @<ver>" command, and cleared using a "clear acl @<ver>" command.
1423 This command cannot be used if the reference <acl> is a file also used with
1424 a map. In this case, the "add map" command must be used instead.
Willy Tarreau44aed902015-10-13 14:45:29 +02001425
Willy Tarreaubb51c442021-04-30 15:23:36 +02001426add map [@<ver>] <map> <key> <value>
1427add map [@<ver>] <map> <payload>
Willy Tarreau44aed902015-10-13 14:45:29 +02001428 Add an entry into the map <map> to associate the value <value> to the key
1429 <key>. This command does not verify if the entry already exists. It is
Willy Tarreaubb51c442021-04-30 15:23:36 +02001430 mainly used to fill a map after a "clear" or "prepare" operation. Entries
1431 are added to the current version of the ACL, unless a specific version is
1432 specified with "@<ver>". This version number must have preliminary been
1433 allocated by "prepare acl", and it will be comprised between the versions
1434 reported in "curr_ver" and "next_ver" on the output of "show acl". Entries
1435 added with a specific version number will not match until a "commit map"
1436 operation is performed on them. They may however be consulted using the
1437 "show map @<ver>" command, and cleared using a "clear acl @<ver>" command.
1438 If the designated map is also used as an ACL, the ACL will only match the
1439 <key> part and will ignore the <value> part. Using the payload syntax it is
1440 possible to add multiple key/value pairs by entering them on separate lines.
1441 On each new line, the first word is the key and the rest of the line is
1442 considered to be the value which can even contains spaces.
Aurélien Nephtali25650ce2018-04-18 14:04:47 +02001443
1444 Example:
1445
1446 # socat /tmp/sock1 -
1447 prompt
1448
1449 > add map #-1 <<
1450 + key1 value1
1451 + key2 value2 with spaces
1452 + key3 value3 also with spaces
1453 + key4 value4
1454
1455 >
Willy Tarreau44aed902015-10-13 14:45:29 +02001456
Amaury Denoyellef99f77a2021-03-08 17:13:32 +01001457add server <backend>/<server> [args]*
1458 Instantiate a new server attached to the backend <backend>. Only supported on
1459 a CLI connection running in experimental mode (see "experimental-mode on").
1460 This method is still in development and may change in the future.
1461
1462 The <server> name must not be already used in the backend. A special
Amaury Denoyelleeafd7012021-04-29 14:59:42 +02001463 restriction is put on the backend which must used a dynamic load-balancing
1464 algorithm. A subset of keywords from the server config file statement can be
1465 used to configure the server behavior. Also note that no settings will be
1466 reused from an hypothetical 'default-server' statement in the same backend.
Amaury Denoyellefc465a52021-03-09 17:36:23 +01001467
Amaury Denoyellee9bb7fb2021-06-10 17:34:10 +02001468 Currently a dynamic server is statically initialized with the "none"
1469 init-addr method. This means that no resolution will be undertaken if a FQDN
1470 is specified as an address, even if the server creation will be validated.
1471
Amaury Denoyellefc465a52021-03-09 17:36:23 +01001472 Here is the list of the currently supported keywords :
1473
1474 - backup
1475 - disabled
1476 - enabled
1477 - id
1478 - maxconn
1479 - maxqueue
1480 - minconn
1481 - pool-low-conn
1482 - pool-max-conn
1483 - pool-purge-delay
Amaury Denoyelle30467232021-03-12 18:03:27 +01001484 - proto
Amaury Denoyellefc465a52021-03-09 17:36:23 +01001485 - proxy-v2-options
1486 - send-proxy
1487 - send-proxy-v2
1488 - source
1489 - tfo
1490 - usesrc
1491 - weight
Amaury Denoyelle69352ec2021-10-18 14:40:29 +02001492 - ws
Amaury Denoyellefc465a52021-03-09 17:36:23 +01001493
1494 Their syntax is similar to the server line from the configuration file,
1495 please refer to their individual documentation for details.
Amaury Denoyellef99f77a2021-03-08 17:13:32 +01001496
William Lallemandaccac232020-04-02 17:42:51 +02001497add ssl crt-list <crtlist> <certificate>
1498add ssl crt-list <crtlist> <payload>
1499 Add an certificate in a crt-list. It can also be used for directories since
1500 directories are now loaded the same way as the crt-lists. This command allow
1501 you to use a certificate name in parameter, to use SSL options or filters a
1502 crt-list line must sent as a payload instead. Only one crt-list line is
1503 supported in the payload. This command will load the certificate for every
1504 bind lines using the crt-list. To push a new certificate to HAProxy the
1505 commands "new ssl cert" and "set ssl cert" must be used.
1506
1507 Example:
1508 $ echo "new ssl cert foobar.pem" | socat /tmp/sock1 -
1509 $ echo -e "set ssl cert foobar.pem <<\n$(cat foobar.pem)\n" | socat
1510 /tmp/sock1 -
1511 $ echo "commit ssl cert foobar.pem" | socat /tmp/sock1 -
1512 $ echo "add ssl crt-list certlist1 foobar.pem" | socat /tmp/sock1 -
1513
1514 $ echo -e 'add ssl crt-list certlist1 <<\nfoobar.pem [allow-0rtt] foo.bar.com
1515 !test1.com\n' | socat /tmp/sock1 -
1516
Willy Tarreau44aed902015-10-13 14:45:29 +02001517clear counters
1518 Clear the max values of the statistics counters in each proxy (frontend &
Willy Tarreaud80cb4e2018-01-20 19:30:13 +01001519 backend) and in each server. The accumulated counters are not affected. The
1520 internal activity counters reported by "show activity" are also reset. This
Willy Tarreau44aed902015-10-13 14:45:29 +02001521 can be used to get clean counters after an incident, without having to
1522 restart nor to clear traffic counters. This command is restricted and can
1523 only be issued on sockets configured for levels "operator" or "admin".
1524
1525clear counters all
1526 Clear all statistics counters in each proxy (frontend & backend) and in each
1527 server. This has the same effect as restarting. This command is restricted
1528 and can only be issued on sockets configured for level "admin".
1529
Willy Tarreauff3feeb2021-04-30 13:31:43 +02001530clear acl [@<ver>] <acl>
Willy Tarreau44aed902015-10-13 14:45:29 +02001531 Remove all entries from the acl <acl>. <acl> is the #<id> or the <file>
1532 returned by "show acl". Note that if the reference <acl> is a file and is
Willy Tarreauff3feeb2021-04-30 13:31:43 +02001533 shared with a map, this map will be also cleared. By default only the current
1534 version of the ACL is cleared (the one being matched against). However it is
1535 possible to specify another version using '@' followed by this version.
Willy Tarreau44aed902015-10-13 14:45:29 +02001536
Willy Tarreauff3feeb2021-04-30 13:31:43 +02001537clear map [@<ver>] <map>
Willy Tarreau44aed902015-10-13 14:45:29 +02001538 Remove all entries from the map <map>. <map> is the #<id> or the <file>
1539 returned by "show map". Note that if the reference <map> is a file and is
Willy Tarreauff3feeb2021-04-30 13:31:43 +02001540 shared with a acl, this acl will be also cleared. By default only the current
1541 version of the map is cleared (the one being matched against). However it is
1542 possible to specify another version using '@' followed by this version.
Willy Tarreau44aed902015-10-13 14:45:29 +02001543
1544clear table <table> [ data.<type> <operator> <value> ] | [ key <key> ]
1545 Remove entries from the stick-table <table>.
1546
1547 This is typically used to unblock some users complaining they have been
1548 abusively denied access to a service, but this can also be used to clear some
1549 stickiness entries matching a server that is going to be replaced (see "show
1550 table" below for details). Note that sometimes, removal of an entry will be
1551 refused because it is currently tracked by a session. Retrying a few seconds
1552 later after the session ends is usual enough.
1553
1554 In the case where no options arguments are given all entries will be removed.
1555
1556 When the "data." form is used entries matching a filter applied using the
1557 stored data (see "stick-table" in section 4.2) are removed. A stored data
1558 type must be specified in <type>, and this data type must be stored in the
1559 table otherwise an error is reported. The data is compared according to
1560 <operator> with the 64-bit integer <value>. Operators are the same as with
1561 the ACLs :
1562
1563 - eq : match entries whose data is equal to this value
1564 - ne : match entries whose data is not equal to this value
1565 - le : match entries whose data is less than or equal to this value
1566 - ge : match entries whose data is greater than or equal to this value
1567 - lt : match entries whose data is less than this value
1568 - gt : match entries whose data is greater than this value
1569
1570 When the key form is used the entry <key> is removed. The key must be of the
1571 same type as the table, which currently is limited to IPv4, IPv6, integer and
1572 string.
1573
1574 Example :
1575 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1576 >>> # table: http_proxy, type: ip, size:204800, used:2
1577 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
1578 bytes_out_rate(60000)=187
1579 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
1580 bytes_out_rate(60000)=191
1581
1582 $ echo "clear table http_proxy key 127.0.0.1" | socat stdio /tmp/sock1
1583
1584 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1585 >>> # table: http_proxy, type: ip, size:204800, used:1
1586 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
1587 bytes_out_rate(60000)=191
1588 $ echo "clear table http_proxy data.gpc0 eq 1" | socat stdio /tmp/sock1
1589 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1590 >>> # table: http_proxy, type: ip, size:204800, used:1
1591
Willy Tarreau7a562ca2021-04-30 15:10:01 +02001592commit acl @<ver> <acl>
1593 Commit all changes made to version <ver> of ACL <acl>, and deletes all past
1594 versions. <acl> is the #<id> or the <file> returned by "show acl". The
1595 version number must be between "curr_ver"+1 and "next_ver" as reported in
1596 "show acl". The contents to be committed to the ACL can be consulted with
1597 "show acl @<ver> <acl>" if desired. The specified version number has normally
1598 been created with the "prepare acl" command. The replacement is atomic. It
1599 consists in atomically updating the current version to the specified version,
1600 which will instantly cause all entries in other versions to become invisible,
1601 and all entries in the new version to become visible. It is also possible to
1602 use this command to perform an atomic removal of all visible entries of an
1603 ACL by calling "prepare acl" first then committing without adding any
1604 entries. This command cannot be used if the reference <acl> is a file also
1605 used as a map. In this case, the "commit map" command must be used instead.
1606
1607commit map @<ver> <map>
1608 Commit all changes made to version <ver> of map <map>, and deletes all past
1609 versions. <map> is the #<id> or the <file> returned by "show map". The
1610 version number must be between "curr_ver"+1 and "next_ver" as reported in
1611 "show map". The contents to be committed to the map can be consulted with
1612 "show map @<ver> <map>" if desired. The specified version number has normally
1613 been created with the "prepare map" command. The replacement is atomic. It
1614 consists in atomically updating the current version to the specified version,
1615 which will instantly cause all entries in other versions to become invisible,
1616 and all entries in the new version to become visible. It is also possible to
1617 use this command to perform an atomic removal of all visible entries of an
1618 map by calling "prepare map" first then committing without adding any
1619 entries.
1620
William Lallemand6ab08b32019-11-29 16:48:43 +01001621commit ssl cert <filename>
William Lallemandc184d872020-06-26 15:39:57 +02001622 Commit a temporary SSL certificate update transaction.
1623
1624 In the case of an existing certificate (in a "Used" state in "show ssl
1625 cert"), generate every SSL contextes and SNIs it need, insert them, and
1626 remove the previous ones. Replace in memory the previous SSL certificates
1627 everywhere the <filename> was used in the configuration. Upon failure it
1628 doesn't remove or insert anything. Once the temporary transaction is
1629 committed, it is destroyed.
1630
1631 In the case of a new certificate (after a "new ssl cert" and in a "Unused"
Ilya Shipitsin2272d8a2020-12-21 01:22:40 +05001632 state in "show ssl cert"), the certificate will be committed in a certificate
William Lallemandc184d872020-06-26 15:39:57 +02001633 storage, but it won't be used anywhere in haproxy. To use it and generate
1634 its SNIs you will need to add it to a crt-list or a directory with "add ssl
1635 crt-list".
William Lallemand6ab08b32019-11-29 16:48:43 +01001636
William Lallemandc184d872020-06-26 15:39:57 +02001637 See also "new ssl cert", "ssl set cert", "abort ssl cert" and
1638 "add ssl crt-list".
William Lallemand6ab08b32019-11-29 16:48:43 +01001639
Willy Tarreau6bdf3e92019-05-20 14:25:05 +02001640debug dev <command> [args]*
Willy Tarreaub24ab222019-10-24 18:03:39 +02001641 Call a developer-specific command. Only supported on a CLI connection running
1642 in expert mode (see "expert-mode on"). Such commands are extremely dangerous
1643 and not forgiving, any misuse may result in a crash of the process. They are
1644 intended for experts only, and must really not be used unless told to do so.
1645 Some of them are only available when haproxy is built with DEBUG_DEV defined
1646 because they may have security implications. All of these commands require
1647 admin privileges, and are purposely not documented to avoid encouraging their
1648 use by people who are not at ease with the source code.
Willy Tarreau6bdf3e92019-05-20 14:25:05 +02001649
Willy Tarreau44aed902015-10-13 14:45:29 +02001650del acl <acl> [<key>|#<ref>]
1651 Delete all the acl entries from the acl <acl> corresponding to the key <key>.
1652 <acl> is the #<id> or the <file> returned by "show acl". If the <ref> is used,
1653 this command delete only the listed reference. The reference can be found with
1654 listing the content of the acl. Note that if the reference <acl> is a file and
1655 is shared with a map, the entry will be also deleted in the map.
1656
1657del map <map> [<key>|#<ref>]
1658 Delete all the map entries from the map <map> corresponding to the key <key>.
1659 <map> is the #<id> or the <file> returned by "show map". If the <ref> is used,
1660 this command delete only the listed reference. The reference can be found with
1661 listing the content of the map. Note that if the reference <map> is a file and
1662 is shared with a acl, the entry will be also deleted in the map.
1663
William Lallemand419e6342020-04-08 12:05:39 +02001664del ssl cert <certfile>
1665 Delete a certificate store from HAProxy. The certificate must be unused and
1666 removed from any crt-list or directory. "show ssl cert" displays the status
1667 of the certificate. The deletion doesn't work with a certificate referenced
1668 directly with the "crt" directive in the configuration.
1669
William Lallemand0a9b9412020-04-06 17:43:05 +02001670del ssl crt-list <filename> <certfile[:line]>
1671 Delete an entry in a crt-list. This will delete every SNIs used for this
1672 entry in the frontends. If a certificate is used several time in a crt-list,
1673 you will need to provide which line you want to delete. To display the line
1674 numbers, use "show ssl crt-list -n <crtlist>".
1675
Amaury Denoyellee5580432021-04-15 14:41:20 +02001676del server <backend>/<server>
1677 Remove a server attached to the backend <backend>. Only valid on a server
1678 added at runtime. The server must be put in maintenance mode prior to its
1679 deletion. The operation is cancelled if the serveur still has active
1680 or idle connection or its connection queue is not empty.
1681
Willy Tarreau44aed902015-10-13 14:45:29 +02001682disable agent <backend>/<server>
1683 Mark the auxiliary agent check as temporarily stopped.
1684
1685 In the case where an agent check is being run as a auxiliary check, due
1686 to the agent-check parameter of a server directive, new checks are only
Dan Lloyd8e48b872016-07-01 21:01:18 -04001687 initialized when the agent is in the enabled. Thus, disable agent will
Willy Tarreau44aed902015-10-13 14:45:29 +02001688 prevent any new agent checks from begin initiated until the agent
1689 re-enabled using enable agent.
1690
1691 When an agent is disabled the processing of an auxiliary agent check that
1692 was initiated while the agent was set as enabled is as follows: All
1693 results that would alter the weight, specifically "drain" or a weight
1694 returned by the agent, are ignored. The processing of agent check is
1695 otherwise unchanged.
1696
1697 The motivation for this feature is to allow the weight changing effects
1698 of the agent checks to be paused to allow the weight of a server to be
1699 configured using set weight without being overridden by the agent.
1700
1701 This command is restricted and can only be issued on sockets configured for
1702 level "admin".
1703
Olivier Houchard614f8d72017-03-14 20:08:46 +01001704disable dynamic-cookie backend <backend>
Ilya Shipitsin2a950d02020-03-06 13:07:38 +05001705 Disable the generation of dynamic cookies for the backend <backend>
Olivier Houchard614f8d72017-03-14 20:08:46 +01001706
Willy Tarreau44aed902015-10-13 14:45:29 +02001707disable frontend <frontend>
1708 Mark the frontend as temporarily stopped. This corresponds to the mode which
1709 is used during a soft restart : the frontend releases the port but can be
1710 enabled again if needed. This should be used with care as some non-Linux OSes
1711 are unable to enable it back. This is intended to be used in environments
1712 where stopping a proxy is not even imaginable but a misconfigured proxy must
1713 be fixed. That way it's possible to release the port and bind it into another
1714 process to restore operations. The frontend will appear with status "STOP"
1715 on the stats page.
1716
1717 The frontend may be specified either by its name or by its numeric ID,
1718 prefixed with a sharp ('#').
1719
1720 This command is restricted and can only be issued on sockets configured for
1721 level "admin".
1722
1723disable health <backend>/<server>
1724 Mark the primary health check as temporarily stopped. This will disable
1725 sending of health checks, and the last health check result will be ignored.
1726 The server will be in unchecked state and considered UP unless an auxiliary
1727 agent check forces it down.
1728
1729 This command is restricted and can only be issued on sockets configured for
1730 level "admin".
1731
1732disable server <backend>/<server>
1733 Mark the server DOWN for maintenance. In this mode, no more checks will be
1734 performed on the server until it leaves maintenance.
1735 If the server is tracked by other servers, those servers will be set to DOWN
1736 during the maintenance.
1737
1738 In the statistics page, a server DOWN for maintenance will appear with a
1739 "MAINT" status, its tracking servers with the "MAINT(via)" one.
1740
1741 Both the backend and the server may be specified either by their name or by
1742 their numeric ID, prefixed with a sharp ('#').
1743
1744 This command is restricted and can only be issued on sockets configured for
1745 level "admin".
1746
1747enable agent <backend>/<server>
1748 Resume auxiliary agent check that was temporarily stopped.
1749
1750 See "disable agent" for details of the effect of temporarily starting
1751 and stopping an auxiliary agent.
1752
1753 This command is restricted and can only be issued on sockets configured for
1754 level "admin".
1755
Olivier Houchard614f8d72017-03-14 20:08:46 +01001756enable dynamic-cookie backend <backend>
n9@users.noreply.github.com25a1c8e2019-08-23 11:21:05 +02001757 Enable the generation of dynamic cookies for the backend <backend>.
1758 A secret key must also be provided.
Olivier Houchard614f8d72017-03-14 20:08:46 +01001759
Willy Tarreau44aed902015-10-13 14:45:29 +02001760enable frontend <frontend>
1761 Resume a frontend which was temporarily stopped. It is possible that some of
1762 the listening ports won't be able to bind anymore (eg: if another process
1763 took them since the 'disable frontend' operation). If this happens, an error
1764 is displayed. Some operating systems might not be able to resume a frontend
1765 which was disabled.
1766
1767 The frontend may be specified either by its name or by its numeric ID,
1768 prefixed with a sharp ('#').
1769
1770 This command is restricted and can only be issued on sockets configured for
1771 level "admin".
1772
1773enable health <backend>/<server>
1774 Resume a primary health check that was temporarily stopped. This will enable
1775 sending of health checks again. Please see "disable health" for details.
1776
1777 This command is restricted and can only be issued on sockets configured for
1778 level "admin".
1779
1780enable server <backend>/<server>
1781 If the server was previously marked as DOWN for maintenance, this marks the
1782 server UP and checks are re-enabled.
1783
1784 Both the backend and the server may be specified either by their name or by
1785 their numeric ID, prefixed with a sharp ('#').
1786
1787 This command is restricted and can only be issued on sockets configured for
1788 level "admin".
1789
Amaury Denoyelle18487fb2021-03-18 15:32:53 +01001790experimental-mode [on|off]
1791 Without options, this indicates whether the experimental mode is enabled or
1792 disabled on the current connection. When passed "on", it turns the
1793 experimental mode on for the current CLI connection only. With "off" it turns
1794 it off.
1795
1796 The experimental mode is used to access to extra features still in
1797 development. These features are currently not stable and should be used with
Ilya Shipitsinba13f162021-03-19 22:21:44 +05001798 care. They may be subject to breaking changes across versions.
Amaury Denoyelle18487fb2021-03-18 15:32:53 +01001799
Willy Tarreauabb9f9b2019-10-24 17:55:53 +02001800expert-mode [on|off]
Amaury Denoyelle18487fb2021-03-18 15:32:53 +01001801 This command is similar to experimental-mode but is used to toggle the
1802 expert mode.
1803
1804 The expert mode enables displaying of expert commands that can be extremely
Willy Tarreauabb9f9b2019-10-24 17:55:53 +02001805 dangerous for the process and which may occasionally help developers collect
1806 important information about complex bugs. Any misuse of these features will
1807 likely lead to a process crash. Do not use this option without being invited
1808 to do so. Note that this command is purposely not listed in the help message.
1809 This command is only accessible in admin level. Changing to another level
1810 automatically resets the expert mode.
1811
Willy Tarreau44aed902015-10-13 14:45:29 +02001812get map <map> <value>
1813get acl <acl> <value>
1814 Lookup the value <value> in the map <map> or in the ACL <acl>. <map> or <acl>
1815 are the #<id> or the <file> returned by "show map" or "show acl". This command
1816 returns all the matching patterns associated with this map. This is useful for
1817 debugging maps and ACLs. The output format is composed by one line par
1818 matching type. Each line is composed by space-delimited series of words.
1819
1820 The first two words are:
1821
1822 <match method>: The match method applied. It can be "found", "bool",
1823 "int", "ip", "bin", "len", "str", "beg", "sub", "dir",
1824 "dom", "end" or "reg".
1825
1826 <match result>: The result. Can be "match" or "no-match".
1827
1828 The following words are returned only if the pattern matches an entry.
1829
1830 <index type>: "tree" or "list". The internal lookup algorithm.
1831
1832 <case>: "case-insensitive" or "case-sensitive". The
1833 interpretation of the case.
1834
1835 <entry matched>: match="<entry>". Return the matched pattern. It is
1836 useful with regular expressions.
1837
1838 The two last word are used to show the returned value and its type. With the
1839 "acl" case, the pattern doesn't exist.
1840
1841 return=nothing: No return because there are no "map".
1842 return="<value>": The value returned in the string format.
1843 return=cannot-display: The value cannot be converted as string.
1844
1845 type="<type>": The type of the returned sample.
1846
Willy Tarreauc35eb382021-03-26 14:51:31 +01001847get var <name>
1848 Show the existence, type and contents of the process-wide variable 'name'.
1849 Only process-wide variables are readable, so the name must begin with
1850 'proc.' otherwise no variable will be found. This command requires levels
1851 "operator" or "admin".
1852
Willy Tarreau44aed902015-10-13 14:45:29 +02001853get weight <backend>/<server>
1854 Report the current weight and the initial weight of server <server> in
1855 backend <backend> or an error if either doesn't exist. The initial weight is
1856 the one that appears in the configuration file. Both are normally equal
1857 unless the current weight has been changed. Both the backend and the server
1858 may be specified either by their name or by their numeric ID, prefixed with a
1859 sharp ('#').
1860
Willy Tarreau0b1b8302021-05-09 20:59:23 +02001861help [<command>]
1862 Print the list of known keywords and their basic usage, or commands matching
1863 the requested one. The same help screen is also displayed for unknown
1864 commands.
Willy Tarreau44aed902015-10-13 14:45:29 +02001865
William Lallemandaccac232020-04-02 17:42:51 +02001866new ssl cert <filename>
1867 Create a new empty SSL certificate store to be filled with a certificate and
1868 added to a directory or a crt-list. This command should be used in
1869 combination with "set ssl cert" and "add ssl crt-list".
1870
Willy Tarreau97218ce2021-04-30 14:57:03 +02001871prepare acl <acl>
1872 Allocate a new version number in ACL <acl> for atomic replacement. <acl> is
1873 the #<id> or the <file> returned by "show acl". The new version number is
1874 shown in response after "New version created:". This number will then be
1875 usable to prepare additions of new entries into the ACL which will then
1876 atomically replace the current ones once committed. It is reported as
1877 "next_ver" in "show acl". There is no impact of allocating new versions, as
1878 unused versions will automatically be removed once a more recent version is
1879 committed. Version numbers are unsigned 32-bit values which wrap at the end,
1880 so care must be taken when comparing them in an external program. This
1881 command cannot be used if the reference <acl> is a file also used as a map.
1882 In this case, the "prepare map" command must be used instead.
1883
1884prepare map <map>
1885 Allocate a new version number in map <map> for atomic replacement. <map> is
1886 the #<id> or the <file> returned by "show map". The new version number is
1887 shown in response after "New version created:". This number will then be
1888 usable to prepare additions of new entries into the map which will then
1889 atomically replace the current ones once committed. It is reported as
1890 "next_ver" in "show map". There is no impact of allocating new versions, as
1891 unused versions will automatically be removed once a more recent version is
1892 committed. Version numbers are unsigned 32-bit values which wrap at the end,
1893 so care must be taken when comparing them in an external program.
1894
Willy Tarreau44aed902015-10-13 14:45:29 +02001895prompt
1896 Toggle the prompt at the beginning of the line and enter or leave interactive
1897 mode. In interactive mode, the connection is not closed after a command
1898 completes. Instead, the prompt will appear again, indicating the user that
1899 the interpreter is waiting for a new command. The prompt consists in a right
1900 angle bracket followed by a space "> ". This mode is particularly convenient
1901 when one wants to periodically check information such as stats or errors.
1902 It is also a good idea to enter interactive mode before issuing a "help"
1903 command.
1904
1905quit
1906 Close the connection when in interactive mode.
1907
Olivier Houchard614f8d72017-03-14 20:08:46 +01001908set dynamic-cookie-key backend <backend> <value>
1909 Modify the secret key used to generate the dynamic persistent cookies.
1910 This will break the existing sessions.
1911
Willy Tarreau44aed902015-10-13 14:45:29 +02001912set map <map> [<key>|#<ref>] <value>
1913 Modify the value corresponding to each key <key> in a map <map>. <map> is the
1914 #<id> or <file> returned by "show map". If the <ref> is used in place of
1915 <key>, only the entry pointed by <ref> is changed. The new value is <value>.
1916
1917set maxconn frontend <frontend> <value>
1918 Dynamically change the specified frontend's maxconn setting. Any positive
1919 value is allowed including zero, but setting values larger than the global
1920 maxconn does not make much sense. If the limit is increased and connections
1921 were pending, they will immediately be accepted. If it is lowered to a value
1922 below the current number of connections, new connections acceptation will be
1923 delayed until the threshold is reached. The frontend might be specified by
1924 either its name or its numeric ID prefixed with a sharp ('#').
1925
Andrew Hayworthedb93a72015-10-27 21:46:25 +00001926set maxconn server <backend/server> <value>
1927 Dynamically change the specified server's maxconn setting. Any positive
1928 value is allowed including zero, but setting values larger than the global
1929 maxconn does not make much sense.
1930
Willy Tarreau44aed902015-10-13 14:45:29 +02001931set maxconn global <maxconn>
1932 Dynamically change the global maxconn setting within the range defined by the
1933 initial global maxconn setting. If it is increased and connections were
1934 pending, they will immediately be accepted. If it is lowered to a value below
1935 the current number of connections, new connections acceptation will be
1936 delayed until the threshold is reached. A value of zero restores the initial
1937 setting.
1938
Willy Tarreau00dd44f2021-05-05 16:44:23 +02001939set profiling { tasks | memory } { auto | on | off }
1940 Enables or disables CPU or memory profiling for the indicated subsystem. This
1941 is equivalent to setting or clearing the "profiling" settings in the "global"
Willy Tarreaucfa71012021-01-29 11:56:21 +01001942 section of the configuration file. Please also see "show profiling". Note
1943 that manually setting the tasks profiling to "on" automatically resets the
1944 scheduler statistics, thus allows to check activity over a given interval.
Willy Tarreau00dd44f2021-05-05 16:44:23 +02001945 The memory profiling is limited to certain operating systems (known to work
1946 on the linux-glibc target), and requires USE_MEMORY_PROFILING to be set at
1947 compile time.
Willy Tarreau75c62c22018-11-22 11:02:09 +01001948
Willy Tarreau44aed902015-10-13 14:45:29 +02001949set rate-limit connections global <value>
1950 Change the process-wide connection rate limit, which is set by the global
1951 'maxconnrate' setting. A value of zero disables the limitation. This limit
1952 applies to all frontends and the change has an immediate effect. The value
1953 is passed in number of connections per second.
1954
1955set rate-limit http-compression global <value>
1956 Change the maximum input compression rate, which is set by the global
1957 'maxcomprate' setting. A value of zero disables the limitation. The value is
1958 passed in number of kilobytes per second. The value is available in the "show
1959 info" on the line "CompressBpsRateLim" in bytes.
1960
1961set rate-limit sessions global <value>
1962 Change the process-wide session rate limit, which is set by the global
1963 'maxsessrate' setting. A value of zero disables the limitation. This limit
1964 applies to all frontends and the change has an immediate effect. The value
1965 is passed in number of sessions per second.
1966
1967set rate-limit ssl-sessions global <value>
1968 Change the process-wide SSL session rate limit, which is set by the global
1969 'maxsslrate' setting. A value of zero disables the limitation. This limit
1970 applies to all frontends and the change has an immediate effect. The value
1971 is passed in number of sessions per second sent to the SSL stack. It applies
1972 before the handshake in order to protect the stack against handshake abuses.
1973
Baptiste Assmann3749ebf2016-08-03 22:34:12 +02001974set server <backend>/<server> addr <ip4 or ip6 address> [port <port>]
Willy Tarreau44aed902015-10-13 14:45:29 +02001975 Replace the current IP address of a server by the one provided.
Michael Prokop4438c602019-05-24 10:25:45 +02001976 Optionally, the port can be changed using the 'port' parameter.
Baptiste Assmann3749ebf2016-08-03 22:34:12 +02001977 Note that changing the port also support switching from/to port mapping
1978 (notation with +X or -Y), only if a port is configured for the health check.
Willy Tarreau44aed902015-10-13 14:45:29 +02001979
1980set server <backend>/<server> agent [ up | down ]
1981 Force a server's agent to a new state. This can be useful to immediately
1982 switch a server's state regardless of some slow agent checks for example.
1983 Note that the change is propagated to tracking servers if any.
1984
William Dauchy7cabc062021-02-11 22:51:24 +01001985set server <backend>/<server> agent-addr <addr> [port <port>]
Misiek43972902017-01-09 09:53:06 +01001986 Change addr for servers agent checks. Allows to migrate agent-checks to
1987 another address at runtime. You can specify both IP and hostname, it will be
1988 resolved.
William Dauchy7cabc062021-02-11 22:51:24 +01001989 Optionally, change the port agent.
1990
1991set server <backend>/<server> agent-port <port>
1992 Change the port used for agent checks.
Misiek43972902017-01-09 09:53:06 +01001993
1994set server <backend>/<server> agent-send <value>
1995 Change agent string sent to agent check target. Allows to update string while
1996 changing server address to keep those two matching.
1997
Willy Tarreau44aed902015-10-13 14:45:29 +02001998set server <backend>/<server> health [ up | stopping | down ]
1999 Force a server's health to a new state. This can be useful to immediately
2000 switch a server's state regardless of some slow health checks for example.
2001 Note that the change is propagated to tracking servers if any.
2002
William Dauchyb456e1f2021-02-11 22:51:23 +01002003set server <backend>/<server> check-addr <ip4 | ip6> [port <port>]
2004 Change the IP address used for server health checks.
2005 Optionally, change the port used for server health checks.
2006
Baptiste Assmann50946562016-08-31 23:26:29 +02002007set server <backend>/<server> check-port <port>
2008 Change the port used for health checking to <port>
2009
Willy Tarreau44aed902015-10-13 14:45:29 +02002010set server <backend>/<server> state [ ready | drain | maint ]
2011 Force a server's administrative state to a new state. This can be useful to
2012 disable load balancing and/or any traffic to a server. Setting the state to
2013 "ready" puts the server in normal mode, and the command is the equivalent of
2014 the "enable server" command. Setting the state to "maint" disables any traffic
2015 to the server as well as any health checks. This is the equivalent of the
2016 "disable server" command. Setting the mode to "drain" only removes the server
2017 from load balancing but still allows it to be checked and to accept new
2018 persistent connections. Changes are propagated to tracking servers if any.
2019
2020set server <backend>/<server> weight <weight>[%]
2021 Change a server's weight to the value passed in argument. This is the exact
2022 equivalent of the "set weight" command below.
2023
Frédéric Lécailleb418c122017-04-26 11:24:02 +02002024set server <backend>/<server> fqdn <FQDN>
Lukas Tribusc5dd5a52018-08-14 11:39:35 +02002025 Change a server's FQDN to the value passed in argument. This requires the
2026 internal run-time DNS resolver to be configured and enabled for this server.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02002027
William Dauchyf6370442020-11-14 19:25:33 +01002028set server <backend>/<server> ssl [ on | off ]
2029 This option configures SSL ciphering on outgoing connections to the server.
William Dauchyf5186442022-01-06 16:57:15 +01002030 When switch off, all traffic becomes plain text; health check path is not
2031 changed.
William Dauchyf6370442020-11-14 19:25:33 +01002032
Andjelko Iharosc4df59e2017-07-20 11:59:48 +02002033set severity-output [ none | number | string ]
2034 Change the severity output format of the stats socket connected to for the
2035 duration of the current session.
2036
William Lallemand6ab08b32019-11-29 16:48:43 +01002037set ssl cert <filename> <payload>
2038 This command is part of a transaction system, the "commit ssl cert" and
2039 "abort ssl cert" commands could be required.
Remi Tricot-Le Breton34459092021-04-14 16:19:28 +02002040 This whole transaction system works on any certificate displayed by the
Remi Tricot-Le Bretonb5f0fac2021-04-14 16:19:29 +02002041 "show ssl cert" command, so on any frontend or backend certificate.
William Lallemand6ab08b32019-11-29 16:48:43 +01002042 If there is no on-going transaction, it will duplicate the certificate
2043 <filename> in memory to a temporary transaction, then update this
2044 transaction with the PEM file in the payload. If a transaction exists with
2045 the same filename, it will update this transaction. It's also possible to
2046 update the files linked to a certificate (.issuer, .sctl, .oscp etc.)
2047 Once the modification are done, you have to "commit ssl cert" the
2048 transaction.
2049
William Lallemand82734752021-09-16 17:30:51 +02002050 Injection of files over the CLI must be done with caution since an empty line
2051 is used to notify the end of the payload. It is recommended to inject a PEM
2052 file which has been sanitized. A simple method would be to remove every empty
2053 line and only leave what are in the PEM sections. It could be achieved with a
2054 sed command.
2055
William Lallemand6ab08b32019-11-29 16:48:43 +01002056 Example:
William Lallemand82734752021-09-16 17:30:51 +02002057
2058 # With some simple sanitizing
2059 echo -e "set ssl cert localhost.pem <<\n$(sed -n '/^$/d;/-BEGIN/,/-END/p' 127.0.0.1.pem)\n" | \
2060 socat /var/run/haproxy.stat -
2061
2062 # Complete example with commit
William Lallemand6ab08b32019-11-29 16:48:43 +01002063 echo -e "set ssl cert localhost.pem <<\n$(cat 127.0.0.1.pem)\n" | \
2064 socat /var/run/haproxy.stat -
2065 echo -e \
2066 "set ssl cert localhost.pem.issuer <<\n $(cat 127.0.0.1.pem.issuer)\n" | \
2067 socat /var/run/haproxy.stat -
2068 echo -e \
2069 "set ssl cert localhost.pem.ocsp <<\n$(base64 -w 1000 127.0.0.1.pem.ocsp)\n" | \
2070 socat /var/run/haproxy.stat -
2071 echo "commit ssl cert localhost.pem" | socat /var/run/haproxy.stat -
2072
Aurélien Nephtali1e0867c2018-04-18 14:04:58 +02002073set ssl ocsp-response <response | payload>
Willy Tarreau44aed902015-10-13 14:45:29 +02002074 This command is used to update an OCSP Response for a certificate (see "crt"
2075 on "bind" lines). Same controls are performed as during the initial loading of
2076 the response. The <response> must be passed as a base64 encoded string of the
Emmanuel Hocdet2c32d8f2017-05-22 14:58:00 +02002077 DER encoded response from the OCSP server. This command is not supported with
2078 BoringSSL.
Willy Tarreau44aed902015-10-13 14:45:29 +02002079
2080 Example:
2081 openssl ocsp -issuer issuer.pem -cert server.pem \
2082 -host ocsp.issuer.com:80 -respout resp.der
2083 echo "set ssl ocsp-response $(base64 -w 10000 resp.der)" | \
2084 socat stdio /var/run/haproxy.stat
2085
Aurélien Nephtali1e0867c2018-04-18 14:04:58 +02002086 using the payload syntax:
2087 echo -e "set ssl ocsp-response <<\n$(base64 resp.der)\n" | \
2088 socat stdio /var/run/haproxy.stat
2089
Willy Tarreau44aed902015-10-13 14:45:29 +02002090set ssl tls-key <id> <tlskey>
2091 Set the next TLS key for the <id> listener to <tlskey>. This key becomes the
2092 ultimate key, while the penultimate one is used for encryption (others just
2093 decrypt). The oldest TLS key present is overwritten. <id> is either a numeric
2094 #<id> or <file> returned by "show tls-keys". <tlskey> is a base64 encoded 48
Emeric Brun9e754772019-01-10 17:51:55 +01002095 or 80 bits TLS ticket key (ex. openssl rand 80 | openssl base64 -A).
Willy Tarreau44aed902015-10-13 14:45:29 +02002096
2097set table <table> key <key> [data.<data_type> <value>]*
2098 Create or update a stick-table entry in the table. If the key is not present,
2099 an entry is inserted. See stick-table in section 4.2 to find all possible
2100 values for <data_type>. The most likely use consists in dynamically entering
2101 entries for source IP addresses, with a flag in gpc0 to dynamically block an
2102 IP address or affect its quality of service. It is possible to pass multiple
2103 data_types in a single call.
2104
2105set timeout cli <delay>
2106 Change the CLI interface timeout for current connection. This can be useful
2107 during long debugging sessions where the user needs to constantly inspect
2108 some indicators without being disconnected. The delay is passed in seconds.
2109
Willy Tarreau4000ff02021-04-30 14:45:53 +02002110set var <name> <expression>
2111 Allows to set or overwrite the process-wide variable 'name' with the result
2112 of expression <expression>. Only process-wide variables may be used, so the
2113 name must begin with 'proc.' otherwise no variable will be set. The
2114 <expression> may only involve "internal" sample fetch keywords and converters
2115 even though the most likely useful ones will be str('something') or int().
2116 Note that the command line parser doesn't know about quotes, so any space in
2117 the expression must be preceded by a backslash. This command requires levels
2118 "operator" or "admin". This command is only supported on a CLI connection
2119 running in experimental mode (see "experimental-mode on").
2120
Willy Tarreau44aed902015-10-13 14:45:29 +02002121set weight <backend>/<server> <weight>[%]
2122 Change a server's weight to the value passed in argument. If the value ends
2123 with the '%' sign, then the new weight will be relative to the initially
2124 configured weight. Absolute weights are permitted between 0 and 256.
2125 Relative weights must be positive with the resulting absolute weight is
2126 capped at 256. Servers which are part of a farm running a static
2127 load-balancing algorithm have stricter limitations because the weight
2128 cannot change once set. Thus for these servers, the only accepted values
2129 are 0 and 100% (or 0 and the initial weight). Changes take effect
2130 immediately, though certain LB algorithms require a certain amount of
2131 requests to consider changes. A typical usage of this command is to
2132 disable a server during an update by setting its weight to zero, then to
2133 enable it again after the update by setting it back to 100%. This command
2134 is restricted and can only be issued on sockets configured for level
2135 "admin". Both the backend and the server may be specified either by their
2136 name or by their numeric ID, prefixed with a sharp ('#').
2137
Willy Tarreau95f753e2021-04-30 12:09:54 +02002138show acl [[@<ver>] <acl>]
Willy Tarreaud6129fc2017-07-28 16:52:23 +02002139 Dump info about acl converters. Without argument, the list of all available
Willy Tarreau95f753e2021-04-30 12:09:54 +02002140 acls is returned. If a <acl> is specified, its contents are dumped. <acl> is
2141 the #<id> or <file>. By default the current version of the ACL is shown (the
2142 version currently being matched against and reported as 'curr_ver' in the ACL
2143 list). It is possible to instead dump other versions by prepending '@<ver>'
2144 before the ACL's identifier. The version works as a filter and non-existing
2145 versions will simply report no result. The dump format is the same as for the
2146 maps even for the sample values. The data returned are not a list of
2147 available ACL, but are the list of all patterns composing any ACL. Many of
2148 these patterns can be shared with maps.
Willy Tarreaud6129fc2017-07-28 16:52:23 +02002149
2150show backend
2151 Dump the list of backends available in the running process
2152
William Lallemand67a234f2018-12-13 09:05:45 +01002153show cli level
2154 Display the CLI level of the current CLI session. The result could be
2155 'admin', 'operator' or 'user'. See also the 'operator' and 'user' commands.
2156
2157 Example :
2158
2159 $ socat /tmp/sock1 readline
2160 prompt
2161 > operator
2162 > show cli level
2163 operator
2164 > user
2165 > show cli level
2166 user
2167 > operator
2168 Permission denied
2169
2170operator
2171 Decrease the CLI level of the current CLI session to operator. It can't be
Amaury Denoyelle18487fb2021-03-18 15:32:53 +01002172 increased. It also drops expert and experimental mode. See also "show cli
2173 level".
William Lallemand67a234f2018-12-13 09:05:45 +01002174
2175user
2176 Decrease the CLI level of the current CLI session to user. It can't be
Amaury Denoyelle18487fb2021-03-18 15:32:53 +01002177 increased. It also drops expert and experimental mode. See also "show cli
2178 level".
William Lallemand67a234f2018-12-13 09:05:45 +01002179
Willy Tarreau4c356932019-05-16 17:39:32 +02002180show activity
2181 Reports some counters about internal events that will help developers and
2182 more generally people who know haproxy well enough to narrow down the causes
2183 of reports of abnormal behaviours. A typical example would be a properly
2184 running process never sleeping and eating 100% of the CPU. The output fields
2185 will be made of one line per metric, and per-thread counters on the same
Thayne McCombscdbcca92021-01-07 21:24:41 -07002186 line. These counters are 32-bit and will wrap during the process's life, which
Willy Tarreau4c356932019-05-16 17:39:32 +02002187 is not a problem since calls to this command will typically be performed
2188 twice. The fields are purposely not documented so that their exact meaning is
2189 verified in the code where the counters are fed. These values are also reset
2190 by the "clear counters" command.
2191
William Lallemand51132162016-12-16 16:38:58 +01002192show cli sockets
2193 List CLI sockets. The output format is composed of 3 fields separated by
2194 spaces. The first field is the socket address, it can be a unix socket, a
2195 ipv4 address:port couple or a ipv6 one. Socket of other types won't be dump.
2196 The second field describe the level of the socket: 'admin', 'user' or
2197 'operator'. The last field list the processes on which the socket is bound,
2198 separated by commas, it can be numbers or 'all'.
2199
2200 Example :
2201
2202 $ echo 'show cli sockets' | socat stdio /tmp/sock1
2203 # socket lvl processes
2204 /tmp/sock1 admin all
2205 127.0.0.1:9999 user 2,3,4
2206 127.0.0.2:9969 user 2
2207 [::1]:9999 operator 2
2208
William Lallemand86d0df02017-11-24 21:36:45 +01002209show cache
Cyril Bonté7b888f12017-11-26 22:24:31 +01002210 List the configured caches and the objects stored in each cache tree.
William Lallemand86d0df02017-11-24 21:36:45 +01002211
2212 $ echo 'show cache' | socat stdio /tmp/sock1
2213 0x7f6ac6c5b03a: foobar (shctx:0x7f6ac6c5b000, available blocks:3918)
2214 1 2 3 4
2215
2216 1. pointer to the cache structure
2217 2. cache name
2218 3. pointer to the mmap area (shctx)
2219 4. number of blocks available for reuse in the shctx
2220
Remi Tricot-Le Bretone3e1e5f2020-11-27 15:48:40 +01002221 0x7f6ac6c5b4cc hash:286881868 vary:0x0011223344556677 size:39114 (39 blocks), refcount:9, expire:237
2222 1 2 3 4 5 6 7
William Lallemand86d0df02017-11-24 21:36:45 +01002223
2224 1. pointer to the cache entry
2225 2. first 32 bits of the hash
Remi Tricot-Le Bretone3e1e5f2020-11-27 15:48:40 +01002226 3. secondary hash of the entry in case of vary
2227 4. size of the object in bytes
2228 5. number of blocks used for the object
2229 6. number of transactions using the entry
2230 7. expiration time, can be negative if already expired
William Lallemand86d0df02017-11-24 21:36:45 +01002231
Willy Tarreauae795722016-02-16 11:27:28 +01002232show env [<name>]
2233 Dump one or all environment variables known by the process. Without any
2234 argument, all variables are dumped. With an argument, only the specified
2235 variable is dumped if it exists. Otherwise "Variable not found" is emitted.
2236 Variables are dumped in the same format as they are stored or returned by the
2237 "env" utility, that is, "<name>=<value>". This can be handy when debugging
2238 certain configuration files making heavy use of environment variables to
2239 ensure that they contain the expected values. This command is restricted and
2240 can only be issued on sockets configured for levels "operator" or "admin".
2241
Willy Tarreau35069f82016-11-25 09:16:37 +01002242show errors [<iid>|<proxy>] [request|response]
Willy Tarreau44aed902015-10-13 14:45:29 +02002243 Dump last known request and response errors collected by frontends and
2244 backends. If <iid> is specified, the limit the dump to errors concerning
Willy Tarreau234ba2d2016-11-25 08:39:10 +01002245 either frontend or backend whose ID is <iid>. Proxy ID "-1" will cause
2246 all instances to be dumped. If a proxy name is specified instead, its ID
Willy Tarreau35069f82016-11-25 09:16:37 +01002247 will be used as the filter. If "request" or "response" is added after the
2248 proxy name or ID, only request or response errors will be dumped. This
2249 command is restricted and can only be issued on sockets configured for
2250 levels "operator" or "admin".
Willy Tarreau44aed902015-10-13 14:45:29 +02002251
2252 The errors which may be collected are the last request and response errors
2253 caused by protocol violations, often due to invalid characters in header
2254 names. The report precisely indicates what exact character violated the
2255 protocol. Other important information such as the exact date the error was
2256 detected, frontend and backend names, the server name (when known), the
2257 internal session ID and the source address which has initiated the session
2258 are reported too.
2259
2260 All characters are returned, and non-printable characters are encoded. The
2261 most common ones (\t = 9, \n = 10, \r = 13 and \e = 27) are encoded as one
2262 letter following a backslash. The backslash itself is encoded as '\\' to
2263 avoid confusion. Other non-printable characters are encoded '\xNN' where
2264 NN is the two-digits hexadecimal representation of the character's ASCII
2265 code.
2266
2267 Lines are prefixed with the position of their first character, starting at 0
2268 for the beginning of the buffer. At most one input line is printed per line,
2269 and large lines will be broken into multiple consecutive output lines so that
2270 the output never goes beyond 79 characters wide. It is easy to detect if a
2271 line was broken, because it will not end with '\n' and the next line's offset
2272 will be followed by a '+' sign, indicating it is a continuation of previous
2273 line.
2274
2275 Example :
Willy Tarreau35069f82016-11-25 09:16:37 +01002276 $ echo "show errors -1 response" | socat stdio /tmp/sock1
Willy Tarreau44aed902015-10-13 14:45:29 +02002277 >>> [04/Mar/2009:15:46:56.081] backend http-in (#2) : invalid response
2278 src 127.0.0.1, session #54, frontend fe-eth0 (#1), server s2 (#1)
2279 response length 213 bytes, error at position 23:
2280
2281 00000 HTTP/1.0 200 OK\r\n
2282 00017 header/bizarre:blah\r\n
2283 00038 Location: blah\r\n
2284 00054 Long-line: this is a very long line which should b
2285 00104+ e broken into multiple lines on the output buffer,
2286 00154+ otherwise it would be too large to print in a ter
2287 00204+ minal\r\n
2288 00211 \r\n
2289
2290 In the example above, we see that the backend "http-in" which has internal
2291 ID 2 has blocked an invalid response from its server s2 which has internal
2292 ID 1. The request was on session 54 initiated by source 127.0.0.1 and
2293 received by frontend fe-eth0 whose ID is 1. The total response length was
2294 213 bytes when the error was detected, and the error was at byte 23. This
2295 is the slash ('/') in header name "header/bizarre", which is not a valid
2296 HTTP character for a header name.
2297
Willy Tarreau1d181e42019-08-30 11:17:01 +02002298show events [<sink>] [-w] [-n]
Willy Tarreau9f830d72019-08-26 18:17:04 +02002299 With no option, this lists all known event sinks and their types. With an
2300 option, it will dump all available events in the designated sink if it is of
Willy Tarreau1d181e42019-08-30 11:17:01 +02002301 type buffer. If option "-w" is passed after the sink name, then once the end
2302 of the buffer is reached, the command will wait for new events and display
2303 them. It is possible to stop the operation by entering any input (which will
2304 be discarded) or by closing the session. Finally, option "-n" is used to
2305 directly seek to the end of the buffer, which is often convenient when
2306 combined with "-w" to only report new events. For convenience, "-wn" or "-nw"
2307 may be used to enable both options at once.
Willy Tarreau9f830d72019-08-26 18:17:04 +02002308
Willy Tarreau7a4a0ac2017-07-25 19:32:50 +02002309show fd [<fd>]
2310 Dump the list of either all open file descriptors or just the one number <fd>
2311 if specified. This is only aimed at developers who need to observe internal
2312 states in order to debug complex issues such as abnormal CPU usages. One fd
2313 is reported per lines, and for each of them, its state in the poller using
2314 upper case letters for enabled flags and lower case for disabled flags, using
2315 "P" for "polled", "R" for "ready", "A" for "active", the events status using
2316 "H" for "hangup", "E" for "error", "O" for "output", "P" for "priority" and
2317 "I" for "input", a few other flags like "N" for "new" (just added into the fd
2318 cache), "U" for "updated" (received an update in the fd cache), "L" for
2319 "linger_risk", "C" for "cloned", then the cached entry position, the pointer
2320 to the internal owner, the pointer to the I/O callback and its name when
2321 known. When the owner is a connection, the connection flags, and the target
2322 are reported (frontend, proxy or server). When the owner is a listener, the
2323 listener's state and its frontend are reported. There is no point in using
2324 this command without a good knowledge of the internals. It's worth noting
2325 that the output format may evolve over time so this output must not be parsed
Willy Tarreau8050efe2021-01-21 08:26:06 +01002326 by tools designed to be durable. Some internal structure states may look
2327 suspicious to the function listing them, in this case the output line will be
2328 suffixed with an exclamation mark ('!'). This may help find a starting point
2329 when trying to diagnose an incident.
Willy Tarreau7a4a0ac2017-07-25 19:32:50 +02002330
Willy Tarreau27456202021-05-08 07:54:24 +02002331show info [typed|json] [desc] [float]
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002332 Dump info about haproxy status on current process. If "typed" is passed as an
2333 optional argument, field numbers, names and types are emitted as well so that
2334 external monitoring products can easily retrieve, possibly aggregate, then
2335 report information found in fields they don't know. Each field is dumped on
Simon Horman05ee2132017-01-04 09:37:25 +01002336 its own line. If "json" is passed as an optional argument then
2337 information provided by "typed" output is provided in JSON format as a
2338 list of JSON objects. By default, the format contains only two columns
2339 delimited by a colon (':'). The left one is the field name and the right
2340 one is the value. It is very important to note that in typed output
2341 format, the dump for a single object is contiguous so that there is no
Willy Tarreau27456202021-05-08 07:54:24 +02002342 need for a consumer to store everything at once. If "float" is passed as an
2343 optional argument, some fields usually emitted as integers may switch to
2344 floats for higher accuracy. It is purposely unspecified which ones are
2345 concerned as this might evolve over time. Using this option implies that the
2346 consumer is able to process floats. The output format used is sprintf("%f").
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002347
2348 When using the typed output format, each line is made of 4 columns delimited
2349 by colons (':'). The first column is a dot-delimited series of 3 elements. The
2350 first element is the numeric position of the field in the list (starting at
2351 zero). This position shall not change over time, but holes are to be expected,
2352 depending on build options or if some fields are deleted in the future. The
2353 second element is the field name as it appears in the default "show info"
2354 output. The third element is the relative process number starting at 1.
2355
2356 The rest of the line starting after the first colon follows the "typed output
2357 format" described in the section above. In short, the second column (after the
2358 first ':') indicates the origin, nature and scope of the variable. The third
2359 column indicates the type of the field, among "s32", "s64", "u32", "u64" and
2360 "str". Then the fourth column is the value itself, which the consumer knows
2361 how to parse thanks to column 3 and how to process thanks to column 2.
2362
2363 Thus the overall line format in typed mode is :
2364
2365 <field_pos>.<field_name>.<process_num>:<tags>:<type>:<value>
2366
Willy Tarreau6b19b142019-10-09 15:44:21 +02002367 When "desc" is appended to the command, one extra colon followed by a quoted
2368 string is appended with a description for the metric. At the time of writing,
2369 this is only supported for the "typed" and default output formats.
2370
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002371 Example :
2372
2373 > show info
2374 Name: HAProxy
2375 Version: 1.7-dev1-de52ea-146
2376 Release_date: 2016/03/11
2377 Nbproc: 1
2378 Process_num: 1
2379 Pid: 28105
2380 Uptime: 0d 0h00m04s
2381 Uptime_sec: 4
2382 Memmax_MB: 0
2383 PoolAlloc_MB: 0
2384 PoolUsed_MB: 0
2385 PoolFailed: 0
2386 (...)
2387
2388 > show info typed
2389 0.Name.1:POS:str:HAProxy
2390 1.Version.1:POS:str:1.7-dev1-de52ea-146
2391 2.Release_date.1:POS:str:2016/03/11
2392 3.Nbproc.1:CGS:u32:1
2393 4.Process_num.1:KGP:u32:1
2394 5.Pid.1:SGP:u32:28105
2395 6.Uptime.1:MDP:str:0d 0h00m08s
2396 7.Uptime_sec.1:MDP:u32:8
2397 8.Memmax_MB.1:CLP:u32:0
2398 9.PoolAlloc_MB.1:MGP:u32:0
2399 10.PoolUsed_MB.1:MGP:u32:0
2400 11.PoolFailed.1:MCP:u32:0
2401 (...)
2402
Simon Horman1084a362016-11-21 17:00:24 +01002403 In the typed format, the presence of the process ID at the end of the
2404 first column makes it very easy to visually aggregate outputs from
2405 multiple processes.
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002406 Example :
2407
2408 $ ( echo show info typed | socat /var/run/haproxy.sock1 ; \
2409 echo show info typed | socat /var/run/haproxy.sock2 ) | \
2410 sort -t . -k 1,1n -k 2,2 -k 3,3n
2411 0.Name.1:POS:str:HAProxy
2412 0.Name.2:POS:str:HAProxy
2413 1.Version.1:POS:str:1.7-dev1-868ab3-148
2414 1.Version.2:POS:str:1.7-dev1-868ab3-148
2415 2.Release_date.1:POS:str:2016/03/11
2416 2.Release_date.2:POS:str:2016/03/11
2417 3.Nbproc.1:CGS:u32:2
2418 3.Nbproc.2:CGS:u32:2
2419 4.Process_num.1:KGP:u32:1
2420 4.Process_num.2:KGP:u32:2
2421 5.Pid.1:SGP:u32:30120
2422 5.Pid.2:SGP:u32:30121
2423 6.Uptime.1:MDP:str:0d 0h01m28s
2424 6.Uptime.2:MDP:str:0d 0h01m28s
2425 (...)
Willy Tarreau44aed902015-10-13 14:45:29 +02002426
Simon Horman05ee2132017-01-04 09:37:25 +01002427 The format of JSON output is described in a schema which may be output
Simon Horman6f6bb382017-01-04 09:37:26 +01002428 using "show schema json".
Simon Horman05ee2132017-01-04 09:37:25 +01002429
2430 The JSON output contains no extra whitespace in order to reduce the
2431 volume of output. For human consumption passing the output through a
2432 pretty printer may be helpful. Example :
2433
2434 $ echo "show info json" | socat /var/run/haproxy.sock stdio | \
2435 python -m json.tool
2436
Simon Horman6f6bb382017-01-04 09:37:26 +01002437 The JSON output contains no extra whitespace in order to reduce the
2438 volume of output. For human consumption passing the output through a
2439 pretty printer may be helpful. Example :
2440
2441 $ echo "show info json" | socat /var/run/haproxy.sock stdio | \
2442 python -m json.tool
2443
Willy Tarreau37857332021-12-28 09:57:10 +01002444show libs
2445 Dump the list of loaded shared dynamic libraries and object files, on systems
2446 that support it. When available, for each shared object the range of virtual
2447 addresses will be indicated, the size and the path to the object. This can be
2448 used for example to try to estimate what library provides a function that
2449 appears in a dump. Note that on many systems, addresses will change upon each
2450 restart (address space randomization), so that this list would need to be
2451 retrieved upon startup if it is expected to be used to analyse a core file.
2452 This command may only be issued on sockets configured for levels "operator"
2453 or "admin". Note that the output format may vary between operating systems,
2454 architectures and even haproxy versions, and ought not to be relied on in
2455 scripts.
2456
Willy Tarreau95f753e2021-04-30 12:09:54 +02002457show map [[@<ver>] <map>]
Willy Tarreau44aed902015-10-13 14:45:29 +02002458 Dump info about map converters. Without argument, the list of all available
2459 maps is returned. If a <map> is specified, its contents are dumped. <map> is
Willy Tarreau95f753e2021-04-30 12:09:54 +02002460 the #<id> or <file>. By default the current version of the map is shown (the
2461 version currently being matched against and reported as 'curr_ver' in the map
2462 list). It is possible to instead dump other versions by prepending '@<ver>'
2463 before the map's identifier. The version works as a filter and non-existing
2464 versions will simply report no result.
2465
2466 In the output, the first column is a unique entry identifier, which is usable
2467 as a reference for operations "del map" and "set map". The second column is
Willy Tarreau44aed902015-10-13 14:45:29 +02002468 the pattern and the third column is the sample if available. The data returned
2469 are not directly a list of available maps, but are the list of all patterns
2470 composing any map. Many of these patterns can be shared with ACL.
2471
Willy Tarreau49962b52021-02-12 16:56:22 +01002472show peers [dict|-] [<peers section>]
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002473 Dump info about the peers configured in "peers" sections. Without argument,
2474 the list of the peers belonging to all the "peers" sections are listed. If
2475 <peers section> is specified, only the information about the peers belonging
Willy Tarreau49962b52021-02-12 16:56:22 +01002476 to this "peers" section are dumped. When "dict" is specified before the peers
2477 section name, the entire Tx/Rx dictionary caches will also be dumped (very
2478 large). Passing "-" may be required to dump a peers section called "dict".
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002479
Michael Prokop4438c602019-05-24 10:25:45 +02002480 Here are two examples of outputs where hostA, hostB and hostC peers belong to
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002481 "sharedlb" peers sections. Only hostA and hostB are connected. Only hostA has
2482 sent data to hostB.
2483
2484 $ echo "show peers" | socat - /tmp/hostA
2485 0x55deb0224320: [15/Apr/2019:11:28:01] id=sharedlb state=0 flags=0x3 \
Emeric Brun0bbec0f2019-04-18 11:39:43 +02002486 resync_timeout=<PAST> task_calls=45122
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002487 0x55deb022b540: id=hostC(remote) addr=127.0.0.12:10002 status=CONN \
2488 reconnect=4s confirm=0
2489 flags=0x0
2490 0x55deb022a440: id=hostA(local) addr=127.0.0.10:10000 status=NONE \
2491 reconnect=<NEVER> confirm=0
2492 flags=0x0
2493 0x55deb0227d70: id=hostB(remote) addr=127.0.0.11:10001 status=ESTA
2494 reconnect=2s confirm=0
Emeric Brun0bbec0f2019-04-18 11:39:43 +02002495 flags=0x20000200 appctx:0x55deb028fba0 st0=7 st1=0 task_calls=14456 \
2496 state=EST
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002497 xprt=RAW src=127.0.0.1:37257 addr=127.0.0.10:10000
2498 remote_table:0x55deb0224a10 id=stkt local_id=1 remote_id=1
2499 last_local_table:0x55deb0224a10 id=stkt local_id=1 remote_id=1
2500 shared tables:
2501 0x55deb0224a10 local_id=1 remote_id=1 flags=0x0 remote_data=0x65
2502 last_acked=0 last_pushed=3 last_get=0 teaching_origin=0 update=3
2503 table:0x55deb022d6a0 id=stkt update=3 localupdate=3 \
2504 commitupdate=3 syncing=0
2505
2506 $ echo "show peers" | socat - /tmp/hostB
2507 0x55871b5ab320: [15/Apr/2019:11:28:03] id=sharedlb state=0 flags=0x3 \
Emeric Brun0bbec0f2019-04-18 11:39:43 +02002508 resync_timeout=<PAST> task_calls=3
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002509 0x55871b5b2540: id=hostC(remote) addr=127.0.0.12:10002 status=CONN \
2510 reconnect=3s confirm=0
2511 flags=0x0
2512 0x55871b5b1440: id=hostB(local) addr=127.0.0.11:10001 status=NONE \
2513 reconnect=<NEVER> confirm=0
2514 flags=0x0
2515 0x55871b5aed70: id=hostA(remote) addr=127.0.0.10:10000 status=ESTA \
2516 reconnect=2s confirm=0
Emeric Brun0bbec0f2019-04-18 11:39:43 +02002517 flags=0x20000200 appctx:0x7fa46800ee00 st0=7 st1=0 task_calls=62356 \
2518 state=EST
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002519 remote_table:0x55871b5ab960 id=stkt local_id=1 remote_id=1
2520 last_local_table:0x55871b5ab960 id=stkt local_id=1 remote_id=1
2521 shared tables:
2522 0x55871b5ab960 local_id=1 remote_id=1 flags=0x0 remote_data=0x65
2523 last_acked=3 last_pushed=0 last_get=3 teaching_origin=0 update=0
2524 table:0x55871b5b46a0 id=stkt update=1 localupdate=0 \
2525 commitupdate=0 syncing=0
2526
Willy Tarreau44aed902015-10-13 14:45:29 +02002527show pools
2528 Dump the status of internal memory pools. This is useful to track memory
2529 usage when suspecting a memory leak for example. It does exactly the same
2530 as the SIGQUIT when running in foreground except that it does not flush
2531 the pools.
2532
Willy Tarreauf1c8a382021-05-13 10:00:17 +02002533show profiling [{all | status | tasks | memory}] [byaddr] [<max_lines>]
Willy Tarreau75c62c22018-11-22 11:02:09 +01002534 Dumps the current profiling settings, one per line, as well as the command
Willy Tarreau1bd67e92021-01-29 00:07:40 +01002535 needed to change them. When tasks profiling is enabled, some per-function
2536 statistics collected by the scheduler will also be emitted, with a summary
Willy Tarreau42712cb2021-05-05 17:48:13 +02002537 covering the number of calls, total/avg CPU time and total/avg latency. When
2538 memory profiling is enabled, some information such as the number of
2539 allocations/releases and their sizes will be reported. It is possible to
2540 limit the dump to only the profiling status, the tasks, or the memory
2541 profiling by specifying the respective keywords; by default all profiling
2542 information are dumped. It is also possible to limit the number of lines
Willy Tarreauf1c8a382021-05-13 10:00:17 +02002543 of output of each category by specifying a numeric limit. If is possible to
2544 request that the output is sorted by address instead of usage, e.g. to ease
2545 comparisons between subsequent calls. Please note that profiling is
2546 essentially aimed at developers since it gives hints about where CPU cycles
2547 or memory are wasted in the code. There is nothing useful to monitor there.
Willy Tarreau75c62c22018-11-22 11:02:09 +01002548
Willy Tarreau87ef3232021-01-29 12:01:46 +01002549show resolvers [<resolvers section id>]
2550 Dump statistics for the given resolvers section, or all resolvers sections
2551 if no section is supplied.
2552
2553 For each name server, the following counters are reported:
2554 sent: number of DNS requests sent to this server
2555 valid: number of DNS valid responses received from this server
2556 update: number of DNS responses used to update the server's IP address
2557 cname: number of CNAME responses
2558 cname_error: CNAME errors encountered with this server
2559 any_err: number of empty response (IE: server does not support ANY type)
2560 nx: non existent domain response received from this server
2561 timeout: how many time this server did not answer in time
2562 refused: number of requests refused by this server
2563 other: any other DNS errors
2564 invalid: invalid DNS response (from a protocol point of view)
2565 too_big: too big response
2566 outdated: number of response arrived too late (after an other name server)
2567
Willy Tarreau69f591e2020-07-01 07:00:59 +02002568show servers conn [<backend>]
2569 Dump the current and idle connections state of the servers belonging to the
2570 designated backend (or all backends if none specified). A backend name or
2571 identifier may be used.
2572
2573 The output consists in a header line showing the fields titles, then one
2574 server per line with for each, the backend name and ID, server name and ID,
2575 the address, port and a series or values. The number of fields varies
2576 depending on thread count.
2577
2578 Given the threaded nature of idle connections, it's important to understand
2579 that some values may change once read, and that as such, consistency within a
2580 line isn't granted. This output is mostly provided as a debugging tool and is
2581 not relevant to be routinely monitored nor graphed.
2582
Willy Tarreau44aed902015-10-13 14:45:29 +02002583show servers state [<backend>]
2584 Dump the state of the servers found in the running configuration. A backend
2585 name or identifier may be provided to limit the output to this backend only.
2586
2587 The dump has the following format:
2588 - first line contains the format version (1 in this specification);
2589 - second line contains the column headers, prefixed by a sharp ('#');
2590 - third line and next ones contain data;
2591 - each line starting by a sharp ('#') is considered as a comment.
2592
Dan Lloyd8e48b872016-07-01 21:01:18 -04002593 Since multiple versions of the output may co-exist, below is the list of
Willy Tarreau44aed902015-10-13 14:45:29 +02002594 fields and their order per file format version :
2595 1:
2596 be_id: Backend unique id.
2597 be_name: Backend label.
2598 srv_id: Server unique id (in the backend).
2599 srv_name: Server label.
2600 srv_addr: Server IP address.
2601 srv_op_state: Server operational state (UP/DOWN/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002602 0 = SRV_ST_STOPPED
2603 The server is down.
2604 1 = SRV_ST_STARTING
2605 The server is warming up (up but
2606 throttled).
2607 2 = SRV_ST_RUNNING
2608 The server is fully up.
2609 3 = SRV_ST_STOPPING
2610 The server is up but soft-stopping
2611 (eg: 404).
Willy Tarreau44aed902015-10-13 14:45:29 +02002612 srv_admin_state: Server administrative state (MAINT/DRAIN/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002613 The state is actually a mask of values :
2614 0x01 = SRV_ADMF_FMAINT
2615 The server was explicitly forced into
2616 maintenance.
2617 0x02 = SRV_ADMF_IMAINT
2618 The server has inherited the maintenance
2619 status from a tracked server.
2620 0x04 = SRV_ADMF_CMAINT
2621 The server is in maintenance because of
2622 the configuration.
2623 0x08 = SRV_ADMF_FDRAIN
2624 The server was explicitly forced into
2625 drain state.
2626 0x10 = SRV_ADMF_IDRAIN
2627 The server has inherited the drain status
2628 from a tracked server.
Baptiste Assmann89aa7f32016-11-02 21:31:27 +01002629 0x20 = SRV_ADMF_RMAINT
2630 The server is in maintenance because of an
2631 IP address resolution failure.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02002632 0x40 = SRV_ADMF_HMAINT
2633 The server FQDN was set from stats socket.
2634
Willy Tarreau44aed902015-10-13 14:45:29 +02002635 srv_uweight: User visible server's weight.
2636 srv_iweight: Server's initial weight.
2637 srv_time_since_last_change: Time since last operational change.
2638 srv_check_status: Last health check status.
2639 srv_check_result: Last check result (FAILED/PASSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002640 0 = CHK_RES_UNKNOWN
2641 Initialized to this by default.
2642 1 = CHK_RES_NEUTRAL
2643 Valid check but no status information.
2644 2 = CHK_RES_FAILED
2645 Check failed.
2646 3 = CHK_RES_PASSED
2647 Check succeeded and server is fully up
2648 again.
2649 4 = CHK_RES_CONDPASS
2650 Check reports the server doesn't want new
2651 sessions.
Willy Tarreau44aed902015-10-13 14:45:29 +02002652 srv_check_health: Checks rise / fall current counter.
2653 srv_check_state: State of the check (ENABLED/PAUSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002654 The state is actually a mask of values :
2655 0x01 = CHK_ST_INPROGRESS
2656 A check is currently running.
2657 0x02 = CHK_ST_CONFIGURED
2658 This check is configured and may be
2659 enabled.
2660 0x04 = CHK_ST_ENABLED
2661 This check is currently administratively
2662 enabled.
2663 0x08 = CHK_ST_PAUSED
2664 Checks are paused because of maintenance
2665 (health only).
Willy Tarreau44aed902015-10-13 14:45:29 +02002666 srv_agent_state: State of the agent check (ENABLED/PAUSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002667 This state uses the same mask values as
2668 "srv_check_state", adding this specific one :
2669 0x10 = CHK_ST_AGENT
2670 Check is an agent check (otherwise it's a
2671 health check).
Willy Tarreau44aed902015-10-13 14:45:29 +02002672 bk_f_forced_id: Flag to know if the backend ID is forced by
2673 configuration.
2674 srv_f_forced_id: Flag to know if the server's ID is forced by
2675 configuration.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02002676 srv_fqdn: Server FQDN.
Frédéric Lécaille31694712017-08-01 08:47:19 +02002677 srv_port: Server port.
Baptiste Assmann6d0f38f2018-07-02 17:00:54 +02002678 srvrecord: DNS SRV record associated to this SRV.
William Dauchyf6370442020-11-14 19:25:33 +01002679 srv_use_ssl: use ssl for server connections.
William Dauchyd1a7b852021-02-11 22:51:26 +01002680 srv_check_port: Server health check port.
2681 srv_check_addr: Server health check address.
2682 srv_agent_addr: Server health agent address.
2683 srv_agent_port: Server health agent port.
Willy Tarreau44aed902015-10-13 14:45:29 +02002684
2685show sess
2686 Dump all known sessions. Avoid doing this on slow connections as this can
2687 be huge. This command is restricted and can only be issued on sockets
Willy Tarreauc6e7a1b2020-06-28 01:24:12 +02002688 configured for levels "operator" or "admin". Note that on machines with
2689 quickly recycled connections, it is possible that this output reports less
2690 entries than really exist because it will dump all existing sessions up to
2691 the last one that was created before the command was entered; those which
2692 die in the mean time will not appear.
Willy Tarreau44aed902015-10-13 14:45:29 +02002693
2694show sess <id>
2695 Display a lot of internal information about the specified session identifier.
2696 This identifier is the first field at the beginning of the lines in the dumps
2697 of "show sess" (it corresponds to the session pointer). Those information are
2698 useless to most users but may be used by haproxy developers to troubleshoot a
2699 complex bug. The output format is intentionally not documented so that it can
2700 freely evolve depending on demands. You may find a description of all fields
2701 returned in src/dumpstats.c
2702
2703 The special id "all" dumps the states of all sessions, which must be avoided
2704 as much as possible as it is highly CPU intensive and can take a lot of time.
2705
Daniel Corbettc40edac2020-11-01 10:54:17 -05002706show stat [domain <dns|proxy>] [{<iid>|<proxy>} <type> <sid>] [typed|json] \
Willy Tarreau698097b2020-10-23 20:19:47 +02002707 [desc] [up|no-maint]
Daniel Corbettc40edac2020-11-01 10:54:17 -05002708 Dump statistics. The domain is used to select which statistics to print; dns
2709 and proxy are available for now. By default, the CSV format is used; you can
Amaury Denoyelle072f97e2020-10-05 11:49:37 +02002710 activate the extended typed output format described in the section above if
2711 "typed" is passed after the other arguments; or in JSON if "json" is passed
2712 after the other arguments. By passing <id>, <type> and <sid>, it is possible
2713 to dump only selected items :
Willy Tarreaua1b1ed52016-11-25 08:50:58 +01002714 - <iid> is a proxy ID, -1 to dump everything. Alternatively, a proxy name
2715 <proxy> may be specified. In this case, this proxy's ID will be used as
2716 the ID selector.
Willy Tarreau44aed902015-10-13 14:45:29 +02002717 - <type> selects the type of dumpable objects : 1 for frontends, 2 for
2718 backends, 4 for servers, -1 for everything. These values can be ORed,
2719 for example:
2720 1 + 2 = 3 -> frontend + backend.
2721 1 + 2 + 4 = 7 -> frontend + backend + server.
2722 - <sid> is a server ID, -1 to dump everything from the selected proxy.
2723
2724 Example :
2725 $ echo "show info;show stat" | socat stdio unix-connect:/tmp/sock1
2726 >>> Name: HAProxy
2727 Version: 1.4-dev2-49
2728 Release_date: 2009/09/23
2729 Nbproc: 1
2730 Process_num: 1
2731 (...)
2732
2733 # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq, (...)
2734 stats,FRONTEND,,,0,0,1000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,1,0, (...)
2735 stats,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,250,(...)
2736 (...)
2737 www1,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,0,250, (...)
2738
2739 $
2740
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002741 In this example, two commands have been issued at once. That way it's easy to
2742 find which process the stats apply to in multi-process mode. This is not
2743 needed in the typed output format as the process number is reported on each
2744 line. Notice the empty line after the information output which marks the end
2745 of the first block. A similar empty line appears at the end of the second
2746 block (stats) so that the reader knows the output has not been truncated.
2747
2748 When "typed" is specified, the output format is more suitable to monitoring
2749 tools because it provides numeric positions and indicates the type of each
2750 output field. Each value stands on its own line with process number, element
2751 number, nature, origin and scope. This same format is available via the HTTP
2752 stats by passing ";typed" after the URI. It is very important to note that in
Dan Lloyd8e48b872016-07-01 21:01:18 -04002753 typed output format, the dump for a single object is contiguous so that there
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002754 is no need for a consumer to store everything at once.
2755
Willy Tarreau698097b2020-10-23 20:19:47 +02002756 The "up" modifier will result in listing only servers which reportedly up or
2757 not checked. Those down, unresolved, or in maintenance will not be listed.
2758 This is analogous to the ";up" option on the HTTP stats. Similarly, the
2759 "no-maint" modifier will act like the ";no-maint" HTTP modifier and will
2760 result in disabled servers not to be listed. The difference is that those
2761 which are enabled but down will not be evicted.
2762
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002763 When using the typed output format, each line is made of 4 columns delimited
2764 by colons (':'). The first column is a dot-delimited series of 5 elements. The
2765 first element is a letter indicating the type of the object being described.
2766 At the moment the following object types are known : 'F' for a frontend, 'B'
2767 for a backend, 'L' for a listener, and 'S' for a server. The second element
2768 The second element is a positive integer representing the unique identifier of
2769 the proxy the object belongs to. It is equivalent to the "iid" column of the
2770 CSV output and matches the value in front of the optional "id" directive found
2771 in the frontend or backend section. The third element is a positive integer
2772 containing the unique object identifier inside the proxy, and corresponds to
2773 the "sid" column of the CSV output. ID 0 is reported when dumping a frontend
2774 or a backend. For a listener or a server, this corresponds to their respective
2775 ID inside the proxy. The fourth element is the numeric position of the field
2776 in the list (starting at zero). This position shall not change over time, but
2777 holes are to be expected, depending on build options or if some fields are
2778 deleted in the future. The fifth element is the field name as it appears in
2779 the CSV output. The sixth element is a positive integer and is the relative
2780 process number starting at 1.
2781
2782 The rest of the line starting after the first colon follows the "typed output
2783 format" described in the section above. In short, the second column (after the
2784 first ':') indicates the origin, nature and scope of the variable. The third
Willy Tarreau589722e2021-05-08 07:46:44 +02002785 column indicates the field type, among "s32", "s64", "u32", "u64", "flt' and
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002786 "str". Then the fourth column is the value itself, which the consumer knows
2787 how to parse thanks to column 3 and how to process thanks to column 2.
2788
Willy Tarreau6b19b142019-10-09 15:44:21 +02002789 When "desc" is appended to the command, one extra colon followed by a quoted
2790 string is appended with a description for the metric. At the time of writing,
2791 this is only supported for the "typed" output format.
2792
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002793 Thus the overall line format in typed mode is :
2794
2795 <obj>.<px_id>.<id>.<fpos>.<fname>.<process_num>:<tags>:<type>:<value>
2796
2797 Here's an example of typed output format :
2798
2799 $ echo "show stat typed" | socat stdio unix-connect:/tmp/sock1
2800 F.2.0.0.pxname.1:MGP:str:private-frontend
2801 F.2.0.1.svname.1:MGP:str:FRONTEND
2802 F.2.0.8.bin.1:MGP:u64:0
2803 F.2.0.9.bout.1:MGP:u64:0
2804 F.2.0.40.hrsp_2xx.1:MGP:u64:0
2805 L.2.1.0.pxname.1:MGP:str:private-frontend
2806 L.2.1.1.svname.1:MGP:str:sock-1
2807 L.2.1.17.status.1:MGP:str:OPEN
2808 L.2.1.73.addr.1:MGP:str:0.0.0.0:8001
2809 S.3.13.60.rtime.1:MCP:u32:0
2810 S.3.13.61.ttime.1:MCP:u32:0
2811 S.3.13.62.agent_status.1:MGP:str:L4TOUT
2812 S.3.13.64.agent_duration.1:MGP:u64:2001
2813 S.3.13.65.check_desc.1:MCP:str:Layer4 timeout
2814 S.3.13.66.agent_desc.1:MCP:str:Layer4 timeout
2815 S.3.13.67.check_rise.1:MCP:u32:2
2816 S.3.13.68.check_fall.1:MCP:u32:3
2817 S.3.13.69.check_health.1:SGP:u32:0
2818 S.3.13.70.agent_rise.1:MaP:u32:1
2819 S.3.13.71.agent_fall.1:SGP:u32:1
2820 S.3.13.72.agent_health.1:SGP:u32:1
2821 S.3.13.73.addr.1:MCP:str:1.255.255.255:8888
2822 S.3.13.75.mode.1:MAP:str:http
2823 B.3.0.0.pxname.1:MGP:str:private-backend
2824 B.3.0.1.svname.1:MGP:str:BACKEND
2825 B.3.0.2.qcur.1:MGP:u32:0
2826 B.3.0.3.qmax.1:MGP:u32:0
2827 B.3.0.4.scur.1:MGP:u32:0
2828 B.3.0.5.smax.1:MGP:u32:0
2829 B.3.0.6.slim.1:MGP:u32:1000
2830 B.3.0.55.lastsess.1:MMP:s32:-1
2831 (...)
2832
Simon Horman1084a362016-11-21 17:00:24 +01002833 In the typed format, the presence of the process ID at the end of the
2834 first column makes it very easy to visually aggregate outputs from
2835 multiple processes, as show in the example below where each line appears
2836 for each process :
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002837
2838 $ ( echo show stat typed | socat /var/run/haproxy.sock1 - ; \
2839 echo show stat typed | socat /var/run/haproxy.sock2 - ) | \
2840 sort -t . -k 1,1 -k 2,2n -k 3,3n -k 4,4n -k 5,5 -k 6,6n
2841 B.3.0.0.pxname.1:MGP:str:private-backend
2842 B.3.0.0.pxname.2:MGP:str:private-backend
2843 B.3.0.1.svname.1:MGP:str:BACKEND
2844 B.3.0.1.svname.2:MGP:str:BACKEND
2845 B.3.0.2.qcur.1:MGP:u32:0
2846 B.3.0.2.qcur.2:MGP:u32:0
2847 B.3.0.3.qmax.1:MGP:u32:0
2848 B.3.0.3.qmax.2:MGP:u32:0
2849 B.3.0.4.scur.1:MGP:u32:0
2850 B.3.0.4.scur.2:MGP:u32:0
2851 B.3.0.5.smax.1:MGP:u32:0
2852 B.3.0.5.smax.2:MGP:u32:0
2853 B.3.0.6.slim.1:MGP:u32:1000
2854 B.3.0.6.slim.2:MGP:u32:1000
2855 (...)
Willy Tarreau44aed902015-10-13 14:45:29 +02002856
Simon Horman05ee2132017-01-04 09:37:25 +01002857 The format of JSON output is described in a schema which may be output
Simon Horman6f6bb382017-01-04 09:37:26 +01002858 using "show schema json".
2859
2860 The JSON output contains no extra whitespace in order to reduce the
2861 volume of output. For human consumption passing the output through a
2862 pretty printer may be helpful. Example :
2863
2864 $ echo "show stat json" | socat /var/run/haproxy.sock stdio | \
2865 python -m json.tool
Simon Horman05ee2132017-01-04 09:37:25 +01002866
2867 The JSON output contains no extra whitespace in order to reduce the
2868 volume of output. For human consumption passing the output through a
2869 pretty printer may be helpful. Example :
2870
2871 $ echo "show stat json" | socat /var/run/haproxy.sock stdio | \
2872 python -m json.tool
2873
William Lallemandd4f946c2019-12-05 10:26:40 +01002874show ssl cert [<filename>]
Remi Tricot-Le Bretonb5f0fac2021-04-14 16:19:29 +02002875 Display the list of certificates used on frontends and backends.
2876 If a filename is prefixed by an asterisk, it is a transaction which is not
2877 committed yet. If a filename is specified, it will show details about the
2878 certificate. This command can be useful to check if a certificate was well
2879 updated. You can also display details on a transaction by prefixing the
2880 filename by an asterisk.
William Lallemandd4f946c2019-12-05 10:26:40 +01002881
2882 Example :
2883
2884 $ echo "@1 show ssl cert" | socat /var/run/haproxy.master -
2885 # transaction
2886 *test.local.pem
2887 # filename
2888 test.local.pem
2889
2890 $ echo "@1 show ssl cert test.local.pem" | socat /var/run/haproxy.master -
2891 Filename: test.local.pem
2892 Serial: 03ECC19BA54B25E85ABA46EE561B9A10D26F
2893 notBefore: Sep 13 21:20:24 2019 GMT
2894 notAfter: Dec 12 21:20:24 2019 GMT
2895 Issuer: /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
2896 Subject: /CN=test.local
2897 Subject Alternative Name: DNS:test.local, DNS:imap.test.local
2898 Algorithm: RSA2048
2899 SHA1 FingerPrint: 417A11CAE25F607B24F638B4A8AEE51D1E211477
2900
2901 $ echo "@1 show ssl cert *test.local.pem" | socat /var/run/haproxy.master -
2902 Filename: *test.local.pem
2903 [...]
2904
William Lallemandc69f02d2020-04-06 19:07:03 +02002905show ssl crt-list [-n] [<filename>]
William Lallemandaccac232020-04-02 17:42:51 +02002906 Display the list of crt-list and directories used in the HAProxy
William Lallemandc69f02d2020-04-06 19:07:03 +02002907 configuration. If a filename is specified, dump the content of a crt-list or
2908 a directory. Once dumped the output can be used as a crt-list file.
2909 The '-n' option can be used to display the line number, which is useful when
2910 combined with the 'del ssl crt-list' option when a entry is duplicated. The
2911 output with the '-n' option is not compatible with the crt-list format and
2912 not loadable by haproxy.
William Lallemandaccac232020-04-02 17:42:51 +02002913
2914 Example:
William Lallemandc69f02d2020-04-06 19:07:03 +02002915 echo "show ssl crt-list -n localhost.crt-list" | socat /tmp/sock1 -
William Lallemandaccac232020-04-02 17:42:51 +02002916 # localhost.crt-list
William Lallemandc69f02d2020-04-06 19:07:03 +02002917 common.pem:1 !not.test1.com *.test1.com !localhost
2918 common.pem:2
2919 ecdsa.pem:3 [verify none allow-0rtt ssl-min-ver TLSv1.0 ssl-max-ver TLSv1.3] localhost !www.test1.com
2920 ecdsa.pem:4 [verify none allow-0rtt ssl-min-ver TLSv1.0 ssl-max-ver TLSv1.3]
William Lallemandaccac232020-04-02 17:42:51 +02002921
William Lallemand62416de2022-10-14 15:29:07 +02002922show startup-logs
2923 Dump all messages emitted during the startup of the current haproxy process,
2924 each startup-logs buffer is unique to its haproxy worker.
2925
Willy Tarreau44aed902015-10-13 14:45:29 +02002926show table
2927 Dump general information on all known stick-tables. Their name is returned
2928 (the name of the proxy which holds them), their type (currently zero, always
2929 IP), their size in maximum possible number of entries, and the number of
2930 entries currently in use.
2931
2932 Example :
2933 $ echo "show table" | socat stdio /tmp/sock1
2934 >>> # table: front_pub, type: ip, size:204800, used:171454
2935 >>> # table: back_rdp, type: ip, size:204800, used:0
2936
Adis Nezirovic1a693fc2020-01-16 15:19:29 +01002937show table <name> [ data.<type> <operator> <value> [data.<type> ...]] | [ key <key> ]
Willy Tarreau44aed902015-10-13 14:45:29 +02002938 Dump contents of stick-table <name>. In this mode, a first line of generic
2939 information about the table is reported as with "show table", then all
2940 entries are dumped. Since this can be quite heavy, it is possible to specify
2941 a filter in order to specify what entries to display.
2942
2943 When the "data." form is used the filter applies to the stored data (see
2944 "stick-table" in section 4.2). A stored data type must be specified
2945 in <type>, and this data type must be stored in the table otherwise an
2946 error is reported. The data is compared according to <operator> with the
2947 64-bit integer <value>. Operators are the same as with the ACLs :
2948
2949 - eq : match entries whose data is equal to this value
2950 - ne : match entries whose data is not equal to this value
2951 - le : match entries whose data is less than or equal to this value
2952 - ge : match entries whose data is greater than or equal to this value
2953 - lt : match entries whose data is less than this value
2954 - gt : match entries whose data is greater than this value
2955
Adis Nezirovic1a693fc2020-01-16 15:19:29 +01002956 In this form, you can use multiple data filter entries, up to a maximum
2957 defined during build time (4 by default).
Willy Tarreau44aed902015-10-13 14:45:29 +02002958
2959 When the key form is used the entry <key> is shown. The key must be of the
2960 same type as the table, which currently is limited to IPv4, IPv6, integer,
2961 and string.
2962
2963 Example :
2964 $ echo "show table http_proxy" | socat stdio /tmp/sock1
2965 >>> # table: http_proxy, type: ip, size:204800, used:2
2966 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
2967 bytes_out_rate(60000)=187
2968 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2969 bytes_out_rate(60000)=191
2970
2971 $ echo "show table http_proxy data.gpc0 gt 0" | socat stdio /tmp/sock1
2972 >>> # table: http_proxy, type: ip, size:204800, used:2
2973 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2974 bytes_out_rate(60000)=191
2975
2976 $ echo "show table http_proxy data.conn_rate gt 5" | \
2977 socat stdio /tmp/sock1
2978 >>> # table: http_proxy, type: ip, size:204800, used:2
2979 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2980 bytes_out_rate(60000)=191
2981
2982 $ echo "show table http_proxy key 127.0.0.2" | \
2983 socat stdio /tmp/sock1
2984 >>> # table: http_proxy, type: ip, size:204800, used:2
2985 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2986 bytes_out_rate(60000)=191
2987
2988 When the data criterion applies to a dynamic value dependent on time such as
2989 a bytes rate, the value is dynamically computed during the evaluation of the
2990 entry in order to decide whether it has to be dumped or not. This means that
2991 such a filter could match for some time then not match anymore because as
2992 time goes, the average event rate drops.
2993
2994 It is possible to use this to extract lists of IP addresses abusing the
2995 service, in order to monitor them or even blacklist them in a firewall.
2996 Example :
2997 $ echo "show table http_proxy data.gpc0 gt 0" \
2998 | socat stdio /tmp/sock1 \
2999 | fgrep 'key=' | cut -d' ' -f2 | cut -d= -f2 > abusers-ip.txt
3000 ( or | awk '/key/{ print a[split($2,a,"=")]; }' )
3001
Willy Tarreau7eff06e2021-01-29 11:32:55 +01003002show tasks
3003 Dumps the number of tasks currently in the run queue, with the number of
3004 occurrences for each function, and their average latency when it's known
3005 (for pure tasks with task profiling enabled). The dump is a snapshot of the
3006 instant it's done, and there may be variations depending on what tasks are
3007 left in the queue at the moment it happens, especially in mono-thread mode
3008 as there's less chance that I/Os can refill the queue (unless the queue is
3009 full). This command takes exclusive access to the process and can cause
3010 minor but measurable latencies when issued on a highly loaded process, so
3011 it must not be abused by monitoring bots.
3012
Willy Tarreau4e2b6462019-05-16 17:44:30 +02003013show threads
3014 Dumps some internal states and structures for each thread, that may be useful
3015 to help developers understand a problem. The output tries to be readable by
Willy Tarreauc7091d82019-05-17 10:08:49 +02003016 showing one block per thread. When haproxy is built with USE_THREAD_DUMP=1,
3017 an advanced dump mechanism involving thread signals is used so that each
3018 thread can dump its own state in turn. Without this option, the thread
3019 processing the command shows all its details but the other ones are less
Willy Tarreaue6a02fa2019-05-22 07:06:44 +02003020 detailed. A star ('*') is displayed in front of the thread handling the
3021 command. A right angle bracket ('>') may also be displayed in front of
3022 threads which didn't make any progress since last invocation of this command,
3023 indicating a bug in the code which must absolutely be reported. When this
3024 happens between two threads it usually indicates a deadlock. If a thread is
3025 alone, it's a different bug like a corrupted list. In all cases the process
3026 needs is not fully functional anymore and needs to be restarted.
3027
3028 The output format is purposely not documented so that it can easily evolve as
3029 new needs are identified, without having to maintain any form of backwards
3030 compatibility, and just like with "show activity", the values are meaningless
3031 without the code at hand.
Willy Tarreau4e2b6462019-05-16 17:44:30 +02003032
William Lallemandbb933462016-05-31 21:09:53 +02003033show tls-keys [id|*]
3034 Dump all loaded TLS ticket keys references. The TLS ticket key reference ID
3035 and the file from which the keys have been loaded is shown. Both of those
3036 can be used to update the TLS keys using "set ssl tls-key". If an ID is
3037 specified as parameter, it will dump the tickets, using * it will dump every
3038 keys from every references.
Willy Tarreau44aed902015-10-13 14:45:29 +02003039
Simon Horman6f6bb382017-01-04 09:37:26 +01003040show schema json
3041 Dump the schema used for the output of "show info json" and "show stat json".
3042
3043 The contains no extra whitespace in order to reduce the volume of output.
3044 For human consumption passing the output through a pretty printer may be
3045 helpful. Example :
3046
3047 $ echo "show schema json" | socat /var/run/haproxy.sock stdio | \
3048 python -m json.tool
3049
3050 The schema follows "JSON Schema" (json-schema.org) and accordingly
3051 verifiers may be used to verify the output of "show info json" and "show
3052 stat json" against the schema.
3053
Willy Tarreauf909c912019-08-22 20:06:04 +02003054show trace [<source>]
3055 Show the current trace status. For each source a line is displayed with a
3056 single-character status indicating if the trace is stopped, waiting, or
3057 running. The output sink used by the trace is indicated (or "none" if none
3058 was set), as well as the number of dropped events in this sink, followed by a
3059 brief description of the source. If a source name is specified, a detailed
3060 list of all events supported by the source, and their status for each action
3061 (report, start, pause, stop), indicated by a "+" if they are enabled, or a
3062 "-" otherwise. All these events are independent and an event might trigger
3063 a start without being reported and conversely.
Simon Horman6f6bb382017-01-04 09:37:26 +01003064
William Lallemand7ceae112021-12-14 15:22:29 +01003065show version
3066 Show the version of the current HAProxy process. This is available from
3067 master and workers CLI.
3068 Example:
3069
3070 $ echo "show version" | socat /var/run/haproxy.sock stdio
3071 2.4.9
3072
3073 $ echo "show version" | socat /var/run/haproxy-master.sock stdio
3074 2.5.0
3075
Willy Tarreau44aed902015-10-13 14:45:29 +02003076shutdown frontend <frontend>
3077 Completely delete the specified frontend. All the ports it was bound to will
3078 be released. It will not be possible to enable the frontend anymore after
3079 this operation. This is intended to be used in environments where stopping a
3080 proxy is not even imaginable but a misconfigured proxy must be fixed. That
3081 way it's possible to release the port and bind it into another process to
3082 restore operations. The frontend will not appear at all on the stats page
3083 once it is terminated.
3084
3085 The frontend may be specified either by its name or by its numeric ID,
3086 prefixed with a sharp ('#').
3087
3088 This command is restricted and can only be issued on sockets configured for
3089 level "admin".
3090
3091shutdown session <id>
3092 Immediately terminate the session matching the specified session identifier.
3093 This identifier is the first field at the beginning of the lines in the dumps
3094 of "show sess" (it corresponds to the session pointer). This can be used to
3095 terminate a long-running session without waiting for a timeout or when an
3096 endless transfer is ongoing. Such terminated sessions are reported with a 'K'
3097 flag in the logs.
3098
3099shutdown sessions server <backend>/<server>
3100 Immediately terminate all the sessions attached to the specified server. This
3101 can be used to terminate long-running sessions after a server is put into
3102 maintenance mode, for instance. Such terminated sessions are reported with a
3103 'K' flag in the logs.
3104
Willy Tarreauf909c912019-08-22 20:06:04 +02003105trace
3106 The "trace" command alone lists the trace sources, their current status, and
3107 their brief descriptions. It is only meant as a menu to enter next levels,
3108 see other "trace" commands below.
3109
3110trace 0
3111 Immediately stops all traces. This is made to be used as a quick solution
3112 to terminate a debugging session or as an emergency action to be used in case
3113 complex traces were enabled on multiple sources and impact the service.
3114
3115trace <source> event [ [+|-|!]<name> ]
3116 Without argument, this will list all the events supported by the designated
3117 source. They are prefixed with a "-" if they are not enabled, or a "+" if
3118 they are enabled. It is important to note that a single trace may be labelled
3119 with multiple events, and as long as any of the enabled events matches one of
3120 the events labelled on the trace, the event will be passed to the trace
3121 subsystem. For example, receiving an HTTP/2 frame of type HEADERS may trigger
3122 a frame event and a stream event since the frame creates a new stream. If
3123 either the frame event or the stream event are enabled for this source, the
3124 frame will be passed to the trace framework.
3125
3126 With an argument, it is possible to toggle the state of each event and
3127 individually enable or disable them. Two special keywords are supported,
3128 "none", which matches no event, and is used to disable all events at once,
3129 and "any" which matches all events, and is used to enable all events at
3130 once. Other events are specific to the event source. It is possible to
3131 enable one event by specifying its name, optionally prefixed with '+' for
3132 better readability. It is possible to disable one event by specifying its
3133 name prefixed by a '-' or a '!'.
3134
3135 One way to completely disable a trace source is to pass "event none", and
3136 this source will instantly be totally ignored.
3137
3138trace <source> level [<level>]
Willy Tarreau2ea549b2019-08-29 08:01:48 +02003139 Without argument, this will list all trace levels for this source, and the
Willy Tarreauf909c912019-08-22 20:06:04 +02003140 current one will be indicated by a star ('*') prepended in front of it. With
Willy Tarreau2ea549b2019-08-29 08:01:48 +02003141 an argument, this will change the trace level to the specified level. Detail
Willy Tarreauf909c912019-08-22 20:06:04 +02003142 levels are a form of filters that are applied before reporting the events.
Willy Tarreau2ea549b2019-08-29 08:01:48 +02003143 These filters are used to selectively include or exclude events depending on
3144 their level of importance. For example a developer might need to know
3145 precisely where in the code an HTTP header was considered invalid while the
3146 end user may not even care about this header's validity at all. There are
3147 currently 5 distinct levels for a trace :
Willy Tarreauf909c912019-08-22 20:06:04 +02003148
3149 user this will report information that are suitable for use by a
3150 regular haproxy user who wants to observe his traffic.
3151 Typically some HTTP requests and responses will be reported
3152 without much detail. Most sources will set this as the
3153 default level to ease operations.
3154
Willy Tarreau2ea549b2019-08-29 08:01:48 +02003155 proto in addition to what is reported at the "user" level, it also
3156 displays protocol-level updates. This can for example be the
3157 frame types or HTTP headers after decoding.
Willy Tarreauf909c912019-08-22 20:06:04 +02003158
3159 state in addition to what is reported at the "proto" level, it
3160 will also display state transitions (or failed transitions)
3161 which happen in parsers, so this will show attempts to
3162 perform an operation while the "proto" level only shows
3163 the final operation.
3164
Willy Tarreau2ea549b2019-08-29 08:01:48 +02003165 data in addition to what is reported at the "state" level, it
3166 will also include data transfers between the various layers.
3167
Willy Tarreauf909c912019-08-22 20:06:04 +02003168 developer it reports everything available, which can include advanced
3169 information such as "breaking out of this loop" that are
3170 only relevant to a developer trying to understand a bug that
Willy Tarreau09fb0df2019-08-29 08:40:59 +02003171 only happens once in a while in field. Function names are
3172 only reported at this level.
Willy Tarreauf909c912019-08-22 20:06:04 +02003173
3174 It is highly recommended to always use the "user" level only and switch to
3175 other levels only if instructed to do so by a developer. Also it is a good
3176 idea to first configure the events before switching to higher levels, as it
3177 may save from dumping many lines if no filter is applied.
3178
3179trace <source> lock [criterion]
3180 Without argument, this will list all the criteria supported by this source
3181 for lock-on processing, and display the current choice by a star ('*') in
3182 front of it. Lock-on means that the source will focus on the first matching
3183 event and only stick to the criterion which triggered this event, and ignore
3184 all other ones until the trace stops. This allows for example to take a trace
3185 on a single connection or on a single stream. The following criteria are
3186 supported by some traces, though not necessarily all, since some of them
3187 might not be available to the source :
3188
3189 backend lock on the backend that started the trace
3190 connection lock on the connection that started the trace
3191 frontend lock on the frontend that started the trace
3192 listener lock on the listener that started the trace
3193 nothing do not lock on anything
3194 server lock on the server that started the trace
3195 session lock on the session that started the trace
3196 thread lock on the thread that started the trace
3197
3198 In addition to this, each source may provide up to 4 specific criteria such
3199 as internal states or connection IDs. For example in HTTP/2 it is possible
3200 to lock on the H2 stream and ignore other streams once a strace starts.
3201
3202 When a criterion is passed in argument, this one is used instead of the
3203 other ones and any existing tracking is immediately terminated so that it can
3204 restart with the new criterion. The special keyword "nothing" is supported by
3205 all sources to permanently disable tracking.
3206
3207trace <source> { pause | start | stop } [ [+|-|!]event]
3208 Without argument, this will list the events enabled to automatically pause,
3209 start, or stop a trace for this source. These events are specific to each
3210 trace source. With an argument, this will either enable the event for the
3211 specified action (if optionally prefixed by a '+') or disable it (if
3212 prefixed by a '-' or '!'). The special keyword "now" is not an event and
3213 requests to take the action immediately. The keywords "none" and "any" are
3214 supported just like in "trace event".
3215
3216 The 3 supported actions are respectively "pause", "start" and "stop". The
3217 "pause" action enumerates events which will cause a running trace to stop and
3218 wait for a new start event to restart it. The "start" action enumerates the
3219 events which switch the trace into the waiting mode until one of the start
3220 events appears. And the "stop" action enumerates the events which definitely
3221 stop the trace until it is manually enabled again. In practice it makes sense
3222 to manually start a trace using "start now" without caring about events, and
3223 to stop it using "stop now". In order to capture more subtle event sequences,
3224 setting "start" to a normal event (like receiving an HTTP request) and "stop"
3225 to a very rare event like emitting a certain error, will ensure that the last
3226 captured events will match the desired criteria. And the pause event is
3227 useful to detect the end of a sequence, disable the lock-on and wait for
3228 another opportunity to take a capture. In this case it can make sense to
3229 enable lock-on to spot only one specific criterion (e.g. a stream), and have
3230 "start" set to anything that starts this criterion (e.g. all events which
3231 create a stream), "stop" set to the expected anomaly, and "pause" to anything
3232 that ends that criterion (e.g. any end of stream event). In this case the
3233 trace log will contain complete sequences of perfectly clean series affecting
3234 a single object, until the last sequence containing everything from the
3235 beginning to the anomaly.
3236
3237trace <source> sink [<sink>]
3238 Without argument, this will list all event sinks available for this source,
3239 and the currently configured one will have a star ('*') prepended in front
3240 of it. Sink "none" is always available and means that all events are simply
3241 dropped, though their processing is not ignored (e.g. lock-on does occur).
3242 Other sinks are available depending on configuration and build options, but
3243 typically "stdout" and "stderr" will be usable in debug mode, and in-memory
3244 ring buffers should be available as well. When a name is specified, the sink
3245 instantly changes for the specified source. Events are not changed during a
3246 sink change. In the worst case some may be lost if an invalid sink is used
3247 (or "none"), but operations do continue to a different destination.
3248
Willy Tarreau370a6942019-08-29 08:24:16 +02003249trace <source> verbosity [<level>]
3250 Without argument, this will list all verbosity levels for this source, and the
3251 current one will be indicated by a star ('*') prepended in front of it. With
3252 an argument, this will change the verbosity level to the specified one.
3253
3254 Verbosity levels indicate how far the trace decoder should go to provide
3255 detailed information. It depends on the trace source, since some sources will
3256 not even provide a specific decoder. Level "quiet" is always available and
3257 disables any decoding. It can be useful when trying to figure what's
3258 happening before trying to understand the details, since it will have a very
3259 low impact on performance and trace size. When no verbosity levels are
3260 declared by a source, level "default" is available and will cause a decoder
3261 to be called when specified in the traces. It is an opportunistic decoding.
3262 When the source declares some verbosity levels, these ones are listed with
3263 a description of what they correspond to. In this case the trace decoder
3264 provided by the source will be as accurate as possible based on the
3265 information available at the trace point. The first level above "quiet" is
3266 set by default.
3267
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003268
William Lallemand142db372018-12-11 18:56:45 +010032699.4. Master CLI
3270---------------
3271
3272The master CLI is a socket bound to the master process in master-worker mode.
3273This CLI gives access to the unix socket commands in every running or leaving
3274processes and allows a basic supervision of those processes.
3275
3276The master CLI is configurable only from the haproxy program arguments with
3277the -S option. This option also takes bind options separated by commas.
3278
3279Example:
3280
3281 # haproxy -W -S 127.0.0.1:1234 -f test1.cfg
3282 # haproxy -Ws -S /tmp/master-socket,uid,1000,gid,1000,mode,600 -f test1.cfg
William Lallemandb7ea1412018-12-13 09:05:47 +01003283 # haproxy -W -S /tmp/master-socket,level,user -f test1.cfg
William Lallemand142db372018-12-11 18:56:45 +01003284
3285The master CLI introduces a new 'show proc' command to surpervise the
3286processes:
3287
3288Example:
3289
3290 $ echo 'show proc' | socat /var/run/haproxy-master.sock -
William Lallemand1dc69632019-06-12 19:11:33 +02003291 #<PID> <type> <relative PID> <reloads> <uptime> <version>
3292 1162 master 0 5 0d00h02m07s 2.0-dev7-0124c9-7
William Lallemand142db372018-12-11 18:56:45 +01003293 # workers
William Lallemand1dc69632019-06-12 19:11:33 +02003294 1271 worker 1 0 0d00h00m00s 2.0-dev7-0124c9-7
3295 1272 worker 2 0 0d00h00m00s 2.0-dev7-0124c9-7
William Lallemand142db372018-12-11 18:56:45 +01003296 # old workers
William Lallemand1dc69632019-06-12 19:11:33 +02003297 1233 worker [was: 1] 3 0d00h00m43s 2.0-dev3-6019f6-289
William Lallemand142db372018-12-11 18:56:45 +01003298
3299
3300In this example, the master has been reloaded 5 times but one of the old
3301worker is still running and survived 3 reloads. You could access the CLI of
3302this worker to understand what's going on.
3303
Willy Tarreau52880f92018-12-15 13:30:03 +01003304When the prompt is enabled (via the "prompt" command), the context the CLI is
3305working on is displayed in the prompt. The master is identified by the "master"
3306string, and other processes are identified with their PID. In case the last
3307reload failed, the master prompt will be changed to "master[ReloadFailed]>" so
3308that it becomes visible that the process is still running on the previous
3309configuration and that the new configuration is not operational.
3310
William Lallemand142db372018-12-11 18:56:45 +01003311The master CLI uses a special prefix notation to access the multiple
3312processes. This notation is easily identifiable as it begins by a @.
3313
3314A @ prefix can be followed by a relative process number or by an exclamation
3315point and a PID. (e.g. @1 or @!1271). A @ alone could be use to specify the
3316master. Leaving processes are only accessible with the PID as relative process
3317number are only usable with the current processes.
3318
3319Examples:
3320
3321 $ socat /var/run/haproxy-master.sock readline
3322 prompt
3323 master> @1 show info; @2 show info
3324 [...]
3325 Process_num: 1
3326 Pid: 1271
3327 [...]
3328 Process_num: 2
3329 Pid: 1272
3330 [...]
3331 master>
3332
3333 $ echo '@!1271 show info; @!1272 show info' | socat /var/run/haproxy-master.sock -
3334 [...]
3335
3336A prefix could be use as a command, which will send every next commands to
3337the specified process.
3338
3339Examples:
3340
3341 $ socat /var/run/haproxy-master.sock readline
3342 prompt
3343 master> @1
3344 1271> show info
3345 [...]
3346 1271> show stat
3347 [...]
3348 1271> @
3349 master>
3350
3351 $ echo '@1; show info; show stat; @2; show info; show stat' | socat /var/run/haproxy-master.sock -
3352 [...]
3353
William Lallemanda57b7e32018-12-14 21:11:31 +01003354You can also reload the HAProxy master process with the "reload" command which
3355does the same as a `kill -USR2` on the master process, provided that the user
3356has at least "operator" or "admin" privileges.
3357
3358Example:
3359
varnav5a3fe9f2021-05-10 10:29:57 -04003360 $ echo "reload" | socat /var/run/haproxy-master.sock stdin
William Lallemanda57b7e32018-12-14 21:11:31 +01003361
3362Note that a reload will close the connection to the master CLI.
3363
William Lallemand142db372018-12-11 18:56:45 +01003364
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200336510. Tricks for easier configuration management
3366----------------------------------------------
3367
3368It is very common that two HAProxy nodes constituting a cluster share exactly
3369the same configuration modulo a few addresses. Instead of having to maintain a
3370duplicate configuration for each node, which will inevitably diverge, it is
3371possible to include environment variables in the configuration. Thus multiple
3372configuration may share the exact same file with only a few different system
3373wide environment variables. This started in version 1.5 where only addresses
3374were allowed to include environment variables, and 1.6 goes further by
3375supporting environment variables everywhere. The syntax is the same as in the
3376UNIX shell, a variable starts with a dollar sign ('$'), followed by an opening
3377curly brace ('{'), then the variable name followed by the closing brace ('}').
3378Except for addresses, environment variables are only interpreted in arguments
3379surrounded with double quotes (this was necessary not to break existing setups
3380using regular expressions involving the dollar symbol).
3381
3382Environment variables also make it convenient to write configurations which are
3383expected to work on various sites where only the address changes. It can also
3384permit to remove passwords from some configs. Example below where the the file
3385"site1.env" file is sourced by the init script upon startup :
3386
3387 $ cat site1.env
3388 LISTEN=192.168.1.1
3389 CACHE_PFX=192.168.11
3390 SERVER_PFX=192.168.22
3391 LOGGER=192.168.33.1
3392 STATSLP=admin:pa$$w0rd
3393 ABUSERS=/etc/haproxy/abuse.lst
3394 TIMEOUT=10s
3395
3396 $ cat haproxy.cfg
3397 global
3398 log "${LOGGER}:514" local0
3399
3400 defaults
3401 mode http
3402 timeout client "${TIMEOUT}"
3403 timeout server "${TIMEOUT}"
3404 timeout connect 5s
3405
3406 frontend public
3407 bind "${LISTEN}:80"
3408 http-request reject if { src -f "${ABUSERS}" }
3409 stats uri /stats
3410 stats auth "${STATSLP}"
3411 use_backend cache if { path_end .jpg .css .ico }
3412 default_backend server
3413
3414 backend cache
3415 server cache1 "${CACHE_PFX}.1:18080" check
3416 server cache2 "${CACHE_PFX}.2:18080" check
3417
3418 backend server
3419 server cache1 "${SERVER_PFX}.1:8080" check
3420 server cache2 "${SERVER_PFX}.2:8080" check
3421
3422
342311. Well-known traps to avoid
3424-----------------------------
3425
3426Once in a while, someone reports that after a system reboot, the haproxy
3427service wasn't started, and that once they start it by hand it works. Most
3428often, these people are running a clustered IP address mechanism such as
3429keepalived, to assign the service IP address to the master node only, and while
3430it used to work when they used to bind haproxy to address 0.0.0.0, it stopped
3431working after they bound it to the virtual IP address. What happens here is
3432that when the service starts, the virtual IP address is not yet owned by the
3433local node, so when HAProxy wants to bind to it, the system rejects this
3434because it is not a local IP address. The fix doesn't consist in delaying the
3435haproxy service startup (since it wouldn't stand a restart), but instead to
3436properly configure the system to allow binding to non-local addresses. This is
3437easily done on Linux by setting the net.ipv4.ip_nonlocal_bind sysctl to 1. This
3438is also needed in order to transparently intercept the IP traffic that passes
3439through HAProxy for a specific target address.
3440
3441Multi-process configurations involving source port ranges may apparently seem
3442to work but they will cause some random failures under high loads because more
3443than one process may try to use the same source port to connect to the same
3444server, which is not possible. The system will report an error and a retry will
3445happen, picking another port. A high value in the "retries" parameter may hide
3446the effect to a certain extent but this also comes with increased CPU usage and
3447processing time. Logs will also report a certain number of retries. For this
3448reason, port ranges should be avoided in multi-process configurations.
3449
Dan Lloyd8e48b872016-07-01 21:01:18 -04003450Since HAProxy uses SO_REUSEPORT and supports having multiple independent
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003451processes bound to the same IP:port, during troubleshooting it can happen that
3452an old process was not stopped before a new one was started. This provides
3453absurd test results which tend to indicate that any change to the configuration
3454is ignored. The reason is that in fact even the new process is restarted with a
3455new configuration, the old one also gets some incoming connections and
3456processes them, returning unexpected results. When in doubt, just stop the new
3457process and try again. If it still works, it very likely means that an old
3458process remains alive and has to be stopped. Linux's "netstat -lntp" is of good
3459help here.
3460
3461When adding entries to an ACL from the command line (eg: when blacklisting a
3462source address), it is important to keep in mind that these entries are not
3463synchronized to the file and that if someone reloads the configuration, these
3464updates will be lost. While this is often the desired effect (for blacklisting)
3465it may not necessarily match expectations when the change was made as a fix for
3466a problem. See the "add acl" action of the CLI interface.
3467
3468
346912. Debugging and performance issues
3470------------------------------------
3471
3472When HAProxy is started with the "-d" option, it will stay in the foreground
3473and will print one line per event, such as an incoming connection, the end of a
3474connection, and for each request or response header line seen. This debug
3475output is emitted before the contents are processed, so they don't consider the
3476local modifications. The main use is to show the request and response without
3477having to run a network sniffer. The output is less readable when multiple
3478connections are handled in parallel, though the "debug2ansi" and "debug2html"
3479scripts found in the examples/ directory definitely help here by coloring the
3480output.
3481
3482If a request or response is rejected because HAProxy finds it is malformed, the
3483best thing to do is to connect to the CLI and issue "show errors", which will
3484report the last captured faulty request and response for each frontend and
3485backend, with all the necessary information to indicate precisely the first
3486character of the input stream that was rejected. This is sometimes needed to
3487prove to customers or to developers that a bug is present in their code. In
3488this case it is often possible to relax the checks (but still keep the
3489captures) using "option accept-invalid-http-request" or its equivalent for
3490responses coming from the server "option accept-invalid-http-response". Please
3491see the configuration manual for more details.
3492
3493Example :
3494
3495 > show errors
3496 Total events captured on [13/Oct/2015:13:43:47.169] : 1
3497
3498 [13/Oct/2015:13:43:40.918] frontend HAProxyLocalStats (#2): invalid request
3499 backend <NONE> (#-1), server <NONE> (#-1), event #0
3500 src 127.0.0.1:51981, session #0, session flags 0x00000080
3501 HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000
3502 HTTP chunk len 0 bytes, HTTP body len 0 bytes
3503 buffer flags 0x00808002, out 0 bytes, total 31 bytes
3504 pending 31 bytes, wrapping at 8040, error at position 13:
3505
3506 00000 GET /invalid request HTTP/1.1\r\n
3507
3508
3509The output of "show info" on the CLI provides a number of useful information
3510regarding the maximum connection rate ever reached, maximum SSL key rate ever
3511reached, and in general all information which can help to explain temporary
3512issues regarding CPU or memory usage. Example :
3513
3514 > show info
3515 Name: HAProxy
3516 Version: 1.6-dev7-e32d18-17
3517 Release_date: 2015/10/12
3518 Nbproc: 1
3519 Process_num: 1
3520 Pid: 7949
3521 Uptime: 0d 0h02m39s
3522 Uptime_sec: 159
3523 Memmax_MB: 0
3524 Ulimit-n: 120032
3525 Maxsock: 120032
3526 Maxconn: 60000
3527 Hard_maxconn: 60000
3528 CurrConns: 0
3529 CumConns: 3
3530 CumReq: 3
3531 MaxSslConns: 0
3532 CurrSslConns: 0
3533 CumSslConns: 0
3534 Maxpipes: 0
3535 PipesUsed: 0
3536 PipesFree: 0
3537 ConnRate: 0
3538 ConnRateLimit: 0
3539 MaxConnRate: 1
3540 SessRate: 0
3541 SessRateLimit: 0
3542 MaxSessRate: 1
3543 SslRate: 0
3544 SslRateLimit: 0
3545 MaxSslRate: 0
3546 SslFrontendKeyRate: 0
3547 SslFrontendMaxKeyRate: 0
3548 SslFrontendSessionReuse_pct: 0
3549 SslBackendKeyRate: 0
3550 SslBackendMaxKeyRate: 0
3551 SslCacheLookups: 0
3552 SslCacheMisses: 0
3553 CompressBpsIn: 0
3554 CompressBpsOut: 0
3555 CompressBpsRateLim: 0
3556 ZlibMemUsage: 0
3557 MaxZlibMemUsage: 0
3558 Tasks: 5
3559 Run_queue: 1
3560 Idle_pct: 100
3561 node: wtap
3562 description:
3563
3564When an issue seems to randomly appear on a new version of HAProxy (eg: every
3565second request is aborted, occasional crash, etc), it is worth trying to enable
Dan Lloyd8e48b872016-07-01 21:01:18 -04003566memory poisoning so that each call to malloc() is immediately followed by the
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003567filling of the memory area with a configurable byte. By default this byte is
35680x50 (ASCII for 'P'), but any other byte can be used, including zero (which
3569will have the same effect as a calloc() and which may make issues disappear).
Dan Lloyd8e48b872016-07-01 21:01:18 -04003570Memory poisoning is enabled on the command line using the "-dM" option. It
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003571slightly hurts performance and is not recommended for use in production. If
Dan Lloyd8e48b872016-07-01 21:01:18 -04003572an issue happens all the time with it or never happens when poisoning uses
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003573byte zero, it clearly means you've found a bug and you definitely need to
3574report it. Otherwise if there's no clear change, the problem it is not related.
3575
3576When debugging some latency issues, it is important to use both strace and
3577tcpdump on the local machine, and another tcpdump on the remote system. The
3578reason for this is that there are delays everywhere in the processing chain and
3579it is important to know which one is causing latency to know where to act. In
3580practice, the local tcpdump will indicate when the input data come in. Strace
3581will indicate when haproxy receives these data (using recv/recvfrom). Warning,
3582openssl uses read()/write() syscalls instead of recv()/send(). Strace will also
3583show when haproxy sends the data, and tcpdump will show when the system sends
3584these data to the interface. Then the external tcpdump will show when the data
3585sent are really received (since the local one only shows when the packets are
3586queued). The benefit of sniffing on the local system is that strace and tcpdump
3587will use the same reference clock. Strace should be used with "-tts200" to get
3588complete timestamps and report large enough chunks of data to read them.
3589Tcpdump should be used with "-nvvttSs0" to report full packets, real sequence
3590numbers and complete timestamps.
3591
3592In practice, received data are almost always immediately received by haproxy
3593(unless the machine has a saturated CPU or these data are invalid and not
3594delivered). If these data are received but not sent, it generally is because
3595the output buffer is saturated (ie: recipient doesn't consume the data fast
3596enough). This can be confirmed by seeing that the polling doesn't notify of
3597the ability to write on the output file descriptor for some time (it's often
3598easier to spot in the strace output when the data finally leave and then roll
3599back to see when the write event was notified). It generally matches an ACK
3600received from the recipient, and detected by tcpdump. Once the data are sent,
3601they may spend some time in the system doing nothing. Here again, the TCP
3602congestion window may be limited and not allow these data to leave, waiting for
3603an ACK to open the window. If the traffic is idle and the data take 40 ms or
3604200 ms to leave, it's a different issue (which is not an issue), it's the fact
3605that the Nagle algorithm prevents empty packets from leaving immediately, in
3606hope that they will be merged with subsequent data. HAProxy automatically
3607disables Nagle in pure TCP mode and in tunnels. However it definitely remains
3608enabled when forwarding an HTTP body (and this contributes to the performance
3609improvement there by reducing the number of packets). Some HTTP non-compliant
3610applications may be sensitive to the latency when delivering incomplete HTTP
3611response messages. In this case you will have to enable "option http-no-delay"
3612to disable Nagle in order to work around their design, keeping in mind that any
3613other proxy in the chain may similarly be impacted. If tcpdump reports that data
3614leave immediately but the other end doesn't see them quickly, it can mean there
Dan Lloyd8e48b872016-07-01 21:01:18 -04003615is a congested WAN link, a congested LAN with flow control enabled and
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003616preventing the data from leaving, or more commonly that HAProxy is in fact
3617running in a virtual machine and that for whatever reason the hypervisor has
3618decided that the data didn't need to be sent immediately. In virtualized
3619environments, latency issues are almost always caused by the virtualization
3620layer, so in order to save time, it's worth first comparing tcpdump in the VM
3621and on the external components. Any difference has to be credited to the
3622hypervisor and its accompanying drivers.
3623
3624When some TCP SACK segments are seen in tcpdump traces (using -vv), it always
3625means that the side sending them has got the proof of a lost packet. While not
3626seeing them doesn't mean there are no losses, seeing them definitely means the
3627network is lossy. Losses are normal on a network, but at a rate where SACKs are
3628not noticeable at the naked eye. If they appear a lot in the traces, it is
3629worth investigating exactly what happens and where the packets are lost. HTTP
3630doesn't cope well with TCP losses, which introduce huge latencies.
3631
3632The "netstat -i" command will report statistics per interface. An interface
3633where the Rx-Ovr counter grows indicates that the system doesn't have enough
3634resources to receive all incoming packets and that they're lost before being
3635processed by the network driver. Rx-Drp indicates that some received packets
3636were lost in the network stack because the application doesn't process them
3637fast enough. This can happen during some attacks as well. Tx-Drp means that
3638the output queues were full and packets had to be dropped. When using TCP it
Dan Lloyd8e48b872016-07-01 21:01:18 -04003639should be very rare, but will possibly indicate a saturated outgoing link.
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003640
3641
364213. Security considerations
3643---------------------------
3644
3645HAProxy is designed to run with very limited privileges. The standard way to
3646use it is to isolate it into a chroot jail and to drop its privileges to a
3647non-root user without any permissions inside this jail so that if any future
3648vulnerability were to be discovered, its compromise would not affect the rest
3649of the system.
3650
Dan Lloyd8e48b872016-07-01 21:01:18 -04003651In order to perform a chroot, it first needs to be started as a root user. It is
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003652pointless to build hand-made chroots to start the process there, these ones are
3653painful to build, are never properly maintained and always contain way more
3654bugs than the main file-system. And in case of compromise, the intruder can use
3655the purposely built file-system. Unfortunately many administrators confuse
3656"start as root" and "run as root", resulting in the uid change to be done prior
3657to starting haproxy, and reducing the effective security restrictions.
3658
3659HAProxy will need to be started as root in order to :
3660 - adjust the file descriptor limits
3661 - bind to privileged port numbers
3662 - bind to a specific network interface
3663 - transparently listen to a foreign address
3664 - isolate itself inside the chroot jail
3665 - drop to another non-privileged UID
3666
3667HAProxy may require to be run as root in order to :
3668 - bind to an interface for outgoing connections
3669 - bind to privileged source ports for outgoing connections
Dan Lloyd8e48b872016-07-01 21:01:18 -04003670 - transparently bind to a foreign address for outgoing connections
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003671
3672Most users will never need the "run as root" case. But the "start as root"
3673covers most usages.
3674
3675A safe configuration will have :
3676
3677 - a chroot statement pointing to an empty location without any access
3678 permissions. This can be prepared this way on the UNIX command line :
3679
3680 # mkdir /var/empty && chmod 0 /var/empty || echo "Failed"
3681
3682 and referenced like this in the HAProxy configuration's global section :
3683
3684 chroot /var/empty
3685
3686 - both a uid/user and gid/group statements in the global section :
3687
3688 user haproxy
3689 group haproxy
3690
3691 - a stats socket whose mode, uid and gid are set to match the user and/or
3692 group allowed to access the CLI so that nobody may access it :
3693
3694 stats socket /var/run/haproxy.stat uid hatop gid hatop mode 600
3695