blob: f77ed4c1b04cc62f8d7324bdd23fdacc0fb659fa [file] [log] [blame]
Willy Tarreau2212e6a2015-10-13 14:40:55 +02001 ------------------------
2 HAProxy Management Guide
3 ------------------------
Willy Tarreau1db55792020-11-05 17:20:35 +01004 version 2.4
Willy Tarreau2212e6a2015-10-13 14:40:55 +02005
6
7This document describes how to start, stop, manage, and troubleshoot HAProxy,
8as well as some known limitations and traps to avoid. It does not describe how
9to configure it (for this please read configuration.txt).
10
11Note to documentation contributors :
12 This document is formatted with 80 columns per line, with even number of
13 spaces for indentation and without tabs. Please follow these rules strictly
14 so that it remains easily printable everywhere. If you add sections, please
15 update the summary below for easier searching.
16
17
18Summary
19-------
20
211. Prerequisites
222. Quick reminder about HAProxy's architecture
233. Starting HAProxy
244. Stopping and restarting HAProxy
255. File-descriptor limitations
266. Memory management
277. CPU usage
288. Logging
299. Statistics and monitoring
Willy Tarreau44aed902015-10-13 14:45:29 +0200309.1. CSV format
Willy Tarreau5d8b9792016-03-11 11:09:34 +0100319.2. Typed output format
329.3. Unix Socket commands
William Lallemand142db372018-12-11 18:56:45 +0100339.4. Master CLI
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003410. Tricks for easier configuration management
3511. Well-known traps to avoid
3612. Debugging and performance issues
3713. Security considerations
38
39
401. Prerequisites
41----------------
42
43In this document it is assumed that the reader has sufficient administration
44skills on a UNIX-like operating system, uses the shell on a daily basis and is
45familiar with troubleshooting utilities such as strace and tcpdump.
46
47
482. Quick reminder about HAProxy's architecture
49----------------------------------------------
50
Willy Tarreau3f364482019-02-27 15:01:46 +010051HAProxy is a multi-threaded, event-driven, non-blocking daemon. This means is
Willy Tarreau2212e6a2015-10-13 14:40:55 +020052uses event multiplexing to schedule all of its activities instead of relying on
53the system to schedule between multiple activities. Most of the time it runs as
54a single process, so the output of "ps aux" on a system will report only one
55"haproxy" process, unless a soft reload is in progress and an older process is
56finishing its job in parallel to the new one. It is thus always easy to trace
Willy Tarreau3f364482019-02-27 15:01:46 +010057its activity using the strace utility. In order to scale with the number of
58available processors, by default haproxy will start one worker thread per
59processor it is allowed to run on. Unless explicitly configured differently,
60the incoming traffic is spread over all these threads, all running the same
61event loop. A great care is taken to limit inter-thread dependencies to the
62strict minimum, so as to try to achieve near-linear scalability. This has some
63impacts such as the fact that a given connection is served by a single thread.
64Thus in order to use all available processing capacity, it is needed to have at
65least as many connections as there are threads, which is almost always granted.
Willy Tarreau2212e6a2015-10-13 14:40:55 +020066
67HAProxy is designed to isolate itself into a chroot jail during startup, where
68it cannot perform any file-system access at all. This is also true for the
69libraries it depends on (eg: libc, libssl, etc). The immediate effect is that
70a running process will not be able to reload a configuration file to apply
71changes, instead a new process will be started using the updated configuration
72file. Some other less obvious effects are that some timezone files or resolver
73files the libc might attempt to access at run time will not be found, though
74this should generally not happen as they're not needed after startup. A nice
75consequence of this principle is that the HAProxy process is totally stateless,
76and no cleanup is needed after it's killed, so any killing method that works
77will do the right thing.
78
79HAProxy doesn't write log files, but it relies on the standard syslog protocol
80to send logs to a remote server (which is often located on the same system).
81
82HAProxy uses its internal clock to enforce timeouts, that is derived from the
83system's time but where unexpected drift is corrected. This is done by limiting
84the time spent waiting in poll() for an event, and measuring the time it really
85took. In practice it never waits more than one second. This explains why, when
86running strace over a completely idle process, periodic calls to poll() (or any
87of its variants) surrounded by two gettimeofday() calls are noticed. They are
88normal, completely harmless and so cheap that the load they imply is totally
89undetectable at the system scale, so there's nothing abnormal there. Example :
90
91 16:35:40.002320 gettimeofday({1442759740, 2605}, NULL) = 0
92 16:35:40.002942 epoll_wait(0, {}, 200, 1000) = 0
93 16:35:41.007542 gettimeofday({1442759741, 7641}, NULL) = 0
94 16:35:41.007998 gettimeofday({1442759741, 8114}, NULL) = 0
95 16:35:41.008391 epoll_wait(0, {}, 200, 1000) = 0
96 16:35:42.011313 gettimeofday({1442759742, 11411}, NULL) = 0
97
98HAProxy is a TCP proxy, not a router. It deals with established connections that
99have been validated by the kernel, and not with packets of any form nor with
100sockets in other states (eg: no SYN_RECV nor TIME_WAIT), though their existence
101may prevent it from binding a port. It relies on the system to accept incoming
102connections and to initiate outgoing connections. An immediate effect of this is
103that there is no relation between packets observed on the two sides of a
104forwarded connection, which can be of different size, numbers and even family.
105Since a connection may only be accepted from a socket in LISTEN state, all the
106sockets it is listening to are necessarily visible using the "netstat" utility
107to show listening sockets. Example :
108
109 # netstat -ltnp
110 Active Internet connections (only servers)
111 Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
112 tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1629/sshd
113 tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2847/haproxy
114 tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 2847/haproxy
115
116
1173. Starting HAProxy
118-------------------
119
120HAProxy is started by invoking the "haproxy" program with a number of arguments
121passed on the command line. The actual syntax is :
122
123 $ haproxy [<options>]*
124
125where [<options>]* is any number of options. An option always starts with '-'
126followed by one of more letters, and possibly followed by one or multiple extra
127arguments. Without any option, HAProxy displays the help page with a reminder
128about supported options. Available options may vary slightly based on the
129operating system. A fair number of these options overlap with an equivalent one
130if the "global" section. In this case, the command line always has precedence
131over the configuration file, so that the command line can be used to quickly
132enforce some settings without touching the configuration files. The current
133list of options is :
134
135 -- <cfgfile>* : all the arguments following "--" are paths to configuration
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200136 file/directory to be loaded and processed in the declaration order. It is
137 mostly useful when relying on the shell to load many files that are
138 numerically ordered. See also "-f". The difference between "--" and "-f" is
139 that one "-f" must be placed before each file name, while a single "--" is
140 needed before all file names. Both options can be used together, the
141 command line ordering still applies. When more than one file is specified,
142 each file must start on a section boundary, so the first keyword of each
143 file must be one of "global", "defaults", "peers", "listen", "frontend",
144 "backend", and so on. A file cannot contain just a server list for example.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200145
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200146 -f <cfgfile|cfgdir> : adds <cfgfile> to the list of configuration files to be
147 loaded. If <cfgdir> is a directory, all the files (and only files) it
Dan Lloyd8e48b872016-07-01 21:01:18 -0400148 contains are added in lexical order (using LC_COLLATE=C) to the list of
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200149 configuration files to be loaded ; only files with ".cfg" extension are
150 added, only non hidden files (not prefixed with ".") are added.
151 Configuration files are loaded and processed in their declaration order.
152 This option may be specified multiple times to load multiple files. See
153 also "--". The difference between "--" and "-f" is that one "-f" must be
154 placed before each file name, while a single "--" is needed before all file
155 names. Both options can be used together, the command line ordering still
156 applies. When more than one file is specified, each file must start on a
157 section boundary, so the first keyword of each file must be one of
158 "global", "defaults", "peers", "listen", "frontend", "backend", and so on.
159 A file cannot contain just a server list for example.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200160
161 -C <dir> : changes to directory <dir> before loading configuration
162 files. This is useful when using relative paths. Warning when using
163 wildcards after "--" which are in fact replaced by the shell before
164 starting haproxy.
165
166 -D : start as a daemon. The process detaches from the current terminal after
167 forking, and errors are not reported anymore in the terminal. It is
168 equivalent to the "daemon" keyword in the "global" section of the
169 configuration. It is recommended to always force it in any init script so
170 that a faulty configuration doesn't prevent the system from booting.
171
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200172 -L <name> : change the local peer name to <name>, which defaults to the local
William Lallemanddaf4cd22018-04-17 16:46:13 +0200173 hostname. This is used only with peers replication. You can use the
174 variable $HAPROXY_LOCALPEER in the configuration file to reference the
175 peer name.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200176
177 -N <limit> : sets the default per-proxy maxconn to <limit> instead of the
178 builtin default value (usually 2000). Only useful for debugging.
179
180 -V : enable verbose mode (disables quiet mode). Reverts the effect of "-q" or
181 "quiet".
182
William Lallemande202b1e2017-06-01 17:38:56 +0200183 -W : master-worker mode. It is equivalent to the "master-worker" keyword in
184 the "global" section of the configuration. This mode will launch a "master"
185 which will monitor the "workers". Using this mode, you can reload HAProxy
186 directly by sending a SIGUSR2 signal to the master. The master-worker mode
187 is compatible either with the foreground or daemon mode. It is
188 recommended to use this mode with multiprocess and systemd.
189
Pavlos Parissisf65f2572018-02-07 21:42:16 +0100190 -Ws : master-worker mode with support of `notify` type of systemd service.
191 This option is only available when HAProxy was built with `USE_SYSTEMD`
192 build option enabled.
193
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200194 -c : only performs a check of the configuration files and exits before trying
195 to bind. The exit status is zero if everything is OK, or non-zero if an
Willy Tarreaubebd2122020-04-15 16:06:11 +0200196 error is encountered. Presence of warnings will be reported if any.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200197
198 -d : enable debug mode. This disables daemon mode, forces the process to stay
Willy Tarreauccf42992020-10-09 19:15:03 +0200199 in foreground and to show incoming and outgoing events. It must never be
200 used in an init script.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200201
202 -dG : disable use of getaddrinfo() to resolve host names into addresses. It
203 can be used when suspecting that getaddrinfo() doesn't work as expected.
204 This option was made available because many bogus implementations of
205 getaddrinfo() exist on various systems and cause anomalies that are
206 difficult to troubleshoot.
207
Dan Lloyd8e48b872016-07-01 21:01:18 -0400208 -dM[<byte>] : forces memory poisoning, which means that each and every
Willy Tarreaubafbe012017-11-24 17:34:44 +0100209 memory region allocated with malloc() or pool_alloc() will be filled with
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200210 <byte> before being passed to the caller. When <byte> is not specified, it
211 defaults to 0x50 ('P'). While this slightly slows down operations, it is
212 useful to reliably trigger issues resulting from missing initializations in
213 the code that cause random crashes. Note that -dM0 has the effect of
214 turning any malloc() into a calloc(). In any case if a bug appears or
215 disappears when using this option it means there is a bug in haproxy, so
216 please report it.
217
218 -dS : disable use of the splice() system call. It is equivalent to the
219 "global" section's "nosplice" keyword. This may be used when splice() is
220 suspected to behave improperly or to cause performance issues, or when
221 using strace to see the forwarded data (which do not appear when using
222 splice()).
223
224 -dV : disable SSL verify on the server side. It is equivalent to having
225 "ssl-server-verify none" in the "global" section. This is useful when
226 trying to reproduce production issues out of the production
227 environment. Never use this in an init script as it degrades SSL security
228 to the servers.
229
Willy Tarreau3eb10b82020-04-15 16:42:39 +0200230 -dW : if set, haproxy will refuse to start if any warning was emitted while
231 processing the configuration. This helps detect subtle mistakes and keep the
232 configuration clean and portable across versions. It is recommended to set
233 this option in service scripts when configurations are managed by humans,
234 but it is recommended not to use it with generated configurations, which
235 tend to emit more warnings. It may be combined with "-c" to cause warnings
236 in checked configurations to fail. This is equivalent to global option
237 "zero-warning".
238
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200239 -db : disable background mode and multi-process mode. The process remains in
240 foreground. It is mainly used during development or during small tests, as
241 Ctrl-C is enough to stop the process. Never use it in an init script.
242
243 -de : disable the use of the "epoll" poller. It is equivalent to the "global"
244 section's keyword "noepoll". It is mostly useful when suspecting a bug
245 related to this poller. On systems supporting epoll, the fallback will
246 generally be the "poll" poller.
247
248 -dk : disable the use of the "kqueue" poller. It is equivalent to the
249 "global" section's keyword "nokqueue". It is mostly useful when suspecting
250 a bug related to this poller. On systems supporting kqueue, the fallback
251 will generally be the "poll" poller.
252
253 -dp : disable the use of the "poll" poller. It is equivalent to the "global"
254 section's keyword "nopoll". It is mostly useful when suspecting a bug
255 related to this poller. On systems supporting poll, the fallback will
256 generally be the "select" poller, which cannot be disabled and is limited
257 to 1024 file descriptors.
258
Willy Tarreau3eed10e2016-11-07 21:03:16 +0100259 -dr : ignore server address resolution failures. It is very common when
260 validating a configuration out of production not to have access to the same
261 resolvers and to fail on server address resolution, making it difficult to
262 test a configuration. This option simply appends the "none" method to the
263 list of address resolution methods for all servers, ensuring that even if
264 the libc fails to resolve an address, the startup sequence is not
265 interrupted.
266
Willy Tarreau70060452015-12-14 12:46:07 +0100267 -m <limit> : limit the total allocatable memory to <limit> megabytes across
268 all processes. This may cause some connection refusals or some slowdowns
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200269 depending on the amount of memory needed for normal operations. This is
Willy Tarreau70060452015-12-14 12:46:07 +0100270 mostly used to force the processes to work in a constrained resource usage
271 scenario. It is important to note that the memory is not shared between
272 processes, so in a multi-process scenario, this value is first divided by
273 global.nbproc before forking.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200274
275 -n <limit> : limits the per-process connection limit to <limit>. This is
276 equivalent to the global section's keyword "maxconn". It has precedence
277 over this keyword. This may be used to quickly force lower limits to avoid
278 a service outage on systems where resource limits are too low.
279
280 -p <file> : write all processes' pids into <file> during startup. This is
281 equivalent to the "global" section's keyword "pidfile". The file is opened
282 before entering the chroot jail, and after doing the chdir() implied by
283 "-C". Each pid appears on its own line.
284
285 -q : set "quiet" mode. This disables some messages during the configuration
286 parsing and during startup. It can be used in combination with "-c" to
287 just check if a configuration file is valid or not.
288
William Lallemand142db372018-12-11 18:56:45 +0100289 -S <bind>[,bind_options...]: in master-worker mode, bind a master CLI, which
290 allows the access to every processes, running or leaving ones.
291 For security reasons, it is recommended to bind the master CLI to a local
292 UNIX socket. The bind options are the same as the keyword "bind" in
293 the configuration file with words separated by commas instead of spaces.
294
295 Note that this socket can't be used to retrieve the listening sockets from
296 an old process during a seamless reload.
297
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200298 -sf <pid>* : send the "finish" signal (SIGUSR1) to older processes after boot
299 completion to ask them to finish what they are doing and to leave. <pid>
300 is a list of pids to signal (one per argument). The list ends on any
301 option starting with a "-". It is not a problem if the list of pids is
302 empty, so that it can be built on the fly based on the result of a command
303 like "pidof" or "pgrep".
304
305 -st <pid>* : send the "terminate" signal (SIGTERM) to older processes after
306 boot completion to terminate them immediately without finishing what they
307 were doing. <pid> is a list of pids to signal (one per argument). The list
308 is ends on any option starting with a "-". It is not a problem if the list
309 of pids is empty, so that it can be built on the fly based on the result of
310 a command like "pidof" or "pgrep".
311
312 -v : report the version and build date.
313
314 -vv : display the version, build options, libraries versions and usable
315 pollers. This output is systematically requested when filing a bug report.
316
Olivier Houchardd33fc3a2017-04-05 22:50:59 +0200317 -x <unix_socket> : connect to the specified socket and try to retrieve any
318 listening sockets from the old process, and use them instead of trying to
319 bind new ones. This is useful to avoid missing any new connection when
William Lallemandf6975e92017-05-26 17:42:10 +0200320 reloading the configuration on Linux. The capability must be enable on the
321 stats socket using "expose-fd listeners" in your configuration.
Olivier Houchardd33fc3a2017-04-05 22:50:59 +0200322
Dan Lloyd8e48b872016-07-01 21:01:18 -0400323A safe way to start HAProxy from an init file consists in forcing the daemon
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200324mode, storing existing pids to a pid file and using this pid file to notify
325older processes to finish before leaving :
326
327 haproxy -f /etc/haproxy.cfg \
328 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
329
330When the configuration is split into a few specific files (eg: tcp vs http),
331it is recommended to use the "-f" option :
332
333 haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
334 -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
335 -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
336 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
337
338When an unknown number of files is expected, such as customer-specific files,
339it is recommended to assign them a name starting with a fixed-size sequence
340number and to use "--" to load them, possibly after loading some defaults :
341
342 haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
343 -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
344 -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
345 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) \
346 -f /etc/haproxy/default-customers.cfg -- /etc/haproxy/customers/*
347
348Sometimes a failure to start may happen for whatever reason. Then it is
349important to verify if the version of HAProxy you are invoking is the expected
350version and if it supports the features you are expecting (eg: SSL, PCRE,
351compression, Lua, etc). This can be verified using "haproxy -vv". Some
352important information such as certain build options, the target system and
353the versions of the libraries being used are reported there. It is also what
354you will systematically be asked for when posting a bug report :
355
356 $ haproxy -vv
357 HA-Proxy version 1.6-dev7-a088d3-4 2015/10/08
358 Copyright 2000-2015 Willy Tarreau <willy@haproxy.org>
359
360 Build options :
361 TARGET = linux2628
362 CPU = generic
363 CC = gcc
364 CFLAGS = -pg -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement \
365 -DBUFSIZE=8030 -DMAXREWRITE=1030 -DSO_MARK=36 -DTCP_REPAIR=19
366 OPTIONS = USE_ZLIB=1 USE_DLMALLOC=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
367
368 Default settings :
369 maxconn = 2000, bufsize = 8030, maxrewrite = 1030, maxpollevents = 200
370
371 Encrypted password support via crypt(3): yes
372 Built with zlib version : 1.2.6
373 Compression algorithms supported : identity("identity"), deflate("deflate"), \
374 raw-deflate("deflate"), gzip("gzip")
375 Built with OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
376 Running on OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
377 OpenSSL library supports TLS extensions : yes
378 OpenSSL library supports SNI : yes
379 OpenSSL library supports prefer-server-ciphers : yes
380 Built with PCRE version : 8.12 2011-01-15
381 PCRE library supports JIT : no (USE_PCRE_JIT not set)
382 Built with Lua version : Lua 5.3.1
383 Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND
384
385 Available polling systems :
386 epoll : pref=300, test result OK
387 poll : pref=200, test result OK
388 select : pref=150, test result OK
389 Total: 3 (3 usable), will use epoll.
390
391The relevant information that many non-developer users can verify here are :
392 - the version : 1.6-dev7-a088d3-4 above means the code is currently at commit
393 ID "a088d3" which is the 4th one after after official version "1.6-dev7".
394 Version 1.6-dev7 would show as "1.6-dev7-8c1ad7". What matters here is in
395 fact "1.6-dev7". This is the 7th development version of what will become
396 version 1.6 in the future. A development version not suitable for use in
397 production (unless you know exactly what you are doing). A stable version
398 will show as a 3-numbers version, such as "1.5.14-16f863", indicating the
399 14th level of fix on top of version 1.5. This is a production-ready version.
400
401 - the release date : 2015/10/08. It is represented in the universal
402 year/month/day format. Here this means August 8th, 2015. Given that stable
403 releases are issued every few months (1-2 months at the beginning, sometimes
404 6 months once the product becomes very stable), if you're seeing an old date
405 here, it means you're probably affected by a number of bugs or security
406 issues that have since been fixed and that it might be worth checking on the
407 official site.
408
409 - build options : they are relevant to people who build their packages
410 themselves, they can explain why things are not behaving as expected. For
411 example the development version above was built for Linux 2.6.28 or later,
Dan Lloyd8e48b872016-07-01 21:01:18 -0400412 targeting a generic CPU (no CPU-specific optimizations), and lacks any
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200413 code optimization (-O0) so it will perform poorly in terms of performance.
414
415 - libraries versions : zlib version is reported as found in the library
416 itself. In general zlib is considered a very stable product and upgrades
417 are almost never needed. OpenSSL reports two versions, the version used at
418 build time and the one being used, as found on the system. These ones may
419 differ by the last letter but never by the numbers. The build date is also
420 reported because most OpenSSL bugs are security issues and need to be taken
421 seriously, so this library absolutely needs to be kept up to date. Seeing a
422 4-months old version here is highly suspicious and indeed an update was
423 missed. PCRE provides very fast regular expressions and is highly
424 recommended. Certain of its extensions such as JIT are not present in all
425 versions and still young so some people prefer not to build with them,
Dan Lloyd8e48b872016-07-01 21:01:18 -0400426 which is why the build status is reported as well. Regarding the Lua
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200427 scripting language, HAProxy expects version 5.3 which is very young since
428 it was released a little time before HAProxy 1.6. It is important to check
429 on the Lua web site if some fixes are proposed for this branch.
430
431 - Available polling systems will affect the process's scalability when
432 dealing with more than about one thousand of concurrent connections. These
433 ones are only available when the correct system was indicated in the TARGET
434 variable during the build. The "epoll" mechanism is highly recommended on
435 Linux, and the kqueue mechanism is highly recommended on BSD. Lacking them
436 will result in poll() or even select() being used, causing a high CPU usage
437 when dealing with a lot of connections.
438
439
4404. Stopping and restarting HAProxy
441----------------------------------
442
443HAProxy supports a graceful and a hard stop. The hard stop is simple, when the
444SIGTERM signal is sent to the haproxy process, it immediately quits and all
445established connections are closed. The graceful stop is triggered when the
446SIGUSR1 signal is sent to the haproxy process. It consists in only unbinding
447from listening ports, but continue to process existing connections until they
448close. Once the last connection is closed, the process leaves.
449
450The hard stop method is used for the "stop" or "restart" actions of the service
451management script. The graceful stop is used for the "reload" action which
452tries to seamlessly reload a new configuration in a new process.
453
454Both of these signals may be sent by the new haproxy process itself during a
455reload or restart, so that they are sent at the latest possible moment and only
456if absolutely required. This is what is performed by the "-st" (hard) and "-sf"
457(graceful) options respectively.
458
William Lallemande202b1e2017-06-01 17:38:56 +0200459In master-worker mode, it is not needed to start a new haproxy process in
460order to reload the configuration. The master process reacts to the SIGUSR2
461signal by reexecuting itself with the -sf parameter followed by the PIDs of
462the workers. The master will then parse the configuration file and fork new
463workers.
464
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200465To understand better how these signals are used, it is important to understand
466the whole restart mechanism.
467
468First, an existing haproxy process is running. The administrator uses a system
Jackie Tapia749f74c2020-07-22 18:59:40 -0500469specific command such as "/etc/init.d/haproxy reload" to indicate they want to
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200470take the new configuration file into effect. What happens then is the following.
471First, the service script (/etc/init.d/haproxy or equivalent) will verify that
472the configuration file parses correctly using "haproxy -c". After that it will
473try to start haproxy with this configuration file, using "-st" or "-sf".
474
475Then HAProxy tries to bind to all listening ports. If some fatal errors happen
476(eg: address not present on the system, permission denied), the process quits
477with an error. If a socket binding fails because a port is already in use, then
478the process will first send a SIGTTOU signal to all the pids specified in the
479"-st" or "-sf" pid list. This is what is called the "pause" signal. It instructs
480all existing haproxy processes to temporarily stop listening to their ports so
481that the new process can try to bind again. During this time, the old process
482continues to process existing connections. If the binding still fails (because
483for example a port is shared with another daemon), then the new process sends a
484SIGTTIN signal to the old processes to instruct them to resume operations just
485as if nothing happened. The old processes will then restart listening to the
486ports and continue to accept connections. Not that this mechanism is system
Dan Lloyd8e48b872016-07-01 21:01:18 -0400487dependent and some operating systems may not support it in multi-process mode.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200488
489If the new process manages to bind correctly to all ports, then it sends either
490the SIGTERM (hard stop in case of "-st") or the SIGUSR1 (graceful stop in case
491of "-sf") to all processes to notify them that it is now in charge of operations
492and that the old processes will have to leave, either immediately or once they
493have finished their job.
494
495It is important to note that during this timeframe, there are two small windows
496of a few milliseconds each where it is possible that a few connection failures
497will be noticed during high loads. Typically observed failure rates are around
4981 failure during a reload operation every 10000 new connections per second,
499which means that a heavily loaded site running at 30000 new connections per
500second may see about 3 failed connection upon every reload. The two situations
501where this happens are :
502
503 - if the new process fails to bind due to the presence of the old process,
504 it will first have to go through the SIGTTOU+SIGTTIN sequence, which
505 typically lasts about one millisecond for a few tens of frontends, and
506 during which some ports will not be bound to the old process and not yet
507 bound to the new one. HAProxy works around this on systems that support the
508 SO_REUSEPORT socket options, as it allows the new process to bind without
509 first asking the old one to unbind. Most BSD systems have been supporting
510 this almost forever. Linux has been supporting this in version 2.0 and
511 dropped it around 2.2, but some patches were floating around by then. It
512 was reintroduced in kernel 3.9, so if you are observing a connection
Dan Lloyd8e48b872016-07-01 21:01:18 -0400513 failure rate above the one mentioned above, please ensure that your kernel
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200514 is 3.9 or newer, or that relevant patches were backported to your kernel
515 (less likely).
516
517 - when the old processes close the listening ports, the kernel may not always
518 redistribute any pending connection that was remaining in the socket's
519 backlog. Under high loads, a SYN packet may happen just before the socket
520 is closed, and will lead to an RST packet being sent to the client. In some
521 critical environments where even one drop is not acceptable, these ones are
522 sometimes dealt with using firewall rules to block SYN packets during the
523 reload, forcing the client to retransmit. This is totally system-dependent,
524 as some systems might be able to visit other listening queues and avoid
525 this RST. A second case concerns the ACK from the client on a local socket
526 that was in SYN_RECV state just before the close. This ACK will lead to an
527 RST packet while the haproxy process is still not aware of it. This one is
Dan Lloyd8e48b872016-07-01 21:01:18 -0400528 harder to get rid of, though the firewall filtering rules mentioned above
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200529 will work well if applied one second or so before restarting the process.
530
531For the vast majority of users, such drops will never ever happen since they
532don't have enough load to trigger the race conditions. And for most high traffic
533users, the failure rate is still fairly within the noise margin provided that at
534least SO_REUSEPORT is properly supported on their systems.
535
536
5375. File-descriptor limitations
538------------------------------
539
540In order to ensure that all incoming connections will successfully be served,
541HAProxy computes at load time the total number of file descriptors that will be
542needed during the process's life. A regular Unix process is generally granted
5431024 file descriptors by default, and a privileged process can raise this limit
544itself. This is one reason for starting HAProxy as root and letting it adjust
545the limit. The default limit of 1024 file descriptors roughly allow about 500
546concurrent connections to be processed. The computation is based on the global
547maxconn parameter which limits the total number of connections per process, the
548number of listeners, the number of servers which have a health check enabled,
549the agent checks, the peers, the loggers and possibly a few other technical
550requirements. A simple rough estimate of this number consists in simply
551doubling the maxconn value and adding a few tens to get the approximate number
552of file descriptors needed.
553
554Originally HAProxy did not know how to compute this value, and it was necessary
555to pass the value using the "ulimit-n" setting in the global section. This
556explains why even today a lot of configurations are seen with this setting
557present. Unfortunately it was often miscalculated resulting in connection
558failures when approaching maxconn instead of throttling incoming connection
559while waiting for the needed resources. For this reason it is important to
Dan Lloyd8e48b872016-07-01 21:01:18 -0400560remove any vestigial "ulimit-n" setting that can remain from very old versions.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200561
562Raising the number of file descriptors to accept even moderate loads is
563mandatory but comes with some OS-specific adjustments. First, the select()
564polling system is limited to 1024 file descriptors. In fact on Linux it used
565to be capable of handling more but since certain OS ship with excessively
566restrictive SELinux policies forbidding the use of select() with more than
5671024 file descriptors, HAProxy now refuses to start in this case in order to
568avoid any issue at run time. On all supported operating systems, poll() is
569available and will not suffer from this limitation. It is automatically picked
Dan Lloyd8e48b872016-07-01 21:01:18 -0400570so there is nothing to do to get a working configuration. But poll's becomes
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200571very slow when the number of file descriptors increases. While HAProxy does its
572best to limit this performance impact (eg: via the use of the internal file
573descriptor cache and batched processing), a good rule of thumb is that using
574poll() with more than a thousand concurrent connections will use a lot of CPU.
575
576For Linux systems base on kernels 2.6 and above, the epoll() system call will
577be used. It's a much more scalable mechanism relying on callbacks in the kernel
578that guarantee a constant wake up time regardless of the number of registered
579monitored file descriptors. It is automatically used where detected, provided
580that HAProxy had been built for one of the Linux flavors. Its presence and
581support can be verified using "haproxy -vv".
582
583For BSD systems which support it, kqueue() is available as an alternative. It
584is much faster than poll() and even slightly faster than epoll() thanks to its
585batched handling of changes. At least FreeBSD and OpenBSD support it. Just like
586with Linux's epoll(), its support and availability are reported in the output
587of "haproxy -vv".
588
589Having a good poller is one thing, but it is mandatory that the process can
590reach the limits. When HAProxy starts, it immediately sets the new process's
591file descriptor limits and verifies if it succeeds. In case of failure, it
592reports it before forking so that the administrator can see the problem. As
593long as the process is started by as root, there should be no reason for this
594setting to fail. However, it can fail if the process is started by an
595unprivileged user. If there is a compelling reason for *not* starting haproxy
596as root (eg: started by end users, or by a per-application account), then the
597file descriptor limit can be raised by the system administrator for this
598specific user. The effectiveness of the setting can be verified by issuing
599"ulimit -n" from the user's command line. It should reflect the new limit.
600
601Warning: when an unprivileged user's limits are changed in this user's account,
602it is fairly common that these values are only considered when the user logs in
603and not at all in some scripts run at system boot time nor in crontabs. This is
604totally dependent on the operating system, keep in mind to check "ulimit -n"
605before starting haproxy when running this way. The general advice is never to
606start haproxy as an unprivileged user for production purposes. Another good
607reason is that it prevents haproxy from enabling some security protections.
608
609Once it is certain that the system will allow the haproxy process to use the
610requested number of file descriptors, two new system-specific limits may be
611encountered. The first one is the system-wide file descriptor limit, which is
612the total number of file descriptors opened on the system, covering all
613processes. When this limit is reached, accept() or socket() will typically
614return ENFILE. The second one is the per-process hard limit on the number of
615file descriptors, it prevents setrlimit() from being set higher. Both are very
616dependent on the operating system. On Linux, the system limit is set at boot
617based on the amount of memory. It can be changed with the "fs.file-max" sysctl.
618And the per-process hard limit is set to 1048576 by default, but it can be
619changed using the "fs.nr_open" sysctl.
620
621File descriptor limitations may be observed on a running process when they are
622set too low. The strace utility will report that accept() and socket() return
623"-1 EMFILE" when the process's limits have been reached. In this case, simply
624raising the "ulimit-n" value (or removing it) will solve the problem. If these
625system calls return "-1 ENFILE" then it means that the kernel's limits have
626been reached and that something must be done on a system-wide parameter. These
627trouble must absolutely be addressed, as they result in high CPU usage (when
628accept() fails) and failed connections that are generally visible to the user.
629One solution also consists in lowering the global maxconn value to enforce
630serialization, and possibly to disable HTTP keep-alive to force connections
631to be released and reused faster.
632
633
6346. Memory management
635--------------------
636
637HAProxy uses a simple and fast pool-based memory management. Since it relies on
638a small number of different object types, it's much more efficient to pick new
639objects from a pool which already contains objects of the appropriate size than
640to call malloc() for each different size. The pools are organized as a stack or
641LIFO, so that newly allocated objects are taken from recently released objects
642still hot in the CPU caches. Pools of similar sizes are merged together, in
643order to limit memory fragmentation.
644
645By default, since the focus is set on performance, each released object is put
646back into the pool it came from, and allocated objects are never freed since
647they are expected to be reused very soon.
648
649On the CLI, it is possible to check how memory is being used in pools thanks to
650the "show pools" command :
651
652 > show pools
653 Dumping pools usage. Use SIGQUIT to flush them.
Willy Tarreau0a93b642018-10-16 07:58:39 +0200654 - Pool cache_st (16 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccc40=03 [SHARED]
655 - Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 0 failures, 2 users, @0x9ccac0=00 [SHARED]
656 - Pool comp_state (48 bytes) : 3 allocated (144 bytes), 3 used, 0 failures, 5 users, @0x9cccc0=04 [SHARED]
657 - Pool filter (64 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 3 users, @0x9ccbc0=02 [SHARED]
658 - Pool vars (80 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccb40=01 [SHARED]
659 - Pool uniqueid (128 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9cd240=15 [SHARED]
660 - Pool task (144 bytes) : 55 allocated (7920 bytes), 55 used, 0 failures, 1 users, @0x9cd040=11 [SHARED]
661 - Pool session (160 bytes) : 1 allocated (160 bytes), 1 used, 0 failures, 1 users, @0x9cd140=13 [SHARED]
662 - Pool h2s (208 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccec0=08 [SHARED]
663 - Pool h2c (288 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cce40=07 [SHARED]
664 - Pool spoe_ctx (304 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccf40=09 [SHARED]
665 - Pool connection (400 bytes) : 2 allocated (800 bytes), 2 used, 0 failures, 1 users, @0x9cd1c0=14 [SHARED]
666 - Pool hdr_idx (416 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cd340=17 [SHARED]
667 - Pool dns_resolut (480 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccdc0=06 [SHARED]
668 - Pool dns_answer_ (576 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccd40=05 [SHARED]
669 - Pool stream (960 bytes) : 1 allocated (960 bytes), 1 used, 0 failures, 1 users, @0x9cd0c0=12 [SHARED]
670 - Pool requri (1024 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cd2c0=16 [SHARED]
671 - Pool buffer (8030 bytes) : 3 allocated (24090 bytes), 2 used, 0 failures, 1 users, @0x9cd3c0=18 [SHARED]
672 - Pool trash (8062 bytes) : 1 allocated (8062 bytes), 1 used, 0 failures, 1 users, @0x9cd440=19
673 Total: 19 pools, 42296 bytes allocated, 34266 used.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200674
675The pool name is only indicative, it's the name of the first object type using
676this pool. The size in parenthesis is the object size for objects in this pool.
677Object sizes are always rounded up to the closest multiple of 16 bytes. The
678number of objects currently allocated and the equivalent number of bytes is
679reported so that it is easy to know which pool is responsible for the highest
680memory usage. The number of objects currently in use is reported as well in the
681"used" field. The difference between "allocated" and "used" corresponds to the
Willy Tarreau0a93b642018-10-16 07:58:39 +0200682objects that have been freed and are available for immediate use. The address
683at the end of the line is the pool's address, and the following number is the
684pool index when it exists, or is reported as -1 if no index was assigned.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200685
686It is possible to limit the amount of memory allocated per process using the
687"-m" command line option, followed by a number of megabytes. It covers all of
688the process's addressable space, so that includes memory used by some libraries
689as well as the stack, but it is a reliable limit when building a resource
690constrained system. It works the same way as "ulimit -v" on systems which have
691it, or "ulimit -d" for the other ones.
692
693If a memory allocation fails due to the memory limit being reached or because
694the system doesn't have any enough memory, then haproxy will first start to
695free all available objects from all pools before attempting to allocate memory
696again. This mechanism of releasing unused memory can be triggered by sending
697the signal SIGQUIT to the haproxy process. When doing so, the pools state prior
698to the flush will also be reported to stderr when the process runs in
699foreground.
700
701During a reload operation, the process switched to the graceful stop state also
702automatically performs some flushes after releasing any connection so that all
703possible memory is released to save it for the new process.
704
705
7067. CPU usage
707------------
708
709HAProxy normally spends most of its time in the system and a smaller part in
710userland. A finely tuned 3.5 GHz CPU can sustain a rate about 80000 end-to-end
711connection setups and closes per second at 100% CPU on a single core. When one
712core is saturated, typical figures are :
713 - 95% system, 5% user for long TCP connections or large HTTP objects
714 - 85% system and 15% user for short TCP connections or small HTTP objects in
715 close mode
716 - 70% system and 30% user for small HTTP objects in keep-alive mode
717
718The amount of rules processing and regular expressions will increase the user
719land part. The presence of firewall rules, connection tracking, complex routing
720tables in the system will instead increase the system part.
721
722On most systems, the CPU time observed during network transfers can be cut in 4
723parts :
724 - the interrupt part, which concerns all the processing performed upon I/O
725 receipt, before the target process is even known. Typically Rx packets are
726 accounted for in interrupt. On some systems such as Linux where interrupt
727 processing may be deferred to a dedicated thread, it can appear as softirq,
728 and the thread is called ksoftirqd/0 (for CPU 0). The CPU taking care of
729 this load is generally defined by the hardware settings, though in the case
730 of softirq it is often possible to remap the processing to another CPU.
731 This interrupt part will often be perceived as parasitic since it's not
732 associated with any process, but it actually is some processing being done
733 to prepare the work for the process.
734
735 - the system part, which concerns all the processing done using kernel code
736 called from userland. System calls are accounted as system for example. All
737 synchronously delivered Tx packets will be accounted for as system time. If
738 some packets have to be deferred due to queues filling up, they may then be
739 processed in interrupt context later (eg: upon receipt of an ACK opening a
740 TCP window).
741
742 - the user part, which exclusively runs application code in userland. HAProxy
743 runs exclusively in this part, though it makes heavy use of system calls.
744 Rules processing, regular expressions, compression, encryption all add to
745 the user portion of CPU consumption.
746
747 - the idle part, which is what the CPU does when there is nothing to do. For
748 example HAProxy waits for an incoming connection, or waits for some data to
749 leave, meaning the system is waiting for an ACK from the client to push
750 these data.
751
752In practice regarding HAProxy's activity, it is in general reasonably accurate
753(but totally inexact) to consider that interrupt/softirq are caused by Rx
754processing in kernel drivers, that user-land is caused by layer 7 processing
755in HAProxy, and that system time is caused by network processing on the Tx
756path.
757
758Since HAProxy runs around an event loop, it waits for new events using poll()
759(or any alternative) and processes all these events as fast as possible before
760going back to poll() waiting for new events. It measures the time spent waiting
761in poll() compared to the time spent doing processing events. The ratio of
762polling time vs total time is called the "idle" time, it's the amount of time
763spent waiting for something to happen. This ratio is reported in the stats page
764on the "idle" line, or "Idle_pct" on the CLI. When it's close to 100%, it means
765the load is extremely low. When it's close to 0%, it means that there is
766constantly some activity. While it cannot be very accurate on an overloaded
767system due to other processes possibly preempting the CPU from the haproxy
768process, it still provides a good estimate about how HAProxy considers it is
769working : if the load is low and the idle ratio is low as well, it may indicate
770that HAProxy has a lot of work to do, possibly due to very expensive rules that
771have to be processed. Conversely, if HAProxy indicates the idle is close to
772100% while things are slow, it means that it cannot do anything to speed things
773up because it is already waiting for incoming data to process. In the example
774below, haproxy is completely idle :
775
776 $ echo "show info" | socat - /var/run/haproxy.sock | grep ^Idle
777 Idle_pct: 100
778
779When the idle ratio starts to become very low, it is important to tune the
780system and place processes and interrupts correctly to save the most possible
781CPU resources for all tasks. If a firewall is present, it may be worth trying
782to disable it or to tune it to ensure it is not responsible for a large part
783of the performance limitation. It's worth noting that unloading a stateful
784firewall generally reduces both the amount of interrupt/softirq and of system
785usage since such firewalls act both on the Rx and the Tx paths. On Linux,
786unloading the nf_conntrack and ip_conntrack modules will show whether there is
787anything to gain. If so, then the module runs with default settings and you'll
788have to figure how to tune it for better performance. In general this consists
789in considerably increasing the hash table size. On FreeBSD, "pfctl -d" will
790disable the "pf" firewall and its stateful engine at the same time.
791
792If it is observed that a lot of time is spent in interrupt/softirq, it is
793important to ensure that they don't run on the same CPU. Most systems tend to
794pin the tasks on the CPU where they receive the network traffic because for
795certain workloads it improves things. But with heavily network-bound workloads
796it is the opposite as the haproxy process will have to fight against its kernel
797counterpart. Pinning haproxy to one CPU core and the interrupts to another one,
798all sharing the same L3 cache tends to sensibly increase network performance
799because in practice the amount of work for haproxy and the network stack are
800quite close, so they can almost fill an entire CPU each. On Linux this is done
801using taskset (for haproxy) or using cpu-map (from the haproxy config), and the
802interrupts are assigned under /proc/irq. Many network interfaces support
803multiple queues and multiple interrupts. In general it helps to spread them
804across a small number of CPU cores provided they all share the same L3 cache.
805Please always stop irq_balance which always does the worst possible thing on
806such workloads.
807
808For CPU-bound workloads consisting in a lot of SSL traffic or a lot of
809compression, it may be worth using multiple processes dedicated to certain
810tasks, though there is no universal rule here and experimentation will have to
811be performed.
812
813In order to increase the CPU capacity, it is possible to make HAProxy run as
814several processes, using the "nbproc" directive in the global section. There
815are some limitations though :
816 - health checks are run per process, so the target servers will get as many
817 checks as there are running processes ;
818 - maxconn values and queues are per-process so the correct value must be set
819 to avoid overloading the servers ;
820 - outgoing connections should avoid using port ranges to avoid conflicts
821 - stick-tables are per process and are not shared between processes ;
822 - each peers section may only run on a single process at a time ;
823 - the CLI operations will only act on a single process at a time.
824
825With this in mind, it appears that the easiest setup often consists in having
826one first layer running on multiple processes and in charge for the heavy
827processing, passing the traffic to a second layer running in a single process.
828This mechanism is suited to SSL and compression which are the two CPU-heavy
829features. Instances can easily be chained over UNIX sockets (which are cheaper
fengpeiyuancc123c62016-01-15 16:40:53 +0800830than TCP sockets and which do not waste ports), and the proxy protocol which is
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200831useful to pass client information to the next stage. When doing so, it is
832generally a good idea to bind all the single-process tasks to process number 1
833and extra tasks to next processes, as this will make it easier to generate
834similar configurations for different machines.
835
836On Linux versions 3.9 and above, running HAProxy in multi-process mode is much
837more efficient when each process uses a distinct listening socket on the same
838IP:port ; this will make the kernel evenly distribute the load across all
839processes instead of waking them all up. Please check the "process" option of
840the "bind" keyword lines in the configuration manual for more information.
841
842
8438. Logging
844----------
845
846For logging, HAProxy always relies on a syslog server since it does not perform
847any file-system access. The standard way of using it is to send logs over UDP
848to the log server (by default on port 514). Very commonly this is configured to
849127.0.0.1 where the local syslog daemon is running, but it's also used over the
850network to log to a central server. The central server provides additional
851benefits especially in active-active scenarios where it is desirable to keep
852the logs merged in arrival order. HAProxy may also make use of a UNIX socket to
853send its logs to the local syslog daemon, but it is not recommended at all,
854because if the syslog server is restarted while haproxy runs, the socket will
855be replaced and new logs will be lost. Since HAProxy will be isolated inside a
856chroot jail, it will not have the ability to reconnect to the new socket. It
857has also been observed in field that the log buffers in use on UNIX sockets are
858very small and lead to lost messages even at very light loads. But this can be
859fine for testing however.
860
861It is recommended to add the following directive to the "global" section to
862make HAProxy log to the local daemon using facility "local0" :
863
864 log 127.0.0.1:514 local0
865
866and then to add the following one to each "defaults" section or to each frontend
867and backend section :
868
869 log global
870
871This way, all logs will be centralized through the global definition of where
872the log server is.
873
874Some syslog daemons do not listen to UDP traffic by default, so depending on
875the daemon being used, the syntax to enable this will vary :
876
877 - on sysklogd, you need to pass argument "-r" on the daemon's command line
878 so that it listens to a UDP socket for "remote" logs ; note that there is
879 no way to limit it to address 127.0.0.1 so it will also receive logs from
880 remote systems ;
881
882 - on rsyslogd, the following lines must be added to the configuration file :
883
884 $ModLoad imudp
885 $UDPServerAddress *
886 $UDPServerRun 514
887
888 - on syslog-ng, a new source can be created the following way, it then needs
889 to be added as a valid source in one of the "log" directives :
890
891 source s_udp {
892 udp(ip(127.0.0.1) port(514));
893 };
894
895Please consult your syslog daemon's manual for more information. If no logs are
896seen in the system's log files, please consider the following tests :
897
898 - restart haproxy. Each frontend and backend logs one line indicating it's
899 starting. If these logs are received, it means logs are working.
900
901 - run "strace -tt -s100 -etrace=sendmsg -p <haproxy's pid>" and perform some
902 activity that you expect to be logged. You should see the log messages
903 being sent using sendmsg() there. If they don't appear, restart using
904 strace on top of haproxy. If you still see no logs, it definitely means
905 that something is wrong in your configuration.
906
907 - run tcpdump to watch for port 514, for example on the loopback interface if
908 the traffic is being sent locally : "tcpdump -As0 -ni lo port 514". If the
909 packets are seen there, it's the proof they're sent then the syslogd daemon
910 needs to be troubleshooted.
911
912While traffic logs are sent from the frontends (where the incoming connections
913are accepted), backends also need to be able to send logs in order to report a
914server state change consecutive to a health check. Please consult HAProxy's
915configuration manual for more information regarding all possible log settings.
916
Dan Lloyd8e48b872016-07-01 21:01:18 -0400917It is convenient to chose a facility that is not used by other daemons. HAProxy
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200918examples often suggest "local0" for traffic logs and "local1" for admin logs
919because they're never seen in field. A single facility would be enough as well.
920Having separate logs is convenient for log analysis, but it's also important to
921remember that logs may sometimes convey confidential information, and as such
Dan Lloyd8e48b872016-07-01 21:01:18 -0400922they must not be mixed with other logs that may accidentally be handed out to
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200923unauthorized people.
924
925For in-field troubleshooting without impacting the server's capacity too much,
926it is recommended to make use of the "halog" utility provided with HAProxy.
927This is sort of a grep-like utility designed to process HAProxy log files at
928a very fast data rate. Typical figures range between 1 and 2 GB of logs per
929second. It is capable of extracting only certain logs (eg: search for some
930classes of HTTP status codes, connection termination status, search by response
931time ranges, look for errors only), count lines, limit the output to a number
932of lines, and perform some more advanced statistics such as sorting servers
933by response time or error counts, sorting URLs by time or count, sorting client
934addresses by access count, and so on. It is pretty convenient to quickly spot
935anomalies such as a bot looping on the site, and block them.
936
937
9389. Statistics and monitoring
939----------------------------
940
Willy Tarreau44aed902015-10-13 14:45:29 +0200941It is possible to query HAProxy about its status. The most commonly used
942mechanism is the HTTP statistics page. This page also exposes an alternative
943CSV output format for monitoring tools. The same format is provided on the
944Unix socket.
945
Amaury Denoyelle072f97e2020-10-05 11:49:37 +0200946Statistics are regroup in categories labelled as domains, corresponding to the
Ilya Shipitsin2272d8a2020-12-21 01:22:40 +0500947multiple components of HAProxy. There are two domains available: proxy and dns.
Amaury Denoyellefbd0bc92020-10-05 11:49:46 +0200948If not specified, the proxy domain is selected. Note that only the proxy
949statistics are printed on the HTTP page.
Willy Tarreau44aed902015-10-13 14:45:29 +0200950
9519.1. CSV format
952---------------
953
954The statistics may be consulted either from the unix socket or from the HTTP
955page. Both means provide a CSV format whose fields follow. The first line
956begins with a sharp ('#') and has one word per comma-delimited field which
957represents the title of the column. All other lines starting at the second one
958use a classical CSV format using a comma as the delimiter, and the double quote
959('"') as an optional text delimiter, but only if the enclosed text is ambiguous
960(if it contains a quote or a comma). The double-quote character ('"') in the
961text is doubled ('""'), which is the format that most tools recognize. Please
962do not insert any column before these ones in order not to break tools which
963use hard-coded column positions.
964
Amaury Denoyelle50660a82020-10-05 11:49:39 +0200965For proxy statistics, after each field name, the types which may have a value
966for that field are specified in brackets. The types are L (Listeners), F
967(Frontends), B (Backends), and S (Servers). There is a fixed set of static
968fields that are always available in the same order. A column containing the
969character '-' delimits the end of the static fields, after which presence or
970order of the fields are not guaranteed.
Willy Tarreau44aed902015-10-13 14:45:29 +0200971
Amaury Denoyelle50660a82020-10-05 11:49:39 +0200972Here is the list of static fields using the proxy statistics domain:
Willy Tarreau44aed902015-10-13 14:45:29 +0200973 0. pxname [LFBS]: proxy name
974 1. svname [LFBS]: service name (FRONTEND for frontend, BACKEND for backend,
975 any name for server/listener)
976 2. qcur [..BS]: current queued requests. For the backend this reports the
977 number queued without a server assigned.
978 3. qmax [..BS]: max value of qcur
979 4. scur [LFBS]: current sessions
980 5. smax [LFBS]: max sessions
981 6. slim [LFBS]: configured session limit
Willy Tarreauc73810f2016-01-11 13:52:04 +0100982 7. stot [LFBS]: cumulative number of sessions
Willy Tarreau44aed902015-10-13 14:45:29 +0200983 8. bin [LFBS]: bytes in
984 9. bout [LFBS]: bytes out
985 10. dreq [LFB.]: requests denied because of security concerns.
986 - For tcp this is because of a matched tcp-request content rule.
987 - For http this is because of a matched http-request or tarpit rule.
988 11. dresp [LFBS]: responses denied because of security concerns.
989 - For http this is because of a matched http-request rule, or
990 "option checkcache".
991 12. ereq [LF..]: request errors. Some of the possible causes are:
992 - early termination from the client, before the request has been sent.
993 - read error from the client
994 - client timeout
995 - client closed connection
996 - various bad requests from the client.
997 - request was tarpitted.
998 13. econ [..BS]: number of requests that encountered an error trying to
999 connect to a backend server. The backend stat is the sum of the stat
1000 for all servers of that backend, plus any connection errors not
1001 associated with a particular server (such as the backend having no
1002 active servers).
1003 14. eresp [..BS]: response errors. srv_abrt will be counted here also.
1004 Some other errors are:
1005 - write error on the client socket (won't be counted for the server stat)
1006 - failure applying filters to the response.
1007 15. wretr [..BS]: number of times a connection to a server was retried.
1008 16. wredis [..BS]: number of times a request was redispatched to another
1009 server. The server value counts the number of times that server was
1010 switched away from.
Willy Tarreaub96dd282016-11-09 14:45:51 +01001011 17. status [LFBS]: status (UP/DOWN/NOLB/MAINT/MAINT(via)/MAINT(resolution)...)
Willy Tarreaubd715102020-10-23 22:44:30 +02001012 18. weight [..BS]: total effective weight (backend), effective weight (server)
Willy Tarreau44aed902015-10-13 14:45:29 +02001013 19. act [..BS]: number of active servers (backend), server is active (server)
1014 20. bck [..BS]: number of backup servers (backend), server is backup (server)
1015 21. chkfail [...S]: number of failed checks. (Only counts checks failed when
1016 the server is up.)
1017 22. chkdown [..BS]: number of UP->DOWN transitions. The backend counter counts
1018 transitions to the whole backend being down, rather than the sum of the
1019 counters for each server.
1020 23. lastchg [..BS]: number of seconds since the last UP<->DOWN transition
1021 24. downtime [..BS]: total downtime (in seconds). The value for the backend
1022 is the downtime for the whole backend, not the sum of the server downtime.
1023 25. qlimit [...S]: configured maxqueue for the server, or nothing in the
1024 value is 0 (default, meaning no limit)
1025 26. pid [LFBS]: process id (0 for first instance, 1 for second, ...)
1026 27. iid [LFBS]: unique proxy id
1027 28. sid [L..S]: server id (unique inside a proxy)
1028 29. throttle [...S]: current throttle percentage for the server, when
1029 slowstart is active, or no value if not in slowstart.
1030 30. lbtot [..BS]: total number of times a server was selected, either for new
1031 sessions, or when re-dispatching. The server counter is the number
1032 of times that server was selected.
1033 31. tracked [...S]: id of proxy/server if tracking is enabled.
1034 32. type [LFBS]: (0=frontend, 1=backend, 2=server, 3=socket/listener)
1035 33. rate [.FBS]: number of sessions per second over last elapsed second
1036 34. rate_lim [.F..]: configured limit on new sessions per second
1037 35. rate_max [.FBS]: max number of new sessions per second
1038 36. check_status [...S]: status of last health check, one of:
1039 UNK -> unknown
1040 INI -> initializing
1041 SOCKERR -> socket error
1042 L4OK -> check passed on layer 4, no upper layers testing enabled
1043 L4TOUT -> layer 1-4 timeout
1044 L4CON -> layer 1-4 connection problem, for example
1045 "Connection refused" (tcp rst) or "No route to host" (icmp)
1046 L6OK -> check passed on layer 6
1047 L6TOUT -> layer 6 (SSL) timeout
1048 L6RSP -> layer 6 invalid response - protocol error
1049 L7OK -> check passed on layer 7
1050 L7OKC -> check conditionally passed on layer 7, for example 404 with
1051 disable-on-404
1052 L7TOUT -> layer 7 (HTTP/SMTP) timeout
1053 L7RSP -> layer 7 invalid response - protocol error
1054 L7STS -> layer 7 response error, for example HTTP 5xx
Daniel Schnellerb6c8b0d2017-09-01 19:13:55 +02001055 Notice: If a check is currently running, the last known status will be
1056 reported, prefixed with "* ". e. g. "* L7OK".
Willy Tarreau44aed902015-10-13 14:45:29 +02001057 37. check_code [...S]: layer5-7 code, if available
1058 38. check_duration [...S]: time in ms took to finish last health check
1059 39. hrsp_1xx [.FBS]: http responses with 1xx code
1060 40. hrsp_2xx [.FBS]: http responses with 2xx code
1061 41. hrsp_3xx [.FBS]: http responses with 3xx code
1062 42. hrsp_4xx [.FBS]: http responses with 4xx code
1063 43. hrsp_5xx [.FBS]: http responses with 5xx code
1064 44. hrsp_other [.FBS]: http responses with other codes (protocol error)
1065 45. hanafail [...S]: failed health checks details
1066 46. req_rate [.F..]: HTTP requests per second over last elapsed second
1067 47. req_rate_max [.F..]: max number of HTTP requests per second observed
Willy Tarreaufb981bd2016-12-12 14:31:46 +01001068 48. req_tot [.FB.]: total number of HTTP requests received
Willy Tarreau44aed902015-10-13 14:45:29 +02001069 49. cli_abrt [..BS]: number of data transfers aborted by the client
1070 50. srv_abrt [..BS]: number of data transfers aborted by the server
1071 (inc. in eresp)
1072 51. comp_in [.FB.]: number of HTTP response bytes fed to the compressor
1073 52. comp_out [.FB.]: number of HTTP response bytes emitted by the compressor
1074 53. comp_byp [.FB.]: number of bytes that bypassed the HTTP compressor
1075 (CPU/BW limit)
1076 54. comp_rsp [.FB.]: number of HTTP responses that were compressed
1077 55. lastsess [..BS]: number of seconds since last session assigned to
1078 server/backend
1079 56. last_chk [...S]: last health check contents or textual error
1080 57. last_agt [...S]: last agent check contents or textual error
1081 58. qtime [..BS]: the average queue time in ms over the 1024 last requests
1082 59. ctime [..BS]: the average connect time in ms over the 1024 last requests
1083 60. rtime [..BS]: the average response time in ms over the 1024 last requests
1084 (0 for TCP)
1085 61. ttime [..BS]: the average total session time in ms over the 1024 last
1086 requests
Willy Tarreau7f618842016-01-08 11:40:03 +01001087 62. agent_status [...S]: status of last agent check, one of:
1088 UNK -> unknown
1089 INI -> initializing
1090 SOCKERR -> socket error
1091 L4OK -> check passed on layer 4, no upper layers testing enabled
1092 L4TOUT -> layer 1-4 timeout
1093 L4CON -> layer 1-4 connection problem, for example
1094 "Connection refused" (tcp rst) or "No route to host" (icmp)
1095 L7OK -> agent reported "up"
1096 L7STS -> agent reported "fail", "stop", or "down"
1097 63. agent_code [...S]: numeric code reported by agent if any (unused for now)
1098 64. agent_duration [...S]: time in ms taken to finish last check
Willy Tarreaudd7354b2016-01-08 13:47:26 +01001099 65. check_desc [...S]: short human-readable description of check_status
1100 66. agent_desc [...S]: short human-readable description of agent_status
Willy Tarreau3141f592016-01-08 14:25:28 +01001101 67. check_rise [...S]: server's "rise" parameter used by checks
1102 68. check_fall [...S]: server's "fall" parameter used by checks
1103 69. check_health [...S]: server's health check value between 0 and rise+fall-1
1104 70. agent_rise [...S]: agent's "rise" parameter, normally 1
1105 71. agent_fall [...S]: agent's "fall" parameter, normally 1
1106 72. agent_health [...S]: agent's health parameter, between 0 and rise+fall-1
Willy Tarreaua6f5a732016-01-08 16:59:56 +01001107 73. addr [L..S]: address:port or "unix". IPv6 has brackets around the address.
Willy Tarreaue4847c62016-01-08 15:43:54 +01001108 74: cookie [..BS]: server's cookie value or backend's cookie name
Willy Tarreauf8211df2016-01-11 14:09:38 +01001109 75: mode [LFBS]: proxy mode (tcp, http, health, unknown)
Willy Tarreauf1516d92016-01-11 14:48:36 +01001110 76: algo [..B.]: load balancing algorithm
Willy Tarreauc73810f2016-01-11 13:52:04 +01001111 77: conn_rate [.F..]: number of connections over the last elapsed second
1112 78: conn_rate_max [.F..]: highest known conn_rate
1113 79: conn_tot [.F..]: cumulative number of connections
Willy Tarreau5b9bdff2016-01-11 14:40:47 +01001114 80: intercepted [.FB.]: cum. number of intercepted requests (monitor, stats)
Willy Tarreau8a90b8e2016-10-21 18:15:32 +02001115 81: dcon [LF..]: requests denied by "tcp-request connection" rules
Willy Tarreaua5bc36b2016-10-21 18:16:27 +02001116 82: dses [LF..]: requests denied by "tcp-request session" rules
Willy Tarreauea96a822018-05-28 15:15:43 +02001117 83: wrew [LFBS]: cumulative number of failed header rewriting warnings
Jérôme Magnin708eb882019-07-17 09:24:46 +02001118 84: connect [..BS]: cumulative number of connection establishment attempts
1119 85: reuse [..BS]: cumulative number of connection reuses
Willy Tarreau72974292019-11-08 07:29:34 +01001120 86: cache_lookups [.FB.]: cumulative number of cache lookups
Jérôme Magnin34ebb5c2019-07-17 14:04:40 +02001121 87: cache_hits [.FB.]: cumulative number of cache hits
Christopher Faulet2ac25742019-11-08 15:27:27 +01001122 88: srv_icur [...S]: current number of idle connections available for reuse
1123 89: src_ilim [...S]: limit on the number of available idle connections
1124 90. qtime_max [..BS]: the maximum observed queue time in ms
1125 91. ctime_max [..BS]: the maximum observed connect time in ms
1126 92. rtime_max [..BS]: the maximum observed response time in ms (0 for TCP)
1127 93. ttime_max [..BS]: the maximum observed total session time in ms
Christopher Faulet0159ee42019-12-16 14:40:39 +01001128 94. eint [LFBS]: cumulative number of internal errors
Pierre Cheynier08eb7182020-10-08 16:37:14 +02001129 95. idle_conn_cur [...S]: current number of unsafe idle connections
1130 96. safe_conn_cur [...S]: current number of safe idle connections
1131 97. used_conn_cur [...S]: current number of connections in use
1132 98. need_conn_est [...S]: estimated needed number of connections
Willy Tarreaubd715102020-10-23 22:44:30 +02001133 99. uweight [..BS]: total user weight (backend), server user weight (server)
Willy Tarreau44aed902015-10-13 14:45:29 +02001134
Amaury Denoyelle50660a82020-10-05 11:49:39 +02001135For all other statistics domains, the presence or the order of the fields are
1136not guaranteed. In this case, the header line should always be used to parse
1137the CSV data.
Willy Tarreau44aed902015-10-13 14:45:29 +02001138
Phil Schererb931f962020-12-02 19:36:08 +000011399.2. Typed output format
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001140------------------------
1141
1142Both "show info" and "show stat" support a mode where each output value comes
1143with its type and sufficient information to know how the value is supposed to
1144be aggregated between processes and how it evolves.
1145
1146In all cases, the output consists in having a single value per line with all
1147the information split into fields delimited by colons (':').
1148
1149The first column designates the object or metric being dumped. Its format is
1150specific to the command producing this output and will not be described in this
1151section. Usually it will consist in a series of identifiers and field names.
1152
1153The second column contains 3 characters respectively indicating the origin, the
1154nature and the scope of the value being reported. The first character (the
1155origin) indicates where the value was extracted from. Possible characters are :
1156
1157 M The value is a metric. It is valid at one instant any may change depending
1158 on its nature .
1159
1160 S The value is a status. It represents a discrete value which by definition
1161 cannot be aggregated. It may be the status of a server ("UP" or "DOWN"),
1162 the PID of the process, etc.
1163
1164 K The value is a sorting key. It represents an identifier which may be used
1165 to group some values together because it is unique among its class. All
1166 internal identifiers are keys. Some names can be listed as keys if they
1167 are unique (eg: a frontend name is unique). In general keys come from the
Dan Lloyd8e48b872016-07-01 21:01:18 -04001168 configuration, even though some of them may automatically be assigned. For
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001169 most purposes keys may be considered as equivalent to configuration.
1170
1171 C The value comes from the configuration. Certain configuration values make
1172 sense on the output, for example a concurrent connection limit or a cookie
1173 name. By definition these values are the same in all processes started
1174 from the same configuration file.
1175
1176 P The value comes from the product itself. There are very few such values,
1177 most common use is to report the product name, version and release date.
1178 These elements are also the same between all processes.
1179
1180The second character (the nature) indicates the nature of the information
1181carried by the field in order to let an aggregator decide on what operation to
1182use to aggregate multiple values. Possible characters are :
1183
1184 A The value represents an age since a last event. This is a bit different
1185 from the duration in that an age is automatically computed based on the
1186 current date. A typical example is how long ago did the last session
1187 happen on a server. Ages are generally aggregated by taking the minimum
1188 value and do not need to be stored.
1189
1190 a The value represents an already averaged value. The average response times
1191 and server weights are of this nature. Averages can typically be averaged
1192 between processes.
1193
1194 C The value represents a cumulative counter. Such measures perpetually
1195 increase until they wrap around. Some monitoring protocols need to tell
1196 the difference between a counter and a gauge to report a different type.
1197 In general counters may simply be summed since they represent events or
1198 volumes. Examples of metrics of this nature are connection counts or byte
1199 counts.
1200
1201 D The value represents a duration for a status. There are a few usages of
1202 this, most of them include the time taken by the last health check and
1203 the time a server has spent down. Durations are generally not summed,
1204 most of the time the maximum will be retained to compute an SLA.
1205
1206 G The value represents a gauge. It's a measure at one instant. The memory
1207 usage or the current number of active connections are of this nature.
1208 Metrics of this type are typically summed during aggregation.
1209
1210 L The value represents a limit (generally a configured one). By nature,
1211 limits are harder to aggregate since they are specific to the point where
1212 they were retrieved. In certain situations they may be summed or be kept
1213 separate.
1214
1215 M The value represents a maximum. In general it will apply to a gauge and
1216 keep the highest known value. An example of such a metric could be the
1217 maximum amount of concurrent connections that was encountered in the
1218 product's life time. To correctly aggregate maxima, you are supposed to
1219 output a range going from the maximum of all maxima and the sum of all
1220 of them. There is indeed no way to know if they were encountered
1221 simultaneously or not.
1222
1223 m The value represents a minimum. In general it will apply to a gauge and
1224 keep the lowest known value. An example of such a metric could be the
1225 minimum amount of free memory pools that was encountered in the product's
1226 life time. To correctly aggregate minima, you are supposed to output a
1227 range going from the minimum of all minima and the sum of all of them.
1228 There is indeed no way to know if they were encountered simultaneously
1229 or not.
1230
1231 N The value represents a name, so it is a string. It is used to report
1232 proxy names, server names and cookie names. Names have configuration or
1233 keys as their origin and are supposed to be the same among all processes.
1234
1235 O The value represents a free text output. Outputs from various commands,
1236 returns from health checks, node descriptions are of such nature.
1237
1238 R The value represents an event rate. It's a measure at one instant. It is
1239 quite similar to a gauge except that the recipient knows that this measure
1240 moves slowly and may decide not to keep all values. An example of such a
1241 metric is the measured amount of connections per second. Metrics of this
1242 type are typically summed during aggregation.
1243
1244 T The value represents a date or time. A field emitting the current date
1245 would be of this type. The method to aggregate such information is left
1246 as an implementation choice. For now no field uses this type.
1247
1248The third character (the scope) indicates what extent the value reflects. Some
1249elements may be per process while others may be per configuration or per system.
1250The distinction is important to know whether or not a single value should be
1251kept during aggregation or if values have to be aggregated. The following
1252characters are currently supported :
1253
1254 C The value is valid for a whole cluster of nodes, which is the set of nodes
1255 communicating over the peers protocol. An example could be the amount of
1256 entries present in a stick table that is replicated with other peers. At
1257 the moment no metric use this scope.
1258
1259 P The value is valid only for the process reporting it. Most metrics use
1260 this scope.
1261
1262 S The value is valid for the whole service, which is the set of processes
1263 started together from the same configuration file. All metrics originating
1264 from the configuration use this scope. Some other metrics may use it as
1265 well for some shared resources (eg: shared SSL cache statistics).
1266
1267 s The value is valid for the whole system, such as the system's hostname,
1268 current date or resource usage. At the moment this scope is not used by
1269 any metric.
1270
1271Consumers of these information will generally have enough of these 3 characters
1272to determine how to accurately report aggregated information across multiple
1273processes.
1274
1275After this column, the third column indicates the type of the field, among "s32"
1276(signed 32-bit integer), "s64" (signed 64-bit integer), "u32" (unsigned 32-bit
1277integer), "u64" (unsigned 64-bit integer), "str" (string). It is important to
1278know the type before parsing the value in order to properly read it. For example
1279a string containing only digits is still a string an not an integer (eg: an
1280error code extracted by a check).
1281
1282Then the fourth column is the value itself, encoded according to its type.
1283Strings are dumped as-is immediately after the colon without any leading space.
1284If a string contains a colon, it will appear normally. This means that the
1285output should not be exclusively split around colons or some check outputs
1286or server addresses might be truncated.
1287
1288
12899.3. Unix Socket commands
Willy Tarreau44aed902015-10-13 14:45:29 +02001290-------------------------
1291
1292The stats socket is not enabled by default. In order to enable it, it is
1293necessary to add one line in the global section of the haproxy configuration.
1294A second line is recommended to set a larger timeout, always appreciated when
1295issuing commands by hand :
1296
1297 global
1298 stats socket /var/run/haproxy.sock mode 600 level admin
1299 stats timeout 2m
1300
1301It is also possible to add multiple instances of the stats socket by repeating
1302the line, and make them listen to a TCP port instead of a UNIX socket. This is
1303never done by default because this is dangerous, but can be handy in some
1304situations :
1305
1306 global
1307 stats socket /var/run/haproxy.sock mode 600 level admin
1308 stats socket ipv4@192.168.0.1:9999 level admin
1309 stats timeout 2m
1310
1311To access the socket, an external utility such as "socat" is required. Socat is
1312a swiss-army knife to connect anything to anything. We use it to connect
1313terminals to the socket, or a couple of stdin/stdout pipes to it for scripts.
1314The two main syntaxes we'll use are the following :
1315
1316 # socat /var/run/haproxy.sock stdio
1317 # socat /var/run/haproxy.sock readline
1318
1319The first one is used with scripts. It is possible to send the output of a
1320script to haproxy, and pass haproxy's output to another script. That's useful
1321for retrieving counters or attack traces for example.
1322
1323The second one is only useful for issuing commands by hand. It has the benefit
1324that the terminal is handled by the readline library which supports line
1325editing and history, which is very convenient when issuing repeated commands
1326(eg: watch a counter).
1327
1328The socket supports two operation modes :
1329 - interactive
1330 - non-interactive
1331
1332The non-interactive mode is the default when socat connects to the socket. In
1333this mode, a single line may be sent. It is processed as a whole, responses are
1334sent back, and the connection closes after the end of the response. This is the
1335mode that scripts and monitoring tools use. It is possible to send multiple
1336commands in this mode, they need to be delimited by a semi-colon (';'). For
1337example :
1338
1339 # echo "show info;show stat;show table" | socat /var/run/haproxy stdio
1340
Dragan Dosena1c35ab2016-11-24 11:33:12 +01001341If a command needs to use a semi-colon or a backslash (eg: in a value), it
Joseph Herlant71b4b152018-11-13 16:55:16 -08001342must be preceded by a backslash ('\').
Chad Lavoiee3f50312016-05-26 16:42:25 -04001343
Willy Tarreau44aed902015-10-13 14:45:29 +02001344The interactive mode displays a prompt ('>') and waits for commands to be
1345entered on the line, then processes them, and displays the prompt again to wait
1346for a new command. This mode is entered via the "prompt" command which must be
1347sent on the first line in non-interactive mode. The mode is a flip switch, if
1348"prompt" is sent in interactive mode, it is disabled and the connection closes
1349after processing the last command of the same line.
1350
1351For this reason, when debugging by hand, it's quite common to start with the
1352"prompt" command :
1353
1354 # socat /var/run/haproxy readline
1355 prompt
1356 > show info
1357 ...
1358 >
1359
1360Since multiple commands may be issued at once, haproxy uses the empty line as a
1361delimiter to mark an end of output for each command, and takes care of ensuring
1362that no command can emit an empty line on output. A script can thus easily
1363parse the output even when multiple commands were pipelined on a single line.
1364
Aurélien Nephtaliabbf6072018-04-18 13:26:46 +02001365Some commands may take an optional payload. To add one to a command, the first
1366line needs to end with the "<<\n" pattern. The next lines will be treated as
1367the payload and can contain as many lines as needed. To validate a command with
1368a payload, it needs to end with an empty line.
1369
1370Limitations do exist: the length of the whole buffer passed to the CLI must
1371not be greater than tune.bfsize and the pattern "<<" must not be glued to the
1372last word of the line.
1373
1374When entering a paylod while in interactive mode, the prompt will change from
1375"> " to "+ ".
1376
Willy Tarreau44aed902015-10-13 14:45:29 +02001377It is important to understand that when multiple haproxy processes are started
1378on the same sockets, any process may pick up the request and will output its
1379own stats.
1380
1381The list of commands currently supported on the stats socket is provided below.
1382If an unknown command is sent, haproxy displays the usage message which reminds
1383all supported commands. Some commands support a more complex syntax, generally
1384it will explain what part of the command is invalid when this happens.
1385
Olivier Doucetd8703e82017-08-31 11:05:10 +02001386Some commands require a higher level of privilege to work. If you do not have
1387enough privilege, you will get an error "Permission denied". Please check
1388the "level" option of the "bind" keyword lines in the configuration manual
1389for more information.
1390
William Lallemand6ab08b32019-11-29 16:48:43 +01001391abort ssl cert <filename>
1392 Abort and destroy a temporary SSL certificate update transaction.
1393
1394 See also "set ssl cert" and "commit ssl cert".
1395
Willy Tarreau44aed902015-10-13 14:45:29 +02001396add acl <acl> <pattern>
1397 Add an entry into the acl <acl>. <acl> is the #<id> or the <file> returned by
1398 "show acl". This command does not verify if the entry already exists. This
1399 command cannot be used if the reference <acl> is a file also used with a map.
1400 In this case, you must use the command "add map" in place of "add acl".
1401
1402add map <map> <key> <value>
Aurélien Nephtali25650ce2018-04-18 14:04:47 +02001403add map <map> <payload>
Willy Tarreau44aed902015-10-13 14:45:29 +02001404 Add an entry into the map <map> to associate the value <value> to the key
1405 <key>. This command does not verify if the entry already exists. It is
1406 mainly used to fill a map after a clear operation. Note that if the reference
1407 <map> is a file and is shared with a map, this map will contain also a new
Aurélien Nephtali25650ce2018-04-18 14:04:47 +02001408 pattern entry. Using the payload syntax it is possible to add multiple
1409 key/value pairs by entering them on separate lines. On each new line, the
1410 first word is the key and the rest of the line is considered to be the value
1411 which can even contains spaces.
1412
1413 Example:
1414
1415 # socat /tmp/sock1 -
1416 prompt
1417
1418 > add map #-1 <<
1419 + key1 value1
1420 + key2 value2 with spaces
1421 + key3 value3 also with spaces
1422 + key4 value4
1423
1424 >
Willy Tarreau44aed902015-10-13 14:45:29 +02001425
William Lallemandaccac232020-04-02 17:42:51 +02001426add ssl crt-list <crtlist> <certificate>
1427add ssl crt-list <crtlist> <payload>
1428 Add an certificate in a crt-list. It can also be used for directories since
1429 directories are now loaded the same way as the crt-lists. This command allow
1430 you to use a certificate name in parameter, to use SSL options or filters a
1431 crt-list line must sent as a payload instead. Only one crt-list line is
1432 supported in the payload. This command will load the certificate for every
1433 bind lines using the crt-list. To push a new certificate to HAProxy the
1434 commands "new ssl cert" and "set ssl cert" must be used.
1435
1436 Example:
1437 $ echo "new ssl cert foobar.pem" | socat /tmp/sock1 -
1438 $ echo -e "set ssl cert foobar.pem <<\n$(cat foobar.pem)\n" | socat
1439 /tmp/sock1 -
1440 $ echo "commit ssl cert foobar.pem" | socat /tmp/sock1 -
1441 $ echo "add ssl crt-list certlist1 foobar.pem" | socat /tmp/sock1 -
1442
1443 $ echo -e 'add ssl crt-list certlist1 <<\nfoobar.pem [allow-0rtt] foo.bar.com
1444 !test1.com\n' | socat /tmp/sock1 -
1445
Willy Tarreau44aed902015-10-13 14:45:29 +02001446clear counters
1447 Clear the max values of the statistics counters in each proxy (frontend &
Willy Tarreaud80cb4e2018-01-20 19:30:13 +01001448 backend) and in each server. The accumulated counters are not affected. The
1449 internal activity counters reported by "show activity" are also reset. This
Willy Tarreau44aed902015-10-13 14:45:29 +02001450 can be used to get clean counters after an incident, without having to
1451 restart nor to clear traffic counters. This command is restricted and can
1452 only be issued on sockets configured for levels "operator" or "admin".
1453
1454clear counters all
1455 Clear all statistics counters in each proxy (frontend & backend) and in each
1456 server. This has the same effect as restarting. This command is restricted
1457 and can only be issued on sockets configured for level "admin".
1458
1459clear acl <acl>
1460 Remove all entries from the acl <acl>. <acl> is the #<id> or the <file>
1461 returned by "show acl". Note that if the reference <acl> is a file and is
1462 shared with a map, this map will be also cleared.
1463
1464clear map <map>
1465 Remove all entries from the map <map>. <map> is the #<id> or the <file>
1466 returned by "show map". Note that if the reference <map> is a file and is
1467 shared with a acl, this acl will be also cleared.
1468
1469clear table <table> [ data.<type> <operator> <value> ] | [ key <key> ]
1470 Remove entries from the stick-table <table>.
1471
1472 This is typically used to unblock some users complaining they have been
1473 abusively denied access to a service, but this can also be used to clear some
1474 stickiness entries matching a server that is going to be replaced (see "show
1475 table" below for details). Note that sometimes, removal of an entry will be
1476 refused because it is currently tracked by a session. Retrying a few seconds
1477 later after the session ends is usual enough.
1478
1479 In the case where no options arguments are given all entries will be removed.
1480
1481 When the "data." form is used entries matching a filter applied using the
1482 stored data (see "stick-table" in section 4.2) are removed. A stored data
1483 type must be specified in <type>, and this data type must be stored in the
1484 table otherwise an error is reported. The data is compared according to
1485 <operator> with the 64-bit integer <value>. Operators are the same as with
1486 the ACLs :
1487
1488 - eq : match entries whose data is equal to this value
1489 - ne : match entries whose data is not equal to this value
1490 - le : match entries whose data is less than or equal to this value
1491 - ge : match entries whose data is greater than or equal to this value
1492 - lt : match entries whose data is less than this value
1493 - gt : match entries whose data is greater than this value
1494
1495 When the key form is used the entry <key> is removed. The key must be of the
1496 same type as the table, which currently is limited to IPv4, IPv6, integer and
1497 string.
1498
1499 Example :
1500 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1501 >>> # table: http_proxy, type: ip, size:204800, used:2
1502 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
1503 bytes_out_rate(60000)=187
1504 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
1505 bytes_out_rate(60000)=191
1506
1507 $ echo "clear table http_proxy key 127.0.0.1" | socat stdio /tmp/sock1
1508
1509 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1510 >>> # table: http_proxy, type: ip, size:204800, used:1
1511 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
1512 bytes_out_rate(60000)=191
1513 $ echo "clear table http_proxy data.gpc0 eq 1" | socat stdio /tmp/sock1
1514 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1515 >>> # table: http_proxy, type: ip, size:204800, used:1
1516
William Lallemand6ab08b32019-11-29 16:48:43 +01001517commit ssl cert <filename>
William Lallemandc184d872020-06-26 15:39:57 +02001518 Commit a temporary SSL certificate update transaction.
1519
1520 In the case of an existing certificate (in a "Used" state in "show ssl
1521 cert"), generate every SSL contextes and SNIs it need, insert them, and
1522 remove the previous ones. Replace in memory the previous SSL certificates
1523 everywhere the <filename> was used in the configuration. Upon failure it
1524 doesn't remove or insert anything. Once the temporary transaction is
1525 committed, it is destroyed.
1526
1527 In the case of a new certificate (after a "new ssl cert" and in a "Unused"
Ilya Shipitsin2272d8a2020-12-21 01:22:40 +05001528 state in "show ssl cert"), the certificate will be committed in a certificate
William Lallemandc184d872020-06-26 15:39:57 +02001529 storage, but it won't be used anywhere in haproxy. To use it and generate
1530 its SNIs you will need to add it to a crt-list or a directory with "add ssl
1531 crt-list".
William Lallemand6ab08b32019-11-29 16:48:43 +01001532
William Lallemandc184d872020-06-26 15:39:57 +02001533 See also "new ssl cert", "ssl set cert", "abort ssl cert" and
1534 "add ssl crt-list".
William Lallemand6ab08b32019-11-29 16:48:43 +01001535
Willy Tarreau6bdf3e92019-05-20 14:25:05 +02001536debug dev <command> [args]*
Willy Tarreaub24ab222019-10-24 18:03:39 +02001537 Call a developer-specific command. Only supported on a CLI connection running
1538 in expert mode (see "expert-mode on"). Such commands are extremely dangerous
1539 and not forgiving, any misuse may result in a crash of the process. They are
1540 intended for experts only, and must really not be used unless told to do so.
1541 Some of them are only available when haproxy is built with DEBUG_DEV defined
1542 because they may have security implications. All of these commands require
1543 admin privileges, and are purposely not documented to avoid encouraging their
1544 use by people who are not at ease with the source code.
Willy Tarreau6bdf3e92019-05-20 14:25:05 +02001545
Willy Tarreau44aed902015-10-13 14:45:29 +02001546del acl <acl> [<key>|#<ref>]
1547 Delete all the acl entries from the acl <acl> corresponding to the key <key>.
1548 <acl> is the #<id> or the <file> returned by "show acl". If the <ref> is used,
1549 this command delete only the listed reference. The reference can be found with
1550 listing the content of the acl. Note that if the reference <acl> is a file and
1551 is shared with a map, the entry will be also deleted in the map.
1552
1553del map <map> [<key>|#<ref>]
1554 Delete all the map entries from the map <map> corresponding to the key <key>.
1555 <map> is the #<id> or the <file> returned by "show map". If the <ref> is used,
1556 this command delete only the listed reference. The reference can be found with
1557 listing the content of the map. Note that if the reference <map> is a file and
1558 is shared with a acl, the entry will be also deleted in the map.
1559
William Lallemand419e6342020-04-08 12:05:39 +02001560del ssl cert <certfile>
1561 Delete a certificate store from HAProxy. The certificate must be unused and
1562 removed from any crt-list or directory. "show ssl cert" displays the status
1563 of the certificate. The deletion doesn't work with a certificate referenced
1564 directly with the "crt" directive in the configuration.
1565
William Lallemand0a9b9412020-04-06 17:43:05 +02001566del ssl crt-list <filename> <certfile[:line]>
1567 Delete an entry in a crt-list. This will delete every SNIs used for this
1568 entry in the frontends. If a certificate is used several time in a crt-list,
1569 you will need to provide which line you want to delete. To display the line
1570 numbers, use "show ssl crt-list -n <crtlist>".
1571
Willy Tarreau44aed902015-10-13 14:45:29 +02001572disable agent <backend>/<server>
1573 Mark the auxiliary agent check as temporarily stopped.
1574
1575 In the case where an agent check is being run as a auxiliary check, due
1576 to the agent-check parameter of a server directive, new checks are only
Dan Lloyd8e48b872016-07-01 21:01:18 -04001577 initialized when the agent is in the enabled. Thus, disable agent will
Willy Tarreau44aed902015-10-13 14:45:29 +02001578 prevent any new agent checks from begin initiated until the agent
1579 re-enabled using enable agent.
1580
1581 When an agent is disabled the processing of an auxiliary agent check that
1582 was initiated while the agent was set as enabled is as follows: All
1583 results that would alter the weight, specifically "drain" or a weight
1584 returned by the agent, are ignored. The processing of agent check is
1585 otherwise unchanged.
1586
1587 The motivation for this feature is to allow the weight changing effects
1588 of the agent checks to be paused to allow the weight of a server to be
1589 configured using set weight without being overridden by the agent.
1590
1591 This command is restricted and can only be issued on sockets configured for
1592 level "admin".
1593
Olivier Houchard614f8d72017-03-14 20:08:46 +01001594disable dynamic-cookie backend <backend>
Ilya Shipitsin2a950d02020-03-06 13:07:38 +05001595 Disable the generation of dynamic cookies for the backend <backend>
Olivier Houchard614f8d72017-03-14 20:08:46 +01001596
Willy Tarreau44aed902015-10-13 14:45:29 +02001597disable frontend <frontend>
1598 Mark the frontend as temporarily stopped. This corresponds to the mode which
1599 is used during a soft restart : the frontend releases the port but can be
1600 enabled again if needed. This should be used with care as some non-Linux OSes
1601 are unable to enable it back. This is intended to be used in environments
1602 where stopping a proxy is not even imaginable but a misconfigured proxy must
1603 be fixed. That way it's possible to release the port and bind it into another
1604 process to restore operations. The frontend will appear with status "STOP"
1605 on the stats page.
1606
1607 The frontend may be specified either by its name or by its numeric ID,
1608 prefixed with a sharp ('#').
1609
1610 This command is restricted and can only be issued on sockets configured for
1611 level "admin".
1612
1613disable health <backend>/<server>
1614 Mark the primary health check as temporarily stopped. This will disable
1615 sending of health checks, and the last health check result will be ignored.
1616 The server will be in unchecked state and considered UP unless an auxiliary
1617 agent check forces it down.
1618
1619 This command is restricted and can only be issued on sockets configured for
1620 level "admin".
1621
1622disable server <backend>/<server>
1623 Mark the server DOWN for maintenance. In this mode, no more checks will be
1624 performed on the server until it leaves maintenance.
1625 If the server is tracked by other servers, those servers will be set to DOWN
1626 during the maintenance.
1627
1628 In the statistics page, a server DOWN for maintenance will appear with a
1629 "MAINT" status, its tracking servers with the "MAINT(via)" one.
1630
1631 Both the backend and the server may be specified either by their name or by
1632 their numeric ID, prefixed with a sharp ('#').
1633
1634 This command is restricted and can only be issued on sockets configured for
1635 level "admin".
1636
1637enable agent <backend>/<server>
1638 Resume auxiliary agent check that was temporarily stopped.
1639
1640 See "disable agent" for details of the effect of temporarily starting
1641 and stopping an auxiliary agent.
1642
1643 This command is restricted and can only be issued on sockets configured for
1644 level "admin".
1645
Olivier Houchard614f8d72017-03-14 20:08:46 +01001646enable dynamic-cookie backend <backend>
n9@users.noreply.github.com25a1c8e2019-08-23 11:21:05 +02001647 Enable the generation of dynamic cookies for the backend <backend>.
1648 A secret key must also be provided.
Olivier Houchard614f8d72017-03-14 20:08:46 +01001649
Willy Tarreau44aed902015-10-13 14:45:29 +02001650enable frontend <frontend>
1651 Resume a frontend which was temporarily stopped. It is possible that some of
1652 the listening ports won't be able to bind anymore (eg: if another process
1653 took them since the 'disable frontend' operation). If this happens, an error
1654 is displayed. Some operating systems might not be able to resume a frontend
1655 which was disabled.
1656
1657 The frontend may be specified either by its name or by its numeric ID,
1658 prefixed with a sharp ('#').
1659
1660 This command is restricted and can only be issued on sockets configured for
1661 level "admin".
1662
1663enable health <backend>/<server>
1664 Resume a primary health check that was temporarily stopped. This will enable
1665 sending of health checks again. Please see "disable health" for details.
1666
1667 This command is restricted and can only be issued on sockets configured for
1668 level "admin".
1669
1670enable server <backend>/<server>
1671 If the server was previously marked as DOWN for maintenance, this marks the
1672 server UP and checks are re-enabled.
1673
1674 Both the backend and the server may be specified either by their name or by
1675 their numeric ID, prefixed with a sharp ('#').
1676
1677 This command is restricted and can only be issued on sockets configured for
1678 level "admin".
1679
Willy Tarreauabb9f9b2019-10-24 17:55:53 +02001680expert-mode [on|off]
1681 Without options, this indicates whether the expert mode is enabled or
1682 disabled on the current connection. When passed "on", it turns the expert
1683 mode on for the current CLI connection only. With "off" it turns it off. The
1684 expert mode enables displaying of expert commands that can be extremely
1685 dangerous for the process and which may occasionally help developers collect
1686 important information about complex bugs. Any misuse of these features will
1687 likely lead to a process crash. Do not use this option without being invited
1688 to do so. Note that this command is purposely not listed in the help message.
1689 This command is only accessible in admin level. Changing to another level
1690 automatically resets the expert mode.
1691
Willy Tarreau44aed902015-10-13 14:45:29 +02001692get map <map> <value>
1693get acl <acl> <value>
1694 Lookup the value <value> in the map <map> or in the ACL <acl>. <map> or <acl>
1695 are the #<id> or the <file> returned by "show map" or "show acl". This command
1696 returns all the matching patterns associated with this map. This is useful for
1697 debugging maps and ACLs. The output format is composed by one line par
1698 matching type. Each line is composed by space-delimited series of words.
1699
1700 The first two words are:
1701
1702 <match method>: The match method applied. It can be "found", "bool",
1703 "int", "ip", "bin", "len", "str", "beg", "sub", "dir",
1704 "dom", "end" or "reg".
1705
1706 <match result>: The result. Can be "match" or "no-match".
1707
1708 The following words are returned only if the pattern matches an entry.
1709
1710 <index type>: "tree" or "list". The internal lookup algorithm.
1711
1712 <case>: "case-insensitive" or "case-sensitive". The
1713 interpretation of the case.
1714
1715 <entry matched>: match="<entry>". Return the matched pattern. It is
1716 useful with regular expressions.
1717
1718 The two last word are used to show the returned value and its type. With the
1719 "acl" case, the pattern doesn't exist.
1720
1721 return=nothing: No return because there are no "map".
1722 return="<value>": The value returned in the string format.
1723 return=cannot-display: The value cannot be converted as string.
1724
1725 type="<type>": The type of the returned sample.
1726
1727get weight <backend>/<server>
1728 Report the current weight and the initial weight of server <server> in
1729 backend <backend> or an error if either doesn't exist. The initial weight is
1730 the one that appears in the configuration file. Both are normally equal
1731 unless the current weight has been changed. Both the backend and the server
1732 may be specified either by their name or by their numeric ID, prefixed with a
1733 sharp ('#').
1734
1735help
1736 Print the list of known keywords and their basic usage. The same help screen
1737 is also displayed for unknown commands.
1738
William Lallemandaccac232020-04-02 17:42:51 +02001739new ssl cert <filename>
1740 Create a new empty SSL certificate store to be filled with a certificate and
1741 added to a directory or a crt-list. This command should be used in
1742 combination with "set ssl cert" and "add ssl crt-list".
1743
Willy Tarreau44aed902015-10-13 14:45:29 +02001744prompt
1745 Toggle the prompt at the beginning of the line and enter or leave interactive
1746 mode. In interactive mode, the connection is not closed after a command
1747 completes. Instead, the prompt will appear again, indicating the user that
1748 the interpreter is waiting for a new command. The prompt consists in a right
1749 angle bracket followed by a space "> ". This mode is particularly convenient
1750 when one wants to periodically check information such as stats or errors.
1751 It is also a good idea to enter interactive mode before issuing a "help"
1752 command.
1753
1754quit
1755 Close the connection when in interactive mode.
1756
Olivier Houchard614f8d72017-03-14 20:08:46 +01001757set dynamic-cookie-key backend <backend> <value>
1758 Modify the secret key used to generate the dynamic persistent cookies.
1759 This will break the existing sessions.
1760
Willy Tarreau44aed902015-10-13 14:45:29 +02001761set map <map> [<key>|#<ref>] <value>
1762 Modify the value corresponding to each key <key> in a map <map>. <map> is the
1763 #<id> or <file> returned by "show map". If the <ref> is used in place of
1764 <key>, only the entry pointed by <ref> is changed. The new value is <value>.
1765
1766set maxconn frontend <frontend> <value>
1767 Dynamically change the specified frontend's maxconn setting. Any positive
1768 value is allowed including zero, but setting values larger than the global
1769 maxconn does not make much sense. If the limit is increased and connections
1770 were pending, they will immediately be accepted. If it is lowered to a value
1771 below the current number of connections, new connections acceptation will be
1772 delayed until the threshold is reached. The frontend might be specified by
1773 either its name or its numeric ID prefixed with a sharp ('#').
1774
Andrew Hayworthedb93a72015-10-27 21:46:25 +00001775set maxconn server <backend/server> <value>
1776 Dynamically change the specified server's maxconn setting. Any positive
1777 value is allowed including zero, but setting values larger than the global
1778 maxconn does not make much sense.
1779
Willy Tarreau44aed902015-10-13 14:45:29 +02001780set maxconn global <maxconn>
1781 Dynamically change the global maxconn setting within the range defined by the
1782 initial global maxconn setting. If it is increased and connections were
1783 pending, they will immediately be accepted. If it is lowered to a value below
1784 the current number of connections, new connections acceptation will be
1785 delayed until the threshold is reached. A value of zero restores the initial
1786 setting.
1787
Willy Tarreaud2d33482019-04-25 17:09:07 +02001788set profiling { tasks } { auto | on | off }
Willy Tarreau75c62c22018-11-22 11:02:09 +01001789 Enables or disables CPU profiling for the indicated subsystem. This is
1790 equivalent to setting or clearing the "profiling" settings in the "global"
1791 section of the configuration file. Please also see "show profiling".
1792
Willy Tarreau44aed902015-10-13 14:45:29 +02001793set rate-limit connections global <value>
1794 Change the process-wide connection rate limit, which is set by the global
1795 'maxconnrate' setting. A value of zero disables the limitation. This limit
1796 applies to all frontends and the change has an immediate effect. The value
1797 is passed in number of connections per second.
1798
1799set rate-limit http-compression global <value>
1800 Change the maximum input compression rate, which is set by the global
1801 'maxcomprate' setting. A value of zero disables the limitation. The value is
1802 passed in number of kilobytes per second. The value is available in the "show
1803 info" on the line "CompressBpsRateLim" in bytes.
1804
1805set rate-limit sessions global <value>
1806 Change the process-wide session rate limit, which is set by the global
1807 'maxsessrate' setting. A value of zero disables the limitation. This limit
1808 applies to all frontends and the change has an immediate effect. The value
1809 is passed in number of sessions per second.
1810
1811set rate-limit ssl-sessions global <value>
1812 Change the process-wide SSL session rate limit, which is set by the global
1813 'maxsslrate' setting. A value of zero disables the limitation. This limit
1814 applies to all frontends and the change has an immediate effect. The value
1815 is passed in number of sessions per second sent to the SSL stack. It applies
1816 before the handshake in order to protect the stack against handshake abuses.
1817
Baptiste Assmann3749ebf2016-08-03 22:34:12 +02001818set server <backend>/<server> addr <ip4 or ip6 address> [port <port>]
Willy Tarreau44aed902015-10-13 14:45:29 +02001819 Replace the current IP address of a server by the one provided.
Michael Prokop4438c602019-05-24 10:25:45 +02001820 Optionally, the port can be changed using the 'port' parameter.
Baptiste Assmann3749ebf2016-08-03 22:34:12 +02001821 Note that changing the port also support switching from/to port mapping
1822 (notation with +X or -Y), only if a port is configured for the health check.
Willy Tarreau44aed902015-10-13 14:45:29 +02001823
1824set server <backend>/<server> agent [ up | down ]
1825 Force a server's agent to a new state. This can be useful to immediately
1826 switch a server's state regardless of some slow agent checks for example.
1827 Note that the change is propagated to tracking servers if any.
1828
Misiek43972902017-01-09 09:53:06 +01001829set server <backend>/<server> agent-addr <addr>
1830 Change addr for servers agent checks. Allows to migrate agent-checks to
1831 another address at runtime. You can specify both IP and hostname, it will be
1832 resolved.
1833
1834set server <backend>/<server> agent-send <value>
1835 Change agent string sent to agent check target. Allows to update string while
1836 changing server address to keep those two matching.
1837
Willy Tarreau44aed902015-10-13 14:45:29 +02001838set server <backend>/<server> health [ up | stopping | down ]
1839 Force a server's health to a new state. This can be useful to immediately
1840 switch a server's state regardless of some slow health checks for example.
1841 Note that the change is propagated to tracking servers if any.
1842
Baptiste Assmann50946562016-08-31 23:26:29 +02001843set server <backend>/<server> check-port <port>
1844 Change the port used for health checking to <port>
1845
Willy Tarreau44aed902015-10-13 14:45:29 +02001846set server <backend>/<server> state [ ready | drain | maint ]
1847 Force a server's administrative state to a new state. This can be useful to
1848 disable load balancing and/or any traffic to a server. Setting the state to
1849 "ready" puts the server in normal mode, and the command is the equivalent of
1850 the "enable server" command. Setting the state to "maint" disables any traffic
1851 to the server as well as any health checks. This is the equivalent of the
1852 "disable server" command. Setting the mode to "drain" only removes the server
1853 from load balancing but still allows it to be checked and to accept new
1854 persistent connections. Changes are propagated to tracking servers if any.
1855
1856set server <backend>/<server> weight <weight>[%]
1857 Change a server's weight to the value passed in argument. This is the exact
1858 equivalent of the "set weight" command below.
1859
Frédéric Lécailleb418c122017-04-26 11:24:02 +02001860set server <backend>/<server> fqdn <FQDN>
Lukas Tribusc5dd5a52018-08-14 11:39:35 +02001861 Change a server's FQDN to the value passed in argument. This requires the
1862 internal run-time DNS resolver to be configured and enabled for this server.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02001863
William Dauchyf6370442020-11-14 19:25:33 +01001864set server <backend>/<server> ssl [ on | off ]
1865 This option configures SSL ciphering on outgoing connections to the server.
1866
Andjelko Iharosc4df59e2017-07-20 11:59:48 +02001867set severity-output [ none | number | string ]
1868 Change the severity output format of the stats socket connected to for the
1869 duration of the current session.
1870
William Lallemand6ab08b32019-11-29 16:48:43 +01001871set ssl cert <filename> <payload>
1872 This command is part of a transaction system, the "commit ssl cert" and
1873 "abort ssl cert" commands could be required.
1874 If there is no on-going transaction, it will duplicate the certificate
1875 <filename> in memory to a temporary transaction, then update this
1876 transaction with the PEM file in the payload. If a transaction exists with
1877 the same filename, it will update this transaction. It's also possible to
1878 update the files linked to a certificate (.issuer, .sctl, .oscp etc.)
1879 Once the modification are done, you have to "commit ssl cert" the
1880 transaction.
1881
1882 Example:
1883 echo -e "set ssl cert localhost.pem <<\n$(cat 127.0.0.1.pem)\n" | \
1884 socat /var/run/haproxy.stat -
1885 echo -e \
1886 "set ssl cert localhost.pem.issuer <<\n $(cat 127.0.0.1.pem.issuer)\n" | \
1887 socat /var/run/haproxy.stat -
1888 echo -e \
1889 "set ssl cert localhost.pem.ocsp <<\n$(base64 -w 1000 127.0.0.1.pem.ocsp)\n" | \
1890 socat /var/run/haproxy.stat -
1891 echo "commit ssl cert localhost.pem" | socat /var/run/haproxy.stat -
1892
Aurélien Nephtali1e0867c2018-04-18 14:04:58 +02001893set ssl ocsp-response <response | payload>
Willy Tarreau44aed902015-10-13 14:45:29 +02001894 This command is used to update an OCSP Response for a certificate (see "crt"
1895 on "bind" lines). Same controls are performed as during the initial loading of
1896 the response. The <response> must be passed as a base64 encoded string of the
Emmanuel Hocdet2c32d8f2017-05-22 14:58:00 +02001897 DER encoded response from the OCSP server. This command is not supported with
1898 BoringSSL.
Willy Tarreau44aed902015-10-13 14:45:29 +02001899
1900 Example:
1901 openssl ocsp -issuer issuer.pem -cert server.pem \
1902 -host ocsp.issuer.com:80 -respout resp.der
1903 echo "set ssl ocsp-response $(base64 -w 10000 resp.der)" | \
1904 socat stdio /var/run/haproxy.stat
1905
Aurélien Nephtali1e0867c2018-04-18 14:04:58 +02001906 using the payload syntax:
1907 echo -e "set ssl ocsp-response <<\n$(base64 resp.der)\n" | \
1908 socat stdio /var/run/haproxy.stat
1909
Willy Tarreau44aed902015-10-13 14:45:29 +02001910set ssl tls-key <id> <tlskey>
1911 Set the next TLS key for the <id> listener to <tlskey>. This key becomes the
1912 ultimate key, while the penultimate one is used for encryption (others just
1913 decrypt). The oldest TLS key present is overwritten. <id> is either a numeric
1914 #<id> or <file> returned by "show tls-keys". <tlskey> is a base64 encoded 48
Emeric Brun9e754772019-01-10 17:51:55 +01001915 or 80 bits TLS ticket key (ex. openssl rand 80 | openssl base64 -A).
Willy Tarreau44aed902015-10-13 14:45:29 +02001916
1917set table <table> key <key> [data.<data_type> <value>]*
1918 Create or update a stick-table entry in the table. If the key is not present,
1919 an entry is inserted. See stick-table in section 4.2 to find all possible
1920 values for <data_type>. The most likely use consists in dynamically entering
1921 entries for source IP addresses, with a flag in gpc0 to dynamically block an
1922 IP address or affect its quality of service. It is possible to pass multiple
1923 data_types in a single call.
1924
1925set timeout cli <delay>
1926 Change the CLI interface timeout for current connection. This can be useful
1927 during long debugging sessions where the user needs to constantly inspect
1928 some indicators without being disconnected. The delay is passed in seconds.
1929
1930set weight <backend>/<server> <weight>[%]
1931 Change a server's weight to the value passed in argument. If the value ends
1932 with the '%' sign, then the new weight will be relative to the initially
1933 configured weight. Absolute weights are permitted between 0 and 256.
1934 Relative weights must be positive with the resulting absolute weight is
1935 capped at 256. Servers which are part of a farm running a static
1936 load-balancing algorithm have stricter limitations because the weight
1937 cannot change once set. Thus for these servers, the only accepted values
1938 are 0 and 100% (or 0 and the initial weight). Changes take effect
1939 immediately, though certain LB algorithms require a certain amount of
1940 requests to consider changes. A typical usage of this command is to
1941 disable a server during an update by setting its weight to zero, then to
1942 enable it again after the update by setting it back to 100%. This command
1943 is restricted and can only be issued on sockets configured for level
1944 "admin". Both the backend and the server may be specified either by their
1945 name or by their numeric ID, prefixed with a sharp ('#').
1946
Willy Tarreaud6129fc2017-07-28 16:52:23 +02001947show acl [<acl>]
1948 Dump info about acl converters. Without argument, the list of all available
1949 acls is returned. If a <acl> is specified, its contents are dumped. <acl> if
1950 the #<id> or <file>. The dump format is the same than the map even for the
1951 sample value. The data returned are not a list of available ACL, but are the
1952 list of all patterns composing any ACL. Many of these patterns can be shared
1953 with maps.
1954
1955show backend
1956 Dump the list of backends available in the running process
1957
William Lallemand67a234f2018-12-13 09:05:45 +01001958show cli level
1959 Display the CLI level of the current CLI session. The result could be
1960 'admin', 'operator' or 'user'. See also the 'operator' and 'user' commands.
1961
1962 Example :
1963
1964 $ socat /tmp/sock1 readline
1965 prompt
1966 > operator
1967 > show cli level
1968 operator
1969 > user
1970 > show cli level
1971 user
1972 > operator
1973 Permission denied
1974
1975operator
1976 Decrease the CLI level of the current CLI session to operator. It can't be
Willy Tarreauabb9f9b2019-10-24 17:55:53 +02001977 increased. It also drops expert mode. See also "show cli level".
William Lallemand67a234f2018-12-13 09:05:45 +01001978
1979user
1980 Decrease the CLI level of the current CLI session to user. It can't be
Willy Tarreauabb9f9b2019-10-24 17:55:53 +02001981 increased. It also drops expert mode. See also "show cli level".
William Lallemand67a234f2018-12-13 09:05:45 +01001982
Willy Tarreau4c356932019-05-16 17:39:32 +02001983show activity
1984 Reports some counters about internal events that will help developers and
1985 more generally people who know haproxy well enough to narrow down the causes
1986 of reports of abnormal behaviours. A typical example would be a properly
1987 running process never sleeping and eating 100% of the CPU. The output fields
1988 will be made of one line per metric, and per-thread counters on the same
Thayne McCombscdbcca92021-01-07 21:24:41 -07001989 line. These counters are 32-bit and will wrap during the process's life, which
Willy Tarreau4c356932019-05-16 17:39:32 +02001990 is not a problem since calls to this command will typically be performed
1991 twice. The fields are purposely not documented so that their exact meaning is
1992 verified in the code where the counters are fed. These values are also reset
1993 by the "clear counters" command.
1994
William Lallemand51132162016-12-16 16:38:58 +01001995show cli sockets
1996 List CLI sockets. The output format is composed of 3 fields separated by
1997 spaces. The first field is the socket address, it can be a unix socket, a
1998 ipv4 address:port couple or a ipv6 one. Socket of other types won't be dump.
1999 The second field describe the level of the socket: 'admin', 'user' or
2000 'operator'. The last field list the processes on which the socket is bound,
2001 separated by commas, it can be numbers or 'all'.
2002
2003 Example :
2004
2005 $ echo 'show cli sockets' | socat stdio /tmp/sock1
2006 # socket lvl processes
2007 /tmp/sock1 admin all
2008 127.0.0.1:9999 user 2,3,4
2009 127.0.0.2:9969 user 2
2010 [::1]:9999 operator 2
2011
William Lallemand86d0df02017-11-24 21:36:45 +01002012show cache
Cyril Bonté7b888f12017-11-26 22:24:31 +01002013 List the configured caches and the objects stored in each cache tree.
William Lallemand86d0df02017-11-24 21:36:45 +01002014
2015 $ echo 'show cache' | socat stdio /tmp/sock1
2016 0x7f6ac6c5b03a: foobar (shctx:0x7f6ac6c5b000, available blocks:3918)
2017 1 2 3 4
2018
2019 1. pointer to the cache structure
2020 2. cache name
2021 3. pointer to the mmap area (shctx)
2022 4. number of blocks available for reuse in the shctx
2023
Remi Tricot-Le Bretone3e1e5f2020-11-27 15:48:40 +01002024 0x7f6ac6c5b4cc hash:286881868 vary:0x0011223344556677 size:39114 (39 blocks), refcount:9, expire:237
2025 1 2 3 4 5 6 7
William Lallemand86d0df02017-11-24 21:36:45 +01002026
2027 1. pointer to the cache entry
2028 2. first 32 bits of the hash
Remi Tricot-Le Bretone3e1e5f2020-11-27 15:48:40 +01002029 3. secondary hash of the entry in case of vary
2030 4. size of the object in bytes
2031 5. number of blocks used for the object
2032 6. number of transactions using the entry
2033 7. expiration time, can be negative if already expired
William Lallemand86d0df02017-11-24 21:36:45 +01002034
Willy Tarreauae795722016-02-16 11:27:28 +01002035show env [<name>]
2036 Dump one or all environment variables known by the process. Without any
2037 argument, all variables are dumped. With an argument, only the specified
2038 variable is dumped if it exists. Otherwise "Variable not found" is emitted.
2039 Variables are dumped in the same format as they are stored or returned by the
2040 "env" utility, that is, "<name>=<value>". This can be handy when debugging
2041 certain configuration files making heavy use of environment variables to
2042 ensure that they contain the expected values. This command is restricted and
2043 can only be issued on sockets configured for levels "operator" or "admin".
2044
Willy Tarreau35069f82016-11-25 09:16:37 +01002045show errors [<iid>|<proxy>] [request|response]
Willy Tarreau44aed902015-10-13 14:45:29 +02002046 Dump last known request and response errors collected by frontends and
2047 backends. If <iid> is specified, the limit the dump to errors concerning
Willy Tarreau234ba2d2016-11-25 08:39:10 +01002048 either frontend or backend whose ID is <iid>. Proxy ID "-1" will cause
2049 all instances to be dumped. If a proxy name is specified instead, its ID
Willy Tarreau35069f82016-11-25 09:16:37 +01002050 will be used as the filter. If "request" or "response" is added after the
2051 proxy name or ID, only request or response errors will be dumped. This
2052 command is restricted and can only be issued on sockets configured for
2053 levels "operator" or "admin".
Willy Tarreau44aed902015-10-13 14:45:29 +02002054
2055 The errors which may be collected are the last request and response errors
2056 caused by protocol violations, often due to invalid characters in header
2057 names. The report precisely indicates what exact character violated the
2058 protocol. Other important information such as the exact date the error was
2059 detected, frontend and backend names, the server name (when known), the
2060 internal session ID and the source address which has initiated the session
2061 are reported too.
2062
2063 All characters are returned, and non-printable characters are encoded. The
2064 most common ones (\t = 9, \n = 10, \r = 13 and \e = 27) are encoded as one
2065 letter following a backslash. The backslash itself is encoded as '\\' to
2066 avoid confusion. Other non-printable characters are encoded '\xNN' where
2067 NN is the two-digits hexadecimal representation of the character's ASCII
2068 code.
2069
2070 Lines are prefixed with the position of their first character, starting at 0
2071 for the beginning of the buffer. At most one input line is printed per line,
2072 and large lines will be broken into multiple consecutive output lines so that
2073 the output never goes beyond 79 characters wide. It is easy to detect if a
2074 line was broken, because it will not end with '\n' and the next line's offset
2075 will be followed by a '+' sign, indicating it is a continuation of previous
2076 line.
2077
2078 Example :
Willy Tarreau35069f82016-11-25 09:16:37 +01002079 $ echo "show errors -1 response" | socat stdio /tmp/sock1
Willy Tarreau44aed902015-10-13 14:45:29 +02002080 >>> [04/Mar/2009:15:46:56.081] backend http-in (#2) : invalid response
2081 src 127.0.0.1, session #54, frontend fe-eth0 (#1), server s2 (#1)
2082 response length 213 bytes, error at position 23:
2083
2084 00000 HTTP/1.0 200 OK\r\n
2085 00017 header/bizarre:blah\r\n
2086 00038 Location: blah\r\n
2087 00054 Long-line: this is a very long line which should b
2088 00104+ e broken into multiple lines on the output buffer,
2089 00154+ otherwise it would be too large to print in a ter
2090 00204+ minal\r\n
2091 00211 \r\n
2092
2093 In the example above, we see that the backend "http-in" which has internal
2094 ID 2 has blocked an invalid response from its server s2 which has internal
2095 ID 1. The request was on session 54 initiated by source 127.0.0.1 and
2096 received by frontend fe-eth0 whose ID is 1. The total response length was
2097 213 bytes when the error was detected, and the error was at byte 23. This
2098 is the slash ('/') in header name "header/bizarre", which is not a valid
2099 HTTP character for a header name.
2100
Willy Tarreau1d181e42019-08-30 11:17:01 +02002101show events [<sink>] [-w] [-n]
Willy Tarreau9f830d72019-08-26 18:17:04 +02002102 With no option, this lists all known event sinks and their types. With an
2103 option, it will dump all available events in the designated sink if it is of
Willy Tarreau1d181e42019-08-30 11:17:01 +02002104 type buffer. If option "-w" is passed after the sink name, then once the end
2105 of the buffer is reached, the command will wait for new events and display
2106 them. It is possible to stop the operation by entering any input (which will
2107 be discarded) or by closing the session. Finally, option "-n" is used to
2108 directly seek to the end of the buffer, which is often convenient when
2109 combined with "-w" to only report new events. For convenience, "-wn" or "-nw"
2110 may be used to enable both options at once.
Willy Tarreau9f830d72019-08-26 18:17:04 +02002111
Willy Tarreau7a4a0ac2017-07-25 19:32:50 +02002112show fd [<fd>]
2113 Dump the list of either all open file descriptors or just the one number <fd>
2114 if specified. This is only aimed at developers who need to observe internal
2115 states in order to debug complex issues such as abnormal CPU usages. One fd
2116 is reported per lines, and for each of them, its state in the poller using
2117 upper case letters for enabled flags and lower case for disabled flags, using
2118 "P" for "polled", "R" for "ready", "A" for "active", the events status using
2119 "H" for "hangup", "E" for "error", "O" for "output", "P" for "priority" and
2120 "I" for "input", a few other flags like "N" for "new" (just added into the fd
2121 cache), "U" for "updated" (received an update in the fd cache), "L" for
2122 "linger_risk", "C" for "cloned", then the cached entry position, the pointer
2123 to the internal owner, the pointer to the I/O callback and its name when
2124 known. When the owner is a connection, the connection flags, and the target
2125 are reported (frontend, proxy or server). When the owner is a listener, the
2126 listener's state and its frontend are reported. There is no point in using
2127 this command without a good knowledge of the internals. It's worth noting
2128 that the output format may evolve over time so this output must not be parsed
Willy Tarreau8050efe2021-01-21 08:26:06 +01002129 by tools designed to be durable. Some internal structure states may look
2130 suspicious to the function listing them, in this case the output line will be
2131 suffixed with an exclamation mark ('!'). This may help find a starting point
2132 when trying to diagnose an incident.
Willy Tarreau7a4a0ac2017-07-25 19:32:50 +02002133
Willy Tarreau698097b2020-10-23 20:19:47 +02002134show info [typed|json] [desc]
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002135 Dump info about haproxy status on current process. If "typed" is passed as an
2136 optional argument, field numbers, names and types are emitted as well so that
2137 external monitoring products can easily retrieve, possibly aggregate, then
2138 report information found in fields they don't know. Each field is dumped on
Simon Horman05ee2132017-01-04 09:37:25 +01002139 its own line. If "json" is passed as an optional argument then
2140 information provided by "typed" output is provided in JSON format as a
2141 list of JSON objects. By default, the format contains only two columns
2142 delimited by a colon (':'). The left one is the field name and the right
2143 one is the value. It is very important to note that in typed output
2144 format, the dump for a single object is contiguous so that there is no
2145 need for a consumer to store everything at once.
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002146
2147 When using the typed output format, each line is made of 4 columns delimited
2148 by colons (':'). The first column is a dot-delimited series of 3 elements. The
2149 first element is the numeric position of the field in the list (starting at
2150 zero). This position shall not change over time, but holes are to be expected,
2151 depending on build options or if some fields are deleted in the future. The
2152 second element is the field name as it appears in the default "show info"
2153 output. The third element is the relative process number starting at 1.
2154
2155 The rest of the line starting after the first colon follows the "typed output
2156 format" described in the section above. In short, the second column (after the
2157 first ':') indicates the origin, nature and scope of the variable. The third
2158 column indicates the type of the field, among "s32", "s64", "u32", "u64" and
2159 "str". Then the fourth column is the value itself, which the consumer knows
2160 how to parse thanks to column 3 and how to process thanks to column 2.
2161
2162 Thus the overall line format in typed mode is :
2163
2164 <field_pos>.<field_name>.<process_num>:<tags>:<type>:<value>
2165
Willy Tarreau6b19b142019-10-09 15:44:21 +02002166 When "desc" is appended to the command, one extra colon followed by a quoted
2167 string is appended with a description for the metric. At the time of writing,
2168 this is only supported for the "typed" and default output formats.
2169
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002170 Example :
2171
2172 > show info
2173 Name: HAProxy
2174 Version: 1.7-dev1-de52ea-146
2175 Release_date: 2016/03/11
2176 Nbproc: 1
2177 Process_num: 1
2178 Pid: 28105
2179 Uptime: 0d 0h00m04s
2180 Uptime_sec: 4
2181 Memmax_MB: 0
2182 PoolAlloc_MB: 0
2183 PoolUsed_MB: 0
2184 PoolFailed: 0
2185 (...)
2186
2187 > show info typed
2188 0.Name.1:POS:str:HAProxy
2189 1.Version.1:POS:str:1.7-dev1-de52ea-146
2190 2.Release_date.1:POS:str:2016/03/11
2191 3.Nbproc.1:CGS:u32:1
2192 4.Process_num.1:KGP:u32:1
2193 5.Pid.1:SGP:u32:28105
2194 6.Uptime.1:MDP:str:0d 0h00m08s
2195 7.Uptime_sec.1:MDP:u32:8
2196 8.Memmax_MB.1:CLP:u32:0
2197 9.PoolAlloc_MB.1:MGP:u32:0
2198 10.PoolUsed_MB.1:MGP:u32:0
2199 11.PoolFailed.1:MCP:u32:0
2200 (...)
2201
Simon Horman1084a362016-11-21 17:00:24 +01002202 In the typed format, the presence of the process ID at the end of the
2203 first column makes it very easy to visually aggregate outputs from
2204 multiple processes.
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002205 Example :
2206
2207 $ ( echo show info typed | socat /var/run/haproxy.sock1 ; \
2208 echo show info typed | socat /var/run/haproxy.sock2 ) | \
2209 sort -t . -k 1,1n -k 2,2 -k 3,3n
2210 0.Name.1:POS:str:HAProxy
2211 0.Name.2:POS:str:HAProxy
2212 1.Version.1:POS:str:1.7-dev1-868ab3-148
2213 1.Version.2:POS:str:1.7-dev1-868ab3-148
2214 2.Release_date.1:POS:str:2016/03/11
2215 2.Release_date.2:POS:str:2016/03/11
2216 3.Nbproc.1:CGS:u32:2
2217 3.Nbproc.2:CGS:u32:2
2218 4.Process_num.1:KGP:u32:1
2219 4.Process_num.2:KGP:u32:2
2220 5.Pid.1:SGP:u32:30120
2221 5.Pid.2:SGP:u32:30121
2222 6.Uptime.1:MDP:str:0d 0h01m28s
2223 6.Uptime.2:MDP:str:0d 0h01m28s
2224 (...)
Willy Tarreau44aed902015-10-13 14:45:29 +02002225
Simon Horman05ee2132017-01-04 09:37:25 +01002226 The format of JSON output is described in a schema which may be output
Simon Horman6f6bb382017-01-04 09:37:26 +01002227 using "show schema json".
Simon Horman05ee2132017-01-04 09:37:25 +01002228
2229 The JSON output contains no extra whitespace in order to reduce the
2230 volume of output. For human consumption passing the output through a
2231 pretty printer may be helpful. Example :
2232
2233 $ echo "show info json" | socat /var/run/haproxy.sock stdio | \
2234 python -m json.tool
2235
Simon Horman6f6bb382017-01-04 09:37:26 +01002236 The JSON output contains no extra whitespace in order to reduce the
2237 volume of output. For human consumption passing the output through a
2238 pretty printer may be helpful. Example :
2239
2240 $ echo "show info json" | socat /var/run/haproxy.sock stdio | \
2241 python -m json.tool
2242
Willy Tarreau44aed902015-10-13 14:45:29 +02002243show map [<map>]
2244 Dump info about map converters. Without argument, the list of all available
2245 maps is returned. If a <map> is specified, its contents are dumped. <map> is
2246 the #<id> or <file>. The first column is a unique identifier. It can be used
2247 as reference for the operation "del map" and "set map". The second column is
2248 the pattern and the third column is the sample if available. The data returned
2249 are not directly a list of available maps, but are the list of all patterns
2250 composing any map. Many of these patterns can be shared with ACL.
2251
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002252show peers [<peers section>]
2253 Dump info about the peers configured in "peers" sections. Without argument,
2254 the list of the peers belonging to all the "peers" sections are listed. If
2255 <peers section> is specified, only the information about the peers belonging
2256 to this "peers" section are dumped.
2257
Michael Prokop4438c602019-05-24 10:25:45 +02002258 Here are two examples of outputs where hostA, hostB and hostC peers belong to
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002259 "sharedlb" peers sections. Only hostA and hostB are connected. Only hostA has
2260 sent data to hostB.
2261
2262 $ echo "show peers" | socat - /tmp/hostA
2263 0x55deb0224320: [15/Apr/2019:11:28:01] id=sharedlb state=0 flags=0x3 \
Emeric Brun0bbec0f2019-04-18 11:39:43 +02002264 resync_timeout=<PAST> task_calls=45122
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002265 0x55deb022b540: id=hostC(remote) addr=127.0.0.12:10002 status=CONN \
2266 reconnect=4s confirm=0
2267 flags=0x0
2268 0x55deb022a440: id=hostA(local) addr=127.0.0.10:10000 status=NONE \
2269 reconnect=<NEVER> confirm=0
2270 flags=0x0
2271 0x55deb0227d70: id=hostB(remote) addr=127.0.0.11:10001 status=ESTA
2272 reconnect=2s confirm=0
Emeric Brun0bbec0f2019-04-18 11:39:43 +02002273 flags=0x20000200 appctx:0x55deb028fba0 st0=7 st1=0 task_calls=14456 \
2274 state=EST
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002275 xprt=RAW src=127.0.0.1:37257 addr=127.0.0.10:10000
2276 remote_table:0x55deb0224a10 id=stkt local_id=1 remote_id=1
2277 last_local_table:0x55deb0224a10 id=stkt local_id=1 remote_id=1
2278 shared tables:
2279 0x55deb0224a10 local_id=1 remote_id=1 flags=0x0 remote_data=0x65
2280 last_acked=0 last_pushed=3 last_get=0 teaching_origin=0 update=3
2281 table:0x55deb022d6a0 id=stkt update=3 localupdate=3 \
2282 commitupdate=3 syncing=0
2283
2284 $ echo "show peers" | socat - /tmp/hostB
2285 0x55871b5ab320: [15/Apr/2019:11:28:03] id=sharedlb state=0 flags=0x3 \
Emeric Brun0bbec0f2019-04-18 11:39:43 +02002286 resync_timeout=<PAST> task_calls=3
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002287 0x55871b5b2540: id=hostC(remote) addr=127.0.0.12:10002 status=CONN \
2288 reconnect=3s confirm=0
2289 flags=0x0
2290 0x55871b5b1440: id=hostB(local) addr=127.0.0.11:10001 status=NONE \
2291 reconnect=<NEVER> confirm=0
2292 flags=0x0
2293 0x55871b5aed70: id=hostA(remote) addr=127.0.0.10:10000 status=ESTA \
2294 reconnect=2s confirm=0
Emeric Brun0bbec0f2019-04-18 11:39:43 +02002295 flags=0x20000200 appctx:0x7fa46800ee00 st0=7 st1=0 task_calls=62356 \
2296 state=EST
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002297 remote_table:0x55871b5ab960 id=stkt local_id=1 remote_id=1
2298 last_local_table:0x55871b5ab960 id=stkt local_id=1 remote_id=1
2299 shared tables:
2300 0x55871b5ab960 local_id=1 remote_id=1 flags=0x0 remote_data=0x65
2301 last_acked=3 last_pushed=0 last_get=3 teaching_origin=0 update=0
2302 table:0x55871b5b46a0 id=stkt update=1 localupdate=0 \
2303 commitupdate=0 syncing=0
2304
Willy Tarreau44aed902015-10-13 14:45:29 +02002305show pools
2306 Dump the status of internal memory pools. This is useful to track memory
2307 usage when suspecting a memory leak for example. It does exactly the same
2308 as the SIGQUIT when running in foreground except that it does not flush
2309 the pools.
2310
Willy Tarreau75c62c22018-11-22 11:02:09 +01002311show profiling
2312 Dumps the current profiling settings, one per line, as well as the command
2313 needed to change them.
2314
Willy Tarreau69f591e2020-07-01 07:00:59 +02002315show servers conn [<backend>]
2316 Dump the current and idle connections state of the servers belonging to the
2317 designated backend (or all backends if none specified). A backend name or
2318 identifier may be used.
2319
2320 The output consists in a header line showing the fields titles, then one
2321 server per line with for each, the backend name and ID, server name and ID,
2322 the address, port and a series or values. The number of fields varies
2323 depending on thread count.
2324
2325 Given the threaded nature of idle connections, it's important to understand
2326 that some values may change once read, and that as such, consistency within a
2327 line isn't granted. This output is mostly provided as a debugging tool and is
2328 not relevant to be routinely monitored nor graphed.
2329
Willy Tarreau44aed902015-10-13 14:45:29 +02002330show servers state [<backend>]
2331 Dump the state of the servers found in the running configuration. A backend
2332 name or identifier may be provided to limit the output to this backend only.
2333
2334 The dump has the following format:
2335 - first line contains the format version (1 in this specification);
2336 - second line contains the column headers, prefixed by a sharp ('#');
2337 - third line and next ones contain data;
2338 - each line starting by a sharp ('#') is considered as a comment.
2339
Dan Lloyd8e48b872016-07-01 21:01:18 -04002340 Since multiple versions of the output may co-exist, below is the list of
Willy Tarreau44aed902015-10-13 14:45:29 +02002341 fields and their order per file format version :
2342 1:
2343 be_id: Backend unique id.
2344 be_name: Backend label.
2345 srv_id: Server unique id (in the backend).
2346 srv_name: Server label.
2347 srv_addr: Server IP address.
2348 srv_op_state: Server operational state (UP/DOWN/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002349 0 = SRV_ST_STOPPED
2350 The server is down.
2351 1 = SRV_ST_STARTING
2352 The server is warming up (up but
2353 throttled).
2354 2 = SRV_ST_RUNNING
2355 The server is fully up.
2356 3 = SRV_ST_STOPPING
2357 The server is up but soft-stopping
2358 (eg: 404).
Willy Tarreau44aed902015-10-13 14:45:29 +02002359 srv_admin_state: Server administrative state (MAINT/DRAIN/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002360 The state is actually a mask of values :
2361 0x01 = SRV_ADMF_FMAINT
2362 The server was explicitly forced into
2363 maintenance.
2364 0x02 = SRV_ADMF_IMAINT
2365 The server has inherited the maintenance
2366 status from a tracked server.
2367 0x04 = SRV_ADMF_CMAINT
2368 The server is in maintenance because of
2369 the configuration.
2370 0x08 = SRV_ADMF_FDRAIN
2371 The server was explicitly forced into
2372 drain state.
2373 0x10 = SRV_ADMF_IDRAIN
2374 The server has inherited the drain status
2375 from a tracked server.
Baptiste Assmann89aa7f32016-11-02 21:31:27 +01002376 0x20 = SRV_ADMF_RMAINT
2377 The server is in maintenance because of an
2378 IP address resolution failure.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02002379 0x40 = SRV_ADMF_HMAINT
2380 The server FQDN was set from stats socket.
2381
Willy Tarreau44aed902015-10-13 14:45:29 +02002382 srv_uweight: User visible server's weight.
2383 srv_iweight: Server's initial weight.
2384 srv_time_since_last_change: Time since last operational change.
2385 srv_check_status: Last health check status.
2386 srv_check_result: Last check result (FAILED/PASSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002387 0 = CHK_RES_UNKNOWN
2388 Initialized to this by default.
2389 1 = CHK_RES_NEUTRAL
2390 Valid check but no status information.
2391 2 = CHK_RES_FAILED
2392 Check failed.
2393 3 = CHK_RES_PASSED
2394 Check succeeded and server is fully up
2395 again.
2396 4 = CHK_RES_CONDPASS
2397 Check reports the server doesn't want new
2398 sessions.
Willy Tarreau44aed902015-10-13 14:45:29 +02002399 srv_check_health: Checks rise / fall current counter.
2400 srv_check_state: State of the check (ENABLED/PAUSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002401 The state is actually a mask of values :
2402 0x01 = CHK_ST_INPROGRESS
2403 A check is currently running.
2404 0x02 = CHK_ST_CONFIGURED
2405 This check is configured and may be
2406 enabled.
2407 0x04 = CHK_ST_ENABLED
2408 This check is currently administratively
2409 enabled.
2410 0x08 = CHK_ST_PAUSED
2411 Checks are paused because of maintenance
2412 (health only).
Willy Tarreau44aed902015-10-13 14:45:29 +02002413 srv_agent_state: State of the agent check (ENABLED/PAUSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002414 This state uses the same mask values as
2415 "srv_check_state", adding this specific one :
2416 0x10 = CHK_ST_AGENT
2417 Check is an agent check (otherwise it's a
2418 health check).
Willy Tarreau44aed902015-10-13 14:45:29 +02002419 bk_f_forced_id: Flag to know if the backend ID is forced by
2420 configuration.
2421 srv_f_forced_id: Flag to know if the server's ID is forced by
2422 configuration.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02002423 srv_fqdn: Server FQDN.
Frédéric Lécaille31694712017-08-01 08:47:19 +02002424 srv_port: Server port.
Baptiste Assmann6d0f38f2018-07-02 17:00:54 +02002425 srvrecord: DNS SRV record associated to this SRV.
William Dauchyf6370442020-11-14 19:25:33 +01002426 srv_use_ssl: use ssl for server connections.
Willy Tarreau44aed902015-10-13 14:45:29 +02002427
2428show sess
2429 Dump all known sessions. Avoid doing this on slow connections as this can
2430 be huge. This command is restricted and can only be issued on sockets
Willy Tarreauc6e7a1b2020-06-28 01:24:12 +02002431 configured for levels "operator" or "admin". Note that on machines with
2432 quickly recycled connections, it is possible that this output reports less
2433 entries than really exist because it will dump all existing sessions up to
2434 the last one that was created before the command was entered; those which
2435 die in the mean time will not appear.
Willy Tarreau44aed902015-10-13 14:45:29 +02002436
2437show sess <id>
2438 Display a lot of internal information about the specified session identifier.
2439 This identifier is the first field at the beginning of the lines in the dumps
2440 of "show sess" (it corresponds to the session pointer). Those information are
2441 useless to most users but may be used by haproxy developers to troubleshoot a
2442 complex bug. The output format is intentionally not documented so that it can
2443 freely evolve depending on demands. You may find a description of all fields
2444 returned in src/dumpstats.c
2445
2446 The special id "all" dumps the states of all sessions, which must be avoided
2447 as much as possible as it is highly CPU intensive and can take a lot of time.
2448
Daniel Corbettc40edac2020-11-01 10:54:17 -05002449show stat [domain <dns|proxy>] [{<iid>|<proxy>} <type> <sid>] [typed|json] \
Willy Tarreau698097b2020-10-23 20:19:47 +02002450 [desc] [up|no-maint]
Daniel Corbettc40edac2020-11-01 10:54:17 -05002451 Dump statistics. The domain is used to select which statistics to print; dns
2452 and proxy are available for now. By default, the CSV format is used; you can
Amaury Denoyelle072f97e2020-10-05 11:49:37 +02002453 activate the extended typed output format described in the section above if
2454 "typed" is passed after the other arguments; or in JSON if "json" is passed
2455 after the other arguments. By passing <id>, <type> and <sid>, it is possible
2456 to dump only selected items :
Willy Tarreaua1b1ed52016-11-25 08:50:58 +01002457 - <iid> is a proxy ID, -1 to dump everything. Alternatively, a proxy name
2458 <proxy> may be specified. In this case, this proxy's ID will be used as
2459 the ID selector.
Willy Tarreau44aed902015-10-13 14:45:29 +02002460 - <type> selects the type of dumpable objects : 1 for frontends, 2 for
2461 backends, 4 for servers, -1 for everything. These values can be ORed,
2462 for example:
2463 1 + 2 = 3 -> frontend + backend.
2464 1 + 2 + 4 = 7 -> frontend + backend + server.
2465 - <sid> is a server ID, -1 to dump everything from the selected proxy.
2466
2467 Example :
2468 $ echo "show info;show stat" | socat stdio unix-connect:/tmp/sock1
2469 >>> Name: HAProxy
2470 Version: 1.4-dev2-49
2471 Release_date: 2009/09/23
2472 Nbproc: 1
2473 Process_num: 1
2474 (...)
2475
2476 # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq, (...)
2477 stats,FRONTEND,,,0,0,1000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,1,0, (...)
2478 stats,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,250,(...)
2479 (...)
2480 www1,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,0,250, (...)
2481
2482 $
2483
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002484 In this example, two commands have been issued at once. That way it's easy to
2485 find which process the stats apply to in multi-process mode. This is not
2486 needed in the typed output format as the process number is reported on each
2487 line. Notice the empty line after the information output which marks the end
2488 of the first block. A similar empty line appears at the end of the second
2489 block (stats) so that the reader knows the output has not been truncated.
2490
2491 When "typed" is specified, the output format is more suitable to monitoring
2492 tools because it provides numeric positions and indicates the type of each
2493 output field. Each value stands on its own line with process number, element
2494 number, nature, origin and scope. This same format is available via the HTTP
2495 stats by passing ";typed" after the URI. It is very important to note that in
Dan Lloyd8e48b872016-07-01 21:01:18 -04002496 typed output format, the dump for a single object is contiguous so that there
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002497 is no need for a consumer to store everything at once.
2498
Willy Tarreau698097b2020-10-23 20:19:47 +02002499 The "up" modifier will result in listing only servers which reportedly up or
2500 not checked. Those down, unresolved, or in maintenance will not be listed.
2501 This is analogous to the ";up" option on the HTTP stats. Similarly, the
2502 "no-maint" modifier will act like the ";no-maint" HTTP modifier and will
2503 result in disabled servers not to be listed. The difference is that those
2504 which are enabled but down will not be evicted.
2505
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002506 When using the typed output format, each line is made of 4 columns delimited
2507 by colons (':'). The first column is a dot-delimited series of 5 elements. The
2508 first element is a letter indicating the type of the object being described.
2509 At the moment the following object types are known : 'F' for a frontend, 'B'
2510 for a backend, 'L' for a listener, and 'S' for a server. The second element
2511 The second element is a positive integer representing the unique identifier of
2512 the proxy the object belongs to. It is equivalent to the "iid" column of the
2513 CSV output and matches the value in front of the optional "id" directive found
2514 in the frontend or backend section. The third element is a positive integer
2515 containing the unique object identifier inside the proxy, and corresponds to
2516 the "sid" column of the CSV output. ID 0 is reported when dumping a frontend
2517 or a backend. For a listener or a server, this corresponds to their respective
2518 ID inside the proxy. The fourth element is the numeric position of the field
2519 in the list (starting at zero). This position shall not change over time, but
2520 holes are to be expected, depending on build options or if some fields are
2521 deleted in the future. The fifth element is the field name as it appears in
2522 the CSV output. The sixth element is a positive integer and is the relative
2523 process number starting at 1.
2524
2525 The rest of the line starting after the first colon follows the "typed output
2526 format" described in the section above. In short, the second column (after the
2527 first ':') indicates the origin, nature and scope of the variable. The third
2528 column indicates the type of the field, among "s32", "s64", "u32", "u64" and
2529 "str". Then the fourth column is the value itself, which the consumer knows
2530 how to parse thanks to column 3 and how to process thanks to column 2.
2531
Willy Tarreau6b19b142019-10-09 15:44:21 +02002532 When "desc" is appended to the command, one extra colon followed by a quoted
2533 string is appended with a description for the metric. At the time of writing,
2534 this is only supported for the "typed" output format.
2535
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002536 Thus the overall line format in typed mode is :
2537
2538 <obj>.<px_id>.<id>.<fpos>.<fname>.<process_num>:<tags>:<type>:<value>
2539
2540 Here's an example of typed output format :
2541
2542 $ echo "show stat typed" | socat stdio unix-connect:/tmp/sock1
2543 F.2.0.0.pxname.1:MGP:str:private-frontend
2544 F.2.0.1.svname.1:MGP:str:FRONTEND
2545 F.2.0.8.bin.1:MGP:u64:0
2546 F.2.0.9.bout.1:MGP:u64:0
2547 F.2.0.40.hrsp_2xx.1:MGP:u64:0
2548 L.2.1.0.pxname.1:MGP:str:private-frontend
2549 L.2.1.1.svname.1:MGP:str:sock-1
2550 L.2.1.17.status.1:MGP:str:OPEN
2551 L.2.1.73.addr.1:MGP:str:0.0.0.0:8001
2552 S.3.13.60.rtime.1:MCP:u32:0
2553 S.3.13.61.ttime.1:MCP:u32:0
2554 S.3.13.62.agent_status.1:MGP:str:L4TOUT
2555 S.3.13.64.agent_duration.1:MGP:u64:2001
2556 S.3.13.65.check_desc.1:MCP:str:Layer4 timeout
2557 S.3.13.66.agent_desc.1:MCP:str:Layer4 timeout
2558 S.3.13.67.check_rise.1:MCP:u32:2
2559 S.3.13.68.check_fall.1:MCP:u32:3
2560 S.3.13.69.check_health.1:SGP:u32:0
2561 S.3.13.70.agent_rise.1:MaP:u32:1
2562 S.3.13.71.agent_fall.1:SGP:u32:1
2563 S.3.13.72.agent_health.1:SGP:u32:1
2564 S.3.13.73.addr.1:MCP:str:1.255.255.255:8888
2565 S.3.13.75.mode.1:MAP:str:http
2566 B.3.0.0.pxname.1:MGP:str:private-backend
2567 B.3.0.1.svname.1:MGP:str:BACKEND
2568 B.3.0.2.qcur.1:MGP:u32:0
2569 B.3.0.3.qmax.1:MGP:u32:0
2570 B.3.0.4.scur.1:MGP:u32:0
2571 B.3.0.5.smax.1:MGP:u32:0
2572 B.3.0.6.slim.1:MGP:u32:1000
2573 B.3.0.55.lastsess.1:MMP:s32:-1
2574 (...)
2575
Simon Horman1084a362016-11-21 17:00:24 +01002576 In the typed format, the presence of the process ID at the end of the
2577 first column makes it very easy to visually aggregate outputs from
2578 multiple processes, as show in the example below where each line appears
2579 for each process :
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002580
2581 $ ( echo show stat typed | socat /var/run/haproxy.sock1 - ; \
2582 echo show stat typed | socat /var/run/haproxy.sock2 - ) | \
2583 sort -t . -k 1,1 -k 2,2n -k 3,3n -k 4,4n -k 5,5 -k 6,6n
2584 B.3.0.0.pxname.1:MGP:str:private-backend
2585 B.3.0.0.pxname.2:MGP:str:private-backend
2586 B.3.0.1.svname.1:MGP:str:BACKEND
2587 B.3.0.1.svname.2:MGP:str:BACKEND
2588 B.3.0.2.qcur.1:MGP:u32:0
2589 B.3.0.2.qcur.2:MGP:u32:0
2590 B.3.0.3.qmax.1:MGP:u32:0
2591 B.3.0.3.qmax.2:MGP:u32:0
2592 B.3.0.4.scur.1:MGP:u32:0
2593 B.3.0.4.scur.2:MGP:u32:0
2594 B.3.0.5.smax.1:MGP:u32:0
2595 B.3.0.5.smax.2:MGP:u32:0
2596 B.3.0.6.slim.1:MGP:u32:1000
2597 B.3.0.6.slim.2:MGP:u32:1000
2598 (...)
Willy Tarreau44aed902015-10-13 14:45:29 +02002599
Simon Horman05ee2132017-01-04 09:37:25 +01002600 The format of JSON output is described in a schema which may be output
Simon Horman6f6bb382017-01-04 09:37:26 +01002601 using "show schema json".
2602
2603 The JSON output contains no extra whitespace in order to reduce the
2604 volume of output. For human consumption passing the output through a
2605 pretty printer may be helpful. Example :
2606
2607 $ echo "show stat json" | socat /var/run/haproxy.sock stdio | \
2608 python -m json.tool
Simon Horman05ee2132017-01-04 09:37:25 +01002609
2610 The JSON output contains no extra whitespace in order to reduce the
2611 volume of output. For human consumption passing the output through a
2612 pretty printer may be helpful. Example :
2613
2614 $ echo "show stat json" | socat /var/run/haproxy.sock stdio | \
2615 python -m json.tool
2616
William Lallemandd4f946c2019-12-05 10:26:40 +01002617show ssl cert [<filename>]
Ilya Shipitsin2a950d02020-03-06 13:07:38 +05002618 Display the list of certificates used on frontends. If a filename is prefixed
2619 by an asterisk, it is a transaction which is not committed yet. If a
William Lallemandd4f946c2019-12-05 10:26:40 +01002620 filename is specified, it will show details about the certificate. This
2621 command can be useful to check if a certificate was well updated. You can
2622 also display details on a transaction by prefixing the filename by an
2623 asterisk.
2624
2625 Example :
2626
2627 $ echo "@1 show ssl cert" | socat /var/run/haproxy.master -
2628 # transaction
2629 *test.local.pem
2630 # filename
2631 test.local.pem
2632
2633 $ echo "@1 show ssl cert test.local.pem" | socat /var/run/haproxy.master -
2634 Filename: test.local.pem
2635 Serial: 03ECC19BA54B25E85ABA46EE561B9A10D26F
2636 notBefore: Sep 13 21:20:24 2019 GMT
2637 notAfter: Dec 12 21:20:24 2019 GMT
2638 Issuer: /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
2639 Subject: /CN=test.local
2640 Subject Alternative Name: DNS:test.local, DNS:imap.test.local
2641 Algorithm: RSA2048
2642 SHA1 FingerPrint: 417A11CAE25F607B24F638B4A8AEE51D1E211477
2643
2644 $ echo "@1 show ssl cert *test.local.pem" | socat /var/run/haproxy.master -
2645 Filename: *test.local.pem
2646 [...]
2647
William Lallemandc69f02d2020-04-06 19:07:03 +02002648show ssl crt-list [-n] [<filename>]
William Lallemandaccac232020-04-02 17:42:51 +02002649 Display the list of crt-list and directories used in the HAProxy
William Lallemandc69f02d2020-04-06 19:07:03 +02002650 configuration. If a filename is specified, dump the content of a crt-list or
2651 a directory. Once dumped the output can be used as a crt-list file.
2652 The '-n' option can be used to display the line number, which is useful when
2653 combined with the 'del ssl crt-list' option when a entry is duplicated. The
2654 output with the '-n' option is not compatible with the crt-list format and
2655 not loadable by haproxy.
William Lallemandaccac232020-04-02 17:42:51 +02002656
2657 Example:
William Lallemandc69f02d2020-04-06 19:07:03 +02002658 echo "show ssl crt-list -n localhost.crt-list" | socat /tmp/sock1 -
William Lallemandaccac232020-04-02 17:42:51 +02002659 # localhost.crt-list
William Lallemandc69f02d2020-04-06 19:07:03 +02002660 common.pem:1 !not.test1.com *.test1.com !localhost
2661 common.pem:2
2662 ecdsa.pem:3 [verify none allow-0rtt ssl-min-ver TLSv1.0 ssl-max-ver TLSv1.3] localhost !www.test1.com
2663 ecdsa.pem:4 [verify none allow-0rtt ssl-min-ver TLSv1.0 ssl-max-ver TLSv1.3]
William Lallemandaccac232020-04-02 17:42:51 +02002664
Christopher Faulet78c43062019-09-27 10:45:47 +02002665show resolvers [<resolvers section id>]
Willy Tarreau44aed902015-10-13 14:45:29 +02002666 Dump statistics for the given resolvers section, or all resolvers sections
2667 if no section is supplied.
2668
2669 For each name server, the following counters are reported:
2670 sent: number of DNS requests sent to this server
2671 valid: number of DNS valid responses received from this server
2672 update: number of DNS responses used to update the server's IP address
2673 cname: number of CNAME responses
2674 cname_error: CNAME errors encountered with this server
2675 any_err: number of empty response (IE: server does not support ANY type)
2676 nx: non existent domain response received from this server
2677 timeout: how many time this server did not answer in time
2678 refused: number of requests refused by this server
2679 other: any other DNS errors
2680 invalid: invalid DNS response (from a protocol point of view)
2681 too_big: too big response
2682 outdated: number of response arrived too late (after an other name server)
2683
2684show table
2685 Dump general information on all known stick-tables. Their name is returned
2686 (the name of the proxy which holds them), their type (currently zero, always
2687 IP), their size in maximum possible number of entries, and the number of
2688 entries currently in use.
2689
2690 Example :
2691 $ echo "show table" | socat stdio /tmp/sock1
2692 >>> # table: front_pub, type: ip, size:204800, used:171454
2693 >>> # table: back_rdp, type: ip, size:204800, used:0
2694
Adis Nezirovic1a693fc2020-01-16 15:19:29 +01002695show table <name> [ data.<type> <operator> <value> [data.<type> ...]] | [ key <key> ]
Willy Tarreau44aed902015-10-13 14:45:29 +02002696 Dump contents of stick-table <name>. In this mode, a first line of generic
2697 information about the table is reported as with "show table", then all
2698 entries are dumped. Since this can be quite heavy, it is possible to specify
2699 a filter in order to specify what entries to display.
2700
2701 When the "data." form is used the filter applies to the stored data (see
2702 "stick-table" in section 4.2). A stored data type must be specified
2703 in <type>, and this data type must be stored in the table otherwise an
2704 error is reported. The data is compared according to <operator> with the
2705 64-bit integer <value>. Operators are the same as with the ACLs :
2706
2707 - eq : match entries whose data is equal to this value
2708 - ne : match entries whose data is not equal to this value
2709 - le : match entries whose data is less than or equal to this value
2710 - ge : match entries whose data is greater than or equal to this value
2711 - lt : match entries whose data is less than this value
2712 - gt : match entries whose data is greater than this value
2713
Adis Nezirovic1a693fc2020-01-16 15:19:29 +01002714 In this form, you can use multiple data filter entries, up to a maximum
2715 defined during build time (4 by default).
Willy Tarreau44aed902015-10-13 14:45:29 +02002716
2717 When the key form is used the entry <key> is shown. The key must be of the
2718 same type as the table, which currently is limited to IPv4, IPv6, integer,
2719 and string.
2720
2721 Example :
2722 $ echo "show table http_proxy" | socat stdio /tmp/sock1
2723 >>> # table: http_proxy, type: ip, size:204800, used:2
2724 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
2725 bytes_out_rate(60000)=187
2726 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2727 bytes_out_rate(60000)=191
2728
2729 $ echo "show table http_proxy data.gpc0 gt 0" | socat stdio /tmp/sock1
2730 >>> # table: http_proxy, type: ip, size:204800, used:2
2731 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2732 bytes_out_rate(60000)=191
2733
2734 $ echo "show table http_proxy data.conn_rate gt 5" | \
2735 socat stdio /tmp/sock1
2736 >>> # table: http_proxy, type: ip, size:204800, used:2
2737 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2738 bytes_out_rate(60000)=191
2739
2740 $ echo "show table http_proxy key 127.0.0.2" | \
2741 socat stdio /tmp/sock1
2742 >>> # table: http_proxy, type: ip, size:204800, used:2
2743 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2744 bytes_out_rate(60000)=191
2745
2746 When the data criterion applies to a dynamic value dependent on time such as
2747 a bytes rate, the value is dynamically computed during the evaluation of the
2748 entry in order to decide whether it has to be dumped or not. This means that
2749 such a filter could match for some time then not match anymore because as
2750 time goes, the average event rate drops.
2751
2752 It is possible to use this to extract lists of IP addresses abusing the
2753 service, in order to monitor them or even blacklist them in a firewall.
2754 Example :
2755 $ echo "show table http_proxy data.gpc0 gt 0" \
2756 | socat stdio /tmp/sock1 \
2757 | fgrep 'key=' | cut -d' ' -f2 | cut -d= -f2 > abusers-ip.txt
2758 ( or | awk '/key/{ print a[split($2,a,"=")]; }' )
2759
Willy Tarreau4e2b6462019-05-16 17:44:30 +02002760show threads
2761 Dumps some internal states and structures for each thread, that may be useful
2762 to help developers understand a problem. The output tries to be readable by
Willy Tarreauc7091d82019-05-17 10:08:49 +02002763 showing one block per thread. When haproxy is built with USE_THREAD_DUMP=1,
2764 an advanced dump mechanism involving thread signals is used so that each
2765 thread can dump its own state in turn. Without this option, the thread
2766 processing the command shows all its details but the other ones are less
Willy Tarreaue6a02fa2019-05-22 07:06:44 +02002767 detailed. A star ('*') is displayed in front of the thread handling the
2768 command. A right angle bracket ('>') may also be displayed in front of
2769 threads which didn't make any progress since last invocation of this command,
2770 indicating a bug in the code which must absolutely be reported. When this
2771 happens between two threads it usually indicates a deadlock. If a thread is
2772 alone, it's a different bug like a corrupted list. In all cases the process
2773 needs is not fully functional anymore and needs to be restarted.
2774
2775 The output format is purposely not documented so that it can easily evolve as
2776 new needs are identified, without having to maintain any form of backwards
2777 compatibility, and just like with "show activity", the values are meaningless
2778 without the code at hand.
Willy Tarreau4e2b6462019-05-16 17:44:30 +02002779
William Lallemandbb933462016-05-31 21:09:53 +02002780show tls-keys [id|*]
2781 Dump all loaded TLS ticket keys references. The TLS ticket key reference ID
2782 and the file from which the keys have been loaded is shown. Both of those
2783 can be used to update the TLS keys using "set ssl tls-key". If an ID is
2784 specified as parameter, it will dump the tickets, using * it will dump every
2785 keys from every references.
Willy Tarreau44aed902015-10-13 14:45:29 +02002786
Simon Horman6f6bb382017-01-04 09:37:26 +01002787show schema json
2788 Dump the schema used for the output of "show info json" and "show stat json".
2789
2790 The contains no extra whitespace in order to reduce the volume of output.
2791 For human consumption passing the output through a pretty printer may be
2792 helpful. Example :
2793
2794 $ echo "show schema json" | socat /var/run/haproxy.sock stdio | \
2795 python -m json.tool
2796
2797 The schema follows "JSON Schema" (json-schema.org) and accordingly
2798 verifiers may be used to verify the output of "show info json" and "show
2799 stat json" against the schema.
2800
Willy Tarreauf909c912019-08-22 20:06:04 +02002801show trace [<source>]
2802 Show the current trace status. For each source a line is displayed with a
2803 single-character status indicating if the trace is stopped, waiting, or
2804 running. The output sink used by the trace is indicated (or "none" if none
2805 was set), as well as the number of dropped events in this sink, followed by a
2806 brief description of the source. If a source name is specified, a detailed
2807 list of all events supported by the source, and their status for each action
2808 (report, start, pause, stop), indicated by a "+" if they are enabled, or a
2809 "-" otherwise. All these events are independent and an event might trigger
2810 a start without being reported and conversely.
Simon Horman6f6bb382017-01-04 09:37:26 +01002811
Willy Tarreau44aed902015-10-13 14:45:29 +02002812shutdown frontend <frontend>
2813 Completely delete the specified frontend. All the ports it was bound to will
2814 be released. It will not be possible to enable the frontend anymore after
2815 this operation. This is intended to be used in environments where stopping a
2816 proxy is not even imaginable but a misconfigured proxy must be fixed. That
2817 way it's possible to release the port and bind it into another process to
2818 restore operations. The frontend will not appear at all on the stats page
2819 once it is terminated.
2820
2821 The frontend may be specified either by its name or by its numeric ID,
2822 prefixed with a sharp ('#').
2823
2824 This command is restricted and can only be issued on sockets configured for
2825 level "admin".
2826
2827shutdown session <id>
2828 Immediately terminate the session matching the specified session identifier.
2829 This identifier is the first field at the beginning of the lines in the dumps
2830 of "show sess" (it corresponds to the session pointer). This can be used to
2831 terminate a long-running session without waiting for a timeout or when an
2832 endless transfer is ongoing. Such terminated sessions are reported with a 'K'
2833 flag in the logs.
2834
2835shutdown sessions server <backend>/<server>
2836 Immediately terminate all the sessions attached to the specified server. This
2837 can be used to terminate long-running sessions after a server is put into
2838 maintenance mode, for instance. Such terminated sessions are reported with a
2839 'K' flag in the logs.
2840
Willy Tarreauf909c912019-08-22 20:06:04 +02002841trace
2842 The "trace" command alone lists the trace sources, their current status, and
2843 their brief descriptions. It is only meant as a menu to enter next levels,
2844 see other "trace" commands below.
2845
2846trace 0
2847 Immediately stops all traces. This is made to be used as a quick solution
2848 to terminate a debugging session or as an emergency action to be used in case
2849 complex traces were enabled on multiple sources and impact the service.
2850
2851trace <source> event [ [+|-|!]<name> ]
2852 Without argument, this will list all the events supported by the designated
2853 source. They are prefixed with a "-" if they are not enabled, or a "+" if
2854 they are enabled. It is important to note that a single trace may be labelled
2855 with multiple events, and as long as any of the enabled events matches one of
2856 the events labelled on the trace, the event will be passed to the trace
2857 subsystem. For example, receiving an HTTP/2 frame of type HEADERS may trigger
2858 a frame event and a stream event since the frame creates a new stream. If
2859 either the frame event or the stream event are enabled for this source, the
2860 frame will be passed to the trace framework.
2861
2862 With an argument, it is possible to toggle the state of each event and
2863 individually enable or disable them. Two special keywords are supported,
2864 "none", which matches no event, and is used to disable all events at once,
2865 and "any" which matches all events, and is used to enable all events at
2866 once. Other events are specific to the event source. It is possible to
2867 enable one event by specifying its name, optionally prefixed with '+' for
2868 better readability. It is possible to disable one event by specifying its
2869 name prefixed by a '-' or a '!'.
2870
2871 One way to completely disable a trace source is to pass "event none", and
2872 this source will instantly be totally ignored.
2873
2874trace <source> level [<level>]
Willy Tarreau2ea549b2019-08-29 08:01:48 +02002875 Without argument, this will list all trace levels for this source, and the
Willy Tarreauf909c912019-08-22 20:06:04 +02002876 current one will be indicated by a star ('*') prepended in front of it. With
Willy Tarreau2ea549b2019-08-29 08:01:48 +02002877 an argument, this will change the trace level to the specified level. Detail
Willy Tarreauf909c912019-08-22 20:06:04 +02002878 levels are a form of filters that are applied before reporting the events.
Willy Tarreau2ea549b2019-08-29 08:01:48 +02002879 These filters are used to selectively include or exclude events depending on
2880 their level of importance. For example a developer might need to know
2881 precisely where in the code an HTTP header was considered invalid while the
2882 end user may not even care about this header's validity at all. There are
2883 currently 5 distinct levels for a trace :
Willy Tarreauf909c912019-08-22 20:06:04 +02002884
2885 user this will report information that are suitable for use by a
2886 regular haproxy user who wants to observe his traffic.
2887 Typically some HTTP requests and responses will be reported
2888 without much detail. Most sources will set this as the
2889 default level to ease operations.
2890
Willy Tarreau2ea549b2019-08-29 08:01:48 +02002891 proto in addition to what is reported at the "user" level, it also
2892 displays protocol-level updates. This can for example be the
2893 frame types or HTTP headers after decoding.
Willy Tarreauf909c912019-08-22 20:06:04 +02002894
2895 state in addition to what is reported at the "proto" level, it
2896 will also display state transitions (or failed transitions)
2897 which happen in parsers, so this will show attempts to
2898 perform an operation while the "proto" level only shows
2899 the final operation.
2900
Willy Tarreau2ea549b2019-08-29 08:01:48 +02002901 data in addition to what is reported at the "state" level, it
2902 will also include data transfers between the various layers.
2903
Willy Tarreauf909c912019-08-22 20:06:04 +02002904 developer it reports everything available, which can include advanced
2905 information such as "breaking out of this loop" that are
2906 only relevant to a developer trying to understand a bug that
Willy Tarreau09fb0df2019-08-29 08:40:59 +02002907 only happens once in a while in field. Function names are
2908 only reported at this level.
Willy Tarreauf909c912019-08-22 20:06:04 +02002909
2910 It is highly recommended to always use the "user" level only and switch to
2911 other levels only if instructed to do so by a developer. Also it is a good
2912 idea to first configure the events before switching to higher levels, as it
2913 may save from dumping many lines if no filter is applied.
2914
2915trace <source> lock [criterion]
2916 Without argument, this will list all the criteria supported by this source
2917 for lock-on processing, and display the current choice by a star ('*') in
2918 front of it. Lock-on means that the source will focus on the first matching
2919 event and only stick to the criterion which triggered this event, and ignore
2920 all other ones until the trace stops. This allows for example to take a trace
2921 on a single connection or on a single stream. The following criteria are
2922 supported by some traces, though not necessarily all, since some of them
2923 might not be available to the source :
2924
2925 backend lock on the backend that started the trace
2926 connection lock on the connection that started the trace
2927 frontend lock on the frontend that started the trace
2928 listener lock on the listener that started the trace
2929 nothing do not lock on anything
2930 server lock on the server that started the trace
2931 session lock on the session that started the trace
2932 thread lock on the thread that started the trace
2933
2934 In addition to this, each source may provide up to 4 specific criteria such
2935 as internal states or connection IDs. For example in HTTP/2 it is possible
2936 to lock on the H2 stream and ignore other streams once a strace starts.
2937
2938 When a criterion is passed in argument, this one is used instead of the
2939 other ones and any existing tracking is immediately terminated so that it can
2940 restart with the new criterion. The special keyword "nothing" is supported by
2941 all sources to permanently disable tracking.
2942
2943trace <source> { pause | start | stop } [ [+|-|!]event]
2944 Without argument, this will list the events enabled to automatically pause,
2945 start, or stop a trace for this source. These events are specific to each
2946 trace source. With an argument, this will either enable the event for the
2947 specified action (if optionally prefixed by a '+') or disable it (if
2948 prefixed by a '-' or '!'). The special keyword "now" is not an event and
2949 requests to take the action immediately. The keywords "none" and "any" are
2950 supported just like in "trace event".
2951
2952 The 3 supported actions are respectively "pause", "start" and "stop". The
2953 "pause" action enumerates events which will cause a running trace to stop and
2954 wait for a new start event to restart it. The "start" action enumerates the
2955 events which switch the trace into the waiting mode until one of the start
2956 events appears. And the "stop" action enumerates the events which definitely
2957 stop the trace until it is manually enabled again. In practice it makes sense
2958 to manually start a trace using "start now" without caring about events, and
2959 to stop it using "stop now". In order to capture more subtle event sequences,
2960 setting "start" to a normal event (like receiving an HTTP request) and "stop"
2961 to a very rare event like emitting a certain error, will ensure that the last
2962 captured events will match the desired criteria. And the pause event is
2963 useful to detect the end of a sequence, disable the lock-on and wait for
2964 another opportunity to take a capture. In this case it can make sense to
2965 enable lock-on to spot only one specific criterion (e.g. a stream), and have
2966 "start" set to anything that starts this criterion (e.g. all events which
2967 create a stream), "stop" set to the expected anomaly, and "pause" to anything
2968 that ends that criterion (e.g. any end of stream event). In this case the
2969 trace log will contain complete sequences of perfectly clean series affecting
2970 a single object, until the last sequence containing everything from the
2971 beginning to the anomaly.
2972
2973trace <source> sink [<sink>]
2974 Without argument, this will list all event sinks available for this source,
2975 and the currently configured one will have a star ('*') prepended in front
2976 of it. Sink "none" is always available and means that all events are simply
2977 dropped, though their processing is not ignored (e.g. lock-on does occur).
2978 Other sinks are available depending on configuration and build options, but
2979 typically "stdout" and "stderr" will be usable in debug mode, and in-memory
2980 ring buffers should be available as well. When a name is specified, the sink
2981 instantly changes for the specified source. Events are not changed during a
2982 sink change. In the worst case some may be lost if an invalid sink is used
2983 (or "none"), but operations do continue to a different destination.
2984
Willy Tarreau370a6942019-08-29 08:24:16 +02002985trace <source> verbosity [<level>]
2986 Without argument, this will list all verbosity levels for this source, and the
2987 current one will be indicated by a star ('*') prepended in front of it. With
2988 an argument, this will change the verbosity level to the specified one.
2989
2990 Verbosity levels indicate how far the trace decoder should go to provide
2991 detailed information. It depends on the trace source, since some sources will
2992 not even provide a specific decoder. Level "quiet" is always available and
2993 disables any decoding. It can be useful when trying to figure what's
2994 happening before trying to understand the details, since it will have a very
2995 low impact on performance and trace size. When no verbosity levels are
2996 declared by a source, level "default" is available and will cause a decoder
2997 to be called when specified in the traces. It is an opportunistic decoding.
2998 When the source declares some verbosity levels, these ones are listed with
2999 a description of what they correspond to. In this case the trace decoder
3000 provided by the source will be as accurate as possible based on the
3001 information available at the trace point. The first level above "quiet" is
3002 set by default.
3003
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003004
William Lallemand142db372018-12-11 18:56:45 +010030059.4. Master CLI
3006---------------
3007
3008The master CLI is a socket bound to the master process in master-worker mode.
3009This CLI gives access to the unix socket commands in every running or leaving
3010processes and allows a basic supervision of those processes.
3011
3012The master CLI is configurable only from the haproxy program arguments with
3013the -S option. This option also takes bind options separated by commas.
3014
3015Example:
3016
3017 # haproxy -W -S 127.0.0.1:1234 -f test1.cfg
3018 # haproxy -Ws -S /tmp/master-socket,uid,1000,gid,1000,mode,600 -f test1.cfg
William Lallemandb7ea1412018-12-13 09:05:47 +01003019 # haproxy -W -S /tmp/master-socket,level,user -f test1.cfg
William Lallemand142db372018-12-11 18:56:45 +01003020
3021The master CLI introduces a new 'show proc' command to surpervise the
3022processes:
3023
3024Example:
3025
3026 $ echo 'show proc' | socat /var/run/haproxy-master.sock -
William Lallemand1dc69632019-06-12 19:11:33 +02003027 #<PID> <type> <relative PID> <reloads> <uptime> <version>
3028 1162 master 0 5 0d00h02m07s 2.0-dev7-0124c9-7
William Lallemand142db372018-12-11 18:56:45 +01003029 # workers
William Lallemand1dc69632019-06-12 19:11:33 +02003030 1271 worker 1 0 0d00h00m00s 2.0-dev7-0124c9-7
3031 1272 worker 2 0 0d00h00m00s 2.0-dev7-0124c9-7
William Lallemand142db372018-12-11 18:56:45 +01003032 # old workers
William Lallemand1dc69632019-06-12 19:11:33 +02003033 1233 worker [was: 1] 3 0d00h00m43s 2.0-dev3-6019f6-289
William Lallemand142db372018-12-11 18:56:45 +01003034
3035
3036In this example, the master has been reloaded 5 times but one of the old
3037worker is still running and survived 3 reloads. You could access the CLI of
3038this worker to understand what's going on.
3039
Willy Tarreau52880f92018-12-15 13:30:03 +01003040When the prompt is enabled (via the "prompt" command), the context the CLI is
3041working on is displayed in the prompt. The master is identified by the "master"
3042string, and other processes are identified with their PID. In case the last
3043reload failed, the master prompt will be changed to "master[ReloadFailed]>" so
3044that it becomes visible that the process is still running on the previous
3045configuration and that the new configuration is not operational.
3046
William Lallemand142db372018-12-11 18:56:45 +01003047The master CLI uses a special prefix notation to access the multiple
3048processes. This notation is easily identifiable as it begins by a @.
3049
3050A @ prefix can be followed by a relative process number or by an exclamation
3051point and a PID. (e.g. @1 or @!1271). A @ alone could be use to specify the
3052master. Leaving processes are only accessible with the PID as relative process
3053number are only usable with the current processes.
3054
3055Examples:
3056
3057 $ socat /var/run/haproxy-master.sock readline
3058 prompt
3059 master> @1 show info; @2 show info
3060 [...]
3061 Process_num: 1
3062 Pid: 1271
3063 [...]
3064 Process_num: 2
3065 Pid: 1272
3066 [...]
3067 master>
3068
3069 $ echo '@!1271 show info; @!1272 show info' | socat /var/run/haproxy-master.sock -
3070 [...]
3071
3072A prefix could be use as a command, which will send every next commands to
3073the specified process.
3074
3075Examples:
3076
3077 $ socat /var/run/haproxy-master.sock readline
3078 prompt
3079 master> @1
3080 1271> show info
3081 [...]
3082 1271> show stat
3083 [...]
3084 1271> @
3085 master>
3086
3087 $ echo '@1; show info; show stat; @2; show info; show stat' | socat /var/run/haproxy-master.sock -
3088 [...]
3089
William Lallemanda57b7e32018-12-14 21:11:31 +01003090You can also reload the HAProxy master process with the "reload" command which
3091does the same as a `kill -USR2` on the master process, provided that the user
3092has at least "operator" or "admin" privileges.
3093
3094Example:
3095
3096 $ echo "reload" | socat /var/run/haproxy-master.sock
3097
3098Note that a reload will close the connection to the master CLI.
3099
William Lallemand142db372018-12-11 18:56:45 +01003100
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200310110. Tricks for easier configuration management
3102----------------------------------------------
3103
3104It is very common that two HAProxy nodes constituting a cluster share exactly
3105the same configuration modulo a few addresses. Instead of having to maintain a
3106duplicate configuration for each node, which will inevitably diverge, it is
3107possible to include environment variables in the configuration. Thus multiple
3108configuration may share the exact same file with only a few different system
3109wide environment variables. This started in version 1.5 where only addresses
3110were allowed to include environment variables, and 1.6 goes further by
3111supporting environment variables everywhere. The syntax is the same as in the
3112UNIX shell, a variable starts with a dollar sign ('$'), followed by an opening
3113curly brace ('{'), then the variable name followed by the closing brace ('}').
3114Except for addresses, environment variables are only interpreted in arguments
3115surrounded with double quotes (this was necessary not to break existing setups
3116using regular expressions involving the dollar symbol).
3117
3118Environment variables also make it convenient to write configurations which are
3119expected to work on various sites where only the address changes. It can also
3120permit to remove passwords from some configs. Example below where the the file
3121"site1.env" file is sourced by the init script upon startup :
3122
3123 $ cat site1.env
3124 LISTEN=192.168.1.1
3125 CACHE_PFX=192.168.11
3126 SERVER_PFX=192.168.22
3127 LOGGER=192.168.33.1
3128 STATSLP=admin:pa$$w0rd
3129 ABUSERS=/etc/haproxy/abuse.lst
3130 TIMEOUT=10s
3131
3132 $ cat haproxy.cfg
3133 global
3134 log "${LOGGER}:514" local0
3135
3136 defaults
3137 mode http
3138 timeout client "${TIMEOUT}"
3139 timeout server "${TIMEOUT}"
3140 timeout connect 5s
3141
3142 frontend public
3143 bind "${LISTEN}:80"
3144 http-request reject if { src -f "${ABUSERS}" }
3145 stats uri /stats
3146 stats auth "${STATSLP}"
3147 use_backend cache if { path_end .jpg .css .ico }
3148 default_backend server
3149
3150 backend cache
3151 server cache1 "${CACHE_PFX}.1:18080" check
3152 server cache2 "${CACHE_PFX}.2:18080" check
3153
3154 backend server
3155 server cache1 "${SERVER_PFX}.1:8080" check
3156 server cache2 "${SERVER_PFX}.2:8080" check
3157
3158
315911. Well-known traps to avoid
3160-----------------------------
3161
3162Once in a while, someone reports that after a system reboot, the haproxy
3163service wasn't started, and that once they start it by hand it works. Most
3164often, these people are running a clustered IP address mechanism such as
3165keepalived, to assign the service IP address to the master node only, and while
3166it used to work when they used to bind haproxy to address 0.0.0.0, it stopped
3167working after they bound it to the virtual IP address. What happens here is
3168that when the service starts, the virtual IP address is not yet owned by the
3169local node, so when HAProxy wants to bind to it, the system rejects this
3170because it is not a local IP address. The fix doesn't consist in delaying the
3171haproxy service startup (since it wouldn't stand a restart), but instead to
3172properly configure the system to allow binding to non-local addresses. This is
3173easily done on Linux by setting the net.ipv4.ip_nonlocal_bind sysctl to 1. This
3174is also needed in order to transparently intercept the IP traffic that passes
3175through HAProxy for a specific target address.
3176
3177Multi-process configurations involving source port ranges may apparently seem
3178to work but they will cause some random failures under high loads because more
3179than one process may try to use the same source port to connect to the same
3180server, which is not possible. The system will report an error and a retry will
3181happen, picking another port. A high value in the "retries" parameter may hide
3182the effect to a certain extent but this also comes with increased CPU usage and
3183processing time. Logs will also report a certain number of retries. For this
3184reason, port ranges should be avoided in multi-process configurations.
3185
Dan Lloyd8e48b872016-07-01 21:01:18 -04003186Since HAProxy uses SO_REUSEPORT and supports having multiple independent
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003187processes bound to the same IP:port, during troubleshooting it can happen that
3188an old process was not stopped before a new one was started. This provides
3189absurd test results which tend to indicate that any change to the configuration
3190is ignored. The reason is that in fact even the new process is restarted with a
3191new configuration, the old one also gets some incoming connections and
3192processes them, returning unexpected results. When in doubt, just stop the new
3193process and try again. If it still works, it very likely means that an old
3194process remains alive and has to be stopped. Linux's "netstat -lntp" is of good
3195help here.
3196
3197When adding entries to an ACL from the command line (eg: when blacklisting a
3198source address), it is important to keep in mind that these entries are not
3199synchronized to the file and that if someone reloads the configuration, these
3200updates will be lost. While this is often the desired effect (for blacklisting)
3201it may not necessarily match expectations when the change was made as a fix for
3202a problem. See the "add acl" action of the CLI interface.
3203
3204
320512. Debugging and performance issues
3206------------------------------------
3207
3208When HAProxy is started with the "-d" option, it will stay in the foreground
3209and will print one line per event, such as an incoming connection, the end of a
3210connection, and for each request or response header line seen. This debug
3211output is emitted before the contents are processed, so they don't consider the
3212local modifications. The main use is to show the request and response without
3213having to run a network sniffer. The output is less readable when multiple
3214connections are handled in parallel, though the "debug2ansi" and "debug2html"
3215scripts found in the examples/ directory definitely help here by coloring the
3216output.
3217
3218If a request or response is rejected because HAProxy finds it is malformed, the
3219best thing to do is to connect to the CLI and issue "show errors", which will
3220report the last captured faulty request and response for each frontend and
3221backend, with all the necessary information to indicate precisely the first
3222character of the input stream that was rejected. This is sometimes needed to
3223prove to customers or to developers that a bug is present in their code. In
3224this case it is often possible to relax the checks (but still keep the
3225captures) using "option accept-invalid-http-request" or its equivalent for
3226responses coming from the server "option accept-invalid-http-response". Please
3227see the configuration manual for more details.
3228
3229Example :
3230
3231 > show errors
3232 Total events captured on [13/Oct/2015:13:43:47.169] : 1
3233
3234 [13/Oct/2015:13:43:40.918] frontend HAProxyLocalStats (#2): invalid request
3235 backend <NONE> (#-1), server <NONE> (#-1), event #0
3236 src 127.0.0.1:51981, session #0, session flags 0x00000080
3237 HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000
3238 HTTP chunk len 0 bytes, HTTP body len 0 bytes
3239 buffer flags 0x00808002, out 0 bytes, total 31 bytes
3240 pending 31 bytes, wrapping at 8040, error at position 13:
3241
3242 00000 GET /invalid request HTTP/1.1\r\n
3243
3244
3245The output of "show info" on the CLI provides a number of useful information
3246regarding the maximum connection rate ever reached, maximum SSL key rate ever
3247reached, and in general all information which can help to explain temporary
3248issues regarding CPU or memory usage. Example :
3249
3250 > show info
3251 Name: HAProxy
3252 Version: 1.6-dev7-e32d18-17
3253 Release_date: 2015/10/12
3254 Nbproc: 1
3255 Process_num: 1
3256 Pid: 7949
3257 Uptime: 0d 0h02m39s
3258 Uptime_sec: 159
3259 Memmax_MB: 0
3260 Ulimit-n: 120032
3261 Maxsock: 120032
3262 Maxconn: 60000
3263 Hard_maxconn: 60000
3264 CurrConns: 0
3265 CumConns: 3
3266 CumReq: 3
3267 MaxSslConns: 0
3268 CurrSslConns: 0
3269 CumSslConns: 0
3270 Maxpipes: 0
3271 PipesUsed: 0
3272 PipesFree: 0
3273 ConnRate: 0
3274 ConnRateLimit: 0
3275 MaxConnRate: 1
3276 SessRate: 0
3277 SessRateLimit: 0
3278 MaxSessRate: 1
3279 SslRate: 0
3280 SslRateLimit: 0
3281 MaxSslRate: 0
3282 SslFrontendKeyRate: 0
3283 SslFrontendMaxKeyRate: 0
3284 SslFrontendSessionReuse_pct: 0
3285 SslBackendKeyRate: 0
3286 SslBackendMaxKeyRate: 0
3287 SslCacheLookups: 0
3288 SslCacheMisses: 0
3289 CompressBpsIn: 0
3290 CompressBpsOut: 0
3291 CompressBpsRateLim: 0
3292 ZlibMemUsage: 0
3293 MaxZlibMemUsage: 0
3294 Tasks: 5
3295 Run_queue: 1
3296 Idle_pct: 100
3297 node: wtap
3298 description:
3299
3300When an issue seems to randomly appear on a new version of HAProxy (eg: every
3301second request is aborted, occasional crash, etc), it is worth trying to enable
Dan Lloyd8e48b872016-07-01 21:01:18 -04003302memory poisoning so that each call to malloc() is immediately followed by the
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003303filling of the memory area with a configurable byte. By default this byte is
33040x50 (ASCII for 'P'), but any other byte can be used, including zero (which
3305will have the same effect as a calloc() and which may make issues disappear).
Dan Lloyd8e48b872016-07-01 21:01:18 -04003306Memory poisoning is enabled on the command line using the "-dM" option. It
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003307slightly hurts performance and is not recommended for use in production. If
Dan Lloyd8e48b872016-07-01 21:01:18 -04003308an issue happens all the time with it or never happens when poisoning uses
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003309byte zero, it clearly means you've found a bug and you definitely need to
3310report it. Otherwise if there's no clear change, the problem it is not related.
3311
3312When debugging some latency issues, it is important to use both strace and
3313tcpdump on the local machine, and another tcpdump on the remote system. The
3314reason for this is that there are delays everywhere in the processing chain and
3315it is important to know which one is causing latency to know where to act. In
3316practice, the local tcpdump will indicate when the input data come in. Strace
3317will indicate when haproxy receives these data (using recv/recvfrom). Warning,
3318openssl uses read()/write() syscalls instead of recv()/send(). Strace will also
3319show when haproxy sends the data, and tcpdump will show when the system sends
3320these data to the interface. Then the external tcpdump will show when the data
3321sent are really received (since the local one only shows when the packets are
3322queued). The benefit of sniffing on the local system is that strace and tcpdump
3323will use the same reference clock. Strace should be used with "-tts200" to get
3324complete timestamps and report large enough chunks of data to read them.
3325Tcpdump should be used with "-nvvttSs0" to report full packets, real sequence
3326numbers and complete timestamps.
3327
3328In practice, received data are almost always immediately received by haproxy
3329(unless the machine has a saturated CPU or these data are invalid and not
3330delivered). If these data are received but not sent, it generally is because
3331the output buffer is saturated (ie: recipient doesn't consume the data fast
3332enough). This can be confirmed by seeing that the polling doesn't notify of
3333the ability to write on the output file descriptor for some time (it's often
3334easier to spot in the strace output when the data finally leave and then roll
3335back to see when the write event was notified). It generally matches an ACK
3336received from the recipient, and detected by tcpdump. Once the data are sent,
3337they may spend some time in the system doing nothing. Here again, the TCP
3338congestion window may be limited and not allow these data to leave, waiting for
3339an ACK to open the window. If the traffic is idle and the data take 40 ms or
3340200 ms to leave, it's a different issue (which is not an issue), it's the fact
3341that the Nagle algorithm prevents empty packets from leaving immediately, in
3342hope that they will be merged with subsequent data. HAProxy automatically
3343disables Nagle in pure TCP mode and in tunnels. However it definitely remains
3344enabled when forwarding an HTTP body (and this contributes to the performance
3345improvement there by reducing the number of packets). Some HTTP non-compliant
3346applications may be sensitive to the latency when delivering incomplete HTTP
3347response messages. In this case you will have to enable "option http-no-delay"
3348to disable Nagle in order to work around their design, keeping in mind that any
3349other proxy in the chain may similarly be impacted. If tcpdump reports that data
3350leave immediately but the other end doesn't see them quickly, it can mean there
Dan Lloyd8e48b872016-07-01 21:01:18 -04003351is a congested WAN link, a congested LAN with flow control enabled and
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003352preventing the data from leaving, or more commonly that HAProxy is in fact
3353running in a virtual machine and that for whatever reason the hypervisor has
3354decided that the data didn't need to be sent immediately. In virtualized
3355environments, latency issues are almost always caused by the virtualization
3356layer, so in order to save time, it's worth first comparing tcpdump in the VM
3357and on the external components. Any difference has to be credited to the
3358hypervisor and its accompanying drivers.
3359
3360When some TCP SACK segments are seen in tcpdump traces (using -vv), it always
3361means that the side sending them has got the proof of a lost packet. While not
3362seeing them doesn't mean there are no losses, seeing them definitely means the
3363network is lossy. Losses are normal on a network, but at a rate where SACKs are
3364not noticeable at the naked eye. If they appear a lot in the traces, it is
3365worth investigating exactly what happens and where the packets are lost. HTTP
3366doesn't cope well with TCP losses, which introduce huge latencies.
3367
3368The "netstat -i" command will report statistics per interface. An interface
3369where the Rx-Ovr counter grows indicates that the system doesn't have enough
3370resources to receive all incoming packets and that they're lost before being
3371processed by the network driver. Rx-Drp indicates that some received packets
3372were lost in the network stack because the application doesn't process them
3373fast enough. This can happen during some attacks as well. Tx-Drp means that
3374the output queues were full and packets had to be dropped. When using TCP it
Dan Lloyd8e48b872016-07-01 21:01:18 -04003375should be very rare, but will possibly indicate a saturated outgoing link.
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003376
3377
337813. Security considerations
3379---------------------------
3380
3381HAProxy is designed to run with very limited privileges. The standard way to
3382use it is to isolate it into a chroot jail and to drop its privileges to a
3383non-root user without any permissions inside this jail so that if any future
3384vulnerability were to be discovered, its compromise would not affect the rest
3385of the system.
3386
Dan Lloyd8e48b872016-07-01 21:01:18 -04003387In order to perform a chroot, it first needs to be started as a root user. It is
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003388pointless to build hand-made chroots to start the process there, these ones are
3389painful to build, are never properly maintained and always contain way more
3390bugs than the main file-system. And in case of compromise, the intruder can use
3391the purposely built file-system. Unfortunately many administrators confuse
3392"start as root" and "run as root", resulting in the uid change to be done prior
3393to starting haproxy, and reducing the effective security restrictions.
3394
3395HAProxy will need to be started as root in order to :
3396 - adjust the file descriptor limits
3397 - bind to privileged port numbers
3398 - bind to a specific network interface
3399 - transparently listen to a foreign address
3400 - isolate itself inside the chroot jail
3401 - drop to another non-privileged UID
3402
3403HAProxy may require to be run as root in order to :
3404 - bind to an interface for outgoing connections
3405 - bind to privileged source ports for outgoing connections
Dan Lloyd8e48b872016-07-01 21:01:18 -04003406 - transparently bind to a foreign address for outgoing connections
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003407
3408Most users will never need the "run as root" case. But the "start as root"
3409covers most usages.
3410
3411A safe configuration will have :
3412
3413 - a chroot statement pointing to an empty location without any access
3414 permissions. This can be prepared this way on the UNIX command line :
3415
3416 # mkdir /var/empty && chmod 0 /var/empty || echo "Failed"
3417
3418 and referenced like this in the HAProxy configuration's global section :
3419
3420 chroot /var/empty
3421
3422 - both a uid/user and gid/group statements in the global section :
3423
3424 user haproxy
3425 group haproxy
3426
3427 - a stats socket whose mode, uid and gid are set to match the user and/or
3428 group allowed to access the CLI so that nobody may access it :
3429
3430 stats socket /var/run/haproxy.stat uid hatop gid hatop mode 600
3431