blob: 147e059f6db198f61fd8a0d493abb13ad97337e0 [file] [log] [blame]
Willy Tarreau2212e6a2015-10-13 14:40:55 +02001 ------------------------
2 HAProxy Management Guide
3 ------------------------
Willy Tarreau73dec762021-11-23 15:50:11 +01004 version 2.6
Willy Tarreau2212e6a2015-10-13 14:40:55 +02005
6
7This document describes how to start, stop, manage, and troubleshoot HAProxy,
8as well as some known limitations and traps to avoid. It does not describe how
9to configure it (for this please read configuration.txt).
10
11Note to documentation contributors :
12 This document is formatted with 80 columns per line, with even number of
13 spaces for indentation and without tabs. Please follow these rules strictly
14 so that it remains easily printable everywhere. If you add sections, please
15 update the summary below for easier searching.
16
17
18Summary
19-------
20
211. Prerequisites
222. Quick reminder about HAProxy's architecture
233. Starting HAProxy
244. Stopping and restarting HAProxy
255. File-descriptor limitations
266. Memory management
277. CPU usage
288. Logging
299. Statistics and monitoring
Willy Tarreau44aed902015-10-13 14:45:29 +0200309.1. CSV format
Willy Tarreau5d8b9792016-03-11 11:09:34 +0100319.2. Typed output format
329.3. Unix Socket commands
William Lallemand142db372018-12-11 18:56:45 +0100339.4. Master CLI
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003410. Tricks for easier configuration management
3511. Well-known traps to avoid
3612. Debugging and performance issues
3713. Security considerations
38
39
401. Prerequisites
41----------------
42
43In this document it is assumed that the reader has sufficient administration
44skills on a UNIX-like operating system, uses the shell on a daily basis and is
45familiar with troubleshooting utilities such as strace and tcpdump.
46
47
482. Quick reminder about HAProxy's architecture
49----------------------------------------------
50
Willy Tarreau3f364482019-02-27 15:01:46 +010051HAProxy is a multi-threaded, event-driven, non-blocking daemon. This means is
Willy Tarreau2212e6a2015-10-13 14:40:55 +020052uses event multiplexing to schedule all of its activities instead of relying on
53the system to schedule between multiple activities. Most of the time it runs as
54a single process, so the output of "ps aux" on a system will report only one
55"haproxy" process, unless a soft reload is in progress and an older process is
56finishing its job in parallel to the new one. It is thus always easy to trace
Willy Tarreau3f364482019-02-27 15:01:46 +010057its activity using the strace utility. In order to scale with the number of
58available processors, by default haproxy will start one worker thread per
59processor it is allowed to run on. Unless explicitly configured differently,
60the incoming traffic is spread over all these threads, all running the same
61event loop. A great care is taken to limit inter-thread dependencies to the
62strict minimum, so as to try to achieve near-linear scalability. This has some
63impacts such as the fact that a given connection is served by a single thread.
64Thus in order to use all available processing capacity, it is needed to have at
65least as many connections as there are threads, which is almost always granted.
Willy Tarreau2212e6a2015-10-13 14:40:55 +020066
67HAProxy is designed to isolate itself into a chroot jail during startup, where
68it cannot perform any file-system access at all. This is also true for the
69libraries it depends on (eg: libc, libssl, etc). The immediate effect is that
70a running process will not be able to reload a configuration file to apply
71changes, instead a new process will be started using the updated configuration
72file. Some other less obvious effects are that some timezone files or resolver
73files the libc might attempt to access at run time will not be found, though
74this should generally not happen as they're not needed after startup. A nice
75consequence of this principle is that the HAProxy process is totally stateless,
76and no cleanup is needed after it's killed, so any killing method that works
77will do the right thing.
78
79HAProxy doesn't write log files, but it relies on the standard syslog protocol
80to send logs to a remote server (which is often located on the same system).
81
82HAProxy uses its internal clock to enforce timeouts, that is derived from the
83system's time but where unexpected drift is corrected. This is done by limiting
84the time spent waiting in poll() for an event, and measuring the time it really
85took. In practice it never waits more than one second. This explains why, when
86running strace over a completely idle process, periodic calls to poll() (or any
87of its variants) surrounded by two gettimeofday() calls are noticed. They are
88normal, completely harmless and so cheap that the load they imply is totally
89undetectable at the system scale, so there's nothing abnormal there. Example :
90
91 16:35:40.002320 gettimeofday({1442759740, 2605}, NULL) = 0
92 16:35:40.002942 epoll_wait(0, {}, 200, 1000) = 0
93 16:35:41.007542 gettimeofday({1442759741, 7641}, NULL) = 0
94 16:35:41.007998 gettimeofday({1442759741, 8114}, NULL) = 0
95 16:35:41.008391 epoll_wait(0, {}, 200, 1000) = 0
96 16:35:42.011313 gettimeofday({1442759742, 11411}, NULL) = 0
97
98HAProxy is a TCP proxy, not a router. It deals with established connections that
99have been validated by the kernel, and not with packets of any form nor with
100sockets in other states (eg: no SYN_RECV nor TIME_WAIT), though their existence
101may prevent it from binding a port. It relies on the system to accept incoming
102connections and to initiate outgoing connections. An immediate effect of this is
103that there is no relation between packets observed on the two sides of a
104forwarded connection, which can be of different size, numbers and even family.
105Since a connection may only be accepted from a socket in LISTEN state, all the
106sockets it is listening to are necessarily visible using the "netstat" utility
107to show listening sockets. Example :
108
109 # netstat -ltnp
110 Active Internet connections (only servers)
111 Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
112 tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1629/sshd
113 tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2847/haproxy
114 tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 2847/haproxy
115
116
1173. Starting HAProxy
118-------------------
119
120HAProxy is started by invoking the "haproxy" program with a number of arguments
121passed on the command line. The actual syntax is :
122
123 $ haproxy [<options>]*
124
125where [<options>]* is any number of options. An option always starts with '-'
126followed by one of more letters, and possibly followed by one or multiple extra
127arguments. Without any option, HAProxy displays the help page with a reminder
128about supported options. Available options may vary slightly based on the
129operating system. A fair number of these options overlap with an equivalent one
130if the "global" section. In this case, the command line always has precedence
131over the configuration file, so that the command line can be used to quickly
132enforce some settings without touching the configuration files. The current
133list of options is :
134
135 -- <cfgfile>* : all the arguments following "--" are paths to configuration
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200136 file/directory to be loaded and processed in the declaration order. It is
137 mostly useful when relying on the shell to load many files that are
138 numerically ordered. See also "-f". The difference between "--" and "-f" is
139 that one "-f" must be placed before each file name, while a single "--" is
140 needed before all file names. Both options can be used together, the
141 command line ordering still applies. When more than one file is specified,
142 each file must start on a section boundary, so the first keyword of each
143 file must be one of "global", "defaults", "peers", "listen", "frontend",
144 "backend", and so on. A file cannot contain just a server list for example.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200145
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200146 -f <cfgfile|cfgdir> : adds <cfgfile> to the list of configuration files to be
147 loaded. If <cfgdir> is a directory, all the files (and only files) it
Dan Lloyd8e48b872016-07-01 21:01:18 -0400148 contains are added in lexical order (using LC_COLLATE=C) to the list of
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200149 configuration files to be loaded ; only files with ".cfg" extension are
150 added, only non hidden files (not prefixed with ".") are added.
151 Configuration files are loaded and processed in their declaration order.
152 This option may be specified multiple times to load multiple files. See
153 also "--". The difference between "--" and "-f" is that one "-f" must be
154 placed before each file name, while a single "--" is needed before all file
155 names. Both options can be used together, the command line ordering still
156 applies. When more than one file is specified, each file must start on a
157 section boundary, so the first keyword of each file must be one of
158 "global", "defaults", "peers", "listen", "frontend", "backend", and so on.
159 A file cannot contain just a server list for example.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200160
161 -C <dir> : changes to directory <dir> before loading configuration
162 files. This is useful when using relative paths. Warning when using
163 wildcards after "--" which are in fact replaced by the shell before
164 starting haproxy.
165
166 -D : start as a daemon. The process detaches from the current terminal after
167 forking, and errors are not reported anymore in the terminal. It is
168 equivalent to the "daemon" keyword in the "global" section of the
169 configuration. It is recommended to always force it in any init script so
170 that a faulty configuration doesn't prevent the system from booting.
171
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200172 -L <name> : change the local peer name to <name>, which defaults to the local
William Lallemanddaf4cd22018-04-17 16:46:13 +0200173 hostname. This is used only with peers replication. You can use the
174 variable $HAPROXY_LOCALPEER in the configuration file to reference the
175 peer name.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200176
177 -N <limit> : sets the default per-proxy maxconn to <limit> instead of the
178 builtin default value (usually 2000). Only useful for debugging.
179
180 -V : enable verbose mode (disables quiet mode). Reverts the effect of "-q" or
181 "quiet".
182
William Lallemande202b1e2017-06-01 17:38:56 +0200183 -W : master-worker mode. It is equivalent to the "master-worker" keyword in
184 the "global" section of the configuration. This mode will launch a "master"
185 which will monitor the "workers". Using this mode, you can reload HAProxy
186 directly by sending a SIGUSR2 signal to the master. The master-worker mode
187 is compatible either with the foreground or daemon mode. It is
188 recommended to use this mode with multiprocess and systemd.
189
Pavlos Parissisf65f2572018-02-07 21:42:16 +0100190 -Ws : master-worker mode with support of `notify` type of systemd service.
191 This option is only available when HAProxy was built with `USE_SYSTEMD`
192 build option enabled.
193
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200194 -c : only performs a check of the configuration files and exits before trying
195 to bind. The exit status is zero if everything is OK, or non-zero if an
Willy Tarreaubebd2122020-04-15 16:06:11 +0200196 error is encountered. Presence of warnings will be reported if any.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200197
Maximilian Maderfc0cceb2021-06-06 00:50:22 +0200198 -cc : evaluates a condition as used within a conditional block of the
199 configuration. The exit status is zero if the condition is true, 1 if the
200 condition is false or 2 if an error is encountered.
201
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200202 -d : enable debug mode. This disables daemon mode, forces the process to stay
Willy Tarreauccf42992020-10-09 19:15:03 +0200203 in foreground and to show incoming and outgoing events. It must never be
204 used in an init script.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200205
Amaury Denoyelle7b01a8d2021-03-29 10:29:07 +0200206 -dD : enable diagnostic mode. This mode will output extra warnings about
207 suspicious configuration statements. This will never prevent startup even in
208 "zero-warning" mode nor change the exit status code.
209
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200210 -dG : disable use of getaddrinfo() to resolve host names into addresses. It
211 can be used when suspecting that getaddrinfo() doesn't work as expected.
212 This option was made available because many bogus implementations of
213 getaddrinfo() exist on various systems and cause anomalies that are
214 difficult to troubleshoot.
215
Willy Tarreau654726d2021-12-28 15:43:11 +0100216 -dL : dumps the list of dynamic shared libraries that are loaded at the end
217 of the config processing. This will generally also include deep dependencies
218 such as anything loaded from Lua code for example, as well as the executable
219 itself. The list is printed in a format that ought to be easy enough to
220 sanitize to directly produce a tarball of all dependencies. Since it doesn't
221 stop the program's startup, it is recommended to only use it in combination
222 with "-c" and "-q" where only the list of loaded objects will be displayed
223 (or nothing in case of error). In addition, keep in mind that when providing
224 such a package to help with a core file analysis, most libraries are in fact
225 symbolic links that need to be dereferenced when creating the archive:
226
227 ./haproxy -W -q -c -dL -f foo.cfg | tar -T - -hzcf archive.tgz
228
Dan Lloyd8e48b872016-07-01 21:01:18 -0400229 -dM[<byte>] : forces memory poisoning, which means that each and every
Willy Tarreaubafbe012017-11-24 17:34:44 +0100230 memory region allocated with malloc() or pool_alloc() will be filled with
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200231 <byte> before being passed to the caller. When <byte> is not specified, it
232 defaults to 0x50 ('P'). While this slightly slows down operations, it is
233 useful to reliably trigger issues resulting from missing initializations in
234 the code that cause random crashes. Note that -dM0 has the effect of
235 turning any malloc() into a calloc(). In any case if a bug appears or
236 disappears when using this option it means there is a bug in haproxy, so
237 please report it.
238
239 -dS : disable use of the splice() system call. It is equivalent to the
240 "global" section's "nosplice" keyword. This may be used when splice() is
241 suspected to behave improperly or to cause performance issues, or when
242 using strace to see the forwarded data (which do not appear when using
243 splice()).
244
245 -dV : disable SSL verify on the server side. It is equivalent to having
246 "ssl-server-verify none" in the "global" section. This is useful when
247 trying to reproduce production issues out of the production
248 environment. Never use this in an init script as it degrades SSL security
249 to the servers.
250
Willy Tarreau3eb10b82020-04-15 16:42:39 +0200251 -dW : if set, haproxy will refuse to start if any warning was emitted while
252 processing the configuration. This helps detect subtle mistakes and keep the
253 configuration clean and portable across versions. It is recommended to set
254 this option in service scripts when configurations are managed by humans,
255 but it is recommended not to use it with generated configurations, which
256 tend to emit more warnings. It may be combined with "-c" to cause warnings
257 in checked configurations to fail. This is equivalent to global option
258 "zero-warning".
259
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200260 -db : disable background mode and multi-process mode. The process remains in
261 foreground. It is mainly used during development or during small tests, as
262 Ctrl-C is enough to stop the process. Never use it in an init script.
263
264 -de : disable the use of the "epoll" poller. It is equivalent to the "global"
265 section's keyword "noepoll". It is mostly useful when suspecting a bug
266 related to this poller. On systems supporting epoll, the fallback will
267 generally be the "poll" poller.
268
269 -dk : disable the use of the "kqueue" poller. It is equivalent to the
270 "global" section's keyword "nokqueue". It is mostly useful when suspecting
271 a bug related to this poller. On systems supporting kqueue, the fallback
272 will generally be the "poll" poller.
273
274 -dp : disable the use of the "poll" poller. It is equivalent to the "global"
275 section's keyword "nopoll". It is mostly useful when suspecting a bug
276 related to this poller. On systems supporting poll, the fallback will
277 generally be the "select" poller, which cannot be disabled and is limited
278 to 1024 file descriptors.
279
Willy Tarreau3eed10e2016-11-07 21:03:16 +0100280 -dr : ignore server address resolution failures. It is very common when
281 validating a configuration out of production not to have access to the same
282 resolvers and to fail on server address resolution, making it difficult to
283 test a configuration. This option simply appends the "none" method to the
284 list of address resolution methods for all servers, ensuring that even if
285 the libc fails to resolve an address, the startup sequence is not
286 interrupted.
287
Willy Tarreau70060452015-12-14 12:46:07 +0100288 -m <limit> : limit the total allocatable memory to <limit> megabytes across
289 all processes. This may cause some connection refusals or some slowdowns
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200290 depending on the amount of memory needed for normal operations. This is
Willy Tarreau70060452015-12-14 12:46:07 +0100291 mostly used to force the processes to work in a constrained resource usage
292 scenario. It is important to note that the memory is not shared between
293 processes, so in a multi-process scenario, this value is first divided by
294 global.nbproc before forking.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200295
296 -n <limit> : limits the per-process connection limit to <limit>. This is
297 equivalent to the global section's keyword "maxconn". It has precedence
298 over this keyword. This may be used to quickly force lower limits to avoid
299 a service outage on systems where resource limits are too low.
300
301 -p <file> : write all processes' pids into <file> during startup. This is
302 equivalent to the "global" section's keyword "pidfile". The file is opened
303 before entering the chroot jail, and after doing the chdir() implied by
304 "-C". Each pid appears on its own line.
305
306 -q : set "quiet" mode. This disables some messages during the configuration
307 parsing and during startup. It can be used in combination with "-c" to
308 just check if a configuration file is valid or not.
309
William Lallemand142db372018-12-11 18:56:45 +0100310 -S <bind>[,bind_options...]: in master-worker mode, bind a master CLI, which
311 allows the access to every processes, running or leaving ones.
312 For security reasons, it is recommended to bind the master CLI to a local
313 UNIX socket. The bind options are the same as the keyword "bind" in
314 the configuration file with words separated by commas instead of spaces.
315
316 Note that this socket can't be used to retrieve the listening sockets from
317 an old process during a seamless reload.
318
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200319 -sf <pid>* : send the "finish" signal (SIGUSR1) to older processes after boot
320 completion to ask them to finish what they are doing and to leave. <pid>
321 is a list of pids to signal (one per argument). The list ends on any
322 option starting with a "-". It is not a problem if the list of pids is
323 empty, so that it can be built on the fly based on the result of a command
324 like "pidof" or "pgrep".
325
326 -st <pid>* : send the "terminate" signal (SIGTERM) to older processes after
327 boot completion to terminate them immediately without finishing what they
328 were doing. <pid> is a list of pids to signal (one per argument). The list
329 is ends on any option starting with a "-". It is not a problem if the list
330 of pids is empty, so that it can be built on the fly based on the result of
331 a command like "pidof" or "pgrep".
332
333 -v : report the version and build date.
334
335 -vv : display the version, build options, libraries versions and usable
336 pollers. This output is systematically requested when filing a bug report.
337
Olivier Houchardd33fc3a2017-04-05 22:50:59 +0200338 -x <unix_socket> : connect to the specified socket and try to retrieve any
339 listening sockets from the old process, and use them instead of trying to
340 bind new ones. This is useful to avoid missing any new connection when
William Lallemandf6975e92017-05-26 17:42:10 +0200341 reloading the configuration on Linux. The capability must be enable on the
342 stats socket using "expose-fd listeners" in your configuration.
William Lallemand2be557f2021-11-24 18:45:37 +0100343 In master-worker mode, the master will use this option upon a reload with
344 the "sockpair@" syntax, which allows the master to connect directly to a
345 worker without using stats socket declared in the configuration.
Olivier Houchardd33fc3a2017-04-05 22:50:59 +0200346
Dan Lloyd8e48b872016-07-01 21:01:18 -0400347A safe way to start HAProxy from an init file consists in forcing the daemon
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200348mode, storing existing pids to a pid file and using this pid file to notify
349older processes to finish before leaving :
350
351 haproxy -f /etc/haproxy.cfg \
352 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
353
354When the configuration is split into a few specific files (eg: tcp vs http),
355it is recommended to use the "-f" option :
356
357 haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
358 -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
359 -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
360 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
361
362When an unknown number of files is expected, such as customer-specific files,
363it is recommended to assign them a name starting with a fixed-size sequence
364number and to use "--" to load them, possibly after loading some defaults :
365
366 haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
367 -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
368 -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
369 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) \
370 -f /etc/haproxy/default-customers.cfg -- /etc/haproxy/customers/*
371
372Sometimes a failure to start may happen for whatever reason. Then it is
373important to verify if the version of HAProxy you are invoking is the expected
374version and if it supports the features you are expecting (eg: SSL, PCRE,
375compression, Lua, etc). This can be verified using "haproxy -vv". Some
376important information such as certain build options, the target system and
377the versions of the libraries being used are reported there. It is also what
378you will systematically be asked for when posting a bug report :
379
380 $ haproxy -vv
Willy Tarreau58000fe2021-05-09 06:25:16 +0200381 HAProxy version 1.6-dev7-a088d3-4 2015/10/08
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200382 Copyright 2000-2015 Willy Tarreau <willy@haproxy.org>
383
384 Build options :
385 TARGET = linux2628
386 CPU = generic
387 CC = gcc
388 CFLAGS = -pg -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement \
389 -DBUFSIZE=8030 -DMAXREWRITE=1030 -DSO_MARK=36 -DTCP_REPAIR=19
390 OPTIONS = USE_ZLIB=1 USE_DLMALLOC=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
391
392 Default settings :
393 maxconn = 2000, bufsize = 8030, maxrewrite = 1030, maxpollevents = 200
394
395 Encrypted password support via crypt(3): yes
396 Built with zlib version : 1.2.6
397 Compression algorithms supported : identity("identity"), deflate("deflate"), \
398 raw-deflate("deflate"), gzip("gzip")
399 Built with OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
400 Running on OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
401 OpenSSL library supports TLS extensions : yes
402 OpenSSL library supports SNI : yes
403 OpenSSL library supports prefer-server-ciphers : yes
404 Built with PCRE version : 8.12 2011-01-15
405 PCRE library supports JIT : no (USE_PCRE_JIT not set)
406 Built with Lua version : Lua 5.3.1
407 Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND
408
409 Available polling systems :
410 epoll : pref=300, test result OK
411 poll : pref=200, test result OK
412 select : pref=150, test result OK
413 Total: 3 (3 usable), will use epoll.
414
415The relevant information that many non-developer users can verify here are :
416 - the version : 1.6-dev7-a088d3-4 above means the code is currently at commit
417 ID "a088d3" which is the 4th one after after official version "1.6-dev7".
418 Version 1.6-dev7 would show as "1.6-dev7-8c1ad7". What matters here is in
419 fact "1.6-dev7". This is the 7th development version of what will become
420 version 1.6 in the future. A development version not suitable for use in
421 production (unless you know exactly what you are doing). A stable version
422 will show as a 3-numbers version, such as "1.5.14-16f863", indicating the
423 14th level of fix on top of version 1.5. This is a production-ready version.
424
425 - the release date : 2015/10/08. It is represented in the universal
426 year/month/day format. Here this means August 8th, 2015. Given that stable
427 releases are issued every few months (1-2 months at the beginning, sometimes
428 6 months once the product becomes very stable), if you're seeing an old date
429 here, it means you're probably affected by a number of bugs or security
430 issues that have since been fixed and that it might be worth checking on the
431 official site.
432
433 - build options : they are relevant to people who build their packages
434 themselves, they can explain why things are not behaving as expected. For
435 example the development version above was built for Linux 2.6.28 or later,
Dan Lloyd8e48b872016-07-01 21:01:18 -0400436 targeting a generic CPU (no CPU-specific optimizations), and lacks any
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200437 code optimization (-O0) so it will perform poorly in terms of performance.
438
439 - libraries versions : zlib version is reported as found in the library
440 itself. In general zlib is considered a very stable product and upgrades
441 are almost never needed. OpenSSL reports two versions, the version used at
442 build time and the one being used, as found on the system. These ones may
443 differ by the last letter but never by the numbers. The build date is also
444 reported because most OpenSSL bugs are security issues and need to be taken
445 seriously, so this library absolutely needs to be kept up to date. Seeing a
446 4-months old version here is highly suspicious and indeed an update was
447 missed. PCRE provides very fast regular expressions and is highly
448 recommended. Certain of its extensions such as JIT are not present in all
449 versions and still young so some people prefer not to build with them,
Dan Lloyd8e48b872016-07-01 21:01:18 -0400450 which is why the build status is reported as well. Regarding the Lua
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200451 scripting language, HAProxy expects version 5.3 which is very young since
452 it was released a little time before HAProxy 1.6. It is important to check
453 on the Lua web site if some fixes are proposed for this branch.
454
455 - Available polling systems will affect the process's scalability when
456 dealing with more than about one thousand of concurrent connections. These
457 ones are only available when the correct system was indicated in the TARGET
458 variable during the build. The "epoll" mechanism is highly recommended on
459 Linux, and the kqueue mechanism is highly recommended on BSD. Lacking them
460 will result in poll() or even select() being used, causing a high CPU usage
461 when dealing with a lot of connections.
462
463
4644. Stopping and restarting HAProxy
465----------------------------------
466
467HAProxy supports a graceful and a hard stop. The hard stop is simple, when the
468SIGTERM signal is sent to the haproxy process, it immediately quits and all
469established connections are closed. The graceful stop is triggered when the
470SIGUSR1 signal is sent to the haproxy process. It consists in only unbinding
471from listening ports, but continue to process existing connections until they
472close. Once the last connection is closed, the process leaves.
473
474The hard stop method is used for the "stop" or "restart" actions of the service
475management script. The graceful stop is used for the "reload" action which
476tries to seamlessly reload a new configuration in a new process.
477
478Both of these signals may be sent by the new haproxy process itself during a
479reload or restart, so that they are sent at the latest possible moment and only
480if absolutely required. This is what is performed by the "-st" (hard) and "-sf"
481(graceful) options respectively.
482
William Lallemande202b1e2017-06-01 17:38:56 +0200483In master-worker mode, it is not needed to start a new haproxy process in
484order to reload the configuration. The master process reacts to the SIGUSR2
485signal by reexecuting itself with the -sf parameter followed by the PIDs of
486the workers. The master will then parse the configuration file and fork new
487workers.
488
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200489To understand better how these signals are used, it is important to understand
490the whole restart mechanism.
491
492First, an existing haproxy process is running. The administrator uses a system
Jackie Tapia749f74c2020-07-22 18:59:40 -0500493specific command such as "/etc/init.d/haproxy reload" to indicate they want to
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200494take the new configuration file into effect. What happens then is the following.
495First, the service script (/etc/init.d/haproxy or equivalent) will verify that
496the configuration file parses correctly using "haproxy -c". After that it will
497try to start haproxy with this configuration file, using "-st" or "-sf".
498
499Then HAProxy tries to bind to all listening ports. If some fatal errors happen
500(eg: address not present on the system, permission denied), the process quits
501with an error. If a socket binding fails because a port is already in use, then
502the process will first send a SIGTTOU signal to all the pids specified in the
503"-st" or "-sf" pid list. This is what is called the "pause" signal. It instructs
504all existing haproxy processes to temporarily stop listening to their ports so
505that the new process can try to bind again. During this time, the old process
506continues to process existing connections. If the binding still fails (because
507for example a port is shared with another daemon), then the new process sends a
508SIGTTIN signal to the old processes to instruct them to resume operations just
509as if nothing happened. The old processes will then restart listening to the
Jonathon Lacherc5b5e7b2021-08-04 00:29:05 -0500510ports and continue to accept connections. Note that this mechanism is system
Dan Lloyd8e48b872016-07-01 21:01:18 -0400511dependent and some operating systems may not support it in multi-process mode.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200512
513If the new process manages to bind correctly to all ports, then it sends either
514the SIGTERM (hard stop in case of "-st") or the SIGUSR1 (graceful stop in case
515of "-sf") to all processes to notify them that it is now in charge of operations
516and that the old processes will have to leave, either immediately or once they
517have finished their job.
518
519It is important to note that during this timeframe, there are two small windows
520of a few milliseconds each where it is possible that a few connection failures
521will be noticed during high loads. Typically observed failure rates are around
5221 failure during a reload operation every 10000 new connections per second,
523which means that a heavily loaded site running at 30000 new connections per
524second may see about 3 failed connection upon every reload. The two situations
525where this happens are :
526
527 - if the new process fails to bind due to the presence of the old process,
528 it will first have to go through the SIGTTOU+SIGTTIN sequence, which
529 typically lasts about one millisecond for a few tens of frontends, and
530 during which some ports will not be bound to the old process and not yet
531 bound to the new one. HAProxy works around this on systems that support the
532 SO_REUSEPORT socket options, as it allows the new process to bind without
533 first asking the old one to unbind. Most BSD systems have been supporting
534 this almost forever. Linux has been supporting this in version 2.0 and
535 dropped it around 2.2, but some patches were floating around by then. It
536 was reintroduced in kernel 3.9, so if you are observing a connection
Dan Lloyd8e48b872016-07-01 21:01:18 -0400537 failure rate above the one mentioned above, please ensure that your kernel
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200538 is 3.9 or newer, or that relevant patches were backported to your kernel
539 (less likely).
540
541 - when the old processes close the listening ports, the kernel may not always
542 redistribute any pending connection that was remaining in the socket's
543 backlog. Under high loads, a SYN packet may happen just before the socket
544 is closed, and will lead to an RST packet being sent to the client. In some
545 critical environments where even one drop is not acceptable, these ones are
546 sometimes dealt with using firewall rules to block SYN packets during the
547 reload, forcing the client to retransmit. This is totally system-dependent,
548 as some systems might be able to visit other listening queues and avoid
549 this RST. A second case concerns the ACK from the client on a local socket
550 that was in SYN_RECV state just before the close. This ACK will lead to an
551 RST packet while the haproxy process is still not aware of it. This one is
Dan Lloyd8e48b872016-07-01 21:01:18 -0400552 harder to get rid of, though the firewall filtering rules mentioned above
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200553 will work well if applied one second or so before restarting the process.
554
555For the vast majority of users, such drops will never ever happen since they
556don't have enough load to trigger the race conditions. And for most high traffic
557users, the failure rate is still fairly within the noise margin provided that at
558least SO_REUSEPORT is properly supported on their systems.
559
560
5615. File-descriptor limitations
562------------------------------
563
564In order to ensure that all incoming connections will successfully be served,
565HAProxy computes at load time the total number of file descriptors that will be
566needed during the process's life. A regular Unix process is generally granted
5671024 file descriptors by default, and a privileged process can raise this limit
568itself. This is one reason for starting HAProxy as root and letting it adjust
569the limit. The default limit of 1024 file descriptors roughly allow about 500
570concurrent connections to be processed. The computation is based on the global
571maxconn parameter which limits the total number of connections per process, the
572number of listeners, the number of servers which have a health check enabled,
573the agent checks, the peers, the loggers and possibly a few other technical
574requirements. A simple rough estimate of this number consists in simply
575doubling the maxconn value and adding a few tens to get the approximate number
576of file descriptors needed.
577
578Originally HAProxy did not know how to compute this value, and it was necessary
579to pass the value using the "ulimit-n" setting in the global section. This
580explains why even today a lot of configurations are seen with this setting
581present. Unfortunately it was often miscalculated resulting in connection
582failures when approaching maxconn instead of throttling incoming connection
583while waiting for the needed resources. For this reason it is important to
Dan Lloyd8e48b872016-07-01 21:01:18 -0400584remove any vestigial "ulimit-n" setting that can remain from very old versions.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200585
586Raising the number of file descriptors to accept even moderate loads is
587mandatory but comes with some OS-specific adjustments. First, the select()
588polling system is limited to 1024 file descriptors. In fact on Linux it used
589to be capable of handling more but since certain OS ship with excessively
590restrictive SELinux policies forbidding the use of select() with more than
5911024 file descriptors, HAProxy now refuses to start in this case in order to
592avoid any issue at run time. On all supported operating systems, poll() is
593available and will not suffer from this limitation. It is automatically picked
Dan Lloyd8e48b872016-07-01 21:01:18 -0400594so there is nothing to do to get a working configuration. But poll's becomes
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200595very slow when the number of file descriptors increases. While HAProxy does its
596best to limit this performance impact (eg: via the use of the internal file
597descriptor cache and batched processing), a good rule of thumb is that using
598poll() with more than a thousand concurrent connections will use a lot of CPU.
599
600For Linux systems base on kernels 2.6 and above, the epoll() system call will
601be used. It's a much more scalable mechanism relying on callbacks in the kernel
602that guarantee a constant wake up time regardless of the number of registered
603monitored file descriptors. It is automatically used where detected, provided
604that HAProxy had been built for one of the Linux flavors. Its presence and
605support can be verified using "haproxy -vv".
606
607For BSD systems which support it, kqueue() is available as an alternative. It
608is much faster than poll() and even slightly faster than epoll() thanks to its
609batched handling of changes. At least FreeBSD and OpenBSD support it. Just like
610with Linux's epoll(), its support and availability are reported in the output
611of "haproxy -vv".
612
613Having a good poller is one thing, but it is mandatory that the process can
614reach the limits. When HAProxy starts, it immediately sets the new process's
615file descriptor limits and verifies if it succeeds. In case of failure, it
616reports it before forking so that the administrator can see the problem. As
617long as the process is started by as root, there should be no reason for this
618setting to fail. However, it can fail if the process is started by an
619unprivileged user. If there is a compelling reason for *not* starting haproxy
620as root (eg: started by end users, or by a per-application account), then the
621file descriptor limit can be raised by the system administrator for this
622specific user. The effectiveness of the setting can be verified by issuing
623"ulimit -n" from the user's command line. It should reflect the new limit.
624
625Warning: when an unprivileged user's limits are changed in this user's account,
626it is fairly common that these values are only considered when the user logs in
627and not at all in some scripts run at system boot time nor in crontabs. This is
628totally dependent on the operating system, keep in mind to check "ulimit -n"
629before starting haproxy when running this way. The general advice is never to
630start haproxy as an unprivileged user for production purposes. Another good
631reason is that it prevents haproxy from enabling some security protections.
632
633Once it is certain that the system will allow the haproxy process to use the
634requested number of file descriptors, two new system-specific limits may be
635encountered. The first one is the system-wide file descriptor limit, which is
636the total number of file descriptors opened on the system, covering all
637processes. When this limit is reached, accept() or socket() will typically
638return ENFILE. The second one is the per-process hard limit on the number of
639file descriptors, it prevents setrlimit() from being set higher. Both are very
640dependent on the operating system. On Linux, the system limit is set at boot
641based on the amount of memory. It can be changed with the "fs.file-max" sysctl.
642And the per-process hard limit is set to 1048576 by default, but it can be
643changed using the "fs.nr_open" sysctl.
644
645File descriptor limitations may be observed on a running process when they are
646set too low. The strace utility will report that accept() and socket() return
647"-1 EMFILE" when the process's limits have been reached. In this case, simply
648raising the "ulimit-n" value (or removing it) will solve the problem. If these
649system calls return "-1 ENFILE" then it means that the kernel's limits have
650been reached and that something must be done on a system-wide parameter. These
651trouble must absolutely be addressed, as they result in high CPU usage (when
652accept() fails) and failed connections that are generally visible to the user.
653One solution also consists in lowering the global maxconn value to enforce
654serialization, and possibly to disable HTTP keep-alive to force connections
655to be released and reused faster.
656
657
6586. Memory management
659--------------------
660
661HAProxy uses a simple and fast pool-based memory management. Since it relies on
662a small number of different object types, it's much more efficient to pick new
663objects from a pool which already contains objects of the appropriate size than
664to call malloc() for each different size. The pools are organized as a stack or
665LIFO, so that newly allocated objects are taken from recently released objects
666still hot in the CPU caches. Pools of similar sizes are merged together, in
667order to limit memory fragmentation.
668
669By default, since the focus is set on performance, each released object is put
670back into the pool it came from, and allocated objects are never freed since
671they are expected to be reused very soon.
672
673On the CLI, it is possible to check how memory is being used in pools thanks to
674the "show pools" command :
675
676 > show pools
677 Dumping pools usage. Use SIGQUIT to flush them.
Willy Tarreau0a93b642018-10-16 07:58:39 +0200678 - Pool cache_st (16 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccc40=03 [SHARED]
679 - Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 0 failures, 2 users, @0x9ccac0=00 [SHARED]
680 - Pool comp_state (48 bytes) : 3 allocated (144 bytes), 3 used, 0 failures, 5 users, @0x9cccc0=04 [SHARED]
681 - Pool filter (64 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 3 users, @0x9ccbc0=02 [SHARED]
682 - Pool vars (80 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccb40=01 [SHARED]
683 - Pool uniqueid (128 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9cd240=15 [SHARED]
684 - Pool task (144 bytes) : 55 allocated (7920 bytes), 55 used, 0 failures, 1 users, @0x9cd040=11 [SHARED]
685 - Pool session (160 bytes) : 1 allocated (160 bytes), 1 used, 0 failures, 1 users, @0x9cd140=13 [SHARED]
686 - Pool h2s (208 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccec0=08 [SHARED]
687 - Pool h2c (288 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cce40=07 [SHARED]
688 - Pool spoe_ctx (304 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccf40=09 [SHARED]
689 - Pool connection (400 bytes) : 2 allocated (800 bytes), 2 used, 0 failures, 1 users, @0x9cd1c0=14 [SHARED]
690 - Pool hdr_idx (416 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cd340=17 [SHARED]
691 - Pool dns_resolut (480 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccdc0=06 [SHARED]
692 - Pool dns_answer_ (576 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccd40=05 [SHARED]
693 - Pool stream (960 bytes) : 1 allocated (960 bytes), 1 used, 0 failures, 1 users, @0x9cd0c0=12 [SHARED]
694 - Pool requri (1024 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cd2c0=16 [SHARED]
695 - Pool buffer (8030 bytes) : 3 allocated (24090 bytes), 2 used, 0 failures, 1 users, @0x9cd3c0=18 [SHARED]
696 - Pool trash (8062 bytes) : 1 allocated (8062 bytes), 1 used, 0 failures, 1 users, @0x9cd440=19
697 Total: 19 pools, 42296 bytes allocated, 34266 used.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200698
699The pool name is only indicative, it's the name of the first object type using
700this pool. The size in parenthesis is the object size for objects in this pool.
701Object sizes are always rounded up to the closest multiple of 16 bytes. The
702number of objects currently allocated and the equivalent number of bytes is
703reported so that it is easy to know which pool is responsible for the highest
704memory usage. The number of objects currently in use is reported as well in the
705"used" field. The difference between "allocated" and "used" corresponds to the
Willy Tarreau0a93b642018-10-16 07:58:39 +0200706objects that have been freed and are available for immediate use. The address
707at the end of the line is the pool's address, and the following number is the
708pool index when it exists, or is reported as -1 if no index was assigned.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200709
710It is possible to limit the amount of memory allocated per process using the
711"-m" command line option, followed by a number of megabytes. It covers all of
712the process's addressable space, so that includes memory used by some libraries
713as well as the stack, but it is a reliable limit when building a resource
714constrained system. It works the same way as "ulimit -v" on systems which have
715it, or "ulimit -d" for the other ones.
716
717If a memory allocation fails due to the memory limit being reached or because
718the system doesn't have any enough memory, then haproxy will first start to
719free all available objects from all pools before attempting to allocate memory
720again. This mechanism of releasing unused memory can be triggered by sending
721the signal SIGQUIT to the haproxy process. When doing so, the pools state prior
722to the flush will also be reported to stderr when the process runs in
723foreground.
724
725During a reload operation, the process switched to the graceful stop state also
726automatically performs some flushes after releasing any connection so that all
727possible memory is released to save it for the new process.
728
729
7307. CPU usage
731------------
732
733HAProxy normally spends most of its time in the system and a smaller part in
734userland. A finely tuned 3.5 GHz CPU can sustain a rate about 80000 end-to-end
735connection setups and closes per second at 100% CPU on a single core. When one
736core is saturated, typical figures are :
737 - 95% system, 5% user for long TCP connections or large HTTP objects
738 - 85% system and 15% user for short TCP connections or small HTTP objects in
739 close mode
740 - 70% system and 30% user for small HTTP objects in keep-alive mode
741
742The amount of rules processing and regular expressions will increase the user
743land part. The presence of firewall rules, connection tracking, complex routing
744tables in the system will instead increase the system part.
745
746On most systems, the CPU time observed during network transfers can be cut in 4
747parts :
748 - the interrupt part, which concerns all the processing performed upon I/O
749 receipt, before the target process is even known. Typically Rx packets are
750 accounted for in interrupt. On some systems such as Linux where interrupt
751 processing may be deferred to a dedicated thread, it can appear as softirq,
752 and the thread is called ksoftirqd/0 (for CPU 0). The CPU taking care of
753 this load is generally defined by the hardware settings, though in the case
754 of softirq it is often possible to remap the processing to another CPU.
755 This interrupt part will often be perceived as parasitic since it's not
756 associated with any process, but it actually is some processing being done
757 to prepare the work for the process.
758
759 - the system part, which concerns all the processing done using kernel code
760 called from userland. System calls are accounted as system for example. All
761 synchronously delivered Tx packets will be accounted for as system time. If
762 some packets have to be deferred due to queues filling up, they may then be
763 processed in interrupt context later (eg: upon receipt of an ACK opening a
764 TCP window).
765
766 - the user part, which exclusively runs application code in userland. HAProxy
767 runs exclusively in this part, though it makes heavy use of system calls.
768 Rules processing, regular expressions, compression, encryption all add to
769 the user portion of CPU consumption.
770
771 - the idle part, which is what the CPU does when there is nothing to do. For
772 example HAProxy waits for an incoming connection, or waits for some data to
773 leave, meaning the system is waiting for an ACK from the client to push
774 these data.
775
776In practice regarding HAProxy's activity, it is in general reasonably accurate
777(but totally inexact) to consider that interrupt/softirq are caused by Rx
778processing in kernel drivers, that user-land is caused by layer 7 processing
779in HAProxy, and that system time is caused by network processing on the Tx
780path.
781
782Since HAProxy runs around an event loop, it waits for new events using poll()
783(or any alternative) and processes all these events as fast as possible before
784going back to poll() waiting for new events. It measures the time spent waiting
785in poll() compared to the time spent doing processing events. The ratio of
786polling time vs total time is called the "idle" time, it's the amount of time
787spent waiting for something to happen. This ratio is reported in the stats page
788on the "idle" line, or "Idle_pct" on the CLI. When it's close to 100%, it means
789the load is extremely low. When it's close to 0%, it means that there is
790constantly some activity. While it cannot be very accurate on an overloaded
791system due to other processes possibly preempting the CPU from the haproxy
792process, it still provides a good estimate about how HAProxy considers it is
793working : if the load is low and the idle ratio is low as well, it may indicate
794that HAProxy has a lot of work to do, possibly due to very expensive rules that
795have to be processed. Conversely, if HAProxy indicates the idle is close to
796100% while things are slow, it means that it cannot do anything to speed things
797up because it is already waiting for incoming data to process. In the example
798below, haproxy is completely idle :
799
800 $ echo "show info" | socat - /var/run/haproxy.sock | grep ^Idle
801 Idle_pct: 100
802
803When the idle ratio starts to become very low, it is important to tune the
804system and place processes and interrupts correctly to save the most possible
805CPU resources for all tasks. If a firewall is present, it may be worth trying
806to disable it or to tune it to ensure it is not responsible for a large part
807of the performance limitation. It's worth noting that unloading a stateful
808firewall generally reduces both the amount of interrupt/softirq and of system
809usage since such firewalls act both on the Rx and the Tx paths. On Linux,
810unloading the nf_conntrack and ip_conntrack modules will show whether there is
811anything to gain. If so, then the module runs with default settings and you'll
812have to figure how to tune it for better performance. In general this consists
813in considerably increasing the hash table size. On FreeBSD, "pfctl -d" will
814disable the "pf" firewall and its stateful engine at the same time.
815
816If it is observed that a lot of time is spent in interrupt/softirq, it is
817important to ensure that they don't run on the same CPU. Most systems tend to
818pin the tasks on the CPU where they receive the network traffic because for
819certain workloads it improves things. But with heavily network-bound workloads
820it is the opposite as the haproxy process will have to fight against its kernel
821counterpart. Pinning haproxy to one CPU core and the interrupts to another one,
822all sharing the same L3 cache tends to sensibly increase network performance
823because in practice the amount of work for haproxy and the network stack are
824quite close, so they can almost fill an entire CPU each. On Linux this is done
825using taskset (for haproxy) or using cpu-map (from the haproxy config), and the
826interrupts are assigned under /proc/irq. Many network interfaces support
827multiple queues and multiple interrupts. In general it helps to spread them
828across a small number of CPU cores provided they all share the same L3 cache.
829Please always stop irq_balance which always does the worst possible thing on
830such workloads.
831
832For CPU-bound workloads consisting in a lot of SSL traffic or a lot of
833compression, it may be worth using multiple processes dedicated to certain
834tasks, though there is no universal rule here and experimentation will have to
835be performed.
836
837In order to increase the CPU capacity, it is possible to make HAProxy run as
838several processes, using the "nbproc" directive in the global section. There
839are some limitations though :
840 - health checks are run per process, so the target servers will get as many
841 checks as there are running processes ;
842 - maxconn values and queues are per-process so the correct value must be set
843 to avoid overloading the servers ;
844 - outgoing connections should avoid using port ranges to avoid conflicts
845 - stick-tables are per process and are not shared between processes ;
846 - each peers section may only run on a single process at a time ;
847 - the CLI operations will only act on a single process at a time.
848
849With this in mind, it appears that the easiest setup often consists in having
850one first layer running on multiple processes and in charge for the heavy
851processing, passing the traffic to a second layer running in a single process.
852This mechanism is suited to SSL and compression which are the two CPU-heavy
853features. Instances can easily be chained over UNIX sockets (which are cheaper
fengpeiyuancc123c62016-01-15 16:40:53 +0800854than TCP sockets and which do not waste ports), and the proxy protocol which is
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200855useful to pass client information to the next stage. When doing so, it is
856generally a good idea to bind all the single-process tasks to process number 1
857and extra tasks to next processes, as this will make it easier to generate
858similar configurations for different machines.
859
860On Linux versions 3.9 and above, running HAProxy in multi-process mode is much
861more efficient when each process uses a distinct listening socket on the same
862IP:port ; this will make the kernel evenly distribute the load across all
863processes instead of waking them all up. Please check the "process" option of
864the "bind" keyword lines in the configuration manual for more information.
865
866
8678. Logging
868----------
869
870For logging, HAProxy always relies on a syslog server since it does not perform
871any file-system access. The standard way of using it is to send logs over UDP
872to the log server (by default on port 514). Very commonly this is configured to
873127.0.0.1 where the local syslog daemon is running, but it's also used over the
874network to log to a central server. The central server provides additional
875benefits especially in active-active scenarios where it is desirable to keep
876the logs merged in arrival order. HAProxy may also make use of a UNIX socket to
877send its logs to the local syslog daemon, but it is not recommended at all,
878because if the syslog server is restarted while haproxy runs, the socket will
879be replaced and new logs will be lost. Since HAProxy will be isolated inside a
880chroot jail, it will not have the ability to reconnect to the new socket. It
881has also been observed in field that the log buffers in use on UNIX sockets are
882very small and lead to lost messages even at very light loads. But this can be
883fine for testing however.
884
885It is recommended to add the following directive to the "global" section to
886make HAProxy log to the local daemon using facility "local0" :
887
888 log 127.0.0.1:514 local0
889
890and then to add the following one to each "defaults" section or to each frontend
891and backend section :
892
893 log global
894
895This way, all logs will be centralized through the global definition of where
896the log server is.
897
898Some syslog daemons do not listen to UDP traffic by default, so depending on
899the daemon being used, the syntax to enable this will vary :
900
901 - on sysklogd, you need to pass argument "-r" on the daemon's command line
902 so that it listens to a UDP socket for "remote" logs ; note that there is
903 no way to limit it to address 127.0.0.1 so it will also receive logs from
904 remote systems ;
905
906 - on rsyslogd, the following lines must be added to the configuration file :
907
908 $ModLoad imudp
909 $UDPServerAddress *
910 $UDPServerRun 514
911
912 - on syslog-ng, a new source can be created the following way, it then needs
913 to be added as a valid source in one of the "log" directives :
914
915 source s_udp {
916 udp(ip(127.0.0.1) port(514));
917 };
918
919Please consult your syslog daemon's manual for more information. If no logs are
920seen in the system's log files, please consider the following tests :
921
922 - restart haproxy. Each frontend and backend logs one line indicating it's
923 starting. If these logs are received, it means logs are working.
924
925 - run "strace -tt -s100 -etrace=sendmsg -p <haproxy's pid>" and perform some
926 activity that you expect to be logged. You should see the log messages
927 being sent using sendmsg() there. If they don't appear, restart using
928 strace on top of haproxy. If you still see no logs, it definitely means
929 that something is wrong in your configuration.
930
931 - run tcpdump to watch for port 514, for example on the loopback interface if
932 the traffic is being sent locally : "tcpdump -As0 -ni lo port 514". If the
933 packets are seen there, it's the proof they're sent then the syslogd daemon
934 needs to be troubleshooted.
935
936While traffic logs are sent from the frontends (where the incoming connections
937are accepted), backends also need to be able to send logs in order to report a
938server state change consecutive to a health check. Please consult HAProxy's
939configuration manual for more information regarding all possible log settings.
940
Dan Lloyd8e48b872016-07-01 21:01:18 -0400941It is convenient to chose a facility that is not used by other daemons. HAProxy
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200942examples often suggest "local0" for traffic logs and "local1" for admin logs
943because they're never seen in field. A single facility would be enough as well.
944Having separate logs is convenient for log analysis, but it's also important to
945remember that logs may sometimes convey confidential information, and as such
Dan Lloyd8e48b872016-07-01 21:01:18 -0400946they must not be mixed with other logs that may accidentally be handed out to
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200947unauthorized people.
948
949For in-field troubleshooting without impacting the server's capacity too much,
950it is recommended to make use of the "halog" utility provided with HAProxy.
951This is sort of a grep-like utility designed to process HAProxy log files at
952a very fast data rate. Typical figures range between 1 and 2 GB of logs per
953second. It is capable of extracting only certain logs (eg: search for some
954classes of HTTP status codes, connection termination status, search by response
955time ranges, look for errors only), count lines, limit the output to a number
956of lines, and perform some more advanced statistics such as sorting servers
957by response time or error counts, sorting URLs by time or count, sorting client
958addresses by access count, and so on. It is pretty convenient to quickly spot
959anomalies such as a bot looping on the site, and block them.
960
961
9629. Statistics and monitoring
963----------------------------
964
Willy Tarreau44aed902015-10-13 14:45:29 +0200965It is possible to query HAProxy about its status. The most commonly used
966mechanism is the HTTP statistics page. This page also exposes an alternative
967CSV output format for monitoring tools. The same format is provided on the
968Unix socket.
969
Amaury Denoyelle072f97e2020-10-05 11:49:37 +0200970Statistics are regroup in categories labelled as domains, corresponding to the
Ilya Shipitsin2272d8a2020-12-21 01:22:40 +0500971multiple components of HAProxy. There are two domains available: proxy and dns.
Amaury Denoyellefbd0bc92020-10-05 11:49:46 +0200972If not specified, the proxy domain is selected. Note that only the proxy
973statistics are printed on the HTTP page.
Willy Tarreau44aed902015-10-13 14:45:29 +0200974
9759.1. CSV format
976---------------
977
978The statistics may be consulted either from the unix socket or from the HTTP
979page. Both means provide a CSV format whose fields follow. The first line
980begins with a sharp ('#') and has one word per comma-delimited field which
981represents the title of the column. All other lines starting at the second one
982use a classical CSV format using a comma as the delimiter, and the double quote
983('"') as an optional text delimiter, but only if the enclosed text is ambiguous
984(if it contains a quote or a comma). The double-quote character ('"') in the
985text is doubled ('""'), which is the format that most tools recognize. Please
986do not insert any column before these ones in order not to break tools which
987use hard-coded column positions.
988
Amaury Denoyelle50660a82020-10-05 11:49:39 +0200989For proxy statistics, after each field name, the types which may have a value
990for that field are specified in brackets. The types are L (Listeners), F
991(Frontends), B (Backends), and S (Servers). There is a fixed set of static
992fields that are always available in the same order. A column containing the
993character '-' delimits the end of the static fields, after which presence or
994order of the fields are not guaranteed.
Willy Tarreau44aed902015-10-13 14:45:29 +0200995
Amaury Denoyelle50660a82020-10-05 11:49:39 +0200996Here is the list of static fields using the proxy statistics domain:
Willy Tarreau44aed902015-10-13 14:45:29 +0200997 0. pxname [LFBS]: proxy name
998 1. svname [LFBS]: service name (FRONTEND for frontend, BACKEND for backend,
999 any name for server/listener)
1000 2. qcur [..BS]: current queued requests. For the backend this reports the
1001 number queued without a server assigned.
1002 3. qmax [..BS]: max value of qcur
1003 4. scur [LFBS]: current sessions
1004 5. smax [LFBS]: max sessions
1005 6. slim [LFBS]: configured session limit
Willy Tarreauc73810f2016-01-11 13:52:04 +01001006 7. stot [LFBS]: cumulative number of sessions
Willy Tarreau44aed902015-10-13 14:45:29 +02001007 8. bin [LFBS]: bytes in
1008 9. bout [LFBS]: bytes out
1009 10. dreq [LFB.]: requests denied because of security concerns.
1010 - For tcp this is because of a matched tcp-request content rule.
1011 - For http this is because of a matched http-request or tarpit rule.
1012 11. dresp [LFBS]: responses denied because of security concerns.
1013 - For http this is because of a matched http-request rule, or
1014 "option checkcache".
1015 12. ereq [LF..]: request errors. Some of the possible causes are:
1016 - early termination from the client, before the request has been sent.
1017 - read error from the client
1018 - client timeout
1019 - client closed connection
1020 - various bad requests from the client.
1021 - request was tarpitted.
1022 13. econ [..BS]: number of requests that encountered an error trying to
1023 connect to a backend server. The backend stat is the sum of the stat
1024 for all servers of that backend, plus any connection errors not
1025 associated with a particular server (such as the backend having no
1026 active servers).
1027 14. eresp [..BS]: response errors. srv_abrt will be counted here also.
1028 Some other errors are:
1029 - write error on the client socket (won't be counted for the server stat)
1030 - failure applying filters to the response.
1031 15. wretr [..BS]: number of times a connection to a server was retried.
1032 16. wredis [..BS]: number of times a request was redispatched to another
1033 server. The server value counts the number of times that server was
1034 switched away from.
Willy Tarreaub96dd282016-11-09 14:45:51 +01001035 17. status [LFBS]: status (UP/DOWN/NOLB/MAINT/MAINT(via)/MAINT(resolution)...)
Willy Tarreaubd715102020-10-23 22:44:30 +02001036 18. weight [..BS]: total effective weight (backend), effective weight (server)
Willy Tarreau44aed902015-10-13 14:45:29 +02001037 19. act [..BS]: number of active servers (backend), server is active (server)
1038 20. bck [..BS]: number of backup servers (backend), server is backup (server)
1039 21. chkfail [...S]: number of failed checks. (Only counts checks failed when
1040 the server is up.)
1041 22. chkdown [..BS]: number of UP->DOWN transitions. The backend counter counts
1042 transitions to the whole backend being down, rather than the sum of the
1043 counters for each server.
1044 23. lastchg [..BS]: number of seconds since the last UP<->DOWN transition
1045 24. downtime [..BS]: total downtime (in seconds). The value for the backend
1046 is the downtime for the whole backend, not the sum of the server downtime.
1047 25. qlimit [...S]: configured maxqueue for the server, or nothing in the
1048 value is 0 (default, meaning no limit)
1049 26. pid [LFBS]: process id (0 for first instance, 1 for second, ...)
1050 27. iid [LFBS]: unique proxy id
1051 28. sid [L..S]: server id (unique inside a proxy)
1052 29. throttle [...S]: current throttle percentage for the server, when
1053 slowstart is active, or no value if not in slowstart.
1054 30. lbtot [..BS]: total number of times a server was selected, either for new
1055 sessions, or when re-dispatching. The server counter is the number
1056 of times that server was selected.
1057 31. tracked [...S]: id of proxy/server if tracking is enabled.
1058 32. type [LFBS]: (0=frontend, 1=backend, 2=server, 3=socket/listener)
1059 33. rate [.FBS]: number of sessions per second over last elapsed second
1060 34. rate_lim [.F..]: configured limit on new sessions per second
1061 35. rate_max [.FBS]: max number of new sessions per second
1062 36. check_status [...S]: status of last health check, one of:
1063 UNK -> unknown
1064 INI -> initializing
1065 SOCKERR -> socket error
1066 L4OK -> check passed on layer 4, no upper layers testing enabled
1067 L4TOUT -> layer 1-4 timeout
1068 L4CON -> layer 1-4 connection problem, for example
1069 "Connection refused" (tcp rst) or "No route to host" (icmp)
1070 L6OK -> check passed on layer 6
1071 L6TOUT -> layer 6 (SSL) timeout
1072 L6RSP -> layer 6 invalid response - protocol error
1073 L7OK -> check passed on layer 7
1074 L7OKC -> check conditionally passed on layer 7, for example 404 with
1075 disable-on-404
1076 L7TOUT -> layer 7 (HTTP/SMTP) timeout
1077 L7RSP -> layer 7 invalid response - protocol error
1078 L7STS -> layer 7 response error, for example HTTP 5xx
Daniel Schnellerb6c8b0d2017-09-01 19:13:55 +02001079 Notice: If a check is currently running, the last known status will be
1080 reported, prefixed with "* ". e. g. "* L7OK".
Willy Tarreau44aed902015-10-13 14:45:29 +02001081 37. check_code [...S]: layer5-7 code, if available
1082 38. check_duration [...S]: time in ms took to finish last health check
1083 39. hrsp_1xx [.FBS]: http responses with 1xx code
1084 40. hrsp_2xx [.FBS]: http responses with 2xx code
1085 41. hrsp_3xx [.FBS]: http responses with 3xx code
1086 42. hrsp_4xx [.FBS]: http responses with 4xx code
1087 43. hrsp_5xx [.FBS]: http responses with 5xx code
1088 44. hrsp_other [.FBS]: http responses with other codes (protocol error)
1089 45. hanafail [...S]: failed health checks details
1090 46. req_rate [.F..]: HTTP requests per second over last elapsed second
1091 47. req_rate_max [.F..]: max number of HTTP requests per second observed
Willy Tarreaufb981bd2016-12-12 14:31:46 +01001092 48. req_tot [.FB.]: total number of HTTP requests received
Willy Tarreau44aed902015-10-13 14:45:29 +02001093 49. cli_abrt [..BS]: number of data transfers aborted by the client
1094 50. srv_abrt [..BS]: number of data transfers aborted by the server
1095 (inc. in eresp)
1096 51. comp_in [.FB.]: number of HTTP response bytes fed to the compressor
1097 52. comp_out [.FB.]: number of HTTP response bytes emitted by the compressor
1098 53. comp_byp [.FB.]: number of bytes that bypassed the HTTP compressor
1099 (CPU/BW limit)
1100 54. comp_rsp [.FB.]: number of HTTP responses that were compressed
1101 55. lastsess [..BS]: number of seconds since last session assigned to
1102 server/backend
1103 56. last_chk [...S]: last health check contents or textual error
1104 57. last_agt [...S]: last agent check contents or textual error
1105 58. qtime [..BS]: the average queue time in ms over the 1024 last requests
1106 59. ctime [..BS]: the average connect time in ms over the 1024 last requests
1107 60. rtime [..BS]: the average response time in ms over the 1024 last requests
1108 (0 for TCP)
1109 61. ttime [..BS]: the average total session time in ms over the 1024 last
1110 requests
Willy Tarreau7f618842016-01-08 11:40:03 +01001111 62. agent_status [...S]: status of last agent check, one of:
1112 UNK -> unknown
1113 INI -> initializing
1114 SOCKERR -> socket error
1115 L4OK -> check passed on layer 4, no upper layers testing enabled
1116 L4TOUT -> layer 1-4 timeout
1117 L4CON -> layer 1-4 connection problem, for example
1118 "Connection refused" (tcp rst) or "No route to host" (icmp)
1119 L7OK -> agent reported "up"
1120 L7STS -> agent reported "fail", "stop", or "down"
1121 63. agent_code [...S]: numeric code reported by agent if any (unused for now)
1122 64. agent_duration [...S]: time in ms taken to finish last check
Willy Tarreaudd7354b2016-01-08 13:47:26 +01001123 65. check_desc [...S]: short human-readable description of check_status
1124 66. agent_desc [...S]: short human-readable description of agent_status
Willy Tarreau3141f592016-01-08 14:25:28 +01001125 67. check_rise [...S]: server's "rise" parameter used by checks
1126 68. check_fall [...S]: server's "fall" parameter used by checks
1127 69. check_health [...S]: server's health check value between 0 and rise+fall-1
1128 70. agent_rise [...S]: agent's "rise" parameter, normally 1
1129 71. agent_fall [...S]: agent's "fall" parameter, normally 1
1130 72. agent_health [...S]: agent's health parameter, between 0 and rise+fall-1
Willy Tarreaua6f5a732016-01-08 16:59:56 +01001131 73. addr [L..S]: address:port or "unix". IPv6 has brackets around the address.
Willy Tarreaue4847c62016-01-08 15:43:54 +01001132 74: cookie [..BS]: server's cookie value or backend's cookie name
Willy Tarreauf8211df2016-01-11 14:09:38 +01001133 75: mode [LFBS]: proxy mode (tcp, http, health, unknown)
Willy Tarreauf1516d92016-01-11 14:48:36 +01001134 76: algo [..B.]: load balancing algorithm
Willy Tarreauc73810f2016-01-11 13:52:04 +01001135 77: conn_rate [.F..]: number of connections over the last elapsed second
1136 78: conn_rate_max [.F..]: highest known conn_rate
1137 79: conn_tot [.F..]: cumulative number of connections
Willy Tarreau5b9bdff2016-01-11 14:40:47 +01001138 80: intercepted [.FB.]: cum. number of intercepted requests (monitor, stats)
Willy Tarreau8a90b8e2016-10-21 18:15:32 +02001139 81: dcon [LF..]: requests denied by "tcp-request connection" rules
Willy Tarreaua5bc36b2016-10-21 18:16:27 +02001140 82: dses [LF..]: requests denied by "tcp-request session" rules
Willy Tarreauea96a822018-05-28 15:15:43 +02001141 83: wrew [LFBS]: cumulative number of failed header rewriting warnings
Jérôme Magnin708eb882019-07-17 09:24:46 +02001142 84: connect [..BS]: cumulative number of connection establishment attempts
1143 85: reuse [..BS]: cumulative number of connection reuses
Willy Tarreau72974292019-11-08 07:29:34 +01001144 86: cache_lookups [.FB.]: cumulative number of cache lookups
Jérôme Magnin34ebb5c2019-07-17 14:04:40 +02001145 87: cache_hits [.FB.]: cumulative number of cache hits
Christopher Faulet2ac25742019-11-08 15:27:27 +01001146 88: srv_icur [...S]: current number of idle connections available for reuse
1147 89: src_ilim [...S]: limit on the number of available idle connections
1148 90. qtime_max [..BS]: the maximum observed queue time in ms
1149 91. ctime_max [..BS]: the maximum observed connect time in ms
1150 92. rtime_max [..BS]: the maximum observed response time in ms (0 for TCP)
1151 93. ttime_max [..BS]: the maximum observed total session time in ms
Christopher Faulet0159ee42019-12-16 14:40:39 +01001152 94. eint [LFBS]: cumulative number of internal errors
Pierre Cheynier08eb7182020-10-08 16:37:14 +02001153 95. idle_conn_cur [...S]: current number of unsafe idle connections
1154 96. safe_conn_cur [...S]: current number of safe idle connections
1155 97. used_conn_cur [...S]: current number of connections in use
1156 98. need_conn_est [...S]: estimated needed number of connections
Willy Tarreaubd715102020-10-23 22:44:30 +02001157 99. uweight [..BS]: total user weight (backend), server user weight (server)
Willy Tarreau44aed902015-10-13 14:45:29 +02001158
Amaury Denoyelle50660a82020-10-05 11:49:39 +02001159For all other statistics domains, the presence or the order of the fields are
1160not guaranteed. In this case, the header line should always be used to parse
1161the CSV data.
Willy Tarreau44aed902015-10-13 14:45:29 +02001162
Phil Schererb931f962020-12-02 19:36:08 +000011639.2. Typed output format
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001164------------------------
1165
1166Both "show info" and "show stat" support a mode where each output value comes
1167with its type and sufficient information to know how the value is supposed to
1168be aggregated between processes and how it evolves.
1169
1170In all cases, the output consists in having a single value per line with all
1171the information split into fields delimited by colons (':').
1172
1173The first column designates the object or metric being dumped. Its format is
1174specific to the command producing this output and will not be described in this
1175section. Usually it will consist in a series of identifiers and field names.
1176
1177The second column contains 3 characters respectively indicating the origin, the
1178nature and the scope of the value being reported. The first character (the
1179origin) indicates where the value was extracted from. Possible characters are :
1180
1181 M The value is a metric. It is valid at one instant any may change depending
1182 on its nature .
1183
1184 S The value is a status. It represents a discrete value which by definition
1185 cannot be aggregated. It may be the status of a server ("UP" or "DOWN"),
1186 the PID of the process, etc.
1187
1188 K The value is a sorting key. It represents an identifier which may be used
1189 to group some values together because it is unique among its class. All
1190 internal identifiers are keys. Some names can be listed as keys if they
1191 are unique (eg: a frontend name is unique). In general keys come from the
Dan Lloyd8e48b872016-07-01 21:01:18 -04001192 configuration, even though some of them may automatically be assigned. For
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001193 most purposes keys may be considered as equivalent to configuration.
1194
1195 C The value comes from the configuration. Certain configuration values make
1196 sense on the output, for example a concurrent connection limit or a cookie
1197 name. By definition these values are the same in all processes started
1198 from the same configuration file.
1199
1200 P The value comes from the product itself. There are very few such values,
1201 most common use is to report the product name, version and release date.
1202 These elements are also the same between all processes.
1203
1204The second character (the nature) indicates the nature of the information
1205carried by the field in order to let an aggregator decide on what operation to
1206use to aggregate multiple values. Possible characters are :
1207
1208 A The value represents an age since a last event. This is a bit different
1209 from the duration in that an age is automatically computed based on the
1210 current date. A typical example is how long ago did the last session
1211 happen on a server. Ages are generally aggregated by taking the minimum
1212 value and do not need to be stored.
1213
1214 a The value represents an already averaged value. The average response times
1215 and server weights are of this nature. Averages can typically be averaged
1216 between processes.
1217
1218 C The value represents a cumulative counter. Such measures perpetually
1219 increase until they wrap around. Some monitoring protocols need to tell
1220 the difference between a counter and a gauge to report a different type.
1221 In general counters may simply be summed since they represent events or
1222 volumes. Examples of metrics of this nature are connection counts or byte
1223 counts.
1224
1225 D The value represents a duration for a status. There are a few usages of
1226 this, most of them include the time taken by the last health check and
1227 the time a server has spent down. Durations are generally not summed,
1228 most of the time the maximum will be retained to compute an SLA.
1229
1230 G The value represents a gauge. It's a measure at one instant. The memory
1231 usage or the current number of active connections are of this nature.
1232 Metrics of this type are typically summed during aggregation.
1233
1234 L The value represents a limit (generally a configured one). By nature,
1235 limits are harder to aggregate since they are specific to the point where
1236 they were retrieved. In certain situations they may be summed or be kept
1237 separate.
1238
1239 M The value represents a maximum. In general it will apply to a gauge and
1240 keep the highest known value. An example of such a metric could be the
1241 maximum amount of concurrent connections that was encountered in the
1242 product's life time. To correctly aggregate maxima, you are supposed to
1243 output a range going from the maximum of all maxima and the sum of all
1244 of them. There is indeed no way to know if they were encountered
1245 simultaneously or not.
1246
1247 m The value represents a minimum. In general it will apply to a gauge and
1248 keep the lowest known value. An example of such a metric could be the
1249 minimum amount of free memory pools that was encountered in the product's
1250 life time. To correctly aggregate minima, you are supposed to output a
1251 range going from the minimum of all minima and the sum of all of them.
1252 There is indeed no way to know if they were encountered simultaneously
1253 or not.
1254
1255 N The value represents a name, so it is a string. It is used to report
1256 proxy names, server names and cookie names. Names have configuration or
1257 keys as their origin and are supposed to be the same among all processes.
1258
1259 O The value represents a free text output. Outputs from various commands,
1260 returns from health checks, node descriptions are of such nature.
1261
1262 R The value represents an event rate. It's a measure at one instant. It is
1263 quite similar to a gauge except that the recipient knows that this measure
1264 moves slowly and may decide not to keep all values. An example of such a
1265 metric is the measured amount of connections per second. Metrics of this
1266 type are typically summed during aggregation.
1267
1268 T The value represents a date or time. A field emitting the current date
1269 would be of this type. The method to aggregate such information is left
1270 as an implementation choice. For now no field uses this type.
1271
1272The third character (the scope) indicates what extent the value reflects. Some
1273elements may be per process while others may be per configuration or per system.
1274The distinction is important to know whether or not a single value should be
1275kept during aggregation or if values have to be aggregated. The following
1276characters are currently supported :
1277
1278 C The value is valid for a whole cluster of nodes, which is the set of nodes
1279 communicating over the peers protocol. An example could be the amount of
1280 entries present in a stick table that is replicated with other peers. At
1281 the moment no metric use this scope.
1282
1283 P The value is valid only for the process reporting it. Most metrics use
1284 this scope.
1285
1286 S The value is valid for the whole service, which is the set of processes
1287 started together from the same configuration file. All metrics originating
1288 from the configuration use this scope. Some other metrics may use it as
1289 well for some shared resources (eg: shared SSL cache statistics).
1290
1291 s The value is valid for the whole system, such as the system's hostname,
1292 current date or resource usage. At the moment this scope is not used by
1293 any metric.
1294
1295Consumers of these information will generally have enough of these 3 characters
1296to determine how to accurately report aggregated information across multiple
1297processes.
1298
1299After this column, the third column indicates the type of the field, among "s32"
1300(signed 32-bit integer), "s64" (signed 64-bit integer), "u32" (unsigned 32-bit
1301integer), "u64" (unsigned 64-bit integer), "str" (string). It is important to
1302know the type before parsing the value in order to properly read it. For example
1303a string containing only digits is still a string an not an integer (eg: an
1304error code extracted by a check).
1305
1306Then the fourth column is the value itself, encoded according to its type.
1307Strings are dumped as-is immediately after the colon without any leading space.
1308If a string contains a colon, it will appear normally. This means that the
1309output should not be exclusively split around colons or some check outputs
1310or server addresses might be truncated.
1311
1312
13139.3. Unix Socket commands
Willy Tarreau44aed902015-10-13 14:45:29 +02001314-------------------------
1315
1316The stats socket is not enabled by default. In order to enable it, it is
1317necessary to add one line in the global section of the haproxy configuration.
1318A second line is recommended to set a larger timeout, always appreciated when
1319issuing commands by hand :
1320
1321 global
1322 stats socket /var/run/haproxy.sock mode 600 level admin
1323 stats timeout 2m
1324
1325It is also possible to add multiple instances of the stats socket by repeating
1326the line, and make them listen to a TCP port instead of a UNIX socket. This is
1327never done by default because this is dangerous, but can be handy in some
1328situations :
1329
1330 global
1331 stats socket /var/run/haproxy.sock mode 600 level admin
1332 stats socket ipv4@192.168.0.1:9999 level admin
1333 stats timeout 2m
1334
1335To access the socket, an external utility such as "socat" is required. Socat is
1336a swiss-army knife to connect anything to anything. We use it to connect
1337terminals to the socket, or a couple of stdin/stdout pipes to it for scripts.
1338The two main syntaxes we'll use are the following :
1339
1340 # socat /var/run/haproxy.sock stdio
1341 # socat /var/run/haproxy.sock readline
1342
1343The first one is used with scripts. It is possible to send the output of a
1344script to haproxy, and pass haproxy's output to another script. That's useful
1345for retrieving counters or attack traces for example.
1346
1347The second one is only useful for issuing commands by hand. It has the benefit
1348that the terminal is handled by the readline library which supports line
1349editing and history, which is very convenient when issuing repeated commands
1350(eg: watch a counter).
1351
1352The socket supports two operation modes :
1353 - interactive
1354 - non-interactive
1355
1356The non-interactive mode is the default when socat connects to the socket. In
1357this mode, a single line may be sent. It is processed as a whole, responses are
1358sent back, and the connection closes after the end of the response. This is the
1359mode that scripts and monitoring tools use. It is possible to send multiple
1360commands in this mode, they need to be delimited by a semi-colon (';'). For
1361example :
1362
1363 # echo "show info;show stat;show table" | socat /var/run/haproxy stdio
1364
Dragan Dosena1c35ab2016-11-24 11:33:12 +01001365If a command needs to use a semi-colon or a backslash (eg: in a value), it
Joseph Herlant71b4b152018-11-13 16:55:16 -08001366must be preceded by a backslash ('\').
Chad Lavoiee3f50312016-05-26 16:42:25 -04001367
Willy Tarreau44aed902015-10-13 14:45:29 +02001368The interactive mode displays a prompt ('>') and waits for commands to be
1369entered on the line, then processes them, and displays the prompt again to wait
1370for a new command. This mode is entered via the "prompt" command which must be
1371sent on the first line in non-interactive mode. The mode is a flip switch, if
1372"prompt" is sent in interactive mode, it is disabled and the connection closes
1373after processing the last command of the same line.
1374
1375For this reason, when debugging by hand, it's quite common to start with the
1376"prompt" command :
1377
1378 # socat /var/run/haproxy readline
1379 prompt
1380 > show info
1381 ...
1382 >
1383
1384Since multiple commands may be issued at once, haproxy uses the empty line as a
1385delimiter to mark an end of output for each command, and takes care of ensuring
1386that no command can emit an empty line on output. A script can thus easily
1387parse the output even when multiple commands were pipelined on a single line.
1388
Aurélien Nephtaliabbf6072018-04-18 13:26:46 +02001389Some commands may take an optional payload. To add one to a command, the first
1390line needs to end with the "<<\n" pattern. The next lines will be treated as
1391the payload and can contain as many lines as needed. To validate a command with
1392a payload, it needs to end with an empty line.
1393
1394Limitations do exist: the length of the whole buffer passed to the CLI must
1395not be greater than tune.bfsize and the pattern "<<" must not be glued to the
1396last word of the line.
1397
1398When entering a paylod while in interactive mode, the prompt will change from
1399"> " to "+ ".
1400
Willy Tarreau44aed902015-10-13 14:45:29 +02001401It is important to understand that when multiple haproxy processes are started
1402on the same sockets, any process may pick up the request and will output its
1403own stats.
1404
1405The list of commands currently supported on the stats socket is provided below.
1406If an unknown command is sent, haproxy displays the usage message which reminds
1407all supported commands. Some commands support a more complex syntax, generally
1408it will explain what part of the command is invalid when this happens.
1409
Olivier Doucetd8703e82017-08-31 11:05:10 +02001410Some commands require a higher level of privilege to work. If you do not have
1411enough privilege, you will get an error "Permission denied". Please check
1412the "level" option of the "bind" keyword lines in the configuration manual
1413for more information.
1414
Remi Tricot-Le Bretone88a2ca2021-04-08 15:30:23 +02001415abort ssl ca-file <cafile>
1416 Abort and destroy a temporary CA file update transaction.
1417
1418 See also "set ssl ca-file" and "commit ssl ca-file".
1419
William Lallemand6ab08b32019-11-29 16:48:43 +01001420abort ssl cert <filename>
1421 Abort and destroy a temporary SSL certificate update transaction.
1422
1423 See also "set ssl cert" and "commit ssl cert".
1424
Remi Tricot-Le Breton3c222bd2021-04-27 16:28:25 +02001425abort ssl crl-file <crlfile>
1426 Abort and destroy a temporary CRL file update transaction.
1427
1428 See also "set ssl crl-file" and "commit ssl crl-file".
1429
Willy Tarreaubb51c442021-04-30 15:23:36 +02001430add acl [@<ver>] <acl> <pattern>
Willy Tarreau44aed902015-10-13 14:45:29 +02001431 Add an entry into the acl <acl>. <acl> is the #<id> or the <file> returned by
Willy Tarreaubb51c442021-04-30 15:23:36 +02001432 "show acl". This command does not verify if the entry already exists. Entries
1433 are added to the current version of the ACL, unless a specific version is
1434 specified with "@<ver>". This version number must have preliminary been
1435 allocated by "prepare acl", and it will be comprised between the versions
1436 reported in "curr_ver" and "next_ver" on the output of "show acl". Entries
1437 added with a specific version number will not match until a "commit acl"
1438 operation is performed on them. They may however be consulted using the
1439 "show acl @<ver>" command, and cleared using a "clear acl @<ver>" command.
1440 This command cannot be used if the reference <acl> is a file also used with
1441 a map. In this case, the "add map" command must be used instead.
Willy Tarreau44aed902015-10-13 14:45:29 +02001442
Willy Tarreaubb51c442021-04-30 15:23:36 +02001443add map [@<ver>] <map> <key> <value>
1444add map [@<ver>] <map> <payload>
Willy Tarreau44aed902015-10-13 14:45:29 +02001445 Add an entry into the map <map> to associate the value <value> to the key
1446 <key>. This command does not verify if the entry already exists. It is
Willy Tarreaubb51c442021-04-30 15:23:36 +02001447 mainly used to fill a map after a "clear" or "prepare" operation. Entries
1448 are added to the current version of the ACL, unless a specific version is
1449 specified with "@<ver>". This version number must have preliminary been
1450 allocated by "prepare acl", and it will be comprised between the versions
1451 reported in "curr_ver" and "next_ver" on the output of "show acl". Entries
1452 added with a specific version number will not match until a "commit map"
1453 operation is performed on them. They may however be consulted using the
1454 "show map @<ver>" command, and cleared using a "clear acl @<ver>" command.
1455 If the designated map is also used as an ACL, the ACL will only match the
1456 <key> part and will ignore the <value> part. Using the payload syntax it is
1457 possible to add multiple key/value pairs by entering them on separate lines.
1458 On each new line, the first word is the key and the rest of the line is
1459 considered to be the value which can even contains spaces.
Aurélien Nephtali25650ce2018-04-18 14:04:47 +02001460
1461 Example:
1462
1463 # socat /tmp/sock1 -
1464 prompt
1465
1466 > add map #-1 <<
1467 + key1 value1
1468 + key2 value2 with spaces
1469 + key3 value3 also with spaces
1470 + key4 value4
1471
1472 >
Willy Tarreau44aed902015-10-13 14:45:29 +02001473
Amaury Denoyellef99f77a2021-03-08 17:13:32 +01001474add server <backend>/<server> [args]*
1475 Instantiate a new server attached to the backend <backend>. Only supported on
1476 a CLI connection running in experimental mode (see "experimental-mode on").
1477 This method is still in development and may change in the future.
1478
1479 The <server> name must not be already used in the backend. A special
Amaury Denoyelleeafd7012021-04-29 14:59:42 +02001480 restriction is put on the backend which must used a dynamic load-balancing
1481 algorithm. A subset of keywords from the server config file statement can be
1482 used to configure the server behavior. Also note that no settings will be
1483 reused from an hypothetical 'default-server' statement in the same backend.
Amaury Denoyellefc465a52021-03-09 17:36:23 +01001484
Amaury Denoyelleefbf35c2021-06-10 17:34:10 +02001485 Currently a dynamic server is statically initialized with the "none"
1486 init-addr method. This means that no resolution will be undertaken if a FQDN
1487 is specified as an address, even if the server creation will be validated.
1488
Amaury Denoyelle14c3c5c2021-08-23 14:10:51 +02001489 To support the reload operations, it is expected that the server created via
1490 the CLI is also manually inserted in the relevant haproxy configuration file.
1491 A dynamic server not present in the configuration won't be restored after a
1492 reload operation.
1493
Amaury Denoyelle56eb8ed2021-07-13 10:36:03 +02001494 A dynamic server may use the "track" keyword to follow the check status of
1495 another server from the configuration. However, it is not possible to track
1496 another dynamic server. This is to ensure that the tracking chain is kept
1497 consistent even in the case of dynamic servers deletion.
1498
Amaury Denoyelle2fc4d392021-07-22 16:04:59 +02001499 Use the "check" keyword to enable health-check support. Note that the
1500 health-check is disabled by default and must be enabled independently from
Amaury Denoyelleb65f4ca2021-08-04 11:33:14 +02001501 the server using the "enable health" command. For agent checks, use the
1502 "agent-check" keyword and the "enable agent" command. Note that in this case
1503 the server may be activated via the agent depending on the status reported,
1504 without an explicit "enable server" command. This also means that extra care
1505 is required when removing a dynamic server with agent check. The agent should
1506 be first deactivated via "disable agent" to be able to put the server in the
1507 required maintenance mode before removal.
Amaury Denoyelle2fc4d392021-07-22 16:04:59 +02001508
Amaury Denoyelle414a6122021-08-06 10:25:32 +02001509 It may be possible to reach the fd limit when using a large number of dynamic
1510 servers. Please refer to the "u-limit" global keyword documentation in this
1511 case.
1512
Amaury Denoyellefc465a52021-03-09 17:36:23 +01001513 Here is the list of the currently supported keywords :
1514
Amaury Denoyelleb65f4ca2021-08-04 11:33:14 +02001515 - agent-addr
1516 - agent-check
1517 - agent-inter
1518 - agent-port
1519 - agent-send
Amaury Denoyelle34897d22021-05-19 09:49:41 +02001520 - allow-0rtt
1521 - alpn
Amaury Denoyelle2fc4d392021-07-22 16:04:59 +02001522 - addr
Amaury Denoyellefc465a52021-03-09 17:36:23 +01001523 - backup
Amaury Denoyelle34897d22021-05-19 09:49:41 +02001524 - ca-file
Amaury Denoyelle2fc4d392021-07-22 16:04:59 +02001525 - check
Amaury Denoyelle79b90e82021-09-20 15:15:19 +02001526 - check-alpn
Amaury Denoyelle2fc4d392021-07-22 16:04:59 +02001527 - check-proto
1528 - check-send-proxy
Amaury Denoyelle79b90e82021-09-20 15:15:19 +02001529 - check-sni
1530 - check-ssl
Amaury Denoyelle2fc4d392021-07-22 16:04:59 +02001531 - check-via-socks4
Amaury Denoyelle34897d22021-05-19 09:49:41 +02001532 - ciphers
1533 - ciphersuites
1534 - crl-file
1535 - crt
Amaury Denoyellefc465a52021-03-09 17:36:23 +01001536 - disabled
Amaury Denoyelle2fc4d392021-07-22 16:04:59 +02001537 - downinter
Amaury Denoyellefc465a52021-03-09 17:36:23 +01001538 - enabled
Amaury Denoyelle725f8d22021-09-20 15:16:12 +02001539 - error-limit
Amaury Denoyelle2fc4d392021-07-22 16:04:59 +02001540 - fall
1541 - fastinter
Amaury Denoyelle34897d22021-05-19 09:49:41 +02001542 - force-sslv3/tlsv10/tlsv11/tlsv12/tlsv13
Amaury Denoyellefc465a52021-03-09 17:36:23 +01001543 - id
Amaury Denoyelle2fc4d392021-07-22 16:04:59 +02001544 - inter
Amaury Denoyellefc465a52021-03-09 17:36:23 +01001545 - maxconn
1546 - maxqueue
1547 - minconn
Amaury Denoyelle34897d22021-05-19 09:49:41 +02001548 - no-ssl-reuse
1549 - no-sslv3/tlsv10/tlsv11/tlsv12/tlsv13
1550 - no-tls-tickets
1551 - npn
Amaury Denoyelle725f8d22021-09-20 15:16:12 +02001552 - observe
1553 - on-error
1554 - on-marked-down
1555 - on-marked-up
Amaury Denoyellefc465a52021-03-09 17:36:23 +01001556 - pool-low-conn
1557 - pool-max-conn
1558 - pool-purge-delay
Amaury Denoyelle2fc4d392021-07-22 16:04:59 +02001559 - port
Amaury Denoyelle30467232021-03-12 18:03:27 +01001560 - proto
Amaury Denoyellefc465a52021-03-09 17:36:23 +01001561 - proxy-v2-options
Amaury Denoyelle2fc4d392021-07-22 16:04:59 +02001562 - rise
Amaury Denoyellefc465a52021-03-09 17:36:23 +01001563 - send-proxy
1564 - send-proxy-v2
Amaury Denoyelle34897d22021-05-19 09:49:41 +02001565 - send-proxy-v2-ssl
1566 - send-proxy-v2-ssl-cn
Amaury Denoyellecd8a6f22021-09-21 11:51:54 +02001567 - slowstart
Amaury Denoyelle34897d22021-05-19 09:49:41 +02001568 - sni
Amaury Denoyellefc465a52021-03-09 17:36:23 +01001569 - source
Amaury Denoyelle34897d22021-05-19 09:49:41 +02001570 - ssl
1571 - ssl-max-ver
1572 - ssl-min-ver
Amaury Denoyellefc465a52021-03-09 17:36:23 +01001573 - tfo
Amaury Denoyelle34897d22021-05-19 09:49:41 +02001574 - tls-tickets
Amaury Denoyelle56eb8ed2021-07-13 10:36:03 +02001575 - track
Amaury Denoyellefc465a52021-03-09 17:36:23 +01001576 - usesrc
Amaury Denoyelle34897d22021-05-19 09:49:41 +02001577 - verify
1578 - verifyhost
Amaury Denoyellefc465a52021-03-09 17:36:23 +01001579 - weight
Amaury Denoyellef9d59572021-10-18 14:40:29 +02001580 - ws
Amaury Denoyellefc465a52021-03-09 17:36:23 +01001581
1582 Their syntax is similar to the server line from the configuration file,
1583 please refer to their individual documentation for details.
Amaury Denoyellef99f77a2021-03-08 17:13:32 +01001584
William Lallemandaccac232020-04-02 17:42:51 +02001585add ssl crt-list <crtlist> <certificate>
1586add ssl crt-list <crtlist> <payload>
1587 Add an certificate in a crt-list. It can also be used for directories since
1588 directories are now loaded the same way as the crt-lists. This command allow
1589 you to use a certificate name in parameter, to use SSL options or filters a
1590 crt-list line must sent as a payload instead. Only one crt-list line is
1591 supported in the payload. This command will load the certificate for every
1592 bind lines using the crt-list. To push a new certificate to HAProxy the
1593 commands "new ssl cert" and "set ssl cert" must be used.
1594
1595 Example:
1596 $ echo "new ssl cert foobar.pem" | socat /tmp/sock1 -
1597 $ echo -e "set ssl cert foobar.pem <<\n$(cat foobar.pem)\n" | socat
1598 /tmp/sock1 -
1599 $ echo "commit ssl cert foobar.pem" | socat /tmp/sock1 -
1600 $ echo "add ssl crt-list certlist1 foobar.pem" | socat /tmp/sock1 -
1601
1602 $ echo -e 'add ssl crt-list certlist1 <<\nfoobar.pem [allow-0rtt] foo.bar.com
1603 !test1.com\n' | socat /tmp/sock1 -
1604
Willy Tarreau44aed902015-10-13 14:45:29 +02001605clear counters
1606 Clear the max values of the statistics counters in each proxy (frontend &
Willy Tarreaud80cb4e2018-01-20 19:30:13 +01001607 backend) and in each server. The accumulated counters are not affected. The
1608 internal activity counters reported by "show activity" are also reset. This
Willy Tarreau44aed902015-10-13 14:45:29 +02001609 can be used to get clean counters after an incident, without having to
1610 restart nor to clear traffic counters. This command is restricted and can
1611 only be issued on sockets configured for levels "operator" or "admin".
1612
1613clear counters all
1614 Clear all statistics counters in each proxy (frontend & backend) and in each
1615 server. This has the same effect as restarting. This command is restricted
1616 and can only be issued on sockets configured for level "admin".
1617
Willy Tarreauff3feeb2021-04-30 13:31:43 +02001618clear acl [@<ver>] <acl>
Willy Tarreau44aed902015-10-13 14:45:29 +02001619 Remove all entries from the acl <acl>. <acl> is the #<id> or the <file>
1620 returned by "show acl". Note that if the reference <acl> is a file and is
Willy Tarreauff3feeb2021-04-30 13:31:43 +02001621 shared with a map, this map will be also cleared. By default only the current
1622 version of the ACL is cleared (the one being matched against). However it is
1623 possible to specify another version using '@' followed by this version.
Willy Tarreau44aed902015-10-13 14:45:29 +02001624
Willy Tarreauff3feeb2021-04-30 13:31:43 +02001625clear map [@<ver>] <map>
Willy Tarreau44aed902015-10-13 14:45:29 +02001626 Remove all entries from the map <map>. <map> is the #<id> or the <file>
1627 returned by "show map". Note that if the reference <map> is a file and is
Willy Tarreauff3feeb2021-04-30 13:31:43 +02001628 shared with a acl, this acl will be also cleared. By default only the current
1629 version of the map is cleared (the one being matched against). However it is
1630 possible to specify another version using '@' followed by this version.
Willy Tarreau44aed902015-10-13 14:45:29 +02001631
1632clear table <table> [ data.<type> <operator> <value> ] | [ key <key> ]
1633 Remove entries from the stick-table <table>.
1634
1635 This is typically used to unblock some users complaining they have been
1636 abusively denied access to a service, but this can also be used to clear some
1637 stickiness entries matching a server that is going to be replaced (see "show
1638 table" below for details). Note that sometimes, removal of an entry will be
1639 refused because it is currently tracked by a session. Retrying a few seconds
1640 later after the session ends is usual enough.
1641
1642 In the case where no options arguments are given all entries will be removed.
1643
1644 When the "data." form is used entries matching a filter applied using the
1645 stored data (see "stick-table" in section 4.2) are removed. A stored data
1646 type must be specified in <type>, and this data type must be stored in the
1647 table otherwise an error is reported. The data is compared according to
1648 <operator> with the 64-bit integer <value>. Operators are the same as with
1649 the ACLs :
1650
1651 - eq : match entries whose data is equal to this value
1652 - ne : match entries whose data is not equal to this value
1653 - le : match entries whose data is less than or equal to this value
1654 - ge : match entries whose data is greater than or equal to this value
1655 - lt : match entries whose data is less than this value
1656 - gt : match entries whose data is greater than this value
1657
1658 When the key form is used the entry <key> is removed. The key must be of the
1659 same type as the table, which currently is limited to IPv4, IPv6, integer and
1660 string.
1661
1662 Example :
1663 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1664 >>> # table: http_proxy, type: ip, size:204800, used:2
1665 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
1666 bytes_out_rate(60000)=187
1667 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
1668 bytes_out_rate(60000)=191
1669
1670 $ echo "clear table http_proxy key 127.0.0.1" | socat stdio /tmp/sock1
1671
1672 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1673 >>> # table: http_proxy, type: ip, size:204800, used:1
1674 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
1675 bytes_out_rate(60000)=191
1676 $ echo "clear table http_proxy data.gpc0 eq 1" | socat stdio /tmp/sock1
1677 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1678 >>> # table: http_proxy, type: ip, size:204800, used:1
1679
Willy Tarreau7a562ca2021-04-30 15:10:01 +02001680commit acl @<ver> <acl>
1681 Commit all changes made to version <ver> of ACL <acl>, and deletes all past
1682 versions. <acl> is the #<id> or the <file> returned by "show acl". The
1683 version number must be between "curr_ver"+1 and "next_ver" as reported in
1684 "show acl". The contents to be committed to the ACL can be consulted with
1685 "show acl @<ver> <acl>" if desired. The specified version number has normally
1686 been created with the "prepare acl" command. The replacement is atomic. It
1687 consists in atomically updating the current version to the specified version,
1688 which will instantly cause all entries in other versions to become invisible,
1689 and all entries in the new version to become visible. It is also possible to
1690 use this command to perform an atomic removal of all visible entries of an
1691 ACL by calling "prepare acl" first then committing without adding any
1692 entries. This command cannot be used if the reference <acl> is a file also
1693 used as a map. In this case, the "commit map" command must be used instead.
1694
1695commit map @<ver> <map>
1696 Commit all changes made to version <ver> of map <map>, and deletes all past
1697 versions. <map> is the #<id> or the <file> returned by "show map". The
1698 version number must be between "curr_ver"+1 and "next_ver" as reported in
1699 "show map". The contents to be committed to the map can be consulted with
1700 "show map @<ver> <map>" if desired. The specified version number has normally
1701 been created with the "prepare map" command. The replacement is atomic. It
1702 consists in atomically updating the current version to the specified version,
1703 which will instantly cause all entries in other versions to become invisible,
1704 and all entries in the new version to become visible. It is also possible to
1705 use this command to perform an atomic removal of all visible entries of an
1706 map by calling "prepare map" first then committing without adding any
1707 entries.
1708
Remi Tricot-Le Bretone88a2ca2021-04-08 15:30:23 +02001709commit ssl ca-file <cafile>
1710 Commit a temporary SSL CA file update transaction.
1711
1712 In the case of an existing CA file (in a "Used" state in "show ssl ca-file"),
1713 the new CA file tree entry is inserted in the CA file tree and every instance
1714 that used the CA file entry is rebuilt, along with the SSL contexts it needs.
1715 All the contexts previously used by the rebuilt instances are removed.
1716 Upon success, the previous CA file entry is removed from the tree.
1717 Upon failure, nothing is removed or deleted, and all the original SSL
1718 contexts are kept and used.
1719 Once the temporary transaction is committed, it is destroyed.
1720
1721 In the case of a new CA file (after a "new ssl ca-file" and in a "Unused"
1722 state in "show ssl ca-file"), the CA file will be inserted in the CA file
1723 tree but it won't be used anywhere in HAProxy. To use it and generate SSL
1724 contexts that use it, you will need to add it to a crt-list with "add ssl
1725 crt-list".
1726
1727 See also "new ssl ca-file", "set ssl ca-file", "abort ssl ca-file" and
1728 "add ssl crt-list".
1729
William Lallemand6ab08b32019-11-29 16:48:43 +01001730commit ssl cert <filename>
William Lallemandc184d872020-06-26 15:39:57 +02001731 Commit a temporary SSL certificate update transaction.
1732
1733 In the case of an existing certificate (in a "Used" state in "show ssl
1734 cert"), generate every SSL contextes and SNIs it need, insert them, and
1735 remove the previous ones. Replace in memory the previous SSL certificates
1736 everywhere the <filename> was used in the configuration. Upon failure it
1737 doesn't remove or insert anything. Once the temporary transaction is
1738 committed, it is destroyed.
1739
1740 In the case of a new certificate (after a "new ssl cert" and in a "Unused"
Ilya Shipitsin2272d8a2020-12-21 01:22:40 +05001741 state in "show ssl cert"), the certificate will be committed in a certificate
William Lallemandc184d872020-06-26 15:39:57 +02001742 storage, but it won't be used anywhere in haproxy. To use it and generate
1743 its SNIs you will need to add it to a crt-list or a directory with "add ssl
1744 crt-list".
William Lallemand6ab08b32019-11-29 16:48:43 +01001745
Remi Tricot-Le Bretone88a2ca2021-04-08 15:30:23 +02001746 See also "new ssl cert", "set ssl cert", "abort ssl cert" and
William Lallemandc184d872020-06-26 15:39:57 +02001747 "add ssl crt-list".
William Lallemand6ab08b32019-11-29 16:48:43 +01001748
Remi Tricot-Le Breton3c222bd2021-04-27 16:28:25 +02001749commit ssl crl-file <crlfile>
1750 Commit a temporary SSL CRL file update transaction.
1751
1752 In the case of an existing CRL file (in a "Used" state in "show ssl
1753 crl-file"), the new CRL file entry is inserted in the CA file tree (which
1754 holds both the CA files and the CRL files) and every instance that used the
1755 CRL file entry is rebuilt, along with the SSL contexts it needs.
1756 All the contexts previously used by the rebuilt instances are removed.
1757 Upon success, the previous CRL file entry is removed from the tree.
1758 Upon failure, nothing is removed or deleted, and all the original SSL
1759 contexts are kept and used.
1760 Once the temporary transaction is committed, it is destroyed.
1761
1762 In the case of a new CRL file (after a "new ssl crl-file" and in a "Unused"
1763 state in "show ssl crl-file"), the CRL file will be inserted in the CRL file
1764 tree but it won't be used anywhere in HAProxy. To use it and generate SSL
1765 contexts that use it, you will need to add it to a crt-list with "add ssl
1766 crt-list".
1767
1768 See also "new ssl crl-file", "set ssl crl-file", "abort ssl crl-file" and
1769 "add ssl crt-list".
1770
Willy Tarreau6bdf3e92019-05-20 14:25:05 +02001771debug dev <command> [args]*
Willy Tarreaub24ab222019-10-24 18:03:39 +02001772 Call a developer-specific command. Only supported on a CLI connection running
1773 in expert mode (see "expert-mode on"). Such commands are extremely dangerous
1774 and not forgiving, any misuse may result in a crash of the process. They are
1775 intended for experts only, and must really not be used unless told to do so.
1776 Some of them are only available when haproxy is built with DEBUG_DEV defined
1777 because they may have security implications. All of these commands require
1778 admin privileges, and are purposely not documented to avoid encouraging their
1779 use by people who are not at ease with the source code.
Willy Tarreau6bdf3e92019-05-20 14:25:05 +02001780
Willy Tarreau44aed902015-10-13 14:45:29 +02001781del acl <acl> [<key>|#<ref>]
1782 Delete all the acl entries from the acl <acl> corresponding to the key <key>.
1783 <acl> is the #<id> or the <file> returned by "show acl". If the <ref> is used,
1784 this command delete only the listed reference. The reference can be found with
1785 listing the content of the acl. Note that if the reference <acl> is a file and
1786 is shared with a map, the entry will be also deleted in the map.
1787
1788del map <map> [<key>|#<ref>]
1789 Delete all the map entries from the map <map> corresponding to the key <key>.
1790 <map> is the #<id> or the <file> returned by "show map". If the <ref> is used,
1791 this command delete only the listed reference. The reference can be found with
1792 listing the content of the map. Note that if the reference <map> is a file and
1793 is shared with a acl, the entry will be also deleted in the map.
1794
Remi Tricot-Le Bretone88a2ca2021-04-08 15:30:23 +02001795del ssl ca-file <cafile>
1796 Delete a CA file tree entry from HAProxy. The CA file must be unused and
1797 removed from any crt-list. "show ssl ca-file" displays the status of the CA
1798 files. The deletion doesn't work with a certificate referenced directly with
1799 the "ca-file" or "ca-verify-file" directives in the configuration.
1800
William Lallemand419e6342020-04-08 12:05:39 +02001801del ssl cert <certfile>
1802 Delete a certificate store from HAProxy. The certificate must be unused and
1803 removed from any crt-list or directory. "show ssl cert" displays the status
1804 of the certificate. The deletion doesn't work with a certificate referenced
1805 directly with the "crt" directive in the configuration.
1806
Remi Tricot-Le Breton3c222bd2021-04-27 16:28:25 +02001807del ssl crl-file <crlfile>
1808 Delete a CRL file tree entry from HAProxy. The CRL file must be unused and
1809 removed from any crt-list. "show ssl crl-file" displays the status of the CRL
1810 files. The deletion doesn't work with a certificate referenced directly with
1811 the "crl-file" directive in the configuration.
1812
William Lallemand0a9b9412020-04-06 17:43:05 +02001813del ssl crt-list <filename> <certfile[:line]>
1814 Delete an entry in a crt-list. This will delete every SNIs used for this
1815 entry in the frontends. If a certificate is used several time in a crt-list,
1816 you will need to provide which line you want to delete. To display the line
1817 numbers, use "show ssl crt-list -n <crtlist>".
1818
Amaury Denoyellee5580432021-04-15 14:41:20 +02001819del server <backend>/<server>
Amaury Denoyelle14c3c5c2021-08-23 14:10:51 +02001820 Remove a server attached to the backend <backend>. All servers are eligible,
1821 except servers which are referenced by other configuration elements. The
1822 server must be put in maintenance mode prior to its deletion. The operation
1823 is cancelled if the serveur still has active or idle connection or its
1824 connection queue is not empty.
Amaury Denoyellee5580432021-04-15 14:41:20 +02001825
Willy Tarreau44aed902015-10-13 14:45:29 +02001826disable agent <backend>/<server>
1827 Mark the auxiliary agent check as temporarily stopped.
1828
1829 In the case where an agent check is being run as a auxiliary check, due
1830 to the agent-check parameter of a server directive, new checks are only
Dan Lloyd8e48b872016-07-01 21:01:18 -04001831 initialized when the agent is in the enabled. Thus, disable agent will
Willy Tarreau44aed902015-10-13 14:45:29 +02001832 prevent any new agent checks from begin initiated until the agent
1833 re-enabled using enable agent.
1834
1835 When an agent is disabled the processing of an auxiliary agent check that
1836 was initiated while the agent was set as enabled is as follows: All
1837 results that would alter the weight, specifically "drain" or a weight
1838 returned by the agent, are ignored. The processing of agent check is
1839 otherwise unchanged.
1840
1841 The motivation for this feature is to allow the weight changing effects
1842 of the agent checks to be paused to allow the weight of a server to be
1843 configured using set weight without being overridden by the agent.
1844
1845 This command is restricted and can only be issued on sockets configured for
1846 level "admin".
1847
Olivier Houchard614f8d72017-03-14 20:08:46 +01001848disable dynamic-cookie backend <backend>
Ilya Shipitsin2a950d02020-03-06 13:07:38 +05001849 Disable the generation of dynamic cookies for the backend <backend>
Olivier Houchard614f8d72017-03-14 20:08:46 +01001850
Willy Tarreau44aed902015-10-13 14:45:29 +02001851disable frontend <frontend>
1852 Mark the frontend as temporarily stopped. This corresponds to the mode which
1853 is used during a soft restart : the frontend releases the port but can be
1854 enabled again if needed. This should be used with care as some non-Linux OSes
1855 are unable to enable it back. This is intended to be used in environments
1856 where stopping a proxy is not even imaginable but a misconfigured proxy must
1857 be fixed. That way it's possible to release the port and bind it into another
1858 process to restore operations. The frontend will appear with status "STOP"
1859 on the stats page.
1860
1861 The frontend may be specified either by its name or by its numeric ID,
1862 prefixed with a sharp ('#').
1863
1864 This command is restricted and can only be issued on sockets configured for
1865 level "admin".
1866
1867disable health <backend>/<server>
1868 Mark the primary health check as temporarily stopped. This will disable
1869 sending of health checks, and the last health check result will be ignored.
1870 The server will be in unchecked state and considered UP unless an auxiliary
1871 agent check forces it down.
1872
1873 This command is restricted and can only be issued on sockets configured for
1874 level "admin".
1875
1876disable server <backend>/<server>
1877 Mark the server DOWN for maintenance. In this mode, no more checks will be
1878 performed on the server until it leaves maintenance.
1879 If the server is tracked by other servers, those servers will be set to DOWN
1880 during the maintenance.
1881
1882 In the statistics page, a server DOWN for maintenance will appear with a
1883 "MAINT" status, its tracking servers with the "MAINT(via)" one.
1884
1885 Both the backend and the server may be specified either by their name or by
1886 their numeric ID, prefixed with a sharp ('#').
1887
1888 This command is restricted and can only be issued on sockets configured for
1889 level "admin".
1890
1891enable agent <backend>/<server>
1892 Resume auxiliary agent check that was temporarily stopped.
1893
1894 See "disable agent" for details of the effect of temporarily starting
1895 and stopping an auxiliary agent.
1896
1897 This command is restricted and can only be issued on sockets configured for
1898 level "admin".
1899
Olivier Houchard614f8d72017-03-14 20:08:46 +01001900enable dynamic-cookie backend <backend>
n9@users.noreply.github.com25a1c8e2019-08-23 11:21:05 +02001901 Enable the generation of dynamic cookies for the backend <backend>.
1902 A secret key must also be provided.
Olivier Houchard614f8d72017-03-14 20:08:46 +01001903
Willy Tarreau44aed902015-10-13 14:45:29 +02001904enable frontend <frontend>
1905 Resume a frontend which was temporarily stopped. It is possible that some of
1906 the listening ports won't be able to bind anymore (eg: if another process
1907 took them since the 'disable frontend' operation). If this happens, an error
1908 is displayed. Some operating systems might not be able to resume a frontend
1909 which was disabled.
1910
1911 The frontend may be specified either by its name or by its numeric ID,
1912 prefixed with a sharp ('#').
1913
1914 This command is restricted and can only be issued on sockets configured for
1915 level "admin".
1916
1917enable health <backend>/<server>
1918 Resume a primary health check that was temporarily stopped. This will enable
1919 sending of health checks again. Please see "disable health" for details.
1920
1921 This command is restricted and can only be issued on sockets configured for
1922 level "admin".
1923
1924enable server <backend>/<server>
1925 If the server was previously marked as DOWN for maintenance, this marks the
1926 server UP and checks are re-enabled.
1927
1928 Both the backend and the server may be specified either by their name or by
1929 their numeric ID, prefixed with a sharp ('#').
1930
1931 This command is restricted and can only be issued on sockets configured for
1932 level "admin".
1933
Amaury Denoyelle18487fb2021-03-18 15:32:53 +01001934experimental-mode [on|off]
1935 Without options, this indicates whether the experimental mode is enabled or
1936 disabled on the current connection. When passed "on", it turns the
1937 experimental mode on for the current CLI connection only. With "off" it turns
1938 it off.
1939
1940 The experimental mode is used to access to extra features still in
1941 development. These features are currently not stable and should be used with
Ilya Shipitsinba13f162021-03-19 22:21:44 +05001942 care. They may be subject to breaking changes across versions.
Amaury Denoyelle18487fb2021-03-18 15:32:53 +01001943
Willy Tarreauabb9f9b2019-10-24 17:55:53 +02001944expert-mode [on|off]
Amaury Denoyelle18487fb2021-03-18 15:32:53 +01001945 This command is similar to experimental-mode but is used to toggle the
1946 expert mode.
1947
1948 The expert mode enables displaying of expert commands that can be extremely
Willy Tarreauabb9f9b2019-10-24 17:55:53 +02001949 dangerous for the process and which may occasionally help developers collect
1950 important information about complex bugs. Any misuse of these features will
1951 likely lead to a process crash. Do not use this option without being invited
1952 to do so. Note that this command is purposely not listed in the help message.
1953 This command is only accessible in admin level. Changing to another level
1954 automatically resets the expert mode.
1955
Willy Tarreau44aed902015-10-13 14:45:29 +02001956get map <map> <value>
1957get acl <acl> <value>
1958 Lookup the value <value> in the map <map> or in the ACL <acl>. <map> or <acl>
1959 are the #<id> or the <file> returned by "show map" or "show acl". This command
1960 returns all the matching patterns associated with this map. This is useful for
1961 debugging maps and ACLs. The output format is composed by one line par
1962 matching type. Each line is composed by space-delimited series of words.
1963
1964 The first two words are:
1965
1966 <match method>: The match method applied. It can be "found", "bool",
1967 "int", "ip", "bin", "len", "str", "beg", "sub", "dir",
1968 "dom", "end" or "reg".
1969
1970 <match result>: The result. Can be "match" or "no-match".
1971
1972 The following words are returned only if the pattern matches an entry.
1973
1974 <index type>: "tree" or "list". The internal lookup algorithm.
1975
1976 <case>: "case-insensitive" or "case-sensitive". The
1977 interpretation of the case.
1978
1979 <entry matched>: match="<entry>". Return the matched pattern. It is
1980 useful with regular expressions.
1981
1982 The two last word are used to show the returned value and its type. With the
1983 "acl" case, the pattern doesn't exist.
1984
1985 return=nothing: No return because there are no "map".
1986 return="<value>": The value returned in the string format.
1987 return=cannot-display: The value cannot be converted as string.
1988
1989 type="<type>": The type of the returned sample.
1990
Willy Tarreauc35eb382021-03-26 14:51:31 +01001991get var <name>
1992 Show the existence, type and contents of the process-wide variable 'name'.
1993 Only process-wide variables are readable, so the name must begin with
1994 'proc.' otherwise no variable will be found. This command requires levels
1995 "operator" or "admin".
1996
Willy Tarreau44aed902015-10-13 14:45:29 +02001997get weight <backend>/<server>
1998 Report the current weight and the initial weight of server <server> in
1999 backend <backend> or an error if either doesn't exist. The initial weight is
2000 the one that appears in the configuration file. Both are normally equal
2001 unless the current weight has been changed. Both the backend and the server
2002 may be specified either by their name or by their numeric ID, prefixed with a
2003 sharp ('#').
2004
Willy Tarreau0b1b8302021-05-09 20:59:23 +02002005help [<command>]
2006 Print the list of known keywords and their basic usage, or commands matching
2007 the requested one. The same help screen is also displayed for unknown
2008 commands.
Willy Tarreau44aed902015-10-13 14:45:29 +02002009
William Lallemandb175c232021-10-19 14:53:55 +02002010httpclient <method> <URI>
2011 Launch an HTTP client request and print the response on the CLI. Only
2012 supported on a CLI connection running in expert mode (see "expert-mode on").
2013 It's only meant for debugging. It currently can't resolve FQDN so your URI must
2014 contains an IP.
2015
Remi Tricot-Le Bretone88a2ca2021-04-08 15:30:23 +02002016new ssl ca-file <cafile>
2017 Create a new empty CA file tree entry to be filled with a set of CA
2018 certificates and added to a crt-list. This command should be used in
2019 combination with "set ssl ca-file" and "add ssl crt-list".
2020
William Lallemandaccac232020-04-02 17:42:51 +02002021new ssl cert <filename>
2022 Create a new empty SSL certificate store to be filled with a certificate and
2023 added to a directory or a crt-list. This command should be used in
2024 combination with "set ssl cert" and "add ssl crt-list".
2025
Remi Tricot-Le Breton3c222bd2021-04-27 16:28:25 +02002026new ssl crl-file <crlfile>
2027 Create a new empty CRL file tree entry to be filled with a set of CRLs
2028 and added to a crt-list. This command should be used in combination with "set
2029 ssl crl-file" and "add ssl crt-list".
2030
Willy Tarreau97218ce2021-04-30 14:57:03 +02002031prepare acl <acl>
2032 Allocate a new version number in ACL <acl> for atomic replacement. <acl> is
2033 the #<id> or the <file> returned by "show acl". The new version number is
2034 shown in response after "New version created:". This number will then be
2035 usable to prepare additions of new entries into the ACL which will then
2036 atomically replace the current ones once committed. It is reported as
2037 "next_ver" in "show acl". There is no impact of allocating new versions, as
2038 unused versions will automatically be removed once a more recent version is
2039 committed. Version numbers are unsigned 32-bit values which wrap at the end,
2040 so care must be taken when comparing them in an external program. This
2041 command cannot be used if the reference <acl> is a file also used as a map.
2042 In this case, the "prepare map" command must be used instead.
2043
2044prepare map <map>
2045 Allocate a new version number in map <map> for atomic replacement. <map> is
2046 the #<id> or the <file> returned by "show map". The new version number is
2047 shown in response after "New version created:". This number will then be
2048 usable to prepare additions of new entries into the map which will then
2049 atomically replace the current ones once committed. It is reported as
2050 "next_ver" in "show map". There is no impact of allocating new versions, as
2051 unused versions will automatically be removed once a more recent version is
2052 committed. Version numbers are unsigned 32-bit values which wrap at the end,
2053 so care must be taken when comparing them in an external program.
2054
Willy Tarreau44aed902015-10-13 14:45:29 +02002055prompt
2056 Toggle the prompt at the beginning of the line and enter or leave interactive
2057 mode. In interactive mode, the connection is not closed after a command
2058 completes. Instead, the prompt will appear again, indicating the user that
2059 the interpreter is waiting for a new command. The prompt consists in a right
2060 angle bracket followed by a space "> ". This mode is particularly convenient
2061 when one wants to periodically check information such as stats or errors.
2062 It is also a good idea to enter interactive mode before issuing a "help"
2063 command.
2064
2065quit
2066 Close the connection when in interactive mode.
2067
Olivier Houchard614f8d72017-03-14 20:08:46 +01002068set dynamic-cookie-key backend <backend> <value>
2069 Modify the secret key used to generate the dynamic persistent cookies.
2070 This will break the existing sessions.
2071
Willy Tarreau44aed902015-10-13 14:45:29 +02002072set map <map> [<key>|#<ref>] <value>
2073 Modify the value corresponding to each key <key> in a map <map>. <map> is the
2074 #<id> or <file> returned by "show map". If the <ref> is used in place of
2075 <key>, only the entry pointed by <ref> is changed. The new value is <value>.
2076
2077set maxconn frontend <frontend> <value>
2078 Dynamically change the specified frontend's maxconn setting. Any positive
2079 value is allowed including zero, but setting values larger than the global
2080 maxconn does not make much sense. If the limit is increased and connections
2081 were pending, they will immediately be accepted. If it is lowered to a value
2082 below the current number of connections, new connections acceptation will be
2083 delayed until the threshold is reached. The frontend might be specified by
2084 either its name or its numeric ID prefixed with a sharp ('#').
2085
Andrew Hayworthedb93a72015-10-27 21:46:25 +00002086set maxconn server <backend/server> <value>
2087 Dynamically change the specified server's maxconn setting. Any positive
2088 value is allowed including zero, but setting values larger than the global
2089 maxconn does not make much sense.
2090
Willy Tarreau44aed902015-10-13 14:45:29 +02002091set maxconn global <maxconn>
2092 Dynamically change the global maxconn setting within the range defined by the
2093 initial global maxconn setting. If it is increased and connections were
2094 pending, they will immediately be accepted. If it is lowered to a value below
2095 the current number of connections, new connections acceptation will be
2096 delayed until the threshold is reached. A value of zero restores the initial
2097 setting.
2098
Willy Tarreau00dd44f2021-05-05 16:44:23 +02002099set profiling { tasks | memory } { auto | on | off }
2100 Enables or disables CPU or memory profiling for the indicated subsystem. This
2101 is equivalent to setting or clearing the "profiling" settings in the "global"
Willy Tarreaucfa71012021-01-29 11:56:21 +01002102 section of the configuration file. Please also see "show profiling". Note
2103 that manually setting the tasks profiling to "on" automatically resets the
2104 scheduler statistics, thus allows to check activity over a given interval.
Willy Tarreau00dd44f2021-05-05 16:44:23 +02002105 The memory profiling is limited to certain operating systems (known to work
2106 on the linux-glibc target), and requires USE_MEMORY_PROFILING to be set at
2107 compile time.
Willy Tarreau75c62c22018-11-22 11:02:09 +01002108
Willy Tarreau44aed902015-10-13 14:45:29 +02002109set rate-limit connections global <value>
2110 Change the process-wide connection rate limit, which is set by the global
2111 'maxconnrate' setting. A value of zero disables the limitation. This limit
2112 applies to all frontends and the change has an immediate effect. The value
2113 is passed in number of connections per second.
2114
2115set rate-limit http-compression global <value>
2116 Change the maximum input compression rate, which is set by the global
2117 'maxcomprate' setting. A value of zero disables the limitation. The value is
2118 passed in number of kilobytes per second. The value is available in the "show
2119 info" on the line "CompressBpsRateLim" in bytes.
2120
2121set rate-limit sessions global <value>
2122 Change the process-wide session rate limit, which is set by the global
2123 'maxsessrate' setting. A value of zero disables the limitation. This limit
2124 applies to all frontends and the change has an immediate effect. The value
2125 is passed in number of sessions per second.
2126
2127set rate-limit ssl-sessions global <value>
2128 Change the process-wide SSL session rate limit, which is set by the global
2129 'maxsslrate' setting. A value of zero disables the limitation. This limit
2130 applies to all frontends and the change has an immediate effect. The value
2131 is passed in number of sessions per second sent to the SSL stack. It applies
2132 before the handshake in order to protect the stack against handshake abuses.
2133
Baptiste Assmann3749ebf2016-08-03 22:34:12 +02002134set server <backend>/<server> addr <ip4 or ip6 address> [port <port>]
Willy Tarreau44aed902015-10-13 14:45:29 +02002135 Replace the current IP address of a server by the one provided.
Michael Prokop4438c602019-05-24 10:25:45 +02002136 Optionally, the port can be changed using the 'port' parameter.
Baptiste Assmann3749ebf2016-08-03 22:34:12 +02002137 Note that changing the port also support switching from/to port mapping
2138 (notation with +X or -Y), only if a port is configured for the health check.
Willy Tarreau44aed902015-10-13 14:45:29 +02002139
2140set server <backend>/<server> agent [ up | down ]
2141 Force a server's agent to a new state. This can be useful to immediately
2142 switch a server's state regardless of some slow agent checks for example.
2143 Note that the change is propagated to tracking servers if any.
2144
William Dauchy7cabc062021-02-11 22:51:24 +01002145set server <backend>/<server> agent-addr <addr> [port <port>]
Misiek43972902017-01-09 09:53:06 +01002146 Change addr for servers agent checks. Allows to migrate agent-checks to
2147 another address at runtime. You can specify both IP and hostname, it will be
2148 resolved.
William Dauchy7cabc062021-02-11 22:51:24 +01002149 Optionally, change the port agent.
2150
2151set server <backend>/<server> agent-port <port>
2152 Change the port used for agent checks.
Misiek43972902017-01-09 09:53:06 +01002153
2154set server <backend>/<server> agent-send <value>
2155 Change agent string sent to agent check target. Allows to update string while
2156 changing server address to keep those two matching.
2157
Willy Tarreau44aed902015-10-13 14:45:29 +02002158set server <backend>/<server> health [ up | stopping | down ]
2159 Force a server's health to a new state. This can be useful to immediately
2160 switch a server's state regardless of some slow health checks for example.
2161 Note that the change is propagated to tracking servers if any.
2162
William Dauchyb456e1f2021-02-11 22:51:23 +01002163set server <backend>/<server> check-addr <ip4 | ip6> [port <port>]
2164 Change the IP address used for server health checks.
2165 Optionally, change the port used for server health checks.
2166
Baptiste Assmann50946562016-08-31 23:26:29 +02002167set server <backend>/<server> check-port <port>
2168 Change the port used for health checking to <port>
2169
Willy Tarreau44aed902015-10-13 14:45:29 +02002170set server <backend>/<server> state [ ready | drain | maint ]
2171 Force a server's administrative state to a new state. This can be useful to
2172 disable load balancing and/or any traffic to a server. Setting the state to
2173 "ready" puts the server in normal mode, and the command is the equivalent of
2174 the "enable server" command. Setting the state to "maint" disables any traffic
2175 to the server as well as any health checks. This is the equivalent of the
2176 "disable server" command. Setting the mode to "drain" only removes the server
2177 from load balancing but still allows it to be checked and to accept new
2178 persistent connections. Changes are propagated to tracking servers if any.
2179
2180set server <backend>/<server> weight <weight>[%]
2181 Change a server's weight to the value passed in argument. This is the exact
2182 equivalent of the "set weight" command below.
2183
Frédéric Lécailleb418c122017-04-26 11:24:02 +02002184set server <backend>/<server> fqdn <FQDN>
Lukas Tribusc5dd5a52018-08-14 11:39:35 +02002185 Change a server's FQDN to the value passed in argument. This requires the
2186 internal run-time DNS resolver to be configured and enabled for this server.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02002187
William Dauchyf6370442020-11-14 19:25:33 +01002188set server <backend>/<server> ssl [ on | off ]
2189 This option configures SSL ciphering on outgoing connections to the server.
2190
Andjelko Iharosc4df59e2017-07-20 11:59:48 +02002191set severity-output [ none | number | string ]
2192 Change the severity output format of the stats socket connected to for the
2193 duration of the current session.
2194
Remi Tricot-Le Bretone88a2ca2021-04-08 15:30:23 +02002195set ssl ca-file <cafile> <payload>
2196 This command is part of a transaction system, the "commit ssl ca-file" and
2197 "abort ssl ca-file" commands could be required.
2198 If there is no on-going transaction, it will create a CA file tree entry into
2199 which the certificates contained in the payload will be stored. The CA file
2200 entry will not be stored in the CA file tree and will only be kept in a
2201 temporary transaction. If a transaction with the same filename already exists,
2202 the previous CA file entry will be deleted and replaced by the new one.
2203 Once the modifications are done, you have to commit the transaction through
2204 a "commit ssl ca-file" call.
2205
2206 Example:
2207 echo -e "set ssl ca-file cafile.pem <<\n$(cat rootCA.crt)\n" | \
2208 socat /var/run/haproxy.stat -
2209 echo "commit ssl ca-file cafile.pem" | socat /var/run/haproxy.stat -
2210
William Lallemand6ab08b32019-11-29 16:48:43 +01002211set ssl cert <filename> <payload>
2212 This command is part of a transaction system, the "commit ssl cert" and
2213 "abort ssl cert" commands could be required.
Remi Tricot-Le Breton34459092021-04-14 16:19:28 +02002214 This whole transaction system works on any certificate displayed by the
Remi Tricot-Le Bretonb5f0fac2021-04-14 16:19:29 +02002215 "show ssl cert" command, so on any frontend or backend certificate.
William Lallemand6ab08b32019-11-29 16:48:43 +01002216 If there is no on-going transaction, it will duplicate the certificate
2217 <filename> in memory to a temporary transaction, then update this
2218 transaction with the PEM file in the payload. If a transaction exists with
2219 the same filename, it will update this transaction. It's also possible to
2220 update the files linked to a certificate (.issuer, .sctl, .oscp etc.)
2221 Once the modification are done, you have to "commit ssl cert" the
2222 transaction.
2223
William Lallemanded8bfad2021-09-16 17:30:51 +02002224 Injection of files over the CLI must be done with caution since an empty line
2225 is used to notify the end of the payload. It is recommended to inject a PEM
2226 file which has been sanitized. A simple method would be to remove every empty
2227 line and only leave what are in the PEM sections. It could be achieved with a
2228 sed command.
2229
William Lallemand6ab08b32019-11-29 16:48:43 +01002230 Example:
William Lallemanded8bfad2021-09-16 17:30:51 +02002231
2232 # With some simple sanitizing
2233 echo -e "set ssl cert localhost.pem <<\n$(sed -n '/^$/d;/-BEGIN/,/-END/p' 127.0.0.1.pem)\n" | \
2234 socat /var/run/haproxy.stat -
2235
2236 # Complete example with commit
William Lallemand6ab08b32019-11-29 16:48:43 +01002237 echo -e "set ssl cert localhost.pem <<\n$(cat 127.0.0.1.pem)\n" | \
2238 socat /var/run/haproxy.stat -
2239 echo -e \
2240 "set ssl cert localhost.pem.issuer <<\n $(cat 127.0.0.1.pem.issuer)\n" | \
2241 socat /var/run/haproxy.stat -
2242 echo -e \
2243 "set ssl cert localhost.pem.ocsp <<\n$(base64 -w 1000 127.0.0.1.pem.ocsp)\n" | \
2244 socat /var/run/haproxy.stat -
2245 echo "commit ssl cert localhost.pem" | socat /var/run/haproxy.stat -
2246
Remi Tricot-Le Breton3c222bd2021-04-27 16:28:25 +02002247set ssl crl-file <crlfile> <payload>
2248 This command is part of a transaction system, the "commit ssl crl-file" and
2249 "abort ssl crl-file" commands could be required.
2250 If there is no on-going transaction, it will create a CRL file tree entry into
2251 which the Revocation Lists contained in the payload will be stored. The CRL
2252 file entry will not be stored in the CRL file tree and will only be kept in a
2253 temporary transaction. If a transaction with the same filename already exists,
2254 the previous CRL file entry will be deleted and replaced by the new one.
2255 Once the modifications are done, you have to commit the transaction through
2256 a "commit ssl crl-file" call.
2257
2258 Example:
2259 echo -e "set ssl crl-file crlfile.pem <<\n$(cat rootCRL.pem)\n" | \
2260 socat /var/run/haproxy.stat -
2261 echo "commit ssl crl-file crlfile.pem" | socat /var/run/haproxy.stat -
2262
Aurélien Nephtali1e0867c2018-04-18 14:04:58 +02002263set ssl ocsp-response <response | payload>
Willy Tarreau44aed902015-10-13 14:45:29 +02002264 This command is used to update an OCSP Response for a certificate (see "crt"
2265 on "bind" lines). Same controls are performed as during the initial loading of
2266 the response. The <response> must be passed as a base64 encoded string of the
Emmanuel Hocdet2c32d8f2017-05-22 14:58:00 +02002267 DER encoded response from the OCSP server. This command is not supported with
2268 BoringSSL.
Willy Tarreau44aed902015-10-13 14:45:29 +02002269
2270 Example:
2271 openssl ocsp -issuer issuer.pem -cert server.pem \
2272 -host ocsp.issuer.com:80 -respout resp.der
2273 echo "set ssl ocsp-response $(base64 -w 10000 resp.der)" | \
2274 socat stdio /var/run/haproxy.stat
2275
Aurélien Nephtali1e0867c2018-04-18 14:04:58 +02002276 using the payload syntax:
2277 echo -e "set ssl ocsp-response <<\n$(base64 resp.der)\n" | \
2278 socat stdio /var/run/haproxy.stat
2279
Willy Tarreau44aed902015-10-13 14:45:29 +02002280set ssl tls-key <id> <tlskey>
2281 Set the next TLS key for the <id> listener to <tlskey>. This key becomes the
2282 ultimate key, while the penultimate one is used for encryption (others just
2283 decrypt). The oldest TLS key present is overwritten. <id> is either a numeric
2284 #<id> or <file> returned by "show tls-keys". <tlskey> is a base64 encoded 48
Emeric Brun9e754772019-01-10 17:51:55 +01002285 or 80 bits TLS ticket key (ex. openssl rand 80 | openssl base64 -A).
Willy Tarreau44aed902015-10-13 14:45:29 +02002286
2287set table <table> key <key> [data.<data_type> <value>]*
2288 Create or update a stick-table entry in the table. If the key is not present,
2289 an entry is inserted. See stick-table in section 4.2 to find all possible
2290 values for <data_type>. The most likely use consists in dynamically entering
2291 entries for source IP addresses, with a flag in gpc0 to dynamically block an
2292 IP address or affect its quality of service. It is possible to pass multiple
2293 data_types in a single call.
2294
2295set timeout cli <delay>
2296 Change the CLI interface timeout for current connection. This can be useful
2297 during long debugging sessions where the user needs to constantly inspect
2298 some indicators without being disconnected. The delay is passed in seconds.
2299
Willy Tarreau4000ff02021-04-30 14:45:53 +02002300set var <name> <expression>
Willy Tarreaue93bff42021-09-03 09:47:37 +02002301set var <name> expr <expression>
2302set var <name> fmt <format>
Willy Tarreau4000ff02021-04-30 14:45:53 +02002303 Allows to set or overwrite the process-wide variable 'name' with the result
Willy Tarreaue93bff42021-09-03 09:47:37 +02002304 of expression <expression> or format string <format>. Only process-wide
2305 variables may be used, so the name must begin with 'proc.' otherwise no
2306 variable will be set. The <expression> and <format> may only involve
2307 "internal" sample fetch keywords and converters even though the most likely
2308 useful ones will be str('something'), int(), simple strings or references to
2309 other variables. Note that the command line parser doesn't know about quotes,
2310 so any space in the expression must be preceded by a backslash. This command
2311 requires levels "operator" or "admin". This command is only supported on a
2312 CLI connection running in experimental mode (see "experimental-mode on").
Willy Tarreau4000ff02021-04-30 14:45:53 +02002313
Willy Tarreau44aed902015-10-13 14:45:29 +02002314set weight <backend>/<server> <weight>[%]
2315 Change a server's weight to the value passed in argument. If the value ends
2316 with the '%' sign, then the new weight will be relative to the initially
2317 configured weight. Absolute weights are permitted between 0 and 256.
2318 Relative weights must be positive with the resulting absolute weight is
2319 capped at 256. Servers which are part of a farm running a static
2320 load-balancing algorithm have stricter limitations because the weight
2321 cannot change once set. Thus for these servers, the only accepted values
2322 are 0 and 100% (or 0 and the initial weight). Changes take effect
2323 immediately, though certain LB algorithms require a certain amount of
2324 requests to consider changes. A typical usage of this command is to
2325 disable a server during an update by setting its weight to zero, then to
2326 enable it again after the update by setting it back to 100%. This command
2327 is restricted and can only be issued on sockets configured for level
2328 "admin". Both the backend and the server may be specified either by their
2329 name or by their numeric ID, prefixed with a sharp ('#').
2330
Willy Tarreau95f753e2021-04-30 12:09:54 +02002331show acl [[@<ver>] <acl>]
Willy Tarreaud6129fc2017-07-28 16:52:23 +02002332 Dump info about acl converters. Without argument, the list of all available
Willy Tarreau95f753e2021-04-30 12:09:54 +02002333 acls is returned. If a <acl> is specified, its contents are dumped. <acl> is
2334 the #<id> or <file>. By default the current version of the ACL is shown (the
2335 version currently being matched against and reported as 'curr_ver' in the ACL
2336 list). It is possible to instead dump other versions by prepending '@<ver>'
2337 before the ACL's identifier. The version works as a filter and non-existing
2338 versions will simply report no result. The dump format is the same as for the
2339 maps even for the sample values. The data returned are not a list of
2340 available ACL, but are the list of all patterns composing any ACL. Many of
Dragan Dosena75eea72021-05-21 16:59:15 +02002341 these patterns can be shared with maps. The 'entry_cnt' value represents the
2342 count of all the ACL entries, not just the active ones, which means that it
2343 also includes entries currently being added.
Willy Tarreaud6129fc2017-07-28 16:52:23 +02002344
2345show backend
2346 Dump the list of backends available in the running process
2347
William Lallemand67a234f2018-12-13 09:05:45 +01002348show cli level
2349 Display the CLI level of the current CLI session. The result could be
2350 'admin', 'operator' or 'user'. See also the 'operator' and 'user' commands.
2351
2352 Example :
2353
2354 $ socat /tmp/sock1 readline
2355 prompt
2356 > operator
2357 > show cli level
2358 operator
2359 > user
2360 > show cli level
2361 user
2362 > operator
2363 Permission denied
2364
2365operator
2366 Decrease the CLI level of the current CLI session to operator. It can't be
Amaury Denoyelle18487fb2021-03-18 15:32:53 +01002367 increased. It also drops expert and experimental mode. See also "show cli
2368 level".
William Lallemand67a234f2018-12-13 09:05:45 +01002369
2370user
2371 Decrease the CLI level of the current CLI session to user. It can't be
Amaury Denoyelle18487fb2021-03-18 15:32:53 +01002372 increased. It also drops expert and experimental mode. See also "show cli
2373 level".
William Lallemand67a234f2018-12-13 09:05:45 +01002374
Willy Tarreau4c356932019-05-16 17:39:32 +02002375show activity
2376 Reports some counters about internal events that will help developers and
2377 more generally people who know haproxy well enough to narrow down the causes
2378 of reports of abnormal behaviours. A typical example would be a properly
2379 running process never sleeping and eating 100% of the CPU. The output fields
2380 will be made of one line per metric, and per-thread counters on the same
Thayne McCombscdbcca92021-01-07 21:24:41 -07002381 line. These counters are 32-bit and will wrap during the process's life, which
Willy Tarreau4c356932019-05-16 17:39:32 +02002382 is not a problem since calls to this command will typically be performed
2383 twice. The fields are purposely not documented so that their exact meaning is
2384 verified in the code where the counters are fed. These values are also reset
2385 by the "clear counters" command.
2386
William Lallemand51132162016-12-16 16:38:58 +01002387show cli sockets
2388 List CLI sockets. The output format is composed of 3 fields separated by
2389 spaces. The first field is the socket address, it can be a unix socket, a
2390 ipv4 address:port couple or a ipv6 one. Socket of other types won't be dump.
2391 The second field describe the level of the socket: 'admin', 'user' or
2392 'operator'. The last field list the processes on which the socket is bound,
2393 separated by commas, it can be numbers or 'all'.
2394
2395 Example :
2396
2397 $ echo 'show cli sockets' | socat stdio /tmp/sock1
2398 # socket lvl processes
2399 /tmp/sock1 admin all
2400 127.0.0.1:9999 user 2,3,4
2401 127.0.0.2:9969 user 2
2402 [::1]:9999 operator 2
2403
William Lallemand86d0df02017-11-24 21:36:45 +01002404show cache
Cyril Bonté7b888f12017-11-26 22:24:31 +01002405 List the configured caches and the objects stored in each cache tree.
William Lallemand86d0df02017-11-24 21:36:45 +01002406
2407 $ echo 'show cache' | socat stdio /tmp/sock1
2408 0x7f6ac6c5b03a: foobar (shctx:0x7f6ac6c5b000, available blocks:3918)
2409 1 2 3 4
2410
2411 1. pointer to the cache structure
2412 2. cache name
2413 3. pointer to the mmap area (shctx)
2414 4. number of blocks available for reuse in the shctx
2415
Remi Tricot-Le Bretone3e1e5f2020-11-27 15:48:40 +01002416 0x7f6ac6c5b4cc hash:286881868 vary:0x0011223344556677 size:39114 (39 blocks), refcount:9, expire:237
2417 1 2 3 4 5 6 7
William Lallemand86d0df02017-11-24 21:36:45 +01002418
2419 1. pointer to the cache entry
2420 2. first 32 bits of the hash
Remi Tricot-Le Bretone3e1e5f2020-11-27 15:48:40 +01002421 3. secondary hash of the entry in case of vary
2422 4. size of the object in bytes
2423 5. number of blocks used for the object
2424 6. number of transactions using the entry
2425 7. expiration time, can be negative if already expired
William Lallemand86d0df02017-11-24 21:36:45 +01002426
Willy Tarreauae795722016-02-16 11:27:28 +01002427show env [<name>]
2428 Dump one or all environment variables known by the process. Without any
2429 argument, all variables are dumped. With an argument, only the specified
2430 variable is dumped if it exists. Otherwise "Variable not found" is emitted.
2431 Variables are dumped in the same format as they are stored or returned by the
2432 "env" utility, that is, "<name>=<value>". This can be handy when debugging
2433 certain configuration files making heavy use of environment variables to
2434 ensure that they contain the expected values. This command is restricted and
2435 can only be issued on sockets configured for levels "operator" or "admin".
2436
Willy Tarreau35069f82016-11-25 09:16:37 +01002437show errors [<iid>|<proxy>] [request|response]
Willy Tarreau44aed902015-10-13 14:45:29 +02002438 Dump last known request and response errors collected by frontends and
2439 backends. If <iid> is specified, the limit the dump to errors concerning
Willy Tarreau234ba2d2016-11-25 08:39:10 +01002440 either frontend or backend whose ID is <iid>. Proxy ID "-1" will cause
2441 all instances to be dumped. If a proxy name is specified instead, its ID
Willy Tarreau35069f82016-11-25 09:16:37 +01002442 will be used as the filter. If "request" or "response" is added after the
2443 proxy name or ID, only request or response errors will be dumped. This
2444 command is restricted and can only be issued on sockets configured for
2445 levels "operator" or "admin".
Willy Tarreau44aed902015-10-13 14:45:29 +02002446
2447 The errors which may be collected are the last request and response errors
2448 caused by protocol violations, often due to invalid characters in header
2449 names. The report precisely indicates what exact character violated the
2450 protocol. Other important information such as the exact date the error was
2451 detected, frontend and backend names, the server name (when known), the
2452 internal session ID and the source address which has initiated the session
2453 are reported too.
2454
2455 All characters are returned, and non-printable characters are encoded. The
2456 most common ones (\t = 9, \n = 10, \r = 13 and \e = 27) are encoded as one
2457 letter following a backslash. The backslash itself is encoded as '\\' to
2458 avoid confusion. Other non-printable characters are encoded '\xNN' where
2459 NN is the two-digits hexadecimal representation of the character's ASCII
2460 code.
2461
2462 Lines are prefixed with the position of their first character, starting at 0
2463 for the beginning of the buffer. At most one input line is printed per line,
2464 and large lines will be broken into multiple consecutive output lines so that
2465 the output never goes beyond 79 characters wide. It is easy to detect if a
2466 line was broken, because it will not end with '\n' and the next line's offset
2467 will be followed by a '+' sign, indicating it is a continuation of previous
2468 line.
2469
2470 Example :
Willy Tarreau35069f82016-11-25 09:16:37 +01002471 $ echo "show errors -1 response" | socat stdio /tmp/sock1
Willy Tarreau44aed902015-10-13 14:45:29 +02002472 >>> [04/Mar/2009:15:46:56.081] backend http-in (#2) : invalid response
2473 src 127.0.0.1, session #54, frontend fe-eth0 (#1), server s2 (#1)
2474 response length 213 bytes, error at position 23:
2475
2476 00000 HTTP/1.0 200 OK\r\n
2477 00017 header/bizarre:blah\r\n
2478 00038 Location: blah\r\n
2479 00054 Long-line: this is a very long line which should b
2480 00104+ e broken into multiple lines on the output buffer,
2481 00154+ otherwise it would be too large to print in a ter
2482 00204+ minal\r\n
2483 00211 \r\n
2484
2485 In the example above, we see that the backend "http-in" which has internal
2486 ID 2 has blocked an invalid response from its server s2 which has internal
2487 ID 1. The request was on session 54 initiated by source 127.0.0.1 and
2488 received by frontend fe-eth0 whose ID is 1. The total response length was
2489 213 bytes when the error was detected, and the error was at byte 23. This
2490 is the slash ('/') in header name "header/bizarre", which is not a valid
2491 HTTP character for a header name.
2492
Willy Tarreau1d181e42019-08-30 11:17:01 +02002493show events [<sink>] [-w] [-n]
Willy Tarreau9f830d72019-08-26 18:17:04 +02002494 With no option, this lists all known event sinks and their types. With an
2495 option, it will dump all available events in the designated sink if it is of
Willy Tarreau1d181e42019-08-30 11:17:01 +02002496 type buffer. If option "-w" is passed after the sink name, then once the end
2497 of the buffer is reached, the command will wait for new events and display
2498 them. It is possible to stop the operation by entering any input (which will
2499 be discarded) or by closing the session. Finally, option "-n" is used to
2500 directly seek to the end of the buffer, which is often convenient when
2501 combined with "-w" to only report new events. For convenience, "-wn" or "-nw"
2502 may be used to enable both options at once.
Willy Tarreau9f830d72019-08-26 18:17:04 +02002503
Willy Tarreau7a4a0ac2017-07-25 19:32:50 +02002504show fd [<fd>]
2505 Dump the list of either all open file descriptors or just the one number <fd>
2506 if specified. This is only aimed at developers who need to observe internal
2507 states in order to debug complex issues such as abnormal CPU usages. One fd
2508 is reported per lines, and for each of them, its state in the poller using
2509 upper case letters for enabled flags and lower case for disabled flags, using
2510 "P" for "polled", "R" for "ready", "A" for "active", the events status using
2511 "H" for "hangup", "E" for "error", "O" for "output", "P" for "priority" and
2512 "I" for "input", a few other flags like "N" for "new" (just added into the fd
2513 cache), "U" for "updated" (received an update in the fd cache), "L" for
2514 "linger_risk", "C" for "cloned", then the cached entry position, the pointer
2515 to the internal owner, the pointer to the I/O callback and its name when
2516 known. When the owner is a connection, the connection flags, and the target
2517 are reported (frontend, proxy or server). When the owner is a listener, the
2518 listener's state and its frontend are reported. There is no point in using
2519 this command without a good knowledge of the internals. It's worth noting
2520 that the output format may evolve over time so this output must not be parsed
Willy Tarreau8050efe2021-01-21 08:26:06 +01002521 by tools designed to be durable. Some internal structure states may look
2522 suspicious to the function listing them, in this case the output line will be
2523 suffixed with an exclamation mark ('!'). This may help find a starting point
2524 when trying to diagnose an incident.
Willy Tarreau7a4a0ac2017-07-25 19:32:50 +02002525
Willy Tarreau27456202021-05-08 07:54:24 +02002526show info [typed|json] [desc] [float]
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002527 Dump info about haproxy status on current process. If "typed" is passed as an
2528 optional argument, field numbers, names and types are emitted as well so that
2529 external monitoring products can easily retrieve, possibly aggregate, then
2530 report information found in fields they don't know. Each field is dumped on
Simon Horman05ee2132017-01-04 09:37:25 +01002531 its own line. If "json" is passed as an optional argument then
2532 information provided by "typed" output is provided in JSON format as a
2533 list of JSON objects. By default, the format contains only two columns
2534 delimited by a colon (':'). The left one is the field name and the right
2535 one is the value. It is very important to note that in typed output
2536 format, the dump for a single object is contiguous so that there is no
Willy Tarreau27456202021-05-08 07:54:24 +02002537 need for a consumer to store everything at once. If "float" is passed as an
2538 optional argument, some fields usually emitted as integers may switch to
2539 floats for higher accuracy. It is purposely unspecified which ones are
2540 concerned as this might evolve over time. Using this option implies that the
2541 consumer is able to process floats. The output format used is sprintf("%f").
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002542
2543 When using the typed output format, each line is made of 4 columns delimited
2544 by colons (':'). The first column is a dot-delimited series of 3 elements. The
2545 first element is the numeric position of the field in the list (starting at
2546 zero). This position shall not change over time, but holes are to be expected,
2547 depending on build options or if some fields are deleted in the future. The
2548 second element is the field name as it appears in the default "show info"
2549 output. The third element is the relative process number starting at 1.
2550
2551 The rest of the line starting after the first colon follows the "typed output
2552 format" described in the section above. In short, the second column (after the
2553 first ':') indicates the origin, nature and scope of the variable. The third
2554 column indicates the type of the field, among "s32", "s64", "u32", "u64" and
2555 "str". Then the fourth column is the value itself, which the consumer knows
2556 how to parse thanks to column 3 and how to process thanks to column 2.
2557
2558 Thus the overall line format in typed mode is :
2559
2560 <field_pos>.<field_name>.<process_num>:<tags>:<type>:<value>
2561
Willy Tarreau6b19b142019-10-09 15:44:21 +02002562 When "desc" is appended to the command, one extra colon followed by a quoted
2563 string is appended with a description for the metric. At the time of writing,
2564 this is only supported for the "typed" and default output formats.
2565
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002566 Example :
2567
2568 > show info
2569 Name: HAProxy
2570 Version: 1.7-dev1-de52ea-146
2571 Release_date: 2016/03/11
2572 Nbproc: 1
2573 Process_num: 1
2574 Pid: 28105
2575 Uptime: 0d 0h00m04s
2576 Uptime_sec: 4
2577 Memmax_MB: 0
2578 PoolAlloc_MB: 0
2579 PoolUsed_MB: 0
2580 PoolFailed: 0
2581 (...)
2582
2583 > show info typed
2584 0.Name.1:POS:str:HAProxy
2585 1.Version.1:POS:str:1.7-dev1-de52ea-146
2586 2.Release_date.1:POS:str:2016/03/11
2587 3.Nbproc.1:CGS:u32:1
2588 4.Process_num.1:KGP:u32:1
2589 5.Pid.1:SGP:u32:28105
2590 6.Uptime.1:MDP:str:0d 0h00m08s
2591 7.Uptime_sec.1:MDP:u32:8
2592 8.Memmax_MB.1:CLP:u32:0
2593 9.PoolAlloc_MB.1:MGP:u32:0
2594 10.PoolUsed_MB.1:MGP:u32:0
2595 11.PoolFailed.1:MCP:u32:0
2596 (...)
2597
Simon Horman1084a362016-11-21 17:00:24 +01002598 In the typed format, the presence of the process ID at the end of the
2599 first column makes it very easy to visually aggregate outputs from
2600 multiple processes.
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002601 Example :
2602
2603 $ ( echo show info typed | socat /var/run/haproxy.sock1 ; \
2604 echo show info typed | socat /var/run/haproxy.sock2 ) | \
2605 sort -t . -k 1,1n -k 2,2 -k 3,3n
2606 0.Name.1:POS:str:HAProxy
2607 0.Name.2:POS:str:HAProxy
2608 1.Version.1:POS:str:1.7-dev1-868ab3-148
2609 1.Version.2:POS:str:1.7-dev1-868ab3-148
2610 2.Release_date.1:POS:str:2016/03/11
2611 2.Release_date.2:POS:str:2016/03/11
2612 3.Nbproc.1:CGS:u32:2
2613 3.Nbproc.2:CGS:u32:2
2614 4.Process_num.1:KGP:u32:1
2615 4.Process_num.2:KGP:u32:2
2616 5.Pid.1:SGP:u32:30120
2617 5.Pid.2:SGP:u32:30121
2618 6.Uptime.1:MDP:str:0d 0h01m28s
2619 6.Uptime.2:MDP:str:0d 0h01m28s
2620 (...)
Willy Tarreau44aed902015-10-13 14:45:29 +02002621
Simon Horman05ee2132017-01-04 09:37:25 +01002622 The format of JSON output is described in a schema which may be output
Simon Horman6f6bb382017-01-04 09:37:26 +01002623 using "show schema json".
Simon Horman05ee2132017-01-04 09:37:25 +01002624
2625 The JSON output contains no extra whitespace in order to reduce the
2626 volume of output. For human consumption passing the output through a
2627 pretty printer may be helpful. Example :
2628
2629 $ echo "show info json" | socat /var/run/haproxy.sock stdio | \
2630 python -m json.tool
2631
Simon Horman6f6bb382017-01-04 09:37:26 +01002632 The JSON output contains no extra whitespace in order to reduce the
2633 volume of output. For human consumption passing the output through a
2634 pretty printer may be helpful. Example :
2635
2636 $ echo "show info json" | socat /var/run/haproxy.sock stdio | \
2637 python -m json.tool
2638
Willy Tarreau6ab7b212021-12-28 09:57:10 +01002639show libs
2640 Dump the list of loaded shared dynamic libraries and object files, on systems
2641 that support it. When available, for each shared object the range of virtual
2642 addresses will be indicated, the size and the path to the object. This can be
2643 used for example to try to estimate what library provides a function that
2644 appears in a dump. Note that on many systems, addresses will change upon each
2645 restart (address space randomization), so that this list would need to be
2646 retrieved upon startup if it is expected to be used to analyse a core file.
2647 This command may only be issued on sockets configured for levels "operator"
2648 or "admin". Note that the output format may vary between operating systems,
2649 architectures and even haproxy versions, and ought not to be relied on in
2650 scripts.
2651
Willy Tarreau95f753e2021-04-30 12:09:54 +02002652show map [[@<ver>] <map>]
Willy Tarreau44aed902015-10-13 14:45:29 +02002653 Dump info about map converters. Without argument, the list of all available
2654 maps is returned. If a <map> is specified, its contents are dumped. <map> is
Willy Tarreau95f753e2021-04-30 12:09:54 +02002655 the #<id> or <file>. By default the current version of the map is shown (the
2656 version currently being matched against and reported as 'curr_ver' in the map
2657 list). It is possible to instead dump other versions by prepending '@<ver>'
2658 before the map's identifier. The version works as a filter and non-existing
Dragan Dosena75eea72021-05-21 16:59:15 +02002659 versions will simply report no result. The 'entry_cnt' value represents the
2660 count of all the map entries, not just the active ones, which means that it
2661 also includes entries currently being added.
Willy Tarreau95f753e2021-04-30 12:09:54 +02002662
2663 In the output, the first column is a unique entry identifier, which is usable
2664 as a reference for operations "del map" and "set map". The second column is
Willy Tarreau44aed902015-10-13 14:45:29 +02002665 the pattern and the third column is the sample if available. The data returned
2666 are not directly a list of available maps, but are the list of all patterns
2667 composing any map. Many of these patterns can be shared with ACL.
2668
Willy Tarreau49962b52021-02-12 16:56:22 +01002669show peers [dict|-] [<peers section>]
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002670 Dump info about the peers configured in "peers" sections. Without argument,
2671 the list of the peers belonging to all the "peers" sections are listed. If
2672 <peers section> is specified, only the information about the peers belonging
Willy Tarreau49962b52021-02-12 16:56:22 +01002673 to this "peers" section are dumped. When "dict" is specified before the peers
2674 section name, the entire Tx/Rx dictionary caches will also be dumped (very
2675 large). Passing "-" may be required to dump a peers section called "dict".
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002676
Michael Prokop4438c602019-05-24 10:25:45 +02002677 Here are two examples of outputs where hostA, hostB and hostC peers belong to
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002678 "sharedlb" peers sections. Only hostA and hostB are connected. Only hostA has
2679 sent data to hostB.
2680
2681 $ echo "show peers" | socat - /tmp/hostA
2682 0x55deb0224320: [15/Apr/2019:11:28:01] id=sharedlb state=0 flags=0x3 \
Emeric Brun0bbec0f2019-04-18 11:39:43 +02002683 resync_timeout=<PAST> task_calls=45122
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002684 0x55deb022b540: id=hostC(remote) addr=127.0.0.12:10002 status=CONN \
2685 reconnect=4s confirm=0
2686 flags=0x0
2687 0x55deb022a440: id=hostA(local) addr=127.0.0.10:10000 status=NONE \
2688 reconnect=<NEVER> confirm=0
2689 flags=0x0
2690 0x55deb0227d70: id=hostB(remote) addr=127.0.0.11:10001 status=ESTA
2691 reconnect=2s confirm=0
Emeric Brun0bbec0f2019-04-18 11:39:43 +02002692 flags=0x20000200 appctx:0x55deb028fba0 st0=7 st1=0 task_calls=14456 \
2693 state=EST
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002694 xprt=RAW src=127.0.0.1:37257 addr=127.0.0.10:10000
2695 remote_table:0x55deb0224a10 id=stkt local_id=1 remote_id=1
2696 last_local_table:0x55deb0224a10 id=stkt local_id=1 remote_id=1
2697 shared tables:
2698 0x55deb0224a10 local_id=1 remote_id=1 flags=0x0 remote_data=0x65
2699 last_acked=0 last_pushed=3 last_get=0 teaching_origin=0 update=3
2700 table:0x55deb022d6a0 id=stkt update=3 localupdate=3 \
2701 commitupdate=3 syncing=0
2702
2703 $ echo "show peers" | socat - /tmp/hostB
2704 0x55871b5ab320: [15/Apr/2019:11:28:03] id=sharedlb state=0 flags=0x3 \
Emeric Brun0bbec0f2019-04-18 11:39:43 +02002705 resync_timeout=<PAST> task_calls=3
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002706 0x55871b5b2540: id=hostC(remote) addr=127.0.0.12:10002 status=CONN \
2707 reconnect=3s confirm=0
2708 flags=0x0
2709 0x55871b5b1440: id=hostB(local) addr=127.0.0.11:10001 status=NONE \
2710 reconnect=<NEVER> confirm=0
2711 flags=0x0
2712 0x55871b5aed70: id=hostA(remote) addr=127.0.0.10:10000 status=ESTA \
2713 reconnect=2s confirm=0
Emeric Brun0bbec0f2019-04-18 11:39:43 +02002714 flags=0x20000200 appctx:0x7fa46800ee00 st0=7 st1=0 task_calls=62356 \
2715 state=EST
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002716 remote_table:0x55871b5ab960 id=stkt local_id=1 remote_id=1
2717 last_local_table:0x55871b5ab960 id=stkt local_id=1 remote_id=1
2718 shared tables:
2719 0x55871b5ab960 local_id=1 remote_id=1 flags=0x0 remote_data=0x65
2720 last_acked=3 last_pushed=0 last_get=3 teaching_origin=0 update=0
2721 table:0x55871b5b46a0 id=stkt update=1 localupdate=0 \
2722 commitupdate=0 syncing=0
2723
Willy Tarreau44aed902015-10-13 14:45:29 +02002724show pools
2725 Dump the status of internal memory pools. This is useful to track memory
2726 usage when suspecting a memory leak for example. It does exactly the same
2727 as the SIGQUIT when running in foreground except that it does not flush
2728 the pools.
2729
Willy Tarreauf1c8a382021-05-13 10:00:17 +02002730show profiling [{all | status | tasks | memory}] [byaddr] [<max_lines>]
Willy Tarreau75c62c22018-11-22 11:02:09 +01002731 Dumps the current profiling settings, one per line, as well as the command
Willy Tarreau1bd67e92021-01-29 00:07:40 +01002732 needed to change them. When tasks profiling is enabled, some per-function
2733 statistics collected by the scheduler will also be emitted, with a summary
Willy Tarreau42712cb2021-05-05 17:48:13 +02002734 covering the number of calls, total/avg CPU time and total/avg latency. When
2735 memory profiling is enabled, some information such as the number of
2736 allocations/releases and their sizes will be reported. It is possible to
2737 limit the dump to only the profiling status, the tasks, or the memory
2738 profiling by specifying the respective keywords; by default all profiling
2739 information are dumped. It is also possible to limit the number of lines
Willy Tarreauf1c8a382021-05-13 10:00:17 +02002740 of output of each category by specifying a numeric limit. If is possible to
2741 request that the output is sorted by address instead of usage, e.g. to ease
2742 comparisons between subsequent calls. Please note that profiling is
2743 essentially aimed at developers since it gives hints about where CPU cycles
2744 or memory are wasted in the code. There is nothing useful to monitor there.
Willy Tarreau75c62c22018-11-22 11:02:09 +01002745
Willy Tarreau87ef3232021-01-29 12:01:46 +01002746show resolvers [<resolvers section id>]
2747 Dump statistics for the given resolvers section, or all resolvers sections
2748 if no section is supplied.
2749
2750 For each name server, the following counters are reported:
2751 sent: number of DNS requests sent to this server
2752 valid: number of DNS valid responses received from this server
2753 update: number of DNS responses used to update the server's IP address
2754 cname: number of CNAME responses
2755 cname_error: CNAME errors encountered with this server
2756 any_err: number of empty response (IE: server does not support ANY type)
2757 nx: non existent domain response received from this server
2758 timeout: how many time this server did not answer in time
2759 refused: number of requests refused by this server
2760 other: any other DNS errors
2761 invalid: invalid DNS response (from a protocol point of view)
2762 too_big: too big response
2763 outdated: number of response arrived too late (after an other name server)
2764
Willy Tarreau69f591e2020-07-01 07:00:59 +02002765show servers conn [<backend>]
2766 Dump the current and idle connections state of the servers belonging to the
2767 designated backend (or all backends if none specified). A backend name or
2768 identifier may be used.
2769
2770 The output consists in a header line showing the fields titles, then one
2771 server per line with for each, the backend name and ID, server name and ID,
2772 the address, port and a series or values. The number of fields varies
2773 depending on thread count.
2774
2775 Given the threaded nature of idle connections, it's important to understand
2776 that some values may change once read, and that as such, consistency within a
2777 line isn't granted. This output is mostly provided as a debugging tool and is
2778 not relevant to be routinely monitored nor graphed.
2779
Willy Tarreau44aed902015-10-13 14:45:29 +02002780show servers state [<backend>]
2781 Dump the state of the servers found in the running configuration. A backend
2782 name or identifier may be provided to limit the output to this backend only.
2783
2784 The dump has the following format:
2785 - first line contains the format version (1 in this specification);
2786 - second line contains the column headers, prefixed by a sharp ('#');
2787 - third line and next ones contain data;
2788 - each line starting by a sharp ('#') is considered as a comment.
2789
Dan Lloyd8e48b872016-07-01 21:01:18 -04002790 Since multiple versions of the output may co-exist, below is the list of
Willy Tarreau44aed902015-10-13 14:45:29 +02002791 fields and their order per file format version :
2792 1:
2793 be_id: Backend unique id.
2794 be_name: Backend label.
2795 srv_id: Server unique id (in the backend).
2796 srv_name: Server label.
2797 srv_addr: Server IP address.
2798 srv_op_state: Server operational state (UP/DOWN/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002799 0 = SRV_ST_STOPPED
2800 The server is down.
2801 1 = SRV_ST_STARTING
2802 The server is warming up (up but
2803 throttled).
2804 2 = SRV_ST_RUNNING
2805 The server is fully up.
2806 3 = SRV_ST_STOPPING
2807 The server is up but soft-stopping
2808 (eg: 404).
Willy Tarreau44aed902015-10-13 14:45:29 +02002809 srv_admin_state: Server administrative state (MAINT/DRAIN/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002810 The state is actually a mask of values :
2811 0x01 = SRV_ADMF_FMAINT
2812 The server was explicitly forced into
2813 maintenance.
2814 0x02 = SRV_ADMF_IMAINT
2815 The server has inherited the maintenance
2816 status from a tracked server.
2817 0x04 = SRV_ADMF_CMAINT
2818 The server is in maintenance because of
2819 the configuration.
2820 0x08 = SRV_ADMF_FDRAIN
2821 The server was explicitly forced into
2822 drain state.
2823 0x10 = SRV_ADMF_IDRAIN
2824 The server has inherited the drain status
2825 from a tracked server.
Baptiste Assmann89aa7f32016-11-02 21:31:27 +01002826 0x20 = SRV_ADMF_RMAINT
2827 The server is in maintenance because of an
2828 IP address resolution failure.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02002829 0x40 = SRV_ADMF_HMAINT
2830 The server FQDN was set from stats socket.
2831
Willy Tarreau44aed902015-10-13 14:45:29 +02002832 srv_uweight: User visible server's weight.
2833 srv_iweight: Server's initial weight.
2834 srv_time_since_last_change: Time since last operational change.
2835 srv_check_status: Last health check status.
2836 srv_check_result: Last check result (FAILED/PASSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002837 0 = CHK_RES_UNKNOWN
2838 Initialized to this by default.
2839 1 = CHK_RES_NEUTRAL
2840 Valid check but no status information.
2841 2 = CHK_RES_FAILED
2842 Check failed.
2843 3 = CHK_RES_PASSED
2844 Check succeeded and server is fully up
2845 again.
2846 4 = CHK_RES_CONDPASS
2847 Check reports the server doesn't want new
2848 sessions.
Willy Tarreau44aed902015-10-13 14:45:29 +02002849 srv_check_health: Checks rise / fall current counter.
2850 srv_check_state: State of the check (ENABLED/PAUSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002851 The state is actually a mask of values :
2852 0x01 = CHK_ST_INPROGRESS
2853 A check is currently running.
2854 0x02 = CHK_ST_CONFIGURED
2855 This check is configured and may be
2856 enabled.
2857 0x04 = CHK_ST_ENABLED
2858 This check is currently administratively
2859 enabled.
2860 0x08 = CHK_ST_PAUSED
2861 Checks are paused because of maintenance
2862 (health only).
Willy Tarreau44aed902015-10-13 14:45:29 +02002863 srv_agent_state: State of the agent check (ENABLED/PAUSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002864 This state uses the same mask values as
2865 "srv_check_state", adding this specific one :
2866 0x10 = CHK_ST_AGENT
2867 Check is an agent check (otherwise it's a
2868 health check).
Willy Tarreau44aed902015-10-13 14:45:29 +02002869 bk_f_forced_id: Flag to know if the backend ID is forced by
2870 configuration.
2871 srv_f_forced_id: Flag to know if the server's ID is forced by
2872 configuration.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02002873 srv_fqdn: Server FQDN.
Frédéric Lécaille31694712017-08-01 08:47:19 +02002874 srv_port: Server port.
Baptiste Assmann6d0f38f2018-07-02 17:00:54 +02002875 srvrecord: DNS SRV record associated to this SRV.
William Dauchyf6370442020-11-14 19:25:33 +01002876 srv_use_ssl: use ssl for server connections.
William Dauchyd1a7b852021-02-11 22:51:26 +01002877 srv_check_port: Server health check port.
2878 srv_check_addr: Server health check address.
2879 srv_agent_addr: Server health agent address.
2880 srv_agent_port: Server health agent port.
Willy Tarreau44aed902015-10-13 14:45:29 +02002881
2882show sess
2883 Dump all known sessions. Avoid doing this on slow connections as this can
2884 be huge. This command is restricted and can only be issued on sockets
Willy Tarreauc6e7a1b2020-06-28 01:24:12 +02002885 configured for levels "operator" or "admin". Note that on machines with
2886 quickly recycled connections, it is possible that this output reports less
2887 entries than really exist because it will dump all existing sessions up to
2888 the last one that was created before the command was entered; those which
2889 die in the mean time will not appear.
Willy Tarreau44aed902015-10-13 14:45:29 +02002890
2891show sess <id>
2892 Display a lot of internal information about the specified session identifier.
2893 This identifier is the first field at the beginning of the lines in the dumps
2894 of "show sess" (it corresponds to the session pointer). Those information are
2895 useless to most users but may be used by haproxy developers to troubleshoot a
2896 complex bug. The output format is intentionally not documented so that it can
2897 freely evolve depending on demands. You may find a description of all fields
2898 returned in src/dumpstats.c
2899
2900 The special id "all" dumps the states of all sessions, which must be avoided
2901 as much as possible as it is highly CPU intensive and can take a lot of time.
2902
Daniel Corbettc40edac2020-11-01 10:54:17 -05002903show stat [domain <dns|proxy>] [{<iid>|<proxy>} <type> <sid>] [typed|json] \
Willy Tarreau698097b2020-10-23 20:19:47 +02002904 [desc] [up|no-maint]
Daniel Corbettc40edac2020-11-01 10:54:17 -05002905 Dump statistics. The domain is used to select which statistics to print; dns
2906 and proxy are available for now. By default, the CSV format is used; you can
Amaury Denoyelle072f97e2020-10-05 11:49:37 +02002907 activate the extended typed output format described in the section above if
2908 "typed" is passed after the other arguments; or in JSON if "json" is passed
2909 after the other arguments. By passing <id>, <type> and <sid>, it is possible
2910 to dump only selected items :
Willy Tarreaua1b1ed52016-11-25 08:50:58 +01002911 - <iid> is a proxy ID, -1 to dump everything. Alternatively, a proxy name
2912 <proxy> may be specified. In this case, this proxy's ID will be used as
2913 the ID selector.
Willy Tarreau44aed902015-10-13 14:45:29 +02002914 - <type> selects the type of dumpable objects : 1 for frontends, 2 for
2915 backends, 4 for servers, -1 for everything. These values can be ORed,
2916 for example:
2917 1 + 2 = 3 -> frontend + backend.
2918 1 + 2 + 4 = 7 -> frontend + backend + server.
2919 - <sid> is a server ID, -1 to dump everything from the selected proxy.
2920
2921 Example :
2922 $ echo "show info;show stat" | socat stdio unix-connect:/tmp/sock1
2923 >>> Name: HAProxy
2924 Version: 1.4-dev2-49
2925 Release_date: 2009/09/23
2926 Nbproc: 1
2927 Process_num: 1
2928 (...)
2929
2930 # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq, (...)
2931 stats,FRONTEND,,,0,0,1000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,1,0, (...)
2932 stats,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,250,(...)
2933 (...)
2934 www1,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,0,250, (...)
2935
2936 $
2937
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002938 In this example, two commands have been issued at once. That way it's easy to
2939 find which process the stats apply to in multi-process mode. This is not
2940 needed in the typed output format as the process number is reported on each
2941 line. Notice the empty line after the information output which marks the end
2942 of the first block. A similar empty line appears at the end of the second
2943 block (stats) so that the reader knows the output has not been truncated.
2944
2945 When "typed" is specified, the output format is more suitable to monitoring
2946 tools because it provides numeric positions and indicates the type of each
2947 output field. Each value stands on its own line with process number, element
2948 number, nature, origin and scope. This same format is available via the HTTP
2949 stats by passing ";typed" after the URI. It is very important to note that in
Dan Lloyd8e48b872016-07-01 21:01:18 -04002950 typed output format, the dump for a single object is contiguous so that there
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002951 is no need for a consumer to store everything at once.
2952
Willy Tarreau698097b2020-10-23 20:19:47 +02002953 The "up" modifier will result in listing only servers which reportedly up or
2954 not checked. Those down, unresolved, or in maintenance will not be listed.
2955 This is analogous to the ";up" option on the HTTP stats. Similarly, the
2956 "no-maint" modifier will act like the ";no-maint" HTTP modifier and will
2957 result in disabled servers not to be listed. The difference is that those
2958 which are enabled but down will not be evicted.
2959
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002960 When using the typed output format, each line is made of 4 columns delimited
2961 by colons (':'). The first column is a dot-delimited series of 5 elements. The
2962 first element is a letter indicating the type of the object being described.
2963 At the moment the following object types are known : 'F' for a frontend, 'B'
2964 for a backend, 'L' for a listener, and 'S' for a server. The second element
2965 The second element is a positive integer representing the unique identifier of
2966 the proxy the object belongs to. It is equivalent to the "iid" column of the
2967 CSV output and matches the value in front of the optional "id" directive found
2968 in the frontend or backend section. The third element is a positive integer
2969 containing the unique object identifier inside the proxy, and corresponds to
2970 the "sid" column of the CSV output. ID 0 is reported when dumping a frontend
2971 or a backend. For a listener or a server, this corresponds to their respective
2972 ID inside the proxy. The fourth element is the numeric position of the field
2973 in the list (starting at zero). This position shall not change over time, but
2974 holes are to be expected, depending on build options or if some fields are
2975 deleted in the future. The fifth element is the field name as it appears in
2976 the CSV output. The sixth element is a positive integer and is the relative
2977 process number starting at 1.
2978
2979 The rest of the line starting after the first colon follows the "typed output
2980 format" described in the section above. In short, the second column (after the
2981 first ':') indicates the origin, nature and scope of the variable. The third
Willy Tarreau589722e2021-05-08 07:46:44 +02002982 column indicates the field type, among "s32", "s64", "u32", "u64", "flt' and
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002983 "str". Then the fourth column is the value itself, which the consumer knows
2984 how to parse thanks to column 3 and how to process thanks to column 2.
2985
Willy Tarreau6b19b142019-10-09 15:44:21 +02002986 When "desc" is appended to the command, one extra colon followed by a quoted
2987 string is appended with a description for the metric. At the time of writing,
2988 this is only supported for the "typed" output format.
2989
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002990 Thus the overall line format in typed mode is :
2991
2992 <obj>.<px_id>.<id>.<fpos>.<fname>.<process_num>:<tags>:<type>:<value>
2993
2994 Here's an example of typed output format :
2995
2996 $ echo "show stat typed" | socat stdio unix-connect:/tmp/sock1
2997 F.2.0.0.pxname.1:MGP:str:private-frontend
2998 F.2.0.1.svname.1:MGP:str:FRONTEND
2999 F.2.0.8.bin.1:MGP:u64:0
3000 F.2.0.9.bout.1:MGP:u64:0
3001 F.2.0.40.hrsp_2xx.1:MGP:u64:0
3002 L.2.1.0.pxname.1:MGP:str:private-frontend
3003 L.2.1.1.svname.1:MGP:str:sock-1
3004 L.2.1.17.status.1:MGP:str:OPEN
3005 L.2.1.73.addr.1:MGP:str:0.0.0.0:8001
3006 S.3.13.60.rtime.1:MCP:u32:0
3007 S.3.13.61.ttime.1:MCP:u32:0
3008 S.3.13.62.agent_status.1:MGP:str:L4TOUT
3009 S.3.13.64.agent_duration.1:MGP:u64:2001
3010 S.3.13.65.check_desc.1:MCP:str:Layer4 timeout
3011 S.3.13.66.agent_desc.1:MCP:str:Layer4 timeout
3012 S.3.13.67.check_rise.1:MCP:u32:2
3013 S.3.13.68.check_fall.1:MCP:u32:3
3014 S.3.13.69.check_health.1:SGP:u32:0
3015 S.3.13.70.agent_rise.1:MaP:u32:1
3016 S.3.13.71.agent_fall.1:SGP:u32:1
3017 S.3.13.72.agent_health.1:SGP:u32:1
3018 S.3.13.73.addr.1:MCP:str:1.255.255.255:8888
3019 S.3.13.75.mode.1:MAP:str:http
3020 B.3.0.0.pxname.1:MGP:str:private-backend
3021 B.3.0.1.svname.1:MGP:str:BACKEND
3022 B.3.0.2.qcur.1:MGP:u32:0
3023 B.3.0.3.qmax.1:MGP:u32:0
3024 B.3.0.4.scur.1:MGP:u32:0
3025 B.3.0.5.smax.1:MGP:u32:0
3026 B.3.0.6.slim.1:MGP:u32:1000
3027 B.3.0.55.lastsess.1:MMP:s32:-1
3028 (...)
3029
Simon Horman1084a362016-11-21 17:00:24 +01003030 In the typed format, the presence of the process ID at the end of the
3031 first column makes it very easy to visually aggregate outputs from
3032 multiple processes, as show in the example below where each line appears
3033 for each process :
Willy Tarreau5d8b9792016-03-11 11:09:34 +01003034
3035 $ ( echo show stat typed | socat /var/run/haproxy.sock1 - ; \
3036 echo show stat typed | socat /var/run/haproxy.sock2 - ) | \
3037 sort -t . -k 1,1 -k 2,2n -k 3,3n -k 4,4n -k 5,5 -k 6,6n
3038 B.3.0.0.pxname.1:MGP:str:private-backend
3039 B.3.0.0.pxname.2:MGP:str:private-backend
3040 B.3.0.1.svname.1:MGP:str:BACKEND
3041 B.3.0.1.svname.2:MGP:str:BACKEND
3042 B.3.0.2.qcur.1:MGP:u32:0
3043 B.3.0.2.qcur.2:MGP:u32:0
3044 B.3.0.3.qmax.1:MGP:u32:0
3045 B.3.0.3.qmax.2:MGP:u32:0
3046 B.3.0.4.scur.1:MGP:u32:0
3047 B.3.0.4.scur.2:MGP:u32:0
3048 B.3.0.5.smax.1:MGP:u32:0
3049 B.3.0.5.smax.2:MGP:u32:0
3050 B.3.0.6.slim.1:MGP:u32:1000
3051 B.3.0.6.slim.2:MGP:u32:1000
3052 (...)
Willy Tarreau44aed902015-10-13 14:45:29 +02003053
Simon Horman05ee2132017-01-04 09:37:25 +01003054 The format of JSON output is described in a schema which may be output
Simon Horman6f6bb382017-01-04 09:37:26 +01003055 using "show schema json".
3056
3057 The JSON output contains no extra whitespace in order to reduce the
3058 volume of output. For human consumption passing the output through a
3059 pretty printer may be helpful. Example :
3060
3061 $ echo "show stat json" | socat /var/run/haproxy.sock stdio | \
3062 python -m json.tool
Simon Horman05ee2132017-01-04 09:37:25 +01003063
3064 The JSON output contains no extra whitespace in order to reduce the
3065 volume of output. For human consumption passing the output through a
3066 pretty printer may be helpful. Example :
3067
3068 $ echo "show stat json" | socat /var/run/haproxy.sock stdio | \
3069 python -m json.tool
3070
Remi Tricot-Le Bretone88a2ca2021-04-08 15:30:23 +02003071show ssl ca-file [<cafile>[:<index>]]
3072 Display the list of CA files used by HAProxy and their respective certificate
3073 counts. If a filename is prefixed by an asterisk, it is a transaction which
3074 is not committed yet. If a <cafile> is specified without <index>, it will show
3075 the status of the CA file ("Used"/"Unused") followed by details about all the
3076 certificates contained in the CA file. The details displayed for every
3077 certificate are the same as the ones displayed by a "show ssl cert" command.
3078 If a <cafile> is specified followed by an <index>, it will only display the
3079 details of the certificate having the specified index. Indexes start from 1.
3080 If the index is invalid (too big for instance), nothing will be displayed.
3081 This command can be useful to check if a CA file was properly updated.
3082 You can also display the details of an ongoing transaction by prefixing the
3083 filename by an asterisk.
3084
3085 Example :
3086
3087 $ echo "show ssl ca-file" | socat /var/run/haproxy.master -
3088 # transaction
3089 *cafile.crt - 2 certificate(s)
3090 # filename
3091 cafile.crt - 1 certificate(s)
3092
3093 $ echo "show ssl ca-file cafile.crt" | socat /var/run/haproxy.master -
3094 Filename: /home/tricot/work/haproxy/reg-tests/ssl/set_cafile_ca2.crt
3095 Status: Used
3096
3097 Certificate #1:
3098 Serial: 11A4D2200DC84376E7D233CAFF39DF44BF8D1211
3099 notBefore: Apr 1 07:40:53 2021 GMT
3100 notAfter: Aug 17 07:40:53 2048 GMT
3101 Subject Alternative Name:
3102 Algorithm: RSA4096
3103 SHA1 FingerPrint: A111EF0FEFCDE11D47FE3F33ADCA8435EBEA4864
3104 Subject: /C=FR/ST=Some-State/O=HAProxy Technologies/CN=HAProxy Technologies CA
3105 Issuer: /C=FR/ST=Some-State/O=HAProxy Technologies/CN=HAProxy Technologies CA
3106
3107 $ echo "show ssl ca-file *cafile.crt:2" | socat /var/run/haproxy.master -
3108 Filename: */home/tricot/work/haproxy/reg-tests/ssl/set_cafile_ca2.crt
3109 Status: Unused
3110
3111 Certificate #2:
3112 Serial: 587A1CE5ED855040A0C82BF255FF300ADB7C8136
3113 [...]
3114
William Lallemandd4f946c2019-12-05 10:26:40 +01003115show ssl cert [<filename>]
Remi Tricot-Le Bretonb5f0fac2021-04-14 16:19:29 +02003116 Display the list of certificates used on frontends and backends.
3117 If a filename is prefixed by an asterisk, it is a transaction which is not
3118 committed yet. If a filename is specified, it will show details about the
3119 certificate. This command can be useful to check if a certificate was well
3120 updated. You can also display details on a transaction by prefixing the
3121 filename by an asterisk.
Remi Tricot-Le Breton6056e612021-06-10 13:51:15 +02003122 This command can also be used to display the details of a certificate's OCSP
3123 response by suffixing the filename with a ".ocsp" extension. It works for
3124 committed certificates as well as for ongoing transactions. On a committed
3125 certificate, this command is equivalent to calling "show ssl ocsp-response"
3126 with the certificate's corresponding OCSP response ID.
William Lallemandd4f946c2019-12-05 10:26:40 +01003127
3128 Example :
3129
3130 $ echo "@1 show ssl cert" | socat /var/run/haproxy.master -
3131 # transaction
3132 *test.local.pem
3133 # filename
3134 test.local.pem
3135
3136 $ echo "@1 show ssl cert test.local.pem" | socat /var/run/haproxy.master -
3137 Filename: test.local.pem
3138 Serial: 03ECC19BA54B25E85ABA46EE561B9A10D26F
3139 notBefore: Sep 13 21:20:24 2019 GMT
3140 notAfter: Dec 12 21:20:24 2019 GMT
3141 Issuer: /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
3142 Subject: /CN=test.local
3143 Subject Alternative Name: DNS:test.local, DNS:imap.test.local
3144 Algorithm: RSA2048
3145 SHA1 FingerPrint: 417A11CAE25F607B24F638B4A8AEE51D1E211477
3146
3147 $ echo "@1 show ssl cert *test.local.pem" | socat /var/run/haproxy.master -
3148 Filename: *test.local.pem
3149 [...]
3150
Remi Tricot-Le Breton3c222bd2021-04-27 16:28:25 +02003151show ssl crl-file [<crlfile>[:<index>]]
3152 Display the list of CRL files used by HAProxy.
3153 If a filename is prefixed by an asterisk, it is a transaction which is not
3154 committed yet. If a <crlfile> is specified without <index>, it will show the
3155 status of the CRL file ("Used"/"Unused") followed by details about all the
3156 Revocation Lists contained in the CRL file. The details displayed for every
3157 list are based on the output of "openssl crl -text -noout -in <file>".
3158 If a <crlfile> is specified followed by an <index>, it will only display the
3159 details of the list having the specified index. Indexes start from 1.
3160 If the index is invalid (too big for instance), nothing will be displayed.
3161 This command can be useful to check if a CRL file was properly updated.
3162 You can also display the details of an ongoing transaction by prefixing the
3163 filename by an asterisk.
3164
3165 Example :
3166
3167 $ echo "show ssl crl-file" | socat /var/run/haproxy.master -
3168 # transaction
3169 *crlfile.pem
3170 # filename
3171 crlfile.pem
3172
3173 $ echo "show ssl crl-file crlfile.pem" | socat /var/run/haproxy.master -
3174 Filename: /home/tricot/work/haproxy/reg-tests/ssl/crlfile.pem
3175 Status: Used
3176
3177 Certificate Revocation List #1:
3178 Version 1
3179 Signature Algorithm: sha256WithRSAEncryption
3180 Issuer: /C=FR/O=HAProxy Technologies/CN=Intermediate CA2
3181 Last Update: Apr 23 14:45:39 2021 GMT
3182 Next Update: Sep 8 14:45:39 2048 GMT
3183 Revoked Certificates:
3184 Serial Number: 1008
3185 Revocation Date: Apr 23 14:45:36 2021 GMT
3186
3187 Certificate Revocation List #2:
3188 Version 1
3189 Signature Algorithm: sha256WithRSAEncryption
3190 Issuer: /C=FR/O=HAProxy Technologies/CN=Root CA
3191 Last Update: Apr 23 14:30:44 2021 GMT
3192 Next Update: Sep 8 14:30:44 2048 GMT
3193 No Revoked Certificates.
3194
William Lallemandc69f02d2020-04-06 19:07:03 +02003195show ssl crt-list [-n] [<filename>]
William Lallemandaccac232020-04-02 17:42:51 +02003196 Display the list of crt-list and directories used in the HAProxy
William Lallemandc69f02d2020-04-06 19:07:03 +02003197 configuration. If a filename is specified, dump the content of a crt-list or
3198 a directory. Once dumped the output can be used as a crt-list file.
3199 The '-n' option can be used to display the line number, which is useful when
3200 combined with the 'del ssl crt-list' option when a entry is duplicated. The
3201 output with the '-n' option is not compatible with the crt-list format and
3202 not loadable by haproxy.
William Lallemandaccac232020-04-02 17:42:51 +02003203
3204 Example:
William Lallemandc69f02d2020-04-06 19:07:03 +02003205 echo "show ssl crt-list -n localhost.crt-list" | socat /tmp/sock1 -
William Lallemandaccac232020-04-02 17:42:51 +02003206 # localhost.crt-list
William Lallemandc69f02d2020-04-06 19:07:03 +02003207 common.pem:1 !not.test1.com *.test1.com !localhost
3208 common.pem:2
3209 ecdsa.pem:3 [verify none allow-0rtt ssl-min-ver TLSv1.0 ssl-max-ver TLSv1.3] localhost !www.test1.com
3210 ecdsa.pem:4 [verify none allow-0rtt ssl-min-ver TLSv1.0 ssl-max-ver TLSv1.3]
William Lallemandaccac232020-04-02 17:42:51 +02003211
Remi Tricot-Le Bretond92fd112021-06-10 13:51:13 +02003212show ssl ocsp-response [<id>]
3213 Display the IDs of the OCSP tree entries corresponding to all the OCSP
3214 responses used in HAProxy, as well as the issuer's name and key hash and the
3215 serial number of the certificate for which the OCSP response was built.
3216 If a valid <id> is provided, display the contents of the corresponding OCSP
3217 response. The information displayed is the same as in an "openssl ocsp -respin
3218 <ocsp-response> -text" call.
3219
3220 Example :
3221
3222 $ echo "show ssl ocsp-response" | socat /var/run/haproxy.master -
3223 # Certificate IDs
3224 Certificate ID key : 303b300906052b0e03021a050004148a83e0060faff709ca7e9b95522a2e81635fda0a0414f652b0e435d5ea923851508f0adbe92d85de007a0202100a
3225 Certificate ID:
Remi Tricot-Le Bretond92fd112021-06-10 13:51:13 +02003226 Issuer Name Hash: 8A83E0060FAFF709CA7E9B95522A2E81635FDA0A
3227 Issuer Key Hash: F652B0E435D5EA923851508F0ADBE92D85DE007A
3228 Serial Number: 100A
3229
3230 $ echo "show ssl ocsp-response 303b300906052b0e03021a050004148a83e0060faff709ca7e9b95522a2e81635fda0a0414f652b0e435d5ea923851508f0adbe92d85de007a0202100a" | socat /var/run/haproxy.master -
3231 OCSP Response Data:
3232 OCSP Response Status: successful (0x0)
3233 Response Type: Basic OCSP Response
3234 Version: 1 (0x0)
3235 Responder Id: C = FR, O = HAProxy Technologies, CN = ocsp.haproxy.com
3236 Produced At: May 27 15:43:38 2021 GMT
3237 Responses:
3238 Certificate ID:
3239 Hash Algorithm: sha1
3240 Issuer Name Hash: 8A83E0060FAFF709CA7E9B95522A2E81635FDA0A
3241 Issuer Key Hash: F652B0E435D5EA923851508F0ADBE92D85DE007A
3242 Serial Number: 100A
3243 Cert Status: good
3244 This Update: May 27 15:43:38 2021 GMT
3245 Next Update: Oct 12 15:43:38 2048 GMT
3246 [...]
3247
Willy Tarreau44aed902015-10-13 14:45:29 +02003248show table
3249 Dump general information on all known stick-tables. Their name is returned
3250 (the name of the proxy which holds them), their type (currently zero, always
3251 IP), their size in maximum possible number of entries, and the number of
3252 entries currently in use.
3253
3254 Example :
3255 $ echo "show table" | socat stdio /tmp/sock1
3256 >>> # table: front_pub, type: ip, size:204800, used:171454
3257 >>> # table: back_rdp, type: ip, size:204800, used:0
3258
Adis Nezirovic1a693fc2020-01-16 15:19:29 +01003259show table <name> [ data.<type> <operator> <value> [data.<type> ...]] | [ key <key> ]
Willy Tarreau44aed902015-10-13 14:45:29 +02003260 Dump contents of stick-table <name>. In this mode, a first line of generic
3261 information about the table is reported as with "show table", then all
3262 entries are dumped. Since this can be quite heavy, it is possible to specify
3263 a filter in order to specify what entries to display.
3264
3265 When the "data." form is used the filter applies to the stored data (see
3266 "stick-table" in section 4.2). A stored data type must be specified
3267 in <type>, and this data type must be stored in the table otherwise an
3268 error is reported. The data is compared according to <operator> with the
3269 64-bit integer <value>. Operators are the same as with the ACLs :
3270
3271 - eq : match entries whose data is equal to this value
3272 - ne : match entries whose data is not equal to this value
3273 - le : match entries whose data is less than or equal to this value
3274 - ge : match entries whose data is greater than or equal to this value
3275 - lt : match entries whose data is less than this value
3276 - gt : match entries whose data is greater than this value
3277
Adis Nezirovic1a693fc2020-01-16 15:19:29 +01003278 In this form, you can use multiple data filter entries, up to a maximum
3279 defined during build time (4 by default).
Willy Tarreau44aed902015-10-13 14:45:29 +02003280
3281 When the key form is used the entry <key> is shown. The key must be of the
3282 same type as the table, which currently is limited to IPv4, IPv6, integer,
3283 and string.
3284
3285 Example :
3286 $ echo "show table http_proxy" | socat stdio /tmp/sock1
3287 >>> # table: http_proxy, type: ip, size:204800, used:2
3288 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
3289 bytes_out_rate(60000)=187
3290 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
3291 bytes_out_rate(60000)=191
3292
3293 $ echo "show table http_proxy data.gpc0 gt 0" | socat stdio /tmp/sock1
3294 >>> # table: http_proxy, type: ip, size:204800, used:2
3295 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
3296 bytes_out_rate(60000)=191
3297
3298 $ echo "show table http_proxy data.conn_rate gt 5" | \
3299 socat stdio /tmp/sock1
3300 >>> # table: http_proxy, type: ip, size:204800, used:2
3301 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
3302 bytes_out_rate(60000)=191
3303
3304 $ echo "show table http_proxy key 127.0.0.2" | \
3305 socat stdio /tmp/sock1
3306 >>> # table: http_proxy, type: ip, size:204800, used:2
3307 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
3308 bytes_out_rate(60000)=191
3309
3310 When the data criterion applies to a dynamic value dependent on time such as
3311 a bytes rate, the value is dynamically computed during the evaluation of the
3312 entry in order to decide whether it has to be dumped or not. This means that
3313 such a filter could match for some time then not match anymore because as
3314 time goes, the average event rate drops.
3315
3316 It is possible to use this to extract lists of IP addresses abusing the
3317 service, in order to monitor them or even blacklist them in a firewall.
3318 Example :
3319 $ echo "show table http_proxy data.gpc0 gt 0" \
3320 | socat stdio /tmp/sock1 \
3321 | fgrep 'key=' | cut -d' ' -f2 | cut -d= -f2 > abusers-ip.txt
3322 ( or | awk '/key/{ print a[split($2,a,"=")]; }' )
3323
Willy Tarreau7eff06e2021-01-29 11:32:55 +01003324show tasks
3325 Dumps the number of tasks currently in the run queue, with the number of
3326 occurrences for each function, and their average latency when it's known
3327 (for pure tasks with task profiling enabled). The dump is a snapshot of the
3328 instant it's done, and there may be variations depending on what tasks are
3329 left in the queue at the moment it happens, especially in mono-thread mode
3330 as there's less chance that I/Os can refill the queue (unless the queue is
3331 full). This command takes exclusive access to the process and can cause
3332 minor but measurable latencies when issued on a highly loaded process, so
3333 it must not be abused by monitoring bots.
3334
Willy Tarreau4e2b6462019-05-16 17:44:30 +02003335show threads
3336 Dumps some internal states and structures for each thread, that may be useful
3337 to help developers understand a problem. The output tries to be readable by
Willy Tarreauc7091d82019-05-17 10:08:49 +02003338 showing one block per thread. When haproxy is built with USE_THREAD_DUMP=1,
3339 an advanced dump mechanism involving thread signals is used so that each
3340 thread can dump its own state in turn. Without this option, the thread
3341 processing the command shows all its details but the other ones are less
Willy Tarreaue6a02fa2019-05-22 07:06:44 +02003342 detailed. A star ('*') is displayed in front of the thread handling the
3343 command. A right angle bracket ('>') may also be displayed in front of
3344 threads which didn't make any progress since last invocation of this command,
3345 indicating a bug in the code which must absolutely be reported. When this
3346 happens between two threads it usually indicates a deadlock. If a thread is
3347 alone, it's a different bug like a corrupted list. In all cases the process
3348 needs is not fully functional anymore and needs to be restarted.
3349
3350 The output format is purposely not documented so that it can easily evolve as
3351 new needs are identified, without having to maintain any form of backwards
3352 compatibility, and just like with "show activity", the values are meaningless
3353 without the code at hand.
Willy Tarreau4e2b6462019-05-16 17:44:30 +02003354
William Lallemandbb933462016-05-31 21:09:53 +02003355show tls-keys [id|*]
3356 Dump all loaded TLS ticket keys references. The TLS ticket key reference ID
3357 and the file from which the keys have been loaded is shown. Both of those
3358 can be used to update the TLS keys using "set ssl tls-key". If an ID is
3359 specified as parameter, it will dump the tickets, using * it will dump every
3360 keys from every references.
Willy Tarreau44aed902015-10-13 14:45:29 +02003361
Simon Horman6f6bb382017-01-04 09:37:26 +01003362show schema json
3363 Dump the schema used for the output of "show info json" and "show stat json".
3364
3365 The contains no extra whitespace in order to reduce the volume of output.
3366 For human consumption passing the output through a pretty printer may be
3367 helpful. Example :
3368
3369 $ echo "show schema json" | socat /var/run/haproxy.sock stdio | \
3370 python -m json.tool
3371
3372 The schema follows "JSON Schema" (json-schema.org) and accordingly
3373 verifiers may be used to verify the output of "show info json" and "show
3374 stat json" against the schema.
3375
Willy Tarreauf909c912019-08-22 20:06:04 +02003376show trace [<source>]
3377 Show the current trace status. For each source a line is displayed with a
3378 single-character status indicating if the trace is stopped, waiting, or
3379 running. The output sink used by the trace is indicated (or "none" if none
3380 was set), as well as the number of dropped events in this sink, followed by a
3381 brief description of the source. If a source name is specified, a detailed
3382 list of all events supported by the source, and their status for each action
3383 (report, start, pause, stop), indicated by a "+" if they are enabled, or a
3384 "-" otherwise. All these events are independent and an event might trigger
3385 a start without being reported and conversely.
Simon Horman6f6bb382017-01-04 09:37:26 +01003386
William Lallemand740629e2021-12-14 15:22:29 +01003387show version
3388 Show the version of the current HAProxy process. This is available from
3389 master and workers CLI.
3390 Example:
3391
3392 $ echo "show version" | socat /var/run/haproxy.sock stdio
3393 2.4.9
3394
3395 $ echo "show version" | socat /var/run/haproxy-master.sock stdio
3396 2.5.0
3397
Willy Tarreau44aed902015-10-13 14:45:29 +02003398shutdown frontend <frontend>
3399 Completely delete the specified frontend. All the ports it was bound to will
3400 be released. It will not be possible to enable the frontend anymore after
3401 this operation. This is intended to be used in environments where stopping a
3402 proxy is not even imaginable but a misconfigured proxy must be fixed. That
3403 way it's possible to release the port and bind it into another process to
3404 restore operations. The frontend will not appear at all on the stats page
3405 once it is terminated.
3406
3407 The frontend may be specified either by its name or by its numeric ID,
3408 prefixed with a sharp ('#').
3409
3410 This command is restricted and can only be issued on sockets configured for
3411 level "admin".
3412
3413shutdown session <id>
3414 Immediately terminate the session matching the specified session identifier.
3415 This identifier is the first field at the beginning of the lines in the dumps
3416 of "show sess" (it corresponds to the session pointer). This can be used to
3417 terminate a long-running session without waiting for a timeout or when an
3418 endless transfer is ongoing. Such terminated sessions are reported with a 'K'
3419 flag in the logs.
3420
3421shutdown sessions server <backend>/<server>
3422 Immediately terminate all the sessions attached to the specified server. This
3423 can be used to terminate long-running sessions after a server is put into
3424 maintenance mode, for instance. Such terminated sessions are reported with a
3425 'K' flag in the logs.
3426
Willy Tarreauf909c912019-08-22 20:06:04 +02003427trace
3428 The "trace" command alone lists the trace sources, their current status, and
3429 their brief descriptions. It is only meant as a menu to enter next levels,
3430 see other "trace" commands below.
3431
3432trace 0
3433 Immediately stops all traces. This is made to be used as a quick solution
3434 to terminate a debugging session or as an emergency action to be used in case
3435 complex traces were enabled on multiple sources and impact the service.
3436
3437trace <source> event [ [+|-|!]<name> ]
3438 Without argument, this will list all the events supported by the designated
3439 source. They are prefixed with a "-" if they are not enabled, or a "+" if
3440 they are enabled. It is important to note that a single trace may be labelled
3441 with multiple events, and as long as any of the enabled events matches one of
3442 the events labelled on the trace, the event will be passed to the trace
3443 subsystem. For example, receiving an HTTP/2 frame of type HEADERS may trigger
3444 a frame event and a stream event since the frame creates a new stream. If
3445 either the frame event or the stream event are enabled for this source, the
3446 frame will be passed to the trace framework.
3447
3448 With an argument, it is possible to toggle the state of each event and
3449 individually enable or disable them. Two special keywords are supported,
3450 "none", which matches no event, and is used to disable all events at once,
3451 and "any" which matches all events, and is used to enable all events at
3452 once. Other events are specific to the event source. It is possible to
3453 enable one event by specifying its name, optionally prefixed with '+' for
3454 better readability. It is possible to disable one event by specifying its
3455 name prefixed by a '-' or a '!'.
3456
3457 One way to completely disable a trace source is to pass "event none", and
3458 this source will instantly be totally ignored.
3459
3460trace <source> level [<level>]
Willy Tarreau2ea549b2019-08-29 08:01:48 +02003461 Without argument, this will list all trace levels for this source, and the
Willy Tarreauf909c912019-08-22 20:06:04 +02003462 current one will be indicated by a star ('*') prepended in front of it. With
Willy Tarreau2ea549b2019-08-29 08:01:48 +02003463 an argument, this will change the trace level to the specified level. Detail
Willy Tarreauf909c912019-08-22 20:06:04 +02003464 levels are a form of filters that are applied before reporting the events.
Willy Tarreau2ea549b2019-08-29 08:01:48 +02003465 These filters are used to selectively include or exclude events depending on
3466 their level of importance. For example a developer might need to know
3467 precisely where in the code an HTTP header was considered invalid while the
3468 end user may not even care about this header's validity at all. There are
3469 currently 5 distinct levels for a trace :
Willy Tarreauf909c912019-08-22 20:06:04 +02003470
3471 user this will report information that are suitable for use by a
3472 regular haproxy user who wants to observe his traffic.
3473 Typically some HTTP requests and responses will be reported
3474 without much detail. Most sources will set this as the
3475 default level to ease operations.
3476
Willy Tarreau2ea549b2019-08-29 08:01:48 +02003477 proto in addition to what is reported at the "user" level, it also
3478 displays protocol-level updates. This can for example be the
3479 frame types or HTTP headers after decoding.
Willy Tarreauf909c912019-08-22 20:06:04 +02003480
3481 state in addition to what is reported at the "proto" level, it
3482 will also display state transitions (or failed transitions)
3483 which happen in parsers, so this will show attempts to
3484 perform an operation while the "proto" level only shows
3485 the final operation.
3486
Willy Tarreau2ea549b2019-08-29 08:01:48 +02003487 data in addition to what is reported at the "state" level, it
3488 will also include data transfers between the various layers.
3489
Willy Tarreauf909c912019-08-22 20:06:04 +02003490 developer it reports everything available, which can include advanced
3491 information such as "breaking out of this loop" that are
3492 only relevant to a developer trying to understand a bug that
Willy Tarreau09fb0df2019-08-29 08:40:59 +02003493 only happens once in a while in field. Function names are
3494 only reported at this level.
Willy Tarreauf909c912019-08-22 20:06:04 +02003495
3496 It is highly recommended to always use the "user" level only and switch to
3497 other levels only if instructed to do so by a developer. Also it is a good
3498 idea to first configure the events before switching to higher levels, as it
3499 may save from dumping many lines if no filter is applied.
3500
3501trace <source> lock [criterion]
3502 Without argument, this will list all the criteria supported by this source
3503 for lock-on processing, and display the current choice by a star ('*') in
3504 front of it. Lock-on means that the source will focus on the first matching
3505 event and only stick to the criterion which triggered this event, and ignore
3506 all other ones until the trace stops. This allows for example to take a trace
3507 on a single connection or on a single stream. The following criteria are
3508 supported by some traces, though not necessarily all, since some of them
3509 might not be available to the source :
3510
3511 backend lock on the backend that started the trace
3512 connection lock on the connection that started the trace
3513 frontend lock on the frontend that started the trace
3514 listener lock on the listener that started the trace
3515 nothing do not lock on anything
3516 server lock on the server that started the trace
3517 session lock on the session that started the trace
3518 thread lock on the thread that started the trace
3519
3520 In addition to this, each source may provide up to 4 specific criteria such
3521 as internal states or connection IDs. For example in HTTP/2 it is possible
3522 to lock on the H2 stream and ignore other streams once a strace starts.
3523
3524 When a criterion is passed in argument, this one is used instead of the
3525 other ones and any existing tracking is immediately terminated so that it can
3526 restart with the new criterion. The special keyword "nothing" is supported by
3527 all sources to permanently disable tracking.
3528
3529trace <source> { pause | start | stop } [ [+|-|!]event]
3530 Without argument, this will list the events enabled to automatically pause,
3531 start, or stop a trace for this source. These events are specific to each
3532 trace source. With an argument, this will either enable the event for the
3533 specified action (if optionally prefixed by a '+') or disable it (if
3534 prefixed by a '-' or '!'). The special keyword "now" is not an event and
3535 requests to take the action immediately. The keywords "none" and "any" are
3536 supported just like in "trace event".
3537
3538 The 3 supported actions are respectively "pause", "start" and "stop". The
3539 "pause" action enumerates events which will cause a running trace to stop and
3540 wait for a new start event to restart it. The "start" action enumerates the
3541 events which switch the trace into the waiting mode until one of the start
3542 events appears. And the "stop" action enumerates the events which definitely
3543 stop the trace until it is manually enabled again. In practice it makes sense
3544 to manually start a trace using "start now" without caring about events, and
3545 to stop it using "stop now". In order to capture more subtle event sequences,
3546 setting "start" to a normal event (like receiving an HTTP request) and "stop"
3547 to a very rare event like emitting a certain error, will ensure that the last
3548 captured events will match the desired criteria. And the pause event is
3549 useful to detect the end of a sequence, disable the lock-on and wait for
3550 another opportunity to take a capture. In this case it can make sense to
3551 enable lock-on to spot only one specific criterion (e.g. a stream), and have
3552 "start" set to anything that starts this criterion (e.g. all events which
3553 create a stream), "stop" set to the expected anomaly, and "pause" to anything
3554 that ends that criterion (e.g. any end of stream event). In this case the
3555 trace log will contain complete sequences of perfectly clean series affecting
3556 a single object, until the last sequence containing everything from the
3557 beginning to the anomaly.
3558
3559trace <source> sink [<sink>]
3560 Without argument, this will list all event sinks available for this source,
3561 and the currently configured one will have a star ('*') prepended in front
3562 of it. Sink "none" is always available and means that all events are simply
3563 dropped, though their processing is not ignored (e.g. lock-on does occur).
3564 Other sinks are available depending on configuration and build options, but
3565 typically "stdout" and "stderr" will be usable in debug mode, and in-memory
3566 ring buffers should be available as well. When a name is specified, the sink
3567 instantly changes for the specified source. Events are not changed during a
3568 sink change. In the worst case some may be lost if an invalid sink is used
3569 (or "none"), but operations do continue to a different destination.
3570
Willy Tarreau370a6942019-08-29 08:24:16 +02003571trace <source> verbosity [<level>]
3572 Without argument, this will list all verbosity levels for this source, and the
3573 current one will be indicated by a star ('*') prepended in front of it. With
3574 an argument, this will change the verbosity level to the specified one.
3575
3576 Verbosity levels indicate how far the trace decoder should go to provide
3577 detailed information. It depends on the trace source, since some sources will
3578 not even provide a specific decoder. Level "quiet" is always available and
3579 disables any decoding. It can be useful when trying to figure what's
3580 happening before trying to understand the details, since it will have a very
3581 low impact on performance and trace size. When no verbosity levels are
3582 declared by a source, level "default" is available and will cause a decoder
3583 to be called when specified in the traces. It is an opportunistic decoding.
3584 When the source declares some verbosity levels, these ones are listed with
3585 a description of what they correspond to. In this case the trace decoder
3586 provided by the source will be as accurate as possible based on the
3587 information available at the trace point. The first level above "quiet" is
3588 set by default.
3589
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003590
William Lallemand142db372018-12-11 18:56:45 +010035919.4. Master CLI
3592---------------
3593
3594The master CLI is a socket bound to the master process in master-worker mode.
3595This CLI gives access to the unix socket commands in every running or leaving
3596processes and allows a basic supervision of those processes.
3597
3598The master CLI is configurable only from the haproxy program arguments with
3599the -S option. This option also takes bind options separated by commas.
3600
3601Example:
3602
3603 # haproxy -W -S 127.0.0.1:1234 -f test1.cfg
3604 # haproxy -Ws -S /tmp/master-socket,uid,1000,gid,1000,mode,600 -f test1.cfg
William Lallemandb7ea1412018-12-13 09:05:47 +01003605 # haproxy -W -S /tmp/master-socket,level,user -f test1.cfg
William Lallemand142db372018-12-11 18:56:45 +01003606
William Lallemanda68a00f2021-11-10 15:42:17 +01003607The master CLI introduces a 'show proc' command to surpervise the
William Lallemand142db372018-12-11 18:56:45 +01003608processes:
3609
3610Example:
3611
3612 $ echo 'show proc' | socat /var/run/haproxy-master.sock -
William Lallemanda68a00f2021-11-10 15:42:17 +01003613 #<PID> <type> <reloads> <uptime> <version>
3614 1162 master 5 [failed: 0] 0d00h02m07s 2.5-dev13
William Lallemand142db372018-12-11 18:56:45 +01003615 # workers
William Lallemanda68a00f2021-11-10 15:42:17 +01003616 1271 worker 1 0d00h00m00s 2.5-dev13
William Lallemand142db372018-12-11 18:56:45 +01003617 # old workers
William Lallemanda68a00f2021-11-10 15:42:17 +01003618 1233 worker 3 0d00h00m43s 2.0-dev3-6019f6-289
3619 # programs
3620 1244 foo 0 0d00h00m00s -
3621 1255 bar 0 0d00h00m00s -
William Lallemand142db372018-12-11 18:56:45 +01003622
3623
3624In this example, the master has been reloaded 5 times but one of the old
3625worker is still running and survived 3 reloads. You could access the CLI of
3626this worker to understand what's going on.
3627
Willy Tarreau52880f92018-12-15 13:30:03 +01003628When the prompt is enabled (via the "prompt" command), the context the CLI is
3629working on is displayed in the prompt. The master is identified by the "master"
3630string, and other processes are identified with their PID. In case the last
3631reload failed, the master prompt will be changed to "master[ReloadFailed]>" so
3632that it becomes visible that the process is still running on the previous
3633configuration and that the new configuration is not operational.
3634
William Lallemand142db372018-12-11 18:56:45 +01003635The master CLI uses a special prefix notation to access the multiple
3636processes. This notation is easily identifiable as it begins by a @.
3637
3638A @ prefix can be followed by a relative process number or by an exclamation
3639point and a PID. (e.g. @1 or @!1271). A @ alone could be use to specify the
3640master. Leaving processes are only accessible with the PID as relative process
3641number are only usable with the current processes.
3642
3643Examples:
3644
3645 $ socat /var/run/haproxy-master.sock readline
3646 prompt
3647 master> @1 show info; @2 show info
3648 [...]
3649 Process_num: 1
3650 Pid: 1271
3651 [...]
3652 Process_num: 2
3653 Pid: 1272
3654 [...]
3655 master>
3656
3657 $ echo '@!1271 show info; @!1272 show info' | socat /var/run/haproxy-master.sock -
3658 [...]
3659
3660A prefix could be use as a command, which will send every next commands to
3661the specified process.
3662
3663Examples:
3664
3665 $ socat /var/run/haproxy-master.sock readline
3666 prompt
3667 master> @1
3668 1271> show info
3669 [...]
3670 1271> show stat
3671 [...]
3672 1271> @
3673 master>
3674
3675 $ echo '@1; show info; show stat; @2; show info; show stat' | socat /var/run/haproxy-master.sock -
3676 [...]
3677
William Lallemanda57b7e32018-12-14 21:11:31 +01003678You can also reload the HAProxy master process with the "reload" command which
3679does the same as a `kill -USR2` on the master process, provided that the user
3680has at least "operator" or "admin" privileges.
3681
3682Example:
3683
varnav5a3fe9f2021-05-10 10:29:57 -04003684 $ echo "reload" | socat /var/run/haproxy-master.sock stdin
William Lallemanda57b7e32018-12-14 21:11:31 +01003685
3686Note that a reload will close the connection to the master CLI.
3687
William Lallemand142db372018-12-11 18:56:45 +01003688
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200368910. Tricks for easier configuration management
3690----------------------------------------------
3691
3692It is very common that two HAProxy nodes constituting a cluster share exactly
3693the same configuration modulo a few addresses. Instead of having to maintain a
3694duplicate configuration for each node, which will inevitably diverge, it is
3695possible to include environment variables in the configuration. Thus multiple
3696configuration may share the exact same file with only a few different system
3697wide environment variables. This started in version 1.5 where only addresses
3698were allowed to include environment variables, and 1.6 goes further by
3699supporting environment variables everywhere. The syntax is the same as in the
3700UNIX shell, a variable starts with a dollar sign ('$'), followed by an opening
3701curly brace ('{'), then the variable name followed by the closing brace ('}').
3702Except for addresses, environment variables are only interpreted in arguments
3703surrounded with double quotes (this was necessary not to break existing setups
3704using regular expressions involving the dollar symbol).
3705
3706Environment variables also make it convenient to write configurations which are
3707expected to work on various sites where only the address changes. It can also
3708permit to remove passwords from some configs. Example below where the the file
3709"site1.env" file is sourced by the init script upon startup :
3710
3711 $ cat site1.env
3712 LISTEN=192.168.1.1
3713 CACHE_PFX=192.168.11
3714 SERVER_PFX=192.168.22
3715 LOGGER=192.168.33.1
3716 STATSLP=admin:pa$$w0rd
3717 ABUSERS=/etc/haproxy/abuse.lst
3718 TIMEOUT=10s
3719
3720 $ cat haproxy.cfg
3721 global
3722 log "${LOGGER}:514" local0
3723
3724 defaults
3725 mode http
3726 timeout client "${TIMEOUT}"
3727 timeout server "${TIMEOUT}"
3728 timeout connect 5s
3729
3730 frontend public
3731 bind "${LISTEN}:80"
3732 http-request reject if { src -f "${ABUSERS}" }
3733 stats uri /stats
3734 stats auth "${STATSLP}"
3735 use_backend cache if { path_end .jpg .css .ico }
3736 default_backend server
3737
3738 backend cache
3739 server cache1 "${CACHE_PFX}.1:18080" check
3740 server cache2 "${CACHE_PFX}.2:18080" check
3741
3742 backend server
3743 server cache1 "${SERVER_PFX}.1:8080" check
3744 server cache2 "${SERVER_PFX}.2:8080" check
3745
3746
374711. Well-known traps to avoid
3748-----------------------------
3749
3750Once in a while, someone reports that after a system reboot, the haproxy
3751service wasn't started, and that once they start it by hand it works. Most
3752often, these people are running a clustered IP address mechanism such as
3753keepalived, to assign the service IP address to the master node only, and while
3754it used to work when they used to bind haproxy to address 0.0.0.0, it stopped
3755working after they bound it to the virtual IP address. What happens here is
3756that when the service starts, the virtual IP address is not yet owned by the
3757local node, so when HAProxy wants to bind to it, the system rejects this
3758because it is not a local IP address. The fix doesn't consist in delaying the
3759haproxy service startup (since it wouldn't stand a restart), but instead to
3760properly configure the system to allow binding to non-local addresses. This is
3761easily done on Linux by setting the net.ipv4.ip_nonlocal_bind sysctl to 1. This
3762is also needed in order to transparently intercept the IP traffic that passes
3763through HAProxy for a specific target address.
3764
3765Multi-process configurations involving source port ranges may apparently seem
3766to work but they will cause some random failures under high loads because more
3767than one process may try to use the same source port to connect to the same
3768server, which is not possible. The system will report an error and a retry will
3769happen, picking another port. A high value in the "retries" parameter may hide
3770the effect to a certain extent but this also comes with increased CPU usage and
3771processing time. Logs will also report a certain number of retries. For this
3772reason, port ranges should be avoided in multi-process configurations.
3773
Dan Lloyd8e48b872016-07-01 21:01:18 -04003774Since HAProxy uses SO_REUSEPORT and supports having multiple independent
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003775processes bound to the same IP:port, during troubleshooting it can happen that
3776an old process was not stopped before a new one was started. This provides
3777absurd test results which tend to indicate that any change to the configuration
3778is ignored. The reason is that in fact even the new process is restarted with a
3779new configuration, the old one also gets some incoming connections and
3780processes them, returning unexpected results. When in doubt, just stop the new
3781process and try again. If it still works, it very likely means that an old
3782process remains alive and has to be stopped. Linux's "netstat -lntp" is of good
3783help here.
3784
3785When adding entries to an ACL from the command line (eg: when blacklisting a
3786source address), it is important to keep in mind that these entries are not
3787synchronized to the file and that if someone reloads the configuration, these
3788updates will be lost. While this is often the desired effect (for blacklisting)
3789it may not necessarily match expectations when the change was made as a fix for
3790a problem. See the "add acl" action of the CLI interface.
3791
3792
379312. Debugging and performance issues
3794------------------------------------
3795
3796When HAProxy is started with the "-d" option, it will stay in the foreground
3797and will print one line per event, such as an incoming connection, the end of a
3798connection, and for each request or response header line seen. This debug
3799output is emitted before the contents are processed, so they don't consider the
3800local modifications. The main use is to show the request and response without
3801having to run a network sniffer. The output is less readable when multiple
3802connections are handled in parallel, though the "debug2ansi" and "debug2html"
3803scripts found in the examples/ directory definitely help here by coloring the
3804output.
3805
3806If a request or response is rejected because HAProxy finds it is malformed, the
3807best thing to do is to connect to the CLI and issue "show errors", which will
3808report the last captured faulty request and response for each frontend and
3809backend, with all the necessary information to indicate precisely the first
3810character of the input stream that was rejected. This is sometimes needed to
3811prove to customers or to developers that a bug is present in their code. In
3812this case it is often possible to relax the checks (but still keep the
3813captures) using "option accept-invalid-http-request" or its equivalent for
3814responses coming from the server "option accept-invalid-http-response". Please
3815see the configuration manual for more details.
3816
3817Example :
3818
3819 > show errors
3820 Total events captured on [13/Oct/2015:13:43:47.169] : 1
3821
3822 [13/Oct/2015:13:43:40.918] frontend HAProxyLocalStats (#2): invalid request
3823 backend <NONE> (#-1), server <NONE> (#-1), event #0
3824 src 127.0.0.1:51981, session #0, session flags 0x00000080
3825 HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000
3826 HTTP chunk len 0 bytes, HTTP body len 0 bytes
3827 buffer flags 0x00808002, out 0 bytes, total 31 bytes
3828 pending 31 bytes, wrapping at 8040, error at position 13:
3829
3830 00000 GET /invalid request HTTP/1.1\r\n
3831
3832
3833The output of "show info" on the CLI provides a number of useful information
3834regarding the maximum connection rate ever reached, maximum SSL key rate ever
3835reached, and in general all information which can help to explain temporary
3836issues regarding CPU or memory usage. Example :
3837
3838 > show info
3839 Name: HAProxy
3840 Version: 1.6-dev7-e32d18-17
3841 Release_date: 2015/10/12
3842 Nbproc: 1
3843 Process_num: 1
3844 Pid: 7949
3845 Uptime: 0d 0h02m39s
3846 Uptime_sec: 159
3847 Memmax_MB: 0
3848 Ulimit-n: 120032
3849 Maxsock: 120032
3850 Maxconn: 60000
3851 Hard_maxconn: 60000
3852 CurrConns: 0
3853 CumConns: 3
3854 CumReq: 3
3855 MaxSslConns: 0
3856 CurrSslConns: 0
3857 CumSslConns: 0
3858 Maxpipes: 0
3859 PipesUsed: 0
3860 PipesFree: 0
3861 ConnRate: 0
3862 ConnRateLimit: 0
3863 MaxConnRate: 1
3864 SessRate: 0
3865 SessRateLimit: 0
3866 MaxSessRate: 1
3867 SslRate: 0
3868 SslRateLimit: 0
3869 MaxSslRate: 0
3870 SslFrontendKeyRate: 0
3871 SslFrontendMaxKeyRate: 0
3872 SslFrontendSessionReuse_pct: 0
3873 SslBackendKeyRate: 0
3874 SslBackendMaxKeyRate: 0
3875 SslCacheLookups: 0
3876 SslCacheMisses: 0
3877 CompressBpsIn: 0
3878 CompressBpsOut: 0
3879 CompressBpsRateLim: 0
3880 ZlibMemUsage: 0
3881 MaxZlibMemUsage: 0
3882 Tasks: 5
3883 Run_queue: 1
3884 Idle_pct: 100
3885 node: wtap
3886 description:
3887
3888When an issue seems to randomly appear on a new version of HAProxy (eg: every
3889second request is aborted, occasional crash, etc), it is worth trying to enable
Dan Lloyd8e48b872016-07-01 21:01:18 -04003890memory poisoning so that each call to malloc() is immediately followed by the
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003891filling of the memory area with a configurable byte. By default this byte is
38920x50 (ASCII for 'P'), but any other byte can be used, including zero (which
3893will have the same effect as a calloc() and which may make issues disappear).
Dan Lloyd8e48b872016-07-01 21:01:18 -04003894Memory poisoning is enabled on the command line using the "-dM" option. It
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003895slightly hurts performance and is not recommended for use in production. If
Dan Lloyd8e48b872016-07-01 21:01:18 -04003896an issue happens all the time with it or never happens when poisoning uses
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003897byte zero, it clearly means you've found a bug and you definitely need to
3898report it. Otherwise if there's no clear change, the problem it is not related.
3899
3900When debugging some latency issues, it is important to use both strace and
3901tcpdump on the local machine, and another tcpdump on the remote system. The
3902reason for this is that there are delays everywhere in the processing chain and
3903it is important to know which one is causing latency to know where to act. In
3904practice, the local tcpdump will indicate when the input data come in. Strace
3905will indicate when haproxy receives these data (using recv/recvfrom). Warning,
3906openssl uses read()/write() syscalls instead of recv()/send(). Strace will also
3907show when haproxy sends the data, and tcpdump will show when the system sends
3908these data to the interface. Then the external tcpdump will show when the data
3909sent are really received (since the local one only shows when the packets are
3910queued). The benefit of sniffing on the local system is that strace and tcpdump
3911will use the same reference clock. Strace should be used with "-tts200" to get
3912complete timestamps and report large enough chunks of data to read them.
3913Tcpdump should be used with "-nvvttSs0" to report full packets, real sequence
3914numbers and complete timestamps.
3915
3916In practice, received data are almost always immediately received by haproxy
3917(unless the machine has a saturated CPU or these data are invalid and not
3918delivered). If these data are received but not sent, it generally is because
3919the output buffer is saturated (ie: recipient doesn't consume the data fast
3920enough). This can be confirmed by seeing that the polling doesn't notify of
3921the ability to write on the output file descriptor for some time (it's often
3922easier to spot in the strace output when the data finally leave and then roll
3923back to see when the write event was notified). It generally matches an ACK
3924received from the recipient, and detected by tcpdump. Once the data are sent,
3925they may spend some time in the system doing nothing. Here again, the TCP
3926congestion window may be limited and not allow these data to leave, waiting for
3927an ACK to open the window. If the traffic is idle and the data take 40 ms or
3928200 ms to leave, it's a different issue (which is not an issue), it's the fact
3929that the Nagle algorithm prevents empty packets from leaving immediately, in
3930hope that they will be merged with subsequent data. HAProxy automatically
3931disables Nagle in pure TCP mode and in tunnels. However it definitely remains
3932enabled when forwarding an HTTP body (and this contributes to the performance
3933improvement there by reducing the number of packets). Some HTTP non-compliant
3934applications may be sensitive to the latency when delivering incomplete HTTP
3935response messages. In this case you will have to enable "option http-no-delay"
3936to disable Nagle in order to work around their design, keeping in mind that any
3937other proxy in the chain may similarly be impacted. If tcpdump reports that data
3938leave immediately but the other end doesn't see them quickly, it can mean there
Dan Lloyd8e48b872016-07-01 21:01:18 -04003939is a congested WAN link, a congested LAN with flow control enabled and
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003940preventing the data from leaving, or more commonly that HAProxy is in fact
3941running in a virtual machine and that for whatever reason the hypervisor has
3942decided that the data didn't need to be sent immediately. In virtualized
3943environments, latency issues are almost always caused by the virtualization
3944layer, so in order to save time, it's worth first comparing tcpdump in the VM
3945and on the external components. Any difference has to be credited to the
3946hypervisor and its accompanying drivers.
3947
3948When some TCP SACK segments are seen in tcpdump traces (using -vv), it always
3949means that the side sending them has got the proof of a lost packet. While not
3950seeing them doesn't mean there are no losses, seeing them definitely means the
3951network is lossy. Losses are normal on a network, but at a rate where SACKs are
3952not noticeable at the naked eye. If they appear a lot in the traces, it is
3953worth investigating exactly what happens and where the packets are lost. HTTP
3954doesn't cope well with TCP losses, which introduce huge latencies.
3955
3956The "netstat -i" command will report statistics per interface. An interface
3957where the Rx-Ovr counter grows indicates that the system doesn't have enough
3958resources to receive all incoming packets and that they're lost before being
3959processed by the network driver. Rx-Drp indicates that some received packets
3960were lost in the network stack because the application doesn't process them
3961fast enough. This can happen during some attacks as well. Tx-Drp means that
3962the output queues were full and packets had to be dropped. When using TCP it
Dan Lloyd8e48b872016-07-01 21:01:18 -04003963should be very rare, but will possibly indicate a saturated outgoing link.
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003964
3965
396613. Security considerations
3967---------------------------
3968
3969HAProxy is designed to run with very limited privileges. The standard way to
3970use it is to isolate it into a chroot jail and to drop its privileges to a
3971non-root user without any permissions inside this jail so that if any future
3972vulnerability were to be discovered, its compromise would not affect the rest
3973of the system.
3974
Dan Lloyd8e48b872016-07-01 21:01:18 -04003975In order to perform a chroot, it first needs to be started as a root user. It is
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003976pointless to build hand-made chroots to start the process there, these ones are
3977painful to build, are never properly maintained and always contain way more
3978bugs than the main file-system. And in case of compromise, the intruder can use
3979the purposely built file-system. Unfortunately many administrators confuse
3980"start as root" and "run as root", resulting in the uid change to be done prior
3981to starting haproxy, and reducing the effective security restrictions.
3982
3983HAProxy will need to be started as root in order to :
3984 - adjust the file descriptor limits
3985 - bind to privileged port numbers
3986 - bind to a specific network interface
3987 - transparently listen to a foreign address
3988 - isolate itself inside the chroot jail
3989 - drop to another non-privileged UID
3990
3991HAProxy may require to be run as root in order to :
3992 - bind to an interface for outgoing connections
3993 - bind to privileged source ports for outgoing connections
Dan Lloyd8e48b872016-07-01 21:01:18 -04003994 - transparently bind to a foreign address for outgoing connections
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003995
3996Most users will never need the "run as root" case. But the "start as root"
3997covers most usages.
3998
3999A safe configuration will have :
4000
4001 - a chroot statement pointing to an empty location without any access
4002 permissions. This can be prepared this way on the UNIX command line :
4003
4004 # mkdir /var/empty && chmod 0 /var/empty || echo "Failed"
4005
4006 and referenced like this in the HAProxy configuration's global section :
4007
4008 chroot /var/empty
4009
4010 - both a uid/user and gid/group statements in the global section :
4011
4012 user haproxy
4013 group haproxy
4014
4015 - a stats socket whose mode, uid and gid are set to match the user and/or
4016 group allowed to access the CLI so that nobody may access it :
4017
4018 stats socket /var/run/haproxy.stat uid hatop gid hatop mode 600
4019