blob: 922c13338d57fbade79625c72521f1b922e33ac0 [file] [log] [blame]
Willy Tarreau2212e6a2015-10-13 14:40:55 +02001 ------------------------
2 HAProxy Management Guide
3 ------------------------
Willy Tarreaufba74ea2018-12-22 11:19:45 +01004 version 2.0
Willy Tarreau2212e6a2015-10-13 14:40:55 +02005
6
7This document describes how to start, stop, manage, and troubleshoot HAProxy,
8as well as some known limitations and traps to avoid. It does not describe how
9to configure it (for this please read configuration.txt).
10
11Note to documentation contributors :
12 This document is formatted with 80 columns per line, with even number of
13 spaces for indentation and without tabs. Please follow these rules strictly
14 so that it remains easily printable everywhere. If you add sections, please
15 update the summary below for easier searching.
16
17
18Summary
19-------
20
211. Prerequisites
222. Quick reminder about HAProxy's architecture
233. Starting HAProxy
244. Stopping and restarting HAProxy
255. File-descriptor limitations
266. Memory management
277. CPU usage
288. Logging
299. Statistics and monitoring
Willy Tarreau44aed902015-10-13 14:45:29 +0200309.1. CSV format
Willy Tarreau5d8b9792016-03-11 11:09:34 +0100319.2. Typed output format
329.3. Unix Socket commands
William Lallemand142db372018-12-11 18:56:45 +0100339.4. Master CLI
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003410. Tricks for easier configuration management
3511. Well-known traps to avoid
3612. Debugging and performance issues
3713. Security considerations
38
39
401. Prerequisites
41----------------
42
43In this document it is assumed that the reader has sufficient administration
44skills on a UNIX-like operating system, uses the shell on a daily basis and is
45familiar with troubleshooting utilities such as strace and tcpdump.
46
47
482. Quick reminder about HAProxy's architecture
49----------------------------------------------
50
Willy Tarreau3f364482019-02-27 15:01:46 +010051HAProxy is a multi-threaded, event-driven, non-blocking daemon. This means is
Willy Tarreau2212e6a2015-10-13 14:40:55 +020052uses event multiplexing to schedule all of its activities instead of relying on
53the system to schedule between multiple activities. Most of the time it runs as
54a single process, so the output of "ps aux" on a system will report only one
55"haproxy" process, unless a soft reload is in progress and an older process is
56finishing its job in parallel to the new one. It is thus always easy to trace
Willy Tarreau3f364482019-02-27 15:01:46 +010057its activity using the strace utility. In order to scale with the number of
58available processors, by default haproxy will start one worker thread per
59processor it is allowed to run on. Unless explicitly configured differently,
60the incoming traffic is spread over all these threads, all running the same
61event loop. A great care is taken to limit inter-thread dependencies to the
62strict minimum, so as to try to achieve near-linear scalability. This has some
63impacts such as the fact that a given connection is served by a single thread.
64Thus in order to use all available processing capacity, it is needed to have at
65least as many connections as there are threads, which is almost always granted.
Willy Tarreau2212e6a2015-10-13 14:40:55 +020066
67HAProxy is designed to isolate itself into a chroot jail during startup, where
68it cannot perform any file-system access at all. This is also true for the
69libraries it depends on (eg: libc, libssl, etc). The immediate effect is that
70a running process will not be able to reload a configuration file to apply
71changes, instead a new process will be started using the updated configuration
72file. Some other less obvious effects are that some timezone files or resolver
73files the libc might attempt to access at run time will not be found, though
74this should generally not happen as they're not needed after startup. A nice
75consequence of this principle is that the HAProxy process is totally stateless,
76and no cleanup is needed after it's killed, so any killing method that works
77will do the right thing.
78
79HAProxy doesn't write log files, but it relies on the standard syslog protocol
80to send logs to a remote server (which is often located on the same system).
81
82HAProxy uses its internal clock to enforce timeouts, that is derived from the
83system's time but where unexpected drift is corrected. This is done by limiting
84the time spent waiting in poll() for an event, and measuring the time it really
85took. In practice it never waits more than one second. This explains why, when
86running strace over a completely idle process, periodic calls to poll() (or any
87of its variants) surrounded by two gettimeofday() calls are noticed. They are
88normal, completely harmless and so cheap that the load they imply is totally
89undetectable at the system scale, so there's nothing abnormal there. Example :
90
91 16:35:40.002320 gettimeofday({1442759740, 2605}, NULL) = 0
92 16:35:40.002942 epoll_wait(0, {}, 200, 1000) = 0
93 16:35:41.007542 gettimeofday({1442759741, 7641}, NULL) = 0
94 16:35:41.007998 gettimeofday({1442759741, 8114}, NULL) = 0
95 16:35:41.008391 epoll_wait(0, {}, 200, 1000) = 0
96 16:35:42.011313 gettimeofday({1442759742, 11411}, NULL) = 0
97
98HAProxy is a TCP proxy, not a router. It deals with established connections that
99have been validated by the kernel, and not with packets of any form nor with
100sockets in other states (eg: no SYN_RECV nor TIME_WAIT), though their existence
101may prevent it from binding a port. It relies on the system to accept incoming
102connections and to initiate outgoing connections. An immediate effect of this is
103that there is no relation between packets observed on the two sides of a
104forwarded connection, which can be of different size, numbers and even family.
105Since a connection may only be accepted from a socket in LISTEN state, all the
106sockets it is listening to are necessarily visible using the "netstat" utility
107to show listening sockets. Example :
108
109 # netstat -ltnp
110 Active Internet connections (only servers)
111 Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
112 tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1629/sshd
113 tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2847/haproxy
114 tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 2847/haproxy
115
116
1173. Starting HAProxy
118-------------------
119
120HAProxy is started by invoking the "haproxy" program with a number of arguments
121passed on the command line. The actual syntax is :
122
123 $ haproxy [<options>]*
124
125where [<options>]* is any number of options. An option always starts with '-'
126followed by one of more letters, and possibly followed by one or multiple extra
127arguments. Without any option, HAProxy displays the help page with a reminder
128about supported options. Available options may vary slightly based on the
129operating system. A fair number of these options overlap with an equivalent one
130if the "global" section. In this case, the command line always has precedence
131over the configuration file, so that the command line can be used to quickly
132enforce some settings without touching the configuration files. The current
133list of options is :
134
135 -- <cfgfile>* : all the arguments following "--" are paths to configuration
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200136 file/directory to be loaded and processed in the declaration order. It is
137 mostly useful when relying on the shell to load many files that are
138 numerically ordered. See also "-f". The difference between "--" and "-f" is
139 that one "-f" must be placed before each file name, while a single "--" is
140 needed before all file names. Both options can be used together, the
141 command line ordering still applies. When more than one file is specified,
142 each file must start on a section boundary, so the first keyword of each
143 file must be one of "global", "defaults", "peers", "listen", "frontend",
144 "backend", and so on. A file cannot contain just a server list for example.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200145
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200146 -f <cfgfile|cfgdir> : adds <cfgfile> to the list of configuration files to be
147 loaded. If <cfgdir> is a directory, all the files (and only files) it
Dan Lloyd8e48b872016-07-01 21:01:18 -0400148 contains are added in lexical order (using LC_COLLATE=C) to the list of
Maxime de Roucy379d9c72016-05-13 23:52:56 +0200149 configuration files to be loaded ; only files with ".cfg" extension are
150 added, only non hidden files (not prefixed with ".") are added.
151 Configuration files are loaded and processed in their declaration order.
152 This option may be specified multiple times to load multiple files. See
153 also "--". The difference between "--" and "-f" is that one "-f" must be
154 placed before each file name, while a single "--" is needed before all file
155 names. Both options can be used together, the command line ordering still
156 applies. When more than one file is specified, each file must start on a
157 section boundary, so the first keyword of each file must be one of
158 "global", "defaults", "peers", "listen", "frontend", "backend", and so on.
159 A file cannot contain just a server list for example.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200160
161 -C <dir> : changes to directory <dir> before loading configuration
162 files. This is useful when using relative paths. Warning when using
163 wildcards after "--" which are in fact replaced by the shell before
164 starting haproxy.
165
166 -D : start as a daemon. The process detaches from the current terminal after
167 forking, and errors are not reported anymore in the terminal. It is
168 equivalent to the "daemon" keyword in the "global" section of the
169 configuration. It is recommended to always force it in any init script so
170 that a faulty configuration doesn't prevent the system from booting.
171
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200172 -L <name> : change the local peer name to <name>, which defaults to the local
William Lallemanddaf4cd22018-04-17 16:46:13 +0200173 hostname. This is used only with peers replication. You can use the
174 variable $HAPROXY_LOCALPEER in the configuration file to reference the
175 peer name.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200176
177 -N <limit> : sets the default per-proxy maxconn to <limit> instead of the
178 builtin default value (usually 2000). Only useful for debugging.
179
180 -V : enable verbose mode (disables quiet mode). Reverts the effect of "-q" or
181 "quiet".
182
William Lallemande202b1e2017-06-01 17:38:56 +0200183 -W : master-worker mode. It is equivalent to the "master-worker" keyword in
184 the "global" section of the configuration. This mode will launch a "master"
185 which will monitor the "workers". Using this mode, you can reload HAProxy
186 directly by sending a SIGUSR2 signal to the master. The master-worker mode
187 is compatible either with the foreground or daemon mode. It is
188 recommended to use this mode with multiprocess and systemd.
189
Pavlos Parissisf65f2572018-02-07 21:42:16 +0100190 -Ws : master-worker mode with support of `notify` type of systemd service.
191 This option is only available when HAProxy was built with `USE_SYSTEMD`
192 build option enabled.
193
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200194 -c : only performs a check of the configuration files and exits before trying
195 to bind. The exit status is zero if everything is OK, or non-zero if an
196 error is encountered.
197
198 -d : enable debug mode. This disables daemon mode, forces the process to stay
199 in foreground and to show incoming and outgoing events. It is equivalent to
200 the "global" section's "debug" keyword. It must never be used in an init
201 script.
202
203 -dG : disable use of getaddrinfo() to resolve host names into addresses. It
204 can be used when suspecting that getaddrinfo() doesn't work as expected.
205 This option was made available because many bogus implementations of
206 getaddrinfo() exist on various systems and cause anomalies that are
207 difficult to troubleshoot.
208
Dan Lloyd8e48b872016-07-01 21:01:18 -0400209 -dM[<byte>] : forces memory poisoning, which means that each and every
Willy Tarreaubafbe012017-11-24 17:34:44 +0100210 memory region allocated with malloc() or pool_alloc() will be filled with
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200211 <byte> before being passed to the caller. When <byte> is not specified, it
212 defaults to 0x50 ('P'). While this slightly slows down operations, it is
213 useful to reliably trigger issues resulting from missing initializations in
214 the code that cause random crashes. Note that -dM0 has the effect of
215 turning any malloc() into a calloc(). In any case if a bug appears or
216 disappears when using this option it means there is a bug in haproxy, so
217 please report it.
218
219 -dS : disable use of the splice() system call. It is equivalent to the
220 "global" section's "nosplice" keyword. This may be used when splice() is
221 suspected to behave improperly or to cause performance issues, or when
222 using strace to see the forwarded data (which do not appear when using
223 splice()).
224
225 -dV : disable SSL verify on the server side. It is equivalent to having
226 "ssl-server-verify none" in the "global" section. This is useful when
227 trying to reproduce production issues out of the production
228 environment. Never use this in an init script as it degrades SSL security
229 to the servers.
230
231 -db : disable background mode and multi-process mode. The process remains in
232 foreground. It is mainly used during development or during small tests, as
233 Ctrl-C is enough to stop the process. Never use it in an init script.
234
235 -de : disable the use of the "epoll" poller. It is equivalent to the "global"
236 section's keyword "noepoll". It is mostly useful when suspecting a bug
237 related to this poller. On systems supporting epoll, the fallback will
238 generally be the "poll" poller.
239
240 -dk : disable the use of the "kqueue" poller. It is equivalent to the
241 "global" section's keyword "nokqueue". It is mostly useful when suspecting
242 a bug related to this poller. On systems supporting kqueue, the fallback
243 will generally be the "poll" poller.
244
245 -dp : disable the use of the "poll" poller. It is equivalent to the "global"
246 section's keyword "nopoll". It is mostly useful when suspecting a bug
247 related to this poller. On systems supporting poll, the fallback will
248 generally be the "select" poller, which cannot be disabled and is limited
249 to 1024 file descriptors.
250
Willy Tarreau3eed10e2016-11-07 21:03:16 +0100251 -dr : ignore server address resolution failures. It is very common when
252 validating a configuration out of production not to have access to the same
253 resolvers and to fail on server address resolution, making it difficult to
254 test a configuration. This option simply appends the "none" method to the
255 list of address resolution methods for all servers, ensuring that even if
256 the libc fails to resolve an address, the startup sequence is not
257 interrupted.
258
Willy Tarreau70060452015-12-14 12:46:07 +0100259 -m <limit> : limit the total allocatable memory to <limit> megabytes across
260 all processes. This may cause some connection refusals or some slowdowns
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200261 depending on the amount of memory needed for normal operations. This is
Willy Tarreau70060452015-12-14 12:46:07 +0100262 mostly used to force the processes to work in a constrained resource usage
263 scenario. It is important to note that the memory is not shared between
264 processes, so in a multi-process scenario, this value is first divided by
265 global.nbproc before forking.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200266
267 -n <limit> : limits the per-process connection limit to <limit>. This is
268 equivalent to the global section's keyword "maxconn". It has precedence
269 over this keyword. This may be used to quickly force lower limits to avoid
270 a service outage on systems where resource limits are too low.
271
272 -p <file> : write all processes' pids into <file> during startup. This is
273 equivalent to the "global" section's keyword "pidfile". The file is opened
274 before entering the chroot jail, and after doing the chdir() implied by
275 "-C". Each pid appears on its own line.
276
277 -q : set "quiet" mode. This disables some messages during the configuration
278 parsing and during startup. It can be used in combination with "-c" to
279 just check if a configuration file is valid or not.
280
William Lallemand142db372018-12-11 18:56:45 +0100281 -S <bind>[,bind_options...]: in master-worker mode, bind a master CLI, which
282 allows the access to every processes, running or leaving ones.
283 For security reasons, it is recommended to bind the master CLI to a local
284 UNIX socket. The bind options are the same as the keyword "bind" in
285 the configuration file with words separated by commas instead of spaces.
286
287 Note that this socket can't be used to retrieve the listening sockets from
288 an old process during a seamless reload.
289
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200290 -sf <pid>* : send the "finish" signal (SIGUSR1) to older processes after boot
291 completion to ask them to finish what they are doing and to leave. <pid>
292 is a list of pids to signal (one per argument). The list ends on any
293 option starting with a "-". It is not a problem if the list of pids is
294 empty, so that it can be built on the fly based on the result of a command
295 like "pidof" or "pgrep".
296
297 -st <pid>* : send the "terminate" signal (SIGTERM) to older processes after
298 boot completion to terminate them immediately without finishing what they
299 were doing. <pid> is a list of pids to signal (one per argument). The list
300 is ends on any option starting with a "-". It is not a problem if the list
301 of pids is empty, so that it can be built on the fly based on the result of
302 a command like "pidof" or "pgrep".
303
304 -v : report the version and build date.
305
306 -vv : display the version, build options, libraries versions and usable
307 pollers. This output is systematically requested when filing a bug report.
308
Olivier Houchardd33fc3a2017-04-05 22:50:59 +0200309 -x <unix_socket> : connect to the specified socket and try to retrieve any
310 listening sockets from the old process, and use them instead of trying to
311 bind new ones. This is useful to avoid missing any new connection when
William Lallemandf6975e92017-05-26 17:42:10 +0200312 reloading the configuration on Linux. The capability must be enable on the
313 stats socket using "expose-fd listeners" in your configuration.
Olivier Houchardd33fc3a2017-04-05 22:50:59 +0200314
Dan Lloyd8e48b872016-07-01 21:01:18 -0400315A safe way to start HAProxy from an init file consists in forcing the daemon
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200316mode, storing existing pids to a pid file and using this pid file to notify
317older processes to finish before leaving :
318
319 haproxy -f /etc/haproxy.cfg \
320 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
321
322When the configuration is split into a few specific files (eg: tcp vs http),
323it is recommended to use the "-f" option :
324
325 haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
326 -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
327 -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
328 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
329
330When an unknown number of files is expected, such as customer-specific files,
331it is recommended to assign them a name starting with a fixed-size sequence
332number and to use "--" to load them, possibly after loading some defaults :
333
334 haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
335 -f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
336 -f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
337 -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) \
338 -f /etc/haproxy/default-customers.cfg -- /etc/haproxy/customers/*
339
340Sometimes a failure to start may happen for whatever reason. Then it is
341important to verify if the version of HAProxy you are invoking is the expected
342version and if it supports the features you are expecting (eg: SSL, PCRE,
343compression, Lua, etc). This can be verified using "haproxy -vv". Some
344important information such as certain build options, the target system and
345the versions of the libraries being used are reported there. It is also what
346you will systematically be asked for when posting a bug report :
347
348 $ haproxy -vv
349 HA-Proxy version 1.6-dev7-a088d3-4 2015/10/08
350 Copyright 2000-2015 Willy Tarreau <willy@haproxy.org>
351
352 Build options :
353 TARGET = linux2628
354 CPU = generic
355 CC = gcc
356 CFLAGS = -pg -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement \
357 -DBUFSIZE=8030 -DMAXREWRITE=1030 -DSO_MARK=36 -DTCP_REPAIR=19
358 OPTIONS = USE_ZLIB=1 USE_DLMALLOC=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
359
360 Default settings :
361 maxconn = 2000, bufsize = 8030, maxrewrite = 1030, maxpollevents = 200
362
363 Encrypted password support via crypt(3): yes
364 Built with zlib version : 1.2.6
365 Compression algorithms supported : identity("identity"), deflate("deflate"), \
366 raw-deflate("deflate"), gzip("gzip")
367 Built with OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
368 Running on OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
369 OpenSSL library supports TLS extensions : yes
370 OpenSSL library supports SNI : yes
371 OpenSSL library supports prefer-server-ciphers : yes
372 Built with PCRE version : 8.12 2011-01-15
373 PCRE library supports JIT : no (USE_PCRE_JIT not set)
374 Built with Lua version : Lua 5.3.1
375 Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND
376
377 Available polling systems :
378 epoll : pref=300, test result OK
379 poll : pref=200, test result OK
380 select : pref=150, test result OK
381 Total: 3 (3 usable), will use epoll.
382
383The relevant information that many non-developer users can verify here are :
384 - the version : 1.6-dev7-a088d3-4 above means the code is currently at commit
385 ID "a088d3" which is the 4th one after after official version "1.6-dev7".
386 Version 1.6-dev7 would show as "1.6-dev7-8c1ad7". What matters here is in
387 fact "1.6-dev7". This is the 7th development version of what will become
388 version 1.6 in the future. A development version not suitable for use in
389 production (unless you know exactly what you are doing). A stable version
390 will show as a 3-numbers version, such as "1.5.14-16f863", indicating the
391 14th level of fix on top of version 1.5. This is a production-ready version.
392
393 - the release date : 2015/10/08. It is represented in the universal
394 year/month/day format. Here this means August 8th, 2015. Given that stable
395 releases are issued every few months (1-2 months at the beginning, sometimes
396 6 months once the product becomes very stable), if you're seeing an old date
397 here, it means you're probably affected by a number of bugs or security
398 issues that have since been fixed and that it might be worth checking on the
399 official site.
400
401 - build options : they are relevant to people who build their packages
402 themselves, they can explain why things are not behaving as expected. For
403 example the development version above was built for Linux 2.6.28 or later,
Dan Lloyd8e48b872016-07-01 21:01:18 -0400404 targeting a generic CPU (no CPU-specific optimizations), and lacks any
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200405 code optimization (-O0) so it will perform poorly in terms of performance.
406
407 - libraries versions : zlib version is reported as found in the library
408 itself. In general zlib is considered a very stable product and upgrades
409 are almost never needed. OpenSSL reports two versions, the version used at
410 build time and the one being used, as found on the system. These ones may
411 differ by the last letter but never by the numbers. The build date is also
412 reported because most OpenSSL bugs are security issues and need to be taken
413 seriously, so this library absolutely needs to be kept up to date. Seeing a
414 4-months old version here is highly suspicious and indeed an update was
415 missed. PCRE provides very fast regular expressions and is highly
416 recommended. Certain of its extensions such as JIT are not present in all
417 versions and still young so some people prefer not to build with them,
Dan Lloyd8e48b872016-07-01 21:01:18 -0400418 which is why the build status is reported as well. Regarding the Lua
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200419 scripting language, HAProxy expects version 5.3 which is very young since
420 it was released a little time before HAProxy 1.6. It is important to check
421 on the Lua web site if some fixes are proposed for this branch.
422
423 - Available polling systems will affect the process's scalability when
424 dealing with more than about one thousand of concurrent connections. These
425 ones are only available when the correct system was indicated in the TARGET
426 variable during the build. The "epoll" mechanism is highly recommended on
427 Linux, and the kqueue mechanism is highly recommended on BSD. Lacking them
428 will result in poll() or even select() being used, causing a high CPU usage
429 when dealing with a lot of connections.
430
431
4324. Stopping and restarting HAProxy
433----------------------------------
434
435HAProxy supports a graceful and a hard stop. The hard stop is simple, when the
436SIGTERM signal is sent to the haproxy process, it immediately quits and all
437established connections are closed. The graceful stop is triggered when the
438SIGUSR1 signal is sent to the haproxy process. It consists in only unbinding
439from listening ports, but continue to process existing connections until they
440close. Once the last connection is closed, the process leaves.
441
442The hard stop method is used for the "stop" or "restart" actions of the service
443management script. The graceful stop is used for the "reload" action which
444tries to seamlessly reload a new configuration in a new process.
445
446Both of these signals may be sent by the new haproxy process itself during a
447reload or restart, so that they are sent at the latest possible moment and only
448if absolutely required. This is what is performed by the "-st" (hard) and "-sf"
449(graceful) options respectively.
450
William Lallemande202b1e2017-06-01 17:38:56 +0200451In master-worker mode, it is not needed to start a new haproxy process in
452order to reload the configuration. The master process reacts to the SIGUSR2
453signal by reexecuting itself with the -sf parameter followed by the PIDs of
454the workers. The master will then parse the configuration file and fork new
455workers.
456
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200457To understand better how these signals are used, it is important to understand
458the whole restart mechanism.
459
460First, an existing haproxy process is running. The administrator uses a system
461specific command such as "/etc/init.d/haproxy reload" to indicate he wants to
462take the new configuration file into effect. What happens then is the following.
463First, the service script (/etc/init.d/haproxy or equivalent) will verify that
464the configuration file parses correctly using "haproxy -c". After that it will
465try to start haproxy with this configuration file, using "-st" or "-sf".
466
467Then HAProxy tries to bind to all listening ports. If some fatal errors happen
468(eg: address not present on the system, permission denied), the process quits
469with an error. If a socket binding fails because a port is already in use, then
470the process will first send a SIGTTOU signal to all the pids specified in the
471"-st" or "-sf" pid list. This is what is called the "pause" signal. It instructs
472all existing haproxy processes to temporarily stop listening to their ports so
473that the new process can try to bind again. During this time, the old process
474continues to process existing connections. If the binding still fails (because
475for example a port is shared with another daemon), then the new process sends a
476SIGTTIN signal to the old processes to instruct them to resume operations just
477as if nothing happened. The old processes will then restart listening to the
478ports and continue to accept connections. Not that this mechanism is system
Dan Lloyd8e48b872016-07-01 21:01:18 -0400479dependent and some operating systems may not support it in multi-process mode.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200480
481If the new process manages to bind correctly to all ports, then it sends either
482the SIGTERM (hard stop in case of "-st") or the SIGUSR1 (graceful stop in case
483of "-sf") to all processes to notify them that it is now in charge of operations
484and that the old processes will have to leave, either immediately or once they
485have finished their job.
486
487It is important to note that during this timeframe, there are two small windows
488of a few milliseconds each where it is possible that a few connection failures
489will be noticed during high loads. Typically observed failure rates are around
4901 failure during a reload operation every 10000 new connections per second,
491which means that a heavily loaded site running at 30000 new connections per
492second may see about 3 failed connection upon every reload. The two situations
493where this happens are :
494
495 - if the new process fails to bind due to the presence of the old process,
496 it will first have to go through the SIGTTOU+SIGTTIN sequence, which
497 typically lasts about one millisecond for a few tens of frontends, and
498 during which some ports will not be bound to the old process and not yet
499 bound to the new one. HAProxy works around this on systems that support the
500 SO_REUSEPORT socket options, as it allows the new process to bind without
501 first asking the old one to unbind. Most BSD systems have been supporting
502 this almost forever. Linux has been supporting this in version 2.0 and
503 dropped it around 2.2, but some patches were floating around by then. It
504 was reintroduced in kernel 3.9, so if you are observing a connection
Dan Lloyd8e48b872016-07-01 21:01:18 -0400505 failure rate above the one mentioned above, please ensure that your kernel
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200506 is 3.9 or newer, or that relevant patches were backported to your kernel
507 (less likely).
508
509 - when the old processes close the listening ports, the kernel may not always
510 redistribute any pending connection that was remaining in the socket's
511 backlog. Under high loads, a SYN packet may happen just before the socket
512 is closed, and will lead to an RST packet being sent to the client. In some
513 critical environments where even one drop is not acceptable, these ones are
514 sometimes dealt with using firewall rules to block SYN packets during the
515 reload, forcing the client to retransmit. This is totally system-dependent,
516 as some systems might be able to visit other listening queues and avoid
517 this RST. A second case concerns the ACK from the client on a local socket
518 that was in SYN_RECV state just before the close. This ACK will lead to an
519 RST packet while the haproxy process is still not aware of it. This one is
Dan Lloyd8e48b872016-07-01 21:01:18 -0400520 harder to get rid of, though the firewall filtering rules mentioned above
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200521 will work well if applied one second or so before restarting the process.
522
523For the vast majority of users, such drops will never ever happen since they
524don't have enough load to trigger the race conditions. And for most high traffic
525users, the failure rate is still fairly within the noise margin provided that at
526least SO_REUSEPORT is properly supported on their systems.
527
528
5295. File-descriptor limitations
530------------------------------
531
532In order to ensure that all incoming connections will successfully be served,
533HAProxy computes at load time the total number of file descriptors that will be
534needed during the process's life. A regular Unix process is generally granted
5351024 file descriptors by default, and a privileged process can raise this limit
536itself. This is one reason for starting HAProxy as root and letting it adjust
537the limit. The default limit of 1024 file descriptors roughly allow about 500
538concurrent connections to be processed. The computation is based on the global
539maxconn parameter which limits the total number of connections per process, the
540number of listeners, the number of servers which have a health check enabled,
541the agent checks, the peers, the loggers and possibly a few other technical
542requirements. A simple rough estimate of this number consists in simply
543doubling the maxconn value and adding a few tens to get the approximate number
544of file descriptors needed.
545
546Originally HAProxy did not know how to compute this value, and it was necessary
547to pass the value using the "ulimit-n" setting in the global section. This
548explains why even today a lot of configurations are seen with this setting
549present. Unfortunately it was often miscalculated resulting in connection
550failures when approaching maxconn instead of throttling incoming connection
551while waiting for the needed resources. For this reason it is important to
Dan Lloyd8e48b872016-07-01 21:01:18 -0400552remove any vestigial "ulimit-n" setting that can remain from very old versions.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200553
554Raising the number of file descriptors to accept even moderate loads is
555mandatory but comes with some OS-specific adjustments. First, the select()
556polling system is limited to 1024 file descriptors. In fact on Linux it used
557to be capable of handling more but since certain OS ship with excessively
558restrictive SELinux policies forbidding the use of select() with more than
5591024 file descriptors, HAProxy now refuses to start in this case in order to
560avoid any issue at run time. On all supported operating systems, poll() is
561available and will not suffer from this limitation. It is automatically picked
Dan Lloyd8e48b872016-07-01 21:01:18 -0400562so there is nothing to do to get a working configuration. But poll's becomes
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200563very slow when the number of file descriptors increases. While HAProxy does its
564best to limit this performance impact (eg: via the use of the internal file
565descriptor cache and batched processing), a good rule of thumb is that using
566poll() with more than a thousand concurrent connections will use a lot of CPU.
567
568For Linux systems base on kernels 2.6 and above, the epoll() system call will
569be used. It's a much more scalable mechanism relying on callbacks in the kernel
570that guarantee a constant wake up time regardless of the number of registered
571monitored file descriptors. It is automatically used where detected, provided
572that HAProxy had been built for one of the Linux flavors. Its presence and
573support can be verified using "haproxy -vv".
574
575For BSD systems which support it, kqueue() is available as an alternative. It
576is much faster than poll() and even slightly faster than epoll() thanks to its
577batched handling of changes. At least FreeBSD and OpenBSD support it. Just like
578with Linux's epoll(), its support and availability are reported in the output
579of "haproxy -vv".
580
581Having a good poller is one thing, but it is mandatory that the process can
582reach the limits. When HAProxy starts, it immediately sets the new process's
583file descriptor limits and verifies if it succeeds. In case of failure, it
584reports it before forking so that the administrator can see the problem. As
585long as the process is started by as root, there should be no reason for this
586setting to fail. However, it can fail if the process is started by an
587unprivileged user. If there is a compelling reason for *not* starting haproxy
588as root (eg: started by end users, or by a per-application account), then the
589file descriptor limit can be raised by the system administrator for this
590specific user. The effectiveness of the setting can be verified by issuing
591"ulimit -n" from the user's command line. It should reflect the new limit.
592
593Warning: when an unprivileged user's limits are changed in this user's account,
594it is fairly common that these values are only considered when the user logs in
595and not at all in some scripts run at system boot time nor in crontabs. This is
596totally dependent on the operating system, keep in mind to check "ulimit -n"
597before starting haproxy when running this way. The general advice is never to
598start haproxy as an unprivileged user for production purposes. Another good
599reason is that it prevents haproxy from enabling some security protections.
600
601Once it is certain that the system will allow the haproxy process to use the
602requested number of file descriptors, two new system-specific limits may be
603encountered. The first one is the system-wide file descriptor limit, which is
604the total number of file descriptors opened on the system, covering all
605processes. When this limit is reached, accept() or socket() will typically
606return ENFILE. The second one is the per-process hard limit on the number of
607file descriptors, it prevents setrlimit() from being set higher. Both are very
608dependent on the operating system. On Linux, the system limit is set at boot
609based on the amount of memory. It can be changed with the "fs.file-max" sysctl.
610And the per-process hard limit is set to 1048576 by default, but it can be
611changed using the "fs.nr_open" sysctl.
612
613File descriptor limitations may be observed on a running process when they are
614set too low. The strace utility will report that accept() and socket() return
615"-1 EMFILE" when the process's limits have been reached. In this case, simply
616raising the "ulimit-n" value (or removing it) will solve the problem. If these
617system calls return "-1 ENFILE" then it means that the kernel's limits have
618been reached and that something must be done on a system-wide parameter. These
619trouble must absolutely be addressed, as they result in high CPU usage (when
620accept() fails) and failed connections that are generally visible to the user.
621One solution also consists in lowering the global maxconn value to enforce
622serialization, and possibly to disable HTTP keep-alive to force connections
623to be released and reused faster.
624
625
6266. Memory management
627--------------------
628
629HAProxy uses a simple and fast pool-based memory management. Since it relies on
630a small number of different object types, it's much more efficient to pick new
631objects from a pool which already contains objects of the appropriate size than
632to call malloc() for each different size. The pools are organized as a stack or
633LIFO, so that newly allocated objects are taken from recently released objects
634still hot in the CPU caches. Pools of similar sizes are merged together, in
635order to limit memory fragmentation.
636
637By default, since the focus is set on performance, each released object is put
638back into the pool it came from, and allocated objects are never freed since
639they are expected to be reused very soon.
640
641On the CLI, it is possible to check how memory is being used in pools thanks to
642the "show pools" command :
643
644 > show pools
645 Dumping pools usage. Use SIGQUIT to flush them.
Willy Tarreau0a93b642018-10-16 07:58:39 +0200646 - Pool cache_st (16 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccc40=03 [SHARED]
647 - Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 0 failures, 2 users, @0x9ccac0=00 [SHARED]
648 - Pool comp_state (48 bytes) : 3 allocated (144 bytes), 3 used, 0 failures, 5 users, @0x9cccc0=04 [SHARED]
649 - Pool filter (64 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 3 users, @0x9ccbc0=02 [SHARED]
650 - Pool vars (80 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccb40=01 [SHARED]
651 - Pool uniqueid (128 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9cd240=15 [SHARED]
652 - Pool task (144 bytes) : 55 allocated (7920 bytes), 55 used, 0 failures, 1 users, @0x9cd040=11 [SHARED]
653 - Pool session (160 bytes) : 1 allocated (160 bytes), 1 used, 0 failures, 1 users, @0x9cd140=13 [SHARED]
654 - Pool h2s (208 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccec0=08 [SHARED]
655 - Pool h2c (288 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cce40=07 [SHARED]
656 - Pool spoe_ctx (304 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccf40=09 [SHARED]
657 - Pool connection (400 bytes) : 2 allocated (800 bytes), 2 used, 0 failures, 1 users, @0x9cd1c0=14 [SHARED]
658 - Pool hdr_idx (416 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cd340=17 [SHARED]
659 - Pool dns_resolut (480 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccdc0=06 [SHARED]
660 - Pool dns_answer_ (576 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccd40=05 [SHARED]
661 - Pool stream (960 bytes) : 1 allocated (960 bytes), 1 used, 0 failures, 1 users, @0x9cd0c0=12 [SHARED]
662 - Pool requri (1024 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cd2c0=16 [SHARED]
663 - Pool buffer (8030 bytes) : 3 allocated (24090 bytes), 2 used, 0 failures, 1 users, @0x9cd3c0=18 [SHARED]
664 - Pool trash (8062 bytes) : 1 allocated (8062 bytes), 1 used, 0 failures, 1 users, @0x9cd440=19
665 Total: 19 pools, 42296 bytes allocated, 34266 used.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200666
667The pool name is only indicative, it's the name of the first object type using
668this pool. The size in parenthesis is the object size for objects in this pool.
669Object sizes are always rounded up to the closest multiple of 16 bytes. The
670number of objects currently allocated and the equivalent number of bytes is
671reported so that it is easy to know which pool is responsible for the highest
672memory usage. The number of objects currently in use is reported as well in the
673"used" field. The difference between "allocated" and "used" corresponds to the
Willy Tarreau0a93b642018-10-16 07:58:39 +0200674objects that have been freed and are available for immediate use. The address
675at the end of the line is the pool's address, and the following number is the
676pool index when it exists, or is reported as -1 if no index was assigned.
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200677
678It is possible to limit the amount of memory allocated per process using the
679"-m" command line option, followed by a number of megabytes. It covers all of
680the process's addressable space, so that includes memory used by some libraries
681as well as the stack, but it is a reliable limit when building a resource
682constrained system. It works the same way as "ulimit -v" on systems which have
683it, or "ulimit -d" for the other ones.
684
685If a memory allocation fails due to the memory limit being reached or because
686the system doesn't have any enough memory, then haproxy will first start to
687free all available objects from all pools before attempting to allocate memory
688again. This mechanism of releasing unused memory can be triggered by sending
689the signal SIGQUIT to the haproxy process. When doing so, the pools state prior
690to the flush will also be reported to stderr when the process runs in
691foreground.
692
693During a reload operation, the process switched to the graceful stop state also
694automatically performs some flushes after releasing any connection so that all
695possible memory is released to save it for the new process.
696
697
6987. CPU usage
699------------
700
701HAProxy normally spends most of its time in the system and a smaller part in
702userland. A finely tuned 3.5 GHz CPU can sustain a rate about 80000 end-to-end
703connection setups and closes per second at 100% CPU on a single core. When one
704core is saturated, typical figures are :
705 - 95% system, 5% user for long TCP connections or large HTTP objects
706 - 85% system and 15% user for short TCP connections or small HTTP objects in
707 close mode
708 - 70% system and 30% user for small HTTP objects in keep-alive mode
709
710The amount of rules processing and regular expressions will increase the user
711land part. The presence of firewall rules, connection tracking, complex routing
712tables in the system will instead increase the system part.
713
714On most systems, the CPU time observed during network transfers can be cut in 4
715parts :
716 - the interrupt part, which concerns all the processing performed upon I/O
717 receipt, before the target process is even known. Typically Rx packets are
718 accounted for in interrupt. On some systems such as Linux where interrupt
719 processing may be deferred to a dedicated thread, it can appear as softirq,
720 and the thread is called ksoftirqd/0 (for CPU 0). The CPU taking care of
721 this load is generally defined by the hardware settings, though in the case
722 of softirq it is often possible to remap the processing to another CPU.
723 This interrupt part will often be perceived as parasitic since it's not
724 associated with any process, but it actually is some processing being done
725 to prepare the work for the process.
726
727 - the system part, which concerns all the processing done using kernel code
728 called from userland. System calls are accounted as system for example. All
729 synchronously delivered Tx packets will be accounted for as system time. If
730 some packets have to be deferred due to queues filling up, they may then be
731 processed in interrupt context later (eg: upon receipt of an ACK opening a
732 TCP window).
733
734 - the user part, which exclusively runs application code in userland. HAProxy
735 runs exclusively in this part, though it makes heavy use of system calls.
736 Rules processing, regular expressions, compression, encryption all add to
737 the user portion of CPU consumption.
738
739 - the idle part, which is what the CPU does when there is nothing to do. For
740 example HAProxy waits for an incoming connection, or waits for some data to
741 leave, meaning the system is waiting for an ACK from the client to push
742 these data.
743
744In practice regarding HAProxy's activity, it is in general reasonably accurate
745(but totally inexact) to consider that interrupt/softirq are caused by Rx
746processing in kernel drivers, that user-land is caused by layer 7 processing
747in HAProxy, and that system time is caused by network processing on the Tx
748path.
749
750Since HAProxy runs around an event loop, it waits for new events using poll()
751(or any alternative) and processes all these events as fast as possible before
752going back to poll() waiting for new events. It measures the time spent waiting
753in poll() compared to the time spent doing processing events. The ratio of
754polling time vs total time is called the "idle" time, it's the amount of time
755spent waiting for something to happen. This ratio is reported in the stats page
756on the "idle" line, or "Idle_pct" on the CLI. When it's close to 100%, it means
757the load is extremely low. When it's close to 0%, it means that there is
758constantly some activity. While it cannot be very accurate on an overloaded
759system due to other processes possibly preempting the CPU from the haproxy
760process, it still provides a good estimate about how HAProxy considers it is
761working : if the load is low and the idle ratio is low as well, it may indicate
762that HAProxy has a lot of work to do, possibly due to very expensive rules that
763have to be processed. Conversely, if HAProxy indicates the idle is close to
764100% while things are slow, it means that it cannot do anything to speed things
765up because it is already waiting for incoming data to process. In the example
766below, haproxy is completely idle :
767
768 $ echo "show info" | socat - /var/run/haproxy.sock | grep ^Idle
769 Idle_pct: 100
770
771When the idle ratio starts to become very low, it is important to tune the
772system and place processes and interrupts correctly to save the most possible
773CPU resources for all tasks. If a firewall is present, it may be worth trying
774to disable it or to tune it to ensure it is not responsible for a large part
775of the performance limitation. It's worth noting that unloading a stateful
776firewall generally reduces both the amount of interrupt/softirq and of system
777usage since such firewalls act both on the Rx and the Tx paths. On Linux,
778unloading the nf_conntrack and ip_conntrack modules will show whether there is
779anything to gain. If so, then the module runs with default settings and you'll
780have to figure how to tune it for better performance. In general this consists
781in considerably increasing the hash table size. On FreeBSD, "pfctl -d" will
782disable the "pf" firewall and its stateful engine at the same time.
783
784If it is observed that a lot of time is spent in interrupt/softirq, it is
785important to ensure that they don't run on the same CPU. Most systems tend to
786pin the tasks on the CPU where they receive the network traffic because for
787certain workloads it improves things. But with heavily network-bound workloads
788it is the opposite as the haproxy process will have to fight against its kernel
789counterpart. Pinning haproxy to one CPU core and the interrupts to another one,
790all sharing the same L3 cache tends to sensibly increase network performance
791because in practice the amount of work for haproxy and the network stack are
792quite close, so they can almost fill an entire CPU each. On Linux this is done
793using taskset (for haproxy) or using cpu-map (from the haproxy config), and the
794interrupts are assigned under /proc/irq. Many network interfaces support
795multiple queues and multiple interrupts. In general it helps to spread them
796across a small number of CPU cores provided they all share the same L3 cache.
797Please always stop irq_balance which always does the worst possible thing on
798such workloads.
799
800For CPU-bound workloads consisting in a lot of SSL traffic or a lot of
801compression, it may be worth using multiple processes dedicated to certain
802tasks, though there is no universal rule here and experimentation will have to
803be performed.
804
805In order to increase the CPU capacity, it is possible to make HAProxy run as
806several processes, using the "nbproc" directive in the global section. There
807are some limitations though :
808 - health checks are run per process, so the target servers will get as many
809 checks as there are running processes ;
810 - maxconn values and queues are per-process so the correct value must be set
811 to avoid overloading the servers ;
812 - outgoing connections should avoid using port ranges to avoid conflicts
813 - stick-tables are per process and are not shared between processes ;
814 - each peers section may only run on a single process at a time ;
815 - the CLI operations will only act on a single process at a time.
816
817With this in mind, it appears that the easiest setup often consists in having
818one first layer running on multiple processes and in charge for the heavy
819processing, passing the traffic to a second layer running in a single process.
820This mechanism is suited to SSL and compression which are the two CPU-heavy
821features. Instances can easily be chained over UNIX sockets (which are cheaper
fengpeiyuancc123c62016-01-15 16:40:53 +0800822than TCP sockets and which do not waste ports), and the proxy protocol which is
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200823useful to pass client information to the next stage. When doing so, it is
824generally a good idea to bind all the single-process tasks to process number 1
825and extra tasks to next processes, as this will make it easier to generate
826similar configurations for different machines.
827
828On Linux versions 3.9 and above, running HAProxy in multi-process mode is much
829more efficient when each process uses a distinct listening socket on the same
830IP:port ; this will make the kernel evenly distribute the load across all
831processes instead of waking them all up. Please check the "process" option of
832the "bind" keyword lines in the configuration manual for more information.
833
834
8358. Logging
836----------
837
838For logging, HAProxy always relies on a syslog server since it does not perform
839any file-system access. The standard way of using it is to send logs over UDP
840to the log server (by default on port 514). Very commonly this is configured to
841127.0.0.1 where the local syslog daemon is running, but it's also used over the
842network to log to a central server. The central server provides additional
843benefits especially in active-active scenarios where it is desirable to keep
844the logs merged in arrival order. HAProxy may also make use of a UNIX socket to
845send its logs to the local syslog daemon, but it is not recommended at all,
846because if the syslog server is restarted while haproxy runs, the socket will
847be replaced and new logs will be lost. Since HAProxy will be isolated inside a
848chroot jail, it will not have the ability to reconnect to the new socket. It
849has also been observed in field that the log buffers in use on UNIX sockets are
850very small and lead to lost messages even at very light loads. But this can be
851fine for testing however.
852
853It is recommended to add the following directive to the "global" section to
854make HAProxy log to the local daemon using facility "local0" :
855
856 log 127.0.0.1:514 local0
857
858and then to add the following one to each "defaults" section or to each frontend
859and backend section :
860
861 log global
862
863This way, all logs will be centralized through the global definition of where
864the log server is.
865
866Some syslog daemons do not listen to UDP traffic by default, so depending on
867the daemon being used, the syntax to enable this will vary :
868
869 - on sysklogd, you need to pass argument "-r" on the daemon's command line
870 so that it listens to a UDP socket for "remote" logs ; note that there is
871 no way to limit it to address 127.0.0.1 so it will also receive logs from
872 remote systems ;
873
874 - on rsyslogd, the following lines must be added to the configuration file :
875
876 $ModLoad imudp
877 $UDPServerAddress *
878 $UDPServerRun 514
879
880 - on syslog-ng, a new source can be created the following way, it then needs
881 to be added as a valid source in one of the "log" directives :
882
883 source s_udp {
884 udp(ip(127.0.0.1) port(514));
885 };
886
887Please consult your syslog daemon's manual for more information. If no logs are
888seen in the system's log files, please consider the following tests :
889
890 - restart haproxy. Each frontend and backend logs one line indicating it's
891 starting. If these logs are received, it means logs are working.
892
893 - run "strace -tt -s100 -etrace=sendmsg -p <haproxy's pid>" and perform some
894 activity that you expect to be logged. You should see the log messages
895 being sent using sendmsg() there. If they don't appear, restart using
896 strace on top of haproxy. If you still see no logs, it definitely means
897 that something is wrong in your configuration.
898
899 - run tcpdump to watch for port 514, for example on the loopback interface if
900 the traffic is being sent locally : "tcpdump -As0 -ni lo port 514". If the
901 packets are seen there, it's the proof they're sent then the syslogd daemon
902 needs to be troubleshooted.
903
904While traffic logs are sent from the frontends (where the incoming connections
905are accepted), backends also need to be able to send logs in order to report a
906server state change consecutive to a health check. Please consult HAProxy's
907configuration manual for more information regarding all possible log settings.
908
Dan Lloyd8e48b872016-07-01 21:01:18 -0400909It is convenient to chose a facility that is not used by other daemons. HAProxy
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200910examples often suggest "local0" for traffic logs and "local1" for admin logs
911because they're never seen in field. A single facility would be enough as well.
912Having separate logs is convenient for log analysis, but it's also important to
913remember that logs may sometimes convey confidential information, and as such
Dan Lloyd8e48b872016-07-01 21:01:18 -0400914they must not be mixed with other logs that may accidentally be handed out to
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200915unauthorized people.
916
917For in-field troubleshooting without impacting the server's capacity too much,
918it is recommended to make use of the "halog" utility provided with HAProxy.
919This is sort of a grep-like utility designed to process HAProxy log files at
920a very fast data rate. Typical figures range between 1 and 2 GB of logs per
921second. It is capable of extracting only certain logs (eg: search for some
922classes of HTTP status codes, connection termination status, search by response
923time ranges, look for errors only), count lines, limit the output to a number
924of lines, and perform some more advanced statistics such as sorting servers
925by response time or error counts, sorting URLs by time or count, sorting client
926addresses by access count, and so on. It is pretty convenient to quickly spot
927anomalies such as a bot looping on the site, and block them.
928
929
9309. Statistics and monitoring
931----------------------------
932
Willy Tarreau44aed902015-10-13 14:45:29 +0200933It is possible to query HAProxy about its status. The most commonly used
934mechanism is the HTTP statistics page. This page also exposes an alternative
935CSV output format for monitoring tools. The same format is provided on the
936Unix socket.
937
938
9399.1. CSV format
940---------------
941
942The statistics may be consulted either from the unix socket or from the HTTP
943page. Both means provide a CSV format whose fields follow. The first line
944begins with a sharp ('#') and has one word per comma-delimited field which
945represents the title of the column. All other lines starting at the second one
946use a classical CSV format using a comma as the delimiter, and the double quote
947('"') as an optional text delimiter, but only if the enclosed text is ambiguous
948(if it contains a quote or a comma). The double-quote character ('"') in the
949text is doubled ('""'), which is the format that most tools recognize. Please
950do not insert any column before these ones in order not to break tools which
951use hard-coded column positions.
952
953In brackets after each field name are the types which may have a value for
954that field. The types are L (Listeners), F (Frontends), B (Backends), and
955S (Servers).
956
957 0. pxname [LFBS]: proxy name
958 1. svname [LFBS]: service name (FRONTEND for frontend, BACKEND for backend,
959 any name for server/listener)
960 2. qcur [..BS]: current queued requests. For the backend this reports the
961 number queued without a server assigned.
962 3. qmax [..BS]: max value of qcur
963 4. scur [LFBS]: current sessions
964 5. smax [LFBS]: max sessions
965 6. slim [LFBS]: configured session limit
Willy Tarreauc73810f2016-01-11 13:52:04 +0100966 7. stot [LFBS]: cumulative number of sessions
Willy Tarreau44aed902015-10-13 14:45:29 +0200967 8. bin [LFBS]: bytes in
968 9. bout [LFBS]: bytes out
969 10. dreq [LFB.]: requests denied because of security concerns.
970 - For tcp this is because of a matched tcp-request content rule.
971 - For http this is because of a matched http-request or tarpit rule.
972 11. dresp [LFBS]: responses denied because of security concerns.
973 - For http this is because of a matched http-request rule, or
974 "option checkcache".
975 12. ereq [LF..]: request errors. Some of the possible causes are:
976 - early termination from the client, before the request has been sent.
977 - read error from the client
978 - client timeout
979 - client closed connection
980 - various bad requests from the client.
981 - request was tarpitted.
982 13. econ [..BS]: number of requests that encountered an error trying to
983 connect to a backend server. The backend stat is the sum of the stat
984 for all servers of that backend, plus any connection errors not
985 associated with a particular server (such as the backend having no
986 active servers).
987 14. eresp [..BS]: response errors. srv_abrt will be counted here also.
988 Some other errors are:
989 - write error on the client socket (won't be counted for the server stat)
990 - failure applying filters to the response.
991 15. wretr [..BS]: number of times a connection to a server was retried.
992 16. wredis [..BS]: number of times a request was redispatched to another
993 server. The server value counts the number of times that server was
994 switched away from.
Willy Tarreaub96dd282016-11-09 14:45:51 +0100995 17. status [LFBS]: status (UP/DOWN/NOLB/MAINT/MAINT(via)/MAINT(resolution)...)
Willy Tarreau44aed902015-10-13 14:45:29 +0200996 18. weight [..BS]: total weight (backend), server weight (server)
997 19. act [..BS]: number of active servers (backend), server is active (server)
998 20. bck [..BS]: number of backup servers (backend), server is backup (server)
999 21. chkfail [...S]: number of failed checks. (Only counts checks failed when
1000 the server is up.)
1001 22. chkdown [..BS]: number of UP->DOWN transitions. The backend counter counts
1002 transitions to the whole backend being down, rather than the sum of the
1003 counters for each server.
1004 23. lastchg [..BS]: number of seconds since the last UP<->DOWN transition
1005 24. downtime [..BS]: total downtime (in seconds). The value for the backend
1006 is the downtime for the whole backend, not the sum of the server downtime.
1007 25. qlimit [...S]: configured maxqueue for the server, or nothing in the
1008 value is 0 (default, meaning no limit)
1009 26. pid [LFBS]: process id (0 for first instance, 1 for second, ...)
1010 27. iid [LFBS]: unique proxy id
1011 28. sid [L..S]: server id (unique inside a proxy)
1012 29. throttle [...S]: current throttle percentage for the server, when
1013 slowstart is active, or no value if not in slowstart.
1014 30. lbtot [..BS]: total number of times a server was selected, either for new
1015 sessions, or when re-dispatching. The server counter is the number
1016 of times that server was selected.
1017 31. tracked [...S]: id of proxy/server if tracking is enabled.
1018 32. type [LFBS]: (0=frontend, 1=backend, 2=server, 3=socket/listener)
1019 33. rate [.FBS]: number of sessions per second over last elapsed second
1020 34. rate_lim [.F..]: configured limit on new sessions per second
1021 35. rate_max [.FBS]: max number of new sessions per second
1022 36. check_status [...S]: status of last health check, one of:
1023 UNK -> unknown
1024 INI -> initializing
1025 SOCKERR -> socket error
1026 L4OK -> check passed on layer 4, no upper layers testing enabled
1027 L4TOUT -> layer 1-4 timeout
1028 L4CON -> layer 1-4 connection problem, for example
1029 "Connection refused" (tcp rst) or "No route to host" (icmp)
1030 L6OK -> check passed on layer 6
1031 L6TOUT -> layer 6 (SSL) timeout
1032 L6RSP -> layer 6 invalid response - protocol error
1033 L7OK -> check passed on layer 7
1034 L7OKC -> check conditionally passed on layer 7, for example 404 with
1035 disable-on-404
1036 L7TOUT -> layer 7 (HTTP/SMTP) timeout
1037 L7RSP -> layer 7 invalid response - protocol error
1038 L7STS -> layer 7 response error, for example HTTP 5xx
Daniel Schnellerb6c8b0d2017-09-01 19:13:55 +02001039 Notice: If a check is currently running, the last known status will be
1040 reported, prefixed with "* ". e. g. "* L7OK".
Willy Tarreau44aed902015-10-13 14:45:29 +02001041 37. check_code [...S]: layer5-7 code, if available
1042 38. check_duration [...S]: time in ms took to finish last health check
1043 39. hrsp_1xx [.FBS]: http responses with 1xx code
1044 40. hrsp_2xx [.FBS]: http responses with 2xx code
1045 41. hrsp_3xx [.FBS]: http responses with 3xx code
1046 42. hrsp_4xx [.FBS]: http responses with 4xx code
1047 43. hrsp_5xx [.FBS]: http responses with 5xx code
1048 44. hrsp_other [.FBS]: http responses with other codes (protocol error)
1049 45. hanafail [...S]: failed health checks details
1050 46. req_rate [.F..]: HTTP requests per second over last elapsed second
1051 47. req_rate_max [.F..]: max number of HTTP requests per second observed
Willy Tarreaufb981bd2016-12-12 14:31:46 +01001052 48. req_tot [.FB.]: total number of HTTP requests received
Willy Tarreau44aed902015-10-13 14:45:29 +02001053 49. cli_abrt [..BS]: number of data transfers aborted by the client
1054 50. srv_abrt [..BS]: number of data transfers aborted by the server
1055 (inc. in eresp)
1056 51. comp_in [.FB.]: number of HTTP response bytes fed to the compressor
1057 52. comp_out [.FB.]: number of HTTP response bytes emitted by the compressor
1058 53. comp_byp [.FB.]: number of bytes that bypassed the HTTP compressor
1059 (CPU/BW limit)
1060 54. comp_rsp [.FB.]: number of HTTP responses that were compressed
1061 55. lastsess [..BS]: number of seconds since last session assigned to
1062 server/backend
1063 56. last_chk [...S]: last health check contents or textual error
1064 57. last_agt [...S]: last agent check contents or textual error
1065 58. qtime [..BS]: the average queue time in ms over the 1024 last requests
1066 59. ctime [..BS]: the average connect time in ms over the 1024 last requests
1067 60. rtime [..BS]: the average response time in ms over the 1024 last requests
1068 (0 for TCP)
1069 61. ttime [..BS]: the average total session time in ms over the 1024 last
1070 requests
Willy Tarreau7f618842016-01-08 11:40:03 +01001071 62. agent_status [...S]: status of last agent check, one of:
1072 UNK -> unknown
1073 INI -> initializing
1074 SOCKERR -> socket error
1075 L4OK -> check passed on layer 4, no upper layers testing enabled
1076 L4TOUT -> layer 1-4 timeout
1077 L4CON -> layer 1-4 connection problem, for example
1078 "Connection refused" (tcp rst) or "No route to host" (icmp)
1079 L7OK -> agent reported "up"
1080 L7STS -> agent reported "fail", "stop", or "down"
1081 63. agent_code [...S]: numeric code reported by agent if any (unused for now)
1082 64. agent_duration [...S]: time in ms taken to finish last check
Willy Tarreaudd7354b2016-01-08 13:47:26 +01001083 65. check_desc [...S]: short human-readable description of check_status
1084 66. agent_desc [...S]: short human-readable description of agent_status
Willy Tarreau3141f592016-01-08 14:25:28 +01001085 67. check_rise [...S]: server's "rise" parameter used by checks
1086 68. check_fall [...S]: server's "fall" parameter used by checks
1087 69. check_health [...S]: server's health check value between 0 and rise+fall-1
1088 70. agent_rise [...S]: agent's "rise" parameter, normally 1
1089 71. agent_fall [...S]: agent's "fall" parameter, normally 1
1090 72. agent_health [...S]: agent's health parameter, between 0 and rise+fall-1
Willy Tarreaua6f5a732016-01-08 16:59:56 +01001091 73. addr [L..S]: address:port or "unix". IPv6 has brackets around the address.
Willy Tarreaue4847c62016-01-08 15:43:54 +01001092 74: cookie [..BS]: server's cookie value or backend's cookie name
Willy Tarreauf8211df2016-01-11 14:09:38 +01001093 75: mode [LFBS]: proxy mode (tcp, http, health, unknown)
Willy Tarreauf1516d92016-01-11 14:48:36 +01001094 76: algo [..B.]: load balancing algorithm
Willy Tarreauc73810f2016-01-11 13:52:04 +01001095 77: conn_rate [.F..]: number of connections over the last elapsed second
1096 78: conn_rate_max [.F..]: highest known conn_rate
1097 79: conn_tot [.F..]: cumulative number of connections
Willy Tarreau5b9bdff2016-01-11 14:40:47 +01001098 80: intercepted [.FB.]: cum. number of intercepted requests (monitor, stats)
Willy Tarreau8a90b8e2016-10-21 18:15:32 +02001099 81: dcon [LF..]: requests denied by "tcp-request connection" rules
Willy Tarreaua5bc36b2016-10-21 18:16:27 +02001100 82: dses [LF..]: requests denied by "tcp-request session" rules
Willy Tarreauea96a822018-05-28 15:15:43 +02001101 83: wrew [LFBS]: cumulative number of failed header rewriting warnings
Jérôme Magninb67a8f32019-07-17 09:24:46 +02001102 84: connect [..BS]: cumulative number of connection establishment attempts
1103 85: reuse [..BS]: cumulative number of connection reuses
Willy Tarreaub896b1a2019-11-08 07:29:34 +01001104 86: cache_lookups [.FB.]: cumulative number of cache lookups
Jérôme Magninb564f1c2019-07-17 14:04:40 +02001105 87: cache_hits [.FB.]: cumulative number of cache hits
Christopher Faulet1bcbb902019-11-08 15:27:27 +01001106 88: srv_icur [...S]: current number of idle connections available for reuse
1107 89: src_ilim [...S]: limit on the number of available idle connections
1108 90. qtime_max [..BS]: the maximum observed queue time in ms
1109 91. ctime_max [..BS]: the maximum observed connect time in ms
1110 92. rtime_max [..BS]: the maximum observed response time in ms (0 for TCP)
1111 93. ttime_max [..BS]: the maximum observed total session time in ms
Willy Tarreau44aed902015-10-13 14:45:29 +02001112
1113
Willy Tarreau5d8b9792016-03-11 11:09:34 +010011149.2) Typed output format
1115------------------------
1116
1117Both "show info" and "show stat" support a mode where each output value comes
1118with its type and sufficient information to know how the value is supposed to
1119be aggregated between processes and how it evolves.
1120
1121In all cases, the output consists in having a single value per line with all
1122the information split into fields delimited by colons (':').
1123
1124The first column designates the object or metric being dumped. Its format is
1125specific to the command producing this output and will not be described in this
1126section. Usually it will consist in a series of identifiers and field names.
1127
1128The second column contains 3 characters respectively indicating the origin, the
1129nature and the scope of the value being reported. The first character (the
1130origin) indicates where the value was extracted from. Possible characters are :
1131
1132 M The value is a metric. It is valid at one instant any may change depending
1133 on its nature .
1134
1135 S The value is a status. It represents a discrete value which by definition
1136 cannot be aggregated. It may be the status of a server ("UP" or "DOWN"),
1137 the PID of the process, etc.
1138
1139 K The value is a sorting key. It represents an identifier which may be used
1140 to group some values together because it is unique among its class. All
1141 internal identifiers are keys. Some names can be listed as keys if they
1142 are unique (eg: a frontend name is unique). In general keys come from the
Dan Lloyd8e48b872016-07-01 21:01:18 -04001143 configuration, even though some of them may automatically be assigned. For
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001144 most purposes keys may be considered as equivalent to configuration.
1145
1146 C The value comes from the configuration. Certain configuration values make
1147 sense on the output, for example a concurrent connection limit or a cookie
1148 name. By definition these values are the same in all processes started
1149 from the same configuration file.
1150
1151 P The value comes from the product itself. There are very few such values,
1152 most common use is to report the product name, version and release date.
1153 These elements are also the same between all processes.
1154
1155The second character (the nature) indicates the nature of the information
1156carried by the field in order to let an aggregator decide on what operation to
1157use to aggregate multiple values. Possible characters are :
1158
1159 A The value represents an age since a last event. This is a bit different
1160 from the duration in that an age is automatically computed based on the
1161 current date. A typical example is how long ago did the last session
1162 happen on a server. Ages are generally aggregated by taking the minimum
1163 value and do not need to be stored.
1164
1165 a The value represents an already averaged value. The average response times
1166 and server weights are of this nature. Averages can typically be averaged
1167 between processes.
1168
1169 C The value represents a cumulative counter. Such measures perpetually
1170 increase until they wrap around. Some monitoring protocols need to tell
1171 the difference between a counter and a gauge to report a different type.
1172 In general counters may simply be summed since they represent events or
1173 volumes. Examples of metrics of this nature are connection counts or byte
1174 counts.
1175
1176 D The value represents a duration for a status. There are a few usages of
1177 this, most of them include the time taken by the last health check and
1178 the time a server has spent down. Durations are generally not summed,
1179 most of the time the maximum will be retained to compute an SLA.
1180
1181 G The value represents a gauge. It's a measure at one instant. The memory
1182 usage or the current number of active connections are of this nature.
1183 Metrics of this type are typically summed during aggregation.
1184
1185 L The value represents a limit (generally a configured one). By nature,
1186 limits are harder to aggregate since they are specific to the point where
1187 they were retrieved. In certain situations they may be summed or be kept
1188 separate.
1189
1190 M The value represents a maximum. In general it will apply to a gauge and
1191 keep the highest known value. An example of such a metric could be the
1192 maximum amount of concurrent connections that was encountered in the
1193 product's life time. To correctly aggregate maxima, you are supposed to
1194 output a range going from the maximum of all maxima and the sum of all
1195 of them. There is indeed no way to know if they were encountered
1196 simultaneously or not.
1197
1198 m The value represents a minimum. In general it will apply to a gauge and
1199 keep the lowest known value. An example of such a metric could be the
1200 minimum amount of free memory pools that was encountered in the product's
1201 life time. To correctly aggregate minima, you are supposed to output a
1202 range going from the minimum of all minima and the sum of all of them.
1203 There is indeed no way to know if they were encountered simultaneously
1204 or not.
1205
1206 N The value represents a name, so it is a string. It is used to report
1207 proxy names, server names and cookie names. Names have configuration or
1208 keys as their origin and are supposed to be the same among all processes.
1209
1210 O The value represents a free text output. Outputs from various commands,
1211 returns from health checks, node descriptions are of such nature.
1212
1213 R The value represents an event rate. It's a measure at one instant. It is
1214 quite similar to a gauge except that the recipient knows that this measure
1215 moves slowly and may decide not to keep all values. An example of such a
1216 metric is the measured amount of connections per second. Metrics of this
1217 type are typically summed during aggregation.
1218
1219 T The value represents a date or time. A field emitting the current date
1220 would be of this type. The method to aggregate such information is left
1221 as an implementation choice. For now no field uses this type.
1222
1223The third character (the scope) indicates what extent the value reflects. Some
1224elements may be per process while others may be per configuration or per system.
1225The distinction is important to know whether or not a single value should be
1226kept during aggregation or if values have to be aggregated. The following
1227characters are currently supported :
1228
1229 C The value is valid for a whole cluster of nodes, which is the set of nodes
1230 communicating over the peers protocol. An example could be the amount of
1231 entries present in a stick table that is replicated with other peers. At
1232 the moment no metric use this scope.
1233
1234 P The value is valid only for the process reporting it. Most metrics use
1235 this scope.
1236
1237 S The value is valid for the whole service, which is the set of processes
1238 started together from the same configuration file. All metrics originating
1239 from the configuration use this scope. Some other metrics may use it as
1240 well for some shared resources (eg: shared SSL cache statistics).
1241
1242 s The value is valid for the whole system, such as the system's hostname,
1243 current date or resource usage. At the moment this scope is not used by
1244 any metric.
1245
1246Consumers of these information will generally have enough of these 3 characters
1247to determine how to accurately report aggregated information across multiple
1248processes.
1249
1250After this column, the third column indicates the type of the field, among "s32"
1251(signed 32-bit integer), "s64" (signed 64-bit integer), "u32" (unsigned 32-bit
1252integer), "u64" (unsigned 64-bit integer), "str" (string). It is important to
1253know the type before parsing the value in order to properly read it. For example
1254a string containing only digits is still a string an not an integer (eg: an
1255error code extracted by a check).
1256
1257Then the fourth column is the value itself, encoded according to its type.
1258Strings are dumped as-is immediately after the colon without any leading space.
1259If a string contains a colon, it will appear normally. This means that the
1260output should not be exclusively split around colons or some check outputs
1261or server addresses might be truncated.
1262
1263
12649.3. Unix Socket commands
Willy Tarreau44aed902015-10-13 14:45:29 +02001265-------------------------
1266
1267The stats socket is not enabled by default. In order to enable it, it is
1268necessary to add one line in the global section of the haproxy configuration.
1269A second line is recommended to set a larger timeout, always appreciated when
1270issuing commands by hand :
1271
1272 global
1273 stats socket /var/run/haproxy.sock mode 600 level admin
1274 stats timeout 2m
1275
1276It is also possible to add multiple instances of the stats socket by repeating
1277the line, and make them listen to a TCP port instead of a UNIX socket. This is
1278never done by default because this is dangerous, but can be handy in some
1279situations :
1280
1281 global
1282 stats socket /var/run/haproxy.sock mode 600 level admin
1283 stats socket ipv4@192.168.0.1:9999 level admin
1284 stats timeout 2m
1285
1286To access the socket, an external utility such as "socat" is required. Socat is
1287a swiss-army knife to connect anything to anything. We use it to connect
1288terminals to the socket, or a couple of stdin/stdout pipes to it for scripts.
1289The two main syntaxes we'll use are the following :
1290
1291 # socat /var/run/haproxy.sock stdio
1292 # socat /var/run/haproxy.sock readline
1293
1294The first one is used with scripts. It is possible to send the output of a
1295script to haproxy, and pass haproxy's output to another script. That's useful
1296for retrieving counters or attack traces for example.
1297
1298The second one is only useful for issuing commands by hand. It has the benefit
1299that the terminal is handled by the readline library which supports line
1300editing and history, which is very convenient when issuing repeated commands
1301(eg: watch a counter).
1302
1303The socket supports two operation modes :
1304 - interactive
1305 - non-interactive
1306
1307The non-interactive mode is the default when socat connects to the socket. In
1308this mode, a single line may be sent. It is processed as a whole, responses are
1309sent back, and the connection closes after the end of the response. This is the
1310mode that scripts and monitoring tools use. It is possible to send multiple
1311commands in this mode, they need to be delimited by a semi-colon (';'). For
1312example :
1313
1314 # echo "show info;show stat;show table" | socat /var/run/haproxy stdio
1315
Dragan Dosena1c35ab2016-11-24 11:33:12 +01001316If a command needs to use a semi-colon or a backslash (eg: in a value), it
Joseph Herlant71b4b152018-11-13 16:55:16 -08001317must be preceded by a backslash ('\').
Chad Lavoiee3f50312016-05-26 16:42:25 -04001318
Willy Tarreau44aed902015-10-13 14:45:29 +02001319The interactive mode displays a prompt ('>') and waits for commands to be
1320entered on the line, then processes them, and displays the prompt again to wait
1321for a new command. This mode is entered via the "prompt" command which must be
1322sent on the first line in non-interactive mode. The mode is a flip switch, if
1323"prompt" is sent in interactive mode, it is disabled and the connection closes
1324after processing the last command of the same line.
1325
1326For this reason, when debugging by hand, it's quite common to start with the
1327"prompt" command :
1328
1329 # socat /var/run/haproxy readline
1330 prompt
1331 > show info
1332 ...
1333 >
1334
1335Since multiple commands may be issued at once, haproxy uses the empty line as a
1336delimiter to mark an end of output for each command, and takes care of ensuring
1337that no command can emit an empty line on output. A script can thus easily
1338parse the output even when multiple commands were pipelined on a single line.
1339
Aurélien Nephtaliabbf6072018-04-18 13:26:46 +02001340Some commands may take an optional payload. To add one to a command, the first
1341line needs to end with the "<<\n" pattern. The next lines will be treated as
1342the payload and can contain as many lines as needed. To validate a command with
1343a payload, it needs to end with an empty line.
1344
1345Limitations do exist: the length of the whole buffer passed to the CLI must
1346not be greater than tune.bfsize and the pattern "<<" must not be glued to the
1347last word of the line.
1348
1349When entering a paylod while in interactive mode, the prompt will change from
1350"> " to "+ ".
1351
Willy Tarreau44aed902015-10-13 14:45:29 +02001352It is important to understand that when multiple haproxy processes are started
1353on the same sockets, any process may pick up the request and will output its
1354own stats.
1355
1356The list of commands currently supported on the stats socket is provided below.
1357If an unknown command is sent, haproxy displays the usage message which reminds
1358all supported commands. Some commands support a more complex syntax, generally
1359it will explain what part of the command is invalid when this happens.
1360
Olivier Doucetd8703e82017-08-31 11:05:10 +02001361Some commands require a higher level of privilege to work. If you do not have
1362enough privilege, you will get an error "Permission denied". Please check
1363the "level" option of the "bind" keyword lines in the configuration manual
1364for more information.
1365
Willy Tarreau44aed902015-10-13 14:45:29 +02001366add acl <acl> <pattern>
1367 Add an entry into the acl <acl>. <acl> is the #<id> or the <file> returned by
1368 "show acl". This command does not verify if the entry already exists. This
1369 command cannot be used if the reference <acl> is a file also used with a map.
1370 In this case, you must use the command "add map" in place of "add acl".
1371
1372add map <map> <key> <value>
Aurélien Nephtali25650ce2018-04-18 14:04:47 +02001373add map <map> <payload>
Willy Tarreau44aed902015-10-13 14:45:29 +02001374 Add an entry into the map <map> to associate the value <value> to the key
1375 <key>. This command does not verify if the entry already exists. It is
1376 mainly used to fill a map after a clear operation. Note that if the reference
1377 <map> is a file and is shared with a map, this map will contain also a new
Aurélien Nephtali25650ce2018-04-18 14:04:47 +02001378 pattern entry. Using the payload syntax it is possible to add multiple
1379 key/value pairs by entering them on separate lines. On each new line, the
1380 first word is the key and the rest of the line is considered to be the value
1381 which can even contains spaces.
1382
1383 Example:
1384
1385 # socat /tmp/sock1 -
1386 prompt
1387
1388 > add map #-1 <<
1389 + key1 value1
1390 + key2 value2 with spaces
1391 + key3 value3 also with spaces
1392 + key4 value4
1393
1394 >
Willy Tarreau44aed902015-10-13 14:45:29 +02001395
1396clear counters
1397 Clear the max values of the statistics counters in each proxy (frontend &
Willy Tarreaud80cb4e2018-01-20 19:30:13 +01001398 backend) and in each server. The accumulated counters are not affected. The
1399 internal activity counters reported by "show activity" are also reset. This
Willy Tarreau44aed902015-10-13 14:45:29 +02001400 can be used to get clean counters after an incident, without having to
1401 restart nor to clear traffic counters. This command is restricted and can
1402 only be issued on sockets configured for levels "operator" or "admin".
1403
1404clear counters all
1405 Clear all statistics counters in each proxy (frontend & backend) and in each
1406 server. This has the same effect as restarting. This command is restricted
1407 and can only be issued on sockets configured for level "admin".
1408
1409clear acl <acl>
1410 Remove all entries from the acl <acl>. <acl> is the #<id> or the <file>
1411 returned by "show acl". Note that if the reference <acl> is a file and is
1412 shared with a map, this map will be also cleared.
1413
1414clear map <map>
1415 Remove all entries from the map <map>. <map> is the #<id> or the <file>
1416 returned by "show map". Note that if the reference <map> is a file and is
1417 shared with a acl, this acl will be also cleared.
1418
1419clear table <table> [ data.<type> <operator> <value> ] | [ key <key> ]
1420 Remove entries from the stick-table <table>.
1421
1422 This is typically used to unblock some users complaining they have been
1423 abusively denied access to a service, but this can also be used to clear some
1424 stickiness entries matching a server that is going to be replaced (see "show
1425 table" below for details). Note that sometimes, removal of an entry will be
1426 refused because it is currently tracked by a session. Retrying a few seconds
1427 later after the session ends is usual enough.
1428
1429 In the case where no options arguments are given all entries will be removed.
1430
1431 When the "data." form is used entries matching a filter applied using the
1432 stored data (see "stick-table" in section 4.2) are removed. A stored data
1433 type must be specified in <type>, and this data type must be stored in the
1434 table otherwise an error is reported. The data is compared according to
1435 <operator> with the 64-bit integer <value>. Operators are the same as with
1436 the ACLs :
1437
1438 - eq : match entries whose data is equal to this value
1439 - ne : match entries whose data is not equal to this value
1440 - le : match entries whose data is less than or equal to this value
1441 - ge : match entries whose data is greater than or equal to this value
1442 - lt : match entries whose data is less than this value
1443 - gt : match entries whose data is greater than this value
1444
1445 When the key form is used the entry <key> is removed. The key must be of the
1446 same type as the table, which currently is limited to IPv4, IPv6, integer and
1447 string.
1448
1449 Example :
1450 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1451 >>> # table: http_proxy, type: ip, size:204800, used:2
1452 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
1453 bytes_out_rate(60000)=187
1454 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
1455 bytes_out_rate(60000)=191
1456
1457 $ echo "clear table http_proxy key 127.0.0.1" | socat stdio /tmp/sock1
1458
1459 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1460 >>> # table: http_proxy, type: ip, size:204800, used:1
1461 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
1462 bytes_out_rate(60000)=191
1463 $ echo "clear table http_proxy data.gpc0 eq 1" | socat stdio /tmp/sock1
1464 $ echo "show table http_proxy" | socat stdio /tmp/sock1
1465 >>> # table: http_proxy, type: ip, size:204800, used:1
1466
Willy Tarreau6bdf3e92019-05-20 14:25:05 +02001467debug dev <command> [args]*
1468 Call a developer-specific command. Only supported when haproxy is built with
1469 DEBUG_DEV defined. Supported commands are then listed in the help message.
1470 All of these commands require admin privileges, and must never appear on a
1471 production system as most of them are unsafe and dangerous.
1472
Willy Tarreau44aed902015-10-13 14:45:29 +02001473del acl <acl> [<key>|#<ref>]
1474 Delete all the acl entries from the acl <acl> corresponding to the key <key>.
1475 <acl> is the #<id> or the <file> returned by "show acl". If the <ref> is used,
1476 this command delete only the listed reference. The reference can be found with
1477 listing the content of the acl. Note that if the reference <acl> is a file and
1478 is shared with a map, the entry will be also deleted in the map.
1479
1480del map <map> [<key>|#<ref>]
1481 Delete all the map entries from the map <map> corresponding to the key <key>.
1482 <map> is the #<id> or the <file> returned by "show map". If the <ref> is used,
1483 this command delete only the listed reference. The reference can be found with
1484 listing the content of the map. Note that if the reference <map> is a file and
1485 is shared with a acl, the entry will be also deleted in the map.
1486
1487disable agent <backend>/<server>
1488 Mark the auxiliary agent check as temporarily stopped.
1489
1490 In the case where an agent check is being run as a auxiliary check, due
1491 to the agent-check parameter of a server directive, new checks are only
Dan Lloyd8e48b872016-07-01 21:01:18 -04001492 initialized when the agent is in the enabled. Thus, disable agent will
Willy Tarreau44aed902015-10-13 14:45:29 +02001493 prevent any new agent checks from begin initiated until the agent
1494 re-enabled using enable agent.
1495
1496 When an agent is disabled the processing of an auxiliary agent check that
1497 was initiated while the agent was set as enabled is as follows: All
1498 results that would alter the weight, specifically "drain" or a weight
1499 returned by the agent, are ignored. The processing of agent check is
1500 otherwise unchanged.
1501
1502 The motivation for this feature is to allow the weight changing effects
1503 of the agent checks to be paused to allow the weight of a server to be
1504 configured using set weight without being overridden by the agent.
1505
1506 This command is restricted and can only be issued on sockets configured for
1507 level "admin".
1508
Olivier Houchard614f8d72017-03-14 20:08:46 +01001509disable dynamic-cookie backend <backend>
Ilya Shipitsin4f9532d2020-03-06 13:07:38 +05001510 Disable the generation of dynamic cookies for the backend <backend>
Olivier Houchard614f8d72017-03-14 20:08:46 +01001511
Willy Tarreau44aed902015-10-13 14:45:29 +02001512disable frontend <frontend>
1513 Mark the frontend as temporarily stopped. This corresponds to the mode which
1514 is used during a soft restart : the frontend releases the port but can be
1515 enabled again if needed. This should be used with care as some non-Linux OSes
1516 are unable to enable it back. This is intended to be used in environments
1517 where stopping a proxy is not even imaginable but a misconfigured proxy must
1518 be fixed. That way it's possible to release the port and bind it into another
1519 process to restore operations. The frontend will appear with status "STOP"
1520 on the stats page.
1521
1522 The frontend may be specified either by its name or by its numeric ID,
1523 prefixed with a sharp ('#').
1524
1525 This command is restricted and can only be issued on sockets configured for
1526 level "admin".
1527
1528disable health <backend>/<server>
1529 Mark the primary health check as temporarily stopped. This will disable
1530 sending of health checks, and the last health check result will be ignored.
1531 The server will be in unchecked state and considered UP unless an auxiliary
1532 agent check forces it down.
1533
1534 This command is restricted and can only be issued on sockets configured for
1535 level "admin".
1536
1537disable server <backend>/<server>
1538 Mark the server DOWN for maintenance. In this mode, no more checks will be
1539 performed on the server until it leaves maintenance.
1540 If the server is tracked by other servers, those servers will be set to DOWN
1541 during the maintenance.
1542
1543 In the statistics page, a server DOWN for maintenance will appear with a
1544 "MAINT" status, its tracking servers with the "MAINT(via)" one.
1545
1546 Both the backend and the server may be specified either by their name or by
1547 their numeric ID, prefixed with a sharp ('#').
1548
1549 This command is restricted and can only be issued on sockets configured for
1550 level "admin".
1551
1552enable agent <backend>/<server>
1553 Resume auxiliary agent check that was temporarily stopped.
1554
1555 See "disable agent" for details of the effect of temporarily starting
1556 and stopping an auxiliary agent.
1557
1558 This command is restricted and can only be issued on sockets configured for
1559 level "admin".
1560
Olivier Houchard614f8d72017-03-14 20:08:46 +01001561enable dynamic-cookie backend <backend>
n9@users.noreply.github.com7c80af02019-08-23 11:21:05 +02001562 Enable the generation of dynamic cookies for the backend <backend>.
1563 A secret key must also be provided.
Olivier Houchard614f8d72017-03-14 20:08:46 +01001564
Willy Tarreau44aed902015-10-13 14:45:29 +02001565enable frontend <frontend>
1566 Resume a frontend which was temporarily stopped. It is possible that some of
1567 the listening ports won't be able to bind anymore (eg: if another process
1568 took them since the 'disable frontend' operation). If this happens, an error
1569 is displayed. Some operating systems might not be able to resume a frontend
1570 which was disabled.
1571
1572 The frontend may be specified either by its name or by its numeric ID,
1573 prefixed with a sharp ('#').
1574
1575 This command is restricted and can only be issued on sockets configured for
1576 level "admin".
1577
1578enable health <backend>/<server>
1579 Resume a primary health check that was temporarily stopped. This will enable
1580 sending of health checks again. Please see "disable health" for details.
1581
1582 This command is restricted and can only be issued on sockets configured for
1583 level "admin".
1584
1585enable server <backend>/<server>
1586 If the server was previously marked as DOWN for maintenance, this marks the
1587 server UP and checks are re-enabled.
1588
1589 Both the backend and the server may be specified either by their name or by
1590 their numeric ID, prefixed with a sharp ('#').
1591
1592 This command is restricted and can only be issued on sockets configured for
1593 level "admin".
1594
1595get map <map> <value>
1596get acl <acl> <value>
1597 Lookup the value <value> in the map <map> or in the ACL <acl>. <map> or <acl>
1598 are the #<id> or the <file> returned by "show map" or "show acl". This command
1599 returns all the matching patterns associated with this map. This is useful for
1600 debugging maps and ACLs. The output format is composed by one line par
1601 matching type. Each line is composed by space-delimited series of words.
1602
1603 The first two words are:
1604
1605 <match method>: The match method applied. It can be "found", "bool",
1606 "int", "ip", "bin", "len", "str", "beg", "sub", "dir",
1607 "dom", "end" or "reg".
1608
1609 <match result>: The result. Can be "match" or "no-match".
1610
1611 The following words are returned only if the pattern matches an entry.
1612
1613 <index type>: "tree" or "list". The internal lookup algorithm.
1614
1615 <case>: "case-insensitive" or "case-sensitive". The
1616 interpretation of the case.
1617
1618 <entry matched>: match="<entry>". Return the matched pattern. It is
1619 useful with regular expressions.
1620
1621 The two last word are used to show the returned value and its type. With the
1622 "acl" case, the pattern doesn't exist.
1623
1624 return=nothing: No return because there are no "map".
1625 return="<value>": The value returned in the string format.
1626 return=cannot-display: The value cannot be converted as string.
1627
1628 type="<type>": The type of the returned sample.
1629
1630get weight <backend>/<server>
1631 Report the current weight and the initial weight of server <server> in
1632 backend <backend> or an error if either doesn't exist. The initial weight is
1633 the one that appears in the configuration file. Both are normally equal
1634 unless the current weight has been changed. Both the backend and the server
1635 may be specified either by their name or by their numeric ID, prefixed with a
1636 sharp ('#').
1637
1638help
1639 Print the list of known keywords and their basic usage. The same help screen
1640 is also displayed for unknown commands.
1641
1642prompt
1643 Toggle the prompt at the beginning of the line and enter or leave interactive
1644 mode. In interactive mode, the connection is not closed after a command
1645 completes. Instead, the prompt will appear again, indicating the user that
1646 the interpreter is waiting for a new command. The prompt consists in a right
1647 angle bracket followed by a space "> ". This mode is particularly convenient
1648 when one wants to periodically check information such as stats or errors.
1649 It is also a good idea to enter interactive mode before issuing a "help"
1650 command.
1651
1652quit
1653 Close the connection when in interactive mode.
1654
Olivier Houchard614f8d72017-03-14 20:08:46 +01001655set dynamic-cookie-key backend <backend> <value>
1656 Modify the secret key used to generate the dynamic persistent cookies.
1657 This will break the existing sessions.
1658
Willy Tarreau44aed902015-10-13 14:45:29 +02001659set map <map> [<key>|#<ref>] <value>
1660 Modify the value corresponding to each key <key> in a map <map>. <map> is the
1661 #<id> or <file> returned by "show map". If the <ref> is used in place of
1662 <key>, only the entry pointed by <ref> is changed. The new value is <value>.
1663
1664set maxconn frontend <frontend> <value>
1665 Dynamically change the specified frontend's maxconn setting. Any positive
1666 value is allowed including zero, but setting values larger than the global
1667 maxconn does not make much sense. If the limit is increased and connections
1668 were pending, they will immediately be accepted. If it is lowered to a value
1669 below the current number of connections, new connections acceptation will be
1670 delayed until the threshold is reached. The frontend might be specified by
1671 either its name or its numeric ID prefixed with a sharp ('#').
1672
Andrew Hayworthedb93a72015-10-27 21:46:25 +00001673set maxconn server <backend/server> <value>
1674 Dynamically change the specified server's maxconn setting. Any positive
1675 value is allowed including zero, but setting values larger than the global
1676 maxconn does not make much sense.
1677
Willy Tarreau44aed902015-10-13 14:45:29 +02001678set maxconn global <maxconn>
1679 Dynamically change the global maxconn setting within the range defined by the
1680 initial global maxconn setting. If it is increased and connections were
1681 pending, they will immediately be accepted. If it is lowered to a value below
1682 the current number of connections, new connections acceptation will be
1683 delayed until the threshold is reached. A value of zero restores the initial
1684 setting.
1685
Willy Tarreaud2d33482019-04-25 17:09:07 +02001686set profiling { tasks } { auto | on | off }
Willy Tarreau75c62c22018-11-22 11:02:09 +01001687 Enables or disables CPU profiling for the indicated subsystem. This is
1688 equivalent to setting or clearing the "profiling" settings in the "global"
1689 section of the configuration file. Please also see "show profiling".
1690
Willy Tarreau44aed902015-10-13 14:45:29 +02001691set rate-limit connections global <value>
1692 Change the process-wide connection rate limit, which is set by the global
1693 'maxconnrate' setting. A value of zero disables the limitation. This limit
1694 applies to all frontends and the change has an immediate effect. The value
1695 is passed in number of connections per second.
1696
1697set rate-limit http-compression global <value>
1698 Change the maximum input compression rate, which is set by the global
1699 'maxcomprate' setting. A value of zero disables the limitation. The value is
1700 passed in number of kilobytes per second. The value is available in the "show
1701 info" on the line "CompressBpsRateLim" in bytes.
1702
1703set rate-limit sessions global <value>
1704 Change the process-wide session rate limit, which is set by the global
1705 'maxsessrate' setting. A value of zero disables the limitation. This limit
1706 applies to all frontends and the change has an immediate effect. The value
1707 is passed in number of sessions per second.
1708
1709set rate-limit ssl-sessions global <value>
1710 Change the process-wide SSL session rate limit, which is set by the global
1711 'maxsslrate' setting. A value of zero disables the limitation. This limit
1712 applies to all frontends and the change has an immediate effect. The value
1713 is passed in number of sessions per second sent to the SSL stack. It applies
1714 before the handshake in order to protect the stack against handshake abuses.
1715
Baptiste Assmann3749ebf2016-08-03 22:34:12 +02001716set server <backend>/<server> addr <ip4 or ip6 address> [port <port>]
Willy Tarreau44aed902015-10-13 14:45:29 +02001717 Replace the current IP address of a server by the one provided.
Michael Prokop4438c602019-05-24 10:25:45 +02001718 Optionally, the port can be changed using the 'port' parameter.
Baptiste Assmann3749ebf2016-08-03 22:34:12 +02001719 Note that changing the port also support switching from/to port mapping
1720 (notation with +X or -Y), only if a port is configured for the health check.
Willy Tarreau44aed902015-10-13 14:45:29 +02001721
1722set server <backend>/<server> agent [ up | down ]
1723 Force a server's agent to a new state. This can be useful to immediately
1724 switch a server's state regardless of some slow agent checks for example.
1725 Note that the change is propagated to tracking servers if any.
1726
Misiek43972902017-01-09 09:53:06 +01001727set server <backend>/<server> agent-addr <addr>
1728 Change addr for servers agent checks. Allows to migrate agent-checks to
1729 another address at runtime. You can specify both IP and hostname, it will be
1730 resolved.
1731
1732set server <backend>/<server> agent-send <value>
1733 Change agent string sent to agent check target. Allows to update string while
1734 changing server address to keep those two matching.
1735
Willy Tarreau44aed902015-10-13 14:45:29 +02001736set server <backend>/<server> health [ up | stopping | down ]
1737 Force a server's health to a new state. This can be useful to immediately
1738 switch a server's state regardless of some slow health checks for example.
1739 Note that the change is propagated to tracking servers if any.
1740
Baptiste Assmann50946562016-08-31 23:26:29 +02001741set server <backend>/<server> check-port <port>
1742 Change the port used for health checking to <port>
1743
Willy Tarreau44aed902015-10-13 14:45:29 +02001744set server <backend>/<server> state [ ready | drain | maint ]
1745 Force a server's administrative state to a new state. This can be useful to
1746 disable load balancing and/or any traffic to a server. Setting the state to
1747 "ready" puts the server in normal mode, and the command is the equivalent of
1748 the "enable server" command. Setting the state to "maint" disables any traffic
1749 to the server as well as any health checks. This is the equivalent of the
1750 "disable server" command. Setting the mode to "drain" only removes the server
1751 from load balancing but still allows it to be checked and to accept new
1752 persistent connections. Changes are propagated to tracking servers if any.
1753
1754set server <backend>/<server> weight <weight>[%]
1755 Change a server's weight to the value passed in argument. This is the exact
1756 equivalent of the "set weight" command below.
1757
Frédéric Lécailleb418c122017-04-26 11:24:02 +02001758set server <backend>/<server> fqdn <FQDN>
Lukas Tribusc5dd5a52018-08-14 11:39:35 +02001759 Change a server's FQDN to the value passed in argument. This requires the
1760 internal run-time DNS resolver to be configured and enabled for this server.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02001761
Andjelko Iharosc4df59e2017-07-20 11:59:48 +02001762set severity-output [ none | number | string ]
1763 Change the severity output format of the stats socket connected to for the
1764 duration of the current session.
1765
Aurélien Nephtali1e0867c2018-04-18 14:04:58 +02001766set ssl ocsp-response <response | payload>
Willy Tarreau44aed902015-10-13 14:45:29 +02001767 This command is used to update an OCSP Response for a certificate (see "crt"
1768 on "bind" lines). Same controls are performed as during the initial loading of
1769 the response. The <response> must be passed as a base64 encoded string of the
Emmanuel Hocdet2c32d8f2017-05-22 14:58:00 +02001770 DER encoded response from the OCSP server. This command is not supported with
1771 BoringSSL.
Willy Tarreau44aed902015-10-13 14:45:29 +02001772
1773 Example:
1774 openssl ocsp -issuer issuer.pem -cert server.pem \
1775 -host ocsp.issuer.com:80 -respout resp.der
1776 echo "set ssl ocsp-response $(base64 -w 10000 resp.der)" | \
1777 socat stdio /var/run/haproxy.stat
1778
Aurélien Nephtali1e0867c2018-04-18 14:04:58 +02001779 using the payload syntax:
1780 echo -e "set ssl ocsp-response <<\n$(base64 resp.der)\n" | \
1781 socat stdio /var/run/haproxy.stat
1782
Willy Tarreau44aed902015-10-13 14:45:29 +02001783set ssl tls-key <id> <tlskey>
1784 Set the next TLS key for the <id> listener to <tlskey>. This key becomes the
1785 ultimate key, while the penultimate one is used for encryption (others just
1786 decrypt). The oldest TLS key present is overwritten. <id> is either a numeric
1787 #<id> or <file> returned by "show tls-keys". <tlskey> is a base64 encoded 48
Emeric Brun9e754772019-01-10 17:51:55 +01001788 or 80 bits TLS ticket key (ex. openssl rand 80 | openssl base64 -A).
Willy Tarreau44aed902015-10-13 14:45:29 +02001789
1790set table <table> key <key> [data.<data_type> <value>]*
1791 Create or update a stick-table entry in the table. If the key is not present,
1792 an entry is inserted. See stick-table in section 4.2 to find all possible
1793 values for <data_type>. The most likely use consists in dynamically entering
1794 entries for source IP addresses, with a flag in gpc0 to dynamically block an
1795 IP address or affect its quality of service. It is possible to pass multiple
1796 data_types in a single call.
1797
1798set timeout cli <delay>
1799 Change the CLI interface timeout for current connection. This can be useful
1800 during long debugging sessions where the user needs to constantly inspect
1801 some indicators without being disconnected. The delay is passed in seconds.
1802
1803set weight <backend>/<server> <weight>[%]
1804 Change a server's weight to the value passed in argument. If the value ends
1805 with the '%' sign, then the new weight will be relative to the initially
1806 configured weight. Absolute weights are permitted between 0 and 256.
1807 Relative weights must be positive with the resulting absolute weight is
1808 capped at 256. Servers which are part of a farm running a static
1809 load-balancing algorithm have stricter limitations because the weight
1810 cannot change once set. Thus for these servers, the only accepted values
1811 are 0 and 100% (or 0 and the initial weight). Changes take effect
1812 immediately, though certain LB algorithms require a certain amount of
1813 requests to consider changes. A typical usage of this command is to
1814 disable a server during an update by setting its weight to zero, then to
1815 enable it again after the update by setting it back to 100%. This command
1816 is restricted and can only be issued on sockets configured for level
1817 "admin". Both the backend and the server may be specified either by their
1818 name or by their numeric ID, prefixed with a sharp ('#').
1819
Willy Tarreaud6129fc2017-07-28 16:52:23 +02001820show acl [<acl>]
1821 Dump info about acl converters. Without argument, the list of all available
1822 acls is returned. If a <acl> is specified, its contents are dumped. <acl> if
1823 the #<id> or <file>. The dump format is the same than the map even for the
1824 sample value. The data returned are not a list of available ACL, but are the
1825 list of all patterns composing any ACL. Many of these patterns can be shared
1826 with maps.
1827
1828show backend
1829 Dump the list of backends available in the running process
1830
William Lallemand67a234f2018-12-13 09:05:45 +01001831show cli level
1832 Display the CLI level of the current CLI session. The result could be
1833 'admin', 'operator' or 'user'. See also the 'operator' and 'user' commands.
1834
1835 Example :
1836
1837 $ socat /tmp/sock1 readline
1838 prompt
1839 > operator
1840 > show cli level
1841 operator
1842 > user
1843 > show cli level
1844 user
1845 > operator
1846 Permission denied
1847
1848operator
1849 Decrease the CLI level of the current CLI session to operator. It can't be
1850 increase. See also "show cli level"
1851
1852user
1853 Decrease the CLI level of the current CLI session to user. It can't be
1854 increase. See also "show cli level"
1855
Willy Tarreau4c356932019-05-16 17:39:32 +02001856show activity
1857 Reports some counters about internal events that will help developers and
1858 more generally people who know haproxy well enough to narrow down the causes
1859 of reports of abnormal behaviours. A typical example would be a properly
1860 running process never sleeping and eating 100% of the CPU. The output fields
1861 will be made of one line per metric, and per-thread counters on the same
1862 line. These counters are 32-bit and will wrap during the process' life, which
1863 is not a problem since calls to this command will typically be performed
1864 twice. The fields are purposely not documented so that their exact meaning is
1865 verified in the code where the counters are fed. These values are also reset
1866 by the "clear counters" command.
1867
William Lallemand51132162016-12-16 16:38:58 +01001868show cli sockets
1869 List CLI sockets. The output format is composed of 3 fields separated by
1870 spaces. The first field is the socket address, it can be a unix socket, a
1871 ipv4 address:port couple or a ipv6 one. Socket of other types won't be dump.
1872 The second field describe the level of the socket: 'admin', 'user' or
1873 'operator'. The last field list the processes on which the socket is bound,
1874 separated by commas, it can be numbers or 'all'.
1875
1876 Example :
1877
1878 $ echo 'show cli sockets' | socat stdio /tmp/sock1
1879 # socket lvl processes
1880 /tmp/sock1 admin all
1881 127.0.0.1:9999 user 2,3,4
1882 127.0.0.2:9969 user 2
1883 [::1]:9999 operator 2
1884
William Lallemand86d0df02017-11-24 21:36:45 +01001885show cache
Cyril Bonté7b888f12017-11-26 22:24:31 +01001886 List the configured caches and the objects stored in each cache tree.
William Lallemand86d0df02017-11-24 21:36:45 +01001887
1888 $ echo 'show cache' | socat stdio /tmp/sock1
1889 0x7f6ac6c5b03a: foobar (shctx:0x7f6ac6c5b000, available blocks:3918)
1890 1 2 3 4
1891
1892 1. pointer to the cache structure
1893 2. cache name
1894 3. pointer to the mmap area (shctx)
1895 4. number of blocks available for reuse in the shctx
1896
1897 0x7f6ac6c5b4cc hash:286881868 size:39114 (39 blocks), refcount:9, expire:237
1898 1 2 3 4 5 6
1899
1900 1. pointer to the cache entry
1901 2. first 32 bits of the hash
1902 3. size of the object in bytes
1903 4. number of blocks used for the object
1904 5. number of transactions using the entry
1905 6. expiration time, can be negative if already expired
1906
Willy Tarreauae795722016-02-16 11:27:28 +01001907show env [<name>]
1908 Dump one or all environment variables known by the process. Without any
1909 argument, all variables are dumped. With an argument, only the specified
1910 variable is dumped if it exists. Otherwise "Variable not found" is emitted.
1911 Variables are dumped in the same format as they are stored or returned by the
1912 "env" utility, that is, "<name>=<value>". This can be handy when debugging
1913 certain configuration files making heavy use of environment variables to
1914 ensure that they contain the expected values. This command is restricted and
1915 can only be issued on sockets configured for levels "operator" or "admin".
1916
Willy Tarreau35069f82016-11-25 09:16:37 +01001917show errors [<iid>|<proxy>] [request|response]
Willy Tarreau44aed902015-10-13 14:45:29 +02001918 Dump last known request and response errors collected by frontends and
1919 backends. If <iid> is specified, the limit the dump to errors concerning
Willy Tarreau234ba2d2016-11-25 08:39:10 +01001920 either frontend or backend whose ID is <iid>. Proxy ID "-1" will cause
1921 all instances to be dumped. If a proxy name is specified instead, its ID
Willy Tarreau35069f82016-11-25 09:16:37 +01001922 will be used as the filter. If "request" or "response" is added after the
1923 proxy name or ID, only request or response errors will be dumped. This
1924 command is restricted and can only be issued on sockets configured for
1925 levels "operator" or "admin".
Willy Tarreau44aed902015-10-13 14:45:29 +02001926
1927 The errors which may be collected are the last request and response errors
1928 caused by protocol violations, often due to invalid characters in header
1929 names. The report precisely indicates what exact character violated the
1930 protocol. Other important information such as the exact date the error was
1931 detected, frontend and backend names, the server name (when known), the
1932 internal session ID and the source address which has initiated the session
1933 are reported too.
1934
1935 All characters are returned, and non-printable characters are encoded. The
1936 most common ones (\t = 9, \n = 10, \r = 13 and \e = 27) are encoded as one
1937 letter following a backslash. The backslash itself is encoded as '\\' to
1938 avoid confusion. Other non-printable characters are encoded '\xNN' where
1939 NN is the two-digits hexadecimal representation of the character's ASCII
1940 code.
1941
1942 Lines are prefixed with the position of their first character, starting at 0
1943 for the beginning of the buffer. At most one input line is printed per line,
1944 and large lines will be broken into multiple consecutive output lines so that
1945 the output never goes beyond 79 characters wide. It is easy to detect if a
1946 line was broken, because it will not end with '\n' and the next line's offset
1947 will be followed by a '+' sign, indicating it is a continuation of previous
1948 line.
1949
1950 Example :
Willy Tarreau35069f82016-11-25 09:16:37 +01001951 $ echo "show errors -1 response" | socat stdio /tmp/sock1
Willy Tarreau44aed902015-10-13 14:45:29 +02001952 >>> [04/Mar/2009:15:46:56.081] backend http-in (#2) : invalid response
1953 src 127.0.0.1, session #54, frontend fe-eth0 (#1), server s2 (#1)
1954 response length 213 bytes, error at position 23:
1955
1956 00000 HTTP/1.0 200 OK\r\n
1957 00017 header/bizarre:blah\r\n
1958 00038 Location: blah\r\n
1959 00054 Long-line: this is a very long line which should b
1960 00104+ e broken into multiple lines on the output buffer,
1961 00154+ otherwise it would be too large to print in a ter
1962 00204+ minal\r\n
1963 00211 \r\n
1964
1965 In the example above, we see that the backend "http-in" which has internal
1966 ID 2 has blocked an invalid response from its server s2 which has internal
1967 ID 1. The request was on session 54 initiated by source 127.0.0.1 and
1968 received by frontend fe-eth0 whose ID is 1. The total response length was
1969 213 bytes when the error was detected, and the error was at byte 23. This
1970 is the slash ('/') in header name "header/bizarre", which is not a valid
1971 HTTP character for a header name.
1972
Willy Tarreau7a4a0ac2017-07-25 19:32:50 +02001973show fd [<fd>]
1974 Dump the list of either all open file descriptors or just the one number <fd>
1975 if specified. This is only aimed at developers who need to observe internal
1976 states in order to debug complex issues such as abnormal CPU usages. One fd
1977 is reported per lines, and for each of them, its state in the poller using
1978 upper case letters for enabled flags and lower case for disabled flags, using
1979 "P" for "polled", "R" for "ready", "A" for "active", the events status using
1980 "H" for "hangup", "E" for "error", "O" for "output", "P" for "priority" and
1981 "I" for "input", a few other flags like "N" for "new" (just added into the fd
1982 cache), "U" for "updated" (received an update in the fd cache), "L" for
1983 "linger_risk", "C" for "cloned", then the cached entry position, the pointer
1984 to the internal owner, the pointer to the I/O callback and its name when
1985 known. When the owner is a connection, the connection flags, and the target
1986 are reported (frontend, proxy or server). When the owner is a listener, the
1987 listener's state and its frontend are reported. There is no point in using
1988 this command without a good knowledge of the internals. It's worth noting
1989 that the output format may evolve over time so this output must not be parsed
1990 by tools designed to be durable.
1991
Simon Horman05ee2132017-01-04 09:37:25 +01001992show info [typed|json]
Willy Tarreau5d8b9792016-03-11 11:09:34 +01001993 Dump info about haproxy status on current process. If "typed" is passed as an
1994 optional argument, field numbers, names and types are emitted as well so that
1995 external monitoring products can easily retrieve, possibly aggregate, then
1996 report information found in fields they don't know. Each field is dumped on
Simon Horman05ee2132017-01-04 09:37:25 +01001997 its own line. If "json" is passed as an optional argument then
1998 information provided by "typed" output is provided in JSON format as a
1999 list of JSON objects. By default, the format contains only two columns
2000 delimited by a colon (':'). The left one is the field name and the right
2001 one is the value. It is very important to note that in typed output
2002 format, the dump for a single object is contiguous so that there is no
2003 need for a consumer to store everything at once.
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002004
2005 When using the typed output format, each line is made of 4 columns delimited
2006 by colons (':'). The first column is a dot-delimited series of 3 elements. The
2007 first element is the numeric position of the field in the list (starting at
2008 zero). This position shall not change over time, but holes are to be expected,
2009 depending on build options or if some fields are deleted in the future. The
2010 second element is the field name as it appears in the default "show info"
2011 output. The third element is the relative process number starting at 1.
2012
2013 The rest of the line starting after the first colon follows the "typed output
2014 format" described in the section above. In short, the second column (after the
2015 first ':') indicates the origin, nature and scope of the variable. The third
2016 column indicates the type of the field, among "s32", "s64", "u32", "u64" and
2017 "str". Then the fourth column is the value itself, which the consumer knows
2018 how to parse thanks to column 3 and how to process thanks to column 2.
2019
2020 Thus the overall line format in typed mode is :
2021
2022 <field_pos>.<field_name>.<process_num>:<tags>:<type>:<value>
2023
2024 Example :
2025
2026 > show info
2027 Name: HAProxy
2028 Version: 1.7-dev1-de52ea-146
2029 Release_date: 2016/03/11
2030 Nbproc: 1
2031 Process_num: 1
2032 Pid: 28105
2033 Uptime: 0d 0h00m04s
2034 Uptime_sec: 4
2035 Memmax_MB: 0
2036 PoolAlloc_MB: 0
2037 PoolUsed_MB: 0
2038 PoolFailed: 0
2039 (...)
2040
2041 > show info typed
2042 0.Name.1:POS:str:HAProxy
2043 1.Version.1:POS:str:1.7-dev1-de52ea-146
2044 2.Release_date.1:POS:str:2016/03/11
2045 3.Nbproc.1:CGS:u32:1
2046 4.Process_num.1:KGP:u32:1
2047 5.Pid.1:SGP:u32:28105
2048 6.Uptime.1:MDP:str:0d 0h00m08s
2049 7.Uptime_sec.1:MDP:u32:8
2050 8.Memmax_MB.1:CLP:u32:0
2051 9.PoolAlloc_MB.1:MGP:u32:0
2052 10.PoolUsed_MB.1:MGP:u32:0
2053 11.PoolFailed.1:MCP:u32:0
2054 (...)
2055
Simon Horman1084a362016-11-21 17:00:24 +01002056 In the typed format, the presence of the process ID at the end of the
2057 first column makes it very easy to visually aggregate outputs from
2058 multiple processes.
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002059 Example :
2060
2061 $ ( echo show info typed | socat /var/run/haproxy.sock1 ; \
2062 echo show info typed | socat /var/run/haproxy.sock2 ) | \
2063 sort -t . -k 1,1n -k 2,2 -k 3,3n
2064 0.Name.1:POS:str:HAProxy
2065 0.Name.2:POS:str:HAProxy
2066 1.Version.1:POS:str:1.7-dev1-868ab3-148
2067 1.Version.2:POS:str:1.7-dev1-868ab3-148
2068 2.Release_date.1:POS:str:2016/03/11
2069 2.Release_date.2:POS:str:2016/03/11
2070 3.Nbproc.1:CGS:u32:2
2071 3.Nbproc.2:CGS:u32:2
2072 4.Process_num.1:KGP:u32:1
2073 4.Process_num.2:KGP:u32:2
2074 5.Pid.1:SGP:u32:30120
2075 5.Pid.2:SGP:u32:30121
2076 6.Uptime.1:MDP:str:0d 0h01m28s
2077 6.Uptime.2:MDP:str:0d 0h01m28s
2078 (...)
Willy Tarreau44aed902015-10-13 14:45:29 +02002079
Simon Horman05ee2132017-01-04 09:37:25 +01002080 The format of JSON output is described in a schema which may be output
Simon Horman6f6bb382017-01-04 09:37:26 +01002081 using "show schema json".
Simon Horman05ee2132017-01-04 09:37:25 +01002082
2083 The JSON output contains no extra whitespace in order to reduce the
2084 volume of output. For human consumption passing the output through a
2085 pretty printer may be helpful. Example :
2086
2087 $ echo "show info json" | socat /var/run/haproxy.sock stdio | \
2088 python -m json.tool
2089
Simon Horman6f6bb382017-01-04 09:37:26 +01002090 The JSON output contains no extra whitespace in order to reduce the
2091 volume of output. For human consumption passing the output through a
2092 pretty printer may be helpful. Example :
2093
2094 $ echo "show info json" | socat /var/run/haproxy.sock stdio | \
2095 python -m json.tool
2096
Willy Tarreau44aed902015-10-13 14:45:29 +02002097show map [<map>]
2098 Dump info about map converters. Without argument, the list of all available
2099 maps is returned. If a <map> is specified, its contents are dumped. <map> is
2100 the #<id> or <file>. The first column is a unique identifier. It can be used
2101 as reference for the operation "del map" and "set map". The second column is
2102 the pattern and the third column is the sample if available. The data returned
2103 are not directly a list of available maps, but are the list of all patterns
2104 composing any map. Many of these patterns can be shared with ACL.
2105
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002106show peers [<peers section>]
2107 Dump info about the peers configured in "peers" sections. Without argument,
2108 the list of the peers belonging to all the "peers" sections are listed. If
2109 <peers section> is specified, only the information about the peers belonging
2110 to this "peers" section are dumped.
2111
Michael Prokop4438c602019-05-24 10:25:45 +02002112 Here are two examples of outputs where hostA, hostB and hostC peers belong to
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002113 "sharedlb" peers sections. Only hostA and hostB are connected. Only hostA has
2114 sent data to hostB.
2115
2116 $ echo "show peers" | socat - /tmp/hostA
2117 0x55deb0224320: [15/Apr/2019:11:28:01] id=sharedlb state=0 flags=0x3 \
Emeric Brun0bbec0f2019-04-18 11:39:43 +02002118 resync_timeout=<PAST> task_calls=45122
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002119 0x55deb022b540: id=hostC(remote) addr=127.0.0.12:10002 status=CONN \
2120 reconnect=4s confirm=0
2121 flags=0x0
2122 0x55deb022a440: id=hostA(local) addr=127.0.0.10:10000 status=NONE \
2123 reconnect=<NEVER> confirm=0
2124 flags=0x0
2125 0x55deb0227d70: id=hostB(remote) addr=127.0.0.11:10001 status=ESTA
2126 reconnect=2s confirm=0
Emeric Brun0bbec0f2019-04-18 11:39:43 +02002127 flags=0x20000200 appctx:0x55deb028fba0 st0=7 st1=0 task_calls=14456 \
2128 state=EST
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002129 xprt=RAW src=127.0.0.1:37257 addr=127.0.0.10:10000
2130 remote_table:0x55deb0224a10 id=stkt local_id=1 remote_id=1
2131 last_local_table:0x55deb0224a10 id=stkt local_id=1 remote_id=1
2132 shared tables:
2133 0x55deb0224a10 local_id=1 remote_id=1 flags=0x0 remote_data=0x65
2134 last_acked=0 last_pushed=3 last_get=0 teaching_origin=0 update=3
2135 table:0x55deb022d6a0 id=stkt update=3 localupdate=3 \
2136 commitupdate=3 syncing=0
2137
2138 $ echo "show peers" | socat - /tmp/hostB
2139 0x55871b5ab320: [15/Apr/2019:11:28:03] id=sharedlb state=0 flags=0x3 \
Emeric Brun0bbec0f2019-04-18 11:39:43 +02002140 resync_timeout=<PAST> task_calls=3
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002141 0x55871b5b2540: id=hostC(remote) addr=127.0.0.12:10002 status=CONN \
2142 reconnect=3s confirm=0
2143 flags=0x0
2144 0x55871b5b1440: id=hostB(local) addr=127.0.0.11:10001 status=NONE \
2145 reconnect=<NEVER> confirm=0
2146 flags=0x0
2147 0x55871b5aed70: id=hostA(remote) addr=127.0.0.10:10000 status=ESTA \
2148 reconnect=2s confirm=0
Emeric Brun0bbec0f2019-04-18 11:39:43 +02002149 flags=0x20000200 appctx:0x7fa46800ee00 st0=7 st1=0 task_calls=62356 \
2150 state=EST
Frédéric Lécaille21dde502019-04-15 13:50:23 +02002151 remote_table:0x55871b5ab960 id=stkt local_id=1 remote_id=1
2152 last_local_table:0x55871b5ab960 id=stkt local_id=1 remote_id=1
2153 shared tables:
2154 0x55871b5ab960 local_id=1 remote_id=1 flags=0x0 remote_data=0x65
2155 last_acked=3 last_pushed=0 last_get=3 teaching_origin=0 update=0
2156 table:0x55871b5b46a0 id=stkt update=1 localupdate=0 \
2157 commitupdate=0 syncing=0
2158
Willy Tarreau44aed902015-10-13 14:45:29 +02002159show pools
2160 Dump the status of internal memory pools. This is useful to track memory
2161 usage when suspecting a memory leak for example. It does exactly the same
2162 as the SIGQUIT when running in foreground except that it does not flush
2163 the pools.
2164
Willy Tarreau75c62c22018-11-22 11:02:09 +01002165show profiling
2166 Dumps the current profiling settings, one per line, as well as the command
2167 needed to change them.
2168
Willy Tarreau44aed902015-10-13 14:45:29 +02002169show servers state [<backend>]
2170 Dump the state of the servers found in the running configuration. A backend
2171 name or identifier may be provided to limit the output to this backend only.
2172
2173 The dump has the following format:
2174 - first line contains the format version (1 in this specification);
2175 - second line contains the column headers, prefixed by a sharp ('#');
2176 - third line and next ones contain data;
2177 - each line starting by a sharp ('#') is considered as a comment.
2178
Dan Lloyd8e48b872016-07-01 21:01:18 -04002179 Since multiple versions of the output may co-exist, below is the list of
Willy Tarreau44aed902015-10-13 14:45:29 +02002180 fields and their order per file format version :
2181 1:
2182 be_id: Backend unique id.
2183 be_name: Backend label.
2184 srv_id: Server unique id (in the backend).
2185 srv_name: Server label.
2186 srv_addr: Server IP address.
2187 srv_op_state: Server operational state (UP/DOWN/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002188 0 = SRV_ST_STOPPED
2189 The server is down.
2190 1 = SRV_ST_STARTING
2191 The server is warming up (up but
2192 throttled).
2193 2 = SRV_ST_RUNNING
2194 The server is fully up.
2195 3 = SRV_ST_STOPPING
2196 The server is up but soft-stopping
2197 (eg: 404).
Willy Tarreau44aed902015-10-13 14:45:29 +02002198 srv_admin_state: Server administrative state (MAINT/DRAIN/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002199 The state is actually a mask of values :
2200 0x01 = SRV_ADMF_FMAINT
2201 The server was explicitly forced into
2202 maintenance.
2203 0x02 = SRV_ADMF_IMAINT
2204 The server has inherited the maintenance
2205 status from a tracked server.
2206 0x04 = SRV_ADMF_CMAINT
2207 The server is in maintenance because of
2208 the configuration.
2209 0x08 = SRV_ADMF_FDRAIN
2210 The server was explicitly forced into
2211 drain state.
2212 0x10 = SRV_ADMF_IDRAIN
2213 The server has inherited the drain status
2214 from a tracked server.
Baptiste Assmann89aa7f32016-11-02 21:31:27 +01002215 0x20 = SRV_ADMF_RMAINT
2216 The server is in maintenance because of an
2217 IP address resolution failure.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02002218 0x40 = SRV_ADMF_HMAINT
2219 The server FQDN was set from stats socket.
2220
Willy Tarreau44aed902015-10-13 14:45:29 +02002221 srv_uweight: User visible server's weight.
2222 srv_iweight: Server's initial weight.
2223 srv_time_since_last_change: Time since last operational change.
2224 srv_check_status: Last health check status.
2225 srv_check_result: Last check result (FAILED/PASSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002226 0 = CHK_RES_UNKNOWN
2227 Initialized to this by default.
2228 1 = CHK_RES_NEUTRAL
2229 Valid check but no status information.
2230 2 = CHK_RES_FAILED
2231 Check failed.
2232 3 = CHK_RES_PASSED
2233 Check succeeded and server is fully up
2234 again.
2235 4 = CHK_RES_CONDPASS
2236 Check reports the server doesn't want new
2237 sessions.
Willy Tarreau44aed902015-10-13 14:45:29 +02002238 srv_check_health: Checks rise / fall current counter.
2239 srv_check_state: State of the check (ENABLED/PAUSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002240 The state is actually a mask of values :
2241 0x01 = CHK_ST_INPROGRESS
2242 A check is currently running.
2243 0x02 = CHK_ST_CONFIGURED
2244 This check is configured and may be
2245 enabled.
2246 0x04 = CHK_ST_ENABLED
2247 This check is currently administratively
2248 enabled.
2249 0x08 = CHK_ST_PAUSED
2250 Checks are paused because of maintenance
2251 (health only).
Willy Tarreau44aed902015-10-13 14:45:29 +02002252 srv_agent_state: State of the agent check (ENABLED/PAUSED/...).
Cyril Bonté5b2ce8a2016-11-02 00:19:58 +01002253 This state uses the same mask values as
2254 "srv_check_state", adding this specific one :
2255 0x10 = CHK_ST_AGENT
2256 Check is an agent check (otherwise it's a
2257 health check).
Willy Tarreau44aed902015-10-13 14:45:29 +02002258 bk_f_forced_id: Flag to know if the backend ID is forced by
2259 configuration.
2260 srv_f_forced_id: Flag to know if the server's ID is forced by
2261 configuration.
Frédéric Lécailleb418c122017-04-26 11:24:02 +02002262 srv_fqdn: Server FQDN.
Frédéric Lécaille31694712017-08-01 08:47:19 +02002263 srv_port: Server port.
Baptiste Assmann6d0f38f2018-07-02 17:00:54 +02002264 srvrecord: DNS SRV record associated to this SRV.
Willy Tarreau44aed902015-10-13 14:45:29 +02002265
2266show sess
2267 Dump all known sessions. Avoid doing this on slow connections as this can
2268 be huge. This command is restricted and can only be issued on sockets
2269 configured for levels "operator" or "admin".
2270
2271show sess <id>
2272 Display a lot of internal information about the specified session identifier.
2273 This identifier is the first field at the beginning of the lines in the dumps
2274 of "show sess" (it corresponds to the session pointer). Those information are
2275 useless to most users but may be used by haproxy developers to troubleshoot a
2276 complex bug. The output format is intentionally not documented so that it can
2277 freely evolve depending on demands. You may find a description of all fields
2278 returned in src/dumpstats.c
2279
2280 The special id "all" dumps the states of all sessions, which must be avoided
2281 as much as possible as it is highly CPU intensive and can take a lot of time.
2282
Simon Horman05ee2132017-01-04 09:37:25 +01002283show stat [{<iid>|<proxy>} <type> <sid>] [typed|json]
2284 Dump statistics using the CSV format; using the extended typed output
2285 format described in the section above if "typed" is passed after the
2286 other arguments; or in JSON if "json" is passed after the other arguments
2287 . By passing <id>, <type> and <sid>, it is possible to dump only selected
2288 items :
Willy Tarreaua1b1ed52016-11-25 08:50:58 +01002289 - <iid> is a proxy ID, -1 to dump everything. Alternatively, a proxy name
2290 <proxy> may be specified. In this case, this proxy's ID will be used as
2291 the ID selector.
Willy Tarreau44aed902015-10-13 14:45:29 +02002292 - <type> selects the type of dumpable objects : 1 for frontends, 2 for
2293 backends, 4 for servers, -1 for everything. These values can be ORed,
2294 for example:
2295 1 + 2 = 3 -> frontend + backend.
2296 1 + 2 + 4 = 7 -> frontend + backend + server.
2297 - <sid> is a server ID, -1 to dump everything from the selected proxy.
2298
2299 Example :
2300 $ echo "show info;show stat" | socat stdio unix-connect:/tmp/sock1
2301 >>> Name: HAProxy
2302 Version: 1.4-dev2-49
2303 Release_date: 2009/09/23
2304 Nbproc: 1
2305 Process_num: 1
2306 (...)
2307
2308 # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq, (...)
2309 stats,FRONTEND,,,0,0,1000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,1,0, (...)
2310 stats,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,250,(...)
2311 (...)
2312 www1,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,0,250, (...)
2313
2314 $
2315
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002316 In this example, two commands have been issued at once. That way it's easy to
2317 find which process the stats apply to in multi-process mode. This is not
2318 needed in the typed output format as the process number is reported on each
2319 line. Notice the empty line after the information output which marks the end
2320 of the first block. A similar empty line appears at the end of the second
2321 block (stats) so that the reader knows the output has not been truncated.
2322
2323 When "typed" is specified, the output format is more suitable to monitoring
2324 tools because it provides numeric positions and indicates the type of each
2325 output field. Each value stands on its own line with process number, element
2326 number, nature, origin and scope. This same format is available via the HTTP
2327 stats by passing ";typed" after the URI. It is very important to note that in
Dan Lloyd8e48b872016-07-01 21:01:18 -04002328 typed output format, the dump for a single object is contiguous so that there
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002329 is no need for a consumer to store everything at once.
2330
2331 When using the typed output format, each line is made of 4 columns delimited
2332 by colons (':'). The first column is a dot-delimited series of 5 elements. The
2333 first element is a letter indicating the type of the object being described.
2334 At the moment the following object types are known : 'F' for a frontend, 'B'
2335 for a backend, 'L' for a listener, and 'S' for a server. The second element
2336 The second element is a positive integer representing the unique identifier of
2337 the proxy the object belongs to. It is equivalent to the "iid" column of the
2338 CSV output and matches the value in front of the optional "id" directive found
2339 in the frontend or backend section. The third element is a positive integer
2340 containing the unique object identifier inside the proxy, and corresponds to
2341 the "sid" column of the CSV output. ID 0 is reported when dumping a frontend
2342 or a backend. For a listener or a server, this corresponds to their respective
2343 ID inside the proxy. The fourth element is the numeric position of the field
2344 in the list (starting at zero). This position shall not change over time, but
2345 holes are to be expected, depending on build options or if some fields are
2346 deleted in the future. The fifth element is the field name as it appears in
2347 the CSV output. The sixth element is a positive integer and is the relative
2348 process number starting at 1.
2349
2350 The rest of the line starting after the first colon follows the "typed output
2351 format" described in the section above. In short, the second column (after the
2352 first ':') indicates the origin, nature and scope of the variable. The third
2353 column indicates the type of the field, among "s32", "s64", "u32", "u64" and
2354 "str". Then the fourth column is the value itself, which the consumer knows
2355 how to parse thanks to column 3 and how to process thanks to column 2.
2356
2357 Thus the overall line format in typed mode is :
2358
2359 <obj>.<px_id>.<id>.<fpos>.<fname>.<process_num>:<tags>:<type>:<value>
2360
2361 Here's an example of typed output format :
2362
2363 $ echo "show stat typed" | socat stdio unix-connect:/tmp/sock1
2364 F.2.0.0.pxname.1:MGP:str:private-frontend
2365 F.2.0.1.svname.1:MGP:str:FRONTEND
2366 F.2.0.8.bin.1:MGP:u64:0
2367 F.2.0.9.bout.1:MGP:u64:0
2368 F.2.0.40.hrsp_2xx.1:MGP:u64:0
2369 L.2.1.0.pxname.1:MGP:str:private-frontend
2370 L.2.1.1.svname.1:MGP:str:sock-1
2371 L.2.1.17.status.1:MGP:str:OPEN
2372 L.2.1.73.addr.1:MGP:str:0.0.0.0:8001
2373 S.3.13.60.rtime.1:MCP:u32:0
2374 S.3.13.61.ttime.1:MCP:u32:0
2375 S.3.13.62.agent_status.1:MGP:str:L4TOUT
2376 S.3.13.64.agent_duration.1:MGP:u64:2001
2377 S.3.13.65.check_desc.1:MCP:str:Layer4 timeout
2378 S.3.13.66.agent_desc.1:MCP:str:Layer4 timeout
2379 S.3.13.67.check_rise.1:MCP:u32:2
2380 S.3.13.68.check_fall.1:MCP:u32:3
2381 S.3.13.69.check_health.1:SGP:u32:0
2382 S.3.13.70.agent_rise.1:MaP:u32:1
2383 S.3.13.71.agent_fall.1:SGP:u32:1
2384 S.3.13.72.agent_health.1:SGP:u32:1
2385 S.3.13.73.addr.1:MCP:str:1.255.255.255:8888
2386 S.3.13.75.mode.1:MAP:str:http
2387 B.3.0.0.pxname.1:MGP:str:private-backend
2388 B.3.0.1.svname.1:MGP:str:BACKEND
2389 B.3.0.2.qcur.1:MGP:u32:0
2390 B.3.0.3.qmax.1:MGP:u32:0
2391 B.3.0.4.scur.1:MGP:u32:0
2392 B.3.0.5.smax.1:MGP:u32:0
2393 B.3.0.6.slim.1:MGP:u32:1000
2394 B.3.0.55.lastsess.1:MMP:s32:-1
2395 (...)
2396
Simon Horman1084a362016-11-21 17:00:24 +01002397 In the typed format, the presence of the process ID at the end of the
2398 first column makes it very easy to visually aggregate outputs from
2399 multiple processes, as show in the example below where each line appears
2400 for each process :
Willy Tarreau5d8b9792016-03-11 11:09:34 +01002401
2402 $ ( echo show stat typed | socat /var/run/haproxy.sock1 - ; \
2403 echo show stat typed | socat /var/run/haproxy.sock2 - ) | \
2404 sort -t . -k 1,1 -k 2,2n -k 3,3n -k 4,4n -k 5,5 -k 6,6n
2405 B.3.0.0.pxname.1:MGP:str:private-backend
2406 B.3.0.0.pxname.2:MGP:str:private-backend
2407 B.3.0.1.svname.1:MGP:str:BACKEND
2408 B.3.0.1.svname.2:MGP:str:BACKEND
2409 B.3.0.2.qcur.1:MGP:u32:0
2410 B.3.0.2.qcur.2:MGP:u32:0
2411 B.3.0.3.qmax.1:MGP:u32:0
2412 B.3.0.3.qmax.2:MGP:u32:0
2413 B.3.0.4.scur.1:MGP:u32:0
2414 B.3.0.4.scur.2:MGP:u32:0
2415 B.3.0.5.smax.1:MGP:u32:0
2416 B.3.0.5.smax.2:MGP:u32:0
2417 B.3.0.6.slim.1:MGP:u32:1000
2418 B.3.0.6.slim.2:MGP:u32:1000
2419 (...)
Willy Tarreau44aed902015-10-13 14:45:29 +02002420
Simon Horman05ee2132017-01-04 09:37:25 +01002421 The format of JSON output is described in a schema which may be output
Simon Horman6f6bb382017-01-04 09:37:26 +01002422 using "show schema json".
2423
2424 The JSON output contains no extra whitespace in order to reduce the
2425 volume of output. For human consumption passing the output through a
2426 pretty printer may be helpful. Example :
2427
2428 $ echo "show stat json" | socat /var/run/haproxy.sock stdio | \
2429 python -m json.tool
Simon Horman05ee2132017-01-04 09:37:25 +01002430
2431 The JSON output contains no extra whitespace in order to reduce the
2432 volume of output. For human consumption passing the output through a
2433 pretty printer may be helpful. Example :
2434
2435 $ echo "show stat json" | socat /var/run/haproxy.sock stdio | \
2436 python -m json.tool
2437
Christopher Faulet7a5303a2019-09-27 10:45:47 +02002438show resolvers [<resolvers section id>]
Willy Tarreau44aed902015-10-13 14:45:29 +02002439 Dump statistics for the given resolvers section, or all resolvers sections
2440 if no section is supplied.
2441
2442 For each name server, the following counters are reported:
2443 sent: number of DNS requests sent to this server
2444 valid: number of DNS valid responses received from this server
2445 update: number of DNS responses used to update the server's IP address
2446 cname: number of CNAME responses
2447 cname_error: CNAME errors encountered with this server
2448 any_err: number of empty response (IE: server does not support ANY type)
2449 nx: non existent domain response received from this server
2450 timeout: how many time this server did not answer in time
2451 refused: number of requests refused by this server
2452 other: any other DNS errors
2453 invalid: invalid DNS response (from a protocol point of view)
2454 too_big: too big response
2455 outdated: number of response arrived too late (after an other name server)
2456
2457show table
2458 Dump general information on all known stick-tables. Their name is returned
2459 (the name of the proxy which holds them), their type (currently zero, always
2460 IP), their size in maximum possible number of entries, and the number of
2461 entries currently in use.
2462
2463 Example :
2464 $ echo "show table" | socat stdio /tmp/sock1
2465 >>> # table: front_pub, type: ip, size:204800, used:171454
2466 >>> # table: back_rdp, type: ip, size:204800, used:0
2467
2468show table <name> [ data.<type> <operator> <value> ] | [ key <key> ]
2469 Dump contents of stick-table <name>. In this mode, a first line of generic
2470 information about the table is reported as with "show table", then all
2471 entries are dumped. Since this can be quite heavy, it is possible to specify
2472 a filter in order to specify what entries to display.
2473
2474 When the "data." form is used the filter applies to the stored data (see
2475 "stick-table" in section 4.2). A stored data type must be specified
2476 in <type>, and this data type must be stored in the table otherwise an
2477 error is reported. The data is compared according to <operator> with the
2478 64-bit integer <value>. Operators are the same as with the ACLs :
2479
2480 - eq : match entries whose data is equal to this value
2481 - ne : match entries whose data is not equal to this value
2482 - le : match entries whose data is less than or equal to this value
2483 - ge : match entries whose data is greater than or equal to this value
2484 - lt : match entries whose data is less than this value
2485 - gt : match entries whose data is greater than this value
2486
2487
2488 When the key form is used the entry <key> is shown. The key must be of the
2489 same type as the table, which currently is limited to IPv4, IPv6, integer,
2490 and string.
2491
2492 Example :
2493 $ echo "show table http_proxy" | socat stdio /tmp/sock1
2494 >>> # table: http_proxy, type: ip, size:204800, used:2
2495 >>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
2496 bytes_out_rate(60000)=187
2497 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2498 bytes_out_rate(60000)=191
2499
2500 $ echo "show table http_proxy data.gpc0 gt 0" | socat stdio /tmp/sock1
2501 >>> # table: http_proxy, type: ip, size:204800, used:2
2502 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2503 bytes_out_rate(60000)=191
2504
2505 $ echo "show table http_proxy data.conn_rate gt 5" | \
2506 socat stdio /tmp/sock1
2507 >>> # table: http_proxy, type: ip, size:204800, used:2
2508 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2509 bytes_out_rate(60000)=191
2510
2511 $ echo "show table http_proxy key 127.0.0.2" | \
2512 socat stdio /tmp/sock1
2513 >>> # table: http_proxy, type: ip, size:204800, used:2
2514 >>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
2515 bytes_out_rate(60000)=191
2516
2517 When the data criterion applies to a dynamic value dependent on time such as
2518 a bytes rate, the value is dynamically computed during the evaluation of the
2519 entry in order to decide whether it has to be dumped or not. This means that
2520 such a filter could match for some time then not match anymore because as
2521 time goes, the average event rate drops.
2522
2523 It is possible to use this to extract lists of IP addresses abusing the
2524 service, in order to monitor them or even blacklist them in a firewall.
2525 Example :
2526 $ echo "show table http_proxy data.gpc0 gt 0" \
2527 | socat stdio /tmp/sock1 \
2528 | fgrep 'key=' | cut -d' ' -f2 | cut -d= -f2 > abusers-ip.txt
2529 ( or | awk '/key/{ print a[split($2,a,"=")]; }' )
2530
Willy Tarreau4e2b6462019-05-16 17:44:30 +02002531show threads
2532 Dumps some internal states and structures for each thread, that may be useful
2533 to help developers understand a problem. The output tries to be readable by
Willy Tarreauc7091d82019-05-17 10:08:49 +02002534 showing one block per thread. When haproxy is built with USE_THREAD_DUMP=1,
2535 an advanced dump mechanism involving thread signals is used so that each
2536 thread can dump its own state in turn. Without this option, the thread
2537 processing the command shows all its details but the other ones are less
Willy Tarreaue6a02fa2019-05-22 07:06:44 +02002538 detailed. A star ('*') is displayed in front of the thread handling the
2539 command. A right angle bracket ('>') may also be displayed in front of
2540 threads which didn't make any progress since last invocation of this command,
2541 indicating a bug in the code which must absolutely be reported. When this
2542 happens between two threads it usually indicates a deadlock. If a thread is
2543 alone, it's a different bug like a corrupted list. In all cases the process
2544 needs is not fully functional anymore and needs to be restarted.
2545
2546 The output format is purposely not documented so that it can easily evolve as
2547 new needs are identified, without having to maintain any form of backwards
2548 compatibility, and just like with "show activity", the values are meaningless
2549 without the code at hand.
Willy Tarreau4e2b6462019-05-16 17:44:30 +02002550
William Lallemandbb933462016-05-31 21:09:53 +02002551show tls-keys [id|*]
2552 Dump all loaded TLS ticket keys references. The TLS ticket key reference ID
2553 and the file from which the keys have been loaded is shown. Both of those
2554 can be used to update the TLS keys using "set ssl tls-key". If an ID is
2555 specified as parameter, it will dump the tickets, using * it will dump every
2556 keys from every references.
Willy Tarreau44aed902015-10-13 14:45:29 +02002557
Simon Horman6f6bb382017-01-04 09:37:26 +01002558show schema json
2559 Dump the schema used for the output of "show info json" and "show stat json".
2560
2561 The contains no extra whitespace in order to reduce the volume of output.
2562 For human consumption passing the output through a pretty printer may be
2563 helpful. Example :
2564
2565 $ echo "show schema json" | socat /var/run/haproxy.sock stdio | \
2566 python -m json.tool
2567
2568 The schema follows "JSON Schema" (json-schema.org) and accordingly
2569 verifiers may be used to verify the output of "show info json" and "show
2570 stat json" against the schema.
2571
2572
Willy Tarreau44aed902015-10-13 14:45:29 +02002573shutdown frontend <frontend>
2574 Completely delete the specified frontend. All the ports it was bound to will
2575 be released. It will not be possible to enable the frontend anymore after
2576 this operation. This is intended to be used in environments where stopping a
2577 proxy is not even imaginable but a misconfigured proxy must be fixed. That
2578 way it's possible to release the port and bind it into another process to
2579 restore operations. The frontend will not appear at all on the stats page
2580 once it is terminated.
2581
2582 The frontend may be specified either by its name or by its numeric ID,
2583 prefixed with a sharp ('#').
2584
2585 This command is restricted and can only be issued on sockets configured for
2586 level "admin".
2587
2588shutdown session <id>
2589 Immediately terminate the session matching the specified session identifier.
2590 This identifier is the first field at the beginning of the lines in the dumps
2591 of "show sess" (it corresponds to the session pointer). This can be used to
2592 terminate a long-running session without waiting for a timeout or when an
2593 endless transfer is ongoing. Such terminated sessions are reported with a 'K'
2594 flag in the logs.
2595
2596shutdown sessions server <backend>/<server>
2597 Immediately terminate all the sessions attached to the specified server. This
2598 can be used to terminate long-running sessions after a server is put into
2599 maintenance mode, for instance. Such terminated sessions are reported with a
2600 'K' flag in the logs.
2601
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002602
William Lallemand142db372018-12-11 18:56:45 +010026039.4. Master CLI
2604---------------
2605
2606The master CLI is a socket bound to the master process in master-worker mode.
2607This CLI gives access to the unix socket commands in every running or leaving
2608processes and allows a basic supervision of those processes.
2609
2610The master CLI is configurable only from the haproxy program arguments with
2611the -S option. This option also takes bind options separated by commas.
2612
2613Example:
2614
2615 # haproxy -W -S 127.0.0.1:1234 -f test1.cfg
2616 # haproxy -Ws -S /tmp/master-socket,uid,1000,gid,1000,mode,600 -f test1.cfg
William Lallemandb7ea1412018-12-13 09:05:47 +01002617 # haproxy -W -S /tmp/master-socket,level,user -f test1.cfg
William Lallemand142db372018-12-11 18:56:45 +01002618
2619The master CLI introduces a new 'show proc' command to surpervise the
2620processes:
2621
2622Example:
2623
2624 $ echo 'show proc' | socat /var/run/haproxy-master.sock -
William Lallemand1dc69632019-06-12 19:11:33 +02002625 #<PID> <type> <relative PID> <reloads> <uptime> <version>
2626 1162 master 0 5 0d00h02m07s 2.0-dev7-0124c9-7
William Lallemand142db372018-12-11 18:56:45 +01002627 # workers
William Lallemand1dc69632019-06-12 19:11:33 +02002628 1271 worker 1 0 0d00h00m00s 2.0-dev7-0124c9-7
2629 1272 worker 2 0 0d00h00m00s 2.0-dev7-0124c9-7
William Lallemand142db372018-12-11 18:56:45 +01002630 # old workers
William Lallemand1dc69632019-06-12 19:11:33 +02002631 1233 worker [was: 1] 3 0d00h00m43s 2.0-dev3-6019f6-289
William Lallemand142db372018-12-11 18:56:45 +01002632
2633
2634In this example, the master has been reloaded 5 times but one of the old
2635worker is still running and survived 3 reloads. You could access the CLI of
2636this worker to understand what's going on.
2637
Willy Tarreau52880f92018-12-15 13:30:03 +01002638When the prompt is enabled (via the "prompt" command), the context the CLI is
2639working on is displayed in the prompt. The master is identified by the "master"
2640string, and other processes are identified with their PID. In case the last
2641reload failed, the master prompt will be changed to "master[ReloadFailed]>" so
2642that it becomes visible that the process is still running on the previous
2643configuration and that the new configuration is not operational.
2644
William Lallemand142db372018-12-11 18:56:45 +01002645The master CLI uses a special prefix notation to access the multiple
2646processes. This notation is easily identifiable as it begins by a @.
2647
2648A @ prefix can be followed by a relative process number or by an exclamation
2649point and a PID. (e.g. @1 or @!1271). A @ alone could be use to specify the
2650master. Leaving processes are only accessible with the PID as relative process
2651number are only usable with the current processes.
2652
2653Examples:
2654
2655 $ socat /var/run/haproxy-master.sock readline
2656 prompt
2657 master> @1 show info; @2 show info
2658 [...]
2659 Process_num: 1
2660 Pid: 1271
2661 [...]
2662 Process_num: 2
2663 Pid: 1272
2664 [...]
2665 master>
2666
2667 $ echo '@!1271 show info; @!1272 show info' | socat /var/run/haproxy-master.sock -
2668 [...]
2669
2670A prefix could be use as a command, which will send every next commands to
2671the specified process.
2672
2673Examples:
2674
2675 $ socat /var/run/haproxy-master.sock readline
2676 prompt
2677 master> @1
2678 1271> show info
2679 [...]
2680 1271> show stat
2681 [...]
2682 1271> @
2683 master>
2684
2685 $ echo '@1; show info; show stat; @2; show info; show stat' | socat /var/run/haproxy-master.sock -
2686 [...]
2687
William Lallemanda57b7e32018-12-14 21:11:31 +01002688You can also reload the HAProxy master process with the "reload" command which
2689does the same as a `kill -USR2` on the master process, provided that the user
2690has at least "operator" or "admin" privileges.
2691
2692Example:
2693
2694 $ echo "reload" | socat /var/run/haproxy-master.sock
2695
2696Note that a reload will close the connection to the master CLI.
2697
William Lallemand142db372018-12-11 18:56:45 +01002698
Willy Tarreau2212e6a2015-10-13 14:40:55 +0200269910. Tricks for easier configuration management
2700----------------------------------------------
2701
2702It is very common that two HAProxy nodes constituting a cluster share exactly
2703the same configuration modulo a few addresses. Instead of having to maintain a
2704duplicate configuration for each node, which will inevitably diverge, it is
2705possible to include environment variables in the configuration. Thus multiple
2706configuration may share the exact same file with only a few different system
2707wide environment variables. This started in version 1.5 where only addresses
2708were allowed to include environment variables, and 1.6 goes further by
2709supporting environment variables everywhere. The syntax is the same as in the
2710UNIX shell, a variable starts with a dollar sign ('$'), followed by an opening
2711curly brace ('{'), then the variable name followed by the closing brace ('}').
2712Except for addresses, environment variables are only interpreted in arguments
2713surrounded with double quotes (this was necessary not to break existing setups
2714using regular expressions involving the dollar symbol).
2715
2716Environment variables also make it convenient to write configurations which are
2717expected to work on various sites where only the address changes. It can also
2718permit to remove passwords from some configs. Example below where the the file
2719"site1.env" file is sourced by the init script upon startup :
2720
2721 $ cat site1.env
2722 LISTEN=192.168.1.1
2723 CACHE_PFX=192.168.11
2724 SERVER_PFX=192.168.22
2725 LOGGER=192.168.33.1
2726 STATSLP=admin:pa$$w0rd
2727 ABUSERS=/etc/haproxy/abuse.lst
2728 TIMEOUT=10s
2729
2730 $ cat haproxy.cfg
2731 global
2732 log "${LOGGER}:514" local0
2733
2734 defaults
2735 mode http
2736 timeout client "${TIMEOUT}"
2737 timeout server "${TIMEOUT}"
2738 timeout connect 5s
2739
2740 frontend public
2741 bind "${LISTEN}:80"
2742 http-request reject if { src -f "${ABUSERS}" }
2743 stats uri /stats
2744 stats auth "${STATSLP}"
2745 use_backend cache if { path_end .jpg .css .ico }
2746 default_backend server
2747
2748 backend cache
2749 server cache1 "${CACHE_PFX}.1:18080" check
2750 server cache2 "${CACHE_PFX}.2:18080" check
2751
2752 backend server
2753 server cache1 "${SERVER_PFX}.1:8080" check
2754 server cache2 "${SERVER_PFX}.2:8080" check
2755
2756
275711. Well-known traps to avoid
2758-----------------------------
2759
2760Once in a while, someone reports that after a system reboot, the haproxy
2761service wasn't started, and that once they start it by hand it works. Most
2762often, these people are running a clustered IP address mechanism such as
2763keepalived, to assign the service IP address to the master node only, and while
2764it used to work when they used to bind haproxy to address 0.0.0.0, it stopped
2765working after they bound it to the virtual IP address. What happens here is
2766that when the service starts, the virtual IP address is not yet owned by the
2767local node, so when HAProxy wants to bind to it, the system rejects this
2768because it is not a local IP address. The fix doesn't consist in delaying the
2769haproxy service startup (since it wouldn't stand a restart), but instead to
2770properly configure the system to allow binding to non-local addresses. This is
2771easily done on Linux by setting the net.ipv4.ip_nonlocal_bind sysctl to 1. This
2772is also needed in order to transparently intercept the IP traffic that passes
2773through HAProxy for a specific target address.
2774
2775Multi-process configurations involving source port ranges may apparently seem
2776to work but they will cause some random failures under high loads because more
2777than one process may try to use the same source port to connect to the same
2778server, which is not possible. The system will report an error and a retry will
2779happen, picking another port. A high value in the "retries" parameter may hide
2780the effect to a certain extent but this also comes with increased CPU usage and
2781processing time. Logs will also report a certain number of retries. For this
2782reason, port ranges should be avoided in multi-process configurations.
2783
Dan Lloyd8e48b872016-07-01 21:01:18 -04002784Since HAProxy uses SO_REUSEPORT and supports having multiple independent
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002785processes bound to the same IP:port, during troubleshooting it can happen that
2786an old process was not stopped before a new one was started. This provides
2787absurd test results which tend to indicate that any change to the configuration
2788is ignored. The reason is that in fact even the new process is restarted with a
2789new configuration, the old one also gets some incoming connections and
2790processes them, returning unexpected results. When in doubt, just stop the new
2791process and try again. If it still works, it very likely means that an old
2792process remains alive and has to be stopped. Linux's "netstat -lntp" is of good
2793help here.
2794
2795When adding entries to an ACL from the command line (eg: when blacklisting a
2796source address), it is important to keep in mind that these entries are not
2797synchronized to the file and that if someone reloads the configuration, these
2798updates will be lost. While this is often the desired effect (for blacklisting)
2799it may not necessarily match expectations when the change was made as a fix for
2800a problem. See the "add acl" action of the CLI interface.
2801
2802
280312. Debugging and performance issues
2804------------------------------------
2805
2806When HAProxy is started with the "-d" option, it will stay in the foreground
2807and will print one line per event, such as an incoming connection, the end of a
2808connection, and for each request or response header line seen. This debug
2809output is emitted before the contents are processed, so they don't consider the
2810local modifications. The main use is to show the request and response without
2811having to run a network sniffer. The output is less readable when multiple
2812connections are handled in parallel, though the "debug2ansi" and "debug2html"
2813scripts found in the examples/ directory definitely help here by coloring the
2814output.
2815
2816If a request or response is rejected because HAProxy finds it is malformed, the
2817best thing to do is to connect to the CLI and issue "show errors", which will
2818report the last captured faulty request and response for each frontend and
2819backend, with all the necessary information to indicate precisely the first
2820character of the input stream that was rejected. This is sometimes needed to
2821prove to customers or to developers that a bug is present in their code. In
2822this case it is often possible to relax the checks (but still keep the
2823captures) using "option accept-invalid-http-request" or its equivalent for
2824responses coming from the server "option accept-invalid-http-response". Please
2825see the configuration manual for more details.
2826
2827Example :
2828
2829 > show errors
2830 Total events captured on [13/Oct/2015:13:43:47.169] : 1
2831
2832 [13/Oct/2015:13:43:40.918] frontend HAProxyLocalStats (#2): invalid request
2833 backend <NONE> (#-1), server <NONE> (#-1), event #0
2834 src 127.0.0.1:51981, session #0, session flags 0x00000080
2835 HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000
2836 HTTP chunk len 0 bytes, HTTP body len 0 bytes
2837 buffer flags 0x00808002, out 0 bytes, total 31 bytes
2838 pending 31 bytes, wrapping at 8040, error at position 13:
2839
2840 00000 GET /invalid request HTTP/1.1\r\n
2841
2842
2843The output of "show info" on the CLI provides a number of useful information
2844regarding the maximum connection rate ever reached, maximum SSL key rate ever
2845reached, and in general all information which can help to explain temporary
2846issues regarding CPU or memory usage. Example :
2847
2848 > show info
2849 Name: HAProxy
2850 Version: 1.6-dev7-e32d18-17
2851 Release_date: 2015/10/12
2852 Nbproc: 1
2853 Process_num: 1
2854 Pid: 7949
2855 Uptime: 0d 0h02m39s
2856 Uptime_sec: 159
2857 Memmax_MB: 0
2858 Ulimit-n: 120032
2859 Maxsock: 120032
2860 Maxconn: 60000
2861 Hard_maxconn: 60000
2862 CurrConns: 0
2863 CumConns: 3
2864 CumReq: 3
2865 MaxSslConns: 0
2866 CurrSslConns: 0
2867 CumSslConns: 0
2868 Maxpipes: 0
2869 PipesUsed: 0
2870 PipesFree: 0
2871 ConnRate: 0
2872 ConnRateLimit: 0
2873 MaxConnRate: 1
2874 SessRate: 0
2875 SessRateLimit: 0
2876 MaxSessRate: 1
2877 SslRate: 0
2878 SslRateLimit: 0
2879 MaxSslRate: 0
2880 SslFrontendKeyRate: 0
2881 SslFrontendMaxKeyRate: 0
2882 SslFrontendSessionReuse_pct: 0
2883 SslBackendKeyRate: 0
2884 SslBackendMaxKeyRate: 0
2885 SslCacheLookups: 0
2886 SslCacheMisses: 0
2887 CompressBpsIn: 0
2888 CompressBpsOut: 0
2889 CompressBpsRateLim: 0
2890 ZlibMemUsage: 0
2891 MaxZlibMemUsage: 0
2892 Tasks: 5
2893 Run_queue: 1
2894 Idle_pct: 100
2895 node: wtap
2896 description:
2897
2898When an issue seems to randomly appear on a new version of HAProxy (eg: every
2899second request is aborted, occasional crash, etc), it is worth trying to enable
Dan Lloyd8e48b872016-07-01 21:01:18 -04002900memory poisoning so that each call to malloc() is immediately followed by the
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002901filling of the memory area with a configurable byte. By default this byte is
29020x50 (ASCII for 'P'), but any other byte can be used, including zero (which
2903will have the same effect as a calloc() and which may make issues disappear).
Dan Lloyd8e48b872016-07-01 21:01:18 -04002904Memory poisoning is enabled on the command line using the "-dM" option. It
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002905slightly hurts performance and is not recommended for use in production. If
Dan Lloyd8e48b872016-07-01 21:01:18 -04002906an issue happens all the time with it or never happens when poisoning uses
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002907byte zero, it clearly means you've found a bug and you definitely need to
2908report it. Otherwise if there's no clear change, the problem it is not related.
2909
2910When debugging some latency issues, it is important to use both strace and
2911tcpdump on the local machine, and another tcpdump on the remote system. The
2912reason for this is that there are delays everywhere in the processing chain and
2913it is important to know which one is causing latency to know where to act. In
2914practice, the local tcpdump will indicate when the input data come in. Strace
2915will indicate when haproxy receives these data (using recv/recvfrom). Warning,
2916openssl uses read()/write() syscalls instead of recv()/send(). Strace will also
2917show when haproxy sends the data, and tcpdump will show when the system sends
2918these data to the interface. Then the external tcpdump will show when the data
2919sent are really received (since the local one only shows when the packets are
2920queued). The benefit of sniffing on the local system is that strace and tcpdump
2921will use the same reference clock. Strace should be used with "-tts200" to get
2922complete timestamps and report large enough chunks of data to read them.
2923Tcpdump should be used with "-nvvttSs0" to report full packets, real sequence
2924numbers and complete timestamps.
2925
2926In practice, received data are almost always immediately received by haproxy
2927(unless the machine has a saturated CPU or these data are invalid and not
2928delivered). If these data are received but not sent, it generally is because
2929the output buffer is saturated (ie: recipient doesn't consume the data fast
2930enough). This can be confirmed by seeing that the polling doesn't notify of
2931the ability to write on the output file descriptor for some time (it's often
2932easier to spot in the strace output when the data finally leave and then roll
2933back to see when the write event was notified). It generally matches an ACK
2934received from the recipient, and detected by tcpdump. Once the data are sent,
2935they may spend some time in the system doing nothing. Here again, the TCP
2936congestion window may be limited and not allow these data to leave, waiting for
2937an ACK to open the window. If the traffic is idle and the data take 40 ms or
2938200 ms to leave, it's a different issue (which is not an issue), it's the fact
2939that the Nagle algorithm prevents empty packets from leaving immediately, in
2940hope that they will be merged with subsequent data. HAProxy automatically
2941disables Nagle in pure TCP mode and in tunnels. However it definitely remains
2942enabled when forwarding an HTTP body (and this contributes to the performance
2943improvement there by reducing the number of packets). Some HTTP non-compliant
2944applications may be sensitive to the latency when delivering incomplete HTTP
2945response messages. In this case you will have to enable "option http-no-delay"
2946to disable Nagle in order to work around their design, keeping in mind that any
2947other proxy in the chain may similarly be impacted. If tcpdump reports that data
2948leave immediately but the other end doesn't see them quickly, it can mean there
Dan Lloyd8e48b872016-07-01 21:01:18 -04002949is a congested WAN link, a congested LAN with flow control enabled and
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002950preventing the data from leaving, or more commonly that HAProxy is in fact
2951running in a virtual machine and that for whatever reason the hypervisor has
2952decided that the data didn't need to be sent immediately. In virtualized
2953environments, latency issues are almost always caused by the virtualization
2954layer, so in order to save time, it's worth first comparing tcpdump in the VM
2955and on the external components. Any difference has to be credited to the
2956hypervisor and its accompanying drivers.
2957
2958When some TCP SACK segments are seen in tcpdump traces (using -vv), it always
2959means that the side sending them has got the proof of a lost packet. While not
2960seeing them doesn't mean there are no losses, seeing them definitely means the
2961network is lossy. Losses are normal on a network, but at a rate where SACKs are
2962not noticeable at the naked eye. If they appear a lot in the traces, it is
2963worth investigating exactly what happens and where the packets are lost. HTTP
2964doesn't cope well with TCP losses, which introduce huge latencies.
2965
2966The "netstat -i" command will report statistics per interface. An interface
2967where the Rx-Ovr counter grows indicates that the system doesn't have enough
2968resources to receive all incoming packets and that they're lost before being
2969processed by the network driver. Rx-Drp indicates that some received packets
2970were lost in the network stack because the application doesn't process them
2971fast enough. This can happen during some attacks as well. Tx-Drp means that
2972the output queues were full and packets had to be dropped. When using TCP it
Dan Lloyd8e48b872016-07-01 21:01:18 -04002973should be very rare, but will possibly indicate a saturated outgoing link.
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002974
2975
297613. Security considerations
2977---------------------------
2978
2979HAProxy is designed to run with very limited privileges. The standard way to
2980use it is to isolate it into a chroot jail and to drop its privileges to a
2981non-root user without any permissions inside this jail so that if any future
2982vulnerability were to be discovered, its compromise would not affect the rest
2983of the system.
2984
Dan Lloyd8e48b872016-07-01 21:01:18 -04002985In order to perform a chroot, it first needs to be started as a root user. It is
Willy Tarreau2212e6a2015-10-13 14:40:55 +02002986pointless to build hand-made chroots to start the process there, these ones are
2987painful to build, are never properly maintained and always contain way more
2988bugs than the main file-system. And in case of compromise, the intruder can use
2989the purposely built file-system. Unfortunately many administrators confuse
2990"start as root" and "run as root", resulting in the uid change to be done prior
2991to starting haproxy, and reducing the effective security restrictions.
2992
2993HAProxy will need to be started as root in order to :
2994 - adjust the file descriptor limits
2995 - bind to privileged port numbers
2996 - bind to a specific network interface
2997 - transparently listen to a foreign address
2998 - isolate itself inside the chroot jail
2999 - drop to another non-privileged UID
3000
3001HAProxy may require to be run as root in order to :
3002 - bind to an interface for outgoing connections
3003 - bind to privileged source ports for outgoing connections
Dan Lloyd8e48b872016-07-01 21:01:18 -04003004 - transparently bind to a foreign address for outgoing connections
Willy Tarreau2212e6a2015-10-13 14:40:55 +02003005
3006Most users will never need the "run as root" case. But the "start as root"
3007covers most usages.
3008
3009A safe configuration will have :
3010
3011 - a chroot statement pointing to an empty location without any access
3012 permissions. This can be prepared this way on the UNIX command line :
3013
3014 # mkdir /var/empty && chmod 0 /var/empty || echo "Failed"
3015
3016 and referenced like this in the HAProxy configuration's global section :
3017
3018 chroot /var/empty
3019
3020 - both a uid/user and gid/group statements in the global section :
3021
3022 user haproxy
3023 group haproxy
3024
3025 - a stats socket whose mode, uid and gid are set to match the user and/or
3026 group allowed to access the CLI so that nobody may access it :
3027
3028 stats socket /var/run/haproxy.stat uid hatop gid hatop mode 600
3029